How to use Terraform or OpenTofu to Manage Datadog

Datadog, a leading monitoring and analytics platform, helps teams implement monitoring for various aspects of their organization whether it is cloud infrastructure, application performance monitoring, or general incident management. Leveraging infrastructure as code (IaC) practices with tools like Terraform or OpenTofu makes the management of Datadog easier to scale.

The Datadog Terraform Provider seamlessly integrates Datadog's monitoring capabilities into your infrastructure provisioning process, allowing you to define and manage monitoring resources alongside your infrastructure code. In this blog, we'll explore how to leverage the Datadog Terraform Provider to streamline your monitoring setup as well as the integrations that Scalr has with Datadog to assist in monitoring your Terraform operations.

Why Use the Datadog Terraform Provider?

Before diving into the implementation details, let's briefly highlight the benefits of using the Datadog Terraform Provider:

1. Infrastructure as Code (IaC): By defining monitoring resources in Terraform configuration files, you treat monitoring infrastructure as part of your overall infrastructure codebase. This promotes consistency, reproducibility, and version control.
2. Simplified Workflow: With Terraform, you can automate the provisioning, updating, and deletion of monitoring resources, reducing manual intervention and potential errors.
3. Scalability: As your infrastructure grows, managing monitoring resources manually becomes cumbersome. Terraform's declarative syntax allows you to scale your monitoring setup effortlessly.
4. Integration: Datadog's Terraform Provider seamlessly integrates with other Terraform-supported providers, enabling you to incorporate monitoring into your existing infrastructure provisioning workflows, like an AWS account.

Getting Started

To start using the Datadog Terraform or OpenTofu Provider, follow these steps:

1. Install Terraform or OpenTofu: If you haven't already, download and install Terraform from the official website (https://www.terraform.io/ or https://opentofu.org/).
2. A Datadog account, which can be created here: https://www.datadoghq.com/
3. Configure your Datadog API Key: Obtain your Datadog API key from the Datadog dashboard. This key will be used by Terraform to authenticate with Datadog.
4. Configure Datadog App Key: Obtain your Datadog App key from the Datadog dashboard. This key will be used by Terraform to authenticate with Datadog.

Using the Datadog Provider: Examples

Define Required Providers and Provider Configuration

Start by configuring the Datadog Terraform provider in your Terraform code. Open your Terraform configuration file (commonly named main.tf) and add the following block:

terraform {
  required_providers {
    datadog = {
      source = "datadog/datadog"
    }
  }
}

provider "datadog" {
   api_key    = "your-datadog-api-key"
   app_key   = "your-datadog-app-key"
}

Replace "your-datadog-api-key" with your Datadog API key and "your-datadog-app-key" with the application keys you generated earlier. The latest version of the official Datadog Terraform documentation can be found in the registry here.

Example: Create a Datadog Monitor

Here's a simple example demonstrating how to create a basic monitor using the Datadog Terraform Provider:

resource "datadog_monitor" "example" {
  name               = "Scalr-Example"
  type               = "metric alert"
  message            = "Alert! Please check the monitor"

  query = "sum(last_1h):sum:system.load.5{host:host0} > 2.0""

  monitor_thresholds {
    warning  = 2
    critical = 4
  }
}

In this example, we define a monitor that triggers when the 5-minute load average on a host named "host0" exceeds 2.0.

Example: Create Downtime

Here's a simple example demonstrating how to set downtime in Datadog using the Datadog Terraform Provider:

resource "datadog_downtime" "example" {
  scope = ["*"]
  start = 1363105000
  end   = 1363234500

  recurrence {
    type   = "days"
    period = 1
  }
}

Example: Pull a Monitor Datasource

In this example, we’ll use the datadog_monitor datasource to pull details about the monitor that was created in a previous step:

data "datadog_monitor" "example" {
  name_filter         = "Scalr-Example"
}

This will return details about the monitor such as the message that the monitor has set as well as the thresholds.

Execute Terraform or Tofu Commands

Once you have created your Terraform or Tofu code for the Datadog resources or datasources, you can then execute it with the following commands:

For Terraform:

terraform init
terraform plan
terraform apply

For OpenTofu:

tofu init
tofu plan
tofu apply

Terraform or Tofu will initialize the Datadog provider and apply the changes to your Datadog environment. Upon a successful Terraform apply, the state file will be created.

Best Practices and Advanced Usage

Variables and Dynamic Configurations

Utilize Terraform variables to make your configurations more dynamic. Instead of hardcoding values, use variables to create reusable and flexible scripts.

provider "datadog" {
   api_key    = "var.api-key"
   app_key   = "var.app-key"
}

These variables can be defined as environment variables to make it more dynamic and avoid having secrets in the code.

For example, you can specify the following before running the Terraform init to specific the Datadog API key:

export TF_VAR_api-key=<your-api-key>

Remote State Management

Consider using remote state management to store your Terraform state files securely. Services like Scalr or AWS S3 can be configured as remote backends to store state files. Here is an example of connecting to Scalr:

terraform {
  backend "remote" {
    hostname = "<account-name>.scalr.io"
    organization = "<scalr-environment-name>"

    workspaces {
      name = "<workspace-name>"
    }
  }
}

Find out more about remote state management here.

Provider Summary

This guide covered the basic steps to configure the Terraform Datadog provider. As you explore further, consider exploring additional Datadog resources supported by the Datadog provider, such as roles, logs, and various integrations with clouds like AWS. The library.tf documentation for the Terraform Datadog provider is a valuable resource for in-depth information and examples.

By integrating Datadog into your Terraform workflows, you're not just managing infrastructure – you're managing monitoring with the efficiency and scalability that infrastructure as code brings.

Scalr’s Integration with Datadog

Scalr, a Terraform Automation and Collaboration platform, provides a best-in-class integration with Datadog. Scalr is featured in the Datadog catalog, making it easier than ever to integrate the two products. Below, we'll explore the seamless integration process and benefits of integrating Scalr with Datadog.

Events

The Scalr to Datadog integration for events can stream event details for Terraform and OpenTofu runs executed in Scalr. Datadog users can build reports based on the source of the run, whether it was from the Terraform CLI, a VCS provider like Github, or manually executed through the UI. Users can also track the result of the run, the execution time, and much more.

‍

‍

The events are sent through the Datadog API integration, see the official documentation here.

Metrics

Scalr will send metrics to Datadog for in-depth analysis and reporting such as queued runs, queue state, the number of environments, and workspace count. These metrics are visualized in their out-of-the-box dashboard to help correlate deployments with other infrastructure changes and to track trends within your pipeline. The metrics functionality is an agent-based integration, which means you must use the Datadog agent. See how to install the Datadog agent and enable the Scalr integration here. 

Audit Logs

Scalr sends all of its audit logs to Datadog for further analysis. Audit logs allow you to get insights into all actions taken, who performed the action, how it was done, and more. The audit log feature can use the same Datadog connection that is used for events or a new one can be created.

For example, you may want to know how and when a Terraform run was discarded in the Scalr pipeline. Scalr will send the following data to Datadog:

{
	"id": "AgAAAY2x_1pxzx914QAAAAAAAAAYAAAAAEFZMnhfMXRKQUFDY1RCNUtZb3FoTlFBQQAAACQ12345656E4ZGIxZmYtNWE3MS00MTU3LWJlMTctODhmMTZhYTU5Nzhl",
	"content": {
		"timestamp": "2024-02-16T12:58:38.577Z",
		"tags": [
			"scalr-workspace-name:demo-ws",
			"scalr-environment-name:cs-m",
			"scalr-workspace:ws-v0o370ouv5kbjmk9h",
			"scalr-user-email:docs@scalr.com",
			"scalr-action:discard-run",
			"scalr-environment:org-sscctbkdgkdr123",
			"source:scalr",
			"datadog.submission_auth:private_api_key"
		],
		"host": "docs.scalr.io",
		"service": "audit-log",
		"attributes": {
			"actor": {
				"id": "user-stp8qjepev3a123",
				"access-token": {
					"type": "session",
					"token": "...B6wifA"
				},
				"type": "user",
				"email": "docs@scalr.com"
			},
			"request": {
				"ip-address": "69.206.111.123",
				"action": "discard-run",
				"id": "a3a473d725c730382558c25d52bd1234",
				"source": "ui",
				"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
			},
			"hostname": "docs.scalr.io",
			"service": "audit-log",
			"outcome": {
				"result": "SUCCESS",
				"status-code": 202
			},
			"target": {
				"display-name": "run-v0o370pcff0e53123",
				"context": {
					"environment": {
						"display-name": "CS-M",
						"id": "org-sscctbkdgkdrqg0"
					},
					"workspace": {
						"display-name": "demo-ws",
						"id": "ws-v0o370ouv5kbjmk9h"
					},
					"account": {
						"display-name": "docs",
						"id": "acc-sscctbisjkl3123"
					}
				},
				"id": "run-v0o370pcff0e53",
				"type": "runs"
			},
			"timestamp": "2024-02-16T12:58:38.577846"
		}
	}
}

Summary

Datadog provides many capabilities to take your Terraform or OpenTofu operations to the next level. Whether you are using the Datadog provider to manage Datadog itself or using the Scalr integration with Datadog to monitor your Terraform and Tofu operations, Datadog is at the forefront of helping users scale.

‍