Terraform Remote-Exec: A Concise Guide
Master Terraform’s remote-exec provisioner: clear setup steps, SSH tips, examples, and best-practice alternatives for post-provision automation.
This article is part of our Terraform Provisioners guide.
Introduction
Terraform's remote-exec provisioner enables infrastructure teams to execute commands directly on newly provisioned resources, bridging the gap between infrastructure creation and configuration. While HashiCorp recommends using provisioners as a "last resort," there are legitimate scenarios where remote-exec provides the flexibility needed for specific configuration tasks.
This guide provides a comprehensive overview of remote-exec provisioners, including practical examples, security considerations, and best practices for managing them at enterprise scale. We'll also explore how modern infrastructure management platforms can help teams implement these patterns more effectively.
Understanding Remote-Exec Provisioners
The remote-exec provisioner invokes scripts or commands on a remote resource after it has been created. Unlike local-exec (which runs commands on the machine executing Terraform), remote-exec connects to and executes commands directly on the target resource.
When to Use Remote-Exec
Appropriate use cases:
- Simple post-deployment configuration tasks
- Software installation on newly created servers
- Service initialization and bootstrapping
- Testing and validation of deployed resources
- One-time setup operations that can't be handled by cloud-init
When to avoid remote-exec:
- Complex configuration management (use specialized tools)
- Ongoing maintenance tasks (not tracked in Terraform state)
- When cloud-init or user data scripts are sufficient
- For changes that should be detected by future Terraform runs
The key limitation is that provisioners aren't declarative, aren't tracked in state, can't detect drift, and aren't idempotent by default.
Basic Configuration and Syntax
Connection Block Structure
Every remote-exec provisioner requires a connection block that defines how Terraform connects to the remote resource:
resource "aws_instance" "web" {
ami = "ami-0c55b159cbfafe1f0"
instance_type = "t2.micro"
key_name = aws_key_pair.deployer.key_name
connection {
type = "ssh"
user = "ubuntu"
private_key = file("${path.module}/private_key.pem")
host = self.public_ip
timeout = "5m"
}
provisioner "remote-exec" {
inline = [
"sudo apt-get update",
"sudo apt-get install -y nginx",
"sudo systemctl start nginx",
"sudo systemctl enable nginx"
]
}
}Execution Methods
Remote-exec supports three execution methods:
1. Inline commands:
provisioner "remote-exec" {
inline = [
"command1",
"command2",
"command3"
]
}2. Single script file:
provisioner "remote-exec" {
script = "path/to/setup.sh"
}3. Multiple script files:
provisioner "remote-exec" {
scripts = [
"path/to/first_script.sh",
"path/to/second_script.sh"
]
}Failure Handling
Control how Terraform handles provisioner failures:
provisioner "remote-exec" {
inline = [
"command1",
"command2"
]
on_failure = "continue" # or "fail" (default)
when = "create" # or "destroy"
}Connection Types: SSH and WinRM
SSH for Linux/Unix Systems
Basic SSH configuration:
connection {
type = "ssh"
user = "ec2-user"
private_key = file("~/.ssh/id_rsa")
host = self.public_ip
port = 22
timeout = "2m"
}SSH with bastion host:
connection {
type = "ssh"
user = "ubuntu"
private_key = file("~/.ssh/id_rsa")
host = self.private_ip
bastion_host = "bastion.example.com"
bastion_user = "bastion-user"
bastion_private_key = file("~/.ssh/bastion_key")
}WinRM for Windows Systems
Basic WinRM configuration:
connection {
type = "winrm"
user = "Administrator"
password = var.admin_password
host = self.public_ip
port = 5985
timeout = "10m"
}Windows instance with WinRM setup:
resource "aws_instance" "windows" {
ami = "ami-windows-server-2019"
instance_type = "t2.medium"
user_data = <<EOF
<powershell>
Enable-PSRemoting -Force
winrm quickconfig -q
winrm set winrm/config/service '@{AllowUnencrypted="true"}'
winrm set winrm/config/service/auth '@{Basic="true"}'
netsh advfirewall firewall add rule name="WinRM-HTTP" dir=in localport=5985 protocol=TCP action=allow
</powershell>
EOF
connection {
type = "winrm"
user = "Administrator"
password = var.admin_password
host = self.public_ip
port = 5985
timeout = "10m"
}
provisioner "remote-exec" {
inline = [
"powershell.exe -Command \"Install-WindowsFeature -Name Web-Server -IncludeManagementTools\"",
"powershell.exe -Command \"Add-Content -Path C:\\inetpub\\wwwroot\\index.html -Value '<h1>Hello from Terraform!</h1>'\""
]
}
}Practical Use Cases with Code Examples
Application Deployment
resource "aws_instance" "app_server" {
ami = "ami-0c55b159cbfafe1f0"
instance_type = "t2.micro"
key_name = aws_key_pair.deployer.key_name
connection {
type = "ssh"
user = "ubuntu"
private_key = file("~/.ssh/id_rsa")
host = self.public_ip
}
# Copy application files
provisioner "file" {
source = "app/"
destination = "/tmp/app"
}
# Install dependencies and deploy
provisioner "remote-exec" {
inline = [
"sudo apt-get update",
"sudo apt-get install -y nodejs npm nginx",
"cd /tmp/app && npm install",
"sudo cp -r /tmp/app /var/www/",
"sudo chown -R www-data:www-data /var/www/app",
"sudo systemctl start nginx",
"sudo systemctl enable nginx"
]
}
}Database Configuration
resource "aws_instance" "database" {
ami = "ami-0c55b159cbfafe1f0"
instance_type = "t2.small"
connection {
type = "ssh"
user = "ubuntu"
private_key = file("~/.ssh/id_rsa")
host = self.public_ip
}
provisioner "remote-exec" {
inline = [
"sudo apt-get update",
"sudo apt-get install -y postgresql postgresql-contrib",
"sudo systemctl start postgresql",
"sudo systemctl enable postgresql",
"sudo -u postgres createdb ${var.database_name}",
"sudo -u postgres psql -c \"CREATE USER ${var.db_user} WITH PASSWORD '${var.db_password}';\"",
"sudo -u postgres psql -c \"GRANT ALL PRIVILEGES ON DATABASE ${var.database_name} TO ${var.db_user};\""
]
}
}Cluster Node Configuration
resource "aws_instance" "k8s_worker" {
count = var.worker_count
ami = "ami-0c55b159cbfafe1f0"
instance_type = "t2.medium"
connection {
type = "ssh"
user = "ubuntu"
private_key = file("~/.ssh/id_rsa")
host = self.public_ip
}
provisioner "remote-exec" {
inline = [
"curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -",
"echo 'deb https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list",
"sudo apt-get update",
"sudo apt-get install -y kubelet kubeadm kubectl",
"sudo kubeadm join ${var.master_ip}:6443 --token ${var.join_token} --discovery-token-ca-cert-hash ${var.cert_hash}"
]
}
depends_on = [aws_instance.k8s_master]
}Security Best Practices
Credential Management
Use variables for sensitive data:
variable "admin_password" {
type = string
sensitive = true
description = "Administrator password for Windows instances"
}
connection {
type = "winrm"
user = "Administrator"
password = var.admin_password
host = self.public_ip
}Leverage external secret management:
data "aws_secretsmanager_secret_version" "db_creds" {
secret_id = "prod/database/credentials"
}
locals {
db_creds = jsondecode(data.aws_secretsmanager_secret_version.db_creds.secret_string)
}
provisioner "remote-exec" {
inline = [
"export DB_USER='${local.db_creds.username}'",
"export DB_PASS='${local.db_creds.password}'",
"bash /tmp/configure_db.sh"
]
}Network Security
Restrict access with security groups:
resource "aws_security_group" "provisioning" {
name = "terraform-provisioning"
description = "Temporary access for Terraform provisioning"
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["${data.external.my_ip.result.ip}/32"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
Name = "terraform-provisioning"
}
}Script Security
Make scripts idempotent and secure:
provisioner "remote-exec" {
inline = [
"set -euo pipefail", # Exit on error, undefined vars, pipe failures
"export DEBIAN_FRONTEND=noninteractive",
"if ! command -v nginx &> /dev/null; then",
" sudo apt-get update",
" sudo apt-get install -y nginx",
"fi",
"sudo systemctl enable nginx",
"sudo systemctl start nginx"
]
}Troubleshooting Common Issues
Connection Problems
SSH timeout errors:
connection {
type = "ssh"
user = "ubuntu"
private_key = file("~/.ssh/id_rsa")
host = self.public_ip
timeout = "10m" # Increase timeout
agent = false # Disable SSH agent
}Host key verification:
connection {
type = "ssh"
user = "ubuntu"
private_key = file("~/.ssh/id_rsa")
host = self.public_ip
host_key = null # Skip host key verification (use carefully)
}Script Execution Issues
Handle missing dependencies:
provisioner "remote-exec" {
inline = [
"which curl || sudo apt-get install -y curl",
"which jq || sudo apt-get install -y jq",
"curl -s https://api.example.com/status | jq '.health'"
]
}Wait for system readiness:
provisioner "remote-exec" {
inline = [
"until [ -f /var/lib/cloud/instance/boot-finished ]; do sleep 1; done",
"sudo apt-get update",
"sudo apt-get install -y nginx"
]
}Alternatives and When to Use Them
Cloud-Init for Initial Configuration
Instead of remote-exec for basic setup:
resource "aws_instance" "web" {
ami = "ami-0c55b159cbfafe1f0"
instance_type = "t2.micro"
user_data = <<-EOF
#!/bin/bash
apt-get update
apt-get install -y nginx
systemctl enable nginx
systemctl start nginx
echo "<h1>Hello World</h1>" > /var/www/html/index.html
EOF
}Configuration Management Tools
Ansible integration:
resource "null_resource" "ansible_playbook" {
provisioner "local-exec" {
command = "ansible-playbook -i '${aws_instance.web.public_ip},' --private-key=${var.private_key_path} playbook.yml"
}
depends_on = [aws_instance.web]
}Provider-Specific Solutions
AWS Systems Manager:
resource "aws_ssm_document" "install_nginx" {
name = "install-nginx"
document_type = "Command"
content = jsonencode({
schemaVersion = "2.2"
description = "Install nginx"
mainSteps = [{
action = "aws:runShellScript"
name = "installNginx"
inputs = {
runCommand = [
"apt-get update",
"apt-get install -y nginx",
"systemctl start nginx"
]
}
}]
})
}
resource "aws_ssm_association" "install_nginx" {
name = aws_ssm_document.install_nginx.name
targets {
key = "InstanceIds"
values = [aws_instance.web.id]
}
}Managing Remote-Exec at Scale
Challenges with Large Deployments
When managing infrastructure at enterprise scale, several challenges emerge with remote-exec provisioners:
- Concurrent connection limits - Most systems limit simultaneous SSH/WinRM connections
- Network reliability - Provisioners can fail due to transient network issues
- Credential management - Securely distributing and rotating access credentials
- Audit and compliance - Tracking what commands were executed where and when
- State management - Provisioners don't integrate with Terraform state for drift detection
Enterprise Solutions
Modern infrastructure management platforms address these challenges by providing:
- Centralized execution - Commands are executed from a secure, managed environment
- Credential injection - Secrets are securely injected at runtime without storage in configuration
- Execution logging - All provisioner activities are logged for audit purposes
- Policy enforcement - Governance policies can prevent risky provisioner configurations
- Scalable orchestration - Handle hundreds of concurrent provisioning operations
For organizations managing complex Terraform deployments, platforms like Scalr provide these capabilities while maintaining the flexibility of remote-exec when needed.
Summary and Recommendations
| Aspect | Recommendation | Notes |
|---|---|---|
| Use Cases | Last resort for simple, one-time configuration | Prefer cloud-init, user data, or configuration management tools |
| Connection Type | SSH for Linux, WinRM for Windows | Use key-based authentication over passwords |
| Script Organization | Use external scripts for complex logic | Inline commands for simple, short operations |
| Error Handling | Always include set -e in bash scripts |
Use on_failure = "continue" judiciously |
| Security | Store credentials in external secret management | Never hardcode passwords or keys |
| Network Access | Restrict to specific IPs via security groups | Use bastion hosts for private resources |
| Timeout Settings | Set realistic timeouts (5-10 minutes) | Consider long-running operations carefully |
| Idempotency | Make scripts idempotent with proper checks | Avoid destructive operations without guards |
| Scalability | Use orchestration platforms for large deployments | Consider centralized execution for enterprise use |
| Alternatives | Evaluate cloud-init, SSM, or config management first | Reserve remote-exec for specific requirements |
Key Takeaways
- Remote-exec is powerful but limited - It provides flexibility but lacks state management and drift detection
- Security is paramount - Always use secure authentication and restrict network access
- Idempotency matters - Scripts should be safe to run multiple times
- Consider alternatives first - Cloud-init, user data, and configuration management tools are often better choices
- Scale requires orchestration - Enterprise deployments benefit from centralized management platforms
For teams managing Terraform at scale, combining remote-exec with modern infrastructure management platforms provides the best of both worlds: the flexibility to handle edge cases while maintaining security, compliance, and operational excellence.
The remote-exec provisioner remains a valuable tool in the Terraform ecosystem when used appropriately. By following the best practices outlined in this guide and considering the broader context of your infrastructure management strategy, you can leverage remote-exec effectively while avoiding common pitfalls.
