Terraform and OpenTofu Pull Request Comments The Right Way

This blog will discuss the various options for automating pull request (PR) comments with Terraform and OpenTofu and, more specifically, the best practices for implementing them. PR comments are not just annotations on lines of code; they are a crucial mechanism for communication and collaboration among developers, reviewers, and stakeholders. PR comments allow developers to operate within their standard context without switching between tools to gather feedback. 

To start, we'll review the overall GitOps flow, which is the reason PR comments exist, and then talk about the various implementations of PR comments in various Terraform automation and collaboration tools (Scalr, HCP Terraform or TFC, and Atlantis).

How do PR comments work?

Let’s review a standard workflow that many Terraform and OpenTofu users are accustomed to: the GitOps workflow. In the GitOps workflow, there are a couple of standard steps:

1. Terraform code is created in a main branch of a VCS provider, such as a GitHub repository. (This same process applies to a Gitlab merge request and Azure DevOps pull request)
2. A developer checks the code into a different branch, changes the Terraform configuration files, and then opens a new pull request against the main branch.
3. Upon opening the pull request, a Terraform automation and collaboration tool (TACO), such as Scalr, Terraform Cloud (TFC), or Atlantis, will execute a speculative/dry run. Depending on the tool you are using, this run will go through the Terraform plan step and check other things, like OPA policies.
4. After the Terraform run has finished, the results are sent back to the VCS provider as PR comments or status checks.
5. Developers will review the comments and decide whether or not to merge the proposed changes; if merged, a Terraform apply will be executed to make the infrastructure changes.
6. The above process will repeat after every new pull request as long as the VCS webhook is linked to the Terraform automation tool.

Note: The flows differ slightly depending on the tool used. For example, Atlantis utilizes the apply before the merge method, whereas Scalr and TFC utilize the merge before the apply. This is a great debate that we will not discuss in this blog.

Solutions for PR Comments

The above shows the general flow of how GitOps and pull request comments work, but there are many different ways to implement the solution for PR comments.

Atlantis:

PR comments are a core feature that draws users to Atlantis. Atlantis provides the complete plan output to the VCS provider, making it rich with information but sometimes overwhelming and noisy.

The other main issue in this scenario is that the VCS providers have comment limits. As any Terraform developer knows, plan outputs can be very lengthy, so with Atlantis, you will not only be ingesting a ton of information but also not even see it all due to comment limits.

Terraform Cloud:

Terraform Cloud only supplies general information about the Terraform plan through a status check but does not report any of the actual Terraform plan details. The users will see if the plan is successful, how many resources will be added/changed/destroyed, and if the run passes the OPA policy checks. For any further information, like the error of a plan output or to see the Terraform changes, the developer will have to log into Terraform Cloud, which then slows down the development process because they are switching contexts.

Scalr:

The Scalr goal with PR comments is to make them as least noisy as possible when runs are successful and as noisy as possible when the Terraform run fails. Just like with standard monitoring solutions, if you constantly receive too much information or alerts, you will eventually ignore them, and they lose their purpose.

To ensure the comments are helpful, Scalr takes a hybrid approach to PR comments by only posting the Terraform plan output when errors occur. Developers will receive the following flags after a plan is executed depending on the result:

• Successful—The plan was successful; there were no OPA failures or destructive changes, so there is nothing to review further.
• Warning - There are destructive changes that the developer needs to be aware of.
• Important - There are OPA policy failures that the developer needs to be aware of.
• Error - The Terraform or OpenTofu run failed, and the error logs were sent back to the pull request comment.

Scalr also has a summary view or per workspace view in the comments. Many developers have a single repository linked to multiple workspaces, resulting in many runs kicking off based on a pull request. Having to review every Terraform run is massively time-consuming, which is why Scalr will report a summary of all the changes across all runs in a single place:

Based on the summary, you can view the per-workspace report, which will break down each run in that workspace, whether it was successful or not, and provide the necessary detail to decide whether to merge the pull request. Here are the various scenarios and information provided based on the Terraform changes:

Scenario 1: The Terraform plan is successful

In this case, Scalr will report back a successful run with general information about the run and workspace:

Scenario 2: The plan is successful, but there are OPA policy violations

Scenario 3: The plan is successful, but there are destructive changes

Scenario 4: The Terraform plan has failed with an error; the Terraform error logs are sent back to the pull request comment

Summary

The Scalr pull request comment feature aims to improve the developer experience by focusing attention only on the areas that require it. If a Terraform plan is successful, why send all of the plan details back with the same level of detail that you get with an error? The proper level of detail reduces noise and ensures that developers will focus on critical issues when they arise. PR comments allow multiple team members to review the Terraform configuration changes without everyone needing access to the tool performing the Terraform deployment. Implementing the best practices outlined above is imperative to the developer experience to ensure the team can push out Terraform changes as quickly as possible with minimal errors.