Embedding Terraform in custom application

Terraform – what? how? what’s it good for?

While we were developing a solution that uses Amazon Web Services, we had quickly found out that relying on their web console can be a nightmare sometimes. We spent much time on operations like code updating, testing, and configuring stuff.

The first way of automating that process is to use their command-line client with some bash scripts. However, this kind of automation introduced other problems – scripts were hard to maintain and get outdated pretty fast. 

That’s why we decided to give Infrastructure as Code a go.

For our first endeavours, we chose to use Terraform by HashiCorp, which despite being a relatively fresh (and not yet mature, as some like to point out), has gained a lot of popularity over the last few years.

Terraform – an Infrastructure As Code (IAC) tool, enables applying changes to your cloud infrastructure without manual UI clicking or sending separate AWS CLI commands. The cloud structure is described in .tf files using HCL syntax – a declarative language that aims to make the structure easy to maintain and modify. But, more importantly, Terraform stores files representing the state of the system. As you change your configurations, “Terraform builds an execution plan that only modifies what is necessary to reach your desired state.”

It’s a perfect tool if you are dealing with an often-changing infrastructure, it helps you keep up with changes made by other developers, (as long as you share the same state file).

Why is it suitable for our application?

We are currently developing a project for managing a network of sensors, that are uploading their measurements to an online database for further visualization/analysis  – a classic Internet of Things example.  For this, we are using AWS IoT Core with Lambda and  DynamoDB that have to be configured and maintained. 

We have split our application into separate Terraform environments (that are separate configuration files). The most common use case is to describe your desired infrastructure in Terraform and deploy it from scratch. One of our Terraform environments is used entirely in this manner – we need to upload Lambda code, define its triggers, and hook its output to DynamoDB. This will be applied only once, at the very moment of service creation.  

But in the case of IoT application, we wanted to be able to add/register new devices to existing infrastructure easily, as well as delete an existing device without any complications. In this case, a Terraform configuration must be modified, in order to introduce changes to the AWS. We could manually modify the .tf files but that is not as pleasant as one would ideally want. So we decided to store all user-defined Things (IoT devices) in a JSON file which can be easily edited (either by hand or script) and then read by Terraform.

And now you have it – just one “terraform apply” to set up the architecture, then adding new Things to JSON and a second “Terraform apply” to register them in AWS IoT Core, create all required certificates, etc.

What if we want more?

Terraform is already automating a lot (a whole lot!) of things when it comes to setting up and managing your AWS infrastructure.

But it still has some limitations. Say we want to decentralize the system, and manage configuration from different machines/instances of our application. The native Terraform’s state locking solution would not work in that case. 

We’ve extended the process of running terraform. Before, it had only two steps: planning and applying. Right now, it downloads the last version of state file from the S3 bucket and after executing terraform, if succeed, uploads the modified version back to S3.

To deal with it we developed a custom Python wrapper, that handles all Terraform actions underneath, leaving a very user-friendly interface, which when crafted for a specified purpose (e.g. adding/removing new devices) can go down to just one command like: “add_device new_device_name”

Such tool will surely simplify a lot for our team during development, but because it is so easy to use, so powerful (and yet so limited to only specified tasks) why not give it to the end-users? 

What we have in mind, is a standalone application that will enable the non-programmer end-user to prepare an AWS infrastructure from scratch with minimum (or no) effort. All he has to do is own a valid AWS account with appropriate permissions to services like IoT Core, Lambda and DynamoDB, and a bunch of devices preprogrammed with firmware developed by our team.

We even developed a GUI for it!

Just a few technical notes

Sometimes Terraform prompts for user input while executing, and because we are using it under the hood of another application, and do not want such distractions, this prompts can be disabled by using -input=false flag.

Also to disable so many outputs from Terraform commands we can use:
TF_IN_AUTOMATION flag (set it to not empty).

We decided also to handle Terraform installation within the application itself. This turned out to be a lot more complicated than we initially thought.
As long as the whole process goes down to downloading .zip and unpacking it, finding the right file to download is not that easy.
By the fact that you cannot change the configuration, that has been modified with a newer version of Terraform than the one you’re currently using, we were encouraged to always use the latest version, and download updates as they appear. But unfortunately, Terraform downloads site does not have a thing as a redirection page for something like “latest” – you have to know the exact version name to download the file. Lucky for us – Terraform has some internal API that will inform about the new available version after calling terraform version command. Sadly this API is not documented, so what we ended up doing is getting new version value directly from the “terraform version” output. 



Images credits:
Banner vector created by upklyak – www.freepik.com
Designed by vectorpocket / Freepik