Back to blog

How to send data from IoT device to AWS cloud?

In one of our previous blog posts, we described how to establish a Wifi connection for ESP32 using Micropython. Today, we will use this connection to send out some sensor data to the AWS IoT Core cloud service.

But first, let’s introduce some of the terms we will use in this post:

“AWS IoT Core is a managed cloud service that lets connected devices easily and securely interact with cloud applications and other devices.”
It helps you easily send data from all devices to one endpoint, and use the incoming messages to trigger AWS Lambda functions, to e.g. send the uploaded values to the database.

AWS IoT Core uses MQTT – a lightweight publish/subscribe messaging protocol. It is useful for use with low power sensors but is applicable to many scenarios. For more info see http://mqtt.org/

The MQTT protocol is based on the principle of publishing messages and subscribing to topics, or “pub/sub”. Multiple clients connect to a broker and subscribe to topics that they are interested in. Clients also connect to the broker and publish messages to topics. Many clients may subscribe to the same topics and do with the information as they please. The broker and MQTT act as a simple, common interface for everything to connect to.

AWS IoT Core Nomenclature

Thing – represents a specific device/logic entity. Things can have parameters (name, version, etc), attributes (model, wattage, etc.), and can be a member of a specified Thing Type.
Example:

Topic – Logical structure holding data. Can be subscribed to or published to. When a Thing subscribes to a topic, all new data published to the topic will be redirected to all subscribed Things. Topics store only the newest piece of information. Topics have a hierarchical structure similar to directories in a file system.
Example topics:
/office/room_7/temperature
/office/room_7/humidity
/office/room_5/temperature
Note: Topics beginning with ‘$’ must not be created, as they are used internally by AWS.

Certificate – registers specific Thing in the AWS IoT, creating a unique entity. A Thing Certificate with Private and Public Key should be created for every device. AWS Root CA, Thing Certificate, and Private key are being stored on an IoT device. Each Thing certificate can have numerous Policies assigned.

Policy – specifies what actions can be performed by devices affected by the policies. The policy can for example, only permit the device to Publish data to the broker, but not to Subscribe to any of the topics (reducing possible data leakage). Policies can be assigned to Certificates (thus, specific Things).

Tags – can be used to categorize. There is no one specific way of adding Tags. They can refer to purpose, owner, environment, or billing group.

Device Shadow – a virtual version of the device that includes the device’s latest state so that applications or other devices can read messages and interact with the device. The Device Shadow persists the last reported state and desired future state of each device even when the device is offline.

Rules Engine – evaluates inbound messages published into AWS IoT Core and transforms and delivers them to another device or a cloud service, based on business rules you define. A rule can apply to data from one or many devices, and it can take one or many actions in parallel.
The Rules Engine can also route messages to AWS endpoints including AWS IoT Analytics, AWS IoT Events, AWS Lambda, Amazon Kinesis, Amazon S3, Amazon DynamoDB, Amazon CloudWatch, Amazon Simple Notification Service (SNS), Amazon Simple Queue Service (SQS), Amazon Elasticsearch Service, and AWS Step Functions. External endpoints can be reached using AWS Lambda, Amazon Kinesis, Amazon SNS, and Rules Engine’s native HTTP action.

Now knowing all of this we can create a minimum setup for sending and receiving MQTT messages to AWS IoT Core:

All of the required steps can be done in a few ways – an AWS Online Console will be used in this example, but we also do this through Terraform. See our previous blogpost about Terraform (with an example on GitHub!) here: https://wizzdev.pl/blog/embedding-terraform-in-custom-application/

Creating a Thing

In AWS IoT Console page select Manage->Things and click “Register a thing” (or “Create” if you have already some Things registered),
Next, choose “Create a single thing”,
Input your Thing Name (you can also specify a Thing Type and a Group it belongs to at this point), and click “Next”.

Congratulations! You have now registered your first Thing to AWS IoT Core. But wait for a second, that does not mean that you can publish anything yet!
Every IoT device must hold a certificate which will allow them to connect to an AWS IoT Core and publish data. What a certificate can/cannot do is defined through Policies assigned to it.

Creating a Certificate

In AWS IoT Console page select Secure>Certificates and click “Create a certificate” (or “Create” if you have already some Certificates available),
Next, choose “Create certificate” (One-click certificate creation),
After the certificate is generated, remember to click “Activate” and to download the certificate with public and private keys, as the keys will not be accessible after closing this page! You can also download AWS root CA from this page. The private key, the device certificate, and the root CA certificate will be later needed to transfer to the device!

Creating a Policy

Now let’s create a policy that will be assigned to the Thing’s certificate. This policy should allow to:

connect,
subscribe,
publish,
receive.

To do that:

In AWS IoT Console page select Secure>Policies and click “Create a policy” (or “Create” if you have already some Policies available),
Input your policy name, and in the Add statements field select “Advanced mode”,
In the new input field paste:

Confirm the creation of your policy by clicking “Create”.

Assigning a Policy to Certificate

To assign the policy to your certificate:

In AWS IoT Console page select Secure>Certificates,
Select your certificate and from Actions menu select “Attach policy”,
Choose a policy to attach and click the “Attach” button.

Attaching a Certificate to a Thing

The certificate is ready but not yet connected with any particular device. Let’s fix it:

In AWS IoT Console page select Secure>Certificates,
In the box for the certificate you created, click on “…” to open a drop-down menu, and then choose “Attach thing”,
Next, select the checkbox for the thing you registered, and choose “Attach”.

Micropython program on ESP

Note: All examples presented here assumes that a WiFi or other internet connection has been already established.
Before starting – upload all required certificates to the ESP32 device. E.g. to a directory

/certs/.

In this example the following certificate files are used:

At the beginning import MQTTClient from umqtt.simple (which is one of Micropython standard libraries).

from umqtt.simple import MQTTClient

Before connecting to the MQTT broker load device certificate and private key. They will be needed to establish the connection:

Next, check your AWS endpoint address. It can be retrieved from the AWS IoT Core console from the Settings tab (the Amazon region can be different, depending on the region you use):

To establish a connection with the MQTT broker replace ‘AWS_ENDPOINT’ with your actual endpoint. Key and cert are the loaded private key and device certificate, and device_name is the name of the previously created IoT Thing.

Now to finally sending the data. Your data should be a formatted string (in our case here – JSON formatted).
Topic to which the data will be sent will have a name ‘topic/data’:

Remember to close the connection after sending the data packets:

mqtt_client.disconnect()

Testing

To check whether the data is received correctly on AWS endpoint you can open the Test tab in AWS IoT Core console and subscribe to any topic. A wildcard ‘#’ can be used to subscribe to all the topics.

Now all upcoming MQTT messages can be observed in the field below!

If you want to explore more and start gathering data from your ESP32-based IoT sensor, please check out our GitHub repository: https://github.com/wizzdev-pl/iot-starter
It contains a sample MircoPython code for ESP32, with clear step-by-step instructions. Moreover, you can use our predefined Terraform template to configure all needed AWS services in one run!