Protocol Buffers, also known as Protobuf, is a tool developed by Google that offers a platform-neutral approach for serializing structured data. It is similar to JSON but smaller, faster, and can automatically generate bindings in your preferred programming language. AWS IoT Core is a managed service that enables you to connect billions of IoT devices and route trillions of messages to AWS services, allowing for seamless scaling of your application to millions of devices. By integrating AWS IoT Core with Protobuf, you can take advantage of Protobuf’s efficient data serialization protocol and automated code binding generation. This week’s article covers using Protobuf with AWS IoT Core and explores the benefits of doing so, such as smaller message size and better performance.
Protobuf’s code generator provides a significant benefit in the form of simplified and more secure program creation. The communication between the various parts of your application can be described in a schema. The schema is read by a code generator (such as protoc) that then creates the necessary code for you to use to encode and decode data. The code generated by Protobuf’s code generators is reliable since it has been used extensively and is well-maintained. By generating code automatically, programmers are spared the tedious task of creating encoding and decoding functions, and cross-language support is ensured.
With the recent launch of AWS IoT Core’s Rule Engine support for the Protocol Buffer messaging format, you can have a producer application written in C running on your device and an AWS Lambda function consumer written in Python, all using generated bindings. Other advantages of using Protocol Buffers instead of JSON with AWS IoT Core include:
- Schema and validation: Protobuf enforces the schema for both the sender and receiver, ensuring proper integration and eliminating bugs. Since the auto-generated code encodes and decodes messages, errors are reduced.
- Adaptability: The schema is mutable, allowing for changes in message content while maintaining backward and forward compatibility.
- Bandwidth optimization: Protobuf messages are smaller than JSON messages, as they do not include headers, resulting in better device autonomy and less bandwidth usage. Recent research on messaging protocols and serialization formats has shown that Protobuf-formatted messages can be up to 10 times smaller than their equivalent JSON-formatted messages, meaning fewer bytes are transmitted to transmit the same content.
- Efficient decoding: Decoding Protobuf messages is more efficient than decoding JSON, allowing recipient functions to run more quickly. A benchmark conducted by Auth0 showed that Protobuf can be up to 6 times more performant than JSON for equivalent message payloads.
How to Use Protocol Buffers with AWS IoT Core
Protobuf can be used in various ways with AWS IoT Core. The easiest method is to publish the message as a binary payload and let the recipient applications decipher it. AWS IoT Core Rules Engine already supports this and works for any binary payload, including Protobuf.
To utilize a more advanced method, you can ingest Protobuf-encoded data from IoT devices, then decode the data into JSON format using AWS IoT Core’s Rules Engine’s decode function. After doing so, you can route the data to a variety of AWS and third-party services. Here are the steps to follow:
-
Download the sample code and install Python requirements.
- https://github.com/aws-samples/aws-iotcore-protobuf-sample
- pip install -r requirements.txt
-
Configure your IOT_ENDPOINT and AWS_REGION environment variables.
- export AWS_REGION=<AWS_REGION>
- export IOT_ENDPOINT=$(aws iot describe-endpoint –endpoint-type iot:Data-ATS –query endpointAddress –region $AWS_REGION –output text)
-
Generate Python bindings and message descriptors using protoc.
- protoc –python_out=. msg.proto
- protoc -o filedescriptor.desc msg.proto
- filedescriptor.desc msg_pb2.py
-
Run the simulated device using Python and the Protobuf generated code bindings.
- python3 simulate_device.py
- Create AWS Resources with AWS CloudFormation and then upload the Protobuf file descriptor.
- Take a look at the AWS IoT Rule that matches, filters, and republishes Protobuf messages as JSON
- Confirm that the transformed messages are being republished.
Once you complete these steps, you’ll be able to convert Protobuf messages from your device to JSON using AWS IoT Core Rules Engine and trigger a Lambda function. The Lambda function will perform an intelligent action based on the message content. For instance, if your device’s temperature exceeds a specific threshold, the Lambda function can send an SNS notification or write data to a DynamoDB table.
This article explores the advantages of utilizing Protocol Buffers in conjunction with AWS IoT Core. These benefits include reduced message sizes, improved performance, schema and validation, adaptability, and optimized bandwidth. Additionally, we’ve covered the process of using Protocol Buffers with AWS IoT Core, including the steps required to convert Protobuf messages from your device to JSON with the AWS IoT Core Rules Engine and how to trigger a Lambda function.
