ESP32 and AWS IoT Tutorial

Apr 18, 2020

|

Volodymyr Rudyi

In this blog post, I’m going to teach you basics of AWS IoT and using it to collect data from microcontrollers. I’m using ESP32 since it’s low-cost, energy-efficient and comes with some very interesting hardware features.

Required knowledge

To fully enjoy this blog post, you need to have some understanding of microcontrollers, Internet of Things and knowledge of the C programming language along with AWS experience.

Additionally, you should be familiar with Linux/Unix command line and the CMake build tool.

Required hardware and software

The setup is rather simple. We need a ESP32-based device, a computer, and a text editor/IDE of your choice.

In my case, I’m using the following combination of hardware and software:

  • M5Stack - a ESP32 development kit in a tiny box form factor
  • MacBook Pro 15 running Mac OS Catalina 10.15.1
  • Visual Studio Code with the C/C++ Extension installed

Before we begin

As you may know, there are several options when it comes to connecting a ESP32 device to the AWS IoT:

  • SDK-based. In this case, your program uses AWS IoT SDK and MQTT to connect to AWS IoT
  • Using Amazon version of FreeRTOS. It’s a FreeRTOS flavour maintained by AWS with some AWS-specific features, like device provisioning.

    In this blog post, we are going with the SDK-based route, mainly to demonstrate how easily it’s possible to use benefits of AWS IoT in existing projects. Besides it, ESP-IDF comes with the AWS IoT component which further simplifies connecting our device to the AWS IoT.

Because we are working with ESP32, we’ll be using ESP-IDF to create a firmware which connects to AWS. ESP-IDF(IoT Development Framework) is built on top of FreeRTOS with some additions, e.g. support of SMP(symmetric multiprocessing).

Setting up ESP-IDF

You need to follow the official documentation to make sure all needed tools are installed. Here is the short version for Mac OS X.

Install XCode Command Line Tools:

xcode-select --install

Install pip and pyserial:

sudo easy_install pip
pip install --user pyserial

Then install CMake and Ninja:

brew install cmake ninja 

Prepare a directory and install ESP-IDF:

mkdir ~/esp
cd ~/esp
git clone --recursive https://github.com/espressif/esp-idf.git
cd ~/esp/esp-idf
./install.sh

Add the following line to your .profile, .bash_profile_ or .zshrc:

export IDF_PATH=$HOME/esp/esp_idf
. $HOME/esp/esp-idf/export.sh

Finally, restart the terminal and run the idf.py command to see if everything works as expected. You should get an output like this:

Creating a project skeleton

The fastest way to create a skeleton project for ESP-IDF is to use one of the examples as a template. All examples are located in the $IDF_PATH/examples. We are going to use the “hello_world” project.

Let’s assume you want to create a project in your home directory. Then you’ll need to copy the example using the following command:

cp -r $IDF_PATH/examples/get-started/hello_world ~/esp-aws-iot
cd ~/esp-aws-iot

I propose to take a minute and make some changes to the project, so it’s not the “hello_world” anymore, but our very own “esp-aws-iot” project. To do so, update CMakeLists.txt in the project root with the following contents:

cmake_minimum_required(VERSION 3.5)

include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(esp-aws-iot)

The project name should be also updated in the Makefile:

PROJECT_NAME := esp-aws-iot

include $(IDF_PATH)/make/project.mk

Then let’s also rename “hello_world_main.c” to “main.c”:

mv main/hello\_world\_main.c main/main.c

Finally, we need to update main/CMakeLists.txt to reflect the renamed file:

idf_component_register(SRCS "main.c"
INCLUDE_DIRS "")

Now it’s time to configure our project:

idf.py menuconfig

You will see the configuration menu similar to the screenshot below:

For our initial firmware, don’t modify anything and just exit the configuration. A default configuration will be written to the sdkconfig file in the project root.

Execute the idf.py build in the root directory of your project to build it. It will take a while, so you can grab a coffee while it’s building.

Uploading the first firmware

Before flashing the firmware, you need to find out the serial port of the device. Plug the device into your PC/laptop and check to which port it’s attached. In my case, it’s /dev/cu.SLAB_USBtoUART

Once you now the port, the firmware can be uploaded to the device using this command:

idf.py -p <your-port> flash

The flashing takes several seconds to complete and produces an output like this:

To verify the ESP32 is running our program, use the idf.py -P <your-port> monitor. You will see chip information and a reboot countdown:

At this point we configured all the needed tools and made sure we can flash the device. Let’s do something more exciting.

Establishing an Internet Connection

To establish an Internet connection using ESP32 we need to perform three steps:

  1. Initialize ESP-IDF TCP/IP stack
  2. Initialize ESP-IDF Wi-Fi driver in a station mode
  3. Connect to a Wi-Fi network

For the sake of simplicity, we will be connecting to a Wi-Fi network set during the configuration/build of the application. This is the approach suggested in the “Getting Started” example for ESP-IDF Wi-Fi applications.

To create a custom configuration section, let’s add a file main/Kconfig.projbuild with the following contents:

 

menu "ESP AWS IoT Example Configuration"

config ESP_WIFI_SSID
string "WiFi SSID"
default "myssid"
help
SSID (network name) for the example to connect to.

config ESP_WIFI_PASSWORD
string "WiFi Password"
default "mypassword"
help
WiFi password (WPA or WPA2) for the example to use.

config ESP_MAXIMUM_RETRY
int "Maximum retry"
default 5
help
Set the Maximum retry to avoid station reconnecting to the AP unlimited when the AP is really inexistent.
endmenu


Then after running the idf.py menuconfig you will see a new configuration section(you may need to scroll down):

Values provided in this configuration section can be accessed in the source code by using corresponding definitions. For example:

printf("Configured Wi-Fi SSID is %s\n", CONFIG_ESP_WIFI_SSID);

So the original configuration name is prepended with “CONFIG_” and can be used anywhere in your source code.

Important: Besides configuring SSID and password, please also disable “WiFi NVS Flash” in the “Component -> Wi-Fi” section. We are not going to use it for now and it will make the code example simpler.

Our first version of the Wi-Fi application will be really simple:

  • Try to connect to a Wi-Fi network
  • After a successful connection print a corresponding message to the serial port
  • In a case if there were 5 failed attempts to connect print a corresponding error message to the serial port

We’ll need to have a simple event loop to properly handle Wi-Fi events. Here is the code example.

Application entry point function:

void app_main()
{
printf("Configured Wi-Fi SSID is %s\n", CONFIG_ESP_WIFI_SSID);
init_wifi();
}

Wi-Fi initialisation code:

void init_wifi()
{
s_app_event_group = xEventGroupCreate();
tcpip_adapter_init();

ESP_ERROR_CHECK(esp_event_loop_create_default());

wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
ESP_ERROR_CHECK(esp_wifi_init(&cfg));

ESP_ERROR_CHECK(esp_event_handler_register(WIFI_EVENT, ESP_EVENT_ANY_ID, &event_handler, NULL));
ESP_ERROR_CHECK(esp_event_handler_register(IP_EVENT, IP_EVENT_STA_GOT_IP, &event_handler, NULL));

wifi_config_t wifi_config = {
.sta = {
.ssid = CONFIG_ESP_WIFI_SSID,
.password = CONFIG_ESP_WIFI_PASSWORD
},
};

ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_STA) );
ESP_ERROR_CHECK(esp_wifi_set_config(ESP_IF_WIFI_STA, &wifi_config) );
ESP_ERROR_CHECK(esp_wifi_start() );

EventBits_t bits = xEventGroupWaitBits(s_app_event_group,
WIFI_CONNECTED_BIT | WIFI_FAIL_BIT,
pdFALSE,
pdFALSE,
portMAX_DELAY);

if (bits & WIFI_CONNECTED_BIT) {
printf("Connection established\n");
} else if (bits & WIFI_FAIL_BIT) {
printf("Failed to connect to a Wi-Fi\n");
}

ESP_ERROR_CHECK(esp_event_handler_unregister(IP_EVENT, IP_EVENT_STA_GOT_IP, &event_handler));
ESP_ERROR_CHECK(esp_event_handler_unregister(WIFI_EVENT, ESP_EVENT_ANY_ID, &event_handler));
vEventGroupDelete(s_app_event_group);
}

Event handler code:

static void event_handler(void* arg, esp_event_base_t event_base,
int32_t event_id, void* event_data)
{
if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_START) {
esp_wifi_connect();
} else if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_DISCONNECTED) {
if (s_retries_count < CONFIG_ESP_MAXIMUM_RETRY) {
esp_wifi_connect();
s_retries_count++;
} else {
xEventGroupSetBits(s_app_event_group, WIFI_FAIL_BIT);
}
} else if (event_base == IP_EVENT && event_id == IP_EVENT_STA_GOT_IP) {
ip_event_got_ip_t* event = (ip_event_got_ip_t*) event_data;
s_retries_count = 0;
xEventGroupSetBits(s_app_event_group, WIFI_CONNECTED_BIT);
}
}

If the connection is established, an output like this will appear in the serial port monitor:

I (3054) tcpip_adapter: sta ip: 192.168.0.110, mask: 255.255.255.0, gw: 192.168.0.1
Connection established

The next step is to connect our application to AWS IoT.

Connecting to AWS IoT

Let’s quickly revise the way interaction between IoT device and the AWS IoT Core happens. Each physical device has a corresponding representation in the AWS IoT service called “Thing”. The actual device connects to AWS IoT endpoint using the MQTT protocol and TLS certificates.

CREATING AWS IOT THING AND CERTIFICATES PROVISIONING

In our case we are going to use a single thing provisioning and certificate generation using AWS IoT console. This is acceptable for development purposes, but is not scalable enough for a production use. The process is rather straightforward. First, go to the Internet of Things section and select the IoT Core item:

Then click the “Create” button in the “Manage” section of the IoT Core Console. Use the “Create a single thing” option:

On the certificate creation step, choose the “One-click certification creation”:

Once certificates are created, download all three files(it won’t be possible to do it later!) and click the “Activate” button before proceeding to the final step.

We’ll leave the certificate without a policy for now. Once the thing is added, it will be displayed in the console:

After the certificate is created, we can add a new policy. Here is a policy template:

{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "iot:Connect",
"Resource": "arn:aws:iot:<region>:<account>:client/ESP-AWS-IoT-Demo"
},
{
"Effect": "Allow",
"Action": "iot:Publish",
"Resource": [
"arn:aws:iot:<region>:<account>:topic/$aws/things/ESP-AWS-IoT-Demo/shadow/update",
"arn:aws:iot:<region>:<account>:topic/$aws/things/ESP-AWS-IoT-Demo/shadow/delete",
"arn:aws:iot:<region>:<account>:topic/$aws/things/ESP-AWS-IoT-Demo/shadow/get"
]
},
{
"Effect": "Allow",
"Action": "iot:Receive",
"Resource": [
"arn:aws:iot:<region>:<account>:topic/$aws/things/ESP-AWS-IoT-Demo/shadow/update/accepted",
"arn:aws:iot:<region>:<account>:topic/$aws/things/ESP-AWS-IoT-Demo/shadow/delete/accepted",
"arn:aws:iot:<region>:<account>:topic/$aws/things/ESP-AWS-IoT-Demo/shadow/get/accepted",
"arn:aws:iot:<region>:<account>:topic/$aws/things/ESP-AWS-IoT-Demo/shadow/update/rejected",
"arn:aws:iot:<region>:<account>:topic/$aws/things/ESP-AWS-IoT-Demo/shadow/delete/rejected"
]
},
{
"Effect": "Allow",
"Action": "iot:Subscribe",
"Resource": [
"arn:aws:iot:<region>:<account>:topicfilter/$aws/things/ESP-AWS-IoT-Demo/shadow/update/accepted",
"arn:aws:iot:<region>:<account>:topicfilter/$aws/things/ESP-AWS-IoT-Demo/shadow/delete/accepted",
"arn:aws:iot:<region>:<account>:topicfilter/$aws/things/ESP-AWS-IoT-Demo/shadow/get/accepted",
"arn:aws:iot:<region>:<account>:topicfilter/$aws/things/ESP-AWS-IoT-Demo/shadow/update/rejected",
"arn:aws:iot:<region>:<account>:topicfilter/$aws/things/ESP-AWS-IoT-Demo/shadow/delete/rejected"
]
},
{
"Effect": "Allow",
"Action": [
"iot:GetThingShadow",
"iot:UpdateThingShadow",
"iot:DeleteThingShadow"
],
"Resource": "arn:aws:iot:<region>:<account>:thing/ESP-AWS-IoT-Demo"

}
]
}


Replace <region> and <account> with your values, create a policy and attach it to a newly created certificate.

This is it for the AWS IoT side, let’s continue with the ESP-IDF.

CONFIGURING ESP-IDF

First, let’s add AWS IoT component to ESP-IDF. To do so, create a “components” directory in the root of your project and execute the following commands:

cd components
git clone --recursive https://github.com/espressif/esp-aws-iot
cd ..

AWS IoT Thing certificate, private and public keys should be placed in the main/certs directory. Be sure to exclude this directory from a version control and never share your private keys with anyone!

I used the following naming convention:

  • root-ca.pem - Amazon Root CA. Need to ensure whatever endpoint we are going to use is a real AWS IoT endpoint and is not provided by some impostor.
  • private.pem.key - private key
  • public.pem.key - public

To embed certificate into the device, main/CMakeLists.txt should be extended with these lines:

target_add_binary_data(${COMPONENT_TARGET} "certs/root-ca.pem" TEXT)
target_add_binary_data(${COMPONENT_TARGET} "certs/certificate.pem.crt" TEXT)
target_add_binary_data(${COMPONENT_TARGET} "certs/private.pem.key" TEXT)

Add two new configuration options to the main/Kconfig.projbuild file:

    config AWS_IOT_THING_NAME
string "AWS IoT Thing Name"
default "ESP-AWS-IoT-Demo"
help
AWS IoT Thing Name

config AWS_IOT_CLIENT_ID
string "AWS IoT Client ID"
default "ESP32-CLIENT"
help
AWS IoT Client Name

Let’s configure AWS IoT related options. Run idf.py menuconfig and go to the “Component Config -> Amazon Web Services IoT Platform” section. An AWS IoT endpoint should be set. To get the information about the endpoint, open the AWS IoT Core Console, select your thing and check the endpoint located in the “Interact” section of the thing.

Once done, return to the main configuration section, proceed to our “ESP AWS IoT Example Configuration” section and specify the AWS IoT thing name there.

CONNECTING TO THE AWS IOT

The connection to AWS IoT is very straightforward. Here is a relevant code fragment:

rc = aws_iot_shadow_init(&mqttClient, &shadowParameters);  
if(SUCCESS != rc) {
ESP_LOGE(TAG, "aws_iot_shadow_init returned error %d, aborting...", rc);
abort();
}

The same is for connecting to the thing shadow:

ShadowConnectParameters_t scp = ShadowConnectParametersDefault;
scp.pMyThingName = CONFIG_AWS_IOT_THING_NAME;
scp.pMqttClientId = CONFIG_AWS_IOT_CLIENT_ID;
scp.mqttClientIdLen = (uint16_t) strlen(CONFIG_AWS_IOT_CLIENT_ID);

rc = aws_iot_shadow_connect(&mqttClient, &scp);
if(SUCCESS != rc) {
ESP_LOGE(TAG, "aws_iot_shadow_connect returned error %d, aborting...", rc);
abort();
}

In this demo we’ll update a field in the thing shadow called “demoField”. Before doing it, we need to prepare some JSON. As you may notice, working with JSON in C is somewhat…interesting:

jsonStruct_t demoField;
demoField.cb = demoField_callback;
demoField.pData = TAG;
demoField.pKey = "demoField";
demoField.type = SHADOW_JSON_STRING;
demoField.dataLength = strlen(TAG);

The value of the field is the same as the application logging tag. Adding this field to a resulting JSON document can be done using this code example:

rc = aws_iot_shadow_add_reported(JsonDocumentBuffer, sizeOfJsonDocumentBuffer, 1, &demoField);

A full example is located in the GitHub repository.

Conclusion

ESP32 is a great microcontroller that comes with a good documentation and a powerful and extensible SDK. By using AWS IoT Embedded SDK it's possible to quickly connect any ESP32-powered device with the AWS IoT Core.