What you will learn

This tutorial teaches you how to build, link, and deploy the Xively Client Library on ESP32 modules, and how to connect your device to Xively. This tutorial supports Linux and macOS.

Following this tutorial on Linux and Windows

This tutorial and the libxively/application code it uses were created on macOS. The steps should be the same on Linux, but we haven't tested it.

If you're a Windows user, consider using cygwin to replicate the steps that use make, and please report any broken steps to us.

Hardware Requirements

  • ESP32 - We used Adafruit's Huzzah development module, but any ESP32 module will do
  • Button connected between GPIO0 and GND-- embedded in most modules
  • LED connected between GPIO17 and GND (pin can be modified in the source code)
  • Jumper cables to connect the LED to the ESP32

Flash memory requirements for this demo

The demo supports Over The Air Firmware updates, which imposes some requirements on the Flash partition table. For out-of-the-box support, we recommend using a module with 4MB of flash.

If you don't need OTA Firmware Updates or Xively's Secure File Transfer, you can free most of the memory required by our partition table. More details and relevant links at the end of this tutorial

Task 1 of 6: Installing the ESP32 Components

You'll need:

  • Espressif's ESP32 SDK: ESP-IDF
  • ESP32-xtensa toolchain

Testing the demo without building it yourself

If you'd like to try the demo, but you'd rather avoid setting up the SDK, toolchain, etc, use the pre-compiled binaries we've provided. They're available in the ESP32 example folder.

If you decide to use those pre-compiled binaries, skip all the build steps in this tutorial and go to Task 6: Flashing and provisioning the device.

Install the ESP-IDF SDK

Set up the esp-idf repository, following Espressif's online documentation

Note: The following code box is a quick reminder for use after initial setup. You must initially follow Espressif's online documentation.

git clone --recursive https://github.com/espressif/esp-idf.git
IDF_PATH=$(YOUR_PATH_TO_FOLDER_ESP-IDF)

Using the same SDK version we did: [16de6bff]

If you run into problems going through this tutorial, it's possible the ESP-IDF was updated with breaking changes we didn't know about.

If that happens, you can use the exact same version of the SDK we did by checking out the same commit. We can do that with these 2 commands:

cd esp-idf
git checkout -b xively_tutorial_version 16de6bff
git submodule update

This will create and select a new branch called xively_tutorial_version, downgraded to the same version we used. If you can find and solve the incompatibilities with the new SDK,

  1. cd into esp-idf again
  2. type git checkout master to go back to the latest code.

Install Toolchain

Follow Espressif's instructions to install the toolchain in your Operating System:

Remember to update your environment PATH as explained in those instructions. If you did it correctly, you should be able to execute the commands xtensa-esp32-elf-gcc and xtensa-esp32-elf-ar from any folder on your system. You will need it later.

Task 2 of 6: Create your Xively digital device

Prerequisites
You should have a Xively account.

Before connecting a physical device to Xively, you must have its digital representation in the system. Log into the Xively management app to complete the following steps.

Create an ESP32 device template using the Product Launcher

This operation will create a device template and a device instance in Xively to represent your hardware setup.

  1. Click on Product Launcher > Add another device.
  1. From the pop-up window select Choose from our template library and click Next.
  1. From the sections tabs at the top of the window go to Quickstart Kits, select your kit, and click Next.

This will automatically create a new Device based on the ESP32's default template, the same way you'd usually do it from the "Device Templates" page.

Get credentials for this device

In order for your device to securely talk to Xively it needs credentials that it will use to authenticate itself as a valid device within your account.

  1. Go to Devices > All devices and look for your sample device. Click on its name.
  2. Click on Get password.
    Save it for later, it will be needed in the next step.
  3. Click on Xively Device ID and Xively Account ID to copy them to clipboard.

The Device ID and the Password will be used to establish the MQTT connection from the device. The Account ID and Device ID are both used to create the name of the device's MQTT topics. That will all be handled by the example application. Write them down, or come back to this page when you're ready to configure your device.

Task 3 of 6: Building the MQTT and TLS libraries

In this step, we're going to build the libwolfssl.a and libxively.a static libraries.

Download Xively C Client

Download the library source code from xively-clientc. Git Clone the repository or download the source archive from the right side of the GitHub page.

Open a command line prompt (Windows) or terminal prompt (Mac/Linux) and use the following commands in order:

cd xively-client-c/
make PRESET=ESP32

You should have 2 static libraries in xively-client-c/bin/esp32/: libxively.a and libwolfssl.a. Those are binaries that will be linked with our applications. If you modify either library or their build configurations, don't forget to re-build them.

Task 4 of 6: Building the Demo Application

Now that the MQTT and TLS libraries are ready, let's compile the application and link everything into a working binary:

Open a command line prompt (Windows) or terminal prompt (Mac/Linux) and use the following commands in order:

cd xively-client-c/examples/esp32/xively_demo/
make

In the build configuration GUI, you can configure the path to your device's serial port as explained in step 6.

The first time you use this makefile, you'll have to configure the Serial Port to your device. A GUI will appear in your terminal after running the command, and you can set your path in Serial Flasher Config -> Default Serial Port:

Set the path, click < Save >, < OK > the default path, and < Exit >.

Task 5 of 6: Flashing the device

First of all, we need to connect the device. If you're using the same Development module as us, you'll only need to connect the board via USB and use the tools provided in the SDK. For other modules, refer to their documentation. You may need to press some button or connect your own USB-UART adapter.

If you've already configured the path to your serial port using the make menuconfig GUI in step 5, you can flash your device very easily using make flash:

cd xively-client-c/examples/esp32/xively_demo
make flash

To flash the pre-compiled binaries we've provided in the xively-client-c repo, use this command to invoke Espressif's flashing tool directly:

Warning: You must not use the make flash command as that would re-build everything.

cd xively-client-c/examples/esp32/xively_demo
python $IDF_PATH/components/esptool_py/esptool/esptool.py \
            --chip esp32 --port /dev/tty.SLAB_USBtoUART --baud 115200 \
            --before default_reset --after hard_reset write_flash -z \
            --flash_mode dio --flash_freq 40m --flash_size detect \
            0x1000 ./build/bootloader/bootloader.bin \
            0x10000 ./build/xively_demo.bin \
            0x8000 ./build/xi_demo_partitions.bin

Task 6 of 6: Provisioning the device via UART

Connect to UART

Now we need to store the Wi-Fi SSID, password, and the Xively Credentials we created earlier in the device . In this example, we'll manually upload them to the device via UART using a serial terminal.

Use a serial terminal (e.g. miniterm) to connect to your device's serial port.

On Linux and OSX, that port can be found at /dev/tty*.

Windows users can use the device manager to find out which COM port has been assigned to the device.

The serial port settings are:

  • Baud Rate: 115200bps
  • 8 data bits, No parity, 1 stop bit

check the documentation for your preferred serial terminal on how to connect a serial port. For example, in miniterm.py, use the following command:

miniterm.py /dev/tty.SLAB_USBtoUART 115200

Remember to replace /dev/cu.SLAB_USBtoUART with the path to your serial port.

Kickstart provisioning mode

The demo application boots in 'provisioning mode' if it's missing any credentials, so it will happen by default on your device. After going through provisioning once, the device will store the credentials in Non-Volatile Storage and never ask for them again.

If you'd like to update any of your credentials, you can manually trigger the provisioning mode using our button. Right after booting, the device will rapidly flash our LED for a few seconds. Press the button while the LED is flashing to kickstart provisioning.

Set your WiFi and Xively Device credentials

Follow the instructions received via UART to set your WiFi and MQTT credentials:

Result

Your ESP32 device is now connected and communicating with Xively.

To see the status of your device, go back to the device page in the Xively management app. Your device should now appear as Connected in your device logs

Now that the device is connected via MQTT, the example code will publish a message to the Button channel whenever the button is pressed. It will also subscribe to the LED topic and turn the LED on/off as instructed over MQTT.

Any Xively User with permission to communicate with this device can subscribe to the Button topic to receive updates and publish to the LED topic to control it.

We can easily test the communication with the device using the Messaging tab in the Device page.

Receiving messages from the device when the button is pressed

Controlling the LED

Communicating with the device through a custom GUI

While the Messaging tab is the most useful option while writing your application, in your final product the users will most likely communicate through a GUI. To replicate that behaviour, you can see graphed button data and control the LED through simple clicks using the Product Launcher:

For more information, check the ESP32 forum topic: https://esp32.com/viewtopic.php?f=18&t=3238

Extra: Over The Air Firmware Updates

Now that your device is up and running, and you've confirmed it can connect to the MQTT broker, you can try more advanced features. We've implemented Xively's Secure File Transfer protocol in this demo, so your device supports and expects OTA firmware updates by default.

The demo is capable of updating itself in the background, while maintaining responsiveness, over the same MQTT connection it's already using to communicate. It will then reboot into the new image.

There's also a FAT Filesystem in flash that you can use to store arbitrary files.

Here's how you can deploy new firmware binaries and other files to your device:

  1. Become a firmware manager for your Xively Account following the steps described in step 2 of the cc3200 SFT tutorial
  2. Create a new Update Package for your Device Template.
    Remember: If you want to update your device's firmware, you must call that file firmware.bin. Otherwise, it will be treated as a regular file
  3. When you start the deployment, your device will download and apply the firmware.bin binary, the device will reboot, and the new code will run.
The ESP32's BSP in the Xively Client defines 'firmware.bin' as the file name for firmware updates. Other files are stored in the filesystem

The ESP32's BSP in the Xively Client defines 'firmware.bin' as the file name for firmware updates. Other files are stored in the filesystem

If everything went well, your device should start downloading its new firmware right away. You can test the device's responsiveness by pressing the button and controlling the LED over MQTT.

Once the update is complete, the device will reboot and start running its new firmware.

After a firmware image has been successfully deployed over the air to your device, the factory default image won't boot anymore. If you'd like to flash a new binary over the serial port (i.e. make flash), you will have to erase the OTA partitions first. The simplest way to do that is running make erase_flash before re-flashing your device. Keep in mind you'll have to re-provision it, because any WiFi/Xively credentials you stored in the device will be erased along with the binaries.

Using Xively's Secure File Transfer in your Project

In order to operate, the Xively Client's implementation of SFT requires a filesystem on the device. This filesystem is also available to the application, and it can store any arbitrary files you may want to update over the air: HTML files, logos, settings, TLS Root Certificates, etc.

Your application only needs to decide which files should be kept up to date by calling xi_set_updateable_files(). In this demo, there's a function that creates the list and configures the library:

static int8_t xt_configure_fw_updates( void )
{
    const char** updateable_files = ( const char* [] ){"firmware.bin"};
    xi_state_t retv = XI_STATE_OK;

    printf( "\n[XT] Registering updateable files list" );

    retv = xi_set_updateable_files( xt_context_handle, updateable_files, 1, NULL );
    if ( XI_STATE_OK != retv )
    {
        printf( "\n[XT] Error [%d] configuring FW updates", retv );
        return -1;
    }
    return 0;
}

If your project needs to keep track of any other files, you can easily add their names to the list. The Xively Client library will download them to the flash filesystem, and keep them up to date with future releases.

The filesystem where files are stored is defined in the demo's partition table: xi_demo_partitions.csv

# Name,   Type, SubType, Offset,   Size
nvs,      data, nvs,     0x9000,   0x4000
otadata,  data, ota,     0xd000,   0x2000
phy_init, data, phy,     0xf000,   0x1000
factory,  app,  factory, 0x10000,  1M
ota_0,    app,  ota_0,   ,         1M
ota_1,    app,  ota_1,   ,         1M
storage,  data, fat,     ,         0xf0000

You can see the partition table assumes a flash chip of 4MB+. 3MB are dedicated to 3 firmware images: Factory default, OTA_0 and OTA_1.

Finally, you can increase the XI_SFT_FILE_CHUNK_SIZE macro in xively-client-c/src/libxively/xi_config.h to speed up SFT file downloads, at the expense of memory and network usage.

You can read all about ESP32 firmware updates and partition tables here. If your flash size is incompatible with our partition table, you can use that documentation to tailor the table to your needs.