Firmware updates: the SFT (Secure File Transfer) protocol

If you want to know the more technical aspects of the Firmware Updates and Secure File Transfer features, this is your doc!

From the device side, firmware and file updates are built to be very easy and flexible to implement, with Xively managing your files' version histories and deployments to individual devices, and rolling that information up to admins in the management app.

What's the general process?

  1. When a device wakes up, it reports on the latest versions of files it has (which is the FILE_INFO message)
  2. If there are newer versions available for any of the files the device has reported, Xively will report back the latest versions: their names, revision numbers, size and fingerprints or checksums (in a FILE_UPDATE_AVAILABLE message)
  3. The device can cycle through the available files to update and request them in chunks (with a FILE_GET_CHUNK message). The device can also refuse the update (with a FILE_STATUS message).
  4. The broker will send back chunked payloads of the files with the requested offset and length (in a FILE_CHUNK message)
  5. (Optional) The device can report back a successful or failed download on a per-file basis when it receives the full file and validates its checksum locally (with a FILE_STATUS message)
  6. When the device has received all of the updates at that time and validated those files, it can update Xively with its latest file statuses (with another FILE_INFO message)

Rollouts of new firmware

When an admin deploys new firmware as a test deployment, devices will receive an updated list of files the next time they check with the broker using a FILE_INFO packet. When an admin deploys new firmware as a fleet-wide deployment, devices will receive an updated list of files sometime over the next 24 hours the next time they check with the broker using a FILE_INFO packet. Some devices will not receive the updates list immediately but should receive the updates up to 24 hours after the deployment starts.

Sequence diagram

The below diagram follows the firmware/file deployment process from the moment of a device powering on and asking the broker if there are any files that it should update, through to the device's confirmation of updated files and installation/reboot.

Message types overview

There are 5 types of messages used in firmware and file deployment:

Message type
Message number
Direction
Description

FILE_INFO

0

Device -> Broker

Used by the device to report what files it has, and at what version.

FILE_UPDATE_AVAILABLE

1

Broker -> Device

Used by the broker to tell a device what file updates are available (if any).

FILE_GET_CHUNK

2

Device -> Broker

Used by the device to request chunks of files it must update.

FILE_CHUNK

3

Broker -> Device

Used by the broker to deliver chunks of files to the device.

FILE_STATUS

4

Device -> Broker

Used by the device to report on the success or failure of a file's delivery. 'Success' is based on whether the downloaded file matches the fingerprint checksum that the broker sent along with the file in the FILE_UPDATE_AVAILABLE message.

How it's done over MQTT

The entire firmware update process is carried out over MQTT, meaning that the device never has to switch contexts or interrupt its normal operation while downloading files.

All messages to and from the broker are encoded as RFC 7049 Concise Binary Object Representation (CBOR) packets. See the RFC for specifications and cbor.io for example codecs.

The Xively client can handle file updates in the background

The Xively embedded client (written in C), is the only embedded client on the market that is capable of multitasking while remaining single-threaded.

This means that (unlike all other lightweight embedded clients that use MQTT), it is capable of downloading file updates in the background without interfering with the devices standard operations.

Updates are managed through two MQTT control topics:

  • xi/ctrl/v1/<deviceId>/cln Xively sends messages to the device on this topic (devices should always subscribe to this topic if they wish to use firmware update capabilities)
  • xi/ctrl/v1/<deviceId>/svc the device sends messages to Xively on this topic
  • Device must publish to the /svc topic using QOS0 or the broker will not respond.

The 'FILE_INFO' message (0)

Devices send the FILE_INFO message periodically to check for available file updates and after performing an update.

These messages consist of:

  • A message type field equal to FILE_INFO
  • A FILE_INFO message version number
  • An array of File name and Revision
[CBOR_TYPE_MAP] Definite, size: 3
    [CBOR_TYPE_STRING] Definite, "msgtype"
    [CBOR_TYPE_UINT] Value 0
    [CBOR_TYPE_STRING] Definite, "msgver"
    [CBOR_TYPE_UINT] Value 1
    [CBOR_TYPE_STRING] Definite, "list"
    [CBOR_TYPE_ARRAY] Indefinite, size:  2
        [CBOR_TYPE_MAP] Definite, size: 2
            [CBOR_TYPE_STRING] Definite, "N"
            [CBOR_TYPE_STRING] Definite, "OS"
            [CBOR_TYPE_STRING] Definite, "R"
            [CBOR_TYPE_STRING] Definite, "1_0_5"
        [CBOR_TYPE_MAP] Definite, size: 2
            [CBOR_TYPE_STRING] Definite, "N"
            [CBOR_TYPE_STRING] Definite, "BSP"
            [CBOR_TYPE_STRING] Definite, "R"
            [CBOR_TYPE_STRING] Definite, "Latest"

The 'FILE_UPDATE_AVAILABLE' message (1)

The FILE_UPDATE_AVAILABLE message is sent to devices in response to a FILE_INFO message when updates are available for the device to download. The device can start updating immediately, wait until it is idle, or wait for user consent.

This message consists of:

  • A message type field equal to FILE_UPDATE_AVAILABLE
  • A FILE_UPDATE_AVAILABLE message version
  • An array of:
    • File name
    • Revision
    • ImageSize
    • Fingerprint
[CBOR_TYPE_MAP] Definite, size: 3
    [CBOR_TYPE_STRING] Definite, msgtype
    [CBOR_TYPE_UINT] Value: 1
    [CBOR_TYPE_STRING] Definite, msgver
    [CBOR_TYPE_UINT] Value: 1
    [CBOR_TYPE_STRING] Definite, list
    [CBOR_TYPE_ARRAY] Indefinite, size:  2
        [CBOR_TYPE_MAP] Definite, size: 4
            [CBOR_TYPE_STRING] Definite, N
            [CBOR_TYPE_STRING] Definite, OS
            [CBOR_TYPE_STRING] Definite, R
            [CBOR_TYPE_STRING] Definite, 1_0_5
            [CBOR_TYPE_STRING] Definite, S
            [CBOR_TYPE_UINT] Width: 4B, Value: 12345
            [CBOR_TYPE_STRING] Definite, F
            [CBOR_TYPE_BYTESTRING] Definite, 561516521652165
        [CBOR_TYPE_MAP] Definite, size: 4
            [CBOR_TYPE_STRING] Definite, N
            [CBOR_TYPE_STRING] Definite, Credentials
            [CBOR_TYPE_STRING] Definite, R
            [CBOR_TYPE_STRING] Definite, 1_0_0
            [CBOR_TYPE_STRING] Definite, S
            [CBOR_TYPE_UINT] Width: 4B, Value: 230
            [CBOR_TYPE_STRING] Definite, F
            [CBOR_TYPE_BYTESTRING] Definite, 11001001010

The 'FILE_GET_CHUNK' message (2)

Devices send the FILE_GET_CHUNK message during the update process to fetch a chunk of a file (i.e. the first message of this kind after a FILE_UPDATE_AVAILABLE indicates that the device starts the file update process). Length must be smaller than or equal to 65536 bytes.

These messages consist of:

  • A message type field equal to FILE_GET_CHUNK
  • FILE_GET_CHUNK message version
  • File name
  • Revision
  • Offset: uint32
  • Length: uint32
[CBOR_TYPE_MAP] Definite, size: 6
     [CBOR_TYPE_STRING] Definite, msgtype
    [CBOR_TYPE_UINT] Value: 2
    [CBOR_TYPE_STRING] Definite, msgver
    [CBOR_TYPE_UINT] Width: 1B, Value: 1
    [CBOR_TYPE_STRING] Definite, N
    [CBOR_TYPE_STRING] Definite, OS
    [CBOR_TYPE_STRING] Definite, R
    [CBOR_TYPE_STRING] Definite, 1_0_5
    [CBOR_TYPE_STRING] Definite, O
    [CBOR_TYPE_UINT] Width: 4B, Value: 12345
    [CBOR_TYPE_STRING] Definite, L
    [CBOR_TYPE_UINT] Width: 4B, Value: 4096

The 'FILE_CHUNK' message (3)

The FILE_CHUNK message is sent in response to a FILE_GET_CHUNK message with the requested chunk of the given file.

These messages consist of:

  • A message type field equal to FILE_CHUNK
  • FILE_CHUNK message version
  • File Name
  • Revision
  • Offset: uint32
  • Length: uint32
  • Status: uint8Chunk: byte array

The status field indicates success or error as follows:
Status = 0: File chunk request is correct.
Status = 1: This block contains the end of the file and so it's shorter than the request length.
Status = 2: File offset is greater than file length.
Status = 3: Requested length is greater than the maximum allowed length of 65536 bytes.
Status = 4: File unavailable.

 [CBOR_TYPE_MAP] Definite, size: 8
    [CBOR_TYPE_STRING] Definite, msgtype
    [CBOR_TYPE_UINT]  Value: 3
    [CBOR_TYPE_STRING] Definite, magver
    [CBOR_TYPE_UINT]  Value: 1
    [CBOR_TYPE_STRING] Definite, N
    [CBOR_TYPE_STRING] Definite, OS
    [CBOR_TYPE_STRING] Definite, R
    [CBOR_TYPE_STRING] Definite, 1_0_5
    [CBOR_TYPE_STRING] Definite, O
    [CBOR_TYPE_UINT] Width: 4B, Value: 12345
    [CBOR_TYPE_STRING] Definite, L
    [CBOR_TYPE_UINT] Width: 4B, Value: 4096
    [CBOR_TYPE_STRING] Definite, S
    [CBOR_TYPE_UINT] Width: 1B, Value: 0
    [CBOR_TYPE_STRING] Definite, C
    [CBOR_TYPE_BYTESTRING] Definite, size: 4096
        [CBOR_TYPE_UINT] Width: 1B, Value: 167
        [CBOR_TYPE_UINT] Width: 1B, Value: 241
                  ...
        [CBOR_TYPE_UINT] Width: 1B, Value: 217
        [CBOR_TYPE_UINT] Width: 1B, Value: 42
        [CBOR_TYPE_UINT] Width: 1B, Value: 130

The 'FILE_STATUS message (4)

The FILE_STATUS message reports the status of a single file. Status codes other than zero (i.e. success) are application-dependent. For example: -3: File lost -14: File Corrupted

These messages in relation to a given file are expected to arrive in the following order: downloaded, processing, done, -and any of these may be skipped.

If status messages arrive in a different order, the report status will be undefined. If the FILE_UPDATE_AVAILABLE message contained multiple files, the overall progress of the update process is considered to be that of the 'smallest' of all files (available < downloading < downloaded < processing < done, where available and downloading are reported by Xively itself).

A message type field equal to FILE_STATUS
Message version
File Name
Revision
File Status Phase: uint8
0-2 - Reserved, set by Xively itself, ignored when received from clients (Unreported, Reported, Downloading)
3 - Downloaded
4 - Processing
5 - Finished (the client will not work on the file anymore, even after an error)
File Status code: int32
If it is greater than or equal to 0, the result returned is a Success
If it is less than 0, the result returned is an Error and if the deployment returns a message less than 0, then it will appear under 'Finished with error'

The Default integer is 0.
You can view the status of your Production Deployments under the Xively web interface. For more information, see Monitoring a Production Deployment. Information relating to Device specific error codes will appear in the Device logs, see Viewing Device Firmware.

In JSON, it looks like:
{
"msgtype": 4,
"msgver": 1,
"N": "OS",
"R": "1_0_5",
"P":5
"S":0
}
In CBOR, it looks like:

 [CBOR_TYPE_MAP] Definite, size: 6
    [CBOR_TYPE_STRING] Definite, msgtype
    [CBOR_TYPE_UINT]  Value: 4
    [CBOR_TYPE_STRING] Definite, msgver
    [CBOR_TYPE_UINT]  Value: 1
    [CBOR_TYPE_STRING] Definite, N
    [CBOR_TYPE_STRING] Definite, OS
    [CBOR_TYPE_STRING] Definite, R
    [CBOR_TYPE_STRING] Definite, 1_0_5
    [CBOR_TYPE_STRING] Definite, P
    [CBOR_TYPE_UINT] Width: 1B, Value: 5
    [CBOR_TYPE_STRING] Definite, S
    [CBOR_TYPE_UINT] Width: 1B, Value: 0

Firmware updates: the SFT (Secure File Transfer) protocol

If you want to know the more technical aspects of the Firmware Updates and Secure File Transfer features, this is your doc!