Firmware updates: the SFT (Secure File Transfer) protocol

This section describes the protocol used for firmware updates.

For device developers, the firmware update protocol is designed to be easy to implement and flexible in implementation options.

The firmware update protocol operates over MQTT, providing the security and efficiency of MQTT TLS connections in conjunction with the device authentication and authorization provided by Xively.

The protocol utilizes the cln and svc control channels. Control channels are channels Xively makes available to all devices.

The protocol allows device implementations to choose MQTT or HTTP for actual file downloads. MQTT can be used for smaller files (<8MB).

Device implementations should implement all protocol messages for proper firmware tracking in Xively CPM.

What's the general process?

  1. When a device powers on or connect to the Xively broker (or at any time, depending on the device's implementation of the protocol), the device sends the FILE_INFO message to report the current versions of the files it has.
  2. If there are newer versions available for any of the files the device has reported, Xively sends a FILE_UPDATE_AVAILABLE message listing the latest versions: their names, revision numbers, size and fingerprints/checksums.
  3. The device transfers the files using MQTT or HTTP. If the device is using MQTT for file transfer, the device and broker exchange a series of FILE_GET_CHUNK and FILE_CHUNK messages. If the device is using HTTP for file transfer, the transfer occurs outside of Xively. Note: a device can refuse an update with the FILE_STATUS message.
  4. The device sends FILE_STATUS messages to report file progress (DOWNLOADING, DOWNLOADED, PROCESSING, FINISHED). Each status may indicate SUCCESS or FAILURE. If there is a FAILURE, an implementation-specific error code can be provided.
  5. When a device has completed all work on a file, it sends the FILE_STATUS FINISHED message. Xively CPM indicates the deployment for a device is FINISHED when Xively receives a FILE_STATUS message for all files in the package, or the device sends a FILE_INFO listing all files in the package.
Protocol and flow

Protocol and flow

Rollouts of new firmware

When an administrator deploys new firmware as a test deployment, devices will receive an updated list of files the next time they send a FILE_INFO message. When an administrator deploys new firmware as a fleet-wide deployment, devices receive an updated list of files sometime over the next 24 hours after sending a FILE_INFO packet. Some devices will not receive the list immediately, but they will receive the list up to 24 hours after the deployment starts.

Message types overview

There are 5 types of messages used in firmware and file deployment:

Message type
Message number
Direction
Description

FILE_INFO

0

Device -> Broker

Used by the device to report what files it has, and at what version.

FILE_UPDATE_AVAILABLE

1

Broker -> Device

Used by the broker to tell a device what file updates are available (if any).

FILE_GET_CHUNK

2

Device -> Broker

Used by the device to request chunks of files it must update.

FILE_CHUNK

3

Broker -> Device

Used by the broker to deliver chunks of files to the device.

FILE_STATUS

4

Device -> Broker

Used by the device to report on the success or failure of a file's delivery.

MQTT Topics and Payloads

By default, the entire firmware update process is carried out over MQTT, meaning that the device never has to switch contexts or interrupt its normal operation while downloading files.

All messages to and from the broker are encoded as RFC 7049 Concise Binary Object Representation (CBOR) packets. See the RFC for specifications and cbor.io for example codecs.

The Xively client can handle file updates in the background

The Xively embedded client (written in C), is the only embedded client on the market that is capable of multitasking while remaining single-threaded.

This means that (unlike all other lightweight embedded clients that use MQTT), it is capable of downloading file updates in the background without interfering with the devices standard operations.

Updates are managed through two MQTT control topics:

  • xi/ctrl/v1/<deviceId>/cln Xively sends messages to the device on this topic (devices should always subscribe to this topic if they wish to use firmware update capabilities)
  • xi/ctrl/v1/<deviceId>/svc the device sends messages to Xively on this topic
  • Device must publish to the /svc topic using QOS0 or the broker will not respond.

Files that exceed the MQTT maximum capabilities must be sent over HTTP.

The 'FILE_INFO' message (0)

Devices send the FILE_INFO message periodically to check for available file updates and after performing an update.

These messages consist of:

  • A message type field equal to FILE_INFO
  • A FILE_INFO message version number
  • An array of File name and Revision
[CBOR_TYPE_MAP] Definite, size: 3
    [CBOR_TYPE_STRING] Definite, "msgtype"
    [CBOR_TYPE_UINT] Value 0
    [CBOR_TYPE_STRING] Definite, "msgver"
    [CBOR_TYPE_UINT] Value 1
    [CBOR_TYPE_STRING] Definite, "list"
    [CBOR_TYPE_ARRAY] Indefinite, size:  2
        [CBOR_TYPE_MAP] Definite, size: 2
            [CBOR_TYPE_STRING] Definite, "N"
            [CBOR_TYPE_STRING] Definite, "OS"
            [CBOR_TYPE_STRING] Definite, "R"
            [CBOR_TYPE_STRING] Definite, "1_0_5"
        [CBOR_TYPE_MAP] Definite, size: 2
            [CBOR_TYPE_STRING] Definite, "N"
            [CBOR_TYPE_STRING] Definite, "BSP"
            [CBOR_TYPE_STRING] Definite, "R"
            [CBOR_TYPE_STRING] Definite, "Latest"

The 'FILE_UPDATE_AVAILABLE' message (1)

The FILE_UPDATE_AVAILABLE message is sent to devices in response to a FILE_INFO message when updates are available for the device to download. The device can start updating immediately, wait until it is idle, or wait for user consent.

This message consists of:

  • A message type field equal to FILE_UPDATE_AVAILABLE
  • A FILE_UPDATE_AVAILABLE message version
  • An array of:
    • File name
    • Revision
    • ImageSize
    • Fingerprint

The fingerprint is referred to as a checksum in Xively CPM. Device implementations should expect to receive base64-encoded data in this field to facilitate entry/display of the field in CPM.

[CBOR_TYPE_MAP] Definite, size: 3
    [CBOR_TYPE_STRING] Definite, msgtype
    [CBOR_TYPE_UINT] Value: 1
    [CBOR_TYPE_STRING] Definite, msgver
    [CBOR_TYPE_UINT] Value: 1
    [CBOR_TYPE_STRING] Definite, list
    [CBOR_TYPE_ARRAY] Indefinite, size:  2
        [CBOR_TYPE_MAP] Definite, size: 4
            [CBOR_TYPE_STRING] Definite, N
            [CBOR_TYPE_STRING] Definite, OS
            [CBOR_TYPE_STRING] Definite, R
            [CBOR_TYPE_STRING] Definite, 1_0_5
            [CBOR_TYPE_STRING] Definite, S
            [CBOR_TYPE_UINT] Width: 4B, Value: 12345
            [CBOR_TYPE_STRING] Definite, F
            [CBOR_TYPE_BYTESTRING] Definite, 561516521652165
        [CBOR_TYPE_MAP] Definite, size: 4
            [CBOR_TYPE_STRING] Definite, N
            [CBOR_TYPE_STRING] Definite, Credentials
            [CBOR_TYPE_STRING] Definite, R
            [CBOR_TYPE_STRING] Definite, 1_0_0
            [CBOR_TYPE_STRING] Definite, S
            [CBOR_TYPE_UINT] Width: 4B, Value: 230
            [CBOR_TYPE_STRING] Definite, F
            [CBOR_TYPE_BYTESTRING] Definite, 11001001010

The 'FILE_GET_CHUNK' message (2)

Devices send the FILE_GET_CHUNK message during the update process to fetch a chunk of a file (i.e. the first message of this kind after a FILE_UPDATE_AVAILABLE indicates that the device starts the file update process). Length must be smaller than or equal to 65536 bytes.

These messages consist of:

  • A message type field equal to FILE_GET_CHUNK
  • FILE_GET_CHUNK message version
  • File name
  • Revision
  • Offset: uint32
  • Length: uint32
[CBOR_TYPE_MAP] Definite, size: 6
     [CBOR_TYPE_STRING] Definite, msgtype
    [CBOR_TYPE_UINT] Value: 2
    [CBOR_TYPE_STRING] Definite, msgver
    [CBOR_TYPE_UINT] Width: 1B, Value: 1
    [CBOR_TYPE_STRING] Definite, N
    [CBOR_TYPE_STRING] Definite, OS
    [CBOR_TYPE_STRING] Definite, R
    [CBOR_TYPE_STRING] Definite, 1_0_5
    [CBOR_TYPE_STRING] Definite, O
    [CBOR_TYPE_UINT] Width: 4B, Value: 12345
    [CBOR_TYPE_STRING] Definite, L
    [CBOR_TYPE_UINT] Width: 4B, Value: 4096

The 'FILE_CHUNK' message (3)

The FILE_CHUNK message is sent in response to a FILE_GET_CHUNK message with the requested chunk of the given file.

These messages consist of:

  • A message type field equal to FILE_CHUNK
  • FILE_CHUNK message version
  • File Name
  • Revision
  • Offset: uint32
  • Length: uint32
  • Status: uint8Chunk: byte array

The status field indicates success or error as follows:
Status = 0: File chunk request is correct.
Status = 1: This block contains the end of the file and so it's shorter than the request length.
Status = 2: File offset is greater than file length.
Status = 3: Requested length is greater than the maximum allowed length of 65536 bytes.
Status = 4: File unavailable.

 [CBOR_TYPE_MAP] Definite, size: 8
    [CBOR_TYPE_STRING] Definite, msgtype
    [CBOR_TYPE_UINT]  Value: 3
    [CBOR_TYPE_STRING] Definite, magver
    [CBOR_TYPE_UINT]  Value: 1
    [CBOR_TYPE_STRING] Definite, N
    [CBOR_TYPE_STRING] Definite, OS
    [CBOR_TYPE_STRING] Definite, R
    [CBOR_TYPE_STRING] Definite, 1_0_5
    [CBOR_TYPE_STRING] Definite, O
    [CBOR_TYPE_UINT] Width: 4B, Value: 12345
    [CBOR_TYPE_STRING] Definite, L
    [CBOR_TYPE_UINT] Width: 4B, Value: 4096
    [CBOR_TYPE_STRING] Definite, S
    [CBOR_TYPE_UINT] Width: 1B, Value: 0
    [CBOR_TYPE_STRING] Definite, C
    [CBOR_TYPE_BYTESTRING] Definite, size: 4096
        [CBOR_TYPE_UINT] Width: 1B, Value: 167
        [CBOR_TYPE_UINT] Width: 1B, Value: 241
                  ...
        [CBOR_TYPE_UINT] Width: 1B, Value: 217
        [CBOR_TYPE_UINT] Width: 1B, Value: 42
        [CBOR_TYPE_UINT] Width: 1B, Value: 130

The 'FILE_STATUS message (4)

The FILE_STATUS message reports the status of a single file. Status codes other than zero (i.e. success) are application-dependent. For example: -3: File lost -14: File Corrupted

The FILE_STATUS messages in relation to a given file are expected to arrive in the following order: downloaded, processing, finished. The protocol does not require devices to send these status messages, but normally a device should be designed to send them for accurate reporting. If the FILE_STATUS status messages arrive in a different order, the report status will be undefined.

If the deployment package contained multiple files (meaning the FILE_UPDATE_AVAILABLE message listed multiple files), the overall progress of the deployment for a device is counted under the FILE_STATUS indicating the least progress. For example, if a package contains three files, and two of the files are FINISHED and one file is DOWNLOADING, the overall deployment status for the device is counted as DOWNLOADING.

A message type field equal to FILE_STATUS

When the File Status Phase is in uint8:
0-2 - Reserved, set by Xively itself, ignored when received from clients (using categories like Unreported, Reported, Downloading)
3 - Downloaded
4 - Processing
5 - Finished (the client will not work on the file anymore, even after an error)

When the File Status code is in int32:

  • If it is greater than or equal to 0, the result returned is a Success.
  • If it is less than 0, the result returned is an Error and if the deployment returns a message less than 0, then it will appear under 'Finished with error'

The Default integer is 0.

You can view the status of your Production Deployments under the Xively web interface. For more information, see Monitoring a Production Deployment. Information relating to Device specific error codes will appear in the Device logs, see Viewing Device Firmware.

 [CBOR_TYPE_MAP] Definite, size: 6
    [CBOR_TYPE_STRING] Definite, msgtype
    [CBOR_TYPE_UINT]  Value: 4
    [CBOR_TYPE_STRING] Definite, msgver
    [CBOR_TYPE_UINT]  Value: 1
    [CBOR_TYPE_STRING] Definite, N
    [CBOR_TYPE_STRING] Definite, OS
    [CBOR_TYPE_STRING] Definite, R
    [CBOR_TYPE_STRING] Definite, 1_0_5
    [CBOR_TYPE_STRING] Definite, P
    [CBOR_TYPE_UINT] Width: 1B, Value: 5
    [CBOR_TYPE_STRING] Definite, S
    [CBOR_TYPE_UINT] Width: 1B, Value: 0