Amazon AWS integration

Users of enterprise systems may want to integrate with Amazon Web Services (AWS) tools to supplement customized parts of their IoT ecosystem.

Xively’s Amazon Kinesis Bridge feature enables you to forward MQTT messages to a Kinesis stream in your AWS account. This way you are able to access a live aggregated stream of specified channels and build custom ingestors/business logic based on these messages that are very easy to scale and can run against billions of messages.

Why might I want to use this?

  • To take advantage of the Simple Notification Service (SNS) to send push notifications to your end user
  • To merge streaming Xively data with information from your other business systems to produce rollups

How do I connect it?

In your Xively account, admins can specify one external (under your own AWS account) Kinesis stream. When the Kinesis Bridge features is successfully set up, all messages published to specially flagged topics are forwarded to that kinesis stream by the Xively Message Broker.

Follow the guide below to forward specified Xively messages to a Kinesis stream you own.

Step 1: Create/choose a Kinesis Stream

If you haven't already, set up a Kinesis stream in your AWS account.

Step 2: Create an AWS policy for Xively

Create a policy that only allows PutRecords to be called on your selected Kinesis stream. Learn more about Overview of IAM polices.

  • Head to the IAM console:
  • In the navigation column on the left, choose Policies.
  • At the top of the page, choose Create Policy.
  • On the Create Policy page choose Select for Policy Generator.
  • Set the policy permissions to the following:
    • Effect: "Allow"
    • AWS service: "Amazon Kinesis"
    • Actions: "PutRecords"
    • Amazon Resource Name: <KinesisStreamARN>

More on ARNs

Note: KinesisStreamARN should always be in the following format:

  • <region> is the region where the Kinesis stream was created,
  • <account-id> is your AWS account ID (12 digits long number without dashes)
  • <stream-name> is the name of your Kinesis stream

Example: arn:aws:kinesis:eu-west-1:392723209456:stream/kinesis- bridge-test-stream

Read more info about ARNs and more about regions

  • The created policy should look like the example below
	"Version": "2012-10-17", 
  "Statement": {
		"Effect": "Allow",
		"Action": "kinesis:PutRecords", 
    "Resource": "<YourKinesisStreamARN>" 

Step 3: Create an AWS role for Xively

Create a role that allows Xively’s AWS account to access your Kinesis stream and attach the previously created policy to it. This will allow our Message Broker to insert records into your Kinesis stream. Learn more about Creating a role in AWS.

  • Head to the IAM console:
  • In the navigation column on the left, choose Roles.
  • At the top of the page, choose Create New Role.
  • Specify Role Name.
  • On the Select Role Type page:
    • Select “Role for Cross-Account Access”
    • Set Account ID to 462322024086. This is Xively’s production AWS account ID.
    • Set External ID to match your Xively Account ID.
  • On the Attach Policy page select the policy you created in step #2.

Step 4: Tell Xively where to forward your messages

  • Log in to the Xively management app:
  • Under "Integrations", find the Amazon Kinesis Bridge Integrations and click "Set up".
  • Enter the ARN of your Kinesis stream from step 1.
  • Enter the ARM of the Role that you set up in step 3.
  • Click "Connect" (or "Save" if you are editing it).

Step 5: Specify which Xively channels should forward to your Kinesis stream

In the Xively management app, select any channels on any Device Templates that you wish to have forwarded to Kinesis.

  • Go to the Device Template for which you want to enable message forwarding.
  • For each channel that should be forwarded to your Kinesis stream, click the pencil icon to edit (or create new ones).
  • On the Edit channel page (or the Add channel page, if you are adding a new one), under the Kinesis Bridge list, select one of the following:
    • Do not send: Messages sent to the topic are not forwarded.
    • Best Effort: The Xively Message Broker will not hold back the acknowledgement of a published message from a publisher, even if it is not able to write the message to the Kinesis stream. This option is beneficial if you would like to optimize for delivery performance and don’t need guaranteed delivery.
    • Guaranteed delivery The Xively Message Broker will only send PUBACK message to the publishers of the marked topic after the message was successfully delivered to the Kinesis stream.

Tip; We recommend selecting "Best Effort" or "Guaranteed".

"Best Effort" or "Guaranteed"?

These two levels of delivery don't affect the quality of forwarding between Xively and your Kinesis stream. Instead, they indicate how Xively will respond to clients (devices and users) that publish messages to Kinesis-flagged channels using QoS1.

If the level of kinesis delivery is set to Best Effort, the Xively message broker will send a PUBACK to publishing clients immediately upon receiving a message, even before attempting to forward that message to your Kinesis stream.

If the level is set to Guaranteed delivery, the broker won't send a PUBACK to publishers until it was able to write the actual message to the Kinesis stream.

Again, this only matters when publishers are publishing messages to the Xively broker at QoS1, so are expecting a PUBACK in the first place. Clients who are publishing messages at QoS0, and clients who are subscribed to these channels, won't notice any difference between "Best Effort" and "Guaranteed".

Step 6: Consume and process Kinesis messages

The messages sent to the Kinesis stream are base64 encoded and in binary format. To start processing the messages you will need to understand our message format. The below table shows the details of our message structure.

Size (in bytes)



Header version



Timestamp: GUID v1 (Cassandra TimeUUID)



Source name length

<Source name length>


Source name: UTF8-encoded string (Xively id of the entity that published the message)



Source properties length

<Source properties length>


Source properties



Target name length

<Target name length>


Target name: UTF8-encoded string



Target properties length

<Target properties length>


Target properties



Content length

<Content length>


Content body (MQTT payload)

] }
} },
   "contentBody":"TEST MESSAGE"

Appendix: message parsing tool

We have written a NodeJS client library written in TypeScript for the Xively Kinesis Bridge feature. It helps with parsing the envelope transported on the Xively-connected Amazon Kinesis stream.

The library is available at:

npm install xively-kinesis-bridge
import {KinesisBridgeEnvelopeParser} from 'xively-kinesis-bridge';

const base64Data = 'some base64 encoded string';
const parser = new KinesisBridgeEnvelopeParser();

const kinesisBridgeEnvelope : KinesisBridgeEnvelope = parser.parseData(base64Data);

Amazon AWS integration