Amazon AWS integration

Some users of enterprise systems to want to integrate with Amazon Web Services (AWS) tools to supplement customized parts of their IoT ecosystem.

Xively’s Amazon Kinesis Bridge feature enables you to forward MQTT messages to a Kinesis stream in your AWS account. This way you are able to access a live aggregated stream of specified channels and build custom ingestors/business logic based on these messages that is very easy to scale and can run against billions of messages.

Why might I want to use this?

  • Take advantage of the Simple Notification Service (SNS) to send push notifications to your end user
  • Merge streaming Xively data with information from your other business systems to produce rollups

How do I connect it?

In your Xively account, admins can specify one external (under your own AWS account) Kinesis stream. When the Kinesis Bridge features is successfully set up, all messages published to specially flagged topics are forwarded to that kinesis stream by the Xively Message Broker.

Follow the guide below to forward specified Xively messages to a Kinesis stream you own.

Step 1: Create/choose a Kinesis Stream

If you haven't already, set up a Kinesis stream in your AWS account.

Step 2: Create an AWS policy for Xively

Create a policy that only allows calling PutRecords on your selected Kinesis stream. Learn more about Overview of IAM polices.

  • Head to the IAM console: https://console.aws.amazon.com/iam.
  • In the navigation column on the left, choose Policies.
  • At the top of the page, choose Create Policy.
  • On the Create Policy page choose Select for Policy Generator.
  • Set the policy permissions to the following:
    • Effect: "Allow"
    • AWS service: "Amazon Kinesis"
    • Actions: "PutRecords"
    • Amazon Resource Name: <KinesisStreamARN>

More on ARNs

Note: KinesisStreamARN should always be in the following format:
arn:aws:kinesis:<region>:<account-id>:stream/<stream-name>

  • region is the region where the Kinesis stream was created,
  • account-id is your AWS account ID (12 digits long number without dashes)
  • stream-name is the name of your Kinesis stream

Example: arn:aws:kinesis:eu-west-1:392723209456:stream/kinesis- bridge-test-stream

Read more info about ARNs and more info about regions

  • The created policy should look like the example below
{
	"Version": "2012-10-17", 
  "Statement": {
		"Effect": "Allow",
		"Action": "kinesis:PutRecords", 
    "Resource": "<YourKinesisStreamARN>" 
  }
}

Step 3: Create an AWS role for Xively

Create a role that allows Xively’s AWS account to access your Kinesis stream and attach the previously created policy to it. This will allow our Message Broker to insert records into your Kinesis stream. Learn more about Creating a role in AWS.

  • Head to the IAM console: https://console.aws.amazon.com/iam.
  • In the navigation column on the left, choose Roles.
  • At the top of the page, choose Create New Role.
  • Specify Role Name.
  • On the Select Role Type page:
    • Select “Role for Cross-Account Access”
    • Set Account ID to 462322024086. This is Xively’s production AWS account ID.
    • Set External ID to match your Xively Account ID.
  • On the Attach Policy page select the policy you created in step #2.

Step 4: Tell Xively where to forward your messages

  • Log in to the Xively management app: https://app.xively.com.
  • Under "Integrations", find the Amazon Kinesis Bridge Integrations and click "Set up".
  • Enter the ARN of your Kinesis stream from step 1.
  • Enter the ARM of the Role that you set up in step 3.
  • Click "Connect" (or "Save" if you are editing it).

Step 5: Specify which Xively channels should forward to your Kinesis stream

In the Xively management app, select any channels on any Device Templates that you wish to have forwarded to Kinesis.

  • Go to the Device Template for which you want to enable message forwarding.
  • For each channel that should be forwarded to your Kinesis stream, click the pencil icon to edit (or create new ones).
  • On the Edit channel page (or the Add channel page, if you are adding a new one), under the Kinesis Bridge list, select "Best Effort" or "Guaranteed".
    • Do not send: Messages sent to the topic are not forwarded.
    • Best Effort: The Xively Message Broker will not hold back the acknowledgement of a published message from a publisher, even if it is not able to write the message to the Kinesis stream. This option is beneficial if you would like to optimize for delivery performance and don’t need guaranteed delivery.
    • Guaranteed delivery The Xively Message Broker will only send PUBACK message to the publishers of the marked topic after the message was successfully delivered to the Kinesis stream.

"Best Effort" or "Guaranteed"?

These two levels of delivery don't affect the quality of forwarding between Xively and your Kinesis stream. Instead, they indicate how Xively will respond to clients (devices and users) that publish messages to Kinesis-flagged channels using QoS1.

If the level of kinesis delivery is set to Best Effort, the Xively message broker will send a PUBACK to publishing clients immediately upon receiving a message, even before attempting to forward that message to your Kinesis stream.

If the level is set to Guaranteed delivery, the broker won't send a PUBACK to publishers until it was able to write the actual message to the Kinesis stream.

Again, this only matters when publishers are publishing messages to the Xively broker at QoS1, so are expecting a PUBACK in the first place. Clients who are publishing messages at QoS0, and clients who are subscribed to these channels, won't notice any difference between "Best Effort" and "Guaranteed".

Step 6: Consume and process Kinesis messages

The messages sent to the Kinesis stream are base64 encoded and in binary format. To start processing the messages you will need to understand our message format. The below table shows the details of our message structure.

Size (in bytes)
Type
Description

1

byte

Header version

16

TimeUUID

Timestamp: GUID v1 (Cassandra TimeUUID)

2

word

Source name length

<Source name length>

string

Source name: UTF8-encoded string (Xively id of the entity that published the message)

2

word

Source properties length

<Source properties length>

JSON

Source properties

2

word

Target name length

<Target name length>

string

Target name: UTF8-encoded string

2

word

Target properties length

<Target properties length>

JSON

Target properties

4

dword

Content length

<Content length>

blob

Content body (MQTT payload)


{
   "headerVersion":1,
   "timeUUID":"95c475af-472e-e611-ab76-0acef479ae3b",
   "sourceNameLength":102,
   "sourceName":"xi/blue/v1/647aa41f-6c4c-4632-a104-f56e6f1e9d2b/d/821550a0-9d3e-
4356-9f2d-aec4e0a919b2/SimpleChannel_2",
   "sourcePropertiesLength":0,
   "sourceProperties":"",
   "targetPropertiesLength":311,
   "targetProperties":{
      "SchemaVersion":1,
      "EntityType":"topic",
      "Owner":{
         "Id":"1b8adadb-657e-4732-a0ac-73012ec77699",
         "Properties":{
            "SchemaVersion":1,
            "EntityType":"device",
            "AccountId":"647aa41f-6c4c-4632-a104-f56e6f1e9d2b",
            "TemplateId":"26116fe5-d7a5-4294-a80c-39299e97a44f",
            "OrganizationIds":[
               "0db639d3-55b8-47b4-bc8b-bacd54bfa3a2"
] }
} },
   "contentLength":12,
   "contentBody":"TEST MESSAGE"
}

Appendix: message parsing tool

We have written a NodeJS client library written in TypeScript for the Xively Kinesis Bridge feature. It helps with parsing the envelope transported on the Xively-connected Amazon Kinesis stream.

The library is available at: https://github.com/xively/xively-kinesis-bridge-client.

npm install xively-kinesis-bridge
import {KinesisBridgeEnvelopeParser} from 'xively-kinesis-bridge';

const base64Data = 'some base64 encoded string';
const parser = new KinesisBridgeEnvelopeParser();

const kinesisBridgeEnvelope : KinesisBridgeEnvelope = parser.parseData(base64Data);

Amazon AWS integration