Using device logs

Record device events, and view lifecycle activity in real time

Introduction

This guide is intended to get you going with the Xively logging service. It will explain how to send some basic log messages from a single device over MQTT as well as retrieve messages that pertain to that device.

This guide will not cover sending logs over HTTP, for that, see Logs in API documentation.

This guide will also not cover the extensive searching capabilities provided by our logging service. Reach out to your Xively rep if you are interested in the advanced query features of device logs.

Assumptions

  • You have a Xively account.
  • You are generally familiar with Xively terminology and concepts

Prerequisites

  • Working cURL or other HTTPS client
  • Device Credentials (ID and Secret)
  • Your personal admin credentials (username and password)

Anatomy of a log

Log messages must match a specific JSON format. We have built this format to be extremely extensible so you can send whatever log messages you want. This guide will call out the major components that are important, for full descriptions, see Log properties.

{
    "sourceId": "71b9da96-f047-4efe-8aae-39eb989ff6a3",
    "accountId": "3db332e6-265b-44dd-a332-6dce033ddd34",
    "sourceType": "deviceId",
    "code": 4,
    "message": "April 5th Test(MQTT)",
    "details": "could not determine largest block. Platform error 4116",
    "severity": "error",
    "tags": ["selfdiagnostic"],
    "sourceTimestamp": 1459877518000
}
  • sourceId The Blueprint ID of your device, this should be the same as the ID for the credential you plan to send the message with.
  • accountId Your Xively Account ID
  • sourceType The source of the log. Optional. Defaults to "deviceId". code This is a customizable error code. You can use this to group errors of specific types or to communicate a code that your logging is already sending. Example: 400 Note: Optional
  • message This is the short semi-human readable description of the log message. Example: MQTT connection successful. Note: Optional
  • details This can be a longer, deeper explanation of the log such as a stack trace. Note: Optional
  • severity This must be one of the standard syslog levels which are: debug, info, notice, warn, error, alert, critical, emergency. Note: Optional
  • tags You can tag your messages with metadata types in order to make them more searchable later. They should be comma separated in an array. Note: Optional
  • sourceTimestamp Timestamp at which the log occurred in milliseconds since epoch. Example: 1459877518123. Note: Optional

Device lifecycle logs

The following logs are attached to devices automatically on operational lifecycle events:

  • Device defined - device is created in Xively.
  • Device activated - device password is generated for the first time.
  • Device associated - an association code is used to move the device into a group (organization).
  • Device deleted - the device is deleted (logs can still be queried for deleted devices).
  • Device connected - the device CONNECTs to the message broker.
  • Device disconnected - the device is disconnected from the messaging broker (either with a DISCONNECT or by failing to respond to an MQTT keepalive).

Sending a log

We will be using Mosquitto to send a log. If you need help using Mosquitto with Xively, refer to the Mosquitto guide.

All devices in Xively have a special channel for logs. You do not need to create this channel, it already exists just for you.

This channel is called _log and the topic takes the following format: xi/blue/v1/<accountId>/d/<deviceId>/_log

Connect to Xively with your device credentials, using the ID as the Client ID and username. Specify this channel for a publish message and use your formatted log message from above as the message. Here is an example mosquitto command for this using our account, device, and log message from above.

mosquitto_pub
    -h broker.xively.com    #Xively broker URL
    -p 8883     #Broker TLS port
    -i 71b9da96-f047-4efe-8aae-39eb989ff6a3    #Device ID
    -u 71b9da96-f047-4efe-8aae-39eb989ff6a3    #Device ID
    -P Ts+5YNJr+AO7gNwdhSECRETJFDu633DEqwkOkv4=    #Device Secret
    -t xi/blue/v1/3db332e6-265b-44dd-a332-6dce033ddd34/d/71b9da96-f047-4efe-8aae-39eb989ff6a3/_log    #Device's log topic
    --cafile ~/path/to/gsr1/cert/gsr1-certificate.crt    #GlobalSign R1 certificate for TLS
    -d     #Turn on mosquitto debugging so we can see the response
    -q 1    #QoS 1 so we know the message was delivered
    -m '{"sourceId":"71b9da96-f047-4efe-8aae-39eb989ff6a3","accountId":"3db332e6-265b-44dd-a332-6dce033ddd34","sourceType":"deviceId","code":4,"message":"April 5th Test(MQTT)","details":"could not determine largest block. Platform error 4116","severity":"error","tags":["selfdiagnostic"],"sourceTimestamp": 1459877518000}'    #Log messaging in format from above
mosquitto_pub -h broker.xively.com -p 8883 -i 71b9da96-f047-4efe-8aae-39eb989ff6a3 -u 71b9da96-f047-4efe-8aae-39eb989ff6a3 -P Ts+5YNJr+AO7gNwdhSECRETJFDu633DEqwkOkv4= -t xi/blue/v1/3db332e6-265b-44dd-a332-6dce033ddd34/d/71b9da96-f047-4efe-8aae-39eb989ff6a3/_log --cafile ~/path/to/gsr1/cert/gsr1-certificate.crt -d -q 1 -m '{"sourceId":"71b9da96-f047-4efe-8aae-39eb989ff6a3","accountId":"3db332e6-265b-44dd-a332-6dce033ddd34","sourceType":"deviceId","code":4,"message":"April 5th Test(MQTT)","details":"could not determine largest block. Platform error 4116","severity":"error","tags":["selfdiagnostic"],   "sourceTimestamp": 1459877518000}'

If everything went well, you should see something like this ...

Client 71b9da96-f047-4efe-8aae-39eb989ff6a3 sending CONNECT
Client 71b9da96-f047-4efe-8aae-39eb989ff6a3 received CONNACK
Client 71b9da96-f047-4efe-8aae-39eb989ff6a3 sending PUBLISH (d0, q1, r0, m1, 'xi/blue/v1/3db332e6-265b-44dd-a332-6dce033ddd34/d/71b9da96-f047-4efe-8aae-39eb989ff6a3/_log', ... (319 bytes))
Client 71b9da96-f047-4efe-8aae-39eb989ff6a3 received PUBACK (Mid: 1)
Client 71b9da96-f047-4efe-8aae-39eb989ff6a3 sending DISCONNECT

If this is what you see, congratulations you have successfully published a log message! If the log message was the right format, you will be able to retrieve it in the next step.

Retrieving a log

Now we will use the logging service's web interface to retrieve the logs for our device. The following will show how to retrieve all the logs for the relevant device. For more information, see Retrieving logs.

It is unlikely you will need to actually do this as all log messages appear, and are sortable directly in the Xively management app. It is included here mostly for future use and for early adopters.

Like all Xively services, the logging service uses JWT authentication. The first step is to use your account user's username and password to Log in via IDM. In response you get your JWT that will be needed in the next API call.

The logging service exists at logging.xively.com. We will be using the search API endpoint: https://logging.xively.com/api/v1/logs/search. You can find examples of searching logs request and response at Retrieving logs.

The logging service uses a query format very similar to ElasticSearch DQL. A query to return all the logs for the device we used above will take the following form:

{
    "query": {
        "bool": {
            "must": [{
                "term": {
                    "sourceId": "71b9da96-f047-4efe-8aae-39eb989ff6a3"  # Your device ID goes here
                }
            }]
        }
    }
}
curl -X POST -H 'Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJleHBpcmVzIjoxNDU5ODg2OTE5MTY5LCJyZW5ld2FsS2V5IjoiSkZSSjROZDFLQktUd2JRM0dKaFMvdz09IiwicmVuZXdhbEV4cCI6MTQ2MTA5NTMxOTE2OSwiY29va2llUHJvcGVydGllcyI6eyJtYXhBZ2UiOjEyMDk2MDAwMDAsImRvbWFpbiI6Ii54aXZlbHkuY29tIiwiSECRETVlLCJodHRwT25seSI6dHJ1ZX0sImlkIjoiNzBlYzllYmUtNDAxZC00ODNhLThkYzItMTgwZWUwYmM5YTM5IiwidXNlcklkIjoiZTZmYTc4MTYtZjY1MC00MjNkLWE3MWUtOGU0ZDAwNmI1YzM4IiwiYWNjb3VudElkIjoiM2RiMzMyZTYtMjY1Yi00NGRkLWEzMzItNmRjZTAzM2RkZDM0Iiwicm9sZXMiOltdLCJjZXJ0IjoiZWMwNWFjZDYtMjViMC00NWRmLWFmMjYtMmYzNDRmZjc5MTI5In0.KW-Lp9N_8RU7GJe82djVWSXNWMbwAq-mE4envYH_oiXasHnV9Rs4M0c_KsS_y9ngozsFkUlPTovOTfYr84qCL10LzeZIkCnPwM9eIsiHUJG7dm-SqTrlVzQ9XWmzyYVZwdYplyWkVJCfih7Ps1zI9StcAX16z_k8Mm5ON4k-0oj0KUxvFmjBXW74rcA3EYC2F0p0HKEx6Zw1E18azW097ug9QlmQCTd44ST2pVaW1UW7KRmWb3FcIK5KE6tn_76y_MpmESJ8CKbFDAlCxIqwU7gSfJ2EEvggQ-OPUz6xRHfcusnB4EXr2UUv20BAIcXq3jJqxitUP8jmI6vL8Scrkw' -H 'Content-Type: application/json' -d '{"query":{"bool":{"must":[{"term":{"sourceId":"71b9da96-f047-4efe-8aae-39eb989ff6a3"}}]}},"from":0,"size":10}' https://logging.xively.com/api/v1/logs/search
curl
    -X POST     # cURL HTTP verb. This is a POST request
    -H 'Authorization: Bearer eyJ0eXAiOiJKV1...TRUNCATED'   # JWT which was returned from IdM /login-user endpoint
    -H 'Content-Type: application/json'     # Required header
    -d '{"query":{"bool":{"must":[{"term":{"sourceId":"71b9da96-f047-4efe-8aae-39eb989ff6a3"}}]}},"from":0,"size":10}'  # Query body from above
    https://logging.xively.com/api/v1/logs/search   # Logging service URL

For this demo we will be using cURL to perform the actual request. We will need the body above, and two headers:

  • Authorization header: "Authorization": "Bearer <your-JWT-here>"
  • Content-Type header: "Content-Type": "application/json"

If this command completes successfully, you should get a response similar to the following:

{
    "took": 17,
    "timed_out": false,
    "_shards": {
        "total": 8,
        "successful": 8,
        "failed": 0
    },
    "hits": {
        "total": 1,
        "max_score": 0.4339554,
        "hits": [{
            "_index": "logs_2016_04_05",
            "_type": "logEntry",
            "_id": "AVPnq8qV19TjRDbGd8Q5",
            "_score": 0.4339554,
            "_source": {
                "sourceId": "71b9da96-f047-4efe-8aae-39eb989ff6a3",
                "accountId": "3db332e6-265b-44dd-a332-6dce033ddd34",
                "sourceType": "deviceId",
                "code": 4,
                "message": "April 5th Test(MQTT)",
                "details": "could not determine largest block. Platform error 4116",
                "severity": "error",
                "tags": ["selfdiagnostic"],
                "sourceTimestamp": "1459880706299",
                "incidents": [null],
                "deviceTemplateId": "a6d9c4de-83a9-4993-b53e-4dc48112bfff",
                "organizationId": "8ed28bf3-49ef-42e9-972d-5fe4d2329928",
                "serviceTimestamp": 1459880708306
            }
        }]
    }
}

As you can see, the endpoint has returned the log message we sent over MQTT earlier. If you see something like this, congratulations! You have successfully sent and retrieved a log message.

Now you can set up your devices to do this programmatically with all your log messages.

Using device logs

Record device events, and view lifecycle activity in real time