REST JSON:API for Commissioner and Diagnostics

Summary

An extended REST API functionality providing capabilities for commissioning and on-mesh diagnostics to generic off-mesh HTTP/HTTPS clients. The implementation is guided by the JSON:API specification. All the resources are described in openapi.yaml specification, also see rendered openapi.yaml. The intent of this document is to provide basic usage and additional background information.

Use cases

The API targets the following consumers and use cases:

• Access to commissioning and network diagnostics for HTTP/HTTPS clients with minimal code dependency (for example, no ot-commissioner needed)
• Field engineers for instant issue analysis and to visualize the mesh network topology
• Continuous monitoring of network quality, for example, localisation of weak mesh regions
• Developers for debugging and understanding network topology
• Testers

A consumer may extract the data from the API and visualize it elsewhere.

Background

• MeshCop needs DTLS and Thread specific sources or binaries
• RESTful HTTP resources following JSON:API specification eases interpretation of the large number of attributes provided and eliminates dependencies to CoAP and DTLS.
• Enables tracking of history on-demand

REST JSON:API resources

/api/devices

The list of Thread network devices is provided as JSON:API collection on the devices resource /api/devices. The collection provides collection items of type threadDevice, one per active Thread device found during discovery. The threadDevice items shall serve as an inventory of devices and provide primarily static information, in particular a static deviceId for each device with some key attributes allowing to identify the device. The collection may contain devices that became inactive. It is discovered, updated, or deleted on-demand.

The Thread Border Router learns the list of network devices on-demand from the active network and returns the full collection or individual items in response to GET requests. The collection may be updated or deleted by a client on request, see updateDeviceCollectionTask in section api/actions below. A DELETE request to /api/devices removes the cached collection.

Device identification

The device collection provides attributes for identifying and selecting a device. Diagnostics are accessible only via mesh local addresses (ML-EID or RLOC16) for on-mesh clients. However, none of them is known to a non-Thread client. Therefore, non-Thread clients may use the deviceId in further diagnostic actions for querying diagnostic attributes from a specific device.

The MAC extended address serves as a stable identifier within the Thread network. In addition, the ML-EID IID (interface identifier) acts as a stable on-mesh identifier, while RLOC16 may change due to mesh network reorganization. Both ML-EID IID and the MAC extended address can be used as the stable identifier for collection items of type threadDevice. Since ML-EID IID is challenging to extract, we use the MAC extended address as the value for the device item deviceId and provide the ML-EID IID as an attribute of threadDevice items. OTBR REST JSON:API uses the ML-EID IID and the on-mesh prefix from the Thread network dataset to construct a mesh-local IPv6 unicast address for gathering on-mesh diagnostics TLVs from the corresponding device.

This proposed deviceId scheme and the limited number of static attributes minimize resource consumption on the OTBR, while still allowing a client to correlate a deviceId to device information provided elsewhere, for example, via a hostname also provided by SRP. A very constrained OTBR may chose to further reduce resource consumption for the device collection and only provide the stable deviceId and the learned ML-EID IID but no further details. A client would then have to refer to the diagnostics for gathering additional information.

A timestamp attribute created should be used when the documents are persisted. After modifications to the document the timestamp attribute updated is added.

Example collection comprising documents of type threadDevice

An example response to curl -G http://otbr.local/api/devices -H 'Accept: application/vnd.api+json' for a network comprising two Thread devices may look similar to:

{
  "data": [
    {
      "id": "de62e016db392476",
      "type": "threadDevice",
      "attributes": {
        "extAddress": "de62e016db392476",
        "mlEidIid": "1d934f57e21e35",
        "omrIpv6Address": <Ipv6Address>,
        "hostname": "<hostname defined by SRP>",
        "role": "router",
        "mode": {
          "fullThreadDevice": true,
          "rxOnWhenIdle": true,
          "fullNetworkData": true
        },
        "created": <ISO 8601 timestamp>,
        "updated": <ISO 8601 timestamp>
      }
    },
    {
      "id": "de62e016db392477",
      "type": "threadDevice",
      "attributes": {
        "extAddress": "de62e016db392477",
        "mlEidIid": "2ea45067f32f46",
        "omrIpv6Address": <Ipv6Address>,
        "eui": "<eui>",
        "hostname": "<hostname defined by SRP>",
        "role": "router",
        "mode": {
          "fullThreadDevice": true,
          "rxOnWhenIdle": true,
          "fullNetworkData": true
        },
        "created": <ISO 8601 timestamp>,
        "updated": <ISO 8601 timestamp>
      }
    },
  ],
  "meta": {
    "collection": {
      "offset": 0,
      "limit": 200,
      "total": 2
    }
  },
}

The threadDevice document does only contain attributes which barely change. For now, hostname attribute values can only be provided if both, SRP-server and REST API, are hosted on the same OTBR.

/api/diagnostics

The diagnostics resource serves a collection of diagnostic items. The implementation comprises items of type networkDiagnostics corresponding to Specification Chapter 10.11 and the TLVs listed in Table Diagnostic core attributes. In addition it serves items of type energyScanReport.

The value of the id field of the collection items is a UUID.

The TLVs are collected and the items appended to the collection based on requests to the /api/actions resource.

/api/actions

An HTTP client may send requests to the actions resource that trigger diagnostic requests into the Thread network or requests to an on-mesh commissioner for onboarding new devices onto the Thread network. Such requests typically have a longer response time and results may be obtained indirectly after some processing time.

Supported action requests to /api/actions are:

• POST
  • addThreadDeviceTask - starts the on-mesh commissioner and adds the joiner candidate into the joiner table.
  • getEnergyScanTask - starts the commissioner and sends energy scan requests to the destination.
  • getNetworkDiagnosticTask - sends diagnostic requests and diagnostic queries to the destination.
  • resetNetworkDiagCounterTask - resets the network diagnostic MLE and/or MAC counters.
  • updateDeviceCollectionTask - discovers the Thread network and adds or updates any responding attached devices in the /api/devices collection.
• GET | DELETE the full actions collection.

Currently, getNetworkDiagnosticTask and getEnergyScanTask must contain a destination attribute which should equal a deviceId contained in the api/devices collection. If the deviceId does not exists, we currently check whether the destination might be a valid mlEidIid or rloc16.

A DELETE request on api/actions cancels an ongoing action and deletes all collection items.

Requests and responses

After receiving the request, ApiActionsPostHandler() parses the request body and validates the received data before we attempt to perform processing. The reception of an accepted request is confirmed with a response status ok and the requests body including a new id for the new request to allow tracking of progress.

Processing of request

A request status attribute is pending when queued and processed one-by-one, except for addThreadDeviceTask which are processed in parallel. The status of a request in processing is active. After processing the request is completed the next queued request is processed.

Request results

A request status attribute turns completed after successful processing, and a reference to the request result is provided as a relationship attribute. Status attribute is set to stopped after timeout, or otherwise failed.

The actual request result must be collected in a separate request to the resource given in the relationship object.

An example response to curl -G http://otbr.local/api/actions -H 'Accept: application/vnd.api+json' after a previous POST of getNetworkDiagnosticTask has completed may look similar to:

{
  "data": [
    {
      "type": "getNetworkDiagnosticTask",
      "attributes": {
        "destination": "32ba6e34fd4c0299",
        "types": [
          "extAddress",
          "rloc16",
          "leaderData",
          "ipv6Addresses",
          "macCounters",
          "batteryLevel",
          "supplyVoltage",
          "channelPages",
          "maxChildTimeout",
          "lDevIdSubject",
          "iDevIdCert",
          "eui",
          "threadVersion",
          "vendorName",
          "vendorModel",
          "vendorSwVersion",
          "threadStackVersion",
          "children",
          "childIpv6Addresses",
          "routerNeighbors",
          "mleCounters"
        ],
        "timeout": 60,
        "status": "completed"
      },
      "id": "54dc649d-c257-4f91-9071-a35a678b6249",
      "relationships": {
        "result": {
          "data": {
            "type": "diagnostics",
            "id": "0a97ef16-1997-43dd-91c4-7fbcb1ec6713"
          }
        }
      }
    }
  ],
  "meta": {
    "collection": {
      "offset": 0,
      "limit": 100,
      "total": 1,
      "pending": 0
    }
  }
}

Examples of general resource features

The request must comprise a header field Accept set to application/vnd.api+json for JSON:API content format, or application/json for JSON content format in the response. Accordingly, the response comprises the header field Content-type.

Get collection

All items of the collection are gettable in a single request, for example:

curl -G http://<otbr-address>/api/<collectionName> -H 'Accept: application/vnd.api+json'

Get collection item by id

Individual items are gettable from the collections, for example:

curl -G http://<otbr-address>/api/<collectionName>/<itemId> -H 'Accept: application/vnd.api+json'

Get items by type

The collection can be filtered by type of the items, for example:

curl -G http://<otbr-address>/api/<collectionName>?fields[<type>] -H 'Accept: application/vnd.api+json'

Get items by type with limited attributes

The collection can be filtered by type of the items, for example:

curl -G http://<otbr-address>/api/<collectionName>?fields[<type>]=attribute1,attribute2 -H 'Accept: application/vnd.api+json'

Detailed request examples are provided in the test folder as a “Bruno Request Collection”. Use the free open-source API client Bruno.

Key file description

• actions_list.cpp, actions_list.hpp

  Implements the action collection, derived from BasicCollection.

• commissioner_manager.cpp, commissioner_manager.hpp

  Handles starting and stopping the on-mesh commissioner as needed by the specific queued action items.

• network_diag_handler.cpp, network_diag_handler.hpp

  Handles collecting network diagnostic TLVs on top of OT core netdiag and meshdiag API.

• rest/actions/*

  Implement the action item stored in action collection. The action item is derived from BasicActions class, which is derived from BasicCollectionItem.

• rest_generic_collection.cpp, rest_generic_collection.hpp

  Implements the BasicCollectionItem and BasicCollection classes.

• rest_*_coll.cpp, rest_*_coll.hpp

  Implements the specific devices and diagnostics collection classes derived from BasicCollection class.

• uuid.cpp, uuid.hpp

  Implements functionality to handle UUID (generation, parse, unparse) used for collection items.

• tests/restjsonapi/*

  Test setup including implementation for various test cases.

How to rebuild and experience this work

1. Follow the Border Router guide to build and install OTBR as usual, for example, natively on a Raspberry Pi.

2. To monitor logs (Errors, Warnings, Info), open a separate terminal instance and use following command:

   tail -f /var/log/syslog | grep otbr
   

3. Send a POST request using BRUNO or CURL, for example, to join a new device into your network.

   curl -X POST -H 'Content-Type: application/vnd.api+json' http://localhost:8081/api/actions -d '{"data": [{"type": "addThreadDeviceTask", "attributes": {"eui": "6234567890AACDEA", "pskd": "J01NME", "timeout": 3600}}]}' | jq
   

   should return

   {
     "data": [
       {
         "id": "2d5a8844-b1bc-4f02-93f0-d87b8c3b4e92",
         "type": "addThreadDeviceTask",
         "attributes": {
           "eui": "6234567890AACDEB",
           "pskd": "J01NME",
           "timeout": 3600,
           "status": "pending"
         },
       }
     ]
   }
   

4. You may check the status and get the full collection of actions.

   curl -X GET -H 'Accept: application/vnd.api+json' http://localhost:8081/api/actions | jq
   

   should return

   {
     "data": [
       {
         "id": "2d5a8844-b1bc-4f02-93f0-d87b8c3b4e92",
         "type": "addThreadDeviceTask",
         "attributes": {
           "eui": "6234567890AACDEB",
           "pskd": "J01NME",
           "timeout": 3600,
           "status": "pending"
         }
       }
     ],
     "meta": {
       "collection": {
         "offset": 0,
         "limit": 100,
         "total": 1
       }
     }
   }
   

5. View the entry added to the commissioner's table sudo ot-ctl commissioner joiner table and expect:

   | ID                    | PSKd                             | Expiration |
   +-----------------------+----------------------------------+------------+
   |      6234567890aacdea |                           J01NME |    3459027 |
   Done
   

6. Start your joiner and after a few seconds repeat above last two steps.

7. For further viewing of the diagnostic endpoints use the Bruno request collection, which you can find in the folder tests/restjsonapi.

8. For running the included test script install Bruno-Cli and run the bash script on your border router:

   cd tests/restjsonapi
   source ./install_bruno_cli
   ./test-restjsonapi-server
   

   All tests should pass.

9. Validating the openapi.yaml specification against the implementation.

   Make sure Schemathesis v4.25.0 or later (4.25.x series) is installed; earlier 4.24.x releases crash during the examples phase:

   pip install --upgrade "schemathesis~=4.25.0"
   

   Then run the tests based on the OpenAPI specification.

   Due to limitations of Schemathesis (v4), we can only filter on the ‘phases’ parameter as follows:

   schemathesis run src/rest/openapi.yaml --url=http://localhost:8081 --include-path-regex="/api/.*" --phases=coverage,examples,fuzzing
   

   Expected results:

   • phases=examples pass
   • phases=coverage fails with wrong error code for request method TRACE (TRACE is not needed and not supported with cpp-httplib)
   • phases=fuzzing causes more failures, in part due to insufficiently strict schema definitions, for example, for SparseFieldset queries

Diagnostic core attributes

The table below is derived from the Thread Specification. The column Short Name defines the attribute names.

Attribute names