Diagnostics

The Diagnostics component provides information about the DroneDeploy Agent itself and about the overall status of the robot. It serves two distinct purposes:

1. Agent diagnostics and instrumentation — visibility into the health and performance of the Agent process (logs, memory, goroutines, profiling), used to diagnose issues or to optimise telemetry performance in view of your network conditions and application requirements.
2. Robot status reporting — a single, canonical report of the robot's health and readiness, which the platform uses to drive the status indicators shown next to each robot, and to make decisions such as not preempting a robot that is busy.

The Diagnostics component replaces the older Rocos Diagnostics (rocos) component. Generally, it exposes the same telemetry and services as the rocos component; but there are a few differences - some topics have been renamed, or their schema altered in order to make them more convenient to use. The new component also has some new functionality: in particular, its robot/status topic supersedes the legacy /rocos/agent/telemetry/heartbeat topic, but rather than only indicating that the robot is online, it has mechanisms for also conveying what state the robot is in.

Note

The diagnostics component cannot be disabled; its configuration can be modified (see Configuration), but in the event the component is not included in the configuration, the diagnostics component will instead run with default settings.

Topic and service naming

Throughout this page, telemetry topics and service endpoints are shown with the /diagnostics/... prefix — for example /diagnostics/robot/status. This prefix is the component's configured name (see Configuration). Every topic and service the component exposes is published beneath this prefix.

Note

The name of the component can be changed via it's configuration; however doing so may cause some functionality which depends on reading the diagnostics component to stop working properly.

Configuration

The Diagnostics component is configured via an appropriate section in the Agent's agent-settings.json configuration file. Unlike most components, it is always enabled — even if the enabled flag is set to false, or if there is no matching section in the configuration file (in which case it uses default settings). For more information on how to configure the Agent's components, see Agent Configuration.

...
"components": [
...
  {
    "enabled": true,
    "id": "diag-host",
    "settings": {
      "name": "diagnostics",
      "minLogBackoff": "1s",
      "status": {
        "defaultHealth": "normal",
        "defaultReadiness": "idle",
        "polling": {
          "missionActivity": {
            "readinessSource": "/dji/mission/state",
            "readinessValue": "msg.active ? \"busy\" : \"idle\"",
            "readinessTimeout": "10s"
          }
        }
      }
    }
  },
...
]

The component understands the following top-level settings:

Name	Description	Default	Units
name	The name under which the component publishes all of its topics and services (the /<name>/... prefix). Leave this as diagnostics.	diagnostics
minLogBackoff	A minimum backoff period enforced after sending each log telemetry message. Use this to prevent runaway feedback loops ("death spirals") when performing telemetry diagnostics at high log levels.	0	duration (e.g. 1s)
status	Configuration for robot status reporting. See that section for the nested fields.

note

The id field for the diagnostics component is diag-host when running on normal robots, but the default name of the component is diagnostics. This is different to most of the other agent components, where the default id and default name are the same.

Robot status reporting

The Diagnostics component maintains a single, aggregated view of the robot's status and publishes it on the robot/status topic. The status is distilled into two orthogonal signals, deliberately kept small so that they are meaningful across every kind of robot:

• Health — is anything wrong with the robot? One of:
  • normal — the robot is healthy.
  • abnormal — something needs attention, but the robot is still functional.
  • critical — the robot requires immediate attention.
  • unknown — health has not been determined.
• Readiness — is the robot available to take on work? One of:
  • idle — the robot is available.
  • busy — the robot is occupied (for example, part-way through a mission) and should not be interrupted.
  • unknown — readiness has not been determined.

How status is determined

The reported status is aggregated from one or more contributors. A contributor is any source that has an opinion about the robot's health or readiness. There are three kinds:

1. Other Agent components. Strongly-opinionated components (for example, the DJI component) can report their own view of the robot's health or readiness directly.
2. Status polling. The Diagnostics component can be configured to watch one or more telemetry topics and derive health or readiness from them. See Polling telemetry for status.
3. Manual service calls. External callers can set a health or readiness value by invoking the status services.

The aggregate is always the worst value reported by any live contributor:

• Health: critical > abnormal > normal > unknown.
• Readiness: busy > idle > unknown.

This means a single component reporting critical health will drive the whole robot's reported health to critical, and a single component reporting busy will mark the robot busy.

Default status

When no contributor expresses an opinion on an axis, the configured default for that axis is used. Defaults are a fallback for an under-instrumented robot — they are not a floor: a contributor reporting normal always wins over a default of critical.

Name	Description	Default
status.defaultHealth	The health reported when no contributor has an opinion on health. Accepts a string (normal, abnormal, critical, unknown) or the equivalent integer.	normal
status.defaultReadiness	The readiness reported when no contributor has an opinion on readiness. Accepts a string (idle, busy, unknown) or the equivalent integer.	idle

The default values should be used for two purposes:

• For a robot that you expect to be instrumented, you are able to distinguish between normal reporting and a failure of the status reporting mechanism.
• For a robot that you know is not configured or capable of reporting status in a useful way, that the robot still appears as normal to users.

Polling telemetry for status

For robots whose components don't report status directly, the Diagnostics component can derive status by polling existing telemetry topics. Each entry under status.polling is a named rule that can drive readiness, health, or both. For each axis, a rule supplies a source telemetry URI to watch and a value expression — a JavaScript expression that receives the latest message as msg and must evaluate to one of the valid status values (as a string or an integer).

Name	Description
healthSource	The telemetry URI to watch in order to determine health.
healthValue	A JavaScript expression, evaluated against the latest message (msg), that returns a health value (e.g. "normal", "critical", or 1).
healthTimeout	If set, reverts this rule's health contribution to unknown when no message has been received on healthSource within this period. A zero or unset timeout disables the watchdog. Useful when the absence of messages is itself a signal.
readinessSource	The telemetry URI to watch in order to determine readiness.
readinessValue	A JavaScript expression, evaluated against the latest message (msg), that returns a readiness value (e.g. "busy", "idle", or 2).
readinessTimeout	As healthTimeout, but for this rule's readiness contribution.

For example, the following rule marks the robot busy whenever its mission state reports an active mission, and reverts to unknown if no mission-state message arrives for 10 seconds:

"polling": {
  "missionActivity": {
    "readinessSource": "/dji/mission/state",
    "readinessValue": "msg.active ? \"busy\" : \"idle\"",
    "readinessTimeout": "10s"
  }
}

Setting status manually

External callers (such as a flow, or an operator tool) can contribute to the robot's status by invoking the following services. Each contribution is keyed by a caller-chosen id, so the same caller can update its contribution repeatedly without stacking up duplicates. The reason is a free-text explanation that is surfaced on the contributors topic to help trace where a value came from.

Service	Request	Description
/diagnostics/robot/status/setHealth	{ "id": string, "health": string | int, "reason": string }	Contributes a health value under the given id.
/diagnostics/robot/status/setReadiness	{ "id": string, "readiness": string | int, "reason": string }	Contributes a readiness value under the given id.

The robot/status topic

The aggregated status is published on /diagnostics/robot/status. This is the canonical topic for "what state is this robot in", and replaces the legacy heartbeat topic. Like any telemetry topic, the rate at which it is delivered is controlled by the subscription (for example by appending ?int=1s); receiving it also confirms the robot's connection to the telemetry broker.

The values are encoded as integers on the wire:

Field	Meaning	Values
health	The aggregate health.	0 = unknown, 1 = normal, 2 = abnormal, 3 = critical
readiness	The aggregate readiness.	0 = unknown, 1 = idle, 2 = busy

Example payload:

{
  "health": 1,
  "readiness": 2
}

Inspecting contributors

To understand why the aggregate has a particular value, the component publishes a per-contributor breakdown on /diagnostics/robot/status/contributors. Each entry identifies a single live contributor, the component it belongs to, its current value, and its reason. Health and readiness contributors are listed in separate arrays.

Example payload:

{
  "health": [
    {
      "id": "battery",
      "component": "dji",
      "value": 2,
      "reason": "battery below 20%"
    }
  ],
  "readiness": [
    {
      "id": "missionActivity",
      "component": "diagnostics",
      "value": 2,
      "reason": "missionActivity"
    }
  ]
}

Agent diagnostics and instrumentation

The Diagnostics component exposes telemetry and services that report on the health and performance of the Agent process itself. These are most useful when investigating Agent issues or tuning telemetry behaviour.

note

This section currently lists the available topics and services. Guidance on how to combine them to diagnose specific classes of problem (for example, identifying a memory leak, or attributing high network usage to a particular component) will be added here over time.

Telemetry topics

Data URI	Description	Unit
/diagnostics/host/logs	A stream of the Agent's console log entries, allowing the Agent's log output to be reconstructed remotely. Sending is rate-limited by minLogBackoff.
/diagnostics/host/diagnostics/numGoRoutines	The number of concurrent routines the Agent is running. A continually climbing value can indicate a leak.
/diagnostics/host/diagnostics/heapAlloc	The amount of heap memory currently allocated to the Agent.	B
/diagnostics/host/diagnostics/proc	Process-level information about the Agent (such as CPU and memory usage).

Services

Service	Description
/diagnostics/host/quit	Requests the Agent to shut down.

Clock synchronisation

The Diagnostics component exposes services for inspecting and correcting the Agent host's clock synchronisation. See also Clock Synchronisation.

Service	Description
/diagnostics/clock/sync/resync	Re-checks the host's time synchronisation and returns the current clock offset (in nanoseconds).
/diagnostics/clock/sync/resetFilter	Resets the clock synchronisation filter.