gRPC WebRTC Signalling Service

Introduction

The P2P Video dashboard widget uses WebRTC to establish a direct connection between robot and browser, allowing video to be streamed to the point of consumption with lowest possible latency. Setting up a WebRTC link requires the sender and receiver to each share information allowing each to find and connect to the other; because during this setup process, no direct connection between sender and receiver yet exists, a signalling service is required to facilitate communication. The Robot Automation platform uses a gRPC service codenamed 'Pigeon' for this purpose.

A brief summary of the Robot Automation WebRTC setup process is show below: to follow this, you will need to understand a bit about how WebRTC works in general, including the terms SDP and ICE. You will also need to know how to use the WebRTC client library of your choice to do such things as discover candidates, generate SDP descriptors, set local/remote offer/answer pairs, and any other mechanics to initiate a connection.

1. Client uses the regular gRPC Telemetry Message Service to discover information about the video streams available on the target robot, which are published to the /video/camera topic.
2. Client determines the unique 'stream ID' string which identifies the video source which should be connected to: this has the form "/project/callsign/streamName".
3. Client connects to the Pigeon signalling service to register as an 'operator'.
4. Client constructs a SDP (Session Description Protocol) descriptor string which describes the type of data the client would like to receive, and uses the Pigeon service to send this "offer" to the target robot, addressed by stream ID.

Hint

The SDP offer and answer which the agent receives and generates for a given P2P video session are published in the /video/camera topic, along with other stream diagnostic information. You can use the value of the SDP offer from a video session streaming to the Robot Automation Portal as a template to use as a basis for your own client.

Note

The DroneDeploy Agent only supports WebRTC streaming using the H264 video encoding; attempting to set up a WebRTC session using an SDP definition which does not accept H264 format video will not be successful.

1. Client uses its native WebRTC library to detect its local ICE candidates (including reflex and relay candidates in addition to host candidates), and uses the Pigeon service to send these candidate addresses to accompany the SDP descriptor.

Hint

DroneDeploy operates a STUN/TURN server, alongside the Pigeon signalling server, which you can use to both generate reflex ICE candidates, and allow relay ICE connectivity. The server requires credentials from authorised clients. If you wish to use this server with your own client, then contact DroneDeploy to discuss usage terms.

1. The robot receives the SDP and ICE candidates, processes this information, and replies with an SDP "answer" which includes the robot's own ICE candidates.
2. Upon receipt of this "answer", the client treats the included ICE candidate information as 'remote' candidates, and begins WebRTC ICE connection negotiation. From this point onward, the signalling service is no longer required.
3. ICE connection negotiation completes, and a WebRTC stream is established over which video begins flowing.

Protocol and Server Selection

Please contact DroneDeploy directly to obtain a copy of the signalling service gRPC protocol definition. Refer to the gRPC documentation regarding how to use the gRPC tools to compile this protocol file into usable library code in your preferred language.

The signalling service which this protocol allows access to is provided by a gRPC server: there is a single service implementation which you will be using:

• The signalling cloud server, a part of the Robot Automation cloud platform (api2.rocos.io:443 for most regions, and used for regular cloud platform connectivity).

Setting Up a Connection

The signalling server will only accept incoming gRPC connections which use some specific parameters, provided by way of Dial Options, Call Options, and Call Context Meta-data:

Keep-alive Parameters

Ensure that you Dial the connection using compatible keep-alive options, or the server may hang up unexpectedly:

PermitWithoutStream: true
Time:                5 sec
Timeout:             5 sec

Credentials and Authentication

You must use a gRPC connection with a secure channel. A default set of TLS/SSL transport credentials must be provided as a Dial Option.

You will need to include an appropriate JWT authentication token as a Call Option each time you make a request to the server (which you can do using the PerRPCCredentials option). Alternatively, this can be provided as a Meta-data field in the Call Context.

Context

Each request to the server must be accompanied with appropriate meta-data:

Field	Type	Description
r-cs	string	The robot callsign of the robot this request relates to.
r-p	string	The project ID of the project containing the robot in question.
authorization	string	JWT token for authentication.
grpc-timeout	int	Optional. This value can be used to set a deadline for the gRPC request to be completed within, after which the request will be aborted.

Using the Signalling Service

After ensuring the connection parameters are correct, you should be able to dial the signalling server and create a new client stub OperatorClient, upon which you can perform service requests. Refer to the diagram below, which shows an overview of the ICE setup process using the signalling service:

Because WebRTC is a peer-to-peer connection protocol, only one client can be receiving each WebRTC video stream from the robot at a time. If you connect while another user is already viewing the video, the existing user's video stream will be terminated abruptly. To check whether a particular video stream ID currently has any clients streaming it, you can call get(), which will return information about a named video stream.

To begin a new WebRTC session, register with the signalling service using connect(), which will allow you to receive messages from the signalling service related to the session (namely the 'answer' SDP coming back from the robot). You will need to provide a globally unique 'client ID' to identify this particular video session.

As described in steps (4) & (5) of the introduction above, you now need to collect the data needed to support the ICE process; namely your SDP 'offer' and list of ICE candidates. Normally, if you want your video to work consistently both in a LAN environment as well as over the internet (through NAT routers), your list of ICE candidates should include at least one host candidate, one reflex candidate, and one relay candidate as a fallback.

Along with the SDP, you will also need to send the agent a "video command" string: this string should describe a GStreamer1.0 pipeline, as described in Video.

First use addIceCandidate() to convey your local ICE candidates to the signalling service, then use offer() to send your SDP offer and the matching video command. Make sure to use the same client ID as you used previously. In your WebRTC library, set the SDP which you sent as your local session descriptor.

Once the Robot has received and processed the request, you will receive a message back via the connect() method you started earlier. Examination of this message should reveal it contains an SDP 'answer' from the Robot: this contains both the stream definition an also the Robot's own ICE candidates.

In your WebRTC library, set the answer received as the remote session descriptor. At this point, the WebRTC library should take over and begin the ICE process.