GStreamer

The Agent's gstreamer component gets the video/audio data from a gstreamer pipeline to publish to a specified topic by pipeline names.

Requirements

In order to use the gstreamer component, the gstreamer plugins are required to be installed on the robot. Please use this guide to installed them, if they are not already installed. These plugins are external libraries and binaries allowing to launch a gstreamer pipeline, and stream the video/audio date to the agent.

Configuration

Below is an example of a section to the robot settings, in order to enable the gstreamer component. This example includes two pipelines, one for video-test (Used for debugging of video issues), and a front-cam pipeline to stream a usb-camera to the portal.

{
    "enabled": true,
    "id": "gstreamer",
    "name":"gstreamer",
    "settings": {
      "logLevel": 6,
      "pipelines": {
      "video-test": {
          "envVariables": {
            "GST_DEBUG": 9
          },
          "command": "videotestsrc ! video/x-raw,framerate=${$channels.video.framerate},width=${$channels.video.width},height=${$channels.video.height} ! x264enc bframes=0 speed-preset=veryfast key-int-max=30 bitrate=${$channels.video.bitrate} ! video/x-h264,stream-format=avc,alignment=au,profile=constrained-baseline ! queue silent=true ! rtph264pay mtu=1400 config-interval=-1 ! application/x-rtp,media=video,clock-rate=${$channels.video.clockRate},encoding-name=${$channels.video.encodingName},ssrc=(uint)${$channels.video.SSRC} ! queue silent=true ! udpsink host=127.0.0.1 port=${$pipeline.port} sync=false async=true",
          "logLevel": 5,
          "channels": {
            "video": {
              "width": 640,
              "height": 480,
              "framerate": "15/1",
              "bitrate": 100
            }
          }
        },
        "front-cam": {
          "envVariables": {
            "GST_DEBUG": 2
          },
          "command": "v4l2src device=/dev/video0 ! video/x-raw,format=YUY2,width=${$channels.video.width}, height=${$channels.video.height}, framerate=${$channels.video.framerate} ! queue ! videoconvert ! video/x-raw,format=I420 ! x264enc bframes=0 speed-preset=veryfast key-int-max=60 ! video/x-h264,stream-format=byte-stream ! queue silent=true ! rtph264pay mtu=1400 config-interval=-1 ! application/x-rtp,media=video,clock-rate=${$channels.video.clockRate},encoding-name=${$channels.video.encodingName},ssrc=(uint)${$channels.video.SSRC} ! queue silent=true ! udpsink host=127.0.0.1 port=${$pipeline.port} sync=false async=true",
          "logLevel": 5,
          "maxRetries": 20,
          "channels": {
            "video": {
              "width": 640,
              "height": 360,
              "framerate": "30/1",
              "bitrate": 100
            }
          }
        }
      }
    }
  }

The Settings section of the config can include the fields below:

Field	Description	Default
enabled		false
name	The gstreamer component's instance name.	gstreamer
logLevel	The component's log level	4
pipelines	A JSON object that specifies the Gstreamer pipelines.

The gstreamer component could have multiple pipelines each for a dedicated media (e.g. front or back camera). The JSON key of the pipeline (front-cam in the example above) is considered the name of the pipeline. The config fields of a pipeline can include the fields below:

Field	Description	Default
command	The Gstreamer pipeline name
envVariables	A JSON object that specifies a set of environment variable pairs ({"key":"value"}) to get passed to the dedicated pipeline Gstreamer processes.
logLevel	The pipeline's specific log level.	4
maxRetries	The maximum number of retries the gstreamer process will have after unsuccessful launch.	-1 (infinity)
maxRetryBackoff	The maximum wait time between each retry. Starts from 100ms up until configured value, and exponentially grows each time. The value is parsed from a string to the duration.	100ms
executable	The name of Gstreamer executable binary.	gst-launch-1.0
channels	The channel (audio/ video) specific sets of value for each pipeline, which could be replaces in ${$channels.video.XXX} fields (See example above).

Please note that upon the subscription of the topic, the agent will launch gst-launch-1.0 as a separate process, and pass in the configured pipeline. The pipeline process will be in a inherited environment, unless environment variables field is configured, and passed to the process execution.

Pipeline Capabilities

As shown in the examples above, each GStreamer pipeline definition should end with a udpsink element, where the target port is determined by a parameterised value which the Agent will select automatically at runtime:

udpsink host=127.0.0.1 port=${$pipeline.port}

This sink element will allow the video output from the pipeline to be captured by the agent.

The sink capabilities are quite strict; the pipeline should generate RTP encapsulated H264 video, satisfying the following capabilities :

video/x-h264,stream-format=byte-stream,alignment=au,profile=constrained-baseline