Skip to main content

GStreamer

The Agent's gstreamer component gets the video/audio data from a gstreamer pipeline to publish to a specified topic by pipeline names.

Requirements​

In order to use the gstreamer component, the gstreamer plugins are required to be installed on the robot. Please use this guide to installed them, if they are not already installed. These plugins are external libraries and binaries allowing to launch a gstreamer pipeline, and stream the video/audio date to the agent.

Configuration​

Below is an example of a section to the robot settings, in order to enable the gstreamer component. This example includes two pipelines, one for video-test (Used for debugging of video issues), and a front-cam pipeline to stream a usb-camera to the portal.

{
    "enabled": true,
    "id": "gstreamer",
    "name": "gstreamer",
    "settings": {
        "logLevel": 6,
        "pipelines": {
            "video-test": {
                "envVariables": {
                    "GST_DEBUG": 9
                },
                "command": "videotestsrc ! video/x-raw,framerate=${$channels.video.framerate},width=${$channels.video.width},height=${$channels.video.height} ! x264enc bframes=0 speed-preset=veryfast key-int-max=30 bitrate=${$channels.video.bitrate} ! video/x-h264,stream-format=avc,alignment=au,profile=constrained-baseline ! queue silent=true ! rtph264pay mtu=1400 config-interval=-1 ! application/x-rtp,media=video,clock-rate=${$channels.video.clockRate},encoding-name=${$channels.video.encodingName},ssrc=(uint)${$channels.video.SSRC} ! queue silent=true ! udpsink host=127.0.0.1 port=${$pipeline.port} sync=false async=true",
                "logLevel": 5,
                "channels": {
                    "video": {
                        "width": 640,
                        "height": 480,
                        "framerate": "15/1",
                        "bitrate": 100
                    }
                }
            },
            "front-cam": {
                "envVariables": {
                    "GST_DEBUG": 2
                },
                "command": "v4l2src device=/dev/video0 ! video/x-raw,format=YUY2,width=${$channels.video.width}, height=${$channels.video.height}, framerate=${$channels.video.framerate} ! queue ! videoconvert ! video/x-raw,format=I420 ! x264enc bframes=0 speed-preset=veryfast key-int-max=60 ! video/x-h264,stream-format=byte-stream ! queue silent=true ! rtph264pay mtu=1400 config-interval=-1 ! application/x-rtp,media=video,clock-rate=${$channels.video.clockRate},encoding-name=${$channels.video.encodingName},ssrc=(uint)${$channels.video.SSRC} ! queue silent=true ! udpsink host=127.0.0.1 port=${$pipeline.port} sync=false async=true",
                "logLevel": 5,
                "maxRetries": 20,
                "channels": {
                    "video": {
                        "width": 640,
                        "height": 360,
                        "framerate": "30/1",
                        "bitrate": 100
                    }
                }
            }
        },
        "services": {
            "record/videoMP4": {
                "pipeline": "-e rtpbin name=rtpbin latency=100 udpsrc port=${$pipeline.port} caps=\"application/x-rtp, media=video, encoding-name=H264, payload=98, clock-rate=90000\" ! rtpbin.recv_rtp_sink_0 rtpbin. ! rtph264depay ! h264parse ! queue ! mp4mux reserved-moov-update-period=1000 faststart=true ! filesink location=${$params.filePath} async=false",
                "topic": "/spot/spotcam/streams/video",
                "duration": "5s"
            },
            "record/audioMP4": {
                "pipeline": "-e rtpbin name=rtpbin latency=100 udpsrc port=${$pipeline.port} caps=\"application/x-rtp, media=audio, encoding-name=OPUS, payload=111, clock-rate=48000\" ! rtpbin.recv_rtp_sink_0 rtpbin. ! rtpopusdepay ! opusparse ! queue ! mp4mux reserved-moov-update-period=1000 faststart=true ! filesink location=${$params.filePath} async=false",
                "topic": "/spot/spotcam/streams/audio",
                "duration": "5s"
            }
        }
    }
}

The Settings section of the config can include the fields below:

FieldDescriptionDefault
enabledfalse
nameThe gstreamer component's instance name.gstreamer
logLevelThe component's log level4
pipelinesA JSON object that specifies the Gstreamer pipelines.
servicesA JSON object that specifies the custom Gstreamer services to create.

Pipelines​

The gstreamer component could have multiple pipelines each for a dedicated media (e.g. front or back camera). The JSON key of the pipeline (front-cam in the example above) is considered the name of the pipeline. The config fields of a pipeline can include the fields below:

FieldDescriptionDefault
commandThe Gstreamer pipeline name
envVariablesA JSON object that specifies a set of environment variable pairs ({"key":"value"}) to get passed to the dedicated pipeline Gstreamer processes.
logLevelThe pipeline's specific log level.4
maxRetriesThe maximum number of retries the gstreamer process will have after unsuccessful launch.-1 (infinity)
maxRetryBackoffThe maximum wait time between each retry. Starts from 100ms up until configured value, and exponentially grows each time. The value is parsed from a string to the duration.100ms
executableThe name of Gstreamer executable binary.gst-launch-1.0
channelsThe channel (audio/ video) specific sets of value for each pipeline, which could be replaces in ${$channels.video.XXX} fields (See example above).

Please note that upon the subscription of the topic, the agent will launch gst-launch-1.0 as a separate process, and pass in the configured pipeline. The pipeline process will be in a inherited environment, unless environment variables field is configured, and passed to the process execution.

Pipeline Capabilities​

As shown in the examples above, each GStreamer pipeline definition should end with a udpsink element, where the target port is determined by a parameterised value which the Agent will select automatically at runtime:

udpsink host=127.0.0.1 port=${$pipeline.port}

This sink element will allow the video output from the pipeline to be captured by the agent.

The sink capabilities are quite strict; the pipeline should generate RTP encapsulated H264 video, satisfying the following capabilities :

video/x-h264,stream-format=byte-stream,alignment=au,profile=constrained-baseline

Services​

The GStreamer component allows you to define custom services. Each entry in the services object results in a corresponding service being advertised by the agent at /gstreamer/<serviceName>.

For example, the configuration above will expose the following services:

  • /gstreamer/record/videoMP4
  • /gstreamer/record/audioMP4

Each service definition should follow the structure below:

FieldDescription
pipelineThe GStreamer pipeline to run for the service. The pipeline should include a templated port using port=${$pipeline.port}. A free port is automatically selected when the pipeline is run.
topicThe telemetry topic from which RTP packets will be piped into the GStreamer pipeline using the auto-selected port.
durationThe duration after which the service will automatically stop. The value is parsed from a string (e.g. "10s"). If no duration is specified, the service runs until cancellation.

Note: Additional parameters can be specified in the agent config if your pipeline requires them. Use ${$params.paramName} in the pipeline to template them. For example, in the config above, the filePath parameter is used to specify the output path for the MP4 file with: location=${$params.filePath}.

Generated Services​

Each configured service will be advertised by the agent and can be called with any required parameters as specified by your custom pipeline.

In the example shown, the user has provided runtime parameters in the service payload, specifying the filePath and overriding the default duration of the audioMP4 service to "10s".

Note: All parameters defined in the agent settings can be overridden at runtime in the service payload.