Skip to main content

Video Widget

The 'Video' dashboard widget gives you the ability to connect to a camera on the robot and stream video using low-latency, peer-to-peer connectivity.

Note

The video plugin must be enabled in the agent configuration: see Video component for details.

This video stream is optimised for single operator teleoperation, as it will find the fastest route across the network, whether via a LAN network connection or via the cloud, but it only allows one operator at a time.

Creating a Video Widget​

On the Robot Dashboard, go to 'Add Widget' and select the 'Video' widget:

Fill in the important parameters to configure the widget:

  • Select the video source from the Camera drop-down list. If the video source is not shown in the list of available options, you may need to manually configure the video plugin to expose an suitable stream (refer here).
  • Optionally, select a suitable Playout Delay value. The receiving web-browser will try to buffer incoming video for approximately this duration in order to help maintain smooth video playback despite momentary network delays. Increasing this value may help improve video stream stability, but will also increase the overall latency of the video stream.
  • In the Video Command field, enter the appropriate GStreamer pipeline descriptor to use to capture video from the selected camera device, and convert it into the H264 format which the Rocos Agent is able to accept (refer here for details on the specific video caps required). The exact GStreamer pipeline configuration will depend on your GStreamer version, choice of camera, computer hardware capabilties and drivers, any other video software on the computer etc. See GStreamer Pipeline Samples for some examples.

** **

Coordinate Picker​

You can now add the coordinate picker functionality to image and video widgets. This allows you to click anywhere on an image or video to capture the percentage position of the X and Y axis. These coordinates can then be sent back down to the robot as a command. This will give you greater control in a range of situations.

For example, the coordinate picker can be used to control robotic arms more easily and help them pick up specific objects in a dynamic environment. You can click on any object in the live video feed and the robotic arm will move to that position.

Using the Coordinate Picker feature:

  • Firstly create a command where you want the coordinates to be sent. This should have two parameters. The payload can then use these parameters to move your robot to a certain location for example.
  • Go back to the Video Widget and toggle the Coordinate Picker feature on if you would like to use it.
  • Select the command that you would like to send this data to.
  • Enter widget.picker.xandwidget.picker.x and widget.picker.y to the corresponding parameter. You can also use expressions here to manipulate the values. e.g. $widget.picker.x * 720.
  • You can also toggle on a visible marker where the click is made. This will highlight on the video widget where the click is made.

Connecting and Playing Video​

To connect to the camera and start a video stream click on the 'Connect' button. After a few moments, the video stream will start playing. While the video stream is playing, some performance metrics are shown in the bar at the top of the widget:

The estimated latency is shown on the left, and the typical (one second average) received data-rate is shown on the right. If the P2P connection is disrupted, then the widget will automatically attempt to reconnect to the agent. You can see more details about the stream status by selecting the Debug option from the kebab menu in the top right corner:

Troubleshooting​

First, make sure that the Agent's video plugin is enabled on the robot (refer here).

There are broadly two types of video issue:

The first is when you cannot obtain a P2P connection between the browser and robot - this is evidenced by the "Network ICE Status" value shown in the video widget Debug panel never changing to "Connected". Possible causes include:

  • Using an incompatible browser type or version.
  • The browser or robot being located behind a particularly aggressive firewall.
  • Very bad network connectivity.

More common are issues related to the GStreamer pipeline itself:

  • The GStreamer pipeline command is invalid and so GStreamer crashes. In this case, the video widget may appear to establish a connection, but no video will appear and the connection will attempt to redial every few seconds.
  • The GStreamer pipeline does not generate H264 video in a format compatible with the Agent. In this case, the video widget may indicate that is has established a connection, but no video frames are received.

Refer to GStreamer Pipeline Samples for samples of compatible GStreamer pipelines to use as a basis for developing values suitable for your robot's specific configuration.

While you are testing potential GStreamer pipeline values, you may find it useful to temporarily edit the pipeline command by selecting the 'Advanced' button on the video widget as shown below. Any changes you make here will not be persisted once you navigate away from the page: