GstTIPlugin Elements

From IGEP - ISEE Wiki

Jump to: navigation, search

Abstract

Gstreamer is a pipeline-based multimedia framework that allows you to create a variety of media-handling software components, including simple audio playback, audio and video playback, recording, streaming, and editing. The pipeline design allows you to easily write any type of streaming multimedia application. GStreamer also provides existing libraries and plugins for features such as A/V synchronization, muxing/demuxing, etc. Using GStreamer allows you to leverage all of this existing code in your product.

The TI GStreamer plugin uses DMAI to make access to the underlying hardware and DSP easy. GStreamer is run as an application on the ARM processor and through the use of DMAI is able to interact with the DSP or acceleration modules for encoding and decoding work. DMAI also provides portability of the code from one processor to another with minimal code changes. DMAI also takes care of using various hardware modules on the SoC to performs tasks such as frame copies using hardware acceleration.

This page will provide an overview of the TI GStreamer plugin elements, properties and usage example.

TI gstreamer plugin includes elements for:

  • Decoding audio, video and image using XDM 0.9 and XDM 1.x codecs.
  • Encoding video and image using XDM 0.9 and XDM 1.x codec.
  • Display sink to display the video using either fbdev or v4l2 standard.

Refer to this Gstreamer article for more information on downloading and building TI Gstreamer elements. The project is hosted at http://gstreamer.ti.com. If you are interested in understanding the design details then watch video presentation http://software-dl.ti.com/sdo/sdo_apps_public_sw/GStreamer_On_TI/FLV1/GStreamer_On_TI.htm

Also, some example GStreamer pipelines are available for cutting-and-pasting on this page.

TIDmaiVideoSink

The TIDmaiVideoSink element is used to display the output video data using fbdev or v4l2 standard.

Element Pads

  • UYVY
 YUV422 interleaved corresponding to V4L2_PIX_FMT_UYVY in v4l2 standard.
  • Y8C8
 YUV 422 semi planar. The dm6467 VDCE outputs this format after a color conversion.The format consists of 
 two planes: one with the Y component and one with the CbCr components interleaved (hence semi).
 Refer TI LSP VDCE documentation for a thorough description of this format

Property Details

Note: The element property can also be queried using gst-inspect command. e.g gst-inspect TIDmaiVideoSink will list all the properties and its default values.

  • displayStd (string)
 Display standard to be used.
 e.g To use Framebuffer “displayStd=fbdev” and to use V4L2 “displayStd=v4l2”
  • displayDevice (string)
 Video device to use 
 e.g To use fb1 “displayDevice=/dev/fb/1” and to use video2 “displayDevice=/dev/video2”
  • videoStd (string)
 Video Standard to use. Display sink supports the entire video standard supported on DMAI library. 
 Note that some standard as specific to one platform only. 
 E.g 720P_60 is supported only DM6467

The table below list the supported standard and its corresponding platform.


DM6446/357 DM355 DM6467 OMAP3530 DM365
D1_NTSC Yes Yes Yes x Yes
D1_PAL Yes Yes Yes x Yes
VGA x x x Yes x
480P x x Yes Yes x
720P_60 x x Yes x Yes
1080I_30 x x Yes x x
  • videoOutput (string)
 Output used for display video.

The table below shows the supported output and its corresponding platform


DM6446 DM355 DM6467 OMAP3530 DM365
Composite Yes Yes Yes x Yes
LCD x x x Yes x
Component x x Yes x Yes
DVI x x x Yes x

Note: You need to know a bit about the relation ship between videoOutput device and its supported standard. E.g Component device supports only 720P_60 and 1080I_30 video standard and similarly LCD supports only VGA resolution.

  • resizer (Boolean)
 Enable hardware resizer to resize the image. If LSP supports hardware resizer then use this option to resize
 video to fit on videoStd resolution. 
 e.g Consider the case where upstream is pushing 320x240 resolution and you have configured videoStd to D1_NTSC. 
In this case if resizer flag is enabled it will instructs display sink to use the hardware resizer to resize
the incoming resolution to D1_NTSC.

Note: There are some constraints on hardware resizer, see LSP user guide on your platform.

  • Autoselect (Boolean)
 Automatically select the Video Standard to use based on the video input. This only works when the upstream 
 element sets the video size attributes in the buffer.
  • numBufs (int)
 Number of video buffers allocated by the driver
  • Rotation (int)
 This property is supported only on OMAP3530 platform. It uses hardware rotation engine to rotate the image. 

Note: In case of fbdev you may need to configure kernel command line argument with correct rotation angle. Refer LSP release note for more information.

  • Framerate (int)
 Frame rate of the video expressed as an integer
 If upstream is not passing the correct framerate then you can set the framerate with this property.

Note: If this property is not specified in command line then display sink auto calculates frame rate depending on platform and video standard. See gst_tidmaivideosink_videostd_get_attrs () for details.

  • accelFrameCopy (boolean)
 Display video sink uses hardware accelerated framecopy module from DMAI library to copy the upstream buffer 
 into a display driver buffer. This greatly offloads the CPU load but it has pre-requisites and limitations. 
 The table below shows the hardware used in framcopy module. This flag in video sink is very useful for controlling
 the framecopy operation when the module is either not implemented or has some limitation on copying the frame. 
 e.g If your OMAP3530 LSP does not have support for ISP resizer then framecopy will not work and in that case
you can use “accelFrameCopy=FALSE” option to disable the accelerated framcopy and this will
fall back to using old memcpy routine to copy the upstream buffer in display driver buffer. 

Note: framecopy module always works with contiguous buffer. If upstream is not sending contiguous buffer then display sink copies the upstream buffer in contiguous buffer before using framecopy module.

Framecopy module in DMAI provides a uniform API’s across the entire platform by hiding the implementation details. It is important to know which peripherals DMAI framecopy module is using for performing the hardware accelerated copy. This information plays a critical role when you are moving from one version of LSP to other.

The table shows the driver used by DMAI framecopy module


DM6446 DM355/DM365 DM6467 OMAP3530

VPSS resizer EDMA VDCE down-scaler ISP resizer
  • contiguousInputFrame (Boolean)
 Set this if elemenet recieves contiguous input frame.
 As we learned that framecopy module always works with contiguous buffer and if upstream is not sending 
 contiguous input buffer then display sink needs to first copy upstream buffer into contiguous memory and 
 for this it uses memcpy which puts lot of cpu load on your system.  With this property you can control 
 this memcpy operation. If you know that upstream is sending contiguous buffer then disable this memcpy.
 E.g v4l2src can be configured to push driver buffer  (which is always contiguous) to downstream.
If this is the case then use this property to get much optimized performance.

e.g gst-launch v4l2src always_copy=FALSE ! TIDmaiVideoSink contiguousInputFrame=TRUE

  • useUserptrBufs (boolean)
 Use USERPTR I/O buffers with the V4L2 display.
  • hideOSD (boolean)
 Hide the video overlay windows during playback.  When using gst-launch, the video overlay may not be initialized properly and could display random data over the video window.  GStreamer-based application that use the video overlay should not set this property to TRUE.

TIAuddec1

Used for decoding XDM 1.x audio

If the upstream element (i.e. demuxer or typefind element) negotiates caps with TIAuddec, the engineName and codecName properties will be auto-detected based on the mime type requested. The engine and codec names used for particular mime types are defined in gsttiauddec.c/gsttiauddec1.c. Currently, they are set to use the engine and codec names provided with the TI evaluation codecs. This element output RAW pcm stream which later can be used by alsasink or osssink.

Element Pads

  • audio/mpeg mpegversion = 1
 Use MP3 Decoder for MPEG1 Layer 2 and Layer 3
  • audio/mpeg mpegversion = 4
 Use AAC decoder for MPEG4

Property Details

Note: The element property can also be queried using gst-inspect command. e.g gst-inspect TIAuddec will list all the properties and its default values.

  • engineName (string)
 Engine name used by codec combo.
  • codecName (string)
 Name of audio codec to use  

Note: If you are using default DVSDK combos then the codec name is the name of directory present in your dvsdk/<platform>_dvsdk_combos_<version>/packages/ti/sdo/codecs.

  • numOutputBufs (int)
 Number of output buffers to allocate for codec.

Note that element push codec buffer directly to the down-stream and if downstream element does not return buffer on-time then you may see "Failed to get free buffer" error from the element. In those case’s you may need to increase the numOutputBufs.

  • displayBuffer (boolean)

Decoder internally uses circular buffer object to keep the data from upstream elements and this property can be used to print the status of circular buffer read-write pointer. This can be useful while debugging the AV sync issues.

R = read pointer
W = write pointer

- or = is data that has been buffered for the codec. When a "-" is being shown, the minimum data requirements for the codec has not been met, so we must wait and buffer more data before invoking the codec. When a "=" is shown, there is enough data for the codec and it will be invoked.

The "|" bar shows how much data is needed to satisfy the codec requirements. When the data reaches the "|" line, the "-" will change to "=" and the codec starts running.

As long as there are always "=" signs between the "R" and the "|", the codec is not being starved, and as long as the "W" is mostly moving faster than "R" you should be able to meet real-time unless there is something else in the system starving the decode. If you let it run long enough, the "W" should catch up the the "R" and the buffer should be full.

Example output:

gst-launch --gst-debug-level=2 rtspsrc location=rtsp://10.5.5.34/media/media.amp ! rtph264depay ! TIViddec2 codecName=h264dec engineName=code
cServer frameRate=5  displayBuffer=true ! TIDmaiVideoSink videoStd=VGA videoOutput=LCD sync=false framerate=5 contiguousInputFrame=true

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[B                     |                                                      ]
[RW                    |                                                      ]
[R-W                   |                                                      ]
[R--W                  |                                                      ]
[R---W                 |                                                      ]
[R----W                |                                                      ]
[R-----W               |                                                      ]
[R------W              |                                                      ]
[R-------W             |                                                      ]
[R--------W            |                                                      ]
[R---------W           |                                                      ]
[R----------W          |                                                      ]
[R-----------W         |                                                      ]
[R------------W        |                                                      ]
[R-------------W       |                                                      ]
[R--------------W      |                                                      ]
[R---------------W     |                                                      ]
[R----------------W    |                                                      ]
[R-----------------W   |                                                      ]
[R------------------W  |                                                      ]
[R-------------------W |                                                      ]
[R--------------------W|                                                      ]
[R=====================W                                                      ]
[R=====================|W                                                     ]
[R=====================|=W                                                    ]
[R=====================|==W                                                   ]
[R=====================|===W                                                  ]
[R=====================|====W                                                 ]
[ R=====================|===W                                                 ]
[ R=====================|====W                                                ]
[ R=====================|=====W                                               ]
[  R=====================|====W                                               ]
[  R=====================|=====W                                              ]
[   R=====================|====W                                              ]
[   R=====================|=====W                                             ]
[    R=====================|====W                                             ]
[    R=====================|=====W                                            ]
[     R=====================|====W                                            ]
[     R=====================|=====W                                           ]
[      R=====================|====W                                           ]
[       R=====================|===W                                           ]
[       R=====================|====W                                          ]
[        R=====================|===W                                          ]
[         R=====================|==W                                          ]
[         R=====================|===W                                         ]
[          R=====================|==W                                         ]
[          R=====================|===W                                        ]
[          R=====================|====W                                       ]
[          R=====================|=====W                                      ]
[           R=====================|====W                                      ]
[           R=====================|=====W                                     ]
[           R=====================|======W                                    ]
[            R=====================|=====W                                    ]
[            R=====================|======W                                   ]
[            R=====================|=======W                                  ]
[            R=====================|========W                                 ]
[             R=====================|=======W                                 ]
[             R=====================|========W                                ]
[             R=====================|=========W                               ]
[             R=====================|==========W                              ]
[             R=====================|===========W                             ]
[              R=====================|==========W                             ]
[              R=====================|===========W                            ]
[              R=====================|============W                           ]
[              R=====================|=============W                          ]
[              R=====================|==============W                         ]
[              R=====================|===============W                        ]
[               R=====================|==============W                        ]
[               R=====================|===============W                       ]
[               R=====================|================W                      ]
[               R=====================|=================W                     ]
[                R=====================|================W                     ]
[                R=====================|=================W                    ]
[                 R=====================|================W                    ]
[                 R=====================|=================W                   ]
[                  R=====================|================W                   ]
[                  R=====================|=================W                  ]
  • genTimeStamps (boolean)
 Add the timestamps on output buffers pushed to down-stream elements.  This timestamp may be 
 later used by downstream elements to sync audio/video clocks.

TIAudenc1

This element encodes audio using an XDM 1.x codec interface.

TIAudenc1 - used for encoding XDM 1.x audio

Currently the only supported audio encoder is AAC-HE on the DM6467.

Element Pads

  • Input
audio/x-raw-int
Raw audio data
  • Output
audio/mpeg mpegversion = 4
Use AAC-HE encoder

Property Details

Note: The element property can also be queried using gst-inspect command. e.g gst-inspect TIAudenc1 will list all the properties and its default values.

  • engineName (string)
 Engine name used by codec combo.
  • codecName (string)
 Name of audio codec to use  

Note: If you are using default DVSDK combos then the codec name is the name of directory present in your dvsdk/<platform>_dvsdk_combos_<version>/packages/ti/sdo/codecs.

  • numChannels (int)
 Number of audio channels
  • bitrate (int)
 Bitrate to encode at in bits/second
  • samplefreq (int)
 Sample frequency of input in KHz
  • numOutputBufs (int)
 Number of output buffers to allocate for codec.

Note that element push codec buffer directly to the down-stream and if downstream element does not return buffer on-time then you may see "Failed to get free buffer" error from the element. In those cases you may need to increase the numOutputBufs.

  • displayBuffer (boolean)
 Encoder internally uses circular buffer object to keep the data from upstream elements and this 
 property can be used to print the status of circular buffer read-write pointer. 
 This can be useful while debugging the AV sync issues.
  • genTimeStamps (boolean)
 Add the timestamps on output buffers pushed to down-stream elements.  This timestamp may be 
 later used by downstream elements to sync audio/video clocks.

TIViddec2

Used for decoding XDM 1.2 video

If the upstream element (i.e. demuxer or typefind element) negotiates caps with TIViddec2, the engineName and codecName properties will be auto-detected based on the mime type requested. The engine and codec names used for particular mime types are defined in gsttividdec2.c. Currently, they are set to use the engine and codec names provided with the TI evaluation codecs.

The element output UYVY, which later can be used by TIDmaiVideoSink or xvimagesink.

Pad Elements

  • video/mpeg mpegversion=2
 Use MPEG2 Video Decoder
 
  • video/mpeg mpegversion=4
 Use MPEG4 Video Decoder
  • video/x-h264
 Use H.264 Video Decoder

Note: You may need to check decoder's supported on your platform. e.g DM355 does not support H.264 decoding

Property Details

Note: The element property can also be queried using gst-inspect command. e.g gst-inspect TIViddec will list all the properties and its default values.

  • engineName (string)
 Engine name used by codec combo.
  • codecName (string)
 Name of video codec to use  

Note: If you are using default DVSDK combos then the codec name is the name of directory present in your dvsdk/<platform>_dvsdk_combos_<version>/packages/ti/sdo/codecs.

  • numOutputBufs (int)
 Number of output buffers to allocate for codec.

Note that element push codec buffer directly to the down-stream and if downstream element does not return buffer on-time then you may see “Failed to get free buffer” error from the element. In those case’s you may need to increase the numOutputBufs.

  • displayBuffer (boolean)
 Decoder internally uses a circular buffer object to keep the data from upstream elements and 
 this property can be used to print the status of circular buffer read-write pointer. 
 This can be useful while debugging the AV sync issues.
  • genTimeStamps (boolean)
 Add the timestamps on output buffers pushed to down-stream elements.  This timestamp may be later 
 used by downstream elements to sync audio/video clocks.
  • framerate (int)
 Communicate this framerate to downstream elements.  The framerate specified should be an integer.
 If 29.97fps is desired specify 30 for this setting.

TIVidenc1

Used for encoding XDM 1.x video

Depending the codecs used, the element will outputs data in MPEG-2, MPEG-4, H.264 formats.

Pad Elements

  • UYVY
 YUV422 interleaved corresponding to V4L2_PIX_FMT_UYVY in v4l2 standard.
  • Y8C8
 YUV 422 semi planar. The dm6467 VDCE outputs this format after a color conversion.  The format 
 consists of two planes: one with the Y component and one with the CbCr components interleaved (hence semi).
 Refer TI LSP VDCE documentation for a thorough description of this format

Property details

Note: The element property can also be queried using gst-inspect command. e.g gst-inspect TIVidenc will list all the properties and its default values.

  • engineName (string)
 Engine name used by codec combo.
  • codecName (string)
 Name of video encoder codec to use  

Note: If you are using default DVSDK combos then the codec name is the name of directory present in your dvsdk/<platform>_dvsdk_combos_<version>/packages/ti/sdo/codecs.

  • numOutputBufs (int)
 Number of output buffers to allocate for codec.

Note that element push codec buffer directly to the down-stream and if downstream element does not return buffer on-time then you may see "Failed to get free buffer" error from the element. In those case’s you may need to increase the numOutputBufs.

  • displayBuffer (boolean)
 Decoder internally uses a circular buffer object to keep the data from upstream and this property
 can be used to print the status of circular buffer read-write pointer. 
 This can be useful while debugging the AV sync issues.
  • genTimeStamps (boolean)
 Add the timestamps on output buffers pushed to down-stream elements.  This timestamp may be later 
 used by downstream elements to sync audio/video clocks.
  • framerate (int)
 Communicate this framerate to downstream elements.  The framerate specified should be an integer.
 If 29.97fps is desired specify 30 for this setting.
  • contiguousInputFrame (Boolean)
 Set this if elemenet recieves contiguous input frame. If this is enabled then encoder element will 
 use hardware accelerator (i.e DMAI framecopy) to copy the data from upstream to circular buffer and this 
 will greatly reduce cpu load.
  • resolution (string)
 If filesrc is used then input resolution for the video ('widthxheight')
  • iColorSpace
 If filesrc is used then input color format to use (uyvy, y8c8)

TIImgdec1

Used for decoding JPEG images using the XDM 1.x JPEG decoder

The element outputs UYVY format, which can be either stored in a file or can be played via TIDmaiVideoSink.

Pad Elements

  • video/x-jpeg
 Input is jpeg image.

Property Details

Note: The element property can also be queried using gst-inspect command. e.g gst-inspect TIImgdec will list all the properties and its default values.

  • engineName (string)
 Engine name used by codec combo.
  • codecName (string)
 Name of image codec to use  

Note: If you are using default DVSDK combos then the codec name is the name of directory present in your dvsdk/<platform>_dvsdk_combos_<version>/packages/ti/sdo/codecs.

  • numOutputBufs (int)
 Number of output buffers to allocate for codec.

Note that element push codec buffer directly to the down-stream and if downstream element does not return buffer on-time then you may see “Failed to get free buffer” error from the element. In those case’s you may need to increase the numOutputBufs.

  • displayBuffer (boolean)
 Decoder internally uses a circular buffer object to keep the data from upstream elements and 
 this property can be used to print the status of circular buffer read-write pointer. 
  • genTimeStamps (boolean)
 Add the timestamps on output buffers pushed to down-stream.
  • resolution (string)
 The resolution of the input image ('width'x'height').

TIImgenc1

Encode raw YUV images into the JPEG format using the XDM 1.x JPEG encoder.

Pad Elements

  • UYVY
 YUV422 interleaved corresponding to V4L2_PIX_FMT_UYVY in v4l2 standard.   
   
  • I420
 YUV420P - 8 bit Y plane followed by 8 bit 2x2 subsampled U and V planes
  • Y42B
 YUV422P - YUV 4:2:2 planar

Property Elements

Note: The element property can also be queried using gst-inspect command. e.g gst-inspect TIImgenc will list all the properties and its default values.

  • engineName (string)
 Engine name used by codec combo.
  • codecName (string)
 Name of image codec to use  

Note: If you are using default DVSDK combo’s then the codec name is the name of directory present in your dvsdk/<platform>_dvsdk_combos_<version>/packages/ti/sdo/codecs.

  • numOutputBufs (int)
 Number of output buffers to allocate for codec.

Note that element push codec buffer directly to the down-stream and if downstream element does not return buffer on-time then you may see "Failed to get free buffer" error from the element. In those cases you may need to increase the numOutputBufs.

  • displayBuffer (boolean)
 Decoder internally uses a circular buffer object to keep the data from upstream elements and 
 this property can be used to print the status of circular buffer read-write pointer. 
  • genTimeStamps (boolean)
 Add the timestamps on output buffers pushed to down-stream.
  • resolution (string)
 The resolution of the input image ('width'x'height').
  • qValue (int)
 The quality to use when encoding the image.
  • iColorSpace
 Colorspace of the input image. e.g YUV422P or YUV420P or UYVY
  • oColorSpace
 Colorspace of the output image. e.g YUV422P or YUV420P or UYVY
 NOTE:  For iColorSpace and oColorSpace not all color spaces are supported by every codec.  For information
        on which color spaces your codec supports see the codec datasheet for that codec.

TIPrepEncBuf

Prepare Encode Buffer: Perform an accelerated copy and/or color-conversion into a physically contigous buffer that can be used by TIVidenc1.

Currently only in the source repository as of revision 873 -- not released yet

Element Pads

  • UYVY
 YUV422 interleaved corresponding to V4L2_PIX_FMT_UYVY in the V4L2 standard.
  • Y8C8 / NV16
 YUV 422 semi planar. The native color format for capture and display on DM6467. The format consists of 
 two planes: one with the Y component and one with the CbCr components interleaved (hence semi).
 Refer TI LSP VDCE documentation for a thorough description of this format
  • NV12
 YUV 420 semi planar.  V4L2_PIX_FMT_NV12 in the V4L2 standard. 

The TIPrepEncBuf element expects the caps to be the same on both the source and sink pads. The only exception is DM6467, where the sink pad can be in the Y8C8/NV16 and the source pad can be NV12. Even in this case, the resolution must be the same.

Property Details

Note: The element property can also be queried using gst-inspect command. e.g gst-inspect TIPrepEncBuf will list all the properties and its default values.

  • numOutputBufs (int)
 Number of video physically contiguous buffers to allocate for the source pad.  When these run out, the element will block until one is released
 by downstream elements.
  • contiguousInputFrame (Boolean)
 Set this if elemenet recieves contiguous input frame.  If this is not set, a hardware-accelerated copy/color conversion cannot be performed,
 and a slow memcpy will be used.

dmaiperf

dmaiperf can be used to capture pipeline performance data. Approximately once a second dmaiperf sends a frames per second, bytes per second, and timestamp data using gst_element_post_message.

The performance messages can be used by your application. However, the dmaiperf element is EXPERIMENTAL and intended for engineers optimizing dmaiperf. Message data format, properties, and other aspects of the dmaiperf element will change. Applications that intend to use the dmaiperf element need to realize these changes will likely break your application.

dmaiperf is available starting in release 1.01. Calculation defects in 1.01 causes the performance values to be inaccurate; recommend dmaiperf from trunk be used.

Element Pads

Identity element that simply passes data through.

Property Details

  • engine-name
 Engine name.
 e.g. engine-name=encode
  • print-arm-load
 Print the ARM CPU load information as a percentage busy.
 e.g. print-arm-load=true

Performance data format

The data format is EXPERIMENTAL and will be changed without regard to backwards compatibility.

The general format is

[<key>: <space> <value>; ]*

The single value keys include:

<key> Value units Meaning
Timestamp HH:MM:SS.xxxxxxxxx Time of day
bps bytes/sec Then data rate at this point in the pipe. The data rate at different points in the pipe can be different, such as before and after a video decoder.
fps frames/sec The video framerate.
CPU
ARM CPU load as a percentage busy.
DSP
DSP CPU load as a percentage busy.

For each memory segment, a set of values are provided:

<key> Value units Meaning
mem_seg text Name of memory segment.
base address Starting physical address.
size bytes Number of bytes in the memory segment.
maxblocklen bytes Size of the biggest block available.
used bytes Number of bytes in the memory segment that are in use.

Example output:

Timestamp: 0:06:30.080919110; bps: 13200933; fps: 30; CPU: 42; DSP: 14; 
mem_seg: DDR2; base: 0x8fb8b0f6; size: 0x20000; maxblocklen: 0x16040; used: 0x9cd0; 
mem_seg: DDRALGHEAP; base: 0x88000000; size: 0x7a00000; maxblocklen: 0x7780c00; used: 0x27f400; 
mem_seg: L1DSRAM; base: 0x11f04000; size: 0x10000; maxblocklen: 0x800; used: 0xf800;

GStreamer Pipelines Examples

See Also