GstTIPlugin Elements
From IGEP - ISEE Wiki
Contents
Abstract
Gstreamer is a pipeline-based multimedia framework that allows you to create a variety of media-handling software components, including simple audio playback, audio and video playback, recording, streaming, and editing. The pipeline design allows you to easily write any type of streaming multimedia application. GStreamer also provides existing libraries and plugins for features such as A/V synchronization, muxing/demuxing, etc. Using GStreamer allows you to leverage all of this existing code in your product.
The TI GStreamer plugin uses DMAI to make access to the underlying hardware and DSP easy. GStreamer is run as an application on the ARM processor and through the use of DMAI is able to interact with the DSP or acceleration modules for encoding and decoding work. DMAI also provides portability of the code from one processor to another with minimal code changes. DMAI also takes care of using various hardware modules on the SoC to performs tasks such as frame copies using hardware acceleration.
This page will provide an overview of the TI GStreamer plugin elements, properties and usage example.
TI gstreamer plugin includes elements for:
- Decoding audio, video and image using XDM 0.9 and XDM 1.x codecs.
- Encoding video and image using XDM 0.9 and XDM 1.x codec.
- Display sink to display the video using either fbdev or v4l2 standard.
Refer to this Gstreamer article for more information on downloading and building TI Gstreamer elements. The project is hosted at http://gstreamer.ti.com. If you are interested in understanding the design details then watch video presentation http://software-dl.ti.com/sdo/sdo_apps_public_sw/GStreamer_On_TI/FLV1/GStreamer_On_TI.htm
Also, some example GStreamer pipelines are available for cutting-and-pasting on this page.
TIDmaiVideoSink
The TIDmaiVideoSink element is used to display the output video data using fbdev or v4l2 standard.
Element Pads
- UYVY
YUV422 interleaved corresponding to V4L2_PIX_FMT_UYVY in v4l2 standard.
- Y8C8
YUV 422 semi planar. The dm6467 VDCE outputs this format after a color conversion.The format consists of two planes: one with the Y component and one with the CbCr components interleaved (hence semi). Refer TI LSP VDCE documentation for a thorough description of this format
Property Details
Note: The element property can also be queried using gst-inspect command. e.g gst-inspect TIDmaiVideoSink will list all the properties and its default values.
- displayStd (string)
Display standard to be used. e.g To use Framebuffer “displayStd=fbdev” and to use V4L2 “displayStd=v4l2”
- displayDevice (string)
Video device to use e.g To use fb1 “displayDevice=/dev/fb/1” and to use video2 “displayDevice=/dev/video2”
- videoStd (string)
Video Standard to use. Display sink supports the entire video standard supported on DMAI library. Note that some standard as specific to one platform only. E.g 720P_60 is supported only DM6467
The table below list the supported standard and its corresponding platform.
| |
DM6446/357 | DM355 | DM6467 | OMAP3530 | DM365 |
|---|---|---|---|---|---|
| D1_NTSC | Yes | Yes | Yes | x | Yes |
| D1_PAL | Yes | Yes | Yes | x | Yes |
| VGA | x | x | x | Yes | x |
| 480P | x | x | Yes | Yes | x |
| 720P_60 | x | x | Yes | x | Yes |
| 1080I_30 | x | x | Yes | x | x |
- videoOutput (string)
Output used for display video.
The table below shows the supported output and its corresponding platform
| |
DM6446 | DM355 | DM6467 | OMAP3530 | DM365 |
|---|---|---|---|---|---|
| Composite | Yes | Yes | Yes | x | Yes |
| LCD | x | x | x | Yes | x |
| Component | x | x | Yes | x | Yes |
| DVI | x | x | x | Yes | x |
Note: You need to know a bit about the relation ship between videoOutput device and its supported standard. E.g Component device supports only 720P_60 and 1080I_30 video standard and similarly LCD supports only VGA resolution.
- resizer (Boolean)
Enable hardware resizer to resize the image. If LSP supports hardware resizer then use this option to resize video to fit on videoStd resolution. e.g Consider the case where upstream is pushing 320x240 resolution and you have configured videoStd to D1_NTSC. In this case if resizer flag is enabled it will instructs display sink to use the hardware resizer to resize the incoming resolution to D1_NTSC.
Note: There are some constraints on hardware resizer, see LSP user guide on your platform.
- Autoselect (Boolean)
Automatically select the Video Standard to use based on the video input. This only works when the upstream element sets the video size attributes in the buffer.
- numBufs (int)
Number of video buffers allocated by the driver
- Rotation (int)
This property is supported only on OMAP3530 platform. It uses hardware rotation engine to rotate the image.
Note: In case of fbdev you may need to configure kernel command line argument with correct rotation angle. Refer LSP release note for more information.
- Framerate (int)
Frame rate of the video expressed as an integer If upstream is not passing the correct framerate then you can set the framerate with this property.
Note: If this property is not specified in command line then display sink auto calculates frame rate depending on platform and video standard. See gst_tidmaivideosink_videostd_get_attrs () for details.
- accelFrameCopy (boolean)
Display video sink uses hardware accelerated framecopy module from DMAI library to copy the upstream buffer into a display driver buffer. This greatly offloads the CPU load but it has pre-requisites and limitations. The table below shows the hardware used in framcopy module. This flag in video sink is very useful for controlling the framecopy operation when the module is either not implemented or has some limitation on copying the frame. e.g If your OMAP3530 LSP does not have support for ISP resizer then framecopy will not work and in that case you can use “accelFrameCopy=FALSE” option to disable the accelerated framcopy and this will fall back to using old memcpy routine to copy the upstream buffer in display driver buffer.
Note: framecopy module always works with contiguous buffer. If upstream is not sending contiguous buffer then display sink copies the upstream buffer in contiguous buffer before using framecopy module.
Framecopy module in DMAI provides a uniform API’s across the entire platform by hiding the implementation details. It is important to know which peripherals DMAI framecopy module is using for performing the hardware accelerated copy. This information plays a critical role when you are moving from one version of LSP to other.
The table shows the driver used by DMAI framecopy module
| |
DM6446 | DM355/DM365 | DM6467 | OMAP3530 |
|---|---|---|---|---|
| |
VPSS resizer | EDMA | VDCE down-scaler | ISP resizer |
- contiguousInputFrame (Boolean)
Set this if elemenet recieves contiguous input frame. As we learned that framecopy module always works with contiguous buffer and if upstream is not sending contiguous input buffer then display sink needs to first copy upstream buffer into contiguous memory and for this it uses memcpy which puts lot of cpu load on your system. With this property you can control this memcpy operation. If you know that upstream is sending contiguous buffer then disable this memcpy. E.g v4l2src can be configured to push driver buffer (which is always contiguous) to downstream. If this is the case then use this property to get much optimized performance.
e.g gst-launch v4l2src always_copy=FALSE ! TIDmaiVideoSink contiguousInputFrame=TRUE
- useUserptrBufs (boolean)
Use USERPTR I/O buffers with the V4L2 display.
- hideOSD (boolean)
Hide the video overlay windows during playback. When using gst-launch, the video overlay may not be initialized properly and could display random data over the video window. GStreamer-based application that use the video overlay should not set this property to TRUE.
TIAuddec1
Used for decoding XDM 1.x audio
If the upstream element (i.e. demuxer or typefind element) negotiates caps with TIAuddec, the engineName and codecName properties will be auto-detected based on the mime type requested. The engine and codec names used for particular mime types are defined in gsttiauddec.c/gsttiauddec1.c. Currently, they are set to use the engine and codec names provided with the TI evaluation codecs. This element output RAW pcm stream which later can be used by alsasink or osssink.
Element Pads
- audio/mpeg mpegversion = 1
Use MP3 Decoder for MPEG1 Layer 2 and Layer 3
- audio/mpeg mpegversion = 4
Use AAC decoder for MPEG4
Property Details
Note: The element property can also be queried using gst-inspect command. e.g gst-inspect TIAuddec will list all the properties and its default values.
- engineName (string)
Engine name used by codec combo.
- codecName (string)
Name of audio codec to use
Note: If you are using default DVSDK combos then the codec name is the name of directory present in your dvsdk/<platform>_dvsdk_combos_<version>/packages/ti/sdo/codecs.
- numOutputBufs (int)
Number of output buffers to allocate for codec.
Note that element push codec buffer directly to the down-stream and if downstream element does not return buffer on-time then you may see "Failed to get free buffer" error from the element. In those case’s you may need to increase the numOutputBufs.
- displayBuffer (boolean)
Decoder internally uses circular buffer object to keep the data from upstream elements and this property can be used to print the status of circular buffer read-write pointer. This can be useful while debugging the AV sync issues.
R = read pointer
W = write pointer
- or = is data that has been buffered for the codec. When a "-" is being shown, the minimum data requirements for the codec has not been met, so we must wait and buffer more data before invoking the codec. When a "=" is shown, there is enough data for the codec and it will be invoked.
The "|" bar shows how much data is needed to satisfy the codec requirements. When the data reaches the "|" line, the "-" will change to "=" and the codec starts running.
As long as there are always "=" signs between the "R" and the "|", the codec is not being starved, and as long as the "W" is mostly moving faster than "R" you should be able to meet real-time unless there is something else in the system starving the decode. If you let it run long enough, the "W" should catch up the the "R" and the buffer should be full.
Example output:
gst-launch --gst-debug-level=2 rtspsrc location=rtsp://10.5.5.34/media/media.amp ! rtph264depay ! TIViddec2 codecName=h264dec engineName=code cServer frameRate=5 displayBuffer=true ! TIDmaiVideoSink videoStd=VGA videoOutput=LCD sync=false framerate=5 contiguousInputFrame=true Setting pipeline to PAUSED ... Pipeline is live and does not need PREROLL ... Setting pipeline to PLAYING ... New clock: GstSystemClock [B | ] [RW | ] [R-W | ] [R--W | ] [R---W | ] [R----W | ] [R-----W | ] [R------W | ] [R-------W | ] [R--------W | ] [R---------W | ] [R----------W | ] [R-----------W | ] [R------------W | ] [R-------------W | ] [R--------------W | ] [R---------------W | ] [R----------------W | ] [R-----------------W | ] [R------------------W | ] [R-------------------W | ] [R--------------------W| ] [R=====================W ] [R=====================|W ] [R=====================|=W ] [R=====================|==W ] [R=====================|===W ] [R=====================|====W ] [ R=====================|===W ] [ R=====================|====W ] [ R=====================|=====W ] [ R=====================|====W ] [ R=====================|=====W ] [ R=====================|====W ] [ R=====================|=====W ] [ R=====================|====W ] [ R=====================|=====W ] [ R=====================|====W ] [ R=====================|=====W ] [ R=====================|====W ] [ R=====================|===W ] [ R=====================|====W ] [ R=====================|===W ] [ R=====================|==W ] [ R=====================|===W ] [ R=====================|==W ] [ R=====================|===W ] [ R=====================|====W ] [ R=====================|=====W ] [ R=====================|====W ] [ R=====================|=====W ] [ R=====================|======W ] [ R=====================|=====W ] [ R=====================|======W ] [ R=====================|=======W ] [ R=====================|========W ] [ R=====================|=======W ] [ R=====================|========W ] [ R=====================|=========W ] [ R=====================|==========W ] [ R=====================|===========W ] [ R=====================|==========W ] [ R=====================|===========W ] [ R=====================|============W ] [ R=====================|=============W ] [ R=====================|==============W ] [ R=====================|===============W ] [ R=====================|==============W ] [ R=====================|===============W ] [ R=====================|================W ] [ R=====================|=================W ] [ R=====================|================W ] [ R=====================|=================W ] [ R=====================|================W ] [ R=====================|=================W ] [ R=====================|================W ] [ R=====================|=================W ]
- genTimeStamps (boolean)
Add the timestamps on output buffers pushed to down-stream elements. This timestamp may be later used by downstream elements to sync audio/video clocks.
TIAudenc1
This element encodes audio using an XDM 1.x codec interface.
TIAudenc1 - used for encoding XDM 1.x audio
Currently the only supported audio encoder is AAC-HE on the DM6467.
Element Pads
- Input
audio/x-raw-int Raw audio data
- Output
audio/mpeg mpegversion = 4 Use AAC-HE encoder
Property Details
Note: The element property can also be queried using gst-inspect command. e.g gst-inspect TIAudenc1 will list all the properties and its default values.
- engineName (string)
Engine name used by codec combo.
- codecName (string)
Name of audio codec to use
Note: If you are using default DVSDK combos then the codec name is the name of directory present in your dvsdk/<platform>_dvsdk_combos_<version>/packages/ti/sdo/codecs.
- numChannels (int)
Number of audio channels
- bitrate (int)
Bitrate to encode at in bits/second
- samplefreq (int)
Sample frequency of input in KHz
- numOutputBufs (int)
Number of output buffers to allocate for codec.
Note that element push codec buffer directly to the down-stream and if downstream element does not return buffer on-time then you may see "Failed to get free buffer" error from the element. In those cases you may need to increase the numOutputBufs.
- displayBuffer (boolean)
Encoder internally uses circular buffer object to keep the data from upstream elements and this property can be used to print the status of circular buffer read-write pointer. This can be useful while debugging the AV sync issues.
- genTimeStamps (boolean)
Add the timestamps on output buffers pushed to down-stream elements. This timestamp may be later used by downstream elements to sync audio/video clocks.
TIViddec2
Used for decoding XDM 1.2 video
If the upstream element (i.e. demuxer or typefind element) negotiates caps with TIViddec2, the engineName and codecName properties will be auto-detected based on the mime type requested. The engine and codec names used for particular mime types are defined in gsttividdec2.c. Currently, they are set to use the engine and codec names provided with the TI evaluation codecs.
The element output UYVY, which later can be used by TIDmaiVideoSink or xvimagesink.
Pad Elements
- video/mpeg mpegversion=2
Use MPEG2 Video Decoder
- video/mpeg mpegversion=4
Use MPEG4 Video Decoder
- video/x-h264
Use H.264 Video Decoder
Note: You may need to check decoder's supported on your platform. e.g DM355 does not support H.264 decoding
Property Details
Note: The element property can also be queried using gst-inspect command. e.g gst-inspect TIViddec will list all the properties and its default values.
- engineName (string)
Engine name used by codec combo.
- codecName (string)
Name of video codec to use
Note: If you are using default DVSDK combos then the codec name is the name of directory present in your dvsdk/<platform>_dvsdk_combos_<version>/packages/ti/sdo/codecs.
- numOutputBufs (int)
Number of output buffers to allocate for codec.
Note that element push codec buffer directly to the down-stream and if downstream element does not return buffer on-time then you may see “Failed to get free buffer” error from the element. In those case’s you may need to increase the numOutputBufs.
- displayBuffer (boolean)
Decoder internally uses a circular buffer object to keep the data from upstream elements and this property can be used to print the status of circular buffer read-write pointer. This can be useful while debugging the AV sync issues.
- genTimeStamps (boolean)
Add the timestamps on output buffers pushed to down-stream elements. This timestamp may be later used by downstream elements to sync audio/video clocks.
- framerate (int)
Communicate this framerate to downstream elements. The framerate specified should be an integer. If 29.97fps is desired specify 30 for this setting.
TIVidenc1
Used for encoding XDM 1.x video
Depending the codecs used, the element will outputs data in MPEG-2, MPEG-4, H.264 formats.
Pad Elements
- UYVY
YUV422 interleaved corresponding to V4L2_PIX_FMT_UYVY in v4l2 standard.
- Y8C8
YUV 422 semi planar. The dm6467 VDCE outputs this format after a color conversion. The format consists of two planes: one with the Y component and one with the CbCr components interleaved (hence semi). Refer TI LSP VDCE documentation for a thorough description of this format
Property details
Note: The element property can also be queried using gst-inspect command. e.g gst-inspect TIVidenc will list all the properties and its default values.
- engineName (string)
Engine name used by codec combo.
- codecName (string)
Name of video encoder codec to use
Note: If you are using default DVSDK combos then the codec name is the name of directory present in your dvsdk/<platform>_dvsdk_combos_<version>/packages/ti/sdo/codecs.
- numOutputBufs (int)
Number of output buffers to allocate for codec.
Note that element push codec buffer directly to the down-stream and if downstream element does not return buffer on-time then you may see "Failed to get free buffer" error from the element. In those case’s you may need to increase the numOutputBufs.
- displayBuffer (boolean)
Decoder internally uses a circular buffer object to keep the data from upstream and this property can be used to print the status of circular buffer read-write pointer. This can be useful while debugging the AV sync issues.
- genTimeStamps (boolean)
Add the timestamps on output buffers pushed to down-stream elements. This timestamp may be later used by downstream elements to sync audio/video clocks.
- framerate (int)
Communicate this framerate to downstream elements. The framerate specified should be an integer. If 29.97fps is desired specify 30 for this setting.
- contiguousInputFrame (Boolean)
Set this if elemenet recieves contiguous input frame. If this is enabled then encoder element will use hardware accelerator (i.e DMAI framecopy) to copy the data from upstream to circular buffer and this will greatly reduce cpu load.
- resolution (string)
If filesrc is used then input resolution for the video ('widthxheight')
- iColorSpace
If filesrc is used then input color format to use (uyvy, y8c8)
TIImgdec1
Used for decoding JPEG images using the XDM 1.x JPEG decoder
The element outputs UYVY format, which can be either stored in a file or can be played via TIDmaiVideoSink.
Pad Elements
- video/x-jpeg
Input is jpeg image.
Property Details
Note: The element property can also be queried using gst-inspect command. e.g gst-inspect TIImgdec will list all the properties and its default values.
- engineName (string)
Engine name used by codec combo.
- codecName (string)
Name of image codec to use
Note: If you are using default DVSDK combos then the codec name is the name of directory present in your dvsdk/<platform>_dvsdk_combos_<version>/packages/ti/sdo/codecs.
- numOutputBufs (int)
Number of output buffers to allocate for codec.
Note that element push codec buffer directly to the down-stream and if downstream element does not return buffer on-time then you may see “Failed to get free buffer” error from the element. In those case’s you may need to increase the numOutputBufs.
- displayBuffer (boolean)
Decoder internally uses a circular buffer object to keep the data from upstream elements and this property can be used to print the status of circular buffer read-write pointer.
- genTimeStamps (boolean)
Add the timestamps on output buffers pushed to down-stream.
- resolution (string)
The resolution of the input image ('width'x'height').
TIImgenc1
Encode raw YUV images into the JPEG format using the XDM 1.x JPEG encoder.
Pad Elements
- UYVY
YUV422 interleaved corresponding to V4L2_PIX_FMT_UYVY in v4l2 standard.
- I420
YUV420P - 8 bit Y plane followed by 8 bit 2x2 subsampled U and V planes
- Y42B
YUV422P - YUV 4:2:2 planar
Property Elements
Note: The element property can also be queried using gst-inspect command. e.g gst-inspect TIImgenc will list all the properties and its default values.
- engineName (string)
Engine name used by codec combo.
- codecName (string)
Name of image codec to use
Note: If you are using default DVSDK combo’s then the codec name is the name of directory present in your dvsdk/<platform>_dvsdk_combos_<version>/packages/ti/sdo/codecs.
- numOutputBufs (int)
Number of output buffers to allocate for codec.
Note that element push codec buffer directly to the down-stream and if downstream element does not return buffer on-time then you may see "Failed to get free buffer" error from the element. In those cases you may need to increase the numOutputBufs.
- displayBuffer (boolean)
Decoder internally uses a circular buffer object to keep the data from upstream elements and this property can be used to print the status of circular buffer read-write pointer.
- genTimeStamps (boolean)
Add the timestamps on output buffers pushed to down-stream.
- resolution (string)
The resolution of the input image ('width'x'height').
- qValue (int)
The quality to use when encoding the image.
- iColorSpace
Colorspace of the input image. e.g YUV422P or YUV420P or UYVY
- oColorSpace
Colorspace of the output image. e.g YUV422P or YUV420P or UYVY
NOTE: For iColorSpace and oColorSpace not all color spaces are supported by every codec. For information
on which color spaces your codec supports see the codec datasheet for that codec.
TIPrepEncBuf
Prepare Encode Buffer: Perform an accelerated copy and/or color-conversion into a physically contigous buffer that can be used by TIVidenc1.
Currently only in the source repository as of revision 873 -- not released yet
Element Pads
- UYVY
YUV422 interleaved corresponding to V4L2_PIX_FMT_UYVY in the V4L2 standard.
- Y8C8 / NV16
YUV 422 semi planar. The native color format for capture and display on DM6467. The format consists of two planes: one with the Y component and one with the CbCr components interleaved (hence semi). Refer TI LSP VDCE documentation for a thorough description of this format
- NV12
YUV 420 semi planar. V4L2_PIX_FMT_NV12 in the V4L2 standard.
The TIPrepEncBuf element expects the caps to be the same on both the source and sink pads. The only exception is DM6467, where the sink pad can be in the Y8C8/NV16 and the source pad can be NV12. Even in this case, the resolution must be the same.
Property Details
Note: The element property can also be queried using gst-inspect command. e.g gst-inspect TIPrepEncBuf will list all the properties and its default values.
- numOutputBufs (int)
Number of video physically contiguous buffers to allocate for the source pad. When these run out, the element will block until one is released by downstream elements.
- contiguousInputFrame (Boolean)
Set this if elemenet recieves contiguous input frame. If this is not set, a hardware-accelerated copy/color conversion cannot be performed, and a slow memcpy will be used.
dmaiperf
dmaiperf can be used to capture pipeline performance data. Approximately once a second dmaiperf sends a frames per second, bytes per second, and timestamp data using gst_element_post_message.
The performance messages can be used by your application. However, the dmaiperf element is EXPERIMENTAL and intended for engineers optimizing dmaiperf. Message data format, properties, and other aspects of the dmaiperf element will change. Applications that intend to use the dmaiperf element need to realize these changes will likely break your application.
dmaiperf is available starting in release 1.01. Calculation defects in 1.01 causes the performance values to be inaccurate; recommend dmaiperf from trunk be used.
Element Pads
Identity element that simply passes data through.
Property Details
- engine-name
Engine name. e.g. engine-name=encode
- print-arm-load
Print the ARM CPU load information as a percentage busy. e.g. print-arm-load=true
Performance data format
The data format is EXPERIMENTAL and will be changed without regard to backwards compatibility.
The general format is
[<key>: <space> <value>; ]*
The single value keys include:
| <key> | Value units | Meaning |
|---|---|---|
| Timestamp | HH:MM:SS.xxxxxxxxx | Time of day |
| bps | bytes/sec | Then data rate at this point in the pipe. The data rate at different points in the pipe can be different, such as before and after a video decoder. |
| fps | frames/sec | The video framerate. |
| CPU | |
ARM CPU load as a percentage busy. |
| DSP | |
DSP CPU load as a percentage busy. |
For each memory segment, a set of values are provided:
| <key> | Value units | Meaning |
|---|---|---|
| mem_seg | text | Name of memory segment. |
| base | address | Starting physical address. |
| size | bytes | Number of bytes in the memory segment. |
| maxblocklen | bytes | Size of the biggest block available. |
| used | bytes | Number of bytes in the memory segment that are in use. |
Example output:
Timestamp: 0:06:30.080919110; bps: 13200933; fps: 30; CPU: 42; DSP: 14; mem_seg: DDR2; base: 0x8fb8b0f6; size: 0x20000; maxblocklen: 0x16040; used: 0x9cd0; mem_seg: DDRALGHEAP; base: 0x88000000; size: 0x7a00000; maxblocklen: 0x7780c00; used: 0x27f400; mem_seg: L1DSRAM; base: 0x11f04000; size: 0x10000; maxblocklen: 0x800; used: 0xf800;
GStreamer Pipelines Examples
See Also
- XDM FAQ - more details about XDM
