Overview of GStreamer

• GStreamer is a very popular open-source multimedia framework that makes performing complicated operations much easier by hiding the complexities. Things like streaming from a camera, playing a .mp4 video or encoding video/audio.​
• GStreamer is LGPL licensed, so you can use it within your application code and do not have to open your application code to the public.
• GStreamer allows you to make 'pipelines' that example where to get input, how to process that data, and how to output it. For example: Capture from a camera -> h.264 encode -> save to file.
• When starting out with GStreamer, you don't even have to create any code to test out different pipelines. You can put together different pipelines all on the command line. However, as you move to using GStreamer in your application, you will move using the GStreamer Library API.

Details of how GStreamer works

One thing to note is that while GStreamer can play all type of media formats, it’s not really the GStreamer software doing the decoding. It’s mostly made up of other open source (and closed source) codecs. GStreamer is essentially a ‘wrapper’ around all those different codecs and video/sound manipulators making it easy to integrate into a product and switch elements in and out as you needed.

The general idea is that elements (grouped together in libraries called ‘plug-ins’) are pipelined together to create the designed encode or decode stream. Knowing which elements plug-ins are available and which need to be ‘piped’ together and how is the trick about learning GStreamer.

Filters are used to change the default format of data as they flow from element to element. You can think of them as like a type cast in programming language. In Linux, the term “Sink” is also used to define the final stage of an output. A screen sink for example would be a video frame buffer. An audio sink could be ALSA (a common Linux audio player). Each plug-in can define as many options as they want. Some plug-ins even take a configuration file location as an argument because the number of options are greater than you really want to specify as individual parameters.

GStreamer comes with a default set of command-line utilities that can help in application development. The application ‘gst-inspect’ can be used to inspect all properties, signals, dynamic parameters and the object hierarchy of an element. This can be very useful to see which GObject properties or which signals (and using what arguments) an element supports. Run "gst-inspect fakesrc” to get an idea of what it does. The app “gst-launch” is a simple script-like command line application that can be used to test pipelines. For example, the command “gst-launch audiotestsrc ! audioconvert ! audio/x-raw-int,channels=2 ! alsasink” will run a pipeline which generates a sine-wave audio stream and plays it to your ALSA audio output. This is very useful in prototyping and creating demos quickly. Note that the command line of gstlaunch uses exclamation points ( ‘!’ ) to separate the plug-ins, kind of like the pipe symbol ( | ) on a shell command line.

The components of GStreamer are grouped together in separate packages for download. The ‘GStreamer’ and ‘Base’ packages are essentially your starting point for the overall framework. For the codec support, there are 3 main plugin packages; Good, Bad and Ugly. The Good contains elements that are well supported, have good licenses and good documentation. The Bad contains less supported code that needs testing or fixing. The Ugly have elements that are well supported and good code, but there may be some license issues you’ll have to look into before using in your product. There are other download packages available on the GStreamer website. Some codes (like FFmpeg) or other specialty utilities have a module package completely for themselves.

There is a master list on the gstreamer web site that show what media support is located in which package so you know what to download. By putting the software in these categories, it at least gives you a general idea of what type of software you’re getting. For the ‘Ugly’ for example, you may not have been aware of the licensing issues for some of the media formats, so this gives you a good heads up in that sense.

The license associated with GStreamer is a GNU LGPL. With the GNU LGPL license, even though the code that you get is free and open source, you can still use it as part of a project that will not be free/open source. Compared to GPL where any software that uses GPL code will automatically be required to also be open source. The importance of this LGPL license is that you get to use free open source code, but it doesn’t hinder you from making a closed source product. It should be noted that while the software for encoding and decoding various types of media are freely available, it does not mean the technology behind the codecs are royalty free. For example, MP3 files are very widely used, but the technology is still patented and licensed (until 2012 anyway). Before releasing a product, you should check with the patent holders for licensing costs.

GStreamer vs FFmpeg

FFmpeg is another popular open source framework for video and audio encoding/decoding. It can be found in many open source projects that run on Desktop PCs. However, many times you will find you must manually download FFmpeg separately and can only use it for personal use due to license and patent issue.

Using FFmpeg in a commercial product is possible, but you must take care as outlined by this page Fmpeg License and Legal Considerations on the FFmpeg website. Someone has to take care to manually build FFmpeg and remove any dangerous portions.

By comparison, GStreamer is more modular and individual technologies and codecs are individual plug-in files. Mean you can simply remove (delete) them on your target board to free yourself of any license or patent issue. You do not have to rebuild Gstreamer itself.

Renesas Support for GStreamer

Renesas plug-in

• Renesas has created a plug-in (“vspmfitler”) for Gstreamer so when you use standard GStreamer functions and features, our specialized hardware is used instead of the CPU seamlessly.​

Video Signal Processor Manager (VSPM) Library​

• A collection of device drivers and libraries that allow the use of Renesas HW for image manipulation such as scaling or rotation​
• This software can handle multiple requests from multiple different applications at once​
• This is what the Gstreamer plug-ins use in order to get access to HW features​
• The API interfaces to this VSPM software is provided to customers in case advanced users have complex requirements beyond what is exposed by the current Gstreamer plug-ins