How to Run a GStreamer Inference Pipeline on Axelera

 Axelera AI and GStreamer logos.

Running a GStreamer inference pipeline on Axelera AI hardware enables powerful deep learning applications at the edge. Axelera AI’s accelerators offer high-performance boards, a diverse Model Zoo of pre-trained models, and the Voyager SDK for deploying AI pipelines. These pipelines are built using GStreamer, a multimedia framework that links modular components into flexible, low-level workflows.

This guide walks you through how to create and run your own GStreamer inference pipeline on Axelera step by step. Before you begin, ensure your Axelera hardware and the Voyager SDK are installed. Read our tutorial on using the Axelera Metis M.2 with Raspberry Pi 5 for more details on how to install the SDK.

Understanding GStreamer for AI Inference Pipelines

GStreamer is a framework for building streaming media applications. At its simplest, it allows you to create a media pipeline, enabling data to flow, via buffers, from one end of the pipeline to the other.

The basic components of GStreamer pipelines are called elements. These process incoming data and output the result. Common GStreamer elements include:

• Source: Generates data, such as reading from disk or capturing from a camera or microphone.

• Sink: Receives data and does not output anything, such as writing to disk or displaying video.

• Filters / Converters: Operate on input data to transform or process it, such as encoders, decoders, or scalers.

  

  Basic GStreamer elements in a media pipeline: source, filter, and sink components.

  
  

  A simple GStreamer pipeline is shown below. videotestsrc generates a video image, videoflip flips it, and autovideosink displays it. The exclamation mark ! denotes the connection between elements in GStreamer syntax.

Pipelines can be run through the gst-launch-1.0 command:

Alternatively, you can implement more advanced applications using GStreamer Daemon or using the C/Python API.

Deploying an Inference Pipeline on Axelera

To begin running AI pipelines on Axelera hardware, the model you plan to use must be compiled along with its corresponding inference pipeline.

1. Travel to the Voyager SDK directory

2. Activate the virtual environment

3. Choose a model from the Model Zoo: To list all available models, run the make command in the root directory of the Voyager SDK. For this guide, we will use the YOLOv9 tiny model in ONNX format.

4. Optionally, download the prebuilt model from Axelera’s cloud service: This avoids the need to compile the model manually.

5. Deploy the inference pipeline

If you did not download the prebuilt model, you must deploy both the pipeline and the model:

Building a GStreamer Inference Pipeline on Axelera

Before running the Axelera inference pipeline through GStreamer, it’s helpful to first run it via the Voyager SDK to understand how it works.

Each model in the Axelera Model Zoo has a corresponding YAML file that defines its inference pipeline. These files are located in the ax_models/zoo directory. To explore these definitions further, check out our guide on using custom weights with Axelera, where we do a deep dive into them.

To view the YOLOv9t ONNX pipeline definition:

At runtime, the Voyager SDK parses this YAML file and converts it into a GStreamer pipeline to run the inference. Fortunately, the SDK also provides a GStreamer pipeline YAML file that you can use directly to build your own pipeline.

1. Begin by running the inference pipeline via the Voyager SDK: You can interrupt the inference with CTRL+C once it starts running. We only need it to barely run for the gst-pipeline.yaml to be generated.

2. Check out the generated GStreamer pipeline YAML file: The gst-pipeline.yaml file will be created under the build directory inside the Voyager SDK root.

This YAML file, shown below, contains a step-by-step breakdown of the GStreamer pipeline, listing each element and its required properties.

Elements like axinplace and axinferencenet are GStreamer components provided by the Voyager SDK. You can read more about them in the Voyager SDK GStreamer Pipeline Operators documentation.

We recommend comparing the original yolov9t-coco-onnx.yaml and the generated gst_pipeline.yaml files to understand how Axelera’s pipeline definitions are translated into GStreamer pipelines.

3. Build the GStreamer pipeline

The gst-pipeline.yaml file can be translated into a working GStreamer pipeline. Start by creating a pipeline.txt file:

Next, open pipeline.txt and add the following content, which contains the valid GStreamer pipeline syntax. Make sure to update the file paths in the filesrc and axinferencenet elements to match your system.

Note: We replaced the original appsink element with a fakesink to avoid preroll issues. This is because the pipeline won't auto-start on its own when using appsink.

4. Run the GStreamer inference pipeline

And just like that, you’ve successfully run a GStreamer inference pipeline on Axelera!

You might be wondering: "Okay, cool, we ran the pipeline. But how do we get the inference results?" Let’s walk through that next.

Obtaining Results from GStreamer Inference Pipeline on Axelera

We’ll now create a sample application that uses Python to:

• Run a GStreamer inference pipeline on an Axelera device

• Extract the resulting inference data

• Print out detection results

1. First, create a pipeline_appsink.txt file

This file is quite similar to pipeline.txt, but now the final element is an appsink named myappsink. Edit pipeline_appsink.txt and add the following content to it.

2. Create an inference.py file:

Open inference.py and add the following content. We will review each section of the code in detail to ensure a full understanding.

3. Import the necessary libraries

4. Create metadata instances for each frame

In the Axelera framework, the AxMeta object travels through the entire pipeline, carrying inference results. Each stage—preprocessing, inference, postprocessing—can attach metadata to it.

AxTaskMeta is a base class for various computer vision tasks that can be performed with the Axelera devices. The AxTaskMeta metadata is then stored within the AxMeta container under unique keys, which are usually the task names defined in the Axelera pipeline. Common task metadata types include:

• ClassificationMeta

• ObjectDetectionMeta

• SemanticSegmentationMeta

• InstanceSegmentationMeta

• TrackerMeta

We define a function to generate metadata instances for each frame. This function reads GStreamer metadata (gst_task_meta) and maps it to Axelera's internal metadata structures.

5. Create metadata instances from GStreamer

The next step is to add the ObjectDetectionMeta to the AxMeta container, since our current model (YOLOv9t) is an object detection model.

In the following function definition, we will:

• Obtain the ordered object detection classes labels

• Create a dictionary with the labels corresponding to the detections task defined in the Axelera pipeline

• Create the ObjectDetectionMeta object

• Add the new metadata instance to AxMeta

6. Print the detections per frame

7. Create the starting point of the application

   The starting point will:

   • Read the Gstreamer pipeline definition from our text file

   • Initialize Gstreamer

   • Run the GSeamer pipeline

   • Initialize the frame counter

   • Create the Axelera decoder for GStreamer metadata

8. Obtain inference results

In the loop, the application will:

• Pull buffers from appsink

• Convert buffers to images

• Decode metadata

• Create a FrameResult

• Print detections

9. Run the inference 

The printed results will look like the following: 

Contact Us

At RidgeRun.ai, we specialize in designing and optimizing GStreamer AI pipelines using specialized hardware such as Axelera hardware modules. If you need help integrating with your custom GStreamer inference pipelines, message us at contactus@ridgerun.ai and let’s start planning your project!