Using DeepStream Skills to Build a Multi-Camera Tracking and Alerting System

If you have not heard the good news yet, NVIDIA DeepStream has a new public home on GitHub starting with version 9. Among its new improvements and examples, DeepStream 9 also includes agentic skills. These skills work with compatible coding agents and harnesses, including tools such as Codex, Claude Code, and OpenCode. They give the agent the knowledge and guidance needed to build, configure, and run applications with DeepStream. Naturally, we had to put them to the test. We set ourselves one rule: build a complex video analytics application using only agentic coding, with no human intervention beyond prompting.

NVIDIA DeepStream vision AI at the edge

The DeepStream Skills Challenge

We gave ourselves the challenge to build a system only using DeepStream skills. We chose an NVIDIA Jetson AGX Thor Developer Kit as our target. The board has a 14-core Arm CPU, 128 GiB of system memory, an NVIDIA Thor GPU, and a 1 TB NVMe drive. For the runtime software environment, we used the nvcr.io/nvidia/deepstream:9.1-triton-multiarch Docker image available from NVIDIA NGC.

We gave ourselves the following requirements:

1. Monitor a collection of cameras placed around a work location.

2. Detect people and track them across all camera views.

3. Use multi-view 3D tracking to preserve each person’s identity as they move between cameras.

4. Render their walking paths on a bird’s-eye view of the warehouse.

5. Build a heat map from the tracked movement.

6. Define restricted zones and raise an alert when someone enters one.

7. Capture a snapshot of the trespasser and ask a vision language model to analyze it.

To present all of this to the user, we also asked the agent to build a web interface. It uses webrtc streaming to show the live pipeline alongside the events generated in real time.

Agent configuration: Throughout this experiment, we used GPT-5.6-Sol with high reasoning effort.

For the camera feeds, we used the 12-camera simulated warehouse videos included with the DeepStream examples.

MV3DT web dashboard with 12 warehouse camera feeds, a bird’s-eye tracking view, a movement heat map, and a restricted-zone entry alert

Multi-Camera 3D Tracking

As a starting point, we tried the MV3DT skill. It provides an example application and reference design for tracking people as they move through a warehouse covered by multiple cameras.

Skill: After asking the agent to run the 12 camera 3D tracking example we could see it using the deepstream-run-mv3dt skill.

It started by downloading all the required dependencies, including the 12 individual CCTV cameras in a simulated warehouse. These videos are fed into a deepstream pipeline that roughly ressembles a structure as shown below:

As the diagram shows, the 12 camera streams are batched first. Person detection then runs across all views together. The detections flow into MV3DT, where BodyPose3DNet estimates each person’s pose in 3D space. This gives the tracker the spatial context it needs to follow people across camera views.

The pipeline then splits in two. One branch tiles the camera feeds, draws the annotations, encodes the result, and saves it to a file for inspection. The other branch turns the tracking data into messages and publishes them through a message broker. This lets the application consume identities, positions, trajectories, and statistics in real time without being tightly coupled to the video pipeline.

Profiling the Demo

We then asked the agent to profile the application.

Skill: The agent loaded the deepstream-profile-pipeline skill and proceeded to generate a report.

Here, we had to iterate quite a few times to to get a comprehensive benchmark of the app since the mentioned skill is more focused on finding the optimal engine configuration rather than profiling the existing app as it is. Nevertheless, it ended up producing a very detailed and useful report.

In summary:

Result	Measured	What it means
Throughput per camera	1.72 FPS	Only 5.7% of the 30 FPS input rate
PeopleNet batch inference	482.41 ms p50	The main bottleneck, with GPU SM utilization at 97% p50

The system as it is is only capable of running at 1.72 FPS per camera, being the person detection GPU usage the main bottleneck.

Replacing the Person Detector

As the profiling results show, PeopleNet Transformer is the main bottleneck. It is a very capable model, however, in this 12-camera setup, the pipeline can process each camera at only 1.72 frames per second. That is far from the 30 frames per second needed to keep up with the source videos.

To improve the throughput, we decided to replace PeopleNet Transformer with RF-DETR. RF-DETR is an open source object detector with open weights. Its core models use the Apache 2.0 license, which allows commercial use. The published checkpoints are pretrained on the COCO dataset, where person is one of the 80 object classes.

At RidgeRun, we created an open source port of RF-DETR for DeepStream. The project is available here: DeepStream-RFDETR. We then asked the agent to integrate this port into the MV3DT application and use it in place of PeopleNet Transformer.

Skill: The agent loaded the deepstream-import-vision-model and started trying to manually port RF-DETR from Huggingface into DeepStream.

Altough the agent did a successful port, we wanted to use our battle-tested implementation, so we asked it to use exaclty the project as it is on GitHub. We also asked it to benchmark the different sizes of the model to help us choose a good balance between accuracy and performance.

Skill: The agent loaded the deepstream-dev and proceeded to correctly replace PeopleNet Transformer with deepstream-rfdetr.

Skill: It then loaded the deepstream-profile-pipeline skill, but this time it automatically generated the comprehensive report, as asked before.

Two RF-DETR bar charts comparing Nano, Small, Medium, and Large models in FP32 and FP16. The first shows maximum single-camera detection rates, and the second shows estimated camera capacity at 30 FPS. Small FP16 is highlighted at 428.91 detections per second and 14.3 cameras

Nano FP16 was faster, but Small FP16 used a larger input and produced better results in our visual review. Small FP16 reached 428.91 detections per second. This is equivalent to roughly 14.3 cameras at 30 frames per second, which is enough for our 12-camera application.

Since we did not have a labeled warehouse dataset for a formal evaluation, we relied on visual inspection. The results were promising, as shown in the GIF below.

Side-by-side animation of the 12-camera RF-DETR Small FP16 tiled inference view and the corresponding bird’s-eye tracking view

Two bar charts comparing PeopleNet Transformer FP16 and RF-DETR Small FP16 in the 12-camera MV3DT pipeline. Per-camera throughput increases from 1.72 to 30.27 FPS, while aggregate throughput increases from 20.67 to 363.20 FPS

With both detectors running in FP16, replacing PeopleNet Transformer with RF-DETR Small increased throughput from 1.72 to 30.27 FPS per camera. Aggregate throughput increased from 20.67 to 363.20 FPS. The new pipeline kept up with all 12 camera feeds at their full 30 FPS source rate.

Building the Web Application

Our target system runs remotely and could eventually be installed in a real warehouse. We still wanted operators to watch the processed live feed from a desktop computer. For that, we needed a web application and a simple way to stream the output of the GStreamer pipeline into a browser with low latency.

We used RidgeRun GStreamer Browser Sink. Our rrbrowsersink element can take H.264 video from a GStreamer pipeline and render it directly in a web page, without any external component.

Skill: The agent loaded the deepstream-dev skill and used it to add the browser streaming output to the pipeline.

Web application showing the live 12-camera RF-DETR Small FP16 feed with person detections and a bird’s-eye view of colored tracks

Adding a Heatmap to the Application

In many commercial vision applications, it is useful to understand where people spend time and which routes they use most often. A heatmap makes these movement patterns easy to see. It turns many individual tracks into a view of activity over time.

We asked the agent to add a heatmap that evolves as new tracks arrive. It produced the visualization below.

Animated bird’s-eye heatmap showing how movement patterns evolve over time

The heatmap reveals three main hotspots. One of them is especially interesting because it is concentrated around an operating lift. That can be a dangerous area for people on foot, so this type of visualization could help identify risky traffic patterns and guide safety improvements.

Adding Restricted Zones and Alerts

The heatmap showed that many workers traveled through the operating lift area. That can be dangerous. We asked the agent to define two restricted zones in the bird’s-eye view. One surrounded the operating lift. The other covered the area where boxes had fallen onto the floor.

We then asked the agent to use NVIDIA’s NVDSAnalytics plugin to detect when a worker entered either zone.

Skill: The agent loaded the deepstream-dev skill and used it to add the restricted-zone logic to the application.

Web application showing a restricted-zone entry alert for a worker near the operating lift, with the 12-camera live view, bird’s-eye tracks, and heatmap

The application emitted an alert whenever a restricted-zone entry occurred.

This analysis runs inside the DeepStream application. It does not run independently in the browser’s bird’s-eye view. That distinction matters when multiple viewers watch the stream. Every browser receives the same alert from the application at the same time. Their alerts do not depend on local latency or processing delays.

Adding Vision Reasoning to Trigger Events

Once we had a reliable trigger for workers entering a restricted zone, we wanted to add another layer of reasoning. A vision-language model could help filter false positives and give the user a clearer description of the scene.

For this, we included NVIDIA Cosmos 3, a reasoning vision-language model. We provided it with a snapshot for every restricted-zone trigger. It could then assess whether the event was genuine and describe what was happening in the image.

Skill: The agent loaded the deepstream-import-vision-model skill to integrate the vision-language model into the application.

Web application showing NVIDIA Cosmos 3 describing a restricted-zone event involving two people in yellow safety vests near an operating lift

We asked the agent to use vLLM as the inference backend for Cosmos. It runs efficiently on the Thor. By the end of the experiment, the agent had assembled the following architecture. The diagram was generated by the agent itself.

Thoughts After the Experiment

Overall, the experiment went smoothly. The agent made little to no mistakes when integrating the new requests.

More specifically, we came away with the following conclusions:

• We studied the available skills beforehand. We knew which skills were available and when they should be used. Working this way made it much more effective to steer the agent and make fast-paced progress.

• By using the DeepStream container and cloning the DeepStream source code, the agent knew its way around the project. It knew where to find the relevant components and how they fit together.

• The agent used the skills when it needed them. We never had to invoke a skill manually. Prompting was enough for the agent to decide when a skill applied.

• The benchmarking skill gave us the only real difficulty. It is designed to benchmark an engine, sweep different batch sizes and quantization levels, and select the best configuration. Our application uses a fixed batch size of 12. We had to iterate a few times to obtain a complete report for that fixed configuration.

• The agent was capable of porting RF-DETR on its own, even though we asked it to use our existing project. We ultimately used our implementation, but the fact that it could complete the port independently was impressive regardless.

• We have built multi-camera tracking systems before, and we know how complex that task can be. Seeing the agent use the skills to build a system of this scope in a few hours or days is genuinely exciting.

• We did not review all of the generated code or stress-test the application to its limits. Even so, building a prototype of this magnitude in such a short time is exciting and impressive.

Need DeepStream or AI Expertise?

If you’re looking for expert support with NVIDIA DeepStream or AI development, RidgeRun.ai can help. Contact us at contactus@ridgerun.ai or visit RidgeRun.ai to discuss your project.

Did an agent write your system and you want to take it to production? We can audit the codebase as part of our engineering services. Let's talk!