Understanding and Optimizing LLM/VLM Memory on NVIDIA Jetson for Edge AI
Updated: Oct 29
A Practical Guide to VLM Memory, NVIDIA Agent Skills, and the Performance-Accuracy Trade-Off Using Qwen3-VL-8B

Why Industrial VLM Deployments Run Out of Memory
Vision-language models (VLMs) can inspect factory images, run PPE checks, read labels, and describe anomalies in hardware or products using natural language. For manufacturers, logistics operators, and robotics teams, that flexibility is attractive: one model can help interpret many visual tasks without a separate classifier for every task.
However, most state-of-the-art VLMs require substantial memory for operation, which translates to expensive hardware requirements in terms of processing capabilities and memory.
Moving from a development platform such as NVIDIA Jetson AGX Thor to a compact Jetson Orin NX introduces a difficult engineering constraint: the model must fit alongside cameras, preprocessing, business logic, and other perception workloads, and still operate correctly.
At RidgeRun.ai, we explored VLM memory optimization on NVIDIA Jetson using the Qwen3-VL-8B-Instruct model, comparing higher-precision deployment considerations with a quantized INT4 AWQ deployment on a 16 GB Orin NX. The lesson is not simply to say “use 4-bit weights”. Successful edge AI optimization requires understanding the complete inference memory budget, measuring a representative image workload, use case constraints, and validating latency and answer quality.
LLM and VLM Memory Is More Than Model Weights
We explored the memory distribution of traditional CNNs in our blog RidgeRun.ai Shares Memory Optimization Best Practices to Run AI Models on NVIDIA Jetson and we experimented with Qwen2.5-3B LLM to run it on a NVIDIA Orin NX in our blog The Winning Inference Framework to Run Generative AI Models on NVIDIA Jetson with Less Memory. Now we are moving a step forward in understanding the memory components of a VLM and how it compares with CNNs and LLMs to reduce memory requirements efficiently.
Figure 1 shows a basic representation of the memory layout of CNN, LLM and VLM. The percentages and proportions are illustrative and depend on the model under study, but they serve well for this comparison.

Traditional CNN inference consumes memory mainly for weights, intermediate activations, feature maps, and execution buffers. For additional details on these components you can refer to our side blog.
In the case of LLMs, input processing like feature maps or image information is not present while adding a new critical memory block: the context / KV-cache.
Now, LLMs are designed to process text as input. This text is first broken down into tokens, which are converted into numerical representations. As these representations pass through the model's attention layers, they are transformed into Queries (Q), Keys (K), and Values (V), in practical terms, they become additional data that must be stored.
Think about your desktop AI assistant, such as ChatGPT. Every message you type, every paragraph you share, and even the contents of documents you upload lead to more tokens and as the model processes these tokens, it generates KV vectors that are stored in what we call the KV cache.
Why is this important? Because the model needs to retain information about previously processed tokens to understand the current conversation and generate coherent responses to the current context.
And here's the catch: the more tokens the model needs to keep in context, the more memory the KV cache consumes.
At this point, it should be intuitively clear why this memory block is so large in our figure. But let's put some numbers behind it with a simple mathematical calculation.
Consider the text you have read so far since you started reading this article. It contains ~500 words. For this case, the approximated required memory usage for tokens and VK vectors is shown in Figure 2. Don’t worry about the math, our goal is to take a look at the memory requirements. Just 500 words require about 83 MiB in memory – I am pretty sure you have provided ChatGPT with much more text than this. Of course, big engines like ChatGPT implement other methods to process really large amounts of text such as RAG systems, but you are already getting the idea.

A final detail to keep in mind is that context and KV-cache memory also scale with the number of concurrent sequences. If, instead of the single request described above, the LLM must serve 10 simultaneous queries of similar length, the memory required to store tokens and KV vectors scales proportionally.
Now let’s get back to our Figure 1. We can say that a VLM is a mix between a CNN and a LLM in terms of its memory composition. It keeps the context/KV-cache space while also adding a memory component for visual processing. Because a VLM can process both text and images, it requires context memory for text as well as visual encoders and feature maps for image content. Moreover, image content also generates tokens and KV vectors that need to be stored in the context memory. As with CNNs, image size also affects memory demand, with an additional effect from visual-token and KV-vector generation. This is why VLM memory requirements are typically larger than LLMs and CNNs.
Four VLM Memory Optimization Levers on NVIDIA Jetson
So far we have already discussed the memory layout for LLMs and VLMs and we have a better understanding of how they are affected by the model and application requirements. Now let’s examine the four main levers for optimizing LLM and VLM memory use.
The following optimization levers are meant to target the main three optimization spots in a VLM as shown in Figure 3.

1. Reduce Weight Precision
As presented in our previous blog, model weights are one of the biggest memory consumers. Quantizing them to lower-precision alternatives can reduce the required memory footprint. This is a key optimization lever that still holds in the case of LLMs and VLMs.
FP16/BF16 weights generally require about two bytes per parameter; 4-bit representations can greatly reduce weight storage. Quantization support also varies by Jetson SoC and inference engine. Smaller weights do not automatically imply faster inference if the kernel implementation is inefficient or not fully supported on your hardware.
Quantization reduces numerical precision and can degrade model quality. After quantization, validate the model on representative application data to confirm that accuracy and response quality remain acceptable.
2. Control Context, Concurrency, and KV-Cache Precision
As we just saw, KV-cache demand grows with the number of stored tokens and active sequences, subject to architecture-specific details such as grouped-query attention. Context length, maximum concurrent requests, and supported KV precision are practical deployment knobs.
Each knob introduces a trade-off. Reducing context length can improve memory headroom, but a shorter context limits the information available to a request, while lower concurrency reduces serving capacity. If the application permits those constraints, the potential savings can be substantial.
Unlike a conventional CNN, a VLM can manage KV-cache precision separately from model-weight precision. Tuning them independently can increase concurrency within the same memory budget, but it can also degrade response quality. Test the model on representative inputs after changing either value.
3. Optimize the Runtime
vLLM, llama.cpp, TensorRT Edge-LLM, and other inference stacks make different trade-offs in memory reservation, kernels, quantization support, batching, and offloading. All these knobs provide a great opportunity to save some memory. Tuning runtime settings like total sequences, batching size or memory allocation based on your specific use case would greatly improve not only your model’s performance but also your memory requirements.
4. Reduce the Visual Workload
VLMs have a lever that text-only LLMs do not: image input. Smaller images, fewer video frames, or region-of-interest crops can reduce visual processing and token count. In PCB inspection, however, excessive downscaling can erase a tiny missing component or solder defect. Cropping a suspected region at sufficient detail is often preferable to shrinking the entire frame. On the other hand, processing unnecessary large images for very obvious requirements is a waste of memory and processing time.
How NVIDIA Agent Skills Fit Into the Workflow
NVIDIA Agent Skills can help turn optimization into a repeatable engineering loop. Start with jetson-memory-audit to inspect the device's memory state; use jetson-inference-mem-tune to guide runtime and memory-related settings; then benchmark and validate the workload with the jetson-llm-benchmark skill. The skills assist with measurement and recommendations;, letting us focus on the real engineering decisions.
NVIDIA Agent Skills can assist throughout the optimization process by helping you collect and analyze performance data, identify optimization opportunities, and evaluate the impact of changes. Figure 4 illustrates a typical optimization workflow and highlights where each NVIDIA Agent Skill can be applied.

A Practical Qwen3-VL Experiment on Jetson Orin NX for Manufacturing Inspection
In order to better understand the impact of VLM memory optimization in an industrial application, let’s analyze the case study of an AI-based manufacturing inspection dashboard like the one shown in Figure 5.

For this case let’s consider the following basic requirements:
VLM Model: Qwen3-VL-8B
Target Hardware: NVIDIA Jetson Orin NX 16GB
Detection time: < 5s
Single request per image
With these requirements in mind, we encounter our first challenge with the selection of the VLM model. Deploying a VLM like Qwen3-VL-8B at its more common BF16 precision with the vLLM framework at its default settings requires an enormous amount of memory of roughly 108 GiB as shown in Figure 6. This is simply unfeasible for our target platform.

Here comes our first opportunity to optimize our model’s memory usage. As Figure 6 shows, the two biggest memory allocations belong to the model’s weights and the KV-cache size.
Let’s start with the KV-cache size. Given that our use case does not require simultaneous sequences, we can greatly optimize this memory allocation. Consider for example Figure 7. It shows that for different sizes of the KV-cache, the throughput of the model is the same. This is true as long as the total number of sequences or context size does not exceed the defined KV-cache size.

This way we can easily move from a memory allocation in Figure 6 to the memory allocation in Figure 8. Given our single-request requirement, we still have room to reduce the KV-cache size, potentially to 2 GiB or less. At that point, however, the model weights become the dominant constraint. Even if we reduce the KV cache to a minimum, the total size of the model’s weights is almost the total memory in our target hardware (16GB).

Here is where we jump to our next level of optimization: model quantization. Using NVIDIA Agent Skills, we identified a more suitable setup: Qwen3-VL-8B Q4_K_M GGUF + llama.cpp. With this setup we are not only moving to 4-bit precision but also changing the inference framework. The result is shown in Figure 9. Now the required memory allocation is about 7GiB, small enough to run on our NVIDIA Jetson Orin NX.

At this point you might think we are done! Well, this is just the beginning of our journey in the optimization of our model. After deploying this configuration on the NVIDIA Jetson Orin NX, however, we found that it ran extremely slowly. For our original image size of 1274 × 960 it takes about 30 seconds to complete an inference.
Now, let’s roll up our sleeves and focus on improving the model’s performance. For this, we will take into account three main techniques: 1) reduce the image size, 2) convert the model to TensorRT, and 3) fine-tune the prompt.
The first obvious step is to move our model to run on TensorRT Edge-LLM to get the most from the NVIDIA Jetson architecture. We used a INT4 AWQ quantization for the language model and FP16 for the vision model, then we built the engine file on our Jetson Thor (you need a powerful NVIDIA-based system for this) and deployed it into our NVIDIA Jetson Orin NX. With this change we reduced the inference time from ~30s to about ~10s.
Next, we resized the images to 320 × 240, leading us to a new inference time of ~9s.
Finally, the real trick. The original prompt we were using was too generic:
This prompt caused the model to generate a verbose answer, increasing execution time. We optimized the prompt to narrow down the answer and limit it to fewer tokens as follows:
The result might surprise you. After applying these optimizations, the manufacturing inspection dashboard now completes inference in approximately 3.8 seconds, a significant improvement over the original 30 seconds. As demonstrated in the video below, the system continues to perform its intended visual inspection tasks while benefiting from substantially faster inference.
Figure 10 shows performance after each change while Figure 11 now shows our updated graph summarizing our journey in memory optimization. Our final setup allocates almost the same amount of memory than our Q4_K_M version but with a dramatic boost in performance.


What These Results Mean for Industrial Edge AI Deployment
Reducing VLM memory requirements and inference latency has implications beyond simply making a model fit on smaller hardware. For industrial applications, these optimizations can enable the use of more compact and potentially lower-cost edge devices while leaving valuable system resources available for camera pipelines, image preprocessing, application logic, and other AI workloads.
In our experiment, we demonstrated how Qwen3-VL-8B could be deployed on an NVIDIA Jetson Orin NX 16 GB, achieving an inference time of approximately 3.8 seconds. This opens opportunities for applications such as automated visual inspection, intelligent video analytics, and robotics, where hardware resources, latency, and deployment costs are important design constraints.
For engineering teams developing industrial AI products, these results illustrate how careful optimization can expand hardware deployment options and help balance application requirements against system cost and resource constraints.
Key Takeaways: Optimizing VLMs for Real-World Edge AI
As demonstrated in this experiment, deploying large Vision-Language Models on resource-constrained edge devices is not simply a matter of reducing model precision. It requires understanding how memory is allocated, how inference frameworks manage resources, and how application requirements influence the optimization decisions.
By combining KV-cache optimization, model quantization, inference runtime selection, image resizing, and prompt engineering, we were able to bring Qwen3-VL-8B from a deployment requiring over 100 GiB of allocated memory to a practical implementation running on a 16 GB NVIDIA Jetson Orin NX, while reducing inference time from approximately 30 seconds to just 3.8 seconds.
However, every optimization introduces trade-offs. Lower precision, reduced image resolution, shorter context windows, and more restrictive prompts can affect model accuracy, visual understanding, and flexibility. This is why memory consumption and inference latency should never be evaluated independently of the application's accuracy and reliability requirements.
Ultimately, successful Edge AI deployment is not about achieving the smallest model or the fastest inference time. It is about finding the right balance between memory, performance, accuracy, and hardware cost for a specific application.
Need to Run a VLM on Production Edge Hardware?
RidgeRun.ai’s AI engineering services help teams evaluate, optimize, and deploy computer vision and multimodal AI on NVIDIA Jetson platforms. If your VLM prototype works on a development system but cannot meet the memory, latency, or reliability targets of your production device, our team can help profile the pipeline, select a practical deployment strategy, and validate the results on target hardware. Contact RidgeRun.ai to discuss an edge AI feasibility assessment or optimization engagement.
