The Winning Inference Framework to Run Generative AI Models on NVIDIA Jetson with Less Memory

Understanding Memory Requirements for AI Models on NVIDIA Jetson

AI models have become a key technology for bringing automated solutions, intelligent monitoring, and computer vision algorithms to life, especially on edge devices. However, as these models continue to improve in capability, so do the computing resources required to deploy them. As models become larger and more capable, their hardware requirements increase, making GPU memory one of the primary deployment constraints on embedded systems. This is particularly true for Large Language Models (LLMs), which have rapidly grown in both size and capability over the past few years, becoming a pivotal tool for day-to-day work. However, their increasing memory requirements have also become a major bottleneck for deployment on resource-constrained devices.

In this blog, we'll walk through a practical optimization example of an LLM using multiple techniques to progressively reduce its memory footprint, enabling larger models to run on NVIDIA Jetson SoMs with lower memory requirements. Throughout the process, we'll measure and compare each technique in terms of memory consumption, inference speed, and benchmark performance using an LLM evaluation harness, providing a comprehensive view of the impact of each optimization.

Using Less Memory for AI Models on NVIDIA Jetson

NVIDIA Jetson platforms provide a powerful foundation for deploying AI applications at the edge. Their GPU acceleration, CUDA ecosystem, and optimized AI software stack enable developers to run complex deep learning models efficiently on compact, low-power devices. However, GPU memory remains a finite resource, making efficient memory utilization an important consideration when deploying AI applications. Reducing the memory footprint of a model not only allows more capable models to run using less memory on the same NVIDIA Jetson hardware, but also leaves additional resources available for the rest of the application, such as preprocessing, post-processing, and simultaneous AI workloads.

One of the most effective techniques for reducing a model’s memory footprint is quantization, which decreases the precision used to represent model weights and activations while striving to preserve model accuracy. By reducing the amount of memory the model requires, quantization lowers the overall system memory requirements, enabling deployment on more resource-constrained NVIDIA Jetson devices.

To demonstrate how quantization affects LLM memory usage, we selected Qwen2.5-3B-Instruct as our reference model. This open-weight language model provides a good balance between capability and resource requirements, making it an excellent candidate for deployment on NVIDIA Jetson platforms. All memory measurements, performance benchmarks, and model evaluations presented in this article were conducted on an NVIDIA Jetson Orin NX (16 GB RAM) system.

We begin our quantization journey with PyTorch, one of the most widely used frameworks for developing, training, and evaluating deep learning models. In addition to serving as the reference implementation for many open-weight LLMs, PyTorch provides an excellent baseline for understanding how different quantization techniques affect memory usage before introducing additional runtime optimizations.

For this first evaluation, we focused exclusively on measuring the memory footprint of the model while running under the PyTorch backend. We deployed the following Qwen2.5-3B-Instruct variants, all publicly available on Hugging Face, and measured their GPU memory consumption on the NVIDIA Jetson Orin NX using the tegrastats utility.

• Qwen2.5-3B-Instruct FP32

• Qwen2.5-3B-Instruct FP16

• Qwen2.5-3B-Instruct INT8 (GPTQ)

• Qwen2.5-3B-Instruct INT4 (AWQ)

The results are shown in Figure 1. They demonstrate that reducing the numerical precision of the model progressively decreases its GPU memory requirements by up to 80%. This allows larger language models to fit within the limited memory available on embedded platforms, freeing valuable GPU resources for additional workloads or enabling the deployment of the model on a NVIDIA Jetson device with lower memory specifications like the NVIDIA Jetson Orin Nano (8GB of RAM).

Figure 1. PyTorch Memory Footprint of Qwen2.5-3B-Instruct.

Optimizing Model Memory Usage with NVIDIA TensorRT

Quantization is only one part of the optimization process. Equally important is selecting an inference runtime that is optimized for the target hardware. To evaluate its impact, we repeated the memory measurements using NVIDIA TensorRT Edge-LLM, NVIDIA's runtime for deploying large language models on embedded GPU platforms.

TensorRT Edge-LLM currently supports the following data types for model quantizations:

• FP16

• INT8

• INT4

  
  

Thus, an FP32 comparison is not included. The FP16 TensorRT model was generated directly from the original Qwen2.5-3B-Instruct weights using the TensorRT Edge-LLM conversion tools. From that FP16 model, we then generated our own INT8 SmoothQuant and INT4 AWQ variants using TensorRT Edge-LLM's quantization workflow and default calibration dataset.

The memory results depicted in Figure 2, show that TensorRT Edge-LLM delivers a similar

memory footprint to PyTorch for the FP16 and INT8 variants, while the INT4 model achieves an additional reduction in memory usage. These results highlight that quantization remains the primary driver for reducing memory consumption, with the choice of runtime providing incremental improvements depending on the quantization strategy.

Figure 2. Cross-Backend Memory Footprint of Qwen2.5-3B-Instruct.

At first glance, these results may suggest that TensorRT Edge-LLM offers only modest memory savings. However, memory consumption is only one aspect of efficient LLM deployment. The real strength of TensorRT Edge-LLM becomes apparent when evaluating inference performance, where its optimizations are specifically designed to maximize the capabilities of NVIDIA Jetson platforms.

Memory Optimization vs. Performance: Finding the Right Balance

We have learned that quantization and TensorRT can significantly reduce the memory footprint of large models, enabling them to run on hardware with more limited memory. However, as the saying goes, there is no such thing as a free lunch. Quantization and the choice of inference framework can also affect a model's accuracy, numerical behavior, and execution time, so these optimizations should always be evaluated against the application's performance requirements.

In the following sections, we’ll explore the primary trade-offs involved in optimizing LLMs.

How Memory Optimization Affects AI Inference Performance

Memory optimization is only one aspect of deploying efficient LLMs on edge devices. In many real-world applications, inference speed is equally important, directly affecting response latency and the number of requests that can be processed. To evaluate the impact of each quantization strategy in the model’s performance, we measured the generation throughput of every model using the same prompt and decoding configuration. This allows us to compare how both quantization and the inference backend influence overall generation speed on the NVIDIA Jetson Orin NX.

Results are shown in Figure 3. All benchmarks used the same prompt and a target output length of 128 generated tokens. For the PyTorch backend, every benchmark was forced to generate exactly 128 tokens. TensorRT Edge-LLM currently allows generation to terminate naturally when the model emits an end-of-sequence (EOS) token, so its output length varies between models. To enable a fair comparison, the reported TensorRT request latency was normalized to an estimated 128-token generation based on the measured decode throughput.

While quantization alone helps reduce memory usage, pairing it with NVIDIA TensorRT Edge-LLM unlocks higher inference performance. The FP16 model delivers nearly twice the generation throughput of the PyTorch implementation, and the performance advantage grows even larger for the INT8 and INT4 variants. This trend highlights how TensorRT Edge-LLM is able to better leverage the underlying NVIDIA Jetson hardware, allowing increasingly quantized models to translate their reduced computational requirements into faster text generation.

Figure 3. Cross-Backend Token Generation Throughput of Qwen2.5-3B-Instruct.

How Memory Optimization Affects Model Accuracy

Reducing memory consumption and increasing inference speed are valuable only if the model continues to generate high-quality responses. To compare model quality across inference backends, we evaluated both the PyTorch and NVIDIA TensorRT Edge-LLM implementations using EleutherAI's lm-evaluation-harness, a widely adopted benchmarking framework for language models. By using the same evaluation methodology across both runtimes, we can assess whether the performance gains achieved through TensorRT Edge-LLM come at the cost of model quality.

Since TensorRT Edge-LLM does not currently expose the token-level probabilities required by many traditional language-model benchmarks, we selected a set of generation-based evaluation tasks that are fully supported by both backends. Specifically, we evaluated every model on GSM8K, IFEval, and DROP, covering mathematical reasoning, instruction following, and reading comprehension. Each model was tested using the same evaluation configuration and dataset subset, allowing for a direct comparison between the PyTorch and TensorRT Edge-LLM implementations.

Each benchmark reports various metrics depending on the task being evaluated:

• GSM8K Flexible Exact Match (EM): Measures the percentage of mathematical problems answered correctly after normalizing formatting differences.

• IFEval Prompt Strict: Evaluates whether the generated response follows the requested prompt format exactly.

• IFEval Instruction Strict: Measures compliance with all instructions contained in the prompt.

• DROP Exact Match (EM): Measures whether the predicted answer exactly matches the expected answer.

• DROP F1: Rewards partially correct answers by measuring the overlap between the predicted and expected responses.

Table 1. Cross-Backend Model Quality Benchmark Results of Qwen2.5-3B-Instruct.

Table 1 summarizes the evaluation results. Overall, both inference backends exhibit similar trends across the evaluated tasks, with the FP16 implementations producing comparable quality scores. As more aggressive quantization is introduced, some degradation becomes visible, particularly for the TensorRT INT8 and INT4 variants. It is important to note, however, that these TensorRT models were generated using the default calibration dataset provided by TensorRT Edge-LLM. In practice, quantization quality depends heavily on the calibration process, and using a representative calibration dataset can significantly improve the accuracy of low-precision models. The key takeaway is that TensorRT Edge-LLM delivers substantial inference speed improvements while maintaining competitive model quality, making it a compelling deployment backend for LLMs on NVIDIA Jetson platforms.

Key Takeaways for Memory Optimization on NVIDIA Jetson

Throughout this article, we explored how progressively quantizing a Large Language Model enables more capable AI applications to run on resource-constrained NVIDIA Jetson devices. By evaluating memory usage, generation throughput, and model quality across multiple quantization strategies and inference backends, we showed that deploying LLMs at the edge requires more than simply reducing model precision.

Our benchmarks demonstrate that quantization significantly lowers GPU memory requirements, allowing larger language models to fit within the limited resources of embedded platforms and reduce the memory requirements for the platform.

NVIDIA TensorRT Edge-LLM is the key to deliver substantial improvements in generation speed while maintaining competitive model quality across a range of generation-based benchmarks. Together, these techniques enable developers to maximize the capabilities of NVIDIA Jetson devices and build more responsive, efficient, and capable edge AI applications.

Contact Us

At RidgeRun.ai, we specialize in deploying high-performance AI and computer vision applications on edge devices, helping organizations optimize inference pipelines, accelerate large language models, and build efficient solutions for resource-constrained platforms such as NVIDIA Jetson. Whether you're exploring LLM deployment, AI acceleration, or end-to-end edge AI systems, our engineering team can help bring your project from prototype to production. Reach out to us at contactus@ridgerun.ai — let’s collaborate!