NVIDIA Enhances Llama 3.1 405B Performance with TensorRT Model Optimizer

Meta’s Llama 3.1 405B giant language mannequin (LLM) is reaching new ranges of efficiency because of NVIDIA’s TensorRT Mannequin Optimizer, based on the NVIDIA Technical Weblog. The enhancements have resulted in as much as a 1.44x enhance in throughput when operating on NVIDIA H200 GPUs.

Excellent Llama 3.1 405B Inference Throughput with TensorRT-LLM

TensorRT-LLM has already delivered exceptional inference throughput for Llama 3.1 405B for the reason that mannequin’s launch. This was achieved by means of varied optimizations, together with in-flight batching, KV caching, and optimized consideration kernels. These strategies have accelerated inference efficiency whereas sustaining decrease precision compute.

TensorRT-LLM added help for the official Llama FP8 quantization recipe, which calculates static and dynamic scaling elements to protect most accuracy. Moreover, user-defined kernels similar to matrix multiplications from FBGEMM are optimized by way of plug-ins inserted into the community graph at compile time.

Boosting Efficiency As much as 1.44x with TensorRT Mannequin Optimizer

NVIDIA’s customized FP8 post-training quantization (PTQ) recipe, obtainable by means of the TensorRT Mannequin Optimizer library, enhances Llama 3.1 405B throughput and reduces latency with out sacrificing accuracy. This recipe incorporates FP8 KV cache quantization and self-attention static quantization, decreasing inference compute overhead.

Desk 1 demonstrates the utmost throughput efficiency, exhibiting important enhancements throughout varied enter and output sequence lengths on an 8-GPU HGX H200 system. The system options eight NVIDIA H200 Tensor Core GPUs with 141 GB of HBM3e reminiscence every and 4 NVLink Switches, offering 900 GB/s of GPU-to-GPU bandwidth.

Equally, Desk 2 presents the minimal latency efficiency utilizing the identical enter and output sequence lengths.

These outcomes point out that H200 GPUs with TensorRT-LLM and TensorRT Mannequin Optimizer are delivering superior efficiency in each latency-optimized and throughput-optimized situations. The TensorRT Mannequin Optimizer FP8 recipe additionally achieved comparable accuracy with the official Llama 3.1 FP8 recipe on the Massively Multitask Language Understanding (MMLU) and MT-Bench benchmarks.

Becoming Llama 3.1 405B on Simply Two H200 GPUs with INT4 AWQ

For builders with {hardware} useful resource constraints, the INT4 AWQ method in TensorRT Mannequin Optimizer compresses the mannequin, permitting Llama 3.1 405B to suit on simply two H200 GPUs. This methodology reduces the required reminiscence footprint considerably by compressing the weights right down to 4-bit integers whereas encoding activations utilizing FP16.

Tables 4 and 5 present the utmost throughput and minimal latency efficiency measurements, demonstrating that the INT4 AWQ methodology gives comparable accuracy scores to the Llama 3.1 official FP8 recipe from Meta.

NVIDIA’s developments in TensorRT Mannequin Optimizer and TensorRT-LLM are paving the way in which for enhanced efficiency and effectivity in operating giant language fashions like Llama 3.1 405B. These enhancements provide builders extra flexibility and cost-efficiency, whether or not they have intensive {hardware} assets or extra constrained environments.

Picture supply: Shutterstock