Demo: Optimizing Gemma inference on NVIDIA GPUs with TensorRT-LLM

Channel:

Google for Developers

Subscribers:

2,370,000

Published on April 2, 2024 4:00:35 AM ● Video Link: https://www.youtube.com/watch?v=y4QljAMsXr0

Duration: 12:21

926 views

31

Even the smallest of Large Language Models are compute intensive significantly affecting the cost of your Generative AI application. Your ability to increase the throughput and reduce latency can make or break many business cases. NVIDIA TensorRT-LLM is an open-source tool that allows you to considerably speed up execution of your models and in this talk we will demonstrate its application to Gemma.

Watch more videos of Gemma Developer Day 2024 → https://goo.gle/440EAIV
Subscribe to Google for Developers → https://goo.gle/developers

#Gemma #GemmaDeveloperDay