Building Flexible & Affordable Infrastructure for Private LLMs

Building Flexible & Affordable Infrastructure for Private LLMs

While many consume Large Language Models (LLMs) as a service, concerns about costs, intellectual property, and security drive many to host their own LLMs in their own datacenters. In this talk, we will discuss how to build a flexible, cost-effective, and secure GPU-intensive infrastructure that plays well in an enterprise environment with multiple groups wanting access to the same type of hardware but different workloads. We will present how a typical infrastructure looks like and also how to automate it so that it can be consumed as a service, using MetalSoft.