When Fraunhofer invested in NVIDIA DGX systems to provide self-hosted AI inferencing to all Fraunhofer employees, the path from hardware to platform was full of pain points. This talk walks through our migration from NVIDIA's Base Command Manager to a Run:ai-based setup integrated into our existing Rancher-based Kubernetes infrastructure, covering the architecture on both sides: a custom stack handling GPU partitioning and inferencing through our own vLLM Helm charts before, and the Run:ai control plane after. We'll wrap up with a look at how the platform is put to use across Fraunhofer, from the FhGenie chatbot to a growing set of internal AI-powered applications.
DevOps Engineer at Fraunhofer specializing in Kubernetes and private cloud infrastructure. His daily work spans IaC, automation, and networking to build and operate the on-premises platform that now powers Fraunhofer's internal AI inferencing services.
