In the world of AI, where resources are often a limiting factor, one enthusiast embarked on a journey to build his own supercomputer in his home basement. Ahmad Osman, a software engineer and AI enthusiast, recently faced a challenge. After experimenting with large language models (LLMs) for almost a year, his 48 GB of VRAM was no longer enough. This led him to build a dedicated LLM server at home, equipped with eight RTX 3090 GPUs and 192 GB of VRAM, in his basement.

Ahmad Osman: Builder at Heart

On his blog, Ahmad introduces himself as a passionate builder:
“I’m a software engineer with experience in machine learning, currently focused on generative AI and large language models. My academic background includes a bachelor’s degree in computer science and data science, and my professional journey has taken me through innovative environments.”

Ahmad’s drive to build an LLM server at home stems from his need for greater computational power, a necessity when working with advanced models like Meta’s Llama-3.1 405B.

Overcoming Hardware Challenges

Ahmad’s decision to upgrade came when he hit the limitations of his existing system. Building an LLM server at home required careful consideration of hardware components. His main focus was on CPU, memory speed, PCIe lanes, and multi-GPU configurations. He eventually chose eight RTX 3090 GPUs for their exceptional VRAM and performance.

Key Questions in System Design

“Which CPU or platform should I buy? Does memory speed really matter? How can I maximize VRAM at home? Why are Nvidia cards so expensive?” Ahmad explains his thought process and decisions that led to building a powerful LLM server at home. After extensive research, he finalized a configuration to maximize processing power for LLM tasks.

• Motherboard: Asrock Rack ROMED8-2T with 128 PCIe lanes.
• CPU: AMD Epyc Milan 7713 with 64 cores.
• Memory: 512 GB DDR4-3200 RAM.
• GPUs: Eight NVIDIA RTX 3090 GPUs.

Software Considerations

Beyond hardware, Ahmad delved into optimizing his software stack. His research revealed that the commonly used software for AI models isn’t always ideal for a home LLM server. Choosing the right software ensures full hardware utilization, crucial for running large language models at home.

Looking to the Future

Ahmad reflects on the rapid pace of technological progress:
“I remember being excited about getting a 60 GB HDD in 2004. Fast forward 20 years, and I now have triple that storage just in my GPUs. I can only imagine what we’ll be doing in another 20 years!”

Ahmad Open to Experimentation

In a recent post on X, Ahmad announced that he’s open to suggestions for experiments on his home LLM server:
“Feel free to suggest any ideas you’d like me to explore. I’m more than willing to run experiments on my server and share the results.”