Project · 2018
Budget Supercomputer
You are looking at a supercomputer, more or less. With the exception of some donated manual labor (for which I am eternally grateful), I built it entirely on my own back in 2018 as a side project. Intel had just abruptly discontinued the Xeon Phi Knights Landing product line, which left computer manufacturers with an unfortunate stockpile of hardware they could not sell through the usual channels. I knew just the right people to get my hands on a lot of that hardware for cheap.
Front aisle — Xeon Phi nodes mounted and cabled
The thermal load was staggering, but water-cooling gave me flexibility. Each of the four shipping containers — fireproof, with suppression — had a cold intake aisle and a hot exhaust aisle, with air pulled through radiators mounted on the central rack. When fully assembled, the system included over 300 Xeon Phi 7210 nodes. Each node weighs in at a maximum 2,662 double-precision GigaFLOPS, for a grand total of nearly 1 PetaFLOP — enough to have ranked on the TOP500 list at the time it was built.
Hot exhaust aisle — cooling infrastructure and cable runs
Of course, I had no budget for fancy interconnect, let alone RAM for 300+ systems. The saving grace: these CPUs shipped with 16 GB of high-bandwidth memory on the package itself (the very same type that is now in shortage for AI), and could boot without any additional RAM installed. To avoid hard drives entirely, I netbooted them with a minimal container Linux distribution.
I settled on RancherOS and used Docker Swarm for orchestration, alongside custom bash/python tooling for scheduling, administration, and IPMI control, with Prometheus and Grafana for monitoring. Networking used a mix of SFP+ and QSFP+ optics.
Full build — one of four shipping containers