Software Developer and lapsed Materials Engineer (MS, Cal Poly SLO, 2014).
Scientific computing was my gateway drug.
I have always imagined that Paradise will be a kind of Python library.
You are looking at a supercomputer, more or less. With the exception of some donated manual labor (for which I am eternally grateful), I built it entirely on my own back in 2018 as a side project. Intel had just abruptly discontinued the Xeon Phi Knights Landing product line, which left computer manufacturers with an unfortunate stockpile of hardware that they could not sell through the usual channels. I knew just the right people to get my hands on some (OK, a lot) of that hardware for cheap.
The thermal load was staggering, but fortunately water-cooling gave me some flexibility. Each of the four shipping containers (fireproof, with suppression!) had a cold intake aisle and hot exhaust aisle, with the air pulled through the radiators on the rack in the center. When this was built, it technically could have ranked on the TOP500 supercomputer list: the entire system included well over 300 Xeon Phi 7210 nodes, and each node weighs in at a maximum 2662 double precision GigaFLOPS for a grand total of nearly 1 PetaFLOPS.
Of course, I didn't have money for fancy supercomputer interconnect, let alone to install RAM in 300+ systems (it was so expensive back then!)... but there was some good news. Since these CPUs were originally intended for HPC workloads, they were designed with 16 GB of high-bandwidth memory on the CPU package itself, and could boot without any additional RAM. To avoid any need to add hard drives, I decided I would netboot them with a minimal container linux distro.
I settled on RancherOS and used Docker Swarm for orchestration, alongside custom bash/python tools for various administrative tasks and to control IPMI, with Prometheus/Grafana for monitoring. The bottleneck was the local servers running the Docker registry and TFTP (for the RancherOS netboot). To avoid choking the network I would avoid switching workloads on everything at once, but who likes waiting? I ended up migrating to QSFP+ network cards on the infrastructure servers and central switch, with SFP+ out to the switches serving the nodes. It was a reasonable compromise for my budget.
This setup made it possible to switch workloads remarkably quickly, which was fortunate because the most profitable workload could change very quickly. I will let you guess what the workload was. Gotta pay the bills...
