12-03-2013, 12:05 AM
Hello guys!
First of all i'd like to thank you for great software. I had alot of headaches and sleepless nights with hashcat and VCL, however the results are very satisfying!
Recently i started hitting a wall because of scaling. I got running various types of hardware, mostly desktop boards, all with HD79**, interconnected with infiniband and ethernet (for PXE boot and scrypt mining). At some point it started to become a complete mess because of wiring mess and desktop hardware reliability issues. Bitcoin style plastic baskets as cases are not very convenient to manage.
I'm looking into rackmountable solutions now. The ultimate FT77 is an option, however i don't like it for a couple of reasons, will describe later. I've made minimal comparison of rackmountable solutions, based on density, cost/efficiency, here's what i came up with:
1) Running generic desktop hardware, the cost to set up for one single chip GPU is ~300$. Mounting these in cheap rackmountable 4u cases gives 3GPU/4u density (4th GPU isn't possible, remember about infiniband card, which consumes one slot). No PSU redundancy, no IPMI...
2) Supermicro 7046GT-TRF (http://www.supermicro.nl/products/system...gt-trf.cfm). Barebone is 2k$, CPUs are cheap LGA1366 from ebay, running rig will be ~2.5k$. I have one running btw. We get about 625$ per card and 4GPU/4u density. Booting generic Wheezy kernel without issues, 4xHD7970.
3) Tyan FT77. I live in Europe and getting one here is around 6k$, thanks to exclusive PNY distributor. If you got these cheaper here, please tell me how you did it. Say, CPUs + RAM will do around 1k. One case will run 7GPUs + infiniband card, which gives us 1k$ per card to setup and 7GPU/4u density. One thing that scares me off is running backported kernel to get it up and running. I'm used to boot universal Debian image over PXE, when you need to update smth you just update one image and you're done. I don't want to maintain an Ubuntu image just for FT77s.
4) Supermicro 2027GR-TRFH (http://www.supermicro.nl/products/system...r-trfh.cfm). This is one big "if", since i don't have one to test and no one tried AMD cards in it. It's 2u, however manufacturer states it will fit 6(!) double width GPUs. In my case 5 GPUs + infiniband card. This barebone costs ~2k$ + 1k$ for CPUs and RAM. Assuming GPUs will fit (possibly custom passive heatsinks will be needed, since the case is designed for passive Teslas and Phi), we get 600$ per card and 10GPU/4u density! Sounds too good to be true
If you have any comments or wanna hit me, do so. About 4th solution, maybe somebody tried that already. If no one did, maybe i will
First of all i'd like to thank you for great software. I had alot of headaches and sleepless nights with hashcat and VCL, however the results are very satisfying!
Recently i started hitting a wall because of scaling. I got running various types of hardware, mostly desktop boards, all with HD79**, interconnected with infiniband and ethernet (for PXE boot and scrypt mining). At some point it started to become a complete mess because of wiring mess and desktop hardware reliability issues. Bitcoin style plastic baskets as cases are not very convenient to manage.
I'm looking into rackmountable solutions now. The ultimate FT77 is an option, however i don't like it for a couple of reasons, will describe later. I've made minimal comparison of rackmountable solutions, based on density, cost/efficiency, here's what i came up with:
1) Running generic desktop hardware, the cost to set up for one single chip GPU is ~300$. Mounting these in cheap rackmountable 4u cases gives 3GPU/4u density (4th GPU isn't possible, remember about infiniband card, which consumes one slot). No PSU redundancy, no IPMI...
2) Supermicro 7046GT-TRF (http://www.supermicro.nl/products/system...gt-trf.cfm). Barebone is 2k$, CPUs are cheap LGA1366 from ebay, running rig will be ~2.5k$. I have one running btw. We get about 625$ per card and 4GPU/4u density. Booting generic Wheezy kernel without issues, 4xHD7970.
3) Tyan FT77. I live in Europe and getting one here is around 6k$, thanks to exclusive PNY distributor. If you got these cheaper here, please tell me how you did it. Say, CPUs + RAM will do around 1k. One case will run 7GPUs + infiniband card, which gives us 1k$ per card to setup and 7GPU/4u density. One thing that scares me off is running backported kernel to get it up and running. I'm used to boot universal Debian image over PXE, when you need to update smth you just update one image and you're done. I don't want to maintain an Ubuntu image just for FT77s.
4) Supermicro 2027GR-TRFH (http://www.supermicro.nl/products/system...r-trfh.cfm). This is one big "if", since i don't have one to test and no one tried AMD cards in it. It's 2u, however manufacturer states it will fit 6(!) double width GPUs. In my case 5 GPUs + infiniband card. This barebone costs ~2k$ + 1k$ for CPUs and RAM. Assuming GPUs will fit (possibly custom passive heatsinks will be needed, since the case is designed for passive Teslas and Phi), we get 600$ per card and 10GPU/4u density! Sounds too good to be true
If you have any comments or wanna hit me, do so. About 4th solution, maybe somebody tried that already. If no one did, maybe i will