PCI Expansion Chassis - Experiences, anyone?

[jodler303]

Anyone got experiences with multi-gpu setup and PCIe expansion chassis? e.g. from MAGMA http://magma.com/products/pcie-expansion...-16-smart/ ? Since the tyan high-end rackmount chassis is so expensive, the idea is to have a standard-server inside one rack-chassis and put multi-gpu's into the magma box. - The two boxes are then interconnected with one single pcie-expansion cable. This type of setup is common for sound studio processors, why not try this for cracking, and might be cheaper than TYAN FT77B7015 & co.

Alternatively, what about PICMG 1.3 / SHB Express based solution?

[epixoip]

They're really expensive and have high latency. The 16-GPU expanders sell for $45k+.

[jodler303]

They're really expensive and have high latency. The 16-GPU expanders sell for $45k+.

epixoip: srsly? a standard expansion chassis itself should not cost much more than $2000-$2500. i did purchase different magma expansion boxes (i think it was 13-slot and 7-slot) some years ago, but for audio production and the standard was pci-x back then.  AFAIK there is nothing GPU-specific about the boxes, it's generic PCIe expansion and the drivers have to be verified working with the expansion. - Are you specifically refering to this box: http://magma.com/products/pcie-expansion...-16-smart/ ? if they want 45 grand, i can only say LOL.  i expected this box to be around $2000-$3000 from my experience.

If it's true what you are saying it would be cheaper to go and look for advantech stuff, chose a nice backplane e.g. this one http://buy.advantech.eu/PCE-7B17-00A1E-P...-00A1E.htm together with a single-board-computer. - However i'm afraid the price will be similar to the TYAN box for this one ... ;-)

My plan B is to use an empty 15-slot 4u rack chassis (i got two very cheap, standard price is around $200 per chassis, see http://www.orbitmicro.com/global/ghi-4f1...se#options), build in a standard mainboard and -if required- use pcie riser cables to connect the cards. But this is not my preferred solution, just plan B.

Plan C was to try and see if the EXSYS expansion boxes are working. They are offering extremely affordable pci(e) expansion solutions. Ok just checked, seems like they're only offering boxes for 2xPCIe + 2xPCI in the 4-slot config unfortunately, so it won't make sense: https://www.exsys.ch/en/expansion-box.html

also just found these: http://www.bhphotovideo.com/c/search?atc...4244844123

and this: http://www.delock.com/produkte/G_61950/m...anguage=en

[epixoip]

Please do not double post (or in your case, triple post.) Instead, use the Edit button to add additional information to your post. I've merged your posts for you.

The $2500 price range you're referring to would be for a small 3-4 GPU exapander chassis, which would be stupid as you could build an entire 4-GPU rig for much cheaper than the price of the expander chassis alone, not even including the price of the GPUs or host system. And actually a good 4-GPU expander chassis (e.g. Cubix XPRM-G3-ELRLE) are around $4600 without the cost of the GPUs, which is more than a Tyan FT77.

Yes, there ARE things that are GPU-specific when dealing with expander chassis, and you SHOULD select a chassis designed specifically for GPUs for three critical reasons: power, cooling, physical dimensions. However, a lot of those expander chassis that are specifically designed for 4+ GPUs cannot physically accommodate desktop cards, as they are typically designed for shorter, low-power, and often passively-cooled Tesla GPUs. Desktop GPUs are longer, are actively cooled, and typically have two top-mounted power connectors vs one rear-mounted power connector found on server GPUs. So desktop GPUs typically cannot physically fit nor can they be powered or adequately cooled in most expander chassis.

I just realized the 16-slot expander you linked to above can only hold 8 GPUs. I don't know if Magma has a true 16-GPU expander chassis, but I do know that the retail price of http://onestopsystems.com/4u-compute-acc...tx-titan-x is $32k with 16x GTX 980, and $45k with 16x Titan X.

I know you're kind of new to this scene, but expander chassis were one of the first things we experimented with ~5 years ago, and we promptly ditched them for the Tyan chassis. An expander is almost always NOT the way to go, the only reason we sell the chassis linked to above is that a host system + 16-GPU chassis + 16 GPUs is cheaper than 2x Brutalis + 1x Magistos + 10Gbit / Infiniband switch. But for less than 9 GPUs, an expander should be out of the question for almost all applications.

Your Plan B chassis almost surely will not work well for GPGPU. Maybe you should check out Chenbro RM41300-FS81 instead.

Plan C is most definitely a no-go.

[jodler303]

That's true, indeed i wasn't thinking of a 16 GPU system, but 16 slots (= 8 GPUs max).

I've mentioned 16 slots from the magma chassis, which really meant to be 16 slots (therefore 8 GPUs), not 16 GPUs. To be honest, i haven't realized until your last post that you were then refering to a 16 GPU system. Having a 4-8 GPU setup in mind, I've compared the 16-slot/8-GPU Magma chassis with TYAN FT77B7015, which also is a 8 GPUs system. I'm not even daring to think about a 16-GPU setup in one box, too complex, too expensive. Even 8 GPUs in one system is complex enough. I'd rather combine multiple smaller systems than even think about building a 16 GPU system. ;-)

Considering the price tag, Cubix XPRM-G3-ELRLE doesn't really attract me. Why would i want to spend so much money for this box, if i can only put 4 GPUs inside? Thats $900 per GPU, so basically more than the investment per GPU (considering GTX 980 or 980ti which are kinda 500-700ish).

Unfortunately, you haven't mentioned anything about the idea with PICMG 1.3 for a 4-8 GTX 980 setup. - That's expensive too, for sure, but for me that sounds pretty good, as long as i put 7-8 GPUs inside and the price tag is comparable to the TYAN box. Have you ever tried a setup like this? If yes, what makes the tyan box better? I'm not asking this to provoke, i'm trying to understand.

[epixoip]

I can't comment on the PICMG 1.3, I've never tried anything like that. But if the price is even at all comparable to the Tyan FT77, even if it was a couple thousand dollars less than the FT77, I'd absolutely just go with the FT77.

[pay]

but I do know that the retail price of http://onestopsystems.com/4u-compute-acc...tx-titan-x is $32k with 16x GTX 980, and $45k with 16x Titan X.

are u saying these price because u allready checked with them, or u'r guessing that would be the price?

[epixoip]

... Why would I guess? As I stated above, we sell that chassis, and that's how much they cost.

[mamexp]

That's true, indeed i wasn't thinking of a 16 GPU system, but 16 slots (= 8 GPUs max).

I've mentioned 16 slots from the magma chassis, which really meant to be 16 slots (therefore 8 GPUs), not 16 GPUs. To be honest, i haven't realized until your last post that you were then refering to a 16 GPU system. Having a 4-8 GPU setup in mind, I've compared the 16-slot/8-GPU Magma chassis with TYAN FT77B7015, which also is a 8 GPUs system. I'm not even daring to think about a 16-GPU setup in one box, too complex, too expensive. Even 8 GPUs in one system is complex enough. I'd rather combine multiple smaller systems than even think about building a 16 GPU system. ;-)

Considering the price tag, Cubix XPRM-G3-ELRLE doesn't really attract me. Why would i want to spend so much money for this box, if i can only put 4 GPUs inside? Thats $900 per GPU, so basically more than the investment per GPU (considering GTX 980 or 980ti which are kinda 500-700ish).

Unfortunately, you haven't mentioned anything about the idea with PICMG 1.3 for a 4-8 GTX 980 setup. - That's expensive too, for sure, but for me that sounds pretty good, as long as i put 7-8 GPUs inside and the price tag is comparable to the TYAN box.  Have you ever tried a setup like this?  If yes, what makes the tyan box better?  I'm not asking this to provoke, i'm trying to understand.

If we're  talking about double digits setup, i dont see why we stick with the limitation with air cooling. 

16 slots can support 16 cards fine. Just cut off the output plates....

[Flomac]

If we're  talking about double digits setup, i dont see why we stick with the limitation with air cooling. 

16 slots can support 16 cards fine. Just cut off the output plates....

This chassis cannot even feed 8 GPUs electrically and needs an outside power source. What a hassle. With 16 GPUs and the even bigger outside power source AND water cooling that hassle turns into a nightmare.