[VCL Cluster] Recommend budget Infiniband hardware?

[alex]

Hi,
I'm building a cluster, first with 3 PCs:
PC 1: My old PC , 2x HD6990 Intel i7 6 cores , 32GB Ram
PC 2: New, just built, 3x HD7970 AMD 8 cores, 16GB Ram
PC 3: New, just built, 3x HD7970 AMD 8 cores, 16GB Ram (Still building, not ready yet)

I'm trying to run a cluster with PC1 and PC2 at computer node, a VM by VirtualBox on PC1 as host (12GB Ram) , and it get only about ~6500Mb/s on SHA1. This is very poor performance, with 3x HD7970 only run at ~600Mb/s . HD 6990 running as about 1200Mb/s x 2 which is pretty OKAY (I think because it is in local host)

I think the problem is because I'm connecting them by an Asus RT-N56U router (no internet connection).

-Can you please suggest any budget InfiniBand hardward (LAN + Switch). Or any other possible solution.
Please recommend a few brand, as in New Zealand, I don't have much choice

-My target is about maximum 15 cards HD7970 on 5 PCs.

And if it is possible, please share any guide how to setup as well. I'm totally newbie on InfiniBand To tell the truth, only heard them when searching on VCL cluster.

Thank you very much.

EDIT:
Here is running speed:

Hashes: 29 total, 1 unique salts, 28 unique digests
Bitmaps: 8 bits, 256 entries, 0x000000ff mask, 1024 bytes
Workload: 128 loops, 80 accel
Watchdog: Temperature abort trigger disabled
Watchdog: Temperature retain trigger disabled
Device #1: Cayman, 1024MB, 830Mhz, 24MCU
Device #2: Cayman, 1024MB, 830Mhz, 24MCU
Device #3: Tahiti, 2048MB, 925Mhz, 32MCU
Device #4: Cayman, 1024MB, 830Mhz, 24MCU
Device #5: Tahiti, 2048MB, 925Mhz, 32MCU
Device #6: Cayman, 1024MB, 830Mhz, 24MCU
Device #7: Tahiti, 2048MB, 925Mhz, 32MCU
Device #1: Kernel ./kernels/4098/m0100_a3.Cayman_1084.4_1084.4.kernel (527272 bytes)
Device #2: Kernel ./kernels/4098/m0100_a3.Cayman_1084.4_1084.4.kernel (527272 bytes)
Device #3: Kernel ./kernels/4098/m0100_a3.Tahiti_1084.4_1084.4 (VM).kernel (369364 bytes)
Device #4: Kernel ./kernels/4098/m0100_a3.Cayman_1084.4_1084.4.kernel (527272 bytes)
Device #5: Kernel ./kernels/4098/m0100_a3.Tahiti_1084.4_1084.4 (VM).kernel (369364 bytes)
Device #6: Kernel ./kernels/4098/m0100_a3.Cayman_1084.4_1084.4.kernel (527272 bytes)
Device #7: Kernel ./kernels/4098/m0100_a3.Tahiti_1084.4_1084.4 (VM).kernel (369364 bytes)

Session.Name...: oclHashcat-plus
Status.........: Aborted
Input.Mode.....: Mask (?1?1?1?1?1?1?1?1?1?d)
Hash.Target....: File (/home/test/sha1.txt)
Hash.Type......: SHA1
Time.Started...: Wed Apr 24 02:54:42 2013 (25 secs)
Time.Estimated.: Thu Apr 25 23:47:01 2013 (1 day, 20 hours)
Speed.GPU.#1...: 1228.8M/s
Speed.GPU.#2...: 1229.1M/s
Speed.GPU.#3...: 533.9M/s
Speed.GPU.#4...: 1265.4M/s
Speed.GPU.#5...: 562.6M/s
Speed.GPU.#6...: 1230.2M/s
Speed.GPU.#7...: 622.7M/s
Speed.GPU.#*...: 6672.6M/s

This is trying to direct connect on a HP Gigabit switch, no router.

[epixoip]

Any IB hardware will do. I picked up a bunch of 10 year old 4x SDR hardware off eBay for less than a grand. Topspin 120 switch plus some old Mellanox HCAs. You really can't go wrong.

As far as setting it up, it's pretty much plug and play.

Drop the following in /etc/modules:

Code:

ib_sa
ib_cm
ib_umad
ib_addr
ib_uverbs
ib_ipoib
ib_ipath
ib_qib

Add a configuration like this one to /etc/network/interfaces:

Code:

auto ib0
iface ib0 inet static
        address 192.168.1.5
        netmask 255.255.255.0
        post-up echo connected >`find /sys -name mode | grep ib0`
        post-up echo 65520 >`find /sys -name mtu | grep ib0`

And that's it.