infiniband setup guide?

[epixoip]

VCL only works with TCP/IP, so to use Infiniband with VCL, you need to do IP over Infiniband.

If you're using Debian/Ubuntu, it's very simple.

Add the following to /etc/modules

Code:

## ipoib
ib_sa
ib_cm
ib_umad
ib_addr
ib_uverbs
ib_ipoib
ib_ipath
ib_qib

Then add the following to /etc/network/interfaces (making appropriate changes for each host):

Code:

auto ib0
iface ib0 inet static
        address 192.168.1.1
        netmask 255.255.255.0
        post-up echo connected >`find /sys -name mode | grep ib0`
        post-up echo 65520 >`find /sys -name mtu | grep ib0`

Then you can bring everything up without rebooting:

Code:

for m in ib_sa ib_cm ib_umad ib_addr ib_uverbs ib_ipoib ib_ipath ib_qib; do modprobe $m; done
ifup ib0

Now you should be able to ping the IPv4 addresses using your normal ICMP ping, scp files back and forth over Infiniband, etc. And most importantly use VCL.


Oh, and I need to mention that the ip space that you use for your Infiniband network needs to be completely separate from your Ethernet network's ip space. Else you'll run into massive issues like Rurapenthe did.

So for example, if your Ethernet network is 192.168.0.0/16, then your Infiniband network should be anything but that. Maybe make it 10.0.0.0/24 or 172.16.0.0/12 or something.

It will NOT work if you use the same address space for both Ethernet and Infiniband.