infiniband setup guide?

[alex]

Hi,
Can anyone point me to the working infiniband setup guide for using vclhashcat?
I've tried several guide that found in google, found this one is quite easy:
http://pkg-ofed.alioth.debian.org/howto/...wto-4.html

And then 2 machines can ping each other, but when running vclhashcat, it only detect 3 cards in local computer, then kernel, and just frozen there
eg:
Device #1: Cayman, 1024MB, 800Mhz, 24MCU
Device #3: Cayman, 1024MB, 800Mhz, 24MCU
Device #3: Cayman, 1024MB, 800Mhz, 24MCU
Device #1: Kernel ./kernels/4098/m0000_a1.Cayman_938.2_CAL 1.4.1741.kernel (490316 bytes)

then it does nothing.

Help please
Will reward if required.

[blandyuk]

epixoip is the man u need

[epixoip]

VCL only works with TCP/IP, so to use Infiniband with VCL, you need to do IP over Infiniband.

If you're using Debian/Ubuntu, it's very simple.

Add the following to /etc/modules

Code:

## ipoib
ib_sa
ib_cm
ib_umad
ib_addr
ib_uverbs
ib_ipoib
ib_ipath
ib_qib

Then add the following to /etc/network/interfaces (making appropriate changes for each host):

Code:

auto ib0
iface ib0 inet static
        address 192.168.1.1
        netmask 255.255.255.0
        post-up echo connected >`find /sys -name mode | grep ib0`
        post-up echo 65520 >`find /sys -name mtu | grep ib0`

Then you can bring everything up without rebooting:

Code:

for m in ib_sa ib_cm ib_umad ib_addr ib_uverbs ib_ipoib ib_ipath ib_qib; do modprobe $m; done
ifup ib0

Now you should be able to ping the IPv4 addresses using your normal ICMP ping, scp files back and forth over Infiniband, etc. And most importantly use VCL.


Oh, and I need to mention that the ip space that you use for your Infiniband network needs to be completely separate from your Ethernet network's ip space. Else you'll run into massive issues like Rurapenthe did.

So for example, if your Ethernet network is 192.168.0.0/16, then your Infiniband network should be anything but that. Maybe make it 10.0.0.0/24 or 172.16.0.0/12 or something.

It will NOT work if you use the same address space for both Ethernet and Infiniband.

[alex]

Hi, epixoip
Everything seem to be okay now. I can ping, can transfer files between nodes.
But when I run oclhashcat-plus or oclhashcat-lite, It only detect cards, and then freeze there, doing nothing.

Device #1: Cayman, 1024MB, 800Mhz, 24MCU
Device #2: Cayman, 1024MB, 800Mhz, 24MCU
Device #3: Cayman, 1024MB, 800Mhz, 24MCU
Device #4: Cayman, 1024MB, 800Mhz, 24MCU
Device #5: Cayman, 1024MB, 800Mhz, 24MCU
Device #6: Cayman, 1024MB, 800Mhz, 24MCU

Please help

[epixoip]

please provide the following:

- description of physical network topology
- /etc/network/interfaces from broker & one compute node
- /etc/vcl/nodes from the broker node

[alex]

Thanks for answer.
- physical network topology:
3 PCs, 1 broker, 2 nodes, all of them connect to an ASUS router via LAN to access internet, IP is 192.168.1.11 , .12 , .13 and connect to a infiniband switch , address 10.10.10.11, .12, .13
ibstat give interface is active , all machines can ping each other on infiniband IPv4

Broker & node are same as I clone them BEFORE setup vcl
of course on nodes, address on /etc/network/interfaces is 10.10.10.12 , other node is 10.10.10.13
/etc/network/interfaces on
Broker:

auto lo
iface lo inet loopback

auto ib0
iface ib0 inet static
address 10.10.10.11
netmask 255.255.255.0
post-up echo connected >`find /sys -name mode | grep ib0`
post-up echo 65520 >`find /sys -name mtu | grep ib0`

/etc/vcl/nodes:
10.10.10.12
10.10.10.13

[epixoip]

looks good from here.

what algorithm are you cracking? how many hashes are you targeting? what is your -n value? how much memory does your broker have?

[alex]

I use the example in guide
http://hashcat.net/wiki/doku.php?id=vcl_cluster_howto

./vclHashcat-plus64.bin -t 32 -a 7 example0.hash ?a?a?a?a example.dict
and
./vclHashcat-lite64.bin -b --benchmark-mode 1 -m 1000 --force

both of them have the same result.

[epixoip]

what version of vcl are you using?

[alex]

Hi,
I'm using the latest one, v1.21. can't find v1.17.
Last time I run with v1.20 , with normal router, not Infiniband, it works, but about 900MB/s only