Posts: 1
Threads: 1
Joined: Oct 2011
When running multiple gfx cards for example four or more 6990's. Is there a problem with pcie bus throughput? I'm doing some calculations on hardware but am unsure about how the bus will be able to handle the communications when having many cards. I've found motherboards that would handle up to 7 cards. Would that be working? Any experience?
10x
Rover
Posts: 5,185
Threads: 230
Joined: Apr 2010
none known, but remember that amd drivers are limited to 8 gpus (= 4 dual gpu cards) per computer.
Posts: 17
Threads: 3
Joined: Apr 2012
04-26-2012, 11:07 AM
(This post was last modified: 04-26-2012, 11:08 AM by splash_.)
(10-28-2011, 10:00 AM)atom Wrote: none known, but remember that amd drivers are limited to 8 gpus (= 4 dual gpu cards) per computer.
Is PCIe 8x is a bottleneck for the password cracking? The motherboard I picked has 4 PCIe (2 x 16x + 2x 8x) slots but when running 4 cards it runs all of them on 8x. Any experience with that?
Posts: 2,301
Threads: 11
Joined: Jul 2010
even PCIe 1x is fine for cracking.
Posts: 6
Threads: 0
Joined: Mar 2012
interesting you ask this question. I just upgraded my machine from 2x7970's to 3x7970's. My motherboard supports two PCIe 16x OR one PCIe 16x and two PCIe 8x. After the upgrade, all of the graphics cards work, but two of them seem to be going at half speed netting in almost no performance gain. Perhaps I am doing something wrong though because before this experience I thought exactly what undeath said.
Posts: 5,185
Threads: 230
Joined: Apr 2010
there is no speed loss due to bus speed
Posts: 649
Threads: 18
Joined: Nov 2010
Here is why it makes no difference.
A pci-e 1.0 1x lane is capable of 250MB/s transfer.
pci-e 2.0 increased this x2, so 500MB/s transfer.
Since the word lists are broken into segments and loaded into GPU memory, the transfer occurs almost instantly. Even if you have a 50GB wordlist, hashcat will scan it, break it into parts then load those parts as needed. If you have a 3GB card, and load a 3 GB segment, it would theoretically take 6 seconds to load it across all cards. I have not actually tested this myself nor do I know if it would even be possible, but using the lane speeds it should be right.
Bus speed is only a factor when wordlist segments are being loaded since all computation is done on the GPU. This includes rule generation, brute force masks, etc. The only thing that you could do to speed up the start time is use an SSD for your wordlist storage, but you will not see a speed increase in recovery speed.
Posts: 6
Threads: 0
Joined: Mar 2012
Yeah, I understand the bus speed should be just fine. No argument there. What I am saying though, is that there is something wrong with putting 3x7970 in my setup. So it is either a ocl, hashcat, driver, or hardware issue. One GPU is always twice as fast as the other two. Even when I only run 1 gpu at a time, gpu 3 is always twice as fast. Here is an example when all 3 are running:
GPU-Loops: 1024
GPU-Accel: 800
Password lengths range: 8 - 55
Platform: AMD compatible platform found
Watchdog: Temperature limit disabled
Device #1: Tahiti, 2048MB, 0Mhz, 32MCU
Device #2: Tahiti, 2048MB, 0Mhz, 32MCU
Device #3: Tahiti, 2048MB, 0Mhz, 32MCU
[s]tatus [p]ause [r]esume [q]uit => s
Status.......: Running
Hash.Target..: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Hash.Type....: MD5
Time.Running.: 5 secs
Time.Left....: 4 days, 10 hours
Plain.Mask...: ?1?1?1?1?1?1?1?1
Plain.Text...: ***aaaaa
Plain.Length.: 8
Progress.....: 85563801600/6634204312890625 (0.00%)
Speed.GPU.#1.: 4484.4M/s
Speed.GPU.#2.: 4483.7M/s
Speed.GPU.#3.: 8233.8M/s
Speed.GPU.#*.: 17318.2M/s
Posts: 5,185
Threads: 230
Joined: Apr 2010
The ~8200Mhash/s is the expected speed @md5@7970@stockclock@oclHashcat-lite
maybe you have some crossfire thing enabled? Maybe some strange setting in the bios?
Posts: 6
Threads: 0
Joined: Mar 2012
Crossfire cables are not connected. Looked at the bios but nothing seems suspicious. Turned off PCI gen 3 but that didnt seem to do anything. Any ideas?