bus speed issues?
When running multiple gfx cards for example four or more 6990's. Is there a problem with pcie bus throughput? I'm doing some calculations on hardware but am unsure about how the bus will be able to handle the communications when having many cards. I've found motherboards that would handle up to 7 cards. Would that be working? Any experience?

none known, but remember that amd drivers are limited to 8 gpus (= 4 dual gpu cards) per computer.
(10-28-2011, 10:00 AM)atom Wrote: none known, but remember that amd drivers are limited to 8 gpus (= 4 dual gpu cards) per computer.

Is PCIe 8x is a bottleneck for the password cracking? The motherboard I picked has 4 PCIe (2 x 16x + 2x 8x) slots but when running 4 cards it runs all of them on 8x. Any experience with that?
even PCIe 1x is fine for cracking.
interesting you ask this question. I just upgraded my machine from 2x7970's to 3x7970's. My motherboard supports two PCIe 16x OR one PCIe 16x and two PCIe 8x. After the upgrade, all of the graphics cards work, but two of them seem to be going at half speed netting in almost no performance gain. Perhaps I am doing something wrong though because before this experience I thought exactly what undeath said.
there is no speed loss due to bus speed
Here is why it makes no difference.

A pci-e 1.0 1x lane is capable of 250MB/s transfer.

pci-e 2.0 increased this x2, so 500MB/s transfer.

Since the word lists are broken into segments and loaded into GPU memory, the transfer occurs almost instantly. Even if you have a 50GB wordlist, hashcat will scan it, break it into parts then load those parts as needed. If you have a 3GB card, and load a 3 GB segment, it would theoretically take 6 seconds to load it across all cards. I have not actually tested this myself nor do I know if it would even be possible, but using the lane speeds it should be right.

Bus speed is only a factor when wordlist segments are being loaded since all computation is done on the GPU. This includes rule generation, brute force masks, etc. The only thing that you could do to speed up the start time is use an SSD for your wordlist storage, but you will not see a speed increase in recovery speed.
Yeah, I understand the bus speed should be just fine. No argument there. What I am saying though, is that there is something wrong with putting 3x7970 in my setup. So it is either a ocl, hashcat, driver, or hardware issue. One GPU is always twice as fast as the other two. Even when I only run 1 gpu at a time, gpu 3 is always twice as fast. Here is an example when all 3 are running:

GPU-Loops: 1024
GPU-Accel: 800
Password lengths range: 8 - 55
Platform: AMD compatible platform found
Watchdog: Temperature limit disabled
Device #1: Tahiti, 2048MB, 0Mhz, 32MCU
Device #2: Tahiti, 2048MB, 0Mhz, 32MCU
Device #3: Tahiti, 2048MB, 0Mhz, 32MCU
[s]tatus [p]ause [r]esume [q]uit => s
Status.......: Running
Hash.Target..: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Hash.Type....: MD5
Time.Running.: 5 secs
Time.Left....: 4 days, 10 hours
Plain.Mask...: ?1?1?1?1?1?1?1?1
Plain.Text...: ***aaaaa
Plain.Length.: 8
Progress.....: 85563801600/6634204312890625 (0.00%)
Speed.GPU.#1.: 4484.4M/s
Speed.GPU.#2.: 4483.7M/s
Speed.GPU.#3.: 8233.8M/s
Speed.GPU.#*.: 17318.2M/s
The ~8200Mhash/s is the expected speed @md5@7970@stockclock@oclHashcat-lite

maybe you have some crossfire thing enabled? Maybe some strange setting in the bios?
Crossfire cables are not connected. Looked at the bios but nothing seems suspicious. Turned off PCI gen 3 but that didnt seem to do anything. Any ideas?