Strange behavior when using two GPUs

[jevgienij]

I started using hashcat with my old GTX670, which does 48kH/s WPA. It is stable and can run 24/7 for a long time. Then I replaced it with a GTX970. It's as stable as the old GPU and does 170kH/s WPA. Improvement is great.

  gtx970.png (Size: 58.62 KB / Downloads: 7)

As you can see the core is at 1316 MHz, temp at 70c (never goes higher) and fan at 61%. Everything is perfect.

But then I thought why not use both cards? So I attached back GTX670 to the second PCIE slot (I have 3 in total; MB is MSI Z77A-GD65) and then GTX970 started behaving strange:

  both-cards.png (Size: 61.59 KB / Downloads: 11)

It quickly reaches 79c, fan runs faster too and gets so loud. Meanwhile GTX670 is running fine, like before, when it was alone. I tried underclocking GTX970 and it didn't help much:

  underclock.png (Size: 66.48 KB / Downloads: 5)

Core is down to 813 MHz, cracking speed to 134kH/s but the temp is still high: 74c.

I see the bus speed is 16 when GTX970 is alone but 8 when both cards are present but I read that even PCIE x1 is fine for hashcat cracking.

What am I missing, what did I do wrong?

[philsmd]

What am I missing [...]?

Founder Edition / Reference Design GPUs

[undeath]

your cooling is probably insufficient

[jevgienij]

So when just one card is in the PC, either 670 or 970, the cooling is okay but when both are present it's not enough? The whole PC case is wide open. It's hard for me to imagine how the existence of one card is making the other go crazy.

Founder Edition / Reference Design GPUs - how is it related to my issue?

[slyexe]

You have two cards stacked on each other blowing hot air onto the other card. Not only that but you Motherboard is putting out more power to the PCI-E slots so there is going to be more heat coming from that too. Reference design cards where the heat is blown out the rear of your chassis is the more efficient route. Turn your fans up and remove any backplates on the card which is stacked.

[jevgienij]

So I decided to do some testing and took everything out of the case: https://imgur.com/a/xioYZTG
This allowed me to get more distance between GPUs - about 10cm / 4in. When I put the hand in between them I didn't feel any particular heat, at least not that kind that would affect the other card. The whole system outside the case should have better cooling in general, right? But still, when both card were present, GTX970 reached over 75c. I have put it back into the case now, single-card only, then generated some more heat by stressing the CPU and GTX970 is running calmly at 60c. So... not convinced.

[jevgienij]

While it seemed to be unlikely to me, you were right. I just bought a cheap PCIe to USB 3.0 riser and took the GTX670 completely outside the case and the temperatures stabilized:

[undeath]

Thank you for reporting back! The problem with those kind of gpu coolers is that they only whirl the air around, but don't actually remove the hot air, as opposed to blower fan coolers. A closed case with proper airflow will work better than an open case for those, but that might still not be enough.