How to diagnose hashcat progress stalling?

[serious-business]

I'm on an Intel MacBook Pro with AMD GPU and running hashcat v6.2.6-1259, built from source.  It was eventually able to crack a real-world WPA2 hash, but instead of taking ~2 hours as estimated, it took a couple days.  Hashcat would be making great progress, and then, suddenly, progress percentage would stop increasing, fans would slow down, and GPU utilization (? Hardware.Mon.#01 & #02.: Util) would drop to 0%

Even leaving hashcat alone overnight, it did not start making progress again.  The abort trigger temp was not exceeded, the battery did not get low, and I prevented the MacBook from going to sleep.  Eventually, I started checkpointing every 10–20%, but sometimes hashcat would stall after the request, and I'd have to quit, restore, and try again.  I think it finally recovered the password at 78% from a restore point of around 60%.

The initial command I used was 

Code:

hashcat --session name -D 2,1 -d 1,2 -m 22000 hash.hc22000 -a 6 dict.txt ?mask

I am now working on a different hash, which again has a reasonable estimated time to completion, but hashcat is again dying on me.  I requested a checkpoint at 10%, and it kept going rapidly until 19.42%, but now it's stopped.  ps says the process is in uninterruptible wait.

What other possibilities are there for what's going on, or where can I find additional clues?  Is there something I should look for in the session log file?

[Chick3nman]

First, I would NOT attempt to use the CPU device at the same time as the GPU device. This will only lead to worse overall performance in the majority of cases, especially on an already weak system.

Second, how big is your dictionary? What is the mask you are trying to run? Perhaps this is a workload issue leading to a low power or idle state, but it's impossible to tell without some more information. I suspect it's not, but worth investigating that avenue first as it's an easy fix.

[serious-business]

First, I would NOT attempt to use the CPU device at the same time as the GPU device. This will only lead to worse overall performance in the majority of cases, especially on an already weak system.

I accidentally did leave 1 on the -D flag (thought it meant order of preference of device type), but I only ended up using the discrete and integrated GPUs with -d 1,2, which are  

Code:

Metal Info:
===========

Metal.Version.: 343.21

Backend Device ID #01 (Alias: #05)
  Type...........: GPU
  Vendor.ID......: 2
  Vendor.........: Apple
  Name...........: AMD Radeon Pro Vega 20
...

Backend Device ID #02 (Alias: #04)
  Type...........: GPU
  Vendor.ID......: 2
  Vendor.........: Apple
  Name...........: Intel(R) UHD Graphics 630
...

I actually did try to use my CPU device (#3) with the GPUs, but -d 1,2,3 would result in hashcat skipping the GPUs and only using the CPU.

Second, how big is your dictionary? What is the mask you are trying to run? Perhaps this is a workload issue leading to a low power or idle state, but it's impossible to tell without some more information. I suspect it's not, but worth investigating that avenue first as it's an easy fix.

Keyspace was 501972000, with both GPUs getting 62357 H/s and an estimated time to completion of just 2 hours, 12 mins.  Doing a quick test now, that estimate seems about right, because it did 4.45% progress in ~5 minutes.  

And that type of performance is fine by me, except that 2 hours extends to 2 days, because I start hc, monitor it for a bit, think, "wow, it sure is working fast", then leave and come back to find progress has stalled.  Then I have to hit 'q' to exit, and restart the session, hoping hc had reached the next restore point.  

The issue shouldn't be low power.  I know this particular MacBook is brilliantly designed, so that even if it's plugged in to a high-wattage power adapter, using the dGPU causes the battery level to drop, and I think at 20% the dGPU gets shut off.  In running hc, I never saw the battery get that low (hc is always idling by then), and anyway, wouldn't the iGPU keep working without the dGPU?

I dealt with idle sleeping by using the macOS caffeinate command, which "creates assertions to alter system sleep behavior."  I even tried preventing the display from sleeping, and that didn't stop hc from stalling.

For this reply, I started an hc session, and it ran 47 mintues before I caught it not working.  At 32.14% progress, it was alive, and at 32.18% progress it was dead.  Battery was 52% (but maybe it had gone down to 50%?), and I was at the laptop the whole time with the display on.

Thank you, I hope I answered all your questions.  I suppose my next step is to check if 50% battery is the killer threshold.

[serious-business]

I suppose my next step is to check if 50% battery is the killer threshold.

Nope, 50% battery isn't a killswitch.  This time, hc worked for about an hour.  At 33.36% progress it was fine, at 33.72%, stalled.  Max GPU temp, 73 C.

[Chick3nman]

This is strange behavior and my best guess is that it has something to do with your system outside of hashcat, perhaps a thermal event or power saving or such. It could even be a "this process appears to be stalled" sort of event, as we've seen with windows and some linux systems before. I'm honestly not sure how to go about diagnosing this, are there system logs you can look for that indicate anything?