hashcat stops after about 37 minutes
#1
Hi,

I've got a large number of hashes and for each one I've got a very serious candidate that just need some mask added to get the real password.
So I've made a script that launch hashcat with my candidates (base word + case toggle) against all hashes that share the same "candidate".
The command looks like this:

Code:
./hashcat64.bin -a 6 --increment -m 120 "/foo/bar/hc-tmp/${f}.h" "/foo/bar/hc-tmp/${f}.wl" -1 ?l?u?d ?1?1?1?1?1?1 --potfile-path=/foo/bar/bouclehc.pot --status --status-timer=60 -w 4

Things go smoothly, at least until I reach an increment that brings time above 37 minutes. Strangely, hashcat will just end the session when it reaches about 37-38 minutes. It means it won't finish the last mask (?1?1?1?1?1?1).
Here are two examples:

Code:
Session..........: hashcat
Status...........: Exhausted
Hash.Type........: sha1($salt.$pass)
Hash.Target......: **********************
Time.Started.....: Thu Jul 27 13:12:01 2017 (37 mins, 49 secs)
Time.Estimated...: Sat Jul 29 02:38:36 2017 (1 day, 12 hours)
Guess.Base.......: File (/foo/bar/hc-tmp/haefxsdrgbh:.wl), Left Side
Guess.Mod........: Mask (?1?1?1?1?1?1) [6], Right Side
Guess.Charset....: -1 ?l?u?d, -2 Undefined, -3 Undefined, -4 Undefined
Speed.Dev.#2.....:   431.4 MH/s (0.93ms)
Recovered........: 0/1 (0.00%) Digests, 0/1 (0.00%) Salts
Progress.........: 988836593664/58163441238016 (1.70%)
Rejected.........: 0/988836593664 (0.00%)
Restore.Point....: 1024/1024 (100.00%)
Candidates.#2....: aefxsdrgbhlxNROR -> AEFXSDRGBHt4G080
HWMon.Dev.#2.....: Temp: 71c Fan: 47% Util: 77% Core:1835MHz Mem:4513MHz Bus:16

Started: Thu Jul 27 12:34:55 2017
Stopped: Thu Jul 27 13:49:51 2017

and

Code:
Session..........: hashcat
Status...........: Exhausted
Hash.Type........: sha1($salt.$pass)
Hash.Target......: **********************
Time.Started.....: Thu Jul 27 14:27:15 2017 (37 mins, 59 secs)
Time.Estimated...: Sat Jul 29 03:54:37 2017 (1 day, 12 hours)
Guess.Base.......: File (/foo/bar/hc-tmp/hnadinejame:.wl), Left Side
Guess.Mod........: Mask (?1?1?1?1?1?1) [6], Right Side
Guess.Charset....: -1 ?l?u?d, -2 Undefined, -3 Undefined, -4 Undefined
Speed.Dev.#2.....:   431.3 MH/s (0.93ms)
Recovered........: 0/1 (0.00%) Digests, 0/1 (0.00%) Salts
Progress.........: 988836593664/58163441238016 (1.70%)
Rejected.........: 0/988836593664 (0.00%)
Restore.Point....: 1024/1024 (100.00%)
Candidates.#2....: nadinejamelxNROR -> NADINEJAMEt4G080
HWMon.Dev.#2.....: Temp: 71c Fan: 47% Util: 77% Core:1835MHz Mem:4513MHz Bus:16

Started: Thu Jul 27 13:50:06 2017
Stopped: Thu Jul 27 15:05:15 2017

The script runs in a screen, and is basically a while true loop.

Any idea why hashcat would abort the session after about 37-38 minutes each time?
#2
Some precision: usually I can make long sessions of hashcat without any problem (several weeks), but I don't usually use -6 mode.
A quick & dirty log analysis shows this timing:
Code:
37.9167
38.8333
37.9167
42.8
37.8167
37.9833
38.0667
38.0833
38.0667
38.5333
No error, nothing special (neither in hashcat.log nor in syslog/dmesg/...)
#3
Please retry with hashcat v3.6.0 release version
#4
Hi atom,
I'm already running v3.6.0
#5
Did you try to run the commands without the for/while loop?
Does the problem also happen without the loop, e.g running the commands one after the other?
#6
I'm currently running the same command outside of a loop, and so far so good: it ran 4 hours on 5 char long mask, and the 6 char mask is running for 2 hours now.
Can't get why hashcat inside a while loop would end spontaneously well before exhaustion…
#7
Ok, so after a while (4 hours) the "out of the loop" command eventually stopped too, well ahead from schedule. Here are the 3 last status printed on screen:

Code:
Session..........: hashcat
Status...........: Running
Hash.Type........: sha1($salt.$pass)
Hash.Target......: /foo/bar/hc-tmp/hcharlestow:.h
Time.Started.....: Fri Jul 28 18:54:43 2017 (4 hours, 16 mins)
Time.Estimated...: Tue Aug  8 07:16:10 2017 (10 days, 8 hours)
Guess.Base.......: File (/foo/bar/hc-tmp/hcharlestow:.wl), Left Side
Guess.Mod........: Mask (?1?1?1?1?1?1) [6], Right Side
Guess.Charset....: -1 ?l?u?d, -2 Undefined, -3 Undefined, -4 Undefined
Speed.Dev.#2.....:   384.1 MH/s (1.08ms)
Recovered........: 2/8 (25.00%) Digests, 2/8 (25.00%) Salts
Progress.........: 6891217223680/465307529904128 (1.48%)
Rejected.........: 0/6891217223680 (0.00%)
Restore.Point....: 0/1024 (0.00%)
Candidates.#2....: charlestow8W3kie -> CHARLESTOWzNoari
HWMon.Dev.#2.....: Temp: 71c Fan: 47% Util: 80% Core:1835MHz Mem:4513MHz Bus:16

[s]tatus [p]ause [r]esume [b]ypass [c]heckpoint [q]uit =>

Session..........: hashcat
Status...........: Running
Hash.Type........: sha1($salt.$pass)
Hash.Target......: /foo/bar/hc-tmp/hcharlestow:.h
Time.Started.....: Fri Jul 28 18:54:43 2017 (4 hours, 17 mins)
Time.Estimated...: Tue Aug  8 07:13:11 2017 (10 days, 8 hours)
Guess.Base.......: File (/foo/bar/hc-tmp/hcharlestow:.wl), Left Side
Guess.Mod........: Mask (?1?1?1?1?1?1) [6], Right Side
Guess.Charset....: -1 ?l?u?d, -2 Undefined, -3 Undefined, -4 Undefined
Speed.Dev.#2.....:   384.2 MH/s (1.08ms)
Recovered........: 2/8 (25.00%) Digests, 2/8 (25.00%) Salts
Progress.........: 6914275409920/465307529904128 (1.49%)
Rejected.........: 0/6914275409920 (0.00%)
Restore.Point....: 0/1024 (0.00%)
Candidates.#2....: charlestow3HtJor -> CHARLESTOWAOoVAR
HWMon.Dev.#2.....: Temp: 71c Fan: 47% Util: 80% Core:1835MHz Mem:4513MHz Bus:16

                                                          
Session..........: hashcat
Status...........: Exhausted
Hash.Type........: sha1($salt.$pass)
Hash.Target......: /foo/bar/hc-tmp/hcharlestow:.h
Time.Started.....: Fri Jul 28 18:54:43 2017 (4 hours, 17 mins)
Time.Estimated...: Tue Aug  8 07:33:11 2017 (10 days, 8 hours)
Guess.Base.......: File (/foo/bar/hc-tmp/hcharlestow:.wl), Left Side
Guess.Mod........: Mask (?1?1?1?1?1?1) [6], Right Side
Guess.Charset....: -1 ?l?u?d, -2 Undefined, -3 Undefined, -4 Undefined
Speed.Dev.#2.....:   383.7 MH/s (1.07ms)
Recovered........: 2/8 (25.00%) Digests, 2/8 (25.00%) Salts
Progress.........: 7910692749312/465307529904128 (1.70%)
Rejected.........: 0/7910692749312 (0.00%)
Restore.Point....: 1024/1024 (100.00%)
Candidates.#2....: charlestowlxNROR -> CHARLESTOWt4G080
HWMon.Dev.#2.....: Temp: 70c Fan: 47% Util: 73% Core:1835MHz Mem:4513MHz Bus:16

Started: Fri Jul 28 14:40:22 2017
Stopped: Fri Jul 28 23:12:19 2017

Full log for this incremental session:
Code:
TOP.597b30b6.0008d9dd   START
TOP.597b30b6.0008d9dd   user_options->separator :
TOP.597b30b6.0008d9dd   user_options->custom_charset_1  ?l?u?d
TOP.597b30b6.0008d9dd   user_options->encoding_from     utf-8
TOP.597b30b6.0008d9dd   user_options->encoding_to       utf-8
TOP.597b30b6.0008d9dd   user_options->potfile_path      /foo/bar/bouclehc.pot
TOP.597b30b6.0008d9dd   user_options->rule_buf_l        :
TOP.597b30b6.0008d9dd   user_options->rule_buf_r        :
TOP.597b30b6.0008d9dd   user_options->session   hashcat
TOP.597b30b6.0008d9dd   user_options->limit     0
TOP.597b30b6.0008d9dd   user_options->skip      0
TOP.597b30b6.0008d9dd   user_options->attack_mode       6
TOP.597b30b6.0008d9dd   user_options->benchmark 0
TOP.597b30b6.0008d9dd   user_options->bitmap_max        24
TOP.597b30b6.0008d9dd   user_options->bitmap_min        16
TOP.597b30b6.0008d9dd   user_options->debug_mode        0
TOP.597b30b6.0008d9dd   user_options->force     0
TOP.597b30b6.0008d9dd   user_options->gpu_temp_abort    90
TOP.597b30b6.0008d9dd   user_options->gpu_temp_disable  0
TOP.597b30b6.0008d9dd   user_options->gpu_temp_retain   0
TOP.597b30b6.0008d9dd   user_options->hash_mode 120
TOP.597b30b6.0008d9dd   user_options->hex_charset       0
TOP.597b30b6.0008d9dd   user_options->hex_salt  0
TOP.597b30b6.0008d9dd   user_options->hex_wordlist      0
TOP.597b30b6.0008d9dd   user_options->increment 1
TOP.597b30b6.0008d9dd   user_options->increment_max     54
TOP.597b30b6.0008d9dd   user_options->increment_min     1
TOP.597b30b6.0008d9dd   user_options->keep_guessing     0
TOP.597b30b6.0008d9dd   user_options->kernel_accel      0
TOP.597b30b6.0008d9dd   user_options->kernel_loops      0
TOP.597b30b6.0008d9dd   user_options->keyspace  0
TOP.597b30b6.0008d9dd   user_options->left      0
TOP.597b30b6.0008d9dd   user_options->logfile_disable   0
TOP.597b30b6.0008d9dd   user_options->loopback  0
TOP.597b30b6.0008d9dd   user_options->machine_readable  0
TOP.597b30b6.0008d9dd   user_options->markov_classic    0
TOP.597b30b6.0008d9dd   user_options->markov_disable    0
TOP.597b30b6.0008d9dd   user_options->markov_threshold  256
TOP.597b30b6.0008d9dd   user_options->nvidia_spin_damp  100
TOP.597b30b6.0008d9dd   user_options->opencl_info       0
TOP.597b30b6.0008d9dd   user_options->opencl_vector_width       0
TOP.597b30b6.0008d9dd   user_options->outfile_autohex   1
TOP.597b30b6.0008d9dd   user_options->outfile_check_timer       5
TOP.597b30b6.0008d9dd   user_options->outfile_format    3
TOP.597b30b6.0008d9dd   user_options->potfile_disable   0
TOP.597b30b6.0008d9dd   user_options->powertune_enable  0
TOP.597b30b6.0008d9dd   user_options->quiet     0
TOP.597b30b6.0008d9dd   user_options->remove    0
TOP.597b30b6.0008d9dd   user_options->remove_timer      60
TOP.597b30b6.0008d9dd   user_options->restore   0
TOP.597b30b6.0008d9dd   user_options->restore_disable   0
TOP.597b30b6.0008d9dd   user_options->restore_timer     60
TOP.597b30b6.0008d9dd   user_options->rp_files_cnt      0
TOP.597b30b6.0008d9dd   user_options->rp_gen    0
TOP.597b30b6.0008d9dd   user_options->rp_gen_func_max   4
TOP.597b30b6.0008d9dd   user_options->rp_gen_func_min   1
TOP.597b30b6.0008d9dd   user_options->rp_gen_seed       0
TOP.597b30b6.0008d9dd   user_options->runtime   0
TOP.597b30b6.0008d9dd   user_options->scrypt_tmto       0
TOP.597b30b6.0008d9dd   user_options->segment_size      33554432
TOP.597b30b6.0008d9dd   user_options->show      0
TOP.597b30b6.0008d9dd   user_options->status    1
TOP.597b30b6.0008d9dd   user_options->status_timer      60
TOP.597b30b6.0008d9dd   user_options->stdout_flag       0
TOP.597b30b6.0008d9dd   user_options->speed_only        0
TOP.597b30b6.0008d9dd   user_options->progress_only     0
TOP.597b30b6.0008d9dd   user_options->usage     0
TOP.597b30b6.0008d9dd   user_options->username  0
TOP.597b30b6.0008d9dd   user_options->veracrypt_pim     0
TOP.597b30b6.0008d9dd   user_options->version   0
TOP.597b30b6.0008d9dd   user_options->weak_hash_threshold       0
TOP.597b30b6.0008d9dd   user_options->workload_profile  4
TOP.597b30b6.0008d9dd   hashes->hashfile        /foo/bar/hc-tmp/hcharlestow:.h
TOP.597b30b6.0008d9dd   hashes->hashlist_mode   4
TOP.597b30b6.0008d9dd   hashes->hashlist_format 0
TOP.597b30b6.0008d9dd   hashes->hashes_cnt      0
TOP.597b30b6.0008d9dd   hashes->digests_cnt     8
TOP.597b30b6.0008d9dd   hashes->digests_done    1
TOP.597b30b6.0008d9dd   hashes->salts_cnt       8
TOP.597b30b6.0008d9dd   hashes->salts_done      1
TOP.597b30b6.0008d9dd   SUB.597b30b7.000c8cf6   START
TOP.597b30b6.0008d9dd   SUB.597b30b7.000c8cf6   straight_ctx->dict      /foo/bar/hc-tmp/hcharlestow:.wl
TOP.597b30b6.0008d9dd   SUB.597b30b7.000c8cf6   mask_ctx->mask  ?1
TOP.597b30b6.0008d9dd   SUB.597b30b7.000c8cf6   runtime_start   1501245624
TOP.597b30b6.0008d9dd   SUB.597b30b7.000c8cf6   runtime_stop    1501245624
TOP.597b30b6.0008d9dd   SUB.597b30b7.000c8cf6   status-after-work       4
TOP.597b30b6.0008d9dd   SUB.597b30b7.000c8cf6   STOP
TOP.597b30b6.0008d9dd   SUB.597b30b8.00080e82   START
TOP.597b30b6.0008d9dd   SUB.597b30b8.00080e82   straight_ctx->dict      /foo/bar/hc-tmp/hcharlestow:.wl
TOP.597b30b6.0008d9dd   SUB.597b30b8.00080e82   mask_ctx->mask  ?1?1
TOP.597b30b6.0008d9dd   SUB.597b30b8.00080e82   runtime_start   1501245628
TOP.597b30b6.0008d9dd   SUB.597b30b8.00080e82   runtime_stop    1501245628
TOP.597b30b6.0008d9dd   SUB.597b30b8.00080e82   status-after-work       4
TOP.597b30b6.0008d9dd   SUB.597b30b8.00080e82   STOP
TOP.597b30b6.0008d9dd   SUB.597b30bc.000e94f8   START
TOP.597b30b6.0008d9dd   SUB.597b30bc.000e94f8   straight_ctx->dict      /foo/bar/hc-tmp/hcharlestow:.wl
TOP.597b30b6.0008d9dd   SUB.597b30bc.000e94f8   mask_ctx->mask  ?1?1?1
TOP.597b30b6.0008d9dd   SUB.597b30bc.000e94f8   runtime_start   1501245633
TOP.597b30b6.0008d9dd   SUB.597b30bc.000e94f8   runtime_stop    1501245637
TOP.597b30b6.0008d9dd   SUB.597b30bc.000e94f8   status-after-work       4
TOP.597b30b6.0008d9dd   SUB.597b30bc.000e94f8   STOP
TOP.597b30b6.0008d9dd   SUB.597b30c5.000bdcb6   START
TOP.597b30b6.0008d9dd   SUB.597b30c5.000bdcb6   straight_ctx->dict      /foo/bar/hc-tmp/hcharlestow:.wl
TOP.597b30b6.0008d9dd   SUB.597b30c5.000bdcb6   mask_ctx->mask  ?1?1?1?1
TOP.597b30b6.0008d9dd   SUB.597b30c5.000bdcb6   runtime_start   1501245642
TOP.597b30b6.0008d9dd   SUB.597b30c5.000bdcb6   runtime_stop    1501245913
TOP.597b30b6.0008d9dd   SUB.597b30c5.000bdcb6   status-after-work       4
TOP.597b30b6.0008d9dd   SUB.597b30c5.000bdcb6   STOP
TOP.597b30b6.0008d9dd   SUB.597b31d9.000244a7   START
TOP.597b30b6.0008d9dd   SUB.597b31d9.000244a7   straight_ctx->dict      /foo/bar/hc-tmp/hcharlestow:.wl
TOP.597b30b6.0008d9dd   SUB.597b31d9.000244a7   mask_ctx->mask  ?1?1?1?1?1
TOP.597b30b6.0008d9dd   SUB.597b31d9.000244a7   runtime_start   1501245917
TOP.597b30b6.0008d9dd   SUB.597b31d9.000244a7   runtime_stop    1501260879
TOP.597b30b6.0008d9dd   SUB.597b31d9.000244a7   status-after-work       4
TOP.597b30b6.0008d9dd   SUB.597b31d9.000244a7   STOP
TOP.597b30b6.0008d9dd   SUB.597b6c4f.00037489   START
TOP.597b30b6.0008d9dd   SUB.597b6c4f.00037489   straight_ctx->dict      /foo/bar/hc-tmp/hcharlestow:.wl
TOP.597b30b6.0008d9dd   SUB.597b6c4f.00037489   mask_ctx->mask  ?1?1?1?1?1?1
TOP.597b30b6.0008d9dd   SUB.597b6c4f.00037489   runtime_start   1501260883
TOP.597b30b6.0008d9dd   SUB.597b6c4f.00037489   runtime_stop    1501276338
TOP.597b30b6.0008d9dd   SUB.597b6c4f.00037489   status-after-work       4
TOP.597b30b6.0008d9dd   SUB.597b6c4f.00037489   STOP
TOP.597b30b6.0008d9dd   proc_start      1501245622
TOP.597b30b6.0008d9dd   proc_stop       1501276339
TOP.597b30b6.0008d9dd   STOP
#8
Does the problem also happen without the --increment command line option?
#9
I've juste finished a test, it shows the same behavior without incremental:

Code:
Session..........: hashcat
Status...........: Exhausted
Hash.Type........: sha1($salt.$pass)
Hash.Target......: /foo/bar/hc-tmp/hcassandraf:.h
Time.Started.....: Sat Jul 29 13:34:07 2017 (2 hours, 8 mins)
Time.Estimated...: Thu Aug  3 19:07:33 2017 (5 days, 3 hours)
Guess.Base.......: File (/foo/bar/hc-tmp/hcassandraf:.wl), Left Side
Guess.Mod........: Mask (?1?1?1?1?1?1) [6], Right Side
Guess.Charset....: -1 ?l?u?d, -2 Undefined, -3 Undefined, -4 Undefined
Speed.Dev.#2.....:   386.1 MH/s (1.07ms)
Recovered........: 3/6 (50.00%) Digests, 3/6 (50.00%) Salts
Progress.........: 5933019561984/348980647428096 (1.70%)
Rejected.........: 0/5933019561984 (0.00%)
Restore.Point....: 1024/1024 (100.00%)
Candidates.#2....: cassandraf991cha -> CASSANDRAFwelfin
HWMon.Dev.#2.....: Temp: 72c Fan: 48% Util: 80% Core:1847MHz Mem:4513MHz Bus:16

Started: Sat Jul 29 13:34:01 2017
Stopped: Sat Jul 29 15:42:34 2017
#10
What I wanted to make sure is that you're not using the latest beta but the release version. Please confirm.