Posts: 38
Threads: 5
Joined: Nov 2013
I recently stood up a 4 GPU system, and noticed something strange. When I run a benchmark -
hashcat64.exe -b -m1000
Quote:OpenCL Platform #1: NVIDIA Corporation
======================================
* Device #1: GeForce RTX 2080, 2048/8192 MB allocatable, 46MCU
* Device #2: GeForce GTX 1080 Ti, 2816/11264 MB allocatable, 28MCU
* Device #3: GeForce GTX 1070, 2048/8192 MB allocatable, 15MCU
* Device #4: GeForce GTX 980 Ti, 1536/6144 MB allocatable, 22MCU
OpenCL Platform #2: Intel(R) Corporation
========================================
* Device #5: Intel(R) HD Graphics 4600, skipped.
* Device #6: Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz, skipped.
Benchmark relevant options:
===========================
* --optimized-kernel-enable
Hashmode: 1000 - NTLM
Speed.#1.........: 69485.6 MH/s (10.40ms) @ Accel:16 Loops:1024 Thr:1024 Vec:2
Speed.#2.........: 39607.4 MH/s (23.18ms) @ Accel:32 Loops:1024 Thr:1024 Vec:2
Speed.#3.........: 36190.7 MH/s (27.03ms) @ Accel:64 Loops:1024 Thr:1024 Vec:2
Speed.#4.........: 31403.6 MH/s (22.95ms) @ Accel:32 Loops:1024 Thr:1024 Vec:2
Speed.#*.........: 176.7 GH/s
Started: Tue Aug 27 14:41:16 2019
Stopped: Tue Aug 27 14:41:31 2019
I get those numbers.
But when actually cracking NTLM hashes, I'll see something very different.
hashcat64.exe -m1000 -a3 -w4
Quote:Session..........: hashcat
Status...........: Quit
Hash.Type........: NTLM
Hash.Target......: (redacted)
Time.Started.....: Tue Aug 27 14:48:11 2019 (10 secs)
Time.Estimated...: Tue Aug 27 14:49:49 2019 (1 min, 28 secs)
Guess.Mask.......: ?1?1?1?1?1?1?1?1?1?1?1 [11]
Guess.Charset....: (redacted)
Guess.Queue......: 1/1 (100.00%)
Speed.#1.........: 14942.9 MH/s (4.96ms) @ Accel:16 Loops:128 Thr:1024 Vec:1
Speed.#2.........: 9676.2 MH/s (10.58ms) @ Accel:32 Loops:128 Thr:1024 Vec:1
Speed.#3.........: 7701.1 MH/s (7.40ms) @ Accel:64 Loops:64 Thr:1024 Vec:1
Speed.#4.........: 8809.2 MH/s (9.49ms) @ Accel:32 Loops:128 Thr:1024 Vec:1
Speed.#*.........: 41129.3 MH/s
Recovered........: 1/5 (20.00%) Digests, 0/1 (0.00%) Salts
Progress.........: 412132311040/4049565169664 (10.18%)
Rejected.........: 0/412132311040 (0.00%)
Restore.Point....: 144474112/1475789056 (9.79%)
Restore.Sub.#1...: Salt:0 Amplifier:1152-1280 Iteration:0-128
Restore.Sub.#2...: Salt:0 Amplifier:1408-1536 Iteration:0-128
Restore.Sub.#3...: Salt:0 Amplifier:1664-1728 Iteration:0-64
Restore.Sub.#4...: Salt:0 Amplifier:1664-1792 Iteration:0-128
Candidates.#1....: kDKkwk!wkdd -> w!LdLw3rr!0
Candidates.#2....: DDrkkWrdWKL -> Rw!r0w!wkdd
Candidates.#3....: !Ll0wk$k$wk -> w$03$RRKkrd
Candidates.#4....: !LlRD0$wWKL -> $lDrkWrdWKL
Hardware.Mon.#1..: Temp: 38c Fan: 71% Util: 92% Core:2010MHz Mem:6800MHz Bus:1
Hardware.Mon.#2..: Temp: 28c Fan: 40% Util: 97% Core:1075MHz Mem:5005MHz Bus:1
Hardware.Mon.#3..: Temp: 43c Fan: 97% Util: 95% Core:2037MHz Mem:3802MHz Bus:1
Hardware.Mon.#4..: Temp: 44c Fan:100% Util: 97% Core:1379MHz Mem:3304MHz Bus:1
I'm aware that multiple hashes in the list can slow it down, but I get the same numbers with 1 or 5 (which is in the list in this case). I can re-run with 1 hash if needed, but does anyone have any ideas as to why I'm seeing SUCH a huge difference? I could understand a few % points slower, but not THAT much of a difference.
I realize there's quite possibly something I'm missing here, but I've been chasing this for the last few days and I'm out of ideas.
Posts: 2,301
Threads: 11
Joined: Jul 2010
Posts: 38
Threads: 5
Joined: Nov 2013
They're actually slower with -O
Quote:Speed.#1.........: 6083.7 MH/s (4.21ms) @ Accel:16 Loops:196 Thr:1024 Vec:2
Speed.#2.........: 8256.4 MH/s (7.03ms) @ Accel:32 Loops:196 Thr:1024 Vec:2
Speed.#3.........: 8173.3 MH/s (5.34ms) @ Accel:64 Loops:98 Thr:1024 Vec:2
Speed.#4.........: 9354.2 MH/s (8.47ms) @ Accel:32 Loops:196 Thr:1024 Vec:2
Speed.#*.........: 31867.5 MH/s
Posts: 38
Threads: 5
Joined: Nov 2013
For sanity's sake here's what version I'm running these with.
Quote:hashcat (v5.1.0) starting in benchmark mode...
Posts: 2,301
Threads: 11
Joined: Jul 2010
can you try with -w3?
Weirdly the loops are much lower for your actual run compared to the benchmark.
Posts: 2,267
Threads: 16
Joined: Feb 2013
08-28-2019, 10:42 AM
(This post was last modified: 08-28-2019, 10:52 AM by philsmd.)
furthermore, this mask is kind of pointing to a possible problem: ?1?1?1?1?1?1?1?1?1?1?1
if --custom-charset1 (or short -1) is defined with a very small number of chars (small charset), that might be the main reason why you don't get full acceleration. You should get a warning in such cases after a few seconds/minutes.
Without knowing how small/huge that keyspace is, it's very difficult to say if you can reach full acceleration.
try with a mask of ?b?b?b?b?b?b?b?b?b instead, just for the testing purpose.
also see https://hashcat.net/faq/morework
btw: single hash vs 2+ hashes makes a huge difference, because of optimizations that can only be performed when having only one hash (depends on the hash type of course)
update: according to this number 4049565169664 the --custom-charset1 only consists of 14 chars because 14^11 = 4,049,565,169,664
that's already quite small number of chars (almost like ?d = 10 digits)
Posts: 38
Threads: 5
Joined: Nov 2013
08-28-2019, 05:32 PM
(This post was last modified: 08-28-2019, 05:32 PM by XakEp.)
I'll boot it up and test different commands.
I'm only posting NTLM hashes, but I see the same issues across the board with other hash types with single hashes as well. Gimme a few and I'll have new data. Thank you so much for your time, this really has me wondering.
Posts: 38
Threads: 5
Joined: Nov 2013
08-28-2019, 07:04 PM
(This post was last modified: 08-28-2019, 07:05 PM by XakEp.)
Ok, new pass with w3 and no mask
Quote:Session..........: hashcat
Status...........: Quit
Hash.Type........: NTLM
Hash.Target......: (redacted)
Time.Started.....: Wed Aug 28 10:50:32 2019 (20 secs)
Time.Estimated...: Wed Aug 28 10:52:27 2019 (1 min, 35 secs)
Guess.Mask.......: ?1?2?2?2?2?2?2?3 [8]
Guess.Charset....: -1 ?l?d?u, -2 ?l?d, -3 ?l?d*!$@_, -4 Undefined
Guess.Queue......: 8/15 (53.33%)
Speed.#1.........: 15715.1 MH/s (33.64ms) @ Accel:16 Loops:1024 Thr:1024 Vec:1
Speed.#2.........: 12668.2 MH/s (61.15ms) @ Accel:32 Loops:1024 Thr:1024 Vec:1
Speed.#3.........: 10444.3 MH/s (83.83ms) @ Accel:64 Loops:1024 Thr:1024 Vec:1
Speed.#4.........: 9128.6 MH/s (65.49ms) @ Accel:32 Loops:1024 Thr:1024 Vec:1
Speed.#*.........: 47956.2 MH/s
Recovered........: 0/1 (0.00%) Digests, 0/1 (0.00%) Salts
Progress.........: 970338598912/5533380698112 (17.54%)
Rejected.........: 0/970338598912 (0.00%)
Restore.Point....: 6750208/68864256 (9.80%)
Restore.Sub.#1...: Salt:0 Amplifier:20480-21504 Iteration:0-1024
Restore.Sub.#2...: Salt:0 Amplifier:37888-38912 Iteration:0-1024
Restore.Sub.#3...: Salt:0 Amplifier:54272-55296 Iteration:0-1024
Restore.Sub.#4...: Salt:0 Amplifier:15360-16384 Iteration:0-1024
Candidates.#1....: evcer2il -> Fmcnjgyl
Candidates.#2....: 0q3w8mxo -> Cjdn8q81
Candidates.#3....: 3dptju62 -> E7g646t1
Candidates.#4....: Rwstjq81 -> httqx4x3
Hardware.Mon.#1..: Temp: 37c Fan: 73% Util: 69% Core:2025MHz Mem:6800MHz Bus:1
Hardware.Mon.#2..: Temp: 29c Fan: 40% Util: 66% Core:1632MHz Mem:5005MHz Bus:1
Hardware.Mon.#3..: Temp: 47c Fan: 99% Util: 98% Core:1999MHz Mem:3802MHz Bus:1
Hardware.Mon.#4..: Temp: 47c Util: 86% Core:1379MHz Mem:3304MHz Bus:1
Started: Wed Aug 28 10:50:07 2019
Stopped: Wed Aug 28 10:50:54 2019
Still far lower than the benchmarks. I monkey'd with it a bit, here's my command -
hashcat64.exe --stfout d:\wordlists\rockyou.txt -r rules\best64.rule| hashcat64.exe -m1000 -a3 d:\hashlists\Random\test.txt -w4 --status --status-timer=2
Quote:Session..........: hashcat
Status...........: Running
Hash.Type........: NTLM
Hash.Target......: (redacted)
Time.Started.....: Wed Aug 28 10:59:00 2019 (5 secs)
Time.Estimated...: Wed Aug 28 11:00:55 2019 (1 min, 50 secs)
Guess.Mask.......: ?1?2?2?2?2?2?2?3 [8]
Guess.Charset....: -1 ?l?d?u, -2 ?l?d, -3 ?l?d*!$@_, -4 Undefined
Guess.Queue......: 8/15 (53.33%)
Speed.#1.........: 15696.5 MH/s (33.75ms) @ Accel:16 Loops:1024 Thr:1024 Vec:1
Speed.#2.........: 12508.2 MH/s (61.46ms) @ Accel:32 Loops:1024 Thr:1024 Vec:1
Speed.#3.........: 10442.6 MH/s (84.05ms) @ Accel:64 Loops:1024 Thr:1024 Vec:1
Speed.#4.........: 9220.6 MH/s (65.98ms) @ Accel:32 Loops:1024 Thr:1024 Vec:1
Speed.#*.........: 47874.7 MH/s
Recovered........: 0/1 (0.00%) Digests, 0/1 (0.00%) Salts
Progress.........: 255241224192/5533380698112 (4.61%)
Rejected.........: 0/255241224192 (0.00%)
Restore.Point....: 0/68864256 (0.00%)
Restore.Sub.#1...: Salt:0 Amplifier:30720-31744 Iteration:0-1024
Restore.Sub.#2...: Salt:0 Amplifier:72704-73728 Iteration:0-1024
Restore.Sub.#3...: Salt:0 Amplifier:56320-57344 Iteration:0-1024
Restore.Sub.#4...: Salt:0 Amplifier:68608-69632 Iteration:0-1024
Candidates.#1....: 7uew3cat -> Xokq7tmo
Candidates.#2....: qzqtwhll -> jo7kp7qu
Candidates.#3....: vu1d2uqu -> V7hqxjpe
Candidates.#4....: Aq7erw56 -> au4z2wat
Hardware.Mon.#1..: Temp: 41c Fan: 80% Util: 74% Core:2025MHz Mem:6800MHz Bus:1
Hardware.Mon.#2..: Temp: 30c Fan: 40% Util: 80% Core:1316MHz Mem:5005MHz Bus:1
Hardware.Mon.#3..: Temp: 47c Fan: 88% Util: 88% Core:2012MHz Mem:3802MHz Bus:1
Hardware.Mon.#4..: Temp: 43c Util: 83% Core:1379MHz Mem:3304MHz Bus:1
The I tried with -O, and its definitely step in the right direction, but still nowhere near the benchmark.
Quote:Session..........: hashcat
Status...........: Running
Hash.Type........: NTLM
Hash.Target......: (redacted)
Time.Started.....: Wed Aug 28 10:57:12 2019 (45 secs)
Time.Estimated...: Wed Aug 28 10:58:06 2019 (9 secs)
Guess.Mask.......: ?1?2?2?2?2?2?2?3 [8]
Guess.Charset....: -1 ?l?d?u, -2 ?l?d, -3 ?l?d*!$@_, -4 Undefined
Guess.Queue......: 8/15 (53.33%)
Speed.#1.........: 27389.1 MH/s (2.82ms) @ Accel:16 Loops:256 Thr:1024 Vec:2
Speed.#2.........: 27719.1 MH/s (9.13ms) @ Accel:32 Loops:256 Thr:1024 Vec:2
Speed.#3.........: 23486.3 MH/s (6.91ms) @ Accel:64 Loops:256 Thr:1024 Vec:2
Speed.#4.........: 21406.3 MH/s (5.67ms) @ Accel:32 Loops:256 Thr:1024 Vec:2
Speed.#*.........: 100.0 GH/s
Recovered........: 0/1 (0.00%) Digests, 0/1 (0.00%) Salts
Progress.........: 4547541729280/5533380698112 (82.18%)
Rejected.........: 0/4547541729280 (0.00%)
Restore.Point....: 2033451008/2479113216 (82.02%)
Restore.Sub.#1...: Salt:0 Amplifier:768-1024 Iteration:0-256
Restore.Sub.#2...: Salt:0 Amplifier:512-1024 Iteration:0-512
Restore.Sub.#3...: Salt:0 Amplifier:1024-1280 Iteration:0-256
Restore.Sub.#4...: Salt:0 Amplifier:1792-2048 Iteration:0-256
Candidates.#1....: vcecp3ps -> 6govujps
Candidates.#2....: g8nvujps -> 6g2ti3qs
Candidates.#3....: Mg5tmios -> Lkqbp3ps
Candidates.#4....: Oshk63os -> mx2tmios
Hardware.Mon.#1..: Temp: 40c Fan: 75% Util: 72% Core:2025MHz Mem:6800MHz Bus:1
Hardware.Mon.#2..: Temp: 31c Fan: 46% Util: 76% Core:1480MHz Mem:5005MHz Bus:1
Hardware.Mon.#3..: Temp: 49c Fan:100% Util: 78% Core:2025MHz Mem:3802MHz Bus:1
Hardware.Mon.#4..: Temp: 49c Util: 65% Core:1379MHz Mem:3304MHz Bus:1
I'm not a stupid person, but I really don't understand what's going on here. Apologies if I'm missing something simple.
Posts: 2,301
Threads: 11
Joined: Jul 2010
try hashcat64.exe -m1000 -a3 d:\hashlists\Random\test.txt ?a?a?a?a?a?a?a?a -w3 -O
Posts: 38
Threads: 5
Joined: Nov 2013
Ho ho, progress!
Quote:Session..........: hashcat
Status...........: Running
Hash.Type........: NTLM
Hash.Target......: (redacted)
Time.Started.....: Wed Aug 28 13:06:28 2019 (2 secs)
Time.Estimated...: Thu Aug 29 04:22:03 2019 (15 hours, 15 mins)
Guess.Mask.......: ?a?a?a?a?a?a?a?a [8]
Guess.Queue......: 1/1 (100.00%)
Speed.#1.........: 49367.6 MH/s (10.82ms) @ Accel:16 Loops:1024 Thr:1024 Vec:2
Speed.#2.........: 25596.0 MH/s (19.26ms) @ Accel:32 Loops:1024 Thr:1024 Vec:2
Speed.#3.........: 25548.6 MH/s (27.59ms) @ Accel:64 Loops:1024 Thr:1024 Vec:2
Speed.#4.........: 20110.2 MH/s (22.47ms) @ Accel:32 Loops:1024 Thr:1024 Vec:2
Speed.#*.........: 120.7 GH/s
Recovered........: 0/1 (0.00%) Digests, 0/1 (0.00%) Salts
Progress.........: 244108623872/6634204312890625 (0.00%)
Rejected.........: 0/244108623872 (0.00%)
Restore.Point....: 21397504/735091890625 (0.00%)
Restore.Sub.#1...: Salt:0 Amplifier:7168-8192 Iteration:0-1024
Restore.Sub.#2...: Salt:0 Amplifier:2048-3072 Iteration:0-1024
Restore.Sub.#3...: Salt:0 Amplifier:7168-8192 Iteration:0-1024
Restore.Sub.#4...: Salt:0 Amplifier:2048-3072 Iteration:0-1024
Candidates.#1....: Pk^u]> -> wv3GR?
Candidates.#2....: E5.;C@ -> 6@HzIA
Candidates.#3....: Pk;L[< -> wv.Ei=
Candidates.#4....: E5bGR? -> 6@G;C@
Hardware.Mon.#1..: Temp: 35c Fan: 45% Util: 85% Core:2040MHz Mem:6800MHz Bus:1
Hardware.Mon.#2..: Temp: 25c Fan: 40% Util: 55% Core:1632MHz Mem:5005MHz Bus:1
Hardware.Mon.#3..: Temp: 38c Fan: 45% Util: 66% Core:2037MHz Mem:3802MHz Bus:1
Hardware.Mon.#4..: Temp: 34c Fan: 61% Util: 59% Core:1379MHz Mem:3304MHz Bus:1
Speeds are higher, but util % are still low.
|
