Help me figure it out vast.ai/-m 23800

[Crew]

Currently working with hash 23800 RAR3-p (Compressed) on cloud service vast.ai
Using docker Hashcat CUDA https://hub.docker.com/r/dizcza/docker-hashcat/

The problem is that I get very strange speed data.
I have tried many Templates already and here is what I get, for example. (for the test I used the mask ?а?а?а?а?а by setting the -O option)

4*4090

Code:

Session..........: hashcat
Status...........: Running
Hash.Mode........: 23800 (RAR3-p (Compressed))
Hash.Target......: $RAR3$*1*2597b58e3fafb7d9*99875cb5*816*2036*1*45b30...6a3*33
Time.Started.....: Fri Apr 11 16:32:45 2025 (1 min, 0 secs)
Time.Estimated...: Sat Apr 12 11:42:52 2025 (19 hours, 9 mins)
Kernel.Feature...: Optimized Kernel
Guess.Mask.......: ?a?a?a?a?a [5]
Guess.Queue......: 1/1 (100.00%)
Speed.#1.........:    28034 H/s (16.54ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#2.........:    28033 H/s (16.55ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#3.........:    28035 H/s (16.49ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#4.........:    28031 H/s (16.45ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#*.........:   112.1 kH/s
Recovered........: 0/1 (0.00%) Digests (total), 0/1 (0.00%) Digests (new)
Progress.........: 6553600/7737809375 (0.08%)
Rejected.........: 0/6553600 (0.00%)
Restore.Point....: 0/81450625 (0.00%)
Restore.Sub.#1...: Salt:0 Amplifier:25-26 Iteration:245760-262144
Restore.Sub.#2...: Salt:0 Amplifier:25-26 Iteration:245760-262144
Restore.Sub.#3...: Salt:0 Amplifier:25-26 Iteration:245760-262144
Restore.Sub.#4...: Salt:0 Amplifier:25-26 Iteration:245760-262144
Candidate.Engine.: Device Generator
Candidates.#1....: 4arie -> 4[tch
Candidates.#2....: 4^$@1 -> 4/#51
Candidates.#3....: 4Xege -> 4Bzst
Candidates.#4....: 4D+45 -> 4zi45
Hardware.Mon.#1..: Temp: 38c Fan: 30% Util:  0% Core:2670MHz Mem:10501MHz Bus:8
Hardware.Mon.#2..: Temp: 41c Fan: 30% Util:  0% Core:2685MHz Mem:10501MHz Bus:8
Hardware.Mon.#3..: Temp: 40c Fan: 30% Util:  0% Core:2715MHz Mem:10501MHz Bus:8
Hardware.Mon.#4..: Temp: 38c Fan: 30% Util:  0% Core:2685MHz Mem:10501MHz Bus:8

12*4090

Code:

Session..........: hashcat
Status...........: Running
Hash.Mode........: 23800 (RAR3-p (Compressed))
Hash.Target......: $RAR3$*1*2597b58e3fafb7d9*99875cb5*816*2036*1*45b30...6a3*33
Time.Started.....: Fri Apr 11 18:01:17 2025 (45 secs)
Time.Estimated...: Sat Apr 12 14:12:27 2025 (20 hours, 10 mins)
Kernel.Feature...: Optimized Kernel
Guess.Mask.......: ?a?a?a?a?a [5]
Guess.Queue......: 1/1 (100.00%)
Speed.#1.........:     8753 H/s (16.17ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#2.........:     8753 H/s (16.31ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#3.........:     8754 H/s (16.04ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#4.........:    10210 H/s (16.03ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#5.........:     8752 H/s (16.19ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#6.........:     8745 H/s (16.01ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#7.........:     8753 H/s (16.05ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#8.........:     8752 H/s (16.12ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#9.........:     8752 H/s (16.10ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#10.........:     8752 H/s (16.12ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#11.........:     8751 H/s (16.15ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#12.........:     8752 H/s (16.14ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#*.........:   106.5 kH/s
Recovered........: 0/1 (0.00%) Digests (total), 0/1 (0.00%) Digests (new)
Progress.........: 4718592/7737809375 (0.06%)
Rejected.........: 0/4718592 (0.00%)
Restore.Point....: 0/81450625 (0.00%)
Restore.Sub.#1...: Salt:0 Amplifier:5-6 Iteration:245760-262144
Restore.Sub.#2...: Salt:0 Amplifier:5-6 Iteration:245760-262144
Restore.Sub.#3...: Salt:0 Amplifier:5-6 Iteration:245760-262144
Restore.Sub.#4...: Salt:0 Amplifier:6-7 Iteration:245760-262144
Restore.Sub.#5...: Salt:0 Amplifier:5-6 Iteration:245760-262144
Restore.Sub.#6...: Salt:0 Amplifier:5-6 Iteration:245760-262144
Restore.Sub.#7...: Salt:0 Amplifier:5-6 Iteration:245760-262144
Restore.Sub.#8...: Salt:0 Amplifier:5-6 Iteration:245760-262144
Restore.Sub.#9...: Salt:0 Amplifier:5-6 Iteration:245760-262144
Restore.Sub.#10...: Salt:0 Amplifier:5-6 Iteration:245760-262144
Restore.Sub.#11...: Salt:0 Amplifier:5-6 Iteration:245760-262144
Restore.Sub.#12...: Salt:0 Amplifier:5-6 Iteration:245760-262144
Candidate.Engine.: Device Generator
Candidates.#1....: aarie -> a[tch
Candidates.#2....: a^$@1 -> a/#51
Candidates.#3....: aXege -> aBzst
Candidates.#4....: 0D+45 -> 0zi45
Candidates.#5....: axe32 -> ap8xa
Candidates.#6....: aw9za -> a8Fma
Candidates.#7....: anFma -> a{]we
Candidates.#8....: a|`!! -> a;iBO
Candidates.#9....: a\hJA -> aF7&m
Candidates.#10....: aZ7&m -> aYiQU
Candidates.#11....: aTdWE -> ag<ST
Candidates.#12....: af:QU -> aytQU
Hardware.Mon.#1..: Temp: 31c Fan: 40% Util:  0% Core:2520MHz Mem:10501MHz Bus:8
Hardware.Mon.#2..: Temp: 31c Fan: 40% Util:  0% Core:2520MHz Mem:10501MHz Bus:8
Hardware.Mon.#3..: Temp: 30c Fan: 33% Util:  0% Core:2565MHz Mem:10501MHz Bus:8
Hardware.Mon.#4..: Temp: 29c Fan: 32% Util:  0% Core:2595MHz Mem:10501MHz Bus:8
Hardware.Mon.#5..: Temp: 32c Fan: 40% Util:  0% Core:2520MHz Mem:10501MHz Bus:8
Hardware.Mon.#6..: Temp: 29c Fan: 30% Util:  0% Core:2565MHz Mem:10501MHz Bus:8
Hardware.Mon.#7..: Temp: 28c Fan: 32% Util:  0% Core:2595MHz Mem:10501MHz Bus:8
Hardware.Mon.#8..: Temp: 32c Fan: 40% Util:  0% Core:2520MHz Mem:10501MHz Bus:8
Hardware.Mon.#9..: Temp: 32c Fan: 40% Util:  0% Core:2565MHz Mem:10501MHz Bus:8
Hardware.Mon.#10..: Temp: 31c Fan: 40% Util:  0% Core:2565MHz Mem:10501MHz Bus:8
Hardware.Mon.#11..: Temp: 32c Fan: 41% Util:  0% Core:2565MHz Mem:10501MHz Bus:8
Hardware.Mon.#12..: Temp: 33c Fan: 41% Util:  0% Core:2565MHz Mem:10501MHz Bus:8

When using 4090 the END-OF-speed is in the range of 110 - 130 kH/s (I tried from 4 to 12 4090, when using 1 4090 the speed does not reach 100 kH/s)

Please tell me, is this a normal speed for 4090?

Next I tried 5090 and got the following results

1*5090

Code:

Session..........: hashcat
Status...........: Running
Hash.Mode........: 23800 (RAR3-p (Compressed))
Hash.Target......: $RAR3$*1*2597b58e3fafb7d9*99875cb5*816*2036*1*45b30...6a3*33
Time.Started.....: Fri Apr 11 17:54:24 2025 (17 secs)
Time.Estimated...: Sat Apr 12 15:40:30 2025 (21 hours, 45 mins)
Kernel.Feature...: Optimized Kernel
Guess.Mask.......: ?a?a?a?a?a [5]
Guess.Queue......: 1/1 (100.00%)
Speed.#1.........:    98737 H/s (13.21ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Recovered........: 0/1 (0.00%) Digests (total), 0/1 (0.00%) Digests (new)
Progress.........: 1740800/7737809375 (0.02%)
Rejected.........: 0/1740800 (0.00%)
Restore.Point....: 0/81450625 (0.00%)
Restore.Sub.#1...: Salt:0 Amplifier:20-21 Iteration:16384-32768
Candidate.Engine.: Device Generator
Candidates.#1....: earie -> esIRI
Hardware.Mon.#1..: Temp: 47c Fan: 36% Util: 25% Core:2842MHz Mem:13801MHz Bus:16

HOWEVER, I accidentally chose Template with 2 5090 and got this speed

Code:

Session..........: hashcat
Status...........: Running
Hash.Mode........: 23800 (RAR3-p (Compressed))
Hash.Target......: $RAR3$*1*2597b58e3fafb7d9*99875cb5*816*2036*1*45b30...6a3*33
Time.Started.....: Fri Apr 11 14:48:35 2025 (1 hour, 13 mins)
Time.Estimated...: Sat Apr 12 00:10:53 2025 (8 hours, 8 mins)
Kernel.Feature...: Optimized Kernel
Guess.Mask.......: ?a?a?a?a?a [5]
Guess.Queue......: 1/1 (100.00%)
Speed.#1.........:   114.7 kH/s (13.00ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#2.........:   114.7 kH/s (13.12ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#*.........:   229.4 kH/s
Recovered........: 0/1 (0.00%) Digests (total), 0/1 (0.00%) Digests (new)
Progress.........: 1008619520/7737809375 (13.03%)
Rejected.........: 0/1008619520 (0.00%)
Restore.Point....: 10357760/81450625 (12.72%)
Restore.Sub.#1...: Salt:0 Amplifier:94-95 Iteration:245760-262144
Restore.Sub.#2...: Salt:0 Amplifier:94-95 Iteration:245760-262144
Candidate.Engine.: Device Generator
Candidates.#1....:  M\YD ->  Zn_t
Candidates.#2....:  w6ru ->  N]HR
Hardware.Mon.#1..: Temp: 51c Fan: 33% Util:  0% Core:2827MHz Mem:13801MHz Bus:16
Hardware.Mon.#2..: Temp: 66c Fan: 37% Util: 86% Core:2745MHz Mem:13801MHz Bus:16

229.4 kH/s

moreover, if I choose Templates with 4 and 8 5090 then I get much lower speed, in the region of 110-130 kH/s

I tried a huge number of available Templates with 2 4 and 8 5090 and did not get such speed

The only time I was able to get close to this was when I ran Template with 8 H200 for comparison.

Code:

Status...........: Running
Hash.Mode........: 23800 (RAR3-p (Compressed))
Hash.Target......: $RAR3$*1*2597b58e3fafb7d9*99875cb5*816*2036*1*45b30...6a3*33
Time.Started.....: Fri Apr 11 16:43:39 2025 (42 secs)
Time.Estimated...: Sat Apr 12 02:08:28 2025 (9 hours, 24 mins)
Kernel.Feature...: Optimized Kernel
Guess.Mask.......: ?a?a?a?a?a [5]
Guess.Queue......: 1/1 (100.00%)
Speed.#1.........:    27169 H/s (13.85ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#2.........:    28804 H/s (13.85ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#3.........:    28817 H/s (13.84ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#4.........:    28877 H/s (13.85ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#5.........:    29718 H/s (13.84ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#6.........:    29808 H/s (13.85ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#7.........:    27353 H/s (13.85ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#8.........:    27785 H/s (13.85ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#*.........:   228.3 kH/s
Recovered........: 0/1 (0.00%) Digests (total), 0/1 (0.00%) Digests (new)
Progress.........: 9259008/7737809375 (0.12%)
Rejected.........: 0/9259008 (0.00%)
Restore.Point....: 0/81450625 (0.00%)
Restore.Sub.#1...: Salt:0 Amplifier:16-17 Iteration:245760-262144
Restore.Sub.#2...: Salt:0 Amplifier:17-18 Iteration:245760-262144
Restore.Sub.#3...: Salt:0 Amplifier:17-18 Iteration:245760-262144
Restore.Sub.#4...: Salt:0 Amplifier:17-18 Iteration:245760-262144
Restore.Sub.#5...: Salt:0 Amplifier:18-19 Iteration:245760-262144
Restore.Sub.#6...: Salt:0 Amplifier:18-19 Iteration:245760-262144
Restore.Sub.#7...: Salt:0 Amplifier:17-18 Iteration:65536-81920
Restore.Sub.#8...: Salt:0 Amplifier:17-18 Iteration:245760-262144
Candidate.Engine.: Device Generator
Candidates.#1....: garie -> gzSMA
Candidates.#2....: fxTCH -> f;|=l
Candidates.#3....: f\|=l -> fy=-t
Candidates.#4....: ft&MA -> f_]te
Candidates.#5....: nG`/1 -> nuSch
Candidates.#6....: nr!?? -> nq]za
Candidates.#7....: fN?:) -> f]+DA
Candidates.#8....: f?oDA -> fd(xa
Hardware.Mon.#1..: Temp: 40c Util:  0% Core:1980MHz Mem:3201MHz Bus:16
Hardware.Mon.#2..: Temp: 35c Util:  0% Core:1980MHz Mem:3201MHz Bus:16
Hardware.Mon.#3..: Temp: 34c Util:  0% Core:1980MHz Mem:3201MHz Bus:16
Hardware.Mon.#4..: Temp: 39c Util:  0% Core:1980MHz Mem:3201MHz Bus:16
Hardware.Mon.#5..: Temp: 41c Util:  0% Core:1980MHz Mem:3201MHz Bus:16
Hardware.Mon.#6..: Temp: 36c Util:  0% Core:1980MHz Mem:3201MHz Bus:16
Hardware.Mon.#7..: Temp: 40c Util:  0% Core:1980MHz Mem:3201MHz Bus:16
Hardware.Mon.#8..: Temp: 35c Util:  0% Core:1980MHz Mem:3201MHz Bus:16

Docker was the same everywhere, the CUDA version too. I was already racking my brains how this could be.
I tried to find significant differences in Template, the only thing that this Template is different is 64GB instead of 32GB, but I don't think it matters that much.

Considering that the docker is the same, could it be drivers? Some additional setting?
What should I pay attention to?

229.4 kH/s turned out to be the maximum speed for this slow hash, I have not seen such before.
As far as I understand now, in my case the TFLOPS value does not affect the number of GPUs?
(for example, my machine has 215.2 TFLOPS, and, conditional 12 4090 have 976.7 TFLOPS)
Is 229.4 kH/s a good speed, or is there some way to try to squeeze out more?

Thank you in advance for your help!

[Crew]

I will add template specifications

[Crew]

I apologize, a small addition, this is not spam, perhaps this will help to better understand the situation. 
Now I found a PC almost identical, even superior in parameters

Total

Result 2*5090

Code:

Session..........: hashcat
Status...........: Running
Hash.Mode........: 23800 (RAR3-p (Compressed))
Hash.Target......: $RAR3$*1*2597b58e3fafb7d9*99875cb5*816*2036*1*45b30...6a3*33
Time.Started.....: Fri Apr 11 14:48:35 2025 (7 hours, 17 mins)
Time.Estimated...: Sat Apr 12 00:10:45 2025 (2 hours, 4 mins)
Kernel.Feature...: Optimized Kernel
Guess.Mask.......: ?a?a?a?a?a [5]
Guess.Queue......: 1/1 (100.00%)
Speed.#1.........:  114.7 kH/s (13.00ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#2.........:  114.7 kH/s (13.12ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#*.........:  229.4 kH/s
Recovered........: 0/1 (0.00%) Digests (total), 0/1 (0.00%) Digests (new)
Progress.........: 6022123520/7737809375 (77.83%)
Rejected.........: 0/6022123520 (0.00%)
Restore.Point....: 63278080/81450625 (77.69%)
Restore.Sub.#1...: Salt:0 Amplifier:14-15 Iteration:49152-65536
Restore.Sub.#2...: Salt:0 Amplifier:14-15 Iteration:81920-98304
Candidate.Engine.: Device Generator
Candidates.#1....: 2wsXq -> 2N#$,
Candidates.#2....: 2MxyQ -> 2Z7aX
Hardware.Mon.#1..: Temp: 51c Fan: 33% Util: 29% Core:2857MHz Mem:13801MHz Bus:16
Hardware.Mon.#2..: Temp: 57c Fan: 37% Util:  0% Core:2910MHz Mem:13801MHz Bus:16

and the result is 4*5090

Code:

Session..........: hashcat
Status...........: Running
Hash.Mode........: 23800 (RAR3-p (Compressed))
Hash.Target......: $RAR3$*1*2597b58e3fafb7d9*99875cb5*816*2036*1*45b30...6a3*33
Time.Started.....: Fri Apr 11 22:17:44 2025 (45 secs)
Time.Estimated...: Sat Apr 12 21:14:10 2025 (22 hours, 55 mins)
Kernel.Feature...: Optimized Kernel
Guess.Mask.......: ?a?a?a?a?a [5]
Guess.Queue......: 1/1 (100.00%)
Speed.#1.........:    23423 H/s (12.99ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#2.........:    23425 H/s (12.73ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#3.........:    23424 H/s (12.71ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#4.........:    23421 H/s (12.86ms) @ Accel:1 Loops:16384 Thr:512 Vec:1
Speed.#*.........:    93693 H/s
Recovered........: 0/1 (0.00%) Digests (total), 0/1 (0.00%) Digests (new)
Progress.........: 4177920/7737809375 (0.05%)
Rejected.........: 0/4177920 (0.00%)
Restore.Point....: 0/81450625 (0.00%)
Restore.Sub.#1...: Salt:0 Amplifier:12-13 Iteration:98304-114688
Restore.Sub.#2...: Salt:0 Amplifier:12-13 Iteration:98304-114688
Restore.Sub.#3...: Salt:0 Amplifier:12-13 Iteration:98304-114688
Restore.Sub.#4...: Salt:0 Amplifier:12-13 Iteration:98304-114688
Candidate.Engine.: Device Generator
Candidates.#1....: rarie -> rsIRI
Candidates.#2....: r7,ke -> rx0st
Candidates.#3....: rRPer -> r*"++
Candidates.#4....: r_?ON -> r=$**
Hardware.Mon.#1..: Temp: 36c Fan: 30% Util:  0% Core:2400MHz Mem:13801MHz Bus:16
Hardware.Mon.#2..: Temp: 36c Fan: 30% Util:  0% Core:2647MHz Mem:13801MHz Bus:16
Hardware.Mon.#3..: Temp: 25c Fan: 30% Util:  0% Core:2647MHz Mem:13801MHz Bus:16
Hardware.Mon.#4..: Temp: 43c Fan: 30% Util:  0% Core:2475MHz Mem:13801MHz Bus:16

docker and CUDA version are the same

How can this be? 
Or rather, I'm even more interested in what exactly is special about 2*5090 that allows it to produce such a difference in speed?

The attached screenshot shows the characteristics of two PCs.