hashcat Forum
Hashcat 6, NVidia GTX 2000 series, and CUDA toolkit - Printable Version

+- hashcat Forum (https://hashcat.net/forum)
+-- Forum: Support (https://hashcat.net/forum/forum-3.html)
+--- Forum: hashcat (https://hashcat.net/forum/forum-45.html)
+--- Thread: Hashcat 6, NVidia GTX 2000 series, and CUDA toolkit (/thread-9375.html)

Hashcat 6, NVidia GTX 2000 series, and CUDA toolkit - jen42 - 07-10-2020

I'm having trouble with hashcat 6 working with any of the NVidia 2000 series GPUs I've tried.  I've tried on a laptop RTX 2060 (Linux Mint) and a hashing rigs with a mix of GTX 1080 Founders and GTX 2080 Supers (Ubuntu 18.04) with the exact same results on the 2080 GPUs only.  The 1080's work just fine...

I have easier access to the laptop so I will use that for the examples below --

First - without nvidia-cuda-toolkit installed:

Code:
$ sudo hashcat-6.0.0/hashcat.bin -m 1000 -b
hashcat (v6.0.0) starting in benchmark mode...

Benchmarking uses hand-optimized kernel code by default.
You can use it in your cracking session by setting the -O option.
Note: Using optimized kernel code limits the maximum supported password length.
To disable the optimized kernel code in benchmark mode, use the -w option.

* Device #1: CUDA SDK Toolkit installation NOT detected.
            CUDA SDK Toolkit installation required for proper device support and utilization
            Falling back to OpenCL Runtime

* Device #1: WARNING! Kernel exec timeout is not disabled.
            This may cause "CL_OUT_OF_RESOURCES" or related errors.
            To disable the timeout, see: https://hashcat.net/q/timeoutpatch
nvmlDeviceGetFanSpeed(): Not Supported

OpenCL API (OpenCL 1.2 CUDA 10.2.185) - Platform #1 [NVIDIA Corporation]
========================================================================
* Device #1: GeForce RTX 2060, 5632/5934 MB (1483 MB allocatable), 30MCU

Benchmark relevant options:
===========================
* --optimized-kernel-enable

Hashmode: 1000 - NTLM

Speed.#1.........: 37744.5 MH/s (53.21ms) @ Accel:64 Loops:1024 Thr:1024 Vec:8

Started: Thu Jul  9 17:51:19 2020
Stopped: Thu Jul  9 17:51:26 2020


And the nvidia packages that are installed when it works:

Code:
$ sudo apt list --installed | grep -i nvidia

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libnvidia-cfg1-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
libnvidia-common-435/bionic,bionic,bionic-updates,bionic-updates,now 435.21-0ubuntu0.18.04.2 all [installed]
libnvidia-common-440/bionic,bionic,bionic-updates,bionic-updates,bionic-security,bionic-security,now 440.100-0ubuntu0.18.04.1 all [installed,automatic]
libnvidia-compute-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
libnvidia-decode-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
libnvidia-encode-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
libnvidia-extra-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
libnvidia-fbc1-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
libnvidia-gl-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
libnvidia-ifr1-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
nvidia-compute-utils-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
nvidia-dkms-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
nvidia-driver-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed]
nvidia-kernel-common-440/bionic,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
nvidia-kernel-source-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
nvidia-prime/bionic-updates,bionic-updates,now 0.8.8.2 all [installed]
nvidia-settings/bionic,now 440.64-0ubuntu0~0.18.04.1 amd64 [installed]
nvidia-utils-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
xserver-xorg-video-nvidia-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]


As you can see, it works, but complains about falling back to OpenCL.  In addition, hashcat 6 pegs the CPU at 100% when running in OpenCL fallback.  This is not ideal.  hashcat 5 can run for weeks (probably forever) without using more than 20% of the CPU, so we are currently reverted back to hashcat 5 because of this problem.

With Cuda Toolkit installed:

Code:
$ sudo hashcat-6.0.0/hashcat.bin -m 1000 -b
hashcat (v6.0.0) starting in benchmark mode...

Benchmarking uses hand-optimized kernel code by default.
You can use it in your cracking session by setting the -O option.
Note: Using optimized kernel code limits the maximum supported password length.
To disable the optimized kernel code in benchmark mode, use the -w option.

* Device #1: WARNING! Kernel exec timeout is not disabled.
            This may cause "CL_OUT_OF_RESOURCES" or related errors.
            To disable the timeout, see: https://hashcat.net/q/timeoutpatch
* Device #2: WARNING! Kernel exec timeout is not disabled.
            This may cause "CL_OUT_OF_RESOURCES" or related errors.
            To disable the timeout, see: https://hashcat.net/q/timeoutpatch
nvmlDeviceGetFanSpeed(): Not Supported

CUDA API (CUDA 10.2)
====================
* Device #1: GeForce RTX 2060, 5644/5934 MB, 30MCU

OpenCL API (OpenCL 1.2 CUDA 10.2.185) - Platform #1 [NVIDIA Corporation]
========================================================================
* Device #2: GeForce RTX 2060, skipped

Benchmark relevant options:
===========================
* --optimized-kernel-enable

Hashmode: 1000 - NTLM

nvrtcCompileProgram(): NVRTC_ERROR_INVALID_OPTION

nvrtc: error: invalid value for --gpu-architecture (-arch)

* Device #1: Kernel /home/jen42/hashcat-6.0.0/OpenCL/shared.cl build failed.

* Device #1: Kernel /home/jen42/hashcat-6.0.0/OpenCL/shared.cl build failed.

Started: Thu Jul  9 18:00:43 2020
Stopped: Thu Jul  9 18:00:44 2020


Here is the package list for all nvidia when the cuda toolkit is installed:

Code:
$ sudo apt list --installed | grep -i nvidia

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libnvidia-cfg1-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
libnvidia-common-435/bionic,bionic,bionic-updates,bionic-updates,now 435.21-0ubuntu0.18.04.2 all [installed]
libnvidia-common-440/bionic,bionic,bionic-updates,bionic-updates,bionic-security,bionic-security,now 440.100-0ubuntu0.18.04.1 all [installed,automatic]
libnvidia-compute-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
libnvidia-decode-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
libnvidia-encode-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
libnvidia-extra-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
libnvidia-fbc1-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
libnvidia-gl-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
libnvidia-ifr1-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
nvidia-compute-utils-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
nvidia-cuda-dev/bionic,now 9.1.85-3ubuntu1 amd64 [installed,automatic]
nvidia-cuda-toolkit/bionic,now 9.1.85-3ubuntu1 amd64 [installed]
nvidia-dkms-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
nvidia-driver-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed]
nvidia-kernel-common-440/bionic,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
nvidia-kernel-source-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
nvidia-opencl-dev/bionic,now 9.1.85-3ubuntu1 amd64 [installed,automatic]
nvidia-prime/bionic-updates,bionic-updates,now 0.8.8.2 all [installed]
nvidia-profiler/bionic,now 9.1.85-3ubuntu1 amd64 [installed,automatic]
nvidia-settings/bionic,now 440.64-0ubuntu0~0.18.04.1 amd64 [installed]
nvidia-utils-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]
xserver-xorg-video-nvidia-440/bionic,bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed,automatic]


As you can see, the GTX 2080's just don't work with nvidia-cuda-toolkit and dependencies installed.  I've tried using the Ubuntu default repositories and installing the NVidia drivers with apt, and also downloading the drivers directly from NVidia and installing them that way.  No change.

I have used the precompiled hashcat binaries and built from source. I've also tried the same with the latest beta of hashcat.  No change.

Additionally, hashcat 6.0 with only the 1080 GPU AND cuda toolkit still pegs the CPU at 100%, so we still can't use hashcat 6 with just those GPU without CPU issues.

Any ideas why this doesn't work?

RE: Hashcat 6, NVidia GTX 2000 series, and CUDA toolkit - Chick3nman - 07-10-2020

Your driver versions and component versions, as shown in your package lists, are not consistent with the versions provided with the current CUDA Toolkit installation.

Grab this file https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=runfilelocal

It's the runfile for installing on Ubuntu 18.04. You should see versions consistent with the following list after installation: https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cuda-major-component-versions

Current Driver version for linux is >= 450.XX

Now, that said, I'm not certain a driver upgrade will purely solve the issue. It sounds more like you've got some other environmental issue at play here. Especially as I can not reproduce with my linux test bench, It's also mixed generation including 20 series cards, running ubuntu 18.04 with latest tookit, hashcat 5.1.0 and 6.0.0 both running perfectly.

RE: Hashcat 6, NVidia GTX 2000 series, and CUDA toolkit - jen42 - 07-10-2020

Thank you!  The 450 driver fixed the hashcat 6.0 cuda errors. Smile

I'm still seeing CPU usage at 100% with hashcat 6 but not 5 for the same tests though:

Hashcat 5.1: is using 6 - 10% CPU:

Code:
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM    TIME+ COMMAND                                       

2448 root      20  0 9209036 1.580g 130340 S  6.6  5.1  0:12.09 hashcat64.bin             


Hashcat 6.0 is using 96-100:

Code:
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM    TIME+ COMMAND                                       
2557 root      20  0 11.330g 990188 105096 S  96.0  3.0  0:11.82 hashcat.bin           


How does your CPU look with hashcat 6 vs 5?

RE: Hashcat 6, NVidia GTX 2000 series, and CUDA toolkit - Chick3nman - 07-10-2020

6.0.0 is definitely using a bit more CPU for me, though not nearly as significant as you are seeing. How many cards are in that system? This may be an issue of scale, where more cards uses more CPU like we had with the Nvidia core spinning issue. The suggestion still, due mostly to that issue, is to have at minimum 1 fast core per GPU in the system, with 1-2 cores left over for the rest of the system.