On-the-fly loading of gz wordlists

On-the-fly loading of gz wordlists - Printable Version

+- hashcat Forum (https://hashcat.net/forum)
+-- Forum: Support (https://hashcat.net/forum/forum-3.html)
+--- Forum: hashcat (https://hashcat.net/forum/forum-45.html)
+--- Thread: On-the-fly loading of gz wordlists (/thread-9248.html)

On-the-fly loading of gz wordlists - Mem5 - 05-23-2020

Quote:* changes v5.1.0 -> v6.0.0
(...)
- Support on-the-fly loading of compressed wordlists in zip and gzip format

Seen in hashcat beta 1803.

Just tried with a gz file:

Code:
$ wc -l  dic*

314'265 dic.gz

19'487'556 dic

Hashcat sees:

Code:
Dictionary cache built:

* Filename..: dic.gz

* Passwords.: 314266

* Keyspace..: 205218

Progress.........: 205218/205218 (100.00%)

Edit:

Code:
$ file dic.gz

dic.gz: gzip compressed data, last modified: Sun Apr 12 05:01:34 2020, from Unix

Looks like the file has not been unzipped?

Thanks.

RE: On-the-fly loading of gz wordlists - royce - 05-23-2020

WFM:

Code:
$ hashcat --version

v5.1.0-1789-gc7da6357

$ file rockyou.txt.gz

rockyou.txt.gz: gzip compressed data, was "rockyou.txt", last modified: Thu Jul  7 14:27:39 2016, from Unix

$ wc -l rockyou.txt.gz

213432 rockyou.txt.gz

$ gunzip -cd rockyou.txt.gz | wc -l

14344391

$ cat brandon.hash

fc275ac3498d6ab0f0b4389f8e94422c

$ hashcat -m 0 -w 4 -O -D 2 brandon.hash rockyou.txt.gz --potfile-disable

[...]

Dictionary cache built:

* Filename..: rockyou.txt.gz

* Passwords.: 14344391

* Bytes.....: 139921497

* Keyspace..: 14344384

* Runtime...: 3 secs

[...]

fc275ac3498d6ab0f0b4389f8e94422c:brandon

RE: On-the-fly loading of gz wordlists - Mem5 - 05-23-2020

Weird.. I may have failed somewhere in the compress step..

RE: On-the-fly loading of gz wordlists - Mem5 - 05-24-2020

From a performance perspective I cannot see any loss of speed; how does hashcat handle gigas of compressed wordlist without spending minutes/hours on decompression?

RE: On-the-fly loading of gz wordlists - royce - 05-24-2020

I'm not sure, but I think that decompressed data can be used "on the fly" - as soon as it's decompressed, without having to wait for the entire file to decompress.

RE: On-the-fly loading of gz wordlists - Mem5 - 05-25-2020

Ok.. but what about "Dictionary cache building", hashcat have to decompress all the file first to know the statistics, number of passwords etc?

RE: On-the-fly loading of gz wordlists - royce - 05-25-2020

Yes, that seems right. It would have to decompress the whole thing first, enough to analyze the statistics and then cache them. so I assume there would be some duplicated work, just like dictionary cache building without compression.