On-the-fly loading of gz wordlists

[Mem5]

* changes v5.1.0 -> v6.0.0
(...)
- Support on-the-fly loading of compressed wordlists in zip and gzip format

Seen in hashcat beta 1803.

Just tried with a gz file:

Code:

$ wc -l  dic* 314'265 dic.gz 19'487'556 dic

Hashcat sees:

Code:

Dictionary cache built: * Filename..: dic.gz * Passwords.: 314266 * Keyspace..: 205218 Progress.........: 205218/205218 (100.00%)

Edit:

Code:

$ file dic.gz dic.gz: gzip compressed data, last modified: Sun Apr 12 05:01:34 2020, from Unix

Looks like the file has not been unzipped?

Thanks.

[royce]

WFM:

Code:

$ hashcat --version v5.1.0-1789-gc7da6357 $ file rockyou.txt.gz rockyou.txt.gz: gzip compressed data, was "rockyou.txt", last modified: Thu Jul 7 14:27:39 2016, from Unix $ wc -l rockyou.txt.gz 213432 rockyou.txt.gz $ gunzip -cd rockyou.txt.gz | wc -l 14344391 $ cat brandon.hash fc275ac3498d6ab0f0b4389f8e94422c $ hashcat -m 0 -w 4 -O -D 2 brandon.hash rockyou.txt.gz --potfile-disable [...] Dictionary cache built: * Filename..: rockyou.txt.gz * Passwords.: 14344391 * Bytes.....: 139921497 * Keyspace..: 14344384 * Runtime...: 3 secs [...] fc275ac3498d6ab0f0b4389f8e94422c:brandon

[Mem5]

Weird.. I may have failed somewhere in the compress step..

[Mem5]

From a performance perspective I cannot see any loss of speed; how does hashcat handle gigas of compressed wordlist without spending minutes/hours on decompression?

[royce]

I'm not sure, but I think that decompressed data can be used "on the fly" - as soon as it's decompressed, without having to wait for the entire file to decompress.

[Mem5]

Ok.. but what about "Dictionary cache building", hashcat have to decompress all the file first to know the statistics, number of passwords etc?

[royce]

Yes, that seems right. It would have to decompress the whole thing first, enough to analyze the statistics and then cache them. so I assume there would be some duplicated work, just like dictionary cache building without compression.