05-19-2013, 10:38 PM
You need to write a program that will hash using SSE2/SSSE3/AVX to make it fast. You should write truncated binary hashes to the file. Since I guess you want to make it useful, you should make a lossy hash table. Indexing around log2(number of passwords) bits of the hash and save a binary value that represents a range of passwords... Unless you are just using an unmodified dictionary then you'll need to just save the full password. Well I guess you could split the dictionary into blocks and compress each of them. Then the "binary value that represents a range of passwords" would be which compressed dictionary part to look at.
Anyway it's not like I've done similar things... http://www.tobtu.com/md5.php. It took about 3 days to generate a database with 50 billion passwords and this was kinda slow. So far this is the best I've come up with.
You can look at this: http://en.wikipedia.org/wiki/Wikipedia_t...ano_coding I really need to go find those papers and read them. So I can add sources and get the article accepted.
I need:
"On binary representations of monotone sequences" by Peter Elias in 1972
I have (but I haven't fully read it yet but the intro was so long and boring that I figured out that in the case of lossy hash tables Huffman encoding of the number of passwords per bin (number of bins is on the order of number of passwords) is more efficient but this leads to an unknown size instead of a size that is known before you start with Elias-Fano):
"Efficient Storage and Retrieval by Content and Address of Static Files" by Peter Elias in 1974
I found (but this doesn't describe anything about the Elias-Fano coding so you "don't" need to read it):
Robert M. Fano. On the number of bits required to implement an associative memory. Memorandum 61, Computer Structures Group, Project MAC, MIT, Cambridge, Mass., n.d., 1971
Anyway it's not like I've done similar things... http://www.tobtu.com/md5.php. It took about 3 days to generate a database with 50 billion passwords and this was kinda slow. So far this is the best I've come up with.
You can look at this: http://en.wikipedia.org/wiki/Wikipedia_t...ano_coding I really need to go find those papers and read them. So I can add sources and get the article accepted.
I need:
"On binary representations of monotone sequences" by Peter Elias in 1972
I have (but I haven't fully read it yet but the intro was so long and boring that I figured out that in the case of lossy hash tables Huffman encoding of the number of passwords per bin (number of bins is on the order of number of passwords) is more efficient but this leads to an unknown size instead of a size that is known before you start with Elias-Fano):
"Efficient Storage and Retrieval by Content and Address of Static Files" by Peter Elias in 1974
I found (but this doesn't describe anything about the Elias-Fano coding so you "don't" need to read it):
Robert M. Fano. On the number of bits required to implement an associative memory. Memorandum 61, Computer Structures Group, Project MAC, MIT, Cambridge, Mass., n.d., 1971