05-20-2013, 05:46 PM
A couple of other minor points, and a question: Why are you storing this in a database?
The minor points; when dealing with large amounts of data, it's important to remember that disk I/O isn't free :-)
On my system, the stock MD5() function in libcrypto runs at 5 million hashes per second, on a single thread. SHA1 is a bit slower, at 2.5 million hashes per second. Both assumed a typical 8-charcter password (longer passwords take longer, of course).
That same system can read a file at about 11 million words per second (again, assuming 8 character passwords, plus a linefeed). It can write a little bit slower.
So, if you just want to create a list of
word:MD5:SHA1
you will need to read ~9 characters, and write ~9+33+41 for each line.
That means you will be able to write about 1 million lines per second, before you run out of disk bandwidth.
In other words, the stock MD5 and SHA1 functions are plenty fast enough to run your disk to saturation, if you have standard hard drives. If you have an SSD, you might need to use one or two threads to bring your speed up.
Using a GPU won't help (at all) in this application.
The minor points; when dealing with large amounts of data, it's important to remember that disk I/O isn't free :-)
On my system, the stock MD5() function in libcrypto runs at 5 million hashes per second, on a single thread. SHA1 is a bit slower, at 2.5 million hashes per second. Both assumed a typical 8-charcter password (longer passwords take longer, of course).
That same system can read a file at about 11 million words per second (again, assuming 8 character passwords, plus a linefeed). It can write a little bit slower.
So, if you just want to create a list of
word:MD5:SHA1
you will need to read ~9 characters, and write ~9+33+41 for each line.
That means you will be able to write about 1 million lines per second, before you run out of disk bandwidth.
In other words, the stock MD5 and SHA1 functions are plenty fast enough to run your disk to saturation, if you have standard hard drives. If you have an SSD, you might need to use one or two threads to bring your speed up.
Using a GPU won't help (at all) in this application.