Wordlist optimizing program?
#1
Is there any program out there that will merge big files and remove duplicates and optimize the wordlist?

I have google the web and find nothing.

I have tried to merge files and optimizing them but all my word programs hang or crash so no point in doing it.

MD5
#2
sort(1)
#3
Take a look at ULM: http://unifiedlm.com/
#4
For big wordlists, use rli2.bin from hashcat-utils.
#5
My assumption is that someone asking the initial question here comes from the Windows part of the universe, like me, where the *nix equivalents don't work on big files, or the characters in rules or password lists.

ULM does some of what the hashcat-utils do, but with more hand-holding, and also more reliably on a Windows system.

If the files are too big for ULM, then blandyuk's HUGE word-lists duplicate remover and merge tool at http://forum.hashkiller.co.uk/topic-view...1842#41842 might be useful.
I haven't tried it, as it wants a version of .NET I don't want to install, and according to Blazer, ULM's dev, may do things differently, e.g. in the handling of NULs.

Windows users need to do some acquiring of other tools, and test them to see if they do what is wanted.

Also, the 64-bit programs, like ULM's Sort64.exe, are much faster than the ordinary 32-bit programs not written for speed or big file handling.

(Speaking of hashcat-utilties, they'd be more useful if the newer tools were documented in the wiki.)
#6
feel free to document them
#7
If I could, I would.
If each programmer/developer could give a one paragraph explanation, with an example, it would be a good start.