+- hashcat Forum (https://hashcat.net/forum)
+-- Forum: Deprecated; Previous versions (https://hashcat.net/forum/forum-29.html)
+--- Forum: General Help (https://hashcat.net/forum/forum-8.html)
+--- Thread: Wordlist optimizing program? (/thread-3546.html)
Wordlist optimizing program? - Heinz - 07-16-2014
Is there any program out there that will merge big files and remove duplicates and optimize the wordlist?
I have google the web and find nothing.
I have tried to merge files and optimizing them but all my word programs hang or crash so no point in doing it.
MD5
RE: Wordlist optimizing program? - undeath - 07-16-2014
sort(1)
RE: Wordlist optimizing program? - Kgx Pnqvhm - 07-16-2014
Take a look at ULM: http://unifiedlm.com/
RE: Wordlist optimizing program? - atom - 07-23-2014
For big wordlists, use rli2.bin from hashcat-utils.
RE: Wordlist optimizing program? - Kgx Pnqvhm - 07-23-2014
My assumption is that someone asking the initial question here comes from the Windows part of the universe, like me, where the *nix equivalents don't work on big files, or the characters in rules or password lists.
ULM does some of what the hashcat-utils do, but with more hand-holding, and also more reliably on a Windows system.
If the files are too big for ULM, then blandyuk's HUGE word-lists duplicate remover and merge tool at http://forum.hashkiller.co.uk/topic-view.aspx?t=5512&m=41842#41842 might be useful.
I haven't tried it, as it wants a version of .NET I don't want to install, and according to Blazer, ULM's dev, may do things differently, e.g. in the handling of NULs.
Windows users need to do some acquiring of other tools, and test them to see if they do what is wanted.
Also, the 64-bit programs, like ULM's Sort64.exe, are much faster than the ordinary 32-bit programs not written for speed or big file handling.
(Speaking of hashcat-utilties, they'd be more useful if the newer tools were documented in the wiki.)
RE: Wordlist optimizing program? - atom - 07-25-2014
feel free to document them
RE: Wordlist optimizing program? - Kgx Pnqvhm - 07-25-2014
If I could, I would.
If each programmer/developer could give a one paragraph explanation, with an example, it would be a good start.