+- hashcat Forum (https://hashcat.net/forum)
+-- Forum: Deprecated; Previous versions (https://hashcat.net/forum/forum-29.html)
+--- Forum: General Help (https://hashcat.net/forum/forum-8.html)
+--- Thread: best duplicate removal tool for wordlists (/thread-4833.html)
best duplicate removal tool for wordlists - turbogiant76 - 11-20-2015
Can anyone recommend one for a windows user?
ULM chokes on large lists. I've also been using the tool found here, but it seems to get stuck in the same place on lists, I'm not sure if it is certain characters or what is causing it.
Thank you.
RE: best duplicate removal tool for wordlists - Mem5 - 11-20-2015
http://gnuwin32.sourceforge.net/packages/coreutils.htm
in particular 'sort' program
RE: best duplicate removal tool for wordlists - Kgx Pnqvhm - 11-21-2015
Have you tried the command line ULM tools?
RE: best duplicate removal tool for wordlists - turbogiant76 - 11-21-2015
I have not. I assumed it used the same engine as the GUI. Can it handle large files?
RE: best duplicate removal tool for wordlists - Kgx Pnqvhm - 11-21-2015
The command line tools are 64-bit executables, and more powerful than the GUI ULM.
RE: best duplicate removal tool for wordlists - Mem5 - 11-22-2015
I had troubles using the ULM command line tools.
I am using sort -u on 20 gb files without issues.