best duplicate removal tool for wordlists
#1
Can anyone recommend one for a windows user?

ULM chokes on large lists.  I've also been using the tool found here, but it seems to get stuck in the same place on lists, I'm not sure if it is certain characters or what is causing it.

Thank you.
#2
http://gnuwin32.sourceforge.net/packages/coreutils.htm
in particular 'sort' program
#3
Have you tried the command line ULM tools?
#4
I have not. I assumed it used the same engine as the GUI. Can it handle large files?
#5
The command line tools are 64-bit executables, and more powerful than the GUI ULM.
#6
I had troubles using the ULM command line tools.
I am using sort -u on 20 gb files without issues.