hashcat Forum › Deprecated; Previous versions › General Help v
best duplicate removal tool for wordlists

Threaded Mode
best duplicate removal tool for wordlists
turbogiant76 Offline
Junior Member
**
Posts: 37
Threads: 21
Joined: Jul 2015
#1
11-20-2015, 10:51 PM
Can anyone recommend one for a windows user?

ULM chokes on large lists.  I've also been using the tool found here, but it seems to get stuck in the same place on lists, I'm not sure if it is certain characters or what is causing it.

Thank you.
Find
Mem5 Offline
Posting Freak
*****
Posts: 803
Threads: 135
Joined: Feb 2011
#2
11-20-2015, 11:19 PM
http://gnuwin32.sourceforge.net/packages/coreutils.htm
in particular 'sort' program
Find
Kgx Pnqvhm Offline
Member
***
Posts: 247
Threads: 59
Joined: Mar 2011
#3
11-21-2015, 03:52 AM
Have you tried the command line ULM tools?
Find
turbogiant76 Offline
Junior Member
**
Posts: 37
Threads: 21
Joined: Jul 2015
#4
11-21-2015, 03:57 AM
I have not. I assumed it used the same engine as the GUI. Can it handle large files?
Find
Kgx Pnqvhm Offline
Member
***
Posts: 247
Threads: 59
Joined: Mar 2011
#5
11-21-2015, 03:14 PM
The command line tools are 64-bit executables, and more powerful than the GUI ULM.
Find
Mem5 Offline
Posting Freak
*****
Posts: 803
Threads: 135
Joined: Feb 2011
#6
11-22-2015, 01:37 AM
I had troubles using the ULM command line tools.
I am using sort -u on 20 gb files without issues.
Find