[solved] Problem sorting dictionary file

[solved] Problem sorting dictionary file - Printable Version

+- hashcat Forum (https://hashcat.net/forum)
+-- Forum: Support (https://hashcat.net/forum/forum-3.html)
+--- Forum: hashcat-utils, maskprocessor, statsprocessor, md5stress, wikistrip (https://hashcat.net/forum/forum-28.html)
+--- Thread: [solved] Problem sorting dictionary file (/thread-3312.html)

[solved] Problem sorting dictionary file - tibit - 04-20-2014

I'm trying to sort a huge (2gb) dictionary file.

I tried the hashcat tools, but had zero result.
With gnuwin utils, I tried sort.exe, gsort.exe, cat.exe

Code:
cat 20_04found.dic | sort | uniq > 20_04founduniq.dic

sort 20_04found.txt | uniq > 20_04foundu.txt

gsort 20_04found.txt | uniq -u >20_04foundu.txt

They all stopped after 163mb from a 2GB file.
I changed the file encoding to utf8, to ascii,and windows ...
but it didn't help.

How can I prevent the sorting process to stop at certain characters?
Or is there a good alternative working on windows 64bit

Thank you for replying

RE: Problem sorting dictionary file - coolbry95 - 04-20-2014

it sounds like you are hitting and EOF character but i could be wrong.

RE: Problem sorting dictionary file - epixoip - 04-20-2014

how much free disk space do you have?

RE: Problem sorting dictionary file - tibit - 04-20-2014

First of all, thank you for your help!

Code:
it sounds like you are hitting and EOF character

If this is the reason, how can I remove these 'End Of File' characters or keyboard characters when they are not read? How can I recognise them?

Code:
how much free disk space do you have?

I have 16Gb Ram, but a primary drive (C) of only 128Gb (SD) with 22Gb free space and a second drive of 1Tb split in two (200=D with 130Gb free, 800=E with 250Gb free space)

RE: Problem sorting dictionary file - Kgx Pnqvhm - 04-20-2014

Somebody usually suggests ULM at this point.
http://unifiedlm.com/

RE: Problem sorting dictionary file - tibit - 04-21-2014

Thanks for solving my problem "Kgx Pnqvhm".

ULM could handle my 2gb file.
I downloaded the latest version of sort64.exe (15.2.2014) from the CLi-package on unifiedlm, which handled the task ....
It also managed to remove the duplicates

Code:
Sort64.exe -i file_in -u -o file_out

Finaly a nice cleaned dic.

Thank you

RE: [solved] Problem sorting dictionary file - Milzo - 04-21-2014

(04-21-2014, 02:42 AM)tibit Wrote: Finaly a nice cleaned dic.

Thank you

you might want to re word that lol