How to Merge 2 Big Wordlists in Windows

How to Merge 2 Big Wordlists in Windows - Printable Version

+- hashcat Forum (https://hashcat.net/forum)
+-- Forum: Support (https://hashcat.net/forum/forum-3.html)
+--- Forum: hashcat-utils, maskprocessor, statsprocessor, md5stress, wikistrip (https://hashcat.net/forum/forum-28.html)
+--- Thread: How to Merge 2 Big Wordlists in Windows (/thread-11458.html)

How to Merge 2 Big Wordlists in Windows - marcusx - 06-15-2023

My intention is to combine 2 large wordlists, remove all duplicate entries, remove everything under 7 characters and over 40 characters and sort the list randomly, at best remove all entries that contain only letters and are not combined.

I work with Windows and have tried PowerShell and The-Distribution-Which-Does-Not-Handle-OpenCL-Well (Kali) Linux.

Via Powershell I have managed the following so far (but works only with small txt files)

Code:
Select-String -Path "D:\test\*.txt" -Pattern "^.{7,40}$" | Select -Unique -Expand Line | Sort-Object -Unique | Set-Content "D:\output.txt"

I have also tried with hashcat-utils via The-Distribution-Which-Does-Not-Handle-OpenCL-Well (Kali) linux via the command:

Code:
./mli2.bin 1.txt 2.txt > out.txt

or via powershell with:

Code:
./mli2.exe 1.txt 2.txt > out.txt

it also creates an out.txt but after 3 hours nothing happens, PowerShell aborts at some point and says that the file is already accessed.

can you help me?

RE: How to Merge 2 Big Wordlists in Windows - lapsikmees - 06-15-2023

Use command line (cmd). "type wordlist1.txt wordlist2.txt | sort /u /o combinedlist.txt"

Sort command converts from utf8 to ansi tho.

You can also do "type wordlist1.txt wordlist2.txt >> wordlist3.txt" and later sort any program you want. I use notepad++ usually.

https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/type
https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/sort