a DIY wordlist generator

[bored_dude]

Hi,

Introduction

A little contributation to whom may be interested to make their own wordlist,  with continuous update, lightweight and simple as hell

The idea behind it was to find a (almost) limitless source of english words with constant update. As the english language continually evolve, having a "tool" to generate new words make sense.

The generator will use as source wikipedia, in particular the random article : https://en.wikipedia.org/wiki/Special:Random

This page on trigger, will redirect to a random article.

This could work on Windows, though the following instruction are for Linux (but definitively adaptable for Windows)

Getting Ready and Started

Create a new directory and go into

Code:

mkdir scrap && cd scrap

Create the wordlist file

Code:

touch dic.txt

Create the scrapping script (I use nano but any editor will work)

Code:

nano wikipedia_scrap.py

Insert inside

Code:

#!/usr/bin/env python3 import pycurl from io import BytesIO from bs4 import BeautifulSoup buffer = BytesIO() c = pycurl.Curl() c.setopt(c.URL, "https://en.wikipedia.org/wiki/Special:Random") c.setopt(c.FOLLOWLOCATION, True) c.setopt(c.WRITEDATA, buffer) c.perform() c.close() html = buffer.getvalue().decode("utf-8") # GET HTML soup = BeautifulSoup(html, "html.parser") # GET URL from <link rel="canonical" rurl = soup.find('link', {'rel' : 'canonical'}).get("href") # Print the current (redirected url) where the scrap will happen print(rurl) # GET all <p> from soup var ptext = soup.find_all('p') # Extract text from <p> for p in ptext:     arr = p.text.split()     for words in arr:         # Need more work, remove punctuation         words = words.replace(',','').replace('.','').replace('(','').replace(')','').lower()         if words.isalpha():             # Only words that have at least 5 chars             if len(words) > 4:                 f = open("dic.txt","a")                 f.write(words + '\n')                 f.close()

Make the script executable

Code:

chmod +x wikipedia_scrap.py

Create the bash script that will act as a process and execute the python script every X seconds (timer can be change here)

Code:

nano exec.sh

Insert inside

Code:

#!/bin/bash # On exit script (ctrl+c or kill), sort alphabetically and clean any double (or triple...) entries trap "sort -u dic.txt > temp && mv temp dic.txt && exit" SIGINT while : do   ./wikipedia_scrap.py   # Set a pause between wikipedia request, can be change for lower value (unsure how many request per minute wikipedia will allow)   sleep 10 done

Make the bash script executable

Code:

chmod +x exec.sh

Now run the bash script

Code:

./exec.sh

Cleaning

Unfortunatelly, determine if a word is in english language is tricky, most of the unwanted foreign words will be easily cleaned as they will be sorted after the last english word.

For example on my current generated dic

Code:

... zygogaster zygomatic zygomorphic zygomycetes zygomycota zygopetalinae zygopetalon zygopetalum zygotaria zygote zymalkowski zynetix zysman zytek zytronic zyuganov zzzero ...

Everything after zzzero

Code:

zábřeh záhady záhony zákupy záleský zámok zánka zápolya ... électorale électrique éliphas élisabeth élite éloize élèves éléonore émigré émigrés émile émilie énergies épinac épiscopale époque épreuves ... διοικητής διοικηταὶ δρουγουβιτεία εβίνα εθνική εκλογική ελλάδα εσφούγγιζε ευθύμης εὐλόγιος εὔρωψ θέλεις θαυμαζω θεσσαλιῶτις θεῖος θρασὺς ...

Starting from there, other non english words can be found sorted between english word, this is the part I think can be improved.

Improving the script

The script can definitively be improved, I'm thinking adding a regex to exclude characters found in words like "divisão" or "phước", or using python library that can do a better job than the function .is_alpha()

Another way could be to change the source, instead of wikipedia, using the API of the New york time to scrap word inside article

I welcome any idea, suggestion or "contribution" to make this little project better. Just keep in mind I like to keep thing as simple as they can be

Here a generated wordlist made with this "tool", with a dirty fast cleaning, over 137770 words generated in ~40 hours of run

https://0x0.st/X6n3.txt

Thank you !

[RealEnder]

Wow, please don't do it this way. No need to hammer Wikipedia's site - it's slow and non-productive.
Wikipedia has dumps here: https://dumps.wikimedia.org . Just parse them and extract the words. The nice thing here is you can also pull article versions and catch mistypings and other interesting stuff.
Back in 2011 I wrote a similar script, which parses the dump on the fly and puts the wordlist in sqlite DB. It's python2 and written for targeted case, but can be easily changed to work. Check it out here: https://sec.stanev.org/?download
If there is an interest, I can shape it a bit.

[DanielG]

Agreed, this is also created at https://github.com/NorthwaveSecurity/wikiraider

"WikiRaider enables you to generate wordlists based on country specific databases of Wikipedia. This will provide you with not only a list of words in a specific language, it will also provide you with e.g. country specific artists, TV shows, places, etc."

[bored_dude]

Thanks guys, very useful infos !

I will definitively remake the script to use https://dumps.wikimedia.org instead and take at the code of wikiraider