Extracting the passwords from a multiple file wordlist (sed & grep).

[epixoip]

I notice that you are attempting to chop the beginning and end from words to try to remove the "password padding". [...] You could keep chopping bits off until you find a word match I suppose. This way applebanana would eventually be found by either apple or banana

I'm happy you picked up on that, because that's what I was hinting at Yes, you could loop through each plain continually chopping it from different positions until we find a match. The important thing to realize is you probably don't need a full match -- a partial match should suffice for detecting if the word is random. This will also help detect words that probably aren't random, but also aren't exactly in your wordlist.

You could write a plethora of different rules for trying to match against your wordlist. Hopefully this got you on the right track though.

At first when you posted I didn't see how I would save any new words that were not in the original dictionary, but now I do since you posted the demo.

Yeah, you can just redirect stdout from this into a new file to get all of the apparently non-random words. I sent all of the 'looks random' messages to stderr, so if you just did >newords you'd still see the 'looks random' messages on stderr without affecting the contents of the file.