BOM markers and hidden characters in front of password

[monyanus]

Dear all,

I am struggling a few days now with trying to test a list of 3137616 passwords which I want to test in combination with prepend hidden characters, most likely BOM (Bite Order Mark) bits for UTF8 or UTF16.

I tried quite a few things which all did not work out.

1) I converted my password list to hex using sed, and prepend the UTF8 and UTF16 BOM bits:
xxd -plain pwdlist.txt > pwdlist_hex.txt
sed '1s/^\(\xff\xfe\)\?/\xff\xfe/' pwdlist.txt > pwdlist_hex2.hex
sed 's/^\(\xff\xfe\)\?/\xff\xfe/' pwdlist.txt >> pwdlist_hex2.hex

When running this file as a password list with or without --hex-charset, hashcat aborts automatically since a byte order mark (BOM) was detected in the password list.

2) I tried to run the password list converted as hex with prepended BOM marker bits as a mask.
I got the error:
“Invalid hex character detected in mask” for every single line

3) I tried running a hyibrid attack with the the mask first followed by the normal password list. so prepending ?b (short for any kind character, I do not even know if this includes BOM markers)
-The resulting attack was way to slow to work since the mask is generate by the processor and not the GPU, so I aborted the attack myself
-I tried running a mask attack with ?b at the end in case the hidden characters are at the end which, surprisingly did run and finished in 2 days.

4) I tried converting the whole plain text password list into a mask, so basically prepending ?b at every line.
The results was the same as in attack 3, the attack was way to slow to ever finish so I aborted.

5) I red this post which is the best information I could find on the forum, the problem is that I do not fully understand the advice given:
https://hashcat.net/forum/archive/index....181-3.html
I tried to create a rule that prepends BOM markers as described below


“.... so now let's look at how to do the append/prepend thing with rules. The rule file "my.rule" should look something like this (you need to adjust it to your needs).
ATTENTION: within this section I just show how the characters would look like, you still need to insert the actual character into the file by replacing the <xy> with the actual characters:

^<bf> ^<bb> ^<ef> ^<09> +0 {

(Note: this just prepends the 3 BOM bytes, in reverse order !!!, and appends the line feed... This is just an example of a more "complicated" rule)”

So without understanding much what I was doing I copied ^<bf> ^<bb> ^<ef> ^<09> +0 { into a new rules file and run a normal password attack with this rule and got the following error:
skipping invalid or unsupported rule in file rules/bom-bit.rule on line 1: ^<bf> ^<bb> ^<ef> ^<09> +0 {
No valid rules left. The rule was found to be invalid

I think with this rule based attack approach  I am on the right track since the rotation would hopefully mean that hashcat does not auto abort when encountering a BOM bits and it uses the GPU which mean the attack should be fast enough.
Can anyone point me in the right direction since these BOM bits are bringing me to the end of my wits?
In generally I think it would be desirably to have a widely available rules to search for prepend and or appended hidden characters and BOM bits. Also I was a bit disappointed that I could not use --force to make haschat accept the input wordlist as hexadecimal character that included BOM bits. I would have expected there would be a way to force hashcat to accept any input the user provides.

Please help.

[philsmd]

you could just use rules to prepend the chars

Code:

^\xfe ^\xff

this can be used with the -j parameter directly or with a rule file (-r).

btw: it's important that the order is like mentioned above... first prepend the 0xfe char and afterwards prepend the 0xff byte to make sure that \xff is at the very beginning (yeah, reversed order when prepending with the ^ rule).
hint: you can also use the command

Code:

hashcat --stdout -j "^\xfe ^\xff" pwdlist.txt

to see what is going on (debug it), or create a new dict with redirect (>) or even better use the output (-o) command line argument to store/append it to a file . but yeah, it's true that storing it to a file doesn't make much sense, because hashcat gives an error if BOM is at the beginning of the dictionary file (so it's just for testing and making sure your rules work).

[monyanus]

@philsmd Thank you so much for your support. I think I start understand a bit better now how to work well with hashcat and how to test and write rules.
Unfortunately when I searched for my word list while prepending the BOM bits '^\xfe ^\xff' (UTF-8 BOM bits) and '^\xff ^\xff' (UTF16 BOM bits), I did not get hit. Meaning there are probably other newline characters prepended or appended in the password. I know the password was copied from a text file on Windows hence, I am pursuing this angle.

Based on the example you provide, I assume that when \x is encountered in a rule or mask, it indicates that a hexadecimal characters follow?

I create a rule set that should look for all possible BOM bits prepended as well as all possible white new line characters prepended or appended, see below:

#####################################################################################
## Rule set to look for passwords with BOM bits and prepended or appended white space characters
#####################################################################################

## BOM Bits UTF-8 and UTF16
^\xfe ^\xff
^\xff ^\xff

## New line characters https://en.wikipedia.org/wiki/Newline
## Prepend all new  line characters from ASCII, ATASCII, EBCDIC, non-ASCII character set
^\x0A
^\x0A ^\x0D
^\x0D
^\x1E
^\x0D ^\x0A
^\x9B
^\x15
^\x76

## Append all new  line characters from ASCII, ATASCII, EBCDIC, non-ASCII character set
$\x0A
$\x0D $\x0A
$\x0D
$\x1E
$\x0A $\x0D
$\x9B
$\x15
$\x76

#########################################################################

Since I did not find the password yet, I create a prepend and append rule to combine having both invisible, white space characters and BOM bits prepended as well as appended. They can simply be run like:

hashcat  -m 15700 -a0 hashcode.txt pwdlist.txt  -w3  --status --status-timer=5 -r rules/bom-bit-newline-characters-prepepend.rule rules/bom-bit-newline-characters-append.rule

In my case the password is used for a My Ether Wallet, hence -m 15700, be sure to change if you have another hash type to crack.

I attached the two rule sets to the original (first post to this thread, replace the file extentions '.txt.' with '.rule') in case any one else would ever encounter a similar issue and might need them. I think this is the case, especially since apparently pre-sold Ethereum wallets more often had the issue that they included hidden characters and white space characters in the password used for encrypting the wallet file:

1) bom-bit-newline-characters-prepend.rule
2) bom-bit-newline-characters-append.rule