Potfile format - to hex or not to hex

[devilsadvocate]

Are there any utilities to convert passwords in a pot file to all hex or all plaintext?

For example, if I have a potfile and it has mixed entries for found passwords and some look like $HEX[hexcharactershere], is there a way to convert those to their plain-text equivalent?

Related to that, would converting from hex to ascii be problematic if some CR/LF/NUL characters are present?
See here: https://hashcat.net/forum/thread-2483.html

Similarly, is there a way to convert all plaintext entries from a potfile to all $HEX[] format?

For hex to ascii conversion, this will do in a pinch (not tested):
From https://hashcat.net/forum/thread-3522.html
perl -ne 'if ($_ =~ m/\$HEX\[([A-Fa-f0-9]+)\]/) {print pack("H*", $1), "\n"}'

But what about ascii to hex?

I will probably write something if this doesn't exist.

*Edited*

Cheers to royce and undeath for making all of the following scripts possible:

Scripts that you will find in this post:
convert_plaintext_2_hex.pl - converts plaintext password files to their equivalent values in hex
convert_hex_2_plaintext.pl - converts passwords in $HEX format to their equivalent in plaintext.
convert_potfile_plain_2_hex.pl - leaves all hashes intact in a potfile and converts all found passwords to $HEX format
convert_potfile_hex_2_plain.pl - leaves all hashes intact in a potfile and converts all found passwords to plaintext format


convert_plaintext_2_hex.pl

Code:

#!/usr/bin/env perl

use utf8;

while (<>) {
 if ($_ =~ m/\$HEX\[([A-Fa-f0-9]+)\]/) {
     print $_;
 } else {
     if ($_ =~ m/(.*)/) {
         print unpack("H*", $1) . "\n"
     }
 }
}

Use convert_plaintext_2_hex.pl like this:

cat passwords.txt | convert_plain_2_hex.pl > passwords_hex_format.txt
or an example with an unsalted md5 potfile (adjust bytes with cut as necessary depending on hash type)
cat md5_found.pot | cut -b 1-33 --complement | convert_plaintext_2_hex.pl > md5_pot_passwords_in_hex.txt


convert_hex_2_plaintext.pl

Code:

#!/usr/bin/env perl

while (<>) {
  if ($_ =~ m/\$HEX\[([A-Fa-f0-9]+)\]/) {
      print pack("H*", $1), "\n"
  } else {
      print $_;
  }
}

Use convert_hex_2_plaintext.pl like this:

cat passwords.txt | convert_hex_2_plain.pl > passwords_plaintext_format.txt
or an example with an unsalted md5 potfile (adjust bytes with cut as necessary depending on hash type)
cat md5_found.pot | cut -b 1-33 --complement | convert_hex_2_plaintext.pl > md5_pot_passwords_in_plaintext.txt


convert_potfile_plain_2_hex.pl

Code:

#!/usr/bin/env perl

use utf8;

while (<>) {
  if ($_ =~ m/(.*):\$HEX\[([A-Fa-f0-9]+)\]/) {
      print $_;
  } else {
      if ($_ =~ m/(.*?):(.*)/) {
          print $1 . ':$HEX[' . unpack("H*", $2) . ']' . "\n"
      }
  }
}

Use convert_potfile_plain_2_hex.pl like this:
cat found_passwords.pot | convert_potfile_plain_2_hex.pl > potfile_hashes_and_plaintext_passwords.txt


convert_potfile_hex_2_plain.pl

Code:

#!/usr/bin/env perl

use utf8;

while (<>) {
   if ($_ =~ m/(.*):\$HEX\[([A-Fa-f0-9]+)\]/) {
       print $1 . ':' . pack("H*", $2), "\n"
   } else {
       print $_;
   }
}

Use convert_potfile_hex_2_plain.pl like this:
cat found_passwords.pot | convert_potfile_hex_2_plain.pl > potfile_hashes_and_HEX_passwords.txt

In addition to the above, there are utilities available from insidepro.  Those are available at: http://www.insidepro.com/download/HM.zip

[royce]

FWIW, I've been using that same Perl one-liner from undeath, expanded for readability and converted to a standalone script:

Code:

#!/usr/bin/env perl

# Credit: undeath, https://hashcat.net/forum/thread-3522.html

while (<>) {
   if ($_ =~ m/\$HEX\[([A-Fa-f0-9]+)\]/) {
       print pack("H*", $1), "\n"
   } else {
       print $_;
   }
}

The inverse could be used to perform the conversion in the other direction.

Out of curiosity - what is the use case for converting to all HEX?

[devilsadvocate]

FWIW, I've been using that same Perl one-liner from undeath, expanded for readability and converted to a standalone script:

Code:

#!/usr/bin/env perl

# Credit: undeath, https://hashcat.net/forum/thread-3522.html

while (<>) {
   if ($_ =~ m/\$HEX\[([A-Fa-f0-9]+)\]/) {
       print pack("H*", $1), "\n"
   } else {
       print $_;
   }
}

The inverse could be used to perform the conversion in the other direction.

Out of curiosity - what is the use case for converting to all HEX?

Storing the list of values in a column in a database table would be one.  Commas and single quote characters that could be in passwords don't play nice with an insert statement.

Either that or a hashcat user would like to use "--hex-wordlist" instead of traditional plain text.

And then, of course, there is trying all plaintext passwords in their hex form up against your hash(es).

For anyone who cares, this will convert all plaintext passwords into hex format.

Code:

#!/usr/bin/env perl

# Credit: devilsadvocate (inspired by royce)

while (<>) {
  if ($_ =~ m/\$HEX\[([A-Fa-f0-9]+)\]/) {
      print $_;
  } else {
      print unpack("H*", $_), "\n"
  }
}

[royce]

Interesting use cases, thanks.

Two things, though:

1. Since the subject of this post starts with "Potfile format", I assumed that the end goal is to convert a potfile, but retain the potfile format. The script that I provided takes a potfile as input and produces an un-HEXed potfile, of the standard form hash:plain. Your script converts both the hash and the plaintext of a potfile to hex, including the colon separator.

EDIT - Yikes! I posted the wrong version!

Code:

#!/usr/bin/env perl

# Credit: undeath, https://hashcat.net/forum/thread-3522.html

use utf8;

while (<>) {
    if ($_ =~ m/(.*):\$HEX\[([A-Fa-f0-9]+)\]/) {
        print $1 . ':' . pack("H*", $2), "\n"
    } else {
        print $_;
    }
}

2. To clarify, are you saying that you try the hex-as-a-string form of passwords against hashes? I would expect very few hits from this technique.

[royce]

And here's a rough pass at the reverse:

Code:

#!/usr/bin/env perl

# Credit: undeath, https://hashcat.net/forum/thread-3522.html
# ... and devilsadvocate ;)

use utf8;

while (<>) {
   if ($_ =~ m/(.*):\$HEX\[([A-Fa-f0-9]+)\]/) {
       print $_;
   } else {
       if ($_ =~ m/(.*?):(.*)/) {
           print $1 . ':$HEX[' . unpack("H*", $2) . ']' . "\n"
       }
   }
}

[devilsadvocate]

Interesting use cases, thanks.

Two things, though:

1. Since the subject of this post starts with "Potfile format", I assumed that the end goal is to convert a potfile, but retain the potfile format. The script that I provided takes a potfile as input and produces an un-HEXed potfile, of the standard form hash:plain. Your script converts both the hash and the plaintext of a potfile to hex, including the colon separator.

EDIT - Yikes! I posted the wrong version!

Code:

#!/usr/bin/env perl

# Credit: undeath, https://hashcat.net/forum/thread-3522.html

use utf8;

while (<>) {
    if ($_ =~ m/(.*):\$HEX\[([A-Fa-f0-9]+)\]/) {
        print $1 . ':' . pack("H*", $2), "\n"
    } else {
        print $_;
    }
}

2. To clarify, are you saying that you try the hex-as-a-string form of passwords against hashes? I would expect very few hits from this technique.

1. Good point.  I took that for granted.  The perl script was only a partial solution.  For md5, I was executing:
cat md5_found.pot | cut -b 1-33 --complement | convert_plaintext_2_hex.pl > md5_pot_passwords_in_hex.txt

That takes out the hash and the colon and just leaves the hex.

2. Try it with "-r ./rules/dive.rule".  You may be surprised at what you get.

Also, the hex conversion leaves "0a" at the end of every conversion due to the CR character.  That may not be so desirable.  I'll look at this more soon and see what can be done to account for that.

[royce]

I know this is going to sound silly, but just to triple-confirm ... you're saying that you convert your plaintext dictionaries to hex strings, producing output like this (from the first ten lines of rockyou):

313233343536
3132333435
313233343536373839
70617373776f7264
696c6f7665796f75
7072696e63657373
31323334353637
726f636b796f75
3132333435363738
616263313233

... and then run those strings themselves, unmodified just as they appear above, through dive.rule?

[devilsadvocate]

I know this is going to sound silly, but just to triple-confirm ... you're saying that you convert your plaintext dictionaries to hex strings, producing output like this (from the first ten lines of rockyou):

313233343536
3132333435
313233343536373839
70617373776f7264
696c6f7665796f75
7072696e63657373
31323334353637
726f636b796f75
3132333435363738
616263313233

... and then run those strings themselves, unmodified just as they appear above, through dive.rule?

Yes, sometimes.  It will occasionally produce a few hits that I wouldn't get otherwise.

Also, I must have missed the part that makes this require triple confirmation.

[devilsadvocate]

I just realized that the conversion from hex to plaintext is probably not handling UTF-8, non-ASCII characters correctly with the conversion process.

A functional workaround will probably be to convert the entire wordlist to hex format and just use that with --hex-wordlist.

I will test and report back... eventually.

[undeath]

hex-encoded words are raw binary data. No guesses about encoding are done. That's up to the user (you).