hashcat iconv

[cryptolovi]

Thought I had the iconv down on hashcat, but since I revisited it lately my assumptions where different.

When using the encoding from and encoding to I expected some like;

printf "boo\n\baa\npo\xebzie\n"|hashcat64.bin -m 0 278fbfac8b69dda968b190e6bb1f4ac9 --encoding-from iso-8859-1 --encoding-to ASCII//TRANSLIT

To find the target, since;
printf "po\xebzie" |iconv -f "iso8859-1" -t "ASCII//TRANSLIT"

is poezie the above md5 and does the correct transliteration. But hashcat iconv module says the character can’t get transliterated and replaces it with a ? when using translit, thus generating 'po?zie' as a candidate.

When not using TRANSLIT hashcat skips the whole word (which is consistant with iconv when not skipping illegal characters)

What am I missing?

[philsmd]

if I generate a md5 hash with your input $HEX[706f657a6965] I get a completely separate result:

Code:

printf "po\xebzie" |iconv -f "iso8859-1" -t "ASCII//TRANSLIT" | md5sum
5453454a253f8840811af32c86d200e4

I think your hash is not correct. First of all, according to the forum rules you shouldn't post hashes and only if asked by admins/mods you NEED to mention the correct password.

I think you are referring to the status display when you say that there is a "?" somewhere. This can be a visual thing (and doesn't necessarily mean that the sequence of bytes that hashcat is trying is incorrect). The status display (prompt) might not show the password "correctly" depending on your shell/terminal encoding.

[cryptolovi]

[quote="philsmd" pid='43006' dateline='1544181928']
if I generate a md5 hash with your input $HEX[706f657a6965] I get a completely separate result:

Code:

printf "po\xebzie" |iconv -f "iso8859-1" -t "ASCII//TRANSLIT" | md5sum
5453454a253f8840811af32c86d200e4

The hash is wrong it's 'echo poezie|md5sum' (forgot the -n creating the hash for the commandline above). So it's searching for the wrong target at the example which I handcrafted. Sorry for posting a hash that might be confusing because the target is not known.

When I say I see the '?' it's not because of terminal translation issues but because I looked at the --stdout output. This option I can't use when giving the hash on the commandline. So, ok the handcrafting of the example errored. But the point still stands I think

For example the hexdump output of --stdout with the options --encoding-to 'ASCII//TRANSLIT' only (using a one word dictionary and the correct hash)
00000000 35 34 35 33 34 35 34 61 32 35 33 66 38 38 34 30 |5453454a253f8840|
00000010 38 31 31 61 66 33 32 63 38 36 64 32 30 30 65 34 |811af32c86d200e4|
00000020 0a 70 6f 3f 7a 69 65 0a |.po?zie.|
00000028

When using --encoding-from too on a dict file it gets a bit more complicated and error prone and you will get translations to;
0a 70 6f 3f 3c 3c 7a 69 65 0a |.po?<<zie.|
when the dict type is wrong (but will leave that out of scope since that's not what I'm questioning)

[philsmd]

well, I'm still not sure if 278fbfac8b69dda968b190e6bb1f4ac9 is the correct hash for your plain. I guess you do not know the password at all (which is bad since it would be a reason to ban you, but I'm not too strict right now since you might have found a potential "problem").

I troubleshooted this a little bit and came to the conclusion that it works with a different LC_ALL (setlocale settings), see:

Code:

#include <stdlib.h>
#include <stdio.h>
#include <iconv.h>
#include <locale.h>

int main (int argc, char *argv[])
{
  char src[] = "\xeb";
  char dst[10];
  size_t srclen = 1;
  size_t dstlen = 10;

  fprintf (stderr, "in:  %02x\n", (unsigned char) src[0]);

  char *pIn  = src;
  char *pOut = (char *) dst;

  // this line is needed for iconv w/ TRANSLIT to work (default LC_ALL is "C")!!!
  // see https://stackoverflow.com/a/7931939
  setlocale (LC_ALL, "en_US.UTF8"); // query it like this printf ("%s\n", setlocale (LC_ALL, NULL));

  iconv_t conv = iconv_open ("ASCII//TRANSLIT", "iso8859-1");
  // check return code !

  iconv (conv, &pIn, &srclen, &pOut, &dstlen);
  // check return code !

  iconv_close (conv);

  fprintf (stderr, "out: %02x\n", (unsigned char) dst[0]);

  return 0;
}

That means that according to https://stackoverflow.com/a/7931939 the locale settings matter a lot (especially for the TRANSLIT iconv configuration). I think the default is just "C" and I'm not sure if hashcat should change this and/or what configuration we should set (because setlocale (LC_ALL, "en_US.UTF8") might not always be the correct one and might lead to other strange behaviour with iconv specific and/or other function calls).

---

update: I found out that it might be enough to just add the

Code:

#include <locale.h>

and

Code:

setlocale (LC_ALL, "");

this might enable the system-wide localization settings (without the need to specify a specific locale).
see https://github.com/bminor/glibc/blob/c37...#L126-L127 for the iconv program in glibc

the disadvantage might be that if we do not use the default "C" locale, other functions might also return different results (if they behave differently with a changed LC_ALL/locale configuration). That means, that this might lead to side-effects beyond the iconv conversion
I guess we would need to investigate if it's worth to change it and which system functions might react differently with another LC_ALL config.

[cryptolovi]

Maybe I wasn’t clear. The above wrong hash (278f..) is the word poezie inclusive the end of line character. The correct target hash is the one you listed (5453..) which is the same word poezie without end of line character.
echo poezie |md5sum <> echo -n poezie|md5sum. Which was the typo I made when handcrafting the example. So I hope I can avoid the ban hammer since I know the target.

The locale settings are interesting, something I overlooked, since I’m almost positive I had hashcat do the proper transliteration in the past on some systems. Maybe that was the case....