hcstatgen not making 32.1MB files

[Hash-IT]

I am playing with hcstatgen today and I am trying to make some custom .hcstat files for use with oclhashcat-plus. i am not sure what I am doing wrong as I think I understand the process, perhaps someone will notice something.

I have tried the following using Win7 64bit, latest 1.0b1 utilities

hcstatgen.exe out.hcstat < infile

The "infile's" I have been using have been of different sizes. I have tried three, 57.7MB, 1.17GB and a tiny 18 line list.

I am not getting the output as described in the wiki "Each outfile will have exactly 32.1mb"

I didn't expect a 32.1MB file from my tiny 18 line list but I did from the other 2. All I got list 1 (57.7MB) was a 328KB hcstat and list 2 (1.17GB) only 4.60MB hcstat.

Despite these lists not being the size described I thought I would try to use them anyway but oclhashcat-plus gives me the error "./hashcat.hcstat: Could not load data"

I also noticed there seems to be a maximum input file size for hcstatgen, does anyone know what it actually is please ?

Thanks.

[atom]

Sounds like you are using a beta hcstatgen which is not compatible to old statsprocessor

[Hash-IT]

Yes I am, sorry I guess that explains it.

Can you tell us what is the largest input (password list) we can use is please ?

Thank you.

[atom]

There is no such limit

[Hash-IT]

I think there must be a limit with the new beta. If I use a 1.7GB input file I can see hcstatgen working away and it displays the lines it is going through, it also takes nearly 60 seconds. However when I try on a 6GB list it pretty much finishes instantly.

I didn't think this was worth a Trac ticket, until I am sure I am using it properly

[atom]

No there is no limit. Can you paste cmdline? Can I download the dictionary?

[Hash-IT]

No there is no limit. Can you paste cmdline?

Sure, but it probably isn't much use. The word list is in the same directory as hcstatgen.exe.

Code:

hcstatgen.exe test.hcstat < large_list.txt

You are right atom, it seems to be only on 2 large lists I have. I have another 5.8GB list that actually works. I have made other lists since posting to test as I was onto the same idea.

I have tested the older hcstatgen.exe V 0.9-64 and it also just stops immediately.

It must be my list, but I am unsure why as it is a cleaned one and it works with oclhashcat-plus.

Sorry this isn't a list I downloaded from the net and I can't upload such a huge list. I will keep cutting it down until it works and try to see what part of it is crashing hcstatgen.

This is something I can test on my own to save you time. If I isolate the problem character I will send a smaller list.

If this means anything to you the list is sorted small to large (word length) and it should all be lower case and unique. There are probably some non UK standard keyboard characters in there to.

To give me an idea where to look can you tell me does hcstatgen work on a list as I would view it ? What I mean is in this case will it start on the smallest words first ? If it does I know I am looking for the problem in the first half.

Also as hcstatgen.exe stops so quickly after starting on this list is there some sort of test it makes on the text file before starting ? My list could be failing for another reason, meaning it isn't actually a character problem at all.


Thanks.

[Hash-IT]

I think I have found the problem !

Its this... Unicode

Code:

•f

Its not showing up on the forum like it does in notepad. I have tried to attach it as a .txt, .7z and a gif but I keep getting forum errors messages ...

Please correct the following errors before continuing:

The file upload failed. Please choose a valid file and try again. Error details: There was a problem moving the uploaded file to its destination.

If you place that character as a line in your password list it will stop hcstatgen.

If this is right then I wonder if people have not been getting the best from hcstatgen up until now ? I guess no one else noticed it was finishing quickly when it hit one of these characters. This would mean they would have been using incomplete statistics as hcstatgen wasn't checking beyond the point it ran into this.

I guess there will be an improvement in this attacks results if we all remake our stats files when it can read or at least ignore this character ?

[epixoip]

can you hexdump that character and find out what it is? it's showing up as two separate characters on the forum, as \xe2\x80\xa2\x66

[Hash-IT]

This what you want ?

95661A0D0A

It actually looks like 3 characters, a large dot, an f and a small arrow pointing to the right.