06-08-2013, 08:29 PM
Tonight there was an interessting conversation on #hashcat IRC. A User "jnpplf" who's new to hashcat joined and asked couple of questions that we often see from people who have some background on hashcracking but either come from other tools or were not active in the hashcracking world for a few years and have outdated information.
I think it's an interessting read so I asked for epixoip's and jnpplf's permission to post it here and they agreed.
I think it's an interessting read so I asked for epixoip's and jnpplf's permission to post it here and they agreed.
Code:
jnpplf (~jnpplf@unaffiliated/jnpplf) has joined #hashcat
jnpplf If my password dictionary is a) numeric only, and b) huge, is there any feasible way of storing it in something that supports integer storage to save space, but also be able to use it with hashcat?
jnpplf Rough math, I believe that would make my wordlist 186GB as text (though I have others that would require much, much more space), or 78GB as ints
epixoip doesn't make any sense to have a large numeric-only dictionary
epixoip straight wordlist attacks are slower than brute force, and brute forcing digits is fast.
epixoip single 7970 can do len 1 - 12 in just over 3m
jnpplf So, use a mask to 'generate' the numbers?
epixoip yes
jnpplf That makes sense
epixoip for example, -a 3 -i ?d?d?d?d?d?d?d?d?d?d?d?d
jnpplf How much benefit would I see from precomputing the hashes?
epixoip zero
jnpplf Really?
epixoip there's a reason we don't use rainbow tables anymore.
jnpplf Is this a disk speed thing?
epixoip no, it's a "tmto is impractical and slow" thing
epixoip gpus have by and large deprecated rainbow tables
jnpplf Interesting. How about if the hashes are generated with some hardcore key stretching / bcrypt with high rounds?
jnpplf I'm off of my original question and into theory now, apparently my info is way out of date
epixoip you can't have a rainbow table for bcrypt anyway
epixoip or any other salted algorithm
epixoip so that question does not apply
jnpplf Oh, you can't run bcrypt without a salt?
epixoip nope
epixoip nor would you want to
jnpplf Also news to me.
jnpplf catches up to 3 years ago
jnpplf Rainbow tables are /really/ redundant?
epixoip yes
jnpplf Dang.
epixoip no one uses them anymore
jnpplf Well that's an item crossed off of my todo list :)
epixoip haha
jnpplf Don't suppose you have a link to the tmto for rainbow tables being redundant so I can read up on it?
epixoip it's common knowledge, don't know if anyone has bothered to write about it
jnpplf It's not THAT common
epixoip it is among password crackers.
epixoip it's not really something you need to read much about. just do the math
jnpplf If I were trivially capable of doing that math, I probably wouldn't be here
epixoip rainbow tables are large, inflexbile, and you have to have one set per length, and per algorithm. they also don't scale well at all with multiple hashes.
epixoip gpus are fast, flexbile, and scale linearly
jnpplf Interesting. Maybe I'll just spend my time getting better with hashcat's mask syntax then
epixoip good idea
jnpplf So if I wanted to, say, crack (dog|cat)1234, would I have to run two separate attempts? I can't figure out a way of writing a mask to match that.
jnpplf That's supposed to be a logical OR rather than a literal
epixoip you wouldn't use a mask attack for that
epixoip you'd use a hybrid attack
jnpplf dictionary + mask
jnpplf I seeeeeeeee
epixoip right
jnpplf Good stuff, thanks
jnpplf : dont read on it, do a benchmark, digit wordlist vs gpu bruteforcing
epixoip no worries
jnpplf oo, had another question before I shut up
jnpplf What if the second half of the password is derived from the first half? I could pipe in valid values somehow, but that means I don't apply either the mask or dictionary? Just generate something from a shell script and get hashcat to accept external input?
epixoip oclhashcat supports reading from stdin
epixoip cpu hashcat can read from a named pipe
jnpplf so generate_passwords.sh > oclhashcat [...] will work?
epixoip |, not >
epixoip but yes
jnpplf Would | be for cpu hashcat?
jnpplf wouldn't*
epixoip no
epixoip for cpu hashcat you'd have to use mkfifo to create a named pipe
epixoip then specify the name of the pipe as a wordlist
jnpplf Ah. Funky, and beyond my understanding.
epixoip generate_passwords.sh > oclhashcat would replace the contents of the oclhashcat binary with the output of generate_passwords.sh
jnpplf so totally_optimised_awesome_script | oclhashcat -awesome_args is my best bet
jnpplf epixoip, good point
epixoip yep
jnpplf Any idea what the fastest language might be for generating those? I feel like bash isn't the best candidate.
jnpplf Or, alternatively, is that not going to be the bottleneck?
epixoip doesn't matter
epixoip it's going to be slow regardless
jnpplf slow?
jnpplf That's exactly what I don't want.
epixoip it's unavoidable in this instance, as you're generating plains on the cpu and then transferring them to the gpu
epixoip you will not be able to fully utilize the gpu in this instance
jnpplf Unless I could pregenerate the list ;)
jnpplf Or is the CPU still faster?
epixoip straight wordlist attacks are also slow on gpu
epixoip becuase you're not giving them enough work
epixoip in order to fully utilize gpus with wordlists you have to add rules
epixoip for straight wordlist attacks, gpu is either same speed or slower than cpu
jnpplf Hmmm, doesn't look like any of the rules would help in my scenario
epixoip you could always write your own rules, if needed.
jnpplf that's the maskprocessor stuff?
epixoip external mask processor
epixoip standalone implementation of the hashcat mask engine
jnpplf Hmmmmmmmm. So the rules still rely on hashcat's mask formatting, which rules out appending the derived part of the password. Might be able to speed up the prefix generation though?
jnpplf Not sure it would help versus using a dictionary for prefixes as they're known
epixoip no
epixoip the rules are used with wordlists, and have their own syntax. it's a small programming language, really.
epixoip see http://hashcat.net/wiki/doku.php?id=rule_based_attack
jnpplf My problem is that, while the rules look suitably wonderful, the help for maskprocessor makes it look extremely limited
epixoip the rules have nothing to do with maskprocessor
epixoip mask processor is just for processing masks.
epixoip the two have nothing to do with eachother
jnpplf Oh
epixoip maskprocessor is mostly so that you can add hashcat's mask functionality to other programs, like pyrit or john the ripper.
jnpplf "Using maskprocessor to generate rules"
jnpplf That's confusing :p
epixoip oh
epixoip yeah i suppose that is misleading
epixoip english is atom's second language so you will have to forgive him.
epixoip oh no no
epixoip i see what you're referring to
epixoip in that instance, that's actually using mask processor to help you generate repetitive rules
epixoip which is most certainly valid
epixoip like if you wanted to generate a rule that appended two lower alpha to each word in a word list
epixoip you wouldn't want to type that shit by hand
jnpplf Wouldn't that just be a hybrid attack though?
jnpplf dictionary + 2 alpha as a mask?
epixoip so you can just use mask processor to generate the rule file for you: ./mp '$?l$?l'
epixoip yeah, but in some instances you can't use a hybrid attack
epixoip then you can use multi-rules
epixoip in some cases it's also more efficient to use rules than hybrid attack
epixoip such as when using mask + dict with a very small mask
epixoip in that case it would be much more efficient to use rules
jnpplf But if the mask is long, it's more efficient to use the hybrid?
epixoip yes
epixoip if possible
jnpplf Brain is about to crawl out of my ear :p
epixoip what if you want to use a hybird attack + rules?
epixoip can't do it
epixoip but you can use rule chaining
epixoip anyway that's not really maskprocessor's raison d'etre, that was really just mentioned as an aside
epixoip just something neat that you could do with mp
jnpplf The examples on the rule attack page make it look like it's most effective if you're trying to fuzz the wordlist. If the format is fairly predictable, does that make it less useful?
epixoip you could also accomplish the same thing with a shell one-liner, but it would be slower.
epixoip if the format is predictable then you can simply write predicatble rules
jnpplf OK, this is going to be much easier if I just explain what I'm doing :p
epixoip for example, if the algorithm is md5(strtoupper(pass)), then you can use the single rule 'u' for all of your attacks
jnpplf dict + known number of digits + single character checksum
epixoip ok, in that case just use -a 6
epixoip what charset is the checksum comprised of?
jnpplf numeric
epixoip ok so then it's really just dict + known number of digits
epixoip -a 6 dict.txt ?d?d?d?d?d
epixoip or whateer
jnpplf Would that be faster than calculating the final digit?
epixoip yes, much faster
jnpplf So I'm solving the wrong problem :p
epixoip with hybrid attacks, the canddiates are generated on the gpu, enabling full acceleration
jnpplf So the GPU would have to be over 10 times faster than the CPU to make that worthwhile
epixoip and the extra digit on the end only adds 1^10 complexity
epixoip yes
epixoip and the GPU is a lot more than 10 times faster than the CPU
jnpplf Interesting.
epixoip in the case of MD5, GPU is probably 100x faster than CPU
epixoip maybe more
jnpplf This is more likely to be SHA-1
jnpplf Just because 2013 :)
epixoip sha-1 should be about 60x faster than cpu
jnpplf And that assumes a nice saucy AMD card, rather than my midrange NVIDIA?
jnpplf ATI*, I guess
epixoip that doesn't mean much. raw md5 is still the most widely used algorithm for password storage, even in 2013.
epixoip amd is correct. ati doesn't exist anymore.
epixoip and yes, those figures are for a 7970
jnpplf I know, but I rarely see them called AMD cards
epixoip we call them amd cards here
jnpplf Check you guys out :p
epixoip nvidia sucks for password cracking, so it definitely won't be 60x faster
epixoip but it will probably be at least 10x faster than cpu
jnpplf But then the AMD benchmarks seem to be about 6 times faster than CUDA
vn but it rocks for gaming
jnpplf Also that :p
epixoip who plays games?
vn not you, obviously.
epixoip obviously.
jnpplf Worst case, as long as using the GPU isn't obviously horrible compared to the CPU, even with cuda, I'd write for AMD and then offload the processing to someone with an AMD card
jnpplf Or 4, as this case may be :)
jnpplf That said, EC2 offers some GPU nodes now, right? Are they AMD based?
epixoip no
epixoip they're super old teslas
epixoip and telsas are worse than their GT* counterparts for crypto
epixoip ec2 is a massive waste of money if you intend to use it for password cracking
epixoip if this is something you plan to do often, go pick up a cheap AMD gpu, like a 7950
epixoip costs less than an ec2 gpu instance.
epixoip 10x the performance
jnpplf I am learning all the things this evening
epixoip good, good
jnpplf Is this related to your day job?
epixoip yes.
jnpplf infosec, or specificially hash cracking?
jnpplf Stricture? :)
epixoip yes, stricture
jnpplf Oh cool. I've sent some business your way :)
epixoip orly?
jnpplf Well, in truth I've referred people to you
jnpplf It may not have become business
epixoip right on, we appreciate it
jnpplf You're the only company I know of that does it as a core offering
epixoip yup
epixoip i don't know of any others, either
epixoip although we're trying to get into hardware sales as well
jnpplf ASICs, or just selling people 7950s? :p
epixoip no, not asics. just dedicated clusters
epixoip we have a few customers we're building clusters for now, using 7970s and 7990s
jnpplf Why would someone buy a cluster of commodity hardware?
epixoip it's not commodity hardware
epixoip we use enterprise grade hardware with commodity gpus, and a warranty
epixoip we use commodity gpus because there's no better option for crypto
epixoip some would say firepro would be more "appropriate," but our customers don't want to spend 4x as much for half the performance.
epixoip and i don't blame them, i certainly wouldn't advise them to do it
jnpplf Had never heard of them until now.
epixoip firepro?
jnpplf Yeah
jnpplf Just googled them up
epixoip that's amd's professional workstation and server product line
epixoip but they still use the same GPUs as the radeon line
epixoip just underclocked
jnpplf So how many people are at Stricture? 1 - 10 on LinkedIn, but only one with a profile
epixoip 5
jnpplf Interesting
jnpplf Time to go fumble my way through some masks then, I guess
epixoip i already gave you the mask :P
jnpplf Yeah but I'm a spaz ;)
epixoip -a 6 dict ?d?d?d?d?d
epixoip plug n play
epixoip (that's assuming it's word + 4 digits + 1 digit checksum)
jnpplf If I need to go through a few different masks/dicts, is there an obvious efficiency to trying to get them all going in the same execution?
epixoip alter mask appropriately.
epixoip yeah. the more work you give it, the more of the gpu you will utilize
epixoip you have to keep it busy to make efficient use of it
jnpplf So... if my dict were (it isn't) one digit long
jnpplf In fact, ignore that
jnpplf If I know the first n characters of a password, at what point does it become more efficient to use a dictionary versus a mask?
jnpplf Can my mask contain a constant?
epixoip yes, your mask can contain a constant
jnpplf So all passwords begin with 'j', use j?d?d
epixoip yup
epixoip and if that's the case, then it would be more efficient to use a mask
jnpplf How about ja..jc?
epixoip -1 abc j?1?d?d/d...
epixoip er, ? not /
jnpplf And still faster than a dictionary?
epixoip yes
epixoip the mask engine generates candidates on the gpu
epixoip which makes it very fast
jnpplf So this comes back to the 10x / 60x / Nx speed increase of the GPU over the CPU?
epixoip right.
jnpplf So I'd need the keyspace of the additional mask to be 60 or more values if my GPU were 60x faster than my CPU
epixoip i'm not sure what you mean
jnpplf Oh, erm...
jnpplf So if I'm cracking dict + mask, it becomes more efficient to use mask+mask as long as the first mask isn't generating more than 60 candidate values
jnpplf assuming a 60x speed increase on the GPU
epixoip no
jnpplf balls
epixoip that only applies if you're doing additional work
epixoip like in the case of just brute forcing the checksum
epixoip we'd have to 1^10 additional work
epixoip if you're doing a straight mask like in the example above, then it would still only be 1^10 additional complexity over the script that calculates the checksum.
jnpplf Sure
jnpplf So.... you're saying the dictionary is FASTER than the mask as a prefix?
epixoip the mask attack will always be much faster than a dictionary or hybrid attack because all of the canddiates are generated on the GPU, there's no memory or host-to-device transfering happening.
epixoip no.
jnpplf So why would you ever use a dictionary?
epixoip because most people don't crack plaintexts that follow some defined pattern like this
epixoip i'm speaking to your specific case, where you have this one specific pattern that you're exploiting
jnpplf OK, so say I'm cracking 8 digit numeric only
jnpplf Oh wait
jnpplf COnfusing myself, gimme a sec :p
epixoip 8 digit numeric you would absolutely want to do a mask attack, no question
jnpplf What if part of it were predictable though?
jnpplf Yep, did it again
epixoip then you have a choice
jnpplf That's when you use the static mask :p
epixoip right
epixoip you can use constants in the mask, or you can do hybrid
jnpplf I think my brain's hitting its new info limit
epixoip the former would likely be faster
epixoip let me give you some advice: stop talking/thinking about it, and just go try it :P
jnpplf Yeah
epixoip once you start doing it you will understand
jnpplf I've done it before, I was looking for efficiency improvements :)
epixoip there are several different ways to skin this cat
epixoip if you know that the password is exactly 8 characters long, starts with a j, and ends in 5 digits, then mask attack is the obvious choice
epixoip you can just blow through it with j?l?l?d?d?d?d?d
jnpplf If it were 8-starting-with-j or 9-starting-with-x, would I run 2 separate jobs or try to force that into a mask?
epixoip two separate masks
epixoip the attacks will run very quickly
jnpplf Does hashcat accept multiple masks in one run?
epixoip no
epixoip but you can use a for loop
epixoip for m in j?l?l?d?d?d?d?d x?l?l?l?d?d?d?d?d; do ./hashcat -a 3 $m ; done
jnpplf Gotcha
jnpplf OK, that's all I'm going to ask :)
jnpplf Going to go melt my lovely graphics card
jnpplf Thanks a lot for the help
epixoip :)
epixoip no worries
epixoip have fun!
jnpplf rages into the wilderness
jnpplf -a 6 -1 abc hashes.txt j?1?d?d?d... doesn't run
jnpplf Just gives me a syntax error
jnpplf Same if I shift -1 abc to after the hashfile name
jnpplf headdesks
jnpplf Wait. I'm missing the hash type.
jnpplf hnnnngh, it always takes me ages to figure out how many args I'm supposed to be passing to this thing
epixoip using sha1 right?
jnpplf Yeah
jnpplf I added the hash type
jnpplf It was acting like it didn't have enough args, I split my mask into two and it tries to open one half as a file...
epixoip ./hashcat -m 120 -n 80 -u 1024 -o jnpplf.pot hash.txt -a 6 -1 abc hashes.txt j?1?d?d?d
epixoip oh
epixoip i just copy/pasted your line
epixoip and i see your problem
epixoip er nm, no i don't.
jnpplf hash.txt, hashes.txt?
epixoip er yes i do
epixoip sorry, i'm tired
epixoip you're using -a 6 instead of -a 3
epixoip and hash.txt, hashes.txt, whatever your file is called
jnpplf Well you have both, is my point
jnpplf You have an output, an input, and something else
epixoip that's because i just copy/pasted your line and tacked it on mine without reading it :)
jnpplf Ah
jnpplf Seems to be running now :)
epixoip ./hashcat -m 120 -n 80 -u 1024 -o jnpplf.pot hash.txt -a 3 -1 abc j?1?d?d?d
jnpplf That's the ticket, though without the performance args at the moment
jnpplf ooo, this is 550MH/s, versus the 300 or so I was getting with my original approach. Fun :)
epixoip add the performance args, should be able to get a bit more.
jnpplf GPU is 99% util
epixoip sure, but -u for example optimizes the number of loops per iteration.
epixoip so you will still likely see better performance
jnpplf Those args are 100% indecipherable to me ;) I'll try with though
jnpplf Seems a tiny bit faster
jnpplf 555 vs 553 at first glance
jnpplf Computer running like crap ;)
epixoip what gpu is this?
jnpplf 660Ti
epixoip ah
jnpplf That's also on a single hash
epixoip oh you're doing a single ahsh?
jnpplf Will generate a whole crapload to test against later
epixoip you don't want to use -plus then
jnpplf Just trying to get stuff stood up for now
epixoip you want to use -lite
jnpplf I don't even have lite. Separately download?
epixoip yes
epixoip lite is optimized for brute forcing a single hash
jnpplf I'm surprised they're separate binaries though
jnpplf Surely that would be easy to switch in code
jnpplf After all, it knows I'm doing a single hash because I had to use --force
epixoip completely different code base
jnpplf Oh really?
r4d1x not it isnt
epixoip no?
jnpplf grabs popcorn
epixoip atom told me they were quite different
r4d1x sry my reply was to "Surely that would be easy to switch in code"
epixoip oh oh okay :)
jnpplf Ah
epixoip you had me very confused :P
r4d1x epixoip: have you tried running amd and nvidia in the same box at all?
epixoip not with hashcat, but with multiforcer we have
epixoip or were you asking at an os level, like installing drivers for both and having xorg see both
r4d1x just wondering if you had tried with hashcat
epixoip ah. no i've not tried with hashcat
r4d1x might have to throw a 580 in and see what happens
jnpplf Hmm. If I write a mask of j?1?d?d?d..., where -1 is abc, that will only try ja, jb, jc, right? Not jab, jac, jabc...
r4d1x so, im listening to JFK tower, and the controller asks this guy to turn left into a storm, pilot comes back, I'm not turning into that storm, controller repeats turn left, pilot declares emergency, controller says he cant, pilots repeats declaring an emergency, controller clears him to land
jnpplf I've got 1 constant, second character is a choice of 2, 3rd onwards should all be ?d, but my Plain.Text starts with ****
r4d1x moral of the story, if you dont want to comply with the controller, declare and emergency
r4d1x s/and/an
epixoip jnpplf: yes, that's correct. it would only try ja, jb, jc, and not jab, jac, jabc, etc.
jnpplf In fact, why would Plain.Text not show me the full value being tested anyway?
epixoip r4d1x: that's awesome
r4d1x I know right?
r4d1x turn left
r4d1x no
r4d1x turn left
r4d1x FUCK YOU
r4d1x cleared to land
epixoip jnpplf: because it's trying 550 million combinations per second; you want it should print all 550 million of them? :)
jnpplf Then what
jnpplf 's Plain.Text even used for?
epixoip to give you a rough indication of where you are.
jnpplf Starting from the back? :-/
jnpplf Surely the first 4 digits are the most important
jnpplf Hmmm, something is up.
jnpplf Why, if I gave it a 10 digit mask, did it try 8 digits?
jnpplf Exhausted that, moved onto 9
r4d1x becuse you didnt set --pw-min=10
r4d1x lite assumes you want to increment
jnpplf I assumed the length was defined by the number of characters I put in the mask
jnpplf Ah
epixoip are you doing cpu or gpu?
epixoip oh you'r eusing lite
jnpplf I am now, yeah
epixoip right, lite increments
jnpplf Gotcha.
jnpplf But the mask will work fine on its own in plus?
r4d1x unless you specify --increment
jnpplf Freaky :p
epixoip yeah, plus requires an explicit switch (-i) to enable incrementing
r4d1x well, when that increment stuff was added I raised hell
r4d1x but I dont care about lite
epixoip yeah i don't like autoincrement
jnpplf Neither do I, normally
jnpplf This would explain why the progress values were so fast ;)