potfile, remove FAQ clarification

[rsberzerker]

I like to run a series of attacks in series. One thing I noticed between runs that takes up some time is the parsing of the potfile. Looking at the help, there is the option to disable it, but should I? I then checked the FAQ and found this section confusing:

Note that using a potfile is very different from the idea which you have in mind when you are used to use --remove. Having a hashlist with only uncracked hashes is fine, but with potfile you can do the same if you use the --left switch. For example, if your cracking session is finished and you want to have a left list, you simply run:

I do run with --remove. So is the FAQ saying if using --remove, then the potfile doesn't matter??? The "very different" doesn't seem to make sense. If I am using --remove and my hash list is already de-dupped, then is there a benefit from the potfile?

[philsmd]

In general, these 2 options (--remove vs. the potfile feature or --potfile-disable) are conceptually used for very different functionalities/feature with were different goals, but you are right that they have some common behaviours/overlaps: i.e. the end result is that if hashes were cracked previously, hashcat doesn't consider the cracked hashes anymore.

Let's step back a little bit and make these basic distinctions; very simplified we can say that this is the goal of the features:
- the remove feature is used to "update" the hash list to keep only the not-yet-cracked hashes (for instance if you need to share it or want to always see the remaining list without using --left separately)
- the potfile feature has the end goal to store the cracked hashes in a kind of "standardized" way (without using different outfile formats, see --outfile-format) and make sure that no "progress"/cracks are lost

Let's make also clear that "dedupe" isn't really the right word here, because we are talking about removing cracks (hashes that are already cracked) and not removing duplicate hashes (which hashcat always does automatically at startup).

You are right that you can simply use --remove... but sometimes that is not a good option (for instance if we keep in mind that the "hashcat brain" feature always need the same original hash list across all nodes/participants).
For many users this is not a problem, they also can just keep a backup of the original list (which is always a good idea btw) and, to be honest, many users do NOT need to run hashcat within these special modes (brain etc, advanced features).

The potfile feature has also always the advantage that if you forgot to use the --outfile (or short -o) option, that the cracked hashes are not "forgotton" and by using the potfile they wont be affected by any such problems (power failures while cracking or forgetting to use -o and therefore the cracks will only go to the screen if potfile is disabled/disallowed).

So yeah, you are kind of right that --remove + --potfile-disable + --outfile could work perfectly fine for you... you just shouldn't forget about their main goals and what the "risks" are if you for instance disable the potfile feature and the hash list won't be update (for instance if the users doesn't use --remove or a rig crash or power failure etc comes before the regular update of the updated/"removed" hashes list... btw the potfile feature normally just appends as the cracks come in... so it happens almost exactly at the time when the user can see the new cracks in the user interface/screen).

I would say that you don't need to worry too much about your command with both --remove and --potfile-disable ... you just should keep in mind what they are really doing and that the potfile disable feature normally should be only used by advanced users that understand why using the potfile most of the times makes sense, but still can be disabled if you know the differences and "risks".

[rsberzerker]

Overall, if I understand correctly, it sounds like the main benefit of leaving the potfile, if using --remove and you already have a backup of your original hash list, is preventing a loss of progress in the event of "a rig crash or power failure etc comes before the regular update of the updated/"removed" hashes list".

This does lead to the question of the session file. Do you know if that is updated before or after the "removed" hashes list?

and not removing duplicate hashes (which hashcat always does automatically at startup).

Did not know this. Nice bit info to have.

[philsmd]

The hashcat.restore (do you call this "session file" ?) file is again a different file... it will also be updated regularly, but only after a new "checkpoint" was reached. How frequently this happens depends a lot on the number of salts/hashes and the hash algorithm itself... so it's not possible to say in general when the restore/session file is updated compared to the update of the hash list (--remove see --remove-timer , default currently is approximately every 1 minute but only if there are some changes i.e. hashes cracked).

If you (@rsberzerker) instead mean when the potfile is updated compared to the hash list (with --remove), I already answered that question above:

the potfile feature normally just appends as the cracks come in

so it's basically an append lines writing mechanism of the hashcat.potfile and happens in general as soon as the cracks are displayed on the display (or, if enabled, appended/written to the --outfile)