String generation, how did you avoid previous outputs?

[polarathene]

With string generation such as markov chains + masks you're able to optimize the generation to more likely results first while still exhausting the keyspace. I get how to iteratively exhaust the keyspace through traditional brute force but when you're effectively starting with random permutations in that keyspace instead of iterating through it, how do you ensure that you're not generating those again during the process?

Simple example, 0-9, you generate 5, then 3 and so on, how would the generator know not to generate 5 or 3 again? It wouldn't make sense to just generate random values and check if they've already been used as that becomes very hit/miss the greater the keyspace as it narrows down,  such as last value in a billion.

[atom]

With hashcat it's solved like this: https://github.com/hashcat/hashcat/blob/...rkov_le.cl

If you want to verify, use --stdout and sort -u the output. The number of passwords in your output before and after sort -u will stay the same.

[polarathene]

With hashcat it's solved like this: https://github.com/hashcat/hashcat/blob/...rkov_le.cl

If you want to verify, use --stdout and sort -u the output. The number of passwords in your output before and after sort -u will stay the same.

Sorry, I'm having trouble making sense of what is going on there, `generate_pw()` is where the password generation is happening and it's fed inputs from the markov left/right methods?

What is C_Markov...complete/combined?
What do the css and cs names mean?

I'd like to try implement a generator myself but I'm not sure how you get around providing guesses with probability/hinting until exhaustion while avoiding that guess repeating. Like an audience in a gameshow each taking a turn to guess the winning number knowing the wrong guesses before them. Hashcat is able to do this with Markov/Masks right? I'm not sure what to search for to read/discover other approaches/implementations, I've asked a few communities with no luck so far.

I was hoping you could clarify how to approach it. Thanks for your time.

[atom]

You're lucky, there's a standalone implementation of it. It should answer all your questions. You can find it here: https://github.com/hashcat/statsprocessor