Limiting the consecutive occurrence

[Pixel]

So, how would sed delete all lines that don't have at least one or more duplicate character any where in the whole line?

This can be done using the command below, add it, slow it, love it xD:

Code:

/\(.\).*\1/!d

Thanks M@LIK, what would it be in regular expression?

I've tried (.).*\1/! and (.).*\1! but it don't work, Im guessing it has something to do with that exclamation mark at the end.

Yes Hash-IT I know it will remove too much, only doing it to get abit more of an understanding of sed and regular expression as they are very similar and powerful tools, plus just to see what happens.

I over looked this, well spotted ntk. We should include this.

Code:

/\(DCBA\|EDCB\|FEDC\|GFED\|HGFE\|IHGF\|JIHG\|KJIH\|LKJI\|MLKJ\|NMLK\|ONML\|PONM\|QPON\|RQPO\|SRQP\|TSRQ\|UTSR\|VUTS\|WVUT\|XWVU\|YXWV\|ZYXW\)/d

also it this the right command?

These kind of command slow it the most! That's why I said I can't do it in the first place (When you asked for alphabetical order rule).

I suppose I could use ULM to remove these...

According to this : "AMD dual core 3GHz" It's your CPU : (
I'm on i7 2.4, I can generate 1GB within couple of hours.

Oh dear and I thought this was a good one !! It looks like I am out of the game chaps as I cannot generate these lists in a reasonable time. I would love to know how come the new processors are so much faster !

It looks like I'm not the only one that needs to upgrade their computer in some way. hehe

Think we should scrap this one... Like M@LIK said it slows it down too much.

Code:

/\(ABCD\|BCDE\|CDEF\|DEFG\|EFGH\|FGHI\|GHIJ\|HIJK\|IJKL\|JKLM\|KLMN\|LMNO\|MNOP\|NOPQ\|OPQR\|PQRS\|QRST\|RSTU\|STUV\|TUVW\|UVWX\|VWXY\|WXYZ\)/d

---

Got another password it's QVHHRIRA got this from a friend, I have seen it so it is real. Tested it and it got striped out again, so we need to loosen the filter abit more so that there is no more than 2 instances of a duplicate character.

M@LIK could you tell me which command I need to change so it will include this?

guessing its this one...

Code:

/\(.\).*\1.*\(.\).*\2/d

[ntk]

the size for each alphabet is each about 54.9 GB, prohibit only repeating more then one time.I have n-~,t-~,k-worldlist it takes between 88min and 106min, on an Athlon(tm) II X2 270 Processor × 2 , overclocked.

"Got another password it's QVHHRIRA got this from a friend"
No worry. It is similar to our strategy of fighting the dragon by packing its head, we very much know that we miss out passwords where repeating is inevitable like when they use DOB in password, so would be 11Mar...,Jun22, or when part of DOB stands next to birth year 01-19xx, 11-19xx, Feb-2-2000xx (a prodigy of cracker) or in fullname McClDold of McCleod Donnald, or MARRichard...,FEBBono, JunNora, SepPerl etc But why worrying? Have we not known, we have strapped all special cases by using filter? we also know they are all strapped in the 20GB parts. So in doubt, run first check in the smaller parts, before commit ourself to the 2,5-time-larger dicts.

---

virgin default uses low-alpha.
Sky uses upper-alpha
BT less then 2 yrs old uses mix-alpha-numeric
Plusnet uses mix-alpha-numeric
talktalk low-alpha-numeric

Can you confirm? Do you know any more? Could that be helpful start?

---

http://ubuntuguru.wordpress.com/2007/03/...on-method/


will start over-night with zip... or will need a new HD soon ...

[arnbju]

An idea for a workaround to get sed to run on multiple cores could be: maskprocessor > sed rule1 > sed rule2 > sed rule3 > output file.

Then you would run 3 instances of sed each on its own core, shouldn't that work?

[M@LIK]

Thanks M@LIK, what would it be in regular expression?

I don't think there's one, this is like "if not", whenever the conditions are not found, delete.

Got another password it's QVHHRIRA got this from a friend, I have seen it so it is real. Tested it and it got striped out again, so we need to loosen the filter abit more so that there is no more than 2 instances of a duplicate character.

M@LIK could you tell me which command I need to change so it will include this?

guessing its this one...

Code:

/\(.\).*\1.*\(.\).*\2/d

I don't understand what you exactly mean there, but if you want to allow "QVHHRIRA", yes, delete that line.

An idea for a workaround to get sed to run on multiple cores could be: maskprocessor > sed rule1 > sed rule2 > sed rule3 > output file.

Then you would run 3 instances of sed each on its own core, shouldn't that work?

I don't think so.

[Hash-IT]

It looks like I'm not the only one that needs to upgrade their computer in some way. hehe

Cheeky git !!! ....

@M@LIK
M@LIK, your work on this thread is tremendous, well done mate keep it up. I am sure there are many people who appreciate your efforts here but none more so then Pixel and I.

Its particularly interesting to see the SED command and regular expression for each filter, very interesting.

[Pixel]

Got another password it's QVHHRIRA got this from a friend, I have seen it so it is real. Tested it and it got striped out again, so we need to loosen the filter abit more so that there is no more than 2 instances of a duplicate character.

M@LIK could you tell me which command I need to change so it will include this?

guessing its this one...

Code:

/\(.\).*\1.*\(.\).*\2/d

I don't understand what you exactly mean there, but if you want to allow "QVHHRIRA", yes, delete that line.

OK I'll explain more, what I meant was two repeat instances total any where in the whole line, so in QVHHRIRA, the two Hs is one instance and the two Rs would be the second instance.

I tried removing

Code:

/\(.\).*\1.*\(.\).*\2/d

but on "A" it gives me this...

AABBCCDD <--- this line has 4 instances
AABBCCDE <--- this line has 3 instances
AABBCCDF <--- this line has 3 instances

Code:

mp64.exe ?u?u?u?u?u?u?u?u --stop-at AAZZZZZZ | sed -f awesome_script.txt >A.txt

heres whats now in the text file...

Code:

/\(.\)\1\1/d
/\(.\).*\1.*\1/d
/\(.\).*\(.\).*\1.*\2/d
/\(.\).*\(.\).*\2.*\1/d

I hope you get me now, Im hopeless at explaining things. Thanks for you time and effort, you are a BIG help M@LIK.

[Hash-IT]

virgin default uses low-alpha.
Sky uses upper-alpha
BT less then 2 yrs old uses mix-alpha-numeric
Plusnet uses mix-alpha-numeric
talktalk low-alpha-numeric

Can you confirm? Do you know any more? Could that be helpful start?

I'll tell you what I know.

virgin default uses low-alpha.

Yes, 8 characters long.

Sky uses upper-alpha

Yes, 8 characters long.

BT less then 2 yrs old uses mix-alpha-numeric

Don't know.

Plusnet uses mix-alpha-numeric

Yes, the only genuine one I have personally seen was 63 characters and Upper + numeric.

talktalk low-alpha-numeric

No I have only seen upper + numeric 8 characters long.

will start over-night with zip... or will need a new HD soon ...

Are you sure you want to start now ? Don't you want to wait and see what the final filter turns out like ?

[Pixel]

BT less then 2 yrs old uses mix-alpha-numeric

Old and the new BTHub3 uses 10 lower HEX characters.

[Hash-IT]

BT less then 2 yrs old uses mix-alpha-numeric

Old and the new BTHub3 uses 10 lower HEX characters.

Thats interesting Pixel, thanks for the info.

Have you seen many new BTHub3's default passwords ? I am surprised they still use the same keys. I know I may be speaking a little too soon but they are almost becoming brute force-able to a wealthy and determined attacker !!

I think another filter is required !!!!

[M@LIK]

OK I'll explain more, what I meant was two repeat instances total any where in the whole line, so in QVHHRIRA, the two Hs is one instance and the two Rs would be the second instance.

Oh I get you, that wasn't very easy to do.
No more than 2 instances of a duplicate character:

Code:

/\(.\).*\1.*\(.\).*\2.*\(.\).*\3/d
/\(.\).*\1.*\(.\).*\(.\).*\2.*\3/d
/\(.\).*\1.*\(.\).*\(.\).*\3.*\2/d
/\(.\).*\(.\).*\1.*\2.*\(.\).*\3/d
/\(.\).*\(.\).*\1.*\(.\).*\2.*\3/d
/\(.\).*\(.\).*\1.*\(.\).*\3.*\2/d
/\(.\).*\(.\).*\2.*\1.*\(.\).*\3/d
/\(.\).*\(.\).*\2.*\(.\).*\1.*\3/d
/\(.\).*\(.\).*\2.*\(.\).*\3.*\1/d
/\(.\).*\(.\).*\(.\).*\1.*\2.*\3/d
/\(.\).*\(.\).*\(.\).*\1.*\3.*\2/d
/\(.\).*\(.\).*\(.\).*\2.*\1.*\3/d
/\(.\).*\(.\).*\(.\).*\2.*\3.*\1/d
/\(.\).*\(.\).*\(.\).*\3.*\1.*\2/d
/\(.\).*\(.\).*\(.\).*\3.*\2.*\1/d

Yup all these, don't forget to remove the previous rule:

Code:

/\(.\).*\(.\).*\1.*\2/d
/\(.\).*\(.\).*\2.*\1/d