--remove not working correctly?

[Waffle]

I ran across a very unusual problem just now.

I had a list I was working on, and wanted to check it. So, I removed my current files, and extracted the passwords from the .pot file, and ran it again with --remove.

cut -c 34- list10.pot >/tmp/t1
rm list10.pot
cp list10.orig list10.txt
./hashcat -a 0 -m 0 --remove -o list10.pot list10.txt /tmp/t1

Added hashes from file list10.txt: 950007 (1 salts)

NOTE: press enter for status-screen

Input.Mode: Dict (/tmp/t1)
Index.....: 1/1 (segment), 724187 (words), 6901757 (bytes)
Recovered.: 723802/950007 hashes, 0/1 salts
Speed/sec.: 36.83k plains, 36.83k words
Progress..: 724187/724187 (100.00%)
Running...: 00:00:00:20
Estimated.: --:--:--:--
Started: Mon Jan 14 10:11:35 2013
Stopped: Mon Jan 14 10:11:57 2013


So far so good - found 723,802 passwords. Then I ran it again, and again, and again over the same list. Each time, it found more passwords!

./hashcat -a 0 -m 0 --remove -o list10.pot list10.txt /tmp/t1
Initializing hashcat v0.42 by atom with 8 threads and 32mb segment-size...

Added hashes from file list10.txt: 226205 (1 salts)

NOTE: press enter for status-screen

Input.Mode: Dict (/tmp/t1)
Index.....: 1/1 (segment), 724187 (words), 6901757 (bytes)
Recovered.: 36048/226205 hashes, 0/1 salts
Speed/sec.: 599.70k plains, 599.70k words
Progress..: 724187/724187 (100.00%)
Running...: 00:00:00:01
Estimated.: --:--:--:--
Started: Mon Jan 14 10:12:26 2013
Stopped: Mon Jan 14 10:12:29 2013
dlr@gp:~/src/hashcat$ ./hashcat -a 0 -m 0 --remove -o list10.pot list10.txt /tmp/t1
Initializing hashcat v0.42 by atom with 8 threads and 32mb segment-size...

Added hashes from file list10.txt: 190157 (1 salts)

NOTE: press enter for status-screen

Input.Mode: Dict (/tmp/t1)
Index.....: 1/1 (segment), 724187 (words), 6901757 (bytes)
Recovered.: 8168/190157 hashes, 0/1 salts
Speed/sec.: 2.00M plains, 2.00M words
Progress..: 724187/724187 (100.00%)
Running...: 00:00:00:01
Estimated.: --:--:--:--
Started: Mon Jan 14 10:12:32 2013
Stopped: Mon Jan 14 10:12:33 2013
dlr@gp:~/src/hashcat$ ./hashcat -a 0 -m 0 --remove -o list10.pot list10.txt /tmp/t1
Initializing hashcat v0.42 by atom with 8 threads and 32mb segment-size...

Added hashes from file list10.txt: 181989 (1 salts)

NOTE: press enter for status-screen

Input.Mode: Dict (/tmp/t1)
Index.....: 1/1 (segment), 724187 (words), 6901757 (bytes)
Recovered.: 3233/181989 hashes, 0/1 salts
Speed/sec.: 4.39M plains, 4.39M words
Progress..: 724187/724187 (100.00%)
Running...: 00:00:00:01
Estimated.: --:--:--:--
Started: Mon Jan 14 10:12:36 2013
Stopped: Mon Jan 14 10:12:38 2013

I can reproduce this on demand, but it works best with a large file.

[undeath]

remove only removes one instance of the hash. If it appears multiple times it will only be removed once. sort -u your hash files before running.

[Waffle]

Of course, I did try that.

sort -u list10.orig >l10.txt

NOTE: press enter for status-screen

Input.Mode: Dict (/tmp/t1)
Index.....: 1/1 (segment), 724187 (words), 6901757 (bytes)
Recovered.: 723802/950007 hashes, 0/1 salts
Speed/sec.: 38.11k plains, 38.11k words
Progress..: 724187/724187 (100.00%)
Running...: 00:00:00:19
Estimated.: --:--:--:--
Started: Mon Jan 14 13:24:42 2013
Stopped: Mon Jan 14 13:25:03 2013
gp:~/src/hashcat$ ./hashcat -a 0 -m 0 --remove -o l10.pot l10.txt /tmp/t1
Initializing hashcat v0.42 by atom with 8 threads and 32mb segment-size...

Added hashes from file l10.txt: 226205 (1 salts)

NOTE: press enter for status-screen

Input.Mode: Dict (/tmp/t1)
Index.....: 1/1 (segment), 724187 (words), 6901757 (bytes)
Recovered.: 36048/226205 hashes, 0/1 salts
Speed/sec.: 677.61k plains, 677.61k words
Progress..: 724187/724187 (100.00%)
Running...: 00:00:00:01
Estimated.: --:--:--:--
Started: Mon Jan 14 13:25:27 2013
Stopped: Mon Jan 14 13:25:29 2013

[atom]

Can you please pack all the files required for reproduce and send me?

[Waffle]

Done yesterday in a private message.

My guess, based on further experiments, is a race condition. Using -n 1 seems to make it happen less often (or at all). Using -n 16 (on my hardware) makes it happen more often.

[atom]

Yeah, very likely. Didnt had the time yet to look into it. Thanks for the files!

[msalman]

i was going to open this error topic
the --remove is not working with me also in hashcat-0.42.7z tested on server 2008
but it was working in hashcat-0.41.7z which is not working not says outdate

32 --remove -m 1000 hash.txt pass/pass1.txt -o cracked.txt -a 0
32 --remove -m 1000 hash.txt pass/pass1.txt -o cracked.txt -a 0

32 -m 1000 hash pass/pass1.txt -o cracked.txt -a 0 --remove
32 -m 1000 hash pass/pass1.txt -o cracked.txt -a 0 --remove

non of them works the hash which is cracked by pass1 didnt removed from hash.txt and pass2 also crack same hash
and i end up with this in cracked.txt two times same hash cracked


f011b3f34c5b6c9580e2afd21f2e7367a$$word1
f011b3f34c5b6c9580e2afd21f2e7367a$$word1
fix this error in new version thx

[atom]

did you sort -u your hashlist before? at this time there is -no- known error for --remove

[msalman]

thanks