hashcat Forum › Misc › User Contributions v
Dump Scraper

Thread Closed 
Threaded Mode
Dump Scraper
vladimir125 Offline
Member
***
Posts: 55
Threads: 17
Joined: Jun 2014
#1
03-19-2015, 11:22 AM (This post was last modified: 03-20-2015, 01:28 AM by vladimir125.)
As you already know, Internet is full of passwords (plain and hashed ones): when a leak occurs, usually it's posted to PasteBin.
The pace of these dumps is so high that it's not humanly possible to collect them all, so we have to rely on a bot, scraping PasteBin site for interesting filea.

Dump Monitor will exactly do this: every time some leaked information are posted on PasteBin, he will tweet the link.

Sadly Dump Monitor is not very efficient: inside its tweets you will find a lot of "false positives" (debug data, log files, Antivirus scan results) or stuff we're not interested into (RSA private keys, API keys, list of email addresses).

Moreover, once you have the raw data you need to extract such information and remove all the garbage.

That's the reason why Dump Scraper was born: inside this repository you will find several scripts to fetch the latest tweets from Dump Monitor, analyze them (discarding useless files) and extract the hashes or the passwords.

https://github.com/tampe125/dump-scraper/releases

Please remember to read the wiki before continuing:
https://github.com/tampe125/dump-scraper/wiki

Finally, this is a super-alpha release, so things may be broken or not working as expected. Moreover, I know it's a kind of "hackish": a single program with a GUI would be 100 times better. Sadly I'm running out of time and I don't know anything about Python GUI development: if anyone wants to contribute, it would be more than welcome!

Please leave here your thoughts and opinions.
Find
atom Offline
Administrator
*******
Posts: 5,185
Threads: 230
Joined: Apr 2010
#2
03-19-2015, 12:13 PM
Many thanks!
Website Find
Si2006 Offline
Member
***
Posts: 80
Threads: 10
Joined: Nov 2012
#3
03-19-2015, 08:52 PM (This post was last modified: 03-19-2015, 08:54 PM by Si2006.)
Can get it to work on ubuntu, I filled in the twitter auth keys and renamed the settings-dist.json and installed dependences.

PHP 5.5.22-1+deb.sury.org~precise+1 | Python 2.7.3

php scrape.php

PHP Warning: require_once(vendor/autoload.php): failed to open stream: No such file or directory in /home/xxxxx/dump-scraper/scrape.php on line 8
PHP Fatal error: require_once(): Failed opening required 'vendor/autoload.php' (include_path='.:/usr/share/php:/usr/share/pear') in /home/xxxxx/dump-scraper/scrape.php on line 8
Find
vladimir125 Offline
Member
***
Posts: 55
Threads: 17
Joined: Jun 2014
#4
03-19-2015, 08:56 PM
ah crap, I forgot to put that in the wiki!
You have to get composer (https://getcomposer.org/download/) and run:

php composer.phar install

sigh, that's the risk of always working on a dev environment...
However don't worry, if everything goes smooth, I think I'll release a new Python only version, with a single entry point.
Find
Si2006 Offline
Member
***
Posts: 80
Threads: 10
Joined: Nov 2012
#5
03-19-2015, 09:01 PM
That done the trick! Thanks
Find
Si2006 Offline
Member
***
Posts: 80
Threads: 10
Joined: Nov 2012
#6
03-19-2015, 09:31 PM
One more problem.

Doesn't seem to create the data folder after processing the tweets with "php scrape.php" and is also displaying a php notice in term.

Code:
processed 2000 tweets
    Found 0 removed tweets in this batch
PHP Notice:  Trying to get property of non-object in /home/xxxxx/dump-scraper/scrape.php on line 97

Notice: Trying to get property of non-object in /home/xxxxxx/dump-scraper/scrape.php on line 97
PHP Notice:  Trying to get property of non-object in /home/xxxxx/dump-scraper/scrape.php on line 100

Notice: Trying to get property of non-object in /home/xxxxx/dump-scraper/scrape.php on line 100
PHP Notice:  Trying to get property of non-object in /home/xxxxx/dump-scraper/scrape.php on line 103

Notice: Trying to get property of non-object in /home/xxxxx/dump-scraper/scrape.php on line 103
PHP Notice:  Trying to get property of non-object in /home/xxxxx/dump-scraper/scrape.php on line 103

Notice: Trying to get property of non-object in /home/xxxxx/dump-scraper/scrape.php on line 103

    processed 2001 tweets
    Found 0 removed tweets in this batch

Total processed tweets: 2001
Find
vladimir125 Offline
Member
***
Posts: 55
Threads: 17
Joined: Jun 2014
#7
03-19-2015, 09:40 PM
ignore the notice error, it seems the tweet doesn't have any data (I'll add a check for it).
Please manually create the folder data/raw

Tomorrow I'll release a new version addressing these issues...
Find
winxp5421 Offline
Junior Member
**
Posts: 32
Threads: 2
Joined: Aug 2014
#8
03-20-2015, 01:17 PM
Got everything working up till "python classify.py"

running Ubuntu 14.04, python 2.7, scipy'0.13.3' , sklearn'0.15.2'.

Error: http://pastebin.com/e2QMSmKs
Find
vladimir125 Offline
Member
***
Posts: 55
Threads: 17
Joined: Jun 2014
#9
03-20-2015, 01:20 PM
can you please post the training/features.csv file?
I think there are some invalid values inside that.
You can upload it to pastebin and put the link here.

Thank you very much!
Find
winxp5421 Offline
Junior Member
**
Posts: 32
Threads: 2
Joined: Aug 2014
#10
03-20-2015, 07:06 PM (This post was last modified: 03-20-2015, 07:11 PM by winxp5421.)
After you had mentioned that the training csv had invalid information i re looked at the Wiki and noticed that the training folder structure was "train" instead of the more logical "trash" i had a hunch this was a typeo so made the adjustment and all works fine now thanks!
http://prntscr.com/6j5lck
Find
Thread Closed