Dump Scraper

[kartan]

I have it up and running, first results are pretty good. Way better then expected. However to actually get it running it is a pain, here are my notes:

apt-get insatll php5-curl

curl -sS https://getcomposer.org/installer | php


mv settings-dist.json settings.json
(insert twitter api keys)

#### dumper-scaper is working now #####
python dumpmon-scraper.py -s 2015-03-19
#######################################

apt-get install python-numpy python-scipy python-matplotlib ipython ipython-notebook python-pandas python-sympy python-nose python-sklearn
#pip install scipy

mkdir train
mkdir data/raw/training

####
manually sort out 20 hash/plain/trash into data/raw/training/[hash/plain/trash]
####
php organize.php --train
####
python classify.py
####
php organize.php -s 2015-03-05 -u 2015-03-20
####
php extract.php -s 2015-03-05 -u 2015-03-20
####
find data/processed/plain/ -name "*.txt" -exec cat {} \; | sort -u > pastebin.dict.txt