ex2tweetwordcount's Introduction
W205 Exercise 2 Readme Sally Hong 1. Create and run EC2 instance with AMI 'ucbw205_complete_plus_postgres_PY2.7 (ami-558fc730)' m3.medium was used for this exercise. 2. Install postgres (follow the below instructions or you can can use shell script from earlier lab) $ yum install postgres $ vim /etc/yum.repos.d/CentOS-Base.repo Append 'exclude=postgresql*' to [base] and [updates] : wq # to save $ yum localinstall https://download.postgresql.org/pub/repos/yum/9.5/redhat/rhel-6-x86_64/pgdg-centos95-9.5-2.noarch.rpm $ yum install postgresql95 postgresql95-server $ ln -sf /usr/pgsql-9.5/bin/psql /usr/bin/psql #to update link to postgresql 9.5 $service postgresql-9.5 initdb #to initialize db $ service postgresql-9.5 start #to start service Edit /var/lib/pgsql/9.5/data/postgresql.conf $ vim /var/lib/pgsql/9.5/data/postgresql.conf change #listen_address = 'localhost' to listen_addresses = โ*โ Change #standard_conforming_strings = off to standard_conforming_strings = off Edit /var/lib/pgsql/9.5/data/pg_hba.conf $ vim /var/lib/pgsql/9.5/data/pg_hba.conf Append to last line host all all 0.0.0.0 0.0.0.0 md5 3. Install anaconda python # this comes with matplotlib which is used to bar plot. Alternatively python 2.7 can be installed via yum along with dependencies in the next step. $ wget https://3230d63b5fc54e62148e-c95ac804525aac4b6dba79b00b39d1d3.ssl.cf1.rackcdn.com/Anaconda2-4.0.0-Linux-x86_64.sh $ bash Anaconda2-4.0.0-Linux-x86_64.sh <Enter> yes <Enter> <Enter> yes <Enter> Restart terminal 4. Install the following if not already installed (dependencies to run exercise 2) python 2.7 a. pip b. ez_setup c. lein $ wget --directory-prefix=/usr/bin https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein $ chmod a+x /usr/bin/lein $ sudo /usr/bin/lein d. virtualenv $ pip install virtualenv e. tweepy $ pip install tweepy f. streamparse $ pip install streamparse g. psycopg2 $ pip install psycopg2 h. matplotlib (This comes with anaconda which was installed earlier) $ pip install matplotlib 5. Run 'sparse quickstart Tweetwordcount' to create folder with quickstart files 6. Copy over files from git repository to replace files in 'src' 'topologies' and the serving scripts $ git clone [email protected]:sallyhong/EX2Tweetwordcount.git 7. In postgres, create database tcount with user postgres. $ su - postgres $ psql postgres # create database tcount; postgres # \q $ exit 7. Run drop_create_table.py as this will drop table tweetwordcount and create table tweetwordcount in database tcount. $ python drop_create_table.py 8. Run 'sparse run' and press ctrl-c to break out of program If running as root, Lein will give warning that the program is being run as root. Press enter to continue running the program. $ sparse run 9. Run serving scripts a. finalresults.py Run without an option to show full results of all words and counts. Run with words you wish to query separated by a space (use double quotes around words that contain single quotes e.g. you're) to return their counts. b. histogram.py Run with argument MIN,MAX (e.g. 3,8) for a list of words and their counts between MIN and MAX inclusive sorted in ascending order by word. c. top20hist.py Will generate PNG file of bar plot of top 20 words and counts in same directory.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.