Git Product home page Git Product logo

oec's Introduction

The Atlas of Economic Complexity Site

The Atlas is a site housing The Observatory, the master's thesis work of Alexander Simoes. This Observatory is a tool that allows users to quickly compose a visual narrative about countries and the products they exchange.

Data

The observatory provides access to bilateral trade data for roughly 200 countries, 50 years and 1000 different products of the SITC4 revision 2 classification. The source of the data we are using is:

1962 - 2000

The Center for International Data from Robert Feenstra

2001 - 2009

UN COMTRADE

Support

The Observatory will run in all modern browsers so long as they have Javascript turned on and have full support for SVG graphics. This includes the latest versions of Firefox, Chrome (Chromium), Safari (WebKit), Opera and IE.

Note: Internet Explorer versions 8 and below will not work as they do not have SVG support built in.

Adding the Observatory to computer via virtualenv

Getting The Observatory Running Locally via Virtualenv

  1. Clone from github (this will create an oec folder in the current directory)

     git clone https://github.com/alexandersimoes/oec.git
    
  2. Create the virtual environment

     virtualenv oec
    
  3. Activate this newly created environment

     . /path/to/oec/bin/activate
    
  4. Install required software

     mysql
     # if using homebrew, not install ONLY the client
     brew install mysql --client-only --universal
    
  5. Install the required Python libraries

     pip install -r requirements.txt
    
  6. Create a MySQL database on your local machine

  7. Import the latest dump of the database from 18.85.28.32/static/db/ (warning this step could take hours!)

     mysql -u username -p -h localhost DB_NAME < oec_xxxx-xx-xx.sql
    
  8. Be sure to create the following local environment variables

     export OEC_SECRET_KEY=some_s3cret_k3y
     export OEC_DB_USER=my_db_username
     export OEC_DB_PW=my_db_password
     export OEC_DB_HOST=localhost
     export OEC_DB_NAME=oec
     * export CACHE_DIR=/home/
    
     * only necessary if using filesystem caching
    
  9. Updating translations (if something is changed)

     pybabel extract -F babel.cfg -o messages.pot oec --no-location --omit-header --no-wrap
     pybabel update -i messages.pot -d oec/translations --no-wrap --no-fuzzy-matching -l [2-LETTER-LANG-CODE]
     pybabel compile -d oec/translations -l [2-LETTER-LANG-CODE] --statistics
    
  10. Running the testing server

    python run.py runserver
    

Deploying to a linux server

Step 1: Install virtual environments and pip

$ sudo apt-get install python-virtualenv python-pip git libmysqlclient-dev python-dev

Step 2: Create virtual environment

$ virtualenv oec $ source oec/bin/activate

Step 3: Pull in app from github

$ git clone https://github.com/alexandersimoes/oec.git -b v3.0 —-single-branch

Step 4: Install python requirements

$ pip install -r requirements.txt

Step 5: Install gunicorn

$ pip install gunicorn

Step 6: Set environment vars

export OEC_SECRET_KEY=yet_another_supers3cret_k35y
export OEC_DB_USER=oec_user
export OEC_DB_PW=oec_pw
export OEC_DB_HOST=localhost
export OEC_DB_NAME=oec
export CACHE_DIR=/home/macro/sites/oec/cache
export OEC_PRODUCTION=1

Step 7: Make cache directory

$ mkdir /home/macro/sites/oec/cache

Step 8: Create nginx config

$ sudo nano /etc/nginx/sites-available/oec.conf

server {
    listen 80;
    server_name 104.239.233.5;
 
    root /home/macro/sites/oec;
 
    access_log /home/macro/sites/oec/logs/access.log;
    error_log /home/macro/sites/oec/logs/error.log;

    location / {
        proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        if (!-f $request_filename) {
            proxy_pass http://127.0.0.1:8000;
            break;
        }
    }
}

$ sudo ln -s /etc/nginx/sites-available/oec.conf /etc/nginx/sites-enabled/

Step 9: Create dir for logs

$ mkdir /home/macro/sites/oec/logs

Step 10: Check nginx config

$ sudo nginx -t

Step 11: Using supervisor to autostart/manage gunicorn process

sudo apt-get install supervisor

Step 11: Create supervisor config

$ sudo nano /etc/supervisor/conf.d/oec.conf

[program:oec]
command = /home/macro/venv/oec/bin/gunicorn oec:app
directory = /home/macro/sites/oec
user = macro
stdout_logfile = /home/macro/sites/oec/logs/gunicorn.log
redirect_stderr = true
environment=PATH="/home/macro/venv/oec/bin", OEC_SECRET_KEY="oec-secret-key", OEC_DB_USER="oec_user", OEC_DB_PW="oec_pw", OEC_DB_HOST="localhost", OEC_DB_NAME="oec", CACHE_DIR="/home/macro/sites/oec/cache", OEC_PRODUCTION="1"

Step 12: Start up supervisor

$ sudo supervisorctl reread
$ sudo supervisorctl update
$ sudo supervisorctl status oec

oec's People

Contributors

alexandersimoes avatar bryant1410 avatar davelandry avatar ericjohnf avatar jspeis avatar mattrdini avatar melteng avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

oec's Issues

Pre-render geo maps for profile pages

Loading world coordinates for only showing one country is extremely browser intensive and since there's no data being shown we should pre-render these as images.

Data on Sierra Leone

I am unable to reconcile the data on exports or imports from and to Sierra Leone with what exists in the national accounts or even World Bank data. Could you please point me to the source of your data.
Thanks

Data discrepancies in SITC classification

Hello.
First of all, I would like to express my deep appreciation of your work.
Working with your data and with your interactive visualization, there is something I haven’t been able to understand. It’s about the number on export and imports when SITC classification is applied. I’ve compared these numbers with the ones of UN-COMTRADE database, where I supposed they came from, and I found huge differences. For instance, the assessment of total exports from USA in 2012 in STIC classification is $2.59T, while it’s $1.23T in HS classification.
Furthermore if you search in COMTRADE database, you can find different numbers for the same SITC code. Here you have an example.

  • In OEC site, exports from USA in 2012 in SITC classification for code 5989 (chemical products) are $36,830,122,589.25.
  • In COMTRADE If you repeat this search you will be reported a trade value of 6,057,293,127 $.*

Can anyone, please, help me to understand such a big difference? Where I’m wrong?
Thank you.

*COMTRADE search and results:
http://comtrade.un.org/api/get?max=50000&type=C&freq=A&px=S4&ps=2012&r=842&p=0&rg=2&cc=AG4

{"pfCode":"S4","yr":2012,"period":2012,"periodDesc":"2012","aggrLevel":4,"IsLeaf":0,"rgCode":2,"rgDesc":"Export","rtCode":842,"rtTitle":"USA","rt3ISO":"USA","ptCode":0,"ptTitle":"World","pt3ISO":"WLD","cmdCode":"5989","cmdDescE":"Chemical products and preparations, n.e.s.","qtCode":8,"qtDesc":"Weight in kilograms","TradeQuantity":2044214354,"NetWeight":2044214354,"TradeValue":6057293127,"estCode":0},

Canada profile data incorrect

Hello,

I just wanted to point out that the data on the Canada profile are incorrect (http://atlas.media.mit.edu/profile/country/can/).

Specifically, there appear to be a lot of products missing on the export side and the breakdown does not match official data from the Canadian government's trade databases. Notably, oil exports, the most important good export, are missing in the breakdown. You can confirm this through the following site:
https://www.ic.gc.ca/eic/site/tdo-dcd.nsf/eng/home

On another note, I suggest you provide a generic e-mail address that people can use to send comments as well. I almost did not submit this comment because I didn't feel like creating a log in.

B

Page will not load completely

When accessing the site home page, I am not able to type in anything in the search bar. And when I access a page via my history, the page will not load completely. The site keeps refreshing itself, before the page loads completely. I have tried both on Safari and Firefox, and both seem to give the same result.

Attempts to access Botswana's data end up in South Africa

When I try to access Botswana's data for 2012, 2011, 2010, and 2009, the page shows up as South Africa's. I didn't check the other years. The same thing happens when I tried to access bilateral trade data for Botswana; South Africa's show up instead.

Is this a glitch? Is Botswana's data missing?

Data discrepancy with source material (UN Comtrade)

Hi,

First of all, let me compliment you with an excellent interface/database for international trade data. Your site has enabled me (originally a history student) to gain insight into the eurasian trade and tackle an economic subject for my Master thesis.

However, in reviewing the trade flow from Russia to Armenia in 2012 I noticed something unusual. It stated that Armenia imported 'Petroleum Gas' (2711) from Russia for an amount of $168.895,86
This is a very big difference with the previous years where Petroleum Gas was the largest import product with amounts varying between 230 and 290 million dollar. I double checked with the UN Comtrade data and here I found a number of over 430 million dollar for the year 2012. Could you explain the missing data? Or maybe this number has been re-categorized?

Tree Map - Bilateral - Imports from Origin
Origin: Armenia
Destination: Russia
Trade flow: Import
Year: 2012
Classification: HS

UN Comtrade search: http://comtrade.un.org/db/dqBasicQueryResults.aspx?px=HS&cc=2711&r=51&p=643&rg=1&y=2012&so=8

Implement Hard Disk Caching

There should be some sort of hard disk caching, so that when data in Redis expires, it doesn't need to be queried from the database again (and will just move back into Redis cache).

Ranking question

Writing this here only because I didn't see a way to contact anyone on the MIT ECI site. I wanted to know why cars have a positive ECI index but planes, helicopters and spacecraft have a negative ECI index number. I looked into this because I found it weird that Brazil was ranked so much lower than Russia, knowing that Embraer exports some decent planes. (Not a Brazilian, no national pride involved, so don't worry about that. Just wondering. Also not a rocket scientist, although given there are probably a few over at MIT I wonder what they might think of this ranking?)

Thanks and Regards,

Paul Antompietri

Exports as a share of world market

In addition to being able to see exports/imports as a share of one particular country's own trade it would be great to have the possibility to visualize exports/imports of one particular country as a share of world market.

Bilateral by product

First, thanks a lot for creating this amazing tool.

When I use the feature Export or Import by product under the bilateral trade section for specific products for example export of (soybean) from Argentina I am getting this error:

The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

For some other products it is working for example cars. However when I am switching from one product to another the system is also changing the country to Afghanistan no matter what country I was looking at first. So then I have to change the country from Afghanistan to the country I am interested in in order to see the visualization.

Rankings and Profiles tabs search functions

OEC is a great instrument! Congratulations on the good work done and many thanks for sharing it with the world!!

Now to the errors:
Rankings tab is giving an internal error 500.
In the profiles tab, it is not possible to select country.

Kind regards

Peru > Germany exports/imports, Molybdenum (2613) in 2001, 2002, 2007

I find those 3 data in the OoEC as Peruvian exports to Germany, but I don't find them as imports of Germany from Peru. Any idea why? (It is vice versa in the UN Comtrade. Peruvian exports don't show these data, but German imports show).

Perhaps related to this. German exports to Peru correspond Peruvian imports from Germany (~ 684 million USD).

But while Peruvian exports to Germany is 1064 million USD, German imports from Peru are given as only 224 million USD. Interestingly copper ore, the main product imported by Germany from Peru, doesnt appear in the import data visualisations.

HS vs SITC

Hi,

I have noticed a drastic difference in numbers between HS and SITC when comparing the same product exported by a country for example export of apples from Argentina when using HS system it totals $147M in 2010 whereas when using SITC it totals $492M I am sure there should be similar discrepancies for other products as well. I understand that HS and SITC use different labeling for different products but same labeling for some products for example apples and I don't know how to explain that huge discrepancy other than error in data. I might be wrong but I wanted to point it out anyway.

thanks,

endrit

Inconsistent data for 2012 SITC Rev. 2

Last year I wrote a monograph using mostly data from the Observatory. I used SITC metadata from the Observatory to calculate share_core (J. Felipe et al. ,2012) for Brazil in different periods of time.

Now I'm updating some of the information but the SITC data has changed! Core products used to be Chemicals (SITC Rev. 2; 51–59), Metals (67 e 69), and Machinery (71–79, 87, 88, e 95).

With the 2012 data update neither HS nor SITC match the classification that I used for my calculations. Are you using later revisions now? SITC Rev. 4? There's nothing in the updates about that!

Internal server error for ranking and profiles for ECI

I was impressed by the Economic Complexity Index when I found it a few weeks ago. Trying to access the rankings and ECI score data for the countries for the different years available I now am encountered by a screen saying 'Internal Server Error The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.' Is there a way this can be fixed so I can access the ECI scores again? Is there another way I can get the data? Also I can't remember exactly which years were available before, but it did not go back as far as the data used in the report on the ECI, can this data from further back also be accessed?

Better Redis Expirations

Items in the Redis cache should have smarter, time-based expirations. Maybe there's even a possibility for different types of data have different expirations (ie. Rankings page data expires quicker, or something like that).

Please add the data manipulation scripts

Please consider posting the scripts used to generate the data from the raw sources, in addition to posting the database dumps and CSV files. A few notes re the World Trade Flows dataset (Feenstra et al. 2005):

  1. The conversion between the country codes in the WTF to ISO3 alone has separate value going well outside the scope of the Observatory.
  2. It is also not clear how you go from the WTF ???A and ???X codes to the purely numeric ones you use. (A lot of the codes in the WTF are not 'true' 4-digit SITC codes but aggregates). This is crucial to understanding the data after 2001 (e.g. when you supplement the WTF with data from COMTRADE).
  3. It seems that the totals in your dataset (calculated as sum of all flows) are rather different than the world totals as reported in the WTF (the bilateral trade flows, wtf_bilat.zip). This seems to be for 1984 onwards, where you do not have the World as a partner in the dataset. The site therefore cannot generate e.g. bilateral flows World to World.

Thanks for the good work :)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.