f4bd3v / humanitas Goto Github PK
View Code? Open in Web Editor NEWA price prediction toolset for developing countries
License: BSD 3-Clause "New" or "Revised" License
A price prediction toolset for developing countries
License: BSD 3-Clause "New" or "Revised" License
shown in this paper
No matter what training algorithm we used, we should implement a sort of 'bagging' by averaging over the results obtained by using multiple different reservoir initializations.
Cross-validation and bootstrapping are both methods for estimating generalization error based on "resampling". The resulting estimates of generalization error are often used for choosing among various models, such as different network architectures.
Bootstrapping seems to work better than cross-validation in many cases (Efron, 1983). In the simplest form of bootstrapping, instead of repeatedly analyzing subsets of the data, you repeatedly analyze subsamples of the data. Each subsample is a random sample with replacement from the full
sample. Depending on what you want to do, anywhere from 50 to 2000 subsamples might be used. There are many more sophisticated bootstrap methods that can be used not only for estimating generalization error but also for estimating confidence bounds for network outputs. There is more on this topic on this website.
Find if there are any supermarkets who publish their prices online. If they are not
in English, browse them through Google Translate.
If you find any, see if web.archive.org's timemachine is archiving it.
Additional words for general food category:
'snack', 'rice'?, 'groceries', 'cook'
work in progress
guys, we have to make absolutely sure we get all the tweets through filtering. I think we could still refine our approachThis pattern library is really powerful and we could run it on the tweets we write to the databaseFor filtering, we should use both suggestion as well as edit distance and PoS-tagging
keep associated PoS tags for all keywords
if a word doesn't fit any keyword, compute a suggestion and check if PoS tags match and edit distance threshold (to keyword) is satisfied, if no suggestion available just check edit distance and compare PoS tags
To make sense of the tweets we're collecting we have to cluster them according to indicators we want to feed into our Neural Networks.
The first step is to filter the tweets hierarchically according to certain categories:
general:
specific:
Price --> Food --> Commodity --> Indicator
Indicator words are "increase", "decrease", "high", "low" and their synonyms.
What are good indicators for making a prediction of a price?
The tweets we group into these categories are then ordered by their timestamps, counted and fed into the network as a sequence for each category. The scaling coefficient will have to be found empirically.
The Question is, given time constraints, do we want to implement a simple filtering or a feature-based clustering algorithm?
If we implement the latter do we use k-means clustering or Spectral clustering?
This is a fairly consistent dataset.
https://github.com/fabbrix/humanitas/blob/master/analysis/statistics/india-daily-retail/num_cities_0.4.csv
https://github.com/fabbrix/humanitas/blob/master/analysis/statistics/india-daily-retail/best_non_na_0.4.csv
Ideally, all the price data should be tabular, with the following columns:
Example of a row:
02/01/2000,week,Indonesia,Banda Aceh,milk,condensed,4000.0
Review of model and training method. Design models.
This part will be universal for other analysis and prediction, so I suggest we come up with a consensus.
Now I construct a multi-indexing dataframe from a csv file like this:
date | price | |||||
---|---|---|---|---|---|---|
product | sub | country | city | freq | ||
Rice | Common/Coarse | India | Chittoor | week | 2013-01-02 | 21.00 |
Rice | Common/Coarse | India | Guntur | week | 2013-01-02 | 24.00 |
Rice | Fine | India | Asansol | week | 2013-04-26 | 23.00 |
Rice | Fine | India | Salem | week | 2013-04-26 | 24.00 |
Where the multi-index is built on 'product', 'sub','country' ,'city' and 'freq'.
And with numexpr installed, we can extract any sub-dataframe we want like this:
sub1 = df.query('product == "Rice"')
sub2 = df.query('product == "Rice" & city == "Asansol" & sub == "Fine"')
where sub1 and sub2 are 2-column dataframes containing all "data, price" sorted by date for each predicate. Finally we can extract a certain time period like this:
after_july = sub1[sub1['date'] > '2013-07-01' ]
I'm quite new to Pandas. Any advice is welcome.
Brainstorm ideas for other places where people may discuss food prices:
News articles, reddit comments, Forums in India etc.
For the India weekly dataset, we may select our target products based on previous statistics and this table:
https://github.com/fabbrix/humanitas/blob/master/analysis/ts/na_table_org.csv
NLP, Sentiment/emotion analysis, Clustering etc.
Research the following questions:
food production (<-- weather), food stocks/distribution, currency exchange rate (w.r.t $), crude oil price
Consumer Price Index (CPI) --> inflation
Record some issues to be solved.
Rice
Has one of the data crunchers checked the csv files for indonesia? is there a special format you'd like me to apply to the data?
Before I dig into prediction, share and discuss some thoughts.
We have wholesale daily (2005-2014) and retail daily (2009-2013) datasets.
The wholesale daily dataset is sparse, but we have some very good series with more than 80%~90% of valid data in over 10 years which also appear very volatile and periodic. Although they are only tiny portions of the whole picture, I suggest we could still make good use of them to produce individual predictions.
Pre-interpolation graphs per region (zoom in or click it to see clearer graphs):
Uttar Pradesh
Apple and onion appear volatile and periodic, but we should discard the rice here, since its price is very stable.
West Bengal
Observe the periodic clustering of high volatility.
Gujarat
Super volatile potato.
Some more to come tomorrow.
Compile a list of news websites, important blogs etc. that publish/post articles on food prices, international commodity markets and food security
Guys,
Please try to follow PEP-8 (http://legacy.python.org/dev/peps/pep-0008/) as a style guide, at least the "Indentation" part (use 4 spaces per indentation level, NOT tabs), otherwise it will be the mess.
Thx.
Map of India with two layers that can be queried by date and commodity
Issue#1: Datamaps (http://datamaps.github.io/) doesn't provide a map of India so far. However it is possible to create maps with TOPOJSON; see this article on how to create a map with D3.js and TOPOJSON
Relate time series we have of different commodities to each other
Good morning team,
quick question concerning the regional scope of the analysis. Basically I'd be able to create interferences between regions, however depending on how we present our system it might not make sense. Are we going to present information to specific regions, i.e. will the user click on a part of India and receive information about commodities to that spcecific area? In that case I would exclude interferences between regions to make it coherent with the rest of the system.
Happy easter!
Alex
It is very sparse.
If we set the valid-threshold to 70% (meaning that only keeps series which has at least 70% of non-NaN data). We only get 15 (product, subproduct), and most regions have only a few (product, subproduct) data.
See the following 2 tables:
num_cities: Each cell represents the number of cities that has at least 70% valid data in that region
https://github.com/fabbrix/humanitas/blob/master/analysis/statistics/india-daily-wholesale/num_cities_0.3.csv
best_non_na: Each cell represents the max among valid percentages of cities in that region
https://github.com/fabbrix/humanitas/blob/master/analysis/statistics/india-daily-wholesale/best_non_na_0.3.csv
Even if I reduce the valid-threshold to 60%, the data is still sparse.
https://github.com/fabbrix/humanitas/blob/master/analysis/statistics/india-daily-wholesale/num_cities_0.4.csv
https://github.com/fabbrix/humanitas/blob/master/analysis/statistics/india-daily-wholesale/best_non_na_0.4.csv
The attempt to reduce time period to 3 years (2011-2014) in order to have less sparsity did not work well. The result looks very similar to the one with the whole time span.
https://github.com/fabbrix/humanitas/blob/master/analysis/statistics/india-daily-wholesale/num_cities_3y_0.4.csv
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.