brezniczky / keyword-trends Goto Github PK

Dates need to be fixed up, they're incorrect (they should cover the same date range as on the right chart).

scrapers/ contains 4 files, one for each (source, document) combination, could be replaced by a config file and a generic configurable implementation.

The current windowed filter requires data such that the edges (half a year on each side) are cut off from the charts. However these values become less reliable, the relationships are interesting and may suggest important, albeit a little speculative trend shifts - an opacity proportionate with the filtering window size seems to be a feasible visualisation of these additional values.

(Similarly to those on the line charts. Or: some other representation of inconfidence.)

Possibly it is the GitHub explosion (if there is one) that should be accounted for at the first place. I.e. number of relevant repositories probably increases due to the increase in total number of users.

Charts are becoming telly and easier to grasp - need for a colour coding is becoming more important as it's now getting too easy to ignore subtle differences in the legend.

Scrape at least some repositories to find out what e.g. analysis means to Java developers. ElasticSearch and libdgx applications have both been found after a quick digging.

The header is missing, so it's manually duplicated, but that appears after the TOC.
Make R knit something better, or add a post-processing step somehow.

The right chart is now drawn by the base package, thus is an oddball - ggplot it.

Need to dig into IEEE's methodology, and check out how much less robust the purely GitHub based observations are against the appearance of new courses and other (perhaps unknown) events.

Use GitHub API, possibly oauth, generate finer grained results, still over 1 year windows to partially eliminate the effects of seasonality.

The major cloud providers I think in early 2016. How fast is MS coming up?

(Because it's a big copy-paste haggis at the minute, obviously.)

The context is the new diagrams and potentially new findings due to SO usage info.

Don't forget to update the server document with the notion of StackOverflow.

Apparently does very well on GitHub, despite (or spot on because of) being a wrapper around other languages in a sense.

language:"Jupyter notebook"

Looks like Ruby is invisible and/or white on the Coursera chart.

Like SO, e.g. see https://api.stackexchange.com/docs .

More data could allow to examine and extract out real trends, fit a model, examine more verbosely, predict, etc.

Modifications to the .RData are not verifiable by reviewing.

It would be easier if the gathered data was in JSON files, or returning to the csv's, as they were cached earlier, perhaps adding a master document. The volume of the data currently allows for either.

(An alternative is to create an .RData plugin for e.g. Meld and then things become a bit better but that's a bit of sci-fi for this project I guess... there may be similar formats with native GitHub/Meld diff viewer support on the other hand - another approach to take.)

For the time being, to reduce maintenance efforts. MarkDown rendering seems bogus.

It is possible to query for the number of users, given language:python and adding created:... filters.

Would be interesting to see how many have created Python/Ruby/etc. repositories over time.

Just to describe in the Server doc. the level of interest in each, to depict how cloud technology conquers the market share from the other.

After scraping this data, should turn out whether an increase in total number of GitHub users appears to be accountable for the changes in specific, e.g. "analysis"-related repository numbers.

htmlpreview.github.io does not seem to work on my mobile. Add a PDF?

• Top weekend programming languages etc.
  https://medium.com/@hoffa/the-top-weekend-languages-according-to-githubs-code-6022ea2e33e8#.47niw3ddz

• Quora on whether it's worth learning R
  https://www.quora.com/Is-learning-R-worth-it

Breakdown charts are drawn by the base plotting package and are a bit nasty at best due to the white frames. E.g. ggplot, plotrix provide alternatives, see http://stackoverflow.com/questions/5030389/getting-a-stacked-area-plot-in-r.

(Only interesting as long as I don't switch to Python etc. too soon.)