Related to evaristoc/fCC_R3#18

Many words are left stuck when they shouldn't

http://solutionfile.trendmicro.com/solutionfile/Consumer/new-web-classification.html
http://luca.ntop.org/LargeScaleWebClassification.pdf
https://link.springer.com/chapter/10.1007/978-3-642-20573-6_55
http://ieeexplore.ieee.org/document/7836714/?reload=true
https://www.slideshare.net/Pachara/webpage-classification

Problem description

There is added code to calculate ranking for EACH of the subjects. However it is inefficient: the ranking is strongly based on relevance to topic, which would hardly change. Better to load it immediately instead.

If saved in database there are many other operations that can be done directly using the existing ranking without having to download the whole database and repeat the calculations.

User Story

• (...)

Technical Specs

• The ranking is extracted directly from the database
• A different record of the data should be found that allows to make different kinds of queries over the database based on the values of the ranking

Possible Solutions

• In the Subject section of each platform add the value that will be used for ranking.
• An apart list with the ranking per subject.

Milestones

• (...)

Warnings and Notes

• By doing so, if indeed a change occurs that might affect ranking, there would not be a general solution, all the data should be reused.
• Relates to evaristoc/fCC_R3#11

References

• (...)

Problem Description

This is a control of several issues already opened and the milestone number 1.

User Story

• (...)

Technical Specs

• add the wikipedia info to the current db
• correct subject word lists
• work on the ranking per subject
• include a list of keywords based on subject relevance not a good idea
• similar platforms: implement kNN/cosine similarity for comparisons between platforms based on ranking per subject; 10 per subject
• other people's selection: identify other mentions of platforms and find similarities as above (?); 10 per subject
• update last date not at this iteration

Possible Solutions

• (...)

Milestones

• https://github.com/evaristoc/fCC_R3_DataAnalysis/milestone/1

Warnings and Notes:

• Refers to issues #6, #7, #8

References:

• (...)

http://commoncrawl.org/the-data/
http://commoncrawl.org/the-data/examples/
https://groups.google.com/forum/#!forum/common-crawl
https://nlp.stanford.edu/pubs/cluster-wsdm09.pdf
http://resources.mpi-inf.mpg.de/d5/teaching/ws09_10/socialnetworks/talks/Thomas_v_Bomhard.pdf

The best access seems to be through Elastic Map-reduce clusters at AWS. Part of the service is paid. There is a python SDK: https://github.com/commoncrawl/cc-mrjob

There is also an API (:tada:!!) https://github.com/ikreymer/pywb/wiki/CDX-Server-API#api-reference

There is an ETL but the reason should be found.

Steps //MAKES POINT PER TABLE, PREPARE A DATA FLOW//

• Social Data Extraction: INPUT: different sources; OUTPUT: different raw data files
• Platformtable Database pre: INPUT: different raw data files, affecting only those links that are included in rules; OUTPUT: a dataset that resembles the database in different aspects, keeping some areas empty for filling in
• Crawled Platformtable Database pre: INPUT: a database-like dataset; OUTPUT: a database-like dataset with crawled data
• Platformtable Dataset for classification: INPUT: a database-like dataset with crawled data required for the classification procedure; OUTPUT: a first dataset for classification
• Crawled Platformtable Database pre with Classes: INPUT: a first dataset for classification + operational definition of classes dataset; OUTPUT: collating a full dataset with ALL the links classified, currently called botdata + Platformtable Database pre filled, including regex forms
• Crawled Platformtable Database pre with Classes and Link Comparisons: INPUT: the dataset currently known as botdata + Crawled Platformtable Database pre with Classes; OUTPUT: collating data calculated based on what was found in botdata
• Curriculum Data Extraction: INPUT: freecodecamp website or similar; OUTPUT: a dictionary per section with some data transforms + subjects, usually called cv
• Crawled Platformtable Database pre with Classes and Link Comparisons with Subject Metrics added INPUT: CV + BOW + Crawled Platformtable Database pre with Classes and Link Comparisons; OUTPUT: collating of data into the dataset pre filled with metrics of subject relevance
• ETL of Crawled Platformtable Database pre with Classes and Link Comparisons with Subject Metrics added: verify compliance before loading, OBS: also includes other tables!!
• Database file: platformstable: INPUT: ETL of Crawled Platformtable Database pre with Classes and Link Comparisons with Subject Metrics added; OUTPUT: updates in the firebase database
• Database file: fcc_subjects: INPUT: ETL of Crawled Platformtable Database pre with Classes and Link Comparisons with Subject Metrics added; OUTPUT: extraction of unique fcc_subjects as captured from the whole list (regex ones)
• Database file: cv: INPUT: CV dict file; OUTPUT: cv dict table

I have been renaming the categories to simplify their name. However that was done in the middle of the data loading so there were data with previous and data with posterior naming.

A hack was made but a more permanent solution should be made.

Evaluating our classes with other classificaitons.

• UK MWA
• learn_anything

Done by @dmesquita :

https://bl.ocks.org/dmesquita/raw/40158689cbd1e53fe47d17a6437029b7/

Problem Description

(...)

User Story

• (...)

Technical Specs

• (...)

Possible Solutions

• A simple solution of nested arrays will be selected. Although not intuitive, it will be easier to manage for firebase possibilities.

Milestones

• (...)

Warnings and Notes:

• Relates to evaristoc/fCC_R3#34

References:

• (...)

Currently they are all csv but delimiters and quotchars are not similar between them. The reason is the information being gathered is varied and hard to standardise. However a simple standard for the information gathered might be found.

Problem Description

(...)

User Story

• The top10 "Similar Platforms" are those in the same category that are similar AND have the highest ranking for the selected subject: Scat_mat x Rankingsubj_diag
• The top10 "Mentioned by" are those of any category mentioned by other users that are similar AND have the highest ranking for the selected subject: Sment_mat x Rankingsubj_diag

Technical Specs

• (...)

Possible Solutions

• Implement on the fly using Lambda functions No, at least not now. Introduce several operational and technological issues that might not be practical
• Look for a cloud service This could be a good option too: to be explored...
• Simple cosine similarity rating, likely kNN; sklearn implementation: all before data is loaded. Some reasons why the solutions above were discharged are:
  • Firebase not good for querying
  • They introduce additional, complex async behaviour and waiting
  • This project is currently based on client browser so it is slow and no much memory
  • The section for which this is implemented is not considered as critical as other areas right now

Milestones

• Prepare a simple demo: top10 "Similar Platforms" the first 10 of same category that are similar; top10 "Mentioned by" just the first 10 more similar no matter the category

Warnings and Notes:

• relates to evaristoc/fCC_R3#11, evaristoc/fCC_R3#16 and https://github.com/evaristoc/fCC_R3_DataAnalysis/milestone/1

References

• https://stackoverflow.com/questions/17627219/whats-the-fastest-way-in-python-to-calculate-cosine-similarity-given-sparse-mat

The source of status code is at the finding_tags function. It should be assigned to crawl_status.

platformstable:

• urls as text? Eg. "Code Conventions for the JavaScript Programming Language"
• urls as numbers
• localhosts?
• urls as no addressing anywhere or wrong format
• empty / imcomplete urls
• us of "wrong" characters in urls, eg. '*'
• duplicate domains (finding canonical better!)
• a standard should be found for crawl_errors showing messages like "403:Forbidden" or "521: Web server is down" or "Site not found · GitHub Page"

Problem Description

When getting data point users from any platform in platformstable, users was a empty list

User Story

• (...)

Technical Specs

• (...)

Possible Solutions

• (...)

Milestones

• (...)

Warnings and Notes:

• the problem probably a missing update in a function like completing_db_with_data_from_botandcv or similar in the main.py

References:

• (...)

currently only category_regex as category (former definition)

Relates to #14