elze / skillclusters Goto Github PK

The parser needs to be able to take new job ads from the directory specified in the IMPORT_FILE_PATH in import_settings.cfg, parse them, and update corresponding rows in the database. When it encounters a skill_term that's already in the skill_post_counters table, it needs to update number_of_postings for that skill_term. And for each secondary term it needs to update the number_of_times field in the corresponding row in the skill_pairs table.

Make the parser recognize that some skill keywords are synonyms of others: e.g. SVN is the same as Subversion, MSSQL is the same as SQL Server.

So far we have identified these synonyms. We can create a Python dict object -- a synonyms dictionary -- in the parser, and for each skill keyword (primary or secondary) we can check whether it is a key in this dictionary:

MS SQL -> SQL Server
MSSQL -> SQL Server
NodeJS -> Node
RESTful -> REST
SVN -> Subversion
TFS -> Team Foundation Server

(Similarly to NodeJS, we should look for alternative spellings of other Javascript framework names: e.g. AngularJS -> Angular, etc.)

When the parser has finished processing job files in the directory specified in the IMPORT_FILE_PATH variable in import_settings.cfg, it needs to move those files to a directory specified in the PROCESSED_FILE_PATH in variable in import_settings.cfg.