Ian Hopkinson 2021-06-18

This module is installed with:

pip install -e .

• configure_db
• write_to_db
• update_to_db
• drop_db_tables,
• read_db
• finalise_db,
• db_config_template

• git_uncommitted_changes
• git_sha

• write_dictionary
• pretty_print_dict
• sort_dict_by_value

• report_input_length in ETL_framework can be really slow for a large file

• Allow ihutilities to install if MySQL is not present

• Use PyMySQL as the connector to allow for updates in Python, currently most recent mysql connector is for 3.4: http://stackoverflow.com/questions/35684400/how-to-use-python-3-5-1-with-a-mysql-database

• I suspect that ETL for large files is slow if a primary key is specified up front because the index is built continuously

• Use setup and teardown on a per test basis for test_db_utils

• NROSH snapshot ETL using do_ETL shows that line count is misleading where we are dropping rows

• At the moment the _id in elasticsearch is autoincrementing, we should be able to explicitly set it (think there is a setting for this)

• read_es could do with returning search metadata

• Build_cache should allow you to easily calculate total run time

• read_db should take templated queries (using the appropriate placeholder and data supplied separately)

• read_db should only allow select statements

• Extend testing to io_utils

• Add in some ngram functionality

• Currently using scripts for survey_csv, could use console scripts: http://python-packaging.readthedocs.org/en/latest/command-line-scripts.html