As much as you can organize your code in such as way it allow easy collaboration and usability. It is recommended to organise the code based on the part of the pipeline you are creating such as (data, training, prediction, etc.). It consits the following folder structures.

artefacts/
code/
dataset/

• artefacts should have all the relevent artefacts of the project.
• code should have all the relevent source-code of the project.
• datasets should have all the datasets of the project here.

This post provide a detailed guideline on how to structure and organise python code like a pro

Documentation provides the best way for organizing the code. It facilitates easy collaboration, code readability and usability. As much as you could, it recommended to provide the following aspects as part of the documentation:

• Comments: Terse descriptions of why a piece of code exists.
• Typing: Specification of a function's inputs and outputs data types, providing insight into what a function consumes and produces at a glance.

def sensor_data(seq: Sequence, max_seq_len: int = 0) -> np.ndarray:
    ...
    return data

• Docstrings: This is a meaningful description that describes the overall function or module and arguments, returns, etc.

If you're using Visual Studio Code (highly recommend), you should get the free Python Docstrings Generator extension so you can type """ under a function and then hit the Shift key to generate a template docstring. It will autofill parts of the docstring using the typing information and even exception in your code!

• Documentation: A rendered webpage that summarizes all the functions, classes, API calls, workflows, examples, etc., so we can view and traverse through the code base without actually having to look at the code just yet. You will create the documentation automatically from the codebase using the docstring provided.

There are many recommended options when it comes to packaging in Python such virtualenv or anaconda environment.

To use virtualen you have to follow the following steps:

1. Installing virtualenv (Documentation https://virtualenv.pypa.io/en/stable/)

   pip install virtualenv

   OR

   sudo apt install virtualenv

2. Create virtualenv with no libraries from your existing python installation

   python3 -m venv imr_venv or virtualenv imr_venv on windows computers.

3. Active virtualenv

   source imr_venv/bin/activate or .\imr_venv\Scripts\activate or windows machine

4. Installing packages

   pip install package_name4.

5. Generate requirements.txt. This gives the list of packages, and it’s version.

   pip freeze > requirements.txt

6. Install requirements from requirements.txt

   pip install -r requirements.txt

If you are interested on using conda environment

1. Install anaconda or miniconda

2. Create conda environment with specific python versions too.

   conda create -n imr_env python=3

3. Activate conda environment ````conda activate mr_env``

4. Install packages conda install package_name

5. Generate environment.yml. This gives the list of packages, and it’s version.

   conda env create -f environment.yml

6. To list the package, we need to use the below command.

   conda env list

Versioning your project will ensure reproducible behavior. Versioning control is a fundamental tool in most software engineering. It tracks any change done to code, allowing the user to review the entire history of changes and possibly revert to an older version.

IMR uses gitlab withich a git cloud-based versioning services. For basic understanding of versionong commands and best practises we recommdend following this tutorial and this video talk

Follow this installation instruction to install git environment in your computer.

After installation make sure yo set up use-mane and email. git config --global user.name "imr_user"

git config --global user.email [email protected]

It is important to version the data and artfects associated with your project. We will use the Data Version Control (DVC) library for it's simplicity, rich features and most importantly modularity. DVC interact perfectly with Git.

DVC has lots of other useful features (metrics, experiments, etc.) so be sure to explore those as well. Versioning your project will ensure reproducible behavior.

To use DVC

1. Install DVC

   pip install dvc

   or download DVC version for your OS by following this link

2. Initialize an existing Git repository.

   dvc init

3. Establish where our remote storage will be. We be using the artefacts directory which won't be checked into our remote repository.

   dvc remote add -d storage artefacts

4. Add files to be tracked by DVC. This will create text pointer files for each file

   dvc add artefacts/data/projects.json

5. Push to remote storage

   dvc push

For more details on how to use DVC you can follow this tutorial and this dvc exercise