ML/modules/ml-pipeline

ML-pipeline

This is a dummy ML-pipeline, consisting of:

• dvc tracking for version control (data and models)
• gto for model registry
• docs, created via sphinx (in pre-commit hooks)
• tests for unit, integration and end to end testing

Within src folder, the structure is as follows:

• multiple pipelines can be defined
  • i.e. for a product, we might require multuple pipelines do deliver a result
  • i.e. multiple models
• these models can be all tracked within the same gto model registry

To enable the virtual envrionemnt created in vscode:

• Open settings
• Search 'env'
• Under the extensions tab, there will be Venv path
• Copy the path of the '.dev_env' folder into there.
• When you select a kernel, clcik through create environment and refresh
• The virutal environment should be there

To use the docker environment for coding in VSCODE:

• Open the "pipeline" folder
• Open with remote container
• Select the Dockerfile
• Add the Git extension (for dvc)

For running experiment, everything will be cached but the workflow will be:

• dvc repro to regenerate the current experiement
• Change parameters if needed
• Use dvc exp run
• Cachec the results by using dvc push -r REMOTE_NAME
• Repeat as needed
• When happy with results, use dvc exp apply EXPERIMENT_NAME
• Use dvc pull
• Commit code