Chapter 8: Version Control and Model Registry Setup

# Chapter 8: Version Control and Model Registry Setup In the data‑science life cycle, *code* and *data* rarely exist in isolation. As projects grow, the sheer volume of experiments, feature‑engineering pipelines, and model artifacts can become a labyrinth if not shepherded by a disciplined version‑control strategy. This chapter lays out a pragmatic framework that blends the rigor of software engineering with the flexibility of data‑science experimentation. --- ## 1. The Why – Why Version Control for Models? | Problem | Consequence | Mitigation with Version Control | |---------|-------------|---------------------------------| | 1️⃣ **Experiment Drift** | Models become “black boxes” with no clear lineage. | Commit‑based lineage tracks every change to scripts, notebooks, and configuration files. | 2️⃣ **Reproducibility Gaps** | A model that works on a dev machine may fail on production. | Tagged commits paired with environment files (`requirements.txt`, `conda.yml`) lock dependencies. | 3️⃣ **Collaboration Bottlenecks** | Teams cannot safely merge feature‑engineering changes. | Branching models (feature, develop, master) isolate work until vetted. | 4️⃣ **Regulatory Audits** | Insufficient audit trails may violate compliance. | Git history + DVC‑tracked data form a tamper‑evident log. > **Pro Tip:** Treat your repository as a *living organism*. Frequent commits, clear commit messages, and rigorous pull‑request reviews are the equivalent of regular check‑ups. --- ## 2. Git: The Backbone of Code Control ### 2.1 Repository Structure text ├── notebooks/ # Exploratory notebooks ├── src/ # Production‑ready scripts │ ├── data/ # Data‑loading modules │ ├── features/ # Feature engineering │ ├── models/ # Model training and inference │ └── utils/ # Helpers and constants ├── tests/ # Unit and integration tests ├── mlruns/ # MLflow run artifacts (if used) ├── .gitignore ├── README.md └── requirements.txt > **Note:** Keep notebooks in a dedicated folder so they don’t clutter the main pipeline. ### 2.2 Branching Strategy - **`main`** – production‑ready models, stable code. - **`develop`** – integration of features before merging to `main`. - **`feature/*`** – isolated work on new experiments. - **`hotfix/*`** – urgent production fixes. All merges to `main` must pass CI checks (unit tests, static analysis, and a model‑evaluation step). ### 2.3 Commit Hygiene | Rule | Example | |------|---------| | 1️⃣ **Atomic Commits** | "Add LGBM baseline with 3‑fold CV" | | 2️⃣ **Verb‑Based Message** | "Refactor data loader to support streaming" | | 3️⃣ **Scope** | Keep a single feature or bug‑fix per commit. | 4️⃣ **Link to Issues** | "Fix #42: handle missing values in feature `age`" | --- ## 3. Data Versioning with DVC (Data Version Control) Data and model artifacts can be hundreds of megabytes to several gigabytes. Storing them directly in Git would bloat the repository and break performance. DVC solves this by tracking metadata in Git while keeping large files in a remote storage (S3, GCS, Azure, or a local NAS). ### 3.1 Quick Start bash # Initialize Git repo git init # Install DVC pip install dvc # Initialize DVC dvc init # Add data folder to DVC dvc add data/raw/train.csv # Commit metadata git add data/raw/train.csv.dvc .gitignore git commit -m "Add raw training data" # Push to remote storage dvc remote add -d myremote s3://mybucket/dvc dvc push ### 3.2 Pipelines DVC pipelines are defined in `dvc.yaml` and automatically track dependencies: yaml stages: preprocess: cmd: python src/data/preprocess.py deps: data/raw/train.csv outs: data/processed/train_preprocessed.csv Running `dvc repro` will re‑execute only the affected stages. --- ## 4. Model Registry: MLflow as the Central Hub A model registry is a formal place to store, version, stage‑manage, and register models. MLflow’s registry is a mature, open‑source solution that integrates with DVC and Git. ### 4.1 Core Concepts | Term | Meaning | |------|---------| | **Model Version** | Immutable snapshot of a model artifact (weights, config). | | **Stage** | Lifecycle state: *None*, *Staging*, *Production*, *Archived*. | | **Tag** | Arbitrary label (e.g., *baseline*, *A/B-test*). | | **Signature** | Input/output schema, ensuring compatibility. | ### 4.2 Registration Flow python import mlflow from mlflow.tracking import MlflowClient # 1️⃣ Log experiment mlflow.set_experiment("customer_churn") # 2️⃣ Train and log model with mlflow.start_run(): # ... training code ... mlflow.sklearn.log_model(model, "model") mlflow.log_param("n_estimators", 200) # 3️⃣ Register model client = MlflowClient() model_uri = f"runs:/{mlflow.active_run().info.run_id}/model" model_details = client.create_registered_model("ChurnClassifier") client.create_model_version("ChurnClassifier", model_uri, mlflow.active_run().info.run_id) ### 4.3 Stage Promotion bash mlflow models transition-stage \ --name ChurnClassifier \ --version 1 \ --stage Production Promoting a model to *Production* triggers automated CI hooks that can, for example, update a Kubernetes deployment. --- ## 5. Integration: From Git to Production Below is a high‑level pipeline that glues everything together: | Step | Tool | Purpose | |------|------|---------| | Code Commit | Git | Source control | | Data Commit | DVC | Data lineage | | Run | DVC | Reproducible execution | | Log | MLflow | Experiment tracking | | Register | MLflow Registry | Model versioning | | Deploy | CI/CD (GitHub Actions, GitLab CI) | Automate promotion | | Monitor | Evidently, Prometheus | Runtime checks | **Sample CI Workflow (GitHub Actions)** yaml name: ML Pipeline on: [push] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Set up Python uses: actions/setup-python@v4 with: python-version: '3.10' - name: Install dependencies run: pip install -r requirements.txt dvc mlflow - name: Pull DVC data run: dvc pull - name: Run pipeline run: dvc repro - name: Log model run: python src/train_and_log.py - name: Promote model if: github.ref == 'refs/heads/main' run: mlflow models transition-stage ... --- ## 6. Case Study: Retail Demand Forecasting **Context:** A mid‑size retailer needed to predict weekly demand for 200 SKUs. The data included sales history, promotions, weather, and holidays. | Challenge | Solution | |-----------|----------| | Large data volume (5 GB raw) | DVC stored raw data in an S3 bucket; only metadata in Git. | Frequent feature‑engineering experiments | Branches for each hypothesis; DVC pipelines re‑ran only affected stages. | Model drift due to seasonality | MLflow registry held multiple model versions; a CI pipeline automatically retrained quarterly and promoted the best. | Stakeholder confidence | Commit‑to‑commit logs were visualized in a lightweight web UI built on Streamlit, showcasing lineage and performance metrics. **Outcome:** The retailer reduced forecasting error from 12 % to 5 % and cut inventory carrying costs by 8 % within a year. --- ## 7. Common Pitfalls and How to Avoid Them | Pitfall | What to Watch For | Fix | |----------|-------------------|-----| | **Blaming Git for data chaos** | Data artifacts are committed directly. | Use DVC; never `git add` large files. | **Unclear branch policies** | Feature branches merge without tests. | Enforce PR reviews + automated test suite. | **Missing signatures in MLflow** | Deploying incompatible models. | Always log a signature; MLflow will warn on mismatch. | **Skipping environment capture** | Re‑runs fail due to version drift. | Commit `environment.yml` or `requirements.txt` with each model run. | **Over‑promoting models** | Production is unstable. | Use a staged promotion workflow with manual gate or automated A/B test. --- ## 8. Looking Ahead Version control and model registries are foundational, but the data‑science ecosystem is evolving rapidly: - **Artifact Registry as a Service** – Managed solutions (AWS S3 with SageMaker Model Registry, GCP Vertex AI, Azure ML) abstract many complexities. - **GitOps for ML** – Treating CI/CD pipelines as declarative Git repos, enabling true rollback and reproducibility. - **Metadata‑centric Observability** – Open‑source projects like Weights & Biases and Evidently extend lineages to runtime metrics. - **Federated Versioning** – Decentralized version control for data that cannot leave its jurisdiction. Staying current means not only mastering the tools but also cultivating a culture of *traceability* and *responsible deployment*. --- ## Key Takeaways 1. **Git is your backbone.** Keep code, notebooks, and experiment configs in a well‑structured repo. 2. **DVC isolates data.** Large datasets live in remote storage, while Git tracks their metadata. 3. **MLflow (or similar) is the heart of model management.** Register, version, stage, and promote models reliably. 4. **Automation is non‑negotiable.** CI/CD pipelines must enforce reproducibility, test coverage, and staged promotions. 5. **Audibility fuels trust.** A clear, immutable lineage from raw data to deployed model is essential for both compliance and stakeholder confidence. By weaving these practices together, you transform chaotic experimentation into a disciplined, auditable, and scalable data‑science operation.