Artifact Management for Data Science: Versioning Models and ETL Jobs with Harbor

In software development, versioning code is standard. However, in data engineering and AI projects, this is not sufficient. A model consists not only of code but also of a specific combination of training data snapshots, library dependencies (Python packages), and the weighted parameters of the model itself.

When an AI model makes an incorrect decision in production quality control, the IT department must be able to demonstrate without gaps: Which code version was running in which Container? Which versions of the libraries were installed? This is where Harbor comes into play - an enterprise-grade Container registry that is much more than just a storage location for images.

1. The Single Source of Truth for Workloads

In our Kubernetes setup, Harbor serves as the central archive for all “artifacts” of the data platform. Every time a data engineer completes a new version of an ETL pipeline (Airflow) or a model, an immutable container image is built and stored in Harbor.

• Versioning: Instead of “model_final_v2.img,” we use unique tags and digests. This ensures that exactly the image validated in the staging environment reaches production.
• Model Storage: Harbor supports OCI artifacts in addition to Docker images. This means we can securely and versioned store the pure model weights directly alongside the execution code.

2. Security Hygiene: Scanning Before Deployment

Industrial companies are primary targets for cyberattacks. Since data science stacks often use hundreds of open-source libraries, the risk of security vulnerabilities (CVEs) is high. Harbor acts as an automated security instance before deployment:

• Vulnerability Scanning: Every image is automatically scanned upon upload (e.g., with Trivy). If critical vulnerabilities are found in a Python library, Harbor can automatically block the deployment of this image to the production cluster.
• Content Trust: Through digital signatures, we ensure that only images created by our build system and not subsequently manipulated are executed in the cluster.

3. Efficiency in the Global Network

For a globally operating corporation with locations in different time zones, the speed of image pulling is crucial.

• Replication: Harbor can automatically synchronize images between different locations or cloud regions. A model developed at headquarters is available in seconds at a local plant without burdening transatlantic lines with every start.
• Garbage Collection: Since data science images often reach several gigabytes due to large libraries, Harbor automatically cleans up old, unused versions to efficiently utilize storage space on the CEPH backend.

Conclusion: Compliance Through Technical Guardrails

Artifact management with Harbor transforms the “experimental field” of data science into a professional release process. It provides the necessary audit security for regulatory requirements and proactively protects the infrastructure from vulnerabilities. For the team, this means full focus on the data while the platform guarantees the integrity and security of the results.

FAQ

Why isn’t a simple registry like Docker Hub sufficient? For industrial companies, data protection and internal governance are crucial. Harbor offers role-based access control (RBAC), integrated security scanning, and runs entirely on-premise or in your own private cloud. It also provides better integration into existing identity management systems.

Does security scanning slow down the development process? The scan usually takes only a few seconds to minutes. Compared to the risk of a security incident or production stoppage, this time investment is negligible and can be seamlessly integrated into the CI/CD pipeline.

Can we use Harbor for Helm Charts as well? Yes, Harbor is a full-fledged repository for Helm Charts. This allows not only the application (the image) but also the infrastructure description (the chart) to be managed and versioned in one central location.

How is Harbor secured in the Kubernetes cluster? Harbor itself runs as a highly available application in the cluster. The data (images and metadata) is stored on S3-compatible CEPH storage, protected against hardware failures through replication.

How does ayedo support artifact management? We implement Harbor as a fixed component of your Kubernetes platform. We configure the scan policies, set up replication rules between your locations, and train your team in the secure handling of Container artifacts.