Let’s Deploy with ayedo, Part 2: ArgoCD GitOps, Monitoring, Observability

TL;DR

• GitOps with ArgoCD anchors the desired state of your applications and infrastructure in Git, making deployments reproducible, auditable, and automatable.
• In the combined workflow of GitLab, Harbor, ArgoCD, and Kubernetes, pipelines handle building, testing, and signing, while ArgoCD manages controlled delivery to various environments.
• ArgoCD ApplicationSets enable scalable multi-environment deployment according to uniform standards—with clear guardrails, selectable automation levels, and clean rollbacks.
• Monitoring & Observability with VictoriaMetrics (metrics), VictoriaLogs (logs), and Grafana provide the technical foundation to reliably meet compliance requirements for availability, traceability, and incident handling.
• ayedo integrates this entire GitOps and observability stack into a consistent platform that combines deployment automation with compliance guardrails—including ready-to-use ArgoCD templates and dashboards.

GitOps with ArgoCD: Control Center for Platform State

What GitOps Means in Practice

GitOps is not just another buzzword, but a structuring approach:
The desired system state—deployments, configuration, policies—is versioned in Git. Automated controllers continuously check whether the actual state in the cluster matches this target image and correct deviations.

For you as a responsible person, this means:

• Every change starts with a commit or merge request.
• Reviews and approvals happen at the Git level, not ad hoc in the cluster.
• The actual state of your Kubernetes clusters can always be reconstructed from Git.

On the ayedo platform, ArgoCD takes on the role of this controller. It observes Git repositories and ensures that defined applications in the target clusters match the state described in the repository.

ArgoCD Applications vs. ApplicationSets

At its core, ArgoCD works with “Applications”—an assignment of:

• Source definition (e.g., a Helm chart or a directory in the Git repo),
• Target cluster and namespace,
• and optionally environment-specific values.

For modern platform operations with multiple environments and clusters, a single application is rarely sufficient. This is where ApplicationSets come into play:

• ApplicationSets generate many individual applications according to a common pattern.
• Sources can be, for example:
  • a list of environments,
  • a cluster registry,
  • or a directory tree in the Git repo.
• You define the deployment pattern once, and ArgoCD automatically creates a separate application for each environment or cluster.

This reduces repetition, prevents copy-paste drift, and ensures that your standards are consistently adhered to across all environments.

Automatic Sync vs. Manual Approval

ArgoCD supports two main operating modes:

1. Automated Sync (auto-sync)

   • Changes in Git are automatically rolled out to the target cluster according to defined rules.
   • Well-suited for:
     • non-critical environments (e.g., Dev, Test),
     • standardized infrastructure components with a high degree of automation.

2. Manual Approval (manual sync)

   • ArgoCD detects deviations between Git and the cluster but only rolls them out when someone actively triggers the sync.
   • Particularly suitable for:
     • production or regulated environments,
     • changes with increased risk or impact,
     • use cases where a four-eyes principle should be implemented.

On the ayedo platform, we often combine these modes:
Pre-production environments are updated automatically, while production is synchronized only after successful completion of the preceding environments and explicit approval. This keeps you agile without drowning in manual detail work.

Targeted Rollbacks Without Chaos

An underestimated advantage of GitOps: Rollbacks become a structured process again.

With ArgoCD, a rollback usually means:

• Reverting to a known chart version or a previous commit in Git.
• ArgoCD then synchronizes the cluster to this defined state.

Instead of “hotfixing” directly in the cluster, you retain:

• History and context of the change,
• Traceability for audits,
• and a clear line between the occasion (ticket/MR) and the technical change.

The Deployment Workflow: From GitLab via Harbor to ArgoCD and Kubernetes

CI Produces Artifacts, GitOps Delivers Them

In the first part of the workflow, the artifact is created:
In GitLab, tests, security scans, and image builds run. At the end, there are at least two results:

• a signed container image in Harbor,
• a versioned Helm chart—also in Harbor or in the Git repo.

At the same time, the deployment specifications (e.g., Helm values per environment) are updated in the Git repository. This combination is crucial:

• Image & Chart: What is technically rolled out.
• Git Config: Where, with which parameters, and in which environment it is rolled out.

ArgoCD as a “Pull” Mechanism

Once new versions are available, nothing needs to be actively “pushed.” Instead:

1. ArgoCD detects the change in the Git repo (or in the chart version).
2. Depending on the configuration:
   • it synchronizes automatically, or
   • marks the application as “out of sync” and waits for approval.
3. The cluster is transitioned to the new target state.

This “pull” architecture is particularly interesting for compliance:

• There is no hidden direct access of the pipeline to productive clusters.
• Access and permissions are concentrated in ArgoCD and can be cleanly controlled and audited there.
• Reviews remain anchored in the familiar GitLab workflow.

Guardrails in the GitOps Flow

Policies as an Integral Part of Deployment

Guardrails are not an afterthought but part of the design principle:
On the ayedo platform, deployments are checked along the entire chain:

• In GitLab pipelines (before pushing the chart):
  • Linting and policy checks for resource limits, security configuration, network rules.
• In the Git repository:
  • Standards for folder structure, naming, environments.
• In the GitOps layer:
  • ArgoCD projects define which clusters and namespaces can be described by which teams.
  • Optional integrations with policy engines that enforce manifest-based rules.

Practically, this means:
A non-compliant resource request (e.g., missing limits) is ideally blocked in the pipeline before ArgoCD can even see it. If a policy violation does slip through to the GitOps layer, the sync can be automatically prevented and reported there.

Multi-Environment Deployments with ArgoCD ApplicationSets

Typical Setup: Dev, Staging, Production

Many organizations today operate with at least three environments:

• Dev: high change frequency, short feedback cycles,
• Staging/Pre-Prod: production-like tests, releases “under real conditions”,
• Production: highly available, regulatory relevant, tightly controlled.

Without ApplicationSets, this often results in three nearly identical but drifting ArgoCD applications. This does not scale well.

ApplicationSets as a Pattern Library

With ApplicationSets, you define a deployment pattern once:

• Source: Your chart in Harbor or a path in the Git repo.
• Parameters: e.g., environment name, target cluster, namespaces, value files.
• Policy: Auto-sync for Dev, manual approval for Prod, tiered resource limits.

ArgoCD automatically generates:

• an application per environment,
• optionally per region or tenant.

The advantage lies less in the technology than in the effect on your organization:

• Changes to the deployment logic are made uniformly and centrally.
• New environments can be added structurally without copy-paste.
• Standards—e.g., “every production application must have resource limits and health checks”—are actually enforced.

Governance Model: Where Is Approval Granted?

In a well-set-up GitOps model, several layers interlock:

• GitLab Merge Requests:
  Technical review of the code and deployment specification.
• ArgoCD Projects and Sync Policies:
  Technical guardrails, which teams are allowed to deploy where.
• ApplicationSets:
  Ensuring that all environments are played according to the same rules.

This allows, for example, to implement that:

• Feature branches are automatically deployed to temporary Dev environments,
• Staging follows after merge and automated tests,
• Production is synchronized only after manual approval in the ArgoCD UI or via a controlled pipeline step.

Monitoring & Observability: Metrics, Logs, Dashboards, Alerts

Deployment automation is only as good as the ability to observe its effects. Especially regulatory requirements for availability, incident response, and traceability (such as under NIS2, which must be fully implemented into national law by 18.10.2024) demand a reliable basis here.

VictoriaMetrics: Metric Backbone

VictoriaMetrics takes on the role of the highly scalable metric store on the ayedo platform:

• Infrastructure metrics:
  Cluster and node status, resource consumption of pods, nodes, storage.
• Application metrics:
  Latencies, error rates, request rates, business KPIs if the application provides them.
• Platform metrics:
  State of ArgoCD, ingress controllers, service meshes, databases.

These metrics form the basis for:

• Service Level Indicators (SLIs) and Service Level Objectives (SLOs),
• Capacity planning and cost control,
• Early detection of performance problems.

VictoriaLogs: Structured Logs Instead of Blackbox

Logs are at least as relevant for audits and incident analyses as metrics. VictoriaLogs collects and indexes:

• Application logs (e.g., from containers in Kubernetes),
• Infrastructure logs (e.g., from ingress, controllers, system services),
• Security-relevant events (e.g., policy violations, failed syncs in ArgoCD).

In combination with metrics, this makes it possible to:

• Correlate peaks in error rates (metric) with specific exceptions (log),
• Trace which version of an application was active at the time of an incident,
• Provide a revision-proof audit trail of system activities.

Grafana: Shared View of System State

Grafana forms the visible layer over metrics and logs:

• Standard dashboards for:
  • Cluster state,
  • ArgoCD health and sync status,
  • central infrastructure components.
• Team-specific dashboards for:
  • individual applications,
  • business processes,
  • compliance-relevant metrics (e.g., availability SLOs).

The important thing here is not just the technology, but the interaction in your team:

• Operations and development teams work on a common data basis.
• Responsible parties see not only “it’s broken,” but can quickly assess:
  • Is a new release the cause?
  • Is there an infrastructure problem?
  • Were the compliance guardrails violated or did they intervene correctly?

Alerting: From Metric to Action

Based on VictoriaMetrics and VictoriaLogs, you can define alerts that fit your organization:

• Technical alerts:
  Error rates, utilization, pod restarts, sync errors in ArgoCD.
• Business-relevant alerts:
  Outage of certain services, exceeding response times