If you’ve spent any time in modern DevOps circles, you’ve heard the siren song of GitOps. The promise is seductively simple: declare your entire infrastructure and application state in version-controlled code, and let an automated operator reconcile reality with your desired state. It’s the pinnacle of declarative infrastructure—clean, auditable, and seemingly infallible. But step outside the pristine demo environment and into the chaotic reality of production, and the cracks begin to show. The truth is, for many real-world, complex systems, pure GitOps is broken. It fails not because the idea is bad, but because it often ignores the messy, stateful, and human-centric nature of actual operations.
The Core Illusion: The World is Declarative
GitOps rests on a foundational assumption: that all systems can be perfectly described in a declarative state and that converging to that state is always safe and deterministic. This is the immutable infrastructure fantasy. For greenfield microservices in Kubernetes? It often works. For a legacy database cluster, a stateful streaming pipeline, or a network appliance with a quirky CLI? The illusion shatters.
Declarative models struggle with emergent state—configuration that is generated or modified by the system itself, outside the Git commit. Think of a database’s auto-increment settings, a cloud service’s auto-generated security keys, or a cache cluster’s self-optimized parameters. A purely declarative GitOps tool sees this divergence and, in its zeal for conformity, might try to “fix” it, leading to catastrophic reconfiguration or data loss.
Where the Wheels Fall Off: Real-World Fracture Points
The theory of GitOps is elegant. The practice is often a series of frustrating workarounds. Here are the key points of failure.
1. The Secret Zero Problem: Bootstrapping and Credentials
GitOps requires a controller (like Argo CD or Flux) to have the power to deploy everything. But how does that controller get deployed? How does it authenticate to your Git repo and your cloud? You need a “secret zero”—an initial credential or configuration outside the GitOps loop. This creates a privileged, manual bootstrap process that is often poorly documented and a single point of failure. The very system designed to eliminate manual toil starts with a manual, magical step.
2. Drift Detection vs. Drift Reality
GitOps champions drift detection. The controller constantly compares Git with reality and alerts on differences. In theory, this is great. In practice, it leads to alert fatigue and ignored alarms. Necessary hotfixes applied directly to production (to mitigate an incident) immediately cause “drift.” Teams are then forced to either back-port the fix to Git (slow) or sync the tool back to reality (defeating the purpose). Soon, the “drift detected” alert becomes background noise.
3. The Monolithic Repository Anti-Pattern
To have a “single source of truth,” teams are often pushed towards a monolithic repository containing all manifests for all environments and services. This creates a massive coordination bottleneck. A simple change to one service requires a PR against a repo with thousands of unrelated files. Merge conflicts become a daily nightmare. While you can use multiple repos, you then lose the atomic cross-service view and complicate dependency management, trading one problem for another.
4. The Slow Feedback Loop of the Git Pipeline
GitOps inserts a mandatory Git commit and often a CI pipeline between a developer’s intent and a deployment. For a critical security patch, this is too slow. The process—PR, review, merge, sync—can take minutes to hours. When a production database is on fire, operators need imperative control now. They will SSH in and fix it, breaking the GitOps model. A process that cannot accommodate emergency response is fundamentally flawed.
5. Stateful Applications: The Kryptonite
This is the most significant failure. Declarative infrastructure assumes statelessness. Try applying a pure GitOps model to:
- Database schema migrations: Rolling back a declarative change does not automatically roll back the data transformation. This requires imperative, ordered scripts.
- Stateful storage: Changing a PersistentVolumeClaim’s storage class or size declaratively can be destructive or unsupported.
- Complex middleware: Products like Kafka or Elasticsearch have intricate, stateful internal configurations that do not map cleanly to a kubectl apply.
Teams end up writing complex, custom operators (essentially imperative code) to manage these “declarative” resources, adding staggering complexity.
Patching the Cracks: The Hybrid Reality
Successful teams aren’t practicing pure GitOps. They’re practicing pragmatic GitOps—a hybrid model that acknowledges these limitations. They use Git as the desired state, but not the only state channel.
Embrace Imperative Escape Hatches
Formalize and secure the imperative path. Instead of pretending SSH access doesn’t exist, create a controlled, audited process for it using tools like ephemeral access bastions or just-in-time privilege elevation. Document the mandatory reconciliation process: any hotfix applied imperatively *must* be followed by a Git commit. The tool serves the team, not the other way around.
Adopt a Multi-Track Deployment Strategy
Not everything belongs in the same GitOps flow.
- Track A (Fully Declarative): Stateless microservices, config maps, network policies. Perfect for GitOps.
- Track B (Operator-Managed): Stateful services (databases, caches). Use a dedicated, mature operator (e.g., Zalando’s Postgres operator) and let it manage the complexity. Git declares the *existence* of the resource, not its intricate internal state.
- Track C (Imperative/CI-Driven): Database migrations, one-off scripts, cloud resource terraforming (where tools like Terraform’s plan/apply are still superior to a blind sync).
Shift Left on Environment-Specific Configuration
Stop storing every environment’s secrets and configs in Git, even encrypted. Use a secret manager (HashiCorp Vault, AWS Secrets Manager) integrated directly with your runtime. The GitOps controller pulls manifests from Git, but the manifests reference external secrets. This reduces repo clutter and centralizes secret lifecycle management.
Use GitOps for Approval, Not Just Deployment
Reframe Git as the audit and approval trail, not the direct deployment trigger. A CI pipeline can deploy upon merge, with the Git commit serving as the immutable record of *what* was deployed and *why* (via PR comments). This maintains auditability while allowing for faster, more flexible deployment patterns.
Conclusion: Beyond the Dogma
GitOps is not a silver bullet; it’s a tool with a specific ideal use case. The dogma of pure declarative infrastructure fails when it meets the entropy of real-world systems, legacy constraints, and urgent human intervention. The goal is not to achieve perfect GitOps purity. The goal is to achieve reliable, auditable, and efficient operations.
Stop trying to force every square peg into the round GitOps hole. Adopt its strengths—version control, audit trails, and a degree of automation—but reject its rigid orthodoxy. Build a hybrid, pragmatic system that allows for both declarative elegance and imperative necessity. Use Git as your source of truth for *desired* state, but always maintain a secure, documented, and realistic path for dealing with the messy, ever-changing *actual* state. That’s not a broken process; that’s professional engineering.
