ArgoCD has established itself as the leading GitOps continuous delivery tool for Kubernetes, but with widespread adoption comes significant responsibility. After years of growth and feedback from a massive user base, the ArgoCD maintainers are taking a bold step with version 3.0 - shifting from flexibility to opinionated defaults that embody security and operational best practices.

This major version is not about flashy new features. Instead, it's about taking the configurations that experienced ArgoCD administrators routinely change right after installation and making them the new defaults. It's about addressing technical debt and aligning the platform with modern Kubernetes security practices.

In this post, we'll explore the key changes in ArgoCD 3.0, with special attention to network policies and secrets management - two areas receiving significant security improvements.

Why a Major Version for Configuration Changes?

You might wonder why the maintainers are incrementing the major version for what appears to be "just" configuration changes. As explained by the ArgoCD team:

"Argo CD is such a critical project of running systems... and changing defaults was something that the contributors, the maintainers were like 'we wish we didn't have that default' and people were using that default."

With so many organizations depending on ArgoCD for critical infrastructure, even small changes to default behaviors require careful consideration. The major version number serves as a signal to users: "Watch out, there's something changing here."

Key Changes in ArgoCD 3.0

1. Resource Tracking Method

One of the most common changes administrators make immediately after installing ArgoCD is switching the resource tracking method from label to annotation or annotation plus label. Why? The label-based approach has length limitations that cause truncation issues for longer application names.

ArgoCD 3.0 makes annotation tracking the default, eliminating this common post-installation tweak.

2. Removal of Legacy Repository Configuration

ArgoCD has long supported storing repository credentials in secrets, but for backward compatibility, it also allowed storing them in config maps (argocd-cm). This practice poses a security risk as sensitive information like passwords and tokens is stored in plain text.

With 3.0, support for the config map approach is being removed entirely. As one maintainer put it:

"Storing it in a config map means you have information that should be secret, like your password and stuff, in plain text in a config map. There's no reason to do that."

This change is part of ArgoCD's broader security initiative, pushing users toward more secure practices by default.

3. Matrix/Merge Generator Defaults

The matrix and merge generators in ArgoCD will now have applyNestedSelectors: true as the default and only option. This setting was already being enabled by virtually all users, making it a prime candidate for becoming the new default.

4. A New Approach to Secrets Management

Perhaps the most significant security evolution in ArgoCD 3.0 is its stronger opinion on secrets management. Historically, ArgoCD remained neutral on how secrets should be handled, leading to a variety of approaches, some more secure than others.

Many users have been using tools like ArgoCD Vault Plugin (AVP) to inject secrets into manifests during generation. While this approach worked, it introduced several security concerns:

1. ArgoCD needs access to the secrets, increasing the risk of leakage
2. Generated manifests with injected secrets are stored in plaintext in ArgoCD's Redis cache
3. The secrets are also available via the repo-server API

The ArgoCD team is now explicitly recommending against this pattern:

"We really think you should not be doing that. That's not a secure way to do it. That's not a stable way to do it."

Instead, they're advocating for operator-based approaches using Kubernetes-native tools like:

• External Secrets Operator
• Vault Secrets Operator
• AWS Secret Operator
• CSI drivers

These approaches keep secrets management independent from application deployment and prevent secrets from passing through ArgoCD's systems.

5. Preventing In-Cluster Usage When Disabled

ArgoCD 3.0 introduces logic to prevent the usage of in-cluster configuration when it's explicitly disabled, removing a potential source of configuration conflicts and security risks.

Network Policies and the Recent Security Vulnerability

While not explicitly highlighted in the 3.0 announcement, the recent critical vulnerability (CVE-2024-31989) in ArgoCD's Redis implementation exemplifies why these security-focused changes are necessary.

The vulnerability, which received a critical score of 9.1, exposed that by default:

1. The Redis database server was not password-protected
2. Any pod within the cluster could potentially connect to the Redis server on port 6379
3. Network policies that should have restricted access weren't being properly enforced in many deployments

ArgoCD has included network policies in its codebase for some time, with separate policies for different components:

• Redis: argocd-redis-network-policy.yaml
• Repo server: argocd-repo-server-network-policy.yaml

However, many users weren't aware that:

1. Their Container Network Interface (CNI) needed to support NetworkPolicy enforcement
2. The CNI required proper configuration to enforce policies
3. These policies should be verified to ensure they're functioning

The issue was patched in May 2024 with versions 2.11.1, 2.10.10, 2.9.15, and 2.8.19, which included an initialized password for the Redis server. This additional security measure helps prevent unauthorized access even when network policies aren't properly enforced.

This vulnerability highlights perfectly why ArgoCD 3.0's shift toward secure defaults is so important. Having security features available isn't enough if they're not properly implemented and enabled by default.

Implementing Network Policies

If you're using ArgoCD and are concerned about network security, you should verify that:

1. Your Kubernetes cluster has a CNI that supports NetworkPolicy (not all do by default)
2. The default ArgoCD network policies are present and enabled:
    

   kubectl -n argocd get networkpolicy argocd-redis-network-policy

3. You've enhanced these policies as needed, potentially adding namespace restrictions

For those continuing to use generation-based secrets management (like ArgoCD Vault Plugin), the ArgoCD team recommends additional mitigations:

1. Set up network policies to prevent direct access to ArgoCD components (Redis and the repo-server)
2. Ensure your cluster supports and properly enforces these network policies
3. Consider running ArgoCD on a dedicated cluster isolated from other applications

Performance Improvements

While security is a major focus, ArgoCD 3.0 also brings performance enhancements:

1. Resource state is being removed from Kubernetes resources and stored in Redis instead, reducing "flapping" in large ArgoCD deployments
2. Improved resource update tracking to ignore expected changes from controllers like Kyverno
3. Batch processing of cluster events

Upgrade Path and Support Model

ArgoCD follows a "current minus 2" support model for security updates and critical bug fixes. This means:

1. When 3.0 is current, versions 2.14 and 2.13 will continue to receive support
2. When 3.1 released, support for 2.13 will end
3. When 3.2 released, support for 2.14 will end

The team has created tools and documentation to help identify if your current ArgoCD configuration will be affected by the 3.0 changes. While the upgrade is designed to be minimally disruptive, users are encouraged to:

1. Review the 2.14 to 3.0 upgrade documentation
2. Use the provided scripts to check for configurations that may need updates
3. Test the upgrade in a non-production environment first

Conclusion

ArgoCD 3.0 represents an important evolution in the project's approach to security and operational practices. By making the best practices the default, the maintainers are reducing the configuration burden on administrators while improving the security posture of new deployments.

The changes to network policies and secrets management are particularly important, addressing real security concerns that have been identified in production environments. While the platform still offers flexibility where needed, the more opinionated defaults will guide users toward safer, more secure implementations.

As one maintainer put it:

"This is meant to be an incremental upgrade in the exact same way that you've gone to Argo 2.13, 2.14 or those other minor version releases. What we've done in this release is we have gone through and basically distilled a lot of best practices into running Argo into the default operation of Argo CD."

For organizations using ArgoCD, now is the time to review your configurations and prepare for the upgrade. The improved defaults in 3.0 will provide a more secure, performant foundation for your GitOps workflows.

References

ArgoCD 3.0: References and Further Reading

Official Resources

1. ArgoCD GitHub Repository - The main source code repository for ArgoCD.
2. ArgoCD 3.0 Upgrade Documentation - Official guide for upgrading from 2.14 to 3.0.
3. ArgoCD Secret Management Documentation - Official documentation on secret management approaches in ArgoCD.

Security Vulnerability Information

1. CVE-2024-31989 - National Vulnerability Database entry for the ArgoCD Redis vulnerability.
2. GitHub Security Advisory: GHSA-9766-5277-j5hr - GitHub's advisory on the "Use of Risky or Missing Cryptographic Algorithms in Redis Cache" vulnerability.
3. Cycode Blog: Redis or Not - Revealing a Critical Vulnerability in Argo CD - Detailed explanation of the CVE-2024-31989 vulnerability from the security researchers who discovered it.

Network Policies in Kubernetes and ArgoCD

1. Kubernetes NetworkPolicy Documentation - Official Kubernetes documentation on network policies.
2. ArgoCD Redis Network Policy YAML - The source file for the Redis network policy in ArgoCD.
3. ArgoCD Repo Server Network Policy YAML - The source file for the repo-server network policy in ArgoCD.

Secrets Management Tools and Approaches

1. External Secrets Operator - A Kubernetes operator that integrates external secret management systems.
2. Vault Secrets Operator - HashiCorp's operator for managing secrets in Kubernetes via Vault.
3. ArgoCD Vault Plugin - Documentation for the ArgoCD Vault Plugin, which uses a generation-based approach to secrets management.

Community and Additional Resources

1. Argo Project Slack - The Argo Project Slack for community discussion and support.
2. CNCF ArgoCD Project Page - The Cloud Native Computing Foundation's page for the Argo Project.
3. OpsMx Blog: Integrating Argo CD with Kubernetes Security and Network Policies - Guide on integrating ArgoCD with security and network policies.

Performance and Scalability

22. ArgoCD Performance Optimization Guide - Documentation on optimizing ArgoCD performance.