Secret Sprawl: Understand It To Reduce Your Risk

Marcel Levy

In the Norwegian folk tale “The Three Billy Goats Gruff,” three goats must travel to a mountain pasture so they can eat grass and fatten up for the winter. There’s just one problem: The only path to the pasture leads over a bridge guarded by a hungry troll, who lives under the bridge and eats anyone who tries to cross.

Shared secrets are the industry’s trolls under the bridges. They’re a constant hidden risk that we have to fight with in order to achieve our actual goals. Like monsters in fairy tales, they’re found everywhere – every organization grapples with shared secret risks.

The folk tale has a happy ending: The goat siblings use brains and brawn to defeat the troll forever.

This article tells you how to defeat your own secret trolls. Read on!

Key Takeaways

  • Today’s companies manage 20-45 times more shared secrets for machines than they do for humans.
  • Many enterprise organizations have suffered data breaches due to compromised shared secrets used by non-human identities (NHIs).
  • Service providers guarantee their systems security, but customers need to guarantee the security of their NHI secrets.
  • If the shared secret is exposed, it has to be changed – and quickly.
  • Secrets managers are the traditional way to prevent secrets sprawl, but they only mitigate and don't prevent sprawl.
  • SPIRL has developed a more secure digital identity solution for each machine so no secrets are ever needed.
  • Cybersecurity teams can move from "managing secrets" to "managing identities" -- the future is identity-first infrastructure where workloads get verifiable identities instead of shared credentials.

Understanding Secrets And Where They Live

When we think of the systems that help our customers, we naturally focus on the application code. But to understand secrets, we turn that on its head and think of the connections between our applications and the other services that allow the applications to work. These services include databases like PostgreSQL or Cosmo, storage like S3, SaaS like SalesForce, AI models like Claude, or Continuous Integration/Continuous Deployment (CI/CD) systems like GitHub or Jenkins. The data in these services is what makes them valuable to businesses and their customers. It needs to be protected from attackers and other unauthorized access.

But the authorized applications need access: The applications need to prove their identity, just like humans do. We call these non-human identities, or NHIs for short.

The easiest way to do that is the oldest trick in the book: Usernames and passwords. They might be called access IDs and keys, or API keys, but they have the following things in common:

  1. They’re shared. The client presents the secret, and the service can compare that against their own records.
  2. Security is only guaranteed if the client can keep the secret safe.
  3. The secret has no expiration date – rotation is manual.
  4. How the secret is stored is up to the client.

This last point is emphasized because it’s part of the “shared responsibility” model that most services follow. The service provider guarantees the security of their systems, but we, the customers, guarantee the security of our systems – including the security of the shared secrets we use to connect to the service provider.

In practice, this means the shared secrets are stored in all kinds of places:

  1. Text files
  2. OS-specific secret managers
  3. General-purpose password managers
  4. Code repositories
  5. Even cloud-based secret managers, which, of course… require secrets to access them.  
Today’s companies manage many more shared secrets for machines than they do for humans. The estimates range from 20 times (Enterprise Strategy Group) to 45 times as many (GitGuardian). Half of the organizations surveyed by the ESG expect their total number to increase by more than 20% over the next 12 months.

What Causes Secret Sprawl?

Think of a developer building an application, or adding a feature. At this phase, she’s primarily focused on getting something to work. Let’s say she’s adding a new type of sensor to a “smart campus” system – one that tracks traffic between buildings, to help determine the optimal placement of departments and offices. Each sensor will send its telemetry to a local gateway. This gateway will then aggregate the data, and publish it to a topic in a streaming service.

And how does the gateway prove to the streaming service that it’s authorized? To put it another way, how does the gateway prove its identity? Through a shared secret of some kind. The developer isn’t thinking of security, availability, and operations yet. She has to get things working first, and there are plenty of problems in the way. 

For instance, before moving to production, she’ll have to set up a test environment. Now there are two secrets that look very similar, and easily confuse the developer. Once the new feature is in production – but before it’s launched – our developer may have to debug the production system. It’s not uncommon for the secret to end up on a developer workstation. Now we have the same secret in two places. While writing code to debug a system, it’s easy to check a secret into a code repository! We can tell people to “be careful,” but good intentions are not a security control.

Once the system is working, this secret is a critical component, like a block at the bottom of the tower in a game of Jenga. It’s difficult to change because of the high risk of failure. A lot of things can degrade a system – a software bug, not enough capacity – but if you present the wrong password, you don’t have a database. And then you don’t have a system at all.

But if the shared secret is exposed, it has to be changed – and quickly. Now we have another problem.

Why Secret Sprawl Is Dangerous

According to the ESG and GitGuardian reports mentioned earlier, most of their respondents had already suffered breaches due to compromised shared secrets used by NHIs. We already mentioned the leakage of a Grok API key, which allowed access to state-of-the-art, undeployed AI models.

Snowflake and their customers learned the hard way in 2024 that secret sprawl must be stopped. A massive data breach hit Snowflake’s customers – household names like AT&T, Neiman Marcus, and Ticketmaster. The attackers behind the breach accessed this data using legitimate credentials that they had stolen using malicious software. 

Between 2017 and 2022, personal details for thousands of Toyota customers was put at risk when a data access key was mistakenly published to a GitHub repository. Since the key was exposed for years, Toyota could not even determine whether or not any data had been accessed.

Again in 2022, Symantec researchers found that almost 2,000 iOS apps contained hardcoded AWS credentials that attackers could use to access databases and services used by the apps, again putting customer data at risk. Two years later, Symantec researchers find that the same problems persist.

The mobile app examples illustrate how difficult it is to mitigate the problem once it’s found, since it requires all the app’s customers to download an updated version of the app. This is an intractable problem, and at some point the app developer has to accept that some customers will be left with a broken app once the credentials are disabled.

In all these examples, note how easy it is to expose a secret. It just needs to be copied across the network, or to another file. These are the basic operations that make computers so useful to us!

Symptoms You Have a Secret Sprawl Problem

How do you know you’re impacted by this?

That’s easy! If your company has a large number of happy customers, and a reasonably complex IT infrastructure, and you don’t know for a fact that you’ve gotten rid of all shared secrets… Then you have a shared secret problem.

Here’s a back-of-the-envelope way to estimate the size of the problem:

  1. Perform a count of all the applications running in production.
  2. Multiply this count of applications by the average number of stages (e.g. development, staging, production) for each application. It’s probably between 2-4, so three is a good estimate.
  3. Estimate the number of services (databases, queues, object storage) used by an average application. Multiply this by the result from step 2, and you’ll have a rough estimate of shared secrets.

For example, if you have roughly 200 applications deployed, with a development and production stage for each, and each application accesses a database, S3, and a message queue, the calculation looks like this:

200 X 2 X 3 = 1200 shared secrets.

This will be an undercount, since it doesn’t include any parts of your CI/CD system that has its own set of shared secrets: System testing, synthetic monitoring, etc. It also doesn’t include the shared secrets used for management, like administrative access to a cloud provider. It also doesn’t tell you which secrets are the riskiest ones – you have no idea what kind of critical data a shared secret allows access to.

Given the sheer number of shared secrets that accompany enterprise application development, finding them in the wrong places will not be too difficult. Anything that humans or machines can read from or write to is fair game. This includes files in the filesystem, code repositories, chats between developers, and, yes, even application logs themselves. Even with years of experience, static analysis will not catch 100 percent of shared secrets. Their patterns change over time, along with the places where they’re stored.

How to Prevent Secret Sprawl

“Treat the symptom with secrets managers, or cure the disease with identities.” - Pieter Kasselman, Director Of Product Engineering

Secret managers and scanners are better than nothing at all. They’re a mitigation tool, like sandbags in a flood. But they’re not going to prevent the sprawl, and they have their limits. We have to stop the sprawl where it starts, and that means solving the original problem, which is caused by forcing applications to use tools designed for humans. 

Prevention means making it easy for developers to use the right tool for the job. For example, SPIRL gives each application its own digital identity document it can use to communicate with other applications, or present to access databases or cloud provider services. In this way, it acts like a passport does for people. This identity document is issued on the basis of information derived from the underlying operating system, hardware, or other platform. It’s cryptographically verifiable by other systems. It expires in a short period of time, in order to minimize the risk of improper use.

Digital identity documents differ from shared secrets in these important ways: 

  1. They separate identity from secret material.
  2. They don’t require the transmission of secrets.
  3. They contain an expiration date.

Two popular identity formats are X.509 certificates and JSON Web Tokens (JWTs). The most important parts of the documents are their digital signature. Just like signatures in the real world, digital signatures are a sign of approval by a trusted party. 

The choice of X.509 or JWT depends mainly on the systems (like databases, or CI/CD systems) that are already in use. An NHI system like SPIRL issues either format, so it integrates with a wide variety of existing systems. 

What is a Non-Human Identity System?

NHI systems aren’t as complex as they sound. They ensure that applications are properly identified, and can use those identities to get access to anything else that’s needed to solve the problems customers care about. That’s it!

When we talk about NHI systems, it’s helpful to know some common terms:

Workload:
This is the word we use to describe the applications that need identities. It could be a web server, a Kubernetes service, a GitHub Action, or a process running on old hardware. 

Identity:
The “name” of a workload, and what sets it apart from anything else. It’s a string of characters that has meaning to your organization. 

Issuer:
Identity documents are provided by issuers, which sign them with their secret key. Issuers also publish a public key that any other application can use to verify that an identity document is valid. 

Trust domain:
Like countries, states, or provinces are to people, trust domains are to workloads. Workloads get their identities from a trust domain, and workload identities are only accepted by applications that explicitly trust the trust domain.

Attestation:
Before an issuer signs an identity document, it makes sure that the identity matches the application. It’s a lot like requiring proof of citizenship and another form of photo id before issuing someone a passport. For NHIs, the specific process depends on the platform where the application is running.

A basic NHI system contains components that perform attestation, issue identities to workloads, and re-issue identities before expiration. It also provides public keys so other systems can validate the identity documents it issues. More advanced features include operational metrics and dashboards, as well as audit logs and other information.

Choosing the Right Non-Human Identity System for Your Stack

In the past, NHI systems have relied on vendor-specific technology, or standards-based public-key infrastructure (PKI). Regardless of the features a vendor provides, there are limits to their flexibility, and they encourage lock-in across the enterprise. PKI systems encourage interoperability, but are not for the faint of heart. 

The limitations of these approaches have led the industry to embrace modern standards like SPIFFE and WIMSE. These standards enable companies to unify NHI across their network, regardless of the underlying platform or vendor.  

When evaluating NHI systems, here are some key dimensions:

Interoperability and Flexibility

  • This includes integration with existing systems and legacy applications, including databases and the identity infrastructure on cloud providers. Future-proofing is harder because it’s speculative, but this is where you ask the NHI provider about their roadmap, and look at their involvement in industry-wide efforts, including standards bodies. 
  • Business is about more than technology. If your company makes an acquisition, how easy will it be to unify identity across the organizations? Conversely, if your company sells a line of business to someone else, what does it mean for those systems? Will it be a nightmarish money pit, or a simple handoff?

Developer Support and CI/CD

  • The developer experience is where things start to go right, or where secrets start to sprawl. How easy is it for developers to use the NHI system’s identities when they first start implementing new features or new systems? Will it be the obvious choice, or will they have to resort to using shared secrets “just to get things working?”
  • There’s a big gulf between code that runs on one developer’s computer, and production code that solves customer problems. That gulf is bridged by Continuous Integration/Continuous Deployment systems, or CI/CD for short. Not only do they need the same kinds of identities that the production systems need, but they also need their own identities to run regression tests, load tests, security scans, and more.

Availability and Scalability

  • Authentication failures tend to be complete failures. When your NHI system is down, workloads can't prove their identity, which means they can't access databases, APIs, or any other protected resources. Unlike a performance degradation, authentication failure is binary—it works or it doesn't. Look for systems with geographic distribution, automatic failover, and local caching capabilities.
  • NHI systems use short-term credentials that expire quickly—ideally within hours. This creates a scaling challenge that's different from traditional authentication systems. Your NHI system needs to handle constant renewal requests from every workload, not just occasional login attempts. If you have 1,000 workloads with certificates that expire every 4 hours, your system needs to process over 6,000 renewals per day just to maintain steady state.

Operations and Maintainability

  • Look for systems where the costs stay flat, or grow much slower than your business does. Traditional secret management creates operational debt that compounds over time—more secrets mean more rotation schedules, more emergency procedures, and more things that can break. NHI systems should save you more money per node as the system grows.
  • The goal is an NHI system that becomes invisible infrastructure—it just works, scales with your business, and doesn't wake up your on-call engineers. Key operational considerations include monitoring and observability capabilities, automated lifecycle management, integration friction, and clear upgrade paths. The best NHI systems are designed to create less operational work as you grow, not more.

Integrating Non-Human Identity Across the Development Lifecycle

The development lifecycle is where secret sprawl begins—and where it can be stopped. Traditional approaches scatter credentials across every stage, creating a trail of vulnerabilities. Non-human identity systems like SPIRL eliminate this by providing workloads with verifiable identities that follow them seamlessly through each phase. What’s more, these identities are clearly different from each other: Unlike a secret key used anywhere in the system, a developer’s local environment is distinguishable from a CI/CD workflow, which is in turn different from a production service.

Good (or Bad) Habits Start In Local Development Environments

This is ground zero for secret sprawl. Developers need access to databases, APIs, and cloud services to build and test features. At this stage, authentication is just one more obstacle to building systems. The default choice has been to copy production credentials into a .env file, and hope it doesn't get committed. But good intentions are not security control. Accidents will happen.

With an NHI system, developers authenticate using short-lived credentials. These credentials are only issued after attestation completes successfully, to make sure that the developer identity ends up in the right place. The credentials are automatically – and safely – renewed before they expire. Developers are happier because they can focus on their work, and the CISO is happier because people don’t have to “be careful” with secret material. 

Continuous Deployment, Not Continuous Breaching 

If the developer environment is where secret sprawl starts, CI/CD accelerates it, assembly-line style. GitHub Actions, Jenkins, and similar systems are secret sprawl factories. Every pipeline needs credentials to deploy code, run tests against databases, or push images to registries. These secrets are available to workflow code, or passed as environment variables. These are all potential leak points. In the best case, these secrets are stored in CI/CD secret managers, which allow the CI/CD systems to redact logs in many cases. However, it doesn’t stop developers from writing the secret to their own logs. And each CI/CD system comes with their own permissions scheme for their secret vaults. 

NHI transforms this. Instead of storing long-lived secrets, your CI/CD system presents its own identity document to access downstream services. A GitHub Action running your test suite gets a time-limited identity that proves it is authorized to access your staging database. When the action completes, the identity expires. No rotation needed. No secret storage required. 

The key insight: Your CI/CD jobs are workloads too. They deserve the same identity-based security as your production applications.

Production Deployment and Monitoring

Production is where the stakes are highest, and where static secrets cause the most damage. Applications need persistent access to databases, cloud service providers, and other services (like AI inference APIs). Traditional secret management requires complex rotation schedules, emergency revocation procedures, and constant monitoring.

With NHI your production applications get identities based on the underlying infrastructure. A Kubernetes pod gets an identity based on its namespace, service account, and pod specifications. An EC2 instance presents an identity derived from its instance metadata. It’s the same with on-premise hosts. These identities are verified in real-time and expire quickly—typically within hours.

The result? No long-lived secrets to rotate, no shared credentials to leak, and complete audit trails showing exactly which workload accessed what service and when. When a security incident occurs, you have forensic data instead of wondering who had access to that database password six months ago.

Governance and Compliance Considerations

Compliance frameworks don't just require access controls—they demand provable access controls. Static secrets make this nearly impossible. Shared secrets can be copied out of band, and proving otherwise requires diving into the code and the logs. 

Universal Audit Requirements

Whether you're dealing with SOC 2's access control requirements, HIPAA's administrative safeguards, or GDPR's integrity and confidentiality principle, the underlying need is the same: comprehensive audit trails that show who accessed what, when, and why.

Traditional secret management fails this test. A database password can be shared across applications, copied to multiple environments, and cached in various systems. When that password is used, you know something accessed your database—but you have no idea what. Was it your legitimate application, or an attacker using stolen credentials?

NHI systems provide cryptographic proof of access. Every authentication event is logged with the specific workload identity, using a signed token. When an auditor asks "Who had access to customer data on March 15th?", you can provide a precise, verifiable answer.

Access Governance That Actually Works

Compliance frameworks require organizations to implement least-privilege access, and perform regular reviews of permissions. A system using shared secrets makes this a never-ending nightmare. What does this API key allow? How many copies exist? Which applications use it? Which environments have access to it, even if they don’t use it? 

NHI systems provide real-time visibility into access patterns. You can see exactly which workloads are accessing which resources, identify over-privileged identities, and automatically revoke access for workloads that no longer exist. When a service is decommissioned, its identity expires—no manual cleanup required.

Incident Response and Forensics

When a security incident occurs, compliance frameworks require detailed incident reports showing the scope of compromise and remediation steps. Secret sprawl makes this investigation a multi-week exercise in digital archaeology. Which systems had access to the compromised credentials? How long were they exposed? What data might have been accessed?

With NHI systems, incident response becomes surgical. The audit logs show exactly which identity was used, which applications or workloads used it, and what resources were accessed. Recovery is equally precise. You revoke the specific compromised identity without affecting other workloads.

The Compliance Advantage

Organizations using NHI systems report significantly faster compliance audits and reduced remediation requirements. Instead of explaining why they can't track secret usage, they demonstrate comprehensive access governance. Instead of manual secret rotation procedures, they show automated identity lifecycle management.

The shift from "we try to manage secrets carefully" to "we don't use shared secrets" represents a fundamental improvement in security posture—one that auditors recognize and value.

This isn't about checking compliance boxes. It's about building systems that make compliance inevitable rather than accidental.

Case Study: What Happens When Secret Sprawl Goes Unchecked

Snowflake is an American cloud data storage and analysis company, and in 2024 it was discovered that 165 of their customers were hit by a data breach. Impacted companies included Ticketmaster, Advance Auto Parts, Neiman Marcus, LendingTree, and AT&T. According to a subsequent federal grand jury indictment, victim data was put up for sale on cybercrime forums, and used for extortion.

It’s important to note that Snowflake was not “hacked” – the attackers used malware to steal credentials from machines owned by the employees or contractors of Snowflake customers. The old adage was proven true again: “Hackers don't break in. They log in.” The reason it was so easy to log in with stolen credentials was because the associated accounts did not use multi-factor authentication (MFA).

And why did these accounts not use MFA? At the time Snowflake didn’t make a clear distinction between accounts used by humans and those used by applications. They were all “user” accounts. Current MFA technology is not designed for applications or services that run without human intervention. It made perfect sense for customers’ applications and services to access Snowflake without MFA.

In January 2025, Snowflake published a blog post describing some of the changes they’re making, including improvements to programmatic access. It’s a great read from a company that learned just how dangerous secret sprawl can be, and how important NHI is to their future. 

Final Thoughts: Take Control Before It’s Too Late

Secret sprawl isn't going away on its own. Every new microservice, every CI/CD pipeline, every cloud integration adds more shared secrets to your infrastructure. The math is simple: more secrets mean more risk, more operational overhead, and more things that can break in the early morning hours.

The fairy tale ending isn't about defeating the troll once. It's about building a bridge with no room for trolls.

Your Action Plan

Immediate Steps

  • Scan repositories and CI/CD systems for hardcoded credentials 
  • Inventory your secrets across all environments and applications
  • Audit access patterns to identify over-privileged secrets and unused credentials

Laying the Foundation

  • Identity the business security goals – these will drive the funding, and where to start
  • Make a list of the critical systems that will benefit the most from an NHI system
  • Understand unique concerns for all the stakeholders impacted by an NHI project

Starting the Project   

  • Evaluate NHI solutions that can eliminate static secrets entirely 
  • Pilot workload identity systems for new applications and services
  • Create a migration plan to move away from shared secrets across your entire infrastructure

The Path Forward

Traditional secret management is like treating the symptoms while ignoring the disease. You can scan for secrets, rotate credentials, and train developers—and you should—but these are defensive measures that don't address the root cause.

The fundamental issue is that we're forcing applications to authenticate like people did in the 1970s: with long-lived, shared passwords. Modern workloads deserve modern identity solutions.

The future is identity-first infrastructure. Instead of managing thousands of static secrets, your applications get cryptographically verifiable identities that prove who they are without sharing secret material. Instead of rotation schedules, you get automatic credential refresh. Instead of forensic archaeology after an incident, you get precise audit logs.

Organizations already making this shift report dramatic improvements: faster incident response, simplified compliance audits, and development teams that can focus on building features instead of managing credentials.

The question isn't whether secret sprawl will cause problems in your organization. The question is whether you'll address it proactively or reactively.

The trolls under the bridge aren't going away. But you don't have to keep feeding them. You can build something better.

SPIRL helps organizations eliminate secret sprawl by replacing shared credentials with workload identities based on open standards. It's designed for rapid deployment across existing infrastructure, with minimal disruption to development workflows.

Ready to stop managing secrets and start managing identities? The bridge to the future is already built. You just need to cross it.