A few years ago, I was testing an Azure Function that pulled secrets from Key Vault. It worked flawlessly on my machine \u2014 every run, every time.<\/p>\n\n\n\n
Confident, I pushed the changes to the build pipeline. The deployment succeeded. But a few minutes later, the production logs screamed \u201cAuthentication failed.\u201d<\/strong><\/p>\n\n\n\n I stared at the error, confused. How could something that worked perfectly in VS Code break the moment it hit the cloud?<\/p>\n\n\n\n That was the day I learned a painful truth about identity in Azure: it\u2019s not just about whether it works \u2014 it\u2019s about who<\/em> it works for.<\/strong><\/p>\n\n\n\n Like most engineers, I started with the path of least resistance.<\/p>\n\n\n\n I\u2019d log into Azure CLI with my personal user account, run the app locally, and watch everything connect smoothly. From my perspective, that felt like \u201ctesting early.\u201d I was validating my code and the Key Vault access logic before deploying it.<\/p>\n\n\n\n But in reality, I wasn\u2019t testing what production would experience \u2014 I was testing what I<\/em> was allowed to do.<\/p>\n\n\n\n As a user, I had Contributor access. I could read, write, and even delete the vault if I wanted. My access token was a golden key. So of course my tests passed \u2014 the system wasn\u2019t proving my permissions, it was simply trusting them.<\/p>\n\n\n\n In production, that same app didn\u2019t run as me<\/em>. It ran under a Managed Identity<\/strong> with very limited Key Vault access \u2014 just enough to read secrets, not manage them.<\/p>\n\n\n\n That subtle difference was enough to break everything. The Managed Identity wasn\u2019t assigned the correct Key Vault role, and since my local tests ran under my overprivileged user identity, the issue never surfaced until it hit PROD.<\/p>\n\n\n\n That\u2019s the danger of developer-led identity testing \u2014 it feels like you\u2019re being proactive, but you\u2019re really just validating your own access, not the system\u2019s configuration.<\/p>\n\n\n\n After that outage, I stopped using my personal account for anything beyond local tinkering. I created a Service Principal (SPN)<\/strong> that mimicked the production identity\u2019s permissions \u2014 nothing more, nothing less.<\/p>\n\n\n\n It was a small change that made a huge difference. A Service Principal is like a synthetic identity. It doesn\u2019t belong to a person; it belongs to the system. It uses a client credential flow<\/em> instead of a login session, and it only has access to what you explicitly grant it.<\/p>\n\n\n\n That means when I test Key Vault access with an SPN, I\u2019m testing exactly what my app will experience \u2014 same flow, same limitations, same RBAC boundaries.<\/p>\n\n\n\n Suddenly, my early tests became meaningful. When they passed, I knew my Managed Identity in SIT or PROD would pass too. When they failed, it was for the right reason.<\/p>\n\n\n\n As my team adopted this approach, something else changed: consistency.<\/p>\n\n\n\n Before, one developer\u2019s \u201clocal success\u201d didn\u2019t guarantee another\u2019s. Our CI\/CD pipelines often failed because they ran under different credentials from what we used locally.<\/p>\n\n\n\n By switching all early access tests \u2014 like Key Vault, Storage, or SQL authentication \u2014 to use SPN credentials, we gained repeatability<\/strong>. The same pytest suite that passed on my laptop would pass on the build agent and in the SIT environment, because the identity context was identical.<\/p>\n\n\n\n It also aligned beautifully with our principle of least privilege<\/strong>. Developers didn\u2019t need high-level access just to run tests \u2014 the SPN handled that, and we could scope it tightly to the resources we needed.<\/p>\n\n\n\n Looking back, the lesson feels obvious, but it reshaped how I approach cloud testing:<\/p>\n\n\n\n That pattern gives you the best of all worlds: speed during development, accuracy in testing, and security in production.<\/p>\n\n\n\n Here\u2019s how the same test can be adapted across those three stages using Run locally after \u2705 Use Case:<\/strong> Early dev loop Run automatically during pipeline build\/test stages. Mimics production access with fixed credentials.<\/p>\n\n\n \u2705 Use Case:<\/strong> CI\/CD validation, environment readiness Run from within an Azure resource (Function, Container, or VM) to test the real identity bound to that resource.<\/p>\n\n\n \u2705 Use Case:<\/strong> Integration \/ SIT Here\u2019s the obvious part that often goes unsaid \u2014 but it\u2019s the real payoff of all this testing.<\/p>\n\n\n\n If your test successfully retrieves a secret like That last point is subtle but critical: when Key Vault returns a connection string successfully, your app can authenticate to the downstream system without touching a single line of code<\/strong>.<\/p>\n\n\n\n In other words, your test doesn\u2019t just validate secrets \u2014 it validates connectivity, configuration, and permission boundaries<\/em> all at once.<\/p>\n\n\n\n But there\u2019s an important boundary here \u2014 and it\u2019s worth making explicit.<\/p>\n\n\n\n If your Key Vault test successfully retrieves a secret, that confirms authentication and authorization to Key Vault<\/strong> \u2014 not necessarily validity of the secret\u2019s contents<\/strong>.<\/p>\n\n\n\n In other words:<\/p>\n\n\n\n \u2705 You\u2019ve proven the identity<\/strong> (SPN\/MI\/user) can access Key Vault and read the secret. That second part is intentionally out of scope<\/em> for this kind of early identity test. Those deeper checks \u2014 validating that the connection string actually connects to SQL, or that a read-only role can\u2019t perform writes \u2014 belong to a different testing layer: the integration and security QA stages<\/strong>.<\/p>\n\n\n\n This keeps the boundaries clean:<\/p>\n\n\n\n That separation means your early \u201cshift-left\u201d tests stay fast, safe, and focused \u2014 while downstream testing covers the rest of the trust chain.<\/p>\n\n\n\n The irony is that the closer I tried to \u201ctest like production,\u201d the more I relied on my own identity \u2014 until I realised that was the problem. Identity testing isn\u2019t about you<\/em>. It\u2019s about replicating who your system really is<\/em>.<\/p>\n\n\n\n So if your Key Vault, Storage, or database tests are still running under your account, enjoy that comfort while it lasts. Because one day, when PROD doesn\u2019t trust you anymore, it\u2019s not going to be your code that fails \u2014 it\u2019s your assumptions.<\/p>\n","protected":false},"excerpt":{"rendered":" A few years ago, I was testing an Azure Function that pulled secrets from Key Vault. It worked flawlessly on...<\/p>\n","protected":false},"author":1,"featured_media":93,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_import_markdown_pro_load_document_selector":0,"_import_markdown_pro_submit_text_textarea":"","footnotes":""},"categories":[3],"tags":[],"class_list":["post-92","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-engineering-systems-that-stay-up"],"yoast_head":"\nThe Developer\u2019s Shortcut<\/h2>\n\n\n\n
The False Sense of Security<\/h2>\n\n\n\n
The Shift to Service Principals<\/h2>\n\n\n\n
The Culture of Repeatable Testing<\/h2>\n\n\n\n
The Three-Stage Identity Pattern<\/h2>\n\n\n\n
Stage<\/th> Identity Used<\/th> Purpose<\/th><\/tr><\/thead> Local Dev<\/strong><\/td> Your AAD User<\/td> Iterate quickly, debug logic<\/td><\/tr> Automated Testing (CI\/CD)<\/strong><\/td> Service Principal (SPN)<\/td> Validate access, repeatably and securely<\/td><\/tr> Runtime (SIT\/PROD)<\/strong><\/td> Managed Identity (MI)<\/td> Enforce least privilege and environment parity<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n Example: Testing Across Identity Types<\/h3>\n\n\n\n
pytest<\/code> and azure-identity<\/code>.<\/p>\n\n\n\n1. Local Dev \u2014 Your AAD User<\/h4>\n\n\n\n
az login<\/code> in VS Code or CLI. Ideal for testing logic flow and secret retrieval.<\/p>\n\n\n\n# test_keyvault_user_identity.py\nimport os, pytest\nfrom azure.identity import DefaultAzureCredential\nfrom azure.keyvault.secrets import SecretClient\n\ndef test_keyvault_access_as_user():\n """Runs locally using your AAD user token (via az login)."""\n vault_name = os.getenv("KEY_VAULT_NAME", "kv-dev-app01")\n kv_uri = f"https:\/\/{vault_name}.vault.azure.net"\n credential = DefaultAzureCredential(exclude_managed_identity_credential=True)\n client = SecretClient(vault_url=kv_uri, credential=credential)\n secret = client.get_secret("db-connection-string")\n assert secret.value, "Secret retrieval failed under user identity"\n<\/pre><\/div>\n\n\n
\u26a0\ufe0f Limitation:<\/strong> May mask permission issues (too much privilege)<\/p>\n\n\n\n2. CI\/CD \u2014 Service Principal (SPN)<\/h4>\n\n\n\n
\n# test_keyvault_spn_identity.py\nimport os, pytest\nfrom azure.identity import ClientSecretCredential\nfrom azure.keyvault.secrets import SecretClient\n\ndef test_keyvault_access_as_spn():\n """Validates Key Vault access using Service Principal credentials."""\n tenant_id = os.getenv("AZURE_TENANT_ID")\n client_id = os.getenv("AZURE_CLIENT_ID")\n client_secret = os.getenv("AZURE_CLIENT_SECRET")\n vault_name = os.getenv("KEY_VAULT_NAME")\n kv_uri = f"https:\/\/{vault_name}.vault.azure.net"\n\n credential = ClientSecretCredential(tenant_id, client_id, client_secret)\n client = SecretClient(vault_url=kv_uri, credential=credential)\n secret = client.get_secret("db-connection-string")\n assert secret.value, "SPN lacks access to required secret"\n<\/pre><\/div>\n\n\n
\u26a0\ufe0f Tip:<\/strong> Scope the SPN to least privilege<\/p>\n\n\n\n3. Runtime \u2014 Managed Identity (MI)<\/h4>\n\n\n\n
\n# test_keyvault_managed_identity.py\nimport os, pytest\nfrom azure.identity import ManagedIdentityCredential\nfrom azure.keyvault.secrets import SecretClient\n\ndef test_keyvault_access_as_mi():\n """Validates Key Vault access using the resource\u2019s Managed Identity."""\n vault_name = os.getenv("KEY_VAULT_NAME", "kv-sit-app01")\n kv_uri = f"https:\/\/{vault_name}.vault.azure.net"\n credential = ManagedIdentityCredential()\n client = SecretClient(vault_url=kv_uri, credential=credential)\n secret = client.get_secret("db-connection-string")\n assert secret.value, "Managed Identity missing permission to Key Vault"\n<\/pre><\/div>\n\n\n
\u26a0\ufe0f Requirement:<\/strong> Must run inside<\/em> Azure with MI enabled<\/p>\n\n\n\nWhy This Test Matters More Than It Looks<\/h3>\n\n\n\n
db-connection-string<\/code>, it\u2019s not just proving that Key Vault access works. It\u2019s also confirming that:<\/p>\n\n\n\n\n
The Core Principle<\/h3>\n\n\n\n
\u274c You haven\u2019t yet proven that the secret value<\/strong> points to a reachable or correctly-permissioned downstream resource.<\/p>\n\n\n\n\n
Lessons Learned<\/h2>\n\n\n\n
\n
Closing Thoughts<\/h2>\n\n\n\n