Daily DevOps Mentor — 2026-08-25

Topic: Software Supply Chain Security — Sigstore, Cosign, SLSA Provenance, and Admission-Time Enforcement on Kubernetes

Daily DevOps Mentor cover — Software Supply Chain Security

1. Topic of the Day

Software supply chain security is the discipline of proving — cryptographically, not by policy document — that the artifact running in production is the exact artifact your CI pipeline built, from source you reviewed, using dependencies you intended to pull. The three pillars are provenance (what built this, from what source, using what inputs — SLSA attestations), identity (who/what signed it — Sigstore's Fulcio-issued short-lived certs bound to OIDC workload identity, not long-lived PGP keys), and verifiable transparency (an immutable, publicly auditable log of every signing event — Rekor).

It exists because the traditional trust model — "the registry says it's myapp:v1.2.3, so it must be myapp:v1.2.3" — has no cryptographic backing. Tags are mutable, registries get compromised, build servers get compromised, and dependency resolution happens automatically at build time with no human in the loop. The attacker's cheapest path to your production cluster in 2026 is not your Kubernetes RBAC or your VPC boundary — it's a maintainer's stolen npm token, an unreviewed postinstall script in a transitive dependency, or a compromised CI runner injecting a backdoor between "tests passed" and "image pushed."

This matters at production scale because supply chain compromises are self-propagating: one poisoned package can reach every service in the org simultaneously, appearing as a routine dependency bump — no perimeter control catches it because the payload arrives through the tooling everyone already trusts. Google, Microsoft, and every major cloud vendor now require SLSA Build Level 3 provenance for internally consumed artifacts. Kubernetes itself, GitHub, and Arch Linux have adopted Sigstore natively, and image-signature verification at admission time has gone from "advanced" to table-stakes for any platform team running regulated or externally-facing workloads.

Sources: Sigstore & SLSA overview — Practical DevSecOps · Sigstore/Cosign end-to-end — Medium · Kubernetes supply chain security — Spectro Cloud

2. Real Business Problem

Scenario: A fintech platform team runs 140 microservices across three EKS clusters. On May 11, 2026-style events (the real npm/PyPI campaign that hit TanStack, Mistral AI's SDKs, UiPath, and OpenSearch packages — 170+ npm packages, 404 malicious versions, in a matter of hours) a transitive dependency several layers deep in a shared internal SDK gets compromised via a stolen maintainer token. The malicious version publishes, gets pulled by three unrelated CI pipelines within the next six hours because Renovate/Dependabot auto-merges patch bumps, and ships to production before anyone notices.

Symptoms:

  • Security scanning (Trivy/Grype, CVE-based SCA) shows nothing — the package has no assigned CVE yet; it's zero-day malicious code, not a known vulnerability. Traditional SCA is structurally blind to this class of attack.
  • SBOMs exist (the org generates them for compliance) but nobody is diffing them against a trust baseline at deploy time — the SBOM is an artifact for audits, not a live gate.
  • The payload exfiltrates CI environment secrets (cloud credentials, npm tokens) during npm install, before a single test runs — by the time build logs are reviewed, the blast radius already includes CI service-account keys.
  • On-call has no way to answer the question "which of our 140 services currently run this compromised package, and are any of them internet-facing" in under a day, because there is no queryable, cryptographically-backed inventory of what's actually running.

The root cause isn't "we didn't scan" — it's that nothing in the pipeline proves what built the artifact or blocks deployment of anything that can't prove it. This is exactly the gap SLSA provenance plus Sigstore keyless signing plus admission-time verification closes: not by catching the malicious code (no scanner reliably does, in the zero-day window), but by making "this image has no verifiable provenance from our trusted build system" itself a hard deployment blocker, and by making retroactive blast-radius analysis a Rekor log query instead of a week of forensics.

3. Production Architecture

Software supply chain security architecture — Sigstore, SLSA, admission enforcement

Developer ──push──▶ CI Build (hermetic, pinned deps, SBOM + SLSA provenance)
                            │
                            ▼
                  Sigstore Trust Root
        Fulcio (OIDC → short-lived cert) ─▶ Cosign signs image + SBOM + attestation
        Rekor (append-only transparency log, Merkle-tree inclusion proof)
                            │
                            ▼
                 OCI Registry (image digest + signature + attestations as referrers)
                            │
                            ▼
        Admission Controller (Kyverno / Gatekeeper) ── verify-image policy
        checks: cosign signature valid, signer identity matches allow-list,
        Rekor inclusion proof present, SLSA level ≥ 3, SBOM attached, digest-pinned
                            │
                     pass ──┴── fail → AdmissionReview DENY, pod never scheduled
                            ▼
                    Cluster Runtime
        kubelet re-pulls by digest (immutable) · Falco/Tetragon runtime drift
        detection · nightly Cosign policy re-audit of all running workloads

Security boundaries:

  • The trust root (Fulcio + Rekor) is the hard boundary — everything downstream (admission policy, registry) is a consumer of that root, never the source of trust. Compromise the registry and an attacker can push a malicious image, but they cannot forge a valid Sigstore signature without also compromising the OIDC identity provider issuing the workload identity token, which is a fundamentally harder target (short-lived, audience-scoped, tied to a specific CI job run).
  • Admission-time enforcement (Kyverno verify-images / Gatekeeper + ratify) is the last line before a container ever executes — it is intentionally redundant with signing at build time, because the whole point of defense-in-depth here is that a compromised registry, a mis-tagged image, or a manually-kubectl apply'd manifest all get caught at the same gate regardless of how they got there.
  • GitOps policy distribution (Argo CD syncing ClusterPolicy/ConstraintTemplate CRDs) means the enforcement rules themselves are version-controlled and reviewed via PR — an attacker who gets cluster access can't quietly widen the signer allow-list without it showing up as a Git diff.

HA/DR: Rekor is a public, mirrored transparency log (Sigstore's public instance, or a self-hosted mirror for air-gapped/regulated environments) — losing connectivity to it fails closed for new deployments (no inclusion proof, no admission) but does not affect already-running workloads, which is the correct failure mode: availability of existing services over admission of new, unverifiable ones. Admission webhooks run with failurePolicy: Fail for signature checks but should be scoped tightly (namespace selectors excluding kube-system) to avoid a webhook outage taking down the entire control plane's ability to schedule anything, including its own remediation tooling.

Multi-region/multi-cluster: Each cluster runs its own Kyverno/Gatekeeper admission webhook (no cross-region synchronous dependency on the packet path), but all clusters consume the same Rekor log and the same GitOps-distributed policy — so signer trust and policy are globally consistent while enforcement stays regional and fails independently per cluster.

Why this shape: Verification has to happen at admission, not just at build/push time, because the registry-to-runtime gap is exactly where tag mutation, manual pushes, and compromised CD pipelines live. Keyless signing (Fulcio) instead of static keys removes the single biggest operational weakness of traditional image signing — key custody and rotation — by binding trust to short-lived, workload-scoped OIDC identity instead of a secret that has to be stored, rotated, and can leak.

Trade-offs: You add a hard dependency on OIDC identity provider availability during every CI run (no token, no signature) and a network dependency on Rekor during every admission (no log entry, no scheduling) — both are the intended fail-closed behavior, but teams need clear runbooks for "Sigstore/Rekor is down, how do we still ship a hotfix" (typically: a documented, audited break-glass bypass policy scoped to specific on-call identities, itself logged).

At scale (500+ services, multiple business units): Move from a single flat signer allow-list to per-namespace/per-team policy scoping via Kyverno policy exceptions, mirror Rekor internally to remove the public-log latency and rate-limit exposure from the admission hot path, and cache verification results keyed by image digest (a digest is immutable — once verified, it never needs re-verification) to keep admission latency off the pod-creation critical path.

4. Solution Design

Design decisions:

  • Keyless signing (Fulcio/OIDC) over long-lived KMS-backed keys for CI-signed artifacts — eliminates key rotation and custody as an operational burden and an attack surface; trust is delegated to the OIDC provider's short-lived token issuance, which is already a hardened, audited system in most orgs (GitHub Actions OIDC, Google Workload Identity Federation).
  • SLSA Build Level 3 as the enforced minimum for anything reaching production — hermetic builds on ephemeral, non-reusable runners with provenance generated by the build platform itself (not the build script), so a compromised build script cannot forge its own provenance.
  • Admission-time policy enforcement via Kyverno (in-cluster, no external service dependency for the policy engine itself) rather than a registry-side-only gate — catches manual kubectl apply, misconfigured CD, and stale manifests that a registry push-time check alone would miss.
  • SBOM generation (Syft, CycloneDX format) attached as a Cosign attestation on the same digest as the signature, not a separate artifact tracked in a spreadsheet — makes the SBOM queryable at the same trust boundary as the signature itself.

Alternatives considered:

  • Static long-lived signing keys (traditional Cosign/Notary v1 mode) stored in a KMS. Pros: works without any OIDC dependency, simpler mental model. Cons: key rotation is a real operational burden that teams defer, a leaked key silently signs anything indefinitely with no built-in expiry, and there's no natural binding between "this specific CI run" and "this specific signature" — you're trusting the key custody process, not a scoped identity.
  • Registry-native content trust (e.g., Docker Content Trust/Notary v1, ECR image scanning alone). Pros: no extra infrastructure. Cons: Notary v1 is effectively deprecated in favor of Notary v2/Notation (which itself increasingly interoperates with Sigstore); registry scanning alone only catches known CVEs, not provenance or identity — it answers "is this vulnerable" not "did we build this."
  • SBOM-only strategy without signing/admission enforcement. Pros: lower engineering lift, satisfies many compliance checklists as-is. Cons: an SBOM inventories what's inside an artifact but proves nothing about who produced it or whether it's the artifact that was actually reviewed — it's forensic material, not a runtime gate, and the May 2026 npm/PyPI campaigns demonstrated exactly this: SBOMs existed, and did not stop deployment.

Scalability: Digest-based verification caching keeps admission overhead near-zero after first verification of a given image; Rekor's Merkle-tree design supports inclusion-proof lookups in logarithmic time regardless of log size, so verification latency doesn't degrade as the org's total signed-artifact volume grows into the millions.

Cost implications: Sigstore's public infrastructure (Fulcio, Rekor) is free for open-source-style keyless signing; self-hosting a private instance (recommended for regulated/air-gapped environments) costs roughly one small HA cluster (3 nodes) plus object storage for the log — materially cheaper than the incident-response cost of a single supply-chain compromise reaching production.

Security implications: This closes the "unauthorized artifact reaches production" gap but does not replace SCA/CVE scanning, secrets scanning, or SAST — it's a complementary control answering a different question ("is this the artifact we built," not "is this artifact free of known vulnerabilities").

Performance implications: Admission webhook verification adds single-digit milliseconds per pod creation once digest-level caching is warm; the first verification of a new digest adds one Rekor lookup (network round-trip, typically <200ms) — negligible against pod scheduling latency budgets, but worth explicit SLOs (verify_latency_p99) so it's visible if Rekor connectivity degrades.

5. Deep Technical Walkthrough

Signing flow (build time):

  1. CI job authenticates to the OIDC provider (GitHub Actions' built-in OIDC token, scoped to repo:org/name:ref:refs/heads/main) and requests a short-lived identity token.
  2. Cosign presents that token to Fulcio, which issues an X.509 certificate valid for ~10 minutes, binding the cert's subject to the OIDC identity (repo, workflow, ref) — not to a human or a static key.
  3. Cosign generates an ephemeral signing keypair in memory, signs the image manifest digest, and immediately discards the private key — the cert (not the key) is what has to be trusted, and it's already expired by the time anyone could misuse it.
  4. The signature, the Fulcio cert, and a SLSA provenance attestation (produced by the build platform — e.g., GitHub's attest-build-provenance action or Tekton Chains) are pushed to Rekor, which returns a signed inclusion proof (a Merkle-tree membership proof plus a signed tree head) recorded permanently and publicly.
  5. Signature + attestations + inclusion proof are attached to the OCI artifact in the registry as referrers (OCI 1.1 subject field) — co-located with the image, addressable by digest.

Verification flow (admission time):

  1. kubectl apply/Argo CD sync triggers pod creation; kube-apiserver calls the Kyverno/Gatekeeper mutating+validating admission webhook.
  2. The policy engine fetches the image's signature and attestations from the registry (by digest, not tag — tags are mutable and must never be trusted for verification).
  3. Cosign verification logic checks: (a) the Fulcio cert chains to the trusted Sigstore (or private) CA, (b) the cert's OIDC identity matches a configured allow-list regex (e.g., https://github.com/org/* for workflow build-and-sign.yml), (c) Rekor has a valid inclusion proof for this exact signature at a timestamp within the cert's validity window, (d) the SLSA provenance attestation declares Build Level ≥ the policy floor, (e) an SBOM attestation is present.
  4. Any check failing → AdmissionReview.allowed = false, pod creation is rejected with a human-readable reason surfaced in the event log — this is the control point where an unsigned, wrong-signer, or provenance-less image is stopped, regardless of how it got into the manifest.

Control-plane vs. data-plane split: The admission webhook is squarely control-plane (blocks scheduling decisions); it has zero data-plane presence — once a pod is running, verification doesn't intercept traffic or syscalls (that's Falco/Tetragon's job for runtime drift, a separate control).

Failure scenarios:

  • Rekor unreachable at admission time: fail-closed by default — new deployments block; document and test the break-glass path (a scoped Kyverno policy exception, itself audited) before you need it at 2 a.m.
  • Fulcio cert expired between signing and verification: not a failure — Rekor's timestamped inclusion proof establishes the signature was valid at signing time, which is what verification checks against, not current time. This is the entire point of the transparency-log design: short-lived certs don't require long-lived trust.
  • Registry returns stale/wrong referrers (registry doesn't support OCI 1.1 referrers API): falls back to the older Cosign tag-based scheme (<digest>.sig) — verify your registry's referrers support explicitly, this is a common silent gap on older self-hosted registries.

Scaling behavior: Verification cost is dominated by network round-trips (registry fetch + Rekor lookup), both of which are independent of cluster size — the pattern scales horizontally by scaling the admission webhook replica count, not by any centralized bottleneck.

6. Production Troubleshooting

Symptom: Deployments across multiple teams start failing admission simultaneously with image verification failed: no matching signatures.

Step-by-step investigation, the way a senior platform engineer would run it:

  1. Scope the blast radius first. kubectl get events -A --field-selector reason=PolicyViolation | tail -50 (Kyverno) or the equivalent Gatekeeper constraint audit — is this one team's images or every image? A cluster-wide failure points at the trust root or the webhook itself, not individual pipelines.

  2. Check the webhook's own health. kubectl get validatingwebhookconfigurations kyverno-policy-validating-webhook-cfg -o yaml and kubectl logs -n kyverno deploy/kyverno -f — look for context deadline exceeded calling out to the registry or Rekor. A failurePolicy: Fail webhook that can't reach Rekor will reject everything, which looks identical to "everyone's signatures are suddenly invalid" from the outside.

  3. Isolate registry vs. Rekor. Manually run cosign verify --certificate-identity-regexp '...' --certificate-oidc-issuer '...' <image>@<digest> from a debug pod in-cluster. If this hangs on the Rekor call specifically, check Rekor's own status page/self-hosted instance health and egress network policy — a recently tightened NetworkPolicy on the kyverno namespace is a very common self-inflicted cause here.

  4. Check for a cert/policy allow-list drift. If only new builds fail (older, already-deployed images verify fine on redeploy), diff the current ClusterPolicy's certificate-identity regex against the CI workflow's actual OIDC subject claim — a workflow rename, a repo transfer, or a branch-protection rule change alters the OIDC subject string and silently breaks the allow-list match. cosign verify with -v will print the actual cert identity it extracted; compare it character-for-character against the policy regex.

  5. Check clock skew if using self-hosted Fulcio/Rekor. Certificate validity windows are ~10 minutes; a node or the Fulcio host itself drifting more than a couple of minutes will intermittently reject valid signatures — check chronyd/ntpd status on the signing infrastructure.

  6. Metrics/dashboards to have pre-built: kyverno_policy_results_total{result="fail"} by policy and namespace (Prometheus), cosign_verify_duration_seconds histogram, Rekor's own /api/v1/log health, and a Grafana panel correlating admission denial spikes against recent policy CRD sync commits in Argo CD — the single most common root cause in practice is a policy change, not an actual attack.

  7. Root cause classes, ranked by real-world frequency: (1) allow-list regex drift after a CI workflow rename, (2) webhook egress NetworkPolicy regression, (3) registry referrers API incompatibility after a registry migration, (4) genuine Rekor/Fulcio outage (rare, but plan for it), (5) an actual unsigned/malicious image — which is the control working as designed.

7. Hands-on Lab

Local lab using kind + Cosign + Kyverno. Validate end-to-end signing and admission-time rejection.

# 1. Create a local cluster
kind create cluster --name supplychain-lab

# 2. Install Kyverno
helm repo add kyverno https://kyverno.github.io/kyverno/ && helm repo update
helm install kyverno kyverno/kyverno -n kyverno --create-namespace

# 3. Install cosign
brew install cosign   # or: curl -O -L https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64

# 4. Build and push a test image to a local registry, then sign it keylessly
docker build -t localhost:5000/demo-app:v1 .
docker push localhost:5000/demo-app:v1
COSIGN_EXPERIMENTAL=1 cosign sign localhost:5000/demo-app:v1   # opens OIDC browser flow (or use --identity-token in CI)

# 5. Verify manually
cosign verify \
  --certificate-identity-regexp "https://github.com/YOUR_ORG/*" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  localhost:5000/demo-app:v1

# 6. Apply a Kyverno policy requiring valid signatures
cat <<'EOF' | kubectl apply -f -
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-image-signatures
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  rules:
  - name: verify-signature
    match:
      any:
      - resources:
          kinds: [Pod]
    verifyImages:
    - imageReferences: ["localhost:5000/demo-app:*"]
      attestors:
      - entries:
        - keyless:
            subject: "https://github.com/YOUR_ORG/*"
            issuer: "https://token.actions.githubusercontent.com"
            rekor:
              url: https://rekor.sigstore.dev
EOF

# 7. Validate enforcement: deploy the signed image (should succeed)
kubectl run signed-test --image=localhost:5000/demo-app:v1

# 8. Validate rejection: deploy an unsigned image (should be denied)
docker tag localhost:5000/demo-app:v1 localhost:5000/demo-app:unsigned
docker push localhost:5000/demo-app:unsigned
kubectl run unsigned-test --image=localhost:5000/demo-app:unsigned
# Expect: Error from server: admission webhook denied the request: ...

# 9. Cleanup
kubectl delete pod signed-test unsigned-test --ignore-not-found
kind delete cluster --name supplychain-lab

Expected result: step 7 schedules normally; step 8 is denied at admission with a policy violation event — proving the gate actually blocks unsigned artifacts rather than merely logging them.

8. Production Case Study

Google (SLSA origin, Binary Authorization): SLSA was born from Google's internal "Borg Binary Authorization for Borg" system, which has enforced provenance-based deployment gating internally for over a decade. Google Cloud's Binary Authorization product productizes the same pattern for GKE — attestor-based policies that require a signed attestation from a specific build/QA pipeline before an image can run, directly informing SLSA's public framework.

GitHub: Adopted Sigstore natively for npm provenance (any package published with --provenance gets a Rekor-logged attestation tying the published tarball back to the exact GitHub Actions run and source commit) and for its own actions/attest-build-provenance — a direct, large-scale production response to the same class of npm supply-chain attacks described in Section 2, aimed at making forged package provenance detectable at install time, not just after the fact.

Financial services / regulated industries: Firms subject to DORA/PCI-DSS increasingly mandate SLSA Level 3 plus signed SBOMs as a contractual requirement for any vendor-supplied container image, with admission-time enforcement (Kyverno/Gatekeeper) as the technical control satisfying the audit requirement — turning what used to be a quarterly manual attestation process into a continuously-enforced, queryable control.

9. Architecture Review

Strengths: Fail-closed by design at the one point (admission) every workload must pass through regardless of how it got there; keyless signing removes key-custody as an ongoing operational failure mode; the transparency log gives you retroactive, queryable forensics ("show me every image ever signed by this compromised CI identity") that a key-only signing scheme cannot provide.

Weaknesses: The system verifies identity and provenance, not safety — a compromised-but-legitimately-signed CI pipeline (attacker gets code execution inside your own trusted build) sails through every check described here. It also adds a hard availability dependency (Rekor/Fulcio, or your self-hosted mirror) to your deploy path, which needs its own SLOs and a tested degraded-mode runbook, or it becomes the reason a critical hotfix can't ship during an incident.

What fails first at 10x scale (1,400+ services): The admission webhook's synchronous registry+Rekor round-trip per new image digest becomes a real latency contributor during large simultaneous rollouts (e.g., a base-image CVE patch touching hundreds of services at once) unless digest-verification caching is aggressively warmed; a flat, org-wide signer allow-list becomes unmanageable and starts requiring per-team policy exceptions, which if not GitOps-reviewed becomes its own governance gap.

At 100M end users: The architecture doesn't change structurally — verification cost is per-deployment, not per-request, so user-facing scale doesn't touch this system directly. What does change is blast-radius tooling: at that scale you need automated Rekor-log correlation ("which currently-running digests were signed by an identity we've since revoked") wired into incident response, not a manual query someone runs during a postmortem.

What I'd redesign: Move from a single global Rekor dependency to a regionally-mirrored, sharded transparency log for latency and blast-radius isolation, and invest early in per-team policy-exception tooling (with mandatory expiry and owner attestation) rather than letting flat allow-lists calcify — the second-most common real-world failure of these programs is policy sprawl, not a bypassed control.

10. Best Practices

Reliability: Design admission-verification for graceful, observable failure — alert on webhook error rate distinct from policy-violation rate; these are different signals (one is "the control is broken," the other is "the control is working").

Scalability: Cache verification results by immutable digest; never re-verify the same digest twice per cluster lifetime.

Observability: Emit structured, per-policy Prometheus metrics from the admission controller, and dashboard admission denials against recent GitOps policy-repo commits so "did our own policy change break this" is a five-second check, not a debugging session.

Security: Enforce digest pinning everywhere (no :latest, no floating tags) as a prerequisite — signature verification against a mutable tag is not meaningfully enforceable. Rotate the OIDC trust configuration (issuer/subject allow-list) through the same PR-reviewed GitOps path as any other security control.

Cost optimization: Self-host Rekor/Fulcio only once verification volume or compliance requirements (air-gapped, data-residency) justify it — the public Sigstore instance is free and sufficient for most orgs below that threshold.

Performance: Keep the admission webhook's registry/Rekor calls off the synchronous pod-creation path where possible by pre-verifying at CD-pipeline time (before kubectl apply) and treating admission-time verification as the redundant safety net, not the primary latency-bearing check.

Maintainability: Treat signer allow-lists and SLSA-level floors as versioned policy-as-code, reviewed exactly like application code — never a manually-edited cluster-side ConfigMap.

Operational excellence: Run quarterly "Rekor is down" game days — the break-glass path is only trustworthy if it's been exercised under realistic pressure, not just written down.

11. Common Production Mistakes

  • Verifying tags instead of digests. A signature check against myapp:prod is meaningless the moment anyone re-pushes that tag — always resolve to digest before verification, and enforce digest-only references in manifests.
  • Treating SBOM generation as the finish line. An SBOM that nobody diffs against a trust baseline at deploy time is a compliance artifact, not a control — the May 2026 npm/PyPI campaigns are the canonical proof this gap is real and exploited.
  • Flipping enforcement to Enforce before auditing in Audit mode. The single most common incident in Kyverno/Gatekeeper rollouts generally (signature policies included) is a legitimate deploy pipeline getting blocked because the allow-list regex didn't match the real OIDC subject string — always run new policies in audit/dry-run first, verify zero unexpected denials across a full deploy cycle, then enforce.
  • Long-lived static signing keys stored in a shared CI secret. Defeats the entire point of moving to keyless — a leaked static key signs anything indefinitely with no expiry and no per-run identity binding.
  • No tested break-glass path for signing/verification infrastructure outages. Teams discover during a real Rekor outage that their only "fix" is disabling the policy cluster-wide — which then quietly stays disabled for weeks because nobody owns re-enabling it.

12. Interview Preparation

Q: Why is keyless signing (Fulcio/OIDC) preferred over long-lived KMS keys for CI-signed artifacts? A: It removes key custody and rotation as an ongoing operational burden and attack surface. Trust is delegated to short-lived, per-run OIDC tokens scoped to a specific workflow/ref — a leaked cert is worthless within minutes, and every signature is provably bound to one specific build execution rather than "whoever had access to the shared key."

Q: What does Rekor actually protect against, given the signing cert is already expired by verification time? A: It provides an immutable, publicly auditable proof that the signature existed and was valid at the time of signing — a signed tree-head inclusion proof anchored to a specific timestamp. This lets verification trust an expired cert's signature retroactively (was it in the log before it expired?) and lets anyone, including third parties outside your org, audit every signing event after the fact — critical for detecting a compromised CI identity signing artifacts it shouldn't.

Q: Your admission controller can't reach Rekor. What's the correct failure mode, and why? A: Fail closed for new deployments — block scheduling of anything requiring fresh verification — but do not affect already-running workloads. The alternative (fail open) means an outage of your transparency-log dependency becomes a trivial way to bypass the entire control; fail-closed converts an availability problem into "we can't deploy right now," which is recoverable and forces a documented, audited break-glass process rather than a silent bypass.

Q: SBOM vs. SLSA provenance — what's the actual difference in what each proves? A: An SBOM lists what's inside an artifact (dependencies, versions, licenses) — useful for CVE matching and audit, but it says nothing about who built the artifact or whether the listed components are what was actually reviewed. SLSA provenance attests to the build process itself — source repo, commit, build platform, build parameters — answering "did our trusted CI system produce this exact artifact," which is the question that stops a compromised or forged build, independent of whether its contents happen to be free of known CVEs.

Q: How would you roll out mandatory image-signature enforcement across an org with 140 services without causing a mass outage? A: Audit-mode first across every namespace for a full deploy cycle (minimum one release cadence per team), instrument denial-would-have-happened metrics, fix allow-list/regex drift surfaced by the audit, communicate a hard enforcement date per namespace (not global, to bound blast radius), then flip to Enforce namespace-by-namespace starting with lowest-risk/internal-only services, with a documented rollback (policy exception, time-boxed and owner-attested) for anything unexpectedly blocked.

13. Latest Industry Updates

  • Coordinated npm/PyPI supply-chain campaign, May 2026: A single attack compromised 170+ npm packages and 2 PyPI packages (404 malicious versions total) across the TanStack router ecosystem, Mistral AI's SDKs (both npm and PyPI), UiPath's automation tooling, OpenSearch, and Guardrails AI — the largest coordinated multi-ecosystem campaign to date, and the strongest real-world argument for admission-time provenance enforcement over CVE-based scanning alone, since none of the malicious versions had assigned CVEs at time of compromise. SafeDep writeup
  • Sustained multi-ecosystem campaign cadence: Six independently confirmed major campaigns landed March–July 2026 across npm, PyPI, Go modules, Crates.io, and Packagist — roughly one major campaign per month, shifting supply-chain compromise from a rare incident to an expected, recurring threat class that platform teams now budget detection and response time for. Phoenix Security analysis
  • Cosign reaching GA with a stable API: Removes a longstanding adoption blocker for enterprises that required API stability guarantees before committing signing infrastructure to it — expect increased default-on adoption in managed Kubernetes offerings through 2026.
  • Native Sigstore integration expanding: Kubernetes, GitHub, and Arch Linux integrating Sigstore directly into their own trust chains signals the ecosystem shift from "bolt-on signing tool" to "expected default" — teams evaluating new build/release tooling should treat native Sigstore support as a baseline requirement going forward, not a nice-to-have.
  • SBOM-alone strategies increasingly called out as insufficient: Multiple 2026 analyses converge on the same finding — SBOMs answer "what's inside" but not "who published it" or "is this what was reviewed," and traditional CVE-focused SCA/SBOM workflows structurally cannot catch zero-day malicious packages, reinforcing provenance + identity + admission enforcement as the necessary complement, not an alternative approach.

14. Summary & Cheat Sheet

Key concepts: Provenance (SLSA — what built this, from what), Identity (Sigstore/Fulcio — who signed it, via short-lived OIDC-bound certs, not static keys), Transparency (Rekor — immutable, publicly auditable log of every signing event), Admission enforcement (Kyverno/Gatekeeper — the last-mile gate that blocks unsigned/unproven images regardless of how they reached the cluster).

Architecture pattern: Developer → hermetic CI build (SBOM + SLSA provenance) → Sigstore keyless signing (Fulcio cert + Rekor log entry) → OCI registry (signature/attestations as digest-addressed referrers) → admission-time verification (Kyverno/Gatekeeper) → cluster runtime (Falco/Tetragon for post-admission drift detection).

Core commands:

cosign sign <image>@<digest>                      # keyless sign (OIDC flow)
cosign verify --certificate-identity-regexp '...' \
  --certificate-oidc-issuer '...' <image>          # verify signature + identity
cosign attest --predicate sbom.json --type cyclonedx <image>  # attach SBOM
cosign tree <image>                                # inspect all attached signatures/attestations
syft <image> -o cyclonedx-json > sbom.json         # generate SBOM

Best-practice checklist:

  • Digest-pinned image references everywhere — no floating tags in production manifests.
  • SLSA Build Level 3 minimum enforced via provenance attestation check at admission.
  • Signer allow-list scoped per-team/namespace, distributed via GitOps, never hand-edited in-cluster.
  • New signature/admission policies rolled out audit-mode first, enforce-mode only after a full deploy-cycle validation with zero unexpected denials.
  • Tested, documented break-glass path for Rekor/Fulcio outages, exercised via game day at least quarterly.
  • Verification results cached by immutable digest to keep admission latency off the pod-scheduling critical path.
  • SBOM + signature attached to the same digest as a Cosign attestation — not tracked separately in a compliance spreadsheet.

Troubleshooting checklist: blast radius (one team vs. cluster-wide) → webhook health/logs → registry vs. Rekor isolation via manual cosign verify → allow-list regex drift vs. actual OIDC subject claim → clock skew on signing infra → dashboard denial spikes against recent policy-repo commits.