title: "Backstage at Scale: Building a Production-Grade Internal Developer Platform with Software Catalog, Golden Paths, and TechDocs" date: 2026-09-22 tags: [Backstage, Platform Engineering, Internal Developer Platform, Developer Experience, Kubernetes, CNCF, GitOps, TechDocs, Software Catalog, AI Infrastructure] cover: ../images/backstage-internal-developer-platform-cover.png

Backstage at Scale: Building a Production-Grade Internal Developer Platform with Software Catalog, Golden Paths, and TechDocs

1. Topic of the Day

Every organization that crosses roughly 150-200 engineers hits the same wall from a different direction than the one covered in this series' Crossplane post: it isn't that infrastructure provisioning is slow, it's that nobody can find anything. Service ownership lives in someone's memory or a stale wiki page. "How do I start a new service" has eleven different answers depending on who you ask and when they joined. Documentation exists, technically, scattered across Confluence, READMEs three commits out of date, and a Notion page abandoned when its author changed teams. A platform team ships excellent infrastructure — Kubernetes, ArgoCD, Crossplane, observability — but developers still burn days rediscovering how to use it, because there is no single, authoritative, developer-facing surface that answers "what exists, who owns it, and how do I build the next one the right way."

Backstage exists to be that surface. Spotify built it internally to solve exactly this problem as their engineering org scaled past a few hundred developers, open-sourced it in 2020, and donated it to the CNCF the same year (accepted September 8, 2020), where it moved to Incubating maturity on March 15, 2022 — the same maturity tier Crossplane and OpenTelemetry sit in today. It is not infrastructure-as-code, it is not a CI/CD engine, and it is not a monitoring tool; it is a developer portal framework: an extensible, plugin-based web application with three foundational capabilities — a Software Catalog (a queryable inventory of every service, API, resource, and team, with ownership and relationships as first-class data), a Scaffolder (a self-service "golden path" template engine for creating new services that already comply with the org's standards), and TechDocs (docs-as-code rendered from Markdown living next to the code it describes, instead of a wiki that drifts away from it). Everything else — Kubernetes visibility, CI/CD status, cost insights, security posture, an AI assistant — is a plugin bolted onto that catalog.

By 2026, Backstage is the closest thing platform engineering has to a default choice: CNCF and SlashData's March 2026 platform engineering report and CNCF's own project data put it in the "adopt" ring with well over 3,000 documented public adopters and roughly 2,000 individual contributors, spanning organizations like LinkedIn, Lego, HCA Healthcare, Airbnb, Booking.com, H&M, Philips, REI, Vodafone, CVS Health, and Siemens. Spotify itself now sells a managed, commercially-supported distribution — Spotify Portal for Backstage, available on AWS Marketplace — for organizations that want Backstage's model without operating the open-source project themselves, complete with an AI knowledge assistant (AiKA) layered on top of the same catalog data. That commercial option matters precisely because it validates the open-source core's architecture is sound enough to build a product on, rather than replacing it.

In production, Backstage exists specifically where platform engineering meets developer experience (DX): it's the "front door" a platform team puts on top of everything they've built elsewhere — Kubernetes clusters, Crossplane-provisioned infrastructure, ArgoCD pipelines, Kyverno policies — so that "self-service" means one form in a portal, not eleven Slack messages to eleven different owning teams. Getting it right is a genuinely hard platform-engineering problem in its own right: catalog data goes stale the moment ingestion breaks, golden paths become shelfware the moment they're not maintained as carefully as production code, and a portal nobody trusts is worse than no portal, because it actively misdirects people.

2. Real Business Problem

Scenario: A 600-engineer B2B SaaS company has grown to roughly 240 microservices across 35 teams over five years, mostly through team-led autonomy with minimal central platform standards. A new VP of Engineering, six weeks into the job, asks a simple question in a leadership review — "how many services do we have, and who owns the one that's paging on-call right now?" — and it takes the platform team four hours and three Slack threads to produce a confident answer, because there is no system of record; the closest thing is a Confluence page last edited fourteen months ago and a Google Sheet three different teams maintain conflicting copies of.

The investigation that follows surfaces the actual shape of the problem:

• New-engineer time-to-first-commit averages 9 business days. Onboarding docs exist but are scattered across a wiki, a "Welcome" repo README, and tribal knowledge; a new hire's first week is largely spent asking teammates which of four internal CI templates to copy, which Slack channel owns the shared auth library, and why their locally-scaffolded service doesn't match what's actually deployed in production.
• Service ownership is unreliable during incidents. In the last quarter, three Sev-2 incidents had a median 22-minute delay in identifying the correct owning team, because PagerDuty escalation policies, the CMDB (last updated by hand, sporadically), and the actual git repository's CODEOWNERS file disagree with each other roughly 30% of the time, by the platform team's own spot-check audit of 50 services.
• "Golden path" is a myth in practice. Four different teams independently maintain four different Cookiecutter-style service templates, each with its own opinions on logging library, base Docker image, and CI pipeline structure — meaning a security vulnerability in a shared dependency requires the platform team to manually track down and patch four different template lineages instead of one.
• Documentation exists but nobody trusts it. An internal survey found 68% of engineers say they "usually" or "always" ask a teammate rather than search docs first, specifically because past experience taught them wiki content is frequently stale — the classic docs-as-side-project failure mode, where documentation isn't reviewed with the same rigor as code because it doesn't live next to the code or go through the same PR process.
• No visibility into what's actually running. The platform team cannot answer, without manually querying six different systems (Kubernetes, the CI system, the cloud billing console, a Terraform state bucket, GitHub, and PagerDuty), which services are actively deployed, which are abandoned-but-still-running (a real cost and security liability), and which have no assigned owner at all after two reorgs.
• AI coding assistants are already in daily use, and they have nothing authoritative to ground against. Developers are pasting service names into an LLM chat and asking "what does this do and who owns it," getting confidently wrong answers, because there's no structured, machine-readable source of truth an AI agent could query instead of hallucinating from stale Slack history.

Leadership's mandate: cut time-to-first-commit for new engineers to under 2 days, produce a single authoritative answer to "what exists and who owns it" within seconds during an incident, converge four service templates into a small number of governed golden paths, and make documentation something people trust enough to use as their first stop — without a multi-year, big-bang platform rewrite, and ideally reusing what the platform team already has: Kubernetes, GitHub, ArgoCD, and Okta SSO.

3. Production Architecture

The frontend is a single-page app; the backend is where the real architecture lives. Since the 2023 "New Backend System" rework (and now, as of the New Frontend System going default in March 2026, on both sides of the stack), Backstage is architected as a thin core runtime plus a set of independently-installed plugin packages — @backstage/plugin-catalog-backend, @backstage/plugin-scaffolder-backend, @backstage/plugin-techdocs-backend, @backstage/plugin-auth-backend, and dozens more, first-party and community. This is a deliberate move away from Backstage's earlier monolithic backend, where every plugin shared one Express app and one process — the new model lets a platform team run the backend as multiple independently-scaled services (a heavy TechDocs build workload doesn't compete for CPU with catalog processing) and, critically, lets them adopt only the plugins they need rather than inheriting the full surface area on day one.

The Software Catalog is a graph database with a plugin-based ingestion pipeline in front of it, not a form anyone fills out by hand. Every entity — a Component, API, Resource, System, Domain, User, or Group — is described by a small YAML file, conventionally named catalog-info.yaml, committed at the root of the repository it describes. Entity Providers (GitHub/GitLab org discovery, a Kubernetes ingestor that turns live cluster objects into catalog entities, or a custom provider hitting an internal system) locate these files and hand them to the catalog's processing pipeline — a chain of processors that resolve implementsApis, dependsOn, providesApis, and ownership relations into the fully-connected entity graph the frontend actually queries. The processed result — the entity, its relations, and its location metadata — persists in PostgreSQL as the catalog's only durable state; nothing about entity data lives solely in a plugin's memory.

Ownership and relations are first-class, not a comment field. Every Component declares a spec.owner (resolved against a Group entity, itself synced from an identity provider or a GroupEntityProvider), and spec.system groups related components into a logical product boundary. This is the direct architectural answer to Section 2's ownership-ambiguity problem: instead of three disagreeing sources of truth (PagerDuty, a hand-maintained CMDB, git CODEOWNERS), the catalog becomes the single graph that PagerDuty escalation policies, the Kubernetes plugin's namespace-to-team mapping, and the org chart (synced from Okta/Azure AD as User and Group entities) all resolve against.

Golden paths are Scaffolder software templates — reviewed, versioned, and tested like any other code. A template.yaml defines a form (service name, owning team, system, target repository), then an ordered sequence of scaffolder actions — fetch:template to render a skeleton from a Cookiecutter-style template repo, publish:github to create the actual repository and push the rendered skeleton, and catalog:register to immediately register the new service's catalog-info.yaml with the catalog, closing the loop from "developer fills out a form" to "service exists in the catalog" without a manual follow-up step. Scaffolder actions run in a sandboxed, allowlisted action registry specifically because a template author's YAML can trigger real side effects (repo creation, cloud resource requests via a custom action calling out to Crossplane or an internal API) — the platform team curates which actions are available, the same way they'd review a library dependency.

TechDocs deliberately does not store rendered documentation in the same database as the catalog. Markdown files live in the same repository as the code, under docs/, following the mkdocs.yml convention; a CI step (or the TechDocs backend itself, in the simpler "in-process" generation mode) runs an MkDocs build inside a container and publishes the static output to object storage — S3, GCS, or Azure Blob — from which a dedicated TechDocs backend serves it to the frontend. This separation is intentional: a broken doc build for one service never touches catalog availability, and static HTML in object storage is cheap, cacheable, and trivially horizontally scaled, unlike serving a live-rendering wiki.

Why this design, and the trade-offs. The alternative evaluated and rejected outright was a commercial internal-developer-portal SaaS (Port, Cortex, OpsLevel, or Atlassian Compass — compared directly in Section 4) that hosts the catalog data externally. That path is genuinely faster to a working demo — no backend to operate — but it means the org's most sensitive metadata (which services exist, who owns them, their dependency graph, security posture) lives in a third-party's database, and it couples the org's ability to customize catalog behavior to that vendor's roadmap. Backstage's self-hosted, plugin-extensible model costs real platform-engineering investment — someone has to run the backend, the Postgres instance, and the TechDocs storage — but it means the catalog can be extended with a custom entity provider for an internal system that will never be a vendor's priority, and it means the org is never negotiating a per-seat license for something as foundational as "where is our service inventory." As adoption scales past a few hundred engineers, the natural evolution is splitting catalog-heavy and scaffolder-heavy backend plugins into separately-scaled deployments (the New Backend System explicitly supports this) and moving from full catalog re-ingestion to incremental entity providers — covered in Section 5 — well before ingestion volume makes that necessary, not after.

4. Solution Design

Design decision: a small number of curated golden-path templates, not a template for every team's preference. The org converged its four independently-maintained service templates into three Scaffolder golden paths — "standard Go microservice," "Node.js API service," and "Python batch job" — each owned and versioned by the platform team as a real, tested artifact with its own CI. This directly targets Section 2's "four templates drifting independently" finding: a shared-dependency CVE now requires patching three golden paths, reviewed and released like a library, instead of chasing four ad hoc lineages. The org deliberately resisted requests to add a fourth and fifth template for teams with "slightly different" needs — the accepted industry guidance, and the org's own prior experience, is that three to five golden paths is the practical ceiling before template sprawl recreates the exact fragmentation the platform was built to remove; a broken or unmaintained template erodes developer trust in the entire platform faster than having no template at all.

Alternative considered and rejected: a commercial IDP SaaS (Port, Cortex, OpsLevel, Atlassian Compass). These products offer the same core value proposition — catalog, scorecards, self-service actions — as a hosted service with materially less operational burden, and were seriously evaluated. Rejected for this org specifically because of data residency concerns (the catalog would hold ownership and dependency data for security-sensitive internal systems) and because the org already operates Kubernetes and Postgres at the scale needed to self-host Backstage's backend competently — for a smaller platform team without that existing operational capacity, or an org that wants a working portal in days rather than a quarter, a hosted IDP product is a legitimate and often better choice; the decision here is a genuine "it depends on what you already operate," not a universal recommendation.

Alternative considered and rejected: keep building on the internal wiki plus a home-grown CMDB. Rejected because it doesn't address the structural cause of Section 2's problems — documentation living apart from code, and ownership data living in a system nothing else can programmatically query, is precisely why the wiki drifted stale and PagerDuty/CMDB/CODEOWNERS disagreed in the first place. A hand-maintained CMDB has no forcing function to stay accurate; a catalog populated by automated entity providers reading from the same git repositories developers already update does.

Scalability considerations. Catalog processing load scales with entity count and, more sharply, with how often full re-ingestion runs against a large external source (a GitHub organization with thousands of repositories, or a Kubernetes cluster with thousands of pods) — this is precisely why incremental entity providers, which mutate the catalog via computed deltas instead of a full teardown-and-rebuild on every cycle, exist, and adopting one early avoided the out-of-memory failures that historically hit orgs doing full-scan ingestion against large GitHub orgs. The org's catalog sits at roughly 1,200 entities (services, APIs, resources, and team/user entities combined) — comfortably within the range Backstage's default processing engine handles without incremental providers, but the platform team adopted the GitHub incremental entity provider anyway, ahead of need, because migrating ingestion strategy after the catalog is load-bearing is materially riskier than adopting it early.

Cost implications. Backstage itself carries no license cost — it's an Apache-2.0 CNCF project — so the real cost is engineering time: roughly 1.5 platform engineers dedicated to operating and extending it, plus modest infrastructure (a small Postgres instance, a handful of backend replicas, S3 storage for TechDocs). Set against the 9-day time-to-first-commit and the 22-minute incident-ownership-resolution delay from Section 2, the ROI case was made on developer-hours reclaimed, not infrastructure savings — the CFO-facing argument was framed as "N engineers times M days of onboarding friction," not "we saved on cloud spend."

Security implications. Backstage's permission framework — evaluated on every catalog read, scaffolder action, and plugin API call — was enabled from the initial rollout rather than bolted on later specifically because retrofitting fine-grained permissions onto an already-open backend that every team has built habits around is a materially harder migration than starting locked down. Scaffolder actions are the sharpest edge here: an action that can create a GitHub repository or call an internal provisioning API is real write access, and the org maintains an explicit allowlist of which actions each golden path may use, reviewed the same way IAM policies are reviewed.

Performance implications. Time-to-first-commit for a net-new service dropped from a multi-day, ask-around process to under an hour for a golden-path-eligible service (Scaffolder form to a running CI pipeline and a catalog-registered repository) — well inside the sub-2-day mandate — with the remaining onboarding time being genuinely necessary ramp-up (understanding the specific business domain), not tooling friction.

Comparing the competitive landscape. Backstage is not the only tool in this space, and the choice matters:

• Backstage (CNCF, self-hosted, open source): maximum extensibility via a large plugin ecosystem and full control over data residency, at the cost of real operational ownership — you run the backend, the database, and you're responsible for the plugin upgrade cadence. The right choice for orgs with existing platform-engineering capacity and a long time horizon.
• Port, Cortex, OpsLevel (commercial, hosted IDP SaaS): faster time-to-value, no backend to operate, generally strong scorecards/maturity-model features out of the box. The right choice for orgs prioritizing speed-to-adoption over deep customization, or without spare platform headcount to run infrastructure for the portal itself.
• Atlassian Compass: the natural choice for an org already deeply invested in the Atlassian suite (Jira, Confluence, Bitbucket) wanting tight native integration over Backstage's broader but more DIY plugin model.
• Spotify Portal for Backstage: a managed distribution of the actual open-source Backstage core, for orgs that want Backstage's specific architecture and plugin compatibility without operating the backend themselves — a middle path between full self-hosting and a fully proprietary competitor.

5. Deep Technical Walkthrough

Entity lifecycle: location, provider, processor, emit. Every catalog entity's journey starts at a location — most commonly a URL pointing at a catalog-info.yaml file, discovered either by manual registration (catalog-import in the UI) or, at scale, by an Entity Provider that programmatically discovers many locations at once (the GitHub org discovery provider scans an entire GitHub organization for matching files on a schedule). Once a location is known, the processing engine reads the raw YAML, runs it through a chain of processors — a PlaceholderProcessor resolves $text/$json includes, an AnnotateLocationEntityProcessor stamps provenance, and validation processors enforce the entity schema — and, critically, a relations-resolving stage turns declarative fields like spec.dependsOn: [component:default/payments-api] into bidirectional graph edges stored alongside both entities. This entire pipeline runs on a polling interval per location (or reactively, for providers with event/webhook support) and writes its result into two Postgres tables that matter operationally: refresh_state (what's due for reprocessing) and final_entities (the last successfully processed, servable version of each entity) — meaning a processing failure for one entity leaves its previous good state servable rather than blanking it out.

Why processors run in series, and why that matters under failure. Entity processors for a given entity execute in a defined, ordered sequence, not in parallel — each processor can depend on the output of the one before it (relations resolution needs validation to have already run). The operational consequence, confirmed directly by Backstage's own engineering guidance, is that a single slow or hanging processor — the canonical example is a Kubernetes-ingesting processor configured to reach a cluster that's since been decommissioned or is network-unreachable — blocks the processing queue for the entities behind it, not just its own. This is precisely the failure mode walked through in Section 6.

Incremental Entity Providers: delta-mutation instead of full rebuild. The original entity provider model was tear-down-and-rebuild: on every refresh cycle, replace the provider's entire contribution to the catalog. For a source with tens of thousands of records (a large GitHub org, an internal CMDB with a hundred thousand rows), doing that as a single in-memory operation on every cycle was the direct cause of out-of-memory failures that several early large-scale Backstage adopters hit. Incremental Entity Providers instead compute and apply a delta — entities added, changed, or removed since the last successful cursor position — closing that scaling gap without requiring the source system to expose webhooks, at the cost of somewhat more complex provider implementation logic (tracking a cursor, computing diffs) that the platform team, not application teams, owns and maintains.

Scaffolder action execution and the security boundary it implies. Each step in a template.yaml invokes a named action (fetch:template, publish:github, github:actions:dispatch, or a custom action) executed server-side by the scaffolder backend inside a scoped working directory per task — never inside the requesting user's browser or with the user's own raw credentials, but via service-account-scoped credentials the platform team provisions for the action (a GitHub App installation token scoped to specific repos and permissions, for instance). This is the exact mechanism that makes golden paths trustworthy as a security control, not just a convenience: because repo creation, branch protection setup, and CI wiring all happen through the same reviewed action code on every golden-path execution, a security requirement (branch protection enabled, a required status check configured) is structurally guaranteed for every service created through the path — something no wiki-documented "please remember to configure branch protection" instruction can guarantee.

TechDocs generation pipeline, in detail. The default, in-process generation mode has the TechDocs backend itself run an MkDocs build (via a Python/MkDocs container image with the techdocs-core plugin bundle) whenever a doc-eligible entity is viewed and its docs are stale, then caches and publishes the output; the recommended production mode instead runs generation as a CI step in the same pipeline that builds and deploys the service, publishing directly to object storage — decoupling documentation freshness from portal traffic patterns and avoiding a burst of concurrent MkDocs builds triggered by page views during a busy on-call incident, which is exactly the wrong moment for the docs backend to be CPU-constrained.

Permission framework evaluation. Every catalog read, scaffolder action execution, and plugin-exposed API call passes through a PermissionPolicy the platform team implements once, centrally — evaluated against the authenticated user's identity (resolved at sign-in via the configured auth provider's signInResolver, typically mapping an Okta/Azure AD group claim to a Backstage Group entity) and the specific resource and action being requested. Conditional decisions (e.g., "can edit catalog-info.yaml metadata only if you're a member of the entity's owning group") are expressed as structured rules the frontend can also partially evaluate to hide UI it knows will be denied, rather than every permission check requiring a round-trip.

Failure scenarios and recovery mechanisms.

• A misconfigured Kubernetes entity-ingesting processor targets a decommissioned cluster: the processor call hangs or times out slowly, and because processors run in series per entity, that specific entity's processing chain stalls — other entities not routed through that processor continue normally, but the affected entities show increasingly stale final_entities data until the processor is fixed or the location removed (Section 6 walks this exact incident).
• A Scaffolder task fails mid-pipeline (e.g., publish:github fails after fetch:template already rendered files): the task is marked failed with the specific step and error surfaced in the Scaffolder UI's task log; because each action's side effects are step-scoped rather than transactional across the whole template, a failed publish:github after a successful fetch:template typically just means retrying the task — no orphaned catalog entity is created, since catalog:register (the step that would create one) only runs after publish succeeds.
• TechDocs build fails for a given service: the previously-published static docs remain servable from object storage; a build failure surfaces as a stale-docs warning in the entity page rather than a broken docs tab, because generation and serving are decoupled (Section 3).
• Postgres (the catalog's only durable store) is lost: since every entity's authoritative definition is a catalog-info.yaml file living in its own service's git repository, catalog data is fundamentally re-derivable — a Postgres restore from backup handles the common case, but even a full-loss scenario is recoverable by re-running full ingestion against every registered location, unlike a hand-maintained CMDB with no external source of truth to rebuild from.

Performance bottlenecks and scaling behavior. At catalog sizes in the low thousands of entities, the binding constraint organizations report is less raw entity count and more processing engine concurrency configuration and inefficient custom processors — Roadie's public engineering writeups document tuning cases achieving up to 48x throughput improvements purely from processing-engine concurrency and caching configuration, without any architectural change. The next anticipated constraint as the catalog grows past the low tens of thousands of entities is search index rebuild time and frontend catalog-listing query performance, which the org is monitoring via Backstage's own exposed processing-duration metrics rather than waiting to discover reactively.

6. Production Troubleshooting

Incident: catalog entities across an entire business unit show increasingly stale data, and nobody notices until an on-call engineer pages the wrong team.

Symptoms. An on-call engineer, paged for a service in the "Payments" system, checks its Backstage entity page to find the owning team and Slack channel — and gets a team that transferred ownership three weeks ago, sending the page to a team with no context on the failing service. Separately, the platform team's own dashboard shows the catalog's overall entity count hasn't grown in three weeks despite four new services being merged and deployed in that window.

Commands and signals to pull first.

• Backstage's own catalog processing metrics (catalog_processing_duration_seconds, catalog_processed_entities_count, exposed via the standard /metrics endpoint if Prometheus integration is enabled) — segmented by processor, to see whether duration is climbing for one specific processor rather than uniformly across all of them.
• refresh_state table inspection (direct Postgres query in a non-production-impacting read replica, or via the catalog's own diagnostic endpoints) — checking next_update_at timestamps to find entities that are significantly overdue for reprocessing, which is the direct symptom of a stuck queue rather than a catalog-wide outage.
• Backend pod logs for the catalog-backend plugin, filtered for the affected entities' namespace/location — looking specifically for timeout or connection-refused errors rather than validation errors, since a validation error fails fast and doesn't block the queue the way a hanging network call does.
• Cross-checking against the Kubernetes ingestor's configured cluster list, since "a Kubernetes-ingesting processor targeting a decommissioned cluster" is the textbook cause of exactly this symptom pattern.

Root cause analysis, step by step. The refresh_state query shows a cluster of entities — all belonging to the Payments business unit specifically — with next_update_at timestamps three weeks in the past, while entities outside that business unit are current. Backend logs for the catalog processor show repeated context deadline exceeded errors against a specific Kubernetes API server hostname. Cross-referencing with the platform team's own infrastructure change log: that Kubernetes cluster was decommissioned three weeks ago as part of an unrelated regional consolidation, but the Kubernetes ingestor's configuration (a static list of cluster API endpoints, not itself dynamically discovered) was never updated to remove it — a coordination gap between the infra-migration project and the catalog configuration, not a bug in Backstage itself. Because processors for affected entities run in series and this processor sits early in the affected entities' pipeline, every entity that pipeline would otherwise touch stalls behind the timeout, one full processing cycle at a time, silently degrading rather than throwing a visible catalog-wide error.

Configuration changes made. Immediate: removed the decommissioned cluster from the Kubernetes ingestor's configuration and restarted the catalog-backend deployment, which immediately unblocked the stuck queue — affected entities caught up to current within one processing cycle. Root-fix: added a synthetic monitoring check specifically for "processor target reachability" (a lightweight periodic health check against every configured Kubernetes cluster and external system the catalog ingests from) that alerts before a decommissioning project silently breaks catalog ingestion, plus a checklist item added to the infrastructure decommissioning runbook itself — "check Backstage catalog configuration for references to this resource" — closing the coordination gap at its actual source rather than only reactively.

Verification. Re-ran a full manual refresh trigger against the previously-stalled entities and confirmed next_update_at timestamps returned to expected intervals, final_entities data matched current git state for spot-checked services, and the on-call escalation test (paging a synthetic test service) resolved to the correct, current owning team. The synthetic reachability check was validated by intentionally pointing it at a known-bad hostname in a staging environment and confirming the alert fired within one check interval.

7. Hands-on Lab

A local Backstage instance exercising the exact catalog, Scaffolder, and TechDocs mechanics used in production — no Kubernetes cluster or cloud account required for the core lab, using SQLite in place of Postgres purely to keep local setup fast.

Setup.

npx @backstage/create-app@latest --skip-install
cd my-backstage-app
yarn install

# Start the app (frontend + backend, using the default SQLite in-memory catalog DB)
yarn dev

Register a service by hand, to see the location -> processor -> entity flow directly.

# catalog-info.yaml, committed at the root of a sample repo
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payments-api
  description: "Handles payment processing and settlement"
  annotations:
    github.com/project-slug: my-org/payments-api
  tags: [payments, tier-1]
spec:
  type: service
  lifecycle: production
  owner: group:default/payments-team
  system: payments
  providesApis: [payments-api]

# Register the location via the Backstage UI ("Register Existing Component"),
# pointing at the raw catalog-info.yaml URL, or via the API directly:
curl -X POST http://localhost:7007/api/catalog/locations \
  -H "Content-Type: application/json" \
  -d '{"type": "url", "target": "https://github.com/my-org/payments-api/blob/main/catalog-info.yaml"}'

# Confirm it processed successfully:
curl http://localhost:7007/api/catalog/entities/by-name/component/default/payments-api

Build a minimal golden-path Scaffolder template.

# templates/simple-service/template.yaml
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: simple-node-service
  title: Simple Node.js Service
  description: Golden path for a standard internal Node.js service
spec:
  owner: platform-team
  type: service
  parameters:
    - title: Service details
      required: [name, owner]
      properties:
        name:
          type: string
          title: Service name
        owner:
          type: string
          title: Owning team
  steps:
    - id: fetch
      name: Fetch skeleton
      action: fetch:template
      input:
        url: ./skeleton
        values:
          name: ${{ parameters.name }}
          owner: ${{ parameters.owner }}
    - id: register
      name: Register in catalog
      action: catalog:register
      input:
        catalogInfoPath: /catalog-info.yaml
  output:
    links:
      - title: Open Catalog Entry
        icon: catalog
        entityRef: ${{ steps.register.output.entityRef }}

# Register the template itself as a catalog Location, then run it from the
# "Create" tab in the Backstage UI, or headlessly:
curl -X POST http://localhost:7007/api/scaffolder/v2/tasks \
  -H "Content-Type: application/json" \
  -d '{"templateRef": "template:default/simple-node-service", "values": {"name": "hello-service", "owner": "platform-team"}}'

Enable TechDocs locally.

# Add a docs/index.md and mkdocs.yml to the sample repo, then in app-config.yaml:
# techdocs:
#   builder: 'local'
#   generator:
#     runIn: 'docker'
#   publisher:
#     type: 'local'

Validation. Confirm the Software Catalog UI lists payments-api with correct owner and system, that the Scaffolder task for hello-service completes and produces a second registered entity, and that the TechDocs tab for a docs-enabled entity renders the MkDocs-built page rather than a "docs not found" placeholder.

Cleanup.

# Stop the dev server (Ctrl+C), then remove local state:
rm -rf my-backstage-app

8. Production Case Study

Spotify is both Backstage's creator and its longest-running production user: the internal tool that became Backstage was built specifically to manage Spotify's own scale — thousands of services and hundreds of squads — and Spotify's continued investment (open-sourcing it, donating it to CNCF, and now selling a commercially-supported distribution, Spotify Portal for Backstage, with an AI knowledge assistant layered on top) is the clearest signal that the underlying catalog-plus-golden-paths model holds up under sustained, large-scale internal use rather than degrading into shelfware.

LinkedIn, Lego, HCA Healthcare, Airbnb, Booking.com, H&M, Philips, REI, Vodafone, CVS Health, and Siemens are among the CNCF-documented public adopters spanning healthcare, retail, travel, and consumer tech — a breadth that reflects Backstage's core value proposition (service catalog plus self-service golden paths) generalizing well beyond any single industry's specific tooling stack, since the plugin architecture lets each org bolt on the specific CI/CD, cloud, and observability integrations relevant to them without forking the core.

The broader industry pattern these adopters share is using Backstage as the integration layer that makes previously-siloed platform investments legible to developers: a company that has already built solid Kubernetes, CI/CD, and observability tooling (the subject of most of this series' other posts) gets comparatively little value from those investments if developers can't discover and use them without platform-team hand-holding — Backstage's bet is that the catalog and a small number of golden paths are the multiplier that makes every other platform investment actually self-service, which is precisely why large organizations already deep into platform engineering gravitate toward it rather than smaller shops still building their first CI pipeline.

9. Architecture Review

Strengths. The plugin architecture is the standout design decision: a platform team adopts exactly the surface area it needs (catalog and Scaffolder on day one, TechDocs and a Kubernetes plugin later) rather than committing to an all-or-nothing platform. Catalog data being re-derivable from catalog-info.yaml files living in the same repositories as the code (Section 5) gives the whole system a resilience property most CMDB-style tools lack: worst-case, the catalog is rebuildable from source of truth, not solely dependent on a database backup. CNCF governance (Incubating maturity, an active security-audit history, and a healthy multi-vendor contributor base rather than single-company control) meaningfully de-risks the "will this project still be maintained in five years" question that matters for a system this foundational.

Weaknesses. The honest industry-wide finding as of mid-2026 — surfaced publicly in analyses like bex.co's "89% market share hides a 10% adoption problem" — is a real gap between installing Backstage and fully adopting it: many organizations stand up a basic catalog and stop there, because the plugin ecosystem's pace of releases creates a genuine "plugin upgrade tax" — keeping a growing set of independently-versioned plugins compatible with each other and with core Backstage releases is ongoing platform-engineering work that doesn't stop once the initial rollout ships, and orgs that under-resource that ongoing maintenance see their instance drift toward exactly the kind of stale, half-trusted state the platform was built to prevent. Backstage also, deliberately, ships you a framework rather than a finished product — meaningful golden paths, a working permission policy, and TechDocs publishing all require real platform-engineering implementation work, unlike a hosted competitor's out-of-the-box scorecards.

What would fail first at 10x scale. For this org's roughly 1,200-entity catalog, 10x is a 12,000-entity catalog — at that scale, the first thing to break is any lingering full-rebuild entity provider (Section 5's incremental-provider migration becomes mandatory, not optional), followed by processing-engine concurrency defaults tuned for a much smaller catalog, and eventually frontend catalog-browsing query performance if search indexing hasn't been moved off the default in-memory/Postgres-FTS setup onto a dedicated search engine (Elasticsearch/OpenSearch, both supported via the search backend's pluggable collator/decorator model).

Reframing "100 million users" for an internal tool. Backstage doesn't serve external end users, so the equivalent stress case is an organization at Google/Amazon/Microsoft scale — on the order of 100,000+ engineers and potentially hundreds of thousands of catalog entities across dozens of business units, each with different compliance and tooling requirements. At that scale, a single Backstage instance with one shared Postgres and one shared permission policy stops being the right architecture; the realistic evolution is federation — either multiple Backstage instances per business unit with a lightweight cross-instance search/discovery layer, or (the direction some of the largest documented deployments have taken) a single instance with strict multi-tenancy enforced through the permission framework and namespaced catalog locations per business unit, accepting the added architectural complexity because a single unified catalog remains valuable even at that scale for cross-org dependency visibility.

What to redesign. Given the plugin-upgrade-tax finding, the most consequential redesign for a scaling org is treating the Backstage instance itself as a product with a dedicated, permanent owning team and an explicit upgrade cadence (tracking upstream releases on a fixed schedule, e.g., monthly), rather than a project that ships once and is maintained on an ad hoc basis — the single biggest predictor of whether an org ends up in the "10% full adoption" cohort or the "89% installed, stalled" cohort in the industry data cited above.

10. Best Practices

Reliability. Run the backend as multiple stateless replicas behind a load balancer from the start (the New Backend System's plugin-per-service model makes this natural), with Postgres as the only durable state and a real backup/restore-tested strategy for it, since it's now load-bearing infrastructure the same way a production database is.

Scalability. Adopt incremental entity providers for any source with more than a few thousand records well before that becomes an incident, and monitor processing-engine duration metrics per-processor from day one so a slow processor is caught in dashboards, not in an on-call escalation gone to the wrong team.

Observability. Treat the catalog's own health as a first-class SLO — freshness (time since last successful processing per entity), not just uptime — since a "healthy" Backstage instance serving three-week-stale ownership data is a silent failure mode, not an obvious one.

Security. Enable the permission framework from initial rollout rather than retrofitting it, maintain an explicit allowlist of Scaffolder actions available to golden paths, and audit which service-account credentials backend plugins hold (a GitHub App installation token scoped to specific repos, not a personal access token with org-wide access).

Cost Optimization. Backstage itself is license-free; the cost discipline that matters is bounding plugin sprawl — every additional community plugin is an additional upgrade-compatibility surface, not just a one-time integration cost, so plugin adoption should go through the same review rigor as a new production dependency.

Performance. Decouple TechDocs generation from serving (build in CI, publish to object storage) rather than relying on the default in-process, on-demand generation mode in production, and tune processing-engine concurrency deliberately rather than leaving default settings that were sized for a much smaller catalog.

Maintainability. Cap golden-path templates at three to five, own them with the same rigor as a shared library (versioned, tested, with a deprecation path for old versions), and resist the pressure to fork a template per team's minor preference.

Operational Excellence. Assign Backstage a permanent owning team and a fixed upgrade cadence against upstream releases — the single practice the industry data most directly ties to landing in the cohort of orgs with genuine, sustained adoption rather than a stalled initial rollout.

11. Common Production Mistakes

Treating the initial rollout as a one-time project rather than an ongoing product is the anti-pattern underlying most of the rest: teams that ship a catalog and a golden path, declare victory, and move the implementing engineers onto other work are the ones whose instance predictably drifts stale within a year, recreating Section 2's original problem with extra steps. Skipping the permission framework at launch because "we'll add security later" is a close second — retrofitting fine-grained permissions onto an instance every team has already built habits around open access to is a materially harder migration, and in practice frequently just doesn't happen. Registering every repository indiscriminately, without requiring a real spec.owner and spec.system, produces a catalog full of orphaned or unowned entities that's technically complete but practically as unreliable as having no catalog — the org's own earlier CMDB failure mode, reproduced inside Backstage instead of solved by it. Building more than a handful of golden paths to satisfy every team's stylistic preference recreates template sprawl under a new name. Finally, using TechDocs' default in-process generation mode in production without moving builds into CI is a scaling trap that works fine in a demo and degrades exactly when it matters most — during a high-traffic incident, when everyone is simultaneously opening docs pages for context.

12. Interview Preparation

Senior DevOps / Platform Engineering: "Design an internal developer platform for a 500-engineer org. What goes in the catalog, and what stays out?" A strong answer centers the catalog on entities with a clear owner and lifecycle — services, APIs, and the resources/systems that group them — and explicitly excludes ephemeral or low-value entities (a personal experiment repo, a one-off script) that would dilute signal; it also addresses how ownership data gets populated (automated entity providers reading from existing systems, not a one-time manual survey that immediately starts going stale) and how the catalog stays trustworthy over time (freshness SLOs, not just initial population).

Kubernetes / Cloud Architecture: "How would you architect Backstage's backend for high availability, and what's the actual single point of failure?" Candidates should identify that the New Backend System's plugin-per-service model supports multiple stateless replicas behind a load balancer, that Postgres is the genuine single point of failure requiring real HA (multi-AZ, tested backup/restore) since it's the catalog's only durable store, and that TechDocs' object-storage-backed static content is comparatively low-risk by design.

Platform Engineering: "A team says the golden-path template doesn't fit their use case and wants their own. How do you decide?" The expected reasoning: weigh the cost of template sprawl (patch N templates instead of one for every shared dependency, developers again face inconsistent tooling) against the genuine cost of forcing a poor fit; a strong answer proposes extending the existing template with an optional parameter or step rather than forking, and only forks when the divergence is structural, not stylistic — with the platform team retaining ownership either way.

AI Infrastructure: "How does an AI coding agent use a service catalog like Backstage's, and why does that change how you should design catalog data?" The key insight: an AI agent querying the catalog needs structured, machine-readable ownership and dependency data (the same spec.owner, spec.dependsOn relations humans use) rather than free-text documentation, which is precisely why Backstage's catalog-as-graph model is becoming more valuable, not less, as AI agents become a primary consumer of internal platform data alongside humans — and why catalog data quality (no orphaned entities, accurate ownership) has a new, higher-stakes consumer that will act on wrong data rather than just display it.

Cloud/Platform Comparison: "When would you choose a hosted IDP product like Port or Cortex over self-hosting Backstage?" A strong answer names the actual trade-off: hosted products trade deep extensibility and full data control for materially lower operational burden and faster time-to-value, making them the better choice for smaller platform teams or orgs prioritizing speed over customization, while self-hosted Backstage is the better choice for orgs with existing platform-engineering capacity, strict data-residency requirements, or the need for deep custom integrations a vendor's roadmap won't prioritize.

13. Latest Industry Updates

The New Frontend System went default in March 2026, the culmination of a multi-year migration away from Backstage's original frontend architecture — the kind of stability and maturity signal that materially eases procurement approval at risk-averse enterprises evaluating Backstage for the first time, since it demonstrates the project can execute a major architectural migration without breaking the ecosystem of existing plugins wholesale.

CNCF released a documentary on Backstage's evolution in March 2026, framing it explicitly as having become "the global open source standard for platform engineering" — alongside a joint CNCF/SlashData report the same month finding platform engineering tooling broadly maturing specifically to support AI-driven infrastructure, a direct signal that catalog-as-context-for-AI-agents (Section 12) is becoming an explicit industry priority, not a speculative future use case.

BackstageCon ran as a co-located event at KubeCon + CloudNativeCon Europe 2026, reflecting the project's growth from a single-vendor tool to one with enough independent community activity to sustain its own dedicated conference track — alongside the CNCF's Certified Backstage Associate (CBA) certification, now an established credential for platform engineers to demonstrate hands-on Backstage competency.

Spotify's plugin marketplace and AI assistant continue to mature the commercial ecosystem around the open-source core: the marketplace now curates vetted plugins from partners including Red Hat and VMware, the broader community plugin ecosystem has grown past 200 plugins, and AiKA — Spotify Portal's AI knowledge assistant, which searches and reasons over the same catalog and TechDocs data a human would — points at the same direction as the CNCF/SlashData report: the software catalog isn't just a UI for humans anymore, it's becoming the structured context layer AI agents query to operate safely inside an organization's real systems.

The honest counter-signal worth tracking: industry analysis through mid-2026 (the "89% market share hides a 10% adoption problem" finding referenced in Section 9) suggests the ecosystem's growth in raw adoption numbers is outpacing the number of organizations achieving deep, sustained adoption — a maturity gap that mirrors what Kubernetes itself went through in its own early years, and one the New Frontend System's stability push and the CBA certification's skills standardization are both direct, deliberate attempts to close.

14. Summary & Cheat Sheet

Core concepts. Backstage is a developer portal framework, not a finished product: Software Catalog (entities, ownership, relations, backed by Postgres), Scaffolder (golden-path software templates executing sandboxed, allowlisted actions), and TechDocs (docs-as-code, built separately from serving) are the three foundational plugins everything else builds on.

Architecture in one line. Entity Providers discover locations → Processing Engine resolves and persists entities to Postgres (series-ordered processors, incremental providers for scale) → the frontend and every other plugin (Kubernetes, ArgoCD, Cost Insights, AiKA-style AI assistants) reads from that catalog rather than querying source systems live.

Key commands / files.

• catalog-info.yaml — the entity definition living in the service's own repo.
• template.yaml — a Scaffolder golden-path definition.
• mkdocs.yml + docs/ — TechDocs source, built via CI, published to object storage.
• refresh_state / final_entities — the Postgres tables to inspect when catalog data looks stale.

Best-practice checklist.

• Permission framework enabled from day one, not retrofitted.
• 3-5 golden paths maximum, owned and versioned like production code.
• Incremental entity providers adopted before catalog size makes full rebuilds unstable.
• TechDocs built in CI, not on-demand in production.
• A permanent owning team and a fixed upstream-upgrade cadence — the single strongest predictor of landing in the "sustained adoption" cohort rather than the "installed and stalled" one.

Troubleshooting checklist.

• Catalog data stale? Check refresh_state.next_update_at for entities stuck behind a hanging processor before assuming a catalog-wide outage.
• Scaffolder task failed mid-pipeline? Check the task log for the specific failed step; failures are step-scoped, not transactional across the template.
• TechDocs missing or stale? Confirm whether generation is in-process (production-risky) or CI-published (recommended), and check object storage directly before assuming a catalog bug.
• Ownership disagreement during an incident? The catalog's spec.owner, not PagerDuty or a wiki, should be the single source of truth — if it isn't yet, that's the actual gap to close.