Dev/Prod Isolation in a Single GCP Project: State, Secrets, IAM, and Workflow Boundaries

This post documents how I kept a Cloud Run + Cloud SQL application in the same GCP project while still cleanly separating dev and prod.

Branch naming alone will not isolate your environments, nor will merely adding two more GitHub workflows. What truly needs to be separated is state, resource naming, secrets, runtime identity, deploy identity, and workflow governance. If even one of these layers leaks, they will eventually collide.

Motivations for Staying in One Cloud Entity

Cost and management overhead usually drive the decision, not technical constraints.

If you go project-per-environment from the start, it’s closer to cloud best practice, but it also adds a lot:

• more IAM setup
• more service accounts
• more budget tracking
• more APIs to enable
• more Terraform bootstrap work
• more operational overhead

If the product is still in early stage, or you just want to get the deployment path, runtime isolation, and branch governance set up first, staying in a single project is a reasonable tradeoff.

Sharing a project forces all resources into the same namespace: as long as naming, state, or secrets contracts are not separated, dev and prod are only separated in appearance.

Six Isolation Boundaries to Enforce

I eventually organized this into 6 boundaries.

If you only do 2-3 of these, you will definitely hit issues later.

Separating Module Definitions from Environment Instantiation

Separating resource definition from environment instantiation forms the core strategy.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

infra/terraform/
  modules/
    shared/
    platform/
    service/
  live/
    shared/
    dev/
      platform/
      service/
    prod/
      platform/
      service/

This layout has two benefits:

1. modules/* focuses on describing reusable infra contracts
2. live/* focuses on describing actual environment state, vars, and target wiring

This way dev and prod are two real live roots, not the same root with a variable switch.

Pitfall 1: Shared Statefile Cross-Contamination

Many people start by writing:

1
2
3
4

backend "gcs" {
  bucket = "shared-state-bucket"
  prefix = "platform"
}

Then use environment = "dev" or environment = "prod" to differentiate resource names.

This is not enough.

Because the state is still one copy. As long as this state manages both shared resources and env-specific resources, you will eventually encounter:

• prod apply touches dev resources
• shared resources and env resources drift together
• import and migration become messy
• destroy blast radius becomes unclear

My final resolution was to split the state prefixes directly:

1
2
3
4
5

shared
platform/dev
service/dev
platform/prod
service/prod

This way, even within the same GCS bucket, the ownership boundary is much clearer.

Pitfall 2: Identifier Overlap Persists

In a single project, naming is not cosmetic — it’s part of isolation.

For example, none of these can share names:

• Cloud SQL instance
• Cloud Run service
• Secret Manager secret
• runtime service account
• deploy service account

I eventually consolidated the naming convention to:

1
2
3
4
5
6

app-dev
app-prod
database-url-dev
database-url-prod
deploy-dev-sa
deploy-prod-sa

This sounds basic, but the real pitfall is secret naming.

If you only change the Cloud Run service to app-dev / app-prod, but secrets are still called:

1
2
3

database-url
auth-backend-secret
admin-access-token

Then dev and prod are still sharing the runtime contract.

Pitfall 3: Credential Keys Diverge While Payloads Converge

This is what I think is most worth recording.

Splitting secret IDs like this:

1
2
3
4

database-url-dev
database-url-prod
admin-access-token-dev
admin-access-token-prod

Only means the resource boundary is separated.

It does not mean the values are necessarily different.

This especially happens during migration. Because when first bringing up prod, it’s common to copy the old values first, thinking “let’s get the flow working first.”

Technically feasible, but be very clear this is only a transitional state, not a completed state.

I separate secret separation into two layers:

For true environment isolation, both layers must hold.

Pitfall 4: Conflating Runtime and Deploy Principals

Cloud Run runtime service account and CI/CD deploy service account are two completely different identities.

I initially tended to think of them as the same category, but later realized this is dangerous.

The cleaner model is:

1
2
3
4
5

runtime-sa-dev   -> used by the running dev service
runtime-sa-prod  -> used by the running prod service

deploy-dev-sa    -> used only by deploy-dev workflow
deploy-prod-sa   -> used only by deploy-prod workflow

Permissions map differently to each role:

• runtime SA needs runtime permissions
• deploy SA needs deployment permissions

These separation gains are direct:

• deploy credentials cannot silently become runtime credentials
• prod deploy cannot affect dev by accident
• IAM review becomes readable

Pitfall 5: Per-Stage Variables Need Explicit Workflow Referencing

This part of GitHub Actions is easy to misunderstand.

Many people think that as long as the repo has environment-scoped vars, any job in any workflow can read them. Not true.

The job must explicitly declare:

1

environment: dev

Or:

1

environment: prod

Only the corresponding job will read that environment’s vars.

So environment-scoped variables and workflow routing are tied together.

If a job doesn’t have environment, even if you’ve set the variables in the GitHub UI, the workflow still can’t access them.

Pitfall 6: VCS Protection Rules Cannot Substitute True Tenancy Segregation

dev branch -> dev main -> prod

This policy is important, but it is not environment isolation itself.

The correct order should be:

1. isolate the environments first
2. then add branch governance

That is, first get:

• state
• names
• secrets
• identities
• workflow entrypoints

properly separated, then add branch boundaries.

Otherwise you get an illusion:

• It looks like main only deploys prod
• But underneath, state or secrets are still being shared

Resulting Pipeline Layout

I converged to this model:

1
2
3
4

CI             -> pull request validation only
Deploy Dev     -> deploy dev service only
Deploy Prod    -> deploy prod service only
Infrastructure -> shared/dev-platform/prod-platform plan/apply

The benefit is that operators looking at the workflow list know exactly what each line does.

I deliberately avoided making environment routing into one overly clever mega-workflow. Because that style looks DRY at first, but later debugging, permission auditing, and branch guards all become harder to read.

Validating That Isolation Is Real

In the end, it’s not about how pretty the repo looks, but whether you can clearly answer “yes” to these questions:

• Does dev deploy never touch prod state
• Does prod deploy never touch dev state
• Are runtime secrets isolated at both the name and actual value layers
• Does the production domain belong only to prod
• Can CI identity and deploy identity be audited separately
• Can operators know which workflow to run without reading source code

If any of these still can’t be answered, the env split is not truly complete.

Main Takeaway

Keeping dev and prod in the same GCP project is doable, but you can’t rely on just branch or surface-level naming to separate them.

What actually works is breaking isolation into multiple boundaries:

• state
• names
• secrets
• runtime identity
• deploy identity
• workflow governance

As long as all these layers are cleaned up together, a single project can run a maintainable dev / prod model. When you’re ready to upgrade to project-per-environment, it’s more of an expansion than a rebuild.

References

• Cloud Run documentation
• Cloud SQL for PostgreSQL
• Secret Manager documentation
• GitHub Actions documentation
• Workload Identity Federation