GCP Custom Domain SSL Certificate Management

Each GCP pipeline's load balancer supports 5 IP addresses. IP 01 is reserved for the Snowplow default domain; IPs 02–05 are available for custom domains.

Each IP can have up to 15 certificates attached in total. This is a GCP limit on the target HTTPS proxy and it is shared across every certificate type Google-managed and customer-managed slots both count against the same 15.

Per IP the pipeline exposes two kinds of slot:

"Customer-managed" means the customer created the certificate resource, not that they uploaded key material. A customer-managed slot holds the name of a classic SSL certificate that already exists in the GCP project, and that certificate can be either Google-managed or imported. Both kinds live in the same compute.sslCertificates collection, so one slot type handles both, the handlers detect which by reading the certificate's type field.

Certificate ordering and the primary certificate

All certificates on one IP attach to the same target HTTPS proxy, in a fixed order:

google-managed 01→05,  then  customer-managed 01→15,  then  snowplow-managed 01→05

The first certificate in that list is the load balancer's primary certificate — the one served to clients that send no SNI or an SNI hostname that matches nothing.

Which path do I use?

There are three options. The first two both auto-renew and need no maintenance.

Default to A. Fewest moving parts: the customer creates one A record and we do everything else. Google proves ownership by checking the domain resolves to the load balancer IP, so we never need access to the customer's DNS zone - this works fine when the customer manages DNS externally.

Use B1 when the customer wants to own the certificate resource themselves, or already has access to the project and prefers to manage it there. It renews automatically just like A - the only difference is who clicked create. Do not describe this to the customer as something they will need to renew.

Use B2 only when the customer requires a specific CA, an EV/OV certificate, a wildcard, or must use a certificate they already hold. This is the only option that creates ongoing renewal work, so treat it as a last resort.

Neither A nor B1 works if the domain sits behind Cloudflare or a similar proxy in non-DNS-only mode - Google cannot validate through a proxy. Use B2 there.

Step 1: Check existing resources

Check which IPs are deployed and their addresses:

@Snowdroid deployHandler gcp/get_ips 0.1.0 false for com.acme-env1 stack:gcp_rt_pipeline

Custom domains start at IP 02. If an IP is already deployed, check its used certificate slots:

@Snowdroid deployHandler gcp/get_certificates 0.1.0 false for com.acme-env1 stack:gcp_rt_pipeline;ip_num:<IP Num>

For example:

@Snowdroid deployHandler gcp/get_certificates 0.1.0 false for com.egoditor-prod1 stack:gcp_rt_pipeline;ip_num:02

Note: get_certificates only lists the customer-managed slots. It does not show Google-managed certificates. To see those, read the Consul keys directly at /input/load_balancer/ip/<NN>/google_managed_certificate/ or use check_certificate with the generated cert name (see Step 4).

Step 2: Deploy a new IP (only if needed)

If no custom-domain IP is deployed, or the existing ones are at capacity, enable the next free IP number:

@Snowdroid deployHandler gcp/enable_ip 0.1.0 false for com.acme-env1 stack:gcp_rt_pipeline;ip_num:<IP Num>

Then retrieve the new address:

@Snowdroid deployHandler gcp/get_ips 0.1.0 false for com.acme-env1 stack:gcp_rt_pipeline

If an IP is already deployed with free certificate slots, reuse it and skip this step.

Step 3: Choose a path

Go to Path A for a Snowplow-provisioned Google-managed certificate, or Path B for a customer-provisioned one.

Path A - Snowplow-provisioned Google-managed certificate

A1. Reply to the customer

Dear <CUSTOMER>,

Could you please create an A record pointing collector-url to IP Address?

That is all we need from you — once the record is live, Google will automatically issue and renew a free TLS certificate for the domain. There is no certificate for you to create, purchase, or renew.

Two things to check on your side:

- If your DNS zone has CAA records, please make sure pki.goog is permitted, e.g. 0 issue "pki.goog". If you have no CAA records at all, nothing is needed.

- The record must point directly at the IP address. If the domain is proxied through Cloudflare or similar, please set it to DNS-only mode, otherwise certificate issuance will fail.

Certificate issuance usually completes within an hour of the record going live, but can take up to 24 hours. We will confirm once it is active.

Looking forward to hearing from you,

A2. Set the domain in Consul

There is no Snowdroid handler for this path. Edit the key directly, using the next free Google-managed slot (01–05) on the chosen IP:

/customer/com_acme/gcp_rt_pipeline_env1/input/load_balancer/ip/<NN>/google_managed_certificate/<MM>/domains

The value is a comma-separated list of domains, e.g. c.acme.com or c.acme.com,c2.acme.com.

A non-empty value creates the certificate; emptying the value destroys it. There is no separate enable flag.

Apply:

@Snowdroid deployUpdate gcp_rt_pipeline true for com_acme-env1
@Snowdroid deployUpdate gcp_rt_pipeline false for com_acme-env1

Order matters. Apply this before the customer points DNS at the load balancer. If they point the record first, that hostname will serve TLS errors until the certificate issues. If they have already created the record, that is not a problem, just apply as soon as possible.

The certificate is created immediately in PROVISIONING state with a per-domain status of FAILED_NOT_VISIBLE. This is expected and is not an error. It resolves on its own once DNS resolves to the load balancer IP.

The certificate name is generated automatically as:

sp-ip-<NN>-google-managed-certificate-<MM>-<env_name>

For example, slot 02 on IP 03 of a prod1 pipeline is sp-ip-03-google-managed-certificate-02-prod1.

Continue to Step 4.

Path B - Customer-provisioned certificate

Use this when the customer wants to create the certificate themselves. Steer them to a Google-managed classic certificate (B1) unless they have a specific reason to import their own (B2) - B1 auto-renews and creates no future work for either side.

B1. Reply to the customer

Dear X,

Could you please create the A record(s) that point collector-url to IP Address?

After creating the A records, please create a Google-managed SSL certificate in the project:

https://cloud.google.com/load-balancing/docs/ssl-certificates/google-managed-certs#create-classic

• Please create a classic certificate, not a Certificate Manager one - we cannot attach the latter.

• Google-managed certificates are free and auto-renew, so there is no renewal work for you or for us.

• If you need several domains, a single multi-domain certificate is easiest.

• Alternatively (not preferred) you can purchase an SSL certificate and import it into the project - but you would then be responsible for replacing it before it expires.

• Either way, please send us the certificate name once created.

• We will attach it to the collector's load balancer.

Two things to check on your side:

• If your DNS zone has CAA records, please make sure pki.goog is permitted, e.g. 0 issue "pki.goog". If you have no CAA records at all, nothing is needed.

• The record must point directly at the IP address. If the domain is proxied through Cloudflare or similar, please set it to DNS-only mode, otherwise certificate issuance will fail.

Hope this all makes sense! Looking forward to hearing from you,

B2. Attach the certificate

Once the customer confirms the certificate exists, attach it to the first free slot on the IP:

@Snowdroid deployHandler gcp/attach_certificate 0.1.0 false for com.acme-env1 stack:gcp_rt_pipeline;ip_num:<IP num>;cert_num:<cert_slot>;cert_name:<cert_name>

For example:

@Snowdroid deployHandler gcp/attach_certificate 0.1.0 false for com.egoditor-prod1 stack:gcp_rt_pipeline;ip_num:02;cert_num:02;cert_name:qrcg-snowplow-cert

The handler validates that the IP is deployed and that the slot is free, writes the certificate name to Consul, and applies the stack.

To remove a certificate later:

@Snowdroid deployHandler gcp/detach_certificate 0.1.0 false for com.acme-env1 stack:gcp_rt_pipeline;ip_num:<IP num>;cert_num:<cert_slot>

Possible error - Null values not allowed

╷
│ Error: Null value found in list
│
│   with module.load_balancer[0].module.ip_02[0].google_compute_target_https_proxy.lb_target_https_proxy[0],
│   on ../../../modules/gcp_load_balancer_ip/1.0.0/ip.tf line 58, in resource "google_compute_target_https_proxy" "lb_target_https_proxy":
│   58:   ssl_certificates = local.ssl_certificates
│
│ Null values are not allowed for this attribute value.

The certificate lookup returned nothing, meaning the certificate was not created correctly. Common causes: created in the wrong GCP project, or created as a Certificate Manager certificate instead of a classic certificate.

The customer will need to upload a new certificate. Detach the bad slot before retrying.

Continue to Step 4.

Step 4: Check the certificate status

@Snowdroid deployHandler gcp/check_certificate 0.1.0 false for com_acme-acc1 cert_name:<cert_name>

For example:

@Snowdroid deployHandler gcp/check_certificate 0.1.0 false for de_zeit-acc1 cert_name:sp-collector-academics-dach

Use the account stack associated with the pipeline here, not the pipeline environment.

Pass the generated name from Path A step A2, or the customer's certificate name for Path B. The handler detects the certificate type: for a MANAGED certificate it prints the managed block with an overall status plus a per-domain status; for an imported one it prints the expiry, SANs, and creation time.

An imported certificate has no provisioning step - it is usable roughly 10 minutes after attaching. Any Google-managed certificate, whether we created it or the customer did, can sit in PROVISIONING for several hours and is only guaranteed within 24 hours of DNS resolving.

Do not proceed to Step 5 until the status is ACTIVE.

Step 5: Add custom domain records

Once all domains are ACTIVE, add the custom DNS and cookie records in Consul:

Custom DNS

/customer/com_acme/gcp_rt_pipeline_env1/input/collector/custom_dns_records/edit

Cookie domains

/customer/com_acme/gcp_rt_pipeline_env1/input/collector_cookie_domains/edit

Apply the changes:

@Snowdroid deployUpdate gcp_rt_pipeline true for com_acme-env1
@Snowdroid deployUpdate gcp_rt_pipeline false for com_acme-env1

What custom_dns_records actually does: it registers the hostname with checkup health monitoring (/health) and publishes it in the pipeline metadata. It does not create DNS records and has no effect on certificate issuance. It is not required for the certificate to work, but the domain will be unmonitored without it - always add it.

Step 6: Check the custom domain

Hit the collector health endpoint on the new domain, e.g. https://c.acme.com/health. If it returns OK and there are no alerts, the domain is live. Inform the customer and close the ticket.

Reference - what is configurable

All keys are under /customer/<client>/gcp_rt_pipeline_<env>/.

IP 01 is a special case. Google-managed slot 01 on IP 01 is hardwired to the pipeline's own dns_record_name and has no Consul key. Slots 02–05 on IP 01 are configurable as normal. IP 01 is also the only IP with an auto-created Route 53 record; IPs 02–05 have no DNS created for them at all, which is why the customer must create the A record.

Renewal

Only B2 generates renewal tickets. If a customer is on B2 without a hard requirement for it, migrating them to B1 or A removes the recurring work.