How to Configure AWS PrivateLink for Snowplow Pipeline Connectivity

Overview

This article aims to help organisations establish secure, private connectivity between their Snowplow pipeline and customer target destinations such as Snowflake, Redshift, Databricks, and more, without exposing data to the public internet. This can be for loading or for datamodeling.

What is PrivateLink?

AWS PrivateLink provides private connectivity between VPCs, AWS services, and on-premises applications, securely on the Amazon network. It eliminates the exposure of data to the public internet and simplifies network architecture.

See official AWS docs for more details: https://aws.amazon.com/privatelink/

When enabled, PrivateLink creates a private endpoint in the Snowplow VPC that connects directly to the service provider's VPC, ensuring all communication remains within AWS's private network infrastructure.

• Traffic stays within the AWS network backbone
• No need for Internet Gateway, NAT devices, or VPN connections
• Simplified network management and improved security posture

Supported Destinations

Snowplow currently supports AWS PrivateLink for the following destinations. Use the tabs below to follow setup instructions based on your destination platform.

Limited PrivateLink Support: While we support PrivateLink connections for data loading to certain destinations, data modelling jobs that run from within your Snowplow VPC may not yet work over PrivateLink endpoints. This means your pipeline can load data privately, but SQL transformations may still require standard connectivity. See destination-specific tabs below for current capabilities and workarounds.

Used when your Snowplow pipeline needs to connect to Snowflake via AWS PrivateLink for enhanced security and performance.

Configure AWS PrivateLink between your Snowplow pipeline and Snowflake data warehouse for secure, private data loading.

This process requires coordination between your organization, Snowplow Support, and Snowflake Support. Setup time may vary depending on establishing connection and working through any customer configuration requirements.

Prerequisites

Before beginning the PrivateLink setup, ensure you have:

Follow the steps outlined in Snowflake's AWS PrivateLink documentation
You will be provided the Snowplow AWS Account ID (available in PMC or from Snowplow Support). Provide this to Snowflake Support to enable PrivateLink between accounts
Since you will perform the initial "Enable Private Link" steps for your Snowflake account as outlined in Snowflake's documentation, which involves generating a get-federation-token via an identity and access management user in AWS or your AWS account root user, we recommend requesting Snowflake Support to enable PrivateLink using the provided Snowplow AWS Account ID.
You will need to provide Snowplow Support with:
- privatelink-vpce-id from the query mentioned in Snowflake docs
- "regionless-privatelink-ocsp-url": "<privatelink_org_ocsp_url>"
- "privatelink-account-url": "<privatelink_account_url>"
- "privatelink_ocsp-url": "<privatelink_ocsp_url>"

Configuration Process

Request the AWS Account ID for your pipeline from Snowplow Support. Contact Snowflake Support and request them to enable PrivateLink between your Snowflake account and the Snowplow AWS Account ID. Once enabled, run this query in Snowflake:

SELECT SYSTEM$GET_PRIVATELINK_CONFIG();

From the query results, extract these four configuration values:

privatelink-vpce-id
regionless-privatelink-ocsp-url
privatelink-account-url
privatelink_ocsp-url

Submit a support ticket with all four configuration values to Snowplow Support. We'll configure the PrivateLink endpoint and update your loader configuration to use the private connection.

Recommended approach: Since Snowplow cannot perform the initial "Enable Private Link" steps in your Snowflake account, we recommend requesting Snowflake Support to complete the entire PrivateLink enablement process using the provided Snowplow AWS Account ID.

All data loading from Snowplow to Snowflake will occur over the private connection
No changes are required to your existing schemas or data models

Used when your Snowplow pipeline needs to connect to Amazon Redshift via AWS PrivateLink for enhanced security and performance.

Setup overview

Snowplow supports two types of AWS PrivateLink connections to Amazon Redshift:

This process requires coordination between your organization and Snowplow Support. Setup time may vary depending on establishing connection and working through any customer configuration requirements.

OPTION 1: Redshift-managed PrivateLink

Prerequisites

Before beginning the PrivateLink setup, ensure:

Your Snowplow pipeline is in the same AWS region as your Redshift cluster - AWS Restriction
Your Redshift cluster is enabled for Relocation or Redshift is Multi-AZ
Your Redshift cluster grants access to the Snowplow AWS Account (either all VPCs or specify the Snowplow VPC ID)
You can only have one AWS PrivateLink endpoint per Snowplow VPC
For additional prerequisites, see Considerations when using Redshift-managed VPC endpoints

Configuration Process

Request AWS PrivateLink setup for Redshift from Snowplow Support. We will provide you with:

The Snowplow AWS Account ID for your pipeline
The Snowplow VPC ID (if you choose not to allow all VPCs from the account)

Navigate to your Redshift cluster in the AWS console
Go to the "Properties" tab
Scroll down to "Cluster Access"
Click "Grant access"
Input the Snowplow AWS Account ID provided by support
Choose either:
- Allow all VPCs from the Snowplow AWS Account, or
- Specify the exact Snowplow VPC ID provided by support

Submit a support ticket or within an existing ticket provide the following information:

Your AWS Account ID where the Redshift cluster resides
Redshift cluster identifier/name
Redshift port (typically 5439)

Snowplow will deploy a Redshift PrivateLink endpoint within the Snowplow pipeline VPC to establish connection.

OPTION 2: Service NLB Interface-type PrivateLink

Prerequisites

Before beginning the PrivateLink setup, ensure:

Your Snowplow pipeline is in the same AWS region as your Redshift cluster - AWS Restriction
Your VPC Endpoint Service subnet availability zones must match your Snowplow VPC availability zones. We currently only deploy to two availability zones, so without matching we won't be able to establish a connection
VPC Endpoint Service configured with the NLB as target
Security Groups on NLB configured to allow traffic from Snowplow VPC CIDRs - Can be supplied by Support team
Target Group with Redshift cluster as target

Configuration Process

Request Service NLB Interface-type PrivateLink setup for Redshift from Snowplow Support. We will provide you with:

The Snowplow AWS Account ID for your pipeline
The VPC Endpoint Service IAM Principal: arn:aws:iam::<SnowplowAWSAccountID>:role/SnowplowDeployment
The Snowplow VPC availability zones that must be matched

Set up the required AWS resources in your account:

Create a Network Load Balancer (NLB) with subnets in availability zones matching Snowplow VPC
Create a Target Group with your Redshift cluster as the target
Create a VPC Endpoint Service pointing to the NLB
Configure Security Groups on the NLB to allow traffic from Snowplow VPC CIDRs - Can be supplied by the Support team

Submit a support ticket with the following information:

VPC Endpoint Service name
Your AWS Account ID
Redshift cluster endpoint and port
NLB availability zones configured

Snowplow will create a VPC Interface Endpoint in the Snowplow pipeline VPC to connect to your VPC Endpoint Service.

Important: The Redshift cluster and Snowplow pipeline must be in the same AWS region for PrivateLink to function properly. This is an AWS requirement that cannot be bypassed.

Post-Configuration

All data loading from Snowplow to Redshift or datamodeling to Redshift will occur over the private connection
No changes are required to your existing schemas or data models