This article describes a feature or service available to Snowplow BDP customers.
Our RDB Loader provides the ability to load batches of events into Redshift, Snowflake, and Databricks. It is our recommended solution for Redshift and Databricks and runs on both AWS and GCP environments.
Architecture
The RDB Loader is made up of two separate components, a transformer that converts the enriched TSV format into SQL statements, and the loader which copies the events into the warehouse.
Configuration Options
The RDB Loader supports user-password authentication to connect and load events to your data warehouse. Below is a list of the configuration options available.
Transformer
| Configuration Option | Description |
|---|---|
| Since Timestamp/Age | The start timestamp and look-back period that the transformer will load events from. |
| Schedule | The schedule the transformer runs on. More frequent runs are required for pipelines with larger event volumes. |
| In-batch Deduplication | Deduplicate events within the same batch by enabling this option. Requires event fingerprinting. |
Loader
| Configuration Option | Description |
|---|---|
| Host / Port | The endpoint used to connect to your data warehouse. |
| Username | User you intend to load data with. |
| Password | Password for the above user. |
| No Operations Schedule | A crontab representation of time periods when the loader shouldn't run. We don't recommend using this option. |
| Redshift Max Errors | Specifically used when loading to Redshift, this option sets the sensitivity of the loader to STL_LOAD_ERRORS before it fails the batch. |
FAQs
See below frequently asked questions about the RDB Loader.
How can I rotate the user password?
If you're wanting to rotate the password for your RDB Loader, you'll need to reach out to our Support team and share the intended password using the secure Credential sharing form in Console. To avoid disruption to your pipeline, we'll coordinate a time with you to update the credentials.