1. Help Center
  2. Community
  3. Ask The Community
Ask The Community
Jahanzeb Younis
Edited

Redshift Loader Fails with "Delimiter not found" Due to Schema Mismatch (1-0-4 vs. 1-0-5)

Description:
Since March 12, 2025, my Snowplow pipeline (using Redshift loader) has stopped loading data into nl_basjes_yauaa_context_1. The issue started after Iglu Central’s Release 165 (March 11, 2025), which added nl.basjes/yauaa_context/jsonschema/1-0-5.
Error Details:
  • Cloudwatch logs: java.sql.SQLException: Load into table 'nl_basjes_yauaa_context_1' failed. Check 'stl_load_errors'.
  • stl_load_errors: err_code: 1214, err_reason: "Delimiter not found", failing on operating_system_version_major with TSV file (schema 1-0-4, 62 fields).
  • Loader error: Table's schema key 'iglu:nl.basjes/yauaa_context/jsonschema/1-0-5' cannot be found in fetched schemas (up to 1-0-4). Migration fails.
Setup:
  • Pipeline: Snowplow with Redshift loader, AWS Redshift, Yauaa enrichment.
  • TSV file: s3://prod-snowplow-events-bucket/transformed/.../yauaa_context/format=tsv/model=1 (schema 1-0-4).
  • Iglu: Using Iglu Central. and i have defined custom schemas for events likes click, view etc
Suspected Cause:
  • Table expects 1-0-5 (post-Release 165), but enrichment outputs 1-0-4.
  • Loader’s Iglu resolver hasn’t synced 1-0-5, causing a field count mismatch.
Steps Taken:
  • Checked stl_load_errors and confirmed TSV has 62 fields.
  • Noted 1-0-5 in Iglu Central changelog but unavailable to loader.
Questions:
  1. How do I ensure the loader fetches 1-0-5 from Iglu Central?
  2. Should I update Yauaa enrichment to 1-0-5 (how?), or revert the table to 1-0-4?
  3. Any known issues with Release 165 and schema migrations?
  •  
0 6 3
Ian Streeter

Hi Jahanzeb Younis, this looks like you are using an old version of RDB Loader.  Older versions had problems with schema evolution, which caused the problems you're describing.

Can you tell us the version of RDB Loader you are using?  I would recommend upgrading to a newer version.  The latest version is 6.1.2.

Jahanzeb Younis

Hi Ian Streeter thank you for the response 
here is the rdb loader version

app_name="rdb-loader-redshift"
app_version="5.6.0"

should i directly upgrade the version in terraform modules ??
will that be a breaking change ??
Josh Beemster

Jahanzeb Younis can you share how you are currently deploying Snowplow? Is this using the Community Edition modules?

Jahanzeb Younis

Hi Josh Beemster, sorry for the late reply—I was away over the weekend. Yes, I’m using the community edition, and deployments are managed through terraform apply. The Iglu server and data pipeline were first deployed in January last year, with a few subsequent deployments for our custom schema updates. I hadn’t faced any issues until this release on March 12, I think.I was able to resolve the issue by forcing the Redshift loader to re-run the migration for the nl_basjes_yauaa_context_1 table. It’s now loading data again with schema version 1-0-4.
However, I’m planning an upgrade for my pipeline since the modules haven’t been updated since their initial deployment. Any guidance on this would be greatly appreciated and would help me avoid hiccups along the way!

Josh Beemster

Hey Jahanzeb Younis I haven't tested this but you can override the versions directly in the quick-start examples by editing the Terraform in the following ways:

1. Navigate to `pipeline/default/target_redshift.tf`

2. Within both the "rs_transformer_stsv" and "rs_loader" add the following input:

```

app_version = "6.1.2"

```

This will override the version from the default which is "5.6.0" currently in the published Community Edition modules.

Jahanzeb Younis

Hi Josh Beemster
I guess i will setup a test pipeline with 5.6.0 version and then update it as you've mentioned.
then i will share any findings for sure