This article outlines the tools used by the Snowplow team to monitor a pipeline.
Monitoring
Components emit statsd metrics which are collected by Container Monitor and pushed to Cloudwatch.
Components are deployed via Terraform stacks with cloud resources wrapped around. These produce managed service metrics form the cloud provider that appear in Cloudwatch.
Snowplow uses the combined metrics to feed into alerts in the cloud provider which when fired will generate OpsGenie alerts that our Support team monitor.
The combined metrics are used within Snowplow to drive a number of Grafana dashboards that show the workload going through components / pipelines and help Support to manage latency / lag issues or investigate service issues.
Snowplow monitors at a workload, an O/S, a host, an infrastructure, and at a service level. This is reflected in the Grafana dashboards and the alerts that we have in place.
A number of time-series based metrics are pushed through to Snowplow operational warehouse for display in BDP Console and used in Data Quality Dashboards. They also feed into datamodels for ensuring that data transformation is keeping to normal times and for calculating that Snowplow is keeping to SLAs, amongst other things. No customer data is pushed to the Snowplow operational warehouse, this data on how the pipeline is functioning and its throughput.
The metrics and alarms within the AWS account that Snowplow is being operated in, are visible to the customer and to Snowplow. Customers can create their own dashboards based on the metrics present.
Access Control
Grafana dashboards are accessible by Snowplow individuals when they are connected to the Snowplow VPN.
BDP Console uses organization based OAuth permissions to control access, so individuals can only see metric information for organizations that they have access to. Individuals in Snowplow will be given access to specific customers to support onboarding or to support ongoing accounts, or in Support they will have wider organization access for day to day support.
The Snowplow operational warehouse has vendor recommended access controls applied.