Atlas Stream Processing Monitoring and Alerting

Atlas Stream Processing provides monitoring and alerts so that users can leverage performance and status insights to refine their workflows.

Monitor Stream Processor Instances in the Atlas UI

For each of your stream processing instances you can monitor your stream processors in the Atlas UI:

In Atlas, go to the Stream Processing page for your project.

WARNING: Navigation Improvements In Progress

We're currently rolling out a new and improved navigation experience. If the following steps don't match your view in the Atlas UI, see the preview documentation.

If it's not already displayed, select the organization that contains your project from the Organizations menu in the navigation bar.
If it's not already displayed, select your project from the Projects menu in the navigation bar.
In the sidebar, click Stream Processing under the Services heading.
The Stream Processing page displays.

Select the stream processing instance you want to monitor.

In the pane of the stream processing instance you want to monitor, click Configure.

Click the Monitoring tab.

The Monitoring tab displays a variety of runtime statistics about a stream processor of your choosing, including, but not limited to:

Number of messages ingested
Number of messages successfully processed
Number of messages sent to your Dead Letter Queue

If your source connection is Apache Kafka, you can monitor the lag between the current offset and the latest offset at the broker for a topic's partition and the sum of all the partition lags.

Filter the monitoring charts as needed.

You can filter the charts by stream processor name, time range, and granularity.

Stream Processor Monitoring Methods

Atlas Stream Processing provides the following methods for on-demand reporting about your stream processors:

The sp.processor.sample() method allows you to see a small sample of the documents output by a currently running stream processor of your choosing. Users can compare the sampled results against their expected results to diagnose any errors in their aggregation pipeline design.

The sp.processor.stats() method returns a variety of runtime statistics about a stream processor of your choosing, including, but not limited to:

Number of messages ingested
Number of messages successfully processed
Number of messages sent to your Dead Letter Queue
In-memory size of your pipeline state
Pipeline definition

If your source connection is Apache Kafka, you can monitor the following optional metrics:

partitionOffsetLag indicates the lag between the current offset and the latest offset at the broker for a topic's partition.
kafkaTotalOffsetLag indicates the sum of all the partition lags.

Stream Processor Metrics in Datadog

You can send metrics to Datadog to monitor your stream processors. To learn how to configure the integration and which metrics are available, see Integrate with Datadog.

Stream Processor Alerts

Atlas Stream Processing triggers alerts when processors change state, or a processor meets various ingestion or output thresholds. For a list of available Atlas Stream Processing alerts, see Atlas Stream Processing Alerts. To learn more about alert configuration, see Configure Alert Settings.

You can target Atlas Stream Processing alerts in the following ways:

All stream processors within a project
All stream processors within a stream processing instance matching the configured predicate
All stream processors the names of which match the configured predicate

For targets other than all stream processors, you can configure multiple targets for the same alert.

Back

Security

Limitations