Atlas Stream Processing Changelog

Adds support for the AWS S3 private link connection type.

Adds the Test Connection button in the Connection Registry UI to initiate a connectivity test for each connection.

Adds the tenant_name, project_name, and org_name fields to Atlas Stream Processing metrics, including those exported to Datadog.

Adds lifetime cache hit/miss statistics to the stats.operatorStats of the :ref`atlas-sp-agg-cachedlookup` stage.

Adds support for the Stream Processor State metric in the Atlas Datadog integration.

Adds the $function operator, allowing for Javascript execution in stream processing pipelines. To learn more, see $function (Stream Processing).

Adds the $cachedLookup operator allowing for $lookup results to be cached for a specific TTL to improve the performance of slowly changing data. To learn more, see $cachedLookup.

Adds the ability for the $emit stage to send null values to Kafka topics to support compacting Kafka Topics by using $emit.config.tombstoneWhen = boolean. To learn more, see $emit Aggregation Stage (Stream Processing).

Adds support for Cross Atlas Project for $source and $merge. To learn more, see Atlas Connection.

Adds support for $source to support Initial Sync when reading from Atlas collections. To learn more, see MongoDB Collection Change Stream.

Adds support for Stream Processing Instances to deploy on Google Cloud. To learn more, see Google Cloud (GCP) Stream Processing Regions.

Adds support for $merge to be able to delete documents in collections. To learn more, see $merge (Stream Processing).

Adds support for Datadog Metrics. To learn more, see Datadog Metrics.

Adds support for AWS S3 connections.

Adds a new $sessionWindow Atlas Stream Processing pipeline aggregation stage that specifies a session window for aggregation of data. This allow you to run a pipeline on each "session" of activity in an input stream. To learn more, see $sessionWindow.

Supports Azure-hosted Confluent Kafka clusters via Private Link connection.

Adds a new $externalFunction Atlas Stream Processing pipeline stage that triggers processes in a specific AWS Lambda resource. To learn more, see $externalFunction.

Supports creating $https connections in the Atlas UI.

Adds the parallelism field to $merge. The field specifies the number of threads to which to distribute write operations, which improves performance.

Allow you to create additional alerts: Output Message Count, DLQ Message Count, Kafka Lag, and Change Stream Delay. To learn more, see Atlas Stream Processing Alerts.

Supports the createUUID expression that takes no arguments and returns UUID BSON type values in Stream Processors.

Adds a new configuration for Window operators to support processingTime in addition to eventTime. To learn more, see Processing Time.

Adds the $meta expression that returns an object containing all streaming metadata for a document.

Adds the parseJsonStrings field to the $https operator, allowing it to parse JSON Strings returned from an API call. To learn more, see $https.

Allows you to add a Kafka Private Link Connection for AWS MSK clusters.

Allows you to deploy Stream Processing Instances on AWS us-east-2. To learn more, see Atlas Stream Processing Regions.

Supports the $currentDate expression that returns the system time of your stream processing instance each time Atlas Stream Processing evaluates it.

Supports reading JSON documents with embedded file signatures (magic bytes).

Fixes an issue that prevented the configuration of hopping windows with hopSize greater than interval.

Changes the executionTimeSecs stat to executionTimeMillis. To view this stat, invoke the sp.processor.stats() command.

Changes the buffering duration for $emit Aggregation Stage (Stream Processing) to Apache Kafka sinks from 1000 milliseconds to five milliseconds.

Supports VPC Peering with Confluent on AWS.

Supports using the Atlas Administration API to edit stream processors.

Supports AWS PrivateLink with Confluent.

Supports Azure Private Link with EventHubs.

Increases Apache Kafka Timeout from 10 seconds to 30 seconds.

Always retries processors when the target cluster is in the REPAIRING state.

Fixes the partitionIdleTimeout after the checkpoint restore.

Adds more watermark and window information to verbose stats.

Ensures calls to produce messages to Apache Kafka don't hang indefinitely, even when the producer queue is full.

Supports ingesting multitopic Apache Kafka sources.

Updates Apache Kafka offsets for consumer groups to near-real time.

Supports the compression.type and acks settings in the Apache Kafka $emit configuration.

Supports collectionless $lookup syntax.

Standardizes the way Atlas Stream Processing serializes Apache Kafka errors into a string.

Increases the $emit and $merge dynamic expression limit to 1000.

Handles $lookup with unique connectionName inside window pipeline.

Improves paused cluster errors.

Supports converting from Binary data to various data types using $convert. This allows you to transform Kafka headers into specific data types as needed.

Stream processors that use Apache Kafka as a source expose per-partition watermarks in sp.processor.stats().

Fixes a bug where users were unable to create stream processing instances in certain regions through the API.