Parallel Kinesis processing without KCL leases or SQS bridges
A pattern I have seen in more than one team: a Lambda function hangs off a Kinesis stream and does nothing except copy records into an SQS queue, so that a pool of workers can process them in parallel. It works, but it is extra code to run, pay for and monitor. The alternative, the Kinesis Client Li

brary, brings its own baggage: shard leases in DynamoDB, checkpoints, and one poisoned record that can hold up an entire shard. On November 17, 2026, AWS launched queues for Kinesis Data Streams, and they target exactly this gap. I tried them out with a small ticket-shop example and wrote everything down in Kinesis Data Streams queues: workers without shard leases. Here are the key points.
A classic stream consumer reads a shard like a lane, front to back, and only one reader can own a lane at a time. More instances than shards gain you nothing. A queue attaches to one on-demand stream and tracks the delivery state of every single record. Workers ask for batches of up to ten records, process them and acknowledge each one.
If a worker does not acknowledge before the visibility timeout (default 30 seconds) expires, the record becomes visible again for the next worker. After too many failed deliveries (six by default, configurable via MaxReceiveCount), it moves to a dead-letter state that you can inspect and restore. AWS says SQS runs underneath, and it feels like it: receive, process, acknowledge. Producers and existing stream consumers notice nothing.
You still write once, and an analytics application can keep reading in order while a queue fans the same records out to parallel workers. Queues give up two things: guaranteed order and exactly-once delivery. That leads to a simple rule of thumb for me: If processing a record depends on earlier records (running totals, sessions, "created" before "cancelled"), stay with a stream consumer. If every record stands on its own (an email, a webhook, a search index update), a queue is a strong fit.
If two components only talk point-to-point and the data is not in Kinesis anyway, SQS remains the starting point. I would not rebuild a KCL application that runs fine just because something new exists. But stream-to-SQS bridges are good candidates to retire. At launch there is no CloudFormation resource and no direct Lambda trigger for queues, so the worker is a long-running process, in my case on Fargate.
Three details make the difference: Partial acknowledgment. Process the batch in parallel and acknowledge only what succeeded. Also check the Failed list of the acknowledge response, because HTTP 200 does not mean every entry worked. Backoff per record.
Instead of waiting for the fixed visibility timeout, failed records get a growing delay via UpdateQueueRecordsVisibility, based on their ReceiveCount. Idempotency. With at-least-once delivery, two workers can hold the same record. A conditional write with a short lease in DynamoDB decides who sends the email.
Simplified: const claim = await store.claim(`ticket-mail#${ticket.orderId}`); if (claim === "done") return; // already sent, just acknowledge if (claim === "busy") throw new BusyError(); try { await sendMail(ticket); await store.complete(key); } catch (err) { await store.release(key); throw err; } Scaling is refreshingly decoupled: the Fargate service scales on the VisibleQueueRecords metric, independent of the shard count. I keep at least one task running, because the queue stops publishing metrics after a while without visible records, and with zero workers no scale-out signal would ever arrive. Queues only work with on-demand streams. Records that are never acknowledged, dead letters included, disappear with the stream retention.
And a queue counts toward the registered-consumer limit it shares with enhanced fan-out. On cost, receiving is priced like regular stream data retrieval, and you save the lease table and the KCL metrics, but a Fargate task running around the clock is on the bill too. A comparison of queues, KCL, Lambda triggers and SQS by order, parallelism and failure behavior A hands-on AWS CLI walkthrough: create, receive, acknowledge, observe redelivery and clean up The complete TypeScript worker with long polling, backoff, partial acknowledgment and graceful shutdown on SIGTERM A CDK stack with a custom resource for the queue, Fargate autoscaling on the backlog and a dead-letter alarm Common mistakes, limits, cost and security notes, plus tests with aws-sdk-client-mock Read it here: Kinesis Data Streams queues: workers without shard leases π This is the condensed version. The full article on sascha-ms.de goes deeper β with all code samples, diagrams and a short knowledge check.
π Next step: Need a schedule in EventBridge? My EventBridge cron helper builds and explains the expression for you. π¬ Stay updated: New articles on serverless, AI and leading tech teams via RSS, browser push or the weekly newsletter (in German). Originally published on sascha-ms.de.
More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en
Key Takeaways
- β’On November 17, 2026, AWS launched queues for Kinesis Data Streams, and they target exactly this gap.
- β’This story was reported by Dev.to, covering developments in the dev space.
- β’AI advancements continue to reshape industries β read the full article on Dev.to for complete coverage.
π Continue reading the full article:
Read Full Article on Dev.to βShare this article



