DEA-C01 Practice Question 1: Kinesis Hot Shard Partition Key
AWS Certified Data Engineer – Associate (DEA-C01) · Test 1 · question 1 of 10
A ride-hailing platform publishes trip telemetry to an Amazon Kinesis Data Streams stream running in provisioned mode with 20 shards. Producers set the partition key to the city_id of the trip. Stream-level IncomingBytes shows the stream is running at roughly 25 percent of its provisioned capacity, yet producers log a steady stream of ProvisionedThroughputExceededException errors, and the WriteProvisionedThroughputExceeded metric is non-zero for only two of the 20 shards. The downstream consumer aggregates trips globally and does not depend on per-city ordering.
What should the data engineer do to eliminate the write throttling at the lowest cost?
Show answer and explanation
Answer:
Kinesis Data Streams throughput limits are enforced per shard (1 MB/second and 1,000 records per second for writes), not per stream. Kinesis hashes the partition key to decide which shard a record lands on, so every record for a given city_id is pinned to one shard. A handful of very busy cities therefore saturate a couple of shards while the other 18 sit idle, exactly the signature described: low stream-level utilization plus WriteProvisionedThroughputExceeded on a small number of shards. The AWS troubleshooting guidance for producers is explicit about this case: make sure you have "selected a partition key that allows you to distribute put operations evenly across all shards", and that you do not have "a particular partition key that's bumping into the service limits when the rest are not". Because the consumer does not need per-city ordering, moving to a high-cardinality key such as trip ID is safe and costs nothing.
Option A treats the symptom. Retries with backoff keep the records from being dropped, but the hot shard is still over its per-shard limit, so producer latency and buffered record age keep growing and the throttling never stops.
Option B doubles the shard count, and the bill, without fixing anything. Resharding changes the hash key ranges, but a single partition key value still maps to exactly one shard. The busiest city simply lands on one of the new shards and throttles it just as before.
Option D addresses the read path. Enhanced fan-out gives a consumer a dedicated 2 MB/second read pipe per shard so it does not share the read quota with other consumers. Read and write quotas are separate, so this has no effect on ProvisionedThroughputExceededException on PutRecord or PutRecords, and it adds per-consumer data retrieval charges.
Reference: Troubleshoot Amazon Kinesis Data Streams producers