Kinesis Data Streams Overview
Amazon Kinesis Data Streams is a foundational service for real-time data ingestion and processing within the AWS ecosystem.
Kinesis data streams collect and process large streams of data in real time, generated continuously by up to thousands upon thousands of data sources. Your applications can process that data in that stream sequentially and incrementally on a record by record basis or over a sliding time window.
Suitable for real-time data analytics, log and data feed intake and processing, real-time metrics and reporting.
Not suitable for small scale consistent throughput and long-term data storage and analytics.
Kinesis Data Streams is not available on the Free Tier and is billed on a per-shard basis.
Kinesis Data Streams offers live and continuous data processing and analysis. It is a managed service, providing elasticity and durability, replicated across three Availability Zones. It's great for implementing process and safety alarms, and allows multiple applications to consume data from the same stream.
Technical Specs: Replicated across three Availability Zones
Data retention period is from a default of 1 day up to a maximum of 7 days (at an additional cost).
Technical Specs: Default 1 day, up to 7 days (additional cost)
Server-side encryption using an AWS KMS customer master key (CMK) is available and simple to implement.
Technical Specs: Uses AWS KMS customer master key (CMK)
Kinesis Data Stream Architecture
A Kinesis Data Stream is composed of shards and data records, forming the fundamental structure for streaming data.
An implementation consists of producers pushing data into the stream, and consumers continuously processing data from the stream.
Stream
The overall Kinesis Data Streams service that collects and processes data. It is made up of shards.
Shards
Shards are mini streams within a stream. The more shards you have, the greater the capacity of that stream. Each shard is uniquely identified and contains a continuous sequence of data records. A shard is not a piece of data; it's a continuous stream of data.
Data Records
Each data record within a shard has three main components: a sequence number, a partition key, and a data blob.
Data Records - Sequence Number
A unique record identifier assigned by Kinesis when the record is created. It is not assigned by your producer.
Data Records - Partition Key
A string that you define in your producer application. Kinesis maps this partition key to a specific shard. Subsequent data with the same partition key prefix will go to the same shard, making data retrieval easier for consumers.
Data Records - Data Blob
Contains the actual data being streamed.
Writing to Kinesis Data Streams
Producers push data into Kinesis Data Streams using various methods, adhering to specific capacity limits and requiring certain parameters.
Data can be put into streams using API calls (HTTP PUT), SDK, Amazon Kinesis Producer Library (KPL) C++ application, or Kinesis Agent java application.
Each shard can support up to 1,000 PutRecords per second and provides a capacity of 1 MB per second of data input.
Technical Specs: 1,000 PUT records per second; 1MB/sec data input per shard
A producer will collect live data and run a PutRecords command. This command requires the name of the stream, a partition key, and the data itself.
Technical Specs: Command: PutRecords
Reading from Kinesis Data Streams
Consumers read data sequentially from Kinesis Data Streams using shard iterators and the GetRecords command.
Each shard can support up to 5 transactions per second and provides a capacity of 2 MB per second of data reads.
Technical Specs: 5 transactions per second; 2MB/sec data reads per shard
A shard iterator identifies a position in the shard from which to start reading data records sequentially. It is supplied by the Kinesis service and is required to read a stream.
Different shard iterator types can be specified depending on where in the stream you want to get data.
Starts reading at a specific sequence number.
Starts reading after a specific sequence number.
Starts reading from a specific point in time.
Retrieves the oldest data in the stream.
Retrieves the latest data in the stream.
The shard iterator is passed to GetRecords, which can be done through the SDK or Kinesis Client Library. GetRecords returns the milliseconds behind the latest record, the next shard iterator, and an array of records.
First, call GetShardIterator, supplying the shard ID, shard iterator type, and stream name. Then, use that shard iterator with a GetRecords command to retrieve records.
Technical Specs: API calls: GetShardIterator, GetRecords
Scaling Kinesis Data Streams
Kinesis Data Streams is a fully managed service that makes it easy to scale up or down depending on capacity needs, a process known as resharding.
Resharding is the process of increasing or decreasing the number of shards in a stream using an UpdateShardCount command in the SDK or CLI.
Technical Specs: Command: UpdateShardCount
A shard split will increase the shard count.
A shard merge will decrease the shard count.
The Kinesis Client Library (KCL) cannot initiate a resharding process but can adapt when it occurs. Kinesis will handle the mapping of partition keys to the new shard count.
It is possible to automate the process of scaling Kinesis by implementing an SDK application that can monitor using CloudWatch and use a CloudWatch alarm to initiate the resharding process.
Before resharding occurs, a shard will be in the OPEN state, allowing read and write operations.
After a resharding operation, the shard will transition to the CLOSED state. New records can no longer be added to this shard; they will be added to the newly created shards. Records can still be read from a closed parent shard.
After the stream's retention period (e.g., one day or up to seven days) has expired, a closed parent shard will no longer be accessible for reading or writing and will not contain data. It's important to read all necessary data from closed shards before they expire.
Kinesis Stream Throughput Calculation
procedure
To identify the number of shards needed for a Kinesis stream, follow this quick process to determine capacity requirements.
A process to determine the number of shards needed for a Kinesis Data Stream based on average record size, records per second, and number of consumer applications.
Prerequisites
- Average record size
- Expected records per second
- Number of concurrent consumer applications
1
Identify the average record size, rounded to the nearest 1 KB (Sizerecords).
This is the baseline unit for bandwidth calculations.
2
Identify the rate of records per second that will be going into the stream (RATErecords).
Determines the total incoming data volume.
3
Calculate the incoming write bandwidth in kilobytes per second (BWwrite). This is Sizerecords multiplied by RATErecords.
This represents the total write capacity needed for the stream.
BWwrite (kB/s) = Sizerecords (kB) x RATErecords (records/sec)
4
Identify the number of Kinesis applications that will concurrently be consuming data from the stream (NUMconsumers).
Each consumer application can impact the read capacity requirements.
5
Calculate the total read bandwidth in kilobytes per second (BWread). This is NUMconsumers multiplied by BWwrite.
This represents the total read capacity needed across all consumers.
BWread (kB/s) = NUMconsumers x BWwrite (kB/s)
6
Determine the number of shards required. This will be the maximum of either the write bandwidth divided by 1000 (1 MB) or the read bandwidth divided by 2000 (2 MB).
A shard has a write capacity of 1 MB/s and a read capacity of 2 MB/s. The calculation ensures enough shards for both read and write demands.
Number of Shards = Max (BWwrite / 1000, BWread / 2000)
Kinesis Data Streams Development Tools
AWS provides a suite of tools and libraries to simplify the development and implementation of Kinesis Data Streams applications.
These tools cater to both producer (data ingestion) and consumer (data processing) applications, as well as integration with other AWS services.
AWS SDKs
Software Development Kits are available for various programming languages to interact with Kinesis Data Streams APIs.
Use Cases:
- Developing custom producer and consumer applications
Kinesis Client Library (KCL)
Recommended by AWS for developing Kinesis consumer applications. It handles complex tasks like adapting to resharding, fault tolerance, and load balancing across multiple consumer instances.
Supported languages
Java, Python, Ruby, .NET, Node.js
Use Cases:
- Building consumer applications
Kinesis Connector Library
Integrates Kinesis Data Streams with other AWS services.
Integrates with
DynamoDB, Redshift, Amazon S3, and Amazon Elasticsearch Service
Use Cases:
- Directing stream data to durable storage or processing services
Kinesis Producer Library (KPL)
Simplifies the process of combining small records into larger ones (aggregation) to maximize effectiveness and efficiency when writing to streams.
Use Cases:
- Optimizing data ingestion for efficiency
Kinesis Agent
A pre-built Java application designed to collect and send data to a Kinesis stream. It's great for creating producer applications without extensive custom code.
Use Cases:
- Collecting and sending data to a stream (e.g., log files)
Kinesis Data Generator
A tool that produces test data to a stream, useful for testing consumer applications.
Use Cases:
- Generating synthetic test data for consumer application development and testing
Kinesis Data Streams Integrations
Kinesis Data Streams can integrate with other AWS services to build comprehensive streaming data solutions.
Kinesis Firehose can capture, transform, and load streaming data into Amazon Kinesis Analytics, Amazon S3, Amazon Redshift, and Amazon Elasticsearch Service. It can reliably deliver data to S3 with minimal operational overhead.
Kinesis Data Analytics enables the processing and analysis of streaming data content using standard SQL statements. It can process streaming location data in near real-time for analytics.
Kinesis Data Streams can collect real-time data which is then delivered to an Amazon S3 data lake via Kinesis Data Firehose for scalable analytics.
After data is collected by Kinesis Data Streams and delivered to an S3 data lake (e.g., via Firehose), it can then be loaded into Amazon Redshift for scalable analytics.