Kinesis acts as a highly available conduit to stream messages between data producers and data consumers.
Data producers can be almost any source of data: system or web log data, social network data, financial trading information, geospatial data, mobile app data, or telemetry from connected IoT devices.
Data consumers will typically fall into the category of data processing and storage applications such as Apache Hadoop, Apache Storm, S3, and ElasticSearch.
Purpose
Kinesis data streams is highly customizable and best suited for developers building custom applications or streaming data for specialized needs.
Kinesis Data Firehose handles loading data streams directly into AWS products for processing. Firehose also allows for streaming to S3, OpenSearch Service, or Redshift, where data can be copied for processing through additional services.
Provisioning & Scaling
Kinesis Data Streams needs you to configure shards, scale, and write your own custom applications for both producers and consumers. KDS requires manual scaling and provisioning.
Kinesis Data Firehose is fully managed and sends data to S3, Redshift, and OpenSearch. Scaling is handled automatically, up to gigabytes per second, and allows for batching, encrypting, and compressing.
Processing Delay
Kinesis Data Streams provides real-time processing with ~200 ms for shared throughput classic single consumer and ~70 ms for the enhanced fan-out consumer.
Kinesis Data Firehose provides near real-time processing with the lowest buffer time of 1 min.
Data Storage
Kinesis Data Streams provide data storage. Data typically is made available in a stream for 24 hours, but for an additional cost, users can gain data availability for up to 365 days.
Kinesis Data Firehose does not provide data storage.
Replay
Kinesis Data Streams supports replay capability
Kinesis Data Firehose does not support replay capability
Producers & Consumers
Kinesis Data Streams & Kinesis Data Firehose support multiple producer options including SDK, KPL, Kinesis Agent, IoT, etc.
Kinesis Data Streams support multiple consumers option including SDK, KCL, and Lambda, and can write data to multiple destinations. However, they have to be coded. Kinesis Data Firehose consumers are close-ended and support destinations like S3, Redshift, OpenSearch, and other third-party tools.
AWS Certification Exam Practice Questions
Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
Open to further feedback, discussion and correction.
Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
Open to further feedback, discussion and correction.
Your organization needs to ingest a big data stream into its data lake on Amazon S3. The data may stream in at a rate of hundreds of megabytes per second. What AWS service will accomplish the goal with the least amount of management?
Amazon Kinesis Firehose
Amazon Kinesis Streams
Amazon CloudFront
Amazon SQS
Your organization is looking for a solution that can help the business with streaming data several services will require access to read and process the same stream concurrently. What AWS service meets the business requirements?
Amazon Kinesis Firehose
Amazon Kinesis Streams
Amazon CloudFront
Amazon SQS
Your application generates a 1 KB JSON payload that needs to be queued and delivered to EC2 instances for applications. At the end of the day, the application needs to replay the data for the past 24 hours. In the near future, you also need the ability for other multiple EC2 applications to consume the same stream concurrently. What is the best solution for this?
11 thoughts on “AWS Kinesis Data Streams vs Kinesis Data Firehose”
A company operates an international business served from a single AWS region. The company wants to expand into a new country. The regulator for that country requires the Data Architect to maintain a log of financial transactions in the country within 24 hours of the product transaction. The production application is latency insensitive. The new country contains another AWS region.
What is the most cost-effective way to meet this requirement?
A. Use CloudFormation to replicate the production application to the new region.
B. Use Amazon CloudFront to serve application content locally in the country; Amazon CloudFront logs will satisfy the requirement.
C. Continue to serve customers from the existing region while using Amazon Kinesis to stream transaction data to the regulator.
D. Use Amazon S3 cross-region replication to copy and persist production transaction logs to a bucket in the new countrys region.
D as only the logs need to be maintained in the new country, S3 cross region replication can be used to copy the data to the AWS region within the new Country.
B. This seems to be for PCI compliance.
Good info. I wish I had read this post prior to taking the SAA-C02 exam the other day. I did pass, but some of the Kinesis related questions made me nervous. I did take the official course but the student guide did not have much info on the differences.
thanks, hope it would help in your next certifications.
This is helpful. I have one question, Can we connect kinesis firehose in one aws account with kinesis stream with another AWS account. is it possible?
A company captures clickstream data from multiple websites and analyzes it using batch processing. The data is loaded nightly into Amazon Redshift and is consumed by business analysts. The company wants to move towards near-real-time data processing for timely insights. The solution should process the streaming data with minimal effort and operational overhead.
Which combination of AWS services are MOST cost-effective for this solution? (Choose two.)
A. Amazon EC2
B. AWS Lambda
C. Amazon Kinesis Data Streams
D. Amazon Kinesis Data Firehose
E. Amazon Kinesis Data Analytics
As per latest update, Data streams retention period increased to 365 days.
Not sure whether the exam will have updated options.
Thanks Jayendra. That was really helpful. Though one of your example Q talks of Kinesis Sreams while the other talks of Kinesis Data Streams. I think they are same, aren’t they?
Yup they are the same. It was referred to Kinesis Streams before the arrival of other kinesis services.
A company operates an international business served from a single AWS region. The company wants to expand into a new country. The regulator for that country requires the Data Architect to maintain a log of financial transactions in the country within 24 hours of the product transaction. The production application is latency insensitive. The new country contains another AWS region.
What is the most cost-effective way to meet this requirement?
A. Use CloudFormation to replicate the production application to the new region.
B. Use Amazon CloudFront to serve application content locally in the country; Amazon CloudFront logs will satisfy the requirement.
C. Continue to serve customers from the existing region while using Amazon Kinesis to stream transaction data to the regulator.
D. Use Amazon S3 cross-region replication to copy and persist production transaction logs to a bucket in the new countrys region.
D as only the logs need to be maintained in the new country, S3 cross region replication can be used to copy the data to the AWS region within the new Country.
B. This seems to be for PCI compliance.
Good info. I wish I had read this post prior to taking the SAA-C02 exam the other day. I did pass, but some of the Kinesis related questions made me nervous. I did take the official course but the student guide did not have much info on the differences.
thanks, hope it would help in your next certifications.
This is helpful. I have one question, Can we connect kinesis firehose in one aws account with kinesis stream with another AWS account. is it possible?
A company captures clickstream data from multiple websites and analyzes it using batch processing. The data is loaded nightly into Amazon Redshift and is consumed by business analysts. The company wants to move towards near-real-time data processing for timely insights. The solution should process the streaming data with minimal effort and operational overhead.
Which combination of AWS services are MOST cost-effective for this solution? (Choose two.)
A. Amazon EC2
B. AWS Lambda
C. Amazon Kinesis Data Streams
D. Amazon Kinesis Data Firehose
E. Amazon Kinesis Data Analytics
As per latest update, Data streams retention period increased to 365 days.
Reference : https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html .
Not sure whether the exam will have updated options.
Thanks Jayendra. That was really helpful. Though one of your example Q talks of Kinesis Sreams while the other talks of Kinesis Data Streams. I think they are same, aren’t they?
Yup they are the same. It was referred to Kinesis Streams before the arrival of other kinesis services.