CloudWatch Logs Use cases

Amazon CloudWatch Logs – Collection & Insights

~ Last updated on : ~ jayendrapatil

Amazon CloudWatch Logs

  • CloudWatch Logs can be used to monitor, store, and access log files from EC2 instances, CloudTrail, Route 53, and other sources.
  • CloudWatch Logs uses the log data for monitoring with no code changes required.
  • CloudWatch Logs requires the unified CloudWatch agent to be installed on EC2 instances and on-premises servers. (Note: The older CloudWatch Logs agent is deprecated and no longer supported. Use the unified CloudWatch agent instead.)
  • The unified CloudWatch agent can collect both logs and metrics from EC2 instances and on-premises servers.
  • A VPC endpoint can be configured to keep traffic between VPC and CloudWatch Logs from leaving the Amazon network. It doesn’t require an IGW, NAT, VPN connection, or Direct Connect connection.
  • CloudWatch Logs allows exporting log data from log groups to an S3 bucket, which can then be used for custom processing and analysis, or to load onto other systems.
  • Log data is encrypted while in transit and while it is at rest.
  • Log data can be encrypted using AWS KMS customer managed keys (CMK).
  • CloudWatch Logs now supports log events up to 1 MB in size (increased from the previous 256 KB limit, as of April 2025).

Required Mainly for SysOps Associate & DevOps Professional Exam

CloudWatch Logs Concepts

Log Events

  • A log event is a record of some activity recorded by the application or resource being monitored.
  • Log event record contains two properties: the timestamp of when the event occurred, and the raw event message.

Log Streams

  • A log stream is a sequence of log events that share the same source for e.g. log events from an Apache access log on a specific host.

Log Groups

  • Log groups define groups of log streams that share the same retention, monitoring, and access control settings for e.g. Apache access logs from each host grouped through log streams into a single log group.
  • Each log stream has to belong to one log group.
  • There is no limit on the number of log streams that can belong to one log group.

Log Classes

  • CloudWatch Logs offers two log classes:
    • Standard – Full-featured log class with real-time monitoring, anomaly detection, Live Tail, Logs Insights, metric filters, subscription filters, and all CloudWatch Logs capabilities.
    • Infrequent Access (IA) – Lower-cost log class (50% lower per GB ingestion price) designed for logs that are queried occasionally, such as forensic investigations and compliance auditing. Supports Logs Insights queries, export to S3, encryption, and data protection.
  • Log class is set at log group creation time and cannot be changed.
  • Infrequent Access does not support metric filters, subscription filters, Live Tail, or anomaly detection.

Metric Filters

  • Metric filters can be used to extract metric observations from ingested events and transform them to data points in a CloudWatch metric.
  • Metric filters are assigned to log groups, and all of the filters assigned to a log group are applied to their log streams.
  • Metric filters are only available for log groups in the Standard log class.

Retention Settings

  • Retention settings can be used to specify how long log events are kept in CloudWatch Logs.
  • Expired log events get deleted automatically.
  • Retention settings are assigned to log groups, and the retention assigned to a log group is applied to their log streams.

CloudWatch Logs Features

CloudWatch Logs Insights

  • CloudWatch Logs Insights is an interactive, purpose-built query language for searching and analyzing log data.
  • Supports queries across multiple log groups and multiple AWS accounts.
  • Automatically discovers fields in JSON-format logs and generates visualizations.
  • Supports OpenSearch SQL and PPL (Piped Processing Language) directly within the Logs Insights console for SQL-based log analytics and JOIN operations across log groups (launched Dec 2024).
  • Field Indexes (Nov 2024) – Index critical log attributes (e.g., requestId, transactionId) to accelerate query performance and reduce scan volume. Up to 20 fields can be indexed per log group. Available at no additional cost.
  • AI-Powered Natural Language Query Generation (June 2024) – Generate Logs Insights queries using plain language, powered by generative AI.
  • Scheduled Queries (Nov 2025) – Automate log analysis by running Logs Insights queries on a recurring schedule, delivering results to S3 or EventBridge.
  • 23 New Query Commands and Functions (June 2026) – Includes hash functions (md5, sha256), string functions (strcontains with case-insensitive search, split), conditional logic (if statement), conversion functions (toNumber, toInt, toLong, toDouble), CSV/XML parsing, histogram, and multi-stage pipes for stats.
  • Facets (June 2026) – Explore log data visually without writing a query for interactive log exploration.
  • Lookup Tables (June 2026) – Enrich query results with external metadata.
  • Parameterized Queries (June 2026) – Save reusable query templates with fill-in variables.
  • JOIN and Sub-queries (June 2026) – Correlate data across log groups in a single query.

Live Tail

  • Live Tail (launched June 2023) provides real-time streaming of log events as they are ingested into log groups.
  • Helps quickly view, filter, and troubleshoot issues in incoming logs without writing queries.
  • Supports fine-grained controls to filter and highlight attributes of interest, and pause/replay logs.
  • A Live Tail session can last for up to 3 hours.
  • Available via console, CLI (streaming support added June 2024), and API.
  • Integrated natively into the AWS Lambda console for real-time function log visibility (Oct 2024).
  • Only available for Standard log class.

Anomaly Detection and Pattern Analytics

  • Anomaly Detection (launched Nov 2023) uses machine learning algorithms trained on decades of Amazon and AWS operational data to automatically identify anomalies in log data.
  • Anomaly detectors train on the past two weeks of log events (training takes up to 15 minutes) and then continuously analyze incoming logs.
  • Default quota: up to 500 anomaly detectors per account (increased from 10 in Oct 2024).
  • Pattern Analytics automatically recognizes and clusters patterns among log records, extracting noteworthy content and trends.
  • Pattern and diff commands use named tokens for easier-to-read results.
  • Only available for Standard log class.

Log Transformation and Enrichment

  • Log Transformation (launched Nov 2024) normalizes logs into a consistent format at the time of ingestion.
  • Supports pre-configured templates for common AWS services (WAF, VPC Flow Logs) and custom transformers (Grok patterns).
  • Each transformer can extract up to 200 fields from a log event.
  • Transformed logs can be leveraged for field indexes, discovered fields in Logs Insights, metric filters, and subscription filters.
  • Included with existing Standard log class ingestion price (no additional cost for transformation).
  • Supports OCSF (Open Cybersecurity Schema Framework) and OTel format conversion for security and compliance data.

Log Analytics (Unified Console)

  • Log Analytics (launched June 2026) is a unified console experience that brings together:
    • CloudWatch Logs Insights for querying and analyzing log data
    • Live Tail for real-time log streaming
    • Contributor Insights for identifying top contributors
  • Supports multiple queries in different tabs simultaneously.
  • Combines facets, saved queries with parameters, natural language query generation, and visualizations in a single interface.

CloudWatch Logs Use Cases

Monitor Logs from EC2 Instances in Real-time

  • Can help monitor applications and systems using log data.
  • Can help track number of errors for e.g. 404, 500, or even specific literal terms “NullReferenceException”, occurring in applications, which can then be matched to a threshold to send notification.

Monitor AWS CloudTrail Logged Events

  • Can be used to monitor particular API activity as captured by CloudTrail by creating alarms in CloudWatch and receiving notifications.

Archive Log Data

  • Can help store log data in highly durable storage, an alternative to S3.
  • Log retention setting can be modified, so that any log events older than this setting are automatically deleted.

Log Route 53 DNS Queries

  • Can help log information about the DNS queries that Route 53 receives.

Managed Syslog Ingestion

  • Managed Syslog Ingestion (launched June 2026) enables sending syslog messages from firewalls, routers, switches, and Linux servers directly into CloudWatch Logs.
  • Eliminates the need for custom syslog collection infrastructure.

Real-time Processing of Log Data with Subscriptions

  • Subscriptions provide access to a real-time feed of log events from CloudWatch Logs and deliver them to other services such as Kinesis Data Streams, Amazon Data Firehose, AWS Lambda, or Amazon OpenSearch Service for custom processing, analysis, or loading to other systems.
  • A subscription filter defines the filter pattern to use for filtering which log events get delivered, as well as the destination for matching log events.
  • CloudWatch Logs log group can be configured to stream data to an Amazon OpenSearch Service cluster in near real-time. (Note: Amazon Elasticsearch Service was renamed to Amazon OpenSearch Service in September 2021.)
  • Account-Level Subscription Filters (Jan 2024) – Deliver real-time log events from all log groups in an account to a Kinesis Data Stream, Amazon Data Firehose, or Lambda using a single account-level filter (one per account per Region).

Cross-Account and Cross-Region Log Centralization

  • Cross-Account Observability enables searching log groups across multiple accounts, running cross-account Logs Insights queries, and creating Contributor Insights rules.
  • Cross-Account and Cross-Region Log Centralization (Sept 2025) copies log data from multiple AWS accounts and regions into a single destination account for centralized management.

Searching and Filtering

  • CloudWatch Logs allows searching and filtering the log data by creating one or more metric filters.
  • Metric filters define the terms and patterns to look for in log data as it is sent to CloudWatch Logs.
  • CloudWatch Logs uses these metric filters to turn log data into numerical CloudWatch metrics that can be graphed or used to set alarms.

AWS Certification Exam Practice Questions

  • Questions are collected from Internet and the answers are marked as per my knowledge and understanding (which might differ with yours).
  • AWS services are updated everyday and both the answers and questions might be outdated soon, so research accordingly.
  • AWS exam questions are not updated to keep up the pace with AWS updates, so even if the underlying feature has changed the question might not be updated
  • Open to further feedback, discussion and correction.
  1. Once we have our logs in CloudWatch, we can do a number of things such as: Choose 3. Choose the 3 correct answers:[CDOP]
    1. Send the log data to AWS Lambda for custom processing or to load into other systems
    2. Stream the log data to Amazon Kinesis
    3. Stream the log data into Amazon OpenSearch Service in near real-time with CloudWatch Logs subscriptions.
    4. Record API calls for your AWS account and delivers log files containing API calls to your Amazon S3 bucket
  2. You have decided to set the threshold for errors on your application to a certain number and once that threshold is reached you need to alert the Senior DevOps engineer. What is the best way to do this? Choose 3. Choose the 3 correct answers: [CDOP]
    1. Set the threshold your application can tolerate in a CloudWatch Logs group and link a CloudWatch alarm on that threshold.
    2. Use the unified CloudWatch agent to send log data from the app to CloudWatch Logs from Amazon EC2 instances
    3. Pipe data from EC2 to the application logs using AWS Data Pipeline and CloudWatch
    4. Once a CloudWatch alarm is triggered, use SNS to notify the Senior DevOps Engineer.
  3. You are hired as the new head of operations for a SaaS company. Your CTO has asked you to make debugging any part of your entire operation simpler and as fast as possible. She complains that she has no idea what is going on in the complex, service-oriented architecture, because the developers just log to disk, and it’s very hard to find errors in logs on so many services. How can you best meet this requirement and satisfy your CTO? [CDOP]
    1. Copy all log files into AWS S3 using a cron job on each instance. Use an S3 Notification Configuration on the PutBucket event and publish events to AWS Lambda. Use the Lambda to analyze logs as soon as they come in and flag issues. (is not fast in search and introduces delay)
    2. Begin using CloudWatch Logs on every service. Stream all Log Groups into S3 objects. Use AWS EMR cluster jobs to perform adhoc MapReduce analysis and write new queries when needed. (is not fast in search and introduces delay)
    3. Copy all log files into AWS S3 using a cron job on each instance. Use an S3 Notification Configuration on the PutBucket event and publish events to AWS Kinesis. Use Apache Spark on AWS EMR to perform at-scale stream processing queries on the log chunks and flag issues. (is not fast in search and introduces delay)
    4. Begin using CloudWatch Logs on every service. Stream all Log Groups into an Amazon OpenSearch Service domain running OpenSearch Dashboards and perform log analysis on a search cluster. (OpenSearch with Dashboards (formerly ELK – Elasticsearch, Kibana stack) is designed specifically for real-time, ad-hoc log analysis and aggregation)
  4. You use Amazon CloudWatch as your primary monitoring system for your web application. After a recent software deployment, your users are getting Intermittent 500 Internal Server Errors when using the web application. You want to create a CloudWatch alarm, and notify an on-call engineer when these occur. How can you accomplish this using AWS services? (Choose three.) [CDOP]
    1. Deploy your web application as an AWS Elastic Beanstalk application. Use the default Elastic Beanstalk CloudWatch metrics to capture 500 Internal Server Errors. Set a CloudWatch alarm on that metric.
    2. Install the unified CloudWatch agent on your servers to stream web application logs to CloudWatch.
    3. Use Amazon Simple Email Service to notify an on-call engineer when a CloudWatch alarm is triggered.
    4. Create a CloudWatch Logs group and define metric filters that capture 500 Internal Server Errors. Set a CloudWatch alarm on that metric.
    5. Use Amazon Simple Notification Service to notify an on-call engineer when a CloudWatch alarm is triggered.
    6. Use AWS Data Pipeline to stream web application logs from your servers to CloudWatch.
  5. A company needs to consolidate logs from multiple AWS accounts for compliance auditing but wants to minimize costs since logs are only queried during investigations. Which CloudWatch Logs approach is most cost-effective? [SAA/SysOps]
    1. Use Standard log class with metric filters for all log groups
    2. Use Infrequent Access log class for the log groups and query with Logs Insights when needed
    3. Export all logs to S3 immediately and use Athena for queries
    4. Use Live Tail to monitor logs in real-time across accounts
  6. A DevOps team wants to quickly identify anomalous log patterns after a deployment without writing custom detection rules. Which CloudWatch Logs feature should they use? [CDOP]
    1. Metric filters with CloudWatch Alarms
    2. CloudWatch Logs Insights scheduled queries
    3. CloudWatch Logs Anomaly Detection with pattern analytics
    4. Subscription filters to Lambda for custom processing
  7. A team needs to stream log events from ALL log groups in their AWS account to a central Kinesis Data Stream without creating individual subscription filters for each log group. What should they use? [CDOP]
    1. Cross-account observability with a monitoring account
    2. CloudWatch Logs transformation with Kinesis destination
    3. Account-level subscription filter policy
    4. CloudWatch Logs Live Tail with CLI streaming
  8. An engineer wants to speed up CloudWatch Logs Insights queries that frequently filter by requestId across large log groups. Which feature should they configure? [CDOP/SysOps]
    1. Log transformation with Grok patterns
    2. Field indexes on the requestId field
    3. Infrequent Access log class for the log group
    4. Scheduled queries with S3 delivery

References