AWS Cost Efficiency
10
mins read
How to reduce AWS CloudWatch Cost?
Amazon CloudWatch enables publishing metrics through logs, ideal for complex application metric ingestion alongside other log data.
Particularly valuable in high-cardinality environments like AWS Lambda and container services such as ECS or EKS, CloudWatch offers a crucial feature: automatic extraction of custom metrics from logs. This capability allows for visualization, alarms, and analysis using Logs Insights, significantly enhancing monitoring efficiency.
Around 25,000 companies rely on Amazon CloudWatch for monitoring. This underscores CloudWatch's pivotal role in real-time metrics, logs, and events collection, vital for global cloud infrastructure management and optimization.
In this post, we will cover how CloudWatch costs are calculated and factors influencing them, why reducing CloudWatch costs is important for businesses using AWS services and share practical tips on how to achieve cost optimization effectively.
CloudWatch costs depend on factors like data usage, storage, and additional features. Charges include ingested data volume, storage, retrieval, monitoring frequency, custom metrics, alarms, dashboards, API requests, and more. Users can also benefit from the AWS Free Tier for limited usage at no cost.
For Detailed insights on paid pricing for CloudWatch visit AWS CloudWatch Pricing page.
Example: If you are monitoring VPCs that send 72TB of ingested VPC flow logs to CloudWatch logs per month and archiving the data for one month, your charges would be as follows:
CloudWatch can incur unnecessary charges across various services such as CloudWatch Custom Metrics, Metric API Calls, Alarms, Dashboards, Logs, Contributor Insights, Synthetics, Internet Monitor, Container Insights, Application Insights, RUM (Real User Monitoring), and Evidently.
Let's explore strategies to minimize these costs effectively.
Take the time to analyze your logging requirements and determine which information is critical for effective monitoring and diagnostics. Configure logging levels to capture only essential data while avoiding excessive verbosity, thereby reducing the volume of logs and minimizing cloudWatch costs.
Utilize CloudWatch Metric Math functionality to consolidate related metrics, which can help reduce the number of custom metrics generated and lower associated expenses. Additionally, implement selective metric filters to limit the ingestion of log data into CloudWatch, focusing only on the most relevant information and further reducing costs without sacrificing monitoring effectiveness.
Amazon CloudWatch logs incur charges from ingestion, archival storage, and analysis. To reduce charges, take the following actions:
To monitor the volume of logs that workloads generate, view this metric in a CloudWatch graph or dashboard.
aws cloudwatch get-metric-statistics --namespace AWS/Logs --metric-name IncomingLogEvents --statistics Sum --start-time <start-time> --end-time <end-time> --period <period>
For high ingestion charges, identify the log group causing a sudden increase in your CloudWatch Logs bill.
aws cloudwatch describe-alarms --query 'MetricAlarms[?MetricName==`IncomingLogEvents`].{AlarmName:AlarmName}'
To reduce ingestion charges, check logging levels and remove unnecessary log ingestion. Consider using infrequent access logs to cost-effectively store your logs.
The retention policy determines how long CloudWatch Logs maintains data. To reduce archival charges, modify the retention policy to a shorter retention period to limit stored data over time.
aws logs put-retention-policy --log-group-name <log-group-name> --retention-in-days <retention-days>
Analysis charges occur when you use Log Insights to query logs or Live Tail to analyze logs in real time.To reduce Log Insights charges, set queries to run with shorter timeframes to scan less data.To reduce Live Tail charges, minimize session usage time.
Setting log retention policies involves defining rules for how long log data should be retained in CloudWatch before it's automatically deleted. By removing old and unnecessary log data, organizations can reduce storage costs and improve the efficiency of log management.
Example: Configure a log retention policy to automatically delete log data older than 30 days using AWS CLI.
aws logs put-retention-policy --log-group-name my-log-group --retention-in-days 30
This command sets a retention period of 30 days for the log group named my-log-group, ensuring that log data older than 30 days is automatically deleted, thus reducing CloudWatch costs.
CloudWatch alarms incur costs based on the number of metrics associated with each alarm. For instance, if you have one alarm with multiple metrics, you'll be charged for each metric.
To reduce CloudWatch costs by reviewing and removing unused alarms, start by assessing the relevance of each alarm and identifying any that are no longer needed. Review the settings of each alarm to ensure they align with current monitoring requirements. Remove or disable alarms that are deemed unnecessary or redundant, ensuring that only essential ones remain active. Regularly monitor alarm usage and adjust configurations as needed to optimize cost-effectiveness while maintaining necessary monitoring functionality.
Here's an example of how you might implement this:
#!/bin/bash
# Get a list of all alarms
aws cloudwatch describe-alarms --query 'MetricAlarms[*].[AlarmName]' --output text > all_alarms.txt
# Loop through each alarm and check if it's necessary
while IFS= read -r alarm_name; do
# Check if the alarm is associated with important metrics
# (replace this condition with your specific criteria)
if [[ "$alarm_name" != *"important_metric"* ]]; then
# Delete the alarm
aws cloudwatch delete-alarms --alarm-names "$alarm_name"
echo "Deleted alarm: $alarm_name"
else
echo "Skipping alarm: $alarm_name (associated with important metrics)"
fi
done < all_alarms.txt
# Clean up: remove temporary file
rm all_alarms.txt
This script lists all alarms, checks if each alarm is associated with important metrics, and deletes those that are not. You should customize the condition inside the loop according to your specific criteria for identifying unnecessary alarms.
To reduce CloudWatch costs through reserved capacity, consider pre-paying for dashboard space or alarms at discounted rates. Reserved Dashboards allow you to secure dashboard capacity for a specific period, while Reserved Alarms enable pre-payment for a set number of alarms over a defined time. These options are advantageous for predictable workloads, providing cost savings over time.
To reduce charges associated with CloudWatch dashboards, consider the following actions:
Regularly review and remove any dashboards that are no longer needed.
aws cloudwatch delete-dashboards --dashboard-names <dashboard-name1> <dashboard-name2> ...
If you're utilizing the AWS Free Tier, ensure that your total number of dashboards does not exceed three.
Utilize dashboard API calls, such as GetDashboard, ListDashboards, PutDashboard, and DeleteDashboards, through the AWS Management Console to avoid exceeding the AWS Free Tier limit.Implementing these actions will help you manage and reduce charges associated with CloudWatch dashboards effectively.
# GetDashboard
aws cloudwatch get-dashboard --dashboard-name <dashboard-name>
# ListDashboards
aws cloudwatch list-dashboards
# PutDashboard
aws cloudwatch put-dashboard --dashboard-name <dashboard-name> --dashboard-body <dashboard-body-json>
# DeleteDashboards
aws cloudwatch delete-dashboards --dashboard-names <dashboard-name1> <dashboard-name2> ...
To reduce CloudWatch costs, implement cost allocation tags by developing a tagging strategy, applying tags to resources, analyzing spending data, optimizing resource usage, and continuously updating tagging strategies.
aws cloudwatch tag-resource --resource-arn arn:aws:cloudwatch:us-west-2:123456789012:dashboard/MyDashboard --tags "Key=Department,Value=Finance" "Key=Project,Value=ABC"
When you exceed one Contributor Insights rule each month or more than one million log events match the rule each month, you incur charges. To reduce charges, follow these steps:
Navigate to the CloudWatch console.Select "Contributor Insights" from the menu.Review your Contributor Insights reports to understand which rules are being triggered frequently.
Identify rules that are not providing significant value or are triggering frequently without actionable insights. Remove unnecessary rules by navigating to the Contributor Insights settings and deleting them.By regularly reviewing and removing unnecessary rules, you can prevent excessive charges associated with Contributor Insights.
Let's consider an example scenario:
An organization uses CloudWatch Contributor Insights to monitor its application logs. They have several rules configured to detect anomalies and issues in their logs. However, they've noticed a significant increase in charges due to exceeding the limits set for Contributor Insights rules and log events matching those rules.
CloudWatch Internet Monitor incurs charges for each monitored resource and all city networks. You might also incur additional charges for diagnostic logs that you publish to CloudWatch Logs. To reduce charges, set a limit for the maximum number of city networks that you monitor.
This action typically requires modifying your monitoring configuration to restrict the number of city networks being monitored. This can be done via custom scripts or through the AWS Management Console.
Example using AWS SDK for Python (Boto3):
import boto3
client = boto3.client('internetmonitor')
response = client.update_monitor(
MonitorName='example-monitor',
Resources=['resource-1', 'resource-2'], # Specify resources
MaxCityNetworks=10 # Set a limit for city networks
)
print(response)
CloudWatch Synthetics incurs charges when you exceed 100 canary runs each month. For each canary run, you might incur additional charges for other AWS services that you use, such as AWS Lambda, Amazon S3, CloudWatch Logs, and CloudWatch metrics. To reduce charges, remove unnecessary canaries.
aws synthetics delete-canary --name <canary-name>
Container Insights incurs charges based on the metrics that are collected and the log data that's ingested. If you use enhanced observability for Amazon Elastic Kubernetes Service (Amazon EKS), then Container Insights metrics and logs are charged for each observation. To reduce charges, turn off enhanced observability on your Amazon Elastic Container Service (Amazon ECS) or Amazon EKS clusters.
Turn Off Enhanced Observability for Amazon ECS:
aws ecs update-cluster-settings --cluster <cluster-name> --settings name=containerInsights,value=disabled
Turn Off Enhanced Observability for Amazon EKS:
aws eks update-cluster-config --name <cluster-name> --resources-vpc-config endpointPublicAccess=true
In conclusion, for businesses utilizing AWS CloudWatch, effective cost optimization involves optimizing logging levels, filtering metrics, and refining log ingestion. Additionally, setting retention policies, removing unused alarms, and considering reserved capacity are key strategies. By also implementing cost allocation tags and leveraging CloudWatch Synthetics, businesses can maintain efficient monitoring within budgetary constraints. Stay proactive, monitor regularly, and adapt your approach to ensure CloudWatch usage remains effective and budget-friendly for your specific needs.
.png)
.png)
