A brief introduction to CloudWatch

Amazon CloudWatch monitors the performance and health of our resources and applications in AWS. As a result it lets us:

  • Track resource and application performance
  • Collect and monitor log files
  • Get notified when an alarm goes off
  • CloudWatch consists of three primary components: metrics, alarms, and events.

When running applications on Amazon EC2 instances, monitoring workload performance is crucial. This involves addressing two key questions: ensuring sufficient EC2 resources for fluctuating performance requirements and automating resource provisioning on demand. While Amazon CloudWatch facilitates performance monitoring and log file collection, it doesn't directly manage EC2 instances. Amazon EC2 Auto Scaling is our solution, as it enables dynamic scaling to maintain fleet health and availability during demand fluctuations. Amazon CloudWatch serves as a distributed statistics-gathering system, collecting and tracking metrics, including custom ones, and triggering notifications for alarms.

CloudWatch has two different monitoring options:

  • Basic Monitoring for Amazon EC2 instances: Seven pre-selected metrics at a 5-minute frequency and three status check metrics at a 1-minute frequency, for no additional charge.

  • Detailed Monitoring for Amazon EC2 instances: All metrics that are available to Basic Monitoring at a 1-minute frequency, for an additional charge. Instances with detailed monitoring enabled provide data aggregation by Amazon EC2, Amazon Machine Image (AMI) ID, and instance type.



The diagram an example of CloudWatch monitoring:

  • EC2 Instance Monitoring:
    • An EC2 instance on the left has the CloudWatch agent installed, with detailed monitoring enabled.
    • Two metrics are highlighted:
      • CPU Utilization: A standard CloudWatch metric collected easily.
      • Memory Utilization: Monitored for the httpd service using a custom-defined metric since it's not visible at the hypervisor layer.
  • CloudWatch Alarm Configuration:
    • A CloudWatch alarm is set up to trigger when CPU utilization exceeds a defined threshold.
    • Upon triggering, an alert is sent via Amazon SNS (Simple Notification Service), generating an email notification.
    • The alert is also sent to an Amazon SQS (Simple Queue Service) topic, creating a work item.

In the diagram above, we can see the following CloudWatch alarm behavior taking place:

  • Alarm Configuration:
    • Threshold is set to 3.
    • Minimum breach condition is set to 3 consecutive periods.

  • Alarm State Changes:
    • Time Periods 1-2: Threshold not breached, alarm state is OK.
    • Time Periods 3-5: Threshold breached consecutively for three periods, alarm state changes to ALARM.
    • Time Period 6: Value dips below the threshold, alarm state reverts to OK.
    • Time Periods 7-8: Threshold not breached, alarm state is OK.
    • Time Period 9: Threshold breached but not for three consecutive periods, alarm state remains OK.


In Amazon CloudWatch, metrics are uniquely defined by a name, a namespace, and zero or more dimensions. Each data point includes a timestamp and optionally a unit of measure. Metrics exist only in the Region where they are created.

  • Namespace: A container for CloudWatch metrics, with metrics from different namespaces isolated to prevent aggregation errors. AWS namespaces follow the AWS/<service> naming convention (e.g., AWS/EC2).

  • Dimension: A name-value pair uniquely identifying a metric, with up to 10 dimensions allowed per metric. Dimensions categorize and describe specific metric characteristics, aiding in structuring statistics plans and filtering result sets.

  • Period: The time length associated with a CloudWatch statistic, defined in seconds. Data aggregation can be adjusted by varying the period, which can range from 1 second to 1 day (86,400 seconds).

  • Common CloudWatch use cases include monitoring account resources for suspicious activity, such as billing alerts to detect potential security violations. Alerts can be set based on estimated billing charges exceeding a specified threshold.

Additionally:

  • CloudWatch Events: Trigger services like AWS Lambda based on near-real-time events, enhancing automation capabilities.
  • CloudWatch Logs: Collect application logs by filtering metric data points for events, providing a comprehensive logging solution.

Amazon CloudWatch Events provides a real-time stream of AWS resource changes. Using configured rules, it can match and route events to various targets for actions. Events can represent changes in AWS resources or be custom application-level events, including scheduled events. Targets include EC2 instances, Lambda functions, SNS topics, and SQS queues. Rules match incoming events, directing them to one or more targets.

In an example, a CloudWatch Events rule triggers an AWS Systems Manager Run Command script each time a new EC2 instance is created.

Amazon CloudWatch Logs allows monitoring, storing, and accessing log files from diverse sources like EC2 instances, CloudTrail, and Route 53. It enables near-real-time log analysis for specific patterns. Users can set alarms based on log data, visualize metrics, and store logs indefinitely externally to EC2 instances, eliminating concerns about storage limitations.


You can think of the process of log analysis as having three distinct phases:

  • Configure – Decide what information you need to capture in your logs, and where and how it will be stored.
  • Collect – Instances are provisioned and removed in a cloud environment. You need a strategy for periodically uploading a server’s log files so that this valuable information is not lost when an instance is eventually terminated.
  • Analyze – After all the data is collected, it is time to analyze it. Using log data gives you greater visibility into the daily health of your systems. It can also provide information on upcoming trends in customer behavior, and insight into how customers currently use your system.

AWS CloudTrail facilitates continuous logging of AWS account activity, capturing information that can be directed to storage services like Amazon S3 or business reporting tools. CloudTrail monitors AWS API calls made through the AWS CLI and Management Console. Despite being comprehensive, CloudTrail isn't able to track events within Amazon EC2 instances, such as shutdowns initiated through SSH sessions. However with configuration, CloudTrail can send auditing logs to Amazon S3, enabling governance, compliance, operational auditing, and risk auditing for AWS accounts.

CloudTrail allows storing API usage logs in an S3 bucket for analysis. Whilst the default CloudTrail event history shows results from the last 90 days, limited to management events and account activity we can configure CloudWatch to capture a complete record of events. We're also able to enhance our analysis by using Athena with CloudTrail logs to query AWS service and Application Load Balancer activity, as well as Amazon Virtual Private Cloud Flow Logs for investigating network traffic patterns with the purpose of identifying threats in your VPC network.

Popular posts from this blog

Network Fundamentals for the Cloud

Image
Computer networking is all around us. From the days of dial up internet, internet connected devices have grown rapidly on the global scale. Despite this, across the globe there are broadly three types of computer network  - LAN/MAN and WAN.  The above diagram does a good example of representing the geographical elements of these networks. But it is worth elaborating bit further. LAN networks are usually used in office locations and connected by physical cables and local wifi. MAN networks are typically interconnected LAN networks with a private network link and a connection to the internet. MAN use cases are mostly reserved for governmental or state institutions, along with huge corporates - I'd imagine Alphabet Inc in Silicon Valley might use this for example. Whilst WAN connects devices in a large geographic area, across multiple cities or countries. WAN's are used to connect LANs. You could argue that the biggest WAN in the world is the internet itself. Some of the fundamen...
Read more

Familiarizing with the Command Line Interface

Image
I'm beyond lucky to have been a successful applicant onto the Amazon re/Start programme. This week on our intensive bootcamp we've been focusing on manoeuvring around the command line interface on Linux. Due to being computer active before YouTube (the delightful days of dial-up internet!) this wasn't all too unfamiliar to me for three specific reasons; Internet Relay Chat , or as the acronym goes - IRC. Before the endless content on YouTube, if you wanted to have a copy of a music video you either had to rely on the artist putting a movie file on a CD single or  you relied on a fan club setting a server/channel up on IRC. To get a music video on IRC, the wizkids of the era set-up a system identical to a Linux CLI. So there I was at 13, deploying my dir and get commands to get my own personal slice of MTV heaven! Raspberry Pi Inspired by the super cool RetroFlag SNES case for the RaspberryPi and all amazing bespoke retro arcades I saw people posting online, I went ahead ...
Read more

Security Fundamentals for the Cloud

Image
Security is the practice of protecting valuable assets. When we talk about information security, cyber security is actually just a subset of this discipline. Cybersecurity is focused on protecting digital related devices – like networks, systems and digital information. Focus on stopping unauthorised access, malicious actions like theft, destruction or alteration or simply disrupting the service. One of the most popular models for thinking about information security is the CIA triad. Confidentiality, Integrity and Availability.  Threats can come in many different forms. The tables below provide some a variety and most interestingly also explore what part of the Triad is impacted.    So, working with the security lifecycle model we can determine what actions we can take at each stage to deliver a robust security infrastructure within our organisation.  Prevention Identify assets – what devices are you using? Are they contemporary? Can you flash newer firmware on them,...
Read more

CLI Fundamentals for the Cloud

Image
Operating a computer with BASH is the reality the end-user faced prior to Window's graphical user interface. In the end however, what we're actually doing is rather much the same. By deploying actions via a CLI, we're empowered in our agency in what we can schedule and automate in the environment.  Ensuring we're starting off at the right directory is key, for that we use; pwd This outputs are current directory which informs us were we're currently sitting at in the system. A basic rule of command syntax is that is delivered in below format; command -> option - > argument eg. sudo usermod -a -G Sales arosalez sudo usermod -> -a -G  -> Sales arosalez The most basic commands revolve around providing quick user or system data. I won't elaborate on the most of these as they are either ubiquitous or pretty self evident. ls, cd, whoami, id, hostname, uptime, date cal, clear, echo, history, touch, cat Don't forget to use the tab autocomplete function...
Read more

DataDog, a Cloud Analytics & Monitoring application

Image
Due to having an industry connection, I had the opportunity to do something really fun this weekend. The application running on AWS is a CRUD API - however, Datadog, a Cloud Analytics & Monitoring agent has been installed to provide feedback on what's actually happening. I found this so beyond awesome, the scope of this product is really impressive.  On the left side you can see a command line running on my Linux VM. On the right you can see a live monitoring of the AWS machine, tracking the inputs received on the API application.  As I send commands into the CRUDE API application, DataDog continues monitoring and updating our dashboard. If you can imagine for the deployment of a much busier service, the possibilities of this kind of real-time monitoring isn't to be underestimated.   Above you can see the CPU usage graphs, but unlike the ones you find on the AWS console, you can see that DataDog manages to also show us the split of the CPU usage.  Even in th...
Read more

A brief introduction to Databases and MySQL

Image
Thanks to completing an IT GNVQ during my secondary school days, databases aren’t all that new to me. I had experience querying a relational database in Microsoft Access already. But for the purpose of this post, I’ll revisit some core principles. The first to example is the types of database, which I’ll use an infographic to explain; Now what we’ll be looking this post is the relational model. This works by having a series of tables linked by public and foreign keys. Each of these keys has to be completely unique. When we update our database, we term this as a transaction. This means one or more changes are performed to a database. To commit a change we need to ensure transactions follow the four standard principles of atomicity, consistency, isolation, and durability. Before we go into talking about querying our relational database, let’s get into the NULL value. When searching we can use IS NULL and IS NOT NULL, but beyond this, there are several useful things to know about this v...
Read more

AWS CodeCommit + Creating a CI/CD pipeline

Image
  Having signed up to the AWS Skill Builder I thought it was something I'd be looking at post-programme. However the above builder labs email landed, I thought, what better way to spend my Sunday afternoon? So the fundamentals lab was working with the AWS service CodeCommit. Having gone through most of gitimmersion  this lab felt very straight forward. It involved creating a local and "remote" (ie. CloudCommit) git repository and synchronising them together. The intermediate lab now proved a little more tricky. First of all was to comprehend that we were going to build a CI/CD pipeline - ie. manifesting the principles of DevOps. Below I've posted a full scale CI/CD pipeline to demonstrate the complexity. However, in this lab we're only using CodeCommit, CodeBuild, CodeDeploy, AWS ECR and ECS. This lab involved the most complex technology stack I've come across thus far, so really appreciated doing something a little more challenging. However, it did involve so...
Read more

Future Orientation: Tips from a AWS re/Start Graduate

Image
  Today we had a recent AWS re/Start Graduate join us for a short presentation and Q&A about their experience both about the programme and their post-programme period before starting a role as a Cloud Engineer.  It was great to hear from someone who less then 12 months ago had also undergone the same exact programme we're a third of the way through now. We got an insight into their day to day (lots of meetings!), how long it took until they were allowed to pick up tickets independently (two months) and how much did their employer support their development (lots!). By chance, they were scheduled to complete the solutions AWS solutions architect certification tomorrow! I found it incredibly motivating and centring to hear their story. I made sure to ask a question myself, which revolved around finding potential roles and then knowing the essential skills that we might need to enhance our employability. The advice provided was; Make sure to continue studying Networking. This ...
Read more

A brief introduction to AWS Cloud Adoption Framework (CAF) and Well-Architected Framework (WAF)

Image
To aid in moving more businesses into the cloud AWS have developed the Cloud Adoption Framework (CAF). There are also competitors’ models, but in the broadest sense they tend to follow the same principles and categories. I’ll start with showing broad categories, then the Azure model and the AWS model.  So now we've familiarised ourselves with the broad categories, we can see the two major cloud providers high-level frameworks presented; Now the 2nd infographic of the AWS CAF begins to show more detail, exposing us to the genuine complexity behind the considerations required during the CAF process. If a business is consequently adopting a cloud, well, it also makes sense to have a structure to think about how the future architecture will be designed and if it matches cloud best practice, in other words AWS's Well-Architected Framework (WAF). Thankfully like many of my posts, there are infographics that excellently and succinctly summaries these ideas for us; It is very likely th...
Read more

Building a VPC in AWS

Image
  For our very last lab about the network, we were given the following customer environment to build. By virtue of the fact the AWS management console GUI is constantly updated it meant that our task instructions were wholly out of date. However, this actually proved to only make it a far more involved and fun process - necessitating lots of troubleshooting when things had not spun up as expected. I felt like I had learnt a lot more this way and for the benefit of my future self, I thought I would make some step by step notes (not necessarily in reference to the diagram above) to serve as prompts for any future cloud VPC's I'll be spinning up. Our challenge was to build the VPC as the diagram above.  Create an elastic IP. We will associate this later with our NAT gateway. Launch your VPC with a private class IP range without forgetting to specific how many availability zones (AZ) you want. Create and label your subnets, again specifying what AZ, as none are public at the ...
Read more