Managing your AWS Resource Consumption

AWS Organizations is a centralized account management service that allows you to consolidate multiple AWS accounts into a structured organization, offering consolidated billing and enhanced account management for improved budget, security, and compliance management. The organizational structure involves a hierarchy of organizational units (OUs) within a root, resembling an upside-down tree. Policies attached to nodes in this hierarchy cascade down to affect all branches and leaves, ensuring consistent controls. The service enables centrally managed access policies, controlled access to AWS services, and automated AWS account creation. However, it doesn't replace AWS Identity and Access Management (IAM) policies, which are applied to individual IAM users, groups, and roles within an account. In contrast, AWS Organizations uses Service Control Policies (SCPs) to regulate access to AWS services for entire accounts or groups of accounts within an OU, affecting all users, groups, and roles, including the AWS account root user.

AWS Organizations is accessible through different interfaces. The AWS Management Console provides a browser-based option and the AWS CLI allows users to perform tasks quickly and conveniently through command-based interactions. Additionally, AWS Software Development Kits (SDKs) offer libraries and sample code for various programming languages and platforms, streamlining tasks such as request signing, error management, and automatic retries. For programmatic access, the HTTPS query API allows users to send requests directly to AWS Organizations.


A tag is a descriptive label that is attached to an AWS resource.

  • Use an IAM policy to mandate the use of specific tags on a resource, and AWS Config to periodically verify that all resources are tagged.

  • A good tagging strategy defines tags along dimensions that enable resource access control, cost-tracking, automation, and organization.

AWS billing and cost management services and tools can help you access, understand, allocate, control, and optimize your AWS costs and usage.

These tools and services include:

  • AWS Bills
  • AWS Cost Explorer
  • AWS Budgets
  • AWS Cost and Usage Reports
  • AWS Trusted Advisor
  • Automated scripts can be used to reduce waste and costs. A serverless model is an efficient way to implement a schedule-based script.

As cloud computing offers the advantage of paying only for necessary resources it makes sense to use strategies for cost optimization include scripting or templating for easy startup and shutdown of environments, turning off unused resources during specific periods, and activating disaster recovery environments only when needed. 


The AWS billing dashboard provides a real-time view of month-to-date AWS expenditure, helping us identify major cost-driving services and trends. Graphs like Spend Summary and Month-to-Date Spend by Service offer insights into spending patterns. AWS Cost Explorer allows detailed analysis, offering default reports like Cost and Usage and Reserved Instance reports. AWS Budgets lets users set custom budgets, providing alerts for exceeding budgeted amounts. AWS Cost and Usage Reports consolidate comprehensive information, customizable by product, usage type, and operation. Amazon CloudWatch monitors estimated AWS charges, generating alerts for surpassing specified thresholds. Billing metric data is stored in the US East Region, triggering alerts only when actual billing surpasses set thresholds, and notifications are sent via email through an SNS topic.

Cost management tools can aid in devising an effective cost-reduction strategy for AWS services, incorporating key principles such as setting custom budgets and embracing the "pay only for what you need, when you need it" concept. Strategies include right-sizing instances by considering T2 or T3 instances for burstable workloads and utilizing Reserved Instances for groups of long-running instances. Spot Instances are recommended for cost-effective batch processing, and AWS Lambda is highlighted for serverless computing with pay-as-you-go pricing. Leveraging managed services like Amazon S3 and Amazon RDS reduces operational burdens and offers cost efficiency. AWS Trusted Advisor provides specific advice on cost reduction. Additionally, eliminating waste is crucial, achieved through monitoring with Amazon CloudWatch metrics and alarms to identify and shut down idle instances. AWS Cost Explorer helps pinpoint costly projects for prioritized examination and cost-reduction opportunities.

AWS Trusted Advisor is an online resource to help you reduce cost, increase performance, and improve security by optimizing your AWS environment. AWS Trusted Advisor analyzes your AWS environment and provides recommendations for best practices in five categories:

• Cost optimization – Advice about how you can save money by eliminating unused and idle resources, or making commitments to reserved capacity.
• Performance – Advice about how to improve the performance of your services by checking your service limits, ensuring that you use provisioned throughput, and monitoring for overutilized instances.
• Security – Advice about how to improve the security of your applications by closing gaps, enabling various AWS security features, and examining your permissions.
• Fault tolerance – Advice about how to increase the availability and redundancy of your AWS applications by using automatic scaling, health checks, Multi-AZ deployment, and backup capabilities.
• Service limits – Advice about the services whose usage exceeds 80 percent of their service limit.

AWS offers various methods for configuring and managing infrastructure on the AWS Cloud, emphasizing proactive planning over reactive approaches. Manual configuration of distributed systems is prone to errors and time-consuming, necessitating automation for consistency. Configuration management, achieved through tools like AWS CloudFormation and AWS OpsWorks, brings benefits such as quick and consistent resource launches. This approach is crucial for deploying configuration changes with minimal server downtime. AWS CloudWatch complements these services by providing monitoring capabilities that trigger actions in response to resource changes. Configuration management is essential for efficient and error-free infrastructure management on AWS.

Many AWS customers use configuration tools to launch new EC2 instances and other resources in an automated, repeatable manner. The flexibility of the configuration management tools enables the implementation of a complete configuration management strategy.

The diagram shows us that configuration software deployed to initialize a set of Amazon EC2 instances as web servers. The process begins with the configuration of a standalone Amazon EC2 instance, designated as the configuration server. This instance hosts templates that outline applications, files, and configurations needed to launch AWS resources. For instance, a web server template specifies the automated launch of an EC2 instance and details tasks like HTTP installation, configuration of the web server's httpd.conf file, and installation of required programming language environments such as PHP or Ruby. The configuration server serves as a centralized hub for managing and automating the setup of multiple EC2 instances with consistent configurations.

In the process of configuring Amazon EC2 instances using a configuration server, the administrator establishes a directory and file structure on the EC2 instance. The configuration server hosts various recipes for configuring different server types, such as MySQL, NAT, Windows IIS, Maven repository, etc. The administrator creates configuration templates, publishing them to the configuration server and checking them into a source control system for version management and change tracking. Configuration may involve custom scripting, AWS CloudFormation, AWS OpsWorks, or similar tools to describe and create AWS Cloud deployments. These deployments encompass resources like EC2 instances, virtual private cloud (VPC) configurations, databases, and other necessary AWS components. This approach ensures standardized configurations and efficient management of diverse AWS Cloud deployments.

Available features and resources include user data for script execution on instance launch, Amazon Machine Images (AMIs) for pre-deployed installations, and configuration and deployment frameworks like Chef, Puppet, Ansible, AWS OpsWorks, and AWS CloudFormation. Configuration servers simplify administrative tasks, such as provisioning and de-provisioning user access to EC2 instances. Idempotent configuration, provided by tools like Chef, Puppet, or Ansible, ensures that repeated commands do not alter the system or environment. Organizations use a combination of these technologies based on factors like production workflow and organizational standards. A recommended approach involves creating custom AMIs for foundational components, using AWS CloudFormation or OpsWorks for rapidly changing components, and maintaining infrastructure as code through version control systems like Git or AWS CodeCommit. Iterative development allows continuous improvement of the configuration management solution, offering benefits like validation, automation, documentation, cost reduction, and enhanced security.


To meet the organization's requirement of having a standardized base configuration on all EC2 instances in the AWS Cloud, the following steps are suggested:

  1. Launch Standard EC2 Instance: Start by launching an EC2 instance from a standard Amazon Machine Image (AMI).

  2. Preconfigure Software: Preconfigure all required software on the EC2 instance, including utilities, in-house tools, and enterprise-scale software for activities like monitoring and intrusion detection.

  3. Create Custom AMI: Generate a custom Amazon Machine Image (AMI) from the configured EC2 instance. This custom AMI represents the base configuration with the pre-installed software.

  4. Enforce Policy for New Instances: To ensure compliance, implement processes that regularly scan running EC2 instances in the AWS account. Terminate any instances that do not use the newly created custom AMI. This enforcement guarantees that all new instances adhere to the organization's policy of launching from the standardized base AMI, maintaining consistency across the AWS environment.

This approach streamlines the deployment of EC2 instances with the required software, facilitating a uniform and controlled environment for the organization's AWS resources.

There several ways to configure and manage your Amazon Machine Images (AMIs) - configuring instances at boot time, and utilizing Infrastructure as Code (IaC) with AWS CloudFormation, AWS Systems Manager, and AWS OpsWorks.

Custom AMIs:

  • Organizations can develop custom AMIs to ensure EC2 instances have a predefined set of software pre-installed.
  • Steps include launching an EC2 instance from a standard AMI, preconfiguring required software, and creating a custom AMI for consistent deployments.
  • Trade-offs involve considerations of build times, boot times, and shelf life, with full AMIs having faster boot times but longer build times.

Configuring Instances at Boot Time:

  • Configuring instances at boot time involves using user data options to run scripts during EC2 instance launch.
  • This approach is often combined with custom AMIs, allowing for a hybrid approach that balances simplicity and flexibility.
  • Factors influencing the choice include build times, boot times, and shelf life, with trade-offs between fully pre-configured AMIs, partially configured AMIs, and OS-only AMIs.

Infrastructure as Code (IaC):

  • IaC is essential for consistent and reliable creation of infrastructure resources.
  • AWS CloudFormation is a service that facilitates IaC by enabling the creation of infrastructure using code.
  • Use cases for IaC include creating replica environments, temporary environments for demonstrations, and lowering costs in development environments.
  • Ongoing configuration management needs are addressed with tools like AWS Systems Manager and AWS OpsWorks, providing insights, automation, and efficiency in managing resources in the cloud.
  • IaC helps ensure consistency and reliability in managing infrastructure throughout its lifecycle.

In summary, organizations can choose between custom AMIs, configuring instances at boot time, or implementing IaC depending on their specific needs, with the goal of achieving consistent, repeatable, and efficient management of AWS resources.

JSON

JSON, or JavaScript Object Notation, is a syntax used for storing and transporting data. It is a text-based, human-readable format that facilitates the exchange of complex data between computer systems. JSON documents consist of key-value pairs and arrays, where keys are unique identifiers and values represent the associated data or pointers to its location. JSON supports structured data, making it suitable for various use cases, especially in the context of APIs and modern RESTful services.

Advantages of JSON include its lightweight nature, with minimal syntax rules, making it easy to read for humans and parse for machines. It excels in data exchange scenarios. However, JSON has some drawbacks, such as verbosity, potentially leading to long files with repetitive braces. Additionally, it lacks built-in support for storing multimedia formats like images.

AWS services, including AWS CloudFormation, leverage JSON as a data format for creating and managing related AWS resources through templates. Understanding JSON is crucial for working effectively with AWS CloudFormation templates, which are formatted text files in JSON or YAML.

YAML

YAML's advantages include native support for embedded comments, enhanced readability, and ease of debugging compared to JSON. YAML uses white-space indentation for denoting and creating data structures, interpreting tabs as white space. Comments can be added using a number sign (#), and lists are created by starting lines with hyphens or by enclosing multiple items in brackets ([]).

While YAML has distinct advantages, JSON remains widely used and has its strengths, such as being more common in computer systems and easier to generate and parse. AWS CloudFormation supports templates written in both JSON and YAML, providing flexibility in infrastructure deployment.

In YAML syntax, key-value pairs are defined using a colon, lists are created with hyphens or brackets, and associative arrays use a colon followed by a space. Strings in YAML are typically not enclosed in quotation marks, although they may be used to escape characters if needed. AWS CloudFormation facilitates the creation and provisioning of AWS infrastructure deployments in a repeatable and predictable manner.

  • Two major terms for AWS CloudFormation are templates and stacks.
  • If an error is encountered when you launch an AWS CloudFormation template, all resources are rolled back by default.
  • Parameters enable you to input custom values to your template each time you create or update a stack.

AWS CloudFormation allows you to create and manage AWS infrastructure deployments consistently and repeatedly. This includes services like Amazon EC2, Amazon EBS, Amazon SNS, Elastic Load Balancing, and Auto Scaling. Utilizing a template file, you can create, update, and delete a set of resources as a unified unit called a stack.

While AWS CloudFormation provides resource management, users might independently modify resources outside of CloudFormation using the underlying service. This could be intentional or accidental changes, potentially complicating stack updates or deletions. To address this, drift detection in CloudFormation helps identify configuration changes made outside of CloudFormation management. Once detected, corrective actions can be taken, such as updating the drifted resources directly to synchronize them with their definitions in the stack template. Resolving drift ensures configuration consistency and enhances the success of stack operations.

In AWS CloudFormation, templates define the AWS resources to be provisioned, while stacks are collections of AWS resources created from templates. A stack can be provisioned multiple times, resulting in the creation of specified AWS resources. Charges for these resources begin when they are created as part of the CloudFormation stack.

During stack deletion, AWS CloudFormation automatically removes the associated resources, following a predetermined order set by the service. Users do not have direct control over the deletion sequence.



The major components of an AWS CloudFormation template include:

  1. Parameters: An optional section containing name-value pairs that can be referenced later in the template.

  2. Mappings: An optional section typically used to refer to the values of the most current Amazon Machine Images (AMIs) in a region or other dynamic data.

  3. Resources: A required section that contains AWS objects to be created, representing various AWS services and their relationships.

  4. Init: The `CloudFormation::Init` resource type facilitates deploying applications, files, and resources onto EC2 instances during the deployment process.

  5. WaitCondition: The `CloudFormation::WaitCondition` resource type coordinates resource creation, allowing synchronization between dependent resources.

  6. Outputs: An optional section that returns string values created by the template, providing information that might be important to users.

Best practices for AWS CloudFormation usage and organization:

  • Utilize lifecycle and ownership concepts to determine resource placement within stacks, avoiding complexity as stacks grow.
  • Reuse templates to replicate infrastructure across multiple environments, such as development, testing, and production.
  • Verify quotas for all resource types before launching a stack to ensure compliance with AWS account limits.
  • Leverage AWS CloudFormation best practices for repeatable and secure infrastructure deployments.
Best practices for creating AWS CloudFormation templates:
  • Avoid embedding credentials; use input parameters for passing information.
  • Use specific AWS parameter types and constraints to quickly validate values.
  • Use `AWS::CloudFormation::Init` to deploy software applications on EC2 instances and describe configurations.
  • Validate templates before use to catch syntax and semantic errors.

Best practices for managing stacks:

  • Manage all stack resources through AWS CloudFormation; avoid making changes outside the service.
  • Create change sets before updating stacks to preview and evaluate proposed changes.
  • Use stack policies to protect critical resources from unintentional updates.
  • Enable AWS CloudTrail to log AWS CloudFormation calls for auditing purposes.
  • Use code reviews and revision controls to manage templates, allowing tracking of changes between different versions.
  • Store AWS CloudFormation templates on Amazon S3 for easier access and management.
  • If a `WaitCondition` times out or returns an error, check for issues in the `CloudFormation::Init` code.

These practices aim to enhance efficiency, security, and reliability when working with AWS CloudFormation.

Popular posts from this blog

Network Fundamentals for the Cloud

Image
Computer networking is all around us. From the days of dial up internet, internet connected devices have grown rapidly on the global scale. Despite this, across the globe there are broadly three types of computer network  - LAN/MAN and WAN.  The above diagram does a good example of representing the geographical elements of these networks. But it is worth elaborating bit further. LAN networks are usually used in office locations and connected by physical cables and local wifi. MAN networks are typically interconnected LAN networks with a private network link and a connection to the internet. MAN use cases are mostly reserved for governmental or state institutions, along with huge corporates - I'd imagine Alphabet Inc in Silicon Valley might use this for example. Whilst WAN connects devices in a large geographic area, across multiple cities or countries. WAN's are used to connect LANs. You could argue that the biggest WAN in the world is the internet itself. Some of the fundamen...
Read more

Familiarizing with the Command Line Interface

Image
I'm beyond lucky to have been a successful applicant onto the Amazon re/Start programme. This week on our intensive bootcamp we've been focusing on manoeuvring around the command line interface on Linux. Due to being computer active before YouTube (the delightful days of dial-up internet!) this wasn't all too unfamiliar to me for three specific reasons; Internet Relay Chat , or as the acronym goes - IRC. Before the endless content on YouTube, if you wanted to have a copy of a music video you either had to rely on the artist putting a movie file on a CD single or  you relied on a fan club setting a server/channel up on IRC. To get a music video on IRC, the wizkids of the era set-up a system identical to a Linux CLI. So there I was at 13, deploying my dir and get commands to get my own personal slice of MTV heaven! Raspberry Pi Inspired by the super cool RetroFlag SNES case for the RaspberryPi and all amazing bespoke retro arcades I saw people posting online, I went ahead ...
Read more

Security Fundamentals for the Cloud

Image
Security is the practice of protecting valuable assets. When we talk about information security, cyber security is actually just a subset of this discipline. Cybersecurity is focused on protecting digital related devices – like networks, systems and digital information. Focus on stopping unauthorised access, malicious actions like theft, destruction or alteration or simply disrupting the service. One of the most popular models for thinking about information security is the CIA triad. Confidentiality, Integrity and Availability.  Threats can come in many different forms. The tables below provide some a variety and most interestingly also explore what part of the Triad is impacted.    So, working with the security lifecycle model we can determine what actions we can take at each stage to deliver a robust security infrastructure within our organisation.  Prevention Identify assets – what devices are you using? Are they contemporary? Can you flash newer firmware on them,...
Read more

CLI Fundamentals for the Cloud

Image
Operating a computer with BASH is the reality the end-user faced prior to Window's graphical user interface. In the end however, what we're actually doing is rather much the same. By deploying actions via a CLI, we're empowered in our agency in what we can schedule and automate in the environment.  Ensuring we're starting off at the right directory is key, for that we use; pwd This outputs are current directory which informs us were we're currently sitting at in the system. A basic rule of command syntax is that is delivered in below format; command -> option - > argument eg. sudo usermod -a -G Sales arosalez sudo usermod -> -a -G  -> Sales arosalez The most basic commands revolve around providing quick user or system data. I won't elaborate on the most of these as they are either ubiquitous or pretty self evident. ls, cd, whoami, id, hostname, uptime, date cal, clear, echo, history, touch, cat Don't forget to use the tab autocomplete function...
Read more

DataDog, a Cloud Analytics & Monitoring application

Image
Due to having an industry connection, I had the opportunity to do something really fun this weekend. The application running on AWS is a CRUD API - however, Datadog, a Cloud Analytics & Monitoring agent has been installed to provide feedback on what's actually happening. I found this so beyond awesome, the scope of this product is really impressive.  On the left side you can see a command line running on my Linux VM. On the right you can see a live monitoring of the AWS machine, tracking the inputs received on the API application.  As I send commands into the CRUDE API application, DataDog continues monitoring and updating our dashboard. If you can imagine for the deployment of a much busier service, the possibilities of this kind of real-time monitoring isn't to be underestimated.   Above you can see the CPU usage graphs, but unlike the ones you find on the AWS console, you can see that DataDog manages to also show us the split of the CPU usage.  Even in th...
Read more

A brief introduction to Databases and MySQL

Image
Thanks to completing an IT GNVQ during my secondary school days, databases aren’t all that new to me. I had experience querying a relational database in Microsoft Access already. But for the purpose of this post, I’ll revisit some core principles. The first to example is the types of database, which I’ll use an infographic to explain; Now what we’ll be looking this post is the relational model. This works by having a series of tables linked by public and foreign keys. Each of these keys has to be completely unique. When we update our database, we term this as a transaction. This means one or more changes are performed to a database. To commit a change we need to ensure transactions follow the four standard principles of atomicity, consistency, isolation, and durability. Before we go into talking about querying our relational database, let’s get into the NULL value. When searching we can use IS NULL and IS NOT NULL, but beyond this, there are several useful things to know about this v...
Read more

AWS CodeCommit + Creating a CI/CD pipeline

Image
  Having signed up to the AWS Skill Builder I thought it was something I'd be looking at post-programme. However the above builder labs email landed, I thought, what better way to spend my Sunday afternoon? So the fundamentals lab was working with the AWS service CodeCommit. Having gone through most of gitimmersion  this lab felt very straight forward. It involved creating a local and "remote" (ie. CloudCommit) git repository and synchronising them together. The intermediate lab now proved a little more tricky. First of all was to comprehend that we were going to build a CI/CD pipeline - ie. manifesting the principles of DevOps. Below I've posted a full scale CI/CD pipeline to demonstrate the complexity. However, in this lab we're only using CodeCommit, CodeBuild, CodeDeploy, AWS ECR and ECS. This lab involved the most complex technology stack I've come across thus far, so really appreciated doing something a little more challenging. However, it did involve so...
Read more

Future Orientation: Tips from a AWS re/Start Graduate

Image
  Today we had a recent AWS re/Start Graduate join us for a short presentation and Q&A about their experience both about the programme and their post-programme period before starting a role as a Cloud Engineer.  It was great to hear from someone who less then 12 months ago had also undergone the same exact programme we're a third of the way through now. We got an insight into their day to day (lots of meetings!), how long it took until they were allowed to pick up tickets independently (two months) and how much did their employer support their development (lots!). By chance, they were scheduled to complete the solutions AWS solutions architect certification tomorrow! I found it incredibly motivating and centring to hear their story. I made sure to ask a question myself, which revolved around finding potential roles and then knowing the essential skills that we might need to enhance our employability. The advice provided was; Make sure to continue studying Networking. This ...
Read more

A brief introduction to AWS Cloud Adoption Framework (CAF) and Well-Architected Framework (WAF)

Image
To aid in moving more businesses into the cloud AWS have developed the Cloud Adoption Framework (CAF). There are also competitors’ models, but in the broadest sense they tend to follow the same principles and categories. I’ll start with showing broad categories, then the Azure model and the AWS model.  So now we've familiarised ourselves with the broad categories, we can see the two major cloud providers high-level frameworks presented; Now the 2nd infographic of the AWS CAF begins to show more detail, exposing us to the genuine complexity behind the considerations required during the CAF process. If a business is consequently adopting a cloud, well, it also makes sense to have a structure to think about how the future architecture will be designed and if it matches cloud best practice, in other words AWS's Well-Architected Framework (WAF). Thankfully like many of my posts, there are infographics that excellently and succinctly summaries these ideas for us; It is very likely th...
Read more

Building a VPC in AWS

Image
  For our very last lab about the network, we were given the following customer environment to build. By virtue of the fact the AWS management console GUI is constantly updated it meant that our task instructions were wholly out of date. However, this actually proved to only make it a far more involved and fun process - necessitating lots of troubleshooting when things had not spun up as expected. I felt like I had learnt a lot more this way and for the benefit of my future self, I thought I would make some step by step notes (not necessarily in reference to the diagram above) to serve as prompts for any future cloud VPC's I'll be spinning up. Our challenge was to build the VPC as the diagram above.  Create an elastic IP. We will associate this later with our NAT gateway. Launch your VPC with a private class IP range without forgetting to specific how many availability zones (AZ) you want. Create and label your subnets, again specifying what AZ, as none are public at the ...
Read more