How to monitor EC2, CloudWatch, EBS, RDS, ELB, ElasticCache using metrics

AWS is the front runners when it comes to have highly available, fault tolerant, secure and high scaling service which can integrate with almost everything in cloud as well as your own data center.

For monitoring your VPC(Virtual Private Cloud), AWS has these two very renowned services:
– CloudWatch and
– CloudTrail

While ‘CloudTrail’ is primarily used to monitor the API calls made to other services or Applications, ‘CloudWatch’ is used for monitoring and logging events periodically (default every 5 minutes, detailed every minute).

It can be used to monitor:
– Compute resources like EB2s, ELBs, Route53, Auto Scaling Groups,
– Storage & CDN resources like  S3, CloudFront, EBS Volumes,
– Database and Analytics services like DynamoDB, Elastic cache, RDS, Elastic MapReduce, Redshift
– SNS, SQS etc.
Let’s cover them one by one on how CloudWatch monitors different services:

A- EC2(Elastic Compute Cloud)

CloudWatch can monitor the following metrics:

# CPU 

– CPUCreditUsage (number of CPU credits consumed by the instance. One CPU credit equals one vCPU running at 100% utilization for one minute)

– CPUCreditBalance (number of CPU credits available for the instance to burst beyond its base CPU utilization, expire every 24 hrs)

– CPUUtilization (percentage of allocated EC2 compute units)

# Network

– NetworkIn (number of bytes received on all network interfaces by the instance)

– NetworkOut (number of bytes sent out on all network interfaces by the instance)

– NetworkPacketsIn (number of packets received on all network interfaces by the instance)

– NetworkPacketsOut (number of packets sent out on all network interfaces by the instance)

# Disk

– DiskReadOps (Completed read operations from all instance volumes)

– DiskWriteOps (Completed write operations to all instance store volumes )

– DiskReadBytes (Bytes read from all instance store volumes )

– DiskWriteBytes (Bytes written to all instance store volumes)

# Status Check

– StatusCheckFailed (Reports whether the instance has passed both the instance status check and the system status check in the last minute)

– StatusCheckFailed_Instance (whether the instance has passed the instance status check in the last minute.)

– StatusCheckFailed_System (whether the instance has passed the system status check in the last minute.)

– You can create alarms based on these above metrics to watch the health of the host and the instances.

Rest of the article to be continued…

 

How to manage website Failover using AWS Route 53 and a website hosted on an external domain

Scenario:

You have 2 websites:

  • “futureCloud.technology”, which is the primary website, is hosted on AWS EC2 Instance supported though an ELB(Elastic Load Balancer). Please keep in mind that GoDaddy.com is just the site registrar for the site name the site is not hosted there.
  • “mohdnaeem.wordpress.com”, which is the secondary/failover website is hosted on WordPress.com domain.
  • You are using Route 53 DNS service to resolve the domains.

Problem:

  • “futureCloud.technology” should be the primary website and should run when the EC2 Instance is healthy.
  • In case of failover the external website “mohdnaeem.wordpress.com” becomes the failover website. The requests for “futureClod.technology” should be routed to this external secondary website in case of the primary websites failure.

Solutions:

  • There can be various solutions. Let’s see them one by one.
  • Solution 1:
    • It is very primitive solution. Just write a domain forwarding rule on your registered websites domain manager panel. Any requests for “futureCloud.Technology” website will automatically be forwarded to “mohdnaeem.wordpress.com”
    • See snapshot below:
    • godaddy forwarding
    • Issue – this is a very basic solution and AWS Route 53 never comes into picture. It should be Amazon’s Route 53 which should dictate the failover logic not registrar.

 

  • Solution 2:
    • We will use AWS Route 53 to dictate the website failover from Primary to Secondary.
    • I am assuming the following:
      • That you created an AWS EC2 instance. (Ideally you create a couple of Instances hosting WordPress website and couple of MySQL Instances for your site to by highly available, fault tolerant and failover tolerant. But assuming that you have installed the WordPress website only on one LAMP server (Linux, Apache, MySQL and PHP). So that in case this primary website goes down then you can run the secondary from an external domain.)
      • That you already installed and configured ‘WordPress’ on the server.
      • That this site is configured behind an ELB (Elastic Load Balancer).
      • That your primary website is running on EC2 instance and failover secondary website (“mohdnaeem.wordpress.com”) is running on WordPress domain.
      • That you have registered a domain name ( say futureCloud.technology) at any domain registrar. I did at Godaddy.com)
    • So now let’s take the things forward from here.
      • First you will have to create a hosted zone exactly matching your registered domain name “futureCloud.technology” for your primary website in Route 53.
      • As you create a hosted zone in Route 53, it creates a couple of NS(Name Server) records. A SOA(Start of Authority) record. You will have to add a ‘A’ record for your EC2 website configured behind the ELB and another ‘A’ record for a S3 bucket (exactly matching the name of the registered website and configured as a static website). We will use this bucket for redirection.
      • See snapshot:
      • AWS Hosted ZOne futureCloud for EC2
      • AWS Hosted ZOne futureCloud for S3
  • Note down the 4 name servers and login to your registrars domain manager control panel(mine was GoDaddy.com) and add those 4 entries as Name Server records.
  • See snapshot:
  • godaddy nameservers
  • Now are configured to run your primary website, go ahead and open a browser and run the website – “futureCloud.technology”.
  • See snapshot:
  • AWS S3 buckets
  • Now you have to configure the failover so that when the EC2 instance is not healthy or not running your secondary failover website should run.
  • To do that first you have to create another bucket (exactly matching the name of the external failover website “mohdnaeem.wordpress.com”.
  • See snapshot:
  • S3 Redirect from futureCloud S3 to wordpress S3
  • In the bucket “futurecloud.technology” configure it as a website with a redirection rule to redirect to the other bucket “mohdnaeem.wordpress.com”.
  • See snapshot
  • S3 Redirect from futureCloud S3 to mohdnaeem.wordpress.com
  • In the bucket “mohdnaeem.wordpress.com” configure it as a website with a redirection rule to redirect to the external website “mohdnaeem.wordpress.com”.
  • See snapshot:
  • AWS S3 buckets
  • In case of EC2 is not healthy or not running the request will first reach “futurecloud.technology” bucket but it will redirect to the other bucket which is configured to redirect it to your external site.
  • Go to the Route 53 to configure another “hosted zone” which exactly matched the name of the external domain name (“mohdnaeem.wordpress.com”) and add an ‘A’ record as failover to the bucket by the same name.
  • As EC2 instance is down, the Route 53 first looks for s3 bucket “futureCloud.technology” but sees a redirection rule for bucket “mohdnaeem.wordpress.com” and which in turn has a redirection to the actual external website “mohdnaeem.wordpress.com”. The hosted zone entry thus maps this failover and redirects you to the external website.
  • Now go to the bowser and run “futureClod.technology”, but this time you will see that the website is redirected to “https://mohdnaeem.wordpress.com”
  • See snapshot:
  • external website running
  • All these redirection setting are being done because the AWS Route 53 is still not a very powerful DNS tool.

Thanks for reading this article. Please contact me in case you need any help. Please go to “About Us” page to view my details.

CloudWatch – a service to do EC2 Instance Health Check/Monitoring , Troubleshooting, Metrics and Analysis

The Health Check/Monitoring , Troubleshooting, Metrics and Analysis of the EC2 instances and getting timely alerts to fix the problems to keep your cloud architecture highly available, auto-scaling and fault tolerant are one of the important roles and responsibilities of a cloud architect or SysOps admin. Let’s check how we can achieve this.

So lets first try to understand what CloudWatch is – Its is a AWS’s health monitoring service to monitor the AWS resources and the applications. It can monitor the following:
– Compute resources like Auto scaling groups, Load balancers, Route 53,
– Storage resources like EBS volumes, storage gateways, Cloud Front,
– Database services like relational RDS instances, non-relational services                   like DynamoDB,
– Analytics services like Elastic Map Reduce, Red Shift,
– In-memory cache services like Elastic Cache to name a few.

The CloudWatch can monitor the following metric:
 – CPU Utilization
 – Disk Reads
 – Network In and Outs
 – Status checks
But it can’t check a few other metrics like Memory Utilization for that we have to add custom metrics, which we will see later in this post.

The default monitoring checks these metrics every 5 minutes whereas the detailed monitoring is every 1 minute.

The status checks listed above can be of two type:
 – System Status Checks – checks related to the host on which the instance                  is virtualized. E.g Loss of network or power,  software or hardware issues               on the host machine. Normally restarting/terminating the instance or                           contacting AWS are the options available.
 – Instance Status Checks – checks related to the VM(Virtual machine) itself.               E.g. memory leaks, corrupted file system, incompatible file system,                                  mis-configured network. Normally restarting/terminating the instance or               checking/trouble shooting your own application for bugs are the options.

On the AWS console go to the CloudWatch service :
 – Click “Create dashboard”
 – Add a widget to dashboard based on the metrics listed above
 – Save the dashboard.(See snapshot below)
 

Now what if we want to monitor a custom metrics(Memory Utilization) which is not monitored by default by CloudWatch. Well then we have to use some custom scripts for it. Lets see how it is done.

- Install the required packages:
 sudo yum install perl-Switch perl-DateTime perl-Sys-Syslog perl-LWP-Protocol-https
- Download the CloudWatch Custom Monitoring Scripts:
 curl http://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.1.zip -O
- Unzip the scripts:
 unzip CloudWatchMonitoringScripts-1.2.1.zip
 rm CloudWatchMonitoringScripts-1.2.1.zip
 cd aws-scripts-mon
- Execute the script(You will get a "Successfully reported metrics to CloudWatch. Reference Id: 84bf63d3-2841-11e7-a20f-7786b8297dbd
" message on success):
 ./mon-put-instance-data.pl --mem-util --mem-used --mem-avail
- Add a crontab job for 5 minutes intervals:
 */5 * * * * ~/aws-scripts-mon/mon-put-instance-data.pl --mem-util --disk-space-util --disk-path=/ --from-cron

Once you have run these scripts successfully, the custom metrics for memory utilization will also be available and you can add it as a widget. See below.

Very Intuitive and Easy going step by step AWS tutorial for Beginners

Please try this link – http://bit.ly/2guw5gY

I have tried a lot of sites like acloud.guru, linuxavademy.com, udemy.com etc to name a few, but the best one is this one.

It is very intuitive and easy going step by step AWS tutorial for Beginners. In which it helps you to create a highly available, auto-scaling WordPress website using EC2 instances, MySQL database over a pair of public subnets hosting the instances, a pair of private subnets hosting the databases enforced with load balance and Route53 DNS Management. Further is also uses CloudFront as CDN for website media, Cloudwatch and SNS for Alerts, Failover management.

I have never seen all this much put into one tutorial. In fact this website is hosted using the same approach. If you are an AWS enthusiast, this website will give you the fire in you to start sailing on this AWS boat and charm to dive deeper to learn more and advanced topics and services in AWS.

Wish you guys best of luck.

Naeem.

Latest AWS Certifications

Now a days I am very fascinated with Cloud Technology and Big Data. These are the technologies of the future and have immense potential and thus have been working hard for the past 6 months and did clear these 2 certifications:

– AWS Certified Solutions Architect – Associate

– AWS Certified Developer – Associate

Next in target are –

– AWS Certified SysOps Administrator – Associate

– AWS Certified Bigdata Certification

– AWS Certified Security Certification