Marks AWS Solution Architect Course notes

Marks AWS Solution Architect Course notes – Apr/May 2017

To give back to the wider community, here are all my course notes for the AWS Solution Architect certification exam. If you read through all of these bullet points and diagrams and understand the lot, you will know enough to pass the exam. to prove that, this is everything I learnt to pass the exam myself, late May 2017.

Contents:
IAM (Identity Access Manager) 2
S3 (Simple Storage Service) 2
CloudFront 3
Storage Gateway 4
Snowball 5
EC2 (Elastic Cloud Compute) 5
Security Groups 7
Volumes and Snapshots 7
AMI (Amazon Machine Image) Types 8
ELB (Elastic Load Balancer) 8
CloudWatch 9
AWS CLI 10
IAM Roles – instead of Programmatic access 10
IAM Roles – instead of creating users in multiple accounts 10
Using Bootstrap/Bash scripts (User data) 11
Instance Metadata 12
Auto-Scaling 12
EC2 Placement Groups 14
EFS (Elastic File System) 14
Lambda 14
DNS Basics 14
Route53 15
Databases 19
RDS – Relational Database System 19
DynamoDB 20
Redshift 20
Elasticache 21
Aurora 21
VPC Overview 22
Build a custom VPC 23
NAT Instances & NAT Gateways 24
Network ACL’s vs Security Groups 27
ELB’s and VPC’s 28
NAT vs Bastion 28
Direct Connect 29
SQS (Simple Queuing Service) 29
SWS (Simple Workflow Service) 30
SNS (Simple Notification Service) 31
Elastic Transcoder 31
API Gateway 31
Kineses 32
Environment setup in course 33
‘Overview of AWSAWS’ Whitepaper 34
Overview of ‘AWS Security Processes’ Whitepaper 35
Overview of ‘Storage Options in the Cloud’ Whitepaper 36
Overview of ‘Architecting for the Cloud Best Practices’ Whitepaper 36
OpsWorks 38
Tags and Resource Groups 38
The well-architected framework – Pillar One, Security 38
The well-architected framework – Pillar Two, Reliability 39
The well-architected framework – Pillar Three, Performance Efficiency 39
The well-architected framework – Pillar Four, Cost Optimization 40
Exam Practicalities 40

IAM (Identity Access Manager)

  • Users, Groups, Roles, Polices.
  • IAM is Global, can’t select a region.
  • https://markbaws.signin.aws.amazon.com/console < account name
  • Console Root login is at: https://console.aws.amazon.com/console/home Root account has Full access.
  • ‘Access key ID’ and ‘Secret access key’ used for API/CLI access, you only get to see and download (.CSV format) the ‘Secret Access key’ when you create it and never again SO DON’T LOOSE IT.
  • Password are for Users to use, Password Polices should be set.
  • You can attach Polices aka Permissions to Users and Groups, Polices are in JSON format e.g.

   {
      “Version”: “2012-10-17”,
      “Statement”: [
         {
            “Effect”: “Allow”,
            “Action”: “*”,
            “Resource”: “*”
         }
      ]
   }

  • Roles allow AWS services to talk to other services, supports across AWS accounts too.

S3 (Simple Storage Service)

  • S3 is Object storage not Block storage
  • Unlimited capacity, Max file size is 5TG
  • Virtual-hosted-style URL format: https:// markybbuckets3-eu-west-1.amazonaws.com/ < region name bucketname which must be unique in the world
  • Path-style URL format: https://s3-eu-west-1.amazonaws.com/markybbucket < region name bucketname which must be unique in the world
  • When you PUT (create) a new object it’s available immediately, when you update/overwrite an object or delete, it may take some time.
  • S3 – 99.99% availability, 99.999999999% of durability (11 x 9’s of durability)
  • S3 infrequently Accessed (S3 IA) – lower cost than S3 but your charged for retrieval
  • S3 Reduced Redundancy (S3 RRS)- design to provide 99.99% availability and a lower 99.99% durability
  • Glacier – very cheap, takes 3-5 hours to restore. No SLA for Glacier. Minimum charge period for Glacier is 90 days, if you store something for less than 90 days you still pay for 90 days.
  • ‘S3 transfer acceleration’ leverages CloudFront to let users upload to a CloudFront edge location and AWS copies it to your bucket in your region. It’s much faster to upload for the users. The ‘Amazon S3 Transfer Acceleration Speed comparison’ site can show how much faster it will be than talking directly to the source buckets location
  • By default, all newly created buckets are PRIVATE.
  • You can setup logging to the same bucket or to another bucket.
  • Static website hosting, website must not need server-side technologies
  • Versioning keeps multiple versions of an object in the same bucket. To restore a deleted file, just delete the ‘delete marker’. Once enabled, Versioning cannot be disabled, only suspended. Versioning’s MFA Delete capability adds MFA for additional security against deleting.
  • Enable ‘Cross-region replication’ to host files in multiple regions. Create the Destination bucket first. Destination bucket must have ‘Versioning enabled’. A new Role will get created for source bucket to talk to destination bucket. Permissions are also replicated across.
  • ‘Lifecycle Management’ for example automatically move after 30 days to ‘Infrequently Accessed’ and after 60 days to ‘Glacier’ and after 425 days delete.
  • Read S3 FAQ before the exam.

CloudFront

  • CloudFront is a CDN (Content Delivery Network), edge locations are in most of the major cities around the world, there are more than 50.
  • Edge locations not just READ only, you can write to them too.
  • ‘Origin’ is where our original files are, it can be the following, you can have multiple Origins:
    • S3 bucket
    • EC2 instance
    • ELB
    • Route53
  • ‘Distribution’ is the name given to a collection of Edge locations that have your content, can be:
    • Web Distribution – used for websites
    • RTMP – Used for Media Streaming
  • You can restrict certain peoples access by using the ‘Restrict Viewer Access (use signed URLs or Signed Cookies)’ option
  • Price classes:
    • US and Europe
    • US, Europe and Asia
    • All Edge locations (Best Performance)
  • Without using Route53 to create a friendly DNS name it will create a name like: https://dw7qq87sxnwz9.cloudfront.net/afilename.jpg
  • ‘Geo-Restriction’ setting allows you to Blacklist countries
  • Encryption options
    • In Transit
      • SSL/TLS (https)
    • At Rest
      • Server Side Encryption
        • SSE-S3 – S3 Managed Key – SSE-S3 provides server side encryption, but Amazon manages the keys of the object storage system, This system makes sure uploaded data is encrypted when stored on Amazon’s servers. The risk of losing the data due to lost keys is eliminated.
        • SSE-KMS – AWS Key Management Service, Managed Keys – SSE-KMS is most advanced, allowing you to manage and audit the keys and providing a level of advanced control over the SSE-S3 service.
        • SSE-C – Server side encryption with customer provided key – Customers provides the encryption keys to Amazon, and they encrypt all data with your public key so that ONLY you can only read the data with your private key.
      • Client side Encryption – User encrypts the data on client-side and uploads to S3

Storage Gateway

  • Storage gateway is an ESX or Hyper-V VM that sits on-prem.
  • 4 different types:
    • File Gateway (NFS) – stores flat files in S3
    • Volumes Gateway (iSCSI) – can be added to any server that supports iSCSI and appears as a disk
      • Stored Volumes – Stores entire data on site and it synchronises to S3 snapshots
      • Cached volumes – Stores only frequently accessed data on site and it synchronises to S3 Volumes and snapshots
    • Tape Gateway (VTL) – Backup software can connect via iSCSI to the VTL.

Snowball

  • Can import to S3 and export from S3
  • Snowball
    • 80Tb capacity
    • 256-bit encryption and uses a TPM
  • Snowball Edge
    • 100TB capacity
    • Has on-board compute capability also
  • Snowmobile
    • Is a truck for 100 Petabytes capacity.

EC2 (Elastic Cloud Compute)

  • On-Demand Instances
  • Reserved Instances
  • Spot Instances – If spot instances are terminated by Amazon and this happens if the spot price goes above your bid price, you will not be charged for a partial hour of usage. However, if you terminate the instance, you will be charged for any hour in which the instance ran.
  • Dedicated Hosts
    • you get the whole host so you don’t share it with others,
    • helpful for fussy licensing,
    • costs more.
  • Instance types, remember ‘Dr Mc GIFT PX’ The Scottish Dr who gifts pictures (PX’s)

    • D for Density Storage
    • R for RAM
    • M – Main choice for general purpose apps
    • C for Compute
    • G for Graphics
    • I for IOPS
    • F for FPGA (Field Programable Gate Arrays) (CPU can be reprogrammed to desired application or functionality)
    • T cheap general purpose (think T2 Micro)
    • P – Graphics (think Pics)
    • X – Extreme Memory
  • EBS (Elastic Block Storage)
    • Provisioned IOPS SSD (IO1)
      • Designed for IO intensive apps
      • If you need more than 10,000 IOPS, max 20,000 IOPS
      • Most expensive
    • General Purpose SSD (GP2)
      • Balances both price and performance
      • 3 IOPS per GB up to 10,000 IOPS, ability to burst up to 3000 IOPS for extended periods of time for volumes under 1 GB
    • Throughput Optimized HDD (ST1) – Magnetic storage
      • Useful for: Big data, data warehouses, Log Processing, sequential writes, cannot be a boot volume
      • Cheaper than General Purpose and PIOPS
    • Cold HDD (SC1) – Magnetic storage
      • Useful for: File Server instead of S3, cannot be a boot volume
      • Lowest Cost
    • Magnetic (Standard) – previous generation but still selectable
    • You cannot mount 1 EBS volume to multiple EC2 instances, instead use EFS
    • Root volume is the boot volume
    • EBS Volumes attached to EC2 instances ‘Delete on Termination’ by default is ticked
  • AMI (Amazon Machine Image)
    • HVM – Hardware Virtual Machine
    • PV – ParaVirtual

Note: (Amazon recommends choosing HVM over PV, HVM makes more virtualisation work for the AWS Xen hypervisor but it is more manoeuvrable around underlying hosting hardware, using PV you will be greatly restricted in your choice of instance types)

  • ‘Amazon Linux AMI’ always updated and supported by AWS
  • ALWAYS one subnet = one availability zone!
  • When you deploy a new Linux VM update packages with:
    • To elevate to super user: sudo su
    • To install all patches: yum update -y

Security Groups

  • ‘Security Groups’ are just firewalls; any change is effective immediately
  • All inbound traffic is blocked by default
  • All outbound traffic is allowed by default
  • Inbound and outbound rules, specify the Type e.g. HTTP, Protocol e.g. TCP Port Range e.g. 80, Source e.g. 0.0.0.0/0 for everything or select another subnet.
  • ‘Security Groups’ are stateful, that means if you allow inbound HTTP, it automatically lets the HTTP traffic back out, you don’t have to add an outbound rule.
  • You can attach a security group to multiple EC2 instances, and add multiple security groups to a EC2 instance.
  • You cannot block specific IP addresses using Security Groups, instead use Network Access Control Lists (ACL’s).

Volumes and Snapshots

  • Volumes exist on EBS (Elastic Block Store)
  • Create a Volume from the available types: Provisioned IOPS SSD (IO1), General Purpose SSD (GP2), Throughput Optimized HDD (ST1), Magnetic storage or Cold HDD (SC1)
  • Attach it to an EC2 instance
  • How to mount and format a new volume on Linux:
    • lsblk to list all drives mounted, you’ll see a drive with no partition
    • file -s /dev/xvdf good habit to check its empty first, it it returns data then its empty
    • mkfs -t ext4 /dev/xvdf to format it
    • mount /dev/xvdf /myfileserver this mounts the new drive to the folder /myfileserver
    • umount /dev/xvdf to unmount a volume
  • Snapshots exist on S3
  • Create a snapshot which is like a photo of the volume, and just give it a name and optional description
  • Snapshots are incremental, this means that only the blocks that have changed since your last snapshot are moved to S3
  • You can create a new volume from a snapshot, this lets you change the AZ, volume size and the volume type as well.
  • RAID types, only use OS level RAID when you’re not getting the disk IO you need:
    • RAID 0 – Striped, No Redundancy, Good Performance
    • RAID 1 – Mirrored, Redundancy
    • RAID 5 – Good for reads, bad for writes, AWS does not recommend ever putting RAID 5 on EBS
    • RAID 10 – Striped & Mirrored, Good Redundancy, Good Performance
  • To take a snapshot of a RAID array you can do this 1 of 3 ways:
    • Freeze the file system
    • Unmount the RAID Array
    • Shutdown the associated EC2 instance
  • To encrypt a root (boot) device:
    • Stop the instance
    • Create a snapshot of the volume
    • Copy the snapshot and tick ‘Encrypt this snapshot’
    • Create an AMI from the snapshot
    • Select the AMI and Launch a new instance
  • Volumes restored from encrypted snapshots are encrypted automatically
  • You can share snapshots to other AWS accounts or made public, but only if they are unencrypted, this is because the encryption key is linked to your AWS account.

AMI (Amazon Machine Image) types

  • Items specified in a AMI:
    • Region
    • Operating system
    • 32bit or 64bit
    • Launch Permissions
  • Storage for root device type
    • Instance store (Ephemeral storage) – You cannot stop the instance, only reboot and terminate. If the instance is stopped from say a host failure, the data is lost. It is created from a template stored in Amazon S3.
    • EBS backed volumes – You can stop and start and the data is persistent. It is created from a AMI which is stored on EBS. EBS volumes should be used for storing data long term.
    • By default, both Root volumes will be deleted on termination (delete a EC2 instance)

ELB (Elastic Load Balancer)

  • Classic Load Balancer – Layer 4
    • Configure ‘Health Check’ the protocol e.g. HTTP, port 80 and a ping path e.g. /healthcheck.html or /index.html
    • Add the EC2 instances
  • Application Load Balancer – Layer 7 which is the Application layer – recommend over the ‘Classic Load Balancer’ by AWS now.
    • Can route requests to one of more ports on each EC2 instance or container instance
    • Create a ‘Target Group’ and give it a name, select the Protocol e.g. HTTP and port 80
    • Create Health check e.g. HTTP and /healthcheck.html
    • Port can be ‘traffic port’ which uses the target groups port, or select ‘override’ and select your own port.
    • ‘Success codes’ enter the HTTP codes to use when checking for a response from a target, can be a range e.g. 200-299
    • ‘Register targets’ you add EC2 instances.
  • ELB’s can only be public or private, it cannot be both.
  • ELB’s have their own DNS name, you are never given an IP address
  • Instances monitored by ELB are reported as; ‘InService’, or ‘Out of Service’

CloudWatch

  • CloudWatch is for monitoring instances and logs, CloudTrail is the security audit trail of AWS
  • Standard Monitoring every 5 minutes is free, Detailed Monitoring every 1 minute is paid for.
  • Create a ‘Dashboard’
    • Add Widgets, types
      • Line, Stacked area, Number, Text
    • Metrics available by default for EC2:
CPU related:
 Disk related: Network related:
 StatusCheck related:
      • RAM not available, this would be a Custom metric, need a bit of code inside the EC2 instance to report on this.
  • Create ‘Alarms’
    • Select the metric to alarm on, e.g. ‘CPUUtilisation’ and the threshold e.g. >= 80
    • Give it a name and optional Description
    • Add Notification to an email address (user must confirm the email address within 72 hours)
  • Create ‘Events’
    • Respond to state changes in your AWS resources.
    • E.g. when an EC2 instance comes online, run a Lambda function
  • For OS and Application level log support, install an Agent on EC2 instances which can pass OS and Application logs to CloudWatch.

AWS CLI

  • Install the AWS CLI on Linux, Mac and Windows or it’s already installed on most AWS marketplace AMI’s
  • User must have ‘Programmatic access’ with a ‘secret key ID’ and ‘secret access key’. These are given to you in a .CSV file/ Don’t forget to attach a Policy to the user e.g. Managed policy ‘AdministratorAccess’
  • Use aws configure command to connect to a AWS account with the CLI, although this can be bad as it stores credentials locally on your PC/server.
    • Add ‘AWS access key ID’
    • Add ‘AWS Secret Access key’
    • Add default region name e.g. ap-southeast-2 for Sydney
  • aws s3 ls < lists everything in S3 on your connected account
  • aws s3 help < type aws then the service then help to give you list of all commands

IAM Service Roles – instead of Programmatic access

  • Using Roles is more secure than using ‘Programmatic access’ with a ‘secret key ID’ and ‘secret access key’, this is as ‘aws configure’ stores these keys in plain text format in a file called ‘credentials’ on the server.
  • 3 Role types can be created:
    • AWS Service Roles e.g. Amazon EC2 allowed access to S3 bucket
    • Role for Cross-Account Access
    • Role for Identity Access
  • When creating an EC2 instance, select your ‘IAM role’. This means when you login to this EC2 instance you don’t need to run the ‘aws configure’ command and add in your secret access key and ID, which is good as you don’t want the credentials stored on your EC2 instance. The role you’ve attached to the EC2 instance gives you access already.
  • In the AWS console, you cannot add/remove/change the IAM role attached to a EC2 instance. You can add/replace the IAM role via the AWS CLI though. You can also attach more polices to an existing role to give more access.
  • Changes to any IAM Roles are effectively immediately.
  • Roles are worldwide, they work across all regions.

IAM Cross-account Roles – instead of creating users in multiple accounts

  • Switching roles enables you to manage resources across AWS accounts using a single user account.
  • When you switch roles, you temporally take on permissions assigned to the new role.
  • When you exit the role, you give up those permissions and get your original permissions back.
  • Steps to enable a role in another account:
    • Identify our account numbers
    • Create a group in IAM e.g. DevGroup
    • Create a user in IAM e.g. John
    • Log in to Production
    • Create the “read-write-app-bucket” policy
    • Create the “UpdateApp” Cross Account Role e.g. MyDevelopersAccess
    • Apply the newly created policy to the role
    • Log in to the Developer Account
    • Create a new inline policy
    • Apply it to the Developer group
    • Login as John
    • Switch Account

      Shows up in the top right of the console you have switched Roles:

Using Bootstrap/Bash scripts (User data)

  • In this example, make sure the instance has an IAM Role attached to be able to read a S3 bucket, as the last line of this user data code reads from a S3 bucket.
  • In User data (this example is for a Linux instance):
    #!/bin/bash < always start with a ‘shebang’ which states the path to the interpreter
    yum update -y     < install all latest patches
    yum install httpd -y     < install Apache web server
    service httpd start     < start Apache
    chkconfig httpd on     < always start Apache
    cd /var/www/html
    aws s3 cp s3://nameofbucket/index.html /var/www/html –region regionname     < copy index.html from S3

Instance Metadata

  • When inside an EC2 instance, run the ‘curl’ or ‘get’ commands to show metadata.
  • curl http://169.254.169.254/latest/meta-data/ < shows the available metadata sections, which are:
    ami-id
    ami-launch-index
    ami-manifest-path
    block-device-mapping/
    hostname
    iam/
    instance-action
    instance-id
    instance-type
    local-hostname
    local-ipv4
    mac
    metrics/
    network/
    placement/
    profile
    public-hostname
    public-ipv4
    public-keys/
    reservation-id
    security-groups
    services/
  • curl http://169.254.169.254/latest/meta-data/public-ipv4 < shows the public IP as an example
  • curl http://169.254.169.254/latest/meta-data/public-ipv4 > mypublicip.html < writes the pubic IP address to a file
  • Key thing to remember is that it’s an instances META DATA, not user data can be retrieved.

Auto-Scaling

  • Before you create an auto-scaling group, create a ‘Launch configuration’, basically is like creating a new EC2 instance, but doesn’t actually deploy it.
    • Select an AMI
    • You can select Spot Instances if you like
    • Select an IAM role
    • Put in your ‘User Data’ bootstrap code
    • Select IP address setting, assign a public IP or not.
    • Select the ‘key pair’.
  • Create ‘Auto scaling group’ details
    • Give it a name e.g. ‘MyAutoScalingGroup’
    • Group size, ‘Start with X instances’
    • Select the Network i.e. VPC
    • Choose subnets in different AZ’s for redundancy
    • Select the ELB
    • The Health Check can be from the ELB or EC2
    • HealthCheck grace period in seconds, this allows time for the EC2 instance to boot and run the ‘User data’ bootstrap code, e.g. 150 seconds is 2.5 minutes
  • Add Auto scaling polices
    • Set ‘Increase Group Size’ and ‘Decrease Group Size’ Policies
    • You can make it send a notification to an email whenever an instance:
      • Launches
      • Terminates
      • Fails to launch
      • Fails to terminate
    • If you delete an Auto scaling group, it will shut down and terminate all the instances in the group.

EC2 Placement Groups

  • Allows you to group EC2 instances together in a low-latency, 10-Gbps network
  • Only some type of instances can be launched in a Placement group (Compute optimised, GPU, Memory optimised, Storage optimised).
  • You can’t move an existing instance into a placement group, you have to create a new instance from an AMI.
  • Placement groups are Single AZ’s only.

EFS (Elastic File System)

  • Supports NFSv4
  • Windows is not supported
  • EFS is block based storage
  • You only pay for what you use
  • Scales up to petabytes capacity
  • Supports thousands of concurrent NFS connections
  • Can be mounted by multiple instances at the same time unlike EBS
  • Data is stored across multiple AZ’s within a region
  • You create ‘Mount Targets’ in the regions AZ’s for instances to connect to
  • Make sure your EC2 instances are in the same security group as the EFS mount targets
  • Mount your file system on Linux using the DNS name:

sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans 2 fs-d69672ef.efs.ap-southeast-2.amazonaws.com:/ efs

  • Do this on multiple instances so they can all read/write to the same EFS folder.

Lambda

  • Is a compute service where you upload your code and create a Lambda function.
  • Can be an event driven compute source that runs your code in response to an event
  • As a compute service to run your code in response to a HTTP request from Amazon API Gateway or API calls made using AWS SDK’s.
  • Languages supported – Node.js, Python, Java, and C#.
  • Priced off:
    • number of requests – 1st million requests are free, $0.20 per 1 million requests thereafter.
    • Duration – calculated from the time your code begins executing to finishes rounded up to the nearest 100ms. Price depends on the amount of memory you allocate to your function. You are charged $0.00001667 for every GB-second used.
  • Its BIG benefits!! NO SERVERS, continuous scaling, Super super cheap.

DNS Basics

  • Pretty much all of AWS is IPV6 compatible now
  • Domain registrars are authorities that can assign domain names directly under one or more top-level domains. Each domain name comes registered in the central ‘WhoIS’ database. AWS is now a Domain registrar.
  • SOA (Start Of Authority) record stores administrator of the zone and other info.
  • NS (Name Server) records are used to direct traffic to the Content DNS server.
  • TTL (Time To Live) in seconds, the length of time the DNS record is cached on a resolving server or users local PC before talking back to the DSN server.
  • A (Address) records translate the name into a IP address. AAAA record is the IPv6 equivalent
  • CNAME (Canonical Name) used to resolve one domain name to another. You get charged per request using Route53.
  • Alias records are similar to CNAME’s, key difference is a CNAME can’t be used for a naked domain name (zone apex) e.g. http://google.com Alias’s can. Also, if using Route53, Alias’s are not charged per request.
  • Remember ELB’s do not have a IPv4 address, you resolve to them using a DNS name e.g. lb1-1234.us-east-1.elb.amazonaws.com, Alias’s are perfect for pointing to a ELB’s DNS name.
  • On AWS, given the choice, always choose an Alias Record over a CNAME.

Route53

  • Route 53 is a global service just like IAM.
  • Route53 got its name as its AWS’s DNS, and DNS runs on port 53
  • There is a limit of 50 domain names available by default, however it is a soft limit and can be raised by contacting AWS support.
  • 2 options, ‘Register domain’ or ‘Transfer domain’.
  • Route 53 Routing Polices are Simple, Weighted, Latency, Failover, Geolocation
  • Simple Routing Policy:
    • The default policy, most commonly used when you have a single resource
    • Create a Record Set, choose Alias Yes or No and select the Alias Target:
  • Weighted Routing Policy:
    • Let’s you split traffic to different regions or ELB’s, they can be in the same region.
    • Good use case is to test out a new web site and send a small percentage of traffic to it.
    • Weight value can be 0 – 255 and AWS calculates the % split
    • E.g. set a weight of 80 to one Alias Target, and a weight of 20 to the other Alias Target.
  • Latency Routing Policy:
    • Allows you to route your traffic based on the lowest network latency for your end user

  • Failover Routing Policy:
    • Used to create an active/passive setup e.g. EU-WEST-2 is primary and AP-SouthEAST-2 is the secondary DR site.
    • Route53 will monitor the health of the primary site using a health check.
    • In Route53 create a ‘Health check’ and point it to the Primary sites ELB’s DNS name checking HTTP port 80.
    • Create a Route 53 Record Set as ‘Failover’ type and point it to the primary sites ELB and select ‘Primary’ site
    • Create another Route 53 Record Set as ‘Failover’ type and point it to the secondary sites ELB and select ‘Secondary’ site
  • Geolocation Routing Policy
    • Lest you route your traffic to servers in a location for example all customers coming from the US will route to US servers with US specific products and prices, and Europe customers will go to Europe server with euro pricing.
    • You can select location by ‘Continent’ or specific ‘countries’ and for the US for specific ‘states’.
    • You can select ‘default’ as the Routing Location for Everywhere else.

Databases

  • RDS – Relational Database System suitable for OLTP (OnLine Transaction Processing) structured data
    • MS SQL, MySQL, PostgreSQL, Oracle, Aurora, MariaDB
  • What are Non-Relational Databases like JSON/NoSQL
    • Collection = Table
    • Document = Row
    • Key Value Pairs =Fields
  • Dynamo DB – fast and flexible NoSQL database
  • Redshift – AWS’s Data warehousing database
    • For OLAP (OnLine Analytics Processing) type
    • Data Warehousing is used for business intelligence. Used to pull in very large and complex data sets. Usually used by management to do queries on data.
  • Elasticache
    • In-memory cache database on the cloud, not on slower disk.
    • Supports two open-source in-memory caching engines, Memcached and Redis
  • DMS – Database Migration Service
    • Handles migration processes like, datatype transformation, compression, and parallel transfer (for faster data transfer)
    • AWS schema conversion tool automatically converts the source database schema including custom code into a different format target database

RDS – Relational Database System

  • AWS RDS supports SQL Server, Oracle, MySQL Server, PostgreSQL, Aurora, MariaDB
  • By default, the inbound rule is just open for the public IP address of the RDS database. Remember to open up security group to your servers trying to communicate to your RDS
  • Connect to the RDS database by its endpoint (hostname), e.g. databasename.eu-west-2.rds.amazonaws.com:3306 < port 3306 is for MySQL
  • To connect you need to supply a username, password, hostname (endpoint) and dbname
  • Automated Backups –
    • Enabled by default.
    • Allows you to recover your database to any point within a ‘retention period’. Retention period can be 1 to 35 days. Replaying transaction logs AWS will recover your database down to the second.
    • The backup data is stored in S3 and you get free storage space equal to the size of the database.
    • Backups are taken within a defined window. During backup window, storage I/O may be suspended/slight latency while your date is being backed up.
  • Snapshots –
    • Are done manually. They are stored even after you delete the original RDS instance, unlike automated backups.
  • Whenever you restore an Automated backup or Snapshot, the restored version will be a new RDS instance with a new end point.
  • RDS Encryption – Is done by the AWS Key Management Service (KMS). All the backups and snapshots get encrypted also. You can’t convert an RDS DB to be encrypted, you have to make a new one and migrate the data over.
  • Supports Multi-AZ for DR – the RDS database is Synchronously replicated to another AZ in the same region. If the failover occurs, the same DNS endpoint is carried over.
  • Read Replica to boost performance– uses asynchronous replication to create an exact ‘read-only’ copy of the RDS DB to boost read performance. You can have up to 5 read replicas of your main DB. Must have automated backups turned on to deploy read replicas. Only supported for MySQL Server, PostgreSQL and MariaDB. Each Read Replica has its own DNS end point. You cannot have Read Replicas that have Multi AZ. You can create Read Replicas of Multi-AZ source databases however. Read Replicas can be promoted to become their own DB, this breaks the replication.
  • DynamoDB vs RDS – DynamoDB offers “push button” scaling, meaning that you can scale your DB on the fly with no down time. RDS is not so easy, you have to use a bigger instance size or add a read replica.
  • Read the RDS FAQ

DynamoDB

  • AWS DynamoDB fast and flexible NoSQL database that gives single-digit millisecond latency at any scale.
  • Supports both document and key-value data models
  • Great fit for mobile, web, gaming, ad-tech, IoT applications.
  • Stored on SSD storage
  • Spread across 3 different geographically district data centres
    • Eventual consistent Reads (Default) – consistency across all copies of data is reached within 1 second. Best Read Performance.
    • Strongly consistent Reads – Returns a result that reflects all writes that received a successful response prior to the read.
  • Pricing based on:
    • Provisioned throughput capacity
      • Write throughput $0.0065 per hour for every 10 units
      • Read Throughput $0.0065 per hour for every 50 units
    • Storage costs $0.25Gb per month
    • In general, DynamoDB can be expensive for writes but is extremely cheap for reads
  • DynamoDB offers “push button” scaling, meaning that you can scale your DB on the fly with no down time.
  • Read the Dynamo RDS FAQ

Redshift

  • Is a fast and powerful petabyte-scale data warehouse service. Really fast as its uses Columnar Data Storage
  • You can start small for just $0.25 per hour and scale up to a petabyte or more for $1,000 per terabyte per year, less than a tenth of most other data warehousing solutions.
  • Single Node (up to 160Gb of data)
  • Multi-Node
    • Leader Node (manages client connections and receives queries)
    • Compute Node (store data and perform queries and computations on that data). Up to 128 Compute Nodes
  • Columnar Data Storage – Redshift stores data in Columns instead of as a series of rows. Column-based is ideal for data warehousing and analytics. Column-based system require far fewer I/O’s, greatly improving performance.
  • Advanced Compression – Columnar data can be compressed much more than row-based data, also doesn’t require indexes to save space also.
  • Massively Parallel Processing (MPP) – Redshift automatically distributes data and query load across all nodes.
  • Pricing based on:
    • Compute Node Hours, you are billed for 1 unit per node per hour, so a 3-node data warehouse cluster running for an entire month would incur 2,160 instance hours. You’re not charged for leader node hours.
    • Backup,
    • Data transfer (only within a VPC, not outside it)
  • Redshift security:
    • Encrypted in transit using SSL
    • Encrypted at rest by using AES-256 encryption, by default Redshift takes care of key management
  • Redshift Availability,
    • Currently only available in 1 AZ
    • Can restore snapshots to new AZ’s in the event of an outage

Elasticache

  • Is an in-memory cache in the cloud. Uses in-memory caches instead of relying on slower disk-based databases.
    • Memcached – Widely adopted. Doesn’t support multi AZ.
    • Redis – Is newer than Memcached and has more features. Elasticache Redis supports Master/State replication and multi AZ to achieve redundancy.
  • Exam tip if you have a database under a lot of stress/load you could use Elasticache to alleviate this.
  • Elasticache is a good choice if your database is read heavy and not prone to frequent changing.
  • Redshift is a good answer if the reason your database is feeling stress is because of management keep running OLAP (OnLine Analytics Processing) transactions on it.

Aurora (MYSQL compatible RDS)

  • Amazon Aurora was developed by Amazon and is a MYSQL compatible RDS. AWS push Aurora quite heavily as the best RDS choice.
  • Provides up to 5 times performance of MySQL at one tenth the price of a commercial database.
  • Aurora scaling:
    • Starts with 10Gb and scales in 10Gb increments up to 64Tb (storage auto scaling)
    • Compute resource can scale up to 32vCPU’s and 244GB of memory
    • 2 copies of your data are contained in each AZ, with minimum of 3 availability zones. = 6 copies of your data
    • Its designed to handle a loss of up to 2 copies of data without affecting database write availability and up to 3 copies without affecting read availability.
    • 2 Replicas types to provide multi AZ redundancy
      • Aurora Replicas – up to 15 replicas (tiers 0-15, 0 is the master/writer, others are readers) (features auto failover)
      • MySQL Read Replicas – up to 5 replicas
    • You can create snapshots and restore to a point in time

VPC Overview

  • Amazon VPC (Virtual Private Cloud)
  • VPC’s can’t span regions but can span AZ’s
  • VPC’s consist of Subnets, IGW’s (or virtual private gateways for VPN’s), Route Tables, Security Groups and ACL’s,
  • Very important exam tip, 1 subnet = 1 availability zone!!
  • You can assign custom IP address ranges in each subnet
  • You can only have one internet gateway per VPC
  • Remember security groups are stateful, so if you create a rule to allow HTTP in, it will automatically allow HTTP out.
  • Remember ACL’s are stateless, so you must create the rule both ways.
  • Default VPC’s:
    • Are always created, is user friendly ready for you to start deploying instances.
    • All subnets have a route out to the internet
    • Each EC2 instance has both a public and private IP address
    • If you delete the default VPC, the only way to get it back is to raise a support ticket with AWS.
  • VPC Peering:
    • VPC Peering is simply a connection between two VPC’s that allows you to route traffic between them using private IP addresses.
    • You cannot create a VPC peering connection between VPC’s that have matching or overlapping CIDR blocks.
    • You can VPC peer over multiple accounts owned by you or others, but not between multiple regions
    • There is no single point of failure or bandwidth bottlenecks
    • Peering is in a star configuration, 1 central VPC peers with others. NO TRANSITIVE PEERING. Simply means you can’t have one VPC talk via another VPC.

Build a custom VPC

  • If you don’t want to use the Default VPC, you can create your own VPC.
  • Create a new VPC:
    • Give it a name
    • Specify the CIDR (Classless Inter Domain Routing) block e.g. 10.0.0.0/16. In AWS, you cannot have a CIDR block larger than /16, for example a /8 will produce an error.
    • The Tenancy can be ‘Default’ which is shared hardware, or ‘Dedicated’ hardware
    • When you create a VPC, it creates a default Route table, VPC security group and ACL.
  • Create a Subnet:
    • Give it a name
    • Select the VPC it will go in
    • Select the AZ
    • Type the CIDR block e.g. 10.0.1.0/24
  • To create a public subnet so it will allow access out to the internet:
    • Create IGW (Internet Gateway) and attach it to a VPC

Note: You cannot attach more than 1 IGW to a VPC

    • Create a new Route table
      • Name it for example ‘MyPublicRoute’
      • Edit it and the Destination will be 0.0.0.0/0 and the Target the name of the IGW e.g. igw-b9daa0d0
      • Associate the subnet you want to make public to the route table
    • Set the subnet to ‘Enable auto-assign Public IP’
  • At this point we have created this:
  • Now deploy another EC2 instance in the Private subnet
  • Edit the Security Group for the EC2 instance in the Private subnet and inbound allow:

    this allows the public EC2 instance to ping the private EC2 instance

NAT Instances & NAT Gateways

  • This is where we have built up to so far:
  • We need a NAT instance or gateway to allow instances in the Private subnet to get out to the internet to get software and updates.
  • NAT Instances have been around for a very long time, NAT gateways were released in 2016 and make it a lot easier, no patching needed, no SPOF, so are preferred.
  • NAT instance:
    • Easiest way, is to use the ‘AWS NAT instance’ from the AWS Marketplace
    • Deploy it into your Public subnet which has auto assign a Public IP turned on
    • It sits behind a security group like all other instances do
    • On the AWS console for the NAT instance you must disable ‘Source/Destination Check’, you must do this as a NAT instance acts to route traffic on behalf of other servers.
    • Edit the route table of the VPC so that 0.0.0.0/0 goes to the NAT instance as the target:
    • This is what we’ve built with a NAT instance. Problem with NAT instance is it’s a single point of failure unless you start using ASG’s and multiple subnets in different AZ’s e.t.c.
  • NAT Gateway:
    • Preferred as is easier to deploy, is highly available handled by AWS, AWS patches it and scales up to 10Gbps
    • Under VPC Dashboard select the ‘create a NAT Gateway’ option
      • Always select the Public subnet
      • Create/allocate an EIP (Elastic IP) which is an internet IP
    • Edit the route table of the VPC so that 0.0.0.0/0 goes to the NAT Gateway as the target:
    • This is what we’ve built with a NAT gateway:

Network ACL’s vs Security Groups

  • Assign Security Groups to instances, assign Network ACL’s to subnets
  • Remember! One subnet can only be associated with one Network ACL! But a Network ACL can be associated with multiple Subnets.
  • Network ACL’s give more granular control over allowing and blocking IP’s, but are more complicated than Security groups.
  • Each subnet in your VPC must be associated with a network ACL. If you don’t associate a subnet with a network ACL, the subnet is automatically associated with the Default network ACL.
  • The default Network ACL that gets created, allows all inbound and outbound traffic:
  • When you create a new custom Network ACL, by default denies all inbound and outbound traffic, as the starting point:
  • Associate a subnet to the Network ACL
  • Typical ports to open for a Public subnet. The 1024-65535 are ephemeral ports used by clients.
  • Rules are evaluated in order lowest first, going up. So, if you have a rule 50 which Denys access and a rule 100 that Allows access, the rule 100 takes effect.

ELB’s and VPC’s

  • Design consideration – when using an ELB If you want higher availability use 2 different subnets in one AZ, or even higher availability 2 AZ’s (2 subnets in each).

NAT vs Bastion

  • Bastion or Jump boxes sit in the Public subnet and are used for coming in for administration only, SSH and RDP most commonly. You can make redundancy by deploying multiple Bastion/jump boxes in different subnets, you can use ASG’s too.
  • A NAT is used to provide internet traffic to EC2 instances in private subnets, usually for software downloads/software updates.

Direct Connect

  • Direct Connect is a dedicated link from your co-location datacentre facility to gain advantages of:
    • Increased reliably over using a VPN
    • Increased bandwidth – 1Gps or 10Gps
    • Lower latency
    • Reduced costs when using large volumes of traffic traversing back and forth from AWS

SQS (Simple Queuing Service)

  • Is a web service that gives you access to a message queue which can act as a buffer for the messages to be stored in temporally, until an instance is ready to process them.
  • This ‘de-couples’ your infrastructure
  • Messages can contain up to 256 KB of text in any format. Billing is per 64KB “Chunks”
  • Any component of an application can retrieve messages programmatically using the AWS SQS API
  • The beauty of using SQS is:
    • if the producer is producing work faster than the consumer can process it
    • if the producer or consumer are only intermittently connected to the network
  • Two types of SQS are now offered, standard (guaranteed once-delivery) and FIFO (First In First Out)
  • SQS Visibility Timeout default is 30 seconds with a maximum of 12 hours.
  • The visibility timeout period only starts when the application server has picked up the message from SQS
  • The instances always pull the messages from the SQS queue, SQS never pushes the message out
  • SQS working in conjunction with auto scaling is how a lot of websites work and handle big traffic
  • SQS is designed to provide “at least once” delivery of all messages in the queue. Although your application should be designed so that if it receives the same message more than once, it won’t be a problem.
  • Pricing
    • First 1 million SQS requests per month are free
    • After that $0.50 per 1 million SQS request per month
    • A single request can have from 1 to 10 messages, up to a maximum total payload of the 256KB
  • Read the SQS FAQ

SWF (Simple WorkFlow service)

  • Is a web service that allows you to coordinate tasks across a variety of components (workers),
  • Task Retention period of up to 1 year
  • SWF Actors:
    • Workflow Starters – An application can initiate (start) a workflow. Could be your e-commerce website when placing an order or a mobile app searching for bus times.
    • Deciders – control the flow of activity tasks in a workflow. If something has finished in a workflow (or fails) a Decider decides what to do next.
    • Activity workers – Carry out the activity tasks, could be EC2 instances or even manual human tasks.

SNS (Simple Notification Service)

  • Is a web service that lets you send notifications from the cloud
  • Let’s you easily deliver to subscribers or other applications
  • Is used a lot in CloudWatch and auto-scaling
  • Can deliver to SMS text message, email, SQS, any HTTP/HTTPS endpoint, or trigger Lambda functions.
  • You create ‘SNS Topics’ and add multiple endpoints like iOS, android and SNS recipients. SNS is smart enough to deliver messages in the right format.
  • ‘Publish’ to a SNS ‘Topic’ to send your messages/emails
  • Key difference between SNS and SQS – SNS is a push-based delivery, SQS the applications polls the queue and pulls the messages
  • Pricing:
    • Users pay $0.50 per 1 million Amazon SNS requests
    • $0.06 per 100,000 notification deliveries over HTTP
    • $0.75 per 100 notification deliveries over SMS
    • $2.00 per 100,000 notification deliveries over Email

Elastic Transcoder

  • A media transcoder in the cloud.
  • Provides transcoding presents for popular outputs formats so you don’t have to research and find out all the different formats to use
  • Pricing based on minutes you transcode and the resolution at which you transcode.

API Gateway

  • Allows you to create a “front door” for applications to access data or functionality from back-end services such as EC2 Instances or code running on AWS Lambda or any web application.
  • API caching – reduces the amount of calls on your endpoint (Lambda, EC2) and also reduces latency of your requests.
  • What’s its features:
    • Low cost & efficient
    • Scales automatically
    • You can throttle requests to prevent attacks
    • Connect to CloudWatch to log all requests
  • If you are using Javascript/AJAX that uses multiple domains with API Gateway, ensure that you have enabled CORS on the API Gateway.

Kineses

  • AWS Kineses is a platform to send your streaming big data to.
  • Streaming Data is data that is generated continuously by thousands of data sources, typically in small sizes (KB’s) like:
    • Stock Prices
    • Game data (as the gamer plays)
    • Social network data
    • Geospatial data (think uber.com)
    • IoT sensor data
  • Kineses Streams – Data is sent to Kinesis Streams and is stored in Shards by default for 24 hours, maximum 7 days. The Consumers take the data from the Shard and turn it into something useful and pass it on. If they are talking about Shards in the exam, then they are talking about Streams. The data capacity of your stream is a total of the number of shards you specify for that stream.
  • Kineses Firehose – it’s completely automated, scales automatically and uses Lambda instead of the EC2 Consumers. As soon as the data comes in it is analysed by Lambda or sent directly onto S3. There’s no data retention window. Can also send to S3 and then onto Redshift, and also to an Elasticsearch cluster
  • Kinesis Analytics – Allows you to run SQL queries on the data as it exists in the Streams or Firehose.
  • In summary:
  • Kinesis – For bringing the data into the cloud
  • Redshift – usually used for applying business intelligence to the data
  • Elastic Map Reduce (EMR) – usually used for Big Data Processing. Note EMT allows you to login as root.

Environment setup in course

OpsWorks

  • Orchestration Service that uses Chef
  • Chef consists of recipes to maintain a consistent state
  • Look for the term “chef” or “recipes” or “cook books” and think of OpsWorks

Tags and Resource Groups

  • Use the Tag editor in the console to find untagged resources and tag them
  • If you Tag Resources, then you can create Resource Groups grouping up all resources with certain tags. E.g. make a Resource Group for all Resources with the Tag key ‘Team’ key value ‘developers’
  • You can view the groups in the console as well as exporting all to csv files.

AWS Workspaces

  • Windows 7 or Windows 10 available
  • By default, you get local administrator access so you can install your own applications.
  • Workspaces are persistent.
  • All data on D:\ is baked up every 12 hours
  • You don’t need an AWS account to login to workspaces.

‘Overview of AWS’ Whitepaper

  • 6 Advantages of Cloud
    • Trade capital expense for variable expense
    • Benefit from massive economies of scale
    • Stop guessing about capacity
    • Increase speed and agility
    • Stop spending money running and maintaining data centres
    • Go global in minutes
  • AWS Regions and AZ’s (May 2017)
  • Physical Security
    • State of the art electronic surveillance and multi factor access control systems
    • Staffed 24 x 7 by security guards
    • Access is authorised on a “least privilege basis”

Overview of ‘AWS Security Processes’ Whitepaper

  • AWS offer a shared security model, AWS is responsible for securing the underlying infrastructure and you the customer are responsible for anything you put on the cloud.
  • All decommissioned magnetic storage devices are degaussed and physically destroyed in accordance with industry standard practices.
  • Logically the AWS Production network is segregated from the Amazon corporate network. Meaning the Amazon.com network is separate from the AWS network.
  • AWS network protection by default:
    • DDoS
    • Man in the middle attacks (MITM)
    • IP Spoofing – AWS will not allow an instance to send traffic with a source IP or MAC address other than its own.
    • Port Scanning – Unauthorised port scans by Amazon EC2 customers are not permitted. You may request permission in advance to conduct vulnerability scans (penetration testing) as required.
  • Trusted Advisor inspects your AWS environment and makes recommendations when opportunities may exist to save money, improve system performance, or close security gaps.
  • AWS does not have any access to your instances or the guest OS.
  • A Risk management overview is conducted by AWS bi-annually.

Overview of ‘Storage Options in the Cloud’ Whitepaper

  • AWS services that provide storage:

    AWS Import/Export you mail AWS USB hard disk of data to upload to S3.
    AWS are preferring customers to use AWS Snowball now and it’s cheaper.
    Pricing: per gateway per month, volume storage usage per GB per month, snapshot storage usage per GB per month, data transfer out of AWS to customer per GB per month

Overview of ‘Architecting for the Cloud Best Practices’ Whitepaper

  • Business Benefits of Cloud
    • Almost zero upfront infrastructure investment
    • Just-in-time Infrastructure
    • More efficient resource utilisation
    • Usage-based costing
    • Reduced time to market
  • Technical benefits of cloud
    • Automation – “Scriptable infrastructure”
    • Proactive Scaling – Auto-scaling
    • More efficient development lifecycle
    • Improved Testability
    • Disaster Recovery and Business Continuity automatic failover between AZ’s and regions
  • Understanding Elasticity benefits:
  • Design for Failure! When designing, you should always design expecting failures to occur and extra load could occur. Design with auto-scaling and use multiple AZ’s and even multiple regions using Route 53.
  • Design for good security using Security groups, only opening needed ports

The well-architected framework – Pillar One, Security

  • Apply security at all layers
  • Enable traceability
  • Automate responses to security events
  • Focus on securing your system
  • Automate security best practices
  • Security in the cloud consists of 4 areas:
    • Data protection
      • Protect data in transit (SSL) and at rest
    • Privilege management
      • Use MFA
      • Use Roles over storing credentials on an instance
    • Infrastructure protection
      • For the AWS console use IAM
    • Detective controls
      • CloudTrail
      • CloudWatch
      • AWS Config

The well-architected framework – Pillar Two, Reliability

  • Test recovery procedures
  • Automatically recover from failure (auto scaling)
  • Scale horizontally to increase performance and availability (auto scaling)
  • Reliability in the cloud consists of 3 areas:
    • Foundations:
      • Plan your AWS VPC/subnet topology up front
      • Provision the right connection to the internet or direct connect
      • Know and manage your AWS service limits
      • Select the right AWS support plan, Basic, Developer, Business & Enterprise
    • Change Management
      • Use AWS CloudTrail
      • Use services like Auto Scaling to automate change when demand changes
    • Failure Management
      • Always architect with the assumption failures will occur
      • Enable backups, use snapshots

The well-architected framework – Pillar Three, Performance Efficiency

  • Compute
    • Choose the right EC2 Instance type based on CPU, RAM e.t.c
    • Use Auto Scaling
    • Switch to Serverless using Lambda if you can
  • Storage
    • Access Method – Block, File or Object
    • Patterns of Access – Random or Sequential
    • Throughput required
    • Frequency of Reading – Online, Offline or Archival
    • Frequency of Update – Worm (Write Once Read Many), Dynamic
  • Database
    • SQL or no SQL or data warehousing
    • With RDS add read replicas to increase performance
  • Network
    • Use Direct Connect to provide greater and predicable bandwidth and reduced latency to AWS
  • Space time trade off – Use more space but place more copies of data closer to your users/customers
    • Use global infrastructure to place environments close to your users/customers
    • Use cashing services such as ElastiCache or CloudFront to reduce latency

The well-architected framework – Pillar Four, Cost Optimization

  • Cost optimisation allows you to pay the cheapest AWS hosting costs possible while not sacrificing your business objectives.
  • Allocate out costs to individual departments/cost centres and encourage them to lower those costs
  • Use managed services like SaaS to reduce costs
  • Trade capital expense for operating expense
  • Matched supply and demand
    • Use Auto scaling to scale with demand
    • Use services like Lambda that only execute (or respond) when a request (demand) comes in.
    • CloudWatch can help you keep watch on what your demand is.
  • Cost-effective resources
    • Use the correct instance type, e.g. on a T2-micro a reporting process might take 7 hours to complete, on a m4.2xlarge is might take 5 minutes and would cost overall less.
    • A well architected system will use the most cost efficient resources to reach the end business goal.
    • Use Reserved Instances if its makes cost sense.
  • Expenditure Awareness
    • With cloud agility comes problems of everyone spending and you’re not aware. Use Consolidated billing (now called AWS Organisations), Cost allocation tags, billing alerts.
  • Optimising over time
    • A service you chose last week might not be the best service to be using today e.g. MySQL RDS was best in 2012, but Aurora launched in 2014 might be better and cheaper.
    • Subscribe to the AWS blog so you know of what new updates there are.
    • Use AWS Trusted Advisor to give you recommendations over time.

Exam Practicalities

  • 80 minutes in length
  • 60 questions in the exam
  • For practice exams aim for 70% right
  • Priced $150 USD no matter what country you’re in
  • Conducted online at an approved centre

My personal site about life, friends & family, living in Australia and a fair bit on cars too :)