Backup files to Amazon S3


After a few years of silence today it came to my mind that I should once again start to write on my blog. During the last 12 odd months I have been involved in developing and revamping several of the websites own by the company I currently work.

In this article I’m going to discuss about the steps I have been using to automate the backing up of the websites that were hosted with Amazon EC2 to Amazon S3 bucket.

The strategy I adapted can be broken down into following 3 steps:

  1. Create a backup copy of each of the database and the website source code (all my websites were developed using PHP) on a daily basis and compress each (database & source code) using tar.gz compression appending the timestamp
  2. Pushing of backup files to Amazon S3 bucket
  3. Set a conjob task to execute to process

Step 1: Create a copy of each of the database and the website source code

To achieve this I created a folder called backups (/home/ubuntu/backups) in the home directory and added the necessary instructions into the shell script as follows.

#!/bin/sh

# (1) set up the required variables
DB_DUMP=<filename>_`date +"_%Y_%m_%d"`.sql
SOURCE_CODE=<filename>_`date +"_%Y_%m_%d"`.tar.gz
DBSERVER=<hostname>
DATABASE=<database name>
USER=<database user>
PASS=<database password>

# (2) use the following command  to create a dump of the database
cd /home/ubuntu/backups/
mysqldump --opt --user=${USER} --password=${PASS} -h ${DBSERVER} ${DATABASE} > ${DB_DUMP}

# (3) compress the mysql database dump using tar.gz compression
tar -zcf ${DB_DUMP}.tar.gz ${DB_DUMP}

# (4) create a copy of the website source, compress it and moved to /home/ubuntu/backups/
cd /var/www/
tar -zcf ${SOURCE_CODE}  <website source code folder>/
mv ${SOURCE_CODE} /home/ubuntu/backups/

# (5) delete the older copies of backups which are more than 3 days old inside /home/ubuntu/backups/
cd /home/ubuntu/backups/
find <filename>_* -mtime +3 -exec rm {} \;

Save the file as backup.sh inside /home/ubuntu/backups

Step 2: Pushing of backup files to Amazon S3 bucket

To achieve this I adapted two approaches and you’ll find that the latter approach is easier. Initially I adapted an approach of using the Amazon AWS’s SDK to move the backup files to Amazon S3 bucket. This approach had an limitation when individual file size (After the initial compression the backup was over 12 GB) exceeded more than 4GB while on a 64 bit architecture Linux box (I used Ubuntu 16.04) since I used PHP. To overcome this I sliced the final output of the compressed file in to multiples of  3.6 GB.

tar czf - / | split -b 3850 MB - ${SOURCE_CODE}.tar.gz.

Approach 1: Using Amazon AWS SDK

Download the appropriate Amazon AWS SDK from here.  In my case I used the PHP SDK using the instructions available here and downloaded the PHP library using the 3rd steps (Installing via Zip file).

<?php
require_once('/home/ubuntu/aws/aws-autoloader.php');
use Aws\S3\S3Client;
use Aws\S3\Exception\S3Exception;

$bucket = '<bucket name>';
$pathToFile = '/home/ubuntu/backups/';
$fileNameSourceCode = ['<filename>_'.date('Y_m_d').'.tar.gz']; // name of the website source code, it should be equal to name of SOURCE_CODE variable found on /home/ubuntu/backups/backup.sh
$fileNameDBDump = '<filename>_'.date('Y_m_d').'.sql.tar.gz';// name of the database dump file, it should be equal to the name of DB_DUMP variable found on /home/ubuntu/backups/backup.sh

$credentials = new Aws\Credentials\Credentials(”, ”);

// Instantiate the client.
$s3 = S3Client::factory([
‘region’ => ‘us-east-1’,  // Since I have create the buckets in US East region (N. Virginia)
‘version’ => ‘2006-03-01’, // Standard version number for the S3 bucket service
‘credentials’ => $credentials
]);

//Pushing the source code file to the Amazon S3 bucket

if(count($fileNameSourceCode) > 0) {
foreach($fileNameSourceCode as $file) {
if(file_exists($pathToFile.$file)) {
try {
// Upload data.
$result = $s3->putObject(array(
‘Bucket’ => $bucket,
‘Key’ => $file,
‘SourceFile’ => $pathToFile.$file,
‘ACL’ => ‘public-read’,
‘Expires’ => gmdate(“D, d M Y H:i:s T”, strtotime(“+15 days”)) //This parameter doesn’t get applied, this we have to set on the bucket from the Amazon S3 account
));

// Print the URL to the object.
echo $result[‘ObjectURL’] . “\n”;
} catch (S3Exception $e) {
echo $e->getMessage() . “\n”;
}
}
}
}

//Pushing the database dump file to the Amazon S3 bucket

if(file_exists($pathToFile.$fileNameDBDump)) {
try {
// Upload data.
$result = $s3->putObject(array(
‘Bucket’ => $bucket,
‘Key’ => $fileNameDBDump,
‘SourceFile’ => $pathToFile.$fileNameDBDump,
‘ACL’ => ‘public-read’,
‘Expires’ => gmdate(“D, d M Y H:i:s T”, strtotime(“+15 days”)) ////This parameter doesn’t get applied, this we have to set on the bucket from the Amazon S3 account
));

// Print the URL to the object.
echo $result[‘ObjectURL’] . “\n”;
} catch (S3Exception $e) {
echo $e->getMessage() . “\n”;
}
}
Save the file as upload_to_s3bucket.php inside /home/ubuntu/backups

Approach 2: Using Amazon S3Tools

The Amazon S3 Tools is a very easy to use command line utility which can be used to push very huge files to Amazon S3 bucket with minimum effort. For Linux & Mac we can use s3cmd while for Windows use S3Express. I found this article on TecAdmin which has comprehensively explained it usage. I followed the following steps to set it up on my server.

  • Setting up of S3tool on the server

Installation

$ sudo apt-get install s3cmd

Configuration

You need to provide the Access Key ID and Secrete Key available with your Amazon AWS account during the configuration by executing the following command. As a best practice it recommends to create an IAM user and provide that creadentials instead of using the root account details.

# s3cmd --configure

  • Setting up the shell script to push the files to S3 Bucket

To achieve this I created a folder called backups (/home/ubuntu/backups) in the home directory and added the necessary instructions into the shell script as follows.


#!/bin/bash

_DB_DUMP=<filename>_`date +"_%Y_%m_%d"`.sql  # name of the website source code, it should be equal to the name of DB_DUMP variable found on /home/ubuntu/backups/backup.sh
_SOURCE_CODE=<filename>_`date +"_%Y_%m_%d"`.tar.gz  # name of the website source code, it should be equal to name of SOURCE_CODE variable found on /home/ubuntu/backups/backup.sh

s3cmd put ${_DB_DUMP} s3://<bucket name>/
s3cmd put ${_SOURCE_CODE} s3://<bucket name>/

Save the file as upload_to_s3bucket.sh inside /home/ubuntu/backups

Step 3: Set a conjob task to execute to process

Now lets set the cronjob task to daily or any required time interval to execute the following two scripts.

Firstly lets make the two shell scripts executable using following command

$ chmod +x /home/ubuntu/backups/backup.sh
$ chmod +x /home/ubuntu/backups/upload_to_s3bucket.sh

Open up the terminal and execute the following command
sudo crontab -e

Enter the following two lines and save.

30 01 * * * /home/ubuntu/backups/backup.sh #set to run the backup 30 minutes passing 1 o'clock in the morning

#use this if used the Amazon AWS SDK approach
00 03 * * * php /home/ubuntu/backups/upload_to_s3bucket.php #set to run the backup daily 3 o'clock in the morning

#use this if used the Amazon S3tools approach

00 03 * * * /home/ubuntu/backups/upload_to_s3bucket.sh #set to run the backup daily 3 o'clock in the morning

Advertisements

Connect To Amazon EC2 via Putty


In this post I’m going to show how to establish a SSH connection to an Amazon EC2 instance using Putty using a Windows box. First of all we need to download following tool. It can be find under the section called For Windows on Intel x86.

Step 1: Generating the Private Key using Puttygen

In Amazon AWS environment each instance (EC2, RDS, ElasticCache, etc…) is attached to a permission/security group, access to each service is provided through Private Key/Public Key authentication. Each Amazon EC2 instance will holds the Public Key of the permission/security group it belongs. To connecting to a specific Amazon EC2 instance need to use the corresponding Private Key (xxx.pem). Puttygen can be used to generate the local Private key out from the Private Key obtained from Amazon AWS required for establishing the connecting to Amazon EC2 instance via Putty.

Open Puttygen as shown in Figure 1 and click Load to select the Private Key obtained form the Amazon AWS.

 Generate private key

Figure 1: Selecting the Amazon AWS Private Key

Next click Save private key, next click OK on the appearing dialog box as shown in Figure 2 and 3

Saving generate private key

Figure 2: Saving the generated Private Key

Provide a suitable name (eg: aws_putty_private_key.ppk) for the newly created Private key, make sure to store all these keys in a well secure place.

Saving generate private key

Figure 3: Saving the generated Private Key

Step 2: Pointing the generated Private key to Putty

Now we have finish creating the private key required by Putty. Open Putty and navigates to Connection -> SSH -> Auth from the left pane of the Putty window as shown in Figure 4. Select the newly created private key (aws_putty_private_key.ppk) from the Options controlling SSH authentication pane as shown in Figure 5

Options controlling SSH authentication

Figure 4: Options controlling SSH authentication

Selecting the generated private key

Figure 5: Selecting the generated Private key

Step 3: Providing Amazon AWS EC2 instance information

Navigates to Session from the left pane of the Putty window as shown in Figure 6 and Provide the Host Name or the IP of the Amazon EC2 instance Connection Type as SSH.

Providing Amazon AWS EC2 instance information

Figure 6: Providing Amazon AWS EC2 instance information

Step 4: Connecting to Amazon AWS EC2 instance

Provides the User name of the EC2 instance.

Connecting to Amazon AWS EC2 instance

Upon successful authentication connection is established. Happy hacking.

Connecting to Amazon AWS EC2 instance

Monitoring and Managing Amazon RDS Databases using MySQL Workbench


Last couple of week I was looking for an easier approch to manage the Databases created on Amazon RDS instances. I had to run through a tedious set of steps in carrying out routine stuff like introducing new updates, taking daily backups and moving it to an Amazon S3 bucket, etc… Before getting into touch with MySQL Workbench had to run through several intermediate hops (Connect to one of the EC2 instance via SSH and connect to the RDS instance from there, ;-( ) even to run a small query on the tables.

Let me eleborate the step I followed in setting up the connect with the DB created on a Amazon RDS instance.

Step 1: Launch MySQL Workbench

Launch MySQL Workbench and select New Server Instance found under Server Administration section form the Welcome screen as shown in Figure 1.

MySQl Workbench Welcome Screen

Figure 1: MySQl Workbench Welcome Screen

Step 2: Specify Host Machine of the Database server

Next form the Create New Server Instance Profile wizard, provide the Public DNS of the Amazon EC2 instance under Remote Host as shown in Figure 2 (this EC2 instance will be as the intermediate point to access DB that resides inside the Amazon RDS instance). To find the Public DNS specific to the EC2 instance can be done as illustrated in Figure 3. Click Next button to procced.
Specify Host Machine

Figure 2: Specify Host Machine of the Database screen

Amazon EC2 Instance

Figure 3: Amazon EC2 admin console screen

Step 3: Creating the DB connection

Fill in the values as described below which illustrated in Figure 4.

  • Connection Name – Provide a name to identify your connection
  • Connection Method – Select Standard TCP/IP over SSH from the drop down list instead of Standard(TCP/IP). Then on the appearing two tabbed pane, fill the following values found under Parameters tab.
    • SSH Hostname – Provide the Public DNS of the Amazon EC2 instace (refer Figure 3) which will be used as the intermediate server instance used to create the connection with the DB instance.
    • SSH Username – Provide the user of the Amazon EC2 instance (refer Figure 3) which will be used as the intermediate server instance used to create the connection with the DB instance.
    • SSH Key File – Provide the Private Key (xxx.pem) used to connect Amazon EC2 instace via SSH
    • MySQL Hostname – Provide the Endpoint of the DB instance created in Amazon RDS(refer Figure 5).
    • Username – Provide the Master Username of the DB instance created in Amazon RDS(refer Figure 5).
    • Password – Click on Store in Keychain.. button and type the password provided while creating the DB instance in Amazon RDS.

Click Next button to procced.

Database Connection

Figure 4: Database Connection screen

Amazon RDS DB Instance

Figure 5: Amazon RDS DB Instance

Step 4: Testing the Database Connection

In this step it validates the parameters provided in the previous step and upon success it returns Database connection tested successfully(refer Figure 6). Upon this message the Next button get activate to procceds to Management and OS.

Test DB Connection

Figure 6: Testing the Database Connection

Step 5: Management and OS

Select SSH login based management option and select the appropriate parameters for Operating System, MySQL Installation Type (refer Figure 7).

Management and OS

Figure 7: Specify remote management type and target operating system

Step 6: Setting up remote SSH Configuration

Provide the following parameters

  • Host Name – Provide the Public DNS of the Amazon EC2 instace (refer Figure 3) which will be used as the intermediate server instance used to create the connection with the DB instance.
  • User Name – Provide the user of the Amazon EC2 instance (refer Figure 3) which will be used as the intermediate server instance used to create the connection with the DB instance.
  • Select Authenticate Using SSH Key and provide the SSH Private Key Path – Provide the Private Key (xxx.pem) used to connect Amazon EC2 instace via SSH

Remote SSH Configuration

Figure 8: Remote SSH Configuration

Step 7: Test Host Machine Settings

In this step it validates the parameters provided in the previous step and upon success it returns Testing host machine setting is done(refer Figure 9). Upon this message the Next button get activate and clicking Next buttin it popup Review Settings dialog box (refer Figure 10).

Test Host Machine Settings

Figure 9: Test Host Machine Settings

Review Settings

Figure 10: Review Settings

Step 8: Creating the Instance Profile

Provide a name for the Profile got created.

Creating the Instance Profile

Figure 11: Creating the Instance Profile

Step 9: Done

Success

Figure 12: Workbench Home screen with the new profile

Setting up Amazon AWS Environment


Introduction

Amazon Web Services (AWS) began offering IT infrastructure services to businesses in the form of web services — now commonly known as cloud computing. One of the key benefits of cloud computing is the opportunity to replace up-front capital infrastructure expenses with low variable costs that scale with your business. Amazon Web Services provides a highly reliable, scalable, low-cost infrastructure platform in the cloud environment.

Location

AWS Management Console

Access and manage Amazon’s growing suite of infrastructure web services through a simple and intuitive, web-based user interface. The AWS Management Console provides convenient management of your compute, storage, and other cloud resources. Figure 1 show the features fund in the AWS Management Console once a user login.

Figure 1: AWS Management Console: Home Page

Login to AWS Management Console

Please click on the either if the two locations shown in the Figure 2 provide the login details.

Figure 2: AWS Management Console: Login locations

In the Figure 3 please provide the following details

  • My e-mail address is: xxxxx
  • I am a returning user and my password is: xxxxx

Figure 3: Sign In or Create an AWS Account

AWS Management Console: Amazon EC2

Amazon EC2 section provide the interface to manage EC2 instances (installation and setting up the operating system environment). In the Figure 4 we can identify 2 main sections as follows.

  • Navigation
  • Amazon EC2 Console Dashboard
Navigation

This section the user will find functionality to manage Amazon EC2 instances. As the first step need to select the Region(datacenter location), the location where the application going to deployed. For example can select Asia Pacific(Singapore) as the region and continue with rest of the activities. Functionality is grouped as follows.

  • Dashboard
  • Scheduled Events
  • INSTANCES
  • IMAGES
    • AMIs
    • Bundle Tasks
  • ELASTIC BLOCK STORE
    • Volumes
    • Snapshots
  • NETWORK & SECURITY
    • Security Groups – Using Security Groups
    • Elastic IPs
    • Placement Groups
    • Load Balancers
    • Key Pairs
    • Network Interfaces
Amazon EC2 Console Dashboard

Based on the selection it will load the respective information falls under each section.

Figure 4: AWS Management Console: Amazon EC2

AWS Management Console: Amazon RDS

Amazon Relational Database Service (RDS) makes it easy to set up, operate, and scale a relational database in the cloud. You can click the button below to launch database instance in minutes with automated backups, turnkey Multi-AZ replication, free monitoring metrics. Amazon RDS gives you access to a familiar MySQL or Oracle database to facilitate compatibility with existing code, applications, and tools.

  • Navigation
  • Amazon RDS Console Dashboard

Navigation

This section the user will find functionality to manage Amazon RDS instances. As the first step need to select the Region(datacenter location), the location where the application going to deployed. For example can select Asia Pacific(Singapore) as the region and continue with rest of the activities. Functionality is grouped as follows.

  • Getting Started Guide – Getting Started Guide
  • RDS Dashboard
  • Databases
    • DB Instances
    • Reserved DB Instances
    • Orderable DB Options
    • DB Snapshots
    • DB Security Groups
    • DB Parameter Groups
    • DB Subnet Groups
    • DB Events

Amazon RDS Console Dashboard

Based on the selection it will load the respective information falls under each section.

Figure 5: AWS Management Console: Amazon RDS

Setting up the database

Please use the following information to login to Amazon RDS instance from our local machine.

$ mysql -u<username> -p<password> -h<xxxxxx.ap-southeast-1.rds.amazonaws.com>
-- user: xxxx
-- password: xxxxx
-- hostname: xxxxxx.ap-southeast-1.rds.amazonaws.com

Next step would be to create the database (eg: elephanti_master) and exit from the instance.


mysql> CREATE DATABASE ;
mysql>..
mysql>..
mysql>quit;

Next transfer the tables to the newly created database on the Amazon RDS instance.


$ mysql -u<username> -p<password> -h<xxxxxx.ap-southeast-1.rds.amazonaws.com> < </path/to/sql_file.sql>;

Configuring the your application

Navigate to the application’s configuration section, eg: config/database.php and update the respective parameters as follows


$db['default']['hostname'] = 'xxxxxxx.ap-southeast-1.rds.amazonaws.com';
$db['default']['username'] = 'xxxxx';
$db['default']['password'] = 'xxxx';
$db['default']['database'] = 'master_db';

AWS Management Console: Amazon ElasticCache

Amazon ElastiCache is a web service that makes it easy to set up, manage, and scale distributed in-memory cache environments in the cloud. It provides a high performance, resizable, and cost-effective in-memory cache, while removing the complexity associated with deploying and managing a distributed cache environment.

  • Navigation
  • My Cache Clusters
Navigation

This section the user will find functionality to manage Amazon ElastiCache instances. As the first step need to select the Region(datacenter location), the location where the application going to deployed. For example select Asia Pacific(Singapore) as the region and continue with rest of the activities. Functionality is grouped as follows.

My Cache Clusters

Based on the selection it will load the respective information falls under each section.

Figure 6: AWS Management Console: Amazon ElasticCache

Connecting to Elastic Cache Cluster

Navigate to the application’s configuration section, eg: config/config.php and update the respective parameters as follows


$config['MEMCACHE_IP'] = "xxxxxxx.cache.amazonaws.com";
$config['MEMCACHE_PORT'] = "11211";