Backup files to Amazon S3


After a few years of silence today it came to my mind that I should once again start to write on my blog. During the last 12 odd months I have been involved in developing and revamping several of the websites own by the company I currently work.

In this article I’m going to discuss about the steps I have been using to automate the backing up of the websites that were hosted with Amazon EC2 to Amazon S3 bucket.

The strategy I adapted can be broken down into following 3 steps:

  1. Create a backup copy of each of the database and the website source code (all my websites were developed using PHP) on a daily basis and compress each (database & source code) using tar.gz compression appending the timestamp
  2. Pushing of backup files to Amazon S3 bucket
  3. Set a conjob task to execute to process

Step 1: Create a copy of each of the database and the website source code

To achieve this I created a folder called backups (/home/ubuntu/backups) in the home directory and added the necessary instructions into the shell script as follows.

#!/bin/sh

# (1) set up the required variables
DB_DUMP=<filename>_`date +"_%Y_%m_%d"`.sql
SOURCE_CODE=<filename>_`date +"_%Y_%m_%d"`.tar.gz
DBSERVER=<hostname>
DATABASE=<database name>
USER=<database user>
PASS=<database password>

# (2) use the following command  to create a dump of the database
cd /home/ubuntu/backups/
mysqldump --opt --user=${USER} --password=${PASS} -h ${DBSERVER} ${DATABASE} > ${DB_DUMP}

# (3) compress the mysql database dump using tar.gz compression
tar -zcf ${DB_DUMP}.tar.gz ${DB_DUMP}

# (4) create a copy of the website source, compress it and moved to /home/ubuntu/backups/
cd /var/www/
tar -zcf ${SOURCE_CODE}  <website source code folder>/
mv ${SOURCE_CODE} /home/ubuntu/backups/

# (5) delete the older copies of backups which are more than 3 days old inside /home/ubuntu/backups/
cd /home/ubuntu/backups/
find <filename>_* -mtime +3 -exec rm {} \;

Save the file as backup.sh inside /home/ubuntu/backups

Step 2: Pushing of backup files to Amazon S3 bucket

To achieve this I adapted two approaches and you’ll find that the latter approach is easier. Initially I adapted an approach of using the Amazon AWS’s SDK to move the backup files to Amazon S3 bucket. This approach had an limitation when individual file size (After the initial compression the backup was over 12 GB) exceeded more than 4GB while on a 64 bit architecture Linux box (I used Ubuntu 16.04) since I used PHP. To overcome this I sliced the final output of the compressed file in to multiples of  3.6 GB.

tar czf - / | split -b 3850 MB - ${SOURCE_CODE}.tar.gz.

Approach 1: Using Amazon AWS SDK

Download the appropriate Amazon AWS SDK from here.  In my case I used the PHP SDK using the instructions available here and downloaded the PHP library using the 3rd steps (Installing via Zip file).

<?php
require_once('/home/ubuntu/aws/aws-autoloader.php');
use Aws\S3\S3Client;
use Aws\S3\Exception\S3Exception;

$bucket = '<bucket name>';
$pathToFile = '/home/ubuntu/backups/';
$fileNameSourceCode = ['<filename>_'.date('Y_m_d').'.tar.gz']; // name of the website source code, it should be equal to name of SOURCE_CODE variable found on /home/ubuntu/backups/backup.sh
$fileNameDBDump = '<filename>_'.date('Y_m_d').'.sql.tar.gz';// name of the database dump file, it should be equal to the name of DB_DUMP variable found on /home/ubuntu/backups/backup.sh

$credentials = new Aws\Credentials\Credentials(”, ”);

// Instantiate the client.
$s3 = S3Client::factory([
‘region’ => ‘us-east-1’,  // Since I have create the buckets in US East region (N. Virginia)
‘version’ => ‘2006-03-01’, // Standard version number for the S3 bucket service
‘credentials’ => $credentials
]);

//Pushing the source code file to the Amazon S3 bucket

if(count($fileNameSourceCode) > 0) {
foreach($fileNameSourceCode as $file) {
if(file_exists($pathToFile.$file)) {
try {
// Upload data.
$result = $s3->putObject(array(
‘Bucket’ => $bucket,
‘Key’ => $file,
‘SourceFile’ => $pathToFile.$file,
‘ACL’ => ‘public-read’,
‘Expires’ => gmdate(“D, d M Y H:i:s T”, strtotime(“+15 days”)) //This parameter doesn’t get applied, this we have to set on the bucket from the Amazon S3 account
));

// Print the URL to the object.
echo $result[‘ObjectURL’] . “\n”;
} catch (S3Exception $e) {
echo $e->getMessage() . “\n”;
}
}
}
}

//Pushing the database dump file to the Amazon S3 bucket

if(file_exists($pathToFile.$fileNameDBDump)) {
try {
// Upload data.
$result = $s3->putObject(array(
‘Bucket’ => $bucket,
‘Key’ => $fileNameDBDump,
‘SourceFile’ => $pathToFile.$fileNameDBDump,
‘ACL’ => ‘public-read’,
‘Expires’ => gmdate(“D, d M Y H:i:s T”, strtotime(“+15 days”)) ////This parameter doesn’t get applied, this we have to set on the bucket from the Amazon S3 account
));

// Print the URL to the object.
echo $result[‘ObjectURL’] . “\n”;
} catch (S3Exception $e) {
echo $e->getMessage() . “\n”;
}
}
Save the file as upload_to_s3bucket.php inside /home/ubuntu/backups

Approach 2: Using Amazon S3Tools

The Amazon S3 Tools is a very easy to use command line utility which can be used to push very huge files to Amazon S3 bucket with minimum effort. For Linux & Mac we can use s3cmd while for Windows use S3Express. I found this article on TecAdmin which has comprehensively explained it usage. I followed the following steps to set it up on my server.

  • Setting up of S3tool on the server

Installation

$ sudo apt-get install s3cmd

Configuration

You need to provide the Access Key ID and Secrete Key available with your Amazon AWS account during the configuration by executing the following command. As a best practice it recommends to create an IAM user and provide that creadentials instead of using the root account details.

# s3cmd --configure

  • Setting up the shell script to push the files to S3 Bucket

To achieve this I created a folder called backups (/home/ubuntu/backups) in the home directory and added the necessary instructions into the shell script as follows.


#!/bin/bash

_DB_DUMP=<filename>_`date +"_%Y_%m_%d"`.sql  # name of the website source code, it should be equal to the name of DB_DUMP variable found on /home/ubuntu/backups/backup.sh
_SOURCE_CODE=<filename>_`date +"_%Y_%m_%d"`.tar.gz  # name of the website source code, it should be equal to name of SOURCE_CODE variable found on /home/ubuntu/backups/backup.sh

s3cmd put ${_DB_DUMP} s3://<bucket name>/
s3cmd put ${_SOURCE_CODE} s3://<bucket name>/

Save the file as upload_to_s3bucket.sh inside /home/ubuntu/backups

Step 3: Set a conjob task to execute to process

Now lets set the cronjob task to daily or any required time interval to execute the following two scripts.

Firstly lets make the two shell scripts executable using following command

$ chmod +x /home/ubuntu/backups/backup.sh
$ chmod +x /home/ubuntu/backups/upload_to_s3bucket.sh

Open up the terminal and execute the following command
sudo crontab -e

Enter the following two lines and save.

30 01 * * * /home/ubuntu/backups/backup.sh #set to run the backup 30 minutes passing 1 o'clock in the morning

#use this if used the Amazon AWS SDK approach
00 03 * * * php /home/ubuntu/backups/upload_to_s3bucket.php #set to run the backup daily 3 o'clock in the morning

#use this if used the Amazon S3tools approach

00 03 * * * /home/ubuntu/backups/upload_to_s3bucket.sh #set to run the backup daily 3 o'clock in the morning

Advertisements

Currency Symbols


Following is a compilation of default currencies used by different countries in the world which I have collected by referring the links found under the References section. The information is organized as follows:

  • Country Name – Name of the country
  • Currency Name – Term used to denote the currency
  • Currency Codes – ISO 4217 Code used to denote Currency
  • Symbol – Default currency symbol that represents the currency
  • HexCode – Hexadecimal Code used to generate the currency symbol. Please note that in order to render it properly needs to append &#x in front of each comma separated character and ; at the end. eg: 41, 66, 73    &#x + 41;&#x + 66;&#x + 73; Afs 
Country Name Currency Name Currency Code Symbol HexCode
Afghanistan Afghani AFN Afs 41, 66, 73
Aland Islands Euro EUR 20ac
Albania Lek ALL Lek 4c, 65, 6b
Algeria Algerian Dinar DZD 2f, 62c, 6
American Samoa United States Dollar USD $ 24
Andorra Euro EUR 20ac
Andorra French franc FRF 20A3
Andorra Andorran/Spanish peseta ESP 20a7
Angola Angolan kwanza AOA Kz 4B, 7A
Anguilla East Caribbean dollar XCD EC$ 45, 43, 24
Antarctica Norwegian Krone NOK ɫɲ 26b, 272
Antigua and Barbuda East Caribbean dollar XCD EC$ 45, 43, 24
Argentina Argentine peso ARS $ 24
Argentina Austral ARA 20b3
Armenia Dram AMD ֏ 58F
Aruba Aruban florin AWG ƒ 192
Australia Australian dollar AUD $ 24
Austria Euro EUR 20ac
Austria Schilling ATS öS F6, 53
Azerbaijan Azerbaijani Manat AZN ман 43c, 430, 43d
Bahamas Bahamian dollar BSD $ 24
Bahrain Bahraini dinar BHD 2e, 2f, 62e, 28, 6
Bangladesh Bangladeshi Taka BDT 9f3
Barbados Barbadian dollar BBD $ 24
Belarus Belarusian ruble BYR p. 70, 2e
Belgium Euro EUR 20ac
Belize Belize dollar BZD BZ$ 42, 5a, 24
Benin West African CFA franc XOF CFA 43, 46, 41
Bermuda Bermudian dollar BMD $ 24
Bhutan Buthanese ngultrum BTN Nu. 4e, 75, 2e
Bolivia Boliviano BOB Bs 42, 73
Bosnia and Herzegovina Marka BAM KM 4b, 4d
Botswana Pula BWP P 50
Bouvet Island Norwegian Krone NOK ɫɲ 26b, 272
Brazil Brazilian real BRL R$ 52, 24
British Indian Ocean Territory Sterling pound GBP £ a3
Brunei Darussalam Brunei dollar BND $ 24
Brunei Darussalam Singapore dollar SGD $ 24
Bulgaria Bulgarian Lev BGN лв 43b, 432
Burkina Faso West African CFA franc XOF CFA 43, 46, 41
Burundi Burundi franc BIF FBu 46, 42, 75
Cambodia Riel KHR 17db
Cameroon Central African CFA franc XAF BEAC 42, 45, 41, 43
Canada Canadian dollar CAD $ 24
Cape Verde Escudo CVE Esc 45, 73, 63
Cayman Islands Cayman Islands dollar KYD $ 24
Central African Republic Central African CFA franc XAF CFA 43, 46, 41
Chad Central African CFA franc XAF BEAC 42, 45, 41, 43
Chile Chilean peso CLP $ 24
China Yuan Renminbi CNY ¥ a5
Christmas Island Australian dollar AUD $ 24
Cocos (Keeling) Islands Australian dollar AUD $ 24
Colombia Colombian peso COP $ 24
Comoros Comoros franc KMF KMF 4b, 4d, 46
Republic of the Congo Central African CFA franc XAF BEAC 42, 45, 41, 43
Democratic Republic of the Congo Congolese franc CDF EB 45, 42
Cook Islands New Zealand dollar NZD $ 24
Cook Islands Cook Islands dollar NZD $ 24
Costa Rica Costa Rican colon CRC 20a1
Cote D’Ivoire West African CFA franc XOF CFA 43, 46, 41
Croatia Croatian kuna HRK kn 6b, 6e
Cuba Cuban peso CUP 20b1
Cyprus European euro EUR 20ac
Czech Republic Czech koruna CZK 4b, 10d
Denmark Danish krone DKK kr 6b, 72
Djibouti Djibouti franc DJF Fdj 46, 64, 6a
Dominica East Caribbean dollar XCD EC$ 45, 43, 24
Dominican Republic Dominican peso DOP RD$ 52, 44, 24
Ecuador US dollar USD $ 24
Egypt Egyptian pound EGP £ a3
El Salvador US dollar USD $ 24
El Salvador Salvadoran colón SVC $ 24
Equatorial Guinea Central African CFA franc GQE CFA 43, 46, 41
Eritrea Eritrean nakfa ERN Nfa 4E, 66, 61
Eritrea Ethiopian Birr ETB Br 42, 72
Estonia Estonian kroon EEK kr 6b, 72
Ethiopia Ethiopian birr ETB Br 42, 72
Europe Euro EUR 20ac
Falkland Islands (Malvinas) Falkland Islands pound FKP £ a3
Faroe Islands Danish krone DKK kr 6b, 72
Fiji Fijian dollar FJD $ 24
Finland Euro EUR 20ac
Finland Finnish markka FIM mk 6d, 6b
France Euro EUR 20ac
France French franc FRF 20A3
French Guiana Euro EUR 20ac
French Guiana French franc FRF 20A3
French Polynesia CFP franc XPF F 46
French Southern Territories Euro EUR 20ac
French Southern Territories French franc FRF 20A3
Gabon Central African CFA franc XAF CFA 43, 46, 41
Gambia Gambian dalasi GMD D 44
Georgia Georgian lari GEL 10DA
Germany Euro EUR 20ac
Germany Deutsche mark DEM DM 44, 4d
Ghana Ghanian cedi GHS 20b5
Gibraltar Gibraltar pound GIP £ a3
Greece Euro EUR 20ac
Greece Greek drachma GRD 20AF
Greenland Danish krone DKK kr 6b, 72
Grenada East Caribbean dollar XCD EC$ 45, 43, 24
Guadeloupe Euro EUR 20ac
Guam United States dollar USD $ 24
Guatemala Guatemalan quetzal GTQ Q 51
Guernsey Sterling pound GBP £ a3
Guinea Guinean franc GNF FG 46, 47
Guinea-Bissau West African CFA franc XOF CFA 43, 46, 41
Guyana Guyanese dollar GYD $ 24
Haiti Haitian gourde HTG G 47
Holy See (Vatican City State) Euro EUR 20ac
Honduras Honduran lempira HNL $ 24
Hong Kong Hong Kong dollar HKD $ 24
Hungary Hungarian forint HUF Ft 46, 74
Iceland Icelandic krona ISK kr 6b, 72
India Indian rupee INR 20B9
Indonesia Indonesian rupiah IDR Rp 52, 70
Iran, Islamic Republic of Iranian rial IRR fdfc
Iraq Iraqi dinar IQD 39, 62e, 2f, 6
Ireland Euro EUR 20ac
Isle of Man Sterling pound GBP £ a3
Isle of Man Manx pound £ a3
Israel Israeli shekel ILS 20aa
Italy Euro EUR 20ac
Jamaica Jamaican dollar JMD J$ 4a, 24
Japan Japanese yen JPY ¥ a5
Jersey Sterling pound GBP £ a3
Jersey Jersey pound £ a3
Jordan Jordanian dinar JOD JOD 4a, 4f, 44
Kazakhstan Kazakhstani tenge KZT лв 43b, 432
Kenya Kenyan shilling KES KSh 4B, 53, 68
Kiribati Australian dollar AUD $ 24
Kiribati Kiribati dollar KBD $ 24
Korea, Democratic People’s Republic of North Korean won KPW 20a9
Korea, Republic of South Korean won KRW 20a9
Kuwait Kuwaiti dinar KWD 2f, 62e, 43, 6
Kyrgyzstan Kyrgyzstani som KGS лв 43b, 432
Lao People’s Democratic Republic Lao kip LAK 20ad
Latvia Latvian lats LVL Ls 4c, 73
Lebanon Lebanese lira LBP £ a3
Lesotho Lesotho loti LSL L 4c
Lesotho South African rand ZAR R 52
Liberia Liberian dollar LRD $ 24
Libyan Arab Jamahiriya Libyan dinar LYD د.ل 62F, 002E, 0644
Liechtenstein Swiss Franc CHF CHF 43, 48, 46
Lithuania Lithuanian litas LTL Lt 4c, 74
Luxembourg Euro EUR 20ac
Macau Macanese pataca MOP $ 24
Macedonia Macedonian denar MKD ден 434, 435, 43d
Madagascar Malagasy ariary MGA Ar 41, 72
Malawi Malawian kwacha MWK MK 4d, 4b
Malaysia Malaysian ringgit MYR RM 52, 4d
Maldives Maldivian rufiyaa MVR Rf 52, 66
Mali West African CFA franc XOF CFA 43, 46, 41
Malta Euro EUR 20ac
Marshall Islands United States dollar USD $ 24
Martinique Euro EUR 20ac
Mauritania Mauritanian ouguiya MRO UM 55, 4d
Mauritius Mauritian rupee MUR 20a8
Mayotte Euro EUR 20ac
Mexico Mexican peso MXN $ 24
Micronesia, Federated States of United States dollar USD $ 24
Moldova, Republic of Moldovan leu MDL MDL 4d, 44, 4c
Monaco Euro EUR 20ac
Mongolia Mongolian tughrik MNT 20ae
Montenegro Euro EUR 20ac
Montserrat East Caribbean dollar XCD EC$ 45, 43, 24
Morocco Moroccan dirham MAD /خخ 2f, 62e, 62e
Mozambique Mozambican metical MZN MT 4d, 54
Myanmar Myanmar Kyat MMK K 4b
Namibia Namibian dollar NAD $ 24
Namibia South African rand ZAR R 52
Nauru Australian dollar AUD $ 24
Nepal Nepalese rupee NPR 20a8
Netherlands Euro EUR 20ac
Netherlands Antilles Netherlands Antilles Guilder ANG ƒ 192
New Caledonia CFP franc XPF F 46
New Zealand New Zealand dollar NZD $ 24
Nicaragua Nicaraguan cordoba NIO C$ 43, 24
Niger West African CFA franc XOF CFA 43, 46, 41
Nigeria Nigerian naira NGN 20a6
Niue New Zealand dollar NZD $ 24
Norfolk Island Australian dollar AUD $ 24
Northern Mariana Islands United States dollar USD $ 24
Norway Norwegian krone NOK ɫɲ 26b, 272
Oman Omani rial OMR fdfc
Pakistan Pakistani rupee PKR 20a8
Palau United States dollar USD $ 24
Palestinian Territory, Occupied Jordanian dinar JOD JOD 4a, 4f, 44
Palestinian Territory, Occupied Egyptian Pound EGP £ a3
Palestinian Territory, Occupied Israeli shekel ILS 20aa
Panama Panamanian balboa PAB B/. 42, 2f, 2e
Panama United States dollar USD $ 24
Papua New Guinea Papua New Guinean kina PGK K 4b
Paraguay Paraguayan guarani PYG Gs 47, 73
Peru Peruvian nuevo sol PEN S/. 53, 2f, 2e
Philippines Philippine peso PHP 20b1
Poland Polish zloty PLN 7a, 142
Portugal Euro EUR 20ac
Puerto Rico United States dollar USD $ 24
Qatar Qatar riyal QAR fdfc
Reunion Euro EUR 20ac
Romania Romanian leu RON lei 6c, 65, 69
Russian Federation Russian ruble RUB руб 440, 443, 431
Rwanda Rwandan franc RWF RF 52, 46
Saint Helena Saint Helena pound SHP £ a3
Saint Kitts and Nevis East Caribbean dollar XCD EC$ 45, 43, 24
Saint Lucia East Caribbean dollar XCD EC$ 45, 43, 24
Saint Martin Euro EUR 20ac
Saint Martin Netherlands Antilles Guilder ANG ƒ 192
Saint Martin United States dollar USD $ 24
Saint Pierre and Miquelon East Caribbean dollar XCD EC$ 45, 43, 24
Saint Vincent and the Grenadines Eastern Caribbean Dollar XCD EC$ 45, 43, 24
Samoa Samoan tala WST WS$ 57, 53, 24
San Marino Euro EUR 20ac
Sao Tome and Principe Dobra STD Db 44, 62
Saudi Arabia Saudi riyal SAR fdfc
Senegal West African CFA franc XOF CFA 43, 46, 41
Serbia Serbian dinar RSD Дин. 414, 438, 43d, 2e
Seychelles Seychelles Rupee SCR 20a8
Sierra Leone Sierra Leone leone SLL Le 4c, 65
Singapore Singapore dollar SGD $ 24
Singapore Brunei dollar BND $ 24
Slovakia Euro EUR 20ac
Slovenia Euro EUR 20ac
Solomon Islands Solomon Islands dollar SBD $ 24
Somalia Somali shilling SOS S 53
South Africa South African rand ZAR R 52
South Georgia and the South Sandwich Islands Sterling pound GBP £ a3
Spain Euro EUR 20ac
Sri Lanka Sri Lankan rupee LKR 20a8
Sudan Sudanese pound SDG SDG 53, 44, 47
Suriname Surinamese dollar SRD $ 24
Svalbard and Jan Mayen Norwegian krone NOK ɫɲ 26b, 272
Swaziland Swazi lilangeni SZL SZL 53, 5a, 4c
Sweden Swedish krona SEK kr 6b, 72
Switzerland Swiss franc CHF CHF 43, 48, 46
Syrian Arab Republic Syrian pound SYP £ a3
Taiwan New Taiwan dollar TWD NT$ 4e, 54, 24
Tajikistan Tajikistani somoni TJS SM 53, 4d
Tanzania, United Republic of Tanzanian shilling TZS x 78
Thailand Thai baht THB ฿ e3f
Timor-Leste United States dollar USD $ 24
Togo West African CFA franc XOF CFA 43, 46, 41
Tokelau New Zealand dollar NZD $ 24
Tonga Tonga Pa’ang TOP T$ 54, 24
Trinidad and Tobago Trinidad and Tobago dollar TTD TT$ 54, 54, 24
Tunisia Tunisian dinar TND 2f, 62e, 2a, 6
Turkey Turkey Lira TRL 20a4
Turkmenistan Turkmenistani manat TMT m 6d
Turks and Caicos Islands United States dollar USD $ 24
Tuvalu Australian dollar AUD $ 24
Tuvalu Tuvalu Dollar TVD $ 24
Uganda Ugandan Shilling UGX USh 55, 53, 68
Ukraine Ukraine Hryvna UAH 20b4
United Arab Emirates UAE dirham AED 2f, 62e, 27, 655, 6
United Kingdom Sterling pound GBP £ a3
United States United States dollar USD $ 24
United States Minor Outlying Islands United States dollar USD $ 24
Uruguay Uruguay Peso UYU $U 24, 55
Uzbekistan Uzbekistan Som UZS лв 43b, 432
Vanuatu Vanuatu vatu VUV Vt 56, 74
Venezuela Venezuela Bolivar Fuerte VEF Bs 42, 73
Vietnam Viet Nam Dong VND 20ab
Virgin Islands, British United States dollar USD $ 24
Virgin Islands, U.S. United States dollar USD $ 24
Wallis and Futuna CFP franc XPF F 46
Western Sahara Euro EUR 20ac
Western Sahara Mauritanian ouguiya MRO UM 55, 4d
Yemen Moroccan dirham MAD /خEخ 2f, 62e, 45, 62e
Zambia Zambian kwacha ZMK ZK 5a, 4b
Zimbabwe Zimbabwe Dollar ZWD Z$ 5a, 24

References

Revision controlling with Subversion


Subversion also more famouly know as SVN is a version management tool used to manage source code, documents, designs, etc.. in software develpment. SVN is a centralized version control system where the main repository resides in a centralized environment and the parties involved in a project can download a copy of the repository to their local machine to introduce the changes. Once the changes are applied to the files/documents on the local copy, next step is to transfer them to the server. As mentioned SVN is a centralized version control system committing the changes to the server means incrementing the revision of the file/document on the main repository, when the other members of the team updates their local copy of the repository the change also get reflected into their local copy, where as with decentralized version control systems like Git, Mercural, etc.. the users can maintain their own local repository which they can commit the changes without pushing directly to the project in the main repository.

In the following sections I’m going to focus your attention on following aspects of setting up and maintaining a version control system taking SVN as a reference.

  • Setting up a new repository
  • Restoring and Backup a repository
  • Basic commands in SVN

Setting up a new repository

Step 1: Lets start by installing SVN(subversion) and required Apache module (libapache2-svn) into a Linux server/desktop (Debian based distributions but with rest of them are also follows the same). All the commands involve in this process needs to be executed as a super user, so always keep in mind to append sudo to the begining.

$ sudo apt-get install subversion libapache2-svn

Step 2: Creating the target directory for SVN repositories. Please note that name given to the SVN repository will be use as the base URL, which will be used to browse the repository via a web browser. The name can be mapped to any domain name or subdomain name (prefferbly via a virtual host) of your choice for easy access, which I’m not going to explore into it. Execute the following command on the terminal window.

$ sudo mkdir /path/to/svn-repository

Step 3: Introducing a new group to manage the activities of the SVN repositories. Let’s call that group as subversion, execute the following command on the terminal window.

$ sudo addgroup subversion

Step 4: Grant the ownership of the directory create to hold the SVN repositoriesy to the newly created subversion group and the www-data, the group which manages the permissions on resources that is requested via Apache Web server.

$ sudo chown -R www-data:subversion /path/to/svn-repository

Step 5: Now change the mode of the folder created to hold the repository using the following command.

$ sudo chmod -R g+rws /path/to/svn-repository

Step 6: Now we have finished the steps involve in setting up the environment, next let’s look at the steps to create a new project repository called svnproject.

$ sudo svnadmin create /path/to/svn-repository/svnproject

Step 7: Adding user(s) to the newly created subversion group. To introduce a new password file along with a new user, include -c and issue the following command, once hitting enter it prompts for the password and it’s confirmation.

$ sudo htpasswd -c /etc/subversion/passwd Hayesha

To introduce a new user to the existing subversion group to access and carry out activities execute the command as follows

$ sudo htpasswd /etc/subversion/passwd Piushan

Step 8: Grant access to the newly created SVN repository (eg: svnproject) via WebDAV protocol, configure the Apache Web server by adding the following snippet to dav_svn.conf file located under /etc/apache2/mods-available/. Place the following block of code snippt inside dav_svn.conf to reflect the parameters of the newly created SVN repository.

$ sudo cp /etc/apache2/mods-available/dav_svn.conf /etc/apache2/mods-available/dav_svn.conf.bk # Creating a backup of the existing configuration file
$ sudo vim /etc/apache2/mods-available/dav_svn.conf


<Location /svn-repository/svnproject>
    DAV svn
      SVNPath /path/to/svn-repository/svnproject
      AuthType Basic
      AuthName "Svnproject Repository"
      AuthUserFile /etc/subversion/passwd
      Require valid-user
</Location>

Step 9: Grant the ownership of the newly created SVN project to subversion and www-data groups.

$ sudo chown -R www-data:subversion /path/to/svn-repository/svnproject/

Step 10: Now change the mode of newly created SVN project using the following command.

$ chmod -R g+rws /path/to/svn-repository/svnproject/

Step 11: Restart Apache server.

$ sudo /etc/init.d/apache2 restart

Reference: https://help.ubuntu.com/community/Subversion

Restoring and Backup a SVN repository

  • Backup a SVN repository
    • The SVN‘s administrative tool named dump will be used to backup a project inside the SVN repository, this can be achieved using following cmmand.

      $ svnadmin dump /path/to/svn-repository/svnproject > /path/to/backup/svnproject_25_11_2012.dump

  • Restoring SVN project to an existing repository using a SVN backup (dump).
    • Create a new project called recoverproject inside the SVN repository as follows

      $ svnadmin create /path/to/svn-repository/recoverproject

    • Load the SVN dump of the project into new repository location as follows. This may take a while depending on the number of revisions had on the SVN dump used for the restoring.

      $ svnadmin load /path/to/svn-repository/recoverproject < /path/to/svn-project-backup.dump

Reference: https://wiki.archlinux.org/index.php/Subversion_backup_and_restore

Basic commands in SVN

Getting a checkout from a main repository of a SVN project. The syntax is svn checkout <url> or by using svn co <url>, co is the shorten form of checkout.

$ svn checkout http://svn.example.com/svn-repository/svnproject/trunk/xxx

To update the local copy (development version) with the main repository of the SVN project. Navigate to the location of the local copy and execute the following command. svn update or by using svn up (up is the shorten form for update.

$ svn update

In the instances where the local copy (development version) of the code base is out of sync (in a lower revision number(s) compared to main repository of the SVN project’s revision numbers), need to execute the svn cleanup command to sync the local copy to bring it to the server’s current revision in order to avoid conflicts that experience while committing the code.

$ svn cleanup

To transfer the modifications in the local copy of the code base to the main repository of the SVN project need to use the svn commit command. It’s always considered a best parctice to introduce a comment using -m ‘comment’ along with the commit command.

$ svn commit -m "adding the modification done to bla_bla-bla.ext" <filename.ext>

To introduce new file(s) and folder(s) to the SVN project on server use svn add folowed with svn commit commands.

$ svn add <filename.ext> or <folder name>
$ svn commit -m "adding the new file(s) or folder(s) called filename.ext or foldername/" <filename.ext> or <foldername/>

To remove file(s) and folder(s) from the main repository of the SVN project use svn update, svn delete followed by svn commit commands.

$ svn update
$ svn delete <filename.ext> or <folder name>
$ svn commit -m "removing existing file(s) or folder(s) called filename.ext or foldername/" <filename.ext> or <foldername>

References:

    Setting up osTicket to handle customer inquires


    osTicket is a widely used open source ticket management systems that can be easily configured to create tickets to inquires via emails, web interface, etc…

    Download

    It is always advisable to download the latest stable version of osTicket and move it to preferred location to continue the installation process.

    Configuration & Database creation

    Move the content inside uploads/ to root of the installation folder.

    Navigate to includes/ folder and rename ost-config.sample.php to ost-config.php and grant write permission.

    $ cp ost-config.sample.php ost-config.php
    $ chmod +w ost-config.php

    Connect to MYSQL server and create a new database to hold the tables and the corresponding data of the ticket system.

    $ mysql -u<username> -p<password>
    mysql> create database <database_name>
    mysql> show databases;

    Basic Installation: Step 1

    As shown in Figure 1 provide the relevant information to suite your requirement and click Install to proceed to next step. The information it captured is categorised in to following subsections.

    • Web path & title – Installation URL path and the Title(provide a meaningful title to describe your exact purpose) of the osTicket system
    • System email – The email address which acts as the sender for all the outgoing email via the ticket system.
    • Admin user
    • Database

    osTicket - Basic Installation

    Figure 1: osTicket – Basic Installation

    Basic Installation: Step 2

    As shown in Figure 2 once the installation is successful carry out the following two tasks.

    • Revoke write permission from ost-config.php using the following command
      • $ chmod 644 include/ost-config.php
    • Delete the setup directory
      • $ rm -fr setup/

    Installation success

    Figure 2: osTicket – Installation Successful

    Reference

    Configuration: General Preferences and Settings

    Login to Administration section (http://www.site-url.com/scp/login.php) or click on the Admin Panel link as shown in Figure 2 or invoke Admin Panel (http://www.site-url.com/scp/admin.php) section using the upper navigation bar found on the top right corner. Refer Figure 3.

    osTicket - Configuration - Admin Panel

    Figure 3: osTicket – Configuration – Admin Panel

    Next click on the Settings tab as shown in Figure 3. This section let you manage parameters such as General Settings, Date & Time, Ticket Options & Settings, Email Settings, Autoresponders and Alerts & Notices. I’ll be mainly focusing on parameters found under General Settings and Email Settings.

    • General Settings – Common parameters like Site URL, Title, Site Online/Off-line, Enable Auto Cron, etc…
      • Helpdesk Status – This lets us manage the ticket system go online or off-line.
      • Helpdesk URL – This allows to update the URL fro the ticket system.
      • Default Department – The categorisation which the system used to managed the Staff/Users in the system.
      • Enable Auto Cron – This enables the cronjob tasks scheduled by the system. I’ll be discussing how to configure and set the conjob tasks on the server.
    • Email Settings – Especially important if planning to use the ticket system to fetch email to generate tickets.
      • Incoming Emails – This settings provide the ability to control fetching of emails via POP/IMAP email and email piping.
      • Outgoing Emails – The mail server which outgoing email will be handled.
      • Default System Email – The email address which acts as the sender for all the outgoing email via the ticket system.

    Configuration: Admin Panel -> Settings -> API

    This section provide the facility to generate the API key used in remote email piping feature. This key needs to be updated in the automail.php or automail.pl files located inside scripts/ folder. Please provide the following two information to generate the API key.

    • Add New IP – Provide the IP address assigned to the server/instance the system is hosted.
    • API Passphrase – This will be used to generated the API key used in email piping feature.

    Setting up Email address(es) to automatically fetch to generate tickets.

    Lets see how to configure an email account in osTicket to automatically generate a ticket(email ticket) for each email received to the email address. Navigates to Admin Panel -> Email -> Add New Email. Please provide the following information described below.

    • Email Info – Settings are mainly for emailed tickets.
      • Email Address
      • Email Name – The text used as the FROM name of the email.
      • New Ticket Priority – The priority level which the ticket should be assigned.
      • New Ticket Dept – The department which the ticket should get categorised.
      • Login info – Required when IMAP/POP and/or SMTP are enabled
        • Username – The email address or the email ID of the email account.
        • Password – The password of the email account.
    • Mail Account – Setting for fetching incoming emails. Mail fetching must be enabled with autocron active or external cron setup.
      • Status – Enable/Disable email fetch feature
      • Host – Hostname of the POP or IMAP service of the email server
      • Port – The port number which POP or IMAP service available
      • Protocol – IMAP or POP
      • Encryption – Yes or NO
      • Fetch Frequency – The time interval which the ticket system to check for new emails.
      • Maximum Emails Per Fetch
    • SMTP Settings – When enabled the email account will use SMTP server instead of internal PHP mail() function for outgoing emails (optional).
      • Status – Enable/Disable the use of following SMPT details for all out going communication
      • SMTP Host – Hostname of the SMTP service of the email server
      • SMTP Port – The port number which SMTP service available
      • Authentication Required? – Yes or NO

    Scheduling the Cron job task

    Under General Preferences and Settings we came across a parameter to enables the cronjob tasks scheduled by the system. Open up your favourite editor (preferably vi). and issue following commands.


    $ crontab -l //to list already scheduled tasks
    $ crontab -e //to scheduled a new task

    If this is the first time it will prompt you to select the editor, select one that preferred by you and it will open up the crontab, schedule the task as follows.


    # m h dom mon dow command
    MAILTO="hayeshais at gmail.com" # this will fire an email upon failure to execute any of the scripts to this email address
    */5 * * * * nobody /usr/bin/php /var/www/api/cron.php
    */5 * * * * nobody wget -q -O /dev/null --user-agent=4816EC4CA293EE2EFCA2C89C88750F4A http://<www.helpdesk.examplesite.com>/api/cron.php
    */5 * * * * nobody /usr/bin/php -q /var/www/api/cron.php

    Enjoy.

    Run nodejs server continuously using forever


    By last Friday morning the open bugs count raised above 150 mark and we managed to take it down to under 25 by the end of the day, thanks to the dedicated effort by the team. Among them, one was to make the Nodejs server run continuously. In our application we are using the Nodejs to implement a near-realtime notification module, which involved pushing notifications and updates to the Web application as well to the Mobile application. Faye module was used to implement the identified channel patterns required to identify different user types and their respective activities each user type executes.

    Nodejs is a powerful, Event-driven, I/O based or JavaScript based server that can be used to develop interactive applications. It was developed on top of the V8 JavaScript engine.

    Installing Nodejs on Ubuntu

    I came across an easier approach to install Nodejs via the Ubuntu Package Manager.

    $ sudo apt-get install python-software-properties
    $ sudo add-apt-repository ppa:chris-lea/node.js
    $ sudo apt-get update
    $ sudo apt-get install nodejs npm

    Install Forever

    In the last command it installed the Node Package Manager(npm). The npm provides the facility to install any module to Nodejs in a manner similar to installing application on Ubuntu via apt-get.

    $ sudo npm install forever --global

    Setting the –global parameter makes the module accessible globally

    Updating Nodejs & the modules to latest stable release

    Nodejs comes with a module called n aka node version manager provides the facility to update the module to its latest or stable release.

    Install n

    $ sudo npm install --global n

    Updating Nodejs using n

    $ sudo n --global latest/stable
        or
    $ sudo n --global custom v0.x.x

    Updating Nodejs modules using n

    $ sudo n --global npm
        or
    $ sudo n --global forever

    « Older entries