Amazon S3 + S3QL on Ubuntu 12.04 mini-HOWTO
Amazon S3 provide object storage with "highly scalable, reliable, secure, fast, inexpensive infrastructure" which very useful for different use cases, e.g. storing your team shared files, your website backup, etc. BTW, it is using different protocal when compare with FTP/FUSE/NFS/iSCSI/etc so we need some trick in order to use it as like as local file system and mount it as usual.
S3QL give you the quick answer: it is simple to install and configure, stable and also good in performance; moreover, S3QL also support other object storage service provider, e.g. Google Storage, Amazon S3 or OpenStack, so we can switch to different vendor if do required; and one of the key benefit when compare with other solution, e.g. S3FS, it is supported by Ubuntu 12.04 so I can set it up within 30mins!
This mini-HOWTO will guide you though some basic for mounting your S3 bucket as local file system, so you can use it for whatever purpose. You need to have Ubuntu 12.04 LTS install correctly.
Install s3cmd and s3ql
NOTE: I found that Ubuntu 12.04 ship with S3QL 1.9-1, which I found unstable for production environment... Therefore I will manually install S3QL from source package.
Just simply execute following commands (I use uuid to create a unique S3 bucket name):
aptitude update aptitude -y install s3cmd uuid apt-get build-dep s3ql
Now download the S3QL source package, build and install it:
cd /usr/local/src wget http://s3ql.googlecode.com/files/s3ql-1.12.tar.bz2 tar jxvf s3ql-1.12.tar.bz2 cd s3ql-1.12 python setup.py build python setup.py test python setup.py install
Get Your AWS Account Information
If you haven't already, sign up with Amazon Web Services and enable S3 for your account.
On the AWS page, hover over the "Your Account" tab, and select "Security Credentials". Find the section labeled "Your access keys" - make a note of your Access Key ID, then click on the link labeled "Show" under "Secret Access Keys" and note that information, too. You will have to provide both pieces of information to the s3 commands you'll be using later.
Setup s3cmd and Create a Bucket
S3 lets you create buckets to store your information; you have to use the AWS API to create buckets, and you won't be able to create buckets with s3fs. If you haven't already created a bucket in your S3 account, you can use the s3cmd program to set one up (Just answer the qustion and fill up with your information):
Once complete s3cmd should show your some message similar as below:
Test access with supplied credentials? [Y/n] Please wait... Success. Your access key and secret key worked fine :-) Now verifying that encryption works... Success. Encryption and decryption worked fine :-) Save settings? [y/N] y Configuration saved to '/root/.s3cfg'
Now you can create your bucket with:
s3cmd mb s3://`uuid`
Will prompt you with a result bucket name as below:
Bucket 's3://915cfdf8-44ca-11e2-9759-12314000610e/' created
Test putting some file to it:
s3cmd put /var/log/syslog s3://915cfdf8-44ca-11e2-9759-12314000610e/
And list object to verify if all works well:
s3cmd ls s3://915cfdf8-44ca-11e2-9759-12314000610e/
Configure s3ql Authentication, Initialize and Test Mount Manually
Now we go to s3ql. First of all create an authentication file, e.g. /root/.s3ql/authinfo2 as below:
[s3] storage-url: s3:// backend-login: joe backend-password: notquitesecret fs-passphrase: ll23bi
Update /root/.s3ql/authinfo2 permission:
chmod 600 /root/.s3ql/authinfo2
Initialize the bucket for S3QL:
mkfs.s3ql --plain --force s3://915cfdf8-44ca-11e2-9759-12314000610e/
And test mount manually:
mount.s3ql s3://915cfdf8-44ca-11e2-9759-12314000610e/ /mnt
For sure you should put some file to it, umount and mount it again and again, also check with df -h ;-)
Setup Automatic Mounting
It is not suitable to setup S3QL as like as normal file system with /etc/fstab. It is recommend to using upstart to handle the automatic mounting.
Create the upstart script /etc/init/s3ql_upstart.conf as below:
# # This file can be placed in /etc/init. It defines an upstart job that # takes care of mounting and unmounting an S3QL file system. # description "S3QL Backup File System" author "Nikolaus Rath <Nikolaus@rath.org>" # This assumes that eth0 provides your internet connection start on (filesystem and net-device-up IFACE=eth0) # We can't use "stop on runlevel " because from that point on we # have only 10 seconds until the system shuts down completely. stop on starting rc RUNLEVEL= # Time to wait before sending SIGKILL to the daemon and # pre-stop script kill timeout 300 env STORAGE_URL="s3://915cfdf8-44ca-11e2-9759-12314000610e/" env MOUNTPOINT="/mnt" env USER="root" env AUTHFILE="/root/.s3ql/authinfo2" expect stop script # Redirect stdout and stderr into the system log DIR=$(mktemp -d) mkfifo "$DIR/LOG_FIFO" logger -t s3ql -p local0.info < "$DIR/LOG_FIFO" & exec > "$DIR/LOG_FIFO" exec 2>&1 rm -rf "$DIR" # Check and mount file system su -s /bin/sh -c 'exec "$0" "$@"' "$USER" -- \ fsck.s3ql --batch --authfile "$AUTHFILE" "$STORAGE_URL" exec su -s /bin/sh -c 'exec "$0" "$@"' "$USER" -- \ mount.s3ql \ --allow-other \ --authfile "$AUTHFILE" \ --cachedir /tmp \ --cachesize 33554432 \ --compress none \ --max-cache-entries 65536 \ --upstart \ "$STORAGE_URL" "$MOUNTPOINT" end script pre-stop script su -s /bin/sh -c 'exec "$0" "$@"' "$USER" -- umount.s3ql "$MOUNTPOINT" end script
Now review /etc/init/s3ql_upstart.conf and setup with your information, reboot, and check the result.
You may also hope to add "--allow-other", "--cachedir /tmp" or other else options to the mount.s3ql line. Remember usually you do need to customize the connection setting.
You can also invoke the upstart script with before reboot:
initctl start s3ql_upstart