Amazon S3 + S3QL on Ubuntu 12.04 mini-HOWTO

Amazon S3 provide object storage with "highly scalable, reliable, secure, fast, inexpensive infrastructure" which very useful for different use cases, e.g. storing your team shared files, your website backup, etc. BTW, it is using different protocal when compare with FTP/FUSE/NFS/iSCSI/etc so we need some trick in order to use it as like as local file system and mount it as usual.

S3QL give you the quick answer: it is simple to install and configure, stable and also good in performance; moreover, S3QL also support other object storage service provider, e.g. Google Storage, Amazon S3 or OpenStack, so we can switch to different vendor if do required; and one of the key benefit when compare with other solution, e.g. S3FS, it is supported by Ubuntu 12.04 so I can set it up within 30mins!

This mini-HOWTO will guide you though some basic for mounting your S3 bucket as local file system, so you can use it for whatever purpose. You need to have Ubuntu 12.04 LTS install correctly.

Install s3cmd and s3ql

NOTE: I found that Ubuntu 12.04 ship with S3QL 1.9-1, which I found unstable for production environment... Therefore I will manually install S3QL from source package.

Just simply execute following commands (I use uuid to create a unique S3 bucket name):

aptitude update
aptitude -y install s3cmd uuid
apt-get build-dep s3ql

Now download the S3QL source package, build and install it:

cd /usr/local/src
wget http://s3ql.googlecode.com/files/s3ql-1.12.tar.bz2
tar jxvf s3ql-1.12.tar.bz2
cd s3ql-1.12
python setup.py build
python setup.py test
python setup.py install

Get Your AWS Account Information

If you haven't already, sign up with Amazon Web Services and enable S3 for your account.

On the AWS page, hover over the "Your Account" tab, and select "Security Credentials". Find the section labeled "Your access keys" - make a note of your Access Key ID, then click on the link labeled "Show" under "Secret Access Keys" and note that information, too. You will have to provide both pieces of information to the s3 commands you'll be using later.

Setup s3cmd and Create a Bucket

S3 lets you create buckets to store your information; you have to use the AWS API to create buckets, and you won't be able to create buckets with s3fs. If you haven't already created a bucket in your S3 account, you can use the s3cmd program to set one up (Just answer the qustion and fill up with your information):

s3cmd --configure

Once complete s3cmd should show your some message similar as below:

Test access with supplied credentials? [Y/n]
Please wait...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'

Now you can create your bucket with:

s3cmd mb s3://`uuid`

Will prompt you with a result bucket name as below:

Bucket 's3://915cfdf8-44ca-11e2-9759-12314000610e/' created

Test putting some file to it:

s3cmd put /var/log/syslog s3://915cfdf8-44ca-11e2-9759-12314000610e/

And list object to verify if all works well:

s3cmd ls s3://915cfdf8-44ca-11e2-9759-12314000610e/

Configure s3ql Authentication, Initialize and Test Mount Manually

Now we go to s3ql. First of all create an authentication file, e.g. /root/.s3ql/authinfo2 as below:

[s3]
storage-url: s3://
backend-login: joe
backend-password: notquitesecret
fs-passphrase: ll23bi

Update /root/.s3ql/authinfo2 permission:

chmod 600 /root/.s3ql/authinfo2

Initialize the bucket for S3QL:

mkfs.s3ql --plain --force s3://915cfdf8-44ca-11e2-9759-12314000610e/

And test mount manually:

mount.s3ql s3://915cfdf8-44ca-11e2-9759-12314000610e/ /mnt

For sure you should put some file to it, umount and mount it again and again, also check with df -h ;-)

Setup Automatic Mounting

It is not suitable to setup S3QL as like as normal file system with /etc/fstab. It is recommend to using upstart to handle the automatic mounting.

Create the upstart script /etc/init/s3ql_upstart.conf as below:

#
# This file can be placed in /etc/init. It defines an upstart job that
# takes care of mounting and unmounting an S3QL file system.
#
description     "S3QL Backup File System"
author          "Nikolaus Rath <Nikolaus@rath.org>"

# This assumes that eth0 provides your internet connection
start on (filesystem and net-device-up IFACE=eth0)

# We can't use "stop on runlevel [016]" because from that point on we
# have only 10 seconds until the system shuts down completely.
stop on starting rc RUNLEVEL=[016]

# Time to wait before sending SIGKILL to the daemon and
# pre-stop script
kill timeout 300

env STORAGE_URL="s3://915cfdf8-44ca-11e2-9759-12314000610e/"
env MOUNTPOINT="/mnt"

env USER="root"
env AUTHFILE="/root/.s3ql/authinfo2"

expect stop

script
    # Redirect stdout and stderr into the system log
    DIR=$(mktemp -d)
    mkfifo "$DIR/LOG_FIFO"
    logger -t s3ql -p local0.info < "$DIR/LOG_FIFO" &
    exec > "$DIR/LOG_FIFO"
    exec 2>&1
    rm -rf "$DIR"

    # Check and mount file system
    su -s /bin/sh -c 'exec "$0" "$@"' "$USER" -- \
        fsck.s3ql --batch --authfile "$AUTHFILE" "$STORAGE_URL"
    exec su -s /bin/sh -c 'exec "$0" "$@"' "$USER" -- \
        mount.s3ql \
          --allow-other \
          --authfile "$AUTHFILE" \
          --cachedir /tmp \
          --cachesize 33554432 \
          --compress none \
          --max-cache-entries 65536 \
          --upstart \
          "$STORAGE_URL" "$MOUNTPOINT"
end script

pre-stop script
    su -s /bin/sh -c 'exec "$0" "$@"' "$USER" -- umount.s3ql "$MOUNTPOINT"
end script

Now review /etc/init/s3ql_upstart.conf and setup with your information, reboot, and check the result.

You may also hope to add "--allow-other", "--cachedir /tmp" or other else options to the mount.s3ql line. Remember usually you do need to customize the connection setting.

You can also invoke the upstart script with before reboot:

initctl start s3ql_upstart

Reference

• http://www.linode.com/wiki/index.php/S3fs
• http://www.rath.org/s3ql-docs/authinfo.html
• http://www.rath.org/s3ql-docs/mkfs.html
• http://www.rath.org/s3ql-docs/mount.html