This post will show you how to mount an S3 bucket to your Ubuntu Server for unlimited pay-as-you-go storage that works just like a locally mounted drive. There are handful of open source projects out there for mounting S3 buckets on Linux. Here we’re highlighting the S3QL project created by Nikolaus Rath.
At the time of writing S3QL is the most actively developed project of the bunch with code commits as recent as the last few days. Its easy to install and setup, has great documentation and even though its Beta software, it comes with a solid set of features including; data de-duplication, dynamic sizing, encryption and a focus on performance.
Before getting started it’s helpful to understand a bit about how S3QL stores files. It works by taking the contents of your files and splitting them up into individual blocks. Each block is then stored inside your S3 bucket. Internally S3QL uses a SQLite database to keep track of how everything is stored so your data can be retrieved quickly and efficiently. Common operations like renaming, moving and copying files don’t even hit the network because they’re all cataloged in the database.
S3QL includes built-in support for data de-duplication. As new blocks of a file are created it intelligently checks to see if any of the new blocks are identical to the existing blocks its keeping track of in its database. If they match, it simply links the new block to the existing block on S3 so it never sends duplicate data over the network. Similarly when files are changed it only transports the individual blocks that changed which is a big performance win over constantly transporting large files when only a few small pieces have changed.
While they’re some great benefits to the approach S3QL takes there are some tradeoffs. Because of the way S3QL splits everything into blocks it tracks with a database it’s not designed with the intention of sharing buckets with other S3 clients. Accessing a S3QL managed bucket with a S3 client like the AWS Management console or tools like Forklift and Expandrive will only reveal the individual blocks. Only S3QL knows how to re-assemble those blocks back into your files. Along the same lines, buckets mounted with S3QL are designed to be mounted in one place at a time so it’s not a solution for sharing data between two servers using S3QL simultaneously. If a sharing is a must-have feature consider the Dropbox approach instead.
With that in mind S3QL is a great solution provided your goals are in-line with it’s features. Let’s start mounting S3 buckets!
S3QL packages are available for Ubuntu 10.04 (Lucid) and newer.
sudo aptitude install python-software-properties
Add S3QL package repositories
sudo add-apt-repository ppa:nikratio/s3ql sudo add-apt-repository ppa:ubuntu-rogerbinns/apsw
Update aptitude and install s3ql
sudo aptitude update sudo aptitude install s3ql
Setup Your Authinfo File
Before you have S3QL create a new bucket it’s a good idea to decide on a bucket name and setup an authinfo file that it’ll use to pick-up your AWS Access Key ID, AWS Secret Access Key and a password you set to encrypt your data. The authinfo file isn’t required but will save you some typing when you’re managing your buckets.
Create the .s3ql directory and authinfo file
mkdir ~/.s3ql vi ~/.s3ql/authinfo
Add these two lines to the authinfo file
backend s3 machine any login AWSAccessKeyID password AWSSecretAccessKey storage-url s3://your-bucket-name password yourEncryptionPassword
Be sure to replace AWSAccessKeyID, AWSSecretAccessKey, your-bucket-name and yourEncryptionPassword in the lines above. Login to your AWS account to grab your Access Key ID and Secret Access Key. You can set the encryption password to anything you want (aim for 10 characters).
Set secure permissions on the authinfo file
chmod 600 ~/.s3ql/authinfo
Create a New File System and S3 Bucket
Use mkfs.s3ql to create a new S3QL file system and bucket
mkfs.s3ql s3://your-bucket-name --s3-location us-west-1
Depending upon where your located you may want to change the S3 location from the default EU to something closer to you. Here we’re setting it to us-west-1. You can read more about the different options available in the mkfs.s3ql documentation.
Mount Your S3 Bucket
The next step is to create the mount point for your bucket and make sure the user you’re logged in as has the correct permissions to access it.
Setup /mnt/cloud-drive as the mount point and set appropriate ownership
sudo mkdir /mnt/cloud-drive sudo chown username /mnt/cloud-drive
Now your ready to mount your bucket to the new mount point!
Use mount.s3ql to mount your bucket
mount.s3ql --cachesize 204800 s3://your-bucket-name /mnt/cloud-drive
The –cachesize 204800 option set the cache size to 200MB instead of the 100MB default. You can read more about the different options available in the mount.s3ql documentation.
Now that your bucket is mounted you can begin using it to store data! If you run into errors you can always run the fsck.s3ql to help resolve them.
Un-mounting is just as easy as mounting.
Use umount.s3ql to un-mount the bucket
Mount Your Bucket Automatically On Boot
Nikolaus created a handy script to automatically mount your bucket on boot.
description "S3QL Backup File System" author "Nikolaus Rath <Nikolaus@rath.org>" # This assumes that eth0 provides your internet connection start on (filesystem and net-device-up IFACE=eth0) stop on runlevel  # Fill in your bucket name and mount point env BUCKET="s3://your-bucket-name" env MOUNTPOINT="/mnt/your-mount-point" expect stop script # Redirect stdout and stderr into the system log DIR=$(mktemp -d) mkfifo "$DIR/LOG_FIFO" logger -t s3ql -p local0.info < "$DIR/LOG_FIFO" & exec > "$DIR/LOG_FIFO" exec 2>&1 rm -rf "$DIR" # Check and mount file system fsck.s3ql --batch "$BUCKET" exec mount.s3ql --upstart --allow-other "$BUCKET" "$MOUNTPOINT" end script pre-stop script umount.s3ql "$MOUNTPOINT" end script
To use this script create a new file in your /etc/init directory.
sudo vi /etc/init/s3qlmount.conf
Copy the contents of the script to the new file. Then make sure to fill in your bucket name, mount point and network interface (eth0, eth1, etc.) with the correct values.
You’ll also need to make sure you setup your authinfo file as described above so S3QL can authenticate correctly. Finally you’ll need to copy the .s3ql folder so it’s also under /root/ which is where the startup script will be looking for the authinfo file.
Put a copy of .s3ql directory under /root/ and adjust permissions
sudo cp -R ~/.s3ql /root/ sudo chown -R root.root /root/.s3ql
When it’s all setup the next time your system boots up your S3 bucket will automatically be mounted for you!