Here is a very simple Bash script that uploads a file to Amazon’s S3. I’ve looked for a simple explanation on how to do that without perl scripts or C# code, and could find none. So after a bit of experimentation and some reverse engineering, here’s the simple sample code:
date="$(LC_ALL=C date -u +"%a, %d %b %Y %X %z")"
md5="$(openssl md5 -binary < "$file" | base64)"
sig="$(printf "PUT\n$md5\n$content_type\n$date\n/$bucket/$path" | openssl sha1 -binary -hmac "$key_secret" | base64)"
curl -T $file http://$bucket.s3.amazonaws.com/$path \
-H "Date: $date" \
-H "Authorization: AWS $key_id:$sig" \
-H "Content-Type: $content_type" \
-H "Content-MD5: $md5"
So what is going on here?
This example script uses what Amazon calls “pre-authenticated URL”, which basically means that instead of performing a “login process” for each upload to generate an an authorization key (OAUTH style), you generate the authorization key yourself by signing the upload details with your AWS secret.
So we first the upload details – these need to be consistent between the signature step and the actual upload request, so we save them in variables instead of using them directly:
key_secretshould be changed to be your actual AWS key detail (no, you can’t have mine)
pathis the path of the file to be stored under the S3 bucket (what Amazon calls the file’s “key”)
bucketis the name of the S3 bucket where we will upload the file
content_typeis the MIME Content-Type of the file, which – unless you want to serve the file through a web site – shouldn’t matter much so I’m using “application/octet-stream” (“binary” for you systems folks)
dateis an RFC2616 compliant date specification. I force
LC_ALLto the POSIX “C” locale to prevent
datefrom genearting localized day and month names, which are not valid in RFC2616
md5is an RFC2616 Content-MD5 compliant MD5 checksum, which means its the binary MD5 digest encoded using Base64 (instead of the hex encoding most people use nowadays)
Finally it is time to generate the authorization key, which we do by creating an “upload details message” that includes:
- The HTTP method (“verb”) that we will use for the upload – which we will use “
- The MD5 signature, which must be the same as the
Content-MD5header in the
- The MIME content type, which must be the same as the
Content-Typeheader in the
- The current date, which must be the same as the
Dateheader in the
- And finally, the S3 resource key for the file that we are uploading, which is composed of the bucket name and the file key
The upload details message is signed using SHA1 HMAC with our AWS secret key, and Base64 ASCII-armoured. After which we can issue the request, where we use the authorization type
AWS (meaning AWS pre-signed URLs) and the authorization token is our AWS key ID and the upload signature.
If all goes well, the script should output nothing, otherwise there is some problem and the AWS response should provide the necessary details. Its important to note that all the headers I’m using in the
curl command are required, though AWS support additional AWS specific headers that start with
X-AMZ- and allow you to provide additional meta-data, such as the uploader’s name and the required permissions for the file. If you use
X-AMZ headers, then you must also add these headers in the “upload details message” that is signed – check the AWS documentation for more details.