Things n' Stuff Thoughts about the universe in general

As anyone who works with the Amazon Web Services API knows, when you submit requests to an AWS service you need to sign the request with your secret key – in order to authenticate your account. The AWS signing process has changed through the years – an earlier version (I think version 1) I implemented in a previous blog post: upload files to Amazon S3 using Bash, with new APIs and newer versions of existing APIs opt in to use the newer signing process.

The current most up to date version of the signing process is known as Signature Version 4 Signing Process and is quite complex, but recently I had the need to use an AWS API that requires requests to be signed using the version 4 process in a bash script¹, so it was time to dust off the old scripting skills and see if I can get this much much much more elaborate signing process to work in bash – and (maybe) surprisingly it is quite doable.

With no further ado, here is the code:

# figure out how to generate SHA256 hashes (support Linux and FreeBSD)
which sha256sum >/dev/null 2>&1 && HASH=sha256sum || HASH=sha256
# shorthand for a verbose mktemp call that works on FreeBSD
MKTEMP="mktemp -t aws-sign.XXXXXX"

function format_date_from_epoch() { # FreeBSD has an annoyingly non GNU-like data utility
	local epoch="$1" format="$2"
	if uname | grep -q FreeBSD; then
		date -u -jf %s $epoch "$format"
	else
		date -u -d @$epoch "$format"
	fi
}

function hash() { # generate a hex-encoded SHA256 hash value
  local data="$1"
  printf "%s" "$data" | $HASH | awk '{print$1}'
}

function hmac() {
  local keyfile="$1" data="$2"
  printf "%s" "$data" | openssl dgst -sha256 -mac HMAC -macopt hexkey:"$( hex < $keyfile )" -binary
}

function hex() { # pipe conversion of binary data to hexencoded byte stream
  # Note: it will mess up if you send more than 256 bytes, which is the maximum column size for xxd output
  xxd -p -c 256
}

function derive_signing_key() {
  local user_secret="$1" message_date="$2" aws_region="$3" aws_service="$4"
  step0="$($MKTEMP)" step1="$($MKTEMP)" step2="$($MKTEMP)" step3="$($MKTEMP)"
  printf "%s" "AWS4${user_secret}" > $step0
  hmac "$step0" "${message_date}" > $step1
  hmac "$step1" "${aws_region}" > $step2
  hmac "$step2" "${aws_service}" > $step3
  hmac "$step3" "aws4_request"
  rm -f $step0 $step1 $step2 $step3
}

function get_authorization_headers() { # the main implementation. Call with all the details to produce the signing headers for an HTTP request
  # Input parameters:
  # User key [required]
  # User secret [required]
  # Timestamp for the request, as an epoch time. If omitted, it will use the current time [optional]
  # AWS region this request will be sent to. If omitted, will use "us-east-1" [optional]
  # AWS service that will receive this request. [required]
  # Request address. If omitted (for example for calls without a path part), "/" is assumed to be congruent with the protocol. [optional]
  # Request query string, after sorting. May be empty for POST requests [optional]
  # POST request body. May be empty for GET requests [optional]
  local user_key="$1" user_secret="$2" timestamp="${3:-$(date +%s)}" aws_region="${4:-us-east-1}"
  local aws_service="$5" address="${6:-/}" query_string="$7" request_payload="$8"

  message_date="$(format_date_from_epoch $timestamp +%Y%m%d)"
  message_time="$(format_date_from_epoch $timestamp +${message_date}T%H%M%SZ)"
  aws_endpoint="${aws_service}.${aws_region}.amazonaws.com"

  # we always add the host header here but not in the output because we expect the HTTP client to send it automatically
  headers="$(printf "host:${aws_endpoint}\nx-amz-date:${message_time}")"
  header_list="host;x-amz-date"

  canonical_request="$(printf "GET\n${address}\n%s\n${headers}\n\n${header_list}\n%s" "${query_string}" "$(hash "$request_payload")")"
  canonical_request_hash="$(hash "$canonical_request")"
  credential_scope="${message_date}/${aws_region}/${aws_service}/aws4_request"
  string_to_sign="$(printf "AWS4-HMAC-SHA256\n${message_time}\n${credential_scope}\n${canonical_request_hash}")"
  signing_key="$($MKTEMP)"
  derive_signing_key "${user_secret}" "${message_date}" "${aws_region}" "${aws_service}" > $signing_key
  signature="$($MKTEMP)"
  hmac "${signing_key}" "${string_to_sign}" > $signature
  authorization_header="Authorization: AWS4-HMAC-SHA256 Credential=${user_key}/${credential_scope}, SignedHeaders=${header_list}, Signature=$( hex < $signature)"
  echo "X-Amz-Date: ${message_time}"
  echo "$authorization_header"
  rm -f $signing_key $signature
}

The get_authorization_headers() function implements the signature generation, and given the correct input will generate header lines for the Authorization and X-Amz-Date headers.

Here is an example usage, that generates a message to SQS and submits it using curl (there’s some heavy handed code to add the required -H flags, which I’m sure can be done in a more elegant way – please suggest):

queue_submitter_key="AKIDEXAMPLE"
queue_submitter_secret="wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY"
aws_service="sqs"
queue_address="/1234567890/queue-name"
queue_url="https://${aws_service}.amazonaws.com${queue_address}"
message="testing"
query_string="Action=SendMessage&MessageBody=$message"

IFS=$'\n' read -rd '' -a curl_headers < <(get_authorization_headers "$queue_submitter_key" "$queue_submitter_secret" "" "" "$aws_service" "$queue_address" "$query_string" "" | sed 's,^,-Hn,')
curl "${curl_headers[@]}" "$queue_url?${query_string}"

This code assumes that no other headers will be sent, which works for my use case where I only submit AWS API calls using HTTP’s GET method. If you want to implement calls that use the POST method, and you want additional headers, there is some API missing to allow you to specify additional headers to be signed and to sort them correctly for the canonical representation – I’m actually not sure if this is required: from reading the documentation (linked above) it seems to me that you can send requests headers that need not be signed and thus you may not need to change the code at all, or maybe just add the Content-Type headers that could be added hard-coded to the implementation.

I would try to explain what this code does and why it does this, but in all truth I find the whole version 4 signing process annoyingly complicated and more “Security by Obscurity” than actual useful defense against cryptoanalysis². So I’ll just say that I believe that by reading the signing document and consulting my verbose variable name you should be able to figure out how everything works.

I’ll also mention one caveat – this code relies on the not-so-common command line utility xxd which is apparently part of the VIM software suite, so if you are missing it on your platform you’d need to install VIM. I’m using it to convert binary digest data to hex encoded byte stream because I couldn’t figure out how to get hexdump to output continuous byte-for-byte hex encoded output. I would very much welcome suggestions on how to implement this functionality using bash only calls (maybe something with read and printf?).

Update

The script in a previous version that was posted had some problem dealing with the binary data generated by the digest, because I was trying to be a smartass and try to store binary data in Bash variables – and apparently this works only some of the time.

The version above uses temporary files for the hmac calculation and the OpenSSL -macopt feature to send binary key data. Because of the requirement for this recent OpenSSL feature, and because FreeBSD 9 base system installs with an outdated OpenSSL version 0.9.8, in order to use the script on FreeBSD you’d need to upgrade your OpenSSL to version 1.0 at least, which is available in the ports system but for the life of me I couldn’t figure out how to use that to replace the base system’s OpenSSL³.

1. I’m trying to use SQS to send change notifications from a FreeBSD jail running on a FreeNAS server – a place were I’m uncomfortable installing the AWS CLI tool or the SDK. This also help explains all the FreeBSD compatibility written into the code [↩]
2. For which I think version 1 was useful enough and version 2 was fine. Come to think of it, I don’t think I’ve ever saw a “version 3” of the process [↩]
3. which is installed in /usr/bin vs. ports which always install stuff in /usr/local/bin. FreeBSD can sometimes be really annoying like that [↩]

This entry was posted on Wednesday, November 19th, 2014 at 12:05 and is filed under Articles, Programming, Script Day. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.