IK.AM

Using IAM roles is recommended over IAM users for service access in AWS. IAM users use fixed access keys, which can lead to leakage and management difficulties. In contrast, IAM roles use temporary tokens to control access, providing higher security and flexibility. Access can be granted only when needed and easily revoked when not, making it a safer and more convenient method.

Let's consider a case where an app on Cloud Foundry (assuming Tanzu Application Service in this article) accesses AWS services.

Table of Contents

• Implementation Method
• Deploying CF Identity Token Service
• Registering CF Identity Token Service as an OIDC Provider in AWS IAM
• Creating IAM Role
• Creating Policy
• Accessing with AWS CLI
• Using a Sidecar to Obtain Tokens

Implementation Method

Apps on Cloud Foundry are launched in containers on VMs called Diego Cells. Internally, containerd and runc are used. When operating Cloud Foundry on AWS, Diego Cells are installed on EC2 instances. Currently, to use IAM roles in Cloud Foundry apps, we have no choice but to use the Instance Profile of these EC2 instances. However, if we set IAM roles at the VM level, all containers on that VM will share the same role, which is generally not acceptable.

One way to assign IAM roles at the app level is to use Web Identity Tokens and AWS STS (Security Token Service). In this method, the app obtains a Web Identity Token (such as a JWT issued by an OIDC provider) and uses that token to request temporary security credentials from STS.

Specifically, the app first obtains a Web Identity Token. Next, it passes that token to the AWS STS AssumeRoleWithWebIdentity API to request to assume an IAM role. STS validates this request and issues a security token with temporary access rights to the app. Using this token, the app can access the necessary AWS resources. This method allows each application to have a unique IAM role, enabling fine-grained access control. Additionally, since it uses temporary tokens instead of fixed access keys, security is enhanced.

How can we implement this method in Cloud Foundry?

While Cloud Foundry apps do not have the Web Identity Token itself, there is a mechanism called Instance Identity to identify application instances. It consists of a TLS certificate. Each container has its public key to prove its identity at the path of the environment variable CF_INSTANCE_CERT, and the private key exists at the path of CF_INSTANCE_KEY.

You can check the contents of the certificate by logging into the container with cf ssh and running the following command. The certificate is issued by Diego's certificate authority.

$ cat $CF_INSTANCE_CERT | openssl x509 -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            fc:4f:ba:ee:97:01:49:2f:74:e2:f8:94:94:ba:37:a9
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = Diego Instance Identity Intermediate CA
        Validity
            Not Before: Aug 28 06:38:44 2024 GMT
            Not After : Aug 29 06:38:44 2024 GMT
        Subject: OU = app:f60fab0d-2937-46f7-ad66-774d910826fd + OU = space:34e1bb23-0e76-4aad-95d7-1abe3ea1dcd8 + OU = organization:4b84793c-f3ea-4a55-92b7-942726aac163, CN = cfd85d7a-6140-4da4-7505-0d7f
...

From the Subject, we can see that it contains the app guid, space guid, and org guid of this app. Since the information to prove its identity is already prepared in the form of a TLS certificate, if we can convert this certificate into a JWT, we can use AssumeRoleWithWebIdentity.

Thus, I implemented a mechanism to perform this conversion (CF Identity Token Service). https://github.com/making/cf-identity-token-service The mechanism is simple: it extracts the app guid, space guid, and org guid from the Subject of the mTLS authenticated client certificate and converts them into JWT claims.

The IAM authentication using CF Identity Token Service follows the flow shown in the diagram below.

(The diagram is referenced from this article)

Now, let's try it out.

Deploying CF Identity Token Service

CF Identity Token Service requires an RSA public key and private key for signing JWTs, so we will generate them with the following commands.

openssl genrsa -out private.pem 2048
openssl rsa -in private.pem -outform PEM -pubout -out public.pem
openssl pkcs8 -topk8 -inform PEM -in private.pem -out private_key.pem -nocrypt
rm -f private.pem

We will use a Docker Image to cf push the CF Identity Token Service. The public and private keys will be set as environment variables here.

cf push cits -o ghcr.io/making/cf-identity-token-service:jvm --no-start
cf set-env cits JWT_PRIVATEKEY base64:$(cat private_key.pem | base64 -w0)
cf set-env cits JWT_PUBLICKEY base64:$(cat public.pem | base64 -w0)
cf start cits

Log in to the app containing the curl command using cf ssh.

Run the following command inside the container.

CITS_DOMAIN=cits.<apps_domain>
curl --cert $CF_INSTANCE_CERT --key $CF_INSTANCE_KEY -XPOST https://$CITS_DOMAIN/token -w '\n'

You should receive a JWT like the following:

eyJraWQiOiJmaXJzdCIsInR5cCI6IkpXVCIsImFsZyI6IlJTMjU2In0.eyJzdWIiOiI0Yjg0NzkzYy1mM2VhLTRhNTUtOTJiNy05NDI3MjZhYWMxNjM6Njc1NWIxOWQtYzU0My00ZTBjLWE0YjMtY2Q2ZTdjOWM2OGEzOjAyNzU2MTkxLWQ4NjktNDgwNi05NzE3LWE2ZWVjNTE0MmU4YSIsImF1ZCI6InN0cy5hbWF6b25hd3MuY29tIiwiYXBwX2d1aWQiOiIwMjc1NjE5MS1kODY5LTQ4MDYtOTcxNy1hNmVlYzUxNDJlOGEiLCJvcmdfZ3VpZCI6IjRiODQ3OTNjLWYzZWEtNGE1NS05MmI3LTk0MjcyNmFhYzE2MyIsImlzcyI6Imh0dHBzOi8vbG9jYWxob3N0Ojg0NDMiLCJleHAiOjE3MjQ0MzAwODcsInNwYWNlX2d1aWQiOiI2NzU1YjE5ZC1jNTQzLTRlMGMtYTRiMy1jZDZlN2M5YzY4YTMiLCJpYXQiOjE3MjQzODY4ODd9.MDvgofP3-NmvJKGn7TuHvdHQJmcQEexC4NEmwMPQNss1gyfoOwcXvUne7LPfSr8OHPc0QSX9L1i6r9nHOa-E9czWGbLYyDldXC_aIoPSOupypRFG2frprBYTDmHS5fooyRjzLf_2e4j6Qlwac8UNqRVEfVyPWH2uxrIK1VStaiP7NvW-q03AL11IFYK1g_S0hW9yWkG03hpbPuwl-kpQUC6T40MD4B4oORaDMwWvM53X3v5gnNyJ2A3N3inhSy2Wkkw5i7HXLXfxJ5HTl26EE0pEKVRswD-d14fP5yGUQjrfG57cePbdX3PvKCV2BtmDRbw9vqd9wxwYM6ZAPLFPUA%

Decode it. You should see a payload like the following.

$ echo "$(curl -s --cert $CF_INSTANCE_CERT --key $CF_INSTANCE_KEY -XPOST https://$CITS_DOMAIN/token | cut -d '.' -f2)==" | base64 -d | jq .
{
  "sub": "4b84793c-f3ea-4a55-92b7-942726aac163:34e1bb23-0e76-4aad-95d7-1abe3ea1dcd8:f60fab0d-2937-46f7-ad66-774d910826fd",
  "aud": "sts.amazonaws.com",
  "app_guid": "f60fab0d-2937-46f7-ad66-774d910826fd",
  "org_guid": "4b84793c-f3ea-4a55-92b7-942726aac163",
  "iss": "https://cits.apps.sandbox.aws.maki.lol",
  "exp": 1724876612,
  "space_guid": "34e1bb23-0e76-4aad-95d7-1abe3ea1dcd8",
  "iat": 1724833412
}

The sub claim is in the format <org_guid>:<space_guid>:<app_guid>.

Registering CF Identity Token Service as an OIDC Provider in AWS IAM

Register the deployed CF Identity Token Service as an ID provider in AWS IAM.

Refer to https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html#oidc-obtain-thumbprint to obtain the thumbprint.

# CITS_DOMAIN=cits.<apps_domain>
CITS_DOMAIN=$(cf curl /v3/apps/$(cf app cits --guid)/routes | jq -r '.resources[0].url')
FINGERPRINT=$(openssl s_client -servername $CITS_DOMAIN -showcerts -connect $CITS_DOMAIN:443 </dev/null 2>/dev/null | openssl x509 -fingerprint -sha1 -noout | sed 's/sha1 Fingerprint=//' | sed 's/://g')

Refer to https://docs.aws.amazon.com/IAM/latest/UserGuide/iam_example_iam_CreateOpenIdConnectProvider_section.html to register the OIDC Provider.

cat <<EOF > oidc-provider.json
{
    "Url": "https://$CITS_DOMAIN",
    "ClientIDList": [
        "sts.amazonaws.com"
    ],
    "ThumbprintList": [
        "$FINGERPRINT"
    ]
}
EOF

aws iam create-open-id-connect-provider --cli-input-json file://oidc-provider.json

Obtain the ARN of the registered OIDC provider.

OIDC_PROVIDER_ARN=$(aws iam list-open-id-connect-providers --query "OpenIDConnectProviderList[?ends_with(Arn, '$CITS_DOMAIN')].Arn" --output text)

Creating IAM Role

Create an IAM Role for the app under the current Org/Space.

# current org/space name
ORG_NAME=$(cat ~/.cf/config.json | jq -r .OrganizationFields.Name)
SPACE_NAME=$(cat ~/.cf/config.json | jq -r .SpaceFields.Name)

ORG_GUID=$(cf org $ORG_NAME --guid)
SPACE_GUID=$(cf space $SPACE_NAME --guid)

cat << EOF > cf-${ORG_NAME}-${SPACE_NAME}-trust-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "${OIDC_PROVIDER_ARN}"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringLike": {
                    "${CITS_DOMAIN}:sub": "${ORG_GUID}:${SPACE_GUID}:*",
                    "${CITS_DOMAIN}:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}
EOF

aws iam create-role --role-name cf-${ORG_NAME}-${SPACE_NAME} --assume-role-policy-document file://cf-${ORG_NAME}-${SPACE_NAME}-trust-policy.json

Creating Policy

Let's grant access rights to the DynamoDB table for this IAM role. The accessible tables will be limited to those with the prefix <org_name>-<space_name> in their names by creating the following policy.

export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export AWS_REGION=ap-northeast-1

cat <<EOF > cf-${ORG_NAME}-${SPACE_NAME}-policy-dynamo.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PrefixFullAccess",
            "Effect": "Allow",
            "Action": "dynamodb:*",
            "Resource": "arn:aws:dynamodb:${AWS_REGION}:${AWS_ACCOUNT_ID}:table/${ORG_NAME}-${SPACE_NAME}-*"
        }
    ]
}
EOF

aws iam put-role-policy --role-name cf-${ORG_NAME}-${SPACE_NAME} --policy-name dynamo-prefix-full-access-${ORG_NAME}-${SPACE_NAME} --policy-document file://cf-${ORG_NAME}-${SPACE_NAME}-policy-dynamo.json

Accessing with AWS CLI

Let's try accessing DynamoDB from the AWS CLI using this IAM role.

Deploy a container image that includes the AWS CLI with cf push, and log in using cf ssh.

# CITS_DOMAIN=cits.<apps_domain>
CITS_DOMAIN=$(cf curl /v3/apps/$(cf app cits --guid)/routes | jq -r '.resources[0].url')

cf push aws-cli -m 128m -o public.ecr.aws/aws-cli/aws-cli --no-route -u process --no-manifest -c 'sleep infinity' --no-start
cf set-env aws-cli AWS_REGION ap-northeast-1
cf set-env aws-cli AWS_ROLE_ARN $(aws iam get-role --role-name cf-${ORG_NAME}-${SPACE_NAME} --query 'Role.Arn' --output text)
cf set-env aws-cli AWS_WEB_IDENTITY_TOKEN_FILE /tmp/token
cf set-env aws-cli AWS_ROLE_SESSION_NAME cf-demo
cf set-env aws-cli CITS_DOMAIN $CITS_DOMAIN
cf start aws-cli
cf ssh aws-cli

export PATH=$PATH:/usr/local/bin
curl -s -XPOST https://${CITS_DOMAIN}/token --cert ${CF_INSTANCE_CERT} --key ${CF_INSTANCE_KEY} -o ${AWS_WEB_IDENTITY_TOKEN_FILE}
aws sts get-caller-identity

You should receive a JSON response like the following.

{
    "UserId": "****:cf-demo",
    "Account": "****",
    "Arn": "arn:aws:sts::****:assumed-role/cf-<org_name>-<space_name>/cf-demo"
}

Now, let's access DynamoDB. Continue running the following commands inside the container.

TABLENAME=<org_name>-<space_name>-movie # changeme
aws dynamodb create-table \
    --table-name ${TABLENAME} \
    --attribute-definitions \
        AttributeName=movieId,AttributeType=S \
        AttributeName=title,AttributeType=S \
        AttributeName=genre,AttributeType=S \
    --key-schema \
        AttributeName=movieId,KeyType=HASH \
    --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 \
    --global-secondary-indexes \
        '[
            {
                "IndexName": "title-index",
                "KeySchema": [{"AttributeName":"title","KeyType":"HASH"}],
                "Projection": {"ProjectionType":"ALL"},
                "ProvisionedThroughput": {"ReadCapacityUnits": 5, "WriteCapacityUnits": 5}
            },
            {
                "IndexName": "genre-index",
                "KeySchema": [{"AttributeName":"genre","KeyType":"HASH"}],
                "Projection": {"ProjectionType":"ALL"},
                "ProvisionedThroughput": {"ReadCapacityUnits": 5, "WriteCapacityUnits": 5}
            }
        ]' | cat

echo "wait until the table is created...."
sleep 20

aws dynamodb put-item \
    --table-name ${TABLENAME} \
    --item \
        '{
            "movieId": {"S": "1e7b56f3-0c65-4fa6-9a32-6d0a65fbb3a5"},
            "title": {"S": "Inception"},
            "releaseYear": {"N": "2010"},
            "genre": {"S": "Science Fiction"},
            "rating": {"N": "8.8"},
            "director": {"S": "Christopher Nolan"}
        }'

aws dynamodb put-item \
    --table-name ${TABLENAME} \
    --item \
        '{
            "movieId": {"S": "2a4b6d72-789b-4a1a-9c7f-74e5a8f7676d"},
            "title": {"S": "The Matrix"},
            "releaseYear": {"N": "1999"},
            "genre": {"S": "Action"},
            "rating": {"N": "8.7"},
            "director": {"S": "The Wachowskis"}
        }'

aws dynamodb put-item \
    --table-name ${TABLENAME} \
    --item \
        '{
            "movieId": {"S": "3f6c8f74-2e6a-48e9-a07f-034f8a67b9e6"},
            "title": {"S": "Interstellar"},
            "releaseYear": {"N": "2014"},
            "genre": {"S": "Adventure"},
            "rating": {"N": "8.6"},
            "director": {"S": "Christopher Nolan"}
        }'

aws dynamodb scan --table-name ${TABLENAME} | cat

You should see that the table has been created and data has been inserted. Now you can access AWS resources using IAM roles with STS.

If you try to access a table without permissions, you will be denied as expected.

bash-4.2# aws dynamodb scan --table-name movie
An error occurred (AccessDeniedException) when calling the Scan operation: User: arn:aws:sts::****:assumed-role/cf-****-****/cf-demo is not authorized to perform: dynamodb:Scan on resource: arn:aws:dynamodb:ap-northeast-1:****:table/movie because no identity-based policy allows the dynamodb:Scan action

Using a Sidecar to Obtain Tokens

In the previous example, token acquisition was done manually. When deploying a web application, we want this token acquisition and renewal to happen automatically in the background. Therefore, we will implement the token acquisition process using a sidecar.

We will deploy the Spring Boot app created in this article as an app that accesses the DynamoDB table created earlier.

git clone https://github.com/making/demo-scylladb-alternator
cd demo-scylladb-alternator

To use AWS STS, we need to add the SDK, so we will add the following dependency to pom.xml.

        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>sts</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>commons-logging</groupId>
                    <artifactId>commons-logging</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

Build the app.

./mvnw clean package

Create the following manifest.yml.

cat <<'EOF' > manifest.yml
applications:
- name: demo-dynamodb
  buildpacks:
  - java_buildpack_offline
  memory: 768m
  path: ./target/demo-scylla-alternator-0.0.1-SNAPSHOT.jar
  env:
    JBP_CONFIG_OPEN_JDK_JRE: '{jre: {version: 17.+}}'
    SPRING_CLOUD_AWS_CREDENTIALS_STS_WEBIDENTITYTOKENFILE: /tmp/token
    SPRING_CLOUD_AWS_CREDENTIALS_STS_ROLEARN: arn:aws:iam::CHANGE_ME:role/cf-${vcap.application.organization_name}-${vcap.application.space_name}
    SPRING_CLOUD_AWS_CREDENTIALS_STS_ROLESESSIONNAME: ${vcap.application.application_name}
    SPRING_CLOUD_AWS_CREDENTIALS_STS_ASYNCCREDENTIALSUPDATE: true
    SPRING_CLOUD_AWS_DYNAMODB_TABLEPREFIX: ${vcap.application.organization_name}-${vcap.application.space_name}-
    SPRING_CLOUD_AWS_DYNAMODB_ENDPOINT: ""
    AWS_REGION: ap-northeast-1
  sidecars:
  - name: issue-token
    command: |
      APP_DOMAIN=$(echo $VCAP_APPLICATION | jq -r '.application_uris[0] | split(".")[1:] | join(".")')
      while true;do
        set +e
        curl -s -XPOST https://cits.${APP_DOMAIN}/token \
          --key $CF_INSTANCE_KEY \
          --cert $CF_INSTANCE_CERT \
          -o ${SPRING_CLOUD_AWS_CREDENTIALS_STS_WEBIDENTITYTOKENFILE} \
          -w 'status:%{http_code}\n'
        set -e
        sleep 7200
      done
    memory: 16m
    process_types:
    - web
EOF
sed -i.bk "s/CHANGE_ME/$(aws sts get-caller-identity --query Account --output text)/" manifest.yml
rm -f manifest.yml.bk

Set the prefix for the tables you want to access in DynamoDB with SPRING_CLOUD_AWS_DYNAMODB_TABLEPREFIX.

Deploy this.

cf push

Check the logs with the following command.

cf logs demo-dynamodb --recent

You should see that the sidecar is able to issue tokens.

...
  2024-08-28T18:07:55.31+0900 [APP/PROC/WEB/SIDECAR/ISSUE-TOKEN/0] OUT status:200
...

Access the app.

DEMO_DOMAIN=$(cf curl /v3/apps/$(cf app demo-dynamodb --guid)/routes | jq -r '.resources[0].url')

curl https://$DEMO_DOMAIN/movies

You should receive a JSON response confirming that the app can successfully access DynamoDB.

[{"movieId":"2a4b6d72-789b-4a1a-9c7f-74e5a8f7676d","title":"The Matrix","releaseYear":1999,"genre":"Action","rating":8.7,"director":"The Wachowskis"},{"movieId":"3f6c8f74-2e6a-48e9-a07f-034f8a67b9e6","title":"Interstellar","releaseYear":2014,"genre":"Adventure","rating":8.6,"director":"Christopher Nolan"},{"movieId":"1e7b56f3-0c65-4fa6-9a32-6d0a65fbb3a5","title":"Inception","releaseYear":2010,"genre":"Science Fiction","rating":8.8,"director":"Christopher Nolan"}]

Except for adding the SDK, we have confirmed that we can access AWS resources using AWS STS without modifying the app.

We have confirmed that by converting Cloud Foundry's Instance Identity into a Web Identity Token, we can access AWS resources using IAM roles. This mechanism can be used not only with AWS but also for authentication with other cloud services or HashiCorp Vault, similar to GitHub Actions' OIDC integration.