Support IAM roles for ECS tasks

[chris-peterson]

When running RabbitMQ in ECS, an optional configuration is to assign an IAM role.

The peer discovery plugin does not currently support this use-case. I observed this error:

2020-03-23 17:11:24.558 [debug] <0.285.0> Started rabbitmq_aws
2020-03-23 17:11:24.558 [debug] <0.285.0> Will use AWS access key of 'undefined'
2020-03-23 17:11:24.558 [debug] <0.285.0> Setting AWS region to "us-west-2"
2020-03-23 17:11:24.594 [debug] <0.285.0> Fetched EC2 instance ID from "http://169.254.169.254/latest/meta-data/instance-id": "i-xxxx"
2020-03-23 17:11:24.691 [error] <0.285.0> Error fetching autoscaling group instance list: "Forbidden"

Enriching with cloudtrail logs, I learned that the API call was happening under the context of the instance profile assigned to the underlying EC2 instance rather than the IAM role assigned to the task:

{
    "recipientAccountId": "xxxx",
    "eventSource": "autoscaling.amazonaws.com",
    "eventName": "DescribeAutoScalingInstances",
    "eventTime": "2020-03-23T16:54:56Z",
    "userIdentity": {
        "arn": "arn:aws:sts::xxxx:assumed-role/myapp-ec2-role/i-yyyy",
        "principalId": "zzzz:i-yyyy",
        "accountId": "xxxx",
        "sessionContext": {
            "attributes": {
                "creationDate": "2020-03-23T16:47:46Z",
                "mfaAuthenticated": "false"
            },
            "sessionIssuer": {
                "userName": "myapp-ec2-role",
                "arn": "arn:aws:iam::xxxx:role/myapp/ec2-role",
                "type": "Role",
                "principalId": "zzzz",
                "accountId": "xxxx"
            }
        },
        "type": "AssumedRole",
        "accessKeyId": "wwww"
    },
    "errorCode": "AccessDenied",
    "eventType": "AwsApiCall",
    "requestID": "5fa37593-4af3-4194-b692-f6a2a71fb51c",
    "requestParameters": null,
    "errorMessage": "User: arn:aws:sts::xxxx:assumed-role/myapp-ec2-role/i-yyyy is not authorized to perform: autoscaling:DescribeAutoScalingInstances",
    "awsRegion": "us-west-2",
    "eventID": "af8970b4-67dd-446d-9444-e96032fea27a",
    "eventVersion": "1.05",
    "responseElements": null,
    "sourceIPAddress": "0.0.0.0"
}

Version information

• RabbitMQ 3.8.1
• Erlang 22.1.8
• rabbitmq-peer-discovery-aws 3.8.3

Suggested Implementation

If the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable is set, then obtain access_key_id and secret_key from the task metadata endpoint, e.g.

curl -s "http://169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"

{
  "RoleArn": "arn:aws:iam::xxxx:role/myapp/myapp-task-role",
  "AccessKeyId": "yyyy",
  "SecretAccessKey": "zzzz",
  "Token": "wwww",
  "Expiration": "2020-03-23T23:54:01Z"
}

Workaround

My current workaround is to create an IAM policy that is a union of myapp-ec2-role and myapp-task-role and use this for both the EC2 instance profile as well as the task IAM role.

[michaelklishin]

I'm afraid such service-specific scenarios are out of scope for this plugin. We'd consider a documentation PR for rabbitmq/rabbitmq-website.

Testing (and debugging failures of) even more scenarios in which this plugin can potentially be used would not be worth the effort for our small team.