Unable to configure AWS secrets in rabbitmq.conf

[lukebakken]

#I'm using the 3.7.0 generic unix package with Erlang 20.1.7 on Arch Linux

I saved rabbitmq.conf.txt as ./etc/rabbitmq/rabbitmq.conf relative to where I unpacked the generic-unix archive:

cluster_formation.proxy.http_proxy = http://localhost:3128
cluster_formation.proxy.https_proxy = http://localhost:3128
proxy_protocol = true

cluster_formation.peer_discovery_backend = rabbit_peer_discovery_aws
cluster_formation.aws.region = us-east-1
cluster_formation.aws.access_key_id = ANIDEXAMPLE
cluster_formation.aws.secret_key = WjalrxuTnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY
cluster_formation.aws.use_autoscaling_group = true
cluster_formation.aws.use_private_ip = true

log.file.level = debug
log.console.level = debug

When you start up the broker, you will see the following in the log:

2017-12-12 14:47:21.903 [info] <0.195.0> Node database directory at /home/lbakken/issues/rmq-generic-unix/rabbitmq_server-3.7.0/var/lib/rabbitmq/mnesia/rabbit@shostakovich is empty. Assuming we need to join an existing cluster or initialise from scratch...
2017-12-12 14:47:21.903 [info] <0.195.0> Configured peer discovery backend: rabbit_peer_discovery_aws
2017-12-12 14:47:21.903 [info] <0.195.0> Will try to lock with peer discovery backend rabbit_peer_discovery_aws
2017-12-12 14:47:21.903 [info] <0.195.0> Peer discovery backend rabbit_peer_discovery_aws does not support registration, skipping randomized startup delay.
2017-12-12 14:47:21.903 [debug] <0.195.0> Setting AWS region to "us-east-1"
2017-12-12 14:47:21.903 [debug] <0.220.0> Supervisor inet_gethost_native_sup started undefined at pid <0.221.0>
2017-12-12 14:47:21.903 [debug] <0.60.0> Supervisor kernel_safe_sup started inet_gethost_native:start_link() at pid <0.220.0>
2017-12-12 14:47:22.106 [error] <0.180.0> Failed to retrieve AWS credentials: undefined
2017-12-12 14:48:10.173 [info] <0.232.0> RabbitMQ hasn't finished starting yet. Waiting for startup to finish before stopping...

The important message is Failed to retrieve AWS credentials: undefined

It appears that the keys in the generated config file may not be what the AWS backend is expecting

rabbitmq.config-generated.txt:

[{rabbit,
     [{log,[{file,[{level,debug}]},{console,[{level,debug}]}]},
      {proxy_protocol,true},
      {cluster_formation,
          [{peer_discovery_backend,rabbit_peer_discovery_aws},
           {peer_discovery_aws,
               [{aws_ec2_region,"us-east-1"},
                {aws_use_private_ip,true},
                {aws_access_key,"ANIDEXAMPLE"},
                {aws_secret_key,"WjalrxuTnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY"},
                {aws_autoscaling,true}]},
           {proxy,
               [{http_proxy,"http://localhost:3128"},
                {https_proxy,"http://localhost:3128"}]}]}]}].

[michaelklishin]

I'm still investigating but so far it looks like the config is generated correctly but rabbitmq_aws:set_credentials/2 doesn't seem to do what I believe it should.

[michaelklishin]

So rabbit_aws:load_credentials/2 ignores config file-provided values because rabbitmq_aws_config:credentials/0 and rabbitmq_aws_config:credentials/1 do. However, rabbitmq_aws:set_credentials/2 should nonetheless override configured state and so far I don't think it does. Exporting more functions and adding more logging for easier debugging.

[michaelklishin]

So the

2017-12-13 20:24:42.976 [error] <0.370.0> Failed to retrieve AWS credentials: undefined

and similar lines mean that an AWS-specific endpoint cannot be reached, which will happen unless you run rabbit_peer_discovery_aws:list_nodes/0 on an EC2 instance.

Additional logging suggests that the configuration is applied successfully. I'm inclined to think that the report runs into rabbitmq/rabbitmq-peer-discovery-common#5.

[michaelklishin]

Also note that when debugging peer discovery plugins it is important to wipe node's directory between starts. Nodes that are already cluster members will try to connect to their known peers.

Invoking list_nodes/0 from the shell is one easy way of triggering a discovery and seeing its return value.

[lukebakken]

Thanks for following-up. It turns out I didn't grep my code correctly and missed references to aws_access_key and aws_secret_key in deps/rabbitmq_peer_discovery_aws.

[mkelderm]

Hi we have a simular issue:

[info] <0.229.0> Peer discovery backend rabbit_peer_discovery_aws does not support registration, skipping registration.

On our RabbitMQ EC2 instance an IAM role is attached with the coorect policy. The rabbitMQ config is:

cluster_formation.peer_discovery_backend = rabbit_peer_discovery_aws
cluster_formation.aws.region = eu-west-1
cluster_formation.aws.use_autoscaling_group = true
cluster_formation.aws.use_private_ip = true

Did anyone face this issue?

[michaelklishin]

The log message above simply states that the backend does not use registration (or rather it happens out of band). This is not an issue or a reason to be concerned with. DNS and classic config backends do not support registration either, for example.

This is not a support forum. Please post more details about what you are trying to do and what the nodes log at debug level to rabbitmq-users.