Fixed RabbitMQ installation when using custom variables (#67)

* Ansible lint (#46)

* resolved conflicts

* Fix ansible-lint issues

* Resolved conflict

* Os (#45)


* Remove yum_update and remove the task to update the current yum packages

---------

Co-authored-by: $GITHUB_ACTOR <[email protected]>

* Fix ansible lint issues in redis auth role (#48)

* Fix lint issues in redis_auth role

* Fix ansible lint issues in platform_app_artifact role (#49)

* Fix ansible lint issues in python role (#51)

* Fix ansible lint issues in rabbitmq_ssl role (#50)

* Add RabbitMQ custom configuration tasks

* Move rabbitmq cluster role tasks to rabbitmq role
Remove rabbitmq cluster role
Update docs
Fix hard-coded rabbitmq ports in platform role
Move rabbitmq ports to common vars role
Add ability to set rabbitmq disk_free_limit.absolute

* Support non-standard mongo port (#65)

* resolved conflicts

* Adding port var to mongo tasks to support non-standard ports

* Fixed typo

* Resolved code review items

* Fixed redis installation when using custom variables (#66)

* Create redis_varlib.te

This file will add SELinux context to add log, data directories in a different place from the default

* Update main.yml to support redis_data_dir, redis_log_dir, redis_pid_dir, different port

Fix support to:

* redis_data_dir, 
* redis_log_dir, 
* redis_pid_dir, 
* different port

* Remove user,group and data,log directories creation.

Remove user,group and data,log directories creation.
Those items now are created in main.yml tasks

* Fixed pid dir, and announce port

Fixed pid dir, and announce port

* Change the installation source from remi_repo to source 

Change the installation source from remi_repo to source

* Added redis_port and sentinel_port variables

Added redis_port and sentinel_port variables

* Fixed the hardcoded redis ports to variable

Fixed the hardcoded redis ports to variable

* Fixed source install

Fixed source install

* Fixed conditions rules for execute directories creation

Fixed conditions rules for execute directories creation

* Optimize the code in announce items

Optimize the code in announce items

* Removed sentinel port, moved for common vars

Removed sentinel port, moved for common vars

* Fixed the hardcoded redis port to a variable

Fixed the hardcoded redis port to a variable

* Fixed hardcoded ports to variable

Fixed hardcoded ports to variable

* Fix sentinel variable name

Fix sentinel variable name

* Fixed trailing spaces

Fixed trailing spaces

* Fix ansible lint issue in redis

* Fix spacing in RabbitMQ guide

---------

Co-authored-by: Steven Schattenberg <[email protected]>
Co-authored-by: Nick-Andreano <[email protected]>
Co-authored-by: $GITHUB_ACTOR <[email protected]>
Co-authored-by: Marcos Dias <[email protected]>

Author: 5 people

docs/iap_guide.md

@@ -55,6 +55,10 @@ The variables in this section may be overridden in the inventory in the `platfor
 | :------- | :---- | :--- | :---------- | :------------ | :--------
 | `iap_bin_file` | `platform` | String | The name of the IAP bin file. | N/A | Yes*
 | `iap_tar_file` | `platform` | String | The name of the IAP tar file. | N/A | Yes*
+| `rabbit_svc_url` | `platform` | String | This variable defines the rabbit service url to use when connecting to an externally provided RabbitMQ cluster. It is intended to be used when the architecture demands that rabbit be hosted elsewhere such as when using AmazonMQ or if the demands of an organization require some other external rabbit solution, like a shared service. | N/A | No
+| `redis_svc_url` | `platform` | String | This variable defines the redis service url to use when connecting to an externally provided redis cluster. It is intended to be used when the architecture demands that redis be hosted elsewhere such as when using Elasticache or if the demands of an organization require some other external redis solution, like a shared service. | N/A | No
+| `mongobdb_svc_url_itential` | `platform` | String | This variable defines the mongodb connection string to use when connecting to the "itential" database. It is intended to be used when the architecture demands that mongo be hosted elsewhere such as when using Mongo Atlas or if the demands of an organization require some other external mongo solution, like a shared service. | N/A | No
+| `mongobdb_svc_url_localaaa` | `platform` | String | This variable defines the mongodb connection string to use when connecting to the "LocalAAA" database. It is intended to be used when the architecture demands that mongo be hosted elsewhere such as when using Mongo Atlas or if the demands of an organization require some other external mongo solution, like a shared service. | N/A | No
 
 Either `iap_bin_file` or `iap_tar_file` must be defined in the inventory, but not both.
 

docs/mongodb_guide.md

@@ -43,8 +43,6 @@ The variables in this section are configured in the inventory in the `all` group
 | :------- | :---- | :--- | :---------- | :------------ | :---------
 | `iap_release` | `all` | Fixed-point | Designates the IAP major version. | N/A | Yes
 | `mongo_root_ca_file_source` | `all` | String | The name of the MongoDB Root CA file.| N/A | No
-| `mongobdb_svc_url_itential` | `all` | String | This variable defines the mongodb connection string to use when connecting to the "itential" database. It is intended to be used when the architecture demands that mongo be hosted elsewhere such as when using Mongo Atlas or if the demands of an organization require some other external mongo solution, like a shared service. | N/A | No
-| `mongobdb_svc_url_localaaa` | `all` | String | This variable defines the mongodb connection string to use when connecting to the "LocalAAA" database. It is intended to be used when the architecture demands that mongo be hosted elsewhere such as when using Mongo Atlas or if the demands of an organization require some other external mongo solution, like a shared service. | N/A | No
 
 The `iap_release` must be defined in the inventory.  This variable, along with the OS major version, is used to determine the static variables.
 

docs/rabbitmq_guide.md

@@ -2,8 +2,7 @@
 
 The playbook and roles in this section install and configure RabbitMQ for the Itential Automation Platform.  There are currently three RabbitMQ-related roles:
 
-* `rabbitmq` – Installs RabbitMQ and performs a base configuration.
-* `rabbitmq_cluster` – Configures RabbitMQ clustering.
+* `rabbitmq` – Installs RabbitMQ, performs a base configuration and configures clustering.
 * `rabbitmq_ssl` – Configures RabbitMQ SSL.
 
 **ⓘ Note:**
@@ -15,19 +14,17 @@ This role is used when installing IAP version 2023.1 and older.
 
 The `rabbitmq` role performs a base install of RabbitMQ including any OS packages required. It installs the appropriate version Erlang.  It creates the appropriate Linux users, directories, log files, and systemd services. It will create the required RabbitMQ users with a default password (see table). It will start the rabbitmq-server service when complete.
 
+The `rabbitmq` also role performs the steps to run RabbitMQ as a cluster of nodes.  It assumes a cluster of 3 and that the first host defined in the inventory will be used as the primary. It will modify the RabbitMQ config file to enable the cluster. It will write the hostname to each RabbitMQ node’s host file (RabbitMQ clustering requires DNS resolution). It creates the required Erlang cookie used by the RabbitMQ nodes to join a cluster. It invokes each RabbitMQ node to join the cluster. It enables queue mirroring. It will restart the rabbitmq-server service when complete.
+
+More info on rabbit cluster: https://www.rabbitmq.com/clustering.html
+
 | User Name | Default Password | Description
 | :-------- | :--------------- | :----------
 | admin | admin | The admin user with root permissions in this RabbitMQ install.
 | itential | itential | The itential user used by IAP to connect.
 
 :::(Warning) (⚠ Warning: ) It is assumed that these default passwords will be changed to meet more rigorous standards. These are intended to be defaults strictly used just for ease of the installation.  It is highly recommended that sensitive data be encrypted using Ansible Vault.
 
-## RabbitMQ Cluster
-
-The `rabbitmq_cluster` role performs the steps to run RabbitMQ as a cluster of nodes.  It assumes a cluster of 3 and that the first host defined in the inventory will be used as the primary. It will modify the RabbitMQ config file to enable the cluster. It will write the hostname to each RabbitMQ node’s host file (RabbitMQ clustering requires DNS resolution). It creates the required Erlang cookie used by the RabbitMQ nodes to join a cluster. It invokes each RabbitMQ node to join the cluster. It enables queue mirroring. It will restart the rabbitmq-server service when complete.
-
-More info on rabbit cluster: https://www.rabbitmq.com/clustering.html
-
 ## RabbitMQ SSL Role
 
 The `rabbitmq_ssl` performs the steps to require TLS when communicating with the RabbitMQ server. It uploads the certificates to the correct location. It is NOT responsible for making the certificates. It will make a number of edits to the RabbitMQ config to enable TLS. It will restart the rabbitmq-server service when complete.
@@ -47,7 +44,6 @@ The variables in this section are configured in the inventory in the `all` group
 | Variable | Group | Type | Description | Default Value | Required?
 | :------- | :---- | :--- | :---------- | :------------ | :--------
 | `iap_release` | `all` | Fixed-point | Designates the IAP major version. | N/A | Yes
-| `rabbit_svc_url` | `all` | String | This variable defines the rabbit service url to use when connecting to an externally provided RabbitMQ cluster. It is intended to be used when the architecture demands that rabbit be hosted elsewhere such as when using AmazonMQ or if the demands of an organization require some other external rabbit solution, like a shared service. | N/A | No
 
 The `iap_release` must be defined in the inventory.  This variable, along with the OS major version, is used to determine the static variables.
 
@@ -59,6 +55,8 @@ The following table lists the default variables that are shared between the Rabb
 
 | Variable | Group | Type | Description | Default Value
 | :------- | :---- | :--- | :---------- | :------------
+| `rabbitmq_port` | `all` | Integer | The RabbitMQ listen port. | `5672`
+| `rabbitmq_mgt_console_port` | `all` | Integer | The RabbitMQ management console listen port. | `15672`
 | `rabbitmq_vhost` | `all` | String | The name of the RabbitmMQ vhost. | `iap`
 | `rabbitmq_ssl` | `all` | Boolean | Flag to enable SSL. <br>`true` - enable HTTPS when connecting to rabbit, disable HTTP. <br>`false` - disable HTTPS when connecting to rabbit, enable HTTP. | `false`
 
@@ -70,9 +68,10 @@ The following table lists the default variables located in `roles/rabbitmq/defau
 
 | Variable | Group | Type | Description | Default Value
 | :------- | :---- | :--- | :---------- | :------------
-| `rabbitmq_config` | `rabbitmq` | String | The location of the RabbitMQ configuration file. | `/etc/rabbitmq/rabbitmq.conf`
-| `rabbitmq_port` | `rabbitmq` | Integer | The default listen port. | `5672`
-| `rabbitmq_mgt_console_port` | `rabbitmq` | Integer | The default management console listen port. | `15672`
+| `rabbitmq_home_dir` | `rabbitmq` | String | The RabbitMQ home directory. | `/var/lib/rabbitmq`
+| `rabbitmq_log_dir` | `rabbitmq` | String | The RabbitMQ log directory. | `/var/log/rabbitmq`
+| `rabbitmq_log_file` | `rabbitmq` | String | The RabbitMQ log file name. | `rabbit.log`
+| `rabbitmq_log_file_level` | `rabbitmq` | String | The RabbitMQ log level. | `info`
 | `rabbitmq_owner` | `rabbitmq` | String | The RabbitMQ Linux user. | `rabbitmq`
 | `rabbitmq_group` | `rabbitmq` | String | The RabbitMQ Linux group. | `rabbitmq`
 | `rabbitmq_bind_ipv6` | `rabbitmq` | Boolean | Flag to enable binding to IPv6. | `true`
@@ -81,21 +80,16 @@ The following table lists the default variables located in `roles/rabbitmq/defau
 | `rabbitmq_password` | `rabbitmq` | String | The default password for the RabbitMQ user. | `itential`
 | `rabbitmq_admin_user` | `rabbitmq` | String | The admin user with root permissions in this RabbitMQ install. | `admin`
 | `rabbitmq_admin_password` | `rabbitmq` | String | The default password for the admin user. | `admin`
-
-:::(Warning) (⚠ Warning: ) It is assumed that these default passwords will be changed to meet more rigorous standards. These are intended to be defaults strictly used just for ease of the installation.  It is highly recommended that sensitive data be encrypted using Ansible Vault.
-
-## RabbitMQ Cluster Role Variables
-
-The variables in this section may be overridden in the inventory in the `rabbmitmq` group vars.
-
-The following table lists the default variables located in `roles/rabbitmq_cluster/defaults/main.yml`.
-
-| Variable | Group | Type | Description | Default Value
-| :------- | :---- | :--- | :---------- | :------------
 | `rabbitmq_cluster` | `rabbitmq` | Boolean | Flag to enable clustering. | `false`
 | `rabbitmq_erlang_cookie` | `rabbitmq` | String | The location of the Erlang cookie file. | `/var/lib/rabbitmq/.erlang.cookie`
 | `rabbitmq_cluster_port` | `rabbitmq` | Integer | The default RabbitMQ cluster listen port. | `25672`
 | `rabbitmq_epmd_port` | `rabbitmq` | Integer | The default RabbitMQ Erlang Port Mapping Daemon listen port. | `4369`
+| `rabbitmq_distribution_buffer_size` | `rabbitmq` | Integer | Inter-node connections use a buffer for data pending to be sent. | N/A
+| `rabbit_max_msg_size` | `rabbitmq` | Integer | The largest allowed message payload size in bytes. | N/A
+| `rabbit_total_mem_available_override` | `rabbitmq` | Integer | Makes it possible to override the total amount of memory availabl in bytes, as opposed to inferring it from the environment using OS-specific means. | N/A
+| `disk_free_limit_absolute` | `rabbitmq` | Integer | Disk free limit in bytes. | N/A
+
+:::(Warning) (⚠ Warning: ) It is assumed that these default passwords will be changed to meet more rigorous standards. These are intended to be defaults strictly used just for ease of the installation.  It is highly recommended that sensitive data be encrypted using Ansible Vault.
 
 ## RabbitMQ SSL Role Variables
 
@@ -194,7 +188,6 @@ ansible-playbook itential.deployer.rabbitmq -i <inventory>
 You can also run select RabbitMQ roles by using the following tags:
 
 * `rabbitmq_install`
-* `rabbitmq_cluster`
 * `rabbitmq_ssl`
 
 To execute only the `rabbitmq` role (skipping the RabbitMQ Cluster and RabbitMQ SSL roles), run the `itential.deployer.rabbitmq` playbook with the `rabbitmq_install` tag:
@@ -203,12 +196,6 @@ To execute only the `rabbitmq` role (skipping the RabbitMQ Cluster and RabbitMQ
 ansible-playbook itential.deployer.rabbitmq -i <inventory> --tags rabbitmq_install
 ```
 
-To execute only the RabbitMQ Cluster role (skipping the RabbitMQ and RabbitMQ SSL roles), run the `itential.deployer.rabbitmq` playbook with the `rabbitmq_cluster` tag:
-
-```
-ansible-playbook itential.deployer.rabbitmq -i <inventory> --tags rabbitmq_cluster
-```
-
 To execute only the RabbitMQ SSL role (skipping the RabbitMQ and RabbitMQ Cluster roles), run the `itential.deployer.rabbitmq` playbook with the `rabbitmq_ssl` tag:
 
 ```

docs/redis_guide.md

@@ -46,7 +46,6 @@ The variables in this section are configured in the inventory in the `all` group
 | Variable | Group | Type | Description | Default Value | Required?
 | :------- | :---- | :--- | :---------- | :------------ | :--------
 | `iap_release` | `all` | Fixed-point | Designates the IAP major version. | N/A | Yes
-| `redis_svc_url` | `all` | String | This variable defines the redis service url to use when connecting to an externally provided redis cluster. It is intended to be used when the architecture demands that redis be hosted elsewhere such as when using Elasticache or if the demands of an organization require some other external redis solution, like a shared service. | N/A | No
 
 The `iap_release` must be defined in the inventory.  This variable, along with the OS major version, is used to determine the static variables.
 

playbooks/install_active_standby.yml

@@ -39,15 +39,6 @@
         - rabbitmq
         - rabbitmq_install
 
-    # Add on any Rabbit features, HA, auth, etc...
-    # Configure Rabbit in a cluster
-    # https://www.rabbitmq.com/clustering.html
-    - role: itential.deployer.rabbitmq_cluster
-      when: rabbitmq_cluster | bool
-      tags:
-        - rabbitmq
-        - rabbitmq_cluster
-
     # Configure rabbit to only accept SSL connections
     - role: itential.deployer.rabbitmq_ssl
       when: rabbitmq_ssl | bool

playbooks/rabbitmq.yml

@@ -15,15 +15,6 @@
         - rabbitmq
         - rabbitmq_install
 
-    # Add on any Rabbit features, HA, auth, etc...
-    # Configure Rabbit in a cluster
-    # https://www.rabbitmq.com/clustering.html
-    - role: itential.deployer.rabbitmq_cluster
-      when: rabbitmq_cluster | bool
-      tags:
-        - rabbitmq
-        - rabbitmq_cluster
-
     # Configure rabbit to only accept SSL connections
     - role: itential.deployer.rabbitmq_ssl
       when: rabbitmq_ssl | bool

roles/common_vars/defaults/main/rabbitmq.yml

@@ -1,6 +1,12 @@
 # Copyright (c) 2024, Itential, Inc
 # GNU General Public License v3.0+ (see LICENSE or https://www.gnu.org/licenses/gpl-3.0.txt)
 ---
+# The RabbitMQ port
+rabbitmq_port: "{{ rabbitmq_default_port }}"
+
+# The RabbitMQ management console port
+rabbitmq_mgt_console_port: "{{ rabbitmq_default_mgt_console_port }}"
+
 # The name of the vhost to use so we do not use the default rabbitmq vhost
 rabbitmq_vhost: iap
 

roles/common_vars/vars/main/rabbitmq.yml

@@ -0,0 +1,6 @@
+# Copyright (c) 2024, Itential, Inc
+# GNU General Public License v3.0+ (see LICENSE or https://www.gnu.org/licenses/gpl-3.0.txt)
+---
+# The RabbitMQ port defaults
+rabbitmq_default_port: 5672
+rabbitmq_default_mgt_console_port: 15672

roles/platform/templates/profile.j2

@@ -9,20 +9,17 @@
 {% set rabbit_hosts = [] %}
 {% if rabbit_svc_url is defined %}
   {{ rabbit_hosts.append(rabbit_svc_url) }}
-  {% set rabbit_port = 5671 %}
   {% set rabbit_protocol = "amqps" %}
   {% set rabbit_certPath = "" %}
   {% set rabbit_keyPath = "" %}
   {% set rabbit_passphrase = "guest" %}
   {% set rabbit_caPath = "" %}
 {% elif groups["rabbitmq"] is defined and groups["rabbitmq"] | length > 0 %}
-  {% set rabbit_port = 5672 %}
   {% set rabbit_protocol = "amqp" %}
   {% for rabbit in groups[vars.rabbit_group_name] %}
     {{ rabbit_hosts.append( hostvars[ rabbit ].inventory_hostname ) }}
   {% endfor %}
   {% if rabbitmq_ssl | bool -%}
-    {% set rabbit_port = 5671 %}
     {% set rabbit_protocol = "amqps" %}
     {% set rabbit_certPath = "" %}
     {% set rabbit_keyPath = "" %}
@@ -200,7 +197,7 @@
   },
 {% if rabbit_hosts is defined and rabbit_hosts | length > 0 -%}
   "rabbitmq": {
-    "adminPort": 15672,
+    "adminPort": {{ rabbitmq_mgt_console_port }},
     "caPath": "",
     "certPath": "",
     "frameMax": 0,
@@ -210,7 +207,7 @@
     "locale": "en_US",
     "passphrase": "guest",
     "password": "{{ rabbitmq_password }}",
-    "port": {{ rabbit_port }},
+    "port": {{ rabbitmq_port }},
     "protocol": "{{ rabbit_protocol }}",
     "username": "{{ rabbitmq_user }}",
     "vhost": "{{ rabbitmq_vhost }}"

roles/rabbitmq/defaults/main.yml

@@ -1,26 +1,34 @@
 # Copyright (c) 2024, Itential, Inc
 # GNU General Public License v3.0+ (see LICENSE or https://www.gnu.org/licenses/gpl-3.0.txt)
 ---
-# The Location of the config rabbitmq config file
-rabbitmq_config: /etc/rabbitmq/rabbitmq.conf
+# The RabbitMQ home directory
+rabbitmq_home_dir: "{{ rabbitmq_default_home_dir }}"
 
-# The default rabbitmq port
-rabbitmq_port: 5672
-
-# The default rabbitmq mgt console port
-rabbitmq_mgt_console_port: 15672
-
-# The default rabbitmq user and group
+# The default RabbitMQ user and group
 rabbitmq_owner: rabbitmq
 rabbitmq_group: rabbitmq
 
 # Flag to enable/disable IPv6
 rabbitmq_bind_ipv6: true
 
-# The hostnames and/or IP address on which rabbitmq should listen for
+# The hostnames and/or IP address on which RabbitMQ should listen for
 # client connections.
 rabbitmq_bind_addr: 127.0.0.1
 
+# Log settings
+rabbitmq_log_dir: /var/log/rabbitmq
+rabbitmq_log_file: rabbit.log
+rabbitmq_log_file_level: info
+
+# Inter-node connections use a buffer for data pending to be sent. Temporary throttling on
+# inter-node traffic is applied when the buffer is at max allowed capacity. The limit is
+# controlled via the RABBITMQ_DISTRIBUTION_BUFFER_SIZE environment variable in kilobytes.
+# Default value is 128 MB (128000 kB).
+#
+# In clusters with heavy inter-node traffic increasing this value may have a positive effect
+# on throughput. Values lower than 64 MB are not recommended.
+# rabbitmq_distribution_buffer_size: 128000
+
 # The largest allowed message payload size in bytes. Messages of larger size will be
 # rejected with a suitable channel exception. Default: 134217728, Max value: 536870912
 # rabbit_max_msg_size: 536870911
@@ -35,6 +43,24 @@ rabbitmq_bind_addr: 127.0.0.1
 # Default: undefined (not set or used).
 # rabbit_total_mem_available_override: 8GB
 
+# Set disk free limit (in bytes). Once free disk space reaches this
+# lower bound, a disk alarm will be set - see the documentation
+# listed above for more details.
+#
+# Absolute watermark will be ignored if relative is defined!
+# disk_free_limit_absolute: 50MB
+
+# Feature flags that can be overriden in the hosts file
+# Essentially, a default installation will not have clustering
+rabbitmq_cluster: false
+
+# The location of the Erlang cookie file
+rabbitmq_erlang_cookie: "{{ rabbitmq_home_dir }}/.erlang.cookie"
+
+## The RabbitMQ cluster ports
+rabbitmq_cluster_port: "{{ rabbitmq_default_cluster_port }}"
+rabbitmq_epmd_port: "{{ rabbitmq_default_epmd_port }}"
+
 # Offline install settings
 # Refer to the offline variables in common_vars for additional settings
 packages_path: "{{ itential_packages_path }}/{{ iap_release }}/rabbitmq"