Add all-in-one CI testing

[markgoddard]

• Add kayobe-automation submodule & configuration
• Add kayobe to requirements.txt
• Add an all-in-one Terraform configuration & README
• Add a playbook to create test resources
• Populate kayobe-config Tempest configuration for refstack
• Add Github actions workflows to build the kayobe image, and run all-in-one tests againsts PRs

[markgoddard]

It probably needs a little more work, but wanted to get your thoughts @jovial

[jovial]

Looks good to me.

[markgoddard]

Is there anything in here that would conflict with existing consumer configuration? Perhaps automation.conf/config.sh?

[jovial]

Is there anything in here that would conflict with existing consumer configuration? Perhaps automation.conf/config.sh?

I've been making things conditional on the kayobe environment:

  if [ "$KAYOBE_ENVIRONMENT" == "aio" ]; then
    # Seem to get servers failing to spawn with higher concurrency
    export TEMPEST_CONCURRENCY=1
  fi

so we could supply a basic configuration that could be used for client side CI and put any upstream specifics behind guards like this. Yes it would merge conflict, but at least that provides a nice way of keeping it up to date. What do you reckon? The one slight gotcha is that you need to supply KAYOBE_ENVIRONMENT instead of relying on .environment as kayobe-env will not have been loaded yet. That could possibly be resolved if we just read that file as well.

Edit: reworded slightly

[markgoddard]

Is there anything in here that would conflict with existing consumer configuration? Perhaps automation.conf/config.sh?

I've been making things conditional on the kayobe environment:

  if [ "$KAYOBE_ENVIRONMENT" == "aio" ]; then
    # Seem to get servers failing to spawn with higher concurrency
    export TEMPEST_CONCURRENCY=1
  fi

so we could supply a basic configuration that could be used for client side CI and put any upstream specifics behind guards like this. Yes it would merge conflict, but at least that provides a nice way of keeping it up to date. What do you reckon? The one slight gotcha is that you need to supply KAYOBE_ENVIRONMENT instead of relying on .environment as kayobe-env will not have been loaded yet. That could possibly be resolved if we just read that file as well.

Edit: reworded slightly

I did that for the concurrency (adapted from SOS), is there anything else it needs to be done for?

[jovial]

Is there anything in here that would conflict with existing consumer configuration? Perhaps automation.conf/config.sh?

I've been making things conditional on the kayobe environment:

  if [ "$KAYOBE_ENVIRONMENT" == "aio" ]; then
    # Seem to get servers failing to spawn with higher concurrency
    export TEMPEST_CONCURRENCY=1
  fi

so we could supply a basic configuration that could be used for client side CI and put any upstream specifics behind guards like this. Yes it would merge conflict, but at least that provides a nice way of keeping it up to date. What do you reckon? The one slight gotcha is that you need to supply KAYOBE_ENVIRONMENT instead of relying on .environment as kayobe-env will not have been loaded yet. That could possibly be resolved if we just read that file as well.
Edit: reworded slightly

I did that for the concurrency (adapted from SOS), is there anything else it needs to be done for?

Don't have any good examples really, but I've started shipping some additional test lists:

KAYOBE_AUTOMATION_TEMPEST_CONF_OVERRIDES="${KAYOBE_AUTOMATION_CONFIG_PATH}/tempest/tempest.overrides.conf"

if [ ! -z ${KAYOBE_ENVIRONMENT:+x} ]; then
  KAYOBE_AUTOMATION_TEMPEST_CONF_OVERRIDES="${KAYOBE_AUTOMATION_CONFIG_PATH}/tempest/tempest-${KAYOBE_ENVIRONMENT}-${KAYOBE_AUTOMATION_TEMPEST_LOADLIST:-}.overrides.conf"

  # Check if loadlist specific overrides exist, if not fallback to environment overrides.
  if [ ! -e "${KAYOBE_AUTOMATION_TEMPEST_CONF_OVERRIDES}" ]; then
      KAYOBE_AUTOMATION_TEMPEST_CONF_OVERRIDES="${KAYOBE_AUTOMATION_CONFIG_PATH}/tempest/tempest-${KAYOBE_ENVIRONMENT}.overrides.conf"
  fi
fi

Trying to establish a convention for where to place tempest overrides. It basically does first found on:

• overrides for specific test list using a specific environment
• overrides for specific environment
• shared overrides

Trying to go for convention over configuration here, so that people don't start doing it in multiple different ways.

Maybe that shouldn't be in the config file, but i started there whilst experimenting.

[jovial]

New changes look good to me.