Skip to content

Conversation

@Pequod55
Copy link

Change Description:

This PR introduces a comprehensive automation system for managing Oracle software and patch definitions. It establishes a new workflow where developers can define new patches in a central YAML file, and a GitHub Action automatically updates the necessary configuration files and user documentation.

Solution Overview:

  • New GitHub Action (.github/workflows/modify_patches.yml): Triggers on pull requests to master and on manual dispatch. It orchestrates the entire automation process.
  • Central Input File (modify_patchlist.yml): A newly structured YAML file where users can add new software and patch definitions. The existing example entries have been commented out to serve as a template.
  • Automation Scripts:
    • modify_patches.py: A new Python script that parses modify_patchlist.yml and inserts the new patch data into the primary Ansible configuration file (roles/common/defaults/main.yml).
    • modify_documentation.py: A new Python script that similarly parses the input YAML and automatically updates the software download tables in the user guide (docs/user-guide.md).
  • Automated Commits: The workflow uses the stefanzweifel/git-auto-commit-action to commit the file changes generated by the scripts directly back to the pull request branch.
  • Testing Framework:
    • Introduces pytest and pytest-mock for testing, with dependencies listed in the new requirements.txt.
    • Includes a new test file, modify_patchlist_test.py, with a comprehensive integration test for the modify_patches.py script.
    • (Note: The .pytest_cache directory has been included. It is recommended to add .pytest_cache/ to the project's .gitignore file.)
  • New Documentation (modify_patchlist.md): A detailed user and technical guide has been added to explain the new automation workflow, file structure, and how to add new patches.

Test Commands:

Test Prep:

  1. Ensure Python 3 is installed. Install dependencies:
pip install -r ./.github/workflows/requirements.txt
  1. Create and check out a new branch from master.
  2. In modify_patchlist.yml, uncomment an example entry or add a new one. For example, add a new RDBMS patch:
rdbms_patches:
  - { category: "RU", base: "21.3.0.0.0", release: "21.99.0.0.0", patchnum: "39999999", patchfile: "p39999999_210000_Linux-x86-64.zip", patch_subdir: "/", prereq_check: false, method: "opatch apply", ocm: false, upgrade: false, md5sum: "a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4" }
  1. (Optional) Add a corresponding entry in the documentation_overrides section to test custom documentation text:
documentation_overrides:
  rdbms_patches:
    category: "Test Patch - MOS"
    software_piece: "Database Release Update 21.99 (Test)"
    file_name: "p39999999_210000_Linux-x86-64.zip"

Test 1: Run unit tests

  1. Execute the test suite to ensure the core logic is sound:
pytest

Test 2: Run the automation scripts locally

  1. Simulate the GitHub Action by running the scripts in order:
python .github/workflows/modify_documentation.py
python .github/workflows/modify_patches.py
  1. Inspect the modified files (roles/common/defaults/main.yml and docs/user-guide.md) to verify the changes.

Test 3: Run the full Git workflow (End-to-End Test)

  1. After completing Test Prep, commit the changes to modify_patchlist.yml:
git add modify_patchlist.yml
git commit -m "feat: Add test RDBMS patch 21.99"
  1. Push the branch and create a Pull Request to master to trigger the "Modify Patches" GitHub Action.

Expected Results:

For Test 1 (Unit Tests):

  • The pytest command completes successfully, with all tests passing.

For Test 2 (Local Run):

  • Both Python scripts execute without errors and print log messages about the updates they are performing.
  • The file docs/user-guide.md is updated to include a new row for the test patch, using the custom text from the documentation_overrides if provided.
  • The file roles/common/defaults/main.yml is updated with the new 21.99.0.0.0 patch entry under the rdbms_patches list.
  • The input file modify_patchlist.yml remains unchanged.

For Test 3 (PR Workflow):

  • The "Modify Patches" GitHub Action runs and completes successfully in the PR checks.
  • A new commit authored by "GitHub Actions" with the message "automation: Update patch files" is pushed to the PR branch.
  • This new commit contains the automated changes to roles/common/defaults/main.yml and docs/user-guide.md.
  • The modify_patchlist.yml file is not changed by the automation commit.

* feat: Add automation for modifying Oracle patch manifests

- Implemented a Python script (`modify_patches.py`) to read, process, and insert patch data into a target YAML file.
- Created a GitHub Actions workflow (`modify_patches.yml`) to automate the execution of the patch modification script on pull requests to the master branch.
- Added unit tests (`modify_patchlist_test.py`) to validate the functionality of the patch modification logic.
- Included a requirements file (`requirements.txt`) for necessary Python dependencies.
- Updated `.gitignore` to exclude Python cache files.
- Added a sample input YAML file (`modify_patchlist.yml`) to serve as a template for patch data.
- Documented the structure and purpose of the patch manifest in `modify_patchlist.md`.
- Cleaned up existing patch entries in `roles/common/defaults/main.yml` to prepare for new entries.

* feat: Add user guide for patch automation process

* fix: Update file extension references from .yaml to .yml in documentation and scripts

* Refactor the GCS download/transfer roles into a more logical and better flowing set of tasks

* Refined quoting for ssh remote commands in Ansible tasks; Minor adjustments to GCS software provisioning tasks

* Set delete_control_node to False for the presubmit test (google#320)

* Set delete_control_node to False for the presubmit test.

* increase loop timeout from 2 to 3 hours

* Add `whenever sqlerror exit sql.sqlcode` to all sqlplus blocks

* Set `db_create_file_dest` when not using ASM storage

* Change from `whenever sqlerror exit sql.sqlcode` to `whenever sqlerror exit failure`

* Configure RAC single-instance presubmit tests to use Oracle Multitenant

To get a bit more test coverage, here we configure the RAC single-instance test case (which is the faster one anyway) to set `--ora-db-container` to true, setting up multitenant.

* Fix multitenant check when saving PDB state (google#326)

* Fix multitenant check when saving PDB state

* Change `container_db` to a boolean

* Set pwd_gen_cmd conditionally based on ora-db-container flag (google#327)

* Set pwd_gen_cmd conditionally based on ora-db-container flag

* remove explicit cast to bool

* Remove `whenever sqlerror` when shutting down from mount; Use PL/SQL to enable flashback and force logging to avoid errors

* Take `oracle_edition` into consideration when building the RDBMS software file list

* Add `list` filter when building `rdbms_edition_sw` for Ansible 2.9 compatibility

* Add new readiness check to ensure that specified storage parameter disk groups exist in `asm_disks`

* Initial commit

* Fix basic syntax errors

* Refactor commit trap and log setup

Modify the common lib to also set up the cleanup trap as part of it's
init, as one less thing the callers might forget.

And talking about forgetting, we had a setup_logging function, and never
called it.  Refactoring so that watch_logs does both the setup and
watching.  It's long, but there's really no need for callers to run part
of it.

* Refactor commit trap and log setup

Modify the common lib to also set up the cleanup trap as part of it's
init, as one less thing the callers might forget.

And talking about forgetting, we had a setup_logging function, and never
called it.  Refactoring so that watch_logs does both the setup and
watching.  It's long, but there's really no need for callers to run part
of it.

* Initial commit

* Fix basic syntax errors

* Clean up merge artifacts

* Update templateing logic

Modify the sed logic to make replacements into a remplrary file and to
use that, allowing the presubmimt to be run multiple times without
clobbering itself.

Also removing a mistaken commented-out apk, adding an error check, and
sending error messages to stderr (&2).

* Add ora_db_container and improve error checks

Free edition runs Oracle 23c, which no longer supports non-CDBs.  Using
the default terraform `ora_db_container` value of false, installs fail.
This change explicitly sets it to `false` for data guard and `true` for
free edition tests.

Also some improved error checking:
* If the sourcing fails, bail out
* If the infra manager apply command fails, run a describe command to
  try and get the actual Terraform error if possible

* Add ora_db_container and improve error checks

Free edition runs Oracle 23c, which no longer supports non-CDBs.  Using
the default terraform `ora_db_container` value of false, installs fail.
This change explicitly sets it to `false` for data guard and `true` for
free edition tests.

Also some improved error checking:
* If the sourcing fails, bail out
* If the infra manager apply command fails, run a describe command to
  try and get the actual Terraform error if possible

* Override and set `ORA_DB_CONTAINER=TRUE` for Free edition installs

* set workload_agent_username conditionally

* Fix DG setup issues related to the staticConnectidentifier value and listener.ora & tnsnames.ora entries

* Use `ansible_hostname` instead of `ansible_fqdn` when building tnsnames.ora entry

* Build and set proper Data Guard StaticConnectIdentifier string

* Changes to support installing Free edition on EL9 with new `os_version` key-value in `rdbms_software`

* Updated filters for building list of RDBMS software to make `os_version` optional

* Make disks larger for free edition test

Oracle 23c's DBCA requires at least 13GB of space on the RECO disk.
Also extending the data disk because, until google#335 lands, the DATA and
RECO disks are classified as ASM disks and ignored, so the full install
actually goes on the boot disk.

Separately, make the free edition script executable in case someone
wants to run it directly using ./

* Address review comments

Check if ${deployment_name} is set in setup_vars()
Set `delete-confrol-node` to `false` to avoid a race condition on
long-running presubmits.

* Data Guard changes required to support deployments with XFS storage

* Change tnsnames.ora entries from `LISTENER_{{ db_name }}` to `LISTENER_{{ oracle_sid }}`

* Break `roles/common/defaults/main.yml` into multiple separate files for manageability

* Changes to support optionally procuring software from a URL

* Remove `url_username` and `url_password` for `uri` module tasks

* Refactor output handling in modify_patches.py to create multiple YAML files for different patch types and update tests accordingly

* fix: merge errors

* fix: merge errors

* fix: merge error

* fix: merge error

---------

Co-authored-by: Simon Pane <[email protected]>
Co-authored-by: Simon Pane <[email protected]>
Co-authored-by: AlexBasinov <[email protected]>
Co-authored-by: Marc Fielding <[email protected]>
Co-authored-by: Alexey Basinov <[email protected]>
@google-oss-prow
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Pequod55
Once this PR has been reviewed and has the lgtm label, please assign alexbasinov for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow
Copy link

Hi @Pequod55. Thanks for your PR.

I'm waiting for a google member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant