(BOLT-459) Create reboot plan #178

MikaelSmith · 2018-10-31T22:34:14Z

Creates a reboot plan that queries last boot time, reboots targets, then waits until all have rebooted or we timeout. Reverts previous work to add a wait function.

…t_function" This reverts commit d4e8444, reversing changes made to 313b52f.

MikaelSmith · 2018-10-31T22:35:43Z

Still needs some testing.

dylanratcliffe

Main concerns detailed in comments, but also if a task fails, but the plan handles that failure, will it still show as a failure in the PE GUI? If so this will generate a lot of failures and be very ugly...

plans/init.pp

tasks/nix.sh

tasks/win.ps1

MikaelSmith · 2018-11-01T18:49:51Z

plans/init.pp

+  }
+
+  # Reboot; catch errors here because the connection may get cut out from underneath
+  $reboot_result = run_task('reboot', $nodes, timeout => $reboot_delay, message => $message)


Having this be a single plan means that if most nodes successfully reboot but one fails, it's hard to recover. May need to split waiting for the reboot into a separate plan. Should we catch errors, wait for reboot on the successful nodes, then fail?

@reidmv any input on this question?

Catch errors -> wait for all nodes to finish -> fail seems like a logical eventflow to me, but I don't have an exact use case in mind.

MikaelSmith · 2018-11-01T20:27:47Z

In PE, it should look like
and the plan will be recorded as Complete.

MikaelSmith · 2018-11-02T22:14:59Z

I think I've added the minimum required testing. Probably won't have time to expand on it for a few weeks.

Out of date

lucywyman · 2018-11-08T17:31:27Z

2 approves: is this ready for merge?

MikaelSmith · 2018-11-08T20:11:59Z

Still working on getting acceptance tests to pass in CI.

MikaelSmith · 2018-11-08T21:28:07Z

spec/acceptance/zplans/init_spec.rb

+    # Ensure Bolt logger is initialized so 'notice' works when running the reboot plan.
+    # Needs to be moved to BoltSpec.
+    require 'bolt/logger'
+    Bolt::Logger.initialize_logging


This stuff is really messy, and doesn't seem to work consistently. @adreyer any idea what's going on with logging in BoltSpec?

I reworked this somewhat. There's still a lot of awkward setup we should probably roll into a BoltSpec helper.

MikaelSmith · 2018-11-09T17:49:41Z

Looks like I got a clean CI run yesterday evening.

MikaelSmith · 2018-11-09T19:19:51Z

Oh, finally figured out what's going on with tests. Some VMs don't like to be rebooted. The reboot acceptance tests themselves catch the reboot command and kill it, so no actual reboot happens.

tasks/last_boot_time.sh

Add bash and powershell implementations of the reboot task so it can be run on systems without Ruby and Puppet installed.

Adds a plan that reboots targets, then waits until they're available again.

Adds rspec unit tests for reboot plan based on BoltSpec::Plans.

Tests are constrained by needing to kill the shutdown command so we don't actually restart VMs.

Update Bolt test dependency to 1.3 to eliminate cludges to work around issues in BoltSpec.

Use Bolt 1.3`s new `wait_until_available` function to reduce the number of task runs we need to do to determine whether targets have rebooted.

Revert "Merge pull request puppetlabs#171 from dylanratcliffe/add_wai…

b04eb99

…t_function" This reverts commit d4e8444, reversing changes made to 313b52f.

This was referenced Oct 31, 2018

Revert "Merge pull request #171 from dylanratcliffe/add_wait_function" #175

Closed

Use TargetSpec type for wait function #174

Closed

MikaelSmith force-pushed the BOLT-957 branch 2 times, most recently from fe00117 to ffd6370 Compare October 31, 2018 22:40

dylanratcliffe suggested changes Nov 1, 2018

View reviewed changes

plans/init.pp Outdated Show resolved Hide resolved

plans/init.pp Show resolved Hide resolved

plans/init.pp Show resolved Hide resolved

tasks/nix.sh Show resolved Hide resolved

tasks/win.ps1 Show resolved Hide resolved

MikaelSmith force-pushed the BOLT-957 branch 3 times, most recently from a0dc39c to a26a2ce Compare November 1, 2018 18:47

MikaelSmith commented Nov 1, 2018

View reviewed changes

MikaelSmith force-pushed the BOLT-957 branch from a26a2ce to 092d76c Compare November 1, 2018 18:55

MikaelSmith force-pushed the BOLT-957 branch 9 times, most recently from e35d86c to 7021a80 Compare November 2, 2018 00:55

MikaelSmith added the needs-tests label Nov 2, 2018

MikaelSmith force-pushed the BOLT-957 branch 4 times, most recently from 92d3f69 to 95146e2 Compare November 2, 2018 22:14

MikaelSmith removed the needs-tests label Nov 2, 2018

MikaelSmith force-pushed the BOLT-957 branch from 95146e2 to b3706a9 Compare November 2, 2018 23:11

MikaelSmith force-pushed the BOLT-957 branch from bd73d19 to c94185e Compare November 8, 2018 00:09

lucywyman approved these changes Nov 8, 2018

View reviewed changes

dylanratcliffe approved these changes Nov 8, 2018

View reviewed changes

MikaelSmith closed this Nov 8, 2018

MikaelSmith reopened this Nov 8, 2018

MikaelSmith force-pushed the BOLT-957 branch from e7d5ad5 to 0b26f6d Compare November 8, 2018 18:54

MikaelSmith commented Nov 8, 2018

View reviewed changes

MikaelSmith force-pushed the BOLT-957 branch 4 times, most recently from 011c59c to 07d7208 Compare November 9, 2018 00:32

MikaelSmith force-pushed the BOLT-957 branch 2 times, most recently from a4850d6 to b923ca4 Compare November 14, 2018 22:55

MikaelSmith commented Nov 16, 2018

View reviewed changes

tasks/last_boot_time.sh Outdated Show resolved Hide resolved

MikaelSmith added 7 commits November 16, 2018 15:50

(BOLT-459) Add bash and powershell implementations of reboot task

bba4efb

Add bash and powershell implementations of the reboot task so it can be run on systems without Ruby and Puppet installed.

(BOLT-459) Add tasks to get last boot time

63794fb

(BOLT-459) Add reboot plan

134cb6e

Adds a plan that reboots targets, then waits until they're available again.

(BOLT-459) Add plan spec tests

30a61ff

Adds rspec unit tests for reboot plan based on BoltSpec::Plans.

(BOLT-459) Add tests for new tasks

a578da7

Tests are constrained by needing to kill the shutdown command so we don't actually restart VMs.

(maint) Remove test cludges with Bolt 1.3.0

7481dd2

Update Bolt test dependency to 1.3 to eliminate cludges to work around issues in BoltSpec.

(BOLT-957) Use wait_until_available to reduce task runs

c9bf10d

Use Bolt 1.3`s new `wait_until_available` function to reduce the number of task runs we need to do to determine whether targets have rebooted.

MikaelSmith force-pushed the BOLT-957 branch from b923ca4 to c9bf10d Compare November 16, 2018 23:51

MikaelSmith merged commit 1284ef0 into puppetlabs:master Nov 19, 2018

MikaelSmith deleted the BOLT-957 branch November 19, 2018 23:49

MikaelSmith mentioned this pull request Nov 21, 2018

(MODULES-8091) Prep module for 2.1.0 release #173

Merged

(BOLT-459) Create reboot plan #178

(BOLT-459) Create reboot plan #178

Uh oh!

Conversation

MikaelSmith commented Oct 31, 2018

Uh oh!

MikaelSmith commented Oct 31, 2018

Uh oh!

dylanratcliffe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MikaelSmith Nov 1, 2018

Choose a reason for hiding this comment

Uh oh!

MikaelSmith Nov 3, 2018

Choose a reason for hiding this comment

Uh oh!

lucywyman Nov 7, 2018

Choose a reason for hiding this comment

Uh oh!

MikaelSmith commented Nov 1, 2018

Uh oh!

MikaelSmith commented Nov 2, 2018

Uh oh!

lucywyman commented Nov 8, 2018

Uh oh!

MikaelSmith commented Nov 8, 2018

Uh oh!

MikaelSmith Nov 8, 2018

Choose a reason for hiding this comment

Uh oh!

MikaelSmith Nov 9, 2018

Choose a reason for hiding this comment

Uh oh!

MikaelSmith commented Nov 9, 2018

Uh oh!

MikaelSmith commented Nov 9, 2018

Uh oh!

Uh oh!

Uh oh!