Want ability to take a sled out of the provisioning pool

[askfongjojo]

There are several situations when we do not want new VM instances to be placed on a certain sled:

1. when the operator or update/migrate program needs to drain a sled prior to SW/HW maintenance
2. when the migrate program wants to reserve the sled so that it has the expected capacity for VMs it's trying to migrate over
3. when the operator wants to reserve the sled to migrate in it a "noisy neighbor" from another sled

Tasks

• Backend implementation and API: [nexus] add sled provision state #4520
• Do we need to expose a read-only view of the provision state on the console?
  • We've decided that we'd like to wait to flesh out our fault management a bit before tackling this.
• Do we need to add support for setting provision state to the Oxide CLI?
  • This is handled automatically.

[askfongjojo]

Per suggestion in RFD 378, we can tentatively introduce reserved as a sled status (and reserve as the action in API/Console).

[jordanhendricks]

@askfongjojo and I chatted a bit about scoping this work out more cleanly for MVP. I believe these are the determinations we arrived at.

There are a couple of use cases I believe we need to prioritize for MVP with a feature that removes a sled from consideration for provisioning:

1. stopping new provisions to a sled so that we can migrate instances off of it, update the sled, and reboot it
2. noticing a sled is unusable (due to a crash, physical pull from the cubby, whatever) and stopping new provisions there (see also: sled-agent should heartbeat to nexus #2986)

Of these two cases, it seems to me that (1) is much more critical for MVP; if we can't stop new provisions to a sled, we create more unbounded work for migrating instances from it during the update path. The situation for (2) is not ideal, but I believe the worst thing that happens in the current provisioning scheme today (i.e., selecting a random sled) is that we would return an error, and it is possible to retry such an operation.

As a starting point, I propose we do the following:

• add a boolean column to the sled table indicating whether the sled is provisionable (and I think we likely want an associated timestamp for its last update, and at least a log message when it is toggled)
• modify sled_reservation_create to consider this field in its query to determine an appropriate sled for provisioning

There are some open questions here about if we want an API endpoint here (internal, or external for fleet operators) to change a sled's state in this way. As I see it, the most important consumer for now is the update process, and my understanding is that this process is still being ironed out. So for now, I propose that we punt on an API for this and I can add the plumbing to update the database schema and query the new field for a first PR. As update evolves and we have a better sense of what is needed, we can make determinations for what type of API we need to support around this behavior.

Lastly, this ticket mentions more complex status of sled state, such as goals outlined in RFD 378. One could also imagine us evolving the "provisioning pool" concept to multiple types of pools that might be operator-specified. I think that is something we can also punt on for the MVP timeframe, and as we move toward supporting such a world, we could design a more general-purpose feature.

[askfongjojo]

The logic of picking a sled for volume create will also have to take into account the sled status. We saw on rack2 that a bad sled was consistently picked because it appears to have the most available disk space:

omicron/nexus/db-queries/src/db/queries/region_allocation.rs

Lines 93 to 94 in cd40ec6

	.order(dataset_dsl::size_used.asc())
	.limit(REGION_REDUNDANCY_THRESHOLD.try_into().unwrap())