org.elasticsearch.xpack.ilm.PolicyStepsRegistry Uses Not-Threadsafe Maps Concurently

[original-brownbear]

Straight forward bug:

These 3 maps:

    private final SortedMap<String, LifecyclePolicyMetadata> lifecyclePolicyMap;
    // keeps track of what the first step in a policy is, the key is policy name
    private final Map<String, Step> firstStepMap;
    // keeps track of a mapping from policy/step-name to respective Step, the key is policy name
    private final Map<String, Map<Step.StepKey, Step>> stepMap;

are all plain hash or tree maps and get updated on every cluster state update on the cluster state applier thread. but are accessed from at least the CS threads and the ILM scheduler threads concurrently.

We need to do something to synchronize them correctly or else at least the period execution will break in subtle ways here and there.

[elasticmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[nielsbauman]

I just learned this issue exists. We already encountered a bug related to this, which was fixed in #126840. I'm not sure how many real bugs can still be caused by this, because ILM is quite fault tolarent nowadays - meaning we may fail once because of a mismatch in the maps, but a following run will pass when the maps are aligned again. But I agree that doing some kind of synchronization and/or atomic updates to the maps here is valuable to rule out any issues.

[original-brownbear]

++ yes please :) it's not like you're just gonna run into an outdated view of these maps here iether, NPE and friends might also show up at some rate. Generally, for this kind of thing the copy-on-write pattern tends to be a a good fit but I don't remember all the details here 4 years later :)