Ensure config reload on ..data symlink switch for CSI driver support

[jfreden]

This PR fixes an issue that was surfaced when using the new CSI driver (not yet in production) used by the Elasticsearch controller to mount config files as secrets. Unlike the current kubelet-based approach, the CSI driver doesn't trigger an ENTRY_MODIFY event on the config file watcher when there are symlink updates in config/operator, causing file changes to go unnoticed by the file-watching service—so the fix ensures a full re-read of the config/operator directory when directory symlinks are updated.

The controller mounts settings.json, project-xyz.json and project-xyz.secrets.json on the Elastcisearch pods in serverless as kubernetes secrets. This is currently done using the kublet that run on each node. The files are are represented by symlinks in the config/operator directory that point to the files in a symlinked directory. For example:

drwxr-xr-x    2 root     root          4096 Apr 29 13:51 ..2025_04_29_13_51_31.1372434508
lrwxrwxrwx    1 root     root            32 Apr 29 13:51 ..data -> ..2025_04_29_13_51_31.1372434508
lrwxrwxrwx    1 root     root            52 Apr 29 13:22 project-ff08458fe99c4e37be0829600b4b5022.json -> ..data/project-ff08458fe99c4e37be0829600b4b5022.json
lrwxrwxrwx    1 root     root            57 Apr 29 13:22 project-ff08458fe99c4e37be0829600b4b5022.roles.yml -> ..data/project-ff08458fe99c4e37be0829600b4b5022.roles.yml
lrwxrwxrwx    1 root     root            60 Apr 29 13:22 project-ff08458fe99c4e37be0829600b4b5022.secrets.json -> ..data/project-ff08458fe99c4e37be0829600b4b5022.secrets.json
lrwxrwxrwx    1 root     root            20 Apr 29 13:22 settings.json -> ..data/settings.json

In Elasticsearch, MultiProjectFileSettingsService (and FileSettingsService in non-multi-project environments) is listening for file updates in the config and config/operator through the AbstractFileWatchingService (by registering a WatchService for the config and operator directories and listen for ENTRY_MODIFY, ENTRY_CREATE and ENTRY_DELETE). Because of the symlink approach, the symlinks for the files that are monitored are not updated, only the ..data symlink is switched to a different timestamped directory, to allow for an atomic update of all files. When running with kublet the ..data symlink is updated (recreated) in the config/operator directory, the AbstractFileWatchingService gets an update for the config directory (ENTRY_MODIFY on the config watch key), this in turn trigger a re-register of the config/operator watch key and a re-read of all files in the config/operator, because we don't know if the file name maps to the same native file system file id.

With the new csi-driver, the ..data symlink switch no longer triggers ENTRY_MODIFY on the config directory after settings.json is changed, so we don't re-register the config/operator watch key and the file changes are never picked up. We're not entirely sure what the difference is in the standard kublet flow vs the csi-driver but I think that ENTRY_MODIFY on the config directory for the ..data symlink switch might have been accidental and that we should always re-read (check if a file has been updated) the full config/operator directory every time a directory symlink is updated in the settings directory.

I've added a FileSettingsServiceIT test to reproduce the issue. This only happens on Linux. On Mac (my local) this doesn't happen because the ENTRY_MODIFY event is triggered for the config watcher when updating the config/operator/..data symlink.

[jfreden]

This test fails on linux without the changes to AbstractFileWatchingService.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[elasticsearchmachine]

Hi @jfreden, I've created a changelog YAML for you.