With the nwcsaf-geo reader, loading an extra file leads to a product becoming unloadable

[gerritholl]

Describe the bug

When I pass only "S_NWC_CT_MSG2_MSG-N-VISIR_20220124T094500Z.nc" to Scene(...), I can load ct. When I pass both "S_NWC_CT_MSG2_MSG-N-VISIR_20220124T094500Z.nc" and "S_NWC_HRW_MSG2_MSG-N-VISIR_20220124T094500Z.nc", I cannot load ct.

To Reproduce

from satpy import Scene
from satpy.utils import debug_on; debug_on()

files_l2 = ["/media/nas/x21308/scratch/NWCSAF/20220124T094500Z/S_NWC_CT_MSG2_MSG-N-VISIR_20220124T094500Z.nc", "/media/nas/x21308/scratch/NWCSAF/20220124T094500Z/S_NWC_HRW_MSG2_MSG-N-VISIR_20220124T094500Z.nc"]
sc = Scene({"nwcsaf-geo": files_l2[0:1]})  # success
sc.load(["ct"])
sc = Scene({"nwcsaf-geo": files_l2[0:2]})  # failure
sc.load(["ct"])

Expected behavior

I expect that adding extra files for the same timeslot should lead to zero or more loadable datasets, but not make previously loadable datasets unloadable.

Actual results

[DEBUG: 2025-08-21 15:28:41 : satpy.readers.core.yaml_reader] Reading ('/home/gholl/checkouts/satpy/satpy/etc/readers/nwcsaf-geo.yaml', '/home/gholl/checkouts/pytroll-dwd-config/oper/readers/nwcsaf-geo.yaml')
[DEBUG: 2025-08-21 15:28:41 : h5py._conv] Creating converter from 7 to 5
[DEBUG: 2025-08-21 15:28:41 : h5py._conv] Creating converter from 5 to 7
[DEBUG: 2025-08-21 15:28:41 : h5py._conv] Creating converter from 7 to 5
[DEBUG: 2025-08-21 15:28:41 : h5py._conv] Creating converter from 5 to 7
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/yaml/constructor.py:559: UserWarning: 'satpy.readers.yaml_reader.FileYAMLReader' has been moved to 'satpy.readers.core.yaml_reader.FileYAMLReader'. Import from the new location instead (ex. 'from satpy.readers.core.yaml_reader import FileYAMLReader'). The old import paths will be removed in Satpy 1.0
  if not hasattr(module, object_name):
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/yaml/constructor.py:563: UserWarning: 'satpy.readers.yaml_reader.FileYAMLReader' has been moved to 'satpy.readers.core.yaml_reader.FileYAMLReader'. Import from the new location instead (ex. 'from satpy.readers.core.yaml_reader import FileYAMLReader'). The old import paths will be removed in Satpy 1.0
  return getattr(module, object_name)
[DEBUG: 2025-08-21 15:28:42 : satpy.readers.core.yaml_reader] Assigning to nwcsaf-geo: ['/media/nas/x21308/scratch/NWCSAF/20220124T094500Z/S_NWC_CT_MSG2_MSG-N-VISIR_20220124T094500Z.nc']
[DEBUG: 2025-08-21 15:28:42 : rasterio.session] Could not import boto3, continuing with reduced functionality.
[DEBUG: 2025-08-21 15:28:42 : satpy.composites.config_loader] Looking for composites config file seviri.yaml
[DEBUG: 2025-08-21 15:28:42 : pyorbital.tlefile] Path to the Pyorbital configuration (where e.g. platforms.txt is found): /home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/pyorbital/etc
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/yaml/constructor.py:559: UserWarning: 'satpy.composites.SingleBandCompositor' has been moved to 'satpy.composites.core.SingleBandCompositor'. Import from the new location instead (ex. 'from satpy.composites.core import SingleBandCompositor'). The old import paths will be removed in Satpy 1.0
  if not hasattr(module, object_name):
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/yaml/constructor.py:563: UserWarning: 'satpy.composites.SingleBandCompositor' has been moved to 'satpy.composites.core.SingleBandCompositor'. Import from the new location instead (ex. 'from satpy.composites.core import SingleBandCompositor'). The old import paths will be removed in Satpy 1.0
  return getattr(module, object_name)
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/yaml/constructor.py:559: UserWarning: 'satpy.composites.DayNightCompositor' has been moved to 'satpy.composites.fill.DayNightCompositor'. Import from the new location instead (ex. 'from satpy.composites.fill import DayNightCompositor'). The old import paths will be removed in Satpy 1.0
  if not hasattr(module, object_name):
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/yaml/constructor.py:563: UserWarning: 'satpy.composites.DayNightCompositor' has been moved to 'satpy.composites.fill.DayNightCompositor'. Import from the new location instead (ex. 'from satpy.composites.fill import DayNightCompositor'). The old import paths will be removed in Satpy 1.0
  return getattr(module, object_name)
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/yaml/constructor.py:559: UserWarning: 'satpy.composites.GenericCompositor' has been moved to 'satpy.composites.core.GenericCompositor'. Import from the new location instead (ex. 'from satpy.composites.core import GenericCompositor'). The old import paths will be removed in Satpy 1.0
  if not hasattr(module, object_name):
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/yaml/constructor.py:563: UserWarning: 'satpy.composites.GenericCompositor' has been moved to 'satpy.composites.core.GenericCompositor'. Import from the new location instead (ex. 'from satpy.composites.core import GenericCompositor'). The old import paths will be removed in Satpy 1.0
  return getattr(module, object_name)
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/yaml/constructor.py:559: UserWarning: 'satpy.composites.Filler' has been moved to 'satpy.composites.fill.Filler'. Import from the new location instead (ex. 'from satpy.composites.fill import Filler'). The old import paths will be removed in Satpy 1.0
  if not hasattr(module, object_name):
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/yaml/constructor.py:563: UserWarning: 'satpy.composites.Filler' has been moved to 'satpy.composites.fill.Filler'. Import from the new location instead (ex. 'from satpy.composites.fill import Filler'). The old import paths will be removed in Satpy 1.0
  return getattr(module, object_name)
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/yaml/constructor.py:559: UserWarning: 'satpy.composites.MaskingCompositor' has been moved to 'satpy.composites.mask.MaskingCompositor'. Import from the new location instead (ex. 'from satpy.composites.mask import MaskingCompositor'). The old import paths will be removed in Satpy 1.0
  if not hasattr(module, object_name):
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/yaml/constructor.py:563: UserWarning: 'satpy.composites.MaskingCompositor' has been moved to 'satpy.composites.mask.MaskingCompositor'. Import from the new location instead (ex. 'from satpy.composites.mask import MaskingCompositor'). The old import paths will be removed in Satpy 1.0
  return getattr(module, object_name)
[DEBUG: 2025-08-21 15:28:42 : satpy.composites.config_loader] Looking for composites config file visir.yaml
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/yaml/constructor.py:559: UserWarning: 'satpy.composites.LongitudeMaskingCompositor' has been moved to 'satpy.composites.mask.LongitudeMaskingCompositor'. Import from the new location instead (ex. 'from satpy.composites.mask import LongitudeMaskingCompositor'). The old import paths will be removed in Satpy 1.0
  if not hasattr(module, object_name):
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/yaml/constructor.py:563: UserWarning: 'satpy.composites.LongitudeMaskingCompositor' has been moved to 'satpy.composites.mask.LongitudeMaskingCompositor'. Import from the new location instead (ex. 'from satpy.composites.mask import LongitudeMaskingCompositor'). The old import paths will be removed in Satpy 1.0
  return getattr(module, object_name)
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/yaml/constructor.py:559: UserWarning: 'satpy.composites.MultiFiller' has been moved to 'satpy.composites.fill.MultiFiller'. Import from the new location instead (ex. 'from satpy.composites.fill import MultiFiller'). The old import paths will be removed in Satpy 1.0
  if not hasattr(module, object_name):
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/yaml/constructor.py:563: UserWarning: 'satpy.composites.MultiFiller' has been moved to 'satpy.composites.fill.MultiFiller'. Import from the new location instead (ex. 'from satpy.composites.fill import MultiFiller'). The old import paths will be removed in Satpy 1.0
  return getattr(module, object_name)
[DEBUG: 2025-08-21 15:28:42 : satpy.readers.nwcsaf_nc] Reading ct.
[DEBUG: 2025-08-21 15:28:42 : satpy.readers.nwcsaf_nc] Reading ct_pal.
[DEBUG: 2025-08-21 15:28:42 : satpy.readers.core.yaml_reader] No coordinates found for DataID(name='ct_pal', resolution=3000, modifiers=())
[DEBUG: 2025-08-21 15:28:42 : satpy.readers.nwcsaf_nc] Reading ct_status_flag.
[DEBUG: 2025-08-21 15:28:42 : satpy.readers.nwcsaf_nc] Reading ct_quality.
[DEBUG: 2025-08-21 15:28:42 : satpy.readers.nwcsaf_nc] Reading ct_conditions.
[DEBUG: 2025-08-21 15:28:42 : satpy.readers.core.yaml_reader] Reading ('/home/gholl/checkouts/satpy/satpy/etc/readers/nwcsaf-geo.yaml', '/home/gholl/checkouts/pytroll-dwd-config/oper/readers/nwcsaf-geo.yaml')
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/yaml/constructor.py:559: UserWarning: 'satpy.readers.yaml_reader.FileYAMLReader' has been moved to 'satpy.readers.core.yaml_reader.FileYAMLReader'. Import from the new location instead (ex. 'from satpy.readers.core.yaml_reader import FileYAMLReader'). The old import paths will be removed in Satpy 1.0
  if not hasattr(module, object_name):
/home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/yaml/constructor.py:563: UserWarning: 'satpy.readers.yaml_reader.FileYAMLReader' has been moved to 'satpy.readers.core.yaml_reader.FileYAMLReader'. Import from the new location instead (ex. 'from satpy.readers.core.yaml_reader import FileYAMLReader'). The old import paths will be removed in Satpy 1.0
  return getattr(module, object_name)
[DEBUG: 2025-08-21 15:28:42 : satpy.readers.core.yaml_reader] Assigning to nwcsaf-geo: ['/media/nas/x21308/scratch/NWCSAF/20220124T094500Z/S_NWC_CT_MSG2_MSG-N-VISIR_20220124T094500Z.nc', '/media/nas/x21308/scratch/NWCSAF/20220124T094500Z/S_NWC_HRW_MSG2_MSG-N-VISIR_20220124T094500Z.nc']
Traceback (most recent call last):
  File "/home/gholl/checkouts/satpy/satpy/scene.py", line 1482, in _update_dependency_tree
    self._dependency_tree.populate_with_keys(needed_datasets, query)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gholl/checkouts/satpy/satpy/dependency_tree.py", line 265, in populate_with_keys
    raise MissingDependencies(unknown_datasets, "Unknown datasets:")
satpy.node.MissingDependencies: Unknown datasets: {DataQuery(name='ct')}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gholl/checkouts/protocode/mwe/nwcsaf-ct-not-found.py", line 8, in <module>
    sc.load(["ct"])
    ~~~~~~~^^^^^^^^
  File "/home/gholl/checkouts/satpy/satpy/scene.py", line 1470, in load
    self._update_dependency_tree(needed_datasets, query)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gholl/checkouts/satpy/satpy/scene.py", line 1484, in _update_dependency_tree
    raise KeyError(str(err))
KeyError: "Unknown datasets: {DataQuery(name='ct')}"

Environment Info:

• OS: openSUSE Leap 15.6
• Satpy Version: v0.58.0-44-g9030005a1

Additional context

I seem to recall having seen this before, but I cannot find a corresponding issue.

[djhoese]

Probably unrelated to your problem, but do you have a custom version of this reader's YAML? You're getting UserWarnings about the reader import path. Either your YAML is custom or Panu missed an import path in his refactor PR.

I'm not familiar with this data, but it is geostationary, right? Are these two different "regions/sectors" of the product? For example, two segments of larger full-disk image? Or are these completely separate L2+ product files?

[djhoese]

I forgot to say, I would assume there is some exception in the reader or file handler that is being hidden from the log. Or maybe a bad available_datasets implementation in a file handler.

[gerritholl]

Probably unrelated to your problem, but do you have a custom version of this reader's YAML? You're getting UserWarnings about the reader import path. Either your YAML is custom or Panu missed an import path in his refactor PR.

Ah, yes, I do. I forgot. When I remove it, the output is:

[DEBUG: 2025-08-25 08:21:58 : satpy.readers.core.yaml_reader] Reading ('/home/gholl/checkouts/satpy/satpy/etc/readers/nwcsaf-geo.yaml',)
[DEBUG: 2025-08-25 08:21:58 : h5py._conv] Creating converter from 7 to 5
[DEBUG: 2025-08-25 08:21:58 : h5py._conv] Creating converter from 5 to 7
[DEBUG: 2025-08-25 08:21:58 : h5py._conv] Creating converter from 7 to 5
[DEBUG: 2025-08-25 08:21:58 : h5py._conv] Creating converter from 5 to 7
[DEBUG: 2025-08-25 08:21:58 : satpy.readers.core.yaml_reader] Assigning to nwcsaf-geo: ['/media/nas/x21308/scratch/NWCSAF/20220124T094500Z/S_NWC_CT_MSG2_MSG-N-VISIR_20220124T094500Z.nc']
[DEBUG: 2025-08-25 08:21:58 : rasterio.session] Could not import boto3, continuing with reduced functionality.
[DEBUG: 2025-08-25 08:21:58 : satpy.composites.config_loader] Looking for composites config file seviri.yaml
[DEBUG: 2025-08-25 08:21:58 : pyorbital.tlefile] Path to the Pyorbital configuration (where e.g. platforms.txt is found): /home/gholl/miniforge3/envs/py313/lib/python3.13/site-packages/pyorbital/etc
[DEBUG: 2025-08-25 08:21:58 : satpy.composites.config_loader] Looking for composites config file visir.yaml
[DEBUG: 2025-08-25 08:21:58 : satpy.readers.nwcsaf_nc] Reading ct.
[DEBUG: 2025-08-25 08:21:58 : satpy.readers.nwcsaf_nc] Reading ct_pal.
[DEBUG: 2025-08-25 08:21:58 : satpy.readers.core.yaml_reader] No coordinates found for DataID(name='ct_pal', resolution=3000, modifiers=())
[DEBUG: 2025-08-25 08:21:58 : satpy.readers.nwcsaf_nc] Reading ct_conditions.
[DEBUG: 2025-08-25 08:21:58 : satpy.readers.nwcsaf_nc] Reading ct_quality.
[DEBUG: 2025-08-25 08:21:58 : satpy.readers.nwcsaf_nc] Reading ct_status_flag.
[DEBUG: 2025-08-25 08:21:58 : satpy.readers.core.yaml_reader] Reading ('/home/gholl/checkouts/satpy/satpy/etc/readers/nwcsaf-geo.yaml',)
[DEBUG: 2025-08-25 08:21:58 : satpy.readers.core.yaml_reader] Assigning to nwcsaf-geo: ['/media/nas/x21308/scratch/NWCSAF/20220124T094500Z/S_NWC_CT_MSG2_MSG-N-VISIR_20220124T094500Z.nc', '/media/nas/x21308/scratch/NWCSAF/20220124T094500Z/S_NWC_HRW_MSG2_MSG-N-VISIR_20220124T094500Z.nc']
Traceback (most recent call last):
  File "/home/gholl/checkouts/satpy/satpy/scene.py", line 1482, in _update_dependency_tree
    self._dependency_tree.populate_with_keys(needed_datasets, query)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gholl/checkouts/satpy/satpy/dependency_tree.py", line 265, in populate_with_keys
    raise MissingDependencies(unknown_datasets, "Unknown datasets:")
satpy.node.MissingDependencies: Unknown datasets: {DataQuery(name='ct')}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gholl/checkouts/protocode/mwe/nwcsaf-ct-not-found.py", line 8, in <module>
    sc.load(["ct"])
    ~~~~~~~^^^^^^^^
  File "/home/gholl/checkouts/satpy/satpy/scene.py", line 1470, in load
    self._update_dependency_tree(needed_datasets, query)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gholl/checkouts/satpy/satpy/scene.py", line 1484, in _update_dependency_tree
    raise KeyError(str(err))
KeyError: "Unknown datasets: {DataQuery(name='ct')}"

I'm not familiar with this data, but it is geostationary, right? Are these two different "regions/sectors" of the product? For example, two segments of larger full-disk image? Or are these completely separate L2+ product files?

Those data are geostationary. The two L2 files should correspond to the same area, as they were produced with the same NWCSAF GEO run (v2018.1). They are not segmented.

[djhoese]

While I'm surprised at the failure, providing two time steps of geostationary data is not supported by Satpy...or at least not yet. Satpy can concatenate swath granules, or segmented geo data, or load single time steps of geo data, but it can't merge multiple time steps together in the plain Scene.

[gerritholl]

It's two different L2 products covering the same timestep for the same sensor. It works for other products (for example CTH, CT), but no HRW.

[pnuu]

The HRW is a big mess as data. I'm not surprised in the least that there are problems with it when loading together with some other data. For example it's not "image" data in the fitst place so it doesn't have the same area as the image-like datasets of NWC SAF GEO.

[djhoese]

Can you give me access to these files?

[pnuu]

@djhoese I've put some data available on Slack.

The HRW reader was implemented in #3070

[djhoese]

Sorry @pnuu, it is your fault:

satpy/satpy/readers/nwcsaf_hrw_nc.py

Lines 197 to 208 in 1de6ac4

	def available_datasets(self, configured_datasets=None):
	"""Form the names for the available datasets."""
	for channel in WIND_CHANNELS:
	prefix = self._get_channel_prefix(channel)
	dset = self.h5f[channel]
	for measurand in dset.dtype.fields.keys():
	if measurand == "trajectory":
	continue
	ds_info = self._measurand_ds_info(prefix, measurand)
	yield True, ds_info
	if self.merge_channels:
	break

The available_datasets method is selfish and completely ignores the configured_datasets that it is provided. That means it throws out all datasets defined in the YAML AND would throw out any datasets defined in other file handlers available_datasets methods (which I think is none).

I can fix it if someone else can write the tests 😉

[djhoese]

Ok so the simplest solution is:

diff --git a/satpy/readers/nwcsaf_hrw_nc.py b/satpy/readers/nwcsaf_hrw_nc.py
--- a/satpy/readers/nwcsaf_hrw_nc.py	(revision 035f9b0d747d2ecef4e93459023b572ea77b9a6c)
+++ b/satpy/readers/nwcsaf_hrw_nc.py	(date 1756226642450)
@@ -196,6 +196,8 @@
 
     def available_datasets(self, configured_datasets=None):
         """Form the names for the available datasets."""
+        if configured_datasets is not None:
+            yield from configured_datasets
         for channel in WIND_CHANNELS:
             prefix = self._get_channel_prefix(channel)
             dset = self.h5f[channel]

BUT this assumes that no YAML configured datasets are important to this file handler and never will be. If someone adds a YAML configured dataset for this file handler it will never be listed as available.