Make uri and serialized_query_parameters available to providers in itemtypes?

[mikemahoney218-usgs]

Is your feature request related to a problem? Please describe.

Related to #1981 , we've been working to move away from limit-offset paging for our Postgres provider, as deep pages using large offsets are inefficient and can cause users to download duplicate records if earlier pages have been updated during a query. In #1982 , we altered itemtypes so that "next" links could be returned by a provider without pygeoapi then appending its own offset-based next links to the response.

The thing we missed was that pygeoapi uses uri and serialized_query_params to generate the next_href object, and these objects aren't available from inside a provider:

pygeoapi/pygeoapi/api/itemtypes.py

Line 567 in c976bd7

next_href = f'{uri}?offset={next_}{serialized_query_params}'

This makes it challenging to generate next links inside a provider.

Describe the solution you'd like

We'd like uri and serialized_query_params to be passed to providers to be used to generate next links if desired.

Describe alternatives you've considered

We could alternatively have some way in itemtypes to pass our "page key" from the provider and have itemtypes use that to generate the link. If offset wasn't validated to be numeric, we'd just need some way for the provider to control next_ here:

pygeoapi/pygeoapi/api/itemtypes.py

Line 567 in c976bd7

next_href = f'{uri}?offset={next_}{serialized_query_params}'

I think that approach is a bit less friendly than letting giving the provider these objects.

[webb-ben]

While not a complete solution, you can fetch the uri from within the provider similar to how it is done for the SensorThings provider with...

from pygeoapi.config import get_config
from pygeoapi.util import get_base_url

pygeoapi/pygeoapi/provider/sensorthings.py

Lines 612 to 613 in 45cac60

	CONFIG = get_config()
	self.rel_link = get_base_url(CONFIG)

Are there specific query parameters that are not making it to the provider? or is it just significantly more complicated to generate the serialized params from what is passed to the provider?

[allenj]

@webb-ben see the PR above. While getting the base URL from the config gets us most of the way there we'd like the provider to have all of the query params so it can create a full URL that includes any search parameters.

itemtypes.py builds the nextlink using these params directly as well. If we wanted to extract that out to a utility method that uses the params sent to the provider's query method we can use that.

We just don't want to lose anything when building the nextlink within the provider.

[webb-ben]

All query params should be passed to the provider. Any query param that matches

pygeoapi/pygeoapi/api/itemtypes.py

Lines 257 to 260 in e84a56c

	reserved_fieldnames = ['bbox', 'bbox-crs', 'crs', 'f', 'lang', 'limit',
	'offset', 'resulttype', 'datetime', 'sortby',
	'properties', 'skipGeometry', 'q',
	'filter', 'filter-lang', 'filter-crs']

will be parsed and passed as function arguments to the provider .query().

Any remaining parameter would be parsed as a property filter. It passes all parameters, regardless of if they are a valid field.

pygeoapi/pygeoapi/api/itemtypes.py

Lines 416 to 424 in e84a56c

	LOGGER.debug('processing property parameters')
	for k, v in request.params.items():
	if k not in reserved_fieldnames:
	if k in list(p.fields.keys()):
	LOGGER.debug(f'Adding property filter {k}={v}')
	else:
	LOGGER.debug(f'Adding additional property filter {k}={v}')
	properties.append((k, v))

[allenj]

I like this idea of utilizing what the provider is given. I haven't fully tested if the translations that happen with fields like filter and datetime before they get to the provider can be relied on for duplicating the query params. I wanted to avoid any of these "lossy" concerns with that with this PR. We can take a closer look at as it certainly smells bad that we'd be passing the entire query param to the provider.