Support deep objects for query parameters with deepObject style

[bajtos]

Background

Many applications expect deeply nested objects in input parameters, see the discussion in swagger-ui starting from this comment: swagger-api/swagger-ui#4064 (comment) In LoopBack, we are running into this problem too, see loopbackio/loopback-next#1679.

Consider a filter parameter defined as follows:

parameters:
 filterParam:
   in: query
   name: filter
   schema:
     type: object
   style: deepObject
   explode: true
   description: Options for filtering the results
   required: false

Let's say the user wants to provide the following value, for example by entering a JSON into the text area rendered by swagger-ui:

{
  "name": "ivan",
  "birth-date": {
    "gte": "1970-01-01"
  }
}

At the moment, the OpenAPI Specification v3 does not describe how to encode such value in a query string. As a result, OpenAPI clients like swagger-ui don't know how to handle this input - see the discussion in swagger-api/swagger-js#1385 for an example.

Proposal

The following query string should be created by swagger-js client for the input value shown above.

filter[name]=ivan&filter[birth-date][qte]=1970-01-01

The proposed serialization style is supported by https://www.npmjs.com/package/qs, which is used by http://expressjs.com as the default query parser, which means that a vast number of Node.js API servers are already expecting this serialization style.

I am not sure if there is any existing formal specification of this format, I am happy to look into that once my proposal gets considered as acceptable in principle.

Additional information

Existing issues that are touching a similar topic:

• style: form + object looses the parameter name? style: form + object looses the parameter name? #1006
• Are indexes in the query parameter array representable? Are indexes in the query parameter array representable? #1501

Two older comments from swagger-js that may be relevant:

swagger-api/swagger-js#1140

Limitations:
deepObject does not handle nested objects. The specification and swagger.io documentation does not provide an example for serializing deep objects. Flat objects will be serialized into the deepObject style just fine.

swagger-api/swagger-js#1140 (comment)

As for deepObject and nested objects - that was explicitly left out of the spec, and it's ok to just Not Support It™.

[earth2marsh]

Are there any other examples of these nested deepObjects outside of Express? The more widespread a pattern, the more likely it is to be considered. Myself, I have an aversion to passing such a complicated object in the query string. Any insight into why Express even landed on this pattern?

[darrelmiller]

We were a bit stuck when allowing the deepObject style. We understood that users wanted this capability but there is no standard definition of what that serialization format looks like. We had a few choices, allow it and define our own standard and hope implementations followed it. Don't allow it because there are no standards, or add it in, say nothing about its format and hope that a default serialization format emerges.

If we can get some confidence that the qs package is becoming a defacto standard and we can create an accurate description of the serialization, then I have no issue recommending that we include that description in a future minor release of the spec.

[rmunix]

What about the URL max length limits? I think one of the reasons why a standard for object serialization in the URL is hard to materialize is because of the URL max length problem, the URL just wasn't intended to pass data in this way. Depending on the browser and/or server software being used the URL max length varies but in general it is small when compared to how much data can be transmitted with other methods like POST or PUT. It certainly would work for small objects but people tend to inadvertently abuse these kinds of APIs by passing bigger than allowed payloads.

[handrews]

@rmunix regarding URL max length problems, I was very happy to see the HTTP SEARCH method draft RFC revived last month: https://tools.ietf.org/html/draft-snell-search-method-01

[louisl]

Are there any other examples of these nested deepObjects outside of Express? The more widespread a pattern, the more likely it is to be considered. Myself, I have an aversion to passing such a complicated object in the query string. Any insight into why Express even landed on this pattern?

http://esbenp.github.io/2016/04/15/modern-rest-api-laravel-part-2/

I use a modified version of https://github.com/esbenp/bruno referenced in the article above in a few apis, it's extremely useful for including related data and search filtering. I'm not really sure how to define those in a spec. I do appreciate that the url strings could get silly long to the point of failing if abused, but without this sort of thing searches and such would have to be actioned as POST requests or limited to basic GET request params. It seems to me no ones really come up with a holy grail API solution for complex search queries so it's a bit of a free for all at the moment.

[bajtos]

Ruby on Rails

Seems to use the same approach.

Reference: https://edgeapi.rubyonrails.org/classes/Hash.html#method-i-to_query
Source code: https://github.com/rails/rails/blob/b5302d5a820b078b6488104dd695a679e5a49623/activesupport/lib/active_support/core_ext/object/to_query.rb#L61-L86

Example code:

require "activesupport"

data = {
  "name" => "David",
  "nationality" => "Danish",
  "address" => {
    "street" => "12 High Street",
    "city" => "London",
  },
  "location" => [10, 20],
}
print data.to_query("person")

Produces the following query string, I have urldecoded and reformatted it for better readability:

person[address][city]=London&
person[address][street]=12+High+Street&
person[location][]=10&
person[location][]=20&
person[name]=David&
person[nationality]=Danish

Notice that array items are using an empty index, i.e. person[location][]=10, instead of person[location][0]=10.

[bajtos]

Python 2.7

AFAICT, Python does not support nested objects in query parameters.

Example code:

from urllib import urlencode
from urlparse import parse_qs

data = {
  'person': {
    'name': 'David',
    'nationality': 'Danish',
    'address': {
      'street': '12 High Street',
      'city': 'London',
    },
    'location': [10, 20],
  }
}

print urlencode(data)

Produces the following query string, I have urldecoded and reformatted it for better readability:

person={
  'nationality':+'Danish',+
  'location':+[10,+20],+
  'name':+'David',+
  'address':+{
    'city':+'London',+
    'street':+'12+High+Street'
  }
}

Interestingly enough, the roundtrip does not preserve the original data.

print parse_qs(urlencode(data))

Outcome:

{'person': ["{'nationality': 'Danish', 'location': [10, 20], 'name': 'David', 'address': {'city': 'London', 'street': '12 High Street'}}"]}

Another example:

print parse_qs('foo[bar]=1&foo[baz]=2')
# {'foo[baz]': ['2'], 'foo[bar]': ['1']}

[louisl]

Not API specific, but jQuery can generate nested array url params from objects.

http://api.jquery.com/jquery.param/

[Stratus3D]

JSONAPI sparse fieldsets require this: https://jsonapi.org/format/#fetching-sparse-fieldsets

[benhaynes]

Hey @earth2marsh and @darrelmiller ... We (Directus team) have been trying to use OpenAPI 3.0 for a while now, but the lack of support for nested deepObjects has kept us from using this spec. We have a dynamic API that allows for relatively complex filtering, for example: filter[<field-name>][<operator>]=<value>

GET /items/users?filter[category][eq]=vip

Our API Reference for this filtering

Is there any hope for this being supported in the future or should we "move on"?