json dump fails for mixed-type keys when sort_keys is specified

[tanzerswingcoat]

BPO	25457
Nosy	@zpincus, @bitdancer, @MojoVampire, @jheiv, @aaronchall, @naught101, @akulakov
PRs	bpo-25457: Allow json.encode() to handle mixed keys when sort_keys=True #8011 bpo-25457: Remove sorting bug in json._iterencode_dict #15691

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2015-10-22.08:01:01.647>
labels = ['3.7', '3.8', 'type-bug', 'library']
title = 'json dump fails for mixed-type keys when sort_keys is specified'
updated_at = <Date 2021-07-10.17:34:02.379>
user = 'https://bugs.python.org/tanzerswingcoat'

bugs.python.org fields:

activity = <Date 2021-07-10.17:34:02.379>
actor = 'tanzer@swing.co.at'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2015-10-22.08:01:01.647>
creator = 'tanzer@swing.co.at'
dependencies = []
files = []
hgrepos = []
issue_num = 25457
keywords = ['patch']
message_count = 17.0
messages = ['253324', '253360', '253362', '253363', '253368', '253369', '297246', '317187', '317216', '317240', '317241', '317243', '317246', '320706', '384635', '397249', '397250']
nosy_count = 8.0
nosy_names = ['zachrahan', 'r.david.murray', 'josh.r', 'jedwards', 'tanzer@swing.co.at', 'Aaron Hall', 'naught101', 'andrei.avk']
pr_nums = ['8011', '15691']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue25457'
versions = ['Python 3.7', 'Python 3.8']

[tanzerswingcoat]

In Python 3, trying to json-dump a dict with keys of different types fails with a TypeError when sort_keys is specified:

python2.7
===========

Python 2.7.10 (default, May 29 2015, 10:02:30) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import json
>>> json.dumps({1 : 42, "foo" : "bar", None : "nada"}, sort_keys = True)
'{"null": "nada", "1": 42, "foo": "bar"}'

python3.5
============

Python 3.5.0 (default, Oct  5 2015, 12:03:13) 
[GCC 4.8.5] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import json
>>> json.dumps({1 : 42, "foo" : "bar", None : "nada"}, sort_keys = True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.5/json/__init__.py", line 237, in dumps
    **kw).encode(obj)
  File "/usr/lib64/python3.5/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib64/python3.5/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
TypeError: unorderable types: str() < int()

Note that the documentation explicitly allows keys of different, if basic, types:

If skipkeys is True (default: False), then dict keys that are not of a basic type (str, int, float, bool, None) will be skipped instead of raising a TypeError.

As all they keys are dumped as strings, a simple solution would be to sort after converting to strings. Looking closely at the output of Python 2, the sort order is a bit strange!

[MojoVampire]

The Python 2 sort order is a result of the "arbitrary but consistent fallback comparison" (omitting details, it's comparing the names of the types), thus the "strange" sort order. Python 3 (justifiably) said that incomparable types should be incomparable rather than silently behaving in non-intuitive ways, hiding errors.

Python is being rather generous by allowing non-string keys, because the JSON spec ( http://json.org/ ) only allows the keys ("names" in JSON parlance) to be strings. So you're already a bit in the weeds as far as compliant JSON goes if you have non-string keys.

Since mixed type keys lack meaningful sort order, I'm not sure it's wrong to reject attempts to sort them. Converting to string is as arbitrary and full of potential for silently incorrect comparisons as the Python 2 behavior, and reintroducing it seems like a bad idea.

[MojoVampire]

As a workaround (should you absolutely need to sort keys by some arbitrary criteria), you can initialize a collections.OrderedDict from the sorted items of your original dict (using whatever key function you like), then dump without using sort_keys=True. For example, your suggested behavior (treat all keys as str) could be achieved by the user by replacing:

    json.dumps(mydict, sort_keys=True)

with:

json.dumps(collections.OrderedDict(sorted(mydict.items(), key=str)))

Particularly in 3.5 (where OrderedDict is a C builtin), this shouldn't incur too much additional overhead (sort_keys has to make a sorted list intermediate anyway), and the output is the same, without introducing implicit hiding of errors.

[MojoVampire]

Oops, minor flaw with that. It's str-ifying the tuples, not the keys, which could (in some cases) cause issues with keys whose reprs have different quoting. So you'd end up with lambdas. Boo. Anyway, corrected version (which would probably not be one-lined in real code):

json.dumps(collections.OrderedDict(sorted(mydict.items(), key=lambda x: str(x[0]))))

[tanzerswingcoat]

Josh Rosenberg wrote at Fri, 23 Oct 2015 02:45:51 +0000:

The Python 2 sort order is a result of the "arbitrary but consistent
fallback comparison" (omitting details, it's comparing the names of
the types), thus the "strange" sort order.

Thanks. I knew that.

Python 3 (justifiably) said that incomparable types should be
incomparable rather than silently behaving in non-intuitive ways,
hiding errors.

"justifiably" is debatable. I consider the change ill-conveived.

Displaying a dictionary (or just its keys) in a readable, or just
reproducible, way is useful in many contexts. Python 3 broke that for
very little, INMNSHO, gain.

I consider "hiding errors" a myth, to say it politely!

Python is being rather generous by allowing non-string keys, because
the JSON spec ( http://json.org/ ) only allows the keys ("names" in
JSON parlance) to be strings. So you're already a bit in the weeds as
far as compliant JSON goes if you have non-string keys.

There are two possibilities:

1. Accepting non-string keys is intended. Then sort_keys shouldn't
   break like it does.

   As far as JSON goes, the output of json.dump[s] contains string keys.

2. Accepting non-string keys is a bug. Then json.dump[s] should be
   changed to not accept them.

Mixing both approaches is the worst of all worlds.

Since mixed type keys lack meaningful sort order, I'm not sure it's
wrong to reject attempts to sort them.

The documentation says:

If sort_keys is True (default False), then the output of dictionaries
will be sorted by key; this is useful for regression tests to ensure
that JSON serializations can be compared on a day-to-day basis.

**Reproducible** is the keyword here.

**Readability** is another one. Even if the sort order is "strange",
it is much better than random order, if you are looking for a specific
key.

For the record, it was a test failing under Python 3.5 that triggered
this bug report.

> As all they keys are dumped as strings, a simple solution would be to
> sort after converting to strings.
Converting to string is as
arbitrary and full of potential for silently incorrect comparisons as
the Python 2 behavior, and reintroducing it seems like a bad idea.

json.dumps already does the conversion::

    >>> json.dumps({1 : 42, "foo" : "bar", None : "nada"})
    '{"foo": "bar", "1": 42, "null": "nada"}'

Another run::

    >>> json.dumps({1 : 42, "foo" : "bar", None : "nada"})
    '{"1": 42, "foo": "bar", "null": "nada"}'

That difference is exactly the reason for sort_keys.

[tanzerswingcoat]

Josh Rosenberg wrote at Fri, 23 Oct 2015 02:56:30 +0000:

As a workaround (should you absolutely need to sort keys by some
arbitrary criteria), you can initialize a collections.OrderedDict from
the sorted items of your original dict (using whatever key function
you like), then dump without using sort_keys=True.

Sigh...

I already implemented a workaround but it's not as simple as you
think — the dictionary in question is nested.

The problem is that this is just another unnecessary difficulty when
trying to move to Python 3.x.

[zpincus]

This one just bit me too. It seems that if JSON serialization accepts non-string dict keys, it should make sure to accept them in all circumstances. Currently, there is an error *only* with mixed-type dicts, *only* when sort_keys=True.

In addition, the error raised in such cases is especially unhelpful. Running the following:
json.dumps({3:1, 'foo':'bar'}, sort_keys=True)

produces a stack trace that terminates in a function defined in C, with this error:
TypeError: '<' not supported between instances of 'str' and 'int'

That error doesn't give non-experts very much to go on...!

The fix is reasonably simple: coerce dict keys to strings *before* trying to sort the keys, not after. The only fuss in making such a patch is that the behavior has to be fixed in both _json.c and in json/encode.py.

The only other consistent behavior would be to disallow non-string keys, but that behavior is at this point very well entrenched. So it only makes sense that encoding should be patched to not fail in corner cases.

[aaronchall]

Now that dicts are sortable, does that make the sort_keys argument redundant?

Should this bug be changed to "won't fix"?

[zpincus]

Well, "wontfix" would be appropriate in the context of deprecating the sort_keys option (over the course of however many releases) and documenting that the new procedure for getting JSON output in a specific order is to ensure that the input dict was created in that order.

Certainly for regression testing, sort_keys is no longer needed, but that's not the only reason people are using that option. (It's certainly not why I use the option -- my use stems from sort_keys improving human readability of the JSON.)

But outside of deprecating sort_keys wholesale, it is still a bug that sort_keys=True can cause an error on input that would otherwise be valid for json.dump[s].

[tanzerswingcoat]

Aaron Hall wrote at Sun, 20 May 2018 16:49:06 +0000:

Now that dicts are sortable, does that make the sort_keys argument redundant?

Should this bug be changed to "won't fix"?

https://bugs.python.org/issue25457#msg317216 is as good an answer as I
could give.

Considering that I openend the bug more than 2.5 years ago, it doesn't
really matter though.

[bitdancer]

I'm fairly certain (though not 100%, obviously :) that coercing first and then sorting would be accepted if someone wants to create a PR for this.