str.split(): allow removing empty strings (when sep is not None)

[warsaw]

BPO	28937
Nosy	@gvanrossum, @warsaw, @karlcow, @serhiy-storchaka, @catherinedevlin, @MarkCBell, @vedgar, @csabella, @corona10, @ZackerySpytz, @akulakov
PRs	gh-73123: Test for keepempty argument to string split #26196 gh-73123: Add a keepempty argument to string, bytes and bytearray split methods #26222
Files	split_prune_1.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ZackerySpytz'
closed_at = None
created_at = <Date 2016-12-11.15:11:42.106>
labels = ['type-feature', 'library', '3.10']
title = 'str.split(): allow removing empty strings (when sep is not None)'
updated_at = <Date 2021-06-16.14:27:14.201>
user = 'https://github.com/warsaw'

bugs.python.org fields:

activity = <Date 2021-06-16.14:27:14.201>
actor = 'andrei.avk'
assignee = 'ZackerySpytz'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2016-12-11.15:11:42.106>
creator = 'barry'
dependencies = []
files = ['45863']
hgrepos = []
issue_num = 28937
keywords = ['patch']
message_count = 48.0
messages = ['282923', '282925', '282926', '282927', '282928', '282929', '282930', '282931', '282932', '282935', '282936', '282954', '282958', '282961', '282962', '282963', '282966', '283011', '283019', '283021', '283025', '283029', '283031', '338412', '338422', '338429', '354907', '354908', '384306', '384321', '384337', '393822', '393871', '393883', '393889', '393892', '393896', '393902', '394012', '394128', '394928', '395159', '395167', '395169', '395184', '395193', '395194', '395926']
nosy_count = 14.0
nosy_names = ['gvanrossum', 'barry', 'syeberman', 'mrabarnett', 'karlcow', 'serhiy.storchaka', 'Catherine.Devlin', 'Mark.Bell', 'veky', 'cheryl.sabella', 'corona10', 'ZackerySpytz', 'Philippe Cloutier', 'andrei.avk']
pr_nums = ['26196', '26222']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue28937'
versions = ['Python 3.10']

[warsaw]

This has finally bugged me enough to file an issue, although I wouldn't be able to use it until Python 3.7. There's a subtle but documented difference in str.split() when sep=None:

>>> help(''.split)
Help on built-in function split:

split(...) method of builtins.str instance
S.split(sep=None, maxsplit=-1) -> list of strings

Return a list of the words in S, using sep as the
delimiter string.  If maxsplit is given, at most maxsplit
splits are done. If sep is not specified or is None, any
whitespace string is a separator and empty strings are
removed from the result.

I.e., that empty strings are removed from the result. This does not happen when sep is given, leading to this type of unfortunate code:

>>> 'foo,bar,baz'.split(',')
['foo', 'bar', 'baz']
>>> 'foo,bar,baz'.replace(',', ' ').split()
['foo', 'bar', 'baz']
>>> ''.split(',')
['']
>>> ''.replace(',', ' ').split()
[]

Specifically, code that wants to split on say commas, but has to handle the case where the source string is empty, shouldn't have to also filter out the single empty string item.

Obviously we can't change existing behavior, so I propose to add a keyword argument prune that would make these two bits of code identical:

>>> ''.split()
[]
>>> ''.split(' ', prune=True)
[]

and would handle the case of ''.split(',') without having to resort to creating an ephemeral intermediate string.

prune should be a keyword-only argument, defaulting to False.

[Vgr255]

I understand the feeling. However, in a project I maintain, we want the other way around - to be able to never have an empty list, even if the string is empty (we resorted to using re.split in the end, which has this behaviour). Consider:

rest = re.split(" +", rest)[0].strip()

This gives us None-like behaviour in splitting, at the cost of not actually using str.split.

I'm +1 on the idea, but I'd like some way to better generalize str.split use (not everyone knows you can pass None and/or an integer).

(At the same time, the counter arguments where str has too many methods, or that methods shouldn't do too much, also apply here.)

But I don't like bikeshedding too much, so let's just count me as +1 for your way, if there's no strong momentum for mine :)

[serhiy-storchaka]

Current behavior is consistent with str.count():

len(string.split(sep)) == string.count(sep) + 1

and re.split():

re.split(re.escape(sep), string) == string.split(sep)

May be the behavior when sep is None should be changed for consistency with the behavior when sep is not None?

[warsaw]

On Dec 11, 2016, at 03:32 PM, Serhiy Storchaka wrote:

Current behavior is consistent with str.count():

len(string.split(sep)) == string.count(sep) + 1

and re.split():

re.split(re.escape(sep), string) == string.split(sep)

Yep. My suggestion is a straight up 'practicality beats purity' request.

May be the behavior when sep is None should be changed for consistency with
the behavior when sep is not None?

I'm very strongly -1 on changing any existing behavior.

[Vgr255]

Changing the behaviour when sep is None is a big backwards-compatibility break, and I'm not sure we'd even want that. It's logical to allow passing None to mean the same thing as NULL (i.e. no arguments), and the behaviour in that case has been like that for... well, long enough that changing it isn't really feasible.

I agree with Barry here, especially since this is a completely opt-in feature, and existing behaviour isn't changed without the user's knowledge.

[serhiy-storchaka]

I meant adding boolean argument that changes the behavior when sep is None, not when it is not None.

[Vgr255]

That would work for my case, but it wouldn't for Barry's (unless I missed something). He wants a non-None argument to not leave empty strings, but I want a None argument to leave empty strings... I don't think there's a one-size-fits-all solution in this case, but feel free to prove me wrong :)

[warsaw]

On Dec 11, 2016, at 03:57 PM, Serhiy Storchaka wrote:

I meant adding boolean argument that changes the behavior when sep is None,
not when it is not None.

Ah, I understand now, thanks. However, I'm not sure that addresses my
particular use case. It's actually kind of handy to filter out the empty
strings. But I'm open to counter arguments.

[Vgr255]

Actually, there might be a way. We could make prune default to True if sep is None, and default to False if sep is not None. That way, we get to keep the existing behaviour for either case, while satisfying both of our use cases :)

If that's a bad idea (and it quite probably is), I'll retract it. But it's an interesting possibility to at least consider.

[mrabarnett]

So prune would default to None?

None means current behaviour (prune if sep is None else don't prune)
True means prune empty strings
False means don't prune empty string

[rhettinger]

A few randomly ordered thoughts about splitting:

• The best general purpose text splitter I've ever seen is in MS Excel and is called "Text to Columns". It has a boolean flag, "treat consecutive delimiters as one" which is off by default.

• There is a nice discussion on the complexities of the current design on StackOverflow: http://stackoverflow.com/questions/16645083 In addition, there are many other SO questions about the behavior of str.split().

• The learning curve for str.split() is already high. The doc entry for it has been revised many times to try and explain what it does. I'm concerned that adding another algorithmic option to it may make it more difficult to learn and use in the common cases (API design principle: giving users more options can impair usability). Usually in Python courses, I recommend using str.split() for the simple, common cases and using regex when you need more control.

• What I do like about the proposal is that that there is no clean way to take the default whitespace splitting algorithm and customize to a particular subset of whitespace (i.e. tabs only).

• A tangential issue is that it was a mistake to expose the maxsplit=-1 implementation detail. In Python 2.7, the help was "S.split([sep [,maxsplit]])". But folks implementing the argument clinic have no way of coping with optional arguments that don't have a default value (like dict.pop), so they changed the API so that the implementation detail was exposed, "S.split(sep=None, maxsplit=-1)". IMO, this is an API regression. We really don't want people passing in -1 to indicate that there are no limits. The Python way would have been to use None as a default or to stick with the existing API where the number of arguments supplied is part of the API (much like type() has two different meanings depending on whether it has an arity of 1 or 3).

Overall, I'm +0 on the proposal but there should be good consideration given to 1) whether there is a sufficient need to warrant increasing API complexity, making split() more difficult to learn and remember, 2) considering whether "prune" is the right word (can someone who didn't write the code read it clearly afterwards), 3) or addressing this through documentation (i.e. showing the simple regexes needed for cases not covered by str.split).

[Vgr255]

Matthew: Yes, that's exactly the way I was going about it.

Thank you Raymond for your comments (and your informative answer on that SO question).

I think that part of the problem is that no delimiter (or None) behaves differently than with a delimiter. If we wanted proper consistency, we would have needed to make passing None (or nothing) the same as passing whitespace, but alas, we have to work with what we have.

As you said, API complexity is a concern that needs to be addressed. I think that the most important part is how it's documented, and, if phrased correctly (which is non-trivial), could actually simplify the explanation.

Consider this draft:

---

The value of the prune keyword argument determines whether or not consecutive delimiters should be grouped together. If prune is not given or None, it defaults to True if sep is None or not given, and False otherwise.

If prune is True, consecutive delimiters (all whitespace if None or not given) are regarded as a single separator, and the result will not contain any empty string. The resulting list may be empty.

Otherwise, if prune is False, consecutive delimiters are not grouped together, and the result may contain one or more empty string. The resulting list will never be empty.

---

I may be oversimplifying this, but it seems to me that this might help some people by hopefully streamlining the explanation.

This still doesn't solve the issue where a user can't say "split on a space or a tab, but not newlines", which IMO is lacking in the design, but that may be for another issue ;)

I've wrapped up a basic patch which probably doesn't work at all; I'll put it up when it's at least partly working for everyone to look at.