argparse: mutually exclusive groups full of help-suppressed args can cause AssertionErrors

[gholms]

BPO	17890
Nosy	@terryjreedy, @bitdancer, @vadmium, @iritkatriel
Files	argparse-assertfail.py: Test case 7691d1d4b955.diff: gholms repo Issue17890-27.patch: Resolution for Python 2.7 issue17890

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-05-02.00:43:02.387>
labels = ['type-bug', 'library']
title = 'argparse: mutually exclusive groups full of help-suppressed args can cause AssertionErrors'
updated_at = <Date 2021-12-10.16:06:27.619>
user = 'https://bugs.python.org/gholms'

bugs.python.org fields:

activity = <Date 2021-12-10.16:06:27.619>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2013-05-02.00:43:02.387>
creator = 'gholms'
dependencies = []
files = ['30102', '30108', '30110']
hgrepos = ['187']
issue_num = 17890
keywords = ['patch']
message_count = 17.0
messages = ['188250', '188253', '188278', '188291', '188295', '188306', '188337', '188379', '188413', '188984', '189054', '189064', '189065', '189355', '193191', '259189', '408217']
nosy_count = 8.0
nosy_names = ['terry.reedy', 'bethard', 'r.david.murray', 'martin.panter', 'paul.j3', 'Yogesh.Chaudhari', 'gholms', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'patch review'
status = 'pending'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue17890'
versions = ['Python 2.7', 'Python 3.3', 'Python 3.4']

Linked PRs

• gh-62090: Simplify argparse usage formatting #105039

[gholms]

When it goes to format a usage message, argparse seems to (correctly) fail to satisfy one of its assertions when all of the following are true:

1. A mutually exclusive group contains only args that are suppressed
2. An unsuppressed arg follows that group
3. The usage is long enough to need to line-wrap

The cause seems to be that the set of regular expressions that argparse uses to clean up mutually exclusive groups' separators doesn't handle the space that follows what would otherwise be an empty pair of square braces, sort of like this:

1. [-h] [ ] [--spam] ...
2. [-h] [] [--spam] ...
3. [-h] [--spam] ...

A test case is attached. I was able to reproduce this with python-2.7.3-13.fc18.x86_64 on Fedora as well as with commit 83588:e6b962fa44bb in 3.4 mainline. I have a small patch for the latter that I'll submit shortly.

Sorry if I missed anything. This is my first bug report against python proper.

[paulj3]

Looks like the

text = text.strip()

at the end of the set of regex (in _format_actions_usage) needs to be replaced with something that removes all excess spaces, e.g.

text = _re.sub( '\s+', ' ', text ).strip()

[YogeshChaudhari]

Made similar required changes for version 2.7

[paulj3]

In the test case: class TestMutuallyExclusiveManySuppressed
even with a short 'eggs' argument, there is a difference

Old usage would be:

usage: PROG [-h] [--eggs EGGS]

new

usage: PROG [-h] [--eggs EGGS]

i.e. 2 v 1 space. But extra spaces are not as dramatic a failure as an assertion error.

It would also be good to check what happens when there are 2 suppressed groups. If the text before all trimming is:

[ -h ] [] () [ --eggs EGGS ]

does it reduce to?

[-h] [--eggs EGGS]

The old code would have left 3 spaces.

I can't think of a situation in which a user would want a (generated) usage line with multiple spaces. If some sort of special formatting is needed, there is always the option of specifying an explicit usage line.

parser = ArugmentParser(usage='one \ttwo  \nthree   four')

[YogeshChaudhari]

A user generated line with multiple spaces may be essential if we are getting the usage data (particularly from) a csv file (that may have random spaces in certain fields) or some other form of stored or auto-generated data.

[paulj3]

I see three solutions -

1. gholms' patch which removes '() ' and [] '

2. Yogesh's patch which removes all duplicated spaces.

3. remove the 2 asserts.

The first 2 do the same thing most of the time, but may differ if the user somehow inserts spaces into names. The third leaves the extra blanks, but renders them innocuous. I doubt if the asserts were written to catch this problem. They probably were included to verify the usage line had been split up as expected prior to reassembling on multiple lines.

As best I can tell test_argparse.py does not test for these spaces. Curiously though a port of argparse to javascript does have a test case with the extra space.

[terryjreedy]

Garrett and Yogesh, please submit contributor license agreements
http://www.python.org/psf/contrib/contrib-form/
if you have not yet. When one is properly recorded, a * appears after your name.

[YogeshChaudhari]

I am not sure if I am missing something. I had filled out the form at http://www.python.org/psf/contrib/contrib-form/ on the day I submitted the patch and even got back an email from Ewa Jodlowska. However, I don't see any "*" after my name. I submitted the same form to contributors@python.org and shot a mail to python-dev mailing list but there is no change. Any suggestions?

[terryjreedy]

Wait a week and see what happens.

[YogeshChaudhari]

@terry: Thanks for the info. I seem to have the elusive "*" after my username now. I am not sure how this works, but can you review/test/apply the patch now?

[terryjreedy]

Argparse is out of my area of competence/experience. I have added a couple of people who have worked on argparse in the past and should be better able to review or suggest another reviewer.

[paulj3]

I'm following a dozen argparse issues with patches. I haven't seen much posting by argparse experts (like bethard, david.murry) since last December.

[YogeshChaudhari]

@terry and @paul:
I have 2 argparse related patches, sitting idle for quiet sometime now. Can you suggest any more names who can take this forward? (I could not find anyone else related to argparse in http://docs.python.org/devguide/experts.html)

[bitdancer]

I've been observing the activity on the argparse issues and am appreciating the work, but I don't have time right now to review the patches. I should have more time next month, and expect to get to them then, if no one else gets to them before I do.

[paulj3]

I just submitted a patch to http://bugs.python.org/issue11874 that substantially rewrites _format_actions_usage(). It generates the group and action parts separately, and does not do the kind of cleanup that produces this issue.

[vadmium]

See duplicate bpo-22363 for the traceback, and a workaround. There is also a patch there similar to Garrett’s original fix, but using the RE r'%s *%s ?'.

I restored the diff from Garrett’s repository, in case it is still useful.

Yogesh: I am not familiar with how the code works, and I struggle to be excited at using regular expressions to build a usage message :). But can you explain why you changed Garrett’s original fix? Does it affect intentional double spaces (or tabs, non-ASCII spaces, etc) in customizable parts of the usage message?

[iritkatriel]

I'm unable to reproduce this problem on 3.11:

>>> import argparse
>>> parser = argparse.ArgumentParser()
>>> group = parser.add_mutually_exclusive_group()
>>> group.add_argument('--spam', help=argparse.SUPPRESS)
_StoreAction(option_strings=['--spam'], dest='spam', nargs=None, const=None, default=None, type=None, choices=None, help='==SUPPRESS==', metavar=None)
>>> parser.add_argument('--' + 'eggs' * 20, dest='eggs')
_StoreAction(option_strings=['--eggseggseggseggseggseggseggseggseggseggseggseggseggseggseggseggseggseggseggseggs'], dest='eggs', nargs=None, const=None, default=None, type=None, choices=None, help=None, metavar=None)
>>> parser.print_usage()
usage: [-h]  [--eggseggseggseggseggseggseggseggseggseggseggseggseggseggseggseggseggseggseggseggs EGGS]
>>>

[encukou]

The reproducer needs a narrow enough console, or a bigger number for the 'eggs' * 20.

This was fixed in #96311.