gh-104683: Argument clinic: cleanup DLSParser state_foo methods

[AlexWaygood]

The DSLParser class has a lot of state_foo methods. All of these have to have the same type annotations, because none of them are actually called directly -- they're all called indirectly via the next() method:

cpython/Tools/clinic/clinic.py

Lines 4644 to 4652 in 6ef8f8c

	def next(
	self,
	state: StateKeeper,
	line: str | None = None
	) -> None:
	# real_print(self.state.__name__, "->", state.__name__, ", line=", line)
	self.state = state
	if line is not None:
	self.state(line)

cpython/Tools/clinic/clinic.py

Line 4670 in 6ef8f8c

self.next(self.state_modulename_name, line)

cpython/Tools/clinic/clinic.py

Line 4731 in 6ef8f8c

self.next(self.state_function_docstring)

(etc. etc.)

All of these are only ever passed a str for their line parameter, and assume that they're only ever passed a str for their line parameter. All, that is -- except one! The state_terminal method is explicitly passed None in the one place that it's called, and will indeed error if it's passed anything that's not None or "":

cpython/Tools/clinic/clinic.py

Lines 4614 to 4615 in 6ef8f8c

	self.next(self.state_terminal)
	self.state(None)

cpython/Tools/clinic/clinic.py

Lines 5576 to 5580 in 6ef8f8c

	def state_terminal(self, line: str | None) -> None:
	"""
	Called when processing the block is done.
	"""
	assert not line

As a result of state_terminal, all of our state_foo methods have to have str | None annotations for line, which means we need to have many pointless assertions everywhere to keep mypy happy. But since state_terminal is special (it's the only one of the state_foo methods that expects to be passed None, and it's only ever called once, after the entire block has been parsed), we can just... special-case it. If we rename it to do_post_block_processing_cleanup(), and call it explicitly after parsing the block rather than indirectly via next(), this allows us to greatly simplify the typing across the rest of the DSLParser state_foo methods, and makes the code generally much more readable for humans as well.

• Issue: Modernise code in Tools/clinic/ #104683

[erlend-aasland]

I don't like this function; it does too many things :)

1. it verifies that we have keyword params after '*'; this should be done in state_parameter() (IIUC, we've got what we need to catch this there)
2. it removes trailing whitespace from the param docstrings; this should be done in state_parameter_docstring()
3. it reformats the docstring and assigns it to the function

IMO, only 3. should be part of the post-processing. Validation should be done by the state machine; that's one of the features of a well-behaving state machine :)

[AlexWaygood]

You know where to file an issue about these problems ;)

[erlend-aasland]

See gh-107550 for 2. Reworking 1. turned out to only make the code worse :(

[erlend-aasland]

It took me some time to wrap my head around the state machine. AFAICS, this change is correct. Thanks!