Limit the number of generated errors if stubs are missing

[JukkaL]

If stubs for third-party libraries have not been installed, mypy may generate thousands of errors, making it hard to figure out what's going on. One reason for this is that missing stubs cause many Any types to be inferred, and these may trigger additional errors, depending on the strictness options being used.

One way to address this is to use some heuristic to silence errors that are not about missing stubs after a threshold. For example, if we've reported N errors about missing stubs and at least M errors overall, hide any remaining errors that are not about stubs.

This could be a significant issue for some users after the next mypy release (0.900), which unbundles stubs for third-party libraries. Thus users are likely to see errors about missing stubs as soon they update to mypy 0.900.

[TheCleric]

@JukkaL possibly stupid question, but is this already happening? I tried to create a small reproducible test where I imported some of these packages and did some basic type checking against their functions, but all I'm seeing is the stub errors.

I'm sure I'm either misunderstanding or doing something wrong, but can you at least point me in the right direction with a small reproducible example (i.e., one import stub error + one other mypy error off of that import) so I can build on it?

Here's a test I tried adding to check-modules.test to start with to get the errors and figure out where/how to limit them:

[case testLimitLegacyStubErrorVolume]
# flags: --config-file tmp/pyproject.toml
from typing import Dict

import certifi
import chardet

where = certifi.where()  # type: str
certifi.contents()  # type: str
chardet.detect(b'chardet')  # type: Dict[str, str]

[file pyproject.toml]
\[tool.mypy]
allow_redefinition = false
allow_untyped_globals = false
check_untyped_defs = true
disallow_any_decorated = true
disallow_any_explicit = true
disallow_any_expr = true
disallow_any_generics = true
disallow_any_unimported = true
disallow_incomplete_defs = true
disallow_subclassing_any = true
disallow_untyped_calls = true
disallow_untyped_decorators = true
disallow_untyped_defs = true
disallow_subclassing_any = true
follow_imports = 'normal'
ignore_missing_imports = false
local_partial_types = true
no_implicit_optional = true
no_implicit_reexport = true
no_strict_optional = false
strict = true
strict_equality = true
strict_optional = true
warn_no_return = true
warn_redundant_casts = true
warn_return_any = true
warn_unreachable = true
warn_unused_configs = true

[builtins fixtures/tuple.pyi]
[builtins fixtures/dict.pyi]

But all I see is the stub errors (even though I was expecting to see errors on the function calls where I would be assigning an implicit Any to the typed vars).

[JukkaL]

Hmm not sure what is going on there. I got syntax errors when I tried the test case, but after fixing them I get errors about missing stubs and Any types, as expected.

Anyway, I have an implementation now and will prepare a PR shortly.