Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MAINTENANCE] only provide py.typed files for fully typed sub-packages #7438

Merged

Conversation

Kilo59
Copy link
Member

@Kilo59 Kilo59 commented Mar 23, 2023

Changes proposed in this pull request:

  • remove top-level py.typed from the top-level great_expectations package
  • only provide py.typed file markers for sub-packages that are fully typed
    • datasource/fluent - datasource/ has one untyped module that prevents us from putting it at the top-level
    • execution_engine

https://peps.python.org/pep-0561/#packaging-type-information

See below for the modules we currently don't type-check.
Note: This doesn't mean that anything not in this list is "typed" it just means we run the type-checker against it. So any types we do declare should be correct. See mypy section on "dynamically typed code"

exclude = [
# BEGIN ALWAYS EXCLUDE SECTION #####################################################
# If pattern should always be excluded add comment explaining why and put
'docs/', # Docs should not be type checked with the rest of the library.
'_version\.py', # generated by `versioneer`
'v012', # legacy code
'tests/datasource/fluent/test_metadatasource\.py', # metaprogramming leads to errors
# END ALWAYS EXCLUDE SECTION ######################################################
#
# #################################################################################
# TODO: complete typing for the following modules and remove from exclude list
# number is the current number of typing errors for the excluded pattern
'checkpoint/actions\.py', # 18
'checkpoint/checkpoint\.py', # 22
'checkpoint/types/checkpoint_result\.py', # 34
'checkpoint/util\.py', # 5
'cli/checkpoint\.py', # 9
'cli/suite\.py', # 24
'cli/upgrade_helpers/upgrade_helper_v11\.py', # 59
'cli/upgrade_helpers/upgrade_helper_v13\.py', # 17
'core/batch\.py', # 29
'core/usage_statistics/anonymizers/action_anonymizer\.py', # 1
'core/usage_statistics/anonymizers/anonymizer\.py', # 6
'core/usage_statistics/anonymizers/base\.py', # 8
'core/usage_statistics/anonymizers/batch_anonymizer\.py', # 10
'core/usage_statistics/anonymizers/batch_request_anonymizer\.py', # 16
'core/usage_statistics/anonymizers/checkpoint_anonymizer\.py', # 16
'core/usage_statistics/anonymizers/data_connector_anonymizer\.py', # 3
'core/usage_statistics/anonymizers/data_docs_anonymizer\.py', # 5
'core/usage_statistics/anonymizers/datasource_anonymizer\.py', # 9
'core/usage_statistics/anonymizers/expectation_anonymizer\.py', # 6
'core/usage_statistics/anonymizers/profiler_anonymizer\.py', # 2
'core/usage_statistics/anonymizers/store_anonymizer\.py', # 6
'core/usage_statistics/anonymizers/store_backend_anonymizer\.py', # 5
'core/usage_statistics/anonymizers/validation_operator_anonymizer\.py', # 5
'core/usage_statistics/usage_statistics\.py', # 19
'dataset/sparkdf_dataset\.py', # 3
'dataset/sqlalchemy_dataset\.py', # 16
'datasource/data_connector/configured_asset_sql_data_connector\.py', # 47
'expectations/core/expect_column_values_to_be_of_type\.py', # 12
'expectations/core/expect_column_values_to_not_match_regex_list\.py', # 2
'expectations/core/expect_column_values_to_not_match_regex\.py', # 2
'expectations/core/expect_column_values_to_not_match_like_pattern_list\.py', # 3
'expectations/core/expect_column_values_to_not_match_like_pattern\.py', # 2
'expectations/core/expect_column_values_to_not_be_null\.py', # 2
'expectations/core/expect_column_values_to_not_be_in_set\.py', # 2
'expectations/core/expect_column_values_to_match_strftime_format\.py', # 2
'expectations/core/expect_column_values_to_match_regex_list\.py', # 2
'expectations/core/expect_column_values_to_match_regex\.py', # 1
'expectations/core/expect_column_values_to_match_like_pattern_list\.py', # 3
'expectations/core/expect_column_values_to_match_like_pattern\.py', # 2
'expectations/core/expect_column_values_to_match_json_schema\.py', # 1
'expectations/core/expect_column_values_to_be_unique\.py', # 1
'expectations/core/expect_column_values_to_be_null\.py', # 3
'expectations/core/expect_column_values_to_be_json_parseable\.py', # 1
'expectations/core/expect_column_values_to_be_increasing\.py', # 1
'expectations/core/expect_column_values_to_be_in_type_list\.py', # 11
'expectations/core/expect_column_values_to_be_in_set\.py', # 1
'expectations/core/expect_column_values_to_be_decreasing\.py', # 1
'expectations/core/expect_column_values_to_be_dateutil_parseable\.py', # 1
'expectations/core/expect_column_values_to_be_between\.py', # 3
'expectations/core/expect_column_unique_value_count_to_be_between\.py', # 1
'expectations/core/expect_column_to_exist\.py', # 4
'expectations/core/expect_column_stdev_to_be_between\.py', # 1
'expectations/core/expect_column_quantile_values_to_be_between\.py', # 15
'expectations/core/expect_column_value_lengths_to_equal\.py', # 1
'expectations/core/expect_column_value_lengths_to_be_between\.py', # 5
'expectations/core/expect_column_proportion_of_unique_values_to_be_between\.py', # 1
'expectations/core/expect_column_values_to_be_in_set\.py', # 2
'expectations/core/expect_column_values_to_be_equal\.py', # 3
'expectations/core/expect_column_values_a_to_be_greater_than_b\.py', # 3
'expectations/core/expect_column_pair_cramers_phi_value_to_be_less_than\.py', # 7
'expectations/core/expect_column_most_common_value_to_be_in_set\.py', # 3
'expectations/core/expect_column_min_to_be_between\.py', # 1
'expectations/core/expect_column_median_to_be_between\.py', # 1
'expectations/core/expect_column_mean_to_be_between\.py', # 1
'expectations/core/expect_column_max_to_be_between\.py', # 1
'expectations/core/expect_column_kl_divergence_to_be_less_than\.py', # 22
'expectations/core/expect_column_pair_values_to_be_in_set\.py', # 2
'expectations/core/expect_column_pair_values_to_be_equal\.py', # 3
'expectations/core/expect_column_pair_values_a_to_be_greater_than_b\.py', # 3
'expectations/core/expect_column_distinct_values_to_equal_set\.py', # 2
'expectations/core/expect_column_distinct_values_to_contain_set\.py', # 4
'expectations/core/expect_column_distinct_values_to_be_in_set\.py', # 1
'expectations/core/expect_compound_columns_to_be_unique\.py', # 3
'expectations/core/expect_multicolumn_sum_to_equal\.py', # 4
'expectations/core/expect_multicolumn_values_to_be_unique\.py', # 3
'expectations/core/expect_select_column_values_to_be_unique_within_record\.py', # 3
'expectations/core/expect_table_columns_to_match_set\.py', # 8
'expectations/core/expect_table_columns_to_match_ordered_list\.py', # 11
'expectations/core/expect_table_column_count_to_equal\.py', # 5
'expectations/core/expect_table_column_count_to_be_between\.py', # 3
'expectations/core/expect_table_row_count_to_equal_other_table\.py', # 11
'expectations/metrics/column_aggregate_metrics/column_value_counts\.py', # 6
'expectations/metrics/column_aggregate_metrics/column_quantile_values\.py', # 1
'expectations/metrics/column_aggregate_metrics/column_partition\.py', # 10
'expectations/metrics/column_aggregate_metrics/column_most_common_value\.py', # 4
'expectations/metrics/column_aggregate_metrics/column_median\.py', # 1
'expectations/metrics/map_metric_provider/column_condition_partial\.py', # 3
'expectations/metrics/map_metric_provider/column_function_partial\.py', # 9
'expectations/metrics/map_metric_provider/column_map_metric_provider\.py',
'expectations/metrics/map_metric_provider/column_pair_condition_partial\.py', # 3
'expectations/metrics/map_metric_provider/column_pair_function_partial\.py', # 9
'expectations/metrics/map_metric_provider/column_pair_map_metric_provider\.py',
'expectations/metrics/map_metric_provider/map_metric_provider\.py', # 2
'expectations/metrics/map_metric_provider/multicolumn_map_metric_provider\.py', # 2
'expectations/metrics/map_metric_provider/multicolumn_condition_partial\.py', # 3
'expectations/metrics/map_metric_provider/multicolumn_function_partial\.py', # 2
'expectations/metrics/map_metric_provider/column_map_condition_auxilliary_methods\.py', # 6
'expectations/metrics/map_metric_provider/map_condition_auxilliary_methods\.py', # 29
'expectations/metrics/map_metric_provider/multicolumn_map_condition_auxilliary_methods\.py', # 2
'expectations/regex_based_column_map_expectation\.py', # 3
'expectations/row_conditions\.py', # 4
'expectations/set_based_column_map_expectation\.py', # 3
'render/renderer/checkpoint_new_notebook_renderer\.py', # 9
'render/renderer/content_block/content_block\.py', # 5
'render/renderer/content_block/exception_list_content_block\.py', # 4
'render/renderer/datasource_new_notebook_renderer\.py', # 4
'render/renderer/notebook_renderer\.py', # 2
'render/renderer/page_renderer\.py', # 10
'render/renderer/profiling_results_overview_section_renderer\.py', # 2
'render/renderer/site_builder\.py', # 3
'render/renderer/slack_renderer\.py', # 9
'render/renderer/suite_edit_notebook_renderer\.py', # 7
'render/renderer/suite_scaffold_notebook_renderer\.py', # 7
'render/renderer/v3/suite_edit_notebook_renderer\.py', # 11
'render/renderer/v3/suite_profile_notebook_renderer\.py', # 4
'render/util\.py', # 17
'render/view/view\.py', # 11
'rule_based_profiler/data_assistant_result/data_assistant_result\.py', # 23
'rule_based_profiler/data_assistant_result/plot_components\.py', # 12
'rule_based_profiler/domain_builder/categorical_column_domain_builder\.py', # 18
'rule_based_profiler/domain_builder/map_metric_column_domain_builder\.py', # 8
'rule_based_profiler/estimators/bootstrap_numeric_range_estimator\.py', # 8
'rule_based_profiler/estimators/kde_numeric_range_estimator\.py', # 7
'rule_based_profiler/expectation_configuration_builder', # 13
'rule_based_profiler/helpers/util\.py', # 46
'rule_based_profiler/parameter_builder/mean_unexpected_map_metric_multi_batch_parameter_builder\.py', # 19
'rule_based_profiler/parameter_builder/metric_multi_batch_parameter_builder\.py', # 15
'rule_based_profiler/parameter_builder/numeric_metric_range_multi_batch_parameter_builder\.py', # 27
'rule_based_profiler/parameter_builder/parameter_builder\.py', # 40
'rule_based_profiler/parameter_builder/partition_parameter_builder\.py', # 9
'rule_based_profiler/parameter_builder/regex_pattern_string_parameter_builder\.py', # 21
'rule_based_profiler/parameter_builder/simple_date_format_string_parameter_builder\.py', # 20
'rule_based_profiler/rule_based_profiler\.py', # 40
'validation_operators/types/validation_operator_result\.py', # 35
'validation_operators/validation_operators\.py', # 16
'validator/validator\.py', # 54
# tests
'tests/datasource/fluent/tasks\.py',
'tests/datasource/fluent/integration/integration_test_utils\.py',
'tests/datasource/fluent/test_spark_datasource\.py',
]

@Kilo59 Kilo59 linked an issue Mar 23, 2023 that may be closed by this pull request
@netlify
Copy link

netlify bot commented Mar 23, 2023

Deploy Preview for niobium-lead-7998 canceled.

Name Link
🔨 Latest commit a1cf159
🔍 Latest deploy log https://app.netlify.com/sites/niobium-lead-7998/deploys/641c6eaa4df95500076c559f

@ghost
Copy link

ghost commented Mar 23, 2023

👇 Click on the image for a new way to code review

Review these changes using an interactive CodeSee Map

Legend

CodeSee Map legend

@Kilo59 Kilo59 marked this pull request as ready for review March 23, 2023 15:46
@Kilo59 Kilo59 requested a review from rdodev March 23, 2023 15:47
Copy link
Contributor

@rdodev rdodev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Kilo59 Kilo59 enabled auto-merge (squash) March 23, 2023 15:53
@Kilo59 Kilo59 merged commit afb54f2 into develop Mar 23, 2023
@Kilo59 Kilo59 deleted the 7436-type-checking-problems-upgrading-from-01550-to-0161 branch March 23, 2023 16:17
Shinnnyshinshin pushed a commit that referenced this pull request Mar 23, 2023
…tion

* develop:
  [MAINTENANCE] only provide `py.typed` files for fully typed sub-packages (#7438)
@Kilo59 Kilo59 mentioned this pull request Sep 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

type-checking problems Upgrading from 0.15.50 to 0.16.1.
2 participants