Skip to content

tea-tasting 0.0.2
Compare
Choose a tag to compare
@e10v e10v released this 15 Apr 20:13
· 197 commits to main since this release
v0.0.2
7b106ce
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.

tea-tasting is a Python package for statistical analysis of A/B tests that features:

  • Student's t-test and Z-test out of the box.
  • Extensible API: Define and use statistical tests of your choice.
  • Delta method for ratio metrics.
  • Variance reduction with CUPED/CUPAC (also in combination with delta method for ratio metrics).
  • Confidence interval for both absolute and percent change.

tea-tasting calculates statistics within data backends such as BigQuery, ClickHouse, PostgreSQL, Snowflake, Spark, and other of 20+ backends supported by Ibis. This approach eliminates the need to import granular data into a Python environment, though Pandas DataFrames are also supported.

tea-tasting is still in alpha, but already includes all the features listed above. The following features are coming soon:

  • Sample ratio mismatch check.
  • More statistical tests:
    • Asymptotic and exact tests for frequency data.
    • Bootstrap.
    • Quantile test (using Bootstrap).
    • Mann–Whitney U test.
  • Power analysis.
  • A/A tests and simulations.
  • Pretty output for experiment results (round etc.).
  • Documentation on how to define metrics with custom statistical tests.
  • Documentation with MkDocs and Material for MkDocs.
  • More examples.
Assets 2
Loading