tea-tasting 0.1.0

What's Changed

tea-tasting is currently in beta. However, I consider it ready for important tasks and use it for the analysis of switchback experiments in my work.

• Analysis of power for RatioOfMeans and Mean by @e10v in #68, #70, #71, #72, #75, and #76
• Fix: make sure that k and n for binomtest are integer by @e10v in #73
• Create a guide on how to use tea-tasting with an arbitrary data backend by @e10v in #79
• Create a guide on custom metrics by @e10v in #81
• Update user guides, docstrings, and readme by @e10v in #77, #78, #80, and #82
• Other minor changes by @e10v in #74 and #83

Full Changelog: v0.0.5...v0.1.0

tea-tasting 0.0.5

What's Changed

• Analysis of a statistic using bootstrap resampling by @e10v in #64
• Analysis of quantiles using bootstrap resampling by @e10v in #65
• Update docs by @e10v in #63 and #67

Full Changelog: v0.0.4...v0.0.5

tea-tasting 0.0.4

What's Changed

• Setup web docs with mkdocs by @e10v in #56
• Experiment result pretty formatting by @e10v in #57 and #59
• Rename is_in to in_ by @e10v in #60
• Update docs and improve naming by @e10v in #61
• Support Ibis 9 by @e10v in #62

Full Changelog: v0.0.3...v0.0.4

tea-tasting 0.0.3

What's Changed

• Sample ratio mismatch check by @e10v in #52
• Handle potential division by zero by @e10v in #53 and #54
• Multiple minor improvements by @e10v in #55 and #51

Full Changelog: v0.0.2...v0.0.3

tea-tasting 0.0.2

tea-tasting is a Python package for statistical analysis of A/B tests that features:

• Student's t-test and Z-test out of the box.
• Extensible API: Define and use statistical tests of your choice.
• Delta method for ratio metrics.
• Variance reduction with CUPED/CUPAC (also in combination with delta method for ratio metrics).
• Confidence interval for both absolute and percent change.

tea-tasting calculates statistics within data backends such as BigQuery, ClickHouse, PostgreSQL, Snowflake, Spark, and other of 20+ backends supported by Ibis. This approach eliminates the need to import granular data into a Python environment, though Pandas DataFrames are also supported.

tea-tasting is still in alpha, but already includes all the features listed above. The following features are coming soon:

• Sample ratio mismatch check.
• More statistical tests:
  • Asymptotic and exact tests for frequency data.
  • Bootstrap.
  • Quantile test (using Bootstrap).
  • Mann–Whitney U test.
• Power analysis.
• A/A tests and simulations.
• Pretty output for experiment results (round etc.).
• Documentation on how to define metrics with custom statistical tests.
• Documentation with MkDocs and Material for MkDocs.
• More examples.

tea-tasting 0.0.1

tea-tasting is a Python package for statistical analysis of A/B tests that features:

• Student's t-test, Z-test, and Bootstrap out of the box.
• Extensible API: Define and use statistical tests of your choice.
• Delta method for ratio metrics.
• Variance reduction with CUPED/CUPAC (also in combination with delta method for ratio metrics).
• Fieller's confidence interval for percent change.
• Sample ratio mismatch check.
• Power analysis.
• A/A tests.

Currently, tea-tasting is in the planning stage, and I'm starting with a README that outlines the proposed API — an approach known as Readme Driven Development (RDD).

Check out my blog post where I explain the motivation for creating this package and the benefits of the RDD approach.