Skip to content

Commit

Permalink
remove commented out section
Browse files Browse the repository at this point in the history
  • Loading branch information
kcormi committed Jun 30, 2023
1 parent 75d0eed commit 697e8c3
Showing 1 changed file with 0 additions and 25 deletions.
25 changes: 0 additions & 25 deletions docs/what_combine_does/statistical_tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,31 +225,6 @@ Where $p_{\mu}$ is the usual probability of observing the observed value of the

Using the $\mathrm{CL}_{s}$ criterion fixes the issue of setting limits much stricter than the experimental sensitivity, because for values of $\mu$ to which the experiment is not sensitive the distribution of the test statistic under the signal hypothesis is nearly the same as under the background hypothesis. Therefore, given the use of opposite tails in the p-value definition, $p_{\mu} \approx 1-p_{b}$, and the ratio approaches 1.

<!---
/// details | **Why $p_{b}$ uses the opposite tail from $p_\mu$**
When considering a null hypothesis and an alternate hypothesis, conventionally the p-values of the two hypotheses are defined using opposite tails.
A diagram showing the expected distributions of the test statistic under a null and alternate hypothesis, as well as their corresponding p-values when defined using opposite tails is shown below.
![](null_alternate_distribution_pvals.png)
By using this convention, for well-separated hypothesis, the p-value for both hypotheses will be small when the test statistic appears in the tails between the two distributions.
If the observed test statistic is on either extreme end, (either larger or smaller than expected from either distribution), then the closer of the two distributions will have a large p-value, and the other distribution will have a small p-value.
If the same tail is used for both distributions, then
By defining the p-values using opposite tails, a small p-value for the null hypothesis, disfavouring the null hypothesis, corresponds to a large p-value for the alternative hypothesis and vice versa.
This is illustrated in the figures below.
When setting limits, the null hypothesis, which we are rejecting by setting a limit, is that there is a signal with strength $\mu$, and the alternate hypothesis is that there is no signal, i.e. the background-only hypothesis.
Therefore, by convention, $p_{b}$ is typically defined using the opposite tail as used for $p_{\mu}$.
///
<br>
-->

Note that this means that a limit set using the $\mathrm{CL}_{s}$ criterion at a given $\mathrm{CL}$ will exclude the true parameter value $\mu$ with a frequency less than the nominal rate of $1-\mathrm{CL}$.
The actual frequency at which it is excluded depends on the sensitivity of the experiment to that parameter value.

Expand Down

0 comments on commit 697e8c3

Please sign in to comment.