Introduce known issues and point people to it

README.md

@@ -1,5 +1,7 @@
 # Benchee [![Hex Version](https://img.shields.io/hexpm/v/benchee.svg)](https://hex.pm/packages/benchee) [![Hex Docs](https://img.shields.io/badge/docs-hexpm-blue.svg)](https://hexdocs.pm/benchee/) [![CI](https://github.com/bencheeorg/benchee/workflows/CI/badge.svg)](https://github.com/bencheeorg/benchee/actions?query=branch%3Amain) [![Coverage Status](https://coveralls.io/repos/github/bencheeorg/benchee/badge.svg?branch=main)](https://coveralls.io/github/bencheeorg/benchee?branch=main) [![Total Download](https://img.shields.io/hexpm/dt/benchee.svg)](https://hex.pm/packages/benchee) [![License](https://img.shields.io/hexpm/l/benchee.svg)](https://github.com/bencheeorg/benchee/blob/main/LICENSE)
 
+**If you have benchmarked using elixir 1.14.0-1.16.0-rc.0, please check out [known issues](#known-issues)**
+
 Library for easy and nice (micro) benchmarking in Elixir. Benchee allows you to compare the performance of different pieces of code at a glance. It is also versatile and extensible, relying only on functions. There are also a bunch of [plugins](#plugins) to draw pretty graphs and more!
 
 Benchee runs each of your functions for a given amount of time after an initial warmup, it then measures their run time and optionally memory consumption. It then shows different statistical values like average, standard deviation etc. See [features](#features).
@@ -66,6 +68,7 @@ The aforementioned [plugins](#plugins) like [benchee_html](https://github.com/be
 
 ## Table of Contents
 
+- [Table of Contents](#table-of-contents)
 - [Features](#features)
   - [Statistics](#statistics)
 - [Installation](#installation)
@@ -85,18 +88,19 @@ The aforementioned [plugins](#plugins) like [benchee_html](https://github.com/be
     - [Suite hooks](#suite-hooks)
     - [Scenario hooks](#scenario-hooks)
       - [What is a scenario?](#what-is-a-scenario)
-      - [before_scenario](#before_scenario)
-      - [after_scenario](#after_scenario)
+      - [before\_scenario](#before_scenario)
+      - [after\_scenario](#after_scenario)
     - [Benchmarking function hooks](#benchmarking-function-hooks)
-      - [before_each](#before_each)
-      - [after_each](#after_each)
+      - [before\_each](#before_each)
+      - [after\_each](#after_each)
     - [Hook arguments and return values](#hook-arguments-and-return-values)
     - [Hook configuration: global versus local](#hook-configuration-global-versus-local)
     - [When does a hook happen? (Complete Example)](#when-does-a-hook-happen-complete-example)
   - [More verbose usage](#more-verbose-usage)
   - [Usage from Erlang](#usage-from-erlang)
+- [Known Issues](#known-issues)
 - [Plugins](#plugins)
-- [Contributing](#contributing-)
+- [Contributing ](#contributing-)
 - [Development](#development)
 - [Copyright and License](#copyright-and-license)
 
@@ -982,6 +986,62 @@ myFunc      289.71 K        3.45 μs   ±250.31%           3 μs
 
 This doesn't seem to be too reliable right now, so suggestions and input are very welcome :)
 
+## Known Issues
+
+There is a known issue affecting elixir version from 1.14.0 to 1.16.0-rc.0: An optimization had been disabled affecting the performance of top level functions. The issue was fixed by re-enabling the optimization in [1.16.0-rc.1](https://github.com/elixir-lang/elixir/blob/v1.16/CHANGELOG.md#v1160-rc1-2023-12-12).
+The issue is best show-cased by the following benchmark where we'd expect ~equal results:
+
+```elixir
+list = Enum.to_list(1..10_000)
+
+defmodule Compiled do
+  def comprehension(list) do
+    for x <- list, rem(x, 2) == 1, do: x + 1
+  end
+end
+
+Benchee.run(%{
+  "module (optimized)" => fn -> Compiled.comprehension(list) end,
+  "top_level (non-optimized)" => fn -> for x <- list, rem(x, 2) == 1, do: x + 1 end
+})
+```
+
+Which yields ~these results on a non fixed elixir version:
+
+```
+Comparison:
+module (optimized)              18.24 K
+top_level (non-optimized)       11.91 K - 1.53x slower +29.14 μs
+```
+
+So, how do you fix it/make sure a benchmark you ran is not affected? All of these work:
+
+* benchmark on an unaffected/fixed versions of elixir (<= 1.13.4 or >= 1.16.0-rc.1)
+* put the code you want to benchmark into a module (just like it is done in `Compiled` in the example above)
+* you can also invoke benchee from within a module, such as:
+```
+defmodule Compiled do
+  def comprehension(list) do
+    for x <- list, rem(x, 2) == 1, do: x + 1
+  end
+end
+
+defmodule MyBenchmark do
+  def run do
+    list = Enum.to_list(1..10_000)
+
+    Benchee.run(%{
+      "module (optimized)" => fn -> Compiled.comprehension(list) end,
+      "top_level (non-optimized)" => fn -> for x <- list, rem(x, 2) == 1, do: x + 1 end
+    })
+  end
+end
+
+MyBenchmark.run()
+```
+
+Also note that even if all your examples are top level functions you should still follow these tips (on affected elixir versions), as the missing optimization might affect them differently.
+
 ## Plugins
 
 Benchee only has small runtime dependencies that were initially extracted from it. Further functionality is provided through plugins that then pull in dependencies, such as HTML generation and CSV export. They help provide excellent visualization or interoperability.