You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Repeated new lines is a common issue when generating JSON so many users end up setting the value of the whitespace_pattern argument. We should probably set a more restrictive limit on the number of consecutive white spaces and new lines.
The text was updated successfully, but these errors were encountered:
Fixes#839#908#690#450
## Problem
A major problem, especially with smaller language models, is the
repetition problem.
For example, let's say a model is generating json and must provide 12
space tokens for indentation in json output. Often a language model will
assign a high probability to a 13th space token, and do the same for a
14th space, and then enter an infinite space generation loop.
This is a problem with NLG that has been known for half a decade, but
only has mitigations (mirostat, repetition penalty, using hundreds of
billions of weights, etc), no absolute solutions (except for
**structured generation**)
## Solution
For structured json generation, we set a sane default whitespace pattern
of `r"[ ]?"`. This removes all newlines and indentation. It disallows
any syntactic whitespace beyond a single space separator.
Users can still set the argument `whitespace_pattern=` if they want
different behavior
Repeated new lines is a common issue when generating JSON so many users end up setting the value of the
whitespace_pattern
argument. We should probably set a more restrictive limit on the number of consecutive white spaces and new lines.The text was updated successfully, but these errors were encountered: