Assertion error in _cleanup_tokens while using Select

[rohitpatki]

The bug
I get assertion error when am I trying to use Select from a list. Interestingly, the error disappears when I change one of the candidates in the list.

To Reproduce

from guidance import models
from guidance import select
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('mistralai/Mistral-7B-v0.1', add_bos_token=True)
mistral = models.Transformers("mistralai/Mistral-7B-v0.1", tokenizer=tokenizer)

candidates = ["nikon", "nike shoes"]
prompt = mistral + "some queries related to camera are"
sugg = prompt + f'''{select(candidates, name="choice")}'''
print(sugg['choice'])

I get the following error:

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
File <timed exec>:6

File /opt/conda/lib/python3.10/site-packages/guidance/models/_model.py:294, in Model.__add__(self, value)
    292                 partial_grammar += string(part)
    293             is_id = not is_id
--> 294         out = lm + partial_grammar
    296 # if we find a null value we do nothing
    297 elif isinstance(value, Null):

File /opt/conda/lib/python3.10/site-packages/guidance/models/_model.py:302, in Model.__add__(self, value)
    300 # run stateless functions (grammar nodes)
    301 elif isinstance(value, StatelessFunction):
--> 302     out = lm._run_stateless(value)
    304 # run stateful functions
    305 else:
    306     out = value(lm)

File /opt/conda/lib/python3.10/site-packages/guidance/models/_model.py:465, in Model._run_stateless(lm, stateless_function, temperature, top_p, n)
    463 delayed_bytes = b""
    464 # last_is_generated = False
--> 465 for new_bytes, is_generated, new_bytes_prob, capture_groups, capture_group_log_probs, new_token_count in gen_obj:
    466 
    467     # we make everything full probability if we are not computing uncertainty
    468     if not lm.compute_log_probs:
    469         new_bytes_prob = 1.0

File /opt/conda/lib/python3.10/site-packages/guidance/models/_model.py:781, in Model.__call__(self, grammar, max_tokens, n, top_p, temperature, ensure_bos_token)
    776 # otherwise we need to compute the logits and sample a valid token
    777 else:
    778 
    779     # if we were forced we might need to clean up the greedy tokenization to match the global tokenization behavior as seen in training
    780     if was_forced:
--> 781         token_ids,token_byte_positions = self._cleanup_tokens(token_ids, token_byte_positions)
    782         was_forced = False
    783     grammar_temp = parser.next_byte_temperature()

File /opt/conda/lib/python3.10/site-packages/guidance/models/_model.py:611, in Model._cleanup_tokens(self, token_ids, token_byte_positions)
    609         for i in range(1, len(token_byte_positions)):
    610             token_byte_positions[i] -= 1
--> 611     assert token_byte_positions[-1] == last_pos
    613 return token_ids, token_byte_positions

AssertionError: 

But when I change the candidates to candidates = ["iphone", "nike shoes"], the error disappears.

System info (please complete the following information):

• AWS Sagemaker notebook

[slundberg]

Thanks! This was due to what seems like a bug in the hugging face tokenizers where the tokenizers add an extra space at the beginning of the prompt if you decode then encode then decode. I pushed a hack patch that address this specific issue in guidance for now.

[aadityabhatia]

Ran into this issue on select calls with version 0.1.11 from github. Version 0.1.10 from pypi works fine. The calls work the first time after the model is loaded but throw AssertionError with no error message on subsequent calls. Following is the specific trace:

File .../lib/python3.11/site-packages/guidance/models/_model.py:335, in Engine.__call__(self, parser, grammar, ensure_bos_token)
    330 # otherwise we need to compute the logits and sample a valid token
    331 else:
    332 
    333     # if we were forced we might need to clean up the greedy tokenization to match the global tokenization behavior as seen in training
    334     if was_forced:
--> 335         token_ids,token_byte_positions = self._cleanup_tokens(token_ids, token_byte_positions)
    336         was_forced = False
    337     grammar_temp = parser.next_byte_temperature()

File .../lib/python3.11/site-packages/guidance/models/_model.py:621, in Engine._cleanup_tokens(self, token_ids, token_byte_positions)
    619         for i in range(1, len(token_byte_positions)):
    620             token_byte_positions[i] -= 1
--> 621     assert token_byte_positions[-1] == last_pos
    623 return token_ids, token_byte_positions

AssertionError: