convert : refactor rope_freqs generation

[compilade]

Follow-up from #9117 (comment).

This isolates handling of generated tensors like rope_freqs for Llama3, Phi-3 and Exaone. This should also fix --vocab-only conversion for Phi-3-128k and Phi-3.5 (which previously generated invalid GGUF files because they included a non-zero tensor count while not including any tensor data).

Note that this will also be relevant for MiniCPM3 (#9322), which re-uses the misbehaving Phi-3 rope tensors insertion.

TODO

• Test Phi-3-mini-128k LoRA
• Test Phi-3-mini-128k vocab-only validity (with llama-tokenize too)
• Test Phi-3-mini-128k conversion
• Test MiniCPM3 conversion
• Test MiniCPM3 vocab-only validity
• Test Llama-3.1 LoRA
• Test Llama-3.1 conversion
  • includes rope_freqs.weight, and has the same checksum as a previous conversion I did a while ago with the same checkout of the upstream model
• Test Llama-3.1 vocab-only (also with llama-tokenize)

---

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low

[ngxson]

LGTM. Thanks for the implementation!

[ngxson]

Btw, you can use scripts/test-lora-conversion-inference.sh. It will perform end-to-end conversion-inference test for lora gemma2, phi3 and llama arch (note: these are toy models, so they don't have rope_freqs)

[ThiloteE]

Related to issue Support for Phi-3 models #6849

[ggerganov]

Should we merge this, or wait for the rest of the tests in OP to be confirmed?

[ngxson]

I ran the test locally and can confirm that it passes. Let's wait for final confirmation from @compilade to merge this.

[compilade]

Should we merge this, or wait for the rest of the tests in OP to be confirmed?

Since #9322 was merged, MiniCPM3's conversion also has to be updated before merging this. I'll update it today.

[bioinformatist]

@compilade Hey bro, when I try to convert Minicpm3 to .gguf format, it says:

The repository for /models contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co//models.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Will it be implemented by this PR? Thanks!

[compilade]

@compilade Hey bro, when I try to convert Minicpm3 to .gguf format, it says:

The repository for /models contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co//models.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Will it be implemented by this PR? Thanks!

@bioinformatist

Yes, actually, in e83d270 I've changed how MiniCPM3's tokenizer is loaded to exactly avoid that custom code. It uses SentencePiece directly instead. I think it results in the same model files, but I didn't test that yet because I can't really run the custom tokenization code since it repends on datamodel_code_generator which is not in Nixpkgs.

That was a single line change in set_vocab.

[bioinformatist]

Got that, as we have been turning to NixOS (especially for production use) these days. Hope everything goes well! ❤️