Training without labels - adapting existing pretrained backbone

[keithkam-yk]

Use case:

• A pretrained model is available for my domain (encoder transformer, but no existing colbert model, vastly different tokenizer etc)
• I would like to initialize a new colbert/ ragatouille model with it as described in What should I do if I want a blank, untrained ColBRET? #206

Question:

1. Can I provide training data not labelled or in pairs/ triplets? What would be the recommended way to freeze the pretrained backbone and only train the linear projection + k-means sections?
2. Is there a way to decouple the initial trained backbone embedding from this? I.e. can I embed my unsupervised training corpus up front with my backbone model and then use those embeddings as my training set, instead of raw "text" input?
   a. use case is to test many configurations of the downstream projection layer + k means indexing, with the aim of reducing the encoding GPU costs

PS. big fan of this + colbert work in general!