Implementation of RRHF

[Guochry]

Firstly, great thanks to your amazing work on alignment!
Besides, I am curious about the implementation of RRHF in your first experiment: is it initialized by the open source Wombat models provided by RRHF team, or the model you trained by RRHF method?

[agi-templar]

We used the open-sourced implementation and trained the model on our dataset for a fair comparison (so that the data factor is controlled). We find the main issue of RRHF is, if the ratings are the same for multiple completions, the loss calculation based on ranking will be wrong (how to decide their rankings since their scores are the same)? Some other users have found the same issue (see GanjinZero/RRHF#25).

Stable Alignment does not have this issue as we use the difference in the scores to modulate the margin. So if the difference is zero (same score, same ranking), the margin for contrastive learning will be canceled. And the floor operation in the loss can make sure we are not overoptimizing because those already aligned samples will be skipped during learning (loss = 0, by torch.max).

[Guochry]

Great thanks again