You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Firstly, great thanks to your amazing work on alignment!
Besides, I am curious about the implementation of RRHF in your first experiment: is it initialized by the open source Wombat models provided by RRHF team, or the model you trained by RRHF method?
The text was updated successfully, but these errors were encountered:
We used the open-sourced implementation and trained the model on our dataset for a fair comparison (so that the data factor is controlled). We find the main issue of RRHF is, if the ratings are the same for multiple completions, the loss calculation based on ranking will be wrong (how to decide their rankings since their scores are the same)? Some other users have found the same issue (see GanjinZero/RRHF#25).
Stable Alignment does not have this issue as we use the difference in the scores to modulate the margin. So if the difference is zero (same score, same ranking), the margin for contrastive learning will be canceled. And the floor operation in the loss can make sure we are not overoptimizing because those already aligned samples will be skipped during learning (loss = 0, by torch.max).
Firstly, great thanks to your amazing work on alignment!
Besides, I am curious about the implementation of RRHF in your first experiment: is it initialized by the open source Wombat models provided by RRHF team, or the model you trained by RRHF method?
The text was updated successfully, but these errors were encountered: