Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

评估方法与位置有很大关系 #32

Open
xiaoyuan1996 opened this issue Jun 6, 2023 · 2 comments
Open

评估方法与位置有很大关系 #32

xiaoyuan1996 opened this issue Jun 6, 2023 · 2 comments

Comments

@xiaoyuan1996
Copy link

image

我们在进行测试后，发现这样的评估方法和位置关系相当密切，把答案进行换位重评后的结果与先前有很大差异。
在使用GPT4等进行评估时，需要进行多次样本翻转来去除位置敏感性。
@GanjinZero
Copy link
Owner

您说的很对，我们当时太穷了，没有那么多资源来跑gpt4/chatgpt。如果可能的话，最好还是翻转位置关系，求平均。

@Guochry
Copy link

Guochry commented Jul 15, 2023

想请问一下您为什么想到用ChatGPT做评估呢？考虑RW曾在额外的偏好数据上训练过，效果应当更好呀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@GanjinZero @xiaoyuan1996 @Guochry