Skip to content
This repository has been archived by the owner on May 17, 2024. It is now read-only.

Add compare-based model eval #36

Open
yujonglee opened this issue Aug 25, 2023 · 2 comments
Open

Add compare-based model eval #36

yujonglee opened this issue Aug 25, 2023 · 2 comments

Comments

@yujonglee
Copy link
Owner

Rather than asking an LLM for a direct evaluation (via giving a score), try giving it a reference and asking for a comparison. This helps with reducing noise.
@yujonglee yujonglee mentioned this issue Aug 25, 2023
Closed
@yujonglee
Copy link
Owner Author

This will be BaseEvalWithReference

@yujonglee yujonglee mentioned this issue Aug 29, 2023
Closed
@yujonglee
Copy link
Owner Author

We can consider using ELO-Ranking, but we need enormous number of LLM Calls.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@yujonglee