Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write new Experiment or ExperimentAnalysis code module #371

Open
mdekstrand opened this issue Apr 10, 2024 · 1 comment
Open

Write new Experiment or ExperimentAnalysis code module #371

mdekstrand opened this issue Apr 10, 2024 · 1 comment

Comments

@mdekstrand
Copy link
Member

mdekstrand commented Apr 10, 2024

Right now RecListAnalysis is good but limited — only computes per-user metrics.

It would help standardization of evaluation procedures if we had a more coherent "analyze" (and maybe "run") tool for experiments. The first version, of course, would just be for analysis.

  • Specify experiment axes instead of inferring them?
  • Support global metrics
  • Specify list lengths as analysis parameter
  • Support metrics with additional data (novelty, etc.)
  • Clean up metric interface design
  • Support analysis (sig tests, CIs, distributions, etc.)
  • Support results in DuckDB?

This ticket is really probably its own epic.

@mdekstrand
Copy link
Member Author

The new RunAnalysis does this for a single run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant