feat: Metric definitions

[eddiebergman]

Changes mfpbench to now support specifying the Metric to use for a benchmark, which will be returned by result.error or result.score. I.e.

benchmark = mfpbench.get(..., value_metric="accuracy")
config = benchmark.sample()
result = benchmark.query(config)

result.error  # Minimize error based on accuracy
result.score  # Maximize score based on accuracy

To help with this, you can see the metrics defined by a benchmark as so:

benchmark = mfpbench.get(...)
print(benchmark.metrics)
# {"accuracy": Metric(minimize=False, bounds=(0, 100), "roc_auc": Metric(minimize=False, bounds=(0, 1)}

You can also specify the value_metric= in the query() call

benchmark = mfpbench.get(...)
config = benchmark.sample()
result = benchmark.query(config, value_metric="roc_auc")

result.error  # Minimize error based on roc_auc
result.score  # Maximize score based on roc_auc

When a Metric is bounded above and below, the .score and .error will always be bounded in (0, 1) as we can normalize the objective.

To view the optimum of all metrics, you can use benchmark.metric_optimums: dict[str, Metric.Value]

# You can also use .score if you need to see the maximization sides of each these
optimum_acc_err = benchmark.metric_optimums["accuracy"].error
optimum_errs = {metric_name: value.error for metric_name, value in benchmark.metric_optimums}

For tabular benchmarks, there is also .table_optimums which acts much the same but does not use theoretical optimums, instead using the best value seen in the table provided.

---

Breaking

Usage of config.dict() or result.dict() should now be config.as_dict() or result.as_dict(). You can also now obtain the id in when calling config.as_dict(with_id=True) which defaults to False.