mlr3proba auc

[ahucephe]

Hi,
I have a question for mlr3proba package. I use "surv.cindex" for calculating Harrell's C-index. Which measurement can I use for "time-dependent AUC", "Brier score", "time-dependent Brier score", "integrated time-dependent AUC" and " integrated time-dependent Brier score" in mlr3proba package?

[RaphaelS1]

Hi, thanks for the question, answer are:

• Time-dependent AUC/Integrated -> There's a few options in mlr3proba but surv.chambless_auc might be the most common
• Brier Score, Time-dependent Brier Score, Integrated Brier Score -> surv.graf

Note that mlr3proba only implements distribution evaluation not evaluation at x-year predictions so it's always integrated over time

[ahucephe]

Thanks Raphael, I tried to use "surv.chambless_auc", but I got an error as "Error in .MeasureSurvChamblessAUC.score(self = self, private = private, :
surv.chambless_auc only compatible with Cox PH models".

My code is below:

at=AutoTuner$new(learner="surv.glmnet",
resampling=rsmp("repeated_cv", repeats = 10, folds = 5),
measure=msr("surv.chambless_auc"),
terminator=trm("evals", n_evals = 5),
tuner=tnr("random_search"),
search_space = tune_spaces
)
at$train(my_task)

[ahucephe]

Can you show a small sample code for using "surv.chambless_auc"? I did got an another error now: "Error: Measure 'surv.chambless_auc' requires a task".
Thank you so much.

[RaphaelS1]

Hi, in response to the first question, time-dependent AUC measures were all designed to evaluate linear predictions from a Cox model only. Soon we will try to implement Antolini's C that is more general (#267).

Regarding the second, the measure has the requires_task property (see this section in our book and this help page). Basically you just need to make sure you pass the task to the score method when using it, it also has the requires_learner property so you must also pass the trained learner:

library(mlr3proba)
#> Loading required package: mlr3
l = lrn("surv.coxph")
t = tsk("rats")
splits = partition(t)
p = l$train(t, splits$train)$predict(t, splits$test)
p$score(msr("surv.chambless_auc"),
  task = t, train_set = splits$train,
  learner = l
)
#> surv.chambless_auc 
#>           0.808036

^{Created on 2023-09-11 with reprex v2.0.2}