How to get MAE, MSE and MAPE of Random Survival Forests

[KeithYouth]

I need to get the mean absolute error(MAE), mean square error(MSE) and mean absolute percentage error(MAPE) as evaluation metrics. I want to know whether scikit-survival can achieve it. My data is uncensored duration. And I want to use random survival forest and random forest separately for prediction, and compare their prediction accuracy. I haved read this https://scikit-survival.readthedocs.io/en/v0.17.0/user_guide/evaluating-survival-models.html But I haven't found a way to implement it yet. Please help me! Thank you!

This is the implementation result I saw in someone else's paper, which is also what I want to achieve

[sebp]

The metrics you mentioned are typically not used in survival analysis, because they cannot deal with censored observations, hence they aren't available in scikit-survival. If you don't have censored observations, you don't need scikit-survival, you can just use a regression model from scikit-learn.

[fkiraly]

You mean, fit one via scikit-survival?

It's available here:
https://scikit-survival.readthedocs.io/en/stable/api/generated/sksurv.ensemble.RandomSurvivalForest.html

[KeithYouth]

Thank you for your answer. What I mean is, can we obtain the MSE value of the survival model in scikit learn?

[fkiraly]

I'm not an expert on this package, but afaik that is not possible, since the type of prediction that MSE requires is not the type of prediction the RandomSurvivalForest produces, or which the data has, unless your data is fully uncensored.

Is your data fully uncensored?

[KeithYouth]

Yes, I have some fully uncensored data. I saw MAE indicators for survival analysis in other people's articles. I am curious about how it is implemented. At present, it seems that it is not achieved through scikit-survival. Thank you very much for your answer.

[fkiraly]

I think there are two main options to use MAE when the test data is censored:

• subset to uncensored, and compute MAE only there
• impute censored under assumption of independent censoring, and compute MAE afterwards

Both options have substantial problems, the first will be biased, even under assumption of independent censoring, and the second depends on the independent censoring assumption as well as the choice of an imputation method (which itself has an error).

However, if your data is fully uncensored, then the issues do not apply - because your task is, strictly speaking, supervised regression rather than survival modelling (which assumes censoring mechanisms). Specifically, instance individual survival predictions are mathematically equivalent to instance individual distributional predictions, and the implied mean is a valid point prediction.

In that case, you can use sklearn to get a regression point prediction. From the next skpro release on, you'll also be able ot use scikit-survival estimators as probabilistic regressors for uncensored data, and evaluate them against C-index, and of course point prediction outcomes via MSE, MAE, etc: https://github.com/sktime/skpro
(the package also supports "non-survival" predictors, though these tend to assume a concentrated distribution like Gaussian rather than shapes of common point process distributions)

[fkiraly]

The metrics you mentioned are typically not used in survival analysis,
because they cannot deal with censored observations, hence they aren't
available in scikit-survival.

Interesting discussion - I do think that, strictly speaking, it can make sense to ask for these.

Imagine a scenario in which none of your outcomes are censored, in this case survival models will still produce a probabilitsic regression prediction. That defines a distribution, which has a mean or location, for which it can make sense to request RMSE or MAE.

Additionally, if your outcomes are censored, there are censored versions of the "classical" regression outcome metrics. They are a bit obscure, and have issues such as not being strictly proper (subject to reasonable definitions), but if the censoring fraction is low, it can still make sense.

[sebp]

You can, but it takes a little bit of work. A Random Survival Forest predicts a survival function S(t), i.e. the probability of experiencing an event up to time t. If you want to compute the MAE with sklearn.metrics.mean_absolute_error you need to get the time point where the survival function (via predict_survival_function) reaches a pre-defined probability. For instance, the median survival time would be the time point t where S(t) = 0.5. Unfortunately, this functionality is not part of scikit-survival, so you would need to implement that yourself.

[fkiraly]

so you would need to implement that yourself

😁 sktime/skpro#237 - the idea is that predict automatically gets the mean from the Empirical distribution (this is simply the integral of x with respect to -dS(x))

(reviews appreciated)

[fkiraly]

PS, @sebp - I think sksurv is a brilliant project!

I do not know details about sksurv's governance, to me it looks like you are carrying most of the weight, and if things are like (non-corporate) open source elsewhere, you're not getting much support or recognition.

I can't find an up-to-date public email, but I would be keen to discuss how to scale and integrate, or generate capacity around sksurv. An option to chat is the sktime discord: https://discord.com/invite/54ACzaFsn7