diff --git a/.github/workflows/pkgdown.yml b/.github/workflows/pkgdown.yml index 4104f98f1..9a21455dc 100644 --- a/.github/workflows/pkgdown.yml +++ b/.github/workflows/pkgdown.yml @@ -44,7 +44,7 @@ jobs: - name: Deploy if: github.event_name != 'pull_request' - uses: JamesIves/github-pages-deploy-action@v4.6.3 + uses: JamesIves/github-pages-deploy-action@v4.7.2 with: clean: false branch: gh-pages diff --git a/.gitignore b/.gitignore index d48cbc176..093a40e42 100644 --- a/.gitignore +++ b/.gitignore @@ -24,7 +24,8 @@ .LSOverride # Icon must end with two \r -Icon +Icon + # Thumbnails ._* @@ -180,3 +181,4 @@ revdep/ # misc Meta/ +Rplots.pdf diff --git a/DESCRIPTION b/DESCRIPTION index f5e9a1185..75aa5bfc4 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -1,6 +1,6 @@ Package: mlr3 Title: Machine Learning in R - Next Generation -Version: 0.20.2.9000 +Version: 0.22.1.9000 Authors@R: c( person("Michel", "Lang", , "michellang@gmail.com", role = "aut", @@ -52,7 +52,7 @@ Imports: future.apply (>= 1.5.0), lgr (>= 0.3.4), mlbench, - mlr3measures (>= 0.6.0), + mlr3measures (>= 1.0.0), mlr3misc (>= 0.15.0), parallelly, palmerpenguins, @@ -69,9 +69,7 @@ Suggests: remotes, RhpcBLASctl, rpart, - testthat (>= 3.1.0) -Remotes: - mlr-org/mlr3measures + testthat (>= 3.2.0) Encoding: UTF-8 Config/testthat/edition: 3 Config/testthat/parallel: false @@ -158,7 +156,7 @@ Collate: 'TaskGeneratorSpirals.R' 'TaskGeneratorXor.R' 'TaskRegr.R' - 'TaskRegr_boston_housing.R' + 'TaskRegr_california_housing.R' 'TaskRegr_mtcars.R' 'TaskUnsupervised.R' 'as_benchmark_result.R' @@ -181,6 +179,7 @@ Collate: 'benchmark.R' 'benchmark_grid.R' 'bibentries.R' + 'default_fallback.R' 'default_measures.R' 'fix_factor_levels.R' 'helper.R' diff --git a/NAMESPACE b/NAMESPACE index 488ef4974..eadee06bf 100644 --- a/NAMESPACE +++ b/NAMESPACE @@ -74,6 +74,9 @@ S3method(col_info,DataBackend) S3method(col_info,data.table) S3method(create_empty_prediction_data,TaskClassif) S3method(create_empty_prediction_data,TaskRegr) +S3method(default_fallback,Learner) +S3method(default_fallback,LearnerClassif) +S3method(default_fallback,LearnerRegr) S3method(default_values,Learner) S3method(default_values,LearnerClassifRpart) S3method(default_values,LearnerRegrRpart) @@ -108,6 +111,11 @@ S3method(set_threads,list) S3method(set_validate,Learner) S3method(summary,Task) S3method(tail,Task) +S3method(task_check_col_roles,Task) +S3method(task_check_col_roles,TaskClassif) +S3method(task_check_col_roles,TaskRegr) +S3method(task_check_col_roles,TaskSupervised) +S3method(task_check_col_roles,TaskUnsupervised) S3method(unmarshal_model,classif.debug_model_marshaled) S3method(unmarshal_model,default) S3method(unmarshal_model,learner_state_marshaled) @@ -241,6 +249,7 @@ export(rsmp) export(rsmps) export(set_threads) export(set_validate) +export(task_check_col_roles) export(tgen) export(tgens) export(tsk) diff --git a/NEWS.md b/NEWS.md index 7ecd4feac..20257998a 100644 --- a/NEWS.md +++ b/NEWS.md @@ -5,25 +5,52 @@ The weights used during training by the Learner are renamed to `weights_learner`, the previous column role `weight` is dysfunctional. Additionally, it is now possible to disable the use of weights via the new hyperparameter `use_weights`. Note that this is a breaking change, but appears to be the less error-prone solution in the long run. -* refactor: Deprecated `data_format` and `data_formats` for Learners, Tasks, and DataBackends. -* feat: The `partition()` function creates training, test and validation sets. -* refactor: Optimize runtime of fixing factor levels. -* refactor: Optimize runtime of setting row roles. -* refactor: Optimize runtime of marshalling. -* refactor: Optimize runtime of `Task$col_info`. -* fix: Column info is now checked for compatibility during `Learner$predict` (#943). + +# mlr3 0.22.1 + +* fix: Extend `assert_measure()` with checks for trained models in `assert_scorable()`. + +# mlr3 0.22.0 + +* fix: Quantiles must not ascend with probabilities. +* refactor: Replace `tsk("boston_housing")` with `tsk("california_housing")`. +* feat: Require unique learner ids in `benchmark_grid()`. +* BREAKING CHANGE: Remove ``$loglik()`` method from all learners. +* fix: Ignore `future.globals.maxSize` when `future::plan("sequential")` is used. +* feat: Add `$characteristics` field to `Task` to store additional information. + +# mlr3 0.21.1 + +* feat: Throw warning when prediction and measure type do not match. +* fix: The `mlr_reflections` were broken when an extension package was not loaded on the workers. + Extension packages must now register themselves in the `mlr_reflections$loaded_packages` field. + +# mlr3 0.21.0 + +* BREAKING CHANGE: Deprecated `data_format` and `data_formats` for `Learner`, `Task`, and `DataBackend` classes. +* feat: The `partition()` function creates training, test and validation sets now. +* perf: Optimize the runtime of fixing factor levels. +* perf: Optimize the runtime of setting row roles. +* perf: Optimize the runtime of marshalling. +* perf: Optimize the runtime of `Task$col_info`. +* fix: column info is now checked for compatibility during `Learner$predict` (#943). * BREAKING CHANGE: The predict time of the learner now stores the cumulative duration for all predict sets (#992). * feat: `$internal_valid_task` can now be set to an `integer` vector. * feat: Measures can now have an empty `$predict_sets` (#1094). This is relevant for measures that only extract information from the model of a learner (such as internal validation scores or AIC / BIC) -* refactor: Deprecated the `$divide()` method -* fix: `Task$cbind()` now works with non-standard primary keys for `data.frames` (#961). +* BREAKING CHANGE: Deprecated the `$divide()` method +* fix: `Task$cbind()` now works with non-standard primary keys for `data.frames` (#961). * fix: Triggering of fallback learner now has log-level `"info"` instead of `"debug"` (#972). -* feat: Added new measure `pinballs `. -* feat: Added new measure `mu_auc`. +* feat: Added new measure `regr.pinball` here and in mlr3measures. +* feat: Added new measure `mu_auc` here and in mlr3measures. * feat: Add option to calculate the mean of the true values on the train set in `msr("regr.rsq")`. * feat: Default fallback learner is set when encapsulation is activated. -* feat: Learners classif.debug and regr.debug have new methods `$importance()` and `$selected_features()` for testing, also in downstream packages +* feat: Learners `classif.debug` and `regr.debug` have new methods `$importance()` and `$selected_features()` for testing, also in downstream packages. +* feat: Create default fallback learner with `default_fallback()`. +* feat: Check column roles when using `$set_col_roles()` and `$col_roles`. +* fix: Add predict set to learner hash. +* BREAKING CHANGE: Encapsulation and the fallback learner are now set with the `$encapsulate(method, fallback)` method. + The `$fallback` field is read-only now and the encapsulate status can be retrieved from the `$encapsulation` field. # mlr3 0.20.2 diff --git a/R/BenchmarkResult.R b/R/BenchmarkResult.R index 72c7d8c3f..9ee35c3ce 100644 --- a/R/BenchmarkResult.R +++ b/R/BenchmarkResult.R @@ -19,7 +19,7 @@ #' @template param_measures #' #' @section S3 Methods: -#' * `as.data.table(rr, ..., reassemble_learners = TRUE, convert_predictions = TRUE, predict_sets = "test")`\cr +#' * `as.data.table(rr, ..., reassemble_learners = TRUE, convert_predictions = TRUE, predict_sets = "test", task_characteristics = FALSE)`\cr #' [BenchmarkResult] -> [data.table::data.table()]\cr #' Returns a tabular view of the internal data. #' * `c(...)`\cr @@ -545,9 +545,17 @@ BenchmarkResult = R6Class("BenchmarkResult", ) #' @export -as.data.table.BenchmarkResult = function(x, ..., hashes = FALSE, predict_sets = "test") { # nolint +as.data.table.BenchmarkResult = function(x, ..., hashes = FALSE, predict_sets = "test", task_characteristics = FALSE) { # nolint + assert_flag(task_characteristics) tab = get_private(x)$.data$as_data_table(view = NULL, predict_sets = predict_sets) - tab[, c("uhash", "task", "learner", "resampling", "iteration", "prediction"), with = FALSE] + tab = tab[, c("uhash", "task", "learner", "resampling", "iteration", "prediction"), with = FALSE] + + if (task_characteristics) { + set(tab, j = "characteristics", value = map(tab$task, "characteristics")) + tab = unnest(tab, "characteristics") + } + + tab[] } #' @export diff --git a/R/DataBackend.R b/R/DataBackend.R index ae8581e34..1ad6f1196 100644 --- a/R/DataBackend.R +++ b/R/DataBackend.R @@ -90,7 +90,8 @@ DataBackend = R6Class("DataBackend", cloneable = FALSE, #' This is deprecated and will be removed in the future. data_formats = deprecated_binding("DataBackend$data_formats", "data.table"), - #' @template field_hash + #' @field hash (`character(1)`)\cr + #' Hash (unique identifier) for this object. hash = function(rhs) { if (missing(rhs)) { if (is.na(private$.hash)) { diff --git a/R/Learner.R b/R/Learner.R index 1a32c507d..4705f2f14 100644 --- a/R/Learner.R +++ b/R/Learner.R @@ -59,9 +59,6 @@ #' * `oob_error(...)`: Returns the out-of-bag error of the model as `numeric(1)`. #' The learner must be tagged with property `"oob_error"`. #' -#' * `loglik(...)`: Extracts the log-likelihood (c.f. [stats::logLik()]). -#' This can be used in measures like [mlr_measures_aic] or [mlr_measures_bic]. -#' #' * `internal_valid_scores`: Returns the internal validation score(s) of the model as a named `list()`. #' Only available for [`Learner`]s with the `"validation"` property. #' If the learner is not trained yet, this returns `NULL`. @@ -463,6 +460,65 @@ Learner = R6Class("Learner", } else { self } + }, + + #' @description + #' Sets the encapsulation method and fallback learner for the train and predict steps. + #' There are currently four different methods implemented: + #' + #' * `"none"`: Just runs the learner in the current session and measures the elapsed time. + #' Does not keep a log, output is printed directly to the console. + #' Works well together with [traceback()]. + #' * `"try"`: Similar to `"none"`, but catches error. + #' Output is printed to the console and not logged. + #' * `"evaluate"`: Uses the package \CRANpkg{evaluate} to call the learner, measure time and do the logging. + #' * `"callr"`: Uses the package \CRANpkg{callr} to call the learner, measure time and do the logging. + #' This encapsulation spawns a separate R session in which the learner is called. + #' While this comes with a considerable overhead, it also guards your session from being teared down by segfaults. + #' + #' The fallback learner is fitted to create valid predictions in case that either the model fitting or the prediction of the original learner fails. + #' If the training step or the predict step of the original learner fails, the fallback is used completely to predict predictions sets. + #' If the original learner only partially fails during predict step (usually in the form of missing to predict some observations or producing some `NA`` predictions), these missing predictions are imputed by the fallback. + #' Note that the fallback is always trained, as we do not know in advance whether prediction will fail. + #' If the training step fails, the `$model` field of the original learner is `NULL`. + #' + #' Also see the section on error handling the mlr3book: + #' \url{https://mlr3book.mlr-org.com/chapters/chapter10/advanced_technical_aspects_of_mlr3.html#sec-error-handling} + #' + #' @param method `character(1)`\cr + #' One of `"none"`, `"try"`, `"evaluate"` or `"callr"`. + #' See the description for details. + #' @param fallback [Learner]\cr + #' The fallback learner for failed predictions. + #' + #' @return `self` (invisibly). + encapsulate = function(method, fallback = NULL) { + assert_choice(method, c("none", "try", "evaluate", "callr")) + + if (method != "none") { + assert_learner(fallback, task_type = self$task_type) + + if (!identical(self$predict_type, fallback$predict_type)) { + warningf("The fallback learner '%s' and the base learner '%s' have different predict types: '%s' != '%s'.", + fallback$id, self$id, fallback$predict_type, self$predict_type) + } + + # check properties + properties = intersect(self$properties, c("twoclass", "multiclass", "missings", "importance", "selected_features")) + missing_properties = setdiff(properties, fallback$properties) + + if (length(missing_properties)) { + warningf("The fallback learner '%s' does not have the following properties of the learner '%s': %s.", + fallback$id, self$id, str_collapse(missing_properties)) + } + } else if (method == "none" && !is.null(fallback)) { + stop("Fallback learner must be `NULL` if encapsulation is set to `none`.") + } + + private$.encapsulation = c(train = method, predict = method) + private$.fallback = fallback + + return(invisible(self)) } ), @@ -540,16 +596,17 @@ Learner = R6Class("Learner", }, - #' @template field_hash + #' @field hash (`character(1)`)\cr + #' Hash (unique identifier) for this object. + #' The hash is calculated based on the learner id, the parameter settings, the predict type, the fallback hash, the parallel predict setting, the validate setting, and the predict sets. hash = function(rhs) { assert_ro_binding(rhs) calculate_hash(class(self), self$id, self$param_set$values, private$.predict_type, - self$fallback$hash, self$parallel_predict, get0("validate", self)) + self$fallback$hash, self$parallel_predict, get0("validate", self), self$predict_sets) }, #' @field phash (`character(1)`)\cr - #' Hash (unique identifier) for this partial object, excluding some components - #' which are varied systematically during tuning (parameter values). + #' Hash (unique identifier) for this partial object, excluding some components which are varied systematically during tuning (parameter values). phash = function(rhs) { assert_ro_binding(rhs) calculate_hash(class(self), self$id, private$.predict_type, @@ -580,58 +637,20 @@ Learner = R6Class("Learner", private$.param_set }, - #' @field encapsulate (named `character()`)\cr - #' Controls how to execute the code in internal train and predict methods. - #' Must be a named character vector with names `"train"` and `"predict"`. - #' Possible values are `"none"`, `"try"`, `"evaluate"` (requires package \CRANpkg{evaluate}) and `"callr"` (requires package \CRANpkg{callr}). - #' When encapsulation is activated, a fallback learner must be set, - # to ensure that some form of valid model / predictions are created, - # after an error of the original learner is caught via encapsulation. - #' If no learner is set in `$fallback`, the default fallback learner is used (see `mlr_reflections$task_types`). - #' See [mlr3misc::encapsulate()] for more details. - encapsulate = function(rhs) { - default = c(train = "none", predict = "none") - - if (missing(rhs)) { - return(insert_named(default, private$.encapsulate)) - } - - assert_character(rhs) - assert_names(names(rhs), subset.of = c("train", "predict")) - private$.encapsulate = insert_named(default, rhs) - if (is.null(private$.fallback)) { - # if there is no fallback, we get a default one from the reflections table - fallback_id = mlr_reflections$learner_fallback[[self$task_type]] - if (!is.null(fallback_id)) { - self$fallback = lrn(mlr_reflections$learner_fallback[[self$task_type]], predict_type = self$predict_type) - } - } - }, #' @field fallback ([Learner])\cr - #' Learner which is fitted to impute predictions in case that either the model fitting or the prediction of the top learner is not successful. - #' Requires encapsulation, otherwise errors are not caught and the execution is terminated before the fallback learner kicks in. - #' If you have not set encapsulation manually before, setting the fallback learner automatically - #' activates encapsulation using the \CRANpkg{evaluate} package. - #' Also see the section on error handling the mlr3book: - #' \url{https://mlr3book.mlr-org.com/chapters/chapter10/advanced_technical_aspects_of_mlr3.html#sec-error-handling} + #' Returns the fallback learner set with `$encapsulate()`. fallback = function(rhs) { - if (missing(rhs)) { - return(private$.fallback) - } + assert_ro_binding(rhs) + return(private$.fallback) + }, - if (!is.null(rhs)) { - assert_learner(rhs, task_type = self$task_type) - if (!identical(self$predict_type, rhs$predict_type)) { - warningf("The fallback learner '%s' and the base learner '%s' have different predict types: '%s' != '%s'.", - rhs$id, self$id, rhs$predict_type, self$predict_type) - } - if (is.null(private$.encapsulate)) { - private$.encapsulate = c(train = "evaluate", predict = "evaluate") - } - } - private$.fallback = rhs + #' @field encapsulation (`character(2)`)\cr + #' Returns the encapsulation settings set with `$encapsulate()`. + encapsulation = function(rhs) { + assert_ro_binding(rhs) + return(private$.encapsulation) }, #' @field hotstart_stack ([HotstartStack])\cr. @@ -647,7 +666,7 @@ Learner = R6Class("Learner", private = list( .use_weights = NULL, - .encapsulate = NULL, + .encapsulation = c(train = "none", predict = "none"), .fallback = NULL, .predict_type = NULL, .param_set = NULL, diff --git a/R/LearnerRegrDebug.R b/R/LearnerRegrDebug.R index f9257282c..293ba4a32 100644 --- a/R/LearnerRegrDebug.R +++ b/R/LearnerRegrDebug.R @@ -108,7 +108,8 @@ LearnerRegrDebug = R6Class("LearnerRegrDebug", inherit = LearnerRegr, return(prediction) } - prediction = setdiff(named_list(mlr_reflections$learner_predict_types[["regr"]][[self$predict_type]]), "quantiles") + predict_types = setdiff(self$predict_type, "quantiles") + prediction = named_list(mlr_reflections$learner_predict_types[["regr"]][[predict_types]]) missing_type = pv$predict_missing_type %??% "na" for (pt in names(prediction)) { diff --git a/R/LearnerRegrFeatureless.R b/R/LearnerRegrFeatureless.R index f5e8827d3..501c48535 100644 --- a/R/LearnerRegrFeatureless.R +++ b/R/LearnerRegrFeatureless.R @@ -28,7 +28,7 @@ LearnerRegrFeatureless = R6Class("LearnerRegrFeatureless", inherit = LearnerRegr super$initialize( id = "regr.featureless", feature_types = unname(mlr_reflections$task_feature_types), - predict_types = c("response", "se"), + predict_types = c("response", "se", "quantiles"), param_set = ps, properties = c("featureless", "missings", "importance", "selected_features"), packages = "stats", @@ -61,6 +61,14 @@ LearnerRegrFeatureless = R6Class("LearnerRegrFeatureless", inherit = LearnerRegr .train = function(task) { pv = self$param_set$get_values(tags = "train") x = task$data(cols = task$target_names)[[1L]] + + quantiles = if (self$predict_type == "quantiles") { + if (is.null(private$.quantiles) || is.null(private$.quantile_response)) { + stop("Quantiles '$quantiles' and response quantile '$quantile_response' must be set") + } + quantile(x, probs = private$.quantiles) + } + if (isFALSE(pv$robust)) { location = mean(x) dispersion = sd(x) @@ -68,11 +76,24 @@ LearnerRegrFeatureless = R6Class("LearnerRegrFeatureless", inherit = LearnerRegr location = stats::median(x) dispersion = stats::mad(x, center = location) } - set_class(list(location = location, dispersion = dispersion, features = task$feature_names), "regr.featureless_model") + + set_class(list( + location = location, + dispersion = dispersion, + quantiles = quantiles, + features = task$feature_names), "regr.featureless_model") }, .predict = function(task) { n = task$nrow + + if (self$predict_type == "quantiles") { + quantiles = matrix(rep(self$model$quantiles, n), nrow = n, byrow = TRUE) + attr(quantiles, "probs") = private$.quantiles + attr(quantiles, "response") = private$.quantile_response + return(list(quantiles = quantiles)) + } + response = rep(self$model$location, n) se = if (self$predict_type == "se") rep(self$model$dispersion, n) else NULL list(response = response, se = se) diff --git a/R/Measure.R b/R/Measure.R index 8ae201c65..688b37519 100644 --- a/R/Measure.R +++ b/R/Measure.R @@ -199,10 +199,13 @@ Measure = R6Class("Measure", #' #' @return `numeric(1)`. score = function(prediction, task = NULL, learner = NULL, train_set = NULL) { - assert_measure(self, task = task, learner = learner) + assert_scorable(self, task = task, learner = learner, prediction = prediction) properties = self$properties assert_prediction(prediction, null.ok = "requires_no_prediction" %nin% properties) + # check should be added to assert_measure() + # except when the checks are superfluous for rr$score() and bmr$score() + # these checks should be added bellow if ("requires_task" %in% properties && is.null(task)) { stopf("Measure '%s' requires a task", self$id) } @@ -211,21 +214,14 @@ Measure = R6Class("Measure", stopf("Measure '%s' requires a learner", self$id) } - if ("requires_model" %in% properties && (is.null(learner) || is.null(learner$model))) { - stopf("Measure '%s' requires the trained model", self$id) - } - if ("requires_model" %in% properties && is_marshaled_model(learner$model)) { - stopf("Measure '%s' requires the trained model, but model is in marshaled form", self$id) + if (!is_scalar_na(self$task_type) && self$task_type != prediction$task_type) { + stopf("Measure '%s' incompatible with task type '%s'", self$id, prediction$task_type) } if ("requires_train_set" %in% properties && is.null(train_set)) { stopf("Measure '%s' requires the train_set", self$id) } - if (!is_scalar_na(self$task_type) && self$task_type != prediction$task_type) { - stopf("Measure '%s' incompatible with task type '%s'", self$id, prediction$task_type) - } - score_single_measure(self, task, learner, train_set, prediction) }, @@ -272,7 +268,10 @@ Measure = R6Class("Measure", private$.predict_sets }, - #' @template field_hash + #' @field hash (`character(1)`)\cr + #' Hash (unique identifier) for this object. + #' The hash is calculated based on the id, the parameter settings, predict sets and the `$score`, `$average`, `$aggregator`, `$obs_loss`, `$trafo` method. + #' Measure can define additional fields to be included in the hash by setting the field `$.extra_hash`. hash = function(rhs) { assert_ro_binding(rhs) calculate_hash(class(self), self$id, self$param_set$values, private$.score, @@ -372,8 +371,6 @@ score_single_measure = function(measure, task, learner, train_set, prediction) { return(NaN) } - - if (!is_scalar_na(measure$predict_type) && measure$predict_type %nin% prediction$predict_types) { # TODO lgr$debug() return(NaN) @@ -384,7 +381,6 @@ score_single_measure = function(measure, task, learner, train_set, prediction) { return(NaN) } - get_private(measure)$.score(prediction = prediction, task = task, learner = learner, train_set = train_set) } @@ -416,7 +412,7 @@ score_measures = function(obj, measures, reassemble = TRUE, view = NULL, iters = tmp = unique(tab, by = c("task_hash", "learner_hash"))[, c("task", "learner"), with = FALSE] for (measure in measures) { - pmap(tmp, assert_measure, measure = measure) + pmap(tmp, assert_scorable, measure = measure) score = pmap_dbl(tab[, c("task", "learner", "resampling", "iteration", "prediction"), with = FALSE], function(task, learner, resampling, iteration, prediction) { diff --git a/R/MeasureAIC.R b/R/MeasureAIC.R index 5a0203d1d..e0a988ebc 100644 --- a/R/MeasureAIC.R +++ b/R/MeasureAIC.R @@ -40,12 +40,14 @@ MeasureAIC = R6Class("MeasureAIC", private = list( .score = function(prediction, learner, ...) { learner = learner$base_learner() - if ("loglik" %nin% learner$properties) { - return(NA_real_) - } - k = self$param_set$values$k %??% 2 - return(stats::AIC(learner$loglik(), k = k)) + + tryCatch({ + return(stats::AIC(stats::logLik(learner$model), k = k)) + }, error = function(e) { + warningf("Learner '%s' does not support AIC calculation", learner$id) + return(NA_real_) + }) } ) ) diff --git a/R/MeasureBIC.R b/R/MeasureBIC.R index 94d67d515..b3d48f205 100644 --- a/R/MeasureBIC.R +++ b/R/MeasureBIC.R @@ -38,11 +38,13 @@ MeasureBIC = R6Class("MeasureBIC", private = list( .score = function(prediction, learner, ...) { learner = learner$base_learner() - if ("loglik" %nin% learner$properties) { - return(NA_real_) - } - return(stats::BIC(learner$loglik())) + tryCatch({ + return(stats::BIC(stats::logLik(learner$model))) + }, error = function(e) { + warningf("Learner '%s' does not support BIC calculation", learner$id) + return(NA_real_) + }) } ) ) diff --git a/R/MeasureElapsedTime.R b/R/MeasureElapsedTime.R index b2c18417d..8f2c0b53f 100644 --- a/R/MeasureElapsedTime.R +++ b/R/MeasureElapsedTime.R @@ -9,12 +9,11 @@ #' #' @description #' Measures the elapsed time during train ("time_train"), predict ("time_predict"), or both ("time_both"). -#' Aggregation of elapsed time defaults to mean but can be configured via the field `aggregator` of the -#' [Measure]. +#' Aggregation of elapsed time defaults to mean but can be configured via the field `aggregator` of the [Measure]. #' -#' When predictions for multiple predict sets were made during [resample()] or [benchmark()], -#' the predict time shows the cumulative duration of all predictions. +#' When predictions for multiple predict sets were made during [resample()] or [benchmark()], the predict time shows the cumulative duration of all predictions. #' If `learner$predict()` is called manually, the last predict time gets overwritten. +#' The elapsed time accounts only for the training duration of the primary learner, excluding the time required for training the fallback learner. #' #' @template param_id #' @templateVar id time_train diff --git a/R/MeasureInternalValidScore.R b/R/MeasureInternalValidScore.R index 2d60bf3e0..94770c1a1 100644 --- a/R/MeasureInternalValidScore.R +++ b/R/MeasureInternalValidScore.R @@ -33,7 +33,7 @@ MeasureInternalValidScore = R6Class("MeasureInternalValidScore", super$initialize( id = select %??% "internal_valid_score", task_type = NA_character_, - properties = c("na_score", "requires_model", "requires_learner", "requires_no_prediction"), + properties = c("na_score", "requires_learner", "requires_no_prediction"), predict_sets = NULL, predict_type = NA_character_, range = c(-Inf, Inf), diff --git a/R/MeasureRegrRSQ.R b/R/MeasureRegrRSQ.R index 01e602eeb..5b5c4450c 100644 --- a/R/MeasureRegrRSQ.R +++ b/R/MeasureRegrRSQ.R @@ -15,7 +15,7 @@ #' where \eqn{\bar{t} = \sum_{i=1}^n t_i}. #' #' Also known as coefficient of determination or explained variation. -#' Subtracts the [rse()] from 1, hence it compares the squared error of the predictions relative to a naive model predicting the mean. +#' Subtracts the [mlr3measures::rse()] from 1, hence it compares the squared error of the predictions relative to a naive model predicting the mean. #' #' This measure is undefined for constant \eqn{t}. #' @@ -29,7 +29,7 @@ #' @template seealso_measure #' @export MeasureRegrRSQ = R6Class("MeasureRSQ", - inherit = Measure, + inherit = MeasureRegr, public = list( #' @description #' Creates a new instance of this [R6][R6::R6Class] class. @@ -40,10 +40,10 @@ MeasureRegrRSQ = R6Class("MeasureRSQ", super$initialize( id = "rsq", - task_type = "regr", properties = if (!private$.pred_set_mean) c("requires_task", "requires_train_set") else character(0), predict_type = "response", minimize = FALSE, + range = c(-Inf, 1), man = "mlr3::mlr_measures_regr.rsq" ) } diff --git a/R/PredictionDataRegr.R b/R/PredictionDataRegr.R index 9602d3375..080dd549b 100644 --- a/R/PredictionDataRegr.R +++ b/R/PredictionDataRegr.R @@ -25,12 +25,8 @@ check_prediction_data.PredictionDataRegr = function(pdata, ...) { # nolint stopf("No probs attribute stored in 'quantile'") } - if (is.null(attr(quantiles, "response"))) { - stopf("No response attribute stored in 'quantile'") - } - - if (any(apply(quantiles, 1L, is.unsorted))) { - stopf("Quantiles are not ascending with probabilities") + if (is.null(attr(quantiles, "response")) && is.null(pdata$response)) { + stopf("No response attribute stored in 'quantile' or response stored in 'pdata'") } colnames(pdata$quantiles) = sprintf("q%g", attr(quantiles, "probs")) diff --git a/R/PredictionRegr.R b/R/PredictionRegr.R index 8aebf8af1..ddf07d774 100644 --- a/R/PredictionRegr.R +++ b/R/PredictionRegr.R @@ -10,7 +10,7 @@ #' @template seealso_prediction #' @export #' @examples -#' task = tsk("boston_housing") +#' task = tsk("california_housing") #' learner = lrn("regr.featureless", predict_type = "se") #' p = learner$train(task)$predict(task) #' p$predict_types @@ -63,7 +63,7 @@ PredictionRegr = R6Class("PredictionRegr", inherit = Prediction, # response is in saved in quantiles matrix if ("quantiles" %in% predict_types) predict_types = union(predict_types, "response") self$predict_types = predict_types - private$.quantile_response = attr(quantiles, "response") + if (is.null(pdata$response)) private$.quantile_response = attr(quantiles, "response") } ), diff --git a/R/Resampling.R b/R/Resampling.R index 776a9d511..011bc8c46 100644 --- a/R/Resampling.R +++ b/R/Resampling.R @@ -246,7 +246,10 @@ Resampling = R6Class("Resampling", !is.null(self$instance) }, - #' @template field_hash + #' @field hash (`character(1)`)\cr + #' Hash (unique identifier) for this object. + #' If the object has not been instantiated yet, `NA_character_` is returned. + #' The hash is calculated based on the class name, the id, the parameter set, and the instance. hash = function(rhs) { assert_ro_binding(rhs) if (!self$is_instantiated) { diff --git a/R/Task.R b/R/Task.R index 151fdda4b..aec3e8676 100644 --- a/R/Task.R +++ b/R/Task.R @@ -16,7 +16,7 @@ #' For example, for a classification task a single column must be marked as target column, and others as features. #' #' Predefined (toy) tasks are stored in the [dictionary][mlr3misc::Dictionary] [mlr_tasks], -#' e.g. [`penguins`][mlr_tasks_penguins] or [`boston_housing`][mlr_tasks_boston_housing]. +#' e.g. [`penguins`][mlr_tasks_penguins] or [`california_housing`][mlr_tasks_california_housing]. #' More toy tasks can be found in the dictionary after loading \CRANpkg{mlr3data}. #' #' @template param_id @@ -253,6 +253,10 @@ Task = R6Class("Task", if (!is.null(private$.internal_valid_task)) { catf(str_indent("* Validation Task:", sprintf("(%ix%i)", private$.internal_valid_task$nrow, private$.internal_valid_task$ncol))) } + + if (!is.null(self$characteristics)) { + catf(str_indent("* Characteristics: ", as_short_string(self$characteristics))) + } }, #' @description @@ -305,7 +309,8 @@ Task = R6Class("Task", data = self$backend$data(rows = rows, cols = query_cols) if (length(query_cols) && nrow(data) != length(rows)) { - stopf("DataBackend did not return the queried rows correctly: %i requested, %i received", length(rows), nrow(data)) + stopf("DataBackend did not return the queried rows correctly: %i requested, %i received. + The resampling was probably instantiated on a different task.", length(rows), nrow(data)) } if (length(rows) && ncol(data) != length(query_cols)) { @@ -657,6 +662,7 @@ Task = R6Class("Task", #' @description #' Modifies the roles in `$col_roles` **in-place**. + #' See `$col_roles` for a list of possible roles. #' #' @param cols (`character()`)\cr #' Column names for which to change the roles for. @@ -670,9 +676,10 @@ Task = R6Class("Task", #' Other column roles are preserved. #' #' @details - #' Roles are first set exclusively (argument `roles`), then added (argument `add_to`) and finally - #' removed (argument `remove_from`) from different roles. + #' Roles are first set exclusively (argument `roles`), then added (argument `add_to`) and finally removed (argument `remove_from`) from different roles. #' Duplicated columns are removed from the same role. + #' For tasks that only allow one target, the target column cannot be set with `$set_col_roles()`. + #' Use the `$col_roles` field to swap the target column. #' #' @return #' Returns the object itself, but modified **by reference**. @@ -826,10 +833,16 @@ Task = R6Class("Task", }) private$.internal_valid_task = rhs + if (private$.internal_valid_task$nrow == 0) { + warningf("Internal validation task has 0 observations.") + } invisible(private$.internal_valid_task) }, - #' @template field_hash + #' @field hash (`character(1)`)\cr + #' Hash (unique identifier) for this object. + #' The hash is calculated based on the complete task object and `$row_ids`. + #' If an internal validation task is set, the hash is recalculated. hash = function(rhs) { if (is.null(private$.hash)) { private$.hash = task_hash(self, self$row_ids, ignore_internal_valid_task = FALSE) @@ -1172,6 +1185,17 @@ Task = R6Class("Task", private$.col_hashes = self$backend$col_hashes[setdiff(unlist(private$.col_roles, use.names = FALSE), self$backend$primary_key)] } private$.col_hashes + }, + + #' @field characteristics (`list()`)\cr + #' List of characteristics of the task, e.g. `list(n = 5, p = 7)`. + characteristics = function(rhs) { + if (missing(rhs)) { + return(private$.characteristics) + } + + private$.characteristics = assert_list(rhs, null.ok = TRUE) + private$.hash = NULL } ), @@ -1183,6 +1207,7 @@ Task = R6Class("Task", .row_roles = NULL, .hash = NULL, .col_hashes = NULL, + .characteristics = NULL, deep_clone = function(name, value) { # NB: DataBackends are never copied! @@ -1249,7 +1274,25 @@ task_set_roles = function(li, elements, roles = NULL, add_to = NULL, remove_from li } -task_check_col_roles = function(self, new_roles) { +#' @title Check Column Roles +#' +#' @description +#' Internal function to check column roles. +#' +#' @param task ([Task])\cr +#' Task. +#' @param new_roles (`list()`)\cr +#' Column roles. +#' +#' @keywords internal +#' @export +task_check_col_roles = function(task, new_roles, ...) { + UseMethod("task_check_col_roles") +} + +#' @rdname task_check_col_roles +#' @export +task_check_col_roles.Task = function(task, new_roles, ...) { if ("weight" %in% names(new_roles)) { stopf("Task role 'weight' is deprecated, use 'weights_learner' instead") } @@ -1260,24 +1303,79 @@ task_check_col_roles = function(self, new_roles) { } } + # check weights for (role in c("weights_learner", "weights_measure", "weights_resampling")) { if (length(new_roles[[role]]) > 0L) { - col_role = self$backend$data(seq(self$backend$nrow), cols = new_roles[[role]])[[1]] - expect_numeric(col_role, lower = 0, any.missing = FALSE) + col = task$backend$data(seq(task$backend$nrow), cols = new_roles[[role]]) + assert_numeric(col[[1]], lower = 0, any.missing = FALSE, .var.name = names(col)) } } - if (inherits(self, "TaskSupervised")) { - if (length(new_roles$target) == 0L) { - stopf("Supervised tasks need at least one target column") - } - } else if (inherits(self, "TaskUnsupervised")) { - if (length(new_roles$target) != 0L) { - stopf("Unsupervised tasks may not have a target column") + # check name + if (length(new_roles[["name"]])) { + row_names = task$backend$data(task$backend$rownames, cols = new_roles[["name"]]) + if (!is.character(row_names[[1L]]) && !is.factor(row_names[[1L]])) { + stopf("Assertion on '%s' failed: Must be of type 'character' or 'factor', not %s", names(row_names), class(row_names[[1]])) } } - new_roles + return(new_roles) +} + +#' @rdname task_check_col_roles +#' @export +task_check_col_roles.TaskClassif = function(task, new_roles, ...) { + + # check target + if (length(new_roles[["target"]]) > 1L) { + stopf("There may only be up to one column with role 'target'") + } + + if (length(new_roles[["target"]]) && any(fget(task$col_info, new_roles[["target"]], "type", key = "id") %nin% c("factor", "ordered"))) { + stopf("Target column(s) %s must be a factor or ordered factor", paste0("'", new_roles[["target"]], "'", collapse = ",")) + } + + NextMethod() +} + +#' @rdname task_check_col_roles +#' @export +task_check_col_roles.TaskRegr = function(task, new_roles, ...) { + + # check target + if (length(new_roles[["target"]]) > 1L) { + stopf("There may only be up to one column with role 'target'") + } + + if (length(new_roles[["target"]]) && any(fget(task$col_info, new_roles[["target"]], "type", key = "id") %nin% c("numeric", "integer"))) { + stopf("Target column '%s' must be a numeric or integer column", paste0("'", new_roles[["target"]], "'", collapse = ",")) + } + + NextMethod() +} + +#' @rdname task_check_col_roles +#' @export +task_check_col_roles.TaskSupervised = function(task, new_roles, ...) { + + # check target + if (length(new_roles$target) == 0L) { + stopf("Supervised tasks need at least one target column") + } + + NextMethod() +} + +#' @rdname task_check_col_roles +#' @export +task_check_col_roles.TaskUnsupervised = function(task, new_roles, ...) { + + # check target + if (length(new_roles$target) != 0L) { + stopf("Unsupervised tasks may not have a target column") + } + + NextMethod() } #' @title Column Information for Backend diff --git a/R/TaskRegr_boston_housing.R b/R/TaskRegr_boston_housing.R deleted file mode 100644 index ec2866343..000000000 --- a/R/TaskRegr_boston_housing.R +++ /dev/null @@ -1,33 +0,0 @@ -#' @title Boston Housing Regression Task -#' -#' @name mlr_tasks_boston_housing -#' @format [R6::R6Class] inheriting from [TaskRegr]. -#' @include mlr_tasks.R -#' -#' -#' @description -#' A regression task for the [mlbench::BostonHousing2] data set. -#' This is the corrected data using the corrected median value (`cmedv`) as target. -#' The uncorrected target (`medv`) is removed from the data. -#' -#' @section Construction: -#' ``` -#' mlr_tasks$get("boston_housing") -#' tsk("boston_housing") -#' ``` -#' -#' @section Meta Information: -#' `r rd_info(tsk("boston_housing"))` -#' -#' @template seealso_task -NULL - -load_task_boston_housing = function(id = "boston_housing") { - b = as_data_backend(remove_named(load_dataset("BostonHousing2", "mlbench"), "medv")) - task = TaskRegr$new(id, b, target = "cmedv", label = "Boston Housing Prices") - b$hash = task$man = "mlr3::mlr_tasks_boston_housing" - task -} - -#' @include mlr_tasks.R -mlr_tasks$add("boston_housing", load_task_boston_housing) diff --git a/R/TaskRegr_california_housing.R b/R/TaskRegr_california_housing.R new file mode 100644 index 000000000..9a8df77e1 --- /dev/null +++ b/R/TaskRegr_california_housing.R @@ -0,0 +1,35 @@ +#' @title Median House Value in California +#' +#' @name california_housing +#' @format [R6::R6Class] inheriting from [TaskRegr]. +#' @aliases mlr_tasks_california_housing +#' +#' @description +#' A regression task to predict the median house value in California. +#' +#' Contains 9 features and 20640 observations. +#' Target column is `"median_house_value"`. +#' +#' @section Construction: +#' ``` +#' mlr_tasks$get("california_housing") +#' tsk("california_housing") +#' ``` +#' +#' @section Meta Information: +#' `r rd_info(tsk("california_housing"))` +#' +#' @source \url{https://www.kaggle.com/datasets/camnugent/california-housing-prices} +#' +#' @template seealso_task +NULL + +load_task_california_housing = function(id = "california_housing") { + b = as_data_backend(readRDS(system.file("extdata", "california_housing.rds", package = "mlr3"))) + task = mlr3::TaskRegr$new(id, b, target = "median_house_value", label = "California House Value") + b$hash = task$man = "mlr3::mlr_tasks_california_housing" + task +} + +#' @include mlr_tasks.R +mlr_tasks$add("california_housing", load_task_california_housing) diff --git a/R/as_prediction.R b/R/as_prediction.R index a5ab76a90..0b87e1b1b 100644 --- a/R/as_prediction.R +++ b/R/as_prediction.R @@ -8,6 +8,8 @@ #' @return [Prediction]. #' @export as_prediction = function(x, check = FALSE, ...) { + if (is.null(x)) return(list()) + UseMethod("as_prediction") } diff --git a/R/assertions.R b/R/assertions.R index cfc798e30..4c257c8eb 100644 --- a/R/assertions.R +++ b/R/assertions.R @@ -107,7 +107,13 @@ test_matching_task_type = function(task_type, object, class) { #' @export #' @param learners (list of [Learner]). #' @rdname mlr_assertions -assert_learners = function(learners, task = NULL, task_type = NULL, properties = character(), .var.name = vname(learners)) { +assert_learners = function(learners, task = NULL, task_type = NULL, properties = character(), unique_ids = FALSE, .var.name = vname(learners)) { + if (unique_ids) { + ids = map_chr(learners, "id") + if (!test_character(ids, unique = TRUE)) { + stopf("Learners need to have unique IDs: %s", str_collapse(ids)) + } + } invisible(lapply(learners, assert_learner, task = task, task_type = NULL, properties = properties, .var.name = .var.name)) } @@ -193,7 +199,7 @@ assert_predictable = function(task, learner) { all(pmap_lgl(list(x = ci_train$levels, y = ci_predict$levels), identical)) if (!ok) { - stopf( "Learner '%s' received task with different column info during train and predict.", learner$id) + lg$warn("Learner '%s' received task with different column info (feature type or level ordering) during train and predict.", learner$id) } } @@ -204,11 +210,13 @@ assert_predictable = function(task, learner) { #' @export #' @param measure ([Measure]). +#' @param prediction ([Prediction]). #' @rdname mlr_assertions -assert_measure = function(measure, task = NULL, learner = NULL, .var.name = vname(measure)) { +assert_measure = function(measure, task = NULL, learner = NULL, prediction = NULL, .var.name = vname(measure)) { assert_class(measure, "Measure", .var.name = .var.name) if (!is.null(task)) { + if (!is_scalar_na(measure$task_type) && !test_matching_task_type(task$task_type, measure, "measure")) { stopf("Measure '%s' is not compatible with type '%s' of task '%s'", measure$id, task$task_type, task$id) @@ -224,6 +232,7 @@ assert_measure = function(measure, task = NULL, learner = NULL, .var.name = vnam } if (!is.null(learner)) { + if (!is_scalar_na(measure$task_type) && measure$task_type != learner$task_type) { stopf("Measure '%s' is not compatible with type '%s' of learner '%s'", measure$id, learner$task_type, learner$id) @@ -246,9 +255,31 @@ assert_measure = function(measure, task = NULL, learner = NULL, .var.name = vnam } } + if (!is.null(prediction) && is.null(learner)) { + # same as above but works without learner e.g. measure$score(prediction) + if (measure$check_prerequisites != "ignore" && measure$predict_type %nin% prediction$predict_types) { + warningf("Measure '%s' is missing predict type '%s' of prediction", measure$id, measure$predict_type) + } + } + invisible(measure) } +#' @export +#' @param measure ([Measure]). +#' @param prediction ([Prediction]). +#' @rdname mlr_assertions +assert_scorable = function(measure, task, learner, prediction = NULL, .var.name = vname(measure)) { + if ("requires_model" %in% measure$properties && is.null(learner$model)) { + stopf("Measure '%s' requires the trained model", measure$id) + } + + if ("requires_model" %in% measure$properties && is_marshaled_model(learner$model)) { + stopf("Measure '%s' requires the trained model, but model is in marshaled form", measure$id) + } + + assert_measure(measure, task = task, learner = learner, prediction = prediction, .var.name = .var.name) +} #' @export #' @param measures (list of [Measure]). diff --git a/R/benchmark_grid.R b/R/benchmark_grid.R index e8d93661b..41556071b 100644 --- a/R/benchmark_grid.R +++ b/R/benchmark_grid.R @@ -67,7 +67,7 @@ #' benchmark_grid = function(tasks, learners, resamplings, param_values = NULL, paired = FALSE) { tasks = assert_tasks(as_tasks(tasks)) - learners = assert_learners(as_learners(learners)) + learners = assert_learners(as_learners(learners), unique_ids = TRUE) resamplings = assert_resamplings(as_resamplings(resamplings)) if (!is.null(param_values)) { assert_param_values(param_values, n_learners = length(learners)) @@ -103,7 +103,8 @@ benchmark_grid = function(tasks, learners, resamplings, param_values = NULL, pai if (!identical(task_nrow, unique(map_int(resamplings, "task_nrow")))) { stop("A Resampling is instantiated for a task with a different number of observations") } - instances = pmap(grid, function(task, resampling) resamplings[[resampling]]$clone()) + # clone resamplings for each task and update task hashes + instances = pmap(grid, function(task, resampling) resampling = resamplings[[resampling]]$clone()) } else { instances = pmap(grid, function(task, resampling) resamplings[[resampling]]$clone()$instantiate(tasks[[task]])) } diff --git a/R/default_fallback.R b/R/default_fallback.R new file mode 100644 index 000000000..5d618a583 --- /dev/null +++ b/R/default_fallback.R @@ -0,0 +1,64 @@ +#' @title Create a Fallback Learner +#' +#' @description +#' Create a fallback learner for a given learner. +#' The function searches for a suitable fallback learner based on the task type. +#' Additional checks are performed to ensure that the fallback learner supports the predict type. +#' +#' @param learner [Learner]\cr +#' The learner for which a fallback learner should be created. +#' @param ... `any`\cr +#' ignored. +#' +#' @return [Learner] +default_fallback = function(learner, ...) { + UseMethod("default_fallback") +} + +#' @rdname default_fallback +#' @export +default_fallback.Learner = function(learner, ...) { + # FIXME: remove when new encapsulate/fallback system is in place + return(NULL) +} + +#' @rdname default_fallback +#' @export +default_fallback.LearnerClassif = function(learner, ...) { + fallback = lrn("classif.featureless") + + # set predict type + if (learner$predict_type %nin% fallback$predict_types) { + stopf("Fallback learner '%s' does not support predict type '%s'.", fallback$id, learner$predict_type) + } + + fallback$predict_type = learner$predict_type + + return(fallback) +} + +#' @rdname default_fallback +#' @export +default_fallback.LearnerRegr = function(learner, ...) { + fallback = lrn("regr.featureless") + + # set predict type + if (learner$predict_type %nin% fallback$predict_types) { + stopf("Fallback learner '%s' does not support predict type '%s'.", fallback$id, learner$predict_type) + } + + fallback$predict_type = learner$predict_type + + # set quantiles + if (learner$predict_type == "quantiles") { + + if (is.null(learner$quantiles) || is.null(learner$quantile_response)) { + stopf("Cannot set quantiles for fallback learner. Set `$quantiles` and `$quantile_response` in %s.", learner$id) + } + + fallback$quantiles = learner$quantiles + fallback$quantile_response = learner$quantile_response + } + + return(fallback) +} diff --git a/R/helper_exec.R b/R/helper_exec.R index aa85c43a2..c59c0d1eb 100644 --- a/R/helper_exec.R +++ b/R/helper_exec.R @@ -4,20 +4,14 @@ allow_partial_matching = list( warnPartialMatchDollar = FALSE ) - set_encapsulation = function(learners, encapsulate) { assert_choice(encapsulate, c(NA_character_, "none", "evaluate", "callr", "try")) if (!is.na(encapsulate)) { - lapply(learners, function(learner) learner$encapsulate = c(train = encapsulate, predict = encapsulate)) - if (encapsulate %in% c("evaluate", "callr")) { - task_type = unique(map_chr(learners, "task_type")) - stopifnot(length(task_type) == 1L) # this should not be possible for benchmarks - fb = get_featureless_learner(task_type) - if (!is.null(fb)) { - lapply(learners, function(learner) if (is.null(learner$fallback)) learner$fallback = fb$clone(TRUE)) - } - } + lapply(learners, function(learner) { + fallback = if (encapsulate != "none") default_fallback(learner) + learner$encapsulate(encapsulate, fallback) + }) } learners } @@ -36,12 +30,19 @@ future_map = function(n, FUN, ..., MoreArgs = list()) { } stdout = if (is_sequential) NA else TRUE + # workaround for sequential plan checking the size of the globals + # see https://github.com/futureverse/future/issues/197 + if (is_sequential) { + old_opts = options(future.globals.maxSize = Inf) + on.exit(options(old_opts), add = TRUE) + } + MoreArgs = c(MoreArgs, list(is_sequential = is_sequential)) lg$debug("Running resample() via future with %i iterations", n) future.apply::future_mapply( FUN, ..., MoreArgs = MoreArgs, SIMPLIFY = FALSE, USE.NAMES = FALSE, - future.globals = FALSE, future.packages = "mlr3", future.seed = TRUE, + future.globals = FALSE, future.packages = mlr_reflections$loaded_packages, future.seed = TRUE, future.scheduling = scheduling, future.chunk.size = chunk_size, future.stdout = stdout ) } diff --git a/R/helper_hashes.R b/R/helper_hashes.R index 324d4ab2b..211264f28 100644 --- a/R/helper_hashes.R +++ b/R/helper_hashes.R @@ -31,7 +31,7 @@ resampling_task_hashes = function(task, resampling, learner = NULL) { task_hash = function(task, use_ids, test_ids = NULL, ignore_internal_valid_task = FALSE) { # order matters: we first check for test_ids and then for the internal_valid_task internal_valid_task_hash = if (!is.null(test_ids)) { - # this does the same as + # this does the same as # task$internal_valid_task = test_ids # $internal_valid_task$hash # but avoids the deep clone @@ -40,6 +40,14 @@ task_hash = function(task, use_ids, test_ids = NULL, ignore_internal_valid_task task$internal_valid_task$hash } - calculate_hash(class(task), task$id, task$backend$hash, task$col_info, use_ids, task$col_roles, - get_private(task)$.properties, internal_valid_task_hash) + calculate_hash( + class(task), + task$id, + task$backend$hash, + task$col_info, + use_ids, + task$col_roles, + get_private(task)$.properties, + internal_valid_task_hash, + task$characteristics) } diff --git a/R/mlr_measures.R b/R/mlr_measures.R index 20f3747fc..46a9252e8 100644 --- a/R/mlr_measures.R +++ b/R/mlr_measures.R @@ -47,7 +47,13 @@ as.data.table.DictionaryMeasure = function(x, ..., objects = FALSE) { m = withCallingHandlers(x$get(key, .prototype = TRUE), packageNotFoundWarning = function(w) invokeRestart("muffleWarning")) insert_named( - list(key = key, label = m$label, task_type = m$task_type, packages = list(m$packages), predict_type = m$predict_type, + list( + key = key, + label = m$label, + task_type = m$task_type, + packages = list(m$packages), + predict_type = m$predict_type, + properties = list(m$properties), task_properties = list(m$task_properties)), if (objects) list(object = list(m)) ) diff --git a/R/mlr_reflections.R b/R/mlr_reflections.R index c979ed7bd..e8a117691 100644 --- a/R/mlr_reflections.R +++ b/R/mlr_reflections.R @@ -8,6 +8,8 @@ #' This environment be modified by third-party packages, e.g. by adding information about new task types #' or by extending the set of allowed feature types. #' +#' Third-party packages that modify the reflections must register themselves in the `loaded_packages` field. +#' #' The following objects are set by \CRANpkg{mlr3}: #' #' * `task_types` (`data.table()`)\cr @@ -116,7 +118,7 @@ local({ ) ### Learner - tmp = c("featureless", "missings", "weights", "importance", "selected_features", "oob_error", "loglik", "hotstart_forward", "hotstart_backward", "validation", "internal_tuning", "marshal") + tmp = c("featureless", "missings", "weights", "importance", "selected_features", "oob_error", "hotstart_forward", "hotstart_backward", "validation", "internal_tuning", "marshal") mlr_reflections$learner_properties = list( classif = c(tmp, "twoclass", "multiclass"), regr = tmp @@ -127,11 +129,6 @@ local({ regr = list(response = "response", se = c("response", "se"), quantiles = c("response", "quantiles"), distr = c("response", "se", "distr")) ) - mlr_reflections$learner_fallback = list( - classif = "classif.featureless", - regr = "regr.featureless" - ) - # Allowed tags for parameters mlr_reflections$learner_param_tags = c("train", "predict", "hotstart", "importance", "threads", "required", "internal_tuning") @@ -160,6 +157,9 @@ local({ ### Logger mlr_reflections$loggers = list() - ### cache package version + ### Cached package version mlr_reflections$package_version = packageVersion("mlr3") + + ### Loaded packages + mlr_reflections$loaded_packages = "mlr3" }) diff --git a/R/mlr_test_helpers.R b/R/mlr_test_helpers.R index 7604bd003..35568ca3c 100644 --- a/R/mlr_test_helpers.R +++ b/R/mlr_test_helpers.R @@ -24,6 +24,7 @@ #' the task, learner and prediction of the returned `result`. #' #' For example usages you can look at the autotests in various mlr3 source repositories such as mlr3learners. +#' More information can be found in the `inst/testthat/autotest.R` file. #' #' **Parameters**: #' @@ -42,7 +43,7 @@ #' Whether to check that running the learner twice with the same seed should result in identical predictions. #' Default is `TRUE`. #' * `configure_learner` (`function(learner, task)`)\cr -#' Before running a `learner` on a `task`, this function allows to change its parameter values depending on the input task. +#' Before running a `learner` on a `task`, this function allows to change its parameter values depending on the input task. #' #' @section run_paramtest(): #' diff --git a/R/partition.R b/R/partition.R index 908566bfa..8eabf1a45 100644 --- a/R/partition.R +++ b/R/partition.R @@ -12,7 +12,7 @@ #' @export #' @examples #' # regression task partitioned into training and test set -#' task = tsk("boston_housing") +#' task = tsk("california_housing") #' split = partition(task, ratio = 0.5) #' data = data.frame( #' y = c(task$truth(split$train), task$truth(split$test)), diff --git a/R/resample.R b/R/resample.R index 0ef5b41fe..cc1bb88f2 100644 --- a/R/resample.R +++ b/R/resample.R @@ -70,6 +70,7 @@ resample = function(task, learner, resampling, store_models = FALSE, store_backe if (!resampling$is_instantiated) { resampling = resampling$instantiate(task) } + n = resampling$iters pb = if (isNamespaceLoaded("progressr")) { # NB: the progress bar needs to be created in this env diff --git a/R/task_converters.R b/R/task_converters.R index cc1eb78b0..40fe9e653 100644 --- a/R/task_converters.R +++ b/R/task_converters.R @@ -35,13 +35,13 @@ convert_task = function(intask, target = NULL, new_type = NULL, drop_original_ta # copy row_roles / col_roles / properties newtask$row_roles = intask$row_roles props = intersect(mlr_reflections$task_col_roles[[intask$task_type]], mlr_reflections$task_col_roles[[new_type]]) - newtask$col_roles[props] = intask$col_roles[props] - newtask$set_col_roles(target, "target") - - # Add the original target(s) as features, only keeping 'new_target'. - if (!all(intask$target_names == target)) { - newtask$set_col_roles(setdiff(intask$col_roles$target, target), "feature") - } + col_roles = intask$col_roles[props] + # add the original target(s) as features, only keeping 'new_target' + col_roles$feature = c(col_roles$feature, setdiff(intask$col_roles$target, target)) + col_roles$target = target + # remove new target from features + col_roles$feature = setdiff(col_roles$feature, target) + newtask$col_roles[props] = col_roles # during prediction, when target is NA, we do not call droplevels if (assert_flag(drop_levels)) { diff --git a/R/warn_deprecated.R b/R/warn_deprecated.R index 0bfd89488..3e9b41259 100644 --- a/R/warn_deprecated.R +++ b/R/warn_deprecated.R @@ -8,7 +8,7 @@ #' `mlr3.warn_deprecated = FALSE`. #' #' The warning is of the format -#' " is deprecated and will be removed in the future." +#' "what is deprecated and will be removed in the future." #' #' Use the 'deprecated_binding()' helper function to create an active binding #' that generates a warning when accessed. diff --git a/R/worker.R b/R/worker.R index d8543e43a..e52daa23f 100644 --- a/R/worker.R +++ b/R/worker.R @@ -18,7 +18,7 @@ learner_train = function(learner, task, train_row_ids = NULL, test_row_ids = NUL stopf("Learner '%s' on task '%s' returned NULL during internal %s()", learner$id, task$id, mode) } - if (learner$encapsulate[["train"]] == "callr") { + if (learner$encapsulation[["train"]] == "callr") { model = marshal_model(model, inplace = TRUE) } @@ -67,7 +67,7 @@ learner_train = function(learner, task, train_row_ids = NULL, test_row_ids = NUL mode, learner$id, task$id, task$nrow, learner = learner$clone()) # call train_wrapper with encapsulation - result = encapsulate(learner$encapsulate["train"], + result = encapsulate(learner$encapsulation["train"], .f = train_wrapper, .args = list(learner = learner, task = task), .pkgs = learner$packages, @@ -195,12 +195,12 @@ learner_predict = function(learner, task, row_ids = NULL) { lg$debug("Calling predict method of Learner '%s' on task '%s' with %i observations", learner$id, task$id, task$nrow, learner = learner$clone()) - if (isTRUE(all.equal(learner$encapsulate[["predict"]], "callr"))) { + if (isTRUE(all.equal(learner$encapsulation[["predict"]], "callr"))) { learner$model = marshal_model(learner$model, inplace = TRUE) } result = encapsulate( - learner$encapsulate["predict"], + learner$encapsulation["predict"], .f = predict_wrapper, .args = list(task = task, learner = learner), .pkgs = learner$packages, @@ -268,6 +268,16 @@ workhorse = function(iteration, task, learner, resampling, param_values = NULL, old_blas_threads = RhpcBLASctl::blas_get_num_procs() on.exit(RhpcBLASctl::blas_set_num_threads(old_blas_threads), add = TRUE) RhpcBLASctl::blas_set_num_threads(1) + } else { # try the bare minimum to disable threading of the most popular blas implementations + old_blas = Sys.getenv("OPENBLAS_NUM_THREADS") + old_mkl = Sys.getenv("MKL_NUM_THREADS") + Sys.setenv(OPENBLAS_NUM_THREADS = 1) + Sys.setenv(MKL_NUM_THREADS = 1) + + on.exit({ + Sys.setenv(OPENBLAS_NUM_THREADS = old_blas) + Sys.setenv(MKL_NUM_THREADS = old_mkl) + }, add = TRUE) } } # restore logger thresholds @@ -316,6 +326,10 @@ workhorse = function(iteration, task, learner, resampling, param_values = NULL, lg$debug("Creating Prediction for predict set '%s'", set) learner_predict(learner, task, row_ids) }, set = predict_sets, row_ids = pred_data$sets, task = pred_data$tasks) + + if (!length(predict_sets)) { + learner$state$predict_time = 0L + } pdatas = discard(pdatas, is.null) # set the model slot after prediction so it can be sent back to the main process @@ -362,7 +376,7 @@ process_model_before_predict = function(learner, store_models, is_sequential, un # and also, do we even need to send it back at all? currently_marshaled = is_marshaled_model(learner$model) - predict_needs_marshaling = isTRUE(all.equal(learner$encapsulate[["predict"]], "callr")) + predict_needs_marshaling = isTRUE(all.equal(learner$encapsulation[["predict"]], "callr")) final_needs_marshaling = !is_sequential || !unmarshal # the only scenario in which we keep a copy is when we now have the model in the correct form but need to transform diff --git a/README.md b/README.md index c71e1972c..ae9be1201 100644 --- a/README.md +++ b/README.md @@ -19,51 +19,53 @@ Status](https://www.r-pkg.org/badges/version-ago/mlr3)](https://cran.r-project.o ## Resources (for users and developers) -- We have written a [book](https://mlr3book.mlr-org.com/). This should - be the central entry point to the package. -- The [mlr-org website](https://mlr-org.com/) includes for example a - [gallery](https://mlr-org.com/gallery.html) with case studies. -- [Reference manual](https://mlr3.mlr-org.com/reference/) -- [FAQ](https://mlr-org.com/faq.html) -- Ask questions on [Stackoverflow (tag - \#mlr3)](https://stackoverflow.com/questions/tagged/mlr3) -- **Extension Learners** - - Recommended core regression, classification, and survival learners - are in [mlr3learners](https://github.com/mlr-org/mlr3learners) - - All others are in - [mlr3extralearners](https://github.com/mlr-org/mlr3extralearners) - - Use the [learner search](https://mlr-org.com/learners.html) to get a - simple overview -- **Cheatsheets** - - [Overview of cheatsheets](https://cheatsheets.mlr-org.com) - - [mlr3](https://cheatsheets.mlr-org.com/mlr3.pdf) - - [mlr3tuning](https://cheatsheets.mlr-org.com/mlr3tuning.pdf) - - [mlr3pipelines](https://cheatsheets.mlr-org.com/mlr3pipelines.pdf) -- **Videos**: - - [useR2019 talk on mlr3](https://www.youtube.com/watch?v=wsP2hiFnDQs) - - [useR2019 talk on mlr3pipelines and - mlr3tuning](https://www.youtube.com/watch?v=gEW5RxkbQuQ) - - [useR2020 tutorial on mlr3, mlr3tuning and - mlr3pipelines](https://www.youtube.com/watch?v=T43hO2o_nZw) - -- **Courses/Lectures** - - The course [Introduction to Machine learning - (I2ML)](https://slds-lmu.github.io/i2ml/) is a free and open flipped - classroom course on the basics of machine learning. `mlr3` is used - in the - [demos](https://github.com/slds-lmu/lecture_i2ml/tree/master/code-demos-pdf) - and - [exercises](https://github.com/slds-lmu/lecture_i2ml/tree/master/exercises). -- **Templates/Tutorials** - - [mlr3-targets](https://github.com/mlr-org/mlr3-targets): Tutorial - showcasing how to use {mlr3} with - [targets](https://docs.ropensci.org/targets/) for reproducible ML - workflow automation. -- [List of extension packages](https://mlr-org.com/ecosystem.html) -- [mlr-outreach](https://github.com/mlr-org/mlr-outreach) contains - public talks and slides resources. -- [Wiki](https://github.com/mlr-org/mlr3/wiki): Contains mainly - information for developers. +- We have written a [book](https://mlr3book.mlr-org.com/). This should + be the central entry point to the package. +- The [mlr-org website](https://mlr-org.com/) includes for example a + [gallery](https://mlr-org.com/gallery.html) with case studies. +- [Reference manual](https://mlr3.mlr-org.com/reference/) +- [FAQ](https://mlr-org.com/faq.html) +- Ask questions on [Stackoverflow (tag + \#mlr3)](https://stackoverflow.com/questions/tagged/mlr3) +- **Extension Learners** + - Recommended core regression, classification, and survival + learners are in + [mlr3learners](https://github.com/mlr-org/mlr3learners) + - All others are in + [mlr3extralearners](https://github.com/mlr-org/mlr3extralearners) + - Use the [learner search](https://mlr-org.com/learners.html) to + get a simple overview +- **Cheatsheets** + - [Overview of cheatsheets](https://cheatsheets.mlr-org.com) + - [mlr3](https://cheatsheets.mlr-org.com/mlr3.pdf) + - [mlr3tuning](https://cheatsheets.mlr-org.com/mlr3tuning.pdf) + - [mlr3pipelines](https://cheatsheets.mlr-org.com/mlr3pipelines.pdf) +- **Videos**: + - [useR2019 talk on + mlr3](https://www.youtube.com/watch?v=wsP2hiFnDQs) + - [useR2019 talk on mlr3pipelines and + mlr3tuning](https://www.youtube.com/watch?v=gEW5RxkbQuQ) + - [useR2020 tutorial on mlr3, mlr3tuning and + mlr3pipelines](https://www.youtube.com/watch?v=T43hO2o_nZw) + +- **Courses/Lectures** + - The course [Introduction to Machine learning + (I2ML)](https://slds-lmu.github.io/i2ml/) is a free and open + flipped classroom course on the basics of machine learning. + `mlr3` is used in the + [demos](https://github.com/slds-lmu/lecture_i2ml/tree/master/code-demos-pdf) + and + [exercises](https://github.com/slds-lmu/lecture_i2ml/tree/master/exercises). +- **Templates/Tutorials** + - [mlr3-targets](https://github.com/mlr-org/mlr3-targets): + Tutorial showcasing how to use {mlr3} with + [targets](https://docs.ropensci.org/targets/) for reproducible + ML workflow automation. +- [List of extension packages](https://mlr-org.com/ecosystem.html) +- [mlr-outreach](https://github.com/mlr-org/mlr-outreach) contains + public talks and slides resources. +- [Wiki](https://github.com/mlr-org/mlr3/wiki): Contains mainly + information for developers. ## Installation @@ -156,16 +158,16 @@ rr$score(measure)[, .(task_id, learner_id, iteration, classif.acc)] ``` ## task_id learner_id iteration classif.acc - ## 1: palmerpenguins::penguins classif.rpart 1 0.9391304 + ## 1: palmerpenguins::penguins classif.rpart 1 0.8956522 ## 2: palmerpenguins::penguins classif.rpart 2 0.9478261 - ## 3: palmerpenguins::penguins classif.rpart 3 0.9298246 + ## 3: palmerpenguins::penguins classif.rpart 3 0.9649123 ``` r rr$aggregate(measure) ``` ## classif.acc - ## 0.938927 + ## 0.9361302 ## Extension Packages @@ -195,71 +197,72 @@ would result in non-trivial API changes. ## Design principles -- Only the basic building blocks for machine learning are implemented in - this package. -- Focus on computation here. No visualization or other stuff. That can - go in extra packages. -- Overcome the limitations of R’s [S3 - classes](https://adv-r.hadley.nz/s3.html) with the help of - [R6](https://cran.r-project.org/package=R6). -- Embrace [R6](https://cran.r-project.org/package=R6) for a clean - OO-design, object state-changes and reference semantics. This might be - less “traditional R”, but seems to fit `mlr` nicely. -- Embrace [`data.table`](https://cran.r-project.org/package=data.table) - for fast and convenient data frame computations. -- Combine `data.table` and `R6`, for this we will make heavy use of list - columns in data.tables. -- Defensive programming and type safety. All user input is checked with - [`checkmate`](https://cran.r-project.org/package=checkmate). Return - types are documented, and mechanisms popular in base R which - “simplify” the result unpredictably (e.g., `sapply()` or `drop` - argument in `[.data.frame`) are avoided. -- Be light on dependencies. `mlr3` requires the following packages at - runtime: - - [`parallelly`](https://cran.r-project.org/package=parallelly): - Helper functions for parallelization. No extra recursive - dependencies. - - [`future.apply`](https://cran.r-project.org/package=future.apply): - Resampling and benchmarking is parallelized with the - [`future`](https://cran.r-project.org/package=future) abstraction - interfacing many parallel backends. - - [`backports`](https://cran.r-project.org/package=backports): Ensures - backward compatibility with older R releases. Developed by members - of the `mlr` team. No recursive dependencies. - - [`checkmate`](https://cran.r-project.org/package=checkmate): Fast - argument checks. Developed by members of the `mlr` team. No extra - recursive dependencies. - - [`mlr3misc`](https://cran.r-project.org/package=mlr3misc): - Miscellaneous functions used in multiple mlr3 [extension - packages](https://mlr-org.com/ecosystem.html). Developed by the - `mlr` team. - - [`paradox`](https://cran.r-project.org/package=paradox): - Descriptions for parameters and parameter sets. Developed by the - `mlr` team. No extra recursive dependencies. - - [`R6`](https://cran.r-project.org/package=R6): Reference class - objects. No recursive dependencies. - - [`data.table`](https://cran.r-project.org/package=data.table): - Extension of R’s `data.frame`. No recursive dependencies. - - [`digest`](https://cran.r-project.org/package=digest) (via - `mlr3misc`): Hash digests. No recursive dependencies. - - [`uuid`](https://cran.r-project.org/package=uuid): Create unique - string identifiers. No recursive dependencies. - - [`lgr`](https://cran.r-project.org/package=lgr): Logging facility. - No extra recursive dependencies. - - [`mlr3measures`](https://cran.r-project.org/package=mlr3measures): - Performance measures. No extra recursive dependencies. - - [`mlbench`](https://cran.r-project.org/package=mlbench): A - collection of machine learning data sets. No dependencies. - - [`palmerpenguins`](https://cran.r-project.org/package=palmerpenguins): - A classification data set about penguins, used on examples and - provided as a toy task. No dependencies. -- [Reflections](https://en.wikipedia.org/wiki/Reflection_%28computer_programming%29): - Objects are queryable for properties and capabilities, allowing you to - program on them. -- Additional functionality that comes with extra dependencies: - - To capture output, warnings and exceptions, - [`evaluate`](https://cran.r-project.org/package=evaluate) and - [`callr`](https://cran.r-project.org/package=callr) can be used. +- Only the basic building blocks for machine learning are implemented + in this package. +- Focus on computation here. No visualization or other stuff. That can + go in extra packages. +- Overcome the limitations of R’s [S3 + classes](https://adv-r.hadley.nz/s3.html) with the help of + [R6](https://cran.r-project.org/package=R6). +- Embrace [R6](https://cran.r-project.org/package=R6) for a clean + OO-design, object state-changes and reference semantics. This might + be less “traditional R”, but seems to fit `mlr` nicely. +- Embrace + [`data.table`](https://cran.r-project.org/package=data.table) for + fast and convenient data frame computations. +- Combine `data.table` and `R6`, for this we will make heavy use of + list columns in data.tables. +- Defensive programming and type safety. All user input is checked + with [`checkmate`](https://cran.r-project.org/package=checkmate). + Return types are documented, and mechanisms popular in base R which + “simplify” the result unpredictably (e.g., `sapply()` or `drop` + argument in `[.data.frame`) are avoided. +- Be light on dependencies. `mlr3` requires the following packages at + runtime: + - [`parallelly`](https://cran.r-project.org/package=parallelly): + Helper functions for parallelization. No extra recursive + dependencies. + - [`future.apply`](https://cran.r-project.org/package=future.apply): + Resampling and benchmarking is parallelized with the + [`future`](https://cran.r-project.org/package=future) + abstraction interfacing many parallel backends. + - [`backports`](https://cran.r-project.org/package=backports): + Ensures backward compatibility with older R releases. Developed + by members of the `mlr` team. No recursive dependencies. + - [`checkmate`](https://cran.r-project.org/package=checkmate): + Fast argument checks. Developed by members of the `mlr` team. No + extra recursive dependencies. + - [`mlr3misc`](https://cran.r-project.org/package=mlr3misc): + Miscellaneous functions used in multiple mlr3 [extension + packages](https://mlr-org.com/ecosystem.html). Developed by the + `mlr` team. + - [`paradox`](https://cran.r-project.org/package=paradox): + Descriptions for parameters and parameter sets. Developed by the + `mlr` team. No extra recursive dependencies. + - [`R6`](https://cran.r-project.org/package=R6): Reference class + objects. No recursive dependencies. + - [`data.table`](https://cran.r-project.org/package=data.table): + Extension of R’s `data.frame`. No recursive dependencies. + - [`digest`](https://cran.r-project.org/package=digest) (via + `mlr3misc`): Hash digests. No recursive dependencies. + - [`uuid`](https://cran.r-project.org/package=uuid): Create unique + string identifiers. No recursive dependencies. + - [`lgr`](https://cran.r-project.org/package=lgr): Logging + facility. No extra recursive dependencies. + - [`mlr3measures`](https://cran.r-project.org/package=mlr3measures): + Performance measures. No extra recursive dependencies. + - [`mlbench`](https://cran.r-project.org/package=mlbench): A + collection of machine learning data sets. No dependencies. + - [`palmerpenguins`](https://cran.r-project.org/package=palmerpenguins): + A classification data set about penguins, used on examples and + provided as a toy task. No dependencies. +- [Reflections](https://en.wikipedia.org/wiki/Reflection_%28computer_programming%29): + Objects are queryable for properties and capabilities, allowing you + to program on them. +- Additional functionality that comes with extra dependencies: + - To capture output, warnings and exceptions, + [`evaluate`](https://cran.r-project.org/package=evaluate) and + [`callr`](https://cran.r-project.org/package=callr) can be used. ## Contributing to mlr3 diff --git a/inst/extdata/california_housing.R b/inst/extdata/california_housing.R new file mode 100644 index 000000000..38125d23b --- /dev/null +++ b/inst/extdata/california_housing.R @@ -0,0 +1,5 @@ +# download data from https://www.kaggle.com/datasets/camnugent/california-housing-prices +root = rprojroot::find_package_root_file() +data = data.table::fread("housing.csv") +data[, ocean_proximity := as.factor(ocean_proximity)] +saveRDS(data, file = file.path(root, "inst", "extdata", "california_housing.rds"), version = 2L) diff --git a/inst/extdata/california_housing.rds b/inst/extdata/california_housing.rds new file mode 100644 index 000000000..39fe3c29c Binary files /dev/null and b/inst/extdata/california_housing.rds differ diff --git a/inst/testthat/helper_autotest.R b/inst/testthat/helper_autotest.R index 11b2060a3..279e0ccf5 100644 --- a/inst/testthat/helper_autotest.R +++ b/inst/testthat/helper_autotest.R @@ -1,17 +1,38 @@ -# Learner autotest suite -# -# `run_experiment(task, learner)` runs a single experiment. -# Returns a list with success flag "status" (`logical(1)`), -# "experiment" (partially constructed experiment), and "error" -# (informative error message). -# -# `run_autotest(learner)` generates multiple tasks, depending on the properties of the learner. -# and tests the learner on each task, with each predict type. -# To debug, simply run `result = run_autotest(learner)` and proceed with investigating -# the task, learner and prediction of the returned `result`. +#' @title Learner Autotest Suite +#' +#' @description +#' The autotest suite is a collection of functions to test learners in a standardized way. +#' Extension packages need to specialize the S3 methods in the file. # -# NB: Extension packages need to specialize the S3 methods in the file. +#' @details +#' `run_autotest(learner)` generates multiple tasks, depending on the properties of the learner and tests the learner on each task, with each predict type. +#' Calls `generate_tasks()` to generate tasks and `run_experiment()` to run the experiments. +#' See `generate_tasks()` for a list of tasks that are generated. +#' To debug, simply run `result = run_autotest(learner)` and proceed with investigating he task, learner and prediction of the returned `result`. +#' +#' `run_experiment(task, learner)` runs a single experiment. +#' Calls `train()` and `predict()` on the learner and checks the prediction with `score()`. +#' The prediction is checked with `sanity_check()`. +#' +#' `generate_tasks(learner)` generates multiple tasks for a given learner. +#' Calls `generate_data()` and `generate_generic_tasks()` to generate tasks with different feature types. +#' +#' @noRd +NULL +#' @title Generate Tasks for a Learner +#' +#' @description +#' Generates multiple tasks for a given [Learner], based on its properties. +#' +#' @param learner [Learner]\cr +#' Learner to generate tasks for. +#' @param proto [Task]\cr +#' Prototype task to generate tasks from. +#' +#' @return (List of [Task]s). +#' +#' @noRd generate_generic_tasks = function(learner, proto) { tasks = list() n = proto$nrow @@ -76,6 +97,20 @@ generate_generic_tasks = function(learner, proto) { }) } +#' @title Generate Data for a Learner +#' +#' @description +#' Generates data for a given [Learner], based on its supported feature types. +#' Data is created for logical, integer, numeric, character, factor, ordered, and POSIXct features. +#' +#' @param learner [Learner]\cr +#' Learner to generate data for. +#' @param N `integer(1)`\cr +#' Number of rows of generated data. +#' +#' @return [data.table::data.table()] +#' +#' @noRd generate_data = function(learner, N) { generate_feature = function(type) { switch(type, @@ -96,14 +131,22 @@ generate_data = function(learner, N) { #' #' @description #' Generates multiple tasks for a given [Learner], based on its properties. -#' This function is primarily used for unit tests, but can also assist while -#' writing custom learners. +#' This function is primarily used for unit tests, but can also assist while writing custom learners. +#' The following tasks are created: +#' * `feat_single_*`: Tasks with a single feature type. +#' * `feat_all_*`: Task with all supported feature types. +#' * `missings_*`: Task with missing values. +#' * `utf8_feature_names_*`: Task with non-ascii feature names. +#' * `sanity`: Task with a simple dataset to check if the learner is working. +#' * `sanity_reordered`: Task with the same dataset as `sanity`, but with reordered columns. +#' * `sanity_switched`: Task with the same dataset as `sanity`, but with the positive class switched. #' -#' @param learner :: [Learner]. -#' @param N :: `integer(1)`\cr +#' @param learner [Learner]\cr +#' Learner to generate tasks for. +#' @param N `integer(1)`\cr #' Number of rows of generated tasks. #' -#' @return (List of [Task]s). +#' @return `list` of [Task]s #' @keywords internal #' @export #' @examples @@ -184,6 +227,17 @@ generate_tasks.LearnerRegr = function(learner, N = 30L) { } registerS3method("generate_tasks", "LearnerRegr", generate_tasks.LearnerRegr) +#' @title Sanity Check for Predictions +#' +#' @description +#' Checks the sanity of a prediction. +#' +#' @param prediction [Prediction]\cr +#' Prediction to check. +#' +#' @return (`logical(1)`). +#' +#' @noRd sanity_check = function(prediction, ...) { UseMethod("sanity_check") } @@ -199,7 +253,34 @@ sanity_check.PredictionRegr = function(prediction, ...) { } registerS3method("sanity_check", "LearnerRegr", sanity_check.PredictionRegr) + +#' @title Run a Single Learner Test +#' +#' @description +#' Runs a single experiment with a given task and learner. +#' +#' @param task [Task]\cr +#' Task to run the experiment on. +#' @param learner [Learner]\cr +#' Learner to run the experiment with. +#' @param seed `integer(1)`\cr +#' Seed to use for the experiment. +#' If `NULL`, a random seed is generated. +#' @param configure_learner `function(learner, task)`\cr +#' Function to configure the learner before training. +#' Useful when learner settings need to be adjusted for a specific task. +#' +#' @return `list` with the following elements: +#' - `ok` (`logical(1)`): Success flag. +#' - `learner` ([Learner]): Learner used for the experiment. +#' - `prediction` ([Prediction]): Prediction object. +#' - `error` (`character()`): Error message if `ok` is `FALSE`. +#' - `seed` (`integer(1)`): Seed used for the experiment. +#' +#' @noRd run_experiment = function(task, learner, seed = NULL, configure_learner = NULL) { + + # function to collect error message and objects err = function(info, ...) { info = sprintf(info, ...) list( @@ -210,6 +291,7 @@ run_experiment = function(task, learner, seed = NULL, configure_learner = NULL) ) } + # seed handling if (is.null(seed)) { seed = sample.int(floor(.Machine$integer.max / 2L), 1L) } @@ -230,8 +312,8 @@ run_experiment = function(task, learner, seed = NULL, configure_learner = NULL) } prediction = NULL score = NULL - learner$encapsulate = c(train = "evaluate", predict = "evaluate") + # check train stage = "train()" # enable weights @@ -247,28 +329,23 @@ run_experiment = function(task, learner, seed = NULL, configure_learner = NULL) } } - ok = try(learner$train(task), silent = TRUE) + ok = suppressWarnings(try(learner$train(task), silent = TRUE)) if (inherits(ok, "try-error")) { return(err(as.character(ok))) } - log = learner$log[stage == "train"] - if ("error" %in% log$class) { - return(err("train log has errors: %s", mlr3misc::str_collapse(log[class == "error", msg]))) - } if (is.null(learner$model)) { return(err("model is NULL")) } + # check predict stage = "predict()" - prediction = try(learner$predict(task), silent = TRUE) - if (inherits(ok, "try-error")) { + prediction = suppressWarnings(try(learner$predict(task), silent = TRUE)) + if (inherits(prediction, "try-error")) { + ok = prediction + prediction = NULL return(err(as.character(ok))) } - log = learner$log[stage == "predict"] - if ("error" %in% log$class) { - return(err("predict log has errors: %s", mlr3misc::str_collapse(log[class == "error", msg]))) - } msg = checkmate::check_class(prediction, "Prediction") if (!isTRUE(msg)) { return(err(msg)) @@ -308,8 +385,9 @@ run_experiment = function(task, learner, seed = NULL, configure_learner = NULL) } } - + # check score stage = "score()" + score = try( prediction$score(mlr3::default_measures(learner$task_type), task = task, @@ -317,7 +395,9 @@ run_experiment = function(task, learner, seed = NULL, configure_learner = NULL) train_set = task$row_ids ), silent = TRUE) if (inherits(score, "try-error")) { - return(err(as.character(score))) + ok = score + score = NULL + return(err(as.character(ok))) } msg = checkmate::check_numeric(score, any.missing = FALSE) if (!isTRUE(msg)) { @@ -325,11 +405,11 @@ run_experiment = function(task, learner, seed = NULL, configure_learner = NULL) } # run sanity check on sanity task - if (startsWith(task$id, "sanity") && ! - sanity_check(prediction, task = task, learner = learner, train_set = task$row_ids)) { + if (startsWith(task$id, "sanity") && !sanity_check(prediction, task = task, learner = learner, train_set = task$row_ids)) { return(err("sanity check failed")) } + # check importance, selected_features and oob_error methods if (startsWith(task$id, "feat_all")) { if ("importance" %in% learner$properties) { importance = learner$importance() @@ -366,6 +446,37 @@ run_experiment = function(task, learner, seed = NULL, configure_learner = NULL) return(list(ok = TRUE, learner = learner, prediction = prediction, error = character(), seed = seed)) } +#' @title Run Autotest for a Learner +#' +#' @description +#' Runs a series of experiments with a given learner on multiple tasks. +#' +#' @param learner ([Learner])\cr +#' The learner to test. +#' @param N (`integer(1)`)\cr +#' Number of rows of generated tasks. +#' @param exclude (`character()`)\cr +#' Regular expression to exclude tasks from the test. +#' Run `generate_tasks(learner)` to see all available tasks. +#' @param predict_types (`character()`)\cr +#' Predict types to test. +#' @param check_replicable (`logical(1)`)\cr +#' Check if the results are replicable. +#' @param configure_learner (`function(learner, task)`)\cr +#' Function to configure the learner before training. +#' Useful when learner settings need to be adjusted for a specific task. +#' +#' @return If the test was successful, `TRUE` is returned. +#' If the test failed, a `list` with the following elements is returned: +#' - `ok` (`logical(1)`): Success flag. +#' - `seed` (`integer(1)`): Seed used for the experiment. +#' - `task` ([Task]): Task used for the experiment. +#' - `learner` ([Learner]): Learner used for the experiment. +#' - `prediction` ([Prediction]): Prediction object. +#' - `score` (`numeric(1)`): Score of the prediction. +#' - `error` (`character()`): Error message if `ok` is `FALSE`. +# +#' @noRd run_autotest = function(learner, N = 30L, exclude = NULL, predict_types = learner$predict_types, check_replicable = TRUE, configure_learner = NULL) { # nolint if (!is.null(configure_learner)) { checkmate::assert_function(configure_learner, args = c("learner", "task")) @@ -373,11 +484,11 @@ run_autotest = function(learner, N = 30L, exclude = NULL, predict_types = learne learner = learner$clone(deep = TRUE) id = learner$id tasks = generate_tasks(learner, N = N) + if (!is.null(exclude)) { tasks = tasks[!grepl(exclude, names(tasks))] } - sanity_runs = list() make_err = function(msg, ...) { run$ok = FALSE @@ -385,11 +496,6 @@ run_autotest = function(learner, N = 30L, exclude = NULL, predict_types = learne run } - # param_tags = unique(unlist(learner$param_set$tags)) - # if (!test_subset(param_tags, mlr_reflections$learner_param_tags)) { - # return(list(ok = FALSE, error = "Invalid parameter tag(s), check `mlr_reflections$learner_param_tags`.")) - # } - for (task in tasks) { for (predict_type in predict_types) { learner$id = sprintf("%s:%s", id, predict_type) @@ -429,26 +535,22 @@ run_autotest = function(learner, N = 30L, exclude = NULL, predict_types = learne } } - - return(TRUE) } #' @title Check Parameters of mlr3 Learners -#' @description Checks parameters of mlr3learners against parameters defined in -#' the upstream functions of the respective learners. +#' +#' @description +#' Checks parameters of mlr3learners against parameters defined in the upstream functions of the respective learners. #' #' @details -#' Some learners do not have all of their parameters stored within the learner -#' function that is called within `.train()`. Sometimes learners come with a -#' "control" function, e.g. [glmnet::glmnet.control()]. Such need to be checked -#' as well since they make up the full ParamSet of the respective learner. +#' Some learners do not have all of their parameters stored within the learner function that is called within `.train()`. +#' Sometimes learners come with a "control" function, e.g. [glmnet::glmnet.control()]. +#' Such need to be checked as well since they make up the full ParamSet of the respective learner. #' -#' To work nicely with the defined ParamSet, certain parameters need to be -#' excluded because these are only present in either the "control" object or the -#' actual top-level function call. Such exclusions should go into argument -#' `exclude` with a comment for the reason of the exclusion. See examples for -#' more information. +#' To work nicely with the defined ParamSet, certain parameters need to be excluded because these are only present in either the "control" object or the actual top-level function call. +#' Such exclusions should go into argument `exclude` with a comment for the reason of the exclusion. +#' See examples for more information. #' #' @param learner ([mlr3::Learner])\cr #' The constructed learner. diff --git a/inst/testthat/helper_expectations.R b/inst/testthat/helper_expectations.R index 91bdee35e..d6c2ca1b0 100644 --- a/inst/testthat/helper_expectations.R +++ b/inst/testthat/helper_expectations.R @@ -474,7 +474,6 @@ expect_resampling = function(r, task = NULL) { expect_hash(r$task_hash, 1L) if (!is.null(task)) { ids = task$row_ids - testthat::expect_equal(task$hash, r$task_hash) } checkmate::expect_count(r$iters, positive = TRUE) @@ -518,7 +517,7 @@ expect_measure = function(m) { testthat::expect_output(print(m), "Measure") if ("requires_no_prediction" %in% m$properties) { - testthat::expect_true(is.null(m$predict_sets)) + testthat::expect_null(m$predict_sets) } expect_id(m$id) diff --git a/man-roxygen/field_hash.R b/man-roxygen/field_hash.R deleted file mode 100644 index 578a8559b..000000000 --- a/man-roxygen/field_hash.R +++ /dev/null @@ -1,2 +0,0 @@ -#' @field hash (`character(1)`)\cr -#' Hash (unique identifier) for this object. diff --git a/man-roxygen/section_parallelization.R b/man-roxygen/section_parallelization.R index 5d2d1f926..6c4fed578 100644 --- a/man-roxygen/section_parallelization.R +++ b/man-roxygen/section_parallelization.R @@ -4,3 +4,5 @@ #' One job is one resampling iteration, and all jobs are send to an apply function #' from \CRANpkg{future.apply} in a single batch. #' To select a parallel backend, use [future::plan()]. +#' More on parallelization can be found in the book: +#' \url{https://mlr3book.mlr-org.com/chapters/chapter10/advanced_technical_aspects_of_mlr3.html} diff --git a/man/BenchmarkResult.Rd b/man/BenchmarkResult.Rd index 630cb02d7..0e1f18d9a 100644 --- a/man/BenchmarkResult.Rd +++ b/man/BenchmarkResult.Rd @@ -20,7 +20,7 @@ Do not modify any extracted object without cloning it first. \section{S3 Methods}{ \itemize{ -\item \code{as.data.table(rr, ..., reassemble_learners = TRUE, convert_predictions = TRUE, predict_sets = "test")}\cr +\item \code{as.data.table(rr, ..., reassemble_learners = TRUE, convert_predictions = TRUE, predict_sets = "test", task_characteristics = FALSE)}\cr \link{BenchmarkResult} -> \code{\link[data.table:data.table]{data.table::data.table()}}\cr Returns a tabular view of the internal data. \item \code{c(...)}\cr diff --git a/man/Learner.Rd b/man/Learner.Rd index 3802365c5..fd602f326 100644 --- a/man/Learner.Rd +++ b/man/Learner.Rd @@ -46,8 +46,6 @@ To filter variables using the importance scores, see package \CRANpkg{mlr3filter The learner must be tagged with property \code{"selected_features"}. \item \code{oob_error(...)}: Returns the out-of-bag error of the model as \code{numeric(1)}. The learner must be tagged with property \code{"oob_error"}. -\item \code{loglik(...)}: Extracts the log-likelihood (c.f. \code{\link[stats:logLik]{stats::logLik()}}). -This can be used in measures like \link{mlr_measures_aic} or \link{mlr_measures_bic}. \item \code{internal_valid_scores}: Returns the internal validation score(s) of the model as a named \code{list()}. Only available for \code{\link{Learner}}s with the \code{"validation"} property. If the learner is not trained yet, this returns \code{NULL}. @@ -306,11 +304,11 @@ Logged warnings as vector.} Logged errors as vector.} \item{\code{hash}}{(\code{character(1)})\cr -Hash (unique identifier) for this object.} +Hash (unique identifier) for this object. +The hash is calculated based on the learner id, the parameter settings, the predict type, the fallback hash, the parallel predict setting, the validate setting, and the predict sets.} \item{\code{phash}}{(\code{character(1)})\cr -Hash (unique identifier) for this partial object, excluding some components -which are varied systematically during tuning (parameter values).} +Hash (unique identifier) for this partial object, excluding some components which are varied systematically during tuning (parameter values).} \item{\code{predict_type}}{(\code{character(1)})\cr Stores the currently active predict type, e.g. \code{"response"}. @@ -319,21 +317,11 @@ Must be an element of \verb{$predict_types}.} \item{\code{param_set}}{(\link[paradox:ParamSet]{paradox::ParamSet})\cr Set of hyperparameters.} -\item{\code{encapsulate}}{(named \code{character()})\cr -Controls how to execute the code in internal train and predict methods. -Must be a named character vector with names \code{"train"} and \code{"predict"}. -Possible values are \code{"none"}, \code{"try"}, \code{"evaluate"} (requires package \CRANpkg{evaluate}) and \code{"callr"} (requires package \CRANpkg{callr}). -When encapsulation is activated, a fallback learner must be set. -If no learner is set in \verb{$fallback}, the default fallback learner is used (see \code{mlr_reflections$task_types}). -See \code{\link[mlr3misc:encapsulate]{mlr3misc::encapsulate()}} for more details.} - \item{\code{fallback}}{(\link{Learner})\cr -Learner which is fitted to impute predictions in case that either the model fitting or the prediction of the top learner is not successful. -Requires encapsulation, otherwise errors are not caught and the execution is terminated before the fallback learner kicks in. -If you have not set encapsulation manually before, setting the fallback learner automatically -activates encapsulation using the \CRANpkg{evaluate} package. -Also see the section on error handling the mlr3book: -\url{https://mlr3book.mlr-org.com/chapters/chapter10/advanced_technical_aspects_of_mlr3.html#sec-error-handling}} +Returns the fallback learner set with \verb{$encapsulate()}.} + +\item{\code{encapsulation}}{(\code{character(2)})\cr +Returns the encapsulation settings set with \verb{$encapsulate()}.} \item{\code{hotstart_stack}}{(\link{HotstartStack})\cr. Stores \code{HotstartStack}.} @@ -352,6 +340,7 @@ Stores \code{HotstartStack}.} \item \href{#method-Learner-predict_newdata}{\code{Learner$predict_newdata()}} \item \href{#method-Learner-reset}{\code{Learner$reset()}} \item \href{#method-Learner-base_learner}{\code{Learner$base_learner()}} +\item \href{#method-Learner-encapsulate}{\code{Learner$encapsulate()}} \item \href{#method-Learner-clone}{\code{Learner$clone()}} } } @@ -598,6 +587,50 @@ Depth of recursion for multiple nested objects.} } } \if{html}{\out{
}} +\if{html}{\out{}} +\if{latex}{\out{\hypertarget{method-Learner-encapsulate}{}}} +\subsection{Method \code{encapsulate()}}{ +Sets the encapsulation method and fallback learner for the train and predict steps. +There are currently four different methods implemented: +\itemize{ +\item \code{"none"}: Just runs the learner in the current session and measures the elapsed time. +Does not keep a log, output is printed directly to the console. +Works well together with \code{\link[=traceback]{traceback()}}. +\item \code{"try"}: Similar to \code{"none"}, but catches error. +Output is printed to the console and not logged. +\item \code{"evaluate"}: Uses the package \CRANpkg{evaluate} to call the learner, measure time and do the logging. +\item \code{"callr"}: Uses the package \CRANpkg{callr} to call the learner, measure time and do the logging. +This encapsulation spawns a separate R session in which the learner is called. +While this comes with a considerable overhead, it also guards your session from being teared down by segfaults. +} + +The fallback learner is fitted to create valid predictions in case that either the model fitting or the prediction of the original learner fails. +If the training step or the predict step of the original learner fails, the fallback is used completely to predict predictions sets. +If the original learner only partially fails during predict step (usually in the form of missing to predict some observations or producing some \verb{NA`` predictions), these missing predictions are imputed by the fallback. Note that the fallback is always trained, as we do not know in advance whether prediction will fail. If the training step fails, the }$model\verb{field of the original learner is}NULL`. + +Also see the section on error handling the mlr3book: +\url{https://mlr3book.mlr-org.com/chapters/chapter10/advanced_technical_aspects_of_mlr3.html#sec-error-handling} +\subsection{Usage}{ +\if{html}{\out{
}}\preformatted{Learner$encapsulate(method, fallback = NULL)}\if{html}{\out{
}} +} + +\subsection{Arguments}{ +\if{html}{\out{
}} +\describe{ +\item{\code{method}}{\code{character(1)}\cr +One of \code{"none"}, \code{"try"}, \code{"evaluate"} or \code{"callr"}. +See the description for details.} + +\item{\code{fallback}}{\link{Learner}\cr +The fallback learner for failed predictions.} +} +\if{html}{\out{
}} +} +\subsection{Returns}{ +\code{self} (invisibly). +} +} +\if{html}{\out{
}} \if{html}{\out{}} \if{latex}{\out{\hypertarget{method-Learner-clone}{}}} \subsection{Method \code{clone()}}{ diff --git a/man/LearnerClassif.Rd b/man/LearnerClassif.Rd index 26680bd79..a62e3b668 100644 --- a/man/LearnerClassif.Rd +++ b/man/LearnerClassif.Rd @@ -85,6 +85,7 @@ Other Learner:
Inherited methods