Version 1.4.0 (#730, #855, #869, #872)

## Features * [BOinG](https://arxiv.org/abs/2111.05834): A two-stage Bayesian optimization approach to allow the optimizer to focus on the most promising regions. * [TurBO](https://arxiv.org/abs/1910.01739): Reimplementaion of TurBO-1 algorithm. * Updated pSMAC: Can pass arbitrary SMAC facades now. Added example and fixed tests. ## Improvements * Enabled caching for multi-objectives (#872). Costs are now normalized in `get_cost` or optionally in `average_cost`/`sum_cost`/`min_cost` to receive a single float value. Therefore, the cached cost values do not need to be updated everytime a new entry to the runhistory was added. ## Interface changes * We changed the location of Gaussian processes and random forests. They are in the folders `epm/gaussian_process` and `epm/random_forest` now. * Also, we restructured the optimizer folder and therefore the location of the acquisition functions and configuration chooser. * Multi-objective functions are located in the folder `multi_objective`. * pSMAC facade was moved to the facade directory. Co-authored-by: Difan Deng <[email protected]> Co-authored-by: Eddie Bergman <[email protected]> Co-authored-by: Carolin Benjamins <[email protected]> Co-authored-by: timruhkopf <[email protected]>
automl · Jul 14, 2022 · 83a9bbe · 83a9bbe
1 parent 99d1129
commit 83a9bbe
Show file tree

Hide file tree

Showing 111 changed files with 7,815 additions and 894 deletions.
diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
@@ -35,7 +35,7 @@ jobs:
 
     - name: Install dependencies
       run: |
-        pip install ".[dev]"
+        pip install ".[gpytorch,dev]"
 
     - name: Make docs
       run: |

diff --git a/.github/workflows/pytest.yml b/.github/workflows/pytest.yml
@@ -23,7 +23,7 @@ on:
 env:
   package-name: smac
   test-dir: tests
-  extra-requires: "[dev]"
+  extra-requires: "[gpytorch,dev]"
 
   # Arguments used for pytest
   pytest-args: >-

diff --git a/.gitignore b/.gitignore
@@ -137,3 +137,6 @@ dmypy.json
 
 # macOS files
 .DS_Store
+
+# Remove docker files
+docker
diff --git a/README.md b/README.md
@@ -15,7 +15,9 @@ efficiently decide which of two configurations performs better.
 SMAC3 is written in Python3 and continuously tested with Python 3.7, 3.8, 3.9, and 3.10. Its Random
 Forest is written in C++. In further texts, SMAC is representatively mentioned for SMAC3.
 
-[Documention](https://automl.github.io/SMAC3)
+> [Documention](https://automl.github.io/SMAC3)
+
+> [Roadmap](https://github.com/orgs/automl/projects/5/views/2)
 
 
 ## Installation

diff --git a/changelog.md b/changelog.md
@@ -1,3 +1,25 @@
+# 1.4.0
+
+## Features
+* [BOinG](https://arxiv.org/abs/2111.05834): A two-stage Bayesian optimization approach to allow the 
+optimizer to focus on the most promising regions.
+* [TurBO](https://arxiv.org/abs/1910.01739): Reimplementaion of TurBO-1 algorithm.
+* Updated pSMAC: Can pass arbitrary SMAC facades now. Added example and fixed tests.
+
+## Improvements
+* Enabled caching for multi-objectives (#872). Costs are now normalized in `get_cost` 
+or optionally in `average_cost`/`sum_cost`/`min_cost` to receive a single float value. Therefore,
+the cached cost values do not need to be updated everytime a new entry to the runhistory was added.
+
+## Interface changes
+* We changed the location of Gaussian processes and random forests. They are in the folders
+`epm/gaussian_process` and `epm/random_forest` now.
+* Also, we restructured the optimizer folder and therefore the location of the acquisition functions
+and configuration chooser.
+* Multi-objective functions are located in the folder `multi_objective`.
+* pSMAC facade was moved to the facade directory.
+
+
 # 1.3.4
 * Added reference to JMLR paper.
 * Typos in documentations.

diff --git a/docs/details/parallelism.rst b/docs/details/parallelism.rst
@@ -2,7 +2,7 @@ Parallelism
 ===========
 
 SMAC also provides a parallel mode to use several parallel computational resources (such as CPU cores).
-This variant of SMAC is called pSMAC (parallel SMAC).
+This variant of SMAC is called pSMAC (parallel SMAC) [1]_.
 The general idea is that all target algorithm run evaluations are shared between the individual SMAC runs
 such that all SMAC runs are better informed and can work together.
 
@@ -19,6 +19,12 @@ such that all SMAC runs are better informed and can work together.
 	SMAC also supports DASH. The documentation is in progress.
 
 
+.. [1] Ramage, S. E. A. (2015). Advances in meta-algorithmic software libraries for
+        distributed automated algorithm configuration (T). University of British
+        Columbia. Retrieved from
+        https://open.library.ubc.ca/collections/ubctheses/24/items/1.0167184.
+
+
 Commandline 
 ~~~~~~~~~~~
 To use pSMAC via the commandline interface, please specify the following two arguments:

diff --git a/examples/python/plot_scalarized_multi_objective.py b/examples/python/plot_scalarized_multi_objective.py
@@ -9,7 +9,7 @@
 
 import logging
 
-from smac.optimizer.multi_objective.parego import ParEGO
+from smac.multi_objective.parego import ParEGO
 
 logging.basicConfig(level=logging.INFO)
 

diff --git a/examples/python/plot_simple_multi_objective.py b/examples/python/plot_simple_multi_objective.py
@@ -27,7 +27,10 @@ def plot(all_x):
     plt.figure()
     for x in all_x:
         f1, f2 = schaffer(x)
-        plt.scatter(f1, f2, c="blue", alpha=0.1)
+        plt.scatter(f1, f2, c="blue", alpha=0.1, zorder=3000)
+
+    plt.vlines([1], 0, 4, linestyles="dashed", colors=["red"])
+    plt.hlines([1], 0, 4, linestyles="dashed", colors=["red"])
 
     plt.show()
 

diff --git a/examples/python/plot_svm_eips.py b/examples/python/plot_svm_eips.py
@@ -21,9 +21,7 @@
 from sklearn.model_selection import cross_val_score
 
 from smac.configspace import ConfigurationSpace
-from smac.epm.uncorrelated_mo_rf_with_instances import (
-    UncorrelatedMultiObjectiveRandomForestWithInstances,
-)
+from smac.epm.random_forest.rf_mo import MultiObjectiveRandomForest
 from smac.facade.smac_ac_facade import SMAC4AC
 
 # EIPS related
@@ -104,7 +102,7 @@ def svm_from_cfg(cfg):
     model_kwargs = {"target_names": ["loss", "time"], "model_kwargs": {"seed": 1}}
     smac = SMAC4AC(
         scenario=scenario,
-        model=UncorrelatedMultiObjectiveRandomForestWithInstances,
+        model=MultiObjectiveRandomForest,
         rng=np.random.RandomState(42),
         model_kwargs=model_kwargs,
         tae_runner=svm_from_cfg,

diff --git a/examples/python/plot_synthetic_function_boing.py b/examples/python/plot_synthetic_function_boing.py
@@ -0,0 +1,74 @@
+"""
+Synthetic Function with BOinG as optimizer
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+An example of applying SMAC with BO inside Grove (BOinG) to optimize a
+synthetic function (2d rosenbrock function).
+
+BOinG optimizer requires a SMAC4BOING wrapper to optimize the target algorithm. It is a two stage BO algorithm.
+In the first stage, BOinG constructs an RF to capture the global loss landscape. Then in the second stage, it only
+optimizes inside a subregion near the candidate suggested by the RF model with a GP model to focus only on the most
+promising region.
+"""
+
+import logging
+
+import numpy as np
+from ConfigSpace import ConfigurationSpace
+from ConfigSpace.hyperparameters import UniformFloatHyperparameter
+
+from smac.facade.smac_boing_facade import SMAC4BOING
+
+# Import SMAC-utilities
+from smac.scenario.scenario import Scenario
+
+
+def rosenbrock_2d(x):
+    """The 2 dimensional Rosenbrock function as a toy model
+    The Rosenbrock function is well know in the optimization community and
+    often serves as a toy problem. It can be defined for arbitrary
+    dimensions. The minimium is always at x_i = 1 with a function value of
+    zero. All input parameters are continuous. The search domain for
+    all x's is the interval [-5, 10].
+    """
+    x1 = x["x0"]
+    x2 = x["x1"]
+
+    val = 100.0 * (x2 - x1**2.0) ** 2.0 + (1 - x1) ** 2.0
+    return val
+
+
+if __name__ == "__main__":
+    logging.basicConfig(level=logging.INFO)  # logging.DEBUG for debug output
+
+    # Build Configuration Space which defines all parameters and their ranges
+    cs = ConfigurationSpace()
+    x0 = UniformFloatHyperparameter("x0", -5, 10, default_value=-3)
+    x1 = UniformFloatHyperparameter("x1", -5, 10, default_value=-4)
+    cs.add_hyperparameters([x0, x1])
+    # Scenario object
+    scenario = Scenario(
+        {
+            "run_obj": "quality",  # we optimize quality (alternatively runtime)
+            "runcount-limit": 20,
+            # max. number of function evaluations; for this example set to a low number
+            "cs": cs,  # configuration space
+            "deterministic": "true",
+        }
+    )
+
+    # Example call of the function
+    # It returns: Status, Cost, Runtime, Additional Infos
+    def_value = rosenbrock_2d(cs.get_default_configuration())
+    print("Default Value: %.2f" % def_value)
+
+    # Optimize, using a SMAC-object
+    print("Optimizing! Depending on your machine, this might take a few minutes.")
+
+    smac = SMAC4BOING(
+        scenario=scenario,
+        rng=np.random.RandomState(42),
+        tae_runner=rosenbrock_2d,
+    )
+
+    smac.optimize()
diff --git a/examples/python/plot_synthetic_function_parallel.py b/examples/python/plot_synthetic_function_parallel.py
@@ -0,0 +1,109 @@
+"""
+Synthetic Function with few Hyperparameters
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+An example of applying SMAC to optimize a synthetic function (2d rosenbrock function).
+
+We use the pSMAC [1]_ facade to demonstrate the parallelization of SMAC.
+Other than that, we use a :term:`Gaussian Process<GP>` to optimize our black-box
+function.
+
+
+.. [1] Ramage, S. E. A. (2015). Advances in meta-algorithmic software libraries for
+    distributed automated algorithm configuration (T). University of British
+    Columbia. Retrieved from
+    https://open.library.ubc.ca/collections/ubctheses/24/items/1.0167184.
+"""
+import importlib
+
+import logging
+
+logging.basicConfig(level=logging.INFO)
+
+import numpy as np
+from ConfigSpace.hyperparameters import UniformFloatHyperparameter
+
+# Import ConfigSpace and different types of parameters
+from smac.configspace import ConfigurationSpace
+from smac.facade.psmac_facade import PSMAC
+from smac.facade.smac_bb_facade import SMAC4BB
+import smac
+
+importlib.reload(smac.facade.psmac_facade)
+from smac.facade.psmac_facade import PSMAC
+
+from smac.optimizer.acquisition import EI
+
+# Import SMAC-utilities
+from smac.scenario.scenario import Scenario
+
+__copyright__ = "Copyright 2021, AutoML.org Freiburg-Hannover"
+__license__ = "3-clause BSD"
+
+
+def rosenbrock_2d(x):
+    """The 2 dimensional Rosenbrock function as a toy model
+    The Rosenbrock function is well know in the optimization community and
+    often serves as a toy problem. It can be defined for arbitrary
+    dimensions. The minimium is always at x_i = 1 with a function value of
+    zero. All input parameters are continuous. The search domain for
+    all x's is the interval [-5, 10].
+    """
+
+    x1 = x["x0"]
+    x2 = x["x1"]
+
+    val = 100.0 * (x2 - x1**2.0) ** 2.0 + (1 - x1) ** 2.0
+    return val
+
+
+if __name__ == "__main__":
+    # Build Configuration Space which defines all parameters and their ranges
+    cs = ConfigurationSpace()
+    x0 = UniformFloatHyperparameter("x0", -5, 10, default_value=-3)
+    x1 = UniformFloatHyperparameter("x1", -5, 10, default_value=-4)
+    cs.add_hyperparameters([x0, x1])
+
+    # Scenario object
+    scenario = Scenario(
+        {
+            "run_obj": "quality",  # we optimize quality (alternatively runtime)
+            "runcount-limit": 20,  # max. number of function evaluations PER WORKER
+            "cs": cs,  # configuration space
+            "deterministic": True,
+        }
+    )
+
+    # Use 'gp' or 'gp_mcmc' here
+    model_type = "gp"
+
+    # Example call of the function
+    # It returns: Status, Cost, Runtime, Additional Infos
+    def_value = rosenbrock_2d(cs.get_default_configuration())
+    print("Default Value: %.2f" % def_value)
+
+    # Optimize, using a SMAC-object
+    print("Optimizing! Depending on your machine, this might take a few minutes.")
+    smac = PSMAC(
+        scenario=scenario,
+        facade_class=SMAC4BB,
+        model_type=model_type,
+        rng=np.random.RandomState(42),
+        acquisition_function=EI,  # or others like PI, LCB as acquisition functions
+        tae_runner=rosenbrock_2d,
+        n_workers=2,  # 2 parallel workers
+    )
+
+    incumbent = smac.optimize()
+    # Get trajectory of optimization (incumbent over time)
+    trajectory_json = smac.get_trajectory()  # trajectory in json format
+
+    # Plot trajectory: cost of incumbent against number of evaluations
+    # import matplotlib.pyplot as plt
+    # X = [t["evaluations"] for t in trajectory_json]
+    # Y = [t["cost"] for t in trajectory_json]
+    # plt.plot(X, Y)
+    # plt.yscale("log")
+    # plt.xlabel("Number of Evaluations")
+    # plt.ylabel("Cost of Incumbent")
+    # plt.show()
diff --git a/examples/python/plot_synthetic_function_turbo.py b/examples/python/plot_synthetic_function_turbo.py
@@ -0,0 +1,80 @@
+"""
+Synthetic Function with TuRBO as optimizer
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+An example of applying SMAC with trust region BO (TuRBO) to optimize a
+synthetic function (2d rosenbrock function).
+
+Eriksson et al. Scalable Global Optimization via Local {Bayesian} Optimization,
+http://papers.nips.cc/paper/8788-scalable-global-optimization-via-local-bayesian-optimization.pdf
+
+TurBO gradually shrinks its search space to the vicinity of the optimum configuration that is ever optimized.
+TuRBO optimizer requires EPMChooserTurBO to suggest the next configuration. Currently, it only supports pure numerical
+hyperparameters.
+"""
+
+import logging
+
+import numpy as np
+from ConfigSpace.hyperparameters import UniformFloatHyperparameter
+
+# Import ConfigSpace and different types of parameters
+from smac.configspace import ConfigurationSpace
+from smac.facade.smac_bb_facade import SMAC4BB
+from smac.optimizer.configuration_chooser.turbo_chooser import TurBOChooser
+
+# Import SMAC-utilities
+from smac.scenario.scenario import Scenario
+
+
+def rosenbrock_2d(x):
+    """The 2 dimensional Rosenbrock function as a toy model
+    The Rosenbrock function is well know in the optimization community and
+    often serves as a toy problem. It can be defined for arbitrary
+    dimensions. The minimium is always at x_i = 1 with a function value of
+    zero. All input parameters are continuous. The search domain for
+    all x's is the interval [-5, 10].
+    """
+    x1 = x["x0"]
+    x2 = x["x1"]
+
+    val = 100.0 * (x2 - x1**2.0) ** 2.0 + (1 - x1) ** 2.0
+    return val
+
+
+if __name__ == "__main__":
+    logging.basicConfig(level=logging.INFO)  # logging.DEBUG for debug output
+
+    # Build Configuration Space which defines all parameters and their ranges
+    cs = ConfigurationSpace()
+    x0 = UniformFloatHyperparameter("x0", -5, 10, default_value=-3)
+    x1 = UniformFloatHyperparameter("x1", -5, 10, default_value=-4)
+    cs.add_hyperparameters([x0, x1])
+
+    # Scenario object
+    scenario = Scenario(
+        {
+            "run_obj": "quality",  # we optimize quality (alternatively runtime)
+            "runcount-limit": 100,
+            "cs": cs,  # configuration space
+            "deterministic": "true",
+        }
+    )
+
+    # Example call of the function
+    # It returns: Status, Cost, Runtime, Additional Infos
+    def_value = rosenbrock_2d(cs.get_default_configuration())
+    print("Default Value: %.2f" % def_value)
+
+    # Optimize, using a SMAC-object
+    print("Optimizing! Depending on your machine, this might take a few minutes.")
+    smac = SMAC4BB(
+        scenario=scenario,
+        rng=np.random.RandomState(42),
+        model_type="gp",
+        smbo_kwargs={"epm_chooser": TurBOChooser},
+        initial_design_kwargs={"init_budget": 0},
+        tae_runner=rosenbrock_2d,
+    )
+
+    smac.optimize()