Merge pull request #354 from automl/development

Prepare release 0.8.0

.travis.yml

@@ -4,11 +4,9 @@ matrix:
 
   include:
   - os: linux
-    env: PYTHON_VERSION="3.4" MINICONDA_URL="https://repo.continuum.io/miniconda/Miniconda3-4.3.21-Linux-x86_64.sh"
+    env: PYTHON_VERSION="3.5" COVERAGE="true" MINICONDA_URL="https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh"
   - os: linux
-    env: PYTHON_VERSION="3.5" COVERAGE="true" MINICONDA_URL="https://repo.continuum.io/miniconda/Miniconda3-4.3.21-Linux-x86_64.sh"
-  - os: linux
-    env: PYTHON_VERSION="3.6" COVERAGE="true" MINICONDA_URL="https://repo.continuum.io/miniconda/Miniconda3-4.3.21-Linux-x86_64.sh"
+    env: PYTHON_VERSION="3.6" COVERAGE="true" MINICONDA_URL="https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh"
 
   # Disable OSX building because it takes too long and hinders progress
   # Set language to generic to not break travis-ci
@@ -18,10 +16,6 @@ matrix:
   #- os: osx
   #  sudo: required
   #  language: generic
-  #  env: PYTHON_VERSION="3.4" MINICONDA_URL="https://repo.continuum.io/miniconda/Miniconda3-4.3.21-MacOSX-x86_64.sh"
-  #- os: osx
-  #  sudo: required
-  #  language: generic
   #  env: PYTHON_VERSION="3.5" MINICONDA_URL="https://repo.continuum.io/miniconda/Miniconda3-4.3.21-MacOSX-x86_64.sh"
   #- os: osx
   #  sudo: required
@@ -43,11 +37,8 @@ before_install:
   - bash miniconda.sh -b -p $HOME/miniconda
   - export PATH="$HOME/miniconda/bin:$PATH"
   - if [[ `which conda` ]]; then echo 'Conda installation successful'; else exit 1; fi
-  - conda create -n testenv --yes python=$PYTHON_VERSION pip wheel nose
+  - conda create -n testenv --yes python=$PYTHON_VERSION pip wheel nose gxx_linux-64 gcc_linux-64 swig
   - source activate testenv
-  - conda install --yes gcc swig
-  - echo "Using GCC at "`which gcc`
-  - export CC=`which gcc`
 
 install:
   - pip install pep8 codecov

README.md

@@ -21,10 +21,10 @@ Status for development branch
 
 # OVERVIEW
 
-SMAC is a tool for algorithm configuration 
-to optimize the parameters of arbitrary algorithms across a set of instances.
-This also includes hyperparameter optimization of ML algorithms.
-The main core consists of Bayesian Optimization in combination with a simple racing mechanism to
+SMAC is a tool for algorithm configuration to optimize the parameters of
+arbitrary algorithms across a set of instances. This also includes
+hyperparameter optimization of ML algorithms. The main core consists of
+Bayesian Optimization in combination with a simple racing mechanism to
 efficiently decide which of two configuration performs better.
 
 For a detailed description of its main idea,
@@ -35,22 +35,25 @@ we refer to
     In: Proceedings of the conference on Learning and Intelligent OptimizatioN (LION 5)
 
 
-SMAC v3 is written in python3 and continuously tested with python3.4 and python3.5. 
-Its [Random Forest](https://bitbucket.org/aadfreiburg/random_forest_run) is written in C++.
+SMAC v3 is written in python3 and continuously tested with python3.5 and
+python3.6. Its [Random Forest](https://github.com/automl/random_forest_run)
+is written in C++.
 
 # Installation
 
-Besides the listed requirements (see `requirements.txt`), the random forest used in SMAC3 requires SWIG (>= 3.0).
+Besides the listed requirements (see `requirements.txt`), the random forest
+used in SMAC3 requires SWIG (>= 3.0).
 
 	apt-get install swig 
 
     cat requirements.txt | xargs -n 1 -L 1 pip install
 
     python setup.py install
 
-If you use Anaconda as your Python environment, you have to install two packages before you can install SMAC:
+If you use Anaconda as your Python environment, you have to install three
+packages before you can install SMAC:
 
-	conda install gcc swig
+	conda install gxx_linux-64 gcc_linux-64 swig
 
 # License
 
@@ -68,15 +71,18 @@ If not, see <https://opensource.org/licenses/BSD-3-Clause>.
 # USAGE
 
 The usage of SMAC v3 is mainly the same as provided with [SMAC v2.08](http://www.cs.ubc.ca/labs/beta/Projects/SMAC/v2.08.00/manual.pdf).
-It supports the same parameter configuration space syntax and interface to target algorithms.
-Please note that we do not support the extended parameter configuration syntax introduced in SMACv2.10.
+It supports the same parameter configuration space syntax and interface to
+target algorithms. Please note that we do not support the extended parameter
+configuration syntax introduced in SMACv2.10.
 
 # Examples
 
 See examples/
 
-  * examples/rosenbrock.py - example on how to optimize a Python function (REQUIRES [PYNISHER](https://github.com/sfalkner/pynisher) )
-  * examples/spear_qcp/run.sh - example on how to optimize the SAT solver Spear on a set of SAT formulas
+  * examples/rosenbrock.py - example on how to optimize a Python function
+    (REQUIRES [PYNISHER](https://github.com/sfalkner/pynisher) )
+  * examples/spear_qcp/run.sh - example on how to optimize the SAT solver Spear
+    on a set of SAT formulas
 
 # Contact
 

changelog.md

@@ -1,36 +1,69 @@
+# 0.8
+
+## Major changes
+
+* Upgrade to ConfigSpace (0.4.X), which is not backwards compatible. On the plus
+  side, the ConfigSpace is about 3-10 times faster, depending on the task.
+* FIX #240: improved output directory structure. If the user does not specify
+  an output directory a SMAC experiment will have the following structure:
+  `smac_/run_<run_id>/*.json`. The user can specify a output directory, e.g.
+  `./myExperiment` or `./myExperiment/` which results in
+  `./myExperiment/run_<run_id>/*.json`.
+* Due to changes in AnaConda's compiler setup we drop the unit tests for
+  python3.4.
+
+## Interface changes
+
+* Generalize the interface of the acquisition functions to work with
+  ConfigSpaces's configuration objects instead of numpy arrays.
+* The acquisition function optimizer can now be passed to the SMBO object.
+* A custom SMBO class can now be passed to the SMAC builder object.
+* `run_id` is no longer an argument to the Scenario object, making the interface
+  a bit cleaner.
+
+## Minor changes
+
+* #333 fixes an incompability with `uncorrelated_mo_rf_with_instances`.
+* #323 fixes #324 and #319, which both improve the functioning of the built-in
+  validation tools.
+* #350 fixes random search, which could accidentaly use configurations found my
+  a local acquisition function optimizer.
+* #336 makes validation more flexible.
+
+
 # 0.7.2
 
-* Introduce version upper bound on ConfigSpace dependency (<0.4)
+* Introduce version upper bound on ConfigSpace dependency (<0.4).
 
 # 0.7.1
 
-* FIX #193, restoring the scenario now possible
-* ADD #271 validation
-* FIX #311 abort on first crash
-* FIX #318, ExecuteTARunOld now always returns a StatusType
+* FIX #193, restoring the scenario now possible.
+* ADD #271 validation.
+* FIX #311 abort on first crash.
+* FIX #318, ExecuteTARunOld now always returns a StatusType.
 
 # 0.6
 
 ## Major changes
 
 * MAINT documentation (nearly every part was improved and extended, 
-  including installation, examples, API) 
-* ADD EPILS as mode (modified version of ParamILS)
+  including installation, examples, API).
+* ADD EPILS as mode (modified version of ParamILS).
 * MAINT minimal required versions of configspace, pyrfr, sklearn increased
-  (several issues fixed in new configspace version)
+  (several issues fixed in new configspace version).
 * MAINT for quality scenarios, the user can specify the objective 
   value for crashed runs 
-  (returned NaN and Inf are replaced by value for crashed runs)
+  (returned NaN and Inf are replaced by value for crashed runs).
 
 ## Minor changes
 
-* FIX issue #220, do not store external data in runhistory 
-* MAINT TAEFunc without pynisher possible
-* MAINT intensification: minimal number of required challengers parameterized
-* FIX saving duplicated (capped) runs
-* FIX handling of ordinal parameters
-* MAINT runobj is now mandatory
-* FIX arguments passed to pyrfr
+* FIX issue #220, do not store external data in runhistory.
+* MAINT TAEFunc without pynisher possible.
+* MAINT intensification: minimal number of required challengers parameterized.
+* FIX saving duplicated (capped) runs.
+* FIX handling of ordinal parameters.
+* MAINT runobj is now mandatory.
+* FIX arguments passed to pyrfr.
 
 # 0.5
 

circle.yml

@@ -27,9 +27,9 @@ dependencies:
     - sudo -E apt-get -yq update
     - sudo -E apt-get -yq --no-install-suggests --no-install-recommends --force-yes install dvipng texlive-latex-base texlive-latex-extra
     # Conda installation
-    - wget http://repo.continuum.io/miniconda/Miniconda3-4.3.21-Linux-x86_64.sh -O ~/miniconda.sh
+    - wget "https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh" -O ~/miniconda.sh
     - bash ~/miniconda.sh -b -p $HOME/miniconda
-    - conda create -n testenv --yes python=3.6 pip wheel nose gcc swig
+    - conda create -n testenv --yes python=3.6 pip wheel nose gxx_linux-64 gcc_linux-64 swig
 
   # The --user is needed to let sphinx see the source and the binaries
   # The pipefail is requested to propagate exit code

doc/apidoc/smac.optimizer.ei_optimization.rst

@@ -0,0 +1,8 @@
+smac\.optimizer\.ei\_optimization module
+========================================
+
+.. automodule:: smac.optimizer.ei_optimization
+    :members:
+    :undoc-members:
+    :show-inheritance:
+    :inherited-members:

doc/apidoc/smac.optimizer.rst

@@ -11,7 +11,7 @@ smac\.optimizer package
 .. toctree::
 
    smac.optimizer.acquisition
-   smac.optimizer.local_search
+   smac.optimizer.ei_optimization
    smac.optimizer.objective
    smac.optimizer.pSMAC
    smac.optimizer.smbo

doc/basic_usage.rst

@@ -53,19 +53,6 @@ The **paramfile**-parameter defines the path to the `PCS-file <options.html#pcs>
 which describes the ranges and default values of the tunable parameters.
 Both will interpret paths *from the execution-directory*.
 
-.. _restorestate:
-
-To restore a previous *SMAC*-run, either because it was interrupted or because
-you want to extend its computation- or time-limits, use the
-``--restore_state FOLDER``-option in the commandline. If you want to increase
-computation- or time-limits, change the scenario-file specified with the
-``--scenario SCENARIOFILE``-option (not the one in the folder to be restored).
-Restarting a *SMAC*-run that quit due to budget-exhaustion will do nothing,
-because the budget is still exhausted.
-**Changing any other options than *wallclock_limit*, *runcount_limit* or
-*tuner-timeout* in the scenario-file is NOT intended and will likely lead
-to unexpected behaviour!**
-
 .. note::
 
     Currently, running *SMAC* via the commandline will register the algorithm with a
@@ -77,6 +64,27 @@ to unexpected behaviour!**
     
         Result for SMAC: <STATUS>, <runtime>, <runlength>, <quality>, <seed>, <instance-specifics>
 
+
+.. _restorestate:
+
+Restoring States
+~~~~~~~~~~~~~~~~
+If a *SMAC*-run was interrupted or you want to extend its computation- or
+time-limits, it can be restored and continued.
+To restore or continue a previous *SMAC*-run, use the
+``--restore_state FOLDER``-option in the commandline. If you want to increase
+computation- or time-limits, change the scenario-file specified with the
+``--scenario SCENARIOFILE``-option (**not the one in the folder to be restored**).
+Restarting a *SMAC*-run that quit due to budget-exhaustion will do nothing,
+because the budget is still exhausted.
+**Changing any other options than *output_dir*, *wallclock_limit*, *runcount_limit* or
+*tuner-timeout* in the scenario-file is NOT intended and will likely lead
+to unexpected behaviour!**
+
+For an example of restoring states from within your Python code, there is an
+implementation with the Branin-example in "examples/branin/restore_state.py".
+
+
 .. _inpython:
 
 Usage in Python

doc/faq.rst

@@ -8,26 +8,31 @@ or try to run the installation first.
 
 .. rubric:: pyrfr raises cryptic import errors.
 
-Ensure that the gcc used to compile the pyrfr is the same as used for linking during execution.
-This often happens with Anaconda -- see `Installation <installation.html>`_ for a solution.
+Ensure that the gcc used to compile the pyrfr is the same as used for linking
+during execution. This often happens with Anaconda -- see
+`Installation <installation.html>`_ for a solution.
 
 .. rubric:: My target algorithm is not accepted, when using the scenario-file.
 
-Make sure that your algorithm accepts commandline options as provided by *SMAC*.
-Refer to `commandline execution <basic_usage.html#commandline>`_ for details on how to wrap your algorithm.
+Make sure that your algorithm accepts commandline options as provided by
+*SMAC*. Refer to `commandline execution <basic_usage.html#commandline>`_ for
+details on how to wrap your algorithm.
 
 You can also run SMAC with :code:`--verbose DEBUG` to see how *SMAC* tried to call your algorithm.
 
 .. rubric:: Can I restore SMAC from a previous state?
 
 Use the `restore-option <basic_usage.html#restorestate>`_.
 
-.. rubric:: I discovered a bug/have criticism or ideas on *SMAC*. Where should I report to?
+.. rubric:: I discovered a bug/have criticism or ideas on *SMAC*. Where should
+I report to?
 
-*SMAC* uses the `GitHub issue-tracker <https://github.com/automl/SMAC3/issues>`_ to take care of bugs and questions. If you
-experience problems with *SMAC*, try to provide a full error report with all the
-typical information (OS, version, console-output, minimum working example, ...).
-This makes it a lot easier to reproduce the error and locate the problem.
+*SMAC* uses the
+`GitHub issue-tracker <https://github.com/automl/SMAC3/issues>`_ to take care
+of bugs and questions. If you experience problems with *SMAC*, try to provide
+a full error report with all the typical information (OS, version,
+console-output, minimum working example, ...). This makes it a lot easier to
+reproduce the error and locate the problem.
 
 
 .. rubric:: **Glossary**

doc/index.rst

@@ -53,7 +53,7 @@ Contents:
      |    howpublished={\\url{https://github.com/automl/SMAC3}} 
      | }
 
-SMAC3 is mainly written in Python 3 and continuously tested with Python 3.4-3.6.
+SMAC3 is mainly written in Python 3 and continuously tested with Python 3.5-3.6.
 Its `Random Forest <https://github.com/automl/random_forest_run>`_ is written in
 C++11.
 

doc/installation.rst

@@ -16,17 +16,18 @@ please call:
     sudo apt-get install build-essential swig
 
 If you use Anaconda, you have to install both gcc and SWIG from Anaconda to
-prevent broken links:
+prevent compilation errors:
 
 .. code-block:: bash
 
-    conda install gcc swig
+    conda install gxx_linux-64 gcc_linux-64 swig
 
 .. _installation_pypi:
 
 Installation from pypi
 ----------------------
-To install SMAC3 from pypi, please use the following command on the command line:
+To install SMAC3 from pypi, please use the following command on the command
+line:
 
 .. code-block:: bash
 
@@ -39,7 +40,8 @@ permissions), please add the option :code:`--user` or create a virtualenv.
 
 Manual Installation
 -------------------
-To install SMAC3 from command line, please type the following commands on the command line
+To install SMAC3 from command line, please type the following commands on the
+command line:
 
 .. code-block:: bash