examples/hposuite_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notebook with Minimal example to run hposuite"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# It is assumed that the datadir is hposuite/data\n",
    "\n",
    "from lib.benchmarks import BENCHMARKS\n",
    "from lib.optimizers import OPTIMIZERS\n",
    "\n",
    "print(BENCHMARKS)\n",
    "print(OPTIMIZERS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running Random Search on a Functional Benchmark (eg: ACKLEY)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "func_benchmark = BENCHMARKS[\"Ex_Functional_Bench\"]\n",
    "rs = OPTIMIZERS[\"RandomSearch\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hposuite import create_study\n",
    "\n",
    "study = create_study(\n",
    "    name=\"hposuite_demo_rs_func\",\n",
    "    output_dir=\"../../hposuite-output\",\n",
    "    optimizers=rs,\n",
    "    benchmarks=func_benchmark,\n",
    "    num_seeds=5,\n",
    "    budget=100,\n",
    ")\n",
    "\n",
    "study.optimize(overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running a BlackBox Optimizer on a Synthetic Benchmark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SMAC provides the BlackBoxFacade which is used as an example of a BlackBox Optimizer. \\\n",
    "Multi-fidelity Hartmann 3D with the correlation \"good\" serves as an example of a Synethetic Multi-fidelity Benchmark."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "bb_optimizer = OPTIMIZERS[\"Ex_Blackbox_Opt\"]\n",
    "mf_syn_benchmark = BENCHMARKS[\"mfh3_good\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hposuite import create_study\n",
    "\n",
    "study = create_study(\n",
    "    name=\"hposuite_demo_bb_mfsyn\",\n",
    "    output_dir=\"../../hposuite-output\",\n",
    "    optimizers=bb_optimizer,\n",
    "    benchmarks=mf_syn_benchmark,\n",
    "    num_seeds=5,\n",
    "    budget=20,\n",
    ")\n",
    "\n",
    "study.optimize(overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running a MultiFidelity Optimizer on a Surrogate Benchmark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use a Multi-fidelity Optimizer like Hyperband from SMAC on the PD1 Cifar 100 Wide Resnet 2048 Benchmark which is a Surrogate Benchmark provided by the MF-Prior-Bench Package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "sg_benchmark = BENCHMARKS[\"Ex_Surrogate_Bench\"]\n",
    "mf_optimizer = OPTIMIZERS[\"Ex_Multifidelity_Opt\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hposuite import create_study\n",
    "\n",
    "study = create_study(\n",
    "    name=\"hposuite_demo_sg_mf\",\n",
    "    output_dir=\"../../hposuite-output\",\n",
    "    optimizers=mf_optimizer,\n",
    "    benchmarks=sg_benchmark,\n",
    "    num_seeds=5,\n",
    "    budget=100,\n",
    ")\n",
    "\n",
    "study.optimize(overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running a Multi-objective Optimizer on a Benchmark with Multiple Objectives"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Optuna's NSGA2 Sampler is used here as an example of a Multi-Objective Optimizer. \\\n",
    "PD1 Cifar 100 Wide Resnet 2048 from MF-Prior-Bench is the Benchmark used with \"valid_error_rate\" and \"train_cost\" as the objectives."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "mo_optimizer = OPTIMIZERS[\"Ex_MO_Opt\"]\n",
    "mo_benchmark = BENCHMARKS[\"Ex_Surrogate_Bench\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hposuite import create_study\n",
    "\n",
    "study = create_study(\n",
    "    name=\"hposuite_demo_sg_mf\",\n",
    "    output_dir=\"../../hposuite-output\",\n",
    "    optimizers=mo_optimizer,\n",
    "    benchmarks=(\n",
    "        mo_benchmark,\n",
    "        {\n",
    "            \"objectives\": [\"valid_error_rate\", \"train_cost\"]\n",
    "        }\n",
    "    ),\n",
    "    num_seeds=5,\n",
    "    budget=100,\n",
    ")\n",
    "\n",
    "study.optimize(overwrite=True)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "hposuite_env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}