notebooks/using-mlflow/train-with-mlflow/xgboost_nested_runs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hyper-parameter optimization using HyperOpt and nested runs in MLflow\n",
    "\n",
    "This notebook demonstrates how to use MLflow for tracking experiment using MLflow in Azure ML when you want to do hyper-parameter optimization using child runs. In MLflow a run can start child runs which will get logged inside of it. This is useful when doing hyper-parameters cause you can get one dedicated run for each trial, while keeping all the runs controlled by a parent run which orchestrates the parameters search. \n",
    "\n",
    "In this example we will show also how you can get artifacts from the child run and log them inside of the parent. We are going to use the library `HyperOpt` for doing the optimization.\n",
    "\n",
    "> **Important:** You may also want to explore using the parameter sweep component from Azure ML for doing hyper-parameter optimization in job-based runs. Check [CLI (v2) sweep job](https://docs.microsoft.com/en-us/azure/machine-learning/reference-yaml-job-sweephttps://docs.microsoft.com/en-us/azure/machine-learning/reference-yaml-job-sweep) for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ensure you have the dependencies for this notebook\n",
    "%pip install -r xgboost_nested_runs.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "\n",
    "warnings.simplefilter(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuring the experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get started. It's always a good idea to start by configuring the name of the experiment we are working with in MLflow. Experiments allows you to organize runs in a comprehensive way so you can compare different experiment's runs with different parameters and configuration. MLflow configures the default experiment named \"Default\" but you can change this name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Experiment: artifact_location='', experiment_id='368f3886-e4e3-449a-bd1e-593c5b0e01ff', lifecycle_stage='active', name='heart-condition-classifier', tags={}>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import mlflow\n",
    "\n",
    "mlflow.set_experiment(experiment_name=\"heart-condition-classifier\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_url = \"http://storage.googleapis.com/download.tensorflow.org/data/heart.csv\"\n",
    "df = pd.read_csv(file_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trestbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thalach</th>\n",
       "      <th>exang</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slope</th>\n",
       "      <th>ca</th>\n",
       "      <th>thal</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>145</td>\n",
       "      <td>233</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>150</td>\n",
       "      <td>0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>fixed</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>67</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>160</td>\n",
       "      <td>286</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>108</td>\n",
       "      <td>1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>normal</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>67</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>120</td>\n",
       "      <td>229</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>129</td>\n",
       "      <td>1</td>\n",
       "      <td>2.6</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>reversible</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>130</td>\n",
       "      <td>250</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>187</td>\n",
       "      <td>0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>normal</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>130</td>\n",
       "      <td>204</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>172</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>normal</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>298</th>\n",
       "      <td>52</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>118</td>\n",
       "      <td>186</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>190</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>fixed</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299</th>\n",
       "      <td>43</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>132</td>\n",
       "      <td>341</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>136</td>\n",
       "      <td>1</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>reversible</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>300</th>\n",
       "      <td>65</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>135</td>\n",
       "      <td>254</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>127</td>\n",
       "      <td>0</td>\n",
       "      <td>2.8</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>reversible</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>301</th>\n",
       "      <td>48</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>130</td>\n",
       "      <td>256</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>150</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>reversible</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>302</th>\n",
       "      <td>63</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>150</td>\n",
       "      <td>407</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>154</td>\n",
       "      <td>0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>reversible</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>303 rows \u00d7 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     age  sex  cp  trestbps  chol  fbs  restecg  thalach  exang  oldpeak  \\\n",
       "0     63    1   1       145   233    1        2      150      0      2.3   \n",
       "1     67    1   4       160   286    0        2      108      1      1.5   \n",
       "2     67    1   4       120   229    0        2      129      1      2.6   \n",
       "3     37    1   3       130   250    0        0      187      0      3.5   \n",
       "4     41    0   2       130   204    0        2      172      0      1.4   \n",
       "..   ...  ...  ..       ...   ...  ...      ...      ...    ...      ...   \n",
       "298   52    1   1       118   186    0        2      190      0      0.0   \n",
       "299   43    0   4       132   341    1        2      136      1      3.0   \n",
       "300   65    1   4       135   254    0        2      127      0      2.8   \n",
       "301   48    1   4       130   256    1        2      150      1      0.0   \n",
       "302   63    0   4       150   407    0        2      154      0      4.0   \n",
       "\n",
       "     slope  ca        thal  target  \n",
       "0        3   0       fixed       0  \n",
       "1        2   3      normal       1  \n",
       "2        2   2  reversible       0  \n",
       "3        3   0      normal       0  \n",
       "4        1   0      normal       0  \n",
       "..     ...  ..         ...     ...  \n",
       "298      2   0       fixed       0  \n",
       "299      2   0  reversible       1  \n",
       "300      2   1  reversible       1  \n",
       "301      1   2  reversible       1  \n",
       "302      2   3  reversible       1  \n",
       "\n",
       "[303 rows x 14 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, some of the variables are categorical. To make it simpler for our model to handle these values, let's use their encoded values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"thal\"] = df[\"thal\"].astype(\"category\").cat.codes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The encoded values looks then as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2, 3, 4, 0, 1], dtype=int8)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"thal\"].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's split our dataset in train and test, so we can assess the performance of the model without overfitting the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    df.drop(\"target\", axis=1), df[\"target\"], test_size=0.3\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining the search space\n",
    "\n",
    "Since we are going to do hyper-parameter optimization, we need to define the search space we are going to use for each parameter. In this simple example, we are optimizing the following parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hyperopt import hp\n",
    "\n",
    "space = {\n",
    "    \"learning_rate\": hp.uniform(\"learning_rate\", 1e-5, 1e-1),\n",
    "    \"max_leaves\": hp.choice(\"max_leaves \", options=[2, 5, 9, 10, 20, 30]),\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training a model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to place all the training code inside of a function so we can use for searching the best parameters configuration. HyperOpt requires us to have **the first positional parameter of this function to receive the parameters of each trial**. Here we are placing them in an argument called `params` which is a dictionary. The rest of the parameters needs to be named parameters. You need to comply with this convention.\n",
    "\n",
    "The function needs to return either a float value which correspond to the loss of the trial, or a dictionary which specific keys. Read the documentation of HyperOpt for other options and details. We are using the dictionary option here cause we want to return some extra information about each trial which is the run ID in MLflow corresponding to the trial. Pay attention how this is indicated in a key `attachments`. This will be handy later to download assets for each run.\n",
    "\n",
    "It's worth noticing how the MLflow run is started with the parameter `nested=True`. This indicates to MLflow that the current run is a child run of a higher level one. The parent run controls the parameter optimization while each child focus on training one particular model. Also, notice how we are using `autolog` to log metrics, parameters and models automatically for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xgboost import XGBClassifier\n",
    "from sklearn.metrics import accuracy_score, recall_score, roc_curve, auc\n",
    "from typing import Any, Dict\n",
    "from hyperopt import STATUS_OK\n",
    "\n",
    "\n",
    "def train(\n",
    "    params: Dict[str, Any], X_train: pd.DataFrame, X_test, y_train: pd.DataFrame, y_test\n",
    ") -> Dict[str, Any]:\n",
    "    run_name = \"_\".join(params.keys())\n",
    "    with mlflow.start_run(nested=True, run_name=f\"hyperopt_{run_name}\") as child_run:\n",
    "        # Using autolog for xgboost model\n",
    "        mlflow.xgboost.autolog(silent=True)\n",
    "\n",
    "        # Using autolog for the metrics calculated with scikit-learn\n",
    "        mlflow.sklearn.autolog(silent=True)\n",
    "\n",
    "        model = XGBClassifier(use_label_encoder=False, eval_metric=\"logloss\", **params)\n",
    "        model.fit(X_train, y_train, eval_set=[(X_test, y_test)], verbose=False)\n",
    "        y_pred = model.predict(X_test)\n",
    "\n",
    "        # Calculate some metrics\n",
    "        accuracy = accuracy_score(y_test, y_pred)\n",
    "        recall = recall_score(y_test, y_pred)\n",
    "        fpr, tpr, thresholds = roc_curve(y_test, y_pred, pos_label=2)\n",
    "        auc_metric = auc(fpr, tpr)\n",
    "\n",
    "        # Return a dictionary HyperOpt can use to better inform the search\n",
    "        # We are also returning the run_id of each iteration so we can use it\n",
    "        # later to access the best run from Mlflow.\n",
    "        return {\n",
    "            \"status\": STATUS_OK,\n",
    "            \"loss\": -recall,\n",
    "            \"attachments\": {\"run_id\": child_run.info.run_id},\n",
    "        }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a search function\n",
    "\n",
    "We are going to create a search function for doing the hyper-parameter optimization. This is not required -  you can place all the code directly on notebook's cells but it is a good choice. The function will:\n",
    "\n",
    "1. Start a new MLflow run to do the optimization this parent run is the one in charge of doing the overall optimization. Notice that if you are running inside of an AML job, you are should not start the run manually cause it is already done for you.\n",
    "2. Creates a `fmin` object which will do the optimization using HyperOpt.\n",
    "3. Gets the best trial and best run ID from MLflow.\n",
    "4. Downloads the best model from the child run which was better and register it in the parent run.\n",
    "5. Logs the best child run in the parent run along with the best hyper-parameters configuration we found.\n",
    "6. Logs some extra artifacts (see details later)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import hyperopt.plotting\n",
    "from hyperopt import fmin, tpe, rand, Trials\n",
    "from functools import partial\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "\n",
    "def search(\n",
    "    space: Dict[str, Any], X_train: pd.DataFrame, X_test, y_train: pd.DataFrame, y_test\n",
    "):\n",
    "    with mlflow.start_run() as run:\n",
    "        print(\"Working in MLflow run:\", run.info.run_id)\n",
    "        trials = Trials()\n",
    "        best_params = fmin(\n",
    "            fn=partial(\n",
    "                train, X_train=X_train, X_test=X_test, y_train=y_train, y_test=y_test\n",
    "            ),\n",
    "            space=space,\n",
    "            algo=tpe.suggest,\n",
    "            max_evals=5,\n",
    "            trials=trials,\n",
    "        )\n",
    "        best_run_id = trials.trial_attachments(trials.best_trial)[\"run_id\"]\n",
    "        best_run = mlflow.get_run(best_run_id)\n",
    "\n",
    "        client = mlflow.tracking.MlflowClient()\n",
    "        best_model = client.download_artifacts(best_run_id, path=\"model\")\n",
    "\n",
    "        mlflow.log_param(\"best_run_id\", best_run_id)\n",
    "        mlflow.log_params({f\"best_{p}\": v for p, v in best_params.items()})\n",
    "        mlflow.log_metric(\"best_loss\", trials.best_trial[\"result\"][\"loss\"])\n",
    "        mlflow.log_artifacts(local_dir=best_model, artifact_path=\"model\")\n",
    "\n",
    "        fig = plt.figure()\n",
    "        ax = fig.add_subplot()\n",
    "        hyperopt.plotting.main_plot_histogram(trials, do_show=False)\n",
    "        mlflow.log_figure(fig, \"loss_histogram.png\")\n",
    "\n",
    "        return trials, best_params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some things to notice about `fmin`:\n",
    "- `fmin` is used to minimize the loss according to the objective indicated in the function `train`. If you pay closer look at how it is indicated, we wrapped the function with `partial`. This is because the function receives extra parameters we are not looking to optimize, but they are directly passed to the function. We are passing here the train and test splits for the features and labels.\n",
    "- `algo` is indicated with `tpe.suggest`, meaning that we are relying on HyperOpt to guess which is the best algorithm to use based on the space we are using.\n",
    "- `max_evals` indicates how many trials will be executed. We are not using early termination in this example, but you are welcome to explore it.\n",
    "- `trials` is passed to the `fmin` function. This is not required but when the argument is present, all the debug information about the trials is stored there. It's a best practice to always indicate it.\n",
    "\n",
    "Some things to notice in the `search` function:\n",
    "- The function `fmin` returns the best parameters combination, but it doesn't provide information about which is the best run. For that, we use `trials.best_trial`. We also added `run_id` as an attachment, so we can use it to get the best run inside of MLflow. This is convenient because we don't have to query all the child runs to guess which is the best one.\n",
    "- We are downloading the best model so we can log it in the parent run. This is done using the function `client.download_artifacts` for the child run and then `mlflow.log_artifacts` for the parent run. So the best model is logged twice, inside of the parent run and also inside of the child run that generated it.\n",
    "- For further reference, the best run ID is also logged as a parameters with the method `mlflow.log_param`.\n",
    "- HyperOpt also has some features for plotting the results. We are logging this plots using `mlflow.log_figure` which will store these plots as `png` files automatically for us."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run the search:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Working in MLflow run: 7eca10a6-d166-4714-a251-10f1585eb101\n",
      "100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 5/5 [01:05<00:00, 13.15s/trial, best loss: -0.6]\n",
      "Showing Histogram of 5 jobs\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAW70lEQVR4nO3de9RddX3n8ffHJKAoiphnBAIhInirFbApIrbKiI54g5bBJYzFy9KV6pJV7dh2QB28VGe0WnW8MlQBtShUBRsFahFhwGlFAuWiMNTIoCQihEu4VLwEv/PH2WnPHM7z5OR5nn1Onuz3a62zcvbev7P395eTnM/Zl/PbqSokSd31kEkXIEmaLINAkjrOIJCkjjMIJKnjDAJJ6jiDQJI6ziCQtlKSk5P810nXIc0Xg0DbpCQ3JXneBLZ7epL3DMxbkaSSLAaoqtdX1Z+PsK6J9EHaWgaBtABtDiVpPhgEWlCS7JjkI0l+0jw+kmTHZtnSJF9PsjHJnUkuTfKQZtl/SbI+yb1Jbkhy2Bxq+Ne9hum2meTzwHLga0nuS/JnTfsjkny/aX9xkif3rffpSf6pqfFLSc7q286hSdY1/fgpcFqSRzfb3pDkrub5nn3ruzjJe5L8Q1PD15I8JskZSe5JcnmSFbP9e9D2wyDQQvM24GDgAGB/4CDg7c2ytwDrgCngscBbgUryROB44LeramfgBcBN81TP0G1W1XHAj4GXVtUjquovkjwB+CLw5qb9efSCYockOwDnAKcDuzbtfn9gW7s1y/YGVtH7/3taM70cuB/4+MBrjgGOA5YBjwf+sXnNrsD1wDvm4y9BC5tBoIXmFcC7q+q2qtoAvIveBx3Ar4Ddgb2r6ldVdWn1BtN6ANgReEqSJVV1U1X9cIZt/EnzjX1jko3ANTO0nW6bw7wcOLeqLqiqXwEfBB4GHEIv3BYDH23Wczbw3YHX/xp4R1X9oqrur6o7quorVfWzqroXeC/wnIHXnFZVP6yqu4HzgR9W1TerahPwJeDAGfqmjjAItNDsAfyob/pHzTyADwBrgb9PcmOSEwCqai29b+HvBG5LcmaSPZjeB6tql80P4GkztB26zVFqr6pfAzfT+7a+B7B+IERuHnj9hqr6+eaJJDsl+Z9JfpTkHuASYJcki/pec2vf8/uHTD9ihnrVEQaBFpqf0DsUstnyZh5VdW9VvaWq9gGOAP7z5nMBVfWFqvqd5rUFvH8+iplpm812pq09SYC9gPXALcCyZt5mew1ubmD6LcATgWdU1SOBZ29e9Wz7o24yCLQtW5LkoX2PxfSOnb89yVSSpcBJwF8DJHlJkn2bD9O76R0S+nWSJyZ5bnNS+ef0vgn/ej4KnG6bzeJbgX36mv8N8OIkhyVZQu+D/BfAP9A7dv8AcHySxUmOpHf+YyY7N33ZmGRXPN6vWTIItC07j94H3ebHO4H3AGvoHbe/FriymQewH/BN4D56H6yfrKqL6J0feB9wO/BT4N8BJ85TjdNtE+C/0wutjUn+pKpuAP4A+FhTy0vpnUz+ZVX9EjgKeC2wsWn3dXpBMZ2P0DvHcDvwHeDv5qlP6ph4Yxpp25TkMuDkqjpt0rVo++YegbSNSPKcJLs1h4ZeRe8ktd/y1Tp/nShtO55I7zzCw4EbgaOr6pbJlqQu8NCQJHWch4YkqeMW3KGhpUuX1ooVKyZdhiQtKFdcccXtVTU1bNmCC4IVK1awZs2aSZchSQtKkh9Nt8xDQ5LUcQaBJHWcQSBJHWcQSFLHGQSS1HEGgSR1XGtB0Awb/N0kVzf3aH3XkDY7NvdlXZvkMu+fKknj1+YewS+A51bV/vTuL3t4koMH2rwWuKuq9gU+zDzdLESSNLrWgqB67msmlzSPwYGNjgQ+2zz/MnDYwB2aJEkta/WXxc29U68A9gU+UVWXDTRZRnNf1qralORu4DH0brTRv55VwCqA5cuXz7qeFSecO+vXztVN73vxxLYtSTNp9WRxVT1QVQcAewIHJXnqLNdzSlWtrKqVU1NDh8qQJM3SWK4aqqqNwEXA4QOL1tPcoLu5H+2jgDvGUZMkqafNq4amkuzSPH8Y8Hzg/ww0Ww28qnl+NPCt8gYJkjRWbZ4j2B34bHOe4CHA31TV15O8G1hTVauBzwCfT7IWuBM4psV6JElDtBYEVXUNcOCQ+Sf1Pf858LK2apAkbZm/LJakjjMIJKnjDAJJ6jiDQJI6ziCQpI4zCCSp4wwCSeo4g0CSOs4gkKSOMwgkqeMMAknqOINAkjrOIJCkjjMIJKnjDAJJ6jiDQJI6ziCQpI4zCCSp4wwCSeo4g0CSOs4gkKSOMwgkqeMMAknqOINAkjrOIJCkjmstCJLsleSiJNcl+X6SNw1pc2iSu5Nc1TxOaqseSdJwi1tc9ybgLVV1ZZKdgSuSXFBV1w20u7SqXtJiHZKkGbS2R1BVt1TVlc3ze4HrgWVtbU+SNDtjOUeQZAVwIHDZkMXPTHJ1kvOT/MY0r1+VZE2SNRs2bGizVEnqnNaDIMkjgK8Ab66qewYWXwnsXVX7Ax8DvjpsHVV1SlWtrKqVU1NTrdYrSV3TahAkWUIvBM6oqrMHl1fVPVV1X/P8PGBJkqVt1iRJ+v+1edVQgM8A11fVh6Zps1vTjiQHNfXc0VZNkqQHa/OqoWcBxwHXJrmqmfdWYDlAVZ0MHA28Ickm4H7gmKqqFmuSJA1oLQiq6ttAttDm48DH26pBkrRl/rJYkjrOIJCkjjMIJKnjDAJJ6jiDQJI6ziCQpI4zCCSp4wwCSeo4g0CSOs4gkKSOMwgkqeMMAknqOINAkjrOIJCkjjMIJKnjDAJJ6jiDQJI6ziCQpI4zCCSp4wwCSeo4g0CSOs4gkKSOMwgkqeMMAknqOINAkjrOIJCkjmstCJLsleSiJNcl+X6SNw1pkyQfTbI2yTVJnt5WPZKk4Ra3uO5NwFuq6sokOwNXJLmgqq7ra/NCYL/m8QzgU82fkqQxaW2PoKpuqaorm+f3AtcDywaaHQl8rnq+A+ySZPe2apIkPdhYzhEkWQEcCFw2sGgZcHPf9DoeHBYkWZVkTZI1GzZsaK1OSeqi1oMgySOArwBvrqp7ZrOOqjqlqlZW1cqpqan5LVCSOq7VIEiyhF4InFFVZw9psh7Yq296z2aeJGlM2rxqKMBngOur6kPTNFsNvLK5euhg4O6quqWtmiRJD9bmVUPPAo4Drk1yVTPvrcBygKo6GTgPeBGwFvgZ8JoW65EkDdFaEFTVt4FsoU0Bb2yrBknSlvnLYknquC0GQZIrkrwxyaPHUZAkabxG2SN4ObAHcHmSM5O8oDkRLEnaDmwxCKpqbVW9DXgC8AXgVOBHSd6VZNe2C5QktWukcwRJngb8JfABer8LeBlwD/Ct9kqTJI3DFq8aSnIFsJHebwJOqKpfNIsuS/KsFmuTJI3BKJePvqyqbhy2oKqOmud6JEljNsqhodcl2WXzRJJHJ3lPeyVJksZplCB4YVVt3DxRVXfR+zWwJGk7MEoQLEqy4+aJJA8DdpyhvSRpARnlHMEZwIVJTmumXwN8tr2SJEnjtMUgqKr3J7kGOKyZ9edV9Y12y5IkjctIg85V1fnA+S3XIkmagFHGGjoqyQ+S3J3kniT3JpnVncYkSdueUfYI/gJ4aVVd33YxkqTxG+WqoVsNAUnafo2yR7AmyVnAV4HNw0swzT2IJUkLzChB8Eh6t5H8D33zCjAIJGk7MMrlo95HWJK2Y6NcNfSEJBcm+V4z/bQkb2+/NEnSOIxysvivgBOBXwFU1TXAMW0WJUkan1GCYKeq+u7AvE1tFCNJGr9RguD2JI+nd4KYJEcDt7RalSRpbEa5auiNwCnAk5KsB/4v8AetViVJGptRrhq6EXhekocDD6mqe9svS5I0LqPcs/ikgWkAqurdW3jdqcBLgNuq6qlDlh8K/C29PQyAs7e0TknS/Bvl0NC/9D1/KL0P91GGnDgd+DjwuRnaXFpVLxlhXZKkloxyaOgv+6eTfBDY4v0IquqSJCtmX5okaRxGuWpo0E7AnvO0/WcmuTrJ+Ul+Y7pGSVYlWZNkzYYNG+Zp05IkGO0cwbU0l44Ci4ApYD6O5V8J7F1V9yV5Eb1B7fYb1rCqTqF35RIrV66sYW0kSbMzyjmC/mP4m+gNSz3nH5RV1T19z89L8skkS6vq9rmuW5I0ulGCYPBy0UduvnIIoKrunM2Gk+xGL1QqyUH0DlPdMZt1SZJmb5QguBLYC7gLCLAL8ONmWQH7DHtRki8ChwJLk6wD3gEsAaiqk4GjgTck2QTcDxxTVR72kaQxGyUILgDOqarzAJK8EPi9qvrDmV5UVcduYfnH6V1eKkmaoFGuGjp4cwgAVNX5wCHtlSRJGqdR9gh+0tx/4K+b6VcAP2mvJEnSOI2yR3AsvUtGz6F3e8qpZp4kaTswyi+L7wTelOThVfUvW2ovSVpYRrlV5SFJrqMZXyjJ/kk+2XplkqSxGOXQ0IeBF9Bc419VVwPPbrMoSdL4jDTWUFXdPDDrgRZqkSRNwChXDd2c5BCgkiwB3sRow1BLkhaAUfYIXk/vdpXLgPXAAc20JGk7MOMeQZJFwP+oqleMqR5J0pjNuEdQVQ8AeyfZYUz1SJLGbJRzBDcC/zvJavpuW1lVH2qtKknS2Ey7R5Dk883TI4CvN2137ntIkrYDM+0R/FaSPegNOf2xMdUjSRqzmYLgZOBC4HHAmr75YYb7EEiSFpZpDw1V1Uer6snAaVW1T9/jcVVlCEjSdmKLvyOoqjeMoxBJ0mSMNMSEJGn7ZRBIUscZBJLUcQaBJHWcQSBJHWcQSFLHGQSS1HEGgSR1nEEgSR3XWhAkOTXJbUm+N83yJPlokrVJrkny9LZqkSRNr809gtOBw2dY/kJgv+axCvhUi7VIkqbRWhBU1SXAnTM0ORL4XPV8B9glye5t1SNJGm6UO5S1ZRlwc9/0umbeLYMNk6yit9fA8uXLx1KcNBsrTjh3Itu96X0vnsh2u2hS7zG09z4viJPFVXVKVa2sqpVTU1OTLkeStiuTDIL1wF5903s28yRJYzTJIFgNvLK5euhg4O6qetBhIUlSu1o7R5Dki8ChwNIk64B3AEsAqupk4DzgRcBa4GfAa9qqRZI0vdaCoKqO3cLyAt7Y1vYlSaNZECeLJUntMQgkqeMMAknqOINAkjrOIJCkjjMIJKnjDAJJ6jiDQJI6ziCQpI4zCCSp4wwCSeo4g0CSOs4gkKSOMwgkqeMMAknqOINAkjrOIJCkjjMIJKnjDAJJ6jiDQJI6ziCQpI4zCCSp4wwCSeo4g0CSOs4gkKSOazUIkhye5IYka5OcMGT5q5NsSHJV83hdm/VIkh5scVsrTrII+ATwfGAdcHmS1VV13UDTs6rq+LbqkCTNrM09goOAtVV1Y1X9EjgTOLLF7UmSZqHNIFgG3Nw3va6ZN+g/JrkmyZeT7DVsRUlWJVmTZM2GDRvaqFWSOmvSJ4u/BqyoqqcBFwCfHdaoqk6pqpVVtXJqamqsBUrS9q7NIFgP9H/D37OZ96+q6o6q+kUz+Wngt1qsR5I0RJtBcDmwX5LHJdkBOAZY3d8gye59k0cA17dYjyRpiNauGqqqTUmOB74BLAJOrarvJ3k3sKaqVgN/lOQIYBNwJ/DqtuqRJA3XWhAAVNV5wHkD807qe34icGKbNUiSZjbpk8WSpAkzCCSp4wwCSeo4g0CSOs4gkKSOMwgkqeMMAknqOINAkjrOIJCkjjMIJKnjDAJJ6jiDQJI6ziCQpI4zCCSp4wwCSeo4g0CSOs4gkKSOMwgkqeMMAknqOINAkjrOIJCkjjMIJKnjDAJJ6jiDQJI6ziCQpI4zCCSp41oNgiSHJ7khydokJwxZvmOSs5rllyVZ0WY9kqQHay0IkiwCPgG8EHgKcGySpww0ey1wV1XtC3wYeH9b9UiShmtzj+AgYG1V3VhVvwTOBI4caHMk8Nnm+ZeBw5KkxZokSQMWt7juZcDNfdPrgGdM16aqNiW5G3gMcHt/oySrgFXN5H1JbphlTUsH1z0umdy+zsT6PEGd63Pe370+4/u8tfaebkGbQTBvquoU4JS5rifJmqpaOQ8lLRj2uRvscze01ec2Dw2tB/bqm96zmTe0TZLFwKOAO1qsSZI0oM0guBzYL8njkuwAHAOsHmizGnhV8/xo4FtVVS3WJEka0NqhoeaY//HAN4BFwKlV9f0k7wbWVNVq4DPA55OsBe6kFxZtmvPhpQXIPneDfe6GVvocv4BLUrf5y2JJ6jiDQJI6brsLgiS7JrkgyQ+aPx89TbvlSf4+yfVJrts8vEVzcvuyZtiLs5oT3du0rejzA0muah6r++YfluTKZv63k+w7vupnZx76nCTvTfLPzb+BPxpf9bMz1z73Lf9okvvar3ju5uF9PqMZ5uZ7SU5NsmR81c/OPPR5qz/DtrsgAE4ALqyq/YALm+lhPgd8oKqeTO9X0Lc1898PfLgZ9uIuesNgbOtG7fP9VXVA8ziib/6ngFdU1QHAF4C3t1rt/Jhrn19N79LlJzX/Bs5stdr5Mdc+k2QlMPSDZRs11z6fATwJ+E3gYcDrWq12fsy1z1v/GVZV29UDuAHYvXm+O3DDkDZPAb49ZH7o/WpvcTP9TOAbk+7TfPS5WXbfDK9/RvP8ROC/TbpPY+jzd4F9J92PMfd5EXBR89qhbba1x1z7PNDmj4H3TrpPbfZ5tp9h2+MewWOr6pbm+U+Bxw5p8wRgY5Kzk/xTkg80g+Q9BthYVZuaduvoDYOxrRulzwAPTbImyXeS/F7f/NcB5yVZBxwHvK+9UufNXPv8eODlzbLzk+zXZrHzZK59Ph5Y3beOhWCufQagOSR0HPB37ZQ5r+bS51l9hi2IISYGJfkmsNuQRW/rn6iqSjLs+tjFwO8CBwI/Bs6id6jgb+e30vkzD30G2Luq1ifZB/hWkmur6of0vim9qKouS/KnwIfYBnahW+7zjsDPq2plkqOAU+n9m5iotvoM3A+8DDh0PuudDy2/z5t9Erikqi6dn6rnpsX3+e7Z1LMgg6CqnjfdsiS3Jtm9qm5Jsjv/duy/3zrgqqq6sXnNV4GD6X0Y7JJkcZOow4bFmIh56DNVtb7588YkFwMHJrkH2L+qLmuancU28q2prT4DP6T3b+Dsptk5wGnzWftstdjn+4F9gbXpDfC7U5K11TuOPFEtv88keQcwBfzhfNc+Wy32+SvM4jNsezw01D9sxasY/i3/cnp/WVPN9HOB66p3UO0iesNdzPT6bc0W+5zk0Ul2bJ4vBZ4FXEfvZNKjkjyhafp84PrWK567ufQZ4KvAv2+ePwf45zaLnSez7nNVnVtVu1XViqpaAfxsWwiBEczpfU7yOuAFwLFV9euxVDx3c3mfZ/cZNukTI/P9oHeM7ELgB8A3gV2b+SuBT/e1ez5wDXAtcDqwQzN/H3onEtcCXwJ2nHSf5qPPwCFNX69u/nxt3+t/v2/ZxcA+k+7TGPq8C3BuM/8f6e0VTbxfbfZ5YF0L5WTxXN/nTfT2DK5qHidNuk9j6PNWf4Y5xIQkddz2eGhIkrQVDAJJ6jiDQJI6ziCQpI4zCCSp4wwCaURZICN2SlvLIJCkjjMIpK2Ung80Y9xfm+Tlzfzdk1zSjA//vSS/m2RRktP72v7xpOuXBi3IsYakCTsKOADYH1gKXJ7kEuA/0Rvy973NaLY7Ne2WVdVTAZLsMomCpZm4RyBtvd8BvlhVD1TVrcD/An6b3hhWr0nyTuA3q+pe4EZgnyQfS3I4cM+kipamYxBI86SqLgGeTW+0x9OTvLKq7qK353Ax8Hrg05OrUBrOIJC23qX0bmqzqBnB9tnAd5PsDdxaVX9F7wP/6c3IkA+pqq/QuwXo0ydWtTQNzxFIW+8cercAvBoo4M+q6qdJXgX8aZJfAfcBr6R3d6jTkmz+0nXiJAqWZuLoo5LUcR4akqSOMwgkqeMMAknqOINAkjrOIJCkjjMIJKnjDAJJ6rj/B2h09s41XECyAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "trials, best_params = search(space, X_train, X_test, y_train, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the results in Azure ML\n",
    "\n",
    "### Using Studio\n",
    "\n",
    "When child runs are used, they are displayed inside of the experiment's run in the tab `child jobs`:\n",
    "\n",
    "![](assets/hyperopt_child_runs.png)\n",
    "\n",
    "You can then view any of the child runs we created and explore metrics, parameters and more:\n",
    "\n",
    "![](assets/hyperopt_child.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploring child runs programmatically\n",
    "\n",
    "We can also access the runs programmatically using MLflow client:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperopt_run = mlflow.last_active_run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see all the child runs by using the tag `mlflow.parentRunId` that gets automatically populated by MLflow and Azure ML."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>run_id</th>\n",
       "      <th>experiment_id</th>\n",
       "      <th>status</th>\n",
       "      <th>artifact_uri</th>\n",
       "      <th>start_time</th>\n",
       "      <th>end_time</th>\n",
       "      <th>metrics.validation_0-logloss</th>\n",
       "      <th>params.max_depth</th>\n",
       "      <th>params.tree_method</th>\n",
       "      <th>params.sampling_method</th>\n",
       "      <th>...</th>\n",
       "      <th>params.n_jobs</th>\n",
       "      <th>params.learning_rate</th>\n",
       "      <th>params.num_boost_round</th>\n",
       "      <th>params.verbosity</th>\n",
       "      <th>tags.mlflow.source.type</th>\n",
       "      <th>tags.mlflow.parentRunId</th>\n",
       "      <th>tags.mlflow.user</th>\n",
       "      <th>tags.mlflow.runName</th>\n",
       "      <th>tags.mlflow.rootRunId</th>\n",
       "      <th>tags.mlflow.source.name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a2f4c25f-a296-4e9b-8379-aff302f87a0b</td>\n",
       "      <td>368f3886-e4e3-449a-bd1e-593c5b0e01ff</td>\n",
       "      <td>FINISHED</td>\n",
       "      <td></td>\n",
       "      <td>2022-06-02 20:38:33.291000+00:00</td>\n",
       "      <td>2022-06-02 20:38:49.621000+00:00</td>\n",
       "      <td>0.317279</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>0.037245679920859816</td>\n",
       "      <td>100</td>\n",
       "      <td>None</td>\n",
       "      <td>LOCAL</td>\n",
       "      <td>7eca10a6-d166-4714-a251-10f1585eb101</td>\n",
       "      <td>Facundo Santiago</td>\n",
       "      <td>hyperopt_learning_rate_max_leaves</td>\n",
       "      <td>7eca10a6-d166-4714-a251-10f1585eb101</td>\n",
       "      <td>/home/santiagxf/miniconda3/envs/heart/lib/pyth...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>27e1ce12-3cef-496a-a1d7-3142cc840566</td>\n",
       "      <td>368f3886-e4e3-449a-bd1e-593c5b0e01ff</td>\n",
       "      <td>FINISHED</td>\n",
       "      <td></td>\n",
       "      <td>2022-06-02 20:38:50.130000+00:00</td>\n",
       "      <td>2022-06-02 20:39:00.107000+00:00</td>\n",
       "      <td>0.313064</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>0.03889419749417589</td>\n",
       "      <td>100</td>\n",
       "      <td>None</td>\n",
       "      <td>LOCAL</td>\n",
       "      <td>7eca10a6-d166-4714-a251-10f1585eb101</td>\n",
       "      <td>Facundo Santiago</td>\n",
       "      <td>hyperopt_learning_rate_max_leaves</td>\n",
       "      <td>7eca10a6-d166-4714-a251-10f1585eb101</td>\n",
       "      <td>/home/santiagxf/miniconda3/envs/heart/lib/pyth...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>62a991d2-d750-4870-b442-8665ba56c25e</td>\n",
       "      <td>368f3886-e4e3-449a-bd1e-593c5b0e01ff</td>\n",
       "      <td>FINISHED</td>\n",
       "      <td></td>\n",
       "      <td>2022-06-02 20:39:01.233000+00:00</td>\n",
       "      <td>2022-06-02 20:39:13.467000+00:00</td>\n",
       "      <td>0.327712</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>0.025487895466987924</td>\n",
       "      <td>100</td>\n",
       "      <td>None</td>\n",
       "      <td>LOCAL</td>\n",
       "      <td>7eca10a6-d166-4714-a251-10f1585eb101</td>\n",
       "      <td>Facundo Santiago</td>\n",
       "      <td>hyperopt_learning_rate_max_leaves</td>\n",
       "      <td>7eca10a6-d166-4714-a251-10f1585eb101</td>\n",
       "      <td>/home/santiagxf/miniconda3/envs/heart/lib/pyth...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>07488ee9-b5f4-4450-beab-8f5811bb0d9f</td>\n",
       "      <td>368f3886-e4e3-449a-bd1e-593c5b0e01ff</td>\n",
       "      <td>FINISHED</td>\n",
       "      <td></td>\n",
       "      <td>2022-06-02 20:39:15.368000+00:00</td>\n",
       "      <td>2022-06-02 20:39:26.214000+00:00</td>\n",
       "      <td>0.324261</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>0.05037740245177806</td>\n",
       "      <td>100</td>\n",
       "      <td>None</td>\n",
       "      <td>LOCAL</td>\n",
       "      <td>7eca10a6-d166-4714-a251-10f1585eb101</td>\n",
       "      <td>Facundo Santiago</td>\n",
       "      <td>hyperopt_learning_rate_max_leaves</td>\n",
       "      <td>7eca10a6-d166-4714-a251-10f1585eb101</td>\n",
       "      <td>/home/santiagxf/miniconda3/envs/heart/lib/pyth...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>e0e59c05-8d34-445d-99dc-9dd60c08a8a0</td>\n",
       "      <td>368f3886-e4e3-449a-bd1e-593c5b0e01ff</td>\n",
       "      <td>FINISHED</td>\n",
       "      <td></td>\n",
       "      <td>2022-06-02 20:39:26.687000+00:00</td>\n",
       "      <td>2022-06-02 20:39:38.601000+00:00</td>\n",
       "      <td>0.321945</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>0.05117885217699967</td>\n",
       "      <td>100</td>\n",
       "      <td>None</td>\n",
       "      <td>LOCAL</td>\n",
       "      <td>7eca10a6-d166-4714-a251-10f1585eb101</td>\n",
       "      <td>Facundo Santiago</td>\n",
       "      <td>hyperopt_learning_rate_max_leaves</td>\n",
       "      <td>7eca10a6-d166-4714-a251-10f1585eb101</td>\n",
       "      <td>/home/santiagxf/miniconda3/envs/heart/lib/pyth...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows \u00d7 49 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 run_id                         experiment_id  \\\n",
       "0  a2f4c25f-a296-4e9b-8379-aff302f87a0b  368f3886-e4e3-449a-bd1e-593c5b0e01ff   \n",
       "1  27e1ce12-3cef-496a-a1d7-3142cc840566  368f3886-e4e3-449a-bd1e-593c5b0e01ff   \n",
       "2  62a991d2-d750-4870-b442-8665ba56c25e  368f3886-e4e3-449a-bd1e-593c5b0e01ff   \n",
       "3  07488ee9-b5f4-4450-beab-8f5811bb0d9f  368f3886-e4e3-449a-bd1e-593c5b0e01ff   \n",
       "4  e0e59c05-8d34-445d-99dc-9dd60c08a8a0  368f3886-e4e3-449a-bd1e-593c5b0e01ff   \n",
       "\n",
       "     status artifact_uri                       start_time  \\\n",
       "0  FINISHED              2022-06-02 20:38:33.291000+00:00   \n",
       "1  FINISHED              2022-06-02 20:38:50.130000+00:00   \n",
       "2  FINISHED              2022-06-02 20:39:01.233000+00:00   \n",
       "3  FINISHED              2022-06-02 20:39:15.368000+00:00   \n",
       "4  FINISHED              2022-06-02 20:39:26.687000+00:00   \n",
       "\n",
       "                          end_time  metrics.validation_0-logloss  \\\n",
       "0 2022-06-02 20:38:49.621000+00:00                      0.317279   \n",
       "1 2022-06-02 20:39:00.107000+00:00                      0.313064   \n",
       "2 2022-06-02 20:39:13.467000+00:00                      0.327712   \n",
       "3 2022-06-02 20:39:26.214000+00:00                      0.324261   \n",
       "4 2022-06-02 20:39:38.601000+00:00                      0.321945   \n",
       "\n",
       "  params.max_depth params.tree_method params.sampling_method  ...  \\\n",
       "0             None               None                   None  ...   \n",
       "1             None               None                   None  ...   \n",
       "2             None               None                   None  ...   \n",
       "3             None               None                   None  ...   \n",
       "4             None               None                   None  ...   \n",
       "\n",
       "  params.n_jobs  params.learning_rate params.num_boost_round params.verbosity  \\\n",
       "0          None  0.037245679920859816                    100             None   \n",
       "1          None   0.03889419749417589                    100             None   \n",
       "2          None  0.025487895466987924                    100             None   \n",
       "3          None   0.05037740245177806                    100             None   \n",
       "4          None   0.05117885217699967                    100             None   \n",
       "\n",
       "  tags.mlflow.source.type               tags.mlflow.parentRunId  \\\n",
       "0                   LOCAL  7eca10a6-d166-4714-a251-10f1585eb101   \n",
       "1                   LOCAL  7eca10a6-d166-4714-a251-10f1585eb101   \n",
       "2                   LOCAL  7eca10a6-d166-4714-a251-10f1585eb101   \n",
       "3                   LOCAL  7eca10a6-d166-4714-a251-10f1585eb101   \n",
       "4                   LOCAL  7eca10a6-d166-4714-a251-10f1585eb101   \n",
       "\n",
       "   tags.mlflow.user                tags.mlflow.runName  \\\n",
       "0  Facundo Santiago  hyperopt_learning_rate_max_leaves   \n",
       "1  Facundo Santiago  hyperopt_learning_rate_max_leaves   \n",
       "2  Facundo Santiago  hyperopt_learning_rate_max_leaves   \n",
       "3  Facundo Santiago  hyperopt_learning_rate_max_leaves   \n",
       "4  Facundo Santiago  hyperopt_learning_rate_max_leaves   \n",
       "\n",
       "                  tags.mlflow.rootRunId  \\\n",
       "0  7eca10a6-d166-4714-a251-10f1585eb101   \n",
       "1  7eca10a6-d166-4714-a251-10f1585eb101   \n",
       "2  7eca10a6-d166-4714-a251-10f1585eb101   \n",
       "3  7eca10a6-d166-4714-a251-10f1585eb101   \n",
       "4  7eca10a6-d166-4714-a251-10f1585eb101   \n",
       "\n",
       "                             tags.mlflow.source.name  \n",
       "0  /home/santiagxf/miniconda3/envs/heart/lib/pyth...  \n",
       "1  /home/santiagxf/miniconda3/envs/heart/lib/pyth...  \n",
       "2  /home/santiagxf/miniconda3/envs/heart/lib/pyth...  \n",
       "3  /home/santiagxf/miniconda3/envs/heart/lib/pyth...  \n",
       "4  /home/santiagxf/miniconda3/envs/heart/lib/pyth...  \n",
       "\n",
       "[5 rows x 49 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "child_runs = mlflow.search_runs(\n",
    "    filter_string=f\"tags.mlflow.parentRunId='{hyperopt_run.info.run_id}'\"\n",
    ")\n",
    "child_runs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8 - AzureML",
   "language": "python",
   "name": "python38-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}