Merge pull request #90 from VIDA-NYU/amlb

AMLB integration.

scripts/amlb/README.md

@@ -0,0 +1,37 @@
+# AMLB Experiments
+
+
+
+We used the AutoML Benchmark (AMLB) to run these experiments locally using Python 3.9. To reproduce these experiments, 
+follow the next steps.
+
+
+1. Clone and install AMLB. See the [AMLB repository](https://github.com/openml/automlbenchmark/tree/v2.0.6/) 
+for additional details about the installation. 
+
+```
+git clone https://github.com/openml/automlbenchmark.git --branch stable --depth 1
+cd automlbenchmark
+pip install -r requirements.txt
+cd ..
+```
+
+2. Create the *openml_datasets* and *results* folder (AMLB will use theses folders).
+```
+mkdir openml_datasets
+mkdir results
+```
+
+
+3. To test the installation, run the following command. You should get valid ML pipelines after running it.
+```
+ python automlbenchmark/runbenchmark.py Alpha-AutoML  openml/t/12  test -f 0 -u user_config/ -i openml_datasets/ -o results/
+```
+
+4. We ran all the systems (AutoWEKA, TPOT, H2O, AutoGluon, Auto-Sklearn, and AlphaD3M) using Singularity containers in 
+SLURM batch jobs in the [NYU Greene Cluster](https://sites.google.com/nyu.edu/nyu-hpc/hpc-systems/greene). To run the 
+experiments in this cluster, run `bash ./run_all_automlbenchmark.sh`.
+All the results will be stored in the `./results/results.csv` file.
+
+
+

scripts/amlb/automl_job.SBATCH

@@ -0,0 +1,8 @@
+#!/bin/bash
+#SBATCH -c 4
+#SBATCH --mem 32GB
+#SBATCH --time 01:15:00
+#SBATCH --output logs/automl_job_%J.out
+#SBATCH [email protected]
+
+singularity exec --bind /scratch/rl3725/alphaautoml_experiments/experiments:/scratch/rl3725/alphaautoml_experiments/experiments --overlay overlay-15GB-500K.ext3:ro /scratch/work/public/singularity/ubuntu-20.04.4.sif /bin/bash -c "source /ext3/env.sh; python automlbenchmark/runbenchmark.py ${1} ${2} 1h4c -f 0 -u user_config/ -i openml_datasets/ -o results/"