Refactor filenames and classes

README.md

@@ -1,7 +1,7 @@
 ## MAAD: A Model for Attended Awareness in Driving
 [Install](#install) // [Datasets](#datasets) // [Training](#training) // [Experiments](#experiments) // [Analysis](#analysis) // [License](#license)
 
-Official [PyTorch](https://pytorch.org/) implementation of *A Model for Attended Awareness in Driving (MAAD)* invented by the RAD Team at [Toyota Research Institute (TRI)](https://www.tri.global/), in particular for [Link to Paper](https://www.overleaf.com/project/5cb74be8dc8abc4e20cab4bc)
+Official [PyTorch](https://pytorch.org/) implementation of *MAAD: A Model and Dataset for "Attended Awareness" in Driving* invented by the RAD Team at [Toyota Research Institute (TRI)](https://www.tri.global/), in particular for [Link to Paper](https://www.overleaf.com/project/5cb74be8dc8abc4e20cab4bc)
 *Deepak Gopinath, Guy Rosman, Simon Stent, Katsuya Terahata, Luke Fletcher, Brenna Argall, John Leonard*.
 
 MAAD affords estimation of attended awareness based on noisy gaze estimates and scene video over time. This learned model additionally affords saliency estimation and refinement of a noisy gaze signal. We demonstrate the performance of the model on a new, annotated dataset that explores the gaze and perceived attended awareness of subjects as they observe a variety of driving scenarios. In this dataset, we provide a surrogate annotated third person estimate of attended awareness as a reproducible supervisory cue.
@@ -42,14 +42,14 @@ Datasets are assumed to be downloaded in `/data/datasets/MAAD_ATT_AWARENESS_LABE
 
 ### Optic Flow
 MAAD uses optic flow of the videos as a side-channel information to perform temporal regularizations. For the purposes of our model, we utilized [[RAFT: Recurrent All Pairs Field Transforms for Optical Flow](https://arxiv.org/pdf/2003.12039.pdf)] to generate optic flow. 
-For each video in the dataset, the optic flow model has to be run all frame pairs N frames apart. The current code assumes that the optic flow generated is at half-resolution with a padding of 2 pixels (on each side) along the y direction. These parameters denoted as `OPTIC_FLOW_SCALE_FACTOR, OPTIC_FLOW_H_PAD, OPTIC_FLOW_W_PAD` can be altered in the `att-aware/src/chm/utils/chm_consts.py` file to suit your needs.
+For each video in the dataset, the optic flow model has to be run all frame pairs N frames apart. The current code assumes that the optic flow generated is at half-resolution with a padding of 2 pixels (on each side) along the y direction. These parameters denoted as `OPTIC_FLOW_SCALE_FACTOR, OPTIC_FLOW_H_PAD, OPTIC_FLOW_W_PAD` can be altered in the `att-aware/src/maad/utils/maad_consts.py` file to suit your needs.
 
-Optic flow is assumed to be cached as `~/chm_cache/optic_flow/VIDEO_ID/frame_N.npy`
+Optic flow is assumed to be cached as `~/maad_cache/optic_flow/VIDEO_ID/frame_N.npy`
 
 ### Segmentation Masks
 MAAD uses segmentation masks for the videos in order to perform diffusivity-based spatial regularization. For the purposes of our model, we used MaskRCNN to generate the segmentation masks for each frame for each video. 
 
-Segmentation masks are assumed to be cached as `~/chm_cache/segmentations_from_video/VIDEO_ID/segmentations_frames/frame_N.png`
+Segmentation masks are assumed to be cached as `~/maad_cache/segmentations_from_video/VIDEO_ID/segmentations_frames/frame_N.png`
 
 During training, lower resolution mask images will be generated by resizing the full sized masks and will be cached back into the same location as `frame_N_ar_{aspect_ratio_reduction_factor}.png`.
 
@@ -58,46 +58,46 @@ MAAD model training can be done using the `train.py` script.
 Run the following command to train a model using all 8 videos (split into a train and test sets) using the parameter settings used in the ICCV paper. 
 ` python train.py --train_sequence_ids 6 7 10 11 26 35 53 60 --use_std_train_test_split --add_optic_flow --use_s3d --enable_amp`
 Default resolution used is `240 x 135`.
-All training args are present in `/att-aware/src/chm/config/args_file.py`
+All training args are present in `/att-aware/src/maad/config/args_file.py`
 
-Models will be saved at `~/cognitive_heatmap/models/TRAINING_HASH_NAME`
+Models will be saved at `~/maad/models/TRAINING_HASH_NAME`
 ## Experiments
 Three different experiments are proposed for MAAD. All experiments are done using the test split. Gaze Denoising and Awareness Estimation uses the trained model for inference. Gaze Calibration experiment involves continued training to optimize the miscalibration transform. 
-All experiment results are saved as jsons in `~/cognitive_heatmap/results/`
+All experiment results are saved as jsons in `~/maad/results/`
 ### Gaze Denoising
 MAAD can be used for denoising noisy gaze estimates by relying on saliency information. 
-The denoising experiment script is located at `att-aware/src/scripts/experiment_chm_denoising.py`
+The denoising experiment script is located at `att-aware/src/scripts/experiment_maad_denoising.py`
 
 The script can be run using the following command:
-`python experiment_chm_denoising.py --train_sequence_ids 6 7 10 11 26 35 53 60 --use_std_train_test_split --add_optic_flow --use_s3d --enable_amp --load_indices_dict_path ~/cognitive_heatmap/logs/TRAINING_HASH/TRAINING_HASH/indices_dict_folder/indices_dict.pkl --load_model_path ~/cognitive_heatmap/models/TRAINING_HASH/MODEL.pt --max_inference_num_batches 1000`
+`python experiment_maad_denoising.py --train_sequence_ids 6 7 10 11 26 35 53 60 --use_std_train_test_split --add_optic_flow --use_s3d --enable_amp --load_indices_dict_path ~/maad/logs/TRAINING_HASH/TRAINING_HASH/indices_dict_folder/indices_dict.pkl --load_model_path ~/maad/models/TRAINING_HASH/MODEL.pt --max_inference_num_batches 1000`
 
 ### Gaze Recalibration
 MAAD can be used for recalibration of a miscalibrated gaze (due to errors in DMS). 
-The calibration experiment script is located at `att-aware/src/scripts/experiment_chm_calibration.py`
+The calibration experiment script is located at `att-aware/src/scripts/experiment_maad_calibration.py`
 The calibration experiment script can be run using the follow command:
 
-`python experiment_chm_calibration_optimization.py --train_sequence_ids 6 7 10 11 26 35 53 60 --use_std_train_test_split --add_optic_flow --use_s3d --enable_amp --load_indices_dict_path ~/cognitive_heatmap/logs/TRAINING_HASH/TRAINING_HASH/indices_dict_folder/indices_dict.pkl --load_model_path ~/cognitive_heatmap/models/TRAINING_HASH/MODEL.pt --dropout_ratio '{"driver_facing":0.0, "optic_flow":0.0}'`
+`python experiment_maad_calibration_optimization.py --train_sequence_ids 6 7 10 11 26 35 53 60 --use_std_train_test_split --add_optic_flow --use_s3d --enable_amp --load_indices_dict_path ~/maad/logs/TRAINING_HASH/TRAINING_HASH/indices_dict_folder/indices_dict.pkl --load_model_path ~/maad/models/TRAINING_HASH/MODEL.pt --dropout_ratio '{"driver_facing":0.0, "optic_flow":0.0}'`
 
 Note that, the above command assumes that the model used for recalibration was trained using the default cost parameters. It is important that the cost coefficients match the original values. Furthermore, the `dropout_ratio` for `driver_facing` gaze module should be set at `0.0` so that gaze is available as a side-channel input to the network at all times. The miscalibration noise levels can be specified using the `miscalibration_noise_levels` argument. 
 
 ### Awareness Estimation
 MAAD can used for attended awareness estimation based on scene context and an imperfect gaze information. 
-The attended awareness estimation script is located at `att-aware/src/scripts/experiment_chm_awareness_estimation.py`
+The attended awareness estimation script is located at `att-aware/src/scripts/experiment_maad_awareness_estimation.py`
 
 The attended awareness estimation script can be run using the following command:
-`python experiment_chm_awareness_estimation.py --train_sequence_ids 6 7 10 11 26 35 53 60 --use_std_train_test_split --add_optic_flow --use_s3d --enable_amp --load_indices_dict_path ~/cognitive_heatmap/logs/TRAINING_HASH/TRAINING_HASH/indices_dict_folder/indices_dict.pkl --load_model_path ~/cognitive_heatmap/models/TRAINING_HASH/MODEL.pt`
+`python experiment_maad_awareness_estimation.py --train_sequence_ids 6 7 10 11 26 35 53 60 --use_std_train_test_split --add_optic_flow --use_s3d --enable_amp --load_indices_dict_path ~/maad/logs/TRAINING_HASH/TRAINING_HASH/indices_dict_folder/indices_dict.pkl --load_model_path ~/maad/models/TRAINING_HASH/MODEL.pt`
 
 ## Analysis
 We have also provided scripts to parse and compute statistics on the results outputted by the experiment scripts. These scripts are available at `att-aware/src/scripts/parse_*_data.py` where `*` could be `denoising, calibration_optimization, awareness_estimation`
 
 The results of the parsing scripts will be outputted directly in the terminal. The parsing scripts can be run using the following commands.
-`python parse_denoising_data.py --results_json_prefix ~/cognitive_heatmap/results/GAZE_DENOISING`. Assumes that the result of the denoising experiment is in `GAZE_DENOISING.json`
+`python parse_denoising_data.py --results_json_prefix ~/maad/results/GAZE_DENOISING`. Assumes that the result of the denoising experiment is in `GAZE_DENOISING.json`
 
-`python parse_awareness_estimation_data.py --results_json_prefix ~/cognitive_heatmap/results/AWARENESS_ESTIMATION`. Assumes that the result of the awareness estimation experiment is in `AWARENESS_ESTIMATION.json`
+`python parse_awareness_estimation_data.py --results_json_prefix ~/maad/results/AWARENESS_ESTIMATION`. Assumes that the result of the awareness estimation experiment is in `AWARENESS_ESTIMATION.json`
 
 The results of the calibration experiments are expected to stored in files with the following filename convention
 `experiment_type_gaze_calibration_miscalibration_noise_level_NOISELEVEL_optimization_run_num_OPTIMIZATIONNUM_FILENAMEAPPEND.json`,
-where `NOISELEVEL` is in the `miscalibration_noise_levels` argument in `experiment_chm_calibration_optimization.py`
+where `NOISELEVEL` is in the `miscalibration_noise_levels` argument in `experiment_maad_calibration_optimization.py`
 `OPTIMIZATIONNUM` goes from 0 to `num_optimization_runs`-1 and `FILENAMEAPPEND` is the `filename_append` argument in the experiment.
 
 `python parse_calibration_optimization_data.py --folder_containing_results FOLDER_CONTAINING_JSONS --num_optimization_runs (same val as used in the experiment) --miscalibration_noise_levels (same val as used in the experiment) --filename_append (same val as used in the experiment)`

src/chm/__init__.py → src/maad/__init__.py

@@ -7,10 +7,10 @@
 __homepage__ = "https://github.com/ToyotaResearchInstitute/att-aware"
 __docs__ = "att-aware is a library for attended awareness estimation"
 __long_docs__ = """
-Official [PyTorch](https://pytorch.org/) implementation of _Cognitive HeatMap_
+Official [PyTorch](https://pytorch.org/) implementation of _MAAD_
 invented by the RAD Team at [Toyota Research Institute (TRI)](https://www.tri.global/),
-in particular for _Cognitive Heatmap_:
-[*Cognitive Heatmap: A Model for Driver Situational Awareness*],
-*Deepak Gopinath, Guy Rosman, Simon Stent, Katsuya Terahata, Luke Fletcher, John Leonard*.
+in particular for _MAAD_:
+[*MAAD: A Model and Dataset for "Attended Awareness" in Driving*],
+*Deepak Gopinath, Guy Rosman, Simon Stent, Katsuya Terahata, Luke Fletcher, Brenna Argall, John Leonard*.
 
 """

src/chm/configs/args_file.py → src/maad/configs/args_file.py

@@ -1,7 +1,7 @@
 # Copyright 2021 Toyota Research Institute.  All rights reserved.
 
 """
-Default chm arguments for training, loss function, dataset creation, inference
+Default maad arguments for training, loss function, dataset creation, inference
 """
 
 import argparse
@@ -23,13 +23,13 @@ def parse_arguments(session_hash, additional_argument_setters=[]):
     parser.add_argument(
         "--precache_dir",
         dest="precache_dir",
-        default=os.path.join(os.path.expanduser("~"), "chm_cache"),
+        default=os.path.join(os.path.expanduser("~"), "maad_cache"),
         help="Full path to the directory in which the cached images, segmentation masks and optic flow are stored",
     )
     parser.add_argument(
         "--att_awareness_labels",
         type=str,
-        default=os.path.join(os.path.expanduser("~"), "data", "CHM_ATT_AWARENESS_LABELS.csv"),
+        default=os.path.join(os.path.expanduser("~"), "data", "MAAD_ATT_AWARENESS_LABELS.csv"),
         help="Path to CSV file containing the attended awareness annotations",
     )
     parser.add_argument(
@@ -65,13 +65,13 @@ def parse_arguments(session_hash, additional_argument_setters=[]):
     parser.add_argument(
         "--log_dir",
         dest="log_dir",
-        default=os.path.join(os.path.expanduser("~"), "cognitive_heatmap", "logs", session_hash),
+        default=os.path.join(os.path.expanduser("~"), "maad", "logs", session_hash),
         help="Full path to the the directory where the tensorboard logs will be written",
     )
     parser.add_argument(
         "--training_hash",
         dest="training_hash",
-        default="chm_train",
+        default="maad_train",
         help="string describing meaningful descriptor string for training session",
     )
     parser.add_argument(
@@ -116,11 +116,7 @@ def parse_arguments(session_hash, additional_argument_setters=[]):
         help="max number of batches during inference",
     )
     parser.add_argument(
-        "--awareness_batch_size",
-        action="store",
-        type=int,
-        default=8,
-        help="Batch size used for the awareness dataset",
+        "--awareness_batch_size", action="store", type=int, default=8, help="Batch size used for the awareness dataset",
     )
     parser.add_argument(
         "--batch_aggregation_size",
@@ -429,11 +425,7 @@ def parse_arguments(session_hash, additional_argument_setters=[]):
     ########################################## LOSS FUNCTION ARGS #######################################
 
     parser.add_argument(
-        "--gt_prior_loss_coeff",
-        action="store",
-        type=float,
-        default=0.0,
-        help="Coeff on gaze transform prior loss",
+        "--gt_prior_loss_coeff", action="store", type=float, default=0.0, help="Coeff on gaze transform prior loss",
     )
     parser.add_argument(
         "--awareness_of_gaze_weight_factor_max",

src/chm/dataset/chm_att_awareness_dataset.py → src/maad/dataset/maad_att_awareness_dataset.py

@@ -2,13 +2,13 @@
 import copy
 import pandas as pd
 
-from chm.dataset.chm_base_dataset import CHMBaseDataset
+from maad.dataset.maad_base_dataset import MAADBaseDataset
 
 
-class CHMAttAwarenessDataset(CHMBaseDataset):
+class MAADAttAwarenessDataset(MAADBaseDataset):
     def __init__(self, dataset_type=None, params_dict=None, **kwargs):
         """
-        CHMAttAwarenessDataset dataset class. Dataset class that encapsulates the attended awareness annotation data.
+        MAADAttAwarenessDataset dataset class. Dataset class that encapsulates the attended awareness annotation data.
 
         Parameters:
         ----------
@@ -73,7 +73,7 @@ def get_metadata_list(self):
 
     def _create_metadata_tuple_list(self):
         """
-        Initializes the metadata_len and metadata_list if needed. The function is called at the very end of the CHMBaseDataset init function
+        Initializes the metadata_len and metadata_list if needed. The function is called at the very end of the MAADBaseDataset init function
 
         Parameters:
         ----------
@@ -116,11 +116,11 @@ def __getitem__(self, idx):
         Returns:
         -------
         data_dict: dict
-            Same keys as the data_dict in chm_base_dataset.
+            Same keys as the data_dict in maad_base_dataset.
             Additional key: att_annotation
 
         auxiliary_info_list: list
-            Same as chm_base_dataset
+            Same as maad_base_dataset
         """
 
         att_label_item = self.att_awareness_labels.iloc[idx]

src/chm/dataset/chm_base_dataset.py → src/maad/dataset/maad_base_dataset.py

@@ -8,13 +8,13 @@
 
 from torch.utils.data import Dataset
 from PIL import Image
-from utils.chm_consts import *
+from utils.maad_consts import *
 
 
-class CHMBaseDataset(Dataset):
+class MAADBaseDataset(Dataset):
     def __init__(self, dataset_type=None, params_dict=None, **kwargs):
         """
-        CHMBaseDataset dataset class. Base class for all the other Dataset classes used for CHM.
+        MAADBaseDataset dataset class. Base class for all the other Dataset classes used for MAAD.
         Implements all the getters for caches and loads up the pandas dataframe with all gaze information
 
         Parameters

src/chm/dataset/chm_gaze_dataset.py → src/maad/dataset/maad_gaze_dataset.py

@@ -1,13 +1,13 @@
 # Copyright 2020 Toyota Research Institute.  All rights reserved.
 import itertools
 
-from chm.dataset.chm_base_dataset import CHMBaseDataset
+from chm.dataset.chm_base_dataset import MAADBaseDataset
 
 
-class CHMGazeDataset(CHMBaseDataset):
+class MAADGazeDataset(MAADBaseDataset):
     def __init__(self, dataset_type=None, params_dict=None, **kwargs):
         """
-        CHMGazeDataset dataset class/
+        MAADGazeDataset dataset class/
         Dataset class for returning gaze and image data for a single sequence
 
         Parameters:
@@ -42,7 +42,7 @@ def _setup_resources(self, **kwargs):
 
     def _create_metadata_tuple_list(self):
         """
-        Initializes the metadata_len and metadata_list if needed. The function is called at the very end of the CHMBaseDataset init function
+        Initializes the metadata_len and metadata_list if needed. The function is called at the very end of the MAADBaseDataset init function
 
         Parameters:
         ----------
@@ -99,4 +99,4 @@ def __getitem__(self, idx):
         """
         (video_id, subject, task), query_frame = self.metadata_list[idx]
         data_dict, auxiliary_info_list = self._get_sequence(video_id, subject, task, query_frame)
-        return data_dict, auxiliary_info_list
+        return data_dict, auxiliary_info_list

src/chm/dataset/chm_pairwise_gaze_dataset.py → src/maad/dataset/maad_pairwise_gaze_dataset.py

@@ -1,13 +1,13 @@
 # Copyright 2020 Toyota Research Institute.  All rights reserved.
 import itertools
 
-from chm.dataset.chm_base_dataset import CHMBaseDataset
+from maad.dataset.maad_base_dataset import MAADBaseDataset
 
 
-class CHMPairwiseGazeDataset(CHMBaseDataset):
+class MAADPairwiseGazeDataset(MAADBaseDataset):
     def __init__(self, dataset_type=None, params_dict=None, **kwargs):
         """
-        CHMPairwiseGazeDataset dataset class.
+        MAADPairwiseGazeDataset dataset class.
         Returns a pair of sequences that is used to compute the consistency cost term.
 
         Parameters:
@@ -37,7 +37,7 @@ def _setup_resources(self, **kwargs):
 
     def _create_metadata_tuple_list(self):
         """
-        Initializes the metadata_len and metadata_list if needed. The function is called at the very end of the CHMBaseDataset init function
+        Initializes the metadata_len and metadata_list if needed. The function is called at the very end of the MAADBaseDataset init function
 
         Parameters:
         ----------
@@ -86,4 +86,4 @@ def __getitem__(self, idx):
 
         data_dict["data_t"] = (data_dict_t, auxiliary_info_list_t)
         data_dict["data_tp1"] = (data_dict_tp1, auxiliary_info_list_tp1)
-        return data_dict
+        return data_dict