Merge pull request #18 from tomas-gajarsky/feature/align-raw

Feature/align

.flake8

@@ -2,5 +2,5 @@
 # exclude directories from being linted
 exclude = internal, venv
 # ignore whitespace before ':'
-ignore = E203, E501
+ignore = E203, E501, W503
 max-line-length = 100

.github/workflows/build.yml

@@ -8,7 +8,7 @@ on:
 
 jobs:
   build:
-    timeout-minutes: 20
+    timeout-minutes: 30
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2

.gitignore

@@ -108,4 +108,5 @@ cython_debug/
 ## User folders
 models/
 outputs/
-tests/data/output/
+tests/data/output/
+data/3dmm/

CHANGELOG.md

@@ -1,5 +1,18 @@
 # Change Log
 
+## 0.1.0
+
+Released on August 22, 2022.
+
+### Added
+* badges for models with SOTA comparison using papers with code
+* predictor for face alignment task - SynergyNet
+* utilizers for face alignment (compute 3d landmarks, mesh and pose) and drawing landmarks
+
+### Changed
+* default logging level to INFO
+* drawing boxes and saving image abstracted to utilizer objects
+
 ## 0.0.8
 
 Released on August 12, 2022.

README.md

@@ -3,7 +3,6 @@
 ![lint](https://github.com/tomas-gajarsky/facetorch/actions/workflows/lint.yml/badge.svg?branch=main)
 [![PyPI](https://img.shields.io/pypi/v/facetorch)](https://pypi.org/project/facetorch/)
 [![Conda (channel only)](https://img.shields.io/conda/vn/conda-forge/facetorch)](https://anaconda.org/conda-forge/facetorch)
-
 [![PyPI - License](https://img.shields.io/pypi/l/facetorch)](https://raw.githubusercontent.com/tomas-gajarsky/facetorch/main/LICENSE)
 <a href="https://github.com/psf/black"><img alt="Code style: black" src="https://img.shields.io/badge/code%20style-black-000000.svg"></a>
 
@@ -43,6 +42,8 @@ Docker Compose provides an easy way of building a working facetorch environment
 * CPU: ```docker compose run facetorch python ./scripts/example.py```
 * GPU: ```docker compose run facetorch-gpu python ./scripts/example.py analyzer.device=cuda```
 
+Check *data/output* for resulting images with bounding boxes and facial 3D landmarks.
+
 ### Configure
 
 The project is configured by files located in *conf* with the main file: *conf/config.yaml*.
@@ -55,6 +56,7 @@ FaceAnalyzer is the main class of facetorch as it is the orchestrator responsibl
 3. Unifier - processor that unifies sizes of all faces and normalizes them
     between 0 and 1.
 4. Predictor dict - set of wrappers around neural networks trained to analyze facial features.
+5. Utilizer dict - set of wrappers around any functionality that requires the output of neural networks e.g. drawing bounding boxes or facial landmarks.
 
 ### Structure
 ```
@@ -65,7 +67,12 @@ analyzer
     └── predictor
             ├── embed
             ├── fer
-            └── deepfake
+            ├── deepfake
+            └── align
+    └── utilizer
+            ├── align
+            ├── draw
+            └── save
 ```
 
 
@@ -80,6 +87,8 @@ analyzer
 1. biubug6
     * code: [Pytorch_Retinaface](https://github.com/biubug6/Pytorch_Retinaface)
     * paper: [Deng et al. - RetinaFace: Single-Shot Multi-Level Face Localisation in the Wild](https://openaccess.thecvf.com/content_CVPR_2020/html/Deng_RetinaFace_Single-Shot_Multi-Level_Face_Localisation_in_the_Wild_CVPR_2020_paper.html)
+    * [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/190500641/face-detection-on-wider-face-hard)](https://paperswithcode.com/sota/face-detection-on-wider-face-hard?p=190500641)
+
 
 
 ### Predictor
@@ -93,6 +102,7 @@ analyzer
 1. 1adrianb
     * code: [unsupervised-face-representation](https://github.com/1adrianb/unsupervised-face-representation)
     * paper: [Bulat et al. - Pre-training strategies and datasets for facial representation learning](https://arxiv.org/abs/2103.16554)
+    * Note: ```include_tensors``` needs to be True in order to include the model prediction in Prediction.logits
 
 
 #### Facial expression recognition (FER)
@@ -105,8 +115,12 @@ analyzer
 1. HSE-asavchenko
     * code: [face-emotion-recognition](https://github.com/HSE-asavchenko/face-emotion-recognition)
     * paper: [Savchenko - Facial expression and attributes recognition based on multi-task learning of lightweight neural networks](https://ieeexplore.ieee.org/abstract/document/9582508)
+    * B2 [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/classifying-emotions-and-engagement-in-online/facial-expression-recognition-on-affectnet)](https://paperswithcode.com/sota/facial-expression-recognition-on-affectnet?p=classifying-emotions-and-engagement-in-online)
+    * B0 [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/facial-expression-and-attributes-recognition/facial-expression-recognition-on-affectnet)](https://paperswithcode.com/sota/facial-expression-recognition-on-affectnet?p=facial-expression-and-attributes-recognition)
+    * B0 [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/facial-expression-and-attributes-recognition/facial-expression-recognition-on-acted-facial)](https://paperswithcode.com/sota/facial-expression-recognition-on-acted-facial?p=facial-expression-and-attributes-recognition)
+
 
-#### Deepfake detection
+#### Deepfake detection (deepfake)
 
     |       deepfake       |      source      |   license   | version |
     | -------------------- | ---------------- | ----------- | ------- |
@@ -116,6 +130,20 @@ analyzer
     * code: [dfdc_deepfake_challenge](https://github.com/selimsef/dfdc_deepfake_challenge)
     * challenge: [Seferbekov - Deepfake Detection Challenge 1st place solution](https://www.kaggle.com/competitions/deepfake-detection-challenge/discussion)
 
+#### Face alignment (align)
+
+    |       align       |      source      |   license   | version |
+    | ----------------- | ---------------- | ----------- | ------- |
+    |    MobileNet v2   |     choyingw     | MIT license |    1    |
+
+1. choyingw
+    * code: [SynergyNet](https://github.com/choyingw/SynergyNet)
+    * challenge: [Wu et al. - Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial Geometry](https://arxiv.org/abs/2110.09772)
+    * [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/synergy-between-3dmm-and-3d-landmarks-for/face-alignment-on-aflw)](https://paperswithcode.com/sota/face-alignment-on-aflw?p=synergy-between-3dmm-and-3d-landmarks-for)
+    * [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/synergy-between-3dmm-and-3d-landmarks-for/head-pose-estimation-on-aflw2000)](https://paperswithcode.com/sota/head-pose-estimation-on-aflw2000?p=synergy-between-3dmm-and-3d-landmarks-for)
+    * [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/synergy-between-3dmm-and-3d-landmarks-for/face-alignment-on-aflw2000-3d)](https://paperswithcode.com/sota/face-alignment-on-aflw2000-3d?p=synergy-between-3dmm-and-3d-landmarks-for)
+    * Note: ```include_tensors``` needs to be True in order to include the model prediction in Prediction.logits
+
 
 
 ### Model download
@@ -125,10 +153,10 @@ You can also download the models manually from a [public Google Drive folder](ht
 
 
 ### Execution time
-Image test.jpg (4 faces) is analyzed in about 400ms and test3.jpg (25 faces) in about 1.1s on NVIDIA Tesla T4 GPU once the default configuration (*conf/config.yaml*) of models is initialized and pre heated to the initial image size 1080x1080. One can monitor the execution times in logs using the DEBUG level.
 
+Image test.jpg (4 faces) is analyzed (including drawing boxes and landmarks) in about 600ms and test3.jpg (25 faces) in about 1.8s (batch_size=8) on NVIDIA Tesla T4 GPU once the default configuration (*conf/config.yaml*) of models is initialized and pre heated to the initial image size 1080x1080 by the first run. One can monitor the execution times in logs using the DEBUG level.
 
-Detailed test.jpg (4 faces) execution times:
+Detailed test.jpg execution times:
 ```
 analyzer
     ├── reader: 27 ms
@@ -137,7 +165,13 @@ analyzer
     └── predictor
             ├── embed: 8 ms
             ├── fer: 22 ms
-            └── deepfake: 109 ms
+            ├── deepfake: 109 ms
+            └── align: 5 ms
+    └── utlizer
+            ├── align: 8 ms
+            ├── draw_boxes: 22 ms
+            ├── draw_landmarks: 152 ms
+            └── save: 0 ms
 ```
 
 
@@ -148,10 +182,11 @@ Run the Docker container:
 
 ### Add predictor
 #### Prerequisites
-1. File of the TorchScript model
-2. Google Drive file ID of the model
+1. file of the TorchScript model
+2. ID of the Google Drive model file
+3. facetorch [fork](https://docs.github.com/en/get-started/quickstart/fork-a-repo)
 
-Facetorch works with models that were exported from PyTorch to TorchScript. You can apply [torch.jit.trace](https://pytorch.org/docs/stable/generated/torch.jit.trace.html) function to compile a PyTorch model as a TorchScript module.
+Facetorch works with models that were exported from PyTorch to TorchScript. You can apply [torch.jit.trace](https://pytorch.org/docs/stable/generated/torch.jit.trace.html) function to compile a PyTorch model as a TorchScript module. Please verify that the output of the traced model equals the output of the original model.
 
 The first models are hosted on my [public Google Drive folder](https://drive.google.com/drive/folders/19qlklR18wYfFsCChQ78it10XciuTzbDM?usp=sharing). You can either send the new model for upload to me, host the model on your 
 Google Drive or host it somewhere else and add your own downloader object to the codebase.
@@ -171,10 +206,11 @@ Google Drive or host it somewhere else and add your own downloader object to the
 to match the requirements of the new model.
 3. Select the postprocessor (or implement a new one based on BasePredPostProcessor) and specify it's parameters e.g. labels in the yaml file to match 
 the requirements of the new model.
+4. (Optional) Add BaseUtilizer derivative that uses output of your model to perform some additional actions.
 
 ##### Configure tests
-1. Add a new predictor to the main *config.yaml* and all *tests.config.<n>.yaml* files. Alternatively, create a new config file e.g. 
-*tests.config.<n>.yaml* and add it to the ```/tests/conftest.py``` file.
+1. Add a new predictor to the main *config.yaml* and all *tests.config.n.yaml* files. Alternatively, create a new config file e.g. 
+*tests.config.n.yaml* and add it to the ```/tests/conftest.py``` file.
 2. Write a test for the new predictor in ```/tests/test_<predictor_name>.py```
 
 #### Test and submit