Demos/AutoLabelDemo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
    "\n",
    "Licensed under the MIT License.\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The AutoLabelling Process Demo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This document describes how to implement the autolabelling process for the Supervised version of Forms Recognizer. By using an autolabelling approach we are able to reduce but not remove the need for a human-in-the-loop. It is strongly recommended that manual labelling still takes place for poorly performing models, but the need for manual labelling should be significantly reduced.\n",
    "\n",
    "We will be referencing code from here https://github.com/microsoft/knowledge-extraction-recipes-forms/tree/master/Training/Auto_Labelling/basic_implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# In summary, the autolabelling process will implement the following steps:\n",
    "\n",
    "1. Prepare the files by converting them from TIF --> JPG --> PDF if required\n",
    "2. Create Storage Containers with a specific naming convention and uploaded the converted files - do this for train and test datasets\n",
    "3. Iterate through every container\n",
    "4. Load the corresponding ground truth record (GT) for an invoice\n",
    "5. Retrieve the values from the GT for the keys we want to extract/tag/label\n",
    "6. Call Read Layout (OCR) for the invoice if no OCR file exists for the invoice\n",
    "7. Search through both the line and word level of the OCR file with formatting to find the ground truth values for the keys to be extracted.\n",
    "8. If a value is found, get the corresponding page, height, width and bounding box attributes for the original unformatted OCR value\n",
    "9. Generate the corresponding ocr.labels.json for the invoice\n",
    "10. Upload the label and json files to the Storage Container and train the Supervised version of Forms Recognizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'fpdf'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-184d97cfba0e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     26\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     27\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mTraining\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mAuto_Labelling\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbasic_implementation\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautolabel_common\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mfind_anchor_keys_in_form\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 28\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mTraining\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mAuto_Labelling\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbasic_implementation\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautolabel_prepare_files\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mcreate_container\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mupload_blobs_to_container\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     29\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mTraining\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mAuto_Labelling\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbasic_implementation\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautolabel_training\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mprocess_folder\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mselect_best_training_set\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mform_recognizerv2_train\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     30\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mExtraction\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSupervised\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprediction_supervised\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mdownload_input_files_from_blob_storage\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprocess_folder_and_predict\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/sources/Knowledge Extraction/microsoft-knowledge-extraction-recipes/knowledge-extraction-recipes-forms/Training/Auto_Labelling/basic_implementation/autolabel_prepare_files.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     10\u001b[0m )\n\u001b[1;32m     11\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mdotenv\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mload_dotenv\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 12\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mfpdf\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mFPDF\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     13\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     14\u001b[0m \u001b[0mload_dotenv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'fpdf'"
     ]
    }
   ],
   "source": [
    "import sys, os\n",
    "import json\n",
    "from os import path\n",
    "__prep_file__ = 'autolabel_prepare_file.py'\n",
    "sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__prep_file__), '..')))\n",
    "\n",
    "__common_file__ = 'autolabel_common.py'\n",
    "sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__common_file__), '..')))\n",
    "\n",
    "__train_file__ = 'autolabel_training.py'\n",
    "sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__train_file__), '..')))\n",
    "\n",
    "__root_common__ = 'common.py'\n",
    "sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__root_common__), '..')))\n",
    "\n",
    "__predict_supervised__ = 'Supervised/prediction_supervised.py'\n",
    "sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__predict_supervised__), '..')))\n",
    "\n",
    "__scoring_supervised__ = 'Scoring/evaluation_gt.py'\n",
    "sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__scoring_supervised__), '..')))\n",
    "\n",
    "from azure.storage.blob import (\n",
    "    BlockBlobService,\n",
    "    ContainerPermissions\n",
    ")\n",
    "\n",
    "from Training.Auto_Labelling.basic_implementation.autolabel_common import find_anchor_keys_in_form\n",
    "from Training.Auto_Labelling.basic_implementation.autolabel_prepare_files import create_container, upload_blobs_to_container\n",
    "from Training.Auto_Labelling.basic_implementation.autolabel_training import process_folder, select_best_training_set, form_recognizerv2_train\n",
    "from Extraction.Supervised.prediction_supervised import download_input_files_from_blob_storage, process_folder_and_predict\n",
    "from Evaluation.Scoring.evaluation_gt import print_results, load_json\n",
    "\n",
    "from common.common import is_phrase_in\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up our environment variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will set up some of the environment variables here and others later in the notebook to keep things clear\n",
    "# Set the values here marked with SET THIS HERE\n",
    "class Config:\n",
    "    \"\"\"\n",
    "    Read from .env file\n",
    "    \"\"\"\n",
    "    FORMS_PATH = '../Data/Invoices/Train' #  The directory where our files to process are\n",
    "    DOC_EXT = '.pdf'  # Change if needed\n",
    "    STORAGE_ACCOUNT_NAME = 'SET THIS HERE'  # Account name for storage\n",
    "    STORAGE_KEY = 'SET THIS HERE'   # The key for the storage account\n",
    "    LOCAL_WORKING_DIR =  '../Data/Invoices/temp' # The local temporary directory to which we write and remove\n",
    "    TRAIN_TEST = 'train'  # Suffixes train or test to container name\n",
    "    CONTAINER_SUFFIX = 'autolabelv1'  # The suffix name of the containers that store the training datasets\n",
    "    KEY_FIELD_NAMES = os.environ.get(\"KEY_FIELD_NAMES\")  # The fields to be extracted e.g. invoicenumber,date,total\n",
    "    GROUND_TRUTH_PATH = os.environ.get(\"GROUND_TRUTH_PATH\")  # This is the path to our Ground Truth\n",
    "    LANGUAGE_CODE = 'en'\n",
    "    REGION = 'eastus'  # The region Form Recognizer and OCR are deployed\n",
    "    SUBSCRIPTION_KEY = 'SET THIS HERE'  # CogSvc key\n",
    "    MULTI_PAGE_FIELDS = os.environ.get(\"MULTI_PAGE_FIELDS\")  # These fields appear over multiple pages\n",
    "    MINIMUM_LABELLED_DATA = os.environ.get(\"MINIMUM_LABELLED_DATA\")  # The minimum number of well labelled samples to\n",
    "    #  train on\n",
    "    SAS_PREFIX = os.environ.get(\"SAS_PREFIX\")\n",
    "    SAS = os.environ.get(\"SAS\")  # SAS for storage train\n",
    "    SAS_TEST = os.environ.get(\"SAS\")  # SAS for storage test\n",
    "    SAMPLE_NUMBER = os.environ.get(\"SAMPLE_NUMBER\")  # Sample number of files for prediction\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Convert files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We do not need to convert the files as they are already in pdf format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Create Storage Containers with a specific naming convention and uploaded the converted files - Train dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get a list on the files to process\n",
    "files = [f for f in os.listdir(Config.FORMS_PATH) if f.endswith(Config.DOC_EXT)]\n",
    "\n",
    "# Create the BlockBlockService that the system uses to call the Blob service for the storage account.\n",
    "block_blob_service = BlockBlobService(\n",
    "    account_name=Config.STORAGE_ACCOUNT_NAME, account_key=Config.STORAGE_KEY)\n",
    "\n",
    "for file_name in files:\n",
    "\n",
    "    container_name = Config.CONTAINER_SUFFIX + Config.TRAIN_TEST\n",
    "    print(f\"Uploading to blob {container_name}\")\n",
    "\n",
    "    # Create container if it doesn't exist and get container sas url\n",
    "    _, _ = create_container(block_blob_service, Config.STORAGE_ACCOUNT_NAME, container_name)\n",
    "\n",
    "    # Upload to container\n",
    "    upload_blobs_to_container(block_blob_service, Config.FORMS_PATH, container_name, Config.DOC_EXT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Create Storage Containers with a specific naming convention and uploaded the converted files - Test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Config.FORMS_PATH = '../Data/Invoices/Test' #  The directory where our files to process are\n",
    "Config.TRAIN_TEST = 'test'  # Suffixes train or test to container name\n",
    "\n",
    "\n",
    "# Get a list on the files to process\n",
    "files = [f for f in os.listdir(Config.FORMS_PATH) if f.endswith(Config.DOC_EXT)]\n",
    "\n",
    "# Create the BlockBlockService that the system uses to call the Blob service for the storage account.\n",
    "block_blob_service = BlockBlobService(\n",
    "    account_name=Config.STORAGE_ACCOUNT_NAME, account_key=Config.STORAGE_KEY)\n",
    "\n",
    "for file_name in files:\n",
    "\n",
    "    container_name = Config.CONTAINER_SUFFIX + Config.TRAIN_TEST\n",
    "    print(f\"Uploading to blob {container_name}\")\n",
    "\n",
    "    # Create container if it doesn't exist and get container sas url\n",
    "    _, _ = create_container(block_blob_service, Config.STORAGE_ACCOUNT_NAME, container_name)\n",
    "\n",
    "    # Upload to container\n",
    "    upload_blobs_to_container(block_blob_service, Config.FORMS_PATH, container_name, Config.DOC_EXT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Iterate through every container"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We now want to get our training datasets, as we are only processing one vendor for this demo, we only want to process the training dataset for AutoLabelling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Revert back to Train dataset\n",
    "Config.TRAIN_TEST = 'train'  # Suffixes train or test to container name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "containers = block_blob_service.list_containers()\n",
    "for container in containers:\n",
    "    if (Config.CONTAINER_SUFFIX + Config.TRAIN_TEST\n",
    "            not in container.name):\n",
    "        continue\n",
    "        \n",
    "    assert container.name == Config.CONTAINER_SUFFIX + Config.TRAIN_TEST\n",
    "    print(container.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now we are going to specific the fields we want to extract, these must match the values in your ground truth, so let's load our ground truth file to see what these are"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Load the corresponding ground truth record (GT) for an invoice"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Config.GROUND_TRUTH_PATH = '../Data/Invoices/Invoice_GT.csv'  # This is the path to our Ground Truth\n",
    "ground_truth_df = pd.read_csv(Config.GROUND_TRUTH_PATH, sep=\",\")\n",
    "ground_truth_df.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We can see our columns names above, these are keys we want to work with, so let's populate the environment variable KEY_FIELD_NAMES"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Config.KEY_FIELD_NAMES = 'INVOICE_NUM, INVOICE_DATE, VENDOR, BILL_TO, TOTAL, VAT_ID, VENDOR_ZIP, BILL_TO_ZIP'\n",
    "key_field_names = Config.KEY_FIELD_NAMES.split(',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Steps 5 - 9: AutoLabel and generate the OCR and labels files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vendor_folder_path_pass1 = f\"{Config.LOCAL_WORKING_DIR}/pass1\"\n",
    "vendor_folder_path_pass2 = f\"{Config.LOCAL_WORKING_DIR}/pass2\"\n",
    "\n",
    "# create training files for all input files\n",
    "pass_level, num_files, num_ground_truth = process_folder(\n",
    "    vendor_folder_path_pass1,\n",
    "    vendor_folder_path_pass2,\n",
    "    key_field_names,\n",
    "    Config.DOC_EXT,\n",
    "    Config.LANGUAGE_CODE,\n",
    "    ground_truth_df,\n",
    "    block_blob_service,\n",
    "    Config.CONTAINER_SUFFIX + Config.TRAIN_TEST,\n",
    "    Config.REGION,\n",
    "    Config.SUBSCRIPTION_KEY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 10: Now we select the best training set from our two passes and upload the files and train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because we are AutoLabelling, we can afford to label more than the minimum 5 forms if we have the data. This can result in a model more robust"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Config.MINIMUM_LABELLED_DATA = 5\n",
    "selected_training_set = select_best_training_set(pass_level, vendor_folder_path_pass1, vendor_folder_path_pass2, Config.MINIMUM_LABELLED_DATA)\n",
    "\n",
    "# Upload the best training set to the container\n",
    "upload_blobs_to_container(block_blob_service, selected_training_set, Config.CONTAINER_SUFFIX + Config.TRAIN_TEST, '.json')\n",
    "print(f\"Uploaded files to blob {Config.CONTAINER_SUFFIX + Config.TRAIN_TEST} training set {selected_training_set}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the model on the optimised dataset\n",
    "Config.SAS = \"\"\"SET THIS HERE - the query string component\"\"\"\n",
    "\n",
    "Config.SAS_PREFIX = 'https://[SET YOUR VALUE HERE].blob.core.windows.net/'\n",
    "\n",
    "sasurl = Config.SAS_PREFIX + Config.CONTAINER_SUFFIX + Config.TRAIN_TEST + Config.SAS\n",
    "train_response = form_recognizerv2_train(Config.REGION,\n",
    "                                         Config.SUBSCRIPTION_KEY,\n",
    "                                         sasurl)\n",
    "\n",
    "modelId = train_response['modelInfo']['modelId']\n",
    "print(f\"\\nModelId is {modelId}\")\n",
    "accuracy = train_response['trainResult']['averageModelAccuracy']\n",
    "print(f\"Average Model Accuracy {accuracy}\")\n",
    "\n",
    "for field in train_response['trainResult']['fields']:\n",
    "    print(f\"Field {field['fieldName']} accuracy {field['accuracy']}\")\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's evaluate against the test set "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sample number of files for prediction - used when you have many files to predict and want to sample\n",
    "Config.SAMPLE_NUMBER = 2   # We have two files in our testset\n",
    "\n",
    "# Revert back to test dataset\n",
    "Config.TRAIN_TEST = 'test'  # Suffixes train or test to container name\n",
    "\n",
    "Config.FORMS_PATH = '../Data/Invoices/Test' #  The directory where our files to process are\n",
    "\n",
    "# Download the files to predict locally\n",
    "input_doc_files = download_input_files_from_blob_storage(\n",
    "    block_blob_service, Config.CONTAINER_SUFFIX + Config.TRAIN_TEST, Config.FORMS_PATH, Config.DOC_EXT,\n",
    "    int(Config.SAMPLE_NUMBER))\n",
    "\n",
    "keys = {}\n",
    "\n",
    "keys = process_folder_and_predict(\n",
    "            keys,\n",
    "            Config.FORMS_PATH,\n",
    "            ground_truth_df,\n",
    "            modelId,\n",
    "            'autolabelv1',\n",
    "            input_doc_files,\n",
    "            Config.KEY_FIELD_NAMES,\n",
    "            Config.REGION,\n",
    "            Config.SUBSCRIPTION_KEY\n",
    "        )\n",
    "\n",
    "\n",
    "result_file = Config.FORMS_PATH  + '/supervised_predict_autolabelv1.json'\n",
    "\n",
    "# Let's save the result file\n",
    "with open(result_file, 'w') as json_file:\n",
    "    json.dump(keys, json_file)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "sepanel",
   "language": "python",
   "name": "sepanel"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "source": [],
    "metadata": {
     "collapsed": false
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}