Skip to content

Latest commit

 

History

History
89 lines (61 loc) · 3.82 KB

README.md

File metadata and controls

89 lines (61 loc) · 3.82 KB

A generic transliteration trainer using Encoder-Decoder transformer model.

Directory organisation

  1. src: Contains source codes for pre-processing and training.
  2. dataset: Contains the datasets use for training.
  3. records: Contains the details of all the trainings.
  4. records/active: Contains the details of current training.

Dataset preparation and Pre-processing

All the tools required for data preprocessing are available. Consider transliteration from lang1 to lang2.

Steps for data preparation:

  1. Prepare a csv file in the following format:
    lang1_word1, lang2_word1
    lang1_word2, lang2_word2

  2. Clean the dataset:
    py src/clean_dataset.py -f path_to_csv_file -l lang-code

    This will create a directory named lang-code in dataset directory. And a file lang-code.csv in the lang-code directory which contents the cleaned and filtered dataset.

  3. Make token file:
    py src/mk_token_file.py -f path_to_clean_csv_file -l lang-code

    This will create a lang-code.tokens file in lang-code directory. Be sure use the cleaned csv file or else unwanted token might be in the token file. This token file will be use to initialise a Tokeniser which will be use to tokenise the datasets.

  4. Shuffle and split:
    /src/split_data.sh -f path_to_clean_csv_file -l lang-code -r val:test

    e.g. For spliting file.csv into training, validation, and testing datasets into 10% validation and 10% testing (remaining will be training set)

    ./src/split_data.sh -f ./file.csv -l lang-code -r 10:10

    This will create three files in the lang-code directory

    1. lang-code-train.csv: trainig dataset
    2. lang-code-val.csv: validation dataset
    3. lang-code-test.csv testing dataset

Training

Now we the trainig dataset ready. We can start training. Training is done by the train.py file. It excepts the following arguments:
  • e: Number of epochs to train.
  • l: Language code
  • i: epoch interval to save checkpoints. Default is 5
  • R: Restart the training. Previous progress will be lost. It will ask for confirmation before proceeding.
  • V: If provided, no validation will be perform during training. Therefore the training will be faster but there will be no validation graph.
  • r: If provided, the training will be done in reverse order (from lang2 to lang1).
  • S: Short training. It will train only for 10 samples. Use for quick testing of the model.

Examples:
To train for lang-code for 100 epochs:
py src/train.py -e 100 -l lang-code

To train for lang-code for 100 epochs in reverse (lang2 to lang1):
py src/train.py -e 100 -l lang-code -r

To restart the training:
py src/train.py -e 100 -l lang-code -R

Evaluation

Use evaluate.py for testing. It will use the dataset/lang-code/lang-code-test.csv file. Valid arguments:
  1. l: lang-code
  2. r: reversre evaluation. Use this if the model was trained reverse
  3. S: short evaluation. Use for quik testing of the model

py src/evaluate.py -l lang-code

When the training and testing is done, move the training records using:

py src/mv_active.py

This will move the records/active directory to records/record_n Then a new training session can be started.

Graphs

The training records all the metrics in records/active/metric file. The graph will be drawn using this file. The graph is drawn using draw_graph.py and save in records directory. It can take following arguments:
  1. d: record directory. Default id records/active
  2. p: if provided, graph will be shown on screen
  3. s: if provided, graph will not be save on disk

py src/draw_graph.py -d records/record_1