custom delft train args

[de-code]

Allows custom args to be passed in to the the training operation.

It adds the following optional configuration options:

• grobid.delft.train.module: Training module (default is grobidTagger.py as per DeLFTModel)
• grobid.delft.train.args: additional arguments to be passed to the training module. To keep it simple, it's split on spaces (which could be improved, should shell parsing be required in the future).

This is also useful for the current training, to configure the training.

The useELMo flag didn't actually work for training because the command list was immutable.

/cc @kermitt2 @lfoppiano

[coveralls]

Coverage increased (+0.04%) to 38.313% when pulling 95534bc on elifesciences:custom-delft-train-args into 506c00a on kermitt2:master.

[lfoppiano]

@de-code so this should run the training using delft directly from Grobid?

[de-code]

@de-code so this should run the training using delft directly from Grobid?

Yes, it would be trained like the Wapiti model. Although I have deferred trying to run it that way. Not least because I then need to create GPU version of the Docker container to run in the cloud. Since I am just using a single training dataset at present, it's okay to do that manually (I still need to run the GROBID training command to get the converted training data).

[lfoppiano]

This PR works in Grobid and on a submodule 👍
Only condition is that the training must be ran after activating the virtual environment.
Better to fix #490 on a separate PR, cause it requires moving around some code.

[de-code]

Hi @lfoppiano I rebased as requested

[de-code]

BTW I would recommend to always squash merge PRs

[lfoppiano]

I wonder, what about we make this an optional method for training the delft model?
Since Jep is not reliable at this stage, this could still be removed in future.

For me, personally the alternative is to poke the files from the tmp directory, and run the training manually, which is also not ideal dd

[de-code]

I wonder, what about we make this an optional method for training the delft model?
Since Jep is not reliable at this stage, this could still be removed in future.

For me, personally the alternative is to poke the files from the tmp directory, and run the training manually, which is also not ideal dd

Just trying to go through some old GitHub notifications. Not sure if it was for me but there seem to be some agreement, that a way to train a DeLFT model without GROBID can be beneficial.

For training via GROBID, as you mentioned JEP is less reliable. Maybe training via the command has the additional advantage that it is more transparent, simpler (given the tool already exists) and could be reproduced separately (via the CLI). I guess for training and evaluation there shouldn't be any noticeable performance difference.

[lfoppiano]

I would parametrize this change via configuration and add a new CLI function in grobid to generate the training data automatically so that we provide multiple ways of training the models.

[lfoppiano]

new CLI is done: https://github.com/lfoppiano/grobid-superconductors/blob/master/src/main/java/org/grobid/service/command/PrepareDelftTraining.java

Since it's in a (yet) private repository, here a (non updated) version: https://gist.github.com/lfoppiano/dd28365a83d7aaf9b459ec0ef846ea34

[de-code]

I was resolving conflicts after the addition of the architecture flag.
Perhaps this could be merged soonish?