Failure in bilingual collection generation

[ronaldtse]

Build failure needs to be fixed here.

Is this a failure in the mn2pdf gem or in isodoc? It seems to be removing the Fontist manifest before the bilingual collection processing is complete. Locally I don't have this issue.

Ping @opoudjis

[relaton] Info: detecting backends:
[relaton-bib] document type "collection" is invalid.
Document Attributes: article is not a legal document type: reverting to 'brochure'
Metanorma XML Syntax: (XML Line 000006:28): element "docidentifier" not allowed yet; expected element "fetched", "formattedref" or "title"
Metanorma XML Syntax: (XML Line 000041:7): element "ext" incomplete; missing required element "editorialgroup"
Metanorma XML Syntax: (XML Line 000063:23): element "sections" incomplete; expected element "clause", "definitions" or "terms"
Document Attributes: article is not a legal document type: reverting to 'brochure'
Metanorma XML Syntax: (XML Line 000006:28): element "docidentifier" not allowed yet; expected element "fetched", "formattedref" or "title"
Metanorma XML Syntax: (XML Line 000041:7): element "ext" incomplete; missing required element "editorialgroup"
Metanorma XML Syntax: (XML Line 000063:23): element "sections" incomplete; expected element "clause", "definitions" or "terms"
java -Xss5m -Xmx2048m -jar /usr/local/bundle/gems/mn2pdf-1.32/lib/../bin/mn2pdf.jar --xml-file "/tmp/collection20210706-2516-1i5755s.presentation.xml" --xsl-file "/usr/local/bundle/gems/metanorma-bipm-1.1.4/lib/isodoc/bipm/bipm.brochure.xsl" --pdf-file "/tmp/collection20210706-2516-1i5755s.pdf" --font-manifest /tmp/fontist_locations20210706-2516-oh01nu.yml
java -Xss5m -Xmx2048m -jar /usr/local/bundle/gems/mn2pdf-1.32/lib/../bin/mn2pdf.jar --xml-file "/tmp/collection20210706-2516-1wxqtis.presentation.xml" --xsl-file "/usr/local/bundle/gems/metanorma-bipm-1.1.4/lib/isodoc/bipm/bipm.brochure.xsl" --pdf-file "/tmp/collection20210706-2516-1wxqtis.pdf" --font-manifest /tmp/fontist_locations20210706-2516-113tcvf.yml
Document Attributes: article is not a legal document type: reverting to 'brochure'
Metanorma XML Syntax: (XML Line 000006:28): element "docidentifier" not allowed yet; expected element "fetched", "formattedref" or "title"
Metanorma XML Syntax: (XML Line 000041:7): element "ext" incomplete; missing required element "editorialgroup"
Metanorma XML Syntax: (XML Line 000063:23): element "sections" incomplete; expected element "clause", "definitions" or "terms"
Document Attributes: article is not a legal document type: reverting to 'brochure'
Metanorma XML Syntax: (XML Line 000006:28): element "docidentifier" not allowed yet; expected element "fetched", "formattedref" or "title"
Metanorma XML Syntax: (XML Line 000041:7): element "ext" incomplete; missing required element "editorialgroup"
Metanorma XML Syntax: (XML Line 000063:23): element "sections" incomplete; expected element "clause", "definitions" or "terms"
Document Attributes: article is not a legal document type: reverting to 'brochure'
Metanorma XML Syntax: (XML Line 000006:28): element "docidentifier" not allowed yet; expected element "fetched", "formattedref" or "title"
Metanorma XML Syntax: (XML Line 000041:7): element "ext" incomplete; missing required element "editorialgroup"
Metanorma XML Syntax: (XML Line 000063:23): element "sections" incomplete; expected element "clause", "definitions" or "terms"
Document Attributes: article is not a legal document type: reverting to 'brochure'
Metanorma XML Syntax: (XML Line 000006:28): element "docidentifier" not allowed yet; expected element "fetched", "formattedref" or "title"
Metanorma XML Syntax: (XML Line 000041:7): element "ext" incomplete; missing required element "editorialgroup"
Metanorma XML Syntax: (XML Line 000063:23): element "sections" incomplete; expected element "clause", "definitions" or "terms"
java -Xss5m -Xmx2048m -jar /usr/local/bundle/gems/mn2pdf-1.32/lib/../bin/mn2pdf.jar --xml-file "bilingual-brochure/collection.presentation.xml" --xsl-file "/usr/local/bundle/gems/metanorma-bipm-1.1.4/lib/isodoc/bipm/bipm.brochure.xsl" --pdf-file "bilingual-brochure/collection.pdf" --font-manifest /tmp/fontist_locations20210706-2516-qoszj5.yml --split-by-language
/usr/local/bundle/gems/mn2pdf-1.32/lib/mn2pdf.rb:40:in `convert': [mn2pdf] Fatal: mn2pdf  (RuntimeError)



Error: Font manifest file '/tmp/fontist_locations20210706-2516-qoszj5.yml' not found!
	from /usr/local/bundle/gems/metanorma-1.3.5/lib/metanorma/output/xslfo.rb:10:in `convert'
	from /usr/local/bundle/gems/isodoc-1.6.7.1/lib/isodoc/xslfo_convert.rb:36:in `convert'
	from /usr/local/bundle/gems/metanorma-1.3.5/lib/metanorma/collection_renderer.rb:82:in `concatenate'
	from /usr/local/bundle/gems/metanorma-1.3.5/lib/metanorma/collection_renderer.rb:61:in `render'
	from /usr/local/bundle/gems/metanorma-1.3.5/lib/metanorma/collection.rb:76:in `render'
	from /usr/local/bundle/bundler/gems/metanorma-cli-ef2ec0dc36da/lib/metanorma/cli/command.rb:73:in `collection'
	from /usr/local/bundle/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
	from /usr/local/bundle/gems/thor-hollaback-0.2.0/lib/thor/hollaback.rb:62:in `run'
	from /usr/local/bundle/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
	from /usr/local/bundle/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
	from /usr/local/bundle/gems/thor-1.0.1/lib/thor/base.rb:485:in `start'
	from /usr/local/bundle/bundler/gems/metanorma-cli-ef2ec0dc36da/lib/metanorma/cli.rb:34:in `start'
	from /usr/local/bundle/bundler/gems/metanorma-cli-ef2ec0dc36da/exe/metanorma:20:in `<top (required)>'
	from /usr/local/bundle/bin/metanorma:23:in `load'
	from /usr/local/bundle/bin/metanorma:23:in `<main>'
##[error]Process completed with exit code 1.
Post job cleanup.

Originally posted by @ronaldtse in #141 (comment)

[CAMOBAP]

Unable to reproduce it locally, tried both docker and macos. After several rerun of docker I see that different files are missing for different runs

I have an assumption that this may be related to/tmp cleanup logic (it fires too soon)

I will:

• Add generate workflow
• Try a couple more docker re-runs

[CAMOBAP]

Well after reading a couple articles, I have found that we cannot rely on /tmp as an IPC approach. There is no guarantee that the temp file will survive the process dead.

Technically we faced with two cases here:

1. ruby's temp file deleted before java executed (happens only once)
2. java's temp file doesn't exist when ruby tried to copy it (happens quiet often on docker and ubuntu)

For the second problem I will try to pass correct output PDF path to avoid unnecessary copy
For the first, we need to doublecheck when unlink called

[ronaldtse]

@CAMOBAP than you for the detailed investigation!

1. ruby's temp file deleted before java executed (happens only once)

The font manifest here is intended only for mn2pdf to consume, so it needs to cross the boundaries between "normal metanorma" (isodoc gem) and "mn2pdf" (mn2pdf gem).

Instead of passing a font file, if you retain the font manifest file pointer until mn2pdf returns to Ruby, the font manifest wouldn't be deleted?

2. java's temp file doesn't exist when ruby tried to copy it (happens quiet often on docker and ubuntu)

We don't have this issue in normal PDF generation because the path of the PDF output file is provided in the arguments. In collection we may have this problem due to temp files and not tracking dependencies. If we do track dependencies like metanorma/metanorma-cli#232 then it wouldn't be a problem.

[kickoke]

Commenting as a reminder of our discussion:

I'm trying to build a standoc metanorma document using docker from this directory: https://github.com/metanorma/mn-samples-standoc

I'm using this command to build the document:
docker run -v "$(pwd)":/metanorma/ metanorma/metanorma metanorma compile document.adoc --agree-to-terms

When I build the document, it only generates .doc and .html output but no PDF.
I've pushed the most recent .err file to the directory as well. I didn't get any error from mn2pdf so I don't know why it doesn't compile PDF...

[CAMOBAP]

PR on review metanorma/metanorma#214

[CAMOBAP]

The original issue is fixed missing pdf will be addressed in #153