Performance merging large files

[hobti01]

Hi and thanks for the valuable tool!

We're seeing minutes or hours to merge 2 large files (1000s or 10000s of lines) with explode, even if there are no anchors:

$ yq --version
yq (https://github.com/mikefarah/yq/) version 4.13.3
$ cat some.file |
  yq eval-all "(select(fileIndex == 0) | explode(.)) * (select(fileIndex == 1) | explode(.))" \
  - \
  other.file >out.file

Do you have any recommendations for speeding this up?

[mikefarah]

Are you able to PM me files (with data anonymized /redacted)? It's hard for me to know why without being able to recreate the issue...

[hobti01]

With 1.yaml of 100k entries and 2.yaml of 1 entry: ...still running...

With 3.yaml of 10k entries and 2.yaml of 1 entry:

$ time cat 3.yaml | yq eval-all "(select(fileIndex == 0) | explode(.)) * (select(fileIndex == 1) | explode(.))" - 2.yaml > out1.yaml

real    1m12.172s
user    1m31.769s
sys     0m1.832s

With 3.yaml of 10k entries and 4.yaml of 10k (same) entries:

$ time cat 3.yaml | yq eval-all "(select(fileIndex == 0) | explode(.)) * (select(fileIndex == 1) | explode(.))" - 4.yaml > out1.yaml

real    3m28.439s
user    4m20.843s
sys     0m3.899s

[hobti01]

With 1.yaml of 100k entries and 2.yaml of 1 entry:

$ time cat 1.yaml | yq eval-all "(select(fileIndex == 0) | explode(.)) * (select(fileIndex == 1) | explode(.))" - 2.yaml > out.yaml

real    114m7.849s
user    144m10.777s
sys     3m16.965s

This is not a common case, but if you have suggestions for improvement we'd appreciate any pointers.

[mikefarah]

Thanks for that - I've been able to replicate those timings.

I found a easy performance fix in explode that makes that op much faster, but unfortunately that isn't what's taking the bulk of the time - its the actual merge. Looking at the merge code, nothing immediately sticks out to me that can be easily fixed. Perhaps there's a smarter way to do it - but I don't know it ;)

[mikefarah]

Just thought of an optimisation for multiply - it should be much faster in the examples used with 2.yaml - and roughly twice as fast for when they files are equally sized!

[mikefarah]

Just released performance fixes in 4.13.4