Toucan Results

[darbyjohnston]

I saw the nice results that @camkerr added for Resolve and NukeStudio, so I thought I would also add some for toucan:

Test                                 | Score   | MFD
----------------------------------------------------
Full clip with no effects........... | 100.00% |  0
Segment at start of media........... | 100.00% |  0
Segment in the middle of media...... | 100.00% |  0
Segment at the end of media......... | 100.00% |  0
Freeze frame at head of media....... | 100.00% |  0
Freeze frame is middle of media..... | 100.00% |  0
Freeze frame at end of media........ | 100.00% |  0
100% speed (no change).............. | 100.00% |  0
Slow to 99% speed................... | 100.00% |  0
Slow to 90% speed................... |  90.83% |  1
Slow to 50% speed................... | 100.00% |  0
Slow to 10% speed................... | 100.00% |  0
Speed up to 101% speed.............. |  98.33% |  1
Speed up to 110% speed.............. |  91.67% |  0
Speed up to 200% speed (2x)......... |  50.00% |  0
Speed up to 1000% speed (10x)....... |  10.00% |  0
Fit-to-fill 99 frames into 100...... | 100.00% |  0
Fit-to-fill 90 frames into 100...... |  91.00% |  1
Fit-to-fill 50 frames into 100...... | 100.00% |  0
Fit-to-fill 33 frames into 100...... | 100.00% |  0
Fit-to-fill 5 frames into 100....... | 100.00% |  0
Fit-to-fill 100 frames into 99...... | 100.00% |  0
Fit-to-fill 100 frames into 50...... | 100.00% |  0
Fit-to-fill 100 frames into 33...... | 100.00% |  0
Fit-to-fill 100 frames into 10...... | 100.00% |  0
Fit-to-fill 100 frames into 9....... | 100.00% |  0
Trim-to-fill 99 frames into 100..... |   1.00% |  1
Trim-to-fill 90 frames into 100..... |   1.00% | 10
Trim-to-fill 50 frames into 100..... |   1.00% | 50
Trim-to-fill 33 frames into 100..... |   1.00% | 67
Trim-to-fill 5 frames into 100...... |   1.00% | 95
Trim-to-fill 100 frames into 99..... |  50.00% |  2
Trim-to-fill 100 frames into 50..... |   2.00% | 50
Trim-to-fill 100 frames into 33..... |   3.03% | 67
Trim-to-fill 100 frames into 10..... |  10.00% | 90
Trim-to-fill 100 frames into 9...... |  11.11% | 91
Reverse 100%........................ | 100.00% |  0
Reverse 50%......................... |  75.00% |  1
Reverse 200%........................ |   2.00% | 49
Reverse 30%......................... |  73.00% |  1
Reverse 120%........................ |   2.00% | 17
----------------------------------------------------
Average score: 67.44%

Score = Percentage of frames that match
MFD   = Maximum difference between frames (ignoring blank frames)

Instead of using diff to compare the overall results, I wrote a small Python script that compares each clip. I don't write a lot of Python, hopefully the logic is correct:

import sys
import opentimelineio as otio

# Command line arguments.
if len(sys.argv) != 4:
    print("usage: time_warp_check_ocr_results.py (.otio) (ocr baseline) (ocr results)")
    exit(1)

otio_file     = sys.argv[1]
baseline_file = sys.argv[2]
results_file  = sys.argv[3]

# Read the files.
timeline = otio.adapters.read_from_file(otio_file)
with open(baseline_file, 'r') as file:
    baseline = file.read().splitlines()
with open(results_file, 'r') as file:
    results = file.read().splitlines()

# Find each clip with a marker and compare the results frame by frame.
scores = []
for item in timeline.video_tracks()[0]:
    if len(item.markers) > 0:
        itemRange = item.trimmed_range_in_parent()
        matches   = 0
        frameDiff = 0

        for i in range(
            int(itemRange.start_time.value),
            int(itemRange.end_time_exclusive().value)):
            if (baseline[i] == results[i]):
                matches = matches + 1
            if (baseline[i].isdigit() and results[i].isdigit()):
                frameDiff = max(frameDiff, abs(int(baseline[i]) - int(results[i])))

        score = matches / itemRange.duration.value * 100.0
        scores.append([item.markers[0].name, score, frameDiff])

# Print the results.
nameLenMax      = 0
scoreStrs       = []
scoreLenMax     = 0
frameDiffStrs   = []
frameDiffLenMax = 0
scoresSum       = 0.0;
for score in scores:
    nameLenMax = max(nameLenMax, len(score[0]))

    scoreStr    = "{:.2f}%".format(score[1])
    scoreLenMax = max(scoreLenMax, len(scoreStr))
    scoreStrs.append(scoreStr)

    frameDiffStr    = "{}".format(score[2])
    frameDiffLenMax = max(frameDiffLenMax, len(frameDiffStr))
    frameDiffStrs.append(frameDiffStr)

    scoresSum += score[1]
averageScore = scoresSum / len(scores)

print(
    '{s:<{width}}'.format(s="Test", width=nameLenMax + 5),
    "|",
    '{s:<{width}}'.format(s="Score", width=7),
    "|",
    '{s:<{width}}'.format(s="MFD", width=frameDiffLenMax))
print('{:{fill}<{width}}'.format("", width=nameLenMax + frameDiffLenMax + 19, fill='-'))
for i in range(0, len(scores)):
    print(
        '{s:{fill}<{width}}'.format(s=scores[i][0], width=nameLenMax + 5, fill='.'),
        "|",
        scoreStrs[i].rjust(7),
        "|",
        frameDiffStrs[i].rjust(frameDiffLenMax))
print('{:{fill}<{width}}'.format("", width=nameLenMax + frameDiffLenMax + 19, fill='-'))
print("Average score:", "{:.2f}%".format(averageScore))
print()
print("Score = Percentage of frames that match")
print("MFD   = Maximum difference between frames (ignoring blank frames)")
print()

Also toucan now supports writing movie files directly, so the example command line in the README can be simplified a bit:

toucan-render time_warp_test_suite.otio toucan_render.mov

[jminor]

Thanks @darbyjohnston!

I updated the toucan render steps in the README.

Do you mind if I add/adapt your Python script into the test suite? I'd like to include an "Almost Right" detection for results that are within some +/- tolerance.

[darbyjohnston]

Thanks for updating the README and feel free to include/modify the script. Adding a tolerance sounds like a great idea.

[darbyjohnston]

I updated the results and the Python script; there was a bug in the original script not handling newlines, and I made the output more granular so it's not just pass/fail.

The new output shows a percentage for each test and the maximum difference between frames. I think the difference is helpful so you can see how far off the frame values are.

Just a note about the Toucan results, most of the low scores are from the "trim to fill" and "reverse" tests which are issues with the source .otio file (#2, #1).