Frame index broken with simple grabber+recorder

[kvr000]

I have pretty simple loop which grabs images, draws some stuff and writes directly to recorder. The resulting video has issues with frame index for some reason - some periods are fine but then the player is unable to seek or takes long time to go through the video and then skips long period of minutes of tens of minutes.

I disabled any specific options and just H264 for video and tune=film and video rate.

Here is the loop (Scala) :

    val recorder = new FFmpegFrameRecorder(filename, grabber.getImageWidth, grabber.getImageHeight, grabber.getAudioChannels)
    recorder.setFormat("mp4")
    recorder.setOption("c", "copy")
    recorder.setFrameRate(grabber.getFrameRate)

    if (grabber.hasVideo) {
      // Keep the rotation from original video, so all the other data including audio match the orientation
      recorder.setDisplayRotation(grabber.getDisplayRotation)
      recorder.setVideoMetadata(grabber.getVideoMetadata)
      recorder.setVideoCodec(avcodec.AV_CODEC_ID_H264)
      recorder.setVideoBitrate((grabber.getVideoBitrate)
      recorder.setVideoOption("tune", "film")
    }
    if (grabber.hasAudio) {
      recorder.setAudioChannels(grabber.getAudioChannels)
      recorder.setAudioMetadata(grabber.getAudioMetadata)
      recorder.setAudioCodec(grabber.getAudioCodec)
      recorder.setAudioBitrate(grabber.getAudioBitrate)
    }

    while ({ stopped = stopIndicator(); !stopped } && { frame = grabber.grab; frame != null }) {
      if (frame.`type` eq Frame.Type.VIDEO) {
        if (imageType != Java2DFrameConverter.getBufferedImageType(frame)) {
          imageType = Java2DFrameConverter.getBufferedImageType(frame)
          bufferedImage = new BufferedImage(grabber.getImageWidth, grabber.getImageHeight, imageType)
        }
        // Convert OpenCV frame to BufferedImage
        Java2DFrameConverter.copy(frame, bufferedImage)
        // Paint
        painter(frame.timestamp / 1000, bufferedImage, grabber.getDisplayRotation)
        // Convert BufferedImage back to OpenCV frame
        val modifiedFrame = frame.clone
        Java2DFrameConverter.copy(bufferedImage, modifiedFrame)
        // Record the modified frame
        recorder.record(modifiedFrame)
      }
      else {
        recorder.record(frame)
      }
    }
    recorder.flush()
    recorder.close()
    grabber.close()

When I recode the video with ffmpeg from command line, it works without problems, so the code must be somehow JavaCv specific (or possibly in my code but this looks pretty straightforward).

[saudet]

You'll need to call stop() and/or close() somewhere to get the index written

[kvr000]

@saudet : Thanks. I provided only the copy loop but there are flush + close in original code. I updated the snippet in the comment.

[saudet]

Maybe the format you're using expects a timestamp for each frame?

[kvr000]

@saudet : Ok, I included also the initialization in the snippet. It's pretty simple H264. Originally, I also had threads (which didn't work) but no matter what are the settings, the result is always the same. I also tried removing c:copy, with no improvement.
So I expect the timestamp to be taken automatically from the original cloned frame. Even when I removed cloning and fully reused the original frame, it still didn't work.
I tested yesterday on 2 minutes video - first 92 seconds were ok, the last half minute was broken.

[saudet]

You're not giving it any audio frames, that's not going to work

[saudet]

Oh, I see you are giving it audio frames, but maybe they are too far apart from the video frames. Typical formats can support up to about 1 second delay between them, so you have to make sure that's what they end up like.

[kvr000]

@saudet : It consumes the original video as is, and within the loop - it either amends the image for VIDEO frame or records the frame directly for any other type. So the frame are stored in the same order as in the original video. The original video works without issues.
I may prepare minimalistic proof-of-concept if that helps.

[saudet]

Like I said, you might need to set the timestamp, so please try to set it

[kvr000]

@saudet : I did add recorder.setTimestamp(grabber.getTimestamp) as the first statement in while loop and it behaves the same.

[kvr000]

@saudet : Please check the proof-of-concept demo at https://github.com/kvr000/zbynek-javacv-poc/tree/main/javacv-broken-index/