UDP_Remote_Protocol.md

UDP Remote Protocol

Overview

Control Commands

Sharktopoda 2 will support a remote protocol that will allow other applications to send commands to it via UDP. The protocol will support the following control commands:

• Connect
• Open
• Close
• Show
• Request information
• Request all information
• Play
• Pause
• Request elapsed time
• Request player state
• Seek elapsed time
• Frame advance
• Frame capture
• Ping

Localization Commands

In addition to the control commands, the remote protocol will also support commands for managing information about localizations, aka rectangular regions of interest, displayed over video during playback.

Localizations

• Add localizations
• Remove localizations
• Update localizations
• Clear localizations
• Select localizations

Command-Response Pattern

sequenceDiagram
    autonumber
    participant R as Remote App
    participant S as Sharktopoda
    
    R->>+S: {"command": "some command", ...}
    S-->>-R: {"response": "some command" ...}

Loading

Command Message Failures

All commands are expected to be valid JSON messages as per the individual command descriptions herein.

Invalid JSON command values will be reported as:

{
  "command": <request command>,
  "status": "failed",
  "cause": <failure cause>
}

Invalid JSON message structure will be reported as:

{
  "command": <request command>,
  "status": "failed",
  "cause": "Invalid message"
}

NOTE: Sharktopoda does not determine or report why the message structure was invalid. It is expected the developer of the control messaging app will consult these requirements to determine the actual cause.

Client Commands

Sharktopoda can also send certain commands to the Remote App. These commands are explicitly sent to the host/port established by a preceding connect control command. The amount of time to wait for a response (i.e. timeout) will be set in the preferences UI. These commands are:

• Add localizations
• Remove localizations
• Update localizations
• Select localizations
• Ping

sequenceDiagram
    autonumber
    participant R as Remote App
    participant S as Sharktopoda
    
    S->>+R: {"command": "some command", ...}
    R-->>S: {"response": "some command" ...}

Loading

Commands Accepted via the UDP port

Sharktopoda will receive JSON messages and respond with JSON via the UDP port configured under Preferences. NOTE: The messages below are pretty formatted for clarity, they do not need to be in the application. It's recommended that the messages are minified. The order of the JSON fields is not important. The maximum message size is 4096 bytes and should be encoded as UTF-8.

The application should support the following commands and corresponding functions:

Back

Connect

Establishes a remote host and port number that Sharktopoda (the video player) can send outgoing UDP messages to another application. When a connect command is received, Sharktopoda should send a ping command to verify that the port is reachable.

sequenceDiagram 
   participant R as Remote App
   participant S as Sharktopoda

   Note over R,S: connect sent to Sharktopoda's UDP port
   R->>+S: {"command": "connect", "port": <port number>}
   S-->>-R: {"response": "connect", "status": "ok"}

   Note over R,S: ping sent to <port number> to verify it is open.
   S->>+R: {"command": "ping"}
   alt Ping timed out
     S->>S: Display error dialog
   else Ping succeeded
     R-->>-S: {"response": "ping"}
     S->>S: Set port/host in memory for later use
     Note over R,S: Use port/host for outgoing commands, e.g.:
     S->>+R: {"command": "add localizations", ...}
     R-->>-S: {"response": "add localizations", ...}
   end

Loading

There are 2 forms of this message. The first form omits the host field; Sharktopoda assumes that the host is localhost.

{
  "command": "connect",
  "port": 8095
}

The second form explicitly specifies the host:

{
  "command": "connect",
  "port": 8095,
  "host": "some.server.org"
}

For both message formats, it should always respond with an ok message:

{
  "response": "connect",
  "status": "ok"
}

Back

Open

Opens the specified video in a new window. The application should associate the url and uuid with the window. (More on that later). If a new window is opened, the video should be immediately paused. (We don't want the video to autoplay when it is first opened)

If a window with the uuid already exits, treat the open command as show command below, and a success response returned.

sequenceDiagram
    participant R as Remote App
    participant S as Sharktopoda
    participant V as Open Videos

    R->>+S: {"command": "open", ...}
    S->V: Check if UUID already exists
    alt UUID exists
      S->>V: Bring window to front/focus
      S-->>R: {"response": "open", "status": "ok"} 
    else UUID is does not exist
      S-)V: Open video and association UUID with window
      alt Successfully opened video
        S-->>R: {"response": "open", "status": "ok"} 
      else Failed to open videoN
        S-->>-R: {"response": "open", "status": "failed"} 
      end
    end

Loading

Open URL

{
  "command": "open",
  "uuid": "b52cf7f1-e19c-40ba-b176-a7e479a3b170",
  "url": "http://someurl/at/moviefile.mov"
}

Open File (using file URL)

{
  "command": "open",
  "uuid": "b52cf7f1-e19c-40ba-b176-a7e479a3b170",
  "url": "file:/somefile/at/moviefile.mp4"
}

Each form of the open command receives an immediate an ok message response.

{
  "response": "open",
  "status": "ok"
}

Sharktopoda will proceed with processing the command on a background thread. Upon completion of background processing, Sharktopoda shall send an open done message to the host/port established via a connect command:

Successfully opened video response

{
  "response": "open done",
  "uuid": "b52cf7f1-e19c-40ba-b176-a7e479a3b170",
  "status": "ok"
}

Failed to open video response

{
  "cause": <cause>,
  "response": "open done",
  "status": "failed"
}

Back

Close

It should close the window with the corresponding uuid:

{
  "command": "close",
  "uuid": "b52cf7f1-e19c-40ba-b176-a7e479a3b170"
}

Close should respond with an status **failewd"" if no window with a matching uuid is found:

{
  "cause":"No video for uuid",
  "response": "close",
  "status": "failed"
}

Back

Show

Focuses the window containing the video with the given uuid and brings it to the front of all open Sharktopoda windows. Some UI toolkits do not grab focus if the app is not already focused. In that case, simply bring the window to the front of the other open Sharktopoda windows.

{
  "command": "show",
  "uuid": "b52cf7f1-e19c-40ba-b176-a7e479a3b170"
}

Show should respond with an ack:

{
  "response": "show",
  "status": "ok"
}

If the window with uuid does not exist it should respond with

{
  "response": "show",
  "status": "failed",
  "cause": ""No video for uuid"
}

Back

Request Video Information for the focused or top most (in Z- order) Window

{"command": "request information"}

It should return the uuid and url of the current or last focused window as well as the length of the video in milliseconds (named as durationMillis) and the frameRate of the video file. The isKey field indicates if the window is currently handling user keyboard input.

{
  "response": "request information",
  "status": "ok",
  "uuid": "b52cf7f1-e19c-40ba-b176-a7e479a3b170",
  "url": "http://someurl/and/moviefile.mov",
  "durationMillis": 150000,
  "frameRate": 29.97,
  "isKey": true
}

If no video windows are currently available (i.e., either no successful open commands or all previously opened videos have been closed), it should respond with

{
  "response": "request information",
  "status": "failed",
  "cause": "No open videos"
}

Back

Request information for all open videos

{"command": "request all information"}

It should return info for all open videos like the following:

{
  "response": "request all information",
  "status": "ok",
  "videos": [
    {
      "uuid": "b52cf7f1-e19c-40ba-b176-a7e479a3b170",
      "url": "http://someurl/and/moviefile.mov"
      "durationMillis": 150000,
      "frameRate": 29.97,
      "isKey": false
    },
    {
      "uuid": "b52cf7f1-e19c-40ba-b176-a7e479a3b170",
      "url": "file://sometoherurl/and/moviefile.mp4"
      "durationMillis": 250300,
      "frameRate": 59.97,
      "isKey": true
    }
  ]
}

If no currently available video windows, it should respond the same as request information.

Back

Play

Play the video associated with the uuid. The play rate will be 1.0 which is normal playback speed.

{
  "command": "play",
  "uuid": "b52cf7f1-e19c-40ba-b176-a7e479a3b170"
}

Optionally the play command can contain a rate for the playback. A positive rate is forward, negative is reverse. 1.0 is normal speed, 2.0 is twice normal speed. -0.5 is half speed in reverse.

{
  "command": "play",
  "uuid": "b52cf7f1-e19c-40ba-b176-a7e479a3b170",
  "rate": -2.4
}

It should respond with:

{
  "response": "play",
  "status":"ok"
}

or

{
  "response": "play",
  "status": "failed",
  "cause": <cause>
}

Back

Pause

Pauses the playback for the video specified by the uuid:

{
  "command": "pause",
  "uuid": "b52cf7f1-e19c-40ba-b176-a7e479a3b170"
}

It should respond with:

{
  "response": "pause",
  "status":"ok"
}

or, in the case of failure, such as the requested video uuid does not exist:

{
  "response": "pause",
  "status": "failed",
  "cause": <cause>
}

Back

Request elapsed time

Return currently viewed moment of the video, i.e. the elapsed time (from the start) of the video as milliseconds.

{
  "command": "request elapsed time",
  "uuid": "b52cf7f1-e19c-40ba-b176-a7e479a3b170"
}

It should respond with:

{
  "response": "request elapsed time",
  "elapsedTimeMillis": 12345,
  "status": "ok"
}

or the following in the uuid does not exist:

{
  "response": "request elapsed time",
  "status": "failed",
  "cause": <cause>
}

Back

Request Player State

Return the current playback state of the video (by uuid) and the actual rate that the video is playing. Possible states are: shuttling forward, shuttling reverse, paused, playing.

• playing is when the video is playing at a rate of 1.0
• shuttling forward is when the video is playing with a positive rate that is not equal to 1.0
• shuttling reverse is when the video is playing with a negative rate.
• paused is obvious. (Not playing)

{
  "command": "request player state", 
  "uuid": "b52cf7f1-e19c-40ba-b176-a7e479a3b170"
}

An example response is:

{
  "response": "request player state", 
  "status": "playing",
  "rate": 1.0
}

or a failed response if the uuid does not exist:

{
  "response": "request player state", 
  "status": "failed",
  "cause": <cause>
}

Back

Seek Elapsed Time

Seek to the provided elapsed time (which will be in milliseconds)

{
  "command": "seek elapsed time",
  "uuid": "b52cf7f1-e19c-40ba-b176-a7e479a3b170",
  "elapsedTimeMillis": 12345
}

Seek should respond with an ok message:

{
  "response": "seek elapsed time",
  "status": "ok"
}

or the following in the uuid does not exist or the elapsedTimeMillis is before/after the videos start/end:

{
  "response": "seek elapsed time",
  "status": "failed",
  "cause": <cause>
}

Back

Frame advance

Advance or regress one frame for the given video. If the direction field is positive (i.e. 1) the the video should be advance one frame. If the direction is negative (-1), then the video should go back one frame. The command is:

{
  "command": "frame advance",
  "uuid": "cb5cf7f1-e19c-40ba-b176-a7e479a3cdef",
  "direction": 1
}

Frame advance should respond with an ok message:

{
  "response": "frame advance",
  "status": "ok"
}

or the following in the uuid does not exist:

{
  "response": "frame advance",
  "status": "failed",
  "cause": "No video for uuid"
}

Back

Frame capture

The frame capture command specifies the local image file path as well as a reference id for the image (used by the calling controller, not Sharktopoda).

{
  "command": "frame capture",
  "uuid": "b52cf7f1-e19c-40ba-b176-a7e479a3b170",
  "imageLocation": "/Some/path/to/save/image.png",
  "imageReferenceUuid": "aa4cf7f1-e19c-40ba-b176-a7e479a3cdef"
}

Sharktopoda should grab the current elapsed time for the specified uuid and respond immediately.

{
  "response": "frame capture",
  "status": "ok"
}

A frame capture request shall report errors as:

{
  "response": "frame capture",
  "status": "failed",
  "cause": <cause>
}

Errors include:

• Video for uuid not open
• Cannot write image to specified location
• Image at specified location exists
• Specified image location is a malformed file path
• Process fails to create a PNG image from capture

Upon sending an ok response message, Sharktopoda will proceed with frame capture processing on a background thread.

Upon completion of background frame capture processing, Sharktopoda shall use the host/port established with a prior connect command to send the response:

{
  "command": "frame capture done",
  "elapsedTimeMillis": 12345,
  "imageReferenceUuid": "aa4cf7f1-e19c-40ba-b176-a7e479a3cdef",
  "imageLocation": "/Some/path/to/save/image.png",
  "status": "ok",
  "uuid": "b52cf7f1-e19c-40ba-b176-a7e479a3b170"
}

If an error occurres in frame capture processing, Sharktopoda shall send a frame capture done response:

{
  "cause": <error message>,
  "command": "frame capture done",
  "imageReferenceUuid": "aa4cf7f1-e19c-40ba-b176-a7e479a3cdef",
  "imageLocation": "/Some/path/to/save/image.png",
  "status": "failed",
  "uuid": "b52cf7f1-e19c-40ba-b176-a7e479a3b170"
}

The frame capture processing flow is as follows:

sequenceDiagram 
    participant R as Remote App
    participant S as Sharktopoda
    participant V as Open Videos
    participant disk as Local Disk

    Note over R,S: Set up. This happens when remote app starts
    R->>+S: {"command": "connect", "port": 8899}
    S-->>-R: {"response": "connect", "status": "ok" }

    S->>S: Ready to send messages to Remote App via port 8899

    Note over R,S: Frame capture sequence
    R->>+S: {"command": "frame capture", ... }
    S->>V: find video by UUID
    alt video UUID does not exist
      S-->>R: {"response": "frame capture", "status": "failed" }
    else video UUID exists
      S-->>-R: {"response": "frame capture", "status": "ok" }
      Note over S,disk: On a separate thread. Don't block incoming UDP commands
      S->>V: Capture image and elapsed time into video
      S->>disk: write lossless PNG to disk at image_location
      alt Unable to write PNG to image_location
        S->>+R: {"command": "frame capture done", "status": "failed", ...}
      else write PNG was successful
        S->>R: {"command": "frame capture done", "status": "ok", ...}
      end
      R-->>-S: {"response": "frame capture done", "status": "ok: }
    end

Loading

Back

Ping

This command checks the Sharktopoda server is responding on the Preferences specfied port:

{
  "command": "ping"
}

{
  "response": "ping",
  "status": "ok"
}

Localizations

A localization defines a rectangular region of interest on the video. Users should be able to draw these regions directly on a video window of the Sharktopoda app. Sharktopoda will, in turn, notify the remote app that a new localization has been created. Sharktopoda needs to be able to handle 10,000s of localizations in a video and have them drawn on the correct frames as the video is playing, shuttling, etc. A Localization is also called an Annotation and has the following properties:

• uuid - The unique identifier for an annotation. UUID v4 (random) is recommended.
• concept - The label associated with a localization that identifies the object in the region of interest. In theory, the concept can be up to 256 characters long, but in practice it is much shorter.
• elapsedTimeMillis - The elapsed time from the start of the video that the localization is to be displayed.
• durationMillis - This field may be present but can be ignored for now. It represents how long the localization is valid. It will span from elapsedTimeMillis to elapsedTimeMillis + durationMillis. The default is 0 which means the localization is valid for a single frame.
• x - The x coordinate of the localization in pixels.
• y - The y coordinate of the localization in pixels.
• width - The width of the localization in pixels.
• height - The height of the localization in pixels.
• color - The color used to draw the localization.

x, y, width, and height are in the same coordinates as the unscaled video. Localizations use an "ocean" coordinate system where origin is the upper-left, +X is right, +Y is down:

Localizations can be added, selected, deleted, or modified from either a remote app or from Sharktopoda. If a localization is created/mutated in Sharktopoda, it will notify the remote app using UDP via the port defined by the connect command.

Incoming messages will be no larger than 4096 bytes. In practice, the remote application will not send more than 10 localizations to Sharktopoda in a single Add or Update message.

Back

Add Localizations

The initiating app (both sharktopoda and the remote app can create localizations) will send a notification of a new localizations to the other app.

{
  "command": "add localizations",
  "uuid": "<the video's uuid>",
  "localizations": [
    {
      "uuid": "<uuid unique to this localization>",
      "concept": "Bathybembix bairdii",
      "elapsedTimeMillis": 49211,
      "durationMillis": 25, // optional, default is 0;
      "x": 1076,
      "y": 13,
      "width": 623,
      "height": 475,
      "color": "#FFDDDD" // optional field. if unspecified, the default is from the Annotation preferences pane
    }
  ]
}

The receiving app should respond with an ok message:

{
  "response": "add localizations",
  "status": "ok"
}

or a failure if the video with uuid does not exist:

{
  "response": "add localizations",
  "status": "failed",
  "cause": <cause>
}

Back

Remove Localizatons

The initiating app will send a notification of localizations to be deleted.

{
  "commmand": "remove localizations",
  "uuid": "<the video's uuid>",
  "localizations": [
    "<uuid for localization A>",
    "<uuid for localization B>"
  ]
}

The receiving app will respond with an ok message:

{
  "response": "remove localizations",
  "status": "ok"
}

or a failure if the video with uuid does not exist:

{
  "response": "remove localizations",
  "status": "failed",
  "cause": <cause>
}

Back

Update Localizations

Update existing localizations in Sharktopoda. If a matching localization's uuid does not already exist in Sharktopoda, ignore that localization. (i.e. do not add, do not update).

NOTE: The Sharktopoda app provides for modification to localization x, y, width, and height values via mouse dragging of a localization region (to either move or resize the region). All other fields may be updated from the Remote App with the exception of the localization uuid value, which is used to identify the localization itself and is therefore immutable.

{
  "command": "update localizations",
  "uuid": "<the video's uuid>",
  "localizations": [
    {
      "uuid": "<uuid unique to this localization>",
      "concept": "Bathybembix bairdii",
      "elapsedTimeMillis": 49211,
      "durationMillis": 25, // optional, default is 0;
      "x": 1076,
      "y": 13,
      "width": 623,
      "height": 475,
      "color": "#FFDDDD" // optional field. if unspecified, the default is from the Annotation preferences pane
    }
  ]
}

The receiving app will respond with an ok message:

{
  "response": "update localizations",
  "status": "ok"
}

or a failure if the video with uuid does not exist:

{
  "response": "update localizations",
  "status": "failed",
  "cause": <cause>
}

Back

Clear Localizations

This will only be sent from the remote app to Sharktopoda (not vice versa). Sharktopoda should remove all cached information about the localizations for a given video.

{
  "command": "clear localizations",
  "uuid": "<the video's uuid>"
}

Sharktopoda will respond with an ok message:

{
  "response": "clear localizations",
  "status": "ok"
}

or a failure if the video with uuid does not exist:

{
  "response": "clear localizations",
  "status": "failed",
  "cause": <cause>
}

Back

Select Localizations

This indicates which localizations are selected. Selected localizations should be drawn in the selected color specified in app Preferences and display the corresponding concept field of the localiztion. If only a single localization is selected, that localization should become editable and be able to be moved and resized. When a select command is received, all previously selected annotations should no longer be selected and should be drawn using their original or default color. Any localization uuids that do not exist in Sharktopoda should be ignored.

{
  "command": "select localizations"
  "uuid": "<the video's uuid>",
  "localizations": [
    "<uuid for localization A>",
    "<uuid for localization B>"
  ]
}

{
  "response": "select localizations",
  "status": "ok"
}

or a failure if the video with uuid does not exist:

{
  "response": "select localizations",
  "status": "failed"
}