Add the ability to specify delay or skipped stops on ADDED or REPLACEMENT trips in GTFS-RT

[miklcct]

Summary

This PR adds the ability to support delays or skipped stops on ADDED or REPLACEMENT trips. The time is the actual estimated time of the trip, and the scheduled time is the time given minus the delay given.

Issue

Closes #6009 . The proposal is submitted to google/transit#504 as well.

Unit tests

Unit tests are added for TripUpdates which specify both time and delay in a StopTimeEvent. This case was previously assumed to be invalid and hence a new method is added to generate items. Tests are also added for ADDED trips containing skipped stops as well.

Documentation

The original // TODO: should we incorporate the delay field if present? is done by this PR.

[codecov]

Codecov Report

Attention: Patch coverage is 80.18868% with 21 lines in your changes missing coverage. Please review.

Project coverage is 69.78%. Comparing base (684b841) to head (16c4dc6).
Report is 1 commits behind head on dev-2.x.

Files with missing lines	Patch %	Lines
...rg/opentripplanner/updater/trip/AddedStopTime.java	77.58%	4 Missing and 9 partials ⚠️
...pplanner/updater/trip/TimetableSnapshotSource.java	83.33%	1 Missing and 7 partials ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             dev-2.x    #6028      +/-   ##
=============================================
- Coverage      69.79%   69.78%   -0.01%     
- Complexity     17786    17812      +26     
=============================================
  Files           2017     2017              
  Lines          76042    76107      +65     
  Branches        7781     7789       +8     
=============================================
+ Hits           53073    53115      +42     
- Misses         20264    20276      +12     
- Partials        2705     2716      +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[optionsome]

ADDED and REPLACEMENT updates don't have many users in the OTP community (I think REPLACEMENT is slowly being replaced by new additions to the GTFS specifications, but I'm not fully up-to-date on the status of those drafts), but we are open to these implementations being fixed and/or improved.

I doubt this will break any existing OTP deployments as I think your interpretation of how delay should be used is the only reasonable one. Similarly, I think previously someone might have just left out a stop in an ADDED trip if it was no longer used but having this possibility to mark them as SKIPPED shouldn't negatively affect anyone. If someone is opposed to these changes, let us know.

The only small question mark is that should we wait for some comments from the GTFS community on your suggestion before this is merged or not. Hopefully, the suggestion to the GTFS specification goes through so there is no longer any gray area around these usages.

[miklcct]

REPLACEMENT is officially deprecated, and the behaviour of ADDED remains unspecified. The proposal addresses DUPLICATED (which has a clearly defined behaviour) as well as ADDED.

Google uses ADDED to duplicate an existing trip to run it at another time, which is now officially migrated to DUPLICATED in the standard, while OpenTripPlanner requires specifying the whole trip and allows arbitrary pattern for ADDED trips, which fits the use case to add trips which are not based on existing patterns (which Google doesn't support).

To my understanding there isn't a way in the official GTFS-RT standard to add arbitrary new realtime trips into a route not based on an existing pattern (even Google doesn't support that). Even in the OpenTripPlanner implementation it is impossible to specify headsigns of added trips (both at trip and stop level) but this will be another issue to solve.

GTFS-TripModification proposal can be used for diverted trips but it isn't supported in OpenTripPlanner yet and it is much more complicated to just replacing the whole trip in case of diversion using the deprecated REPLACEMENT feature of OpenTripPlanner.

I am thinking that we should wait for a while to see how the GTFS community progresses on trip additions / diversions / replacement, but I will have a production deployment in the next few months using this change in order to support real time added / diverted trains, which few other commercial providers support (neither Google nor Citymapper could route me via the official diversion to Battersea Park when the line to Clapham Junction was blocked unexpectedly).

Could you please let me know if there are any existing OpenTripPlanner deployments which do handle real-time diverted trips?

[miklcct]

Possibly need further testing for REPLACEMENT trips:

[leonardehrenfried]

We talked about this in today's dev meeting (to which you are welcome). We are generally ok with this interpreation of the spec but would like to see it codified in the spec.

Are you interested in un-deprecating the ADDED schedule relationship in the GTFS spec repo? google/transit#490 is a good start but can you create an actual PR which can be discussed and critiqued line by line by the community? To me the most important change I would like to see is an actually description of what ADDED means. The OTP implementation thinks that it allows you to create completely new trips but I would like to clarify.
If you're okay with trying the spec change I would support you with it.

Since the realtime code is old and has incurred lots of technical debt, we would also like to see some clean up. Are you interested in that?

[miklcct]

I will be interested in cleaning up the code if it means a chance for me to get headsigns, shapes, completely new stops, etc., into the realtime updates but I would like to see how the GTFS-ServiceChanges v3.1 goes first because we commit to anything.

A major limitation of the product we are developing is the inability to specify the headsign for modified trips under the current specification, so I am relying on frontend tweaks to "correct" the headsign for trains terminating short.

[leonardehrenfried]

I also found a thread on Slack about ADDED: https://mobilitydata-io.slack.com/archives/C3D321CKB/p1715957220876199

I think there will be interest in specifying it.

[leonardehrenfried]

Before I review any further: are you going to convert google/transit#490 into a spec PR?

[miklcct]

I plan to do so but not in these few days.

[miklcct]

Reverting this to a draft now until a decision has been made on google/transit#504

[miklcct]

PR updated to fully implement my proposal at google/transit#504

This should remain as draft now as I need to at least wait until #5393 is merged in (for TripOnServiceDate API - this will affect replacement trip headsigns) and the proposal being accepted as an experimental feature.