Specify the behaviour of TripUpdate.schedule_relationship = ADDED, and un-deprecate REPLACEMENT

[miklcct]

The use of TripUpdate.schedule_relationship = ADDED was unspecified and different producers / consumers used it in different ways. For example, it is sometimes used to specify additional departures on an existing route, but it is also used to specify departures which can't be matched to any existing trips.

This PR attempts to specify the behaviour of ADDED, and un-deprecate REPLACEMENT, based on the implementation of OpenTripPlanner which specifies the whole journey to be added or replaced. Additional fields, such as headsigns, and pickup / drop-off types, are introduced as required to support the full specification of completely new trips.

ADDED

In this proposal, TripUpdate.schedule_relationship = ADDED should be used to add trips which do not duplicate an existing trip. Such trips are considered to be unrelated to any existing trips in the GTFS Static and can serve an arbitrary pattern, including completely new patterns not found in the GTFS Static.

A typical use case is for relief trips for extra demand, typically after big events.

As with the current OpenTripPlanner implementation, trip_id in the TripDescriptor for added trips must be completely new (not found in GTFS static) and unique, and a start_date should also be specified as well (I am not using the word "must" here because it is permitted not to specify start_date to match scheduled trips, in this case the trip is assumed to run today).

The whole journey of the added trip must be specified, in stop order, as StopTimeUpdates inside the TripUpdate without any omission. Fields are added to TripProperties and StopTimeProperties for esssential information such as names, headsigns, pickup / drop off types.

REPLACEMENT

I propose to un-deprecate TripUpdate.schedule_relationship = REPLACEMENT as well. It works in the same way as ADDED, apart from that the TripDescriptor must match one instance of a scheduled trip (like other values of TripUpdate.schedule_relationship), and that instance is replaced with the complete replacement trip specified in form of StopTimeUpdates like an added trip. The original stop times in the GTFS static are not considered by the replacement trip in any form to avoid confusion. The replacement trip can serve an arbitrary pattern with an arbitrary schedule, the only expectation is that the passenger should associate the replacement trip to actually be a replacement of the original trip.

A typical use case is for short-term timetable change, or short-term (near real-time) diversion, where the fact that the trip_id remains the same can be used by journey planners to notify the user that the booked service has been changed. (In particular, I have successfully used this feature to handle real-time train diversions in GB in OpenTripPlanner and route users to alight at diverted stops, which is something neither Google Maps nor Citymapper can do now)

This is the behaviour implemented in OpenTripPlanner, which is equivalent to deleting the matched trip, and processing the replacement TripUpdate as an ADDED trip mentioned above.

Relationship to TripModification

TripModification provides a way to modify trips en-masse by specifying a list of trip IDs where the same detour can be applied. However, it is not suited to change the schedule on a per-trip basis, replacing the trip with a completely different schedule after any diversions with different running times (common due to pathing constraints on railways).

It should be forbidden to modify the same trip via a REPLACEMENT trip update and also via a TripModification.

[gcamp]

Nice to see some movement in that direction!

I think you used a markdown editor that changed formatting on a lot of tables which makes it hard to see the actual diff from your proposal. Would it be possible to fix that?

You put a lot in the PR description that's not actually in the proposed changes. Is that just to start the discussion? Some of it is quite consequential, like the whole journey of the added trip must be specified.

I'm a bit puzzled on how a consumer is supposed to ingest ADDED changes like this with arbitrary trips with no more information than an headsign. Which route is that on? Is those added trips supported only on existing routes in the GTFS? If the answer is no, we're getting quite close to the service change proposal : https://bit.ly/gtfs-service-changes-v3_1

[leonardehrenfried]

Thanks for opening this PR!

OTP has had an implementation of ADDED for a long time but its behaviour is severely underspecified. I'd love to formalise it.

Yes, OTP allows you to create completely new free form trips that have no relation to an existing pattern or trip. It tries match the given route id to an existing one but if none is in the message a dummy one is created. For once, OTP is really following the "just give us what you have, and we will try to work it out" strategy.

The only requirement we have is that the stop ids must match the static GTFS. The question is what should happen when they don't. Should the entire update be dropped or individual stops? Does that even need to be specified?

I agree with what @gcamp said about the markdown tables and the issue description.

Lastly, you might find it easier to get this through review if you split it into two PRs: one for ADDED and one for REPLACEMENT. That's just a guess though.

[miklcct]

I think that the requirement for the whole trip to be specified is written in the code. Let me know if it is not clear enough.

I'll fix the formatting later today.

[miklcct]

Nice to see some movement in that direction!

I think you used a markdown editor that changed formatting on a lot of tables which makes it hard to see the actual diff from your proposal. Would it be possible to fix that?

You put a lot in the PR description that's not actually in the proposed changes. Is that just to start the discussion? Some of it is quite consequential, like the whole journey of the added trip must be specified.

I'm a bit puzzled on how a consumer is supposed to ingest ADDED changes like this with arbitrary trips with no more information than an headsign. Which route is that on? Is those added trips supported only on existing routes in the GTFS? If the answer is no, we're getting quite close to the service change proposal : https://bit.ly/gtfs-service-changes-v3_1

"The whole journey of the added trip must be specified" is a fact in the core of this PR, noted in the updated definition of StopTimeUpdate:

Updates to StopTimes for the trip (both future, i.e., predictions, and in some cases, past ones, i.e., those that already happened). The updates must be sorted by stop_sequence, and apply for all the following stops of the trip up to the next specified stop_time_update.
If trip.schedule_relationship is SCHEDULED, at least one stop_time_update must be provided for the trip.
If trip.schedule_relationship is ADDED or REPLACEMENT, stop_time_updates must be provided for all stops in the added or replacement trip, and the stop times in the static GTFS are not used.
If the trip is canceled or deleted, no stop_time_updates need to be provided. If stop_time_updates are provided for a canceled or deleted trip then the trip.schedule_relationship takes precedence over any stop_time_updates and their associated schedule_relationship. If the trip is duplicated, stop_time_updates may be provided to indicate real-time information for the new trip.

The route and direction of the ADDED trip is specified in TripDescriptor.route_id. Sorry I didn't make it clear and I'll going to refine the PR. It should not be possible to replace a trip to work on a different route in a REPLACEMENT trip as it may confuse consumers.

[miklcct]

I do not want to specify the behaviour of missing stops at this moment because it may depend on the client's capability for dynamically adding stops via Stop messages. Theoretically the stop_id must refer to a stop in GTFS static, or a stop added via Stop messages.

[leonardehrenfried]

So this is pretty much a codifcation of what OTP has been supporting for several years. This would of course be very convenient for us but I would like to hear more voices from the community, in particular producers.

I know that HSL (Helsinki) is using this as both a producer and consumer (OTP) for many years.

MBTA has also indicated that they use ADDED as a producer.

@optionsome @jfabi @sam-hickey-ibigroup

[skinkie]

So this is pretty much a codifcation of what OTP has been supporting for several years. This would of course be very convenient for us but I would like to hear more voices from the community, in particular producers.

Producing it for over 12 years too.

[optionsome]

I know that HSL (Helsinki) is using this as both a producer and consumer (OTP) for many years.

HSL doesn't produce or consume ADDED or REPLACEMENT updates currently, if that was what you were referring to.

[leonardehrenfried]

What happens when you ADD a trip and then CANCEL it again? Should it be become invisible in the system (DELETED?) or show up as a CANCELLED?

[miklcct]

That's a good question. I still need to think about how things will work.

My producer implementation cancels an added trip using TripUpdate.schedule_relationship = ADDED with all StopTimeUpdate having a SKIPPED relationship.

The questions are that:

1. What if a later version of the full dataset real time feed doesn't contain the ADDED feed? (My intention is that it no longer exists and should be considered as DELETED. A GTFS-RT full dataset should only be applied into the original static data.)
2. How do I cancel an ADDED trip? I think that a TripUpdate with a trip id not found in static GTFS, schedule_relationship = CANCELED and the original planned stops marked with StopTimeUpdate.schedule_relationship = 'SKIPPED' makes it clear that the trip was added then cancelled. (Use DELETED instead of CANCELED to hide it from the board)

[skinkie]

As we are considering FULL_DATASET shouldn't that just replace?

[leonardehrenfried]

Actually, @skinkie is right. If you use FULL_DATASET then the moment you fetch the new version of the RT feed the old ADDED trip will completely vanish and it neither exists as DELETED nor CANCELLED. It's like it never existed.

However, once there is movement towards specifying INCREMENTAL we will have to revisit this.

[leonardehrenfried]

Does anyone know how Google and Apple handle ADDED?

@bdferris-v2

@eliasmbd I don't know who the relevant person from Apple would be. Could you tag them?

[skinkie]

@leonardehrenfried at Google the thing was limited to stop sequences previously observed. Hence if the ADDED trip was an instance of a stop sequence that is part of the database, it could be processed. I don't know if it is already capable of processing a partial instance of a stop sequence.

https://support.google.com/transitpartners/answer/10106497?hl=en#zippy=%2Cadd-with-tripupdates

[tzujenchanmbd]

Recently we’ve heard from the community about the need for best practices regarding when to use static vs. realtime data. The changes included in this PR (such as adding new trips in real-time) seem to underscore the importance of such guidance. While the goal of ADDED is to address "unscheduled trips", could it be that producers over-rely on ADDED and use it in situations where static data updates should be applied?

(Recent Issue #512, while not directly related to this proposal, illustrates a case of "misuse" of realtime data. The Caltrans report pointed out that some producers rely on service alerts to present "scheduled" holiday services.)

The currently proposed wording for ADDED is: "An extra trip unrelated to any existing trips, for example, to respond to sudden passenger load."

Curious if the community is interested in further discussing here the timing of using ADDED versus updating trips in static data and adding details on best practices? (Issue #113 seems to have had relevant discussions on this.)

[Arilith]

Personally I'm greatly in favour of developing the specification for both ADDED and REPLACEMENT trips further. Without the current implementation in OTP for those relations, we (in The Netherlands) would not be able to correctly utilise GTFS-RT for realtime information for our rail network.

Maybe a few examples of our current use-cases will help shed some light on the situation:

Currently, for just the next three days there's 124 trips that could make use of the ADDED schedulerelationship, and 1235! REPLACEMENT trips. These are mostly trips where one journey has been split into two separate parts due to a disruption along the route.

E.g. Journey 1 is Station A - B - C - D - E - F normally.
There's a big disruption at station C.
Journey 1 is modified using SCHEDULED (or REPLACEMENT in the case of platform changes) into Journey 1a with the original trip number and halting at stations A - B and using SKIPPED for C, D, E, F.
Journey 1b is created using ADDED matching onto the original route with a new trip_id, but only halting at stations D - E - F. Using SKIPPED for A - B - C.

This is an extremely common use case in The Netherlands ands happens tens to hundreds of times per day.

Besides, the ADDED relationship is also relatively commonly used for rail replacement services that were unplanned (e.g. busses during extremely big disruptions)

Additionally, we also make great use of the REPLACEMENT trips for indicating platform changes or small diversions. Without using both ADDED and / or REPLACEMENT, platform changes would not work in our current implementation of OpenTripPlanner.

See this file of my GTFS-RT generating code to get a quick overview of some of the hoops we jump through to get GTFS-RT clients to accept 99.95+% of our raw train journey updates.

[miklcct]

Recently we’ve heard from the community about the need for best practices regarding when to use static vs. realtime data. The changes included in this PR (such as adding new trips in real-time) seem to underscore the importance of such guidance. While the goal of ADDED is to address "unscheduled trips", could it be that producers over-rely on ADDED and use it in situations where static data updates should be applied?

(Recent Issue #512, while not directly related to this proposal, illustrates a case of "misuse" of realtime data. The Caltrans report pointed out that some producers rely on service alerts to present "scheduled" holiday services.)

The currently proposed wording for ADDED is: "An extra trip unrelated to any existing trips, for example, to respond to sudden passenger load."

Curious if the community is interested in further discussing here the timing of using ADDED versus updating trips in static data and adding details on best practices? (Issue #113 seems to have had relevant discussions on this.)

From my understanding, changes known at least a week in advance should be put in the static, and a day or less should be in the RT only. Can someone please point me out if this is recorded in the best practices?

For example, there is a landslide and an emergency timetable has been applied as a result. This is clearly a RT scenario and the emergency timetable should be ADDED with the original DELETED. However, in the case of non-urgent track defect found on Monday and a closure is decided to take place on the coming Sunday, a new static file should be published to give advance information to passengers.

[Sergiodero]

Hi @miklcct!

We noticed the announcement in the GTFS-Realtime Google Group regarding the intention to open a vote next week. While we appreciate the effort and initiative to advance the proposal, from our end at MobilityData we would like to recommend postponing the vote until at least after the holiday season.

Opening the vote now might result in lower participation and fewer reviews, as many community members may be away or have limited availability during this time. We believe this timing adjustment could help ensure a more inclusive and robust decision-making process.

On a related note, could you please clarify which organizations are acting as the first consumer and first producer for this proposal (and the person who represents each of them)? Clarifying this could help confirm the status of the proposal before opening a vote and ensure the change is well-supported by the community. Typically, these roles are fulfilled by different individuals or organizations.

[skinkie]

@Arilith I think we can act as both consumer and producer right?

[leonardehrenfried]

While it probably works to use REPLACEMENT for platform changes, I think it's worth pointing out that there is also assigned_stop_id to model it without having to replace the entire trip.

[skinkie]

While it probably works to use REPLACEMENT for platform changes, I think it's worth pointing out that there is also assigned_stop_id to model it without having to replace the entire trip.

Worth mentioning that for example for the repository owner a platform change historically only works if the replacement platform has an equal parent_station.

[miklcct]

While it probably works to use REPLACEMENT for platform changes, I think it's worth pointing out that there is also assigned_stop_id to model it without having to replace the entire trip.

Worth mentioning that for example for the repository owner a platform change historically only works if the replacement platform has an equal parent_station.

This requirement has since been dropped, with the newest requirement that it shouldn't result in a significant change in journey experience for the passenger (i.e. the passenger should treat that as a day-to-day operation detail rather than something not expected normally).

[skinkie]

This requirement has since been dropped, with the newest requirement that it shouldn't result in a significant change in journey experience for the passenger (i.e. the passenger should treat that as a day-to-day operation detail rather than something not expected normally).

The standard dropped it or did the consumer implement it?

[miklcct]

There is no longer a strict requirement that the assigned_stop_id must be in the same station in the standard.

[miklcct]

While it probably works to use REPLACEMENT for platform changes, I think it's worth pointing out that there is also assigned_stop_id to model it without having to replace the entire trip.

This is something OpenTripPlanner doesn't support yet, and I plan to work on it if I have spare time.

[miklcct]

I am now bringing this proposal to a vote, on behalf of Aubin MaaS Limited. The voting period will end at 2025-01-15 23:59:59 UTC.

This proposal has been implemented in opentripplanner/OpenTripPlanner#6028 and is deployed on https://test.open-trip-planner.jnction.co.uk/ for National Rail network in Great Britain.

The data we are using can be downloaded below:
GTFS static: https://dev.aubin.app/gtfs/great_britain_gtfs.zip
Real-time feed: https://dev.aubin.app/national_rail_gtfsrt.binpb

[doconnoronca]

Where can the GTFS real time feeds that implement this be found?

[miklcct]

Where can the GTFS real time feeds that implement this be found?

We are still working on the pipeline for the data production, however, I have made a temporary arrangement to make it available for reference (it is for development purpose only - the real feed is generated locally every 15 minutes on the OTP server but I am copying it using a cron job every minute)

GTFS static: https://dev.aubin.app/gtfs/great_britain_gtfs.zip
Real-time feed: https://dev.aubin.app/national_rail_gtfsrt.binpb

[tzujenchanmbd]

According to the current governance process:

Before calling for a vote, at least one GTFS-realtime producer and one GTFS-realtime consumer should implement the proposed change. It is expected that the GTFS-realtime producer(s) include the change in a public-facing GTFS-realtime feed and provide a link to that data within the pull request comments, and that the GTFS-realtime consumer(s) provides a link in the pull request comments to an application that is utilizing the change in a non-trivial manner (i.e, it is supporting new or improved functionality).

I would like to confirm the producer and consumer supporting this vote:

• Producer: Aubin MaaS Limited
• Consumer: OpenTripPlanner (@leonardehrenfried could you help confirm the OTP link?)

@miklcct is this correct?

Additionally, since initiating a vote should include data, I suggest everyone vote after the data published.

[skinkie]

You can add a few more.
Producer: OVapi / OpenGeo, R-OV
Consumer: Bliksem Labs, R-OV

https://gtfs.ovapi.nl/nl/trainUpdates.pb

[miklcct]

Yes that's correct.

[Arilith]

As Stefan mentioned R-OV is both a consumer as producer (also with ADDED and REPLACEMENT in use):

https://gtfs-rt.r-ov.nl/trainUpdates.pb

And supporting this vote.

[Arilith]

Also on behalf of InfoPlaza Mobility, which is a consumer, we support this proposal.

[miklcct]

Additionally, since initiating a vote should include data, I suggest everyone vote after the data published.

Our data is now ready:

GTFS static (updated daily): https://dev.aubin.app/gtfs/great_britain_gtfs.zip
Real-time feed (updated every minute): https://dev.aubin.app/national_rail_gtfsrt.binpb

The voting is now open and will end at 2025-01-15 23:59:59 UTC.

[gcamp]

+1 Transit, thanks for taking the time to clarify all of this.

[leonardehrenfried]

+1 OpenTripPlanner

[Arilith]

+1 R-OV

[Arilith]

+1 on behalf of InfoPlaza Mobility

[skinkie]

+1 @StichtingOpenGeo

[mads14]

Sorry to join the conversation so late. Overall Swiftly (a producer) thinks this is a good idea to have an explicit way to specify added non-duplicated trips. Thank you for your work on this!

We did have 2 questions:

1. For trips that are ADDED and not duplicated, do you all think it would be a good idea if the spec encouraged producers to use trip_properties.shape_id, particularly if the stop pattern is not a "substring" of an existing trip on the route?

2. I know that some producers still use ADDED as a way to communicate duplicated trips and some consumers still read it/expect it and do not read DUPLICATED, even though DUPLICATED has been the preferred strategy for some time.
   Should this PR give any guidance on how to transition to this new world given there will be a transition period in which producers will continue to produce ADDED trips (for duplicated) and consumers may not yet support DUPLICATED? Is there an amount of time in which the community will expect producers to completely stop using ADDED for trips that are DUPLICATED and for consumers to stop expecting it?

[miklcct]

Sorry to join the conversation so late. Overall Swiftly (a producer) thinks this is a good idea to have an explicit way to specify added non-duplicated trips. Thank you for your work on this!

We did have 2 questions:

1. For trips that are ADDED and not duplicated, do you all think it would be a good idea if the spec encouraged producers to use trip_properties.shape_id, particularly if the stop pattern is not a "substring" of an existing trip on the route?

shape_id is optional in the GTFS static anyway so I don't want to specifically encourage the use here. It is to be treated like any other trips if the real-time updater has the capability to digest real-time shapes.

2. I know that some producers still use ADDED as a way to communicate duplicated trips and some consumers still read it/expect it and do not read DUPLICATED, even though DUPLICATED has been the preferred strategy for some time.
   Should this PR give any guidance on how to transition to this new world given there will be a transition period in which producers will continue to produce ADDED trips (for duplicated) and consumers may not yet support DUPLICATED? Is there an amount of time in which the community will expect producers to completely stop using ADDED for trips that are DUPLICATED and for consumers to stop expecting it?

I can make a guidance later but I am not going to change anything in the pull request except typos and clarifications as it is now voting period. Basically https://gtfs.org/documentation/realtime/examples/migration-duplicated is to be followed using option 2 (a new trip_id in the added trip), with the ADDED trip specifying the complete journey, if I use OpenTripPlanner implementation as a reference (it supports ADDED trips but not DUPLICATED trips at this moment).

[mads14]

Thanks for the response and clarification!