Add Pekko streaming support for chat completions #128
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kciesielski
reviewed
Nov 7, 2023
|
|
||
| private def deserializeEvent: Flow[ServerSentEvent, ChatChunkResponse, Any] = | ||
| Flow[ServerSentEvent].collect { | ||
| case ServerSentEvent(Some(data), _, _, _) if data != DoneEventMessage => |
There was a problem hiding this comment.
** issue ** This makes me thinking - the DoneEventMessage is simply filtered here, and we basically rely on the stream end to be signalised. Maybe we should rather takeWhile instead of collect, which would guarantee that DoneEventMessage completes the stream right away.
For that, we may need a test where there are more messages after DoneEventMessage, it's unexpected but we would be sure that "done" means "done" for us, no matter what happens.
There was a problem hiding this comment.
Similarly with fs2 and zio, these modules may also need such an improvement.
There was a problem hiding this comment.
Thanks for that suggestion, I added takeWhile to fs2, zio and pekko.
kciesielski
approved these changes
Nov 8, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds streaming support for Create chat completion endpoint using Pekko Streams as a streams implementation.
Here are the main changes made in this PR:
PekkoHttpBackend.stubMinor change:
DeserializationExceptionwas replaced with more detailedDeserializationOpenAIExceptionduring tests