StreamCallback issue if :usage is nothing in PT.

[Sixzero]

MVP:

using PromptingTools
using StreamCallbacks
const PT = PromptingTools

cb = PT.StreamCallback(flavor = StreamCallbacks.OpenAIStream())

msg = aigenerate("Write a story about a space cat in 100 chars";
    streamcallback=cb,
    model="gpt4om",
    api_kwargs=(stream=true,)  # Enable streaming mode!
)
ERROR: MethodError: no method matching getindex(::Nothing, ::Symbol)
The function `getindex` exists, but no method is defined for this combination of argument types.
Stacktrace:
 [1] response_to_message(schema::PromptingTools.OpenAISchema, MSG::Type{…}, choice::JSON3.Object{…}, resp::OpenAI.OpenAIResponse{…}; model_id::String, time::Float64, run_id::Int64, sample_id::Nothing, name_assistant::Nothing)
   @ PromptingTools ~/repo/PromptingTools.jl/src/llm_openai.jl:181
 [2] response_to_message
   @ ~/repo/PromptingTools.jl/src/llm_openai.jl:160 [inlined]
...
 [6] aigenerate(prompt_schema::PromptingTools.OpenAISchema, prompt::String; verbose::Bool, api_key::String, model::String, return_all::Bool, dry_run::Bool, conversation::Vector{…}, streamcallback::StreamCallback{…}, no_system_message::Bool, name_user::Nothing, name_assistant::Nothing, http_kwargs::@NamedTuple{…}, api_kwargs::@NamedTuple{…}, kwargs::@Kwargs{})
   @ PromptingTools ~/repo/PromptingTools.jl/src/llm_openai.jl:340
 [7] aigenerate(prompt::String; model::String, kwargs::@Kwargs{…})
   @ PromptingTools ~/repo/PromptingTools.jl/src/llm_interface.jl:498
 [8] top-level scope
   @ ~/repo/jl_pkgs/StreamCallbacksExt.jl/test.jl:18

If I don't add the stream=true which I think is rarely mentioned anywhere in the StreamCallbacks.jl if even mentioned that it is needed.

msg = aigenerate("Write a story about a space cat in 100 chars";
    streamcallback=cb,
    model="gpt4om",
    api_kwargs=(stream=true,)  # Enable streaming mode!
)

ERROR: HTTP.RequestError:
HTTP.Request:
HTTP.Messages.Request:
"""
POST /v1/chat/completions HTTP/1.1
Authorization: ******
Content-Type: application/json
Host: api.openai.com
Accept: */*
User-Agent: HTTP.jl/1.11.2
Accept-Encoding: gzip
Cookie: ******
Transfer-Encoding: chunked

[Message Body was streamed]"""Underlying error:
AssertionError: Content-Type header include the type text/event-stream
Stacktrace:
  [1] (::HTTP.ConnectionRequest.var"#connections#4"{…})(req::HTTP.Messages.Request; proxy::Nothing, socket_type::Type, socket_type_tls::Nothing, readtimeout::Int64, connect_timeout::Int64, logerrors::Bool, logtag::Nothing, closeimmediately::Bool, kw::@Kwargs{…})
    @ HTTP.ConnectionRequest ~/.julia/packages/HTTP/FSzDg/src/clientlayers/ConnectionRequest.jl:143
...
 [25] streamed_request!(cb::StreamCallback{…}, url::String, headers::Vector{…}, input::IOBuffer; kwargs::@Kwargs{…})
    @ StreamCallbacks ~/.julia/packages/StreamCallbacks/TSTvV/src/shared_methods.jl:197
...

As a quick fix I did this in llm_openai.jl:180

    usage = get(resp.response, :usage, nothing)
    usage = usage isa Dict ? usage : Dict()
    tokens_prompt = get(usage, :prompt_tokens, 0)
    tokens_completion = get(usage, :completion_tokens, 0)

But I am not sure if I this is the right way to fix this issue.
The problem is that resp.response.usage == nothing, so get doesn't default to Dict but return the nothing.

[Sixzero]

Although what is interesting is that:

cb = PT.StreamCallback()

msg = aigenerate("Write a story about a space cat in 100 chars"; streamcallback=cb)

This works.

If I specify the flavor it won't work anymore.

[svilupp]

Thanks for reporting! It's a bug in StreamCallbacks that should be fixed by this PR: svilupp/StreamCallbacks.jl#9 (new patch version has been tagged).

The issue was that I wasn't checking if we had any usage data or not, so I was adding a usage=nothing.

If you want usage from the server, you must explicitly set it api_kwargs = (stream = true, stream_options = (include_usage = true,)). This is done automatically by the callback configurator (if you let it).

[Sixzero]

thx for fast fix!

[Sixzero]

Tested this exact call:

using PromptingTools
using PromptingTools: SystemMessage, UserMessage, AIMessage
using StreamCallbacks
using StreamCallbacksExt
cb = StreamCallbackWithHooks(
    content_formatter = text -> begin
    text
end,
on_done = () -> nothing,
on_stop_sequence = (stop_sequence) -> (println(stop_sequence)),
)
cb = StreamCallbackChannelWrapper(callback=cb)
conv = [SystemMessage("YOU ARE JUST GUD."), UserMessage("How you so gud."), AIMessage("You no know?"), UserMessage("ohno")]
msg = aigenerate(conv;
    streamcallback=cb,
    model="claude",
    api_kwargs=(; stop_sequences=["that"], top_p=0.8, max_tokens=8192) 
)

And no issues.