One way is to eagerly call JSON.parse as fragments are coming in. If you also split on json semantic boundaries like quotes/closing braces/closing brackets, you can detect valid objects and start processing them while the stream continues.

Interesting approach! thanks for sharing

Not the original commenter but I’ve done this too with Pydantic AI (actually the library does it for you). See “Streaming Structured Output” here https://ai.pydantic.dev/output/#streaming-structured-output

Thanks yes! Im aware of structured outputs, llama.cpp has also great support with GBNF and several languages beyond json.

I've been trying to create go/rust ones but its way harder than just json due to all the context/state they carry over