[Bug]: Anthropic streaming fails with JSONDecodeError on partial chunks #17473
Description
What happened?
This issue is similar to #16562, my product is using Anthropic models with streaming mode, sometimes partial chunks are returned but LiteLLM tries to parse the JSON instantly, leading to an json.decoder.JSONDecodeError.
I saw in #16562, handle_accumulated_json_chunk has been added to support Gemini models partial chunks, can you please also add this capability to support Anthropic models?
Thanks!
Relevant log output
litellm.ServiceUnavailableError: litellm.MidStreamFallbackError: litellm.APIConnectionError: Error parsing chunk: Unterminated string starting at: line 1 column 32 (char 31),
Received chunk: data: {\"type\":\"content_block_delta\",\"i
Traceback (most recent call last):
File \"/app/.venv/lib/python3.12/site-packages/litellm/llms/anthropic/chat/handler.py\", line 903, in __anext__
data_json = json.loads(str_line[5:])
^^^^^^^^^^^^^^^^^^^^^^^^
File \"/usr/local/lib/python3.12/json/__init__.py\", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/usr/local/lib/python3.12/json/decoder.py\", line 338, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/usr/local/lib/python3.12/json/decoder.py\", line 354, in raw_decode
obj, end = self.scan_once(s, idx)
^^^^^^^^^^^^^^^^^^^^^^
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 32 (char 31)Are you a ML Ops Team?
No
What LiteLLM version are you on ?
1.76.1
Twitter / LinkedIn details
No response