Long-lived streaming responses (LLM completions) can stall when the socket goes silent but the connection is never closed, leaving a read that never returns to hang the request indefinitely. Add an opt-in read_timeout parameter. Release Notes: - N/A
Long-lived streaming responses (LLM completions) can stall when the socket goes silent but the connection is never closed, leaving a read that never returns to hang the request indefinitely. Add an opt-in read_timeout parameter. Release Notes: - N/A