language_models: Clear speed for OpenAI-compatible providers that don't support fast mode (#59496)
`OpenAiCompatibleLanguageModel::stream_completion` (used for custom OpenAI-compatible providers) forwarded `request.speed` straight into `into_open_ai`/`into_open_ai_response`, which translate `Speed::Fast` into OpenAI's `service_tier` field. Custom OpenAI-compatible endpoints don't recognize that field and reject the request. `speed` can end up set even for a provider that doesn't support fast mode: `Thread::inherit_parent_settings` copies `speed` from a parent thread to a subagent without checking whether the subagent's model supports it. Every other provider sharing this conversion code already guards against this (`open_ai.rs`, `anthropic.rs`, `anthropic_compatible.rs`, `language_models_cloud.rs`). This adds the same guard for `OpenAiCompatibleLanguageModel`. Release Notes: - Fixed requests to OpenAI-compatible providers (e.g. Baseten) sometimes including an unsupported `service_tier` parameter
This commit is contained in:
1 parent
39ea28735d
commit
c4e30593b1
1 file changed
+7
-1
@@ -322,7 +322,7 @@ impl LanguageModel for OpenAiCompatibleLanguageModel {
|
||||
|
||||
fn stream_completion(
|
||||
&self,
|
||||
request: LanguageModelRequest,
|
||||
mut request: LanguageModelRequest,
|
||||
cx: &AsyncApp,
|
||||
) -> BoxFuture<
|
||||
'static,
|
||||
@@ -334,6 +334,12 @@ impl LanguageModel for OpenAiCompatibleLanguageModel {
|
||||
LanguageModelCompletionError,
|
||||
>,
|
||||
> {
|
||||
// `speed` can leak in from a parent thread's model; this provider never
|
||||
// supports fast mode, and arbitrary compatible endpoints reject `service_tier`.
|
||||
if !self.supports_fast_mode() {
|
||||
request.speed = None;
|
||||
}
|
||||
|
||||
if self.model.capabilities.chat_completions {
|
||||
let request = into_open_ai(
|
||||
request,
|
||||
|
||||
Reference in new issue
Block a user