When images are resized to meet provider size constraints (Anthropic's
1568px limit or the 5MB encoded-PNG cap), the stored ImageSize was still
recording the original width/height rather than the final post-downscale
dimensions. This caused incorrect token estimation via estimate_tokens()
since it uses width * height / 750.
Use processed_image.dimensions() after all downscale passes so that
ImageSize reflects the actual image sent to the provider.
Release Notes:
- Fixed an issue where token estimation would be incorrect in case where
the thread contained downscaled images.
- `language_model` no longer depends on provider-specific crates such as
`anthropic` and `open_ai` (inverted dependency)
- `language_model_core` was extracted from `language_model` which
contains the types for the provider-specific crates to convert to/from.
- `gpui::SharedString` has been extracted into its own crate (still
exposed by `gpui`), so `language_model_core` and provider API crates
don't have to depend on `gpui`.
- Removes some unnecessary `&'static str` | `SharedString` -> `String`
-> `SharedString` conversions across the codebase.
- Extracts the core logic of the cloud `LanguageModelProvider` into its
own crate with simpler dependencies.
Release Notes:
- N/A
---------
Co-authored-by: John Tur <john-tur@outlook.com>
This PR moves the `CompletionIntent` enum from the `cloud_llm_client`
crate to the `language_model` crate, as it is no longer part of the
Cloud interface.
Release Notes:
- N/A
We were sending the raw tool debug output as JSON to the model rather
than whatever the tool intended as content for the model.
Which meant we were sending unneeded information to the model, which
matters in the edit tool case.
Release Notes:
- N/A
This is a staff only toggle for now, since the consequences of
activating it are not obvious and quite dire (tokens costs 6 times
more).
Also, persist thinking, thinking effort and fast mode in DbThread so the
thinking mode toggle and thinking effort are persisted.
Release Notes:
- Agent: The thinking mode toggle and thinking effort are now persisted
when selecting a thread from history.
Before you mark this PR as ready for review, make sure that you have:
- [x] Added a solid test coverage and/or screenshots from doing manual
testing
- [x] Done a self-review taking into account security and performance
aspects
- [x] Aligned any UI changes with the [UI
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
Release Notes:
- N/A
---------
Co-authored-by: Zed Zippy <234243425+zed-zippy[bot]@users.noreply.github.com>
The rate limiter's semaphore guard was being held for the entire
duration of a turn, including during tool execution. This caused
deadlocks when subagents tried to acquire permits while parent requests
were waiting for them to complete.
## The Problem
In `run_turn_internal`, the stream (which contains the `RateLimitGuard`
holding the semaphore permit) was kept alive throughout the entire loop
iteration - including during **tool execution**:
1. Parent request acquires permit
2. Parent starts streaming, consumes response
3. Parent starts executing tools (subagents)
4. **Stream/guard still held** while tools execute
5. Subagents try to acquire permits → blocked because parent still holds
permit
6. Deadlock if all permits are held by parents waiting for subagent
children
## The Fix
Two changes were made:
1. **Drop the stream early**: Added an explicit `drop(events)` after the
stream is fully consumed but before tool execution begins. This releases
the rate limit permit so subagents can acquire it.
2. **Removed the `bypass_rate_limit` workaround**: Since the root cause
is now fixed, the bypass mechanism is no longer needed.
Note: no release notes because subagents are still feature-flagged, and
this rate limiting change isn't actually observable without them.
Release Notes:
- N/A
## Problem
When subagents use the `edit_file` tool, it creates an `EditAgent` that
makes its own model request to get the edit instructions. These "nested"
requests compete with the parent subagent conversation requests for rate
limiter permits.
The rate limiter uses a semaphore with a limit of 4 concurrent requests
per model instance. When multiple subagents run in parallel:
1. 3 subagents each hold 1 permit for their ongoing conversation streams
(3 permits used)
2. When all 3 try to use `edit_file` simultaneously, their edit agents
need permits too
3. Only 1 edit agent can get the 4th permit; the other 2 block waiting
4. The blocked edit agents can't complete, so their parent subagent
conversations can't complete
5. The parent conversations hold their permits, so the blocked edit
agents stay blocked
6. **Deadlock**
## Solution
Added a `bypass_rate_limit` field to `LanguageModelRequest`. When set to
`true`, the request skips the rate limiter semaphore entirely. The
`EditAgent` sets this flag because its requests are already "part of" a
rate-limited parent request.
(No release notes because subagents are still feature-flagged.)
Release Notes:
- N/A
---------
Co-authored-by: Zed Zippy <234243425+zed-zippy[bot]@users.noreply.github.com>
## Problem
When users paste or drag large images into the agent panel, the encoded
payload can exceed upstream provider limits (e.g., Anthropic's 5MB
per-image limit), causing API errors.
## Solution
Enforce a default 5MB limit on encoded PNG bytes in
`LanguageModelImage::from_image`:
1. Apply existing Anthropic dimension limits first (1568px max in either
dimension)
2. Iteratively downscale by ~15% per pass until the encoded PNG is under
5MB
3. Return `None` if the image can't be shrunk within 8 passes
(fail-safe)
The limit is enforced at the `LanguageModelImage` conversion layer,
which is the choke point for all image ingestion paths (agent panel
paste/drag, file mentions, text threads, etc.).
## Future Work
The 5MB limit is a conservative default. Provider-specific limits can be
introduced later by adding a `from_image_with_constraints` API.
## Testing
Added a regression test that:
1. Generates a noisy 4096x4096 PNG (guaranteed >5MB)
2. Converts it via `LanguageModelImage::from_image`
3. Asserts the result is ≤5MB and was actually downscaled
---
**Note:** This PR builds on #45312 (prompt store fail-open fix). Please
merge that first.
cc @rtfeldman
---------
Co-authored-by: Zed Zippy <234243425+zed-zippy[bot]@users.noreply.github.com>
I noticed we had some typos that were getting through CI, but it looks
like the new version of `typos` catches them. So I updated it and fixed
them.
Release Notes:
- N/A
Closes#5355
Release Notes:
- Fixed rendering glitches with files with more than 16 million lines
(that occured due to floating number rounding errors).
---------
Co-authored-by: Smit Barmase <heysmitbarmase@gmail.com>
This introduces a new field `thinking_allowed` on `LanguageModelRequest`
which lets us control whether thinking should be enabled if the model
supports it.
We permit thinking in the Inline Assistant, Edit File tool and the Git
Commit message generator, this should make generation faster when using
a thinking model, e.g. `claude-sonnet-4-thinking`
Release Notes:
- N/A
cc @osyvokon
We were seeing a bunch of errors in our backend when people were using
Claude models with thinking enabled.
In the logs we would see
> an error occurred while interacting with the Anthropic API:
invalid_request_error: messages.x.content.0.type: Expected `thinking` or
`redacted_thinking`, but found `text`. When `thinking` is enabled, a
final `assistant` message must start with a thinking block (preceeding
the lastmost set of `tool_use` and `tool_result` blocks). We recommend
you include thinking blocks from previous turns. To avoid this
requirement, disable `thinking`. Please consult our documentation at
https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking
However, this issue did not occur frequently and was not easily
reproducible. Turns out it was triggered by us not correctly handling
[Redacted Thinking
Blocks](https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking#thinking-redaction).
I could constantly reproduce this issue by including this magic string:
`ANTHROPIC_MAGIC_STRING_TRIGGER_REDACTED_THINKING_46C9A13E193C177646C7398A98432ECCCE4C1253D5E2D82641AC0E52CC2876CB
` in the request, which forces `claude-3-7-sonnet` to emit redacted
thinking blocks (confusingly the magic string does not seem to be
working for `claude-sonnet-4`). As soon as we hit a tool call Anthropic
would return an error.
Thanks to @osyvokon for pointing me in the right direction 😄!
Release Notes:
- agent: Fixed an issue where Anthropic models would sometimes return an
error when thinking was enabled
This PR adds a new `intent` field to completion requests to assist in
categorizing them correctly.
Release Notes:
- N/A
---------
Co-authored-by: Ben Brandt <benjamin.j.brandt@gmail.com>
This expands our deserialization of JSON from models to be more tolerant
of different variations that the model may send, including
capitalization, wrapping things in objects vs. being plain strings, etc.
Also when deserialization fails, it reports the entire error in the JSON
so we can see what failed to deserialize. (Previously these errors were
very unhelpful at diagnosing the problem.)
Finally, also removes the `WrappedText` variant since the custom
deserializer just turns that style of JSON into a normal `Text` variant.
Release Notes:
- N/A
Some providers sometimes send `{ "type": "text", "text": ... }` instead
of just the text as a string. Now we accept those instead of erroring.
Release Notes:
- N/A
This is very basic support for them. There are a number of other TODOs
before this is really a first-class supported feature, so not adding any
release notes for it; for now, this PR just makes it so that if
read_file tries to read a PNG (which has come up in practice), it at
least correctly sends it to Anthropic instead of messing up.
This also lays the groundwork for future PRs for more first-class
support for images in tool calls across more image file formats and LLM
providers.
Release Notes:
- N/A
---------
Co-authored-by: Agus Zubiaga <hi@aguz.me>
Co-authored-by: Agus Zubiaga <agus@zed.dev>
Problem Statement:
Support for image analysis (vision) is currently restricted to Anthropic
and Gemini models. This limits users who wish to leverage vision
capabilities available in other models, such as Copilot, for tasks like
attaching image context within the agent message editor.
Proposed Change:
This PR extends vision support to include Copilot models that are
already equipped with vision capabilities. This integration will allow
users within VS Code to attach and analyze images using supported
Copilot models via the agent message editor.
Scope Limitation:
This PR does not implement controls within the message editor to ensure
that image context (e.g., through copy-paste or attachment) is
exclusively enabled or prompted only when a vision-supported model is
active. Long term the message editor should have access to each models
vision capability and stop the users from attaching images by either
greying out the context saying it's not support or not work through both
copy paste and file/directory search.
Closes#30076
Release Notes:
- Add vision support for Copilot Chat models
---------
Co-authored-by: Bennet Bo Fenner <bennet@zed.dev>
Release Notes:
- agent: Add support for @mentioning images
- agent: Add support for including images via file context picker
---------
Co-authored-by: Oleksiy Syvokon <oleksiy.syvokon@gmail.com>
This PR adds a "max mode" toggle to the Agent panel, for models that
support it.
Only visible to folks in the `new-billing` feature flag.
Icon is just a placeholder.
Release Notes:
- N/A
Our provider code in `language_models` filters out messages for which
`LanguageModelRequestMessage::contents_empty` returns `false`. This
doesn't seem wrong by itself, but `contents_empty` was returning `false`
for messages whose first segment didn't contain non-whitespace text even
if they contained other non-empty segments. This caused requests to fail
when a message with a tool call didn't contain any preceding text.
Release Notes:
- N/A
This PR attaches the thread ID and the new prompt ID to telemetry events
for completions in the Agent panel.
Release Notes:
- N/A
---------
Co-authored-by: Mikayla Maki <mikayla.c.maki@gmail.com>
This PR removes the dependency on the `ui` crate from the
`language_model` crate.
We were only depending on it to import `IconName`—which now lives in
`icons`—and some re-exported GPUI items.
Release Notes:
- N/A
This PR removes the dependencies on the individual model provider crates
from the `language_model` crate.
The various conversion methods for converting a `LanguageModelRequest`
into its provider-specific request type have been inlined into the
various provider modules in the `language_models` crate.
The model providers we provide via Zed's cloud offering get to stay, for
now.
Release Notes:
- N/A
Done automatically with
> ast-grep -p '$A.background_executor().spawn($B)' -r
'$A.background_spawn($B)' --update-all --globs "\!crates/gpui"
Followed by:
* `cargo fmt`
* Unexpected need to remove some trailing whitespace.
* Manually adding imports of `gpui::{AppContext as _}` which provides
`background_spawn`
* Added `AppContext as _` to existing use of `AppContext`
Release Notes:
- N/A
- Added support for DeepSeek as a new language model provider in Zed
Assistant
- Implemented streaming API support for real-time responses from
DeepSeek models.
- Added a configuration UI for DeepSeek API key management and settings.
- Updated documentation with detailed setup instructions for DeepSeek
integration.
- Added DeepSeek-specific icons and model definitions for seamless
integration into the Zed UI.
- Integrated DeepSeek into the language model registry, making it
available alongside other providers like OpenAI and Anthropic.
Release Notes:
- Added support for DeepSeek to the Assistant.
---------
Co-authored-by: Marshall Bowers <git@maxdeviant.com>
There's still a bit more work to do on this, but this PR is compiling
(with warnings) after eliminating the key types. When the tasks below
are complete, this will be the new narrative for GPUI:
- `Entity<T>` - This replaces `View<T>`/`Model<T>`. It represents a unit
of state, and if `T` implements `Render`, then `Entity<T>` implements
`Element`.
- `&mut App` This replaces `AppContext` and represents the app.
- `&mut Context<T>` This replaces `ModelContext` and derefs to `App`. It
is provided by the framework when updating an entity.
- `&mut Window` Broken out of `&mut WindowContext` which no longer
exists. Every method that once took `&mut WindowContext` now takes `&mut
Window, &mut App` and every method that took `&mut ViewContext<T>` now
takes `&mut Window, &mut Context<T>`
Not pictured here are the two other failed attempts. It's been quite a
month!
Tasks:
- [x] Remove `View`, `ViewContext`, `WindowContext` and thread through
`Window`
- [x] [@cole-miller @mikayla-maki] Redraw window when entities change
- [x] [@cole-miller @mikayla-maki] Get examples and Zed running
- [x] [@cole-miller @mikayla-maki] Fix Zed rendering
- [x] [@mikayla-maki] Fix todo! macros and comments
- [x] Fix a bug where the editor would not be redrawn because of view
caching
- [x] remove publicness window.notify() and replace with
`AppContext::notify`
- [x] remove `observe_new_window_models`, replace with
`observe_new_models` with an optional window
- [x] Fix a bug where the project panel would not be redrawn because of
the wrong refresh() call being used
- [x] Fix the tests
- [x] Fix warnings by eliminating `Window` params or using `_`
- [x] Fix conflicts
- [x] Simplify generic code where possible
- [x] Rename types
- [ ] Update docs
### issues post merge
- [x] Issues switching between normal and insert mode
- [x] Assistant re-rendering failure
- [x] Vim test failures
- [x] Mac build issue
Release Notes:
- N/A
---------
Co-authored-by: Antonio Scandurra <me@as-cii.com>
Co-authored-by: Cole Miller <cole@zed.dev>
Co-authored-by: Mikayla <mikayla@zed.dev>
Co-authored-by: Joseph <joseph@zed.dev>
Co-authored-by: max <max@zed.dev>
Co-authored-by: Michael Sloan <michael@zed.dev>
Co-authored-by: Mikayla Maki <mikaylamaki@Mikaylas-MacBook-Pro.local>
Co-authored-by: Mikayla <mikayla.c.maki@gmail.com>
Co-authored-by: joão <joao@zed.dev>
This PR restructures the storage of the tool uses and results in
`assistant2` so that they don't live on the individual messages.
It also introduces a `LanguageModelToolUseId` newtype for better type
safety.
Release Notes:
- N/A
Bennet Bo Fenner