This change adds metrics for evaluating editable context collectors and
the collectors themselves:
- `cursor excerpt`: small area around the cursor.
- `current-file`: the full file that contains the cursor.
- `edit-history`: excerpts around edit history edits.
- `edit-history-file`: full files referenced by the edit history.
- `git-log`: files related to the cursor file according to recent git
co-change history.
- `oracle-file`: full files referenced by the expected patches. This is
an oracle/evaluation-only source because it uses the target patch.
Individual collectors can be enabled by passing a comma-separated list
like this: `ep context --type=edit-history,current-file`. There's also a
preset, `--type=editable`, which will enable all collectors except
oracle.
This change also adds the
`.prompt_input.related_files[0].excerpts[0].context_source` field.
Editable context is only accessible via ep-cli for now.
Release Notes:
- N/A
---------
Co-authored-by: Eric Holk <eric@zed.dev>
Needed for use in cloud
Self-Review Checklist:
- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [x] The content is consistent with the [UI/UX
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable
Closes #ISSUE
Release Notes:
- N/A or Added/Fixed/Improved ...
Stacked on https://github.com/zed-industries/zed/pull/50566.
Begin collecting kept chars rate, as well as the count of tree-sitter
errors in the code before and after applying the prediction.
Self-Review Checklist:
- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [ ] The content is consistent with the [UI/UX
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable
Closes #ISSUE
Release Notes:
- N/A or Added/Fixed/Improved ...
- `language_model` no longer depends on provider-specific crates such as
`anthropic` and `open_ai` (inverted dependency)
- `language_model_core` was extracted from `language_model` which
contains the types for the provider-specific crates to convert to/from.
- `gpui::SharedString` has been extracted into its own crate (still
exposed by `gpui`), so `language_model_core` and provider API crates
don't have to depend on `gpui`.
- Removes some unnecessary `&'static str` | `SharedString` -> `String`
-> `SharedString` conversions across the codebase.
- Extracts the core logic of the cloud `LanguageModelProvider` into its
own crate with simpler dependencies.
Release Notes:
- N/A
---------
Co-authored-by: John Tur <john-tur@outlook.com>
The watcher had been broken for some time, but became even more broken
after the recent move of the queries.
This PR restores the reloading behavior for debug builds so that
languages are reloaded once a scheme file is changed.
Release Notes:
- N/A
Duplicates are defined as cursor positions that have an approximate
Jaccard similarity greater than 0.5 (over token 5-grams).
From the resulting clusters of near-duplicates, we select up to N
examples that are maximally different from each other.
Release Notes:
- N/A
#2874 on steroids
Before you mark this PR as ready for review, make sure that you have:
- [ ] Added a solid test coverage and/or screenshots from doing manual
testing
- [ ] Done a self-review taking into account security and performance
aspects
- [ ] Aligned any UI changes with the [UI
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
Release Notes:
- N/A
---------
Co-authored-by: Eric Holk <eric@zed.dev>
This allows us to switch the prompt format without client-side changes.
If we want to experiment with prompt formats or models other than the
currently-deployed one, we can use the raw endpoint, and do prompt
construction and output processing on the client.
This also adds an optional environment parameter to the raw endpoint, so
that we can use that endpoint in the new scheme where we're deploying to
separate environments for different zeta prompt versions.
Release Notes:
- N/A
Some code got added to `workspace` that prevents us from running tests
for the `edit_prediction(cli)` crates specifically without the
`test-support` feature flag.
Release Notes:
- N/A
Co-authored-by: Ben Kunkle <ben@zed.dev>
Adds a new `ep split` command that splits JSONL datasets into multiple
output files with stratification by `repository_url` when present.
Example usage:
ep split input.jsonl train.jsonl=80% valid.jsonl=rest
Release Notes:
- N/A
Generates a training or evaluation example from a
chronologically-ordered commit. This is a port from the Python codebase
(except for the reorder_patch.rs, which was originally written in Rust
in).
Release Notes:
- N/A
This PR adds a staff-only button to the edit prediction menu for
capturing your current editing session as edit prediction example file.
When you click that button, it opens a markdown tab with the example. By
default, the most recent change that you've made is used as the expected
patch, and all of the previous events are used as the editing history.
<img width="303" height="123" alt="Screenshot 2025-12-14 at 6 58 33 PM"
src="https://github.com/user-attachments/assets/600c7bf2-7cf4-4d27-8cd4-8bb70d0b20b0"
/>
Release Notes:
- N/A
- Limit status lines to 10 in case `max_parallelism` is specified with a
grater value
- Handle logging gracefully rather than writing over it when clearing
status lines
Release Notes:
- N/A
This PR restructures the commands of the Edit Prediction CLI (now called
`ep`), to support some flows that are important for the training
process:
* generating zeta2 prompt and expected output, without running
predictions
* scoring outputs that are generated by a system other than the
production code (to evaluate the model during training)
To achieve this, we've restructured the CLI commands so that they all
take as input, and produce as output, a consistent, uniform data format:
a set of one or more `Example` structs, expressible either as the
original markdown format, or as a JSON lines. The `Example` struct
starts with the basic fields that are in human-readable eval format, but
contain a number of optional fields that are filled in by different
steps in the processing pipeline (`context`, `predict`, `format-prompt`,
and `score`).
### To do
* [x] Adjust the teacher model output parsing to use the full buffer
contents
* [x] Move udiff to cli
* [x] Align `format-prompt` with Zeta2's production code
* [x] Change score output to assume same provider
* [x] Move pretty reporting to `eval` command
* [x] Store cursor point in addition to cursor offset
* [x] Rename `edit_prediction_cli2` -> `edit_prediction_cli` (nuke the
old one)
Release Notes:
- N/A
---------
Co-authored-by: Oleksiy Syvokon <oleksiy@zed.dev>
Co-authored-by: Agus Zubiaga <agus@zed.dev>
Co-authored-by: Ben Kunkle <ben@zed.dev>
This PR partially implements a knowledge distillation data pipeline.
`zeta distill` gets a dataset of chronologically ordered commits and
generates synthetic predictions with a teacher model (one-shot Claude
Sonnet).
`zeta distill --batches cache.db` will enable Message Batches API. Under
the first run, this command will collect all LLM requests and upload a
batch of them to Anthropic. On subsequent runs, it will check the batch
status. If ready, it will download the result and put them into the
local cache.
Release Notes:
- N/A
---------
Co-authored-by: Piotr Osiewicz <24362066+osiewicz@users.noreply.github.com>
Co-authored-by: Ben Kunkle <ben@zed.dev>
