# Objective
Part of EP-10
Closes EP-118
- Describe the objective or issue this PR addresses.
- If you're fixing a specific issue, use "Fixes #X" for each issue as
[described in the GitHub
docs](https://docs.github.com/en/issues/tracking-your-work-with-issues/using-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword).
## Solution
- Describe the solution used to achieve the objective above.
## Testing
- Did you test these changes? If so, how?
- Are there any parts that need more testing?
- How can other people (reviewers) test your changes? Is there anything
specific they need to know?
- If relevant, what platforms did you test these changes on, and are
there any important ones you can't test?
## Self-Review Checklist:
- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [x] The content adheres to Zed's UI standards
([UX/UI](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
and
[icon](https://github.com/zed-industries/zed/blob/main/crates/icons/README.md)
guidelines)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable
## Showcase
> This section is optional. If this PR does not include a visual change
or does not add a new user-facing feature, you can delete this section.
- Help others understand the result of this PR by showcasing your
awesome work!
- If this PR includes a visual change, consider adding a screenshot,
GIF, or video
- A before/after comparison is very useful for changes to existing
features!
While a showcase should aim to be brief and digestible, you can use a
toggleable section to save space on longer showcases:
<details>
<summary>Click to view showcase</summary>
My super cool demos here
</details>
---
Release Notes:
- N/A or Added/Fixed/Improved ...
Co-authored-by: Nbarry <nbarry@zed.dev>
# Objective
Closes EP-9
- Describe the objective or issue this PR addresses.
- If you're fixing a specific issue, use "Fixes #X" for each issue as
[described in the GitHub
docs](https://docs.github.com/en/issues/tracking-your-work-with-issues/using-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword).
## Solution
- Describe the solution used to achieve the objective above.
## Testing
- Did you test these changes? If so, how?
- Are there any parts that need more testing?
- How can other people (reviewers) test your changes? Is there anything
specific they need to know?
- If relevant, what platforms did you test these changes on, and are
there any important ones you can't test?
## Self-Review Checklist:
- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [x] The content adheres to Zed's UI standards
([UX/UI](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
and
[icon](https://github.com/zed-industries/zed/blob/main/crates/icons/README.md)
guidelines)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable
## Showcase
> This section is optional. If this PR does not include a visual change
or does not add a new user-facing feature, you can delete this section.
- Help others understand the result of this PR by showcasing your
awesome work!
- If this PR includes a visual change, consider adding a screenshot,
GIF, or video
- A before/after comparison is very useful for changes to existing
features!
While a showcase should aim to be brief and digestible, you can use a
toggleable section to save space on longer showcases:
<details>
<summary>Click to view showcase</summary>
My super cool demos here
</details>
---
Release Notes:
- N/A or Added/Fixed/Improved ...
---------
Co-authored-by: Oleksiy Syvokon <oleksiy.syvokon@gmail.com>
Co-authored-by: Oleksiy Syvokon <oleksiy@zed.dev>
This PR adds the `correctly_deleted_chars` field and updates `kept_rate`
to account for it, not just inserted chars.
It also adds `recall_rate` to measure coverage of reference
insertions/deletions.
Finally, it renames "final" to "reference" and "prediction" to
"candidate".
Self-Review Checklist:
- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [x] The content is consistent with the [UI/UX
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable
Release Notes:
- N/A
For now, the only source of logprobs is `./run.py infer`, which writes
them to under the `.predictions` struct. `ep score` copies these values
to `.score`. There is some duplication (same value stored in two
places), which is unfortunate, but can't be fixed without reworking how
scores are stored.
Release Notes:
- N/A
Previously, we were not computing excerpt regions correctly for EP
examples captured from prod. This PR fixes that, and also simplifies the
data flow in the EP CLI. Examples either come from a concise spec (like
the markdown evals), or are collected from prod. Either way, we compute
from them a `ZetaPromptInput`, and the downstream steps like
prompt-formatting and scoring are derived from that.
Release Notes:
- N/A
---------
Co-authored-by: Ben Kunkle <ben@zed.dev>
Other changes:
- Changed tokenization to more code-aware tokenization from split-commit
- Fixed word-diff implementation which was inefficient and sometimes
incorrect
Release Notes:
- N/A
This also fixes the isolated whitespace change metric, which is not
takes cursor position into account (it's okay to insert a newline if it
has a cursor on it)
Release Notes:
- N/A
* [x] capture and store teacher model's predicted cursor position
* [x] provide cursor position to student during distillation
* [x] eval cursor positions
* [x] parse and apply cursor position predictions at runtime
Release Notes:
- N/A
Closes #ISSUE
Measure the model reversing user edits programatically, for a more
robust and consistent measurement than LLM-as-judge
Release Notes:
- N/A *or* Added/Fixed/Improved ...
This command takes raw LLM outputs (`predictions.actual_output`) that
could be generated elsewhere and parses them into a canonical unified
diff (`predictions.actual_patch`).
This is useful for simplifying the evaluation pipeline and for rerunning
the parser without having to generate LLM outputs.
Release Notes:
- N/A
We were including changes due to Buffer.reload in the edit history.
Release Notes:
- N/A
---------
Co-authored-by: Oleksiy Syvokon <oleksiy.syvokon@gmail.com>
Co-authored-by: Agus Zubiaga <agus@zed.dev>
Co-authored-by: Ben Kunkle <ben@zed.dev>
Co-authored-by: Zed Zippy <234243425+zed-zippy[bot]@users.noreply.github.com>
This one does `fim_prefix`, `fim_middle`, and `fim_suffix` in that
order, in the prompt, instead of putting the current middle last.
Release Notes:
- N/A
---------
Co-authored-by: Agus Zubiaga <agus@zed.dev>
Co-authored-by: Ben Kunkle <ben@zed.dev>
This fixes an issue that we noticed in particular with Mercury edit
predictions.
* [x] fix storage to not go stale
* [x] exclude excerpts that intersect the cursor excerpt
* [x] see if string representation of excerpts can be cached, to avoid
rebuilding it on every prediction
Release Notes:
- N/A
---------
Co-authored-by: Ben Kunkle <ben@zed.dev>
- Replace `is_new_file: bool` with `FileStatus` enum
(Created/Modified/Deleted) in udiff.rs to properly track file status
through diff operations
- Handle deleted files in `apply_diff` by calling
`project.delete_file()`
- Fix diff serialization in reorder_patch.rs to output `+++ /dev/null`
for file deletions and parse both `--- /dev/null` and `+++ /dev/null`
correctly
- Add bounds check for edit ranges exceeding buffer length
Also includes edit_prediction_cli improvements:
- Track `context_range` and `editable_range` in ExampleBuffer for more
precise prompt formatting
- Export MAX_CONTEXT_TOKENS and MAX_REWRITE_TOKENS from zeta2
- Wait for buffer parsing before computing ranges
- Respect NO_COLOR env var and enable info-level logging
Release Notes:
- N/A
Co-authored-by: Agus Zubiaga <agus@zed.dev>
* Fix some bugs in capture of EP examples from running app
* Tweak markdown format for EP examples
* Store repo and revision in TOML front matter
* Represent cursor position using a comment line
* Allow multiple expected patches in evals
* Remove line-based scoring criteria for evals
* Add a `synthesize` subcommand to the EP cli that generates examples
from git commits
Release Notes:
- N/A
This PR adds a staff-only button to the edit prediction menu for
capturing your current editing session as edit prediction example file.
When you click that button, it opens a markdown tab with the example. By
default, the most recent change that you've made is used as the expected
patch, and all of the previous events are used as the editing history.
<img width="303" height="123" alt="Screenshot 2025-12-14 at 6 58 33 PM"
src="https://github.com/user-attachments/assets/600c7bf2-7cf4-4d27-8cd4-8bb70d0b20b0"
/>
Release Notes:
- N/A
This PR restructures the commands of the Edit Prediction CLI (now called
`ep`), to support some flows that are important for the training
process:
* generating zeta2 prompt and expected output, without running
predictions
* scoring outputs that are generated by a system other than the
production code (to evaluate the model during training)
To achieve this, we've restructured the CLI commands so that they all
take as input, and produce as output, a consistent, uniform data format:
a set of one or more `Example` structs, expressible either as the
original markdown format, or as a JSON lines. The `Example` struct
starts with the basic fields that are in human-readable eval format, but
contain a number of optional fields that are filled in by different
steps in the processing pipeline (`context`, `predict`, `format-prompt`,
and `score`).
### To do
* [x] Adjust the teacher model output parsing to use the full buffer
contents
* [x] Move udiff to cli
* [x] Align `format-prompt` with Zeta2's production code
* [x] Change score output to assume same provider
* [x] Move pretty reporting to `eval` command
* [x] Store cursor point in addition to cursor offset
* [x] Rename `edit_prediction_cli2` -> `edit_prediction_cli` (nuke the
old one)
Release Notes:
- N/A
---------
Co-authored-by: Oleksiy Syvokon <oleksiy@zed.dev>
Co-authored-by: Agus Zubiaga <agus@zed.dev>
Co-authored-by: Ben Kunkle <ben@zed.dev>
This PR partially implements a knowledge distillation data pipeline.
`zeta distill` gets a dataset of chronologically ordered commits and
generates synthetic predictions with a teacher model (one-shot Claude
Sonnet).
`zeta distill --batches cache.db` will enable Message Batches API. Under
the first run, this command will collect all LLM requests and upload a
batch of them to Anthropic. On subsequent runs, it will check the batch
status. If ready, it will download the result and put them into the
local cache.
Release Notes:
- N/A
---------
Co-authored-by: Piotr Osiewicz <24362066+osiewicz@users.noreply.github.com>
Co-authored-by: Ben Kunkle <ben@zed.dev>
