In some evals, the teacher produced hallucinations, seemingly due to
context rot. This makes the zeta prompt crate's budgeted rendering
usable by the teacher, so that it can truncate the list of excerpts.
I've also cleaned up the implementation of zeta_prompt's
`format_related_files_within_budget`, and changed the behavior so that
it filters the the excerpts by priority but renders the files in their
original order.
Release Notes:
- N/A
Previously, we were not computing excerpt regions correctly for EP
examples captured from prod. This PR fixes that, and also simplifies the
data flow in the EP CLI. Examples either come from a concise spec (like
the markdown evals), or are collected from prod. Either way, we compute
from them a `ZetaPromptInput`, and the downstream steps like
prompt-formatting and scoring are derived from that.
Release Notes:
- N/A
---------
Co-authored-by: Ben Kunkle <ben@zed.dev>
This will not affect how Zeta 2 behaves in production until we update
Cloud to pull in the changes to the `zeta_prompt` crate. But from some
early testing, it seems to improve behavior, not worsen it, even though
the editable region size differs from the currently-deployed model's
training data.
Release Notes:
- N/A
---------
Co-authored-by: Oleksiy Syvokon <oleksiy.syvokon@gmail.com>
Co-authored-by: Ben Kunkle <ben@zed.dev>
Co-authored-by: Zed Zippy <234243425+zed-zippy[bot]@users.noreply.github.com>
Duplicates are defined as cursor positions that have an approximate
Jaccard similarity greater than 0.5 (over token 5-grams).
From the resulting clusters of near-duplicates, we select up to N
examples that are maximally different from each other.
Release Notes:
- N/A
Closes #ISSUE
Before you mark this PR as ready for review, make sure that you have:
- [ ] Added a solid test coverage and/or screenshots from doing manual
testing
- [ ] Done a self-review taking into account security and performance
aspects
- [ ] Aligned any UI changes with the [UI
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
Release Notes:
- N/A *or* Added/Fixed/Improved ...
Closes #ISSUE
Before you mark this PR as ready for review, make sure that you have:
- [ ] Added a solid test coverage and/or screenshots from doing manual
testing
- [ ] Done a self-review taking into account security and performance
aspects
- [ ] Aligned any UI changes with the [UI
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
Release Notes:
- N/A *or* Added/Fixed/Improved ...
Closes #ISSUE
Before you mark this PR as ready for review, make sure that you have:
- [ ] Added a solid test coverage and/or screenshots from doing manual
testing
- [ ] Done a self-review taking into account security and performance
aspects
- [ ] Aligned any UI changes with the [UI
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
Release Notes:
- N/A *or* Added/Fixed/Improved ...
Release Notes:
- Added the ability to use a self-hosted OpenAI-compatible server for
edit predictions.
---------
Co-authored-by: Ben Kunkle <ben@zed.dev>
Co-authored-by: Zed Zippy <234243425+zed-zippy[bot]@users.noreply.github.com>
#2874 on steroids
Before you mark this PR as ready for review, make sure that you have:
- [ ] Added a solid test coverage and/or screenshots from doing manual
testing
- [ ] Done a self-review taking into account security and performance
aspects
- [ ] Aligned any UI changes with the [UI
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
Release Notes:
- N/A
---------
Co-authored-by: Eric Holk <eric@zed.dev>
Closes #ISSUE
Before you mark this PR as ready for review, make sure that you have:
- [ ] Added a solid test coverage and/or screenshots from doing manual
testing
- [ ] Done a self-review taking into account security and performance
aspects
- [ ] Aligned any UI changes with the [UI
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
Release Notes:
- N/A *or* Added/Fixed/Improved ...
* More conservative predictions for prose
* Explain "user accepted prediction" in the teacher prompt
* Sonnet 4.6 support
* Don't strip comments in teacher prompt's edit history
Release Notes:
- N/A
This allows us to just pull requests that have the latest EP request
schema with the `predicted` boolean on events in the edit history.
Release Notes:
- N/A
The cache isn't needed, now that we have a better way of reducing
resource consumption (disabling worktree scanning), and it adds race
conditions.
Release Notes:
- N/A
It now creates multi-turn conversation, where first two messages are the
original teacher prompt and output.
This way, we'll have changes in teacher prompt automatically applied to
repair without having to explicitly sync them.
Release Notes:
- N/A
Other changes:
- Changed tokenization to more code-aware tokenization from split-commit
- Fixed word-diff implementation which was inefficient and sometimes
incorrect
Release Notes:
- N/A
This allows us to switch the prompt format without client-side changes.
If we want to experiment with prompt formats or models other than the
currently-deployed one, we can use the raw endpoint, and do prompt
construction and output processing on the client.
This also adds an optional environment parameter to the raw endpoint, so
that we can use that endpoint in the new scheme where we're deploying to
separate environments for different zeta prompt versions.
Release Notes:
- N/A
This also fixes the isolated whitespace change metric, which is not
takes cursor position into account (it's okay to insert a newline if it
has a cursor on it)
Release Notes:
- N/A
* [x] capture and store teacher model's predicted cursor position
* [x] provide cursor position to student during distillation
* [x] eval cursor positions
* [x] parse and apply cursor position predictions at runtime
Release Notes:
- N/A
Max Brunsfeld