Sarah WeskerandKirill Bulatov
59920bc9b5
# Objective - Zed can hang (and eventually get force-killed) when opening certain binary files, because `analyze_byte_content`'s UTF-16 heuristic misclassifies them as UTF-16LE/BE text. - Reproduced with a real-world case: a ~92 MB OTBM game map file (the binary map format used by OpenTibia/Tibia servers), which interleaves short ASCII strings with small u16 length/type fields. Its byte pattern (mostly-zero high bytes, very few control characters) passed the existing check, so Zed read the entire file, decoded it as UTF-16, and opened it as an editable buffer with tens of millions of characters and effectively no line breaks — a pathological case for the text layout/renderer that hangs or crashes the app (most noticeably on Windows). ## Solution `is_plausible_utf16_text` in `crates/language/src/file_content.rs` previously only rejected the UTF-16 hypothesis when too many code units were control characters (> 2%). That's not sufficient on its own: binary formats that interleave short ASCII fragments with small numeric fields can have a very low control-character ratio while still not being real text — most of their "characters" land on stray symbol/high-byte values rather than letters, digits, or spaces. This PR adds a second, independent requirement: at least 30% of the analyzed code units must be letters, digits, or spaces (the bulk of any real UTF-16 text sample). Both conditions now have to hold for a byte sequence to be classified as UTF-16 text — otherwise it falls through to `ByteContent::Binary`, and file loading is rejected early, as intended for binary files, instead of decoding the whole file as garbled text. ## Testing - Added `test_length_prefixed_binary_not_misdetected_as_utf16le` in `crates/worktree/src/worktree.rs`, using a synthetic byte pattern that reproduces the same statistical shape as the real file (null high bytes, low control-character ratio, no word-like low bytes) — asserts it is now classified `Binary`. - Verified against the real 92 MB `.otbm` file that triggered the bug (not committed, since it's user data): before the fix it was classified `Utf16Le`, after the fix it's classified `Binary`. - Ran the full existing `analyze_byte_content` / `is_plausible_utf16_text` test suite (`cargo test -p worktree --lib tests::`) — all 7 tests pass, including the pre-existing positive UTF-16LE/UTF-16BE detection tests, so legitimate UTF-16 files are unaffected. - Built a full `--release` binary on Windows and confirmed opening the real file now shows "Binary files are not supported" immediately instead of hanging. ## Self-Review Checklist: - [x] I've reviewed my own diff for quality, security, and reliability - [x] Unsafe blocks (if any) have justifying comments — N/A, no unsafe code - [x] The content adheres to Zed's UI standards — N/A, no UI change - [x] Tests cover the new/changed behavior - [x] Performance impact has been considered and is acceptable — only affects classification of the first 1 KB of a file, negligible cost --- Release Notes: - Fixed: Zed no longer hangs when opening certain binary files (e.g. game asset/map formats) that were previously misdetected as UTF-16 text. --------- Co-authored-by: Kirill Bulatov <kirill@zed.dev>
Zed
Welcome to Zed, a high-performance, multiplayer code editor from the creators of Atom and Tree-sitter.
Installation
On macOS, Linux, and Windows you can download Zed directly or install Zed via your local package manager (macOS/Linux/Windows).
Other platforms are not yet available:
- Web (tracking discussion)
Developing Zed
Contributing
See CONTRIBUTING.md for ways you can contribute to Zed.
Also... we're hiring! Check out our jobs page for open roles.
Licensing
Zed source code is licensed primarily under GPL-3.0-or-later, with Apache-2.0 components where marked.
License information for third party dependencies must be correctly provided for CI to pass.
We use cargo-about to automatically comply with open source licenses. If CI is failing, check the following:
- Is it showing a
no license specifiederror for a crate you've created? If so, addpublish = falseunder[package]in your crate's Cargo.toml. - Is the error
failed to satisfy license requirementsfor a dependency? If so, first determine what license the project has and whether this system is sufficient to comply with this license's requirements. If you're unsure, ask a lawyer. Once you've verified that this system is acceptable add the license's SPDX identifier to theacceptedarray inscript/licenses/zed-licenses.toml. - Is
cargo-aboutunable to find the license for a dependency? If so, add a clarification field at the end ofscript/licenses/zed-licenses.toml, as specified in the cargo-about book.
Sponsorship
Zed is developed by Zed Industries, Inc., a for-profit company.
If you’d like to financially support the project, you can do so via GitHub Sponsors. Sponsorships go directly to Zed Industries and are used as general company revenue. There are no perks or entitlements associated with sponsorship.