yenru0/zed
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
dc0bbb2b43d8a40fe0bd4664770cc4af4f907e9f
zed/crates/language/src/language.rs
T
Kirill Bulatov 318c664ec3 markdown: Cache code block syntax highlighting at parse time instead of re-highlighting every frame (#63138) 
Part of https://github.com/zed-industries/zed/issues/57349

`MarkdownElement` re-highlights every code block with tree-sitter on
every frame.
Each visible code block pays a full parse per frame, which makes agent
panel streaming and hover popovers janky once the content grows.

This PR computes code block highlights once, in the background parse
task, instead of at render time:

- Highlights are resolved into `(Range<usize>, HighlightId)` runs at
parse time; each entry stores the grammar and `HighlightMap` it was
resolved against and is served only while the grammar still holds the
pointer-identical map (`ResolvedHighlights::is_current`), so theme
changes need no invalidation hooks, and a block whose entry went stale
falls back to direct fragment highlighting at render until the next
parse.
- `Language::highlight_text_resolved` memoizes results in a
content-keyed LRU (4MB budget, 512KB per-entry cap), so the re-parses
triggered by streaming appends reuse the highlights of unchanged blocks
instead of re-running tree-sitter on them.
- Blocks larger than the per-entry cap bypass the cache and are
highlighted directly, as before.

`CostBudgetedLru` is generic rather than specialized because a follow-up
PR reuses it for per-buffer highlight caching in
`BufferSnapshot::chunks`; the capture flattening keyed by grammar index
(`HighlightCaptureRef`) is shared with that follow-up as well.

## Benchmarks

`cargo bench -p benchmarks --bench markdown_renderer` renders generated
documents that mix paragraphs, headings, tables, and Rust code blocks.

**Run 1 — baseline, `main` (`d6449a9e3f`):**

```
Markdown render/min_bytes/5000    time: [1.2956 ms 1.2974 ms 1.3016 ms]
Markdown render/min_bytes/10000   time: [1.7919 ms 1.7955 ms 1.8026 ms]
Markdown render/min_bytes/50000   time: [6.5510 ms 6.6241 ms 6.6671 ms]
Markdown render/min_bytes/250000  time: [37.644 µs 2.4819 ms 4.3791 ms]   <- bimodal already on main
```

**Run 2 — this branch:**

```
Markdown render/min_bytes/5000    time: [940.11 µs 945.28 µs 948.79 µs]
                                  change: [-29.317% -28.385% -27.607%] (p = 0.00 < 0.05)
Markdown render/min_bytes/10000   time: [1.1162 ms 1.1331 ms 1.1407 ms]
                                  change: [-40.859% -39.151% -37.719%] (p = 0.00 < 0.05)
Markdown render/min_bytes/50000   time: [2.4943 ms 2.9225 ms 3.1051 ms]
                                  change: [-80.088% -67.885% -58.058%] (p = 0.00 < 0.05)
Markdown render/min_bytes/250000  time: [43.442 µs 44.661 µs 45.514 µs]
                                  change: [-98.210% -95.806% +5.5167%] (p = 0.17 > 0.05)
```

Over the same runs, GPUI reports window draw mean 2.22 ms → 1.26 ms, p99
31.2 ms → 16.0 ms, and the share of frames that blow the frame budget
drops from 3.9% to 2.0%.

Note: at the 250,000-byte case root blocks render lazily, which makes
the measurement bimodal and inconclusive in both configurations.

Release Notes:

- Improved rendering performance of markdown code blocks (agent panel,
hover popovers, markdown preview)
2026-09-09 14:31:14 +00:00

2224 lines
78 KiB
Rust
Raw Blame History

//! The `language` crate provides a large chunk of Zed's language-related
//! features (the other big contributors being project and lsp crates that revolve around LSP features).
//! Namely, this crate:
//! - Provides [`Language`], [`Grammar`] and [`LanguageRegistry`] types that
//! use Tree-sitter to provide syntax highlighting to the editor; note though that `language` doesn't perform the highlighting by itself. It only maps ranges in a buffer to colors. Treesitter is also used for buffer outlines (lists of symbols in a buffer)
//! - Exposes [`LanguageConfig`] that describes how constructs (like brackets or line comments) should be handled by the editor for a source file of a particular language.
//!
//! Notably we do *not* assign a single language to a single file; in real world a single file can consist of multiple programming languages - HTML is a good example of that - and `language` crate tends to reflect that status quo in its API.
mod available_languages;
mod buffer;
mod diagnostic;
mod diagnostic_set;
mod file_content;
mod language_registry;
pub mod language_settings;
mod manifest;
pub mod modeline;
mod outline;
pub mod proto;
mod runnable;
mod syntax_map;
mod task_context;
mod text_diff;
mod toolchain;
#[cfg(test)]
pub mod buffer_tests;
#[cfg(test)]
mod proto_diagnostics_tests;
pub use crate::language_settings::{
AutoIndentMode, EditPredictionPromptFormat, EditPredictionsMode, IndentGuideSettings,
ZetaVersion,
};
use anyhow::{Context as _, Result};
use async_trait::async_trait;
use collections::{HashMap, HashSet};
use futures::Future;
use futures::future::LocalBoxFuture;
use futures::lock::OwnedMutexGuard;
use gpui::{App, AsyncApp, Entity, EntityId};
use http_client::HttpClient;
pub use language_core::{
SymbolKind,
highlight_cache::ResolvedHighlights,
highlight_map::{CaptureId, HighlightId, HighlightMap},
};
use futures::future::FutureExt as _;
use language_core::highlight_cache::{MAX_TEXT_HIGHLIGHT_ENTRY_BYTES, TextHighlightKey};
pub use language_core::{
BlockCommentConfig, BracketPair, BracketPairConfig, BracketPairContent, BracketsConfig,
BracketsPatternConfig, CodeLabel, CodeLabelBuilder, DebugVariablesConfig, DebuggerTextObject,
DecreaseIndentConfig, Grammar, GrammarId, HighlightsConfig, IndentConfig, InjectionConfig,
InjectionPatternConfig, JsxTagAutoCloseConfig, LanguageConfig, LanguageConfigOverride,
LanguageId, LanguageMatcher, OrderedListConfig, OutlineConfig, Override, OverrideConfig,
OverrideEntry, RedactionConfig, RunnableCapture, RunnableConfig, SoftWrap, Symbol,
TaskListConfig, TextObject, TextObjectConfig, WrapCharactersConfig, default_true,
deserialize_regex, deserialize_regex_vec, regex_json_schema, regex_vec_json_schema,
serialize_regex,
};
pub use language_registry::{
LanguageLoader, LanguageName, LanguageServerStatusUpdate, LoadedLanguage, ServerHealth,
};
use lsp::{
CodeActionKind, InitializeParams, LanguageServerBinary, LanguageServerBinaryOptions, Uri,
};
pub use manifest::{ManifestDelegate, ManifestName, ManifestProvider, ManifestQuery};
pub use modeline::{ModelineSettings, parse_modeline};
use parking_lot::Mutex;
use regex::Regex;
pub use runnable::{ResolvedRunnable, RunnableMatchCapture, RunnableRange, RunnableResolver};
use semver::Version;
use serde_json::Value;
use settings::WorktreeId;
use std::{
ffi::OsStr,
fmt::Debug,
hash::Hash,
mem,
ops::{DerefMut, Range},
path::{Path, PathBuf},
str,
sync::{Arc, LazyLock},
};
use syntax_map::{QueryCursorHandle, SyntaxSnapshot, flattened_highlight_regions};
use task::RunnableTag;
pub use task_context::{ContextLocation, ContextProvider};
pub use text_diff::{
DiffOptions, apply_diff_patch, apply_reversed_diff_patch, char_diff, line_diff, text_diff,
text_diff_with_options, unified_diff, unified_diff_with_context, unified_diff_with_offsets,
word_diff_ranges,
};
use theme::SyntaxTheme;
pub use toolchain::{
LanguageToolchainStore, LocalLanguageToolchainStore, Toolchain, ToolchainList, ToolchainLister,
ToolchainMetadata, ToolchainScope,
};
use tree_sitter::{self, QueryCursor, WasmStore, wasmtime};
use util::rel_path::RelPath;
pub use available_languages::AvailableLanguage;
pub use buffer::Operation;
pub use buffer::*;
pub use diagnostic::{
Diagnostic, DiagnosticMessage, DiagnosticSourceKind, RelatedInformation, RelatedLocation,
};
pub use diagnostic_set::{DiagnosticEntry, DiagnosticEntryRef, DiagnosticGroup};
pub use file_content::{
ByteContent, DecodedText, FILE_ANALYSIS_BYTES, analyze_byte_content, decode_text, encode_text,
};
pub use language_registry::{
BinaryStatus, LanguageNotFound, LanguageQueries, LanguageRegistry, QueryFile,
QueryFileContents, QueryFiles,
};
pub use lsp::{LanguageServerId, LanguageServerName};
pub use outline::*;
pub use syntax_map::{
OwnedSyntaxLayer, SyntaxLayer, SyntaxMapMatches, ToTreeSitterPoint, TreeSitterOptions,
};
pub use text::{AnchorRangeExt, LineEnding};
pub use tree_sitter::{Node, Parser, QueryCapture, Tree, TreeCursor};
pub(crate) fn to_settings_soft_wrap(value: language_core::SoftWrap) -> settings::SoftWrap {
match value {
language_core::SoftWrap::None => settings::SoftWrap::None,
language_core::SoftWrap::PreferLine => settings::SoftWrap::PreferLine,
language_core::SoftWrap::EditorWidth => settings::SoftWrap::EditorWidth,
language_core::SoftWrap::Bounded => settings::SoftWrap::Bounded,
}
}
static QUERY_CURSORS: Mutex<Vec<QueryCursor>> = Mutex::new(vec![]);
static PARSERS: Mutex<Vec<Parser>> = Mutex::new(vec![]);
#[ztracing::instrument(skip_all)]
pub fn with_parser<F, R>(func: F) -> R
where
F: FnOnce(&mut Parser) -> R,
{
let mut parser = PARSERS.lock().pop().unwrap_or_else(|| {
let mut parser = Parser::new();
parser
.set_wasm_store(WasmStore::new(&WASM_ENGINE).unwrap())
.unwrap();
parser
});
// Tree-sitter auto-resets the parser at the end of a successful parse,
// but the cancellation paths (progress callback returning `Break`,
// cancelled balancing) leave outstanding state on the parser. The next
// call to `parse_with_options` would then *resume* that cancelled parse
// instead of starting fresh.
parser.reset();
parser.set_included_ranges(&[]).unwrap();
let result = func(&mut parser);
PARSERS.lock().push(parser);
result
}
pub fn with_query_cursor<F, R>(func: F) -> R
where
F: FnOnce(&mut QueryCursor) -> R,
{
let mut cursor = QueryCursorHandle::new();
func(cursor.deref_mut())
}
static WASM_ENGINE: LazyLock<wasmtime::Engine> = LazyLock::new(|| {
wasmtime::Engine::new(&wasmtime::Config::new()).expect("Failed to create Wasmtime engine")
});
/// A shared grammar for plain text, exposed for reuse by downstream crates.
pub static PLAIN_TEXT: LazyLock<Arc<Language>> = LazyLock::new(|| {
Arc::new(Language::new(
LanguageConfig {
name: "Plain Text".into(),
soft_wrap: Some(SoftWrap::EditorWidth),
autoclose_before: ")]}".into(),
matcher: (LanguageMatcher {
path_suffixes: vec!["txt".to_owned()],
first_line_pattern: None,
modeline_aliases: vec!["text".to_owned(), "txt".to_owned()],
})
.into(),
brackets: BracketPairConfig {
pairs: vec![
BracketPair {
start: "(".to_string(),
end: ")".to_string(),
close: true,
surround: true,
newline: false,
},
BracketPair {
start: "[".to_string(),
end: "]".to_string(),
close: true,
surround: true,
newline: false,
},
BracketPair {
start: "{".to_string(),
end: "}".to_string(),
close: true,
surround: true,
newline: false,
},
BracketPair {
start: "\"".to_string(),
end: "\"".to_string(),
close: true,
surround: true,
newline: false,
},
BracketPair {
start: "'".to_string(),
end: "'".to_string(),
close: true,
surround: true,
newline: false,
},
],
disabled_scopes_by_bracket_ix: Default::default(),
},
..Default::default()
},
None,
))
});
pub fn symbol_kind_to_lsp(kind: SymbolKind) -> lsp::SymbolKind {
match kind {
SymbolKind::File => lsp::SymbolKind::FILE,
SymbolKind::Module => lsp::SymbolKind::MODULE,
SymbolKind::Namespace => lsp::SymbolKind::NAMESPACE,
SymbolKind::Package => lsp::SymbolKind::PACKAGE,
SymbolKind::Class => lsp::SymbolKind::CLASS,
SymbolKind::Method => lsp::SymbolKind::METHOD,
SymbolKind::Property => lsp::SymbolKind::PROPERTY,
SymbolKind::Field => lsp::SymbolKind::FIELD,
SymbolKind::Constructor => lsp::SymbolKind::CONSTRUCTOR,
SymbolKind::Enum => lsp::SymbolKind::ENUM,
SymbolKind::Interface => lsp::SymbolKind::INTERFACE,
SymbolKind::Function => lsp::SymbolKind::FUNCTION,
SymbolKind::Variable => lsp::SymbolKind::VARIABLE,
SymbolKind::Constant => lsp::SymbolKind::CONSTANT,
SymbolKind::String => lsp::SymbolKind::STRING,
SymbolKind::Number => lsp::SymbolKind::NUMBER,
SymbolKind::Boolean => lsp::SymbolKind::BOOLEAN,
SymbolKind::Array => lsp::SymbolKind::ARRAY,
SymbolKind::Object => lsp::SymbolKind::OBJECT,
SymbolKind::Key => lsp::SymbolKind::KEY,
SymbolKind::Null => lsp::SymbolKind::NULL,
SymbolKind::EnumMember => lsp::SymbolKind::ENUM_MEMBER,
SymbolKind::Struct => lsp::SymbolKind::STRUCT,
SymbolKind::Event => lsp::SymbolKind::EVENT,
SymbolKind::Operator => lsp::SymbolKind::OPERATOR,
SymbolKind::TypeParameter => lsp::SymbolKind::TYPE_PARAMETER,
}
}
pub fn lsp_to_symbol_kind(kind: lsp::SymbolKind) -> SymbolKind {
match kind {
lsp::SymbolKind::FILE => SymbolKind::File,
lsp::SymbolKind::MODULE => SymbolKind::Module,
lsp::SymbolKind::NAMESPACE => SymbolKind::Namespace,
lsp::SymbolKind::PACKAGE => SymbolKind::Package,
lsp::SymbolKind::CLASS => SymbolKind::Class,
lsp::SymbolKind::METHOD => SymbolKind::Method,
lsp::SymbolKind::PROPERTY => SymbolKind::Property,
lsp::SymbolKind::FIELD => SymbolKind::Field,
lsp::SymbolKind::CONSTRUCTOR => SymbolKind::Constructor,
lsp::SymbolKind::ENUM => SymbolKind::Enum,
lsp::SymbolKind::INTERFACE => SymbolKind::Interface,
lsp::SymbolKind::FUNCTION => SymbolKind::Function,
lsp::SymbolKind::VARIABLE => SymbolKind::Variable,
lsp::SymbolKind::CONSTANT => SymbolKind::Constant,
lsp::SymbolKind::STRING => SymbolKind::String,
lsp::SymbolKind::NUMBER => SymbolKind::Number,
lsp::SymbolKind::BOOLEAN => SymbolKind::Boolean,
lsp::SymbolKind::ARRAY => SymbolKind::Array,
lsp::SymbolKind::OBJECT => SymbolKind::Object,
lsp::SymbolKind::KEY => SymbolKind::Key,
lsp::SymbolKind::NULL => SymbolKind::Null,
lsp::SymbolKind::ENUM_MEMBER => SymbolKind::EnumMember,
lsp::SymbolKind::STRUCT => SymbolKind::Struct,
lsp::SymbolKind::EVENT => SymbolKind::Event,
lsp::SymbolKind::OPERATOR => SymbolKind::Operator,
lsp::SymbolKind::TYPE_PARAMETER => SymbolKind::TypeParameter,
_ => SymbolKind::Null,
}
}
/// Commands that the client (editor) handles locally rather than forwarding
/// to the language server. Servers embed these in code lens and code action
/// responses when they want the editor to perform a well-known UI action.
#[derive(Debug, Clone)]
pub enum ClientCommand {
/// Open a location list (references panel / peek view).
ShowLocations,
/// Schedule a task from an LSP command's arguments.
ScheduleTask(task::TaskTemplate),
}
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct Location {
pub buffer: Entity<Buffer>,
pub range: Range<Anchor>,
}
/// Context provided to LSP adapters when a user responds to a ShowMessageRequest prompt.
/// This allows adapters to intercept preference selections (like "Always" or "Never")
/// and potentially persist them to Zed's settings.
#[derive(Debug, Clone)]
pub struct PromptResponseContext {
/// The original message shown to the user
pub message: String,
/// The action (button) the user selected
pub selected_action: lsp::MessageActionItem,
}
type ServerBinaryCache = futures::lock::Mutex<Option<(bool, LanguageServerBinary)>>;
type DownloadableLanguageServerBinary = LocalBoxFuture<'static, Result<LanguageServerBinary>>;
pub type LanguageServerBinaryLocations = LocalBoxFuture<
'static,
(
Result<LanguageServerBinary>,
Option<DownloadableLanguageServerBinary>,
),
>;
/// Represents a Language Server, with certain cached sync properties.
/// Uses [`LspAdapter`] under the hood, but calls all 'static' methods
/// once at startup, and caches the results.
pub struct CachedLspAdapter {
pub name: LanguageServerName,
pub disk_based_diagnostic_sources: Vec<String>,
pub disk_based_diagnostics_progress_token: Option<String>,
language_ids: HashMap<LanguageName, String>,
pub adapter: Arc<dyn LspAdapter>,
cached_binary: Arc<ServerBinaryCache>,
}
impl Debug for CachedLspAdapter {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("CachedLspAdapter")
.field("name", &self.name)
.field(
"disk_based_diagnostic_sources",
&self.disk_based_diagnostic_sources,
)
.field(
"disk_based_diagnostics_progress_token",
&self.disk_based_diagnostics_progress_token,
)
.field("language_ids", &self.language_ids)
.finish_non_exhaustive()
}
}
impl CachedLspAdapter {
pub fn new(adapter: Arc<dyn LspAdapter>) -> Arc<Self> {
let name = adapter.name();
let disk_based_diagnostic_sources = adapter.disk_based_diagnostic_sources();
let disk_based_diagnostics_progress_token = adapter.disk_based_diagnostics_progress_token();
let language_ids = adapter.language_ids();
Arc::new(CachedLspAdapter {
name,
disk_based_diagnostic_sources,
disk_based_diagnostics_progress_token,
language_ids,
adapter,
cached_binary: Default::default(),
})
}
pub fn name(&self) -> LanguageServerName {
self.adapter.name()
}
pub async fn get_language_server_command(
self: Arc<Self>,
delegate: Arc<dyn LspAdapterDelegate>,
toolchains: Option<Toolchain>,
binary_options: LanguageServerBinaryOptions,
cx: &mut AsyncApp,
) -> LanguageServerBinaryLocations {
let cached_binary = self.cached_binary.clone().lock_owned().await;
self.adapter.clone().get_language_server_command(
delegate,
toolchains,
binary_options,
cached_binary,
cx.clone(),
)
}
pub fn code_action_kinds(&self) -> Option<Vec<CodeActionKind>> {
self.adapter.code_action_kinds()
}
pub fn process_diagnostics(
&self,
params: &mut lsp::PublishDiagnosticsParams,
server_id: LanguageServerId,
) {
self.adapter.process_diagnostics(params, server_id)
}
pub fn retain_old_diagnostic(&self, previous_diagnostic: &Diagnostic) -> bool {
self.adapter.retain_old_diagnostic(previous_diagnostic)
}
pub fn underline_diagnostic(&self, diagnostic: &lsp::Diagnostic) -> bool {
self.adapter.underline_diagnostic(diagnostic)
}
pub fn diagnostic_message_to_markdown(&self, message: &str) -> Option<String> {
self.adapter.diagnostic_message_to_markdown(message)
}
pub async fn process_completions(&self, completion_items: &mut [lsp::CompletionItem]) {
self.adapter.process_completions(completion_items).await
}
pub async fn labels_for_completions(
&self,
completion_items: &[lsp::CompletionItem],
language: &Arc<Language>,
) -> Result<Vec<Option<CodeLabel>>> {
self.adapter
.clone()
.labels_for_completions(completion_items, language)
.await
}
pub async fn labels_for_symbols(
&self,
symbols: &[Symbol],
language: &Arc<Language>,
) -> Result<Vec<Option<CodeLabel>>> {
self.adapter
.clone()
.labels_for_symbols(symbols, language)
.await
}
pub fn language_id(&self, language_name: &LanguageName) -> String {
self.language_ids
.get(language_name)
.cloned()
.unwrap_or_else(|| language_name.lsp_id())
}
pub async fn initialization_options_schema(
&self,
delegate: &Arc<dyn LspAdapterDelegate>,
cx: &mut AsyncApp,
) -> Option<serde_json::Value> {
self.adapter
.clone()
.initialization_options_schema(
delegate,
self.cached_binary.clone().lock_owned().await,
cx,
)
.await
}
pub async fn settings_schema(
&self,
delegate: &Arc<dyn LspAdapterDelegate>,
cx: &mut AsyncApp,
) -> Option<serde_json::Value> {
self.adapter
.clone()
.settings_schema(delegate, self.cached_binary.clone().lock_owned().await, cx)
.await
}
pub fn process_prompt_response(&self, context: &PromptResponseContext, cx: &mut AsyncApp) {
self.adapter.process_prompt_response(context, cx)
}
}
/// [`LspAdapterDelegate`] allows [`LspAdapter]` implementations to interface with the application
// e.g. to display a notification or fetch data from the web.
#[async_trait]
pub trait LspAdapterDelegate: Send + Sync {
fn show_notification(&self, message: &str, cx: &mut App);
fn http_client(&self) -> Arc<dyn HttpClient>;
fn worktree_id(&self) -> WorktreeId;
fn worktree_root_path(&self) -> &Path;
fn resolve_relative_path(&self, path: PathBuf) -> PathBuf;
fn status_source_id(&self) -> EntityId;
fn update_status(&self, language: LanguageServerName, status: BinaryStatus);
fn registered_lsp_adapters(&self) -> Vec<Arc<dyn LspAdapter>>;
async fn language_server_download_dir(&self, name: &LanguageServerName) -> Option<Arc<Path>>;
async fn npm_package_installed_version(
&self,
package_name: &str,
) -> Result<Option<(PathBuf, Version)>>;
async fn which(&self, command: &OsStr) -> Option<PathBuf>;
async fn shell_env(&self) -> HashMap<String, String>;
async fn read_text_file(&self, path: &RelPath) -> Result<String>;
async fn try_exec(&self, binary: LanguageServerBinary) -> Result<()>;
}
#[async_trait(?Send)]
pub trait LspAdapter: 'static + Send + Sync + DynLspInstaller {
fn name(&self) -> LanguageServerName;
fn process_diagnostics(&self, _: &mut lsp::PublishDiagnosticsParams, _: LanguageServerId) {}
/// When processing new `lsp::PublishDiagnosticsParams` diagnostics, whether to retain previous one(s) or not.
fn retain_old_diagnostic(&self, _previous_diagnostic: &Diagnostic) -> bool {
false
}
/// Whether to underline a given diagnostic or not, when rendering in the editor.
///
/// https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#diagnosticTag
/// states that
/// > Clients are allowed to render diagnostics with this tag faded out instead of having an error squiggle.
/// for the unnecessary diagnostics, so do not underline them.
fn underline_diagnostic(&self, _diagnostic: &lsp::Diagnostic) -> bool {
true
}
/// Post-processes completions provided by the language server.
async fn process_completions(&self, _: &mut [lsp::CompletionItem]) {}
fn diagnostic_message_to_markdown(&self, _message: &str) -> Option<String> {
None
}
async fn labels_for_completions(
self: Arc<Self>,
completions: &[lsp::CompletionItem],
language: &Arc<Language>,
) -> Result<Vec<Option<CodeLabel>>> {
let mut labels = Vec::new();
for (ix, completion) in completions.iter().enumerate() {
let label = self.label_for_completion(completion, language).await;
if let Some(label) = label {
labels.resize(ix + 1, None);
*labels.last_mut().unwrap() = Some(label);
}
}
Ok(labels)
}
async fn label_for_completion(
&self,
_: &lsp::CompletionItem,
_: &Arc<Language>,
) -> Option<CodeLabel> {
None
}
async fn labels_for_symbols(
self: Arc<Self>,
symbols: &[Symbol],
language: &Arc<Language>,
) -> Result<Vec<Option<CodeLabel>>> {
let mut labels = Vec::new();
for (ix, symbol) in symbols.iter().enumerate() {
let label = self.label_for_symbol(symbol, language).await;
if let Some(label) = label {
labels.resize(ix + 1, None);
*labels.last_mut().unwrap() = Some(label);
}
}
Ok(labels)
}
async fn label_for_symbol(
&self,
_symbol: &Symbol,
_language: &Arc<Language>,
) -> Option<CodeLabel> {
None
}
/// Returns initialization options that are going to be sent to a LSP server as a part of [`lsp::InitializeParams`]
async fn initialization_options(
self: Arc<Self>,
_: &Arc<dyn LspAdapterDelegate>,
_cx: &mut AsyncApp,
) -> Result<Option<Value>> {
Ok(None)
}
/// Returns the JSON schema of the initialization_options for the language server.
async fn initialization_options_schema(
self: Arc<Self>,
_delegate: &Arc<dyn LspAdapterDelegate>,
_cached_binary: OwnedMutexGuard<Option<(bool, LanguageServerBinary)>>,
_cx: &mut AsyncApp,
) -> Option<serde_json::Value> {
None
}
/// Returns the JSON schema of the settings for the language server.
/// This corresponds to the `settings` field in `LspSettings`, which is used
/// to respond to `workspace/configuration` requests from the language server.
async fn settings_schema(
self: Arc<Self>,
_delegate: &Arc<dyn LspAdapterDelegate>,
_cached_binary: OwnedMutexGuard<Option<(bool, LanguageServerBinary)>>,
_cx: &mut AsyncApp,
) -> Option<serde_json::Value> {
None
}
async fn workspace_configuration(
self: Arc<Self>,
_: &Arc<dyn LspAdapterDelegate>,
_: Option<Toolchain>,
_: Option<Uri>,
_cx: &mut AsyncApp,
) -> Result<Value> {
Ok(serde_json::json!({}))
}
async fn additional_initialization_options(
self: Arc<Self>,
_target_language_server_id: LanguageServerName,
_: &Arc<dyn LspAdapterDelegate>,
) -> Result<Option<Value>> {
Ok(None)
}
async fn additional_workspace_configuration(
self: Arc<Self>,
_target_language_server_id: LanguageServerName,
_: &Arc<dyn LspAdapterDelegate>,
_cx: &mut AsyncApp,
) -> Result<Option<Value>> {
Ok(None)
}
/// Returns a list of code actions supported by a given LspAdapter
fn code_action_kinds(&self) -> Option<Vec<CodeActionKind>> {
None
}
fn disk_based_diagnostic_sources(&self) -> Vec<String> {
Default::default()
}
fn disk_based_diagnostics_progress_token(&self) -> Option<String> {
None
}
fn language_ids(&self) -> HashMap<LanguageName, String> {
HashMap::default()
}
/// Support custom initialize params.
fn prepare_initialize_params(
&self,
original: InitializeParams,
_: &App,
) -> Result<InitializeParams> {
Ok(original)
}
fn client_command(
&self,
_command_name: &str,
_arguments: &[serde_json::Value],
) -> Option<ClientCommand> {
None
}
/// Method only implemented by the default JSON language server adapter.
/// Used to provide dynamic reloading of the JSON schemas used to
/// provide autocompletion and diagnostics in Zed setting and keybind
/// files
fn is_primary_zed_json_schema_adapter(&self) -> bool {
false
}
/// True for the extension adapter and false otherwise.
fn is_extension(&self) -> bool {
false
}
/// Called when a user responds to a ShowMessageRequest from this language server.
/// This allows adapters to intercept preference selections (like "Always" or "Never")
/// for settings that should be persisted to Zed's settings file.
fn process_prompt_response(&self, _context: &PromptResponseContext, _cx: &mut AsyncApp) {}
}
pub trait LspInstaller {
type BinaryVersion;
fn check_if_user_installed(
&self,
_: &Arc<dyn LspAdapterDelegate>,
_: Option<Toolchain>,
_: &AsyncApp,
) -> impl Future<Output = Option<LanguageServerBinary>> {
async { None }
}
fn fetch_latest_server_version(
&self,
delegate: &Arc<dyn LspAdapterDelegate>,
pre_release: bool,
cx: &mut AsyncApp,
) -> impl Future<Output = Result<Self::BinaryVersion>>;
fn check_if_version_installed(
&self,
_version: &Self::BinaryVersion,
_container_dir: &PathBuf,
_delegate: &Arc<dyn LspAdapterDelegate>,
) -> impl Send + Future<Output = Option<LanguageServerBinary>> + use<Self> {
async { None }
}
fn fetch_server_binary(
&self,
latest_version: Self::BinaryVersion,
container_dir: PathBuf,
_delegate: &Arc<dyn LspAdapterDelegate>,
) -> impl Send + Future<Output = Result<LanguageServerBinary>> + use<Self>;
fn cached_server_binary(
&self,
container_dir: PathBuf,
delegate: &dyn LspAdapterDelegate,
) -> impl Future<Output = Option<LanguageServerBinary>>;
}
#[async_trait(?Send)]
pub trait DynLspInstaller {
async fn try_fetch_server_binary(
&self,
delegate: &Arc<dyn LspAdapterDelegate>,
container_dir: PathBuf,
pre_release: bool,
cx: &mut AsyncApp,
) -> Result<LanguageServerBinary>;
fn get_language_server_command(
self: Arc<Self>,
delegate: Arc<dyn LspAdapterDelegate>,
toolchains: Option<Toolchain>,
binary_options: LanguageServerBinaryOptions,
cached_binary: OwnedMutexGuard<Option<(bool, LanguageServerBinary)>>,
cx: AsyncApp,
) -> LanguageServerBinaryLocations;
}
#[async_trait(?Send)]
impl<LI, BinaryVersion> DynLspInstaller for LI
where
BinaryVersion: Send + Sync,
LI: LspInstaller<BinaryVersion = BinaryVersion> + LspAdapter,
{
async fn try_fetch_server_binary(
&self,
delegate: &Arc<dyn LspAdapterDelegate>,
container_dir: PathBuf,
pre_release: bool,
cx: &mut AsyncApp,
) -> Result<LanguageServerBinary> {
let name = self.name();
log::debug!("fetching latest version of language server {:?}", name.0);
delegate.update_status(name.clone(), BinaryStatus::CheckingForUpdate);
let latest_version = self
.fetch_latest_server_version(delegate, pre_release, cx)
.await?;
if let Some(binary) = cx
.background_executor()
.spawn(self.check_if_version_installed(&latest_version, &container_dir, &delegate))
.await
{
log::debug!("language server {:?} is already installed", name.0);
delegate.update_status(name.clone(), BinaryStatus::None);
Ok(binary)
} else {
log::debug!("downloading language server {:?}", name.0);
delegate.update_status(name.clone(), BinaryStatus::Downloading);
let binary = cx
.background_executor()
.spawn(self.fetch_server_binary(latest_version, container_dir, delegate))
.await;
delegate.update_status(name.clone(), BinaryStatus::None);
binary
}
}
fn get_language_server_command(
self: Arc<Self>,
delegate: Arc<dyn LspAdapterDelegate>,
toolchain: Option<Toolchain>,
binary_options: LanguageServerBinaryOptions,
mut cached_binary: OwnedMutexGuard<Option<(bool, LanguageServerBinary)>>,
mut cx: AsyncApp,
) -> LanguageServerBinaryLocations {
async move {
let cached_binary_deref = cached_binary.deref_mut();
// First we check whether the adapter can give us a user-installed binary.
// If so, we do *not* want to cache that, because each worktree might give us a different
// binary:
//
// worktree 1: user-installed at `.bin/gopls`
// worktree 2: user-installed at `~/bin/gopls`
// worktree 3: no gopls found in PATH -> fallback to Zed installation
//
// We only want to cache when we fall back to the global one,
// because we don't want to download and overwrite our global one
// for each worktree we might have open.
if binary_options.allow_path_lookup
&& let Some(binary) = self
.check_if_user_installed(&delegate, toolchain, &mut cx)
.await
{
log::info!(
"found user-installed language server for {}. path: {:?}, arguments: {:?}",
self.name().0,
binary.path,
binary.arguments
);
return (Ok(binary), None);
}
if let Some((pre_release, cached_binary)) = cached_binary_deref
&& *pre_release == binary_options.pre_release
{
return (Ok(cached_binary.clone()), None);
}
if !binary_options.allow_binary_download {
return (
Err(anyhow::anyhow!("downloading language servers disabled")),
None,
);
}
let Some(container_dir) = delegate.language_server_download_dir(&self.name()).await
else {
return (
Err(anyhow::anyhow!("no language server download dir defined")),
None,
);
};
let last_downloaded_binary = self
.cached_server_binary(container_dir.to_path_buf(), delegate.as_ref())
.await
.context(
"did not find existing language server binary, falling back to downloading",
);
let download_binary = async move {
let mut binary = self
.try_fetch_server_binary(
&delegate,
container_dir.to_path_buf(),
binary_options.pre_release,
&mut cx,
)
.await;
if let Err(error) = binary.as_ref() {
if let Some(prev_downloaded_binary) = self
.cached_server_binary(container_dir.to_path_buf(), delegate.as_ref())
.await
{
log::info!(
"failed to fetch newest version of language server {:?}. \
error: {:?}, falling back to using {:?}",
self.name(),
error,
prev_downloaded_binary.path
);
binary = Ok(prev_downloaded_binary);
} else {
delegate.update_status(
self.name(),
BinaryStatus::Failed {
error: format!("{error:?}"),
},
);
}
}
if let Ok(binary) = &binary {
*cached_binary = Some((binary_options.pre_release, binary.clone()));
}
binary
}
.boxed_local();
(last_downloaded_binary, Some(download_binary))
}
.boxed_local()
}
}
/// Represents a language for the given range. Some languages (e.g. HTML)
/// interleave several languages together, thus a single buffer might actually contain
/// several nested scopes.
#[derive(Clone, Debug)]
pub struct LanguageScope {
language: Arc<Language>,
override_id: Option<u32>,
}
#[doc(hidden)]
#[cfg(any(test, feature = "test-support"))]
pub struct FakeLspAdapter {
pub name: &'static str,
pub initialization_options: Option<Value>,
pub additional_initialization_options: HashMap<LanguageServerName, Value>,
pub additional_workspace_configuration: HashMap<LanguageServerName, Value>,
pub prettier_plugins: Vec<&'static str>,
pub disk_based_diagnostics_progress_token: Option<String>,
pub disk_based_diagnostics_sources: Vec<String>,
pub language_server_binary: LanguageServerBinary,
pub capabilities: lsp::ServerCapabilities,
pub initializer: Option<Box<dyn 'static + Send + Sync + Fn(&mut lsp::FakeLanguageServer)>>,
pub label_for_completion: Option<
Box<
dyn 'static
+ Send
+ Sync
+ Fn(&lsp::CompletionItem, &Arc<Language>) -> Option<CodeLabel>,
>,
>,
}
pub struct Language {
pub(crate) id: LanguageId,
pub(crate) config: LanguageConfig,
pub(crate) grammar: Option<Arc<Grammar>>,
pub(crate) context_provider: Option<Arc<dyn ContextProvider>>,
pub(crate) toolchain: Option<Arc<dyn ToolchainLister>>,
pub(crate) manifest_name: Option<ManifestName>,
}
impl Language {
pub fn new(config: LanguageConfig, ts_language: Option<tree_sitter::Language>) -> Self {
Self::new_with_id(LanguageId::new(), config, ts_language)
}
pub fn id(&self) -> LanguageId {
self.id
}
fn new_with_id(
id: LanguageId,
config: LanguageConfig,
ts_language: Option<tree_sitter::Language>,
) -> Self {
Self {
id,
config,
grammar: ts_language.map(|ts_language| Arc::new(Grammar::new(ts_language))),
context_provider: None,
toolchain: None,
manifest_name: None,
}
}
pub fn with_context_provider(mut self, provider: Option<Arc<dyn ContextProvider>>) -> Self {
self.context_provider = provider;
self
}
pub fn with_toolchain_lister(mut self, provider: Option<Arc<dyn ToolchainLister>>) -> Self {
self.toolchain = provider;
self
}
pub fn with_manifest(mut self, name: Option<ManifestName>) -> Self {
self.manifest_name = name;
self
}
pub fn with_queries(mut self, queries: LanguageQueries) -> Result<Self> {
if let Some(grammar) = self.grammar.take() {
let grammar =
Arc::try_unwrap(grammar).map_err(|_| anyhow::anyhow!("cannot mutate grammar"))?;
let grammar = grammar.with_queries(queries, &mut self.config)?;
self.grammar = Some(Arc::new(grammar));
}
Ok(self)
}
pub fn with_highlights_query(self, source: &str) -> Result<Self> {
self.with_grammar_query(|grammar| grammar.with_highlights_query(source))
}
pub fn with_runnable_query(self, source: &str) -> Result<Self> {
self.with_grammar_query(|grammar| grammar.with_runnable_query(source))
}
pub fn with_outline_query(self, source: &str) -> Result<Self> {
self.with_grammar_query_and_name(|grammar, name| grammar.with_outline_query(source, name))
}
pub fn with_text_object_query(self, source: &str) -> Result<Self> {
self.with_grammar_query_and_name(|grammar, name| {
grammar.with_text_object_query(source, name)
})
}
pub fn with_debug_variables_query(self, source: &str) -> Result<Self> {
self.with_grammar_query_and_name(|grammar, name| {
grammar.with_debug_variables_query(source, name)
})
}
pub fn with_brackets_query(self, source: &str) -> Result<Self> {
self.with_grammar_query_and_name(|grammar, name| grammar.with_brackets_query(source, name))
}
pub fn with_indents_query(self, source: &str) -> Result<Self> {
self.with_grammar_query_and_name(|grammar, name| grammar.with_indents_query(source, name))
}
pub fn with_injection_query(self, source: &str) -> Result<Self> {
self.with_grammar_query_and_name(|grammar, name| grammar.with_injection_query(source, name))
}
pub fn with_override_query(mut self, source: &str) -> Result<Self> {
if let Some(grammar_arc) = self.grammar.take() {
let grammar = Arc::try_unwrap(grammar_arc)
.map_err(|_| anyhow::anyhow!("cannot mutate grammar"))?;
let grammar = grammar.with_override_query(
source,
&self.config.name,
&self.config.overrides,
&mut self.config.brackets,
&self.config.scope_opt_in_language_servers,
)?;
self.grammar = Some(Arc::new(grammar));
}
Ok(self)
}
pub fn with_redaction_query(self, source: &str) -> Result<Self> {
self.with_grammar_query_and_name(|grammar, name| grammar.with_redaction_query(source, name))
}
fn with_grammar_query(
mut self,
build: impl FnOnce(Grammar) -> Result<Grammar>,
) -> Result<Self> {
if let Some(grammar_arc) = self.grammar.take() {
let grammar = Arc::try_unwrap(grammar_arc)
.map_err(|_| anyhow::anyhow!("cannot mutate grammar"))?;
self.grammar = Some(Arc::new(build(grammar)?));
}
Ok(self)
}
fn with_grammar_query_and_name(
mut self,
build: impl FnOnce(Grammar, &LanguageName) -> Result<Grammar>,
) -> Result<Self> {
if let Some(grammar_arc) = self.grammar.take() {
let grammar = Arc::try_unwrap(grammar_arc)
.map_err(|_| anyhow::anyhow!("cannot mutate grammar"))?;
self.grammar = Some(Arc::new(build(grammar, &self.config.name)?));
}
Ok(self)
}
pub fn name(&self) -> LanguageName {
self.config.name.clone()
}
pub fn manifest(&self) -> Option<&ManifestName> {
self.manifest_name.as_ref()
}
pub fn code_fence_block_name(&self) -> Arc<str> {
self.config
.code_fence_block_name
.clone()
.unwrap_or_else(|| self.config.name.as_ref().to_lowercase().into())
}
pub fn matches_kernel_language(&self, kernel_language: &str) -> bool {
let kernel_language_lower = kernel_language.to_lowercase();
if self.code_fence_block_name().to_lowercase() == kernel_language_lower {
return true;
}
if self.config.name.as_ref().to_lowercase() == kernel_language_lower {
return true;
}
self.config
.kernel_language_names
.iter()
.any(|name| name.to_lowercase() == kernel_language_lower)
}
pub fn context_provider(&self) -> Option<Arc<dyn ContextProvider>> {
self.context_provider.clone()
}
pub fn toolchain_lister(&self) -> Option<Arc<dyn ToolchainLister>> {
self.toolchain.clone()
}
pub fn highlight_text<'a>(
self: &'a Arc<Self>,
text: &'a Rope,
range: Range<usize>,
) -> Vec<(Range<usize>, HighlightId)> {
self.highlight_text_resolved(text, range).runs.to_vec()
}
pub fn highlight_text_resolved(
self: &Arc<Self>,
text: &Rope,
range: Range<usize>,
) -> ResolvedHighlights {
let Some(grammar) = &self.grammar else {
return ResolvedHighlights::default();
};
let Some(highlights_config) = &grammar.highlights_config else {
return ResolvedHighlights::default();
};
let highlights = if text.len() > MAX_TEXT_HIGHLIGHT_ENTRY_BYTES {
self.compute_resolved_highlights(grammar, text)
} else {
let key = TextHighlightKey::new(text.chunks(), text.len());
match highlights_config
.text_highlight_cache
.get(&key, text.chunks())
{
Some(highlights) => highlights,
None => highlights_config.text_highlight_cache.insert(
key,
Arc::from(text.chunks().collect::<String>()),
self.compute_resolved_highlights(grammar, text),
),
}
};
if range.start == 0 && range.end >= text.len() {
return highlights;
}
ResolvedHighlights {
sources: highlights.sources.clone(),
runs: highlights
.runs
.iter()
.filter(|(run_range, _)| run_range.start < range.end && run_range.end > range.start)
.map(|(run_range, highlight_id)| {
(
run_range.start.max(range.start) - range.start
..run_range.end.min(range.end) - range.start,
*highlight_id,
)
})
.collect(),
}
}
fn compute_resolved_highlights(
self: &Arc<Self>,
grammar: &Arc<Grammar>,
text: &Rope,
) -> ResolvedHighlights {
let highlight_map = grammar.highlight_map();
let tree = parse_text(grammar, text, None);
let captures =
SyntaxSnapshot::single_tree_captures(0..text.len(), text, &tree, self, |grammar| {
grammar
.highlights_config
.as_ref()
.map(|config| &config.query)
});
let mut runs = Vec::<(Range<usize>, HighlightId)>::new();
for region in flattened_highlight_regions(captures, 0..text.len()) {
let highlight_id = region
.stack
.iter()
.rev()
.find_map(|capture| highlight_map.get(capture.capture_id));
let Some(highlight_id) = highlight_id else {
continue;
};
match runs.last_mut() {
Some((last_range, last_highlight_id))
if *last_highlight_id == highlight_id
&& last_range.end == region.range.start =>
{
last_range.end = region.range.end;
}
_ => runs.push((region.range, highlight_id)),
}
}
ResolvedHighlights {
sources: [(Arc::clone(grammar), highlight_map)].into_iter().collect(),
runs: runs.into(),
}
}
pub fn path_suffixes(&self) -> &[String] {
&self.config.matcher.path_suffixes
}
pub fn should_autoclose_before(&self, c: char) -> bool {
c.is_whitespace() || self.config.autoclose_before.contains(c)
}
pub fn set_theme(&self, theme: &SyntaxTheme) {
if let Some(grammar) = self.grammar.as_ref()
&& let Some(highlights_config) = &grammar.highlights_config
{
*grammar.highlight_map.lock() =
build_highlight_map(highlights_config.query.capture_names(), theme);
}
}
pub fn grammar(&self) -> Option<&Arc<Grammar>> {
self.grammar.as_ref()
}
pub fn default_scope(self: &Arc<Self>) -> LanguageScope {
LanguageScope {
language: self.clone(),
override_id: None,
}
}
pub fn lsp_id(&self) -> String {
self.config.name.lsp_id()
}
pub fn snippet_scope_id(&self) -> String {
self.config.name.snippet_scope_id()
}
pub fn prettier_parser_name(&self) -> Option<&str> {
self.config.prettier_parser_name.as_deref()
}
pub fn config(&self) -> &LanguageConfig {
&self.config
}
}
#[inline]
pub fn build_highlight_map(capture_names: &[&str], theme: &SyntaxTheme) -> HighlightMap {
HighlightMap::from_ids(
capture_names
.iter()
.map(|capture_name| theme.highlight_id(capture_name).map(HighlightId::new)),
)
}
impl LanguageScope {
pub fn path_suffixes(&self) -> &[String] {
self.language.path_suffixes()
}
pub fn language_name(&self) -> LanguageName {
self.language.config.name.clone()
}
pub fn collapsed_placeholder(&self) -> &str {
self.language.config.collapsed_placeholder.as_ref()
}
/// Returns line prefix that is inserted in e.g. line continuations or
/// in `toggle comments` action.
pub fn line_comment_prefixes(&self) -> &[Arc<str>] {
Override::as_option(
self.config_override().map(|o| &o.line_comments),
Some(&self.language.config.line_comments),
)
.map_or([].as_slice(), |e| e.as_slice())
}
/// Config for block comments for this language.
pub fn block_comment(&self) -> Option<&BlockCommentConfig> {
Override::as_option(
self.config_override().map(|o| &o.block_comment),
self.language.config.block_comment.as_ref(),
)
}
/// Config for documentation-style block comments for this language.
pub fn documentation_comment(&self) -> Option<&BlockCommentConfig> {
self.language.config.documentation_comment.as_ref()
}
/// Returns list markers that are inserted unchanged on newline (e.g., `- `, `* `, `+ `).
pub fn unordered_list(&self) -> &[Arc<str>] {
&self.language.config.unordered_list
}
/// Returns configuration for ordered lists with auto-incrementing numbers (e.g., `1. ` becomes `2. `).
pub fn ordered_list(&self) -> &[OrderedListConfig] {
&self.language.config.ordered_list
}
/// Returns configuration for task list continuation, if any (e.g., `- [x] ` continues as `- [ ] `).
pub fn task_list(&self) -> Option<&TaskListConfig> {
self.language.config.task_list.as_ref()
}
/// Returns additional regex patterns that act as prefix markers for creating
/// boundaries during rewrapping.
///
/// By default, Zed treats as paragraph and comment prefixes as boundaries.
pub fn rewrap_prefixes(&self) -> &[Regex] {
&self.language.config.rewrap_prefixes
}
/// Returns a list of language-specific word characters.
///
/// By default, Zed treats alphanumeric characters (and '_') as word characters for
/// the purpose of actions like 'move to next word end` or whole-word search.
/// It additionally accounts for language's additional word characters.
pub fn word_characters(&self) -> Option<&HashSet<char>> {
Override::as_option(
self.config_override().map(|o| &o.word_characters),
Some(&self.language.config.word_characters),
)
}
/// Returns a list of language-specific characters that are considered part of
/// a completion query.
pub fn completion_query_characters(&self) -> Option<&HashSet<char>> {
Override::as_option(
self.config_override()
.map(|o| &o.completion_query_characters),
Some(&self.language.config.completion_query_characters),
)
}
/// Returns a list of language-specific characters that are considered part of
/// identifiers during linked editing operations.
pub fn linked_edit_characters(&self) -> Option<&HashSet<char>> {
Override::as_option(
self.config_override().map(|o| &o.linked_edit_characters),
Some(&self.language.config.linked_edit_characters),
)
}
/// Returns whether to prefer snippet `label` over `new_text` to replace text when
/// completion is accepted.
///
/// In cases like when cursor is in string or renaming existing function,
/// you don't want to expand function signature instead just want function name
/// to replace existing one.
pub fn prefers_label_for_snippet_in_completion(&self) -> bool {
self.config_override()
.and_then(|o| o.prefer_label_for_snippet)
.unwrap_or(false)
}
/// Returns a list of bracket pairs for a given language with an additional
/// piece of information about whether the particular bracket pair is currently active for a given language.
pub fn brackets(&self) -> impl Iterator<Item = (&BracketPair, bool)> {
let mut disabled_ids = self
.config_override()
.map_or(&[] as _, |o| o.disabled_bracket_ixs.as_slice());
self.language
.config
.brackets
.pairs
.iter()
.enumerate()
.map(move |(ix, bracket)| {
let mut is_enabled = true;
if let Some(next_disabled_ix) = disabled_ids.first()
&& ix == *next_disabled_ix as usize
{
disabled_ids = &disabled_ids[1..];
is_enabled = false;
}
(bracket, is_enabled)
})
}
pub fn should_autoclose_before(&self, c: char) -> bool {
c.is_whitespace() || self.language.config.autoclose_before.contains(c)
}
pub fn language_allowed(&self, name: &LanguageServerName) -> bool {
let config = &self.language.config;
let opt_in_servers = &config.scope_opt_in_language_servers;
if opt_in_servers.contains(&name.0) {
if let Some(over) = self.config_override() {
over.opt_into_language_servers.contains(&name.0)
} else {
false
}
} else {
true
}
}
pub fn override_name(&self) -> Option<&str> {
let id = self.override_id?;
let grammar = self.language.grammar.as_ref()?;
let override_config = grammar.override_config.as_ref()?;
override_config.values.get(&id).map(|e| e.name.as_str())
}
fn config_override(&self) -> Option<&LanguageConfigOverride> {
let id = self.override_id?;
let grammar = self.language.grammar.as_ref()?;
let override_config = grammar.override_config.as_ref()?;
override_config.values.get(&id).map(|e| &e.value)
}
}
impl Hash for Language {
fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
self.id.hash(state)
}
}
impl PartialEq for Language {
fn eq(&self, other: &Self) -> bool {
self.id.eq(&other.id)
}
}
impl Eq for Language {}
impl Debug for Language {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Language")
.field("name", &self.config.name)
.finish()
}
}
pub(crate) fn parse_text(grammar: &Grammar, text: &Rope, old_tree: Option<Tree>) -> Tree {
with_parser(|parser| {
parser
.set_language(&grammar.ts_language)
.expect("incompatible grammar");
let mut chunks = text.chunks_in_range(0..text.len());
parser
.parse_with_options(
&mut move |offset, _| {
chunks.seek(offset);
chunks.next().unwrap_or("").as_bytes()
},
old_tree.as_ref(),
None,
)
.unwrap()
})
}
pub trait CodeLabelExt {
fn fallback_for_completion(
item: &lsp::CompletionItem,
language: Option<&Language>,
) -> CodeLabel;
}
impl CodeLabelExt for CodeLabel {
fn fallback_for_completion(
item: &lsp::CompletionItem,
language: Option<&Language>,
) -> CodeLabel {
let highlight_id = item.kind.and_then(|kind| {
let grammar = language?.grammar()?;
use lsp::CompletionItemKind as Kind;
match kind {
Kind::CLASS => grammar.highlight_id_for_name("type"),
Kind::CONSTANT => grammar.highlight_id_for_name("constant"),
Kind::CONSTRUCTOR => grammar.highlight_id_for_name("constructor"),
Kind::ENUM => grammar
.highlight_id_for_name("enum")
.or_else(|| grammar.highlight_id_for_name("type")),
Kind::ENUM_MEMBER => grammar
.highlight_id_for_name("variant")
.or_else(|| grammar.highlight_id_for_name("property")),
Kind::FIELD => grammar.highlight_id_for_name("property"),
Kind::FUNCTION => grammar.highlight_id_for_name("function"),
Kind::INTERFACE => grammar.highlight_id_for_name("type"),
Kind::METHOD => grammar
.highlight_id_for_name("function.method")
.or_else(|| grammar.highlight_id_for_name("function")),
Kind::OPERATOR => grammar.highlight_id_for_name("operator"),
Kind::PROPERTY => grammar.highlight_id_for_name("property"),
Kind::STRUCT => grammar.highlight_id_for_name("type"),
Kind::VARIABLE => grammar.highlight_id_for_name("variable"),
Kind::KEYWORD => grammar.highlight_id_for_name("keyword"),
_ => None,
}
});
let label = &item.label;
let label_length = label.len();
let runs = highlight_id
.map(|highlight_id| vec![(0..label_length, highlight_id)])
.unwrap_or_default();
let text = if let Some(detail) = item.detail.as_deref().filter(|detail| detail != label) {
format!("{label} {detail}")
} else if let Some(description) = item
.label_details
.as_ref()
.and_then(|label_details| label_details.description.as_deref())
.filter(|description| description != label)
{
format!("{label} {description}")
} else {
label.clone()
};
let filter_range = item
.filter_text
.as_deref()
.and_then(|filter| text.find(filter).map(|ix| ix..ix + filter.len()))
.unwrap_or(0..label_length);
CodeLabel {
text,
runs,
filter_range,
}
}
}
#[cfg(any(test, feature = "test-support"))]
impl Default for FakeLspAdapter {
fn default() -> Self {
Self {
name: "the-fake-language-server",
capabilities: lsp::LanguageServer::full_capabilities(),
initializer: None,
disk_based_diagnostics_progress_token: None,
initialization_options: None,
additional_initialization_options: HashMap::default(),
additional_workspace_configuration: HashMap::default(),
disk_based_diagnostics_sources: Vec::new(),
prettier_plugins: Vec::new(),
language_server_binary: LanguageServerBinary {
path: "/the/fake/lsp/path".into(),
arguments: vec![],
env: Default::default(),
},
label_for_completion: None,
}
}
}
#[cfg(any(test, feature = "test-support"))]
impl LspInstaller for FakeLspAdapter {
type BinaryVersion = ();
async fn fetch_latest_server_version(
&self,
_: &Arc<dyn LspAdapterDelegate>,
_: bool,
_: &mut AsyncApp,
) -> Result<Self::BinaryVersion> {
unreachable!()
}
async fn check_if_user_installed(
&self,
_: &Arc<dyn LspAdapterDelegate>,
_: Option<Toolchain>,
_: &AsyncApp,
) -> Option<LanguageServerBinary> {
Some(self.language_server_binary.clone())
}
fn fetch_server_binary(
&self,
_: (),
_: PathBuf,
_: &Arc<dyn LspAdapterDelegate>,
) -> impl Send + Future<Output = Result<LanguageServerBinary>> + use<> {
async {
unreachable!();
}
}
async fn cached_server_binary(
&self,
_: PathBuf,
_: &dyn LspAdapterDelegate,
) -> Option<LanguageServerBinary> {
unreachable!();
}
}
#[cfg(any(test, feature = "test-support"))]
#[async_trait(?Send)]
impl LspAdapter for FakeLspAdapter {
fn name(&self) -> LanguageServerName {
LanguageServerName(self.name.into())
}
fn disk_based_diagnostic_sources(&self) -> Vec<String> {
self.disk_based_diagnostics_sources.clone()
}
fn disk_based_diagnostics_progress_token(&self) -> Option<String> {
self.disk_based_diagnostics_progress_token.clone()
}
async fn initialization_options(
self: Arc<Self>,
_: &Arc<dyn LspAdapterDelegate>,
_cx: &mut AsyncApp,
) -> Result<Option<Value>> {
Ok(self.initialization_options.clone())
}
async fn additional_initialization_options(
self: Arc<Self>,
target_language_server_id: LanguageServerName,
_: &Arc<dyn LspAdapterDelegate>,
) -> Result<Option<Value>> {
Ok(self
.additional_initialization_options
.get(&target_language_server_id)
.cloned())
}
async fn additional_workspace_configuration(
self: Arc<Self>,
target_language_server_id: LanguageServerName,
_: &Arc<dyn LspAdapterDelegate>,
_cx: &mut AsyncApp,
) -> Result<Option<Value>> {
Ok(self
.additional_workspace_configuration
.get(&target_language_server_id)
.cloned())
}
async fn label_for_completion(
&self,
item: &lsp::CompletionItem,
language: &Arc<Language>,
) -> Option<CodeLabel> {
let label_for_completion = self.label_for_completion.as_ref()?;
label_for_completion(item, language)
}
fn is_extension(&self) -> bool {
false
}
}
pub fn point_to_lsp(point: PointUtf16) -> lsp::Position {
lsp::Position::new(point.row, point.column)
}
pub fn point_from_lsp(point: lsp::Position) -> Unclipped<PointUtf16> {
Unclipped(PointUtf16::new(point.line, point.character))
}
pub fn range_to_lsp(range: Range<PointUtf16>) -> Result<lsp::Range> {
anyhow::ensure!(
range.start <= range.end,
"Inverted range provided to an LSP request: {:?}-{:?}",
range.start,
range.end
);
Ok(lsp::Range {
start: point_to_lsp(range.start),
end: point_to_lsp(range.end),
})
}
pub fn range_from_lsp(range: lsp::Range) -> Range<Unclipped<PointUtf16>> {
let mut start = point_from_lsp(range.start);
let mut end = point_from_lsp(range.end);
if start > end {
// We debug instead of warn so that this is not logged by default unless explicitly requested.
// Using warn would write to the log file, and since we receive an enormous amount of
// range_from_lsp calls (especially during completions), that can hang the main thread.
//
// See issue #36223.
zlog::debug!("range_from_lsp called with inverted range {start:?}-{end:?}");
mem::swap(&mut start, &mut end);
}
start..end
}
#[doc(hidden)]
#[cfg(any(test, feature = "test-support"))]
pub fn rust_lang() -> Arc<Language> {
test_language("rust", tree_sitter_rust::LANGUAGE.into())
}
#[doc(hidden)]
#[cfg(any(test, feature = "test-support"))]
pub fn json_lang() -> Arc<Language> {
test_language("json", tree_sitter_json::LANGUAGE.into())
}
#[doc(hidden)]
#[cfg(any(test, feature = "test-support"))]
pub fn markdown_lang() -> Arc<Language> {
test_language("markdown", tree_sitter_md::LANGUAGE.into())
}
#[cfg(any(test, feature = "test-support"))]
fn test_language(name: &str, grammar: tree_sitter::Language) -> Arc<Language> {
Arc::new(
Language::new(grammars::load_config(name), Some(grammar))
.with_queries(grammars::load_queries(name))
.unwrap_or_else(|error| panic!("could not parse queries for language {name}: {error}")),
)
}
#[cfg(test)]
mod tests {
use super::*;
use gpui::{TestAppContext, rgba};
use pretty_assertions::assert_matches;
#[test]
fn test_highlight_map() {
let theme = SyntaxTheme::new(
[
("function", rgba(0x100000ff)),
("function.method", rgba(0x200000ff)),
("function.async", rgba(0x300000ff)),
("variable.builtin.self.rust", rgba(0x400000ff)),
("variable.builtin", rgba(0x500000ff)),
("variable", rgba(0x600000ff)),
]
.iter()
.map(|(name, color)| (name.to_string(), (*color).into())),
);
let capture_names = &[
"function.special",
"function.async.rust",
"variable.builtin.self",
];
let map = build_highlight_map(capture_names, &theme);
assert_eq!(
theme.get_capture_name(map.get(CaptureId(0)).unwrap()),
Some("function")
);
assert_eq!(
theme.get_capture_name(map.get(CaptureId(1)).unwrap()),
Some("function.async")
);
assert_eq!(
theme.get_capture_name(map.get(CaptureId(2)).unwrap()),
Some("variable.builtin")
);
}
#[test]
fn test_highlight_text_resolves_nested_captures_with_theme_fallback() {
let language = Arc::new(
Language::new(
LanguageConfig {
name: "Rust".into(),
..LanguageConfig::default()
},
Some(tree_sitter_rust::LANGUAGE.into()),
)
.with_highlights_query(
r#"
(function_item) @function.definition
(identifier) @variable
"fn" @keyword
"#,
)
.unwrap(),
);
let theme = SyntaxTheme::new(
[
("function", rgba(0x100000ff)),
("keyword", rgba(0x200000ff)),
]
.iter()
.map(|(name, color)| (name.to_string(), (*color).into())),
);
language.set_theme(&theme);
let code = "fn main() {}";
let highlights = language.highlight_text_resolved(&Rope::from(code), 0..code.len());
assert!(highlights.is_current());
let named = highlights
.runs
.iter()
.map(|(range, highlight_id)| {
(
range.clone(),
theme.get_capture_name(*highlight_id).unwrap(),
)
})
.collect::<Vec<_>>();
assert_eq!(
named,
vec![(0..2, "keyword"), (2..12, "function")],
"an inner capture missing from the theme must fall back to its outer capture"
);
let memoized = language.highlight_text_resolved(&Rope::from(code), 0..code.len());
assert!(
Arc::ptr_eq(&highlights.runs, &memoized.runs),
"repeated highlighting of the same text must be memoized"
);
let partial = language.highlight_text_resolved(&Rope::from(code), 0..2);
assert!(
!Arc::ptr_eq(&highlights.runs, &partial.runs),
"a different range must not reuse the memoized highlights"
);
assert_eq!(partial.runs.as_ref(), &[(0..2, highlights.runs[0].1)]);
let richer_theme = SyntaxTheme::new(
[
("function", rgba(0x100000ff)),
("keyword", rgba(0x200000ff)),
("variable", rgba(0x300000ff)),
]
.iter()
.map(|(name, color)| (name.to_string(), (*color).into())),
);
language.set_theme(&richer_theme);
assert!(
!highlights.is_current(),
"a theme change must invalidate previously resolved highlights"
);
let rethemed = language.highlight_text_resolved(&Rope::from(code), 0..code.len());
assert!(rethemed.is_current());
let renamed = rethemed
.runs
.iter()
.map(|(range, highlight_id)| {
(
range.clone(),
richer_theme.get_capture_name(*highlight_id).unwrap(),
)
})
.collect::<Vec<_>>();
assert_eq!(
renamed,
vec![
(0..2, "keyword"),
(2..3, "function"),
(3..7, "variable"),
(7..12, "function"),
],
"a theme change must invalidate memoized highlights and resolve against the new theme"
);
}
#[test]
fn test_oversized_text_is_highlighted_without_caching() {
let language = rust_lang();
let theme = SyntaxTheme::new(
[
("function", rgba(0x100000ff)),
("keyword", rgba(0x200000ff)),
]
.iter()
.map(|(name, color)| (name.to_string(), (*color).into())),
);
language.set_theme(&theme);
let small_code = "fn main() {}";
let small_highlights =
language.highlight_text_resolved(&Rope::from(small_code), 0..small_code.len());
assert!(
!small_highlights.runs.is_empty(),
"texts within the size cap must be highlighted"
);
let oversized_code = format!(
"fn main() {{}}{}",
" ".repeat(MAX_TEXT_HIGHLIGHT_ENTRY_BYTES)
);
let oversized_rope = Rope::from(oversized_code.as_str());
let first_highlights =
language.highlight_text_resolved(&oversized_rope, 0..oversized_code.len());
assert_eq!(
first_highlights.runs.as_ref(),
small_highlights.runs.as_ref(),
"texts over the cache entry cap must still be highlighted"
);
let second_highlights =
language.highlight_text_resolved(&oversized_rope, 0..oversized_code.len());
assert_eq!(
second_highlights.runs.as_ref(),
first_highlights.runs.as_ref()
);
assert!(
!Arc::ptr_eq(&first_highlights.runs, &second_highlights.runs),
"texts over the cache entry cap must not be memoized"
);
}
#[test]
fn test_with_parser_resets_after_cancellation() {
use std::ops::ControlFlow;
use tree_sitter::{Language as TsLanguage, ParseOptions};
let rust_language: TsLanguage = tree_sitter_rust::LANGUAGE.into();
// Drain the shared pool so this test sees a deterministic LIFO order:
// the parser we push at the end of the first `with_parser` call is the
// one we pop at the start of the second call.
PARSERS.lock().clear();
// Large enough that tree-sitter invokes the progress callback before
// the parse completes; otherwise the cancellation never fires.
let large_input = format!("fn a() {{ {} }}", "b(c, d); e(f, g); ".repeat(5000));
let small_input = "fn z() {}";
// Cancel a parse via the progress callback. Tree-sitter retains the
// in-progress parse state on the parser (its `canceled_balancing` flag
// and/or non-empty parse stack), and the next call to
// `parse_with_options` will *resume* that parse unless the parser is
// reset first.
let cancelled = with_parser(|parser| {
parser.set_language(&rust_language).unwrap();
let bytes = large_input.as_bytes();
let mut break_immediately = |_: &_| ControlFlow::Break(());
parser.parse_with_options(
&mut |offset, _| {
if offset < bytes.len() {
&bytes[offset..]
} else {
&[]
}
},
None,
Some(ParseOptions {
progress_callback: Some(&mut break_immediately),
}),
)
});
assert!(
cancelled.is_none(),
"first parse should be cancelled by the progress callback"
);
// Deliberately do NOT call `set_language` here: tree-sitter's
// `ts_parser_set_language` internally calls `ts_parser_reset`, which
// would mask the very bug we're checking for. Instead we rely on the
// language being preserved across `parser.reset()` (it is) and verify
// that `with_parser` itself produces a clean parser for the next user.
let tree = with_parser(|parser| {
let bytes = small_input.as_bytes();
parser
.parse_with_options(
&mut |offset, _| {
if offset < bytes.len() {
&bytes[offset..]
} else {
&[]
}
},
None,
None,
)
.expect("parse of small_input should succeed")
});
assert_eq!(tree.root_node().byte_range(), 0..small_input.len());
assert_eq!(tree.root_node().kind(), "source_file");
assert!(
!tree.root_node().has_error(),
"tree should be error-free, got: {}",
tree.root_node().to_sexp()
);
}
#[gpui::test(iterations = 10)]
async fn test_language_loading(cx: &mut TestAppContext) {
let languages = LanguageRegistry::test(cx.executor());
let languages = Arc::new(languages);
languages.register_native_grammars([
("json", tree_sitter_json::LANGUAGE),
("rust", tree_sitter_rust::LANGUAGE),
]);
languages.register_test_language(LanguageConfig {
name: "JSON".into(),
grammar: Some("json".into()),
matcher: (LanguageMatcher {
path_suffixes: vec!["json".into()],
..Default::default()
})
.into(),
..Default::default()
});
languages.register_test_language(LanguageConfig {
name: "Rust".into(),
grammar: Some("rust".into()),
matcher: (LanguageMatcher {
path_suffixes: vec!["rs".into()],
..Default::default()
})
.into(),
..Default::default()
});
assert_eq!(
languages.language_names(),
&[
LanguageName::new_static("JSON"),
LanguageName::new_static("Plain Text"),
LanguageName::new_static("Rust"),
]
);
let rust1 = languages.language_for_name("Rust");
let rust2 = languages.language_for_name("Rust");
// Ensure language is still listed even if it's being loaded.
assert_eq!(
languages.language_names(),
&[
LanguageName::new_static("JSON"),
LanguageName::new_static("Plain Text"),
LanguageName::new_static("Rust"),
]
);
let (rust1, rust2) = futures::join!(rust1, rust2);
assert!(Arc::ptr_eq(&rust1.unwrap(), &rust2.unwrap()));
// Ensure language is still listed even after loading it.
assert_eq!(
languages.language_names(),
&[
LanguageName::new_static("JSON"),
LanguageName::new_static("Plain Text"),
LanguageName::new_static("Rust"),
]
);
// Loading an unknown language returns an error.
assert!(languages.language_for_name("Unknown").await.is_err());
}
#[gpui::test]
async fn test_completion_label_omits_duplicate_data() {
let regular_completion_item_1 = lsp::CompletionItem {
label: "regular1".to_string(),
detail: Some("detail1".to_string()),
label_details: Some(lsp::CompletionItemLabelDetails {
detail: None,
description: Some("description 1".to_string()),
}),
..lsp::CompletionItem::default()
};
let regular_completion_item_2 = lsp::CompletionItem {
label: "regular2".to_string(),
label_details: Some(lsp::CompletionItemLabelDetails {
detail: None,
description: Some("description 2".to_string()),
}),
..lsp::CompletionItem::default()
};
let completion_item_with_duplicate_detail_and_proper_description = lsp::CompletionItem {
detail: Some(regular_completion_item_1.label.clone()),
..regular_completion_item_1.clone()
};
let completion_item_with_duplicate_detail = lsp::CompletionItem {
detail: Some(regular_completion_item_1.label.clone()),
label_details: None,
..regular_completion_item_1.clone()
};
let completion_item_with_duplicate_description = lsp::CompletionItem {
label_details: Some(lsp::CompletionItemLabelDetails {
detail: None,
description: Some(regular_completion_item_2.label.clone()),
}),
..regular_completion_item_2.clone()
};
assert_eq!(
CodeLabel::fallback_for_completion(&regular_completion_item_1, None).text,
format!(
"{} {}",
regular_completion_item_1.label,
regular_completion_item_1.detail.unwrap()
),
"LSP completion items with both detail and label_details.description should prefer detail"
);
assert_eq!(
CodeLabel::fallback_for_completion(&regular_completion_item_2, None).text,
format!(
"{} {}",
regular_completion_item_2.label,
regular_completion_item_2
.label_details
.as_ref()
.unwrap()
.description
.as_ref()
.unwrap()
),
"LSP completion items without detail but with label_details.description should use that"
);
assert_eq!(
CodeLabel::fallback_for_completion(
&completion_item_with_duplicate_detail_and_proper_description,
None
)
.text,
format!(
"{} {}",
regular_completion_item_1.label,
regular_completion_item_1
.label_details
.as_ref()
.unwrap()
.description
.as_ref()
.unwrap()
),
"LSP completion items with both detail and label_details.description should prefer description only if the detail duplicates the completion label"
);
assert_eq!(
CodeLabel::fallback_for_completion(&completion_item_with_duplicate_detail, None).text,
regular_completion_item_1.label,
"LSP completion items with duplicate label and detail, should omit the detail"
);
assert_eq!(
CodeLabel::fallback_for_completion(&completion_item_with_duplicate_description, None)
.text,
regular_completion_item_2.label,
"LSP completion items with duplicate label and detail, should omit the detail"
);
}
#[test]
fn test_deserializing_comments_backwards_compat() {
// current version of `block_comment` and `documentation_comment` work
{
let config: LanguageConfig = ::toml::from_str(
r#"
name = "Foo"
block_comment = { start = "a", end = "b", prefix = "c", tab_size = 1 }
documentation_comment = { start = "d", end = "e", prefix = "f", tab_size = 2 }
"#,
)
.unwrap();
assert_matches!(config.block_comment, Some(BlockCommentConfig { .. }));
assert_matches!(
config.documentation_comment,
Some(BlockCommentConfig { .. })
);
let block_config = config.block_comment.unwrap();
assert_eq!(block_config.start.as_ref(), "a");
assert_eq!(block_config.end.as_ref(), "b");
assert_eq!(block_config.prefix.as_ref(), "c");
assert_eq!(block_config.tab_size, 1);
let doc_config = config.documentation_comment.unwrap();
assert_eq!(doc_config.start.as_ref(), "d");
assert_eq!(doc_config.end.as_ref(), "e");
assert_eq!(doc_config.prefix.as_ref(), "f");
assert_eq!(doc_config.tab_size, 2);
}
// former `documentation` setting is read into `documentation_comment`
{
let config: LanguageConfig = ::toml::from_str(
r#"
name = "Foo"
documentation = { start = "a", end = "b", prefix = "c", tab_size = 1}
"#,
)
.unwrap();
assert_matches!(
config.documentation_comment,
Some(BlockCommentConfig { .. })
);
let config = config.documentation_comment.unwrap();
assert_eq!(config.start.as_ref(), "a");
assert_eq!(config.end.as_ref(), "b");
assert_eq!(config.prefix.as_ref(), "c");
assert_eq!(config.tab_size, 1);
}
// old block_comment format is read into BlockCommentConfig
{
let config: LanguageConfig = ::toml::from_str(
r#"
name = "Foo"
block_comment = ["a", "b"]
"#,
)
.unwrap();
assert_matches!(config.block_comment, Some(BlockCommentConfig { .. }));
let config = config.block_comment.unwrap();
assert_eq!(config.start.as_ref(), "a");
assert_eq!(config.end.as_ref(), "b");
assert_eq!(config.prefix.as_ref(), "");
assert_eq!(config.tab_size, 0);
}
}
}
Reference in new issue View Git Blame Copy Permalink