The tokens are organization-specific (the organization id in the JWT
claims). Before this commit, the `LlmApiToken` cache was an
`Option<String>`, and `cached()` ignored the `organization_id` argument
on cache hit. Correctness relied on `RefreshLlmTokenListener` clearing
and refreshing the token when it observed an `OrganizationChanged`
event.
That leaves two windows where a token minted for the wrong organization
could be served from the cache:
- the refresh runs in a spawned task, so between the organization switch
and the task acquiring the write lock, any request calling `cached()`
would get the previous organization's token.
- `Client::authenticated_llm_request` snapshots the organization id at
call time; if the server demanded a token refresh after the user had
switched organizations mid-request, the retry path would mint a fresh
token for the old organization and write it into the shared cache,
poisoning it for all subsequent requests until the next refresh event.
The fix is to store the organization id corresponding to the token next
to it in the cache, and to check that the cached token is for the
correct organization in `cached()`.
Release Notes:
- Fixed a race where LLM and edit prediction requests made immediately
after switching organizations could be attributed to the previously
selected organization.
This PR makes it so we always pass up an organization ID when creating
an LLM token.
We should have an organization ID in all cases.
Release Notes:
- N/A
Code had been assuming (erroneously, but understandably) that
LlmApiToken::acquire would give them a valid token.
This is not true, as those tokens expire and you must call refresh
explicitly.
Add some helpers to do the retry for you, and rename acquire to cached
to be
clearer about the intent.
Closes #ISSUE
Release Notes:
- Fixed some rare cases where API requests would fail with Unauthorized
- `language_model` no longer depends on provider-specific crates such as
`anthropic` and `open_ai` (inverted dependency)
- `language_model_core` was extracted from `language_model` which
contains the types for the provider-specific crates to convert to/from.
- `gpui::SharedString` has been extracted into its own crate (still
exposed by `gpui`), so `language_model_core` and provider API crates
don't have to depend on `gpui`.
- Removes some unnecessary `&'static str` | `SharedString` -> `String`
-> `SharedString` conversions across the codebase.
- Extracts the core logic of the cloud `LanguageModelProvider` into its
own crate with simpler dependencies.
Release Notes:
- N/A
---------
Co-authored-by: John Tur <john-tur@outlook.com>
Tom Houlé