LiveKit Cloud rejects room-join tokens as revoked when their `nbf`
predates a participant revocation. Zed generated those LiveKit JWTs with
`nbf: 0`, so a fresh participant token minted after stale connection
cleanup could still appear older than the cleanup and leave a user
joined at the collab layer without audio or screen sharing. This sets
`nbf` to the issuance time for room-join tokens while leaving admin/API
tokens unchanged, and adds regression coverage at the token, mock
LiveKit, and channel rejoin layers.
Closes FR-83
Release Notes:
- Fixed calls getting stuck without audio or screen sharing after
restarting Zed and rejoining a channel.
- **Revert "livekit: Preserve tokens on channel rejoin (#59388)"**
- **Revert "call: Log LiveKit connection info refresh outcomes in retry
loop (#59205)"**
- **Revert "audio: Fix phantom presence in channels (#59195)"**
- **proto: Reserve RejoinRoomResponse field 4 after revert**
We've observed a spike of phantom collaborator issues after this fixes.
This sucks and I'm quite unhappy about it.
# Objective
- Describe the objective or issue this PR addresses.
- If you're fixing a specific issue, use "Fixes #X" for each issue as
[described in the GitHub
docs](https://docs.github.com/en/issues/tracking-your-work-with-issues/using-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword).
## Solution
- Describe the solution used to achieve the objective above.
## Testing
- Did you test these changes? If so, how?
- Are there any parts that need more testing?
- How can other people (reviewers) test your changes? Is there anything
specific they need to know?
- If relevant, what platforms did you test these changes on, and are
there any important ones you can't test?
## Self-Review Checklist:
- [ ] I've reviewed my own diff for quality, security, and reliability
- [ ] Unsafe blocks (if any) have justifying comments
- [ ] The content adheres to Zed's UI standards
([UX/UI](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
and
[icon](https://github.com/zed-industries/zed/blob/main/crates/icons/README.md)
guidelines)
- [ ] Tests cover the new/changed behavior
- [ ] Performance impact has been considered and is acceptable
## Showcase
> This section is optional. If this PR does not include a visual change
or does not add a new user-facing feature, you can delete this section.
- Help others understand the result of this PR by showcasing your
awesome work!
- If this PR includes a visual change, consider adding a screenshot,
GIF, or video
- A before/after comparison is very useful for changes to existing
features!
While a showcase should aim to be brief and digestible, you can use a
toggleable section to save space on longer showcases:
<details>
<summary>Click to view showcase</summary>
My super cool demos here
</details>
---
Release Notes:
- N/A
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- **collab: Don't revoke LiveKit token when rejoining the same channel**
- **call: Bound LiveKit reconnect attempts and decouple from collab
rejoin**
# Objective
- Describe the objective or issue this PR addresses.
- If you're fixing a specific issue, use "Fixes #X" for each issue as
[described in the GitHub
docs](https://docs.github.com/en/issues/tracking-your-work-with-issues/using-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword).
## Solution
- Describe the solution used to achieve the objective above.
## Testing
- Did you test these changes? If so, how?
- Are there any parts that need more testing?
- How can other people (reviewers) test your changes? Is there anything
specific they need to know?
- If relevant, what platforms did you test these changes on, and are
there any important ones you can't test?
## Self-Review Checklist:
- [ ] I've reviewed my own diff for quality, security, and reliability
- [ ] Unsafe blocks (if any) have justifying comments
- [ ] The content adheres to Zed's UI standards
([UX/UI](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
and
[icon](https://github.com/zed-industries/zed/blob/main/crates/icons/README.md)
guidelines)
- [ ] Tests cover the new/changed behavior
- [ ] Performance impact has been considered and is acceptable
## Showcase
> This section is optional. If this PR does not include a visual change
or does not add a new user-facing feature, you can delete this section.
- Help others understand the result of this PR by showcasing your
awesome work!
- If this PR includes a visual change, consider adding a screenshot,
GIF, or video
- A before/after comparison is very useful for changes to existing
features!
While a showcase should aim to be brief and digestible, you can use a
toggleable section to save space on longer showcases:
<details>
<summary>Click to view showcase</summary>
My super cool demos here
</details>
---
Release Notes:
- N/A or Added/Fixed/Improved ...
# Objective
You know that feeling? You join the channel, crack a banger joke to
break the ice - yet, nobody laughs. Worse yet, you're swarmed with Slack
messages like "Piotr, we can't hear you". After a while you realize that
maybe it's for the best as the joke was not as good as you've initially
thought, but still: it's possible to get into a state where other
participants see you as muted. You don't hear anybody. Bummer.
That's what happened when we were hoping to pair with @dinocosta today.
I could not hear him. However, he had some logs to share which indicated
that while collab thought he's in the room with me, live-kit was
rejecting his token as it was revoked.
What clanker helped us figure out is that there's a race-y code path
where if you disconnect from a given room abruptly, collab can put you
in a limbo.
<details><summary>Clanker's idea of what's going on</summary>
<p>
Found the whole story. Here's what those logs mean and where the bug
surface is.
What's happening
The error at room.rs:1809 is the detach_and_log_err on the task spawned
by spawn_room_connection (crates/call/src/call_impl/room.rs:1754-1810).
So this is the initial livekit::Room::connect call failing — the LiveKit
SDK internally retried 3 times (that's the retrying... (1/3) lines), got
401 invalid token: revoked every time, gave up, and the error propagated
out of the ? at line 1762 and was logged-and-dropped.
The "revoked" comes from the collab server: leave_room_for_session
(crates/collab/src/rpc.rs:4091-4095) calls
live_kit.remove_participant(livekit_room,
session.user_id().to_string()), and LiveKit Cloud treats a removed
participant's token as revoked. Crucially, the LiveKit identity is just
the user id, not the connection id, while the DB-side
leave_room(connection_id) is per-connection. leave_room_for_session
fires from three places:
1. Explicit leave (rpc.rs:1666)
2. connection_lost, after RECONNECT_TIMEOUT expires (rpc.rs:1342)
3. Stale-connection cleanup when re-joining a channel (rpc.rs:3401)
So there's a race: an old connection's delayed cleanup (path 2) can fire
after the same user has rejoined the room on a new connection and been
issued a fresh token — and remove_participant(room, user_id)
kicks/revokes the user's current LiveKit participant, because the
identity is shared across connections. Quick restart of Zed, two
instances, or a reconnect that straddles the server's RECONNECT_TIMEOUT
all set this up.
Client-side consequences (this connects to your first question)
When that initial connect fails:
- this.live_kit stays None and this.diagnostics is never created — lines
1780-1791 only run on success. So in this failure mode CallDiagnostics
doesn't exist at all; the "stats never overridden" path I described
earlier isn't even reached.
- There is no retry and no token refresh. The failure is
detach_and_log_err'd and that's the end of it. The user stays in the
room at the collab level (status is still Online, they appear in the
channel), but with no audio and no way to recover short of leaving and
rejoining.
- Even a collab-level reconnect doesn't help: RejoinRoomResponse
(crates/proto/proto/call.proto:58) carries no live_kit_connection_info,
and rejoin() never calls spawn_room_connection — the LiveKit connection
is established exactly once, in Room::new, with the token from the
original join response.
So the user-visible symptom is: you're "in" the call, everyone sees you,
but you can't hear or be heard, and there's no error surfaced in the UI
— just this log line.
If you want to fix it, the two angles are server-side (make
leave_room_for_session skip the LiveKit removal when the user still has
a live connection in that room, or scope LiveKit identities
per-connection) and client-side (treat a failed spawn_room_connection as
fatal-but-recoverable: surface it, and have rejoin request fresh
connection info instead of silently carrying on). Happy to dig into
either if you want.
</p>
</details>
So tl;dr: our token was swapped out underneath us. One way to "fix" it
was to just go into another channel, but that was a bummer.
We have a good and reliable repro for it though: `kill -9 $ZED_PID`
followed by an attempt to rejoin the same channel within 30s would
consistently put us in that state.
Put another way: you panic and thus do not send a clean "leave room"
message to LK. if you rejoin the channel within 30s (after restarting),
LK will invalidate the very token you're attempting to use (as it cleans
up old tokens).
It may also happen without crashing, but that's the most reliable way to
repro the issue. Long story short your LK token gets tainted and you
can't share the audio anyhow.
## Solution
The fix is both client and server-side.
On client's side, we now retry the reconnection with a back-off and we
grab a fresh token off of reconnect attempts.
On server's side, we share the current token on room reconnect attempts.
We've also tweaked the call diagnostics to not use `unwrap_or_default`
so much: it was hard for us to tell that we're in a totally bogus state
and we've only reached that conclusion based on source code analysis. We
now show some generic "ok you're in a borked state pls report a bug"
message instead.
## Testing
We added tests that present the flacky scenario. We've also tried to
setup an infra to test the new behaviour, but sadly, local LK instance
seems to revoke tokens more leniently than the prod..
OTOH, we are quite confident that once new collab is deployed, we'll be
able to mince the new tokens and life is going to be good. Even when
running against the current prod instance we could see that our code
changed the behaviour for the better, as now we'll actually try to use a
new token if one is provided by collab. For that to happen we need to
redeploy though.
## Self-Review Checklist:
- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [x] The content adheres to Zed's UI standards
([UX/UI](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
and
[icon](https://github.com/zed-industries/zed/blob/main/crates/icons/README.md)
guidelines)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable
Release Notes:
- Fixed a race condition that caused collab users to not receive/send
any audio to their peers.
---------
Co-authored-by: dino <dinojoaocosta@gmail.com>
Co-authored-by: Dino <dino@zed.dev>
Co-authored-by: zed-zippy[bot] <234243425+zed-zippy[bot]@users.noreply.github.com>
This PR renames the `UserId` type in the `client` crate to
`LegacyUserId`.
The `id` field on the `User` has also been renamed to `legacy_id`.
This is strictly a rename, no change in behavior.
Release Notes:
- N/A
Richard Feldman