Add SSE Handler for Real-Time Leaderboard and Revise Cache Invalidation Architecture

[yulmwu]

To provide clients with a real-time scoreboard (timeline and leaderboard) using SSE (Server-Sent Events), a new SSE handler GET /api/scoreboard/stream has been added. Once a client successfully establishes a connection to this endpoint, the following events are delivered whenever there is a change to the scoreboard. (scope is reserved for future extensibility, while reason and ts are included for debugging purposes.)

event: ready
data: {}

event: scoreboard
data: {"scope":"all","reason":"submission_correct","ts":"2026-02-27T18:00:00Z"}

Previously, when changes occurred in the scoreboard, the cache was simply invalidated. Users then had to manually call /api/leaderboard, /api/leaderboard/teams, /api/timeline, or /api/timeline/teams for the data to be recalculated and cached.

Now, when an action that may affect the scoreboard occurs (such as a correct flag submission or a challenge update), an event is published to the scoreboard.events channel via Redis Pub/Sub. To prevent burst load, events are debounced by 300ms.

A Redis distributed lock is then used to ensure that only one node (one application instance among the distributed backends) performs the actual scoreboard recalculation and caching. The distributed lock uses the leaderboard:rebuild:lock key with a randomly generated token.

Once the recalculation is complete, that node publishes an event to the scoreboard.rebuilt channel. All nodes subscribe to this channel and, upon receiving the message, emit an SSE event indicating that the scoreboard has changed (more precisely, that cache recomputation has completed).

Clients then call the four scoreboard-related APIs mentioned earlier. In most cases, the recalculated results are already cached, allowing for fast responses.

Please refer to the diagram below for the detailed flow. A separate technical blog post will be published later to further explain this architecture.

---

Q1. Wouldn’t it be sufficient to handle change occurs -> cache invalidation + SSE event -> client refetch?
A. It is possible. However, when multiple clients refetch simultaneously during a cache miss, the same recalculation may be executed multiple times. In this architecture, debouncing and distributed locking reduce load and duplicate computation, and events are emitted only after the cache has been rebuilt, improving consistency.

Q2. Why not let the node that processes the change immediately recalculate and send notifications without using scoreboard.events?
A. That would work in a single-node setup. In a distributed environment, however, simultaneous requests could trigger duplicate recalculations. By standardizing events through scoreboard.events, debouncing, locking, and SSE publishing can be centrally coordinated, resulting in more stable behavior.

Q3. Shouldn’t WebSocket be used instead of SSE?
A. In this case, only server-to-client one-way notifications are required, making SSE a simpler and easier-to-operate choice. WebSocket is more appropriate when bidirectional, real-time interaction is needed.

Q4. How is the Thundering Herd problem caused by multiple SSE-connected clients addressed?
A. This architectural change was primarily introduced to address that issue. As explained earlier, SSE events are sent only after the scoreboard has been recalculated and cached in advance, which prevents the herd effect. Additionally, debouncing mitigates burst traffic when APIs that affect the scoreboard are triggered in rapid succession.

Q5. Wouldn’t it be safer to omit a TTL on the distributed lock?
A. Without a TTL, if the node holding the lock fails, the lock could remain indefinitely and block recalculation. A TTL acts as a safeguard to ensure system recoverability.

Q6. Why introduce debouncing at all? Doesn’t adding a scoreboard.events channel increase maintenance complexity?
A. That is a valid concern. However, when a CTF competition starts or just before it begins requests such as user or team creation can spike significantly. This can be especially noticeable at scale. Since scoreboard aggregation requires substantial database access, the goal is to minimize DB load and maintain stable operation even in large environments. For that reason, the debouncing strategy was introduced.

We would like to thank the anonymous Discord user who provided several suggestions for improvement and thoughtful questions.

[codecov]

Codecov Report

❌ Patch coverage is 75.54585% with 56 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/realtime/scoreboard_bus.go	71.42%	24 Missing and 16 partials ⚠️
internal/http/handlers/sse_handler.go	77.77%	6 Missing and 2 partials ⚠️
internal/http/router.go	0.00%	3 Missing ⚠️
internal/realtime/sse.go	86.95%	2 Missing and 1 partial ⚠️
internal/http/handlers/handler.go	92.59%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[Copilot]

Pull request overview

This PR introduces a Redis Pub/Sub–driven “scoreboard rebuild” pipeline and an SSE endpoint so clients can be notified when leaderboard/timeline caches have been rebuilt (instead of polling and potentially triggering duplicate recomputations across nodes).

Changes:

• Added an SSE hub and /api/scoreboard/stream handler that emits ready and scoreboard events.
• Added a LeaderboardBus that debounces scoreboard.events, uses a Redis distributed lock to rebuild caches, and then publishes scoreboard.rebuilt for SSE fanout.
• Updated handlers to publish scoreboard.events on scoreboard-affecting actions, plus docs and tests.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
internal/realtime/sse.go	In-memory subscriber hub for broadcasting SSE payloads.
internal/realtime/sse_test.go	Unit tests for subscribe/broadcast/unsubscribe behavior.
internal/realtime/leaderboard_bus.go	Redis Pub/Sub debounce + distributed lock + cache rebuild + rebuilt broadcast.
internal/realtime/leaderboard_bus_test.go	Tests for publish, lock semantics, rebuild, and debounce behavior via miniredis.
internal/http/handlers/sse_handler.go	SSE endpoint implementation for /api/scoreboard/stream.
internal/http/handlers/sse_handler_test.go	Tests that SSE stream sends ready and scoreboard events and handles nil hub.
internal/http/router.go	Wires the new SSE route and injects the SSE hub into the router.
internal/http/handlers/handler.go	Publishes scoreboard.events when scoreboard-affecting actions occur.
internal/http/handlers/handler_test.go	Adds test asserting scoreboard.events is published on scoreboard-change notification.
internal/http/integration/testenv_test.go	Updates router construction with new SSE hub argument (nil in integration env).
internal/http/integration/stacks_test.go	Updates router construction with new SSE hub argument (nil in integration env).
docs/docs/scoreboard.md	Documents the SSE endpoint and event formats.
cmd/server/main.go	Creates SSE hub + leaderboard bus and disables server WriteTimeout for SSE streaming.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[yulmwu]

Refs. https://articles.swua.kr/development/2026-02-27-development-mitigating-thundering-herd-with-redis (Blog)