feat(telemetry): add CodeQ telemetry collection system #5
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add qBraid-specific telemetry module to collect session data for analysis and model improvement. Components: - config: Add qbraid.telemetry config section under a separate namespace to maintain upstream compatibility when merging from opencode - types: TypeScript types matching the telemetry service schema - sanitizer: Strip secrets, redact sensitive files, truncate large content - signals: Track implicit feedback (retries, errors, abandonment) - uploader: Batch and upload data with retry logic and offline handling - consent: Tier-based consent (free=forced opt-in, paid=opt-in) - collector: Main module that coordinates all components Design choices for upstream compatibility: - All telemetry code is in a separate src/telemetry/ directory - Config uses qbraid.* namespace that upstream will ignore - No modifications to existing session/processor logic yet Implements: [CodeQ] Add telemetry config schema Implements: [CodeQ] Implement TelemetryCollector module Implements: [CodeQ] Implement content sanitizer Implements: [CodeQ] Implement batch uploader with retry
Complete the telemetry integration with OpenCode: Integration module (integration.ts): - Subscribe to Session, Message, and File events from Event Bus - Automatic tracking of sessions, turns, tool calls, and file changes - Uses Instance.state for automatic cleanup on disposal - Graceful shutdown flushes pending telemetry data Bootstrap integration: - Initialize telemetry during InstanceBootstrap - Telemetry is now automatically enabled on startup (respects consent) - Errors during initialization are logged but don't block startup Exports: - Telemetry.initIntegration() - Initialize with Event Bus integration - Telemetry.shutdownIntegration() - Explicit shutdown (automatic via Instance.dispose) - Telemetry.completeTurn() - Finalize assistant response - Telemetry.userMessage() - Record user input - Telemetry.retry() - Record turn retry Implements: [CodeQ] Wire up telemetry module
- Fix tool call status: now properly detects error state - Remove unused Bus import from collector.ts
|
Thanks for your contribution! This PR doesn't have a linked issue. All PRs must reference an existing issue. Please:
See CONTRIBUTING.md for details. |
- Update integration to properly record user messages with content - Record assistant responses with model ID and token counts - Track tool calls with status, duration, and input/output sizes - Finalize turns and upload to telemetry service - Deduplicate user message recording - Get user info (userId, organizationId) from consent endpoint - Read qBraid API key from config (provider.qbraid.options.apiKey) - Update default telemetry endpoint to production Cloud Run URL The telemetry now captures: - Full user message content (sanitized) - Full assistant response content - Tool call metadata (name, status, duration, sizes) - File changes with path hashes - Session metrics (turn count, token counts, tool counts) - Implicit signals (retries, compactions, abandonment)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Add comprehensive telemetry collection system for CodeQ to collect agent interactions for model improvement, analytics, and debugging.
Architecture
All telemetry code is isolated in
packages/opencode/src/telemetry/with aqbraid.*config namespace to maintain easy upstream merges from opensource opencode.New Files
telemetry/types.ts- TypeScript types matching the microservice schematelemetry/sanitizer.ts- Redacts secrets, sensitive files, truncates large contenttelemetry/signals.ts- Tracks implicit feedback (retries, errors, abandonment)telemetry/consent.ts- Tier-based consent (free=forced opt-in, paid=opt-in)telemetry/uploader.ts- Batch upload with retry and offline handlingtelemetry/collector.ts- Main orchestration moduletelemetry/integration.ts- Event Bus subscriptions for automatic data collectiontelemetry/index.ts- Public API (Telemetrynamespace)Modified Files
src/config/config.ts- Addedqbraid.telemetryconfig section (apiUrl, enabled, batchSize, etc.)src/project/bootstrap.ts- AddedTelemetry.initIntegration()callFeatures
Configuration
Related PRs