Release 0.6.0: Community Dashboard, BIDS, Metrics#156
Merged
neuromechanist merged 29 commits intomainfrom Feb 5, 2026
Merged
Conversation
Tests were patching src.knowledge.faq_summarizer.create_openrouter_llm but the import is now inside the function from src.core.services.litellm_llm. Fixed both test instances to patch the correct module.
Implement hybrid rate limiting approach: - Per-minute (bot protection): Built-in Rate Limiting API - Per-hour (human abuse): Workers KV - Rate limits: 10/min, 20/hour (prod), 60/min, 100/hour (dev) - 50% reduction in KV writes - Auto-deploy on worker file changes Closes #129
Problem: Git diff fails with 'fatal: bad object' because GitHub Actions uses shallow checkout (depth=1) and github.event.before doesn't exist. Solution: Set fetch-depth=0 to fetch full history, allowing git diff to work properly when detecting worker file changes. This will trigger deployment on next push.
Remove branch filter from pull_request trigger in tests.yml so lint and unit tests run on PRs targeting any branch, not just main/develop. This ensures feature branch PRs to epic branches still get CI coverage.
Remove branch filter from pull_request trigger in tests.yml so lint and unit tests run on PRs targeting any branch, not just main/develop. This ensures feature branch PRs to epic branches still get CI coverage.
Skip test_documentation_urls_accessible; HED docs URL returns 404 due to upstream repo change. See #139.
Skip test_documentation_urls_accessible; HED docs URL returns 404 due to upstream repo change. See #139.
* feat: add backend metrics collection and request logging
- Add src/metrics/ package with SQLite storage (WAL mode),
aggregation queries, and request timing middleware
- Add global /metrics/overview and /metrics/tokens endpoints
- Add per-community /{id}/metrics and /{id}/metrics/usage
- Log token usage, model, key_source, tools for ask/chat
- Streaming handlers log metrics at end of generator
- Middleware captures timing for all requests
- All metrics endpoints require admin auth
Closes #134
* Address PR review: error handling and type safety fixes
- Wrap middleware dispatch in try/except so metrics never crash requests
- Wrap init_metrics_db() in try/except for graceful degradation
- Always return AssistantWithMetrics (remove return_metrics flag)
- Narrow log_request except to sqlite3.Error
- Add try/except to _log_streaming_metrics
- Log metrics on streaming error paths (400/500)
- Add sqlite3 error handling to all metrics endpoints (503)
- Add logger + warning for malformed JSON in queries.py
- Fix middleware ordering comment
- Remove redundant inline import uuid
- Move get_metrics_connection to top-level import
* CI: run lint and tests on all PRs
Remove branch filter from pull_request trigger in tests.yml
so lint and unit tests run on PRs targeting any branch, not
just main/develop. This ensures feature branch PRs to epic
branches still get CI coverage.
* Disable broken URL test until upstream fix
Skip test_documentation_urls_accessible; HED docs URL
returns 404 due to upstream repo change. See #139.
Knowledge databases live inside Docker containers, not locally. Added SSH + docker exec examples for listing tables, querying docstrings, and searching symbols to avoid wasted debugging time.
) * Add dashboard frontend with public metrics endpoints - Add public query functions (no tokens/costs/models exposed) - Create /metrics/public/* endpoints (no auth required) - Build /dashboard page with Chart.js, community tabs, admin unlock - Register new routers in main.py - Add tests for public endpoints and dashboard page (28 new tests) * Restructure dashboard as standalone static site - Move per-community public metrics to community router (/{community_id}/metrics/public, /{community_id}/metrics/public/usage) - Keep only global /metrics/public/overview in metrics_public router - Remove FastAPI dashboard router - Add dashboard/ as standalone static site for Cloudflare Pages: / = aggregate overview, /{community} = community detail - Client-side routing with configurable API base URL - Add _redirects for Cloudflare Pages SPA routing - Update tests for new route structure * Add CI workflow for dashboard Cloudflare Pages deploy Deploys dashboard/ to osa-dash.pages.dev via wrangler. Same pattern as existing deploy-pages.yml for the demo widget: - main -> osa-dash.pages.dev (production) - develop -> develop.osa-dash.pages.dev - PRs -> {branch}.osa-dash.pages.dev with preview URL comment * Add dynamic community tab bar to dashboard Tabs are populated from /metrics/public/overview API so new communities appear automatically. Navigation uses simple links (All -> /, community -> /{id}) with active tab highlighting. * Address PR review findings: XSS, error handling, tests - Fix XSS: add escapeHtml() helper, sanitize all innerHTML interpolations, use encodeURIComponent() for URL path segments - Move get_metrics_connection() inside try blocks in all metrics endpoints - Add console.error/warn to all JavaScript catch blocks (no silent failures) - Improve admin section UX: defer visibility until data loads successfully - Extract shared helpers in queries.py (_count_tools, _validate_period) - Add test classes: TestPublicAdminBoundary, TestEmptyDatabase, TestCommunityMetricsValues with dynamic community cross-checks - Fix admin boundary tests to use auth_env fixture with test API key * Address round-2 review: XSS, error logging, auth tests, simplify - Fix single-quote XSS in onclick handlers: use encodeURIComponent for communityId in changePeriod calls, decode in changePeriod - Validate health status against known values instead of escapeHtml - Add console.warn to sync/health .catch() blocks - Add console.error to loadCommunityView catch block - Add auth-enabled tests proving public endpoints stay accessible when REQUIRE_API_AUTH=true (core security contract) - Add metrics_connection() context manager in db.py; simplify all endpoint handlers from nested try/try/finally to with-statement - Use tuple unpacking in _count_tools for clarity
* Serve dashboard from /osa/ base path for status.osc.earth Move dashboard/index.html to dashboard/osa/index.html and add BASE_PATH constant to strip /osa prefix in client-side router. Update all internal links (tabs, community cards) to use the /osa/ prefix. Update _redirects for SPA routing under /osa/. Update dashboard tests for new file location. * Handle /osa without trailing slash in _redirects Add explicit /osa rule alongside /osa/* to ensure the path without trailing slash also serves the SPA index.
* Add per-community auth, quality metrics, and budget alerting Phase 4 of the community dashboard: per-community scoped authentication (AuthScope + community admin keys), LangFuse observability wiring, quality metrics (error rates, latency percentiles, tool call tracking), cost estimation with model pricing table, budget checking with configurable limits, and automated GitHub issue alerting when spend thresholds are exceeded. Includes scheduled budget check job (every 15 min) and sample budget configs for HED and EEGLAB communities. Tested: 1152 passed, 68% coverage. * Address PR review: simplify code and fix error handling - Fix _issue_exists to return True on error (prevent duplicate spam) - Simplify redundant exception tuples in alerts.py - Extract _require_community_access helper (4 duplicated blocks) - Extract parse_admin_keys method on Settings (3 duplicated parsers) - Strengthen AuthScope with Literal type, frozen, validation - Make BudgetStatus frozen (immutable snapshot) - Add BudgetConfig cross-field validation (daily <= monthly) - Share single DB connection in budget check loop - Add budget check failure escalation (matching sync pattern) - Split LangFuse except into ImportError vs Exception - Improve _migrate_columns to re-raise unexpected errors - Log warnings for malformed community_admin_keys entries - Bump unknown model fallback logging from debug to warning * Fix streaming metrics fields and add quality endpoint tests Add tool_call_count and langfuse_trace_id to streaming metrics logging so streaming requests capture the same quality data as non-streaming. Add 14 endpoint tests covering community quality, quality summary, and global quality API routes. * Address round 2 review: fix alerts, docstrings, tests, simplify - Fix _issue_exists to return None on failure instead of True, with warning-level logging when dedup check fails - Fix stale pricing date and deduplicate fallback branches - Extract shared _fetch_latency_percentiles helper in queries - Make BudgetConfig frozen (immutable after parsing) - Fix inaccurate docstrings: maintainers usage, _percentile method name, _migrate_columns idempotency, regex claims - Fix get_quality_summary docstring key name mismatch - Track per-community scheduler failures for critical alerting - Upgrade malformed config entry log from WARNING to ERROR - Add tests: AuthScope validation, BudgetConfig daily>monthly, community-scoped keys on global endpoints, dedup failure
Code fixes:
- Handle HTTPException in streaming generators as SSE error events
(cannot re-raise after response headers sent)
- Extract _match_wildcard_origin helper, AgentResult dataclass,
_extract_agent_result/_set_metrics_on_request to deduplicate
ask/chat endpoints
- Use metrics_connection() context manager in metrics router
- Add failure counting with escalation logging to log_request()
- Refactor check_budget() to accept BudgetConfig instead of 3 floats
- Add __post_init__ validation to BudgetStatus for non-negative spend
- Simplify list_sessions to reuse _evict_expired_sessions
- Move inline imports (re, os) to top-level
- Simplify _get_communities_with_sync to list comprehension
Docstring fixes:
- Clarify verify_api_key handles only global admin keys
- Update scheduler module docstring to mention budget checks
New tests:
- check_budget with today's timestamps (exercises date('now') SQL)
- Budget alert trigger with current-day spend
- BudgetStatus rejects negative spend values
- _percentile edge cases (single element, two elements, empty list)
- _count_tools with malformed JSON
- _extract_community_id documents intentional None for metrics paths
…-epic-community-dashboard Community Dashboard: metrics, auth, budget alerting
…local testing guide (#150) Add community development section to CLAUDE.md with links to the documentation site registry guides (adding a community, local testing, schema reference, extensions). Generalize .context/local-testing-guide.md from EEGLAB-specific to community-agnostic with placeholder COMMUNITY_ID.
* feat: add BIDS community assistant (Phase 1) Add BIDS (Brain Imaging Data Structure) as a new community with: - 45 documentation sources (2 preloaded, 43 on-demand) covering the specification, all 12 modalities, derivatives, website getting-started guides, FAQs, tools, schema docs, and BEP process - 4 GitHub repos for issue/PR sync (specification, validator, website, examples) - 14 citation DOIs (canonical paper + 12 modality-specific extension papers + BIDS Apps) - System prompt with modality awareness, schema awareness, validator guidance, converter recommendations, and explicit anti-hallucination instructions for GitHub references - NeuroStars discourse integration (bids tag) - Budget and maintainer configuration All documentation URLs verified (raw GitHub + readthedocs stable). Config validates correctly and community discovery works (336 tests pass, no failures). Closes #149 * fix: address PR review findings for BIDS config - Fix phenotypic data description (phenotype/ directory, not participants.tsv) - Fix code description (data preparation scripts, not analysis code) - Add participants.tsv to data summary files description - Fix behavioral wording (no neural recordings, not no neuroimaging) - Fix genetics wording in system prompt (brain imaging data) - Add missing physiological recordings documentation entry (13 modality docs) - Add Physiological to system prompt modality list
Allows manual re-triggering when automated runs fail due to transient issues (e.g. expired tokens, push race conditions).
Replace "What are the required metadata fields?" with "What are the BIDS Common Principles?" as a more foundational starting question for new users.
The model was not calling knowledge tools when asked about specific PR/issue numbers. Add explicit patterns showing how to search by number.
* feat: add number-based lookup to GitHub item search When query contains a PR/issue number (e.g. "2022", "#500", "PR 2022"), search now does a direct number lookup first, then falls back to full-text search for remaining slots. Deduplicates results. Fixes #153 * feat: switch all communities from Qwen to Claude Haiku 4.5 Qwen was not reliably calling knowledge tools when asked about specific PRs/issues. Claude Haiku 4.5 via Anthropic provider is more stable for tool calling and has caching enabled. * Address PR review: add number index, skip FTS for pure number queries, expand tests - Add idx_github_items_number index for direct number lookups - Add _is_pure_number_query() to skip FTS when query is just a number pattern - Add debug logging when number lookup finds no results - Add tests for status filter, nonexistent numbers, limits, bug/feature prefixes - Strengthen assertions on existing number lookup tests * Fix ruff SIM103: return condition directly in _is_pure_number_query * Fix ruff formatting
* fix: add dashboard CORS origin and missing worker routes
Worker:
- Add osa-dash.pages.dev to CORS allowlist (dashboard origin)
- Add routes for dashboard endpoints: /metrics/public/overview,
/metrics/{overview,tokens,quality}, /sync/{status,health},
/{community}/metrics/public, /{community}/sessions
Dashboard:
- Auto-detect API backend from hostname instead of falling back
to window.location.origin (which fails for deployed dashboards)
- osa-dash.pages.dev -> api.osc.earth/osa (prod)
- *.osa-dash.pages.dev -> api.osc.earth/osa-dev (dev/preview)
* fix: add osa-dash.pages.dev to backend CORS allowlist
The dashboard (osa-dash.pages.dev) was being rejected by the
backend's CORS middleware. Add both exact and wildcard patterns
for the dashboard Pages project.
* fix: harden worker security and improve dashboard errors
Worker:
- Split proxy into worker-key vs client-key passthrough modes
- Admin/sessions endpoints now forward client key (not worker key)
- Extract RESERVED_PATHS constant, rateLimitOrReject helper
- Extract validateCommunityId with consistent reserved path checks
- Add rate limiting to all endpoints including admin and sync
- Validate https:// protocol on subdomain CORS checks
- Only forward Origin header if CORS-validated
- Replace bare catch blocks with console.warn logging
Dashboard:
- Add status-code-aware error messages (429, 404, 401, 500)
- Replace generic "Failed to load" with actionable user guidance
Contributor
🚀 Preview Deployment
This preview will be updated automatically when you push new commits. |
Contributor
Dashboard Preview
This preview will be updated automatically when you push new commits. |
Re-apply changes from d12c98e that were listed in the squash merge commit message but not included in the actual diff: - Split proxy into worker-key vs client-key passthrough modes - Admin/sessions endpoints forward client key (not worker key) - Extract RESERVED_PATHS, rateLimitOrReject, validateCommunityId - Rate limiting on all endpoints including admin and sync - Validate https:// protocol on subdomain CORS checks - Only forward Origin header if CORS-validated - Replace bare catch blocks with console.warn logging
Contributor
|
This PR modifies community CORS origins. Worker changes detected. After merging to Manual deployment (if needed): cd workers/osa-worker
wrangler deploy --env dev # for develop branch
wrangler deploy # for main branch |
Member
Author
PR Review Summary (6 agents, parallel)Already Fixed (during review)
Critical Issues (2)1. [error-handling] Rate limiter fail-open without alerting 2. [error-handling] Turnstile verification silently skipped when secret key missing High Issues (5)3. [error-handling] Worker top-level catch leaks internal error messages 4. [error-handling] Worker 5. [error-handling] Metrics DB init failure silently degrades system 6. [error-handling] Metrics middleware broad exception catch 7. [error-handling] Streaming metrics not logged on session limit Medium Issues (8)8. [code-reviewer] KV rate limit double-read race condition 9. [code-reviewer] Dashboard sync info not HTML-escaped 10. [error-handling] Scheduler job registration failures silently degraded 11. [error-handling] 12. [test-analyzer] No tests for 13. [test-analyzer] Per-community scoped auth endpoints untested (criticality 7) 14. [comment-analyzer] Worker README endpoints table incomplete 15. [comment-analyzer] Stale pricing date (2025-07) in cost.py Suggestions (from simplifier, type-design, comments)Types:
Simplification:
Comments:
Positive Observations
|
- Total now counts only community-scoped requests, not health checks, sync, and other infrastructure endpoints - All registered communities appear in overview even with 0 requests - Middleware _extract_community_id now recognizes metrics, sessions, and config endpoints (not just /ask and /chat)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Release 0.6.0 brings the community dashboard epic, BIDS community support, and metrics infrastructure.
Major Features
osa-dash.pages.devwith public metrics overview, per-community views, usage charts, admin token/cost views, and sync statusCOMMUNITY_ADMIN_KEYS),AuthScopesystem for role-based access controlInfrastructure
Files Changed
Test plan