BigBrotr

Nostr Relay Discovery, Monitoring, and Event Archiving System

Discovers relays across clearnet and overlay networks, monitors health with NIP-11/NIP-66 compliance checks, and archives events into PostgreSQL.

What It Does

BigBrotr runs a pipeline of five async services that continuously map and monitor the Nostr relay ecosystem:

Seeder ──> Finder ──> Validator ──> Monitor ──> Synchronizer
(seed URLs)  (discover)  (test)    (health)     (archive events)

Seeder loads relay URLs from a seed file (one-shot)
Finder discovers new relays from stored events (NIP-65 relay lists, kind 2/3) and external APIs
Validator tests WebSocket connectivity for each candidate, promoting valid relays
Monitor performs NIP-11 info document fetches and NIP-66 health checks (RTT, SSL, DNS, GeoIP, ASN, HTTP headers), then publishes results as kind 10166/30166 Nostr events
Synchronizer connects to all validated relays, subscribes to events, and archives them with per-relay cursor tracking for incremental sync

All services expose Prometheus metrics, run behind PGBouncer connection pooling, and support clearnet + Tor + I2P + Lokinet connectivity.

Architecture

Diamond DAG with strict import direction (top to bottom only):

              services         src/bigbrotr/services/
             /   |   \
          core  nips  utils    src/bigbrotr/{core,nips,utils}/
             \   |   /
              models           src/bigbrotr/models/

models -- Pure frozen dataclasses. Zero I/O, zero package dependencies, stdlib logging only.
core -- Pool (asyncpg), Brotr (DB facade), BaseService, Logger, Metrics, Exceptions.
nips -- NIP-11 relay info fetch/parse, NIP-66 health checks (RTT, SSL, DNS, Geo, Net, HTTP). Has I/O.
utils -- DNS resolution, Nostr key management, WebSocket/HTTP transport with SOCKS5 proxy.
services -- Business logic: Seeder, Finder, Validator, Monitor (+ Publisher + Tags), Synchronizer.

Quick Start

Prerequisites

Docker and Docker Compose
(Optional) Python 3.11+ for local development

Deploy with Docker Compose

git clone https://github.com/BigBrotr/bigbrotr.git
cd bigbrotr/deployments/bigbrotr

# Configure secrets
cp .env.example .env
# Edit .env: set DB_PASSWORD, PRIVATE_KEY, GRAFANA_PASSWORD

# Start everything
docker compose up -d

# Watch the pipeline start
docker compose logs -f seeder

This starts PostgreSQL, PGBouncer, Tor proxy, all 5 services, Prometheus, and Grafana.

Endpoint	URL
Grafana	`http://localhost:3000`
Prometheus	`http://localhost:9090`
PostgreSQL	`localhost:5432`
PGBouncer	`localhost:6432`

Run a Single Service Locally

pip install -e ".[dev]"
cd deployments/bigbrotr

# One cycle
python -m bigbrotr seeder --once

# Continuous with debug logging
python -m bigbrotr finder --log-level DEBUG

Deployments

BigBrotr supports multiple deployment configurations from the same codebase via a single parametric Dockerfile (deployments/Dockerfile with ARG DEPLOYMENT).

BigBrotr (Full Archive)

Stores complete Nostr events (id, pubkey, created_at, kind, tags, content, sig). 7 materialized views for analytics. Tor enabled. All 5 services + Prometheus + Grafana.

cd deployments/bigbrotr && docker compose up -d

LilBrotr (Lightweight)

Stores event metadata only (id, pubkey, created_at, kind, tagvalues). Omits tags JSON, content, and sig for approximately 60% disk savings. No materialized views. Same service pipeline.

cd deployments/lilbrotr && docker compose up -d

Custom Deployment

cp -r deployments/_template deployments/myrelay
# Edit config, SQL schema, docker-compose.yaml
cd deployments/myrelay && docker compose up -d

Database

PostgreSQL 16 with PGBouncer (transaction-mode pooling) and asyncpg async driver. All mutations via stored functions with bulk array parameters.

Schema

Table	Purpose
`relay`	Validated relay URLs with network type and discovery timestamp
`event`	Nostr events (BYTEA ids/pubkeys/sigs for space efficiency)
`event_relay`	Junction: which events were seen at which relays
`metadata`	Content-addressed NIP-11/NIP-66 documents (SHA-256 dedup, `payload` JSONB)
`relay_metadata`	Time-series snapshots linking relays to metadata records (`metadata_type` column)
`service_state`	Per-service operational data (candidates, cursors, checkpoints)

Stored Functions (22)

1 utility: tags_to_tagvalues (extracts single-char tag values for GIN indexing)
10 CRUD: relay_insert, event_insert, metadata_insert, event_relay_insert, relay_metadata_insert, event_relay_insert_cascade, relay_metadata_insert_cascade, service_state_upsert, service_state_get, service_state_delete
3 cleanup: orphan_event_delete, orphan_metadata_delete, relay_metadata_delete_expired (all batched)
8 refresh: one per materialized view + all_statistics_refresh

All functions use SECURITY INVOKER, bulk array parameters, and ON CONFLICT DO NOTHING.

Materialized Views (7, BigBrotr Only)

relay_metadata_latest, event_stats, relay_stats, kind_counts, kind_counts_by_relay, pubkey_counts, pubkey_counts_by_relay -- all support REFRESH CONCURRENTLY via unique indexes.

Monitoring

Prometheus Metrics

Every service exposes /metrics on its configured port with four metric types:

Metric	Type	Description
`service_info`	Info	Static service metadata
`service_gauge`	Gauge	Point-in-time state (consecutive_failures, last_cycle_timestamp, progress)
`service_counter`	Counter	Cumulative totals (cycles_success, cycles_failed, errors by type)
`cycle_duration_seconds`	Histogram	Cycle latency with 10 buckets (1s to 1h)

Alert Rules (4)

Alert	Condition	Severity
ServiceDown	`up == 0` for 5m	critical
HighFailureRate	error rate > 0.1/s for 5m	warning
PoolExhausted	zero available connections for 2m	critical
DatabaseSlow	p99 query latency > 5s for 5m	warning

Grafana Dashboard

Auto-provisioned dashboard with per-service panels: last cycle time, cycle duration, error counts (24h), consecutive failures. Validator has additional candidate progress panels.

Structured Logging

info finder cycle_completed relay_count=100 duration=2.5
error validator retry_failed attempt=3 url="wss://relay.example.com"

JSON mode available for cloud aggregation:

{"timestamp": "2026-02-09T12:34:56+00:00", "level": "info", "service": "finder", "message": "cycle_completed", "relay_count": 100}

Nostr Protocol Support

NIPs Implemented

NIP	Usage
NIP-01	Event model, relay communication
NIP-02	Contact list relay discovery (kind 3)
NIP-11	Relay information document fetch and parse
NIP-65	Relay list metadata (kind 10002)
NIP-66	Relay monitoring and discovery (kinds 10166, 30166)

Event Kinds

Kind	Direction	Purpose
0	Published	Monitor profile metadata
2	Consumed	Deprecated relay recommendation (content = URL)
3	Consumed	Contact list (content = JSON with relay URLs as keys)
10002	Consumed	NIP-65 relay list ("r" tags)
10166	Published	Monitor announcement (capabilities, timeouts)
30166	Published	Relay discovery (addressable, one per relay, full metadata tags)

NIP-66 Health Checks

Check	What It Measures
RTT	WebSocket open/read/write latency (ms)
SSL	Certificate validity, expiry, issuer, cipher suite
DNS	A/AAAA/CNAME/NS/PTR records, query time
Geo	Country, city, coordinates, timezone, geohash (GeoLite2)
Net	IP address, ASN, organization (GeoLite2 ASN)
HTTP	Server header, X-Powered-By

Configuration

Environment Variables

Variable	Required	Description
`DB_PASSWORD`	Yes	PostgreSQL password
`PRIVATE_KEY`	For Monitor	Nostr private key (hex or nsec) for event publishing and RTT write tests
`GRAFANA_PASSWORD`	No	Grafana admin password

Configuration Files

deployments/bigbrotr/config/
+-- brotr.yaml                  # Pool, batch size, timeouts
+-- services/
    +-- seeder.yaml             # Seed file path
    +-- finder.yaml             # API sources, scan interval (default: 1h)
    +-- validator.yaml          # Validation interval (8h), cleanup, networks
    +-- monitor.yaml            # Check interval (1h), retry per check type, networks
    +-- synchronizer.yaml       # Sync interval (15m), per-relay overrides, concurrency

All configs use Pydantic v2 validation with typed defaults and constraints.

Development

Setup

git clone https://github.com/BigBrotr/bigbrotr.git && cd bigbrotr
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pre-commit install

Quality Checks

make lint          # ruff check src/ tests/
make format        # ruff format src/ tests/
make typecheck     # mypy src/bigbrotr (strict mode)
make test-unit        # pytest unit tests (2049 tests)
make test-integration # pytest integration tests (requires Docker)
make test-fast        # pytest -m "not slow"
make coverage         # pytest --cov with HTML report
make ci               # all checks: lint + format + typecheck + test-unit
make docs             # build MkDocs documentation site
make docs-serve       # serve docs locally with live reload
make build            # build Python package (sdist + wheel)
make docker-build     # build Docker image (DEPLOYMENT=bigbrotr)
make docker-up        # start Docker stack
make docker-down      # stop Docker stack
make clean            # remove build artifacts and caches

Test Suite

2049 unit tests + 8 integration tests (testcontainers PostgreSQL)
asyncio_mode = "auto" -- no @pytest.mark.asyncio needed
Global timeout: 120s per test
Shared fixtures via tests/fixtures/relays.py (registered as pytest plugin)
Coverage threshold: 80% (branch coverage enabled)

CI/CD Pipeline

Stage	Tool	Purpose
Pre-commit	ruff, mypy, yamllint, detect-secrets, markdownlint, hadolint, sqlfluff	Code quality gates
Unit Test	pytest (Python 3.11--3.14 matrix)	Unit tests + coverage
Integration Test	pytest + testcontainers	PostgreSQL integration tests
Build	Docker Buildx (matrix)	Multi-deployment image builds + Trivy scan
Security	pip-audit, Trivy, CodeQL	Dependency vulns, container scanning, static analysis
Release	PyPI (OIDC) + GHCR	Package + Docker image publishing, SBOM generation
Docs	MkDocs Material	Auto-generated API docs deployed to GitHub Pages
Dependencies	Dependabot	Weekly updates for pip, Docker, GitHub Actions

Project Structure

bigbrotr/
+-- src/bigbrotr/                    # Main package (namespace)
|   +-- __main__.py                  # CLI entry point
|   +-- core/                        # Foundation
|   |   +-- pool.py                  # asyncpg connection pool with retry
|   |   +-- brotr.py                 # High-level DB facade (stored procedures)
|   |   +-- base_service.py          # Abstract service with run_forever loop
|   |   +-- exceptions.py            # Exception hierarchy (10 classes)
|   |   +-- metrics.py               # Prometheus metrics server
|   |   +-- logger.py                # Structured key=value / JSON logging
|   |   +-- yaml.py                  # YAML config loader
|   +-- models/                      # Pure frozen dataclasses (zero I/O)
|   |   +-- relay.py                 # URL validation, network detection
|   |   +-- event.py                 # Nostr event wrapper
|   |   +-- metadata.py              # Content-addressed metadata (SHA-256)
|   |   +-- service_state.py         # ServiceState, EventKind, StateType
|   |   +-- constants.py             # NetworkType enum
|   |   +-- event_relay.py           # Event-relay junction
|   |   +-- relay_metadata.py        # Relay-metadata junction
|   +-- nips/                        # NIP protocol implementations (I/O)
|   |   +-- nip11/                   # Relay information document
|   |   +-- nip66/                   # Health checks: rtt, ssl, dns, geo, net, http
|   +-- utils/                       # DNS, keys, transport
|   +-- services/                    # Business logic
|       +-- seeder.py
|       +-- finder.py
|       +-- validator.py
|       +-- monitor.py               # Health check orchestration
|       +-- monitor_publisher.py     # Nostr event broadcasting
|       +-- monitor_tags.py          # NIP-66 tag building
|       +-- synchronizer.py
|       +-- common/                  # Shared queries, configs, mixins
+-- deployments/
|   +-- Dockerfile                   # Single parametric (ARG DEPLOYMENT)
|   +-- bigbrotr/                    # Full archive deployment
|   |   +-- config/                  # YAML configs
|   |   +-- postgres/init/           # SQL schema (10 files, 22 functions)
|   |   +-- monitoring/              # Prometheus + Grafana provisioning
|   |   +-- docker-compose.yaml
|   +-- lilbrotr/                    # Lightweight deployment
|   +-- _template/                   # Custom deployment template
+-- tests/
|   +-- fixtures/relays.py           # Shared relay fixtures
|   +-- unit/                        # 2049 tests (mirrors src/ structure)
|   +-- integration/                 # 8 tests (testcontainers PostgreSQL)
+-- docs/                            # Architecture, Database, Deployment, Development, Configuration
+-- Makefile                         # Development targets
+-- pyproject.toml                   # All config: deps, ruff, mypy, pytest, coverage

Exception Hierarchy

BigBrotrError
+-- ConfigurationError          # YAML, env vars, CLI
+-- DatabaseError
|   +-- ConnectionPoolError     # Transient (retry)
|   +-- QueryError              # Permanent (don't retry)
+-- ConnectivityError
|   +-- RelayTimeoutError       # Connection/response timed out
|   +-- RelaySSLError           # TLS/SSL failures
+-- ProtocolError               # NIP parsing/validation
+-- PublishingError             # Event broadcast failures

Docker Infrastructure

Container Stack

Container	Image	Purpose	Resources
postgres	`postgres:16-alpine`	Primary storage	2 CPU, 2 GB
pgbouncer	`edoburu/pgbouncer:v1.25.1-p0`	Transaction-mode connection pooling	0.5 CPU, 256 MB
tor	`osminogin/tor-simple:0.4.8.10`	SOCKS5 proxy for .onion relays	0.5 CPU, 256 MB
finder	bigbrotr (parametric)	Relay discovery	1 CPU, 512 MB
validator	bigbrotr (parametric)	Candidate validation	1 CPU, 512 MB
monitor	bigbrotr (parametric)	Health monitoring	1 CPU, 512 MB
synchronizer	bigbrotr (parametric)	Event archiving	1 CPU, 512 MB
prometheus	`prom/prometheus:v2.51.0`	Metrics collection (30d retention)	0.5 CPU, 512 MB
grafana	`grafana/grafana:10.4.1`	Dashboards	0.5 CPU, 512 MB

Networks

data-network -- postgres, pgbouncer, tor, all services
monitoring-network -- prometheus, grafana, all services (metrics scraping)

Security

All ports bound to 127.0.0.1 (no external exposure)
Non-root container execution (UID 1000)
tini as PID 1 for proper signal handling
SCRAM-SHA-256 authentication (PostgreSQL + PGBouncer)
Real healthchecks via /metrics endpoint (not fake PID checks)

Technology Stack

Category	Technologies
Language	Python 3.11+ (fully typed, strict mypy)
Database	PostgreSQL 16, asyncpg, PGBouncer
Async	asyncio, aiohttp, aiomultiprocess
Nostr	nostr-sdk (Rust FFI via PyO3/uniffi)
Validation	Pydantic v2, rfc3986
Monitoring	Prometheus, Grafana, structured logging
Networking	aiohttp-socks (SOCKS5), dnspython, geoip2, tldextract
Testing	pytest, pytest-asyncio, pytest-cov, testcontainers
Quality	ruff (lint+format), mypy (strict), pre-commit (21 hooks)
CI/CD	GitHub Actions, pip-audit, Trivy, CodeQL, Dependabot
Containers	Docker, Docker Compose, tini

Contributing

Fork and clone
pip install -e ".[dev]" and pre-commit install
Write tests for new functionality
make ci -- all checks must pass
Submit a pull request

Conventional commits: feat:, fix:, refactor:, docs:, test:, chore:

License

MIT -- see LICENSE.

Name		Name	Last commit message	Last commit date
Latest commit History 892 Commits
.github		.github
.vscode		.vscode
deployments		deployments
docs		docs
src/bigbrotr		src/bigbrotr
tests		tests
tools		tools
.dockerignore		.dockerignore
.editorconfig		.editorconfig
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
.secrets.baseline		.secrets.baseline
.trivyignore		.trivyignore
.yamllint		.yamllint
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
mkdocs.yml		mkdocs.yml
pyproject.toml		pyproject.toml

License

BigBrotr/bigbrotr

Folders and files

Latest commit

History

Repository files navigation

BigBrotr

What It Does

Architecture

Quick Start

Prerequisites

Deploy with Docker Compose

Run a Single Service Locally

Deployments

BigBrotr (Full Archive)

LilBrotr (Lightweight)

Custom Deployment

Database

Schema

Stored Functions (22)

Materialized Views (7, BigBrotr Only)

Monitoring

Prometheus Metrics

Alert Rules (4)

Grafana Dashboard

Structured Logging

Nostr Protocol Support

NIPs Implemented

Event Kinds

NIP-66 Health Checks

Configuration

Environment Variables

Configuration Files

Development

Setup

Quality Checks

Test Suite

CI/CD Pipeline

Project Structure

Exception Hierarchy

Docker Infrastructure

Container Stack

Networks

Security

Technology Stack

Contributing

License

Links

About

Topics

Resources

License

Code of conduct

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases 10

Packages 0

Uh oh!

Contributors 3

Uh oh!

Languages

Packages