RA-8279: feat(otel): Add OpenTelemetry distributed tracing and ctutils logging integration #3

himaschal · 2025-12-08T14:01:16Z

ℹ️ Release Coordination (Downstream of ctutils)

This feature depends on digicert/ctutils (RA-8279).
Status: Ready for review (ensure ctutils v1.0.0 tag is available).
Plan:

Wait for digicert/ctutils PR1 merge & v1.0.0 tag.
Update go.mod in this PR to use digicert/ctutils v1.0.0.
Merge this PR.

Summary

Integrates the digicert/ctutils shared logging library to enable OpenTelemetry distributed tracing across Trillian Log Server and Log Signer. This allows full request tracing from the frontend (CTFE) through to the storage layer.

Features

Distributed Tracing: End-to-end trace propagation (gRPC interceptors).
Unified Logging: Replaces ad-hoc logging with ctutils adapters.
Database Tracing: (If applicable, check code for SQL wrappers).
Configuration: Standard OTEL_* environment variables.

Configuration

See the README Observability Section for full details.

Variable	Description
`OTEL_ENABLED`	Master switch for tracing
`OTEL_EXPORTER`	`otlp` (collector) or `stdout` (debug)
`LOG_LEVEL`	Minimum log severity (Parsed by app, correctly implemented)

Testing

Integration verified in typical deployment scenarios:

Local K8s: Verified traces appear in Jaeger when triggered from CTFE.

See full e2e testing here

Refs: RA-8279

… integration This change integrates the digicert/ctutils shared logging library to enable OpenTelemetry-compliant distributed tracing across the Trillian log server and signer components. Key changes: - Add config/config.go with InitLogging() for centralized OTEL configuration - Update log_server and log_signer main.go to call config.InitLogging() - Add chained gRPC interceptors for trace context propagation - Add Dockerfile.unified with SSH access for private ctutils dependency - Update go.mod/go.sum for ctutils v0.1.6 and OTEL dependencies The logging configuration is driven by environment variables: - OTEL_ENABLED: Enable/disable OpenTelemetry (default: false) - OTEL_EXPORTER: Exporter type ('otlp' or 'stdout') - OTEL_COLLECTOR_ENDPOINT: OTLP collector URL - OTEL_SERVICE_NAME: Service name for traces - OTEL_SAMPLE_RATIO: Sampling ratio (0.0-1.0) This enables end-to-end request tracing from CTFE through Trillian backends, allowing operators to correlate logs and traces across the CT infrastructure. Refs: RA-8279

chore(deps): update ctutils

Copilot

Pull request overview

This PR integrates OpenTelemetry distributed tracing into Trillian log server and signer components through the digicert/ctutils shared logging library, enabling end-to-end request tracing and structured logging.

Changes:

Added OTEL-compliant distributed tracing via ctutils dependency
Configured gRPC and HTTP middleware for trace propagation and logging
Added environment-variable based OTEL configuration

Reviewed changes

Copilot reviewed 16 out of 18 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
config/config.go	New centralized logging initialization with OTEL setup
cmd/trillian_log_server/main.go	Initialize logging on server startup
cmd/trillian_log_signer/main.go	Initialize logging on signer startup
cmd/internal/serverutil/main.go	Add gRPC/HTTP interceptors for trace propagation
go.mod/go.sum	Add ctutils v0.1.13-test and updated OTEL dependencies
examples/deployment/docker//Dockerfile	Docker build configuration for private ctutils dependency
.github/workflows/*.yaml	CI authentication for private ctutils repository
README.md	Documentation for OTEL configuration
experimental/batchmap/batchmap.shims.go	Auto-generated code formatting updates

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

config/config.go

cmd/internal/serverutil/main.go

examples/deployment/docker/log_server/Dockerfile.unified

examples/deployment/docker/log_signer/Dockerfile

go.mod

config/config.go

examples/deployment/docker/log_server/Dockerfile

examples/deployment/docker/log_signer/Dockerfile

cmd/trillian_log_server/main.go

cmd/internal/serverutil/main.go

chore(deps): update ctutils

Removed accidental duplicate comments introduced during previous edits to ensure clean and readable Dockerfiles.

Removed a commented-out require statement for ctutils that was causing confusion, as the actual requirement is correctly defined later in the file.

replaced os.Setenv with t.Setenv to fix errcheck lint errors and improve test cleanup.

Copilot

Pull request overview

Copilot reviewed 16 out of 18 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

README.md

config/config.go

config/config_test.go

examples/deployment/docker/log_server/Dockerfile

examples/deployment/docker/log_signer/Dockerfile

README.md

…ment variables to workflows and enhance `ctutils` update logic.

This reverts commit 4236c87.

…ssue" This reverts commit 646baff.

Co-authored-by: himaschal <himaschal@users.noreply.github.com>

cmd/trillian_log_signer/main.go

cmd/internal/serverutil/main.go

.github/workflows/codeql.yml

.github/workflows/govulncheck.yml

.github/workflows/golangci-lint.yml

experimental/batchmap/batchmap.shims.go

…ockerfiles for log server and signer

…argument and streamlining build commands

Co-authored-by: himaschal <himaschal@users.noreply.github.com> Test version for golem rollout

…cessary and not recommended according to best practice. This will also keep the logserver / logsigner logs clean from unnecessary logs when scheduled jobs run like health checks or metrics scraping

sadhana-angara · 2026-02-11T15:32:51Z

QA Evidence for CT Log :

Test Case 1 : Deploy Jaeger and Configure OTLP Export :
Status : PASS
Evidence :

Jaeger pod deployed successfully -

Services configured with OTLP export -

Port forwarding established -

Jaeger UI accessible at http://localhost:16686

CTFE API accessible at http://localhost:6962

========================================================================================

Test 2: Verify Distributed Tracing in Jaeger UI
Status : PASS
Evidence :

Multiple traces visible in search results (6+ traces) : Yes
Traces show both ctfe and trillian-logserver services : Yes
Span hierarchy shows parent-child relationships : Yes
Timing data present for each span (typically 3-25ms) : Yes
Operation names visible (e.g., ctfe_internal_sth_force) : Yes

========================================================================================

Test 3: W3C Trace Context Propagation
Status : PASS
Evidence :

Request succeeds (200 OK) : Yes
Custom trace_id appears in Jaeger UI : Yes
Trace shows in search results within 10 seconds : Yes

========================================================================================

Test 4: Cross-Service Trace Propagation
Status : PASS
Evidence :

Same trace_id found in CTFE logs : Yes
Same trace_id found in Trillian logserver logs : Yes
Logs show parent-child relationship via span_id : Yes

========================================================================================

Test 5: Structured Logging with Trace Context
Status : PASS
Evidence :

Logs in JSON format : Yes
trace_id field present (32-char hex) : Yes
span_id field present (16-char hex) : Yes
parent_source field present (values: client_header, grpc_metadata, or system_generated) : Yes
elapsed_ms field present (timing data) : Yes

========================================================================================

Test 6: Multiple Requests with Shared Trace
Status : PASS
Evidence :

**- All 3 requests succeed : **

- Logs show 3+ entries with same trace_id :

- Each entry has different span_id :

========================================================================================

Test 7: Performance and Timing Analysis
Status : PASS
Evidence : In Jaeger UI, examine 5-10 different traces

1. Trace ID : 4dd4c18687c1d0f1b54bba5f7223efd7 :
Typical get-sth operation: 5.47ms
gRPC call to Trillian: 3.39ms on server side
CTFE overhead : 5.1 - 3.39 = 1.71ms
No traces showing errors or timeouts.

2.Trace ID : 089ff8e936e3df81afbfb9fe416f3194 :
Typical get-sth operation: 3.26ms
gRPC call to Trillian: 2.2ms on server side
CTFE overhead : 3.04 - 2.2 = 0.84ms
No traces showing errors or timeouts.

3.Trace ID : b3aec802d23c219702cc36f7d7a3b1e6 :
Typical get-sth operation: 5.1ms
gRPC call to Trillian: 3.37ms on server side
CTFE overhead : 4.75 - 3.37 = 1.38ms
No traces showing errors or timeouts.

4.Trace ID : 9a4723af4335407b39318948fb92d9b6 :
Typical get-sth operation: 4.52ms
gRPC call to Trillian: 2.94ms on server side
CTFE overhead : 4.52 - 2.94 = 1.58ms
No traces showing errors or timeouts.

5.Trace ID : bea9a080ada3f6739735ab113f1ad7c2 :
Typical get-sth operation: 7.95ms
gRPC call to Trillian: 3.78ms on server side
CTFE overhead : 7.09 - 3.78 = 3.31ms
No traces showing errors or timeouts.

6.Trace ID : 4ae35982188663fee3d8eeb622866356 :
Typical get-sth operation: 8.76ms
gRPC call to Trillian: 4.69ms on server side
CTFE overhead : 7.83 - 4.69 = 3.14ms
No traces showing errors or timeouts.

7.Trace ID : 8018b3ff51e0b343206a6b984802c18b :
Typical get-sth operation: 6.14ms
gRPC call to Trillian: 4.21ms on server side
CTFE overhead : 6.14 - 4.21 = 1.93ms
No traces showing errors or timeouts.

8.Trace ID : a3699b3aef49a80fc73aae7bb5bd6e5f :
Typical get-sth operation: 5.2ms
gRPC call to Trillian: 3.61ms on server side
CTFE overhead : 4.97 - 3.61 = 1.36ms
No traces showing errors or timeouts.

9.Trace ID : 4a4f82d3b182303ed0b773c0460b9169 :
Typical get-sth operation: 7.65ms
gRPC call to Trillian: 4.19ms on server side
CTFE overhead : 7.24 - 4.19 = 3.05ms
No traces showing errors or timeouts.

10.Trace ID : 26a99abfc5ad08be508a00b301abb032 :
Typical get-sth operation: 5.08ms
gRPC call to Trillian: 2.26ms on server side
CTFE overhead : 4.59 - 2.26 = 2.33ms
No traces showing errors or timeouts.

========================================================================================

Test 8: Service Dependencies Visualization
Status : PASS
Evidence :

Clear call chain: ctfe → trillian-logserver : Yes
gRPC communication visible in spans : Pending
No unexpected service dependencies : Pending

himaschal mentioned this pull request Dec 8, 2025

RA-8279: Add OpenTelemetry distributed tracing and ctutils logging to CTFE digicert/certificate-transparency-go#3

Open

himaschal and others added 17 commits December 10, 2025 17:35

Update ctutils to v0.1.7

518b1d8

ci: add workflow to update ctutils dependencies

c64d10e

fix(ci): use oauth2 syntax for authentication and debug token

86dc36a

fix(ci): use GITHUB_TOKEN for PR creation

fea9c9d

fix(ci): auth to private ctutils for lint and vuln checks

c370095

fix(ci): auth to private ctutils for database integration tests

0991a95

fix(ci): auth to private ctutils and set GOPRIVATE for CodeQL

03dc543

test: clean run trigger for v0.1.11-test

13a414e

chore(deps): update ctutils to latest version

c71aacb

Merge pull request #4 from digicert/chore/update-ctutils

6ae67f5

chore(deps): update ctutils

chore: test pipeline run with ctutils v0.1.12-test

febb1a9

chore(deps): update ctutils to latest version

683f18f

Merge pull request #5 from digicert/chore/update-ctutils

f04e183

chore(deps): update ctutils

chore: test pipeline run with ctutils v0.1.13-test

66422c4

chore(deps): update ctutils to latest version

53ae555

Merge pull request #6 from digicert/chore/update-ctutils

c2fab3b

chore(deps): update ctutils

chore: revert pipeline testing configurations to production ready state

d81bf44

himaschal requested a review from Copilot January 13, 2026 16:03

Copilot started reviewing on behalf of himaschal January 13, 2026 16:04 View session

Copilot AI reviewed Jan 13, 2026

View reviewed changes

himaschal and others added 8 commits January 14, 2026 18:56

RA-8279: Modified to use github app

996c9dc

ci: use github app for private repo access

ac7a3ca

ci: update all workflows to use github app token

8448274

chore(deps): update ctutils to latest version

4300dd0

Merge pull request #8 from digicert/chore/update-ctutils-test

80f9e8f

chore(deps): update ctutils

chore(deps): update ctutils to latest version

221044d

Merge pull request #9 from digicert/chore/update-ctutils-test

ef8cfbb

chore(deps): update ctutils

chore(deps): update ctutils to latest version

c16b7e8

himaschal added 3 commits January 17, 2026 22:15

Clean up duplicate comments in Dockerfiles

cb3ffff

Removed accidental duplicate comments introduced during previous edits to ensure clean and readable Dockerfiles.

Remove misleading comment in go.mod

586f41b

Removed a commented-out require statement for ctutils that was causing confusion, as the actual requirement is correctly defined later in the file.

Fix lint errors in config_test.go

deb3c87

replaced os.Setenv with t.Setenv to fix errcheck lint errors and improve test cleanup.

himaschal requested a review from Copilot January 18, 2026 09:27

Copilot started reviewing on behalf of himaschal January 18, 2026 09:27 View session

Copilot AI reviewed Jan 18, 2026

View reviewed changes

himaschal added 2 commits January 22, 2026 16:04

Add shutdown handling and config consistency

0e27de6

add Dockerfile.local to .gitignore. Used for local dev / builds.

ff55056

himaschal changed the title ~~feat(otel): Add OpenTelemetry distributed tracing and ctutils logging integration~~ RA-8279: feat(otel): Add OpenTelemetry distributed tracing and ctutils logging integration Jan 23, 2026

himaschal and others added 10 commits January 23, 2026 11:35

build: update ctutils dependency to fix CI build

2c11080

RA-8279: Update README with minor changes.

fef833f

RA-8279: Address PR comments. Add GOPRIVATE and GONOSUMDB environ…

ba2ccae

…ment variables to workflows and enhance `ctutils` update logic.

fix(ci): robust latest release resolution for ctutils

beb4e3f

fix(ci): add GONOPROXY for private ctutils module access

800d4ea

test: remove GONOPROXY to verify if it's necessary

4236c87

Revert "test: remove GONOPROXY to verify if it's necessary"

abf3be0

This reverts commit 4236c87.

test: remove GONOPROXY again to check if fixed tag resolves issue

646baff

Revert "test: remove GONOPROXY again to check if fixed tag resolves i…

abaae85

…ssue" This reverts commit 646baff.

chore(deps): update ctutils to v0.1.27-test (#17)

abe8d6d

Co-authored-by: himaschal <himaschal@users.noreply.github.com>

digilob approved these changes Jan 27, 2026

View reviewed changes

himaschal and others added 5 commits January 29, 2026 14:25

chore: update GONOPROXY indentation in workflow files and add local D…

1ddf0a9

…ockerfiles for log server and signer

RA-8179: improve Dockerfiles for local development by adding GOPROXY …

d1e92cb

…argument and streamlining build commands

chore(deps): update ctutils to v0.0.1-golem (#18)

7fdd44d

Co-authored-by: himaschal <himaschal@users.noreply.github.com> Test version for golem rollout

chore(deps): update ctutils to v0.0.2-golem (#19)

7b49d54

RA-8279: Do not trace admin endpoints on the backend. These are unnec…

bfbfe7a

…cessary and not recommended according to best practice. This will also keep the logserver / logsigner logs clean from unnecessary logs when scheduled jobs run like health checks or metrics scraping

sadhana-angara self-requested a review February 11, 2026 15:33

sadhana-angara approved these changes Feb 11, 2026

View reviewed changes

digilob approved these changes Feb 11, 2026

View reviewed changes

RA-8279: feat(otel): Add OpenTelemetry distributed tracing and ctutils logging integration #3

Are you sure you want to change the base?

RA-8279: feat(otel): Add OpenTelemetry distributed tracing and ctutils logging integration #3

Uh oh!

Conversation

himaschal commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ℹ️ Release Coordination (Downstream of ctutils)

Summary

Features

Configuration

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sadhana-angara commented Feb 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

himaschal commented Dec 8, 2025 •

edited

Loading