RA-8279: Add OpenTelemetry distributed tracing and ctutils logging to CTFE by himaschal · Pull Request #3 · digicert/certificate-transparency-go

himaschal · 2025-12-08T14:01:46Z

ℹ️ Release Coordination (Downstream of ctutils)

This feature depends on digicert/ctutils (RA-8279).
Status: Ready for review (ensure ctutils v1.0.0 tag is available).
Plan:

Wait for digicert/ctutils PR 1 merge & v1.0.0 tag.
Update go.mod in this PR to use digicert/ctutils v1.0.0.
Merge this PR.

Summary

Integrates OpenTelemetry distributed tracing and standardized logging into the Certificate Transparency Frontend (CTFE), leveraging the shared digicert/ctutils library. This enables end-to-end observability from HTTP requests down to the Trillian backend.

Key Features

Trace Propagation: CTFE extracts incoming trace context (if any) and propagates it to Trillian Log Server/Signer via gRPC.
Request Logging: Semantic logging for HTTP requests (method, status, duration).
Configuration: Standard OTEL_* env vars + LOG_LEVEL.

Configuration

See trillian/README.md for details.

Variable	Description	Default
`OTEL_ENABLED`	Enable tracing	`false`
`OTEL_EXPORTER`	`otlp`, `stdout`	`stdout`
`OTEL_COLLECTOR_ENDPOINT`	OTLP endpoint (grpc)	`localhost:4317`
`LOG_LEVEL`	`DEBUG`, `INFO`, `WARN`, `ERROR`	`INFO`

How It Works

HTTP Handlers: Wrapped with otelhttp.NewHandler for automatic span creation on incoming requests.
gRPC Clients: Use chained interceptors (ChainedGRPCClientInterceptor) to propagate trace context to backends.
Flow: HTTP Request → CTFE → gRPC → Trillian (Log Server/Signer).

Related PRs

ctutils#1 - Shared logging library
trillian#3 - Trillian backend OTEL integration
[trillian-ctlog#32](https://github.com/digicert/trillian

Testing

Integration: Verified in Clean Run 9.0 environment. Traces successfully appear in Jaeger for /ct/v1/get-sth and other endpoints.

See full e2e testing here

… to CTFE This change integrates the digicert/ctutils shared logging library to enable OpenTelemetry-compliant distributed tracing in the Certificate Transparency Frontend (CTFE). Key changes: - Add trillian/ctfe/config/config.go with InitLogging() for OTEL configuration - Update ct_server/main.go to initialize logging and wrap HTTP handlers with otelhttp - Add chained gRPC client interceptors for trace context propagation to Trillian - Add Dockerfile.unified with SSH access for private ctutils dependency - Update go.mod/go.sum for ctutils v0.1.6 and OTEL dependencies The logging configuration is driven by environment variables: - OTEL_ENABLED: Enable/disable OpenTelemetry (default: false) - OTEL_EXPORTER: Exporter type ('otlp' or 'stdout') - OTEL_COLLECTOR_ENDPOINT: OTLP collector URL - OTEL_SERVICE_NAME: Service name for traces - OTEL_SAMPLE_RATIO: Sampling ratio (0.0-1.0) HTTP handlers are wrapped with otelhttp.NewHandler for automatic span creation, and gRPC clients use chained interceptors to propagate trace context to Trillian backends. This enables end-to-end request tracing across the CT infrastructure. Refs: RA-8279

chore(deps): update ctutils

Copilot

Pull request overview

This PR integrates OpenTelemetry distributed tracing into the Certificate Transparency Frontend (CTFE) using the digicert/ctutils shared logging library. It enables end-to-end request tracing from HTTP requests through gRPC calls to Trillian backends, with configurable trace exporters and sampling.

Changes:

Added OpenTelemetry support with environment-based configuration for tracing
Integrated ctutils library for shared logging functionality
Updated CI/CD workflows to handle private ctutils repository authentication

Reviewed changes

Copilot reviewed 13 out of 15 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
trillian/examples/deployment/docker/ctfe/Dockerfile.unified	New unified Dockerfile with support for private ctutils dependency
trillian/examples/deployment/docker/ctfe/Dockerfile	Added GitHub token authentication for private module access
trillian/docs/ManualDeployment.md	Added OpenTelemetry distributed tracing documentation section
trillian/README.md	Added Observability section with OTEL configuration reference
trillian/ctfe/instance.go	Wrapped scheduled tasks with span tracing
trillian/ctfe/handlers.go	Minor comment adjustments (commented import)
trillian/ctfe/ct_server/main.go	Initialized OTEL logging, wrapped HTTP handlers, added gRPC interceptors
trillian/ctfe/config/config.go	New centralized logging configuration with OpenTelemetry support
go.mod	Added ctutils v0.1.13-test and updated OTEL dependencies
go.sum	Updated checksums for new and upgraded dependencies
.gitignore	Added entry for ctfe_server binary
.github/workflows/update-ctutils.yaml	New workflow for automated ctutils dependency updates
.github/workflows/govulncheck.yml	Added ctutils authentication
.github/workflows/golangci-lint.yml	Added ctutils authentication
.github/workflows/codeql.yml	Added ctutils authentication

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

go.mod

Copilot · 2026-01-13T16:12:39Z

trillian/ctfe/handlers.go

 	"github.com/google/certificate-transparency-go/asn1"
 	"github.com/google/certificate-transparency-go/tls"
+
+	//"github.com/google/certificate-transparency-go/trillian/ctfe/logging"


This commented-out import statement should be removed. Leaving commented code in the codebase creates confusion and reduces maintainability. If this import is not needed, it should be deleted entirely.

Suggested change

//"github.com/google/certificate-transparency-go/trillian/ctfe/logging"

trillian/ctfe/ct_server/main.go

trillian/examples/deployment/docker/ctfe/Dockerfile.unified

trillian/examples/deployment/docker/ctfe/Dockerfile

chore(deps): update ctutils

Copilot

Pull request overview

Copilot reviewed 12 out of 14 changed files in this pull request and generated 11 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.github/workflows/codeql.yml

.github/workflows/govulncheck.yml

trillian/examples/deployment/docker/ctfe/Dockerfile

trillian/ctfe/ct_server/main.go

.github/workflows/update-ctutils.yaml

trillian/ctfe/config/config.go

trillian/docs/ManualDeployment.md

.github/workflows/golangci-lint.yml

…s` module across workflows and enhance the `update-ctutils` workflow to explicitly resolve and report the latest version.

trillian/ctfe/ct_server/main.go

…ly order tracing and CORS middleware, and update deployment documentation with new OpenTelemetry and logging environment variables.

This reverts commit 6a215da.

…ssue" This reverts commit 9ac12a9.

Co-authored-by: himaschal <himaschal@users.noreply.github.com>

trillian/ctfe/ct_server/main.go

…ne is not double-wrapping but serve different purposes.

…ving unnecessary dependency download step

Co-authored-by: himaschal <himaschal@users.noreply.github.com> Test version for golem rollout

sadhana-angara · 2026-02-11T15:31:48Z

QA Evidence for CT Log :

Test Case 1 : Deploy Jaeger and Configure OTLP Export :
Status : PASS
Evidence :

Jaeger pod deployed successfully -

Services configured with OTLP export -

Port forwarding established -

Jaeger UI accessible at http://localhost:16686

CTFE API accessible at http://localhost:6962

========================================================================================

Test 2: Verify Distributed Tracing in Jaeger UI
Status : PASS
Evidence :

Multiple traces visible in search results (6+ traces) : Yes
Traces show both ctfe and trillian-logserver services : Yes
Span hierarchy shows parent-child relationships : Yes
Timing data present for each span (typically 3-25ms) : Yes
Operation names visible (e.g., ctfe_internal_sth_force) : Yes

========================================================================================

Test 3: W3C Trace Context Propagation
Status : PASS
Evidence :

Request succeeds (200 OK) : Yes
Custom trace_id appears in Jaeger UI : Yes
Trace shows in search results within 10 seconds : Yes

========================================================================================

Test 4: Cross-Service Trace Propagation
Status : PASS
Evidence :

Same trace_id found in CTFE logs : Yes
Same trace_id found in Trillian logserver logs : Yes
Logs show parent-child relationship via span_id : Yes

========================================================================================

Test 5: Structured Logging with Trace Context
Status : PASS
Evidence :

Logs in JSON format : Yes
trace_id field present (32-char hex) : Yes
span_id field present (16-char hex) : Yes
parent_source field present (values: client_header, grpc_metadata, or system_generated) : Yes
elapsed_ms field present (timing data) : Yes

========================================================================================

Test 6: Multiple Requests with Shared Trace
Status : PASS
Evidence :

**- All 3 requests succeed : **

- Logs show 3+ entries with same trace_id :

- Each entry has different span_id :

========================================================================================

Test 7: Performance and Timing Analysis
Status : PASS
Evidence : In Jaeger UI, examine 5-10 different traces

1. Trace ID : 4dd4c18687c1d0f1b54bba5f7223efd7 :
Typical get-sth operation: 5.47ms
gRPC call to Trillian: 3.39ms on server side
CTFE overhead : 5.1 - 3.39 = 1.71ms
No traces showing errors or timeouts.

2.Trace ID : 089ff8e936e3df81afbfb9fe416f3194 :
Typical get-sth operation: 3.26ms
gRPC call to Trillian: 2.2ms on server side
CTFE overhead : 3.04 - 2.2 = 0.84ms
No traces showing errors or timeouts.

3.Trace ID : b3aec802d23c219702cc36f7d7a3b1e6 :
Typical get-sth operation: 5.1ms
gRPC call to Trillian: 3.37ms on server side
CTFE overhead : 4.75 - 3.37 = 1.38ms
No traces showing errors or timeouts.

4.Trace ID : 9a4723af4335407b39318948fb92d9b6 :
Typical get-sth operation: 4.52ms
gRPC call to Trillian: 2.94ms on server side
CTFE overhead : 4.52 - 2.94 = 1.58ms
No traces showing errors or timeouts.

5.Trace ID : bea9a080ada3f6739735ab113f1ad7c2 :
Typical get-sth operation: 7.95ms
gRPC call to Trillian: 3.78ms on server side
CTFE overhead : 7.09 - 3.78 = 3.31ms
No traces showing errors or timeouts.

6.Trace ID : 4ae35982188663fee3d8eeb622866356 :
Typical get-sth operation: 8.76ms
gRPC call to Trillian: 4.69ms on server side
CTFE overhead : 7.83 - 4.69 = 3.14ms
No traces showing errors or timeouts.

7.Trace ID : 8018b3ff51e0b343206a6b984802c18b :
Typical get-sth operation: 6.14ms
gRPC call to Trillian: 4.21ms on server side
CTFE overhead : 6.14 - 4.21 = 1.93ms
No traces showing errors or timeouts.

8.Trace ID : a3699b3aef49a80fc73aae7bb5bd6e5f :
Typical get-sth operation: 5.2ms
gRPC call to Trillian: 3.61ms on server side
CTFE overhead : 4.97 - 3.61 = 1.36ms
No traces showing errors or timeouts.

9.Trace ID : 4a4f82d3b182303ed0b773c0460b9169 :
Typical get-sth operation: 7.65ms
gRPC call to Trillian: 4.19ms on server side
CTFE overhead : 7.24 - 4.19 = 3.05ms
No traces showing errors or timeouts.

10.Trace ID : 26a99abfc5ad08be508a00b301abb032 :
Typical get-sth operation: 5.08ms
gRPC call to Trillian: 2.26ms on server side
CTFE overhead : 4.59 - 2.26 = 2.33ms
No traces showing errors or timeouts.

========================================================================================

Test 8: Service Dependencies Visualization
Status : PASS
Evidence :

Clear call chain: ctfe → trillian-logserver : Yes
gRPC communication visible in spans : Pending
No unexpected service dependencies : Pending

himaschal and others added 19 commits December 8, 2025 15:59

Update ctutils to v0.1.7

85a4ee1

ci: add workflow to update ctutils dependencies

8df6935

test: trigger workflow run for v0.1.9-test

816e020

fix(ci): use oauth2 syntax for authentication and debug token

4521100

fix(ci): revert to standardized secret name CTUTILS_PRIVATE_REPO_TOKEN

3520e32

fix(ci): use GITHUB_TOKEN for PR creation

22ff749

fix(ci): auth to private ctutils for lint and vuln checks

dd0ec76

fix(ci): auth to private ctutils and set GOPRIVATE for CodeQL

769842f

test: clean run trigger for v0.1.11-test

7b12f4b

chore(deps): update ctutils to latest version

1f76b3c

Merge pull request #4 from digicert/chore/update-ctutils

07db48e

chore(deps): update ctutils

chore: test pipeline run with ctutils v0.1.12-test

4ec222c

chore(deps): update ctutils to latest version

8cd756b

Merge pull request #5 from digicert/chore/update-ctutils

b628160

chore(deps): update ctutils

chore: test pipeline run with ctutils v0.1.13-test

9b0f1eb

chore(deps): update ctutils to latest version

e52583c

Merge pull request #6 from digicert/chore/update-ctutils

a20155b

chore(deps): update ctutils

chore: revert pipeline testing configurations to production ready state

8cdad3d

himaschal requested a review from Copilot January 13, 2026 16:04

Copilot started reviewing on behalf of himaschal January 13, 2026 16:05 View session

Copilot AI reviewed Jan 13, 2026

View reviewed changes

himaschal and others added 8 commits January 14, 2026 18:56

RA-8279: Modified to use github app

d88d1e8

ci: use github app for private repo access

4e46fe3

chore: temp add push trigger for test

ecbdc34

ci: update all workflows to use github app token

7016b92

ci: add debug step for failed token

228e0cb

chore: remove debug step

27e2b9d

chore(deps): update ctutils to latest version

b7a8314

Merge pull request #8 from digicert/chore/update-ctutils-test

97cb3ed

chore(deps): update ctutils

Copilot started reviewing on behalf of himaschal January 18, 2026 09:27 View session

Copilot AI reviewed Jan 18, 2026

View reviewed changes

himaschal added 2 commits January 22, 2026 16:04

Add shutdown handling, config consistency, and tests

48bc2be

Add dockerfile.local to .gitignore. Only used for local dev / build.

61c6765

himaschal changed the title ~~feat(otel): Add OpenTelemetry distributed tracing and ctutils logging to CTFE~~ RA-8279: Add OpenTelemetry distributed tracing and ctutils logging to CTFE Jan 23, 2026

himaschal added 2 commits January 23, 2026 11:35

build: update ctutils dependency to fix CI build

11c02e1

RA-8279: Address PR comments. Add GONOSUMDB for the private `ctutil…

c84255b

…s` module across workflows and enhance the `update-ctutils` workflow to explicitly resolve and report the latest version.

himaschal commented Jan 26, 2026

View reviewed changes

trillian/ctfe/ct_server/main.go Show resolved Hide resolved

himaschal and others added 10 commits January 26, 2026 11:50

RA_8279: Addressed PR comments. Refactor CTFE HTTP server to explicit…

15d72e4

…ly order tracing and CORS middleware, and update deployment documentation with new OpenTelemetry and logging environment variables.

fix(ci): robust latest release resolution for ctutils

9e42da9

fix(ci): add GONOPROXY for private ctutils module access

c860af5

fix(ci): add GONOPROXY for private ctutils module access

85decb3

test: remove GONOPROXY to verify if it's necessary

6a215da

Revert "test: remove GONOPROXY to verify if it's necessary"

4e22424

This reverts commit 6a215da.

test: remove GONOPROXY again to check if fixed tag resolves issue

9ac12a9

Revert "test: remove GONOPROXY again to check if fixed tag resolves i…

bd12cbc

…ssue" This reverts commit 9ac12a9.

chore(deps): remove invalid checksum for v0.1.26-test from go.sum

dc26be0

chore(deps): update ctutils to v0.1.27-test (#17)

9081cf4

Co-authored-by: himaschal <himaschal@users.noreply.github.com>

digilob reviewed Jan 26, 2026

View reviewed changes

trillian/ctfe/ct_server/main.go Show resolved Hide resolved

trillian/ctfe/ct_server/main.go Outdated Show resolved Hide resolved

RA-8279: Added a comment areound the mux handler and why the given li…

304b523

…ne is not double-wrapping but serve different purposes.

digilob approved these changes Jan 26, 2026

View reviewed changes

himaschal and others added 6 commits January 29, 2026 14:11

docker: local CTFE Dockerfile tweak to skip licensing for local builds

571bb13

Update .gitignore

0ea38b3

RA-8279: Fix formatting on the yamls.

f43856d

RA-8279: refactor(Dockerfile.local): streamline build process by remo…

6b596de

…ving unnecessary dependency download step

chore(deps): update ctutils to v0.0.1-golem (#18)

e7fcc53

Co-authored-by: himaschal <himaschal@users.noreply.github.com> Test version for golem rollout

chore(deps): update ctutils to v0.0.2-golem (#19)

fdcea5b

sadhana-angara self-requested a review February 11, 2026 15:33

sadhana-angara approved these changes Feb 11, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RA-8279: Add OpenTelemetry distributed tracing and ctutils logging to CTFE#3

RA-8279: Add OpenTelemetry distributed tracing and ctutils logging to CTFE#3
himaschal wants to merge 58 commits intomasterfrom
RA-8279_improve_application_visibility_timing_logs

himaschal commented Dec 8, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI Jan 13, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sadhana-angara commented Feb 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

himaschal commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ℹ️ Release Coordination (Downstream of ctutils)

Summary

Key Features

Configuration

How It Works

Related PRs

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sadhana-angara commented Feb 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

himaschal commented Dec 8, 2025 •

edited

Loading