Skip to content

Conversation

@morrisonlevi
Copy link
Collaborator

@morrisonlevi morrisonlevi commented Jan 30, 2026

Description

On master, we send every sample to another thread and attempt to wake that thread to process the sample. The problem is the syscall here is almost as expensive as collecting the sample if the other thread is asleep, which is often the case in NTS builds. Here's a profile showing this (but note it's not always this close):

Screenshot 2026-01-30 at 4 30 44 PM

This git branch avoids that. Instead it writes to a queue and samples are handled when that background thread wakes up for other reasons. Notably one of those reasons is every 10ms when wall-time is enabled (default) so probably this should work pretty well without filling the queues. The queues are larger on ZTS builds, though by a fixed multiplier (16).

Reviewer checklist

  • Test coverage seems ok.
  • Appropriate labels assigned.

On every sample we send data to another thread and attempt to wake that
thread. The problem is the syscall here is almost as expensive as
collecting the sample if the other thread is asleep, which is often the
case.

This branch avoids that. Instead it writes to a queue and samples are
handled when that background thread wakes up for other reasons. Notably
one of those reasons is every 10ms when wall-time is enabled (default) so
_probably_ this should work pretty well without filling the queues.
@morrisonlevi morrisonlevi added cat:performance profiling Relates to the Continuous Profiler labels Jan 30, 2026
@datadog-datadog-prod-us1
Copy link

datadog-datadog-prod-us1 bot commented Jan 30, 2026

⚠️ Tests

Fix all issues with Cursor

⚠️ Warnings

🧪 1024 Tests failed

    testSearchPhpBinaries from integration.DDTrace\Tests\Integration\PHPInstallerTest (Fix with Cursor)

    testSimplePushAndProcess from laravel-58-test.DDTrace\Tests\Integrations\Laravel\V5_8\QueueTest (Fix with Cursor)

testSimplePushAndProcess from laravel-8x-test.DDTrace\Tests\Integrations\Laravel\V8_x\QueueTest (Datadog) (Fix with Cursor)
DDTrace\Tests\Integrations\Laravel\V8_x\QueueTest::testSimplePushAndProcess
Test code or tested code printed unexpected output: spanLinksTraceId: 697e42550000000053500fa8f04af4d2
tid: 697e425500000000
hexProcessTraceId: 53500fa8f04af4d2
hexProcessSpanId: ad599ca97e9520f7
processTraceId: 6003315521545237714
processSpanId: 12491187293275889911

phpvfscomposer://tests/vendor/phpunit/phpunit/phpunit:106
View all

ℹ️ Info

❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: b622e0f | Docs | Datadog PR Page | Was this helpful? Give us feedback!

@codecov-commenter
Copy link

codecov-commenter commented Jan 30, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.12%. Comparing base (f7f9271) to head (b622e0f).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3611      +/-   ##
==========================================
- Coverage   62.20%   62.12%   -0.08%     
==========================================
  Files         141      141              
  Lines       13387    13387              
  Branches     1753     1753              
==========================================
- Hits         8327     8317      -10     
- Misses       4263     4270       +7     
- Partials      797      800       +3     

see 3 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f7f9271...b622e0f. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Under sustained load in ZTS, this could otherwise run forever.
@pr-commenter
Copy link

pr-commenter bot commented Jan 30, 2026

Benchmarks [ profiler ]

Benchmark execution time: 2026-01-31 18:02:19

Comparing candidate commit b622e0f in PR branch levi/messages with baseline commit f7f9271 in branch master.

Found 9 performance improvements and 6 performance regressions! Performance is the same for 14 metrics, 7 unstable metrics.

scenario:php-profiler-exceptions-with-profiler

  • 🟩 cpu_usage_percentage [-4.235%; -3.725%]

scenario:php-profiler-exceptions-with-profiler-and-timeline

  • 🟩 cpu_usage_percentage [-4.033%; -3.589%]

scenario:php-profiler-timeline-memory-with-profiler

  • 🟩 cpu_usage_percentage [-50.395%; -49.401%]
  • 🟩 cpu_user_time [-452.483ms; -430.893ms] or [-38.554%; -36.714%]
  • 🟩 execution_time [-189.360ms; -176.112ms] or [-20.186%; -18.774%]

scenario:php-profiler-timeline-memory-with-profiler-and-timeline

  • 🟩 cpu_system_time [-421.040ms; -386.969ms] or [-79.435%; -73.007%]
  • 🟩 cpu_usage_percentage [-56.624%; -56.165%]
  • 🟩 cpu_user_time [-732.226ms; -690.810ms] or [-46.885%; -44.234%]
  • 🟩 execution_time [-378.042ms; -364.195ms] or [-30.718%; -29.593%]

scenario:walk_stack/1

  • 🟥 wall_time [+8.706µs; +8.711µs] or [+68.682%; +68.723%]

scenario:walk_stack/50

  • 🟥 wall_time [+6.503µs; +6.517µs] or [+42.016%; +42.105%]

scenario:walk_stack/99

  • 🟥 wall_time [+6.487µs; +6.498µs] or [+41.949%; +42.017%]

scenario:walk_stack_instructions/1

  • 🟥 instructions [+61.9K instructions; +61.9K instructions] or [+58.773%; +58.781%]

scenario:walk_stack_instructions/50

  • 🟥 instructions [+54.0K instructions; +54.0K instructions] or [+51.221%; +51.227%]

scenario:walk_stack_instructions/99

  • 🟥 instructions [+53.6K instructions; +53.6K instructions] or [+50.749%; +50.752%]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cat:performance profiling Relates to the Continuous Profiler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants