docs: update release process documentation to match recent changes #16271
Conversation
emmettbutler
requested review from
RamyElkest,
brettlangdon and
christophe-papazian
Codeowners resolved as |
|
Performance SLOsComparing candidate emmett.butler/release-process-update (4b09610) with baseline main (1aa8dec) 📈 Performance Regressions (3 suites)📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 496.167µs (SLO: <700.000µs 📉 -29.1%) vs baseline: 📈 +18.8% Memory: ✅ 42.841MB (SLO: <46.000MB -6.9%) vs baseline: +5.2% ✅ ospathbasename_noaspectTime: ✅ 432.519µs (SLO: <700.000µs 📉 -38.2%) vs baseline: +2.8% Memory: ✅ 42.566MB (SLO: <46.000MB -7.5%) vs baseline: +4.3% ✅ ospathjoin_aspectTime: ✅ 634.586µs (SLO: <700.000µs -9.3%) vs baseline: +3.5% Memory: ✅ 42.585MB (SLO: <46.000MB -7.4%) vs baseline: +4.6% ✅ ospathjoin_noaspectTime: ✅ 639.182µs (SLO: <700.000µs -8.7%) vs baseline: +3.4% Memory: ✅ 42.841MB (SLO: <46.000MB -6.9%) vs baseline: +5.2% ✅ ospathnormcase_aspectTime: ✅ 350.433µs (SLO: <700.000µs 📉 -49.9%) vs baseline: +1.3% Memory: ✅ 42.880MB (SLO: <46.000MB -6.8%) vs baseline: +5.4% ✅ ospathnormcase_noaspectTime: ✅ 362.330µs (SLO: <700.000µs 📉 -48.2%) vs baseline: +3.1% Memory: ✅ 42.585MB (SLO: <46.000MB -7.4%) vs baseline: +4.7% ✅ ospathsplit_aspectTime: ✅ 496.356µs (SLO: <700.000µs 📉 -29.1%) vs baseline: +3.8% Memory: ✅ 42.880MB (SLO: <46.000MB -6.8%) vs baseline: +5.6% ✅ ospathsplit_noaspectTime: ✅ 505.633µs (SLO: <700.000µs 📉 -27.8%) vs baseline: +3.9% Memory: ✅ 42.920MB (SLO: <46.000MB -6.7%) vs baseline: +5.6% ✅ ospathsplitdrive_aspectTime: ✅ 378.787µs (SLO: <700.000µs 📉 -45.9%) vs baseline: +2.7% Memory: ✅ 42.900MB (SLO: <46.000MB -6.7%) vs baseline: +5.5% ✅ ospathsplitdrive_noaspectTime: ✅ 72.121µs (SLO: <700.000µs 📉 -89.7%) vs baseline: -0.8% Memory: ✅ 42.920MB (SLO: <46.000MB -6.7%) vs baseline: +5.6% ✅ ospathsplitext_aspectTime: ✅ 461.874µs (SLO: <700.000µs 📉 -34.0%) vs baseline: +2.5% Memory: ✅ 42.920MB (SLO: <46.000MB -6.7%) vs baseline: +5.7% ✅ ospathsplitext_noaspectTime: ✅ 468.575µs (SLO: <700.000µs 📉 -33.1%) vs baseline: +1.2% Memory: ✅ 42.625MB (SLO: <46.000MB -7.3%) vs baseline: +4.7% 📈 iastaspectssplit - 12/12✅ rsplit_aspectTime: ✅ 162.432µs (SLO: <250.000µs 📉 -35.0%) vs baseline: 📈 +11.7% Memory: ✅ 42.664MB (SLO: <46.000MB -7.3%) vs baseline: +5.1% ✅ rsplit_noaspectTime: ✅ 159.997µs (SLO: <250.000µs 📉 -36.0%) vs baseline: +3.1% Memory: ✅ 42.546MB (SLO: <46.000MB -7.5%) vs baseline: +4.8% ✅ split_aspectTime: ✅ 150.949µs (SLO: <250.000µs 📉 -39.6%) vs baseline: +2.9% Memory: ✅ 42.546MB (SLO: <46.000MB -7.5%) vs baseline: +4.8% ✅ split_noaspectTime: ✅ 153.283µs (SLO: <250.000µs 📉 -38.7%) vs baseline: +1.1% Memory: ✅ 42.625MB (SLO: <46.000MB -7.3%) vs baseline: +5.0% ✅ splitlines_aspectTime: ✅ 145.453µs (SLO: <250.000µs 📉 -41.8%) vs baseline: +0.2% Memory: ✅ 42.743MB (SLO: <46.000MB -7.1%) vs baseline: +4.8% ✅ splitlines_noaspectTime: ✅ 149.541µs (SLO: <250.000µs 📉 -40.2%) vs baseline: -1.9% Memory: ✅ 42.546MB (SLO: <46.000MB -7.5%) vs baseline: +4.7% 📈 telemetryaddmetric - 30/30✅ 1-count-metric-1-timesTime: ✅ 3.385µs (SLO: <20.000µs 📉 -83.1%) vs baseline: 📈 +10.5% Memory: ✅ 35.606MB (SLO: <38.000MB -6.3%) vs baseline: +5.9% ✅ 1-count-metrics-100-timesTime: ✅ 206.459µs (SLO: <220.000µs -6.2%) vs baseline: +1.9% Memory: ✅ 35.429MB (SLO: <38.000MB -6.8%) vs baseline: +5.6% ✅ 1-distribution-metric-1-timesTime: ✅ 3.243µs (SLO: <20.000µs 📉 -83.8%) vs baseline: -4.9% Memory: ✅ 35.527MB (SLO: <38.000MB -6.5%) vs baseline: +5.9% ✅ 1-distribution-metrics-100-timesTime: ✅ 215.631µs (SLO: <230.000µs -6.2%) vs baseline: -1.0% Memory: ✅ 35.566MB (SLO: <38.000MB -6.4%) vs baseline: +5.6% ✅ 1-gauge-metric-1-timesTime: ✅ 2.150µs (SLO: <20.000µs 📉 -89.2%) vs baseline: -3.1% Memory: ✅ 35.547MB (SLO: <38.000MB -6.5%) vs baseline: +6.0% ✅ 1-gauge-metrics-100-timesTime: ✅ 135.979µs (SLO: <150.000µs -9.3%) vs baseline: -1.0% Memory: ✅ 35.507MB (SLO: <38.000MB -6.6%) vs baseline: +5.9% ✅ 1-rate-metric-1-timesTime: ✅ 3.020µs (SLO: <20.000µs 📉 -84.9%) vs baseline: -5.8% Memory: ✅ 35.507MB (SLO: <38.000MB -6.6%) vs baseline: +5.7% ✅ 1-rate-metrics-100-timesTime: ✅ 220.182µs (SLO: <250.000µs 📉 -11.9%) vs baseline: +2.9% Memory: ✅ 35.547MB (SLO: <38.000MB -6.5%) vs baseline: +5.9% ✅ 100-count-metrics-100-timesTime: ✅ 20.667ms (SLO: <22.000ms -6.1%) vs baseline: +1.9% Memory: ✅ 35.488MB (SLO: <38.000MB -6.6%) vs baseline: +5.7% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.220ms (SLO: <2.550ms 📉 -13.0%) vs baseline: -0.8% Memory: ✅ 35.468MB (SLO: <38.000MB -6.7%) vs baseline: +4.6% ✅ 100-gauge-metrics-100-timesTime: ✅ 1.400ms (SLO: <1.550ms -9.7%) vs baseline: ~same Memory: ✅ 35.527MB (SLO: <38.000MB -6.5%) vs baseline: +5.8% ✅ 100-rate-metrics-100-timesTime: ✅ 2.246ms (SLO: <2.550ms 📉 -11.9%) vs baseline: +1.5% Memory: ✅ 35.507MB (SLO: <38.000MB -6.6%) vs baseline: +5.6% ✅ flush-1-metricTime: ✅ 4.488µs (SLO: <20.000µs 📉 -77.6%) vs baseline: -2.9% Memory: ✅ 35.586MB (SLO: <38.000MB -6.4%) vs baseline: +4.7% ✅ flush-100-metricsTime: ✅ 173.434µs (SLO: <250.000µs 📉 -30.6%) vs baseline: -0.6% Memory: ✅ 35.586MB (SLO: <38.000MB -6.4%) vs baseline: +5.0% ✅ flush-1000-metricsTime: ✅ 2.185ms (SLO: <2.500ms 📉 -12.6%) vs baseline: ~same Memory: ✅ 36.412MB (SLO: <38.750MB -6.0%) vs baseline: +5.3% 🟡 Near SLO Breach (1 suite)🟡 tracer - 6/6✅ largeTime: ✅ 31.635ms (SLO: <32.950ms -4.0%) vs baseline: ~same Memory: ✅ 36.785MB (SLO: <39.250MB -6.3%) vs baseline: +5.1% ✅ mediumTime: ✅ 3.133ms (SLO: <3.200ms -2.1%) vs baseline: -0.6% Memory: ✅ 35.193MB (SLO: <38.750MB -9.2%) vs baseline: +4.7% ✅ smallTime: ✅ 363.652µs (SLO: <370.000µs 🟡 -1.7%) vs baseline: +3.3% Memory: ✅ 35.252MB (SLO: <38.750MB -9.0%) vs baseline: +4.7%
|
| that comes next on the currently under-development minor release line. For example: | ||
|
|
||
| * Latest release: ``4.4.0`` -> version string on main: ``4.5.0rc1`` | ||
| * Latest release: ``4.3.0rc4`` -> version string on main: ``4.3.0rc5`` |
There was a problem hiding this comment.
This will create problems with system tests one day, for new features, if new features are merged at that point in main but no 4.3.0rc5 is released.
The problem comes from the fact that suddenly, when 4.3.0rc4 is promoted to stable, the release branch is ahead (4.3.0) of the main version, but behind in term of commits.
There was a problem hiding this comment.
I think we are going to always have a race condition somewhere with system-tests.
If you open a PR for a new feature that includes a change to system-tests, the manifest/skips/etc will be based around the next version to be released on main. So if main is 4.4.0rc1 would you set the version constraint to >=4.4.0[rc1]? or 4.5.0[rc1]?
However, the version on main can change between when you open a PR and when your PR is merged.
We also don't require being up to date with main in order to merge a change in. So a PR which reports 4.4.0rc1 to system-tests, and can pass system-tests with that version constraint, could still get merged in after main has been bumped to 4.5.0rc1.
So what is the right order for managing the versioning / test activation with system-tests?
I think the general system-tests workflow which "works" is:
- Create dd-trace-py change + system-tests change at the same time
- Open system-tests PR with changes, and dd-trace-py PR with system-tests version bump to your system-tests dev commit
- Validate CI for both PRs are passing
- Merge system-tests PR
- Update your dd-trace-py PR to now point to the HEAD of
system-testsmain(with your change merged in) - Get dd-trace-py PR merged
But the final min version needed for your change is still going to potentially change before your dd-trace-py PR is merged in.
I wrote up the below before thinking about the above, figured worth keeping for rough train of thought sharing.
I think a lot of this hinges on whether the default is to include new features into the next rc release or not.
I am not sure. Once we cut an rc we should only include explicit changes ontop of rc1 for any further rcs or the actual minor release.
This way we'd end up with:
- Bump
mainto the next minor release rc14.4.0rc1->4.5.0rc1 - Cut
4.4.0rc1from the commit before the version change to4.5.0rc- the order of these two steps likely don't matter much, just depends on how sensitive we want to be to the "correctness" of version on
mainfor any merged change
- the order of these two steps likely don't matter much, just depends on how sensitive we want to be to the "correctness" of version on
- Cut
4.4release branch from the4.4.0rc1tag
This way, main always represents the next minor version to be released, such that anyone who merges anything into main can automatically know which minor version their change will 100% be included in.
This should solve Christophe's concerns/system-tests issue.
| $ git checkout <branch> | ||
| $ reno report --branch=origin/<branch> | pandoc -f rst -t gfm --wrap=none | less |
There was a problem hiding this comment.
Do we need the checkout if we are passing origin/<branch> anyway? This steps (reno report) confuses me a bit every time.
There was a problem hiding this comment.
just a fetch is needed to make sure you have latest
| $ git checkout A.B # previous release branch | ||
| $ git pull | ||
| $ git checkout -b X.Y # new release branch | ||
| $ git merge main -Xtheirs # this keeps the tags intact so that reno will work properly | ||
| $ git push -u origin X.Y |
There was a problem hiding this comment.
Are these steps correct?
Would it just be:
git checkout main #origin/main _could_ be better ?
git pull
git checkout -b X.Y
git push -u origin X.Y?
there shouldn't be any commits on X.Y-1 which aren't on main, but due to how backports work, the commit sha on X.Y-1 would be different than what was merged into main
| * main's version string is a release candidate ``rc`` version. | ||
| * Pull request a change to the version string on the main branch that sets it to ``rc1`` on the next release line. | ||
| For example: in step 1 we created branch ``1.3``, and we will change the version string on main to ``1.4.0rc1``. | ||
| * The version string on the release branch is set to a patch version on the previous minor release line. |
There was a problem hiding this comment.
if we grab from main, shouldn't it be the most recent rc for that version? e.g. 1.3.0rc1, then we update to 1.3.0 ?
| that comes next on the currently under-development minor release line. For example: | ||
|
|
||
| * Latest release: ``4.4.0`` -> version string on main: ``4.5.0rc1`` | ||
| * Latest release: ``4.3.0rc4`` -> version string on main: ``4.3.0rc5`` |
There was a problem hiding this comment.
I think we are going to always have a race condition somewhere with system-tests.
If you open a PR for a new feature that includes a change to system-tests, the manifest/skips/etc will be based around the next version to be released on main. So if main is 4.4.0rc1 would you set the version constraint to >=4.4.0[rc1]? or 4.5.0[rc1]?
However, the version on main can change between when you open a PR and when your PR is merged.
We also don't require being up to date with main in order to merge a change in. So a PR which reports 4.4.0rc1 to system-tests, and can pass system-tests with that version constraint, could still get merged in after main has been bumped to 4.5.0rc1.
So what is the right order for managing the versioning / test activation with system-tests?
I think the general system-tests workflow which "works" is:
- Create dd-trace-py change + system-tests change at the same time
- Open system-tests PR with changes, and dd-trace-py PR with system-tests version bump to your system-tests dev commit
- Validate CI for both PRs are passing
- Merge system-tests PR
- Update your dd-trace-py PR to now point to the HEAD of
system-testsmain(with your change merged in) - Get dd-trace-py PR merged
But the final min version needed for your change is still going to potentially change before your dd-trace-py PR is merged in.
I wrote up the below before thinking about the above, figured worth keeping for rough train of thought sharing.
I think a lot of this hinges on whether the default is to include new features into the next rc release or not.
I am not sure. Once we cut an rc we should only include explicit changes ontop of rc1 for any further rcs or the actual minor release.
This way we'd end up with:
- Bump
mainto the next minor release rc14.4.0rc1->4.5.0rc1 - Cut
4.4.0rc1from the commit before the version change to4.5.0rc- the order of these two steps likely don't matter much, just depends on how sensitive we want to be to the "correctness" of version on
mainfor any merged change
- the order of these two steps likely don't matter much, just depends on how sensitive we want to be to the "correctness" of version on
- Cut
4.4release branch from the4.4.0rc1tag
This way, main always represents the next minor version to be released, such that anyone who merges anything into main can automatically know which minor version their change will 100% be included in.
This should solve Christophe's concerns/system-tests issue.
| See [the support level definitions](https://docs.datadoghq.com/tracing/trace_collection/compatibility/python/#releases) for more information. | ||
|
|
||
| .. code-block:: bash | ||
| Where the EOL month is calculated thus: <this major release line's start month> + <18 months> |
There was a problem hiding this comment.
should we leave a note that you should just steal the version from the most recent minor release for that major version?
The calculation only actually matters for X.0.0 releases anyways.
This change updates the release process documentation for accuracy and clarity, particularly around guarantees related to the
project.versionstring.