HBASE-29838 Run Hadoop Check as a GitHub Action #7651

ndimiduk · 2026-01-20T07:05:05Z

No description provided.

ndimiduk · 2026-01-20T14:09:02Z

Looks like we occupied the runner for 6h and then it was aborted.

Apache9 · 2026-01-20T16:27:28Z

https://infra.apache.org/github-actions-policy.html

The policy here does not say about the 6 hours timeout...

We can ask infra about the rules and the size of the github runners, our jenkins runners finished in "343m 23s", which was very close to 6 hours, so if the machine of the github runner is weaker, the build will be very easy to cost more than 6 hours...

ndimiduk · 2026-01-20T16:35:32Z

My action itself has timeout-minutes: 600 . i'll try bumping this up.

ndimiduk · 2026-01-20T16:38:31Z

or not. 6h is GH's hard limit, https://docs.github.com/en/actions/reference/limits

Apache9 · 2026-01-21T01:52:22Z

Then maybe we should try self hosted github runners? For self hosted runners the execution time limit is 5 days...

ndimiduk · 2026-01-23T07:07:12Z

Then maybe we should try self hosted github runners? For self hosted runners the execution time limit is 5 days...

Yes we should bring this back to our CI discussions with Infra. Maybe we can borrow from the pool of new Jenkins workers while we continue to build this out.

Yetus is supposed to provide smart, selective detection of module changes when it decided which tests to run. I think the new .github directory broke that for this run, so I've pushed a change to exclude it, maybe that will help.

I'm also going to see if I can manually parallelize the unit test runs -- maybe break out three separate checks for the three main unit test groups or something like that.

Apache9 · 2026-01-23T09:05:27Z

Then maybe we should try self hosted github runners? For self hosted runners the execution time limit is 5 days...

Yes we should bring this back to our CI discussions with Infra. Maybe we can borrow from the pool of new Jenkins workers while we continue to build this out.

Yetus is supposed to provide smart, selective detection of module changes when it decided which tests to run. I think the new .github directory broke that for this run, so I've pushed a change to exclude it, maybe that will help.

I'm also going to see if I can manually parallelize the unit test runs -- maybe break out three separate checks for the three main unit test groups or something like that.

Maybe we could split the UTs run as a seperated github check? Or even more, we could split the UTs run as two seperated check, one has -PrunDevTests(for small/medium tests) and one has -PrunLargeTests(for large tests).

ndimiduk · 2026-01-23T12:37:28Z

Maybe we could split the UTs run as a seperated github check? Or even more, we could split the UTs run as two seperated check, one has -PrunDevTests(for small/medium tests) and one has -PrunLargeTests(for large tests).

Yep, that's exactly my thinking as well. Landing these other cleanup issues and I'll be back.

ndimiduk · 2026-01-23T13:20:52Z

Okay this is better. Module selection chose only hbase-examples for running the unit tests.

-1 overall
  
  | Vote |      Subsystem |  Runtime   | Comment
  ============================================================================
  |   0  |        reexec  |   1m 13s   | Docker mode activated. 
  |  -0  |         yetus  |   0m  4s   | Unprocessed flag(s): --brief-report-file 
  |      |                |            | --spotbugs-strict-precheck
  |      |                |            | --author-ignore-list
  |      |                |            | --blanks-eol-ignore-file
  |      |                |            | --blanks-tabs-ignore-file
  |      |                |            | --quick-hadoopcheck
  +---------------------------------------------------------------------------
  |      |                |            | Prechecks 
  +---------------------------------------------------------------------------
  +---------------------------------------------------------------------------
  |      |                |            | master Compile Tests 
  +---------------------------------------------------------------------------
  |  +1  |    mvninstall  |   3m 25s   | master passed 
  |  +1  |       compile  |   0m 19s   | master passed 
  |  +1  |       javadoc  |   0m 17s   | master passed 
  |  +1  |    shadedjars  |   4m 47s   | branch has no errors when building our 
  |      |                |            | shaded downstream artifacts.
  +---------------------------------------------------------------------------
  |      |                |            | Patch Compile Tests 
  +---------------------------------------------------------------------------
  |  +1  |    mvninstall  |   3m  7s   | the patch passed 
  |  +1  |       compile  |   0m 18s   | the patch passed 
  |  +1  |         javac  |   0m 18s   | the patch passed 
  |  +1  |       javadoc  |   0m 15s   | the patch passed 
  |  +1  |    shadedjars  |   4m 51s   | patch has no errors when building our 
  |      |                |            | shaded downstream artifacts.
  +---------------------------------------------------------------------------
  |      |                |            | Other Tests 
  +---------------------------------------------------------------------------
  |  -1  |          unit  |   0m 24s   | hbase-examples in the patch failed. 
  |      |                |  19m 12s   |
  
  
  || Subsystem || Report/Notes ||
  ============================================================================
  |  unit  | /home/runner/work/hbase/hbase/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-examples.txt |
  | Docker | ClientAPI=1.48 ServerAPI=1.48 base: /home/runner/work/hbase/hbase/yetus-jdk17-hadoop3-check/output/Dockerfile |
  | GITHUB PR | https://github.com/apache/hbase/pull/7651 |
  | Optional Tests | javac javadoc unit compile shadedjars |
  | uname | Linux 9b6b1363e88a 6.11.0-1018-azure #18~24.04.1-Ubuntu SMP Sat Jun 28 04:46:03 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
  | Build tool | maven |
  | Personality | dev-support/hbase-personality.sh |
  | git revision | master / adfd0560c941109283928946bd671a5836604e46 |
  | Default Java | Eclipse Adoptium-17.0.11+9 |
  | Max. process+thread count | 72 (vs. ulimit of 30000) |
  | modules | C: hbase-examples U: hbase-examples |
  | versions | git=2.34.1 maven=3.9.8 |
  | Powered by | Apache Yetus 0.15.0 https://yetus.apache.org/ |

ndimiduk · 2026-01-23T15:51:15Z

Looking over out last successful nightly on master, the Large tests on hbase-server still took 7h. I'm looking for other ways to partition this up.

ndimiduk · 2026-01-25T14:10:12Z

The crazy idea has legs. Let me post about it on the JIRA.

Apache9 · 2026-01-25T14:15:01Z

dev-support/hbase-personality.sh

+  if [[ -n "${INCLUDE_TESTS_PATTERN}" ]]; then
+    yetus_debug "Using include pattern: ${INCLUDE_TESTS_PATTERN}"
+    eval "${__resultvar}='-Dtest=${INCLUDE_TESTS_PATTERN}'"
+  elif [[ -n "${EXCLUDE_TESTS_URL}" ]]; then


We should not use elif here? We use this to exclude the flaky tests in pre commit and nightly...

-Dtest.include.pattern=xxx -Dtest.exclude.pattern=xxx

Apache9 · 2026-01-25T14:35:59Z

.github/workflows/yetus-jdk17-hadoop3-check.yml

+    runs-on: ubuntu-latest
+    timeout-minutes: 360
+
+    strategy:


I'm not very familiar with the mechanism here, but I think we should run JDK17 hadoop checks other than the unit plugin first, and then start the below for runs with unit plugin only?

Apache-HBase · 2026-01-27T15:27:14Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/6/console in case of problems.

Apache-HBase · 2026-01-27T15:28:36Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/6/console in case of problems.

Apache-HBase · 2026-01-27T16:32:23Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 44s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+0 🆗	yamllint	0m 0s		yamllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ master Compile Tests _
+1 💚	mvninstall	4m 50s		master passed
+1 💚	compile	14m 0s		master passed
+1 💚	spotless	1m 11s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	4m 39s		the patch passed
+1 💚	compile	12m 29s		the patch passed
+1 💚	javac	12m 29s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	pylint	0m 12s	/results-pylint.txt	The patch generated 57 new + 0 unchanged - 0 fixed = 57 total (was 0)
+1 💚	shellcheck	0m 2s		No new issues.
+1 💚	xmllint	0m 0s		No new issues.
+1 💚	hadoopcheck	15m 54s		Patch does not cause any errors with Hadoop 3.3.6 3.4.1.
-1 ❌	spotless	0m 17s	/patch-spotless.txt	patch has 55 errors when running spotless:check, run spotless:apply to fix.
			_ Other Tests _
-1 ❌	asflicense	0m 25s	/results-asflicense.txt	The patch generated 2 ASF License warnings.
		65m 34s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/6/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7651
Optional Tests	dupname asflicense codespell detsecrets shellcheck shelldocs spotless pylint markdownlint yamllint javac xmllint hadoopcheck compile
uname	Linux 7851ab68384d 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `1134084`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	191 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/6/console
versions	git=2.34.1 maven=3.9.8 pylint=2.15.5 shellcheck=0.8.0 xmllint=20913
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2026-01-27T19:34:03Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 12s		Docker mode activated.
-0 ⚠️	yetus	0m 4s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+1 💚	mvninstall	2m 36s		master passed
+1 💚	compile	1m 38s		master passed
+1 💚	javadoc	1m 33s		master passed
+1 💚	shadedjars	4m 24s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 16s		the patch passed
+1 💚	compile	1m 40s		the patch passed
+1 💚	javac	1m 40s		the patch passed
+1 💚	javadoc	1m 29s		the patch passed
-1 ❌	shadedjars	0m 6s	/patch-shadedjars.txt	patch has 7 errors when building our shaded downstream artifacts.
			_ Other Tests _
-1 ❌	unit	226m 54s	/patch-unit-root.txt	root in the patch failed.
		247m 35s

Subsystem	Report/Notes
Docker	ClientAPI=1.53 ServerAPI=1.53 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/6/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7651
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 16ff0856b062 6.14.0-1018-aws #18~24.04.1-Ubuntu SMP Mon Nov 24 19:46:27 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `1134084`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/6/testReport/
Max. process+thread count	6244 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/6/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2026-01-27T19:36:12Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/7/console in case of problems.

Apache-HBase · 2026-01-27T19:36:46Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/7/console in case of problems.

Apache-HBase · 2026-01-27T20:09:54Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+0 🆗	yamllint	0m 0s		yamllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ master Compile Tests _
+1 💚	mvninstall	2m 41s		master passed
+1 💚	compile	6m 26s		master passed
+1 💚	spotless	0m 38s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 16s		the patch passed
+1 💚	compile	6m 26s		the patch passed
+1 💚	javac	6m 26s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	pylint	0m 4s	/results-pylint.txt	The patch generated 57 new + 0 unchanged - 0 fixed = 57 total (was 0)
+1 💚	shellcheck	0m 1s		No new issues.
+1 💚	xmllint	0m 0s		No new issues.
+1 💚	hadoopcheck	8m 27s		Patch does not cause any errors with Hadoop 3.3.6 3.4.1.
-1 ❌	spotless	0m 5s	/patch-spotless.txt	patch has 55 errors when running spotless:check, run spotless:apply to fix.
			_ Other Tests _
-1 ❌	asflicense	0m 10s	/results-asflicense.txt	The patch generated 2 ASF License warnings.
		33m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/7/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7651
Optional Tests	dupname asflicense codespell detsecrets shellcheck shelldocs spotless pylint markdownlint yamllint javac xmllint hadoopcheck compile
uname	Linux a8e6a820f3c9 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `a80e268`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	196 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/7/console
versions	git=2.34.1 maven=3.9.8 pylint=2.15.5 shellcheck=0.8.0 xmllint=20913
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2026-01-27T23:44:24Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 13s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+1 💚	mvninstall	2m 34s		master passed
+1 💚	compile	1m 43s		master passed
+1 💚	javadoc	1m 31s		master passed
+1 💚	shadedjars	4m 22s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 17s		the patch passed
+1 💚	compile	1m 39s		the patch passed
+1 💚	javac	1m 39s		the patch passed
+1 💚	javadoc	1m 29s		the patch passed
-1 ❌	shadedjars	0m 6s	/patch-shadedjars.txt	patch has 7 errors when building our shaded downstream artifacts.
			_ Other Tests _
-1 ❌	unit	228m 19s	/patch-unit-root.txt	root in the patch failed.
		248m 58s

Subsystem	Report/Notes
Docker	ClientAPI=1.53 ServerAPI=1.53 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/7/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7651
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 742e2d25895a 6.14.0-1018-aws #18~24.04.1-Ubuntu SMP Mon Nov 24 19:46:27 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `a80e268`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/7/testReport/
Max. process+thread count	5829 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/7/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

This comment has been minimized.

Sign in to view

ndimiduk force-pushed the 29838-jdk-checks-master branch from 1829212 to 352a9a1 Compare January 23, 2026 07:03

This comment has been minimized.

Sign in to view

ndimiduk force-pushed the 29838-jdk-checks-master branch from 352a9a1 to adfd056 Compare January 23, 2026 12:48

This comment has been minimized.

Sign in to view

ndimiduk force-pushed the 29838-jdk-checks-master branch from adfd056 to ad51411 Compare January 23, 2026 16:28

This comment has been minimized.

Sign in to view

Apache9 reviewed Jan 25, 2026

View reviewed changes

ndimiduk added 3 commits January 27, 2026 16:16

HBASE-29838 Run Hadoop Check as a GitHub Action

dd43220

try something crazy

1de8f7a

add a job to compute test group "waves"

ab00f4b

ndimiduk force-pushed the 29838-jdk-checks-master branch from ad51411 to 1134084 Compare January 27, 2026 15:18

ndimiduk added 2 commits January 27, 2026 16:39

Fix include/exclude test args to work together

ce12346

force test whole project

a80e268

ndimiduk force-pushed the 29838-jdk-checks-master branch from 1134084 to a80e268 Compare January 27, 2026 15:39

HBASE-29838 Run Hadoop Check as a GitHub Action #7651

Are you sure you want to change the base?

HBASE-29838 Run Hadoop Check as a GitHub Action #7651

Conversation

ndimiduk commented Jan 20, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ndimiduk commented Jan 20, 2026

Uh oh!

Apache9 commented Jan 20, 2026

Uh oh!

ndimiduk commented Jan 20, 2026

Uh oh!

ndimiduk commented Jan 20, 2026

Uh oh!

Apache9 commented Jan 21, 2026

Uh oh!

ndimiduk commented Jan 23, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Apache9 commented Jan 23, 2026

Uh oh!

ndimiduk commented Jan 23, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

ndimiduk commented Jan 23, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

ndimiduk commented Jan 23, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ndimiduk commented Jan 25, 2026

Uh oh!

Apache9 Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

Apache9 Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

Apache9 Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

Apache-HBase commented Jan 27, 2026

Uh oh!

Apache-HBase commented Jan 27, 2026

Uh oh!

Apache-HBase commented Jan 27, 2026

Uh oh!

Apache-HBase commented Jan 27, 2026

Uh oh!

Apache-HBase commented Jan 27, 2026

Uh oh!

Apache-HBase commented Jan 27, 2026

Uh oh!

Apache-HBase commented Jan 27, 2026

Uh oh!

Apache-HBase commented Jan 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!