feat: node-exporter into vhd build #7704

chmill-zz · 2026-01-22T00:14:23Z

What this PR does / why we need it:

this is adding node-exporter into the vhdbuild by default. At the end of the vhdbuild we systemctl disable to allow a fresh start during cse letting the node-exporter-startup script run and gather up node specific details needed.

Which issue(s) this PR fixes:

Fixes #

cameronmeissner · 2026-01-23T23:17:07Z

parts/linux/cloud-init/artifacts/node-exporter/baseline/etc/node-exporter.d/web-config.yml

@@ -0,0 +1,5 @@
+tls_server_config:
+  cert_file: "/etc/kubernetes/certs/kubeletserver.crt"


these paths aren't necessarily correct - they depend on whether kubelet serving certificate rotation is enabled - when it's disabled these paths are correct, however when it's enabled both cert_file and key_file should point towards: /var/lib/kubelet/pki/kubelet-server-current.pem

interesting this is copy paste from the aks-vm-extension repo and what is on my node today. I don't see any where it's touched, just a static file.

root@aks-sys-41317600-vmss000000:/etc/node-exporter.d# cat web-config.yml tls_server_config: cert_file: "/etc/kubernetes/certs/kubeletserver.crt" key_file: "/etc/kubernetes/certs/kubeletserver.key" client_auth_type: "RequireAndVerifyClientCert" client_ca_file: "/etc/kubernetes/certs/ca.crt"

i think we could address this in the node-exporter-startup.sh and check for the existence of /var/lib/kubelet/pki/kubelet-server-current.pem and if it exists use it. And if not use /etc/kubernetes/certs/kubeletserver.crt

that's tricky - kubeelet-server-currnet.pem won't exist until after kubelet requests it from the control plane after CSE exists, though we can check IMDS during CSE to see whether the feature itself will be enabled (like we currently do in configureKubeletServing)

alrighty. Adjusted the service to wait for kubelet. Startup script to check the imds cache file first. If that's not around then call imds ourselves. And at the very end check if /var/lib/kubelet/pki/kubelet-server-current.pem exists and everything else said no for some reason.

...ux/cloud-init/artifacts/node-exporter/baseline/etc/systemd/system/node-exporter-restart.path

parts/linux/cloud-init/artifacts/node-exporter/node-exporter-startup.sh

cameronmeissner · 2026-01-23T23:21:17Z

parts/linux/cloud-init/artifacts/cse_config.sh

+configureNodeExporter() {
+    echo "Configuring Node Exporter"
+    # Check for skip file to determine if node-exporter was installed on this VHD
+    if [ ! -f /etc/node-exporter.d/skip_vhd_node_exporter ]; then


if the skip file isn't there, meaning we don't want to skip, we end up skipping? this seems backwards

also, rather than commit and manage an empty flag file in version control, can we instead pivot off some other node exporter asset file's existence?

skip file is for the extension logic as a signal that the component is agentbaker managed and extension should skip all managing.

existing nodes should have all components from extension previously. If we were to check for node-exporter running it should always be there vhd or extension install originally and we wouldn't be able to tell which. The skip file becomes a clear sign that this was vhd baked and extension should ignore.

once we get to a point where extension isn't needed anymore we can easily remove it

parts/linux/cloud-init/artifacts/cse_helpers.sh

cameronmeissner · 2026-01-23T23:24:01Z

vhdbuilder/packer/test/linux-vhd-content-test.sh

+  # Check for the skip sentinel file - this is essential for e2e testing
+  # If this file doesn't exist, e2e tests will silently skip node-exporter validation. Which is bad.
+  local skip_file="/etc/node-exporter.d/skip_vhd_node_exporter"
+  if [ ! -f "$skip_file" ]; then


again, this seems backwards

cameronmeissner · 2026-01-23T23:24:50Z

parts/linux/cloud-init/artifacts/node-exporter/node-exporter-startup.sh

@@ -0,0 +1,32 @@
+#!/bin/sh
+
+if [ "$(cat /etc/os-release | grep ^ID= | cut -c 4-)" = "flatcar" ]; then


do we need the flatcar check? it looks like we don't install node exporter on flatcar according to what's currently in install-dependencies.sh

yeah for the initial setup in AB i'm just looking at getting ubuntu and mariner working. Plan is to get all distros and this is copy paste from existing extension so eventually needed

vhdbuilder/packer/install-dependencies.sh

r2k1 · 2026-01-24T06:54:03Z

vhdbuilder/packer/install-node-exporter.sh

+    [ -n "${VHD_LOGS_FILEPATH:-}" ] && echo "  - node-exporter ${NODE_EXPORTER_VERSION}-${NODE_EXPORTER_REVISION}" >> "${VHD_LOGS_FILEPATH}"
+
+    rm -rf "${NODE_EXPORTER_BUILD_ROOT}"
+}


A random question. Would you prefer to write this logic in bash or golang?

…cker Packer file provisioner cannot upload directories with nested subdirectories. Replace directory upload with individual file entries for all node-exporter baseline files.

Copilot

Pull request overview

This PR adds node-exporter to the VHD build process by default for Ubuntu, Mariner, and Azure Linux distributions. The node-exporter service is installed during VHD build but disabled, then enabled and started during CSE (Cluster Service Extension) provisioning. The implementation includes automatic TLS configuration using kubelet certificates, with support for both static and rotating certificates.

Changes:

Added node-exporter installation to VHD builder configurations for all supported platforms (base, arm64, mariner, flatcar, CVM variants)
Implemented installation script with package version management from components.json
Created startup script with automatic TLS configuration and certificate detection logic
Added CSE integration to enable and start node-exporter during cluster provisioning
Included VHD build tests and E2E validation for node-exporter functionality

Reviewed changes

Copilot reviewed 24 out of 88 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
vhdbuilder/packer/vhd-image-builder-*.json (8 files)	Added node-exporter file provisioning steps to all VHD builder variants
vhdbuilder/packer/install-node-exporter.sh	New installation script handling architecture detection, package download, and baseline asset deployment
vhdbuilder/packer/install-dependencies.sh	Integrated node-exporter installation into dependency installation flow
vhdbuilder/packer/imagecustomizer/azlosguard/azlosguard.yml	Added node-exporter assets to OSGuard image customizer configuration
parts/linux/cloud-init/artifacts/node-exporter/*	New node-exporter configuration files, systemd services, and startup script
parts/linux/cloud-init/artifacts/cse_*.sh	CSE integration: added configureNodeExporter function and error code
vhdbuilder/packer/test/linux-vhd-content-test.sh	Added testNodeExporter function to validate installation in VHD builds
parts/common/components.json	Added node-exporter package definitions with version information for Ubuntu, Mariner, and Azure Linux
e2e/validators.go	Added ValidateNodeExporter function for E2E testing
pkg/agent/testdata/CustomizedImage/CustomData	Updated test data with node-exporter installation

Copilot · 2026-01-27T23:56:40Z

parts/linux/cloud-init/artifacts/cse_config.sh

+configureNodeExporter() {
+    echo "Configuring Node Exporter"
+    # Check for skip file to determine if node-exporter was installed on this VHD
+    if [ ! -f /etc/node-exporter.d/skip_vhd_node_exporter ]; then
+        echo "Node Exporter assets not found on this VHD (missing /etc/node-exporter.d/skip_vhd_node_exporter); skipping configuration."
+        return 0
+    fi
+
+    if ! systemctlEnableAndStart node-exporter 30; then
+        echo "Failed to start node-exporter service"
+        return $ERR_NODE_EXPORTER_START_FAIL
+    fi
+    if ! systemctlEnableAndStart node-exporter-restart.path 30; then
+        echo "Failed to start node-exporter-restart.path"
+        return $ERR_NODE_EXPORTER_START_FAIL
+    fi
+    echo "Node Exporter started successfully"
+}


The configureNodeExporter function returns early (line 816) when the skip file is not found, but it doesn't indicate this in logs during actual failures. The function returns error code ERR_NODE_EXPORTER_START_FAIL (lines 821, 825) when services fail to start, but this will cause CSE to fail even though node-exporter is optional/non-critical.

Consider whether node-exporter failures should be fatal to CSE. If node-exporter is meant to be optional, these failures should be logged as warnings but not cause CSE to exit with an error code. If it's meant to be required, the current behavior is correct but should be clearly documented.

cse shouldn't fail if the skip file doesn't exist. This avoids a scenario with an old vhd but new cse. which i've heard is possible though awkward. If the skip files doesn't exist then aks-extension will install node-exporter

...ux/cloud-init/artifacts/node-exporter/baseline/etc/systemd/system/node-exporter-restart.path

e2e/validators.go

parts/linux/cloud-init/artifacts/node-exporter/node-exporter-startup.sh

...cloud-init/artifacts/node-exporter/baseline/etc/systemd/system/node-exporter-restart.service

parts/linux/cloud-init/artifacts/node-exporter/node-exporter-startup.sh

chmill-zz requested review from AbelHu, Devinwong, YaoC, awesomenix, cameronmeissner, djsly, ganeshkumarashok, juan-lee, junjiezhang1997, lilypan26, phealy, r2k1, timmy-wright, yewmsft and zachary-bailey as code owners January 22, 2026 00:14

chmill-zz changed the title ~~Nodeexportershift~~ feat: node-exporter into vhd build Jan 22, 2026

chmill-zz force-pushed the nodeexportershift branch from 0d52887 to 60cc001 Compare January 22, 2026 00:40

chmill-zz temporarily deployed to test January 22, 2026 00:41 — with GitHub Actions Inactive

github-actions bot added the components This pull request updates cached components on Linux or Windows VHDs label Jan 22, 2026

chmill-zz temporarily deployed to test January 22, 2026 02:09 — with GitHub Actions Inactive

chmill-zz force-pushed the nodeexportershift branch from f0f23e8 to eb8e316 Compare January 22, 2026 19:34

chmill-zz temporarily deployed to test January 22, 2026 19:34 — with GitHub Actions Inactive

chmill-zz force-pushed the nodeexportershift branch 2 times, most recently from 68a2704 to d494735 Compare January 23, 2026 21:59

chmill-zz requested a review from yagmurbaydogan as a code owner January 23, 2026 21:59

chmill-zz force-pushed the nodeexportershift branch from d494735 to 277e340 Compare January 23, 2026 22:21

chmill-zz temporarily deployed to test January 23, 2026 22:21 — with GitHub Actions Inactive

chmill-zz force-pushed the nodeexportershift branch from 277e340 to 1ae3542 Compare January 23, 2026 23:16

cameronmeissner reviewed Jan 23, 2026

View reviewed changes

...ux/cloud-init/artifacts/node-exporter/baseline/etc/systemd/system/node-exporter-restart.path Outdated Show resolved Hide resolved

cameronmeissner reviewed Jan 23, 2026

View reviewed changes

parts/linux/cloud-init/artifacts/node-exporter/node-exporter-startup.sh Outdated Show resolved Hide resolved

cameronmeissner reviewed Jan 23, 2026

View reviewed changes

parts/linux/cloud-init/artifacts/cse_helpers.sh Outdated Show resolved Hide resolved

cameronmeissner reviewed Jan 23, 2026

View reviewed changes

vhdbuilder/packer/install-dependencies.sh Outdated Show resolved Hide resolved

r2k1 reviewed Jan 24, 2026

View reviewed changes

chmill-zz force-pushed the nodeexportershift branch 8 times, most recently from 5d91c2a to 16b29eb Compare January 27, 2026 16:05

chmill-zz added 3 commits January 27, 2026 16:10

feat: install node-exporter during VHD build

c586f91

fix: use explicit file uploads for node-exporter baseline files in pa…

446079d

…cker Packer file provisioner cannot upload directories with nested subdirectories. Replace directory upload with individual file entries for all node-exporter baseline files.

chore: remove accidentally committed log file

eeef055

chmill-zz force-pushed the nodeexportershift branch from 16b29eb to eeef055 Compare January 27, 2026 16:13

chmill-zz temporarily deployed to test January 27, 2026 16:14 — with GitHub Actions Inactive

startup more better in cert checking with imds now

d208abf

Copilot AI review requested due to automatic review settings January 27, 2026 23:51

chmill-zz temporarily deployed to test January 27, 2026 23:51 — with GitHub Actions Inactive

Copilot started reviewing on behalf of chmill-zz January 27, 2026 23:51 View session

Copilot AI reviewed Jan 27, 2026

View reviewed changes

Address PR review comments for node-exporter

393c846

chmill-zz temporarily deployed to test January 28, 2026 06:28 — with GitHub Actions Inactive

		@@ -0,0 +1,5 @@
		tls_server_config:
		cert_file: "/etc/kubernetes/certs/kubeletserver.crt"

		@@ -0,0 +1,32 @@
		#!/bin/sh

		if [ "$(cat /etc/os-release \| grep ^ID= \| cut -c 4-)" = "flatcar" ]; then

feat: node-exporter into vhd build #7704

Are you sure you want to change the base?

feat: node-exporter into vhd build #7704

Uh oh!

Conversation

chmill-zz commented Jan 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cameronmeissner Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cameronmeissner Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cameronmeissner Jan 27, 2026 •

edited

Loading

cameronmeissner Jan 23, 2026 •

edited

Loading