profile::elastic::service doesn't actually disable unattended upgrades

[akuzminsky]

Problem

The current Puppet manifest in profile::elastic::service stops unattended-upgrades.service, but this only controls the shutdown hook — not the actual upgrade process. Unattended upgrades continue to run via apt-daily-upgrade.timer and the config in /etc/apt/apt.conf.d/20auto-upgrades.

service { 'unattended-upgrades.service':
  ensure => stopped,
}

Impact

On 2026-02-27, unattended-upgrade upgraded libstdc++6 and libgcc-s1 on production ES nodes, causing Elasticsearch to restart across the cluster. The timeline:

• 06:11 — ip-10-2-3-206 went down
• 06:21 — ip-10-2-3-217 bounced (removed then re-added to cluster)
• 06:24:21 — unattended-upgrade started on ip-10-2-2-111 (gcc-13 packages)
• 06:24:34-06:24:45 — GC pressure escalating (267ms → 316ms → 348ms per second)
• 06:24:45 — unattended-upgrade upgraded libstdc++6, libgcc-s1 (core shared libraries used by ES JVM/ML controller)
• 06:24:56 — Elasticsearch began stopping

All three ES nodes were affected in sequence, appearing as a rolling restart.

Evidence from apt history.log:

Start-Date: 2026-02-27  06:24:45
Commandline: /usr/bin/unattended-upgrade
Upgrade: libstdc++6:amd64, libgcc-s1:amd64, ...
End-Date: 2026-02-27  06:24:52

The apt-daily-upgrade.timer is active and ran at 06:23:48, and /etc/apt/apt.conf.d/20auto-upgrades has Unattended-Upgrade "1".

Suggested Fix

Replace the ineffective service stop with:

# Disable the actual upgrade timer
service { 'apt-daily-upgrade.timer':
  ensure => stopped,
  enable => false,
}

# Override apt config to disable unattended upgrades
file { '/etc/apt/apt.conf.d/20auto-upgrades':
  ensure  => file,
  content => "APT::Periodic::Update-Package-Lists \"1\";\nAPT::Periodic::Unattended-Upgrade \"0\";\n",
}

This keeps package list updates (so you can see what's available) but prevents automatic installation.

Affected Files

• modules/profile/manifests/elastic/service.pp
• environments/development/modules/profile/manifests/elastic/service.pp
• environments/sandbox/modules/profile/manifests/elastic/service.pp

Affected Nodes

All ES nodes (ip-10-2-2-111, ip-10-2-3-217, ip-10-2-3-206 in production).

[akuzminsky]

Update: root cause is needrestart, not unattended-upgrades directly

New incident today on a production ES data node repeats the pattern. Evidence from the node shows the restart is issued by needrestart as a post-upgrade hook, not by unattended-upgrades or apt itself.

Direct proof

From /var/log/unattended-upgrades/unattended-upgrades-dpkg.log during the incident window, after libcap2 / libcap2-bin / libpam-cap were upgraded:

Restarting services...
 /etc/needrestart/restart.d/systemd-manager
 systemctl restart chrony.service elasticsearch.service fwupd.service irqbalance.service
                   multipathd.service packagekit.service polkit.service postfix@-.service
                   rsyslog.service systemd-journald.service systemd-networkd.service
                   systemd-resolved.service systemd-udevd.service udisks2.service

systemd PID-1 log confirms the sequence: Stopping elasticsearch.service → Stopped → Starting → Started (~2 min unavailable on this node).

The elasticsearch package itself was not upgraded — last dpkg change was weeks earlier. needrestart restarts services whose mapped shared libraries have been replaced (this event: libcap2; the 2026-02-27 event: libstdc++6 / libgcc-s1).

Why the originally proposed fix works but is heavy-handed

Disabling apt-daily-upgrade.timer + APT::Periodic::Unattended-Upgrade "0" stops the whole chain, but it also stops all security patches on the node. We lose CVE coverage on kernel/libc/openssl to fix a service-restart problem.

Narrower fix — configure needrestart to leave ES/Kibana alone

Drop a file at /etc/needrestart/conf.d/99-elastic.conf:

# Do not auto-restart stateful services; operators will do it in maintenance windows.
\$nrconf{override_rc} = {
    %{\$nrconf{override_rc} // {}},
    qr(^elasticsearch\.service\$) => 0,
    qr(^kibana\.service\$) => 0,
};

This lets library upgrades install normally, so security patches still land — only the restart is suppressed. Operators then schedule a rolling restart in a maintenance window. This matches how needrestart already treats apt-daily.service, cloud-init.service, systemd-logind, etc. in the default config.

Puppet changes proposed

In modules/profile/manifests/elastic/service.pp (and the env copies already listed in the issue), replace the service { 'unattended-upgrades.service': ensure => stopped } block with:

file { '/etc/needrestart/conf.d/99-elastic.conf':
  ensure  => file,
  owner   => 'root',
  group   => 'root',
  mode    => '0644',
  content => @(EOT)
    # Managed by Puppet (profile::elastic::service).
    # Prevent needrestart from auto-restarting stateful services after library upgrades.
    \$nrconf{override_rc} = {
        %{\$nrconf{override_rc} // {}},
        qr(^elasticsearch\.service\$) => 0,
    };
    | EOT
  ,
}

And the analogous profile for Kibana should ship the same override for kibana.service.

Additional findings worth capturing

1. All data and master nodes share the same apt timer, so a cluster-wide simultaneous restart is possible if RandomizedDelaySec happens to align. The needrestart fix eliminates this risk regardless.
2. /etc/apt/apt.conf.d/50unattended-upgrades has an empty Package-Blacklist on the affected node, so anything in -security / -apps-security is fair game. Not changing this — letting patches land is the point.
3. Packages that triggered the latest event: libcap2, libcap2-bin, libpam-cap (noble security pocket).