Ubuntu Phased Rollout Debugging

1 Ubuntu packages hold back due unmet depencies

• Problem: Not all packages on a system do upgrade and are mentioned as hold back

root@srvtestl12p:~# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
  base-files cloud-init coreutils cryptsetup cryptsetup-bin cryptsetup-initramfs distro-info dpkg ethtool firmware-sof-signed iptables kpartx landscape-common ldap-utils libcryptsetup12 libcups2
  libip4tc2 libip6tc2 libldap-2.5-0 libldap-common libmm-glib0 libnss-systemd libpam-systemd libsystemd0 libudev1 libxtables12 modemmanager motd-news-config multipath-tools python-apt-common
  python3-apt python3-distro-info python3-distupgrade python3-gssapi python3-software-properties python3-update-manager snapd software-properties-common systemd systemd-hwe-hwdb systemd-sysv
  ubuntu-advantage-tools ubuntu-pro-client-l10n ubuntu-release-upgrader-core udev update-manager-core update-notifier-common vim vim-common vim-runtime vim-tiny xxd
0 upgraded, 0 newly installed, 0 to remove and 52 not upgraded.

root@srvtestl12p:~# apt-cache policy vim-tiny
vim-tiny:
  Installed: 2:8.2.3995-1ubuntu2.13
  Candidate: 2:8.2.3995-1ubuntu2.17
  Version table:
     2:8.2.3995-1ubuntu2.17 500 (phased 20%)
        500 katello://foreman.domain.tld/pulp/deb/ORG/lc_env_prod/ubuntu22-04/custom/Ubuntu_22_04/jammy-updates default/all amd64 Packages
     2:8.2.3995-1ubuntu2.16 500
        500 katello://foreman.domain.tld/pulp/deb/ORG/lc_env_prod/ubuntu22-04/custom/Ubuntu_22_04/jammy-security default/all amd64 Packages
 *** 2:8.2.3995-1ubuntu2.13 100
        100 /var/lib/dpkg/status
     2:8.2.3995-1ubuntu2 500
        500 katello://foreman.domain.tld/pulp/deb/ORG/lc_env_prod/ubuntu22-04/custom/Ubuntu_22_04/jammy default/all amd64 Packages

1.1 WHY

Updates are just rolled out to a subset of client machines (grouped by machine-id) to minimize impact in case of broken packages. Example, if broken package get rolled out with "phase forex: 20%" and in first week, several clients report errors, other 80% get not affected and phased rollout get stopped.

• There is a fix, see link (3) for ubuntu 24.04 and later
• https://ubuntu.com/server/docs/about-apt-upgrade-and-phased-updates
• https://ubuntu-archive-team.ubuntu.com/phased-updates.html
• https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1988819

1.2 HOW TO GET RID OF

vim /etc/apt/apt.conf.d/99no-phased-updates
------
APT::Get::Always-Include-Phased-Updates "true";
------

Problem is that if you enable that, you always get "critical/buggy" updates first. Maybe its not what you want.

1.3 OUR SOLUTION / WAY TO GO

• We probably get affected by this issue when
  • Moving repos from "public" to "Satellite/Foreman/local mirror"
  • Update systems which have big gaps in software versions (not updated for a long period)

So we decided to:

• Work around

echo 'APT::Get::Always-Include-Phased-Updates "true";' > /etc/apt/apt.conf.d/99no-phased-updates
apt-get clean all
apt-get update
apt-get dist-upgrade # must show holded packages as upgradable
apt-get clean all
rm -fv /etc/apt/apt.conf.d/99no-phased-updates
apt-get update
apt-get dist-upgrade # must show no packages

• To get informed about systems affected by this issue, we create a rundeck job, pointing to this systems

1.4 Down the rabbit hole

This Phase out information gets carried out by Packages files in Repository meta data.
See example here:

curl -s http://archive.ubuntu.com/ubuntu/dists/jammy-updates/main/binary-amd64/Packages.gz | gunzip - | grep -e ^Phased -e Package: | grep -B1 ^Phased
Package: python3-update-manager
Phased-Update-Percentage: 0
--
Package: update-manager
Phased-Update-Percentage: 0
Package: update-manager-core
Phased-Update-Percentage: 0

• https://git.launchpad.net/ubuntu-archive-tools/tree/phased-updater

This script, running on package distribution systems, is looking for bugs/heat-level and decreases the Phased-Update-Percentage if heat-level rises up. Otherwise it is incrementing the Phased-Update-Percentage step by step over time.