LEAPP to Conclusions
Picking up where I left off with my last article, I have a fresh batch of RHEL 8.10 servers that began life as Oracle Linux. The next thing I wanted to accomplish was then upgrading them in place from RHEL 8 to RHEL 9, but with a catch; I again wanted to do this entirely using Satellite and no external repos. The benefits of performing the upgrade with Satellite are first; you can perform the upgrade in batches and second, it means that the upgrade can be done in a disconnected environment. Also, this allows you to peform the migration and upgrade in place, saving you the trouble of deploying new servers and "lift-n-shift-ing" (that's a verb) your workloads, which (although my crabby greybeard sysadmin instincts would tell you is the correct approach - start from known good config, don't carry old "cruft" with you through the upgrade, etc.) can definitely save a lot of time and effort. It also saves you the hassle of playing musical chairs with subscriptions - less of an implication for customers using Virtual Datacenter subscriptions, but absolutely something to consider for those who use standalone RHEL subscriptions, which would be the case for anyone running bare metal or cloud infrastructure. The upgrade is accomplished using a Red Hat utility called LEAPP. Maddeningly, LEAPP isn't actually an acronym for anything. Even more maddening, we are pretty inconsistent with referring to it as either "LEAPP" or "Leapp". To the best of my knowledge, when it was first being developed back in the RHEL 7 lifecycle it was initially an internal POC and referred to "Legacy Application" which got shortened to "LeApp" and the name just kind of stuck. The rest is just another part of Red Hat lore alongside the epic flame wars on memolist and the Unsub Squirrel.
Performing LEAPP upgrades with Satellite follows a similar workflow to my previous work with Convert2RHEL but is actually a bit more straightforward. In broad strokes, you first need an activation key with both the repos for both RHEL 8 and 9. Then you can run the three predefined tasks from the Satellite UI; Leapp - install, Leapp - Preupgrade and Leapp - Upgrade.
Leapp - Install does exactly what it sounds like. It installs the necessary packages to perform the upgrade.
Leapp - Preupgrade is a crucial part of the upgrade process and makes the process as painless as possible. The preupgrade runs a series of checks and generates a report of configuration issues on the server that will cause the upgrade to fail. Not only that, but it will even fix most of them for you.
Leapp - Upgrade is the final part of the process where you actually upgrade from RHEL 8 to RHEL 9. It is by far the most time consuming part of the process but it's really pretty straightforward. As long as you've cleared all the inhibitors identified in the preupgrade phase, it should go off without a hitch.
Create an Activation Key
The first thing we need to do is create an activation key containing both the RHEL 8 and RHEL 9 repos - at a minimum they need to have BaseOS and AppStream. You don't need to create a content view. Just like with Convert2RHEL, you will just use Library > Default Organization View.
Then subscribe your servers using the new activation key. You can do so in a batch job from the Satellite UI using remote execution to do all of them at once.
LEAPP Install
With our freshly baked batch of RHEL 8 servers registered with our LEAPP key, we can proceed to install the LEAPP packages. Schedule a remote job to run on the hosts you want to upgrade and select 'Leapp - Install'
Leapp - Preupgrade
Next we can run the LEAPP preupgrade checks. From the Satellite UI, select the hosts you want to upgrade and select 'Preupgrade check with Leapp'. Run the job and wait. It took about 10 minutes to run on my hosts, YMMV.
Once the task has completed, check the output of the 'Leapp preupgrade report' tab and you'll see a list of potential issues and inhibitors. You can select all the inhibitors and click on 'Fix Selected' to automatically remediate most, if not all of the issues identified that will prevent a successful upgrade. What this does in the background is generate an Ansible playbook that will remediate all of the things it can automatically.
Once the remediation has completed, run another Leapp preupgrade check to see if there are any remaining inhibitors. Upon running the second scan, the Leapp preupgrade check only identified one remaining inhibitor blocking the upgrade. In my case, it was sshd being configured by default to allow root login with a password. Not really sure why that's considered a "blocker" and not just against recommended best practice, but whatever...
Worth noting, the Leapp Preupgrade Check will also identify several warnings and notes, but unless they are explicitly identified as inhibitors, in my experience they can be safely ignored. That being said, if you really wanted to, you can always drill down on them and follow the links in our documentation and follow the remediation steps for all the ones that are annotated "has remediation". The rest are just informational and shouldn't affect LEAPP.
While the Leapp preupgrade tool can't automatically remediate this issue, it's easy enough to do it as a batch job with Satellite. Schedule a remote job against your servers to do execute the following
sed -i 's/PermitRootLogin yes/PermitRootLogin prohibit-password/g' /etc/ssh/sshd_config
systemctl restart sshd
Then, once that task has been completed, run one final Leapp Preupgrade Check for good measure (yes, it's probably unnecessary at this point but best to just verify. It only takes a couple minutes to run a scan.)
Finally, with no inhibitors identified by the scan, we can proceed to actually perform the upgrade from RHEL 8 to RHEL 9 in place.
LEAPP Upgrade
This is where the magic happens. You can actually just kick off the upgrade job from the Preupgrade report screen. Just click the 'Run Upgrade' button, go grab a coffee and wait for the job to complete. It's going to take a little while to finish.
Unfortunately the console logs don't provide much useful output. You'll be staring at this message for most of the upgrade process.
However, if you want to see what it's actually doing on your hosts, you can monitor processes in real time but running top on your servers while they're being upgraded.
Or if you really want to see exactly what's going on in real time, you can run tail -f /var/log/leapp/leapp-upgrade.log , but it's a little bit like watching paint dry. Seriously, go grab a coffee. Take the dog for a walk. Get some fresh air, because it's going to be a little while.
You can also watch the KVM console which provides some additional output, but again... it's about as interesting as watching grass grow.
In my case, I clocked the upgrade at a little over a half hour from start to finish. Your servers are going to reboot multiple times throughout the process, so getting an exact time for the upgrade is tough, short of using a stopwatch and staring at the screen for a half hour or more, which doesn't sound like fun to me. These are VMs running on Openshift Virt with 1CPU and 2GB RAM running on ODF storage with 1Gbe networking. I'm sure that throwing more compute and memory at these servers, or running on faster storage/network would probably speed things up a bit, but at the end of the day it's going to take about as long as it takes to install a fresh copy of RHEL, because that is in essence what you're doing.
At long last, the servers have landed on RHEL 9 but we're still not done. There are going to be a few housekeeping tasks to do after running LEAPP.
Post Upgrade
Once the upgrade has completed, there are some steps you're going to want to take. First, you're going to need to re-register your new RHEL 9 hosts using a RHEL 9 activation key.
Optionally, any settings you changed previously, eg; disabling secure boot you may want to re-enable at this time.
You're also going to want to remove the orphaned LEAPP packages from RHEL 8 since they don't exist in the RHEL 9 repos. Because they're orphaned now, you're going to need to remove them with rpm as opposed to cleanly removing them with dnf. This part of the process could use a little improvement, but it is what it is. You must remove the packages in the correct order of dependencies, eg;
rpm -e leapp-upgrade-el8toel9-0.22.0-1.el8_10.noarch
rpm -e leapp-repository-deps-el9-5.0.9-100.202502140957Z.51872205.main.el9.noarch
rpm -e leapp-0.19.0-1.el8_10.noarch
rpm -e python3-leapp-0.19.0-1.el8_10.noarch
rpm -e leapp-deps-el9-5.0.9-100.202502140957Z.51872205.main.el9.noarch
Et Voila! Finally, we have 3 full fledged RHEL 9 servers ready to go.
Wrapping Up
In my 2 most recent blog posts, I've covered how to leverage Satellite to convert Oracle Linux to RHEL and then subsequently how to upgrade from RHEL 8 to RHEL 9 in place. Leveraging Satellite to perform the migration and upgrades allows you to both accomplish these tasks at scale, but also within a disconnected environment, which may be a very attractive option for customers with strict security requirements (I'm thinking government/defense use cases, or anyone with sensitive air-gapped development environments).