So I have an EC2 instance running RHEL 8.9 which was recently upgraded from RHEL 7.9. Now, I want to upgrade it all the way to RHEL 9. Here is the worklog which recording the steps I performed to achieve that goal.

Step 0, perform a snapshot of the root volume of the instance, or create an AMI from it. In my case, this is just a test instance, so I skip taking the snapshot.

Let’s verify the current running RHEL on the instance:

[root@ip-172-31-60-71 ~]# cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.9 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.9"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.9 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.9
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.9"

Next, using the official guide from Red Hat [1], I proceed to install the Leapp RHUI package.

[root@ip-172-31-60-71 ~]# dnf config-manager --set-enabled rhui-client-config-server-8
[root@ip-172-31-60-71 ~]# dnf -y install leapp-rhui-aws
RHUI Client Configuration Server 8
                  42 kB/s | 2.0 kB     00:00
Error:
 Problem: package leapp-rhui-aws-1.0.7-1.el8.noarch from rhui-client-config-server-8 requires leapp-upgrade-el8toel9, but none of the providers can be installed
  - package leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch from @System conflicts with leapp-upgrade-el8toel9 provided by leapp-upgrade-el8toel9-0.16.0-6.el8_6.noarch from rhel-8-appstream-rhui-rpms
  - package leapp-upgrade-el8toel9-0.16.0-6.el8_6.noarch from rhel-8-appstream-rhui-rpms conflicts with leapp-upgrade-el7toel8 provided by leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch from @System
  - package leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch from @System conflicts with leapp-upgrade-el8toel9 provided by leapp-upgrade-el8toel9-0.17.0-3.el8.noarch from rhel-8-appstream-rhui-rpms
  - package leapp-upgrade-el8toel9-0.17.0-3.el8.noarch from rhel-8-appstream-rhui-rpms conflicts with leapp-upgrade-el7toel8 provided by leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch from @System
  - package leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch from @System conflicts with leapp-upgrade-el8toel9 provided by leapp-upgrade-el8toel9-0.17.0-1.el8_6.2.noarch from rhel-8-appstream-rhui-rpms
  - package leapp-upgrade-el8toel9-0.17.0-1.el8_6.2.noarch from rhel-8-appstream-rhui-rpms conflicts with leapp-upgrade-el7toel8 provided by leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch from @System
  - package leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch from @System conflicts with leapp-upgrade-el8toel9 provided by leapp-upgrade-el8toel9-0.18.0-1.el8.noarch from rhel-8-appstream-rhui-rpms
  - package leapp-upgrade-el8toel9-0.18.0-1.el8.noarch from rhel-8-appstream-rhui-rpms conflicts with leapp-upgrade-el7toel8 provided by leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch from @System
  - package leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch from @System conflicts with leapp-upgrade-el8toel9 provided by leapp-upgrade-el8toel9-0.18.0-1.el8_8.2.noarch from rhel-8-appstream-rhui-rpms
  - package leapp-upgrade-el8toel9-0.18.0-1.el8_8.2.noarch from rhel-8-appstream-rhui-rpms conflicts with leapp-upgrade-el7toel8 provided by leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch from @System
  - package leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch from @System conflicts with leapp-upgrade-el8toel9 provided by leapp-upgrade-el8toel9-0.19.0-1.el8.noarch from rhel-8-appstream-rhui-rpms
  - package leapp-upgrade-el8toel9-0.19.0-1.el8.noarch from rhel-8-appstream-rhui-rpms conflicts with leapp-upgrade-el7toel8 provided by leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch from @System
  - package leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch from @System conflicts with leapp-upgrade-el8toel9 provided by leapp-upgrade-el8toel9-0.19.0-4.el8_9.noarch from rhel-8-appstream-rhui-rpms
  - package leapp-upgrade-el8toel9-0.19.0-4.el8_9.noarch from rhel-8-appstream-rhui-rpms conflicts with leapp-upgrade-el7toel8 provided by leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch from @System
  - cannot install the best candidate for the job
  - problem with installed package leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

And of course, there were some errors. Which is kinda expected, otherwise, I would not even create this post. ;)

Well, remember, as stated in the beginning this RHEL 8 instance was an upgrade from RHEL 7.9 where I had to use the Leapp utility as well. Now, those previous installed Leap related packages are still lingering in the system.

So, let’s try to uninstall the leapp-upgrade-el7toel8 package.

[root@ip-172-31-60-71 ~]# yum remove leapp-upgrade-el7toel8
All matches were filtered out by exclude filtering for argument: leapp-upgrade-el7toel8
No packages marked for removal.
Dependencies resolved.
Nothing to do.
Complete!
[root@ip-172-31-60-71 ~]# rpm -qi leapp-upgrade-el7toel8
Name        : leapp-upgrade-el7toel8
Version     : 0.19.0
Release     : 1.el7_9
Architecture: noarch
Install Date: Wed 22 Nov 2023 02:31:13 AM UTC
Group       : Unspecified
Size        : 9754684
License     : ASL 2.0
Signature   : RSA/SHA256, Thu 24 Aug 2023 09:57:30 AM UTC, Key ID 199e2f91fd431d51
Source RPM  : leapp-repository-0.19.0-1.el7_9.src.rpm
Build Date  : Wed 23 Aug 2023 06:41:08 PM UTC
Build Host  : x86-vm-43.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : https://oamg.github.io/leapp/
Summary     : Leapp repositories for the in-place upgrade
Description :
Leapp repositories for the in-place upgrade to the next major version
of the Red Hat Enterprise Linux system.

Sure, it would not work. I did notice a hint about being exclude. Let’s keep digging, and I found that the package was being excluded from 2 files yum.conf and dnf.conf.

[root@ip-172-31-60-71 ~]# grep -n leapp-upgrade-el7toel8 /etc/yum.conf
7:exclude=python2-leapp,snactor,leapp-upgrade-el7toel8,leapp

[root@ip-172-31-60-71 ~]# sed -i '7d' /etc/yum.conf
[root@ip-172-31-60-71 ~]# grep -n leapp-upgrade-el7toel8 /etc/yum.conf

[root@ip-172-31-60-71 ~]# grep leapp /etc/dnf/dnf.conf  -n
7:exclude=python2-leapp,snactor,leapp-upgrade-el7toel8,leapp

[root@ip-172-31-60-71 ~]# sed -i '7d' /etc/dnf/dnf.conf
[root@ip-172-31-60-71 ~]# grep leapp /etc/dnf/dnf.conf  -n

Let’s try removing the (old) leap-upgrade-el7toel8 package again:

[root@ip-172-31-60-71 ~]# yum remove leapp-upgrade-el7toel8
Dependencies resolved.
==============================================================================================================================================================================
 Package                                            Architecture                       Version                                      Repository                           Size
==============================================================================================================================================================================
Removing:
 leapp-upgrade-el7toel8                             noarch                             0.19.0-1.el7_9                               @System                             9.3 M
Removing dependent packages:
 leapp                                              noarch                             0.16.0-1.el7_9                               @System                              62 k

Transaction Summary
==============================================================================================================================================================================
Remove  2 Packages

Freed space: 9.4 M
Is this ok [y/N]: y
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                      1/1
  Erasing          : leapp-0.16.0-1.el7_9.noarch                                                                                                                          1/2
  Erasing          : leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch                                                                                                         2/2
  Running scriptlet: leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch                                                                                                         2/2
  Verifying        : leapp-0.16.0-1.el7_9.noarch                                                                                                                          1/2
  Verifying        : leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch                                                                                                         2/2
Installed products updated.

Removed:
  leapp-0.16.0-1.el7_9.noarch                                                   leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch

Complete!

Great. Now I should be able to install the new leapp-rhui-aws package, right?

[root@ip-172-31-60-71 ~]# dnf -y install leapp-rhui-aws
Last metadata expiration check: 0:02:24 ago on Wed 22 Nov 2023 06:17:49 AM UTC.
Error:
 Problem: package leapp-rhui-aws-1.0.7-1.el8.noarch from rhui-client-config-server-8 requires leapp, but none of the providers can be installed
  - package leapp-0.14.0-1.el8_6.noarch from rhel-8-appstream-rhui-rpms requires python3-leapp = 0.14.0-1.el8_6, but none of the providers can be installed
  - package leapp-0.15.0-2.el8.noarch from rhel-8-appstream-rhui-rpms requires python3-leapp = 0.15.0-2.el8, but none of the providers can be installed
  - package leapp-0.15.0-1.el8_6.1.noarch from rhel-8-appstream-rhui-rpms requires python3-leapp = 0.15.0-1.el8_6.1, but none of the providers can be installed
  - package leapp-0.15.1-1.el8.noarch from rhel-8-appstream-rhui-rpms requires python3-leapp = 0.15.1-1.el8, but none of the providers can be installed
  - package leapp-0.16.0-2.el8.noarch from rhel-8-appstream-rhui-rpms requires python3-leapp = 0.16.0-2.el8, but none of the providers can be installed
  - package python2-leapp-0.16.0-1.el7_9.noarch from @System conflicts with python3-leapp provided by python3-leapp-0.14.0-1.el8_6.noarch from rhel-8-appstream-rhui-rpms
  - package python3-leapp-0.14.0-1.el8_6.noarch from rhel-8-appstream-rhui-rpms conflicts with python2-leapp provided by python2-leapp-0.16.0-1.el7_9.noarch from @System
  - package python2-leapp-0.16.0-1.el7_9.noarch from @System conflicts with python3-leapp provided by python3-leapp-0.15.0-2.el8.noarch from rhel-8-appstream-rhui-rpms
  - package python3-leapp-0.15.0-2.el8.noarch from rhel-8-appstream-rhui-rpms conflicts with python2-leapp provided by python2-leapp-0.16.0-1.el7_9.noarch from @System
  - package python2-leapp-0.16.0-1.el7_9.noarch from @System conflicts with python3-leapp provided by python3-leapp-0.15.0-1.el8_6.1.noarch from rhel-8-appstream-rhui-rpms
  - package python3-leapp-0.15.0-1.el8_6.1.noarch from rhel-8-appstream-rhui-rpms conflicts with python2-leapp provided by python2-leapp-0.16.0-1.el7_9.noarch from @System
  - package python2-leapp-0.16.0-1.el7_9.noarch from @System conflicts with python3-leapp provided by python3-leapp-0.15.1-1.el8.noarch from rhel-8-appstream-rhui-rpms
  - package python3-leapp-0.15.1-1.el8.noarch from rhel-8-appstream-rhui-rpms conflicts with python2-leapp provided by python2-leapp-0.16.0-1.el7_9.noarch from @System
  - package python2-leapp-0.16.0-1.el7_9.noarch from @System conflicts with python3-leapp provided by python3-leapp-0.16.0-2.el8.noarch from rhel-8-appstream-rhui-rpms
  - package python3-leapp-0.16.0-2.el8.noarch from rhel-8-appstream-rhui-rpms conflicts with python2-leapp provided by python2-leapp-0.16.0-1.el7_9.noarch from @System
  - cannot install the best candidate for the job
  - problem with installed package python2-leapp-0.16.0-1.el7_9.noarch
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
[root@ip-172-

Nope, but there is enough hint why the installation could not proceed. So, let’s fix that.

[root@ip-172-31-60-71 ~]# rpm -q python2-leapp-0.16.0-1.el7_9.noarch
python2-leapp-0.16.0-1.el7_9.noarch
[root@ip-172-31-60-71 ~]# d

[root@ip-172-31-60-71 ~]# dnf remove python2-leapp-0.16.0-1.el7_9.noarch
Dependencies resolved.
[root@ip-172-31-60-71 ~]# dnf -y install leapp-rhui-aws
Last metadata expiration check: 0:05:05 ago on Wed 22 Nov 2023 06:17:49 AM UTC.
Dependencies resolved.
==============================================================================================================================================================================
 Package                                       Architecture                  Version                                 Repository                                          Size
==============================================================================================================================================================================
Installing:
 leapp-rhui-aws                                noarch                        1.0.7-1.el8                             rhui-client-config-server-8                         25 k
Installing dependencies:
 leapp                                         noarch                        0.16.0-2.el8                            rhel-8-appstream-rhui-rpms                          33 k
 leapp-upgrade-el8toel9                        noarch                        0.19.0-4.el8_9                          rhel-8-appstream-rhui-rpms                         1.0 M
 python3-leapp                                 noarch                        0.16.0-2.el8                            rhel-8-appstream-rhui-rpms                         182 k

Transaction Summary
==============================================================================================================================================================================
Install  4 Packages

Total download size: 1.3 M
Installed size: 9.3 M
Downloading Packages:
(1/4): python3-leapp-0.16.0-2.el8.noarch.rpm                                                                                                  2.6 MB/s | 182 kB     00:00
(2/4): leapp-0.16.0-2.el8.noarch.rpm                                                                                                          448 kB/s |  33 kB     00:00
(3/4): leapp-upgrade-el8toel9-0.19.0-4.el8_9.noarch.rpm                                                                                        10 MB/s | 1.0 MB     00:00
(4/4): leapp-rhui-aws-1.0.7-1.el8.noarch.rpm                                                                                                  528 kB/s |  25 kB     00:00
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                         7.8 MB/s | 1.3 MB     00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                      1/1
  Installing       : python3-leapp-0.16.0-2.el8.noarch                                                                                                                    1/4
  Installing       : leapp-0.16.0-2.el8.noarch                                                                                                                            2/4
  Installing       : leapp-upgrade-el8toel9-0.19.0-4.el8_9.noarch                                                                                                         3/4
  Installing       : leapp-rhui-aws-1.0.7-1.el8.noarch                                                                                                                    4/4
  Running scriptlet: leapp-rhui-aws-1.0.7-1.el8.noarch                                                                                                                    4/4
  Verifying        : leapp-0.16.0-2.el8.noarch                                                                                                                            1/4
  Verifying        : python3-leapp-0.16.0-2.el8.noarch                                                                                                                    2/4
  Verifying        : leapp-upgrade-el8toel9-0.19.0-4.el8_9.noarch                                                                                                         3/4
  Verifying        : leapp-rhui-aws-1.0.7-1.el8.noarch                                                                                                                    4/4
Installed products updated.

Installed:
  leapp-0.16.0-2.el8.noarch         leapp-rhui-aws-1.0.7-1.el8.noarch         leapp-upgrade-el8toel9-0.19.0-4.el8_9.noarch         python3-leapp-0.16.0-2.el8.noarch

Complete!

Next step is to run the leapp preupgrade command to check for issues before the actual upgrade. Note this is a EC2 instance on AWS, and RHEL uses RHUI for the repositories, it does not use the Red Hat Subscription, therefore the option --no-rhsm is passed to the leapp command.

[root@ip-172-31-60-71 ~]# leapp preupgrade --no-rhsm
...snip...
Downloading Packages:
Check completed.
==> Processing phase `Reports`
====> * verify_check_results
        Check all dialogs and notify that user needs to make some choices.
====> * verify_check_results
        Check all generated results messages and notify user about them.

Debug output written to /var/log/leapp/leapp-preupgrade.log

============================================================
                      REPORT OVERVIEW
============================================================

HIGH and MEDIUM severity reports:
    1. Packages not signed by Red Hat found on the system
    2. GRUB2 core will be automatically updated during the upgrade
    3. Usage of deprecated Model "RequiredTargetUserspacePackages" at /etc/leapp/repos.d/system_upgrade/common/actors/cloud/checkrhui/actor.py:123

Reports summary:
    Errors:                      0
    Inhibitors:                  0
    HIGH severity reports:       3
    MEDIUM severity reports:     0
    LOW severity reports:        1
    INFO severity reports:       4

Before continuing consult the full report:
    A report has been generated at /var/log/leapp/leapp-report.json
    A report has been generated at /var/log/leapp/leapp-report.txt

============================================================
                   END OF REPORT OVERVIEW
============================================================

Answerfile has been generated at /var/log/leapp/answerfile

It looks like there is no issue. So I can proceed with the actual upgrade of the OS.

[root@ip-172-31-60-71 ~]# leapp upgrade --no-rhsm
...snip...

Running transaction test
Transaction test succeeded.
Complete!
====> * add_upgrade_boot_entry
        Add new boot entry for Leapp provided initramfs.
====> * remove_upgrade_artifacts
        Removes artifacts left over by previous leapp runs
A reboot is required to continue. Please reboot your system.


Debug output written to /var/log/leapp/leapp-upgrade.log

============================================================
                      REPORT OVERVIEW
============================================================

HIGH and MEDIUM severity reports:
    1. Packages not signed by Red Hat found on the system
    2. GRUB2 core will be automatically updated during the upgrade
    3. Usage of deprecated Model "RequiredTargetUserspacePackages" at /etc/leapp/repos.d/system_upgrade/common/actors/cloud/checkrhui/actor.py:123

Reports summary:
    Errors:                      0
    Inhibitors:                  0
    HIGH severity reports:       3
    MEDIUM severity reports:     0
    LOW severity reports:        1
    INFO severity reports:       4

Before continuing consult the full report:
    A report has been generated at /var/log/leapp/leapp-report.json
    A report has been generated at /var/log/leapp/leapp-report.txt

============================================================
                   END OF REPORT OVERVIEW
============================================================

Answerfile has been generated at /var/log/leapp/answerfile

It says “a reboot is required to continue.” Let’s do that. After rebooting the instance and wait for about 20 minutes, the instance was automatically restarted, and voilla, it’s now running RHEL 9.3.

$ ssh -i ./keys/mykey.pem ec2-user@1.2.3.4
Register this system with Red Hat Insights: insights-client --register
Create an account or view all your systems at https://red.ht/insights-dashboard
Last login: Wed Nov 22 22:20:22 2023 from 2.3.4.5
[ec2-user@ip-172-31-60-71 ~]$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="9.3 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.3 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.3"

One thing worth notice is that some packages related to leapp utility are stilled installed and mark as excluding in the system.

[root@ip-172-31-60-71 ~]# grep leapp /etc/{dnf/dnf.conf,yum.conf}  -n
/etc/dnf/dnf.conf:7:exclude=leapp,leapp-upgrade-el8toel9,python3-leapp,snactor
/etc/yum.conf:7:exclude=leapp,leapp-upgrade-el8toel9,python3-leapp,snactor

To remove all those Leapp packages, follow the steps from [2]. Here is how to do it:

[root@ip-172-31-60-71 ~]# dnf config-manager --save --setopt exclude=''
[root@ip-172-31-60-71 ~]# grep leapp /etc/{dnf/dnf.conf,yum.conf}  -n

[root@ip-172-31-60-71 ~]# dnf remove leapp-deps-el9 leapp-repository-deps-el9
Dependencies resolved.
==============================================================================================================================================================================
 Package                                  Architecture          Version                                                     Repository                                   Size
==============================================================================================================================================================================
Removing:
 leapp-deps-el9                           noarch                5.0.9-100.202301271308Z.f7c82a24.master.el9                 @System                                       0
 leapp-repository-deps-el9                noarch                5.0.9-100.202301271308Z.f7c82a24.master.el9                 @System                                       0
Removing dependent packages:
 leapp-rhui-aws                           noarch                1.0.7-1.el8                                                 @rhui-client-config-server-8                 36 k
Removing unused dependencies:
 leapp                                    noarch                0.16.0-2.el8                                                @rhel-8-appstream-rhui-rpms                  61 k
 leapp-upgrade-el8toel9                   noarch                0.19.0-4.el8_9                                              @rhel-8-appstream-rhui-rpms                 8.6 M
 python3-leapp                            noarch                0.16.0-2.el8                                                @rhel-8-appstream-rhui-rpms                 596 k

Transaction Summary
==============================================================================================================================================================================
Remove  6 Packages

Freed space: 9.3 M

That’s it!

References: