RHEL Image Mode, Satellite and Openshift Virtualization

There is some very cool innovation taking place across the Red Hat portfolio. One of the new features that was announced with the release of RHEL 10 is "image mode", aka; bootc. In essence, image mode allows you to create server images the way you would create a container image. This makes your server images smaller and faster, and benefit from the immutable nature of containers offering better security. Not only that, but patching becomes a thing of the past. Updating your server is simply a matter of changing your container image tag, which also allows for instant rollback in case something breaks. No more having to restore from backups or manage VM snapshots. Just change an image tag and restart the server and you're back in business. This also allows customers to adopt a unified devops approach to managing both their container workloads and servers, reducing toil and administrative burden.

With the release of Satellite 6.19, one new feature that has been added is the ability to leverage Openshift Virtualization as a compute resource, allowing you to finally manage your virtual machines running on Openshift from the same web console you manage the rest of your servers.

It's great to see the move toward deeper integration across our flagship products; RHEL, Satellite and Openshift. The configuration though does take some work - something that I bet Ansible could help with automating, bringing in the last major player in our portfolio, AAP. One big cohesive family of products that work seamlessly together.

Openshift Compute Resource on Satellite

The process of setting up Openshift Virtualization as a compute resource in Satellite closely follows the process for setting up virt-who with kubevirt which I previously blogged about.

Prerequisites:

• Satellite 6.19 server
• Openshift Virtualization (only tested on 4.22 but previous versions may work)
• A RHEL 10 workstation with image-builder and podman - registered to your Satellite server
• Your Red Hat Service Account token (for podman auth to registry.redhat.io)

Log in as a cluster admin on your openshift cluster and create a satellite service account with a non-expiring authentication token. Keep in mind, VMs on Openshift are namespaced, so the service account must reside in the same namespace as your VMs (or you'd need to set up multiple compute resources per namespace)

oc create serviceaccount satellite-sa -n virtualization
oc adm policy add-cluster-role-to-user kubevirt.io:admin system:serviceaccount:default:satellite-sa -n virtualization
oc adm policy add-cluster-role-to-user cluster-admin -z satellite-sa -n virtualization

then create the non-expiring token

cat << EOF > satellite-sa-token.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: satellite-sa-token
  Namespace: virtualization
  annotations:
    kubernetes.io/service-account.name: satellite-sa
type: kubernetes.io/service-account-token
EOF
oc apply -f satellite-sa-token.yaml

and finally, retrieve your new token

oc get secret satellite-sa-token -n virtualization -oyaml | grep token: | awk -F ":" '{print $NF}' | base64 -d

Configure Satellite

the kubevirt plugin is not enabled by default on Satellite, so you'll need to install it. SSH into the satellite server

satellite-installer --enable-foreman-plugin-kubevirt

Then configure compute resource as follows

You need to copy your non-expiring service account token you created in the previous step, and you'll need your Openshift API certificate chain. Then click submit.

You should now be able to go to Virtual Machines tab under your new compute resource and see all the VMs running in your namespace

RHEL Image Mode

Now I'm going to show you how to get started with bootc, aka; image mode. Log in to your RHEL 10 workstation and make sure you have podman and image-builder installed

dnf -y install podman image-builder

This is only intended as a quick introduction into how to use image mode, and like containers, there is virtually no limit to what you can do in your Dockerfile, but at a minimum you're going to need to have at least one way to log in to your servers once deployed. Also keep in mind; like containers, bootc images are immutable and ephemeral. In fact, the entire filesystem with the exception of /var and /etc is read only. In short, if you didn't bake it in when you built the image, you can't change or add it after the fact, which has it's up sides and downsides. From a security and compliance perspective, that is amazing! However, I wouldn't try to run a workstation using image mode. Not unless you wanted to be rebuilding every time you need to make a change.

Let's start by creating a root password hash so you'll be able to at least log in to the console

openssl passwd -6 'MYSUPERSECRETPASSWORD' > rootpwdhash

Then create your Dockerfile. Here's my very simple example.

FROM registry.redhat.io/rhel10/rhel-bootc:10.2

# Set root password
RUN usermod --password '$6$K51fGlTvUK2aKCFs$VqheKYSd5KZwttNb2uJI...' root

# Enable SSH root login
RUN echo "PermitRootLogin yes" > /etc/ssh/sshd_config.d/01-permit-root.conf

# Standard DNF install (Podman automatically injects host RHSM certs)
RUN dnf -y install     httpd     firewalld     openssh-server     cloud-init     NetworkManager     systemd     && dnf clean all

# Enable services
RUN systemctl enable sshd NetworkManager cloud-init httpd firewalld

# Create landing page
RUN echo "<h1>Hello from RHEL 10.2 bootc!</h1>" > /var/www/html/index.html

# Configure firewall offline
RUN firewall-offline-cmd --zone=public --add-service=http     && firewall-offline-cmd --zone=public --add-service=https

As you can see, my Dockerfile does the following:

• sets the root password
• enables root login
• installs packages
• configures the web server
• configures firewalld

Now, before we can actually build this bootc image, we need to get our Red Hat Registry Service Account token for podman to authenticate to registry.redhat.io

open a web browser and navigate to https://access.redhat.com/terms-based-registry/

Find (or create) your service account and grab your token

Then once you have your token, authenticate with podman

podman login -u='11009103|rcuda' -p=eyJhbGciOiJSUzUxMiJ9.e...InzqC744L7l8Y9zSk2YRM3zNZ5C1bM registry.redhat.io

Then you can proceed to build your first bootc image!

podman build -t gitea.lab.cudanet.org/cudanet/imagemode/rhel-10_2:0.1 .

Now here's the first place where things get a little "choose-your-own-adventure". SSL is important here, and whatever registry you choose to push your bootc image to must be trusted by your Openshift cluster. By default, Satellite ships it's own self signed certs, and yes - Satellite does have it's own registry built in. So you have a couple choices here:

1. add your Satellite root CA to the Openshift
2. sign both your Openshift cluster and Satellite server with your own CA
3. use a different registry

Since I already have Gitea set up with LetsEncrypt certs and don't currently have an internal CA that I use, Gitea was the path of least resistance in my case.

Once you've decided where you're going to host this image, push it to your registry of choice

podman login gitea.lab.cudanet.org
podman push gitea.lab.cudanet.org/cudanet/imagemode/rhel-10_2:0.1 .

Congratulations, you've just built your first bootc image.

Great! ...how do I actually use it?

This is the part where things get even more complicated. Ideally, you would deploy a VM and declaratively specify your image tag as a containerdisk in YAML, eg;

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: rhel10-bootc-vm
  namespace: virtualization
spec:
  running: true
  template:
    metadata:
      labels:
        kubevirt.io/vm: rhel10-bootc-vm
    spec:
      domain:
        cpu:
          cores: 1
          sockets: 1
          threads: 1
        resources:
          requests:
            memory: 2Gi
        devices:
          disks:
            - name: containerdisk
              disk:
                bus: virtio
          interfaces:
            - name: nic0
              bridge: {}
      networks:
        - name: nic0
          multus:
            networkName: virtualization/machine-net
      volumes:
        - name: containerdisk
          containerDisk:
            image: gitea.lab.cudanet.org/cudanet/imagemode/rhel-10_2:0.1

But in my testing, this didn't work as expected because at this point, it's technically not a containerdisk. I kept running into this error:

E0912 00:48:02.533914 1 transport.go:248] Failed to find VM disk image file in the container image
E0912 00:48:02.533985 1 data-processor.go:256] Failed to find VM disk image file in the container image

This is because there is no disk image, it's just a raw ostree filesystem and it is expecting to find a disk - so you'd need to create an actual containerdisk, which is another couple of steps, and yet another image to upload.

image-builder build qcow2 --bootc-ref gitea.lab.cudanet.org/cudanet/imagemode/rhel-10_2:0.1

then create a new Dockerfile for the containerdisk

FROM scratch
ADD --chown=107:107 bootc-rhel-10.2-qcow2-x86_64.qcow2 /disk/disk.qcow2

and then build and push the containerdisk to your registry

podman build -t gitea.lab.cudanet.org/cudanet/imagemode/rhel-10_2-containerdisk:0.1 .
podman push gitea.lab.cudanet.org/cudanet/imagemode/rhel-10_2-containerdisk:0.1

Then just update the image tag on the VM to the new containerdisk you just uploaded, eg;

gitea.lab.cudanet.org/cudanet/imagemode/rhel-10_2-containerdisk:0.1

and boom! Image Mode VM running on OCP!

But wait, we're not done yet! I wanted to fully integrate this with Satellite, allowing you to provision and manage these VMs from the Satellite webUI, which... it turns out took a lot more work.

The first hurdle you're going to run into is the fact that Satellite has no way of specifying a container disk image. The only way (that I've found thus far) to provision a VM using Satellite is to configure your compute resource to point to a datasource that exists in the same namespace. A datasource can be either a PVC or a snapshot (like the default template images on OCP virt if you're using a compatible CSI driver like ODF).

Upload your disk image to Openshift

virtctl image-upload dv rhel10-bootc-dv --namespace=virtualization --size=30Gi --storage-class=ocs-storagecluster-ceph-rbd-virtualization --image-path=/root/imagemode/bootc-rhel-10.2-qcow2-x86_64/bootc-rhel-10.2-qcow2-x86_64.qcow2 --access-mode=ReadWriteMany --insecure

Then once the PVC has been created, create the corresponding datasource

cat <<EOF> datasource.yaml
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataSource
metadata:
  name: custom-rhel10-bootc
  namespace: virtualization
spec:
  source:
    pvc:
      name: rhel10-bootc-dv
      namespace: virtualization
EOF
oc apply -f datasource.yaml

Then with our datasource created, in Satellite, navigate to your Compute Resource > Images and configure your image to point to the name of the datasource you just created (rhel10-bootc-dv)

Then you need to configure one or more compute profiles to use the image

Make sure to configure the following settings: Image, Network, Storage Class - and make sure to check the bootable radio button. Otherwise your VM images will not deploy because at least one device must be marked bootable in Openshift.

Then Finally, we can proceed to deploy a new VM. Go to Hosts > Create Host and configure the first tab (host) appropriately, making sure to select Openshift Virtualization under 'Deploy On'.

On the Virtual Machine tab, here are a couple gotchas. First, make sure to unselect 'Power ON this machine' because it will simply fail to deploy since kubernetes is declarative and you can't power on a VM that is still being provisioned.

Second, you MUST set the disk size to something larger than the size of the PVC you uploaded or the CDI importer will fail provisioning the disk and you'll have to start over.

Under the 'Operating System' tab, make sure to set Provisioning Method to 'Image Based' and select your image from the drop down

Then under the Network tab you'll have to set the device name, which will be enp1s0 for a standard VM on Openshift, you'll also have to set the DNS name and domain name.

Finally, click 'Submit' and you're new VM should provision on Openshift in a few seconds

All in all, I'm not super jazzed about the current state of integration between Openshift and Satellite, but consider this feature is pretty much brand new, it's only going to improve from here. Ideally, I'd like to be able to point the disk image at a container registry, rather than an existing PVC and be able to manage upgrades/rollbacks as intended, but for demonstration purposes, there you have it. RHEL image mode, Satellite and Openshift all working together. Now if you'll excuse me, I have some Ansible playbooks to write to make this process less painful (and now we've come full circle 😂)