IBM Z machines in Fedora Copr
Late in May 2021, IBM LinuxONE experts contacted our team with the intentions to have the native s390x architecture support in Fedora Copr — to be able to process some of their Copr use-cases. After the initial discussions, due to the “cloud nature” of the Copr build system (and the need for short-lived VMs), IBM Cloud was quickly chosen as the best fit. Later that year IBM folks managed to provide us an IBM Cloud account and sponsor the s390x (Z13) compute power (for all Copr users), thank you! The year 2021 was full of operational work in Copr so we managed to finalize this task in January 2022.
This post is technical, about how do we start Fedora builder machines in IBM Cloud, and some initial problems we had (that you might face, too).
Spawning IBM Cloud machines
The notation used in IBM Cloud is very similar to OpenStack or AWS. One peculiarity compared to the other clouds: The new instances don’t get a public IP address automatically. A Floating IP needs to be reserved explicitly and then assigned to the instance (fortunately this can be done via a single API call, even though not available in the Python API). The script is automatically executed by Resalloc.
The s390x architecture is only supported in
Tokyo in IBM Cloud, locations
( Tokyo, São
Paulo and London, all pretty far away from the rest of the
Copr infrastructure (hosted in North Carolina mostly), though things seem to
work pretty fine (the heavy network stuff goes through mirrored DNF
repositories). We just need to get the package sources to the builder machine,
and later download the built results (usually small stuff, but beware if you
maintain a large package(s)!).
jp-tok-3). A bit (Edit 2022-03-15)
Working with the QCOW2 images in IBM Cloud
TL;DR, cloud-init-based, even the default Fedora
s390x QCOW2 images work!
When preparing a custom image, one can spawn a fresh machine, modify it accordingly (e.g. with Ansible) and then snapshot it. This approach broke subscription-manager, see below.
But fortunately, we observed we can run our guestfish scripting on
s390x virtual machines (normally we need to run those scripts on a
bare-metal box). Feel free to take a look at our related
The generated QCOW2 image file needs to be uploaded into the S3 bucket, and then
a new IBM Cloud “image” is created from that bucket. Unfortunately, the
ibmcloud-vpc library lacks the support for this. The web UI doesn’t help much
either (even if you can afford to spend a few manual clicks), because there’s
a “200 MB” upload limit.
Therefore we use the official (not yet open)
utility. To simplify our automation, we use this image
(note that this Podman image must be built on an s390x box before it is
uploaded to quay.io!). Using that image, it is just a single command to upload
the locally generated QCOW2 file twist it into a new VM image.
Several notable things before you start.
Fedora SSO has not been implemented so far, one needs to create the user+password access assigned to a Fedora SIG’s e-mail.
Occasionally the cloud leaks some resources. We had to create two tickets so far to clean up some stuff because we couldn’t clean ourselves (4 images, and one instance).
The “OS type” category for Fedora 35 is missing. One can upload a Fedora F35 image (or even Rawhide), but the metadata will claim it is Fedora 34 at best (or you can pick RHEL). I filled a ticket for this, but it is WONTFIX (adding a new OS type is not possible until at least one of the officially supported images is also of that OS type)
If you need subscription-manager, be careful with snapshots. When a snapshot image is generated in IBM Cloud, some very weird, not-GPG-signed, katello-related package is installed automatically into the snapshot image. It breaks RHSM (at least on Fedora). This problem is still to be reported.
Uploading images is not very straight-forward on Fedora.
The s390x support in IBM Cloud, controlled via Python API (with some tweaks with
/bin/ibmcloud) seems pretty solid. Things work fine for us, (mostly) since
the beginning. We run from 6 up to 18 VMs there (s390x, z13) with 2xCPU,
4G RAM, and plenty of swap space.