Building RHEL on Fedora?

Historically, this wasn’t easy. E.g. Fedora Koji—while building Fedora EPEL packages against the official Red Hat Enterprise Linux (RHEL) packages—is used to maintain their own “private” mirror of RHEL repositories. This approach though brings a great level of inconsistency between “how the official Fedora EPEL is built,” and how the contributors can actually “reproduce the build locally.” Contributors usually aren’t able to mirror the RHEL content locally, so they just build against a different (a fork) distribution (CentOS, Alma Linux, Rocky Linux, …).

Some time ago we added support for Subscription Manager to the Mock tool. But not many EPEL contributors are aware of this, and only until recently, this feature has not been used in any “production” code.

The current Copr status and configuration

The Copr builders are “cloud” machines, spawned from a VM image, kept working for a while, and then destroyed. Thus any Red Hat subscription needs to be automatically taken, and automatically destroyed:

  1. We use a very insistent snippet for taking subscriptions. Yes, from time to time the attempt to subscribe or attach a subscription fails and we can not afford such a failure because a lot of time was already spent on starting the VM, and would be completely wasted. This is also a reason why we don’t use the Ansible community general module that turned out to be extremely unreliable from this perspective.

  2. We also try our best to unregister the VM before we delete it.

  3. Because we start thousands of machines every day, we can not blindly rely on the flawlessness of step 2 (unregistering may fail, but also some small percentage of our builders die without letting us know what happened). Therefore we also install a cron job (run twice per hour) that automatically removes the leftover entitlements (automatically lists the remaining systems, and subscriptions for systems that appear to be already deleted are also deleted, using the RHSM API).

When the system is registered (see step 1. above), Mock is able to lift up the generated PEM certificates and properly work with the official RHEL repositories hosted in Red Hat’s CDN—as configured in the mock-core-configs package (see rpm -ql mock-core-configs | grep rhel, and related docs, we, in particular, rely on /etc/mock/rhel+epel-8* files for the epel-8-x86_64 chroots).

Conclusion

Copr is trying the best to be as close as possible to the end-user perspective, doing mostly what the user would do with mock locally (this way we get the reports that “something is broken” in Mock or Copr really fast). And this small RHSM step allowed us to make the EPEL builds in Koji, Copr, and local Mock builds (even though only optional) much, much close to each other. Till now it seems to be working well! Feel free to try mock -r rhel-8-x86_64 yourself.