Pavel's Space

Reinitialize rpmautospec in your package

Tue, 11 Nov 2025 00:00:00 +0000

From time to time, you might want to re-initialize rpmautospec in your RPM package’s (Dist)Git repo (rpmautospec is the tool that expands %autorelease and %autochangelog templates in specfiles).

The motivations for this step might be simple:

the tool is analyzing an excessively long Git history
you are attempting to handle shallow Git clones
you want to migrate the package from one DistGit to another (e.g., from Fedora to CentOS Stream)
you want to avoid executing Turing-Complete macros in Version or Epoch fields.

The principle

TL;DR: we need to prevent rpmautospec logic from analyzing beyond the last two Git commits. The tool behaves this way only when the very last commit (a) modifies the Version: and (b) changes the changelog file (the file with expanded %changelog).

This implies we need to make two precisely formatted commits.

Step-by-step guide

(Re)generate the changelog file (do not ‘git add’ the file for now; this change belongs in the second commit):
```
rpmautospec generate-changelog > changelog
```
TODO: if we move from rpm to norpm, we need to use the full RPM parser one last time here; document how.
Set a placeholder version, e.g., Version: 0.

Do not worry; no builds are performed for the first commit. You might want to apply other changes here as well, such as dropping %lua macros from the Epoch if present.
Remove the %autochangelog and %autorelease templates from the spec file. To keep things simple, you can just set Release: 0.
Make the first commit (spec file changes only, keep the changelog file changes uncommitted!).
Set the Version: field appropriately (think of Turing-Complete macros).
Reintroduce the %autochangelog and %autorelease templates.
Make the second commit. Include the changelog change. The Version reset we did means that the calculated %autorelease is re-started from Release: 1—if you need a different value, say 5, add the string [bump release: 5] into the commit message.
Optional step: You might want to add one more empty commit using git commit --allow-empty to bump the Release and generate %changelog entry. (This depends on your distribution’s policy.)

You are done! See also an example merge-request.

Double-check

Check that rpmautospec works by:

$ rpmautospec --debug process-distgit *spec *spec
===========================================================
Extracting linear history snippets from branched history...
===========================================================
commit 124279c: reenable autospec with Lua-free Version
Keep processing: commit 124279cc62be84852bb702f49fca5aaae0d93f6c
  epoch_version: 25.04.0.0.84
  child must continue: False
  changelog changed: True
  child changelog removed: None
  our changelog removed: False
  merge unresolvable: False
  spec file present: True
  child must continue (incoming): True
  child must continue (outgoing): False
  parent to follow: da9b55c
commit da9b55c: Disable rpmautospec for one commit
children_visitors_info[]['child_must_continue']: [False, False]
Only traversing: commit da9b55c3a6175643788eed8cbcf5475a574cf333
...

The second commit, da9b55c, was analyzed, resulting in False, False. This indicates there’s no need to analyze any older variants of the spec file.

$ git diff
+## START: Set by rpmautospec
+## (rpmautospec version 0.8.1)
+## RPMAUTOSPEC: autorelease, autochangelog
+%define autorelease(e:s:pb:n) %{?-p:0.}%{lua:
+    release_number = 5;
+    base_release_number = tonumber(rpm.expand("%{?-b*}%{!?-b:1}"));
+    print(release_number + base_release_number - 1);
+}%{?-e:.%{-e*}}%{?-s:.%{-s*}}%{!?-n:%{?dist}}
+## END: Set by rpmautospec
...
 %changelog
-%autochangelog
+## START: Generated by rpmautospec
+* Thu Jul 03 2025 Kamal Heib <kheib@redhat.com> - 25.04.0.0.84-1
+- Update to upstream 25.04.0-0.84
...

Please note the release_number = 5 line and the version-release of the last %changelog entry—that’s what we needed.

Use git checkout -p to reset the changes made to the spec file by the previous rpmautospec process-distgit call. Done.

Disk partitioning on Fedora Copr Power9 machines

Tue, 21 Oct 2025 00:00:00 +0000

Warning: Never directly copy-paste commands from this guide, always think twice the action you do to avoid DATA LOSS.

Fedora Infrastructure / Red Hat IT folks recently racked three POWER9 servers for Fedora Copr. Thank you! I’m going to quickly recap the steps I took to make them (almost) work.

This is a (somewhat easier, this time) follow-up to my previous action of that kind.

Getting Machines Installed

With Kevin (Fedora Infra), we made a few installation attempts via the kickstart he prepared.

The first attempt was with a single large RAID10. The machine was booting fine, but that layout would likely be slow for both (a) libvirt disks and (b) SWAP. We need striping for both.
Instead of doing the very slow manual re-creation work like previously, we decided to reinstall the machine with a modified kickstart—without --growfs and keeping the / partition small. I then manually created striped SWAP and RAID0 for /libvirt-images. This initially worked fine, and we even started building RPMs on those machines. However, the machine did not survive a reboot (for newly added clevis & tang). We noticed:
```
mdadm: failed to add /dev/sde5 to /dev/md/3_0: Invalid argument
```
We thought that the manually created RAIDs on Fedora 43 had broken the Petitboot bootloader. Hence…
We reinstalled for the 3rd time, with all the RAIDs created by Anaconda kickstart (that attempt seemed reasonable, because the other MDs also created by Anaconda were just working fine). Well, even now the machine failed to boot, with the very same error.

Yeah, so neither manually created MDs nor the Anaconda installation could make the Petitboot bootloader happy with our five MDs use-case.

Debugging

When I switched to the Petitboot command-line shell and re-scanned the MDs (failing the mdadm command), things became a bit more normal. At least /boot and the root filesystem (/) on the RAID arrays started working!

Here is the full mdadm --assemble --scan error output.

mdadm: /dev/md/unused-249:1 has been started with 8 drives.
mdadm: /dev/md/unused-249:0 has been started with 8 drives.
mdadm: /dev/md/unused-249:2 has been started with 8 drives.
mdadm: failed to add /dev/sdb3 to /dev/md/unused-249:3: Invalid argument
mdadm: failed to add /dev/sdc3 to /dev/md/unused-249:3: Invalid argument
mdadm: failed to add /dev/sdd3 to /dev/md/unused-249:3: Invalid argument
mdadm: failed to add /dev/sde3 to /dev/md/unused-249:3: Invalid argument
mdadm: failed to add /dev/sdf3 to /dev/md/unused-249:3: Invalid argument
mdadm: failed to add /dev/sdg3 to /dev/md/unused-249:3: Invalid argument
mdadm: failed to add /dev/sdh3 to /dev/md/unused-249:3: Invalid argument
mdadm: failed to add /dev/sda3 to /dev/md/unused-249:3: Invalid argument
mdadm: failed to RUN_ARRAY /dev/md/unused-249:3: Invalid argument
mdadm: failed to add /dev/sdb2 to /dev/md/unused-249:4: Invalid argument
mdadm: failed to add /dev/sdc2 to /dev/md/unused-249:4: Invalid argument
mdadm: failed to add /dev/sdd2 to /dev/md/unused-249:4: Invalid argument
mdadm: failed to add /dev/sde2 to /dev/md/unused-249:4: Invalid argument
mdadm: failed to add /dev/sdf2 to /dev/md/unused-249:4: Invalid argument
mdadm: failed to add /dev/sdg2 to /dev/md/unused-249:4: Invalid argument
mdadm: failed to add /dev/sdh2 to /dev/md/unused-249:4: Invalid argument
mdadm: failed to add /dev/sda2 to /dev/md/unused-249:4: Invalid argument
mdadm: failed to RUN_ARRAY /dev/md/unused-249:4: Invalid argument
mdadm: failed to add /dev/sdb3 to /dev/md/unused-249:3: Invalid argument
mdadm: failed to add /dev/sdc3 to /dev/md/unused-249:3: Invalid argument
mdadm: failed to add /dev/sdd3 to /dev/md/unused-249:3: Invalid argument
mdadm: failed to add /dev/sde3 to /dev/md/unused-249:3: Invalid argument
mdadm: failed to add /dev/sdf3 to /dev/md/unused-249:3: Invalid argument
mdadm: failed to add /dev/sdg3 to /dev/md/unused-249:3: Invalid argument
mdadm: failed to add /dev/sdh3 to /dev/md/unused-249:3: Invalid argument
mdadm: failed to add /dev/sda3 to /dev/md/unused-249:3: Invalid argument
mdadm: failed to RUN_ARRAY /dev/md/unused-249:3: Invalid argument
mdadm: failed to add /dev/sdb2 to /dev/md/unused-249:4: Invalid argument
mdadm: failed to add /dev/sdc2 to /dev/md/unused-249:4: Invalid argument
mdadm: failed to add /dev/sdd2 to /dev/md/unused-249:4: Invalid argument
mdadm: failed to add /dev/sde2 to /dev/md/unused-249:4: Invalid argument
mdadm: failed to add /dev/sdf2 to /dev/md/unused-249:4: Invalid argument
mdadm: failed to add /dev/sdg2 to /dev/md/unused-249:4: Invalid argument
mdadm: failed to add /dev/sdh2 to /dev/md/unused-249:4: Invalid argument
mdadm: failed to add /dev/sda2 to /dev/md/unused-249:4: Invalid argument
mdadm: failed to RUN_ARRAY /dev/md/unused-249:4: Invalid argument

Kevin also observed these weird disk errors:

fdisk: device has more than 2^32 sectors, can't use all of them

Anyway, after a manual post-start --assemble --scan manually, and I got the most important MDs working so the machine could boot. At least something!

Planning the fix

The error we saw seemed to be related to some 32-bit limit. What if we re-created the non-working RAIDs once more, manually, directly inside Petitboot’s shell? I believed that MDs created in Petitboot would later be readable by Petitboot..

On the booted machine:

[root@unused-249 ~]# lsblk /dev/sdh

[root@unused-249 ~]# lsblk /dev/sdh
NAME                                            MAJ:MIN RM   SIZE RO TYPE   MOUNTPOINTS
sdh                                               8:112  0   3.6T  0 disk
├─sdh1                                            8:113  0     8M  0 part
├─sdh2                                            8:114  0     3T  0 part
│ └─md4                                           9:4    0    24T  0 raid0
│   └─luks-bd506244-b18a-4ea8-9af1-15db95c63d7d 252:2    0    24T  0 crypt  /vmvolumes
├─sdh3                                            8:115  0    64G  0 part
│ └─md3                                           9:3    0 511.5G  0 raid0
│   └─luks-3d68312d-7c05-42d2-86b9-be636676d5bf 252:0    0 511.5G  0 crypt
├─sdh4                                            8:116  0    32G  0 part
│ └─md2                                           9:2    0 127.9G  0 raid10
│   └─luks-fde52936-8c89-479d-9d70-cede214f7314 252:1    0 127.9G  0 crypt  /
├─sdh5                                            8:117  0  1000M  0 part
│ └─md0                                           9:0    0   999M  0 raid1  /boot
└─sdh6                                            8:118  0   500M  0 part
  └─md1                                           9:1    0 499.9M  0 raid1  /boot/efi

Ok, we’ll recreate md3 and md4, i.e. everything on sd*2 and sd*3.

Cleanup first

Stop swap:

swapoff -a

Comment-out things from /etc/fstab, /etc/crypttab and /etc/mdadm.conf appropriately. All the stuff related to md{3,4}! Then:

umount /vmvolumes

cryptsetup close luks-bd506244-b18a-4ea8-9af1-15db95c63d7d
cryptsetup close luks-3d68312d-7c05-42d2-86b9-be636676d5bf

mdadm --stop md3
mdadm --stop md4

mdadm --zero-superblock /dev/sd*2
mdadm --zero-superblock /dev/sd*3

Reboot to Petitboot!

Back in Petitboot shell

# mdadm --create /dev/md3 /dev/sd*2 --level=0 --raid-devices=8
# mdadm --create /dev/md4 /dev/sd*3 --level=0 --raid-devices=8

Reboot once more, and… voilà, even though we have five MDs (boot/efi/vmvolumes/swap/root), the machine boots, and Petitboot is happy!

So let’s re-configure mdadm.conf to cover the two new md3 and md4:

[root@vmhost-p09-copr03 ~][PROD-RDU3]# cat /etc/mdadm.conf
# mdadm.conf written out by anaconda
MAILADDR root
ARRAY /dev/md0 metadata=1.2 UUID=aaaaaaaa:bbbbbbbb:cccccccc:dddddddd
ARRAY /dev/md1 metadata=1.0 UUID=bbbbbbbb:cccccccc:dddddddd:aaaaaaaa
ARRAY /dev/md2 metadata=1.2 UUID=aaaaaaaa:bbbbbbbb:cccccccc:dddddddd
ARRAY /dev/md3 metadata=1.2 UUID=bbbbbbbb:cccccccc:dddddddd:aaaaaaaa
ARRAY /dev/md4 metadata=1.2 UUID=aaaaaaaa:bbbbbbbb:cccccccc:dddddddd

Encrypt SWAP with a random key:

cryptsetup open --type plain /dev/md4 swap_crypt --key-file /dev/urandom

Setup automatic decryption of SWAP:

# echo swap_crypt   UUID=$(mdadm --detail --scan | grep md/4 | sed 's/.*UUID=//') /dev/urandom  swap,plain,cipher=aes-xts-plain64,size=256 >> /etc/crypttab

Create the SWAP:

mkswap /dev/mapper/swap_crypt
echo /dev/mapper/swap_crypt   none    swap    sw      0       0 >> /etc/fstab
swapon -a

Create auto-decryption for the /libvirt-images (note we don’t mount, that’s done by copr-hypervisor playbook).

dd if=/dev/urandom of=/root/libvirt-luks-images.key bs=4096 count=1
cryptsetup luksFormat /dev/md3 /root/libvirt-luks-images.key
cryptsetup luksOpen /dev/md3 vmimages_crypt --key-file /root/libvirt-luks-images.key
mkfs.ext4 -L vmvolumes /dev/mapper/vmimages_crypt
chmod 0600 /root/libvirt-luks-images.key
echo vmimages_crypt  UUID=$(mdadm --detail --scan | grep md/3 | sed 's/.*UUID=//') /root/libvirt-luks-images.key  luks >> /etc/crypttab

Backup old grub config! And then fix it (all MDs but efi mentioned?).

echo $(for i in $(mdadm --detail --scan | sort | grep -e md/4 -e md/3 | sed 's/.*UUID=//'); do echo rd.md.uuid=$i; done)
rd.md.uuid=48a41879:9301e678:bf629bf1:b05eb5ae rd.md.uuid=1e0bd440:a22ea797:fef650fd:5df66d17
grubby --update-kernel=ALL --args 'rd.md.uuid=663b53dc:bab1869d:84e94f2a:7fee5168 rd.luks.uuid=luks-1483fbff-398d-49bf-8a6b-e16e839c261a rd.md.uuid=06df0fa7:0a1ac337:18840fd9:96e5f9e5 net.ifnames=0 console=ttyS1,115200'

Rebuild initramfs:

dracut -f

Uh! UUIDs don’t work reliably in /etc/crypttab, but since we fixed intitramfs, it is OK to use symbolic /dev/mdX references:

vmimages_crypt /dev/md3 /root/libvirt-luks-images.key luks
swap_crypt /dev/md4 /dev/urandom swap,plain,cipher=aes-xts-plain64,size=256

Reboot, and enjoy! Now, let’s fix the ipv6 issues, but that’s a different story.

Fedora Copr builders - September 2025

Fri, 05 Sep 2025 00:00:00 +0000

It’s been more than four years since my last post like this.

A few changes have happened since then, so let me do a quick recap.

Changes in last 4 years, a.k.a. “News”

Namely, Fedora Copr now provides native build support for all supported Fedora architectures: aarch64, ppc64le, s390x, and x86_64. This means we no longer have to emulate any architecture using qemu-user-static. It’s faster, and more reliable.

Some package builds are known to be especially demanding, so we’ve created a way to request high-performance workers.

Although it’s not news (it’s over three years old), IBM has started sponsoring the s390x architecture through a community account in IBM Cloud. Since then, we’ve managed to bump and reorganize our quota a few times, and we also have high-performance s390x workers. Thank you, IBM!

A more recent development is the ppc64le high-performance profile support, thanks to OSUOSL. By the way, OSUOSL is our only source of Power10 machines right now. Thank you, OSUOSL!

Since 2022 we’ve started using Red Hat-subscribed Fedora builders to allow users to build against the official Red Hat Enterprise Linux content. Since then, we’ve booted and subscribed millions of ephemeral Fedora VMs. Thank you, Red Hat.

Mirek has become one of the main administrators for the Fedora community EC2 account. He’s also started moving our EC2 machines from spot/on-demand to reserved instances, which allows for more cost-effective planning. EC2 machines handle a large portion of x86_64 builds and all of the aarch64 builds, and EC2 also sponsors other parts of Fedora Copr infrastructure. Thank you, Amazon AWS team! Mirek is also paying close attention to accounting for resources in Fedora community account and cleaning up unused things there. Thank you, Mirek.

The Current Status

The number of machines has roughly doubled since the last time:

Thanks to a few optimizations, our throughput has more than doubled—Fedora Copr can now handle tens of thousands of Mock builds daily:

We currently have machines dedicated to the following architectures:

aarch64
- 37 EC2 reserved machines
- Up to 30 EC2 spot machines
- Up to 8 EC2 on-demand machines
s390x
- Up to 18 IBM Cloud on-demand machines (across 6 locations in São Paulo and Madrid)
- Up to 2 IBM Cloud high-performance machines
ppc64le
- Our own hardware:
  - 28 Power8 machines
  - 31 Power9 machines
- OSUOSL machines:
  - Up to 15 standard machines
  - Up to 3 high-performance machines
x86_64
- 80 of our own machines, always running in the Fedora Lab
- 46 EC2 reserved machines
- Up to 70 EC2 spot machines
- Up to 20 EC2 on-demand machines
- Up to 20 powerful EC2 machines

In total, we have 235 machines running at all times and the ability to auto-scale up to 428 machines.

Future Plans

The Power8 machines are aging, and with the help of the Fedora Infrastructure team, we plan on replacing them with newer Power9 machines.

The number of workers we have is mostly sufficient for current load. While we would like to have more s390x machines, computational power is not the main concern for the Fedora Copr team. Our primary challenge is storage (our current RAID deployment provides about 40TB). Another area of concern is the computational power required on the copr-backend to keep the RPM repository metadata up-to-date. To address these, we plan to “outsource” our storage via PULP, which should offload these things to the subject-matter-expert team—and allow us to concentrate on achieving an even higher build throughput.

Additionally, we may want to dedicate a separate VM for the Resalloc machine. The process of machine allocation is quite demanding, involving hundreds of shell and Python processes to control the machines being started.

Empower your writing with Ramalama

Mon, 23 Jun 2025 00:00:00 +0000

Have you ever asked ChatGPT or other online AI services to review and correct your emails or posts for you? Have you ever pondered over what the service or company, such as OpenAI, does with the text you provide them “for free”? What are the potential risks of sharing private content, possibly leading to copyright headaches?

I am going to demonstrate how I use Ramalama and local models instead of ChatGPT for English corrections, on a moderately powerful laptop. The latest container-based tooling surrounding the Ramalama project makes running Language Models (LLMs) on Fedora effortless and free of those risks.

Installation and starting

Ramalama is packaged for Fedora! Just go with:

dnf install -y ramalama

It is a fairly swift action, I assure you. Do not be misled though. You will need to download gigabytes of data later on, during the Ramalama runtime :-)

I take privacy seriously, the more isolation the better. Even though Ramalama claims it isolates the model I’d like to isolate Ramalama itself. Therefore, I create a new user called “ramalama” with the command:

useradd ramalama && sudo su - ramalama

This satisfies me, as I trust the Ramalama/Podman team’s that they do not escalate privileges (at least I don’t see any setuid bits, etc.).

Experimenting with models

ramalama run: error: the following arguments are required: MODEL

The tool does a lot for you, but you still need to research which model is most likely the one you require. I haven’t done an extensive research myself, but I’ve heard rumors that LLama3 or Mistral are relatively good options for English on laptops. Lemme try:

$ ramalama run llama3
Downloading ollama://llama3:latest ...
Trying to pull ollama://llama3:latest ...
 99% |█████████████████████████████████████████████████████████████████████████ |    4.33 GB/   4.34 GB 128.84 MB/s
Getting image source signatures
Copying blob 97024d81bb02 done   |_
Copying blob 97971704aef2 done   |_
Copying blob 2b2bdebbc339 done   |_
Copying config d13a3de051 done   |_
Writing manifest to image destination
🦭 > Hello, who is there?
Hello! I'm an AI, not a human, so I'm not "there" in the classical sense. I exist solely as a digital entity,
designed to understand and respond to natural language inputs. I'm here to help answer your questions, provide
information, and chat with you about various topics. What's on your mind?

Be prepared for that large download (6GB in my case, not just the 4.3GB model; I’m including the Ramalama Podman image as well).

The command-line sucks here, server helps

Very early on, you’ll find that the command-line prompt doesn’t make it easy for you to type new lines; therefore, asking the model for help with composing multi-line emails isn’t straightforward. Yes, you can use Python’s multi\nline strings\nlike this one, but for this you’ll at least need a conversion tool.

I want to have a similar UI with the ChatGPT, and it is possible!

$ ramalama serve llama3
OpenAI RESTAPI: http://localhost:8080
... starting ...

Getting this in web browser:

But after a while:

...
srv  log_server_r: request: POST /v1/chat/completions 192.168.0.4 500
srv  log_server_r: request: GET / 192.168.0.4 200
srv  log_server_r: request: GET /props 192.168.0.4 200

Admittedly, that’s just too much. I don’t need a full prompt; I’d still prefer a simple command-line interface that would let me provide multiline strings and respond with the model’s answer. Nah, we need to package python-openai into Fedora (but it is not yet there).

Performance concerns

Both llama3 and Mistral respond surprisingly quickly. The reply starts immediately, and they print approximately 30 tokens per second. Contrastingly, Deepseek takes much longer to respond, approximately a minute, but the token rate is roughly equivalent.

I was surprised to find that while the GPU was fully utilized, NVTOP did not report any additional GPU memory consumption (before, during or after model provided the answer). Does anyone have any ideas as to why this might be the case?

Summary

The mentioned models perform exceptionally well for my use-case. My interactions with the model look like:

fix english:
the multi-line text
that I want to correct

and the outputs are noticeably superior to the inputs :-).

More experimentation is possible with different models, temperature settings, RAG, and more. Refer to ramalama run –help for details.

However, I have been encountering some hardware issues with my Radeon 780M. If I run my laptop for an extended period, starting the prompt with a lengthy question can trigger a black screen situation, leaving no other interactions possible (reboot needed). If you have any suggestions on how to debug these issues, please let me know.

Access to serial console in EC2

Sun, 01 Dec 2024 00:00:00 +0000

Even in the cloud, it is sometimes convenient to monitor systemd logs via the serial console (or even log into the machine) when services like sshd fail or disks fail to mount. With EC2, you can use SSH for this purpose.

Get the instance ID

Either go to the console (web-ui) and get the instance ID there, or just ssh to the machine and query the Metadata Service:

$ ssh <user>@<host>
$ TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
$ curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id ; echo
i-015xxxxxxxxxxxxxx
$ exit

Access the serial console

On your machine, setup a few environment variables:

$ instance_id=i-015xxxxxxxxxxxxxx
$ pubkey=/home/praiskup/.ssh/id_rsa.pub
$ region=us-east-1

Tell EC2 what SSH key you want to use first, and then ssh to the console:

$ aws ec2-instance-connect send-serial-console-ssh-public-key \
    --instance-id "$instance_id" \
    --serial-port 0 \
    --ssh-public-key file://"$pubkey" \
    --region "$region"
-----------------------------------------------------
|           SendSerialConsoleSSHPublicKey           |
+----------------------------------------+----------+
|                RequestId               | Success  |
+----------------------------------------+----------+
|  xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  |  True    |
+----------------------------------------+----------+

$ ssh "$instance_id".port0@serial-console.ec2-instance-connect."$region".aws
copr-fe-dev login: 

Work with the console over SSH

Use the neat SSH control keys, start with ~?, quit the session with ~.:

copr-fe-dev login: ~?
Supported escape sequences:
 ~.   - terminate connection (and any multiplexed sessions)
 ~B   - send a BREAK to the remote system
 ~R   - request rekey
 ~V/v - decrease/increase verbosity (LogLevel)
 ~^Z  - suspend ssh
 ~#   - list forwarded connections
 ~&   - background ssh (when waiting for connections to terminate)
 ~?   - this message
 ~~   - send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)

Watch the systemd logs over SSH:

[root@copr-fe-dev ~][STG]# reboot 
         Stopping session-160.scope - Session 160 of User root...
         Stopping session-49.scope - Session 49 of User root...
[  OK  ] Removed slice system-modprobe.slice - Slice /system/modprobe.
[  OK  ] Removed slice system-sshd\x2dkeygen.slice - Slice /system/sshd-keygen.
[  OK  ] Removed slice system-systemd\x2dzr…- Slice /system/systemd-zram-setup.
...

Currently there’s a problem with Grub in EC2, but in general you would be doing something like:

$ cp /boot/grub2/grub.cfg /var/tmp/     # backup
$ grub2-editenv - unset menu_auto_hide  # https://fedoraproject.org/wiki/Changes/HiddenGrubMenu
$ vim /etc/default/grub                 # change the $GRUB_TIMEOUT
$ grub2-mkconfig > /boot/grub2/grub.cfg # re-generate
$ vim -d /boot/grub2/grub.cfg /var/tmp/grub.cfg  # review!

You might notice a problem that I did, systemd disallows reboot over EC2 console, not reported for now. The EC2 console has no “send ctrl+alt+delete” button, nor an on-screen keyboard. You can’t send ctrl+alt+delte over the SSH-console. SysRq is disabled of course (by default). Could we have something like ctrl+R for rebooting? Not sure.

Debugging Jekyll posts locally

Fri, 11 Oct 2024 00:00:00 +0000

Just a quick post about a small side-project. For a few years now, I’ve been maintaining a Jekyll Container image. Mostly for my own convenience—whether I’m working on posts for this blog or writing documentation for Mock (or other). I thought I’d share a few words about it now.

Keep My Box Clean! (and DRY)

The motivation was simple: to be able to debug Jekyll/GitHub Pages posts locally before pushing them to GitHub. I wanted to do this consistently across multiple pages, and I didn’t want to repeat myself in the future (following the DRY principle).

Back then, I realized that running Jekyll locally wasn’t a trivial task—at least not on Fedora, if, like me, you prefer staying on a “pure” Fedora system (meaning you only install software distributed through Fedora repositories). Notably, installing the GitHub Pages additions from gems wasn’t easy either, and it still isn’t (as of autumn 2024). Building your own container can also cause some headaches.

Jekyll made easy

So here we are—assuming you have a blog post or any documentation root directory, you can run the Jekyll server in a container, and available on http://localhost:4000/, using just:

$ jekyll-host ./your-jekyll-root
Installing deps, may take several minutes
=========================================
 Server listens on http://127.0.0.1:4000
 Jekyll Log: /tmp/jekyll-server.log (in container)
 Install logs: /tmp/bundler-install.log (in container)
=========================================

The jekyll-host script (which must be in your $PATH) is just a one-line wrapper around a podman run command that uses a pre-built container image hosted and built by https://quay.io/.

I prefer to stay 100% focused on writing, not on the infrastructure. After a quick chat with my colleagues, it seems this setup could be helpful to others as well. If that’s the case, enjoy!

Cleaning up rolling distro builds in Copr

Mon, 01 Jul 2024 00:00:00 +0000

Need something extra? Employ your own Copr builders!

Sat, 22 Jun 2024 00:00:00 +0000

Each Copr deployment needs to manage a farm of builders (normally virtual machines) across various clouds, with various architectures and performance profiles.

For example, in the Fedora Copr instance, where there is normally a reasonable amount of x86_64/amd64 builders, users very often wait for s390x/Z builders.

The recent Copr contribution has given us option to allocate high-performance builders for specific build tasks. And as a nice side effect, it also opened the door to providing any other custom workers. And, if you don’t want to wait in the queue anymore, you can also have workers dedicated to your team!

What’s Needed?

Copr administrators should be able to configure a new pool(s) of VMs (builders) for you, and then make your builds use them.

Of course, admins will need your permission to start the machines in your name. There’s already a built-in support for working with IBM Cloud, OpenStack, EC2 or LibVirt (so a cloud token is needed, or SSH access to hypervisor). Support for other clouds could be added, too — we’d just need a bit more help from you (scripting and golden-image preparation).

You need to make a budget decision. Your builders can either be booted on demand (which may delay the build processing by 2 or 3 minutes, but it depends…) or use the VM pre-allocation mechanism (a few builders are booted in advance, you pay the cloud costs for them, but they are pre-prepared to take your tasks immediately).

Then one note (similar like with the high-performance builders). For the configured builds, your cloud account (and only that) will be used for virtual machine allocation, and it probably isn’t limitless.

Community Sponsors

Resalloc also allows Copr admins to configure additional pools of workers for the general public. This can be beneficial for interesting use-cases (platform enablement, specific hardware needs, AI-related builds, etc.), or even just to boost the overall Copr throughput. If you plan to help the community this way, or you already do, THANK YOU very much!

What next?

Please consult the options with us, either on the copr-devel list, the team e-mail copr-team@redhat.com, or submit an issue.

RFC: Planning Copr → PULP RPM content movement

Fri, 15 Mar 2024 00:00:00 +0000

The Fedora Copr storage hosts (beginning of Y2024) approximately 33TB of data (RPM repositories and RPM files). The data is stored on a Fedora server, utilizing a rather complicated lvm+raid setup, all hosted on AWS EC2, typically hidden behind CDN (AWS CloudFront). Modifying this setup and further maintenance is non-trivial (adding new arrays of disks into LVM, migrating the box to new Fedora releases, etc).

We’d like to use something that auto-scales long-term, either something like Amazon EFS (seems more expensive compared to our current setup) or S3 (implies a significant rewrite of Copr code).

Also, things used to be relatively trivial years ago. With the current data payload though, even some of our logic needs to be updated (both for convenience and performance).

Wait, we are not the first team hosting RPMs, right? There’s PULP! Why not delegate the heavy-lifting to the project designed for RPM content hosting (actually having the S3 support)? Sure, the switch is not going to be trivial. In this post, I’d like to at least propose the basic “stages” for the hostname migration (how users consume the storage).

The old state

The results are served by lighttpd on the Copr Backend box, in the non-cached results directory, and cached by CloudFronts (fp.o = fedoraproject.org, fic.o = fedorainfracloud.org).

All users consume content through CDN, download.copr.fedorainfracloud.org.

The migration time

We’ll start with a new hostname related to the PULP server. We are not sure whether we’ll use the PULP built-in HA, CloudFront, or both. Initially, we’ll begin moving a small set of projects, and the copr-be.cloud.fp.o box will have to start redirecting (some) requests to them to, e.g., pulp.copr.fp.o. You may wonder if this “redirection” is supported by DNF5, DNF, and YUM customers — it is (tested even on CentOS 6).

Migrated Copr projects will provide updated .repo files (like this), pointing to the PULP hostname. So, when users start consuming project content, e.g., with dnf copr enable, their machines will not have to follow the redirects!

Once done

The download.corp.fedorainfracloud.org no longer points to the CloudFront cache but directly to the PULP host. This change will be necessary for some indefinite amount of time to remain compatible with the old .repo file installations.

Bad idea?

The primary motivation for this flow is to ensure the smoothest possible experience throughout all phases (repositories available all the time).

Please let us know if you have any concerns or questions. Otherwise, just stay tuned for further updates!

AI-driven RPM Build Failure diagnosis

Tue, 16 Jan 2024 00:00:00 +0000