Currently, you have RabbitMQ server installed, which is not directly
upgradable to the newer version. Upgrading may prevent the server
from starting due to missing feature flags.
Exactly my point.. Well it's more: "Server is fine, except for RabbitMQ service". Anyway, I think it's nice to see they try to prevent me upgrading to a breaking system I guess.
Update: I removed the RabbitMQ server, performed the upgrade to Ubuntu 24.04. And reinstalled RabbitMQ server again. Deleting the queues that are causing issues, restarted RabbitMQ and Mbin. And now everything seems to work just fine, running Ubuntu 24.04!