Pipewire freezes after initial login or screen blank

For a few months now, Pipewire has been acting in weird ways. Before I upgraded to Plasma 6, after a screen blank (not suspend) no media can be played. For example, YouTube player is in the loading state for a long time, eventually playing without sound, VLC doesn't start for a few minutes.

I have to restart the Pipewire Systemd service to fix it. This restart process takes much longer than usual (<1 seconds). Sometimes it takes up to 5 minutes.

Now with Plasma 6, it also does the same after the initial login. I'm guessing this is because Plasma ignores some KWIN_X11_* env var?

However, if I restart the Pipewire service prior to playing any media after login or screen blank, it takes <1 seconds and everything plays correctly.

This seems to be the problem with the HDMI audio only, as switching to the USB headphones (before screen blanks) prevents this from happening.

How should I start troubleshooting this? What other details should I provide?