How significant is it that the server code is open-source or not? It's possible for Signal to publish their server code while running completely different software on their servers. The point of the client is being open source and audited on a regular basis by the community, which is why it doesn't make sense to trust the server-side software.
The entire point is that we don’t have to trust the sever at all. The client is open source and regularly audited by the community. As long as the client stays fully open source, everything’s fine. Also, the closed source dependencies are part of a spam reduction effort which IMO is well worth it. Prior to this, Signal had a spam problem and the client itself remains fully open source.
Signal could have very well not even told people that they added a closed source dependency on Google to its servers and just lied by publishing fake server code that omits the closed source dependency., but instead they were very transparent about the spam problem. In terms of they "why?" regarding the closed source dependencies, their argument is that making it open source would almost immediately result in all anti-spam measures being thwarted. Frankly I'm inclined to agree and again, as long as the client is fully open source and regularly audited, the server code is irrelevant to user privacy/security.
https://community.signalusers.org/t/spam-scam-on-signal/26665
https://signal.org/blog/keeping-spam-off-signal/