I'm on Signal (obviously not self hosted) and even if I really wanted to move to another platform be it self hosted or yet another privacy focussed one, I can't ask my friends and family to move to another platform again. I already asked them to move away from WhatsApp, can't do it again..
This is what I told most of my friend when they asked me to move to signal. Is is going to be a very shitty company managed by a shitty egocentric person and you are going to regret. But you will make people move and they won't do it again and won't understand the reasons
Matrix. With its bridges you can "wire-in" networks like WhatsApp, Slack, Signal, Telegram, Discord, iMessage, SMS, e-mail, ... and have a single app that interacts with them all. You can have a single group chat with users from all those networks participating and no one would be any the wiser.
From my experience (with Dendrite, not synapse, so keep that in mind), bridges create "fake" users to replicate your contacts on these platform as matrix users, and they are visible on the whole instance by all their users (but you might not be able to talk to them). Also, in puppeted mode (which is what you want to "replace" your app with matrix), only a single user can use the bridge at a time, so the other users cannot use it.
If you're already using Nextcloud, it has a chat w/ video chat as well.
Matrix / Synapse / Element.io is also pretty cool. The UX might not be on par with what some family expects though. I don't know if voice/video chat is built-in yet or not, but it was at least an option before.
My family has been using Synapse since before Covid hit, including TURN server (coturn) for audio and video calls. No complaints about the UI so far, except for Element on iPhone reportedly "freezing" for a few moments when sending messages - and that was over a year ago.
I tried setting up coturn with docker but I can't remember offhand what I had issues with. I am running a reverse proxy (Traefik) with wildcard certs. I should dig into it again. Do you happen to have any pointers or any good guides I could try following?
Just for a family and friends I'd go for xmpp. Matrix is still an enormous greavy piece of software, hard to self host if you don't want to pay for a gigantic server just for it. Also the UI is more like gamer/company chat (discord, slack...), what may not be what your family expect, coming from whatsapp, telegram, or plain sms. In the contrary xmpp is very light and nowadays a lot of tutorial exists on how to configure it, even with voice/video. Plus mobile apps like conversation match the habbits of other messengers.
Another Nextcloud user here. If you setup a Nextcloud server you will see many benefits, with the Talk app being just one of them. Install OnlyOffice and you can even collaboratively edit documents in the web browser similar to Google Docs or o365. I really can't recommend Nextcloud enough. Anyone who is into self hosting should at least give it a try.
Nextcloud looks really great and it has a chat / video chat too, I want to give it a spin in the future, as it also allows you to self-host a lot of things that people usually outsource to Microsoft, Google or Apple.
Matrix works, but it's way harder and more expensive to selfhost than for example XMPP, which can be hosted even on cheapest VPS or first RPi. I would definitely take the cost and "how hard is it to maintain in the long run" into consideration.
Mattermost also works and is pretty easy to selfhost, but it doesn't have federation.
Another option is always an email with delta.chat - I don't think it offers voice calling, but email is one of the most basic services one can host, and many automated solutions to help with that exist.
Matrix. With its bridges you can "wire-in" networks like WhatsApp, Slack, Signal, Telegram, Discord, iMessage, SMS, e-mail, ... and have a single app that interacts with them all. You can have a single group chat with users from all those networks participating and no one would be any the wiser.