Not the OP, but I'll add my 2c
I've got various components on my server that do have CLI counterparts, but functionality is limited or annoying at best.
For example, managing stuff under KVM can be done with "virsh", editing a bunch of XML, and configuring SSH from the master to every underlying system...
Or I could just RDP/X11-forward from my other device and do it in a couple clicks.
That said, SSH (or a VPN) is still partly the answer as I don't like exposing RDP/VNC services to the internet, but XRDP or x11vnc etc with port forwarding through SSH work nicely.
If one is connecting from Android then ConnectBot supports port forwarding and can also be used with an X server application (i.e X11 XSDL) if you wanted to just go with just SSH+X-Forwarding