How do they know you verified by decrypting?
The client has to be able to verify. They can't trust your result. Imagine a man-in-the-middle attack; if someone intercepted all traffic between you and the client, including the cert exchange, how would either party figure out that traffic was being intercepted?
Client connects to website, but gets intercepted. Attacker provides own self-signed certificate to client. Client asks you to verify the certificate, but attacker can intercept that too and just reply the certificate is "totes cool bro just trust me". You are none the wiser either, because the attacker can just decrypt client traffic and pretend they are the client by re-encrypting the data themselves.
With a Certificate Authority, the client can take the received cert and ask the CA "did you sign this?". The CA will then tell you they didn't, exposing the attacker's fake cert. This works, because the CA is already a trusted entity. That trust is being extended to your website's certificate validity and thus the website identity.