1y ago

Server notification system

Hey all, For context, I am self-hosting a lot of different applications on a headless server and some of them are just scripts scraping content or doing automatic tasks.

I've been wondering for some time about how I should handle errors and how to be notified when they happen.

Currently, I need to connect to the server and read my logs to know that something has gone wrong. But I would like to get near instant notifications on my phone or computer.

I know I could send emails, try to use signal-cli or other messaging services but some of you might have better ideas.

Should I use one of these or is there some great technology I have never heard of?

17 comments

Use ntfy.sh. It's open source and has a free server.
Disclaimer: I made it ;-)
- This looks brilliant! I love how simple the sending API is.
  It had occurred to me that I could do something like this, but I'm happy you did all the hard work for me (especially the always-on feature of the Android app).
- Wow really nice project! Do you work on this full time?
  
  Thanks. I don't work on it full time, no. It's a side gig project I've been doing for a year and a half. I recently added paid plans to get a little side income, but it's not really taken off. Likely because the free tier is too generous hehe.
- Oh great ! I will look into it, it seems very cool.

https://grafana.com/docs/grafana/latest/alerting/
- Didn't know grafana provided this kind of feature. I don't think i'm going to use it but it's really good to know. Thanks !
  
  Prometheus has alerting and that’s really the preferred system for sending the alerts, IMO.

You could setup a status monitoring system and then configured it to send out messages for critical alerts. For example, I'm using Vigil to monitor my services and it's configured to send email alerts when something is down and then sms alerts when things are still down for too long (in case I didn't read the email).
- You're right ! I didn't think about status page tools... It might be overkill for my setup but I will definitely look into it. Thank you.

I use Seq. You can use it for free for personal use and it has a docker image available. I can use their libraries to send logs my scripts create to Seq, you can ingest logs from other docker containers and syslog, and set it up to send emails for certain log message patterns.
Also is very easy to install and set up. It's just an exe in windows and docker container elsewhere.

Might be overkill but you could setup and run a Jenkins server to run your scripts and use one of the many notification plugins that are available.

https://grafana.com/docs/grafana/latest/alerting/

17 comments