I’ve been noticing reddthat going down for a short time (a few minutes or so) more than before - is everything okay?

https://lestat.org/status/lemmy shows our uptime at 88.82% right now - not horrible, but not great either. We of course stunt need to be up 99.9999% of the time, but still.

Is there anything we can help with besides donating? I like it here, so I want to make sure this place stays up for a long time 👍

  • Tiff@reddthat.comMEnglish
    3·
    11 months ago

    Updates hiding in the comments again!

    We are now using v0.18.3!

    There was extended downtime because docker wouldn’t cooperate AT ALL.

    The nginx proxy container would not resolve the DNS. So after rebuilding the containers twice and investigating the docker network settings, a “simple” reboot of the server fixed it!

    1. Our database on the filesystem went from 33GB to 5GB! They were not kidding about the 80% reduction!
    2. The compressed database backups went from 4GB to ~0.7GB! Even bigger space savings.
    3. The changes to backend/frontend has resulted in less downtime when performing big queries on the database so far.
    4. The “proxy” container is nginx, and because it utilises the configuration upstream lemmy-ui & upstream lemmy. These are DNS entries which are cached for a period of time. So if a new container comes online it doesn’t actually find the new containers because it cached all the IPs that lemmy-ui resolves too. (In this example it would have been only 1, and then we add more containers the proxy would never find them). 4.1 You can read more here: http://forum.nginx.org/read.php?2,215830,215832#msg-215832
    5. The good news is that https://serverfault.com/a/593003 is the answer to the question. I’ll look at implementing this over the next day(s).

    I get notified whenever reddthat goes down, most of the time it coincided with me banning users and removing content. So I didn’t look into it much, but honestly the uptime isn’t great. (Red is <95% uptime, which means we were down for 1 hour!).

    Actually, it is terrible.

    With the changes we’ve made i’ll be monitoring it over the next 48 hours and confirm that we no longer have any real issues. Then i’ll make a real announcement.

    Thanks all for joining our little adventure!
    Tiff