What’s happening?
feddit.uk will be down for maintainence between 19:00 and 21:00 UTC (GMT in old money) on Saturday the 27th of January.
We are aiming to move the site between hosts, and restore missing images.
How long do you expect it to take?
The target time is 1 hour. The window is 2 hours. That’s enough time for about 1 season of Peep Show, if you’re bored.
What is the rollback plan?
If we’re not happy with things by 20:30, we will bring the site up in it’s previous state, and schedule a longer window in the future. We’ll let you know if/when the decision is made.
Where can I see status updates?
We have a public channel on Matrix for people who wish to keep up with the status. It’s part of the feddit.uk Matrix space which is there as a fallback in case of any downtime and to provide end-to-end encrypted communication where required.
edit: there is now a WhatsApp channel.
Why are you doing this?
As previously discussed in here, it’s been quite the journey but, in summary: Tom, who set up the instance and was the only Admin, went AWOL. GreatAlbatross and I contacted him offering to take over and Tom made us Admins but we didn’t have any access to the server, so we were limited in what we could do.
The situation got increasingly problematic and plans were drawn up to take over the whole running of the site or to start a new one from scratch. It took a while but just as we were about to pull the trigger on the new instance Tom gave us access. GA is now in charge of the domain and I have overall responsibility for the new server.
The site has been suffering outages because the size of the image folders ate up all the available space and Tom removed the first image folder (why various graphics are still missing). That made moving the site to a new hosting plan more urgent and we’ve been working away in the background to get to the point that we were in a position to start the process. Hence the downtime.
We’d like to thank everyone for their patience and enthusiasm - it has helped carry us through what hasn’t been a straightforward process.
I have more questions
There will be more information after the move about how we are going to move forward with feddit.uk to ensure it will thrive for as long as people want it. However, if there’s anything that is unclear about current events then fire away in the comments below.
You must log in or register to comment.
If this message is posted…Then we are migrated. I just need to sort a few missing pictures.
Looking good to me. Well done folks!
And… we’re back!
Top job everyone. Thanks a million.
Good work guys
I actually thought the site was down for the last few weeks. It was just showing an error.
It was only when I cleared the cookies, and relogged in that it would load again. Think it was something to do with the upgrade to 0.19, but I do wonder how many other users are just seeing the error and making new accounts elsewhere.
We were able to update the error message and point people to Matrix if they had any issues.
We don’t know how many people we’ve lost in this whole process but we will try and spread the word around and hopefully hook people back in (their accounts and all our communities will still be there, so folks will wander back, even if they may be using a new account). Fingers crossed, everything should be a lot more stable going forward and we can all get back to the more interesting stuff.
I responded to somebody a while back with the same issue. The server error message is very unhelpful!
The image folder has now been restored on the new location, for anybody wondering.
It was deemed to be a safer way to do things, than try to fold thousands of images into a folder on a live site.Do either you or Emperor have any experience with web servers or are you muddling along? Or maybe getting help from Tom?
What Linux distro is the new server running?
Do either you or Emperor have any experience with web servers or are you muddling along?
My previous answer stands.
Or maybe getting help from Tom?
Other than providing us with access to the relevant bits and bobs Tom is not involved with the move.
What Linux distro is the new server running?
We will only be revealing some general technical information about the new server to reduce the attack surface.
My previous answer stands.
What previous answer? Nevermind - found it!
Bloody hell you’ve got a good memory!
Bloody hell you’ve got a good memory!
I’m like an elephant, in as much as I am grey, wrinkly and do large poos.
I’m like a seagull, in as much that I see something going down, swoop in, shit everywhere, and leave.
And steal chips.
Issues with image uploads should now be fixed.
Unfortunately, this has interrupted my long-running task restoring the images uploaded between when I made the image backup, and when the old server was turned off (about 5 days).
We’re aiming to restore them soon, but it’s going to be a background task, and it’s going to take a while (there are millions of images, and the sync task needs to scan each one, see if it’s already present, and if not put it in place).
I’m sacrificing a goat to appease the gods of live server maintenance as I type
For anyone worried about this, no animals will be harmed during the migration, other than some potential gnashing of teeth and/or tearing hair out (all human, all self-inflicted). However, any “spare” goat meat can be sent to my PO Box.
We now have a WhatsApp channel. Seems like the right time to roll it out.
There will be other ways of engaging with feddit.uk coming up
Aw man, why WhatsApp? Facebook apps should be avoided by everyone.
Meta apps cause bone cancer. I think Emperor帝 is using it to trace people for GCHQ too!
The irony of using WhatsApp as a fallback here
We aren’t - that’s Matrix, a WhatsApp channel just seemed a quick and easy extra way of getting information out.
While I’d love it if everyone I knew abandoned WhatsApp, it’ll never happen, so I and many others use it.
We are going to be increasing our… outreach to other services going forward and this will be one option amongst many. If people don’t want it then they don’t have to use it - we’d still encourage people to join our space on Matrix.
When I was browsing instances, I vaguely remember that feddit.uk was hosted in Finland. Where will the new hosting be based?
Germany. Testing so far, the ping round trip is improved a fair chunk.
I have no particular opinion but was this based on cost or prior experience with the host? Wouldn’t a UK host have been even better ping wise?
Perhaps, but it would be absolutely negligible, compared to the savings. My guess is the new host is Hetzner, which is good. The difference in ping wouldn’t even matter for most videogames tbh, as long as it’s the same continent
Partly cost, partly experience.
Hosting it in the UK would have been 95% the same price for coloc alone, dedicated server rental from a non-crap company would have been considerably more.
You’re not using VPS or cloud? What kind of 90s, BBS-loving attitude is this?
Explain yourself Old Abatross
Good luck team. I’ve been waiting for this day so I can create new banners for [email protected].
Does that mean all missing avatars and icons have been there all along but just been disabled? Will that get repopulated from other servers? Just curious.
P.S. I like the gateway message. Nice touch.
P.S. I like the gateway message. Nice touch.
Thanks! I was having a ponder on what the site “was”. And thinking of it as a gateway, with a some social spaces too, felt right.
Kinda early web style, perhaps?Does that mean all missing avatars and icons have been there all along but just been disabled?
The first folder of images (which held a lot of early, core graphics) was zipped up and removed from the server.
Will that get repopulated from other servers?
I don’t want to guarantee anything until we can see that everything has worked, but that folder is currently reintegrated into the new server. It may be caches need to ge cleared or something needs the delicate technical equivalent of a swift kick but all graphics should work within a day or so (or immediately for some people, with luck).
We’ll be able to update everyone on this tomorrow if the move has been successful, so if caches need clearing for some folks they’ll know pretty quickly.
@[email protected] and @[email protected] - why did you not clear this with me before you made the plan?
This is the first I’ve heard of it so I find this quite a surprise which is unacceptable considering my importance to the smooth running of the site!
I provide this site with much needed wit and wisdom.
Could you explain this oversight please?
If I don’t receive an adequate explanation I’ll not give my blessing for the move.
If I don’t receive an adequate explanation I’ll not give my blessing for the move.
So if something goes wrong, it’s your fault? Duly noted if we need to apportion blame tomorrow evening.
i will be truthful, i had lost complete confidence in the instance after seeing the crashes back in december. i had moved to using lemmy anonymously since then, to “wait and see” if i had to setup an account elsewhere.
seeing this though? full confidence restored. 👍
i am glad there is now more than one single point of failure. i hope that this adventure becomes a valuable lesson to other instances that have just one admin running everything. if you’re reading this, and your instance has one admin - you need to fix that, yesterday!
once you have an OpenCollective or something like it setup, i will be able to contribute. make sure to let us know when a fresh donation link is up (because i don’t trust the old ko-fi link in the sidebar)
thank you so much for your hard work. ❤️
And thanks for your kind words.
i am glad there is now more than one single point of failure.
And that was a definite concern for us and other Lemmy instance Admins.
Going forward feddit.uk is going to be future-proofed so it can survive and thrive. It’s now everyone’s site, the Admins are just running it for you. So it’s not just a change in getting to the point where the site can survive the loss of an Admin, it’s a change in… philosophy.
once you have an OpenCollective or something like it setup, i will be able to contribute. make sure to let us know when a fresh donation link is up (because i don’t trust the old ko-fi link in the sidebar)
Yes the Ko-fi link will go once we are 100% confident we don’t need access to the old server and so we are only using our own resources. I’ll get in the case with researching Open Collective so we can use the momentum of the changes to ensure the finances are solid.