PSA: Lemmy.world is having some heavy federation issues lately.

As per this post, it seems Lemmy World is having federation issues. I've had some of my posts and comments get suspiciously low response, so I've looked into it and it turns out that many of them aren't showing on other instances.

Here's an example: my Payday 3 post is only visible on Lemmy.world:

Meanwhile, in other instances, it doesn't show up at all:

It's not clear if this issue is exclusive to this instance but I thought I'd give people a heads up. I reposted and now it showed up in other instances. If you feel like you're getting no response from other communities, check on another instance if your content is visible, otherwise repost it.

42 comments

It isn't just Lemmy.world, and isn't limited to v0.17 either. Lemmy.ca, which has been running v0.18 for days, is also not seeing a lot of federated content, nor accurate community subscription counts from other instances, including lemmy.ml and others running v0.18. I'm posting this from my .world alt because .ca is showing zero comments in this thread.
There have been several not-so-friendly comments made about the issues by someone in !lemmy_administration@lemmy.ml, calling out the site admins of .world and .ml for not sharing their server logs, and the overall silence on the issue. The user's concerns are genuine, but I don't agree with their behaviour.
I suspect the devs were/are completely overwhelmed by the mass influx of redditors these past two weeks. I assume someone is looking into it, but frankly I wouldn't blame anyone if they chose to step away for a few days. I think a lot of folks (particularly "the masses") forget that these are largely volunteer efforts, by people with day jobs.
- Thanks for your reply. This site isn’t even a month old and the user growth is spectacular. Lemmy hasn’t ever been tested with these amounts of users.
  I’m sure the devs are doing their best in getting it all more performant. Meanwhile the team at lemmy.world is trying to get a grip on what’s happening with federation. There’s not much to be found in logs, anyone who can help troubleshoot this let us know. The only real adjustable setting is the federation workers. We’ve apparently either set that too high or too low. I’m gonna set the log level back to info on the hopes we get some queue stats…
  To be continued
  
  No worries, and I completely sympathize with the situation. My hat is off to you site admins, and the lemmy devs, for keeping things afloat these past couple of weeks, often at great expense of both time and money. I am confident that all of these kinks will be ironed out in the coming months, and in the meantime I am happy to have a few alts to hold me over.
  
  Back in my day 1 month old sites had 0% downtime and no reported glitches. 1 million daily active users! There's no need for beta software either cause 2.0 was always the perfect release.
  Those were the days!
  
  So to be clear, we're fairly certain this is a technical issue and not because of something other instances are doing?
- for not sharing their server logs
  WTF do users think they have a right to see these at all? Just because Lemmy is open source doesn't mean you get to run the damn thing.
  Also my condolonces on you being a Canadian.
- I sympathize. I just hope it's not true. Friday and Saturday of this week are likely the second and bigger wave of redditors coming over as their 3rd party apps get disabled on/after 30th. I hope the admins are preparing for it.

Correct me if I’m wrong but .18 lemmy doesn’t support captcha, so lemmy.world is waiting for a patch before updating so we don’t get flooded with bots.
Edit: fixed typo

Just clicked on a Futurama related post on Lemmy.world front page, it opened wtih this MLB Game Thread as content instead of what I clicked on. **This is happened more than once. Previously I thought I derp clicked on the wrong item **
- This has been happening to me for at least a day now. Reloading fixes it and brings you to the post you thought you were in.
  Funny thing is sports communities are the ones I block as I come across them, and this glitch circumvents blocked communities.
- I'm not kidding when I say I opened a meme post and the picture was from the "bulges" page. The comments were the correct meme post comments though. I went back and forward again in my browser and the image was correct.

I believe this is a side effect of the 0.17 vs 0.18 split which unfortunately wasn't backwards compatible. Once 0.18.1 rolls out with readded captcha everything should be hunky-dory again.
- Fingers crossed it doesn't take that long!
  
  Interesting that’s it’s been days without a minor beta at all. Where in the race for 0.18 the updates were 12 hourly at one point. Guess all the devs burnt the candle at both end. Much appreciated ❤️
- Was !games@sh.itjust.works the instance that was not showing the post?
  
  Yes, I ended up reposting there and it shows up now.

Lemmy.world is definitely having federation issues. Lemmy in general is having a lot of federation issues, but it looks like lemmy.world is being effected more so due to the number of accounts and activity on lemmy.world in combination with different versions of lemmy handling messages a bit differently with different implicit assumptions.

Workaround, I had to do the same on kbin.
Editing your comment triggers another sync, often an edit will make the comment or post synchronise to all other instances.

That's pretty much what I yesterday called the lemmy.world bubble.

To add some specificity to this:
I've been unable to post to !arduino@lemmy.ca or !avr@lemmy.ca successfully. And the comments seem to be failing to load at all.
Here's to hoping its the .18.x vs .17 issue. But I figured I need to post an official report of my problems somewhere, and this topic seems the most applicable location.
I've also been "Subscription pending" on !avr@lemmy.ca for... at least 20 minutes now. Suspiciously long times to get that resolved.
- 5 hours later, I'm still "Subscription pending". I dunno if this is related to the issues, but I figure I should document the problem here.
  
  Almost a day later, "Subscription pending". I'm guessing its this slow-federation issue that probably will be fixed with .18 upgrade (or maybe lemmy.ca needs to increase its federation workers?)

There should be a fix in a day or two, there was an issue with captcha to my understanding.
- Ruud has said he doesn't know if any .18x update will fix these issues

If it was reposted and worked that’s not a federation issue right? Could it be that world isn’t on .18 yet?

I'm very new to Lemmy and have already been experiencing this problem just testing how commenting to a federated community on another server works. It doesn't/hasn't worked for me yet in controlled testing. I've been assuming it was a problem with the other lemmy instance where the community is created/hosted but now I'm rethinking that assumption.
Usually, not assigning sufficient resources at a workload just causes the backlog queue to get longer, but in the case of lemmy it seems to result in some items never getting processed at all. That's an odd behavior for a backlog to have.

I've had difficulties with responding or commenting on posts on federated servers. Comments work just fine if I'm on this instance, but threads on others it just never posts.

I got a report earlier that my community wasn't showing up on https://browse.feddit.de earlier as well, but it's back now. I'm not sure how it works but I assume it's related.
- Any idea on that 0.18.1 release? I have no idea how much work is required to add re:capcha. Are we talking days, weeks or months off work?
  
  Pull request which restored Captchas but was reverted shortly before 0.18.0 got released
  Pull request which hopefully restores Captchas in 0.18.1

This is why its a bad idea to have massive servers

Just wanted to let you all know that the federation with kbin.social is working fine.
- I'm not sure, I can't see their original Payday 3 post, only the reposted one. Looking a little closer, they might have just deleted it since I can't see it from lemmy.world either.
  
  It might be interesting to have one of the monitoring tools that people are banging check on post latency between various instances.

42 comments