Lemmy via Docker Compose, using Traefik and CloudFlare

Disclaimers:

First thing first, I'm new to the whole Fediverse, and Lemmy thing, so please don't hesitate to point out any problems you're foreseeing.

Secondly, I'm by no means saying this is the ideal implementation, something something see above. Please don't hesitate to make recommendations for improvements.

Lastly, I'm not sure if it is completely working. I'm still noticing a few issues that I will document and monitor towards the end of the post. If you know of the cause or how to debug further, please do let me know!

Notes and Assumptions:

I am using an ARM server. So I'm using ARM images, you will need to make sure you're using the correct architecture image.
I assume you have Traefik up and running in a separate network. I used docker compose to bring traefik up, minimal configurations, and I'm just hijacking the default network there (project folder was gateway so the complete network name is gateway_default)... there's probably better ways to do this.
On note of networks, I really don't like the fact that the default postgres was left wide open on the lemmyexternalproxy network. I think I've locked my down, but you may wish to double check my work.
I'm not sure if what I am doing with the hostnames are correct, but it seems to work for the most part, so I'm not complaining. If there is a better way, please do advise!
I used an override file for docker compose to apply extra settings. This allows me to keep the original docker-compose.yml untouched, and I can just pull in new changes (theoretically).
Since I'm using traefik, I don't need nginx running doing nothing. I replaced it with a light weight alpine image that just shuts down successfully, so it doesn't use resources.

Without further delays, here's my files:

docker-compose.override.yml:

 undefined

    
version: "3.3"

networks:
  lemmyexternalproxy:
    internal: true
  lemmygateway:
    name: gateway_default
    external: true

services:
  lemmy:
    image: dessalines/lemmy:0.17-linux-arm64
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.lemmy.entrypoints=websecure"
      - "traefik.http.routers.lemmy.rule=Host(`lemmy.chiisana.net`) && HeadersRegexp(`Accept`, `^application/`) || Host(`lemmy.chiisana.net`) && Method(`POST`) || Host(`lemmy.chiisana.net`) && PathPrefix(`/{path:(api|pictrs|feeds|nodeinfo|.well-known)}`)"
      - "traefik.http.routers.lemmy.tls=true"
      - "traefik.http.services.lemmy-svc.loadbalancer.server.port=8536"
      - "traefik.docker.network=gateway_default"
    networks:
      - lemmygateway
  lemmy-ui:
    image: dessalines/lemmy-ui:0.17-linux-arm64
    environment:
      - LEMMY_UI_HOST=0.0.0.0:1234
      - LEMMY_UI_LEMMY_INTERNAL_HOST=lemmy:8536
      - LEMMY_UI_LEMMY_EXTERNAL_HOST=lemmy.chiisana.net
      - LEMMY_UI_HTTPS=true
      - LEMMY_UI_DEBUG=false
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.lemmy-ui.entrypoints=websecure"
      - "traefik.http.routers.lemmy-ui.rule=Host(`lemmy.chiisana.net`)"
      - "traefik.http.routers.lemmy-ui.tls=true"
      - "traefik.http.services.lemmy-ui-svc.loadbalancer.server.port=1234"
      - "traefik.docker.network=gateway_default"
    networks:
      - lemmygateway
  proxy:
    image: alpine:latest
    command: "true"
    entrypoint: "true"
    restart: "no"
  pictrs:
    image: asonix/pictrs:0.4.0-rc.3

lemmy.hjson:

    
  setup: {
    admin_username: "chiisana"
    admin_password: "password-redacted-duh"
    site_name: "chiisana lemmy site"
  }
  database: {
    host: "postgres"
    user: "lemmy"
    password: "password-redacted-duh"
    database: "lemmy"
  }
  email: {
    smtp_server: "smtp.mailgun.org:587"
    smtp_login: "lemmy@chiisana.net"
    smtp_password: "password-redacted-duh"
    smtp_from_address: "lemmy@chiisana.net"
    tls_type: "tls"
  }
  pictrs: {
    url: "http://pictrs:8080/"
    api_key: "API_KEY"
  }
  hostname: "lemmy.chiisana.net"
  bind: "0.0.0.0"
  port: 8536
  tls_enabled: true
}

Known issue(s)?

I have my registration disabled as the instance is supposed to be just for my own auth not be depended on other instances. In my /admin section, I'm seeing a ton of users from endlesstalk.org pop up as banned users. I have no idea what that is about, as endlesstalk.org seems to also be used only by one user. I'll be monitoring this and see what's to come of it. Edit: Looks like this is just the way the system is designed, and not a configuration error on my part! All good here. Thanks for clarifying it @lemmy@endlesstalk.org !
I'm not sure if I'm getting all the messages federated. In this community, for example, I can see most if not all recent threads. However, most threads have no comments in it. Some newer threads, I see comments, but it seems to be incomplete. I'm not sure if I'm only supposed to receive new messages, or if something else is happening. I'll be monitoring this, and hoping the federation will just catch up over time.
Edit: It would appear this post itself is not federating to !selfhosted@lemmy.world for some reason... I'm partially hoping it is just caught in some kind of moderation queue, but seeing other posts made after this appear on the list leads me to believe there's still something amiss.

If you encounter any other issue, please do post back so we can try to debug it together. Hope this helps someone!

12 comments

I have also tons of banned users from @endlesstalk.org and @rabbitea.rs
- Rabbitears haven't popped up for me yet. I think the "issue" is that in a federated environment, Lemmy federates the banned users as well.
  If you look at Lemmy World's mod log and Endless Talk's mod log it becomes pretty quick to see where these bans are coming from.
  Frankly, I'm not sure if this is the "right" approach, but this appears to be the approach the developers of Lemmy have chosen. I'm probably just going to ignore the banned users section moving forward for the time being, and pray that no one can ban my user from their server, resulting in a cascade Fediverse wide ban... I don' think that should be possible, yet here we are discussing the oddities of unknown banned users... 🤷

Regarding the banned users from endlesstalk.org
There were a bot creating around 400 users on my instance(endlesstalk.org). At first I banned them all, but later I figured the better solution was to remove the users from my database(Which I have now done)
- Hey thank you for this note Philip! I've also noticed chatters on other threads and this github issue, so I guess it is just a design of the network kind of thing, rather than a @chiisana misconfigured this kind of thing. I'll go ahead and cross that part out as non-issue :)

Thanks for this. Will try it in a bit. Gave it a shot myself a few days ago but wasn’t able to log in when accessing through traefik via web
- Unfortunately, it would appear that there's not without very significant problems... I'm commenting to your comment via my lemmy.world account because I'm not seeing your comment on my instance.
  If you do get it working, and find ways to resolve issues I'm having, please do share back so I can get my instance fixed as well! Thanks!
  
  Will do. Trying to find a solution for my search error

This helped, thank you! Especially the Traefik rules, so that we can ditch the unneccessary nginx proxy! My issue currently is, that I don't get any comments from other instances. Apparently comments on my instance in communities like !lemmysupport@lemmy.ml can be seen there, but nothing comes back onto my instance...
Not sure what's going on there.
Btw. I'm using a rather fresh version of 0.17.4, but nothing seems to work there...
- Yeah, I'm getting mixed results as well. Federation seems to be super finicky right now. A lot of finger pointing going on and some posts I've seen suggests it is Cloudflare being the culprit. As much as I'd like to shed Cloudflare to get federation working, I just don't see that being something that's viable long term. It is very easy to DDOS someone, and I do not want to expose my instance IP publicly.
  Looking at the commit logs, the difference between 0.17.3 and 0.17.4 seems to be just some database optimizations, so I think the problem we're seeing is still something else.
  Also, the lemmy.ml instance is acting up across the board, even from the lemmy.world instance, or other major instances, the subscribe doesn't seem to return properly... so I wouldn't necessarily use them as the benchmark.

Okay your guide helped with my login error when coming through the proxy, thanks for that. But now I'm getting this when searching for communities: ERROR HTTP request{http.method=GET http.scheme="https" http.host=lemmy.tld.com http.target=/api/v3/ws otel.kind="server" request_id=d794b124-3b67-46f2-85fa-08b3c39707bd http.status_code=101 otel.status_code="OK"}: lemmy_server::api_routes_websocket: couldnt_find_object: invalid query
- Do you have federation enabled? I think the checkbox in /admin isn't checked by default.
  
  yep, its enabled

12 comments