Remote View

1y ago

Lemmy World SysOp

Lemmy.World is looking for 4 new Systems operators to help with our growing community.

Volunteers will assist our existing systems team with monitoring and maintenance.

We’re ideally looking for chill folks that want to give back to their community and work on our back-end infrastructure. Must have 4+ years of professional experience working in systems administration. We are not looking for junior admins at this time. Please keep in mind that, while this is a volunteer gig, we would ask you to be able to help at least 5-10 hours a week. We also understand this is a hobby and that family and work comes first.

Applicants must be okay with providing their CV and/or LinkedIn profile AND sitting for a video interview. This is due to the sensitivity of the infrastructure you will have access to.

We are an international team that works from both North America EST time (-4) and Europe CEST (+2) so we would ask that candidates be flexible with their availability.

If you are in AEST (+10) or JST (+9) please let us know, as we are looking for at least one Sysadmin to help out during our overnight.

You may be asked to participate in an on-call pool. Please keep in mind that this is a round-robin style pool, so it's alright if you're busy as it will just move along the chain.

If you're interested and want to apply, click here.

102 comments

I have the experience, but not the energy nor passion as I am almost burned out already. I hope you find some awesome people.
- If it was a paying gig would you consider it? 5 to 10 hours a week, let's say 10. What kind of salary would you expect?
  Just curious.
  
  I also have the desired skill set and experience far surpassing what they’re asking for but not the time or energy to do this since my work already demands 60+ hours a week and on-call from me. Yes I’m American.
  To answer your pay question; around 4-500 would be the average pay for 10 hours this position in the working world. Since the fediverse instances have next to zero reliable income (donations can’t be counted as reliable) I understand this is a difficult if not impossible bill to pay. This is why they’re asking for volunteers whose work schedule is more sane and therefore have the energy and time to commit. I wish I was available to do so, maybe if my current job search is successful at finding something more chill.
  
  Knowing this is a volunteer project, I'd never request renumeration. If I were contracting with a large company, I guess I'd charge 300-500 per day. That's just based on quotes I get on LinkedIn, as I've never worked as a contractor. Also I couldn't have it interfere with my main job, where I'm also on call, so it would be lower priority.
  
  So I'm a systems engineer in the real world for an (almost) unicorn (current valuation might even have tossed us over that magic number). My salary is on the lower end of the spectrum but I'm happy with it because normally the work life balances is dandy. My total comp is well into 6 figures USD. Oh and I'm fully remote.
  Now, this is not something you can get out of highschool. I've been working with Linux for 10+ years, built (and maintained) entire AD forests, have a fairly deep understanding of networking and containerization, etc.
  Again. You don't start like me. You start getting a gig in front line help desk and answer questions. In your free time at work you learn (that's never going to stop). Eventually your outgrow help desk and move into some other role (and keep learning). The people who are successful in this field A) can always be learning, B) have a means to destress/avoid burnout and C) have customer service skills.
- oof its like you're either me or i'm you. hope you find your way past the burn out or out of it if you end up sinking into it. i'm going on like 3.5 years of battling it and there are better days and worse days, but i have no idea what else to even do. managing infra and writing code have been my entire career up to now.
  
  I'm about 1.5 years into it. Lost most passion for the job, but there are flashes of motivation here and there. Considering trying to move into full time development, but that would take a lot of effort. Tired of keeping up with the kubernetes ecosystem too.

In my opinion Lemmy.world should start selling a bit of merchandising (t-shirt and so on), just to add a little on the donation side.
BTW. the donation links are in the group info.

I read it as Lemmy World PsyOp at first and thought there’s some conspiracy bullshit happening on the instance. lol. Good luck on your search!
- Lemmy World psychological operations, the secret communist agenda of the administration team!!! leaked!
  
  i knew it!
- I did as well! After I read your post I went back up to review the post headline
- Glad to know I wasn't the only one, lol.

A CTO of 5 years with many more years of experience here. I would be really glad to help, but not in scenario where I have to prepare a CV for international readers and have no pay at all as this looks to me like a job application with no job.
Considering you are running on Digitalocean infrastructure, I am completely unsure why you would ever need Ansible and Terraform as it just adds complexity without certain benefit, especially if you mention Kubernetes which DO already provides with two clicks.
I'd personally suggest trying out ArgoCD for declarative clusters. With this thing, I've seen 2 companies maintained by a single DevOps engineer with no problems. Huge timesaver and makes everything transparent.
In case this process changes and becomes less corporate-y and more transparent, I'll be ready to apply. Hope you're going to find the right people! Long live Lemmy World!
- Ansible/Teraform are portable. I don't see it's usage as a failing, rather as avoiding DO lock in.
  Agreed with the rest though. This is quite the ask.
- I'm a big fan of Kubernetes, and for larger projects the flexibility and power it brings is unrivalled. But for smaller projects, assuming equal levels of competence, delivery teams using managed Kubernetes are almost universally later and have more issues than teams that use simpler solutions. Container-as-a-service solutions like GCP CloudRun or AWS FarGate help somewhat, but are not cheap for a given amount of compute time.
  Terraform (or IaC in general) absolutely has a place, because even if you use Kubernetes, most projects have more infrastructure to manage than just the cluster - at the very least, lemmy.world has a CloudFlare proxy to manage - and clicking buttons in a management portal is not a repeatable way of deploying that, or deploying the Kubernetes clusters themselves.
  Ansible also has a place, particularly if you're deploying onto bare metal. I wouldn't use it for new deployments unless I had bare metal to configure and maintain, but lemmy.world is deployed onto a bare metal server as I understand it. Plus, the most effective tooling is generally the one your team understands.
- IaC is the right move. It transfers complexity, it necessarilt doesn't add it. It makes your deployments reproduceable and automated.
  Which is a baseline to having highly available infrastructure. Not everyone will be familiar, or have the right mindset for that sort of DevOps.

Senior Network engineer with lots of experience in the field (servers + network), 15+ years if you need help let me know I'm happy to lend a hand.

I'm qualified, but 5-10 hours can mean a lot of different things.
Are you looking just for oncall/incident response, or are there more active reliability projects that you need help on?

4 applicants x 5-10 hours is .5 to 1 full time employees. Very generously speaking the ask here is for 100k/yr in free labor. The stringent interview process is going to be very limiting on potential candidates.
The experience isn't going to be a learning experience since you're looking for people that already know it all and I wouldn't even put it on a resume, it just advertises to employers you're ok being lowballed.
Perhaps this is a necessity for an instance of this size, but to me that seems to indicate that lemmy.world has reached the upper end of reasonable scalability, which given the workings of the fediverse would be fine.
- Yeah, I agree. I really doubt they're going to find anyone with these conditions

Out of curiosity, will you be able to weed out bad faith volunteers? I am sure there are a variety of interests that would be more than willing to pay a junior admin to be a Lemmy Sysop and it’s not like the candidate will volunteer that information.
- I think they'll be fine, the form asks for a CV + video call + lemmy user name and optional github profile.
  I'd be surprised if a bad faith candidate got through that. A probation period could work here, where their access is a bit restricted at first
  
  Hum, I hate to be that guy, but the good (experienced) bad actors have multiple profiles curated exactly for this reason. There's a reason why companies require proof of identity and a background security check.
- We are not looking for junior admins at this time.

5-10 hours a week isnt that crazy on a fun hobby project tbh. Some people can sink 30+ hours a week into their github repositories, streaming video games, browsing lemmy, or whatever else they're into.
It might not work for everyone, but I'm sure there is someone who is passionate, willing to learn, and wants to help this community thrive.
- Sounds like they're looking for someone doing the same in their 9-5 job though.

I threw my hat in the ring. Hope you all get some good people!

How about another approach?
There is no good reason for Lemmyworld to keep on growing to an extent that this kind of overhead is necessary. The idea of Lemmy is decentralization and not creating a new reddit instance. Close your registration, limit your amount of communities and let Lemmy grow in other directions.
- The thing is. Not to grow. LW wants to get stable.
  
  Understandable, but aren't growth and instability related in this case? There are many instances with capacity that are already run by capable people. Just spread the load (ahem) across the Lemmy verse and only handle as much as you can.
  But maybe I'm missing a point, I just think that this would be the best for Lemmy in the long run.

Definitely misread this as Lemmy World PsyOp
- MKULTRA lemmings
- Lol, same.

You may be asked to participate in an on-call pool
Unfortunately, this is where I noped out. But I ditch most paid positions where I can't avoid standby-time.

Threw my hat in the ring, I'm a senior devops engineer.
Don't have any Lemmy experience though. I have no desire to self host it, but I wouldn't mind being part of the team to maintain a large instance.

Get rid of TLS 1.0 and 1.1 - good luck 👍

I spent 4-5 years running a high traffic server using Linux, nginx, apache, php and whatever we did with Python, and would be glad to help. This was in 2010 though, so….
- I know, I meet all of the Soft skills, like four of the Systems knowledge, and maybe 0.5 of the Ideal Devops skills. But I have certifications, love 90s cartoons, and hate oatmeal raisin cookies - so I’m thinking I’m perfect.

Lol. I read the header as "Lemmy World PsyOp" and was like "well, that's disappointing," lol.

Very cool! Would be nice to have folks from different timezones to help with outages that currently occur mostly when everyone in the team is sleeping. Good luck!

Sysop... reminds of the good ol' BBS days. What a great time that was.
- I operated a few in the nineties. Sacred Grounds running pcboard BBS software. Kinda miss those days, it was great 😃
  
  It was certainly iconic. Felt like the glory days of computers, networking, and user groups. I always got that excited feeling like when you're a kid and anything is possible! It was before extreme specialization, and you literally could learn everything. It was a time when we felt like we had something special before the masses were even aware of it.
  Then ISPs popped up, and we ditched our modems and BBS's for IRC and Usenet. Still before the masses were aware. Still a special time.
  Then, eventually, corporations got wind of a new market to target, and the masses moved in. They used ridiculous names like portals, email chains, properties, social media, etc., and everything went to shit fast. It felt like something special was taken away from those of us who were at the beginning. It lost that special feeling and turned toxic in no time.
  Some embraced it, but more rejected it and would not dare join Facebook or MySpace or wherever corporate, ad-infested places the lemmings amassed. Instead, we created niche places a bit too technical for the sheeple to follow. We became rebels of conformity. And then, we became old and obsolete.
  It was a concurrent evolution and devolution as we were thrown toward the modern-day internet.
  Regardless of how it turned out, I am very thankful and glad I was a part of that early, amazing culture!

This is really tempting actually. Do you by chance need someone with skills in various storage technologies?
- Like empty vans, and storage lockers?
  
  And butts?
  
  Object storage, POSIX, and various flavors of RAID. I also dabble in RubberMaid bins from time to time as well. Don’t get me started on Docker containers either.
  
  My man!

Look forward to talking to you!

Do you get paid in Recognition Bucks?
- They cant currently afford like another 4k expenses for a dedicated sysop engineer. It will destroy their savings for the instance and lw would get shut down in 3 months. The server costs are high too.

I'm like a junior in experience but would love to help and especially learn how to help. I also have a lot of free time

I'm not a a professional in the area of question, but if you need moderators for the instance I would love to help in any capacity that I can.
I've been a moderator on a large forum, one of the biggest in my country, and would love to help whatever way I can. But sysop is not something I've got experience unfortunately.

I'm curious what the backend looks like based on your requirements here... Ansible is always a red flag for me that your servers are pets not cattle. Just maintain a golden image, especially since you mention kubernetes. And if you're using self managed kubernetes USE REMOTE ETCD. Trust me, it will save you so much time and drama.
I wouldn't be interested unless it's paid so I am just throwing that out there for y'all to consider.
- This made me laugh. Configuration management systems like ansible, chef, salt, and puppet only exist because people wanted to manage a large numbers of systems and keep them consistent and replaceable, i.e treat them like cattle instead of pets. They were born out of the pets vs cattle analogy.
  I realize containerization has taken that a step further but it's funny to hear someone talk about these tools like they're something archaic.
  
  Probably more accurate to say "mutable" vs "immutable".
  Harder to have pets in immutable if it's actually immutable.
  
  If you want to replace them why would you not use a golden image? The same thing goes for VMs, not only containerized. You can sit there and wait for ansible to run, or just have your image come up immediately with what you need. It makes it take longer with 0 benefit.
- Or just run everything in containers and use stock ubuntu or aws Linux or whatever.

I'm a noobie sysadmin so I don't have enough experience yet. Hope you guys find some people to help out!

No offense but you're asking for some crazy free labor. This would only make sense for college students or new grads trying to get experience. Why would I add another on-call shift to my existing career?
- Everyone involved is a volunteer. If it's not for you it's not for you. All good!
- It's a FOSS project, of course they're asking for volunteers. They're all volunteers themselves. It's not like the lemmy devs are making gobs of money from it.

If you need a janitor I can help but back end stuff is over my head.

I'd love to help out one day but right now my experience is just hosting random things for fun that I find on github

I totally hope you'll get enough admins to help out. Alas, I'm way to short on time to commit to this now. After 28y sysadmin and developing on Unix and Linux, I could use some project that is useful, but alas, no time to spare now. (Maybe later)

wow nice, hope you get some qualified folks! regrettably I dont have time or I'd toss my hat in the ring

On anti work this would be a request for 1/4th FTE with no benefits. I'm sure you'll get there best of the best.

102 comments