3w ago

Frustratingly bad at self hosting. Can someone help me access LLMs on my rig from my phone

tl-dr

-Can someone give me step by step instructions (ELI5) on how to get access to my LLM's on my rig from my phone?

Jan seems the easiest but I've tried with Ollama, librechat, etc.

.....

I've taken steps to secure my data and now I'm going the selfhosting route. I don't care to become a savant with the technical aspects of this stuff but even the basics are hard to grasp! I've been able to install a LLM provider on my rig (Ollama, Librechat, Jan, all of em) and I can successfully get models running on them. BUT what I would LOVE to do is access the LLM's on my rig from my phone while I'm within proximity. I've read that I can do that via wifi or LAN or something like that but I have had absolutely no luck. Jan seems the easiest because all you have to do is something with an API key but I can't even figure that out.

Any help?

43 comments

Self hosting IS hard, don't beat yourself too much because of it.. After all you're trying to serve services for yourself that are usually served by companies with thousands of employees.
A server requires knowledge, maintainance and time, it's okay to feel frustrated sometimes.

Why don't you ask your LLMs how to do it.
- lol I have! They all say the same similar thing but it’s just not working for me.
  
  How strange.
  
  Why do you want to set it up if your experience is bad results?
  
  They are trolling you. They are probably radical anti AI folk.

Just do like me - Install Ollama and OpenWebUI, install Termux on Android, connect through Termux with port forwarding.
ssh -L 0.0.0.0:3000:ServerIP_OnLAN:3000
And access OpenWebUI at http://127.0.0.1:3000/ on your phone browser. Or SSH forward the Ollama port to use the Ollama Android app. This requires you to be on the same LAN as the server. If you port forward SSH through your router, you can access it remotely through your public IP (If so, I'd recommend only allowing login through certs or have a rate limiter for SSH login attempts.
The shell command will then be ssh -L 0.0.0.0:3000:YourPublicIP:3000
But what are the chances that you run the LLM on a Linux machine and use an android to connect, like me, and not a windows machine and use an iPhone? You tell me. No specs posted...
- Oh! Also, I’m using windows on my PC. And my phone is an iPhone.
  I’m not using Linux yet, but that is in my todo list for the future! After I get more comfortable with some more basics of self hosting.
  
  Oh! Also, I’m using windows on my PC. And my phone is an iPhone.
  Okay, that's a starting place. So if this is Windows, and if you only care about access on the wireless network, then I suppose that it's probably easiest to just expose the stuff directly to other machines on the wireless network, rather than tunneling through SSH.
  You said that you have ollama running on the Windows PC. I'm not familiar with LibreChat, but it has a Web-based interface? Are you wanting to access that from a web browser on the phone?
- Bet, I’ll try that when I get home tonight. If I don’t have success can I message you directly ?
- ssh -L 0.0.0.0:3000:YOURPUBLICIP:3000
  If you can SSH to the LLM machine, I'd probably recommend ssh -L127.0.0.1:11434:127.0.0.1:11434 <remote hostname>. If for some reason you don't have or inadvertently bring down a firewall on your portable device, you don't want to be punching a tunnel from whatever can talk to your portable device to the LLM machine.
  (Using 11434 instead of 3000, as it looks like that's ollama's port.)
  EDIT: OP, it's going to be hard to give a reliable step-by-step, because I have no idea what your network looks like. So, for example, it's possible to have your wireless access point set up so that devices can't talk to each other at all. You might have some kind of firewall on your LLM machine, so that if they can talk to each other from the WAP's standpoint, the firewall will block traffic from your phone; you'd need to punch a hole in that. At least something (sshd for the example here, or ollama itself to the network) needs to be listening on a routable address. As DrDystopia points out, we don't even know what OS the LLM machine is running (Linux?) so giving any kind of step-by-step is going to be hard there.
  I have had absolutely no luck.
  Problem is, that doesn't say much. Like, doesn't say what you've seen.
  Do you know what the LAN IP address of your LLM machine is? Can you ping that IP address from Termux on your phone when both are on the same WiFi network ($ ping <ip-address>?) What OS is the LLM machine? If Linux, do you have sshd installed? It sounds like you do have ollama on it and that it's working if you use it from the LLM machine? When you said that it didn't work, what did you try and what errors or behavior did you see?
  
  3000 is the OpenWebUI port, never got it to work by using either 127.0.0.1 or localhost, only 0.0.0.0. Ollama's port 11434 on 127.x worked fine though.
  you don’t want to be punching a tunnel from whatever can talk to your portable device to the LLM machine.
  Fair point.

Ollama + open webui + tailscale/netbird
Open webui provides a fully functional docker with ollama, so just find the section that applies to you (amd, nvidia, etc) https://github.com/open-webui/open-webui?tab=readme-ov-file#quick-start-with-docker-
And on that host install netbird or Tailscale, install the same on your phone, in tailscale you need to enable magicdns but in netbird I think it provides dns by default.
Once the docker is running and both your server and phone are connected to the vpn (netbird or tailscale) you just type the dns of your server in your phone's browser (in netbird it would be "yourserver.netbird.cloud" and in tailscale it would be "yourserver.yourtsnet.ts.net")
Checkout networkchuck on youtube as he has a lot of simple tutorials.
- Bet. I believe what you mentioned is best for accessing my LLM no matter where I am in the world, correct? If so I will try this one after I try what the other person suggested.
  Thank you!

What OS is your server running? Do you have an Android phone or an iPhone?
In either case all you likely need to do is expose the port and then access your server by IP on that port with an appropriate client.
In Ollama you can expose the port to your local network by changing the bind address from 127.0.0.1 to 0.0.0.0
Regarding clients: on iOS you can use Enchanted or Apollo to connect to Ollama.
On Android there are likely comparable apps.
- Sever is my rig which is running windows. Phone is iPhone.
  Exposing the port is something I’ve tried to do in the past with no success! When you say, change the bind address, do I do that in the windows defender firewall in the inbound rules section?
  
  I believe you just need to set the env var OLLAMA_HOST to 0.0.0.0:11434 and then restart Ollama.
  
  When on your wifi, try navigating in your browser to your windows computer's address with a colon and the port 11434 at the end. Would look something like this:
  http://192.168.xx.xx:11434/
  If it works your browser will just load the text: Ollama is running
  From there you just need to figure out how you want to interact with it. I personally pair it with OpenWebUI for the web interface
  
  I believe you set env vars on Windows through System Properties -> Advanced -> Environment Variables.

Sounds like the issue is getting to the server, not the LLM server itself. If so, may want to look into running a reverse proxy, or if you want to access it remotely, tunnels: https://github.com/anderspitman/awesome-tunneling

At risk of getting more technical, ik_llama.cpp has a good built in webui:
https://github.com/ikawrakow/ik_llama.cpp/

Getting more technical, its also way better than ollama. You can run models way smarter than ollama can on the same hardware.
For reference, I'm running GLM-4 (667 GB of raw weights) on a single RTX 3090/Ryzen gaming rig, at reading speed, with pretty low quantization distortion.
And if you want a 'look this up on the internet for me' assistant (which you need for them to be truly useful), you need another docker project as well.
...That's just how LLM self hosting is now. It's simply too hardware intense and ad hoc to be easy and smart and cheap. You can indeed host a small 'default' LLM without much tinkering, but its going to be pretty dumb, and pretty slow on ollama defaults.
- Bet. Looking into that now. Thanks!
  I believe I have 11g of vram, so I should be good to run decent models from what I’ve been told by the other AIs.
  
  How much system RAM, and what kind? DDR5?
  ik doesn't have great documentation, so it'd be a lot easier for me to just point you places, heh.
  
  In case I miss your reply, assuming a 3080 + 64 GB of RAM, you want the IQ4_KSS (or IQ3_KS, for more RAM for tabs and stuff) version of this:
  https://huggingface.co/ubergarm/GLM-4.5-Air-GGUF
  Part of it will run on your GPU, part will live in system RAM, but ik_llama.cpp does the quantizations split and GPU offloading in a particularly efficient way for these kind of 'MoE' models. Follow the instructions on that page.
  If you 'only' have 32GB RAM or less, that's tricker, and the next question is what kind of speeds do you want. But it's probably best to wait a few days and see how Qwen3 80B looks when it comes out. Or just go with the IQ4_K version of this: https://huggingface.co/ubergarm/Qwen3-30B-A3B-Thinking-2507-GGUF
  And you don't strickly need the hyper optimization of ik_llama.cpp for a small model like Qwen3 30B. Something easier like lm studio or the llama.cpp docker image would be fine.
  Alternatively, you could try to squeeze Gemma 27B into that 11GB VRAM, but it would be tight.
- Ollama does have some features that make it easier to use for a first-time user, including:
  Calculating automatically how many layers can fit in VRAM and loading that many layers and splitting between main memory/CPU and VRAM/GPU. llama.cpp can't do that automatically yet.
  Automatically unloading the model from VRAM after a period of inactivity.
  I had an easier time setting up ollama than other stuff, and OP does apparently already have it set up.
  
  Yeah. But it also messes stuff up from the llama.cpp baseline, and hides or doesn't support some features/optimizations, and definitely doesn't support the more efficient iq_k quants of ik_llama.cpp and its specialzied MoE offloading.
  And that's not even getting into the various controversies around ollama (like broken GGUFs or indications they're going closed source in some form).
  ...It just depends on how much performance you want to squeeze out, and how much time you want to spend on the endeavor. Small LLMs are kinda marginal though, so IMO its important if you really want to try; otherwise one is probably better off spending a few bucks on an API that doesn't log requests.

I recommend technotim's docker AI stack.

43 comments