Marvin Damschen

Website: https://marvin.damschen.net
LinkedIn: https://www.linkedin.com/in/damschen/

Public Key

npub12xugn8qwwvwm8jte5fyea6xjxka9v7g6q22h22ulsfq2f9ajskrshjlcya

NIP-05 Address

Profile Code

nprofile1qqs9rwyfns88x8dne9u6yjv7arfrtwjk0ydq99t49w0cys9yj7egtpcprpmhxue69uhhyetvv9ujuumwdae8gtnnda3kjctvqy28wumn8ghj7un9d3shjtnyv9kh2uewd9hsew8e4p

Publishing to

relay.snort.social

relay.damus.io

nos.lol

Author Public Key

npub12xugn8qwwvwm8jte5fyea6xjxka9v7g6q22h22ulsfq2f9ajskrshjlcya

Show more details

Last Notes

2026-01-24 13:57:03 UTC

- reply

If that works well you could move to a bigger one like Qwen3-4B-Instruct-2507-GGUF (2507 stands for the July 2025 update here)
or Qwen3-8B you mentioned. You should not need to worry about the format (GGUF or not), Jan.ai should pick the right one.
HuggingFace can help to see what other people are using, e.g., trending LLMs max 6B:
https://huggingface.co/models?pipeline_tag=text-generation&num_parameters=min:0,max:6B&sort=trending
I have no experience with fine-tuning, yet, sorry.

2026-01-24 13:51:51 UTC

- reply

Yes, it is a mess... Most of the time the structure is similar to this:
[model name]-[number of parameters in B]-[thinking/instruct]-[format]-[quantization]
Not all parts are always present. Qwen3-8B-GGUF would be the Qwen3 model with 8 billion parameters in GGUF format. 8B is big for CPU but quantizations reduce the size at the cost of output quality. Jan.ai should pick a quantization suitable for your hardware.
LFM2.5-1.2B-Instruct-GGUF is a recent small model you could start with. 🙂

2026-01-23 20:51:12 UTC

- reply

sorry what did you mean with "lost"? I can easily see the noise around LLMs being scary, though. 🙂

2026-01-23 19:43:13 UTC

I have been curious about running LLMs locally for a while, not just for privacy reasons. I revisited this topic recently and would say it has become quite straight forward and usable, even on Intel GPUs.
So this is a blog post focusing on running gpt‑oss‑20b on the Intel Arc Pro B60:
https://marvin.damschen.net/post/intel-arc-llm/
#llm #selfhosting #privacy #intel