Marvin Damschen

Website: https://marvin.damschen.net
LinkedIn: https://www.linkedin.com/in/damschen/

Last Notes

2026-01-24 13:57:03 UTC

- reply

If that works well you could move to a bigger one like Qwen3-4B-Instruct-2507-GGUF (2507 stands for the July 2025 update here)
or Qwen3-8B you mentioned. You should not need to worry about the format (GGUF or not), Jan.ai should pick the right one.
HuggingFace can help to see what other people are using, e.g., trending LLMs max 6B:
https://huggingface.co/models?pipeline_tag=text-generation&num_parameters=min:0,max:6B&sort=trending
I have no experience with fine-tuning, yet, sorry.

2026-01-24 13:51:51 UTC

- reply

Yes, it is a mess... Most of the time the structure is similar to this:
[model name]-[number of parameters in B]-[thinking/instruct]-[format]-[quantization]
Not all parts are always present. Qwen3-8B-GGUF would be the Qwen3 model with 8 billion parameters in GGUF format. 8B is big for CPU but quantizations reduce the size at the cost of output quality. Jan.ai should pick a quantization suitable for your hardware.
LFM2.5-1.2B-Instruct-GGUF is a recent small model you could start with. 🙂

2026-01-23 20:51:12 UTC

- reply

sorry what did you mean with "lost"? I can easily see the noise around LLMs being scary, though. 🙂

2026-01-23 19:43:13 UTC

I have been curious about running LLMs locally for a while, not just for privacy reasons. I revisited this topic recently and would say it has become quite straight forward and usable, even on Intel GPUs.
So this is a blog post focusing on running gpt‑oss‑20b on the Intel Arc Pro B60:
https://marvin.damschen.net/post/intel-arc-llm/
#llm #selfhosting #privacy #intel