Nostr notes by Marvin Damschen

If that works well you could move to a bigger one like ...

2026-01-24T13:57:03Z

In reply to nevent1q…5kpj
_________________________

If that works well you could move to a bigger one like Qwen3-4B-Instruct-2507-GGUF (2507 stands for the July 2025 update here)
or Qwen3-8B you mentioned. You should not need to worry about the format (GGUF or not), Jan.ai should pick the right one.
HuggingFace can help to see what other people are using, e.g., trending LLMs max 6B:
https://huggingface.co/models?pipeline_tag=text-generation&num_parameters=min:0,max:6B&sort=trending

I have no experience with fine-tuning, yet, sorry.

Yes, it is a mess... Most of the time the structure is similar to ...

2026-01-24T13:51:51Z

In reply to nevent1q…gq5v
_________________________

Yes, it is a mess... Most of the time the structure is similar to this:
[model name]-[number of parameters in B]-[thinking/instruct]-[format]-[quantization]

Not all parts are always present. Qwen3-8B-GGUF would be the Qwen3 model with 8 billion parameters in GGUF format. 8B is big for CPU but quantizations reduce the size at the cost of output quality. Jan.ai should pick a quantization suitable for your hardware.

LFM2.5-1.2B-Instruct-GGUF is a recent small model you could start with. 🙂

sorry what did you mean with "lost"? I can easily see the ...

2026-01-23T20:51:12Z

In reply to nevent1q…80yx
_________________________

sorry what did you mean with "lost"? I can easily see the noise around LLMs being scary, though. 🙂

I have been curious about running LLMs locally for a while, not ...

2026-01-23T19:43:13Z

I have been curious about running LLMs locally for a while, not just for privacy reasons. I revisited this topic recently and would say it has become quite straight forward and usable, even on Intel GPUs.

So this is a blog post focusing on running gpt‑oss‑20b on the Intel Arc Pro B60:
https://marvin.damschen.net/post/intel-arc-llm/

#llm #selfhosting #privacy #intel