https://huggingface.co/collections/inclusionAI/ling-31

https://huggingface.co/collections/inclusionAI/ling-30

LM Studio is good to start with, I do wish it was open source. Internally it just uses llama.cpp except with a GUI for configuration so it’s a bit easier to use. Unsloth Studio is similar to LM Studio except open source, but I haven’t tried it yet.

I actually have LM Studio on a Linux live boot USB along with Ling 3.0 Tiny, and it’s preconfigured to run with the Vulkan backend (no CUDA in a live boot, and might plug it into an Intel or AMD graphics). I haven’t needed to use it yet but I figure it could be nice in a pinch away from home.

try the Strata engine instead of llama.cpp

https://github.com/Niko1221/Strata/blob/main/docs/MODELS.md#pick-by-memory

You should be able to run Qwen 3.8 Flash Next decently

With Strata, things have gotten more flexible. Similar optimizations are coming to llama.cpp eventually.

https://github.com/Niko1221/Strata/blob/main/docs/MODELS.md#pick-by-memory

It says Qwen 3.8 Flash Next can run with just 8GB VRAM and 96GB RAM for a good quant of it. This is a really good model, and the new version 4 should be coming in the next month or so, probably the same size but smarter.

DeepSeek v4.1 Flash is probably even smarter, says it needs 12+GB VRAM and 64GB of RAM. For decent speed I think you’ll want at least 32GB VRAM and 256GB RAM for this one. 96GB VRAM should be very good.

You’re gonna have big diminishing returns chasing for even bigger models, partly because Strata doesn’t support anything bigger and llama.cpp is currently not nearly as optimized. There might be an inference engine I don’t know about for the biggest models.

what are the specs? Try Ling 3.0 Tiny, that’s probably the fastest decent model, it should be both smarter and faster than Qwen 3.5 4b

MoE models are flexible, see my guide for llama.cpp https://lemmus.org/post/24235317

Now there’s also --moe-cache-mib that could beat --n-cpu-moe depending on your exact hardware. I’ll need to write an updated guide.

And now there’s Strata and PolyStrata which get even better performance than that.

I just use llama.cpp with Zoo Code, the VSCode extension

join us !localllama@sh.itjust.works

It’s a MoE model so it’s much more flexible than a dense model, see my guide for llama.cpp https://lemmus.org/post/24235317

Now there’s also --moe-cache-mib that could beat --n-cpu-moe depending on your exact hardware. I’ll need to write an updated guide.

And now there’s Strata and PolyStrata which get even better performance than that.

yea I would guess LoRA training uses bf16

I didn’t write down my exact tokens/s in llama.cpp before installing the unlocker, but the numbers I got after were roughly the same as I remembered, which also makes sense since I use quantized models that afaik use integer tensor ops.

token generation is bottlenecked by memory speed not compute power. your prompt processing/prefill speeds might’ve gone up

https://piefed.social/comment/13343280

PieFed has the concept of silencing another instance. A silenced instance is not defederated from but their posts do not show in the Popular or All feeds and their communities are not shown during onboarding. Their communities can still be found in the communities list (if you unblock the instance first) and joined in the normal way. Once joined, posts in there show up in the Subscribed feed of the joiner as usual.

Since I added silencing functionality I feel like there’s almost no need for (more) defederation. I can offer protection to newbies who don’t yet know which places to avoid and they can explore the dark corners when they’re ready. It’s a low-drama way to put a bit of distance between groups that rub each other the wrong way.

Yeah default blocks or silencing or something like that

PieFed allows admins different levels of blocking instances, they can do defederation but there are levels softer than that

oh yea feddit blocks shitjustworks https://feddit.online/instance/sh.itjust.works/posts

maybe @xSikes@feddit.online could ask their admins to convert the defederation to a “silenced” or whatever that feature is on PieFed

32GB RAM, RTX 2080 8GB VRAM, I use Tiel Coder 35b-a3b (it’s a fine tune of Qwen 3.6) with Zoo Code

I see the last activity was 2 months ago.

that means no one from your instance is subscribed, if you subscribe then you’ll get new posts

you can see that there are new posts here https://sh.itjust.works/c/localllama

Well it’s a MoE, so at least in llama.cpp you can offload some experts and use the new MoE caching --moe-cache-mib and you’ll probably still get at least 90% of the speed

Unsloth has their Q4 quant at 506 GB 💀

Active parameters aren’t that high, you’ll need KV cache in VRAM too, so maybe it would run in 64GB VRAM

I’m excited to test it. Maybe next time I need to do some coding, unless maybe Qwen 4 is out by then!

Yeah someone said they’re way better at 3d modeling, but how do we know the closed models aren’t actually 5 separate models in a trenchcoat? Behind the scenes they could be making tool calls to each other, or they could have a small router model that decides which model to send each request to. They could literally have 100 different fine-tunes to choose from dynamically and we would still think it’s all a single model that’s just really good at everything. (100 is unlikely, just an example)