- cross-posted to:
- technology@lemmy.ml
- cross-posted to:
- technology@lemmy.ml
*Western AI companies are terrified
Anyway, what do you all think of the news?
Video is 8 minutes and 30 seconds long if you’re interested.
Is it true that the DeepSeek models and that other AI model (I forgot the name already lol) can be run locally? How do they compete with Claude and OpenAI?
Also, what about that other AI thing, OpenClaw?
I am interested in AI but still very much skeptical.
I have only used Duck AI sometimes but try to avoid it.
You must log in or # to comment.
The fact the majority of people on REDDIT tech subs have become straight up pro-China is truly telling in how dogshit and godawful the US is going about AI compared to China.
You should’ve said:
“Reddit TECH subs” instead
See, the “TECH” part is more astonishing to me
I mean, it’s astonishing because these aren’t just TECH subs, they’re also TECH subs on fucking REDDIT!!
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Kimi K3 is truly impressive, latest Qwen and GLM5.2 is also worth noting. The US has lost pretty much its entire moat to open weight models.
The only way I see the US companies staying relevant long term is through agressive anti-competetive actions and subsidies from the government.
Why use the term “open-weight”?
Why not say “open-source”?
Is there a difference?
According to the Open Source Initiative:
Open Weights differ significantly from Open Source AI because they do not include:
-
Training code – The scripts or frameworks used to create and curate the training dataset.
-
Training dataset – The full dataset used for training, when legally possible. As an alternative, when distribution of the training dataset is not legally possible,
- Comprehensive data transparency – Full details about dataset composition, such as source domains, cleaning methods, or balancing techniques.
coolio
-
I think its because the training data is not accessible, but you should fact check my claim
interesting, interesting…
ChatGPT and Claude are programs that are not publicly available. They run on privately owned, powerful computers (servers), and your only access to them is through those servers. For better-performing models, you have to pay a subscription. They also cost a lot for the companies to run because they waste a lot of energy.
Chinese AIs like deepseek or qwen or glm, on the other hand, are very efficient while performing pretty well, which means they’re cheaper to run. At the same time, they’re publicly available, so anyone can download the models and run them on their own computer.
How much VRAM or RAM do you need?
you need terabytes of vram if you want to run a full foundational model… otherwise you’re running a quantized version that introduces loss of information and accuracy… some models can be quantized low enough to run on 8gbof vram, and many smaller models can run fine on 16, 24, or 32gb of VRAM
Depends on the model, but maybe it’s around 16 gigs of vram, and a good gpu, but you’d need a lot more resources to make it perform as good as the one in the web interface, i tried with my nvidia gtx 1660 ti 6 GB vram and the models that were available to me weren’t really clever
You need 128GB of VRAM for aggressive quantization of DeepSeek Flash. 16GB might be enough for aggressive quantization of smaller models. In theory you can run Qwen3.6 27B in 2-bit quantization (or something like https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf ) and have some space for context but it’s a toy. You need at least 24GB of VRAM to use reasonable quant of Qwen3.6 27B (or 32GB of shared VRAM to handle Qwen3.6 35BA3B which is faster but takes more VRAM than 27B) in actual tasks.
I’ve had great luck running qwen2.5-coder:7b on my work laptop. It’s great for a wide range of dev and operational tasks with opencode.
So basically needs more power, huh…
Based on the other comments a lot more, the whole ai crash isn’t really about regular consumers but big companies that prefer to host their own ai instead of paying open ai and anthropic
Honestly, the development of AI (not it happening but HOW it happened) has been very interesting and somewhat hopeful to me.
This is because the companies that control this technology don’t have full control and basically are fighting each other for it.
Really fascinating ngl
DeepSeek V4 flash is feasiblish to run it personally. A minipc costing $3000 can do a working quantization, two such computers can run what is in practice a full model. You could even run a (weak but working) quantization of GLM5.2 in this configuration.
What you get running models this way is control over what exactly runs (you never know how aggresively quantized are models running with API, for example) and possibility of running anything that fits (e.g. models with refusals removed with methods like heretic).
What you lose is that you need to buy special hardware for thousands of dollars (which might or might not age well) and loading prompt (including the history of conversation) can be really slow compared to what bigger machines that host models for many users can achieve. Loading past context of 100,000s tokens takes actual minutes while using the same model through API will feel instant.
It shows how important is for people to own the datacenters. Running model on bigger hardware for multiple people is much more efficient than running it on personal hardware.
It shows how important is for people to own the datacenters. Running model on bigger hardware for multiple people is much more efficient than running it on personal hardware.
Which also is why the AI companies are panicking a little bit. They don’t necessarily target private consumer anyways, but are trying to expand into the business-licensing space. Thats why they are competing so hard to create the most complete ecosystem of tools, i.e. coding, legal work, picture+movie, defense.
But no reasonably lead company is going to waste $9999999 billions on tokens for ever more expensive models when you can get similar results by expanding your serverhall with a few m^2.
$3000 in today’s prices or of six months ago?
Current prices, I put example under the neighbour comment.
A minipc costing $3000 can do a working quantization, two such computers can run what is in practice a full model…
Would you able to please spec such a PC?
https://www.bosgamepc.com/products/bosgame-m5-ai-mini-desktop-ryzen-ai-max-395 it’s still below $3K. It’s not the optimal Strix Halo variant if you want to cluster a few of them though (you still can use thunderbolt in this one to connect it to another one).
they’d be pretty slow though… lpddr has really poor bandwidth compared to what you get in graphics cards
Thank you! I had no idea!
(I was curious - wanted to have an idea what would be considered feasible these days but personally would rather pay for the tokens if needed as you suggested. I am however going to use the specs of the models you suggested to learn more about how AI works in such a machine)
Interesting…
Can you tweak or change the AI completely?
Or even somehow notifies revamp it into your own AI completely?
Maybe build off of it somehow and improve it maybe?
If you mean using different models, yes. If you mean finetuning model weights… No, finetuning LLMs is hard in general and Strix Halo (device that I had in mind that costs 3K$ for 128GB of VRAM) is also kinda bad for it now due to having an AMD (and not really strong when it comes to compute power) GPU.
In practice, people who try finetuning on top of newest models usually cripple them. There are plenty of finetunes of Qwen 3.6 on huggingface trying to distill stronger models and they are just bad.
Heretic, the method for removing refusals I mentioned, also requires more VRAM to run it than it’s required for inference.
“AMD GPU”
oof
About how strong these models are compared to claude - DeepSeek flash is much better than Haiku, GLM 5.2 is comparable at level of Opus, definitely better than Sonnet. In a quantization you could run with 256GB of VRAM, it would still be better than Sonnet I think.
You can run these models with API and save a lot of money compared to Claude too if you swap: Haiku -> DeepSeek Flash, Sonnet -> DeepSeek Pro, Opus -> GLM 5.2, Fable -> Kimi K3
I found a YouTube link in your post. Here are links to the same video on alternative frontends that protect your privacy:
