In fairness, unless you have about 800GB of VRAM/HBM you’re not running true Deepseek yet. The smaller models are Llama or Qwen distilled from Deepseek R1.
I’m really hoping Deepseek releases smaller models that I can fit on a 16GB GPU and try at home.
Well, honestly: I have this kind of computational power at my university, and we are in dire need of a locally hosted LLM for a project, so at least for me as a researcher, its really really cool to have that.
Lucky you! I need to check my university’s current GPU power but sadly my thesis won’t be needing that kind of horsepower, so I won’t be able to give it a try unless I pay AWS or someone else for it on my own dime.
Sure but i can run the decensored quants of those distils on my pc, I dont need to even open the article to know that openai isnt going to allow me to do that and so isnt really relevant.
Qwen 2.5 is already amazing for a 14B, so I don’t see how deepseek can improve that much with a new base model, even if they continue train it.
Perhaps we need to meet in the middle, and have quad channel APUs like Strix Halo become more common, and maybe release like 40-80GB MoE models. Perhaps bitnet ones?
Or design them for asynchronous inference.
I just don’t see how 20B-ish models can perform like one orders of magnitude bigger without a paradigm shift.
nVidia’s new Digits workstation, while expensive from a consumer standpoint, should be a great tool for local inferencing research. $3000 for 128GB isn’t a crazy amount for a university or other researcher to spend, especially when you look at the price of the 5090.
Why would you buy a single use behemoth when you can buy a strix halo 128GB that can work as an actual tablet/laptop and have all the functionality of the behemoth?! while supporting decades of legacy x86 software. Truly wondering why anyone would buy that NVIDIA thing other than pure ignorance and marketing says NV is the AI company.
In fairness, unless you have about 800GB of VRAM/HBM you’re not running true Deepseek yet. The smaller models are Llama or Qwen distilled from Deepseek R1.
I’m really hoping Deepseek releases smaller models that I can fit on a 16GB GPU and try at home.
Well, honestly: I have this kind of computational power at my university, and we are in dire need of a locally hosted LLM for a project, so at least for me as a researcher, its really really cool to have that.
Lucky you! I need to check my university’s current GPU power but sadly my thesis won’t be needing that kind of horsepower, so I won’t be able to give it a try unless I pay AWS or someone else for it on my own dime.
Sure but i can run the decensored quants of those distils on my pc, I dont need to even open the article to know that openai isnt going to allow me to do that and so isnt really relevant.
Qwen 2.5 is already amazing for a 14B, so I don’t see how deepseek can improve that much with a new base model, even if they continue train it.
Perhaps we need to meet in the middle, and have quad channel APUs like Strix Halo become more common, and maybe release like 40-80GB MoE models. Perhaps bitnet ones?
Or design them for asynchronous inference.
I just don’t see how 20B-ish models can perform like one orders of magnitude bigger without a paradigm shift.
nVidia’s new Digits workstation, while expensive from a consumer standpoint, should be a great tool for local inferencing research. $3000 for 128GB isn’t a crazy amount for a university or other researcher to spend, especially when you look at the price of the 5090.
Why would you buy a single use behemoth when you can buy a strix halo 128GB that can work as an actual tablet/laptop and have all the functionality of the behemoth?! while supporting decades of legacy x86 software. Truly wondering why anyone would buy that NVIDIA thing other than pure ignorance and marketing says NV is the AI company.
Dense models that would fit in 100-ish GB like mistral large would be really slow on that box, and there isn’t a SOTA MoE for that size yet.
So, unless you need tons of batching/parallel requests, its… kinda neither here nor there?
As someone else said, the calculus changes with cheaper Strix Halo boxes (assuming those mini PCs are under $3K).