Software could be the easiest fix for hyperscalers' AI power squeeze, researchers say — data center demand is expected to rival Japan's electricity usage by 2030

Avid Amoeba@lemmy.ca · 5 days

Thats kinda how the whole Chinese competition started - optimization to make due with limited hw.

Also I just changed the inference engine I use for local models and Qwen 3.8 27B went from 50tps to 150tps. The new engine is optimized for my hw.

Lydia_K@lemmy.world · 5 days

What did you switch to?

Avid Amoeba@lemmy.ca · 5 days

Radiance. Was using the latest llama.cpp. This thread caught my eye as simple enough to try.

Agent641@lemmy.world · 5 days

What are the specs of the machine that runs it?

Avid Amoeba@lemmy.ca · 5 days

2x R9700, Ryzen 7700, 64GB RAM. The absolute best I’ve seen from llama.cpp was 60tps but the typical was 45-55tps. Radiance gave me 150tps on the first try. It can do parallel output while keeping 80-120tps with 2-4 streams. It can do more in parallel but I heven’t tested what the numbers look like. As far as I understand these numbers are possible on a single R9700 with the MXFP4 model. Haven’t tried it.