this post was submitted on 01 Dec 2024
48 points (88.7% liked)

Futurology

1854 readers
48 users here now

founded 1 year ago
MODERATORS
voidx
Lugh
Espiritdescali
AwesomeLowlander
48
Multiple LLMs voting together on content validation catch each other’s mistakes to achieve 95.6% accuracy. (arxiv.org)
submitted 3 weeks ago by Lugh to c/futurology
27 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] DavidGarcia@feddit.nl 6 points 3 weeks ago (2 children)

For the small ones, with GPUs a couple hundred watts when generating. For the large ones, somewhere between 10 to 100 times that.

With specialty hardware maybe 10x less.

[–] pennomi@lemmy.world 3 points 3 weeks ago (2 children)

A lot of the smaller LLMs don’t require GPU at all - they run just fine on a normal consumer CPU.

[–] copygirl@lemmy.blahaj.zone 3 points 3 weeks ago (1 children)

Wouldn't running on a CPU (while possible) make it less energy efficient, though?

[–] pennomi@lemmy.world 3 points 3 weeks ago

It depends. A lot of LLMs are memory-constrained. If you’re constantly thrashing the GPU memory it can be both slower and less efficient.

[–] DavidGarcia@feddit.nl 1 points 3 weeks ago

yeah but 10x slower, at speeds that just don't work for many use cases. When you compare energy consumption per token, there isn't much difference.

[–] kippinitreal@lemmy.world 2 points 3 weeks ago

Good god. Thanks for the info.