Technology

59578 readers

3420 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 1 year ago

MODERATORS

198

Microsoft sets 16GB default for RAM for AI PCs – machines will also need 40 TOPS of AI compute: Report (www.tomshardware.com)

submitted 10 months ago by throws_lemy@lemmy.nz to c/technology@lemmy.world

69 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] Secret300@sh.itjust.works 6 points 10 months ago (1 children)

Personally I really want it to but only locally run AI like lamma or whatever it's called

[–] evranch@lemmy.ca 4 points 10 months ago (1 children)

Do it, it's easy and fun and you'll learn about the actual capabilities of the tech. Started a week ago and I'm a convert on the utility of local AI. Got to go back to Reddit for it but r/localllama has tons of good info. You can actually run useful models at a conversational pace.

This whole thread is silly because VRAM is what you need, I'm running some pretty good coding and general knowledge models in a 12GB Radeon. Almost none of my 32GB system ram is used lol either Microsoft is out of touch or hiding an amazing new algorithm

Running in system ram works but the processing is painfully slow on the regular CPU, over 10x slower

[–] Secret300@sh.itjust.works 1 points 10 months ago (2 children)

Just downloaded gpt4all and lm studio or whatever. I'm learning slowly but there's a lot of jargon. I only have the 4GB Rx 5500 and I'm not sure how to get it to run on my GPU. I think I really just need to upgrade my PC tho. I have 16GB of ram but an i5-6500. Shit be slow

[–] evranch@lemmy.ca 2 points 10 months ago

Start off with the Tinyllama model, it's under 1gb. It will even run on raspberry pi so on real PCs it rips even on CPU. You need a "quantized" model, they are distributed as GGUF files.

I would recommend 5 bit quantized. The less bits, the stupider to put it simply, and Tinyllama is already pretty stupid. But it's still impressive for what it is, and you can learn the jargon which is the hard part.

Fastest software to run the model on is llama.cpp which is a rewrite from python to C++. Use -ngl to offload layers from cpu to GPU.

Not sure what system you're using, most AI development is done on Linux so if you're on Windows I can't guarantee anything will work.

Working right now on making a voice assistant for my house that can read all my MQTT data and give status reports, it's neat when you get it running. Fun to tweak it with prompts and see what it can do. Tinyllama can't seem to reliably handle MQTT and JSON but slightly smarter models can with ease.

[–] evranch@lemmy.ca 1 points 10 months ago* (last edited 10 months ago)

Ok, I walked over to my PC to give you a working command line for llama.cpp. You need to make sure it is compiled with support for hipBLAS / ROCm which is the equivalent AMD framework to CUDA, if you want it to run on your GPU.

./main -ngl 24 -m models/tinyllama-1.1b-chat-v1.0.Q5_K_M.gguf --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -i -ins

This will put it into interactive mode so you can try to chat with it. Running on my GPU it cranks out almost 160 tokens per second, which is way faster than anyone can type. On CPU (-ngl 0) it will make 90 which is still fast. TinyLlama is not a great chatter and should be treated more as a prediction or answer engine. i.e:

>Write a paragraph about borscht.
Borscht is a traditional Russian soup made with beetroot, potatoes, and a variety of spices. It is often served during the winter months in Russia, Ukraine, Belarus, and other Soviet-style countries. Borscht is similar to borscht in Poland, but has different ingredients and a slightly different preparation method. In Poland, beets are boiled until they become tender, then blended with potatoes and vegetable broth to create the soup. In Russia, beetroot is removed from the pot before cooking and replaced with other vegetables such as carrots, celery, and onions. The resulting mixture is then simmered until it is thickened, creating a hearty and flavorful soup. Borscht is usually served cold or at room temperature, and can be accompanied by sour cream, slices of crusty bread, or grilled meats such as kebabs.

It does know a surprising amount, considering it would fit on a CDROM