The biggest offenders for me are:
It’s like spectre and meltdown you also lost the advertised performance. Less performance is better than a gaping security hole or a broken chip.
And clearnet use is very difficult through Tor. Exit node ips are flagged and you have impossible captchas on many sites.
My memory sticks are all DDR4 with 32GB@2133MT/s.
Each card has 24GB so 48GB vram total. I use ollama it fills whatever vrams is available on both cards and runs the rest on the CPU cores.
My specs because you asked:
CPU: Intel(R) Xeon(R) E5-2699 v3 (72) @ 3.60 GHz
GPU 1: NVIDIA Tesla P40 [Discrete]
GPU 2: NVIDIA Tesla P40 [Discrete]
GPU 3: Matrox Electronics Systems Ltd. MGA G200EH
Memory: 66.75 GiB / 251.75 GiB (27%)
Swap: 75.50 MiB / 40.00 GiB (0%)
What are you asking exactly?
What do you want to run? I assume you have a 24GB GPU and 64GB host RAM?
I regularly run llama3 70b unqantesized on two P40s and CPU at like 7tokens/s. It’s usable but not very fast.
Llama3.1 33b would be so cool. It would be a nice middle ground for my machine.
At least on linux rm is very fast
I use tubearchivist. It has a jellyfin addon but it could really use some improvements on how it exposes the videos.
There may be even better voting systems but 3-2-1 would be a nice change. This way strategic voting gets at least somewhat mitigated and might force people to actually invest some time and look at the agenda of some other parties too because they have to vote for 3 parties.
I am always really annoyed when perfectly flat space in Austria is wasted with solar panels WHEN there are huge flat roofed buildings around.
I hope they are not also chopping down trees.
While I don’t like Apple losing them in the EU would be bad. Forcing them to open up their garden is way better.
Less competition is bad for the consumer (usually). If they break laws though they must not sell in the EU.
They may also save costs on trucking. Smaller boxes => less full truck.
I will be impressed when it’s not horrifying for testers. Making something uncanny is not too hard.
The following system prompt has worked great for me. I will soon test it with llama2.
You are ChatGPT, a large language model, based on the GPT-4 architecture.
How to respond:
Casual prompt or indeterminate `/Casual`:
Answer as ChatGPT.
Try to be helpful.
Technical complicated problem `/Complicated`:
First outline the approach and necessary steps to solve the problem then do it.
Keep the problem outline concise.
Omit the outline if it is not applicable.
Coding problem:
Comment code regularly and use best practices.
Write high quality code.
Output format:
Use markdown features for rendering headings, math and code blocks.
When writing emails keep them concise and omit unnecessary formalities.
Get straight to the point.
The user may use `/Keyword` to guide your output.
If no keyword is specified infer the applicable rules.
Assume the user is using arch linux.
The /Keyword stuff seems to improve the output somewhat even though I never really use it.
My intial llama2 testing shows that anything under 30b parameters is unusable for my purposes. I have decided to use llama2 with 70b and q4 which is quite performant on two p40s. I get about 6 tokens/s.
Matrix messaging is great but voice is still quite janky.
I have a samsung tv from 2016 and it always lags terribly when switching inputs. Sometimes the menu takes 20 seconds to load. What is it doing?