Why Size Matters in LLMs?

How the Number of Parameters Influences LLM Performance

Friendly llamas (guanacos). Photo by Miranda Salzgeber on Unsplash

Andrej Karpathy describes in a recent video LLMs as lossy compressors, which helps explain why having more parameters is so important and why Small Language Models (SLMs) can't quite match their capabilities, even with better pre-training data.