Theory Thursday: A New Phase of Model Development

Art by @basilonmypizza: https://lnkd.in/eF8FkWzN - https://basilhefti.ch/

A recent experiment in “autoresearch” caught my attention: parts of the ML research loop can be automated.

Karpathy’s setup optimizes the validation loss. What is this? Picture training as learning from text by predicting a blacked out word (GPTs are trained to predict the next token, but the idea is the same). Because you have the full text, validation is easy: has the model guessed the blacked out word, yes or no? The better the model gets, the better the validation metric becomes.

And because validation is done on text not used for training, the loss shows how well the model generalizes (rather than just memorizes).

By the way: the optimisation can be compared to a skier searching for a good, fun descent. At each step, you follow the steepest local direction. In model training, the loss landscape is the mountain. And over many steps, the training “skis” toward weights that perform well.

So, how does autoresearch work? Each iteration runs a short training run, validates the resulting model, and keeps code changes that lead to a lower validation loss. This works well. In one run over 2 days, Andrej saw ~700 experiments and 20 relevant (and sometimes surprising) improvements.

The new: the system is optimizing the training process (the day-to-day use of GPT optimizes prompts and outputs).

My read: over time, this idea will lower the barrier and make model development accessible beyond a small group of experts. And coding and training are merging into one loop: systems that write code, run experiments, and refine it.

And the future is already here. For for example in everyday data work, from churn prediction and data quality detection to understanding how data connects across systems - in short, wherever you have to “ski” through parameter space.

Which LLM have you always wanted to train for yourself?

— Art: https://lnkd.in/eF8FkWzN https://basilhefti.ch/

— on connecting data: https://lnkd.in/ehG_SzqB