Theory Thursday: How to Build Your Own Specialised LLM
Art by @basilonmypizza: https://lnkd.in/eF8FkWzN - https://basilhefti.ch/
Most of us use ready-made LLMs - they’re reliable, versatile and tested. But sometimes, you need to train your own. Andrej Karpathy’s llm.c shows how to train a small model on sequences, like lines from Shakespeare, by turning each word into a number and learning which comes next.
Despite the name language models, LLMs can be trained for many kinds of sequences. Delphi-2M applies the same idea to healthcare by treating each medical event - a diagnosis, a treatment, a lifestyle marker - as a token in a patient’s timeline. A transformer learns to predict the next event just as it would the next word. Because it only sees past events, it captures usable patterns (this is what makes it a causal transformer).
So, we have lego blocks we can reuse to apply this pattern in other domains, such as customer analytics (CRM). What are these blocks?
First, the “vocabulary block”: define the state space for the topic at hand. For CRM, this might include contract_signed, invoice_sent, payment_late, complaint, churn, and upgrade. The model learns how these states connect - some naturally follow each other, others almost never do.
Next comes the “embedding block”. Here, we convert the states into numbers. It sounds technical, but we simply count through the vocabulary. The event chain contract_signed → invoice_sent → complaint → upgrade becomes [0, 1, 3, 5]. These pieces now fit into a numerical form the model can work with.
In the same spirit, we store date differences. The above sequence might be [0, 30, 2, 90] days apart. Another set of pieces added.
Then comes the “transformer block”. We can use Andrej Karpathy’s llm.c or the Delphi-2M transformer to train the model (both will require adjustments).
Putting these pieces together, we get a trained model. For new data, it can generate the next events. Staying with the CRM example, you can ask what happens after contract_signed → invoice_sent → ?. If the next predicted event is churn, you can try and act before it happens. In the Delphi-2M case, the model can forecast the likelihood and timing of diseases and was validated across countries, showing strong generalisation.
Causal transformers turn timelines into prediction engines. Where would you use yours? 🚀
Art: https://lnkd.in/eF8FkWzN https://basilhefti.ch/
