Writing
Notes
Models, memory, and the boring infrastructure that makes them useful.
MoE is probably good enough.
I spent a lot of time trying to make sparsity work in dense models. Mixture-of-experts leaves some intelligence on the table. For almost everything I care about shipping, it is enough.
Memory is All You Need.
Chat products grew a memory of you. Taken seriously, that pattern is how you get business intelligence that is actually useful.
What should AI models actually do?
Frontier models keep getting bigger so they can hold more of the world in their weights. I am not convinced that is the job.