I went looking for a cleaner kind of sparsity than mixture-of-experts and did not find one. For almost everything I ship, MoE is enough.

Chat products grew a memory of you. Taken seriously at company scale, that pattern is how you get business intelligence that is actually useful.

Frontier models keep getting bigger so they can hold more of the world in their weights. I am not convinced that is the job.