MoE is probably good enough.

I wanted a denser kind of sparsity. Not "this token sees 2 of 64 experts," but a model that could stay fully capable without paying full price on every forward pass. There is a romantic version of this where every parameter is always a little bit awake, just cheaply, and you do not have to gamble on a router.

I did not find a version of that story I would bet a training run on.

Mixture-of-experts has obvious sins. The router is a single point of stupidity. Experts collapse. Load balancing turns into a second loss function you babysit. Some tokens get the B-team and you only notice in evals you did not think to run. If you stare at it long enough, you can convince yourself that a dense model of the same activated size would have been cleaner, more even, more "actually intelligent."

Maybe. I also ship things.

A production model has to be smart enough, fast enough, and cheap enough that you can call it from a phone, or from a batch job, or from a tool loop that might run 40 times before breakfast. MoE is a slightly ugly way to buy capacity you do not activate. That is a very old systems trick. We have been doing it with caches, with indexes, with anycast, with every CDN on earth. You do not keep the whole world hot. You keep a way to get to the part you need.

Dense models feel more honest. They also feel like a luxury belief. If you have the money and the energy budget to keep everything awake, congratulations. Most of the useful work I see is closer to: get a good enough specialist on the token, do not blow the latency budget, leave room for retrieval and tools to do the factual work.

I am not saying the research is done. Better routers would help. Better training so experts do not become decorative would help. I would like a world where sparsity is a property of the computation, not a pile of expert islands and a traffic cop. Until that world shows up in a checkpoint I can run, MoE is the available compromise.

There is also a taste issue. People treat architectural purity as a moral stance. Dense is "real." MoE is a hack. Transformers were a hack that worked. Every production system I have loved has been a hack that worked, then got cleaned up just enough that you could sleep.

If MoE leaves a bit of intelligence on the table, fine. I will take the intelligence I can afford to serve, and spend the leftover budget on memory, tools, and evals against the actual job. Those have been the bigger leaks in every product I have touched.

Probably good enough is not a slogan. It is the sentence you say when you have stared at a prettier alternative long enough to notice it does not have a ship date.