Published:
I spent January and February 2026 interning at the LUMIA Lab at Shanghai Jiao Tong University. It was a busy time focused on ICML submissions, where I finished a first-author paper on dLLM and a third-author paper on PonderLM, among others. I chatted a lot with the seniors and my mentor in the group and learned a great deal.
Published:
I applied for countless internships in Shanghai, but somehow, I ended up in Beijing. I still remember the excitement when I received the internship offer in early July. I thought I was blazing a new trail, so I came here, rented an apartment, and started my internship life.
Published:
Doing the obvious things right is harder than it sounds. I have watched “model companies” that barely build models. I have seen placeholder papers with no real experiments—work you cannot reproduce, or results that do not add up. Plenty of people train models by hoping for a lucky run. And I am sure many of us have seen bad ideas pushed forward anyway, because someone cared more about their own gain than about whether the thing actually works.
Published:
I believe pretraining for large language models is far from finished. A lot of work today focuses on building agents and on post-training. Those steps can help on specific tasks, but the gains are often narrow. The deeper abilities we still need—making sense of the world and dealing with messy real situations—depend on large-scale pretraining with careful data and real changes to how the model is built. That includes new versions of the basic Transformer design, memory inside the model that ties ideas together more cleanly, and flexible ways to blend representations when the setting shifts.
Published:
This post will show up by default. To disable scheduling of future posts, edit config.yml and set future: false.
Published:
This is a sample blog post. Lorem ipsum I can’t remember the rest of lorem ipsum and don’t have an internet connection right now. Testing testing testing this blog post. Blog posts are cool.
Published:
This post will show up by default. To disable scheduling of future posts, edit config.yml and set future: false.
Published:
This is a sample blog post. Lorem ipsum I can’t remember the rest of lorem ipsum and don’t have an internet connection right now. Testing testing testing this blog post. Blog posts are cool.
Published:
This post will show up by default. To disable scheduling of future posts, edit config.yml and set future: false.
Published:
This is a sample blog post. Lorem ipsum I can’t remember the rest of lorem ipsum and don’t have an internet connection right now. Testing testing testing this blog post. Blog posts are cool.
Published:
论文记录《Parallelizing Linear Transformers with the Delta Rule over Sequence Length》