benfreu.com benfreu.com
benfreu.com
Blog

benfreu.com Blog

Transformers using the first few tokens as attention sinks

Attention sinks in LLMs

The attention mechanism has a small flaw that recent models fix with one extra token.
Alexander the Great Cutting the Gordian Knot by André Castaigne (1898–1899)

What I learned this week 2026/35

The Gordian Knot was also untied and you can print your own magazine.
YouTube retention graphs explained

What I learned this week 2026/34

Nautical miles and Earth's size, filmmaking is cooperative, Pixar's story, everything is a YouTube video, and AI is used for low-priority work.

New posts straight to your inbox

Subscribe and I'll send you my latest posts about twice a month – written by a human. (I typed that en dash into my keyboard)

By signing up, you agree to benfreu's Terms of Service and Privacy Policy. You'll only get new posts and from time to time I'll tell you about a project I'm working on, but I'll never spam you. And you can unsubscribe at any time.