My Top Algorithms Across Transformer Architectures

---

Algorithms can be evaluated across many dimensions: computational and memory efficiency, mathematical elegance, creativity, practical impact, and, of course, personal preference.

This is my list of favorite algorithms and mechanisms across Transformer architectures, judged by some combination of those factors and my own human bias.

1. IPA — Attention With a Point

This is my single favorite algorithm across them all.

Invariant Point Attention (IPA) is an attention mechanism used in protein structure models. I encountered it later than standard attention, but implementing protein structure models with IPA actually made me understand attention at a much deeper level.

The reason is that IPA forced me to think about attention geometrically.

With ordinary attention, it can sometimes be difficult to attach an intuitive meaning to what the Query and Key actually represent. They are learned projections of vectors, and depending on the architecture, there may not even be a useful word for what they correspond to.

But with IPA, we have a point!!!

Literally.

The attention mechanism operates not only on learned features but also on points in three-dimensional space. Suddenly, part of the attention calculation has a geometric interpretation: these points have positions, distances, and relationships in physical space.

That was exciting to me.

It also made the mathematics feel more tangible. Instead of thinking about attention purely as operations on abstract vectors, I could see how the mechanism was reasoning about the geometry of a protein.

I’m probably somewhat emotionally attached to IPA for the same reason I’m attached to the Euler totient function, $\phi$: there is something unusually satisfying about the mathematics and I abused that function when I did number theory.

2. CLIP

Another emotional one.

If I were to pursue research, representation learning is one of the areas that interests me most. When I first encountered CLIP, I remember thinking:

Yeah. That’s the path.

CLIP itself isn’t necessarily the most sophisticated vision algorithm anymore. Its importance to me is more fundamental than that.

The idea of learning a shared representation between images and text is incredibly elegant. Instead of explicitly programming a model to understand every possible relationship between the two modalities, we can construct a learning objective that encourages semantically corresponding representations to become aligned.

That way of thinking about representation learning is what caught my attention because I always thought about it.

It also connects naturally to the ideas behind many newer architectures. CLIP isn’t simply “an image model” to me; it’s an example of a broader idea: learn a representation that captures the structure we actually care about, rather than forcing the model to solve every downstream task independently.

That is a direction I find particularly interesting, especially as I move toward architectures such as JEPA.

3. RoPE

I like this one partly because I don’t like the alternative.

Yes, getting a set of sine and cosine pairs works mathematically to encode a unique time/position, but it’s just not elegant to code.

It’s a small transformation with a surprisingly large consequence.

Besides, the math hugs the most beautiful equation in math:

$$e^{i\pi} = -1$$

That alone probably earns RoPE some points from me.

More importantly, the fact that positional relationships emerge through the interaction of rotated queries and keys makes the mechanism feel much less arbitrary than simply adding a positional vector to every token.

4. Attention

It would be unfair to put IPA at the top without giving some love to its ancestor.

IPA would not exist without attention.

The first time I really understood the Transformer architecture, I remember thinking: wow.

Before attention, I already understood that vectors and matrices could represent information and that transformations could change the representation of a concept.

But using vectors to essentially ask questions about other vectors was mind-blowing.

Given a representation $X$, we can transform it into a Query:

$$Q = XW_Q$$

and a Key:

$$K = XW_K$$

The transformations are related because they originate from the same underlying representation, but they give that representation different roles inside the attention mechanism.

One can think about this imperfect linguistic analogy.

Suppose the representation corresponds to the concept of Leyla. Depending on the context, we might want to ask something like:

Who?

while another representation provides something like:

She.

The underlying concept is related, but its role in the interaction changes.

But the analogy captures something beautiful about the mechanism: the same underlying representation can be projected into different spaces for different computational roles.

The tranformations make the vector interactions “grammatically correct”.

That combination of simple linear algebra and surprisingly expressive behavior is what made attention so interesting to me in the first place. I also spent some time deeply impressed by the authors of the attention paper.

And ultimately, that is probably why IPA became my favorite: it takes an already beautiful idea and gives part of it a physical, geometric interpretation and I love geometry.