Google's MIRAS and TITANS
The next step forward?
After eating my words, I now have to reframe something that I have taken for granted.
My research has been about designing deep learning models for networks and multimodal data that evolve across time. I had used RNNs, transformers as well as hybrids for such sequential data.
These two papers from Google (MIRAS and TITANS) have made me rethink how I view such models.
The MIRAS paper is a conceptual framework. The TITANS paper is an instantiation of the framework that has the potential to shift the field.
Both are related to a paper on ‘Nested Learning’ that I posted a while back (that led me to eat my words, see link).
MIRAS frames AI models for sequential data as being about just 4 questions.
1️⃣Memory - How to remember?
2️⃣Attention - What to pay attention to?
3️⃣Retention - What to forget?
4️⃣Learning - How to update what you know?
It highlights that even models prior to transformers (such as RNNs) addressed these 4 questions (e.g., knowing what to pay attention to, even if they did not use the attention mechanism in transformers).
The TITANS paper proposes concrete approaches for each of these questions.
But its most interesting contribution is the element of surprise(!)
Inspired by human memory (surprising events are more memorable), TITANS uses momentum of change as a way to track surprise and learn. The basic idea is that more surprising information is more important, and so TITANS prioritizes such surprising information when learning.
TITANS is also interesting in that it uses a neural network as dynamic memory that can still learn after training, in real time.
Links to papers below. Worth a read if you work with sequences, time-series, text, videos …
Google article - https://research.google/blog/titans-miras-helping-ai-have-long-term-memory/
TITANS paper - https://arxiv.org/abs/2501.00663
MIRAS paper - https://arxiv.org/pdf/2504.13173
#AI #Attention #Google #DeepLearning