Task Superposition Effect in LLMs

The Surprising Ability of LLMs to Perform Different Tasks simultaneously from a Single Inference Call

Pete Weishaupt
2 min readOct 13, 2024

In the paper “EVERYTHING EVERYWHERE ALL AT ONCE: LLMs CAN IN-CONTEXT LEARN MULTIPLE TASKS IN SUPERPOSITION” the authors explore the surprising ability of large language models (LLMs) to perform multiple distinct tasks simultaneously during a single inference call. This phenomenon, referred to as “task superposition,” is demonstrated through empirical evidence across different LLM families and scales.

The authors posit theoretical explanations for this capability, highlighting the expressive power of transformer models. They demonstrate that even models trained to learn one task at a time can exhibit task superposition. Further, the authors investigate how LLMs internally combine task vectors during superposition, showing that larger models are better at solving more tasks in parallel and calibrating their output distributions.

The paper concludes by discussing limitations and future directions for research, particularly in developing decoding strategies that can leverage task superposition effectively.

Dig Deeper:

The phenomenon of “task superposition” is the ability of Large Language Models (LLMs) to perform multiple, distinct in-context learning (ICL) tasks simultaneously during a single inference call.

--

--