| Stewards | Ryan Healey, Jonah Henkle |
| Editors | Jonah Henkle |
Superposition in machine learning refers to the phenomenon whereby, “models represent more features than they have dimensions” [1]. Within this definition is an operative metaphor borrowed from quantum mechanics: the representation of more features than dimensions is like the condition of superposition in quantum mechanics where a physical entity like an atom or electron exists in multiple states at once. In quantum mechanics, physical objects have energy values which are quantized, meaning the values are not on a continuous spectrum but limited to certain set values. At a definite state, objects have one of those finite values, but in superposition they can have multiple of these values at once [2]. Quantum states are described as the condition of having multiple values at once.
In the realm of machine learning, the metaphor, “superposition,” is used to describe the way in which relatively small models can represent the content of, or appear as, much larger models. In their paper, “Toy Models and Superposition,” Hughes et al. deploy the metaphor of superposition to describe a very real phenomenon in artificial neural networks. The superposition metaphor is meant to capture how, in noisy environments, small models are able to compute in quantum states, operating like larger models. The superposition thesis stands on the assumption that network representations are the culmination of discrete features (decomposability) and features are represented by direction (linearity) [1]. These two assumptions stem from a desire for model interpretability. The superposition thesis is then a further attempt to give a name to seemingly unexplainable opaque phenomena in learning models.
What are features? It appears proponents of the superposition thesis for machine learning have not settled on a single coherent definition. The term “features” gestures at several distinct but interrelated concepts. Features suggest:
We suggest none of these definitions of features produce a stable foundation for the theory of superposition in machine learning. Hence, where superposition is wielded as a strong metaphor for behavior in machine learning, it functions as a weak and somewhat obscure analogy.
In his 1935 paper, “Die gegenwartige Situation in der Quantenmechanik” [3], Erwin Schrödinger cautioned, “Above all, a determinate model and a determinate state of the same must not be confused” [3]. Of course this paper is most famous for its “cat paradox” thought experiment—a provocative heuristic for superposition. We want to focus instead, on how the discourse of superposition emerges as part of the discourse of the determinacy of models. Schrödinger continues, “The clear view over the totality of possible states—yet without relationship among them—constitutes ‘the model’ or ‘the model in any state whatsoever.’ But the concept of the model then amounts to more than merely: the two points in certain positions, endowed with certain velocities. It embodies also knowledge for every state how it will change with time in absence of outside interference” [3].
Superposition seeks to capture the “knowledge” of models or systems in the absence of direct observation. Similarly, interpretability seeks to uncover the hidden phenomena inside the so-called “black box” of learning models.
[IDEA] We can think of the ambition of the superposition thesis for machine learning as an attempt to enter machine learning into Schrödinger’s discourse, or the discourse of physics more broadly. Hedging, we can advance a weak version of the superposition thesis for machine learning: not that “superposition is a real observed phenomenon”[1], but that machine learning models, like quantum models, are characterized not only by the totality of states, or the totality of neurons, but all of the possible states, or all the entailments produced of the network.