A Treatise on Quantization — Quantum Physics and Artificial Intelligence

 

This article is more of a philosophical exploration of Quantization in Quantum Mechanics and Large Language Models.

Humans have a knack for finding surprising conceptual parallels between different areas of study. Here, we want to trace a thread linking the mathematical structure of quantization — as it manifests in quantum mechanics and current large language models (LLMs). In thinking about how quantization relates across these two domains, I’ve been quite excited to draw on my work as a philosopher of science, focusing on the crucial but delicate differences between these two kinds of quantization.

The Nature of Quantization — From Continuous to Discrete

Beneath the bluster is a discretization — a mode of passage from the continuous to the discrete. In quantum mechanics, quantization is the overarching singular step that characterizes the transition from classical physics to the quantum world, where physical quantities become discrete. In the LLMs, quantization is merely a more mundane technique to reduce the computational complexity of these giant neural networks, which, thanks to their size, are computationally extremely expensive.

However, though these two examples might be analogous in their broad outlines, this is only superficial; Quantization is much more profound and raises more questions in these two domains. This leads to central puzzles about reality, knowledge, and computation.

Quantum Mechanics — The Quantization of Reality

Nothing ‘boils away,’ as in Aristotelian philosophy. Quantization is not just a mathematical trick or an approximation; it’s actual quantum mechanics. (For those interested, we focused briefly on the details of this process in a document that sums up the main points here.)

Non-uniqueness and Ambiguity — The checklists and recipes of the textbook version of quantum theory stand in sharp contrast with the quantum mechanical state of affairs involving the realization of quantization. Quantizing a classical system is inherently ambiguous since no unique, natural quantization procedure exists. This non-uniqueness is a telltale of a deep philosophical issue: the quantum and the classical worlds are either truly disjointed or may need the right angle to reveal the connection.

Geometric and Algebraic Structures — When trying to achieve quantization in physics, we usually need to add some other mathematical structures, such as choices of coordinates or representations. Geometric quantization and the role played by symplectic geometry illustrate how tightly physics is linked to mathematics. This interplay reinforces the idea that nature might be much more mathematical than we realize.

It adds another layer. There is this deep representation-theory connection between quantization and the representation theory of groups. Going back to the mathematical analogy of Geometry and Arithmetic, you may not know the nature of the many-dimensional space we inhabit — but you know enough about reflection symmetries and their role in tackling difficult questions (that is, you know geometry). Quantum theory connects left and right, past and future, particle and field, vibrators and stars, all in ways that classical theory cannot understand. If this symmetry notion is fundamental, what is its nature? It’s a mathematical question, so what does that mean?

Complex by Branes is an approach that searches for a valid solution to the quantum problem by embodying it once again in a highly complex space with levels of complexity, all embedded in a richly complex background. These mathematical basic intuitions also investigate the question that provokes all this with further reach: the reach beyond what seems adequate. And so on. Once again, this new synthesis demands that we accommodate it by carefully expanding the conceptual apparatus at our disposal. Complexification, indeed.

Large Language Models — The Quantization of Knowledge

Unlike quantization in quantum mechanics, which is a fundamental property of quantum theory, the quantization of LLMs is the homebrew response to a practical need, and it raises similar philosophical questions to those raised by quantization in quantum mechanics.

Approximation of continuous knowledge — At the source of LLMs, one finds an attempt to capture continuity, movement, flow, and the intricate dynamics of human language discretely and computationally. Quantization here seeks to find efficient representations of an infinite, continuous space of meanings. In this respect, there’s a conceptual connection between the problem of translation with LLMs and a continuum of physical states using the discrete language of quantum mechanics.

Trade Precision and Efficiency — The motivation behind LLM quantization is to drastically reduce the computational complexity (in the amount of storage, time, energy, hardware accelerators, data center real estate, etc.) without a significant loss of performance — that is, by reducing the precision of model parameters and activations from, say, 32-bit floating-point numbers to 8-bit integers. This essential trade-off of precision for efficiency uses the amount of quantization error as a proxy. What does this tell us about the nature of intelligence and understanding? How much precision is needed to capture meaningful knowledge?

Non-linear Effects and Emergent Behavior — Unlike in quantum mechanics, where the effects of the quantization are fundamental and (‘in principle’) predictable, the effects of the quantization on LLM performance are often non-linear and emergent — the behavior is not ‘determined’ by the quantization itself. Like many other complex systems, it is hard to predict the effects of changes at the micro-scale on behaviors at the macro-scale.

Adaptive and Learning-Based Approaches — Some sophisticated approaches to quantizing LLMs involve fine-tuning or retraining the model after it has been quantized. This is unlike any comparable process in quantum mechanical quantization. It reflects the ever-changing nature of artificial intelligence that is not part of the fixed laws of quantum mechanics.

Philosophical Implications — Reality, Knowledge, and Computation

The contrast of quantized LLMs raises some significant philosophical questions.

Both domains, however, wrestle with the trade-off between continuous and discrete representations, and quantum mechanics, in particular, makes this trade-off inherent to reality. The same does not seem to apply to LLMs, which are constrained by the tension between continuous and discrete representations only insofar as it’s helpful to do so. Does this mean discreteness is a hard limit of human understanding or just some nifty approximations?

Levels of Abstraction — Quantum mechanical quantization operates at the foundational level of physical reality, while LLM quantization operates at a very high level of language and knowledge. Are these levels fundamentally connected, or are they irreducibly different?

What is the Role of Mathematics? Just as mathematical formalisms play an essential role in Quantum Mechanics (QM), they also play a critical role in LLMs. Yet, while QM seems to enable one to reach deep truths concerning the nature of reality, in LLMs, they are employed more as a pragmatic tool. What are the implications concerning the relationship between mathematics, reality, and the models we construct representing it?

The central limitation of uncertain knowledge in quantum mechanics concerns a physics problem, but it is one of another sort. Quantum uncertainty is an ineliminable feature of the quantum formalism. By contrast, LLM uncertainty is (typically) a question of approximation and statistical patterns; it is more akin to Bayesian uncertainty concerning information about the world rather than a peculiar and profound fundamental property of our physical interactions with the world. How different is this? How different is this uncertainty’s function in our interaction with the world? What type of knowledge does it correspond to? What types of predictions can we make based on it?

Conclusion — The Convergence of Physics and Artificial Intelligence

Future discoveries at the intersection of quantum mechanics and artificial intelligence may bring us closer to solving some of the most profound questions about science and the philosophy of reality. The remarkable fact that a single phenomenon called quantization appears so differently and critically in these two fields is already hugely suggestive.

However, the most profound thing to take away from the comparison is that the idea of quantization is, in a sense, everywhere. From the tiniest aspects of physical reality to the most significant models of human knowledge we have managed to build into our AI systems, there is this tension between the continuous and the discrete, the analog and the digital.

It could end up being the case that as we push the envelope in two seemingly unrelated and disparate fields, quantum mechanics and artificial intelligence, those two fields might teach each other more than we now first suspect. Some of the deep philosophical questions that arise with their juxtaposition might give us a whole new outlook on questions about knowledge and reality and, ultimately, on whether we’re just patterns of unfathomable computation.

All in all, the story of quantization in physics and AI is a testament to the power of abstract thought, to the rich connections that can be discovered when you look at seemingly unrelated fields through a standard lens, even in complex phenomena, seemingly diverse objects can fit into a unified picture. The take-home lesson is that, when exploring nature, it always pays to assume little, be uncomfortable with complexity — and always keep in mind unity in diversity.