Humans may be just the bootloader for digital superintelligence, but we have to work awfully hard to make sand think. To do so required following an exponential curve of progress for more than fifty generations of chips, a new one every year for more than half a century. No other human endeavor comes close to sustaining this level of progress over so many iterations. The result is that carefully arranged silicon, with a few other atoms added in the right places, can now perform something that resembles human thought. AI chips — like Nvidia’s monstrous accelerator package with more than 200 billion transistors — are the most complex thing people can manufacture today.

The story that follows is about the collision of two kinds of exponential progress. One is rooted in atoms, chemistry, and the science of advanced manufacturing. The other is driven by data, statistics, and algorithms. Each advanced along its own trajectory, yet their convergence produced capabilities that neither could have unlocked alone. The modern AI boom is best understood by viewing both forms of scaling as parts of one continuous technological evolution.

The first type of scaling, compute, is based on a simple on/off switch, the transistor. Transistors are the fundamental building blocks of digital logic. They use an electrical input signal to determine if current can flow through the transistor or not. This “on” or “off” current flow can store a binary state, a simple 0 or 1. Basic primitives — logic gates — can be constructed by wiring transistors together. Connecting two in series, with the output of one flowing to the input of another, forms a logical “AND” — the output is only a 1 if both inputs are 1. Similarly, a logical “NAND” — which stands for ‘not and’ — can be formed with 4 transistors wired in a complex pattern.

Amazingly, any digital circuit, from the simplest binary adder to the most complex AI accelerator, can be constructed with NAND gates. It’s an elegant and beautiful result of information theory: the NAND gate itself is universal and functionally complete as it can express every possible boolean function. Simple functions are intuitive: a logical AND — output 1 if and only if both inputs are 1 — can be formed with two NAND gates in series, the output of the first used as both inputs to the second. Logical OR can be produced with two ANDs and another NAND, for a total of five NANDs. And so on for other basic logic operations, NOT, NOR, XOR, and more.

Combining many simple logic operations in a circuit forms logic blocks, with simple functions like adding and subtracting. Logic blocks in turn can be combined into simple subsystems that load and store instructions. After that, microarchitectures, instruction-set architectures, and further until, after ten or so layers of abstraction, you are at the application level: what users see. This is a web browser, word processor — or now an AI performing humanlike tasks. But at the bottom of these systems are just simple on/off switches, wired together in the right way. There are billions of NAND gates in a chip the size of your thumbnail.

A long series of innovations were required to reach the point where billions of switches could be packed on a chip. Early computers used vacuum tubes which were fragile and power-hungry. Vacuum tubes relied on heated cathodes in a vacuum to emit electrons towards anodes, a mechanism that is slow, inefficient, and crude by modern standards. The transistor, invented in 1947, replaced clunky vacuum tubes with a solid piece of silicon that could switch reliably and efficiently. Rather than waiting for a cathode to heat up, in a transistor an electric field opens a “channel” for current to flow inside silicon. And instead of an airtight glass tube, a transistor needs just a literal sliver of silicon — which we now can manipulate at the nanometer scale. Transistors use far less power, can be made smaller, and operate at far higher speeds than vacuum tubes, switching on and off more than a billion times every second.

A decade after the invention of the transistor came the integrated circuit, which put multiple transistors on a single chip to form logic gates (like NAND). Soon after logic gates came logic blocks that could perform simple math or route digital signals , then microarchitectures executing instructions and interfacing with memory, and so on. And within twenty years of the creation of the first silicon transistor, simple operating systems were running applications on integrated circuits of 100,000 transistors.

Modern chips begin as highly purified silicon. Manufacturers convert silicon dioxide into material that is almost entirely free of defects, then melt it and form it into a single crystal cylinder called an “ingot.” The crystal is then sliced into wafers and polished until defects on the surface of these wafers are measured in fractions of an atom. This level of flatness is not cosmetic; even small imperfections would distort later steps and cause device failures, so the starting surface must be close to perfect.

Fabrication proceeds layer by layer atop the bare wafer. Transistors and wires are built through repeated cycles of material deposition, lithography, etching, and doping. Lithography, a process that uses light to map circuit patterns onto a silicon wafer, transfers the chip design — a series of simple lines, spaces, and circles that will be transformed into transistors, wires, and vertical connections from layer to layer. Using the pattern defined via lithography, other processes adjust the electrical behavior of the silicon itself. Ion implantation inserts controlled amounts of elements like boron or phosphorus to create regions where current can or cannot flow. Thin film deposition forms conductive wires and insulating layers only a few atoms thick. Chemical-mechanical polishing removes bumps, ensuring a pristine, atomically flat surface for the next layer. A leading-edge chip requires thousands of these steps, each executed with nanometer-scale precision.

After completing all layers — as many as one hundred in modern chips — the wafer is tested. Functional chips are cut out from the wafer, bonded to tiny solder bumps, and packaged so they can be connected to circuit boards. Although the end product may be small, construction of modern chips pushes physics, chemistry, and manufacturing equipment to their limits. The fact that these steps can be repeated millions of times per day with just one or two defective transistors in a billion is one of the most impressive achievements in modern industry.

Training a neural network is largely a matter of multiplying and summing vast tables of numbers. Although the results look almost like magic, this is just statistical pattern recognition at scale. Artificial intelligence took off once hardware could deliver enough raw computation. What made today’s AI possible was the ability to place enormous numbers of compute units side by side, close to memory, and to keep them all running in parallel. As chip capabilities grew, researchers built models large enough to capture language and vision. As models demanded more, the hardware advanced again. This feedback loop spurred the present AI boom.

Scaling laws add a second exponential to the story. Moore’s law concerned the number of transistors engineers could pack onto a chip. Scaling laws in machine learning describe how model performance improves as you increase three quantities: the size of the model, the amount of data it is trained on, and the compute used during training. In 2020, researchers studying large language models noticed something striking: when these three variables rose concurrently, performance improved following smooth power law curves. The models did not improve in fits and starts. And they did not level off unexpectedly. Instead, they followed clean mathematical relationships that predicted how much better the next, larger system would be. If you doubled the compute and data in the right ratio, you could predict the reduction in error before doing the training run.

The parallel between Moore’s law and scaling laws is not superficial. Both describe a world where steady, compounding improvements produce outcomes that seem impossible when viewed from the starting point. Moore’s exponential made processors a million times more capable over five decades. Scaling laws did something similar for AI, except the exponent has played out over a handful of years. Early language models were clumsy pattern matchers that struggled to keep track of long sentences. Their training datasets were a fraction of today’s, and their parameter counts were measured in millions. By the time ChatGPT arrived in November 2022, models had grown to hundreds of billions of parameters. Training runs consumed millions of GPU hours. The compute used just to train one model was larger than the total used to train every neural network in the world just a decade prior. Yet instead of chaos or diminishing returns, researchers found that bigger models almost always performed better, and that the gains were predictable.

There is an important difference between Moore’s law and scaling laws, though. The former was partly a roadmap. The semiconductor industry organized itself around Moore’s law, coordinating investments so that each generation of chips arrived roughly on schedule. But scaling laws in AI were discovered after the fact. They described what researchers were already stumbling into as they pushed models larger. Once recognized, they became a guide. Labs now forecast the performance of future models using these laws, estimate the compute budgets needed to reach certain capabilities, and design data collection efforts accordingly — the same as Moore’s law used to guide semiconductors. The laws do not guarantee success. But they do offer a rare quantitative framework that brings structure to a field long guided by intuition and luck.

Understanding why scaling laws work requires a brief look at what a large language model actually does. The model learns to predict the next word in a sequence by adjusting internal weights so that its predictions better match the patterns in its training data. Those weights give the model a sort of high dimensional memory of language structure. As models get larger, they can “memorize” and represent more subtle patterns. With more data, they can generalize across a greater span of contexts. With more compute, they can train on that data with a model size large enough to encode all of it. Scaling laws describe the quantitative relationship among these factors. Crucially, they show that models remain “underfitted” at every size we have tried. In plain terms, we have not yet reached a point where adding more data or compute ceases to help. Moore’s law eventually slowed when transistors ran into atomic limits in the 2010s. Scaling laws have yet to hit a similar plateau.

Remarkably, once models grew past certain scaling thresholds, their behavior improved in ways that went beyond steady reductions in prediction error. They began to show abilities that had never been programmed into them at all. Emergent abilities. Tasks like multi-step reasoning, translation between obscure language pairs, and writing coherent code, that were not explicitly programmed in new models. They just surfaced when models grew large enough to internalize the structure of these problems from data alone. These phase transitions echo the qualitative jumps seen in computing during the early years of Moore’s law — when added transistor budgets made entirely new classes of applications feasible. Just as the shift from thousands of transistors to millions enabled graphical interfaces and early networking, the leap from billions of parameters to hundreds of billions enabled systems that interact with humans in fluent language.

With the right architecture and training process, relentless scaling can carry a technology into regimes that feel qualitatively different from what came before. The semiconductor industry proved this over half a century of squeezing more transistors into less space. Machine learning is now showing the same pattern of relentless scaling in compressed form.

The foundation for all of this progress, the machines that “think," is itself extraordinary. Modern chips pack 100 billion transistors into the area of a thumbnail, each one patterned with atomic precision. A single misplaced atom can cause a defect, yet a finished chip typically has only a handful. Factories repeat this feat across wafer after wafer, every day, yielding hundreds of millions of nearly perfect devices each year. And this reliability and scale are what make the relentless expansion of AI models possible. The models might look miraculous when they cross new capability thresholds, but they depend on the second miracle of modern chip manufacturing. Taken together, these two forms of scaling have created systems whose behavior and reach would have been difficult to imagine only a decade ago. Only time will tell what the next decade will bring.