What is a neural network?

A neural network is a function. Numbers go in, numbers come out, and in between there is a pile of multiplication and addition. Everything else you have heard about them is either a detail of how the pile is arranged, or a story about where the numbers in the middle came from.

The whole mechanism, in one paragraph

Take a list of input numbers. Multiply each by a weight and add the results together, which gives you one number. Do that many times with different weights, and you have a layer of outputs. Squash each one through a simple curve so it cannot grow without limit. Feed those into another layer the same way. At the end, read off the outputs and interpret them as whatever you decided they mean - a letter, a probability, a direction to move.

That is all. There is no reasoning step hidden anywhere, no symbolic logic, no part of the computation that is doing something other than arithmetic. The intelligence, where there is any, is entirely in the particular values of the weights.

Where the weights come from

This is the only interesting question, and it has several answers.

• Training by gradient descent, which is how almost every neural network you have heard of was made. Show it examples, measure how wrong it was, adjust every weight slightly in the direction that would have been less wrong, repeat a few billion times.
• Evolution, which is how this project's organisms got theirs. Weights are inherited, mutated slightly at birth, and whichever ones happen to belong to organisms that leave descendants are the ones that persist. Nobody computes how wrong anything was, because nothing here defines wrong.
• Learning during life. Some networks also adjust while running, so an individual changes over its own lifetime rather than only across generations. Both happen here, separately, which is unusual and makes it possible to tell them apart.

Small enough to look at

Modern language models have hundreds of billions of weights, which is why nobody can inspect one and say what it is doing. An organism here has a recurrent network with 24 hidden units and about two thousand weights - small enough to print.

That is not a toy version of the real thing. It is the real thing at a size where the usual excuse disappears. You can open any living organism's brain while the world runs, see every weight, and watch which inputs are driving its behaviour right now. Whatever it is doing, nothing is hidden behind scale.

Why "recurrent" matters here

A plain network maps inputs to outputs with no memory: the same input always gives the same output. A recurrent network feeds some of its own activity back in at the next step, so what it does depends on what just happened to it.

For a creature in a world, that difference is the difference between a reflex and a short-term state. It is also why two organisms with identical weights can behave differently: they have been through different things.

Recurrent networks in more depth · Do they think like brains? · How one here actually decides