Can neural networks compute anything a computer can?

The answer

The Universal Approximation Theorem, proven by George Cybenko in 1989 and extended by Kurt Hornik in 1991, shows that a feedforward neural network with a single hidden layer containing enough neurons can approximate any continuous function to arbitrary precision. This means neural networks are theoretically capable of representing any computable function - they're Turing complete. However, 'can approximate' doesn't mean 'will learn efficiently.' A network that can theoretically represent a function may need impractically many neurons or training steps to learn it. Recurrent neural networks and transformers with unbounded computation are formally Turing complete, meaning they can simulate any algorithm given sufficient resources, though real-world constraints always limit this theoretical power.