How a purist solution may not be the answer to inventing true machine intelligence.
My interest in machine intelligence has long been shaped by texts like Jeff Hawkins’ On Intelligence (2004). I used to hold the view that achieving truly intelligent machines required building systems that mirrored the human brain down to the deepest computational details—that only “pure” neural net technology could suffice. But recently, a counter-thought has begun to take shape: perhaps technological compatibility is more important than biological fidelity in the pursuit of better machine intelligence algorithms.
The problem lies in our current computer architecture choices. Von Neumann architecture separates processing and memory with a relatively slow bus that is the result of legacy computer design which utilizes storage that was inherently much slower than the processor. With the development of (relatively) cheap RAM this problem partially goes away, but is still inherent in the design.
Neuromorphic computing
I am personally very interested in developing architectures where the processing and memory are co-located and highly intertwined. Something with the parallelism of a GPU but with neuron-like specialization at each node specifically made for the application of neural networks and natural language processing. Perhaps Intel’s neuromorphic chips will pioneer the way, or IBM’s NorthPole chip.
What Large Language Models do RIght
What I have begun to learn, is that modern Large Language Models are actually very efficient at utilizing current computer architecture in a way I hadn’t anticipated. In fact, the Transformer technology they use in particular, is custom made for running quick inference on GPUs and were a major breakthrough for machine learning research. This technology was spearheaded by a team of researchers at Google, who published their findings in the landmark 2017 paper Attention Is All You Need.
While today’s LLMs do not closely follow the design of the human brain like that envisioned by myself and Jeff Hawkins and others, they are able leverage the hardware in a way that a more pure solution would simply not be able to. This may be superior to models that closely follow the brain in a way I haven’t envisioned, and lead to better intelligent machines.
Historical Architecture Decisions
While presenting the earlier draft of this article to Claude.ai it pointed out an interesting insight I hadn’t seen.
This is the part I find most interesting. You’re essentially arguing that intelligence isn’t substrate-independent in a naive way — that a given computational paradigm and a given hardware architecture co-evolve and constrain each other. Transformers happen to map extremely well onto GPU/TPU parallelism (matrix multiplications, all the way down).
A more biologically faithful architecture might be algorithmically superior in some abstract sense and still lose because it doesn’t leverage the hardware ecosystem we’ve spent 70 years optimizing.
This is what I find most interesting. The the hardware we have now is limited by Von Neumann memory bottlenecks. It is what we have had to work with so far and has deeply influenced our software design. In a matter of fact, the two are actually inextricable.
History actually supports this. The brain-inspired connectionist approaches of the 80s partly lost ground to symbolic AI not because they were wrong but because the hardware wasn’t there yet. Then deep learning “won” in the 2010s partly because GPUs happened to be perfect for it — almost accidentally, since GPUs were built for gaming.
It’s remarkable that a category of hardware designed for gaming and 3D graphics became the cornerstone of machine learning and our understanding of how to leverage existing hardware. Perhaps God is in the details sometimes.
When you write a piece of software you are bound to the hardware you’re working with – you can’t just dream pie-in-the-sky dreams. And conversely, when you design hardware, engineers are often constrained to make the hardware backwards compatible with existing codebases. Of course, abstraction layers like virtual machines and operating systems can shield software from the underlying hardware — but even these are ultimately running on physical silicon with real constraints.
A New Thought
This has led to a new thought for me: maybe we don’t have to follow the design of the brain so closely and can instead rely on iterative improvement to come up with a superior design that effectively leverages the underlying hardware. The brain is one solution to the problem of intelligence — but it may not be the only one, and it may not be the best one for silicon. Designing around the hardware we have, rather than reverse-engineering biology, could open doors we haven’t thought to look for yet.
The brain’s design differs greatly from current silicon — it is relatively slow and remarkably power-efficient, yet it still runs on electrical impulses in a way that echoes semiconductor design. It is also built for a very specific purpose: a hierarchical, sparse-distributed memory system that handles both high-order thought and fine-grained memory detail exceptionally well. This architecture is amazing but does not reflect current computer architecture well.
I think maybe if modern computer hardware was more parallel by design and had memory that was co-located, we might be able to even rival the design of the human brain someday in certain applications. Such designs might be able to work right alongside current Von Neumann silicon. But that is only a thought for now.
Conclusion
The idea that intelligence isn’t substrate-independent perfectly captures what I’ve been grappling with, and it has propelled my understanding of machine learning forward. And perhaps we will be able to build better intelligent machines because of it. Both the opportunity to build better traditional sequential algorithms and brain inspired network algorithms exists.
I always appreciate when a new insight like this emerges and gives me something to think about.
Timothy Grindall is a technology writer and self-taught computer programmer, photographer, filmmaker, thinker, and writer who wants to see the world change for the better.
This article is a rewrite of an earlier, rougher draft. With help from the Gemma 4 LLM running on my laptop and some assistance from Claude, I reworked it into something clearer and more satisfying.
Leave a Reply