{"id":2248,"date":"2023-04-03T09:02:27","date_gmt":"2023-04-03T09:02:27","guid":{"rendered":"https:\/\/mpelembe.net\/?p=2248"},"modified":"2023-04-18T14:28:56","modified_gmt":"2023-04-18T14:28:56","slug":"ai-will-soon-become-impossible-for-humans-to-comprehend-the-story-of-neural-networks-tells-us-why","status":"publish","type":"post","link":"https:\/\/mpelembe.net\/index.php\/ai-will-soon-become-impossible-for-humans-to-comprehend-the-story-of-neural-networks-tells-us-why\/","title":{"rendered":"AI will soon become impossible for humans to comprehend \u2013 the story of neural networks tells us\u00a0why"},"content":{"rendered":"

David Beer<\/a>, University of York<\/a><\/em><\/span><\/p>\n

In 1956, during a year-long trip to London and in his early 20s, the mathematician and theoretical biologist Jack D. Cowan visited Wilfred Taylor and his strange new \u201clearning machine<\/a>\u201d. On his arrival he was baffled by the \u201chuge bank of apparatus\u201d that confronted him. Cowan could only stand by and watch \u201cthe machine doing its thing\u201d. The thing it appeared to be doing was performing an \u201cassociative memory scheme\u201d \u2013 it seemed to be able to learn how to find connections and retrieve data.<\/p>\n

<\/p>\n

It may have looked like clunky blocks of circuitry, soldered together by hand in a mass of wires and boxes, but what Cowan was witnessing was an early analogue form of a neural network \u2013 a precursor to the most advanced artificial intelligence of today, including the much discussed ChatGPT<\/a> with its ability to generate written content in response to almost any command. ChatGPT\u2019s underlying technology is a neural network. <\/p>\n

As Cowan and Taylor stood and watched the machine work, they really had no idea exactly how it was managing to perform this task. The answer to Taylor\u2019s mystery machine brain can be found somewhere in its \u201canalog neurons\u201d, in the associations made by its machine memory and, most importantly, in the fact that its automated functioning couldn\u2019t really be fully explained. It would take decades for these systems to find their purpose and for that power to be unlocked.<\/p>\n

\n \"Black<\/a>
\n Jack Cowan, who played a key part in the development of neural networks from the 1950s onwards.<\/span>
\n University of Chicago Photographic Archive, Hanna Holborn Gray Special Collections Research Center.<\/a><\/span>
\n <\/figcaption><\/figure>\n

The term neural network incorporates a wide range of systems, yet centrally, according to IBM<\/a>, these \u201cneural networks \u2013 also known as artificial neural networks (ANNs) or simulated neural networks (SNNs) \u2013 are a subset of machine learning and are at the heart of deep learning algorithms\u201d. Crucially, the term itself and their form and \u201cstructure are inspired by the human brain, mimicking the way that biological neurons signal to one another\u201d.<\/p>\n

There may have been some residual doubt of their value in its initial stages, but as the years have passed AI fashions have swung firmly towards neural networks. They are now often understood to be the future of AI. They have big implications for us and for what it means to be human. We have heard echoes of these concerns recently<\/a> with calls to pause new AI developments for a six month period to ensure confidence in their implications. <\/p>\n

\n
\n \"\"
\n <\/span><\/p>\n<\/figcaption><\/figure>\n

This article is part of Conversation Insights<\/em><\/strong>
\n
The Insights team generates long-form journalism<\/a> derived from interdisciplinary research. The team is working with academics from different backgrounds who have been engaged in projects aimed at tackling societal and scientific challenges.<\/em><\/p>\n

\n

It would certainly be a mistake to dismiss the neural network as being solely about glossy, eye-catching new gadgets. They are already well established in our lives. Some are powerful in their practicality. As far back as 1989, a team led by Yann LeCun at AT&T Bell Laboratories used back-propagation techniques to train a system to recognise handwritten postal codes<\/a>. The recent announcement by Microsoft<\/a> that Bing searches will be powered by AI, making it your \u201ccopilot for the web\u201d, illustrates how the things we discover and how we understand them will increasingly be a product of this type of automation.<\/p>\n<\/p>\n

Drawing on vast data to find patterns AI can similarly be trained to do things like image recognition at speed \u2013 resulting in them being incorporated into facial recognition<\/a>, for instance. This ability to identify patterns has led to many other applications, such as predicting stock markets<\/a>.<\/p>\n

Neural networks are changing how we interpret and communicate too. Developed by the interestingly titled Google Brain Team<\/a>, Google Translate<\/a> is another prominent application of a neural network. <\/p>\n

You wouldn\u2019t want to play Chess or Shogi with one either. Their grasp of rules and their recall of strategies and all recorded moves means that they are exceptionally good at games (although ChatGPT seems to struggle with Wordle<\/a>). The systems that are troubling human Go players (Go is a notoriously tricky strategy board game) and Chess grandmasters, are made from neural networks<\/a>.<\/p>\n

But their reach goes far beyond these instances and continues to expand. A search of patents restricted only to mentions of the exact phrase \u201cneural networks\u201d produces 135,828 results. With this rapid and ongoing expansion, the chances of us being able to fully explain the influence of AI may become ever thinner. These are the questions I have been examining in my research and my new book on algorithmic thinking<\/a>.<\/p>\n

Mysterious layers of \u2018unknowability\u2019<\/h2>\n

Looking back at the history of neural networks tells us something important about the automated decisions that define our present or those that will have a possibly more profound impact in the future. Their presence also tells us that we are likely to understand the decisions and impacts of AI even less over time. These systems are not simply black boxes, they are not just hidden bits of a system that can\u2019t be seen or understood.<\/p>\n

It is something different, something rooted in the aims and design of these systems themselves. There is a long-held pursuit of the unexplainable. The more opaque, the more authentic and advanced the system is thought to be. It is not just about the systems becoming more complex or the control of intellectual property limiting access (although these are part of it). It is instead to say that the ethos driving them has a particular and embedded interest in \u201cunknowability\u201d. The mystery is even coded into the very form and discourse of the neural network. They come with deeply piled layers \u2013 hence the phrase deep learning \u2013 and within those depths are the even more mysterious sounding \u201chidden layers\u201d. The mysteries of these systems are deep below the surface.<\/p>\n

There is a good chance that the greater the impact that artificial intelligence comes to have in our lives the less we will understand how or why. Today there is a strong push for AI that is explainable. We want to know how it works and how it arrives at decisions and outcomes. The EU is so concerned by the potentially \u201cunacceptable risks\u201d and even \u201cdangerous\u201d applications that it is currently advancing a new AI Act<\/a> intended to set a \u201cglobal standard\u201d for \u201cthe development of secure, trustworthy and ethical artificial intelligence\u201d.<\/p>\n

Those new laws will be based on a need for explainability, demanding that<\/a> \u201cfor high-risk AI systems, the requirements of high quality data, documentation and traceability, transparency, human oversight, accuracy and robustness, are strictly necessary to mitigate the risks to fundamental rights and safety posed by AI\u201d. This is not just about things like self-driving cars (although systems that ensure safety fall into the EU\u2019s category of high risk AI), it is also a worry that systems will emerge in the future that will have implications for human rights. <\/p>\n

This is part of wider calls for transparency in AI so that its activities can be checked, audited and assessed. Another example would be the Royal Society\u2019s policy briefing on explainable AI<\/a> in which they point out that \u201cpolicy debates across the world increasingly see calls for some form of AI explainability, as part of efforts to embed ethical principles into the design and deployment of AI-enabled systems\u201d.<\/p>\n

But the story of neural networks tells us that we are likely to get further away from that objective in the future, rather than closer to it.<\/p>\n

Inspired by the human brain<\/h2>\n

These neural networks may be complex systems yet they have some core principles. Inspired by the human brain, they seek to copy or simulate forms of biological and human thinking. In terms of structure and design they are, as IBM also explains<\/a>, comprised of \u201cnode layers, containing an input layer, one or more hidden layers, and an output layer\u201d. Within this, \u201ceach node, or artificial neuron, connects to another\u201d. Because they require inputs and information to create outputs they \u201crely on training data to learn and improve their accuracy over time\u201d. These technical details matter but so too does the wish to model these systems on the complexities of the human brain.<\/p>\n

Grasping the ambition behind these systems is vital in understanding what these technical details have come to mean in practice. In a 1993 interview<\/a>, the neural network scientist Teuvo Kohonen concluded that a \u201cself-organising\u201d system \u201cis my dream\u201d, operating \u201csomething like what our nervous system is doing instinctively\u201d. As an example, Kohonen pictured how a \u201cself-organising\u201d system, a system that monitored and managed itself, \u201ccould be used as a monitoring panel for any machine \u2026 in every airplane, jet plane, or every nuclear power station, or every car\u201d. This, he thought, would mean that in the future \u201cyou could see immediately what condition the system is in\u201d. <\/p>\n

\n \"A
\n Early computing often involved a large apparatus of assembled parts.<\/span>
\n Aalto University Archives<\/span><\/span>
\n <\/figcaption><\/figure>\n

The overarching objective was to have a system capable of adapting to its surroundings. It would be instant and autonomous, operating in the style of the nervous system. That was the dream, to have systems that could handle themselves without the need for much human intervention. The complexities and unknowns of the brain, the nervous system and the real world would soon come to inform the development and design of neural networks.<\/p>\n

\u2018Something fishy about it\u2019<\/h2>\n

But jumping back to 1956 and that strange learning machine, it was the hands-on approach that Taylor had taken when building it that immediately caught Cowan\u2019s attention. He had clearly sweated over the assembly of the bits and pieces. Taylor, Cowan observed<\/a> during an interview on his own part in the story of these systems, \u201cdidn\u2019t do it by theory, and he didn\u2019t do it on a computer\u201d. Instead, with tools in hand, he \u201cactually built the hardware\u201d. It was a material thing, a combination of parts, perhaps even a contraption. And it was \u201call done with analogue circuitry\u201d taking Taylor, Cowan notes, \u201cseveral years to build it and to play with it\u201d. A case of trial and error.<\/p>\n

Understandably Cowan wanted to get to grips with what he was seeing. He tried to get Taylor to explain this learning machine to him. The clarifications didn\u2019t come. Cowan couldn\u2019t get Taylor to describe to him how the thing worked. The analogue neurons remained a mystery. The more surprising problem, Cowan thought, was that Taylor \u201cdidn\u2019t really understand himself what was going on\u201d. This wasn\u2019t just a momentary breakdown in communication between the two scientists with different specialisms, it was more than that.<\/p>\n

In an interview from the mid-1990s<\/a>, thinking back to Taylor\u2019s machine, Cowan revealed that \u201cto this day in published papers you can\u2019t quite understand how it works\u201d. This conclusion is suggestive of how the unknown is deeply embedded in neural networks. The unexplainability of these neural systems has been present even from the fundamental and developmental stages dating back nearly seven decades. <\/p>\n

This mystery remains today and is to be found within advancing forms of AI. The unfathomability of the functioning of the associations made by Taylor\u2019s machine led Cowan to wonder if there was \u201csomething fishy about it\u201d.<\/p>\n

Long and tangled roots<\/h2>\n

Cowan referred back to his brief visit with Taylor when asked about the reception of his own work some years later. Into the 1960s people were, Cowan reflected, \u201ca little slow to see the point of an analogue neural network\u201d. This was despite, Cowan recalls, Taylor\u2019s 1950s work on \u201cassociative memory\u201d being based on \u201canalog neurons\u201d. The Nobel Prize-winning neural systems expert, Leon N. Cooper, concluded<\/a> that developments around the application of the brain model in the 1960s, were regarded \u201cas among the deep mysteries\u201d. Because of this uncertainty there remained a scepticism about what a neural network might achieve. But things slowly began to change.<\/p>\n

Some 30 years ago the neuroscientist Walter J. Freeman, who was surprised by the \u201cremarkable<\/a>\u201d range of applications that had been found for neural networks, was already commenting on the fact that he didn\u2019t see them as \u201ca fundamentally new kind of machine\u201d. They were a slow burn, with the technology coming first and then subsequent applications being found for it. This took time. Indeed, to find the roots of neural network technology we might head back even further than Cowan\u2019s visit to Taylor\u2019s mysterious machine. <\/p>\n

The neural net scientist James Anderson and the science journalist Edward Rosenfeld have noted<\/a> that the background to neural networks goes back into the 1940s and some early attempts to, as they describe, \u201cunderstand the human nervous systems and to build artificial systems that act the way we do, at least a little bit\u201d. And so, in the 1940s, the mysteries of the human nervous system also became the mysteries of computational thinking and artificial intelligence.<\/p>\n

Summarising this long story, the computer science writer Larry Hardesty has pointed out<\/a> that deep learning in the form of neural networks \u201chave been going in and out of fashion for more than 70 years\u201d. More specifically, he adds, these \u201cneural networks were first proposed in 1944 by Warren McCulloch and Walter Pitts, two University of Chicago researchers who moved to MIT in 1952 as founding members of what\u2019s sometimes called the first cognitive science department\u201d.<\/p>\n

\n \"Black
\n The inventors of the neural network Walter Pitts and Warren McCulloch pictured here in 1949.<\/span>
\n Semantic Scholar<\/a><\/span>
\n <\/figcaption><\/figure>\n

Elsewhere, 1943<\/a> is sometimes the given date as the first year for the technology. Either way, for roughly 70 years accounts suggest that neural networks have moved in and out of vogue, often neglected but then sometimes taking hold and moving into more mainstream applications and debates. The uncertainty persisted. Those early developers frequently describe the importance of their research as being overlooked, until it found its purpose often years and sometimes decades later.<\/p>\n

Moving from the 1960s into the late 1970s we can find further stories of the unknown properties of these systems. Even then, after three decades, the neural network was still to find a sense of purpose. David Rumelhart, who had a background in psychology and was a co-author of a set of books published in 1986 that would later drive attention back again towards neural networks, found himself collaborating on the development of neural networks with his colleague Jay McClelland<\/a>. <\/p>\n

As well as being colleagues they had also recently encountered each other at a conference in Minnesota where Rumelhart\u2019s talk on \u201cstory understanding\u201d had provoked some discussion among the delegates.<\/p>\n

Following that conference McClelland returned with a thought about how to develop a neural network that might combine models to be more interactive. What matters here is Rumelhart\u2019s recollection<\/a> of the \u201chours and hours and hours of tinkering on the computer\u201d.<\/p>\n

\n

We sat down and did all this in the computer and built these computer models, and we just didn\u2019t understand them. We didn\u2019t understand why they worked or why they didn\u2019t work or what was critical about them.<\/p>\n<\/blockquote>\n

Like Taylor, Rumelhart found himself tinkering with the system. They too created a functioning neural network and, crucially, they also weren\u2019t sure how or why it worked in the way that it did, seemingly learning from data and finding associations.<\/p>\n

Mimicking the brain – layer after layer<\/h2>\n

You may already have noticed that when discussing the origins of neural networks the image of the brain and the complexity this evokes are never far away. The human brain acted as a sort of template for these systems. In the early stages, in particular, the brain \u2013 still one of the great unknowns \u2013 became a model for how the neural network might function. <\/p>\n

\n \"Design
\n The model of the brain became a model for the layering within artificial neural networks.<\/span>
\n Shutterstock\/CYB3RUSS<\/a><\/span>
\n <\/figcaption><\/figure>\n

So these experimental new systems were modelled on something whose functioning was itself largely unknown. The neurocomputing engineer Carver Mead has spoken revealingly<\/a> of the conception of a \u201ccognitive iceberg\u201d that he had found particularly appealing. It is only the tip of the iceberg of consciousness of which we are aware and which is visible. The scale and form of the rest remains unknown below the surface.<\/p>\n

In 1998, James Anderson<\/a>, who had been working for some time on neural networks, noted that when it came to research on the brain \u201cour major discovery seems to be an awareness that we really don\u2019t know what is going on\u201d.<\/p>\n

In a detailed account in the Financial Times in 2018<\/a>, technology journalist Richard Waters noted how neural networks \u201care modelled on a theory about how the human brain operates, passing data through layers of artificial neurons until an identifiable pattern emerges\u201d. This creates a knock-on problem, Waters proposed, as \u201cunlike the logic circuits employed in a traditional software program, there is no way of tracking this process to identify exactly why a computer comes up with a particular answer\u201d. Waters\u2019 conclusion is that these outcomes cannot be unpicked. The application of this type of model of the brain, taking the data through many layers, means that the answer cannot readily be retraced. The multiple layering is a good part of the reason for this.<\/p>\n

Hardesty<\/a> also observed these systems are \u201cmodelled loosely on the human brain\u201d. This brings an eagerness to build in ever more processing complexity in order to try to match up with the brain. The result of this aim is a neural net that \u201cconsists of thousands or even millions of simple processing nodes that are densely interconnected\u201d. Data moves through these nodes in only one direction. Hardesty observed that an \u201cindividual node might be connected to several nodes in the layer beneath it, from which it receives data, and several nodes in the layer above it, to which it sends data\u201d.<\/p>\n

Models of the human brain were a part of how these neural networks were conceived and designed from the outset. This is particularly interesting when we consider that the brain was itself a mystery of the time (and in many ways still is). <\/p>\n

\u2018Adaptation is the whole game\u2019<\/h2>\n

Scientists like Mead and Kohonen wanted to create a system that could genuinely adapt to the world in which it found itself. It would respond to its conditions. Mead was clear that the value in neural networks was that they could facilitate this type of adaptation. At the time, and reflecting on this ambition, Mead added<\/a> that producing adaptation \u201cis the whole game\u201d. This adaptation is needed, he thought, \u201cbecause of the nature of the real world\u201d, which he concluded is \u201ctoo variable to do anything absolute\u201d.<\/p>\n

This problem needed to be reckoned with especially as, he thought, this was something \u201cthe nervous system figured out a long time ago\u201d. Not only were these innovators working with an image of the brain and its unknowns, they were combining this with a vision of the \u201creal world\u201d and the uncertainties, unknowns and variability that this brings. The systems, Mead thought, needed to be able to respond and adapt to circumstances without<\/em> instruction.<\/p>\n

Around the same time in the 1990s, Stephen Grossberg \u2013 an expert in cognitive systems working across maths, psychology and bioemedical engineering \u2013 also argued that<\/a> adaptation was going to be the important step in the longer term. Grossberg, as he worked away on neural network modelling, thought to himself that it is all \u201cabout how biological measurement and control systems are designed to adapt quickly and stably in real time to a rapidly fluctuating world\u201d. As we saw earlier with Kohonen\u2019s \u201cdream\u201d of a \u201cself-organising\u201d system, a notion of the \u201creal world\u201d becomes the context in which response and adaptation are being coded into these systems. How that real world is understood and imagined undoubtedly shapes how these systems are designed to adapt.<\/p>\n

Hidden layers<\/h2>\n

As the layers multiplied, deep learning plumbed new depths. The neural network is trained using training data that, Hardesty explained<\/a>, \u201cis fed to the bottom layer \u2013 the input layer \u2013 and it passes through the succeeding layers, getting multiplied and added together in complex ways, until it finally arrives, radically transformed, at the output layer\u201d. The more layers, the greater the transformation and the greater the distance from input to output. The development of Graphics Processing Units (GPUs), in gaming for instance, Hardesty added, \u201cenabled the one-layer networks of the 1960s and the two to three- layer networks of the 1980s to blossom into the ten, 15, or even 50-layer networks of today\u201d. <\/p>\n

Neural networks are getting deeper. Indeed, it\u2019s this adding of layers, according to Hardesty, that is \u201cwhat the \u2018deep\u2019 in \u2018deep learning\u2019 refers to\u201d. This matters, he proposes, because \u201ccurrently, deep learning is responsible for the best-performing systems in almost every area of artificial intelligence research\u201d.<\/p>\n

But the mystery gets deeper still. As the layers of neural networks have piled higher their complexity has grown. It has also led to the growth in what are referred to as \u201chidden layers\u201d within these depths. The discussion of the optimum number of hidden layers in a neural network is ongoing. The media theorist Beatrice Fazi has written<\/a> that \u201cbecause of how a deep neural network operates, relying on hidden neural layers sandwiched between the first layer of neurons (the input layer) and the last layer (the output layer), deep-learning techniques are often opaque or illegible even to the programmers that originally set them up\u201d. <\/p>\n

As the layers increase (including those hidden layers) they become even less explainable \u2013 even, as it turns out, again, to those creating them. Making a similar point, the prominent and interdisciplinary new media thinker Katherine Hayles also noted<\/a> that there are limits to \u201chow much we can know about the system, a result relevant to the \u2018hidden layer\u2019 in neural net and deep learning algorithms\u201d.<\/p>\n

Pursuing the unexplainable<\/h2>\n

Taken together, these long developments are part of what the sociologist of technology Taina Bucher<\/a> has called the \u201cproblematic of the unknown\u201d. Expanding his influential research on scientific knowledge into the field of AI, Harry Collins has pointed out that<\/a> the objective with neural nets is that they may be produced by a human, initially at least, but \u201conce written the program lives its own life, as it were; without huge effort, exactly how the program is working can remain mysterious\u201d. This has echoes of those long-held dreams of a self-organising system. <\/p>\n

I\u2019d add to this that the unknown and maybe even the unknowable have been pursued as a fundamental part of these systems from their earliest stages. There is a good chance that the greater the impact that artificial intelligence comes to have in our lives the less we will understand how or why.<\/p>\n

But that doesn\u2019t sit well with many today. We want to know how AI works and how it arrives at the decisions and outcomes that impact us. As developments in AI continue to shape our knowledge and understanding of the world, what we discover, how we are treated, how we learn, consume and interact, this impulse to understand will grow. When it comes to explainable and transparent AI, the story of neural networks tells us that we are likely to get further away from that objective in the future, rather than closer to it.<\/p>\n

\n
\n \"\"
\n <\/span><\/p>\n<\/figcaption><\/figure>\n

For you: more from our Insights series<\/a>:<\/em><\/p>\n