Humans have always been fascinated by themselves and their unique abilities. Understanding how reason and human thought work would be equivalent in scope to decoding the human genome and the Big Bang.
Unfortunately, looking at our DNA doesn’t help. As the evolutionary psychologist Thomas Suddendorf put it so beautifully: Our genome “uses the same dictionary as a tulip”. Human programming code is 99 percent identical to that of a chimpanzee and 50 percent identical to that of a banana. Many of the things we once thought made us unique, like language and tool use, are also found in other animals and in a growing number of artificial intelligence (AI) systems.
Science podcast: “The subtle difference. What makes us human?”
Kristian Kersting is a professor of AI and machine learning at the TU Darmstadt, co-director of the Hessian Center for AI (hessian.ai), investor in the Heidelberg AI start-up Aleph Alpha, author of the book (“How machines learn”) and sponsor of the German AI Prize 2019.
There are different ways to do this in AI. You can try to find AI models that are very good at a particular application and maybe even better than a human. But you can also strive for a “general AI” (Artificial General Intelligence, short: AGI) that can be used universally and has something like common sense.
Large AI language models fuel the dream of a general AI
Large Language Models (LLMs) are currently fueling the dream of general AI. Variants such as OpenAI’s GPT-3, Beijing Academy of Artificial Intelligence’s Wu Dao 2.0, Microsoft and Nvidia’s Megatron-Turing NLG, DeepMind’s AlphaCode and Gopher, or more recently OpenAI’s DALL·E 2, the Imagen AI system for image synthesis and Google’s PaLM rely on large amounts of data and gigantic neural networks with hundreds of billions of parameters. Megatron-Turing NLG consists of 530 billion parameters and PaLM of 540 billion, and the development is rapid: GPT-2 was presented in June 2020, Megatron-Turing NLG in 2021, and PaLM in April 2022.
After that came DeepMind’s Chinchilla, and Meta AI released its Open Pretrained Transformer (OPT-175B), a 175 billion parameter language model that is freely accessible and trained on publicly available datasets. Based on chinchilla hat DeepMind just released its Flamingo multimodal model and Gato presented in mid-May 2022. The AI agent Gato is not only multimodal, but also multitasking – he can play Atari, annotate images, chat, stack blocks with a real robotic arm and much more, deciding whether to use text, joint torques, button presses or other depending on the context issues tokens.
The next breakthroughs: “Game over”?
Nando de Freitas, a senior researcher at Google’s AI research division DeepMind and co-creator of Gato, confidently tweeted “Game Over.” According to him, Gato and his motto of “more data, parameters, modalities and tasks” are the path to general AI that we could still experience in our lifetime.
What the next technical breakthroughs in AI will be is still unclear, but the credo of scaling is grounded in past experience that LLMs’ capabilities scale with their size. The hope is not unjustified, because our human brains are currently much larger: we have an estimated 86 billion neurons with 100 trillion synapses. That is at least 185 times as many parameters as with current LLMs.
According to the scaling laws for neural language models, size and capabilities are correlated. Therefore, there is still some room for improvement in the development of large language models, and upcoming, even larger LLMs may surprise us with many more capabilities! On the other hand, the question remains open as to whether the scaling of size will continue indefinitely or whether (and from where) a limit of technological progress will be reached through scaling.
Natural intelligence at the heart of the debate
This progress not only harbors great economic potential for an AI circular economy, but also prompts intensive discussion in cognitive science: Do LLMs process words, sentences, language and images like the human brain? Are they building blocks of natural intelligence?
Brenden M. Lake, a cognitive scientist at New York University, was able to show that visual AI language models prefer to categorize objects by shape – a phenomenon known as shape bias, which is observed in children from the age of two. Gary Marcus, professor emeritus at New York University and founder and CEO of Robust.AI, counters that current LLMs like Gato don’t have much to do with natural intelligence because Gato doesn’t work like our brains and doesn’t learn like a child .
Whether isolated talents or general AI, to put it in the words of Federal Chancellor Olaf Scholz, we are experiencing a turning point: the world after the LLMs is no longer the same world as before.
AI circular economy “Made in Europe”
This turning point thrives on powerful and state-of-the-art AI ecosystems, as we know them from the USA and China. Only when universities, companies, start-ups and “AI supercomputers” come together can AI models of any size be designed to be open, secure, energy and storage efficient, fed with our data, our knowledge and our values and with other AI Combine techniques freely. LLMs need special AI hardware, which is currently available almost nowhere in Germany, and setting up and operating it requires special knowledge.
There are AI language models like EleutherAI’s GPT-NeoX-20B whose weights, training and evaluation code are open source and untrained by companies. But that’s a smaller order of magnitude. The AI systems are also becoming more European: The multimodal AI language model Luminous from the Heidelberg AI start-up Aleph Alpha speaks German, English, French, Italian and Spanish, among other things, and is based on the MAGMA language developed by Aleph Alpha together with the University of Heidelberg. Model that uses adapters to adapt pre-trained LLMs to new tasks. The Heidelberg-based company is currently the only European company that develops and offers this technology.
Design your own AI ecosystem for Germany
We now have to build a powerful and state-of-the-art AI ecosystem in Germany and in Europe. That costs money and requires the political will to shape things. Yes! But it’s worth it. This is the only way we can design AI systems according to our ideas and establish an AI circular economy “Made in Europe”: train once, use productively again and again. Ultimately, our own large language models also ensure our progress in decoding artificial and human intelligence.
To home page
#Commentary #Kristian #Kersting #turning #point #artificial #intelligence