Prologue: The Age of Artificial Intelligence

The last couple of years undoubtedly mark the begining of the Age of Artificial Intelligence. The release of ChatGPT was the iPhone moment of AI. While it isn’t necessarily a fundamental breakthrough in the scientific or technical advancement of the field, it undeniably marks the inception of the widespread of AI. It has already had a significant cultural impact. It has reshaped public perception and discourse surrounding AI, bringing awareness of its potential and ability to influence our lives.

For a long time, Artificial Intelligence has been in the collective imagination, from its portrayal in science fiction to many mildly successful attempts at practical applications. However, with ChatGPT’s release and the subsequent surge in large language models and chatbots, AI has finally become a fundamental technology for the general population, even if it is nascent.

Reasonably enough, many skeptics –myself included on occasions– question AI’s authenticity and transformative potential in its current form beyond the hype and excitement of a shiny new toy. Debates on whether AI is merely an illusion or a revolutionary force happen daily across all traditional and digital media.

In this last issue of Mostly Harmless for 2023, we will review the evolving promises and expectations of Generative AI, highlighting the essential challenges and unresolved issues necessary for the technology to realize its intended potential.

Approaching the subject with a healthy balance of skepticism and optimism, I will critique the current state of generative AI from the perspective of a deeply optimist machine learning researcher who loves the field. I will re-examine the most impactful milestones that brought us here and assess the challenges and obstacles ahead.

By the end, I hope you share with me the excitement of being in the early days of what could potentially be the most important invention of our civilization, with the necessary suspicion and open-mindedness to understand the path forward isn’t guaranteed.

The road so far

Let’s start by looking back at the last few years in generative AI, from the inception of the transformer architecture to the release of the first widely useful AI application.

When discussing generative AI, we can address different domains: language/text, images, and audio/music. Arguably, the most impactful advancements are in the text domain due to the emergence of large language models. The other domains have also seen incredible progress with the development of, e.g., diffusion models allowing for controllable text-to-image generation and realistic text-to-speech and speech-to-text conversion.

Thus, I want to specifically focus on language, first because it’s my area of expertise, and second, because the majority of economically valuable jobs projected to be automated in the near future will be due to the existence of language models. While the other domains certainly have exciting applications in art, marketing, video games, and content creation, the potential reach of very powerful language models with real language understanding surpasses that of generative sound, music, and image combined.

The reason is simple. Language is the primary means of human communication, interaction with the social environment, collaboration, and task-solving. Natural language is frequently used to describe tasks and their solutions. Therefore, solving natural language understanding and developing a machine capable of comprehending instructions, following them, and communicating back in a natural language could potentially be the most transformative technology we could ever invent. We’re still far from full natural language understanding, though. But, at the same time, we’re so much father ahead than anyone seriously involved in AI in the late 2010’s could predict.

Let’s turn our attention to language modeling, then. At the moment of writing, there is no doubt GPT-4 is the state of the art, at least in terms of being the most powerful, widely available general-purpose language model. That might not be the case in a couple of months when Gemini is finally out, but still, looking over at the story of the GPT series seems like the best way to understand how we got here and where we might go next.

How to train your chatbot

The history of chatbots goes all the way back to ELIZA in 1962, but their most recent incarnation started, at least in the public mind, with the advent of GPT-2 in 2019. Its ability to fabricate stories about talking unicorns found in the Peruvian mountains, a significant leap from anything we’d seen before, blew our minds. Sure, we had made strides in translation and text classification with bare transformers, but nothing came close to GPT-2’s coherence in generating general-purpose text. Suddenly, an AI model could write a short story!

This was also the first time researchers decided not to immediately release the weights of the model, apparently not for any commercial reasons but for fear of it being misused to generate fake news and impersonate people online. Oh, the irony!

Next year, GPT-3 shook things up with its paper on large language models as few-shot learners, marking the first significant breakthrough in transfer learning. Transfer learning is a pivotal area in AI, exploring how quickly we can adapt to new tasks similar to ones we’ve already learned without starting from scratch. This was the focus of the GPT-3 paper, and the first time we had a hint that such a thing as general-purpose language models could exist.

The GPT-3 paper reveals something many linguists already believed: language understanding is a general-purpose task. Furthermore, text completion is also a very general-purpose task. If you excel at completing sentences across various domains, you also excel at a wide range of tasks that aren’t trivially just text completion, as many tasks can essentially be boiled down to “please complete the following text.” Summarizing text, translating to another language, or solving math problems are all examples of tasks easily framed as text completion.

However, there’s a catch. While models like GPT-3 excel at predicting and completing text, they struggle to follow specific instructions. This happens because the training data doesn’t resemble instructions. So, even though the model may know how to summarize or translate, you have to figure out how these tasks are presented in the training data to prompt the model effectively. For example, to summarize, you would give the model your text and append “TL-DR;.”

Learning to follow instructions

What if you want to give clear instructions to the model that allow for some flexibility in how you provide them? This is crucial because multiple ways exist to fill in a given text. For instance, when you ask GPT-3 a question, the text can be completed by providing an answer. Still, it is also valid to continue with similar questions, as vast collections of questions are available on the internet. The model doesn’t know whether you want more questions or the answer to the previous question.

That’s where instruction fine-tuning comes in, as described in the InstructGPT paper. This approach enables you to train your model to better understand and respond to your instructions, whether asking it to solve an equation, summarize a text, generate ideas, or perform other tasks.

With instruction fine-tuning, the model can be guided to prioritize answering questions over simply executing completion. This approach functions by adjusting the likelihood of responses to questions. For this method to be effective, the model must possess a probable and correct answer to the question; we are just nudging it to value that answer more likely than other non-answer completions.

However, this is still not enough. For widespread use, you need a model willing to reject problematic instructions, friendly, and committed to providing factual and accurate information. The desired responses should align with user preferences and maintain a respectful and non-discriminatory tone, using generally informal yet polite language. In other words, you need a model that is knowledgeable (pretrained in the whole internet), helpful (can follow instructions), and nice. This is where reinforcement learning with human feedback (RLHF) comes into play, also presented in this paper.

Don’t be evil… please?

In RLHF, we take an instruction fine-tuned model and further adjust it to score better answers that align with user preferences —such as being more or less concise, using more or less formal language, or choosing to answer specific topics. This cannot be accomplished with supervised training because the range of possible human preferences is extensive and challenging to model in a dataset.

Researchers tackled this problem by turning to reinforcement learning. In this approach, we allow the model to provide multiple responses to a prompt rather than showing the model examples of good answers. A human evaluator then ranks and scores these responses based on their preferences for factuality, relevance, length and depth, language style, non-discrimination, etc. This ranking process encodes the human values we desire in a model but cannot program explicitly.

RLHF thus involves teaching the language model to rank its answers based on human preferences. The ranking model is then used to further train the language model, resulting in answers more aligned with user preferences.

From an engineering standpoint, this final puzzle piece transformed language models into usable applications. Just like, we had all the components leading to the iPhone, but the genius move was to put them together in a commercially viable and user-friendly format. ChatGPT is the iPhone of the AI revolution because it embodies several innovations in a commercially plausible manner, making it accessible and valuable to all users, not just experts or researchers.

And that’s the very brief story of how we got to ChatGPT. But that’s only the beginning. In the last year since ChatGPT was released, a new industry has grown around large language models. Let’s review some of the most interesting trends that make 2023 the year of artificial intelligence.

The Rise of AI

Following ChatGPT, models have significantly diversified, with big players like Google, Meta, Apple, Twitter/X, Amazon, and many more training their large models. This competition has driven innovation: AI is now where the Big Tech giants fight each other.

Various open-source models have also emerged, stemming from Facebook Llama 1 and spinning off into a full Llamaverse. This has led to developing more specialized models, fine-tuned for specific domains and purposes, such as code generation, question-answering from databases, and more.

The open-source and closed-source models offer a wide range of trade-offs, and projects and wrappers are in place to seamlessly switch between these models, allowing for greater transparency and flexibility. This mirrors the trend in open source of creating diverse infrastructure and tools, as well as the emergence of frameworks that support various technologies. Just as we now have numerous options for databases, programming languages, and web frameworks, the field of language models is experiencing a similar surge in diversity and innovation.

The second major trend is multimodality. Although we focus on language models here, we must recognize the incredible development in visual and audio models. These models can recognize and analyze images, generate images and videos, and create audio outputs based on text prompts. Combining these separate models into multidomain models can solve problems requiring multiple modalities simultaneously. For instance, giving it a picture, asking questions about the image, modifying it, and generating new images based on it.

A third major trend is the rise of language models as application platforms. This means interfacing LLMs with plugins and applications and connecting with external code. ChatGPT introduced the idea of plugins that let you interact with external services such as knowledge bases, search engines, or APIs for specific products like a store website or booking service.

This concept has since evolved into language chains, where you can connect a linguistic interface with non-linguistic tools or services. This involves using a large language model to link with databases, retrieve relevant information, call APIs, perform actions, and compute results. With this setup, you can build applications, known as agents, that are language-driven and capable of interacting with the Internet to carry out tasks on your behalf. These agents go beyond just processing language; they can manage finances, build websites, and more, functioning closer to real assistants.

A final trend I want to highlight is the emergence of a new field: prompt engineering. It started as an informal practice due to the need for writing effective prompts, but has since evolved into a semi-formal discipline. Since these models aren’t perfect and don’t understand everything perfectly yet, the way you ask for information can affect the results you get. As a result, folks are working on figuring out best practices for crafting good prompts as these models continue to evolve.

Some experts are skeptical about the long-term future of prompt engineering, believing that as models get better at understanding, it won’t matter how you prompt them. Others think prompt engineering will always be necessary, in the same sense that design patterns, naming and style conventions, and other programming practices remain relevant regardless of how powerful programming languages get.