The AI Awakening: Stanford's ECON295/CS323 Study Guide

The professor is Eric Bolson.

The professor sees the class as an exercise in co-creation and believes most of the knowledge will ultimately come from the students, the readings, and the work they do on exercises.

Many people have the impression that AI progress and Tech progress are going a little faster these days and having a bigger impact on the economy and society.

Potential reasons suggested include the buzz around large language models, Euphoria with stock like Nvidia, many large tech companies pursuing AI, easier access to compute and infrastructure, a self-fulfilling cycle where people realize the potential of using more compute and large-scale data sets, and increased investment in infrastructure.

A Pew survey found that about 18% of people in the US Workforce have used ChatGPT at work at least once.

More recently, consumers have been feeling the impact of AI more because of accessible interfaces like ChatGPT, which was one of the first examples available for free.

One perspective is that the perception of AI changing the workforce and revolutionizing the economy is far greater than the reality, with estimates suggesting only about $3 billion in generative AI software revenues in 2023, and most people's jobs not fundamentally changing today, despite tremendous future potential.

The 'bitter lesson' idea is that many advances in AI are due to building models that can better leverage data and scale with current compute, rather than purely algorithmic advances. Progress has often come from using more data and more compute, overwhelming techniques aimed at explicitly capturing knowledge.

The three things identified are more compute, a lot more data (especially digital data), and better algorithms/more parameters, such as advances in the Transformer model.

The Transformer model is considered a big invention, possibly one of the biggest in history, because it allows for managing more data and compute effectively, although it took time to realize its full power after its initial publication.

The commercialization of language models like ChatGPT and Gemini has made the technology accessible and interdisciplinary, allowing startups and companies to build upon existing infrastructure and technology, which contributes to the public buzz.

The economic changes and impact are lagging way behind the potential of the technology. Even if AI tech progress froze, business innovation and economic productivity growth would continue as people implement existing inventions.

A potential issue is running out of data, as recent models have been trained on almost all available data on the internet and in books. There's a question about where to get additional data and whether synthetic data can be effectively used.

AlphaZero is an example of synthetic data training; it was trained on zero human data but knew the rules of games like chess and Go, generating its own games to learn from billions or trillions of times.

A risk is that much web content is now generated by LLMs, and if web scrapers use this data to train the next generation of models, it could become dysfunctional.

A major socioeconomic risk is the unknown unknowns – not knowing how the technology will malfunction, which is scary despite expectations for policy or business applications.

Almost invariably, AI researchers say they were surprised and did not expect this level of improvement, indicating a genuine inflection or sea change in capabilities.

The main gap lies between the improving rate of technology (perhaps exponential) and the slower change or improvement in our business institutions, culture, and economic understanding.

Part of the professor's mission, and the goal of the class, is to close this gap by speeding up our understanding of AI and how economics, business processes, and institutions need to be updated, rather than trying to stop the technology.

The Industrial Revolution, ignited around 1775-1776, is used as an example where living standards began to grow at a compounded exponential rate, unlike the relatively stagnant standards before then despite major historical events.

The steam engine, specifically James Watt's improvement, is highlighted as the invention that ignited the Industrial Revolution by allowing the use of machines for physical work instead of human or animal muscles.

These are referred to as General Purpose Technologies (GPTs).

GPTs are pervasive (affecting broad sectors), able to be improved over time, and most importantly, able to spawn complimentary innovations.

AI might be the most general GPT because if one can 'solve intelligence' (as DeepMind's slogan suggests), it could be used to solve many other problems in various fields like the environment, health care, poverty, and inventing other things.

ImageNet is a dataset of about 14 million labeled images used for a contest on machine identification. Around 2012, Jeff Hinton's team introduced deep learning techniques (neural networks) which led to a steep inflection point, making machines very good at image and face recognition, in some cases better than humans.

When the field was founded in 1956, it was mostly focused on symbolic methods, with neural networks being very shallow due to limited computational power.

Expert systems were rule-based systems painstakingly hand-coded with if-then rules derived from human experts to perform tasks. They didn't scale well and were prone to errors.

With machine learning (sometimes called software 2.0), instead of humans telling the machine exactly what to do, the machine learns relationships (statistical relationships) between inputs and outputs from large amounts of data, allowing it to make predictions.

There is a 'Gold Rush' to find more and more applications where machine learning can be applied, particularly in areas with sufficient digital data on inputs and outputs.

An example is a robot weed killer that uses machine learning image recognition to identify unwanted plant species and destroy them with lasers automatically.

Unlike traditional machine learning which often requires labeled data (supervised learning), generative AI uses unsupervised or self-supervised learning, which scales a lot better as it doesn't require human annotators to label the data.

LLMs are trained using a self-supervised approach by predicting the next token or word in a sequence (like fill-in-the-blank). By covering up words in a large text corpus and having the model predict them, it learns relationships without requiring human labeling.

On the Uniform Bar Exam, GPT-3.5 performed better than about 10% of humans, while GPT-4 performed better than about 90% of humans, showing a significant improvement in its ability to solve such practical tasks.

'Scaling laws' are predictable relationships observed between increasing the amount of compute power, dataset size, and model parameters, and the resulting improvement in the LLM's ability to predict the next word, which correlates with other performance metrics.

'Stargate' is a very large data center being built by Microsoft and OpenAI, costing $100 billion, based on the belief that having more compute applied will lead to further progress in LLMs.

Predictions for when a general AI system (defined by specific difficult criteria including embodiment) will be devised and released have significantly moved closer. A couple of years ago, the estimate was 2057, last year 2040, and more recently 2031, likely influenced by the unexpected progress in LLMs.

Although Yan LeCun reportedly believes LLMs might be a 'dead end' scientifically and scaling laws won't continue indefinitely, he thinks they will still be worth 'trillions of dollars of impact', indicating significant economic value.

No, there is no economic law or theory that guarantees everyone will benefit evenly from technology. It's possible some people could stagnate or be made worse off, as has been true for some with high school education or less in the US over the past couple decades despite overall productivity growth.

The Turing test is the idea of making a machine indistinguishable from a human in conversation. More broadly, it's associated with the idea that AI means replicating humans as closely as possible.

The professor considers it a bad goal because it points research in the wrong direction by focusing primarily on creating technologies that are substitutes for humans (doing the same tasks), rather than complements that amplify human capabilities and make human labor more valuable.

Through most of history, most technologies have been complements, amplifying human labor and increasing its value rather than driving it down.

The 'Turing trap' is a scenario where technology primarily focuses on substituting human labor, potentially leading to infinite productivity but also zero labor income, concentrating wealth and power, and diminishing the bargaining power of laborers who become inessential to production.

The study looked at a call center where an LLM provided suggestions to human operators by analyzing call transcripts and identifying patterns from successful outcomes. The humans still talked to the customers.

The study found a significant benefit: people with access to the LLM technology were about 14% more productive, answering questions more accurately and solving problems more efficiently within a few months.

The least skilled and least experienced workers had about a 35% productivity improvement, while the most skilled workers had close to a 0% improvement. The system helped less experienced workers by making tacit knowledge from successful operators available.

Machine learning was better at solving very common problems because it had enough data to train on. Humans were better at dealing with rare, one-off, or 'tail case' problems where there were few or no examples in the training data.

Customers were happier (higher satisfaction and better sentiment), and operators were also happier and less likely to quit.

One paper estimated that about 80% of the US workforce would have at least 10% of their tasks affected by GPTs, impacting most workers to some extent, especially high-income workers.

The six levels describe automation ranging from no automation to full automation where the machine does everything (Level 5). This framework can potentially be applied to most tasks in the economy to describe the degree of human-machine collaboration versus full automation.

Freestyle chess was a period where humans and machines working together could beat the best chess computers and human players, demonstrating the power of human-machine collaboration. This is reportedly no longer true as machines have advanced.

The required components include typically two to four readings per week, weekly assignments, and a team project.

Students can submit questions via Slido and upvote other questions. They can also raise their hand to ask questions in class.

Students need to form teams by April 12th.

Teams should be diverse, with members from different programs (computer science, economics, business, engineering, etc.) and other dimensions of diversity.

Students can choose to work on research proposals or business plans.

No, the discussion sections are completely optional.

Topics include how to write a business plan and how to think about government policy.