Paper Trail

The Innovation Illusion: Why Ideas Aren’t Actually Getting Harder to Find

May 19, 202611:16Paper Trail

This episode challenges the common perception that the pace of innovation is slowing, arguing instead that the effort required to generate new ideas has increased exponentially. Listeners will learn that while productivity growth has indeed decelerated, this is due to the ever-growing investment in research and development needed to push the frontier, rather than a lack of new ideas or declining human ingenuity. Using examples like Moore's Law and agricultural yields, the discussion illustrates how maintaining progress now demands significantly more resources just to stay in place.

Key Takeaways

Detailed Report

The Cost of Progress: Why Innovation Isn't Slowing, Just Getting More Expensive

Unpacking the "Innovation Illusion"

There's a common belief that the pace of innovation is slowing, with many suggesting that the "low-hanging fruit" of discovery has already been picked. However, recent research challenges this perception, arguing that it's an illusion. The paper suggests that new ideas aren't inherently harder to find; instead, the effort required to discover them has dramatically increased. This situation is akin to "running faster just to stay in the same place," where significant effort is expended merely to maintain the current rate of progress.

The Productivity Paradox and Research Effort

Macroeconomic data generally supports the observation that productivity growth, particularly Total Factor Productivity (TFP), has slowed significantly since the mid-2000s. The research paper reconciles this widely accepted fact by introducing a crucial missing variable: the enormous increase in global research effort. The authors propose a framework where the "cost of new ideas" has been rising exponentially. This means that to maintain any given rate of productivity growth, an ever-larger share of resources—including scientists, engineers, and R&D budgets—has been needed for research and development. It's not that the marginal productivity of an individual researcher has plummeted, but rather that the aggregate system requires substantially more inputs to achieve similar outputs as before.

Evidence Across Sectors

#### The True Cost of Moore's Law

Moore's Law, which predicts the doubling of transistors on a microchip approximately every two years, has long been a bedrock of technological progress. While it appears as a consistent exponential trend, the paper highlights that this is not an automatic process. Maintaining this doubling rate has required an exponentially increasing investment in research and development. For instance, the number of researchers required to sustain Moore's Law has been doubling roughly every 1.5 years. The authors estimate that the research effort behind doubling transistor density today is more than 100 times greater than it was in the early 1970s. Without this escalating effort, Moore's Law would have ceased to hold true decades ago.

#### Agriculture and Medicine

The patterns observed in high-tech sectors like microchips are not isolated. The research extends its analysis to other critical domains, including agriculture and medical research. In agriculture, incredible gains in crop yields have been achieved over the last century, but the rate of increase in yields per researcher has slowed significantly. To achieve the next percentage point of improvement, more plant geneticists, agronomists, and sophisticated lab work are required. Similarly, in medicine, while the discovery of penicillin was a landmark achievement that was relatively straightforward in its time, developing a new drug today involves decades of research, massive clinical trials, and multi-billion dollar investments. The "low-hanging fruit" metaphor, often implying a finite number of easy discoveries, is challenged; even if the fruit is higher up, humanity has developed longer ladders, better fruit-picking machines, and exponentially more pickers to reach it.

Quantifying the Cost of Ideas

The researchers developed a structural model that connects research effort, measured by the number of researchers, to TFP growth. The core of their approach involved estimating a parameter that quantifies how many researchers are needed to produce a given amount of TFP growth. This allowed them to create a counterfactual scenario: what TFP growth would have been if the number of researchers had remained constant or grown at a slower rate. Their findings revealed a robust pattern: the number of researchers required to generate a constant rate of TFP growth has been rising exponentially. This implies that the "research productivity" – the TFP growth generated per researcher – has been falling significantly. The "illusion" stems from comparing today's TFP growth rates with those of the past without accounting for the vastly different inputs of research effort.

Implications for the Future of Innovation

The paper does not offer specific policy prescriptions on *how* to make research cheaper. Instead, its primary implication is that *more* research is needed, not less. The authors argue that the observed slowdown in productivity growth would be far worse without the massive increase in research effort that has already occurred. Therefore, reducing research investment would be catastrophic for future growth. The research serves as a powerful reminder that economic growth and improved living standards are not inevitable; they are the result of conscious, sustained, and increasingly costly investments in knowledge creation. Humanity's ability to innovate is not exhausted, but the process is becoming more resource-intensive, requiring a continuous push of the boundaries of what's possible, even if it feels like more effort is required for each marginal gain. This presents a significant challenge: how can research efforts continue to be scaled in a way that sustains innovation, given that the returns per individual researcher appear to be diminishing?

Show Notes

Works Referenced

  • Are Ideas Getting Harder to Find?: This seminal paper argues that while Total Factor Productivity (TFP) growth has slowed, the effort (measured by researchers and R&D investment) required to maintain innovation has increased exponentially, suggesting that ideas are not harder to find but more costly to discover.

Glossary

  • Total Factor Productivity (TFP): A measure of how efficiently inputs like labor and capital are used in production, indicating overall economic productivity and technological progress.
  • Moore's Law: An observation that the number of transistors on a microchip doubles approximately every two years, leading to exponential growth in computing power.
  • Cost of New Ideas: The exponentially increasing research effort, human capital, and financial resources required to generate a constant rate of technological progress and economic growth.
  • Low-hanging fruit: A metaphor describing easily achievable goals or discoveries that require minimal effort, often implying that these have already been exploited.

Full Transcript

HostThere's a common perception that the pace of innovation is slowing down. You hear it everywhere, from think tanks to tech blogs, this idea that the "low-hanging fruit" has been picked, and new ideas are just harder to come by.
ExpertThat perception, as compelling as it sounds, turns out to be an illusion, at least according to a recent paper. What the research suggests is that ideas aren't actually getting *harder* to find, but rather, the effort required to find them has dramatically increased.
HostSo, it's not that people are less clever or the well is running dry, but that they're simply working a lot harder for the same, or even slightly less, output? That really turns the common narrative on its head.
ExpertPrecisely. It's a critical distinction. The paper argues that if this exponentially increasing effort *weren't* being put in, the productivity slowdown observed would be far more severe, potentially even negative. The situation is essentially one of running faster just to stay in the same place.
HostThat "running faster to stay in the same place" analogy is quite vivid. It's worth unpacking this idea. The general consensus, supported by macroeconomic data, is that productivity growth has indeed slowed significantly since the mid-2000s. Total Factor Productivity, or TFP, which measures how efficiently inputs are used, has seen a marked deceleration. How does this paper reconcile that widely accepted fact with its "innovation isn't harder" claim?
ExpertThe paper doesn't dispute the slowdown in TFP growth. What it does, instead, is introduce a crucial missing variable: the enormous increase in research effort. The authors propose a framework where the "cost of new ideas" has been rising exponentially. To maintain any given rate of productivity growth, an ever-larger share of resources—scientists, engineers, R&D budgets—has been needed for research and development.
HostSo, it’s not that the rate of discovery per researcher has plummeted to zero, but that the overall *system* needs more inputs to achieve similar outputs as before? It's a distinction between the marginal productivity of an individual researcher and the aggregate innovative capacity.
ExpertExactly. Think of it this way: if you plot the number of researchers against the rate of TFP growth, you see that while TFP growth has flattened, the number of researchers has exploded. The paper then models what TFP growth *would have been* without that massive increase in research effort. The finding is stark: without the dramatic rise in R&D, TFP growth would be far lower, possibly even negative. This implies that innovation is still happening, and in significant ways, but it’s becoming incredibly expensive.
HostOne of the most compelling examples the paper highlights is Moore's Law. For decades, it's been the bedrock of technological progress, predicting that the number of transistors on a microchip would double approximately every two years. Many view this as almost a natural law, a given. But the paper suggests something much more active is at play.
ExpertMoore's Law, while it appears as a consistent exponential trend in computing power, is not an automatic process. The paper points out that maintaining this doubling rate has required an exponentially increasing investment in research and development. For instance, the number of researchers required to sustain Moore's Law has been doubling roughly every 1.5 years.
HostSo, the perceived "naturalness" of Moore's Law masks a gargantuan, ever-growing effort. It's not a free lunch; it's a lunch that costs more and more to prepare each time.
ExpertPrecisely. The authors estimate that the research effort behind doubling transistor density today is more than 100 times greater than it was in the early 1970s. If this effort hadn't scaled up, Moore's Law would have ceased to hold true decades ago. This isn't evidence that ideas are getting harder to find *per se*, but that the frontier of knowledge is pushed forward by a proportionally greater mobilization of human capital and financial resources.
HostThis isn't just about microchips, though, is it? Does the paper find similar patterns in other sectors that might seem less "high-tech" or research-intensive?
ExpertIt does. The research extends its analysis to a couple of other critical domains: agricultural yields and medical research. Take agriculture, for example. Incredible gains in crop yields have been seen over the last century, feeding a rapidly growing global population. However, the rate of increase in yields per researcher has slowed significantly.
HostSo, to get that next percentage point of yield improvement, more plant geneticists, more agronomists, and more sophisticated lab work are needed?
ExpertExactly. The low-hanging fruit of basic hybridization or simple crop rotation might have been picked, but advanced genomic sequencing, precision agriculture, and complex agro-ecological systems require enormous, coordinated research efforts. The gains are still there, and they're crucial for food security, but the cost in terms of research input has climbed. The same pattern holds in medicine. The discovery of penicillin was a landmark, relatively straightforward achievement in its time. Now, developing a new drug involves decades of research, massive clinical trials, and multi-billion dollar investments.
HostThat's a powerful point. The "low-hanging fruit" metaphor often implies a finite number of easy discoveries, after which everything becomes exponentially harder. But what the paper seems to be saying is that even if the fruit is higher up, longer ladders, better fruit-picking machines, and exponentially more pickers have been developed and deployed to get it.
ExpertThat's a good way to frame it. The question isn't whether the fruit exists, but at what cost it can be reached. The paper’s central argument is that the observation of slowing productivity growth, when viewed alongside the massive increase in research effort, suggests that the underlying “difficulty” of innovation hasn't necessarily changed. What has changed is the *resource intensity* required to maintain or even slightly reduce the pace of progress. It challenges the inherent pessimism of the "ideas are getting harder to find" narrative by pointing to the sheer scale of effort being exerted.
HostRegarding the methodology, how did the researchers actually quantify this "cost of new ideas" and separate it from raw TFP growth? What was their identification strategy to make this claim?
ExpertThey developed a structural model that connects research effort—measured by the number of researchers—to TFP growth. The core of their approach involves estimating a parameter that captures how many researchers are needed to produce a given amount of TFP growth. This parameter allows them to essentially back-calculate what TFP growth would have been if the number of researchers had remained constant, or grown at a slower rate.
HostSo, they're essentially creating a counterfactual scenario. They're asking: if R&D departments and scientific communities hadn't scaled up to this unprecedented degree, where would the economy be?
ExpertThat's right. By doing so, they found a very robust pattern: the number of researchers required to generate a constant rate of TFP growth has been rising exponentially. This implies that the "research productivity" – the TFP growth generated per researcher – has been falling significantly. However, because the total number of researchers has grown even faster, the overall TFP growth has been maintained, albeit at a slower rate than in previous golden ages of innovation.
HostAnd that's where the "illusion" comes in, because the research effort is not typically held constant when TFP numbers are examined. Only the output side is usually seen.
ExpertExactly. The illusion stems from comparing today's TFP growth rates with those of the past without accounting for the vastly different inputs of research effort. When you factor in the sheer volume of human capital and financial resources now dedicated to R&D, what looks like a slowdown is actually a testament to sustained effort preventing an even more dramatic decline.
HostSo, if the cost of new ideas is rising exponentially, what are the implications? Does this mean society is trapped on a treadmill, where more and more resources have to be poured into research just to stay afloat? Or does the paper offer any solutions or policy implications?
ExpertThe paper doesn't necessarily offer policy prescriptions in terms of *how* to make research cheaper. Instead, its primary implication is that *more* research is needed, not less. The authors argue that the observed slowdown in productivity growth would be far worse without the massive increase in research effort that has already occurred. Therefore, reducing research investment would be catastrophic for future growth.
HostThat's a powerful warning. It suggests that if the current pace of progress is to be maintained, let alone accelerated, ways need to be found to sustain or even increase this exponentially rising research investment. It's not a question of whether innovation is possible, but whether it can be afforded.
ExpertIt's a reminder that economic growth and improved living standards are not inevitable. They are the result of conscious, sustained, and increasingly costly investments in knowledge creation. The paper highlights that humanity's ability to innovate is not exhausted, but rather, the process is becoming more resource-intensive, requiring a continuous push of the boundaries of what's possible, even if it feels like more effort is required for each marginal gain.
HostThe research offers several key insights. First, the perceived slowdown in innovation isn't due to ideas being inherently harder to find, but rather the exponentially increasing resources required to discover them.
ExpertSecond, the 'cost of new ideas' has risen dramatically across sectors, from microchips to medicine, meaning significantly more researchers and R&D investment are needed to achieve the same rate of progress as in the past.
HostAnd third, without this sustained and growing investment in research, the productivity slowdown observed would be far more severe, underscoring the critical role of R&D in maintaining economic growth.
ExpertUltimately, the paper presents a challenge: how can research efforts continue to be scaled in a way that sustains innovation, given that the returns per individual researcher appear to be diminishing? And what does this mean for how scientific discovery is structured and funded in the decades to come?