A recent article (Lessons from 60 years of pharmaceutical innovation. Nature Rev. Drug Discov. 8, 959–968 (2009))1 presents a corporate perspective on the reasons why the annual output of new drugs during the past few decades has not increased, despite vast increases in spending on pharmaceutical research and development (R&D). Here, I provide a researcher's perspective on the issue, based on my experience in the industry during this time.

In my view the key cause is the change in the way research and researchers are treated by management. During the five decades I have spent in the industry, the primary goal of research has changed from discovery and social usefulness to making money, and the ruling philosophy today is micromanagement, efficiency and timelines. This is suited to selling automobiles but not to research, which is a creative enterprise.

First, real innovation requires contemplation. Creation is an intuitive leap beyond deductive reasoning from known facts. This cannot be forced. Sometimes a great idea appears, unbidden, that is off the prescribed track. Such ideas were seriously considered in the past, but are seldom today. Indeed, this can even lead researchers to pursue such ideas against orders, in secret.

Why did this change occur? The increase in size of the major pharmaceutical companies gave rise to more and more management layers between research heads and the bench scientists who are the creators. The sheer volume of periodic research reports makes it impossible for the top management layer to absorb them effectively, and so the data percolate slowly upwards through managers in the middle layers, who digest them for their superiors. These middle managers receive little or no credit for new discoveries, but get blamed if they support something that eventually fails. Therefore, the safest thing to do is to disparage anything new. If they kill an Einstein in the cradle, nobody will ever know.

The only ways to avoid this problem are to allow flexibility in planning and reduce the depth of the middle layers, either by splitting the companies into smaller ones or by their division into units with real autonomy. Of course, this principle has already been recognized and several large companies have aimed to implement strategies to become more 'biotech-like' in their structure. However, in my experience senior management have been loath to grant power to groups below them.

It is clear that in recent years real creativity has taken place increasingly in small, flexible companies that sometimes succeed in their goals. This is why the share of new drugs originating from small companies has increased from ∼23% to ∼70%1. However, once a small company becomes part of a large one — which is common once it has demonstrated some success (for example, by bringing a drug to market) — its innovative capacity is often lost.

Second, if size is detrimental to an innovative research culture, mergers between large companies should make things worse — and they do2,3. They have a strong negative personal impact on researchers and, consequently, the innovative research environment. For example, the merger of Bristol-Myers with Squibb in 1989, which I witnessed, was a scene of power grabs and disintegrating morale. Researchers who could get a good offer left the company, and the positions of those who remained were often decided by favouritism rather than talent. Productivity fell so low that an outside firm was hired to find out why. Of course, everyone knew what was wrong but few — if any — had the nerve to say it.

A third group of negative management-related developments are those related to assessing performance. Bonuses for performance have been structured in such a way as to encourage research managers to press their subordinates for something that looks good in the short term. The problem is that it does not actually have to be good, and so the bonuses do not create an incentive for managers to care whether a project fails or not, because they keep the money anyway. Thus the incentives of this group are pitted directly against those of the company. In my view bonuses related to position rather than performance would be better. Whether a bonus is granted or not should depend on overall company performance, so that a bonus for one person is a bonus for all (proportional to salary). In this way, everyone has incentives to pull together, rather than act just for themselves, and cheating is not rewarded. Detailed annual appraisals can also be destructive if they favour cheap success over more profound but risky research.

Dismissals are a fourth critical area. A confidential 6-month notice period would allow someone time to look elsewhere without the stigma of dismissal, but job terminations have become brutal, often done with only a few hours' notice. This is bad not only for those losing their jobs but also for the survivors and thus the company itself. The terrified survivors' impulse will be to concentrate on easy things, rather than those that are more innovative and thus more risky, to maximize their accomplishments. Prescriptive timelines produce the same result. This is a guaranteed way to crush innovative research. For innovation to flourish, you must treat people with respect and abolish fear.

Finally, another negative management trend has been to hire 'academic geniuses' into high positions to make 'quantum leaps' in R&D productivity. In my experience this has always been disastrous. They promised great things, spent recklessly, wrecked morale by belittling the regular staff and left with no significant accomplishments.

A better source of new ideas is to ask external expert research (not management) consultants who give scientific advice but do not have power over the staff, and the staff themselves. In the ranks are many excellent, creative scientists who should be encouraged to read, think and create new proposals that they can test, using ∼20% of their working time. Spending 20% less time on assigned projects is a small price to pay for a potentially groundbreaking innovation. This will occasionally produce something really valuable. Diversity in personnel will produce diversity in discoveries. It will also increase overall productivity by raising morale.

The converse — micromanagement — is the worst attribute of contemporary research management; it is both demeaning and unproductive. When I publicly proposed this '20% programme' many years ago, our then head of research was astonished: “People will do crazy things!” My reply was: “We have a talented staff. You have to trust their judgment on what's worth doing.”

People go into science not just to make a living, but because it is interesting, beneficial and beautiful. They really want to do worthwhile things. The industry will prosper if they are allowed to.