Innovation is being talked about everywhere. The US Senate is working on a biomedical innovation bill. Australia’s main funding agency has just announced that it will cut hundreds of climate scientists as part of its National Science and Innovation Agenda. The National Council of Science Museums in India will add Innovation Hubs at its centres. More and more organizations are using innovation in their names and brands. Innovation is a central plank of national and local policies and it consumes billions of dollars of investment worldwide. Yet the evidence base for these innovation efforts is close to nothing. We simply do not know how the innovation happens. We should do more to find out.

Innovation is commonly confused with invention and creativity. Creativity is the ability to generate original ideas, concepts and objects. It spurs invention, which is most evident in the areas of technology and business. Artists enjoy creativity, whereas engineers and scientists focus on inventions. But innovation demands a third ingredient: market success.

History contains examples of the rapid transformation of creativity into market success: Picasso managed to earn an income from his creations, and Disney’s theme parks are lucrative tourist attractions. But disruptive innovations — those that have transformative impacts, such as the steam engine or the Apple iPhone — are rare.

The path to innovation is currently more art than science. That might explain why it is shockingly inefficient: the chance of an invention attaining enough commercial or social success to be recognized as an innovation reaches no more than low single percentages. In the United States’ Small Business Innovation Research programme, a very low proportion of grants results in a viable economic activity, product or service. In markets that are saturated, such as those of mobile phones or medical discoveries, the success rate is even lower.

Governments want innovation that not only transforms the industry, but also offers solutions to the ‘grand challenge’ problems that the world faces: alternative energy sources, mitigating climate change, eliminating poverty and improving health care and security. Many research programmes in these and other areas claim to be innovative, because they seek and apply new approaches to a specific problem. Others seem to believe that the research itself is innovative because it produces new findings, or that the results will inevitably lead to innovative outcomes.

But it is not that simple. There is no deep understanding of the innovation process, which is complex and has not been well captured or formalized. There is no unified theory or reliable model for innovation. There is no innovation science.

How could the science of innovation happen? There are several areas in which research could initiate and potentially formalize it: for example, the study of patents or creative individuals such as musicians and painters. We could use results from these studies to conceptualize and model the innovation process from generalizations identified across the engineering, arts, science and social domains.

There is no deep understanding of the innovation process.

The task is complex and enormous. The first step could be to look for shared factors and to devise rules and hints that support innovation. It might help to look backwards. Firms and individuals often claim that they have learned from mistakes, but how many analyse failure systematically? Patent libraries are packed with submissions that never get used, and many research programmes and clinical trials do not lead to success. Analysis of these failures could help others to succeed, and could contribute to an understanding of what drives innovation.

Another backward-looking approach is imagining the best, then scaling it back to reality. Imagine an item of office furniture with all-encompassing functionality: it fulfils all the needs of the worker, but also changes colour according to the weather and adapts to the height and weight of the person. Limitations of technology, prohibitive cost and anticipated market response are then used to scale it back into marketable items: a chair whose height is adjustable and a desk that can be set at two height levels, and both would be available in different colours. Computer printers were innovated in a similar way by incorporating functions beyond printing.

A building block of innovation science is connecting seemingly unrelated ideas. We are flooded with discoveries in isolated domains. Making quick connections between, for instance, biology and technology, could lead to bigger ideas and redirect research and development.

Innovation-science researchers must develop models of the market and products to predict successful outcomes. These models could be based on emerging evolutionary computation, and would be developed, validated and tested using streams of data, such as consumer interests and preferences.

Over the long term, private foundations should establish a global initiative at a scale similar to that of the Breakthrough Energy Coalition spearheaded by Microsoft co-founder Bill Gates. (In fact, that coalition could itself greatly benefit from innovation science.)

In commercial terms — comparing investment to output — the innovation process might have a failure rate of 99%. That would simply not be tolerated by any other commercial enterprise. Mainstream industry has moved to six-sigma programmes and beyond, barely tolerating one or two errors in a million. Governments and scientists should focus less on discussing various forms of innovation and more on how to innovate.