One night in 1851, Henry David Thoreau woke from a dream. In it, astride two ungovernable horses — literal nightmares — he galloped through the woods, “but the horses bit each other and occasioned endless trouble and anxiety, and it was my employment to hold their heads apart”. At that time, the 34-year-old naturalist–writer was trying to reconcile two contending forces in his life: the transcendentalism he had long espoused and the rigours of science he had recently discovered.

Walden Pond in Concord, Massachusetts, where the naturalist Henry David Thoreau lived for two years in the 1840s as an experiment. Credit: Tetra Images/Getty

Transcendentalism emerged in the mid-1830s as an intoxicating set of philosophical, literary and spiritual tendencies unified by discontent with American life. Its core tenet, derived from Romantic philosophy, was that God permeated everything. As the movement's best-known spokesperson, philosopher Ralph Waldo Emerson, put it, “behind nature, throughout nature, spirit is present”. Transcendent divinity could be perceived only through intuition and inspiration — not through reason — and was best found in nature. This set of beliefs fuelled the experiment in self-reliance and simple living on Walden Pond in Concord, Massachusetts, in the mid-1840s that led to Thoreau's masterpiece, Walden, published in 1854.

But something happened to Thoreau before his best-known book was finished, subtly changing its final form, making it more empirical, and thus scientific. Always fascinated by the natural world, he began to ponder more on its physical processes. Every day, rain or shine and usually for at least four hours, he tramped through the woods and fields surrounding his native Concord, collecting specimens and writing down what he observed in a series of small notebooks. He measured and weighed, pressing the leaves of red currant, poison sumac and many other species for his herbarium. In the evening, he transferred his observations into larger journals.

Occasionally he worried about this new tendency. “I fear,” he wrote on 19 August 1851, “that the character of my knowledge is from year to year becoming more distinct and scientific — that, in exchange for views as wide as heaven's cope, I am being narrowed down to the field of the microscope.” He tried to fit the data he gathered into a larger, transcendentalist vision of the cosmos, but his focus was inexorably drawn to the material world.

In 1860, he encountered Charles Darwin's On the Origin of Species, published the previous year. He was already familiar with Darwin, having devoured The Voyage of the Beagle (1839) almost a decade earlier. Thoreau first learnt of Origin at a dinner party on New Year's Day 1860, a gathering that included radical abolitionist Franklin Sanborn, child-welfare reformer Charles Loring Brace and transcendental philosopher Bronson Alcott. This quartet of progressive intellectuals discussed the book at length, and Thoreau was immediately captivated. He pestered the local library to acquire a copy, and was the first person to check it out when it arrived later that month.

Engine of creation

By February, Thoreau was copying extracts from Origin into his natural-history notebook. He seems to have been drawn to Darwin's examples of the delicate interplay of causes that lead to the survival or extinction of species. For instance, Darwin wrote: “The number of humble-bees in any district depends in a great degree on the number of field-mice, which destroy their combs and nests.”

Thoreau was most taken, however, with Darwin's idea that the struggle between species was an engine of creation. It undermined transcendentalist assumptions that nature was inherently good, that the Universe bent toward moral goodness. It also corroborated Thoreau's own observations. While living on Walden Pond, he had tried to discover the “unbroken harmony” of the environment, the “celestial dews” of the pond, but sometimes a darker reality intruded.

“Not a pickerel or shiner picks an insect from this smooth surface but it manifestly disturbs the equilibrium of the whole lake,” he wrote in Walden. Something ominous lurks around this sentence. That animals must consume other animals to survive disturbed the transcendentalist urge for harmony. But Darwin provided an explanation for nature's murderous subtext: competition and struggle, he wrote, influenced “the whole economy of nature”. It drove species to change and adapt. It was the cost of doing nature's business.

Thoreau was also struck by Darwin's discussion of geographical distribution — the way in which plants and animals are dispersed across the globe. In 1850, Thoreau had noticed a pine seedling in his yard, far from any other pine, prompting him to wonder how it had arrived there. He began to study how squirrels transported nuts and seeds, then followed the aerial voyages of milkweed and dandelion seeds. Soon he was observing cockleburs and other barbed seeds that attach themselves to animals and clothing. The question he was trying to answer was one he had asked in Walden: “Why do precisely these objects which we behold make a world?”

Darwin reinvigorated the issue for him in 1860. “When an American forest is cut down,” the British biologist wrote, “a very different vegetation springs up ... What a struggle between the several kinds of trees must here have gone on during long centuries, each annually scattering its seeds by the thousand.” Thoreau took special note.

As soon as he finished Darwin's book, he began consolidating the data he had gathered over the years. He re-read his journals — thousands of pages — and copied every observation about plants and animals on to slips of paper. He transcribed this material again in a series of prototype 'spreadsheets' (it is possible that he invented the concept). On each page, Thoreau listed a month, with a column for every year from 1852 to 1860. Here he recorded all the information he had gathered, in tiny, nearly illegible handwriting.

Henry David Thoreau, pictured in 1856. Credit: Granger/Rex/Shutterstock

What was he up to? We don't entirely know. But the painstaking work enabled Thoreau to capture and quantify the process of growth and death in nature — to discern patterns in nature's chaotic creativity. It also allowed him to verify Darwin's theories as they applied to the natural environs he knew so well. Six months after reading Origin, he incorporated the book's ideas into a new essay, 'The succession of forest trees'. Delivered as a lecture at Concord's annual cattle show and printed throughout the nation's newspapers, this was one of Thoreau's most popular essays in his lifetime. It refuted a common theory that the appearance of new forests was the product of spontaneous generation. Nonsense, Thoreau declared: forests are “regularly planted each year by various quadrupeds and birds”. In a series of detailed descriptions, he proceeded to corroborate and explain Darwin's statement about American forests.

More importantly, Thoreau suggested how a transcendentalist might begin to appreciate a natural world that did not rely on God to function. He did this by locating mystery and wonder within materialism. His touchstone was the seed: “Though I do not believe that a plant will spring up where no seed has been,” he wrote, “I have great faith in a seed.”

Self-directing wonder

There is a wondrousness in a natural world that is blindly self-directing, driven by struggle and contingency, authored by itself.

That passage has much in common with the final paragraph of Origin, describing an “entangled bank, clothed with plants of many kinds, with birds singing on the bushes”. Thoreau's point — like Darwin's — was that nature's fecundity had produced a world of wondrous scope and intricacy. Millions of seeds and spores are produced and broadcast by wind and animals; only a few plants find their niche. The countless complex interactions necessary to produce a single maple, he discovered, need involve nothing intelligent, omniscient or all-seeing. And there is a wondrousness in a natural world that is blindly self-directing, driven by struggle and contingency, authored by itself. Thoreau's essay on forest succession was an early response to a world Darwin had introduced, made wonderful by science. We will never know how far Thoreau might have extended these ideas. Nor can we know what insights he might have extracted by applying the principles of variation and natural selection to his beloved woods and fields. On 3 December 1860, he caught a cold, which exacerbated a dormant case of tuberculosis. He died in 1862.

It is clear, however, that Thoreau had begun to control the wild horses of his earlier dream. Throughout 1860, he carefully recorded the ways in which oaks and pines advanced, retreated and produced hundreds of thousands of acorns and pinecones to gain a foothold. He marvelled that Darwin's theory “implies a greater vital force in nature, because it is more flexible and accommodating, and equivalent to a sort of constant new creation”. This declaration marks an epoch in American intellectual life. In one short year, Darwin had propelled Thoreau to a new understanding: that nature just was — and that was enough.