In 833 AD, the Abbasid Caliph, al-Mamun, was at the height of his power. He ruled a vast empire, and a clear majority of the world’s Muslims considered him the leader of the faithful. However, there was a problem. While he enjoyed considerable influence on questions of religious law and doctrine, the highest human authority in Islam wasn’t the caliph but an amorphous group of respected scholars, the ulema, who were supposed to reach a consensus on contentious issues.

Membership of the ulema wasn’t by official appointment, but by something approaching popular acclamation. And, on some seminal matters, consensus proved impossible. For example, rationalist theologians, the Mutazilites, believed humans have free will. Ranged against them were some of the experts on Sharia law, who insisted that God had determined all things, including who was destined for hell. This meant debates could continue without agreement for centuries.

Al-Mamun wasn’t happy about this. He wanted to enforce his right to adjudicate on religious controversies but instead found himself having to negotiate with the various factions. It was time to press the issue. As it happened, he enjoyed the support of the Mutazilites and so picked one question upon which they agreed against the majority of the ulema: Is the Koran eternal, or was it created by God?

Most Muslims said that the Arabic text of the holy book had always existed in the mind of the deity. For all eternity, he knew exactly what the political situation in 7th century Arabia was going to be and had the Koran ready for dictation to the prophet Muhammad. This strongly implied that history was determined in advance. The Mutazilites disagreed. They said that although God had composed the Koran, it had not always existed. They further argued that determinism implied humans have no choice about whether to sin and no chance to earn redemption, which conflicted with God’s mercy.

Islamic Inquisition

Mercy wasn’t at the forefront of al-Mamun’s mind when he sought to impose his will on the ulema. He issued a proclamation that the Koran was created and insisted everyone consent to it. The most distinguished religious and legal scholars were hauled in for questioning. Those who refused to accept the caliph’s position were tortured and imprisoned until they recanted. The episode is called the mihna, meaning “trial” or “ordeal” in Arabic but often translated as “inquisition.” In this case, the inquisitors were on the side of rationalism and their victims were religious conservatives refusing to disown traditional dogma. Resistance to the caliph was led by Ahmad ibn Hanbal, a venerable expert on Sharia law, whose day job was running a bakery.

Al-Mamun died on campaign within months of instituting the mihna, so it was left to his successors to enforce his ordinance. Many of the ulema backed down, but not ibn Hanbal. Despite being beaten until he passed out and interrogated by the caliph in person, he refused to admit the Koran is created. Too influential to be ignored and too stubborn to recant, ibn Hanbal eventually forced the caliphs to accept defeat. After about 15 years, they wound down the mihna. There would never be a central doctrinal authority in Islam.

No central authority

As for ibn Hanbal, he is recognized as the founder of one of the four main schools of Sunni jurisprudence. Sharia law is based on the Koran (which doesn’t actually contain very much legal material) and the hadiths — that is, sayings of the prophet Muhammad and his companions. Each hadith is a snippet of a conversation, often involving Muhammad responding to a query from one of his followers. If the Prophet said it, it’s authoritative and enjoys the force of law. The trouble was, by ibn Hanbal’s time, fake hadiths had been proliferating as they were invented to support various agendas. He took it upon himself to collate the authentic hadiths, supporting their authority by showing how they had been passed down to his own day.

Ibn Hanbal was not the only collector of hadiths. There are at least six canonical collections, not to mention several thousand sayings dismissed as poorly authenticated. Sharia is far from being a single edifice. Any recognized scholar can issue an opinion, or fatwa, and whether any other scholar concurs is up to him. Today, you can seek a fatwa to address a particular concern over the internet. And if you don’t like the result, you can ask someone else. Consensus remains elusive, although Muslims do now agree the Koran is uncreated.

A wide variety of Islamic thought

It’s the sheer variety of Islamic thought that takes many Westerners by surprise. Without a Muslim version of the pope to delineate heresy from orthodoxy, the ulema have been left to argue it out among themselves. Sometimes rulers would get involved and lend their authority to favored scholars. But even then, dissenters could simply move to a more favorable jurisdiction. Indeed, the biographies of famous thinkers like Avicenna and al-Ghazali are full of peregrinations from one part of the Muslim world to another, as they sought patronage or just to be left alone.

The Shia, who dominate in Iran, originally split from mainline Islam over the question of who was the rightful caliph. But in time, they developed their own theological and legal heritage, which has generally bent more toward rationalism than their Sunni equivalents. The Shia have themselves suffered from schism: most significantly, the Ismaili sect that enjoyed its greatest success under the Fatimid caliphate in Egypt. Meanwhile, the mystical practices of Sufism have permeated all branches of Islam. Sufism can’t be characterized as either theologically conservative or liberal since the Sufi masters wended their own way through the doctrinal morass, with some ending up very far from the mainstream.

Islam has also absorbed outside influences. The translation movement, whereby a huge stock of Persian, Sanskrit, and Greek writing was rendered in Arabic under the Abbasid caliphate, brought governance, science, and medicine to a Muslim audience. All three subjects thrived as they were turned to issues like administration, the calendar, and good health. Today, medieval Muslim mathematicians and natural philosophers are rightfully celebrated, even if some exaggerations about their achievements have crept into the record. More controversially, pseudoscience like astrology was perennially popular. One caliph used a crack squad of astrologers to determine the most auspicious time to lay the foundation stone of his new capital of Baghdad. Alchemy was especially welcomed by Shia savants, who generated a vast corpus of esoteric texts under the name of a quasi-mythical figure called Jabir.

Foreign science and philosophy

Areas of knowledge that had originated outside the Islamic world were called the “foreign sciences.” This was not necessarily detrimental to their position in society even if they had to fight for their corner in the bazaar of ideas. For instance, the Brethren of Purity were a 10th-century sect from Basra, who blended Greek science with Islamic mysticism to create a unique ideology. While they gathered quite a following (despite no one knowing who they actually were), orthodox leaders tried unsuccessfully to suppress their ideas.

Philosophy, called falsafa in Arabic, built on the thought of Greek sages, especially Aristotle, whose work had been translated under the Abbasids. Its greatest exponent was Avicenna, the medieval Persian polymath whose achievements laid the foundation for almost all subsequent Muslim philosophy. Within Islam itself, falsafa had to jostle with theology and law for academic kudos. (Some Arab thinkers celebrated in the West, such as Averroes, were much less influential among Muslims.)

While it is true that falsafa and foreign sciences like astronomy were opposed by traditionalists, the critique misses the point: Pretty much every Muslim thinker aroused opposition from other scholars somewhere. Throwing fatwas at opponents was part and parcel of debate. Nor should we forget that, for centuries after the Islamic conquests, the majority of the people ruled by the caliphs were not Muslims at all. Christians, Jews, Hindus, and Zoroastrians may have been second-class citizens, but that did not put a stop to their intellectual life.

Yet today, despite the range of Islamic thought, few would deny that Muslim countries are generally more socially and religiously conservative than the West — a reversal of the position in the Middle Ages. What happened?

An attack on rationality

A superficial and flippant answer might be “nothing much.” The big changes took place in the West rather than the East, which stayed much the same. But there is more to it than that.

The failure of the mihna meant that caliphs would never enjoy the dogmatic authority of medieval popes. But it was also a humiliation for the Mutazilite rationalists, whose influence deteriorated over the following decades. Meanwhile, the triumphant followers of ibn Hanbal and other scholars pushed their twin agendas of determinism and Koranic literalism. In place of the disgraced Mutazilites, new Ashari and Maturidi schools of theology, each named after their 10th-century founders, came to represent Sunni orthodoxy. While the Maturidis, in particular, were sympathetic toward falsafa, both schools reflected a significant move away from the rationalism of their Mutazilite predecessors. Thus, as consensus among the ulema coalesced slowly over the centuries, it inclined toward conservatism. In any case, traditionalists maintained their hold over Muslim laypeople, who never showed much interest in highfalutin speculative divinity.

By far, the most renowned of the Ashari theologians is al-Ghazali. In 1095, he was a confidant of sultans and viziers but then left his well-paid job as a teacher in Baghdad, swearing that he would never again be beholden to the state. Shortly afterward, he published a trenchant attack on falsafa, The Incoherence of the Philosophers. In this book, he took 20 theses, which followers of the philosopher Avicenna thought were proven, and showed where their reasoning was faulty. It was not that al-Ghazali disagreed with all the theses, although he considered most of them wrong and some to be outright heresy. Rather, he sought to puncture the pretensions of those thinkers who thought that they could use reason to discover absolute truth. Such certainty, he maintained, was only available through divine revelation.

Al-Ghazali has been blamed for diverting Islamic thought toward mysticism and religious obscurantism. In truth, he is hard to pigeonhole. Western scholars in the Middle Ages thought he was a devoted follower of Avicenna because a summary of the latter’s thought circulated under the name of Algazel. In the Muslim world, his most influential work is the enormous Revival of the Religious Sciences, a complete guide to Muslim ethics and ritual intended to integrate Sufi practice with Sunni belief. Not everyone was impressed. In a fine example of the tensions among different parts of the ulema, the authorities in Cordoba, Spain, had the book burnt in the courtyard of the city’s mosque in 1109.

While al-Ghazali’s reputation as the destroyer of philosophy in the Muslim world is unfair, there is little doubt that rationalism remained a minority pursuit. Al-Ghazali himself was hardly the most conservative of the medieval ulema. The 14th-century polemicist ibn Taymiyyah made him look like a dripping wet liberal. Ibn Taymiyyah railed against Sufism, philosophy, and any kind of religious novelty (by which he meant something that was less than 600 years old). From his base in Damascus, he preached that the only source of religious authority was the Koran, together with the genuine hadiths. His cantankerous attacks on his contemporaries got him into endless trouble. He was regularly thrown into prison and accused of violating the authority of the ulema. And yet, today, his trenchant dogma is extremely influential among traditionalists.

Nostalgia meets reality

The term ibn Taymiyyah used for the earliest Muslims who composed the hadiths was “the pious predecessors,” or al-salaf al-salah in Arabic, so his modern admirers are called Salafists. Salafism seeks to return Islam to its roots, inspired by nostalgia for an imaginary past that even Muslim history makes clear never existed.

An early Salafist movement was Wahhabism, founded during the 18th century as a joint venture between the cleric Abd al-Wahhab and the Arabian warlord Muhammad bin Saud. Reacting against the decadence of the Ottoman Empire, they carved an emirate out of the desert that survives to this day as the Kingdom of Saudi Arabia. Elsewhere, Salafi principles have been adopted by a broad range of modern Islamist groups, whether or not they consider themselves formally affiliated with that movement.

Since Islam is such a broad church, why is it that these conservative revivalists have become so successful? To find the answer, we need to remember that until the 18th century, a Muslim would have been justified in thinking that Islam was ordained by God to convert the whole world. Despite local setbacks, such as the reconquest of Spain by Christians, Islam continued to grow strongly in sub-Saharan Africa and India, as well as Central and Southeast Asia.

However, by the 1800s, it was impossible to maintain the dream of superiority. In July 1798, Napoleon Bonaparte invaded Egypt. The climactic Battle of the Pyramids lasted barely an hour and ended with the cream of the Egyptian cavalry decimated to the cost of about 40 French lives. In India, the British encroached upon and finally annexed the once mighty Mughal Empire, while Persia had to acquiesce to subjugation by one European power to protect it from the others. It must have been difficult for the faithful to process the enormity of these sudden reversals.

Despots and stability, not democracy and liberalism

Nineteenth-century Muslim modernists, such as India’s Syed Ahmad Khan and Egypt’s Muhammad Abduh, advocated the imitation of the West, arguing that since Islam is a religion of reason it is fully compatible with the Enlightenment. However, modernization has been hampered by the fact that much of the Muslim world is ruled by despots who are only interested in reform to the extent that it shores up their own power. Science was one thing, liberalism quite another. In the 20th century, as Europe continued to accelerate away, political change was suppressed in places like Egypt and Iran.

While there are few democratic institutions in the Islamic world, we have seen that authority within Islam itself is broadly spread among those ulema who have acquired a certain level of prestige among their peers and the people. The failure of modernism (as evidenced most recently by the collapse of the so-called Arab Spring) has meant religion has been the main outlet for protest. Politicized Islam pursues the twin goals of social reform and religious renewal. The most prominent example is the Muslim Brotherhood, founded in 1928 by the Egyptian Hassan al-Banna. The Brotherhood combines a demand for social justice with a fundamentalist theology. It was clear threat to Egypt’s secular rulers, who tried suppress it, only to make it more radical. Al-Banna was murdered by the secret police at age 42, while its leading intellectual, Sayyid Qutb, was hanged by the regime.

Preoccupied with the struggle against Nazism and then Communism, the West valued stability over democracy in the Muslim world, leading it to support despotic regimes such as the military juntas in Egypt and the Wahhabist House of Saud. That said, where Western power has been brought to bear to impose democracy, such as in Afghanistan and Iraq, the results have been disappointing.

Modernization is possible

So, it is hard to escape the conclusion that the solutions to the Muslim world’s problems will have to come from within. Glib assertions that Islam needs a Reformation are misguided, not least because Salafism itself is analogous to the return to the Bible and early Church Fathers advocated by the first Protestants. And the 17th-century European wars of religion are not an example anyone would like to see followed.

Instead, advocates for reform need to have patience. Religious consensus, developed over the centuries, has never been in the gift of rulers like Caliph al-Mamun. It can only change gradually and has always had to accommodate the views of common folk. This means that until ordinary people living in Muslim countries feel their lives are being improved by modernity, they are likely to remain suspicious of liberalism. More democracy will probably mean more Islamists in power.

Luckily, stupendous intellectual resources already exist in Islam, as does the tradition of debate. These are the ingredients that can help reconcile Islam to the modern world. It’s a process that has been going on quietly for many years. But it will take time to come to fruition.