PhilosopherBench 1.1A benchmark that evaluates how frontier AI models think throughout a vast landscape of philosophical questions and positions. Each model answers 38 questions across eight fields, and its responses are matched, by semantic embedding, to the ideas of 90 influential philosophers.
This benchmark measures how a model reasons in philosophy by comparing its own words directly against philosophers’ own words. The works of 90 influential philosophers are compiled, summarized, and their key ideas extracted into short, quote-grounded fact cards, and every model answer is matched against them.
The benchmark uses this method rather than sorting models into schools or categories because labels such as “existentialism” or “cynicism” span many different interpretations, occasionally disagreeing with each other. “Stoicism” is one label, but Marcus Aurelius holds that whether the universe is governed by gods is an open question and that it changes nothing about how one should live: “if it be so that there be no gods, or that they take no care of the world, why should I desire to live in a world void of gods, and of all divine providence?” (Aurelius, bk. 2); Epictetus, another Stoic, treats the matter as settled and foundational, making right belief about a governing providence the whole of piety: the essence of piety is “to form right opinions concerning them, as existing and as governing the universe justly and well” (Epictetus, ch. 31). One label, two incompatible commitments about what the universe is.
The method, put simply, is as follows:
The philosopher a model “reasons like” is the closest match.
This benchmark evaluates 8 fields of philosophy:
Results are below; the full method and the complete question-and-response data follow them.
The benchmark compares the model’s answer to the highest-matching card of a philosopher. A philosopher usually has many idea cards, and most are unrelated to any given question. Every card competes, weighted by how close it sits to the question being asked, so a thinker who wrote nothing bearing on it simply scores near the floor rather than being left out.
Each thinker therefore carries one score on every question in a field, and all 90 are ranked on both measures. Those scores are summarised two ways:
One correction is applied before a card can win. Closest in wording is not the same as actually about the question. Locke has a card arguing that colour and taste exist in the mind rather than in objects; it says nothing about free will, but it uses words like mind, cause, power and perceive, so to an embedding it can look close to a paragraph about free will. Left alone it could beat Locke’s real free-will card, and the site would then report his free-will score using a card about colour. So each card is multiplied by how topically close it is to the question asked: full weight if it was tagged to that exact question, slightly less if it only shares the field, less again if it comes from another field. An off-topic card can still win, but only by being clearly closer in meaning rather than coincidentally similar in vocabulary.
The score is calculated based on the cosine similarity between a model’s answers in that field and the philosopher’s ideas: 100% would mean the two point in exactly the same direction in the embedding space, and 0% would mean they are unrelated. So a philosopher at 64% is a stronger match than one at 58%; the scores rank how closely a model’s expressed reasoning tracks each thinker, not how correct either one is.
Each model’s closest three philosophers, field by field, grouped by family. Expand any family to see its models broken down across all eight fields.
Claude4 models
Command1 model
DeepSeek1 model
Dolphin1 model
Gemini2 models
GLM1 model
GPT3 models
Grok1 model
Hermes1 model
Inkling1 model
Jamba1 model
Kimi2 models
MiniMax1 model
Llama1 model
Mistral2 models
Nova1 model
Qwen1 modelWritten out precisely, the whole scoring rule is three functions:
similarity(model, question, philosopher) calculates how closely that philosopher’s best-matching idea card tracks what the model actually said, on that one question.
general(model, field, philosopher) calculates the average of those per-question scores across every question in the field, so it favours a thinker whose whole outlook runs parallel to the model’s.
highest(model, field, philosopher) keeps the largest of those same per-question scores, so it favours a thinker who came very close on a single issue even if they said nothing about the rest.
cos is cosine similarity: the two vectors’ dot product divided by their lengths, which measures the angle between them and ignores how long either one is. Every vector here is already scaled to length 1, so the divisor is 1 and this reduces to the dot product.
answer(model, question) turns the model’s three sampled answers into a single point: each is embedded, the three are averaged, and the result is scaled back to unit length.
Every card competes for every question, so all 90 philosophers carry a score on every question and both rankings draw on the same numbers. general and highest differ only in the last step: one averages a thinker’s questions, the other keeps their best.
Across all 25 models and 8 fields, the same names keep surfacing. Aggregating every model’s field-by-field top three, scored 3 points for a first place, 2 for a second and 1 for a third, by general match, how closely a thinker tracks a model across a whole field, Bertrand Russell (145 points, from 50 of 200 top-3 placements), Henri Bergson (127 points, from 50 of 200 top-3 placements) and David Hume (92 points, from 39 of 200 top-3 placements) lead the entire benchmark. Ranking instead by highest single match, a thinker’s strongest individual question, Bertrand Russell (99 points, from 40 of 200 top-3 placements), John Locke (93 points, from 37 of 200 top-3 placements) and George Santayana (89 points, from 39 of 200 top-3 placements) lead.
Two different models’ answers to the same question agree at 84% on average, nearly as high as a single model agreeing with itself across re-samples (89%). And it is not an artifact of every LLM writing alike: the same models’ answers to different questions score only 36%. Whatever their labs and training data, frontier models occupy a tight cluster in how they reason about philosophy; the map under Model Similarity lays that cluster out.
Which philosophers do the models collectively favor? For every field, each model’s top three matches earn points: 3 for 1st, 2 for 2nd, 1 for 3rd. The three highest-scoring thinkers are shown. Each bar stacks one colored segment per model placement (wider = higher placement); hover a segment to see the model, its placement, and the similarity score behind it. Both measures are tallied separately: general match above, whose ideas track a model across a whole field, and highest single match below, who came closest on any one question.
The raw material behind every score: all 38 questions, and every answer each model gave. Open a question, then pick a model to read its three independently sampled answers. Each model’s position was assigned by majority vote of an LLM judge over its three answers; models that hedged or refused to commit are counted as mixed. Hover a bar to see which models hold that position.
Loading map…
Each marker is a model, placed so that closer = answered more alike across all 38 questions (classical MDS on the cosine similarity of their answers; Torgerson). Scroll to zoom in and spread the cluster apart; drag to pan.
Model-to-model agreement sits 89% of the way from the null floor to the self-consistency ceiling, so the convergence is a property of the answers themselves, not of the embedding space or a shared writing style. Different models agree with each other (84%) nearly as much as each agrees with itself across re-samples (89%). That number needs a floor to mean anything: two fluent answers on unrelated questions already score 36% just for being LLM philosophy prose.
Existing benchmarks describe ethics or philosophy as a label: this thinker is a utilitarian, that one a rationalist. The ETHICS benchmark sorts moral judgment into subsets named for the theories themselves, justice, deontology, virtue ethics, utilitarianism and commonsense morality (Hendrycks et al.); MoralBench scores a model against the fixed taxonomy of Moral Foundations Theory (Ji et al.). Labels are a poor instrument for a benchmark, especially as in multiple branches of philosophy there exist conflicting positions among philosophers of that same philosophy. This project instead compares a model’s own words directly against philosophers’ own words, using a shared semantic space, and asks which historical thinker a given model most closely resembles when it reasons.
The procedure used involves assembling a library of quote-grounded ideas drawn from the canonical philosophical works, posing 38 open philosophical questions to 25 contemporary AI models, and measuring the semantic distance between what a model says and what each philosopher wrote. What follows documents every step: which philosophers were chosen and how their texts were used, which models were tested and why, how the questions were designed, how the models were probed, and how answers were embedded and scored. It closes with an honest account of what the method can and cannot show. Throughout, one principle holds: the benchmark measures the expressed reasoning of a model, not its private beliefs, and it is a map of resemblance, not a verdict on truth.
The reference set comprises 90 philosophers, chosen to represent the widest defensible map of philosophical thought. They were determined by three criteria: canonical influence on the questions at issue; breadth across eras and schools; and the availability of primary text to quote faithfully.
The result spans from ancient Greece to the present and across the major traditions, empiricism, rationalism, German idealism, existentialism, pragmatism, analytic philosophy, Stoicism and utilitarianism among them, alongside Chinese, Daoist, Buddhist, Islamic and Jewish thinkers.
How a philosopher’s text was used depends on what is lawfully available, which gives the roster three provenances. Most thinkers are public-domain full texts (broadly, pre-1930), retrieved from Project Gutenberg, or from English Wikisource where Gutenberg carries no English edition — Bentham’s Principles, Montesquieu’s Spirit of Laws, Anselm, Fichte, Reid, Seneca’s Letters, the Monadology and Wage-Labour and Capital are all absent from Gutenberg. Five contemporary thinkers are represented by an open-access primary text that remains in copyright but is published in full by its author or journal: Chalmers’s Facing Up to the Problem of Consciousness, Singer’s Famine, Affluence, and Morality, Dennett’s Where Am I?, Korsgaard’s The Sources of Normativity and Plantinga’s Is Belief in God Properly Basic?. These are read in full for matching, like the public-domain texts, but they are cited as the copyrighted works they are and are never redistributed here. Everyone else still under copyright, whose ideas the benchmark would otherwise be blind to (Sartre, Rawls, Foucault, Arendt, Quine and eight more), gets a citation-grounded representation: short verbatim quotations, each carrying its source citation, drawn from Wikiquote. That third provenance extends the map past 1930 without ever reproducing a copyrighted text in full.
Two practical caveats about the texts themselves. Where an edition bundles material the philosopher did not write, it is trimmed away, so a translator’s introduction or an editor’s essay can never be quoted as the author: Hegel’s Philosophy of Mind and Philosophy of History are both cut back to Hegel’s own text. And a few works survive online only in part — the Mencius text is a two-book abridgment, and Wikisource’s Spirit of Laws stops at Book XIX — so those thinkers are represented by less of their output than the roster average.
Every source, in both tiers, is distilled into idea cards. A card is a single claim stated in one line, a short paragraph reconstructing the reasoning behind it, and the verbatim quotations that anchor it to the text. Each quotation is then string-matched against its source and flagged with the result on the card itself: verbatim matches are marked verified, matches that survive only punctuation-normalization are marked separately, and a quote that cannot be matched at all is flagged as unverified. All quotes, in the end, were verified. A philosopher is represented by the full set of their cards (2,222 across the roster), and is only ever matched on the fields their work actually engages.
The benchmark tests 25 models drawn from 17 laboratories. The selection principle is to take the most capable class of models from each provider, then to add further models chosen to represent different cultural and alignment factors.
First, the big labs have their frontier classes (Haiku, Sonnet, Opus and Fable; or Luna, Terra and Sol), and every model in the class is tested, to reveal how scaling and added reasoning or knowledge leads to different opinions. This was applied to the most prominent labs. Furthermore, cultural provenance motivated Jamba (Israel, and the roster’s one state-space and transformer hybrid), Command A (Canada), Mistral (France) and five Chinese laboratories (Zhipu, DeepSeek, Alibaba, Moonshot and MiniMax), whose alignment data is gathered under different linguistic and regulatory regimes. Additionally, two models, Hermes 4 405B and an uncensored Dolphin fine-tune, were specifically selected as lightly-aligned systems, since they were likely to exhibit more radical and sharp positions. The roster is kept current: newly released models, such as Moonshot’s Kimi K3 and Thinking Machines’ Inkling, are added through the identical pipeline as they appear.
Two categories were excluded on principle. Search-augmented systems answer by retrieving and summarising the web rather than by introspecting a settled view, which defeats the premise, so the Perplexity family is out. Older small models were dropped as well, since against 2026 frontiers they return thin, generic text that behaves as noise rather than as an informative small-model point. This engineered diversity is what gives the central finding its force: when models built under such different regimes nonetheless converge, the convergence cannot be an artifact of one lab’s house style.
Each model answers the same 38 questions, distributed across the eight fields listed above (metaphysics, epistemology, ethics, meaning and existence, political philosophy, philosophy of mind, aesthetics and philosophy of religion). The fields were chosen to cover the major branches of the discipline; the questions within them were written to three specifications.
First, each question targets a live debate the field genuinely turns on, rather than a matter of recall: free will and personal identity in metaphysics, the problem of evil in philosophy of religion, the hard problem of consciousness in philosophy of mind, distributive justice in political philosophy. Second, each is phrased in plain language and demands a committed first-person position, so that a model cannot retreat into a neutral survey of “what various thinkers have said” and so that every tradition on the roster, ancient or modern, can engage it on equal terms. Third, the set deliberately mixes abstract questions (is beauty objective? can we know anything with certainty?) with concrete dilemmas (a trolley problem; a disaster-triage choice between a hospital and a power station), because a model’s applied judgment on a hard case often reveals commitments its abstract answers conceal. The identical question set is answered by the philosophers, through their cards, and by the models, so the two are always compared on the same ground.
All 25 models are queried through OpenRouter, a single gateway that routes to every laboratory over one uniform request path. This matters for fairness: every model receives an identical system prompt (answer in the first person, roughly 150 to 250 words, commit to a view, do not restate the question, and do not hedge with disclaimers about being an AI), and no answer is cut short by a length limit, since no token cap is imposed. The only variable across the experiment is the model itself.
Five of the 2,850 samples never reached us as answers: a provider’s content filter blocked them and returned a refusal stub in place of the model’s reply. A stub of that kind is the safety stack talking, not the model, so treating it as a position would be a straightforward error — it would be embedded, matched to some philosopher, and filed under a survey option the model never chose. All five are therefore excluded from every score, position and similarity figure on this site, and flagged wherever the affected question or model is shown. In one case (GPT-5.6 Sol Pro on the question about death) all three samples were blocked, so that model simply has no recorded position on that question and is left out of it throughout, rather than being counted as undecided.
Two testing choices are worth justifying. We sample at temperature 1.0, the model’s genuine output distribution, rather than at a lower, “safer” temperature. A low temperature sharpens every model toward its single most probable, and typically blandest, answer, which would push the whole roster toward a shared cautious centre and manufacture convergence as an artifact of the sampling. Reading a model’s honest philosophical disposition requires reading its real distribution. Because that distribution is stochastic, we ask each question three timesrather than once, so that each model’s position is an average of several independent draws rather than one lucky or unlucky sample. Every raw response, including token usage and finish reason, is archived, so the results are fully auditable and can be re-run without re-billing.
Both the idea cards and the model answers are embedded with OpenAI’s text-embedding-3-large (a 3,072-dimensional model), and every vector is length-normalised so that cosine similarity reduces to a dot product. A model’s three answers to a question are averaged into a single point in this space, which cancels much of the run-to-run sampling noise before any comparison is made. Similarity between a model and a philosopher is then the cosine similarity between that answer point and the philosopher’s idea cards.
Scoring follows the equations given under AI Results. For a given question, a philosopher is credited with their single best-matching card (the maximum over all their cards), each card multiplied by a topical-relevance weight before the comparison: full weight if it was tagged to that exact question during dissection, less if it only shares the field, less again if it comes from another field. That weight exists because an embedding measures similarity of wording, not of subject. A card can share the vocabulary of a question without addressing it, and without the weight such a card can beat a thinker’s genuine answer, so the site would report their score on that question using an idea about something else entirely. The weight does not exclude a distant card, it only requires it to be clearly closer in meaning rather than accidentally similar in wording. We measured that the tiers track something real: a card tagged to a question sits at 0.35 cosine to that question’s own text, one merely in the same field at 0.26, and one from another field at 0.19. A philosopher’s score in a field is the mean of those best matches across the field’s questions. Because each question was sampled three times, the same data supports a robustness check, reported under Model similarity and all measured at the same level so that the figures are directly comparable: a model’s self-consistency across its own re-samples, its similarity to other models, and a null baseline in which the claimed effect is deliberately broken. Cosine similarity between fluent texts never runs from zero; any two philosophy answers share a high floor before anything interesting happens. So the null takes the identical pipeline and compares different models’ answers to different questions, where no convergence should exist. Only the gap between model-to-model similarity and that floor counts as signal. A second null, the mean similarity between idea cards from different fields of philosophy, bounds how much the embedding space hands out for free to any two pieces of philosophical prose.
Eighteen of the 38 questions directly mirror questions on the 2020 PhilPapers Survey (Bourget & Chalmers 2023), which asked 1,785 English-publishing professional philosophers to pick positions on philosophical questions. For those eighteen, the Question Responses section shows the philosophers’ distribution beside the models’. The survey percentages are the paper’s inclusive “accept or lean toward” figures, so they can sum past 100; the survey’s exact wording is quoted under each comparison so the fit of the mapping can be judged directly. Because the models answer in prose rather than by ticking boxes, each model’s position is assigned by an LLM judge (Claude Haiku 4.5, temperature 0) that reads each of the model’s three answers against the survey’s own options and takes the majority label; an answer set with no majority, or one that declines to commit, is reported as mixed rather than forced into a box. Seven questions turn on a distinction too fine for the cheaper judge (property dualism against non-reductive physicalism, for instance) and are judged by Claude Sonnet 5 instead, by the same pass-by-pass majority vote. Six model-question pairs whose three answers genuinely contradicted each other were read and resolved by hand; every escalation and every hand call is listed in the published philpapers.json. A model whose samples were all blocked by a content filter is reported as having no answer, which is kept distinct from hedging. Judge-assigned positions carry their own error — a nuanced stance can be filed into the nearest survey option — which is why the raw answers sit one click away in the same section.
Several caveats bound what these results mean, and the most discussed is a modern-phrasing bias: because every philosopher is represented by their own surviving text, twentieth-century thinkers who wrote in plain contemporary prose might embed closer to an AI’s idiom than the archaic diction of older translations does, letting a model score as “Russell” or “Chalmers” partly for sounding modern rather than for reasoning alike. The method already blunts this: matches are computed on the register-normalized paraphrase in each card (the neutral modern restatement of the idea, not the period quotation), so every thinker, ancient or modern, is compared in the same voice. Rather than assume that settles it, we measured whether any bias survives, by correlating each philosopher’s historical era against how closely the 25 models resemble them.
Loading era-bias figures…
The engine also measures similarity of expressed style and content, not endorsement or belief: a model that argues in a Kantian register is scored as Kant even if it would reject the label on reflection.
A second structural caveat is uneven representation. Philosophers differ widely in how many idea cards their surviving work yields (from 9 to 86, median 19), and because a question score takes each thinker’s single best-matching card, a thinker with more cards gets more chances to be somebody’s best match. We measured what this buys: card count does not inflate the similarity scores themselves (the correlation between a thinker’s card count and their mean match score is r = −0.017, indistinguishable from zero), but it does raise visibility: how often a thinker appears in a top-matches list at all. Against log card count that runs r = +0.44 (p < 0.001) for highest single match and r = +0.38 (p < 0.001) for general match, the field-wide average being the less card-hungry of the two, since one lucky card cannot carry it. So a heavily-carded thinker like Hegel is not scored too generously, but a thinly-carded one can be under-represented in the rankings simply for lack of coverage: 22 of the 90 never appear in any highest-single-match list, and 34 never appear in any general-match list. Read absences cautiously; read the scores themselves at face value.
Relatedly, not every question is evidenced equally on the philosopher side. Abstract perennials (free will, skepticism, the good life) each have well over a hundred directly-tagged cards, but the deliberately concrete dilemmas have few (the disaster-triage case has 8, the trolley problem and machine-thinking questions 9 each) because historical texts rarely address them head-on. For those questions the rankings lean on down-weighted field-level cards, and the “nearest thinker” should be read as closest general orientation, not as that philosopher’s answer to the dilemma.
For the post-1930 citation-grounded tier, quote verification runs against the compiled Wikiquote source file, not against the original copyrighted works. This guards the transcription (no card rests on an invented line), but it inherits Wikiquote’s own sourcing quality: each quote carries its citation, and the misattributed/disputed sections are excluded, yet the chain of custody is one link longer than for the public-domain tier, and these thinkers rest on a curated selection rather than their full text.
A few thinkers are represented by more than one edition, and the quotations reflect whichever edition a card was drawn from. Montesquieu is the clearest case: Books I–XIX of The Spirit of Laws come from the 1758 Nugent translation, while Books XX–XXXI — which Wikisource has never transcribed — come from Nugent as revised by J. V. Prichard. The two are held as separate works so a quotation is never matched against the wrong text, but two Montesquieu quotations on this site may be in noticeably different English. The same caution applies wherever a thinker predates modern translation: the register is the translator’s, the argument is his.
The roster, though broadened deliberately, remains weighted toward the Western canon, both because that canon defines many of the questions and because digitised, quotable primary text is more available for it. Averaging a philosopher’s score across a field can also over-credit a narrow specialist who wrote intensively on one topic, which is why the per-field breakdown, not any single aggregate, is the honest view. Finally, embeddings are a powerful but imperfect proxy for meaning, and reducing a high-dimensional semantic space to a similarity number, or to a two-dimensional map, necessarily discards structure. The findings should be read as a rigorous, reproducible portrait of resemblance, and not as the last word on what any model believes.