Query Design Audit

Navigation

Hub: 00 - Fact-Check MOC · Previous: 06 - Open Questions · Next: 08 - Search Terms and Their Allegations

Whether each search pair is commensurable, and where asymmetry is guaranteed by circumstance rather than chosen by editors.

A separate question from whether the counts are accurate: were the right things counted? A term-frequency comparison is only evidence of editorial choice if both terms could plausibly have appeared for either side. Where the asymmetry is guaranteed by the structure of the events or by the meaning of the words, the count measures circumstance rather than bias.

This is the strongest general objection to the book’s method. It applies unevenly, and the pattern of where it applies is itself informative.

The test

For each search pair, ask: is this the same claim, in the same vocabulary, available to be made about either party? If yes, asymmetry is evidence. If no, asymmetry is partly definitional.

Searches where the objection lands

”Right to defend itself” — 755:8 on TV, 459:4 in print

The weakest of the headline findings, and the one the book leans on hardest rhetorically.

The phrase is a term of art drawn from UN Charter Article 51, which frames self-defense as a right of states responding to armed attack. Israel is a state and had just been attacked. Palestinians are not a state in the frame US media operates in, and in the narrative structure of the coverage they occupy the position of the party being acted upon rather than responding. The phrase surfaces when a state answers an attack militarily. Its near-absence for Palestinians is substantially predicted by legal vocabulary and narrative role before any editorial sympathy enters.

Johnson’s actual claim is more interesting than the ratio: that the phrase was “typically followed by the rationalization of mass civilian killing.” That is an assertion about function — what work the phrase does in a sentence. Frequency counts cannot test function. The number is doing rhetorical work the method cannot support.

”Israeli hostages” 835 vs “Palestinian prisoners” 52

The weakest comparison in the book. These are not substitutable terms describing the same phenomenon: people seized during an attack and held by a non-state group, versus people held in a state detention system. Different legal categories, different vocabularies, no reason to expect parity in a corpus about a specific hostage crisis.

The substantive point underneath is real — Johnson notes thousands held without trial and no reporting on their families — but that is an argument about editorial neglect, not something a word ratio demonstrates.

Emotive terms — massacre, slaughter, barbaric, savage

Partly vulnerable. October 7 was a discrete, single-day, close-range attack with identifiable victims; the Israeli campaign was diffuse and ran for months. Massacre-vocabulary attaches to the former structure regardless of victim identity.

Weakened, though not eliminated, by the Bucha test in 04 - Results Stage 1: a close-range attack on civilians five weeks into a different war drew the vocabulary at comparable rates, so violence-form alone does not explain the gap.

Searches where the objection does not land

”Human shields” — 231:5

Johnson’s strongest search, and the objection has almost no purchase on it.

This is the identical factual allegation, in identical vocabulary, available to be made about either party on the same evidentiary basis. Nothing about the structure of the conflict makes the phrase inapplicable to one side. And IDF human-shield use was documented by Human Rights Watch, Amnesty International, and the UN Commission of Inquiry, and reported in Israeli media during the survey window. The claim was available and sourced.

231 to 5 across four outlets in 100 days is not explained by circumstance.

”Hamas-run Ministry of Health” — qualifier applied to one side only

Equally strong. A doubt-sowing modifier attached to one party’s casualty figures while the other party’s are reported unqualified and unsourced. Both sides produce casualty figures through government bodies. The asymmetry is in the hedging, not in the underlying facts, and no feature of the events requires it.

The deeper problem: no account of term selection

Beneath the commensurability question sits a systemic one the book never addresses.

Nothing states how the search terms were chosen. There is no pre-registration, no list of terms tested and discarded, no rule generating the term set. With GDELT and Nexis one can run hundreds of term pairs in an afternoon and publish the ones that came out lopsided. That is a garden-of-forking-paths problem, and it applies to every finding regardless of commensurability.

This is not an accusation that Johnson did it. It is that the design gives a reader no way to rule it out — and a book claiming a “beyond a reasonable doubt” standard has to.

The partial defense available to him: several findings are near-total (231:5, zero uses of “massacre” for any Israeli strike), and extreme results are harder to produce by cherry-picking than moderate ones. But that argument is his to make and he does not make it.

What this changes

The objection damages the book’s presentation more than its case. Johnson leads with the emotive-terms finding and the right-to-defend ratio — the two most vulnerable to a commensurability challenge — and buries “human shields” and “Hamas-run,” which are far more defensible and much harder to explain away.

A hostile reviewer will attack the opening chapters and never reach the strong material. That is a structural weakness in how the argument is built, not in whether it is true.

Test worth running

Whether US media applied “human shields” to Russian forces in Ukraine during the first 100 days of that war. Russian human-shield use was reported. If the phrase was used freely there and appears 5 times for Israel across four outlets in 100 days, circumstance is ruled out as an explanation and Johnson’s strongest finding gets stronger. GDELT can answer this in minutes.