The Civilians Query Problem

Navigation

Hub: 00 - Fact-Check MOC · Previous: 08 - Search Terms and Their Allegations · Next: 10 - Session Log

A design error rather than a typo, and the only finding so far that inflates a headline effect in the direction of the argument.

The most substantive methodological finding in the audit so far. A headline claim is labeled as measuring one thing and its search terms measure something broader, and the broadening is not symmetric between the two conflicts being compared.

The claim in the book

Chapter 7 carries this as a full-caps section heading at p. 125 (file not included in this release):

CNN AND MSNBC COVERED THE IMPACT OF RUSSIA’S INVASION ON CIVILIANS TWICE AS OFTEN AS THEY DID THE IMPACT OF ISRAEL’S BOMBING AND SIEGE OF GAZA ON CIVILIANS, DESPITE THE LATTER HAVING A DEATH TOLL FIVE TIMES THAT OF THE FORMER

The supporting sentence on the same page: “CNN anchors, reporters, and on-air guests mentioned Ukrainian civilians 15,593 times from February 24, 2022, to June 4, 2022. In contrast, they mentioned civilians in Gaza 7,569 times from October 7, 2023, to January 14, 2024. MSNBC anchors, reporters, and on-air guests mentioned Ukrainian civilians 13,941 times versus 7,493 mentions of civilians in Gaza.”

Supporting figures, first 100 days of each conflict:

Ratio as published: approximately 1.96 to 1.

How the underlying search works

The Internet Archive TV News Archive divides every broadcast into 15-second clips of closed-caption text. A query is a list of terms; the system returns every clip containing any term on the list, and the count is the number of clips returned.

So the count is only as good as the term list. Whatever is on the list defines what is actually being measured, regardless of what the finding is called. The label can name one thing while the term list captures another.

Label versus measurement

The heading measures coverage of civilian suffering: it says CNN and MSNBC covered the impact of Russia’s invasion on civilians twice as often as Israel’s bombing of Gaza on civilians.

The term list is recorded in the header of data/nation-tv-dataset/Ukrainian Civilians - 100 days.csv. Eleven of its twelve terms are explicitly about civilians: “ukrainian civilians”, “civilians in ukraine”, “civilians of ukraine”, “ukrainian people”, “people of ukraine”, “people in ukraine”, “ukrainian refugees”, “refugees of ukraine”, “refugees in ukraine”, “refugees from ukraine”.

The twelfth term is the bare demonym: “ukrainians”.

That term is not a civilian reference. It denotes the nationality — civilians, soldiers, and officials alike — and it appears far more often than all eleven civilian terms combined, so it dominates the result set.

The Gaza equivalent has the same structure, with two bare demonyms — “palestinians” and “gazans” — alongside ten civilian-specific terms.

Why the bare demonym breaks it

Sampled clips that the search counted as mentions of Ukrainian civilians:

• discussion of an $800 million security package and heavy weapons deliveries
• whether Ukrainians could win the war without air power
• Ukrainians reporting that Russian troops were careless around a nuclear facility
• supplying the Ukrainians while holding the EU together

None of these is about civilians. They are weapons, military strategy, and diplomacy — counted only because the word “Ukrainians” appears in the clip.

Composition of the counts, computed directly from the published rows:

Roughly three quarters of the Ukraine total consists of clips containing no civilian reference of any kind.

Why the distortion is asymmetric

The natural objection is that the same noise inflates both sides equally, so it cancels. It does not, and that is the crux.

This is the crux

US television refers to Ukraine’s army, government, and leadership collectively as “the Ukrainians.” One word spans civilians, soldiers, and officials. US television refers to Palestinian armed and governing actors as “Hamas” — a term absent from the Gaza query. Military and political coverage therefore flows into the Ukraine count and is filtered out of the Gaza count by construction.

The same term list counts more categories of thing on one side than the other, because the two conflicts are not described in parallel vocabulary. The bare demonym pulls Ukraine’s military and diplomatic coverage into a bucket labeled “civilians”; the Gaza equivalent has no parallel term doing the same work, because that coverage is filed under “Hamas.”

Corrected comparison

Restricting both sides to clips containing explicit civilian or refugee language:

“Almost twice as often” becomes approximately 40 percent more often. Roughly half the reported gap is an artifact of the bare demonym — in plain terms, about half of it was created by counting Ukrainian weapons shipments, strategy, and diplomacy as coverage of Ukrainian civilians.

The denominator has a parallel problem

The death toll comparison — Gaza’s civilian toll 500 percent greater — appears to compare Gaza Health Ministry reported totals against UN OHCHR verified civilian deaths for Ukraine. OHCHR states that its verified figures considerably understate actual totals, since each death requires corroboration. Comparing a reported count against a verified count inflates the ratio.

Two caveats. The exact figures need checking against primary sources; this has not been done. And the passage carries no visible citation for either death toll, though the text extraction may not preserve superscript markers.

Worth noting the tension: Chapter 2 criticizes media for hedging Gaza Health Ministry figures as unreliable. Here the stricter counting standard is applied to the other side of Johnson’s own comparison.

Assessment

Not fabrication. The published figure of 29,534 reproduces exactly from the published data — this dataset, unlike the emotive-words file, is complete and verifiable.

Not a refutation. A 1.40x coverage gap against a substantially larger civilian death toll still supports the directional claim, and the Ukraine comparison remains the best-designed element in the book: matched windows, same networks, victim group varied.

What it is: a mismatch between a stated measurement and an actual one, inflating a headline effect by roughly 40 percent, in the direction of the argument. It differs in kind from the items in 05 - Findings Register, which are typographical and arithmetic. This is a design problem in a claim the book sets in a full-caps heading.

Reproduction

import pandas as pd
U = (r'civilians in ukraine|ukrainian civilians|civilians of ukraine|'
     r'ukrainian people|people of ukraine|people in ukraine|ukrainian refugees|'
     r'refugees of ukraine|refugees in ukraine|refugees from ukraine')
df = pd.read_csv('Ukrainian Civilians - 100 days.csv', skiprows=2,
                 engine='python', on_bad_lines='skip')
s = df['Snippet'].fillna('').str.lower()
print(len(df), s.str.contains(U, regex=True).sum())

Note the header offset: these files need skiprows=2, unlike the emotive-words files which need skiprows=3.

Open

Whether the same bare-demonym inflation affects the other Ukraine-Gaza comparisons in Chapter 7 — children at p. 123 (file not included in this release), journalists at p. 123 (file not included in this release), refugees at p. 125 (file not included in this release). The refugee comparison (1,663 vs 211) is the next to check, since its term list may have the same structure.

The genocide/war-crime comparison at pp. 127–128 (file not included in this release) does not have this problem and was checked clean on 2026-08-04: the stated 17.2x reproduces from its own components (Ukraine 1,515 war crimes + 275 genocide = 1,790; Gaza 92 + 12 = 104; 1,790 / 104 = 17.2), and Fig. 7.3 matches the text at every value.