WikiNPOV Drift Detector

Measuring whether Wikipedia articles can be seen drifting away from a neutral point of view (NPOV) before an editor flags them — and what the data actually says when you build the evaluation honestly.

A research project by Mahima Agarwal (@mahi01agarwal, User:AgarwalMahima) and Dev (@mishradev1, Devdoingdev) Tracking task: phab:T428896.

The task was originally posed as drift detection: catch an article as it slides away from neutrality. We built a dated benchmark first, then tested one signal at a time against it. The result reframes the problem:

Flagged articles don't drift — they start out different. An article that later gets an NPOV tag already reads as more subjective than a matched, never-tagged twin a full year before any human flags it, and that gap stays flat right up to the tag. The text is a static property of dispute-prone articles, not a changing warning sign. What moves around the time of a dispute is editor activity, not the prose.

That makes the text a risk factor you can triage on in advance, not a drift detector.

Each stage tests one question on the same matched dataset. Stages 1–2 are Jupyter notebooks (built for PAWS); 3–6 are scripts.

Subjectivity measures tone ("does it read loaded?"). A separate question is one-sidedness — whether an article represents the relevant viewpoints proportionally, or leans one way by emphasis and omission (due weight). We scored every snapshot with an LLM judge instructed to rate balance only, ignoring tone, and ran the identical lead-time test. The answer is the same: a later-tagged article already reads as more one-sided than its matched twin a full year out, with no rise toward the tag (difference-in-differences +0.02, 95% CI [-0.003, 0.043]). Bias direction, like bias tone, is a static property of dispute-prone articles, not a drift signal — a finding that converges with independent work in the Wikimedia Research team's NPOV workstream (phab:T433445).

The static picture above holds for every tone and direction signal we built. But when we stopped borrowing an off-the-shelf neutrality classifier and fine-tuned a detector on the task itself (RoBERTa, tagged vs. matched-control text), a small pre-onset drift appeared and survived every check we could throw at it: a difference-in-differences of +0.008 [0.002, 0.014] from a year out to the tag, holding under a strict temporal train/test split and under a length control (tagged articles grow ~7× more words, but the drift is not explained by that growth).

The interesting part is what kind of signal it is. A plain TF-IDF bag-of-words model reproduces the same drift (DiD +0.013 [0.006, 0.022]) and matches — even beats — the transformer on same-day detection. And when we grew the training set ~40% (481 → 688 matched pairs), the transformer's drift collapsed to zero (+0.002 [-0.001, 0.005]) while the bag-of-words drift stayed significant (+0.0095 [0.003, 0.016]). More data and more model capacity surfaced no hidden semantic signal — they confirmed there is none beyond the words.

The one thing that drifts is vocabulary. Dispute-era words (but, claims, alleged, controversy, …) accumulate in the prose before a tag, and a 1990s-style lexical model captures that as well as a transformer. So the refined claim is: bias is largely a static risk factor, plus a small, purely lexical pre-onset drift — and we stop short of calling it drift in the meaning of the text.

npov_data/full_npov_events_en.csv + full_npov_controls_en.csv: roughly two thousand English NPOV-dispute events, each dated to the exact revision that introduced the tag, each paired with a matched control. Building it surfaced three measurement pitfalls that any study of maintenance templates inherits:

1. Bot backdating. ~26% of dated maintenance templates are placed by a bot, so dating an event by the template alone measures the bot's run, not the dispute.
2. Edit-summary recall. Regexes over edit summaries recover only ~53% of human template additions — a biased ground truth if used alone.
3. Dormant controls. Matching controls on lifetime totals leaves them inactive in the window a detector actually examines, which inflates results.

Everything derives from public Wikipedia revisions, collected read-only through the MediaWiki Action API. The code never logs in and never edits any wiki.

pip install -r requirements.txt

Scripts read and write under ./npov_data/ and cache every API response under ./npov_cache/ (set NPOV_ROOT or edit ROOT at the top of a script to relocate). Caching means interrupted runs cost nothing on restart. A transient API failure is never written to the cache as a feature value — only real responses are cached, and transient errors are retried.

The neural subjectivity scoring (score_bias_gpu.py, score_sections_gpu.py) uses a Wiki-Neutrality-Corpus-trained classifier and benefits from a GPU; it auto-selects CUDA → Apple MPS → CPU.

A typical order: full_localise.py → build_controls.py → run_nb2.py → build_text_features.py → score_bias_gpu.py → run_nb3.py → build_leadtime_text.py → score_bias_gpu.py → run_nb4.py → build_sections.py → score_sections_gpu.py → run_sections_size.py → run_nb4_sections.py → run_watchlist.py.

notebooks/   NB1 (ground truth) and NB2 (null model) as Jupyter notebooks
*.py         NB1b–NB6 pipeline scripts
npov_data/   dataset, per-stage feature tables, result curves and figures

Large intermediate files (per-sentence dumps, the API cache, logs) are not tracked; they regenerate from the scripts.

Code is released under the MIT License (see LICENSE). The dataset is derived from Wikipedia and is therefore available under CC BY-SA 4.0, consistent with its source.