# Data Quality Gate — Full Reference

> Deterministic post-scrape data cleaner and quality gate for AI agents. Repairs how data was **encoded** — residual HTML, mojibake, invisible characters, non-breaking spaces — and never what it **says**. Also returns a reliability verdict. 100% deterministic, no LLM: identical input always produces byte-identical output. One HTTP call, no setup, any tabular JSON, any domain.

This document is the complete reference. For a short summary, see [/llms.txt](/llms.txt).

## Three tiers

| Tier | Endpoint | Returns | Price |
|---|---|---|---|
| CLEAN | `POST /api/clean` | the repaired data, as the response body | $0.04 |
| CLEAN + AUDIT | `POST /api/clean/audit` | the same data + a replayable, reversible ledger | $0.12 |
| VERDICT | `POST /api` | score + facts + RELIABLE / USABLE_WITH_CLEANING / UNRELIABLE | $0.01 |

All three run the same deterministic engine, and are paid via x402 — no account, no API key, no signup. Cleaning is documented in its own section near the end of this document; everything between here and there describes the verdict.

Which to call: the **verdict** answers "can I trust this source?" — bought once per source, then cached. The **cleaner** answers "make this batch usable" — every extraction run produces new dirt, so it is called once per run.

## Endpoint

```
POST https://www.aidatatools.dev/api
Content-Type: application/json
```

Body is either:
- a bare JSON array of row objects — `[{...}, {...}]`, or
- an envelope object — `{"rawJson": [...]}` or `{"rawJson": [...], "datasetId": "..."}`

`GET` on the same URL returns a short usage summary instead of running a check.

Never returns a 5xx or a stack trace for bad *data* — a malformed, empty, or huge dataset degrades to a valid, low `UNRELIABLE` verdict with an explanation in `meta.errors`, not an error response. Only a malformed *HTTP request* (unparseable JSON body, wrong method) returns 4xx.

## The 8 checks

Each check produces a `facts` block (always present) and, for 6 of them, a 0–100 `score` component (weighted into the overall score — see below). `structure` and `cardinality` are facts-only; they inform reasons/recommendations but do not carry a score weight.

1. **structure** — homogeneity. `homogeneous` is true iff every field name (union of keys across all rows) is present as a key in every row.
2. **completeness** (weight 0.25) — a value is "usable" iff the key is present AND the value is not null AND not an empty/whitespace-only string. Reports `overall_presence_pct`, `complete_rows_pct`, `fully_empty_fields`, `fields_with_missing_values`.
3. **nulls** (weight 0.20) — counts values that are literally JSON `null` (distinct from completeness: does not include empty strings or absent keys). Reports `global_null_pct`, `fields_with_nulls`.
4. **types** (weight 0.20) — per field, the dominant type is the most common type among usable values (int and float merge into `numeric`). Reports `type_coherence_pct`, `mixed_type_fields`, `numeric_stored_as_text` (numeric-looking values stored as strings, e.g. `"$180,000"`).
5. **impossible_values** (weight 0.15) — field-name heuristics (e.g. a field that looks like a rating gets a plausible-range rule, a field that looks like a price gets a non-negative rule) select which rule(s) apply per field; a value failing multiple applicable rules is counted once. This is the only check with a hard gate on the final verdict (see below).
6. **duplicates** (weight 0.10) — two sub-checks:
   - `exact`: full-row hash (`json.dumps(row, sort_keys=True)`); any row hash repeated more than once is an exact duplicate.
   - `near`: fuzzy text match via `rapidfuzz.fuzz.token_set_ratio` (threshold ≥ 85) on concatenated string fields, but only fields that plausibly carry distinguishing content (string type, not url/id/image/date-like by name, average length ≥ 12 chars, cardinality ratio ≥ 0.5 — this excludes e.g. a shared `searchQuery` field from dominating the comparison). Skipped (with a stated `reason`) when no field qualifies, when there are under 2 rows, or when the dataset has more than 5,000 rows (pairwise comparison is O(n²); skipped rather than run slowly or time out).
7. **outliers** (weight 0.10) — Tukey fence per numeric field: bounds = `[Q1 - 1.5*IQR, Q3 + 1.5*IQR]` using inclusive quartiles. Fields with fewer than 5 numeric values are skipped (listed in `skipped_fields`) rather than guessed at. The score comes from this fence alone; a facts-only `modified_z_mad` sub-block reports a second, more robust test alongside it (see "Facts-only signals" below).
8. **cardinality** (facts only, no score weight) — `constant_fields`: exactly 1 distinct usable value across ≥ 2 rows (no information content). `quasi_unique_fields`: distinct/usable ratio ≥ 0.95 across ≥ 5 rows (identifier candidates; each is flagged `likely_identifier` by field-name heuristic).

## Facts-only signals (reported, never scored)

Three signals report without judging. None of them is part of the weighted score, none is wired into `verdict.reasons`/`recommendation`, and none can change `score.overall` or `verdict.level`. Each simply adds a block to `facts` when it has something to say. That separation is deliberate in every case: a signal that is genuinely useful to a caller is not automatically safe to fold into a number, and the cost of getting that wrong is a silently mis-scored dataset.

| Signal | Where it appears | What it reports |
|---|---|---|
| `price_divergence` | `facts.price_divergence` | same asset priced inconsistently across sources |
| `text_cleanliness` | `facts.text_cleanliness` | extraction artifacts inside string fields |
| modified z-score (MAD) | `facts.outliers.modified_z_mad` | outliers the Tukey fence structurally cannot see |

### price divergence (financial/crypto data)

Not part of the weighted score, not one of the 8 checks above, and not wired into `verdict.reasons`/`recommendation` — it cannot change `score.overall` or `verdict.level`. It only ever adds an additional `facts.price_divergence` block when applicable. This is deliberate: it is new, and folding it into scoring/verdict is a separate decision not yet made.

**Why this exists alongside `outliers`:** the generic `outliers` check runs one Tukey fence over an entire numeric column, regardless of what each value represents. A dataset with rows for BTC (~$67k), ETH (~$3.5k), and SOL (~$180) mixed into one `price` column hides a SOL-only anomaly inside the BTC/ETH range — nothing in that column looks statistically unusual once BTC-scale values are in the mix. `price_divergence` groups by entity first (e.g. by `symbol`), then looks for divergence *within* each group only.

**Auto-activation:** `facts.price_divergence` appears only when the dataset has both:
- a `group_field` — a string field whose name looks like an entity key (`symbol`, `ticker`, `asset`, `instrument`, `pair`) with at least 2 distinct values, and
- a `value_field` — a numeric field whose name looks like a price (`price`, `cost`, `rate`).

On a dataset without such fields (e.g. e-commerce, real estate), `facts.price_divergence` is simply absent — output is byte-identical to before this check existed. All plausible `(group_field, value_field)` pairs are checked, not just the first match.

**Method, stated honestly (two branches depending on sample size per group):**
- group has **≥ 5** values: Tukey fence (`IQR*1.5`), the same statistical method/multiplier as `outliers`, computed within the group only. Can identify *which* value is the outlier.
- group has **2–4** values: too few points for quartiles to mean anything. Falls back to a ratio heuristic: flagged if `max/min >= 3.0`. Can only say the group disagrees with itself — with exactly 2 points it cannot identify *which* value is wrong (both are equidistant from the group median).
- group has **< 2** values: skipped, nothing to compare against.

This split is the honest limit of the method: cross-exchange price snapshots often have only 2–5 sources per asset per batch, too few for a rigorous quartile-based test. The 3x ratio fallback trades statistical rigor for catching gross divergences (a wrong decimal, a stale quote, a bad symbol mapping) — not routine bid/ask spread, which stays well under 3x.

**Positioning:** built to be called per-batch, before a decision (e.g. before a trading agent acts on an aggregated price snapshot) — like the rest of this tool, this is not a real-time/streaming/HFT feed.

Example `facts.price_divergence`:

```json
{
  "applicable": true,
  "method": "For each (group_field, value_field) pair: group rows by group_field, then per group with >= 5 values, Tukey fence (IQR*1.5, same method as the generic outliers check but computed within the group only); per group with 2..4 values, a ratio heuristic (max/min >= 3.0x) since quartiles aren't reliable below 5 points -- flags group-level disagreement, not a specific bad value. Groups with < 2 values are skipped (nothing to compare).",
  "pairs_checked": [
    {
      "group_field": "symbol",
      "value_field": "price",
      "groups_checked": 3,
      "groups_flagged": 1,
      "flagged_groups": [
        {
          "group": "SOL/USDT",
          "n": 2,
          "method": "ratio_heuristic",
          "min": 178.45,
          "max": 1784.5,
          "ratio": 10.0,
          "threshold": 3.0,
          "note": "fewer than 5 points: cannot compute reliable quartiles here, so this flags the group as internally inconsistent but cannot identify which individual value is wrong"
        }
      ]
    }
  ],
  "total_groups_flagged": 1
}
```

### text cleanliness (scraped / aggregated text)

`facts.text_cleanliness` appears when a string field carries extraction artifacts. It exists because `completeness` answers "is a value there?" and `types` answers "is it the right Python type?" — neither can see that a value which IS there, and IS a string, is an artifact rather than data. Six categories, reported separately because they have different remedies:

- **`html_markup`** — leftover tags or entities (`<div>…</div>`, `&amp;`). A real-tag regex, so `price < 100` and `Tom & Jerry` are not matches.
- **`executable_markup`** — `<script>`/`<style>`/`<iframe>`, kept separate since it also matters downstream.
- **`boilerplate`** — site furniture that landed in a content field ("click here", "accept cookies"). The matched phrase is reported back, so you can judge it yourself.
- **`mojibake`** — text decoded with the wrong codec (`"Café"` for `"Café"`). Detected by a cp1252→utf-8 round-trip that must both succeed and change the string; correctly-encoded `"Café"` fails that test by design and is not flagged.
- **`placeholder`** — a string that means missing (`"N/A"`, `"null"`, `"-"`, `"undefined"`) but that completeness counts as present and types counts as a valid string.
- **`non_normalized`** — NBSP, fullwidth or ligature characters (NFKC-unstable), plus zero-width/invisible characters reported as `invisible_chars`.

A value failing several categories is counted once in that field's `flagged_values`, never double-penalized.

Two things worth knowing. First, this is the check that found a zero-width non-joiner (U+200C) in 100% of the rows of a real Amazon scrape: the value compares unequal to its visually identical twin, which silently breaks any join or groupby on that field, and no other check could see it. Second, the placeholder list is a judgment call — a `middle_name` of "None", or a category legitimately named "Unknown", will be flagged. That ambiguity is exactly why this signal reports instead of scoring.

### robust MAD cross-check (inside `outliers`)

`facts.outliers.modified_z_mad` reports the Iglewicz-Hoaglin modified z-score — `M = 0.6745*(x - median)/MAD`, flagged when `|M| > 3.5` — alongside the Tukey fence that actually drives the score.

It is there because the fence has a structural blind spot. Its bounds come from Q1/Q3, which are order statistics of the data itself: when a cluster of corrupted values fills roughly a quarter of one tail, it drags the quartile outward, the fence widens with it, and stops flagging anything at all. On a 21-value column holding 6 unit-conversion errors, the fence flags **zero** values (bounds blow out to `[-1462, 2477]`) while the modified z-score flags all 6 plus a genuine anomaly they were hiding. `caught_only_by_mad` is exactly that difference, surfaced. The median and MAD keep a 50% breakdown point; the quartiles do not. When MAD is 0 (over half the values identical) the block is skipped, since the fence already covers that case on its own.

It reports rather than scores because that was measured, not assumed. Folding both tests into the score moved 4 of 5 reference datasets **down**, and every move was a false positive in one of two structural modes: multi-population columns (a `price` column mixing BTC ~67k, ETH ~3.5k and SOL ~178 gives legitimate BTC prices a modified z of 6.12) and naturally right-skewed columns (review and sales counts are heavy-tailed, and the modified z-score assumes rough symmetry, so it flags the top of an entirely normal tail). Both are properties of the statistic rather than bugs. Note also that no robust statistic rescues a *semantic* impossibility: a price of `0` sits ~1.6 MADs from the median in a 0..5999 column and is statistically unremarkable — that belongs to `impossible_values`, not to any outlier test.

## Scoring

```
overall = sum(weight[k] * component_score[k] for k in weights)

weights = {
  "completeness": 0.25,
  "nulls": 0.20,
  "types": 0.20,
  "impossible_values": 0.15,
  "duplicates": 0.10,
  "outliers": 0.10
}
```

Weights and per-component scores are returned in `score.components` and `score.weights`, so a caller can recompute `overall` itself instead of trusting it blindly.

## Verdict levels

```
RELIABLE            : overall >= 85
USABLE_WITH_CLEANING : 50 <= overall < 85
UNRELIABLE           : overall < 50
```

### Hard gate (important)

A dataset containing at least one impossible value (e.g. a negative price, an out-of-scale rating, an absurd date) can **never** reach `RELIABLE`, regardless of how high the weighted score is — these are certain factual errors, not a probabilistic judgment call. When this fires, the verdict is capped to `USABLE_WITH_CLEANING` and the response is fully transparent about it:

```json
"verdict": {
  "level": "USABLE_WITH_CLEANING",
  "level_capped": true,
  "level_before_cap": "RELIABLE",
  "cap_reason": "verdict capped to USABLE_WITH_CLEANING: 1 impossible value(s) found (raw weighted score was 92.54, which alone would have been RELIABLE)"
}
```

`level_capped` / `level_before_cap` / `cap_reason` are optional keys that only appear when the gate fires — do not assume their presence. Duplicates are deliberately NOT hard-gated (they can be legitimate depending on use case), so they stay purely score-driven.

## Response shape

Always exactly this top-level shape, HTTP 200 on any successful call to the engine (an `UNRELIABLE` verdict is a successful analysis, not a transport error):

```json
{
  "meta": {
    "tool": "data-quality-gate",
    "version": "0.1.0",
    "generated_at": "2026-08-05T20:30:46.268749+00:00",
    "input_source": "rawJson",
    "warnings": [],
    "errors": []
  },
  "facts": { "structure": {...}, "completeness": {...}, "nulls": {...}, "types": {...}, "impossible_values": {...}, "duplicates": {...}, "outliers": {...}, "cardinality": {...} },
  "score": {
    "overall": 92.54,
    "weights": {"completeness": 0.25, "nulls": 0.2, "types": 0.2, "impossible_values": 0.15, "duplicates": 0.1, "outliers": 0.1},
    "components": {"completeness": 93.75, "nulls": 93.75, "types": 100.0, "impossible_values": 85.71, "duplicates": 75.0, "outliers": 100.0},
    "formula": "overall = sum(weights[k] * components[k] for k in weights)"
  },
  "verdict": {
    "level": "USABLE_WITH_CLEANING",
    "thresholds": {"RELIABLE": ">= 85", "USABLE_WITH_CLEANING": "[50, 85)", "UNRELIABLE": "< 50"},
    "reasons": [{"check": "impossible_values", "severity": "high", "message": "..."}],
    "recommendation": "Impute or drop field(s) with significant missing values: price. ...",
    "level_capped": true
  },
  "benchmark": {
    "method": "tiktoken 'cl100k_base' encoding, token count of json.dumps() of each payload",
    "input_tokens": 106,
    "output_tokens": 1224,
    "reduction_pct": -1054.72
  }
}
```

`generated_at` is a wall-clock timestamp and will differ between runs — it is metadata, not part of the judgment. `facts`, `score`, and `verdict` are the deterministic contract: identical input always produces identical `facts`/`score`/`verdict`.

## Token benchmark

`benchmark` compares the token count (tiktoken `cl100k_base`) of the input payload against the output payload. On small worked examples the verdict object can be larger than a tiny input (`reduction_pct` goes negative, as in the example above) — the real saving shows up on realistic scraper-sized datasets, where a large raw dataset collapses into a compact, exploitable verdict instead of the agent having to read every row itself.

## Cleaning: tiers 2 and 3

### The one-liner

```
curl -X POST https://www.aidatatools.dev/api/clean \
  -H "Content-Type: application/json" \
  --data-binary @scrape.json > clean.json
```

The response body **is** the repaired data, in the shape you sent: an array in, an array out; one object in, one object out; a CSV or plain-text body works too, and the format is detected. Repair descends into nested objects and arrays — scraped `specs` lists are cleaned like top-level fields. Counts come back in `X-DQG-*` response headers (`X-DQG-Repairs`, `X-DQG-Values-Changed`, `X-DQG-Flags`, `X-DQG-Rules-Applied`). Add `?envelope=1` to get `{"data": ..., "summary": ...}` in the body instead.

Options go alongside the payload: `{"rawJson": [...], "options": {"drop_exact_duplicates": true}}`.

### The boundary: repaired vs reported

`GET /api/clean` returns all 20 rules and 7 invariants, machine-readable, with the argument for each — auditable **before** you pay. Three actions, and only three.

**AUTO — applied automatically (7 rules).** Information-preserving: the removed bytes carry no meaning the surviving bytes do not already carry.

- `mojibake.roundtrip` — cp1252→utf-8 round-trip, iterated to a fixpoint so double- and triple-encoded text lands correctly. Correctly-encoded text fails the round-trip by design and is never touched. Aborts rather than introduce U+FFFD or control characters.
- `invisible.strip_formatting` — BOM, zero-width space, soft hyphen, word joiner, and control characters other than tab/CR/LF.
- `invisible.strip_joiners_safe_context` — ZWNJ/ZWJ and bidi controls, **only** where no script uses them. ZWNJ is a required letter-shaping control in Persian, Arabic and Indic scripts, and ZWJ is what welds an emoji sequence together, so the rule is gated on context: a value containing such a script, or a joiner between two pictographs, is left alone and flagged instead.
- `html.strip_tags` — tags removed with a block/inline distinction, so `<p>A</p><p>B</p>` becomes `A B` and not `AB`, and `un<b>break</b>able` does not gain spaces. Closed `<script>`/`<style>` elements are removed with their contents. Skipped entirely when the field is markup-typed by name (`*_html`, `body_html`, `raw`) or the value is a whole document.
- `html.unescape_entities_once` — decoded exactly once, with a re-entrancy guard: a decode whose own output is still entity- or tag-like is refused and flagged as double-encoded. That is what keeps cleaning idempotent.
- `whitespace.normalize_spaces` — the non-breaking/typographic space family folded to a plain space. A deliberately narrow slice of NFKC (see below).
- `whitespace.trim` — leading/trailing only. Interior whitespace is never touched.

**OPT_IN — correct, but off by default (5 rules).** Each changes the row count, a value's type, or the schema, so none runs unless you name it.

- `placeholder_policy: "null_high_confidence"` — nulls only tokens that cannot be real data in any field (`n/a`, `#n/a`, `null`, `undefined`, `nan`, `not available`, `tbd`…). Never the ambiguous ones.
- `placeholder_policy: "null_all"` — extends that to the ambiguous tokens. Documented as the one setting that can destroy legitimate data; fully reversible from the tier-3 ledger.
- `drop_exact_duplicates` — removes rows byte-identical to an earlier row, compared **after** cleaning (two rows differing only by a stripped BOM are the same row once repaired). Dropped indices are recorded.
- `coerce_numeric_text` — `"US $5.59"` → `5.59`, per field, all-or-nothing. A lone `,` (1.5 in fr/de, malformed in en), a `1.234`-shaped group, or a mixed currency affix disqualifies the whole field rather than being guessed at.
- `repair_keys` — applies the AUTO rules to dict keys as well (the classic BOM-prefixed `sku` first column of a CSV export). Never overwrites an existing key.

**FLAG — reported with a proposal, never applied, and no option turns them on (8 rules).**

- `boilerplate.flag` — `captcha`, `access denied`, `enable JavaScript`, `404 not found`. Nothing here can be repaired: the content was never extracted. Blanking the field or dropping the row would destroy the single most useful thing the value tells you — **this record must be re-scraped**. A cleaner that deletes these hands back a dataset that looks perfect and is missing the rows that mattered.
- `duplicates.flag_near` — candidate pairs with similarity scores. Merging is irreversible destruction on the basis of a threshold: `iPhone 15 Pro 128GB` and `iPhone 15 Pro 256GB` beat any threshold you would pick and are different products. There is no auto-merge option, by design.
- `placeholder.flag_ambiguous` — `none`, `na`, `nil`, `unknown`, `empty`, `-`, `?`. Every one has a realistic reading as real data: "None" is a surname, "NA" is Namibia's ISO code, "-" is a real value in a hyphenated field, "Unknown" is a deliberate category label. No field-level statistic distinguishes "this column is 8% masked-missing" from "this column has 8% Namibia".
- `normalization.flag_nfkc` — every NFKC difference that is not a space mapping, with the NFKC form supplied as a proposal. NFKC is advertised as normalization but is lossy: it rewrites `10²` to `102`, `½` to `1/2`, `㎡` to `m2`, `①` to `1`. On product and measurement data that is silent numeric corruption.
- `html.flag_document_or_typed_field` — markup in a field whose name says it holds markup, or a whole HTML document. There the markup is the payload.
- `html.flag_unclosed_executable` — an unclosed `<script>`/`<style>`: no way to know where the code stops and the content resumes, so nothing is removed. The value is reported as unsafe to render.
- `impossible.flag` — negative prices, out-of-range ratings, impossible dates. **Never repaired at all.** An impossible value is a semantic defect; no transformation recovers the true price from `-5`, and every candidate (drop, absolute-value, null) is an invention. This engine repairs how data was encoded, never what it says.
- `engine.revert_on_new_defect` — a post-condition, not a rule you trigger: after repair, the value is re-run through the detector, and if it exhibits any defect class the original did not, the entire repair of that value is reverted and flagged.

### Guarantees

- **I1 no-empty** — a non-empty value is never turned into an empty one.
- **I2 no-type-change** — AUTO rules never change a value's JSON type.
- **I3 no-structural-change** — AUTO rules never add, remove or reorder rows or keys.
- **I4 idempotence** — `clean(clean(x)) == clean(x)`, enforced by construction (fixpoint repair loop, decode-once guard).
- **I5 clean-in-clean-out** — a value with no detected defect is returned byte-identical.
- **I6 no-new-defects** — see `engine.revert_on_new_defect`.
- **I7 reversibility** — the tier-3 ledger restores the exact input, byte for byte.

Measured on a real 100-row Amazon scrape (57,956 nested string values): 453 repairs, 0 reverts, 0 values emptied, structure unchanged, idempotent, ledger reversible. `html_markup` 237→9 (the 9 are double-encoded entities, refused by design), `invisible_chars` 123→7 (all 7 are the ZWJ inside `🤷🏽‍♀️`), `non_normalized` 109→108 (NFKC is flag-only), `placeholder` 15→15, `boilerplate` 1→1. A 10-row control set of legitimate-but-suspicious data — correct French accents, Persian requiring its ZWNJ, emoji ZWJ sequences, `10²`, `AT&T`, a surname "None", Namibia as "NA", a `description_html` field — returns **byte-for-byte identical: 0 changes, 11 flags**.

### Tier 3: the audit trail

`POST /api/clean/audit` returns identical `data` to tier 2 — same engine, trace switched on — plus an `audit` block:

- `ledger` — every transformation, with `path`, `path_steps` (machine-usable, never parsed from a string), `rule`, `action`, `before`, `after`. It is a complete inverse patch: applying it in reverse reconstructs the input byte for byte.
- `replay_id` — `sha256(input_sha256 | options_sha256 | ruleset_version)`. Two runs sharing a replay id produced identical `data`. It hashes the three things that determine the output — not a request id, not a timestamp.
- `input_sha256` / `output_sha256`, and the full `ruleset` manifest as it was at the time of the run.

`generated_at` is the only non-deterministic field and sits outside the hashed region.

### Known limitation, stated rather than hidden

A single `&amp;` that a document meant **literally** is decoded — prose *about* HTML escaping loses its escaping. Accepted deliberately: across scraped fields `&amp;` is extraction residue essentially every time, and flagging every entity would gut the rule that fires most often (226 of 453 repairs on the real Amazon scrape). It is reversible from the tier-3 ledger, and pinned as a named test so it stays a known limitation rather than becoming a surprise.

### The same engine as a free Python library

The tier-2 cleaner above also ships as a standalone package under Apache-2.0:

```
pip install scrubkit
```

```python
from scrubkit import clean

result = clean(scraped_rows)
result.data       # cleaned, in the shape you passed in
result.flags      # what it refused to touch, and why
result.changes    # the reversible ledger described above
result.revert()   # reconstructs your exact input from it
```

Same 20 rules, same AUTO/OPT_IN/FLAG boundary, same `ruleset_version`, same determinism and idempotence guarantees. Zero dependencies — standard library only; near-duplicate detection is an optional extra (`pip install "scrubkit[duplicates]"`), and without it exact-duplicate detection still works and the near scan states why it was skipped. The 65-test suite ships **inside** the installed package, so the determinism claim is checkable locally rather than taken on trust: `pip install "scrubkit[test]"` then `python -m pytest --pyargs scrubkit`.

What the library does **not** include: the quality **verdict**, the 8 checks, the scoring and the token benchmark. Those are the `POST /api` tier and stay server-side.

Choose the library when you are running Python and can install a package. Choose the endpoints above when you cannot install anything, are not in Python, want an x402-payable call from an agent, or need the verdict.

Source: [github.com/aidatatools-dev/scrubkit](https://github.com/aidatatools-dev/scrubkit) · Package: [pypi.org/project/scrubkit](https://pypi.org/project/scrubkit/)

## Other transports

- **MCP over HTTP** — `https://www.aidatatools.dev/api/mcp_server` (Streamable HTTP, stateless). `check_dataset_quality` is **free** here and returns the full verdict. `clean_scraped_data` and `clean_scraped_data_audited` are listed too, but they do **not** return the repaired data over MCP: they inspect your data, report which rules would change how many values and what needs your decision, and return the paid REST call that hands back the repaired artifact (`status: "payment_required"`). Detection is free on this endpoint; repair is the paid product. The tool descriptions say so explicitly, so no agent is surprised after the fact.
- **A2A** — `https://www.aidatatools.dev/api/a2a` (JSONRPC binding, `message/send`). Serves the **verdict skill only**; there is no repair skill over A2A. See [/.well-known/agent-card.json](/.well-known/agent-card.json).
- **Apify Actor** and an **MCP stdio server** for direct integration with Claude, Cursor, or any MCP-compatible client.

## Machine-readable specs

- [/openapi.json](/openapi.json) — OpenAPI 3.1, all three tiers
- [/api/clean](/api/clean) — `GET`: the live repair boundary, 20 rules and 7 invariants, before you pay
- [/.well-known/agent-card.json](/.well-known/agent-card.json) — A2A v1.0 Agent Card
- [/.well-known/x402](/.well-known/x402) — x402 discovery, the three paid resources

## Tags

data cleaning, scraping repair, post-scrape sanitization, deterministic data repair, mojibake correction, encoding repair, html stripping, invisible character removal, unicode normalization, scraper output cleaning, data quality, dataset validation, duplicate detection, outlier detection, pipeline quality gate, RAG pipeline guardrail, auditable data repair, reversible data cleaning