A
B
` does not become `AB`; HTML entities decoded exactly once, with a re-entrancy guard; zero-width, BOM, soft-hyphen and control characters removed; the non-breaking space family folded to a plain space; leading/trailing whitespace trimmed. - **OPT_IN (5 rules)** — correct, but they change the row count, a value's type, or the schema, so they are never applied unless you ask by name: nulling unambiguous placeholders, dropping byte-identical duplicate rows, coercing `"US $5.59"` → `5.59` (per field, all-or-nothing, only where unambiguous), repairing dict keys. - **FLAG (8 rules)** — reported with a concrete proposal, **never applied, and no option turns them on**. Near-duplicate rows are never merged (`iPhone 15 Pro 128GB` and `256GB` beat any threshold and are different products). Ambiguous placeholders are never nulled (`None` is a surname, `NA` is Namibia, `-` is a real value). Full NFKC is never applied (it rewrites `10²` to `102`). Failed extractions (`captcha`, `access denied`, `enable JavaScript`) are never deleted — that value is the most useful thing in the row: **re-scrape that record**. Impossible values — a negative price, an out-of-range rating — are never "repaired" at all. This engine repairs how data was encoded, never what it says. That line is why the output is still your data. ## Guarantees - **Deterministic** — no LLM, no sampling. Same input + same options + same `ruleset_version` → byte-identical output. Cacheable, replayable, auditable. - **Idempotent** — cleaning the result again changes nothing. - **Non-destructive** — no value is ever emptied, retyped, or reordered by an automatic rule. - **No new defects** — every repaired value is re-checked by the detector; a repair that would introduce a defect the original did not have is reverted and flagged instead. - **Reversible** — tier 3's ledger reconstructs your input byte for byte. Measured on a real 100-row Amazon scrape (57,956 nested string values): 453 repairs, 0 reverts, 0 values emptied, structure unchanged. A 10-row control set of legitimate-but-suspicious data — correct French accents, Persian requiring its ZWNJ, emoji ZWJ sequences, `10²`, `AT&T`, a surname "None", Namibia as "NA" — comes back **byte-for-byte identical: 0 changes, 11 flags**. ## Verdict: when to call it instead - Before feeding a dataset to an LLM or agent for reasoning or decision-making - Before loading data into a RAG pipeline - Before a trading agent acts on aggregated price data pulled from multiple sources - Whenever an agent receives data from an unknown or untrusted source `POST /api` returns `meta`, `facts`, `score`, `verdict`, `benchmark`. Some checks report without judging — `price_divergence` (cross-source disagreement on the same asset), `text_cleanliness` (the artifacts the cleaner above repairs), and `outliers.modified_z_mad` (a robust median/MAD cross-check). These appear in `facts` and are deliberately **not** folded into `score`. ``` curl -X POST https://www.aidatatools.dev/api \ -H "Content-Type: application/json" \ -d '[{"id": 1, "price": 10}, {"id": 2, "price": null}]' ``` ## MCP Streamable HTTP MCP server at `https://www.aidatatools.dev/api/mcp_server`. Three tools: - `check_dataset_quality` — **free**, returns the full verdict. - `clean_scraped_data` / `clean_scraped_data_audited` — these do **not** return the repaired data over MCP. They inspect your data, tell you exactly which rules would change how many values and what needs your decision, and return the paid REST call that hands back the repaired artifact. Detection is free here; repair is the paid product. The tool descriptions say so, and the response carries `status: "payment_required"`. ## Docs - [Full reference](/llms-full.txt) — every rule, every check, complete output schemas, worked examples - [OpenAPI 3.1](/openapi.json) — machine-readable contract for all three tiers - [Repair boundary](/api/clean) — live, machine-readable: the 20 rules and 7 invariants, before you pay - [A2A Agent Card](/.well-known/agent-card.json) — agent-to-agent discovery (verdict only; cleaning is REST/MCP) - [x402 discovery](/.well-known/x402) — the three paid resources Tags: data cleaning, scraping repair, post-scrape sanitization, deterministic data repair, mojibake correction, encoding repair, data quality, scraper output validation, pipeline quality gate.