A
B
' into 'AB' (a new, wrong word) or 'unbreakable' into 'un break able' (a broken one). Deliberately NOT applied when the field is markup-typed by name (*_html, body_html, raw, markup) or the value is a whole document ( '&', 'é' -> '\u00e9'), and refuse the decode when its own result still looks like an entity or a tag.", "why": "Decoding to a fixpoint is the classic data-corruption bug: '&' becomes '&' becomes '&', so cleaning twice gives a different answer than cleaning once, and any value carrying a deliberately double-encoded entity is destroyed. Decoding exactly once is right, but on its own it still breaks idempotence -- clean('&') is '&', and cleaning THAT gives '&'. The re-entrancy guard closes both holes at once: a decode whose own output is still entity-like or tag-like is refused and FLAGged as double-encoded, so a second pass can never dig deeper than the first. Idempotence here is enforced by construction, not hoped for.\nKNOWN LIMITATION, stated rather than hidden: this rule DOES decode a single '&' that a document meant literally -- prose *about* HTML escaping ('write & to get an ampersand') comes back with the ampersand decoded. That case is real, it is rare next to the millions of scraped fields where '&' is pure extraction residue, and it is the one place in the AUTO tier where the engine can be wrong about intent. It stays AUTO for that ratio, and the ledger makes it reversible; a caller who ingests HTML documentation should clean with the tier-3 envelope and check the html.unescape_entities_once entries.", "reversible_from_ledger": true, "invariants": ["decode-once", "refuses re-entrant decodes", "idempotent", "reversible from the ledger"]}, {"rule_id": "whitespace.normalize_spaces", "detects": "non_normalized", "action": "AUTO", "what": "Map the non-breaking / typographic space family (U+00A0, U+202F, U+2007, U+2009, U+2002-2006, U+205F, U+3000) to a plain space.", "why": "A DELIBERATELY NARROW slice of what NFKC would do, because full NFKC is the second trap in this domain. NFKC is advertised as 'normalization' but it is lossy: it rewrites '10\u00b2' to '102', '\u00bd' to '1/2', '\u33a1' to 'm2', '\u2460' to '1'. On a product feed -- exactly the data this tier is for -- that silently changes measurements and prices. So the engine repairs only the space characters, where the mapping is genuinely meaning-preserving, and reports every OTHER NFKC difference as a FLAG with the NFKC form as a proposal. Anyone who wants full NFKC can apply the proposal themselves; this engine will not do it to them by surprise.", "reversible_from_ledger": true, "invariants": ["strict subset of NFKC", "never applies compatibility folding"]}, {"rule_id": "whitespace.trim", "detects": "(new signal) leading/trailing whitespace", "action": "AUTO", "what": "Strip leading and trailing whitespace. Interior whitespace is never touched, except inside a value where tag removal created the run.", "why": "The lowest-risk repair in the set and the most common scraping artifact (' Nice lamp\\n' from a text node). Interior whitespace is left alone because it can be meaningful (addresses, code, poetry); leading/trailing whitespace in a scraped field never is. Recorded in the ledger like everything else, and switchable off with trim_whitespace=False for the caller who disagrees.", "reversible_from_ledger": true}, {"rule_id": "placeholder.null_high_confidence", "detects": "placeholder", "action": "OPT_IN", "what": "Replace a masked-missing string with JSON null, restricted to tokens that cannot plausibly be real data in any field: n/a, #n/a, n.a., n/d, nan, null, undefined, (none), (empty), not available, not applicable, no data, tbd, tba, to be determined/announced, missing.", "why": "Nulling a placeholder is the fix that makes the OTHER checks honest -- completeness counts 'N/A' as present, types counts it as a fine string. But it changes the value's type (str -> null), which is a schema-level event, so it is opt-in even for the unambiguous tokens. The genuinely ambiguous tokens are not merely gated behind this option: they are excluded from it entirely -- see placeholder.flag_ambiguous.", "option": "placeholder_policy='null_high_confidence'", "reversible_from_ledger": true, "invariants": ["changes type", "restricted to the high-confidence token set"]}, {"rule_id": "placeholder.null_all", "detects": "placeholder", "action": "OPT_IN", "what": "Same, extended to the ambiguous tokens (none, na, unknown, -, ?, ...).", "why": "Offered because some callers know their domain ('this is a price feed, \"-\" is never a price') and are right to want it. Documented as the one setting in this engine that can destroy legitimate data, named so that nobody arrives at it by accident, and never a default. The ledger makes it fully reversible if the caller was wrong.", "option": "placeholder_policy='null_all'", "reversible_from_ledger": true, "invariants": ["caller-asserted domain knowledge", "fully reverted by the ledger"]}, {"rule_id": "duplicates.drop_exact", "detects": "duplicates.exact", "action": "OPT_IN", "what": "Keep the first occurrence of each byte-identical row (comparison made AFTER cleaning, so rows that differ only by a stripped BOM collapse correctly), drop the rest, and record every dropped index.", "why": "Byte-identical rows are usually a paginated scrape overlapping itself. But 'usually' is not 'always': two identical line items in an order, or two identical sensor readings, are legitimate data, and dropping them changes every count and sum computed downstream. Row count is exactly the kind of thing another system depends on, so this stays opt-in -- and when off, the duplicates are still reported so the caller can flip one flag and re-run.", "option": "drop_exact_duplicates=True", "reversible_from_ledger": true, "invariants": ["stable: first occurrence wins", "dropped indices recorded"]}, {"rule_id": "types.coerce_numeric_text", "detects": "types.numeric_stored_as_text", "action": "OPT_IN", "what": "Parse 'US $5.59' -> 5.59, field by field, all-or-nothing, and only for a field where every value shares one currency/unit affix and every separator resolves unambiguously.", "why": "High-value (a text price cannot be compared or summed) but three real hazards: locale ('1,234' is 1234 in en-US and 1.234 in de-DE), unit loss (dropping '$' discards which currency this is), and ranges ('5-7'). The engine refuses rather than guesses: a single ambiguous value disqualifies the whole field, so a field is never left half-numeric and half-text -- which would be worse than what it started as. Off by default because it changes the field's type.", "option": "coerce_numeric_text=True", "reversible_from_ledger": true, "invariants": ["all-or-nothing per field", "no ambiguous separator is ever guessed", "mixed currencies disqualify the field"]}, {"rule_id": "keys.repair", "detects": "invisible_chars / mojibake in a key", "action": "OPT_IN", "what": "Apply the AUTO text rules to dict KEYS as well as values (the classic '\\ufeffsku' first column of a BOM-prefixed CSV export), skipping any repair that would collide with an existing key.", "why": "A key is a contract with every downstream consumer, and renaming one can break a mapping that a human wrote by hand -- including one that was written against the broken name. That is a bigger blast radius than any value-level fix, so it is opt-in despite the repair itself being just as safe as on a value. Collisions are never resolved by overwriting.", "option": "repair_keys=True", "reversible_from_ledger": true, "invariants": ["never overwrites an existing key", "never reorders keys"]}, {"rule_id": "boilerplate.flag", "detects": "boilerplate", "action": "FLAG", "what": "Report the value and the phrase that matched ('click here', 'access denied', 'captcha', '404 not found', 'enable javascript'...).", "why": "There is nothing to repair: the content was never extracted. The only transformations available are to blank the field or drop the row, and both destroy the single most useful thing this value tells you -- that the scrape failed on this record and must be re-run. A cleaner that deletes these hands back a dataset that looks 100% clean and is missing the rows that mattered. Flagging them is not a gap in the product; on a scraping pipeline it is the highest-value output the engine produces.", "reversible_from_ledger": false}, {"rule_id": "duplicates.flag_near", "detects": "duplicates.near", "action": "FLAG", "what": "Report candidate near-duplicate row pairs with their similarity score.", "why": "Merging fuzzy duplicates is irreversible destruction of a record on the basis of a threshold. 'iPhone 15 Pro 128GB' and 'iPhone 15 Pro 256GB' score above any threshold you would pick and are different products; so do two genuinely distinct listings from two sellers. There is no deterministic, domain-free rule that separates the duplicate from the variant, so the engine offers NO option to auto-merge -- not even an opt-in one. Deciding this requires knowing the domain, which is the caller's job, and the pairs are handed over so it is a cheap one.", "reversible_from_ledger": false}, {"rule_id": "placeholder.flag_ambiguous", "detects": "placeholder", "action": "FLAG", "what": "Report -- and never null, under any policy short of the explicitly reckless null_all -- the ambiguous tokens: none, na, nil, unknown, empty, -, --, ---, ?, ??, ???.", "why": "Every one of these has a realistic dataset where it is real data. 'None' is a surname and a genuine enum value. 'NA' is Namibia's ISO code, North America, and sodium. 'Unknown' is a legitimate category label used on purpose. '-' is a real value in a hyphenated field. No field-level statistic distinguishes 'this column is 8% masked-missing' from 'this column has 8% Namibia', because both look identical. This is the textbook case for propose-don't-apply.", "reversible_from_ledger": false}, {"rule_id": "normalization.flag_nfkc", "detects": "non_normalized", "action": "FLAG", "what": "Report every NFKC difference that is NOT a space-family mapping, with the NFKC form supplied as a proposal.", "why": "See whitespace.normalize_spaces. NFKC folds superscripts, fractions, unit ligatures and circled numbers into ASCII lookalikes, which on measurement and price data is silent numeric corruption ('10\u00b2' -> '102'). The proposal is handed over because on a pure-text corpus NFKC is often exactly right -- the engine just will not be the one to decide that for a dataset it knows nothing about.", "reversible_from_ledger": false}, {"rule_id": "html.flag_document_or_typed_field", "detects": "html_markup", "action": "FLAG", "what": "Report, without stripping, markup in a field whose name says it holds markup (*_html, html, raw, markup, body_html) or whose value is a whole document ().", "why": "Here the markup is the payload. Stripping tags from `description_html` does not clean the field, it empties it of everything the consumer asked for. Name-based and shape-based detection are both cheap and both conservative in the safe direction.", "reversible_from_ledger": false}, {"rule_id": "html.flag_unclosed_executable", "detects": "executable_markup", "action": "FLAG", "what": "Report an unclosed is removed with its contents (AUTO, via html.strip_tags) because everything inside it is code, not content. An UNCLOSED one gives no way to know where the code stops and the content resumes, so removing to end-of-string would delete arbitrary real text. The safe fallback is to report it -- with the note that this value is unsafe to render.", "reversible_from_ledger": false}, {"rule_id": "impossible.flag", "detects": "impossible_values (tier-1 check)", "action": "FLAG", "what": "Never repaired by this engine at all: negative prices, out-of-range ratings, impossible dates stay exactly as they are.", "why": "An impossible value is a *semantic* defect, not an encoding artifact. There is no transformation that recovers the true price from -5; every candidate (drop it, absolute-value it, null it) is an invention. This engine repairs how data was encoded, never what it says. That line is why the repaired output can be trusted to still be your data.", "reversible_from_ledger": false}, {"rule_id": "engine.revert_on_new_defect", "detects": "(post-condition, all rules)", "action": "FLAG", "what": "After repair, the value is re-run through the tier-1 detector. If it exhibits any defect class the original did not, the ENTIRE repair of that value is reverted and the value is flagged instead.", "why": "The backstop that turns the existing detector into the guardrail on the repairer. It means no future rule -- including one added carelessly -- can make a value dirtier than it found it, and it costs one extra classify() call per changed value. Together with the no-empty invariant, it is why the engine can promise that a repair never makes things worse, rather than merely intending to.", "reversible_from_ledger": false}], "engine_invariants": ["I1 no-empty: a non-empty value is never turned into an empty one", "I2 no-type-change: AUTO rules never change a value's JSON type", "I3 no-structural-change: AUTO rules never add/remove/reorder rows or keys", "I4 idempotence: clean(clean(x)) == clean(x), by construction", "I5 clean-in-clean-out: a value with no detected defect is returned byte-identical", "I6 no-new-defects: a repair that trips a detector the original did not is reverted", "I7 reversibility: the tier-3 ledger restores the exact input, byte for byte"]}, "payment": {"x402Enabled": true, "activeNetworks": ["solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp", "eip155:8453"]}}