What this tool removes

What it removes

Everything in this list is a character in your text. It is really there, and taking it out is deterministic and verifiable.

• Zero-width characters: U+200B, U+200C, U+200D, U+FEFF and the word joiner U+2060
• Soft hyphens, combining grapheme joiners and invisible operators
• Bidirectional controls, U+202A to U+202E and U+2066 to U+2069
• Unicode tag characters, U+E0000 to U+E007F, which can carry a hidden message
• Private-use codepoints and noncharacters
• Sixteen non-standard spaces, folded to an ordinary space rather than deleted
• Confusable letters (Cyrillic and Greek lookalikes), but only inside an otherwise-Latin word
• Fullwidth forms and mathematical alphanumerics, folded back to plain letters
• Optionally, on request: em dashes, en dashes and curly quotes

What it deliberately leaves alone

Some invisible characters are load-bearing. Removing them corrupts the text, so the rules are context-aware rather than blind.

• Zero-width joiners between emoji, because a family emoji is several people held together by joiners
• Zero-width joiners and non-joiners in Persian, Arabic and Devanagari, where they are real letters
• Variation selectors following an emoji, which decide whether it renders as an emoji or an outline glyph
• Genuine Cyrillic and Greek text: a lookalike is only folded when its neighbours are Latin letters
• Em dashes and curly quotes, unless you explicitly ask for them, because they are correct punctuation

How the rules are applied

Every rule is a separate switch, and every switch has a default chosen so that running the cleaner on ordinary text is safe. Invisible characters, spaces, confusables and styled letterforms are on. Punctuation is off, because em dashes and curly quotes are correct punctuation and folding them is a style choice rather than a repair.

Through the API each switch is individually controllable per request, so a publishing pipeline can be strict where it needs to be and conservative everywhere else.

A note on provenance data

The C2PA standard has begun defining how a content credential can be embedded in plain text, using Unicode variation selectors. That is a signed provenance record rather than a covert mark, and stripping it would remove information a reader might legitimately want.

Our variation-selector handling is already conservative (selectors following an emoji are preserved), and we treat provenance data as something to report rather than quietly delete.