Normalize & Zawgyi

burmesenlp.normalize.normalize

normalize(text: str, *, warn_zawgyi: bool = True) -> str

Normalize Myanmar text for segmentation.

• Validates the input type (raises TypeError for non-str).
• Strips zero-width space/joiner/non-joiner, word-joiner and BOM.
• Applies Unicode NFC (e.g. composes U+1025 U+102E into U+1026).
• Logs a warning if the text looks Zawgyi-encoded; conversion to Unicode must be done by the caller. Pass warn_zawgyi=False when normalizing dictionary keys in bulk (avoids noisy false positives).

NFC does not canonicalize Myanmar syllable-mark order: every medial/vowel/anusvara/visarga sign has Canonical_Combining_Class 0 (only asat and dot below are non-zero), so Unicode's canonical reordering algorithm never touches their relative order. Two different input-method key orders for the same syllable stay distinct strings through NFC forever. Call :func:canonical_order separately (opt-in; not applied here) if that matters for your data.

Source code in src/burmesenlp/normalize/__init__.py

def normalize(text: str, *, warn_zawgyi: bool = True) -> str:
    """Normalize Myanmar text for segmentation.

    - Validates the input type (raises ``TypeError`` for non-``str``).
    - Strips zero-width space/joiner/non-joiner, word-joiner and BOM.
    - Applies Unicode NFC (e.g. composes U+1025 U+102E into U+1026).
    - Logs a warning if the text looks Zawgyi-encoded; conversion to
      Unicode must be done by the caller.  Pass ``warn_zawgyi=False`` when
      normalizing dictionary keys in bulk (avoids noisy false positives).

    NFC does **not** canonicalize Myanmar syllable-mark order: every
    medial/vowel/anusvara/visarga sign has Canonical_Combining_Class 0
    (only asat and dot below are non-zero), so Unicode's canonical
    reordering algorithm never touches their relative order. Two
    different input-method key orders for the same syllable stay
    distinct strings through NFC forever. Call :func:`canonical_order`
    separately (opt-in; not applied here) if that matters for your data.
    """
    if not isinstance(text, str):
        raise TypeError(f"expected str, got {type(text).__name__}")
    if not text:
        return ""
    if warn_zawgyi and looks_like_zawgyi(text):
        logger.warning(
            "Input looks like Zawgyi-encoded text; segmentation results "
            "will be unreliable. Convert to Unicode first."
        )
    text = text.translate(_ZERO_WIDTH_TABLE)
    return unicodedata.normalize("NFC", text)

burmesenlp.normalize.looks_like_zawgyi

looks_like_zawgyi(text: str) -> bool

Heuristically detect Zawgyi-encoded text.

This is a lightweight rule-based check, not a trained detector. Use a dedicated converter (e.g. ICU transliteration, myanmar-tools) for authoritative detection and conversion.

Source code in src/burmesenlp/normalize/__init__.py

def looks_like_zawgyi(text: str) -> bool:
    """Heuristically detect Zawgyi-encoded text.

    This is a lightweight rule-based check, not a trained detector.  Use a
    dedicated converter (e.g. ICU transliteration, myanmar-tools) for
    authoritative detection and conversion.
    """
    return bool(_ZAWGYI_HINTS.search(text))

burmesenlp.zawgyi.zg2uni

zg2uni(text: str) -> str

Convert Zawgyi-encoded text to standard Unicode Myanmar.

Source code in src/burmesenlp/zawgyi/zawgyi.py

def zg2uni(text: str) -> str:
    """Convert Zawgyi-encoded text to standard Unicode Myanmar."""
    if not isinstance(text, str):
        raise TypeError(f"expected str, got {type(text).__name__}")
    return _apply_rules(text, _zg2uni_rules())

burmesenlp.zawgyi.uni2zg

uni2zg(text: str) -> str

Convert standard Unicode Myanmar text to Zawgyi encoding.

Source code in src/burmesenlp/zawgyi/zawgyi.py

def uni2zg(text: str) -> str:
    """Convert standard Unicode Myanmar text to Zawgyi encoding."""
    if not isinstance(text, str):
        raise TypeError(f"expected str, got {type(text).__name__}")
    return _apply_rules(text, _uni2zg_rules())

burmesenlp.zawgyi.to_unicode

to_unicode(text: str, *, normalize: bool = True) -> str

Ensure text is standard Unicode Myanmar.

Detects Zawgyi and converts only if needed, then applies NFC by default. Safe to call on text of unknown encoding.

Source code in src/burmesenlp/zawgyi/zawgyi.py

def to_unicode(text: str, *, normalize: bool = True) -> str:
    """Ensure *text* is standard Unicode Myanmar.

    Detects Zawgyi and converts only if needed, then applies NFC by default.
    Safe to call on text of unknown encoding.
    """
    if not isinstance(text, str):
        raise TypeError(f"expected str, got {type(text).__name__}")
    if is_zawgyi(text):
        text = zg2uni(text)
    if normalize:
        text = unicodedata.normalize("NFC", text)
    return text

burmesenlp.zawgyi.is_zawgyi

is_zawgyi(text: str) -> bool

Heuristic check for whether text is Zawgyi-encoded.

Source code in src/burmesenlp/zawgyi/zawgyi.py

def is_zawgyi(text: str) -> bool:
    """Heuristic check for whether *text* is Zawgyi-encoded."""
    if not isinstance(text, str):
        raise TypeError(f"expected str, got {type(text).__name__}")
    if not text:
        return False

    length = len(text)
    for i, ch in enumerate(text):
        cp = ord(ch)
        if cp in _ZAWGYI_INDICATOR_RANGE or cp in _ZAWGYI_INDICATOR_EXTRA:
            return True
        if cp == _VIRAMA:
            next_cp = ord(text[i + 1]) if i + 1 < length else None
            if next_cp not in _CONSONANT_RANGE:
                return True

    return bool(_ZAWGYI_ORDER.search(text))