Skip to content

Normalize & Zawgyi

burmesenlp.normalize.normalize 

normalize(text: str, *, warn_zawgyi: bool = True) -> str

Normalize Myanmar text for segmentation.

  • Validates the input type (raises TypeError for non-str).
  • Strips zero-width space/joiner/non-joiner, word-joiner and BOM.
  • Applies Unicode NFC (e.g. composes U+1025 U+102E into U+1026).
  • Logs a warning if the text looks Zawgyi-encoded; conversion to Unicode must be done by the caller. Pass warn_zawgyi=False when normalizing dictionary keys in bulk (avoids noisy false positives).

NFC does not canonicalize Myanmar syllable-mark order: every medial/vowel/anusvara/visarga sign has Canonical_Combining_Class 0 (only asat and dot below are non-zero), so Unicode's canonical reordering algorithm never touches their relative order. Two different input-method key orders for the same syllable stay distinct strings through NFC forever. Call :func:canonical_order separately (opt-in; not applied here) if that matters for your data.

Source code in src/burmesenlp/normalize/__init__.py 
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
def normalize(text: str, *, warn_zawgyi: bool = True) -> str:
    """Normalize Myanmar text for segmentation.

    - Validates the input type (raises ``TypeError`` for non-``str``).
    - Strips zero-width space/joiner/non-joiner, word-joiner and BOM.
    - Applies Unicode NFC (e.g. composes U+1025 U+102E into U+1026).
    - Logs a warning if the text looks Zawgyi-encoded; conversion to
      Unicode must be done by the caller.  Pass ``warn_zawgyi=False`` when
      normalizing dictionary keys in bulk (avoids noisy false positives).

    NFC does **not** canonicalize Myanmar syllable-mark order: every
    medial/vowel/anusvara/visarga sign has Canonical_Combining_Class 0
    (only asat and dot below are non-zero), so Unicode's canonical
    reordering algorithm never touches their relative order. Two
    different input-method key orders for the same syllable stay
    distinct strings through NFC forever. Call :func:`canonical_order`
    separately (opt-in; not applied here) if that matters for your data.
    """
    if not isinstance(text, str):
        raise TypeError(f"expected str, got {type(text).__name__}")
    if not text:
        return ""
    if warn_zawgyi and looks_like_zawgyi(text):
        logger.warning(
            "Input triggered a Zawgyi heuristic that also fires on Shan, "
            "Mon, Kayah, Karen, and Rumai Palaung text -- this check cannot "
            "tell those apart from Zawgyi. If this is genuinely Zawgyi, "
            "convert it first (zg2uni()/to_unicode()); if it's one of "
            "those other languages, segmentation results may still be "
            "unreliable but no conversion is needed. For a calibrated "
            "answer instead of this heuristic, use "
            "burmesenlp.zawgyi.get_zawgyi_probability()."
        )
    text = text.translate(_ZERO_WIDTH_TABLE)
    return unicodedata.normalize("NFC", text)

burmesenlp.normalize.looks_like_zawgyi 

looks_like_zawgyi(text: str) -> bool

Heuristically detect Zawgyi-encoded text.

This is a lightweight rule-based check, not a trained detector, and it is measurably wrong on minority-language text: measured false-positive rate on real Wikipedia text (paragraph-level) is Shan ~97.5% (119/122), Mon ~72% (283/393). See _ZAWGYI_HINTS above for why no cheap rewrite of this pattern fixes that (dropping just the codepoint-range clause still leaves ~59%/~45%). This function only gates a warning in normalize() -- nothing branches on it -- so the fix under consideration is different from is_zawgyi()'s (see burmesenlp.zawgyi.zawgyi, which DOES branch on its equivalent check and has been changed accordingly). For a calibrated, non-heuristic answer, use :func:burmesenlp.zawgyi.detector.get_zawgyi_probability.

Source code in src/burmesenlp/normalize/__init__.py 
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
def looks_like_zawgyi(text: str) -> bool:
    """Heuristically detect Zawgyi-encoded text.

    This is a lightweight rule-based check, not a trained detector, and
    it is measurably wrong on minority-language text: measured
    false-positive rate on real Wikipedia text (paragraph-level) is
    **Shan ~97.5% (119/122), Mon ~72% (283/393)**. See ``_ZAWGYI_HINTS``
    above for why no cheap rewrite of this pattern fixes that (dropping
    just the codepoint-range clause still leaves ~59%/~45%). This
    function only gates a warning in ``normalize()`` -- nothing branches
    on it -- so the fix under consideration is different from
    ``is_zawgyi()``'s (see ``burmesenlp.zawgyi.zawgyi``, which DOES
    branch on its equivalent check and has been changed accordingly).
    For a calibrated, non-heuristic answer, use
    :func:`burmesenlp.zawgyi.detector.get_zawgyi_probability`.
    """
    return bool(_ZAWGYI_HINTS.search(text))

burmesenlp.zawgyi.zg2uni 

zg2uni(text: str) -> str

Convert Zawgyi-encoded text to standard Unicode Myanmar.

Source code in src/burmesenlp/zawgyi/zawgyi.py 
120
121
122
123
124
def zg2uni(text: str) -> str:
    """Convert Zawgyi-encoded text to standard Unicode Myanmar."""
    if not isinstance(text, str):
        raise TypeError(f"expected str, got {type(text).__name__}")
    return _apply_rules(text, _zg2uni_rules())

burmesenlp.zawgyi.uni2zg 

uni2zg(text: str) -> str

Convert standard Unicode Myanmar text to Zawgyi encoding.

Source code in src/burmesenlp/zawgyi/zawgyi.py 
113
114
115
116
117
def uni2zg(text: str) -> str:
    """Convert standard Unicode Myanmar text to Zawgyi encoding."""
    if not isinstance(text, str):
        raise TypeError(f"expected str, got {type(text).__name__}")
    return _apply_rules(text, _uni2zg_rules())

burmesenlp.zawgyi.to_unicode 

to_unicode(text: str, *, normalize: bool = True, threshold: float = DEFAULT_ZAWGYI_THRESHOLD) -> str

Ensure text is standard Unicode Myanmar.

Detects Zawgyi and converts only if needed, then applies NFC by default. Safe to call on text of unknown encoding.

Detection uses :func:burmesenlp.zawgyi.detector.get_zawgyi_probability (a bigram Markov model), NOT :func:is_zawgyi -- that heuristic's codepoint-range signal is also the Unicode block for Shan/Mon/Kayah/ Karen/Rumai Palaung letters (see :func:is_zawgyi's docstring for measured false-positive rates), and a false positive here is irreversible, unlike a false negative. See :data:DEFAULT_ZAWGYI_THRESHOLD for why the default isn't 0.5.

Source code in src/burmesenlp/zawgyi/zawgyi.py 
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
def to_unicode(text: str, *, normalize: bool = True, threshold: float = DEFAULT_ZAWGYI_THRESHOLD) -> str:
    """Ensure *text* is standard Unicode Myanmar.

    Detects Zawgyi and converts only if needed, then applies NFC by default.
    Safe to call on text of unknown encoding.

    Detection uses :func:`burmesenlp.zawgyi.detector.get_zawgyi_probability`
    (a bigram Markov model), NOT :func:`is_zawgyi` -- that heuristic's
    codepoint-range signal is also the Unicode block for Shan/Mon/Kayah/
    Karen/Rumai Palaung letters (see :func:`is_zawgyi`'s docstring for
    measured false-positive rates), and a false positive here is
    irreversible, unlike a false negative. See
    :data:`DEFAULT_ZAWGYI_THRESHOLD` for why the default isn't 0.5.
    """
    if not isinstance(text, str):
        raise TypeError(f"expected str, got {type(text).__name__}")
    if get_zawgyi_probability(text) > threshold:
        text = zg2uni(text)
    if normalize:
        text = unicodedata.normalize("NFC", text)
    return text

burmesenlp.zawgyi.is_zawgyi 

is_zawgyi(text: str) -> bool

Heuristic check for whether text is Zawgyi-encoded.

Fast, but NOT reliable on minority-language text: its codepoint-range signal is also the Unicode block for Shan/Mon/Kayah/Karen/Rumai Palaung letters, and there is no way to narrow that range to exclude them, because Zawgyi's own glyph-variant reuse and those languages' genuine letters are frequently the identical codepoint -- only authorial intent tells them apart, not the codepoint itself (see zg2uni's rule 5 and its 14 Karen-range rules). Measured false-positive rate on real Wikipedia text, paragraph-level: Shan ~100% (119/122), Mon ~72% (283/393).

For a calibrated answer, use :func:burmesenlp.zawgyi.detector.get_zawgyi_probability instead -- it scores 0.0 on both samples above. This function is kept unchanged for callers who rely on its exact boolean semantics; to_unicode() no longer calls it internally for exactly this reason (a false positive here has no recovery path -- see that function).

Source code in src/burmesenlp/zawgyi/zawgyi.py 
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
def is_zawgyi(text: str) -> bool:
    """Heuristic check for whether *text* is Zawgyi-encoded.

    Fast, but NOT reliable on minority-language text: its codepoint-range
    signal is also the Unicode block for Shan/Mon/Kayah/Karen/Rumai
    Palaung letters, and there is no way to narrow that range to exclude
    them, because Zawgyi's own glyph-variant reuse and those languages'
    genuine letters are frequently the identical codepoint -- only
    authorial intent tells them apart, not the codepoint itself (see
    zg2uni's rule 5 and its 14 Karen-range rules). Measured false-positive
    rate on real Wikipedia text, paragraph-level: **Shan ~100%
    (119/122), Mon ~72% (283/393)**.

    For a calibrated answer, use
    :func:`burmesenlp.zawgyi.detector.get_zawgyi_probability` instead --
    it scores 0.0 on both samples above. This function is kept unchanged
    for callers who rely on its exact boolean semantics; ``to_unicode()``
    no longer calls it internally for exactly this reason (a false
    positive here has no recovery path -- see that function).
    """
    if not isinstance(text, str):
        raise TypeError(f"expected str, got {type(text).__name__}")
    if not text:
        return False

    length = len(text)
    for i, ch in enumerate(text):
        cp = ord(ch)
        if cp in _ZAWGYI_INDICATOR_RANGE or cp in _ZAWGYI_INDICATOR_EXTRA:
            return True
        if cp == _VIRAMA:
            next_cp = ord(text[i + 1]) if i + 1 < length else None
            if next_cp not in _CONSONANT_RANGE:
                return True

    return bool(_ZAWGYI_ORDER.search(text))