Shortcuts on this page
r m x toggle line displays
j k next/prev highlighted chunk
0 (zero) top of page
1 (one) first highlighted chunk
1"""Stage: script_segment (#271, #272, #308, #312).
3Consumes: tokens, segments, structure, interpunct_offsets, segmenter.
4Produces: tokens, by two independent splits into sub-slices -- a
5listed honorific peeled off the END of the name's last
6non-post-nominal token, in whichever of the name's runs that falls
7(the tail token also carries this module's _PEELED_TAG), and the
8first activated-script token of the name segment split into n+1 --
9segments (index runs remapped past the insertions), ambiguities
10(indices likewise remapped, plus a SEGMENTATION report when more than
11one split was vocabulary-supported, or when a segmenter's answer
12scored under the confidence floor).
13Reads: Policy.segment_scripts, Lexicon.surnames,
14Lexicon.honorific_tails, ParseState.segmenter, and Lexicon suffix
15vocabulary through TWO predicates, which are NOT one another's
16singular and plural. _vocab.is_suffix_strict asks whether a single
17token is a post-nominal, initial veto included (the peel's scan-back
18and the surname site). _vocab.is_wholly_suffix asks segment's own
19suffix-comma question of a whole RUN, through the POLICY-selected
20token test plus period_joined_vocab, delimiter handling and the
21Ph./D. merge -- so the run predicate says yes both to tokens the
22token predicate VETOES ("V.", "V", "I") and to tokens it never sees
23as suffixes at all ("Msc.Ed.", "J.씨", which reach it through
24period_joined_vocab). The initial-shaped words are the class #319 was
25ABOUT, not the whole of the disagreement, and the extra routes listed
26above are where the rest of it comes from;
27test_is_wholly_suffix_is_not_the_plural_of_is_post_nominal pins the
28"V." half. The run predicate is how the peel declines a run that is
29not name text (#319), but only where the name's own run offers a
30site, since a glued honorific is itself part of what makes a run read
31as suffix-shaped; it owns two further Policy fields
32(lenient_comma_suffixes picks the strict or lenient token test -- it
33flips "田中さん, V." -- and extra_suffix_delimiters both counts a bare
34delimiter-core token as a suffix and splits a token on a core, the
35split being the half that flips "田中さん, Jr./V." under {"/"} and the
36bare core the half that flips "田中さん, /").
38Implements rules W1 (the vocabulary/segmenter division), W2 (the
39glued-honorific peel) and W3 (the writer's divisions are respected)
40of docs/design/rules.md, cited at their code below; the decision
41chain (#308, #312, #319, the vetting bars, the measured
42spaced-honorific trade) is decisions.md#W1, #W2 and #W3. Both splits
43make sub-slices of one token, rewriting nothing -- spans still index
44the original exactly, so the anti-#100 invariant holds by
45construction. Both also match on the token's CORE, its
46text.rstrip(FULL_STOPS) (#323) -- rstrip and never strip, because
47both offsets are measured from the text's START, so the core has to
48stay a prefix of the text for the offset the match yields to land
49where the match did. The peel runs FIRST, so suffix classification
50can claim the tail and the surname match or segmenter consult sees
51the name rather than name-plus-honorific; its
52ASCII bail sits above everything here, so a caller-added LATIN tail
53fires only on a name carrying at least one non-ASCII character (see
54the bail's own comment, and honorific_tails' field note).
55"""
56from __future__ import annotations
58import dataclasses
59import functools
60from collections.abc import Sequence
62from nameparser._lexicon import FULL_STOPS
63from nameparser._pipeline._state import (
64 ParseState, PendingAmbiguity, Structure, WorkToken,
65)
66from nameparser._pipeline._vocab import (
67 effective_script, is_suffix_strict, is_wholly_suffix,
68)
69from nameparser._types import AmbiguityKind, Segmentation, Span
71#: Marks the tail token the peel below MANUFACTURED, so the segmenter's
72#: neighbour test can tell it from a token somebody wrote. Namespaced,
73#: therefore unstable provenance rather than API (_types.STABLE_TAGS is
74#: the whole stable set, and FOLDED_TAG is the precedent for a
75#: structural marker carrying this prefix); it is vocabulary-derived
76#: besides, since honorific_tails is what licensed the split. Emitter
77#: and reader are both in this module, so unlike FOLDED_TAG it needs no
78#: home in _types.
79_PEELED_TAG = "vocab:peeled-honorific"
81#: Segmenter answers scoring below this attach a SEGMENTATION report
82#: (amendment 2026-07-29 section 3). Kept at the drafted 0.9 after
83#: measuring namedivider 0.4.1 over 112 names (#272 Task 5), which
84#: found a distribution the amendment did not anticipate: the scores
85#: are BIMODAL, not clustered near 1. A rule-based division (the kana
86#: boundary in 高橋みなみ, or a two-character name) scores exactly 1.0;
87#: a kanji-statistics division scores a softmax over the candidate cut
88#: positions, observed in 0.23-0.68 and driven more by name LENGTH
89#: (median 0.61 at three characters, 0.37 at five -- the softmax is
90#: over len-1 candidates) than by correctness: the wrong answers in
91#: the sample scored 0.32/0.60/0.60/0.61, straddling the correct
92#: median of 0.52. So no floor INSIDE that band separates error from
93#: success, and the only cut the data supports is between a stated
94#: certainty and a statistical guess -- which is what section 3's
95#: epistemic argument asks for anyway. 0.9 sits in the empty gap
96#: (0.68 to 1.0), far from both modes, and reads as "confident" for a
97#: third-party segmenter with a calibrated score too. Consequence,
98#: pinned by tests and stated so it can be checked: every
99#: STATISTICALLY divided name carries a SEGMENTATION report, and every
100#: RULE-divided one -- the kana boundary, a two-character name,
101#: namedivider's specific-name rules -- carries none.
102#: namedivider scores its own two-character rule 1.0, so this floor
103#: keeps that division silent -- see locales/ja.py for why the
104#: presumption is accepted as stated.
105#: Not configurable in 2.x (YAGNI).
106_SEGMENTER_CONFIDENCE_FLOOR = 0.9
109def _remap(run: tuple[int, ...], split_at: int,
110 added: int) -> tuple[int, ...]:
111 """One segment run after token `split_at` split into `added` + 1
112 pieces: the extra pieces join the run immediately after it, and
113 every later index shifts by as many."""
114 out: list[int] = []
115 for j in run:
116 if j == split_at:
117 out.extend(range(split_at, split_at + added + 1))
118 else:
119 out.append(j + added if j > split_at else j)
120 return tuple(out)
123def _pieces(text: str, splits: tuple[int, ...]) -> tuple[str, ...]:
124 """`text` cut at every offset in `splits`: n offsets, n+1 pieces."""
125 cuts = (0, *splits, len(text))
126 return tuple(text[a:b] for a, b in zip(cuts, cuts[1:]))
129def _split(state: ParseState, i: int, splits: tuple[int, ...],
130 detail: str | None, tail_tag: str | None = None) -> ParseState:
131 """Cut token `i` at every offset in `splits`, recording `detail` as
132 a SEGMENTATION report when there is one, and adding `tail_tag` to
133 the LAST piece when there is one of those.
135 The offsets arrive non-empty, ascending and interior whatever chose
136 them. From a segmenter: non-empty is the caller's own check,
137 strictly ascending with each >= 1 is Segmentation.__post_init__'s,
138 and the last offset is < len(text) -- checked by the caller. From
139 the vocabulary: the single offset is >= 1 and < len(text) by the
140 range(cap, 0, -1) construction and its len-1 cap.
142 The ONE split path: the vocabulary hit is the single-offset case
143 and the segmenter's answer the general one, so neither can drift
144 from the other's index arithmetic. `tail_tag` is part of keeping it
145 one path: the piece is tagged HERE, where it is built, rather than
146 by the caller, which would have to re-derive where its own tail
147 landed after handing the index arithmetic off. Only the peel passes
148 one, and the piece it wants is the last by construction -- the
149 honorific it cut off the end; the surname path passes nothing and
150 the default leaves it exactly as it was."""
151 token = state.tokens[i]
152 base = token.span.start
153 parts: list[WorkToken] = []
154 start = 0
155 for piece in _pieces(token.text, splits):
156 end = start + len(piece)
157 parts.append(dataclasses.replace(
158 token, text=piece, span=Span(base + start, base + end)))
159 start = end
160 if tail_tag is not None:
161 parts[-1] = dataclasses.replace(
162 parts[-1], tags=parts[-1].tags | {tail_tag})
163 added = len(splits)
164 tokens = state.tokens[:i] + tuple(parts) + state.tokens[i + 1:]
165 # Every index the earlier stages recorded is now stale past the
166 # split point: the segment runs group's own iteration rests on, and
167 # the ambiguities from extract_delimited (resolved to indices by
168 # tokenize) and segment. An ambiguity ON the split token keeps
169 # pointing at the head.
170 segments = tuple(_remap(run, i, added) for run in state.segments)
171 ambiguities = tuple(
172 dataclasses.replace(a, indices=tuple(
173 j + added if j > i else j for j in a.indices))
174 for a in state.ambiguities)
175 if detail is not None:
176 ambiguities += (PendingAmbiguity(
177 AmbiguityKind.SEGMENTATION, detail,
178 tuple(range(i, i + added + 1))),)
179 return dataclasses.replace(state, tokens=tokens, segments=segments,
180 ambiguities=ambiguities)
183@functools.lru_cache(maxsize=16)
184def _longest_entry(entries: frozenset[str]) -> int:
185 """The longest entry in a vocabulary, cached per-vocabulary rather
186 than recomputed per parse (Lexicon is frozen and slotted, so it
187 cannot carry a cached_property of its own). The frozenset is
188 hashable and a process holds only a handful of distinct
189 vocabularies -- the default one, plus one per constructed pack
190 parser -- so maxsize=16 bounds pathological many-lexicon churn
191 without ever evicting in normal use. Keyed by the frozenset VALUE,
192 not by (lexicon, field): the two callers pass surnames and
193 honorific_tails, but every lexicon that leaves KOREAN_SURNAMES
194 alone shares one entry, so the count is distinct SETS rather than
195 lexicons times fields.
197 Callers must pass a NON-EMPTY vocabulary: max() of an empty set
198 raises, and both call sites sit under a match guard that cannot
199 reach it with one."""
200 return max(map(len, entries))
203def _is_post_nominal(state: ParseState, i: int) -> bool:
204 """Whether token `i` is post-nominal VOCABULARY. Two of this
205 stage's decisions ask it and neither is about script: an honorific
206 is neither a surname SITE nor the token a glued honorific hangs off
207 (#308).
209 It answers a VOCABULARY question, not a positional one, and the two
210 callers do not spend the answer the same way. In TRAILING position
211 vocabulary and position agree, so the peel site steps over a True
212 and goes on scanning. In LEADING position they disagree -- 양 is
213 the family name there, whatever the suffix set says -- so the
214 surname site reads a True as an ANSWER and declines, rather than as
215 a token to step past.
217 STRICT, not lenient: the initial veto applies, so "V." is not a
218 post-nominal here though bare "v" is a suffix word. That agrees
219 with what classify does with the same token downstream -- "V." is
220 a middle initial -- and the difference is reachable under the
221 default lexicon: "田中さん V." stops its scan-back at the initial
222 and does not peel, while "田中さん II" steps over II and does. That
223 pair is the case table's ja_honorific_glued_before_an_initial and
224 ja_honorific_glued_before_a_roman_suffix, added because the clause
225 could NAME the discriminating input while swapping in
226 is_suffix_lenient still failed no test -- the shape a "unify the
227 two suffix predicates" refactor would have walked straight past.
229 Suffixes only, deliberately. Should a CJK entry ever join titles,
230 the surname site would want it excluded while the peel site must
231 not: a title never trails, so widening the peel's scan-back would
232 move it off the real last name token. Split the predicate then, not
233 before."""
234 return is_suffix_strict(state.tokens[i].text, state.lexicon)
237# rules.md#W2: "A part that is not name text — a post-nominal word
238# standing on its own — is never the name's end: the split-off steps
239# past it to the name word behind, and never dissects it." (history:
240# decisions.md#W2)
241# The scan-back below is that clause, and it needs no punctuation to
242# fire. '김민준 박사님' steps past 박사님 to 김민준, finds no listed
243# tail there and returns None -- so 박사님 is left whole for suffix
244# classification rather than cut into 박사 + 님, which is what the
245# same input gives when the step is removed. '선생님' is post-nominal
246# entire with no name word behind it, so the scan yields no site at
247# all and the token stays whole. Both are contract-tier corpus names
248# and rules.md#W2 example lines; decisions.md#cjk-comma-demotion
249# carries the forced-predicate measurement behind them.
250def _peel_site(state: ParseState, flat: Sequence[int],
251 tails: frozenset[str]) -> tuple[int, int] | None:
252 """Where a peel would land in the token run `flat`: the index of the
253 token to cut and the OFFSET to cut it at (the listed tail, and any
254 full stops riding behind it, are what comes off the end), or None
255 where that run offers no peel.
257 Two callers, one answer, which is the point of naming it. The peel
258 itself asks it once, of the runs it decided to scan. The gate above
259 that decision asks it of segments[0] ALONE, to find out whether
260 declining the second run would cost the only site. Sharing the scan
261 with the peel rather than approximating it there is what makes the
262 gate's answer mean what it says: a gate that merely looked for a
263 token ENDING in a listed tail would count a token that is a tail
264 entire, which the scan-back skips as a site -- "선생님, J.씨" would
265 decline on a site the peel then cannot use, and lose the peel in
266 exactly the way the gate exists to prevent. (Only the lone-token
267 shape of that divergence is reachable from the gate, and the gate's
268 own other conjunct is what makes that so: SUFFIX_COMMA wants BOTH
269 a suffix-shaped second run and more than one word before the
270 comma, and the gate is asked only where the first of those is
271 already true -- so a run this call ever sees under FAMILY_COMMA
272 failed on the second, and segments[0] holds at most one token. The
273 word-count alone would not say that: "Dr 김민준, 지훈" has two
274 words before the comma and is FAMILY_COMMA.)
276 Callers must pass a NON-EMPTY `tails` -- _longest_entry's
277 precondition, which the stage's own early return supplies."""
278 i = next((j for j in reversed(flat)
279 if not _is_post_nominal(state, j)), None)
280 if i is None:
281 # nothing but post-nominals, or no tokens at all
282 return None
283 text = state.tokens[i].text
284 # The tail is matched on the token's core (#323): a stop glued after
285 # the honorific ('김민준씨.', '田中さん.') stands between the listed
286 # tail and the token's end and used to defeat the match. What the
287 # text read instead depended on the stop: the ASCII spelling went to
288 # a title downstream (H2's shape is ASCII-period-only), while a
289 # fullwidth or ideographic stop left the whole text a lone name word
290 # ('田中さん。' read given). The cut lands BEFORE the tail, so the
291 # stops ride with the honorific piece and the token text is never
292 # rewritten. TRAILING only: a leading stop is not between the name
293 # and its honorific, and the offset returned below is the core's
294 # length less the tail's, one subtraction.
295 core = text.rstrip(FULL_STOPS)
296 # range/cap construction identical to the surname match below, and
297 # for the same two reasons: longest-first, and a len-1 cap that
298 # makes the offset interior by construction (_split's contract). An
299 # empty or one-character core caps below 1 and the loop is empty.
300 cap = min(_longest_entry(tails), len(core) - 1)
301 for length in range(cap, 0, -1):
302 if core[-length:] in tails:
303 return i, len(core) - length
304 return None
307# rules.md#W2: "a listed honorific glued to the end of the name's
308# last name word splits off once and reads as a suffix." (history:
309# decisions.md#W2)
310# That the peel also reaches ACROSS a family comma is stated at
311# rules.md#W3 instead, which is a tolerated rule since the
312# 2026-09-01 comma demotion -- the crossing is what the parser does
313# today, not something W2 promises. W3 also carries the reading of a
314# period a listing leaves behind (decisions.md#cjk-comma-demotion,
315# amended by decisions.md#cjk-full-stops): a period on a SEPARATE
316# post-nominal word rides into the suffix and moves nothing ('様.'
317# is post-nominal-strict and the scan steps past it as it steps past
318# '様'), and since #323 a period glued to the honorific's OWN token
319# rides with the honorific too -- _peel_site matches the tail through
320# the trailing stop and cuts before it, so '田中さん.' divides where
321# '田中さん' does, and '김민준씨.' where '김민준씨' does (both are
322# case rows now; before 2026-09-10 they read as a title, measured
323# 2026-09-05 and pinned by nothing, and for one commit of the #323
324# branch the surname site read '김민준씨.' as 김 + 민준씨.). Neither
325# reading is a promise; the step past the post-nominal word itself
326# is W2's, above, and is.
327def _peel_honorific_tail(state: ParseState) -> ParseState:
328 """#308: split a listed honorific off the END of the name's last
329 NON-POST-NOMINAL token -- 田中さん -> 田中 + さん -- and let
330 the existing machinery do the rest. Suffix classification claims
331 the tail downstream (every honorific_tails entry is a suffix word
332 too, enforced by Lexicon), and the segmentation half below then
333 sees the remainder rather than the glued whole, so 김민준씨 splits
334 김 + 민준 and a configured segmenter is handed 山田太郎 rather
335 than 山田太郎様.
337 Scanning back over post-nominals rather than taking the last token
338 outright does three things at once. An unrelated trailing suffix
339 cannot hide the peel site, so "김민준씨 Jr." answers as the
340 comma-written "Dr 김민준씨, Jr." does -- one name, two spellings,
341 one parse. A token that IS a tail (씨, さん, and the nested 선생님,
342 which the cap alone would peel to 선생 + 님) is skipped as a site
343 and stays whole: every tail is a suffix word by the Lexicon
344 invariant, which is what makes that guard hold, and the cap below
345 only keeps the offset interior for _split. And the scan answers
346 None rather than indexing anything, which two reachable inputs
347 need: a name that is nothing but post-nominals ("씨"), and one
348 whose name runs are empty (", , 씨" scopes to two empty ones).
349 Nothing here rests on a structural gate landing first, then, which
350 is what let #312 move both of the stage's gates below it.
352 WHICH tokens are scanned is a separate decision, and the one a
353 reader is likeliest to undo: the NAME's segment runs, flattened,
354 and never state.tokens or every segment. The alternatives agree
355 except where extract_delimited has already claimed a token or a
356 comma has closed the name, both of which this stage can still see
357 -- so "김민준씨 (Jimmy)" and "Dr 김민준씨, V." are the inputs that
358 tell the three apart. See the note at the scan itself.
360 That last token is the last of the NAME, which reaches into a
361 maiden clause: maiden tokens are still main-stream here
362 (extract_delimited has masked only bracketed content), so
363 "김민준 née 박씨" peels 씨 off the MAIDEN name 박씨 and hands it to
364 the person's suffix list -- "née Ms. Park". Intended rather than
365 incidental in that direction: the honorific is the reader's
366 regardless of which of her names it was glued to, and a
367 name-final honorific is exactly what this peels. Since #312 that
368 reach extends to FAMILY_COMMA along with the rest of the crossing
369 -- "김, 민준 née 박씨" now routes 씨 to suffix, where before #312
370 the stage returned at the family-comma gate above the peel and
371 left it in maiden "박씨".
373 The other direction is a LIMIT, stated rather than fixed: a maiden
374 clause pushes the site off the person's own name, so "김민준씨 née
375 박" does NOT peel and gives given "민준씨" -- the original bug,
376 intact behind a marker. "김민준씨 née 박씨" shows both at once, two
377 identical honorifics of which only the maiden's is routed. Chasing
378 the marker into the site scan is scope creep for an uncommon input,
379 and it could not be done here anyway: classify has not run, so the
380 marker tokens carry no tag this stage could read.
382 Longest-first, and ONE peel: a remainder that itself ends in a
383 listed tail is accepted rather than chased. No SHIPPED input
384 witnesses that any more -- 박사님 was the last one, and adding it
385 is what removed the case: 김민준박사님 now gives up 박사님 entire,
386 since longest-first reaches the whole honorific. The pin needs a
387 tail whose remainder ends in a DIFFERENT tail, the two not
388 themselves a listed entry, and no pair in the shipped vocabulary
389 has that shape; the stage test test_one_peel_never_a_stack carries
390 it on a synthetic lexicon, which is now its only witness. No
391 script precondition on the remainder either, since the tail alone
392 is the license: Andersonさん peels.
394 Emits no ambiguity, unlike the surname fork below, though
395 longest-first does CHOOSE here too -- 김선생님 gives 선생님 where
396 님 also matches. The difference is what the runner-up is: a second
397 matching surname is a competing READING of the name, which a
398 caller may prefer, while a shorter tail leaves a remainder that is
399 not a name at all (김선생), so there is nothing to adjudicate."""
400 tails = state.lexicon.honorific_tails
401 if not tails:
402 return state
403 # The NAME's runs, which is more than segments[0] but never every
404 # segment. A comma is how a writer says which runs are the name: a
405 # FAMILY comma splits the name itself across two of them ("김,
406 # 민준씨" is (0,) and (1,)), and the honorific is as often glued to
407 # the given name as to the family, so the peel has to cross that
408 # boundary (#312). Every OTHER structure keeps the whole name in
409 # segments[0], so no boundary has to be crossed to find the site
410 # there -- which is why the asymmetry is the point rather than an
411 # accident of two cases. Note that "the rest is post-nominals" is
412 # true of the SUFFIX comma only: under NO_COMMA segment returns
413 # exactly one run and any trailing post-nominal is INSIDE it
414 # ("김민준씨 Jr." is one run of two tokens), which is why the
415 # scan-back above steps over such a token rather than simply never
416 # reaching it.
417 # The second run is only NAME text when segment read it as one,
418 # which the structure alone does not say: SUFFIX_COMMA also wants
419 # more than one word before the comma, so a one-word part turns a
420 # wholly suffix-shaped remainder into FAMILY_COMMA anyway ("田中さん,
421 # V." is that input). So ask segment's own predicate instead of
422 # inferring the answer from the structure it produced (#319).
423 # is_wholly_suffix, NOT the plural of _is_post_nominal: the two
424 # disagree on the initial-shaped suffix words ("V.", "V", "I"),
425 # which is the class #319 was reported about, and on everything
426 # the run predicate's extra routes reach and the token predicate
427 # does not ("Msc.Ed." and "J.씨" by period_joined_vocab, both of
428 # which this change also moves). Reaching into such a
429 # run put the site on "V.", which ends in no listed tail, so the
430 # peel silently abandoned and さん stayed glued to the family --
431 # while "田中さん, PhD" peeled all along, because "PhD" satisfies
432 # the strict test and the scan-back stepped over it. One credential,
433 # two spellings, two answers FROM THE PEEL -- and the peel's answer
434 # is the only one that moved: the peeled remainder is 田中 and lands
435 # in family under every spelling, while where the CREDENTIAL lands
436 # is assign's question and still differs ("PhD" a title, "V." a
437 # given, "Ph. D." a suffix beside さん).
438 # A junk tail is the worse shape of the same reach: in
439 # "김민준씨, J.씨" the site lands on the junk "J.씨", so master
440 # peeled THAT 씨 and left the person's own glued inside family
441 # "김민준씨". Reachable only where the run is genuinely
442 # suffix-shaped, which "J.씨" is by period_joined_vocab; a run of
443 # ordinary name text is scanned on purpose, and a junk tail further
444 # out than the second run is held off by the scope rule instead
445 # ("Dr 김민준씨, Jr., 박씨" is SUFFIX_COMMA with 박씨 in a third
446 # run, so it never reaches here at all).
447 # An EMPTY second run stays in scope and contributes nothing:
448 # is_wholly_suffix is False on it by its own contract (v1 read
449 # "Doe,, Jr." as a family comma), which is the reading this line
450 # wants anyway -- flattening an empty run adds no site.
451 # Policy(lenient_comma_suffixes=False) keeps the old answer for the
452 # INITIAL-shaped suffixes specifically, which is where the
453 # strict/lenient gap lives: the knob drops this call to the strict
454 # predicate too, so is_wholly_suffix(["V."]) is False, the run reads
455 # as name text, it IS scanned, and the peel is abandoned on "V." as
456 # before -- family "田中さん", given "V.". It is not a blanket
457 # freeze of the old behavior, and "田中さん, Ph. D." is the input
458 # that shows the difference: the Ph./D. merge folds that pair to a
459 # form is_suffix_strict accepts, so the run is declined and the peel
460 # fires under the strict knob as well.
461 # Flattening the SEGMENTS rather than state.tokens is load-bearing
462 # too: extracted nickname and maiden content is in tokens but in NO
463 # segment, and scanning tokens would put the peel site on a
464 # nickname ("김민준씨 (Jimmy)" -> the site becomes Jimmy and
465 # nothing peels). ko_honorific_glued_given_nickname pins that.
466 # And declining takes a SECOND condition: segments[0] must hold a
467 # peel site of its own. is_wholly_suffix reaches period_joined_vocab,
468 # which calls a run suffix-shaped when ANY period-chunk is suffix
469 # VOCABULARY -- and every honorific tail is a suffix word by the
470 # Lexicon invariant, so a glued honorific is itself the evidence.
471 # The predicate is circular at THIS call site alone -- not because
472 # segment asks it any earlier (nothing has peeled at either call)
473 # but because segment SPENDS the answer differently: it reads the
474 # run's shape and stops, the answer being the structure, while the
475 # peel reads the same shape and then decides whether to go strip
476 # the very honorific that produced it. "이, J.씨" reads as wholly suffix
477 # only because of the 씨 the peel exists to remove, and declining a
478 # run that holds the only site does not fall back to some other
479 # site -- it loses the peel outright, and with it the given name,
480 # which lands in suffix as "J.씨". Asking for a site in segments[0]
481 # keeps the #319 answer wherever the peel has somewhere else to go
482 # ("田中さん, V." still declines, さん is right there) and gives the
483 # circular case back to master's reading. The two-honorific input
484 # "김민준씨, J.씨" is where the choice is visible and deliberate:
485 # both runs offer a site, so the decline stands and the person's own
486 # 씨 is peeled rather than the junk one behind the comma.
487 runs = state.segments[:1]
488 if state.structure is Structure.FAMILY_COMMA:
489 second = [state.tokens[j].text for j in state.segments[1]]
490 # a site here is asked about, not used: the offset it carries is
491 # >= 1 by the cap, so a site is always truthy and None never is
492 if not (is_wholly_suffix(second, state.lexicon, state.policy)
493 and _peel_site(state, state.segments[0], tails)):
494 runs = state.segments[:2]
495 site = _peel_site(state, [j for seg in runs for j in seg], tails)
496 if site is None:
497 return state
498 i, offset = site
499 # The tail carries a tag because this stage MANUFACTURED it. The
500 # segmenter's neighbour test below needs to tell it from a token
501 # somebody wrote, and no vocabulary question can: the two spellings
502 # put the same word in the same place, and only the provenance
503 # differs.
504 return _split(state, i, (offset,), None, tail_tag=_PEELED_TAG)
507def _split_surname_site(state: ParseState) -> ParseState:
508 """The stage's other split: the first activated-script token of the
509 name part is matched longest-first against Lexicon.surnames, and a
510 hit splits it in two; where the vocabulary declines, an optional
511 Parser(segmenter=...) is consulted instead.
513 A sibling of _peel_honorific_tail rather than a continuation of it.
514 The two answer different questions -- this one asks where a name
515 divides into surname and given, the peel asks whether a token ends
516 in a word that can never end a name -- which is why they carry
517 different gates: the FAMILY comma and the 间隔号 gate this half
518 alone (#312), and segment_scripts below gates it alone too. That
519 is also why the stage entry below interleaves gates with its two
520 calls rather than running one cascade."""
521 scripts = state.policy.segment_scripts
522 # an empty VOCABULARY deliberately does not bail here -- see below
523 if not scripts:
524 return state
525 # segments[0] is the NAME part under both remaining structures
526 # (everything, under NO_COMMA); later segments are suffixes. Its
527 # members are main-stream token indices by construction, so
528 # extracted nickname/maiden content is unreachable from here. For
529 # this site that is merely tidy -- the first script-written token
530 # is the same either way. It is the PEEL above that reads this run
531 # load-bearingly, and in the two structures that reach here it
532 # reads exactly this one, only backwards; the argument lives at
533 # that scan.
534 i = next((i for i in state.segments[0]
535 if effective_script(state.tokens[i].text) in scripts), None)
536 # A post-nominal in the surname's own position is not a site to
537 # skip past but an answer: a surname LEADS, so if the leading
538 # script-written token is an honorific there is no surname here to
539 # find. Scanning ON would reach the given name, which is exactly
540 # what the first-token rule exists to prevent -- 지 is a listed
541 # surname, so "양 지훈" (양 is a surname AND a shipped honorific)
542 # would have its own given name split in half. Declining also
543 # covers the token the peel above manufactures, which is the first
544 # and only script-written one in "Anderson선생님".
545 if i is None or _is_post_nominal(state, i):
546 return state
547 token = state.tokens[i]
548 text = token.text
549 surnames = state.lexicon.surnames
550 # The vocabulary match reads the token's CORE (#323). Without it
551 # the stop WAS the remainder: '김.' matched its own head and became
552 # 김 + '.'; the stop rides with the remainder instead, so '김민준.'
553 # divides as 김 + '민준.'. A leading stop never reaches THIS site --
554 # the classification fold in front rstrips too
555 # (_vocab._normalized_for_script) and hides such a token before it
556 # arrives -- while the peel above has no such gate in front of it
557 # and its own rstrip is what does the work there.
558 core = text.rstrip(FULL_STOPS)
559 # A token that IS a surname never splits: a bare "남궁" must not
560 # become 남 + 궁 just because the single-syllable surname also
561 # matches -- there is nothing to split off, and a lone token's
562 # role is the order resolution's call, stop or no stop.
563 if core in surnames:
564 return state
565 # Longest-first (compound-before-single falls out of it), capped
566 # so the remainder is never empty. Direct membership, no
567 # _normalize: the script gate admits only CJK text, and a CJK
568 # entry's stored form is its NFC composition with no case or
569 # edge-stop change (#322) -- so an entry AUTHORED in NFD is
570 # composed on the way in, raw NFD input matches nothing here, and
571 # the name goes unsplit, which is the no-split rules.md#W1 already
572 # accepts for NFD text. An empty vocabulary skips the match rather
573 # than bailing the stage (_longest_entry's max() has nothing to
574 # take): a surname-less lexicon declines every token, which is
575 # exactly the condition the segmenter is consulted on, so an early
576 # bail would make a configured segmenter silently inert under
577 # Lexicon.empty() -- the JA pack's own shape.
578 matches: list[int] = []
579 if surnames:
580 cap = min(_longest_entry(surnames), len(core) - 1)
581 matches = [length for length in range(cap, 0, -1)
582 if core[:length] in surnames]
583 if matches:
584 take = matches[0]
585 detail = None
586 if len(matches) > 1:
587 # more than one vocabulary-supported split: longest-first
588 # DECIDED a fork, and the deciding stage records it. A
589 # single-match split chose nothing, so it stays silent --
590 # a dictionary certainty and the statistical guess below
591 # are different epistemic states, and these two emission
592 # rules say so.
593 chosen = " + ".join(map(repr, _pieces(text, (take,))))
594 other = " + ".join(map(repr, _pieces(text, (matches[1],))))
595 detail = (f"{text!r} splits as {chosen} on the longest "
596 f"surname; {other} also reads")
597 return _split(state, i, (take,), detail)
598 if state.segmenter is None:
599 return state # the first activated-script token decides
600 # The segmenter's precondition, which the vocabulary has no twin of:
601 # it is asked where an UNDIVIDED name divides, so it may only be
602 # shown a token that is the whole name. Where the name part carries
603 # a second script-written token the writer already drew this
604 # stage's missing boundary -- "山田 太郎" is divided, and dividing
605 # its family again yields 山 + 田 + 太郎 (namedivider answers for
606 # any string, and scores a two-character one 1.0 by rule, so no
607 # confidence check would catch it). The neighbour counts whatever
608 # script it is written in, ACTIVATED or not -- effective_script is
609 # merely non-None -- because a katakana or hangul neighbour is a
610 # boundary its writer drew just as deliberately as a Han one: under
611 # the JA pack "山田太郎 マイケル" declines, though katakana is in no
612 # activation set. A Latin title or suffix is NOT such a boundary:
613 # it says nothing about where the CJK name splits, so "Dr 阿明日,
614 # Jr." still reaches the segmenter. Vocabulary keeps its own rule
615 # -- a listed surname is a certainty about that exact string,
616 # whoever else stands beside it.
617 # The one neighbour that does not count is the one this stage
618 # MANUFACTURED (#308): a glued 山田太郎様 has no writer-drawn
619 # boundary anywhere, so the 様 the peel just cut off cannot be read
620 # as one -- the precondition must see what the writer wrote, which
621 # was a single undivided token. A SPACED 様 does count, and the
622 # reason is weaker than "its writer drew that boundary and chose
623 # to write 山田太郎 as a unit": in "山田太郎 様" the unit the writer
624 # drew is the WHOLE NAME, honorific and all, so that story is false
625 # for the very input it describes. What the code relies on is that
626 # by POSITION a spaced honorific is indistinguishable from a spaced
627 # name element, so the test conservatively counts it. Measured, the
628 # trade is worth it: counting them keeps 佐藤 氏, 田中 様, 鈴木 先生
629 # and 中村 教授 whole -- all four divide bare under the JA pack (佐
630 # + 藤, 田 + 中, 鈴 + 木, 中 + 村) -- and costs the single division
631 # 山田太郎 様. Four real surnames against one.
632 # Provenance, not vocabulary: the two spellings put the same word
633 # in the same place, and asking the suffix set instead cannot
634 # separate them.
635 if any(j != i and effective_script(state.tokens[j].text) is not None
636 and _PEELED_TAG not in state.tokens[j].tags
637 for j in state.segments[0]):
638 return state
639 # No try/except around the call: rules.md#A1's Accepted clause
640 # ("a user-supplied segmenter's own error propagates"). The two
641 # checks below are that same doctrine, curated,
642 # and they are where the line this module draws is easiest to state:
643 # a PROTOCOL VIOLATION BY THE SEGMENTER AUTHOR RAISES, while an
644 # ADAPTER'S DEFENSE AGAINST ITS LIBRARY DECLINES. Both checks here
645 # are the first kind -- a wrong answer TYPE and an answer indexing
646 # past the token it was handed are stage-detectable bugs in
647 # user-supplied code, inside the declared totality exception, so
648 # they get the same treatment the callable's own exceptions get.
649 # locales/ja.py is the second kind: its repertoire, length,
650 # reconstruction and score guards all return None, because what
651 # they defend against is namedivider answering a question nobody
652 # asked it, which is a fact about the CONTENT, not a broken
653 # protocol. Bounded like every message here: the type's NAME, never
654 # its contents.
655 # The CORE, not the raw token (#323): handed the raw token, a
656 # segmenter answering len-1 would make the stop a piece of its own
657 # ('田中太郎' + '。'), where against the core the trailing stops
658 # ride with the last piece -- '山田太郎.' answered at 2 divides as
659 # 山田 + '太郎.'.
660 answer = state.segmenter(core)
661 if answer is not None and not isinstance(answer, Segmentation):
662 # a duck-typed answer carrying a .splits of its own would
663 # otherwise wander into the split path and surface as a
664 # ValueError naming Token, pointing the reader at nameparser's
665 # insides instead of at their segmenter
666 raise TypeError(
667 f"segmenter must return Segmentation or None, got "
668 f"{type(answer).__name__}")
669 if answer is None or not answer.splits:
670 return state # declined, or confidently one token
671 # splits[-1] is the max -- Segmentation enforced ascending. The
672 # upper bound is the half Segmentation cannot check, since it never
673 # sees the text; an offset at or past the end would make an empty
674 # piece. Declining silently here (as this did before the review)
675 # made an off-by-one segmenter undebuggable: every answer it gave
676 # vanished, and the parse merely looked unsegmented.
677 # The bound is the CORE's length, the string the segmenter was
678 # actually handed: an offset it could not have derived from its own
679 # input is the same author bug whether or not `text` is longer.
680 if answer.splits[-1] >= len(core):
681 raise ValueError(
682 f"segmenter returned splits beyond the token: last offset "
683 f"{answer.splits[-1]}, the segmenter was given "
684 f"{len(core)} characters")
685 conf = answer.confidence
686 detail = None
687 if conf is not None and conf < _SEGMENTER_CONFIDENCE_FLOOR:
688 reading = " + ".join(map(repr, _pieces(text, answer.splits)))
689 detail = (f"{text!r} splits as {reading} on a segmenter answer "
690 f"scoring {conf:.2f}, under the "
691 f"{_SEGMENTER_CONFIDENCE_FLOOR} confidence floor")
692 return _split(state, i, answer.splits, detail)
695# rules.md#W1: "an undivided word in the family position of a name
696# written in an activated script divides after a recognized surname,
697# the longest recognized surname first; where the vocabulary
698# recognizes nothing, an optional segmenter may divide instead"
699# rules.md#W3: "under a family comma the pre-comma text is the
700# family by declaration and never divides, and the post-comma side
701# is given text with no family to find" (history: decisions.md#W3)
702def script_segment(state: ParseState) -> ParseState:
703 if state.original.isascii():
704 # spans index the original exactly (the anti-#100 invariant),
705 # so an ASCII original has only ASCII tokens: nothing here is
706 # in any script's ranges. It also short-circuits the PEEL,
707 # which has no script gate of its own -- so a caller-configured
708 # ASCII tail never fires, and one non-ASCII character anywhere
709 # in the name switches it on. Correct for the CJK vocabulary
710 # that ships, stated because honorific_tails is public: see
711 # that field's own note. Latin orthography SPACES its
712 # post-nominals ("John Smith PhD"), so the glued position this
713 # peel exists for is a CJK one to begin with. What Latin does
714 # glue is PREnominal and period-joined (Mr.Smith, Lt.Gov.),
715 # which _vocab.period_joined_vocab classifies as one token
716 # rather than splitting -- a different mechanism, and no peel
717 # site either way. A glued Latin POST-nominal is spelled the
718 # same way, so it reaches that same mechanism rather than
719 # nothing: period_joined_vocab reads "Smith.Jr." as a title
720 # ('jr' is title vocabulary as well as suffix vocabulary), and
721 # where position allows, that wins -- "Smith.Jr. Anderson"
722 # gives title "Smith.Jr.", family "Anderson". Whatever it
723 # decides, this bail is what settles the question here: it
724 # returns above all of it, so no ASCII input reaches the peel.
725 return state
726 if not state.segments:
727 return state
728 # #312: the peel runs in front of both gates below, because both
729 # answer where a name divides into surname and given, and the peel
730 # does not ask that. It asks whether a token ends in a word that
731 # can never end a name, and a comma or a dot elsewhere in the
732 # string does not change the answer. Placing it above the block
733 # also keeps a future segmentation gate from silently capturing
734 # it: a new gate lands with its siblings, below.
735 state = _peel_honorific_tail(state)
736 if state.structure is Structure.FAMILY_COMMA:
737 return state # the comma already drew the SURNAME boundary
738 if state.interpunct_offsets:
739 # #298: a 间隔号-divided name is a transcription -- its pieces
740 # are syllable groups, not surname+given, so neither the
741 # vocabulary nor the segmenter applies (codepoint-scoped: the
742 # nakaguro records nothing and gates nothing, spec decision 5).
743 # State-global like the FAMILY_COMMA gate above: a marker
744 # anywhere in the name reads the WHOLE name as a transcription
745 # listing, so even an un-dotted hangul token beside a dotted
746 # one stays whole. Scoped to the surname split since #312: an
747 # honorific glued to a transcription is still an honorific.
748 return state
749 return _split_surname_site(state)