Coverage for /pythoncovmergedfiles/medio/medio/usr/local/lib/python3.11/site-packages/nameparser/_pipeline/_script_segment.py: 14%

1"""Stage: script_segment (#271, #272, #308, #312). 

2 

3Consumes: tokens, segments, structure, interpunct_offsets, segmenter, 

4one_case where segment recorded it, and comma_offsets, for the 

5own-words span `_one_case` asks the fact over where segment did not 

6(#613; the answer is not recorded, classify recording its own), and 

7original, for the peel's ASCII bail. 

8Produces: tokens, by four independent splits into sub-slices -- a 

9word of the part before any comma glued to the next at its full stop 

10divided there (rules/T8), with a SUFFIX_OR_NAME report where an 

11initial is cut from the word ending the name, a listed title peeled 

12off the START of the name's first word or the given part's (rules/W6), a 

13listed honorific peeled off the END of the name's last 

14non-post-nominal token, in whichever of the name's runs that falls 

15(the tail token also carries this module's _PEELED_TAG), and the 

16first activated-script token of the name segment split into n+1 

17(into its two surnames and, undivided, the given name behind them, 

18where a Han surname pair stands there, the second surname carrying 

19SURNAME_PAIR_TAG for post_rules, rules/W5) -- segments (index runs remapped past the insertions), ambiguities 

20(indices likewise remapped, plus a SEGMENTATION report when more than 

21one split was vocabulary-supported, or when a segmenter's answer 

22scored under the confidence floor). 

23Reads: the vocabulary `_vocab.glued_cuts` reads for T8 (Lexicon.titles, 

24titles_ambiguous, title_modifiers, title_styles, saint_particles, 

25given_name_abbreviations, suffix_acronyms and its ambiguous set, 

26suffix_words and its ambiguous set, credential_words, and 

27period_joined_vocab's), and Lexicon.suffix_words, suffix_acronyms and 

28titles DIRECTLY (the end-of-name report's tests), Policy.segment_scripts, Lexicon.surnames, Policy.script_orders 

29and Policy.name_order (the order W5's pair needs, rules/W4), 

30Lexicon.honorific_tails, Lexicon.glued_titles and Lexicon.titles (the 

31front peel's words and the titles a site steps past and never cuts), 

32ParseState.segmenter, Lexicon.maiden_markers 

33DIRECTLY (the own-words span `_one_case` takes, _pieces.own_words), 

34everything classify and `_comma.decide` read, through 

35`_postnominal_behind_a_whole_name` (the comma's own decision, asked of 

36a classified copy of the state, #613), and Lexicon suffix 

37vocabulary through TWO predicates, which are NOT one another's 

38singular and plural. _vocab.is_suffix_strict asks whether a single 

39token is a post-nominal, initial veto included (the peel's scan-back 

40and the surname site). _vocab.is_wholly_suffix asks C1's suffix 

41test of a whole RUN -- the one `_comma.decide` and segment's C2 flag 

42ask -- through the POLICY-selected 

43token test plus period_joined_vocab, delimiter handling and the 

44Ph./D. merge -- so the run predicate says yes both to tokens the 

45token predicate VETOES ("V.", "V", "I") and to tokens it never sees 

46as suffixes at all ("Msc.Ed.", "J.씨", which reach it through 

47period_joined_vocab). The initial-shaped words are the class #319 was 

48ABOUT, not the whole of the disagreement, and the extra routes listed 

49above are where the rest of it comes from; 

50test_is_wholly_suffix_is_not_the_plural_of_is_post_nominal pins the 

51"V." half. The run predicate is how the peel declines a run that is 

52not name text (#319), but only where the name's own run offers a 

53site, since a glued honorific is itself part of what makes a run read 

54as suffix-shaped; it owns two further Policy fields 

55(lenient_comma_suffixes picks the strict or lenient token test -- it 

56flips "田中さん, V." -- and extra_suffix_delimiters both counts a bare 

57delimiter-core token as a suffix and splits a token on a core, the 

58split being the half that flips "田中さん, Jr./V." under {"/"} and the 

59bare core the half that flips "田中さん, /"). 

60 

61Implements rules T8 (a word glued at its full stop), W1 (the 

62vocabulary/segmenter division), W2 (the glued-honorific peel), W6 (the 

63glued-title peel), W3 (the 

64writer's divisions are respected) and W5's division (a Han surname 

65pair) of docs/design/rules/, cited at their code below; the decision 

66chain (#308, #312, #319, the vetting bars, the measured 

67spaced-honorific trade) is decisions/W1, #W2 and #W3, and T8's 

68decisions/T8. Every split makes sub-slices of one token, rewriting nothing -- spans still index 

69the original exactly, so the anti-#100 invariant holds by 

70construction. The script splits also match on the token's CORE, its 

71text.rstrip(FULL_STOPS) (#323) -- rstrip and never strip, because 

72both offsets are measured from the text's START, so the core has to 

73stay a prefix of the text for the offset the match yields to land 

74where the match did. The peel runs before the script splits, so suffix classification 

75can claim the tail and the surname match or segmenter consult sees 

76the name rather than name-plus-honorific; its ASCII bail sits above 

77every script split, T8's division alone running ahead of it, since 

78it divides glued words in any cased script, ASCII ones included, so 

79a caller-added LATIN tail 

80fires only on a name carrying at least one non-ASCII character (see 

81the bail's own comment, and honorific_tails' field note). 

82""" 

83from __future__ import annotations 

84 

85import functools 

86import unicodedata 

87from collections.abc import Callable, Sequence 

88 

89from nameparser._lexicon import FULL_STOPS, _normalize 

90import nameparser._pipeline._comma as _comma 

91from nameparser._pipeline._classify import classify 

92from nameparser._pipeline._state import ( 

93 SURNAME_PAIR_TAG, ParseState, PendingAmbiguity, PendingReading, 

94 Structure, WorkToken, copy_with, 

95) 

96from nameparser._pipeline._vocab import ( 

97 GLUED_SHAPE, dotted_title_end, effective_script, 

98 glued_cuts, glued_saint_cut, title_prefixes, 

99 is_initial_chunk, is_one_case, 

100 period_joined_vocab, 

101 is_suffix_strict, is_wholly_suffix, resolve_script_set, 

102) 

103from nameparser._pipeline._pieces import own_words 

104from nameparser._policy import Script 

105from nameparser._types import ( 

106 AmbiguityKind, ReadingKind, ReadingSource, Role, Segmentation, Span, 

107) 

108 

109#: Marks the tail token the peel below MANUFACTURED, so the segmenter's 

110#: neighbour test can tell it from a token somebody wrote. Namespaced, 

111#: therefore unstable provenance rather than API (_types.STABLE_TAGS is 

112#: the whole stable set, and FOLDED_TAG is the precedent for a 

113#: structural marker carrying this prefix); it is vocabulary-derived 

114#: besides, since honorific_tails is what licensed the split. Emitter 

115#: and reader are both in this module, so unlike FOLDED_TAG it needs no 

116#: home in _types. 

117_PEELED_TAG = "vocab:peeled-honorific" 

118 

119#: Segmenter answers scoring below this attach a SEGMENTATION report 

120#: (amendment 2026-07-29 section 3). Kept at the drafted 0.9 after 

121#: measuring namedivider 0.4.1 over 112 names (#272 Task 5), which 

122#: found a distribution the amendment did not anticipate: the scores 

123#: are BIMODAL, not clustered near 1. A rule-based division (the kana 

124#: boundary in 高橋みなみ, or a two-character name) scores exactly 1.0; 

125#: a kanji-statistics division scores a softmax over the candidate cut 

126#: positions, observed in 0.23-0.68 and driven more by name LENGTH 

127#: (median 0.61 at three characters, 0.37 at five -- the softmax is 

128#: over len-1 candidates) than by correctness: the wrong answers in 

129#: the sample scored 0.32/0.60/0.60/0.61, straddling the correct 

130#: median of 0.52. So no floor INSIDE that band separates error from 

131#: success, and the only cut the data supports is between a stated 

132#: certainty and a statistical guess -- which is what section 3's 

133#: epistemic argument asks for anyway. 0.9 sits in the empty gap 

134#: (0.68 to 1.0), far from both modes, and reads as "confident" for a 

135#: third-party segmenter with a calibrated score too. Consequence, 

136#: pinned by tests and stated so it can be checked: every 

137#: STATISTICALLY divided name carries a SEGMENTATION report, and every 

138#: RULE-divided one -- the kana boundary, a two-character name, 

139#: namedivider's specific-name rules -- carries none. 

140#: namedivider scores its own two-character rule 1.0, so this floor 

141#: keeps that division silent -- see locales/ja.py for why the 

142#: presumption is accepted as stated. 

143#: Not configurable in 2.x (YAGNI). 

144_SEGMENTER_CONFIDENCE_FLOOR = 0.9 

145 

146 

147def _remap(run: tuple[int, ...], split_at: int, 

148 added: int) -> tuple[int, ...]: 

149 """One segment run after token `split_at` split into `added` + 1 

150 pieces: the extra pieces join the run immediately after it, and 

151 every later index shifts by as many.""" 

152 out: list[int] = [] 

153 for j in run: 

154 if j == split_at: 

155 out.extend(range(split_at, split_at + added + 1)) 

156 else: 

157 out.append(j + added if j > split_at else j) 

158 return tuple(out) 

159 

160 

161def _pieces(text: str, splits: tuple[int, ...]) -> tuple[str, ...]: 

162 """`text` cut at every offset in `splits`: n offsets, n+1 pieces.""" 

163 cuts = (0, *splits, len(text)) 

164 return tuple(text[a:b] for a, b in zip(cuts, cuts[1:])) 

165 

166 

167def _split(state: ParseState, i: int, splits: tuple[int, ...], 

168 detail: str | None, tail_tag: str | None = None) -> ParseState: 

169 """Cut token `i` at every offset in `splits`, recording `detail` as 

170 a SEGMENTATION report when there is one, and adding `tail_tag` to 

171 the LAST piece when there is one of those. 

172 

173 The offsets arrive non-empty, ascending and interior whatever chose 

174 them. From a segmenter: non-empty is the caller's own check, 

175 strictly ascending with each >= 1 is Segmentation.__post_init__'s, 

176 and the last offset is < len(text) -- checked by the caller. From 

177 the vocabulary: the single offset is >= 1 and < len(text) by the 

178 range(cap, 0, -1) construction and its len-1 cap. From T8's 

179 division: `_vocab.glued_cuts` appends each offset after a matched 

180 stop, so each is >= 2, ascending, and < len(text), a name following. 

181 

182 The ONE split path: the vocabulary hit is the single-offset case, 

183 and the segmenter's answer and T8's division the general one, so 

184 none can drift from the others' index arithmetic. `tail_tag` is part of keeping it 

185 one path: the piece is tagged HERE, where it is built, rather than 

186 by the caller, which would have to re-derive where its own tail 

187 landed after handing the index arithmetic off. Only the peel passes 

188 one, and the piece it wants is the last by construction -- the 

189 honorific it cut off the end; the surname path passes nothing and 

190 the default leaves it exactly as it was.""" 

191 token = state.tokens[i] 

192 base = token.span.start 

193 parts: list[WorkToken] = [] 

194 start = 0 

195 for piece in _pieces(token.text, splits): 

196 end = start + len(piece) 

197 parts.append(copy_with( 

198 token, text=piece, span=Span(base + start, base + end))) 

199 start = end 

200 if tail_tag is not None: 

201 parts[-1] = copy_with( 

202 parts[-1], tags=parts[-1].tags | {tail_tag}) 

203 added = len(splits) 

204 tokens = state.tokens[:i] + tuple(parts) + state.tokens[i + 1:] 

205 # Every index the earlier stages recorded is now stale past the 

206 # split point: the segment runs group's own iteration rests on, and 

207 # the ambiguities from extract_delimited (resolved to indices by 

208 # tokenize) and segment. An ambiguity ON the split token keeps 

209 # pointing at the head. 

210 segments = tuple(_remap(run, i, added) for run in state.segments) 

211 ambiguities = tuple( 

212 copy_with(a, indices=tuple( 

213 j + added if j > i else j for j in a.indices)) 

214 for a in state.ambiguities) 

215 if detail is not None: 

216 ambiguities += (PendingAmbiguity( 

217 AmbiguityKind.SEGMENTATION, detail, 

218 tuple(range(i, i + added + 1))),) 

219 return copy_with(state, tokens=tokens, segments=segments, 

220 ambiguities=ambiguities) 

221 

222 

223@functools.lru_cache(maxsize=16) 

224def _longest_entry(entries: frozenset[str]) -> int: 

225 """The longest entry in a vocabulary, cached per-vocabulary rather 

226 than recomputed per parse (Lexicon is frozen and slotted, so it 

227 cannot carry a cached_property of its own). The frozenset is 

228 hashable and a process holds only a handful of distinct 

229 vocabularies -- the default one, plus one per constructed pack 

230 parser -- so maxsize=16 bounds pathological many-lexicon churn 

231 without ever evicting in normal use. Keyed by the frozenset VALUE, 

232 not by (lexicon, field): the two callers pass surnames and 

233 honorific_tails, but every lexicon that leaves KOREAN_SURNAMES 

234 alone shares one entry, so the count is distinct SETS rather than 

235 lexicons times fields. 

236 

237 Callers must pass a NON-EMPTY vocabulary: max() of an empty set 

238 raises, and both call sites sit under a match guard that cannot 

239 reach it with one.""" 

240 return max(map(len, entries)) 

241 

242 

243def _is_post_nominal(state: ParseState, i: int) -> bool: 

244 """Whether token `i` is post-nominal VOCABULARY. Two of this 

245 stage's decisions ask it and neither is about script: an honorific 

246 is neither a surname SITE nor the token a glued honorific hangs off 

247 (#308). 

248 

249 It answers a VOCABULARY question, not a positional one, and the two 

250 callers do not spend the answer the same way. In TRAILING position 

251 vocabulary and position agree, so the peel site steps over a True 

252 and goes on scanning. In LEADING position they disagree -- 양 is 

253 the family name there, whatever the suffix set says -- so the 

254 surname site reads a True as an ANSWER and declines, rather than as 

255 a token to step past. 

256 

257 STRICT, not lenient: the initial veto applies, so "V." is not a 

258 post-nominal here though bare "v" is a suffix word. That agrees 

259 with what classify does with the same token downstream -- "V." is 

260 a middle initial -- and the difference is reachable under the 

261 default lexicon: "田中さん V." stops its scan-back at the initial 

262 and does not peel, while "田中さん II" steps over II and does. That 

263 pair is the case table's ja_honorific_glued_before_an_initial and 

264 ja_honorific_glued_before_a_roman_suffix, added because the clause 

265 could NAME the discriminating input while swapping in 

266 is_suffix_lenient still failed no test -- the shape a "unify the 

267 two suffix predicates" refactor would have walked straight past. 

268 

269 Suffixes only, deliberately. Should a CJK entry ever join titles, 

270 the surname site would want it excluded while the peel site must 

271 not: a title never trails, so widening the peel's scan-back would 

272 move it off the real last name token. Split the predicate then, not 

273 before.""" 

274 # rules/T4, #727: a bare lowercase letter in a name written in one 

275 # case is vetoed as its capital is; the case fact is asked only of 

276 # such a letter, so a CJK honorific (no case) never asks it 

277 text = state.tokens[i].text 

278 return is_suffix_strict(text, state.lexicon, 

279 len(text) == 1 and text.islower() 

280 and _one_case(state)) 

281 

282 

283# rules/W2: "A part that is not name text — a post-nominal word 

284# standing on its own — is never the name's end: the split-off steps 

285# past it to the name word behind, and never dissects it." (history: 

286# decisions/W2) 

287# The scan-back below is that clause, and it needs no punctuation to 

288# fire. '김민준 박사님' steps past 박사님 to 김민준, finds no listed 

289# tail there and returns None -- so 박사님 is left whole for suffix 

290# classification rather than cut into 박사 + 님, which is what the 

291# same input gives when the step is removed. '선생님' is post-nominal 

292# entire with no name word behind it, so the scan yields no site at 

293# all and the token stays whole. Both are corpus names and 

294# rules/W2 example lines; decisions/cjk-comma-demotion 

295# carries the forced-predicate measurement behind them. 

296def _peel_site(state: ParseState, flat: Sequence[int], 

297 tails: frozenset[str]) -> tuple[int, int] | None: 

298 """Where a peel would land in the token run `flat`: the index of the 

299 token to cut and the OFFSET to cut it at (the listed tail, and any 

300 full stops riding behind it, are what comes off the end), or None 

301 where that run offers no peel. 

302 

303 Two callers, one answer, which is the point of naming it. The peel 

304 itself asks it once, of the runs it decided to scan. The gate above 

305 that decision asks it of segments[0] ALONE, to find out whether 

306 declining the second run would cost the only site. Sharing the scan 

307 with the peel rather than approximating it there is what makes the 

308 gate's answer mean what it says: a gate that merely looked for a 

309 token ENDING in a listed tail would count a token that is a tail 

310 entire, which the scan-back skips as a site -- "선생님, J.씨" would 

311 decline on a site the peel then cannot use, and lose the peel in 

312 exactly the way the gate exists to prevent. (Only the lone-token 

313 shape of that divergence is reachable from the gate, and the gate's 

314 own other conjunct is what makes that so: SUFFIX_COMMA wants BOTH 

315 a suffix-shaped second run and more than one word before the 

316 comma, and the gate is asked only where the first of those is 

317 already true -- so a run this call ever sees under FAMILY_COMMA 

318 failed on the second, and segments[0] holds at most one token. The 

319 word-count alone would not say that: "Dr 김민준, 지훈" has two 

320 words before the comma and is FAMILY_COMMA.) 

321 

322 Callers must pass a NON-EMPTY `tails` -- _longest_entry's 

323 precondition, which the stage's own early return supplies.""" 

324 i = next((j for j in reversed(flat) 

325 if not _is_post_nominal(state, j)), None) 

326 if i is None: 

327 # nothing but post-nominals, or no tokens at all 

328 return None 

329 text = state.tokens[i].text 

330 # The tail is matched on the token's core (#323): a stop glued after 

331 # the honorific ('김민준씨.', '田中さん.') stands between the listed 

332 # tail and the token's end and used to defeat the match. What the 

333 # text read instead depended on the stop: the ASCII spelling went to 

334 # a title downstream (H2's shape is ASCII-period-only), while a 

335 # fullwidth or ideographic stop left the whole text a lone name word 

336 # ('田中さん。' read given). The cut lands BEFORE the tail, so the 

337 # stops ride with the honorific piece and the token text is never 

338 # rewritten. TRAILING only: a leading stop is not between the name 

339 # and its honorific, and the offset returned below is the core's 

340 # length less the tail's, one subtraction. 

341 core = text.rstrip(FULL_STOPS) 

342 # range/cap construction identical to the surname match below, and 

343 # for the same two reasons: longest-first, and a len-1 cap that 

344 # makes the offset interior by construction (_split's contract). An 

345 # empty or one-character core caps below 1 and the loop is empty. 

346 cap = min(_longest_entry(tails), len(core) - 1) 

347 for length in range(cap, 0, -1): 

348 if core[-length:] in tails: 

349 return i, len(core) - length 

350 return None 

351 

352 

353# rules/W2: "a listed honorific glued to the end of the name's 

354# last name word splits off once and reads as a suffix." (history: 

355# decisions/W2) 

356# That the peel also reaches ACROSS a family comma is stated at 

357# rules/W3 instead, which is a tolerated rule since the 

358# 2026-09-01 comma demotion -- the crossing is what the parser does 

359# today, not something W2 promises. W3 also carries the reading of a 

360# period a listing leaves behind (decisions/cjk-comma-demotion, 

361# amended by decisions/cjk-full-stops): a period on a SEPARATE 

362# post-nominal word rides into the suffix and moves nothing ('様.' 

363# is post-nominal-strict and the scan steps past it as it steps past 

364# '様'), and since #323 a period glued to the honorific's OWN token 

365# rides with the honorific too -- _peel_site matches the tail through 

366# the trailing stop and cuts before it, so '田中さん.' divides where 

367# '田中さん' does, and '김민준씨.' where '김민준씨' does (both are 

368# case rows now; before 2026-09-10 they read as a title, measured 

369# 2026-09-05 and pinned by nothing, and for one commit of the #323 

370# branch the surname site read '김민준씨.' as 김 + 민준씨.). Neither 

371# reading is a promise; the step past the post-nominal word itself 

372# is W2's, above, and is. 

373def _one_case(state: ParseState) -> bool: 

374 """ParseState.one_case, asked here where no earlier stage did, as 

375 classify asks it: segment records the fact only where C2's flag 

376 asked for it (#613).""" 

377 if state.one_case is not None: 

378 return state.one_case 

379 return is_one_case(own_words(state.tokens, state.comma_offsets, 

380 state.lexicon._sets.maiden_markers)[0]) 

381 

382 

383def _postnominal_behind_a_whole_name(state: ParseState) -> bool: 

384 """Whether the part after the comma is the postnominal part of a 

385 whole name, so the part before it is a name and not a surname -- 

386 rules/C1's question, which group decides once the words are 

387 tagged (#613). This stage runs first, so it asks THAT decision, on 

388 the name as written: `_comma.decide` over a classified copy of the 

389 state, which is the one implementation of the question. A hand copy 

390 asked the vocabulary here until #613's /simplify pass, and was 

391 patched four times in review for drifting from decide (the ambiguous 

392 class's count, the numeral refusal, a suffix before the comma, the 

393 split 'Ph. D.'), each drift dividing a name decide then read as a 

394 family comma, or leaving whole one it read as a suffix comma. 

395 

396 The copy costs a classify pass and a decision, so the callers ask 

397 only where the answer can change something: the peel once it has 

398 found a site, the division once there is a token it could divide 

399 (Latin and accented-Latin comma names never pay it).""" 

400 return (_comma.decide(classify(state)).structure 

401 is Structure.SUFFIX_COMMA) 

402 

403 

404def _peel_honorific_tail(state: ParseState, 

405 postnominal: Callable[[], bool]) -> ParseState: 

406 """#308: split a listed honorific off the END of the name's last 

407 NON-POST-NOMINAL token -- 田中さん -> 田中 + さん -- and let 

408 the existing machinery do the rest. Suffix classification claims 

409 the tail downstream (every honorific_tails entry is a suffix word 

410 too, enforced by Lexicon), and the segmentation half below then 

411 sees the remainder rather than the glued whole, so 김민준씨 splits 

412 김 + 민준 and a configured segmenter is handed 山田太郎 rather 

413 than 山田太郎様. 

414 

415 Scanning back over post-nominals rather than taking the last token 

416 outright does three things at once. An unrelated trailing suffix 

417 cannot hide the peel site, so "김민준씨 Jr." answers as the 

418 comma-written "Dr 김민준씨, Jr." does -- one name, two spellings, 

419 one parse. A token that IS a tail (씨, さん, and the nested 선생님, 

420 which the cap alone would peel to 선생 + 님) is skipped as a site 

421 and stays whole: every tail is a suffix word by the Lexicon 

422 invariant, which is what makes that guard hold, and the cap below 

423 only keeps the offset interior for _split. And the scan answers 

424 None rather than indexing anything, which two reachable inputs 

425 need: a name that is nothing but post-nominals ("씨"), and one 

426 whose name runs are empty (", , 씨" scopes to two empty ones). 

427 Nothing here rests on a structural gate landing first, then, which 

428 is what let #312 move both of the stage's gates below it. 

429 

430 WHICH tokens are scanned is a separate decision, and the one a 

431 reader is likeliest to undo: the NAME's segment runs, flattened, 

432 and never state.tokens or every segment. The alternatives agree 

433 except where extract_delimited has already claimed a token or a 

434 comma has closed the name, both of which this stage can still see 

435 -- so "김민준씨 (Jimmy)" and "Dr 김민준씨, V." are the inputs that 

436 tell the three apart. See the note at the scan itself. 

437 

438 That last token is the last of the NAME, which reaches into a 

439 maiden clause: maiden tokens are still main-stream here 

440 (extract_delimited has masked only bracketed content), so 

441 "김민준 née 박씨" peels 씨 off the MAIDEN name 박씨 and hands it to 

442 the person's suffix list -- "née Ms. Park". Intended rather than 

443 incidental in that direction: the honorific is the reader's 

444 regardless of which of her names it was glued to, and a 

445 name-final honorific is exactly what this peels. Since #312 that 

446 reach extends to FAMILY_COMMA along with the rest of the crossing 

447 -- "김, 민준 née 박씨" now routes 씨 to suffix, where before #312 

448 the stage returned at the family-comma gate above the peel and 

449 left it in maiden "박씨". 

450 

451 The other direction is a LIMIT, stated rather than fixed: a maiden 

452 clause pushes the site off the person's own name, so "김민준씨 née 

453 박" does NOT peel and gives given "민준씨" -- the original bug, 

454 intact behind a marker. "김민준씨 née 박씨" shows both at once, two 

455 identical honorifics of which only the maiden's is routed. Chasing 

456 the marker into the site scan is scope creep for an uncommon input, 

457 and it could not be done here anyway: classify has not run, so the 

458 marker tokens carry no tag this stage could read. 

459 

460 Longest-first, and ONE peel: a remainder that itself ends in a 

461 listed tail is accepted rather than chased. No SHIPPED input 

462 witnesses that any more -- 박사님 was the last one, and adding it 

463 is what removed the case: 김민준박사님 now gives up 박사님 entire, 

464 since longest-first reaches the whole honorific. The pin needs a 

465 tail whose remainder ends in a DIFFERENT tail, the two not 

466 themselves a listed entry, and no pair in the shipped vocabulary 

467 has that shape; the stage test test_one_peel_never_a_stack carries 

468 it on a synthetic lexicon, which is now its only witness. No 

469 script precondition on the remainder either, since the tail alone 

470 is the license: Andersonさん peels. 

471 

472 Emits no ambiguity, unlike the surname fork below, though 

473 longest-first does CHOOSE here too -- 김선생님 gives 선생님 where 

474 님 also matches. The difference is what the runner-up is: a second 

475 matching surname is a competing READING of the name, which a 

476 caller may prefer, while a shorter tail leaves a remainder that is 

477 not a name at all (김선생), so there is nothing to adjudicate.""" 

478 tails = state.lexicon._sets.honorific_tails 

479 if not tails: 

480 return state 

481 # The NAME's runs: more than segments[0], never every segment. A 

482 # FAMILY comma splits the name across two runs and the honorific 

483 # glues to either side (#312); every other structure keeps the 

484 # name whole in segments[0], a trailing post-nominal under 

485 # NO_COMMA included, which the scan-back steps over. Flattening 

486 # SEGMENTS rather than state.tokens keeps the site off extracted 

487 # nickname and maiden content. 

488 # 

489 # rules/W3: "A part that is nothing but suffix words is not the 

490 # name's end: it is declined as the site, and the split-off falls 

491 # back to the part before the comma and takes the honorific there, 

492 # provided that part offers a site of its own." 

493 # The structure does not say whether segments[1] is name text, so 

494 # the run predicate is asked (`is_wholly_suffix`, not the plural of 

495 # `_is_post_nominal`, which vetoes an initial-shaped 'V.' and never 

496 # sees period_joined_vocab's 'J.씨'), with the case fact and policy 

497 # `_comma.decide` reads it with; an empty run is in 

498 # scope and adds no site. The second condition is there because 

499 # the predicate is circular at this site alone: a glued honorific 

500 # is itself suffix evidence. decisions/W2 (#319). 

501 site = _peel_site(state, state.segments[0], tails) 

502 if state.structure is Structure.FAMILY_COMMA: 

503 # Under a family comma both runs may be scanned. The comma's 

504 # decision is asked only where it can matter -- where scanning 

505 # one run and scanning both find different sites (#613's 

506 # /simplify: it is a classify pass and a decision) 

507 both = _peel_site(state, [j for seg in state.segments[:2] 

508 for j in seg], tails) 

509 if both != site and not postnominal(): 

510 second = [state.tokens[j].text for j in state.segments[1]] 

511 # `one_case` is passed for the reason the field exists: the 

512 # credential lean is part of what "wholly suffix" MEANS since 

513 # #289, and a stage that asks the question without it gets a 

514 # different answer from `_comma.decide`, which reads the same 

515 # run with the fact in hand at group's head. Left out, 'Kim김민준씨, MA' read 

516 # 'MA' as name material, declined nothing, and scanned the 

517 # second run -- where the site is 'MA' itself, which carries no 

518 # listed tail -- so the person's own 씨 went unpeeled (family 

519 # 'Kim김민준씨') while 'Kim김민준씨, PhD' peeled it, one name in 

520 # two spellings parsed two ways (review round, #289/#516). 

521 # `segment` records the fact only where C2's flag asks for it, 

522 # so `_one_case` asks it here for every other name (#613). 

523 # the site test first: it is the cheaper half, and false 

524 # for every name whose part before the comma holds no glued 

525 # honorific, which spares those the case fact 

526 # one fact, asked for the lean and for the strict test's 

527 # initial veto alike (rules/T4, #727) 

528 if not (site is not None 

529 and is_wholly_suffix(second, state.lexicon, 

530 state.policy, 

531 one_case=(case := _one_case(state)), 

532 letters_one_case=case)): 

533 site = both 

534 if site is None: 

535 return state 

536 i, offset = site 

537 # The tail carries a tag because this stage MANUFACTURED it. The 

538 # segmenter's neighbour test below needs to tell it from a token 

539 # somebody wrote, and no vocabulary question can: the two spellings 

540 # put the same word in the same place, and only the provenance 

541 # differs. 

542 return _split(state, i, (offset,), None, tail_tag=_PEELED_TAG) 

543 

544 

545#: The Thai vowels written after their consonant (U+0E30 to U+0E3A, 

546#: U+0E45), which no word opens with: where one follows a title, the 

547#: title's last letters belong to the word, so the peel falls back to a 

548#: shorter title ('นางสาวิตรี' is นาง + สาวิตรี) or keeps the word whole 

549#: ('นางิ', Nagi; 'นางาเซะ', Nagase), as it does behind a combining mark. 

550_AFTER_A_CONSONANT = frozenset(map(chr, (*range(0x0E30, 0x0E3B), 0x0E45))) 

551 

552 

553def _head_sites(state: ParseState) -> list[int]: 

554 """The tokens a glued title may open: the first word of the name and, 

555 after a family comma, the first word of the given part, each past 

556 any listed title written apart in front of it, alone or joined at its 

557 full stops ('รองศาสตราจารย์ นายแพทย์สมชาย', 'รศ.ดร. นายสมชาย').""" 

558 titles = state.lexicon._sets.titles 

559 runs = (state.segments[:2] if state.structure is Structure.FAMILY_COMMA 

560 else state.segments[:1]) 

561 sites = [] 

562 for run in runs: 

563 j = next((j for j in run 

564 if not _all_titles(state.tokens[j].text, titles)), None) 

565 if j is not None: 

566 sites.append(j) 

567 return sites 

568 

569 

570def _all_titles(text: str, titles: frozenset[str]) -> bool: 

571 """Whether `text` is a listed title, whole or chunk by chunk between 

572 its full stops ('รองศาสตราจารย์', 'พ.อ.หญิง', 'รศ.ดร.'): a title 

573 written apart, which the site walk passes over; a word holding a 

574 name ('ดร.สมชาย') is a site, never passed over.""" 

575 n = _normalize(text) 

576 if n in titles: 

577 return True 

578 chunks = [c for c in n.split(".") if c] 

579 return bool(chunks) and all(c in titles for c in chunks) 

580 

581 

582# rules/W6: "a title the vocabulary marks to peel, glued to the start of 

583# the name's first word, splits off once and reads as a title" (history: 

584# decisions/W6 (#317)) 

585def _peel_title_head(state: ParseState) -> ParseState: 

586 """#317: split a listed title off the START of the name's first word 

587 -- นายสมชาย -> นาย + สมชาย -- the mirror of `_peel_honorific_tail`, 

588 licensed the same way: only a title written `attached` peels 

589 (`Lexicon.glued_titles`), the vocabulary carrying its own license. 

590 Longest first, ONE peel, and never off a word the vocabulary lists 

591 whole (`นายแพทย์`, which opens with `นาย`), nor where what is left 

592 opens with no letter, a Thai vowel written after its consonant 

593 (`_AFTER_A_CONSONANT`) taking the next shorter title instead. The head 

594 piece needs no tag: classify reads it as the listed title it is. 

595 

596 T8's division ran on the word whole, before this peel, and found no 

597 chunk at its head (`อาจารย์ดร.สมชาย`), so a remainder still glued at 

598 a full stop (`ดร.สมชาย`) is divided here, as T8 divides the same 

599 word written apart; behind a family comma it stays whole, as T8 

600 leaves every word after a comma.""" 

601 sets = state.lexicon._sets 

602 divide = False 

603 for i in reversed(_head_sites(state)): 

604 text = state.tokens[i].text 

605 low = text.lower() 

606 if len(low) != len(text): 

607 # a letter lowered into two: the offsets of `low` would not 

608 # be the text's 

609 continue 

610 cut, w = _peel_cut(low, state) 

611 if not cut: 

612 continue 

613 divide = divide or bool(state.segments and i in state.segments[0] 

614 and GLUED_SHAPE.search(text, cut)) 

615 state = _split(state, i, (cut,), None) 

616 if w in sets.title_modifiers: 

617 # a title only in a phrase peeled with its head behind it: 

618 # the head peels too, as written apart ('ว่าที่จ่าสิบตรีสมชาย', 

619 # 'ใจดี, ว่าที่ร.ท.วินัย') 

620 second, _ = _peel_cut(low[cut:], state) 

621 if second: 

622 state = _split(state, i + 1, (second,), None) 

623 return _divide_glued(state) if divide else state 

624 

625 

626def _peel_cut(low: str, state: ParseState) -> tuple[int, str]: 

627 """Where rules/W6's peel cuts the lower-cased word `low`, and the 

628 title it takes, or (0, ''). 

629 

630 Longest first; a title that would leave a Thai vowel or a mark at 

631 the opening falls back to the next shorter one ('นางสาวิตรี' is นาง + 

632 สาวิตรี), while one followed by anything else that opens no word peels 

633 nothing: a hyphen, a digit, the repetition mark ('นางสาว-สุดา', 

634 'นางสาวๆ').""" 

635 sets = state.lexicon._sets 

636 for w in _longest_first(sets.glued_titles): 

637 if len(w) >= len(low) or not low.startswith(w): 

638 continue 

639 end = len(w) 

640 if "." in w: 

641 # a title written short ends where T8 reads it end: past its 

642 # own stop, or behind a last chunk of two or more letters 

643 # (#1319: 'พล.ต.ต.สุรศักดิ์', 'พ.อ.หญิงวาสนา', never 'พ.ต.อ' 

644 # in 'พ.ต.อนุชา') 

645 e = dotted_title_end(low, 0, (w,)) 

646 if e is None or e >= len(low): 

647 continue 

648 end = e 

649 elif low[end] == "." and end + 1 < len(low): 

650 end += 1 

651 if w in sets.title_modifiers and not low.startswith( 

652 title_prefixes(sets.titles), end): 

653 # a title only in a phrase (rules/H9) peels only with a 

654 # listed title behind it ('ว่าที่ร.ท.วินัย', 'ว่าที่ดร.สมชาย') 

655 continue 

656 nxt = low[end] 

657 if nxt in _AFTER_A_CONSONANT or unicodedata.category(nxt) == "Mn": 

658 continue 

659 return (end, w) if nxt.isalpha() and nxt != "\u0e46" else (0, "") 

660 return 0, "" 

661 

662 

663@functools.lru_cache(maxsize=16) 

664def _longest_first(heads: frozenset[str]) -> tuple[str, ...]: 

665 """The glued titles longest first, sorted once per vocabulary.""" 

666 return tuple(sorted(heads, key=len, reverse=True)) 

667 

668 

669def _split_candidate(state: ParseState) -> int | None: 

670 """The token `_split_surname_site` would try to divide, or None: 

671 asked on its own so a caller can learn cheaply that there is 

672 nothing to divide before asking anything costlier (#613's 

673 /simplify).""" 

674 scripts = state.policy.segment_scripts 

675 # an empty VOCABULARY deliberately does not bail here -- see below 

676 if not scripts: 

677 return None 

678 # segments[0] is the NAME part under both remaining structures 

679 # (everything, under NO_COMMA); later segments are suffixes. Its 

680 # members are main-stream token indices by construction, so 

681 # extracted nickname/maiden content is unreachable from here. For 

682 # this site that is merely tidy -- the first script-written token 

683 # is the same either way. It is the PEEL above that reads this run 

684 # load-bearingly, and in the two structures that reach here it 

685 # reads exactly this one, only backwards; the argument lives at 

686 # that scan. 

687 i = next((i for i in state.segments[0] 

688 if effective_script(state.tokens[i].text) in scripts), None) 

689 # A post-nominal in the surname's own position is not a site to 

690 # skip past but an answer: a surname LEADS, so if the leading 

691 # script-written token is an honorific there is no surname here to 

692 # find. Scanning ON would reach the given name, which is exactly 

693 # what the first-token rule exists to prevent -- 지 is a listed 

694 # surname, so "양 지훈" (양 is a surname AND a shipped honorific) 

695 # would have its own given name split in half. Declining also 

696 # covers the token the peel above manufactures, which is the first 

697 # and only script-written one in "Anderson선생님". 

698 if i is None or _is_post_nominal(state, i): 

699 return None 

700 # A token that IS a surname never splits: a bare "남궁" must not 

701 # become 남 + 궁 just because the single-syllable surname also 

702 # matches -- there is nothing to split off, and a lone token's 

703 # role is the order resolution's call, stop or no stop. Asked here, 

704 # with the core the split reads (#323), so the comma gate in front 

705 # learns it before asking the comma's decision (#613's /simplify) 

706 if state.tokens[i].text.rstrip(FULL_STOPS) in state.lexicon._sets.surnames: 

707 return None 

708 return i 

709 

710 

711def _surname_pair(state: ParseState, i: int, core: str, 

712 surnames: frozenset[str]) -> tuple[int, ...] | None: 

713 """rules/W5: "whose first two are each a listed single surname" -- 

714 the offsets at which `_split` divides token `i` into its two 

715 surnames and, written undivided, the given name behind them: (1, 2) 

716 for four characters, (1,) for two with the given name after the 

717 space; or None where it holds no second single surname the family 

718 can take, by the length, script and order tests below (#845). Token 

719 `i`'s longest listed surname is its first character, so no listed 

720 compound opens it. 

721 

722 Four characters written undivided, or two with a word of two 

723 written after the space: a Han given name is nearly always one or 

724 two characters, and a three-character name is nearly always a 

725 single surname before a two-character given name (李林伟, and 

726 `李林 伟` written with a stray space), the reason no married pair is 

727 listed as a compound. 

728 

729 Only where the name will be read family-first, since only there is 

730 the first surname the family for the second to join: every word of 

731 the name segment but the post-nominals ending it written in the 

732 scripts assign's order resolution keys (rules/W4), resolving to an 

733 entry, or failing one, to a declared order, that puts the family 

734 first -- the resolution assign makes over its name pieces, asked of 

735 the words before titles are known. A word in another 

736 script, a Latin title included, keeps W1's division, which reads 

737 the same under any order, so the stage never cuts a name the order 

738 then reads otherwise. History: decisions/W5 (#845).""" 

739 # the length tests first: they are C-level, and refuse a 

740 # three-character token, the usual single surname and given name, 

741 # before the script walk below 

742 if core[1] not in surnames or len(core) not in (2, 4): 

743 return None 

744 run = state.segments[0] 

745 at = run.index(i) 

746 if len(core) == 2 and ( 

747 at + 1 == len(run) 

748 or len(state.tokens[run[at + 1]].text.rstrip(FULL_STOPS)) != 2): 

749 return None 

750 # post-nominals ending the name, a peeled honorific among them, are 

751 # no name pieces of assign's, so they key no order (王李小明さん, 

752 # 王李小明 PhD); anything before them does 

753 end = len(run) 

754 while (end > at + 1 

755 and (_PEELED_TAG in state.tokens[run[end - 1]].tags 

756 or _is_post_nominal(state, run[end - 1]))): 

757 end -= 1 

758 found: set[Script] = set() 

759 for j in run[:end]: 

760 script = effective_script(state.tokens[j].text) 

761 if script is None: 

762 return None 

763 found.add(script) 

764 if resolve_script_set(found) is not Script.HAN: 

765 return None 

766 order = next((o for s, o in state.policy.script_orders 

767 if s is Script.HAN), state.policy.name_order) 

768 if order[0] is Role.GIVEN: 

769 return None 

770 if len(core) == 4: 

771 return (1, 2) 

772 return None if _is_post_nominal(state, run[at + 1]) else (1,) 

773 

774 

775def _split_surname_pair(state: ParseState, i: int, 

776 splits: tuple[int, ...]) -> ParseState: 

777 """Token `i` divided at `_surname_pair`'s cuts into its two surnames 

778 and, unspaced, the given name behind them, the second surname 

779 tagged for post_rules to read as the family's second unit. No 

780 report here: whether the pair is read is post_rules' decision, 

781 under the order the name is read in, so it reports there 

782 (mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE).""" 

783 state = _split(state, i, splits, None) 

784 second = state.tokens[i + 1] 

785 tokens = list(state.tokens) 

786 tokens[i + 1] = copy_with(second, tags=second.tags | {SURNAME_PAIR_TAG}) 

787 return copy_with(state, tokens=tuple(tokens)) 

788 

789 

790def _split_surname_site(state: ParseState) -> ParseState: 

791 """The stage's other split: the first activated-script token of the 

792 name part is matched longest-first against Lexicon.surnames, and a 

793 hit splits it in two; where the vocabulary declines, an optional 

794 Parser(segmenter=...) is consulted instead. 

795 

796 A sibling of _peel_honorific_tail rather than a continuation of it. 

797 The two answer different questions -- this one asks where a name 

798 divides into surname and given, the peel asks whether a token ends 

799 in a word that can never end a name -- which is why they carry 

800 different gates: the FAMILY comma and the 间隔号 gate this half 

801 alone (#312), and segment_scripts below gates it alone too. That 

802 is also why the stage entry below interleaves gates with its two 

803 calls rather than running one cascade.""" 

804 i = _split_candidate(state) 

805 if i is None: 

806 return state 

807 token = state.tokens[i] 

808 text = token.text 

809 surnames = state.lexicon._sets.surnames 

810 # The vocabulary match reads the token's CORE (#323). Without it 

811 # the stop WAS the remainder: '김.' matched its own head and became 

812 # 김 + '.'; the stop rides with the remainder instead, so '김민준.' 

813 # divides as 김 + '민준.'. A leading stop never reaches THIS site -- 

814 # the classification fold in front rstrips too 

815 # (_vocab._normalized_for_script) and hides such a token before it 

816 # arrives -- while the peel above has no such gate in front of it 

817 # and its own rstrip is what does the work there. 

818 core = text.rstrip(FULL_STOPS) 

819 # Longest-first (compound-before-single falls out of it), capped 

820 # so the remainder is never empty. Direct membership, no 

821 # _normalize: the script gate admits only CJK text, and a CJK 

822 # entry's stored form is its NFC composition with no case or 

823 # edge-stop change (#322) -- so an entry AUTHORED in NFD is 

824 # composed on the way in, raw NFD input matches nothing here, and 

825 # the name goes unsplit, the no-split NFD degradation that 

826 # decisions/W1's 2026-07-29 NFD bullet (ja amendment 1a) accepts. An empty vocabulary skips the match rather 

827 # than bailing the stage (_longest_entry's max() has nothing to 

828 # take): a surname-less lexicon declines every token, which is 

829 # exactly the condition the segmenter is consulted on, so an early 

830 # bail would make a configured segmenter silently inert under 

831 # Lexicon.empty() -- the JA pack's own shape. 

832 matches: list[int] = [] 

833 if surnames: 

834 cap = min(_longest_entry(surnames), len(core) - 1) 

835 matches = [length for length in range(cap, 0, -1) 

836 if core[:length] in surnames] 

837 if matches: 

838 take = matches[0] 

839 if take == 1: 

840 pair = _surname_pair(state, i, core, surnames) 

841 if pair is not None: 

842 return _split_surname_pair(state, i, pair) 

843 detail = None 

844 if len(matches) > 1: 

845 # more than one vocabulary-supported split: longest-first 

846 # DECIDED a fork, and the deciding stage records it. A 

847 # single-match split chose nothing, so it stays silent -- 

848 # a dictionary certainty and the statistical guess below 

849 # are different epistemic states, and these two emission 

850 # rules say so. 

851 chosen = " + ".join(map(repr, _pieces(text, (take,)))) 

852 other = " + ".join(map(repr, _pieces(text, (matches[1],)))) 

853 detail = (f"{text!r} splits as {chosen} on the longest " 

854 f"surname; {other} also reads") 

855 return _split(state, i, (take,), detail) 

856 if state.segmenter is None: 

857 return state # the first activated-script token decides 

858 # The segmenter's precondition, which the vocabulary has no twin of: 

859 # it is asked where an UNDIVIDED name divides, so it may only be 

860 # shown a token that is the whole name. Where the name part carries 

861 # a second script-written token the writer already drew this 

862 # stage's missing boundary -- "山田 太郎" is divided, and dividing 

863 # its family again yields 山 + 田 + 太郎 (namedivider answers for 

864 # any string, and scores a two-character one 1.0 by rule, so no 

865 # confidence check would catch it). The neighbour counts whatever 

866 # script it is written in, ACTIVATED or not -- effective_script is 

867 # merely non-None -- because a katakana or hangul neighbour is a 

868 # boundary its writer drew just as deliberately as a Han one: under 

869 # the JA pack "山田太郎 マイケル" declines, though katakana is in no 

870 # activation set. A Latin title or suffix is NOT such a boundary: 

871 # it says nothing about where the CJK name splits, so "Dr 小明日, 

872 # Jr." still reaches the segmenter. Vocabulary keeps its own rule 

873 # -- a listed surname is a certainty about that exact string, 

874 # whoever else stands beside it. 

875 # The one neighbour that does not count is the one this stage 

876 # MANUFACTURED (#308): a glued 山田太郎様 has no writer-drawn 

877 # boundary anywhere, so the 様 the peel just cut off cannot be read 

878 # as one -- the precondition must see what the writer wrote, which 

879 # was a single undivided token. A SPACED 様 does count, and the 

880 # reason is weaker than "its writer drew that boundary and chose 

881 # to write 山田太郎 as a unit": in "山田太郎 様" the unit the writer 

882 # drew is the WHOLE NAME, honorific and all, so that story is false 

883 # for the very input it describes. What the code relies on is that 

884 # by POSITION a spaced honorific is indistinguishable from a spaced 

885 # name element, so the test conservatively counts it. Measured, the 

886 # trade is worth it: counting them keeps 佐藤 氏, 田中 様, 鈴木 先生 

887 # and 中村 教授 whole -- all four divide bare under the JA pack (佐 

888 # + 藤, 田 + 中, 鈴 + 木, 中 + 村) -- and costs the single division 

889 # 山田太郎 様. Four real surnames against one. 

890 # Provenance, not vocabulary: the two spellings put the same word 

891 # in the same place, and asking the suffix set instead cannot 

892 # separate them. 

893 if any(j != i and effective_script(state.tokens[j].text) is not None 

894 and _PEELED_TAG not in state.tokens[j].tags 

895 for j in state.segments[0]): 

896 return state 

897 # No try/except around the call: rules/A1's Accepted clause 

898 # ("a user-supplied segmenter's own error, which propagates"). The 

899 # two checks below are that same doctrine, curated, and they are 

900 # where the line this module draws is easiest to state: 

901 # a PROTOCOL VIOLATION BY THE SEGMENTER AUTHOR RAISES, while an 

902 # ADAPTER'S DEFENSE AGAINST ITS LIBRARY DECLINES. Both checks here 

903 # are the first kind -- a wrong answer TYPE and an answer indexing 

904 # past the token it was handed are stage-detectable bugs in 

905 # user-supplied code, inside the declared totality exception, so 

906 # they get the same treatment the callable's own exceptions get. 

907 # locales/ja.py is the second kind: its repertoire, length, 

908 # reconstruction and score guards all return None, because what 

909 # they defend against is namedivider answering a question nobody 

910 # asked it, which is a fact about the CONTENT, not a broken 

911 # protocol. Bounded like every message here: the type's NAME, never 

912 # its contents. 

913 # The CORE, not the raw token (#323): handed the raw token, a 

914 # segmenter answering len-1 would make the stop a piece of its own 

915 # ('田中太郎' + '。'), where against the core the trailing stops 

916 # ride with the last piece -- '山田太郎.' answered at 2 divides as 

917 # 山田 + '太郎.'. 

918 answer = state.segmenter(core) 

919 if answer is not None and not isinstance(answer, Segmentation): 

920 # a duck-typed answer carrying a .splits of its own would 

921 # otherwise wander into the split path and surface as a 

922 # ValueError naming Token, pointing the reader at nameparser's 

923 # insides instead of at their segmenter 

924 raise TypeError( 

925 f"segmenter must return Segmentation or None, got " 

926 f"{type(answer).__name__}") 

927 if answer is None or not answer.splits: 

928 return state # declined, or confidently one token 

929 # splits[-1] is the max -- Segmentation enforced ascending. The 

930 # upper bound is the half Segmentation cannot check, since it never 

931 # sees the text; an offset at or past the end would make an empty 

932 # piece. Declining silently here (as this did before the review) 

933 # made an off-by-one segmenter undebuggable: every answer it gave 

934 # vanished, and the parse merely looked unsegmented. 

935 # The bound is the CORE's length, the string the segmenter was 

936 # actually handed: an offset it could not have derived from its own 

937 # input is the same author bug whether or not `text` is longer. 

938 if answer.splits[-1] >= len(core): 

939 raise ValueError( 

940 f"segmenter returned splits beyond the token: last offset " 

941 f"{answer.splits[-1]}, the segmenter was given " 

942 f"{len(core)} characters") 

943 conf = answer.confidence 

944 detail = None 

945 if conf is not None and conf < _SEGMENTER_CONFIDENCE_FLOOR: 

946 reading = " + ".join(map(repr, _pieces(text, answer.splits))) 

947 detail = (f"{text!r} splits as {reading} on a segmenter answer " 

948 f"scoring {conf:.2f}, under the " 

949 f"{_SEGMENTER_CONFIDENCE_FLOOR} confidence floor") 

950 return _split(state, i, answer.splits, detail) 

951 

952 

953# rules/T8: "the word reads as it does with a space there" -- the part 

954# before any comma, where the division acts (decisions/T8 (#1230, #1235)) 

955def _divide_glued(state: ParseState) -> ParseState: 

956 """Every word of the part before the first comma divided where 

957 `_vocab.glued_cuts` finds a chunk read alone meeting a name, last 

958 word first, so the indices still to visit do not move.""" 

959 head = state.segments[0] 

960 lexicon = state.lexicon 

961 # the words as written, before any of them divides: the report reads 

962 # the words after its own, which may have divided already 

963 texts = [state.tokens[j].text for j in head] 

964 for k in range(len(head) - 1, -1, -1): 

965 i = head[k]