Shortcuts on this page
r m x toggle line displays
j k next/prev highlighted chunk
0 (zero) top of page
1 (one) first highlighted chunk
1"""Stage: assign.
3Consumes: pieces + piece_tags (grouped), segments, structure, tokens,
4one_case, and tail_reads -- rules/S2's trailing run as group read it
5once, after P3's joins and before the particle chain (#614), which the
6main segment's placement takes rather than peeling again, save where
7group's joins left no name word in front of it; and interpunct_offsets,
8whose 间隔号 keeps a transcription in its declared order (rules/W4).
9Produces: tokens with roles set on every main-stream token, and the
10folded-middle and attached tags on the particles P6's attachment joins
11to the family after a family comma (#613; the attached tag, 3.0's 4c,
12tells rules/R9 the attachment from O3's fold); and order, the order
13the positional read used, None where none happened (ParseState.order).
14Reads: Policy.name_order (#270), Policy.lone_name_word (rules/O5,
15the field a lone name word takes, #1111), Policy.initials_as_family
16(rules/O4's order report stands down under it, #1254), is_suffix_lenient on the trailing
17piece of a two-part comma name, and Policy.script_orders (#271, which
18overrides it when every name piece is written wholly in one script, or
19in the Han/Hiragana/Katakana repertoire the #272 kana license shares
20across pieces); token/piece tags; Lexicon only through tags already
21applied by classify (plus the leading-title period rule), and
22Lexicon.given_name_titles, which an ambiguity's alternative asks for
23rules/P5's given-name-title licence; and Lexicon.title_phrases, titles
24and titles_ambiguous through _vocab.in_a_title_phrase, for rules/H9's
25TITLE_OR_NAME report on a contested word opening the name.
27Implements rules H2, H4, H5, H6, H7, H8, N3, O4, O5, S10, W4, and P6 after a family comma,
28of docs/design/rules/,
29each cited at its code below. Ports v1's assignment loops.
30NO_COMMA (per name_order):
31leading title pieces chain while no given-position name has been seen
32(a title needs a following piece, unless the whole name is one title);
33then positional assignment per name_order with the trailing-suffix
34rule: the piece from which everything after is a strict suffix is the
35last name-position piece, the rest are suffixes. Behind that peel a
36trailing run of the title words H5's chain takes
37(`_pieces.trailing_titles`) chains into the title from
38the end, leaving one name piece standing; where the run TAKES
39something the peel was only provisional and runs again over the pieces
40with the titled ones spliced out, the two alternating until the run
41takes nothing, so a trailing title is transparent to the suffix
42reading however many titles are written (_pieces.tail_reading). Where
43the run takes nothing -- almost every name -- the first peel is the
44only one and its answer stands.
45The v1 single-name+nickname rule lives here (decisions/N3): a
46nonempty nickname beside exactly one piece in total puts that piece
47in FAMILY.
48FAMILY_COMMA: segment 0 FAMILY but for a leading run of listed titles
49(rules/H6, #701; v1 kept them in the family) and the post-nominals
50behind its first family piece (SUFFIX, 'Smith Jr., John'). A part after the
51comma that holds no name word, or that a credential opens (#603), is
52not read here: group's head binds its SUFFIX and TITLE roles before
53any join (`_comma.decide`, #613), making the comma a suffix comma
54behind a whole name ('John Smith, Dr.', 'John Smith, PhD Jones') and
55leaving the family comma behind one name word ('Smith, Jr.', 'Smith,
56PhD Jones'), where segment 1 then reaches this stage with no pieces.
57A segment 1 that does reach it gets leading titles, then the same
58trailing title run over the pieces this segment does not read as
59suffixes ('Smith, John Prof.'), then given, then middles with those
60suffix-reading pieces to suffix -- one predicate for both -- and
61#602's run after the given word ('Smith, John PhD Jones'); segments 2+
62are suffixes but for their title words (#603), lenient -- segment
63already flagged the ones that are neither COMMA_STRUCTURE.
64SUFFIX_COMMA: segment 0 as NO_COMMA; segment 1 arrives bound by group,
65and segments 2+ are read as after a family comma.
66Emits PARTICLE_OR_GIVEN when the leading name piece is a lone
67particles_ambiguous token with more pieces following ("Van Johnson",
68and since #367 "Dr. Van Johnson" too, a title no longer displacing the
69particle out of that position) -- whatever role name_order assigns,
70save a lowercase one in a mixed-case name under a family-first order,
71which P1's fold takes (rules/P11, #670) --
72and, since #613, at P6's attachment after a family comma where the run
73holds an ambiguous particle ("Beethoven, Ludwig van").
74Emits SUFFIX_OR_NAME at these sites: the trailing roman numeral, taken
75or declined (since #718 for want of words to spare, since #752 for its
76Title case), a numeral ending the family part before a family comma,
77one of two or more letters ending a given part's walk, a bare I or V
78read as the generation ending that walk behind a name word (since
79#877), each
80ambiguous acronym the trailing peel had to resolve, the bare-suffix
81carve-out where an input that is nothing but post-nominal vocabulary
82gets its first word made into the name (H4's suffix half, #491),
83-- since #289 -- the FAMILY-COMMA path's own read of the first
84post-comma piece as the given name, -- since #531 -- the class member
85ENDING that path's given part, which the first-piece emitter could
86never reach, -- since #602 -- each name word the credential run
87absorbs, in the no-comma name and in the given part after a family
88comma, and -- since #613 -- P6's attachment after a family comma,
89where it declines a post-nominal reading or stands down inside a
90credential run.
91Further emitters of the same kind live in `_comma.py` (called at
92group's head: the comma's own decision, #544's anchored members and
93#603's absorbed words) and `_group.py`; they are not assign's and are
94not counted here. And
95at the one site that places a LONE name word, GIVEN_OR_FAMILY for the
96field the convention picked (O5, #449) and TITLE_OR_NAME for the two
97shapes where the doubt is whether a word is a title instead (H4 and
98H7, #491): the peel leaving one title-vocabulary word standing, and a
99joined unit carrying title vocabulary.
100"""
101from __future__ import annotations
103import unicodedata
105from collections.abc import Sequence, Set
106from typing import NamedTuple
108from nameparser._lexicon import Lexicon, _run_addresses_by_given
109from nameparser._pipeline._vocab import (
110 _ROMAN, effective_script, in_a_title_phrase, in_initialless_script,
111 is_initial_shaped,
112 TITLE_READING, is_listed_numeral, particle_reading,
113 is_suffix_lenient, name_reading, suffix_reading,
114 numeral_leans_to_the_name, numeral_settled_by_case,
115 reads_as_a_reported_numeral, reads_as_generation, resolve_script_set,
116)
117from nameparser._pipeline._pieces import (
118 split_credential_report,
119 anchor_in_reach, credential_at_the_given_slot, given_slot_anchors,
120 _NOT_A_RUN_START, has_name_content, is_lone_never_given_particle,
121 is_lowercase_ambiguous_particle,
122 Peel, TailRead, is_prefix_piece, is_suffix_piece, is_title_piece,
123 leading_titles,
124 particle_tail, listed_lean, peel_walk, settled_by_case,
125 starts_a_credential_run, starts_the_run_here,
126 tail_reading, trailing_titles,
127)
128from nameparser._pipeline._state import (
129 AMBIGUOUS_ACRONYM_TAG, ATTACHED_TAG, DOTTED_TITLE_TAG, NAME_ROLES, TITLE_AMBIGUOUS_TAG, ParseState,
130 Beside, Deferred, PendingAmbiguity, PendingReading, Structure,
131 WorkToken, _AMBIGUOUS_CREDENTIAL_TAGS, _NEVER_FLIPPED, copy_with,
132)
133from nameparser._policy import Policy, Script
134from nameparser._types import (FOLDED_TAG, SHAPE_ACRONYM_TAG, AmbiguityKind,
135 ReadingKind, ReadingSource, Role)
137def _set_roles(tokens: list[WorkToken], piece: tuple[int, ...],
138 role: Role) -> None:
139 for i in piece:
140 tokens[i] = copy_with(tokens[i], role=role)
143def _name_pieces(state: ParseState, tokens: Sequence[WorkToken],
144 seg_idx: int) -> list[tuple[int, ...]]:
145 """The pieces of segment `seg_idx` the parse read as name words."""
146 return [q for q in state.pieces[seg_idx]
147 if tokens[q[0]].role in NAME_ROLES]
150def _slot_field(state: ParseState, seg_idx: int, pos: int, count: int,
151 order: tuple[Role, Role, Role] | None) -> Role:
152 """The field position `pos` of `count` name words takes in segment
153 `seg_idx`, every other word read as the parse read it: the family
154 before a family comma, the given part's given or middle name after
155 one, and otherwise the order the main segment's read set.
157 `count` is two or more wherever no family comma decides, so rules/O5's
158 lone-word field is never asked here: `as_a_name` counts one more than
159 the name pieces, of which assign always places one (its lone-piece
160 branches index the first, so a parse with none would raise there); a split
161 pair stands behind a name word H4's carve-out keeps; and
162 `_resolve_deferred` counts a chained word one more than its name's,
163 and reads a join that would leave the name one word as no join
164 (rules/P3) before it asks."""
165 if state.structure is Structure.FAMILY_COMMA:
166 return (Role.FAMILY if seg_idx == 0
167 else Role.GIVEN if pos == 0 else Role.MIDDLE)
168 assert order is not None # the main segment's read sets it
169 return _name_positions(order, count)[pos]
172def _name_alt(token: WorkToken, role: Role,
173 front: Sequence[int] | None, tokens: Sequence[WorkToken],
174 first: bool, licensed: bool) -> PendingReading:
175 """The name reading of a word read as something else, landing in
176 `role` unless the name piece `front` it stands behind would take it
177 in, so that it lands where that piece lands: a particle chain runs
178 to the end of the part's name words (rules/P2), and a lone particle
179 stands unchained only for want of a word behind it, unless it opens
180 the part, where it never chains (rules/P4); and a bound given name
181 opening the part joins the word after it where it is `licensed`, a
182 family comma having fixed the family or a given-name title standing
183 ahead (rules/P5),
184 since the word would otherwise be the family word the join
185 reserves; and a never-given particle opening the part folds it into
186 the family (rules/P1). The front's field is named at
187 assemble, where P6 may since have attached it to the family, which
188 it would not have done with a word behind it (`Beside`)."""
189 alt = name_reading(token, role)
190 if front is None:
191 return alt
192 tags = tokens[front[0]].tags
193 # rules/P1's fold: a never-given particle opening the part takes the
194 # word behind it into the family, whatever the order and the part
195 if first and is_lone_never_given_particle(front, tokens):
196 return copy_with(alt, role=Role.FAMILY)
197 # a piece opening the part with a bound given name is P5's join,
198 # not a chain, though `abu` is a particle too: the word stands apart
199 bound = first and "vocab:bound-given" in tags
200 if ("particle" in tags and (len(front) > 1 or not first) and not bound
201 or licensed and bound and len(front) == 1):
202 # a front P6 has already attached held `role`'s field before it;
203 # `last` names the front's last word, which rules/P10 leaves the
204 # family where it reads the chain's opening phrase as a middle
205 # name (decisions/P10, #704)
206 return copy_with(alt, role=Beside(
207 front[0], role if FOLDED_TAG in tags else tokens[front[0]].role,
208 last=front[-1]))
209 return alt
212def _absorbed(piece: Sequence[int], tokens: Sequence[WorkToken],
213 alt: PendingReading | None) -> PendingAmbiguity:
214 """rules/S8's report for a name word #602's run absorbs, one
215 spelling for the no-comma and the given-part run. `alt` is its
216 name reading, whose field the caller's positional read knows (None
217 where the caller fills it in once it does)."""
218 text = " ".join(tokens[i].text for i in piece)
219 return PendingAmbiguity(
220 AmbiguityKind.SUFFIX_OR_NAME,
221 f"{text!r} follows a credential, so it reads as part of the "
222 f"suffix run; it may be a name word", tuple(piece),
223 alternative=alt)
226#: Tags that say the word's own reading was claimed before position
227#: could speak, so O5's convention decided nothing. Built from M4's
228#: `_NEVER_FLIPPED` pair rather than respelling it: the two sets answer
229#: different questions -- may M4 retag the word, and did anything
230#: decide the field -- and share the pair because a bound given name,
231#: or a word written as an initial, answers both. The
232#: third, `particle`, is here alone because a lone particle's reading
233#: is P4's. None is a predicate this emitter owns: they are read off
234#: the tags classify already recorded (mechanisms.md#TWO-LAYER-ASSIGN).
235_WORD_ALREADY_CLAIMED = _NEVER_FLIPPED | frozenset({"particle"})
238# rules/H2: "an abbreviation opening the part of the name that
239# carries the given name — the whole name, or the part after a
240# family comma — reads as a title even when unlisted" -- the count is
241# _pieces.leading_titles since #424 (the chain's leading position is
242# read off the same run, _pieces.chain_lead, #624); the roles are set
243# here.
244def _peel_leading_titles(pieces: tuple[tuple[int, ...], ...],
245 ptags: tuple[frozenset[str], ...],
246 tokens: list[WorkToken],
247 ambiguities: list[PendingAmbiguity] | None,
248 lexicon: Lexicon) -> int:
249 """Assign TITLE to the leading title pieces and return the first
250 non-title index. `ambiguities` is None where a family comma has
251 already marked the part as the given name, so no fork is left to
252 report."""
253 n = leading_titles(pieces, ptags, tokens)
254 for k in range(n):
255 _set_roles(tokens, pieces[k], Role.TITLE)
256 # rules/H9: "read as a name word where it is the first name word
257 # after any titles" -- the peel stops at it because classify read it
258 # out of the title, so it is the first piece the peel leaves
259 # ('Dean Smith', 'Mr Dean Smith'); and rules/H9: "a word no listed
260 # phrase holds is a title only behind a style or a modifier, or
261 # before `of`", so standing alone it reports nothing ('Grace Smith')
262 if (ambiguities is not None and n < len(pieces) and len(pieces[n]) == 1
263 and TITLE_AMBIGUOUS_TAG in tokens[pieces[n][0]].tags
264 and "vocab:title" not in tokens[pieces[n][0]].tags
265 and in_a_title_phrase(tokens[pieces[n][0]].text, lexicon)):
266 i = pieces[n][0]
267 ambiguities.append(PendingAmbiguity(
268 AmbiguityKind.TITLE_OR_NAME,
269 f"{tokens[i].text!r} is a title that may also be a given name, "
270 f"and stands in no title phrase; read as ", (i,),
271 field_tail=" rather than a title", alternative=TITLE_READING))
272 return n
275class EffectiveOrder(NamedTuple):
276 """What _effective_order made of a name's scripts. `order` is the
277 order the positional read uses; `by_script` says a script_orders
278 entry RESOLVED it, which is not the same as `order` happening to
279 equal the declared name_order -- under a declared family-first
280 order a Han name's W4 entry and the declaration agree, and only
281 this flag distinguishes the script rule DECIDING the reading from
282 the caller's declaration standing unopposed. O5's convention
283 report reads it (#449)."""
285 order: tuple[Role, Role, Role]
286 by_script: bool
289# rules/W4: "a name written wholly in one East Asian script, or in
290# the kana-licensed Japanese repertoire, reads family-first whatever
291# order the caller declared; a wholly-katakana name keeps the declared
292# order" (history: decisions/W4)
293def _effective_order(policy: Policy,
294 pieces: Sequence[tuple[int, ...]],
295 name_pieces: Sequence[int],
296 tokens: list[WorkToken],
297 *, dot_divided: bool) -> EffectiveOrder:
298 """script_orders resolution (#271): when every name piece is
299 written wholly in ONE script that has an entry, that script's
300 order governs the positional read; anything else -- Latin, mixed
301 scripts, no entry -- falls back to name_order. A 间隔号-divided
302 name (`dot_divided`, #298) suppresses the whole lookup first: the
303 dot marks a transcription -- playing the role pure katakana plays
304 in the kana license, orthography naming the convention -- so the
305 license yields to name_order. Piece-level, after
306 title/suffix peeling: 'Dr. 毛泽东' is a wholly-Han NAME under a
307 Latin title. Kana-licensed tokens (高橋みなみ, #272) resolve to
308 HIRAGANA the same way a wholly-Han or wholly-Hangul token resolves
309 to its own script -- and so does a kana-licensed NAME split across
310 separately single-script PIECES ('高橋 みなみ', Han piece plus
311 Hiragana piece): resolve_script_set generalizes the license from
312 one token's characters to the whole found-script set below, which
313 is why Han+Hangul ('毛 김') still declines even though both
314 individually read family-first -- the license is specific to the
315 Han/Hiragana/Katakana repertoire, not "the entries happen to
316 agree".
318 Naming note, since the two are easy to conflate: THIS function
319 resolves the ORDER for a whole name; `_vocab.effective_script`
320 resolves the SCRIPT for a single token. This function calls that
321 one per token below.
323 Takes the segment's pieces and WHICH of them the name kept, the
324 shape every piece-layer predicate takes: a caller holding those
325 indices had to build a second list of the same pieces to hand
326 over otherwise, which on 3.11 is a comprehension frame on every
327 parse (decisions/parse-cost).
329 Returns an EffectiveOrder: the order triple, and `by_script` set only on
330 the one path where an entry answered. Every fallback below is a
331 script rule DECLINING, and reports it as such.
332 """
333 declared = EffectiveOrder(policy.name_order, by_script=False)
334 # #298 transcription marker -- see the docstring; codepoint-scoped
335 # (only U+00B7 records; decisions/T3)
336 if dot_divided:
337 return declared
338 if not policy.script_orders:
339 return declared
340 # Collect every token's script rather than comparing pairwise as
341 # tokens are seen: the kana license needs the WHOLE set (a Han
342 # piece and a Hiragana piece only license together, never one at a
343 # time), so resolution is deferred to resolve_script_set below.
344 found: set[Script] = set()
345 for piece_idx in name_pieces:
346 for i in pieces[piece_idx]:
347 script = effective_script(tokens[i].text)
348 if script is None:
349 # Latin, mixed, or a script with no entry: never a key
350 return declared
351 found.add(script)
352 resolved = resolve_script_set(found)
353 if resolved is None:
354 # e.g. Han+Hangul: two scripts, neither the kana license's
355 # Han/Hiragana/Katakana repertoire -- no single tradition
356 return declared
357 for script, order in policy.script_orders:
358 if script is resolved:
359 return EffectiveOrder(order, by_script=True)
360 # the resolved script has no entry: the declaration stands, and
361 # nothing about the writing system decided the reading
362 return declared
365# rules/O4: "words no vocabulary has claimed read by position. In
366# the default given-first order the first name word is the given name,
367# the last is the family name, and everything between is middle names"
368def _name_positions(order: tuple[Role, Role, Role],
369 count: int) -> list[Role]:
370 """Roles for `count` name pieces (titles/suffixes already peeled),
371 per name_order. GIVEN_FIRST: given, middles..., family.
372 FAMILY_FIRST: family, given, middles... FAMILY_FIRST_GIVEN_LAST:
373 family, middles..., given. One piece takes order[0], which
374 `_assign_main`, placing a lone name word, overrides with rules/O5's
375 field (every other caller lays out two or more, or slices the first
376 slot off); two pieces take order[0] and the other primary."""
377 first, second = order[0], order[1]
378 if count == 1:
379 return [first]
380 if first is Role.GIVEN: # GIVEN_FIRST
381 return ([Role.GIVEN] + [Role.MIDDLE] * (count - 2)
382 + [Role.FAMILY])
383 if second is Role.GIVEN: # FAMILY_FIRST
384 return ([Role.FAMILY, Role.GIVEN]
385 + [Role.MIDDLE] * (count - 2))
386 return ([Role.FAMILY] + [Role.MIDDLE] * (count - 2) # F_F_GIVEN_LAST
387 + [Role.GIVEN])
390def _placed(read: TailRead, rest: list[int],
391 pieces: tuple[tuple[int, ...], ...],
392 ) -> tuple[list[int], tuple[int, ...], Peel]:
393 """`tail_reading`'s three answers, taken off the run group read at
394 its head (#614) rather than read again over the joined pieces: the
395 walk with the name pieces first and the run's suffixes after them,
396 the pieces the H5 chain took, and the read's peel with `names` and
397 `run_titles` renumbered onto these pieces. Group split the run off
398 before its joins and put it back after them unchanged, so each of
399 its pieces is whole here and a membership test on its first token
400 places it."""
401 names: list[int] = []
402 suffixes: list[int] = []
403 titled: list[int] = []
404 run_titles: list[int] = []
405 for k in rest:
406 first = pieces[k][0]
407 if first not in read.tail:
408 names.append(k)
409 elif first in read.titles:
410 titled.append(k)
411 else:
412 suffixes.append(k)
413 if first in read.run_titles:
414 run_titles.append(k)
415 peel = read.peel
416 count = len(names)
417 names.extend(suffixes)
418 # built, not `_replace`d: that is two frames on every read
419 return names, tuple(titled), Peel(
420 count, peel.numerals, peel.picks, peel.anchors, tuple(run_titles),
421 peel.absorbed)
424def _assign_main(seg_idx: int, state: ParseState,
425 tokens: list[WorkToken],
426 ambiguities: list[PendingAmbiguity],
427 ) -> tuple[Role, Role, Role] | None:
428 """Returns the order the positional read used, for ParseState.order
429 -- None on every path that returns before resolving one."""
430 pieces = state.pieces[seg_idx]
431 ptags = state.piece_tags[seg_idx]
432 n = _peel_leading_titles(pieces, ptags, tokens, ambiguities,
433 state.lexicon)
434 if n == len(pieces):
435 return None
436 # group-flagged suffix pieces (the ph-d merge) are suffixes at ANY
437 # position -- v1's fix_phd extracted the credential from the string
438 # before parsing, so position never mattered (PR review I3).
439 # Walked rather than collected first: the list was never read
440 # again, and on 3.11 a comprehension is a frame of its own, which
441 # is one frame back against the one the H5 reading below costs. A
442 # 3.11 FACT and not a portable one: PEP 709 inlines comprehensions
443 # from 3.12, where this tree's reference parse costs 395 written
444 # either way (measured 2026-09-09, against 416 and 417 on 3.11).
445 # 3.11 is the interpreter the band is quoted for, so 3.11 is what
446 # the shape is chosen on (decisions/parse-cost).
447 for k in range(n, len(pieces)):
448 if "suffix" in ptags[k]:
449 _set_roles(tokens, pieces[k], Role.SUFFIX)
450 rest = peel_walk(n, ptags)
451 if not rest:
452 return None
453 # rules/N3: "a name that is only a nickname and one name word
454 # reads that word as the family name" (history: decisions/N3)
455 # -- v1's p_len == 1 counted
456 # the WHOLE segment before any title peeling -- 'Xyz. (Bud) Smith'
457 # has two pieces, so the title peel wins and this rule declines;
458 # H1 then names Smith the family (rules/N3's title bullet)
459 # A suffix makes two pieces as a title does: that shape is N3's
460 # suffix clause, at O5's site below.
461 # The nickname scan sits LAST in the test: it is a generator, which
462 # every interpreter resumes once per token (seven call events on
463 # the reference name), and only a one-piece segment ever reads its
464 # answer. Hoisting it to the top of the read, where it once stood,
465 # is what put 3.12-3.15 one call over the band (decisions/parse-cost).
466 # Behind `state.extracted`, as at N3's suffix clause below: a
467 # nickname is a clause extract set aside.
468 if (len(pieces) == 1 and len(rest) == 1 and state.extracted
469 and any(t.role is Role.NICKNAME for t in tokens)):
470 _set_roles(tokens, pieces[rest[0]], Role.FAMILY)
471 return None
472 # rules/S2's trailing peel and rules/H5's title chain, read
473 # together to their fixed point by _pieces.tail_reading -- one
474 # function since the /simplify round, shared with the bound-given
475 # reserve (P5), which must count the name words this leaves.
476 #
477 # rules/H5: "successive single words that wear the abbreviation
478 # shape and are title vocabulary chain into the title from the end,
479 # leaving one name word standing" -- the titles are set BEFORE
480 # _name_positions, so the shortened list is what the positional
481 # read and the script test both see (a trailing Latin title must
482 # not make a wholly-CJK name look mixed-script, the same reason the
483 # leading peel runs first). Read ahead of the bare-suffix carve-out
484 # because with no name piece left there is nothing for the chain to
485 # read: its floor keeps 0 pieces of an empty list, so the branch
486 # below is reached exactly as before.
487 #
488 # Every bare ambiguous acronym the FINAL peel had to resolve is one
489 # coin-flip each, in either direction, so the report collects
490 # rather than overwrites. Deferred to after assignment because what
491 # a pick is declined as depends on the role it took, and which role
492 # "not peeled" means depends on name_order; the field itself is
493 # worded by assemble (#626). (The roman-numeral fork's TAKEN branch
494 # needs no such deferral and is reported here; its declined branch
495 # is a pick, #718, and is reported with the others.)
496 #
497 # Read once in group, after P3's connective joins and ahead of the
498 # particle chain, on a main segment of three or more pieces whose
499 # run is one block at its end (#614): the run is placed off that
500 # reading. Where the read left name pieces and
501 # group's joins then left none past the leading titles -- a join
502 # that made the name a title ('Freiherr von Berg Dr. and Ed.
503 # Prof.') -- the reading is taken here over what stands, as it was
504 # taken everywhere before #614, so the name keeps a word. The same
505 # procedure and not always the same answer: the run group read
506 # split off was kept out of group's joins, so what stands differs
507 # from what stood before #614 (that name read family 'von Berg Dr.
508 # and Ed. Prof.' then).
509 read = (state.tail_reads[seg_idx]
510 if seg_idx < len(state.tail_reads) else None)
511 if read is not None:
512 placed = _placed(read, rest, pieces)
513 if read.peel.names and not placed[2].names:
514 read = None
515 else:
516 rest, titled_tail, peeled = placed
517 if read is None:
518 rest, titled_tail, peeled = tail_reading(rest, pieces, ptags, tokens,
519 state.one_case)
520 for piece_idx in titled_tail:
521 _set_roles(tokens, pieces[piece_idx], Role.TITLE)
522 # rules/S8: "except a title word, which reads as a title" -- the
523 # credential run (#602) is placed here, so a name word it absorbed
524 # is reported here too
525 for piece_idx in peeled.run_titles:
526 _set_roles(tokens, pieces[piece_idx], Role.TITLE)
527 # the name reading of a word the run or the numeral fork took lands
528 # where one more name word after the name's would: its field is
529 # filled in below, once the positional read has an order
530 trailing_alts = len(ambiguities)
531 for piece in peeled.absorbed:
532 ambiguities.append(_absorbed(piece, tokens, None))
533 for numeral in peeled.numerals:
534 # a single letter is a name part unless it happens to be a
535 # roman numeral -- and V/X/I are ordinary middle initials, so
536 # taking it as a suffix is a call, not a fact; a numeral read
537 # by its shape alone is in no wordlist, and may be a name word
538 text = tokens[numeral[0]].text
539 ambiguities.append(PendingAmbiguity(
540 AmbiguityKind.SUFFIX_OR_NAME,
541 f"{text!r} is a roman numeral, so it reads as a "
542 f"generational suffix; "
543 + ("any other single letter there would be a middle initial"
544 if len(text.rstrip(".")) == 1
545 else "it may be a name word"),
546 numeral))
547 name_pieces, suffix_pieces = rest[:peeled.names], rest[peeled.names:]
548 if peeled.run_titles:
549 suffix_pieces = [q for q in suffix_pieces
550 if q not in peeled.run_titles]
551 if peeled.names == 0:
552 # everything suffix-shaped after titles: first one is the name
553 name_pieces, suffix_pieces = suffix_pieces[:1], suffix_pieces[1:]
554 # AFTER the whole tail reading, and load-bearing: the script test
555 # sees the NAME pieces only, so a Latin title or suffix ('Dr. 毛
556 # 泽东', '毛 泽东, PhD') cannot make a wholly-CJK name look
557 # mixed-script.
558 resolved = _effective_order(state.policy, pieces, name_pieces, tokens,
559 dot_divided=bool(state.interpunct_offsets))
560 order = resolved.order
561 # rules/O5: "a name of one name word that nothing else has
562 # decided reads that word as the field the caller's lone-word
563 # setting names" -- `Policy.lone_name_word`, a convention, not a
564 # determination: O4 has no positions to compare at one word, so
565 # this line is where the library picks one of two equally
566 # consistent readings and picks it the same way every time. A
567 # script that settles the order decides a lone word's field too
568 # (O5's W4 Except). The rules that DO decide such a name (H1, M4
569 # and N3's suffix clause; N3's bare shape returns before reaching
570 # here) run after this and retag (decisions/O5, #1111).
571 lone_field = (order[0] if resolved.by_script
572 else state.policy.lone_name_word)
573 roles = ([lone_field] if len(name_pieces) == 1
574 else _name_positions(order, len(name_pieces)))
575 for pos, piece_idx in enumerate(name_pieces):
576 _set_roles(tokens, pieces[piece_idx], roles[pos])
577 for piece_idx in suffix_pieces:
578 _set_roles(tokens, pieces[piece_idx], Role.SUFFIX)
580 # rules/O4: "a name word followed by nothing but initials reports
581 # the order" -- the positional read takes the word as the given
582 # name, where a byline writes its family name first (`Armitage J
583 # O`); declared nowhere, so the default given-first order picked it
584 # (decisions/O4, #1254). Bare capitals (`Armitage JO`) are no
585 # initials here: they are written as a short surname is (`Zhi LI`).
586 # Inline and nested so a plain name exits at its second piece
587 # (the parse-cost band, decisions/parse-cost). No test of the comma
588 # or the segment is needed: assign calls this for segment 0 under
589 # no comma or a suffix comma alone. Nor of a script's order: one
590 # needs the name written wholly in Han, Hangul or kana, which hold
591 # no initial (rules/T5) for the walk below to find (#1348 review).
592 order_report: PendingAmbiguity | None = None
593 if (len(name_pieces) >= 2 and order[0] is Role.GIVEN
594 and not state.policy.initials_as_family):
595 first = pieces[name_pieces[0]]
596 # a particle in front is PARTICLE_OR_GIVEN's fork, reported
597 # there; an initial in front is refused below by its one letter
598 initialed = (len(first) == 1
599 and "particle" not in tokens[first[0]].tags)
600 for k in name_pieces[1:]:
601 if not initialed:
602 break
603 piece = pieces[k]
604 letters = tokens[piece[0]]
605 # chunks of the composed spelling, as classify measures a
606 # letter (`composed_len`): a decomposed `É.S.` is one run;
607 # the shape tag is given only to letters (period_joined_vocab's
608 # alphabetic gate), so a chunk's length is the whole test
609 initialed = len(piece) == 1 and (
610 "initial" in letters.tags
611 or SHAPE_ACRONYM_TAG in letters.tags
612 and all(len(chunk) == 1
613 for chunk in unicodedata.normalize(
614 "NFC", letters.text).rstrip(".").split(".")))
615 text = unicodedata.normalize("NFC", tokens[first[0]].text)
616 # the word in front is a name word a byline could write first:
617 # no run of initials itself (rules/T7's `J.A.`), and in a script
618 # that writes initials, as no byline in Han or Hangul does
619 if (initialed and any(c.isalpha() for c in text)
620 and not all(len(chunk) == 1
621 for chunk in text.rstrip(".").split("."))
622 and not in_initialless_script(text)):
623 order_report = PendingAmbiguity(
624 AmbiguityKind.ORDER,
625 f"{tokens[first[0]].text!r} is the only name word before "
626 f"initials, which a byline writes after the family name "
627 f"and a South Asian name writes as the family; read as ",
628 tuple(first),
629 # rules/A1: "the order (O4), whose other reading lays out
630 # the whole name rather than moving one word" -- so the
631 # report names none and says it in words (decisions/O4)
632 field_tail=(" by the given-first order; read family "
633 "first (name_order=FAMILY_FIRST), the word in "
634 "front is the family name and the letters "
635 "the given and middle initials"))
637 def as_a_name(i: int) -> PendingReading:
638 # every other word read as the parse reads it -- the carve-out's
639 # name word too -- the name word the reading declines would be
640 # the last of one more than the name's; asked per report, so a
641 # name that makes no report here pays nothing
642 front = (pieces[name_pieces[-1]] if name_pieces
643 and i not in pieces[name_pieces[-1]] else None)
644 # rules/P5's other licence, a given-name title ahead, asked of
645 # the leading run through the predicate group's join asks, so
646 # the two cannot disagree about what one run asserts; asked
647 # only where the bound branch could take it, so a report per
648 # word of a long run never walks the name (the scaling test)
649 licensed = (front is not None and len(name_pieces) == 1
650 and "vocab:bound-given" in tokens[front[0]].tags
651 and _run_addresses_by_given(
652 (tokens[j].text for k in range(name_pieces[0])
653 for j in pieces[k]),
654 state.lexicon._sets.given_name_titles))
655 return _name_alt(tokens[i],
656 _slot_field(state, seg_idx, len(name_pieces),
657 len(name_pieces) + 1, order),
658 front, tokens, len(name_pieces) == 1, licensed)
660 for r in range(trailing_alts, len(ambiguities)):
661 report = ambiguities[r]
662 ambiguities[r] = copy_with(
663 report, alternative=as_a_name(report.indices[0]))
664 # after the trailing reports are given their name reading, which
665 # the order report has none of
666 if order_report is not None:
667 ambiguities.append(order_report)
668 # Both emitters below turn on the PEEL's count, so neither can
669 # reach a name that kept two name pieces, and the two scans are
670 # nested under the count rather than run beside it: they cost the
671 # call budget on every parse otherwise (decisions/parse-cost).
672 if peeled.names <= 1:
673 head = pieces[name_pieces[0]]
674 # Both conventions here turn on a lone name word, so both
675 # report at the site that places one
676 # (mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE), and one
677 # predicate carries what says nothing else decided the FIELD:
678 # a title (segment 0's own peel, or one a family comma left in
679 # the next segment -- either is a Role.TITLE by now, and
680 # either makes the reading H1's), a maiden name (M4's), or a
681 # script order convention. `resolved.by_script` rather than a
682 # comparison against name_order: a declared family-first order
683 # agreeing with a Han name's entry is agreement, not
684 # authorship, and for a lone CJK honorific ('さん', '씨') it
685 # scopes W2's reading out rather than excusing it (#271/#308).
686 titled = any(t.role is Role.TITLE for t in tokens)
687 field_undecided = (
688 not resolved.by_script and not titled
689 and not any(t.role is Role.MAIDEN for t in tokens))
690 # rules/H4: "an input whose every word is post-nominal
691 # vocabulary reads its first word as a name and reports
692 # `suffix-or-name`" (history: decisions/H4) -- only the
693 # word made into a name reports, and no name piece survived
694 # the peel, so the carve-out above made the first post-nominal
695 # the name. Its field is worded by assemble from the final role
696 # (`PendingAmbiguity.field_tail`, #626).
697 if peeled.names == 0 and field_undecided:
698 text = " ".join(tokens[i].text for i in head)
699 ambiguities.append(PendingAmbiguity(
700 AmbiguityKind.SUFFIX_OR_NAME,
701 f"{text!r} is post-nominal vocabulary with no name word "
702 f"beside it; read as ", tuple(head),
703 field_tail=" rather than a post-nominal, nothing else "
704 "being left to be the name",
705 alternative=suffix_reading(tokens[head[0]])))
706 # One name piece off the peel: the convention placed a lone
707 # name word. A suffix beside it is not a decision -- 'Smith
708 # Jr.' IS this convention -- and the count comes off the peel
709 # rather than off name_pieces, which is what keeps the carve-out
710 # above out of these branches. Title vocabulary in the unit makes
711 # the doubt H4's or H7's (is the WORD a title), anything else
712 # O5's (which FIELD it took, or N3's where a nickname decides
713 # it), so only the second reads `field_undecided`: a title or a
714 # maiden name beside a lone title word is placed by H1 or M4
715 # whatever N3 does here, and a script order never reaches this
716 # branch (#1169).
717 if peeled.names == 1 and not resolved.by_script:
718 # rules/H4: "an input whose only remaining name word
719 # after the title peel is itself title vocabulary reads
720 # that word as the name by convention and reports
721 # `title-or-name`" (history: decisions/H4), and
722 # rules/H7: "one name UNIT of more than one word carrying
723 # title vocabulary reports `title-or-name`" (history:
724 # decisions/H7, decisions/H4 #518) -- one branch, its detail
725 # naming a field only in H7's shape ('John of Bishop',
726 # 'Attorney General of Minnesota'), where the unit is more
727 # than the title word.
728 # A LONE title word never reaches here ('Dr.', 'Prince of
729 # Wales'): the leading-title peel took the whole name.
730 if any("vocab:title" in tokens[i].tags for i in head):
731 text = " ".join(tokens[i].text for i in head)
732 lone = len(head) == 1
733 # rules/N3: "a name that is only a nickname and one name
734 # word reads that word as the family name, whatever
735 # suffix stands beside them" -- a title word left as
736 # that word too, its doubt (is it a title) still H4's
737 # report; the scan behind `state.extracted`, as at N3's
738 # other sites (decisions/N3, #1169)
739 # The word's own reading, as O5's clause below asks it:
740 # a title-particle ('St', 'Graf') or an initial keeps
741 # the reading it has, as N3's Except has it
742 if (lone and state.extracted
743 and tokens[head[0]].tags.isdisjoint(
744 _WORD_ALREADY_CLAIMED)
745 and any(t.role is Role.NICKNAME for t in tokens)):
746 _set_roles(tokens, head, Role.FAMILY)
747 ambiguities.append(PendingAmbiguity(
748 AmbiguityKind.TITLE_OR_NAME,
749 f"{text!r} is title vocabulary and the only name word "
750 f"the title peel left standing; read as the name by "
751 f"convention rather than as more title"
752 if lone else
753 f"{text!r} is the only name unit and joins title "
754 f"vocabulary to a name word; read as ",
755 tuple(head),
756 field_tail=None if lone else " by convention",
757 alternative=TITLE_READING))
758 # rules/O5: "a name of one name word that nothing else has
759 # decided reads that word as the field the caller's lone-word
760 # setting names" -- the report for that reading (history:
761 # decisions/O5, #449, #1111). Past
762 # `field_undecided`, two clauses of O5's own: the word's
763 # own reading may have claimed it, and A2's content test
764 # -- a piece with no alphanumeric character is no name
765 # word, so parse("(") keeps its unbalanced-delimiter
766 # report and gains nothing here.
767 elif (field_undecided
768 and all(tokens[i].tags.isdisjoint(_WORD_ALREADY_CLAIMED)
769 for i in head)
770 and any(c.isalnum() for i in head
771 for c in tokens[i].text)):
772 # rules/N3: "a name that is only a nickname and one name
773 # word reads that word as the family name, whatever
774 # suffix stands beside them" -- the suffixed shape the
775 # early count above declines, decided here, where the
776 # peel has set the suffix aside and O5's clauses have
777 # said nothing else decided the word (decisions/N3,
778 # #815). The early count's scan, behind
779 # `state.extracted` as at N3's bare site above (a
780 # nickname is only ever a clause extract), so a name
781 # with no clause pays no frame for it.
782 if state.extracted and any(t.role is Role.NICKNAME
783 for t in tokens):
784 _set_roles(tokens, head, Role.FAMILY)
785 else:
786 text = " ".join(tokens[i].text for i in head)
787 ambiguities.append(PendingAmbiguity(
788 AmbiguityKind.GIVEN_OR_FAMILY,
789 f"{text!r} is the only name word and nothing else "
790 f"decides it; read as ", tuple(head),
791 field_tail=" by convention, the field "
792 "Policy.lone_name_word names",
793 # the other primary field, named from the final
794 # role by assemble
795 alternative=PendingReading(ReadingKind.NAME,
796 ReadingSource.POSITION,
797 Deferred.OTHER_PRIMARY)))
798 # rules/S10: "a one-letter numeral before a credential is the
799 # family name, and the reading is reported" -- the part's last name
800 # word with nothing but the trailing run behind it (#721, Derek:
801 # 'John Smith V PhD' keeps family 'V' and reports, as 'john i, phd'
802 # does). Decided here, where the trailing read has said what that
803 # run is
804 # `_ROMAN` first, C-level, so an ordinary name word pays no frame
805 if (len(name_pieces) >= 2
806 and len(last := pieces[name_pieces[-1]]) == 1
807 and _ROMAN.match((word := tokens[last[0]].text).rstrip("."))
808 is not None
809 # the trailing run, or the Ph./D. pair, which the walk
810 # leaves out, behind it
811 and (suffix_pieces or titled_tail
812 or name_pieces[-1] + 1 < len(pieces)
813 and "suffix" in ptags[name_pieces[-1] + 1])
814 and tuple(last) not in peeled.picks
815 and reads_as_a_reported_numeral(
816 word, tokens[last[0]].tags,
817 tokens[pieces[name_pieces[-2]][0]].text, state.one_case)):
818 ambiguities.append(PendingAmbiguity(
819 AmbiguityKind.SUFFIX_OR_NAME,
820 f"{word!r} is a roman numeral, but a post-nominal or title "
821 f"follows it; read as ", tuple(last),
822 field_tail=" rather than a generational suffix",
823 alternative=suffix_reading(tokens[last[0]])))
824 for piece in peeled.picks:
825 # every pick is in rest, so the loops above just gave it a
826 # role; whether it was read as a suffix or a name settles what
827 # it was declined as, and which field assemble names (#626)
828 token = tokens[piece[0]]
829 assert token.role is not None
830 # rules/S10: "either reading is reported" -- the numeral fork's
831 # declined pick (#718), which carries neither credential tag
832 # (the acronym fork's picks carry one) and is never a suffix
833 if token.tags.isdisjoint(_AMBIGUOUS_CREDENTIAL_TAGS):
834 # a pick a credential run in front then took is the run's
835 # absorbed name word (rules/S8); otherwise it is declined
836 # for want of words to spare (one letter) or for its Title
837 # case (two or more, rules/S10's lean)
838 if token.role is Role.SUFFIX:
839 ambiguities.append(_absorbed(piece, tokens,
840 as_a_name(piece[0])))
841 continue
842 why = ("is a roman numeral, but the name has no other word "
843 "to spare for it" if len(token.text.rstrip(".")) == 1
844 else "is a roman numeral written in Title case, which "
845 "leans to the name")
846 ambiguities.append(PendingAmbiguity(
847 AmbiguityKind.SUFFIX_OR_NAME,
848 f"{token.text!r} {why}; read as ", piece,
849 field_tail=" rather than a generational suffix",
850 alternative=suffix_reading(token)))
851 continue
852 declined = ("a name part" if token.role is Role.SUFFIX
853 else "a post-nominal")
854 # rules/S9: "An unlisted word joins the ambiguous class (S4) by
855 # SHAPE" (and rules/S3); such a word is in no
856 # wordlist and may be written with its periods, so the listed
857 # member's wording would misdescribe it on both counts (#563)
858 what = ("is shaped like a post-nominal but listed in no "
859 "vocabulary, so it may be an ordinary name"
860 if SHAPE_ACRONYM_TAG in token.tags
861 else "written without periods is both a post-nominal "
862 "and an ordinary name")
863 ambiguities.append(PendingAmbiguity(
864 AmbiguityKind.SUFFIX_OR_NAME,
865 f"{token.text!r} {what}; read as ", piece,
866 field_tail=f" rather than {declined}",
867 alternative=(as_a_name(piece[0])
868 if token.role is Role.SUFFIX
869 else suffix_reading(token))))
870 # leading ambiguous particle read as a name (#121 surfaced)
871 if name_pieces:
872 head = pieces[name_pieces[0]]
873 # rules/P11: "an ambiguous particle written in lower case, in a
874 # name of mixed case, opens the name as a never-given one does"
875 # -- and "nothing is reported" there, the writing having settled
876 # the fork (decisions/P11, #670)
877 if (len(head) == 1 and len(name_pieces) > 1
878 and "vocab:particle-ambiguous" in tokens[head[0]].tags
879 and not (order is not None and order[0] is Role.FAMILY
880 and is_lowercase_ambiguous_particle(
881 head, tokens, state.one_case))):
882 # the field is assemble's to word from the final role
883 # (`PendingAmbiguity.field_tail`, #626), never assumed given:
884 # `order` above is _effective_order's answer, not
885 # necessarily name_order's, and a later rule may still move
886 # the word ('Van Ivan Petrovich' under the East Slavic rule
887 # rotates 'Van' into the family)
888 ambiguities.append(PendingAmbiguity(
889 AmbiguityKind.PARTICLE_OR_GIVEN,
890 f"leading {tokens[head[0]].text!r} may be a family-name "
891 f"particle; read as ", tuple(head), field_tail="",
892 alternative=particle_reading(tokens[head[0]].tags)))
893 return order
896def _names_before(fam_pieces: Sequence[Sequence[int]],
897 tokens: Sequence[WorkToken], lead: int, k: int) -> int:
898 """How many name words stand in front of the numeral at
899 `fam_pieces[k]`, the title run's `lead` pieces left out: the pieces
900 that are not wholly linking words (`_is_linking_word`)."""
901 # rules/S10: "a particle or a connective is no name word there,
902 # wherever it stands in the part" -- the count of the family part's
903 # numeral site; a chain or join holding a name word is one piece and
904 # counts (decisions/P9 (#1080), decisions/S10 (#1093))
905 return sum(1 for j in range(lead, k)
906 if not all(_is_linking_word(tokens[t].tags, j == k - 1)
907 for t in fam_pieces[j]))
910def _is_linking_word(tags: Set[str], beside: bool) -> bool:
911 """Whether a word of the family part is a linking word, which the
912 numeral's count leaves out: a particle, or a connective, save one
913 that is also suffix vocabulary standing `beside` the numeral."""
914 # rules/S10: "A connective that is also suffix vocabulary, as the
915 # generational 'i' is, stays in the count where P13 has not linked
916 # the letter" -- the exception's half of this predicate
917 # (decisions/P13 (#1097))
918 return "particle" in tags or ("conjunction" in tags and not (
919 beside and "vocab:suffix" in tags))
922def _family_part_numeral(word: WorkToken, preceding: str, names: int,
923 one_case: bool | None) -> Role | None:
924 """The role of a roman numeral ending the family part before a
925 family comma, behind `names` name words of that part (`_names_before`,
926 which leaves its linking words out), as the comma-less name reads one
927 ending it (rules/S10), or None where the numeral's
928 predicate reads no numeral there (an initial in front, a letter with
929 its period): the suffix where it reads as the generation, a
930 one-letter numeral needing two name words in front of it, save one
931 its written case settles (T4), and the family name where that count
932 fails or its Title case leans to the name."""
933 if reads_as_generation(word.text, word.tags, preceding, one_case):
934 if (names < 2 and len(word.text) == 1
935 and not numeral_settled_by_case(word.text, one_case)):
936 return Role.FAMILY
937 return Role.SUFFIX
938 if numeral_leans_to_the_name(word.text, word.tags, preceding, one_case):
939 return Role.FAMILY
940 return None
943def _reads_as_a_trailing_suffix(piece: Sequence[int],
944 prev_piece: Sequence[int],
945 prev_ptags: Set[str],
946 tokens: Sequence[WorkToken],
947 lexicon: Lexicon,
948 one_case: bool | None) -> bool:
949 """The lenient tail test for a trailing one-token piece after a
950 family comma, and #432's carve-out from it.
952 A word that could be a middle initial, written with the period that
953 marks an abbreviation, is name material -- so 'Smith, John V.' is
954 middle 'V.' where 'Smith, John V' stays suffix 'V' (v1 parity,
955 #144). Only behind a NAME word: behind a suffix the credential run
956 owns it, and 'Smith, John PhD I.' keeps suffix 'PhD, I.'.
958 The period is the whole carve-out. is_initial_shaped would be
959 redundant beside it: the caller only consults this where
960 is_suffix_piece said no, and a lenient single token it refuses is
961 one carrying the `initial` tag, so the shape is already implied.
963 reads_as_generation is asked with no word in front (`preceding`
964 None): its initial-in-front refusal is a no-comma rule and the
965 opposite of this path's v1 parity -- 'Chang, Andy C I' is first
966 Andy, middle C, suffix I, and asking it with the word in front
967 made the numeral a middle. The #401/#421 entry
968 under decisions/P5 records the same fork declining to transfer
969 to this walk under LENIENT.
970 """
971 # rules/C9: "behind a name word it is the generation only when
972 # written bare, a period marking it the abbreviation of a name and
973 # so a middle initial"
974 tok = tokens[piece[0]]
975 text = tok.text
976 if text.endswith(".") and not is_suffix_piece(
977 prev_piece, prev_ptags, tokens):
978 return False
979 # the numeral's predicate, which in this slot reads a bare
980 # one-letter numeral by the vocabulary ('Smith, John X' keeps
981 # middle 'X'); `_ROMAN` first, C-level
982 if len(text) == 1 and _ROMAN.match(text) is not None:
983 return reads_as_generation(text, tok.tags, None, one_case)
984 return is_suffix_lenient(text, lexicon)
987def _inside_a_credential_run(seg: Sequence[Sequence[int]],
988 tokens: list[WorkToken], given_at: int,
989 k: int, end: int,
990 ambiguities: list[PendingAmbiguity]) -> bool:
991 """rules/P9 (P6's third exception) for the run `seg[k:end]`: a
992 run assign read as post-nominals, with a credential read in
993 front of it and another behind, stands INSIDE the credential run
994 assign read whole rather than ending the name, so it keeps that
995 reading -- 'DOE, JANE PHD VD MA' reads suffix 'PHD VD MA' where the
996 attachment had pulled VD out of the middle of it (#573). Behind as
997 well as in front: with nothing behind it the word ends the name,
998 and 'Doe, Jane PhD vd' keeps family 'vd Doe'. A title between it
999 and the post-nominal in front is transparent, as it is to every
1000 trailing reading (H5): 'DOE, JANE PHD PROF. VD MA' reads as 'DOE,
1001 JANE PHD VD MA' does. The run is never empty: the caller asks
1002 only where `k < end`, and over an empty range the test below would
1003 pass vacuously and report a fork on the word behind (#613's
1004 review, ', Jane PhD do MA'). (Behind it no title is skipped: a title
1005 there leaves the word a name in assign, so nothing reaches here.)
1007 The test is the ROLES assign gave, not S7's company query: the
1008 question is whether assign read the run whole, and a member in
1009 front that the writing or a count made the credential ('DOE, JANE
1010 MA VD PHD') says so as plainly as a degree does. So does a
1011 generation, in front or behind ('Berg, Jan PhD vd Jr.'): the
1012 tussenvoegsel P6 is about stands right behind the given name, and
1013 a post-nominal between them says this word is not one. The run's own
1014 role is what keeps a plain particle out ('Doe, Jane PhD de PhD',
1015 whose 'de' is a name word, not a post-nominal), and the given
1016 name in front is what keeps 'Doe, Jane vd PhD' attaching.
1017 Reported as S4's credential fork, at the site that declines the
1018 attachment."""
1019 def suffix_read(q: int) -> bool:
1020 return all(tokens[i].role is Role.SUFFIX for i in seg[q])
1022 def is_title(q: int) -> bool:
1023 return all(tokens[i].role is Role.TITLE for i in seg[q])
1025 if not all(suffix_read(q) for q in range(k, end)):
1026 return False
1027 front = k - 1
1028 while front > given_at and is_title(front):
1029 front -= 1
1030 if (front <= given_at or end == len(seg)
1031 or not suffix_read(front) or not suffix_read(end)):
1032 return False
1033 run = seg[k]
1034 text = " ".join(tokens[i].text for i in run)
1035 ambiguities.append(PendingAmbiguity(
1036 AmbiguityKind.SUFFIX_OR_NAME,
1037 f"{text!r} written without periods is both a post-nominal and a "
1038 f"family-name particle; between credentials after a family "
1039 f"comma it reads as a post-nominal",
1040 tuple(run), alternative=particle_reading(tokens[run[0]].tags)))
1041 return True
1044def _attach_particle_tail(pieces: Sequence[Sequence[int]],
1045 tokens: list[WorkToken], given_at: int,
1046 k: int, end: int,
1047 ambiguities: list[PendingAmbiguity]) -> None:
1048 """rules/P6's attachment after a family comma: the particle run
1049 `pieces[k:end]` ending the given part joins the family the comma
1050 named and is written before it -- the Dutch alphabetized listing,
1051 "Beethoven, Ludwig van" being how "Ludwig van Beethoven" is filed.
1052 The caller has found the run (`particle_tail`) and checked that a
1053 given word stands ahead of it and that P1's fold will not take the
1054 part; this declines only where the run stands inside a credential
1055 run the walk read whole (`_inside_a_credential_run`, the third
1056 exception).
1058 Keyed on the tokens' VOCABULARY, not their roles, which is what
1059 gives the attachment its stated precedence over S2: `vd`, `mc` and
1060 `do` are the three words in both vocabularies; the walk reads a
1061 trailing `vd` or `mc` as a post-nominal, so those two need the
1062 override, and `do` is in the AMBIGUOUS acronym half, which already
1063 leaves it a name word, so it attaches by the plain rule. After a
1064 family comma the tussenvoegsel is the commoner reading.
1066 The words-to-spare guard is a piece test, not a count: every
1067 trailing piece that is wholly particles attaches, and the run must
1068 leave a GIVEN word ahead of it, so "Nguyen, Van" keeps its only
1069 given word. Only the DEGENERATE Vietnamese listing is protected by
1070 that -- "Nguyen, Thi Van" has a given word to spare, so `Van`
1071 attaches and the given name is lost. rules/P6 records why that
1072 is accepted.
1074 mechanisms.md#FOLDED_TAG does the rest: tokens never move, so the
1075 family view reads the tag and renders these before the base.
1077 Decided in assign since #613, in the pass that read the given
1078 part; until then post_rules took it a stage later, and assign held
1079 the tail back from #602's run by predicting it."""
1080 if _inside_a_credential_run(pieces, tokens, given_at, k, end,
1081 ambiguities):
1082 return
1083 run = [i for piece in pieces[k:end] for i in piece]
1084 # mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE: "Emit at
1085 # the site that takes the branch, not where an ambiguous
1086 # tag sits" -- this attachment is where the fork is
1087 # decided, so the report is raised here, at the attachment,
1088 # rather than by the given-part walk's own emitters, which
1089 # read the run before the attachment decides it (#405).
1090 #
1091 # Two arms, keyed on what the attachment OVERRODE rather
1092 # than on what the words are, so each names a branch the
1093 # parse actually weighed. `declined_suffix` reads the role
1094 # the walk gave the run, which is why both lists are built
1095 # BEFORE the re-roling loop below overwrites it.
1096 #
1097 # ORDERED, not asserted disjoint. The two cannot both hold
1098 # under the shipped vocabulary -- `vd` and `mc` are the
1099 # only words in both the particle and the unambiguous
1100 # suffix vocabularies, and neither is ambiguous particle
1101 # vocabulary -- but a caller's Lexicon may put one word in
1102 # both, and rules/A1 says parsing never fails on any
1103 # input, so this decides instead of raising. (Measured: an
1104 # assert here DID fire on `Berg, Jan zz` under a Lexicon
1105 # adding `zz` to particles_ambiguous and suffix_acronyms.)
1106 # The suffix arm wins because it names the reading the
1107 # parse actually took: assign had read the word as a
1108 # post-nominal, so the name-word reading was never on the
1109 # table for the attachment to decline.
1110 #
1111 # Both details name the ATTACHMENT and stop there. What
1112 # P6 decides is that the run joins the family the comma
1113 # named instead of standing on its own; how the joined
1114 # words then READ is R2's call, taken by the UNJOINED_TAG
1115 # loop at the end of post_rules. The two can disagree:
1116 # `de la, Jan van` attaches `van` and leaves an
1117 # all-particle family, which R2 marks, so every other view
1118 # -- `family_base`, the initials, case repair -- reads
1119 # those words as ordinary name words ('Jan Van De La'
1120 # forced). A detail promising "read as the family's
1121 # particle" would contradict all three.
1122 ambiguous = [i for i in run
1123 if "vocab:particle-ambiguous" in tokens[i].tags]
1124 declined_suffix = [i for i in run
1125 if tokens[i].role is Role.SUFFIX]
1126 text = " ".join(tokens[i].text for i in run)
1127 if declined_suffix:
1128 ambiguities.append(PendingAmbiguity(
1129 AmbiguityKind.SUFFIX_OR_NAME,
1130 f"{text!r} written without periods is both a "
1131 f"post-nominal and a family-name particle; after a "
1132 f"family comma it joins the family the comma named "
1133 f"rather than standing as a post-nominal",
1134 tuple(run), alternative=suffix_reading(tokens[run[0]])))
1135 elif ambiguous:
1136 word = tokens[ambiguous[0]].text
1137 walked = tokens[run[0]].role
1138 assert walked is not None # the given part's walk roled the run
1139 ambiguities.append(PendingAmbiguity(
1140 AmbiguityKind.PARTICLE_OR_GIVEN,
1141 f"{word!r} is both a family-name particle and an "
1142 f"ordinary given name; after a family comma "
1143 f"{text!r} joins the family the comma named "
1144 f"rather than standing as a name word of its own",
1145 tuple(run),
1146 # where the given part's walk placed it, before this moves
1147 # it: a name word, or a title the walk read off its period
1148 # ('Doe, Jane nee Smith St.'); a name word is taken in by
1149 # the piece in front where that piece would take it
1150 alternative=(TITLE_READING if tokens[run[0]].role is Role.TITLE
1151 else _name_alt(
1152 tokens[ambiguous[0]],
1153 walked,
1154 pieces[k - 1] if k - 1 >= given_at
1155 and tokens[pieces[k - 1][0]].role in NAME_ROLES
1156 else None,
1157 tokens, k - 1 == given_at, True))))
1158 for i in run:
1159 tokens[i] = copy_with(
1160 tokens[i], role=Role.FAMILY,
1161 tags=tokens[i].tags | {FOLDED_TAG, ATTACHED_TAG})
1164def _split_name_alt(state: ParseState, tokens: Sequence[WorkToken],
1165 seg_idx: int, piece: Sequence[int],
1166 order: tuple[Role, Role, Role] | None) -> PendingReading:
1167 """The name reading of a split credential assign placed, every
1168 other word read as the parse read it: the family before a family
1169 comma, the given part's given or middle name after one, and
1170 otherwise the last of the part's name words -- or wherever the name
1171 piece in front of it would take it in (`_name_alt`)."""
1172 names = [q for q in _name_pieces(state, tokens, seg_idx)
1173 if q[0] < piece[0]]
1174 role = _slot_field(state, seg_idx, len(names), len(names) + 1, order)
1175 return _name_alt(tokens[piece[0]], role, names[-1] if names else None,
1176 tokens, len(names) == 1,
1177 state.structure is Structure.FAMILY_COMMA)
1180def _resolve_deferred(state: ParseState, tokens: Sequence[WorkToken],
1181 ambiguities: list[PendingAmbiguity],
1182 order: tuple[Role, Role, Role] | None) -> None:
1183 """Name the field of each alternative classify or group deferred,
1184 from the positions just read: a connective's join takes in the name
1185 word on each side, so where those stand in two pieces the joined
1186 unit takes the place of the first, and where both are inside the
1187 letter's own piece, or the join would leave the name one word, the
1188 letter keeps its own field (named at assemble); a chained particle
1189 stands as a name word of its own in front of its piece, one more
1190 unit. Every other word keeps its reading, so the field is
1191 the one the positional read gives that place: the family before a
1192 family comma, the given part's given or middle name after one, and
1193 the declared order's otherwise. A referent outside every name
1194 position is left deferred, and surfaces as no alternative."""
1195 for r, report in enumerate(ambiguities):
1196 alt = report.alternative
1197 if (alt is None or alt.role not in (Deferred.JOINED,
1198 Deferred.UNCHAINED)
1199 or not report.indices):
1200 continue
1201 i = report.indices[0]
1202 for seg_idx in range(len(state.pieces)):
1203 names = _name_pieces(state, tokens, seg_idx)
1204 at = next((k for k, q in enumerate(names) if i in q), None)
1205 if at is not None:
1206 break
1207 else:
1208 continue
1209 if alt.role is Deferred.UNCHAINED:
1210 pos, count = at, len(names) + 1
1211 else:
1212 # the join takes in the name word on each side: a word of
1213 # the connective's own piece, or the piece beside it. One
1214 # inside its piece joins nothing the piece did not hold, so
1215 # it lands where its piece lands; one opening or ending the
1216 # name has no word to join on that side, and no field
1217 piece = names[at]
1218 k = piece.index(i)
1219 first = at - 1 if k == 0 else at
1220 last = at + 1 if k == len(piece) - 1 else at
1221 if first < 0 or last >= len(names):
1222 continue
1223 pos, count = first, len(names) - (last - first)
1224 # rules/P3: "a single-letter connective in a three-word name"
1225 # stays a name word, so a join that would leave the name one
1226 # word is no join, and the letter keeps its own field; a
1227 # family comma's surname is a word of the name the given
1228 # part's count does not see
1229 if first == last or (
1230 count == 1
1231 and state.structure is not Structure.FAMILY_COMMA):
1232 ambiguities[r] = copy_with(report, alternative=copy_with(
1233 alt, role=Beside(i)))
1234 continue
1235 ambiguities[r] = copy_with(report, alternative=copy_with(
1236 alt, role=_slot_field(state, seg_idx, pos, count, order)))
1239def assign(state: ParseState) -> ParseState:
1240 tokens = list(state.tokens)
1241 ambiguities = list(state.ambiguities)
1242 if not state.segments:
1243 return state
1244 order: tuple[Role, Role, Role] | None = None
1245 if state.structure is Structure.NO_COMMA:
1246 order = _assign_main(0, state, tokens, ambiguities)
1247 tail = len(state.segments)
1248 elif state.structure is Structure.SUFFIX_COMMA:
1249 order = _assign_main(0, state, tokens, ambiguities)
1250 tail = 1
1251 else: # FAMILY_COMMA
1252 # PARTICLE_OR_GIVEN is deliberately not emitted on the
1253 # wholly-family read: after a comma that fixed the family, a
1254 # leading given-position particle is not meaningfully
1255 # ambiguous, and script_orders is not consulted for the parallel
1256 # reason. Scoped, not silent: the comma fixed WHICH PIECE is the
1257 # family and said nothing about a particle trailing the given
1258 # name, so P6's attachment after the given-part walk below
1259 # decides that fork and reports it there, at the site that
1260 # takes the branch (#405). Behind a whole name, a comma followed
1261 # by no name word is not this path since #613: `_comma.decide`
1262 # makes it the suffix comma, read positionally by the branch
1263 # above, which emits and consults both as the comma-less name
1264 # does -- and group's chain emitter with them, the comma being
1265 # decided before the chain runs. Behind one name word it is
1266 # this path, with segment 1 already bound ('Smith, Jr.').
1267 # v1: "lastname part may have suffixes in it" -- the first
1268 # piece is always the family even if suffix-shaped; any later
1269 # strict-suffix piece goes to SUFFIX per piece ('Smith Jr.,
1270 # John' -> family=Smith, suffix=Jr.)
1271 fam_pieces = state.pieces[0]
1272 fam_tags = state.piece_tags[0]
1273 # The part after the comma reaches here as the given part, or
1274 # bound and with no pieces left: group's `_comma.decide` binds a
1275 # postnominal one before any join (#613, rules/C7), so this
1276 # path is the listing form's, and the positional read #296 gave
1277 # a comma followed by no name word behind two name words is
1278 # decide's suffix comma now, read by the NO_COMMA branch above.
1279 # rules/C11's report, scoped to the ambiguous credential
1280 # class: this is the given-name half of the comma's OWN
1281 # decision (listing or credential run), where the writing left
1282 # the fork open. `_comma.decide` reports the credential half,
1283 # where it binds the part, and P6's attachment fork reports
1284 # from `_attach_particle_tail` above ("Berg, Jan vd") -- a
1285 # different fork. Each half is reported where its branch is
1286 # taken, so this DECISION is never reported twice; a second ambiguous
1287 # token elsewhere in the name is a second fork and reports on
1288 # its own (#289, mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE).
1289 #
1290 # The report tracks the FORK BEING CONSULTED, not the lean --
1291 # exactly as the trailing slot has always done (`Jack MA`
1292 # reported before #289 too, even where the pick was declined
1293 # for want of words to spare). So membership alone gates it:
1294 # a caseless script or an all-lower spelling still called this
1295 # fork and read it positionally.
1296 #
1297 # The gate reads EITHER tag (`_AMBIGUOUS_CREDENTIAL_TAGS`, the
1298 # same pair `_group`'s chain emitter asks), and the shape one
1299 # is what reaches a by-shape member -- under EITHER 2.4
1300 # switch, the dotted and the caps alike, since classify writes
1301 # it from both branches. It reaches one with the dotted switch
1302 # OFF as well: classify writes the shape tag whether or not
1303 # the switch admits the token to the class, which is what lets
1304 # a declined fork be REPORTED without being taken ('Smith,
1305 # A.B.' under `unlisted_dotted_suffixes=False` reports and
1306 # keeps its given). An earlier wording said the class reaches
1307 # a by-shape member "once `Policy.unlisted_dotted_suffixes`
1308 # admits it", which is true of the CLASS and false of this
1309 # report.
1310 #
1311 # Read off the FIRST post-comma piece only -- on this path it
1312 # is the given name, `_comma.decide` having read the part and
1313 # found a name word in it, and it is the one piece the lean
1314 # can reach at one word before the comma. So `"Smith, MA PhD"` reports ONCE, for 'MA' alone:
1315 # 'PhD' is settled vocabulary and carries neither tag, and
1316 # even a second CLASS member there would not be read here
1317 # (test_assign.py asserts the count).
1318 #
1319 # The #531 emitter at the far end of this branch counts
1320 # differently, and the difference is the slot rather than a
1321 # second policy: it reports once per member of the trailing
1322 # run it reads, so 'Doe, John MA JD' reports TWICE, matching
1323 # the comma-less 'John Smith MA JD'. A member the writing
1324 # keeps as a name stops that run, which is why
1325 # 'Doe, John MA Ma' reports once and for 'Ma' alone -- 'MA'
1326 # then has a name word behind it and is never asked
1327 # (rules/S5).
1328 if state.pieces[1] and len(state.pieces[1][0]) == 1:
1329 i = state.pieces[1][0][0]
1330 if not tokens[i].tags.isdisjoint(_AMBIGUOUS_CREDENTIAL_TAGS):
1331 # the field is assemble's to word: with nothing before
1332 # the comma and a title after the word, H1 moves it to
1333 # the family (', Ma Dr.', #626's review)
1334 ambiguities.append(PendingAmbiguity(
1335 AmbiguityKind.SUFFIX_OR_NAME,
1336 f"{tokens[i].text!r} after the comma is also an "
1337 f"ordinary name word; read as ", (i,), field_tail="",
1338 # the comma's other branch: the part a credential run
1339 alternative=suffix_reading(tokens[i])))
1340 # Segment 1 is read before segment 0's family pass. It
1341 # consumes piece tags and text only -- nothing segment 0's
1342 # read writes. (A title standing after the comma behind a whole
1343 # name, 'John V, Dr.', is bound at group's head since #613 and
1344 # never reaches this path.)
1345 if len(state.segments) > 1:
1346 pieces = state.pieces[1]
1347 ptags = state.piece_tags[1]
1348 # Both are the walk's, filled once the title run is found.
1349 titled_idx: frozenset[int] = frozenset()
1350 walkable: frozenset[int] = frozenset()
1351 #: Where the trailing suffix run starts, for the #531
1352 #: report below: `trailing_floor`'s answer, read once on
1353 #: the first member the loop meets and -1 until then (no
1354 #: piece index can be negative, and the loop's own floor is
1355 #: n + 1). Once is enough for the same reason one number is
1356 #: enough inside that walk: the question is MONOTONE, the
1357 #: loop below ascends, and a later member can only ask for
1358 #: LESS of the walk than the first one did.
1359 run_floor = -1
1361 def previous_kept(m: int, titled: frozenset[int]) -> int:
1362 """The piece before `m` that the H5 chain did NOT
1363 take. Both readings this segment needs are that one:
1364 the piece the lenient tail test measures against, and
1365 -- asked of one past the end -- where the segment's
1366 name ENDS, since a title the chain took is not where a
1367 name ends (#144). One walk for both, so 'as if the
1368 titled pieces were absent' cannot come to mean two
1369 things.
1371 DEFENSIVE, and measured inert: over 191,146 generated
1372 inputs the skip fired on 153 of the 140,227 walks the
1373 lenient test's `prev` asked for, and deleting it
1374 changed no parse among them (2026-09-09, on the shape
1375 this replaced, where the name's END walked past the
1376 same pieces in a copy of this loop). Kept because "as
1377 if the titled pieces were absent" is the rule the
1378 predicate below implements, and a caller's vocabulary
1379 reaches shapes the sweep's word list does not -- an
1380 inert branch is cheaper than a rule with a hole in it.
1381 """
1382 m -= 1
1383 while m in titled:
1384 m -= 1
1385 return m
1387 #: State of the #531 walk below, per `titled` value: how
1388 #: far down the trailing suffix run has been walked, and
1389 #: whether that walk has SETTLED (it stopped on a piece
1390 #: that refuses, so no lower piece can end the given part
1391 #: either and the refusal is never re-asked).
1392 #:
1393 #: The key carries `titled` for the same reason the
1394 #: predicate takes it as a parameter: the two passes ask
1395 #: about the same pieces with different ones spliced out.
1396 #: It CANNOT go stale. Everything the recorded verdict
1397 #: rests on is tags and text -- `is_suffix_piece` reads
1398 #: ptags and token tags, `_reads_as_a_trailing_suffix`
1399 #: reads text plus `is_suffix_piece`, `listed_lean` reads
1400 #: tags, text and `state.one_case` -- and none of them
1401 #: reads `.role`, verified by reading all three
1402 #: (2026-09-19). The one thing this segment's code rewrites
1403 #: between the two passes is the role, through `_set_roles`,
1404 #: which is a `copy_with(role=...)` and leaves
1405 #: text and tags identical.
1406 floors: dict[frozenset[int], tuple[int, bool]] = {}
1407 #: #544's anchors, per `titled` value like `floors` and for
1408 #: the same reason: `given_slot_anchors` over the pieces the
1409 #: chain kept, computed once, the first time a member's own
1410 #: writing declines, so a run of members is read in one
1411 #: forward pass rather than one look-behind per member.
1412 anchor_memo: dict[frozenset[int], list[bool]] = {}
1414 def anchored(m: int, titled: frozenset[int]) -> bool:
1415 memo = anchor_memo.get(titled)
1416 if memo is None:
1417 # nothing in front the pass could read as an
1418 # anchor: the ordinary 'Smith, John Ma' stops here.
1419 # Asked only before the pass exists -- once it
1420 # does it answers in one lookup, where the reach
1421 # test walks back through the run
1422 if not anchor_in_reach(range(m - 1, -1, -1), pieces,
1423 ptags, tokens, titled):
1424 return False
1425 # from past the leading title run, which `n` already
1426 # counts: this closure is reached only from the walk
1427 # below `_peel_leading_titles` sets it on
1428 memo = given_slot_anchors(pieces, ptags, tokens, n,
1429 skip=titled)
1430 anchor_memo[titled] = memo
1431 return memo[m]
1433 def trailing_floor(m: int, titled: frozenset[int]) -> int:
1434 """Where the trailing suffix run starts, walked as far
1435 down as `m` needs it: `m >= trailing_floor(m, titled)`
1436 is exactly "every kept piece behind `m` reads as a
1437 suffix", which is what ENDING the given part means
1438 (rules/S5, #531).
1440 ONE walk per `titled` value, shared by the predicate
1441 below and by that rule's report at the foot of this
1442 segment, and the reason both are LINEAR in the run's
1443 length. The question is MONOTONE -- a piece ends the
1444 given part whenever the piece behind it does -- so a
1445 single descent answers for every member, each piece
1446 read at most once. Asked member by member instead, the
1447 reading is recursive (a member ends the given part iff
1448 everything kept behind it reads as a suffix, and a
1449 piece behind it is a member asking the same of its own
1450 tail): a walk per member, which cost O(run**2) with the
1451 per-member memo this replaced and 2**run without one.
1452 Measured on `'Doe, John ' + 'MA '*k`, k doubling from
1453 200: 2.1/4.1/8.3/17.1ms here, against 8.6/31.4/120/463
1454 with the memo and 1.5/3.2/7.2/17.3 at cc78c960, where
1455 no such walk existed at all (2026-09-19).
1457 `low` descends only to `m`, so what comes back is a
1458 floor FOR `m` rather than the run's own first piece
1459 whenever the run reaches past it; that is all either
1460 caller asks, and stopping there is what keeps the
1461 member's own frame count where it was. `final` carries
1462 the other half: a piece that refuses settles the floor
1463 for everything in front of it.
1465 Re-entrant by construction, and it has to be: the
1466 descent asks the predicate below about a piece that is
1467 itself often a member, which asks this back. `floors`
1468 names the piece under test BEFORE that call, so the
1469 re-entrant reading is "this piece ends the given part",
1470 which is what the descent has just established of it.
1471 """
1472 entry = floors.get(titled)
1473 if entry is None:
1474 # previous_kept() of one past the end, spelled out
1475 # here rather than called: the frame budget again,
1476 # this walk being asked of every family-comma name
1477 # with a member in the given part, and the skip is
1478 # two lines. Keep the two in step.
1479 low = len(pieces) - 1
1480 while low in titled:
1481 low -= 1
1482 entry = (low, False)
1483 floors[titled] = entry
1484 low, final = entry
1485 while not final and low > m:
1486 if reads_as_a_suffix(low, titled):
1487 low -= 1
1488 while low in titled:
1489 low -= 1
1490 else:
1491 final = True
1492 floors[titled] = (low, final)
1493 return low
1495 #: where the particle run opening the given part ends,
1496 #: per `titled` value -- `attaches_the_lead`'s one walk
1497 lead_run_ends: dict[frozenset[int], int] = {}
1499 def attaches_the_lead(m: int, titled: frozenset[int]) -> bool:
1500 """Is piece `m` the name word a never-given particle
1501 opening the given part attaches to? post_rules' P1
1502 fold takes such a particle forward into the family and
1503 needs a name word for it to attach to, so the given
1504 name the #531 slot below counts on is not there, and
1505 the first word past the particle run is the surname it
1506 heads rather than a word to spare ('SMITH, VD MA'
1507 reads as 'Smith, vd Ma' does, #573). Only where the
1508 member's writing says nothing: capitals in a mixed-case
1509 name still make it the credential ('Smith, de MA'
1510 keeps suffix 'MA'), as they do with no words to spare
1511 anywhere (rules/S5). Asked only of a class member
1512 the slot would otherwise take."""
1513 if not is_lone_never_given_particle(pieces[n], tokens):
1514 return False
1515 if listed_lean(tokens[pieces[m][0]],
1516 state.one_case) == "credential":
1517 return False
1518 # The member is the particle run's word when it is the
1519 # first kept piece past that run. Where the run ends is
1520 # one answer per `titled` value, walked once and kept,
1521 # so a run of members costs one walk rather than one
1522 # each (the second review measured the per-member scan
1523 # quadratic on 'SMITH, VD ' + 'DE '*k + 'MA '*k). Every
1524 # token, not the piece's tags: a chain of particles
1525 # ('DE LA') is one piece whose tags say nothing of it
1526 # ('SMITH, VD DE LA MA').
1527 run_end = lead_run_ends.get(titled)
1528 if run_end is None:
1529 run_end = n + 1
1530 while run_end < len(pieces) and (
1531 run_end in titled
1532 or all("particle" in tokens[i].tags
1533 for i in pieces[run_end])):
1534 run_end += 1
1535 lead_run_ends[titled] = run_end
1536 return m <= run_end
1538 def reads_as_a_suffix(m: int, titled: frozenset[int]) -> bool:
1539 """Does this segment's walk read piece `m` as a suffix?
1541 Asked twice, and by one predicate rather than by two
1542 conditions written to match
1543 (mechanisms.md#ONE-PREDICATE-PER-QUESTION): once to
1544 find the pieces the H5 title chain must not reach
1545 past, and once by the walk order below, which is the
1546 site that places them.
1548 `titled` is a PARAMETER because the two passes hand it
1549 different values -- an empty set on the first, the
1550 chain's own pieces on the second, which is what makes
1551 that second reading the one 'as if the titled pieces
1552 were absent'.
1553 A closure over the caller's local said the same thing,
1554 but only by WHEN it was rebound.
1555 """
1556 if is_suffix_piece(pieces[m], ptags[m], tokens):
1557 return True
1558 # rules/S10: "A numeral of two or more letters reads as
1559 # the generation" in the given part, which asks no
1560 # initial in front -- 'Smith, John VI' as 'Smith, John
1561 # III' (#752); `_ROMAN` first, C-level
1562 if (len(pieces[m]) == 1
1563 and len(stem := (tok := tokens[pieces[m][0]]).text
1564 .rstrip(".")) >= 2
1565 and _ROMAN.match(stem) is not None):
1566 return reads_as_generation(tok.text, tok.tags, None,
1567 state.one_case)
1568 # rules/S5: "a word that ENDS that part reads as the credential
1569 # unless its WRITING says otherwise" -- the given part's
1570 # trailing slot (#531).
1571 # INLINE, and that is the frame budget talking rather
1572 # than taste: the candidates pass asks this closure once
1573 # per piece of every family-comma segment, so a helper
1574 # call would cost a frame on every non-member piece and
1575 # a generator expression would cost its own on 3.11.
1576 # Membership is therefore a bare `in` on tags already
1577 # in hand, after a `len` -- 'Doe, John Q.' and
1578 # 'Smith, John V' measure +0 with this shape and +1
1579 # with a helper (2026-09-18).
1580 #
1581 # That "once per piece" is the NON-MEMBER cost, and
1582 # only it. A member is asked a second time by
1583 # trailing_floor()'s descent above, and twice is the
1584 # whole of it: the descent takes each piece once and
1585 # the candidates pass asks each piece once, which is what
1586 # makes the run linear. What the slot costs, measured
1587 # against cc78c960: 'Smith, John', 'Doe, John Q.',
1588 # 'Smith, John V', 'Smith, MA' and 'Berg, Jan vd' are
1589 # all +0 frames, while 'Doe, John MA' is +6 and 'Doe,
1590 # John MA PhD' +23 -- the member's descent, its lean,
1591 # and the report below (2026-09-19).
1592 #
1593 # The comma has already named the family and the first
1594 # piece after it is the given name, so the words to
1595 # spare S4's count asks about are there by
1596 # construction and the count says nothing at this
1597 # slot. What is left is the writing, which is the same
1598 # evidence the comma-less spelling of the same name
1599 # reads. #144's two-segment restriction below is NOT
1600 # inherited: it exists because a trailing 'V' before a
1601 # third comma part is likely a middle initial, and a
1602 # class member is not initial-shaped while a
1603 # credential list behind it makes the credential
1604 # reading likelier rather than less.
1605 piece = pieces[m]
1606 # rules/S12: a split credential reads "as a member of the
1607 # ambiguous class", here read as the class is with no
1608 # writing to lean on and in silence, as group merges it
1609 # only behind the given word
1610 if len(piece) == 2 and "split" in ptags[m]:
1611 return (m >= trailing_floor(m, titled)
1612 and not attaches_the_lead(m, titled))
1613 if len(piece) == 1:
1614 tok = tokens[piece[0]]
1615 if AMBIGUOUS_ACRONYM_TAG in tok.tags:
1616 # 'ending the given part' reaches past the
1617 # credentials behind it and past a trailing
1618 # title, which trailing_floor() skips the way
1619 # previous_kept() does -- so 'Doe, John MA
1620 # Prof.' and 'Doe, John Prof. MA' land on one
1621 # answer without a second notion of trailing.
1622 # A name word behind the member ends the
1623 # reach, and the member is an ordinary middle
1624 # name read in silence.
1625 if (m >= trailing_floor(m, titled)
1626 and not attaches_the_lead(m, titled)):
1627 # A member that is ALSO particle
1628 # vocabulary reads as the credential only
1629 # on a POSITIVE credential lean: P6's
1630 # attachment outranks this reading in
1631 # every other spelling, and 'Doe, John do'
1632 # leans nothing, so the positional reading
1633 # would take it -- the wrong answer there,
1634 # not merely a stray report
1635 # (decisions/S5, 2026-09-18).
1636 #
1637 # A FUNCTION since #533 rather than a
1638 # condition written here (mechanisms.md
1639 # #ONE-PREDICATE-PER-QUESTION). The call
1640 # costs one frame PER MEMBER asked at this
1641 # slot, not one per name: against
1642 # 2f57ff21, 'Doe, John MA' is 310 -> 311
1643 # and 'Doe, John MA Ma MA', which asks
1644 # four times, 439 -> 443, while a name
1645 # with no member here never reaches it
1646 # ('Smith, John' 206, 'MA JD' 185, both
1647 # unchanged). Measured 2026-09-19 per
1648 # `Parser.parse`; Derek took that trade
1649 # deliberately.
1650 # #544: an unambiguous credential in front
1651 # of it in the same run anchors it -- asked
1652 # through a thunk, so only a member the
1653 # writing declines pays for the pass
1654 if credential_at_the_given_slot(
1655 tok, state.one_case,
1656 lambda: anchored(m, titled)):
1657 return True
1658 prev = previous_kept(m, titled)
1659 # trailing piece of a two-part name is unambiguously
1660 # positioned: v1 accepts the lenient test there
1661 # ('Smith, John V' -> suffix='V', #144); with a third
1662 # comma part the trailing token is more likely a middle
1663 # initial, so strict only
1664 return (m == previous_kept(len(pieces), titled)
1665 and len(state.segments) == 2
1666 and len(pieces[m]) == 1
1667 and _reads_as_a_trailing_suffix(
1668 pieces[m], pieces[prev], ptags[prev],
1669 tokens, state.lexicon, state.one_case))
1671 # rules/C1: "anything else after the comma means the
1672 # listing form" -- and this walk reads it. A segment of no
1673 # name word never reaches here: it is `_comma.decide`'s, read piece by piece and bound at
1674 # group's head (#613, rules/C8), the suffix verdict
1675 # before the title reading of the same word (#296, #325).
1676 # A name word in the segment makes it v1's walk ('Smith,
1677 # John Jr.').
1678 # rules/H9: "no comma has marked the given name" -- here the
1679 # family comma has, so the peel reports nothing
1680 n = _peel_leading_titles(pieces, ptags, tokens, None,
1681 state.lexicon)
1682 # rules/H5: "the title is TRANSPARENT to the suffix
1683 # reading: where two or more name words stand, what
1684 # stands once the chain is taken reads exactly as it
1685 # would read written without the title, plus the title"
1686 # -- the trailing title run, on this walk
1687 # too. A name word in segment 1 is what keeps
1688 # `_comma.decide` from reading the segment as the
1689 # postnominal part, so 'Smith, John Prof.' had no route
1690 # to title at all and read middle 'Prof.' at every
1691 # baseline; the 'Smith, Dr.' family of rows are decide's
1692 # doing, a different mechanism.
1693 #
1694 # The candidates are the pieces this walk would NOT
1695 # read as a suffix, which is this path's answer to the
1696 # peel the no-comma path runs first -- 'Smith, John
1697 # Prof. Jr.' must reach `Prof.` past the postnominal
1698 # behind it, and 'Smith, John Prof. V' past the
1699 # numeral the lenient tail test claims (#144), which
1700 # is why the filter is the walk's own predicate and
1701 # not the strict suffix test alone. Piece `n` is
1702 # always the given below, whatever that predicate
1703 # would say of it, so it is always a candidate. That
1704 # `k == n` is LOAD-BEARING, not defensive: it is what
1705 # the walk's floor stands on when the given piece
1706 # itself reads as a suffix, and dropping it leaves
1707 # 'Smith, II Mr. V' a middle 'Mr.' where the title is
1708 # (24 inputs of that shape move, of 191,146 generated,
1709 # measured 2026-09-09).
1710 #
1711 # `candidates` is the ORDER, which `trailing_titles` and
1712 # its slice read; everything after asks MEMBERSHIP,
1713 # once per piece, and a list or tuple answers that by
1714 # scanning, so the walk was quadratic in the part's
1715 # length (#553). Hence the two sets. A tuple keying
1716 # `floors` or `anchor_memo` above would also re-hash
1717 # its whole length at every lookup, where a frozenset
1718 # caches its hash.
1719 candidates = [k for k in range(n, len(pieces))
1720 if k == n
1721 or not reads_as_a_suffix(k, frozenset())]
1722 # rules/H8: "the word joins the title field, and the given
1723 # part reads as it would without it" -- a bare salutation
1724 # chains here as a period-marked title does
1725 kept = trailing_titles(candidates, pieces, ptags, tokens,
1726 salutations=True)
1727 walkable = frozenset(candidates)
1728 titled_idx = frozenset(candidates[kept:])
1729 for k in titled_idx:
1730 _set_roles(tokens, pieces[k], Role.TITLE)
1731 # v1 walk order: the first non-title piece is ALWAYS the
1732 # given, before any suffix check -- 'Smith, V. Jones' keeps
1733 # first='V.'. Two 2.x deviations are `_comma.decide`'s
1734 # reading now (#613), so the walk here never meets
1735 # either: a last piece
1736 # that is unambiguously suffix-shaped is a suffix, where v1
1737 # made it the given ('Andrews, M.D.', 'Smith, Dr. Jr.';
1738 # classified fix(comma-family)), a segment whose only
1739 # non-title piece is a suffix piece holding no name word;
1740 # and a credential opening the segment makes it the
1741 # postnominal part whatever follows ('Hardman, RN - CRNA'
1742 # kept first='RN' until #603).
1743 if n < len(pieces):
1744 _set_roles(tokens, pieces[n], Role.GIVEN)
1745 # The chain's floor leaves a name piece standing, so the
1746 # given above is never one of the pieces it took. What the
1747 # chain DOES move is where this segment's name ends, which
1748 # the lenient tail test turns on (#144) -- so where it took
1749 # something the question is re-asked with the pieces it
1750 # left. Where it took nothing, `walkable` already IS this
1751 # predicate's answer for every piece: the first pass's own
1752 # memo, not a second spelling of the question, and it
1753 # cannot have gone stale because nothing the chain does
1754 # moved the end of the name.
1755 # rules/S8: "or after the given word in the part after a
1756 # family comma" -- the given part's own credential run
1757 # (#602), the comma having fixed the family. The tag tests are inline ahead of the
1758 # predicate, as in `_pieces.credential_run`, so a name word
1759 # pays no frame: this loop runs on every family-comma name.
1760 sticky_from = len(pieces)
1761 for m in range(n + 1, len(pieces)):
1762 tags = tokens[pieces[m][0]].tags
1763 if (m not in titled_idx
1764 and ("suffix" in ptags[m] or "split" in ptags[m]
1765 or (("vocab:suffix" in tags
1766 or AMBIGUOUS_ACRONYM_TAG in tags
1767 or _ROMAN.match(tokens[pieces[m][0]].text
1768 .rstrip(".")) is not None)
1769 and tags.isdisjoint(_NOT_A_RUN_START)))
1770 and starts_a_credential_run(pieces[m], ptags[m],
1771 tokens, None,
1772 state.one_case)):
1773 sticky_from = m
1774 break
1775 # rules/P6: "a particle ending the name attaches to that
1776 # family name" -- decided here, in the pass that reads the
1777 # given part, rather than predicted for post_rules to take
1778 # a stage later (#613). The tail is `pieces[k6:end6]`, found
1779 # by `particle_tail` ONCE, at the moment the walk reaches
1780 # #602's run (or, with no run, once the walk has placed
1781 # every piece): the pieces in front of the run hold the roles
1782 # this walk gave them and the run's own words hold none
1783 # yet, which is the state in which the walk's answer equals
1784 # the one it would give over the finished roles -- the run
1785 # reads every word it keeps as a suffix or a title, neither
1786 # a name role, and a class member in it as the credential,
1787 # where the walk's #531 stop wants it. The tail's words are
1788 # left to the walk, not the run: absorbing them reported a
1789 # suffix reading the attachment then overrode ('Smith, John
1790 # PhD de', 'Smith, John PhD de Jr.'). A particle the walk
1791 # does not end on -- one with a credential behind it and
1792 # another particle past that, 'Smith, John PhD de PhD van'
1793 # -- stays in the run, as does a lone member of the
1794 # ambiguous credential class (`do`), which the run reads as
1795 # the credential.
1796 k6 = end6 = len(pieces)
1798 def as_a_middle(m: int, token: WorkToken) -> PendingReading:
1799 # the name reading of a word the walk read as a suffix:
1800 # a middle name, taken in by the name word in front, or
1801 # the family where P1's fold opens the part
1802 front = pieces[m - 1]
1803 # P1's fold after a family comma takes the rest of the
1804 # part into the family whatever the order, so a word past
1805 # the opening particle's own piece lands there; the word
1806 # directly behind it is `_name_alt`'s, which takes it into
1807 # that piece; decisions/C9 (#877)
1808 if n < m - 1 and is_lone_never_given_particle(pieces[n],
1809 tokens):
1810 return name_reading(token, Role.FAMILY)
1811 return _name_alt(
1812 token, Role.MIDDLE,
1813 front if tokens[front[0]].role in NAME_ROLES else None,
1814 tokens, m - 1 == n, True)
1816 for m in range(n + 1, len(pieces)):
1817 if m == sticky_from:
1818 k6, end6 = particle_tail(pieces, tokens)
1819 if m in titled_idx:
1820 continue
1821 suffix_here = (reads_as_a_suffix(m, titled_idx)
1822 if titled_idx else m not in walkable)
1823 # rules/S10: "A numeral read by its shape alone reports
1824 # too" -- one of two or more letters the walk reads as
1825 # the generation; `_ROMAN` first, C-level
1826 if (suffix_here and len(pieces[m]) == 1
1827 and len(stem := (tok := tokens[pieces[m][0]]).text
1828 .rstrip(".")) >= 2
1829 and _ROMAN.match(stem) is not None
1830 and not is_listed_numeral(tok.text, tok.tags)
1831 and not numeral_settled_by_case(tok.text,
1832 state.one_case)):
1833 ambiguities.append(PendingAmbiguity(
1834 AmbiguityKind.SUFFIX_OR_NAME,
1835 f"{tok.text!r} is a roman numeral, so it reads as "
1836 f"a generational suffix; it may be a name word",
1837 (pieces[m][0],),
1838 # a name word the walk keeps past the given
1839 # name is a middle one
1840 alternative=as_a_middle(m, tok)))
1841 # rules/C9: "a lone letter ending the given part ahead of
1842 # a further comma is the middle initial, whatever stands
1843 # before it" -- the further comma has named the suffix,
1844 # so the credential run in front stops short of it
1845 # (#819); `len` first, C-level
1846 if (m > sticky_from and len(state.segments) > 2
1847 and len(pieces[m]) == 1
1848 and len(tokens[pieces[m][0]].text.rstrip(".")) == 1
1849 and m == previous_kept(len(pieces), titled_idx)
1850 and is_initial_shaped(
1851 tokens[pieces[m][0]].text.upper())):
1852 _set_roles(tokens, pieces[m], Role.MIDDLE)
1853 continue
1854 # rules/C9: "The bare letter read as the generation behind
1855 # a name word is reported" -- the shape a middle initial
1856 # is written in, as S10 reports it with no comma;
1857 # decisions/C9 (#877); `len` first, C-level
1858 if (len(pieces[m]) == 1
1859 and len((tok := tokens[pieces[m][0]]).text) == 1
1860 and (m >= sticky_from and not k6 <= m < end6
1861 or m < sticky_from and suffix_here)
1862 and _ROMAN.match(tok.text) is not None
1863 and tokens[pieces[m - 1][0]].role in NAME_ROLES
1864 # a particle in front may take the letter into
1865 # its chain or stand alone, the field the name
1866 # reading names depending on which; decisions/C9
1867 # (#877)
1868 and not is_prefix_piece(pieces[m - 1], ptags[m - 1],
1869 tokens)
1870 and not numeral_settled_by_case(tok.text,
1871 state.one_case)):
1872 ambiguities.append(PendingAmbiguity(
1873 AmbiguityKind.SUFFIX_OR_NAME,
1874 f"{tok.text!r} is a roman numeral, so it reads as a "
1875 f"generational suffix; any other single letter "
1876 f"there would be a middle initial",
1877 (pieces[m][0],), alternative=as_a_middle(m, tok)))
1878 if m >= sticky_from and not k6 <= m < end6:
1879 # inside the run: a title word reads as a title,
1880 # every other word as a suffix, and a word the walk
1881 # would have kept as a name is reported, here where
1882 # the run absorbs it
1883 if (m > sticky_from
1884 and not is_suffix_piece(pieces[m], ptags[m],
1885 tokens)
1886 and is_title_piece(pieces[m], ptags[m],
1887 tokens)):
1888 _set_roles(tokens, pieces[m], Role.TITLE)
1889 continue
1890 piece = pieces[m]
1891 if (len(piece) == 1
1892 and not tokens[piece[0]].tags.isdisjoint(
1893 _AMBIGUOUS_CREDENTIAL_TAGS)):
1894 # a class member inside the run is still the
1895 # fork #531's report names, now read as the
1896 # credential, as #544's anchor reads one --
1897 # unless its capitals settled it (rules/S6)
1898 if not settled_by_case(tokens[piece[0]],
1899 state.one_case):
1900 ambiguities.append(PendingAmbiguity(
1901 AmbiguityKind.SUFFIX_OR_NAME,
1902 f"{tokens[piece[0]].text!r} ending the given "
1903 f"part is also an ordinary name word; read "
1904 f"as a credential", (piece[0],),
1905 alternative=as_a_middle(
1906 m, tokens[piece[0]])))
1907 # the run's own first word is the credential that
1908 # starts it, a split one ('Pharm D') included
1909 elif (not suffix_here and m != sticky_from
1910 and has_name_content(piece, tokens)):
1911 ambiguities.append(
1912 _absorbed(piece, tokens,
1913 as_a_middle(m, tokens[piece[0]])))
1914 _set_roles(tokens, piece, Role.SUFFIX)
1915 continue
1916 _set_roles(tokens, pieces[m],
1917 Role.SUFFIX if suffix_here else Role.MIDDLE)
1918 # #531's report, one of this module's SUFFIX_OR_NAME
1919 # sites (the header lists them). The gate reads EITHER tag, as the
1920 # first-post-comma emitter's does: classify writes the
1921 # shape tag whether or not the dotted switch admits
1922 # the token, which is what lets a declined fork be
1923 # reported without being taken.
1924 #
1925 # The report tracks the FORK CONSULTED, not the lean
1926 # (#289's rule), so a member the writing kept as a
1927 # name reports too -- 'Doe, John Ma' stays a middle
1928 # name and says so. It reports only in the TRAILING
1929 # RUN: with a name word behind it no fork was
1930 # consulted, and AGENTS.md's "a kind is worth adding
1931 # only if a reader would hesitate too" is why that
1932 # must stay silent rather than why it happens to (the
1933 # sentence is the 2.0-conventions section's, not
1934 # rules/A1's, which this cited until 2026-09-19).
1935 #
1936 # The `do` carve-out is a REPORT carve-out on top of
1937 # the reading one above: a particle-tagged member this
1938 # walk did not itself take is P6's fork, and P6
1939 # reports it in its own kind, which is
1940 # mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE read
1941 # strictly. Without it 'Doe, John do' reported both
1942 # kinds.
1943 piece = pieces[m]
1944 if (len(piece) == 1
1945 and not tokens[piece[0]].tags.isdisjoint(
1946 _AMBIGUOUS_CREDENTIAL_TAGS)
1947 and (suffix_here
1948 or "particle" not in tokens[piece[0]].tags)
1949 # rules/S6: "a clash settled by the writing is
1950 # not reported"
1951 and not (suffix_here and settled_by_case(
1952 tokens[piece[0]], state.one_case))):
1953 # The SAME floor the predicate measures members
1954 # against, so this report has no walk of its own to
1955 # regress: it had one, and on a run the predicate's
1956 # memo had already made quadratic the report was
1957 # CUBIC, because its per-member walk re-scanned
1958 # `walkable` -- a list -- at every step. Measured
1959 # then on `'Doe, ' + 'John '*r + 'MA '*r`, r
1960 # doubling from 100: 5.8/28/175/1229ms, 4.9x then
1961 # 6.2x then 7.0x per doubling and heading for the
1962 # 8x a cubic gives, against 1.8/3.7/8.0ms here
1963 # (2026-09-19). Nothing guarded it: the scan was a
1964 # C-level `in` over a list and emitted no frame, so
1965 # the frame-ratio test in tests/v2/test_benchmark.py
1966 # was structurally blind to it, and the clock-based
1967 # shapes beside it repeat ONE unit where this cost
1968 # needs a name holding two runs (`_PREFIXED_SHAPES`
1969 # adds a prefix but still repeats one run, so it
1970 # does not reach this either). What keeps it gone
1971 # is the structure: one walk, read by both callers,
1972 # so a second would have to be written on purpose.
1973 #
1974 # Read once for the whole loop, `run_floor` being
1975 # where the FIRST member's walk stopped and the
1976 # floor for every later member too: if the walk
1977 # reached that member, nothing behind it refuses
1978 # and no later member can be refused either; if it
1979 # stopped short, it stopped at the LAST piece that
1980 # refuses, which is exactly what a later member's
1981 # own walk would have found (2026-09-19).
1982 if run_floor < 0:
1983 run_floor = trailing_floor(m, titled_idx)
1984 if m >= run_floor:
1985 i2 = piece[0]
1986 ambiguities.append(PendingAmbiguity(
1987 AmbiguityKind.SUFFIX_OR_NAME,
1988 f"{tokens[i2].text!r} ending the given "
1989 f"part is also an ordinary name word; "
1990 f"read as "
1991 f"{'a credential' if suffix_here else 'a name'}",
1992 (i2,),
1993 alternative=(
1994 as_a_middle(m, tokens[i2])
1995 if suffix_here
1996 else suffix_reading(tokens[i2]))))
1997 if sticky_from == len(pieces):
1998 k6, end6 = particle_tail(pieces, tokens)
1999 # Behind every family comma, one with nothing before it
2000 # too, which then makes the particle the whole family
2001 # (', Dr. Jones vd'; decisions/C15, #892). GIVEN ahead of
2002 # the tail -- rules/P6: "at least one given word must
2003 # remain" -- the given is piece `n`, so a tail starting
2004 # there leaves none ('Nguyen, Van').
2005 # And not where a never-given particle opens the part: P1's
2006 # fold takes every given and middle word into the family in
2007 # post_rules, so no given word remains for this one either
2008 # ('Smith, de Mesnil van' keeps 'van' where the fold puts
2009 # it, not hoisted in front of 'Smith', the 2026-08-18
2010 # defect); with no name word behind it to fold, the tail
2011 # stays the suffix, behind an empty head as behind its twin
2012 # (', de PhD vd' as 'Smith, de PhD vd', #892).
2013 if (n < k6 < end6
2014 and not is_lone_never_given_particle(pieces[n],
2015 tokens)):
2016 _attach_particle_tail(pieces, tokens, n, k6, end6,
2017 ambiguities)
2018 # rules/H6: "A run of title words opening the part before a
2019 # family comma is the title" -- by vocabulary only, never H2's
2020 # shape, leaving one piece for the family, and stopping at a
2021 # word that is a particle too ('St Pierre', 'Freiherr von
2022 # Berg'), or where everything behind it is post-nominal, which
2023 # leaves no family word to address -- H3's concern, by a
2024 # stricter test than H3's one-word give-back, so this is not
2025 # `leading_titles` ('RA PhD Jr., John' keeps family 'RA', where
2026 # 'RA PhD Jr. John' reads title 'RA'). A dual is a title wherever
2027 # a name word follows it ('Rev. Sr. Smith, Mary').
2028 # `is_title_piece` is H5's vocabulary read, so the two slots that
2029 # refuse the shape agree on what a title word is, save H5's
2030 # written-short clause, which this slot does not read
2031 # (decisions/H9, #999: 'Maj. Smith, John' keeps family 'Maj. Smith').
2032 #
2033 # rules/H10: "where a title the run keeps stands behind it" --
2034 # a dotted run ('U.S. Senator Smith, John') counts as a title
2035 # word here as in the leading run, and one the run ends on is
2036 # given back ('U.S. Grant Smith, John' keeps family 'U.S. Grant
2037 # Smith'), so the roles are set once the run is known.
2038 lead = 0
2039 floor = len(fam_pieces)
2040 # the length test and the tag test first, both C-level: a part
2041 # no title opens ('Smith, John', 'Smith Jones, John') pays no
2042 # frame. "title" and "vocab:title" are the two arms
2043 # `is_title_piece` answers yes from, so for a listed title the
2044 # predicate stays the one that answers; a lone piece carrying
2045 # classify's DOTTED_TITLE_TAG is taken on the tag alone
2046 # (rules/H10) and given back below unless a title the run keeps
2047 # stands behind it
2048 tags0 = tokens[fam_pieces[0][0]].tags if fam_pieces else frozenset()
2049 if (floor > 1
2050 and ("title" in fam_tags[0] or "vocab:title" in tags0
2051 or DOTTED_TITLE_TAG in tags0)
2052 and ((len(fam_pieces[0]) == 1 and DOTTED_TITLE_TAG in tags0)
2053 or is_title_piece(fam_pieces[0], fam_tags[0], tokens))):
2054 # where the trailing post-nominal run starts: walked once,
2055 # and only for a part a title opens
2056 while floor > 1 and is_suffix_piece(fam_pieces[floor - 1],
2057 fam_tags[floor - 1],
2058 tokens):
2059 floor -= 1
2060 # the first piece is a title or a dotted run, asked above; a
2061 # dotted run the run ends on is given back after the walk
2062 while (lead + 1 < floor
2063 and (lead == 0 or is_title_piece(
2064 fam_pieces[lead], fam_tags[lead], tokens)
2065 or (len(fam_pieces[lead]) == 1 and DOTTED_TITLE_TAG
2066 in tokens[fam_pieces[lead][0]].tags))
2067 and "particle" not in tokens[fam_pieces[lead][0]].tags):
2068 lead += 1
2069 while (lead and len(fam_pieces[lead - 1]) == 1
2070 and DOTTED_TITLE_TAG
2071 in tokens[fam_pieces[lead - 1][0]].tags):
2072 lead -= 1
2073 for k in range(lead):
2074 _set_roles(tokens, fam_pieces[k], Role.TITLE)
2075 # rules/P9: "a particle that is also suffix vocabulary heads
2076 # the name word behind it" -- with a name piece behind it in
2077 # this part (vd, mc), it heads that name, as P2's join does,
2078 # and is not peeled from between two family words
2079 # (#573: 'SMITH VD MA, JOHN', where the uniform case left the
2080 # chain stopped before VD; 'SMITH VD JR, JOHN', a suffix word
2081 # behind it, keeps suffix 'VD JR'). Read here because only
2082 # this family pass needs it: a comma followed by no name
2083 # word behind a whole name is group's suffix comma since #613,
2084 # and the NO_COMMA read takes segment 0, trailing run and all
2085 # ('Berg de MA, Prof.'). The first piece the title run leaves
2086 # is the family's first and is family whatever it is, as piece
2087 # 0 is with no title ('Dr. PhD Smith, John' as 'PhD Smith, John').
2088 # A piece at or past the floor is already known to be a suffix
2089 # with nothing but suffixes behind it, so it is not asked again.
2091 # Whether a particle read as family stands in front of the
2092 # piece the loop is at, and whether the piece right in front was
2093 # read as family, for P2's arm below.
2094 chained = False
2095 family_before = False
2096 for k in range(lead, len(fam_pieces)):
2097 piece = fam_pieces[k]
2098 if k >= floor:
2099 _set_roles(tokens, piece, Role.SUFFIX)
2100 # rules/C1: "a numeral ending the family part is read as the
2101 # comma-less name reads it" -- the numeral's predicate, a
2102 # one-letter numeral with two name words in front of it
2103 # (rules/S10; name words only, `_names_before`), so 'John
2104 # Smith V, M.J.' keeps suffix 'V' as 'John Smith III, M.J.'
2105 # keeps 'III' (#774). `_ROMAN`
2106 # first, C-level, so a family word pays no frame
2107 elif (k > lead and len(piece) == 1
2108 and _ROMAN.match(
2109 (word := tokens[piece[0]]).text.rstrip("."))
2110 is not None
2111 and all(is_suffix_piece(fam_pieces[j], fam_tags[j],
2112 tokens)
2113 for j in range(k + 1, len(fam_pieces)))
2114 and (numeral := _family_part_numeral(
2115 word, tokens[fam_pieces[k - 1][0]].text,
2116 _names_before(fam_pieces, tokens, lead, k),
2117 state.one_case)) is not None):
2118 _set_roles(tokens, piece, numeral)
2119 # reported where taken: a name word here is the family
2120 # name the comma has just named, which nobody misreads
2121 if (numeral is Role.SUFFIX
2122 and not is_listed_numeral(word.text, word.tags)
2123 and not numeral_settled_by_case(word.text,
2124 state.one_case)):
2125 ambiguities.append(PendingAmbiguity(
2126 AmbiguityKind.SUFFIX_OR_NAME,
2127 f"{word.text!r} ending the family part is a roman "
2128 f"numeral, so it reads as a generational suffix; "
2129 f"it may be a name word", tuple(piece),
2130 alternative=name_reading(word, Role.FAMILY)))
2131 elif (k > lead and is_suffix_piece(piece, fam_tags[k], tokens)
2132 and not (k + 1 < len(fam_pieces)
2133 and "particle" in tokens[piece[0]].tags
2134 and not is_suffix_piece(
2135 fam_pieces[k + 1], fam_tags[k + 1],
2136 tokens))
2137 # rules/P2: "before a family comma the part keeps
2138 # either in the family behind a particle, as the
2139 # chain does" -- a suffix word S8 starts no run at
2140 # or declines for its writing, anything but
2141 # suffixes behind it ('bin Abd Rahman, Ahmad',
2142 # 'bin Md Rahman, Ahmad'); a particle that is suffix
2143 # vocabulary too is P9's (decisions/P2, #1177)
2144 and not (chained and family_before
2145 and "particle" not in tokens[piece[0]].tags
2146 and not all(
2147 is_suffix_piece(fam_pieces[j], fam_tags[j],
2148 tokens)
2149 for j in range(k + 1, len(fam_pieces)))
2150 and not starts_the_run_here(
2151 k, fam_pieces, fam_tags, tokens,
2152 state.one_case, False))):
2153 _set_roles(tokens, piece, Role.SUFFIX)
2154 else:
2155 _set_roles(tokens, piece, Role.FAMILY)
2156 # the tag tests are is_prefix_piece's own arms, inline
2157 # so a family word pays no frame
2158 chained = chained or ("prefix" in fam_tags[k]
2159 or len(piece) == 1
2160 and "particle"
2161 in tokens[piece[0]].tags)
2162 family_before = tokens[piece[0]].role is Role.FAMILY
2163 tail = 2
2164 # rules/C2: segments past the structure's name segments are
2165 # consumed as suffixes, except "a word of the title vocabulary there
2166 # that is not also suffix vocabulary is a title" (#603) -- a part
2167 # past the second comma holds no name word, so the surnames H5
2168 # keeps a bare trailing title word from misreading are not there to
2169 # protect ('Eric H. Holder, Jr., Attorney General'). A
2170 # piece holding a suffix word stays the postnominal the slot makes
2171 # it, a dual alone as after a family comma (C8) and a dual group's
2172 # connective join took into a title ('PhD - and MD') alike.
2173 for seg_idx in range(tail, len(state.segments)):
2174 for piece, ptags_ in zip(state.pieces[seg_idx],
2175 state.piece_tags[seg_idx]):
2176 # the tag tests inline ahead of the predicate: every tail
2177 # past the name segments runs this ('John Smith, Jr., PhD'),
2178 # the part after a suffix comma being bound by group, and the call
2179 # would be its frame. Both arms are needed -- a connective
2180 # join tags a piece a title though its first token is none
2181 # ('and Secretary of State')
2182 _set_roles(tokens, piece,
2183 Role.TITLE
2184 if (("title" in ptags_
2185 or "vocab:title" in tokens[piece[0]].tags)
2186 and is_title_piece(piece, ptags_, tokens)
2187 and not any("vocab:suffix" in tokens[i].tags
2188 for i in piece))
2189 else Role.SUFFIX)
2190 # rules/S12: "where the count takes the pair, the reading is reported" -- at
2191 # the end of a name, at the end of the given part, as a run start,
2192 # here where its roles are final (the part a comma's decision binds reports in `_comma`, a
2193 # pair a maiden clause releases in group). A loop over the piece
2194 # tags, so a name with no pair pays no frame for it.
2195 for seg_idx in range(min(tail, len(state.segments))):
2196 for piece, ptags_ in zip(state.pieces[seg_idx],
2197 state.piece_tags[seg_idx]):
2198 if ("split" in ptags_ and len(piece) == 2
2199 and tokens[piece[0]].role is Role.SUFFIX
2200 and tokens[piece[1]].role is Role.SUFFIX):
2201 ambiguities.append(split_credential_report(
2202 piece, tokens, _split_name_alt(state, tokens, seg_idx,
2203 piece, order)))
2204 # the reports earlier stages left a field for: only a name that
2205 # has one pays the walk
2206 if state.ambiguities:
2207 _resolve_deferred(state, tokens, ambiguities, order)
2208 return copy_with(state, tokens=tuple(tokens),
2209 order=order,
2210 ambiguities=tuple(ambiguities))