Shortcuts on this page
r m x toggle line displays
j k next/prev highlighted chunk
0 (zero) top of page
1 (one) first highlighted chunk
1"""Shared piece-level predicates for pipeline stages.
3How a PIECE reads -- its tokens, plus the tags classify wrote on them
4and the tags group derived for the piece -- where _vocab answers how a
5WORD reads from text alone. Both are consulted by more than one
6stage; the split is by what the question takes, not by
7which stage happens to ask (mechanisms.md#ONE-PREDICATE-PER-QUESTION).
8_vocab points here from its own side: "Text-level tests used by more
9than one stage; piece-level ones live in _pieces, the sibling layer
10over tokens-plus-tags." own_words (#289/#516) answers over the whole
11token STREAM rather than one piece -- pieces do not exist yet at the
12stages that call it -- but the question is still piece-shaped, not
13word-shaped: it reads token ROLE, which _vocab's text-level tests
14never take.
16Before this module those predicates lived in _group, not because
17grouping owned them but because assign imported group and could not be
18imported back, so group was the only place both stages could reach.
19They arrived there that way across three PRs -- #424 brought
20is_leading_title, leading_titles and trailing_start, #425 the peel
21(peel_walk, peel_trailing), #429 the no-name-segment test that
22#430 turned into segment_suffix_reading.
23is_title_piece and is_suffix_piece are older than any of that: they
24were group's from its first commit, and travel because the others
25call them.
27The import that forced all of it is the one #439 removed: assign no
28longer names _group at all. What still holds is the rule that replaced
29it, and tests/v2/test_layering.py is where it is written down -- a
30piece predicate may not depend on a stage, in either direction.
32The S2 trailing peel travels as the unit the decisions describe --
33peel_walk, peel_trailing and trailing_start together -- though only
34the first two cross a stage boundary. trailing_titles joins them
35because it reads what that peel left: the two answer one question
36between them, where the tail of a name stops being the name -- and
37tail_reading is that one question, running them against each other to
38their fixed point for the two stages that must not disagree about the
39answer.
41Layering: imports _state and _vocab only; segment, script_segment,
42classify, group, assign, post_rules and _comma import it (segment,
43script_segment and classify for the #289/#516 own-words span alone) -- and neither of the two it imports imports it
44back.
46Naming follows _vocab's: inside an already-private module the leading
47underscore marks module-PRIVATE, so the names other stages call are
48bare and only the internals keep it (_PERIOD_ABBREV here). Getting that
49backwards -- which this module did until the underscores came off --
50costs a reader the one cheap way to tell a shared predicate from a
51helper.
52"""
53from __future__ import annotations
55import unicodedata
56from collections.abc import (
57 Callable, Container, Iterable, Mapping, Sequence, Set)
58from typing import NamedTuple
60from nameparser._pipeline._state import (
61 AMBIGUOUS_ACRONYM_TAG, AMBIGUOUS_WORD_TAG,
62 GENERATION_WORD_TAG, TITLE_CASE_NAME_TAG, GIVEN_ABBREVIATION_TAG, NAME_ROLES, SHAPE_ACRONYM_TAG, SUFFIX_OR_UNREAD, AmbiguityKind, PendingAmbiguity,
63 CHAIN_TITLE_TAG, DOTTED_TITLE_TAG, PendingReading,
64 WorkToken,
65)
66from nameparser._pipeline._vocab import (
67 _DIGRAPH_INITIALS, _INDIC_BRAHMIC, _PERIOD_ABBREV, _ROMAN,
68 Lean, ambiguous_lean,
69 in_initialless_script,
70 is_listed_numeral, is_single_letter_numeral, numeral_leans_to_the_name,
71 numeral_settled_by_case, reads_as_generation, tag_marker_runs,
72)
75# rules/P3: "both questions asked of a name here and in P8 — how
76# many words it has, and whether it is written in one case — are
77# asked of the name's OWN words: a maiden marker taken as one, and the
78# words it takes (M2), are not among them, and neither is a delimited
79# clause (N1, M1)" (history: decisions/P3)
80def own_words(tokens: Sequence[WorkToken], comma_offsets: Sequence[int],
81 markers: frozenset[str],
82 marker_tags: Mapping[int, str] | None = None,
83 ) -> tuple[list[str], int]:
84 """The name's OWN word texts and the index the maiden clause
85 starts at -- one span for the two stages that ask about it
86 (#289/#516).
88 Own words are the role-less tokens before the clause: a delimited
89 clause's tokens arrive from extract with a role already set, and
90 everything from a maiden marker on is the clause. Appending a
91 clause to a name must not change how a word in the name reads.
93 `marker_tags` is the map `_vocab.tag_marker_runs` already built,
94 index -> "vocab:maiden-marker"/"...-cont"; a caller that has it
95 (classify) hands it over and pays no second walk. A caller that
96 runs BEFORE those tags exist (segment) omits it, and this
97 function calls `tag_marker_runs` itself to build the SAME map
98 classify would -- not an approximation of it, which is what a
99 from-scratch text walk (this module's earlier `first_marker_head`)
100 could disagree with on a name where a marker's head opens an entry
101 but no run completes ('z' of 'z domu'): measured, 'ANNA z Nowak,
102 MD' flipped the recorded one-case verdict under that approximation
103 (decisions/P3). Sharing the exact function instead makes the
104 two paths agree by construction, not by corpus luck.
106 Takes tokens/comma_offsets/markers rather than a whole ParseState:
107 both call sites have all three already, and passing them lets this
108 function sit beside the piece predicates rather than in _vocab
109 (mechanisms.md#ONE-PREDICATE-PER-QUESTION; the
110 _post_rules.suffix_entries precedent, AGENTS.md's named exception)
111 -- it answers with the SPAN, where `tag_marker_runs` answers only
112 which tokens open a run.
114 `marker_tags`' keys must arrive in index order for the walk below
115 to find the SMALLEST head in one pass: `tag_marker_runs` walks its
116 tokens left to right, so the first head it records is already the
117 smallest, and a caller building its own map must preserve that
118 order too.
120 A plain tuple, not a NamedTuple: measured 2026-09-17, wrapping
121 this in a `NamedTuple` (this module's `Peel` is one) cost the
122 reference name one more frame (413.00 vs the 412.00 band this
123 commit must hold) -- a NamedTuple's `__new__` is itself a call,
124 where a bare tuple literal is not. `Peel` can afford the frame
125 because assign builds one only where the trailing peel actually
126 ran; `own_words` returns on every parse.
127 """
128 if marker_tags is None:
129 marker_tags = tag_marker_runs(tokens, comma_offsets, markers)
130 clause_at = len(tokens)
131 for i, tag in marker_tags.items():
132 if tag == "vocab:maiden-marker" and tokens[i].role is None:
133 clause_at = i
134 break
135 return ([t.text for t in tokens[:clause_at] if t.role is None],
136 clause_at)
139# rules/H3: "successive title words at the start of the part
140# carrying the given name chain into one title; a title word
141# elsewhere in the name does not"
142def is_title_piece(piece: Sequence[int], ptags: Set[str],
143 tokens: Sequence[WorkToken]) -> bool:
144 if "title" in ptags:
145 return True
146 return len(piece) == 1 and "vocab:title" in tokens[piece[0]].tags
149# rules/P4: "a particle in the name's leading position chains
150# nothing" -- WHERE that position is, asked once, by group's chain and
151# the trailing read's unit count (#624)
152def chain_lead(pieces: Sequence[Sequence[int]], ptags: Sequence[Set[str]],
153 tokens: Sequence[WorkToken], n: int) -> int:
154 """The particle chain's leading position, given `n`, the end of
155 the leading title run (`leading_titles`): the last title in the run
156 that is also a particle, else the first piece past the run. A title
157 is no name word, so the first piece past the run leads; a word in
158 both vocabularies ('Freiherr', 'St') leads in its stead and puts
159 the particle behind it inside a name ('Freiherr von Berg', 'St van
160 Johnson'), and of several such words the last leads, all being
161 titles ('Freiherr St John Smith' as 'Dr. St John Smith'). No
162 particle stands between the lead and the run's end, so the chain
163 opens no unit inside the titles.
165 The run is read as walked, before H3's give-back: a title the run
166 hands back to the name because only suffix words follow it is
167 still a title to the chain, so the particle behind it is the
168 name's leading piece and chains nothing (P4: 'Dr. Mc Mc' keeps
169 'Mc' and 'Mc' apart, as before #624). `leading_titles` stops at a
170 piece that is itself a leading title only where it gave that piece
171 back, before a suffix piece (the tags it tests inline first), or at
172 the segment's last piece, which a title may not be unless it is the
173 whole segment -- the `n + 1 < len(pieces)` test rules that one out.
174 The same tags go first here, so a name with no suffix behind its
175 titles pays no frame for the test."""
176 if (n + 1 < len(pieces)
177 and ("suffix" in ptags[n + 1]
178 or "vocab:suffix" in tokens[pieces[n + 1][0]].tags)
179 and is_leading_title(pieces[n], ptags[n], tokens)):
180 n += 1
181 for k in range(n - 1, -1, -1):
182 if is_prefix_piece(pieces[k], ptags[k], tokens):
183 return k
184 return n
187# A particle, or a piece a join made one: what group's prefix chain
188# (at its loop in _group_segment) chains and stops at, what P3's join
189# derives a prefix from, and what group's rootname count and leading
190# scan step past.
191def is_prefix_piece(piece: Sequence[int], ptags: Set[str],
192 tokens: Sequence[WorkToken]) -> bool:
193 if "prefix" in ptags:
194 return True
195 return len(piece) == 1 and "particle" in tokens[piece[0]].tags
198# rules/P3: "a recognized connective joins its neighbors into one
199# name part, connective runs included — except a single-letter
200# connective in a three-word name, which stays a name word where P13
201# has not joined it first, and a single-letter connective that reads
202# as an initial instead (P8), which never joins" (history:
203# decisions/P3)
204def is_conj_piece(piece: Sequence[int], ptags: Set[str],
205 tokens: Sequence[WorkToken]) -> bool:
206 if "conjunction" in ptags:
207 return True
208 return len(piece) == 1 and "conjunction" in tokens[piece[0]].tags
211def joined_tags(ptags: Sequence[Set[str]], lo: int, hi: int,
212 add: Set[str] = frozenset(),
213 drop: Set[str] = frozenset()) -> set[str]:
214 # the ONE definition of a merged piece's tags: merge_pieces applies
215 # it, and P5's reserve reads it to model the join it is
216 # weighing (#425) -- so the view cannot drift from the merge.
217 # A merged piece inherits every part's tags, so a site whose
218 # product is not what its parts were drops what no longer
219 # applies: the particle chain drops `prefix`, the bound join
220 # `title` (a derived title tag on the pair would have assign
221 # peel the given name as a leading title).
222 return (set().union(*ptags[lo:hi]) | add) - drop
225def merge_pieces(pieces: list[list[int]], ptags: list[set[str]],
226 lo: int, hi: int, add: Set[str] = frozenset(),
227 drop: Set[str] = frozenset()) -> None:
228 # pieces/ptags are parallel arrays; every merge must update
229 # both in lockstep.
230 #
231 # Extend the first piece IN PLACE rather than rebuilding the
232 # merged list. The obvious spelling --
233 # pieces[lo:hi] = [[i for p in pieces[lo:hi] for i in p]]
234 # -- re-flattens everything accumulated so far on every call, so
235 # a chain that merges into the same piece n times copies
236 # 1+2+...+n and the stage goes quadratic in the length of the
237 # chain. A conjunction run ("and " * n) did exactly that while
238 # `join_connectives` merged through here: it measured 2.4x-2.9x
239 # per doubling against the 2.0x every other shape holds. No piece
240 # list is aliased outside this function (each caller builds every
241 # piece as a fresh list -- group's [i] -- and reads pieces[k] only
242 # before a merge), so mutating is safe; verified
243 # identical token/role/tag/span/ambiguity output over 54,877
244 # names.
245 #
246 # Every call site passes lo < hi, and this REQUIRES it: with
247 # lo >= hi the slice assignment would insert rather than
248 # replace, putting a second reference to pieces[lo] into the
249 # array, and the next merge to touch either index would extend
250 # the same list twice. The old rebuild-a-fresh-list spelling
251 # was harmless there. Keep the bound if you add a caller.
252 #
253 # The slice assignment itself still shifts every piece behind
254 # `hi`, C-level work no frame guard counts, so a caller that
255 # merges once per member of a run whose length grows with the
256 # input goes quadratic all the same. `join_connectives` and
257 # group's split-pair merge did, and build their lists in one pass
258 # instead (decisions/parse-cost, 2026-11-03); the callers left
259 # merge once per run (the particle chain, the bound join). A
260 # caller that would merge per member builds its output that way,
261 # with a row in tests/v2/test_benchmark.py's shift guards.
262 combined = pieces[lo]
263 for piece in pieces[lo + 1:hi]:
264 combined.extend(piece)
265 pieces[lo:hi] = [combined]
266 ptags[lo:hi] = [joined_tags(ptags, lo, hi, add, drop)]
269def join_connectives(pieces: list[list[int]], ptags: list[set[str]],
270 tokens: Sequence[WorkToken], *,
271 letter_stays: bool, titles_only: bool,
272 frozen: Set[int] = frozenset()) -> None:
273 """rules/P3's connective joins, for both parts that make them:
274 group's name segments, and the part after a comma, which
275 `_comma.decide` reads before group joins anything (#617, where the
276 two had been copies that differed). Contiguous connectives merge
277 into one first (v1: "of the"); then each connective joins its
278 neighbours, taking its kind from the one on its left, or from the
279 one on its right where it opens the part.
281 `letter_stays` is the single-letter carve-out's answer for this
282 part, and it reaches a LONE single-letter connective only: one
283 beside another connective merges into their run first and joins
284 with it. Group reads it from the name's rootname count; the comma
285 part, which a title joins and a name word never does, always keeps
286 the lone letter. `titles_only` is the comma part's: a join is made
287 only where the neighbour is a title, and makes the joined piece
288 one ('Mr. and Mrs.') with no prefix kind, nothing in that part
289 being a particle chain; any other connective there is left a word,
290 the reading deciding what it is. Otherwise every join is made and
291 takes the neighbour's title and prefix kinds. `frozen` holds the
292 TOKEN index of each connective that is to make no join or run of
293 its own -- group's, the generational ones with no name word on one
294 side; the comma part's, every generational one -- though a
295 neighbour's join may still take one in."""
296 # contiguous conjunction runs merge first
297 #
298 # The first token (`run_pieces[-1][0] in frozen`) and not
299 # `frozen.isdisjoint(...)`: the piece this loop extends GROWS
300 # with every merge, so a
301 # test over its tokens costs 1+2+...+n and the stage goes
302 # quadratic in the length of a connective run -- measured,
303 # 'and ' x3200 took 41.8ms against 21.7ms, 6.2x per 4x input
304 # where the shape reads 4.1x, and tests/v2/test_benchmark.py's
305 # "and " shape is the guard that caught it. Reading the first
306 # token alone is exact rather than an approximation: a frozen
307 # piece is one token and this branch never merges it, and this
308 # loop finishes before the join below can take one into a
309 # neighbour's piece, so a piece holding a frozen token here IS
310 # that token.
311 #
312 # Built in one pass, not through `merge_pieces`, whose slice
313 # assignment would shift the rest of the list once per member
314 # (its comment). The run being built is `run_pieces[-1]`.
315 run_pieces: list[list[int]] = []
316 run_tags: list[set[str]] = []
317 for piece, tags in zip(pieces, ptags):
318 if (run_pieces
319 and is_conj_piece(run_pieces[-1], run_tags[-1], tokens)
320 and is_conj_piece(piece, tags, tokens)
321 and run_pieces[-1][0] not in frozen
322 and piece[0] not in frozen):
323 run_pieces[-1].extend(piece)
324 run_tags[-1] = joined_tags((run_tags[-1], tags), 0, 2,
325 add={"conjunction"})
326 else:
327 run_pieces.append(piece)
328 run_tags.append(tags)
329 pieces[:] = run_pieces
330 ptags[:] = run_tags
331 # each conjunction joins its neighbors, rules/P3: "except a
332 # single-letter connective in a three-word name, which stays a
333 # name word" (v1's Google Code issue 11 carve-out, the
334 # "john e smith" bug).
335 #
336 # The joins are written into fresh lists in one pass, for the
337 # same reason: a run of joins ('Mr. and ' x n) through
338 # `merge_pieces` would shift the rest of the list once per join.
339 # A join takes the piece behind it off `out`, so the piece it
340 # produces is `out[-1]`, the left neighbour of the next piece
341 # read, and `len(out)` is always the index `pieces[k]` would
342 # stand at had every join so far been made in the list itself.
343 out: list[list[int]] = []
344 out_tags: list[set[str]] = []
345 n = len(pieces)
346 k = 0
347 while k < n:
348 piece, tags = pieces[k], ptags[k]
349 # first token again, exact for the run loop's reason (a
350 # frozen piece is one token) and for a second: the piece a
351 # join produces is left on `out`, BEHIND `k`, so no joined
352 # piece is ever tested here.
353 if (not is_conj_piece(piece, tags, tokens)
354 or piece[0] in frozen):
355 out.append(piece)
356 out_tags.append(tags)
357 k += 1
358 continue
359 if letter_stays and len(piece) == 1:
360 text = tokens[piece[0]].text
361 if not text.isascii():
362 # a decomposed 'й' is one letter in two code points
363 # (#665); compose it, as the lookup does
364 text = unicodedata.normalize("NFC", text)
365 if len(text) == 1 and text.isalpha():
366 out.append(piece)
367 out_tags.append(tags)
368 k += 1
369 continue
370 has_left = bool(out)
371 has_right = k + 1 < n
372 if has_left:
373 neighbor, neighbor_tags = out[-1], out_tags[-1]
374 elif has_right:
375 neighbor, neighbor_tags = pieces[k + 1], ptags[k + 1]
376 else:
377 # a connective alone in its part is its own neighbour,
378 # so it takes its own kinds
379 neighbor, neighbor_tags = piece, tags
380 derived = set()
381 if is_title_piece(neighbor, neighbor_tags, tokens):
382 derived.add("title")
383 elif titles_only:
384 out.append(piece)
385 out_tags.append(tags)
386 k += 1
387 continue
388 if (not titles_only
389 and is_prefix_piece(neighbor, neighbor_tags, tokens)):
390 derived.add("prefix")
391 parts = [piece]
392 part_tags = [tags]
393 if has_left:
394 parts.insert(0, out.pop())
395 part_tags.insert(0, out_tags.pop())
396 if has_right:
397 parts.append(pieces[k + 1])
398 part_tags.append(ptags[k + 1])
399 combined = parts[0]
400 for part in parts[1:]:
401 combined.extend(part)
402 out.append(combined)
403 out_tags.append(joined_tags(part_tags, 0, len(part_tags),
404 add=derived))
405 k += 2 if has_right else 1
406 pieces[:] = out
407 ptags[:] = out_tags
410# _PERIOD_ABBREV: imported from _vocab, not redefined here (#289/#516,
411# quality-review finding) -- _vocab.name_word_count needed the SAME
412# shape test is_leading_title asks (_vocab.is_title_shaped), and
413# layering only allows the move in that direction (_pieces may import
414# _vocab; _vocab may not import _pieces). tests/v2/test_regex_sync.py
415# declares `_pieces._PERIOD_ABBREV` as an import of _vocab's, which an
416# import binds here under the same name. Out of
417# assign since #424 and in the piece layer since #439: the test is
418# assign's, and the chain's leading position (`chain_lead`) and the
419# trailing-run walk must start where assign starts.
422# rules/H2: "an abbreviation opening the part of the name that
423# carries the given name — the whole name, or the part after a
424# family comma — reads as a title even when unlisted"
425# (history: decisions/H2)
426def is_leading_title(piece: Sequence[int], ptags: Set[str],
427 tokens: Sequence[WorkToken]) -> bool:
428 if is_title_piece(piece, ptags, tokens):
429 return True
430 if len(piece) != 1:
431 return False
432 text = tokens[piece[0]].text
433 # INLINED rather than calling _vocab.is_title_shaped, which asks
434 # the exact same question (#289/#516, quality-review finding: the
435 # two must not drift, and did once -- name_word_count's own
436 # vocabulary-only title test read 'Xyz.' as a name word where this
437 # predicate reads it as a title, and the disagreement flipped a
438 # comma structure `Dr. Smith, Ed`'s LISTED spelling did not).
439 # THIS is the one home for the number, `is_title_shaped` pointing
440 # here rather than restating it: routing this hot path through the
441 # shared function costs one frame per call (`is_leading_title`
442 # runs on every leading piece of every parse, unlike
443 # name_word_count's comma-only path), moving the reference name
444 # from 412/449 to 417/454 -- five calls on `Dr. Juan de la Vega
445 # III`, recomputable with `uv run python
446 # tools/perf/call_count.py`. Kept as two spellings of ONE test
447 # instead -- if you touch one, touch both, and
448 # `test_is_title_shaped_and_is_leading_title_agree` (this module's
449 # own test file) checks it over the union of both predicates'
450 # example tables rather than leaving it to a sentence.
451 return (bool(_PERIOD_ABBREV.match(text))
452 and (text.isascii() or not (_INDIC_BRAHMIC.search(text)
453 or in_initialless_script(text)))
454 # rules/H2's transliteration-digraph Except (#1136) -- a
455 # C-level membership test
456 and text[:-1].lower() not in _DIGRAPH_INITIALS
457 # rules/H2's precedence: a given name written short outranks
458 # the shape ('Chas.', #789) -- classify's tag, a C-level test
459 and GIVEN_ABBREVIATION_TAG not in tokens[piece[0]].tags)
462def leading_titles(pieces: Sequence[Sequence[int]],
463 ptags: Sequence[Set[str]],
464 tokens: Sequence[WorkToken]) -> int:
465 """How many leading pieces assign peels as titles: the first
466 non-title index. A title needs a following piece, unless the whole
467 segment is one title (v1 parity). And the run gives back its last
468 piece when that piece is a name candidate: where everything behind
469 the run is suffix pieces, the run gives back its last piece, when
470 that piece is one word and is not itself suffix vocabulary
471 (rules/H3, decisions/H3 -- the block at the floor below
472 carries the examples of each half, and the ordering its two inline
473 tag reads were measured on). A dotted run (rules/H10) the run ends
474 on is given back after the floor.
475 One definition, read by assign (which sets the roles), by the
476 chain's trailing-run walk, and by `chain_lead`, which finds the
477 chain's leading position inside this run for group's chain and the
478 trailing read alike (P4, #367, #424, #624)."""
479 n = 0
480 while n < len(pieces):
481 if ((n + 1 < len(pieces) or len(pieces) == 1)
482 and (is_leading_title(pieces[n], ptags[n], tokens)
483 or (len(pieces[n]) == 1 and DOTTED_TITLE_TAG
484 in tokens[pieces[n][0]].tags))):
485 n += 1
486 continue
487 break
488 # rules/H3: "where everything behind the run is post-nominal,
489 # the run gives its last word back to the name, provided that word
490 # stands alone and is not itself suffix vocabulary"
491 #
492 # ONE WORD, because a joined unit led by a title is a title run and
493 # handing it back would lose the title: 'Prince of Wales Jr' reads
494 # title 'Prince of Wales', family 'Jr', not given 'Prince of Wales'
495 # with no title at all.
496 #
497 # Two residuals. A run whose last word IS suffix vocabulary is not
498 # given back, so 'Dr Pope RA PhD' still reads title 'Dr Pope RA',
499 # family 'PhD'. And the floor asks is_suffix_piece, which vetoes a
500 # bare initial-shaped numeral, so 'Dr Pope V' keeps the whole run
501 # as the title and reads the numeral as the name -- given 'V',
502 # 'pope' being a given-name title and the run's last word, where
503 # 'Dr Smith V' reads family 'V' (a title being no word to spare
504 # for a one-letter numeral, #718). That numeral fork is outside
505 # this floor (decisions/H3).
506 #
507 # The two inline tag reads are the cheapest NECESSARY condition for
508 # the piece behind the run to be a suffix piece at all --
509 # is_suffix_piece cannot answer yes without one of them -- so the
510 # ordinary titled name, whose next piece is no kind of suffix,
511 # leaves this branch without entering a frame. Measured on the
512 # plan's ordering rather than the shape below: asking the
513 # authoritative predicate first cost 8 calls per parse of the
514 # reference name (leading_titles runs four times), against a band
515 # with room for two (decisions/parse-cost). is_suffix_piece
516 # stays the predicate that ANSWERS, here and in the walk.
517 if (n and n < len(pieces)
518 and ("suffix" in ptags[n]
519 or "vocab:suffix" in tokens[pieces[n][0]].tags)
520 and len(pieces[n - 1]) == 1
521 and not is_suffix_piece(pieces[n - 1], ptags[n - 1],
522 tokens)):
523 for k in range(n, len(pieces)):
524 if not is_suffix_piece(pieces[k], ptags[k], tokens):
525 break
526 else:
527 # nothing behind the run but suffix pieces
528 n -= 1
529 # rules/H10: "where a title the run keeps stands behind it, after
530 # H3's floor has given its word back" -- so a dotted run the run
531 # ends on is given back too, and so is each dotted run before it,
532 # while a title before them stays ('U.S. Grant', 'Hon. U.S. Grant');
533 # the tag is classify's, on a
534 # word no other title test takes (decisions/H10, #490)
535 while (n and len(pieces[n - 1]) == 1
536 and DOTTED_TITLE_TAG in tokens[pieces[n - 1][0]].tags):
537 n -= 1
538 return n
541def is_suffix_piece(piece: Sequence[int], ptags: Set[str],
542 tokens: Sequence[WorkToken]) -> bool:
543 if "suffix" in ptags:
544 return True
545 if len(piece) != 1:
546 return False
547 tags = tokens[piece[0]].tags
548 return "vocab:suffix" in tags and "initial" not in tags
551# rules/S8: "A title word never starts the run, and neither does a
552# member of the ambiguous class its writing leaves unsettled, a single
553# letter, a connective" -- what
554# starts #602's credential run, one predicate for the trailing peel and
555# the given-part walk (mechanisms.md#ONE-PREDICATE-PER-QUESTION)
556def starts_a_credential_run(piece: Sequence[int], ptags: Set[str],
557 tokens: Sequence[WorkToken],
558 preceding: str | None = None,
559 one_case: bool | None = None) -> bool:
560 """Whether one piece starts #602's run: an unambiguous suffix
561 word of two or more letters, in any script -- a credential acronym,
562 a generational word or an honorific -- that no other vocabulary
563 claims as a name.
565 `is_suffix_piece` already refuses the ambiguous class and an
566 initial-shaped letter. Refused here: any single letter (a lowercase
567 `i` or `v` mid-name is an initial or a connective far more often
568 than a numeral, and carries no `initial` tag), a connective, a
569 word that is also bound given-name or particle vocabulary (`abd`
570 heads `Abd Allah`; `vd`, `mc`). No script is refused: an honorific
571 starts the run as a credential does (#679, decisions/S8), so
572 `Ана Хорват старији Џонс` reads as `Ana Horvat stariji Jones` does.
574 The split credential group merges ('Ph. D.', the one piece carrying
575 the "suffix" piece tag) starts it as its one-token spelling 'PhD'
576 does: a run that started for one spelling of a word and not the
577 other is the vocabulary-against-shape split nameparser/AGENTS.md's
578 spelling sweep names (#603 found it, 'Smith, John Ph. D. Jones' keeping
579 middle 'Jones' where 'Smith, John PhD Jones' reads suffix
580 'PhD Jones'). The no-comma spelling is not reached: `run_start`
581 asks only of the pieces `peel_walk` keeps, and the walk drops the
582 merged piece, so 'John Smith Ph. D. Jones' still reads family
583 'Jones'.
585 A roman numeral of two or more letters starts it where
586 `_vocab.reads_as_generation` reads it as the generation, with
587 `preceding` and `one_case` that predicate's (#784): 'John Smith VI
588 Jones' as 'John Smith III Jones'.
589 """
590 if "suffix" in ptags or "split" in ptags:
591 # a split credential alone: a connective join keeps the tag on
592 # the wider piece ('Ph. D. and Mary'), which starts nothing, as
593 # 'PhD and Mary' does not
594 return len(piece) == 2
595 if len(piece) != 1:
596 return False
597 tok = tokens[piece[0]]
598 if not tok.tags.isdisjoint(_NOT_A_RUN_START):
599 return False
600 # rules/S8: "a roman numeral of two or more letters that reads as
601 # the generation" -- asked of the numeral's predicate, which the
602 # vocabulary's own numerals answer too
603 stem = tok.text.rstrip(".")
604 if len(stem) >= 2 and _ROMAN.match(stem) is not None:
605 return reads_as_generation(tok.text, tok.tags, preceding, one_case)
606 # rules/S8: "and so does a listed member of the ambiguous class its
607 # writing settles as the credential" -- the capitals mark the
608 # post-nominals as periods do; the particle test above still
609 # refuses 'DO'
610 if AMBIGUOUS_ACRONYM_TAG in tok.tags:
611 return (settled_by_case(tok, one_case)
612 and sum(map(str.isalpha, tok.text)) >= 2)
613 if not is_suffix_piece(piece, ptags, tokens):
614 return False
615 # C-level, no generator frame per character
616 letters = "".join(filter(str.isalpha, tok.text.lower()))
617 return len(letters) >= 2
620#: tags of suffix words another vocabulary claims as a name word or a
621#: joiner, which therefore never start #602's run
622_NOT_A_RUN_START = frozenset({"conjunction", "particle",
623 "vocab:bound-given"})
626def split_credential_report(piece: Sequence[int],
627 tokens: Sequence[WorkToken],
628 alternative: PendingReading) -> PendingAmbiguity:
629 """rules/S12's report on a split credential the count took, one
630 spelling for the three sites that take one (assign's placement, the
631 comma's binding, the maiden clause's release). `alternative` is the
632 words' name reading, whose field each site knows."""
633 text = " ".join(tokens[i].text for i in piece)
634 return PendingAmbiguity(
635 AmbiguityKind.SUFFIX_OR_NAME,
636 f"{text!r} is two words whose letters spell a credential, so it "
637 f"reads as a credential; they may be name words", tuple(piece),
638 alternative=alternative)
641def is_lone_never_given_particle(piece: Sequence[int],
642 tokens: Sequence[WorkToken]) -> bool:
643 """A piece that is one never-given particle (rules/P1's fold
644 site). One predicate, asked by the fold in post_rules and by two
645 sites in assign that predict the fold will take the particle
646 forward: the given slot, which then leaves no given name in front
647 of a member (#573), and P6's attachment after a family comma,
648 which then leaves no given word for the tail to stand behind
649 (#613) -- copies would drift silently, each site's own tests
650 still passing (mechanisms.md#ONE-PREDICATE-PER-QUESTION)."""
651 return (len(piece) == 1
652 and "particle" in tokens[piece[0]].tags
653 and "vocab:particle-ambiguous" not in tokens[piece[0]].tags)
656# rules/P11: "an ambiguous particle written in lower case, in a name of
657# mixed case, opens the name as a never-given one does" -- the site
658# test, asked by the fold in post_rules and by assign's report, which
659# the fold settles (decisions/P11, #670)
660def is_lowercase_ambiguous_particle(piece: Sequence[int],
661 tokens: Sequence[WorkToken],
662 one_case: bool | None) -> bool:
663 """A piece that is one ambiguous particle written in lower case in
664 a name of mixed case: rules/P1's fold site under a family-first
665 order, where the writing settles the particle's fork as a
666 never-given particle's vocabulary does (#670). `one_case` is
667 ParseState.one_case; None (nobody asked) settles nothing."""
668 return (len(piece) == 1
669 and "vocab:particle-ambiguous" in tokens[piece[0]].tags
670 and one_case is False and tokens[piece[0]].text.islower())
673def _numeral_behind_the_initial_veto(piece: Sequence[int],
674 tokens: Sequence[WorkToken]) -> bool:
675 """Suffix vocabulary that is_suffix_piece refuses because it is
676 also initial-shaped: a ONE-CHARACTER entry, bare or with a period.
678 Named for the shape rather than enumerated, because the shape is
679 what the code tests and the enumeration goes stale -- in the
680 shipped lexicon it reaches i, v and 2, and NOT x or ix (roman, but
681 not suffix vocabulary) nor ii/iii/iv (suffix vocabulary, but two
682 characters, so never initial-shaped and never vetoed in the first
683 place). A caller adding a one-character suffix in a script that
684 has initials extends it.
686 The veto is right where such a word could be a middle initial, and
687 wrong where it is describing the suffix in front of it, which is
688 the only place this is asked from. The len(piece) != 1 guard is
689 defensive: a merged multi-token piece carries "suffix" in ptags, so
690 is_suffix_piece claims it one branch earlier and no reachable input
691 arrives here with one.
692 """
693 if len(piece) != 1:
694 return False
695 # A one-letter numeral here is the numeral's predicate's yes by
696 # construction: in a comma part it reads one by the vocabulary, and
697 # the tag below is that vocabulary (#752), so it is not asked again.
698 tags = tokens[piece[0]].tags
699 return "vocab:suffix" in tags and "initial" in tags
702def _anchors(piece: Sequence[int], tokens: Sequence[WorkToken]) -> bool:
703 """Whether a suffix piece may ANCHOR an ambiguous member behind it
704 (#544): not a connective, not a particle, and not a single-letter
705 roman numeral.
707 A connective anchors nothing because between two name words it is
708 a link -- the generational 'i' is also Catalan's 'i' (rules/P3)
709 -- and 'Jane Doe nee Puig i Ma' must keep its clause. A particle
710 anchors nothing for the same reason from the other side: it joins
711 the name word behind it (rules/P2), so a word that is both
712 particle and suffix vocabulary ('vd', 'mc') standing in front of
713 a member is the head of a family name, not a credential list --
714 'Smith vd Ma, John' keeps family 'Smith vd Ma' and 'Jan vd Ma'
715 family 'vd Ma'. A particle MEMBER is still anchored by a
716 credential in front of it ('doe, jane v phd do'); the exclusion is
717 of the anchor only. A single-letter roman numeral, in any case
718 ('V', 'v', 'I.'), anchors nothing for its SHAPE: one letter is
719 the shape a middle initial is written in ('V' can be one), and S3
720 retired single-character vocabulary matches for the same reason.
721 Not because a numeral is no credential -- a multi-letter suffix
722 word ('III', 'Jr') anchors like any other suffix piece
723 (`is_single_letter_numeral`). A title/suffix DUAL ('ms', 'ra',
724 'sr') does anchor, except in the given part's
725 leading title run, where it stands in title position ('Smith, Ms
726 Ma' is Ms. Ma Smith); that exclusion is the callers', which start
727 their walks past the leading title run or, in
728 `segment_suffix_reading`, read no anchor at all in a part whose
729 leading title run holds one.
731 Asked only of a piece `is_suffix_piece` accepted, and only once a
732 member's own writing has declined, so an ordinary name never pays
733 for it."""
734 if len(piece) != 1:
735 return True
736 tok = tokens[piece[0]]
737 return ("conjunction" not in tok.tags
738 and "particle" not in tok.tags
739 and not is_single_letter_numeral(tok.text))
742# rules/S7: "Only a credential in FRONT speaks; one behind the
743# member says nothing about it." #544: an ambiguous member standing
744# BEHIND an unambiguous credential in one run is described by that
745# credential's company, not by its own writing -- 'PhD MEng' is a list of degrees whatever case 'MEng'
746# is written in. ONE forward pass per question, so a run of members
747# is read in linear time: a look-behind per member is quadratic in
748# the run, which test_benchmark's frame-ratio guard is built to catch.
749def credential_anchors(order: Sequence[int],
750 pieces: Sequence[Sequence[int]],
751 ptags: Sequence[Set[str]],
752 tokens: Sequence[WorkToken],
753 first_kept: bool = True) -> list[bool]:
754 """For each position of `order` (piece indices in text order),
755 whether a lone listed member standing there is ANCHORED: the
756 contiguous run of pieces in FRONT of it that are suffix pieces or
757 lone ambiguous members ends in -- counting back past the members --
758 a suffix piece that may anchor (`_anchors`). Any other piece ends
759 the run. The anchor is in front only: 'Wang Ma PhD' leaves 'Ma'
760 unanchored.
762 The caller decides where the walk starts, and starts it past the
763 leading title run, where a title/suffix dual reads as a title
764 ('Smith, RA MA Ma' keeps its middle name). Where `order` holds the
765 name part, its own leading position is the name the reserve keeps
766 -- `rest[0]` at the no-comma peel, the given part's own first
767 piece at the given-slot call sites -- so ITS writing never anchors
768 what stands behind it, whatever vocabulary it carries: 'PhD Ma'
769 keeps the count's reading, family 'Ma', rather than reading as a
770 credential list headed by the given name itself. `first_kept`
771 False is the one caller with no reserve, `segment_suffix_reading`,
772 asking whether a comma part is a credential run at all: there the
773 first piece is a run member like any other ('Smith, PhD MEng')."""
774 out: list[bool] = []
775 anchor = False
776 for pos, idx in enumerate(order):
777 out.append(anchor)
778 if pos == 0 and first_kept:
779 continue
780 piece = pieces[idx]
781 if is_suffix_piece(piece, ptags[idx], tokens):
782 # a suffix piece that may not anchor still ENDS the run it
783 # would have anchored: 'PhD v Ma' is not a credential list
784 # through the numeral
785 anchor = _anchors(piece, tokens)
786 elif not (len(piece) == 1
787 and AMBIGUOUS_ACRONYM_TAG in tokens[piece[0]].tags):
788 anchor = False
789 return out
792def given_slot_anchors(pieces: Sequence[Sequence[int]],
793 ptags: Sequence[Set[str]],
794 tokens: Sequence[WorkToken],
795 start: int, end: int | None = None,
796 skip: Container[int] = ()) -> list[bool]:
797 """`credential_anchors` for the given part's slot, indexed by PIECE
798 rather than by position: the pass over `start` to `end`, `skip`
799 spliced out, and False before `start`, from `end` on and at every
800 skipped piece. `start` is where the leading title run ends, so a
801 title/suffix dual standing in it anchors nothing ('Smith, RA MA
802 Ma'), and the piece at `start` is the given name the reserve keeps.
803 The one home of that query for assign's given slot."""
804 stop = len(pieces) if end is None else end
805 order: Sequence[int] = range(start, stop)
806 if skip:
807 order = [q for q in order if q not in skip]
808 out = [False] * len(pieces)
809 for q, anchored in zip(order, credential_anchors(order, pieces, ptags,
810 tokens)):
811 out[q] = anchored
812 return out
815def anchor_in_reach(back: Iterable[int],
816 pieces: Sequence[Sequence[int]],
817 ptags: Sequence[Set[str]],
818 tokens: Sequence[WorkToken],
819 skip: Container[int] = ()) -> bool:
820 """False where `credential_anchors` must answer False for the
821 member standing just behind `back` -- the piece indices in front
822 of it, nearest first, `skip` spliced out: past the lone members
823 in front, the first other piece is no suffix piece, or there is
824 none. True means only "ask the pass".
826 Exact in that direction because the test is weaker than the
827 pass's: a suffix piece carries "suffix" in its ptags or is one
828 token carrying "vocab:suffix" (`is_suffix_piece`), so a piece
829 failing both can neither anchor nor carry a run on. Read from tags
830 alone, with no frame per piece, so the ordinary name ending in a
831 Title-case member ('John Smith Ma', 'Doe, John Q. Ma') spends one
832 frame here rather than the pass's one per piece. Callers pass the
833 positions the pass itself would read or a superset reaching
834 further back: the walk meets any anchor the pass would find before
835 it reaches past the pass's start, so reaching further can cost a
836 pass that finds nothing and never a reading."""
837 for q in back:
838 if q in skip:
839 continue
840 piece = pieces[q]
841 if "suffix" in ptags[q]:
842 return True
843 if len(piece) != 1:
844 return False
845 tags = tokens[piece[0]].tags
846 if "vocab:suffix" in tags:
847 return True
848 if AMBIGUOUS_ACRONYM_TAG not in tags:
849 return False
850 return False
853def segment_suffix_reading(pieces: Sequence[Sequence[int]],
854 ptags: Sequence[Set[str]],
855 tokens: Sequence[WorkToken],
856 lenient: bool,
857 one_case: bool | None,
858 anchored: list[int] | None = None,
859 absorbed: list[int] | None = None,
860 licensed: Container[int] = (),
861 head_named: bool = True, # rules/S11
862 runs: Container[int] = (), # rules/C16
863 ) -> tuple[bool, ...] | None:
864 """How each piece of a no-name segment reads: True a suffix, False
865 a title. None when the segment holds a name word and so is not a
866 credential run at all.
868 `one_case` admits #289's credential lean: an ALL-CAPS member of
869 the ambiguous set inside a mixed-case name is a credential in this
870 slot even with one word before the comma, because the writing is
871 evidence the count does not have ('Smith, MA' -> family 'Smith',
872 suffix 'MA'). Only the LEAN reaches here on its own: a token
873 admitted to the class by SHAPE takes the count instead, which
874 `_comma.decide` takes and hands in as `licensed`, below ('Smith,
875 A.B.' -> given 'A.B.', one name word being no count).
877 A listed member ANCHORED by an unambiguous credential in front of
878 it in the same run reads as a credential too, whatever its writing
879 (#544, `credential_anchors`): 'Smith, PhD MEng' is family 'Smith'
880 with two degrees. A title/suffix dual standing in the part's
881 LEADING TITLE RUN -- every piece ahead of it a title or another
882 such dual -- is a title there, and in a part that holds one this
883 reading anchors nothing at all, not by the dual and not by a
884 later credential: the dual makes the next name-position word the
885 given name, and reading the part wholly as credentials would
886 stack the anchor's guess on the dual's ('Smith, Ms Ma', 'Smith,
887 Ms RA Ma' and 'Smith, RA PhD Ma' keep a given name, 'Smith,
888 Prof. RA Ma' too). The walk then reads the part, and its given
889 slot's company starts past that given name ('Smith, RA PhD Jr
890 Ma' reads suffix 'Jr Ma'). A plain title ahead of the credential
891 does not stop this reading ('Smith, Dr. PhD LAc' reads title
892 'Dr.', suffix 'PhD LAc').
894 `anchored`, when the caller passes a list, receives the index of
895 every piece the anchor read as a credential after its own writing
896 declined -- the picks the caller reports, since this decides them
897 (mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE). A member whose own
898 capitals lean credential is not among them. Meaningful only when
899 the answer is not None: a segment the walk abandons part-way may
900 have appended to it first.
902 ONE reader since #613: `_comma.decide`, which reads the part once
903 at group's head and binds the roles this returns, so the decision
904 that the part holds no name word and the roles its words get
905 cannot disagree. Until then it answered for two readers in
906 _assign.py, the no-name gate and the router, and #429 shipped the
907 inverse of its own fix by deriving that agreement twice
908 (mechanisms.md#ONE-PREDICATE-PER-QUESTION); it answered for a
909 third until #436, group's one-entry join.
911 `licensed` holds the pieces rules/C4's count has licensed as
912 credentials -- ambiguous-class words in a credential run behind
913 two name words -- and each reads as a suffix whatever its writing
914 or company.
916 `runs` holds a run of initials spelling a listed credential that
917 stands behind one word (rules/C16, #1238), passed only there:
918 reached while the part is still in its leading run of titles, it
919 makes the part the given part and the answer None.
921 The initial veto (`_vocab.is_initial`) keeps a roman numeral out
922 of a suffix reading, which is right after a NAME word (rules/C9's
923 one-character suffix word, read by what stands before it): 'Smith, John V.' is a
924 middle initial (#432). After a SUFFIX word the numeral is
925 describing that suffix -- 'PSM I' is Professional Scrum Master
926 level I -- so the run continues through it, period included, an
927 initial there being no shape anyone writes (#430). A title resets
928 that: what follows a bare title is not continuing a credential.
930 A part OPENED by a credential is the postnominal part however it
931 goes on (#603, rules/C13): where an unambiguous suffix word that
932 starts #602's run (`starts_a_credential_run`) is the first suffix
933 word of the part, with only titles in front of it (a title/suffix
934 dual among them ends the chance: a credential behind one opens
935 nothing), every later
936 word is read as #602's run reads the given part -- a title word as
937 a title, anything else as a suffix -- so 'John Smith, PhD Jones'
938 and 'Smith, PhD Jones' read suffix 'PhD Jones'. A word of the
939 title vocabulary as well opens nothing (#603, Derek): in front of
940 a given name it is the listing form's title ('Smith, Ms Jane'),
941 and no count of the words before the comma can tell a two-word
942 surname from a given name and a family name, so none decides it.
943 `absorbed`, when passed, receives the index
944 of every piece the opened part takes that would otherwise have
945 made this None -- the caller reports them, as #602's run reports
946 the name words it takes.
948 None covers both ways a segment can fail to be a run: a name word
949 anywhere in it (in a part no credential opened), and no pieces at
950 all ('Doe,, Jr.', which holds no
951 title to read by).
953 `lenient` is Policy.lenient_comma_suffixes, and only the numeral
954 continuation consults it. A part after the comma that is nothing
955 but suffix words, rules/C14: "stops being a credential run at such a word"
956 -- such a word being a suffix word written as an initial. Under
957 strict the numeral gets no suffix reading behind a suffix word, so
958 a part no credential opened is no run and the reading is None
959 ('Smith, RA I'); a part a credential opened (C13) still takes the
960 word in through `opened` below and hands it to `absorbed`
961 ('Smith, PSM I.'), which implements
962 rules/C14's Except bullet "A credential opening the part". The walk reads the token's `initial`
963 tag, so a one-case lowercase letter is vetoed as its capital is
964 (rules/T4). Reading no policy here silently overrode the one knob a
965 caller sets to prevent exactly this (#430). History:
966 decisions/C14 (#761), decisions/C13 (#603).
968 The FAMILY_COMMA rule that segment 0 is the family name rests
969 on the writer having said where the family name ends. A comma
970 followed by no name word said no such thing -- 'John Smith, Dr.' is
971 'Dr. John Smith' with the honorific moved -- so the pre-comma name
972 keeps its positional read instead of being merged (through
973 `_comma.decide`'s suffix comma, behind a whole name). Uses the same
974 is_leading_title predicate the peel does, period-abbreviation
975 inference included, so the two cannot disagree about what a title
976 is; a mixed run like 'Smith, Dr. Jr.' is a title and a postnominal,
977 each read where it stands, never a title run 'Dr. Jr.'.
978 """
979 if not pieces:
980 return None
981 out: list[bool] = []
982 # #544: `credential_anchors` over the whole part, computed the
983 # first time a member's writing declines with a suffix piece in
984 # reach in front of it (`anchor_in_reach`, asked only while the
985 # pass does not exist yet). A member opening the part has nothing
986 # in front and asks neither ('Smith, Ed', 'Smith, Ma John'); a
987 # name word in front returns None before the member is reached;
988 # and a title in front ends the reach test, so 'Smith, Dr. Ma'
989 # pays that one test and no pass.
990 anchors: list[bool] | None = None
991 # Whether every piece so far stands in the part's leading title
992 # run (titles, and title/suffix duals), and whether a dual has
993 # stood there: once one has, nothing in the part anchors. Tracked
994 # here rather than read off `leading_titles`, whose period-shape
995 # inference counts a suffix like 'Esq.' into the run ('Smith, Esq.
996 # RA Ma' keeps its anchored 'Ma' only by this walk's reading).
997 leading = True
998 dual_led = False
999 # #603: the first suffix word of the leading run, which may open
1000 # the part unless a dual stood ahead of it; asked whether it does
1001 # only once a piece reaches the title and name branches below, so a
1002 # part of nothing but suffix words ('Smith, Jr.', 'Smith, PhD MA')
1003 # never pays for the question
1004 opener: tuple[Sequence[int], Set[str]] | None = None
1005 opened = False
1006 for piece, tags in zip(pieces, ptags):
1007 # the verdict just recorded IS "stands behind a suffix" -- keeping
1008 # a separate flag meant maintaining that equality by hand at three
1009 # sites, and a fourth branch that appended without assigning would
1010 # have diverged silently
1011 after_suffix = bool(out) and out[-1]
1012 # rules/C16: "behind one word, initials run together are the
1013 # given name whatever they spell" -- `runs`, which the
1014 # caller passes only there; a suffix word in front of the run
1015 # makes it the credential run, titles and duals leave it a name
1016 # (decisions/C16 (#1238))
1017 if leading and piece[0] in runs:
1018 return None
1019 if is_suffix_piece(piece, tags, tokens):
1020 if (leading and len(piece) == 1
1021 and "vocab:title" in tokens[piece[0]].tags):
1022 dual_led = True
1023 else:
1024 if leading and not dual_led:
1025 opener = (piece, tags)
1026 leading = False
1027 out.append(True)
1028 continue
1029 member = (len(piece) == 1
1030 and AMBIGUOUS_ACRONYM_TAG in tokens[piece[0]].tags)
1031 # rules/S11: "The floor holds first" -- at the first slot after
1032 # a comma a generation word's writing adds nothing to the count
1033 # C4 reads first, `head_named` being that count's two name
1034 # words before the comma (', Dr. fils' and 'Sir Alexandre, fils'
1035 # keep a family word)
1036 if member and listed_lean(tokens[piece[0]], one_case) \
1037 == "credential" and (
1038 head_named
1039 or AMBIGUOUS_WORD_TAG not in tokens[piece[0]].tags):
1040 leading = False
1041 out.append(True)
1042 continue
1043 # A word rules/C4's count licenses as a credential: the
1044 # caller (`_comma.decide`) counted two name words before the
1045 # comma and found the part a credential run, which settles the
1046 # ambiguous word here before S7's anchors are asked (#613)
1047 if piece[0] in licensed and (len(piece) == 1 or "split" in tags):
1048 leading = False
1049 out.append(True)
1050 continue
1051 if (member and not dual_led
1052 and SHAPE_ACRONYM_TAG not in tokens[piece[0]].tags):
1053 if anchors is None and out and anchor_in_reach(
1054 range(len(out) - 1, -1, -1), pieces, ptags, tokens):
1055 anchors = credential_anchors(range(len(pieces)), pieces,
1056 ptags, tokens, first_kept=False)
1057 if anchors is not None and anchors[len(out)]:
1058 leading = False
1059 if anchored is not None:
1060 anchored.append(len(out))
1061 out.append(True)
1062 continue
1063 if (lenient and after_suffix
1064 and _numeral_behind_the_initial_veto(piece, tokens)):
1065 leading = False
1066 out.append(True)
1067 continue
1068 if opener is not None:
1069 # asked once: cleared, as it is set at most once
1070 opened = starts_a_credential_run(opener[0], opener[1], tokens)
1071 opener = None
1072 if opened:
1073 # #602's run: a title word reads as a title, anything else
1074 # as a suffix -- vocabulary only, as in the given part
1075 if is_title_piece(piece, tags, tokens):
1076 out.append(False)
1077 else:
1078 if absorbed is not None:
1079 absorbed.append(len(out))
1080 out.append(True)
1081 elif is_leading_title(piece, tags, tokens):
1082 out.append(False)
1083 else:
1084 return None
1085 return tuple(out)
1088class Peel(NamedTuple):
1089 """What assign's trailing peel made of a walk. `names` is a count
1090 of positions in the caller's `rest`: rest[:names] are the name
1091 pieces and rest[names:] the suffixes. The other two are pieces --
1092 token-index tuples, as PendingAmbiguity wants them -- and each is
1093 one token long: `numerals` are the pieces the roman-numeral fork
1094 took as a call, each reported, in peel order (none for a numeral
1095 its lower case settled or the vocabulary lists in two or more
1096 letters, which is taken unrecorded; a one-letter one is always the
1097 walk's last piece, one of two or more letters anywhere in its
1098 trailing run, #752),
1099 `picks` the bare ambiguous acronyms the peel had to resolve, in
1100 peel order, either way (the last may sit at rest[names - 1]), and
1101 a numeral the fork declined, a one-letter one for want of words to
1102 spare (#718), always the walk's last piece, or one of two or more
1103 letters for its Title case (#752), anywhere in the trailing run,
1104 and a name unless a credential run in front of it takes it.
1105 `anchors` is `peel_trailing`'s working state for `tail_reading`
1106 alone -- the walk's `credential_anchors` pass over `rest` where a
1107 member needed one, handed to the next pass of the fixed point
1108 (`peel_trailing`'s `start`). No caller reads it, and the Peel
1109 `tail_reading` returns carries None."""
1111 names: int
1112 numerals: tuple[tuple[int, ...], ...]
1113 picks: tuple[tuple[int, ...], ...]
1114 anchors: list[bool] | None
1115 #: #602 (`credential_run`): the pieces in rest[names:] that are
1116 #: title words inside the credential run, and the name pieces the
1117 #: run absorbed, which assign reports
1118 run_titles: tuple[int, ...] = ()
1119 absorbed: tuple[tuple[int, ...], ...] = ()
1122# rules/S2: "a trailing word of the suffix vocabulary reads as a
1123# suffix — generational forms and credential acronyms alike" -- and
1124# rules/S3: "an ambiguous acronym written with its periods, one after
1125# each letter or one after each of two or more letter chunks, counts
1126# unambiguously; a single trailing period is the abbreviation shape
1127# any word can wear and does not" -- and rules/S4: "a BARE ambiguous acronym is consumed only when
1128# the name has words to spare"
1129# (v1's are_suffixes tail rule, with the roman-numeral special)
1130def peel_walk(start: int, ptags: Sequence[Set[str]]) -> list[int]:
1131 """The indices peel_trailing walks: `start` to the segment's end,
1132 minus the group-flagged credential pieces (the Ph. D. merge),
1133 which assign reads as suffixes at any position. Built here and nowhere
1134 else, so the walk's input cannot drift between assign and the
1135 group sites that read it: the numeral fork is a last-piece
1136 test that reads the piece before as rest[k - 2], which holds only
1137 over this list."""
1138 return [j for j in range(start, len(ptags))
1139 if "suffix" not in ptags[j]]
1142def trailing_start(start: int, pieces: Sequence[Sequence[int]],
1143 ptags: Sequence[Set[str]], tokens: Sequence[WorkToken],
1144 *, one_case: bool | None) -> int:
1145 """Where assign's trailing suffix run begins, read over the pieces
1146 as they stand from `start`: the index of the first piece the S2
1147 peel takes, or len(pieces) when it takes none (#424). What P2's
1148 chain stops before where S2's run was not read once ahead of it
1149 (#614: the part before a family comma, a part past the second
1150 comma, a run holding a name piece; the given part after a family
1151 comma stops at `trailing_run_start` since #623), and
1152 what M2's walk stops before where no trailing rule reads the
1153 clause -- each had asked "is this a
1154 suffix?" with the suffix-piece test, which vetoes a bare 'V' as
1155 an initial (the #401 question), and so took a trailing numeral,
1156 or a bare acronym with words to spare, into the family or the
1157 maiden name.
1159 Both forks, always, and #602's credential run on top
1160 (`credential_run`), so P2's chain stops where the run starts.
1161 Readers that also need H5's title chain read the same pieces
1162 through `tail_reading`, which runs this pair's peel and the chain
1163 to a fixed point -- the maiden take among them (#601). So this
1164 function's answer is the whole answer only where no trailing
1165 title chain also reads the pieces."""
1166 rest = peel_walk(start, ptags)
1167 names = run_start(rest, peel_trailing(rest, pieces, ptags, tokens,
1168 one_case).names,
1169 pieces, ptags, tokens, one_case=one_case)
1170 return rest[names] if names < len(rest) else len(pieces)
1173# #289/#516: the "listed member, not by-shape" test both
1174# peel_trailing and segment_suffix_reading ask before reading the
1175# lean -- shared here so the two cannot drift on what counts
1176# (quality-review finding: it was spelled twice, once per site,
1177# before this). Those two test "vocab:suffix-ambiguous" in tags
1178# INLINE, before calling this, rather than leaving that cheap check to
1179# this function's own body: measured, a caller whose `elif` reaches
1180# this on every piece (segment_suffix_reading's does, one per
1181# family-comma segment 1, member or not) pays one frame for the call
1182# regardless of what is inside it, and the inline pre-check is what
1183# keeps a non-member piece ("Smith, John"'s "John") from ever making
1184# the call at all.
1185#
1186# A further caller since #533 -- credential_at_the_given_slot just
1187# below -- deliberately does NOT pre-check: it owns #531's reading
1188# and leaves membership to its own callers (its docstring says so),
1189# and the frame argument holds transitively because its caller asks
1190# inline -- `AMBIGUOUS_ACRONYM_TAG in tok.tags` after a
1191# `len(piece) == 1` at assign's given-part trailing slot. So no
1192# non-member piece reaches this function down that route either.
1193def listed_lean(token: WorkToken, one_case: bool | None) -> Lean | None:
1194 """`ambiguous_lean` for a LISTED bare-ambiguous token, or None if
1195 the token is not tagged a listed member, is admitted by SHAPE
1196 instead (`SHAPE_ACRONYM_TAG`, a switch's doing, not the writing's),
1197 or there is no case fact to ask at all."""
1198 if (one_case is None or AMBIGUOUS_ACRONYM_TAG not in token.tags
1199 or SHAPE_ACRONYM_TAG in token.tags):
1200 return None
1201 # rules/S11: "Any other's writing never speaks, and the count decides
1202 # alone" -- only a word written `title_case_is_name` (`fils`, `kc`)
1203 # takes the suffix word's cue; an agnomen and `junior` take none
1204 # (decisions/S11, #1039)
1205 if AMBIGUOUS_WORD_TAG in token.tags:
1206 return ambiguous_lean(token.text, one_case,
1207 "word" if TITLE_CASE_NAME_TAG in token.tags
1208 else "none")
1209 return ambiguous_lean(token.text, one_case)
1212# rules/S6: "a clash settled by the writing is not reported" -- the
1213# gate's form step for a listed acronym, asked where a reading is
1214# chosen, so the word keeps its tags and with them its other readings
1215# (history: decisions/S6 (#785))
1216def settled_by_case(token: WorkToken, one_case: bool | None) -> bool:
1217 """Whether the writing settles a LISTED ambiguous acronym as the
1218 credential: written in capitals in a name of mixed case
1219 (`listed_lean`). Only the clash between the credential and the
1220 name reading is settled; the word stays a member of the class, so
1221 a particle reading ('DO') or the place a maiden clause gives its
1222 first word still answers as before. A suffix word that is also a
1223 name ('fils') is S11's and is not asked here."""
1224 return (AMBIGUOUS_WORD_TAG not in token.tags
1225 and listed_lean(token, one_case) == "credential")
1228def credential_at_the_given_slot(
1229 token: WorkToken, one_case: bool | None,
1230 anchored: Callable[[], bool] | None = None) -> bool:
1231 """#531's reading of a class MEMBER ending the given part after a
1232 family comma: the credential unless the writing says otherwise.
1233 The caller decides membership and that the piece ends that part.
1235 The words to spare are there by construction at that slot, so the
1236 count says nothing and only the lean does; a member that is also
1237 particle vocabulary reads as the credential on a POSITIVE lean
1238 alone, P6's attachment keeping every other spelling.
1240 Or where the member is ANCHORED (#544): an unambiguous credential
1241 in front of it in the same run (`credential_anchors`) makes it the
1242 credential whatever its writing says, a particle member included
1243 -- a degree in front outranks P6's attachment, as the capitals do
1244 ('doe, jane v phd do' reads suffix 'v phd do'). Only the caller
1245 knows the run, so it computes the anchor; `anchored` is a THUNK,
1246 asked last, so a member the writing already settles never pays
1247 for the walk.
1249 Called from assign's walk over the given part, as one function
1250 rather than a condition written at the site
1251 (mechanisms.md#ONE-PREDICATE-PER-QUESTION). It is a text-and-tags
1252 question, which is what puts it in this module rather than beside
1253 any caller.
1255 The membership half of that contract is CHECKED rather than
1256 trusted, because getting it wrong is silent: all three of
1257 `listed_lean`'s None reasons fall through to the `particle` test
1258 below, so a non-member handed in by mistake is answered True --
1259 "read it as the credential" -- for a word the class never admitted.
1260 An assert rather than a raise or a branch: it enters no Python
1261 frame (measured -- 'Doe, John MA' stays at 311), it states the
1262 contract where a reader of the function body meets it, and under
1263 -O it is exactly the code that was here before.
1264 """
1265 assert AMBIGUOUS_ACRONYM_TAG in token.tags, (
1266 f"credential_at_the_given_slot is #531's reading of a LISTED "
1267 f"class member; {token.text!r} carries {sorted(token.tags)} "
1268 f"and is not one. The caller decides membership -- test "
1269 f"AMBIGUOUS_ACRONYM_TAG before calling")
1270 lean = listed_lean(token, one_case)
1271 if lean == "credential" or (lean is None
1272 and "particle" not in token.tags):
1273 return True
1274 return anchored is not None and anchored()
1277#: `_chain_units`' marks: a piece opening a name unit, and a name word
1278#: the chain joins to the run in front of it
1279OPENS, JOINED = 1, 0
1282def second_unit(rest: Sequence[int], units: Sequence[int]) -> int:
1283 """The position in `rest` of the piece opening its second name
1284 unit, `len(rest)` where there is none: `rest[:m]` holds two units
1285 exactly when this is below `m`. The first piece always counts as
1286 a unit's, being where the name the walk reads starts."""
1287 for i in range(1, len(rest)):
1288 if units[rest[i]] == OPENS:
1289 return i
1290 return len(rest)
1293def _kept_in_front(rest: Sequence[int], front: int,
1294 pieces: Sequence[Sequence[int]],
1295 ptags: Sequence[Set[str]],
1296 tokens: Sequence[WorkToken],
1297 one_case: bool | None) -> int:
1298 """The position in `rest` of the last word the trailing walk keeps
1299 at or before `front`, which stands in front of a generation word
1300 that is also a name, so that word's count looks past what the walk
1301 takes there (rules/S11, #890); -1 where it keeps none. Taken by
1302 itself: a suffix piece, a numeral of two or more letters read as the
1303 generation ('Pedro VI Filho'), a member of the ambiguous class its
1304 writing makes a suffix ('Pedro MA Filho'). Taken when carried: a
1305 bare member of the class behind any of those, the members between
1306 passing the run on, as rules/S7's company reads it ('Pedro Junior
1307 Ma Filho'). One pass, so a carried run costs a step a member."""
1308 members: int | None = None # where an uncarried run of members began
1309 j = front
1310 while j >= 0:
1311 piece = pieces[rest[j]]
1312 if is_suffix_piece(piece, ptags[rest[j]], tokens):
1313 members = None # a carrier: the run behind it is taken
1314 j -= 1
1315 continue
1316 if len(piece) != 1:
1317 break
1318 tok = tokens[piece[0]]
1319 stem = tok.text.rstrip(".")
1320 if ((len(stem) > 1 and _ROMAN.match(stem) is not None
1321 and reads_as_generation(
1322 tok.text, tok.tags,
1323 tokens[pieces[rest[j - 1]][0]].text if j >= 1 else None,
1324 one_case))
1325 or (AMBIGUOUS_ACRONYM_TAG in tok.tags
1326 and listed_lean(tok, one_case) == "credential")):
1327 members = None
1328 j -= 1
1329 continue
1330 # a bare member that leans nowhere is taken only if something in
1331 # front carries the run; an agnomen carries nothing and is kept
1332 if (AMBIGUOUS_ACRONYM_TAG not in tok.tags
1333 or AMBIGUOUS_WORD_TAG in tok.tags):
1334 break
1335 if members is None:
1336 members = j
1337 j -= 1
1338 return j if members is None else members
1341def peel_trailing(rest: Sequence[int], pieces: Sequence[Sequence[int]],
1342 ptags: Sequence[Set[str]],
1343 tokens: Sequence[WorkToken],
1344 one_case: bool | None,
1345 start: int | None = None,
1346 anchors: list[bool] | None = None,
1347 units: Sequence[int] | None = None,
1348 second: int = 1,
1349 count_numeral: bool = True) -> Peel:
1350 """The S2 trailing peel over `rest`, a peel_walk list. In the
1351 piece layer rather than in assign because group's bound-given
1352 reserve asks the same question of the view the join would leave
1353 (#425): one walk, so the reserve and the assignment cannot drift. Pure -- the ambiguities are
1354 returned for assign to report, in the order it always reported
1355 them.
1357 `one_case` is ParseState.one_case, the recorded fact: None means
1358 nobody asked, which is every caller that has no state to ask with,
1359 and reads as "no lean" -- rules/S4's count alone, the behavior
1360 of every release before this one.
1362 `start` and `anchors` resume a walk rather than begin one: the
1363 walk picks up at `rest[start - 1]`, as though the pieces from
1364 `start` on were already peeled, with `anchors` the pass an earlier
1365 walk built (or None). Only `tail_reading` passes them, and its
1366 docstring states when that answers what a fresh walk would. The
1367 returned picks and numerals are the resumed walk's own; a
1368 one-letter numeral is never read, `start` standing short of the
1369 walk's last piece.
1371 `second` is where the second name unit opens in `rest`
1372 (`second_unit`), the caller's to find once; positions are units
1373 where `units` is None, and it is 1.
1375 `units`, where given, marks each piece by the unit the chain will
1376 put it in (`_chain_units`, #620): the words to spare are counted in
1377 units, so a particle chain counts as the one name word P2 makes of
1378 it, and a word the chain joins to a run without the reading
1379 weighing it is a name word the walk stops at. The numeral fork still reads the piece in front of the
1380 numeral as written: an initial there keeps it a name word whatever
1381 unit the initial is in ('John van der J. V', rules/P2).
1383 `count_numeral` False takes a one-letter numeral without asking
1384 for words to spare: the given part after a family comma is read
1385 without the count, and its chain bound must not ask it (#718).
1386 Every numeral is weighed by `_vocab.reads_as_generation`, the one
1387 predicate every numeral rule asks (#752).
1388 """
1389 picks: list[tuple[int, ...]] = []
1390 numerals: list[tuple[int, ...]] = []
1391 # where the second name unit opens in `rest`, found once a member
1392 # asks; positions are units where `units` is None
1393 k = len(rest) if start is None else start
1394 while k > 0:
1395 piece = pieces[rest[k - 1]]
1396 # a word the chain joins to a run, and the reading does not
1397 # weigh, is a name word
1398 if units is not None and units[rest[k - 1]] != OPENS:
1399 break
1400 if is_suffix_piece(piece, ptags[rest[k - 1]], tokens):
1401 k -= 1
1402 continue
1403 # rules/S10: "the numeral reads as the generation" -- every
1404 # numeral the suffix test above refused, the listed ones it took
1405 # being the predicate's yes too. One of two or more letters is
1406 # read anywhere in the trailing run ('John Smith VI MD' as 'John
1407 # Smith III MD', #784); one of one letter only where it ends the
1408 # walk, since with a credential behind it it is the name word
1409 # S10's Accepted clause keeps (#721). `_ROMAN` on the stem first,
1410 # C-level, so a name word pays no frame. A one-letter numeral
1411 # "needs words to spare, as a bare ambiguous acronym does" --
1412 # the acronym fork's own count, below, so 'John V' keeps family
1413 # 'V' (#718). Declined, it is a pick, since "either reading is
1414 # reported", and no branch below can take a numeral
1415 if (k >= 2 and len(piece) == 1
1416 and _ROMAN.match(text := tokens[piece[0]].text.rstrip("."))
1417 is not None and text
1418 and (len(text) > 1
1419 or k == len(rest)
1420 # the walk leaves out the Ph./D. pair, which is the
1421 # credential behind the numeral all the same
1422 and not (rest[k - 1] + 1 < len(ptags)
1423 and "suffix" in ptags[rest[k - 1] + 1]))):
1424 tok = tokens[piece[0]]
1425 if reads_as_generation(tok.text, tok.tags,
1426 tokens[pieces[rest[k - 2]][0]].text,
1427 one_case):
1428 # rules/S10: "a numeral written in lower case in a name
1429 # of mixed case is the generation and reports nothing"
1430 # -- decided here once: taken as any suffix word is, and
1431 # not recorded as the fork's call, so no reporter meets
1432 # it
1433 if numeral_settled_by_case(tok.text, one_case):
1434 k -= 1
1435 continue
1436 if (len(text) > 1 or not count_numeral
1437 or k - 2 >= second):
1438 numerals.append(tuple(piece))
1439 k -= 1
1440 continue
1441 picks.append(tuple(piece))
1442 break
1443 # rules/S10: "a Title-case numeral in a name of mixed case
1444 # leans to the name" -- weighed and declined, so reported
1445 if numeral_leans_to_the_name(
1446 tok.text, tok.tags,
1447 tokens[pieces[rest[k - 2]][0]].text, one_case):
1448 picks.append(tuple(piece))
1449 break
1450 # A bare ambiguous acronym ("MA", not "M.A.") is a credential
1451 # only when peeling it still leaves a given AND a family name.
1452 # With two pieces, "one of them is a credential" is the less
1453 # likely reading, so it stays the family name -- "Jack MA" is a
1454 # person, "John Smith MA" is a person with a degree. This is
1455 # v1's reserve_last narrowed to the ambiguous set: 2.0
1456 # deliberately peels UNambiguous suffixes even when nothing is
1457 # left ("Smith PhD" -> suffix, a classified fix), because there
1458 # the vocabulary is not in doubt.
1459 bare_ambiguous = (len(piece) == 1
1460 and AMBIGUOUS_ACRONYM_TAG in tokens[piece[0]].tags)
1461 # rules/S12: a split credential reads "as a member of the
1462 # ambiguous class" -- read by the count alone, its writing carrying no
1463 # case to lean on, and in silence: its other reading is two
1464 # name words, which the count already refuses it behind one
1465 if len(piece) == 2 and "split" in ptags[rest[k - 1]]:
1466 if k >= 2 and k - 2 >= second:
1467 k -= 1
1468 continue
1469 break
1470 # #516, switch off: the writing still makes this token
1471 # credential-SHAPED, and the parser is choosing the name
1472 # reading over that one -- the fork the caller asked to be
1473 # told about. Reported here, unconsumed, rather than folded
1474 # into `bare_ambiguous` above: with the switch off the token
1475 # never carries "vocab:suffix-ambiguous" (classify's own
1476 # gate), so `bare_ambiguous` is already False and this is the
1477 # ONLY place the report can be recorded. Switch ON, the token
1478 # carries BOTH tags, so `not bare_ambiguous` is what stands
1479 # this branch down and lets the consuming branch below take
1480 # it; the two are not exclusive.
1481 if (not bare_ambiguous and k >= 2 and len(piece) == 1
1482 and SHAPE_ACRONYM_TAG in tokens[piece[0]].tags):
1483 picks.append(tuple(piece))
1484 break
1485 # #289: written case is evidence the count does not have, and
1486 # it overrides the count in BOTH directions -- an all-caps
1487 # member of a mixed-case name is taken with nothing to spare
1488 # ("Jack MA"), a Title-case one is declined with plenty
1489 # ("John Smith Ma"). The lean is the LISTED set's alone: a
1490 # token admitted to this class by SHAPE carries no writing
1491 # convention to read, so it takes the count (decisions/S6).
1492 #
1493 # k < 2 means it is the only piece left, which is not the fork
1494 # this reports -- and not a floor the lean moves: the walk
1495 # starts after the leading title run, so one piece behind a
1496 # title ("Mr MA") is exactly this case and must stay a name.
1497 # The lean is computed only past this floor -- membership,
1498 # then the floor, then the count-or-lean, in that order, with
1499 # nothing computed a step earlier could discard.
1500 if bare_ambiguous and k >= 2:
1501 # rules/S6: "a clash settled by the writing is not
1502 # reported" -- taken as any suffix word is, and no pick
1503 if settled_by_case(tokens[piece[0]], one_case):
1504 k -= 1
1505 continue
1506 picks.append(tuple(piece))
1507 lean = listed_lean(tokens[piece[0]], one_case)
1508 # peeling still leaves given + family, or the writing says
1509 # to peel anyway
1510 # rules/S11: "only where the name has words to spare" -- a
1511 # generation word counts the NAME words in front of it, so
1512 # what the walk takes there ('Pedro Junior Filho', 'Pedro VI
1513 # Filho', 'Pedro Junior Ma Filho') is not counted as the word
1514 # spared and then peeled as well (#890). One scan to the
1515 # last word the walk keeps, so a run costs a step a word
1516 front = k - 2
1517 if AMBIGUOUS_WORD_TAG in tokens[piece[0]].tags:
1518 front = _kept_in_front(rest, front, pieces, ptags, tokens,
1519 one_case)
1520 # rules/S11: "the floor holds first" -- a generation word's
1521 # case may settle it as the suffix only where the name keeps
1522 # a family word ('Pedro Junior fils' keeps family 'fils',
1523 # Derek, #890); an acronym's capitals take it with none
1524 gated = AMBIGUOUS_WORD_TAG in tokens[piece[0]].tags
1525 if (lean == "credential" and (not gated or front >= second)
1526 or (lean is None and front >= second)):
1527 k -= 1
1528 continue
1529 # #544: or an unambiguous credential stands IN FRONT of it
1530 # in the same run -- 'John Smith PhD MEng' is a list of
1531 # degrees, not a family name 'MEng'. Computed once per
1532 # walk, and only when a member has declined and a suffix
1533 # piece is in reach, so an ordinary name never pays for
1534 # it. Still a pick: the fork is reported as a counted
1535 # member's is.
1536 #
1537 # `rest[k - 2:0:-1]` is what stands in front of the member
1538 # down to, not including, `rest[0]`, the name the reserve
1539 # keeps, which never anchors (`credential_anchors`). So a
1540 # member at k == 2 finds nothing in reach. A by-shape member
1541 # (a lean of None) gets here at k == 2, or, counted in the
1542 # chain's units (#620), behind a run the chain joins into
1543 # one word ('Freiherr von Berg G.J.', at k == 3), where
1544 # what it finds in reach is that run's words.
1545 # The pass, once built, answers first: asked of every
1546 # member, the reach test would walk back through the run
1547 # each time, one look-behind per member.
1548 # A generation word that is also a name has no company to
1549 # keep (rules/S7's credentials): a suffix word in front of
1550 # it is no evidence, its count above having looked past it
1551 if AMBIGUOUS_WORD_TAG in tokens[piece[0]].tags:
1552 break
1553 if anchors is None and anchor_in_reach(
1554 rest[k - 2:0:-1], pieces, ptags, tokens):
1555 anchors = credential_anchors(rest, pieces, ptags, tokens)
1556 if anchors is not None and anchors[k - 1]:
1557 k -= 1
1558 continue
1559 break
1560 return Peel(k, tuple(numerals), tuple(picks), anchors)
1563def behind_the_merged_credential(rest: Sequence[int], p: int,
1564 ptags: Sequence[Set[str]]) -> bool:
1565 """Whether the Ph./D. pair stands right in front of `rest[p]`,
1566 left out of the walk (`peel_walk`) though it is a credential there
1567 all the same -- the test `run_start` asks inline, its hot loop
1568 paying no frame for it (#827)."""
1569 return (p > 0 and rest[p] - rest[p - 1] == 2
1570 and "suffix" in ptags[rest[p] - 1])
1573def _name_word_behind(piece: Sequence[int], ptags: Set[str],
1574 tokens: Sequence[WorkToken],
1575 one_case: bool | None) -> bool:
1576 """Whether the piece behind a member its capitals settle, or behind
1577 a credential acronym written in Title case, reads as a word of the
1578 name rather than a further credential, asked of its
1579 first word, since group has merged particle runs and connective
1580 joins by then ('de la Vega', 'Ortega y Gasset'): a particle, a word
1581 of a script with no case, or a cased word not written in lower case
1582 (Title case or capitals, title vocabulary included), that is no
1583 suffix word and no member its own capitals settle. A credential
1584 merged across a space ('Pharm D', 'Ph. D.') is no name word. Asked
1585 only through `declined_by_the_name_behind`, of a settled member or a
1586 credential acronym written in Title case (`acronym_in_title_case`)."""
1587 if "suffix" in ptags or "split" in ptags:
1588 return False
1589 tok = tokens[piece[0]]
1590 if "particle" in tok.tags:
1591 return True
1592 text = tok.text
1593 if text.islower():
1594 return False
1595 return ("vocab:suffix" not in tok.tags
1596 and SHAPE_ACRONYM_TAG not in tok.tags
1597 and not (AMBIGUOUS_ACRONYM_TAG in tok.tags
1598 and settled_by_case(tok, one_case)))
1601# rules/S8: "nor does a credential acronym written in Title case" --
1602# the clause this implements (decisions/S8, #1019)
1603def acronym_in_title_case(token: WorkToken) -> bool:
1604 """Whether an unambiguous credential ACRONYM is written as a name
1605 word is: one run of letters, a trailing period allowed, its first
1606 letter a capital and every later one lower case ('Md', 'Phd',
1607 'Md.'), where a credential's writing has capitals after the first
1608 letter ('MD', 'PhD') or periods or joiners inside it ('M.D.',
1609 'M.d.', 'Pa-c'). The case mask is not consulted, so 'Bt', 'Kt' and
1610 'Knt', whose own masks are Title case, read as names here too. A
1611 suffix WORD (a generation, honorific or other word written out:
1612 'Jr', 'Esq', 'III', 'Старший') has no capitals of its own for Title
1613 case to drop, so its case says nothing and it is not asked. Not
1614 `_vocab.written_as_a_name`, #564's wider case contrast, which
1615 admits 'McDonald'. Only `run_start` asks it, where a name word
1616 stands behind the acronym."""
1617 if "vocab:suffix" not in token.tags or "vocab:suffix-word" in token.tags:
1618 return False
1619 stem = token.text.rstrip(".")
1620 return (len(stem) >= 2 and stem.isalpha() and stem[0].isupper()
1621 and stem[1:].islower())
1624# rules/S8: "nor does a generation word written out" -- the clause
1625# this implements (decisions/S8)
1626def generation_word_before_a_name(token: WorkToken,
1627 piece: Sequence[int], ptags: Set[str],
1628 tokens: Sequence[WorkToken],
1629 one_case: bool | None) -> bool:
1630 """Whether a generation WORD ('Junior', 'Jr', 'Snr', 'mlađi') has
1631 behind it, in `piece`, a word of the name, asked of the piece's
1632 first word as `_name_word_behind` asks it: a word that is no suffix
1633 word, no numeral S10 reads as the generation, no initial, no connective, no title word
1634 (the run reads that as a title: 'Eric H. Holder Jr. Attorney
1635 General') and no maiden marker, so no particle that is also title
1636 or suffix vocabulary ('St.', 'Mc'), and is a particle or is written
1637 in a name of one case or,
1638 in a name of mixed case, in Title case or a script with no case --
1639 capitals there are a credential's writing as likely as a name's
1640 ('de Planta jr. CF'), and lower case is no name word's. A credential
1641 merged across a space is no name word. The numerals are S10's, read
1642 by their own predicate, and stay out. Asked only through
1643 `declined_by_the_name_behind`."""
1644 if (GENERATION_WORD_TAG not in token.tags
1645 or _ROMAN.match(token.text.rstrip(".")) is not None
1646 or "suffix" in ptags or "split" in ptags):
1647 return False
1648 tags = tokens[piece[0]].tags
1649 text = tokens[piece[0]].text
1650 if (not tags.isdisjoint(_NOT_A_NAME_BEHIND)
1651 # a numeral S10 reads by its shape carries no suffix tag
1652 # ('JOHN SMITH JUNIOR VI JONES'); asked of S10's predicate,
1653 # so a Title-case 'Vi' leaning to the name stays one
1654 or (_ROMAN.match(text.rstrip(".")) is not None
1655 and reads_as_generation(text, tags, token.text,
1656 one_case))):
1657 return False
1658 if "particle" in tags:
1659 return True
1660 return bool(one_case) or not (text.islower() or text.isupper())
1663#: what a word behind a generation word is that is no word of the name
1664_NOT_A_NAME_BEHIND = frozenset({"vocab:suffix", SHAPE_ACRONYM_TAG,
1665 "initial", "conjunction", "vocab:title",
1666 "vocab:maiden-marker"})
1669# rules/S8: "not where a name word stands behind it" -- the three
1670# writings that decline the run there, one predicate for `run_start`,
1671# P2's chain stop (`starts_the_run_here`) and the family part's suffix
1672# pass in assign, which borrows it under rules/P2 before a family comma,
1673# where S8's declines do not apply (decisions/P2, #1177)
1674def declined_by_the_name_behind(token: WorkToken, piece: Sequence[int],
1675 ptags: Set[str],
1676 tokens: Sequence[WorkToken],
1677 one_case: bool | None) -> bool:
1678 """Whether a word that would start #602's run declines it because
1679 of the word behind it, `piece`: a member its capitals settle, or a
1680 credential acronym written in Title case, with a word of the name
1681 behind it (`_name_word_behind`), or a generation word written out
1682 before a name (`generation_word_before_a_name`). The caller asks
1683 whether the word starts a run at all."""
1684 return ((AMBIGUOUS_ACRONYM_TAG in token.tags
1685 and settled_by_case(token, one_case)
1686 # rules/S8: "nor does a credential acronym written in
1687 # Title case" -- the writing of a name word
1688 or acronym_in_title_case(token))
1689 and _name_word_behind(piece, ptags, tokens, one_case)
1690 # rules/S8: "nor does a generation word written out" -- a
1691 # filer's middle-name box, the family name behind it
1692 or generation_word_before_a_name(token, piece, ptags, tokens,
1693 one_case))
1696# rules/S8: "A credential after the name core starts a run to the
1697# end of its part" -- applied to a finished peel (#602), so every
1698# reader of the trailing run (assign, P5's reserve, the maiden take)
1699# sees one answer; P2's chain stop also asks the run's start word by
1700# word (`starts_the_run_here`), the count of name words aside.
1701def run_start(rest: Sequence[int], names: int,
1702 pieces: Sequence[Sequence[int]],
1703 ptags: Sequence[Set[str]],
1704 tokens: Sequence[WorkToken],
1705 units: Sequence[int] | None = None,
1706 one_case: bool | None = None,
1707 initials_cue: bool = True) -> int:
1708 """Where #602's run starts in `rest[:names]`: the position of the
1709 first credential that starts one (`starts_a_credential_run`) with
1710 two name pieces in front of it, or `names` where none does -- with
1711 no comma the family has to exist first. A lone particle does not
1712 count toward the two, 'de Mesnil' being one surname, unless the
1713 run starts right behind it (#627). The position
1714 alone, for readers that need only where the name ends
1715 (`trailing_start`); `credential_run` builds the rest of the answer.
1717 The tag tests are INLINE ahead of the predicate, so a name word --
1718 every piece of an ordinary name -- and a connective or particle
1719 never pay the predicate's frame: this runs on every parse, once
1720 per piece, and a clause holding a run of links ('Puig i i i ...
1721 Soler') otherwise paid two frames per link (decisions/parse-cost).
1722 The predicate repeats the refusal; the inline copy is only the
1723 cheap half of the same question. Under three names no run can
1724 start, two core pieces having to stand in front of it.
1726 `units` (`_chain_units`, #620) counts over pieces the chain has not
1727 joined yet: a word the chain will join to the run in front of it
1728 is no name word of its own, and a particle opening such a run is
1729 the one the run makes ('der la', merged into one piece ahead of
1730 the read, one surname)."""
1731 if names < 3:
1732 return names
1733 core = 0
1734 lone = False
1735 for p in range(names):
1736 q = rest[p]
1737 tags = tokens[pieces[q][0]].tags
1738 if core + lone >= 2:
1739 # the Ph./D. pair standing in front of this piece starts the
1740 # run there, though the walk leaves it out (#827): 'John
1741 # Smith Ph. D. Jones' as 'John Smith PhD Jones'
1742 # (`behind_the_merged_credential`, inline)
1743 if (q - rest[p - 1] == 2 and "suffix" in ptags[q - 1]
1744 and starts_a_credential_run(pieces[q - 1], ptags[q - 1],
1745 tokens)):
1746 return p
1747 if (("vocab:suffix" in tags or "split" in ptags[q]
1748 or AMBIGUOUS_ACRONYM_TAG in tags
1749 or _ROMAN.match(tokens[pieces[q][0]].text.rstrip("."))
1750 is not None)
1751 and tags.isdisjoint(_NOT_A_RUN_START)
1752 # p >= 2 here: two name words stand in front
1753 and starts_a_credential_run(
1754 pieces[q], ptags[q], tokens,
1755 tokens[pieces[rest[p - 1]][0]].text, one_case)
1756 # rules/S8: "not where a name word stands behind it"
1757 # -- a settled member followed by one is the
1758 # writer's initials as likely as a credential
1759 and not (initials_cue and p + 1 < len(rest)
1760 # the Ph./D. pair behind it is a credential
1761 # the walk leaves out (#827)
1762 and not behind_the_merged_credential(
1763 rest, p + 1, ptags)
1764 and declined_by_the_name_behind(
1765 tokens[pieces[q][0]], pieces[rest[p + 1]],
1766 ptags[rest[p + 1]], tokens, one_case))):
1767 return p
1768 # a lone particle is not yet a name: 'de Mesnil' is one surname
1769 # -- unless it opens a chain run, which is then the surname
1770 # (with no `units` every piece opens its own), or the run
1771 # starts right behind it, leaving it nothing to join: then it
1772 # counts as a name word of its own, as 'van der' and 'von und
1773 # zu' do (#627, 'John von PhD Jones')
1774 if units is None or units[q] == OPENS:
1775 lone = (len(pieces[q]) == 1 and "particle" in tags
1776 and (units is None or q + 1 == len(units)
1777 or units[q + 1] == OPENS))
1778 core += not lone
1779 return names
1782# rules/P6: "a particle ending the name attaches to that family
1783# name" -- WHICH particles, asked once, by assign's given-part walk,
1784# which both leaves them out of #602's run and attaches them (#613;
1785# #610 had first made the run and post_rules' attachment one walk)
1786def particle_tail(seg: Sequence[Sequence[int]],
1787 tokens: Sequence[WorkToken]) -> tuple[int, int]:
1788 """`seg[lo:hi]`, the run of wholly-particle pieces P6 attaches: back
1789 from the end past pieces that hold no name word and are not
1790 themselves particles -- a post-nominal is written BEHIND the
1791 particle in this listing -- then back over particle pieces,
1792 stopping at a lone member of the ambiguous credential class read as
1793 the credential (#531: the capitals or a degree made it one, so it
1794 is not the tussenvoegsel).
1796 The class-member stop keys on the VOCABULARY tag and not on the
1797 suffix role alone, which is what gives P6's attachment precedence
1798 over S2: `vd` and `mc` are unambiguous suffix vocabulary, also
1799 particles, also suffix-roled, and the tag is what keeps them inside
1800 the run, while the role alone would have stood the attachment down
1801 for #531's member too ('Doe, John DO' read family 'DO Doe' with the
1802 condition absent, verified when #531 landed). A trailing piece that
1803 IS particle vocabulary ends the first walk rather than being looked
1804 past, since `vd` arrives suffix-roled and is the run.
1806 Read off ROLES, asked at the moment assign's walk reaches #602's
1807 run (or, with no run, once the walk has placed every piece): every
1808 word in front of the run holds the role the walk gave it, and the
1809 run's words hold none yet -- no name role, and a class member not
1810 yet read as anything, the run being what will read it as the
1811 credential. That is why the answer equals the one this would give
1812 over the finished roles, which is what the attachment needs.
1813 `lo == hi` where there is no such run.
1815 The single-token tests are inline (the common piece is one token),
1816 so the walk pays no frame per piece; it runs on every family-comma
1817 parse."""
1818 hi = len(seg)
1819 while hi > 0:
1820 piece = seg[hi - 1]
1821 if (all("particle" in tokens[i].tags for i in piece)
1822 if len(piece) > 1 else
1823 "particle" in tokens[piece[0]].tags):
1824 break
1825 if (any(tokens[i].role in NAME_ROLES for i in piece)
1826 if len(piece) > 1 else
1827 tokens[piece[0]].role in NAME_ROLES):
1828 break
1829 hi -= 1
1830 lo = hi
1831 while lo > 0:
1832 piece = seg[lo - 1]
1833 if len(piece) == 1:
1834 tok = tokens[piece[0]]
1835 if "particle" not in tok.tags or (
1836 AMBIGUOUS_ACRONYM_TAG in tok.tags
1837 and tok.role in SUFFIX_OR_UNREAD):
1838 break
1839 elif not all("particle" in tokens[i].tags for i in piece):
1840 break
1841 lo -= 1
1842 return lo, hi
1846def has_name_content(piece: Sequence[int],
1847 tokens: Sequence[WorkToken]) -> bool:
1848 """Whether a piece holds a letter or digit: a piece with none is
1849 no name word (rules/A2), so a run that takes it in absorbs no
1850 name and reports nothing. C-level, no frame per character."""
1851 return any(map(str.isalnum, "".join([tokens[i].text for i in piece])))
1854def credential_run(rest: Sequence[int], peel: Peel, p: int,
1855 pieces: Sequence[Sequence[int]],
1856 ptags: Sequence[Set[str]],
1857 tokens: Sequence[WorkToken],
1858 one_case: bool | None = None) -> Peel:
1859 """`peel` with its name count cut back to `p`, where #602's run
1860 starts (`run_start`, which the caller asks first so that a name
1861 with no run pays one frame, not two).
1863 Every word from the credential on is then in the suffix run,
1864 except a title word, which is recorded in `run_titles` for assign
1865 to read as a title; a name word the run absorbs is recorded in
1866 `absorbed` for assign to report -- not a word with no letter or
1867 digit in it, and not a member the peel already reported as a pick.
1868 One pass, the suffix test first: a credential, the common word in
1869 a run, leaves both lists without the title test's frame."""
1870 run_titles: list[int] = []
1871 absorbed: list[tuple[int, ...]] = []
1872 # a numeral starting the run is read as the generation, which a
1873 # shape alone makes a call, reported as the peel's are (rules/S10)
1874 numerals = peel.numerals
1875 # where the Ph./D. pair the walk leaves out starts the run
1876 # (`run_start`), rest[p] is the first word behind it, read like the
1877 # rest; else rest[p] is the credential
1878 first = p if behind_the_merged_credential(rest, p, ptags) else p + 1
1879 start = pieces[rest[p]]
1880 if (first > p and len(start) == 1 and _ROMAN.match(
1881 (tok := tokens[start[0]]).text.rstrip(".")) is not None
1882 and not is_listed_numeral(tok.text, tok.tags)
1883 and not numeral_settled_by_case(tok.text, one_case)):
1884 numerals = (tuple(start), *numerals)
1885 for q in rest[first:peel.names]:
1886 if is_suffix_piece(pieces[q], ptags[q], tokens):
1887 continue
1888 if is_title_piece(pieces[q], ptags[q], tokens):
1889 run_titles.append(q)
1890 continue
1891 piece = tuple(pieces[q])
1892 if piece not in peel.picks and has_name_content(piece, tokens):
1893 absorbed.append(piece)
1894 return Peel(p, numerals, peel.picks, peel.anchors,
1895 tuple(run_titles), tuple(absorbed))
1898# rules/H5: "only a word the vocabulary knows as a title is one,
1899# and a bare title word is a name word"
1900# -- the trailing run's own predicate. NOT is_leading_title:
1901# that predicate carries H2's unlisted-abbreviation inference, which is
1902# the LEADING slot's shape rule and has no trailing counterpart, so
1903# with it 'John Smith Xyz.' would lose its family name to a title
1904# (decisions/H5). Classify's CHAIN_TITLE_TAG reads a word's title
1905# vocabulary through the same "vocab:title" tag is_title_piece reads,
1906# so the leading and trailing runs cannot disagree about what a title
1907# WORD is while disagreeing, deliberately, about what a title SHAPE is
1908# -- save the word H5's written-short clause adds, classify's
1909# TRAILING_TITLE_TAG, which no leading read admits (#999).
1910def trailing_titles(rest: Sequence[int], pieces: Sequence[Sequence[int]],
1911 ptags: Sequence[Set[str]],
1912 tokens: Sequence[WorkToken],
1913 end: int | None = None,
1914 salutations: bool = False) -> int:
1915 """How many pieces of `rest` the trailing title chain LEAVES
1916 standing: `rest[:kept]` are the name pieces and `rest[kept:]` the
1917 title words the chain took (wearing the abbreviation shape, a title
1918 listed whole with a period inside it, or under `salutations` a bare
1919 salutation), in piece order. Counted
1920 the way `peel_trailing` counts, so the two answers compose without
1921 arithmetic at the call site. `rest` is the caller's NAME pieces:
1922 on the no-comma path what the S2 peel left, after a family comma
1923 the segment's pieces that the segment's own suffix reading does
1924 not claim, and in `tail_reading` the leftovers of whichever peel
1925 is current. `end`, where given, reads `rest[:end]` without the
1926 copy, for `tail_reading`'s passes. `salutations` is the given
1927 part after a family comma asking, where a bare salutation the
1928 vocabulary marks `after_given` chains as well (rules/H8).
1929 Floor: the first position of `rest` is never taken, so one name
1930 piece stands and a name is never all title. An empty
1931 `rest` returns 0, which is what leaves assign's bare-suffix
1932 carve-out reached exactly as before.
1934 ONE WORD per piece, the same gate the leading peel's give-back
1935 uses: a joined unit is not the shape this reads, and the tokens of
1936 one are not each a title word.
1938 Every parse with a name word to place enters this frame -- assign
1939 asks the question here rather than answering a cheaper version of
1940 it inline (mechanisms.md#ONE-PREDICATE-PER-QUESTION) -- so what it
1941 costs an ordinary name is one frame and one tag test. The
1942 exceptions return before it: a segment that is all title, and a
1943 comma part read wholly as a credential run, have no name piece to
1944 hand this (52 of the 1289 corpus parses, measured 2026-09-09 --
1945 'Coach', 'Lord of the Universe', 'Smith, Jr.', 'MD, PHD').
1946 The word test is classify's CHAIN_TITLE_TAG, a set membership test
1947 (a C call, no Python frame), so the ordinary parse pays one tag
1948 test and stops (decisions/parse-cost, #1171).
1949 """
1950 k = len(rest) if end is None else end
1951 while k > 1:
1952 idx = rest[k - 1]
1953 piece = pieces[idx]
1954 # classify's CHAIN_TITLE_TAG is the word test, a C-level read
1955 # (#1171); no #323 veto on the shape there, unlike
1956 # is_leading_title's: the shape is ANDed with a vocabulary
1957 # read, so the word is listed, and a listed CJK title wearing a
1958 # stop should read as a title
1959 if len(piece) == 1 and (
1960 CHAIN_TITLE_TAG in tokens[piece[0]].tags
1961 # rules/H8: "the given part after it ends in a word whose
1962 # title reading is a salutation" -- the salutation chains
1963 # as a period-marked title does ("It reads as the
1964 # period-marked spelling reads"); a tag test, so a name
1965 # word pays no frame
1966 or (salutations and "vocab:after-given-title"
1967 in tokens[piece[0]].tags)):
1968 k -= 1
1969 continue
1970 break
1971 return k
1974# rules/H5: "successive single words that wear the abbreviation
1975# shape and are title vocabulary chain into the title from the end",
1976# and "a title the vocabulary lists whole with a period inside it
1977# chains without a final one" -- one word's half of that, for a caller
1978# asking it of a piece the chain did not walk to: the maiden take,
1979# choosing the words the clause-free view is read over (#601), and
1980# P13's walk back to a connective ending the name words (#1168).
1981# Both read classify's CHAIN_TITLE_TAG, the one statement of the word
1982# test (#1171); `trailing_titles` reads it inline, for the frame budget
1983# its docstring states, and adds its `salutations` arm, which no
1984# caller of this one asks.
1985def is_trailing_title_word(piece: Sequence[int],
1986 tokens: Sequence[WorkToken]) -> bool:
1987 """Whether one piece is a word H5's trailing chain takes: a lone
1988 token wearing the abbreviation shape that the title vocabulary
1989 lists, or that classify marks as a title written short that may
1990 be a given name (TRAILING_TITLE_TAG), or a title the vocabulary
1991 lists whole with a period inside it, which classify marks
1992 (JOINED_TITLE_TAG) and which needs no final period -- the words
1993 classify marks CHAIN_TITLE_TAG. Position is the caller's question
1994 -- this answers only what the word is."""
1995 return len(piece) == 1 and CHAIN_TITLE_TAG in tokens[piece[0]].tags
1998#: lone-word tags `trailing_candidates` admits on: the ambiguous class,
1999#: listed or by shape, and any suffix word -- an initial-shaped one
2000#: included, which `is_suffix_piece` vetoes and the numeral fork reads
2001_TRAILING_WORD_TAGS = frozenset({AMBIGUOUS_ACRONYM_TAG, SHAPE_ACRONYM_TAG,
2002 "vocab:suffix"})
2005# rules/M2: "It takes them up to the trailing run of post-nominals
2006# and titles that the end of the name reads as if the clause were not
2007# written" -- which words of a clause might be in that run, written
2008# beside the peel and the chain whose admissions it has to cover (#610)
2009def trailing_candidates(lo: int, pieces: Sequence[Sequence[int]],
2010 ptags: Sequence[Set[str]],
2011 tokens: Sequence[WorkToken]) -> int:
2012 """The first of the pieces after `lo` that `tail_reading` might read
2013 into the trailing run: `pieces[c:]`, a SUPERSET of what it takes,
2014 for the maiden take to build its clause-free view over and let
2015 `tail_reading` decide. `pieces[lo]` itself is never a candidate.
2017 Back from the end over every piece the peel or the chain admits on
2018 its own terms -- a suffix piece, a member of the ambiguous class,
2019 an initial-shaped suffix word, a period-marked title word or a
2020 title listed whole with a period inside it -- and
2021 over a roman numeral the numeral's predicate admits, asked with no
2022 case so it stays a superset: one of two or more letters wherever it
2023 stands, the peel reading it anywhere in the run (#752), and one of
2024 one letter where nothing stands behind it but such title words and
2025 group-flagged suffix pieces: the chain takes the titles first and
2026 the walk never holds the flagged pieces ('Ph. D.'), so either way
2027 that numeral is the walk's last piece and the fork reads it. Then
2028 back to the first
2029 credential that starts #602's run, whose words the run takes
2030 whatever they are. A BARE title word is no candidate: H5's chain
2031 does not take one, and in the view it would only inflate the count
2032 of words to spare.
2034 The superset is pinned by tests/v2/pipeline/test_pieces.py over a
2035 generated grid against `tail_reading` itself, so a new admission
2036 in the peel or the chain fails there until this covers it."""
2037 c = len(pieces)
2038 titles_behind = True
2039 while c - 1 > lo:
2040 k = c - 1
2041 piece = pieces[k]
2042 title = is_trailing_title_word(piece, tokens)
2043 if not (title or "suffix" in ptags[k] or "split" in ptags[k]
2044 or is_suffix_piece(piece, ptags[k], tokens)
2045 or (len(piece) == 1 and (
2046 not tokens[piece[0]].tags.isdisjoint(_TRAILING_WORD_TAGS)
2047 or ((titles_behind
2048 or len(tokens[piece[0]].text.rstrip(".")) > 1)
2049 and reads_as_generation(
2050 tokens[piece[0]].text, tokens[piece[0]].tags,
2051 tokens[pieces[k - 1][0]].text, None))))):
2052 break
2053 # a group-flagged suffix piece is outside the peel's walk
2054 # altogether (`peel_walk` drops it), so it is behind the
2055 # numeral the way a chained title is
2056 titles_behind = titles_behind and (title or "suffix" in ptags[k])
2057 c = k
2058 # the tag test is the predicate's own necessary half, inline so a
2059 # name word pays no frame
2060 return next((j for j in range(lo + 1, c)
2061 if ("suffix" in ptags[j] or "split" in ptags[j]
2062 or "vocab:suffix" in tokens[pieces[j][0]].tags
2063 or _ROMAN.match(tokens[pieces[j][0]].text.rstrip("."))
2064 is not None)
2065 and starts_a_credential_run(
2066 pieces[j], ptags[j], tokens,
2067 tokens[pieces[j - 1][0]].text)
2068 # a member its capitals settle, asked with the case as
2069 # mixed so the superset holds whatever the name's case
2070 or AMBIGUOUS_ACRONYM_TAG in tokens[pieces[j][0]].tags
2071 and starts_a_credential_run(pieces[j], ptags[j], tokens,
2072 None, False)),
2073 c)
2076# rules/H5: "the title is TRANSPARENT to the suffix reading: where
2077# two or more name words stand, what stands once the chain is taken
2078# reads exactly as it would read written without the title, plus the
2079# title"
2080def tail_reading(rest: list[int], pieces: Sequence[Sequence[int]],
2081 ptags: Sequence[Set[str]],
2082 tokens: Sequence[WorkToken],
2083 one_case: bool | None,
2084 units: Sequence[int] | None = None,
2085 count_numeral: bool = True,
2086 salutations: bool = False,
2087 initials_cue: bool = True,
2088 ) -> tuple[list[int], tuple[int, ...], Peel]:
2089 """The S2 peel and the H5 chain read together to a FIXED POINT:
2090 peel, chain, splice the chained pieces out, peel again over what
2091 is left -- the name pieces the chain kept, then the pieces the
2092 peel had taken, in original order -- until the chain takes
2093 nothing. Where it takes nothing on the first pass, which is almost
2094 every name, that first peel is the answer and the loop costs one
2095 comparison.
2097 Returns the walk with the chained pieces spliced out, the pieces
2098 the chain took, and the FINAL peel -- whose numeral fork and
2099 ambiguous picks are the ones assign reports. The walk is
2100 partitioned by that peel exactly as a caller partitions its own:
2101 `rest[:peel.names]` the name pieces, `rest[peel.names:]` the
2102 suffixes. A bare tuple rather than a named one because a
2103 NamedTuple's __new__ is a frame of its own on every parse
2104 (decisions/parse-cost), and `_group_segment` returns its three
2105 the same way.
2107 Transparency is what the fixed point buys: 'X Prof. Y' reads
2108 exactly as 'X Y' reads plus the title, however many titles are
2109 written and wherever the peel then stops. Iterating ONCE reads a
2110 second title only half way -- 'John Prof. MA Prof.' un-peeled the
2111 acronym and re-exposed the first title, reading family 'Prof.'
2112 with suffix 'MA' where 'John Prof. MA' read family 'MA' (written
2113 in one case today: since #289 capitals in a mixed-case name take
2114 the acronym whatever the count, so 'john prof. ma prof.' and
2115 'john prof. ma' are the pair that still reads family 'ma').
2117 One function for assign's placement, group's bound-given reserve
2118 (P5), which must count the name words assign will leave, and the
2119 maiden take (rules/M2), which must end the clause where the
2120 reading of the name will begin -- and since #614, for a main
2121 segment, the one read group takes between P3's joins and the
2122 particle chain (`read_trailing_run`), which assign and the reserve
2123 take their answer from instead. Deriving that agreement twice is what left the two
2124 disagreeing at S4's bare-ambiguous reserve: 'abdul rahman MA'
2125 declined the join and 'abdul rahman MA Prof.' took it
2126 (mechanisms.md#ONE-PREDICATE-PER-QUESTION).
2128 `rest` is a peel_walk list and is not mutated -- the splice
2129 rebinds this local -- so a caller's own reference still names the
2130 walk it built. Every caller reads the one returned here instead,
2131 which is the one the final peel partitions.
2133 The chain's floor is `trailing_titles`' own, one name piece left
2134 standing.
2136 #602's credential run is applied to the final peel on the way out
2137 (`credential_run`), so every reader sees the run where it starts.
2139 Linear in the walk (#558). Re-peeling the whole walk every pass
2140 re-read each suffix the passes before had already peeled, so a
2141 name ending 'MA Prof. MA Prof. ...' cost the square of its tail.
2142 A pass instead RESUMES the walk where the splice left it, over
2143 `rest` as written: the pieces in front of the splice are the ones
2144 a fresh walk would meet, and `credential_anchors` reads each
2145 position off the pieces in front of it, so the carried pass
2146 answers for them too. What a fresh walk reads differently is the
2147 position COUNT of the pieces it already peeled, which the splice
2148 lowers, and two tests of the walk read it. The acronym fork's
2149 words to spare: a member whose count falls below three can
2150 decline where it was taken -- 'john prof. ma ma prof.' is exactly
2151 that, keeping family 'ma' only because the second pass walks
2152 afresh -- so a pass resumes only with two or more pieces in front
2153 of the splice, every peeled piece then counting three or more.
2154 And the numeral fork, which reads the walk's last piece against
2155 the one before it: a pass resumes only with two or more peeled
2156 pieces behind the splice, so that pair is the one the first walk
2157 read. Elsewhere -- a splice near the front of the walk, or before
2158 two pieces have been peeled -- the pass walks afresh over the
2159 spliced pieces. The fuzz that checked this against the
2160 re-peeling loop, the three conditions each shown load-bearing by
2161 a weakened copy failing it, is recorded on the PR closing #558.
2162 `count_numeral` is `peel_trailing`'s, passed to every pass, and
2163 `salutations` is `trailing_titles`', passed to every chain.
2164 The splice itself is never materialized in between: the runs are
2165 collected back to front and joined once.
2166 """
2167 # where the second name unit opens, found once for the read: a
2168 # pass keeps `rest[:hi]`, so it holds while it stands in front of
2169 # the splice, and each pass rescanning for it from the front cost
2170 # the square of a run behind a long joined surname (#620's
2171 # /simplify, 'Freiherr von Berg Berg ... MA Dr. MA Dr. ...')
2172 second = 1 if units is None else second_unit(rest, units)
2173 peeled = peel_trailing(rest, pieces, ptags, tokens, one_case,
2174 units=units, second=second,
2175 count_numeral=count_numeral)
2176 kept = trailing_titles(rest, pieces, ptags, tokens,
2177 end=peeled.names, salutations=salutations)
2178 if kept == peeled.names:
2179 p = run_start(rest, peeled.names, pieces, ptags, tokens, units,
2180 one_case, initials_cue)
2181 return rest, (), (peeled if p == peeled.names else credential_run(
2182 rest, peeled, p, pieces, ptags, tokens, one_case))
2183 # `rest[:hi]` stands as written; `behind` holds the peeled runs
2184 # the splices left after it, and `titled` the chained ones, each
2185 # back to front -- a pass's run goes in FRONT of what the pass
2186 # before it took
2187 hi = len(rest)
2188 behind: list[list[int]] = []
2189 behind_count = 0
2190 titled: list[list[int]] = []
2191 picks = list(peeled.picks)
2192 numerals = list(peeled.numerals)
2193 while kept < peeled.names:
2194 names = peeled.names
2195 titled.append(rest[kept:names])
2196 behind.append(rest[names:hi])
2197 behind_count += hi - names
2198 # the walk stopped AT the chain's last title, and a fresh walk
2199 # will not meet that piece: drop the pick it made there
2200 if picks and picks[-1] == tuple(pieces[rest[names - 1]]):
2201 picks.pop()
2202 hi = kept
2203 # Counted in units (#620) the position test still answers: a
2204 # splice lowers a count only by taking out titles that open
2205 # units of their own, and with fewer than two units in front of
2206 # it every piece there is in the first piece's chain run, which
2207 # the titles behind it join -- they count nothing, so nothing
2208 # the walk peeled loses a word to spare (fuzzed over 372,330
2209 # names against a units test here: no difference)
2210 if kept >= 2 and behind_count >= 2:
2211 peeled = peel_trailing(rest, pieces, ptags, tokens, one_case,
2212 start=hi, anchors=peeled.anchors,
2213 units=units, second=second,
2214 count_numeral=count_numeral)
2215 picks.extend(peeled.picks)
2216 numerals.extend(peeled.numerals)
2217 else:
2218 rest = rest[:hi] + [j for run in reversed(behind) for j in run]
2219 if units is not None and second >= hi:
2220 second = second_unit(rest, units)
2221 hi = len(rest)
2222 behind, behind_count = [], 0
2223 peeled = peel_trailing(rest, pieces, ptags, tokens, one_case,
2224 units=units, second=second,
2225 count_numeral=count_numeral)
2226 picks = list(peeled.picks)
2227 numerals = list(peeled.numerals)
2228 kept = trailing_titles(rest, pieces, ptags, tokens,
2229 end=peeled.names, salutations=salutations)
2230 if behind:
2231 rest = rest[:hi] + [j for run in reversed(behind) for j in run]
2232 final = Peel(peeled.names, tuple(numerals), tuple(picks), None)
2233 p = run_start(rest, final.names, pieces, ptags, tokens, units,
2234 one_case, initials_cue)
2235 return (rest, tuple(j for run in reversed(titled) for j in run),
2236 final if p == final.names else credential_run(
2237 rest, final, p, pieces, ptags, tokens, one_case))
2240class TailRead(NamedTuple):
2241 """S2's trailing run as group reads it ONCE, after P3's connective
2242 joins and before the particle chain and the bound-given join
2243 (#614), handed to assign so neither join can move it and assign
2244 reads no second time -- save where group's joins left no name word
2245 in front of the run, and assign reads the name again so it keeps
2246 one ('Prince of Wales Jr'). Token indices rather than piece
2247 indices, since group's joins renumber the pieces in front of the
2248 run: `tail` holds every token the run took, `titles` those the H5
2249 chain took and `run_titles` the title words inside #602's run (both
2250 inside `tail`), and `peel` the final peel, whose numeral fork, bare
2251 ambiguous picks and absorbed name words are what assign reports;
2252 its `names` counts the name pieces as the read left them."""
2254 tail: frozenset[int]
2255 titles: frozenset[int]
2256 run_titles: frozenset[int]
2257 peel: Peel
2260# rules/P2: "A particle joins the words after it into one name part,
2261# the join running until the next particle starts a group of its own"
2262# -- how far, asked by the chain and by S2's read alike (#620), so the
2263# units the read counts are the ones the chain then builds
2264def chain_run_end(k: int, pieces: Sequence[Sequence[int]],
2265 ptags: Sequence[Set[str]],
2266 tokens: Sequence[WorkToken], end: int,
2267 one_case: bool | None = None,
2268 given_slot: bool = False) -> int:
2269 """Where the chain run opened by the particle `pieces[k]` ends: past
2270 the particles straight behind it, then past the name words up to
2271 `end`, a suffix piece S8 would start its run at, a member its
2272 capitals settle as the credential in the given part after a family
2273 comma, or the next particle. `pieces[k:j]` is the one name unit P2
2274 makes of them; `j == k + 1` where it makes none. `one_case` is
2275 ParseState's.
2277 A suffix piece stops it only where `starts_the_run_here` says S8
2278 would start its run there, so a suffix word S8 starts no run at
2279 ('Abd') or declines for its writing with a name word behind it
2280 ('Md', 'Jr.') is the chain's (rules/P2, #1177), save the second in
2281 the given part after a family comma (`given_slot`), where S8's run
2282 starts behind the given word whatever stands behind it; and a
2283 particle that is suffix vocabulary too opens no chain over a suffix
2284 word right behind it.
2286 Only the given part after a family comma stops at a settled member:
2287 there the member starts S8's run behind the given word whatever
2288 stands in front, so the chain leaves it to the run. Elsewhere the
2289 trailing read's `end` stops the chain at a member the read takes,
2290 and a stop at one it declines would strand it as a name word between
2291 the particle run and the words behind it (#785's review).
2293 The particle test is `is_prefix_piece`'s definition written inline:
2294 the walk runs per piece of every chained name (AGENTS.md's frame
2295 budget, question 1), and so is the member tag ahead of the
2296 settled test."""
2297 count = len(pieces)
2298 j = k + 1
2299 while j < count and ("prefix" in ptags[j]
2300 or len(pieces[j]) == 1
2301 and "particle" in tokens[pieces[j][0]].tags):
2302 j += 1
2303 settled_case = one_case if given_slot else None
2304 while (j < end and "prefix" not in ptags[j]
2305 and not (len(pieces[j]) == 1
2306 and "particle" in tokens[pieces[j][0]].tags)
2307 # rules/P2: "a trailing suffix begins" -- at a suffix word
2308 # S8 would start its run at; one it declines, a name word
2309 # behind it, is the chain's (decisions/P2, #1177)
2310 and not (is_suffix_piece(pieces[j], ptags[j], tokens)
2311 # rules/P2: "a particle that is suffix vocabulary
2312 # too opens no chain over a suffix word right behind
2313 # it" -- P9's reading before a comma, kept wherever
2314 # the chain runs (decisions/P2, #1177)
2315 and (j == k + 1 and len(pieces[k]) == 1
2316 and "vocab:suffix" in tokens[pieces[k][0]].tags
2317 or starts_the_run_here(j, pieces, ptags, tokens,
2318 one_case, given_slot)))
2319 # rules/S6: "a clash settled by the writing is not reported"
2320 # -- and the chain does not take the settled credential in
2321 and not (len(pieces[j]) == 1
2322 and AMBIGUOUS_ACRONYM_TAG in tokens[pieces[j][0]].tags
2323 and settled_by_case(tokens[pieces[j][0]],
2324 settled_case))):
2325 j += 1
2326 return j
2329def starts_the_run_here(j: int, pieces: Sequence[Sequence[int]],
2330 ptags: Sequence[Set[str]],
2331 tokens: Sequence[WorkToken],
2332 one_case: bool | None,
2333 given_slot: bool) -> bool:
2334 """Whether the suffix piece `pieces[j]` would start #602's run
2335 where it stands, as `run_start` asks it, the count of name words in
2336 front aside: it starts one (`starts_a_credential_run`) and, outside
2337 the given part after a family comma, the word behind it, if any,
2338 does not decline it (`declined_by_the_name_behind`). A word written
2339 as a name ('Md', 'Abd', 'Jr.' before a name) is no stop for P2's
2340 chain there, and none for assign's family-part pass behind a
2341 particle, the two callers."""
2342 if not starts_a_credential_run(pieces[j], ptags[j], tokens,
2343 tokens[pieces[j - 1][0]].text,
2344 one_case):
2345 return False
2346 if given_slot or j + 1 == len(pieces) or len(pieces[j]) != 1:
2347 return True
2348 return not declined_by_the_name_behind(
2349 tokens[pieces[j][0]], pieces[j + 1], ptags[j + 1], tokens, one_case)
2352# rules/P2: "A particle joins the words after it into one name part"
2353# -- the particles straight behind it first, merged ahead of S2's read
2354# so the read counts and absorbs the run as the one piece the chain
2355# makes of it (#625)
2356def merge_particle_runs(pieces: list[list[int]], ptags: list[set[str]],
2357 tokens: Sequence[WorkToken],
2358 n: int) -> tuple[set[int], int | None]:
2359 """Merge each run of two or more adjacent particles past the
2360 chain's leading position into one piece, the one `chain_run_end`
2361 would open its run with ('van der', 'van mc'). The merged piece is
2362 a particle still (`prefix`), and no title: a P3 join can carry the
2363 tag in from a word in both vocabularies ('St und Do'), and the
2364 merged run would then read as a leading title.
2366 `n` is the end of the leading title run (`leading_titles`), read
2367 before anything merges. Returns the first token of each merged
2368 piece -- the chain's PARTICLE_OR_GIVEN report keys on a run that
2369 claimed something past its particle, which a merged run has though
2370 `chain_run_end` then claims nothing more -- and the leading
2371 position (`chain_lead`) where it asked it, else None.
2373 The particle test is `is_prefix_piece`'s definition written
2374 inline, and the leading position is asked only where two particles
2375 stand together past the first piece (AGENTS.md's frame budget,
2376 question 1)."""
2377 prefix: list[bool] = []
2378 for k in range(len(pieces)):
2379 prefix.append("prefix" in ptags[k]
2380 or len(pieces[k]) == 1
2381 and "particle" in tokens[pieces[k][0]].tags)
2382 merged: set[int] = set()
2383 k = 1
2384 while k + 1 < len(prefix) and not (prefix[k] and prefix[k + 1]):
2385 k += 1
2386 if k + 1 >= len(prefix):
2387 return merged, None
2388 lead = chain_lead(pieces, ptags, tokens, n)
2389 k = lead + 1
2390 while k < len(prefix):
2391 if prefix[k]:
2392 q = k + 1
2393 while q < len(prefix) and prefix[q]:
2394 q += 1
2395 if q > k + 1:
2396 merged.add(pieces[k][0])
2397 merge_pieces(pieces, ptags, k, q, add={"prefix"},
2398 drop={"title"})
2399 del prefix[k + 1:q]
2400 k += 1
2401 return merged, lead
2404def _chain_units(pieces: Sequence[Sequence[int]],
2405 ptags: Sequence[Set[str]],
2406 tokens: Sequence[WorkToken],
2407 n: int, lead: int | None = None,
2408 one_case: bool | None = None) -> list[int] | None:
2409 """The name units P2's chain will make of `pieces`, as a mark per
2410 piece -- OPENS where a piece opens a unit, and JOINED where the
2411 chain will join a name word to the run in front of it and the
2412 reading does not weigh the word (`_weighed`, which keeps a word of
2413 its own what a position always counted as one). The particles
2414 straight behind each other are one piece already
2415 (`merge_particle_runs`, #625), so a particle run is one unit and a
2416 dual inside it ('van mc', 'von vd') no word the read can take.
2417 `n` is the end of the leading title run, and the chain starts past
2418 its leading position (`chain_lead`, the one answer group's chain
2419 takes too, #624; `lead` where the caller has asked it already),
2420 which opens no unit inside the titles: the name starts at `n`.
2422 Nothing else is merged: the read counts with the flags and reads
2423 every other piece as written, so no word it weighs is hidden
2424 inside a unit
2425 (#620; the merged copy #614 read through hid a title inside a
2426 credential run and a shape-only numeral, and its indices drifted
2427 from the pieces'). None where no particle stands past the first
2428 piece, every piece then its own unit."""
2429 count = len(pieces)
2430 prefix = [("prefix" in ptags[k]
2431 or len(pieces[k]) == 1
2432 and "particle" in tokens[pieces[k][0]].tags)
2433 for k in range(count)]
2434 if not any(prefix[1:]):
2435 return None
2436 units = [OPENS] * count
2437 k = (chain_lead(pieces, ptags, tokens, n) if lead is None
2438 else lead) + 1
2439 while k < count:
2440 if prefix[k]:
2441 j = chain_run_end(k, pieces, ptags, tokens, count, one_case)
2442 q = k + 1
2443 # a split credential is weighed as the class it is a
2444 # member of (rules/S12); a suffix word with an unweighed
2445 # word behind it inside the chain's run is one the chain
2446 # took because S8 starts no run there or declines one, so
2447 # the peel cannot reach it and it is no word of its own
2448 # (rules/P2, #1177's
2449 # review: 'Freiherr von Md Berg V' as 'Freiherr von Abdul
2450 # Berg V')
2451 while (q < j and "split" not in ptags[q]
2452 and (not _weighed(pieces[q], tokens)
2453 or q + 1 < j
2454 and not _weighed(pieces[q + 1], tokens)
2455 and is_suffix_piece(pieces[q], ptags[q], tokens))):
2456 units[q] = JOINED
2457 q += 1
2458 k = j
2459 continue
2460 k += 1
2461 return units
2464def _weighed(piece: Sequence[int], tokens: Sequence[WorkToken]) -> bool:
2465 """Whether a word the chain will join is one the trailing reading
2466 WEIGHS, and so counts as a word of its own, as a position always
2467 counted it: a suffix word of either kind, the ambiguous class, a
2468 word in it by shape, an initial, a roman numeral by shape, or a
2469 period-marked title word. A word the chain joins and the reading
2470 weighs is a word the reading may yet take ('Freiherr von Berg MA
2471 X.Y.Z.' keeps suffix 'MA X.Y.Z.'), so it ends the plain run the
2472 count folds into the particle (#620's review); a title written
2473 short that may be a given name is one too, H5's chain taking it
2474 as it takes a listed title (#999), and so is a title listed whole
2475 with a period inside it, which the chain takes without a final one
2476 (#1104). A connective join is one name word, P3's own count. The
2477 tests are inline: this asks once per word behind a particle run."""
2478 if len(piece) > 1:
2479 return False
2480 tok = tokens[piece[0]]
2481 return (not tok.tags.isdisjoint(_WEIGHED_TAGS)
2482 or _ROMAN.match(stem := tok.text.rstrip(".")) is not None
2483 and stem != "")
2486#: the tags any one of which makes `_weighed` True for a one-word
2487#: piece: the trailing reading's lone-word tags, an initial, and a word
2488#: H5's chain takes (#999, #1104, #1171)
2489_WEIGHED_TAGS = _TRAILING_WORD_TAGS | {"initial", CHAIN_TITLE_TAG}
2492def read_trailing_run(pieces: Sequence[Sequence[int]],
2493 ptags: Sequence[Set[str]],
2494 tokens: Sequence[WorkToken],
2495 one_case: bool | None, n: int,
2496 lead: int | None = None,
2497 ) -> tuple[TailRead, int] | None:
2498 """`tail_reading` read once over a segment's pieces as group holds
2499 them after rules/P3's connective joins and before the particle
2500 chain (#614), counting name words by the units the chain will make
2501 (`_chain_units`, #620): a `TailRead`, and the index in `pieces` of
2502 the first piece of the run, which runs to the end. None where there
2503 is nothing to read (no name piece past the leading titles), and
2504 where the run is not one block at the end of `pieces`: a title the
2505 H5 chain takes standing in front of a name word the peel declined
2506 ('John Dr. G.J.'), the one shape whose run holds a name piece --
2507 there group joins as before #614 and assign reads for itself. A
2508 segment the read takes nothing from reads as a run of no pieces, at
2509 the end.
2511 `n` is the end of the leading title run (`leading_titles`) and
2512 `lead` the chain's leading position where the caller has asked it,
2513 both read before `merge_particle_runs` merged the pieces: the merge
2514 starts past the lead and drops the title tag, so it moves the lead
2515 never, and the title run only where H3 had given a title back
2516 before a run of particles that are suffix vocabulary, which the
2517 read takes as written: 'Dr. mc Jr mc mc' reads family 'mc', suffix
2518 'Jr mc mc', where the run read after the merge would leave 'mc Jr
2519 mc mc' in the family."""
2520 if n == len(pieces):
2521 return None
2522 units = _chain_units(pieces, ptags, tokens, n, lead, one_case)
2523 rest, titled, peel = tail_reading(peel_walk(n, ptags), pieces, ptags,
2524 tokens, one_case, units)
2525 tail: set[int] = set()
2526 for k in rest[peel.names:]:
2527 tail.update(pieces[k])
2528 titles: set[int] = set()
2529 for k in titled:
2530 titles.update(pieces[k])
2531 tail |= titles
2532 run_titles: set[int] = set()
2533 for k in peel.run_titles:
2534 run_titles.update(pieces[k])
2535 start = len(pieces)
2536 for k, piece in enumerate(pieces):
2537 if piece[0] in tail:
2538 start = k
2539 break
2540 for k in range(start, len(pieces)):
2541 if pieces[k][0] not in tail and "suffix" not in ptags[k]:
2542 return None
2543 return TailRead(frozenset(tail), frozenset(titles),
2544 frozenset(run_titles), peel), start
2547def trailing_run_start(start: int, pieces: Sequence[Sequence[int]],
2548 ptags: Sequence[Set[str]],
2549 tokens: Sequence[WorkToken],
2550 one_case: bool | None) -> int:
2551 """Where the run `tail_reading` takes over `pieces[start:]` begins,
2552 its trailing titles included: the first index of a piece the peel
2553 or the H5 chain took, `len(pieces)` where they take none. The
2554 bound for a reading that takes the trailing titles as well as the
2555 suffixes, as assign's walk does in the given part after a family
2556 comma (#623), where `trailing_start` sees only the suffixes and
2557 `trailing_start_past_titles` leaves the titles inside the bound.
2558 The chain takes a bare salutation too, as that walk does (rules/H8)."""
2559 # rules/S4: "the given part's trailing slot after a family comma is
2560 # read without it (S5)" -- the numeral half of the count, which
2561 # assign's walk there never asks either, so the chain this bounds
2562 # stops where that walk will read the suffix (#718's review); the
2563 # acronym half is still counted here, as before #718
2564 rest, titled, peel = tail_reading(peel_walk(start, ptags), pieces,
2565 ptags, tokens, one_case,
2566 count_numeral=False, salutations=True,
2567 initials_cue=False)
2568 first = len(pieces)
2569 for k in rest[peel.names:]:
2570 first = min(first, k)
2571 for k in titled:
2572 first = min(first, k)
2573 return first
2576# rules/H5: "the title is TRANSPARENT to the suffix reading: where
2577# two or more name words stand, what stands once the chain is taken
2578# reads exactly as it would read written without the title, plus the
2579# title"
2580def trailing_start_past_titles(start: int,
2581 pieces: Sequence[Sequence[int]],
2582 ptags: Sequence[Set[str]],
2583 tokens: Sequence[WorkToken],
2584 *, one_case: bool | None) -> int:
2585 """`trailing_start` read through H5's chain: where assign's
2586 trailing suffix run begins once a trailing TITLE has stopped
2587 hiding it.
2589 `trailing_start` reads the pieces as WRITTEN, so a title standing
2590 behind the suffix run makes the peel take nothing and the answer
2591 is `len(pieces)` -- the reading assign itself has not had since
2592 H5, because assign runs the peel and the chain to their fixed
2593 point instead (`tail_reading`). A caller using that answer as the
2594 right bound of the NAME is told a credential is a name word:
2595 'John Quincy Adams i MA Prof.' read family 'Adams i MA' where
2596 'John Quincy Adams i MA' reads family 'Adams' and suffix 'i MA'
2597 (#397 second review). Every caller that bounds the name wants
2598 this one; `trailing_start` stays for the callers that count a
2599 trailing run of the pieces as they stand.
2601 The returned index bounds the name from the right, and the
2602 trailing titles the chain spliced out are not under it: they end
2603 the segment, so they stand at or past the first suffix piece
2604 whenever there is one. Where the peel takes nothing even past the
2605 chain this returns `len(pieces)` as `trailing_start` does, and a
2606 trailing title is then inside the bound and refused by the title
2607 test the callers already run beside it.
2608 """
2609 rest, _titled, peeled = tail_reading(peel_walk(start, ptags),
2610 pieces, ptags, tokens, one_case)
2611 return rest[peeled.names] if peeled.names < len(rest) else len(pieces)