Shortcuts on this page
r m x toggle line displays
j k next/prev highlighted chunk
0 (zero) top of page
1 (one) first highlighted chunk
1"""Shared piece-level predicates for pipeline stages.
3How a PIECE reads -- its tokens, plus the tags classify wrote on them
4and the tags group derived for the piece -- where _vocab answers how a
5WORD reads from text alone. Both are consulted by more than one
6stage; the split is by what the question takes, not by
7which stage happens to ask (mechanisms.md#ONE-PREDICATE-PER-QUESTION).
8_vocab points here from its own side: "Text-level tests used by more
9than one stage; piece-level ones live in _pieces, the sibling layer
10over tokens-plus-tags."
12Before this module those predicates lived in _group, not because
13grouping owned them but because assign imported group and could not be
14imported back, so group was the only place both stages could reach.
15They arrived there that way across three PRs -- #424 brought
16is_leading_title, leading_titles and trailing_start, #425 the peel
17(peel_walk, peel_trailing), #429 the no-name-segment test that
18#430 turned into segment_suffix_reading.
19is_title_piece and is_suffix_piece are older than any of that: they
20were group's from its first commit, and travel because the others
21call them.
23The import that forced all of it is the one #439 removed: assign no
24longer names _group at all. What still holds is the rule that replaced
25it, and tests/v2/test_layering.py is where it is written down -- a
26piece predicate may not depend on a stage, in either direction.
28The S2 trailing peel travels as the unit decisions.md describes --
29peel_walk, peel_trailing and trailing_start together -- though only
30the first two cross a stage boundary. trailing_titles joins them
31because it reads what that peel left: the two answer one question
32between them, where the tail of a name stops being the name -- and
33tail_reading is that one question, running them against each other to
34their fixed point for the two stages that must not disagree about the
35answer.
37Layering: imports _state and _vocab only; _group and _assign import
38it, and neither of the two it imports imports it back.
40Naming follows _vocab's: inside an already-private module the leading
41underscore marks module-PRIVATE, so the names other stages call are
42bare and only the internals keep it (_PERIOD_ABBREV here). Getting that
43backwards -- which this module did until the underscores came off --
44costs a reader the one cheap way to tell a shared predicate from a
45helper.
46"""
47from __future__ import annotations
49import re
50from collections.abc import Sequence, Set
51from typing import NamedTuple
53from nameparser._pipeline._state import WorkToken
54from nameparser._pipeline._vocab import (
55 in_initialless_script, is_trailing_numeral_suffix,
56)
59# rules.md#H3: "successive title words at the start of the part
60# carrying the given name chain into one title; a title word
61# elsewhere in the name does not"
62def is_title_piece(piece: Sequence[int], ptags: Set[str],
63 tokens: Sequence[WorkToken]) -> bool:
64 if "title" in ptags:
65 return True
66 return len(piece) == 1 and "vocab:title" in tokens[piece[0]].tags
69# Ported verbatim from v1 (nameparser/config/regexes.py
70# "period_abbreviation") -- layering forbids the config import; keep
71# in sync by hand (tests/v2/test_regex_sync.py). Out of assign since
72# #424 and in the piece layer since #439: the test is assign's, and group's
73# leading-particle scan and trailing-run walk must start where assign
74# starts.
75_PERIOD_ABBREV = re.compile(r'^[^\W\d_]{2,}\.$')
78# rules.md#H2: "an abbreviation opening the part of the name that
79# carries the given name — the whole name, or the part after a
80# family comma — reads as a title even when unlisted"
81# (history: decisions.md#H2)
82def is_leading_title(piece: Sequence[int], ptags: Set[str],
83 tokens: Sequence[WorkToken]) -> bool:
84 if is_title_piece(piece, ptags, tokens):
85 return True
86 if len(piece) != 1:
87 return False
88 text = tokens[piece[0]].text
89 # The shape reads a Latin convention: a period marks an
90 # abbreviation. Scripts with no initials have no period
91 # abbreviations either (_policy._NO_INITIALS, the #320 veto
92 # is_initial carries), so a CJK word wearing a period is a name
93 # word, not a title -- a lone '田中.' is the family name (#323).
94 # _PERIOD_ABBREV stays ASCII-period only: a word wearing '。' never
95 # matched it, and the veto is what makes the ASCII spelling agree.
96 # ASCII text can carry no _NO_INITIALS character (every range sits
97 # above U+3000), so the C-level test declines before the regex
98 # search runs -- four frames per unlisted-abbreviation opener per
99 # parse, is_leading_title running four times per piece.
100 return (bool(_PERIOD_ABBREV.match(text))
101 and (text.isascii() or not in_initialless_script(text)))
104def leading_titles(pieces: Sequence[Sequence[int]],
105 ptags: Sequence[Set[str]],
106 tokens: Sequence[WorkToken]) -> int:
107 """How many leading pieces assign peels as titles: the first
108 non-title index. A title needs a following piece, unless the whole
109 segment is one title (v1 parity). And the run gives back its last
110 piece when that piece is a name candidate: where everything behind
111 the run is suffix pieces, the run gives back its last piece, when
112 that piece is one word and is not itself suffix vocabulary
113 (rules.md#H3, decisions.md#H3 -- the block at the floor below
114 carries the examples of each half, and the ordering its two inline
115 tag reads were measured on).
116 One definition, read by assign (which sets the roles) and by the
117 chain's trailing-run walk; the leading-particle scan shares the
118 predicate, is_leading_title, but stops at a title-and-particle
119 word (P4, #367, #424)."""
120 n = 0
121 while n < len(pieces):
122 if ((n + 1 < len(pieces) or len(pieces) == 1)
123 and is_leading_title(pieces[n], ptags[n], tokens)):
124 n += 1
125 continue
126 break
127 # rules.md#H3: "where everything behind the run is post-nominal,
128 # the run gives its last word back to the name, provided that word
129 # stands alone and is not itself suffix vocabulary"
130 #
131 # ONE WORD, because a joined unit led by a title is a title run and
132 # handing it back would lose the title: 'Prince of Wales Jr' reads
133 # title 'Prince of Wales', family 'Jr', not given 'Prince of Wales'
134 # with no title at all.
135 #
136 # Two residuals. A run whose last word IS suffix vocabulary is not
137 # given back, so 'Dr King MD PhD' still reads title 'Dr King MD',
138 # family 'PhD'. And the floor asks is_suffix_piece, which vetoes a
139 # bare initial-shaped numeral, so 'Dr King V' keeps the whole run
140 # as the title and reads the numeral as the name -- given 'V',
141 # 'king' being a given-name title and the run's last word, where
142 # 'Dr Smith V' reads suffix 'V'. That numeral fork is outside this
143 # floor (decisions.md#H3).
144 #
145 # The two inline tag reads are the cheapest NECESSARY condition for
146 # the piece behind the run to be a suffix piece at all --
147 # is_suffix_piece cannot answer yes without one of them -- so the
148 # ordinary titled name, whose next piece is no kind of suffix,
149 # leaves this branch without entering a frame. Measured on the
150 # plan's ordering rather than the shape below: asking the
151 # authoritative predicate first cost 8 calls per parse of the
152 # reference name (leading_titles runs four times), against a band
153 # with room for two (decisions.md#parse-cost). is_suffix_piece
154 # stays the predicate that ANSWERS, here and in the walk.
155 if (n and n < len(pieces)
156 and ("suffix" in ptags[n]
157 or "vocab:suffix" in tokens[pieces[n][0]].tags)
158 and len(pieces[n - 1]) == 1
159 and not is_suffix_piece(pieces[n - 1], ptags[n - 1],
160 tokens)):
161 for k in range(n, len(pieces)):
162 if not is_suffix_piece(pieces[k], ptags[k], tokens):
163 break
164 else:
165 # nothing behind the run but suffix pieces
166 n -= 1
167 return n
170def is_suffix_piece(piece: Sequence[int], ptags: Set[str],
171 tokens: Sequence[WorkToken]) -> bool:
172 if "suffix" in ptags:
173 return True
174 if len(piece) != 1:
175 return False
176 tags = tokens[piece[0]].tags
177 return "vocab:suffix" in tags and "initial" not in tags
180def _numeral_behind_the_initial_veto(piece: Sequence[int],
181 tokens: Sequence[WorkToken]) -> bool:
182 """Suffix vocabulary that is_suffix_piece refuses because it is
183 also initial-shaped: a ONE-CHARACTER entry, bare or with a period.
185 Named for the shape rather than enumerated, because the shape is
186 what the code tests and the enumeration goes stale -- in the
187 shipped lexicon it reaches i, v and 2, and NOT x or ix (roman, but
188 not suffix vocabulary) nor ii/iii/iv (suffix vocabulary, but two
189 characters, so never initial-shaped and never vetoed in the first
190 place). A caller adding a one-character suffix in a script that
191 has initials extends it.
193 The veto is right where such a word could be a middle initial, and
194 wrong where it is describing the suffix in front of it, which is
195 the only place this is asked from. The len(piece) != 1 guard is
196 defensive: a merged multi-token piece carries "suffix" in ptags, so
197 is_suffix_piece claims it one branch earlier and no reachable input
198 arrives here with one.
199 """
200 if len(piece) != 1:
201 return False
202 tags = tokens[piece[0]].tags
203 return "vocab:suffix" in tags and "initial" in tags
206def segment_suffix_reading(pieces: Sequence[Sequence[int]],
207 ptags: Sequence[Set[str]],
208 tokens: Sequence[WorkToken],
209 lenient: bool,
210 ) -> tuple[bool, ...] | None:
211 """How each piece of a no-name segment reads: True a suffix, False
212 a title. None when the segment holds a name word and so is not a
213 credential run at all.
215 ONE answer for two readers, both in _assign.py -- the no-name gate
216 and the router -- because they must agree piece for piece. #429
217 shipped the inverse of its own fix by deriving that agreement twice
218 (mechanisms.md#ONE-PREDICATE-PER-QUESTION). It answered for a third
219 until #436: group's one-entry join asked it too, and the render's
220 entry boundary is a rule over the written commas in post_rules now
221 (rules.md#R1), which asks this nothing.
223 rules.md#S2's initial veto keeps a roman numeral out of a suffix
224 reading, which is right after a NAME word: 'Smith, John V.' is a
225 middle initial (#432). After a SUFFIX word the numeral is
226 describing that suffix -- 'PSM I' is Professional Scrum Master
227 level I -- so the run continues through it, period included, an
228 initial there being no shape anyone writes (#430). A title resets
229 that: what follows a bare title is not continuing a credential.
231 None covers both ways a segment can fail to be a run: a name word
232 anywhere in it, and no pieces at all ('Doe,, Jr.', which holds no
233 title to read by).
235 `lenient` is Policy.lenient_comma_suffixes, and only the numeral
236 continuation consults it. C1: "by default a recognized suffix word
237 counts even written like an initial, while strict mode vetoes
238 initial-shaped words" -- so under strict the veto stands and the
239 run ends where it always did. Reading no policy here silently
240 overrode the one knob a caller sets to prevent exactly this.
242 The FAMILY_COMMA rule "segment 0 is wholly the family name" rests
243 on the writer having said where the family name ends. A comma
244 followed by no name word said no such thing -- 'John Smith, Dr.' is
245 'Dr. John Smith' with the honorific moved -- so the pre-comma name
246 keeps its positional read instead of being merged. Uses the same
247 is_leading_title predicate the peel does, period-abbreviation
248 inference included, so the two cannot disagree about what a title
249 is; a mixed run like 'Smith, Dr. Jr.' is a title and a postnominal,
250 each read where it stands, never a title run 'Dr. Jr.'.
251 """
252 if not pieces:
253 return None
254 out: list[bool] = []
255 for piece, tags in zip(pieces, ptags):
256 # the verdict just recorded IS "stands behind a suffix" -- keeping
257 # a separate flag meant maintaining that equality by hand at three
258 # sites, and a fourth branch that appended without assigning would
259 # have diverged silently
260 after_suffix = bool(out) and out[-1]
261 if is_suffix_piece(piece, tags, tokens):
262 out.append(True)
263 elif (lenient and after_suffix
264 and _numeral_behind_the_initial_veto(piece, tokens)):
265 out.append(True)
266 elif is_leading_title(piece, tags, tokens):
267 out.append(False)
268 else:
269 return None
270 return tuple(out)
273class Peel(NamedTuple):
274 """What assign's trailing peel made of a walk. `names` is a count
275 of positions in the caller's `rest`: rest[:names] are the name
276 pieces and rest[names:] the suffixes. The other two are pieces --
277 token-index tuples, as PendingAmbiguity wants them -- and each is
278 one token long: `numeral` is the piece the roman-numeral fork
279 took (None when it did not fire; always the walk's last piece),
280 `picks` the bare ambiguous acronyms the peel had to resolve, in
281 peel order, either way (the last may sit at rest[names - 1])."""
283 names: int
284 numeral: tuple[int, ...] | None
285 picks: tuple[tuple[int, ...], ...]
288# rules.md#S2: "a trailing word of the suffix vocabulary reads as a
289# suffix — generational forms and credential acronyms alike, and an
290# ambiguous acronym written with its periods, one after each letter,
291# counts unambiguously; a single trailing period is the abbreviation
292# shape any word can wear and does not. A BARE ambiguous acronym is
293# consumed only when the name has words to spare"
294# (v1's are_suffixes tail rule, with the roman-numeral special)
295def peel_walk(start: int, ptags: Sequence[Set[str]],
296 skip: Set[int] = frozenset()) -> list[int]:
297 """The indices peel_trailing walks: `start` to the segment's end,
298 minus the group-flagged credential pieces (the Ph. D. merge),
299 which assign reads as suffixes at any position, and minus `skip`
300 -- a tail segment's delimiter cores, which are structure rather
301 than words (the maiden walk's case, #424). Built here and nowhere
302 else, so the walk's input cannot drift between assign and the
303 group sites that read it: the numeral fork is a last-piece
304 test that reads the piece before as rest[k - 2], which holds only
305 over this list."""
306 return [j for j in range(start, len(ptags))
307 if j not in skip and "suffix" not in ptags[j]]
310def trailing_start(start: int, pieces: Sequence[Sequence[int]],
311 ptags: Sequence[Set[str]], tokens: Sequence[WorkToken],
312 skip: Set[int] = frozenset(),
313 numeral_only: bool = False) -> int:
314 """Where assign's trailing suffix run begins, read over the pieces
315 as they stand from `start`: the index of the first piece the S2
316 peel takes, or len(pieces) when it takes none (#424). What P2's
317 chain and M2's walk stop before -- each had asked "is this a
318 suffix?" with the suffix-piece test, which vetoes a bare 'V' as
319 an initial (the #401 question), and so took a trailing numeral,
320 or a bare acronym with words to spare, into the family or the
321 maiden name.
323 `numeral_only` is the maiden walk's reading: the bare-acronym
324 fork counts pieces, and the walk removes the very pieces it
325 counted, so an acronym peeled over the pieces as they stand may
326 be the family of what is left ('John née Jones Smith Ma' read
327 maiden 'Jones Smith', family 'Ma'). The numeral fork reads one
328 piece, the one before the numeral, and _maiden_take re-asks it
329 with the piece the take leaves there; the acronym is left to
330 assign."""
331 rest = peel_walk(start, ptags, skip)
332 peeled = peel_trailing(rest, pieces, ptags, tokens)
333 if numeral_only:
334 return rest[-1] if peeled.numeral is not None else len(pieces)
335 return rest[peeled.names] if peeled.names < len(rest) else len(pieces)
338def peel_trailing(rest: Sequence[int], pieces: Sequence[Sequence[int]],
339 ptags: Sequence[Set[str]],
340 tokens: Sequence[WorkToken]) -> Peel:
341 """The S2 trailing peel over `rest`, a peel_walk list. In the
342 piece layer rather than in assign because group's bound-given
343 reserve asks the same question of the view the join would leave
344 (#425): one walk, so the reserve and the assignment cannot drift. Pure -- the ambiguities are
345 returned for assign to report, in the order it always reported
346 them."""
347 picks: list[tuple[int, ...]] = []
348 numeral: tuple[int, ...] | None = None
349 k = len(rest)
350 while k > 0:
351 piece = pieces[rest[k - 1]]
352 if is_suffix_piece(piece, ptags[rest[k - 1]], tokens):
353 k -= 1
354 continue
355 # a final single letter that is a roman numeral, after a piece
356 # that is not initial-shaped; the predicate's docstring carries
357 # the is_initial_shaped reasoning (#320)
358 if (k == len(rest) and k >= 2 and len(piece) == 1
359 and is_trailing_numeral_suffix(
360 tokens[piece[0]].text,
361 tokens[pieces[rest[k - 2]][0]].text)):
362 numeral = tuple(piece)
363 k -= 1
364 continue
365 # A bare ambiguous acronym ("MA", not "M.A.") is a credential
366 # only when peeling it still leaves a given AND a family name.
367 # With two pieces, "one of them is a credential" is the less
368 # likely reading, so it stays the family name -- "Jack MA" is a
369 # person, "John Smith MA" is a person with a degree. This is
370 # v1's reserve_last narrowed to the ambiguous set: 2.0
371 # deliberately peels UNambiguous suffixes even when nothing is
372 # left ("Smith PhD" -> suffix, a classified fix), because there
373 # the vocabulary is not in doubt.
374 bare_ambiguous = (len(piece) == 1
375 and "vocab:suffix-ambiguous" in tokens[piece[0]].tags)
376 # k < 2 means it is the only piece left, which is not the fork
377 # this reports.
378 if bare_ambiguous and k >= 2:
379 picks.append(tuple(piece))
380 if k >= 3: # peeling still leaves given + family
381 k -= 1
382 continue
383 break
384 return Peel(k, numeral, tuple(picks))
387# rules.md#H5: "only a word the vocabulary knows as a title is one,
388# and a bare title word is a name word"
389# -- the trailing run's own predicate. NOT is_leading_title:
390# that predicate carries H2's unlisted-abbreviation inference, which is
391# the LEADING slot's shape rule and has no trailing counterpart, so
392# with it 'John Smith Xyz.' would lose its family name to a title
393# (decisions.md#H5). The vocabulary read is is_title_piece's, shared
394# with the leading run so the two cannot disagree about what a title
395# WORD is while disagreeing, deliberately, about what a title SHAPE is.
396def trailing_titles(rest: Sequence[int], pieces: Sequence[Sequence[int]],
397 ptags: Sequence[Set[str]],
398 tokens: Sequence[WorkToken]) -> int:
399 """How many pieces of `rest` the trailing title chain LEAVES
400 standing: `rest[:kept]` are the name pieces and `rest[kept:]` the
401 period-marked title words the chain took, in piece order. Counted
402 the way `peel_trailing` counts, so the two answers compose without
403 arithmetic at the call site. `rest` is the caller's NAME pieces:
404 on the no-comma path what the S2 peel left, after a family comma
405 the segment's pieces that the segment's own suffix reading does
406 not claim, and in `tail_reading` the leftovers of whichever peel
407 is current.
408 Floor: one name piece stands, so a name is never all title -- and
409 an empty `rest` returns 0, which is what leaves assign's
410 bare-suffix carve-out reached exactly as before.
412 ONE WORD per piece, the same gate the leading peel's give-back
413 uses: a joined unit is not the shape this reads, and the tokens of
414 one are not each a title word.
416 Every parse with a name word to place enters this frame -- assign
417 asks the question here rather than answering a cheaper version of
418 it inline (mechanisms.md#ONE-PREDICATE-PER-QUESTION) -- so what it
419 costs an ordinary name is one frame and one regex match. The
420 exceptions return before it: a segment that is all title, and a
421 comma part read wholly as a credential run, have no name piece to
422 hand this (52 of the 1289 corpus parses, measured 2026-09-09 --
423 'Coach', 'Lord of the Universe', 'Smith, Jr.', 'MD, PHD').
424 The shape test
425 runs BEFORE the vocabulary one to keep it at that: _PERIOD_ABBREV
426 is a compiled regex (a C call, no Python frame) where
427 is_title_piece is a call, and almost no name ends in a
428 period-marked word, so the ordinary parse pays the one match and
429 stops (decisions.md#parse-cost).
430 """
431 k = len(rest)
432 while k > 1:
433 idx = rest[k - 1]
434 piece = pieces[idx]
435 # no #323 veto on the shape here, unlike is_leading_title's:
436 # the shape is ANDed with is_title_piece, so the word is listed
437 # vocabulary, and a listed CJK title wearing a stop should read
438 # as a title.
439 if (len(piece) == 1
440 and _PERIOD_ABBREV.match(tokens[piece[0]].text)
441 and is_title_piece(piece, ptags[idx], tokens)):
442 k -= 1
443 continue
444 break
445 return k
448# rules.md#H5: "the title is TRANSPARENT to the suffix reading: where
449# two or more name words stand, what stands once the chain is taken
450# reads exactly as it would read written without the title, plus the
451# title"
452def tail_reading(rest: list[int], pieces: Sequence[Sequence[int]],
453 ptags: Sequence[Set[str]],
454 tokens: Sequence[WorkToken],
455 ) -> tuple[list[int], tuple[int, ...], Peel]:
456 """The S2 peel and the H5 chain read together to a FIXED POINT:
457 peel, chain, splice the chained pieces out, peel again over what
458 is left -- the name pieces the chain kept, then the pieces the
459 peel had taken, in original order -- until the chain takes
460 nothing. Where it takes nothing on the first pass, which is almost
461 every name, that first peel is the answer and the loop costs one
462 comparison.
464 Returns the walk with the chained pieces spliced out, the pieces
465 the chain took, and the FINAL peel -- whose numeral fork and
466 ambiguous picks are the ones assign reports. The walk is
467 partitioned by that peel exactly as a caller partitions its own:
468 `rest[:peel.names]` the name pieces, `rest[peel.names:]` the
469 suffixes. A bare tuple rather than a named one because a
470 NamedTuple's __new__ is a frame of its own on every parse
471 (decisions.md#parse-cost), and `_group_segment` returns its three
472 the same way.
474 Transparency is what the fixed point buys: 'X Prof. Y' reads
475 exactly as 'X Y' reads plus the title, however many titles are
476 written and wherever the peel then stops. Iterating ONCE reads a
477 second title only half way -- 'John Prof. MA Prof.' un-peeled the
478 acronym and re-exposed the first title, reading family 'Prof.'
479 with suffix 'MA' where 'John Prof. MA' reads family 'MA'.
481 One function for two readers -- assign's placement and group's
482 bound-given reserve (P5), which must count the name words assign
483 will leave. Deriving that agreement twice is what left the two
484 disagreeing at S2's bare-ambiguous reserve: 'abdul rahman MA'
485 declined the join and 'abdul rahman MA Prof.' took it
486 (mechanisms.md#ONE-PREDICATE-PER-QUESTION).
488 `rest` is a peel_walk list and is not mutated -- the splice
489 rebinds this local -- so a caller's own reference still names the
490 walk it built. Both callers read the one returned here instead,
491 which is the one the final peel partitions.
492 """
493 titled: list[int] = []
494 while True:
495 peeled = peel_trailing(rest, pieces, ptags, tokens)
496 kept = trailing_titles(rest[:peeled.names], pieces, ptags,
497 tokens)
498 if kept == peeled.names:
499 return rest, tuple(titled), peeled
500 # the chain's pieces reach this list back to front, so each
501 # run goes in FRONT of what the pass before it took
502 titled[:0] = rest[kept:peeled.names]
503 rest = rest[:kept] + rest[peeled.names:]