Shortcuts on this page
r m x toggle line displays
j k next/prev highlighted chunk
0 (zero) top of page
1 (one) first highlighted chunk
1"""Stage: classify.
3Consumes: tokens, segments (the part a title phrase or a split
4credential's pair is read within), extracted (whether a delimited
5clause holds a word with a comma on it, rules/S1), comma_offsets (with token roles, the two halves of
6the structural-boundary test the marker pass applies -- see
7_vocab.tag_marker_runs), and one_case where an earlier stage recorded
8it -- segment writes it lazily where C2's flag on a part past the
9second comma asks for it (#289/#516, #613), so this read is the
10fallback for every other name.
11Produces: tokens with vocabulary tags added (text/span/role unchanged),
12plus ambiguities (CONJUNCTION_OR_INITIAL) and
13one_case -- whether the name's own words are written in one case,
14recorded for the later stages that read it (#289/#516) and left alone
15where an earlier stage already asked.
16Reads: every Lexicon vocabulary field except surnames and
17honorific_tails, which script_segment consumes upstream, and
18given_names, which only post_rules reads (rules/O6); and the kinds
19of its suffix words and particles (Lexicon's generation_words,
20credential_words, honorific_words and patronymic_particles, internal
21sets, 3.0) and the
22suffix words whose Title case leans to the name
23(title_case_name_words, rules/S6, #1039) and the
24titles written short (title_abbreviations, #999) and the names a
25community gives every member (shared_names) and the given names
26written behind a particle (particle_given_names, #704) and the
27two-word given names a bound word completes into (bound_phrases,
28#694) and the titles a run of single letters may spell
29(dotted_titles, rules/H10, #490); and, since
302.4, Policy.unlisted_dotted_suffixes and Policy.unlisted_caps_suffixes,
31which decide whether an UNLISTED dotted or all-caps token joins the
32ambiguous credential class by SHAPE (#516) -- the all-caps half here
33only under CapsSuffixes.EVERYWHERE, the default's part after a comma
34being tagged by `_comma.decide` when group binds it (#613); and
35Policy.extra_suffix_delimiters, with the built-in semicolon
36(_vocab.glued_cores), so a word after the first comma with a
37delimiter glued to it is tagged as the word without it (rules/C10).
38is_initial also consults
39the _policy module's _NO_INITIALS constant. It is named apart from the
40fields above because it is not CONFIGURATION -- no Lexicon or Policy
41carries it and no caller can change it -- and not because it decides
42nothing: the tags this stage writes do vary by it ('씨.' is not tagged
43`initial`, which is the whole of #320).
45Tags emitted -- stable (API): "particle", "conjunction", "initial";
46namespaced (unstable): "vocab:title", "vocab:given-title",
47"vocab:after-given-title", "vocab:title-ambiguous",
48"vocab:suffix", "vocab:suffix-word", "vocab:suffix-ambiguous",
49"vocab:suffix-word-ambiguous", "vocab:agnomen",
50"vocab:generation-word", "vocab:credential-word",
51"vocab:honorific-word", "vocab:patronymic", "vocab:trailing-title",
52"vocab:joined-title" and "vocab:chain-title" (read by the later stages
53and dropped by assemble, Token.reading carrying the kind out, save the
54last three, which mark no kind), "vocab:title-case-name" (a suffix word whose
55Title case leans to the name, rules/S6, kept on the published token),
56"vocab:particle-ambiguous", "vocab:bound-given",
57"vocab:given-abbreviation", "vocab:shared-name" (which rules/P10, O6
58and O7 read), "vocab:particle-given" (which P10 and O6 read) and
59"vocab:bound-phrase" (on a bound word and the word after it being a
60listed pair, which P12 joins and O6 counts once, #694), each
61dropped by assemble and carrying no kind out, "vocab:maiden-marker",
62"vocab:maiden-marker-cont"; in a namespace of its own,
63"shape:acronym"; and in another, "split:credential" and
64"split:member", written on the first word of a credential split across
65a space (`_vocab.split_pair`) to name the piece the pair merges into:
66the Ph./D. pair's unambiguous suffix where the lexicon lists `phd` as
67an unambiguous credential, or any other pair's member of the ambiguous
68class (rules/S12).
69"vocab:maiden-marker" tags the HEAD of a maiden marker, which is a
70whole marker whenever the marker is one word; the continuation tag
71carries the rest of a PHRASE marker ("z domu"), so a site asking
72"does a marker start here" reads the same tag it always did and a site
73asking "where does it end" walks the continuations.
74"vocab:suffix" means "counts as a suffix as written": unambiguous
75suffix vocabulary, or an ambiguous acronym written with periods --
76at the TAG level 'M.A.' gets "vocab:suffix" while 'Ma' gets only
77"vocab:suffix-ambiguous"; what assign then does with a trailing
78ambiguous tag is rules S4 to S7 (the words-to-spare guard, the
79case, the company) and their Accepted consequences.
80The initial veto is assign's job, not classify's: 'V' carries both
81"vocab:suffix" and "initial".
82"shape:acronym" is the one tag in the shape: namespace and it records
83WHERE a class claim came from rather than what the vocabulary holds:
84an unlisted token the writing makes credential-shaped. It rides
85beside "vocab:suffix-ambiguous" where a Policy switch admits the
86token to that class, and stands alone where the switch is off, which
87is what lets the fork be reported without being taken.
88"""
89from __future__ import annotations
91import unicodedata
93from nameparser._lexicon import _normalize, _spells_an_initial
94from nameparser._policy import CapsSuffixes
95from nameparser._pipeline._state import (
96 AGNOMEN_TAG, AMBIGUOUS_ACRONYM_TAG, AMBIGUOUS_WORD_TAG, COMMA_STRIP, CREDENTIAL_WORD_TAG,
97 GENERATION_WORD_TAG, GIVEN_ABBREVIATION_TAG, HONORIFIC_WORD_TAG,
98 BOUND_PHRASE_TAG, PARTICLE_GIVEN_TAG, PATRONYMIC_TAG, SHAPE_ACRONYM_TAG, SHARED_NAME_TAG, SPLIT_CREDENTIAL_TAG, SPLIT_MEMBER_TAG, TITLE_AMBIGUOUS_TAG,
99 TITLE_CASE_NAME_TAG,
100 CHAIN_TITLE_TAG, DOTTED_TITLE_TAG, JOINED_TITLE_TAG, TRAILING_TITLE_TAG,
101 ParseState,
102 Deferred, PendingAmbiguity, PendingReading,
103 WorkToken, copy_with,
104)
105from nameparser._types import AmbiguityKind, ReadingKind, ReadingSource
106from nameparser._pipeline._vocab import (
107 BUILTIN_GLUED_DELIMITERS, SPLIT_CHUNK, SPLIT_PAIR_OPENS, _INITIAL, _JOINED_WORD,
108 _TRAILING_TITLE_SHAPE, caps_shape_candidate,
109 composed_len, glued_cores, glued_stem, is_cased_paired_initials,
110 is_dotted_title, is_initial, is_one_case,
111 period_joined_title, period_joined_vocab,
112 split_pair,
113 suffix_as_written, tag_marker_runs, title_readings,
114)
115from nameparser._pipeline._pieces import own_words
120# rules/S2: "a trailing word of the suffix vocabulary reads as a
121# suffix — generational forms and credential acronyms alike" -- and
122# rules/S3: "an ambiguous acronym written with its periods, one after
123# each letter or one after each of two or more letter chunks, counts
124# unambiguously; a single trailing period is the abbreviation shape
125# any word can wear and does not" -- and rules/S4: "a
126# bare ambiguous acronym is consumed only when the name has words to
127# spare"
128def _tags_for(token: WorkToken, n: str, state: ParseState,
129 marker_tag: str | None, one_case_own: bool,
130 one_case: bool, letter_one_case: bool) -> frozenset[str]:
131 """`n` is _normalize(token.text), folded once by the caller and
132 shared with the marker pass; `marker_tag` is what that pass decided
133 for this token, or None. The marker DECISION is entirely
134 `_vocab.tag_marker_runs`'; only the writing happens here, so the
135 two tokens of a phrase are built once rather than replaced twice.
137 `one_case_own` is true when the name's OWN words are written in one
138 case AND this token is one of the name's own words -- a maiden clause
139 and any delimited (nickname) content are not, so the fork never
140 reads them either (rules/P3): a clause's words are not the
141 name's own words, and appending one must not change how THIS token
142 reads.
144 `one_case` is the bare NAME-level fact alone -- P3's own-words
145 span is not this question's business. #516's caps branch reads
146 THIS, not `one_case_own`: a maiden clause's own words are outside
147 `one_case_own`'s span by construction (`i < clause_at` fails for
148 every one of them), so a token past the clause cut reads
149 `one_case_own` as False regardless of whether the WHOLE name is
150 written in one case -- 'JOHN SMITH NEE' flipped 'NEE' to a
151 credential reading with the switch on, one case and all, because
152 `not one_case_own` was true for it purely from being past the
153 clause cut, never from the name's own writing (#516 review round,
154 a second reviewer's finding). `single_letter_connective` below
155 keeps `one_case_own`: that fork is genuinely about the OWN-WORDS
156 span, and reads a clause's word as no evidence on purpose.
158 `letter_one_case` is rules/T4's question for THIS token: the
159 own-words fact for one of the name's own words, and for a word
160 outside them -- a clause's, or past a declined marker -- the fact
161 about the WHOLE name, since the own words' case says nothing
162 about how the clause beside them was written: 'y' in 'MARIA
163 GARCIA NEE LOPEZ y FERNANDEZ' is a lower-case letter among
164 capitals, which T4 reads as no initial (#723's review)."""
165 lex = state.lexicon
166 tags = set(token.tags)
167 if marker_tag is not None:
168 tags.add(marker_tag)
169 if n in lex._sets.titles:
170 tags.add("vocab:title")
171 # rules/H5: "a title the vocabulary lists whole with a period
172 # inside it chains without a final one" -- marked here, where
173 # the listing is known whole; a period, since a hyphen alone
174 # joins whole words as often as abbreviations ('co-founder'), and
175 # the one hyphenated abbreviation ending a name in the corpora,
176 # 'д-р', is a credential there to CLDR (decisions/H5, #1104)
177 if "." in n and _JOINED_WORD.match(n):
178 tags.add(JOINED_TITLE_TAG)
179 # rules/H10: "a run of single letters joined by full stops whose
180 # spelling without them is a title the vocabulary marks as written
181 # that way" -- the tag; whether a title the run keeps follows it is
182 # for the title runs to ask (`_pieces.leading_titles`, and H6's in
183 # assign). The stop and set tests first, C-level, so a word with no
184 # interior stop, or one spelling no marked title, pays no frame.
185 elif ("." in n and n.replace(".", "") in lex._sets.dotted_titles
186 and is_dotted_title(token.text, n, lex)):
187 tags.add(DOTTED_TITLE_TAG)
188 if n in lex._sets.given_name_titles:
189 tags.add("vocab:given-title")
190 # rules/H5: "a title that may be a given name is title vocabulary
191 # there where its title reading is written short" -- marked here,
192 # where the sets are at hand, so the chain reads a tag
193 if (n in lex._sets.title_abbreviations
194 and n in lex._sets.titles_ambiguous):
195 tags.add(TRAILING_TITLE_TAG)
196 # rules/H8: "the word joins the title field" -- tagged only where the
197 # word is a title too, so an entry outside `titles` is inert
198 if n in lex._sets.after_given_titles and n in lex._sets.titles:
199 tags.add("vocab:after-given-title")
200 if suffix_as_written(n, token.text, lex):
201 tags.add("vocab:suffix")
202 if n in lex._sets.suffix_words_ambiguous:
203 # rules/S11: "it reads as the suffix only where the name has words
204 # to spare" -- the ambiguous class's tag, so every site
205 # that weighs a member weighs this one, and the cue tags that
206 # tell `_pieces.listed_lean` how its writing leans
207 tags.add(AMBIGUOUS_ACRONYM_TAG)
208 tags.add(AMBIGUOUS_WORD_TAG)
209 if n in lex._sets.title_case_name_words:
210 tags.add(TITLE_CASE_NAME_TAG)
211 if n in lex._sets.agnomens:
212 tags.add(AGNOMEN_TAG)
213 # rules/R8: "The kind of a word listed as a suffix word is the
214 # lexicon's" -- a gated credential (`kc`, #965) reads as a
215 # credential, not the generation the class otherwise holds
216 elif n in lex._sets.credential_words:
217 tags.add(CREDENTIAL_WORD_TAG)
218 elif n in lex._sets.suffix_words:
219 tags.add("vocab:suffix-word")
220 # rules/R8: "The kind of a word listed as a suffix word is the
221 # lexicon's" -- read here, from the word as this stage reads it
222 # (decisions/R8, #326; #754, 4i-b)
223 if n in lex._sets.generation_words:
224 tags.add(GENERATION_WORD_TAG)
225 elif n in lex._sets.credential_words:
226 tags.add(CREDENTIAL_WORD_TAG)
227 elif n in lex._sets.honorific_words:
228 tags.add(HONORIFIC_WORD_TAG)
229 if n in lex._sets.suffix_acronyms_ambiguous:
230 tags.add(AMBIGUOUS_ACRONYM_TAG)
231 # rules/P7: "A one-letter word written with its period is an
232 # initial and not a particle" -- both particle tags, so 'Ó.' is
233 # neither half of the vocabulary
234 if n in lex._sets.particles and not _spells_an_initial(n, token.text):
235 tags.add("particle")
236 if n in lex._sets.particles_ambiguous:
237 tags.add("vocab:particle-ambiguous")
238 # rules/R9: "a particle the lexicon holds as a patronymic" -- its
239 # kind, read here as a suffix word's is (decisions/R9, #754)
240 if n in lex._sets.patronymic_particles:
241 tags.add(PATRONYMIC_TAG)
242 # rules/P8: "a single-letter connective reads as an initial
243 # where the writing says so: written as a bare Latin capital in a
244 # name that is not written wholly in one case, or — in a name
245 # written wholly in one case, where nothing says so — where the
246 # letter is one the vocabulary marks as reading both ways"
247 # (#383/#479; history: decisions/P8)
248 # one letter as the writer sees it: a decomposed 'й' is two code
249 # points (#665), so non-ASCII text is counted composed, after the
250 # C-level tests have declined everything else
251 single_letter_connective = (n in lex._sets.conjunctions
252 and token.text.upper() != token.text.lower()
253 and (len(token.text) == 1
254 or not token.text.isascii()
255 and composed_len(token.text) == 1))
256 if single_letter_connective and one_case_own:
257 # No case evidence, so the vocabulary decides. No namespaced
258 # tag beside it: the emitted ambiguity IS the record of the
259 # decision (mechanisms.md#MARK-DONT-STRIP is satisfied by the
260 # report) -- mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE says
261 # emit where the branch is taken, not where an ambiguous tag
262 # sits, and a vocab: tag records MEMBERSHIP, not the branch
263 # taken, so it is the wrong shape of record here.
264 if n in lex._sets.conjunctions_ambiguous:
265 tags.add("initial")
266 else:
267 # a bare capital y joins here, where mixed case vetoes it
268 tags.add("conjunction")
269 else:
270 # the mixed-case rule, unchanged. v1's is_conjunction excludes
271 # initials: 'e.' in 'john e. smith' is a middle initial, not
272 # the Spanish conjunction 'e'
273 # rules/T4: "in a name written wholly in one case, which says
274 # nothing about any letter in it, a bare letter is an initial
275 # wherever its capital would be" -- so every reader of the tag
276 # after classify reads 'j' as it reads 'J' (#723; the roman
277 # numerals first, #721). Readers that run before the tag
278 # exists, or ask the word's text rather than its tag -- the
279 # strict suffix test, which the honorific peel and
280 # `lenient_comma_suffixes=False` ask -- take the same fact as
281 # an argument (#727). `letter_one_case` is the own-words fact
282 # for an own word and the whole name's for a clause's
283 # (docstring). The capital is asked of the
284 # bare pattern at C level, no frame per one-letter word, and
285 # needs no script test: no capital `_INITIAL` matches is in a
286 # script without initials. 'é' and 'ж' stay as they were,
287 # their capitals being no initial either. A clause's letter
288 # asks the whole name's case (`letter_one_case`, docstring)
289 initial = is_initial(token.text) or (
290 letter_one_case and len(token.text) == 1
291 and _INITIAL.fullmatch(token.text.upper()) is not None)
292 if n in lex._sets.conjunctions and not initial:
293 tags.add("conjunction")
294 if initial:
295 tags.add("initial")
296 if n in lex._sets.bound_given_names:
297 tags.add("vocab:bound-given")
298 # rules/H2: "an abbreviation the vocabulary reads as a given name
299 # written short is that given name" -- the reading H2's shape test
300 # weighs (`_pieces.is_leading_title`); a membership, so any spelling
301 # carries it and only that test acts on it
302 if n in lex._sets.given_name_abbreviations:
303 tags.add(GIVEN_ABBREVIATION_TAG)
304 # rules/O6, O7 and P10 read this: a name a community gives every
305 # member is neither taken nor completed by the two-name switches,
306 # nor completed by P10's particle phrase (#704)
307 if n in lex._sets.shared_names:
308 tags.add(SHARED_NAME_TAG)
309 # rules/P10 and O6 read this: a given name written behind a particle
310 # completes the given name it follows (#704)
311 if n in lex._sets.particle_given_names:
312 tags.add(PARTICLE_GIVEN_TAG)
313 # maiden markers are NOT tagged here: an entry may be a phrase whose
314 # words are not markers on their own, and this function sees one
315 # token with no neighbours. `_vocab.tag_marker_runs` does the whole
316 # field, single words included, so there is one place that decides
317 # it (mechanisms.md#ONE-PREDICATE-PER-QUESTION).
318 # The block a WHOLE-TOKEN vocabulary match skips, and four
319 # readings inside it, in precedence order. The first two are v1's
320 # period-joined derivation (parse_pieces): a token with a period
321 # not at the end, ANY of whose period chunks is a title, is a
322 # title as a whole ('Lt.Gov.', and by the ANY rule 'Mr.Smith');
323 # else a suffix where EVERY chunk counts ('JD.CPA', #1131). Title wins
324 # (v1's continue). The third and fourth are 2.4's by-shape halves
325 # (#516; rules/S9: "joins the ambiguous class (S4) by SHAPE, in
326 # the positions each shape is admitted") and reach only a
327 # token NO vocabulary claimed: a dotted token of two or more
328 # alphabetic chunks, and -- where its switch is on -- an unlisted
329 # all-caps word. They are an `elif` chain
330 # with the dotted one first, so a dotted token never reaches the
331 # caps test and the two shapes stay disjoint.
332 #
333 # Both by-shape branches carry SHAPE_ACRONYM_TAG BESIDE the
334 # membership tag rather than instead of it: `vocab:` records
335 # membership (the roster above) and the peel reads membership,
336 # while the shape tag records that the claim came from the
337 # WRITING. The dotted branch writes it even with its switch off,
338 # which is what lets a declined fork be reported without being
339 # taken. `role is None` guards both: a token already carrying a
340 # role is delimited content, decided by extract's escape and never
341 # at the trailing slot -- 'Bridge (A.B)' is the control that
342 # proves the guard load-bearing (without it the nickname reading
343 # of 'A.B' gains a spurious SUFFIX_OR_NICKNAME report in post_rules), and
344 # 'Bridge (1.4)' cannot exercise it, a digit chunk never reaching
345 # the shape verdict at all (period_joined_vocab's alphabetic
346 # gate).
347 if "vocab:title" not in tags and "vocab:suffix" not in tags:
348 derived = period_joined_vocab(token.text, lex)
349 if derived == "title":
350 tags.add("vocab:title")
351 # rules/H5: "a word with interior periods every one of whose
352 # chunks is a title chains without a final period" -- marked
353 # as a title listed whole is (decisions/H5, #1131)
354 if period_joined_title(token.text, lex):
355 tags.add(JOINED_TITLE_TAG)
356 elif derived == "suffix":
357 tags.add("vocab:suffix")
358 elif (derived == "shape" and token.role is None
359 and n not in lex._sets.suffix_acronyms_ambiguous):
360 # This branch and `_vocab.ambiguous_class_candidate` ask
361 # the SAME question twice, of necessity -- `segment` runs
362 # before `classify` and has no tags to read yet -- kept
363 # from drifting by `test_classify.
364 # test_ambiguous_class_candidate_agrees_with_the_tag`
365 # rather than by this sentence alone.
366 #
367 # `n not in suffix_acronyms_ambiguous` is the third guard,
368 # and it is about a LISTED member rather than a role: a
369 # caller may list a dotted entry ('a.b'), which the whole-
370 # token membership test above matches while
371 # `suffix_as_written`'s period-free acronym lookup ('ab')
372 # misses, so the chunk view reached here and called the
373 # word by-shape -- silencing `_pieces.listed_lean`, which
374 # declines wherever SHAPE_ACRONYM_TAG rides, and costing
375 # the caller's own listing its case lean ('Jack A.B.' read
376 # family where 'Jack MA' reads suffix). One frozenset
377 # lookup on a branch only a dotted token reaches; the
378 # shipped ambiguous vocabulary carries no periods, so
379 # nothing default changes.
380 tags.add(SHAPE_ACRONYM_TAG)
381 # rules/S3: "Paired initials are the exception, as C5 defines
382 # them, in a script that writes capitals" -- the shape
383 # tag alone, as with the switch off, so the trailing slots
384 # read them as a name word and report the fork; the comma
385 # restores the class tag where C5 reads them
386 # (decisions/S3 (#864)); `is_cased_paired_initials` asks the
387 # script half ('พ.บ.' is no initial)
388 if (state.policy.unlisted_dotted_suffixes
389 and not is_cased_paired_initials(token.text)):
390 tags.add(AMBIGUOUS_ACRONYM_TAG)
391 elif (state.policy.unlisted_caps_suffixes is CapsSuffixes.EVERYWHERE
392 and token.role is None
393 # what the predicate would decline anyway, asked first in
394 # C: a listed member ('John Smith, MA'), and a word that
395 # is not alphabetic capitals ('John Smith, Ph. D.')
396 and n not in lex._sets.suffix_acronyms_ambiguous
397 and (token.text.isupper()
398 and (token.text.isalpha() or "-" in token.text
399 or "/" in token.text)
400 or not unicodedata.is_normalized("NFC", token.text))
401 and caps_shape_candidate(token.text, lex, state.policy,
402 one_case)):
403 # #516's all-caps half: an unlisted word written in capitals
404 # inside a mixed-case name. EVERYWHERE tags it in every
405 # slot; the default admits it only in the part after a
406 # comma, which `_comma.decide` reads after this stage and
407 # tags there when it binds the part (#564, #613), so the
408 # reading carries the same marks under either setting --
409 # case repair keeps 'XYZ' in 'John Smith, XYZ'
410 # (rules/R4) rather than title-casing a credential the
411 # default admitted.
412 # The setting comes FIRST, then C-level checks of what the
413 # predicate would decline, so the default never calls it
414 # and EVERYWHERE calls it only for an unlisted all-caps
415 # word (that is why this half is a call where the dotted branch
416 # above stays inline: the dotted caller has no such cheap
417 # first conjunct to hide behind). The predicate's own
418 # docstring carries the whole-vocabulary roster and what
419 # each measured entry would have cost unfixed.
420 tags.add(SHAPE_ACRONYM_TAG)
421 tags.add(AMBIGUOUS_ACRONYM_TAG)
422 return frozenset(tags)
425#: The tags of a word that joins the word after it, which a delimiter
426#: glued to the word stands between (rules/C10). A particle's are among
427#: them though it also attaches backward (rules/P6): kept, it chained
428#: forward across the delimiter ('Berg, de; Smith' read family 'Berg de;
429#: Smith'), and blocking the forward joins alone would guard every site
430#: that reads the tag (decisions/C10).
431_JOINING_TAGS = frozenset({"particle", "conjunction",
432 "vocab:particle-ambiguous", "vocab:bound-given"})
434#: What a word read out of the title gives up (rules/H9): the title tag
435#: and the two that qualify one, which are read only beside it.
436_TITLE_TAGS = frozenset({"vocab:title", "vocab:given-title",
437 "vocab:after-given-title"})
440def classify(state: ParseState) -> ParseState:
441 # One fold per token, shared by the marker pass and the vocabulary
442 # tags -- the shape suffix_as_written already asks for ("n is
443 # _normalize(text), passed in so callers normalize once").
444 folded = [_normalize(t.text) for t in state.tokens]
445 marker_tags = tag_marker_runs(state.tokens, state.comma_offsets,
446 state.lexicon._sets.maiden_markers, folded)
447 # rules/P3 says a maiden marker, taken as one, and the words it
448 # takes, are not among the name's own words -- so the span and its
449 # clause cut are _pieces.own_words', shared with the site that
450 # needs the same answer two stages earlier (#289/#516). The marker
451 # map goes with it: this stage has already decided which tokens
452 # are run HEADS, so the helper reads that decision rather than
453 # walking the texts again, and classify's answer is the one it was
454 # before the helper existed.
455 own, clause_at = own_words(state.tokens, state.comma_offsets,
456 state.lexicon._sets.maiden_markers, marker_tags)
457 # ONE fact per parse, and it is recorded now (ParseState.one_case):
458 # segment writes it first where C2's flag on a tail part asks, and
459 # a fact two stages decide apart is what recording it prevents
460 # (decisions/S6).
461 one_case = state.one_case
462 if one_case is None:
463 one_case = is_one_case(own)
464 # The fork itself must not read a clause's words either, so the
465 # `one_case and ...` argument below repeats `own`'s membership test
466 # per token, and the fork and its emitter then agree with the case
467 # class they consult. No extra frame -- it is one more boolean in a
468 # comprehension that already walks every token.
469 # rules/T4 asks a word outside the own words about the WHOLE
470 # name's case (`_tags_for`'s `letter_one_case`). Where every token
471 # is an own word the two facts are one, so only a name with a
472 # clause pays the second test's frame.
473 whole_one_case = one_case and (
474 len(own) == len(state.tokens)
475 or is_one_case([t.text for t in state.tokens]))
476 # rules/C10: "the word keeps the delimiter in its text and reads as
477 # it would without it" -- tagged as the word without it ('MD;' as
478 # 'MD'), so every reading that asks a word's vocabulary asks the
479 # stem's; only where the delimiter "stands after the first comma",
480 # a semicolon before it being a word (decisions/C10, D15)
481 readers = list(state.tokens)
482 stemmed: set[int] = set()
483 if state.comma_offsets:
484 policy_cores = state.policy.extra_suffix_delimiters
485 cores = (BUILTIN_GLUED_DELIMITERS if not policy_cores
486 else glued_cores(policy_cores))
487 first = state.comma_offsets[0]
488 for i, t in enumerate(state.tokens):
489 text = t.text
490 if t.span.start < first:
491 continue
492 for core in cores:
493 if core in text:
494 stem = glued_stem(text, cores)
495 if stem is not None:
496 readers[i] = copy_with(t, text=stem)
497 folded[i] = _normalize(stem)
498 stemmed.add(i)
499 break
500 # rules/S1: "the comma is no part of the word it is written against"
501 # -- a word of a delimited clause keeps a comma written beside it
502 # (N1), and is tagged as the word without it, so a clause S1 sets apart
503 # gives each word its vocabulary whichever side of the comma it
504 # stands ('(Jr, PhD)', '(MBA, PhD)'); a maiden marker's comma
505 # unmakes the marker (M3), which the marker pass above has already
506 # read off the word as written (decisions/S1, #792)
507 if state.extracted:
508 for i, t in enumerate(state.tokens):
509 if (t.role is not None and i not in marker_tags
510 and (bare := t.text.strip(COMMA_STRIP)) and bare != t.text):
511 readers[i] = copy_with(t, text=bare)
512 folded[i] = _normalize(bare)
513 tokens = tuple(
514 copy_with(
515 t, tags=_tags_for(readers[i], folded[i], state, marker_tags.get(i),
516 one_case_own=one_case and i < clause_at
517 and t.role is None, one_case=one_case,
518 letter_one_case=whole_one_case or (
519 one_case and i < clause_at
520 and t.role is None)))
521 for i, t in enumerate(state.tokens))
522 # A word with a delimiter glued to it joins no neighbour across the
523 # delimiter, so the stem's joining tags stay off it: 'y;' is no
524 # connective to the word after it (rules/C10: "no join, maiden
525 # clause or connective reaches across it")
526 if stemmed:
527 tokens = tuple(
528 copy_with(t, tags=t.tags - _JOINING_TAGS) if i in stemmed else t
529 for i, t in enumerate(tokens))
530 # rules/P12: "the two words are one name unit past the first name word"
531 # -- the phrase test over each word and the word after it, asked
532 # only of a name holding a bound word, in a plain loop, so that no
533 # other name pays a frame; the tag goes on the bound word, and
534 # group joins the pair (decisions/P12, #694)
535 phrases = state.lexicon._sets.bound_phrases
536 if phrases and not state.lexicon._sets.bound_given_names.isdisjoint(
537 folded):
538 heads = set()
539 for i in range(len(folded) - 1):
540 # rules/C10: "no join, maiden clause or connective reaches
541 # across it", so a delimiter glued to the bound word ends it
542 if (f"{folded[i]} {folded[i + 1]}" in phrases
543 and i not in stemmed):
544 heads.add(i)
545 if heads:
546 tokens = tuple(
547 copy_with(t, tags=t.tags | {BOUND_PHRASE_TAG})
548 if i in heads else t for i, t in enumerate(tokens))
549 # rules/H9: "the word is a title only inside a title phrase, and
550 # elsewhere a name word" -- the phrase test (_vocab.title_readings)
551 # asked per part, of the words carrying no role, and only of a part
552 # holding one, so that a name with neither pays no frame. A word read out of the title keeps no
553 # title tag, and every later stage reads it as the name word it is;
554 # a contested word keeps TITLE_AMBIGUOUS_TAG either way, for the
555 # report where it opens the name (assign)
556 lex = state.lexicon
557 # rules/H10: "a title that is also a given name reads it as the
558 # title it spells when asking for a partner in its title phrase
559 # (H9)" -- the run's spelling without stops goes to the phrase test
560 # wherever it stands, as its undotted twin does. Only where the name
561 # holds a word that test reads, so a run alone ('U.S. Grant') runs
562 # no phrase test; whether the run is kept as a title is the title
563 # runs' question, not this one (decisions/H10)
564 if not (lex._sets.titles_ambiguous.isdisjoint(folded)
565 and lex._sets.title_modifiers.isdisjoint(folded)):
566 for i, t in enumerate(tokens):
567 if DOTTED_TITLE_TAG in t.tags:
568 folded[i] = folded[i].replace(".", "")
569 if not (lex._sets.titles_ambiguous.isdisjoint(folded)
570 and lex._sets.title_modifiers.isdisjoint(folded)):
571 written = list(tokens)
572 for at, seg in enumerate(state.segments):
573 own_part = [i for i in seg if written[i].role is None]
574 norms = [folded[i] for i in own_part]
575 if (lex._sets.titles_ambiguous.isdisjoint(norms)
576 and lex._sets.title_modifiers.isdisjoint(norms)):
577 continue
578 readings = title_readings(
579 [readers[i].text for i in own_part], norms, lex,
580 head=at == 0,
581 titled=["vocab:title" in written[i].tags for i in own_part])
582 for i, title in zip(own_part, readings):
583 tags = written[i].tags
584 n = folded[i]
585 if n not in lex._sets.titles_ambiguous and n not in lex._sets.title_modifiers:
586 continue
587 if n in lex._sets.titles_ambiguous and n not in lex._sets.titles:
588 tags = tags | {TITLE_AMBIGUOUS_TAG}
589 if title:
590 # in a title phrase: a title (its given-name title
591 # tag, if any, `_tags_for` wrote already)
592 tags = tags | {"vocab:title"}
593 if not title:
594 tags = tags - _TITLE_TAGS
595 if tags != written[i].tags:
596 written[i] = copy_with(written[i], tags=tags)
597 tokens = tuple(written)
598 # rules/S12: "a credential written split across a space reads as one
599 # word" -- each adjacent pair of a segment asked once, here, and
600 # the answer written on its first word for group's merge and the
601 # comma's reading of its part (`_vocab.split_pair`). The shape and
602 # the fold's membership are asked first, in C, so an ordinary pair
603 # of words pays no frame; a token carrying a role is delimited
604 # content, which no pair reaches.
605 split: dict[int, str] | None = None
606 acronyms = state.lexicon._sets.suffix_acronyms
607 words = state.lexicon._sets.suffix_words
608 for seg in state.segments:
609 for j in range(len(seg) - 1):
610 a, b = tokens[seg[j]], tokens[seg[j + 1]]
611 if (SPLIT_CHUNK.fullmatch(a.text) is not None
612 and SPLIT_CHUNK.fullmatch(b.text) is not None
613 and a.role is None and b.role is None
614 # the opening letter, the length a superset of the
615 # predicate's count of composed letters ('i i' folds
616 # to the listed 'ii' and opens with one)
617 and (len(a.text.rstrip(".")) >= 2
618 or a.text.endswith(".") and b.text.endswith(".")
619 and (len(b.text) == 2 or not b.text.isascii()))
620 # the Ph./D. pair, which `split_pair` answers by the
621 # lexicon's `phd` (#1000); any other only where its words
622 # are not two suffix words, two credentials already,
623 # and its fold is listed, composed in C where it is
624 # not ASCII
625 and (SPLIT_PAIR_OPENS.fullmatch(a.text) is not None
626 or not ("vocab:suffix" in a.tags
627 and "vocab:suffix" in b.tags)
628 and ((fold := (a.text + b.text).replace(".", "")
629 .lower()) in acronyms or fold in words
630 or not fold.isascii()
631 and ((nfc := unicodedata.normalize(
632 "NFC", fold)) in acronyms
633 or nfc in words)))
634 and (kind := split_pair(a.text, b.text,
635 state.lexicon)) is not None
636 # a member pair a connective borders is no credential:
637 # the connective joins the words beside it
638 and (kind == "suffix" or (
639 j == 0
640 or "conjunction" not in tokens[seg[j - 1]].tags)
641 and (j + 2 >= len(seg)
642 or "conjunction" not in tokens[seg[j + 2]].tags))
643 # the Ph./D. pair outranks a pair overlapping it:
644 # 'Dr. Ph. D.' is no 'Dr. Ph.' ('drph') and a 'D.'
645 and not (kind == "split" and j + 2 < len(seg)
646 and split_pair(b.text, tokens[seg[j + 2]].text,
647 state.lexicon) == "suffix")):
648 if split is None:
649 split = {}
650 split[seg[j]] = (SPLIT_CREDENTIAL_TAG if kind == "suffix"
651 else SPLIT_MEMBER_TAG)
652 if split is not None:
653 # a loop, not a generator, which is a frame per token resumed
654 written = list(tokens)
655 for i, tag in split.items():
656 written[i] = copy_with(written[i], tags=written[i].tags | {tag})
657 tokens = tuple(written)
658 # rules/H5: "successive single words that wear the abbreviation
659 # shape and are title vocabulary chain into the title from the end"
660 # -- which words those are, written once as CHAIN_TITLE_TAG for
661 # every reader of the chain (#1171). After H9's phrase test, which
662 # settles "vocab:title"; against the text as written, which the
663 # readers saw before the tag ('Prof.;' wears no shape, C10's stem
664 # notwithstanding). The tag tests first, in C, so only a title word
665 # pays the regex, and only one wearing the shape pays a frame, in
666 # any position ('Dr.' too: position is the readers' question).
667 chain: list[WorkToken] | None = None
668 for i, token in enumerate(tokens):
669 tags = token.tags
670 if (JOINED_TITLE_TAG in tags
671 or ("vocab:title" in tags or TRAILING_TITLE_TAG in tags)
672 and _TRAILING_TITLE_SHAPE.match(token.text) is not None):
673 if chain is None:
674 chain = list(tokens)
675 chain[i] = copy_with(token, tags=tags | {CHAIN_TITLE_TAG})
676 if chain is not None:
677 tokens = tuple(chain)
678 ambiguities = list(state.ambiguities)
679 for i, token in enumerate(tokens):
680 # #383/#479: narrows the fork's own "initial" tag with the same
681 # inputs the fork used, rather than re-deciding from scratch.
682 # Only the casedness test is inherited from the tag --
683 # `is_initial(token.text)` also tags a bare capital "initial"
684 # in the fork's else branch, so the tag alone does not tell
685 # this apart from that.
686 #
687 # Of the two clauses beside it, one is load-bearing and one is
688 # not. `conjunctions_ambiguous` is IMPLIED by the others for a
689 # fresh parse -- the contrapositive of what it looks like:
690 # `is_initial` matches an ASCII capital only, so a bare
691 # LOWERCASE letter is tagged "initial" only by the fork's
692 # branch ("jose e maria santos" tags a lowercase e), for a
693 # conjunctions_ambiguous member, or by the one-case letter
694 # (#721, #723), which never tags an own word that is a
695 # single-letter connective (the fork's branch takes those
696 # first). So for a letter of EITHER case in `conjunctions`,
697 # "initial" plus len 1 implies membership. It is kept only because `_tags_for`
698 # starts from `set(token.tags)`, so
699 # this clause is what keeps the emitter honest if a runner ever
700 # hands classify tokens it did not build; no such path exists
701 # today. `conjunctions` is the clause doing real work: it keeps
702 # an orphan marker (a conjunctions_ambiguous entry no longer in
703 # conjunctions) inert rather than reported (decisions/P8).
704 # `i < clause_at and token.role is None` mirrors the fork's own
705 # "own words" test above, for the same reason (rules/P3): a
706 # clause's words were never eligible for the fork, so they must
707 # never be eligible to report either. Emitted at the decision
708 # site (mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE), per
709 # token -- 'e and e e' emits twice; assemble withdraws the
710 # report on a letter no name word follows (rules/P8, #721).
711 if (one_case and "initial" in token.tags
712 and folded[i] in state.lexicon._sets.conjunctions_ambiguous
713 and folded[i] in state.lexicon._sets.conjunctions
714 and i < clause_at and token.role is None
715 and (len(token.text) == 1
716 or not token.text.isascii()
717 and composed_len(token.text) == 1)):
718 ambiguities.append(PendingAmbiguity(
719 AmbiguityKind.CONJUNCTION_OR_INITIAL,
720 f"{token.text!r} is both a connective and an initial; "
721 f"the name is written in one case, so nothing marks "
722 f"which, and it is read as an initial",
723 (i,),
724 # the field of the join it would make is assign's to
725 # name, once the name has positions
726 alternative=PendingReading(ReadingKind.CONJUNCTION,
727 ReadingSource.VOCABULARY,
728 Deferred.JOINED)))
729 # The write rides the replace this stage already makes, so
730 # recording the fact costs no frame of its own.
731 return copy_with(state, tokens=tokens,
732 ambiguities=tuple(ambiguities),
733 one_case=one_case)