1,000 sampled decisions from 215 sources, drawn from a 1.6% sample of the materialized candidate clusters and re-solved with the production rule.
Each row is one decision the store makes. Left is the document that gets dropped,
right is the document kept in its place. The rule removes a document only when a
longer document in the same candidate cluster already holds at least 75% of its
word 3-grams, so the kept side is always the longer of the two.
The sample takes up to 5 decisions from each of the
215 sources that produced any, spread across the kinds of duplicate each source
makes, so it is a tour of the corpus rather than a ranking. How much a source actually
loses is a separate number, counted over the whole marker tree and shown beside every
source: on the card, in the dropdown, and in the table at the foot of the page.
The page opens on the source with the highest drop rate, with the picker ordered the same way,
so the first thing on screen is the corpus at its most affected. Switch the source to
every source, mixed for the even tour.
Drop rate for every source
Counted over the whole marker tree against the normalized corpus, so these are exact and
independent of the sample above. % of source is what the store drops from that source.
% of clustered is what it drops from the part of that source the rule can act on at all —
a source with a low first number and a high second one is one the clustering barely reached.
The corpus-wide rate is 12.74%, but it averages over a range spanning two orders of magnitude.
Hardest hit: massive_function_calling 94.5%, nemotron_cc_v2_1/high_quality_translated 79.9%, nemotron_cc_v2/medium_high_quality 71.1%, cp/arxiv_papers 68.1%, cp/public_domain_review 67.3%. At the other end, 66 sources lose nothing at all and
86 lose under 1%. Which end a source sits at moves the mixture more than the average does.
Fuzzy duplicate analysis of the datakit store built from
rav/hero_store_dup_075, marker tree
user/rav/dedup/verified/v11-c075-restored,
rule containment_ngram3_v2 at word 3-gram containment 0.75.