PLoS ONEplosplosonePLOS ONE1932-6203Public Library of ScienceSan Francisco, CA USAPONE-D-23-0861010.1371/journal.pone.0291337Research ArticleSocial sciencesLinguisticsSemanticsBiology and life sciencesPsychologyBehaviorAntisocial behaviorSocial sciencesPsychologyBehaviorAntisocial behaviorSocial sciencesLinguisticsComputational linguisticsSocial sciencesSociologyCommunicationsSocial communicationSocial mediaComputer and information sciencesNetwork analysisSocial networksSocial mediaSocial sciencesSociologySocial networksSocial mediaSocial sciencesSociologyCommunicationsSocial communicationSocial mediaTwitterComputer and information sciencesNetwork analysisSocial networksSocial mediaTwitterSocial sciencesSociologySocial networksSocial mediaTwitterPeople and placesPopulation groupingsEthnicitiesEuropean peopleItalian peopleComputer and information sciencesNetwork analysisSocial networksSocial sciencesSociologySocial networksPhysical sciencesMathematicsProbability theoryStatistical distributionsBenford’s Law applies to word frequency rank in English, German, French, Spanish, and ItalianBenford’s Law applies to word frequency rank in English, German, French, Spanish, and Italianhttps://orcid.org/0000-0003-3684-307XGolbeckJenniferConceptualizationData curationFormal analysisInvestigationMethodologyProject administrationResourcesSoftwareValidationWriting – original draftWriting – review & editing*University of Maryland, College Park, MD, United States of AmericaDe BenedictisLucaEditorUniversity of Macerata, ITALY
The authors have declared that no competing interests exist.
* E-mail: jgolbeck@umd.edu20231492023189e0291337223202329820232023Jennifer GolbeckThis is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Benford’s Law states that, in many real-world data sets, the frequency of numbers’ first digits is predicted by the formula log(1 + (1/d)). Numbers beginning with a 1 occur roughly 30% of the time, and are six times more common than numbers beginning with a 9. We show that Benford’s Law applies to the the frequency rank of words in English, German, French, Spanish, and Italian. We calculated the frequency rank of words in the Google Ngram Viewer corpora. Then, using the first significant digit of the frequency rank, we found the FSD distribution adhered to the expected Benford’s Law distribution. Over a series of additional corpora from sources ranging from news to books to social media and across the languages studied, we consistently found adherence to Benford’s Law. Furthermore, at the user-level on social media, we found Benford’s Law holds for the vast majority of users’ collected posts and significant deviations from Benford’s Law tends to be a mark of spam bots.
The author(s) received no specific funding for this work.Data AvailabilityWe used 23 public corpra in this study Google - English http://storage.googleapis.com/books/ngrams/books/datasetsv2.html Version 20120701 Google - German http://storage.googleapis.com/books/ngrams/books/datasetsv2.html Version 20120701 Google - Spanish http://storage.googleapis.com/books/ngrams/books/datasetsv2.html Version 20120701 Big English https://www.kaggle.com/datasets/rtatman/english-word-frequency?resource=download Google - French http://storage.googleapis.com/books/ngrams/books/datasetsv2.html Version 20120701 Google -Italian http://storage.googleapis.com/books/ngrams/books/datasetsv2.html Version 20120701 French Reddit https://www.kaggle.com/datasets/breandan/french-reddit-discussion Cresci http://mib.projects.iit.cnr.it Twitter abortion https://www.kaggle.com/datasets/mcantoni81/italian-tweets-discussion-on-the-roevswade LELU https://www.kaggle.com/datasets/breandan/french-reddit-discussion Spanish Wikipedia https://www.kaggle.com/datasets/rtatman/120-million-word-spanish-corpus COW https://www.webcorpora.org/opendata/frequencies/ Aranea corpra http://unesco.uniba.sk/aranea_about/ DeReKo https://www.corpusfinder.ugent.be/corpus/69 BNC http://www.kilgarriff.co.uk/bnc-readme.html parler https://zenodo.org/record/4442460.Introduction
Benford’s Law states that, in many real-world data sets, the frequency of numbers’ first digits is not evenly distributed. The exact frequency P predicted for a first significant digit (FSD) d is given by this formula:
P(d)=log10(1+(1/d))
Benford’s Law is most widely used in forensic accounting, where a violation may indicate fraud [1]. Research has also shown that it applies to Covid reporting [2, 3], genome data [4], scientific regression coefficients [5], the stock market in some cases [6] but not others [7], and can be used to detect AI-generated images [8]. Generally, if Benford’s Law typically applies to a system, deviations from it can be indicators of “unnatural” activity.
In statistical linguistics, both word frequency (the number of times a word appears in a corpus) and word frequency rank (which word is most popular, 2nd most popular, and so on) are common measures. For example, these are the ten most popular words with their frequency rank and overall occurrence in an English corpus:
the (53,097,401,461)
of (30,966,074,232)
and (22,632,024,504)
to (19,347,398,077)
in (16,891,065,263)
a (15,310,087,895)
is (8,384,246,685)
that (8,000,768,228)
for (6,545,282,031)
it (5,740,085,369)
For this research, we measured frequency and frequency rank using the Google 1-gram datasets for English, German, French, Spanish, and Italian. Described further below, these corpora contain the count of how often every work appears in a large collection of around 8 million books published between 1500 and 2008, (roughly 6% of all books ever published).
Then, we looked at the distribution of frequency rank FSDs within a variety of additional corpora. We found across corpora and languages, these distributions closely followed Benford’s expected distribution and were statistically indistinguishable. Furthermore, the FSD distribution of the frequency rank of words posted by individual users also closely follow Benford’s Law. In two corpora, we show that when a user’s posts deviate from Benford’s Law, it is associated with spam bot behavior. This echoes previous work that shows violations of Benford’s Law in social network connection behavior is also an indicator of bot activity [9].
We discuss how these results lead to the possibility of using Benford’s Law to detect bots and other unusual or unnatural behavior.
Background and related workBenford’s Law
Benford’s Law was first detected by astronomer Simon Newcomb and later validated by physicist Frank Benford. Benford studied naturally occurring numbers from many sources: the surface area of rivers, atomic weights, and numbers appearing in Reader’s Digest, and found all followed the same FSD distribution [10].
The formula for the law, P(d) = log10(1 + (1/d)), provides a theoretical distribution of expected first digits, shown in Table 1.
10.1371/journal.pone.0291337.t001
Frequency of first significant digits (FSD) expected by Benford’s Law.
FSD
1
2
3
4
5
6
7
8
9
Frequency
0.301
0.176
0.125
0.097
0.07918
0.067
0.05799
0.051
0.046
Benford’s Law is frequently applied in financial and accounting applications, specifically to detect fraud [11]. However, Benford’s law is not always present even when researchers may expect it to be based on the form of the data. For example, [7] examined S&P500 daily closing values and found non-conformity with Benford. Thus, it is important to apply and test with Benford using a careful eye.
Benford’s Law often applies to systems that follow a power law distribution [12]. Power laws are commonly found in social network structures [13] social media [14], and online behavior [15]. In our previous studies, we have shown that Benford’s Law applies to friend and follower counts in social media across platforms, both at the platform level and the individual level [16]. We also showed that violations of Benford’s Law in network structure can be used to identify malicious bots [9].
Rank-frequency distribution, Zipf’s Law, and language
The frequency of a word in a corpus is the number of times it occurs. The rank-frequency, or frequency rank, is the rank of a word when the frequencies of all words are sorted from most to least frequent. The most frequent word in a corpus would have a rank of 1. Frequency rank distributions, not just in languages but of most data, tend to follow a power law distribution [17]. Power laws appear frequently in nature and occur when one variable varies as a power of the other.
Zipf’s law is a particular type of power law distribution that states that frequency is inversely proportional to rank. It was originally formulated with respect to linguistic analysis, addressing the frequency of words in a corpus [18]. Researchers have found that word frequency adheres to Zipf’s law [19].
Zipf’s Law and Benford’s Law have some relationship. Studies have shown that in some contexts, Benford’s Law and Zipf’s law both apply, but not all Zipfian distributions follow Benford’s Law [12].
Despite Zipf’s Law originating from the study of language, and a statistical connection between Benford’s Law and Zipf’s law, there is almost no research into Benford’s Law applying to language. One study found that Chinese character and word frequency could be described by Benford’s Law on small-scale texts [20]. However, there are no studies on large corpora, that look at other languages, or that address frequency rank and Benford’s Law.
Methods and data
Our foundational question in this research is if Benford’s Law applies to word frequency rank across languages. If Benford’s Law consistently holds across diverse datasets in a given language, it is evidence that we can expect Benford to hold for most (but not necessarily all!) datasets in that language. That, in turn, would indicate that we may be able to use Benford testing to look for “abnormal” behavior which, in turn, could be an initial indicator of fraudulent or manipulated behavior.
The benefit of using frequency rank is that it allows comparison across corpora. The frequency count of the word “that”, for example, will vary in every corpus, but it’s rank should be approximately the same. In this analysis, using frequency rank also allows us to establish a standard frequency rank for the language as a whole, using very large datasets, which we can then use to evaluate the use in each corpus based on the same values. For example, “that” will be ranked 8 in English and considered the 8th most frequent English word in every corpus, rather than re-measuring it within each corpus.
Establishing word frequency rank
The frequency of a given word in a language can never really be measured since, for example, the vast majority of words spoken are never recorded. Even in recorded language, frequency depends on many factors, including time period, context, region, and format (e.g. written or spoken). It must be measured within a corpus, and corpora vary with respect to all of these factors. For this research, we opted to use the largest corpora available to establish a baseline frequency and frequency rank for words.
We used the Google Books Ngram Viewer dataset [21]. These datasets are built from a large collection of around 8 million books published between 1500 and 2008, which account for roughly 6% of all books ever published. The English dataset is the largest, with approximately 1 trillion words. The German, Spanish, and French datasets each have over 100 billion words, and the Italian dataset has over 60 billion. These are orders of magnitude larger than any other available corpora.
Google makes frequency counts available for each word. To calculate frequency and frequency rank, we first removed tokens that were not words (numbers, punctuation) and then sorted by frequency. We assigned each word a frequency rank beginning at 1 for the most popular word (e.g. “the” in the English corpus).
Analyzing one dataset is not enough to make claims about general rules for language use [22]. In this case, if English in one dataset follows a Benford’s Law distribution, that does not necessarily mean we can draw conclusions about language or even English generally. To mitigate these concerns, we followed the guidelines of [21], considering multiple languages and validating our results on different datasets in the target languages.
We applied the same ranking process to the Google datasets for German, French, Spanish, and Italian. We used these frequency ranks for our Benford’s Law analysis of each language. We also used these frequency ranks as a basis for analyzing other corpora in each language to see if the Benford’s distribution was observable in smaller datasets drawn from a variety of sources. These corpora are discussed in the next section.
Additional corpora
To check if Benford’s Law applies to word usage with respect to frequency rank, we tested on a number of additional large corpora across all five languages. These are drawn from news, social media, books, and the web. The number of total words and unique words in each dataset are given in Tables 2–6. The purpose of this analysis was to validate results obtained from the Google Ngrams analysis on very diverse corpora. If Benford’s Law applies robustly to language use, it should be observable in many contexts, corpora of many sizes, and from a variety of sources. Thus, the purpose in choosing these was not to select the biggest or most balanced or those from a specific source. Rather, they were chosen to test how well Benford works in a range of contexts and languages.
10.1371/journal.pone.0291337.t002
Distribution of FSD of word’s frequency rank based on use in Google Ngram Viewer corpora.
Total word counts are measured in billions (B).
FSD
Benford
English
German
Spanish
Italian
French
1
0.30
0.32
0.30
0.33
0.32
0.32
2
0.18
0.18
0.19
0.17
0.18
0.18
3
0.12
0.13
0.13
0.11
0.12
0.12
4
0.10
0.10
0.10
0.09
0.09
0.09
5
0.08
0.08
0.07
0.08
0.08
0.08
6
0.07
0.07
0.06
0.07
0.06
0.07
7
0.06
0.05
0.05
0.06
0.06
0.06
8
0.05
0.04
0.05
0.05
0.05
0.05
9
0.05
0.04
0.04
0.04
0.04
0.04
Total Words
755 B
103 B
107 B
64 B
167 B
Unique words
4,999,751
4,540,689
2,250,889
1,825,177
2,741,602
Pearson ρ
1.000
0.998
0.996
0.999
0.999
K-S value
0.222
0.111
0.111
0.111
0.111
K-S p-value
0.9895
1
1
1
1
10.1371/journal.pone.0291337.t003
Distribution of FSD of word’s frequency rank based on use in additional English corpora.
FSD
Benford
Parler
Cresci
BNC
Leipzig
1
0.30
0.30
0.29
0.31
0.31
2
0.18
0.15
0.17
0.17
0.16
3
0.12
0.14
0.13
0.13
0.13
4
0.10
0.11
0.11
0.10
0.10
5
0.08
0.07
0.07
0.08
0.08
6
0.07
0.07
0.07
0.07
0.07
7
0.06
0.06
0.07
0.05
0.05
8
0.05
0.05
0.04
0.05
0.05
9
0.05
0.05
0.04
0.04
0.04
Total Words
1,760,010,838
73,520,025
95,168,106
17,310,077
Unique words
1,848,342
620,828
927,596
417,396
Pearson ρ
0.989
0.993
0.998
0.997
K-S Value
0.111
0.222
0.222
0.111
K-S p-value
1
0.9895
0.9895
1
10.1371/journal.pone.0291337.t004
Distribution of FSD of word’s frequency rank based on use in additional German corpora.
FSD
Benford
DeRoKo
Aranea
COW
Leipzig
1
0.30
0.31
0.30
0.30
0.32
2
0.18
0.19
0.19
0.19
0.18
3
0.12
0.13
0.14
0.14
0.13
4
0.10
0.10
0.10
0.10
0.09
5
0.08
0.08
0.08
0.08
0.08
6
0.07
0.06
0.06
0.06
0.06
7
0.06
0.05
0.05
0.05
0.05
8
0.05
0.04
0.04
0.05
0.04
9
0.05
0.04
0.04
0.04
0.05
Total Words
7,301,006,311
100,379,842
11,331,629,659
15,089,080
Unique words
98,072
483,408
71,092,150
838,937
Pearson ρ
0.999
0.997
0.997
0.999
K-S Value
0.222
0.222
0.222
0.222
K-S p-value
0.9895
0.9895
0.9895
0.9895
10.1371/journal.pone.0291337.t005
Distribution of FSD of word’s frequency rank based on use in additional Spanish corpora.
FSD
Benford
COW
Aranea
LELU
Leipzig
1
0.30
0.34
0.33
0.36
0.33
2
0.18
0.16
0.17
0.16
0.17
3
0.12
0.11
0.11
0.11
0.11
4
0.10
0.09
0.09
0.09
0.09
5
0.08
0.09
0.08
0.08
0.08
6
0.07
0.07
0.07
0.05
0.08
7
0.06
0.06
0.06
0.05
0.06
8
0.05
0.04
0.04
0.04
0.05
9
0.05
0.03
0.04
0.04
0.04
Total Words
3,163,716,419
97,268,117
75,709,619
19,817,544
Unique words
8,540,745
272,471
3,800,336
413,687
Pearson ρ
0.989
0.995
0.991
0.994
K-S Value
0.222
0.222
0.222
0.111
K-S p-value
0.9895
0.9895
0.9895
1
10.1371/journal.pone.0291337.t006
Distribution of FSD of word’s frequency rank based on use in additional French corpora.
FSD
Benford
Aranea
Reddit
Leipzig
1
0.30
0.32
0.34
0.32
2
0.18
0.18
0.19
0.19
3
0.12
0.14
0.11
0.13
4
0.10
0.09
0.11
0.09
5
0.08
0.07
0.06
0.07
6
0.07
0.06
0.06
0.06
7
0.06
0.06
0.05
0.06
8
0.05
0.05
0.05
0.04
9
0.05
0.03
0.03
0.03
Total Words
104,152,457
66,518,066
19,419,591
Unique words
261,637
772,033
360,383
Pearson ρ
0.997
0.995
0.998
K-S Value
0.111
0.222
0.222
K-S p-value
1
0.9895
0.9895
Many of these corpora contain non-standard words like usernames, hashtags, misspellings, and URLs. They also have “words” which are numbers, codes, punctuation, emoji, etc. We applied a high level filter to remove numbers and punctuation. Other non-standard words that did not appear in the Google datasets were ignored. While this can constitute a significant percentage of the unique words in a corpus, it usually reflects a small percentage of the total words. For example, in the Parler dataset, there were 1,848,342 unique words but only 690,590 were in the Google corpus. While it may appear that the majority of words were excluded from analysis, these excluded words made up only 0.79% of all the words used on Parler. Most were usernames and hashtags (e.g. jreidusmct, funnyquotes, ycgoiszxsvipw, oyrgdjxfgxsl, xfajndhbbzz, saddayforamerica). Thus, because the removed words are such a small percentage of all words used, their exclusion should have a negligible impact on the Benford’s analysis. (See also [7] for an analogous discussion of the presence of zeroes in financial data).
Cresci (English—Social Media)
The Fake Project dataset is a dataset of Twitter bots and genuine accounts [23]. Across four categories of accounts, the dataset contains 6.6 million tweets.
Parler (English—Social Media)
Parler was a Twitter-like microblogging platform that became popular among Trump supporters and conservatives, particularly in 2020. It tagged itself as “Parler: Speak freely and express yourself openly without fear of being deplatformed for your views” and was seen as a haven after far-right figures were suspended from Twitter for violating their Terms of Service. Parler was a major planning platform for the January 6 insurrection in the United States and it was shut down within a few days of the attacks on the Capitol when their service providers broke ties.
We analyzed a dataset of Parler posts that spans August 2018 to January 2021 and was collected and described in [24]. It includes 183 million posts from 4 million users. Most posts are in English, and we only considered the English language posts in our analysis.
British National Corpus (English—wide coverage)
The British National Corpus (BNC) [25] is a large corpus of samples of English, both written and spoken. It is a balanced corpus which is supposed be representative of how English is used. The written samples, which make up around 90% of the corpus, are collected from newspapers, magazines, journals, fiction and non-fiction books, letters, school essays, and other text. The spoken portion also seeks to use a wide ranging sample, including lectures, radio shows, call-in shows and classroom discussions.
We used the unlemmatized frequency list from the BNC provided at [26].
The German Reference Corpus DeRoKo (German—books and newspapers)
The German Reference Corpus or Deutsches Referenzkorpus (DeRoKo) [27] is a large reference corpus of German texts. It is built from fiction, scientific manuscripts, and newspapers. We conducted our analysis using the DeReKo-2014-II-MainArchive-STT.100000.freq list, which contains approximately 7 billion words.
The Aranea corpora comprise web pages crawled beginning in 2013 [28–30]. We used the German, Spanish, French, and Italian corpora in our analysis.
Leipzig News Corpus (German, Spanish, French, Italian, English) The Leipzig corpora are collected from newspaper text and released yearly for each language [31, 32]. We used the 1 million sentence corpora from 2008 for English, Spanish and German, and the integrated 2005-2008 corpus for French and Italian.
Corpora from the Web—COW (German, Spanish—web pages)
The Corpora from the Web project provides web corpora for multiple European languages, including German and Spanish [33, 34].
This focused corpus of around 47,000 tweets was collected in June 2022 after the overturn of Roe v Wade. The tweets are in Italian and were selected if they contained “abortion”, “aborto” or “abortire”. We intentionally include this smaller and more focused dataset as a test of whether Benford’s Law applies to word use in specific contexts.
LÉLU French dialog corpus (French—Social Media)
The LÉLU French dialog corpus [35] is a collection of 556,621 comments from French-language subreddits: /r/france, /r/FrancaisCanadien, /r/truefrance, /r/paslegorafi, and /r/rance.
120M Word Corpus (Spanish—Wikipedia)
The 120M word Spanish Wikicorpus [36] is collected from Spanish-language Wikipedia pages.
ResultsBenford’s Law applies across corpora
To investigate the presence of Benford’s Law in language, we considered words’ frequency rank, as described above.
We then represented each word by the FSD of its frequency rank. In this case, both “the” (rank 1) and “it” (rank 10) would be represented as a 1, “of” would be 2, etc. We have frequency counts for each word, so we summed the frequencies for all words with each FSD to get the total occurrence of each FSD in the corpus. Dividing this by the total number of words in the corpus produces the frequency of each FSD.
For the Google corpora, we computed these values for each language. The frequency of each FSD is shown in Table 2. We first computed the Pearson correlation between the observed frequency and the frequency expected if Benford’s Law applies. Pearson correlations are a common way to measure how closely a distribution adheres to Benford’s Law [37–39].
As shown in Table 2, all correlations are almost perfect, with the lowest ρ = 0.996. All correlations were significant for p < 0.001 with a Bonferroni correction of 23 to account for all corpora correlations performed in this study.
We also ran Kolmogorov–Smirnov (K-S) tests to check the fit of the data with the Benford’s Law distribution. The K-S p-values, which indicate the probability that the social network’s FSD distribution is the same as Benford’s, are also shown in Table 2. These values are all > 0.98, indicating a near perfect match with the expected distribution.
We performed the same analysis on additional corpora using the frequency ranks from the Google dataset. As with the large Google corpora, the Pearson correlation was strong on every additional corpus. Every value was > 0.96 and all were significant for p < 0.001 with the Bonferroni correction described above. Similarly, the K-S tests showed strong adherence to Benford’s Law, with p-values > 0.98. Results for each language are shown in Tables 3 to 7.
10.1371/journal.pone.0291337.t007
Distribution of FSD of word’s frequency rank based on use in additional Italian corpora.
FSD
Benford
Aranea
Twitter—Abortion
Leipzig
1
0.30
0.33
0.35
0.32
2
0.18
0.17
0.16
0.17
3
0.12
0.10
0.12
0.11
4
0.10
0.06
0.09
0.11
5
0.08
0.09
0.07
0.08
6
0.07
0.09
0.07
0.06
7
0.06
0.06
0.06
0.06
8
0.05
0.08
0.04
0.05
9
0.05
0.04
0.05
0.05
Total Words
94,921,972
1,175,775
18,716,950
Unique words
314,426
55,373
430,832
Pearson ρ
0.967
0.989
0.996
K-S Value
0.111
0.111
0.222
K-S p-value
1
1
0.9895
User-level analysis on social media
In previous studies, we showed that Benford’s Law applied to the network structure of social media platforms as a whole and also to individual users’ egocentric networks [16]. When individual users significantly deviated from the expected Benford distribution, it was often an indicator that the accounts were bots. This inspired similar questions for this analysis. Does Benford’s Law hold for the frequency rank of individual users’ word choice and, if so, does deviation from Benford indicate bot-like behavior?
Among the social media corpora considered in this analysis, Parler and Cresci had many tweets per individual user. The French Reddit and Italian Twitter corpora often had only one post per user which was not enough text to conduct this type of analysis.
The Cresci data has an additional benefit in that it is a labeled dataset for analyzing bots. Accounts are labeled as “genuine” or one of several types of bots. We considered genuine accounts, social spam bots, and traditional spam bots.
For each user in these groups, we built a set of their tweets. We excluded any users who had posted fewer than 1,000 words to ensure our statistics would be meaningful. This left us with 1,048 genuine accounts, 2,214 social spam accounts, and 160 traditional spam accounts.
For each user, we measured the distribution of FSDs of the frequency rank of each word they used. We then compared each individual user’s FSD distribution against the expected Benford distribution. As is common practice, we used the Pearson correlation to measure how well the distributions fit.
The mean correlation coefficient for genuine accounts (0.977) was significantly higher than that of social spam bots (0.947) or traditional spam bots (0.908). A one-way ANOVA revealed that there was a statistically significant difference in the Pearson correlation coefficients between at least two groups (F(2, 3419) = [281.1596], p = 0). Tukey’s HSD Test for multiple comparisons found that the mean correlation coefficient was significantly different between genuine users and social spam bots (p =< 0.001, 95%C.I. = [0.061, 0.078]) and between genuine users and traditional spam bots (p =< 0.001, 95%C.I. = [0.026, 0.033]).
In the Parler dataset, accounts are not labeled as bots or not. As a preliminary investigation, we sorted accounts by their Pearson ρ and qualitatively analyzed 100 accounts—the 50 with the strongest correlations and the 50 accounts with the weakest correlations. Based on visual inspection of the content, there did not appear to be any spam bots among the top Benford-correlated Parler users. Among the 50 weakest correlates, inspection of their posts indicated 45 were obvious spam bots and 3 more were bots posting other automated content. Among the obvious spam bots were accounts that posted the same message over and over. These are representative examples of the repeated posts from five different accounts:
Welcome! Follow me for conservative Memes and News.
Welcome to Parler, Patriot! Follow me and I will definitely give you a follow back!
Follow me for funny conservative comedy on parler Don’t forget to echo!
Follow me for more Trump tweets! GOD bless all you Patriots!
The White Rabbit welcomes you to Parler. Hope to see you around the hole! Follow the White Rabbit…
Discussion and conclusions
We have shown that Benford’s Law applies to the frequency rank distribution of words in English, French, German, Spanish, and Italian. Using large datasets from a range of sources, we consistently found a strong correlation and statistical adherence to the expected Benford’s Law distribution. We also showed that Benford’s Law applies to the smaller samples of individuals’ social media posts in the vast majority of cases. However, spam bots tend to have a lower correlation with Benford’s distribution and strong deviations from the expected distribution are indicators of bot activity.
While many bots do closely adhere to the FSD distribution predicted by Benford’s Law, analysis for the Cresci bot dataset shows that there is some signal present indicating that a deviation from the expected distribution is tied to bot status. Furthermore, among the biggest deviators in the Parler dataset, almost all were spam bots. While correlation with Benford alone is not strong enough to indicate an account is a bot or not, these results suggest this could be one factor to include in bot detection algorithms to potentially improve their performance.
These results suggest a number of avenues for future work. First, given there is some relationship between data that follows Zipf’s Law and Benford’s Law [12], Benford tests can be expanded further. We have focused on Western European languages and a natural next step would be to verify this in a wider range of languages. Existing work suggests this likely would work, given the word frequency results in Chinese [20]. Zipf’s Law has also been shown to hold in the vocalizations of other species, including dolphins [40] and penguins [41]; does Benford’s Law apply as well? Additionally, Zipf himself posited psychological theories for why his law works. He that the relationship exists to minimize effort both for the speaker and hearer. The speaker wants to use the fewest number of words, while the hearer wants the words with the clearest meaning used [42]. There is an intriguing line of work to connect psychological research on Zipf’s Law and language with Benford’s Law to see if there are insights to be uncovered that can psychologically explain Benfordian behavior.
ReferencesDurtschiC, HillisonW, PaciniC. The effective use of BenfordÕs law to assist in detecting fraud in accounting data. Journal of forensic accounting. 2004;5(1):17–34.KochC, OkamuraK. Benford’s law and COVID-19 reporting. Economics letters. 2020;196:109573. doi: 10.1016/j.econlet.2020.10957332952242SambridgeM, JacksonA. National COVID numbers–Benford’s law looks for errors. Nature. 2020;581(7809):384–385. doi: 10.1038/d41586-020-01565-532457528HoyleDC, RattrayM, JuppR, BrassA. Making sense of microarray data distributions. Bioinformatics. 2002;18(4):576–584. doi: 10.1093/bioinformatics/18.4.57612016055DiekmannA. Not the First Digit! Using Benford’s Law to Detect Fraudulent Scientif ic Data. Journal of Applied Statistics. 2007;34(3):321–329. doi: 10.1080/02664760601004940HillTP. The First Digit Phenomenon A century-old observation about an unexpected pattern in many numerical tables applies to the stock market, census statistics and accounting data. American Scientist. 1998;86(4):358–363. doi: 10.1511/1998.31.358AusloosM, FiccadentiV, DhesiG, ShakeelM. Benford’s laws tests on S&P500 daily closing values and the corresponding daily log-returns both point to huge non-conformity. Physica A: Statistical Mechanics and its Applications. 2021;574:125969. doi: 10.1016/j.physa.2021.125969Bonettini N, Bestagini P, Milani S, Tubaro S. On the use of Benford’s law to detect GAN-generated images. In: 2020 25th international conference on pattern recognition (ICPR). IEEE; 2021. p. 5495–5502.Golbeck J. Benford’s law can detect malicious social bots. First Monday. 2019;.BenfordF. The law of anomalous numbers. Proceedings of the American Philosophical Society. 1938; p. 551–572.NigriniMJ. Benford’s Law: Applications for forensic accounting, auditing, and fraud detection. vol. 586. John Wiley & Sons; 2012.PietroneroL, TosattiE, TosattiV, VespignaniA. Explaining the uneven distribution of numbers in nature: the laws of Benford and Zipf. Physica A: Statistical Mechanics and its Applications. 2001;293(1):297–304. doi: 10.1016/S0378-4371(00)00633-6BarabásiAL, AlbertR. Emergence of scaling in random networks. science. 1999;286(5439):509–512. doi: 10.1126/science.286.5439.50910521342Asur S, Huberman BA, Szabo G, Wang C. Trends in social media: Persistence and decay. Available at SSRN 1755748. 2011;.GilesDE. Benford’s law and naturally occurring prices in certain ebaY auctions. Applied Economics Letters. 2007;14(3):157–161. doi: 10.1080/13504850500425667GolbeckJ. Benford’s law applies to online social networks. PloS one. 2015;10(8):e0135169. doi: 10.1371/journal.pone.013516926308716LiW. Random texts exhibit Zipf’s-law-like word frequency distribution. IEEE Transactions on information theory. 1992;38(6):1842–1845. doi: 10.1109/18.165464FaganS, GençayR. An introduction to textual econometrics. Handbook of empirical economics and finance. 2010; p. 133–153.PiantadosiST. Zipf’s word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review. 2014;21:1112–1130. doi: 10.3758/s13423-014-0585-6Gao J, Zhao Y, Cui R. Research on the Applicability of Benford’s Law in Chinese Texts. In: 2020 2nd International Conference on Artificial Intelligence and Advanced Manufacture (AIAM). IEEE; 2020. p. 13–17.YounesN, ReipsUD. Guideline for improving the reliability of Google Ngram studies: Evidence from religious terms. PloS one. 2019;14(3):e0213554. doi: 10.1371/journal.pone.021355430901329PechenickEA, DanforthCM, DoddsPS. Characterizing the Google Books corpus: Strong limits to inferences of socio-cultural and linguistic evolution. PloS one. 2015;10(10):e0137041. doi: 10.1371/journal.pone.013704126445406Cresci S, Di Pietro R, Petrocchi M, Spognardi A, Tesconi M. The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race. In: Proceedings of the 26th international conference on world wide web companion; 2017. p. 963–972.Aliapoulios M, Bevensee E, Blackburn J, Bradlyn B, De Cristofaro E, Stringhini G, et al. A large open dataset from the parler social network. In: Proceedings of the International AAAI Conference on Web and Social Media. vol. 15; 2021. p. 943–951.Consortium B, et al. British national corpus. Oxford Text Archive Core Collection. 2007;.Kilgarriff A. BNC database and word frequency lists;. Available from: http://www.kilgarriff.co.uk/bnc-readme.html.Kupietz M, Belica C, Keibel H, Witt A. The German Reference Corpus DeReKo: A Primordial Sample for Linguistic Research. In: Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10); 2010.Benko V. Aranea: Yet another family of (comparable) web corpora. In: Text, Speech and Dialogue: 17th International Conference, TSD 2014, Brno, Czech Republic, September 8-12, 2014. Proceedings 17. Springer; 2014. p. 247–256.Benko V. Compatible sketch grammars for comparable corpora. In: Proceedings of the XVI EURALEX International Congress: The User In Focus; 2014. p. 15–19.Rychlỳ P. Manatee/Bonito-A Modular Corpus Manager. In: RASLAN; 2007. p. 65–70.Biemann C, Heyer G, Quasthoff U, Richter M. The Leipzig corpora collection-monolingual corpora of standard size. Proceedings of Corpus Linguistic. 2007;2007.Collection LC. News corpus based on material from 2022;. https://corpora.uni-leipzig.de.Schäfer R, Bildhauer F. Building large corpora from the web using a new efficient tool chain. In: Lrec; 2012. p. 486–493.Schäfer R. Processing and querying large web corpora with the COW14 architecture. In: Proceedings of the 3rd Workshop on Challenges in the Management of Large Corpora; 2015. p. 28–34.El Bawab A, Considine B, Saleh Z. LELU: A French Dialogue Corpus from Reddit; 2017.Reese S, Boleda G, Cuadros M, Padró L, Rigau G. Wikicorpus: A word-sense disambiguated multilingual wikipedia corpus. In: Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10); 2010.JudgeG, SchechterL. Detecting problems in survey data using BenfordÕs Law. Journal of Human Resources. 2009;44(1):1–24. doi: 10.3368/jhr.44.1.1BhattacharyaS, XuD, KumarK. An ANN-based auditor decision support system using Benford’s law. Decision support systems. 2011;50(3):576–584. doi: 10.1016/j.dss.2010.08.011JolionJM. Images and Benford’s law. Journal of Mathematical Imaging and Vision. 2001;14(1):73–81. doi: 10.1023/A:1008363415314Vradi AA. Dolphin communication: a quantitative linguistics approach. Universitat Politècnica de Catalunya; 2021.FavaroL, GambaM, CrestaE, FumagalliE, BandoliF, PilengaC, et al. Do penguins’ vocal sequences conform to linguistic laws?Biology letters. 2020;16(2):20190589. doi: 10.1098/rsbl.2019.058932019463LindersGM, LouwerseMM. Zipf’s law revisited: Spoken dialog, linguistic units, parameters, and the principle of least effort. Psychonomic Bulletin & Review. 2023;30(1):77–101. doi: 10.3758/s13423-022-02142-93584083710.1371/journal.pone.0291337.r001Decision Letter 0De BenedictisLucaAcademic Editor2023Luca De BenedictisThis is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.Submission Version0
1 Jun 2023
PONE-D-23-08610Benford's Law Applies to Word Frequency Rank in English, German, French, Spanish, and ItalianPLOS ONE
Dear Dr. Golbeck,
Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.
Please submit your revised manuscript by Jul 16 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.
Please include the following items when submitting your revised manuscript:
A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.
If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.
If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.
We look forward to receiving your revised manuscript.
Kind regards,
Luca De Benedictis, PhD
Academic Editor
PLOS ONE
Journal Requirements:
When submitting your revision, we need you to address these additional requirements.
1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at
https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and
1. Is the manuscript technically sound, and do the data support the conclusions?
The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.
Reviewer #1: Yes
**********
2. Has the statistical analysis been performed appropriately and rigorously?
Reviewer #1: Yes
**********
3. Have the authors made all data underlying the findings in their manuscript fully available?
The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.
Reviewer #1: Yes
**********
4. Is the manuscript presented in an intelligible fashion and written in standard English?
PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.
Reviewer #1: Yes
**********
5. Review Comments to the Author
Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)
Reviewer #1: PONE-D-23-08610
Reviewer comments
The present paper demonstrates that Benford's Law applies to ranked word frequencies in English, German, French, Spanish, and Italian. The author(s) computed the word frequency ranks using the Google Ngram Viewer corpus to prove this. By examining the first significant digit of the frequency ranks, the author(s) shows that the distribution of frequencies adheres to the expected pattern outlined by Benford's Law. Furthermore, across various sources such as news, books, and social media the author(s) investigated, we consistently found conformity to Benford's Law. Moreover, the authors analysed users' posts on social media and discovered that the vast majority of collected posts followed Benford's Law, while significant deviations from this pattern often indicated the presence of spam bots.
Introduction
In referring to Benford’s law studies in general and specifically on the stock market, more recent works may help contextualise. For example, Ausloos, M., Ficcadenti, V., Dhesi, G., & Shakeel, M. (2021). Benford’s laws tests on S&P500 daily closing values and the corresponding daily log-returns both point to huge non-conformity. Physica A: Statistical Mechanics and its Applications, 574, 125969.
The following statement should be simplified in favour of readers not familiar with Google 1-gram corpora. Furthermore, it is unclear what is intended by word’s frequency rank; namely, have you applied Benford’s law on the ranks or the frequencies? It may become clearer later in the text, but I feel it is important to say it here. Finally, what is “gold standard” in this context?
“We conducted an analysis of language use in a variety of corpra. To determine a word's frequency rank (e.g. what is the 314th most popular word in English), we measured frequency using the Google 1-gram corpra (reviewer’s note: CORRECT THE TYPO HERE) for English, German, French, 13 Spanish, and Italian. These frequency ranks were used as the gold standard rank in each language.”
A similar comment on the clarity of what “rank” and what “frequency” are under investigation should be applied to the paragraph related to social media posts.
Background and Related Work
This section could be improved by adding a practical example of the type of analysis that can be done with Benford’s law in linguistics. For example, mentioning a word, its rank in a corpus, and the way to consider its frequency. This comment is connected with the one made above about clarity.
Establishing Word Frequency Rank
One reads, “Google makes frequency counts available for each word. To calculate frequency and frequency rank, we first removed tokens that were not words (numbers, punctuation) and then sorted by frequency. We assigned each word a frequency rank beginning at 1 for the most popular word (e.g. “the" in the English corpus).” taken that a corpus is a book and that a corpora is a collection of books in a given language, does one have a rank for each word in every corpus? So, for example, one has N English books, each having a number of words and associated frequencies. I understand from the Results sections that words are considered altogether. Namely, one gets “approximately 1 trillion words” with their frequencies and ranks for English, correct? If so, I suggest to clarify here.
Additional Corpra
When the author(s) starts this section, the analysis and its results are not presented yet, but the basement of the sub-section is a need to validate the results. This is understandable, but a more thoughtful discussion of the potential results and their generalisability is necessary before introducing a “validation” argument.
In the following statement, “If Benford's Law applies robustly to language use, it should be observable in many contexts, corpra of many sizes, and both balanced and unbalanced corpra.” what does “balanced” mean?
Results
In this section, one finally understands, with an example, the concept of “frequency rank”. The usage of this approach is partially justified in the section “Rank-frequency distribution, Zipf's Law, and Language” but it is not enough because the obvious thought I had by reading is “why not use the frequencies directly?” I like the idea of using the ranking, so I am not discouraging, but some more on the reasons, the benefit of it, might be beneficial.
User-Level Analysis on Social Media
The methodology here must be clarified. Is the analysis based again on the “frequency rank”?
I do not get if the conclusion related to the fact that “bot” social network accounts are less Benford’s law compliant comes from a visual inspection or if it is possible to have some statistical test to validate that point.
Conclusions:
If the author(s) decides to keep any of the comments on board, please revise the conclusion accordingly. Furthermore, I suggest that, given the quality of the results in the field, more academically informed conclusive remarks can be given on the areas covered: the linguistic aspect and the social network component.
General comments:
When one removes non-standard text, I refer, e.g. to “Other non-standard words are unlikely to appear in the Google corpra and will be ignored” there might be a “manipulation” of the data that somewhat connect to what is done in the case of accountancy when some figures are systematically changed. Therefore, extra care should be used. Some of those words might be neologisms, words that are outside current dictionaries but in use in “hurban” conversation; this is especially relevant for the component of the analysis related to social networks. My question would be, does consider words in official dictionaries or including those not-in-dictionaries would change the results drastically? If it is not possible to re-do the analysis, I understand. I am not necessarily asking for that, but a few comments on the impact on Benford’s law validity would help. This is partially there already when the %s are reported, but a clear statement where one says that these numbers are not enough for breaking Benford’s law may reassure a reader. For example in “Ausloos, M., Ficcadenti, V., Dhesi, G., & Shakeel, M. (2021). Benford’s laws tests on S&P500 daily closing values and the corresponding daily log-returns both point to huge non-conformity. Physica A: Statistical Mechanics and its Applications, 574, 125969” a point regarding the inclusion or exclusion of the “zeros” from the returns analysis is made when evaluating the second digit law.
Check the usage of the words “corpora” and “corpus” I feel sometimes the usage of the words is interchanged, but the meaning is different.
Congratulations on using genuine and non-genuine accounts for the analysis related to the bots! I really like it!
**********
6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.
If you choose “no”, your identity will remain anonymous but your review may still be made public.
Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.
Reviewer #1: No
**********
[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]
While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.
10.1371/journal.pone.0291337.r002Author response to Decision Letter 0Submission Version1
2 Jun 2023
(this response is also included as a submission document)
Response to Reviewers
Thank you so much for these very helpful comments. They have allowed us to make the paper much better and have been a tremendous help to the clarity. Below we have listed the reviewer comments in black and our responses in blue. We made all the changes requested by the reviewers.
Introduction
In referring to Benford’s law studies in general and specifically on the stock market, more recent works may help contextualise. For example, Ausloos, M., Ficcadenti, V., Dhesi, G., & Shakeel, M. (2021). Benford’s laws tests on S&P500 daily closing values and the corresponding daily log-returns both point to huge non-conformity. Physica A: Statistical Mechanics and its Applications, 574, 125969.
Thank you for this reference. We have added this to the introduction and also as a useful reference in the background to point out that Benford's law does not simply always apply in this type of data.
The following statement should be simplified in favour of readers not familiar with Google 1-gram corpora. Furthermore, it is unclear what is intended by word’s frequency rank; namely, have you applied Benford’s law on the ranks or the frequencies? It may become clearer later in the text, but I feel it is important to say it here. Finally, what is “gold standard” in this context?
“We conducted an analysis of language use in a variety of corpra. To determine a word's frequency rank (e.g. what is the 314th most popular word in English), we measured frequency using the Google 1-gram corpra (reviewer’s note: CORRECT THE TYPO HERE) for English, German, French, 13 Spanish, and Italian. These frequency ranks were used as the gold standard rank in each language.”
A similar comment on the clarity of what “rank” and what “frequency” are under investigation should be applied to the paragraph related to social media posts.
We have clarified this sentence, added background on the 1-gram corpora, and dropped "gold standard" from the paragraph for clarity.
We also moved the entire frequency rank discussion up to the introduction to address the many points about clarity raised by the reviewer.
Background and Related Work
This section could be improved by adding a practical example of the type of analysis that can be done with Benford’s law in linguistics. For example, mentioning a word, its rank in a corpus, and the way to consider its frequency. This comment is connected with the one made above about clarity.
This point should be addressed by the addition of the frequency rank discussion to the introduction.
Establishing Word Frequency Rank
One reads, “Google makes frequency counts available for each word. To calculate frequency and frequency rank, we first removed tokens that were not words (numbers, punctuation) and then sorted by frequency. We assigned each word a frequency rank beginning at 1 for the most popular word (e.g. “the" in the English corpus).” taken that a corpus is a book and that a corpora is a collection of books in a given language, does one have a rank for each word in every corpus? So, for example, one has N English books, each having a number of words and associated frequencies. I understand from the Results sections that words are considered altogether. Namely, one gets “approximately 1 trillion words” with their frequencies and ranks for English, correct? If so, I suggest to clarify here.
Google refers so their 1 Trillion Word dataset as a single corpus, rather than a corpora of all the books. After reviewing your comments (including one about corpora vs corpus below) think this is likely the source of the confusion. To clarify, we've updated the text to refer to the google data as a "dataset" (since it has more data in it than just words, anyway).
Additional Corpora
When the author(s) starts this section, the analysis and its results are not presented yet, but the basement of the sub-section is a need to validate the results. This is understandable, but a more thoughtful discussion of the potential results and their generalisability is necessary before introducing a “validation” argument.
We have added this discussion to the beginning of the Methods and Data section, just above Establishing Word Frequency Rank.
In the following statement, “If Benford's Law applies robustly to language use, it should be observable in many contexts, corpra of many sizes, and both balanced and unbalanced corpra.” what does “balanced” mean?
"Balanced" corpora are intended to reflect overall usage, including speaking, fictional writing, non-fiction, etc. Honestly, it's only a concept we encountered in the process of doing this work and is not critical to the paper, so we've updated this sentence to instead refer to data "from a variety of sources" which captures the idea without the jargon.
Results
In this section, one finally understands, with an example, the concept of “frequency rank”. The usage of this approach is partially justified in the section “Rank-frequency distribution, Zipf's Law, and Language” but it is not enough because the obvious thought I had by reading is “why not use the frequencies directly?” I like the idea of using the ranking, so I am not discouraging, but some more on the reasons, the benefit of it, might be beneficial.
Generally, statistical linguistics use frequency rank rather than just frequency because it allows for easy comparison between different datasets. The frequency of the word "and" will be quite different in a 1 trillion word dataset vs. a 10 million word one. Of course, that doesn’t really matter here for Benford's Law analysis, but it does matter for cross-dataset comparison. We could do a Benford test on the frequency counts within each dataset (which, incidentally, we did in early stages of this work and everything looks pretty Benford-y), but we wouldn't be comparing apples-to-apples across datasets then. Using the frequency rank lets us say for a language overall, here's the rank of each word, and we can check every dataset with that same rank.
This is an important point and we have added a more formal discussion of this point to the Methods and Data section.
User-Level Analysis on Social Media
The methodology here must be clarified. Is the analysis based again on the “frequency rank”?
I do not get if the conclusion related to the fact that “bot” social network accounts are less Benford’s law compliant comes from a visual inspection or if it is possible to have some statistical test to validate that point.
We have re-written this section to clarify the methodology.
For the Parler accounts, we made the conclusion about bot status based on a human analysis of their posts. We have re-written this paragraph to make that clearer.
Conclusions:
If the author(s) decides to keep any of the comments on board, please revise the conclusion accordingly. Furthermore, I suggest that, given the quality of the results in the field, more academically informed conclusive remarks can be given on the areas covered: the linguistic aspect and the social network component.
We have updated the conclusion to include additional discussion along these lines.
General comments:
When one removes non-standard text, I refer, e.g. to “Other non-standard words are unlikely to appear in the Google corpra and will be ignored” there might be a “manipulation” of the data that somewhat connect to what is done in the case of accountancy when some figures are systematically changed. Therefore, extra care should be used. Some of those words might be neologisms, words that are outside current dictionaries but in use in “hurban” conversation; this is especially relevant for the component of the analysis related to social networks. My question would be, does consider words in official dictionaries or including those not-in-dictionaries would change the results drastically? If it is not possible to re-do the analysis, I understand. I am not necessarily asking for that, but a few comments on the impact on Benford’s law validity would help. This is partially there already when the %s are reported, but a clear statement where one says that these numbers are not enough for breaking Benford’s law may reassure a reader. For example in “Ausloos, M., Ficcadenti, V., Dhesi, G., & Shakeel, M. (2021). Benford’s laws tests on S&P500 daily closing values and the corresponding daily log-returns both point to huge non-conformity. Physica A: Statistical Mechanics and its Applications, 574, 125969” a point regarding the inclusion or exclusion of the “zeros” from the returns analysis is made when evaluating the second digit law.
The challenge with these words is that, since they aren't in the Google dataset for each language, they do not have a frequency rank. These do not make a difference to the analysis – we point out in the Additional Corpora section that they are <1% of the data. We agree that additional description of their impact on the validity is important and we have added that text along with a reference to the paper mentioned.
Check the usage of the words “corpora” and “corpus” I feel sometimes the usage of the words is interchanged, but the meaning is different.
Thanks for this – we had a consistent typo with "corpra" instead of "corpora" on top of all the other confusion. We were trying to use "corpora" simply as the plural of corpus. We hope this has been clarified in this version, but if we are unintentionally mis-using this still, we'd appreciate the feedback!
Congratulations on using genuine and non-genuine accounts for the analysis related to the bots! I really like it!
Thank you! We are so excited about these results!
Submitted filename: Response to Reviewers.pdf
10.1371/journal.pone.0291337.r003Decision Letter 1De BenedictisLucaAcademic Editor2023Luca De BenedictisThis is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.Submission Version1
29 Aug 2023
Benford's Law Applies to Word Frequency Rank in English, German, French, Spanish, and Italian
PONE-D-23-08610R1
Dear Dr. Golbeck,
We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Please, a careful final reading is recommended to spot remaining typos.
Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.
An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.
If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.
Kind regards,
Luca De Benedictis, PhD
Academic Editor
PLOS ONE
Comments to the Author
Reviewer #1: All comments have been addressed
**********
2. Is the manuscript technically sound, and do the data support the conclusions?
The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.
Reviewer #1: Yes
**********
3. Has the statistical analysis been performed appropriately and rigorously?
Reviewer #1: Yes
**********
4. Have the authors made all data underlying the findings in their manuscript fully available?
The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.
Reviewer #1: Yes
**********
5. Is the manuscript presented in an intelligible fashion and written in standard English?
PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.
Reviewer #1: Yes
**********
6. Review Comments to the Author
Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)
Reviewer #1: Very good work, extensive and properly executed. A careful final reading is recommended to spot remaining typos.
**********
7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.
If you choose “no”, your identity will remain anonymous but your review may still be made public.
Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.
Reviewer #1: No
**********
10.1371/journal.pone.0291337.r004Acceptance letterDe BenedictisLucaAcademic Editor2023Luca De BenedictisThis is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
4 Sep 2023
PONE-D-23-08610R1
Benford's Law Applies to Word Frequency Rank in English, German, French, Spanish, and Italian
Dear Dr. Golbeck:
I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.
If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.
If we can help with anything else, please email us at plosone@plos.org.
Thank you for submitting your work to PLOS ONE and supporting open access.