Our motivating example is the DARPA Network Challenge [7], [9]. This challenge required teams to provide coordinates of red weather balloons placed at different locations in the continental United States, offering a reward of to the first team to report all correct locations.

This large-scale mobilization required the ability to spread information about the tasks widely and quickly, and to incentivize individuals to act. The MIT team completed the challenge in hours and minutes. In approximately hours prior to the beginning of the challenge, the MIT team was able to recruit almost individuals through a recursive incentive mechanism.

The MIT team's approach was based on the idea that achieving large-scale mobilization towards a task requires diffusion of information about the tasks through social networks, as well as incentives for individuals to act, both towards the task and towards the recruitment of other individuals. This was achieved through a recursive incentive mechanism, which is illustrated in Figure 1. The mechanism distributes up to per balloon to people along the referral path that leads to the balloon. The person who finds the balloon gets , his immediate recruiter (or, parent) gets one half of the finder's compensation, etc. In Figure 1, agent recruits all of his neighbors, namely , and , while agent recruits , who finds balloon . The finder receives . Since recruited , it gets . From this sequence, receives .

Likewise, looking at the left recruitment path, the finder receives . As above, we have for and for . From this sequence, receives . Adding up its payments from the two sequences it initiated, receives a total payment of . Notice that the amount distributed to the agents never exceeds per balloon. In this example was paid for the first balloon and — for the second. The MIT team donated the undistributed money to charity.

The contracts described above can be dubbed split contracts, specifying the percentage of total child's reward that must be passed back to the parent. In particular, the MIT's winning strategy used a -split contract across all nodes.

However, as mentioned above, the MIT's strategy assumed that any balloon citing report is correct. Yet, in the actual challenge, verifying balloon reports turned out to be a major obstacle [10]. Indeed, 124 out of 186 reports turned out to be false either by sabotage or by mistake. Figure 2 shows all of the reported locations, highlighting the prevalence of false reports in this kind of time-critical task (see [9] for examples of misreports). In this paper, we make a first attempt to model and tackle the verification problem.

We model scenarios where the center has an information gathering task. The information-seeking entity is represented by the root of a tree, and each node in the tree holds the answer required to complete the task with a fixed probability . Information about the task (or the question) is propagated through referrals sent from parents to their children until a node holding the answer is found. The nodes along the path are compensated to ensure that once an answer-holding node is reached, it reports the answer (thus, is the probability of holding and returning the answer). Unlike the other models, we allow for the possibility of false reports.

A crucial modeling choice regards the cause of the false reports. A rational agent that has no stake in the mechanism except for the compensation received, has no incentive to lie as false reports never result in a compensation. One may consider a richer population of agents where some agents derive utility from the mechanism not succeeding in recovering the true answer, or from increasing the time until a true answer is discovered. Designing a compelling utility function for this “lying” type of agent is an option. However, in such models, false reporting can be overcome by a payment that is high enough to make the agents' utility from truthful reporting a better option.

Given this, we pursue a different, simpler modeling avenue which does not rely on agent utilities. Instead, with probability each node happens to be “irrational”. Such a node does not hold the true answer, but claims that he does and sends a false report to its parent. In other words, is the probability that the node does not hold the answer and generates a false report. These irrational agents are not affected by penalties and compensation: they lie irrespective of the incentives. In this model, misreports cannot be prevented but have to be discovered. Such a model of misreports is consistent with ignorant rather than malicious behavior: e.g., misreports due to mistakes and noise. As we alluded to earlier, modeling malicious behavior requires making assumption about the utility functions of the malicious agents, and it remains an avenue for future research. A good starting point may be the work on spiteful bidding (e.g., see [17], [18]).

As soon as a “reporting” node is recruited and generates an answer (correct or mistaken), it can no longer recruit other nodes as, from its point of view, the answer has already been recovered. Therefore, a reporting node is always a leaf node and only leaf nodes can generate false reports.

Now that we settled on how false reports arise, we need to model the verification process. We are going to assume that a node other than the reporter can verify the report with certainty. Naturally, verification requires some effort which we model with the cost incurred by the node performing verification. Note that under the “perfect” verification, it is sufficient to obtain just one verification. The next question is which node should perform the verification. The most immediate candidate is the parent of the reporting node. After all, it is the parent node who decides which children to invite, and it is reasonable to hold him accountable for his invitees. Also, from the point of view of invited children, the first point of task-related contact for them is the parent. Furthermore, nobody except for the recruiter may have the authority/ability to question the recruit.

Given the assumptions above, we model the sequence of events next. A report goes from the reporting node to the node who recruited it — its immediate parent in the referral tree. On receiving a report, the immediate parent can verify whether the report is correct incurring the cost . If the report is verified to be false, it is dropped. Otherwise, the immediate parent submits it directly to the root. To encourage verification, we assume the mechanism supports penalties. If a false report is propagated from a leaf node back to the root, the immediate parent of has to pay the penalty . Penalizing the leaf node does not make sense as a node submitting a false report is irrational and, thus, indifferent to monetary incentives. Penalizing ancestors other than the immediate parent is not fair as they cannot verify the report or control its submission to the root.

Following [11], we propagate rewards using split contracts (we discuss why the choice of split contracts is justified in the Related Work section). Suppose node has the correct answer. Let refer to split contracts offered on the path from the root node to node : i.e., the root offers -split to the first node on the path, who offers -split to the second node, etc. The fraction of the reward received by each node is shown in Table 1.

We will be concerned with incentivizing the immediate parent of the reporting node to participate and perform verification. His share of the reward is . For example, under the -split contract, the parent of a reporting node receives a quarter of the reward.

The model detailed above is specified by the probabilities of false and true reports, the verification cost, the penalty level, the reward provided by the root, and the split contract determining allocation of the reward (see Table 2). While the first 3 parameters are exogenous, the root is likely to have control over the penalty level, as well as distribution of the reward. Clearly, it is in the root's interest to minimize the reward given out. In this section we derive the split contract which minimizes the reward required to recover the answer. We also describe the penalty level sufficient to ensure that verification takes place and no false reports are propagated.

Recall that and refer to the probabilities of submitting true and false answers respectively. These events are disjoint and the probability that a report is correct is . Let denote the reward offered by the root. Consider a reporting node and his parent . If the report is correct, the parent will receive resulting in the expected reward of . When deciding on whether or not to verify the report, the parent must consider the verification cost , and the penalty for propagating false reports. Verification cost is incurred regardless of the report accuracy, while the penalty is paid only if the report is false. The utility of the parent performing verification appears on the left hand side while the utility when no verification is performed is on the right:(1)Thus, the parent prefers verifying the report when the verification cost is below the expected penalty .

Our work can be seen as an application of mechanism design [20] to social networks and information gathering tasks. The model has similarities to the model of Query Incentive Networks (QIN) presented by Kleinberg and Raghavan [19]. In that model, the root needs to recover an answer from a network of nodes where each node has a small probability of holding the answer. In order to encourage nodes to return the answer, the root proposes a reward that is propagated down the tree. Once an answer-holding node is recruited, it sends the answer to its parent, who forwards it to the grandparent, and so on until the root is reached. There is a constant (integer-valued) cost incurred by each node on the path from the answer-holding node back to the root. The authors describe the minimum reward required to obtain the answer with high probability when each parent can offer a reward to its children.

Our model is similar to the QIN model in that we are searching to retrieve an answer from the network where the question is propagated via invitations that parents send to their children. The main novel ingredients in our model are (i) the possibility of false reports; (ii) the option to verify the reports at a cost; and (iii) the ability of the root to penalize false report submissions. These attributes appear in many real-world settings making our model more readily applicable.

It is interesting to note that the introduction of costly verification and penalties allowed us to dispense with one of the assumptions of the QIN model: the costly propagation of the answer is no longer required. Without this assumption, the QIN model admits degenerate solutions, where the root gets the answer for an arbitrarily small cost (indeed, the required reward would also be zero in our model if we set the probability of false reports to zero). Our disposal of this assumption is important, for example in situations where propagating the answer has negligible cost (e.g., forwarding an email or re-tweeting) relative to a demanding verification task (e.g., checking if a balloon report is authentic by personally sighting it), or when the nodes can send the answer directly to the root without propagating it up the referral path.

Our model restricts attention to split contracts. However, this seems to be the right class of contracts to focus on for the following reasons. The simplest and most common alternative is fixed rewards: each parent promises its children a fixed amount. While original work on QIN considered fixed contracts [19], Cebrian et al. [11] showed that a significantly lower reward is required when using split contracts. Also, in our setting, where verification and recovery of the answer must occur with certainty, fixed contracts are inappropriate: any contract that offers a fixed amount to each node will require an infinite investment to be recovered with probability one as the answer may be arbitrarily deep within the tree. Another natural idea is to share the reward equally among all the nodes along the path. In this context, this division of the reward coincides with the Shapley value [21]. However, such a division would also require an infinite reward in our model: for an answer that is levels deep, the root would have to pay each of the nodes along the path the minimum reward that a node will expect to undertake verification. Notice that may be arbitrarily large.

Another justification of split contracts comes from the work of Emek et al. [22]. The authors take an axiomatic approach to show that a special case of split contracts with an equal split at each level arises naturally in multi-level marketing. Similar to our model, in multi-level marketing recursive referrals are sought from the participants. A fundamental difference is that in the marketing model participants are compensated for each referral they make, while in QIN and our context, only referrals that contribute to finding the answer generate a reward.

We acknowledge that in reality recruitment trees are finite and potentially not very deep, while our assumption is that propagation of referrals can produce arbitrarily deep trees, reaching very large (possibly infinite) numbers of individuals. Indeed, recent work suggests that we live in a “small but slow world”, as social network topology and human burstiness can actually hinder information propagation, effectively reducing the population reached [23]–[28]. However, as the DARPA Network Challenge showed, wide dissemination of information does occur in certain scenarios [29]–[33].

Since the seminal experiments by social psychologist Stanley Milgram in the 1960s, it has been established that social networks are very effective at finding target individuals through short paths [1]. Various explanations of this phenomenon have been given [3], [4], [6], [34]. However, it has also been recognized that the success of social search requires individuals to be motivated to actually conduct the search and participate in the information diffusion. Indeed, it has been shown experimentally that while successful chains happen to be short, the majority of chains observed empirically terminate prematurely [5]. Dodds et al. conclude that “although global social networks are, in principle, searchable, actual success depends sensitively on individual incentives” [5]. In other words, a key challenge in social search is the incentive challenge. However, while models like the Query Incentive Networks model [19], and the split-contract approach [7] both provide incentives for diffusion, the problem of verifying the received reports has not previously received any formal treatment.

The issue of verification arises in many real-world crowdsourcing scenarios (e.g., mapping social uprisings [35], [36] or gathering disaster response requests [37], [38]). Indeed, some competitive scenarios have even been subject to larger levels of sabotage, as illustrated by the attack on the crowdsourced strategy to tackle the 2011 DARPA Shredder Challenge [39]. For such settings, our paper provides the first steps towards formally analyzing verification schemes. Specifically, we introduced a model for studying verification in referral-based crowdsourcing. We explored the relationship between various parameters, including the size of the reward offered by the mechanism, the probability of possessing the answer, the probability of false reports, the cost of verifying the correctness of reports, and the penalty imposed by the mechanism on false reports. Our main theoretical result is the proof that the optimal distribution of the reward in our model is given by the -split contract. This contract happened to be the one used by the winner of the Red Balloon Challenge, showing that this way of sharing the reward is also appealing in practice. Our paper provides the first theoretical justification of this mechanism in the presence of misinformation. Our second contribution is in initiating a formal study of verification in information-gathering scenarios. Our model provides a starting point for future research where various assumptions may be relaxed. We outline some directions next.

We provided results for the uniform and known verification cost. Bitcoin provides an example of a real system where this assumption holds: the expected computational cost of authorizing a transaction is uniform and known (see [40] for more details on Bitcoin). In other scenarios such as quality verification of crowdsourced tasks (e.g., accuracy of a translation, deciding whether a photo is authentic, or evaluating a programming job) costs may be heterogeneous as well as the private information of the agents. Our model can be directly extended to introduce heterogeneous and private costs for the analysis of such scenarios.

In online scenarios one may easily create multiple identities. Thus, it is particularly important to consider mechanisms that are resilient to coalitions of lying nodes. Since coalitions may easily control an entire referral path, verification from nodes outside the path is likely to be required. This question has been tackled with “uniform” rather than split contracts in [40]. Moreover, for split-contracts, some lessons may be drawn from the results on false-name-proof mechanisms for multi-level marketing [22].

It is also interesting to weaken some of our other modeling assumptions. For example, if we no longer assume that the probability that a node holds the answer (or lies) is uniform, the mechanism should encourage the recruitment of individuals more likely to possess the answer, perhaps based on the knowledge that agents have about the abilities and reliability of their peers. An even more selective recruitment is likely to arise if the cost of recruiting others is non-zero.

Our model makes the assumption that the split contract (e.g., -split) is selected by the mechanism and cannot be modified by other nodes. This contrasts with the models of [11], [19], where each node chooses which contract to offer to its children, and the resulting equilibrium contract is analyzed. An important direction for further study, therefore, is to perform equilibrium analysis in our model, when nodes not only choose who to recruit and whether to verify, but also what split to offer.

Finally, an important extension of our work is to explore the dynamics of strategic behavior in the context of repeated interaction. In particular, a threat of non-monetary punishment may be sufficient to encourage verification. For example, in permanent systems such as Amazon Mechanical Turk, Wikipedia or Bitcoin, the penalty may be imposed in the form of decreased reputation, which diminishes future earning potential or the influence a user exercises [41].