Empirical Legal Research : The Gap between Facts and Values and Legal Academic Training

‘Traditional legal scholarship is under pressure. Debates are taking place on the aims and methods of the academic study of law.’ These are some of the introductory words in Smits.1 Debates range from the ‘disruptive’ influence that digitization, machine learning and Big Data may have on the profession2 and the methodology of legal studies;3 to the relationship between empirical research and normative questions and the difficulties lawyers have in incorporating results from empirical studies into normative scholarship and practice.4 However, at the same time it seems like a ‘revolution’ is taking place in law, at least according to Ho & Kramer.5 They counted the proportion of Stanford Law Review articles mentioning the word ‘empirical’ over a period of almost 60 years, which made them call the development a revolution (Figure 1). A word count is a thin indicator of the ‘empirical engagement’ of legal researchers, as using this word does not necessarily imply that empirical research has been carried out. Seidman Diamond & Mueller6 searched deeper and analyzed the content of 60 law review volumes published between 1998 and 2008. ‘Our content analysis revealed that by 2008 nearly half of law review articles included some empirical content. Production of original research is less common.’7 They also reported that ‘evaluating the place of empirical scholarship in law reviews, has attracted a flurry of attention and a variety of approaches (...).’8 Klick9 studied the content of eight journals publishing in the field of law and economics (like the Review of Law and Economics and the Journal of Law, Economics and Organization) and analyzed the ‘empirical


Introduction: A discipline in crisis and an empirical revolution in law?
'Traditional legal scholarship is under pressure.Debates are taking place on the aims and methods of the academic study of law.' These are some of the introductory words in Smits. 1 Debates range from the 'disruptive' influence that digitization, machine learning and Big Data may have on the profession 2 and the methodology of legal studies; 3 to the relationship between empirical research and normative questions and the difficulties lawyers have in incorporating results from empirical studies into normative scholarship and practice. 4owever, at the same time it seems like a 'revolution' is taking place in law, at least according to Ho & Kramer. 5They counted the proportion of Stanford Law Review articles mentioning the word 'empirical' over a period of almost 60 years, which made them call the development a revolution (Figure 1).
A word count is a thin indicator of the 'empirical engagement' of legal researchers, as using this word does not necessarily imply that empirical research has been carried out.Seidman Diamond & Mueller 6 searched deeper and analyzed the content of 60 law review volumes published between 1998 and 2008.'Our content analysis revealed that by 2008 nearly half of law review articles included some empirical content.Production of original research is less common.' 7 They also reported that 'evaluating the place of empirical scholarship in law reviews, has attracted a flurry of attention and a variety of approaches (…).' 8 Klick 9 studied the content of eight journals publishing in the field of law and economics (like the Review of Law and Economics and the Journal of Law, Economics and Organization) and analyzed the 'empirical market share' of these journals.Data were collected from the journals' first issue onwards.He found that the share of empirical articles only declined for one journal while for the others it increased, though not dramatically or revolutionarily.

Figure 1 The proportion of Stanford Law Review articles mentioning the word 'empirical' over a period of almost 60 years
Engel studied the coverage of behavioural law and economics within one journal (Journal of Empirical Legal Studies) devoted to empirical contributions. 10He found 77 articles since the journal's inception (in 2004) that focus on this topic, 32 addressing behavioural papers on private law, 27 covering an issue from criminal law and 5 dealing with public law.In the field of international law, Chilton & Tingley11 interpreted the development as an 'empirical turn in the study of international law' that took place in the first decade of the 21st century.Simmons & Breidenbach12 also mention the 'empirical turn in legal scholarship [that] generally has been pretty well-documented.Indeed, there is even a law school ranking [in the USA] based on institutional strength in empirical legal studies.In the specific area of international economic law, the trend is less noted, but is on the rise.' 13 Although the impression may arise that the empirical turn in legal studies is something which marks the last two decades, the reality is different.American Legal Realism14 goes back to the first part of the 20 th century and is a tradition with a focus on empirical research, including 'sociological jurisprudence' .Legal realists were also active in applied legal work related to the New Deal policy and its implementation.Kritzer puts it as follows: 'In the 1920s and 1930s, and in a few cases even earlier, one can find a wide range of empirically-oriented research on law [in the USA].The specific topics of this early research include: appellate courts and appellate decision making, automobile accident compensation and litigation, bankruptcy, criminal courts, divorce, judicial staffing and judicial selection, juries and legal needs and legal aid.' 15 In Europe, sociology of law studies by Weber, Durkheim, Petrazycki, Gurvitch and Ehrlich were sometimes also empirical in nature, while the same holds true for parts of criminology, legal psychology, law and economics and other 'law and …' specialties.Although civilology is seen as a new 'kid on the block' , not only is the concept over 100 years old 16 but work done by Dutch-German law scholars like Meijers, Hijmans and Hamaker in the first part of the 20 th century showed their inclination towards empirical work. 17lthough it is an exaggeration to call these developments a 'revolution', empirical legal research is blossoming.Van Dijck refers to a 'booming' empirical legal studies movement. 18Experimental legislation, regulatory impact assessments and experiments in and with private law arrangements but also studies describing and analyzing in an empirical way treaties and protocols, including transnational governance and international law can easily be found in the literature.Research on theories underlying legal arrangements 19 must be added, as is the case with systematic reviews of existing research on the impact and societal acceptance of (penal) sanctions and behavioural modification programmes implemented in prisons and elsewhere.Specialized institutions producing these reviews like the Campbell Collaboration and several other 'Clearinghouses' have been established over the last few decades.
Empirical legal research (ELR) is rooted in a diversity of disciplines, sub-disciplines and specialties, some going back for centuries.Figure 2 shows the roots of ELR.The more empirical legal research is a 'growth industry' , the more important it is to understand and discuss epistemological, methodological and translational problems of this field of study.Epistemological problems deal with the kind of knowledge that is produced and the accumulation of knowledge over time.Methodology addresses how research problems are related to designs of studies, the role of theories, data collection and data analysis, including how to operationalize legal concepts and where to find data (stored, but also 'Big Data').Problems of a translational character are how to bring empirical evidence to the fore, in such a way that it can be understood and used by lawyers, legislators and regulators.A crucial element of translational activities is the gap between facts ('evidence') and values, also known as the factvalue dichotomy and the 'Bewertungsproblem':21 how does one link empirical (including causal) evidence to the normativity of legal arrangements and legal scholarship?
It is this problem that we focus on in this paper.Our perspective is what students of law, including PhD candidates and legal practitioners (in training), need to know about this problem and how to address it.As the field of empirical legal research is blossoming, the more necessary it is that students and practitioners are not only familiarized with methodological aspects of empirical legal research (research designs, data collection, Big Data and analysis, statistics and visualization), but also with this issue.Burns22 and van Gestel et al. 23 are of the opinion that within current academic-legal education, there is room for improvement.This includes the problem of how to link empirical evidence to the normativity of legal arrangements and legal scholarship.24

The gap between facts and values
Concerning the normative character of legal scholarship and legal arrangements Smits recently said the following: 'The legal discipline reflects what it is that individuals, firms, states, and other organizations ought to do, or ought to refrain from doing.Typical legal questions are thus: whether disinheriting one's children should be permitted, whether the death penalty should be imposed for criminal offences, under which circumstances it is justified to go to war, when constitutional review should be allowed, and whether ship-wrecked sailors may eat their weakest companion if they are likely to die of starvation.' 25 The gap between robust, empirical evidence on -for example -the deterrent effect of the death penalty or the consequences for the well-being of children, when they are disinherited, on the one hand, and the legal-normative argumentation to be in favour or against the death penalty or disinheritization, is serious.Giesen formulated the gap problem (for private law and psychological research) as follows: ' An intriguing, and as yet unresolved question underlying all these kinds of studies is whether it is in fact possible -and if so, how, why and when -to leap from extralegal (e.g.psychological) insights to normative legal conclusions.Given that facts in themselves cannot generate values, how and when can any decision maker or researcher step over from, for example, empirical psychological facts to legal normative value judgments as one is required to do from a legal end, for instance as a judge, or from a public policy perspective?If psychological research tells us -to give but one example -that warning signs are only followed by those people who have been given the warning if the costs of complying with that warning are low, could a judge then conclude that a legal duty to warn should be rejected, as being superfluous, in all other circumstances?' 26 The topic of how to relate results from empirical (descriptive/causal) research to normative legal questions is by no means a novel one.Hume (1888) may be one of the first to have addressed it, referring to what seems to be a significant difference between descriptive statements (about what is) and prescriptive or normative statements. 27Weber, in his study on 'Die Objektivität sozialwissenschaftlicher und sozialpolitischer Erkenntnis' (1904),28 also discussed this problem and made the point that 'eine empirische Wissenschaft vermag niemanden zu lehren, was er soll, sondern nur, was er kann und -unter Umständen -was er will.' According to Weber, the social and cultural sciences including economics are never capable 'bindende Normen und Ideale (...) [zu] ermitteln, um daraus für die Praxis Rezepte ableiten zu können' .Lepsius concluded in 2005 that even a thorough way of establishing facts will not do away with the Bewertungsproblem, 'the problem of adding normative value to facts: no legal obligation follows from empirical facts.'29

Approaches to the gap between facts and values 30
Three issues are on the agenda: -Is it necessary that students of law know about this dichotomy and approaches to deal with it?Our answer is yes.Not being aware of this problem will either lead to refraining from using empirical evidence, to compare apples and oranges without knowing the differences or to pick and choose certain evidence but deny other empirical findings (for example, because they seem not to fit or 'verify' the normative statements already formulated).
-Is it necessary to bridge the gap?Our answer is yes and a great many other authors agree. 31One argument goes back to Roscoe Pound and Oliver Holmes: law in the books and blackletter law is important but not enough to understand how 'law' develops, what it does and does not to society, how it can be made (more) effective and -sometimesless harmful?A second argument is that when lawyers decide on normative issues, formulate verdicts, and introduce rules and legislation, often with far-reaching consequences, without knowing right from wrong regarding behavioural mechanisms, pathways, consequences and side-effects, this will create legal arrangements without a 'reality check' . 32an the gap between facts and values be bridged?It will probably depend on the availability and applicability of (translational) approaches.A number of them have been presented by Giesen33 and I will add several more.

The Giesen collection
In his paper 'The use and incorporation of extralegal insights in private law reasoning' Giesen lists a number of authors addressing this problem.We have summarized several of them in the next table.

Table 1 Approaches to address the gap between facts and values problem based on Giesen
Author Approach

Robbennolt
In 2002, 'Robbennolt set out to discuss "the persistent tension between the methods of social science and the theory, goals, and settings of law and policy".She starts with the warning that to utilize empirical research means that there are trade-offs to be made.The question is "how to appropriately use well-done but inherently imperfect research, for legal and policy purposes".
If one evaluates empirical research, for instance as a judge in a tort case on the perceived effectiveness of a warning sign, one should be concerned about different forms of the validity of the research in question, such as construct validity, internal validity and external validity.
The person (thinking about) using the data from, for instance, experimental studies should not uncritically accept the results of such studies as actually representing the way judges make decisions.However, uncritically rejecting results is equally bad since experimental research provides useful information about how people decide, understand instructions, etc.Thus, neither accepting results at face value, nor rejecting results out of hand is sensible; more systematic consideration is needed.'34

Lepsius
Lepsius addressed the Bewertungsproblem 'by actually reformulating the issue as a mere problem of (legal) evidence: it is for the law to decide which facts (at stake in legal proceedings) need proof from a legal-normative angle since these are the facts which are needed to determine the existence of some form of legal consequences.It is those facts so decided upon that would need to be "proven" by the social sciences, much in the same way as a judge would call upon a medical expert to determine medical facts.The judge (or more broadly: a lawyer) should not be meddling in this terrain himself as is now often still the case.' 35 Engel 'Engel aims to find out why his (…) fellow lawyers are so reluctant to use social sciences, even for descriptive purposes (…) He (…) elaborates on the (possible) reasons for the reticence in the legal community towards the use of social science (…) [and] claims that the integration of social science is in fact an art, incapable of resting on a "one size fits all" answer.In fact "every new case, every new topic and every new academic paper must find the individually best way to carry off the integration." (…) Engel does provide us with some generalizations that might be useful in some cases.First, he points to the use of a procedural instead of a substantive governance of this complex issue, which would be typical for lawyers.Second, he encourages us to treat different sorts of cases differently (…).Third, he proposes to distinguish between the generation and the representation of court decisions, writing down a more accessible justification for a decision that was based on methods from social science.Fourth and foremost in this regard, he proposes that legal academics, trained in social sciences, serve as intermediaries, as so-called interface actors serving both lawyers as well as methodological standards when integrating law and social science.'36 Mertz '[Mertz] introduces a new (…) vantage point which insists "that we study the process of interdisciplinary translation itself " (…) "(…) Analysis from diverse disciplinary points of view teaches us that this translation process is far from transparent.The important task ahead of us, then, is to develop better understandings of legal and social scientific 'transduction' -or translation in the more complex sense (…)." (…) Mertz has proposed (…) in a very broad fashion, to use insights from linguistic anthropology in thinking about how to make the transition from social science to law, and to avoid problems while doing so (…).'37

Vranken
Vranken refers to the importance of translating the evidence produced by empirical (legal) researchers (and links that also with the Daubert standard) (see below).His approach is to distinguish between categories of translational activities instead of trying the one size fits all.First, there is a difference between thinking and operations of judges (who have the explicit task of judging, i.e. formulating normative statements) and researchers, whose explicit task it is to deliver an adequate research product.A second category are the researchers who also want to present recommendations (of a normative nature).While judges, according to Vranken, apply current legislation and jurisprudence as a criterion, researchers cannot do the same (as the current law and jurisprudence may be the cause of the problems that they investigate).How translational activities precisely take place and how transparent they are, remains unclear in Vranken's paper. 38See also Van Boom 39 who is critical of Vranken's references to conventions, rules of the game and informal practices.
One of Giesen's conclusions as to what these and a few other authors had to offer in addressing the gap problem is that the reader '(…) probably feels at least slightly disappointed.Scanning the available methodological literature does not really get us much further.We do know that the issue is real and serious enough; we do know that we need to work on it.But how?' 40 One reason is that the authors discussed by Giesen reformulate the problem in somewhat different terms without coming closer to a solution. 41Or they see the problem as 'a mere problem of (legal) evidence' 42 and as a weighting process of factors and arguments 'lawyers should not be afraid of, because they are 'by nurture' already trained in weighing all sorts of arguments, principles, factors, points of view, figures, and so on, when deciding cases.' 43 Another reason is that the solutions suggested appear to have the same or similar difficulties as the 'original' problem or are, at their best, first steps, 44 to be developed further.I largely agree with Giesen's analysis. 45ith respect to the Daubert standard that Giesen only mentioned briefly (by saying that 'the judge himself already [is] the gatekeeper [about the type of evidence allowed to be used]' , 46 there is more to say.It is the standard used by trial judges in the USA to make a preliminary assessment of whether an expert's scientific testimony is based on reasoning or methodology that is scientifically valid and can properly be applied to the facts at issue.Under this standard, the factors that may be considered in determining whether the methodology is valid are: (1) whether the theory or technique in question can be and has been tested; (2) whether it has been subjected to peer review and publication; (3) its known or potential error rate; (4) the existence and maintenance of standards controlling its operation; and (5) whether it has attracted widespread acceptance within a relevant scientific community.Faigman 47 is of the opinion that 'although it has taken more than 200 years, [it] initiated a scientific revolution in the law.'

Giesen's due process approach
Being not very comfortable with the approaches reviewed, Giesen took the challenge to develop his own approach, the due process approach.The first part of it is to be '(…) cautious when using insights from elsewhere in a legal discussion leading to legal consequences; law is not only about psychology, or sociology or economics, it is also (and perhaps mainly) about value judgments being made at a given point in time at a given place.This cautious approach would then have it that a judge, practitioner or legal scholar is only "allowed" -in the scientific sense of the word -to leap from extralegal insights to legal solutions if certain (formal, procedural) criteria have been satisfied: if due process is attended to.The following non-exhaustive set of criteria that ultimately deal with rather common methodological problems (such as construct validity, internal validity and external validity biases) might be listed here as relevant criteria that the judge or scholar should consider and weigh, taken together, before using empirical insights in his legal reasoning: -whether the empirical work is in fact relevant for the question of law that arises, -whether the work is up to the current state of the art in the field methodologically, as well as regards its research design, etc., and its implications, -whether (more generally) the research is valid and reliable, -whether there is conflicting empirical work on the same issue, -whether the study has been replicated and confirmed or not, -whether the study is but one building block of a larger set of studies needed for policy implications, -whether the researcher is both an expert and objective and independent, and so on.
With regard to all of these factors, and others that might of course be added, the reasoned justification provided by the user of the extralegal information (the judge deciding the case, and so on) would be crucial.That justification would, for instance, need to deal with the issue, raised above, that aggregated data are used in individual cases.But if and when these criteria have been duly considered, weighed against one another, and justified, the extralegal materials can be considered reliable (enough) and may thus be used in the decision-making process (again: there would be no obligation to do so).The legal or public policy outcome may then be inspired by the empirical insights found.To put it differently: the Sein can then be used to answer the Sollen, basically because all possible safeguards have been put in place.' (…) An important consequence of the due process approach advocated here is of course that it asks of judges, practitioners and scholars to be or at least become "somewhat" (…) familiar with the methodology of the social science at stake.That hurdle might also prove to be gigantic and insurmountable.But as long as that is the case (…) this difficulty might still be overcome by using court-appointed experts to collect or at least evaluate the usefulness of the extralegal materials available (…). ' 48 Although Giesen's approach to the gap problem is interesting (and related to the Daubert standard), it basically runs into the same difficulties as discussed above.Although probably nobody is against 'due process' and is positive about adequate translation/transduction activities, there continues to be a problem, i.e. leaping from valid (and relevant) empirical evidence (compliant with due process criteria or the Daubert standard) to formulating normative statements.Why is this?An example clarifies my point.It concerns microcredit and lookalike microfinance programmes, which are well-known in the developing world as policy instruments based on (soft) laws and regulation.Recently, four systematic research reviews 49 were published.A systematic review summarizes the results of empirical studies evaluating the impact of microcredit programmes.The primary studies have been reviewed and scrutinized on the basis of a protocol which includes Giesen's methodological due process criteria and others. 50The review process is to distinguish the wheat from the chaff; only those studies that pass the methodological criteria are used for the analysis and synthesis.The four systematic reviews raise serious doubts about what the impact of microcredits on women and society is.'(…) there appears to be a gap between the often optimistic societal belief in the capacity of microcredit to ameliorate the position of women in decision-making processes within the household on the one hand, and the empirical evidence base on the other hand.We conclude that there is no consistent evidence for an effect of microcredit on women's control over household spending.Given the overall lack of evidence for an effect of microcredit on women's control over household resources it is therefore very unlikely that, overall, microcredit has a meaningful and substantial impact on empowerment processes in a broader sense.' 51 Although the evidence from this and the other systematic reviews is strong and convincing, the question remains how legal counselors, legal scholars and policy advisors in the field of microcredits and development aid, will operate when they are confronted with these results?How do they 'move' from the sophisticated and crystal-clear findings about the absence or near absence of microcredits' impact on women's empowerment to answering the question whether or not to continue, to abandon or to modify microcredit programmes?How can the empirical results help or guide the persons who have to advise on or decide about such a question?
A simple answer would be: 'abandon this intervention, as robust research has found that…' .This is too simple, as there can be other factors, both normative (including ideological) and political ones, at stake that may nevertheless 'value' microcredits so highly that instead of abandoning the intervention, the robust research evidence is 'abandoned' , i.e. not used.Normative beliefs may and sometimes outweigh the evidence.Exactly as Giesen has indicated: 'But the novel insight [in this case: robust results from several systematic reviews] itself is not enough; there might be one or more good reasons not to follow up on that insight, given the other arguments presented to the decision maker.' 52 Even when following a due process approach, the core of the gap problem continues to exist.
And there is another development contributing to the gap problem not addressed by Giesen.The more lawyers and legal scholars collaborate with applied social scientists carrying out evaluations, assessments and regulatory research, 53 the larger the likelihood that they are confronted with the practice of presenting recommendations that (sometimes) are normative in nature.As they strive for the utilization of their findings by policy makers and others, the goal to influence decision-makers' knowledge, attitudes, norms and values is high on their agenda.Patton 54 refers to utilization-focused evaluation, which is based on the principle that an evaluation should be judged on its usefulness to its intended users.Therefore, evaluations should be planned and conducted in ways that enhance the likely utilization of both the findings and of the process itself to inform decisions and improve performance.Of a different nature is 'nudging' , an approach in the behavioural and economic sciences which argues that positive reinforcement and indirect suggestions to achieve non-forced compliance can influence the motives, incentives and decision making of groups and individuals, at least as effectively -if not more effectively -than direct instruction, legislation, or enforcement.However, Whyte 55 makes the point that this kind of policymaking 'provides a mechanism for academic elites to impose their own values on society as a whole' .Earlier, Van de Vall & Bolas 56 defended the idea that applied research should primarily focus on persuading and influencing policy makers in the 'right' direction and referred to the researcher as a change agent.These authors even suggested that the types of methods used by social scientists should be partly dependent upon what helps to realize this goal.Again, values and norms of researchers are dominant but what exactly are the links between the empirical research and the normative statements, remains unclear.Therefore, the gap or the Bewertungsproblem is not only a problem for lawyers working with empirical researchers, but also for social scientists believing that it is commendable to present recommendations (of a normative, value-laden content).In both situations there is a (mystical) leap from IST to SOLL.
To take the discussion on the gap problem a step further, I have added to Giesen's collection several other approaches.The first characterizes law as an argumentative discipline, two others have as their background evaluation studies and the fourth empirically informed ethics.57

The law as an argumentative discipline
Smits defines legal science as an argumentative discipline.
'The core of legal science is the behaviour of the homo juridicus (what it is that people should do as a matter of law) (…) If one's research question is not what the law says, but what it should say, empirical material can be used to test whether some idea or argument was already used elsewhere and how it was received in that other jurisdiction.In my view, the most important research method to evaluate arguments is therefore the comparative one.(…) Other jurisdictions should in this respect be seen as "experimenting laboratories".' 58 He adds that 'the aim of legal studies is not to put an end to normative uncertainty but to take this uncertainty as a starting point.(…) This leads to a characterization of legal studies as the discipline of conflicting arguments.' 59 A crucial question which Smits asks is: 'how we should establish what is the better argument?For some part the answer must be found in the normative presuppositions underlying the acceptance of an argument (…) I agree that each argument can only be assessed within a certain normative framework.But in doing so, we should not forget that in many jurisdictions there is already such a framework available in the form of a doctrinal system.Each jurisdiction has its own 'internal morality' as a reflection of the prevailing normative views within that jurisdiction.(...) This view of legal methodology implies that each normative scholarly exercise consists of two steps.
-The first is to identify the relevant arguments in favour of and against a certain solution.
Several methods can be used to do this, including empirical approaches, but in the end the comparative method is the most promising one.-The second step is to see whether these arguments fit into an already existing normative setting.' 60 What, then, are the criteria to compare and weigh normative statements like values?Smits did not answer that question but Ball 61 and Lint 62 present several criteria that can be used.The first is completeness.To what extent does the argument address all key aspects that are at stake and to what extent are important ones left out?This requires that the analyst is aware of all important values, by, for example, studying the history of the policy or regulation and by examining public opinion.The second criterion is that of relevance.The question is to what extent the embodied values are appropriate, checkable in a similar vein as for completeness.In Ball's words, do the 'reasons offered in support of the value goals of a policy argument appertain to those goals'?The third criterion is the matter of consonance.'To what degree do the claimed values contradict each other?'The idea is that while complete consonance is difficult to attain, inconsistencies in the set of values would undermine the argument's power.Insights from the world of decision-support systems can also help to address this weighting problem. 63In principle (multi-actor) multi-criteria analysis (MAMCA) and decision maps can make this process transparent, in particular now that digital tools are available to engage professionals in such a process simultaneously. 64mits also addressed the relationship between empirical legal research and normativity.He describes the contribution of several subfields of legal research to the analysis of conflicting arguments, including values.Although he is of the opinion that ELR always implies normative judgments (on, for example, the needs of people or the values that are at stake as soon as one evaluates rules in terms of their 'success'), 'what ELR can do, is [to] show how effective or ineffective it is to use law as an instrument to achieve a set policy goal.This can inform the debate about which alternative may be the better one to adopt, without giving any conclusive evidence.This is indeed the way in which empirical legal research is often used: it measures the effectiveness of different (possible) solutions.' 65

Unravelling and unpacking arguments and speech acts
This approach is strongly related to the family of 'theory-driven evaluations'. 66It starts with unravelling arguments that are said to be normative (or ideological) into parts that can be empirically tested and parts that cannot (as they are sui generis normative 67 ).Let us take, as an example, voting by ethnic minorities in a Western industrialized country.Suppose that the voting rate of ethnic minorities in this country is considered 'too low' compared to the voting rate of the rest of the population.It is seen as desirable ('good') to have it increased by at least 30%.Suppose also that a law is implemented to help realize this goal.The law specifies two actions.One is to have all documents on voting and the programmes of political parties translated into every language spoken by ethnic minorities in the country.The second action is to have three times more ballot boxes than are currently operating in geographic areas populated by an x percentage of ethnic minorities.
The research problem is to investigate if this law (and its two interventions) leads to an increase of at least 30% in voting by ethnic minorities and what can be done if this goal is not realized.The first question is whether this law can be empirically tested.That is doubtful, as one of the law's underlying central assumptions (it is good, desirable or commendable to stimulate voting by ethnic minorities with at least 30%) is normative in nature.What is good for believers can be 'bad' for non-believers.Framed in this way, it may be concluded that an empirical evaluation of the law is not possible, given its normative character.However, by using Searle's theory on speech acts, 68 recently discussed by Hage, 69 a different conclusion can be reached.
Searle distinguished between several speech acts. 70The first are assertives: they commit the speaker (i.e. the policy maker or lawyer) to something being the case.For instance, the statement that less than x% of the eligible ethnic minorities in country X vote.As Searle puts it, assertives have the word-to-world direction of fit; they are successful if they are true.That means that they can be put to an empirical test, which applies to our case.
Directives is a second category of speech acts: attempts by someone to get the other to do something.This is a statement that 'I, as a policy maker or legislator, will do something about the underrepresentation in the voting of minorities' .Directives have the world-to-word direction of fit, and are successful if they are effective.Again, the success of the law can be measured in an empirical sense.
Commissives (the third category) commit a person to some future course of action.They have, according to Searle, also the world-to-word direction of fit.For instance, the sentence 'I promise to realize that I will help minorities in making voting easier, so that underrepresentation will disappear' is such a commissive (presented by a policy maker).And again, this type of statement can be empirically researched as it stays in the domain of facts and not of oughts.
What the evaluation also can look into are the two practical interventions and their consequences.First, by reconstructing (and testing) the underlying intervention theory: why is it believed that adding ballot boxes and translating voting documents will contribute to a higher voting rate by ethnic minorities (up to 30%)?This can be done by searching for empirical evidence on these and lookalike approaches in research repositories in the field of law and politics and by collecting new data through experimental or other research designs measuring the impact of these interventions.
Contrary to the original answer (this law cannot be evaluated because it is inherently normative in nature), now there is another situation: by unravelling the assumptions and speech acts, it appears that almost every item of this 'normative' law can be empirically tested.The only issue that cannot be handled in this way is the normative adjective that it is 'good' to stimulate voting, as the number of voting minorities is believed to be 'too low'.That belongs to the core of the (sui generis) normative part of the law.71Decision-makers therefore still have to weigh empirical findings and this normative statement.This Bewertungsproblem has not been solved, but strongly trimmed.

Deliberative democratic evaluation
Evaluators like House & Howe 72 and Greene 73 have suggested a 'deliberative democratic evaluation' approach.The rationale behind this approach is 'a rejection of the fact-value dichotomy and thus the possibility of a value-free evaluative science.Instead, we contend that evaluation incorporates value judgments (even if implicitly) both in the methodological and in the concepts employed, concepts such as "intelligence" or "community" or "disadvantaged"' . 74House & Howe 'reject both extreme relativism (radical constructivism) and post modernism as viable frameworks for a value-engaged evaluation practice, and instead emphasize the importance of legitimizing values as intrinsic to evaluative knowledge claims, but also subjecting them to reasoned deliberation, using appropriate rules of evidence, argument and negotiation.' Part of the model is the answer to the question what values should an evaluation promote?House has argued for two fundamental democratic values, namely social justice and equality. 75he democratic evaluation model gives procedures by which stakeholders' interests are articulated, shared, and advanced in evaluation, even when, or perhaps especially when, they conflict.However, they are broader than Giesen's due process approach.These procedures rest on three principles: inclusion, dialogue, and deliberation.
'Inclusion means that the interests of all legitimate stakeholders are included in the evaluation.Dialogue (among stakeholders) is offered as the process through which the real or authentic interests, as compared to the perceived interests, of diverse stakeholders are identified.And deliberation is the rational, cognitive process by which varying, even conflicting stakeholder claims are negotiated.These may be claims of values, interpretations of evaluation results, or action implications.Deliberation means that all such claims are subject to reasoned discussion, with evidence and argument.In deliberative democratic evaluation thus, the evaluator's role is crucial and challenging, as he/she is charged with ensuring these principles of inclusion, dialogue, and deliberation through skilful facilitation and diplomatic leadership.' 76 To some extent 'virtue ethics' 77 and this approach are related, as the two democratic values which House takes as points of departure can be seen as 'societal virtues' .A virtue is a trait so that, whatever else is true of those among whom we live, it is better if they have it. 78Virtue ethics focuses on evaluating agents in terms of values like 'rightness' or 'goodness' , or something being inherently admirable or deplorable or noble or ignoble.It emphasizes an individual's character as the key element of ethical thinking, rather than rules about the acts themselves or their consequences.Applied to evaluation, this approach would not focus on individuals only, but also on societies.While House & Howe restrict their approach to only two values (social justice and equality), others may be added.
Following this approach, the gap is believed to be 'solved' , because it is believed that there is no such thing as a dichotomy between facts and values.This point has been strenuously criticized.One point made is that the promotion of democracy is not the main purpose of evaluations; another is the concern about the imposition of the evaluator's own values in the process of weighting arguments.Why are 'social justice' and 'equality' the basic values for society and for evaluators?And what about the tradeoff from realizing these values for other values that people and society deem to be important?And there is practical criticism that the approach is 'idealistic and difficult to implement in wholesale in today's democracies, with their special-interest politics and sound-bite media domination' . 79

Empirically informed ethics
In the last 25 years, ethicists have increasingly combined empirical (usually social science) research with normative-ethical analysis and reflection. 80Christen & Alfano 81 distinguished three ways as to how this is done.The first is to empirically describe the framing of a normative problem: what are the concepts and variables related to -for example -the discussion on whether or not it is legally and normatively acceptable to carry out research on stem cells and what are the societal discourses about this question?Finding empirical data as an indicator of the feasibility of ethical thought is a second potential involvement.One of the examples Christen & Alfano give is this.'Data emerging from patients with focal lesions in the prefrontal cortex that play a significant role in arguments for the significance of emotions as a "foundation" of moral intuitions and for practical decision making are remarkably imprecise with respect to what kind of emotions are affected.Such findings are also highly prone to misinterpretations driven by prejudices about what the data should demonstrate.' 82 Related to this second approach is to study how lawyers (and others) in practice handle ethical and normative issues.Perry et al. 's study on 'the Ethical Health Lawyer' is an example. 83The primary research question was how health lawyers respond when they encounter ethical or moral dilemmas in their practice for which the law fails to offer a brightline solution.The authors developed 'a survey instrument aimed at capturing empirical data about how health lawyers deliberate and act when they encounter an ethical or moral dilemma in their practice (…).We drafted hypothetical scenarios and questions designed to highlight the tension between what the law and rules of professional conduct might allow and what might more broadly be understood as the right or just course of action.Most questions (in these scenarios) used the term "ethical dilemmas" and several measurement scales used by the researchers applied the options "Definitely ethical" or "Definitely unethical".' 84 The third way of relating empirical research to ethical and normative topics is that data can be seen as foundations of normative theories, in particular when performing thought experiments.Such experiments are set up in such a way as to elicit assent to or even certitude in certain (normative) judgments. 85euroscientists are engaged in this work as they are interested in what is happening inside the brain when persons who have relevant empirical evidence available are confronted with moral dilemmas.The Trolley Problem (aka the Fat Man problem) is an example of such an approach.It originated with Foot's article entitled 'The Problem of Abortion and the Doctrine of Double Effect' . 86It consists of two scenarios.The first (called the Switch case) is this.A runaway trolley is about to kill five workers on the track.A bystander notices that he can throw a switch, thereby turning the trolley onto a spur where there is only one worker who would be killed (= the empirical evidence).What to do? (= the normative /ethical problem).Neuroscientists and cognitive scientists have designed experiments that look into the workings of 'moral grammars' that people use to guide their normative opinions and behaviour. 87The idea is that encountering such a conflict evokes both a strong emotional response as well as a reasoned cognitive response that tend to oppose one another.In neuro-scientific studies Greene et al. 88 asked research subjects to contemplate both the Switch case and the Footbridge case.'In Footbridge, you and a man are standing on a footbridge over the tracks.His body is large enough to stop the trolley if you push him onto the tracks.He will die but the five others will be spared.Many people believe that it would be morally wrong for you to push the big man onto the tracks.' The core of this approach is to address the Bewertungsproblem by studying what is happening inside the brain, when people have to deal with normative/ethical problems while having evidence on the case.Instead of producing normative statements on what people should do to leap from empirics to values (when confronted with the trolley or lookalike problems 89 ), cognitive and neuroscientists open the black box of decision-making mechanisms that guide a person's perspectives and behaviour.Knowing which mechanisms are 'at stake' and how they work when linking (or delinking) IST and SOLL can help in finding the pathways that judges, prosecutors, regulators and legal scholars are following when they are confronted with the option to cross the bridge between facts, values (and emotions) or refrain from that behaviour.

The gap problem and legal education
Empirical legal research is a growth industry.Papers and handbooks are published on methods, theories and data collection and analysis, including legal evaluations.The (brand new) 'Law as Big Data' movement 90 will have a serious impact on legal scholarship and practice.These developments confront legal scholarship and legal education with important challenges.They include the relationship between empirical findings and the normativity of the law.Part of this relationship is the gap problem: can facts and values be combined and integrated and how does one leap from evidence to norm(ativity)?Giesen 91 not only discussed several approaches to understand and deal with this problem, but also developed an approach himself.Although his proposal is interesting and worthwhile, in the evaluation literature, philosophy and the field of (empirically informed) ethics several other approaches have been developed that contribute to a better understanding (and handling) of this problem, both for (legal) research and (legal) education.When ELR is blossoming and the 'Law as Big data' movement may revolutionize the world of law, legal education cannot do without addressing the gap problem.
How relevant are the approaches we outlined?The democratic deliberative evaluation approach challenges the very dichotomy between facts and values and follows, to some extent, a virtue ethics approach.Certain values are interpreted as given and foundational, and are used as reference points (or 'benchmarks') in evaluations of policies, laws and programmes.Two of the most serious difficulties of this approach is why certain values are 'chosen' and not others and how the 'deliberations' can be organized in practice.
The second approach applies insights from theory-driven evaluations (opening up the black boxes of policies and legal arrangements) and speech theory to the gap problem.It was shown that this approach does not 'solve' the problem, but trims it down.The third approach takes the law (and legal studies) as an argumentative discipline and specifies criteria that can be used during the weighting process of empirical evidence and normative statements (including multi-actor multi-criteria analysis).Finally, we presented insights from a relatively new field, empirically informed ethics.One such insight is that it is relevant to know which processes take place, when one is confronted with the fact/value problematique.By understanding which brain and cognition-oriented mechanisms 92 are active, when one is confronted with moral and social dilemmas (like the trolley problem), one can search for pathways of how to handle the problematique.
What are the topics for academic legal training that result from this?
The first is to make students knowledgeable about the existence of a gap problem and its backgrounds.One of these backgrounds is the empirical legal researcher who produces evidence that contradicts the assumed 'workability' of certain legal arrangements, practices, positions or expectations.It basically functions as a 'reality check' 93 , but may not always be a welcome guest.A second background is that, due to limited experience with data (collection and analysis), a (normative) legal scholar's interpretation of some (segments of the) evidence may be methodologically incorrect.A third possibility is that the fragmented nature of empirical legal research makes it difficult to find robust and relevant evidence that is capable of passing the 'due diligence' 'test' .Presuppositions about the type of empirical evidence (qualitative, quantitative) that 'should' be preferred as evidence in the legal world is another background of the gap problem (with an example from the field of neurolaw regarding the impact on judges and juries attached to brain imaging techniques and their role in forensic investigations versus the more restricted impact that 'traditional' (narrative) evidence seems to have).
The second topic for academic legal training is to teach participants how to unravel the gap problem.As we showed earlier, what looks like a complex (normative) problem may be reduced to a much smaller problem, after unpacking its assumptions and types of statements ('speech acts').
Understanding that very probably the solution to the gap problem does not exist, can be the third and final aspect of this segment of the legal curriculum.¶

Figure 2
Figure 2 Roots of empirical legal research 20 They challenge received wisdoms.One of the conclusions Vaessen et al. present is that 48 Giesen 2015, supra note 26, pp.16-17 (footnotes omitted, emphasis added).49 R. Stewart et al., What is the impact of microfinance on poor people?A systematic review of evidence from Sub-Saharan Africa, EPPI- Centre, London, 2010; R. Stewart et al., Do micro-credit, micro-savings and micro-leasing serve as effective financial inclusion interventions enabling poor people, and especially women, to engage in meaningful economic opportunities in low-and middle-income countries -A systematic review of the evidence, EPPI-Centre, London, 2010; M. Duvendack et al., What is the evidence of the impact of microfinance on the well-being of poor people?, EPPI-Centre, London, 2011; J. Vaessen et al., 'The Effects of Microcredit on Women's Control over Household Spending in Developing Countries: A Systematic Review and Meta-analysis', Campbell Systematic Reviews 2014:8.50 D. Gough et al., An Introduction to Systematic Reviews, 2012.