This essay is based on my September 3 talk, and proceedings contribution, for the 2011 ISFG conference in Vienna Austria.

Keywords: forensic mathematics; forensic statistics; DNA mixture; rare haplotype; refereeing; misconduct

- Introduction DNA identification is interesting in offering, more than any other forensic area, scope for explicit mathematical treatment. Many specific problems have received mathematical attention (variously analysis, discussion, and debate) in the literature, beginning with straightforward linking suspect to unknown DNA profile, adding difficulties and complications such as rare haplotypes, database search or mixed sample, and identification via kinship with its many attendant intricacies. Mathematics as a tool has several potential attractions and advantages. Mathematical writing is explicit in definitions and assumptions, hence should provide clear unambiguous communication. The reader should know what is being claimed. Mathematical exposition is logical and deductive. Ideally the reader is led irresistibly along a linear deductive path. If not irresistibly, at least the exact point of resistance is manifest. Then the reader can say “I disagree with your premise” or can argue that step D doesn’t follow from step C; a productive discussion is then possible with a good chance for resolution of disagreement.
- On validation I believe that the main and possibly the only criterion for validation is that the method is fair to innocent suspects. The test is the
- Good and bad
- Mathematical paradigm
- Offensive style
- Mathematical deficiency about the “exclusion” method
- Unprofessional rare haplotype literature
- SWGDAM on rare haplotypes SWGDAM instructions on rare haplotype matching [S-Yhap] don’t even state a problem but instead begin by proposing how to calculate a haplotype frequency. The reader who realizes that (population) frequency is not (matching) probability and that the evidential problem concerns probability, is left slack-jawed at the post with nothing with which to disagree while the paper gallops off into (irrelevant) mentions of formulas and ideas lacking not only foundation but lacking any chain of reasoning.
- BKW wrongly denies validity Much of the recent [BKW] is a sophisticated discussion of my [FP], resting largely on unsupported opinion (“we believe”, “it is difficult to perceive”), factual errors, incompetent mathematics and logic, and apparently scientific misbehaviour – a performance impossible for any of the able authors individually so I attribute it to the multiple author effect. I’ll review here only its discussion of validity. It concludes with the claim that “we have shown Brenner’s approach ... suffers from potential anti-conservativeness.” Mystified, I consulted the authors and learned that “shown” refers to a section where the paper tries a little math, a hopeful analytical approach for assessing my method, but the analysis runs aground – dead end. I sympathize. My own notebooks are littered with ideas that didn’t pan out. But I don’t try to publish them and I conclude nothing from them except my own limitations. Not so [BKW]. If their analysis goes nowhere it must be a shortcoming of the thing they are trying to analyze! In any case, it was an odd effort. They should instead have read the part of my paper in which I claim to establish validity, and if my argument is convincing it follows that any attempt to debunk validity is bound to fail.
- BKW wrongly asserts validity
- Discussion Important properties and components for a methodology paper in forensic mathematics include
- Statement of the problem
- State premises In most cases the most important component of the premises will be the model. Writers who skip this step — i.e. most writers — deserve to be maligned because I believe they do so for a combination of bad reasons, of which the most innocuous is laziness — They vaguely imagine that the premises are known to all or are clear from their mathematics. But that's just asking the reader to do their work for them.
- Derive consequences and results Emphasis here is on the idea of
- Validate The premises and model are adequate if the evidential value implied by them is in general fair to an innocent suspect.
- References [FP] Brenner CH,

The paradigm should be: State the problem, formulate it mathematically, state premises (inevitably including a model since this is applied mathematics), justify the premises (i.e. validate the model), derive the result.

I tried to follow the mathematical tradition in my recent paper on rare haplotype evidence and
(as I modestly define) the fundamental problem of forensic mathematics [FP]. The main problem is to
find the evidentiary value of a previously unseen haplotype linking suspect to crime scene.
Expressed mathematically it comes down to a conditional probability that a random innocent person
will match. For the evidential likelihood ratio (the reciprocal of that probability) I derive the
expression LR=*n*/(1-κ) where *n*=size of reference database after extending it with the crime
scene type, and κ=proportion of that database that is singletons, which derivation is valid
under a stated modelling condition. Then I show that the modelling condition holds, hence the
formula is valid, for a wide range of theoretical populations which encompasses the plausible range
of real populations.

No doubt my paper isn’t a perfect example of the paradigm, nor is it a unique example. But disappointingly, contrary examples abound.

A paper which simply gives a recipe for calculation without any stated justification (let alone careful justification) is professionally deficient. We find mixture [B-mix], [S-mix] and rare haplotype papers [S-Yhap] of this sort. Also, there is a prevalent style of sophisticated writing that is entertaining but literally pointless: e.g. “an attractive point of view” (Is it attractive to the innocent suspect who is the victim of the entertainment?), or “An alternative framework is to suppose ... a prior distribution that is conveniently taken to be Beta” [BKW]. Is convenience of statisticians a valid criterion for deciding who goes to jail?

In the ‘90s the exclusion method for mixtures was simple: If a suspect is “included” then report RMNE, calculated per-locus as the squared sum of the allele frequencies for alleles observed above 100rfu or so. No one actually wrote down the model but the formula is simple enough that it can be reverse-engineered to deduce what the model must be: The formula assumes that all alleles of a donor will be conspicuous (e.g. >100rfu) and “included” means all of ones alleles are conspicuous in the mixture. Obviously this is an absurd model. That it survived and was popular and accepted for years – perhaps still – proves the importance of explicitly writing down models and explicitly deriving and justifying the consequences. With nothing written down, nothing wrong is written down and errors are less obvious.

The recent appearance of a refined “exclusion” method [B-mix] [S-mix] suggests that RMNE enthusiasts have woken up to the folly and unfairness of the original approach. They have not, though, woken up to the importance of models, let alone to justifying the shibboleths of the RMNE faith such as “easy to understand!”, “needs no assumption about number of contributors!”, “conservative!” These papers give no mathematical analysis at all; only recipes which apparently we are supposed to trust.

And my point isn’t that the method fails. My point is that the adherents of a method have a positive responsibility to show why it works. They have not done a respectably professional scientific job if they don’t explain coherently. Otherwise the rest of us – reader, analyst in the laboratory, judge, and accused – ought to be suspicious of the validity of the method. And I am.

Finally and equally mysterious [BKW] falsely asserts that a simultaneous
publication, [Ysurvey], validates an alternative approach, “frequency
surveying.” There is nothing about validation in [Ysurvey]. An author of [Ysurvey] says that it includes no validation (M Anderson, *pers comm*).
The method seems [FP] *prima facie*
implausible. An author of [BKW] recalls that the assertion was added at the
behest of the referee (B Weir, *pers comm*).
If so the referee overlooked confused reasoning, an algebraic blunder and
numerous factual errors in [BKW] but managed to join as anonymous co-author with a
mistaken claim that could be important in a judicial setting. A judge might be falsely reassured
with unjust effect. Or the defense, armed with the true and unflattering story, might argue
convincingly that a breakdown in the peer-review process amounts to a
shoddy lack of professionalism
about statistical methods in forensic genetics as a whole. That’s harsh reasoning from one incident,
but is the conclusion wrong?

Explain the problem first in words. This important step is difficult, but important in order to give reader and writer alike something to latch onto.

It requires careful thought to find the right words to say exactly what you are trying to do. If those words are missing, that doesn't indicate mere carelessness. Rather take it as a sign of vague and fuzzy thinking, and a warning of more to come.

If the words are carefully chosen it will be straightforward to give mathematical expression to the evidential value that is the goal.

I can imagine a reasonable forensic mathematics paper that gives a good argument for some method without giving a model, just based on statistical analysis of some experiments for example. But it would always be much better with a model. The model provides insight into why the suggested idea works, and it guides understanding into the range of circumstances under which the idea is valid.

By the way, the concern in particular for the *innocent* suspect isn't simply because
we care more about them. It is because fairness to the innocent is almost certain, for
a subtle but pervasive philosophical reason, to be more difficult than fair evaluation
of evidence when the suspect is really guilty. A model is necessarily an idealization,
an approximation to reality that omits some details. Omitting details means omitting
evidence. For example, in the case of DNA mixture evidence, a simple model doesn't
worry about signal intensity. When the suspect is really the donor to a mixture,
the intensity information will in practice nearly always be further evidence so ignoring
it is doing him a favor, being "conservative." But for an innocent suspect the tendency
is exactly the opposite: the more we look at details the more we are likely to see things
revealing the truth. That may be why it is very hard to find a simple method — a
method based on a very much simplified model — that is fair to the innocent.

The above points seem pretty obvious; it is hard to imagine anyone arguing that it is ok to omit stating the problem. To see the vital importance of stating premises and giving a model, look at any paper which does not do so and see immediately how chaotic and difficult to evaluate it is. Even for a quite simple paper I find that as I lay out the mathematics explicitly, I am forced to refine my results and conclusions. So I am convinced that slapdash papers — i.e. most papers — are riddled with unsoundness. Those criticized in this essay and many (most?) others are nothing more than guesswork, and the more closely the guesses are examined the more it is apparent that shoddy reasoning leads not just to poorly supported conclusions, but to poor conclusions.

Fundamental problem of forensic mathematics – The evidential value of a rare haplotype, Forensic Sci. Int. Genet. 4 281-291

[B-mix] Budowle B, Onorato AJ, Callaghan TF, Della Manna A, et al,

Mixture interpretation: defining
the relevant features for guidelines for the assessment of mixed DNA profiles in forensic casework,
J Forensic Sci 2009; 54(4): 810–21

[S-mix] SWGDAM Interpretation Guidelines for Autosomal STR Typing by Forensic DNA Testing
Laboratories §3.5. Interpretation of DNA Typing Results for Mixed Samples *at*

http://www.fbi.gov/about-us/lab/codis/swgdam-interpretation-guidelines

[S-Yhap] SWGDAM Y-chromosome Short Tandem Repeat (Y-STR) Interpretation Guidelines. (§5.3. The basis
for the haplotype frequency estimation is the counting method. *at*

http://www.fbi.gov/about-us/lab/forensic-science-communications/fsc/oct2009/standards/2009_01_standards01.htm/

[BKW] Buckleton JS, Krawczak M, Weir BS,

The interpretation of lineage markers in forensic DNA testing, FSI Genetics 5 (2011) 78-83

[Ysurvey] Willuweit S, Caliebe A, Anderson MM, et al,

Y-STR frequency surveying method: a critical reappraisal, FSI Genetics 5 (2011) 84-90