Questions about the Design Matrix
by MikeGeneThe recent thread about the "Man on Mars" has raised some good questions about The Design Matrix. First, hrun asks:
I am not trying to strike a round-house blow at design detection or the design matrix here. I am asking a genuine question: How do we know that we are operating at the right resolution to detect design? How can we avoid false positives or false negatives due to being at the wrong resolution? Is a single positive result at any resolution sufficient to conclude design?
I think these are excellent and only by asking them, pondering them, and attempting to answer them shall we make progress in trying to infer design in a fair and open-ended manner. So let me address each one.
How do we know that we are operating at the right resolution to detect design?
We can't. I think we are dealing with a topic that is deeply ambiguous and we don't have the luxury and convenience of experience with known non-human, intelligent design. Nevertheless, we have to begin somewhere and various people, from both sides of the aisle, will use different starting points when trying to pass judgment on the origin of some system. In comes the Matrix"¦"¦.
How can we avoid false positives or false negatives due to being at the wrong resolution?
I don't think we can ever truly "avoid" such risks, but the Matrix does make significant progress in this area. This is because the Matrix, as a function of its design, allows us to sample from different resolutions.
The Matrix draws upon four different lens: Analogy, Discontinuity, Rationality, and Foresight. I use chapters 8 and 9 (ca. 80 pages) to explain how each perspective can be employed independently to bring about a higher resolution analysis along that dimension through greater focus. But the four focused analyses are then expanded into a wider survey, as all four scores are considered together, as an average, and can be visually represented. This allows us to step back ("lower resolution") and consider the larger picture. In fact, it's this act of "stepping back" that helps control against an over-eager conclusion built only on one lens that is highly magnified: "The Design Matrix thus buffers against hasty conclusions in either directions. To eliminate design, you need something more than evidence of evolution. To conclude design, you need something more than evidence against evolution." (p. 275).
Is the Design Matrix perfect? Of course not. I offer it only as a next step and an improvement. With use, the Matrix itself can be better refined to factor for different resolutions.
Is a single positive result at any resolution sufficient to conclude design?
I don't think so. The DM is not about establishing any conclusion with a single score (either from one person or a group). Here are three excerpts from the book that better explain the objectives/uses of the Matrix:
It is important to again stress that the Design Matrix is not an objective, physical measurement that detects design. The Matrix is a scoring system and, as such, is ultimately subjective. For example, when I give the genetic code a Design Matrix score of 3, this does not mean that all people everywhere are obligated to concur. Nor does it mean that the score cannot change as a consequence of new information or further consideration. Nevertheless, the Matrix focuses our thinking process and helps to clarify why people would and would not infer design in any particular instance. And while in the end the scoring is subjective, it is not whimsical. The Matrix allows us to examine a biological feature from four different perspectives and points of emphasis. In the spirit of the Explanatory Continuum (Chapter 2), it searches for the subtleties crucial to any investigation and helps to both gauge and illustrate our convictions. For those open-minded about various forms of design amid life, it allows such investigators the opportunity to compare notes and perhaps arrive at some kind of consensus.
the Design Matrix can not only bring focus for research, but is also receptive to the findings of research.
Lastly, the Design Matrix score is not intended to be a static figure. It merely represents a score that is dependent on the scorekeeper's knowledge at the time. As such, a Design Matrix score can change, in either direction, in light of new information pertaining to any of the criteria. What would thus matter most is not a Design Matrix score at any point in time, but the persistence or trending of a score as new information is acquired.
This all nicely leads to the questions bipod raised:
When do we know that we have enough epistemic resolution? How deep must we go, how many levels of resolution must we traverse, before we are justified?
Let me try to get to that one next. In the meantime, here is an old essay that is relevant.
January 26th, 2008 at 12:45 pm
Hi, Mike. You wrote (in The Design Matrix):
This sounds analogous to the method used to diagnose a particular mental illness. There are no laboratory tests which can detect schizophrenia or bipolar disorder, for example. Rather there are sets of descriptors which serve as diagnostic criteria. Whether or not a given descriptor applies to an individual is ultimately a subjective decision which may vary from one rater to another.
However, a measure of inter-rater reliability can be obtained, which identifies the variability between raters for a given descriptor or set of descriptors. By refining the descriptors in order to achieve a high inter-rater reliability, we can approach something akin to objectivity…although, of course, we can never get there. Our measurements will never be as precise as those who are detecting physical illness…but even there, imprecision and errors go with the territory.
Comment by Lutepisc — January 26, 2008 @ 12:45 pm
January 26th, 2008 at 1:23 pm
Hi Lutepisc,
Thanks for sharing that. The main criticism of the Matrix will be that it is "too subjective," yet I view this as a knee-jerk reaction. When I clearly write in the book, "The Matrix is a scoring system and, as such, is ultimately subjective," you would think it would be obvious that I am fully aware of this problem and thus perhaps, just perhaps, the Matrix is not somehow discredited simply because it is "subjective."
There are very few things we can be truly objective about. You provide an excellent example to illustrate this. Do we have an epistemic ability to refrain from addressing mental illness and making diagnoses because they are ultimately "subjective?" Do we say that unless someone can come up with a purely objective measure that detects mental illness, we should all shut up about it and go about life as if it did not exist? After all, making a faulty diagnosis can be quiet damaging, as even history shows.
No, we do the best that we can, such that while our diagnoses are ultimately subjective, they are not whimsical. We work with and through our subjectivity, and with practice, we try to make the diagnoses less and less subjective.
Comment by MikeGene — January 26, 2008 @ 1:23 pm
January 26th, 2008 at 7:06 pm
Well, I have not read the "Design Matrix" yet, but I have thought about the issue of detection. I think probability and statistics can go a long way to help. That is, we can calculate the probability of seeing what we see under the null hypothesis: that there is no design. In this situation the context is fixed, and this fixture permits such a calculation to go forward. Now it may happen that the context can be adjusted, but the adjustment will again permit a calculation. However, calculating the power of a statistical test is more problematic, as the sample space is always context dependent. One must put forward a sample space to permit any calculation. Nevertheless, the continual adjustment to the sample space is permitted to protect the null hypothesis, within limits. When those limits become unreasonable, e.g., when we require a collective of infinite universe to protect our accidental universe, then we reject the null hypothesis.
Comment by Stephen — January 26, 2008 @ 7:06 pm
January 26th, 2008 at 7:29 pm
Stephen, the problem with your approach of calculating probabilities is that it's pretty much impossible to do so. Consider this example:
You find a piece of paper on which you see the sentence 'Methinks it is like a weasel' written in black ink.
How would you go about calculating the probabilities of this scenario? And I'm not asking for numbers here, just an idea about how to approach this problem.
Comment by hrun — January 26, 2008 @ 7:29 pm
January 26th, 2008 at 9:23 pm
I guess what I am saying is that probability is context dependent. This is something we should be able to agree on. The rest is just frosting on the cake.
Assuming we want to look deeper, first comes quantitative analysis: If someone says something is random (or random with laws) then that someone carries the burden to demonstrate a sample space. If the someone is unable to provide a sample space then the default is that quantitative analysis fails, just like hrun will tell us. But once a sample space is specified then it can be tested, but in no way should we assume a sample space in advance the way Richard Dawkins does in the "Blind Watchmaker". If any thing, we might approach a sample space given a series of refinements, but if each improved sample space eventually fails then that effort demonstrates something that is not random (or with laws). We are forced to turn to a qualitative analysis, assuming we don't start at one.
In a qualitative analysis we might attempt to revive the quantitative analysis as a conservative approximation. In this case we only put limits on the sample space, so as to bound the probability of an event below some number that can be calculated. If the calculated number is infinitesimal we reject the randomness hypothesis and welcome revisions to the restrictions put on the sample space. This notes that the advocates of randomness carry the burden to provide information on the sample space; we then continue as we did in the quantitative analysis. I believe we can study the following with this approach: anthropic coincidence, synchronicity, the odds of intelligent life else where in the universe. In any event, these approaches ought to be open to dialectical refinement given that context must be specified.
I suspect that real-world context cannot be specified as an absolute fixture, meaning that we live in a universe where freewill is possible; it is hard wired into the very fabric of space and time. Therefore, my hypothesis predicts failure with the above attempts as "design" is recognized given an innate synthesis that holds content to its context. In other words, I predict we can contrive something, like saying 'Methinks it is like a weasel,' in a way that contradicts law and chance.
Comment by Stephen — January 26, 2008 @ 9:23 pm
January 27th, 2008 at 12:26 am
Let me take a stab at bipod's questions:
My answer? I don't know. But I do know one thing "“ this is not a problem that is restricted to design inferences (as argued in my link from above).
I liken such questions to another one "“ how can I know if I am a good father? I don't think that I ever can. All I can do is to wake up everyday and do the best that I can.
When it comes to a sense of justification in belief, I think (at least) three variables come into play. First, how ambiguous is the issue in question? Second, what is the degree of conviction that comes with the belief in question? Third, how much do I expect others to agree with me?
Comment by MikeGene — January 27, 2008 @ 12:26 am
January 27th, 2008 at 8:25 pm
Just a footnote: I don't want to minimize the importance of a fuller qualitative analysis (beyond the simple revival I hint at above), because we are forced into it once we understand that the "front-loaded" contextual features that support the quantitative analysis are all too contrived for the deeper questions. In other words, real quantitative analysis works in the negative, to defeat its foundation on our search for a deeper truth. Quantitative analysis only works when the context can be contrived, and this is good enough for its purpose (the self discovery of its own negation).
The real reason for my footnote is this: I have ordered "The Design Matix", and I hope to have a book review posted on Amazon once I have fishised reading it. So be looking for it in a few weeks.
Comment by Stephen — January 27, 2008 @ 8:25 pm
January 28th, 2008 at 1:00 am
Hi Stephen,
Thanks! I'll be looking forward to that review.
Comment by MikeGene — January 28, 2008 @ 1:00 am
January 29th, 2008 at 1:17 pm
Hrun spoke of the Nazca Lines and how we determined that they were designed. I was curious about how the Matrix would score them. Here's my stab at it:
Analogy: not extremely detailed, but recognizable as monkeys, birds etc.
+4
Discontinuity: Definitely stand out from the normal desert landscape.
+5
Rationality: No known function, not even viewable before the advent of flight, negative geoglyph design not very robust.
-5
Foresight: No clues as to whether the lines were intended for posterity, except that their design exploits the non-erosive climate of the area.
-4
Total = 0
So are we jumping to conclusions in saying that the lines were designed?
Comment by chunkdz — January 29, 2008 @ 1:17 pm