I went to Cornell and I'm one of the many students that participated in this guy's experiments (although not this particular one with the erotic pictures. I got regular pictures.)
I can tell you that every semester that I was there he was running a version of the "Are you psychic?" experiment. I'm sure he's been doing it every semester for a very long time. Undoubtedly there have been loads of experiments where it didn't pan out. (If you're curious about my results, I got 54% and a cheerful grad student greeted me after the fact by saying "congrats! you're psychic!")
The fact is, if you run an experiment like this enough times you are going to get a significant result eventually. That's why you have alpha values. If it's at .05, that means that 5% of the time you're going to get a false positive. I think that's what this is.
The only way this kind of experiment can have meaning is if 1) somebody can hit a very high percentage correct a high percentage of the time or 2) over all, there is a trend that is unexpected.
I would be very surprised if said grad student actually believed you were psychic. A result in the significant region is NOT significant. A set of results in the significant region is.
I'm sure he gives the size in the paper somewhere. I expect it was probably similar to what I did. It was 5 years ago and the pictures weren't numbered so I really have no idea.
The paper says it's 36 trials. 30% of experiments with 36 dichotomous decisions where the answers are chosen randomly would be expected to have 20/36 correct (= 56% correct) or better.
I consider myself a true skeptic, in the sense outlined in that article; no, I don't really currently believe in systematic psi effects, but yes, there is a threshold of evidence that will convince me.
That said, to engage in a bit of meta-skepticism, I see an article like this about every three years or so, and so far nothing's come of any of them that I am aware of. And by "like this", I mean the whole article, appeals to skeptics to be "true" skeptics and not just unthinking unbelievers (which is a real thing, certainly), acknowledgments of the poor history of the field but assertions that it's different this time, trotting out various people with various credentials who assert there's something there... I'll wait for the consistent replication.
Would you be "skeptical" if a meta-analysis of physics paper showed a statistically significant outcome of a particular unexpected particle being created in different high-energy atomic collisions? Or would you look for where the theories are wrong that didn't predict the outcome?
The only answer to your question I can imagine would really be "it depends". I'd probably wait for more data to come in on the grounds that I don't have any compelling need to make a decision anyhow, so whatever you're trying to get at this is probably not a great question to do it with. I am interested in particle physics but only as a passive observer.
(Further edit: Oh, you're probably trying to say I shouldn't be skeptical in light of the meta-analysis. The correct thing to do here is to do some more experiments that continue to show the effect shown in the metaanalysis, there's too many ways a metaanalysis can screw up. Particle physics is a really different world than psi research. If it remains, great. I'm actually not too invested in the "time flows in one direction" theory, because in point of fact relativity itself disproves it even before you get to quantum mechanics; it is at best partially ordered and in fact some things happen in quite counterintuitive ways in straight-up relativity even though they ultimately resolve into something that matches our intuitive understanding of time. In ways that would exceed an HN posting to describe, though, I think that relativity ultimately forbids reverse-time information flow even more strongly than you'd think as a result, not less strongly. I'd suggest googling up "Reflections on Relativity" and be prepared to spend some serious hours with it; it is not light reading.)
As this article points out, the problem with these studies is not statistics. The issue is non-reproducibility, and the ever-present possibility of deception (fraud, incompetence, cheating participants, etc.) Studies "demonstrating" ESP are probably as old as science itself.
On the other hand, I've heard one eminent psychology professor at McGill tell us that "there's something there", and that he had personally tried a few seemingly conclusive experiments. Unfortunately, there have been so many ESP cranks before, that no modern-era psychologist would dare touch that field for fear that their career be forever sidetracked.
Also relevant, regarding the claims of homeopathy:
What about a memory of water? Is it possible? In 1988 the scientific journal Nature, had received an article written by celebrated French scientist Jacques Benveniste. He claimed to have found the evidence that made a homeopathy scientifically credible. Benveniste experimented with very dilute solutions. To his surprise, his research showed that even when the allergic substance was diluted down to homeopathic quantities, it could still trigger a reaction in the basophils. Nature's editor Sir John Maddox decided to publish a paper, but under one condition, to be allowed to Nature's team of investigators to inspect Beveniste's laboratory. When Maddox named his team, he took everyone by surprise. Included on the team was a man who was not a professional scientist: magician and paranormal investigator James Randi. Randi and the team watched Benveniste's team repeat the experiment. They went to extraordinary lengths to ensure that none of the scientists involved knew which samples were the homeopathic solutions, and which ones were the controls - even taping the sample codes to the ceiling for the duration of the experiment. This time, Benveniste's results were inconclusive, and the scientific community remained unconvinced by Benveniste's memory of water theory.
Call me skeptic but none of the experiments or the so called "significant laboratory evidence of psi" are conclusive at all. Premonition deals mostly with feeling something before it happens and precognition deals with seeing things before it happens. I believe the former is side effect of a mind that analyzes and understands patterns with fairly good accuracy, while the latter is nothing than pure coincidence. It's easy to say random things like "my dog is looking at a turtle" 100 times and get some percentage correct, but I bet you that if I asked you about it, you would only remember the correct ones and mostly discard all the times you where wrong.
Hell not only do it all the time, I did it three times in the last hour while I was taking a shower. I "saw" my dog pee on the studio, I "saw" my significant other arrive with chicken, and I "saw" a call from my sister. I leave the room and my dog peed on the studio, my girlfriend is eating some roasted chicken and my sister called me while I was reading this article. Does it mean I'm a psychic? HELL NO! I've done that same exact thing before, probably by analyzing patterns, and have been utterly wrong.
Believing in this stuff would mean that our futures are ruled by some kind of destiny, which is fine for Prince of Persia, but really makes absolutely no contribution to our collective sanity.
Wohooo I just imagined my girlfriend is going to come out of a giant cake wearing a spandex catsuit. I hope I hit this one too!
It's ridiculous that such an article could get published in the first place, though it seems the magazine is "new-age". They admit that all the evidence doesn't support the point they are arguing for:
"But the long history of parapsychology lab research, going back far before Bem to Rhine’s ESP work in the 1930s, shows that when you bring psi into the lab, it tends to become more of a systematic statistical biasing factor than a source of individual mind-blowing “miracle events.”" - in other words "Evidence of PSI is indistinguishable from stochastic noise".
"The chief bugaboo of scientific psi research has been replicability. It has proved frustratingly difficult to precisely replicate the results of many psi experiments." - In other words, the studies rely on chance events to get results in the first place.
"Perhaps the experiments ... will finally resolve this problem and provide robustly replicable experiments demonstrating psi phenomena. I certainly hope so." - And the author is biased. How is this worth discussing?
On an email list of local psychologists, one psychologist pointed out that the showings of different stimuli were almost certainly not random in the way that Bernoulli trials would have to be for the calculation of statistical significance to be correct. I'll try to paraphrase some of his words below:
We have to find out whether or not Bem was really using Bernoulli trials. Did he allow for cases comparable to having an experimental subject guess heads versus tails in a sequence of twelve coin tosses, when heads might come up twelve times in a row? Did he also allow stimuli to alternate back and forth, like a coin toss sequence of H T H T H T H T H T H T? Many times when experimenters try to make sure sequence LOOKS random, it actually makes a sequence more predictable by excluding some possibilities. This always makes the sequence more predictable.
If an experimenter shows subjects sequences that are meant to look random, but for every subject there are just as many "off" states as "on" states, in twelve trials, then there are fewer possible sequences. With Bernoulli trials, there would be 2^12 = 4096 possible sequences of 12 trials, but with "balanced" choices of stimuli there would be only 12-choose-6 = 942. Experimenter-designed sequences can also create dependencies so that knowing any part of the sequence helps to predict the rest of it. A subject who guessed that over twelve trials in a sequence there are equal numbers of trials in each of two states would be right 64.3 percent of the time, on average, not 50 percent.
The psychologist went on to note that such studies are very hard to design properly, but so far information isn't available on how the study was done to that level of detail. He thinks it is more likely than not that the experimenter gave his subjects clues to the structure of the sequences of trials he set up, and that he set up conditions for which performance above the 50 percent accuracy level is expected and not remarkable.
From the article :
"The sequencing of the pictures on these trials was randomly determined by a randomizing algorithm … and their left/right target positions were determined by an Araneus Alea I hardware-based random number generator."
At the very least they were using Araneus Alea wich is a hardware random number generator, so the numbers were not predictable. It's possible that the "randomizing algorithm" did something dumb and made the sequence not random, but I doubt it. I'm downloading the replication package to see if it has the source code. If it doesn't, that's a pretty big red flag for me.
I think that it's more likely thet the sudy was done so many times that it eventually gave significant results than it is that the sequence was not random. Or maybe prescience is real to some degree, or the study is a statistical glitch.
EDIT: the replication package doesn't have the source code for the progrma that randomizes the sequence.
Thank you for pointing to that statement from the article. I'm still pondering what scope that statement leaves for imbalance in the number of pictures of one kind or another during the trails with each experimental subject.
Is anyone else highly skeptical of the bar chart indicating that just > 50% of professors in 'natural science' disciplines believe in ESP and that social science (except psychology) and humanities professors are even worse?
FWIW the author only tested one of the nine experiments. The replication experiment was also done using volunteers over the Internet, meaning it wasn't a true replication. (Not Ivy league students, further away from the random number generator, etc.) Anyway, it'll be interesting to see if any of them can be replicated in the future.
I look at this study and think "If this effect was real, it would change the world as we know it. It would definitely win the Nobel prize, and once understood could almost certainly be harnessed to produce almost infinite riches."
This doesn't in and of itself affect the probability that it's a true effect, but why would you publish this study based on such a limited single trial if you truly believed you had discovered such a world changing thing? Wouldn't you at least run a second trial to confirm? Wouldn't you drop everything you were doing to try to firm it up so solidly that no rational person could doubt that you had just changed history?
This leaves me to conclude that even the original author does not believe it to be true. For if he believes he's made a discovery of this magnitude and is still standing up in front of Psych 101 courses giving the same lecture he's given for the last 20 years instead of pursuing this with every ounce of his being, he would be a fool. I do not believe he could be so large a fool, ergo, the effect is merely a statistical mirage.
Given the prior expectation of approximately zero, the posterior remains approximately zero. Fantastical claims require durable evidence that can be repeated by different people at different times in different places.
There was actually prior expectation. To quote the article:
"Bem was unable to find any fatal flaw in Honorton’s work. He became more and more interested in extending his research focus from personality and social psychology to psi research. In 1994, Bem and Honorton co-authored a landmark article on psi in the mainstream psychology journal Psychological Bulletin. The article described the results of a thorough statistical meta-analysis of eleven ganzfeld studies. (A meta-analysis involves combining data from a series of similar experiments conducted over a period of time, to come to an overall conclusion.) The result of the meta-analysis was striking: subjects obtained overall target “hit” rates of approximately 35 percent, far above the 25 percent that chance performance would predict."
Based on economics, and the fact that nobody is exploiting this for massive gains, I'd say that the prior expectation for this being anything but an error or a fraud is much lower than you seem to be implying.
Well how exactly would you exploit being able to see porn from the future 53% of the time? My understanding is that there is a ton of military research also indicating a weak psi effect, as seen in The Men Who Stare At Goats, but I don't think they've found any use for it. I'm not a true believer by any means, but I don't think it's especially unreasonable for there to be some sort of weak psi effect either. You should skim over a copy of Stan Grof's book When The Impossible Happens. Most of the chapters aren't at all convincing, but there are a few things that jump out as being kind of interesting. At the very least reading stuff like that helps one to develop a more noetic understanding of the history of thought.
From the perspective of information, the bits that you have decoded into pornography could be decoded differently into another coherent image. Therefore, the fact that the particular choice of decoder produced pornography should not much matter.
The military uses for this are obvious; the military abandoned it because it could not produce results.
And even if this only works for porn, that's fine. I could make a killing betting mathematicians and scientists that I could pick the porn > 50% of the time.
What do you propose to be the propagator of information? Photons? If so, what is the receptor? If not, then what? Also, how is this information propagated from the source? If propagated through time, how does this reconcile with the forward-propagation in time of everything else that has ever been observed?
"The military abandoned it because it could not produce results."
According to the movie, the military originally abandoned the research because of internal conflicts, and they have in fact since resumed research again. Not sure what the veracity of this is, but they stated it as being a true fact at the end of the film.
"What do you propose to be the propagator of information?"
Terence McKenna once said, "Not only is reality stranger than we suppose, it's stranger than we can suppose." Betting on current scientific paradigms as accurately portraying reality is a suckers bet every time. I have no idea where the information would come from, though presumably it would come from the same place that consciousness comes from.
The reason HN is mostly logical positivists is because it's dominated by engineers and people with engineer-like personalities, and that's what most engineers think science advocates. (Even though most scientists themselves don't believe this, and post modern philosophy makes pretty short work of it.)
The records show that this was transferred to the CIA in 1995 and discontinued. If they secretly continued it, they sure did a great job in the run-up to 9/11.
If these guys are open with their data, it will be rather straightforward to identify the error that they committed that led to these results. If not, a few other people will try to replicate their results and fail. At least some things in their field of study are always consistent.
Also, I'm a scientist, not an engineer.
Oh, also, let's be clear here. If what these fellows propose is true, then they will have simultaneously discovered both of the following:
1) Reverse causality
2) Evidence for biological receptors for the mediator of reverse causality
It's basically hilarious, except for the fact that the money that funds this could be spent more productively on literally anything else. In general, parapsychology consists of the practice of accidentally or willfully misunderstanding the scientific method.
"If what these fellows propose is true, then they will have simultaneously discovered 1) Reverse causality 2) Evidence for biological receptors for the mediator of reverse causality"
If either is true, then history suggests that we'll later look back and realize that we already had a ton of evidence for both of these hypotheses, but that it was all being used instead as evidence that the old paradigm was correct.
"If these guys are open with their data, it will be rather straightforward to identify the error that they committed that led to these results."
Maybe. With a lot of these studies it hasn't been obvious if there has been an error or if so what it was, which is why the field keeps going.
"Parapsychology consists of the practice of accidentally or willfully misunderstanding the scientific method."
The hypotheses put forward by parapsychology might well be false, but that's no reason to trash talk the researchers. I don't think there's any reason to think they're less intelligent, competent, or ethical than any other researchers, and a lot of them are highly respected, e.g. Stan Grof.
Time and again, researchers in this field have shown that they cannot or choose not to conduct reproducible science. In its decades of existence, parapsychology has never had a reproducible finding that cannot be explained by common physics or psychology. J Archibald Wheeler was right to try to eject parapsychology from the AAAS.
Why do you say that Stan Grof is associated with parapsychology? I had only heard of him in connection to psychedelic research previously, which is something else entirely.
He co-founded the field of transpersonal psychology, which is very similar. His research in this area was highly regarded even by Carl Sagan, who was probably the most famous skeptic of his day. He was especially interested in birth memories, memories of past lives, and also syncronicities.
Interesting. I wasn't familiar with transpersonal psychology before, but the wikipedia article on the topic specifically cautions against conflating parapsychology with transpersonal psychology. Consequently, I don't think that what I've said could reasonably be interpreted as an attack on people who you say are reputable, like Stan Grof.
Transpersonal psychology sounds harmless: a soft science that makes soft claims. Unless they barge onto the scene and start claiming that spirituality has a physical basis that (only) they can perceive, I'm probably not going to have any concerns with what they do.
In contrast, parapsychology makes claims about the physical world that would have real impact if true, but in the end what they claim is never true. If they've simply been suffering from "bad luck" for the past several decades, I think that real scientists can be forgiven for having a hard time distinguishing that run of bad luck from from incompetence or fraud.
One thing is true: on HN, you and I always seem to get into very interesting discussions (from my perspective, at least). I hope that my stern words for the field of parapsychology don't cloud the fact that I wish you well.
Yeah good discussion, I wish you well also. Anyway time for bed, but definitely go check out some of Grof's work. He seems to be really interested in reincarnation, which to me seems a step beyond psi in terms of plausibility. You almost couldn't have reincarnation without psi, or at least whatever underlying mechanism was powering it.
On a quick scan of the start of this article, the one thing that jumped out was that the pictures were randomized using the programming language's native random number generator.
I didn't see what language was used here, but it could be they just found a flaw in that code?
"The sequencing of the pictures on these trials was randomly determined by a randomizing algorithm … and their left/right target positions were determined by an Araneus Alea I hardware-based random number generator."
It is possible, but that would not easily explain why they only varied for one type of photo. Non-chance results should have been observed for all trial types, or with an obvious pattern of the human getting better over time. Though I can't prove that didn't happen.
If I'm reading this article correctly, it claims that there is statistically significant evidence that:
1) Precognition (at least when it comes to guessing where erotic photos will show up on a computer screen) is a real phenomenon
2) Precognition abilities are related to changes in the Earth's magnetic field
This certainly sounds hard to believe, but IF these experiments do turn out to be repeatable in the future (that remains to be seen), what effect would this have on our fundamental understanding of how the world works? It's pretty mind-blowing.
It is (potentially) mind-blowing. Our models, understandings, and predictive capability w.r.t. time, space, matter, energy, and causality would need revising. Jobs and reputations would be lost; new theories, paradigms, technologies, and industries would be born. That's all happened before, and one can look forward to it happening again. ;)
I believe that it will have a lot of influence in how we understand time and causation. I'm not a quantum physicist or a psychologist, but that's the gist of what I'm taking from it. If you look at the study as less of a study into precognition, and more as a study of how time operates then it seems a lot more plausible.
Even assuming that there were some precognition effects, we're talking about digital pictures displayed pseudo-randomly. What exactly are the subjects purported to precognit? The electrical states of ram circuits? Are they purported to have a built-in jpeg decompression algo in their brains, that allows them to know which images are erotic?
If the goal of the study is to show that people are able to access purportedly unknowable information, then this is a terrible design. There should be some much more easily knowable condition than binary memory states. If, on the other hand, they are trying to show that effect can precede cause, then physicists should be running the experiment, not psychologists. Introducing people to the equation introduces too much error.
Gosh. So from the one experiment that article described, people had premonition/precognition about an upcoming event when it had 'psychological valence.' Ie. when they were about to see an erotic picture, but not when they were about to see a picture from a less remarkable category (in the control experiments).
What I'd be interested in is if the level of premonition success showed any difference if the next picture was being selected in real time by another person, and not by a random computer algorithm.
It would be interesting if they combined these types of studies with brain scans to see if there are any patterns within the human brain during positives and negatives. This would help to find if there is a neurological basis for precognition, or if it is all really just random chance.
A lot of commenters don't appear to have read the paper on which the article is based on - critiquing at its best. Some factoids of note:
The Bem study isn't unique, there have been 1000s of studies exploring psi that have arrived at significant results. Look up Dean Radin for starters in case you're interested.
The Bem study is unique in that it is getting published in a mainstream journal, which is unusual since the mainstream psychology community doesn't believe or like the idea of researching psi, i.e. most consider it impossible. But that mainstream attitude is just that, an attitude.
Bem is fairly unusual as most researchers that do psi-research are considered on the fringe of the community and don't get in to the mainstream journals.
Researchers that do explore psi risk their reputations and careers.
Bem started out as a skeptic when he was invited to critique another researchers psi experiments. He's a well reputed researcher who has contributed his own theories that have been a real contribution to the field of psychology(non-psi related), something few scientists achieve in their careers.
So far, 2 papers have been written in response to Bem's study, as noted by another commenter. The first uses the Bem study to argue an entirely different agenda: the need for changes in the way general studies use statistics. The core of their argument would affect hundreds of thousands of papers across the entire field of psychology (and others). How valid the authors points are in relation to the Bem study is open to debate.
In the 2nd paper by other authors, it was attempted to replicate their findings. If you read the paper, you can't even tell how closely they followed Bem's original experiment, ie. what software did they use, how did the users install it etc? (and they didn't replicate all of Bem's experiments). One big warning bell is that they used internet participants, which is a shoddy way to do a psi experiment, or most any experiment for that matter.
For those that critique the design: Bem took 9 classic psychological experiments that the field is extremely familiar with and reversed them. Designing a good psi-experiment is incredibly hard and a lot more thought and scrutiny than the average experiment. Using classic experiments makes a great deal of sense.
@araneae:
Welcome to psychology research. Almost all the experiments, findings and studies that make it into journals were paved in copious amounts of (pilot)studies and experiments that yielded no results. Such is the nature of doing research in the field of psychology. Designing a psi experiment that stands up to scrutiny requires even more work, failures etc.
The concept of backward causation has been verified in other fields, like quantum biology and from a physics point of view, it is widely accepted both by theory and empirical study. It's not that big of stretch of the imagination that the most sophisticated organ that we have encountered in the universe (our brain) might be capable of it.
Fact #389: number one reason start ups fail and random critiques are flawed: they're based on relentlessly operating on unproven, unverified assumptions.
I can tell you that every semester that I was there he was running a version of the "Are you psychic?" experiment. I'm sure he's been doing it every semester for a very long time. Undoubtedly there have been loads of experiments where it didn't pan out. (If you're curious about my results, I got 54% and a cheerful grad student greeted me after the fact by saying "congrats! you're psychic!")
The fact is, if you run an experiment like this enough times you are going to get a significant result eventually. That's why you have alpha values. If it's at .05, that means that 5% of the time you're going to get a false positive. I think that's what this is.