The V1 to V2 judge migration

While developing our V2 judge, we found a source of bias in the V1 judge. The V1 judge was blind by design: it received the real target shuffled in with nine decoys and did not have any way to know which one was the true target. The bias did not come from the judge knowing the answer. It came from how we handled the judge's errors.

In very rare circumstances, the V1 judge hallucinated targets while ordering the set of decoys: some of the candidates in its reply were not candidates we had sent. When it passed its results back to the code that called the judge, that code only ever looked for one thing, the position of the true target. It never confirmed that all of the other decoys were still valid. A run where the judge invented half the lineup looked identical to a clean run, as long as the true target appeared somewhere in it.

What this meant was that in error scenarios where the LLM hallucinated while ordering the decoys, there was a slight bias in choosing the true target, because we were not detecting the hallucination error and re-running the judging in that case. Runs that hallucinated the true target away failed loudly and got retried. Runs that hallucinated only decoys passed silently. The errors that survived were the ones where the true target held its position against a partly imaginary field, and across thousands of sessions that selection effect nudged the true target upward.

We fix this in the V2 judge in two ways.

First, when the V2 judge returns its results, we confirm that all 10 targets it ordered in the decoy set are valid: every entry in the ranking has to be a candidate we actually sent. If any are invalid, we re-run the whole thing. No error scenario can slip through silently anymore, which reduces the bias completely.

Second, the new models are simply better and hallucinate less often. The verification pass is the guarantee, but in practice it rarely has to fire.

We have null tested the V2 judge, and this null testing is actually how we found the bias in the V1 judge in the first place.

To null test a judge, we take our existing session results and swap in a new random target and new random decoys: a full new set of target and decoys that does not include the previous target or the previous decoys. We then run the judge over those sessions and see if we get chance results. The session was never about that target, so if there is no bias in the judge, we should see exactly chance results when evaluating it against the wrong target, no matter whether remote viewing is real.

Our new judge does properly null to chance when tested against the wrong targets, and we get a better-than-chance result when we run it against the true targets. During development, we also only tested it against the first 10,000 sessions in our dataset, to make sure we do not overfit the judge to our sessions.

When we developed the V2 judge, we saw that it performed above chance. When we were testing it in isolation and evaluating it, we were using the same decoys that the V1 judge used. When we rolled out the V2 judge in production and backfilled all the old sessions with V2 scores, we let it pick new decoys. We did not think this would matter, but it turns out it mattered a lot.

Within the remote viewing community, it is commonly believed that you can get "displaced" onto your decoys, meaning some of the decoys could come through in the session. Because of this, we had assumed that re-running judging with a new set of decoys should not alter the result, and in some cases might even improve it. We saw the exact opposite: picking new decoys and re-running the judging reduced the effect to chance.

This is the primary reason we were delayed in releasing the new stats page and had to hide it. At Social RV, we never want to host data that we know to be invalid, but we also did not want to put up data that says "remote viewing is not better than chance" without being highly confident and exhausting our research options. While working through it, we then hit a subsequent, unrelated bug in the fix itself, which is what truly borked the page and delayed bringing it back.

Why the original decoys carry the effect is now an open research question in its own right, and one of the more interesting findings to come out of this platform.

In our stats page today, sessions that were previously run on the V1 judge have been re-run on the V2 judge using the same decoy set that was used with the V1 judge. New sessions are scored by V2 with full verification from the start. These are the results you see on the stats page now: a judge that verifies its own output, the original decoys, and an effect that holds above chance.

The whole point of Social RV is for the results to be trustless: even if Social RV were actively a bad actor in the space, it should be possible to verify that none of our data was influenced, modified, or tampered with. Our blockchain-backed sessions already solve this for proving sessions are blind. What we did not realize is that we also needed to solve it for proving which decoys were originally picked for judging, and that they were picked truly randomly. Before this incident, we did not think the decoy set mattered enough to record.

Given this learning about the impact of the decoys, our blockchain-based provenance is no longer adequate. There is no funny business in this data that we are aware of, but researchers are currently left trusting that we have not done something weird with the decoys, and they should not have to.

To solve this, we will be releasing a new version of the blockchain-backed sessions which also records the decoys we pick, and uses the blockchain randomness to pick them in the first place. Until that ships, the decoy sets are the one part of the custody chain that requires trusting the platform.

If you see any other part of the data custody chain we are still missing, tell us at contact@social-rv.com.