Reward delay parameters not set correctly

[alexpiet]

Consolidating discussions having in multiple places

@bwtan-allen commented here poor performance over the past few weeks

@mlee10201 opened an issue here

@XX-Yin believe this PR is the source of the problem: #1620 (comment)

@rachelstephlee commented on teams here

I believe that PR was finally merged to main here: #1666

To resolve:

• Revert last week's PR (Revert "[update main] 2025-11-11" #1672)
• Determine a list of sessions that were impacted, and track in the data-utils repo discussion page @rachelstephlee
• Verify that this was the only cause of poor performance recently. @bwtan-allen please review your data and make a recommendation
• Understand what went wrong in our testing process, and make a plan to avoid future issues
  • Was manual RA action required for all sessions to set these parameters? Where/how was this documented?
  • What open issues were mentioned on the PR and not addressed?
  • How was this not identified during review, testing (including isolated testing, production testing, and integration testing) ?
• Sync main and production_testing and develop @micahwoodard

Before we add these updates back to main: @hagikent @rachelstephlee

• Make sure the default of these parameters are set to be off, or whatever their original state was
• Make sure AutoTrain system sets these parameters correctly. RA should NOT manually set parameters!
• Make sure these parameters are being logged in NWBs

[ellahiltonvano]

@hagikent I'll let RAs running CS2 know that they will need to switch branches again.

[rachelstephlee]

much thanks everyone. I just tested the PR with the secondary stimuli last Monday 11/10. What dates would have this PR have impacted? (i.e. should I correctly assume that the PR was merged in by Tuesday morning 11/11? Just wanted to get a range of dates of affected sessions.

[ellahiltonvano]

@rachelstephlee we merged in these updates Wednesday (11/12) morning. I manually switched all of the boxes to main, so Weds, Thurs, Fri, and Mon sessions were affected (11/12/25-11/17/25).

[rachelstephlee]

Thanks. @bwtan-allen said he noticed performance changes in the fiber boxes (all D boxes) prior 11/12 date. He was wondering if RA's might have accidentally swapped to the dev branch the week prior and might have run a few sessions on the dev, not main branch.

Kenta and I should look over mouse performance starting when the PR was merged in dev just in case, but I would be surprised if RAs would accidentally do this given your strict SOP.

[alexpiet]

• RAs would never run on the dev branch. @bwtan-allen Can you quantify or document the performance changes prior to 11/12?
• The production_testing branch was run on the testing rigs on 11/3 (and maybe 11/4), 10/11, and 11/11.

[ellahiltonvano]

@rachelstephlee agree with Alex about the Dev branch. RAs were running a subset of mice on the second_stimulus_merge branch per Kenta's request (I can get you these IDs if you'd like). @bwtan-allen could you be referring to these mice? RAs have noticed poor performance in these animals compared to other mice on stage graduated.

[ellahiltonvano]

@alexpiet for this question: Was manual RA action required for all sessions to set these parameters? Where/how was this documented?

1. Documented here, SOP distributed to all RAs, I walked through the process when people ran their first mouse on CS2.
2. I communicated changes at the NSB team meeting to all foraging RAs (i.e., RAs no longer have to switch branches before training, but need to manually set parameters outlined in the above SOP).

[ellahiltonvano]

For this question: How was this not identified during review, testing (including isolated testing, production testing, and integration testing)?

Margaret was able to notice the misalignment of reward times on the ephys rig only, because there is only one box (one tone), and the solenoids in that rig emit a clearer clicking sound, allowing the user to more easily identify the exact timing of reward delivery.

Some ideas for why it would have been difficult to identify in 446/7 alone:

1. It is not in our SOP to use the command prompt window (this would have been the easiest way to determine timing of trial events). I have discouraged RAs from opening this because closing it the wrong way can result in data loss.
2. Related to 1, even if an RA is checking this window, it is nearly impossible to extract any information because of the presence of the environment sensor. This clutters the screen and is the only information that a user can see.
3. It is hard to determine exact timing of rewards when you are 1) running multiple mice and 2) can't tell which box the GoCue for a specific trial is coming from.

We should probably consider expanding our integration testing SOP as a result, it is not surprising to me that the current SOP did not capture this bug - this seems like something that would be easier identified through scientist analysis of behavior data.

[alexpiet]

@alexpiet for this question: Was manual RA action required for all sessions to set these parameters? Where/how was this documented?

1. Documented here, SOP distributed to all RAs, I walked through the process when people ran their first mouse on CS2.
2. I communicated changes at the NSB team meeting to all foraging RAs (i.e., RAs no longer have to switch branches before training, but need to manually set parameters outlined in the above SOP).

!

We should not ever set parameters manually, thats the whole purpose of the autotrain system. It seems like the big problem was that those parameters needed to be manually set to be off for non-CS2 mice.

[ellahiltonvano]

@alexpiet agreed we should integrate to autotrain, but I think that the default once we merged was still to have CS2 set to off - I remember that RAs had to turn it on before running these mice, but I could be wrong.

[rachelstephlee]

I did the software integration test with @henryloeffler , and I faintly recall that it might have been ON by default, which I really regret not flagging during the software integration test.

@henryloeffler do you happen to remember if it was on or off by default? I am 80% sure that the default was OFF but I really cannot remember off the top of my head.

Agree with ella thiat should be caught by scientists. I would recommend for software integration testing-- for big PR changes like this we may want to collect data for a few sessions to test ahead of time and validate correctly. My understanding is that Kenta did that, but perhaps we didn't take a close enough look at this.