Every project at EverOasis answers one question: is the person more able when the tool is put away? This paper is about measuring that question without lying to ourselves.
Most product metrics reward dependence. Engagement, session length, and assisted-task completion all rise when the tool does more of the thinking. Capability retention moves in the opposite direction. If people cannot perform without the assistant a week later, we did not build a tool for growth. We built a crutch with a polished UI.
A practical method
We use a remove-and-retest protocol. Learners complete a sequence with the tool available, then return after a fixed interval with the tool withheld. Scoring focuses on transfer: can they explain, apply, and adapt — not merely recall the assisted answer.
The hard part is not the interval. It is choosing tasks that the tool could have completed alone. If the assisted path is pure output delivery, retention tests only memory of that output. Useful tests leave a residue of judgment.
What we discard
We discard any study design where the control group is “no tool” from day one and the treatment group never loses access. That comparison measures performance under assistance, not growth. We also discard self-report confidence as a primary endpoint. People feel more capable long before they are.
The method is slower than A/B tests built for conversion. It is also the only test that matches the claim we make about our work.