From Predicted to Confirmed

1. Context

Before the SCHOLAR documentation platform reached youth participants at Assemble, a children's museum makerspace in Pittsburgh, the broader Maker Corps network — adult and near-peer mentors across four site cohorts in Pittsburgh, Detroit, Baltimore, and Albemarle — had already been using an early version of the platform for several months. With a full youth rollout approaching, I designed and ran a structured usability survey of this adult tester group to catch problems while the cost of fixing them was still low.

2. Problem & Constraints

Core problem: How might we know, before deploying to youth, whether a documentation platform would actually hold up in a low-support, drop-in environment — using only the small tester group available?

• Small sample. Five adult respondents, a convenience sample drawn from staff already on hand, not a recruited study population.
• Hard deadline. The survey needed to run and be analyzed before what internal project documents called an "imminent launch."
• Distributed testers. Five respondents spread across four sites with different device and network setups, mirroring the environment the youth pilot would later face.

Five responses is not a sample size that supports statistical generalization — the judgment call was how much weight a small, non-representative sample deserved.

— method note from this evaluation

3. Method Rationale

Mixed-methods by necessity

With only five respondents, scaled ratings alone would have been too thin to act on, and open text alone would have been hard to prioritize. Pairing a Likert-scale item with an open-response item on the same question let me measure both the size of a problem and its likely cause.

Treat small data as hypothesis-generating, not conclusive

Rather than discard the sample as too small to matter, or present it as more definitive than it was, I used it to generate specific predictions and then looked for an independent source to confirm or disconfirm them.

Triangulate instead of stacking

I deliberately paired the survey with field observation from a different population (youth, not adult testers) so that a shared finding across both would mean something a single source could not show alone.

4. Process

Five adult testers completed a 13-question survey mixing Likert-scale ratings with open response. Weeks later, I observed a youth field pilot run into the same class of problems in real conditions:

5. Synthesis

• The adult survey is anticipatory and mechanism-focused — it explains why the platform would struggle before a single youth user touched it.
• The youth pilot data is outcome-focused — it shows what those mechanisms actually produced in the field.
• Presented together rather than separately, neither source could be dismissed the way it might have been alone: the survey as "just adult opinion," the field notes as "one difficult week."

6. Findings & Design Decisions

• Overall experience rating: mean 2.2 of 3 across five adult testers, with zero respondents selecting the top rating — a ceiling effect suggesting no tester came away confident in the tool.
• Instruction clarity: mean 1.8 of 2, dominated by "sorta" rather than "yes" — a signal that onboarding, not just the interface, needed work.
• Open-text themes converged on two mechanisms — unclear submission state and buried actions — that recurred as concrete technical failures in the youth pilot weeks later.
• A tester's prediction that mentors "won't engage with it, or not willingly" anticipated the disengagement later observed directly in the field.

7. Outcome

The convergence of predicted and confirmed failure gave the recommendation to move off SCHOLAR a stronger evidentiary basis than either source alone — a small internal survey plus a short field pilot, read together, carried more weight than a larger single study might have on its own. That recommendation directly informed the broader platform decision that followed.

Time to ditch Scholar?

— field reflection memo, August 2014

8. Reflection

This is the clearest example I have of using a small, imperfect dataset well rather than waiting for a bigger one, and of naming a convenience sample's limits explicitly rather than overselling it. If I were extending this work, the next step would be a pre-registered prediction log — writing down what a small pilot suggests before the field data comes in, to make the triangulation auditable rather than reconstructed after the fact.