Choosing the Right Tool

1. Context

Learning in the Making studied how young people learn by making, across informal sites including a children's museum makerspace (Assemble, Pittsburgh), a Fab Lab in an underserved Detroit neighborhood (Mt. Elliott), a public school makerspace (Digital Harbor, Baltimore), and MAKESHOP at the Children's Museum of Pittsburgh. A design question sat underneath the research questions: could a web platform capture youth documentation and peer critique well enough to become part of the making process itself, rather than a separate reporting burden?

Nothing connected the informal-making experience to a lightweight way to reflect on it and get feedback.

— framing drawn from the project's NSF proposal

As platform liaison, I owned that question end to end: evaluating candidate tools, testing them by hand, observing real deployments in the field, and turning findings into recommendations the research team acted on — across three successive platforms.

2. Problem & Constraints

Core problem: How might youth makers document their process and receive structured critique, in a drop-in informal setting, without adding technical friction heavy enough to break engagement?

• Informal, drop-in environments. Kids could arrive and leave mid-project; the tool couldn't assume a fixed class schedule.
• Non-technical facilitators. Sites were staffed by volunteers and near-peer mentors with little capacity to troubleshoot software.
• Device limitations. Several sites had only Chromebooks, which the first platform tested had not been built or tested for.
• Young users. Ages roughly 8–17, requiring an interface simple enough to use with minimal onboarding.
• No engineering budget. Every option had to be an existing, third-party platform, adapted rather than built.

A platform for this kind of setting needs scaffolding for critique built in — it cannot be a fully open-ended platform — and its operation must be almost immediately apparent.

— design principle extracted from the evaluation, later published

3. Method Rationale

Layered evaluation before commitment

Rather than trust a feature list, I staged the evaluation to catch failures at the cheapest point possible: comparative analysis first, then a hands-on usability walkthrough, then field observation during a live pilot, then a stakeholder conversation once a specific limitation needed a decision.

Field evidence outweighs the comparison matrix

When a tool scored well on paper but failed in the field, I weighted the field result — a platform's spec sheet only matters if it survives contact with an actual Chromebook and an actual ten-year-old.

Escalate to the vendor only after the data pointed somewhere fixable

I brought a specific, evidence-backed gap to each vendor rather than a general complaint, which is what made a workaround conversation productive rather than just a support ticket.

4. Process

The evaluation moved through three sequential iterations:

5. Synthesis

• For the research team: a clear, evidence-based rationale for abandoning two already-configured tools rather than settling — expensive in the short term, but what made the third choice actually work.
• For future tool decisions: a reusable design principle (critique scaffolding is required; onboarding friction is not negotiable) that outlived this specific project.
• For the field: a documented account, later published, of what happens when a general-purpose platform is asked to do specialized work without the right scaffolding.

6. Findings & Design Decisions

• SCHOLAR's export function actively misled users: the "Journal Article" format disabled in-browser viewing and produced only error screens, found through direct hands-on testing rather than a feature audit.
• Device assumptions baked into a platform can silently fail an entire site — SCHOLAR's desktop-browser design meant Chromebook-only sites lost core functionality without warning.
• Ease of use and critique scaffolding traded off directly across all three tools; no platform delivered both, which justified moving to a purpose-built third option instead of settling for tool #2.
• Observed behavioral contrast — disengaged during platform use, engaged during hands-on making — provided evidence for the abandonment recommendation that survey data alone would not have captured as vividly.

7. Outcome

The recommendation to move off SCHOLAR was acted on within the same project cycle. The move to Build in Progress led to a multi-year collaboration with the tool's creator, a national conference presentation, and a peer-reviewed publication co-authored with the project's principal investigator. Deployment expanded beyond the original research sites into public libraries and children's museums.

LitM will begin testing Build in Progress in the Fall of 2015.

— Johnson & Halverson, Learning in the Making: Leveraging Technologies for Impact

8. Reflection

This project taught me to treat a platform decision as a hypothesis to test, not a one-time procurement choice, and to recommend killing a tool the team had already invested in when the field evidence supported it. If I were extending this evaluation today, the highest-value next step would be outcome data from the actual Build in Progress deployment — the archive establishes intent and collaboration through 2016 but not yet how that pilot performed.