Community tests / arxiv-prompt-to-product-2025

Community test · not ours

From Prompt to Product: Human-Centered Benchmark (Replit, Bolt, Firebase Studio)

By Marcos Ortiz, Justin Hill, Collin Overbay, Ingrida Semenec, Frederic Sauve-Hoover, Jim Schwoebel, Joel Shor · 2025-12 · method: mixed

Open original source Creator program (30%)

Task

96 prompts spanning domains/difficulty; 288 generated apps; 205 human raters; 1,071 pairwise comparisons after quality filtering.

Single-shot generation per system+prompt; deployed; human isolated ratings + side-by-side preferences (ease, trust, visual appeal, visual appropriateness). Bradley-Terry / LMM analysis.

Creator credit

This page is unclaimed. If you ran this test, you can claim it and — once the creator program is live — earn 30% of affiliate commission generated from outbound clicks on this page.

Program status: launching — we do not promise earnings.

Claim code (paste into the form): BUILDOFF-CLAIM-ARXIV-PROMPT-TO-PRODUCT-2025

Is this your test? Claim it

Result (as stated)

Pairwise win rates (Table 2): Firebase highest on all four metrics (Ease 42.5%, Trust 41.2%, Visual Appeal 45.5%, Visual Appropriateness 45.1%); Bolt second; Replit third. Authors' hierarchy: 'Firebase > Bolt > Replit'. Composite scores stated: Firebase 1.933, Bolt 1.561, Replit 1.302.

Places we recorded: #1 Firebase Studio (composite 1.933 / top win rates) · #2 Bolt (composite 1.561) · #3 Replit (composite 1.302)

Conflict of interest: Study funded by Quome Inc. (acknowledgements). Quome Cloud used for deployment.

Largest academic human study found. Does not include Lovable/v0/Base44.

Try the builders mentioned

Outbound links include a tracking param tied to this test (bo_test=arxiv-prompt-to-product-2025 ). Real affiliate IDs are TODO until programs are approved. Disclosure.

Replit Bolt → Firebase Studio

Claim this test

Ownership verification (free, no backend yet): post this unique code from the same X account (or on the original page) that published the test, then submit the claim form. 

BUILDOFF-CLAIM-ARXIV-PROMPT-TO-PRODUCT-2025
  1. Post a public X post (or add a line on the original article) containing the code above.
  2. Open the claim form and paste the code into the “claim code” field (prefill is best-effort).
  3. We match the code to the author handle on the listed source. Manual review until automation ships.
Open claim form