Methodology

Today: BuildOff is a sourced compilation of public third-party tests. We open each source, copy the result as stated, flag conflicts of interest, and aggregate 1st / top-3 / last among roster builders that appeared in that test's ranking. We never invent or paraphrase scores into numbers the author did not give.

Coming later: our own Playwright / Lighthouse / security harness (already in harness/) — not publishing BuildOff scores until we run it for real. We are not paying for builder plans yet.

Community tallies (live now)

Only tests with includeInTallies: true and a declared ranking enter the home aggregates.
“1st / top-3 / last” are among builders on our roster that appeared in that test — not among every tool the author mentioned.
Vendor-published or self-preferencing tests stay in the dataset and are flagged (e.g. Converge Slack clone, Emergent bakeoff).
Sources we could not open are listed but marked unverified and excluded from tallies.
Full dataset: /community-tests/ · CSV.

Planned BuildOff harness (coming later)

Weight

Functional · 50%

Playwright suite shipped with each weekly spec. Pass rate × 100.

Weight

Security · 30%

Start at 100, subtract fixed penalties per finding. Critical findings also cap overall at 40.

Weight

Lighthouse · 20%

Mean of Performance, Accessibility, Best Practices, SEO (mobile).

Overall formula (when we publish)

overall = functional×0.5 + security×0.3 + lighthouse×0.2
if any critical security finding: overall = min(overall, 40)

Security scan (planned)

Passive (always): secrets grep, JWT classification, HTTPS + headers, source maps.
Active (only on apps BuildOff itself built, --owned): read-only probes. Never pointed at third-party sites.

Penalties

Severity	Points subtracted
critical	100
high	40
medium	15
low	5
info	0

What is not scored

Aesthetic preference is never scored. Cost and build time will be reported separately when we run our own builds and will not enter the overall score. Community tallies are not a BuildOff score.

Reproducibility (harness, when used)

node harness/run.mjs \
  --url https://YOUR-APP.example \
  --spec 01-habit-tracker \
  --builder lovable \
  --owned --lh-runs 3 \
  --cost 4.20 --time 37 --prompts 2

Harness source: harness/. Specs: specs/.