How we test

1. Pick a builder question. Candidates come from what indie builders are actually asking about.
2. Run it in isolation. Each tool runs in its own locked-down container: non-root, no extra privileges, and network access limited to the tool's own vendor.
3. Score with checks. Automated checks, including hidden tests the tool never sees, decide pass or fail. Tools aren't allowed to edit the tests.
4. Cap the cost. Each run is capped at $2 and 15 minutes, with up to 3 runs per task.
5. Record the evidence. Each run writes a validated evidence record: tool and version, plan or API billing, checks, time and cost. Records are scanned for secrets before publishing.
6. Write the verdict. An editor drafts Keep, Ship or Skip. Every Skip gets human review before it goes out.

Tools we can't test end-to-end go to a review queue instead of getting a "we tested it" verdict.

FAQ

What does a "tested" verdict mean?

The tool completed a real task in our isolated test bench and was scored by automated checks, including hidden tests it never saw. No completed test means no "we tested it" verdict.

Who runs the tests?

AI agents run the tests and draft the verdicts under human editorial approval. Every issue discloses this.

Can sponsors buy a verdict?

No. Sponsors buy a clearly labeled slot. They never see or influence verdicts.

What do Keep, Ship and Skip mean?

Keep: worth paying for long-term. Ship: good enough to try now. Skip: not worth it at its price.