Holmes vs Checksum
Both use AI agents to build end-to-end tests, repair them when the app changes and run them on pull requests. Checksum delivers Playwright code into your repo, checked by its own engineers. Holmes adds native mobile and accessibility testing, context from specs and designs, and public prices with a free start.
Holmes publishes its Free, Starter and Team prices, and you can set it up yourself with €250 in free credits. Checksum's pricing page lists workflow tiers without dollar amounts, and its docs say every customer starts with a demo and a Proof of Value run.
Holmes tests native Android and iOS apps on emulators and simulators with the same plain-language goals it uses on the web. Checksum's pages describe web, API and unit tests; native mobile apps aren't described.
Holmes runs WCAG audits in a real browser, keyboard navigation included, and walks your product as the personas you build for. Checksum's pages don't describe accessibility testing.
Checksum learns from your app and, if connected, your front-end code. Holmes also reads the ticket, the spec in Notion or Confluence, the design in Figma and your top pages in PostHog before it decides what to test.
Based on each product's public pages on 17 November 2026. A red cross means Checksum's pages don't describe it, not that it can't be done.
| Capability | Holmes | Checksum |
|---|---|---|
| Testing | ||
| How tests are written | Yes: Plain-language goals. No test code to write | Yes: An AI agent writes Playwright (or Cypress) tests, engineers verify them, and they arrive as pull requests[1][2] |
| Finds bugs nobody scripted | Yes: Agents explore and test paths nobody wrote down | Partly: Proposes the key flows from your app; production errors become tests[2][1] |
| Keeps tests working when the UI changes | Yes: Flows repair themselves when a label or layout changes | Yes: A healing agent fixes the test code and opens a pull request; about 70% of failures resolve on their own[1][11] |
| Web browsers | Yes: Chromium, Firefox and WebKit, the engine behind Safari | Yes: Playwright, described as cross-browser; the browser list isn't on its pages[3] |
| Native mobile apps | Yes: Native Android and iOS apps on emulators and simulators | Not described: Not on its pages. Native iOS or Android apps |
| APIs | Yes: Every request behind a journey, checked in context | Partly: An API agent chains requests across endpoints; Enterprise plans only[4][5] |
| Accessibility | Yes: WCAG audits in a real browser, with keyboard navigation | Not described: Not on its pages. WCAG audits |
| Workflow | ||
| Pull requests and previews | Yes: Check and comment on GitHub and Bitbucket pull requests. GitLab supported | Yes: Runs on GitHub pull requests and GitLab merge requests, and can test each preview deployment[6] |
| Schedules | Yes: On every deploy, on a schedule, or by webhook | Yes: Pull requests, merges and schedules; a REST API for Jenkins, CircleCI and other CI[6] |
| Context it reads before testing | Yes: Reads the code change, ticket, spec (Notion, Confluence), design (Figma) and analytics (PostHog) | Partly: Your app and, if connected, its source code. Figma, Notion or analytics aren't on its integrations page[2][3] |
| Coding agents (MCP) | Yes: Start runs and read findings from Claude and Cursor via MCP | Yes: Hosted MCP server and plugins for Claude Code, Cursor and VS Code[7] |
| Evidence with each bug | Yes: Recording, screenshots, console and network logs, steps to reproduce | Yes: Videos, screenshots, network recordings and Playwright traces[8] |
| Where findings go | Yes: Findings on the pull request, the ticket, Slack, Microsoft Teams and email | Yes: Jira, Linear and ClickUp, synced both ways; Slack, Microsoft Teams, Discord and Google Chat[9][10] |
| Managed service | Not covered: Not covered. Self-serve, with guided onboarding on Team and up | Yes: Engineers verify each test, and Checksum submits the fixes when tests break[1][11] |
| Pricing and security | ||
| What you pay for | Yes: Priced on how much Holmes tests, never on seats | Note: Workflows maintained, each up to 30 actions; unlimited runs, heals and users[5] |
| Entry price | Yes: Free plan. Paid plans from €199 ($219) a month | Note: Emerging plan for 50 workflows; dollar prices aren't on its pricing page[5] |
| Getting started | Yes: €250 in free credits, no credit card | Note: A demo and a Proof of Value run with its team; plans list a free 30-day trial[2][5] |
| Inside your network | Not covered: Not covered. Cloud only | Partly: The CLI runs on your own CI runners to reach apps on your network[6] |
Holmes sells QA software, so we compete with Checksum. We've tried to be fair: every statement about Checksum comes from its own public pages, linked below with the date we checked them. Products change. If something here is out of date, email support@useholmes.com and we'll fix it.
Checksum’s Results as a Service solution includes human engineer final verification so Checksum delivers 100% ready-to-go, working Playwright tests.Checked 17 November 2026.
Test code is delivered to your repository as pull requests. No vendor lock-in: run the tests anywhere.Checked 17 November 2026.
Core browser automation engine powering Checksum's cross-browser test execution.Checked 17 November 2026.
Checksum tests whether your system actually works, chaining requests across multiple endpoints to verify that real user flows behave correctly end to end.Checked 17 November 2026.
Unlimited test runs. Unlimited auto-healings. Unlimited users.Checked 17 November 2026.
Run your Checksum suite automatically on pull requests, merges, and schedules.Checked 17 November 2026.
You can use Checksum from Claude Code, Cursor, VS Code, and other AI coding tools.Checked 17 November 2026.
per-test status, videos, screenshots, network recordings, and Playwright tracesChecked 17 November 2026.
Create and update Jira, Linear, or ClickUp tickets automatically when Checksum finds bugs, and keep their statuses in sync in both directions.Checked 17 November 2026.
Slack, Microsoft Teams, Discord, and Google Chat are supported.Checked 17 November 2026.
Checksum detects the break, updates the flow, and submits a fix as a PR automatically.Checked 17 November 2026.
Checksum's agent writes Playwright tests, its engineers verify them, and the code lands in your repo. Holmes works from plain-language goals with no test code to maintain, tests native Android and iOS apps and accessibility as well as the web, and reads specs, designs and analytics before it tests.[1]
With Checksum, yes in the code sense: test code is delivered to your repository as pull requests, and you can run it anywhere. Holmes keeps plain-language goals in Holmes instead of test code. You can import existing test cases from a file, from Jira, Linear, Notion or Confluence, or from the top journeys in your analytics.[2]
Checksum prices by the number of workflows it maintains, where one workflow is an end-to-end test of up to 30 actions, with unlimited runs, heals and users. Its tiers start at 50 workflows, its pricing page shows no dollar amounts, and plans list a free 30-day trial. Holmes has a Free plan, Starter at €199 a month for up to 3 products, and Team from €499 a month, all with unlimited users.[5]
Holmes tests native Android and iOS apps on emulators and simulators. Checksum's pages describe end-to-end, API and unit tests for web apps; native mobile testing isn't described there.[1]
If you want a team that writes and verifies your tests for you, want standard Playwright code in your own repo, want flat pricing per workflow with unlimited runs, need chained API tests on an Enterprise plan, or send alerts to Discord or Google Chat, Checksum covers those on its own pages. If you want to start on your own today, test mobile apps and accessibility, and give the agent your specs and designs, try Holmes.[1][2][5][4][10]
Evaluating Checksum? See what Holmes finds first.
Free, no scripts, no SDK, and no test account needed to start.