Holmes vs e2e
Both use AI agents that work from plain-language goals, and both report on pull requests. e2e is free Apache-2.0 code that runs on your machines with your own model, and it also runs on physical phones. Holmes adds a hosted service with no test code, context from your specs and designs, and findings in Slack and Microsoft Teams for the whole team.
e2e tests are TypeScript files in your repository, with goals, locators or both. Holmes tests are plain-language goals, so there is no test code to write or maintain.
e2e runs on your machine or in your CI, and you connect the model and any hosted browsers or devices. Holmes runs in its own cloud, in isolated containers destroyed after each session.
e2e agents use the context you write into their config. Holmes reads the code change, the ticket, the spec in Notion or Confluence, the design in Figma and your top pages in PostHog before it decides what to test.
e2e reports on the pull request and in a JSON report and markdown summary. Holmes also sends findings to the ticket, Slack, Microsoft Teams and email, writes a plain-language summary of every release, and has unlimited users on every plan.
Based on each product's public pages on 17 November 2026. A red cross means e2e's pages don't describe it, not that it can't be done.
| Capability | Holmes | e2e |
|---|---|---|
| Testing | ||
| How tests are written | Yes: Plain-language goals. No test code to write | Yes: TypeScript test files with plain-language goals, locators and assertions, or a mix[1] |
| Finds bugs nobody scripted | Yes: Agents explore and test paths nobody wrote down | Yes: e2e explore runs an agent from a goal and reports findings with severity[3] |
| Keeps tests working when the UI changes | Yes: Flows repair themselves when a label or layout changes | Yes: Replay hands a step back to the agent when the screen no longer matches, then records the healed step[4] |
| Web browsers | Yes: Chromium, Firefox and WebKit, the engine behind Safari | Yes: Chromium, Firefox and WebKit through Playwright[1] |
| Native Android apps | Yes: Native Android apps on an Android emulator | Yes: Emulators and phones connected to your machine; hosted devices through integrations[2][10] |
| Native iOS apps | Yes: Native iOS apps on iPhone and iPad simulators | Yes: Simulators and paired iPhones; hosted devices through integrations[2][10] |
| APIs | Yes: Every request behind a journey, checked in context | Yes: API checks with fetch and value matchers, in the same files as UI tests[5] |
| Accessibility | Yes: WCAG audits in a real browser, with keyboard navigation | Not described: Not on its pages. WCAG audits are listed as planned[12] |
| Workflow | ||
| Pull requests and previews | Yes: Check and comment on GitHub and Bitbucket pull requests. GitLab supported | Yes: One comment per run on GitHub pull requests, results in GitLab merge requests, and Vercel preview deployments. Bitbucket isn't on its CI page[6] |
| Schedules | Yes: On every deploy, on a schedule, or by webhook | Partly: No built-in scheduler is described; runs come from your CI[6] |
| Context it reads before testing | Yes: Reads the code change, ticket, spec (Notion, Confluence), design (Figma) and analytics (PostHog) | Partly: Context and system prompts you write for each agent in its config. Jira, Linear, Figma, Notion or analytics aren't on its pages[11][5] |
| Coding agents (MCP) | Yes: Start runs and read findings from Claude and Cursor via MCP | Yes: MCP server that gives Claude Code, Cursor and other coding agents live app sessions[1] |
| Evidence with each bug | Yes: Recording, screenshots, console and network logs, steps to reproduce | Yes: Trace page, screenshots, video when recording is on, console output and failed requests[7][3] |
| Where findings go | Yes: Findings on the pull request, the ticket, Slack, Microsoft Teams and email | Partly: The pull or merge request, a JSON report and a markdown summary. Slack, Microsoft Teams or issue trackers aren't on its pages[1][6] |
| Where tests run | Yes: Holmes' cloud, in isolated containers destroyed after each session | Note: Your machine or your CI. Hosted browsers and devices through third-party integrations[12][6][10] |
| Pricing and security | ||
| What you pay for | Yes: Priced on how much Holmes tests, never on seats | Note: Apache-2.0 open source. Agent steps use your own model subscription, API key or local server; tests without agent steps need no model[14][8] |
| Users | Yes: Unlimited users on every plan | Note: User accounts or seats aren't described; tests live in your repository[1] |
| Entry price | Yes: Free plan. Paid plans from €199 ($219) a month | Note: No prices on its pages; the code is Apache-2.0 open source. Enterprise rollout help is sold through a call[14][15] |
| Getting started | Yes: €250 in free credits, no credit card | Yes: npx e2e init, then npx e2e run. The example needs no sign-in or API key[9] |
| Company base | Yes: EU company (Ghent, Belgium), GDPR compliant | Note: Not stated. TesterArmy's site says the same team built e2e[13] |
Holmes sells QA software, so we compete with e2e. We've tried to be fair: every statement about e2e comes from its own public pages, linked below with the date we checked them. Products change. If something here is out of date, email support@useholmes.com and we'll fix it.
The open source AI testing framework. Customizable TypeScript testing framework for web, mobile apps and more.Checked 17 November 2026.
runs tests on iOS simulators, Android emulators, and phones connected to your machine.Checked 17 November 2026.
Each finding has the steps to reproduce it and, when possible, a screenshot.Checked 17 November 2026.
Then it hands the rest of the step to the agent, which finishes the goal from the current screen.Checked 17 November 2026.
Check your API with fetch and value matchers, in the same files and run as UI tests.Checked 17 November 2026.
e2e runs on each CI service that can run Node 24.8 or later.Checked 17 November 2026.
Console output, uncaught exceptions, failed requests, and 4xx and 5xx responsesChecked 17 November 2026.
Tests without agent steps need no model.Checked 17 November 2026.
The example makes no model calls, so it needs no sign-in or API key.Checked 17 November 2026.
@e2e-dev/browserstack: BrowserStack App Automate real Android and iOS devices for mobile targets.Checked 17 November 2026.
An entry holds the model, an optional judge, system, context, tools, budgets, and providerOptions.Checked 17 November 2026.
Tests run on your machine or in your CI, not in TesterArmy's cloud.Checked 17 November 2026.
TesterArmy is winding down. New sign-ups are closed. Existing accounts keep working until January 15, 2027. The same team built e2e, an open source testing framework you run yourself.Checked 17 November 2026.
Apache License Version 2.0Checked 17 November 2026.
our engineers roll it out with you, then step back.Checked 17 November 2026.
e2e is an open source TypeScript framework. You install it in your repository, write tests that mix plain-language goals with locators, and run them on your machine or in CI with your own model. Holmes is a hosted platform. You describe journeys in plain language, and Holmes reads the ticket, spec, design and analytics, then reports on the pull request, in Slack, Microsoft Teams and email.[1][12]
Holmes tests native Android and iOS apps on emulators and simulators. e2e runs tests on iOS simulators, Android emulators and phones connected to your machine. Hosted devices, including BrowserStack real devices, come through its integrations.[2][10]
e2e's pages list no price; the code is Apache-2.0. Agent steps use your own model subscription, API key or local server, and tests without agent steps need no model. Hosted browsers and devices come from third-party providers. Holmes has a Free plan, Starter at €199 a month for up to 3 products, and Team from €499 a month, all with unlimited users. Holmes is priced on how much it tests, never on seats.[1][8][10]
TesterArmy's site says it is winding down: sign-ups are closed and existing accounts keep working until 15 January 2027. The same team built e2e, which has a guide for turning saved TesterArmy tests into tests in your repository that run on your machine or in your CI.[13][12]
Yes. Holmes imports existing test cases from a file (PDF, Word, Excel, CSV, JSON or text), from Jira, Linear, Notion or Confluence, or from the top journeys in your analytics. You can also write new ones in plain language; there is no test code to maintain.
If you want open source test code in your own repository, physical phones, your own choice of model, or nothing hosted by a vendor, e2e covers those on its own pages. If you want no test code, nothing to run yourself, context from specs and designs, and findings in Slack and Microsoft Teams for the whole team, try Holmes.[1][2][8]
Evaluating e2e? See what Holmes finds first.
Free, no scripts, no SDK, and no test account needed to start.