Playwright vs Vitest vs Web Test Runner vs Karma - Comparing Browser Testing Strategies
Libraries and Strategies for Testing in the Browser
I write this article because in my research to improve my integration tests in Fluski, I found little content, and because in that little content, the concept of parallelization is not well defined.
Until recently, I thought there were only two options for doing tests that involve UI: E2E tests in the browser (slow but reliable) or integration tests in Jsdom (fast but insecure).
It turns out that there are several options for doing integration tests (or component tests) in a real browser.
This combines the best of both worlds: reliability and speed. So fast that it allows the people at Preact to run 1000 tests in 1 second.
ILP and TLP
If you're wondering what the last 2 columns mean:
- ILP (instruction-level parallelism): Even if two tests use the same CPU thread, they could save time if, when one is waiting for a synchronous instruction, the other can advance its execution.
- TLP (thread-level parallelism): As the name indicates, the tests are executed in different threads.
The ideal would be a solution combining ILP and TLP, but this does not exist as of today.
Achieving ILP+TLP can be challenging because of the way browsers work. For example, Chrome sometimes uses the same thread in different tabs and sometimes it doesn't. Even so, I believe that with the efforts that are being made in this area, this will come.
Where Are the Tests Run?
In this article, I focused only on strategies and libraries for testing in the browser (that's why I didn't mention Jest).
However, there is an important distinction to make. All the libraries I mentioned run 100% in the browser, except Playwright, which interacts with the browser through an API in another language (which can be Node.js).
An advantage of this is that Playwright can be used, in addition to JavaScript (Node.js), with other languages such as Python or Java.
A possible disadvantage is that this forces the tests to follow the library's API. Migrating from web-test-runner to @vitest/browser shouldn't be that difficult. Migrating from any of those 2 to Playwright may not be trivial.
Although to be fair, most people using web-test-runner or @vitest/browser probably combine it with something like Testing Library whose API is syntactic sugar for interacting with the browser.
I'm not sure how running the tests 100% in the browser, or explicitly through an API like in Playwright, can affect the reliability of the tests.
If you have more information about this or can think of other advantages or disadvantages of these approaches I would love to hear from you.
Future Work
This article is a work in progress. Some things I would like to add:
- A benchmark comparing these options in the table.
- Document missing strategies: Safetest (from Netflix), StoryBook, and Cypress Components.
- Document what isolation level they have.
- A GitHub Action workflow that caches browsers so as not to have to install them on every CI (it's complicated).
You may want to follow me on Twitter to find out when this happens. Contributions on these points are welcome.
Acknowledgments
Many thanks to the people on the Vitest Discord (Vladimir, James, Joaquin and Mohammad), Playwright (mxschmitt, Mindaugas) and Twitter (justinfagnani, _rschristian) for their comments on this topic.