diff --git a/test-strategy.md b/test-strategy.md index 480db4233c..41cef3c926 100644 --- a/test-strategy.md +++ b/test-strategy.md @@ -107,4 +107,4 @@ For instructions on how to write and run tests see the [contributing guide](http As we formalise our testing strategy and bring / keep our test up to standard, it is important that we have some principles on managing defects as they occur/ are reported. For now we can have one very simple rule: -- Every bug fix should be accompanied by a test that failed before the fix and passes afterwards. This test should _typically_ be a dynamic code test but it could be a linting rule or new type if that is appropriate. There are always exceptions but we should think very carefully before ignoring this rule. +- Every bug fix should be accompanied by a test that failed before the fix and passes afterwards. This test should _typically_ be a dynamic code test but it could be a linting rule or new type if that is appropriate. There are always exceptions but we should think very carefully before ignoring this rule. \ No newline at end of file diff --git a/testing-guidelines/playwright.md b/testing-guidelines/playwright.md new file mode 100644 index 0000000000..51b4e064a2 --- /dev/null +++ b/testing-guidelines/playwright.md @@ -0,0 +1,98 @@ +## Playwright Tips + +Gradio uses [playwright](https://playwright.dev/docs/intro) to interact with gradio applications programmatically to ensure that both the frontend and backend function as expected. +Playwright is very powerful but it can be a little intimidating if you haven't used it before. +No one on the team is a testing expert so don't be afraid to ask if you're unsure how to do something. +Likewise, if you learn something new about playwright, please share with the team! + +### Tip 1 - Retrying Assertions + +Playwright tests are written imperatively - first type into this textbox, then click this button, then check this textbox has this output. +This is nice because it matches how users interact with Gradio applications. +However, playwright carries out these steps much faster than any human can! +This can cause you to check whether a textbox has the correct output before the server is finished processing the request. + +For this reason, playwright ships with some [retrying assertions](https://playwright.dev/docs/test-assertions#auto-retrying-assertions). +These assertions will retry until they pass or a timeout is reached, by default 5 seconds. +So even if playwright checks a DOM element before the server is done, it gives the server a chance to finish by retrying. + +An example of a retrying assertion is `toBeChecked`. Note that you can manually increase the timeout as well: +```js +// 5 seconds +await expect(page.getByTestId('checkbox')).toBeChecked({timeout?: 5000}); +``` + +An example of a non-retrying assertion is `isChecked`: + +```js +await expect(page.getByTestId('checkbox').isChecked()) +``` + +Sometimes there may not be a retrying assertion for what you need to check. +In that case, you can retry any custom async function until it passes using `toPass` ([docs](https://playwright.dev/docs/test-assertions#expecttopass)). + +```js +await expect(async () => { + const response = await page.request.get('https://api.example.com'); + expect(response.status()).toBe(200); +}).toPass(); +``` + +### Tip 2 - Don't rely on internal network calls to check if something is done +Internal network calls are not visible to the user, so they can be refactored whenever. +If we have tests that rely on a request to a given route finishing before moving on, for example, they will fail if we ever change the route name or some other implementation detail. + +It's much better to use a retrying assertion that targets a visible DOM element with a larger timeout to check if some work is done. + +Avoid this: +```js +const uploadButton = page... +await uploadButton.click(); +await page.waitForRequest("**/upload?*"); +await expect(page.getByTestId("file-component")).toHaveValue(...) +``` + +Do This: +```js +const uploadButton = page... +await uploadButton.click(); +await expect(page.getByTestId("file-component")).toHaveValue(..., {timeout?: 5000}); +``` + +### Tip 3 - Use the playwright trace viewer +Whenever a test fails locally, playwright will write out some details about the test to the `test-results` directory at the top level of the repo. + +You can view the trace using the following command: + +```bash +npx playwright show-trace test-results//trace.zip +``` + +You can see a "video" of the failing test, a screenshot of when it failed, as well as all the network calls and console messages. + +![local_trace_viewer](https://github.com/gradio-app/gradio/assets/41651716/31ed5fa8-e1d9-43a0-9757-469905678683) + +If a test fails on CI, you can obtain the same trace by downloading the artifact from github actions. + +1. From the failing Github Actions page, go to the `Summary` page +2. Scroll down to the bottom to where it says `Artifacts` +3. Click on `playwright-screenshots` to download a zip archive. +4. Unzip it and use the `show-trace` command. + +![download_trace](https://github.com/gradio-app/gradio/assets/41651716/20c279a8-9a56-4dcf-8df0-c4711e305515) + +### Tip 4 - Playwright can write the test for you + +You can write the basic skeleton of the test automatically by just interacting with the UI! + +First, start a gradio demo from the command line. Then use the following command and point it to the URL of the running demo: + +```bash +npx playwright codegen +``` + +This will open up a Chromium session where each interaction with the page will be converted into a playwright accessor. + +NOTE: Only copy the `test("test-name", ....)` not the imports. For playwright to work when running in the gradio CI, `test` and `expect` need to be imported from `@gradio/tootils`. + +![code_gen_demo](https://github.com/gradio-app/gradio/assets/41651716/96003fba-d17c-46b9-9c6d-35218fbdfb6f) \ No newline at end of file