Skip to content

Testing

Testing ensures that our code works as expected, and continues to do so as we make changes.

Running Tests

In order for tests to be valuable, they must be pleasant to read, write, and run. Unpleasant tests will generally be ignored, grow rusty, and create friction, driving development away from the code under test and creating a false sense of security.

We use Pants to run tests, and leverage its features where possible:

  • Dependency inference and caching: Pants will only run the tests that are affected by the changes you've made.
  • Filesystem watching: Pants will re-run tests when you save a file, speeding up the feedback loop. This makes TDD a breeze.
  • Parallelism: Pants will run tests in parallel, speeding up the test suite.

The structure of the repository is also designed to support clean and rapid testing:

  • service_utils provides shared code logic for services and utilities for testing, making all services and their tests consistent.
  • Liberal use of fixtures to reduce boilerplate while setting up non-trivial test environments.
  • Pants aliases provide canned commands to run for common testing scenarios.

Test Types

EDR-26 Testing Approaches summarises the types of testing we use and expect at Marqo. This section provides more specifics on testing for the Control Plane.

It's worth noting that pytest and Pants are used to run all tests, regardless of their type.

Unit Tests

Unit tests typically involve testing a single function or class in isolation, mocking all immediate dependencies (with exceptions for pure functions). They are very fast and should be run often during development and in CI.

Integration Tests

Integration tests verify entire services work by calling their APIs and mocking their network dependencies (DBs and other APIs). They are relatively fast and should be run often during development and in CI.

End-to-End Tests

End-to-end tests stand up a full AWS environment and call public-facing APIs. They are slow to run, so they mostly assume that all components are correct, and verify that they are wired together correctly and respect the contracts between them.

End-to-end tests include both API calls and UI automation. We use the end-to-end test suite as a smoke test to verify deployments. test@marqo.ai is used as the test user for SSO sign in, and the service account (controller-ui-test) for LambdaTest.

Common Testing Commands

Note: See pants.toml for the full list of aliases and more detailed descriptions.

To constantly run all Control Plane microservice tests during development:

PANTS_CONCURRENT=True pants test services --loop

Console UI Tests (Playwright)

The console UI tests are located in tests/console/ and use Playwright with TypeScript.

cd tests/console
npm install
npm test                 # Run all tests
npm run test:headed      # Run with browser visible
npm run test:ui          # Run with Playwright UI
npm run test:signup      # Run signup tests only
npm run test:data-plane  # Run data plane tests only

Legacy UI Tests (Selenium - Deprecated)

The legacy Selenium tests in components/ui_tests are deprecated. Use the Playwright tests in tests/console/ instead.

# Legacy command (deprecated)
PANTS_CONCURRENT=True pants test components/ui_tests --local --no-watch