Skip to content

Usage

To run the Model Validation pipeline, invoke the "Model Validation Run" GitHub workflow. You can configure it with the following inputs:

  • environment: The Marqo Cloud environment to run the model validation on. Choose from development or staging. With development, the config will be driven more by the ENV_BRANCH environment variable, while staging uses a known configuration (relying only on API_KEY).
  • inference_type: The type of inference to run the model on. Choose from cpu (aka CPU.small), cpu-large (aka CPU) or gpu (aka GPU).
  • models: A comma-separated list of model names to test. If not provided, all models in the registry will be tested.
  • only_failed_models: Only run the tests for models that have previously failed.
  • create_new_index: Create a new index for each model instead of reusing an existing index.

Environments

Typically the model validation pipeline is run on the staging environment. This is pre-configured and requires no additional inputs.

However there may be times you want to run the pipeline somewhere else (e.g. staging is broken; staging is overloaded; you want to test against features or infrastructure not yet in staging). In this case, select the dev environment option, and provide the additional inputs as needed:

  • dev_name: The suffix (or prefix) used when creating the environment. By default, this is your username (without the leading or trailing -).
  • dev_system_account_id: The system account ID for the test account that has been set up in the dev cell.
  • dev_api_key: The API key of the test account that has been set up in the dev cell.