Skip to content

Enable ETL pipeline to push automated updates to an index

This is a document for solutions engineers to enable ETL pipeline with automatic score modifier updates when onboarding a new customer. This will send updates to a chosen customer index.

Before running this, pixel must already be set up and sending data to the appropriate customer’s S3 bucket in pixel-ingestion-events-production. Link to bucket here

Steps to Create Customer Mapping

  1. Grant Administrator Access:
    • Use Escalator to obtain admin access.
    • Account (pixel prod): 761200394628
  2. Create Mapping via Lambda Function:
    • Use the CreateCustomerDataLambda-prod function.
    • Provide the following input format:
      • Notes:
        • To turn on automatic updates, set automatic_updates_top_queries_field and/or automatic_updates_score_modifier_fields depending on what fields in the index need to be updated. WARNING: THIS WILL UPDATE LIVE CUSTOMER INDEX DATA. If you want to test out the generated journey/score modifiers data first without affecting a customer index, set these to false then inspect generated files in marqo-analytics-data-bucket-prod.
        • automatic_updates_frequency is in hours (24 is the default).
{
  "action": "write",
  "data": {
    "pixel_account_id": "<PIXEL_ACCOUNT_ID>",
    "system_account_id": "<SYSTEM_ACCOUNT_ID>",
    "pixel_document_type": "marqo",
    "index_name": "<INDEX_NAME>",
    "automatic_updates_frequency": 24,
    "automatic_updates_top_queries_field": False,
    "automatic_updates_score_modifier_fields": False
  }
}
  1. Verification:
    • After running the Lambda, querying the Analytics API should return valid results.

⚠️ Warning: Do not map customer Pixel Account IDs to other System Account IDs. The score modifier logic assumes a many-to-one relationship between these IDs. Incorrect mappings may break score modifier updates.

3. Enabling Automatic Score Modifier Updates

To enable automatic score modifier updates, you must create or update the customer entry with automatic_updates set to true.

Option 1: Create via Lambda

  • Use the CreateCustomerDataLambda-prod function as described in section 2, but set:
"automatic_updates_top_queries_field": True,
"automatic_updates_score_modifier_fields": True

Option 2: Update via DynamoDB

  1. Open the DynamoDB console.
  2. Locate the customer entry (by pixel_account_id).
  3. Modify the automatic_updates_top_queries_field and automatic_updates_score_modifier_fields fields from false → true.

The row should now look like this:

image.png

Note: The automatic_updates parameter is deprecated, and can be set to anything.

After enabling, the ETL pipeline will include an additional step to perform score modifier updates automatically.

etlpipeline #analytics

Before JS pixel setup:

  1. Run the Onboard Customer and Manage Customer Infrastructure GH Actions in the pixel repo, as described in this runbook.