A/B Tests

These endpoints are to create, read, update, delete A/B tests parameters.

A/B tests are used for recommendation through a special type of scenarios: see A/B Test Scenarios.

To segment analytics using an A/B tests, see Analytics Segmentations. Resulting analytics will be using these segmentations; see Analytics (a.k.a. KPIs).

About the necessity of a User ID

An A/B test is supposed to run a test on users, thus a user_id is always expected when using an ab_test scenario, The core reason is that later on, analytics tasks will segment users the same way thanks to the same user_id’s. Any infringement of this principle may distort efferent analytics: if anonymous users logged in after receiving recommendations, their interactions will impact analytics reports, as analytics tasks perform calculations on all interactions. To protect the integrity of your analytics, the recommendation requests should raise an error.

Still, at times, it makes sense to request item_to_* or session_to_* recommendations under an A/B test and still expect recommendations, for example when the engineering cost of specifying another scenario at runtime or skip_default_scenario=True is non-negligible.

For this reason the API offers an optional A/B test parameter missing_user_id_rule to prescribe an action rule when the expected user_id goes missing, set at creation of the A/B test; see POST ab-tests/params/.

The existing rules are:

  • raise is best to keep the default behavior of raising an WrongData error;

  • scenario_a to always use scenario A;

  • scenario_b to always use scenario B;

  • random picks a scenario at random; scenario A is picked with probability_a, scenario B with probability 1-probability_a.

The three rules allowing recommendations (A/B/random) are equivalent as far as analytics are concerned: they will only reduce the significance and not add bias, merely adding noise. Abide by the one that best fits your use.

[Internal] List all A/B Tests Parameters

GET ab-tests/params/

Note

Authorized Roles: root, manager, backend

List all A/B tests parameters.

Response JSON Object

  • warnings (list-of-string) – Optional. List of warnings

  • ab_tests (list-of-object) –

    All A/B test parameters

    Inner fields

    • warnings (list-of-string) – List of warnings

    • id (string) – A/B test unique ID

    • name (string) – A/B test name

    • probability_a (float32) – [min: 1e-12 max: 0.999999999999] Probability of belonging to A group

    • missing_user_id_rule (enum) – choices (case insensitive): [scenario_a, scenario_b, raise, random] Rule to select a scenario when user_id is missing in item_to_* and session_to_* scenarios. More information in Missing User ID Rule.

EXAMPLE RESPONSE
  {
      "ab_tests": [
          {
              "id": "gw5456hftt",
              "name": "my_ab_test1",
              "probability_a": 0.66
          }
      ]
  }

[Internal] Create Parameters for an A/B Test

POST ab-tests/params/

Note

Authorized Roles: root, manager, backend

Create parameters for an A/B test.

Request JSON Object

  • name (string) – A/B test name

  • probability_a (float32) – Optional. [min: 1e-12 max: 0.999999999999] Probability of belonging to A group

  • missing_user_id_rule (enum) – Optional. choices (case insensitive): [scenario_a, scenario_b, raise, random] Rule to select a scenario when user_id is missing in item_to_* and session_to_* scenarios. More information in Missing User ID Rule.

EXAMPLE QUERY BODY
  {
      "name": "my_ab_test2",
      "probability_a": 0.66,
      "missing_user_id_rule": "random"
  }
Response JSON Object

  • warnings (list-of-string) – Optional. List of warnings

  • id (string) – A/B test unique ID

EXAMPLE RESPONSE
  {
      "id": "m43qfwe"
  }

Errors:

[Internal] Get Parameters of an A/B Test

GET ab-tests/params/<str:ab_test_id>/

Note

Authorized Roles: root, manager, backend

Get the parameters of an A/B test.

Response JSON Object

  • warnings (list-of-string) – Optional. List of warnings

  • id (string) – A/B test unique ID

  • name (string) – A/B test name

  • probability_a (float32) – Optional. [min: 1e-12 max: 0.999999999999] Probability of belonging to A group

  • missing_user_id_rule (enum) – Optional. choices (case insensitive): [scenario_a, scenario_b, raise, random] Rule to select a scenario when user_id is missing in item_to_* and session_to_* scenarios. More information in Missing User ID Rule.

EXAMPLE RESPONSE
  {
      "id": "gw5456hftt",
      "name": "my_ab_test1",
      "probability_a": 0.66
  }

Errors:

[Internal] Rename an A/B Test

PATCH ab-tests/params/<str:ab_test_id>/

Note

Authorized Roles: root, manager, backend

Rename an A/B test.

Request JSON Object

  • name (string) – A/B test new name

EXAMPLE QUERY BODY
  {
      "name": "my_ab_test1"
  }

Errors:

[Internal] Delete an A/B Test

DELETE ab-tests/params/<str:ab_test_id>/

Note

Authorized Roles: root, manager, backend

Delete an A/B test.

Cannot be done if a resource (analytics segmentation or scenarios) is using this A/B test.

Errors: