Run Replay Cases in CI

Replay cases can run from any CI system that can make an authenticated HTTP request. This lets you turn real webhook payloads captured in Hooklistener into regression checks for staging, preview, or production-like webhook handlers.

The examples below use GitHub Actions, but the same API works in GitLab CI, CircleCI, Buildkite, Jenkins, or a local script.

What the CI job does

A typical CI workflow:

Deploys or starts a webhook handler that Hooklistener can reach.
Calls the Hooklistener Request Cases API with wait: true.
Fails the CI job when the case run reports failed or timeout.
Prints the case run report URL and aggregate assertion counts.

Use wait: true when you want CI to block until queued replays finish. The wait timeout defaults to 30 seconds and is capped at 120 seconds.

Required secrets

Create these secrets in your CI provider:

Secret	Description
`HOOKLISTENER_API_KEY`	Organization API key from Organization Settings > API Keys
`HOOKLISTENER_ENDPOINT_ID`	Debug endpoint ID that owns the saved cases
`HOOKLISTENER_TARGET_URL`	Public URL that should receive each replayed webhook

The target URL must be reachable from Hooklistener's servers. For hosted CI, use a deployed preview URL, staging URL, or tunnel URL rather than localhost.

GitHub Actions workflow

Save this as .github/workflows/hooklistener-replay-cases.yml in the application repo that owns your webhook handler:

name: Hooklistener replay cases

on:
  pull_request:
  workflow_dispatch:

jobs:
  replay-cases:
    runs-on: ubuntu-latest
    env:
      HOOKLISTENER_API_KEY: ${{ secrets.HOOKLISTENER_API_KEY }}
      HOOKLISTENER_ENDPOINT_ID: ${{ secrets.HOOKLISTENER_ENDPOINT_ID }}
      HOOKLISTENER_TARGET_URL: ${{ secrets.HOOKLISTENER_TARGET_URL }}
    steps:
      - name: Run saved replay cases
        shell: bash
        run: |
          set -euo pipefail

          response_file="$(mktemp)"
          request_body="$(jq -n \
            --arg target_url "$HOOKLISTENER_TARGET_URL" \
            '{
              target_url: $target_url,
              target_name: "GitHub Actions",
              wait: true,
              timeout_ms: 120000
            }')"

          status_code="$(
            curl --silent --show-error --location \
              --output "$response_file" \
              --write-out "%{http_code}" \
              --request POST "https://app.hooklistener.com/api/v1/endpoints/${HOOKLISTENER_ENDPOINT_ID}/cases/run" \
              --header "Authorization: Bearer ${HOOKLISTENER_API_KEY}" \
              --header "Content-Type: application/json" \
              --data "$request_body"
          )"

          cat "$response_file"
          echo

          if [[ "$status_code" -lt 200 || "$status_code" -ge 300 ]]; then
            echo "Hooklistener API returned HTTP $status_code"
            exit 1
          fi

          result_status="$(jq -r '.data.result_status // "unknown"' "$response_file")"
          report_url="$(jq -r '.data.case_suite_run_url // empty' "$response_file")"
          queued_count="$(jq -r '.data.queued_count // 0' "$response_file")"
          failed_count="$(jq -r '.data.failed_count // 0' "$response_file")"
          passed_count="$(jq -r '.data.passed_count // 0' "$response_file")"
          assertion_failed_count="$(jq -r '.data.assertion_failed_count // 0' "$response_file")"
          assertion_error_count="$(jq -r '.data.assertion_error_count // 0' "$response_file")"

          echo "Hooklistener result: $result_status"
          echo "Queued cases: $queued_count"
          echo "Queue failures: $failed_count"
          echo "Passed assertions: $passed_count"
          echo "Failed assertions: $assertion_failed_count"
          echo "Assertion errors: $assertion_error_count"

          if [[ -n "$report_url" ]]; then
            echo "Report: https://app.hooklistener.com${report_url}"
          fi

          case "$result_status" in
            passed|completed)
              exit 0
              ;;
            failed|timeout)
              exit 1
              ;;
            *)
              echo "Unexpected Hooklistener result_status: $result_status"
              exit 1
              ;;
          esac

completed means all deliveries finished but no replay assertions were configured. Add assertions to your saved cases when you want CI to verify response status or response body fields.

Run one named suite

To run only a named suite, add a HOOKLISTENER_CASE_SUITE_ID secret:

env:
  HOOKLISTENER_API_KEY: ${{ secrets.HOOKLISTENER_API_KEY }}
  HOOKLISTENER_ENDPOINT_ID: ${{ secrets.HOOKLISTENER_ENDPOINT_ID }}
  HOOKLISTENER_CASE_SUITE_ID: ${{ secrets.HOOKLISTENER_CASE_SUITE_ID }}
  HOOKLISTENER_TARGET_URL: ${{ secrets.HOOKLISTENER_TARGET_URL }}

Then replace the request_body assignment in the workflow with:

request_body="$(jq -n \
  --arg case_suite_id "$HOOKLISTENER_CASE_SUITE_ID" \
  --arg target_url "$HOOKLISTENER_TARGET_URL" \
  '{
    case_suite_id: $case_suite_id,
    target_url: $target_url,
    target_name: "GitHub Actions",
    wait: true,
    timeout_ms: 120000
  }')"

Use suites for focused checks such as Billing webhooks, GitHub issue lifecycle, or Shopify order events.

Use a saved replay target

If your endpoint already has a saved replay target, pass target_id instead of target_url:

request_body="$(jq -n \
  --arg target_id "$HOOKLISTENER_TARGET_ID" \
  '{
    target_id: $target_id,
    wait: true,
    timeout_ms: 120000
  }')"

curl --request POST "https://app.hooklistener.com/api/v1/endpoints/${HOOKLISTENER_ENDPOINT_ID}/cases/run" \
  --header "Authorization: Bearer ${HOOKLISTENER_API_KEY}" \
  --header "Content-Type: application/json" \
  --data "$request_body"

Saved targets are useful when the same staging or preview URL is reused across workflows.

Generic shell script

For CI systems that do not use GitHub Actions, the core command is:

request_body="$(jq -n \
  --arg target_url "$HOOKLISTENER_TARGET_URL" \
  '{
    target_url: $target_url,
    wait: true,
    timeout_ms: 120000
  }')"

response="$(
  curl --fail-with-body --silent --show-error \
    --request POST "https://app.hooklistener.com/api/v1/endpoints/${HOOKLISTENER_ENDPOINT_ID}/cases/run" \
    --header "Authorization: Bearer ${HOOKLISTENER_API_KEY}" \
    --header "Content-Type: application/json" \
    --data "$request_body"
)"

result_status="$(jq -r '.data.result_status' <<<"$response")"

case "$result_status" in
  passed|completed)
    exit 0
    ;;
  failed|timeout)
    exit 1
    ;;
  *)
    echo "$response"
    exit 1
    ;;
esac

Troubleshooting

The run returns No saved cases exist for this endpoint - save at least one captured request as a replay case before running CI.

The job passes with result_status: completed but does not verify anything - the cases delivered successfully, but no assertions were configured. Add expected status codes or expected JSON body subsets to the saved cases.

The run returns timeout - the cases were queued, but not all forwards completed before timeout_ms. Increase the timeout up to 120000, or inspect the report URL to see which forward is still waiting.

The target never receives a request - make sure HOOKLISTENER_TARGET_URL is publicly reachable from Hooklistener. A local development URL such as http://localhost:3000/webhooks will only work when replaying through an active CLI listener, not from hosted CI.

The API returns 401 or 403 - verify that HOOKLISTENER_API_KEY starts with hklst_, belongs to the same organization as the endpoint, and has not been revoked.

What the CI job does​

Required secrets​

GitHub Actions workflow​

Run one named suite​

Use a saved replay target​

Generic shell script​

Troubleshooting​

Related docs​