Running Tests - Revyl Documentation

Run a Single Test

Execute a test by name or ID:

revyl test run login-flow

The CLI will:

Resolve the test name (from aliases or API)
Queue the test for execution
Stream progress to your terminal
Report the final result and, for a blocking human-terminal run, open its report

Example Output

Running test: login-flow
Platform: android

Step 1/5: Tap the login button ✓
Step 2/5: Enter email address ✓
Step 3/5: Enter password ✓
Step 4/5: Tap Submit ✓
Step 5/5: Dashboard is visible ✓

Build: v1.2.3 from branch main
✓ Test passed in 45s
Report: https://app.revyl.ai/report/abc123

Run Options

Flag	Description
`--branch <name>`	Run on this branch, for this run only
`--build-selection inherit\|latest`	Automatic (the branch’s default build) or Latest build (its newest, even when pinned)
`--all-branches`	Run the newest build across all of the app’s branches
`--build-id <id>`	Run one exact build by ID
`--retries <n>`	Number of retry attempts (default: 1)
`--timeout <seconds>`	Maximum execution time (default: 3600)
`--no-wait`	Queue test and exit immediately
`--open`	Explicitly request report opening for a supported blocking human-terminal run
`--no-open`	Do not open the completed report
`--json`	Output results as JSON
`--verbose`	Show detailed execution logs
`--launch-var <key-or-id>`	Apply a stored organization launch variable (repeatable)
`--launch-env KEY=VALUE`	Apply an inline launch variable (repeatable)
`--launch-arg-set <name-or-id>`	Apply a stored ordered iOS argument set (repeatable)
`--launch-arg=<token>`	Append one exact, non-secret iOS argument token (repeatable)
`--no-inherited-launch-vars`	Ignore `REVYL_INHERITED_LAUNCH_ENV_VAR_IDS` entirely; explicit launch variables still apply

Blocking human-terminal runs open completed reports by default. --no-open, --no-wait, --json, --github-actions, CI, SSH, and other headless executions suppress browser opening. Stored argument sets are flattened in attachment and flag order, followed by inline --launch-arg tokens in flag order. Explicit arguments precede iOS -KEY value pairs generated from environment configurations. Inline tokens are non-secret and can appear in shell history, process arguments, and reports. Android-only test runs reject iOS arguments with an actionable error.

Run a Workflow

Execute a workflow (multiple tests). To build, upload, and run in one command:

revyl workflow run smoke-tests --build

This builds your app, uploads it, then runs all tests in the workflow. Without --build, the workflow runs on its saved builds:

revyl workflow run smoke-tests

Example Output

Running workflow: smoke-tests
Tests: 5

[1/5] login-flow ✓ (32s)
[2/5] browse-products ✓ (28s)
[3/5] add-to-cart ✓ (41s)
[4/5] checkout ✗ (55s) - Validation failed
[5/5] logout ✓ (12s)

Results: 4 passed, 1 failed
Total time: 2m 48s
Report: https://app.revyl.ai/workflow-report/xyz789

Workflow Options

Launch-configuration overrides apply to every compatible child test in the workflow. Mixed workflows apply iOS argument sets to iOS children and skip them for Android children.

Flag	Description
`--github-actions`	Format output for GitHub Actions
`--branch <name>`	Run every app on this branch, for this run only
`--build-selection inherit\|latest`	Automatic or Latest build for every app
`--all-branches`	Run the newest build across all branches of each app
`--ios-app <id>` / `--android-app <id>`	Replace the iOS or Android app of every test
`--ios-build-id <id>` / `--android-build-id <id>`	Run one exact build by ID (needs the matching app flag)
`--ios-build <label>` / `--android-build <label>`	Run one exact build by version label (needs the matching app flag)
`--launch-var <key-or-id>`	Apply a stored environment configuration to every child (repeatable)
`--launch-env KEY=VALUE`	Apply a non-secret inline environment value (repeatable)
`--launch-arg-set <name-or-id>`	Apply a stored iOS argument set to iOS children (repeatable)
`--launch-arg=<token>`	Append a non-secret inline iOS token to iOS children (repeatable)
`--no-inherited-launch-vars`	Ignore `REVYL_INHERITED_LAUNCH_ENV_VAR_IDS` entirely; explicit launch variables still apply
`--open`	Explicitly request report opening for a supported blocking human-terminal run
`--no-open`	Do not open the completed report
`--no-wait`	Queue the workflow and exit without opening a report

Branch and build selection

A run uses the saved choice (the test’s, or the workflow’s App Override) unless you pass one for the run. On Default branch it tries your Git branch first. See How a run picks its build. The run prints its build and branch, such as Build: v1.4 from branch main, and a notice when it couldn’t use your Git branch. Tests created in the dashboard start on All branches · Latest build, so a pull request job can run another pull request’s upload. Pass the ID of the build the job uploaded, or save the test on Default branch so it runs on the pull request’s branch: revyl test build set <test> --build-selection inherit. See New saved choices. revyl test build set <test> saves a test’s choice, and revyl test build clear <test> resets it to All branches · Latest build. For how each run option behaves, including --build and per-platform overrides on workflows, see Run options.

Full Pipeline: Build → Upload → Test

Use the --build flag with revyl test run to run the complete pipeline:

revyl test run login-flow --build --platform android

This executes:

Build - Run your build command
Upload - Upload the artifact to Revyl
Test - Run the test against the new build

Pipeline Options

Flag	Description
`--build`	Build and upload before running
`--profile <name>`	Named local build profile (requires `--build`)
`--platform ios\|android`	Platform recipe inside the selected profile (requires `--build`)
`--retries <n>`	Retry attempts for the test
`--no-wait`	Don’t wait for test completion
`--open`	Explicitly request report opening for a supported blocking human-terminal run
`--no-open`	Do not open the completed report
`--timeout <seconds>`	Test timeout
`--json`	JSON output

Example

# Build Android, upload, and run test
revyl test run checkout-flow --build --platform android

# Run test without building (against existing build)
revyl test run checkout-flow

The CLI validates that the test exists before starting the build process. If you specify a test name that doesn’t exist, you’ll get an immediate error with a list of available tests.

Full Pipeline: Build → Upload → Workflow

To build and run a workflow in one command:

revyl workflow run smoke-tests --build
revyl workflow run smoke-tests --build --platform android

This executes:

Build - Run your build command
Upload - Upload the artifact to Revyl
Workflow - Run all tests in the workflow against the new build

Leave out --build to run the workflow without rebuilding.

Workflow pipeline options

Flag	Description
`--build`	Build and upload before running workflow
`--profile <name>`	Named local build profile (requires `--build`)
`--platform ios\|android`	Platform recipe inside the selected profile (requires `--build`)
`--retries <n>`	Retry attempts for each test
`--no-wait`	Don’t wait for workflow completion
`--open`	Explicitly request report opening for a supported blocking human-terminal run
`--no-open`	Do not open the completed report
`--timeout <seconds>`	Workflow timeout
`--json`	JSON output

Build or Upload Without Running Tests

Upload a build without running tests:

revyl build --profile release --platform android

This builds in the cloud. Test/workflow --build remains local. See Build Configuration.

Upload Options

Flag	Description
`--profile <name>`	Named local build profile to use
`--platform ios\|android`	Platform recipe inside the selected profile
`--version <string>`	Version label (default: `<git-branch-slug>-YYYYMMDD-HHMMSS`, or the timestamp alone on a detached checkout, as in most CI jobs); doesn’t choose the branch
`--branch <name>`	Branch to upload to; without it, your Git branch (created if needed), otherwise the app’s default branch
`--pin`	Pin the branch to the new build. Uploads don’t pin otherwise; see Build branches.
`--json`	JSON output

Upload Output

revyl build upload --file ./app-release.apk --app <id> ends with the build and the branch it landed on:

Upload complete!
App:             <app-id>
Build Version:   feature-new-login-20260227-153000
Build ID:        <build-id>
Branch:          feature/new-login (Latest build: runs use this upload)
Package ID:      com.example.app

Build Version is the version label; Build ID identifies the build.

Default debug builds from every framework satisfy Revyl’s artifact rules. For release builds, .ipa archives, or narrowed abiFilters, see Build Artifact Requirements.

List Builds

View uploaded build versions across all branches of an app:

revyl build list --app "My App"

Options:

Flag	Description
`--platform <ios\|android>`	Filter the app list by platform (without `--app`)
`--app <name-or-id>`	App whose builds to list
`--branch <name>`	List one Revyl branch (default: all branches). If no Revyl branch has that name, lists builds built from a Git branch of that name. `HEAD` means your checked-out Git branch.
`--all-branches`	All branches (the default)
`--page`, `--page-size`	Paginate build versions
`--json`	JSON output

Without --app, the command lists app summaries. Cross-branch build tables show both GIT BRANCH and REVYL BRANCH (the dashboard’s Branch column). See Build branches from the CLI for filters and JSON fields.

Exit Codes

The CLI uses standard exit codes for scripting:

Code	Meaning
`0`	Test/workflow passed
`1`	Test/workflow failed or error

CI/CD Example

#!/bin/bash
revyl workflow run smoke-tests

if [ $? -eq 0 ]; then
  echo "All tests passed!"
  ./deploy.sh
else
  echo "Tests failed, aborting deployment"
  exit 1
fi

JSON Output

For programmatic use, get structured JSON output:

revyl test run login-flow --json

{
  "success": true,
  "task_id": "task_abc123",
  "test_id": "test_abc123",
  "test_name": "login-flow",
  "status": "completed",
  "report_link": "https://app.revyl.ai/report/abc123",
  "duration": "45.0s",
  "error": "",
  "branch_name": "main",
  "build_version": "1.4.2",
  "build_id": "build_abc123",
  "branch_source": "default_branch",
  "git_branch_name": "feature-login"
}

branch_name, build_version, and build_id name the build the run used. branch_source says where its branch came from: git_branch, explicit_branch, or default_branch. git_branch_name names the Git branch the run consulted, so default_branch with a git_branch_name, as above, means your Git branch couldn’t be used. See JSON fields. revyl workflow run --json reports these fields for each entry in tests.

Open Results in Browser

Blocking human-terminal test and workflow runs open the completed report by default. To state the intent explicitly:

revyl test run login-flow --open

Suppress browser opening with --no-open. No-wait, JSON, GitHub Actions, CI, SSH, and other headless runs never open a browser. Or open a previous report:

revyl test open login-flow

Background Execution

Queue a test without waiting for results:

revyl test run login-flow --no-wait

Output:

Test queued: login-flow
Task ID: task_abc123
Track progress: https://app.revyl.ai/task/abc123

Common Workflows

Quick Smoke Test

# Build, upload, and run workflow
revyl workflow run smoke-tests --build

# Or run workflow without rebuilding
revyl workflow run smoke-tests

Test New Changes

# Build, upload, and test
revyl test run login-flow --build --platform android

CI/CD Pipeline

# In your CI script
export REVYL_API_KEY=${{ secrets.REVYL_API_KEY }}

# Build, upload, and run the workflow on that build; fail the pipeline if tests fail
revyl workflow run regression --build --profile release --platform android --retries 2

Local Development Loop

# Make code changes
# ...

# Quick test without rebuilding
revyl test run login-flow

# Full rebuild and test
revyl test run login-flow --build --platform android

Troubleshooting

Cancelling a Running Test

Use revyl test cancel or revyl workflow cancel to request cancellation of a running test or workflow:

# Cancel a running test
revyl test cancel <task_id>

# Request cancellation of a workflow and its tests
revyl workflow cancel <task_id>

Where to find the task ID:

CLI output when starting a test: Task ID: abc123-def456...
Report URL: https://app.revyl.ai/tests/report?taskId=abc123...

A successful command can acknowledge a durable cancellation request while execution is still stopping. Check the returned test status or the workflow’s child statuses for terminal completion. Retrying a stop preserves the session’s existing terminal outcome and can resume pending cleanup.

Pressing Ctrl+C only stops the CLI from monitoring - the test continues running on the server. Use revyl test cancel <task_id> to actually stop the test.

Test Times Out

Increase timeout: --timeout 900
Check device availability in Revyl dashboard
Verify the test runs manually in the web UI

Build Upload Fails

Check your build command produces output at the expected path
Verify file permissions on the artifact
Try --dry-run to see what would be uploaded

”Test not found” Error

Run revyl test list to see available tests
Check spelling and case sensitivity
Ensure the test exists in your config.yaml aliases or on remote

​Run a Single Test

​Example Output

​Run Options

​Run a Workflow

​Example Output

​Workflow Options

​Branch and build selection

​Full Pipeline: Build → Upload → Test

​Pipeline Options

​Example

​Full Pipeline: Build → Upload → Workflow

​Workflow pipeline options

​Build or Upload Without Running Tests

​Upload Options

​Upload Output

​List Builds

​Exit Codes

​CI/CD Example

​JSON Output

​Open Results in Browser

​Background Execution

​Common Workflows

​Quick Smoke Test

​Test New Changes

​CI/CD Pipeline

​Local Development Loop

​Troubleshooting

​Cancelling a Running Test

​Test Times Out

​Build Upload Fails

​”Test not found” Error

Run a Single Test

Example Output

Run Options

Run a Workflow

Example Output

Workflow Options

Branch and build selection

Full Pipeline: Build → Upload → Test

Pipeline Options

Example

Full Pipeline: Build → Upload → Workflow

Workflow pipeline options

Build or Upload Without Running Tests

Upload Options

Upload Output

List Builds

Exit Codes

CI/CD Example

JSON Output

Open Results in Browser

Background Execution

Common Workflows

Quick Smoke Test

Test New Changes

CI/CD Pipeline

Local Development Loop

Troubleshooting

Cancelling a Running Test

Test Times Out

Build Upload Fails

”Test not found” Error