Test definitions from Git

A test definition is everything a run executes for one test: its steps and run settings, and the modules and scripts its steps name. When a project turns on Run tests from Git, each run takes its test definitions from the files in .revyl/tests/, .revyl/modules/ and .revyl/scripts/ instead of the copies saved in Revyl, and records which files it used. Run tests from Git turns it on.

Which files a run usesCopy Anchor Link

Which branch a run readsCopy Anchor Link

Each run has one branch, and it takes both its build and its files from it. The run's branch is the first of these that applies:

1. The branch chosen or saved for the run: --branch, Run on a branch, or the branch saved on the test or on the workflow's app override.
2. Your Git branch, when the app follows Git branches.
3. The app's default branch.

The app's default branch reads your repository's default branch, whatever it's called in Revyl. Any other branch reads the Git branch with the same name, so --branch qa reads Git branch qa. A run on an exact build reads the branch that build is on, and in a run on all branches each test reads the branch its own build is on.

In a run that reads Git, the build in a test's file takes the place of the test's saved build settings. When the test's file on the run's branch picks a build on another branch, the run reads the test's file on that branch instead, so the test's build and files still come from one branch. A file that picks a build on the app's default branch, such as an older build pinned by ID, keeps the run's branch, as when your Git branch has no build. If the file there picks a build on yet another branch, the run doesn't start, as Troubleshoot runs from Git shows. A run whose branch is chosen with --branch or Run on a branch, and a run that sends the checkout's files, read the files on their own branch or checkout.

The build and the files fall back to the default branch on their own. When your Git branch has no usable build, the run takes its build from the app's default branch, and a run that reads Git still reads its files from your Git branch. The CLI says both:

Git branch "<branch>" has no usable build; using the app's default branch "<default-branch>". The tests come from Git branch "<branch>".

When Git doesn't have the run's branch, the run reads the files on your repository's default branch, as Fallbacks shows.

Which files belong to a testCopy Anchor Link

A test file belongs to the saved test whose ID is in its _meta.remote_id. A run that reads Git reads the .yaml files directly inside .revyl/tests/ and .revyl/modules/ and the script files directly inside .revyl/scripts/, not subfolders. It reads them only for tests on an app the project builds or receives uploads for, and a test on any other app runs its saved copy.

Modules and scriptsCopy Anchor Link

A test file names its modules and scripts by name. A module named sign-in uses .revyl/modules/sign-in.yaml when that file is in the same checkout or commit, and the saved module otherwise. A script named make-email works the same way with .revyl/scripts/make-email.py, or .js, .ts or .sh for its runtime.

Variables and secretsCopy Anchor Link

When a test's file sets variables, a run that uses the file takes the test's plain-text variables from the file instead of from the saved test. Secret variables stay in Revyl, and a file never holds a secret's value: revyl test pull leaves the test's secrets out of the file, and revyl test push keeps them, in Revyl CLI 0.1.234 or later. When a test's file sets variables, the revyl test push of Revyl CLI 0.1.218 to 0.1.233 deletes the test's saved secrets and saves the file's variables as plain text, and Revyl CLI 0.1.217 and older stop in a checkout whose .revyl/config.yaml sets run_test_definitions_from_git. Every run uses the test's saved secrets, and a file variable with a secret's name is ignored. To add or change a secret, open the test in Revyl and choose Variables. When the test's file sets variables, that view lists the file's variables and, under Secrets, the test's saved secrets.

FallbacksCopy Anchor Link

When a run reads Git and the test's file isn't there, the run uses another copy and says why:

A run never falls back when Revyl can't read Git: it fails before it starts. Troubleshoot runs from Git lists the errors and their fixes.

Saved copiesCopy Anchor Link

The saved test, module and script stay in Revyl as the copies a run uses when Git has no file. With the setting on, revyl test push and the MCP tool update_test update only those saved copies, and both say so. Sync tests describes what revyl test push creates and updates. When you turn the setting off, runs use the saved copies again; Turn off Run tests from Git shows how to save your files' changes to them first.

What a run showsCopy Anchor Link

The CLI prints a Definition: line for each run of a project that runs tests from Git, even with --json. The terminal UI shows the same line under the run, and the MCP tools run_test and run_workflow return its text as test_definition_summary. A run that sent the checkout's files names the checkout:

Definition: Checkout @ <commit> · <branch> · uncommitted changes · 1 test, 1 module and 1 script from files

A run that read Git names the branch and commit it read, and a run that fell back to the saved test says so:

Definition: 1 test from Git branch <branch> @ <commit>

Definition: 1 test saved (no file in Git)

revyl test run --json adds test_definition_source (cli_upload, git_commit_read or saved_test) and test_definition_fallback_reason. A run that read Git also adds test_definition_git_branch_name and test_definition_git_commit_sha, plus test_definition_missing_git_branch_name when Git doesn't have the run's branch. revyl workflow run --json adds the source and reason counted across the workflow's tests, as test_count_by_definition_source and test_count_by_definition_fallback_reason, and the three Git fields when every test that read Git read the same branch and commit. run_test and run_workflow return the same fields, and list problems with the files they sent, such as a file Git ignores, in test_definition_warnings.

In a report, Info has a Test definition row with the repository, the file, the Git branch and commit, and any fallback reason. It opens the exact steps and run settings the run used. Editing a saved test, module or script later never changes what an earlier run shows.

In the dashboardCopy Anchor Link

• A test, module or script that every run reads from a file opens read-only. It shows Runs from with the file and its commit, and Edit in repo opens the file in your repository.
• A session you start in the test editor reads the test's file at your default branch's head when it starts. Start and every Run Steps in that session use that file's steps, modules, scripts, variables, device and location, even after your default branch moves, and the editor shows that file while the session runs. When the session ends, the editor shows the branch's head again.
• A module or script that other tests still run from its saved copy stays editable, with a notice that edits reach only those other tests.
• The run dialogs show Tests from Git branch with the branch the run's tests will be read from, or Tests from the repo's default branch when that branch is the app's default branch.

LimitsCopy Anchor Link

• A run that sends the checkout's files sends at most 10 MiB, and each test, module or script file it sends can be at most 1 MiB.
• A run that reads Git reads at most 16 MiB from the project's .revyl/tests/, .revyl/modules/ and .revyl/scripts/ at one commit, and each file there can be at most 1 MiB. Each test, module and script file there must be a regular file, not a symbolic link. Past these limits the run fails with test_definition_git_read_unavailable.
• When more than 256 projects in an organization have Run tests from Git on, runs in the organization fail with test_definition_git_project_limit_exceeded.

RelatedCopy Anchor Link

• Run tests from Git: turn it on.
• Troubleshoot runs from Git
• Run tests in CI
• Branch and build selection