Troubleshoot runs from Git

Start with what the run used: the Definition: line the CLI printed, or the Test definition row under Info in the run's report. Both name where each test came from and why it fell back, as Test definitions from Git describes.

When an error is in one file, revyl test run and revyl workflow run start its message with the file's path, followed by the line when the error names one:

Error: .revyl/tests/<test-name>.yaml:30: Module '<module-name>' is not in the files or in the organization

An API call that starts a run gets the same error as JSON with code and message, plus path and line when the error is in one file:

{"code":"module_not_found","message":"Module '<module-name>' is not in the files or in the organization","path":".revyl/tests/<test-name>.yaml","line":30}

run_test_definitions_from_git is not a supported fieldCopy Anchor Link

Error: Invalid configuration
  <repository>/.revyl/config.yaml

  run_test_definitions_from_git is not a supported field

The CLI that read .revyl/config.yaml is version 0.1.217 or older, and the file sets run_test_definitions_from_git.

Every Revyl CLI that reads .revyl/config.yaml in the repository, in CI and on developers' machines, must be version 0.1.234 or later while the file sets run_test_definitions_from_git, with any value. What an older CLI does there depends on its version:

• Revyl CLI 0.1.217 and older stop with run_test_definitions_from_git is not a supported field when they read the file.
• Revyl CLI 0.1.218 to 0.1.233 run tests without sending the checkout's files, as --from-checkout=false does. When a test's file sets variables, their revyl test push deletes the test's saved secrets and saves the file's variables as plain text.

Run revyl upgrade to update a CLI. The Revyl GitHub Actions install the latest CLI unless their cli-version input pins an older one; with an older pin, the step stops before the run starts.

To fix it, run revyl upgrade, or raise the cli-version your CI pins. On a branch created before you turned Run tests from Git off, merge your default branch into the branch instead, as Open branches after you turn it off describes.

To confirm the fix, run revyl config validate. It prints ✓ Configuration is valid locally.

The run_test_definitions_from_git key in .revyl/config.yaml needs Revyl CLI 0.1.234 or newerCopy Anchor Link

A Revyl GitHub Action step fails before the run starts, with this error and error_message output:

The run_test_definitions_from_git key in .revyl/config.yaml needs Revyl CLI 0.1.234 or newer (found 0.1.210). Set cli-version to 0.1.234 or newer, or remove the key.

The step's cli-version input pins a CLI older than 0.1.234, and the .revyl/config.yaml the CLI reads sets run_test_definitions_from_git, with any value. That file is the first one from the folder in the step's working-directory input, or the workspace root when the input is unset, up to the repository root, and the message names its path from the repository root, for example <project-root>/.revyl/config.yaml for a project in a subfolder. The from-checkout input fails the same way:

The from-checkout input needs Revyl CLI 0.1.234 or newer (found 0.1.210).

To fix it:

1. Set the step's cli-version to 0.1.234 or later, or remove the input so the step installs the latest CLI.
2. On a branch created before you turned Run tests from Git off, merge your default branch into the branch instead, as Open branches after you turn it off describes.

To confirm the fix, rerun the job. The step starts the run.

Run tests from Git is off for this project, but this checkout's .revyl/config.yaml turns it onCopy Anchor Link

revyl config validate prints:

⚠ Run tests from Git is off for this project, but this checkout's .revyl/config.yaml turns it on, so CLI runs from this checkout send its files. To match, remove `run_test_definitions_from_git: true` from .revyl/config.yaml, or run `revyl config push` to turn it on.

For a project whose configuration is managed from the default branch, the line ends with the merge that makes them match instead: To match, merge this branch into the default branch, which turns it on for the project. on the branch that adds the line, and To match, merge the default branch into this branch. on a branch created before you turned the setting off. In a checkout that can't tell which branch changed the setting, such as a shallow clone, it names both merges. revyl test run prints the same kind of line after Definition:, saying that this run sent the checkout's files.

The checkout's .revyl/config.yaml sets run_test_definitions_from_git: true, but the project's published setting is off, so CLI and CI runs from this checkout send its files anyway. This happens on a branch created before you turned the setting off, after you turned it off in the dashboard without removing the line, and while you turn it on, before the setting is published. To fix it:

• If you turned the setting off, remove the line from .revyl/config.yaml and commit the change. On a branch created before you turned it off, merging your default branch into the branch removes it.
• If you're turning the setting on, merge the branch into your default branch, then check that the merge published the setting, as step 7 of Turn on Run tests from Git describes.

To confirm the fix, run revyl config validate. It prints ✓ Published: Revyl runs this exact configuration and no line that starts ⚠ Run tests from Git is.

Git ignores a pulled module or script fileCopy Anchor Link

The CLI warns when a module or script file it writes is hidden from Git:

⚠ Git ignores .revyl/modules/<module-name>.yaml because of .gitignore:12:modules/. Remove that rule, or add `!.revyl/modules/` and `!.revyl/modules/**` after it, then run git add -A.

The file matches an ignore rule, so git add -A cannot stage it. An older Revyl CLI also wrote a .revyl/.gitignore that ignored modules and scripts. The current CLI updates that exact Revyl-owned file automatically and prints:

Updated .revyl/.gitignore so modules and scripts can be committed with git add -A.

To fix a different ignore rule:

1. Apply the negation rules named in the warning after the matching rule, or remove that rule. Do not replace an ignore file that you wrote yourself.

2. Stage the definition files:

   Copy Text

   git add -A
   git status --short

3. Confirm that .revyl/modules/ and .revyl/scripts/ files appear in the staged changes.

If a checkout run sends a definition file that Git ignores, the CLI names the file and marks the run as having uncommitted definition changes. Fix the ignore rule and stage the file before committing.

Run tests from Git is on for this project, but this checkout's .revyl/config.yaml doesn't turn it onCopy Anchor Link

revyl config validate prints:

⚠ Run tests from Git is on for this project, but this checkout's .revyl/config.yaml doesn't turn it on, so CLI runs from this checkout don't send its files. To match, run `revyl config push` to turn it off, or add `run_test_definitions_from_git: true` to .revyl/config.yaml.

For a project whose configuration is managed from the default branch, the line ends with the merge that makes them match instead: To match, merge this branch into the default branch, which turns it off for the project. on the branch that removes the line, and To match, merge the default branch into this branch. on a branch created before you turned the setting on. In a checkout that can't tell which branch changed the setting, such as a shallow clone, it names both merges. revyl test run prints the same kind of line after Definition:, saying that this run didn't send the checkout's files.

The project's published setting is on, but this checkout's .revyl/config.yaml doesn't set it, so CLI and CI runs from this checkout read the files from Git instead of sending the checkout's. This happens on a branch created before you turned the setting on, after you removed the line but before you published the change, and after you turned it on in the dashboard without adding the line. To fix it:

• If you're turning the setting off, merge the branch into your default branch or publish the change, as step 3 of Turn off Run tests from Git describes.
• If the setting stays on, add the line to .revyl/config.yaml and commit the change. On a branch created before you turned it on, merging your default branch into the branch adds it.

To confirm the fix, run revyl config validate. It prints ✓ Published: Revyl runs this exact configuration and no line that starts ⚠ Run tests from Git is.

A test ran its saved copyCopy Anchor Link

Definition: 1 test saved (no file in Git)

No file linked the test, so it ran the copy saved in Revyl. A file links a test when its _meta.remote_id is the test's ID and the project builds or receives uploads for the test's app. To fix it:

1. If the test has no file in .revyl/tests/, write one for every saved test on the test's app, where <app-name> is the app's name or ID from revyl app list:

   Copy Text

   revyl test pull --all --app <app-name>

   Copy Text

   Pulling tests...
   ✓ <test-name>: pulled v<version>

2. Commit the file on the branch you run on. Also commit it on the default branch, because runs on other branches fall back to the default branch's files.

3. If the file was already there, check that one of the project's build profiles sets the test's app as its app_id, or that its pull-request review takes your CI's uploads for that app.

To confirm the fix, run the test again. Its Definition: line says from Git with no (no file in Git).

test_definition_git_read_unavailableCopy Anchor Link

The run fails with this message, and the API returns it with the code test_definition_git_read_unavailable:

The test definition files in Git could not be read; retry the run

Revyl couldn't read the run's files from Git, so the run didn't start. It never runs the saved copy instead. The repository's GitHub or GitLab connection may be missing or have lost access to the repository, the provider may have failed or limited the read, or the files may be past the limits. The response's path names a file or folder that broke a limit. To fix it:

1. Retry the run.
2. If it fails again, run revyl github status and check that the repository shows access granted, or run revyl gitlab status and check for access: true. If it doesn't, reconnect it as GitHub or GitLab describes.
3. If the response names a path, bring that file or folder within the limits.

To confirm the fix, run the test again. It starts and prints a Definition: line.

Run tests from Git doesn't turn off while Revyl can't read the repositoryCopy Anchor Link

When Revyl can't read the repository, for example because the GitHub App lost access to it, runs that read their files from Git fail with this message, and the API returns it with the code test_definition_git_read_unavailable:

The test definition files in Git could not be read; retry the run

You can't turn Run tests from Git off until Revyl can read the repository again. revyl config push and the dashboard check Revyl's access to the repository before they change the setting, so they fail too. On a project whose configuration is managed from the default branch, Revyl turns the setting off only when it reads the merged change from the default branch. To fix it:

1. Restore Revyl's access to the repository. Run revyl github status and check that the repository shows access granted, or run revyl gitlab status and check for access: true. If it doesn't, reconnect it as GitHub or GitLab describes.
2. Turn it off as Turn off Run tests from Git describes. If you merged the change that removes the line while Revyl couldn't read the repository, Revyl turns the setting off at the next push to the default branch.

To confirm the fix, run revyl config validate in an up-to-date checkout of your default branch. It prints ✓ Published: Revyl runs this exact configuration.

Module '<module-name>' is not in the files or in the organizationCopy Anchor Link

Error: .revyl/tests/<test-name>.yaml:30: Module '<module-name>' is not in the files or in the organization

The test names a module that has no file in .revyl/modules/ in the same checkout or commit, and no saved module with that name. The error starts with the file and line of the reference. Scripts fail the same way:

Error: .revyl/tests/<test-name>.yaml:27: Script '<script-name>' is not in the files or in the organization

The API returns module_not_found or script_not_found. To fix it:

1. Open the file at the line the error starts with. That line holds the module: or script: reference.

2. List the modules or scripts saved in Revyl:

   Copy Text

   revyl module list
   revyl script list

   Each prints a table with a NAME column.

3. If the name in the reference is wrong, correct it. If the name is in the list, write its file with revyl module pull <module-name> or revyl script pull <script-name>. Otherwise, add its file to .revyl/modules/ or .revyl/scripts/.

4. For a run that reads Git, commit the change on the branch the run reads.

To confirm the fix, run the test again. It starts and prints a Definition: line.

MODULE_NOT_FOUND from revyl test pushCopy Anchor Link

✗ <project-root>/.revyl/tests/<test-name>.yaml:27 test.blocks[4].module - MODULE_NOT_FOUND: Module '<module-name>' was not found in this organization.
  Use one of: <saved-module-name>, <other-saved-module-name>
Error: YAML validation failed

The test names a module that has a file in .revyl/modules/ but no saved copy, and the CLI is older than 0.1.234, so revyl test push doesn't create it. A script that has only a file fails the same way, with SCRIPT_NOT_FOUND. To fix it:

1. Update the CLI:

   Copy Text

   revyl upgrade

2. Push the test again, where <test-name> is the test's name:

   Copy Text

   revyl test push <test-name>

   Before it saves the test, the push creates each script, then each module, that the test uses and that has only a file:

   Copy Text

   ✓ Created script <script-name> from .revyl/scripts/<script-name>.py
   ✓ Created module <module-name> from .revyl/modules/<module-name>.yaml as v1
   Pushing tests...
   
   ✓ <test-name>: pushed to v<version>

To confirm the fix, run revyl module list or revyl script list. The new module or script is in the NAME column.

Skipped module <module-name>: .revyl/modules/<module-name>.yaml differs from the saved copyCopy Anchor Link

⚠ Skipped module <module-name>: .revyl/modules/<module-name>.yaml differs from the saved copy; add --include-modules-and-scripts to update it

revyl test push creates a module or script that has a file but no saved copy, and updates a saved one only with --include-modules-and-scripts, because other saved tests may use it. The push still saved the test, and the saved test uses the saved module, not the file. A script shows the same warning with Skipped script. To fix it:

1. See what the push would update, and which other saved tests use it:

   Copy Text

   revyl test push <test-name> --force --include-modules-and-scripts --dry-run

   Copy Text

   Scripts and modules this push would save from files first:
     update module <module-name> from .revyl/modules/<module-name>.yaml (saved copy is v<version>)
   ⚠ Other saved tests use module <module-name> too: <other-test-name>

2. If those tests should run the file's version too, push with the flag:

   Copy Text

   revyl test push <test-name> --force --include-modules-and-scripts

   Copy Text

   ✓ Updated module <module-name> from .revyl/modules/<module-name>.yaml to v<version>
   ⚠ Other saved tests use module <module-name> too: <other-test-name>

To confirm the fix, run revyl test push <test-name> --dry-run. It lists no skip module line for that module.

both link test <test-id> (duplicate_test_definition_file)Copy Anchor Link

Error: .revyl/tests/<file-name>.yaml and .revyl/tests/<other-file-name>.yaml both link test <test-id> (duplicate_test_definition_file); keep one of them

Two test files have the same _meta.remote_id, so the run can't tell which one is the test. Two module or script files with the same name fail the same way, for example <script-name>.py and <script-name>.sh:

Error: .revyl/scripts/<script-name>.py and .revyl/scripts/<script-name>.sh both define "<script-name>"; keep one of them

A run that reads Git, such as one with --from-checkout=false, names one of the two files, and the API returns duplicate_test_definition_file with that file's path:

Error: .revyl/tests/<file-name>.yaml: More than one test definition file names this test in _meta.remote_id

The other test file has the same _meta.remote_id as the file the error names. It can be in another project that has the setting on and builds the same app, in this repository or another; Several apps or repositories says where each test's file goes. To fix it, delete the test file you don't need. For two module or script files, delete or rename one of them. For a run that reads Git, commit the change on the branch the run reads.

To confirm the fix, run the test again. It starts and prints a Definition: line.

failed to parse test fileCopy Anchor Link

Error: .revyl/tests/<test-name>.yaml: failed to parse test file: yaml: line 12: did not find expected ',' or ']'

The CLI couldn't parse the test file before the run. To fix it, correct the file at that line, then check it:

revyl test validate .revyl/tests/<test-name>.yaml

✓ Validated 1 YAML test file

invalid_test_definition_fileCopy Anchor Link

Error: .revyl/tests/<test-name>.yaml:25: download_file block must have either 'step_description' or 'file', not both

The file parses but breaks the test format, so the run doesn't start. The API returns the code invalid_test_definition_file, and the error starts with the file and line. To fix it:

1. Correct the file at that line.

2. Check it:

   Copy Text

   revyl test validate .revyl/tests/<test-name>.yaml

   Copy Text

   ✓ Validated 1 YAML test file

To confirm the fix, run the test again. It starts and prints a Definition: line.

build picks a build on Git branch "<branch>"Copy Anchor Link

Error: .revyl/tests/<test-name>.yaml:13: build picks a build on Git branch "<branch>", but the test's file there picks a build on another branch. Make both files pick the same branch, or pick the run's branch with --branch

The test's file on the run's branch picks a build on Git branch <branch>, so the run reads the test's file there, as Test definitions from Git describes. That file picks a build on a third branch, so no one branch gives the test both its build and its files, and the run doesn't start. When <branch> has no file for the test, the message says which has no file for this test instead. The error starts with the file and the line of its build, and the API returns the code invalid_test_definition_file. To fix it, do one of these:

• Make the test's files on both branches pick a build on the same branch, or add the test's file on <branch>, and commit the change on each branch.

• Pick the run's branch, so the run takes its build and its files from that branch:

  Copy Text

  revyl test run <test-name> --branch <branch>

To confirm the fix, run the test again. It starts and prints a Definition: line.

is <bytes> bytes; a test, module or script file can be at most 1 MiBCopy Anchor Link

Error: .revyl/scripts/<script-name>.py is <bytes> bytes; a test, module or script file can be at most 1 MiB

A file the run sends from the checkout is over 1 MiB. To fix it, make the file smaller, for example by moving large data out of a script.

over the 10 MiB a run can sendCopy Anchor Link

Error: the run's <count> test, module and script files total <bytes> bytes, over the 10 MiB a run can send; rerun with --from-checkout=false to run without sending them

The files the run sends from the checkout total more than 10 MiB. An API call that sends files past either size limit gets test_definition_files_too_large. To fix it, make the files smaller, or rerun with --from-checkout=false to run the files committed at the head of the run's branch in Git.

test_definition_git_project_limit_exceededCopy Anchor Link

The run fails with this message, and the API returns it with the code test_definition_git_project_limit_exceeded:

Too many projects run tests from Git in this organization

More than 256 projects in the organization have Run tests from Git on, so runs in the organization fail until fewer do. To fix it, remove run_test_definitions_from_git from the projects that don't need it, and publish each change the way you turned the setting on in Run tests from Git.