LLM Schema Reference

# Revyl YAML Test Schema - LLM Reference

## Purpose

This document provides a structured, machine-readable reference for generating Revyl YAML tests programmatically.

## Recommended Approach for AI Test Generation

Before writing test YAML, **read the app's source code** to understand:

1. **Screen structure** — What screens exist and their navigation flow
2. **UI elements** — Actual button labels, text content, input fields
3. **State transitions** — Loading states, animations, timeouts
4. **User-facing outcomes** — What the user sees when actions succeed or fail

This produces better tests than guessing from descriptions — you get exact labels, real navigation flows, and know which states are stable enough to validate.

## Critical Behavior for Test Generation

**IMPORTANT**: When generating tests, DO NOT include manual navigation at the start:
- **Mobile tests**: Automatically open the app at test start

**Manual steps are ONLY needed for**:
- `navigate`: Opening a deep link URL mid-test
- `open_app`: Reopening app after `kill_app` or `go_home`
- `kill_app`: Completely terminating the app (cold restart on reopen)
- `go_home`: Going to home screen while keeping app in background (warm restart on reopen)

**Correct pattern**: Tests should start with `instructions`, `validation`, or `extraction` blocks.

**Supported platforms**: `ios` and `android` only.

## Complete Schema Structure

    test:
      metadata:
        name: string                    # Required. Test name
        platform: enum                  # Required. Values: "ios" | "android"

      build:
        name: string                    # Required. Build variable name
        pinned_version: string          # Optional. Specific version

      blocks:
        - # Block objects (see Block Type Definitions below)

## Block Type Definitions

### instructions
    type: "instructions"
    step_description: string          # Required. Natural language instruction

### validation
    type: "validation"
    step_description: string          # Required. Condition to validate

### extraction
    type: "extraction"
    step_description: string          # Required. What to extract
    variable_name: string             # Required. Use kebab-case

### manual
    type: "manual"
    step_type: enum                   # Required. See values below
    step_description: string          # Value/parameter (format depends on step_type)

**step_type values**: `wait`, `open_app`, `kill_app`, `go_home`, `navigate`, `set_location`

**step_description format by step_type**:
- `wait`: Number of seconds (e.g., `"2"`, `"5"`, `"10"`)
- `open_app`: Bundle ID for system apps, or omit/empty for installed app
- `kill_app`: Not used (omit)
- `go_home`: Not used (omit)
- `navigate`: URL or deep link (e.g., `"myapp://settings/profile"`)
- `set_location`: Latitude,Longitude (e.g., `"37.7749,-122.4194"`)

**kill_app vs go_home**:
- `kill_app`: Completely terminates the app process. App will cold-start when reopened.
- `go_home`: Goes to home screen but keeps app in background. App resumes from where it left off.

**System App Bundle IDs** (for open_app step_description):
- iOS: `com.apple.Preferences` (Settings), `com.apple.mobilesafari` (Safari), `com.apple.mobileslideshow` (Photos), `com.apple.MobileAddressBook` (Contacts), `com.apple.Maps` (Maps)
- Android: `com.android.settings` (Settings), `com.android.chrome` (Chrome), `com.google.android.apps.photos` (Photos), `com.google.android.contacts` (Contacts), `com.google.android.apps.maps` (Maps)

**Examples**:

    # Wait 3 seconds
    - type: manual
      step_type: wait
      step_description: "3"

    # Kill app completely and reopen (cold start)
    - type: manual
      step_type: kill_app

    - type: manual
      step_type: open_app

    # Go home and return to app (warm resume)
    - type: manual
      step_type: go_home

    - type: manual
      step_type: open_app

    # Open a system app (Photos on iOS)
    - type: manual
      step_type: open_app
      step_description: "com.apple.mobileslideshow"

    # Open a deep link
    - type: manual
      step_type: navigate
      step_description: "myapp://settings/profile"

    # Set GPS location (lat,long format)
    - type: manual
      step_type: set_location
      step_description: "37.7749,-122.4194"

**IMPORTANT**: Do NOT use `open_app` at test start - the app opens automatically.

### if
    type: "if"
    condition: string                 # Required. Natural language condition
    then: array                       # Required. Array of blocks
    else: array                       # Optional. Array of blocks

### while
    type: "while"
    condition: string                 # Required. Loop condition
    body: array                       # Required. Array of blocks

### code_execution
    type: "code_execution"
    step_description: string          # Required. Script ID (UUID)
    variable_name: string             # Optional. Store script output

Code execution steps run pre-created scripts (Python, JavaScript, TypeScript, or Bash) in a secure sandbox. The step_description must be a valid script ID from your Revyl account. Scripts can make HTTP requests to your backend, generate dynamic test data, and perform calculations. The script stdout output becomes the variable value.

## Variable System

**CRITICAL**: Variables MUST use double curly brace syntax: `{{variable-name}}`

**Definition** (in extraction block):

    - type: extraction
      step_description: "The product price"
      variable_name: product-price

**Usage** (in any step_description):

    - type: validation
      step_description: "Total is {{product-price}}"

**Variable naming rules**:
- Use kebab-case: `order-number`, `user-email`, `phone-number`
- NO spaces, underscores, or special characters
- Must be defined via extraction block before use

## High-Level Instructions (Recommended for Complex Flows)

When there is indeterminism about what actions need to be taken, **use high-level instructions** instead of specifying every step. The AI will intelligently handle all the intermediate steps.

**Example - Instead of guessing every step**:

    # DON'T do this - too specific and may not match actual flow
    - type: instructions
      step_description: "Tap sign up button"
    - type: instructions
      step_description: "Enter email in email field"
    - type: instructions
      step_description: "Enter password in password field"
    # etc... you may not know all the steps

**DO this - Use high-level instruction**:

    - type: instructions
      step_description: "Complete the sign up flow"

The system will automatically navigate through all required screens and states without needing to know the exact flow in advance.

**When to use high-level instructions**:
- Multi-step flows where exact screens are unknown (sign up, checkout, onboarding)
- Flows that may vary based on user state
- Complex processes where you want the AI to handle the details

## Validation Best Practices

### Validate Outcomes, Not Transient States

Validate the **result** of an action, not loading/transition states. Transient UI (spinners, progress text) may vanish before screenshot capture.

    # BAD - transient state
    - type: validation
      step_description: "Loading spinner is visible"

    # GOOD - meaningful outcome
    - type: validation
      step_description: "The product list is displayed"

### Use Broad Validations When Uncertain

When you don't know exactly what to verify, **use broad validations** instead of guessing specific components.

**Broad validation examples**:

    # General state validations - GOOD
    - type: validation
      step_description: "The checkout screen is displayed"

    - type: validation
      step_description: "Login was successful"

    - type: validation
      step_description: "The user is on the dashboard"

**Negative validations (what shouldn't be visible)**:

    # Verify errors are NOT shown - GOOD
    - type: validation
      step_description: "No error message is displayed"

    - type: validation
      step_description: "The login screen is no longer visible"

**Avoid overly specific validations when uncertain**:

    # DON'T guess specific components you're not sure about
    - type: validation
      step_description: "The blue confirmation button in the top right is visible"

    # DO use general validations
    - type: validation
      step_description: "The confirmation screen is displayed"

## Wait Block Guidelines

Steps have **built-in retry/wait logic**. Only add waits for known significant delays (after `kill_app`, system-level pauses). Do NOT add waits after regular taps or between instruction-validation pairs.

## Test Generation Rules

1. **NEVER start with manual navigation** - app opens automatically at test start
2. **First block should be**: instructions, validation, or extraction
3. **Always specify platform** - Use `ios` or `android` in metadata.platform
4. **Use high-level instructions for complex flows** - "Complete the checkout process" instead of specifying every step
5. **Use broad validations when uncertain** - "Login was successful" instead of guessing specific UI elements
6. **Validate outcomes, not loading states** - "Data is displayed" not "Loading spinner is visible"
7. **Negative validations are powerful** - "No error message is displayed" confirms success
8. **Wait step_description must be a NUMBER** - Use "3" not "Wait for page to load" (represents seconds)
9. **Use waits sparingly** - Instruction/validation steps have built-in retries. Only add waits for known significant delays
10. **Use open_app ONLY when**: Reopening app after `kill_app` or `go_home` mid-test
11. **Always include**: test object, metadata.name, metadata.platform, build.name, blocks array
12. **Variables MUST use {{variable-name}} syntax**
13. **Platform awareness**: Only use platform-appropriate step_types (ios/android only)

## Complete Example

    test:
      metadata:
        name: "E-commerce Purchase"
        platform: ios

      build:
        name: "Shopping App - iOS"

      blocks:
        # App opens automatically - NO open_app needed

        - type: instructions
          step_description: "Search for 'laptop'"

        - type: instructions
          step_description: "Tap on the first result"

        - type: extraction
          step_description: "The product price"
          variable_name: product-price

        - type: instructions
          step_description: "Tap 'Add to Cart'"

        - type: validation
          step_description: "Cart shows price {{product-price}}"

        - type: if
          condition: "Discount field is visible"
          then:
            - type: instructions
              step_description: "Enter 'SAVE10'"

        - type: instructions
          step_description: "Tap 'Checkout'"

        - type: validation
          step_description: "Checkout screen displayed"

## Pre-Generation Quality Check

**IMPORTANT**: Before generating the final YAML, review the test for these common issues. Revyl uses AI/computer-use to execute tests, which is different from traditional Selenium automation.

### Missing Information (ASK USER)

If you detect these issues, **stop and ask the user** before generating the YAML:

- **"Log in" without credentials** - How should the AI log in? Provide username/password or specify test account
- **"Search for product" without specifying which** - What product name or search term?
- **"Enter address" without the address** - What address should be entered?
- **"Fill out form" without form data** - What values for each field?
- **Using `{{variable}}` that was never extracted** - Every variable must have an extraction block first

**BAD**: "Log in to the app" (missing credentials)
**GOOD**: "Log in with username '[email protected]' and password 'Test123!'"

### AI/Computer-Use Specific Issues

Revyl uses AI vision to execute tests. Unlike Selenium, the AI:

- **Can only see the current screen** - Don't assume what's behind menus or on other screens
- **Interprets instructions literally** - "Click the button" fails if there are multiple buttons; be specific
- **Has no memory between test runs** - Each test starts fresh from app launch
- **Cannot access backend/database** - All validations must check something VISIBLE on screen

**BAD**: "Verify the order was saved to the database"
**GOOD**: "Verify the confirmation message 'Order saved successfully' is displayed"

**BAD**: "Click the button"
**GOOD**: "Tap the 'Submit Order' button"

### Vague Instructions (ASK USER)

These will cause test failures. **Ask the user to clarify** before proceeding:

- **"Verify it works"** - What does "works" look like on screen?
- **"Check the page"** - Check what specifically?
- **"Make sure everything is correct"** - What should be visible to confirm correctness?
- **"Complete the process"** - What indicates the process is complete?

**BAD**: "Verify the payment works"
**GOOD**: "Verify the text 'Payment successful' is displayed"

### Assumptions That Break Tests

- **Assuming app state** - Tests start from app launch. Don't assume user is logged in or on a specific screen
- **Hardcoding dynamic data** - Prices, stock status, dates change. Extract them first or use validations that don't depend on exact values
- **Assuming specific UI positions** - "Click the third item" may break if list order changes

**Note on popups**: Revyl automatically handles common system popups (permissions, notifications). However, if you need SPECIFIC behavior (e.g., "Deny location permission" vs "Allow"), you must explicitly include that step.

### Quality Checklist

Before outputting the YAML, verify each item. **If any check fails, ask the user to provide the missing information** - do not guess or make assumptions:

1. [ ] Login steps include credentials OR explain how to authenticate
2. [ ] All `{{variables}}` are extracted before being used
3. [ ] Validations describe what should be VISIBLE on screen
4. [ ] Instructions are specific enough for only ONE interpretation
5. [ ] Test does NOT assume pre-existing app state
6. [ ] Dynamic data is either extracted or validated flexibly
7. [ ] Platform is specified (ios or android)
8. [ ] Build name matches exactly

**Do not generate YAML until all issues are resolved with the user.**

## Common Validation Errors to Avoid

1. Starting test with unnecessary manual navigation (open_app)
2. Using descriptive text for wait instead of a number - "3" not "Wait briefly"
3. Missing required fields (type, step_description, platform, etc.)
4. Invalid block type values
5. Invalid step_type values
6. Missing variable_name in extraction blocks
7. Empty then/body arrays in if/while blocks
8. **Using wrong variable syntax** - MUST be `{{variable-name}}`
9. Missing platform in metadata (always required)