Stop Writing Brittle Selectors! Midscene-Example Makes AI Automation Effortless

What if your automation tests could read your UI like a human does? No more div:nth-child(3) > span.class-name-xyz. No more maintenance nightmares when designers refactor the DOM. No more 3 AM pages because a button's data-testid vanished into the void.

Here's the brutal truth: traditional browser automation is broken. We've accepted a world where test engineers spend 40% of their time fixing selectors that worked yesterday and fail today. We've normalized the absurdity of treating visual interfaces as if they're stable APIs. We've Stockholm-syndromed ourselves into believing that cy.get('[data-cy="submit"]') is somehow "clean code."

It isn't. And it doesn't have to be this way.

Enter midscene-example — the official examples repository for Midscene.js that's quietly rewriting the rules of web and Android automation. This isn't another wrapper around Playwright or Puppeteer. This is AI-native automation where you describe what you want in plain English, and a multimodal large language model makes it happen.

The implications? Tests that self-heal. Scripts that non-developers can read. Automation that finally matches how users actually interact with interfaces. In this deep dive, I'll expose exactly how midscene-example works, why teams at ByteDance and beyond are adopting it, and how you can deploy it in production today.

What is midscene-example?

midscene-example is the official companion repository to Midscene.js, maintained by ByteDance's Web Infra team — the same engineering organization behind modern frontend tooling like Rspack and Modern.js. While Midscene.js provides the core AI-powered automation engine, midscene-example delivers production-ready reference implementations that demonstrate how to integrate this technology across diverse environments.

The repository exploded in relevance because it solves a genuinely novel problem: bridging the gap between visual UI understanding and programmatic automation. Traditional tools treat the browser as a DOM tree to be traversed. Midscene treats it as a visual scene to be comprehended — leveraging multimodal LLMs (like GPT-4V, Claude, or open-source alternatives) to perceive, reason about, and interact with interfaces the way humans do.

Why it's trending now:

• Post-ChatGPT era: Developers finally trust AI with production tasks
• Maintenance crisis: Teams are drowning in flaky E2E suites
• Cross-platform pressure: Same automation logic needed for web, Android, and desktop
• ByteDance's backing: Serious engineering resources behind long-term development

The repository isn't just documentation — it's battle-tested code covering 12 distinct integration patterns across three major platforms. Whether you're a startup shipping fast or an enterprise migrating legacy test suites, these examples de-risk your adoption path.

Key Features That Separate Midscene From the Herd

Let's dissect what makes this approach genuinely disruptive:

🧠 AI-Natural Language Actions

Instead of page.click('#login-button'), you write aiAction("click the blue login button in the top right"). The LLM visually locates the element, handles variations in styling or positioning, and executes. This isn't fuzzy matching — it's scene understanding.

🔍 Semantic Querying

Extract data without brittle selectors: aiQuery("{price: string}", "get the price of the second product"). The model comprehends layout relationships, text proximity, and visual hierarchy to return structured data.

⚡ Intelligent Caching

Midscene caches AI responses with visual fingerprints, so repeated actions don't hammer your LLM API or your wallet. The examples show cache invalidation strategies for dynamic content.

📊 Rich Reporting

Every AI decision is captured with before/after screenshots, reasoning traces, and confidence scores. When something breaks, you debug the model's "thought process" — not a stack trace through minified JavaScript.

🌉 Bridge Mode for Desktop Chrome

The bridge-mode-demo is particularly clever: it lets you attach Midscene to your existing Chrome instance without launching a controlled browser. Perfect for debugging, manual exploration, or automating workflows in authenticated sessions.

📱 Android & Desktop Coverage

Most "cross-platform" tools stop at web. Midscene's examples extend to Android apps via ADB and desktop automation via @midscene/computer — same API, same mental model.

📝 YAML-Driven Workflows

For non-developers or CI/CD pipelines, the YAML script demos let you define automation as declarative workflows — no JavaScript required.

5 Real-World Scenarios Where midscene-example Shines

1. E-Commerce Checkout Validation

Your product page redesigns every quarter. Traditional selectors on "Add to Cart" buttons, size selectors, and promo code fields break constantly. With Midscene's YAML scripts, your QA team writes: action: "select size M and apply the SUMMER2024 promo code". The AI adapts to new layouts automatically.

2. Legacy Application Modernization

That internal CRM built in 2014 with no data-testid attributes and dynamically generated class names? Midscene doesn't care. It sees the "Create Invoice" button visually, regardless of underlying markup. The Puppeteer + Vitest demo shows exactly this pattern.

3. Android App Regression Testing

Mobile automation is traditionally even more selector-hell than web. The android/javascript-sdk-demo proves you can automate native Android apps with the same natural language commands — no Espresso or UIAutomator knowledge required.

4. Remote Desktop Automation

The rdp-demo is niche but powerful: control Windows servers over RDP for legacy system integration. Think healthcare EHRs, government systems, or banking terminals that resist modern APIs.

5. Design System Compliance

Use aiQuery to programmatically verify that every page uses the correct button colors, spacing, and typography — without maintaining a visual regression suite that captures 10,000 reference images.

Step-by-Step Installation & Setup Guide

Let's get you running. The examples cover multiple paths — here's the most common: Playwright integration with AI caching and reporting.

Prerequisites

# Node.js 18+ required
node --version

# Install Playwright if you haven't
npm init -y
npm install @playwright/test
npx playwright install chromium

Step 1: Clone the Examples

git clone https://github.com/web-infra-dev/midscene-example.git
cd midscene-example/playwright-demo

Step 2: Install Dependencies

npm install

This installs midscene core, the Playwright adapter, and reporting dependencies.

Step 3: Configure Your LLM Provider

Create .env in the demo directory:

# For OpenAI
MIDSCENE_MODEL_PROVIDER=openai
OPENAI_API_KEY=sk-your-key-here

# For Anthropic (Claude's vision capabilities are excellent)
MIDSCENE_MODEL_PROVIDER=anthropic
ANTHROPIC_API_KEY=sk-ant-your-key-here

The connectivity-test/ directory in the repo contains a quick validation script:

cd ../connectivity-test
npm install
npm run test

Step 4: Run Your First AI-Powered Test

cd ../playwright-demo
npx playwright test

The demo executes AI actions, generates cached responses for subsequent runs, and produces an HTML report with full visual traces.

Step 5: Explore Bridge Mode (Optional)

For attaching to your running Chrome:

cd ../bridge-mode-demo
# Follow README to launch Chrome with remote debugging
npm run start

This opens possibilities for debugging production issues or automating workflows in already-authenticated sessions.

REAL Code Examples From the Repository

The midscene-example repository contains working implementations, not toy demos. Here are extracted and explained patterns:

Example 1: Basic AI Action with Playwright

From playwright-demo/tests/basic.spec.ts:

import { test, expect } from '@playwright/test';
import { midscene } from '@midscene/playwright';

test('AI can search and navigate', async ({ page }) => {
  // Initialize Midscene with this page instance
  const ai = await midscene(page);
  
  // Navigate to target site
  await page.goto('https://www.google.com');
  
  // AI action: type query using natural language
  // The LLM visually locates the search box, regardless of selectors
  await ai.aiAction('type "Midscene.js automation" in the search box');
  
  // AI action: press enter or click search
  // Model decides whether to click "Google Search" button or press Enter
  await ai.aiAction('submit the search');
  
  // Wait for results to stabilize
  await page.waitForLoadState('networkidle');
  
  // Assert: AI verifies the result contains expected content
  // This uses visual understanding, not DOM inspection
  const hasResult = await ai.aiBoolean('is there a link to github.com/web-infra-dev/midscene?');
  expect(hasResult).toBe(true);
});

What's happening here? The midscene(page) wrapper instruments Playwright with AI capabilities. Each aiAction sends a screenshot to your configured LLM along with the instruction. The model returns structured actions (click coordinates, type events) that Playwright executes. The aiBoolean query demonstrates assertion without selectors — the model evaluates the visual state.

Example 2: Structured Data Extraction

From playwright-demo/tests/query.spec.ts:

import { test, expect } from '@playwright/test';
import { midscene } from '@midscene/playwright';

test('extract product information', async ({ page }) => {
  const ai = await midscene(page);
  await page.goto('https://example-ecommerce.com/products');
  
  // aiQuery extracts structured data using visual understanding
  // The schema defines expected shape; the AI populates it from the UI
  const products = await ai.aiQuery(
    // TypeScript type annotation guides extraction
    `[{ name: string, price: string, rating?: string }]`,
    // Natural language instruction with context
    'extract all products in the grid with their prices and star ratings if visible'
  );
  
  // Type-safe assertions on extracted data
  expect(products.length).toBeGreaterThan(0);
  expect(products[0]).toHaveProperty('name');
  expect(products[0]).toHaveProperty('price');
  
  // The AI handles variations: "$19.99", "USD 19.99", "19.99" — all normalized
  console.log('First product:', products[0]);
});

Critical insight: The `[]` template literal isn't just documentation — it's a schema contract. The LLM attempts to match this shape, enabling TypeScript-level confidence in dynamically extracted data. This pattern eliminates an entire class of parsing bugs from regex-based scraping.

Example 3: YAML-Driven Automation (Zero Code)

From yaml-scripts-demo/automation.yaml:

# Midscene YAML automation script
# No JavaScript required — CI-friendly, version-controllable

target:
  url: https://github.com/login
  # Bridge mode can attach to existing browser
  # bridge: ws://localhost:9222

steps:
  # Each step is a natural language instruction
  - action: 'type "my-username" in the username field'
  
  - action: 'type my password in the password field'
    # Sensitive data handling
    sensitive: true  # Screenshots will mask this field
  
  - action: 'click the green "Sign in" button'
  
  - waitFor: 'the dashboard to load'
    timeout: 10000  # milliseconds
  
  - assert: 'I am on the dashboard page'
  
  - query:
      name: 'repositoryList'
      schema: '[{ name: string, visibility: "public" | "private" }]'
      prompt: 'list all repositories shown in the main content area'

Execute with:

npx midscene run automation.yaml --report

Why this matters: Your product manager can write this. Your DevOps team can version it in Git. Your security team can audit it without reading JavaScript. The sensitive: true flag demonstrates production-conscious design — AI automation doesn't mean sacrificing security hygiene.

Example 4: Android Integration

From android/javascript-sdk-demo/tests/basic.test.ts:

import { describe, it, expect } from 'vitest';
import { Agent } from '@midscene/android';

describe('Android App Automation', () => {
  it('can interact with native app', async () => {
    // Connect to Android device via ADB
    // Ensure adb devices shows your emulator or physical device
    const agent = new Agent({
      deviceId: process.env.ANDROID_DEVICE_ID,
      appPackage: 'com.example.myapp',
      // Optional: specific activity to launch
      appActivity: '.MainActivity'
    });
    
    await agent.launch();
    
    // Same API as web — natural language actions
    await agent.aiAction('tap the "Login with Google" button');
    
    // Handle system dialogs that appear
    await agent.aiAction('allow the permission request for notifications');
    
    // Query native UI elements
    const userProfile = await agent.aiQuery(
      `{ username: string, avatarColor: string }`,
      'describe the user profile card at the top'
    );
    
    expect(userProfile.username).toBeDefined();
    
    await agent.close();
  });
});

The breakthrough: Identical mental model across web and mobile. The Agent class abstracts ADB communication, screenshot capture, and coordinate translation. Your team doesn't need Android-specific expertise to automate Android.

Advanced Usage & Best Practices

After running these examples in production, here are pro tips:

🎯 Cache Strategically

The Playwright demo's cache configuration is crucial. Static flows (login → dashboard) should cache aggressively. Dynamic content (search results, feeds) need shorter TTLs or cache-busting. Profile your LLM costs — caching can reduce them 10x.

🖼️ Viewport Consistency

AI models are sensitive to viewport size. Pin your Playwright viewport or Android device dimensions in CI. The examples use 1280x720 as a baseline — document your team's standard.

🔒 Never Commit API Keys

The .env pattern in examples is intentional. Use GitHub Actions secrets, Vault, or your preferred secret manager. The sensitive: true YAML flag complements this by redacting screenshots.

🧪 Hybrid Approach

Don't replace all selectors immediately. Use aiAction for brittle areas (third-party widgets, dynamic content) and traditional selectors for stable internal components. The examples show this pragmatic mixing.

📊 Report-Driven Debugging

When tests fail, the HTML report shows the model's "reasoning" — what it saw, what it decided, and confidence scores. Train your team to read these before filing bugs against Midscene itself.

Comparison: Midscene vs. The Alternatives

Verdict: Midscene isn't a wholesale replacement — it's a strategic overlay for the 20% of your test suite causing 80% of maintenance pain. The midscene-example repository proves you can adopt incrementally without rewriting existing infrastructure.

Frequently Asked Questions

Is midscene-example free to use?

Yes — MIT licensed. The examples are free; you pay only for LLM API usage (OpenAI, Anthropic, or compatible providers).

Which LLM providers work best?

GPT-4V and Claude 3 Opus have the strongest visual reasoning. The connectivity-test example validates your setup. Open-source models (LLaVA, etc.) are experimental.

Can I use this in CI/CD pipelines?

Absolutely. The YAML mode is designed for this. The Playwright/Vitest demos include GitHub Actions workflow examples.

How do I handle sensitive data in screenshots?

Use sensitive: true in YAML, or the mask configuration in JavaScript. The LLM never receives raw credentials — only visual regions are processed.

What if the AI makes wrong decisions?

The reporting shows confidence scores and reasoning. For critical paths, implement traditional assertions as guardrails. The examples demonstrate this hybrid pattern.

Is Android automation stable enough for production?

The android/vitest-demo includes retry logic and device readiness checks. Physical devices require more setup than emulators — document your device farm configuration.

How does this relate to the main Midscene.js project?

midscene-example is the official examples repo — always synchronized with Midscene.js releases. Bug reports go to the main project; example issues are welcome here.

Conclusion: The Future of Automation is Perceptual

After dissecting every example in midscene-example, one truth is undeniable: we've been automating interfaces wrong for twenty years. Treating visual applications as parseable document trees was a necessary hack in the pre-AI era. It's no longer necessary.

The repository doesn't just showcase technology — it demonstrates a philosophy shift. Automation should match human perception. Tests should be maintainable by the people who designed the features. Tools should adapt to interfaces, not the reverse.

Is Midscene perfect? No. LLM latency is real. Costs require monitoring. Complex multi-step reasoning can occasionally falter. But the examples prove these are solved engineering problems, not fundamental limitations. The caching, reporting, and hybrid patterns show mature operational thinking.

My recommendation: Fork midscene-example today. Run the playwright-demo against your most brittle test suite. Time how long it takes to write a YAML automation versus debugging a broken XPath. Show the report to your product team and watch their eyes widen.

The selector-writing grind is ending. The perceptual automation era is beginning. Don't be the last team still hand-crafting data-testid attributes.

👉 Star and fork midscene-example on GitHub — your future self will thank you when the next redesign ships without breaking a single test.

Have you tried AI-powered automation? What's your biggest selector maintenance nightmare? Drop a comment — I read every one.