mediar-ai/mcp-server-macos-use: Native macOS Control for AI Agents

mediar-ai/mcp-server-macos-use: Native macOS Control for AI Agents

Most AI agents that interact with operating systems rely on visual pipelines—screenshots, OCR, and coordinate guessing. That approach is brittle: it breaks when themes change, windows move, or resolution shifts. For developers building automation that needs to survive real-world conditions, a more robust layer is needed. mcp-server-macos-use addresses this by controlling macOS applications natively through accessibility APIs rather than pixels, exposing that control through the Model Context Protocol (MCP) so any compatible client can drive it.

This article covers what mcp-server-macos-use does, how it works under the hood, and how to integrate it into your agent stack. The focus keyword mcp-server-macos-use refers to the Swift-based MCP server maintained by mediar-ai, currently at 340 GitHub stars and 36 forks as of its last commit on April 26, 2026.

What is mediar-ai/mcp-server-macos-use?

mcp-server-macos-use is an MCP server implementation written in Swift that enables AI agents to control macOS applications programmatically. It was created by mediar-ai (contact: matt@mediar.ai, Discord: m13v_) and is distributed under a license categorized as "Other" in repository metadata—meaning you should review the actual license file before commercial use.

The server sits in the growing ecosystem of [INTERNAL_LINK: MCP servers] that standardize how LLMs invoke external tools. Rather than requiring model-specific integrations, MCP defines a protocol: the client (Claude Desktop, Cursor, or any compatible host) sends JSON-RPC messages over stdio, and the server translates those into native macOS actions.

What distinguishes this implementation is its reliance on macOS accessibility APIs via the MacosUseSDK dependency. These APIs provide structured, semantic information about UI elements—buttons, text fields, menus—rather than raw bitmaps. The server exposes five core tools for application control, each returning an accessibility tree traversal that gives the calling agent a structured view of the interface state.

The project's primary language is Swift, which matters for two reasons: it can link directly against Apple's native frameworks (AppKit, AX APIs), and it produces a single binary with no runtime interpreter overhead. The stdio transport keeps deployment simple—no network ports, no TLS certificates, no container orchestration.

Key Features

The server exposes functionality through five MCP tools, each following a consistent pattern: perform an action, then traverse and return the accessibility tree. This design lets agents observe the concrete effects of their actions rather than guessing whether a click registered.

macos-use_open_application_and_traverse launches or activates applications by name, bundle ID, or file path. Unlike AppleScript or shell-based open commands, this tool immediately returns the accessibility tree, so an agent knows what elements are available without a separate inspection step.

macos-use_click_and_traverse simulates mouse clicks at specified coordinates within a target application's window, identified by PID. The coordinate system behavior depends on MacosUseSDK implementation—agents should verify whether coordinates are window-relative or screen-relative during integration testing.

macos-use_type_and_traverse and macos-use_press_key_and_traverse handle text input and keyboard events respectively. The key press tool supports modifier combinations (Shift, Control, Option, Command, Function, CapsLock, NumericPad, Help) with multiple alias names per modifier for flexibility in agent-generated calls.

macos-use_refresh_traversal performs tree traversal without action side effects—essentially a "read-only" state poll. This is critical for agents that need to wait for asynchronous UI changes or verify preconditions before acting.

All action tools accept optional override parameters through the arguments object: traverseBefore/traverseAfter to control when snapshots occur, showDiff to highlight changes between traversals, onlyVisibleElements to reduce tree size, and animation controls (showAnimation, animationDuration) for debugging visibility. A delayAfterAction parameter helps with applications that need settling time.

The stdio transport means the server runs as a child process of the MCP client, inheriting its lifecycle. There's no daemon to manage, no port conflicts, and no authentication beyond the OS user context.

Use Cases

Automated UI Testing Without Fragile Selectors Traditional web-based testing tools (Selenium, Playwright) don't reach native macOS applications. mcp-server-macos-use fills this gap for teams shipping Mac desktop software. Because it uses accessibility APIs rather than visual matching, tests survive theme changes, Retina scaling adjustments, and window repositioning. The returned accessibility trees provide semantic element information that assertions can target directly.

Agentic Workflow Automation Knowledge workers running Claude Desktop can delegate multi-step macOS tasks to an agent with mcp-server-macos-use configured. Examples include: generating reports that require copying data from Numbers, formatting in TextEdit, and attaching to Mail messages; or batch-processing images through Preview's export dialog with parameter variations. The agent sees structured UI state after each action, enabling recovery from unexpected dialogs.

Accessibility-First RPA Replacement Robotic process automation on macOS often relies on brittle AppleScript or GUI scripting that breaks with OS updates. mcp-server-macos-use's MCP-based architecture decouples the automation logic (in the client LLM) from the execution layer (the Swift server). When macOS APIs change, only the server needs updates—the agent's reasoning remains stable.

Development and Debugging Aid Developers building macOS applications can use macos-use_refresh_traversal to inspect their own app's accessibility tree programmatically. This reveals how VoiceOver and other assistive technologies perceive the interface, helping catch AX labeling issues early.

Cross-Model Compatibility Because it speaks MCP rather than a vendor-specific protocol, mcp-server-macos-use works with any model behind a compatible client. Teams aren't locked to Claude or GPT-4—they can switch models or run local LLMs without changing their macOS automation layer.

Installation & Setup

The build process follows standard Swift Package Manager conventions. The README provides these exact commands:

# Example build command (adjust as needed, use 'debug' for development)
swift build -c debug # Or 'release' for production

# Running the server (it communicates via stdin/stdout)
./.build/debug/mcp-server-macos-use

Step-by-step breakdown:

1. Clone the repository to your local machine. The server requires macOS with Xcode Command Line Tools installed for Swift compilation.

2. Build in debug mode with swift build -c debug. This produces an executable at .build/debug/mcp-server-macos-use. Use -c release for optimized builds; the debug build is recommended during initial integration to capture any assertion failures.

3. Run the server directly from the build output. It immediately begins listening for MCP messages on stdin and writing responses to stdout. There is no configuration file or environment variable setup required for basic operation.

4. Configure your MCP client to launch this binary. For Claude Desktop, add the following to your configuration (replacing the path placeholder):

{
    "mcpServers": {
        "mcp-server-macos-use": {
            "command": "/path/to/your/project/mcp-server-macos-use/.build/debug/mcp-server-macos-use"
        }
    }
}

The command field must contain the absolute path to the built binary. Relative paths or ~ expansion may fail depending on how the client launches subprocesses. The README explicitly notes: "Replace /path/to/your/project/ with the actual absolute path to your mcp-server-macos-use directory."

5. Grant accessibility permissions when prompted. macOS will request permission for the binary to control your computer via System Preferences > Security & Privacy > Accessibility. This is mandatory—the AX APIs are gated behind this permission.

Real Code Examples

The README provides configuration and conceptual usage rather than extensive inline code samples. The following examples are derived directly from its documented tool schemas and setup instructions.

Example 1: Claude Desktop Configuration

This JSON configures the MCP client to launch the server on demand:

{
    "mcpServers": {
        "mcp-server-macos-use": {
            "command": "/Users/dev/projects/mcp-server-macos-use/.build/debug/mcp-server-macos-use"
        }
    }
}

The server process starts when Claude Desktop initializes MCP connections, and terminates when the client closes. No manual server management is required. The absolute path prevents working-directory ambiguity that breaks relative resolution in sandboxed app contexts.

Example 2: Opening an Application via MCP Tool Call

While the README doesn't provide raw JSON-RPC payloads, the tool schema defines this structure for macos-use_open_application_and_traverse:

{
    "name": "macos-use_open_application_and_traverse",
    "arguments": {
        "identifier": "Safari"
    }
}

The identifier parameter accepts flexible targeting: application name ("Safari"), bundle ID ("com.apple.Safari"), or absolute file path ("/Applications/Safari.app"). The tool activates the application if running, or launches it if not, then returns the accessibility tree. This single call replaces the traditional two-step pattern of open -a Safari followed by a separate AX inspection.

Example 3: Typing with Post-Action Delay

For applications with slow focus transitions, the optional parameters allow timing adjustments:

{
    "name": "macos-use_type_and_traverse",
    "arguments": {
        "pid": 12345,
        "text": "Quarterly report draft",
        "delayAfterAction": 0.5
    }
}

The pid identifies the target process (obtainable from the open_application tool's return or pgrep). The delayAfterAction of 0.5 seconds gives the destination field time to process focus change before tree traversal captures the new state. This parameter is particularly relevant for web views or complex document-based applications.

The README notes that these optional parameters map to ActionOptions in the MacosUseSDK source. Developers needing precise behavior should consult that implementation for default values and validation logic.

Advanced Usage & Best Practices

Tree Diffing for Efficient Agents Enable showDiff: true when calling action tools to receive only changed accessibility elements between pre- and post-action traversals. This reduces token consumption in the LLM context window and focuses the agent's attention on relevant UI mutations. The README mentions this as a capability but doesn't specify diff format—inspect actual responses during development.

Visibility Filtering for Performance Set onlyVisibleElements: true to exclude off-screen or hidden elements from traversals. This is essential for complex applications like Xcode or Photoshop where the full accessibility tree can contain thousands of nodes. The trade-off: agents may miss elements that become visible through scrolling.

Coordinate System Verification The README notes that x/y coordinates for click operations depend on MacosUseSDK behavior. Before deploying agents that click precisely, verify through test calls whether coordinates are window-relative, screen-relative, or use another reference frame. This is a common integration failure point.

Process Lifecycle Management Because the server communicates over stdio, it terminates when the MCP client disconnects. For long-running automations, ensure your client maintains the connection or implements restart logic. The server itself is stateless regarding application control—all state lives in the target applications' actual UI.

Error Handling Strategy The accessibility APIs can fail silently if permissions are revoked or applications become unresponsive. Agents should implement retry logic with macos-use_refresh_traversal to verify application state before assuming an action succeeded.

Comparison with Alternatives

Playwright dominates web automation but cannot interact with native macOS applications. Teams needing both web and native coverage would use both tools in complementary roles.

AppleScript and Shortcuts provide native control but return unstructured results (boolean success, plain text) rather than semantic UI trees. They're suitable for simple triggers, not agentic reasoning loops that need to observe and respond to interface state.

Anthropic's computer-use (available through their API) takes the screenshot-based approach that mcp-server-macos-use specifically avoids. It works across platforms but incurs latency from image encoding/decoding and can misclick when visual layouts shift. mcp-server-macos-use offers lower latency and higher precision for macOS-specific workflows, at the cost of platform restriction and self-hosting responsibility.

FAQ

What macOS versions are supported? The README doesn't specify minimum versions. Accessibility APIs have evolved significantly; test on your target deployment version.

Is there a prebuilt binary? No—the README shows only source compilation via swift build. You'll need Xcode Command Line Tools.

What license applies? The repository metadata lists "Other." Review the actual license file in the repository before commercial use or redistribution.

Can this control sandboxed App Store apps? Accessibility API access is restricted for some sandboxed applications. System apps and most third-party apps work; test your specific targets.

How do I get the PID for click/type operations? Use the PID returned by macos-use_open_application_and_traverse, or query via pgrep/ps in your agent's shell tool.

Does it work with Apple Silicon and Intel Macs? Swift compiles universally. The built binary architecture depends on your build machine; use swift build --arch x86_64 or --arch arm64 for specific targets.

Who maintains this? mediar-ai, with direct contact at matt@mediar.ai and Discord m13v_. The project welcomes issue submissions for feature tailoring.

Conclusion

mcp-server-macos-use occupies a specific, valuable niche: AI agent control of macOS applications through native accessibility APIs, exposed via the open Model Context Protocol. For developers and power users frustrated by screenshot-based automation's fragility, it offers a structured alternative that returns semantic UI state rather than pixels to interpret.

The tool is best suited for: teams shipping macOS desktop software needing automated testing; Claude Desktop users building personal automation workflows; and developers constructing multi-step agents that require precise interaction with native applications. It's not a drop-in replacement for cross-platform solutions, and it requires comfort with Swift compilation and MCP client configuration.

The project is actively maintained with recent commits, and the maintainer explicitly invites feature requests via issues or direct contact. If your stack includes macOS targets and MCP-compatible clients, evaluate whether mcp-server-macos-use's structured approach outperforms visual alternatives for your reliability requirements.

Get started at the official repository or visit macos-use.dev for additional documentation.