Joinly Exposed: The Open-Source Secret Weapon for AI Agents in Video Calls

What if your AI assistant could actually join your standups, take notes in your client calls, and execute tasks while the meeting is still running? Not after. Not from a transcript. In real time.

Here's the brutal truth: most AI "meeting assistants" are glorified recording devices. They sit on the sidelines, capture audio, and dump a summary in your inbox three hours later. They're passive. They're slow. And they sure as hell can't hop into a Zoom call and create a GitHub issue while your product manager is still talking.

That changes now.

Enter joinly.ai — a connector middleware that's quietly becoming the infrastructure layer for AI-native meetings. Built by a team in Osnabrück, Germany, this open-source project lets any AI agent join Google Meet, Zoom, or Microsoft Teams as an active participant. Not a recorder. A participant. It can speak, listen, chat, mute itself, read screenshares, and execute tasks through MCP (Model Context Protocol) servers — all while the meeting is live.

If you're building AI agents or just tired of passive meeting bots, keep reading. This is the technical deep dive you didn't know you needed.

What is joinly.ai?

joinly.ai is an open-source connector middleware designed to bridge the gap between AI agents and video conferencing platforms. At its core, it's an MCP (Model Context Protocol) server that exposes meeting-specific tools and resources to any LLM-powered agent, enabling real-time voice and text interaction inside live meetings.

The project was created by joinly-ai, a team focused on making meetings natively accessible to AI agents rather than retrofitting post-meeting analysis onto legacy workflows. It's trending right now because it solves a genuinely hard problem: bidirectional, real-time AI participation in human meetings.

Unlike traditional meeting assistants that rely on calendar integrations and post-processing, joinly operates as a browser-based participant. It joins meetings through the same web interfaces humans use — Google Meet, Zoom, Teams — which means no API keys from meeting providers, no enterprise partnerships, and no platform lock-in. This architecture choice is both brilliant and subversive: it democratizes access to meeting infrastructure.

The project is 100% open-source, self-hosted, and privacy-first. No data leaves your infrastructure unless you want it to. You can run it locally with Ollama, use your own GPU for transcription, and even host your own TTS models. The Docker images are pre-packaged with Chromium, Whisper, and Kokoro, making deployment genuinely one-command simple.

With MIT licensing, active Discord community, and a public roadmap that includes camera support, screen sharing, and A2A protocol integration, joinly is positioning itself as the foundational infrastructure for the emerging "AI coworker" category.

Key Features That Make joinly.ai Insane

Let's break down what makes this tool technically remarkable:

• 🔴 Live Interaction: Your agents don't just listen — they execute tasks and respond by voice or chat in real-time. This isn't post-processing; this is in-meeting action. The agent can create tickets, update documentation, or query databases while stakeholders are still discussing.

• 🗣️ Conversational Flow Engine: Built-in logic handles the messiness of human conversation — interruptions, overlapping speech, multi-speaker dynamics. Most voice AI fails here. Joinly's turn-taking detection and end-of-utterance handling make interactions feel natural rather than robotic.

• 🌐 Cross-Platform by Design: Google Meet, Zoom, Microsoft Teams — any platform accessible via browser. No SDK dependencies, no API rate limits, no enterprise approval cycles. The browser-as-participant model is elegantly simple.

• 🧠 Bring-Your-Own-LLM: Full compatibility with OpenAI, Anthropic Claude, and local models via Ollama. You're not locked into a proprietary model. Swap providers based on cost, latency, or privacy requirements.

• 🎙️ Modular TTS/STT Stack:

  • STT: Whisper (local, default) or Deepgram (cloud)
  • TTS: Kokoro (local, default), ElevenLabs, or Deepgram This modularity means you optimize for speed, quality, or cost per use case.

• 🚀 100% Open-Source & Self-Hosted: Full control over data. No SaaS subscription traps. The Docker image (~2.3GB) packages everything — browser, models, runtime — for true zero-dependency deployment.

• 🛠️ MCP-Native Architecture: Built on the Model Context Protocol, making it composable with any MCP-compatible tool ecosystem. Connect Notion, GitHub, Tavily search, or custom internal tools.

5 Real-World Use Cases Where joinly.ai Dominates

1. The Autonomous Engineering Standup

Imagine an AI agent that joins your daily standup, hears "we need to investigate the auth bug," creates a GitHub issue with proper labels, assigns it to the on-call engineer, and posts the link in Slack — before the standup ends. The demo video shows exactly this: joinly answering "What is Joinly?" by web search, then creating a demo repository issue.

2. Live Documentation Editing

Connect joinly to your Notion workspace via MCP. During a product review, when the PM says "update the Q3 roadmap with the new priority," the agent pulls up the page and edits it live. The Notion demo proves this isn't theoretical — it's shipping today.

3. Sales Call Intelligence

Deploy an agent that joins sales calls, transcribes in real-time, detects competitor mentions, queries your battlecard database via MCP, and whispers talking points to the rep through a side channel. The get_video_snapshot tool even lets it see screenshares — perfect for reviewing proposals live.

4. Accessibility-First Meeting Participation

For team members who can't attend live, joinly doesn't just record — it participates on their behalf. With configurable voice, language support (--lang de, --lang ja), and persistent memory (roadmap), agents can represent stakeholders who are in different time zones or have scheduling conflicts.

5. Regulatory Compliance & Privacy-Critical Environments

Finance, healthcare, government — sectors where SaaS meeting bots are non-starters. Joinly's self-hosted model means zero third-party data exposure. Run entirely on-prem with local Whisper + Kokoro, Ollama for LLM inference, and audit every byte that leaves your network.

Step-by-Step Installation & Setup Guide

Ready to deploy? Here's the complete path from zero to live AI agent in your next meeting.

Prerequisites

• Docker installed on your machine
• A valid API key from your preferred LLM provider (OpenAI, Anthropic, or Ollama for local)

Step 1: Create Project Directory

mkdir joinly && cd joinly

Step 2: Configure Environment Variables

Create a .env file with your LLM credentials:

# .env — OpenAI configuration (default)
JOINLY_LLM_MODEL=gpt-4o
JOINLY_LLM_PROVIDER=openai
OPENAI_API_KEY=your-openai-api-key

Pro tip: Check the .env.example file for Anthropic Claude and Ollama configurations. Delete unused provider placeholders to avoid conflicts.

Step 3: Pull the Docker Image

# Standard image (~2.3GB, includes Chromium + models)
docker pull ghcr.io/joinly-ai/joinly:latest

# OR: CUDA-enabled image for GPU-accelerated transcription/TTS
docker pull ghcr.io/joinly-ai/joinly:latest-cuda

Step 4: Launch Your Meeting and Deploy the Agent

Start your meeting in Zoom, Google Meet, or Teams. Copy the meeting URL. Then run:

# Basic client mode — agent joins directly
docker run --env-file .env ghcr.io/joinly-ai/joinly:latest --client <MeetingURL>

# With GPU support (requires NVIDIA Container Toolkit, CUDA >= 12.6)
docker run --gpus all --env-file .env ghcr.io/joinly-ai/joinly:latest-cuda --client <MeetingURL>

That's it. Your agent is now in the meeting.

Step 5: Verify and Debug

# Enable verbose logging
-v    # or -vv, -vvv for deeper levels

# Launch with VNC for visual debugging
--vnc-server --vnc-server-port 5900
# Then connect via VNC client to localhost:5900

REAL Code Examples from the Repository

Let's examine actual code patterns from joinly's README and implementation, with detailed breakdowns.

Example 1: Server-Client Architecture with External MCP Tools

The quickstart uses --client for simplicity, but the real power comes from running joinly as an MCP server and connecting external clients. This enables composing multiple tool ecosystems:

# Terminal 1: Start joinly as MCP server (port 8000)
docker run -p 8000:8000 ghcr.io/joinly-ai/joinly:latest

# Terminal 2: Connect external client with uv (ultrafast Python package manager)
# Prerequisites: uv installed (https://github.com/astral-sh/uv)
uvx joinly-client --env-file .env <MeetingUrl>

Why this matters: Running as server decouples the meeting infrastructure from your agent logic. You can now layer on Notion, GitHub, Tavily search, or any MCP-compatible tool without rebuilding the core meeting participant.

Example 2: MCP Server Configuration for Tool Composition

Here's the exact JSON configuration pattern for adding external MCP servers to your agent:

{
    "mcpServers": {
        "localServer": {
            "command": "npx",
            "args": ["-y", "package@0.1.0"]
        },
        "remoteServer": {
            "url": "http://mcp.example.com",
            "auth": "oauth"
        }
    }
}

Practical implementation with Tavily web search:

# 1. Create config.json with Tavily settings (see examples/config_tavily.json)
# 2. Run client with MCP configuration
uvx joinly-client --env-file .env --mcp-config config.json <MeetingUrl>

Technical insight: The "mcpServers" key follows the fastmcp client specification. Each entry becomes available as callable tools in your agent's context. The command pattern uses npx for Node.js-based servers, while url enables remote service integration. This is how the GitHub demo achieves live web search → issue creation in a single conversational flow.

Example 3: Customizing TTS/STT Providers via CLI

Joinly's modular audio stack is configured through command-line arguments. Here's how to optimize for different scenarios:

# High-quality cloud TTS with ElevenLabs
# Requires ELEVENLABS_API_KEY in .env
docker run --env-file .env ghcr.io/joinly-ai/joinly:latest \
  --client <MeetingURL> \
  --tts elevenlabs \
  --tts-arg voice_id=EXAVITQu4vr4xnSDxMaL  # Sarah voice

# Local GPU-accelerated transcription with larger Whisper model
# Uses CUDA image for distil-large-v3 default
docker run --gpus all --env-file .env ghcr.io/joinly-ai/joinly:latest-cuda \
  --client <MeetingURL> \
  --stt whisper \
  --stt-arg model_name=large-v3  # Best accuracy, requires download on first run

# German-language meeting with local everything (privacy max)
docker run --env-file .env ghcr.io/joinly-ai/joinly:latest \
  --client <MeetingURL> \
  --name "KI-Assistent" \
  --lang de \
  --tts kokoro \
  --stt whisper

Critical implementation detail: The --tts-arg and --stt-arg patterns pass provider-specific parameters through to the underlying service. For Kokoro, voice selects from available speakers. For Whisper, model_name controls the accuracy/speed tradeoff: base (default, CPU-friendly), small, medium, large-v3 (best, GPU-recommended).

Example 4: Building Custom Agents with Native MCP Tools

For developers building beyond the joinly-client wrapper, the MCP server exposes these precise tools:

# Conceptual usage based on client_example.py pattern
# See: https://github.com/joinly-ai/joinly/client/README.md#code-usage

# Tools available to your agent:
# - join_meeting(url, name, passcode=None)
# - leave_meeting()
# - speak_text(text)  # TTS output
# - send_chat_message(message)  # Text chat
# - mute_yourself() / unmute_yourself()
# - get_chat_history() → JSON
# - get_participants() → JSON
# - get_transcript(minutes=None) → JSON with timestamps
# - get_video_snapshot() → Image (screenshare capture!)

# Resources (subscribable for real-time updates):
# - transcript://live  # Streaming transcript with speaker attribution

The get_video_snapshot tool is particularly underappreciated — it enables visual understanding of shared screens, making joinly capable of "seeing" presentations, diagrams, and demos. Combined with multimodal LLMs, this creates agents that can discuss visual content in real-time.

Advanced Usage & Best Practices

Performance Optimization

• GPU is worth it: The CUDA image with distil-large-v3 provides dramatically better transcription accuracy than CPU base. For production deployments, the NVIDIA Container Toolkit setup pays dividends in comprehension quality.
• Model caching: First container start downloads model weights. Mount a volume to /root/.cache to persist across restarts:

  docker run -v joinly-cache:/root/.cache ...
  

Security Hardening

• Network isolation: Run in dedicated Docker network, restrict egress to known LLM provider endpoints
• Secret management: Use Docker secrets or external vaults instead of .env files in production
• VNC exposure: Never expose --vnc-server on public interfaces; use SSH tunneling

Memory and Persistence (Roadmap-Aware)

The public roadmap shows "Improve client memory" as active development. For now, implement conversation persistence by:

1. Subscribing to transcript://live resource
2. Storing utterances in external vector DB via MCP tool
3. Injecting relevant context into subsequent meeting joins

Multi-Agent Orchestration

Run multiple joinly instances with different --name values and MCP configurations:

• "Documentarian" agent: focused on get_transcript, Notion MCP, write access
• "Action" agent: GitHub + Slack MCPs, task execution focused
• "Analyst" agent: Tavily + internal DB, research and Q&A focused

Comparison with Alternatives

The verdict: If you need a polished SaaS product with zero setup, incumbents work. If you need AI agents that actually do things in meetings, with full control over data, models, and tools — joinly is the only open-source game in town.

FAQ

Is joinly.ai free to use?

Yes. The project is MIT-licensed and 100% free. You pay only for your infrastructure (LLM API calls, GPU if used) and optional cloud TTS/STT services.

Can I run joinly entirely offline?

Absolutely. Use Ollama for local LLM inference, Whisper for local STT, and Kokoro for local TTS. Zero external API calls required.

What meeting platforms are supported?

Any platform accessible via web browser: Google Meet, Zoom, Microsoft Teams, and theoretically any WebRTC-based service.

How does joinly handle meeting authentication?

It joins as a browser participant, so it handles authentication the same way a human would — meeting links, passcodes, and waiting rooms. The join_meeting tool accepts optional passcode parameters.

Is GPU required?

No, but recommended. The CPU image runs Whisper base model adequately. GPU with CUDA >= 12.6 enables distil-large-v3 for significantly better transcription.

Can multiple agents join the same meeting?

Yes. Run separate container instances with different --name values. Each appears as an independent participant.

What's the difference between --client and server mode?

--client runs a built-in agent directly. Server mode exposes the MCP interface on port 8000, enabling external clients and multi-tool composition.

Conclusion

The future of work isn't AI that summarizes what happened. It's AI that participates while it happens.

joinly.ai represents a fundamental architectural shift: from passive meeting bots to active, tool-equipped agents that join, listen, speak, and execute in real-time. Its MCP-native design, cross-platform browser-based approach, and uncompromising open-source ethos make it the infrastructure layer that AI-native teams have been waiting for.

The project is early — camera support, screen sharing, and A2A protocol integration are on the roadmap — but what's shipping today is already more capable than most proprietary alternatives. The demos don't lie: this thing creates GitHub issues from voice commands and edits Notion pages live.

My take? If you're building AI agents, you need to understand this architecture. If you're running meetings, you need to try this deployment. The barrier to entry is one Docker command.

👉 Star the repository: github.com/joinly-ai/joinly

👉 Join the community: Discord

👉 Skip the setup, try cloud: cloud.joinly.ai

The age of AI agents that actually show up to work starts here. Don't be the last team to deploy one.

Made with ❤️ in Osnabrück. Built for builders everywhere.