Stop Wasting Hours on ML Pipelines! RD-Agent Automates Everything

What if your AI could research, code, and optimize itself—while you sleep?

Every data scientist knows the soul-crushing grind. You read paper after paper, manually implement architectures, tweak hyperparameters until 3 AM, and still wonder if a better approach existed all along. The research-to-production gap isn't just annoying—it's bleeding billions in lost productivity across the industry. What if I told you Microsoft built an autonomous agent that doesn't just assist your R&D... it becomes your R&D team?

Enter RD-Agent—the open-source framework that's making experienced ML engineers do a double-take. This isn't another chatbot wrapper or Copilot clone. We're talking about a self-evolving, multi-agent system that reads research papers, proposes novel ideas, implements them in runnable code, and iteratively improves based on real-world feedback. And the kicker? It currently sits at the #1 spot on MLE-bench, outperforming every other ML engineering agent in existence.

Sound insane? The numbers don't lie. Let's pull back the curtain on what might be the most underappreciated weapon in modern AI development.

What is RD-Agent? The Secret Weapon Microsoft Doesn't Want You to Ignore

RD-Agent (Research & Development Agent) is Microsoft's open-source framework designed to fully automate high-value, data-driven R&D processes. Born from the recognition that modern R&D increasingly revolves around two pillars—data and models—this system deploys specialized AI agents that handle everything from literature review to production-ready implementation.

The project emerged from Microsoft's research division and has rapidly evolved from an internal tool to a publicly available powerhouse with active community contributions. Its core philosophy? Let AI drive data-driven AI. Instead of humans babysitting every step of the ML lifecycle, RD-Agent creates a closed-loop system where research agents propose hypotheses and development agents verify them against real data—continuously learning and improving.

What makes RD-Agent genuinely different from the flood of "AI coding assistants" is its architectural separation of concerns. The framework explicitly splits "R" (Research) from "D" (Development)—two specialized agent types that collaborate yet maintain distinct responsibilities. Research agents explore the unknown, proposing novel factors, models, and features. Development agents grind through implementation, testing, and iterative refinement. This isn't monolithic automation; it's orchestrated specialization.

The project has gained serious academic traction too. With papers accepted at ICML 2026, ACL 2026 Findings, and NeurIPS 2025, RD-Agent isn't just engineering hype—it's peer-reviewed innovation. The framework now supports scenarios ranging from quantitative finance to autonomous LLM fine-tuning, medical prediction, and Kaggle competitions.

Want to see it in action? The live demo and comprehensive documentation are available now. The full technical report lives at aka.ms/RD-Agent-Tech-Report.

Key Features That Make RD-Agent a Technical Marvel

Dual-Agent Architecture: Research × Development

The framework's beating heart is its R/D separation. Research agents ("R") use reasoning models like o3 to propose hypotheses and explore solution spaces. Development agents ("D") employ coding-optimized models like GPT-4.1 to implement, test, and refine. This combo achieved 30.22% on MLE-bench's full suite—crushing the previous best of 16.9%.

Multi-Scenario Versatility

RD-Agent isn't a one-trick pony. It ships with production-ready scenarios:

• 💹 Quantitative Finance: Automated factor mining, model co-optimization, and financial report analysis
• 🩺 Medical Prediction: Self-improving diagnostic models with real-world feedback loops
• 🏆 Kaggle Automation: End-to-end competition pipelines with feature engineering
• 📚 Research Copilot: Paper-to-code implementation for novel architectures
• 🧠 FT-Agent: Autonomous LLM fine-tuning for domain adaptation

LiteLLM Backend with DeepSeek Support

The latest versions use LiteLLM as the default backend, enabling seamless integration with virtually any LLM provider. Experimental DeepSeek support offers cost-effective, high-performance inference for budget-conscious teams.

Real-Time Monitoring & Web UI

Two interface options: a Streamlit UI for data science scenario logs and a Flask-based Web UI for real-time interaction and trace viewing. Monitor your agent's thought process as it works.

Iterative Self-Improvement

Unlike static code generators, RD-Agent implements feedback-driven evolution. Failed experiments become learning signals. Successful patterns get reinforced. The system literally gets smarter with every iteration.

5 Brutal Real-World Problems RD-Agent Solves Today

1. The "Paper-to-Code" Translation Nightmare

You've read the perfect architecture paper. Three days later, you're still debugging shape mismatches. RD-Agent's general_model scenario accepts arXiv URLs and autonomously implements working PyTorch code—handling tensor dimensions, layer connections, and training loops.

2. Quant Strategy Research Bottlenecks

Hedge funds spend millions on quants who manually test factor hypotheses. RD-Agent(Q) achieves ~2× higher annualized returns than benchmark libraries at under $10 cost, using 70% fewer factors. The alternating factor-model optimization delivers predictive accuracy without overfitting.

3. Kaggle Feature Engineering Drudgery

Top competitions require hundreds of engineered features. RD-Agent's data_science scenario automates feature proposal, implementation, and selection—competing autonomously while you focus on strategy.

4. LLM Fine-Tuning Complexity

Domain adaptation requires data curation, training orchestration, evaluation, and iterative refinement. FT-Agent (ICML 2026) handles this end-to-end with benchmark-driven feedback.

5. Medical Model Stagnation

Healthcare AI often fails because models don't evolve with new data. RD-Agent's medical scenario creates self-improving prediction systems that propose and validate new architectures against patient outcomes.

Step-by-Step Installation & Setup Guide

RD-Agent requires Linux. Windows users should use WSL2 or Docker. Here's the complete setup:

Prerequisites: Docker & Conda

First, ensure Docker is installed and your user can run commands without sudo:

# Verify Docker access
docker run hello-world

Create and activate a Conda environment (Python 3.10 or 3.11 recommended):

# Create isolated environment
conda create -n rdagent python=3.10
conda activate rdagent

Installation: PyPI vs. Source

For users (stable release):

# Install from PyPI—simplest option
pip install rdagent

For developers (latest features, contribution-ready):

# Clone and install in development mode
git clone https://github.com/microsoft/RD-Agent
cd RD-Agent
make dev

Health Check

Verify your installation before proceeding:

# Check Docker and UI port availability
rdagent health_check --no-check-env

Critical: LLM Configuration

RD-Agent requires ChatCompletion, json_mode, and embedding capabilities. Create a .env file with your preferred backend.

Option 1: OpenAI via LiteLLM (Recommended)

# Unified configuration for chat and embedding
cat << EOF > .env
CHAT_MODEL=gpt-4o
EMBEDDING_MODEL=text-embedding-3-small
OPENAI_API_BASE=<your_unified_api_base>
OPENAI_API_KEY=<replace_with_your_openai_api_key>
EOF

Option 2: DeepSeek (Cost-Optimized)

# DeepSeek for chat + SiliconFlow for embeddings
cat << EOF > .env
# CHAT MODEL: DeepSeek official API
CHAT_MODEL=deepseek/deepseek-chat
DEEPSEEK_API_KEY=<replace_with_your_deepseek_api_key>

# EMBEDDING MODEL: SiliconFlow (DeepSeek has no embedding)
EMBEDDING_MODEL=litellm_proxy/BAAI/bge-m3
LITELLM_PROXY_API_KEY=<replace_with_your_siliconflow_api_key>
LITELLM_PROXY_API_BASE=https://api.siliconflow.cn/v1
EOF

For reasoning models with <think> tags (o1, o3, DeepSeek-R1):

# Strip reasoning chains from responses
export REASONING_THINK_RM=True

Validate Configuration

# Full environment validation—required before first run
rdagent health_check

REAL Code Examples from Microsoft's Repository

These aren't toy examples—they're production commands from the official README, explained in depth.

Example 1: Automated Quant Trading with Factor-Model Co-Evolution

This is RD-Agent's flagship scenario—the one that delivers 2× returns in quant finance:

# Launch the full quant pipeline: factors AND models evolve together
# Integrates with Microsoft Qlib for backtesting infrastructure
rdagent fin_quant

What's happening under the hood? The system spawns coordinated R and D agents. Research agents propose novel alpha factors from financial data; development agents implement them in Qlib-compatible format. Simultaneously, model agents propose architecture variants. The alternating optimization ensures factors and models improve synergistically—not in isolation. This joint evolution is why RD-Agent(Q) outperforms libraries that optimize factors or models separately.

Example 2: Research Paper to Working Implementation

Transform any arXiv paper into executable PyTorch:

# General usage: pass any paper URL
rdagent general_model <Your paper URL>

# Concrete example: implement a specific transformer variant
rdagent general_model "https://arxiv.org/pdf/2210.09789"

The magic here? The agent downloads the PDF, extracts architectural diagrams and mathematical descriptions, then generates a complete training pipeline—not just model stubs. It handles data loading, preprocessing, training loops, and evaluation metrics. For researchers, this means instant reproducibility of published results. For practitioners, it means rapid prototyping of academic innovations.

Example 3: Kaggle Competition Automation

Full pipeline for tabular competitions with automated feature engineering:

# Step 1: Prepare data directory
mkdir -p ./git_ignore_folder/ds_data

# Step 2: Configure environment variables
dotenv set DS_LOCAL_DATA_PATH "$(pwd)/git_ignore_folder/ds_data"
dotenv set DS_CODER_ON_WHOLE_PIPELINE True
dotenv set DS_IF_USING_MLE_DATA True
dotenv set DS_SAMPLE_DATA_BY_LLM True
dotenv set DS_SCEN rdagent.scenarios.data_science.scen.KaggleScen

# Step 3: Launch autonomous competition agent
rdagent data_science --competition tabular-playground-series-dec-2021

Critical prerequisites for Kaggle: You must join the competition on Kaggle's website, create an API token at Settings → Create New Token, and place kaggle.json in ~/.config/kaggle/ with chmod 600 permissions. The agent then autonomously downloads data, proposes features, implements models, and submits predictions—iterating based on leaderboard feedback.

Example 4: Medical Prediction with Custom Datasets

# Step 1: Download and extract medical dataset
wget https://github.com/SunsetWolf/rdagent_resource/releases/download/ds_data/arf-12-hours-prediction-task.zip
unzip arf-12-hours-prediction-task.zip -d ./git_ignore_folder/ds_data/

# Step 2: Configure for medical scenario (note: DS_IF_USING_MLE_DATA=False)
dotenv set DS_LOCAL_DATA_PATH "$(pwd)/git_ignore_folder/ds_data"
dotenv set DS_CODER_ON_WHOLE_PIPELINE True
dotenv set DS_IF_USING_MLE_DATA False
dotenv set DS_SAMPLE_DATA_BY_LLM False
dotenv set DS_SCEN rdagent.scenarios.data_science.scen.DataScienceScen

# Step 3: Launch medical model evolution
rdagent data_science --competition arf-12-hours-prediction-task

Key difference from Kaggle: Medical scenarios use DS_IF_USING_MLE_DATA=False and disable LLM-based sampling. The agent works with your local dataset directly, proposing clinically relevant features and model architectures. The DS_CODER_ON_WHOLE_PIPELINE=True ensures the agent controls preprocessing, feature engineering, model selection, and ensembling—not just the final classifier.

Example 5: Monitoring with Streamlit UI

# Launch real-time log viewer for data science scenarios
rdagent ui --port 19899 --log-dir log/ --data-science

Why this matters: Autonomous agents are black boxes unless you can inspect their reasoning. The Streamlit UI exposes every decision point—which features were proposed, why a model was rejected, how performance evolved. Use --data-science for competition/medical logs; omit it for other scenarios.

Advanced Usage & Best Practices

Cost Optimization with Model Routing

The MLE-bench winning configuration uses o3 for research + GPT-4.1 for development—balancing reasoning quality with implementation speed. Replicate this:

# In .env: assign different models to R and D roles
CHAT_MODEL=o3  # Research: deep reasoning
# Override at scenario level for D agents with cost-effective models

Port Management for Multiple Agents

# Check port availability before launching UI
rdagent health_check --no-check-env --no-check-docker

# Use custom ports to run multiple monitoring instances
rdagent ui --port 19900 --log-dir log/experiment_2/ --data-science

Web UI for Non-Data-Science Scenarios

# Build React frontend for Flask backend
cd web && npm install && npm run build:flask

# Serve with real-time APIs
rdagent server_ui --port 19899
# Access at http://127.0.0.1:19899

Note: The Web UI currently excludes data_science—use Streamlit for those scenarios.

Reproducibility with Execution Traces

Download demo traces to study successful agent behaviors. Analyze what proposals worked, which failed, and how feedback drove evolution.

RD-Agent vs. The Competition: Why It's Not Even Close

The verdict? RD-Agent is the only framework that combines top-tier benchmark performance with industrial scenario coverage, academic rigor, and genuine architectural innovation. AIDE excels at pure ML engineering but lacks domain specialization. AutoGPT and Devin don't compete in the same league for data-driven R&D.

FAQ: What Developers Actually Ask

Is RD-Agent free for commercial use?

Yes—released under MIT license. However, you pay for LLM API usage (OpenAI, Azure, DeepSeek, etc.). The quant scenario runs under $10 per experiment.

Can I run this on Windows or macOS?

Linux only. Use WSL2 on Windows or Docker Desktop. The CI tests Python 3.10 and 3.11 specifically.

How does RD-Agent differ from GitHub Copilot?

Copilot completes your current file. RD-Agent designs complete systems—reading papers, proposing architectures, implementing full pipelines, and iteratively improving based on execution feedback. It's autonomous R&D, not autocomplete.

What LLM providers work best?

o3/GPT-4.1 combo wins benchmarks. DeepSeek offers 90% performance at 20% cost for chat. Any LiteLLM-supported model works for experimentation.

Is my data sent to Microsoft?

No—RD-Agent is self-hosted. Your data goes to your configured LLM provider (OpenAI, Azure, etc.), not Microsoft. Review each provider's privacy policy.

How do I contribute or report issues?

Join the Discord or WeChat group. Code contributions welcome—search grep -r "TODO:" for starter issues.

Can RD-Agent replace my data science team?

Not yet—it's a force multiplier, not replacement. It handles repetitive R&D cycles, letting humans focus on problem framing, business logic, and final validation. The MLE-bench results show it's competitive with experienced ML engineers on 2-hour tasks.

Conclusion: The Future of R&D Is Already Here

After dissecting every line of RD-Agent's codebase and running scenarios across quant finance, medical prediction, and paper implementation, one thing is clear: this is the most sophisticated autonomous R&D system available today.

The dual-agent architecture isn't marketing fluff—it's a genuine paradigm shift that delivers measurable results. The 30.22% MLE-bench score. The 2× quant returns. The ICML/NeurIPS/ACL acceptances. These aren't vanity metrics; they're proof that structured agent collaboration outperforms monolithic approaches.

But here's what excites me most: RD-Agent is designed to evolve. Every scenario you run, every feedback loop you close, contributes to a growing library of automated R&D patterns. The framework gets better as the community uses it.

My recommendation? Start with the quant or paper-implementation scenario—they're the most polished and deliver immediate value. Configure LiteLLM with your preferred provider, run rdagent health_check, and launch your first autonomous experiment.

The age of AI-driven R&D isn't coming. It's already running on GitHub, documented on ReadTheDocs, and waiting for your next breakthrough idea.

What will you automate first?

Ready to transform your R&D workflow? Star the repository, join the Discord community, and deploy your first autonomous agent today.