MLflow 概述
MLflow 是一个开源平台,旨在帮助机器学习从业者和团队处理机器学习过程中的复杂性。 它提供了一个追踪功能,通过捕获应用程序服务执行的详细信息,增强了您生成式 AI 应用程序中的 LLM 可观察性。追踪功能能够记录请求的每个中间步骤的输入、输出和元数据,使您能够轻松地定位错误和意外行为的来源。
功能
- 追踪仪表板:通过包含跨度(span)的输入、输出和元数据的详细仪表板,监控您的 crewAI agent 的活动。
- 自动追踪:与 crewAI 完全自动化的集成,可通过运行
mlflow.crewai.autolog()启用。 - 轻松实现手动追踪插桩:通过 MLflow 的高级流畅 API(如装饰器、函数包装器和上下文管理器)自定义追踪插桩。
- OpenTelemetry 兼容性:MLflow Tracing 支持将追踪数据导出到 OpenTelemetry Collector,然后可用于将追踪数据导出到各种后端,如 Jaeger、Zipkin 和 AWS X-Ray。
- 打包和部署 Agent:将您的 crewAI agent 打包并部署到具有多种部署目标的推理服务器。
- 安全托管 LLM:通过 MLflow 网关,在统一的端点中托管来自不同提供商的多个 LLM。
- 评估:使用便捷的 API
mlflow.evaluate(),通过广泛的指标来评估您的 crewAI agent。
设置说明
1
安装 MLflow 包
复制
询问 AI
# The crewAI integration is available in mlflow>=2.19.0
pip install mlflow
2
启动 MLflow 追踪服务器
复制
询问 AI
# This process is optional, but it is recommended to use MLflow tracking server for better visualization and broader features.
mlflow server
3
在您的应用程序中初始化 MLflow
在您的应用程序代码中添加以下两行追踪 CrewAI Agent 的用法示例有关更多配置和用例,请参阅 MLflow 追踪文档。
复制
询问 AI
import mlflow
mlflow.crewai.autolog()
# Optional: Set a tracking URI and an experiment name if you have a tracking server
mlflow.set_tracking_uri("https://:5000")
mlflow.set_experiment("CrewAI")
复制
询问 AI
from crewai import Agent, Crew, Task
from crewai.knowledge.source.string_knowledge_source import StringKnowledgeSource
from crewai_tools import SerperDevTool, WebsiteSearchTool
from textwrap import dedent
content = "Users name is John. He is 30 years old and lives in San Francisco."
string_source = StringKnowledgeSource(
content=content, metadata={"preference": "personal"}
)
search_tool = WebsiteSearchTool()
class TripAgents:
def city_selection_agent(self):
return Agent(
role="City Selection Expert",
goal="Select the best city based on weather, season, and prices",
backstory="An expert in analyzing travel data to pick ideal destinations",
tools=[
search_tool,
],
verbose=True,
)
def local_expert(self):
return Agent(
role="Local Expert at this city",
goal="Provide the BEST insights about the selected city",
backstory="""A knowledgeable local guide with extensive information
about the city, it's attractions and customs""",
tools=[search_tool],
verbose=True,
)
class TripTasks:
def identify_task(self, agent, origin, cities, interests, range):
return Task(
description=dedent(
f"""
Analyze and select the best city for the trip based
on specific criteria such as weather patterns, seasonal
events, and travel costs. This task involves comparing
multiple cities, considering factors like current weather
conditions, upcoming cultural or seasonal events, and
overall travel expenses.
Your final answer must be a detailed
report on the chosen city, and everything you found out
about it, including the actual flight costs, weather
forecast and attractions.
Traveling from: {origin}
City Options: {cities}
Trip Date: {range}
Traveler Interests: {interests}
"""
),
agent=agent,
expected_output="Detailed report on the chosen city including flight costs, weather forecast, and attractions",
)
def gather_task(self, agent, origin, interests, range):
return Task(
description=dedent(
f"""
As a local expert on this city you must compile an
in-depth guide for someone traveling there and wanting
to have THE BEST trip ever!
Gather information about key attractions, local customs,
special events, and daily activity recommendations.
Find the best spots to go to, the kind of place only a
local would know.
This guide should provide a thorough overview of what
the city has to offer, including hidden gems, cultural
hotspots, must-visit landmarks, weather forecasts, and
high level costs.
The final answer must be a comprehensive city guide,
rich in cultural insights and practical tips,
tailored to enhance the travel experience.
Trip Date: {range}
Traveling from: {origin}
Traveler Interests: {interests}
"""
),
agent=agent,
expected_output="Comprehensive city guide including hidden gems, cultural hotspots, and practical travel tips",
)
class TripCrew:
def __init__(self, origin, cities, date_range, interests):
self.cities = cities
self.origin = origin
self.interests = interests
self.date_range = date_range
def run(self):
agents = TripAgents()
tasks = TripTasks()
city_selector_agent = agents.city_selection_agent()
local_expert_agent = agents.local_expert()
identify_task = tasks.identify_task(
city_selector_agent,
self.origin,
self.cities,
self.interests,
self.date_range,
)
gather_task = tasks.gather_task(
local_expert_agent, self.origin, self.interests, self.date_range
)
crew = Crew(
agents=[city_selector_agent, local_expert_agent],
tasks=[identify_task, gather_task],
verbose=True,
memory=True,
knowledge={
"sources": [string_source],
"metadata": {"preference": "personal"},
},
)
result = crew.kickoff()
return result
trip_crew = TripCrew("California", "Tokyo", "Dec 12 - Dec 20", "sports")
result = trip_crew.run()
print(result)
4
可视化 Agent 的活动
现在,您的 crewAI agent 的追踪数据已被 MLflow 捕获。让我们访问 MLflow 追踪服务器以查看追踪数据并深入了解您的 Agent。在浏览器中打开 
127.0.0.1:5000 访问 MLflow 追踪服务器。
MLflow 追踪仪表板
