Diuse/zhenxun_bot
1
0
Fork 0
mirror of https://github.com/zhenxun-org/zhenxun_bot.git synced 2026-10-22 12:40:09 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

♻️ refactor(tools): 重构工具装饰器系统并优化沙箱与文档注释 (#2147)

Browse Source 
* ♻️ refactor(ai): 重构 AI 服务模块并完善文档注释

- 【重构】统一清理并优化所有 AI 服务模块文件的导入语句，将其移至文件顶部
- 【重构】重构 `hooks.py` 中的 `Hooks` 派发逻辑，使用通用管道函数消除重复代码，并引入 `HookPoint` 描述符
- 【重构】重构工具装饰器实现，新增 `toolkit` 类装饰器，优化 `BaseToolkit` 配置合并与前缀处理
- 【功能】Docker 沙箱容器创建时支持自动注入系统代理环境变量并配置 `ExtraHosts`
- 【功能】Jupyter 服务启动前自动清理旧进程并初始化临时目录权限
- 【修复】优化 Pydantic 结构化输出校验失败时的错误信息提取，提供更详细的字段级错误反馈
- 【修复】在 `api.py` 中避免将 `ModelRetry` 和 `ControlFlowExit` 异常错误地包装为 `LLMException`
- 【文档】为 AI 服务、沙箱、工具链、工作流等核心模块补充完整的 Docstring 和类型注释

* 📝 docs(ai): 补全核心模块文档注释并清理冗余代码

- 补全 `run/context`、`run/hooks` 和 `tools/engine/registry` 中类与方法的中文文档注释
- 清理 `tools/providers/builtin/sandbox` 中未使用的 `PythonPluginProtocol` 协议及相关导入
- 规范化部分代码的格式与尾随逗号

* ♻️ refactor!(flow): 重构 Task 为 AgentTask 并优化工作流元数据定义

- 【Breaking Change】将 `Task` 重命名为 `AgentTask` 以避免命名冲突
- 更新 Agent、Team、Workflow 等模块中的类型声明与相关逻辑
- 引入 `AutoNodeMeta` 强类型元数据，替换工作流装饰器中的裸字典定义
- 将 `StepMeta`、`ConditionMeta` 和 `RouterMeta` 统一移动至 `types.py`
- 优化 `RunnableNode` 对上游 `AgentTask` 的处理与拼接逻辑
- 调整团队协作策略中 `FinishAction` 的返回值为完整结果对象

* ♻️ refactor(workflow): 移除人工确认机制并重构错误策略

- 移除工作流节点的人工确认（HITL）与挂起继续机制
- 删除 `auto` 自动化工作流及相关装饰器文件
- 将错误处理策略类从 `types.py` 拆分并移动到新文件 `policies.py`
- 优化节点执行失败时的异常信息格式化输出
- 移除 `WorkflowRunResult` 和 `StepOutput` 中与挂起相关的状态字段

* 🚨 auto fix by pre-commit hooks

---------

Co-authored-by: webjoin111 <455457521@qq.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Rumio
2026-07-05 11:47:21 +08:00
committed by GitHub
co-authored by webjoin111 pre-commit-ci[bot]
parent 80fc5b86a7
commit 0b32d69c9c
61 changed files with 1659 additions and 1511 deletions
Download Patch File Download Diff File Expand all files Collapse all files
zhenxun/services/ai/context/memory/storage/backends.py
+69 -41
View File
@@ -2,6 +2,7 @@ import asyncio
import base64
from collections.abc import Callable
import datetime
from pathlib import Path
import time
from typing import TYPE_CHECKING, Any, cast
@@ -26,6 +27,7 @@ from zhenxun.services.ai.core.messages import (
LLMContentPart,
LLMMessage,
SystemMessage,
TextPart,
ToolMessage,
UserMessage,
)
@@ -34,13 +36,47 @@ from zhenxun.services.db_context import Model
from zhenxun.utils.pydantic_compat import TypeAdapter, model_dump
class AbstractMemoryRecord(Model):
"""Tortoise ORM 短期记忆持久化基类 (Mixin)。"""
id = fields.UUIDField(pk=True, description="主键")
session_id = fields.CharField(max_length=255, index=True)
role = fields.CharField(max_length=32)
content = fields.JSONField()
api_context = fields.JSONField(null=True)
created_at = fields.DatetimeField(auto_now_add=True)
metadata = fields.JSONField(null=True)
class Meta: # type: ignore
abstract = True
class AbstractSlotRecord(Model):
"""Tortoise ORM 记忆槽持久化基类 (Mixin)。"""
id = fields.CharField(
pk=True, max_length=128, description="复合主键: session_id + label"
)
session_id = fields.CharField(max_length=255, index=True)
label = fields.CharField(max_length=64, index=True)
content = fields.TextField()
size_limit = fields.IntField(default=2000)
pinned = fields.BooleanField(default=True)
scope = fields.CharField(max_length=255)
description = fields.CharField(max_length=255, default="")
created_at = fields.FloatField()
updated_at = fields.FloatField()
class Meta: # type: ignore
abstract = True
class DBMessageSerializer:
"""将 LLMMessage 与数据库 JSON 格式进行序列化/反序列化的帮助类"""
@staticmethod
def deserialize_content(content_raw: Any) -> list[LLMContentPart]:
from zhenxun.services.ai.core.messages import TextPart
"""反序列化数据库中的 JSON 数据为 LLMMessage 消息内容部件列表"""
content_parts: list[LLMContentPart] = []
if isinstance(content_raw, list):
adapter = TypeAdapter(LLMContentPart)
@@ -59,8 +95,7 @@ class DBMessageSerializer:
@staticmethod
def serialize_content(content_payload: Any) -> list[dict[str, Any]]:
from pathlib import Path
"""将 LLMMessage 消息内容序列化为可存储于数据库的 JSON 格式"""
if isinstance(content_payload, str):
return [{"type": "text", "text": content_payload}]
elif isinstance(content_payload, list):
@@ -96,6 +131,7 @@ class MemoryScope:
self,
rag_client: "ScopedRAGClient",
):
"""初始化长期记忆作用域与 RAG 客户端"""
self.rag_client = rag_client
self._background_tasks: set[Any] = set()
@@ -164,13 +200,13 @@ class MemoryScope:
async def forget(
self, session: SessionMetadata, record_ids: list[str] | None = None
) -> int:
"""从 RAG 向量数据库中删除指定的记忆记录"""
return await self.rag_client.delete(
record_ids=record_ids,
)
async def _reinforce_memories(self, records: list[BaseRecord]):
import time
"""惰性强化记忆：更新被检索记忆的访问次数和最后访问时间"""
now = time.time()
for r in records:
r.metadata["access_count"] = r.metadata.get("access_count", 0) + 1
@@ -179,15 +215,20 @@ class MemoryScope:
class InMemoryChatContext(BaseChatContext):
"""基于内存的聊天上下文存储后端"""
def __init__(self):
"""初始化内存聊天上下文"""
self._messages: dict[str, list[LLMMessage]] = {}
async def get_messages(self, session: SessionMetadata) -> list[LLMMessage]:
"""获取指定会话的所有短期历史消息"""
return list(self._messages.get(session.session_id, []))
async def search(
self, query: str, session: SessionMetadata, limit: int = 10
) -> list[LLMMessage]:
"""在内存中简单检索包含查询词的历史消息"""
results = []
for msg in self._messages.get(session.session_id, []):
if query in msg.extract_text:
@@ -199,6 +240,7 @@ class InMemoryChatContext(BaseChatContext):
async def add_messages(
self, session: SessionMetadata, messages: list[LLMMessage]
) -> None:
"""向指定会话中追加历史消息"""
if session.session_id not in self._messages:
self._messages[session.session_id] = []
self._messages[session.session_id].extend(messages)
@@ -206,9 +248,11 @@ class InMemoryChatContext(BaseChatContext):
async def set_messages(
self, session: SessionMetadata, messages: list[LLMMessage]
) -> None:
"""覆盖设置指定会话的历史消息"""
self._messages[session.session_id] = list(messages)
async def clear(self, session: SessionMetadata) -> None:
"""清空指定会话的全部历史消息"""
self._messages.pop(session.session_id, None)
async def clear_by_query(self, query: ScopeSelector) -> None:
@@ -221,22 +265,9 @@ class InMemoryChatContext(BaseChatContext):
self._messages.pop(sid, None)
class AbstractMemoryRecord(Model):
"""Tortoise ORM 短期记忆持久化基类 (Mixin)。"""
id = fields.UUIDField(pk=True, description="主键")
session_id = fields.CharField(max_length=255, index=True)
role = fields.CharField(max_length=32)
content = fields.JSONField()
api_context = fields.JSONField(null=True)
created_at = fields.DatetimeField(auto_now_add=True)
metadata = fields.JSONField(null=True)
class Meta: # type: ignore
abstract = True
class TortoiseChatContext(BaseChatContext):
"""基于 Tortoise ORM 的聊天上下文存储后端"""
def __init__(
self,
model_class: type[AbstractMemoryRecord],
@@ -245,10 +276,12 @@ class TortoiseChatContext(BaseChatContext):
]
| None = None,
):
"""初始化 Tortoise ORM 聊天上下文存储后端"""
self.model_class = model_class
self.custom_save_hook = custom_save_hook
def _row_to_message(self, row: AbstractMemoryRecord) -> LLMMessage:
"""将数据库记录转换为 LLMMessage 实例"""
content_parts = DBMessageSerializer.deserialize_content(row.content)
metadata: dict[str, Any] | None = (
row.metadata if isinstance(row.metadata, dict) else None
@@ -270,6 +303,7 @@ class TortoiseChatContext(BaseChatContext):
return cast(LLMMessage, LLMMessage(role=role, **kwargs))
async def get_messages(self, session: SessionMetadata) -> list[LLMMessage]:
"""从数据库中查询并获取指定会话的短期历史消息"""
rows = (
await self.model_class.filter(session_id=session.session_id)
.order_by("created_at")
@@ -280,6 +314,7 @@ class TortoiseChatContext(BaseChatContext):
async def search(
self, query: str, session: SessionMetadata, limit: int = 10
) -> list[LLMMessage]:
"""在数据库中检索包含查询词的历史消息"""
rows = (
await self.model_class.filter(
session_id=session.session_id, content__icontains=query
@@ -293,6 +328,7 @@ class TortoiseChatContext(BaseChatContext):
async def add_messages(
self, session: SessionMetadata, messages: list[LLMMessage]
) -> None:
"""向数据库中批量追加指定会话的历史消息"""
if not messages:
return
@@ -331,10 +367,12 @@ class TortoiseChatContext(BaseChatContext):
async def set_messages(
self, session: SessionMetadata, messages: list[LLMMessage]
) -> None:
"""覆盖设置指定会话的数据库历史消息"""
await self.clear(session)
await self.add_messages(session, messages)
async def clear(self, session: SessionMetadata) -> None:
"""删除指定会话在数据库中的全部历史消息"""
await self.model_class.filter(session_id=session.session_id).delete()
async def clear_by_query(self, query: ScopeSelector) -> None:
@@ -357,31 +395,15 @@ def get_orm_chat_context(
)
class AbstractSlotRecord(Model):
"""Tortoise ORM 记忆槽持久化基类 (Mixin)。"""
id = fields.CharField(
pk=True, max_length=128, description="复合主键: session_id + label"
)
session_id = fields.CharField(max_length=255, index=True)
label = fields.CharField(max_length=64, index=True)
content = fields.TextField()
size_limit = fields.IntField(default=2000)
pinned = fields.BooleanField(default=True)
scope = fields.CharField(max_length=255)
description = fields.CharField(max_length=255, default="")
created_at = fields.FloatField()
updated_at = fields.FloatField()
class Meta: # type: ignore
abstract = True
class TortoiseSlotContext(BaseSlotContext):
"""基于 Tortoise ORM 的记忆槽存储后端"""
def __init__(self, model_class: type[AbstractSlotRecord]):
"""初始化 Tortoise ORM 记忆槽存储后端"""
self.model_class = model_class
def _row_to_slot(self, row: AbstractSlotRecord) -> MemorySlot:
"""将数据库记忆槽记录转换为 MemorySlot 实例"""
return MemorySlot(
label=row.label,
content=row.content,
@@ -394,6 +416,7 @@ class TortoiseSlotContext(BaseSlotContext):
)
async def get_slot(self, session: SessionMetadata, label: str) -> MemorySlot | None:
"""查询并获取指定会话及作用域下 label 对应的记忆槽"""
rows = await self.model_class.filter(
session_id__in=session.accessible_scopes, label=label
).all()
@@ -405,6 +428,7 @@ class TortoiseSlotContext(BaseSlotContext):
return None
async def set_slot(self, session: SessionMetadata, slot: MemorySlot) -> None:
"""保存或更新指定会话的记忆槽到数据库"""
composite_id = f"{slot.scope}_{slot.label}"
await self.model_class.update_or_create(
@@ -425,10 +449,12 @@ class TortoiseSlotContext(BaseSlotContext):
async def delete_slot(
self, session: SessionMetadata, label: str, scope: str
) -> None:
"""从数据库中删除指定作用域和 label 的记忆槽"""
composite_id = f"{scope}_{label}"
await self.model_class.filter(id=composite_id).delete()
async def list_pinned_slots(self, session: SessionMetadata) -> list[MemorySlot]:
"""获取指定会话所有可访问的、置顶且非空的记忆槽"""
rows = await self.model_class.filter(
session_id__in=session.accessible_scopes, pinned=True
).all()
@@ -442,6 +468,7 @@ class TortoiseSlotContext(BaseSlotContext):
return [s for s in merged.values() if s.content.strip()]
async def list_all_slots(self, session: SessionMetadata) -> list[MemorySlot]:
"""获取指定会话所有可访问的记忆槽列表"""
rows = await self.model_class.filter(
session_id__in=session.accessible_scopes
).all()
@@ -454,6 +481,7 @@ class TortoiseSlotContext(BaseSlotContext):
return list(merged.values())
async def clear_by_query(self, query: ScopeSelector) -> None:
"""批量清理匹配指定前缀的所有记忆槽"""
scope_prefix = query.scope_prefix
await self.model_class.filter(session_id__startswith=scope_prefix).delete()
zhenxun/services/ai/context/rag/ingestion.py
+26 -6
View File
@@ -8,11 +8,15 @@ from zhenxun.services.log import logger
class ChunkingStrategy(ABC):
"""分块策略抽象基类，用于将长文本记录切分为多个短的 BaseRecord"""
@abstractmethod
def chunk(self, record: BaseRecord) -> list[BaseRecord]:
"""将输入的记录切分为子记录列表。由子类具体实现"""
raise NotImplementedError
def clean_text(self, text: str) -> str:
"""清洗和规范化文本，去除多余的空行和空白字符"""
cleaned_text = re.sub(r"\n+", "\n", text)
cleaned_text = re.sub(r"[ \t]+", " ", cleaned_text)
return cleaned_text.strip()
@@ -20,6 +24,7 @@ class ChunkingStrategy(ABC):
def _create_chunk_record(
self, original_record: BaseRecord, chunk_number: int, content: str
) -> BaseRecord:
"""根据原始记录创建分块后的 BaseRecord，并自动附带切片索引和父级ID等元数据"""
meta_data = original_record.metadata.copy()
meta_data["chunk_index"] = chunk_number
meta_data["chunk_size"] = len(content)
@@ -44,6 +49,7 @@ class DocumentChunking(ChunkingStrategy):
self.chunk_size = chunk_size
def chunk(self, record: BaseRecord) -> list[BaseRecord]:
"""按双换行将内容切分为段落，并将相邻段落合并为符合最大字符长度限制的分块"""
if len(record.content) <= self.chunk_size:
return [
self._create_chunk_record(record, 0, self.clean_text(record.content))
@@ -113,7 +119,7 @@ class RecursiveCharacterChunking(ChunkingStrategy):
]
def _split_text(self, text: str, separators: list[str]) -> list[str]:
"""核心递归切分逻辑"""
"""核心递归切分逻辑。尝试使用给定的分隔符列表按优先级切分文本"""
final_chunks = []
separator = separators[-1]
new_separators = []
@@ -187,6 +193,7 @@ class RecursiveCharacterChunking(ChunkingStrategy):
return chunks
def chunk(self, record: BaseRecord) -> list[BaseRecord]:
"""使用递归字符切分方式，将文本切分为带有重叠部分的分块记录"""
content = record.content.strip()
if len(content) <= self.chunk_size:
@@ -219,6 +226,7 @@ class RowChunking(ChunkingStrategy):
self.rows_per_chunk = rows_per_chunk
def chunk(self, record: BaseRecord) -> list[BaseRecord]:
"""将表格行数据按指定行数切分为块，每个块都带有相同的表头首部"""
lines = record.content.splitlines()
lines = [line for line in lines if line.strip()]
@@ -260,6 +268,7 @@ class DeduplicationProcessor:
self.threshold = threshold
async def process(self, records: list[BaseRecord]) -> list[BaseRecord]:
"""对输入记录列表进行批处理内去重，过滤相似度达到或超过阈值的重复记录"""
if not records or len(records) <= 1:
return records
@@ -297,7 +306,9 @@ class BaseBatchNode(ABC):
"""批处理节点基类：一次性接收并处理全部记录"""
@abstractmethod
async def process_batch(self, records: list[BaseRecord]) -> list[BaseRecord]: ...
async def process_batch(self, records: list[BaseRecord]) -> list[BaseRecord]:
"""批量处理 BaseRecord 记录列表"""
...
class BaseMapNode(ABC):
@@ -306,7 +317,9 @@ class BaseMapNode(ABC):
@abstractmethod
async def process_one(
self, record: BaseRecord
) -> BaseRecord | list[BaseRecord] | None: ...
) -> BaseRecord | list[BaseRecord] | None:
"""处理单条 BaseRecord 记录，可返回修改后的记录、拆分后的多条记录，或 None（表示过滤该记录）""" # noqa: E501
...
class DynamicChunkingNode(BaseBatchNode):
@@ -329,6 +342,7 @@ class DynamicChunkingNode(BaseBatchNode):
self.strategies = custom_strategies or {".csv": RowChunking(rows_per_chunk=30)}
async def process_batch(self, records: list[BaseRecord]) -> list[BaseRecord]:
"""根据记录的元数据扩展名动态匹配并执行切块策略"""
chunks = []
for record in records:
ext = record.metadata.get("extension", "")
@@ -338,7 +352,7 @@ class DynamicChunkingNode(BaseBatchNode):
class BaseEmbeddingBatchNode(BaseBatchNode):
"""批量向量化抽象基类：提取文本、分批请求 API 并将结果写回的公共逻辑"""
"""批量向量化抽象基类：提取文本、分批请求 API 并将结果 write 回的公共逻辑"""
def __init__(self, embedder, batch_size: int = 80):
"""
@@ -357,6 +371,7 @@ class BaseEmbeddingBatchNode(BaseBatchNode):
pass
async def process_batch(self, records: list[BaseRecord]) -> list[BaseRecord]:
"""批量将文本记录提取并请求向量化接口，写回向量数据"""
if not self.embedder or not records:
return records
@@ -382,14 +397,15 @@ class BaseEmbeddingBatchNode(BaseBatchNode):
class EmbeddingNode(BaseEmbeddingBatchNode):
"""并发向量化初次构建节点"""
"""并发向量化初次构建节点，只对有实际内容的记录进行向量化。"""
def _filter_target_records(self, records: list[BaseRecord]) -> list[BaseRecord]:
"""筛选出非空内容的记录进行向量化"""
return [r for r in records if r.content.strip()]
class DedupNode(BaseBatchNode):
"""批次内查重节点"""
"""批次内查重节点。在流水线中作为去重节点使用。"""
def __init__(self, threshold: float):
"""
@@ -401,6 +417,7 @@ class DedupNode(BaseBatchNode):
self.processor = DeduplicationProcessor(threshold=threshold)
async def process_batch(self, records: list[BaseRecord]) -> list[BaseRecord]:
"""调用去重处理器过滤本批次中的高度重复记录"""
return await self.processor.process(records)
@@ -417,6 +434,7 @@ class StorageCommitNode(BaseBatchNode):
self.storage = storage
async def process_batch(self, records: list[BaseRecord]) -> list[BaseRecord]:
"""根据记录的操作类型（保存、更新或删除）分类，并批量提交至存储后端"""
if not records:
return records
@@ -468,9 +486,11 @@ class IndexPipeline:
self.max_workers = max_workers
def add_node(self, node: BaseBatchNode | BaseMapNode):
"""向处理流水线中追加一个节点"""
self.nodes.append(node)
async def run(self, records: list[BaseRecord]) -> list[BaseRecord]:
"""并发调度处理引擎，运行并执行流水线中的所有处理节点，返回处理后的记录"""
if not records:
return []