提交 b21f2fe9 authored 作者: 陈泽健's avatar 陈泽健

fix(performance): 全库扫描 order_by 兜底排查——26 处两段式改造防 MySQL 1038

针对含大 JSON/TEXT 列 + 整行 ORDER BY 的列表查询做两段式改造(防 MySQL 1038 Out of sort memory),含非 performance 模块:
- services 12 文件:case/api_test/execution(2处)/report/scheduler/security/security_report/deploy/device_sim/login_template/api_preset/performance(漏网点2处)
- routers 5 文件:report_template/reports/scheduled_tasks/security_scheduled_tasks/stats
- 白名单 13 处归档(聚合查询/无大列表),pytest 423 passed 零回归
Co-Authored-By: 's avatarClaude <noreply@anthropic.com>
上级 bab0b25b
......@@ -3,13 +3,81 @@
> **生成时间**: 2026-09-07
> **当前分支**: `platform-auto-test`
> **最近提交**: `67d87399` fix(performance): MySQL 1038 整行排序爆内存修复收官 + 截图清理闭环(7 处两段式 / 真实混合场景 E2E 确认目标机曲线)(已推送 origin/platform-auto-test)
> **会话窗口**: 性能测试 — 待办闭环 + 提交推送闭环(截图清理坏测试修复+部署 / MySQL 1038 三轮修复 7 处 / 目标机曲线 E2E 确认 / `67d87399` 提交推送 ✅
> **状态**: ✅ 全部闭环:三项待办 + 代码提交推送(`67d87399`,10 文件 +825/-59)+ 5.60 热修复已部署验证(report 接口 200)
> **会话窗口(上一窗口)**: 2026-09-07 早 目标机监控开启 + X 轴修复(`0aa655a5`,已推送部署
> **会话窗口**: 性能测试 — P2 待办①全库扫描 order_by 兜底排查闭环(26 处两段式修复 / 17 文件 +425/-193 + 白名单 13 处归档,**未提交待 /GitCommit**
> **状态**: ⏳ 代码已改完并全量回归通过(423 passed),**改动未提交**,待用户确认后 /GitCommit;未部署 5.60
> **会话窗口(上一窗口)**: 2026-09-07 续2 提交推送闭环(`67d87399`)+ 2026-09-07 早 目标机监控开启 + X 轴修复(`0aa655a5`
---
## ⚡ 最新会话更新(2026-09-07 续2)— 代码提交推送闭环 ✅
## ⚡ 最新会话更新(2026-09-07 续3)— P2 待办①全库扫描 order_by 兜底排查闭环 ✅
### A. 排查范围与方法
- 全库扫描 `backend/app` 35 个含 `select(` 的文件,提取全部 35 处 `select(...).order_by(...)`
- 判断口诀:**「表含大 JSON/TEXT 列 + 整行 ORDER BY」即风险,与列表/详情/有无 LIMIT 无关**
- 修复模式统一沿用 performance 模块已固化的两段式:第一段 `select(Model.id)` 轻量排序分页,第二段 `id.in_(ids)` 回查整行并按 ids 还原顺序。
### B. 修复清单(24 处 / 16 文件,另 performance 漏网点 2 处 = 共 26 处 / 17 文件)
| # | 文件 | 表 / 大列 | 场景 |
|---|------|-----------|------|
| 1 | `services/case_service.py` | test_cases(steps/config/parameters/condition JSON) | 用例列表 |
| 2 | `services/api_test_service.py` | test_cases | API 用例列表 |
| 3 | `services/api_test_service.py` | case_results(steps_result JSON 大列) | 结果列表 |
| 4 | `services/api_test_service.py` | executions(config JSON) | 执行列表 |
| 5 | `services/execution_service.py` | executions | 执行列表 |
| 6 | `services/execution_service.py` | test_cases | create_execution 批量取用例 |
| 7-8 | `services/execution_service.py` ×2 | case_results + JOIN TestCase | 执行引擎取 pending 结果(join 仅保留在第一段排序) |
| 9 | `services/execution_service.py` | case_results | get_case_results(详情/报告) |
| 10 | `services/report_service.py` | case_results | 报告生成取全部结果 |
| 11 | `services/scheduler_service.py` | test_cases | 定时任务批量取用例 |
| 12 | `services/security_service.py` | vulnerability_results(request_info/response_info 大 TEXT) | 漏洞列表 |
| 13 | `services/security_report_service.py` | vulnerability_results | 安全报告取全部漏洞 |
| 14 | `services/deploy_service.py` | deploy_logs(content 大 TEXT 日志) | 部署日志列表 |
| 15-16 | `services/device_sim_service.py` ×2 | device_simulators / report_logs(payload JSON) | 设备列表 / 上报日志 |
| 17 | `services/login_template_service.py` | login_templates(steps JSON) | 模板列表 |
| 18 | `services/api_preset_service.py` | api_presets(body/capture_rules JSON) | 预设列表 |
| 19-20 | `routers/report_template.py` ×2 | report_templates(config 大 TEXT) | 模板列表 / 删除默认模板回退 |
| 21 | `routers/reports.py` | executions | 报告可选执行列表 |
| 22 | `routers/scheduled_tasks.py` | executions | 定时任务运行历史 |
| 23 | `routers/security_scheduled_tasks.py` | executions | 安全定时任务运行历史 |
| 24 | `routers/stats.py` | executions | Dashboard 最近执行 limit(5)(LIMIT 不豁免) |
**⚡ performance 模块漏网点 2 处(本轮新发现)**`performance_service.py``get_project_report()` / `list_project_reports()` —— `performance_project_reports` 表含 `task_summaries` 大 TEXT(任务摘要 JSON 数组),前 7 处修复时遗漏。`get_batch_report()` 按 batch_id 直查无 ORDER BY,无需改。
**说明**:修复清单 1-24 全部为「两段式」改造(`select(id)` 排序分页 → `id.in_()` 回查整行还原顺序);上表行 7-8 两处 join 仅保留在第一段排序。**变更规模**:17 文件 +425/-193(`git diff --stat`)。
### C. 白名单(确认安全,不改 + 理由)
| 位置 | 理由 |
|------|------|
| `stats.py` 趋势/模块统计 ×2 | 聚合查询,只取聚合列,无大列 |
| `module_service` / `project_service` / `document_service` / `dependency_service` | 表无大 JSON/TEXT 列(description 短文本)或按 case_id 过滤行数极少 |
| `scheduled_tasks` / `security_scheduled_tasks` 任务列表 ×2 | ScheduledTask 仅 module_ids/weekdays 小 JSON 数组,任务数少 |
| `deploy_service` 服务器/脚本/部署执行 ×3 | DeployServer(ssh_key 数 KB)/DeployScript/DeployExecution(error_message 短文本),行数少 |
| `device_sim_service` EnvConfig | 环境配置行数极少 |
| `security_service` SecurityConfig | 4 个 JSON 但为个位数配置行 |
### D. 验证
| 项 | 结果 |
|----|------|
| `py_compile` 17 个修改文件 | ✅ 全过 |
| 复查脚本再扫整行 select+order_by | ✅ 仅剩白名单 13 处 |
| 全量 `pytest tests/ -q` | ✅ **423 passed**(52s,零回归) |
| `git diff --stat` | ✅ 17 文件 +425/-193 |
### E. 剩余待办(更新)
| # | 任务 | 优先级 | 说明 |
|---|------|--------|------|
| 1 | **本窗口 17 文件提交推送** | P1 | 待用户 /GitCommit(纯后端 services/routers,无前端);提交后按需部署 5.60 |
| 2 | performance 大 JSON 列拆表 | P3 | csv_content / *_summary / task_summaries 长期方案(任务/执行行只留指针) |
| 3 | 跟踪其他窗口 progress | — | recorder/projects 为其他窗口工作,合并时注意冲突 |
---
## ⚡ 会话更新(2026-09-07 续2)— 代码提交推送闭环 ✅
### F. 提交推送(/GitCommit 三重门控)
......
......@@ -74,8 +74,21 @@ async def list_templates(db: AsyncSession = Depends(get_db)):
Returns:
List[ReportTemplateListItem]: 模板列表
"""
result = await db.execute(select(ReportTemplate).order_by(ReportTemplate.updated_at.desc()))
templates = result.scalars().all()
# 两段式防 MySQL 1038:report_templates 表含 config 大 TEXT(完整 JSON),
# 整行 ORDER BY 会撑爆 sort buffer。第一段只取 id 轻量排序,第二段按
# id 回查整行并还原顺序。
ids = list(
(
await db.execute(
select(ReportTemplate.id).order_by(ReportTemplate.updated_at.desc())
)
).scalars().all()
)
templates = []
if ids:
rows = await db.execute(select(ReportTemplate).where(ReportTemplate.id.in_(ids)))
by_id = {t.id: t for t in rows.scalars().all()}
templates = [by_id[i] for i in ids if i in by_id]
return [t.to_list_item() for t in templates]
......@@ -247,9 +260,15 @@ async def delete_template(template_id: int, db: AsyncSession = Depends(get_db)):
logger.info(f"删除报告模板: {template.name} (id={template_id})")
# 如果删除的是默认模板,把剩余的第一个设为默认
# 两段式防 MySQL 1038:config 大 TEXT 不参与排序,第一段只取 id
if was_default:
remaining = await db.execute(select(ReportTemplate).order_by(ReportTemplate.updated_at.desc()))
first = remaining.scalars().first()
first_id = (
await db.execute(
select(ReportTemplate.id).order_by(ReportTemplate.updated_at.desc()).limit(1)
)
).scalar_one_or_none()
if first_id is not None:
first = await db.get(ReportTemplate, first_id)
if first:
first.is_default = True
await db.commit()
......
......@@ -152,15 +152,25 @@ async def list_reports(
if case_type:
conditions.append(Execution.case_type == case_type)
query = (
select(Execution)
# 两段式防 MySQL 1038:executions 表含 config JSON / error_message TEXT
# 列,整行 ORDER BY 会撑爆 sort buffer。第一段只取 id 轻量排序分页,
# 第二段按 id 回查整行并还原顺序。
ids = list(
(
await db.execute(
select(Execution.id)
.where(*conditions)
.order_by(desc(Execution.created_at))
.offset(skip)
.limit(limit)
)
result = await db.execute(query)
items = result.scalars().all()
).scalars().all()
)
items = []
if ids:
rows = await db.execute(select(Execution).where(Execution.id.in_(ids)))
by_id = {e.id: e for e in rows.scalars().all()}
items = [by_id[i] for i in ids if i in by_id]
count_query = select(func.count(Execution.id)).where(*conditions)
result = await db.execute(count_query)
......
......@@ -325,8 +325,13 @@ async def list_task_runs(
if not task:
raise HTTPException(status_code=404, detail=f"定时任务不存在: {task_id}")
query = (
select(Execution)
# 两段式防 MySQL 1038:executions 表含 config JSON / error_message TEXT
# 列,整行 ORDER BY 会撑爆 sort buffer。第一段只取 id 轻量排序分页,
# 第二段按 id 回查整行并还原顺序。
ids = list(
(
await db.execute(
select(Execution.id)
.where(
Execution.trigger_type == "scheduled",
Execution.trigger_by == task.name,
......@@ -335,8 +340,13 @@ async def list_task_runs(
.offset(skip)
.limit(limit)
)
result = await db.execute(query)
executions = list(result.scalars().all())
).scalars().all()
)
executions = []
if ids:
rows = await db.execute(select(Execution).where(Execution.id.in_(ids)))
by_id = {e.id: e for e in rows.scalars().all()}
executions = [by_id[i] for i in ids if i in by_id]
return {
"total": len(executions),
......
......@@ -296,8 +296,13 @@ async def list_task_runs(
if not task:
raise HTTPException(status_code=404, detail=f"定时任务不存在: {task_id}")
query = (
select(Execution)
# 两段式防 MySQL 1038:executions 表含 config JSON / error_message TEXT
# 列,整行 ORDER BY 会撑爆 sort buffer。第一段只取 id 轻量排序分页,
# 第二段按 id 回查整行并还原顺序。
ids = list(
(
await db.execute(
select(Execution.id)
.where(
Execution.trigger_type == "scheduled",
Execution.trigger_by == task.name,
......@@ -307,8 +312,13 @@ async def list_task_runs(
.offset(skip)
.limit(limit)
)
result = await db.execute(query)
executions = list(result.scalars().all())
).scalars().all()
)
executions = []
if ids:
rows = await db.execute(select(Execution).where(Execution.id.in_(ids)))
by_id = {e.id: e for e in rows.scalars().all()}
executions = [by_id[i] for i in ids if i in by_id]
return {
"total": len(executions),
......
......@@ -81,13 +81,20 @@ async def get_overview_stats(
result = await db.execute(module_count_query)
total_modules = result.scalar() or 0
# 最近执行
recent_query = (
select(Execution)
.order_by(Execution.created_at.desc())
.limit(5)
# 最近执行 — 两段式防 MySQL 1038:executions 表含 config JSON /
# error_message TEXT 列,整行 ORDER BY 会撑爆 sort buffer。第一段只取
# id 轻量排序,第二段按 id 回查所需行并还原顺序。
recent_ids = list(
(
await db.execute(
select(Execution.id).order_by(Execution.created_at.desc()).limit(5)
)
result = await db.execute(recent_query)
).scalars().all()
)
recent_executions = []
if recent_ids:
rows = await db.execute(select(Execution).where(Execution.id.in_(recent_ids)))
by_id = {e.id: e for e in rows.scalars().all()}
recent_executions = [
{
"id": e.id,
......@@ -98,7 +105,9 @@ async def get_overview_stats(
"trigger_by": e.trigger_by,
"created_at": e.created_at.isoformat() if e.created_at else None,
}
for e in result.scalars().all()
for i in recent_ids
if i in by_id
for e in [by_id[i]]
]
return {
......
......@@ -80,13 +80,28 @@ class ApiPresetService:
total = (await self.db.execute(count_query)).scalar() or 0
query = (
query.order_by(desc(ApiPreset.updated_at))
.offset((page - 1) * page_size)
.limit(page_size)
# 查询列表 — 两段式防 MySQL 1038:本表含 headers/body/body_template/
# capture_rules 大 JSON 列,整行 ORDER BY 会撑爆 sort buffer。第一段只
# 取 id 轻量排序分页,第二段按 id 回查整行并还原顺序。
id_query = select(ApiPreset.id).order_by(desc(ApiPreset.updated_at))
if keyword:
id_query = id_query.where(
(ApiPreset.name.like(like_pattern)) | (ApiPreset.target_url.like(like_pattern))
)
ids = list(
(
await self.db.execute(
id_query.offset((page - 1) * page_size).limit(page_size)
)
).scalars().all()
)
items: List[ApiPreset] = []
if ids:
rows = await self.db.execute(
select(ApiPreset).where(ApiPreset.id.in_(ids))
)
result = await self.db.execute(query)
items = list(result.scalars().all())
by_id = {p.id: p for p in rows.scalars().all()}
items = [by_id[i] for i in ids if i in by_id]
return items, total
......
......@@ -83,15 +83,25 @@ class ApiTestService:
count_result = await self.db.execute(count_query)
total = count_result.scalar() or 0
# 查询列表
result = await self.db.execute(
select(TestCase)
# 查询列表 — 两段式防 MySQL 1038:test_cases 表含 steps/config 等
# 大 JSON 列,整行 ORDER BY 会撑爆 sort buffer。第一段只取 id 轻量
# 排序分页,第二段按 id 回查整行并还原顺序。
ids = list(
(
await self.db.execute(
select(TestCase.id)
.where(*conditions)
.order_by(desc(TestCase.updated_at))
.offset(skip)
.limit(limit)
)
cases = list(result.scalars().all())
).scalars().all()
)
cases: List[TestCase] = []
if ids:
rows = await self.db.execute(select(TestCase).where(TestCase.id.in_(ids)))
by_id = {c.id: c for c in rows.scalars().all()}
cases = [by_id[i] for i in ids if i in by_id]
return cases, total
......@@ -375,15 +385,25 @@ class ApiTestService:
count_result = await self.db.execute(count_query)
total = count_result.scalar() or 0
# 查询列表
result = await self.db.execute(
select(CaseResult)
# 查询列表 — 两段式防 MySQL 1038:case_results 表含 steps_result 大
# JSON 列,整行 ORDER BY 会撑爆 sort buffer。第一段只取 id 轻量排序
# 分页,第二段按 id 回查整行并还原顺序。
ids = list(
(
await self.db.execute(
select(CaseResult.id)
.where(*conditions)
.order_by(CaseResult.created_at)
.offset(skip)
.limit(limit)
)
results = list(result.scalars().all())
).scalars().all()
)
results: List[CaseResult] = []
if ids:
rows = await self.db.execute(select(CaseResult).where(CaseResult.id.in_(ids)))
by_id = {r.id: r for r in rows.scalars().all()}
results = [by_id[i] for i in ids if i in by_id]
return results, total
......@@ -409,14 +429,24 @@ class ApiTestService:
count_result = await self.db.execute(count_query)
total = count_result.scalar() or 0
# 查询列表
result = await self.db.execute(
select(Execution)
# 查询列表 — 两段式防 MySQL 1038:executions 表含 config JSON 列,
# 整行 ORDER BY 会撑爆 sort buffer。第一段只取 id 轻量排序分页,
# 第二段按 id 回查整行并还原顺序。
ids = list(
(
await self.db.execute(
select(Execution.id)
.where(*conditions)
.order_by(desc(Execution.created_at))
.offset(skip)
.limit(limit)
)
executions = list(result.scalars().all())
).scalars().all()
)
executions: List[Execution] = []
if ids:
rows = await self.db.execute(select(Execution).where(Execution.id.in_(ids)))
by_id = {e.id: e for e in rows.scalars().all()}
executions = [by_id[i] for i in ids if i in by_id]
return executions, total
\ No newline at end of file
......@@ -120,17 +120,26 @@ class CaseService:
total_result = await self.db.execute(count_query)
total = total_result.scalar() or 0
# 查询列表
query = (
select(TestCase)
.order_by(TestCase.updated_at.desc())
# 查询列表 — 两段式防 MySQL 1038:本表含 steps/config/parameters 等
# 大 JSON 列,整行 ORDER BY 会撑爆 sort buffer。第一段只取 id 轻量排序
# 分页,第二段按 id 回查整行并还原顺序。
id_query = select(TestCase.id)
if conditions:
id_query = id_query.where(*conditions)
ids = list(
(
await self.db.execute(
id_query.order_by(TestCase.updated_at.desc())
.offset(skip)
.limit(limit)
)
if conditions:
query = query.where(*conditions)
result = await self.db.execute(query)
cases = list(result.scalars().all())
).scalars().all()
)
cases: List[TestCase] = []
if ids:
rows = await self.db.execute(select(TestCase).where(TestCase.id.in_(ids)))
by_id = {c.id: c for c in rows.scalars().all()}
cases = [by_id[i] for i in ids if i in by_id]
return cases, total
......
......@@ -548,15 +548,26 @@ class DeployService:
total_result = await self.db.execute(count_query)
total = total_result.scalar() or 0
query = (
select(DeployLog)
# 两段式防 MySQL 1038:deploy_logs 表含 content 大 TEXT 列,整行
# ORDER BY 会撑爆 sort buffer。第一段只取 id 轻量排序分页,第二段
# 按 id 回查整行并还原顺序。
ids = list(
(
await self.db.execute(
select(DeployLog.id)
.where(*conditions)
.order_by(DeployLog.timestamp.asc())
.offset(skip)
.limit(limit)
)
result = await self.db.execute(query)
return list(result.scalars().all()), total
).scalars().all()
)
logs: List[DeployLog] = []
if ids:
rows = await self.db.execute(select(DeployLog).where(DeployLog.id.in_(ids)))
by_id = {l.id: l for l in rows.scalars().all()}
logs = [by_id[i] for i in ids if i in by_id]
return logs, total
# ==================== 核心执行逻辑 ====================
......
......@@ -378,13 +378,20 @@ class DeviceSimService:
total_result = await self.db.execute(count_query)
total = total_result.scalar() or 0
# 查询列表
query = select(DeviceSimulator).order_by(DeviceSimulator.created_at.desc())
# 查询列表 — 两段式防 MySQL 1038:本表含 topics/default_topic_params/
# report_config 等大 JSON 列,整行 ORDER BY 会撑爆 sort buffer。第一段
# 只取 id 轻量排序分页,第二段按 id 回查整行并还原顺序。
id_query = select(DeviceSimulator.id).order_by(DeviceSimulator.created_at.desc())
if conditions:
query = query.where(and_(*conditions))
query = query.offset(skip).limit(limit)
result = await self.db.execute(query)
items = list(result.scalars().all())
id_query = id_query.where(and_(*conditions))
ids = list((await self.db.execute(id_query.offset(skip).limit(limit))).scalars().all())
items: List[DeviceSimulator] = []
if ids:
rows = await self.db.execute(
select(DeviceSimulator).where(DeviceSimulator.id.in_(ids))
)
by_id = {d.id: d for d in rows.scalars().all()}
items = [by_id[i] for i in ids if i in by_id]
# 更新实时运行状态
with _running_simulators_lock:
......@@ -1381,13 +1388,18 @@ class DeviceSimService:
total_result = await self.db.execute(count_query)
total = total_result.scalar() or 0
# 查询列表
query = select(ReportLog).order_by(ReportLog.created_at.desc())
# 查询列表 — 两段式防 MySQL 1038:report_logs 表含 payload 大 JSON 列,
# 整行 ORDER BY 会撑爆 sort buffer。第一段只取 id 轻量排序分页,
# 第二段按 id 回查整行并还原顺序。
id_query = select(ReportLog.id).order_by(ReportLog.created_at.desc())
if conditions:
query = query.where(and_(*conditions))
query = query.offset(skip).limit(limit)
result = await self.db.execute(query)
items = list(result.scalars().all())
id_query = id_query.where(and_(*conditions))
ids = list((await self.db.execute(id_query.offset(skip).limit(limit))).scalars().all())
items: List[ReportLog] = []
if ids:
rows = await self.db.execute(select(ReportLog).where(ReportLog.id.in_(ids)))
by_id = {r.id: r for r in rows.scalars().all()}
items = [by_id[i] for i in ids if i in by_id]
return items, total
......
......@@ -490,17 +490,20 @@ class ExecutionService:
total_result = await self.db.execute(count_query)
total = total_result.scalar() or 0
query = (
select(Execution)
.order_by(desc(Execution.created_at))
.offset(skip)
.limit(limit)
)
# 两段式防 MySQL 1038:executions 表含 config JSON / error_message
# TEXT 列,整行 ORDER BY 会撑爆 sort buffer。第一段只取 id 轻量排序
# 分页,第二段按 id 回查整行并还原顺序。
id_query = select(Execution.id).order_by(desc(Execution.created_at))
if conditions:
query = query.where(*conditions)
result = await self.db.execute(query)
return list(result.scalars().all()), total
id_query = id_query.where(*conditions)
ids = list(
(await self.db.execute(id_query.offset(skip).limit(limit))).scalars().all()
)
if not ids:
return [], total
rows = await self.db.execute(select(Execution).where(Execution.id.in_(ids)))
by_id = {e.id: e for e in rows.scalars().all()}
return [by_id[i] for i in ids if i in by_id], total
async def get_execution(self, execution_id: str) -> Optional[Execution]:
"""
......@@ -546,11 +549,18 @@ class ExecutionService:
# 先查询用例,取首个用例的 case_type 作为执行记录的类型(用于按类型筛选)
# 按模块分组 + order 排序,确保同模块内首用例(含登录)先执行
cases_query = select(TestCase).where(TestCase.id.in_(case_ids)).order_by(
# 两段式防 MySQL 1038:TestCase 表含 steps/config/parameters 大 JSON
# 列,整行 ORDER BY 会撑爆 sort buffer。第一段只取 id 轻量排序,
# 第二段按 id 回查整行并还原顺序。
id_query = select(TestCase.id).where(TestCase.id.in_(case_ids)).order_by(
TestCase.module_id, TestCase.order, TestCase.created_at
)
cases_result = await self.db.execute(cases_query)
cases = list(cases_result.scalars().all())
ids = list((await self.db.execute(id_query)).scalars().all())
cases: List[TestCase] = []
if ids:
rows = await self.db.execute(select(TestCase).where(TestCase.id.in_(ids)))
by_id = {c.id: c for c in rows.scalars().all()}
cases = [by_id[i] for i in ids if i in by_id]
first_case_type = cases[0].case_type if cases else "ui"
execution = Execution(
......@@ -649,8 +659,11 @@ class ExecutionService:
# 获取所有待执行的用例结果
# JOIN TestCase 按模块分组 + order 排序,确保同模块内首用例(含登录)先执行
results_query = (
select(CaseResult)
# 两段式防 MySQL 1038:case_results 表含 steps_result 大 JSON 列,整行
# ORDER BY 会撑爆 sort buffer。第一段只取 id 轻量排序,第二段按 id
# 回查整行并还原顺序(回查无需 join,多对一 join 不产生重复行)。
id_query = (
select(CaseResult.id)
.join(TestCase, CaseResult.case_id == TestCase.id)
.where(
CaseResult.execution_id == execution_id,
......@@ -658,8 +671,12 @@ class ExecutionService:
)
.order_by(TestCase.module_id, TestCase.order, TestCase.created_at)
)
results_data = await self.db.execute(results_query)
case_results = list(results_data.scalars().all())
ids = list((await self.db.execute(id_query)).scalars().all())
case_results: List[CaseResult] = []
if ids:
rows = await self.db.execute(select(CaseResult).where(CaseResult.id.in_(ids)))
by_id = {r.id: r for r in rows.scalars().all()}
case_results = [by_id[i] for i in ids if i in by_id]
# 获取用例定义
case_ids = [r.case_id for r in case_results]
......@@ -1027,13 +1044,23 @@ class ExecutionService:
Returns:
Tuple[List[CaseResult], int]: (用例结果列表, 总数)
"""
query = (
select(CaseResult)
# 两段式防 MySQL 1038:case_results 表含 steps_result 大 JSON 列,整行
# ORDER BY 会撑爆 sort buffer。第一段只取 id 轻量排序,第二段按 id
# 回查整行并还原顺序。
ids = list(
(
await self.db.execute(
select(CaseResult.id)
.where(CaseResult.execution_id == execution_id)
.order_by(CaseResult.created_at.asc())
)
result = await self.db.execute(query)
items = list(result.scalars().all())
).scalars().all()
)
items: List[CaseResult] = []
if ids:
rows = await self.db.execute(select(CaseResult).where(CaseResult.id.in_(ids)))
by_id = {r.id: r for r in rows.scalars().all()}
items = [by_id[i] for i in ids if i in by_id]
return items, len(items)
......@@ -1137,8 +1164,11 @@ class ExecutionService:
})
# 获取所有待执行的用例结果(按模块分组排序)
results_query = (
select(CaseResult)
# 两段式防 MySQL 1038:case_results 表含 steps_result 大 JSON 列,整行
# ORDER BY 会撑爆 sort buffer。第一段只取 id 轻量排序,第二段按 id
# 回查整行并还原顺序。
id_query = (
select(CaseResult.id)
.join(TestCase, CaseResult.case_id == TestCase.id)
.where(
CaseResult.execution_id == execution_id,
......@@ -1146,8 +1176,12 @@ class ExecutionService:
)
.order_by(TestCase.module_id, TestCase.order, TestCase.created_at)
)
results_data = await self.db.execute(results_query)
case_results = list(results_data.scalars().all())
ids = list((await self.db.execute(id_query)).scalars().all())
case_results: List[CaseResult] = []
if ids:
rows = await self.db.execute(select(CaseResult).where(CaseResult.id.in_(ids)))
by_id = {r.id: r for r in rows.scalars().all()}
case_results = [by_id[i] for i in ids if i in by_id]
# 获取用例定义
case_ids = [r.case_id for r in case_results]
......
......@@ -155,14 +155,18 @@ class LoginTemplateService:
total_result = await self.db.execute(count_query)
total = total_result.scalar()
# 查询列表
query = select(LoginTemplate).order_by(LoginTemplate.created_at.desc())
# 查询列表 — 两段式防 MySQL 1038:本表含 steps 大 JSON 列,整行
# ORDER BY 会撑爆 sort buffer。第一段只取 id 轻量排序分页,第二段
# 按 id 回查整行并还原顺序。
id_query = select(LoginTemplate.id).order_by(LoginTemplate.created_at.desc())
if conditions:
query = query.where(*conditions)
query = query.offset(skip).limit(limit)
result = await self.db.execute(query)
templates = list(result.scalars().all())
id_query = id_query.where(*conditions)
ids = list((await self.db.execute(id_query.offset(skip).limit(limit))).scalars().all())
templates: List[LoginTemplate] = []
if ids:
rows = await self.db.execute(select(LoginTemplate).where(LoginTemplate.id.in_(ids)))
by_id = {t.id: t for t in rows.scalars().all()}
templates = [by_id[i] for i in ids if i in by_id]
return templates, total
......
......@@ -1520,13 +1520,20 @@ class PerformanceService:
Returns:
dict: 合并报告数据,无则返回 None
"""
result = await self.db.execute(
select(PerformanceProjectReport)
# 两段式防 MySQL 1038:performance_project_reports 表含 task_summaries
# 大 TEXT(JSON 数组),整行 ORDER BY 会撑爆 sort buffer。第一段只取
# id 轻量排序,第二段按 id 回查整行。
report_id = (
await self.db.execute(
select(PerformanceProjectReport.id)
.where(PerformanceProjectReport.project_id == project_id)
.order_by(PerformanceProjectReport.created_at.desc())
.limit(1)
)
report = result.scalar_one_or_none()
).scalar_one_or_none()
report = None
if report_id:
report = await self.db.get(PerformanceProjectReport, report_id)
if not report:
return None
return self._report_to_dict(report)
......@@ -1565,10 +1572,6 @@ class PerformanceService:
Returns:
dict: {total, items}
"""
base_query = select(PerformanceProjectReport).where(
PerformanceProjectReport.project_id == project_id
)
# 总数
count_result = await self.db.execute(
select(func.count()).select_from(PerformanceProjectReport).where(
......@@ -1577,14 +1580,26 @@ class PerformanceService:
)
total = count_result.scalar() or 0
# 分页列表
result = await self.db.execute(
base_query
# 分页列表 — 两段式防 MySQL 1038:task_summaries 大 TEXT 不参与排序,
# 第一段只取 id 轻量排序分页,第二段按 id 回查整行并还原顺序。
ids = list(
(
await self.db.execute(
select(PerformanceProjectReport.id)
.where(PerformanceProjectReport.project_id == project_id)
.order_by(PerformanceProjectReport.created_at.desc())
.offset((page - 1) * page_size)
.limit(page_size)
)
reports = list(result.scalars().all())
).scalars().all()
)
reports = []
if ids:
rows = await self.db.execute(
select(PerformanceProjectReport).where(PerformanceProjectReport.id.in_(ids))
)
by_id = {r.id: r for r in rows.scalars().all()}
reports = [by_id[i] for i in ids if i in by_id]
return {
"total": total,
......
......@@ -400,14 +400,25 @@ class ReportService:
if not execution:
raise ValueError(f"执行记录不存在: {execution_id}")
# 获取用例结果
results_query = (
select(CaseResult)
# 获取用例结果 — 两段式防 MySQL 1038:case_results 表含 steps_result
# 大 JSON 列,整行 ORDER BY 会撑爆 sort buffer。第一段只取 id 轻量
# 排序,第二段按 id 回查整行并还原顺序。
result_ids = list(
(
await db.execute(
select(CaseResult.id)
.where(CaseResult.execution_id == execution_id)
.order_by(CaseResult.created_at.asc())
)
results_result = await db.execute(results_query)
case_results = list(results_result.scalars().all())
).scalars().all()
)
case_results: List[CaseResult] = []
if result_ids:
rows = await db.execute(
select(CaseResult).where(CaseResult.id.in_(result_ids))
)
by_id = {r.id: r for r in rows.scalars().all()}
case_results = [by_id[i] for i in result_ids if i in by_id]
# 构建 用例ID → 模块名 映射(用于按模块分组展示)
module_map: dict = {}
......
......@@ -123,8 +123,11 @@ async def collect_module_cases(
"""
if not module_ids:
return []
query = (
select(TestCase)
# 两段式防 MySQL 1038:TestCase 表含 steps/config/parameters 大 JSON 列,
# 整行 ORDER BY 会撑爆 sort buffer。第一段只取 id 轻量排序,第二段按 id
# 回查整行并还原顺序。
id_query = (
select(TestCase.id)
.where(
TestCase.module_id.in_(module_ids),
TestCase.status == "active",
......@@ -132,8 +135,12 @@ async def collect_module_cases(
)
.order_by(TestCase.module_id, TestCase.order, TestCase.created_at)
)
result = await db.execute(query)
return list(result.scalars().all())
ids = list((await db.execute(id_query)).scalars().all())
if not ids:
return []
rows = await db.execute(select(TestCase).where(TestCase.id.in_(ids)))
by_id = {c.id: c for c in rows.scalars().all()}
return [by_id[i] for i in ids if i in by_id]
async def _has_running_security_execution(db) -> bool:
......
......@@ -183,12 +183,25 @@ class SecurityReportService:
db: AsyncSession,
) -> List[VulnerabilityResult]:
"""获取漏洞测试结果列表"""
result = await db.execute(
select(VulnerabilityResult)
# 两段式防 MySQL 1038:本表含 request_info/response_info 大 TEXT 列,
# 整行 ORDER BY 会撑爆 sort buffer。第一段只取 id 轻量排序,第二段
# 按 id 回查整行并还原顺序。
ids = list(
(
await db.execute(
select(VulnerabilityResult.id)
.where(VulnerabilityResult.execution_id == execution_id)
.order_by(VulnerabilityResult.level, VulnerabilityResult.test_id)
)
return list(result.scalars().all())
).scalars().all()
)
if not ids:
return []
rows = await db.execute(
select(VulnerabilityResult).where(VulnerabilityResult.id.in_(ids))
)
by_id = {r.id: r for r in rows.scalars().all()}
return [by_id[i] for i in ids if i in by_id]
async def _get_security_config(
self,
......
......@@ -511,15 +511,28 @@ class SecurityService:
count_result = await self.db.execute(count_query)
total = count_result.scalar() or 0
# 查询列表(按风险等级排序)
result = await self.db.execute(
select(VulnerabilityResult)
# 查询列表(按风险等级排序)— 两段式防 MySQL 1038:本表含
# request_info/response_info/description/fix_suggestion 大 TEXT 列,
# 整行 ORDER BY 会撑爆 sort buffer。第一段只取 id 轻量排序分页,
# 第二段按 id 回查整行并还原顺序。
ids = list(
(
await self.db.execute(
select(VulnerabilityResult.id)
.where(*conditions)
.order_by(VulnerabilityResult.level, VulnerabilityResult.test_id)
.offset(skip)
.limit(limit)
)
results = list(result.scalars().all())
).scalars().all()
)
results: List[VulnerabilityResult] = []
if ids:
rows = await self.db.execute(
select(VulnerabilityResult).where(VulnerabilityResult.id.in_(ids))
)
by_id = {r.id: r for r in rows.scalars().all()}
results = [by_id[i] for i in ids if i in by_id]
return results, total
......
Markdown 格式
0%
您添加了 0 到此讨论。请谨慎行事。
请先完成此评论的编辑!
注册 或者 后发表评论