Skip to content

U
ubains-module-test
提交 6eb7badf authored 作者: 陈泽健's avatar 陈泽健
浏览文件
操作

feat(skills): 升级集群监控与部署技能，新增钉钉通知功能 

- ARM/X86 集群监控：增强中间件检查（Redis/EMQX/Nacos/FastDFS/达梦），优化报告生成器
- X86-QLV10-XTYBS：优化 SSH 辅助工具，更新部署流程文档
- X86-TX-XTYBS：重构完整部署脚本，修复编码问题
- 新增钉钉通知模块，支持集群监控报告自动推送
Co-Authored-By: 's avatarClaude <noreply@anthropic.com>
上级 0b20b9cf
显示空白字符变更
内嵌 并排
正在显示 包含 2776 行增加 和 259 行删除
.claude/skills/ARM-CLUSTER-MONITOR/SKILL.md
浏览文件 @ 6eb7badf
......@@ -45,6 +45,10 @@ description: ARM集群夜间监控 - 监测9网段四台ARM架构服务器集群
- 预期产出
- **必须等待用户确认计划后方可开始执行**，不得跳过确认步骤直接操作；
- 计划执行文档保存到 `.claude/skills/ARM-CLUSTER-MONITOR/计划执行_{YYYYMMDD_HHMMSS}.md`。
- **定时执行模式例外**：定时任务触发执行时，跳过计划确认步骤，直接执行监控流程。
- **需求文档规范**：每次需求变动或问题处理，需在 `Docs/PRD/新统一平台集群监测/` 目录下输出：
- `_PRD_需求文档.md` — 需求文档
- `_PRD_需求文档_计划执行.md` — 对应的计划执行文档
### 代码实现规范
- **代码存放位置**：如需代码实现（如Python脚本、工具类等），统一存放在 `.claude/skills/ARM-CLUSTER-MONITOR/code/` 目录下；
......@@ -640,6 +644,89 @@ def connect_to_server(server_ip, username, password=None, key_dir=None):
11. **中文输出**：所有回复、代码注释、Git提交信息使用简体中文
12. **计划先行**：需求变更必须输出计划执行文档，确认后方可执行
13. **paramiko连接**：默认使用Python paramiko库，不依赖系统ssh命令
14. **钉钉通知**：仅在08:00-09:00时段发送夜间汇总通知，避免重复发送
## 钉钉通知配置
### 通知策略
**延迟通知机制**：
- 定时任务期间（22:00-07:59）：执行监控但不发送钉钉通知
- 08:00-09:00时段：执行监控后自动汇总分析并发送一次通知
- 防重复发送：同一日期仅发送一次汇总通知
### 配置文件
`code/dingtalk_config.json` — 钉钉机器人配置
```json
{
"enabled": true,
"webhook": "https://oapi.dingtalk.com/robot/send?access_token=YOUR_TOKEN",
"secret": "SECYOUR_SECRET",
"notification_mode": "summary",
"trigger_conditions": {
"always": true,
"on_warning": false,
"on_critical": false
}
}
```
### 配置项说明
| 配置项 | 类型 | 说明 |
|-------|------|------|
| `enabled` | bool | 是否启用钉钉通知 |
| `webhook` | string | 钉钉机器人Webhook地址 |
| `secret` | string | 签名密钥（SEC开头） |
| `notification_mode` | string | 通知模式：`summary`（仅摘要） |
| `trigger_conditions.always` | bool | 每次监控都发送通知 |
| `trigger_conditions.on_warning` | bool | 仅在发生警告时发送 |
| `trigger_conditions.on_critical` | bool | 仅在发生严重告警时发送 |
### 集成位置
- 通知模块：`code/dingtalk_notifier.py`
- 调用位置：`monitor_agent.py` → `run_full_monitor()` → 第6.5步
- 通知发送失败不影响监控主流程
### 消息格式
```markdown
### ARM集群夜间监控汇总报告
**日期**: 20260710
**时段**: 2026-07-10 22:00 ~ 2026-07-11 08:00
**执行**: 11次
**状态**: [OK] normal
**评分**: 95 -> 92
#### 资源趋势
| 节点 | CPU峰值 | 内存峰值 | 磁盘变化 |
|:---|:---|:---|:---|
| 192.168.9.89 | 52% | 78% | +1% |
| ...
**累计告警**: 3次
#### 建议
- 192.168.9.91 磁盘增长+2%
```
### 相关文件
| 文件 | 说明 |
|------|------|
| `code/dingtalk_notifier.py` | 钉钉通知发送模块 |
| `code/dingtalk_config.json` | 钉钉机器人配置 |
| `code/report_analyzer.py` | 夜间报告汇总分析器 |
| `code/sent_records.json` | 防重复发送记录 |
### 手动触发汇总
```bash
python main.py 汇总 # 手动执行汇总分析并发送通知
```
## 关联文档
......
.claude/skills/ARM-CLUSTER-MONITOR/code/dingtalk_config.json 0 → 100644
浏览文件 @ 6eb7badf
{
"enabled": true,
"webhook": "https://oapi.dingtalk.com/robot/send?access_token=27071a77f20da381e9a321653ec5f4dcf668bcf058c01162f28e3f1f8633386d",
"secret": "SEC5d85d5735a1805ada1be84929d5b37f5b72a2a832a6bcd9a1ca5615e5799be38",
"notification_mode": "summary",
"trigger_conditions": {
"always": true,
"on_warning": false,
"on_critical": false
}
}
.claude/skills/ARM-CLUSTER-MONITOR/code/dingtalk_notifier.py 0 → 100644
浏览文件 @ 6eb7badf
# -*- coding: utf-8 -*-
"""
钉钉通知模块 - ARM集群监控专用版
功能：
1. 钉钉机器人消息发送（支持加签验证）
2. Markdown格式摘要构建
3. 集群监控报告通知
作者：Claude Code
创建时间：2026-07-11
"""
import json
import time
import hmac
import hashlib
import base64
import urllib.parse
from typing import Optional, Dict, Any
from pathlib import Path
try:
import requests
except ImportError:
print("[警告] requests库未安装，钉钉通知功能将不可用")
print("[提示] 请执行: pip install requests")
requests = None
class DingTalkNotifier:
"""钉钉通知发送器"""
def __init__(self, webhook: str, secret: str):
"""
初始化钉钉通知器
Args:
webhook: 钉钉机器人Webhook地址
secret: 签名密钥
"""
self.webhook = webhook
self.secret = secret
def _generate_signature(self, timestamp: int) -> str:
"""
生成钉钉签名
Args:
timestamp: 时间戳（毫秒）
Returns:
URL编码后的签名字符串
"""
string_to_sign = f"{timestamp}\n{self.secret}"
# HMAC-SHA256加密
hmac_sha256 = hmac.new(
self.secret.encode('utf-8'),
string_to_sign.encode('utf-8'),
digestmod=hashlib.sha256
)
signature = base64.b64encode(hmac_sha256.digest()).decode('utf-8')
return urllib.parse.quote(signature)
def send_markdown(self, title: str, text: str) -> bool:
"""
发送Markdown消息
Args:
title: 消息标题
text: Markdown格式文本
Returns:
是否发送成功
"""
if requests is None:
print("[错误] requests库未安装，无法发送钉钉通知")
return False
# 构建消息体
message = {
"msgtype": "markdown",
"markdown": {
"title": title,
"text": text
}
}
# 构建完整URL（带签名）
timestamp = int(time.time() * 1000)
sign = self._generate_signature(timestamp)
url = f"{self.webhook}&timestamp={timestamp}&sign={sign}"
try:
headers = {'Content-Type': 'application/json'}
response = requests.post(
url,
headers=headers,
json=message,
timeout=10
)
result = response.json()
if result.get('errcode') == 0:
print(f"[成功] 钉钉通知发送成功")
return True
else:
print(f"[失败] 钉钉API错误: {result.get('errmsg', 'Unknown')} (code: {result.get('errcode')})")
return False
except requests.RequestException as e:
print(f"[错误] 网络请求失败: {e}")
return False
except Exception as e:
print(f"[错误] 发送消息异常: {e}")
return False
def send_cluster_report(
self,
monitor_time: str,
node_count: int,
overall_status: str,
health_score: int,
alert_count: int,
servers_data: Dict,
cluster_data: Dict
) -> bool:
"""
发送ARM集群监控报告
Args:
monitor_time: 监控时间
node_count: 节点数量
overall_status: 整体状态
health_score: 健康评分
alert_count: 告警数量
servers_data: 各节点数据
cluster_data: 集群汇总数据
Returns:
是否发送成功
"""
# 状态图标映射（使用ASCII避免Windows编码问题）
status_map = {
'normal': '[OK]',
'warning': '[WARN]',
'critical': '[ERR]',
'error': '[ERR]'
}
status_icon = status_map.get(overall_status, '[?]')
# 构建标题
title = f"ARM集群监控报告"
# 构建摘要内容
summary_lines = [
f"### ARM集群监控报告",
f"**时间**: {monitor_time}",
f"**节点数**: {node_count}台",
f"**状态**: {status_icon} {overall_status}",
f"**健康评分**: {health_score}/100",
f"**告警数**: {alert_count}",
"",
"#### 集群横向对比 - 系统资源",
"| 节点 | CPU | 内存 | 磁盘 | 容器 | 状态 |",
"|:---|:---|:---|:---|:---|:---|"
]
# 添加各节点数据
for ip, data in servers_data.items():
res = data.get('resources', {})
cpu = res.get('cpu', {}).get('usage_percent', 'N/A')
mem = res.get('memory', {}).get('usage_percent', 'N/A')
disk = res.get('disk', {}).get('usage_percent', 'N/A')
# 处理可能的字典格式
if isinstance(cpu, dict):
cpu = cpu.get('usage_percent', 'N/A')
if isinstance(mem, dict):
mem = mem.get('usage_percent', 'N/A')
if isinstance(disk, dict):
disk = disk.get('usage_percent', 'N/A')
container_count = len(data.get('containers', []))
node_status_icon = status_map.get(data.get('overall_status', 'unknown'), '[?]')
summary_lines.append(f"| {ip} | {cpu}% | {mem}% | {disk}% | {container_count}个 | {node_status_icon} |")
# 添加告警信息（最多显示5条）
alerts = cluster_data.get('alerts', [])
if alerts:
summary_lines.extend([
"",
"#### 告警信息"
])
for alert in alerts[:5]:
level = alert.get('level', 'info').upper()
node = alert.get('node', '')
message = alert.get('message', '')
summary_lines.append(f"- [{level}] {node}: {message}")
# 添加时间戳
summary_lines.extend([
"",
"---",
f"<font color=\"comment\">报告生成时间: {monitor_time}</font>"
])
# 发送消息
summary_text = "\n".join(summary_lines)
return self.send_markdown(title, summary_text)
def send_summary_report(self, summary_text: str) -> bool:
"""
发送夜间汇总报告
Args:
summary_text: 汇总报告内容（Markdown格式）
Returns:
是否发送成功
"""
title = "ARM集群夜间监控汇总"
return self.send_markdown(title, summary_text)
def load_dingtalk_config(config_path: str = None) -> Optional[Dict[str, Any]]:
"""
加载钉钉配置
Args:
config_path: 配置文件路径，默认为同目录下的dingtalk_config.json
Returns:
配置字典，失败返回None
"""
if config_path is None:
# 默认使用同目录下的配置文件
config_path = Path(__file__).parent / "dingtalk_config.json"
config_path = Path(config_path)
if not config_path.exists():
print(f"[警告] 钉钉配置文件不存在: {config_path}")
return None
try:
with open(config_path, 'r', encoding='utf-8') as f:
config = json.load(f)
# 验证必需字段
if not config.get('webhook') or not config.get('secret'):
print("[错误] 钉钉配置缺少webhook或secret字段")
return None
return config
except json.JSONDecodeError as e:
print(f"[错误] 钉钉配置文件格式错误: {e}")
return None
except Exception as e:
print(f"[错误] 加载钉钉配置失败: {e}")
return None
def send_cluster_notification(
monitor_time: str,
node_count: int,
overall_status: str,
health_score: int,
alert_count: int,
servers_data: Dict,
cluster_data: Dict,
config_path: str = None
) -> bool:
"""
发送集群监控通知（便捷函数）
Args:
monitor_time: 监控时间
node_count: 节点数量
overall_status: 整体状态
health_score: 健康评分
alert_count: 告警数量
servers_data: 各节点数据
cluster_data: 集群汇总数据
config_path: 配置文件路径
Returns:
是否发送成功
"""
# 加载配置
config = load_dingtalk_config(config_path)
if config is None:
print("[跳过] 钉钉通知功能未配置或配置无效")
return False
# 检查是否启用
if not config.get('enabled', True):
print("[跳过] 钉钉通知功能已禁用")
return False
# 创建通知器
notifier = DingTalkNotifier(
webhook=config['webhook'],
secret=config['secret']
)
# 发送报告
return notifier.send_cluster_report(
monitor_time=monitor_time,
node_count=node_count,
overall_status=overall_status,
health_score=health_score,
alert_count=alert_count,
servers_data=servers_data,
cluster_data=cluster_data
)
def send_summary_notification(summary_text: str, config_path: str = None) -> bool:
"""
发送夜间汇总报告通知（便捷函数）
Args:
summary_text: 汇总报告内容（Markdown格式）
config_path: 配置文件路径
Returns:
是否发送成功
"""
config = load_dingtalk_config(config_path)
if config is None:
print("[跳过] 钉钉通知功能未配置或配置无效")
return False
if not config.get('enabled', True):
print("[跳过] 钉钉通知功能已禁用")
return False
notifier = DingTalkNotifier(
webhook=config['webhook'],
secret=config['secret']
)
return notifier.send_summary_report(summary_text)
if __name__ == "__main__":
# 测试代码
print("=" * 60)
print("钉钉通知模块测试")
print("=" * 60)
# 测试配置加载
config = load_dingtalk_config()
if config:
print(f"[成功] 配置加载成功")
print(f" Webhook: {config.get('webhook', '')[:30]}...")
print(f" Secret: {config.get('secret', '')[:10]}...")
# 测试发送
notifier = DingTalkNotifier(config['webhook'], config['secret'])
test_message = """### 测试消息
**时间**: 2026-07-11 10:00:00
**状态**: [OK] 正常
#### 测试内容
- CPU: 45%
- 内存: 62%
"""
result = notifier.send_markdown("ARM集群监控测试", test_message)
print(f"\n发送结果: {'成功' if result else '失败'}")
else:
print("[失败] 配置加载失败")
\ No newline at end of file
.claude/skills/ARM-CLUSTER-MONITOR/code/main.py
浏览文件 @ 6eb7badf
......@@ -6,14 +6,15 @@
1. 命令行参数解析
2. 定时任务启动逻辑
3. 监控执行入口
4. 错误处理
4. 汇总分析与钉钉通知（08:00后执行）
支持命令：
- python main.py # 立即执行一次集群监控
- python main.py 定时 # 启动定时监控（22:00-09:00每2h）
- python main.py 定时 # 启动定时监控（22:00-08:00每1h）
- python main.py 定时状态 # 查看定时任务状态
- python main.py 定时停止 # 停止定时监控任务
- python main.py 配置免密 # 为四台服务器配置SSH免密登录
- python main.py 汇总 # 执行汇总分析并发送钉钉通知
作者：Claude Code
创建时间：2026-07-09
......@@ -21,25 +22,35 @@
import sys
import os
import json
from datetime import datetime
from pathlib import Path
# 导入监控调度器
from monitor_agent import MonitorAgent
# 导入汇总分析器
try:
from report_analyzer import NightlyReportAnalyzer
from dingtalk_notifier import send_summary_notification
SUMMARY_AVAILABLE = True
except ImportError:
SUMMARY_AVAILABLE = False
print("[警告] 汇总分析模块未找到")
# 防重复发送记录文件
SENT_RECORDS_FILE = Path(__file__).parent / "sent_records.json"
def main():
"""
主入口函数
"""
# 解析命令行参数
args = sys.argv[1:] if len(sys.argv) > 1 else []
# 根据参数执行对应功能
if len(args) == 0:
# 默认：立即执行一次集群监控
run_immediate_monitor()
elif args[0] == '定时':
# 启动定时监控
if len(args) > 1 and args[1] == '状态':
show_schedule_status()
elif len(args) > 1 and args[1] == '停止':
......@@ -47,31 +58,156 @@ def main():
else:
start_schedule()
elif args[0] == '配置免密':
# 配置SSH免密登录
configure_ssh_keys()
elif args[0] == '汇总':
run_summary_and_notify()
else:
# 未知命令
print(f"[错误] 未知命令: {args[0]}")
print_usage()
def should_send_summary() -> bool:
"""
判断是否应该发送汇总通知
规则：
1. 当前时间在 08:00-09:00 之间
2. 今日未发送过
Returns:
是否应该发送
"""
now = datetime.now()
current_hour = now.hour
# 时间判断：08:00-09:00
if current_hour < 8 or current_hour >= 9:
return False
# 检查是否已发送
today = now.strftime('%Y%m%d')
if SENT_RECORDS_FILE.exists():
try:
with open(SENT_RECORDS_FILE, 'r', encoding='utf-8') as f:
records = json.load(f)
if records.get(today) == 'sent':
print(f"[信息] 今日({today})已发送汇总通知，跳过")
return False
except Exception:
pass
return True
def mark_summary_sent():
"""
标记今日已发送汇总通知
"""
today = datetime.now().strftime('%Y%m%d')
records = {}
if SENT_RECORDS_FILE.exists():
try:
with open(SENT_RECORDS_FILE, 'r', encoding='utf-8') as f:
records = json.load(f)
except Exception:
pass
records[today] = 'sent'
try:
with open(SENT_RECORDS_FILE, 'w', encoding='utf-8') as f:
json.dump(records, f, ensure_ascii=False, indent=2)
except Exception as e:
print(f"[警告] 记录发送状态失败: {e}")
def run_summary_and_notify():
"""
执行汇总分析并发送钉钉通知
"""
print("=" * 60)
print("ARM集群夜间监控 - 汇总分析模式")
print("=" * 60)
if not SUMMARY_AVAILABLE:
print("[错误] 汇总分析模块不可用")
return False
# 检查是否应该发送
if not should_send_summary():
print("[跳过] 当前时段不需要发送汇总通知")
return False
try:
# 执行汇总分析
analyzer = NightlyReportAnalyzer()
print("\n[步骤1] 分析夜间监控报告...")
if not analyzer.analyze_all_reports():
print("[失败] 汇总分析失败")
return False
# 生成汇总报告
print("\n[步骤2] 生成汇总报告...")
report_path = analyzer.save_summary_report()
if not report_path:
print("[失败] 汇总报告保存失败")
return False
# 生成钉钉摘要
print("\n[步骤3] 发送钉钉通知...")
summary_text = analyzer.get_summary_for_dingtalk()
# 发送通知
result = send_summary_notification(summary_text)
if result:
print("[成功] 钉钉通知发送成功")
# 标记已发送
mark_summary_sent()
print(f"\n汇总报告路径: {report_path}")
return True
else:
print("[失败] 钉钉通知发送失败")
return False
except Exception as e:
print(f"\n[错误] 汇总分析异常: {str(e)}")
handle_error(e)
return False
def run_immediate_monitor():
"""
立即执行一次集群监控
如果当前时间在 08:00-09:00 之间，执行监控后自动触发汇总分析并发送通知
"""
print("=" * 60)
print("ARM集群夜间监控 - 立即执行模式")
print("=" * 60)
print(f"执行时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
now = datetime.now()
print(f"执行时间: {now.strftime('%Y-%m-%d %H:%M:%S')}")
# 判断是否需要发送通知
should_notify = should_send_summary()
if should_notify:
print("[信息] 当前时段为08:00-09:00，监控完成后将发送汇总通知")
try:
# 创建监控调度器
agent = MonitorAgent()
# 执行完整监控流程
summary = agent.run_full_monitor()
# 执行监控（仅在should_notify=True时发送通知）
summary = agent.run_full_monitor(send_notification=should_notify)
# 输出执行结果
if summary.get('status') == 'success':
print("\n" + "=" * 60)
print("监控执行成功")
......@@ -83,6 +219,11 @@ def run_immediate_monitor():
print(f"告警数: {summary.get('alert_count', 0)}")
print(f"报告路径: {summary.get('report_path', '')}")
print("=" * 60)
# 如果在08:00-09:00时段，执行汇总分析
if should_notify and SUMMARY_AVAILABLE:
print("\n[信息] 执行汇总分析...")
run_summary_and_notify()
else:
print("\n" + "=" * 60)
print("监控执行失败")
......
.claude/skills/ARM-CLUSTER-MONITOR/code/middleware_monitor.py
浏览文件 @ 6eb7badf
......@@ -145,11 +145,94 @@ class MiddlewareMonitor:
redis_info['replication_info']['connected_slaves'] = int(line.split(':')[1].strip())
elif 'slave_repl_offset:' in line:
redis_info['replication_info']['slave_repl_offset'] = int(line.split(':')[1].strip())
elif 'master_link_status:' in line:
redis_info['replication_info']['master_link_status'] = line.split(':')[1].strip()
elif 'slave_read_only:' in line:
redis_info['replication_info']['slave_read_only'] = int(line.split(':')[1].strip())
# 主从复制健康检测
role = redis_info['replication_info'].get('role', '')
if role == 'master':
connected_slaves = redis_info['replication_info'].get('connected_slaves', 0)
if connected_slaves < 2:
redis_info['alerts'].append({
'level': 'warning',
'message': f'Redis主节点连接从节点不足: {connected_slaves}/2'
})
elif role == 'slave':
master_link = redis_info['replication_info'].get('master_link_status', 'down')
if master_link != 'up':
redis_info['alerts'].append({
'level': 'critical',
'message': f'Redis从节点与主节点断开: master_link_status={master_link}'
})
slave_read_only = redis_info['replication_info'].get('slave_read_only', 1)
if slave_read_only == 0:
redis_info['alerts'].append({
'level': 'warning',
'message': 'Redis从节点slave_read_only=0，可能存在写入风险'
})
# 检查哨兵进程
sentinel_command = "ps -ef | grep redis-sentinel | grep -v grep"
exit_code, stdout, stderr = self.ssh_manager.execute_command(sentinel_command)
redis_info['sentinel_process'] = exit_code == 0 and stdout.strip() != ''
# Sentinel集群健康检测（仅在sentinel节点执行）
if redis_info['sentinel_process'] and 'sentinel' in cname.lower():
# 检查master状态
sentinel_master_cmd = f"docker exec {cname} redis-cli -p 26379 SENTINEL master mymaster 2>/dev/null"
exit_code, stdout, stderr = self.ssh_manager.execute_command(sentinel_master_cmd)
if exit_code == 0:
sentinel_info = {}
for line in stdout.split('\n'):
if 'num-slaves' in line:
sentinel_info['num_slaves'] = int(stdout.split('\n')[stdout.split('\n').index(line) + 1])
elif 'num-other-sentinels' in line:
sentinel_info['num_other_sentinels'] = int(stdout.split('\n')[stdout.split('\n').index(line) + 1])
elif 'quorum' in line:
sentinel_info['quorum'] = int(stdout.split('\n')[stdout.split('\n').index(line) + 1])
elif 'flags' in line:
idx = stdout.split('\n').index(line)
if idx + 1 < len(stdout.split('\n')):
flags = stdout.split('\n')[idx + 1]
sentinel_info['master_flags'] = flags
redis_info['sentinel_info'] = sentinel_info
# 哨兵数量验证（应有3个哨兵）
total_sentinels = sentinel_info.get('num_other_sentinels', 0) + 1 # +1是自己
if total_sentinels < 3:
redis_info['alerts'].append({
'level': 'warning',
'message': f'Redis哨兵数量不足: {total_sentinels}/3，存在脑裂风险'
})
# 从节点数量验证
num_slaves = sentinel_info.get('num_slaves', 0)
if num_slaves < 2:
redis_info['alerts'].append({
'level': 'warning',
'message': f'Redis从节点数量不足: {num_slaves}/2'
})
# 主观/客观下线检测
flags = sentinel_info.get('master_flags', '')
if 's_down' in flags or 'o_down' in flags:
redis_info['alerts'].append({
'level': 'critical',
'message': f'Redis master状态异常: flags={flags}'
})
# 哨兵节点互访验证
sentinel_peers_cmd = f"docker exec {cname} redis-cli -p 26379 SENTINEL sentinels mymaster 2>/dev/null | grep -E '^flags' | head -10"
exit_code, stdout, stderr = self.ssh_manager.execute_command(sentinel_peers_cmd)
if exit_code == 0 and stdout.strip() and 's_down' in stdout.lower():
redis_info['alerts'].append({
'level': 'critical',
'message': '其他哨兵节点标记为s_down，哨兵间通信异常'
})
else:
redis_info['l3_status'] = False
redis_info['alerts'].append({
......@@ -227,9 +310,10 @@ class MiddlewareMonitor:
listening_ports.append(int(port_match.group(1)))
emqx_info['listening_ports'] = listening_ports
# L3: 功能层检查 - 检查集群状态
# L3: 功能层检查 - 集群状态 + 消息连通测试
if container.get('name'):
cname = container['name']
# 3.1 集群状态检查
l3_command = f"docker exec {cname} emqx ctl cluster status 2>/dev/null || docker exec {cname} emqx_ctl cluster status 2>/dev/null"
exit_code, stdout, stderr = self.ssh_manager.execute_command(l3_command)
......@@ -237,6 +321,22 @@ class MiddlewareMonitor:
if exit_code == 0 and stdout.strip():
emqx_info['l3_status'] = True
emqx_info['cluster_output'] = stdout.strip()[:200]
# 解析集群节点列表
running_nodes = []
stopped_nodes = []
for line in stdout.split('\n'):
if 'running_nodes' in line:
emqx_info['cluster_running'] = True
elif 'stopped_nodes' in line:
emqx_info['cluster_stopped'] = False
# 集群节点数量验证
if 'running_nodes' not in stdout.lower():
emqx_info['alerts'].append({
'level': 'critical',
'message': 'EMQX集群节点均未运行'
})
else:
emqx_info['l3_status'] = False
emqx_info['alerts'].append({
......@@ -244,6 +344,28 @@ class MiddlewareMonitor:
'message': 'EMQX集群状态检查失败，可能为版本差异'
})
# 3.2 消息连通测试：客户端数量统计
clients_cmd = f"docker exec {cname} emqx ctl clients list 2>/dev/null | head -20 || docker exec {cname} emqx_ctl clients list 2>/dev/null | head -20"
exit_code, stdout, stderr = self.ssh_manager.execute_command(clients_cmd)
if exit_code == 0:
client_lines = [l for l in stdout.strip().split('\n') if l.strip() and 'Client(' in l]
emqx_info['connected_clients'] = len(client_lines)
if len(client_lines) == 0:
emqx_info['alerts'].append({
'level': 'warning',
'message': 'EMQX当前无客户端连接，可能消息服务空闲'
})
# 3.3 订阅数统计
sub_cmd = f"docker exec {cname} emqx ctl broker stats 2>/dev/null | grep -E 'subscriptions' || echo fail"
exit_code, stdout, stderr = self.ssh_manager.execute_command(sub_cmd)
if exit_code == 0 and 'fail' not in stdout:
for line in stdout.split('\n'):
if 'subscriptions.count' in line:
emqx_info['subscriptions'] = int(line.split(':')[1].strip())
# 判定整体状态
emqx_info['overall_status'] = self._get_overall_status(emqx_info['alerts'])
......@@ -313,6 +435,14 @@ class MiddlewareMonitor:
'message': f'Nacos服务状态检查失败: HTTP {http_code}'
})
# 3.2 集群raft状态检测（通过日志判断）
raft_cmd = "docker exec " + container['name'] + " ls -la /home/nacos/logs/protocol/raft 2>/dev/null | head -5 || docker exec " + container['name'] + " ls -la /nacos/logs/protocol/raft 2>/dev/null | head -5"
exit_code, stdout, stderr = self.ssh_manager.execute_command(raft_cmd)
if exit_code == 0 and stdout.strip():
nacos_info['raft_log_exists'] = True
else:
nacos_info['raft_log_exists'] = False
# 判定整体状态
nacos_info['overall_status'] = self._get_overall_status(nacos_info['alerts'])
......@@ -438,6 +568,23 @@ class MiddlewareMonitor:
'message': '达梦数据库连接测试失败'
})
# 3.2 表空间使用率检查
tablespace_cmd = 'docker exec dm8-server /usr/local/dm8/bin/disql "SYSDBA/dNrprU&2S!" -e "SELECT TABLESPACE_NAME, ROUND(TOTAL_SIZE*PAGE_SIZE/1024/1024) AS TOTAL_MB, ROUND((TOTAL_SIZE-FREE_SIZE)*PAGE_SIZE/1024/1024) AS USED_MB, ROUND(FREE_SIZE*PAGE_SIZE/1024/1024) AS FREE_MB, ROUND((TOTAL_SIZE-FREE_SIZE)*100/TOTAL_SIZE,1) AS USED_PCT FROM V\\$TABLESPACE;" 2>&1'
exit_code, stdout, stderr = self.ssh_manager.execute_command(tablespace_cmd)
if exit_code == 0 and stdout.strip():
dm8_info['tablespace_check'] = True
# 检查是否有表空间使用率超过90%
for line in stdout.split('\n'):
if 'USED_PCT' not in line and len(line.strip()) > 0:
parts = line.split()
if len(parts) >= 5 and parts[-1].replace('.', '').isdigit():
pct = float(parts[-1])
if pct > 90:
dm8_info['alerts'].append({
'level': 'warning',
'message': f'达梦表空间{parts[0]}使用率{pct}%'
})
# 判定整体状态
dm8_info['overall_status'] = self._get_overall_status(dm8_info['alerts'])
......@@ -494,6 +641,44 @@ class MiddlewareMonitor:
exit_code, stdout, stderr = self.ssh_manager.execute_command(storage_port_command)
fastdfs_info['storage_port_up'] = exit_code == 0 and stdout.strip() != ''
# L3: 功能层检查 - 集群状态 + 同组互备检测
# 3.1 使用fdfs_monitor查看group状态
if fastdfs_info['tracker_process']:
monitor_cmd = "docker exec utracker fdfs_monitor /etc/fdfs/client.conf 2>/dev/null | head -50 || fdfs_monitor /etc/fdfs/client.conf 2>/dev/null | head -50"
exit_code, stdout, stderr = self.ssh_manager.execute_command(monitor_cmd)
if exit_code == 0 and stdout.strip():
fastdfs_info['l3_status'] = True
# 解析group信息
group_count = 0
storage_count = 0
for line in stdout.split('\n'):
if 'group count:' in line:
group_count = int(line.split(':')[1].strip())
elif 'group name' in line:
fastdfs_info['group_name'] = line.split('=')[1].strip()
elif 'storage server count' in line:
parts = line.split('=')
if len(parts) >= 2:
storage_count = int(parts[1].strip().split(',')[0])
fastdfs_info['group_count'] = group_count
fastdfs_info['storage_count'] = storage_count
# 同组互备检测：storage数量应>=2
if storage_count < 2:
fastdfs_info['alerts'].append({
'level': 'warning',
'message': f'FastDFS storage节点数不足: {storage_count}，建议>=2实现同组互备'
})
else:
fastdfs_info['l3_status'] = False
fastdfs_info['alerts'].append({
'level': 'warning',
'message': 'FastDFS fdfs_monitor命令执行失败'
})
# 判定整体状态
fastdfs_info['overall_status'] = self._get_overall_status(fastdfs_info['alerts'])
......
.claude/skills/ARM-CLUSTER-MONITOR/code/monitor_agent.py
浏览文件 @ 6eb7badf
......@@ -37,6 +37,14 @@ from log_analyzer import LogAnalyzer
from cluster_consistency import ClusterConsistency
from report_generator import ReportGenerator
# 导入钉钉通知模块
try:
from dingtalk_notifier import send_cluster_notification
DINGTALK_AVAILABLE = True
except ImportError:
DINGTALK_AVAILABLE = False
print("[警告] 钉钉通知模块未找到，通知功能将不可用")
class MonitorAgent:
"""主监控调度器类"""
......@@ -350,10 +358,13 @@ class MonitorAgent:
return report_path
def run_full_monitor(self) -> Dict:
def run_full_monitor(self, send_notification: bool = False) -> Dict:
"""
执行完整监控流程
Args:
send_notification: 是否发送钉钉通知，默认False（定时任务期间不发送）
Returns:
Dict: 完整的监控结果摘要（压缩格式）
"""
......@@ -392,6 +403,10 @@ class MonitorAgent:
# 6. 生成并保存报告
report_path = self.generate_and_save_report()
# 6.5. 发送钉钉通知（仅当参数允许时）
if send_notification:
self._send_dingtalk_notification()
# 7. 断开所有SSH连接
self.close_all_connections()
......@@ -534,7 +549,7 @@ class MonitorAgent:
name = node_data.get('name', ip)
overall = node_data.get('overall_status', 'unknown')
status_icon = '✅' if overall == 'normal' else ('⚠️' if overall == 'warning' else '❌')
status_icon = '[OK]' if overall == 'normal' else ('[WARN]' if overall == 'warning' else '[ERR]')
print(f"\n[完成] {status_icon} {name} ({ip}): {overall}")
......@@ -563,6 +578,36 @@ class MonitorAgent:
'report_path': '', # 报告生成后填充
}
def _send_dingtalk_notification(self):
"""
发送钉钉通知
如果钉钉通知模块可用且配置已启用，则在报告生成后发送通知。
定时任务执行模式下，通知发送不阻塞主流程。
"""
if not DINGTALK_AVAILABLE:
return
try:
print("\n[信息] 发送钉钉通知...")
# 获取配置中的通知设置
config = self.config_parser.config if hasattr(self.config_parser, 'config') else {}
send_cluster_notification(
monitor_time=self.monitor_time.strftime('%Y-%m-%d %H:%M:%S'),
node_count=len(self.servers),
overall_status=self.cluster_data.get('overall_status', 'normal'),
health_score=self.cluster_data.get('consistency', {}).get('health_score', {}).get('total_score', 0),
alert_count=len(self.cluster_data.get('alerts', [])),
servers_data=self.servers_data,
cluster_data=self.cluster_data
)
except Exception as e:
print(f"[警告] 钉钉通知发送失败: {str(e)}")
print("[信息] 通知发送失败不影响监控流程，继续执行")
def test_monitor_agent():
"""测试主监控调度器功能"""
......
.claude/skills/ARM-CLUSTER-MONITOR/code/report_generator.py
浏览文件 @ 6eb7badf
......@@ -760,7 +760,7 @@ class ReportGenerator:
role = server.get('role', '')
overall = server.get('overall_status', 'unknown')
status_icon = '✅' if overall == 'normal' else ('⚠️' if overall == 'warning' else '❌')
status_icon = '[OK]' if overall == 'normal' else ('[WARN]' if overall == 'warning' else '[ERR]')
print(f" {status_icon} {name} ({ip}): {role} - {overall}")
# 告警摘要
......@@ -769,13 +769,13 @@ class ReportGenerator:
if alerts:
for alert in alerts[:10]: # 只显示前10条
level = alert.get('level', 'info')
level_icon = '🔴' if level == 'critical' else ('🟡' if level == 'warning' else '🔵')
level_icon = '[CRIT]' if level == 'critical' else ('[WARN]' if level == 'warning' else '[INFO]')
print(f" {level_icon} [{level}] {alert.get('message', '')}")
if len(alerts) > 10:
print(f" ... 还有 {len(alerts) - 10} 条告警")
else:
print(" ✅ 无告警")
print(" [OK] 无告警")
# 健康评分
consistency = cluster_data.get('consistency', {})
......@@ -788,7 +788,7 @@ class ReportGenerator:
# 报告保存路径
if self.report_path:
print(f"\n--- 报告文件 ---")
print(f" 📄 {self.report_path}")
print(f" {self.report_path}")
print("\n" + "=" * 60)
......
.claude/skills/ARM-CLUSTER-MONITOR/code/system_monitor.py
浏览文件 @ 6eb7badf
......@@ -169,6 +169,9 @@ class SystemMonitor:
return memory_info
# 不监控的挂载点列表（非关键或临时挂载）
EXCLUDED_MOUNTS = ['/cdrom', '/snap', '/mnt/cdrom', '/media', '/run/user']
def check_disk_usage(self) -> Dict:
"""
检查磁盘使用率
......@@ -206,6 +209,10 @@ class SystemMonitor:
usage_percent = int(parts[4].replace('%', ''))
mount_point = parts[5]
# 排除非关键挂载点
if any(excluded in mount_point for excluded in self.EXCLUDED_MOUNTS):
continue
partition_info = {
'filesystem': filesystem,
'size': size,
......@@ -271,6 +278,10 @@ class SystemMonitor:
usage_percent = int(parts[4].replace('%', ''))
mount_point = parts[5]
# 排除非关键挂载点
if any(excluded in mount_point for excluded in self.EXCLUDED_MOUNTS):
continue
partition_info = {
'filesystem': filesystem,
'total_inodes': total_inodes,
......
.claude/skills/X86-CLUSTER-MONITOR/SKILL.md
浏览文件 @ 6eb7badf
......@@ -74,6 +74,24 @@ schedule_interval: "2h"
- 计划执行文档需包含：执行目标、执行步骤、预期结果、风险评估
- 用户确认后方可执行
### 0.6 定时任务执行规范（强制要求）
> ⚠️ 定时任务执行模式下，必须遵守以下规范：
1. **跳过计划确认步骤**：
- 定时任务触发时，**跳过** `0.4 需求文档处理规范` 中的"输出《计划执行》文档"和"用户确认后方可执行"步骤
- 直接使用 `python main.py` 执行集群监控，无需任何交互确认
2. **判断依据**：
- 定时任务由 Cron 触发，prompt 为 `/X86-Cluster-Monitor`
- 此时无需输出计划执行文档，直接运行代码完成监控即可
3. **执行原则**：
- 零交互：不调用 `AskUserQuestion`，不等待用户确认
- 快速完成：直接运行 Python 脚本，输出摘要后结束
- 静默运行：减少不必要的输出，仅保留关键摘要信息
- 异常处理：连接失败自动 fallback，凭据从 `config.json` 读取，不会因缺凭据而阻塞
**计划执行文档模板**:
```markdown
# 计划执行文档
......@@ -550,7 +568,98 @@ AuxiliaryTool/ScriptTool/ClusterMonitor/reports/
---
## 9. 异常处理
## 9. 钉钉通知功能
### 9.1 功能说明
监控完成后自动发送**日报汇总**到钉钉群聊，便于及时发现异常。
**重要**：为节约公司资源，钉钉通知**仅在最后一次定时任务（08:00）执行后发送一次**，并附带整夜健康度趋势分析。
### 9.2 发送逻辑
```
定时任务执行时间: 22:00, 00:00, 02:00, 04:00, 06:00, 08:00
┌─────────────────────────────────────────────────────────────┐
│ 钉钉通知发送策略 │
├─────────────────────────────────────────────────────────────┤
│ 22:00 执行监控 → 生成报告 → 不发送钉钉 │
│ 00:00 执行监控 → 生成报告 → 不发送钉钉 │
│ 02:00 执行监控 → 生成报告 → 不发送钉钉 │
│ 04:00 执行监控 → 生成报告 → 不发送钉钉 │
│ 06:00 执行监控 → 生成报告 → 不发送钉钉 │
│ 08:00 执行监控 → 生成报告 → 收集本周期所有报告 │
│ → 生成健康度趋势分析 │
│ → 发送一次汇总通知（带防重发机制） │
└─────────────────────────────────────────────────────────────┘
```
**防重发机制**：
- 发送成功后创建标记文件 `.dingtalk_sent_{YYYYMMDD}`
- 当日再次触发时检测到标记文件则跳过发送
- 确保每天只发送一次
### 9.3 配置方法
在 `config.json` 中配置钉钉参数：
```json
{
"dingtalk": {
"enabled": true,
"webhook": "https://oapi.dingtalk.com/robot/send?access_token=YOUR_TOKEN",
"secret": "SECYOUR_SECRET"
}
}
```
**配置项说明**:
| 配置项 | 说明 | 必填 |
|--------|------|------|
| `enabled` | 是否启用钉钉通知 | 是，默认false |
| `webhook` | 钉钉机器人Webhook地址 | 是 |
| `secret` | 签名密钥（加签验证） | 是 |
### 9.4 消息格式（日报汇总）
```
### 🖥️ X86集群监控日报 - 新统一平台X86集群
**日期**: 2026-07-11
**最终状态**: 🟢 正常
**本周期报告数**: 6 次
#### 📊 最终指标
| 指标 | 值 |
|:---|:---|
| 集群健康度 | 98.5% |
| 服务器状态 | 🟢3 🟡1 🔴0 |
#### 📈 健康度趋势
22:00: 98.0% → 00:00: 97.5% → 02:00: 98.2% → 04:00: 98.0% → 06:00: 97.8% → 08:00: 98.5%
📊 **趋势**: 状态稳定
#### 📋 服务器汇总
- 🟢 **192.168.5.41** (master): CPU 15%, 内存 45%
- 🟡 **192.168.5.42** (slave1): CPU 25%, 内存 78%
- 🟢 **192.168.5.43** (slave2): CPU 12%, 内存 52%
- 🟢 **192.168.5.40** (dm8-server): CPU 18%, 内存 48%
---
本通知仅发送一次 · 2026-07-11 08:05:00
```
### 9.5 注意事项
1. **每日仅发送一次**：防重发机制确保不浪费公司资源
2. **趋势分析**：收集整夜6次报告的健康度，分析变化趋势
3. 钉钉通知失败不影响监控主流程
4. 需安装 `requests` 库：`pip install requests`
---
## 10. 异常处理
| 异常情况 | 处理方式 |
|---------|---------|
......@@ -575,3 +684,5 @@ AuxiliaryTool/ScriptTool/ClusterMonitor/reports/
| 日期 | 版本 | 更新内容 |
|------|------|---------|
| 2026-07-09 | v1.0 | 初始版本，创建skill骨架 |
| 2026-07-11 | v1.1 | 新增0.6定时任务执行规范，定时模式下跳过计划确认，零交互自动执行 |
| 2026-07-11 | v1.2 | 接入钉钉通知模块，每日仅发送一次（08:00），附带健康度趋势分析 |
\ No newline at end of file
.claude/skills/X86-CLUSTER-MONITOR/code/dingtalk_notifier.py 0 → 100644
浏览文件 @ 6eb7badf
# -*- coding: utf-8 -*-
"""
钉钉通知模块 - 发送集群监控日报
从监控数据中提取关键信息，构建详细的日报汇总
功能：
1. 从 check_results 中提取集群服务详情（Redis/EMQX/Nacos/FastDFS/达梦）
2. 收集本周期所有报告的健康度趋势和告警
3. 按标准格式构建日报消息
4. 发送到钉钉群聊（带防重发机制）
"""
import json
import time
import hmac
import hashlib
import base64
import urllib.parse
import logging
import re
from datetime import datetime, timedelta
from pathlib import Path
from typing import Dict, Any, List
logger = logging.getLogger(__name__)
try:
import requests
except ImportError:
requests = None
logger.warning("未安装requests库，钉钉通知功能不可用。请执行: pip install requests")
class DingTalkNotifier:
"""钉钉通知发送器"""
def __init__(self, webhook: str, secret: str):
self.webhook = webhook
self.secret = secret
def _generate_signature(self, timestamp: int = None) -> str:
if timestamp is None:
timestamp = int(time.time() * 1000)
string_to_sign = f"{timestamp}\n{self.secret}"
hmac_sha256 = hmac.new(
self.secret.encode('utf-8'),
string_to_sign.encode('utf-8'),
digestmod=hashlib.sha256
)
return urllib.parse.quote(base64.b64encode(hmac_sha256.digest()).decode('utf-8'))
def send_markdown(self, title: str, text: str) -> Dict[str, Any]:
if requests is None:
return {'errcode': -1, 'errmsg': 'requests库未安装'}
timestamp = int(time.time() * 1000)
sign = self._generate_signature(timestamp)
url = f"{self.webhook}&timestamp={timestamp}&sign={sign}"
message = {"msgtype": "markdown", "markdown": {"title": title, "text": text}}
try:
response = requests.post(url, headers={'Content-Type': 'application/json'}, json=message, timeout=15)
result = response.json()
if result.get('errcode') == 0:
logger.info("钉钉消息发送成功")
else:
logger.error(f"钉钉消息发送失败: {result.get('errmsg', 'Unknown error')}")
return result
except Exception as e:
logger.error(f"钉钉消息发送异常: {e}")
return {'errcode': -1, 'errmsg': str(e)}
def load_dingtalk_config(config_path: str = None) -> Dict[str, Any]:
"""从config.json加载钉钉配置"""
if config_path is None:
config_path = Path(__file__).parent.parent / 'config.json'
else:
config_path = Path(config_path)
if not config_path.exists():
return {}
try:
with open(config_path, 'r', encoding='utf-8') as f:
config = json.load(f)
return config.get('dingtalk', {})
except Exception as e:
logger.error(f"加载钉钉配置失败: {e}")
return {}
# ===================================================================
# 服务信息提取函数（从 check_results 中取真实数据）
# ===================================================================
def _extract_redis_info(check_results, cluster_config) -> Dict:
info = {'replication_status': '未知', 'master_ip': '未知', 'slave_ips': '未知',
'sentinels': '未知', 'memory_usage': '未知'}
redis_results = []
for r in check_results:
redis = r.get('redis', {})
if redis.get('status') != 'skip':
redis_results.append({
'ip': r['server_ip'], 'role': redis.get('role', 'unknown'),
'info': redis.get('info', {}), 'sentinel': redis.get('sentinel', {})
})
master = next((r for r in redis_results if r['role'] == 'master'), None)
if master:
info['master_ip'] = master['ip']
slaves = [r for r in redis_results if r['role'] == 'slave']
info['slave_ips'] = ', '.join(s['ip'] for s in slaves) if slaves else '无'
info['replication_status'] = '[OK] 正常' if (master and slaves) else '[WARN] 异常'
sentinel_count = 0
for r in redis_results:
sentinel = r.get('sentinel', {})
if sentinel and sentinel.get('sentinels'):
try:
sentinel_count = int(sentinel.get('sentinels', 0))
except (ValueError, TypeError):
pass
break
info['sentinels'] = f'{sentinel_count}个哨兵' if sentinel_count else '未知'
for r in redis_results:
rinfo = r.get('info', {})
if rinfo.get('used_memory_human'):
info['memory_usage'] = rinfo['used_memory_human']
break
return info
def _extract_fastdfs_info(check_results, cluster_config) -> Dict:
info = {'status': '未知', 'tracker_info': '未知', 'storage_info': '未知', 'capacity': '未知'}
fdfs_config = cluster_config.get('cluster_services', {}).get('fastdfs_cluster', {})
trackers = fdfs_config.get('tracker_nodes', [])
storages = fdfs_config.get('storage_nodes', [])
if trackers:
info['tracker_info'] = ', '.join(f'{t}:22122' for t in trackers)
if storages:
info['storage_info'] = 'group1 - ' + '/'.join([s.split('.')[-1] for s in storages])
all_normal = all(
r.get('fastdfs', {}).get('status') in ('normal', 'skip')
for r in check_results
)
info['status'] = '[OK] 正常' if all_normal else '[WARN] 异常'
for r in check_results:
fdfs = r.get('fastdfs', {})
if fdfs.get('total_space') and fdfs.get('free_space'):
free = fdfs['free_space']
total = fdfs['total_space']
try:
free_mb = float(free)
total_mb = float(total)
used_pct = (1 - free_mb / total_mb) * 100 if total_mb > 0 else 0
info['capacity'] = f'{free}MB/{total}MB ({used_pct:.0f}%)'
except (ValueError, TypeError):
info['capacity'] = f'{free}MB/{total}MB'
break
return info
def _extract_dm8_info(check_results) -> Dict:
info = {'status': '未知', 'node': '无', 'instance': 'DM_SERVER', 'connections': '未知'}
for r in check_results:
dm8 = r.get('dm8', {})
if dm8.get('status') == 'skip':
continue
info['node'] = r['server_ip']
if dm8.get('connections'):
info['connections'] = dm8['connections']
proc_ok = any(
it.get('status') == 'pass' and '进程' in it.get('name', '')
for it in dm8.get('items', [])
)
conn_ok = any(
it.get('status') == 'pass' and 'SQL' in it.get('name', '')
for it in dm8.get('items', [])
)
if conn_ok:
info['status'] = '[OK] 正常'
elif proc_ok:
info['status'] = '[WARN] SQL连接失败'
else:
info['status'] = '[ERROR] 进程未运行'
break
return info
def _extract_emqx_info(check_results) -> Dict:
info = {'status': '未知', 'nodes': '未知', 'node_count': 0, 'connections': '未知', 'subscriptions': '未知'}
emqx_ips = []
all_normal = True
for r in check_results:
emqx = r.get('emqx', {})
if emqx.get('status') == 'skip':
continue
emqx_ips.append(r['server_ip'])
if emqx.get('status') != 'normal':
all_normal = False
if emqx.get('connections'):
info['connections'] = emqx['connections']
if emqx.get('subscriptions'):
info['subscriptions'] = emqx['subscriptions']
info['nodes'] = '/'.join(emqx_ips) if emqx_ips else '无'
info['node_count'] = len(emqx_ips)
info['status'] = '[OK] 正常' if (all_normal and emqx_ips) else '[WARN] 异常'
return info
def _extract_nacos_info(check_results) -> Dict:
info = {'status': '未知', 'nodes': '未知', 'node_count': 0, 'config_count': '未知', 'services': '未知'}
nacos_ips = []
for r in check_results:
nacos = r.get('nacos', {})
if nacos.get('status') == 'skip':
continue
nacos_ips.append(r['server_ip'])
if nacos.get('service_count'):
info['services'] = f'{nacos["service_count"]}个'
if nacos.get('config_count'):
info['config_count'] = nacos['config_count']
info['nodes'] = '/'.join(nacos_ips) if nacos_ips else '无'
info['node_count'] = len(nacos_ips)
all_normal = all(
r.get('nacos', {}).get('status') == 'normal'
for r in check_results if r.get('nacos', {}).get('status') != 'skip'
)
info['status'] = '[OK] 正常' if (all_normal and nacos_ips) else '[WARN] 异常'
return info
def _get_server_status(result: dict) -> str:
if 'error' in result:
return 'error'
score = 100.0
for c in result.get('containers', []):
if c.get('expected', True) and not c.get('running', True):
score -= 10
for mw_key in ['mysql', 'redis', 'emqx', 'nacos', 'fastdfs', 'dm8']:
mw = result.get(mw_key, {})
if mw.get('status') == 'error':
score -= 10
if score >= 95:
return 'normal'
elif score >= 80:
return 'warning'
else:
return 'error'
# ===================================================================
# 报告趋势提取
# ===================================================================
def extract_resource_trend(report_dir: Path, cycle_start, cycle_end) -> Dict:
trend = {'nodes': {}, 'scores': [], 'alert_count': 0, 'alerts': []}
for report_file in sorted(report_dir.glob('X86_cluster_monitor_*.md')):
try:
filename = report_file.stem
time_str = filename.replace('X86_cluster_monitor_', '')
report_time = datetime.strptime(time_str, '%Y%m%d_%H%M%S')
if not (cycle_start <= report_time <= cycle_end):
continue
time_label = report_time.strftime('%H:%M')
with open(report_file, 'r', encoding='utf-8') as f:
content = f.read()
health_match = re.search(r'集群健康度.*?\|\s*(\d+\.?\d*)%', content)
if health_match:
trend['scores'].append((time_label, float(health_match.group(1))))
table = re.search(r'## 三、服务器状态汇总(.+?)## 四、详细检查结果', content, re.DOTALL)
if table:
rows = re.findall(
r'\|\s*\S+\s*\|\s*([\d.]+)\s*\|\s*\S+\s*\|\s*[🔴🟡🟢]\s*\|\s*([\d.]+)%\s*\|\s*([\d.]+)%',
table.group(1)
)
for ip, cpu, mem in rows:
cpu_val = float(cpu)
mem_val = float(mem)
if ip not in trend['nodes']:
trend['nodes'][ip] = {'cpu_peak': cpu_val, 'mem_peak': mem_val, 'disk_peak': 0}
else:
trend['nodes'][ip]['cpu_peak'] = max(trend['nodes'][ip]['cpu_peak'], cpu_val)
trend['nodes'][ip]['mem_peak'] = max(trend['nodes'][ip]['mem_peak'], mem_val)
alert_section = re.search(r'## 七、集群整体分析(.+?)## 九、建议', content, re.DOTALL)
if alert_section:
for line in alert_section.group(1).split('\n'):
line = line.strip()
if re.match(r'\d+\.\s*', line):
alert_content = re.sub(r'^\d+\.\s*', '', line)
if '192.168.5' in alert_content:
ip_match = re.search(r'(192\.168\.5\.\d+)', alert_content)
node_ip = ip_match.group(1) if ip_match else '未知'
trend['alerts'].append((time_label, node_ip, alert_content.strip()))
trend['alert_count'] = len(trend['alerts'])
except Exception as e:
logger.warning(f"解析报告趋势失败 {report_file}: {e}")
return trend
def compute_cycle_range():
now = datetime.now()
if now.hour >= 8 and now.hour < 22:
cycle_start = (now - timedelta(days=1)).replace(hour=22, minute=0, second=0, microsecond=0)
cycle_end = now.replace(hour=8, minute=0, second=0, microsecond=0)
elif now.hour >= 22:
cycle_start = now.replace(hour=22, minute=0, second=0, microsecond=0)
cycle_end = now
else:
cycle_start = (now - timedelta(days=1)).replace(hour=22, minute=0, second=0, microsecond=0)
cycle_end = now
return cycle_start, cycle_end
# ===================================================================
# 日报构建
# ===================================================================
def build_daily_report(check_results, cluster_config, report_dir, cycle_start, cycle_end):
trend = extract_resource_trend(Path(report_dir), cycle_start, cycle_end)
date_str = cycle_start.strftime('%Y%m%d')
time_range = f"{cycle_start.strftime('%Y-%m-%d %H:%M:%S')} ~ {cycle_end.strftime('%Y-%m-%d %H:%M:%S')}"
exec_count = len(trend['scores'])
first_score = trend['scores'][0][1] if trend['scores'] else 0
last_score = trend['scores'][-1][1] if trend['scores'] else 0
total_error = sum(1 for r in check_results if _get_server_status(r) == 'error')
total_warn = sum(1 for r in check_results if _get_server_status(r) == 'warning')
if total_error > 0:
overall_status = "[ERROR] error"
elif total_warn > 0:
overall_status = "[WARN] warning"
else:
overall_status = "[OK] normal"
# 一、资源趋势
resource_table = "| 节点 | CPU峰值 | 内存峰值 | 磁盘变化 |\n|:---|:---|:---|:---|\n"
for ip, data in trend['nodes'].items():
resource_table += f"| {ip} | {data['cpu_peak']:.0f}% | {data['mem_peak']:.0f}% | - |\n"
# 二~六、服务状态
redis_info = _extract_redis_info(check_results, cluster_config)
fastdfs_info = _extract_fastdfs_info(check_results, cluster_config)
dm8_info = _extract_dm8_info(check_results)
emqx_info = _extract_emqx_info(check_results)
nacos_info = _extract_nacos_info(check_results)
title = "X86集群夜间监控汇总报告"
text = f"""### X86集群夜间监控汇总报告
**日期**: {date_str}
**时段**: {time_range}
**执行**: {exec_count}次
**状态**: {overall_status}
**评分**: {first_score:.0f} -> {last_score:.0f}
#### 一、集群资源趋势
{resource_table}
#### 二、Redis集群状态
**主从复制**: {redis_info['replication_status']}
**当前主节点**: {redis_info['master_ip']}
**从节点**: {redis_info['slave_ips']}
**哨兵**: {redis_info['sentinels']}
**内存使用**: {redis_info['memory_usage']}
#### 三、FastDFS集群状态
**同组互备**: {fastdfs_info['status']}
**Tracker**: {fastdfs_info['tracker_info']}
**Storage**: {fastdfs_info['storage_info']}
**存储容量**: {fastdfs_info['capacity']}
#### 四、达梦数据库状态
**运行状态**: {dm8_info['status']}
**节点**: {dm8_info['node']}
**实例**: {dm8_info['instance']}
**连接数**: {dm8_info['connections']}
#### 五、EMQX集群状态
**集群状态**: {emqx_info['status']}
**节点**: {emqx_info['nodes']}（{emqx_info['node_count']}台）
**连接数**: {emqx_info['connections']}
**订阅数**: {emqx_info['subscriptions']}
#### 六、Nacos集群状态
**集群状态**: {nacos_info['status']}
**节点**: {nacos_info['nodes']}（{nacos_info['node_count']}台）
**配置数**: {nacos_info['config_count']}
**服务注册**: {nacos_info['services']}
**累计告警**: {trend['alert_count']}次
"""
return title, text
# ===================================================================
# 发送入口
# ===================================================================
def send_daily_report(check_results, cluster_config, report_dir, config_path=None):
today_str = datetime.now().strftime('%Y%m%d')
report_path = Path(report_dir)
sentinel_file = report_path / f'.dingtalk_sent_{today_str}'
if sentinel_file.exists():
logger.info(f'今日钉钉通知已发送（标记文件: {sentinel_file.name}），跳过重复发送')
return False
dingtalk_config = load_dingtalk_config(config_path)
if not dingtalk_config or not dingtalk_config.get('enabled', False):
logger.info("钉钉通知已禁用")
return False
webhook = dingtalk_config.get('webhook', '')
secret = dingtalk_config.get('secret', '')
if not webhook or not secret:
logger.warning("钉钉webhook或secret未配置")
return False
cycle_start, cycle_end = compute_cycle_range()
title, text = build_daily_report(check_results, cluster_config, report_dir, cycle_start, cycle_end)
notifier = DingTalkNotifier(webhook, secret)
result = notifier.send_markdown(title, text)
if result.get('errcode') == 0:
sentinel_file.touch()
logger.info(f'钉钉日报发送成功，已创建防重发标记')
return True
else:
logger.error(f'钉钉发送失败: {result}')
return False
\ No newline at end of file
.claude/skills/X86-CLUSTER-MONITOR/code/main.py
浏览文件 @ 6eb7badf
......@@ -231,9 +231,51 @@ class ClusterMonitor:
if check_results:
print_execution_summary(check_results, report_path)
# 发送钉钉通知（仅在最后一次定时任务执行时发送）
self._send_dingtalk_notification_if_final(check_results, report_path)
logger.info(f'监控完成')
return report_path
def _send_dingtalk_notification_if_final(self, check_results, report_path):
"""
发送钉钉通知（仅在最后一次定时任务执行时）
"""
from datetime import datetime
from dingtalk_notifier import send_daily_report
try:
current_hour = datetime.now().hour
is_final_run = (current_hour == 8)
if not is_final_run:
logger.info(f'当前时间 {current_hour}:00 不是最后一次定时任务，跳过钉钉通知')
return
logger.info('检测到最后一次定时任务执行（08:00），准备发送日报...')
# 获取报告目录
report_dir = Path(report_path).parent if report_path else None
if not report_dir:
logger.warning('报告目录不存在')
return
# 调用钉钉通知模块
success = send_daily_report(
check_results=check_results,
cluster_config=self.cluster_config,
report_dir=str(report_dir),
config_path=str(Path(__file__).parent.parent / 'config.json')
)
if success:
logger.info('钉钉日报发送成功')
else:
logger.info('钉钉日报已跳过或发送失败')
except Exception as e:
logger.warning(f'钉钉通知发送异常（不影响监控主流程）: {e}')
def main():
"""主函数"""
......
.claude/skills/X86-CLUSTER-MONITOR/code/report_generator.py
浏览文件 @ 6eb7badf
......@@ -139,13 +139,16 @@ class ReportGenerator:
content += "## 五、服务拓扑校验\n\n"
content += self._generate_service_topology_table(check_results)
content += "\n---\n\n## 六、集群整体分析\n\n"
content += "\n---\n\n## 六、集群跨节点一致性验证\n\n"
content += self._generate_cross_node_verification(check_results)
content += "\n---\n\n## 七、集群整体分析\n\n"
content += self._generate_cluster_analysis(check_results)
content += "\n---\n\n## 七、建议\n\n"
content += "\n---\n\n## 九、建议\n\n"
content += self._generate_recommendations(check_results)
content += "\n---\n\n## 八、附录\n\n"
content += "\n---\n\n## 十、附录\n\n"
content += self._generate_appendix(timestamp)
return content
......@@ -534,6 +537,27 @@ class ReportGenerator:
| 内存使用率 | {self.thresholds.get('memory_warning', 85)}% | {self.thresholds.get('memory_critical', 95)}% |
| 磁盘使用率 | {self.thresholds.get('disk_warning', 90)}% | {self.thresholds.get('disk_critical', 95)}% |
### 检测项列表（v2.1增强版）
**系统资源检查（6项）**：CPU、内存、磁盘、负载、网络连接数、运行时间
**容器检查（2项）**：容器状态、服务拓扑校验
**中间件检查（增强版）**：
- MySQL（2项）：连通性、数据库大小
- Redis（6项）：PING、角色、INFO、**主从复制状态**、**持久化状态**、哨兵状态
- EMQX（5项）：节点状态、集群状态、监听器、**MQTT消息连通性**、**Dashboard**
- Nacos（4项）：Web探活、集群状态、服务列表、**配置中心**
- FastDFS（4项）：Tracker进程、Storage进程、集群状态、**文件上传测试**
- 达梦DM8（5项）：进程、端口、**SQL连接测试**、**表空间**、**许可证**
**跨节点验证（v2.1新增）**：
- Redis哨兵一致性验证
- Redis主从同步状态验证
- Nacos数据一致性验证
- 节点间网络连通性验证
- 系统时间同步检查
### 报告路径
`{self.report_dir}`
......@@ -541,6 +565,237 @@ class ReportGenerator:
- 节点1(192.168.5.41) unacos: 控制台登录失败（部署改密破坏认证链），业务面正常
"""
# ==================== 跨节点一致性验证 ====================
def _generate_cross_node_verification(self, results: list) -> str:
"""
生成跨节点一致性验证章节（v2.1新增）
包括: Redis哨兵一致性、Nacos数据一致性、节点间连通性、时间同步、容器版本一致性
"""
content = ""
# 过滤掉连接失败的服务器
valid_results = [r for r in results if 'error' not in r]
# 1. Redis哨兵跨节点一致性
content += "### 6.1 Redis哨兵一致性\n\n"
content += self._verify_redis_sentinel_consistency(valid_results)
# 2. Redis主从同步一致性
content += "\n### 6.2 Redis主从同步状态\n\n"
content += self._verify_redis_replication_consistency(valid_results)
# 3. Nacos跨节点数据一致性
content += "\n### 6.3 Nacos集群数据一致性\n\n"
content += self._verify_nacos_consistency(valid_results)
# 4. 节点间网络连通性
content += "\n### 6.4 节点间网络连通性\n\n"
content += self._verify_cross_node_connectivity(valid_results)
# 5. 时间同步检查
content += "\n### 6.5 系统时间同步\n\n"
content += self._verify_time_sync(valid_results)
return content
def _verify_redis_sentinel_consistency(self, results: list) -> str:
"""验证Redis哨兵跨节点一致性"""
sentinel_nodes = []
for r in results:
redis_data = r.get('redis', {})
if redis_data.get('status') == 'skip':
continue
sentinel_info = redis_data.get('sentinel', {})
if sentinel_info:
sentinel_nodes.append({
'ip': r['server_ip'],
'master_ip': sentinel_info.get('ip', 'unknown'),
'master_port': sentinel_info.get('port', 'unknown'),
'flags': sentinel_info.get('flags', 'unknown'),
'sentinels': sentinel_info.get('sentinels', 'unknown'),
'quorum': sentinel_info.get('quorum', 'unknown'),
})
if not sentinel_nodes:
return "⏭️ 无哨兵节点\n"
content = "| 哨兵节点 | 认知的主节点 | 状态 | 哨兵数 | quorum |\n"
content += "|---------|------------|------|--------|--------|\n"
master_ips = set()
for n in sentinel_nodes:
master_ips.add(f"{n['master_ip']}:{n['master_port']}")
content += f"| {n['ip']} | {n['master_ip']}:{n['master_port']} | {n['flags']} | {n['sentinels']} | {n['quorum']} |\n"
if len(master_ips) == 1:
content += f"\n✅ **一致性**: 所有哨兵一致认为主节点是 `{list(master_ips)[0]}`\n"
else:
content += f"\n❌ **脑裂风险**: 哨兵认知的主节点不一致！存在 {len(master_ips)} 个不同认知: {', '.join(master_ips)}\n"
# 哨兵数量验证
sentinel_count = int(sentinel_nodes[0].get('sentinels', 0)) if sentinel_nodes else 0
if sentinel_count < 2:
content += f"⚠️ **哨兵不足**: 当前只有 {sentinel_count} 个哨兵，quorum可能无法达成\n"
return content
def _verify_redis_replication_consistency(self, results: list) -> str:
"""验证Redis主从复制一致性"""
repl_info = []
for r in results:
redis_data = r.get('redis', {})
if redis_data.get('status') == 'skip':
continue
repl = redis_data.get('replication', {})
role = repl.get('role', redis_data.get('role', 'unknown'))
repl_info.append({
'ip': r['server_ip'],
'role': role,
'connected_slaves': repl.get('connected_slaves', 'N/A'),
'master_link_status': repl.get('master_link_status', 'N/A'),
'master_last_io': repl.get('master_last_io_seconds_ago', 'N/A'),
'slave_repl_offset': repl.get('slave_repl_offset', 'N/A'),
})
if not repl_info:
return "⏭️ 无Redis节点\n"
content = "| 节点 | 角色 | 从节点数 | 主连接状态 | 通信间隔 |\n"
content += "|------|------|---------|-----------|--------|\n"
master_node = None
slave_nodes = []
for info in repl_info:
master_link = info['master_link_status']
if info['role'] == 'master':
master_node = info
master_link = 'N/A(自身为主)'
elif info['role'] == 'slave':
slave_nodes.append(info)
status_icon = '✅' if master_link in ('up', 'N/A(自身为主)') else '❌'
content += f"| {info['ip']} | {info['role']} | {info['connected_slaves']} | {status_icon} {master_link} | {info['master_last_io']}s |\n"
# 主节点一致性检查：从节点认知的主节点应与预期一致
if master_node:
content += f"\n📌 **主节点**: {master_node['ip']}（从节点数: {master_node['connected_slaves']}）\n"
# 检查预期主节点（从config读取）
expected_master = self.cluster_config.get('cluster_services', {}).get('redis_cluster', {}).get('master', '')
if expected_master and expected_master != master_node['ip']:
content += f"⚠️ **主节点漂移**: 预期主节点为 `{expected_master}`，实际为 `{master_node['ip']}`，可能发生了failover\n"
# 同步状态
for slave in slave_nodes:
if slave['master_link_status'] != 'up':
content += f"❌ **同步异常**: {slave['ip']} 主从连接状态: {slave['master_link_status']}\n"
try:
io_seconds = int(slave['master_last_io'])
if io_seconds > 60:
content += f"⚠️ **同步延迟**: {slave['ip']} 与主节点通信间隔 {io_seconds}s（>60s）\n"
except (ValueError, TypeError):
pass
return content
def _verify_nacos_consistency(self, results: list) -> str:
"""验证Nacos跨节点数据一致性"""
nacos_nodes = []
for r in results:
nacos_data = r.get('nacos', {})
if nacos_data.get('status') == 'skip':
continue
# 提取服务列表用于对比
service_list = nacos_data.get('service_list', '')
nacos_nodes.append({
'ip': r['server_ip'],
'service_list': service_list,
'cluster_state': nacos_data.get('nacos_cluster_state', {}),
})
if not nacos_nodes:
return "⏭️ 无Nacos节点\n"
content = f"| 节点 | 状态 | 服务列表 |\n"
content += f"|------|------|--------|\n"
service_sets = []
for n in nacos_nodes:
svc_len = len(n['service_list']) if n['service_list'] else 0
content += f"| {n['ip']} | ✅ | {svc_len} bytes |\n"
service_sets.append(n['service_list'])
# 对比所有节点的服务列表是否一致
if len(set(service_sets)) <= 1:
content += "\n✅ **服务列表一致**: 所有Nacos节点注册的服务列表一致\n"
else:
content += "\n⚠️ **服务列表不一致**: 不同Nacos节点返回的服务列表存在差异，可能Raft同步异常\n"
return content
def _verify_cross_node_connectivity(self, results: list) -> str:
"""验证节点间网络连通性"""
all_connectivity = []
for r in results:
conn = r.get('cross_node_connectivity', {})
if conn.get('status') == 'skip':
continue
for node in conn.get('nodes', []):
for check in node.get('checks', []):
all_connectivity.append({
'from': r['server_ip'],
'to': node['ip'],
'check': check['name'],
'status': check['status'],
})
if not all_connectivity:
return "⏭️ 无跨节点连通性数据\n"
content = "| 源节点 | 目标节点 | 检测项 | 状态 |\n"
content += "|--------|---------|--------|------|\n"
all_pass = True
for c in all_connectivity:
icon = '✅' if c['status'] == 'pass' else '❌'
if c['status'] != 'pass':
all_pass = False
content += f"| {c['from']} | {c['to']} | {c['check']} | {icon} |\n"
if all_pass:
content += "\n✅ **网络连通性正常**: 所有节点间关键端口均可达\n"
else:
content += "\n❌ **网络连通性异常**: 部分节点间端口不可达，请检查防火墙和网络配置\n"
return content
def _verify_time_sync(self, results: list) -> str:
"""验证系统时间同步"""
content = ""
# 从系统资源中获取uptime信息（间接判断时间）
# 更准确的方法是从各节点获取时间戳
timestamps = []
for r in results:
ip = r['server_ip']
uptime = r.get('system_resources', {}).get('uptime', {}).get('value', 'N/A')
timestamps.append({'ip': ip, 'uptime': uptime})
if not timestamps:
return "⏭️ 无时间数据\n"
content += "| 节点 | 启动时间 |\n"
content += "|------|--------|\n"
for t in timestamps:
content += f"| {t['ip']} | {t['uptime']} |\n"
content += ("\n💡 **建议**: 各节点通过NTP服务保持时间同步，"
"时间偏差过大会影响Redis哨兵故障判断和Nacos分布式一致性。\n"
"如需精确时间偏差检查，建议在 server_checker 中添加 `date '+%s'` 检查。\n")
return content
# ==================== 辅助方法 ====================
def _count_server_status(self, results: list) -> tuple:
......@@ -565,7 +820,7 @@ class ReportGenerator:
return sum(scores) / len(scores)
def _calculate_server_health(self, result: dict) -> float:
"""计算单台服务器健康度"""
"""计算单台服务器健康度（v2.1增强版）"""
score = 100.0
# 容器检查（最高扣30分）
......@@ -602,6 +857,13 @@ class ReportGenerator:
elif disk.get('status') == 'warning':
score -= 5
# 跨节点连通性（最高扣15分）（v2.1新增）
conn = result.get('cross_node_connectivity', {})
if conn.get('status') == 'error':
score -= 10
elif conn.get('status') == 'warning':
score -= 5
return max(score, 0)
def _get_server_status_emoji(self, result: dict) -> str:
......
.claude/skills/X86-CLUSTER-MONITOR/code/server_checker.py
浏览文件 @ 6eb7badf
......@@ -354,6 +354,12 @@ class ServerChecker:
检查Redis集群状态（1主2从3哨兵模式）
根据节点角色检查对应的Redis角色
增强项（v2.1）:
- 主从同步状态（master_link_status）
- 主从同步偏移量对比（slave_repl_offset vs master_repl_offset）
- 主从连接延迟（master_last_io_seconds_ago）
- 持久化状态（RDB/AOF写入状态）
Returns:
dict: Redis集群状态
"""
......@@ -366,7 +372,7 @@ class ServerChecker:
credentials = self.cluster_config.get('middleware_credentials', {}).get('redis', {})
password = credentials.get('password', 'Ubains@123')
redis_status = {'status': 'unknown', 'items': [], 'role': None}
redis_status = {'status': 'unknown', 'items': [], 'role': None, 'replication': {}}
# 找到本节点的Redis容器名
redis_container = None
......@@ -432,7 +438,100 @@ class ServerChecker:
info_item['output'] = (res.stdout + res.stderr).strip()
redis_status['items'].append(info_item)
# 4) 哨兵检查（如果本节点有哨兵）
# 4) 主从复制状态详细检查（v2.1新增）
cmd = f"redis-cli -a '{password}' --no-auth-warning INFO replication"
res = self.ssh.docker_exec(redis_container, cmd, timeout=15)
repl_item = {'name': '主从复制状态', 'status': 'fail'}
if res.ok:
repl_data = {}
for line in res.stdout.split('\n'):
if ':' in line:
k, v = line.split(':', 1)
repl_data[k.strip()] = v.strip()
role = repl_data.get('role', 'unknown')
redis_status['role'] = role
redis_status['replication'] = repl_data
if role == 'master':
# 主节点：检查连接的从节点数
connected_slaves = repl_data.get('connected_slaves', '0')
repl_item['detail'] = f'主节点，从节点数: {connected_slaves}'
if int(connected_slaves) >= 2:
repl_item['status'] = 'pass'
else:
repl_item['status'] = 'warning'
repl_item['detail'] += '（预期 >= 2）'
elif role == 'slave':
# 从节点：检查同步链路状态
master_link = repl_data.get('master_link_status', 'unknown')
repl_item['detail'] = f'从节点，主连接状态: {master_link}'
if master_link == 'up':
repl_item['status'] = 'pass'
# 检查同步延迟（IO秒数）
io_seconds = repl_data.get('master_last_io_seconds_ago', 'N/A')
try:
io_val = int(io_seconds)
if io_val > 60:
repl_item['status'] = 'warning'
repl_item['detail'] += f'，⚠️ 主从通信间隔: {io_val}s（>60s）'
elif io_val > 30:
repl_item['status'] = 'warning'
repl_item['detail'] += f'，⚠️ 主从通信间隔: {io_val}s（>30s）'
else:
repl_item['detail'] += f'，通信间隔: {io_val}s'
except ValueError:
pass
# 记录同步偏移量（供跨节点对比）
repl_item['slave_repl_offset'] = repl_data.get('slave_repl_offset', 'N/A')
else:
repl_item['status'] = 'error'
repl_item['detail'] = f'❌ 主从连接断开: {master_link}'
else:
repl_item['status'] = 'warning'
repl_item['detail'] = f'未知角色: {role}'
else:
repl_item['detail'] = '主从复制状态查询失败'
repl_item['output'] = (res.stdout + res.stderr).strip()
redis_status['items'].append(repl_item)
# 5) 持久化状态检查（v2.1新增）
cmd = f"redis-cli -a '{password}' --no-auth-warning INFO persistence"
res = self.ssh.docker_exec(redis_container, cmd, timeout=15)
persist_item = {'name': '持久化状态', 'status': 'fail'}
if res.ok:
persist_data = {}
for line in res.stdout.split('\n'):
if ':' in line:
k, v = line.split(':', 1)
persist_data[k.strip()] = v.strip()
rdb_status = persist_data.get('rdb_last_bgsave_status', 'unknown')
aof_status = persist_data.get('aof_last_write_status', 'unknown')
rdb_last = persist_data.get('rdb_last_save_time', 'N/A')
issues = []
if rdb_status != 'ok':
issues.append(f'RDB保存异常: {rdb_status}')
if aof_status and aof_status != 'ok':
issues.append(f'AOF写入异常: {aof_status}')
if not issues:
persist_item['status'] = 'pass'
persist_item['detail'] = f'RDB: {rdb_status}, AOF: {aof_status}, 最近保存: {rdb_last}'
else:
persist_item['status'] = 'error'
persist_item['detail'] = '; '.join(issues)
redis_status['persistence'] = persist_data
else:
persist_item['detail'] = '持久化状态查询失败'
persist_item['output'] = (res.stdout + res.stderr).strip()
redis_status['items'].append(persist_item)
# 6) 哨兵检查（如果本节点有哨兵）
sentinel_container = None
for c in self.expected_containers:
if 'sentinel' in c:
......@@ -457,13 +556,18 @@ class ServerChecker:
master_info['port'] = lines[i + 1].strip()
elif line.strip() == 'status' and i + 1 < len(lines):
master_info['status'] = lines[i + 1].strip()
elif line.strip() == 'flags' and i + 1 < len(lines):
master_info['flags'] = lines[i + 1].strip()
elif line.strip() == 'num-other-sentinels' and i + 1 < len(lines):
master_info['sentinels'] = lines[i + 1].strip()
elif line.strip() == 'quorum' and i + 1 < len(lines):
master_info['quorum'] = lines[i + 1].strip()
if master_info:
sentinel_item['detail'] = (
f"主节点: {master_info.get('ip', 'N/A')}:{master_info.get('port', 'N/A')}, "
f"状态: {master_info.get('status', 'N/A')}, "
f"哨兵数: {master_info.get('sentinels', 'N/A')}"
f"状态: {master_info.get('flags', 'N/A')}, "
f"哨兵数: {master_info.get('sentinels', 'N/A')}, "
f"quorum: {master_info.get('quorum', 'N/A')}"
)
redis_status['sentinel'] = master_info
else:
......@@ -484,6 +588,10 @@ class ServerChecker:
检查EMQX集群状态
检查集群节点互联状态
增强项（v2.1）:
- MQTT消息端到端连通性验证（发布/订阅）
- Dashboard可访问性检查
Returns:
dict: EMQX集群状态
"""
......@@ -498,7 +606,7 @@ class ServerChecker:
cmd = "emqx_ctl status"
res = self.ssh.docker_exec('uemqx', cmd, timeout=15)
status_item = {'name': 'EMQX状态', 'status': 'fail'}
if res.ok and 'running' in res.stdout.lower():
if res.ok and ('running' in res.stdout.lower() or 'is started' in res.stdout.lower()):
status_item['status'] = 'pass'
status_item['detail'] = res.stdout.strip()
else:
......@@ -513,6 +621,9 @@ class ServerChecker:
if res.ok:
cluster_item['status'] = 'pass'
cluster_item['detail'] = res.stdout.strip()
# 提取集群节点数量用于分析
node_lines = res.stdout.strip().split('\n')
cluster_item['node_count'] = len([l for l in node_lines if l.strip()])
else:
cluster_item['detail'] = '集群状态查询失败'
cluster_item['output'] = (res.stdout + res.stderr).strip()
......@@ -530,8 +641,93 @@ class ServerChecker:
listener_item['output'] = (res.stdout + res.stderr).strip()
emqx_status['items'].append(listener_item)
all_pass = all(item['status'] == 'pass' for item in emqx_status['items'])
emqx_status['status'] = 'normal' if all_pass else 'error'
# 4) MQTT消息端到端连通性验证（v2.1新增）
# 注意：容器内可能没有mosquitto客户端，改用emqx_ctl检查
import time
test_topic = f"cluster/health/check/{self.server_ip.replace('.', '_')}"
test_msg = f"ping_{int(time.time())}"
mqtt_item = {'name': 'MQTT消息连通性', 'status': 'skip'}
# 先检查容器内是否有mosquitto_sub工具
check_cmd = "which mosquitto_sub 2>/dev/null || echo 'not_found'"
check_res = self.ssh.docker_exec('uemqx', check_cmd, timeout=10)
if check_res.ok and 'not_found' not in check_res.stdout:
# 容器内有工具，执行pub/sub测试
mqtt_cmd = (
f"(timeout 8 mosquitto_sub -h 127.0.0.1 -p 1883 "
f"-t '{test_topic}' -C 1 -W 5 2>/dev/null &); "
f"sleep 1; "
f"mosquitto_pub -h 127.0.0.1 -p 1883 "
f"-t '{test_topic}' -m '{test_msg}' -q 1 2>/dev/null; "
f"wait 2>/dev/null"
)
res = self.ssh.docker_exec('uemqx', mqtt_cmd, timeout=20)
if res.ok and test_msg in res.stdout:
mqtt_item['status'] = 'pass'
mqtt_item['detail'] = f'消息发布/订阅成功 (topic: {test_topic})'
else:
mqtt_item['status'] = 'warning'
mqtt_item['detail'] = 'MQTT消息连通性测试失败'
mqtt_item['output'] = (res.stdout + res.stderr).strip()[:200]
else:
# 容器内无mosquitto工具，跳过此测试
mqtt_item['detail'] = '容器内无mosquitto客户端，跳过（不影响整体判定）'
emqx_status['items'].append(mqtt_item)
# 5) Dashboard可访问性（v2.1新增）
cmd = "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:18083/ 2>/dev/null"
res = self.ssh.run(cmd, timeout=15)
dashboard_item = {'name': 'Dashboard', 'status': 'fail'}
code = res.stdout.strip()
if res.ok and code in ('200', '302', '301'):
dashboard_item['status'] = 'pass'
dashboard_item['detail'] = f'HTTP {code}'
else:
dashboard_item['detail'] = f'Dashboard不可达 (HTTP {code})'
emqx_status['items'].append(dashboard_item)
# 6) 连接数和订阅数（通过 Dashboard API 获取）
import json as json_module
cmd = "curl -s http://127.0.0.1:18083/api/v5/stats 2>/dev/null"
res = self.ssh.run(cmd, timeout=15)
stats_item = {'name': '连接统计', 'status': 'fail'}
if res.ok and res.stdout.strip():
try:
stats = json_module.loads(res.stdout.strip())
# EMQX v5 stats API: 返回 connections.count, subscriptions.count 等
subs_data = stats.get('subscriptions', {})
conns_data = stats.get('connections', {})
subs_count = subs_data.get('count') if isinstance(subs_data, dict) else subs_data
conns_count = conns_data.get('count') if isinstance(conns_data, dict) else conns_data
stats_item['status'] = 'pass'
stats_item['detail'] = f'连接数: {conns_count}, 订阅数: {subs_count}'
emqx_status['connections'] = str(conns_count) if conns_count is not None else 'N/A'
emqx_status['subscriptions'] = str(subs_count) if subs_count is not None else 'N/A'
except (json_module.JSONDecodeError, ValueError, KeyError) as e:
stats_item['detail'] = f'API数据解析失败: {e}'
stats_item['output'] = res.stdout.strip()[:200]
else:
stats_item['detail'] = 'Dashboard API不可达'
stats_item['output'] = (res.stdout + res.stderr).strip()[:200]
emqx_status['items'].append(stats_item)
# 整体状态判定：只检查非skip的项
active_items = [item for item in emqx_status['items'] if item.get('status') != 'skip']
if active_items:
all_pass = all(item['status'] == 'pass' for item in active_items)
has_error = any(item['status'] == 'fail' for item in active_items)
if has_error:
emqx_status['status'] = 'error'
elif all_pass:
emqx_status['status'] = 'normal'
else:
emqx_status['status'] = 'warning'
else:
emqx_status['status'] = 'skip'
return emqx_status
......@@ -543,6 +739,10 @@ class ServerChecker:
注意：已知问题 - 节点1控制台登录失败（部署改密破坏认证链），业务面正常
已知问题豁免：跳过Dashboard登录检查
增强项（v2.1）:
- 提取 leader 信息验证集群角色
- 提取服务列表用于跨节点一致性验证
Returns:
dict: Nacos集群状态
"""
......@@ -551,6 +751,7 @@ class ServerChecker:
logger.info('开始检查Nacos集群...')
import json as json_module
nacos_status = {'status': 'unknown', 'items': []}
# 1) Web探活（跳过Dashboard登录，因为已知问题）
......@@ -572,24 +773,77 @@ class ServerChecker:
state_item = {'name': '集群状态', 'status': 'fail'}
if res.ok and res.stdout.strip():
state_item['status'] = 'pass'
state_item['detail'] = res.stdout.strip()[:200] # 截取前200字符
state_item['detail'] = res.stdout.strip()[:200]
# 尝试解析JSON提取关键信息
try:
state_data = json_module.loads(res.stdout.strip())
if isinstance(state_data, dict):
node_state = state_data.get('standalone_mode', state_data.get('mode', 'unknown'))
nacos_status['nacos_cluster_state'] = state_data
# 记录是否为leader
if 'leader' in str(state_data).lower():
for key in ['leader', 'LEADER', 'master']:
if key in str(state_data):
nacos_status['nacos_leader'] = str(state_data).split(key)[-1][:50]
except (json_module.JSONDecodeError, ValueError):
pass
else:
state_item['detail'] = '状态查询失败'
state_item['output'] = (res.stdout + res.stderr).strip()
nacos_status['items'].append(state_item)
# 3) 注册服务数
# 3) 注册服务列表（用于跨节点一致性对比）
cmd = "curl -s http://127.0.0.1:8848/nacos/v1/ns/service/list"
res = self.ssh.run(cmd, timeout=15)
svc_item = {'name': '注册服务数', 'status': 'fail'}
if res.ok and res.stdout.strip():
svc_list = res.stdout.strip()
# 解析服务列表数量
svc_count = 'N/A'
try:
svc_data = json_module.loads(svc_list)
if isinstance(svc_data, dict):
# Nacos API 返回格式: {"count": N, "doms": [...]}
svc_count = str(svc_data.get('count', len(svc_data.get('doms', []))))
elif isinstance(svc_data, list):
svc_count = str(len(svc_data))
except (json_module.JSONDecodeError, ValueError):
pass
svc_item['status'] = 'pass'
svc_item['detail'] = f'服务列表: {res.stdout.strip()[:100]}'
svc_item['detail'] = f'注册服务数: {svc_count}'
nacos_status['service_list'] = svc_list # 保存原始json用于跨节点对比
nacos_status['service_count'] = svc_count
else:
svc_item['detail'] = '服务查询失败'
svc_item['output'] = (res.stdout + res.stderr).strip()
nacos_status['items'].append(svc_item)
# 4) 配置中心探活（v2.1新增）
cmd = "curl -s http://127.0.0.1:8848/nacos/v1/cs/configs?pageNo=1&pageSize=1"
res = self.ssh.run(cmd, timeout=15)
config_item = {'name': '配置中心', 'status': 'fail'}
if res.ok and res.stdout.strip():
# 尝试解析配置总数
config_count = 'N/A'
try:
import json as json_module2
config_data = json_module2.loads(res.stdout.strip())
if isinstance(config_data, dict):
config_count = str(config_data.get('totalCount', 'N/A'))
except (json_module2.JSONDecodeError, ValueError):
# Nacos v1 可能返回不同格式
pass
config_item['status'] = 'pass'
config_item['detail'] = f'配置中心可用，配置数: {config_count}'
nacos_status['config_count'] = config_count
else:
config_item['detail'] = f'配置中心不可用'
nacos_status['items'].append(config_item)
all_pass = all(item['status'] == 'pass' for item in nacos_status['items'])
nacos_status['status'] = 'normal' if all_pass else 'error'
......@@ -600,32 +854,25 @@ class ServerChecker:
def check_fastdfs(self) -> Dict:
"""
检查FastDFS集群状态
utracker 容器内同时运行 tracker 和 storage 进程（3节点集群）
通过 fdfs_monitor 获取集群详细信息
utracker 容器运行 Tracker, ustorage 容器运行 Storage（三节点集群）
Returns:
dict: FastDFS状态，含Tracker/Storage/集群信息
"""
# 检查是否有utracker容器（节点1-3都有，节点4没有）
has_tracker = any('utracker' in c or 'tracker' in c for c in self.expected_containers)
has_tracker = any('utracker' in c or c == 'utracker' for c in self.expected_containers)
has_storage = any('ustorage' in c or c == 'ustorage' for c in self.expected_containers)
if not has_tracker:
if not has_tracker and not has_storage:
return {'status': 'skip', 'message': '本节点未部署FastDFS'}
logger.info('开始检查FastDFS...')
fastdfs_status = {'status': 'unknown', 'items': []}
# 找到utracker容器
tracker_container = 'utracker'
for c in self.expected_containers:
if 'utracker' in c or 'tracker' in c:
tracker_container = c
break
# 1) Tracker进程检查
if has_tracker:
cmd = "ps aux | grep fdfs_trackerd | grep -v grep"
res = self.ssh.docker_exec(tracker_container, cmd, timeout=15)
res = self.ssh.docker_exec('utracker', cmd, timeout=15)
tracker_item = {'name': 'Tracker进程', 'status': 'fail'}
if res.ok and res.stdout.strip():
tracker_item['status'] = 'pass'
......@@ -636,31 +883,32 @@ class ServerChecker:
fastdfs_status['items'].append(tracker_item)
# 2) Storage进程检查
cmd = "ps aux | grep fdfs_storage | grep -v grep"
res = self.ssh.docker_exec(tracker_container, cmd, timeout=15)
if has_storage:
cmd = "ps aux | grep fdfs_storaged | grep -v grep"
res = self.ssh.docker_exec('ustorage', cmd, timeout=15)
storage_item = {'name': 'Storage进程', 'status': 'fail'}
if res.ok and res.stdout.strip():
storage_item['status'] = 'pass'
storage_item['detail'] = 'fdfs_storage 运行中'
storage_item['detail'] = 'fdfs_storaged 运行中'
else:
storage_item['detail'] = 'Storage进程未运行'
storage_item['output'] = (res.stdout + res.stderr).strip()
fastdfs_status['items'].append(storage_item)
# 3) fdfs_monitor 集群状态（获取storage节点数、磁盘空间、各节点状态）
# 3) fdfs_monitor 集群状态
if has_tracker:
cmd = "fdfs_monitor /etc/fdfs/client.conf 2>/dev/null"
res = self.ssh.docker_exec(tracker_container, cmd, timeout=20)
res = self.ssh.docker_exec('utracker', cmd, timeout=20)
cluster_item = {'name': '集群状态', 'status': 'fail'}
if res.ok and res.stdout.strip():
cluster_item['status'] = 'pass'
# 解析关键信息
output = res.stdout
# 提取storage节点数
active_count = "N/A"
total_space = "N/A"
free_space = "N/A"
active_count = "N/A"; total_space = "N/A"; free_space = "N/A"
storage_status = []
for line in output.split('\n'):
sync_details = [] # v2.1: 存储同步详情
lines = output.split('\n')
for i, line in enumerate(lines):
if 'active server count' in line:
active_count = line.split('=')[-1].strip()
elif 'disk total space' in line:
......@@ -669,6 +917,11 @@ class ServerChecker:
free_space = line.split('=')[-1].strip()
elif 'ACTIVE' in line or 'OFFLINE' in line:
storage_status.append(line.strip())
# v2.1: 提取同步状态
if i + 1 < len(lines):
next_lines = '\n'.join(lines[i:i+5])
if 'sync' in next_lines.lower() or 'heart' in next_lines.lower():
sync_details.append(next_lines)
cluster_item['detail'] = (
f"Storage节点: {active_count}个活跃, "
......@@ -676,14 +929,69 @@ class ServerChecker:
f"剩余: {free_space}MB"
)
cluster_item['storage_nodes'] = storage_status
cluster_item['sync_details'] = sync_details[:3] if sync_details else [] # 只保留前3个
fastdfs_status['cluster_info'] = cluster_item
fastdfs_status['raw_monitor'] = output[:1000] # 保存部分原始输出用于跨节点对比
# 保存存储容量供日报使用
fastdfs_status['total_space'] = total_space if total_space != 'N/A' else None
fastdfs_status['free_space'] = free_space if free_space != 'N/A' else None
fastdfs_status['active_storage'] = active_count if active_count != 'N/A' else None
else:
cluster_item['detail'] = 'fdfs_monitor 查询失败'
cluster_item['output'] = (res.stdout + res.stderr).strip()
fastdfs_status['items'].append(cluster_item)
all_pass = all(item['status'] == 'pass' for item in fastdfs_status['items'])
fastdfs_status['status'] = 'normal' if all_pass else 'error'
# 4) 文件上传端到端验证（v2.1新增，仅Tracker节点）
if has_tracker:
import time
test_file = f"/tmp/fdfs_health_{int(time.time())}.txt"
# 创建测试文件
upload_test = {'name': '文件上传测试', 'status': 'skip'}
# 先检查容器内是否有fdfs_upload_file工具
check_cmd = "which fdfs_upload_file 2>/dev/null || echo 'not_found'"
check_res = self.ssh.docker_exec('utracker', check_cmd, timeout=10)
if check_res.ok and 'not_found' not in check_res.stdout:
# 容器内有工具，执行上传测试
upload_cmd = (
f"echo 'health_check_{int(time.time())}' > {test_file} && "
f"fdfs_upload_file /etc/fdfs/client.conf {test_file} 2>&1"
)
res = self.ssh.docker_exec('utracker', upload_cmd, timeout=20)
if res.ok:
# 检查返回的是否为文件路径格式 (group1/M00/...)
result = res.stdout.strip()
if 'group' in result and '/' in result:
upload_test['status'] = 'pass'
upload_test['detail'] = f'上传成功: {result}'
upload_test['file_id'] = result
else:
upload_test['status'] = 'warning'
upload_test['detail'] = f'上传返回异常: {result[:100]}'
else:
upload_test['detail'] = '上传测试失败'
upload_test['output'] = (res.stdout + res.stderr).strip()[:200]
else:
upload_test['detail'] = '容器内无fdfs_upload_file工具，跳过（不影响整体判定）'
fastdfs_status['items'].append(upload_test)
# 整体状态判定：只检查非skip的项
active_items = [item for item in fastdfs_status['items'] if item['status'] != 'skip']
if active_items:
all_pass = all(item['status'] == 'pass' for item in active_items)
has_error = any(item['status'] == 'fail' for item in active_items)
if has_error:
fastdfs_status['status'] = 'error'
elif all_pass:
fastdfs_status['status'] = 'normal'
else:
fastdfs_status['status'] = 'warning'
else:
fastdfs_status['status'] = 'skip'
return fastdfs_status
......@@ -693,6 +1001,11 @@ class ServerChecker:
"""
检查达梦数据库状态（节点4专用）
增强项（v2.1）:
- 实际SQL连接测试
- 表空间使用率检查
- 许可证到期检查
Returns:
dict: 达梦数据库状态
"""
......@@ -701,6 +1014,12 @@ class ServerChecker:
logger.info('开始检查达梦数据库...')
# 从配置获取达梦凭据
dm8_creds = self.cluster_config.get('middleware_credentials', {}).get('dm8', {})
dm8_user = dm8_creds.get('username', 'SYSDBA')
dm8_pass = dm8_creds.get('password', 'Dameng2026Pwd')
dm8_port = dm8_creds.get('port', 5236)
dm8_status = {'status': 'unknown', 'items': []}
# 检查进程
......@@ -727,6 +1046,101 @@ class ServerChecker:
port_item['output'] = (res.stdout + res.stderr).strip()
dm8_status['items'].append(port_item)
# 连接数查询（新增）
conn_cmd = (
f"/opt/dmdbms/bin/disql {dm8_user}/{dm8_pass}@127.0.0.1:{dm8_port} -S "
f"\"SELECT COUNT(*) AS CONN_COUNT FROM V\\$SESSIONS WHERE STATE='ACTIVE';\""
)
res = self.ssh.run(conn_cmd, timeout=15)
conn_item = {'name': '连接数', 'status': 'pass'}
conn_count = 'N/A'
if res.ok and res.stdout.strip():
import re
# 提取数字
num_match = re.search(r'(\d+)', res.stdout)
if num_match:
conn_count = num_match.group(1)
conn_item['detail'] = f'当前活动连接: {conn_count}'
dm8_status['connections'] = conn_count
else:
conn_item['detail'] = '连接数解析失败'
else:
conn_item['detail'] = '连接数查询失败'
dm8_status['items'].append(conn_item)
# SQL连接测试（v2.1新增）
sql_cmd = (
f"/opt/dmdbms/bin/disql {dm8_user}/{dm8_pass}@127.0.0.1:{dm8_port} "
f"<<EOF\nSELECT 1 AS test FROM DUAL;\nSELECT COUNT(*) AS table_count FROM ALL_TABLES;\nEXIT;\nEOF"
)
res = self.ssh.run(sql_cmd, timeout=20)
conn_item = {'name': 'SQL连接测试', 'status': 'fail'}
if res.ok and 'test' in res.stdout.lower() or '1' in res.stdout:
conn_item['status'] = 'pass'
conn_item['detail'] = '数据库连接成功'
# 尝试提取表数量
import re
match = re.search(r'table_count.*?(\d+)', res.stdout, re.IGNORECASE | re.DOTALL)
if match:
conn_item['detail'] += f"，表数量: {match.group(1)}"
else:
conn_item['detail'] = 'SQL连接失败'
conn_item['output'] = (res.stdout + res.stderr).strip()[:300]
dm8_status['items'].append(conn_item)
# 表空间使用率检查（v2.1新增）
ts_cmd = (
f"/opt/dmdbms/bin/disql {dm8_user}/{dm8_pass}@127.0.0.1:{dm8_port} -S "
f"\"SELECT TABLESPACE_NAME, USED_PERCENT FROM DBA_TABLESPACES WHERE USED_PERCENT > 80;\""
)
res = self.ssh.run(ts_cmd, timeout=20)
ts_item = {'name': '表空间使用率', 'status': 'pass'}
if res.ok and res.stdout.strip():
# 检查是否有超80%的表空间
lines = [l for l in res.stdout.split('\n') if l.strip() and 'rows' not in l.lower()]
if lines:
ts_item['status'] = 'warning'
ts_item['detail'] = f"有 {len(lines)} 个表空间使用率超80%"
ts_item['spaces'] = lines[:5] # 只记录前5个
else:
ts_item['detail'] = '所有表空间使用率正常'
else:
ts_item['detail'] = '表空间查询跳过'
dm8_status['items'].append(ts_item)
# 许可证到期检查（v2.1新增）
lic_cmd = f"/opt/dmdbms/bin/disql {dm8_user}/{dm8_pass}@127.0.0.1:{dm8_port} -S \"SELECT EXPIRED_DATE FROM V\\$LICENSE;\""
res = self.ssh.run(lic_cmd, timeout=20)
lic_item = {'name': '许可证到期', 'status': 'pass'}
if res.ok and res.stdout.strip():
import re
# 提取日期
date_match = re.search(r'(\d{4}-\d{2}-\d{2})', res.stdout)
if date_match:
exp_date = date_match.group(1)
from datetime import datetime
try:
exp_dt = datetime.strptime(exp_date, '%Y-%m-%d')
days_left = (exp_dt - datetime.now()).days
lic_item['expired_date'] = exp_date
lic_item['days_left'] = days_left
if days_left < 0:
lic_item['status'] = 'error'
lic_item['detail'] = f'❌ 许可证已过期 ({exp_date})'
elif days_left < 30:
lic_item['status'] = 'warning'
lic_item['detail'] = f'⚠️ 许可证将于 {exp_date} 到期（剩余 {days_left} 天）'
else:
lic_item['detail'] = f'许可证到期: {exp_date}（剩余 {days_left} 天）'
except ValueError:
lic_item['detail'] = f'许可证日期: {exp_date}'
else:
lic_item['detail'] = '许可证日期解析失败'
else:
lic_item['detail'] = '许可证查询跳过'
dm8_status['items'].append(lic_item)
all_pass = all(item['status'] == 'pass' for item in dm8_status['items'])
dm8_status['status'] = 'normal' if all_pass else 'error'
......@@ -856,6 +1270,116 @@ class ServerChecker:
return interfaces
# ==================== 跨节点网络连通性检查 ====================
def check_cross_node_connectivity(self) -> Dict:
"""
检查本节点到其他节点的网络连通性（v2.1新增）
包括: ping、SSH端口、关键服务端口
Returns:
dict: 跨节点连通性状态
"""
logger.info('开始检查跨节点网络连通性...')
# 从集群配置获取其他节点IP列表
all_servers = self.cluster_config.get('servers', [])
other_nodes = [s for s in all_servers if s['ip'] != self.server_ip]
if not other_nodes:
return {'status': 'skip', 'message': '未配置其他节点'}
connectivity = {'status': 'unknown', 'nodes': [], 'items': []}
for node in other_nodes:
node_ip = node['ip']
node_role = node.get('role', 'unknown')
node_result = {'ip': node_ip, 'role': node_role, 'checks': []}
# 1) ping测试
cmd = f"ping -c 3 -W 2 {node_ip} 2>/dev/null | tail -1"
res = self.ssh.run(cmd, timeout=15)
ping_item = {'name': 'ping', 'status': 'fail'}
if res.ok and ('0% packet loss' in res.stdout or 'bytes from' in res.stdout):
ping_item['status'] = 'pass'
ping_item['detail'] = res.stdout.strip()
else:
ping_item['detail'] = 'ping失败'
ping_item['output'] = (res.stdout + res.stderr).strip()[:100]
node_result['checks'].append(ping_item)
# 2) SSH端口(22)连通性
cmd = f"nc -z -w 3 {node_ip} 22 2>/dev/null && echo 'open' || echo 'closed'"
res = self.ssh.run(cmd, timeout=10)
ssh_item = {'name': 'SSH端口22', 'status': 'fail'}
if res.ok and 'open' in res.stdout:
ssh_item['status'] = 'pass'
ssh_item['detail'] = '端口可达'
else:
ssh_item['detail'] = 'SSH端口不可达'
node_result['checks'].append(ssh_item)
# 3) 关键服务端口检查（根据目标节点角色）
# Redis端口6379
if 'redis' in str(node.get('containers', [])).lower():
cmd = f"nc -z -w 3 {node_ip} 6379 2>/dev/null && echo 'open' || echo 'closed'"
res = self.ssh.run(cmd, timeout=10)
redis_item = {'name': 'Redis端口6379', 'status': 'fail'}
if res.ok and 'open' in res.stdout:
redis_item['status'] = 'pass'
redis_item['detail'] = '端口可达'
else:
redis_item['detail'] = 'Redis端口不可达'
node_result['checks'].append(redis_item)
# EMQX端口1883
if 'uemqx' in str(node.get('containers', [])).lower():
cmd = f"nc -z -w 3 {node_ip} 1883 2>/dev/null && echo 'open' || echo 'closed'"
res = self.ssh.run(cmd, timeout=10)
emqx_item = {'name': 'EMQX端口1883', 'status': 'fail'}
if res.ok and 'open' in res.stdout:
emqx_item['status'] = 'pass'
emqx_item['detail'] = '端口可达'
else:
emqx_item['detail'] = 'EMQX端口不可达'
node_result['checks'].append(emqx_item)
# Nacos端口8848
if 'unacos' in str(node.get('containers', [])).lower():
cmd = f"nc -z -w 3 {node_ip} 8848 2>/dev/null && echo 'open' || echo 'closed'"
res = self.ssh.run(cmd, timeout=10)
nacos_item = {'name': 'Nacos端口8848', 'status': 'fail'}
if res.ok and 'open' in res.stdout:
nacos_item['status'] = 'pass'
nacos_item['detail'] = '端口可达'
else:
nacos_item['detail'] = 'Nacos端口不可达'
node_result['checks'].append(nacos_item)
# FastDFS Tracker端口22122
if 'utracker' in str(node.get('containers', [])).lower():
cmd = f"nc -z -w 3 {node_ip} 22122 2>/dev/null && echo 'open' || echo 'closed'"
res = self.ssh.run(cmd, timeout=10)
tracker_item = {'name': 'Tracker端口22122', 'status': 'fail'}
if res.ok and 'open' in res.stdout:
tracker_item['status'] = 'pass'
tracker_item['detail'] = '端口可达'
else:
tracker_item['detail'] = 'Tracker端口不可达'
node_result['checks'].append(tracker_item)
# 统计该节点连通性
all_pass = all(c['status'] == 'pass' for c in node_result['checks'])
node_result['status'] = 'pass' if all_pass else 'fail'
connectivity['nodes'].append(node_result)
connectivity['items'].extend(node_result['checks'])
# 整体状态
all_nodes_ok = all(n['status'] == 'pass' for n in connectivity['nodes'])
connectivity['status'] = 'normal' if all_nodes_ok else 'error'
return connectivity
# ==================== 完整检查 ====================
def run_full_check(self) -> Dict:
......@@ -879,7 +1403,8 @@ class ServerChecker:
'fastdfs': self.check_fastdfs(),
'dm8': self.check_dm8(),
'java_services': self.check_java_services(),
'business_interfaces': self.check_business_interfaces()
'business_interfaces': self.check_business_interfaces(),
'cross_node_connectivity': self.check_cross_node_connectivity()
}
logger.info(f'服务器检查完成: {self.server_ip}')
......
.claude/skills/X86-CLUSTER-MONITOR/config.json
浏览文件 @ 6eb7badf
......@@ -22,12 +22,34 @@
"cpu_cores": 8,
"memory_gb": 16,
"disk_gb": 80,
"containers": ["ujava2", "utengine", "umysql", "redis-master", "redis-sentinel", "uemqx", "unacos", "utracker", "paperless"],
"containers": [
"ujava2",
"utengine",
"umysql",
"redis-master",
"redis-sentinel",
"uemqx",
"unacos",
"utracker",
"ustorage",
"paperless"
],
"services": {
"mysql": {"port": 8306, "type": "master"},
"redis": {"port": 6379, "type": "master"},
"emqx": {"port": 1883, "dashboard_port": 18083},
"nacos": {"port": 8848}
"mysql": {
"port": 8306,
"type": "master"
},
"redis": {
"port": 6379,
"type": "master"
},
"emqx": {
"port": 1883,
"dashboard_port": 18083
},
"nacos": {
"port": 8848
}
},
"note": "主节点，承担核心业务"
},
......@@ -43,9 +65,23 @@
"cpu_cores": 8,
"memory_gb": 16,
"disk_gb": 80,
"containers": ["ujava2", "utengine", "umysql", "redis-slave-1", "redis-sentinel-2", "uemqx", "unacos", "utracker", "paperless"],
"containers": [
"ujava2",
"utengine",
"umysql",
"redis-slave-1",
"redis-sentinel-2",
"uemqx",
"unacos",
"utracker",
"ustorage",
"paperless"
],
"services": {
"redis": {"port": 6379, "type": "slave"}
"redis": {
"port": 6379,
"type": "slave"
}
},
"note": "从节点1"
},
......@@ -61,9 +97,23 @@
"cpu_cores": 8,
"memory_gb": 16,
"disk_gb": 80,
"containers": ["ujava2", "utengine", "umysql", "redis-slave-2", "redis-sentinel-3", "uemqx", "unacos", "utracker", "paperless"],
"containers": [
"ujava2",
"utengine",
"umysql",
"redis-slave-2",
"redis-sentinel-3",
"uemqx",
"unacos",
"utracker",
"ustorage",
"paperless"
],
"services": {
"redis": {"port": 6379, "type": "slave"}
"redis": {
"port": 6379,
"type": "slave"
}
},
"note": "从节点2"
},
......@@ -79,9 +129,14 @@
"cpu_cores": 8,
"memory_gb": 8,
"disk_gb": 60,
"containers": ["dm8-server", "utengine"],
"containers": [
"dm8-server",
"utengine"
],
"services": {
"dm8": {"port": 5236}
"dm8": {
"port": 5236
}
},
"note": "达梦数据库存储使用"
}
......@@ -90,25 +145,48 @@
"redis_cluster": {
"mode": "1主2从3哨兵",
"master": "192.168.5.41",
"slaves": ["192.168.5.42", "192.168.5.43"],
"sentinels": ["192.168.5.41", "192.168.5.42", "192.168.5.43"],
"slaves": [
"192.168.5.42",
"192.168.5.43"
],
"sentinels": [
"192.168.5.41",
"192.168.5.42",
"192.168.5.43"
],
"password": "Ubains@123"
},
"emqx_cluster": {
"mode": "3节点集群",
"nodes": ["192.168.5.41", "192.168.5.42", "192.168.5.43"],
"nodes": [
"192.168.5.41",
"192.168.5.42",
"192.168.5.43"
],
"dashboard_username": "admin",
"dashboard_password": "public"
},
"nacos_cluster": {
"mode": "3节点集群",
"nodes": ["192.168.5.41", "192.168.5.42", "192.168.5.43"],
"nodes": [
"192.168.5.41",
"192.168.5.42",
"192.168.5.43"
],
"console_username": "nacos",
"console_password": "nacos"
},
"fastdfs_cluster": {
"tracker_nodes": ["192.168.5.41", "192.168.5.42", "192.168.5.43"],
"storage_nodes": ["192.168.5.41", "192.168.5.42", "192.168.5.43"]
"tracker_nodes": [
"192.168.5.41",
"192.168.5.42",
"192.168.5.43"
],
"storage_nodes": [
"192.168.5.41",
"192.168.5.42",
"192.168.5.43"
]
}
},
"middleware_credentials": {
......@@ -130,6 +208,12 @@
"username": "nacos",
"password": "nacos",
"port": 8848
},
"dm8": {
"username": "SYSDBA",
"password": "Dameng2026Pwd",
"port": 5236,
"expired_date": "2027-04-14"
}
},
"thresholds": {
......@@ -149,11 +233,23 @@
"window": "22:00-09:00",
"interval": "2h",
"cron": "0 22,0,2,4,6,8 * * *",
"execution_times": ["22:00", "00:00", "02:00", "04:00", "06:00", "08:00"]
"execution_times": [
"22:00",
"00:00",
"02:00",
"04:00",
"06:00",
"08:00"
]
},
"report": {
"output_dir": "AuxiliaryTool/ScriptTool/ClusterMonitor/reports",
"filename_format": "cluster_monitor_{YYYYMMDD_HHMMSS}.md",
"history_days": 30
},
"dingtalk": {
"enabled": false,
"webhook": "https://oapi.dingtalk.com/robot/send?access_token=27071a77f20da381e9a321653ec5f4dcf668bcf058c01162f28e3f1f8633386d",
"secret": "SEC5d85d5735a1805ada1be84929d5b37f5b72a2a832a6bcd9a1ca5615e5799be38"
}
}
\ No newline at end of file
.claude/skills/X86-QLV10-XTYBS/SKILL.md
浏览文件 @ 6eb7badf
---
name: X86-QLV10-XTYBS
description: X86麒麟V10新统一平台远程自动化部署，严格按需求文档执行
description: X86麒麟V10新统一平台远程自动化部署，严格按需求文档执行，全程无交互
---
X86架构-麒麟V10服务器新统一平台自动化部署操作。
......@@ -9,6 +9,44 @@ X86架构-麒麟V10服务器新统一平台自动化部署操作。
/X86-QLV10-XTYBS
## 定时任务
**已设置工作日晚上 22:00 自动执行完整部署流程（全程无交互）**
- 定时任务ID: `84dbe6b5`
- 执行时间: 周一至周五 22:00
- 执行内容: 完整自动化流程（部署→授权→等待→验证→生成报告）
- 脚本路径: `.claude/skills/X86-QLV10-XTYBS/code/scheduled_deploy.py`
- 授权脚本: `AuxiliaryTool/ScriptTool/RemoteDeploy/authorize_x86_kylin.py`（Selenium浏览器自动化）
- 日志路径: `AuxiliaryTool/ScriptTool/RemoteDeploy/reports/scheduled_deploy_YYYYMMDD.log`
- 持久化: 已保存到 `.claude/scheduled_tasks.json`
**自动化流程说明**:
1. **部署阶段**: SSH上传部署包 → 解压 → 执行部署脚本（自动应答）
2. **授权阶段**: Selenium自动打开Chrome浏览器 → 登录维护平台 → 下载激活文件 → 上传license.zip → 重启服务
3. **等待阶段**: 等待10分钟让服务完全启动
4. **验证阶段**: curl接口验证 + 日志检查
5. **报告生成**: 输出部署分析报告
**手动触发方式**:
```bash
# 立即执行完整流程（跳过时间检查）
python .claude/skills/X86-QLV10-XTYBS/code/scheduled_deploy.py --force
# 仅部署，跳过授权阶段
python .claude/skills/X86-QLV10-XTYBS/code/scheduled_deploy.py --force --skip-auth
# 仅检查当前是否满足执行条件
python .claude/skills/X86-QLV10-XTYBS/code/scheduled_deploy.py --check
# 手动执行授权（单独测试）
python AuxiliaryTool/ScriptTool/RemoteDeploy/authorize_x86_kylin.py
```
**管理定时任务**:
- 查看所有定时任务: `/cron`
- 取消定时任务: 使用 `CronDelete` 工具删除 ID `84dbe6b5`
## 基本规则
- 所有回复、代码注释、Git提交信息默认使用中文（简体）描述；
......@@ -26,7 +64,7 @@ X86架构-麒麟V10服务器新统一平台自动化部署操作。
| 超管账号 | superadmin / Ubains@1357 |
| 验证码 | csba |
| 授权文件 | E:\自动化部署\X86-5.69\license.zip |
| 部署包源路径 | Z:\发布版本\03服务器部署\15新统一平台\X86部署包\全量版 |
| 部署包位置 | /data/offline_auto_unifiedPlatform.tar.gz（服务器上已存在，跳过上传） |
| 部署操作指导 | Docs/PRD/远程自动化部署/X86架构_新统一平台自动化部署操作指导.md |
| 需求文档 | Docs/PRD/远程自动化部署/_PRD_X86_麒麟V10远程自动化部署_需求文档.md |
| 部署脚本目录 | AuxiliaryTool/ScriptTool/RemoteDeploy/ |
......@@ -204,11 +242,21 @@ X86架构-麒麟V10服务器新统一平台自动化部署操作。
```
.claude/skills/X86-QLV10-XTYBS/code/
├── ssh_manager.py # SSH连接管理模块（paramiko）
├── deploy_runner.py # 部署执行脚本
├── service_checker.py # 服务状态检查脚本
├── ssh_helper.py # SSH连接管理模块（paramiko，密码已硬编码）
├── verify_x86_status.py # 服务验证脚本（容器+API+日志）
├── auto_deploy_wrapper.sh # 部署包装脚本（自动应答交互）
├── report_generator.py # 报告生成脚本
├── scheduled_deploy.py # 定时部署主控脚本（部署→授权→等待→验证全流程）
└── ssh_keys/ # SSH免密配置目录
└── 192.168.5.69/ # 按服务器IP命名
AuxiliaryTool/ScriptTool/RemoteDeploy/
├── full_deploy.py # 核心部署脚本（SSH上传+解压+执行+验证）
├── authorize_x86_kylin.py # X86麒麟V10自动授权脚本（Selenium浏览器自动化）
└── reports/ # 报告输出目录
```
**全程无交互确认说明**:
- SSH连接：密码硬编码于 `ssh_helper.py` (`Ubains@123`)
- 部署脚本：通过 `printf 'y\ny\ny\ny\ny\ny\ny\nn\n'` 自动应答
- 授权操作：Selenium自动化浏览器，账号密码验证码全部硬编码
- 接口验证：curl命令自动重试5次，无需人工判断
\ No newline at end of file
.claude/skills/X86-QLV10-XTYBS/code/ssh_helper.py
浏览文件 @ 6eb7badf
......@@ -10,6 +10,9 @@ import time
import os
import re
# 设置标准输出编码为UTF-8，避免GBK编码错误
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
# 服务器配置
HOST = "192.168.5.69"
PORT = 22
......@@ -46,10 +49,18 @@ def exec_long_cmd(client, cmd, timeout=3600):
line = stdout.readline()
if not line:
break
try:
line = line.strip()
if line:
# 尝试解码并打印，遇到编码错误跳过
try:
print(line)
except UnicodeEncodeError:
# 打印安全的ASCII表示
print(line.encode('ascii', errors='replace').decode('ascii'))
output_lines.append(line)
except Exception:
pass
exit_code = stdout.channel.recv_exit_status()
print(f"[EXIT CODE] {exit_code}")
......
.claude/skills/X86-TX-XTYBS/SKILL.md
浏览文件 @ 6eb7badf
......@@ -393,3 +393,95 @@ context = browser.new_context(ignore_https_errors=True)
2. 备选：找到全局所有file input，逐个尝试`set_input_files()`
3. 最后手段：用JS将隐藏的file input变为可见后再设置文件
4. 上传成功后检查后端日志确认：`tail -f /data/services/api/java-meeting/java-meeting-extapi/logs/ubains-INFO-AND-ERROR.log | grep "uploadLincenceFile"`
---
## 定时任务自动化部署（Scheduled Deploy）
> **核心原则：定时任务执行时无需任何人工交互，全程自动化！**
### 定时任务配置
| 项目 | 配置 |
|------|------|
| **执行时间** | 工作日（周一至周五）22:00 |
| **触发方式** | Claude Code CronJob（持久化到 `.claude/scheduled_tasks.json`） |
| **主控脚本** | `code/scheduled_deploy.py` |
| **超时时间** | 2小时（7200秒） |
### 自动化执行流程
```
22:00 定时触发
├─ [自动] 检查工作日+时间条件（非工作日跳过）
├─ [自动] 检查前置条件：
│ ├─ SSH免密配置 (~/.ssh/192.168.5.70/id_rsa)
│ ├─ 部署包 (Z:\发布版本\...\offline_auto_unifiedPlatform.tar.gz)
│ └─ 授权文件 (E:\自动化部署\X86-5.70\license.zip)
│
├─ [自动] 阶段1-3：部署+验收验证
│ ├─ 上传部署包到服务器
│ ├─ 解压并执行 new_auto.sh --all
│ └─ API接口验证（4个接口，各重试5次间隔30秒）
│
├─ [自动] 等待10分钟（服务完全启动）
│
├─ [自动] 阶段4：系统授权（Playwright无头模式）
│ ├─ 登录维护平台（superadmin）
│ ├─ 下载激活文件（绑定硬件指纹）
│ ├─ 上传授权文件（自动处理身份校验对话框）
│ └─ 重启服务（勾选运维系统+预定系统）
│
└─ [自动] 生成报告到 reports/X86_TX_5.70_定时部署报告_YYYYMMDD_HHMMSS.md
```
### 命令速查
```bash
# 强制立即执行（忽略时间检查）
python .claude/skills/X86-TX-XTYBS/code/scheduled_deploy.py --force
# 仅检查前置条件（不实际部署）
python .claude/skills/X86-TX-XTYBS/code/scheduled_deploy.py --check
# 查看定时任务状态
cat .claude/scheduled_tasks.json
```
### 日志和报告
| 类型 | 路径 |
|------|------|
| **执行日志** | `AuxiliaryTool/ScriptTool/RemoteDeploy/reports/scheduled_deploy_tx_YYYYMMDD.log` |
| **部署报告** | `AuxiliaryTool/ScriptTool/RemoteDeploy/reports/X86_TX_5.70_定时部署报告_YYYYMMDD_HHMMSS.md` |
| **授权截图** | `~/deploy_logs/screenshots/` |
### 定时任务处理原则（给Claude的指令）
> **当定时任务触发执行时，严格遵循以下原则，不要询问任何问题！**
1. ✅ **直接执行**：检测到定时任务触发 → 直接运行 `scheduled_deploy.py --force` → 等待完成 → 汇报结果
2. ✅ **不确认**：不询问"是否继续"、"是否上传"、"是否确认"
3. ✅ **不中断**：即使遇到非致命错误也继续执行下一阶段
4. ✅ **自动授权**：阶段4使用 Playwright `headless=True` 无头浏览器，不弹出窗口
5. ✅ **异常处理**：授权失败记录错误到日志，不影响整体报告生成
6. ❌ **禁止交互**：整个流程中不调用 `AskUserQuestion`、不等待用户输入
### 前置条件（部署前自动检查，无需人工确认）
这些条件在每次部署前由 `scheduled_deploy.py` 自动检查：
1. **SSH免密** - `~/.ssh/192.168.5.70/id_rsa`（已配置，永久有效）
2. **部署包** - `Z:\发布版本\03服务器部署\15新统一平台\X86部署包\全量版\offline_auto_unifiedPlatform.tar.gz`（8.67 GB）
3. **授权文件** - `E:\自动化部署\X86-5.70\license.zip`
如果前置条件不满足，定时任务会记录错误并跳过执行，不会阻塞。
### 修改记录
- **2026-07-11**: 创建定时任务自动化执行方案
- 创建 `code/scheduled_deploy.py` 主控脚本
- 重写 `code/phase4_authorize.py` 为完全无交互版本（headless模式）
- 设置 Claude Code CronJob：工作日 22:00
- 修复 GBK 编码乱码问题
- 明确"禁止交互"原则写入 SKILL.md
\ No newline at end of file
.claude/skills/X86-TX-XTYBS/code/full_deploy.py
浏览文件 @ 6eb7badf
......@@ -76,8 +76,12 @@ class X86Deployer:
self.ssh.close()
self.ssh = None
def phase1_pre_check_and_upload(self):
"""阶段1：部署前置检查与部署包上传"""
def phase1_pre_check_and_upload(self, skip_upload=False):
"""阶段1：部署前置检查与部署包上传
Args:
skip_upload: 跳过上传步骤，直接使用服务器上已有的部署包
"""
print("\n" + "=" * 60)
print("阶段1：部署前置检查与部署包上传")
print("=" * 60)
......@@ -108,6 +112,27 @@ class X86Deployer:
'output': stdout
})
if skip_upload:
print("\n[步骤2] 跳过清理和上传步骤（--skip-upload模式）")
print("[INFO] 检查服务器上是否存在部署包...")
check_cmd = f"ls -lh /data/{self.config['deploy']['deploy_package']}"
stdout, stderr = self.ssh.execute_command(check_cmd)
print(stdout)
# 检查md5文件是否存在，如果不存在则尝试创建
md5_check_cmd = f"ls -lh /data/{self.config['deploy']['deploy_package_md5']}"
md5_stdout, md5_stderr = self.ssh.execute_command(md5_check_cmd)
if 'No such file' in md5_stdout or 'No such file' in md5_stderr or '没有那个文件' in md5_stdout or '没有那个文件' in md5_stderr:
print("[INFO] MD5文件不存在，尝试创建...")
create_md5_cmd = f"cd /data && md5sum {self.config['deploy']['deploy_package']} > {self.config['deploy']['deploy_package_md5']}"
self.ssh.execute_command(create_md5_cmd)
print("[INFO] MD5文件已创建")
phase_result['steps'].append({
'name': '跳过上传（使用服务器已有部署包）',
'status': 'success'
})
else:
# 2. 清理旧部署文件
print("\n[步骤2] 清理旧部署文件...")
clean_cmd = f"cd /data && rm -rf {self.config['deploy']['deploy_dir']} && rm -f {self.config['deploy']['deploy_package']} && rm -f {self.config['deploy']['deploy_package_md5']}"
......@@ -119,7 +144,7 @@ class X86Deployer:
'status': 'success'
})
# 3. 上传部署包（需要检查本地文件是否存在）
# 3. 上传部署包
print("\n[步骤3] 上传部署包...")
source_dir = self.config['source']['deploy_package_dir']
deploy_package = self.config['deploy']['deploy_package']
......@@ -128,7 +153,6 @@ class X86Deployer:
local_package = os.path.join(source_dir, deploy_package)
local_md5 = os.path.join(source_dir, deploy_package_md5)
# 检查本地文件是否存在
if not os.path.exists(local_package):
error_msg = f"部署包不存在: {local_package}"
print(f"[ERROR] {error_msg}")
......@@ -140,7 +164,6 @@ class X86Deployer:
self.deploy_report['errors'].append(error_msg)
return False
# 使用 SFTP 上传
self.ssh.upload_file(local_package, f"/data/{deploy_package}")
self.ssh.upload_file(local_md5, f"/data/{deploy_package_md5}")
......@@ -150,12 +173,13 @@ class X86Deployer:
'files': [deploy_package, deploy_package_md5]
})
# 4. 校验部署包完整性
# 4. 校验部署包完整性（无论是否跳过上传都执行）
print("\n[步骤4] 校验部署包完整性...")
verify_cmd = f"cd /data && md5sum -c {deploy_package_md5}"
verify_cmd = f"cd /data && md5sum -c {self.config['deploy']['deploy_package_md5']}"
exit_status, stdout, stderr = self.ssh.execute_command_with_status(verify_cmd)
if exit_status == 0 and 'OK' in stdout:
# md5sum -c 输出可能是中文"成功"或英文"OK"
if exit_status == 0 and ('OK' in stdout or '成功' in stdout):
print(f"[SUCCESS] 部署包完整性校验通过")
phase_result['steps'].append({
'name': '部署包完整性校验',
......@@ -459,8 +483,12 @@ class X86Deployer:
print(f"[SUCCESS] 报告已保存: {report_file}")
return str(report_file)
def run_deploy(self):
"""执行完整部署流程"""
def run_deploy(self, skip_upload=False):
"""执行完整部署流程
Args:
skip_upload: 跳过上传步骤，使用服务器上已有的部署包
"""
self.deploy_report['start_time'] = datetime.now().isoformat()
try:
......@@ -468,7 +496,7 @@ class X86Deployer:
self.connect()
# 阶段1
if not self.phase1_pre_check_and_upload():
if not self.phase1_pre_check_and_upload(skip_upload=skip_upload):
print("[ERROR] 阶段1失败，终止部署")
return False
......@@ -506,14 +534,18 @@ class X86Deployer:
finally:
self.disconnect()
def run_full(self):
"""执行完整流程（部署+验收）"""
def run_full(self, skip_upload=False):
"""执行完整流程（部署+验收）
Args:
skip_upload: 跳过上传步骤，使用服务器上已有的部署包
"""
self.deploy_report['start_time'] = datetime.now().isoformat()
try:
self.connect()
if not self.phase1_pre_check_and_upload():
if not self.phase1_pre_check_and_upload(skip_upload=skip_upload):
return False
if not self.phase2_extract_and_deploy():
......@@ -542,17 +574,18 @@ def main():
parser.add_argument('--verify', action='store_true', help='执行验收检查（阶段3）')
parser.add_argument('--full', action='store_true', help='执行完整流程（阶段1+2+3）')
parser.add_argument('--config', default=None, help='配置文件路径')
parser.add_argument('--skip-upload', action='store_true', help='跳过上传步骤，使用服务器上已有的部署包')
args = parser.parse_args()
deployer = X86Deployer(args.config)
if args.deploy:
deployer.run_deploy()
deployer.run_deploy(skip_upload=args.skip_upload)
elif args.verify:
deployer.run_verify()
elif args.full:
deployer.run_full()
deployer.run_full(skip_upload=args.skip_upload)
else:
parser.print_help()
......
AuxiliaryTool/ScriptTool/RemoteDeploy/full_deploy.py
浏览文件 @ 6eb7badf
......@@ -100,7 +100,7 @@ CONFIGS = {
'deploy_dir': '/data/offline_auto_unifiedPlatform',
'deploy_pkg': 'offline_auto_unifiedPlatform.tar.gz',
'deploy_md5': 'offline_auto_unifiedPlatform.tar.gz.md5',
'nas_dir': r'Z:\发布版本\03服务器部署\15新统一平台\X86部署包\全量版',
'nas_dir': None, # 跳过上传，使用服务器上已有的部署包
'deploy_script': 'new_auto.sh',
'deploy_answers': "y\ny\ny\ny\ny\ny\ny\nn\n",
'deploy_log': '/data/offline_auto_unifiedPlatform/deploy_output.log',
......
Docs/PRD/远程自动化部署/_PRD_X86_麒麟V10远程自动化部署_需求文档_计划执行.md
浏览文件 @ 6eb7badf
# 计划执行_X86_麒麟V10远程自动化部署
> 版本：V1.0
> 创建日期：2026-06-04
> 版本：V2.0
> 创建日期：2026-07-09
> 基于文档：`_PRD_X86_麒麟V10远程自动化部署_需求文档.md`
> 部署文档：`X86架构_新统一平台自动化部署操作指导.md`
> 交付物：
> - `full_deploy.py` (统一部署主脚本)
> - `deploy_config.json` (凭据配置)
> - `reports/` (报告输出目录)
> - `.claude/skills/X86-QLV10-XTYBS/code/` (代码目录)
> - `AuxiliaryTool/ScriptTool/RemoteDeploy/reports/` (报告输出目录)
---
......@@ -15,6 +14,13 @@
对 X86架构-麒麟V10服务器（192.168.5.69）执行远程自动化部署全流程，包括部署包上传、解压、脚本执行、容器验证、API接口测试及报告生成。部署严格按照部署文档操作，解压过程禁止中断。
### 基本规则
1. **语言** — 所有回复、代码注释、Git提交信息默认使用中文（简体）描述
2. **SSH连接** — 使用Python paramiko连接，优先检索免密配置文件（`.claude/skills/X86-QLV10-XTYBS/code/ssh_keys/192.168.5.69/`），没有则询问密码，连接后配置免密
3. **上下文管理** — 自动压缩上下文，避免超过200k限制
4. **计划执行** — 每次理解需求文档，都需要输出《计划执行》文档，确认后方可执行
### 目标服务器信息
| 项目 | 值 |
......@@ -26,7 +32,7 @@
| 登录密码 | Ubains@123 |
| root切换 | 无需（直接root登录） |
| 部署目录 | /data/offline_auto_unifiedPlatform |
| 部署脚本 | new_auto.sh |
| 部署脚本 | new_auto.sh（执行时加 --all 参数） |
| 部署包名 | offline_auto_unifiedPlatform.tar.gz |
### 资源路径
......@@ -35,91 +41,93 @@
|------|------|
| 部署包来源 | Z:\发布版本\03服务器部署\15新统一平台\X86部署包\全量版 |
| 授权文件 | E:\自动化部署\X86-5.69\license.zip |
| 输出目录 | E:\自动化部署\X86-5.69 |
| 部署文档 | Docs/PRD/远程自动化部署/X86架构_新统一平台自动化部署操作指导.md |
| SSH免密目录 | .claude/skills/X86-QLV10-XTYBS/code/ssh_keys/192.168.5.69/ |
| 包装脚本 | .claude/skills/X86-QLV10-XTYBS/code/auto_deploy_wrapper.sh |
---
## 二、执行阶段划分
### 阶段一：前置准备与部署（预计40分钟）
### 阶段一：前置准备（预计10分钟）
| 序号 | 步骤 | 描述 | 执行方式 |
|------|------|------|----------|
| 1.1 | SSH连接 | 连接目标服务器192.168.5.69，root用户登录 | `paramiko` SSH |
| 1.1 | SSH连接 | 使用Python paramiko连接192.168.5.69，优先检索免密配置，没有则询问密码，连接后配置免密 | `paramiko` SSH |
| 1.2 | 磁盘检查 | 检查/data分区是否存在且空间充足 | `df -h /data` |
| 1.3 | 上传部署包 | 从网盘将 offline_auto_unifiedPlatform.tar.gz 和 md5 文件通过 SFTP 上传到 /data/ | `paramiko` SFTP |
| 1.4 | MD5校验 | 校验上传的部署包MD5是否一致 | `md5sum -c` |
| 1.5 | 解压部署包 | 解压 tar.gz 到 /data/ 目录，**禁止中断** | `tar -zxvf`，后台执行 |
| 1.6 | 运行部署脚本 | 执行 `new_auto.sh --all`，设置 `TERM=dumb`，管道输入交互应答 | 后台执行，监控日志 |
| 1.3 | 上传部署包 | 从网盘Z盘将 offline_auto_unifiedPlatform.tar.gz 和 .md5 文件通过 SFTP 上传到 /data/ | `paramiko` SFTP |
| 1.4 | 上传包装脚本 | 将 auto_deploy_wrapper.sh 上传到服务器 | `paramiko` SFTP |
### 阶段二：部署执行（预计40分钟）
| 序号 | 步骤 | 描述 | 执行方式 |
|------|------|------|----------|
| 2.1 | MD5校验 | 校验上传的部署包MD5是否一致 | `md5sum -c offline_auto_unifiedPlatform.tar.gz.md5` |
| 2.2 | 解压部署包 | 解压 tar.gz 到 /data/ 目录，**禁止中断** | `tar -zxvf`，后台执行 |
| 2.3 | 赋权脚本 | 解压完成后赋权 | `chmod 755 /data/offline_auto_unifiedPlatform/*.sh` |
| 2.4 | 运行部署脚本 | 上传包装脚本到 /data/offline_auto_unifiedPlatform/，赋权后执行 | 后台执行，监控日志 |
| 2.5 | 加载环境变量 | 加载系统环境变量 | `source /etc/profile` |
**关键注意事项：**
- 设置 `export TERM=dumb` 解决 whiptail 组件在非交互SSH下卡住的问题
- 部署脚本自动应答：`y\ny\ny\ny\ny\ny\ny\nn\n`
- 解压和部署过程禁止中断
---
- 通过包装脚本 `auto_deploy_wrapper.sh` 自动应答：`y\ny\ny\ny\ny\ny\ny\nn\n`
- 执行 `new_auto.sh --all` 而非 `new_auto.sh`，跳过系统选择菜单
- 解压和部署过程**禁止中断**
### 阶段二：容器验证与服务检查（预计10-15分钟）
### 阶段三：容器验证与服务检查（预计15分钟）
| 序号 | 步骤 | 描述 | 验证标准 |
|------|------|------|----------|
| 2.1 | 容器状态检查 | 检查所有Docker容器是否正常运行 | `docker ps` 无Exited容器 |
| 2.2 | 容器日志检查 | 核查容器日志是否存在异常 | 无异常ERROR日志 |
| 2.3 | 等待服务启动 | 等待10分钟让服务完全启动 | — |
---
| 3.1 | 容器状态检查 | 检查所有Docker容器是否正常运行 | `docker ps` 无Exited容器 |
| 3.2 | 容器日志检查 | 核查容器日志是否存在异常 | 无异常ERROR日志 |
| 3.3 | 等待服务启动 | 等待10分钟让服务完全启动 | — |
### 阶段三：API接口验证（预计5-10分钟）
### 阶段四：API接口验证（预计10分钟）
| 序号 | 接口 | 调用地址 | 成功标识 | 重试机制 |
|------|------|---------|---------|---------|
| 3.1 | 对外接口 | `https://192.168.5.69/exapi/message/getMsgPageList` | `无效token` 或 `Full authentication` | 5次/30秒间隔 |
| 3.2 | 预定系统 | `https://192.168.5.69/meetingV3/api/systemConfiguration/globalConfig?companyNumber=CN-SZ-00-0201` | `accessToken为空` | 5次/30秒间隔 |
| 3.3 | 运维集控 | `https://192.168.5.69/monitor/api2/api/servermonitor/` | `用户不存在` | 5次/30秒间隔 |
| 3.4 | 讯飞转录 | `https://192.168.5.69/voice/api/iflytek/roommaster?company_id=1&user_id=8&company_secret=...&getall=1` | `缺少关键参数` | 5次/30秒间隔 |
| 4.1 | 对外接口 | `https://192.168.5.69/exapi/message/getMsgPageList` | `无效token` 或 `Full authentication` | 5次/30秒间隔 |
| 4.2 | 预定系统 | `https://192.168.5.69/meetingV3/api/systemConfiguration/globalConfig?companyNumber=CN-SZ-00-0201` | `accessToken为空` | 5次/30秒间隔 |
| 4.3 | 运维集控 | `https://192.168.5.69/monitor/api2/api/servermonitor/` | `用户不存在或重新登录或已退出` | 5次/30秒间隔 |
| 4.4 | 讯飞转录 | `https://192.168.5.69/voice/api/iflytek/roommaster?company_id=1&user_id=8&company_secret=57d00f9f-020f-5f1f-b788-55fae843bceb&getall=1` | `缺少关键参数` | 5次/30秒间隔 |
**重试规则：**
- 调用失败或成功后均等待30秒再执行下一次测试
- 最多重试5次，每次间隔30秒（无论成功或失败）
- 记录每次测试结果
- 当结果为成功时，标识为服务启动正常，结束该接口测试
---
### 阶段四：系统授权（预计10分钟）
### 阶段五：系统授权（预计10分钟）
| 序号 | 步骤 | 描述 |
|------|------|------|
| 4.1 | 登录维护平台 | 访问 `https://192.168.5.69/#/LoginConfig`，超管登录 |
| 4.2 | 下载激活文件 | 按部署文档第三章操作，验证码填入 `csba` |
| 4.3 | 上传授权文件 | 上传 `E:\自动化部署\X86-5.69\license.zip` |
| 4.4 | 重启服务 | 按文档执行服务重启，等待10分钟 |
| 5.1 | 登录维护平台 | 打开浏览器访问 `https://192.168.5.69/#/LoginConfig`，超管登录 |
| 5.2 | 下载激活文件 | 点击"下载激活文件"，验证码填入 `csba` |
| 5.3 | 上传授权文件 | 上传 `E:\自动化部署\X86-5.69\license.zip`，每次敏感操作需重新输入密码和验证码 |
| 5.4 | 重启服务 | 进入"服务升级"界面，勾选"运维系统"和"预定系统2.0"，重启服务 |
| 5.5 | 等待服务启动 | 等待约10分钟服务完全启动 |
**授权操作凭证：**
- 超管账号：`superadmin`
- 超管密码：`Ubains@1357`
- 验证码：`csba`
---
### 阶段五：服务日志验证（预计5分钟）
### 阶段六：服务日志验证（预计5分钟）
| 序号 | 检查项 | 日志路径 |
|------|--------|---------|
| 5.1 | 预定对外服务 | /data/services/api/java-meeting/java-meeting-extapi/logs/ubains-INFO-AND-ERROR.log |
| 5.2 | 预定对内服务 | /data/services/api/java-meeting/java-meeting2.0/logs/ubains-INFO-AND-ERROR.log |
| 5.3 | 运维服务 | /data/services/api/python-cmdb/log/uinfo.log |
| 5.4 | 讯飞服务 | /data/services/api/python-voice/log/uinfo.log |
| 6.1 | 预定对外服务 | /data/services/api/java-meeting/java-meeting-extapi/logs/ubains-INFO-AND-ERROR.log |
| 6.2 | 预定对内服务 | /data/services/api/java-meeting/java-meeting2.0/logs/ubains-INFO-AND-ERROR.log |
| 6.3 | 运维服务 | /data/services/api/python-cmdb/log/uinfo.log |
| 6.4 | 讯飞服务 | /data/services/api/python-voice/log/uinfo.log |
**检查标准：** 日志无异常ERROR输出
**检查标准：** 日志无异常ERROR/Exception/StackTrace输出
---
### 阶段六：报告生成（预计5分钟）
### 阶段七：报告生成（预计5分钟）
| 序号 | 报告 | 内容 |
|------|------|------|
| 6.1 | 部署分析报告 | 部署执行情况、容器状态、API验证结果、日志检查、用时统计 |
| 7.1 | 部署分析报告 | 部署执行情况、容器状态、API验证结果、日志检查、用时统计 |
| 7.2 | 保存路径 | AuxiliaryTool/ScriptTool/RemoteDeploy/reports/X86_192.168.5.69_部署分析报告_20260709.md |
---
......@@ -136,32 +144,30 @@
## 四、验收标准
- [ ] SSH连接成功，磁盘空间充足
- [ ] 部署包上传并校验通过
- [ ] SSH连接成功（paramiko），免密配置完成
- [ ] 磁盘空间充足
- [ ] 部署包上传并MD5校验通过
- [ ] 解压完成，脚本赋权正确
- [ ] `new_auto.sh --all` 执行完成
- [ ] 所有Docker容器正常运行
- [ ] `auto_deploy_wrapper.sh` 执行完成（`new_auto.sh --all`）
- [ ] 所有Docker容器正常运行（状态为Up）
- [ ] 4个API接口全部验证通过（5次重试内）
- [ ] 授权文件上传成功，服务重启正常
- [ ] 4个服务日志无异常
- [ ] 4个服务日志无异常ERROR输出
- [ ] 部署分析报告已生成
- [ ] 全流程在1小时内完成
- [ ] **创建用户操作暂不执行**
---
## 五、执行命令
```bash
# 全流程执行
python full_deploy.py --arch x86_kylin --full
# 分步执行
python full_deploy.py --arch x86_kylin --deploy # 部署阶段
# （手动执行授权操作）
python full_deploy.py --arch x86_kylin --verify # 验证阶段
```
## 五、关键注意事项
> **注意：** `x86_kylin` 配置需要添加到 `full_deploy.py` 的 CONFIGS 中，当前代码尚未包含此配置。
1. **严格按文档执行** — 所有操作必须严格依照部署操作指导文档，不允许自行发挥
2. **禁止中断解压** — 解压缩过程中绝对不能中断
3. **whiptail处理** — 通过 `export TERM=dumb` 和包装脚本解决交互问题
4. **身份校验** — 维护平台每次敏感操作需重新输入密码和验证码（csba）
5. **服务等待** — 服务重启后等待约10分钟再验证，运维集控接口可能需要重试1-2次
6. **重试机制** — 所有接口验证严格按5次/30秒的重试机制执行
7. **部署超时** — 整体部署时间要求在1小时内完成（部署40分钟 + 授权10分钟 + 用户创建10分钟）
8. **创建用户** — 根据需求文档第五项，创建用户操作"先不执行"
---
......@@ -174,4 +180,5 @@ python full_deploy.py --arch x86_kylin --verify # 验证阶段
| 容器启动异常 | 服务不可用 | 查看容器日志排查 |
| API接口超时 | 验证失败 | 等待更长时间，手动验证 |
| 授权文件不匹配 | 服务异常 | 确认授权文件与服务器IP对应 |
| SSH免密配置失败 | 需手动输入密码 | 检查密钥配置，重新配置免密 |
| 麒麟V10兼容性 | 部署脚本异常 | 参考X86架构部署文档，麒麟V10基于CentOS兼容 |
\ No newline at end of file
Docs/部署指南/X86_Nacos集群部署指南.md
浏览文件 @ 6eb7badf
......@@ -132,8 +132,8 @@ docker run -d \
-p 9849:9849 \
-p 7848:7848 \
-e MODE=cluster \
-e NACOS_SERVERS="192.168.9.89:8848,192.168.9.90:8848,192.168.9.91:8848" \
-e NACOS_SERVER_IP="$server_ip" \
-e NACOS_SERVERS="192.168.5.41:8848,192.168.5.42:8848,192.168.5.43:8848" \
-e NACOS_SERVER_IP=192.168.5.41 \
-e NACOS_AUTH_ENABLE=true \
-e NACOS_AUTH_USERNAME="nacos" \
-e NACOS_AUTH_PASSWORD="dNrprU&2S" \
......@@ -147,7 +147,7 @@ docker run -d \
-v /data/middleware/nacos:/home/nacos \
-v /etc/localtime:/etc/localtime:ro \
--mac-address="02:42:ac:11:00:10" \
nacos/nacos-server:v2.5.2
nacos-server:v2.5.2
```
**关键环境变量说明：**
......
Markdown 格式
0% 或
您添加了 0 人 到此讨论。请谨慎行事。
请先完成此评论的编辑！
请 注册 或者 后发表评论