--- todos: - id: "redis-keys" content: "RedisKeyManagerService 增加 duration buckets/sum/count 三个 key 常量" status: pending - id: "prometheus-service" content: "PrometheusService 实现 recordHttpRequestDuration + render histogram 输出" status: pending - id: "middleware-timing" content: "PrometheusMiddleware 计时并在 finally 记录 duration" status: pending - id: "grafana-panels" content: "app_http_services_dashboard.json 增加 P50/P95/P99/Avg 与 route 耗时面板" status: pending - id: "verify" content: "curl /metrics 验证 + 跑 completion-report 门禁" status: pending isProject: false --- # slot_wallet HTTP 响应时长监控 ## 现状 当前链路： ```mermaid sequenceDiagram participant Client participant Middleware as PrometheusMiddleware participant Handler participant Redis participant Metrics as MetricsController Client->>Middleware: HTTP Request Middleware->>Handler: handler() Handler-->>Middleware: Response Middleware->>Redis: hIncrBy requests_total Middleware-->>Client: Response Note over Metrics,Redis: Prometheus scrape Metrics->>Redis: hGetAll Metrics-->>Client: text/plain metrics ``` - 中间件：[`slot_wallet/app/middleware/PrometheusMiddleware.php`](slot_wallet/app/middleware/PrometheusMiddleware.php) 在 `finally` 中调用 `recordHttpRequest()`，只记请求数。 - 指标服务：[`slot_wallet/app/service/PrometheusService.php`](slot_wallet/app/service/PrometheusService.php) 用 Redis Hash 存 counter，经 [`slot_wallet/app/controller/MetricsController.php`](slot_wallet/app/controller/MetricsController.php) 的 `/metrics` 输出。 - Grafana：[`slot_wallet/doc/grafana/app_http_services_dashboard.json`](slot_wallet/doc/grafana/app_http_services_dashboard.json) 仅查询 `app_http_requests_total` / `app_business_responses_total`。 **结论：可以加响应时长**，且与现有架构兼容；推荐新增标准 Prometheus Histogram，而不是改现有 counter。 --- ## 目标指标 新增 metric：`app_http_request_duration_seconds`（histogram） | 子指标 | 含义 | |--------|------| | `_bucket{le="..."}` | 各耗时桶累计次数 | | `_sum` | 累计耗时（秒） | | `_count` | 观测次数 | **Labels** 与现有请求 counter 保持一致：`service`, `method`, `route`, `status`，便于在 Grafana 与请求量 join。 **Histogram buckets（秒）**（Webman API 常用区间）： `0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, +Inf` --- ## 代码改动 ### 1. Redis Key — [`slot_wallet/app/service/RedisKeyManagerService.php`](slot_wallet/app/service/RedisKeyManagerService.php) 新增 3 个常量（与现有 `PROMETHEUS_*` 命名一致）： - `PROMETHEUS_HTTP_REQUEST_DURATION_BUCKETS` → `prometheus:http:duration:buckets` - `PROMETHEUS_HTTP_REQUEST_DURATION_SUM` → `prometheus:http:duration:sum` - `PROMETHEUS_HTTP_REQUEST_DURATION_COUNT` → `prometheus:http:duration:count` ### 2. 记录耗时 — [`slot_wallet/app/service/PrometheusService.php`](slot_wallet/app/service/PrometheusService.php) 新增 `recordHttpRequestDuration(string $method, string $route, int $status, float $durationSeconds)`： - field 仍用 `json_encode` 存 labels（bucket 额外带 `le`） - `_bucket` / `_count`：`Redis::hIncrBy(..., 1)` - `_sum`：`Redis::hIncrByFloat(..., $durationSeconds)`（避免整数毫秒精度损失） - 对每个 `le` bucket，若 `$durationSeconds <= le` 则递增；最后递增 `+Inf` bucket - 抽取私有方法 `buildHttpRequestLabels()`，供 `recordHttpRequest` 与 duration 复用，避免重复 JSON 拼装 扩展 `render()`，在现有 counter 之后输出： ```text # HELP app_http_request_duration_seconds HTTP request latency in seconds. # TYPE app_http_request_duration_seconds histogram app_http_request_duration_seconds_bucket{...,le="0.1"} N ... app_http_request_duration_seconds_sum{...} X.XXX app_http_request_duration_seconds_count{...} N ``` ### 3. 中间件计时 — [`slot_wallet/app/middleware/PrometheusMiddleware.php`](slot_wallet/app/middleware/PrometheusMiddleware.php) 在 `process()` 开头 `$startedAt = microtime(true)`，`finally` 中： ```php $durationSeconds = microtime(true) - $startedAt; PrometheusService::recordHttpRequestDuration($method, $route, $status, $durationSeconds); ``` 优化：将 `resolveRoute()` 结果缓存到局部变量，避免 `finally` 里重复解析；`recordHttpRequest` 与 `recordHttpRequestDuration` 共用同一份 `method/route/status`。 耗时口径：**中间件包裹的完整 handler 执行时间**（含业务逻辑，不含 Prometheus 写 Redis 本身；写在 `finally` 末尾，影响极小）。 --- ## Grafana 仪表盘扩展 更新 [`slot_wallet/doc/grafana/app_http_services_dashboard.json`](slot_wallet/doc/grafana/app_http_services_dashboard.json)，在现有请求量面板下方新增一行「Latency」区域（`version` 递增为 3）： | 面板 | 类型 | PromQL 示例 | |------|------|-------------| | P50 Latency | stat | `histogram_quantile(0.50, sum by (le) (rate(app_http_request_duration_seconds_bucket{service="$service"}[$__rate_interval])))` | | P95 Latency | stat | 同上 `0.95` | | P99 Latency | stat | 同上 `0.99` | | Avg Latency | stat | `sum(rate(..._sum...)) / sum(rate(..._count...))` | | Latency Trend (P50/P95/P99) | timeseries | 三条 quantile 曲线，unit=`s` | | Top Slow Routes (P95) | bargauge | `topk(10, histogram_quantile(0.95, sum by (route, le) (rate(..._bucket{service="$service"}[$__range]))))` | | Route Latency Details | table | 合并 route 的 P95、Avg、Requests（与现有 Route Request Details 面板并列或扩列） | 所有新面板 `unit` 设为 `s`（秒），threshold 可按业务再调。 --- ## 数据流（改后） ```mermaid sequenceDiagram participant Middleware as PrometheusMiddleware participant Redis participant Grafana Middleware->>Middleware: startedAt = microtime(true) Middleware->>Middleware: handler() Middleware->>Redis: incr requests_total Middleware->>Redis: incr duration buckets/sum/count Note over Grafana: rate + histogram_quantile Grafana->>Grafana: P50/P95/P99/Avg ``` --- ## 验证步骤 1. 本地发若干 HTTP 请求到 slot_wallet（含快/慢接口）。 2. `curl /metrics`，确认出现 `app_http_request_duration_seconds_bucket/_sum/_count`。 3. Prometheus scrape 后，在 Grafana 导入更新后的 dashboard JSON，选择 `service=slot_wallet`，确认 P50/P95 有数据。 4. 跑 `slot-backend-completion-report` 门禁（`php -l` + verify 脚本）。 --- ## 注意事项 - **仅改 slot_wallet**（按你的选择）；`slot_console` / `slot_pwa` 结构相同，后续可 copy 同一套改动。 - Redis Hash 会随 `route × status × bucket` 增长；与现有 `app_http_requests_total` 同一量级，可接受。 - Histogram 在 Prometheus 侧用 `rate()` + `histogram_quantile()` 算分位；Grafana 查询写法与标准 Prometheus 一致。 - 无需改 Prometheus scrape 配置（metric 名新增，非替换）。