168 lines
7.0 KiB
Markdown
168 lines
7.0 KiB
Markdown
<!-- c8f2b444-d26a-4f73-b3e8-317e7228c8ab -->
|
||
---
|
||
todos:
|
||
- id: "redis-keys"
|
||
content: "RedisKeyManagerService 增加 duration buckets/sum/count 三个 key 常量"
|
||
status: pending
|
||
- id: "prometheus-service"
|
||
content: "PrometheusService 实现 recordHttpRequestDuration + render histogram 输出"
|
||
status: pending
|
||
- id: "middleware-timing"
|
||
content: "PrometheusMiddleware 计时并在 finally 记录 duration"
|
||
status: pending
|
||
- id: "grafana-panels"
|
||
content: "app_http_services_dashboard.json 增加 P50/P95/P99/Avg 与 route 耗时面板"
|
||
status: pending
|
||
- id: "verify"
|
||
content: "curl /metrics 验证 + 跑 completion-report 门禁"
|
||
status: pending
|
||
isProject: false
|
||
---
|
||
# slot_wallet HTTP 响应时长监控
|
||
|
||
## 现状
|
||
|
||
当前链路：
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant Client
|
||
participant Middleware as PrometheusMiddleware
|
||
participant Handler
|
||
participant Redis
|
||
participant Metrics as MetricsController
|
||
|
||
Client->>Middleware: HTTP Request
|
||
Middleware->>Handler: handler()
|
||
Handler-->>Middleware: Response
|
||
Middleware->>Redis: hIncrBy requests_total
|
||
Middleware-->>Client: Response
|
||
Note over Metrics,Redis: Prometheus scrape
|
||
Metrics->>Redis: hGetAll
|
||
Metrics-->>Client: text/plain metrics
|
||
```
|
||
|
||
- 中间件：[`slot_wallet/app/middleware/PrometheusMiddleware.php`](slot_wallet/app/middleware/PrometheusMiddleware.php) 在 `finally` 中调用 `recordHttpRequest()`，只记请求数。
|
||
- 指标服务：[`slot_wallet/app/service/PrometheusService.php`](slot_wallet/app/service/PrometheusService.php) 用 Redis Hash 存 counter，经 [`slot_wallet/app/controller/MetricsController.php`](slot_wallet/app/controller/MetricsController.php) 的 `/metrics` 输出。
|
||
- Grafana：[`slot_wallet/doc/grafana/app_http_services_dashboard.json`](slot_wallet/doc/grafana/app_http_services_dashboard.json) 仅查询 `app_http_requests_total` / `app_business_responses_total`。
|
||
|
||
**结论：可以加响应时长**，且与现有架构兼容；推荐新增标准 Prometheus Histogram，而不是改现有 counter。
|
||
|
||
---
|
||
|
||
## 目标指标
|
||
|
||
新增 metric：`app_http_request_duration_seconds`（histogram）
|
||
|
||
| 子指标 | 含义 |
|
||
|--------|------|
|
||
| `_bucket{le="..."}` | 各耗时桶累计次数 |
|
||
| `_sum` | 累计耗时（秒） |
|
||
| `_count` | 观测次数 |
|
||
|
||
**Labels** 与现有请求 counter 保持一致：`service`, `method`, `route`, `status`，便于在 Grafana 与请求量 join。
|
||
|
||
**Histogram buckets（秒）**（Webman API 常用区间）：
|
||
|
||
`0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, +Inf`
|
||
|
||
---
|
||
|
||
## 代码改动
|
||
|
||
### 1. Redis Key — [`slot_wallet/app/service/RedisKeyManagerService.php`](slot_wallet/app/service/RedisKeyManagerService.php)
|
||
|
||
新增 3 个常量（与现有 `PROMETHEUS_*` 命名一致）：
|
||
|
||
- `PROMETHEUS_HTTP_REQUEST_DURATION_BUCKETS` → `prometheus:http:duration:buckets`
|
||
- `PROMETHEUS_HTTP_REQUEST_DURATION_SUM` → `prometheus:http:duration:sum`
|
||
- `PROMETHEUS_HTTP_REQUEST_DURATION_COUNT` → `prometheus:http:duration:count`
|
||
|
||
### 2. 记录耗时 — [`slot_wallet/app/service/PrometheusService.php`](slot_wallet/app/service/PrometheusService.php)
|
||
|
||
新增 `recordHttpRequestDuration(string $method, string $route, int $status, float $durationSeconds)`：
|
||
|
||
- field 仍用 `json_encode` 存 labels（bucket 额外带 `le`）
|
||
- `_bucket` / `_count`：`Redis::hIncrBy(..., 1)`
|
||
- `_sum`：`Redis::hIncrByFloat(..., $durationSeconds)`（避免整数毫秒精度损失）
|
||
- 对每个 `le` bucket，若 `$durationSeconds <= le` 则递增；最后递增 `+Inf` bucket
|
||
- 抽取私有方法 `buildHttpRequestLabels()`，供 `recordHttpRequest` 与 duration 复用，避免重复 JSON 拼装
|
||
|
||
扩展 `render()`，在现有 counter 之后输出：
|
||
|
||
```text
|
||
# HELP app_http_request_duration_seconds HTTP request latency in seconds.
|
||
# TYPE app_http_request_duration_seconds histogram
|
||
app_http_request_duration_seconds_bucket{...,le="0.1"} N
|
||
...
|
||
app_http_request_duration_seconds_sum{...} X.XXX
|
||
app_http_request_duration_seconds_count{...} N
|
||
```
|
||
|
||
### 3. 中间件计时 — [`slot_wallet/app/middleware/PrometheusMiddleware.php`](slot_wallet/app/middleware/PrometheusMiddleware.php)
|
||
|
||
在 `process()` 开头 `$startedAt = microtime(true)`，`finally` 中：
|
||
|
||
```php
|
||
$durationSeconds = microtime(true) - $startedAt;
|
||
PrometheusService::recordHttpRequestDuration($method, $route, $status, $durationSeconds);
|
||
```
|
||
|
||
优化：将 `resolveRoute()` 结果缓存到局部变量，避免 `finally` 里重复解析；`recordHttpRequest` 与 `recordHttpRequestDuration` 共用同一份 `method/route/status`。
|
||
|
||
耗时口径：**中间件包裹的完整 handler 执行时间**（含业务逻辑，不含 Prometheus 写 Redis 本身；写在 `finally` 末尾，影响极小）。
|
||
|
||
---
|
||
|
||
## Grafana 仪表盘扩展
|
||
|
||
更新 [`slot_wallet/doc/grafana/app_http_services_dashboard.json`](slot_wallet/doc/grafana/app_http_services_dashboard.json)，在现有请求量面板下方新增一行「Latency」区域（`version` 递增为 3）：
|
||
|
||
| 面板 | 类型 | PromQL 示例 |
|
||
|------|------|-------------|
|
||
| P50 Latency | stat | `histogram_quantile(0.50, sum by (le) (rate(app_http_request_duration_seconds_bucket{service="$service"}[$__rate_interval])))` |
|
||
| P95 Latency | stat | 同上 `0.95` |
|
||
| P99 Latency | stat | 同上 `0.99` |
|
||
| Avg Latency | stat | `sum(rate(..._sum...)) / sum(rate(..._count...))` |
|
||
| Latency Trend (P50/P95/P99) | timeseries | 三条 quantile 曲线，unit=`s` |
|
||
| Top Slow Routes (P95) | bargauge | `topk(10, histogram_quantile(0.95, sum by (route, le) (rate(..._bucket{service="$service"}[$__range]))))` |
|
||
| Route Latency Details | table | 合并 route 的 P95、Avg、Requests（与现有 Route Request Details 面板并列或扩列） |
|
||
|
||
所有新面板 `unit` 设为 `s`（秒），threshold 可按业务再调。
|
||
|
||
---
|
||
|
||
## 数据流（改后）
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant Middleware as PrometheusMiddleware
|
||
participant Redis
|
||
participant Grafana
|
||
|
||
Middleware->>Middleware: startedAt = microtime(true)
|
||
Middleware->>Middleware: handler()
|
||
Middleware->>Redis: incr requests_total
|
||
Middleware->>Redis: incr duration buckets/sum/count
|
||
Note over Grafana: rate + histogram_quantile
|
||
Grafana->>Grafana: P50/P95/P99/Avg
|
||
```
|
||
|
||
---
|
||
|
||
## 验证步骤
|
||
|
||
1. 本地发若干 HTTP 请求到 slot_wallet（含快/慢接口）。
|
||
2. `curl /metrics`，确认出现 `app_http_request_duration_seconds_bucket/_sum/_count`。
|
||
3. Prometheus scrape 后，在 Grafana 导入更新后的 dashboard JSON，选择 `service=slot_wallet`，确认 P50/P95 有数据。
|
||
4. 跑 `slot-backend-completion-report` 门禁（`php -l` + verify 脚本）。
|
||
|
||
---
|
||
|
||
## 注意事项
|
||
|
||
- **仅改 slot_wallet**（按你的选择）；`slot_console` / `slot_pwa` 结构相同，后续可 copy 同一套改动。
|
||
- Redis Hash 会随 `route × status × bucket` 增长；与现有 `app_http_requests_total` 同一量级，可接受。
|
||
- Histogram 在 Prometheus 侧用 `rate()` + `histogram_quantile()` 算分位；Grafana 查询写法与标准 Prometheus 一致。
|
||
- 无需改 Prometheus scrape 配置（metric 名新增，非替换）。
|