168 lines
7.0 KiB
Markdown
168 lines
7.0 KiB
Markdown
<!-- c8f2b444-d26a-4f73-b3e8-317e7228c8ab -->
|
||
---
|
||
todos:
|
||
- id: "redis-keys"
|
||
content: "RedisKeyManagerService 增加 duration buckets/sum/count 三个 key 常量"
|
||
status: pending
|
||
- id: "prometheus-service"
|
||
content: "PrometheusService 实现 recordHttpRequestDuration + render histogram 输出"
|
||
status: pending
|
||
- id: "middleware-timing"
|
||
content: "PrometheusMiddleware 计时并在 finally 记录 duration"
|
||
status: pending
|
||
- id: "grafana-panels"
|
||
content: "app_http_services_dashboard.json 增加 P50/P95/P99/Avg 与 route 耗时面板"
|
||
status: pending
|
||
- id: "verify"
|
||
content: "curl /metrics 验证 + 跑 completion-report 门禁"
|
||
status: pending
|
||
isProject: false
|
||
---
|
||
# slot_wallet HTTP 响应时长监控
|
||
|
||
## 现状
|
||
|
||
当前链路:
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant Client
|
||
participant Middleware as PrometheusMiddleware
|
||
participant Handler
|
||
participant Redis
|
||
participant Metrics as MetricsController
|
||
|
||
Client->>Middleware: HTTP Request
|
||
Middleware->>Handler: handler()
|
||
Handler-->>Middleware: Response
|
||
Middleware->>Redis: hIncrBy requests_total
|
||
Middleware-->>Client: Response
|
||
Note over Metrics,Redis: Prometheus scrape
|
||
Metrics->>Redis: hGetAll
|
||
Metrics-->>Client: text/plain metrics
|
||
```
|
||
|
||
- 中间件:[`slot_wallet/app/middleware/PrometheusMiddleware.php`](slot_wallet/app/middleware/PrometheusMiddleware.php) 在 `finally` 中调用 `recordHttpRequest()`,只记请求数。
|
||
- 指标服务:[`slot_wallet/app/service/PrometheusService.php`](slot_wallet/app/service/PrometheusService.php) 用 Redis Hash 存 counter,经 [`slot_wallet/app/controller/MetricsController.php`](slot_wallet/app/controller/MetricsController.php) 的 `/metrics` 输出。
|
||
- Grafana:[`slot_wallet/doc/grafana/app_http_services_dashboard.json`](slot_wallet/doc/grafana/app_http_services_dashboard.json) 仅查询 `app_http_requests_total` / `app_business_responses_total`。
|
||
|
||
**结论:可以加响应时长**,且与现有架构兼容;推荐新增标准 Prometheus Histogram,而不是改现有 counter。
|
||
|
||
---
|
||
|
||
## 目标指标
|
||
|
||
新增 metric:`app_http_request_duration_seconds`(histogram)
|
||
|
||
| 子指标 | 含义 |
|
||
|--------|------|
|
||
| `_bucket{le="..."}` | 各耗时桶累计次数 |
|
||
| `_sum` | 累计耗时(秒) |
|
||
| `_count` | 观测次数 |
|
||
|
||
**Labels** 与现有请求 counter 保持一致:`service`, `method`, `route`, `status`,便于在 Grafana 与请求量 join。
|
||
|
||
**Histogram buckets(秒)**(Webman API 常用区间):
|
||
|
||
`0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, +Inf`
|
||
|
||
---
|
||
|
||
## 代码改动
|
||
|
||
### 1. Redis Key — [`slot_wallet/app/service/RedisKeyManagerService.php`](slot_wallet/app/service/RedisKeyManagerService.php)
|
||
|
||
新增 3 个常量(与现有 `PROMETHEUS_*` 命名一致):
|
||
|
||
- `PROMETHEUS_HTTP_REQUEST_DURATION_BUCKETS` → `prometheus:http:duration:buckets`
|
||
- `PROMETHEUS_HTTP_REQUEST_DURATION_SUM` → `prometheus:http:duration:sum`
|
||
- `PROMETHEUS_HTTP_REQUEST_DURATION_COUNT` → `prometheus:http:duration:count`
|
||
|
||
### 2. 记录耗时 — [`slot_wallet/app/service/PrometheusService.php`](slot_wallet/app/service/PrometheusService.php)
|
||
|
||
新增 `recordHttpRequestDuration(string $method, string $route, int $status, float $durationSeconds)`:
|
||
|
||
- field 仍用 `json_encode` 存 labels(bucket 额外带 `le`)
|
||
- `_bucket` / `_count`:`Redis::hIncrBy(..., 1)`
|
||
- `_sum`:`Redis::hIncrByFloat(..., $durationSeconds)`(避免整数毫秒精度损失)
|
||
- 对每个 `le` bucket,若 `$durationSeconds <= le` 则递增;最后递增 `+Inf` bucket
|
||
- 抽取私有方法 `buildHttpRequestLabels()`,供 `recordHttpRequest` 与 duration 复用,避免重复 JSON 拼装
|
||
|
||
扩展 `render()`,在现有 counter 之后输出:
|
||
|
||
```text
|
||
# HELP app_http_request_duration_seconds HTTP request latency in seconds.
|
||
# TYPE app_http_request_duration_seconds histogram
|
||
app_http_request_duration_seconds_bucket{...,le="0.1"} N
|
||
...
|
||
app_http_request_duration_seconds_sum{...} X.XXX
|
||
app_http_request_duration_seconds_count{...} N
|
||
```
|
||
|
||
### 3. 中间件计时 — [`slot_wallet/app/middleware/PrometheusMiddleware.php`](slot_wallet/app/middleware/PrometheusMiddleware.php)
|
||
|
||
在 `process()` 开头 `$startedAt = microtime(true)`,`finally` 中:
|
||
|
||
```php
|
||
$durationSeconds = microtime(true) - $startedAt;
|
||
PrometheusService::recordHttpRequestDuration($method, $route, $status, $durationSeconds);
|
||
```
|
||
|
||
优化:将 `resolveRoute()` 结果缓存到局部变量,避免 `finally` 里重复解析;`recordHttpRequest` 与 `recordHttpRequestDuration` 共用同一份 `method/route/status`。
|
||
|
||
耗时口径:**中间件包裹的完整 handler 执行时间**(含业务逻辑,不含 Prometheus 写 Redis 本身;写在 `finally` 末尾,影响极小)。
|
||
|
||
---
|
||
|
||
## Grafana 仪表盘扩展
|
||
|
||
更新 [`slot_wallet/doc/grafana/app_http_services_dashboard.json`](slot_wallet/doc/grafana/app_http_services_dashboard.json),在现有请求量面板下方新增一行「Latency」区域(`version` 递增为 3):
|
||
|
||
| 面板 | 类型 | PromQL 示例 |
|
||
|------|------|-------------|
|
||
| P50 Latency | stat | `histogram_quantile(0.50, sum by (le) (rate(app_http_request_duration_seconds_bucket{service="$service"}[$__rate_interval])))` |
|
||
| P95 Latency | stat | 同上 `0.95` |
|
||
| P99 Latency | stat | 同上 `0.99` |
|
||
| Avg Latency | stat | `sum(rate(..._sum...)) / sum(rate(..._count...))` |
|
||
| Latency Trend (P50/P95/P99) | timeseries | 三条 quantile 曲线,unit=`s` |
|
||
| Top Slow Routes (P95) | bargauge | `topk(10, histogram_quantile(0.95, sum by (route, le) (rate(..._bucket{service="$service"}[$__range]))))` |
|
||
| Route Latency Details | table | 合并 route 的 P95、Avg、Requests(与现有 Route Request Details 面板并列或扩列) |
|
||
|
||
所有新面板 `unit` 设为 `s`(秒),threshold 可按业务再调。
|
||
|
||
---
|
||
|
||
## 数据流(改后)
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant Middleware as PrometheusMiddleware
|
||
participant Redis
|
||
participant Grafana
|
||
|
||
Middleware->>Middleware: startedAt = microtime(true)
|
||
Middleware->>Middleware: handler()
|
||
Middleware->>Redis: incr requests_total
|
||
Middleware->>Redis: incr duration buckets/sum/count
|
||
Note over Grafana: rate + histogram_quantile
|
||
Grafana->>Grafana: P50/P95/P99/Avg
|
||
```
|
||
|
||
---
|
||
|
||
## 验证步骤
|
||
|
||
1. 本地发若干 HTTP 请求到 slot_wallet(含快/慢接口)。
|
||
2. `curl /metrics`,确认出现 `app_http_request_duration_seconds_bucket/_sum/_count`。
|
||
3. Prometheus scrape 后,在 Grafana 导入更新后的 dashboard JSON,选择 `service=slot_wallet`,确认 P50/P95 有数据。
|
||
4. 跑 `slot-backend-completion-report` 门禁(`php -l` + verify 脚本)。
|
||
|
||
---
|
||
|
||
## 注意事项
|
||
|
||
- **仅改 slot_wallet**(按你的选择);`slot_console` / `slot_pwa` 结构相同,后续可 copy 同一套改动。
|
||
- Redis Hash 会随 `route × status × bucket` 增长;与现有 `app_http_requests_total` 同一量级,可接受。
|
||
- Histogram 在 Prometheus 侧用 `rate()` + `histogram_quantile()` 算分位;Grafana 查询写法与标准 Prometheus 一致。
|
||
- 无需改 Prometheus scrape 配置(metric 名新增,非替换)。
|