幂等性设计知识库

Appearance

网络超时与重试机制 ​

概述 ​

网络超时是分布式系统中最常见的故障场景之一。当请求因网络延迟、服务器繁忙或中间件故障而超时时，客户端通常会进行重试。如果重试机制设计不当，就会导致重复执行的问题。

为什么会出现超时重试 ​

常见超时场景 ​

客户端                    网络                     服务端
  |                        |                         |
  |-- 请求 ---------------->|------------------------>|
  |                        |                         |-- 处理中...
  |                        |                         |-- 处理中...
  |                        |  (网络延迟/丢包)         |
  |  (等待响应...)          |                         |-- 处理完成 ✓
  |                        |                         |
  |  [超时!]               |                         |
  |-- 重试请求 ------------>|------------------------>|
  |                        |                         |-- 再次处理 ❌
  |                        |                         |-- 产生重复数据

超时原因分类 ​

类型原因发生概率解决方案
网络层丢包、延迟、DNS 解析慢高重试 + 幂等性
传输层TCP 连接超时、TLS 握手慢中连接池优化
应用层服务器处理慢、数据库锁中异步处理 + 幂等性
网关层负载均衡器超时、限流低调整超时配置

C# 中的超时处理 ​

1. HttpClient 超时配置 ​

csharp
public class ApiService
{
    private readonly HttpClient _httpClient;
    private readonly ILogger<ApiService> _logger;
    
    public ApiService(HttpClient httpClient, ILogger<ApiService> logger)
    {
        _httpClient = httpClient;
        _logger = logger;
        
        // 设置合理的超时时间
        _httpClient.Timeout = TimeSpan.FromSeconds(30);
    }
    
    /// <summary>
    /// 带重试的请求（非幂等 - 危险！）
    /// </summary>
    public async Task<Order> CreateOrderUnsafe(CreateOrderRequest request)
    {
        var maxRetries = 3;
        
        for (int i = 0; i < maxRetries; i++)
        {
            try
            {
                var response = await _httpClient.PostAsJsonAsync(
                    "/api/orders", 
                    request);
                
                response.EnsureSuccessStatusCode();
                return await response.Content.ReadFromJsonAsync<Order>();
            }
            catch (TaskCanceledException ex) when (ex.InnerException is TimeoutException)
            {
                _logger.LogWarning(ex, "Request timeout, attempt {Attempt}/{MaxRetries}", 
                    i + 1, maxRetries);
                
                if (i == maxRetries - 1) throw;
                
                // 等待后重试
                await Task.Delay(TimeSpan.FromSeconds(Math.Pow(2, i)));
            }
        }
        
        throw new Exception("Failed after retries");
    }
}

问题：如果第一次请求实际上已经成功（只是响应丢失），重试会导致重复创建订单。

2. 带幂等性的重试（推荐） ​

csharp
public class IdempotentApiService
{
    private readonly HttpClient _httpClient;
    private readonly ILogger<IdempotentApiService> _logger;
    
    public async Task<Order> CreateOrderWithIdempotency(
        CreateOrderRequest request)
    {
        var maxRetries = 3;
        
        // 生成确定性幂等键（基于业务数据）
        var idempotencyKey = GenerateIdempotencyKey(request);
        
        for (int i = 0; i < maxRetries; i++)
        {
            try
            {
                var httpRequest = new HttpRequestMessage(HttpMethod.Post, "/api/orders");
                httpRequest.Headers.Add("Idempotency-Key", idempotencyKey);
                httpRequest.Content = JsonContent.Create(request);
                
                var response = await _httpClient.SendAsync(httpRequest);
                response.EnsureSuccessStatusCode();
                
                return await response.Content.ReadFromJsonAsync<Order>();
            }
            catch (TaskCanceledException ex) when (IsTimeout(ex))
            {
                _logger.LogWarning(ex, 
                    "Request timeout with key {Key}, retry {Attempt}/{Max}", 
                    idempotencyKey, i + 1, maxRetries);
                
                if (i == maxRetries - 1) throw;
                
                // 指数退避
                await Task.Delay(TimeSpan.FromSeconds(Math.Pow(2, i)));
            }
            catch (HttpRequestException ex) when (ex.StatusCode == HttpStatusCode.Conflict)
            {
                // 409 Conflict - 请求正在处理中
                _logger.LogInformation("Request still processing, waiting...");
                await Task.Delay(TimeSpan.FromSeconds(2));
            }
        }
        
        throw new Exception("Failed after retries");
    }
    
    private string GenerateIdempotencyKey(CreateOrderRequest request)
    {
        // 基于业务数据生成确定性键
        var data = $"{request.UserId}_{request.ProductId}_{request.Quantity}_{DateTime.UtcNow:yyyyMMdd}";
        using var sha256 = SHA256.Create();
        var hash = sha256.ComputeHash(Encoding.UTF8.GetBytes(data));
        return Convert.ToHexString(hash)[..32].ToLowerInvariant();
    }
    
    private bool IsTimeout(TaskCanceledException ex)
    {
        return ex.InnerException is TimeoutException || 
               ex.CancellationToken.IsCancellationRequested == false;
    }
}

优势：即使重试多次，服务端也只会处理一次。

重试策略最佳实践 ​

1. 指数退避（Exponential Backoff） ​

csharp
public class RetryPolicy
{
    /// <summary>
    /// 带指数退避的重试
    /// </summary>
    public static async Task<T> ExecuteWithRetry<T>(
        Func<Task<T>> action,
        int maxRetries = 3,
        TimeSpan? baseDelay = null)
    {
        var delay = baseDelay ?? TimeSpan.FromSeconds(1);
        
        for (int i = 0; i <= maxRetries; i++)
        {
            try
            {
                return await action();
            }
            catch (Exception ex) when (IsRetryable(ex) && i < maxRetries)
            {
                var waitTime = delay * Math.Pow(2, i); // 1s, 2s, 4s, 8s...
                
                // 添加随机抖动，避免惊群效应
                var jitter = TimeSpan.FromMilliseconds(new Random().Next(0, 1000));
                waitTime += jitter;
                
                Console.WriteLine($"Retry {i + 1}/{maxRetries} after {waitTime.TotalSeconds}s");
                
                await Task.Delay(waitTime);
            }
        }
        
        throw new Exception($"Failed after {maxRetries} retries");
    }
    
    private static bool IsRetryable(Exception ex)
    {
        return ex is TaskCanceledException ||  // 超时
               ex is HttpRequestException httpEx && 
               httpEx.StatusCode.HasValue &&
               (int)httpEx.StatusCode >= 500;   // 服务器错误
    }
}

// 使用示例
var order = await RetryPolicy.ExecuteWithRetry(
    async () => await _apiClient.CreateOrder(request),
    maxRetries: 3,
    baseDelay: TimeSpan.FromSeconds(1));

2. 断路器模式（Circuit Breaker） ​

csharp
using Polly;

public class ResilientApiService
{
    private readonly AsyncPolicy _retryPolicy;
    private readonly AsyncPolicy _circuitBreakerPolicy;
    private readonly HttpClient _httpClient;
    
    public ResilientApiService(HttpClient httpClient)
    {
        _httpClient = httpClient;
        
        // 重试策略：最多3次，指数退避
        _retryPolicy = Policy
            .Handle<HttpRequestException>()
            .OrResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)
            .WaitAndRetryAsync(
                retryCount: 3,
                sleepDurationProvider: (attempt, context) => 
                    TimeSpan.FromSeconds(Math.Pow(2, attempt)),
                onRetry: (outcome, timespan, retryNumber, context) =>
                {
                    Console.WriteLine($"Retry {retryNumber} after {timespan.TotalSeconds}s");
                });
        
        // 断路器策略：连续5次失败后打开断路器，30秒后尝试恢复
        _circuitBreakerPolicy = Policy
            .Handle<HttpRequestException>()
            .OrResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)
            .CircuitBreakerAsync(
                exceptionsAllowedBeforeBreaking: 5,
                durationOfBreak: TimeSpan.FromSeconds(30),
                onBreak: (outcome, breakDelay) =>
                {
                    Console.WriteLine($"Circuit broken for {breakDelay.TotalSeconds}s");
                },
                onReset: () =>
                {
                    Console.WriteLine("Circuit reset");
                });
    }
    
    public async Task<Order> CreateOrderResilient(
        CreateOrderRequest request,
        string idempotencyKey)
    {
        // 组合策略：先经过断路器，再重试
        var combinedPolicy = Policy.WrapAsync(_retryPolicy, _circuitBreakerPolicy);
        
        return await combinedPolicy.ExecuteAsync(async () =>
        {
            var httpRequest = new HttpRequestMessage(HttpMethod.Post, "/api/orders");
            httpRequest.Headers.Add("Idempotency-Key", idempotencyKey);
            httpRequest.Content = JsonContent.Create(request);
            
            var response = await _httpClient.SendAsync(httpRequest);
            response.EnsureSuccessStatusCode();
            
            return await response.Content.ReadFromJsonAsync<Order>();
        });
    }
}

服务端超时处理 ​

1. 设置合理的超时时间 ​

csharp
// Program.cs
builder.Services.AddControllers(options =>
{
    // 全局请求超时
    options.Filters.Add(new RequestTimeoutFilter 
    { 
        TimeoutSeconds = 30 
    });
});

// 或者在控制器级别设置
[ApiController]
[Route("api/[controller]")]
public class OrdersController : ControllerBase
{
    [HttpPost]
    [RequestTimeout(60)] // 此接口允许 60 秒
    public async Task<ActionResult<Order>> CreateOrder([FromBody] CreateOrderRequest request)
    {
        // 长时间运行的操作
        var order = await _orderService.CreateOrderAsync(request);
        return CreatedAtAction(nameof(GetOrder), new { id = order.Id }, order);
    }
}

2. 异步取消支持 ​

csharp
public class OrderService
{
    public async Task<Order> CreateOrderAsync(
        CreateOrderRequest request,
        CancellationToken cancellationToken = default)
    {
        // 验证请求
        ValidateRequest(request);
        
        // 数据库操作（支持取消）
        var order = new Order { /* ... */ };
        _dbContext.Orders.Add(order);
        await _dbContext.SaveChangesAsync(cancellationToken);
        
        // 发送通知（支持取消）
        await _notificationService.SendAsync(
            order.UserId, 
            "Order created",
            cancellationToken);
        
        return order;
    }
}

// 控制器中使用
[HttpPost]
public async Task<ActionResult<Order>> CreateOrder(
    [FromBody] CreateOrderRequest request,
    CancellationToken cancellationToken)
{
    try
    {
        var order = await _orderService.CreateOrderAsync(request, cancellationToken);
        return CreatedAtAction(nameof(GetOrder), new { id = order.Id }, order);
    }
    catch (OperationCanceledException)
    {
        // 客户端取消了请求
        return StatusCode(499, "Client closed connection");
    }
}

监控与告警 ​

1. 记录超时和重试 ​

csharp
public class RetryMetrics
{
    private readonly Counter<long> _timeoutCount;
    private readonly Counter<long> _retryCount;
    private readonly Histogram<double> _retryDuration;
    
    public void RecordTimeout(string endpoint)
    {
        _timeoutCount.Add(1, new KeyValuePair<string, object?>("endpoint", endpoint));
    }
    
    public void RecordRetry(string endpoint, int attemptNumber, double durationMs)
    {
        _retryCount.Add(1, new KeyValuePair<string, object?>("endpoint", endpoint));
        _retryDuration.Record(durationMs, 
            new KeyValuePair<string, object?>("attempt", attemptNumber));
    }
}

public class MonitoringHandler : DelegatingHandler
{
    private readonly RetryMetrics _metrics;
    private readonly ILogger<MonitoringHandler> _logger;
    
    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request,
        CancellationToken cancellationToken)
    {
        var stopwatch = Stopwatch.StartNew();
        var endpoint = $"{request.Method} {request.RequestUri?.PathAndQuery}";
        
        try
        {
            var response = await base.SendAsync(request, cancellationToken);
            stopwatch.Stop();
            
            // 记录超时
            if (response.StatusCode == HttpStatusCode.RequestTimeout)
            {
                _metrics.RecordTimeout(endpoint);
                _logger.LogWarning("Request timeout: {Endpoint}, Duration: {Ms}ms", 
                    endpoint, stopwatch.ElapsedMilliseconds);
            }
            
            return response;
        }
        catch (TaskCanceledException ex) when (!cancellationToken.IsCancellationRequested)
        {
            stopwatch.Stop();
            _metrics.RecordTimeout(endpoint);
            _logger.LogError(ex, "Request timeout: {Endpoint}, Duration: {Ms}ms", 
                endpoint, stopwatch.ElapsedMilliseconds);
            throw;
        }
    }
}

2. Prometheus 告警规则 ​

yaml
groups:
  - name: timeout_alerts
    rules:
      - alert: HighTimeoutRate
        expr: rate(http_request_timeout_total[5m]) > 0.05
        for: 5m
        annotations:
          summary: "High HTTP timeout rate"
          description: "More than 5% of requests are timing out"
      
      - alert: ExcessiveRetries
        expr: rate(http_retry_total[5m]) > 100
        for: 5m
        annotations:
          summary: "Excessive HTTP retries"
          description: "More than 100 retries per second"

实际案例分析 ​

案例 1：支付超时导致重复扣款 ​

问题：

用户发起支付 -> 银行扣款成功 -> 响应在网络中丢失 -> 客户端超时重试 -> 再次扣款

解决方案：

csharp
public class PaymentService
{
    public async Task<PaymentResult> ProcessPayment(
        PaymentRequest request,
        string idempotencyKey)
    {
        // 检查是否已处理
        var existingPayment = await _payments
            .FirstOrDefaultAsync(p => p.IdempotencyKey == idempotencyKey);
        
        if (existingPayment != null)
        {
            return new PaymentResult 
            { 
                Success = true,
                PaymentId = existingPayment.Id 
            };
        }
        
        // 调用银行 API（传递相同的幂等键）
        var bankResponse = await _bankGateway.ChargeAsync(new BankChargeRequest
        {
            Amount = request.Amount,
            IdempotencyKey = idempotencyKey  // 关键：银行也支持幂等
        });
        
        // 保存结果
        var payment = new Payment
        {
            IdempotencyKey = idempotencyKey,
            TransactionId = bankResponse.TransactionId,
            Status = "success"
        };
        
        await _payments.AddAsync(payment);
        await _dbContext.SaveChangesAsync();
        
        return new PaymentResult { Success = true, PaymentId = payment.Id };
    }
}

案例 2：订单创建超时 ​

问题：

创建订单 -> 插入数据库成功 -> 返回前超时 -> 客户端重试 -> 重复订单

解决方案：

sql
-- PostgreSQL: 使用唯一约束
ALTER TABLE orders 
ADD CONSTRAINT uk_orders_idempotency_key 
UNIQUE (idempotency_key);
csharp
try
{
    await _dbContext.SaveChangesAsync();
}
catch (DbUpdateException ex) when (IsUniqueViolation(ex))
{
    // 订单已存在，返回已有订单
    var existingOrder = await _dbContext.Orders
        .FirstOrDefaultAsync(o => o.IdempotencyKey == idempotencyKey);
    
    return existingOrder;
}

最佳实践总结 ​

✅ DO ​

  1. 始终使用幂等键：所有可能重试的操作都必须有幂等键
  2. 实现指数退避：避免立即重试加重服务器负担
  3. 设置合理超时：根据操作类型设置不同的超时时间
  4. 支持取消令牌：及时释放资源
  5. 记录重试日志：便于问题排查
  6. 监控超时率：及时发现系统问题

❌ DON'T ​

  1. 不要无限重试：设置最大重试次数
  2. 不要固定间隔重试：使用指数退避 + 抖动
  3. 不要忽略超时：超时不等于失败，可能已成功
  4. 不要在重试时改变参数：保持请求一致性
  5. 不要假设第一次失败：先检查状态再重试

总结 ​

网络超时重试是分布式系统的常态，关键在于：

  1. 识别：区分超时和真正的失败
  2. 防护：使用幂等性保证安全重试
  3. 策略：采用指数退避和断路器
  4. 监控：及时发现异常模式

通过合理的超时配置和幂等性设计，可以构建出高度可靠的分布式系统。

Released under the MIT License.

Copyright © 2026 Jarod Chen