Observabilité des Service Mesh

Guide complet des patterns d'observabilité pour Istio, Linkerd et déploiements de service mesh.

Quand utiliser cette compétence

• Configurer le tracing distribué entre services
• Implémenter les métriques et dashboards du service mesh
• Déboguer les problèmes de latence et d'erreurs
• Définir des SLOs pour la communication entre services
• Visualiser les dépendances entre services
• Dépanner les problèmes de connectivité du mesh

Concepts fondamentaux

1. Les trois piliers de l'observabilité

┌─────────────────────────────────────────────────────┐
│                 Observabilité                        │
├─────────────────┬─────────────────┬─────────────────┤
│    Métriques    │     Traces      │      Logs       │
│                 │                 │                 │
│ • Taux requête  │ • Contexte span │ • Logs accès    │
│ • Taux erreur   │ • Latence       │ • Détails erreur│
│ • Latence P50   │ • Dépendances   │ • Info debug    │
│ • Saturation    │ • Goulots       │ • Audit trail   │
└─────────────────┴─────────────────┴─────────────────┘

2. Signaux d'or pour le Mesh

Templates

Template 1 : Istio avec Prometheus & Grafana

# Installer Prometheus
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus
  namespace: istio-system
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
    scrape_configs:
      - job_name: 'istio-mesh'
        kubernetes_sd_configs:
          - role: endpoints
            namespaces:
              names:
                - istio-system
        relabel_configs:
          - source_labels: [__meta_kubernetes_service_name]
            action: keep
            regex: istio-telemetry
---
# ServiceMonitor pour Prometheus Operator
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: istio-mesh
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: istiod
  endpoints:
    - port: http-monitoring
      interval: 15s

Template 2 : Requêtes clés de métriques Istio

# Taux requête par service
sum(rate(istio_requests_total{reporter="destination"}[5m])) by (destination_service_name)

# Taux erreur (5xx)
sum(rate(istio_requests_total{reporter="destination", response_code=~"5.."}[5m]))
  / sum(rate(istio_requests_total{reporter="destination"}[5m])) * 100

# Latence P99
histogram_quantile(0.99,
  sum(rate(istio_request_duration_milliseconds_bucket{reporter="destination"}[5m]))
  by (le, destination_service_name))

# Connexions TCP
sum(istio_tcp_connections_opened_total{reporter="destination"}) by (destination_service_name)

# Taille requête
histogram_quantile(0.99,
  sum(rate(istio_request_bytes_bucket{reporter="destination"}[5m]))
  by (le, destination_service_name))

Template 3 : Jaeger Distributed Tracing

# Installation Jaeger pour Istio
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    enableTracing: true
    defaultConfig:
      tracing:
        sampling: 100.0 # 100% en dev, inférieur en prod
        zipkin:
          address: jaeger-collector.istio-system:9411
---
# Déploiement Jaeger
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: jaeger
  template:
    metadata:
      labels:
        app: jaeger
    spec:
      containers:
        - name: jaeger
          image: jaegertracing/all-in-one:1.50
          ports:
            - containerPort: 5775 # UDP
            - containerPort: 6831 # Thrift
            - containerPort: 6832 # Thrift
            - containerPort: 5778 # Config
            - containerPort: 16686 # UI
            - containerPort: 14268 # HTTP
            - containerPort: 14250 # gRPC
            - containerPort: 9411 # Zipkin
          env:
            - name: COLLECTOR_ZIPKIN_HOST_PORT
              value: ":9411"

Template 4 : Linkerd Viz Dashboard

# Installer l'extension Linkerd viz
linkerd viz install | kubectl apply -f -

# Accéder au dashboard
linkerd viz dashboard

# Commandes CLI pour l'observabilité
# Top requêtes
linkerd viz top deploy/my-app

# Métriques par route
linkerd viz routes deploy/my-app --to deploy/backend

# Inspection du trafic live
linkerd viz tap deploy/my-app --to deploy/backend

# Edges service (dépendances)
linkerd viz edges deployment -n my-namespace

Template 5 : Grafana Dashboard JSON

{
  "dashboard": {
    "title": "Service Mesh Overview",
    "panels": [
      {
        "title": "Request Rate",
        "type": "graph",
        "targets": [
          {
            "expr": "sum(rate(istio_requests_total{reporter=\"destination\"}[5m])) by (destination_service_name)",
            "legendFormat": "{{destination_service_name}}"
          }
        ]
      },
      {
        "title": "Error Rate",
        "type": "gauge",
        "targets": [
          {
            "expr": "sum(rate(istio_requests_total{response_code=~\"5..\"}[5m])) / sum(rate(istio_requests_total[5m])) * 100"
          }
        ],
        "fieldConfig": {
          "defaults": {
            "thresholds": {
              "steps": [
                { "value": 0, "color": "green" },
                { "value": 1, "color": "yellow" },
                { "value": 5, "color": "red" }
              ]
            }
          }
        }
      },
      {
        "title": "P99 Latency",
        "type": "graph",
        "targets": [
          {
            "expr": "histogram_quantile(0.99, sum(rate(istio_request_duration_milliseconds_bucket{reporter=\"destination\"}[5m])) by (le, destination_service_name))",
            "legendFormat": "{{destination_service_name}}"
          }
        ]
      },
      {
        "title": "Service Topology",
        "type": "nodeGraph",
        "targets": [
          {
            "expr": "sum(rate(istio_requests_total{reporter=\"destination\"}[5m])) by (source_workload, destination_service_name)"
          }
        ]
      }
    ]
  }
}

Template 6 : Kiali Service Mesh Visualization

# Installation Kiali
apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali
  namespace: istio-system
spec:
  auth:
    strategy: anonymous # ou openid, token
  deployment:
    accessible_namespaces:
      - "**"
  external_services:
    prometheus:
      url: http://prometheus.istio-system:9090
    tracing:
      url: http://jaeger-query.istio-system:16686
    grafana:
      url: http://grafana.istio-system:3000

Template 7 : Intégration OpenTelemetry

# Collecteur OpenTelemetry pour mesh
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
      zipkin:
        endpoint: 0.0.0.0:9411

    processors:
      batch:
        timeout: 10s

    exporters:
      jaeger:
        endpoint: jaeger-collector:14250
        tls:
          insecure: true
      prometheus:
        endpoint: 0.0.0.0:8889

    service:
      pipelines:
        traces:
          receivers: [otlp, zipkin]
          processors: [batch]
          exporters: [jaeger]
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [prometheus]
---
# Istio Telemetry v2 avec OTel
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  tracing:
    - providers:
        - name: otel
      randomSamplingPercentage: 10

Règles d'alerte

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: mesh-alerts
  namespace: istio-system
spec:
  groups:
    - name: mesh.rules
      rules:
        - alert: HighErrorRate
          expr: |
            sum(rate(istio_requests_total{response_code=~"5.."}[5m])) by (destination_service_name)
            / sum(rate(istio_requests_total[5m])) by (destination_service_name) > 0.05
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Taux d'erreur élevé pour {{ $labels.destination_service_name }}"

        - alert: HighLatency
          expr: |
            histogram_quantile(0.99, sum(rate(istio_request_duration_milliseconds_bucket[5m]))
            by (le, destination_service_name)) > 1000
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Latence P99 élevée pour {{ $labels.destination_service_name }}"

        - alert: MeshCertExpiring
          expr: |
            (certmanager_certificate_expiration_timestamp_seconds - time()) / 86400 < 7
          labels:
            severity: warning
          annotations:
            summary: "Certificat mesh expirant dans moins de 7 jours"

Bonnes pratiques

À faire

• Échantillonner de manière appropriée - 100% en dev, 1-10% en prod
• Utiliser le contexte de trace - Propager les en-têtes de manière cohérente
• Configurer des alertes - Pour les signaux d'or
• Corréler métriques/traces - Utiliser les exemplaires
• Conserver stratégiquement - Niveaux de stockage chaud/froid

À ne pas faire

• Ne pas sur-échantillonner - Les coûts de stockage s'accumulent
• Ne pas ignorer la cardinalité - Limiter les valeurs de labels
• Ne pas négliger les dashboards - Visualiser les dépendances
• Ne pas oublier les coûts - Surveiller les coûts d'observabilité