Kubernetes Telemetry

Kubernetes Telemetry#

Overview#

To deploy OpenTelemetry services for Graphistry in a Kubernetes environment, you will need to configure the system using Helm values. For comprehensive documentation on deploying Graphistry with Helm, refer to the official documentation at Graphistry Helm Documentation. Additionally, you can explore the open-source Helm project for Graphistry on GitHub at Graphistry Helm GitHub.

Telemetry Deployment Modes#

Graphistry services export telemetry data (metrics and traces) using the OpenTelemetry standard. In Kubernetes, the telemetry data is pushed to the OpenTelemetry Collector (otel-collector), which forwards it to observability tools such as Prometheus, Jaeger, Grafana Cloud, etc.

Kubernetes supports two primary modes of telemetry deployment, similar to Docker Compose:

Forwarding to External Services (Cloud Mode)#

When the Helm value telemetryStack.OTEL_CLOUD_MODE is true, telemetry data is forwarded to external services like Grafana Cloud, similar to Docker Compose’s Forwarding to External Services mode.

Using Packaged Observability Tools#

When the Helm value telemetryStack.OTEL_CLOUD_MODE is false, the stack bundled with Graphistry (Prometheus, Jaeger, Grafana) is deployed, and telemetry data is exported to these tools, similar to Docker Compose’s Using Packaged Observability Tools mode.

Hybrid Mode#

You can also configure a Hybrid Mode, combining both local tools and external services. This requires custom Helm chart adjustments to forward data to both local and external observability services. See Docker Compose’s Hybrid Mode for more information.

Prerequisites#

Before deploying OpenTelemetry services for Graphistry on Kubernetes, ensure you have the following prerequisites in place:

  1. Kubernetes Cluster: You must have access to a running Kubernetes cluster.

  2. Helm: Helm is the package manager for Kubernetes that simplifies the deployment and management of applications.

  3. Graphistry Helm Project: You must have the graphistry-helm project cloned or downloaded to your local machine. This project contains the necessary Helm charts and configurations for deploying Graphistry services with Kubernetes. You can find the project and instructions in the official Graphistry Helm GitHub repository.

  4. Access to Required Resources: Ensure you have the necessary permissions to deploy applications to the Kubernetes cluster. You may need appropriate access rights to the cloud provider’s Kubernetes resources or administrative permissions for your self-hosted Kubernetes environment.

Helm Values for OpenTelemetry in Kubernetes#

To deploy OpenTelemetry for Graphistry in a Kubernetes environment, you’ll need to configure the Helm deployment with specific values. These values are typically defined in a values.yaml file, which will replace the Docker Compose configuration in your setup.

The following is an example of the configuration you would include in your values.yaml file to deploy OpenTelemetry services within Kubernetes:

global:  ## global settings for all charts
  ENABLE_OPEN_TELEMETRY: false

  # Graphistry Telemetry values and environment variables for observability tools
  # can be set like helm upgrade -i chart_name --name release_name \
  #--set stENVPublic.LOG_LEVEL="FOO"
  # Telemetry documentation:
  # https://github.com/graphistry/graphistry-cli/blob/master/docs/tools/telemetry.md#kubernetes-deployment
  telemetryStack:
    OTEL_CLOUD_MODE: false   # false: deploy our stack: jaeger, prometheus, grafana etc.; else fill OTEL_COLLECTOR_OTLP_HTTP_ENDPOINT and credentials bellow
    openTelemetryCollector:
      image: "otel/opentelemetry-collector-contrib:0.87.0"
      # Settings for cloud mode (when OTEL_CLOUD_MODE: true)
      OTEL_COLLECTOR_OTLP_HTTP_ENDPOINT: ""   # e.g. Grafana OTLP HTTP endpoint for Graphistry Hub https://otlp-gateway-prod-us-east-0.grafana.net/otlp
      OTEL_COLLECTOR_OTLP_USERNAME: ""   # e.g. Grafana Cloud Instance ID for OTLP
      OTEL_COLLECTOR_OTLP_PASSWORD: ""   # e.g. Grafana Cloud API Token for OTLP
      # Settings for cluster mode (when graphistry-helm.global.ENABLE_CLUSTER_MODE: true)
      LEADER_OTEL_EXPORTER_OTLP_ENDPOINT: "" # All followers's collectors will export to this leader collector, and from there the telemetry data will be exported to Grafana, Prometheus, Jaeger, etc; e.g. "otel-collector.graphistry1.svc.cluster.local:4317"

    grafana:
      image: "grafana/grafana:11.0.0"
      GF_SERVER_ROOT_URL: "/grafana"
      GF_SERVER_SERVE_FROM_SUB_PATH: "true"
    dcgmExporter:
      image: "nvcr.io/nvidia/k8s/dcgm-exporter:3.3.5-3.4.1-ubuntu22.04"
      DCGM_EXPORTER_CLOCK_EVENTS_COUNT_WINDOW_SIZE: 1000  # milliseconds
    jaeger:
      image: "jaegertracing/all-in-one:1.50.0"
      OTEL_EXPORTER_JAEGER_ENDPOINT: "jaeger:4317"
    nodeExporter:
      image: "prom/node-exporter:v1.8.2"
    prometheus:
      image: "prom/prometheus:v2.47.2"

Configuration Overview#

  1. global: This section in the values.yaml file is used to define values that are accessible across all charts within the parent-child hierarchy. Both the parent chart (e.g., charts/graphistry-helm) and its child charts (e.g., charts/graphistry-helm/charts/telemetry) can reference these global values using .Values.global.<value_name>, providing a unified configuration across the deployment.

  2. telemetryStack: This section defines environment variables that control the OpenTelemetry configuration in Kubernetes. These variables replicate the settings that were originally defined in the Docker Compose setup.

  3. global.ENABLE_OPEN_TELEMETRY: Set to true to enable the OpenTelemetry stack within the Kubernetes environment. This will ensure that telemetry data is collected and processed by the relevant tools in your stack.

  4. telemetryStack.OTEL_CLOUD_MODE:

  • When set to false, the internal observability stack (Jaeger, Prometheus, Grafana, NVIDIA DCGM Exporter and Node Exporter) is deployed locally within your Kubernetes cluster. So, setting it to false is similar to using packaged observability tools within the Kubernetes environment.

  • When set to true, telemetry data is forwarded to external services, such as Grafana Cloud or other OTLP-compatible services. So, setting this to true is equivalent to forwarding telemetry to external services.

  1. telemetryStack.openTelemetryCollector.OTEL_COLLECTOR_OTLP_HTTP_ENDPOINT, telemetryStack.openTelemetryCollector.OTEL_COLLECTOR_OTLP_USERNAME, and telemetryStack.openTelemetryCollector.OTEL_COLLECTOR_OTLP_PASSWORD: These fields are required only if OTEL_CLOUD_MODE is set to true. They provide the necessary connection details (such as the endpoint, username, and password) for forwarding telemetry data to external services like Grafana Cloud or other OTLP-compatible services.

  2. telemetryStack.openTelemetryCollector.LEADER_OTEL_EXPORTER_OTLP_ENDPOINT: This field is used by all follower collectors when global.ENABLE_CLUSTER_MODE is set to true. In this case, all follower collectors will export their telemetry data to the leader’s collector, which will then export the data to Grafana, Prometheus, Jaeger, etc. For example: "otel-collector.graphistry1.svc.cluster.local:4317". See the guide on Configuring Telemetry for a Graphistry Cluster on Kubernetes.

  3. telemetryStack.grafana.GF_SERVER_ROOT_URL and telemetryStack.grafana.GF_SERVER_SERVE_FROM_SUB_PATH: These settings are used to configure Grafana, especially when it’s deployed behind a reverse proxy or using an ingress controller.

  • telemetryStack.grafana.GF_SERVER_ROOT_URL defines the root URL for accessing Grafana (e.g., /grafana).

  • telemetryStack.grafana.GF_SERVER_SERVE_FROM_SUB_PATH should be set to true if Grafana is accessed from a sub-path (e.g., /grafana) behind a reverse proxy or ingress.

  1. telemetryStack.dcgmExporter.DCGM_EXPORTER_CLOCK_EVENTS_COUNT_WINDOW_SIZE: This environment variable controls the GPU metric sampling resolution for dcgm-exporter, which exports GPU telemetry to Prometheus. It defines the window size (in milliseconds) for counting clock events on the GPU.

  • A smaller value (e.g., 500) results in higher-resolution telemetry with more frequent GPU metric updates.

  • A larger value (e.g., 2000) reduces the data rate but lowers monitoring overhead. This setting applies regardless of OTEL_CLOUD_MODE and affects both local and cloud-based telemetry setups.

  1. telemetryStack.*.image: These values allow to change the image versions of the observability tools.

Caddyfile - reverse proxy set up#

In Kubernetes, you can customize the Caddy configuration to expose or route telemetry data to different observability endpoints, offering flexibility for your deployment. By default, the Kubernetes setup includes ingress configurations for Prometheus, Jaeger, and Grafana dashboards. However, if you need more control over the routing or wish to modify the reverse proxy settings, you can refer to the Docker Compose section for guidance on configuring Caddy. To modify the Caddy configuration in Kubernetes, such as on GKE (Google Kubernetes Engine), follow these steps:

  1. Edit the Caddy ConfigMap and update the configuration as needed.

  2. Delete the existing Caddy ConfigMap (kubectl delete configmap caddy-config -n graphistry).

  3. Update the Graphistry Helm chart to apply the new configuration.

  4. Delete the current Caddy pod to trigger a restart with the updated settings (kubectl delete $(kubectl get pods -n graphistry -o name | grep caddy-graphistry) -n graphistry).

  5. Verify that the new ConfigMap is created and applied to the new Caddy pod (kubectl get configmap caddy-config -n graphistry -o yaml).

Additionally, review the general and global values in the Graphistry chart, as some are related to the Caddy configuration.