SUSE® Observability

Introduction

SUSE® Observability, formerly known as StackState can be used for Observability of your Kubernetes clusters and their workloads.

The installation of SUSE® Observability, the SUSE® Observability UI extension and the SUSE® Observability Agents takes about 30 minutes in total.

Getting help

For support please file a support case in SUSE Customer Center (SCC).

Prerequisites

License key

A license key for SUSE® Observability server can be obtained via the SUSE Customer Center in the Subscription tab and is be shown as "SUSE® Observability" Registration Code. This license is valid until the end of your Rancher Prime subscription.

Requirements

To install SUSE® Observability, ensure that the cluster has enough CPU and memory capacity.

For information on requirements, see System requirements and sizing.

Storage

SUSE® Observability uses persistent volume claims for the services that need to store data. The default storage class for the cluster will be used for all services unless this is overridden by values specified on the command line or in a values.yaml file. All services come with a pre-configured volume size that should be good to get you started, but can be customized later using variables as required.

SUSE® Observability requires the underlying storage to be based on flash memory (SSD) or similar in performance.

For production environments, NFS is not recommended and supported for storage provisioning in SUSE® Observability due to the potential risk of data corruption.

For our different installation profiles, the following are the defaulted storage requirements:

trial 10 non-HA 20 non-HA 50 non-HA 100 non-HA 150 HA 250 HA 500 HA 4000 HA

Retention (days)

3

30

30

30

30

30

30

30

30

Storage requirement

154Gi

335Gi

375Gi

475Gi

525Gi

2595Gi

2695Gi

3595Gi

6995Gi

For more details on the defaults used, see the page Configure storage.

Helm

SUSE® Observability is installed through Helm, which needs to be installed with a minimum version of 3.13.1.

The different components

SUSE® Observability Server

This is the on-prem hosted server part of the installation. It contains a set of services to store observability data:

  • Topology (StackGraph)

  • Metrics (VictoriaMetrics)

  • Traces (ClickHouse)

  • Logs (ElasticSearch)

Next to this, it contains a set of services for all the observability tasks. e.g. Notifications, State management, Monitoring, etc.

SUSE® Observability Agent

The lightweight SUSE® Observability agent is installed on your downstream worker nodes. It collects and reports metrics, events, traces and logs, and it provides real-time observability and insights, enabling proactive monitoring and troubleshooting of your IT environment.

The SUSE® Observability version of the Agent also uses eBPF as a lightweight way to monitor all your workloads and their communication. It also decodes the RED (Rate, Errors and Duration) signals for most of the common L7 protocols like TCP, HTTP, TLS, Redis, etc.

Rancher Prime - Observability UI extension

This is an UI extension to Rancher Manager that integrates the health signals observed by SUSE® Observability. It gives direct access to the health of any resource and a link to SUSE® Observability’s UI for further investigation.

Installing SUSE® Observability server and SUSE® Observability

For information on installing SUSE® Observability along with the server, see Installing SUSE Observability.

Accessing SUSE® Observability

The SUSE® Observability Helm chart has support for creating an Ingress resource to make SUSE® Observability accessible outside of the cluster. Follow these instructions to set that up when you have an ingress controller in the cluster. Make sure that the resulting URL uses TLS with a valid, not self-signed, certificate.

If you prefer to use a load balancer instead of ingress, expose the suse-observability-router service. The URL for the loadbalancer needs to use a valid, not self-signed, TLS certificate.

Installing UI Extensions

For information on installing the SUSE® Observability UI, see Installing UI extensions.

Installing the SUSE® Observability Agent

For information on installing the SUSE® Observability agent, see Quick start guide.

Required privileges

Two of the SUSE® Observability Agent workloads need privileges that the restricted Pod Security Standard does not allow. The other Agent workloads — the cluster agent, the checks agent, the Kubernetes RBAC agent, the HTTP header injector and the OpenTelemetry components — need none of them.

Node agent

The node agent is a DaemonSet running the node agent and process agent containers. It requires the following system privileges:

  1. hostPID: true: This privilege is required to associate process identifiers (PIDs) with their corresponding control groups (cgroups). This association is essential for accurately mapping processes to their respective containers.

  2. hostNetwork: true (Optional): By default, the node agent runs with hostNetwork: true to facilitate scraping of open metrics data from all configured pods on the node without requiring additional network policies. If disabled, appropriate network policies must be defined to ensure the agent can access the necessary endpoints.

  3. securityContext.privileged: true on the process agent container: This elevated privilege is required for several critical functions. Primarily, it permits the agent to inject eBPF (extended Berkeley Packet Filter) programs into each network namespace for monitoring purposes. It is also necessary for reading the connection tracking (conntrack) tables across all network namespaces. While this list is not exhaustive, future development aims to replace this broad privilege with more granular Linux capabilities where feasible.

Furthermore, the node agent requires container runtime sockets to be mounted within its pod. This configuration is essential as it facilitates direct communication with the container runtime daemons, which is a prerequisite for scraping metrics and metadata from all containers on the host system.

Logs agent

The logs agent, enabled with logsAgent.enabled, is a DaemonSet that collects container logs from the node filesystem. It mounts the host paths /var/log and /var/lib/docker/containers, and its container sets securityContext.seLinuxOptions.type to spc_t so that it can read those paths on SELinux-enforcing nodes. It does not need hostPID, hostNetwork or privileged.

Restrictive Pod Security policies

This section applies to any cluster that enforces the restricted Pod Security Standard by default, including:

Every container in the SUSE® Observability and SUSE® Observability Agent Helm charts sets the following securityContext values, so that no namespace exemption is needed for them:

  1. securityContext.capabilities.drop is ["ALL"]

  2. securityContext.seccompProfile.type is "RuntimeDefault"

  3. securityContext.runAsNonRoot is true

  4. securityContext.allowPrivilegeEscalation is false

Exceptions

Three containers cannot satisfy the restricted profile.

Container Chart What to do

Elasticsearch configure-sysctl init container

SUSE® Observability

Disable it with elasticsearch.sysctlInitContainer.enabled: false and raise vm.max_map_count on the nodes yourself. See Elasticsearch Required Permissions.

Node agent DaemonSet (node-agent and process-agent containers)

SUSE® Observability Agent

Install the Agent into a namespace that is exempt from the policy. See Node agent.

Logs agent DaemonSet (logs-agent container)

SUSE® Observability Agent

Install the Agent into a namespace that is exempt from the policy, or leave logsAgent.enabled at false. See Logs agent.

Because of the node agent, the recommended approach is to install the SUSE® Observability Agent into a namespace that you exempt from the restrictive policy, alongside the other required Rancher namespaces. The SUSE® Observability chart itself needs no exemption once the Elasticsearch init container is disabled.

A few optional init containers fix up data volume ownership by running chown as root, so they cannot satisfy the restricted profile either. They are all disabled by default, and you only need to consider them if you switch one on to repair volume permissions:

  • clickhouse.volumePermissions.enabled

  • kafka.volumePermissions.enabled

  • zookeeper.volumePermissions.enabled

  • hbase.hdfs.volumePermissions.enabled together with hbase.hdfs.volumePermissions.securityContext.enabled

The Agent chart’s all.fullPrivilegesMode.enabled option is a debugging aid: it replaces the restricted securityContext on the agent containers with a privileged one. Leave it at false on any cluster that enforces a restrictive policy.

Single Sign On

To enable Single sign-on with your own authentication provider please see here.

Frequently asked questions & Observations:

  1. Is it mandatory to install a SUSE® Observability agent before proceeding with adding the UI extension?

    • No this is not mandatory, the UI extension can be installed independent.

  2. Is it mandatory to install SUSE® Observability Server before we proceed with UI extensions?

    • Yes this is mandatory since you need to provide a SUSE® Observability endpoint in the configuration

  3. Can we install SUSE® Observability on a local cluster or on a downstream cluster?

    • Both options are possible.

  4. To monitor the downstream clusters, should we install the SUSE® Observability agent from the app store or add a new instance from the SUSE® Observability UI?

    • Both options are possible depending on users preference.

Open Issues

  1. When you uninstall and reinstall the UI extensions for Observability, we noticed that service token is not deleted and is reused upon reinstallation. Whenever we uninstall the extensions, service token should be removed.

    • This information should be deleted when the UI extensions are uninstalled.

  2. After the extensions are installed, the SUSE® Observability UI opens in the same tab as the Rancher UI.

    • You can use shift-click to open in a new tab, this will become the default behaviour

  3. Be aware upgrading or downgrading from HA to NON-HA and vice-versa is not yet supported.

Troubleshooting

For any queries regarding the installation of the UI extension for Observability, see Extension troubleshooting guide.