An open cloud-native web console improving on the life of SAP Applications administrators.

Trento is a city on the Adige River in Trentino-Alto Adige/Südtirol in Italy. [...] It is one of the nation's wealthiest and most prosperous cities, [...] often ranking highly among Italian cities for quality of life, standard of living, and business and job opportunities. (source)

This project is a reboot of the "SUSE Console for SAP Applications", also known as the Blue Horizon for SAP prototype, which is focused on automated infrastructure deployment and provisioning for SAP Applications.

As opposed to that first iteration, this new one will focus more on operations of existing clusters, rather than deploying new one.

• Features
• Introduction
• Installation
  • Requirements
  • Quick-Start installation
    • Trento Server installation
    • Trento Agent installation
      • Starting Trento Agent service
  • Manual installation
    • Pre-built binaries
    • Compile from source
  • Docker images
  • RPM Packages
  • Helm chart
    • Install K3S
    • Install Helm and chart dependencies
    • Install the Trento Server Helm chart
    • Other Helm chart usage examples:
  • Manually running Trento
    • Consul
      • Server Consul Agent
      • Client Consul Agent
    • Trento Agents
    • Trento Runner
      • Setting up and starting ARA
      • Starting the Trento Runner
    • Trento Web UI
• Configuration
• Development
  • Helm development chart
  • Build system
  • Development dependencies
  • Docker
  • SAPControl web service
• Support
• Contributing
• License

• Automated discovery of SAP HANA HA clusters;
• SAP Systems and Instances overview;
• Configuration validation for Pacemaker, Corosync, SBD, SAPHanaSR and other generic SUSE Linux Enterprise for SAP Application OS settings (a.k.a. the HA Config Checks);
• Specific configuration audits for SAP HANA Scale-Up Performance-Optimized scenarios deployed on MS Azure cloud.

Trento is a comprehensive monitoring solution made by two main components, the Trento Server and the Trento Agent.

The Trento Server is an independent, cloud-native, distributed system and should run on dedicated infrastructure resources. It is in turn composed by the following sub-systems:

• The trento web application;
• The trento runner worker;
• A Consul data-plane;
• An ARA service.

The Trento Agent is a single background process (trento agent) running in each host of the target infrastructure the user desires to monitor.

Please note that, except for the third-party ones like Consul and ARA, all the components are embedded within one single trento binary.

See the architecture document for additional details.

Being the project in development, all of the above might be subject to change!

The Trento Server is intended to run in many ways, depending on users' already existing infrastructure, but it's designed to be cloud-native and OS agnostic. As such, our default installation method provisions a minimal, single node, K3S Kubernetes cluster to run its various components in Linux containers.
The suggested physical resources for running all the Trento Server components are 2GB of RAM and 2 CPU cores. The Trento Server needs to reach the target infrastructure.

The Trento Agent component, on the other hand, needs to interact with a number of low-level system components which are part of the SUSE Linux Enterprise Server for SAP Applications Linux distribution. These could in theory also be installed and configured on other distributions providing the same functionalities, but this use case is not within the scope of the active development. The resource footprint of the Trento Agent should not impact the performance of the host it runs on.

Installation scripts are provided to automatically install and update the latest version of Trento. Please follow the instructions in the given order.

The script installs a single node K3s cluster and uses the trento-server Helm chart to bootstrap a complete Trento server component.

You can curl | bash if you want to live on the edge.

curl -sfL https://raw.githubusercontent.com/trento-project/trento/main/install-server.sh | bash

Or you can fetch the script, and then execute it manually.

curl -O https://raw.githubusercontent.com/trento-project/trento/main/install-server.sh
chmod 700 install-server.sh
sudo ./install-server.sh

The script will ask you for a private key that is used by the runner service to perform checks in the agent hosts via ssh.

Note: if a Trento server is already installed in the host, it will be updated.

Please refer to the Trento Runner section for more information. Please refer to the Helm chart section for more information about the Helm chart.

After the server installation, you might want to install Trento agents in a running cluster. Please add the public key to the ssh authorized_keys to enable the runner checks in the agent host, as mentioned in the server installation above.

As for the server component an installation script is provided, you can curl | bash it if you want to live on the edge.

curl -sfL https://raw.githubusercontent.com/trento-project/trento/main/install-agent.sh | sudo bash

Or you can fetch the script, and then execute it manually.

curl -O https://raw.githubusercontent.com/trento-project/trento/main/install-agent.sh
chmod 700 install-agent.sh
sudo ./install-agent.sh

The script will ask you for two IP addresses.

• agent bind IP: the private address to which the trento-agent should be bound for internal communications. This is an IP address that should be reachable by the other hosts, including the trento server. Note for the Pacemaker users: this IP address should not be a floating IP.

• trento server IP: the address where Trento server can be reached.

You can pass these arguments as flags or env variables too:

curl -sfL https://raw.githubusercontent.com/trento-project/trento/main/install-agent.sh | sudo bash -s - --agent-bind-ip=192.168.33.10 --server-ip=192.168.33.1

AGENT_BIND_IP=192.168.33.10 SERVER_IP=192.168.33.1 sudo ./install-agent.sh

The installation script does not start the agent automatically.

You can enable boot startup and launch it with systemd:

sudo systemctl enable --now trento-agent

Please, make sure the server is running before starting the agent.

That's it! You can now reach the Trento web UI and start using it.

Pre-built statically linked binaries are made available via GitHub releases.

You clone also clone and build it manually:

git clone https://github.com/trento-project/trento.git
cd trento
make build

See the section below to know more about the build dependencies.

T.B.D.

T.B.D.

The packaging/helm directory contains the Helm chart for installing Trento Server in a Kubernetes cluster.

If installing as root:

# curl -sfL https://get.k3s.io | sh

If installing as non-root user:

curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644

Export KUBECONFIG env variable:

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

Please refer to the K3S official documentation for more information about the installation.

Install Helm:

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

Please refer to the Helm official documentation for more information about the installation.

Add third-party Helm repositories:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

Install chart dependencies:

cd packaging/helm/trento-server/
helm dependency update

The Runner component of Trento server needs ssh access to the agent nodes to perform the checks. You need to pass a valid private key used for ssh authentication to the Helm chart, and it will be stored in the K3s cluster as a secret. Please refer to the Trento Runner section for more information.

Install the Trento Server chart:

helm install trento . --set-file trento-runner.privateKey=/your/path/id_rsa_runner

or perform a rolling update:

helm upgrade trento . --set-file trento-runner.privateKey=/your/path/id_rsa_runner

Now you can connect to the web server via http://localhost and point the agents to the cluster IP address.

Use a different container image (e.g. the rolling one):

helm install trento . --set trento-web.image.tag="rolling" --set trento-runner.image.tag="rolling" --set-file trento-runner.privateKey=id_rsa_runner

Use a different container registry:

helm install trento . --set trento-web.image.repository="ghcr.io/myrepo/trento-web" --set trento-runner.image.repository="ghcr.io/myrepo/trento-runner" --set-file trento-runner.privateKey=id_rsa_runner

Please refer to the the subcharts values.yaml for an advanced usage.

What follows are explicit instructions on how to run all the various components without any automated orchestration.

The Trento system needs leverages a Consul data-plane for service discovery and persistent data storage.

Consul processes are all called "agents", and they can run either in server mode or in client mode.

We have only tested version Consul version 1.9.x and, while it should work with any version implementing Consul Protocol version 3, we can´t make any guarantee in that regard.

To start a Consul agent in server mode:

mkdir consul.d
./consul agent -server -ui -bootstrap-expect=1 -client=0.0.0.0 -data-dir=consul-data -config-dir=consul.d

This will start Consul listening to connections from any IP address on the default network interface, using consul-data as directory to persist data, and consul.d to autoload configuration files from.

Each Trento Agent instance also needs a Consul agent in client mode, on each target node we want to connect Trento to.

You can start Consul in client mode as follows:

export SERVER_IP_ADDRESS=#your Consul server IP address here
mkdir consul.d
./consul agent -retry-join=$SERVER_IP_ADDRESS -bind='{{ GetInterfaceIP "eth0" }}' -data-dir=consul-agent-data -config-dir=consul.d

Since the client Consul Agent will most likely run on a machine with multiple IP addresses and/or network interfaces, you will need to specify one with the -bind flag.

Production deployments will require at least three instances of the Consul agent in server mode to ensure fault-tolerance. Be sure to check Consul's deployment guide.

While Consul provides a -dev flag to run a standalone, stateless server agent, Trento does not support this mode: it needs a persistent server even during development.

For development purposes, when running everything on a single host machine, no Client Consul Agent is required: the Server Agent exposes the same API and can be consumed by both trento web and trento agent.

Trento Agents are responsible for discovering SAP systems, HA clusters and some additional data. These Agents need to run in the same systems hosting the HA Cluster services, so running them in isolated environments (e.g. serverless, containers, etc.) makes little sense, as they won't be able as the discovery mechanisms will not be able to report any host information.

To start the trento agent:

./trento agent start

If the discovery loop is being executed too frequently, and this impacts the Web interface performance, the agent has the option to configure the discovery loop mechanism using the --discovery-period flag. Increasing this value improves the overall performance of the application

This feature is still under heavy development

Trento Agents publish discovery data to a Collector on Trento Server

./test/certs/ folder contains some dummy Server, Client and CA Certificates and Keys. Those are useful in order to test mTLS communication between the Agent and the DataCollector.

As of now mtls needs to explicitly be activated with --enable-mtls

$> ./trento web serve [...] --enable-mtls --cert /path/to/certs/server-cert.pem --key /path/to/certs/server-key.pem --ca /path/to/certs/ca-cert.pem

$> ./trento agent start [...] --enable-mtls --cert /path/to/certs/client-cert.pem --key /path/to/certs/client-key.pem --ca /path/to/certs/ca-cert.pem

The Trento Runner is a worker process responsible for driving automated configuration audits. It is based on Ansible and ARA. These 2 components (the Runner and ARA) can be executed in the same host as the Web UI, but it is not mandatory: they can be executed in any other host with network access to the Trento Agents.

The Runner itself and ARA can be even executed in different hosts too, as long as the network connection is available between them.

Find more information about how to create more Trento health checks here.

In order to start them, some packages must be installed and started. Here a quick go through:

# Install ARA with server dependencies
pip install 'ara[server]~=1.5.7'
# Setup ARA database
ara-manage migrate
# Start ARA server. This process can be started in background or in other shell terminal
ara-manage runserver ip:port

The installed ara version should be at least ara~=1.5.7

If the requests to ARA server fail with a message like the next one, it means that the server address must be allowed:

2021-09-02 07:13:48,715 ERROR django.security.DisallowedHost: Invalid HTTP_HOST header: '10.74.1.5:8000'. You may need to add '10.74.1.5' to ALLOWED_HOSTS.
2021-09-02 07:13:48,732 WARNING django.request: Bad Request: /api/

To fix it run:

export ARA_ALLOWED_HOSTS="['10.74.1.5']"
# Or allow all the addresses with
export ARA_ALLOWED_HOSTS=['*']

Independently where you decide to run ARA, the Runner needs the ansible and ara Python packages available locally:

pip install 'ansible~=4.6.0' 'ara~=1.5.7'

The installed ansible components versions should be at least ansible~=4.6.0 and ansible-core~=2.11.5

Once dependencies are in place, you can start the Runner itself:

./trento runner start --ara-server http://$ARA_IP:$ARA_PORT --api-host $WEB_IP --api-port $WEB_PORT -i 5
# If the connection to consul fails try set the CONSUL_HTTP_ADDR environment variable
CONSUL_HTTP_ADDR=$CONSUL_IP:$CONSUL_PORT ./trento runner start --ara-server http://$ARA_IP:$ARA_PORT --api-host $WEB_IP --api-port $WEB_PORT -i 5

Note: The Trento Runner component must have SSH access to all the agents via a password-less SSH key pair.

At this point, we can start the web application as follows:

./trento web serve
# If ARA server is not running in the same machine set the ara-addr flag
./trento web serve --ara-addr araIP:port

Please consult the help CLI command for more insights on the various options.

Trento can be run with a config file in replacement of command-line arguments.

Configuration, if not otherwise specified by the --config=/path/to/config.yaml option, would be searched in following locations:

Note that order represents priority

• /etc/trento/ <-- first location looked
• /usr/etc/trento/ <-- fallback here if config not found in previous location
• ~/.config/trento/ aka user's home <-- fallback here

yaml is the only supported format at the moment.

Each component of trento supports its own configuration, so the expected config files in the chosen location must be called after the component it is supporting.

So supported names are (agent|web|runner).yaml

Example locations:

/etc/trento/agent.yaml

/etc/trento/web.yaml

/etc/trento/runner.yaml

or

/usr/etc/trento/agent.yaml

/usr/etc/trento/web.yaml

/usr/etc/trento/runner.yaml

Note: runner still not supported for now

# /etc/trento/agent.yaml

enable-mtls: true
cert: /path/to/certs/client-cert.pem
key: /path/to/certs/client-key.pem
ca: /path/to/certs/ca-cert.pem
collector-host: localhost
collector-port: 8443

# /etc/trento/web.yaml

db-port: 5432

collector-port: 8443
enable-mtls: true
cert: /path/to/certs/server-cert.pem
key: /path/to/certs/server-key.pem
ca: /path/to/certs/ca-cert.pem

All of the options supported by the command line and configuration file can be provided as environment variables as well.

The rule is: get the option name eg. enable-mtls, replace dashes - with underscores _, make it uppercase and add a TRENTO_ prefix.

Examples:

enable-mtls -> TRENTO_ENABLE_MTLS=true ./trento agent start

collector-host -> TRENTO_COLLECTOR_HOST=localhost ./trento web serve

cert -> TRENTO_CERT=/path/to/certs/server-cert.pem ./trento web serve

A development Helm chart is available at ./hack/helm/trento-dev. The chart is based on the official Helm chart package and overrides certain values to provide a development environment.

# Update dependencies in the official Helm chart
cd ./packaging/helm/trento-server
helm dep update

# Install the development Helm chart
cd ./hack/helm/trento-dev
helm dep update
helm install trento-dev .

Since integration tests require a running PostgreSQL instance, please make sure the chart is installed prior to running the integration tests. The PostgreSQL instance will be accessible at port localhost:5432.

If you want to skip database integration tests, you can use a dedicated environment variable as follows:

TRENTO_DB_INTEGRATION_TESTS=false make test

We use GNU Make as a task manager; here are some common targets:

make # clean, test and build everything

make clean # removes any build artifact
make test # executes all the tests
make fmt # fixes code formatting
make web-assets # invokes the frontend build scripts
make generate # refresh automatically generated code (e.g. static Go mocks)

Feel free to peek at the Makefile to know more.

Additionally, for the development we use mockery for the generate target, which in turn is required for the test target. You can install it with go install github.com/vektra/mockery/v2.

Be sure to add the mockery binary to your $PATH environment variable so that make can find it. That usually comes with configuring $GOPATH, $GOBIN, and adding the latter to your $PATH.

The Dockerfile will automatically fetch all the required compile-time dependencies to build the binary and finally a container image with the fresh binary in it.

We use a multi-stage build with two main targets: trento-runner and trento-web. The latter is the default.

You can build the component like follows:

docker build --target trento-runner -t trento-runner .
docker build -t trento-web . # same as specifying --target trento-web

Please note that the trento agent component requires to be running on the OS (not inside a container) so, while it is technically possible to run trento agent commands in the container, it makes little sense because most of its internals require direct access to the host of the HA Cluster components.

The SAPControl web service soap client was generated by hooklift/gowsdl, then the methods and structs needed were cherry-picked and adapted. For reference, you can find the full, generated, web service code here.

Please only report bugs via GitHub issues; for any other inquiry or topic use GitHub discussion.

See CONTRIBUTING.md

Copyright 2021 SUSE LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.