trinodb / trino-gateway Goto Github PK

Blog post
linkedin
slack

Anything since after the rename to Trino from PrestoSQL should be compatible. We do not want to support anything older anyway.. users of those old versions can stick to the old Presto Gateway or upgrade.

An optional feature to support retrying another backend url (idk maybe to default one?) if a query is requested to gateway, but the backend url it routed dies and/or there is no response from that trino server.

Besides the executable JAR we should at least provide a container image to run Trino Gateway.

based on lyft version plus enhancements from Bloomberg

includes refactor of package space to io.trino.gateway

Add to release process as well

#67 (comment)

• How to install using the JAR
• How to install using the Docker container
• Requirements (external DB...)
• How to start/stop..
• Minimal configuration
• Required Java and OS and more

Problem: Currently, if the backends are not responding to healthcheck, trino-gateway will inactivate the backends. Since healthcheck only checks against active backends, inactive backends will never be turned to active again.

Solution: it is better to introduce healthy/pending/unhealthy states depending on the health of the backends. These states will be independent from the active/inactive states, so it ensures that unhealthy but active backends can still be checked and possibly turned active again.

Healthy state is defined as backends are healthy and ready to be served
Pending state is defined as when the backend is switched from inactive to active. It will wait until healthcheck returns success before turning backend to healthy
Unhealthy state is defined as backends are returning error or not responding to healthcheck. At this point, if the backends are still active, healthcheck will still check on these backends.

I use Form/Basic authentication

following:
presetUsers:
user1:
password:
privileges: "lb_admin, lb_user"
user2:
password:
privileges: "lb_api"

The authorized account and password pop up.
Input user and password ,401 Unauthorized will always appear.

my setting sample:
authentication:
defaultType: "form"

presetUsers:
user1:
password: "pass"
privileges: "lb_admin, lb_user, lb_api"

Lombok uses private API in Javac and that blocks the upgrade to Java 21. We should not need Lombok in general anyway.

resourceGroupId in /trino/selector/read/{resourceGroupId} is wrongly annotated with @QueryParam, when it's actually a path parameter:

@GET
@Path("/selector/read/{resourceGroupId}")
public Response readSelector(@QueryParam("resourceGroupId") String resourceGroupIdStr,
    ...

As a result, /trino/selector/read/X returns all selectors, and not just those in the X resource group.

https://github.com/trinodb/trino-gateway/actions/runs/7361226191/job/20038470483?pr=117

Error:  io.trino.gateway.ha.router.TestRoutingGroupSelector.testByRoutingRulesEngineFileChange -- Time elapsed: 0.007 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <etl2> but was: <etl>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
	at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1141)
	at io.trino.gateway.ha.router.TestRoutingGroupSelector.testByRoutingRulesEngineFileChange(TestRoutingGroupSelector.java:138)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

Follow along the Java 21 requirement update for Trino itself. Currently the codebase does not compile with Java 21. Probably needs more dependency updates as well.

After airbase pom PR is verified and release process tested.

@martint and @mosabua to collaborate

Maybe do a competition by submitting a logo as comment here.

Approved and agreed upon names are

• "Trino Gateway" overall
• "trino-gateway" for repository and similar technical usage
• "io.trino.gateway.*" as Java package, groupId and similar namespace usage

When connecting to a Trino webpage secured by OAuth from a browser, the OAuth handshake will fail if the Gateway forwards requests to different backends. This can be worked around by setting up a routing group with a single backend.

Hi, we originally had oidc to okta working with v3 of trino-gateway with the following config

authentication:
 defaultType: oauth
 oauth:
   issuer: ${ISSUER}
   clientId: ${TRINO_GATEWAY_OIDC_CLIENT_ID}
   clientSecret: ${TRINO_GATEWAY_OIDC_CLIENT_SECRET}
   tokenEndpoint: ${ISSUER}/oauth2/v1/token
   authorizationEndpoint: ${ISSUER}/oauth2/v1/authorize
   jwkEndpoint: ${ISSUER}/oauth2/v1/keys
   redirectUrl: https://${URL_OF_GATEWAY}/oidc/callback
   userIdField: email
   scopes:
     - openid
     - email
     - offline_access

After we updated to latest version 4 [latest commit of main on trino-gateway at the time] b6bbb14 when we hit the url of the gateway we had deployed it would redirect to http://localhost:8090/sso instead of http://${URL_OF_GATEWAY}/sso

I believe the update of jetty/dropwizard introduced this
434eec9

I tested that theory by checking out the latest v3 and applying all commits one at a time and that was the commit that introduced that behavior.

Hopefully theres just a config option I'm missing but I couldn't get the redirect fixed.

after dependencies like dropwizard are updated.

Needs architectural discussion how to best add this feature. Probably should talk to @electrum who added this to Trino.

@vishalya is interested on working on this feature.

The current DEBUG log level is too verbose.

after airbase PR is merged

Whenever Trino gateway runs a health check with the trino backend clusters, it returns an 'Unauthorized to fetch cluster stats' error and sets the status of the cluster to 'false'. When changing trino authentication to 'form' from 'oauth2' the health check works.

The TestSpecificDbResourceGroupsManager tests intermittently fail their assertions on the number of resource groups. This appears to potentially be a timing bug due to interaction between TestSpecificDbResourceGroupsManager and TestResourceGroupsManager.

For release tooling and much more.

The link in https://github.com/trinodb/trino-gateway/blob/main/docs/development.md#contributing return 404.

The MySQL driver requires enabling TLS1 and/or 1.1 in the JDK, or setting SSLMode=Disabled.

The project build from #14 needs to be verified to produce a usable and working binary. The build already passes.

This automatically also passes testing of the usage of Java 17.

This recent PR causes verify command to fail because maven-surefire-plugin was removed.
Is there a way to add jvm ops for tests?

I noticed that -DargLine is a way to add ops when using mvn cmd

• mvn clean verify -DargLine="$(tr -s '\r\n' ' ' < ".mvn/jvm.config")" works for ci
• https://stackoverflow.com/a/24892321

error log for main branch

https://github.com/trinodb/trino-gateway/actions/runs/6086764118/job/16513838158

Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make protected final java.lang.Class java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) throws java.lang.ClassFormatError accessible: module java.base does not "opens java.lang" to unnamed module @2c4d0347

When the header exceeds 8192, it will cause an exception
WARN [2023-11-29 09:24:27,248] org.eclipse.jetty.http.HttpParser: Header is too large 8193>8192

ProxyServer
In line 81, SSL is not enabled.
The following configuration will not be set

• outputBufferSize
• requestHeaderSize
• responseHeaderSize

in some cases various back-ends may support different credential sets.

it should be possible to override backendState per gateway back-end instance. this should be optional to allow fallback to the global setting.

It might be useful to add a Helm chart to manage Trino Gateway deployment on k8s.

To allow dependency plugin analyze to be enabled. For development you can set

<air.check.skip-dependency>false</air.check.skip-dependency>

in root pom and the use.

mvn install

Or you can use

mvn dependency:analyze

For now that produces LOTS of failures. PRs will remove problems one at a time.

Some improvements (also towards Java 21 support) will come from updating dependencies.

mvn versions:display-dependency-updates

At the end we will remove the skip setting in the root pom.

In new release notes page in docs

What's the load balancing scheme used by Trino gateway? And is there a way to change it?

Like Trino itself.

Currently the project uses an old version of Dropwizard. Airlift is a superior alternative that is actively maintained and also used in Trino itself.

Rather than upgrading Dropwizard it might be better to move to Airlift and then take advantage of aspects such as easy tarball creation, config system, opentelemety support and much more. This is the preferred approach by @martint.

Potential an initial upgrade of dropwizard to fix the worst issues might still be a good step.

we are planning to run multiple replicas in ocp, our environment requires full deployment to update the container in anyway which would require a redeployment any time we want to update the routing rules so if we could use the api calls and have them stored in postgres than it would make admin work for gateway much better.

Description

Trino Gateway routes queries randomly within a routing group and does not consider the number of already running and queued queries on a cluster within a routing group

Issue

It can happen that some queries take longer and and queue of the routed cluster keeps growing

Example scenario:

Assumption 3 Clusters in a routing group.

• Each cluster receive 50 queries
• cluster 1 and 2 process all the queries and have 0 queries remaining in queued queries
• cluster 3 has received some complex queries and processing needs longer and blocks other queries
• Gateway routes randomly again additional 50 queries to each cluster
• all clusters receive 50 queries to process again
• cluster 3 has 100 queries now, because the one or couple complex queries are in running state and blocking others.

Feature Request

Implement a solution which routes queries based on health status of the backend. It should consider the number of running and queued queries and route new queries to clusters with less load

Findings on current code

There is already code for this, but it is not used as of now. ClusterStateListenerModule does not have code for it.
https://github.com/trinodb/trino-gateway/blob/main/gateway-ha/src/main/java/io/trino/gateway/ha/router/TrinoQueueLengthRoutingTable.java
https://github.com/trinodb/trino-gateway/blob/main/gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/TrinoQueueLengthChecker.java

We should remove the following 2 lines from pom.xml:

Hello folks, we hit a bug using the new trino-gateway when opening query details from the gateway's UI displaying query history.

The trino-gateway uses backendUrl instead of externalUrl in the links it shows in the UI (see code) which means when opened from our internal users, they do not resolve properly since it's trying to use the internal endpoints that the gateway uses to communicate with the clusters.

The previous presto-gateway was using the gateway's URL for this (see code), which works so far in our case, and then it routes the UI calls to fetch the query details to the right coordinator cluster using the query mapping table/cache. At some point there is a request http://<gateway-url>/ui/api/query/<query_id>, which is matched by the request handler and gets sent to the right coordinator that has the query details.

Solution

I suggest one of the following:

1. Use the externalUrl of the backend cluster for the links in the query history, and then we will need to set those to something resolvable by the expected users of the admin UI.
2. Revert to using the gateway's URL and let the routing take care of finding the right backend cluster.

I believe the better solution is 2, so that the gateway handles all the query routing requests, same as it does for the actual execution. With 1, we will need to make sure all clusters have user-resolvable endpoints, which in our case is impossible, so we would anyway set the externalUrl to the gateway URL if that's the solution you pick.

• As part of GitHub Actions
• Use docker containers for Trino and Trino Gateway

Some setup for manual testing and also automated testing already exists and @vishalya will help with this task

I tried to run the trino-gateway according to the documentation, but I failed with the following error message:

java --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED -jar gateway-ha-4-SNAPSHOT-jar-with-dependencies.jar server ../gateway-ha-config.yml
io.dropwizard.configuration.ConfigurationParsingException: ../gateway-ha-config.yml has an error:
  * Failed to parse configuration at: logging.appenders.[0].filterFactories.[0]; Could not resolve type id 'Log-filter-factory' as a subtype of `io.dropwizard.logging.common.filter.FilterFactory<ch.qos.logback.classic.spi.ILoggingEvent>`: known type ids = [uri] (for POJO property 'filterFactories')
 at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain: io.trino.gateway.ha.config.HaGatewayConfiguration["logging"]->io.dropwizard.logging.common.DefaultLoggingFactory["appenders"]->java.util.ArrayList[0]->io.dropwizard.logging.common.ConsoleAppenderFactory["filterFactories"]->java.util.ArrayList[0])

	at io.dropwizard.configuration.ConfigurationParsingException$Builder.build(ConfigurationParsingException.java:277)
	at io.dropwizard.configuration.BaseConfigurationFactory.build(BaseConfigurationFactory.java:177)
	at io.dropwizard.configuration.BaseConfigurationFactory.build(BaseConfigurationFactory.java:94)
	at io.dropwizard.core.cli.ConfiguredCommand.parseConfiguration(ConfiguredCommand.java:139)
	at io.dropwizard.core.cli.ConfiguredCommand.run(ConfiguredCommand.java:85)
	at io.dropwizard.core.cli.Cli.run(Cli.java:78)
	at io.dropwizard.core.Application.run(Application.java:94)
	at io.trino.gateway.ha.HaGatewayLauncher.main(HaGatewayLauncher.java:25)
Caused by: com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve type id 'Log-filter-factory' as a subtype of `io.dropwizard.logging.common.filter.FilterFactory<ch.qos.logback.classic.spi.ILoggingEvent>`: known type ids = [uri] (for POJO property 'filterFactories')
 at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain: io.trino.gateway.ha.config.HaGatewayConfiguration["logging"]->io.dropwizard.logging.common.DefaultLoggingFactory["appenders"]->java.util.ArrayList[0]->io.dropwizard.logging.common.ConsoleAppenderFactory["filterFactories"]->java.util.ArrayList[0])
	at com.fasterxml.jackson.databind.exc.InvalidTypeIdException.from(InvalidTypeIdException.java:43)
	at com.fasterxml.jackson.databind.DeserializationContext.invalidTypeIdException(DeserializationContext.java:2084)
	at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownTypeId(DeserializationContext.java:1575)
	at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._handleUnknownTypeId(TypeDeserializerBase.java:298)
	at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._findDeserializer(TypeDeserializerBase.java:165)
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:151)
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:136)
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:263)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:361)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28)
	at com.fasterxml.jackson.module.blackbird.deser.SettableObjectProperty.deserializeAndSet(SettableObjectProperty.java:44)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:215)
	at com.fasterxml.jackson.module.blackbird.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:120)
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:170)
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:136)
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:263)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:361)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28)
	at com.fasterxml.jackson.module.blackbird.deser.SettableObjectProperty.deserializeAndSet(SettableObjectProperty.java:44)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:215)
	at com.fasterxml.jackson.module.blackbird.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:120)
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedUsingDefaultImpl(AsPropertyTypeDeserializer.java:226)
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:145)
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:263)
	at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:138)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:278)
	at com.fasterxml.jackson.module.blackbird.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:155)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
	at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4801)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2974)
	at io.dropwizard.configuration.BaseConfigurationFactory.build(BaseConfigurationFactory.java:148)

The latest version of Athena's query engine has incorporated many features of Trino: https://aws.amazon.com/about-aws/whats-new/2022/10/amazon-athena-announces-upgraded-query-engine/

It would be great if the Trino Gateway would support redirection to AWS Athena as well.

Offloading queries to Athena could mitigate overloading of the Trino cluster(s) in case of very high usage.

Changing version to MySQL 8.x has causes issue with index size in exact_match_source_selectors table's unique index, as it is innodb with utf8mb4 (4 bytes)

trinodb/trino solves this issue by giving limits to each unique key index UNIQUE (source(128), environment, query_type(128), resource_group_id)

However, in gateway there is syntax issue when activejdbc executes sql.

• #53 (comment)