trinodb / trino-gateway Goto Github PK
View Code? Open in Web Editor NEWHome Page: https://trinodb.github.io/trino-gateway/
License: Apache License 2.0
Home Page: https://trinodb.github.io/trino-gateway/
License: Apache License 2.0
Whenever Trino gateway runs a health check with the trino backend clusters, it returns an 'Unauthorized to fetch cluster stats' error and sets the status of the cluster to 'false'. When changing trino authentication to 'form' from 'oauth2' the health check works.
The latest version of Athena's query engine has incorporated many features of Trino: https://aws.amazon.com/about-aws/whats-new/2022/10/amazon-athena-announces-upgraded-query-engine/
It would be great if the Trino Gateway would support redirection to AWS Athena as well.
Offloading queries to Athena could mitigate overloading of the Trino cluster(s) in case of very high usage.
The TestSpecificDbResourceGroupsManager
tests intermittently fail their assertions on the number of resource groups. This appears to potentially be a timing bug due to interaction between TestSpecificDbResourceGroupsManager
and TestResourceGroupsManager
.
To allow dependency plugin analyze to be enabled. For development you can set
<air.check.skip-dependency>false</air.check.skip-dependency>
in root pom and the use.
mvn install
Or you can use
mvn dependency:analyze
For now that produces LOTS of failures. PRs will remove problems one at a time.
Some improvements (also towards Java 21 support) will come from updating dependencies.
mvn versions:display-dependency-updates
At the end we will remove the skip setting in the root pom.
after airbase PR is merged
The link in https://github.com/trinodb/trino-gateway/blob/main/docs/development.md#contributing return 404.
When the header exceeds 8192, it will cause an exception
WARN [2023-11-29 09:24:27,248] org.eclipse.jetty.http.HttpParser: Header is too large 8193>8192
ProxyServer
In line 81, SSL is not enabled.
The following configuration will not be set
When connecting to a Trino webpage secured by OAuth from a browser, the OAuth handshake will fail if the Gateway forwards requests to different backends. This can be worked around by setting up a routing group with a single backend.
Changing version to MySQL 8.x has causes issue with index size in exact_match_source_selectors table's unique index, as it is innodb with utf8mb4 (4 bytes)
trinodb/trino solves this issue by giving limits to each unique key index UNIQUE (source(128), environment, query_type(128), resource_group_id)
However, in gateway there is syntax issue when activejdbc executes sql.
Anything since after the rename to Trino from PrestoSQL should be compatible. We do not want to support anything older anyway.. users of those old versions can stick to the old Presto Gateway or upgrade.
Maybe do a competition by submitting a logo as comment here.
The current DEBUG log level is too verbose.
Problem: Currently, if the backends are not responding to healthcheck, trino-gateway will inactivate the backends. Since healthcheck only checks against active backends, inactive backends will never be turned to active again.
Solution: it is better to introduce healthy/pending/unhealthy states depending on the health of the backends. These states will be independent from the active/inactive states, so it ensures that unhealthy but active backends can still be checked and possibly turned active again.
Healthy state
is defined as backends are healthy and ready to be served
Pending state
is defined as when the backend is switched from inactive to active. It will wait until healthcheck returns success before turning backend to healthy
Unhealthy state
is defined as backends are returning error or not responding to healthcheck. At this point, if the backends are still active, healthcheck will still check on these backends.
Some setup for manual testing and also automated testing already exists and @vishalya will help with this task
What's the load balancing scheme used by Trino gateway? And is there a way to change it?
Currently the project uses an old version of Dropwizard. Airlift is a superior alternative that is actively maintained and also used in Trino itself.
Rather than upgrading Dropwizard it might be better to move to Airlift and then take advantage of aspects such as easy tarball creation, config system, opentelemety support and much more. This is the preferred approach by @martint.
Potential an initial upgrade of dropwizard to fix the worst issues might still be a good step.
Besides the executable JAR we should at least provide a container image to run Trino Gateway.
This recent PR causes verify
command to fail because maven-surefire-plugin
was removed.
Is there a way to add jvm ops for tests?
I noticed that -DargLine
is a way to add ops when using mvn cmd
mvn clean verify -DargLine="$(tr -s '\r\n' ' ' < ".mvn/jvm.config")"
works for cihttps://github.com/trinodb/trino-gateway/actions/runs/6086764118/job/16513838158
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make protected final java.lang.Class java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) throws java.lang.ClassFormatError accessible: module java.base does not "opens java.lang" to unnamed module @2c4d0347
I use Form/Basic authentication
following:
presetUsers:
user1:
password:
privileges: "lb_admin, lb_user"
user2:
password:
privileges: "lb_api"
The authorized account and password pop up.
Input user and password ,401 Unauthorized will always appear.
my setting sample:
authentication:
defaultType: "form"
presetUsers:
user1:
password: "pass"
privileges: "lb_admin, lb_user, lb_api"
Like Trino itself.
The MySQL driver requires enabling TLS1 and/or 1.1 in the JDK, or setting SSLMode=Disabled
.
An optional feature to support retrying another backend url (idk maybe to default one?) if a query is requested to gateway, but the backend url it routed dies and/or there is no response from that trino server.
For release tooling and much more.
Hello folks, we hit a bug using the new trino-gateway
when opening query details from the gateway's UI displaying query history.
The trino-gateway
uses backendUrl
instead of externalUrl
in the links it shows in the UI (see code) which means when opened from our internal users, they do not resolve properly since it's trying to use the internal endpoints that the gateway uses to communicate with the clusters.
The previous presto-gateway
was using the gateway's URL for this (see code), which works so far in our case, and then it routes the UI calls to fetch the query details to the right coordinator cluster using the query mapping table/cache. At some point there is a request http://<gateway-url>/ui/api/query/<query_id>
, which is matched by the request handler and gets sent to the right coordinator that has the query details.
I suggest one of the following:
externalUrl
of the backend cluster for the links in the query history, and then we will need to set those to something resolvable by the expected users of the admin UI.I believe the better solution is 2, so that the gateway handles all the query routing requests, same as it does for the actual execution. With 1, we will need to make sure all clusters have user-resolvable endpoints, which in our case is impossible, so we would anyway set the externalUrl
to the gateway URL if that's the solution you pick.
based on lyft version plus enhancements from Bloomberg
includes refactor of package space to io.trino.gateway
Approved and agreed upon names are
Add to release process as well
It might be useful to add a Helm chart to manage Trino Gateway deployment on k8s.
resourceGroupId
in /trino/selector/read/{resourceGroupId}
is wrongly annotated with @QueryParam
, when it's actually a path parameter:
@GET
@Path("/selector/read/{resourceGroupId}")
public Response readSelector(@QueryParam("resourceGroupId") String resourceGroupIdStr,
...
As a result, /trino/selector/read/X
returns all selectors, and not just those in the X
resource group.
in some cases various back-ends may support different credential sets.
it should be possible to override backendState per gateway back-end instance. this should be optional to allow fallback to the global setting.
Follow along the Java 21 requirement update for Trino itself. Currently the codebase does not compile with Java 21. Probably needs more dependency updates as well.
Lombok uses private API in Javac and that blocks the upgrade to Java 21. We should not need Lombok in general anyway.
we are planning to run multiple replicas in ocp, our environment requires full deployment to update the container in anyway which would require a redeployment any time we want to update the routing rules so if we could use the api calls and have them stored in postgres than it would make admin work for gateway much better.
Blog post
linkedin
slack
The project build from #14 needs to be verified to produce a usable and working binary. The build already passes.
This automatically also passes testing of the usage of Java 17.
We should remove the following 2 lines from pom.xml:
Lines 119 to 120 in 7e30bbe
Trino Gateway routes queries randomly within a routing group and does not consider the number of already running and queued queries on a cluster within a routing group
It can happen that some queries take longer and and queue of the routed cluster keeps growing
Assumption 3 Clusters in a routing group.
Implement a solution which routes queries based on health status of the backend. It should consider the number of running and queued queries and route new queries to clusters with less load
There is already code for this, but it is not used as of now. ClusterStateListenerModule does not have code for it.
https://github.com/trinodb/trino-gateway/blob/main/gateway-ha/src/main/java/io/trino/gateway/ha/router/TrinoQueueLengthRoutingTable.java
https://github.com/trinodb/trino-gateway/blob/main/gateway-ha/src/main/java/io/trino/gateway/ha/clustermonitor/TrinoQueueLengthChecker.java
In new release notes page in docs
Hi, we originally had oidc to okta working with v3 of trino-gateway with the following config
authentication:
defaultType: oauth
oauth:
issuer: ${ISSUER}
clientId: ${TRINO_GATEWAY_OIDC_CLIENT_ID}
clientSecret: ${TRINO_GATEWAY_OIDC_CLIENT_SECRET}
tokenEndpoint: ${ISSUER}/oauth2/v1/token
authorizationEndpoint: ${ISSUER}/oauth2/v1/authorize
jwkEndpoint: ${ISSUER}/oauth2/v1/keys
redirectUrl: https://${URL_OF_GATEWAY}/oidc/callback
userIdField: email
scopes:
- openid
- email
- offline_access
After we updated to latest version 4 [latest commit of main on trino-gateway at the time] b6bbb14 when we hit the url of the gateway we had deployed it would redirect to http://localhost:8090/sso
instead of http://${URL_OF_GATEWAY}/sso
I believe the update of jetty
/dropwizard
introduced this
434eec9
I tested that theory by checking out the latest v3 and applying all commits one at a time and that was the commit that introduced that behavior.
Hopefully theres just a config option I'm missing but I couldn't get the redirect fixed.
I tried to run the trino-gateway according to the documentation, but I failed with the following error message:
java --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED -jar gateway-ha-4-SNAPSHOT-jar-with-dependencies.jar server ../gateway-ha-config.yml
io.dropwizard.configuration.ConfigurationParsingException: ../gateway-ha-config.yml has an error:
* Failed to parse configuration at: logging.appenders.[0].filterFactories.[0]; Could not resolve type id 'Log-filter-factory' as a subtype of `io.dropwizard.logging.common.filter.FilterFactory<ch.qos.logback.classic.spi.ILoggingEvent>`: known type ids = [uri] (for POJO property 'filterFactories')
at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain: io.trino.gateway.ha.config.HaGatewayConfiguration["logging"]->io.dropwizard.logging.common.DefaultLoggingFactory["appenders"]->java.util.ArrayList[0]->io.dropwizard.logging.common.ConsoleAppenderFactory["filterFactories"]->java.util.ArrayList[0])
at io.dropwizard.configuration.ConfigurationParsingException$Builder.build(ConfigurationParsingException.java:277)
at io.dropwizard.configuration.BaseConfigurationFactory.build(BaseConfigurationFactory.java:177)
at io.dropwizard.configuration.BaseConfigurationFactory.build(BaseConfigurationFactory.java:94)
at io.dropwizard.core.cli.ConfiguredCommand.parseConfiguration(ConfiguredCommand.java:139)
at io.dropwizard.core.cli.ConfiguredCommand.run(ConfiguredCommand.java:85)
at io.dropwizard.core.cli.Cli.run(Cli.java:78)
at io.dropwizard.core.Application.run(Application.java:94)
at io.trino.gateway.ha.HaGatewayLauncher.main(HaGatewayLauncher.java:25)
Caused by: com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve type id 'Log-filter-factory' as a subtype of `io.dropwizard.logging.common.filter.FilterFactory<ch.qos.logback.classic.spi.ILoggingEvent>`: known type ids = [uri] (for POJO property 'filterFactories')
at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain: io.trino.gateway.ha.config.HaGatewayConfiguration["logging"]->io.dropwizard.logging.common.DefaultLoggingFactory["appenders"]->java.util.ArrayList[0]->io.dropwizard.logging.common.ConsoleAppenderFactory["filterFactories"]->java.util.ArrayList[0])
at com.fasterxml.jackson.databind.exc.InvalidTypeIdException.from(InvalidTypeIdException.java:43)
at com.fasterxml.jackson.databind.DeserializationContext.invalidTypeIdException(DeserializationContext.java:2084)
at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownTypeId(DeserializationContext.java:1575)
at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._handleUnknownTypeId(TypeDeserializerBase.java:298)
at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._findDeserializer(TypeDeserializerBase.java:165)
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:151)
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:136)
at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:263)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:361)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28)
at com.fasterxml.jackson.module.blackbird.deser.SettableObjectProperty.deserializeAndSet(SettableObjectProperty.java:44)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314)
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:215)
at com.fasterxml.jackson.module.blackbird.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:120)
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:170)
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:136)
at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:263)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:361)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244)
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28)
at com.fasterxml.jackson.module.blackbird.deser.SettableObjectProperty.deserializeAndSet(SettableObjectProperty.java:44)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314)
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:215)
at com.fasterxml.jackson.module.blackbird.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:120)
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedUsingDefaultImpl(AsPropertyTypeDeserializer.java:226)
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:145)
at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:263)
at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:138)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:278)
at com.fasterxml.jackson.module.blackbird.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:155)
at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4801)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2974)
at io.dropwizard.configuration.BaseConfigurationFactory.build(BaseConfigurationFactory.java:148)
https://github.com/trinodb/trino-gateway/actions/runs/7361226191/job/20038470483?pr=117
Error: io.trino.gateway.ha.router.TestRoutingGroupSelector.testByRoutingRulesEngineFileChange -- Time elapsed: 0.007 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <etl2> but was: <etl>
at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1141)
at io.trino.gateway.ha.router.TestRoutingGroupSelector.testByRoutingRulesEngineFileChange(TestRoutingGroupSelector.java:138)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.