beltran / gohive Goto Github PK
View Code? Open in Web Editor NEWGo driver for Apache Hive and the Hive Metastore
License: MIT License
Go driver for Apache Hive and the Hive Metastore
License: MIT License
Hello,
I'm a golang beginner, so please excuse me if this is an issue on my end.
I'm running Hive locally and want to connect to it from my code.
I installed the package with 'go get github.com/beltran/gohive' and tried to run the example code in my project.
When I run the code, I get this error:
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x6525e0]
goroutine 1 [running]:
github.com/apache/thrift/lib/go/thrift.(*TSocket).IsOpen(...)
C:/Users/user/go/pkg/mod/github.com/apache/[email protected]/lib/go/thrift/socket.go:111
github.com/apache/thrift/lib/go/thrift.(*TSocket).Open(0x0, 0x10, 0x0)
C:/Users/user/go/pkg/mod/github.com/apache/[email protected]/lib/go/thrift/socket.go:85 +0x30
github.com/beltran/gohive.innerConnect(0x75c2a7, 0xa, 0x2710, 0x75aaff, 0x4, 0xc00010e000, 0x0, 0x1, 0x16)
C:/Users/user/go/pkg/mod/github.com/beltran/[email protected]/hive.go:165 +0x1c3
github.com/beltran/gohive.Connect(...)
C:/Users/user/go/pkg/mod/github.com/beltran/[email protected]/hive.go:118
main.main()
c:/Users/user/go/src/value-mapping-service-hdfs-writer/main.go:16 +0x133
exit status 2
[Done] exited with code=1 in 1.314 seconds
It seems that "connection, errConn := gohive.Connect("localhost:", 10000, "NONE", configuration)" starts this issue and goes deeper.
I noticed that if using "localhost" instead of "localhost:", I get the error:
2020/05/08 14:32:56 Required field 'sessionId' is unset! Struct:TSessionHandle(sessionId:null)
exit status 1
I'm asking because I'm slightly confused how to connect to my locally hosted Hive. Any help is appreciated.
Hello again!
I'm trying to use GoHive with Kerberos. Unfortunately, there seems to be no libkrb5-dev for Windows 10.
I do have MIT Kerberos with SDK and TDM64-GCC installed.
The file gssapi.h is located in C:\Program Files\MIT\Kerberos\include\gssapi.
I have env variable MITKRB5 pointing to C:\Program Files\MIT\Kerberos\
But I still get the following erros when I try to run:
go get -tags kerberos github.com/beltran/gohive
..\..\pkg\mod\github.com\beltran\[email protected]\buffer.go:9:10: fatal error: gssapi/gssapi.h: No such file or directory
9 | #include <gssapi/gssapi.h>
| ^~~~~~~~~~~~~~~~~
compilation terminated.
go get -tags kerberos github.com/beltran/gosasl
# github.com/beltran/gssapi
..\..\pkg\mod\github.com\beltran\[email protected]\buffer.go:9:10: fatal error: gssapi/gssapi.h: No such file or directory
9 | #include <gssapi/gssapi.h>
| ^~~~~~~~~~~~~~~~~
compilation terminated.
I try to run the demo in NOSASL
, but get the following error. How to resolve it ?
Invalid method name: 'OpenSession'
configuration := gohive.NewConnectConfiguration()
configuration.Service = "hive"
configuration.FetchSize = 1000
connection, errConn := gohive.Connect("xx.xx.xx.xx", 10000, "KERBEROS", configuration)
llog.Debug("--------",errConn)
kinit have done: kinit -kt ./secret.keytab hive/[email protected]
return err info:
019/02/23 23:14:17 Bad SASL negotiation status: 3 (GSS initiate failed)
I use "github.com/beltran/gohive v1.5.3" at the "NONE" mode.
I run main.go in the example folder.
the sql "SELECT * FROM myTable" have no result, but I can get 4 rows in the hive client use the same sql.
we connect hive with kerberos, kinit was done, but some wrong happend:
Failed to open Apache Hive connection: Bad SASL negotiation status: 3 (Final handshake step failed)
We experience an error when connecting to hive using Kerberos auth:
Bad SASL negotiation status: 3 (GSS initiate failed)
Below is our code and before executing our code, we execute kinit:
kinit -kt {{ .Values.krb5.keytabFile }} {{ .Values.krb5.principal }};
configuration.Username = cfg.HiveUsername
configuration.Password = cfg.HivePassword
configuration.Service = cfg.HiveService
configuration.FetchSize = cfg.HiveFetchsize
if cfg.HiveAuth == "KERBEROS" || cfg.HiveAuth == "kerberos" {
configuration.TLSConfig = &tls.Config{
InsecureSkipVerify: true,
}
}
connection, errConn := gohive.Connect(cfg.HiveHost, cfg.HivePort, cfg.HiveAuth, configuration)
if errConn != nil {
return nil, fmt.Errorf("Could not connect to Hive. %v", errConn)
}
return &HiveClient{
Configuration: configuration,
Connection: connection,
}, nil
I was running something like this:
i:=make([]interface{},100)
for cursor.HasMore(ctx) {
cursor.FetchOne(ctx,i...)
if cursor.Err != nil {
log.Fatal(cursor.Err)
}
}
and the return was:
Unexpected data type <nil> for value 2019 (should be string)
Can you please add support for the interface{} type?
Thank you very much ^^
file: hive.go
166 line
if err = transport.Open(); err != nil {
return
}
modify
if err = transport.Open(); err != nil && err.Error() != "Socket already connected." {
return
}
Reproduce
1. create table and insert data in mysql
create table t(a int, b int)
insert into t values(1,2)
2. use gohive client to execute
cursor.Exec(ctx,"select * from t as x left join t as y on x.a=y.b")
cursor.RowMap(ctx)
Expected Result
x.a: 1 x.b:2 y.a: nil y.b:nil
But Get
a:nil b.nil
Get error message:
../../go/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1044:36: not enough arguments in call to iprot.ReadStructBegin
have ()
want (context.Context)
../../go/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1049:55: not enough arguments in call to iprot.ReadFieldBegin
have ()
want (context.Context)
../../go/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1063:25: not enough arguments in call to iprot.Skip
have (thrift.TType)
want (context.Context, thrift.TType)
../../go/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1082:31: not enough arguments in call to iprot.ReadFieldEnd
have ()
want (context.Context)
../../go/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1086:31: not enough arguments in call to iprot.ReadStructEnd
have ()
want (context.Context)
../../go/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1093:28: not enough arguments in call to iprot.ReadI32
have ()
want (context.Context)
../../go/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1102:31: not enough arguments in call to iprot.ReadString
have ()
want (context.Context)
../../go/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1114:34: not enough arguments in call to oprot.WriteStructBegin
have (string)
want (context.Context, string)
../../go/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1125:32: not enough arguments in call to oprot.WriteFieldStop
have ()
want (context.Context)
../../go/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1128:32: not enough arguments in call to oprot.WriteStructEnd
have ()
want (context.Context)
../../go/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1128:32: too many errors
Hi,
Is there any way to set the cursor timeout to let the hive client wait for the query to execute?
cursor.Exec(ctx, "select * from table1")
Does the HiveConfiguration support this?
Thanks :)
Getting the following error when doing go get github.com/beltran/gohive
# github.com/beltran/gohive/vendor/github.com/beltran/gssapi ../../beltran/gohive/vendor/github.com/beltran/gssapi/buffer.go:9:10: fatal error: gssapi/gssapi.h: No such file or directory #include <gssapi/gssapi.h> ^~~~~~~~~~~~~~~~~ compilation terminated.
Edit: Apologies. Github gave an error and double posted. This one can be safely deleted,
Hello again. I'm trying to build my program in a multistage container build.
I'm using golang:1.15.2-buster
as builder image.
pip list shows that gssapi is installed.
---> Running in cedd3c0e325d
Package Version
------------- -------
asn1crypto 0.24.0
configparser 3.5.0b2
cryptography 2.6.1
decorator 4.4.2
entrypoints 0.3
enum34 1.1.6
gssapi 1.6.2
ipaddress 1.0.17
keyring 17.1.1
keyrings.alt 3.1.1
mercurial 4.8.2
pip 18.1
pycrypto 2.6.1
PyGObject 3.30.4
pyxdg 0.25
SecretStorage 2.3.1
setuptools 40.8.0
six 1.12.0
wheel 0.32.3
I get the following errors:
Step 19/28 : RUN go get -tags kerberos github.com/beltran/gohive
---> Running in b8866152dcec
go: github.com/beltran/gohive upgrade => v1.2.0
go: downloading github.com/beltran/gohive v1.2.0
# github.com/beltran/gosasl
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:62:16: undefined: gssapi.Name
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:181:2: undefined: gssapi.Options
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:183:2: undefined: gssapi.Lib
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:187:18: undefined: gssapi.CredId
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:190:18: undefined: gssapi.CtxId
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:198:20: undefined: gssapi.GSS_C_INTEG_FLAG
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:198:54: undefined: gssapi.GSS_C_MUTUAL_FLAG
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:198:89: undefined: gssapi.GSS_C_SEQUENCE_FLAG
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:198:126: undefined: gssapi.GSS_C_CONF_FLAG
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:326:44: undefined: gssapi.Name
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:198:126: too many errors
Am I missing something important, am I running a wrong version of GoHive or is there some other problem?
To me it seems that gssapi cannot be found despite it being installed.
In the standard library(database/sql), we have a pool of connections. Therefore, I can create a connection once and use it for all requests, even in several goroutines (it is convenient for mock testing). For example, using the MongoDB driver, I can also use one connection for all requests.
If I make one connection using your library and try to use it in several goroutines at the same time, I will get an error. Errors at each start can be different.
Can I use one connection(connection pool) for all queries in your library?
I create a connect with HiveConfiguration.
conf.HiveConfiguration = map[string]string{"tez.queue.name": yarnQueue}
But queries with the connect can not submit to the correct queue,sometimes they are correct sometimes wrong.
panic: runtime error: invalid memory address or nil pointer dereference
goroutine 4793785 [running]:
git.ddxq.mobi/ddmc-engine/chain/utils.SqlHive.func1.1.1(0xc012e36800, 0x7e, 0xc5587eb4a0)
/data1/jenkins/workspace/golang-service_2/chain/utils/sql.go:41 +0xba
panic(0x1375d40, 0x1fd2460)
/usr/local/go1.15.2/src/runtime/panic.go:969 +0x175
github.com/beltran/gohive.(Cursor).executeAsync(0xc54e821c80, 0x1762280, 0xc0000340b0, 0xc012e36800, 0x7e)
/var/lib/jenkins/go/pkg/mod/github.com/beltran/gohive@v1.5.1/hive.go:538 +0x26a
github.com/beltran/gohive.(Cursor).Execute(0xc54e821c80, 0x1762280, 0xc0000340b0, 0xc012e36800, 0x7e, 0x0)
/var/lib/jenkins/go/pkg/mod/github.com/beltran/gohive@v1.5.1/hive.go:464 +0x57
git.ddxq.mobi/ddmc-engine/chain/utils.sqlHive(0x1762280, 0xc0000340b0, 0xc000208310, 0x12bd760, 0xc4d9c61040, 0xc012e36800, 0x7e, 0x0, 0x0)
/data1/jenkins/workspace/golang-service_2/chain/utils/sql.go:116 +0x21b
git.ddxq.mobi/ddmc-engine/chain/utils.SqlHive.func1.1(0xc012e36800, 0x7e, 0xc5587eb4a0, 0x1762280, 0xc0000340b0, 0xc000208310, 0x12bd760, 0xc4d9c61040)
/data1/jenkins/workspace/golang-service_2/chain/utils/sql.go:49 +0xc8
created by git.ddxq.mobi/ddmc-engine/chain/utils.SqlHive.func1
/data1/jenkins/workspace/golang-service_2/chain/utils/sql.go:37 +0x171
chain/utils/...
It's my own stack that you can ignore.
This exception occasionally occurs on the line.
how to executing multiple queries maintaining the session ...
EX
SET mapred.job.queue.name = default;
set hive.exec.parallel = true;
SET mapred.job.name = myjob;
SET hive.exec.max.dynamic.partitions = 100000;
SET hive.exec.max.dynamic.partitions.pernode = 100000;
set hive.vectorized.execution.enabled = true;
set hive.vectorized.execution.reduce.enabled = true;
SET hive.execution.engine = mr;
set hive.exec.dynamic.partition.mode = nonstrict;
insert overwrite table scm.mytable PARTITION(eventdate)
select * from stg.stage_table;
package main
import (
"github.com/beltran/gohive"
"log"
)
func main() {
configuration := gohive.NewConnectConfiguration()
configuration.Service = "hive"
configuration.FetchSize = 1000
connection, errConn := gohive.Connect("10.21.89.89", 10000, "NOSASL", configuration)
if errConn != nil {
log.Fatal(errConn)
}
connection.Close()
}
My hive cluster did not enable kerberos authentication, I used this code to connect the hive cluster and port with 10000, it returns an exception with EOF, what's wrong ?
hi i exec go test to query is ok,but exec main func has error:write: broken pipe
Hello author, now we have a problem, that kinit has been successful, gohive.NewConnectConfiguration() ,the configuration.Principal = "abc" or configuration.Principal = "xyz" will connect to hive successfully no matter what value is set, why is this? Please help to answer thank you
gohive upgrade thrift version to 0.14.0,but my project require thrift version under 0.12.0,can you add this support in v1.0.1 ?
Is it possible to connect to Hive running in embedded mode (connection string jdbc:hive2://
) with this package?
Build error when using v1.5.3 . It seems there is a bug. The below is build output
# github.com/beltran/gohive ../../go/pkg/mod/github.com/beltran/[email protected]/hive.go:210:18: assignment mismatch: 2 variables but thrift.NewTSSLSocketConf returns 1 value ../../go/pkg/mod/github.com/beltran/[email protected]/hive.go:216:18: assignment mismatch: 2 variables but thrift.NewTSocketConf returns 1 value
Hello once again :). I finally got back to this and face another problem.
Despite having go build -tags kerberos in Dockerfile, I get:
Panic: gosasl may have been installed without kerberos support please reinstall with `go get` using the flags `build kerberos`. Alternatively if `go run` is being ran it should be ran with `go run -tags kerberos ...` and the binary should have been build with `go build -tags kerberos ...`.
goroutine 1 [running]:
github.com/beltran/gosasl.NewGSSAPIMechanism(...)
/go/pkg/mod/github.com/beltran/[email protected]/no_gssapi.go:14
github.com/beltran/gohive.NewTSaslTransport(0x8c7440, 0xc0000a0240, 0xc00001e00a, 0x24, 0x842f8c, 0x6, 0xc0001a3290, 0x4bf785, 0xc00007c120, 0x2)
/go/pkg/mod/github.com/beltran/[email protected]/sasl_transport.go:48 +0x4bb
github.com/beltran/gohive.innerConnect(0xc00001e00a, 0x24, 0x1f49, 0xc00001a9c8, 0x8, 0xc0001b6000, 0xc00011e540, 0x40, 0x40)
/go/pkg/mod/github.com/beltran/[email protected]/hive.go:246 +0xda8
github.com/beltran/gohive.Connect(...)
/go/pkg/mod/github.com/beltran/[email protected]/hive.go:119
SANITIZED/internal/hive.New(0xc00001e00a, 0x24, 0x1f49, 0xc00001a9c8, 0x8, 0xc00001c0ce, 0x4, 0xc00001c10e, 0x4, 0xc00001c00e, ...)
/build/internal/hive/client.go:61 +0x1f9
main.createClients(0xc00001e00a, 0x24, 0x1f49, 0xc00001a9c8, 0x8, 0xc00001c0ce, 0x4, 0xc00001c10e, 0x4, 0xc00001c00e, ...)
/build/main.go:234 +0x58
main.main()
/build/main.go:78 +0x46d
I have tried running go get gosasl and go get gohive in Dockerfile with -tags kerberos.
Using ENTRYPOINT ["/main"] or CMD in Dockerfile outputs the error above.
Here is the Dockerfile:
FROM SANITIZED/golang:1.15.2-buster AS builder
# Set necessary environmet variables needed for our image
ENV GO111MODULE=on \
CGO_ENABLED=1 \
GOOS=linux \
GOARCH=amd64 \
GOPROXY="SANITIZED"
# Move to working directory /build
WORKDIR /build
RUN sed -i 'SANITIZED' /etc/apt/sources.list
RUN sed -i 'SANITIZED' /etc/apt/sources.list
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends apt-utils
RUN apt-get install python-pip libkrb5-dev -y python-gssapi libgssapi-krb5-2
RUN pip install gssapi
RUN pip list
RUN go get -tags kerberos "github.com/beltran/gosasl"
RUN go get -tags kerberos "github.com/beltran/gohive"
# Copy and download dependency using go mod
COPY go.mod .
COPY go.sum .
RUN go mod download
# Copy the code into the container
COPY . .
# Build binary
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -tags kerberos -tags netgo -a -ldflags '-w -extldflags "-static"' -o main .
# Move to /bin for the resulting binary file
WORKDIR /bin
# Copy binary from build to main folder
RUN cp /build/main .
COPY . .
# Start from scratch
FROM scratch
# Copy the binary
COPY --from=builder /bin/main /
# Add user
USER 1000
# Command to run
ENTRYPOINT [ "/main"]
Hi, I try to insert array type of field into table, it returns
Unable to create temp file for insert values Expression of type TOK_FUNCTION not supported in insert
i have an issue when i use "m := cursor.RowMap(context.Background())" for the query like "select a,b,c from tableName"(data type: a int32,b string,c string).I only receive the data from column a, but when i use "cursor.FetchOne(ctx, &a, &b, &c)" i recevie the data for all column
Hello, i think that this code on file hive.go line 424 is not right:
if msg == nil { *msg = fmt.Sprintf("gohive: operation in state (%v) without task status or error message", operationStatus.OperationState) }
You shouldn't derreference a nil pointer, this can lead to a panic.
Regards
Hi.
I just compiled your example, via
$go build
.
Then I tried to launch the executable:
./example
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x442d6c]goroutine 1 [running]:
github.x2ecom..z2fapache..z2fthrift..z2flib..z2fgo..z2fthrift.TSocket.IsOpen
/home/oceanfish81/go_projects/gohive/vendor/github.com/apache/thrift/lib/go/thrift/socket.go:111
github.x2ecom..z2fapache..z2fthrift..z2flib..z2fgo..z2fthrift.TSocket.Open
/home/oceanfish81/go_projects/gohive/vendor/github.com/apache/thrift/lib/go/thrift/socket.go:85
gohive.innerConnect
/home/oceanfish81/go_projects/gohive/hive.go:166
github.x2ecom..z2fbeltran..z2fgohive.Connect
/home/oceanfish81/go_projects/gohive/hive.go:119
main.main
/home/oceanfish81/go_projects/gohive/example/main.go:16
. If there is an uncaught exception - it is just hard to distinguish misconfigured external resources (no open port, for the socket) from something else.
panic: gosasl was installed without kerberos support please reinstall with go get
using the flags build kerberos
Could you teach me how to solve this problem? Thank you very much. btw, i am a golang beginner
hi,
My env:
export GOROOT=/usr/local/go
export GOPATH=/go
export PATH=$PATH:$GOROOT/bin/:$GOPATH/bin
sudo yum install -y krb5-devel
go get -t kerberos github.com/beltran/gohive
package kerberos: unrecognized import path "kerberos" (import path does not begin with hostname)
then run example main.go.
go run main.go
panic: gosasl was installed without kerberos support please reinstall with go get
using the flags build kerberos
goroutine 1 [running]:
github.com/beltran/gohive/vendor/github.com/beltran/gosasl.NewGSSAPIMechanism(0x707397, 0x4, 0x707ccf, 0x7, 0xc420098bc8)
/go/src/github.com/beltran/gohive/vendor/github.com/beltran/gosasl/no_gssapi.go:14 +0x39
github.com/beltran/gohive.NewTSaslTransport(0x864660, 0xc42007f410, 0x7117b4, 0x23, 0x707736, 0x6, 0xc42007f500, 0x0, 0x0, 0x200000000c)
/go/src/github.com/beltran/gohive/sasl_transport.go:47 +0xd3
github.com/beltran/gohive.Connect(0x7117b4, 0x23, 0x2710, 0x707e86, 0x8, 0xc420104000, 0x7fb66d13d000, 0x0, 0xc420063e90)
/go/src/github.com/beltran/gohive/hive.go:159 +0x1596
main.main()
/go/src/project/test/main.go:18 +0x122
exit status 2
It returns this error
go/src/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1044:36: not enough arguments in call to iprot.ReadStructBegin
have ()
want (context.Context)
go/src/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1049:55: not enough arguments in call to iprot.ReadFieldBegin
have ()
want (context.Context)
go/src/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1063:25: not enough arguments in call to iprot.Skip
have (thrift.TType)
want (context.Context, thrift.TType)
go/src/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1082:31: not enough arguments in call to iprot.ReadFieldEnd
have ()
want (context.Context)
go/src/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1086:31: not enough arguments in call to iprot.ReadStructEnd
have ()
want (context.Context)
go/src/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1093:28: not enough arguments in call to iprot.ReadI32
have ()
want (context.Context)
go/src/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1102:31: not enough arguments in call to iprot.ReadString
have ()
want (context.Context)
go/src/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1114:34: not enough arguments in call to oprot.WriteStructBegin
have (string)
want (context.Context, string)
go/src/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1125:32: not enough arguments in call to oprot.WriteFieldStop
have ()
want (context.Context)
go/src/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1128:32: not enough arguments in call to oprot.WriteStructEnd
have ()
want (context.Context)
go/src/src/github.com/beltran/gohive/hiveserver/HiveServer.go:1128:32: too many errors
Hi there
I have two general questions:
kinit, klist, kdestroy1
installed locally. hs2.example.com is an alias for 127.0.0.1 in /etc/hosts.`+ counter=19
+ [[ 19 -gt 18 ]]
+ echo 'Waited for three minutes and hive didn'\''t appear to start'
Waited for three minutes and hive didn't appear to start
+ docker logs hs2.example
2020-11-05T22:54:14,291 INFO [NotificationEventPoll 0] metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hive/[email protected] (auth:KERBEROS) retries=1 delay=1 lifetime=0
2020-11-05T22:54:14,292 INFO [NotificationEventPoll 0] metastore.HiveMetaStore: 2: get_config_value: name=metastore.batch.retrieve.max defaultValue=50
2020-11-05T22:54:14,292 INFO [NotificationEventPoll 0] HiveMetaStore.audit: ugi=hive/[email protected] ip=unknown-ip-addr cmd=get_config_value: name=metastore.batch.retrieve.max defaultValue=50
2020-11-05T22:54:21,902 ERROR [HiveServer2-Handler-Pool: Thread-66] server.TThreadPoolServer: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Invalid status 71
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:694) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:691) ~[hive-exec-3.1.2.jar:3.1.2]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_272]
at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_272]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742) ~[hadoop-common-2.7.7.jar:?]
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:691) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) ~[hive-exec-3.1.2.jar:3.1.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_272]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_272]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_272]
Caused by: org.apache.thrift.transport.TTransportException: Invalid status 71
at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:184) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ~[hive-exec-3.1.2.jar:3.1.2]
... 10 more
2020-11-05T22:54:31,937 ERROR [HiveServer2-Handler-Pool: Thread-66] server.TThreadPoolServer: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Invalid status 71
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:694) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:691) ~[hive-exec-3.1.2.jar:3.1.2]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_272]
at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_272]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742) ~[hadoop-common-2.7.7.jar:?]
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:691) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) ~[hive-exec-3.1.2.jar:3.1.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_272]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_272]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_272]
Caused by: org.apache.thrift.transport.TTransportException: Invalid status 71
at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:184) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ~[hive-exec-3.1.2.jar:3.1.2]
... 10 more
2020-11-05T22:54:41,966 ERROR [HiveServer2-Handler-Pool: Thread-66] server.TThreadPoolServer: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Invalid status 71
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:694) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:691) ~[hive-exec-3.1.2.jar:3.1.2]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_272]
at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_272]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742) ~[hadoop-common-2.7.7.jar:?]
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:691) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) ~[hive-exec-3.1.2.jar:3.1.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_272]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_272]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_272]
Caused by: org.apache.thrift.transport.TTransportException: Invalid status 71
at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:184) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ~[hive-exec-3.1.2.jar:3.1.2]
... 10 more
+ exit 1
Hi,
Can I connect the hive with kerberos authentication under centos7 by the following operation?
yum install -y krb5-devel
go get -tags kerberos github.com/beltran/gohive
Thanks!
"gosasl may have been installed without kerberos support please reinstall with go get
using the flags build kerberos
. Alternatively if go run
is being ran it should be ran with go run -tags kerberos ...
and the binary should have been build with `go build -tags kerberos" No_gssapi will be called directly. What is the reason? Please answer, thank you very much
Hi beltran,
Recently I want to operate cdh hiveserver2, which requires kerberos authentication.
With the kinit
command help, I can connect to hiveserver2 using gohive successfully.
However, I want to communicate with hs2 all with codes without running kinit
, so I spend a lot of time to learn about kerberos and the gokrb5 repo, https://github.com/jcmturner/gokrb5.
The following is my code example:
https://gist.github.com/nice-journey/40148566b92e894a16cb28aa682a2740
The main idea of the previous code includes three standard krb auth steps:
But the hs2 throw error log as the following:
2022-06-04 01:07:16,919 DEBUG org.apache.thrift.transport.TSaslServerTransport: [HiveServer2-Handler-Pool: Thread-43]: transport map does not contain key 2022-06-04 01:07:16,919 DEBUG org.apache.thrift.transport.TSaslTransport: [HiveServer2-Handler-Pool: Thread-43]: opening transport org.apache.thrift.transport.TSaslServerTransport@5839534b 2022-06-04 01:07:16,923 DEBUG org.apache.thrift.transport.TSaslTransport: [HiveServer2-Handler-Pool: Thread-43]: SERVER: Writing message with status ERROR and payload length 17 2022-06-04 01:07:16,923 DEBUG org.apache.thrift.transport.TSaslServerTransport: [HiveServer2-Handler-Pool: Thread-43]: failed to open server transport org.apache.thrift.transport.TTransportException: Invalid status 96 at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:184) at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) at ...............
The error log indicates the header of message is not correct.
After some research I found the fact that the hs2 authentication is not fully krb mode, and it is gssapi mode. The krb is just one achievement of GSS API, so hs2 throw error of wrong header.
And I also found that the krb server does not output any log when I use gohive to operate hive. Can it prove the fact that auth progress of hive has nothing to do with kerberos server? So my attempt in the gist code maybe totally wrong?
Could you give some advices to communicate with hs2 without kinit
and totally with codes? which is similar to the UserGroupInformation package of java. It read keytab file and krb config to communicate to hs2 server, and it does not need to run kinit
previously.
Sorry I'm only skilled in cpp/go/python and not skilled in java.
Thanks!
Hi beltran,
I use beltra/gohive heavy, and I use go module.
Now if I execute go mod init
go 1.13
require github.com/beltran/gohive v1.0.0
This version v1.0.0
has connection failure.
https://github.com/beltran/gohive/blob/v1.0.0/hive.go#L166
Please create new tag and make me happy!!
Thanks!!
Hello Team,
On june 15th, "github.com/apache/thrift/lib/go/thrift" recenlty updated its file. Causing file hiveserver.go to fail. Just curious if this Its been addressed at the moment? . I'm Kind of nubbie, so maybe a workaround exist for it at the moment?.
thrift update - link: apache/thrift@e79f764#diff-8a01e4124163225974263ecb577e8bf4
affected file: https://github.com/beltran/gohive/blob/master/hiveserver/HiveServer.go
error:
.....\go\src\github.com\beltran\gohive\hiveserver\HiveServer.go:1125:32: not enough arguments in call to oprot.WriteFieldStop have () want (context.Context) ......\go\src\github.com\beltran\gohive\hiveserver\HiveServer.go:1128:32: not enough arguments in call to oprot.WriteStructEnd have () want (context.Context) ......\go\src\github.com\beltran\gohive\hiveserver\HiveServer.go:1128:32: too many errors
Regards,
I want to connect hive on A server (kinit command cannot be executed). How to connect via kerberos?
hive is on B server
How to get nil field value, not default?
Hello again. I'm trying to build my program in a multistage container build.
I'm using golang:1.15.2-buster
as builder image.
pip list shows that gssapi is installed.
---> Running in cedd3c0e325d
Package Version
------------- -------
asn1crypto 0.24.0
configparser 3.5.0b2
cryptography 2.6.1
decorator 4.4.2
entrypoints 0.3
enum34 1.1.6
gssapi 1.6.2
ipaddress 1.0.17
keyring 17.1.1
keyrings.alt 3.1.1
mercurial 4.8.2
pip 18.1
pycrypto 2.6.1
PyGObject 3.30.4
pyxdg 0.25
SecretStorage 2.3.1
setuptools 40.8.0
six 1.12.0
wheel 0.32.3
I get the following errors:
Step 19/28 : RUN go get -tags kerberos github.com/beltran/gohive
---> Running in b8866152dcec
go: github.com/beltran/gohive upgrade => v1.2.0
go: downloading github.com/beltran/gohive v1.2.0
# github.com/beltran/gosasl
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:62:16: undefined: gssapi.Name
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:181:2: undefined: gssapi.Options
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:183:2: undefined: gssapi.Lib
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:187:18: undefined: gssapi.CredId
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:190:18: undefined: gssapi.CtxId
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:198:20: undefined: gssapi.GSS_C_INTEG_FLAG
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:198:54: undefined: gssapi.GSS_C_MUTUAL_FLAG
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:198:89: undefined: gssapi.GSS_C_SEQUENCE_FLAG
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:198:126: undefined: gssapi.GSS_C_CONF_FLAG
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:326:44: undefined: gssapi.Name
/go/pkg/mod/github.com/beltran/[email protected]/gssapi.go:198:126: too many errors
Am I missing something important, am I running a wrong version of GoHive or is there some other problem?
To me it seems that gssapi cannot be found despite it being installed.
here is my description on my hive: i can connect to my hive without username and password in normal, the setting hive.server2.authentication in hive-site.xml is NOSASL
a error Socket already connected appeared when I used go connect to hive, below is my main.go
hope you give me some advice after you view this issue, thanks
package main
import (
"context"
"log"
"github.com/beltran/gohive"
)
func main() {
async := false
ctx := context.Background()
configuration := gohive.NewConnectConfiguration()
// is this a database?
configuration.Service = "bulut"
configuration.FetchSize = 1000
connection, errConn := gohive.Connect("10.125.0.15", 10000, "NOSASL", configuration)
if errConn != nil {
// the error Socket already connected was occured here
log.Fatal(errConn)
}
cursor := connection.Cursor()
cursor.Execute(ctx, "SELECT * FROM goTest", async)
if cursor.Err != nil {
log.Fatal(cursor.Err)
}
var users []User
for cursor.HasMore(ctx) {
var user User
if cursor.Err != nil {
log.Fatal(cursor.Err)
}
cursor.FetchOne(ctx, &user.Id, &user.Name)
if cursor.Err != nil {
log.Fatal(cursor.Err)
}
users = append(users,user)
}
log.Println(users)
cursor.Close()
connection.Close()
}
type User struct {
Id int
Name string
}
if use zookeeper implement hiveserver2 ha , the host is more then one.
eg)
beeline -u "jdbc:hive2://devnode2,devnode1,devnode3/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2_zk;principal=hive/[email protected]
How can I rewrite this code in this case,can I write all hosts direct ?
gohive.Connect("hs2.example.com", 10000, "KERBEROS", configuration)
thanks !
Hello,
I use apache livy to start a thirft server for spark, the hive jdbc url just like this:
jdbc:hive2://localhost:10090/?livy.server.sessionId=0
And the hive url is localhost:10090
Does this package support to set the query parameter? (ex: livy.server.sessionId)
Should I set this in HiveConfiguration?
The conf.HiveConfiguration = map[string]string{"key": value}
How do I find out which keys can be set?
Thanks :)
connect kdc and hive has the same host can be ok, but how to do when they are different
We are running into an issue with connecting to Hive using LDAP + TLS.
The following snippet shows how we are utilizing the library:
cfg := gohive.NewConnectConfiguration()
cfg.Username = u.user
cfg.Password = u.password
cfg.Service = "hive"
cfg.Database = h.dbName
cfg.TransportMode = "http"'
cfg.TLSConfig = &tls.Config{}
connection, err := gohive.Connect(h.host, h.port, "NONE", cfg)
if err != nil {
log.Fatalf("gohive.Connect : Error : %s\n", err)
}
This yields the following exception on the client side:
2022-04-14T15:36:03.566-0700 FATAL pkg/log.go:143 gohive.Connect : Error : Error while executing query: TStatus({StatusCode:ERROR_STATUS InfoMessages:[*org.apache.hive.service.cli.HiveSQLException:Invalid SessionHandle: SessionHandle [463e200c-1693-430a-a3a0-3e13982353ce]:39:38 org.apache.hive.service.cli.session.SessionManager:getSession:SessionManager.java:579 org.apache.hive.service.cli.CLIService:executeStatementAsync:CLIService.java:308 org.apache.hive.service.cli.thrift.ThriftCLIService:ExecuteStatement:ThriftCLIService.java:562 org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1557 org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1542 org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39 org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39 org.apache.thrift.server.TServlet:doPost:TServlet.java:83 org.apache.hive.service.cli.thrift.ThriftHttpServlet:doPost:ThriftHttpServlet.java:207 javax.servlet.http.HttpServlet:service:HttpServlet.java:523 javax.servlet.http.HttpServlet:service:HttpServlet.java:590 org.eclipse.jetty.servlet.ServletHolder:handle:ServletHolder.java:852 org.eclipse.jetty.servlet.ServletHandler:doHandle:ServletHandler.java:544 org.eclipse.jetty.server.handler.ScopedHandler:nextHandle:ScopedHandler.java:233 org.eclipse.jetty.server.session.SessionHandler:doHandle:SessionHandler.java:1581 org.eclipse.jetty.server.handler.ScopedHandler:nextHandle:ScopedHandler.java:233 org.eclipse.jetty.server.handler.ContextHandler:doHandle:ContextHandler.java:1307 org.eclipse.jetty.server.handler.ScopedHandler:nextScope:ScopedHandler.java:188 org.eclipse.jetty.servlet.ServletHandler:doScope:ServletHandler.java:482 org.eclipse.jetty.server.session.SessionHandler:doScope:SessionHandler.java:1549 org.eclipse.jetty.server.handler.ScopedHandler:nextScope:ScopedHandler.java:186 org.eclipse.jetty.server.handler.ContextHandler:doScope:ContextHandler.java:1204 org.eclipse.jetty.server.handler.ScopedHandler:handle:ScopedHandler.java:141 org.eclipse.jetty.server.handler.gzip.GzipHandler:handle:GzipHandler.java:772 org.eclipse.jetty.server.handler.HandlerWrapper:handle:HandlerWrapper.java:127 org.eclipse.jetty.server.Server:handle:Server.java:494 org.eclipse.jetty.server.HttpChannel:handle:HttpChannel.java:374 org.eclipse.jetty.server.HttpConnection:onFillable:HttpConnection.java:268 org.eclipse.jetty.io.AbstractConnection$ReadCallback:succeeded:AbstractConnection.java:311 org.eclipse.jetty.io.FillInterest:fillable:FillInterest.java:103 org.eclipse.jetty.io.ChannelEndPoint$2:run:ChannelEndPoint.java:117 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:runTask:EatWhatYouKill.java:336 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:doProduce:EatWhatYouKill.java:313 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:tryProduce:EatWhatYouKill.java:171 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:run:EatWhatYouKill.java:129 org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread:run:ReservedThreadExecutor.java:367 java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1149 java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:624 java.lang.Thread:run:Thread.java:750] SqlState:<nil> ErrorCode:0xc00029c0ec ErrorMessage:0xc000308180})
Looking at the Hive logs it yields:
2022-04-14T17:53:28,518 INFO [etp1337659716-50] thrift.ThriftHttpServlet: Could not validate cookie sent, will try to generate a new cookie
2022-04-14T17:53:29,017 INFO [etp1337659716-50] thrift.ThriftHttpServlet: Cookie added for clientUserName tahmed
2022-04-14T17:53:29,131 WARN [etp1337659716-50] thrift.ThriftCLIService: Error executing statement:
org.apache.hive.service.cli.HiveSQLException: Invalid SessionHandle: SessionHandle [8fc66ab0-f620-4714-9799-2d1c5eed6345]
at org.apache.hive.service.cli.session.SessionManager.getSession(SessionManager.java:579) ~[hive-service-3.1.2-amzn-4.jar:3.1.2-amzn-4]
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:308) ~[hive-service-3.1.2-amzn-4.jar:3.1.2-amzn-4]
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:562) ~[hive-service-3.1.2-amzn-4.jar:3.1.2-amzn-4]
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557) ~[hive-exec-3.1.2-amzn-4.jar:3.1.2-amzn-4]
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542) ~[hive-exec-3.1.2-amzn-4.jar:3.1.2-amzn-4]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) ~[hive-exec-3.1.2-amzn-4.jar:3.1.2-amzn-4]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[hive-exec-3.1.2-amzn-4.jar:3.1.2-amzn-4]
at org.apache.thrift.server.TServlet.doPost(TServlet.java:83) ~[hive-exec-3.1.2-amzn-4.jar:3.1.2-amzn-4]
at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:207) ~[hive-service-3.1.2-amzn-4.jar:3.1.2-amzn-4]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:523) ~[jakarta.servlet-api-4.0.3.jar:4.0.3]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) ~[jakarta.servlet-api-4.0.3.jar:4.0.3]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:852) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:544) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1581) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1307) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:482) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1549) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1204) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.server.Server.handle(Server.java:494) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:374) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:268) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) ~[jetty-io-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:367) ~[jetty-runner-9.4.20.v20190813.jar:9.4.20.v20190813]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_322]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_322]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
After a lot of time spent debugging we determined the root cause was there being no cookie jar. We tried reusing the inMemoryCookieJar but kept seeing the following error message on the client side:
HTTP Response code: 500
We eventually realized the current implementation of inMemoryCookieJar is a little strange, namely it is not propagating all cookies received from the server. Finally, after adding the following implementation of inMemoryCookieJar, it worked for async=true queries.
type inMemoryCookieJar struct {
storage map[string][]*http.Cookie
}
func (jar inMemoryCookieJar) SetCookies(u *url.URL, cookies []*http.Cookie) {
jar.storage[u.Host] = cookies
}
func (jar inMemoryCookieJar) Cookies(u *url.URL) []*http.Cookie {
return jar.storage[u.Host]
}
There are still many questions that are yet to be answered:
HTTP 431
for sync queries but works perfectly for async queries. Why?A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.