doesn't it maintain a session while executing multiple queries? about gohive HOT 12 CLOSED

Hello, I believe this is a hive limitation, so it can't be done, but you can create several connections and have a query per connection

connection1, _ := gohive.Connect("hs2.example.com", 10000, "KERBEROS", configuration)
connection2, _ := gohive.Connect("hs2.example.com", 10000, "KERBEROS", configuration)

cursor1 := connection1.Cursor()
cursor2 := connection2.Cursor()

cursor1.Exec(ctx, "insert overwrite table scm.mytable PARTITION(eventdate)")
cursor2.Exec(ctx, "select * from stg.stage_table")

from gohive.

I don't think its hive limitation because it works when tried executing the same query in python using pyhs2 package.

from gohive.

For some reason I thought you wanted to run concurrent queries on the same connection.

What you want to do should be working right now, let me look into why it's not.

from gohive.

Hello, this is actually working, but you have to remove the final ; at the end, it's the same in pyhs2, instead of
cursor.Exec(ctx, "set hive.exec.parallel = true;") it should be:
cursor.Exec(ctx, "set hive.exec.parallel = true")
Otherwhise hive.exec.parallel is set to true; instead of true. You also have can only do one SET statement at once.

from gohive.

Hi,
Actually, queries are executed individually without a semicolon at the end by converting into a string array using the split function. (same as pyhs2/pyhive)
so tried a couple of more permutation and combination

1. Each query without a semicolon at the end with a single connection and a single cursor
   result: no error but nothing happens, the query "insert overwrite .." yields nothing
   2 With a semicolon at the end of each query and with a single connection and a single cursor
   result: error for the last query ( insert overwrite... )
2. Without a semicolon at the end of each query and with a single connectin and new cursor for each query
   result: no error but nothing happens, the query "insert overwrite .." yields nothing

from gohive.

Hello, I'm not sure what's going on, I would think this test proves that setting the properties works as expected. If you could provide a POC of unexpected behavior that would be helpful.

The setting are set per session, so all of them should be executed on the same connection, then the queries should be executed on this same connection, if a new connection is created you'd need to set them all again. You could set the settings here when creating the connection so you don't have to do cursor.Exec("SET hive..... = value") for every single setting.

from gohive.

Please find attached code snippet
test .txt

from gohive.

I've been playing around with it for a bit. Everything seems fine. I've added at the end:

        cursor.Exec(context.Background(), "SET")
        for cursor.HasMore(context.Background()) {
                fmt.Println(cursor.RowMap(context.Background()))
        }

That lists all the hive properties and they are all set as expected, is this the case for you as well? I can't see any difference with pyhs2. I guess you see the difference in the contents of scm.mytable.

from gohive.

Yes, i could see expected hive properties are set but getting an error as attached
Though the same set of queries works fine in python

map[set:SET hive.exec.dynamic.partition.mode=nonstrict]
map[set:SET hive.exec.max.dynamic.partitions=100000]
map[set:SET hive.exec.max.dynamic.partitions.pernode=100000]
map[set:SET hive.execution.engine=mr]
map[set:SET mapred.job.name=testjob]
map[set:SET mapred.job.queue.name=default]

in python follow the same flow and it works over there
error.txt

EX
multipleQueries = jobQuery["Query"].split(";")
for query in multipleQueries:
with conn.cursor() as cur:
cur.execute(query)

from gohive.

Hello, when I run this script: main.txt, it ends without failure, however when I remove the line set hive.exec.dynamic.partition.mode = nonstrict; from the file I get the error from the previous error.txt file, something like:

panic: Error while executing query: TStatus({StatusCode:ERROR_STATUS InfoMessages:[*org.apache.hive.service.cli.HiveSQLException:Error while compiling statement: FAILED: SemanticException [Error 10096]: Dynamic partition strict mode requires at least one static partition column. To turn this off set hive.exec.dynamic.partition.mode=nonstrict:17:16 org.apache.hive.service.cli.operation.Operation:toSQLException:Operation

from gohive.

There is no issue with connection state, it is maintained ... the issue was leading or trailing space in hive set command doesn't set that particular property property

from gohive.

Thank you for debugging this and reporting back

from gohive.