Time difference b/w cursor.execute() function in PyAthena and Query execution time displayed in Athena Console about pyathena HOT 4 CLOSED

Is the value of execution_time_in_millis that can be retrieved from the cursor object the same as the value of Athena's web console?
Perhaps it seems that fetching data of PyAthena is slow. If the result set is large, using PandasCursor is faster.
https://github.com/laughingman7743/PyAthena#pandascursor

For the comparison result between Cursor and PandasCursor, refer to the following GIST.
https://gist.github.com/laughingman7743/2e4d83ca4e394dc645e9ea9a45fe78ba

from pyathena.

@laughingman7743 Yes, the value of execution_time_in_millis and the value of Athena's web console is the same.

I edited the execute function of the Cursor class as below and logging in the execution times. The conclusion of 2-5 seconds delay is based on this.

    @synchronized
    def execute(self, operation, parameters=None):
        start = time.time()
        self._reset_state()
        self._query_id = self._execute(operation, parameters)
        query_execution = self._poll(self._query_id)
        diff1 = time.time() - start
        logging.info("Time taken for polling to finish: " + str(diff1))
        if query_execution.state == AthenaQueryExecution.STATE_SUCCEEDED:
            self._result_set = AthenaResultSet(
                self._connection, self._converter, query_execution, self.arraysize,
                self.retry_exceptions, self.retry_attempt, self.retry_multiplier,
                self.retry_max_delay, self.retry_exponential_base)
            diff2 = time.time() - start
            logging.info("Time taken for returning the result " + str(diff2))
        else:
            raise OperationalError(query_execution.state_change_reason)

from pyathena.

PyAthena only internally calls boto3 API (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/athena.html). Is there a delay when executing directly in boto3?
Since PyAthena fetches 1000 rows when creating a ResultSet object, I think it is somewhat slow.

How about trying a new JDBC driver because it seems to process result set streaming?
https://docs.aws.amazon.com/athena/latest/ug/release-note-2018-08-16.html

from pyathena.

There is a delay even while executing directly from boto3. Please refer to this gist for the same.
https://gist.github.com/abhilash1994/3007d93352edea1dfbae04e132da41b7

from pyathena.