Git Product home page Git Product logo

monetdbe-python's Introduction

MonetDB/e-Python

MonetDB/e-Python - the serverless Python embedded MonetDB

alt text

Want to store and search a massive amount of numbers? You don't want to run a database service? Is SQLite too slow for what you want to do? Search no further! MonetDBe-Python is here. Just pip install the binary wheel on your Windows, Linux or OS X system and you are ready to go, no compilation needed.

MonetDBe-Python internally relies on a serverless and trimmed-down version of the blazingly fast MonetDB, the open-source column-store database.

The documentation can be found at: https://monetdbe.readthedocs.io/ Documentation Status

The source code can be found at: https://github.com/monetdBSolutions/MonetDBe-Python/

install

you need:

  • An up-to-date Linux, OSX or Windows
  • pip >= 19.3
  • Python >= 3.7

to make sure you have a recent pip first upgrade pip:

$ pip install --upgrade pip

now you can install MonetDBe-Python with:

$ pip install monetdbe

usage

Just import and get started, no running a server required. Connecting without an argument starts an in-memory storage instance:

>>> from monetdbe import connect
>>> con = connect()
>>> con.execute('select * from tables').fetchdf()
      id               name  schema_id                                              query  type  system  commit_action  access  temporary
0   2001            schemas       2000                                               None    10    True              0       0          0
1   2007              types       2000                                               None    10    True              0       0          0
2   2016          functions       2000                                               None    10    True              0       0          0
3   2029               args       2000                                               None    10    True              0       0          0
4   2038          sequences       2000                                               None    10    True              0       0          0
..   ...                ...        ...                                                ...   ...     ...            ...     ...        ...
81  6650       storagemodel       2000  create view sys.storagemodel as\nselect "schem...    11    True              0       0          0
82  6661  tablestoragemodel       2000  create view sys.tablestoragemodel as\nselect "...    11    True              0       0          0
83  6675         statistics       2000                                               None    10    True              0       0          0
84  6734           compinfo       6698  create view logging.compinfo as select * from ...    11    True              0       0          0
85  6739    systemfunctions       2000  create view sys.systemfunctions as select id a...    11    True              0       0          0

See a another simple example illustrating the Pandas support in this notebook:

https://github.com/MonetDBSolutions/MonetDBe-Python/blob/master/notebooks/basic_example.ipynb

monetdbe-python's People

Contributors

aris-koning avatar arjenderijke avatar gijzelaerr avatar hannes avatar joerivanruth avatar njnes avatar sstalin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

monetdbe-python's Issues

Out of scenario slots when running test suite

halfway running the test suite MonetDB starts to give errors:

#main thread: createExceptionInternal: !ERROR: MALException:sql.start:42000!out of scenario slots
#main thread: createExceptionInternal: !ERROR: SQLException:SQLinitClient:42000!Catalogue not available
#main thread: createExceptionInternal: !ERROR: SQLException:SQLexitClient:42000!Catalogue not available
SQLException:SQLinitClient:42000!Catalogue not available

This is due to not cleaning up properly, could be datasets or open connections.

Basis issue with fetchone()

Example taken from the Python SQLite documentation. No matter what, an empty result set should be properly handled

import monetdbe
conn = monetdbe.connect('example.db')
conn.execute('''CREATE TABLE stocks (date text, trans text, symbol text, qty real, price real)''')
c = conn.cursor()
symbol = 'RHAT'
c.execute("SELECT * FROM stocks WHERE symbol = '%s'" % symbol)
t = ('RHAT',)
c.execute('SELECT * FROM stocks WHERE symbol=?', t)
print(c.fetchone())

Improve ? and : argument handling

I've made a very simple argument parsing implementation, but this doesn't probably support escaping. Should we do this server-side?

sqlite.intransaction

This property should be handled at the mapiclient library. It is necessary when
the autocommit mode is turned off.

A c-function is needed in the interface. Just assume it for now.

Crashing on MONETDB url (monetdb://localhost:5000/)

import monetdbe
remote = monetdbe.connect("monetdb://localhost:5000/sf1?user=monetdb&password=monetdb")

Gives:

monetdbe.exceptions.OperationalError: MALException:monetdbe.monetdbe_startup:Cannot add farm /home/gijs/Work/MonetDBe-Python/monetdb:/localhost:5000/sf1?user=monetdb&password=monetdb

Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

Compiling monetdb on centos7 fails, missing dl* symbols

[ 94%] Building C object sql/backends/monet5/UDF/udf/CMakeFiles/udf.dir/udf.c.o                                         
[ 94%] Building C object sql/backends/monet5/generator/CMakeFiles/generator.dir/90_generator_hge.sql.c.o                
[ 94%] Linking C shared library libembedded.so                                                                          
[ 94%] Building C object sql/backends/monet5/UDF/udf/CMakeFiles/udf.dir/80_udf.sql.c.o                                  
[ 94%] Building C object sql/backends/monet5/UDF/udf/CMakeFiles/udf.dir/80_udf_hge.sql.c.o                              
[ 94%] Built target embedded                                                                                            
Scanning dependencies of target example1                                                                                
[ 94%] Building C object ctest/tools/embedded/CMakeFiles/example1.dir/example1.c.o                                      
Scanning dependencies of target example2                                                                                
[ 94%] Building C object ctest/tools/embedded/CMakeFiles/example2.dir/example2.c.o                                      
[ 95%] Linking C executable example1                                                                                    
[ 96%] Linking C shared module lib_udf.so                                                                               
/usr/bin/ld: /build/MonetDB-default/build/monetdb5/tools/libmonetdb5.so.29: undefined reference to `dlsym'              
/usr/bin/ld: /build/MonetDB-default/build/monetdb5/tools/libmonetdb5.so.29: undefined reference to `dlerror'            
/usr/bin/ld: /build/MonetDB-default/build/monetdb5/tools/libmonetdb5.so.29: undefined reference to `dlopen'             
/usr/bin/ld: /build/MonetDB-default/build/monetdb5/tools/libmonetdb5.so.29: undefined reference to `dlclose'            
collect2: error: ld returned 1 exit status                                                                              
make[2]: *** [ctest/tools/embedded/CMakeFiles/example1.dir/build.make:85: ctest/tools/embedded/example1] Error 1        
make[1]: *** [CMakeFiles/Makefile2:3409: ctest/tools/embedded/CMakeFiles/example1.dir/all] Error 2                      
make[1]: *** Waiting for unfinished jobs....                                                                            
[ 96%] Linking C shared module lib_generator.so                                                                         
[ 96%] Linking C executable example2                                                                                    
[ 96%] Built target udf                                                                                                 
/usr/bin/ld: /build/MonetDB-default/build/monetdb5/tools/libmonetdb5.so.29: undefined reference to `dlsym'              
/usr/bin/ld: /build/MonetDB-default/build/monetdb5/tools/libmonetdb5.so.29: undefined reference to `dlerror'            
/usr/bin/ld: /build/MonetDB-default/build/monetdb5/tools/libmonetdb5.so.29: undefined reference to `dlopen'             
/usr/bin/ld: /build/MonetDB-default/build/monetdb5/tools/libmonetdb5.so.29: undefined reference to `dlclose'            
collect2: error: ld returned 1 exit status                                                                              
make[2]: *** [ctest/tools/embedded/CMakeFiles/example2.dir/build.make:85: ctest/tools/embedded/example2] Error 1        
make[1]: *** [CMakeFiles/Makefile2:3436: ctest/tools/embedded/CMakeFiles/example2.dir/all] Error 2                      
[ 96%] Built target generator                                                                                           
make: *** [Makefile:163: all] Error 2                                                                                   

monetdbe.complete_statement missing

From the Python SQLite documentation
3rd example to illustrate a simple frontend.

con = monetdbe.connect(":memory:")
con.isolation_level = None
cur = con.cursor()

buffer = ""

print("Enter your SQL commands to execute in monetdbe.")
print("Enter a blank line to exit.")

while True:
    line = input()
    if line == "":
        break
    buffer += line
    if monetdbe.complete_statement(buffer):
        try:
            buffer = buffer.strip()
            cur.execute(buffer)

            if buffer.lstrip().upper().startswith("SELECT"):
                print(cur.fetchall())
        except monetdbe.Error as e:
            print("An error occurred:", e.args[0])
        buffer = ""

con.close()

Enter your SQL commands to execute in monetdbe.
Enter a blank line to exit.
select 1;
Traceback (most recent call last):
  File "x.py", line 16, in <module>
    if monetdbe.complete_statement(buffer):
AttributeError: module 'monetdbe' has no attribute 'complete_statement'

INSERT OR REPLACE crashes monetdb embedded

create table test(id integer auto_increment primary key, name text, income float, unique_test text unique);
INSERT OR REPLACE INTO test(id, unique_test) VALUES (1, 'foo);

Building module gives compilation warnings

In file included from /home/gijs/opt/monetdb/include/monetdb/monetdb_embedded.h:12,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:541:
/home/gijs/opt/monetdb/include/monetdb/monetdb_config.h:13: warning: "HAVE_SYS_TYPES_H" redefined
   13 | #define HAVE_SYS_TYPES_H /usr/include/x86_64-linux-gnu
      | 
In file included from /usr/include/python3.8/pyconfig.h:3,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:22:
/usr/include/x86_64-linux-gnu/python3.8/pyconfig.h:1194: note: this is the location of the previous definition
 1194 | #define HAVE_SYS_TYPES_H 1
      | 
In file included from /home/gijs/opt/monetdb/include/monetdb/monetdb_embedded.h:12,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:541:
/home/gijs/opt/monetdb/include/monetdb/monetdb_config.h:31: warning: "HAVE_DLFCN_H" redefined
   31 | #define HAVE_DLFCN_H /usr/include
      | 
In file included from /usr/include/python3.8/pyconfig.h:3,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:22:
/usr/include/x86_64-linux-gnu/python3.8/pyconfig.h:274: note: this is the location of the previous definition
  274 | #define HAVE_DLFCN_H 1
      | 
In file included from /home/gijs/opt/monetdb/include/monetdb/monetdb_embedded.h:12,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:541:
/home/gijs/opt/monetdb/include/monetdb/monetdb_config.h:32: warning: "HAVE_FCNTL_H" redefined
   32 | #define HAVE_FCNTL_H /usr/include
      | 
In file included from /usr/include/python3.8/pyconfig.h:3,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:22:
/usr/include/x86_64-linux-gnu/python3.8/pyconfig.h:337: note: this is the location of the previous definition
  337 | #define HAVE_FCNTL_H 1
      | 
In file included from /home/gijs/opt/monetdb/include/monetdb/monetdb_embedded.h:12,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:541:
/home/gijs/opt/monetdb/include/monetdb/monetdb_config.h:36: warning: "HAVE_LANGINFO_H" redefined
   36 | #define HAVE_LANGINFO_H /usr/include
      | 
In file included from /usr/include/python3.8/pyconfig.h:3,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:22:
/usr/include/x86_64-linux-gnu/python3.8/pyconfig.h:573: note: this is the location of the previous definition
  573 | #define HAVE_LANGINFO_H 1
      | 
In file included from /home/gijs/opt/monetdb/include/monetdb/monetdb_embedded.h:12,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:541:
/home/gijs/opt/monetdb/include/monetdb/monetdb_config.h:38: warning: "HAVE_LIBINTL_H" redefined
   38 | #define HAVE_LIBINTL_H /usr/include
      | 
In file included from /usr/include/python3.8/pyconfig.h:3,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:22:
/usr/include/x86_64-linux-gnu/python3.8/pyconfig.h:603: note: this is the location of the previous definition
  603 | #define HAVE_LIBINTL_H 1
      | 
In file included from /home/gijs/opt/monetdb/include/monetdb/monetdb_embedded.h:12,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:541:
/home/gijs/opt/monetdb/include/monetdb/monetdb_config.h:44: warning: "HAVE_POLL_H" redefined
   44 | #define HAVE_POLL_H /usr/include
      | 
In file included from /usr/include/python3.8/pyconfig.h:3,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:22:
/usr/include/x86_64-linux-gnu/python3.8/pyconfig.h:750: note: this is the location of the previous definition
  750 | #define HAVE_POLL_H 1
      | 
In file included from /home/gijs/opt/monetdb/include/monetdb/monetdb_embedded.h:12,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:541:
/home/gijs/opt/monetdb/include/monetdb/monetdb_config.h:47: warning: "HAVE_STRINGS_H" redefined
   47 | #define HAVE_STRINGS_H /usr/include
      | 
In file included from /usr/include/python3.8/pyconfig.h:3,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:22:
/usr/include/x86_64-linux-gnu/python3.8/pyconfig.h:1039: note: this is the location of the previous definition
 1039 | #define HAVE_STRINGS_H 1
      | 
In file included from /home/gijs/opt/monetdb/include/monetdb/monetdb_embedded.h:12,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:541:
/home/gijs/opt/monetdb/include/monetdb/monetdb_config.h:49: warning: "HAVE_SYS_FILE_H" redefined
   49 | #define HAVE_SYS_FILE_H /usr/include/x86_64-linux-gnu
      | 
In file included from /usr/include/python3.8/pyconfig.h:3,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:22:
/usr/include/x86_64-linux-gnu/python3.8/pyconfig.h:1118: note: this is the location of the previous definition
 1118 | #define HAVE_SYS_FILE_H 1
      | 
In file included from /home/gijs/opt/monetdb/include/monetdb/monetdb_embedded.h:12,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:541:
/home/gijs/opt/monetdb/include/monetdb/monetdb_config.h:50: warning: "HAVE_SYS_IOCTL_H" redefined
   50 | #define HAVE_SYS_IOCTL_H /usr/include/x86_64-linux-gnu
      | 
In file included from /usr/include/python3.8/pyconfig.h:3,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:22:
/usr/include/x86_64-linux-gnu/python3.8/pyconfig.h:1121: note: this is the location of the previous definition
 1121 | #define HAVE_SYS_IOCTL_H 1
      | 
In file included from /home/gijs/opt/monetdb/include/monetdb/monetdb_embedded.h:12,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:541:
/home/gijs/opt/monetdb/include/monetdb/monetdb_config.h:52: warning: "HAVE_SYS_MMAN_H" redefined
   52 | #define HAVE_SYS_MMAN_H /usr/include/x86_64-linux-gnu
      | 
In file included from /usr/include/python3.8/pyconfig.h:3,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:22:
/usr/include/x86_64-linux-gnu/python3.8/pyconfig.h:1139: note: this is the location of the previous definition
 1139 | #define HAVE_SYS_MMAN_H 1
      | 
In file included from /home/gijs/opt/monetdb/include/monetdb/monetdb_embedded.h:12,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:541:
/home/gijs/opt/monetdb/include/monetdb/monetdb_config.h:53: warning: "HAVE_SYS_PARAM_H" redefined
   53 | #define HAVE_SYS_PARAM_H /usr/include/x86_64-linux-gnu
      | 
In file included from /usr/include/python3.8/pyconfig.h:3,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:22:
/usr/include/x86_64-linux-gnu/python3.8/pyconfig.h:1149: note: this is the location of the previous definition
 1149 | #define HAVE_SYS_PARAM_H 1
      | 
In file included from /home/gijs/opt/monetdb/include/monetdb/monetdb_embedded.h:12,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:541:
/home/gijs/opt/monetdb/include/monetdb/monetdb_config.h:54: warning: "HAVE_SYS_RESOURCE_H" redefined
   54 | #define HAVE_SYS_RESOURCE_H /usr/include/x86_64-linux-gnu
      | 
In file included from /usr/include/python3.8/pyconfig.h:3,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:22:
/usr/include/x86_64-linux-gnu/python3.8/pyconfig.h:1158: note: this is the location of the previous definition
 1158 | #define HAVE_SYS_RESOURCE_H 1
      | 
In file included from /home/gijs/opt/monetdb/include/monetdb/monetdb_embedded.h:12,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:541:
/home/gijs/opt/monetdb/include/monetdb/monetdb_config.h:56: warning: "HAVE_SYS_TIMES_H" redefined
   56 | #define HAVE_SYS_TIMES_H /usr/include/x86_64-linux-gnu
      | 
In file included from /usr/include/python3.8/pyconfig.h:3,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:22:
/usr/include/x86_64-linux-gnu/python3.8/pyconfig.h:1188: note: this is the location of the previous definition
 1188 | #define HAVE_SYS_TIMES_H 1
      | 
In file included from /home/gijs/opt/monetdb/include/monetdb/monetdb_embedded.h:12,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:541:
/home/gijs/opt/monetdb/include/monetdb/monetdb_config.h:57: warning: "HAVE_SYS_UIO_H" redefined
   57 | #define HAVE_SYS_UIO_H /usr/include/x86_64-linux-gnu
      | 
In file included from /usr/include/python3.8/pyconfig.h:3,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:22:
/usr/include/x86_64-linux-gnu/python3.8/pyconfig.h:1197: note: this is the location of the previous definition
 1197 | #define HAVE_SYS_UIO_H 1
      | 
In file included from /home/gijs/opt/monetdb/include/monetdb/monetdb_embedded.h:12,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:541:
/home/gijs/opt/monetdb/include/monetdb/monetdb_config.h:58: warning: "HAVE_SYS_UN_H" redefined
   58 | #define HAVE_SYS_UN_H /usr/include/x86_64-linux-gnu
      | 
In file included from /usr/include/python3.8/pyconfig.h:3,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:22:
/usr/include/x86_64-linux-gnu/python3.8/pyconfig.h:1200: note: this is the location of the previous definition
 1200 | #define HAVE_SYS_UN_H 1
      | 
In file included from /home/gijs/opt/monetdb/include/monetdb/monetdb_embedded.h:12,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:541:
/home/gijs/opt/monetdb/include/monetdb/monetdb_config.h:60: warning: "HAVE_SYS_WAIT_H" redefined
   60 | #define HAVE_SYS_WAIT_H /usr/include/x86_64-linux-gnu
      | 
In file included from /usr/include/python3.8/pyconfig.h:3,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:22:
/usr/include/x86_64-linux-gnu/python3.8/pyconfig.h:1206: note: this is the location of the previous definition
 1206 | #define HAVE_SYS_WAIT_H 1
      | 
In file included from /home/gijs/opt/monetdb/include/monetdb/monetdb_embedded.h:12,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:541:
/home/gijs/opt/monetdb/include/monetdb/monetdb_config.h:61: warning: "HAVE_TERMIOS_H" redefined
   61 | #define HAVE_TERMIOS_H /usr/include/x86_64-linux-gnu
      | 
In file included from /usr/include/python3.8/pyconfig.h:3,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:22:
/usr/include/x86_64-linux-gnu/python3.8/pyconfig.h:1221: note: this is the location of the previous definition
 1221 | #define HAVE_TERMIOS_H 1
      | 
In file included from /home/gijs/opt/monetdb/include/monetdb/monetdb_embedded.h:12,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:541:
/home/gijs/opt/monetdb/include/monetdb/monetdb_config.h:63: warning: "HAVE_UUID_UUID_H" redefined
   63 | #define HAVE_UUID_UUID_H /usr/include
      | 
In file included from /usr/include/python3.8/pyconfig.h:3,
                 from build/temp.linux-x86_64-3.8/_monetdbe_cffi.c:22:
/usr/include/x86_64-linux-gnu/python3.8/pyconfig.h:1300: note: this is the location of the previous definition
 1300 | #define HAVE_UUID_UUID_H 1
      | 

Leftover .gdk_lock and uuid

If you abort an open database connection in the Python application (or hangup) then
the database directory may still contain the .gdk_lock and UUID file

They should have been removed automatically with a signal handler

Fetching empy table: : Assertion `size != 0' failed.

from tempfile import TemporaryDirectory
from monetdbe import connect
cur = connect(TemporaryDirectory().name).cursor()
cur.execute("create table test(i int)")
cur.execute("select i from test")
res = cur.fetchall()

crashes with:

python: /home/gijs/Work/MonetDB/gdk/gdk_utils.c:1669: GDKmalloc_internal: Assertion `size != 0' failed.

on the current default branch

error: unknown type name ‘monetdb_table’;

$ python ./setup.py bdist_wheel -running bdist_wheel                                                                                                                                                                                                                                                          
running build                                                                                                                                                                                                                                                                
running build_py                                                                                                                                                                                                                                                             
running build_ext                                                                                                                                                                                                                                                            
generating cffi module 'build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c'                                                                                                                                                                                                    
already up-to-date                                                                                                                                                                                                                                                           
building 'monetdbe._lowlevel' extension                                                                                                                                                                                                                                      
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/opt/python/cp36-cp36m/include/python3.6m -c build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c -o build/temp.linux-x86_64-3.6/build/temp.linux-x86_64-3.6/monetdbe._lowlevel.o           
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c: In function ‘_cffi_d_monetdb_append’:                                                                                                                                                                                      
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c:1055:37: warning: passing argument 4 of ‘monetdb_append’ from incompatible pointer type [-Wincompatible-pointer-types]                                                                                                      
   return monetdb_append(x0, x1, x2, x3, x4);                                                                                                                                                                                                                                
                                     ^~                                                                                                                                                                                                                                      
In file included from build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c:541:                                                                                                                                                                                                  
/usr/local/include/monetdb/monetdb_embedded.h:154:23: note: expected ‘bat *’ {aka ‘int *’} but argument is of type ‘monetdb_column **’ {aka ‘struct <anonymous> **’}                                                                                                         
 embedded_export char* monetdb_append(monetdb_connection conn, const char* schema, const char* table, bat *batids, size_t column_count);                                                                                                                                     
                       ^~~~~~~~~~~~~~                                                                                                                                                                                                                                        
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c: In function ‘_cffi_f_monetdb_append’:                                                                                                                                                                                      
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c:1121:41: warning: passing argument 4 of ‘monetdb_append’ from incompatible pointer type [-Wincompatible-pointer-types]                                                                                                      
   { result = monetdb_append(x0, x1, x2, x3, x4); }                                                                                                                                                                                                                          
                                         ^~                                                                                                                                                                                                                                  
In file included from build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c:541:                                                                                                                                                                                                  
/usr/local/include/monetdb/monetdb_embedded.h:154:23: note: expected ‘bat *’ {aka ‘int *’} but argument is of type ‘monetdb_column **’ {aka ‘struct <anonymous> **’}                                                                                                         
 embedded_export char* monetdb_append(monetdb_connection conn, const char* schema, const char* table, bat *batids, size_t column_count);                                                                                                                                     
                       ^~~~~~~~~~~~~~                                                                                                                                                                                                                                        
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c: At top level:                                                                                                                                                                                                              
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c:1461:52: error: unknown type name ‘monetdb_table’; did you mean ‘monetdb_types’?                                                                                                                                            
 static char * _cffi_d_monetdb_get_table(void * x0, monetdb_table * * x1, char const * x2, char const * x3)                                                                                                                                                                  
                                                    ^~~~~~~~~~~~~                                                                                                                                                                                                            
                                                    monetdb_types                                                                                                                                                                                                            
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c: In function ‘_cffi_f_monetdb_get_table’:                                                                                                                                                                                   
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c:1470:3: error: unknown type name ‘monetdb_table’; did you mean ‘monetdb_types’?                                                                                                                                             
   monetdb_table * * x1;                                                                                                                                                                                                                                                     
   ^~~~~~~~~~~~~                                                                                                                                                                                                                                                             
   monetdb_types                                                                                                                                                                                                                                                             
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c:1497:39: error: ‘monetdb_table’ undeclared (first use in this function); did you mean ‘monetdb_time’?                                                                                                                       
     x1 = ((size_t)datasize) <= 640 ? (monetdb_table * *)alloca((size_t)datasize) : NULL;                                                                                                                                                                                    
                                       ^~~~~~~~~~~~~                                                                                                                                                                                                                         
                                       monetdb_time                                                                                                                                                                                                                          
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c:1497:39: note: each undeclared identifier is reported only once for each function it appears in                                                                                                                             
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c:1497:56: error: expected expression before ‘)’ token                                                                                                                                                                        
     x1 = ((size_t)datasize) <= 640 ? (monetdb_table * *)alloca((size_t)datasize) : NULL;                                                                                                                                                                                    
                                                        ^                                                                                                                                                                                                                    
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c:1523:36: warning: passing argument 2 of ‘monetdb_get_table’ from incompatible pointer type [-Wincompatible-pointer-types]                                                                                                   
   { result = monetdb_get_table(x0, x1, x2, x3); }                                                                                                                                                                                                                           
                                    ^~                                                                                                                                                                                                                                       
In file included from build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c:541:                                                                                                                                                                                                  
/usr/local/include/monetdb/monetdb_embedded.h:156:23: note: expected ‘sql_table **’ {aka ‘struct sql_table **’} but argument is of type ‘int **’                                                                                                                             
 embedded_export char* monetdb_get_table(monetdb_connection conn, sql_table** table, const char* schema_name, const char* table_name);                                                                                                                                       
                       ^~~~~~~~~~~~~~~~~                                                                                                                                                                                                                                     
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c: In function ‘_cffi_d_monetdb_in_transaction’:                                                                                                                                                                              
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c:1538:10: warning: implicit declaration of function ‘monetdb_in_transaction’; did you mean ‘_cffi_d_monetdb_in_transaction’? [-Wimplicit-function-declaration]                                                               
   return monetdb_in_transaction(x0);                                                                                                                                                                                                                                        
          ^~~~~~~~~~~~~~~~~~~~~~                                                                                                                                                                                                                                             
          _cffi_d_monetdb_in_transaction                                                                                                                                                                                                                                     
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c: In function ‘_cffi_d_monetdb_result_fetch’:                                                                                                                                                                                
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c:1688:35: warning: passing argument 2 of ‘monetdb_result_fetch’ from incompatible pointer type [-Wincompatible-pointer-types]                                                                                                
   return monetdb_result_fetch(x0, x1, x2, x3);                                                                                                                                                                                                                              
                                   ^~                                                                                                                                                                                                                                        
In file included from build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c:541:                                                                                                                                                                                                  
/usr/local/include/monetdb/monetdb_embedded.h:148:23: note: expected ‘monetdb_column **’ {aka ‘struct <anonymous> **’} but argument is of type ‘monetdb_result *’ {aka ‘struct <anonymous> *’}                                                                               
 embedded_export char* monetdb_result_fetch(monetdb_connection conn, monetdb_column** res, monetdb_result* mres, size_t column_index);                                                                                                                                       
                       ^~~~~~~~~~~~~~~~~~~~                                                                                                                                                                                                                                  
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c:1688:39: warning: passing argument 3 of ‘monetdb_result_fetch’ from incompatible pointer type [-Wincompatible-pointer-types]                                                                                                
   return monetdb_result_fetch(x0, x1, x2, x3);                                                                                                                                                                                                                              
                                       ^~                                                                                                                                                                                                                                    
In file included from build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c:541:                                                                                                                                                                                                  
/usr/local/include/monetdb/monetdb_embedded.h:148:23: note: expected ‘monetdb_result *’ {aka ‘struct <anonymous> *’} but argument is of type ‘monetdb_column **’ {aka ‘struct <anonymous> **’}                                                                               
 embedded_export char* monetdb_result_fetch(monetdb_connection conn, monetdb_column** res, monetdb_result* mres, size_t column_index);                                                                                                                                       
                       ^~~~~~~~~~~~~~~~~~~~                                                                                                                                                                                                                                  
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c: In function ‘_cffi_f_monetdb_result_fetch’:                                                                                                                                                                                
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c:1743:39: warning: passing argument 2 of ‘monetdb_result_fetch’ from incompatible pointer type [-Wincompatible-pointer-types]                                                                                                
   { result = monetdb_result_fetch(x0, x1, x2, x3); }                                                                                                                                                                                                                        
                                       ^~                                                                                                                                                                                                                                    
In file included from build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c:541:                                                                                                                                                                                                  
/usr/local/include/monetdb/monetdb_embedded.h:148:23: note: expected ‘monetdb_column **’ {aka ‘struct <anonymous> **’} but argument is of type ‘monetdb_result *’ {aka ‘struct <anonymous> *’}                                                                               
 embedded_export char* monetdb_result_fetch(monetdb_connection conn, monetdb_column** res, monetdb_result* mres, size_t column_index);                                                                                                                                       
                       ^~~~~~~~~~~~~~~~~~~~                                                                                                                                                                                                                                  
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c:1743:43: warning: passing argument 3 of ‘monetdb_result_fetch’ from incompatible pointer type [-Wincompatible-pointer-types]                                                                                                
   { result = monetdb_result_fetch(x0, x1, x2, x3); }                                                                                                                                                                                                                        
                                           ^~                                                                                                                                                                                                                                
In file included from build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c:541:                                                                                                                                                                                                  
/usr/local/include/monetdb/monetdb_embedded.h:148:23: note: expected ‘monetdb_result *’ {aka ‘struct <anonymous> *’} but argument is of type ‘monetdb_column **’ {aka ‘struct <anonymous> **’}                                                                               
 embedded_export char* monetdb_result_fetch(monetdb_connection conn, monetdb_column** res, monetdb_result* mres, size_t column_index);                                                                                                                                       
                       ^~~~~~~~~~~~~~~~~~~~                                                                                                                                                                                                                                  
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c: At top level:                                                                                                                                                                                                              
build/temp.linux-x86_64-3.6/monetdbe._lowlevel.c:1939:108: error: ‘_cffi_d_monetdb_get_table’ undeclared here (not in a function); did you mean ‘_cffi_f_monetdb_get_table’?                                                                                                 
   { "monetdb_get_table", (void *)_cffi_f_monetdb_get_table, _CFFI_OP(_CFFI_OP_CPYTHON_BLTN_V, 52), (void *)_cffi_d_monetdb_get_table },                                                                                                                                     
                                                                                                            ^~~~~~~~~~~~~~~~~~~~~~~~~                                                                                                                                        
                                                                                                            _cffi_f_monetdb_get_table                                                                                                                                        
error: command 'gcc' failed with exit status 1                                                                                                                                                                                                                               
make: *** [Makefile:20: wheels] Error 1                                                                                                                                                                                                                                      

Unable to load the data

Running the taxi.py example, the system aborts on the copy into. The manual interaction with the server works great.

def loaddb():
    clk = time.time()
    cursor = conn.cursor()
    cursor.execute("""
    COPY OFFSET 2 INTO yellow_tripdata_2016_01 FROM '/ufs/mk/repository/monetdbe-examples/data/yellow_tripdata_2016-01.csv' delimiters ',','\n'  best effort
    """)
    cursor.execute("SELECT count(*) FROM yellow_tripdata_2016_01")
    print(cursor.fetchone())
    cursor.close()
    print("Load database %6.3f milliseconds"  %(time.time() - clk))
(python) mk@toke::~/repository/monetdbe-examples/src/python> python taxi.py
Create database  0.001 milliseconds
#main thread: GDKfdlocate: !ERROR: no name specified
#main thread: GDKsave: !ERROR: failed name=, ext=tail, mode 0
!FATAL: COMMIT: transaction commit failed (perhaps your disk is full?) exiting (kernel error: (null))

Iteration over a result set causes issues

Another one is taken from the Python SQL documentation. monetdbe 0.6

import monetdbe
conn = monetdbe.connect('example.db')
conn.execute('''CREATE TABLE stocks (date text, trans text, symbol text, qty real, price real)''')
c = conn.cursor()
purchases = [('2006-03-28', 'BUY', 'IBM', 1000, 45.00),
             ('2006-04-05', 'BUY', 'MSFT', 1000, 72.00),
             ('2006-04-06', 'SELL', 'IBM', 500, 53.00),
            ]
c.executemany('INSERT INTO stocks VALUES (?,?,?,?,?)', purchases)
for row in c.execute('SELECT * FROM stocks ORDER BY price'):
    print(row)

gives:

... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/ufs/mk/.virtualenvs/mk-OjYiXP4V/lib/python3.7/site-packages/monetdbe/cursor.py", line 59, in __iter__
    self._set_description(columns)
  File "/ufs/mk/.virtualenvs/mk-OjYiXP4V/lib/python3.7/site-packages/monetdbe/cursor.py", line 53, in _set_description
    *list(zip(name, type_code, display_size, internal_size, precision, scale, null_ok)))
TypeError: _make() takes 2 positional arguments but 6 were given

Nested exceptions

The following code:

import monetdbe
conn = monetdbe.connect('example.db')
conn.commit()

triggers a nested Exception handling message:

#main thread: createExceptionInternal: !ERROR: SQLException:sql.trans:2DM30!COMMIT: not allowed in auto commit mode
SQLException:sql.trans:2DM30!COMMIT: not allowed in auto commit mode
Traceback (most recent call last):
  File "/ufs/mk/.virtualenvs/mk-OjYiXP4V/lib/python3.7/site-packages/monetdbe/cursor.py", line 94, in execute
    self.result, self.rowcount, self.prepare_id = self.connection.lowlevel.query(formatted, make_result=True)
  File "/ufs/mk/.virtualenvs/mk-OjYiXP4V/lib/python3.7/site-packages/monetdbe/_cffi.py", line 170, in query
    check_error(lib.monetdb_query(_connection, query.encode(), p_result, affected_rows, prepare_id))
  File "/ufs/mk/.virtualenvs/mk-OjYiXP4V/lib/python3.7/site-packages/monetdbe/_cffi.py", line 48, in check_error
    raise exceptions.DatabaseError(decoded)
monetdbe.exceptions.DatabaseError: SQLException:sql.trans:2DM30!COMMIT: not allowed in auto commit mode

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/ufs/mk/.virtualenvs/mk-OjYiXP4V/lib/python3.7/site-packages/monetdbe/connection.py", line 81, in commit
    return self.execute("COMMIT")
  File "/ufs/mk/.virtualenvs/mk-OjYiXP4V/lib/python3.7/site-packages/monetdbe/connection.py", line 68, in execute
    cur = Cursor(con=self).execute(query, args)
  File "/ufs/mk/.virtualenvs/mk-OjYiXP4V/lib/python3.7/site-packages/monetdbe/cursor.py", line 96, in execute
    raise OperationalError(e)
monetdbe.exceptions.OperationalError: SQLException:sql.trans:2DM30!COMMIT: not allowed in auto commit mode

Get MonetDBlite test suite working (tests/test_lite/test_pylite00.py)

for MonetDBlite these queries are all executed on the same database:

        monetdbe.sql('CREATE TABLE pylite00 (i INTEGER)')
        monetdbe.sql('INSERT INTO pylite00 VALUES (1), (2), (3), (4), (5)')
        result = monetdbe.sql('SELECT * FROM pylite00')

because how we manage multiple connection with monetdbe, every statement is executed in a new in-memory database.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.