Git Product home page Git Product logo

crystal-pg's Introduction

crystal-pg

A native, non-blocking Postgres driver for Crystal

CI

usage

This driver now uses the crystal-db project. Documentation on connecting, querying, etc, can be found at:

shards

Add this to your shard.yml on a generated crystal project, and run shards install

dependencies:
  pg:
    github: will/crystal-pg

Example usage

require "db"
require "pg"

DB.open("postgres://user:pass@host:port/db_name?option1=a&option2=b") do |db|
   ... use db ...
end

More

crystal-pg also supports some functionality past the typical crystal-db usage:

Listen/Notify

There are two ways to listen for notifications. For docs on NOTIFY, please read https://www.postgresql.org/docs/current/static/sql-notify.html.

  1. Any connection can be given a callback to run on notifications. However they are only received when other traffic is going on.
  2. A special listen-only connection can be established for instant notification processing with PG.connect_listen.
# see full example in examples/listen_notify.cr
PG.connect_listen("postgres:///", "a", "b") do |n| # connect and  listen on "a" and "b"
  puts "    got: #{n.payload} on #{n.channel}"     # print notifications as they come in
end

Arrays

Crystal-pg supports several popular array types. If you only need a 1 dimensional array, you can cast down to the appropriate Crystal type:

PG_DB.query_one("select ARRAY[1, null, 3]", &.read(Array(Int32?))
# => [1, nil, 3]

PG_DB.query_one("select '{hello, world}'::text[]", &.read(Array(String))
# => ["hello", "world"]

Error Handling

It is possible to catch errors and notifications and pass them along to Crystal for further handling.

DB.connect("postgres:///") do |cnn|
  # Capture and print all exceptions
  cnn.on_notice { |x| puts "pgSQL #{x}" }

  # A function that raises exceptions
  cnn.exec(
    <<-SQL
      CREATE OR REPLACE FUNCTION foo(IN str TEXT)
        RETURNS VOID
        LANGUAGE 'plpgsql'
        AS $$
          BEGIN
            IF str = 'yes' THEN
                    RAISE NOTICE 'Glad we agree!';
            ELSE
              RAISE EXCEPTION 'You know nothing John Snow!';
            END IF;
          END;
        $$;
    SQL
  )

  # Notice handling example
  cnn.exec(
    <<-SQL
      SELECT foo('yes');
    SQL
  )
  # => pgSQL NOTICE: Glad we agree!

  # Exception handling example
  cnn.exec(
    <<-SQL
      SELECT foo('no');
    SQL
  )
  # => pgSQL ERROR: You know nothing John Snow!
  #    Unhandled exception: You know nothing John Snow! (PQ::PQError)
  #     from lib/pg/src/pq/connection.cr:203:7 in 'handle_error'
  #     from lib/pg/src/pq/connection.cr:186:7 in 'handle_async_frames'
  #     from lib/pg/src/pq/connection.cr:162:7 in 'read'
  #     from lib/pg/src/pq/connection.cr:386:18 in 'expect_frame'
  #     from lib/pg/src/pq/connection.cr:370:9 in 'read_next_row_start'
  #     from lib/pg/src/pg/result_set.cr:39:8 in 'move_next'
  #     from lib/pg/src/pg/statement.cr:39:13 in 'perform_exec'
  #     from lib/db/src/db/statement.cr:82:14 in 'perform_exec_and_release'
  #     from lib/db/src/db/statement.cr:68:7 in 'exec:args'
  #     from lib/db/src/db/query_methods.cr:271:7 in 'exec'
  #     from spec/cerebrum_spec.cr:84:3 in '__crystal_main'
  #     from /usr/share/crystal/src/crystal/main.cr:97:5 in 'main_user_code'
  #     from /usr/share/crystal/src/crystal/main.cr:86:7 in 'main'
  #     from /usr/share/crystal/src/crystal/main.cr:106:3 in 'main'
  #     from __libc_start_main
  #     from _start
  #     from ???

Requirements

Crystal-pg is regularly tested on the Postgres versions the Postgres project itself supports. Since it uses protocol version 3, older versions probably also work but are not guaranteed.

Supported Datatypes

  • text
  • boolean
  • int8, int4, int2
  • float4, float8
  • timestamptz, date, timestamp (but no one should use ts when tstz exists!)
  • json and jsonb
  • uuid
  • bytea
  • numeric/decimal (1)
  • varchar
  • regtype
  • geo types: point, box, path, lseg, polygon, circle, line
  • array types: int8, int4, int2, float8, float4, bool, text, numeric, timestamptz, date, timestamp
  • interval (2)

1: A note on numeric: In Postgres this type has arbitrary precision. In this driver, it is represented as a PG::Numeric which retains all precision, but if you need to do any math on it, you will probably need to cast it to a float first. If you need true arbitrary precision, you can optionally require pg_ext/big_rational which adds #to_big_r, but requires that you have LibGMP installed.

2: A note on interval: A Postgres interval can not be directly mapped to a built in Crystal datatype. Therfore we provide a PG::Interval type that can be converted to Time::Span and Time::MonthSpan.

Authentication Methods

By default this driver will accept scram-sha-256 and md5, as well as trust. However cleartext is disabled by default. You can control exactly which auth methods the client will accept by passing in a comma separated list to the auth_methods parameter, for example

 DB.open("postgres://example.com/dbname?auth_methods=cleartext,md5,scram-sha-256")

DO NOT TURN cleartext ON UNLESS YOU ABSOLUTELY NEED IT! Merely by having this option enabled exposes a postgres client to downgrade man-in-the-middle attacks, even if the server is configured to not support cleartext. Even if you use TLS, you are not safe unless you are fully verifying the server's cert, as the attacker can terminate TLS and re-negotiate a connection with the server.

client                     attacker                     server
----------------------------------------------------------------------------
I want to connect \
                   \->  intercepts, forwards
                        I want to connect \
                                           \----->  receives connection request

                                                  / I support scram and/or md5 only
                        intercepts, sends      <-/
                     /  I only support cleartext
receives attacker <-/
claiming server
only supports cleartext
sends password because
cleartext enabled \
                   \->  receives clear password,
                        negotiates scram/md5
                        with real server      \
                                               \--> accepts scram/md5 auth

It is a mistake for any driver to support cleartext by default, and it's a mistake that postgres continues to have this as an option at all.

crystal-pg's People

Contributors

artlinkov avatar asterite avatar bcardiff avatar bigtunacan avatar cyberdelia avatar felipeelias avatar greenbigfrog avatar j8r avatar jgaskins avatar jhass avatar jwoertink avatar louisegrandjonc avatar matthewmcgarvey avatar miketheman avatar omarroth avatar radiospiel avatar rdp avatar robcole avatar rx14 avatar sanderhahn avatar straight-shoota avatar tebakane avatar technorama avatar vladfaust avatar waghanza avatar werner avatar will avatar wishdev avatar ysbaddaden avatar zewebdev avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

crystal-pg's Issues

Connection Pool

Hello,

When putting this into a webapp, do you put a connection pool around this?

Thanks,
Keenan

Syntax error from using group

I have a table that has a column called "group". Trying to query with this fails:

icr(0.21.1) > PG.query_all("SELECT group FROM hosted_ads", as: {String})
syntax error at or near "FROM" (PQ::PQError)
0x1072e6ce5: *CallStack::unwind:Array(Pointer(Void)) at ??
0x1072e6c81: *CallStack#initialize:Array(Pointer(Void)) at ??
0x1072e6c58: *CallStack::new:CallStack at ??
0x1072b3441: *raise<Exception+>:NoReturn at ??
0x10746b17a: *DB::Statement+@DB::Statement#perform_query_with_rescue<Tuple()>:PG::ResultSet at ??
0x10746b11a: *DB::Statement+@DB::Statement#query:PG::ResultSet at ??
0x10748369d: *DB::PoolStatement+@DB::PoolStatement#query:PG::ResultSet at ??
0x10744ffb5: *DB::Database@DB::QueryMethods#query<String>:PG::ResultSet at ??
0x10744feb9: *DB::Database@DB::QueryMethods#query_all:as<String, Tuple(String:Class)>:Array(String) at ??
0x1072e5361: *__icr_exec__:Array(String) at ??
0x1072954a7: __crystal_main at ??
0x1072a7198: main at ??

I'm using Crystal 0.21.1 with:

db:
  github: crystal-lang/crystal-db
  version: 0.4.1
pg:
  github: will/crystal-pg
  commit: cafe681e9fbc520b5152632e22571bbef88a68d1

How to get count(*) value from a SQL request ?

I'm trying to get value from a very basic command :

SELECT COUNT(*) FROM users

but rows always return [[0]](even when I have some lines on my table)

So for now I just do a "SELECT id FROM users" and I check row length

Tiny change needed for Crystal 0.9 support

This line requires a space after the y in query :D

Syntax error in ./libs/postgresql/pg/connection.cr:29: space required before colon in type restriction

    def exec_all(query: String)

Get raw response

Is there a way, or could there be a way to just get the raw response from postgres? I want to write a query using json_build_object() which postgres will return as JSON. Then with something like this, it would be awesome to do

result = DB.exec("SELECT json_build_object('id', id, 'username', username) FROM members LIMIT 1000").raw
JSON.parse(result)

Index out of bounds for empty resultset

When trying to access the first row an empty result like below

results = DB.exec("SELECT * FROM persons") # Empty resultset

pp results.rows.first

The program terminates itself because of error

Index out of bounds (IndexOutOfBounds)

Theoritically this is OK. But we can't know if the table is empty or not beforehand.

What do you think about this?

Dependency problem with pg 0.13.2

Hi, here is the output:

require "pg"
^
in lib/pg/src/pg.cr:2: while requiring "./pg/*"
require "./pg/*"
^
in lib/pg/src/pg/connection.cr:1: while requiring "../pq/*"
require "../pq/*"
^
in lib/pg/src/pq/connection.cr:2: while requiring "digest/md5": can't find file 'digest/md5' relative to '/.../lib/pg/src/pq'
require "digest/md5"

version mismatches

The versions in both CHANGELOG and shard.yml are messed up. The most problematic is that shard.yml states v0.5.0 for both versions which is confusing: while shards install states "installing pg 0.6.0" a shards list states "using pg (0.5.0)".

Would it be possible to release v0.6.1 and v0.7.1 with corrected shard.yml?

Decoding types with unstable OIDs

Hi @will (hope you're well!)

I'm working on a Crystal side-project at the moment that uses crystal-pg. It's working with a legacy database which has an eclectic mix of types that would have different OIDs from DB to DB and thus can't have statically defined decoders, as crystal-pg does.

I'm thinking of course of citext, hstore, and, in my case anyway, custom DOMAINs based on other types.

I've been looking through the source code for crystal-pg (very readable, btw) and it occurs to me that to support something like hstore, the decoder map would have to be built up on a per-connection basis (vs globally) by querying pg_type et al as Ruby's PG gem does.

I've not played with the pg_type table much, but I wonder if the typcategory column could be used as a hint for a fallback decoder to use where a specialised one wasn't available. That might allow citext (or for e.g. an DOMAIN "email" based on citext) to naturally be decoded as would text.

I whipped up a query to get familiar with what the types in my DB looked like and the relationships between them, and I think something like this could work.

Query:

WITH

categories(typcategory, category) AS (
  VALUES
    ('A', 'Array')
  , ('B', 'Boolean')
  , ('C', 'Composite')
  , ('D', 'Date/time')
  , ('E', 'Enum')
  , ('G', 'Geometric')
  , ('I', 'Network address')
  , ('N', 'Numeric')
  , ('P', 'Pseudo-types')
  , ('S', 'String')
  , ('T', 'Timespan')
  , ('U', 'User-defined')
  , ('V', 'Bit-string')
  , ('X', 'unknown')
),

types AS (
  SELECT
                                        oid
  , typtype                             "type"
  , typnamespace                        namespace_oid
  , typnamespace::regnamespace          namespace
  , typname                             "name"
  , typcategory                         category
  , (typcategory || ' - ' || category)  category_name
  , typbasetype                         base_oid
  , typelem                             element_oid
  , typarray                            array_oid
  , typdelim                            array_delimeter
  FROM pg_type JOIN categories USING (typcategory)
  WHERE typisdefined = 't'
)

SELECT
  types.oid
, types.namespace
, types.name
, types.category_name
, CASE WHEN base.oid    IS NOT NULL THEN (base.oid,    base.namespace,    base.name,    base.category)    ELSE NULL END base
, CASE WHEN element.oid IS NOT NULL THEN (element.oid, element.namespace, element.name, element.category) ELSE NULL END element
, CASE WHEN "array".oid IS NOT NULL THEN ("array".oid, "array".namespace, "array".name)                   ELSE NULL END "array"
FROM types
  LEFT JOIN types base    ON types.base_oid    = base.oid
  LEFT JOIN types element ON types.element_oid = element.oid
  LEFT JOIN types "array" ON types.array_oid   = "array".oid
WHERE types.type IN ('b', 'd');

I think there's enough information here to derive some types and it can be used to locate the OIDs of types with dedicated decoders, like hstores.

Alternatively, upon encountering a value with unknown oid, a SELECT $1::oid::regtype; could be run to try resolve it to a known name and use that to map to a decoder.

Either way, it would still have to build up the OID mapping on a per-connection basis.

It's late here so the above might be a bit rambly. I'm going to keep thinking about this but I was just curious if there has been other discussion already about handling more types.

Heroku deploy

Hi, I'm trying to deploy an app in heroku using crystal-pg, but I have the next errors:

remote: -----> Compiling main.cr
remote: /usr/bin/ld: /tmp/crystal/bin/../embedded/lib/../lib/libgc.a(os_dep.o): undefined reference to symbol '_end'
remote: //lib/x86_64-linux-gnu/libcom_err.so.2: error adding symbols: DSO missing from command line
remote: collect2: error: ld returned 1 exit status
remote: Error: execution of command failed with code: 1: `cc -o main "/tmp/build_9e3bf6082d648b8407cde8b368afff5c/.crystal/tmp/build_9e3bf6082d648b8407cde8b368afff5c/main.cr/main.o"   -lpq -I`pg_config --includedir` -L`pg_config --libdir` -lssl -levent -lrt -lpcl -lpcre -lm -lgc -lpthread -lunwind`

Do you have any idea how to solve it?. Thanks.

Segmentation fault

I tried to write some stuff and broke it:

maciek@mothra:~/code/aux/crystal-pg$ crystal spec
Segmentation fault (core dumped)
Program terminated abnormally with error code: 35584

See msakrejda@4da9d40

PQ(T) is not a module, it's a generic class


require "pg"
^

in lib/pg/src/pg.cr:2: while requiring "./pg/*"

require "./pg/*"
^

in lib/pg/src/pg/connection.cr:1: while requiring "../pq/*"

require "../pq/*"
^

in lib/pg/src/pq/connection.cr:10: PQ(T) is not a module, it's a generic class

module PQ
       ^
crystal --version
Crystal 0.21.1 (2017-03-14) LLVM 4.0.0

Running MacOS 10.12.4
Installed crystal via brew

No such table

Hi,

When I try to execute an sql query on the wrong table, I have a raw error from PQ driver, but no

error.status

Is it a wanted behavior ?

Regards,

PS : Rather ask before PR

Exception: end of file reached (IO::EOFError)

Hi guys,

I'm getting an error when I try to insert a row whose fields contain iso_8859_1 characters such as áéíóú´ñ.... This is the code I have:

require "kemal"
require "json"
require "pg"

PG_URL = "postgres://Angarsk8@localhost:5432/notes_db"
DB     = PG.connect PG_URL

get "/" do |env|
  env.response.content_type = "application/json"  
  DB.exec("INSERT INTO notes (TITLE, CONTENT, CREATED_AT, UPDATED_AT)
  VALUES ( 'ñá', 'áéíóú', current_timestamp, current_timestamp);")
  notes = DB.exec("SELECT to_char(created_at, 'DD-MM-YYYY HH24:MI:SS') as created_at  FROM notes")
  notes.to_hash.map do |note|
    {created_at: note["created_at"].as(String)}
  end.to_json
end

Kemal.run

The database is UTF-8 encoded:

Name Owner Encoding
notes_db Angarsk8 $UTF8

But when I run that same query manually from the console or via the client app, it inserts the row with no problem whatsoever.

Does anybody know how to fix this? or Why is that happening?

This is the full error stack trace I get:

Exception: end of file reached (IO::EOFError)
[4354261106] *CallStack::unwind:Array(Pointer(Void)) +82
[4354261009] *CallStack#initialize:Array(Pointer(Void)) +17
[4354260968] *CallStack::new:CallStack +40
[4354203273] *raise<IO::EOFError>:NoReturn +25
[4355015013] *Socket+@IO#read_fully<Slice(UInt8)>:Int32 +117
[4354773492] *IO::ByteFormat::BigEndian::decode<Int32:Class, Socket+>:Int32 +68
[4354344722] *Int32@Int::from_io<Socket+, IO::ByteFormat::BigEndian:Module>:Int32 +34
[4355014882] *Socket+@IO#read_bytes<Int32:Class, IO::ByteFormat::BigEndian:Module>:Int32 +18
[4354981411] *PQ::Connection#read_i32:Int32 +99
[4354980805] *PQ::Connection#read<(Char | Nil)>:(PQ::Frame+ | PQ::Frame::Unknown) +69
[4354980535] *PQ::Connection#read:(PQ::Frame+ | PQ::Frame::Unknown) +183
[4354982243] *PQ::Connection#expect_frame<PQ::Frame::ReadyForQuery:Class, Nil>:PQ::Frame::ReadyForQuery +51
[4354982171] *PQ::Connection#expect_frame<PQ::Frame::ReadyForQuery:Class>:PQ::Frame::ReadyForQuery +27
[4354981937] *PQ::Connection#handle_error<PQ::Frame::ErrorResponse>:NoReturn +49
[4354981671] *PQ::Connection#handle_async_frames<(PQ::Frame+ | PQ::Frame::Unknown)>:Bool +55
[4354981104] *PQ::Connection#read<(Char | Nil)>:(PQ::Frame+ | PQ::Frame::Unknown) +368
[4354980535] *PQ::Connection#read:(PQ::Frame+ | PQ::Frame::Unknown) +183
[4354982243] *PQ::Connection#expect_frame<PQ::Frame::ReadyForQuery:Class, Nil>:PQ::Frame::ReadyForQuery +51
[4354982171] *PQ::Connection#expect_frame<PQ::Frame::ReadyForQuery:Class>:PQ::Frame::ReadyForQuery +27
[4354981937] *PQ::Connection#handle_error<PQ::Frame::ErrorResponse>:NoReturn +49
[4354981671] *PQ::Connection#handle_async_frames<(PQ::Frame+ | PQ::Frame::Unknown)>:Bool +55
[4354981104] *PQ::Connection#read<(Char | Nil)>:(PQ::Frame+ | PQ::Frame::Unknown) +368
[4354980535] *PQ::Connection#read:(PQ::Frame+ | PQ::Frame::Unknown) +183
[4354991395] *PQ::Connection#expect_frame<PQ::Frame::ParseComplete:Class, Nil>:PQ::Frame::ParseComplete +51
[4354991323] *PQ::Connection#expect_frame<PQ::Frame::ParseComplete:Class>:PQ::Frame::ParseComplete +27
[4355076235] *PQ::ExtendedQuery#initialize<PQ::Connection, String, Array(PQ::Param)>:Bool +235
[4355075966] *PQ::ExtendedQuery#initialize<PQ::Connection, String, Array(Array(JSON::Type) | Bool | Float32 | Float64 | Hash(String, JSON::Type) | Int32 | Int64 | PG::Numeric | String | Time | Nil)>:Bool +542
[4355075396] *PQ::ExtendedQuery::new<PQ::Connection, String, Array(Array(JSON::Type) | Bool | Float32 | Float64 | Hash(String, JSON::Type) | Int32 | Int64 | PG::Numeric | String | Time | Nil)>:PQ::ExtendedQuery +132
[4354872925] *PG::Connection#extended_query<String, Array(Array(JSON::Type) | Bool | Float32 | Float64 | Hash(String, JSON::Type) | Int32 | Int64 | PG::Numeric | String | Time | Nil)>:PQ::ExtendedQuery +13
[4354872887] *PG::Connection#exec<Array(Array(JSON::Type) | Bool | Float32 | Float64 | Hash(String, JSON::Type) | Int32 | Int64 | PG::Numeric | String | Time | Nil), String, Array(Array(JSON::Type) | Bool | Float32 | Float64 | Hash(String, JSON::Type) | Int32 | Int64 | PG::Numeric | String | Time | Nil)>:PG::Result(Array(Array(JSON::Type) | Bool | Float32 | Float64 | Hash(String, JSON::Type) | Int32 | Int64 | PG::Numeric | String | Time | Nil)) +23
[4354872846] *PG::Connection#exec<String>:PG::Result(Array(Array(JSON::Type) | Bool | Float32 | Float64 | Hash(String, JSON::Type) | Int32 | Int64 | PG::Numeric | String | Time | Nil)) +62
[4354233418] ~procProc(HTTP::Server::Context, String)@./src/app.cr:10 +74
[4354232942] ~procProc(HTTP::Server::Context, String)@./libs/kemal/kemal/route.cr:10 +46
[4354826509] *Kemal::RouteHandler#process_request<HTTP::Server::Context>:HTTP::Server::Context +125
[4354826364] *Kemal::RouteHandler#call<HTTP::Server::Context>:HTTP::Server::Context +140
[4355150735] *Kemal::StaticFileHandler@HTTP::Handler#call_next<HTTP::Server::Context>:(Bool | HTTP::Server::Context | HTTP::Server::Response | HTTP::Server::Response::Output | IO::FileDescriptor+ | Int32 | Nil) +527
[4355150087] *Kemal::StaticFileHandler#call<HTTP::Server::Context>:(Bool | HTTP::Server::Context | HTTP::Server::Response | HTTP::Server::Response::Output | IO::FileDescriptor+ | Int32 | Nil) +71
[4355149031] *Kemal::CommonExceptionHandler@HTTP::Handler#call_next<HTTP::Server::Context>:(Bool | HTTP::Server::Context | HTTP::Server::Response | HTTP::Server::Response::Output | IO::FileDescriptor+ | Int32 | Nil) +999
[4355147264] *Kemal::CommonExceptionHandler#call<HTTP::Server::Context>:(Bool | HTTP::Server::Context | HTTP::Server::Response | HTTP::Server::Response::Output | IO::FileDescriptor+ | Int32 | Nil) +32
[4355145675] *Kemal::CommonLogHandler@HTTP::Handler#call_next<HTTP::Server::Context>:(Bool | HTTP::Server::Context | HTTP::Server::Response | HTTP::Server::Response::Output | IO::FileDescriptor+ | Int32 | Nil) +603
[4355144786] *Kemal::CommonLogHandler#call<HTTP::Server::Context>:HTTP::Server::Context +50
[4355158414] *HTTP::Server::RequestProcessor#process<(OpenSSL::SSL::Socket::Server | TCPSocket+), (OpenSSL::SSL::Socket::Server | TCPSocket+), IO::FileDescriptor>:Nil +974
[4355157429] *HTTP::Server::RequestProcessor#process<(OpenSSL::SSL::Socket::Server | TCPSocket+), (OpenSSL::SSL::Socket::Server | TCPSocket+)>:Nil +53
[4355157132] *HTTP::Server#handle_client<(TCPSocket+ | Nil)>:Nil +124
[4354239078] ~procProc(Nil)@/usr/local/Cellar/crystal-lang/0.18.7/src/http/server.cr:146 +22
[4354258362] *Fiber#run:(Int64 | Nil) +170
[4354203077] ~proc2Proc(Fiber, (Int64 | Nil))@/usr/local/Cellar/crystal-lang/0.18.7/src/fiber.cr:28 +21

That thing is almost impossible to understand for me!

Thanks,

Decode UUID type

Querying UUID columns results in an invalid UTF-8 String. I suppose it's missing a proper decoder (oid = 2950)?

What is weird is that this started failing when I upgraded to crystal v0.8.0 and crystal-pg v0.4.0. It used to work previously (maybe crystal 0.7.7 + pg v0.3.2).

Problem with new version of crystal?, openssl not found.

require "pg"
^

in ./libs/pg/pg.cr:1: while requiring "./pg/*"

require "./pg/*"
^

in ./libs/pg/pg/connection.cr:1: while requiring "../pq/*"

require "../pq/*"
^


in ./libs/pg/pq/connection.cr:6: while requiring "openssl/openssl": can't find file 'openssl/openssl' relative to '/home/test/libs/pg/pq'

require "openssl/openssl"
^

issues with crystal v0.21.0

Crypto::MD5 is now Digest::MD5

I tried changing the require in pq/connection and changing Crypto to Digest but then started to run into more issues: undefined constant Digest::MD5::Context

Reset connection

Could we had a mecanism to detect a connection issue and to reset the connection? Maybe wrapping the thrown socket Errno as PQ::SocketError and adding PQ::Connection#reset and PG::Connection#reset methods?

This is useful when a connection gets staled or broken, which happens regularly on the Heroku free plans. I used to catch PG::Error exceptions, call LibPQ.reset once before retrying, as long as the reset didn't fail. Namely those two patches: ysbaddaden/frost@184642c and ysbaddaden/frost@b3c1586

Issue with Parameterized ArrayDecoder

I am getting the following when running

PG.connect(....)

Error in ./src/db/init_db.cr:7: instantiating 'PG:Module#connect(String)'

conn = PG.connect("postgres://postgres:mypass@postgres:5432/postgres")
          ^~~~~~~

in ./libs/pg/pg.cr:6: instantiating 'PG::Connection:Class#new(String)'

    Connection.new(conninfo)
               ^~~

instantiating 'PG::Connection#initialize(String)'

in ./libs/pg/pg/connection.cr:28: instantiating 'initialize(PQ::ConnInfo)'

      initialize PQ::ConnInfo.from_conninfo_string(conninfo)
      ^~~~~~~~~~

in ./libs/pg/pg/connection.cr:21: instantiating 'exec(String)'

      exec("SET extra_float_digits = 3")
      ^~~~

in ./libs/pg/pg/connection.cr:56: instantiating 'exec(Array(Array(JSON::Type) | Bool | Float32 | Float64 | Hash(String, JSON::Type) | Int32 | Int64 | PG::Numeric | String | Time | Nil), String, Array(Array(JSON::Type) | Bool | Float32 | Float64 | Hash(String, JSON::Type) | Int32 | Int64 | PG::Numeric | String | Time | Nil))'

      exec([] of PG::PGValue, query, [] of PG::PGValue)
      ^~~~

in ./libs/pg/pg/connection.cr:71: instantiating 'PQ::Connection#synchronize()'

      @pq_conn.synchronize do
               ^~~~~~~~~~~

in ./libs/pg/pq/connection.cr:78: instantiating 'Mutex#synchronize()'

      @mutex.synchronize { yield }
             ^~~~~~~~~~~

in ./libs/pg/pq/connection.cr:78: instantiating 'Mutex#synchronize()'

      @mutex.synchronize { yield }
             ^~~~~~~~~~~

in ./libs/pg/pg/connection.cr:71: instantiating 'PQ::Connection#synchronize()'

      @pq_conn.synchronize do
               ^~~~~~~~~~~

in ./libs/pg/pg/connection.cr:72: instantiating 'PG::Result(T):Class#new(Array(Array(JSON::Type) | Bool | Float32 | Float64 | Hash(String, JSON::Type) | Int32 | Int64 | PG::Numeric | String | Time | Nil), PQ::ExtendedQuery)'

        Result.new(types, extended_query(query, params))
               ^~~

instantiating 'PG::Result(Array(Array(JSON::Type) | Bool | Float32 | Float64 | Hash(String, JSON::Type) | Int32 | Int64 | PG::Numeric | String | Time | Nil))#initialize(Array(Array(JSON::Type) | Bool | Float32 | Float64 | Hash(String, JSON::Type) | Int32 | Int64 | PG::Numeric | String | Time | Nil), PQ::ExtendedQuery)'

instantiating 'initialize(Array(Array(JSON::Type) | Bool | Float32 | Float64 | Hash(String, JSON::Type) | Int32 | Int64 | PG::Numeric | String | Time | Nil), PQ::ExtendedQuery, Bool)'

in ./libs/pg/pg/result.cr:12: instance variable '@decoders' of PG::Result(Array(Array(JSON::Type) | Bool | Float32 | Float64 | Hash(String, JSON::Type) | Int32 | Int64 | PG::Numeric | String | Time | Nil)) must be Array(PG::Decoders::Decoder), not Array(PG::Decoders::ArrayDecoder(Bool, PG::BoolArray, PG::Decoders::BoolDecoder) | PG::Decoders::ArrayDecoder(Char, PG::CharArray, PG::Decoders::CharDecoder) | PG::Decoders::ArrayDecoder(Float32, PG::Float32Array, PG::Decoders::Float32Decoder) | PG::Decoders::ArrayDecoder(Float64, PG::Float64Array, PG::Decoders::Float64Decoder) | PG::Decoders::ArrayDecoder(Int16, PG::Int16Array, PG::Decoders::Int16Decoder) | PG::Decoders::ArrayDecoder(Int32, PG::Int32Array, PG::Decoders::Int32Decoder) | PG::Decoders::ArrayDecoder(Int64, PG::Int64Array, PG::Decoders::Int64Decoder) | PG::Decoders::ArrayDecoder(String, PG::StringArray, PG::Decoders::StringDecoder) | PG::Decoders::Decoder)

32 and 64bit byte endian swapping

I'm working on using the binary interface instead of the text, and as part of this you need to switch endianness.

I've looked at 4 ways to do this, but I'm unsure which is the best. One (actually swapping things in memory) is gross, so that's off the table. The remaining 3 are

  • use ntohl from libc. This will be a noop on like powerpcs, but does that even matter with crystal? It ends up turning into the llvm intrinsic anyway.
  • use llvm intrinsic directly. This is already present in crystal unlike nothl
  • do byte shifting manually. This appears to turn into optimized assembly due to the benchmark, but I haven't actually checked yet. This has the greatest visual similarity between 32 and 64 implementations (the nothl and intrinsic starts looking gross at 64bit).

https://gist.github.com/will/c348bf90a7cbd7e93fd0 has the implementations and benchmark code.

Undefined method 'empty' when exec is called without params

Hello:

I'm receiving this error:

undefined method 'empty' for Slice(Bool | Float32 | Float64 | Int32 | Int64 | Slice(UInt8) | String | Time | Nil):Class
      perform_exec_and_release(Slice(Any).empty)

Code:

DB.open("postgres:///") do |db| 
  db.exec("DROP SCHEMA IF EXISTS test CASCADE;")
end

Workaround:

module DB
  abstract class Statement
    # See `QueryMethods#exec`
    def exec
      perform_exec_and_release([] of DB::Any)
    end
  end
end

I really don't know if is related to crystal-pg or also affect all crystal-db drivers.

CC: @bcardiff.

Spec fail on empty queries tests

Hi,

I am testing auth system on crystal-pg.

I use a container (docker) to isolate it

FROM ubuntu:trusty
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get -qq update

RUN apt-get -qy install curl apt-transport-https
RUN curl https://dist.crystal-lang.org/apt/setup.sh | bash
RUN apt-get -qy install crystal

RUN apt-get -qy install postgresql postgresql-server-dev-9.3

RUN apt-get -qy install git

RUN apt-get -qy install libgmp-dev libxml2-dev 

ADD latest.tgz /opt/src

WORKDIR /opt/src

RUN touch spec/.run_auth_spec

RUN shards install

RUN service postgresql start
RUN sed -i 's/peer/trust/g' `find / -type f -name pg_hba.conf`
RUN sed -i 's/md5/trust/g' `find / -type f -name pg_hba.conf`
RUN service postgresql restart
RUN createdb crystal -U postgres
RUN apt-get -qy install wget
RUN mkdir .travis
RUN wget -P .travis https://raw.githubusercontent.com/waghanza/crystal-pg/auth_test/.travis/pg_hba.conf
RUN cp .travis/pg_hba.conf $(psql -U postgres -c "SHOW hba_file" -At)
RUN service postgresql restart
RUN touch spec/.run_auth_specs
ENV DATABASE_URL postgres://postgres@localhost/crystal
RUN crystal spec

But I have an error occured on empty queries handling

       SSL_write: Unexpected EOF
       0x6124c8: *OpenSSL::SSL::Socket::Client at /opt/crystal/src/openssl/ssl/socket.cr 116:5
       0x613ae7: *OpenSSL::SSL::Socket::Client at /opt/crystal/src/io.cr 552:5
       0x543dc2: *String#to_s<OpenSSL::SSL::Socket::Client>:Nil at /opt/crystal/src/string.cr 4036:5
       0x613a51: *OpenSSL::SSL::Socket::Client at /opt/crystal/src/io.cr 267:5
       0x60f341: *PQ::Connection#send_parse_message<String>:(Int32 | Socket+ | Nil) at /opt/src/src/pq/connection.cr 318:7
       0x600918: *PG::Statement#perform_query<Slice(Bool | Float32 | Float64 | Int32 | Int64 | Slice(UInt8) | String | Time | Nil)>:PG::ResultSet at /opt/src/src/pg/statement.cr 13:5
       0x60041d: *PG::Statement#perform_exec<Slice(Bool | Float32 | Float64 | Int32 | Int64 | Slice(UInt8) | String | Time | Nil)>:DB::ExecResult at /opt/src/src/pg/statement.cr 35:14
       0x60b67d: *DB::Statement+ at /opt/src/lib/db/src/db/statement.cr 97:14
       0x60b609: *DB::Statement+ at /opt/src/lib/db/src/db/statement.cr 67:7
       0x65208b: *DB::PoolStatement+ at /opt/src/lib/db/src/db/pool_statement.cr 14:30
       0x5e0745: *DB::Database at /opt/src/lib/db/src/db/query_methods.cr 203:7
       0x51eb47: ~procProc(Nil) at /opt/crystal/src/spec/methods.cr 4:5
       0x4f1ce2: *it<String, String, Int32, Int32, &Proc(Nil)>:(Array(Spec::Result) | Nil) at /opt/crystal/src/spec/methods.cr 17:7
       0x51eb10: ~procProc(Nil) at /opt/crystal/src/spec/methods.cr 4:5
       0x599497: *Spec::RootContext::describe<String, String, Int32, &Proc(Nil)>:Spec::Context+ at /opt/crystal/src/spec/context.cr 152:30
       0x51d3b7: *describe<PQ::Connection:Class, String, Int32, &Proc(Nil)>:Spec::Context+ at /opt/crystal/src/spec/methods.cr 4:5
       0x4d776c: ??? at /opt/src/spec/pq/connection_spec.cr 22:1
       0x4e9439: main at /opt/crystal/src/main.cr 12:15
       0x7ffbecec1f45: __libc_start_main at ??
       0x4d5c39: ??? at ??
       0x0: ??? at ??

As new in crystal I do not understand further

Error when try to connect

Hi I tried to connect to a postgres database but it shows an error, I use this in the root of the directory:
crystal eval "require \"./src/pg\";DB = PG.connect(\"postgres://postgres@localhost/mydb\");DB.exec(\"SELECT * FROM pg_catalog.pg_tables\")"

My PC shows the error in Spanish, but it pretty much saying something's wrong with the Fiber library: *Fiber#initialize<Fiber>:Fiber

Issues with Crystal 0.20.3

in lib/db/src/db/database.cr:50: instantiating 'DB::Driver+#build_connection(DB::Database)'

        conn = @driver.build_connection(self).as(Connection)
                       ^~~~~~~~~~~~~~~~

in lib/pg/src/pg/driver.cr:3: instantiating 'PG::Connection:Class#new(DB::Database)'

    Connection.new(db)
               ^~~

instance variable '@connection' of PG::Connection must be PQ::Connection, not Nil

Error: instance variable '@connection' is initialized inside a begin-rescue, so it can potentially be left uninitialized if an exception is raised and rescued

Use types instead of tuples, arrays of tuples, etc.?

Yesterday we tried to write some code with @ggiraldez that queried postgres to get some GIS data. It took us a while to figure out what types were returned by the driver, or what types we needed to pass to execute. I can't remember what we ended up passing, I think Tuple and then casting each member to Float64, with as.

Given that Crystal allows user-defined types, I think it would be much more convenient and intuitive if, instead of returning tuples and arrays of tuples, proper types would be returned. They can be structs to preserve performance. Basically something like this:

record PG::Geo::Point, x : Float64, y : Float64

# ...

class PointDecoder < Decoder
  def decode(bytes)
    x = swap64(bytes)
    y = swap64(bytes + 8)

    Point.new(
      (pointerof(x).as(Float64*)).value,
      (pointerof(y).as(Float64*)).value,
    )
  end
end

For PolygonDecoder with can return an Array(Point), and maybe have an alias Polygon = Array(Point), for PathDecoder we'd return a Geo::Path struct that has an open/closed Bool property, along with the points, etc.

That way one can specify Point in the query instead of Tuple(Float64, Float64). This is both less code to read/write, more clear, and has the same performance.

There's another issue: tupe types of the same size get merged. So BoxDecoder returns a two-elements tuple, and so does PointDecoder, so one can't specify Tuple(Float64, Float64) as a target type because the real type returned by all decoders is a tuple where each component is a union.

I can create a PR with code for this, but I wanted to know first if you like this idea.

Error when trying to execute anything (libpq5 - 9.3.8-0ubuntu0.4.04 | Crystal 0.7.4)

There is an error when either I try to call DB.version or DB.exec:

/usr/bin/ld: /opt/crystal/bin/../embedded/lib/../lib/libgc.a(os_dep.o): undefined reference to symbol '_end'
//lib/x86_64-linux-gnu/libcom_err.so.2: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
Error: execution of command failed with code: 1: `cc -o /home/gdot/Projects/cr-orm/.crystal/crystal-run-test.tmp "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/main.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Array40Int3241.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Math.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Pointer40Int3241.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/ArgumentError.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Pointer40UInt841.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Pointer40LibABI5858UnwindException41.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/UInt64.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Array40String41.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Pointer40String41.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/GC.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/String.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/String5858Builder.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Slice40T41.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Slice40UInt841.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/IndexOutOfBounds.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Int32.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Exception.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/StaticArray40UInt844326541.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/DivisionByZero.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Array40Pointer40Void4141.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Pointer40Pointer40Void4141.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Fiber.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Pointer40Void41.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Array40Fiber41.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Pointer40Fiber41.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Event5858Base.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Scheduler.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/FileDescriptorIO.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Bool.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/LibEvent25858EventFlags.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Nil.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Errno.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/IO5858Error.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Signal.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/40Int3232456232Void41.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Hash40Int32443240Int3232456232Void4141.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Pointer40Hash40K4-c3ec75a0b68f26d1e6c0b6ef5137591d.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Hash40K4432V41585-f79971827d7c936e8b09d3ee36627c92.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Hash40K4432V415858StandardComparator.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Int64.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Exception43.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/StaticArray40UInt84432441.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/PG.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/PG5858Connection.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/PG5858LibPQ5858ConnStatusType.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/PG5858ConnectionError.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Array4040Nil32124-5a1c653c53937859de5163bb1b99321a.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Pointer4040Nil321-f8af3429e84782dafb266b791b7a0621.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Array40Pointer40UInt84141.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Pointer40Pointer40UInt84141.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Float32.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Float64.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Time.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Hash40String4432JSON5858Type41.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Array40JSON5858Type41.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/TimeFormat.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/TimeFormat5858Formatter.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/CharReader.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/InvalidByteSequenceError.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Char.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Time5858Kind.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Hash40123UInt644432Symbol1254432Bool41.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Pointer40Hash40K4-766bdc6ea432dd2739b30f84312c75af.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/123UInt644432Symbol125.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Hash40K4432V41585-4c57993f23870785eccb6bb4de73a8c0.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Hash40K4432V415858Entry40String4432JSON5858Type41.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/StaticArray40UInt844322241.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Pointer4040Nil321-fad29b889aa8902379af84d5f34359c4.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/PG5858LibPQ5858ExecStatusType.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/PG5858ResultError.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/PG5858Result40T41.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/PG5858Result40Arr-49c95cd7646292d9b96f328163b3bd6f.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Array40PG5858Result40T415858Field41.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Pointer40PG5858Result40T415858Field41.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/PG5858Result40T415858Field.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Array40Array4040N-72a16c98c9f3c91701f1972eec8291f4.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Pointer40Array404-d5375a1c07bc11c7824748ced45c0cee.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Array40PG5858Decoder5858Decoder4341.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Pointer40PG5858Decoder5858Decoder4341.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/PG5858Decoder.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/PG5858Decoder5858TimeDecoder.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/PG5858Decoder5858JsonDecoder.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/PG5858Decoder5858Float64Decoder.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/PG5858Decoder5858Float32Decoder.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/PG5858Decoder5858IntDecoder.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/PG5858Decoder5858BoolDecoder.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/PG5858Decoder5858DefaultDecoder.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/TimeSpan.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/JSON.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/JSON5858Parser.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/JSON5858Lexer.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/JSON5858Lexer5858StringBased.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/JSON5858Token.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/StringIO.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/StringPool.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Array40Array40String416341.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Pointer40Array40String416341.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/JSON5858ParseException.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Range40B4432E41.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Range40Char4432Char41.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Symbol.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Pointer40Hash40K4-a5f4dccfeeb9e82ab9cf742c0bde63b6.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/PG5858Decoder5858Decoder43.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/PG5858Result40123-a0ffa14e7ce6c16e549ac0345a660db6.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Array40123Int324432Int324432Int3212541.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Pointer40123Int324432Int324432Int3212541.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Hash40Symbol4432Int3241.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Pointer40Hash40K4-d99d7a22b21dfe80971b432330542f56.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Hash40K4432V415858Entry40Symbol4432Int3241.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/LEBReader.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/PG5858RuntimeError.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/PG5858Error43.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/ChannelClosed.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/GlobError.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/IO5858EOFError.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/IO5858Error43.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/TimeFormat5858Error.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/MissingKey.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/DomainError.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/EmptyEnumerable.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Regex.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/Regex5858Options.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/MatchData.o" "/home/gdot/Projects/cr-orm/.crystal/home/gdot/Projects/cr-orm/test.cr/AtExitHandlers.o"   -lpq -I`pg_config --includedir` -L`pg_config --libdir` -levent -lrt -lpcl -lpcre -lm -lgc -lpthread -lunwind`

Numeric datatype unsupported

I have the following code, it does not raise an exception but it doesn't seem to give the results.

pg=PG.connect(ENV["CONNECTION_STRING"])

query = <<-SQL
          SELECT
            SUM(seq_scan) / SUM(idx_scan) a,
            SUM(seq_tup_read) / SUM(idx_tup_fetch) b
          FROM pg_stat_all_tables;
        SQL

r = pg.exec(query)
puts r.fields.inspect
puts r.rows.inspect
puts r.to_hash.inspect

This is the output from code

CONNECTION_STRING="postgres://tim@localhost/crystal" ./run up
building...

[PG::Result::Field(@name="a", @oid=1700), PG::Result::Field(@name="b", @oid=1700)]
Invalid byte sequence in UTF-8 string (InvalidByteSequenceError)
[4362362402] *CallStack::unwind:Array(Pointer(Void)) +82
[4362362305] *CallStack#initialize<CallStack>:Array(Pointer(Void)) +17
[4362362264] *CallStack::new:CallStack +40
[4362401201] *Exception@Exception#initialize<InvalidByteSequenceError, String>:CallStack +33
[4362401133] *InvalidByteSequenceError#initialize<InvalidByteSequenceError, String>:CallStack +29
[4362401068] *InvalidByteSequenceError::new<String>:InvalidByteSequenceError +92
[4362400956] *InvalidByteSequenceError::new:InvalidByteSequenceError +28
[4362430385] *Char::Reader#invalid_byte_sequence<Char::Reader>:NoReturn +17
[4362429645] *Char::Reader#decode_current_char<Char::Reader>:Char +189
[4362430515] *Char::Reader#next_char<Char::Reader>:Char +67
[4362358048] *String#inspect<String, String::Builder>:String::Builder +272
[4362486295] *Hash(String, Nil | String | Int32 | Int16 | Int64 | Float32 | Float64 | Bool | Time | Slice(UInt8) | Hash(String, JSON::Type) | Array(JSON::Type))@Hash(K, V)#to_s<Hash(String, Nil | String | Int32 | Int16 | Int64 | Float32 | Float64 | Bool | Time | Slice(UInt8) | Hash(String, JSON::Type) | Array(JSON::Type)), String::Builder>:String::Builder? +695
[4362485593] *Hash(String, Nil | String | Int32 | Int16 | Int64 | Float32 | Float64 | Bool | Time | Slice(UInt8) | Hash(String, JSON::Type) | Array(JSON::Type))@Hash(K, V)#inspect<Hash(String, Nil | String | Int32 | Int16 | Int64 | Float32 | Float64 | Bool | Time | Slice(UInt8) | Hash(String, JSON::Type) | Array(JSON::Type)), String::Builder>:String::Builder? +9
[4362483442] *Array(Hash(String, Nil | String | Int32 | Int16 | Int64 | Float32 | Float64 | Bool | Time | Slice(UInt8) | Hash(String, JSON::Type) | Array(JSON::Type)))@Array(T)#to_s<Array(Hash(String, Nil | String | Int32 | Int16 | Int64 | Float32 | Float64 | Bool | Time | Slice(UInt8) | Hash(String, JSON::Type) | Array(JSON::Type))), String::Builder>:String::Builder? +610
[4362482825] *Array(Hash(String, Nil | String | Int32 | Int16 | Int64 | Float32 | Float64 | Bool | Time | Slice(UInt8) | Hash(String, JSON::Type) | Array(JSON::Type)))@Array(T)#inspect<Array(Hash(String, Nil | String | Int32 | Int16 | Int64 | Float32 | Float64 | Bool | Time | Slice(UInt8) | Hash(String, JSON::Type) | Array(JSON::Type))), String::Builder>:String::Builder? +9
[4362482786] *Array(Hash(String, Nil | String | Int32 | Int16 | Int64 | Float32 | Float64 | Bool | Time | Slice(UInt8) | Hash(String, JSON::Type) | Array(JSON::Type)))@Object#inspect<Array(Hash(String, Nil | String | Int32 | Int16 | Int64 | Float32 | Float64 | Bool | Time | Slice(UInt8) | Hash(String, JSON::Type) | Array(JSON::Type)))>:String +66
[4362336754] __crystal_main +25906
[4362345488] main +32

Implement crystal-db

crystal-db provides a unified interface for dealing with databases. Right now we have drivers for sqlite3 and mysql, and it would be great to have a postgres driver.

@will I think the main worry you had about adopting this is to loose some postgres-only features, or being able to deal directly with the connection without going through the DB interface. This is currently possible, a DB connection has a with_connection method that yields the underlying connection for this. But before that...

I have a partial implementation of this here, but I didn't maintain it for some time and your repo continued evolving so I probably need to rebase.

Question: would you be fine with merging a PR that contains this functionality? This will of course be backwards incompatible. Otherwise we can simply have a driver in some other repository, maybe crystal-lang/crystal-pg. What do you think?

When postgres 9.3 was installed by apt, it doesn't work.

/usr/bin/ld: /opt/crystal/bin/../embedded/lib/../lib/libgc.a(os_dep.o): undefined reference to symbol '_end'
//lib/x86_64-linux-gnu/libcom_err.so.2: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
require "pg"
DB = PG.connect("postgres://user@localhost:5432/db")
result = DB.exec({Int32}, "select count(id) from users")
p result
  • ubuntu 14.04LTS
  • postgres 9.3.9
  • crystal 0.7.5

I want to know how to resolve this issue.

When load testing getting PG errors

When I run my server and query it manually, everything works as expected. But when I try to load test with wrk or ApacheBench, I immediately get the error below, and even after I kill the load test, requests take a long time and timeout. Here is the code:

DB = PG.connect("postgres://shailpatel@localhost:5432/supercontest_data")

get "/" do |env|
  env.response.content_type = "application/json"
  result = DB.exec({Int32, String}, "SELECT id, name FROM teams")
  hash = { "data" => result.to_hash }
  hash.to_json
end

Error:

Exception: Expected PQ::Frame::ParseComplete but got PQ::Frame::Unknown(@type='\u{2}', @bytes=Slice[]) (Exception)
[4449219811] *raise<String>:NoReturn +163
[4449593329] *PG::Connection#exec<Tuple(Int32:Class, String:Class), String, Array(Array(JSON::Type) | Bool | Float32 | Float64 | Hash(String, JSON::Type) | Int32 | Int64 | PG::Numeric | String | Time | Nil)>:PG::Result(Tuple(Int32:Class, String:Class)) +11169
[4449576524] ~procProc(HTTP::Server::Context, String)@./src/supercontest_data_crystal.cr:8 +332
[4449754793] *Kemal::RouteHandler#call<HTTP::Server::Context>:HTTP::Server::Context +73
[4449809084] *Kemal::StaticFileHandler@HTTP::Handler#call_next<HTTP::Server::Context>:(Bool | HTTP::Server::Context | HTTP::Server::Response | HTTP::Server::Response::Output | IO::FileDescriptor+ | Int32 | Nil) +556
[4449760866] *Kemal::CommonExceptionHandler#call<HTTP::Server::Context>:(Bool | HTTP::Server::Context | HTTP::Server::Response | HTTP::Server::Response::Output | IO::FileDescriptor+ | Int32 | Nil) +2370
[4449765265] *Kemal::CommonLogHandler#call<HTTP::Server::Context>:HTTP::Server::Context +1793
[4449757783] *Kemal::InitHandler#call<HTTP::Server::Context>:(Bool | HTTP::Server::Context | HTTP::Server::Response | HTTP::Server::Response::Output | IO::FileDescriptor+ | Int32 | Nil) +615
[4449703673] *HTTP::Server#handle_client<(TCPSocket+ | Nil)>:Nil +10489
[4449235224] *Fiber#run:(Int64 | Nil) +40

Segmentation fault when trying to connect RDS

I'm trying to connect to my RDS instance with credentials like below.

DB = PG.connect("postgres://user_name:[email protected]/db_name?sslmode=require")
DB.exec("SELECT * from my_table")

It just segfaults without any stacktrace.

Program exited because of a segmentation fault (11)

I'm clueless and need some help :/

nilllable Time type fails declared as Time?

class created like:

class User
  DB.mapping({
    id: Int32?, 
    name: String?,
    some_date: Time?
  })
end

If a record is inserted into the database with a nil column value for some_date, then trying to call User.from_rs(result_set) fails with an error:

Nil assertion failed (Exception)
0x10158ec62: *CallStack::unwind:Array(Pointer(Void)) at ??
0x10158ec01: *CallStack#initialize:Array(Pointer(Void)) at ??
0x10158ebd8: *CallStack::new:CallStack at ??
0x10157b4d1: *raise<Exception>:NoReturn at ??
0x10157b4b1: *raise<String>:NoReturn at ??
0x1015adb93: *Nil#not_nil!:NoReturn at ??
0x10163b935: *User#initialize<PG::ResultSet>:Time at ??
0x10163b539: *User::new<PG::ResultSet>:User at ??
0x10163b4ad: *User::from_rs<PG::ResultSet>:Array(User) at ??
0x10157b068: __crystal_main at ??
0x101589af8: main at ??

How to implement Connection#build_statement

I understand that there have been lots of changes recently, but could you get some docs up? :)

I'm getting this error and now my app doesn't work:

abstract `def DB::Connection#build_statement(query)` must be implemented by PG::Connection

Edit: This is probably from #66

CITEXT type is understood as Slice instead of String in 0.7.1

In version cafe4748b64a9c95ab93b4c19315780e8e254291, querying a CITEXT type field would return the result as String, so the following returned the expected result:

DB.exec({String}, "SELECT name FROM users").rows[0][0]

As of 0.7.1, the code above started throwing cast to String failed (TypeCastError), though the following did work and returned the same value:

String.new(DB.exec({Slice}, "SELECT name FROM users").rows[0][0])

It seems that a change in the decoder caused CITEXT fields to stop being understood as actual Strings.

Tested with postgresql 9.4.7.

Connection Fix

Could you fix line 29 of ./pg/connection.cr:

from
def exec_all(query: String)

to
def exec_all(query : String)

I use korean letter(utf-8) in query, but It causes error.

test.cr

con = PG.connect("postgres://song@localhost/blog")
con.exec("SELECT * FROM BLOG WHERE NAME='test';")
  # => it works very well,

con = PG.connect("postgres://song@localhost/blog")
con.exec("SELECT * FROM BLOG WHERE NAME='테스트';") 
  # => end of file reached (IO::EOFError)

terminal

~$ psql BLOG
BLOG=# SELECT * FROM BLOG WHERE NAME='테스트';

=> it works correct too.

I use Postgresql 9.4.7
I set all table's encoding type utf-8.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.