Git Product home page Git Product logo

mpg's Introduction

mpg - more modern postgres to the gallon

Eases interoperability between clojure/java datatypes and postgres datatypes. No more boilerplate!

Handles the following:

  • DATE <-> java.time.LocalDate
  • TIMESTAMP/TIMESTAMPTZ <-> java.time.ZonedDateTime
  • JSON/JSONB <-> clojure map/vector
  • ARRAY (e.g. int[]) <-> clojure vector
  • BYTEA <-> byte array
  • HSTORE <-> clojure map (limited support - jdbc stringifies all contents)

Can also insert (but not retrieve) the following types:

  • java.util.Date -> DATE/TIMESTAMP/TIMESTAMPTZ
  • java.sql.Timestamp -> DATE/TIMESTAMP/TIMESTAMPTZ
  • java.nio.ByteBuffer -> BYTEA

Build Status

Note: this library was once called jdbc-pg-sanity, mpg is the new version.

Installation

Add mpg as a leiningen or boot dependency:

[mpg "1.3.0"]

Clojars Project

Usage

Just require the mpg.core namespace and call patch

(ns whatever.db
    (require [clojure.java.jdbc :as j]
             [mpg.core :as mpg]]))
(mpg/patch) ;; take the default settings
(mpg/patch {:default-map :hstore}) ;; custom settings are merged with the defaults
;; valid settings:
    :data        - boolean, default true. auto-map maps and vectors?
    :datetime    - boolean, default true. auto-map java.time.{LocalDate, ZonedDateTime} ?
    :default-map - keyword. one of :json, :jsonb, :hstore. Default :jsonb

Limitations

The current clojure.java.jdbc interface imposes some limitations on us.

  1. You only get the autoconversion when using clojure.java.jdbc or something built on it
  2. When using unbound statements, we cannot save a vector as an array type (we therefore use json)
  3. When using unbound statements, you must choose between storing maps as json or hstore (default: json)
  4. All applications that have written to the database are assumed to have correctly saved timestamps in UTC. If you only use this library, you won't have to worry about that. Most applications can be configured with the TZ environment variable

Running tests

You need a database and a user on it with which we can run tests.

You can provide information about these with environment variables:

MPG_TEST_DB_URI # default is '//127.0.0.1:5432/mpg_test'
MPG_TEST_DB_USER
MPG_TEST_DB_PASS

You can create a postgres database to test with createdb and give your user permissions with GRANT as per normal postgres.

Running the tests is the same as any leiningen-based project:

boot test

Contributing

Contributions and improvements welcome, just open an issue! :)

If this library should do something, and it doesn't currently do it, please fork and open a pull request. All reasonable contributions will be accepted.

Please run the tests before opening a pull request :)

Acknowledgements

This library was originally extracted from luminus boilerplate which hails from code floating around the internet generally. James Laver basically rewrote it.

License

Copyright © 2016 Shane Kilkelly, James Laver

Distributed under the MIT license.

mpg's People

Contributors

enragedginger avatar jjl avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

mpg's Issues

clojure.java.jdbc vs clojure.jdbc

It looks like your library targets the clojure.java.jdbc library, but your documentation and announcements say clojure.jdbc. These are two separate libraries, so this is bound to add to the confusion that already exists. You should change any reference from clojure.jdbc to clojure.java.jdbc.

clojure.java.jdbc
clojure.jdbc

Thanks for the library. I think it does meet a need for postgresql boilerplate.

clojure.java.jdbc/insert! does not play nicely

And best of all, I get a different error from the reporter of the bug in irc.

I get a NullPointerException because PgPreparedStatement.getMetaData() returns null. The utterly quality pgjdbc code this goes through is not enlightening. Anyway, it appears .getMetaData is returning null inside an insert! which is breaking everything.

The probable outcome of this will be fixes to clojure.java.jdbc :/

Testing destructive changes in isolated environments

One of the things I'm not satisfied about with the current test suite is the fact that we only test with the one configuration - we proclaim that you can choose hstore as the default, and while we test roundtripping through hstore, we don't actually have the ability to test whether when it's set as the default, it actually does what it says on the tin.

Because of the protocolly nature of this, I'm not entirely sure how to go about this. My current estimate is that we'll need to spin up isolated JVMs in the way that boot does, for which there are a couple of libraries of which I'm aware. Any other ideas I'd be interested in hearing, but I'd definitely like better test coverage in the wake of a conversation I had on github yesterday where they were experiencing some odd bugs and their "fix" was to add a catch-all case that just passed the input through unmodified. There are clearly some holes in our abstraction, and I want to flush them out.

Abstraction holes

As I alluded to in my last ticket, I'm aware that there are some holes in our abstraction and I want to plug them. See the discussion on this commit:
enragedginger@6137596

Firstly, I'd like to gather a list of what is falling through. @enragedginger - is it possible you could log anything that goes through the default case on your app? it would be really handy to figure out what's going wrong - i'm not deploying anything busy enough to flush it out until early next year.

hstore support, internal representations

As detailed in PR #4, I've added limited hstore support. There is a question of what we should do with hstore because it is a bit inconsistently treated compared to json. cheshire unpicks keywords and such when they are keys in maps. That's rather nice, should we try to emulate it? On the other hand, we can't even have actual numbers because jdbc boxes it into HashMap<String,String>

Travis badge

Travis sucks, but it sucks less than anything I don't have to host myself because I am lazy. My release manager is looking into how to fix travis builds that rely on postgres at the minute (because we need it for utrecht).

It would be nice to show off that our tests pass on a clean envrionment.

Do we want more unidirectional casts?

By this, I mean do we want to be able to accept as input something that we could convert to a given column, but not be able to generate as output?

Examples:

  1. I can (as of pull requests) pass in a byte-array for a bytea and get one back out when it is queried. Do we want to be able to pass in a ByteBuffer, even though we will not generate one from a bytea field?
  2. There are other date and time types in the standard library that we could use

Further question: do some of these indicate we should have some more options like :default-map? I am loathe to make it so easy to shoot yourself in the foot with DateTime-y things, but I can also see a valid use case for wanting e.g. java.util.Date even though it gives us a massive caveat

Clojure.spec tests

I really like clojure.spec. It would be pretty awesome if we could find a way to use it to provide clearer error messages if you misconfigure things, especially since with the one-way casts we're likely to add quite a few more keys.

We can't actually release it with spec in yet because it requires an alpha clojure 1.9, but when 1.9 is out, I think we should seriously consider requiring it, not least for the generative testing.

Thoughts? @ShaneKilkelly

License conflict

The README and the project.clj say it's MIT-Licensed, but the LICENSE file is way too long for that. Looks vaguely like the Eclipse License from memory.

What is it supposed to be licensed under?

Automated releases when the postgres jdbc driver gets bumped

I haven't given too much thought to how this would work, but given that they usually contain security fixes, we should probably keep everything up to date and ensure the code still passes tests.

Ideal scenario is we find an RSS feed somewhere of pg jdbc releases and add a cronjob which pings it, updates it and submits a PR.

Queries dependent upon PG array data type work first time but fail on subsequent calls

The following problem description references the mpg code found here: "mpg/src/mpg/data.clj:56-64".

When querying a Postgres array of strings (varchar[]), the initial query works, but all subsequent queries throw the following error: 'org.postgresql.util.PSQLException: ERROR: column "" is of type character varying[] but expression is of type json. Hint: You will need to rewrite or cast the expression.'. A little investigation revealed that calling .getParameterTypeName on a JDBC ParameterMetaData object from a new PreparedStatement object returns "_varchar". This is an expected response for the mpg implementation of IPersistentVector#'set-parameter, which then uses "varchar" as the element type to create an array object that gets set on the PreparedStatement object. Once that array is set on the PreparedStatment, however, subsequent calls to get the parameter type name from the ParameterMetaData return "varchar[]". This is true for new instances of PreparedStatment objects using the same sql string and db connection. As a result, mpg's implementation of IPersistentVector#'set-parameter defaults to setting a JSON object instead of an array on the PreparedStatement. Hence, the Postgres error mentioned above. Since PreparedStatement objects seem to be cached (for the life of the database connection?) and reused with different query params, some changed state in the PreparedStatement object affects all future queries of that sql statement until the db connection is reset. Updating the sql statement to specify data type (i.e., <field-name>::varchar[]) doesn't seem to have an effect.

You can reproduce the issue with the following steps:

  1. with postgres running, create a new database.
    createdb repro_bug

  2. at a terminal window type:
    psql repro_bug -c 'create table foo (bar character varying(255)[])'

  3. then evaluate the clojure code in the following gist: https://gist.github.com/cwyckoff/faa1cad80b64b726a4900f6fd30c841d

Docs directory is looking a bit bare

At the minute, I'd say the README is doing a better job. It might be nice to include a few fully working basic examples in an 'examples' directory, but I don't think we can add much more value by having just markdown, given the "gets out of your way" nature of the library.

I think the best value from this module is given in conjunction with utrecht and this might make a nice base for samples.

Thoughts?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.