Git Product home page Git Product logo

orm's Introduction

The ORM library provides a storage backend to persist ML values. This backend is integrated seamlessly with OCaml and currently uses SQLite (although other backends are easily possible). The user does not have to worry about writing any SQL queries manually.

Installation

You can download the latest distribution from Github at http://github.com/mirage/orm. It also depends on the following libraries:

  • dyntype : version 0.9.0+, available from http://github.com/mirage/dyntype

  • ocaml-sqlite3: version 1.5.7+, available from http://www.ocaml.info/home/ocaml_sources.html. Earlier versions had crash bugs which are easily triggered by the ORM library, so please ensure you are up-to-date before reporting bugs.

  • type-conv: version 108.07.00+, available from http://www.ocaml.info/home/ocaml_sources.html

  • sqlite3: version 3.6.22+, available from http://www.sqlite.org/download.html. Earlier versions had crash bugs which are easily triggered by the ORM library, so please ensure you are up-to-date before reporting bugs. Note that you may also need pkg-config installed for this to compile, so on MacOS X do brew install pkg-config sqlite3 to get the latest dependencies for the OCaml bindings.

The library installs an ocamlfind META file, so use it with the orm.syntax package. To compile a file foo.ml with the ORM and findlib, do:

ocamlfind ocamlopt -syntax camlp4o -package orm.syntax -c t.ml

To link it into a standalone executable:

ocamlfind ocamlopt -syntax camlp4o -linkpkg -package orm.syntax t.ml

You can report issues using the Github issue tracker at http://github.com/mirage/orm/issues, or mail the authors at mailto:[email protected]. If you use the ORM somewhere, feel free to drop us a short line and we can add your project to the Wiki as well.

We recommend installation via the OPAM package manager, available at http://opam.ocamlpro.com. Simply do opam install orm.

Usage

For each type definition t annotated with the keyword orm, a tuple of functions to persist and query the saved values are automatically generated:

(* User-defined datatype *)
type t = ... with orm

(* Auto-generated signatures *)
val t_init: string -> (t, [ `RW ]) db
val t_init_read_only: string -> (t, [ `RO ]) db
val t_get: (t, [< `RW | `RO ]) db -> ... -> t list
val t_save: (t, [ `RW ]) db -> t -> unit
val t_delete: (t, [ `RW ]) db -> t -> unit

Example

This example define a basic ML types corresponding to a photo gallery:

type image = string
and gallery = {
    name: string;
    date: float;
    contents: image list;
} with orm

We hold an image as a binary string, and a gallery is a named list of images. First, init functions are generated for both image and gallery:

val image_init : string -> (image, [ `RW ]) db
val gallery_init : string -> (gallery, [ `RW ]) db
val image_init_read_only : string -> (image, [ `RO ]) db
val gallery_init_read_only : string -> (gallery, [ `RO ]) db

Query functions are generated with signatures matching the various fields in the record or object, for example:

val gallery_get : (gallery, [< `RO | `RW ]) db ->
    ?name:[ `Eq string | `Contains string] ->
    ?date:[ `Le float | `Ge float | `Eq float | `Neq float] ->
    ?custom:(gallery -> bool) ->
    gallery list

let my_pics db = gallery_get ~name:(`Contains "Anil") db
let my_pics db = gallery_get ~custom:(fun g -> String.lowercase g.name = "anil") db

To use this, you simply pass the database handle and specify any constraints to the optional variables. More complex functions can be specified using the custom function which filters the full result set (as seen in the second example above).

Be aware that custom functions currently disable the query optimizer and force a full scan. We are investigating ways of exposing relational operations in a future release, and ideas (or even better, patches) are always appreciated.

How It Works

Intuitively, calling gallery_init will:

  1. use dyntype.type-of to translate the type definitions into:

     let type_of_image = Ext ( "image", String )
     let type_of_gallery =
         Ext("gallery", Dict [ 
             ("name", String); ("date", Float) ; ("contents", Enum type_of_image)
     ])
    
  2. use some basic inductive rules to generate the database schema:

     CREATE TABLE image (__id__ INTEGER PRIMARY KEY, image TEXT);
     CREATE TABLE gallery (__id__ INTEGER PRIMARY KEY, gallery__name TEXT, 
         gallery__date REAL, gallery__contents__0 INTEGER);
     CREATE TABLE gallery__contents__0 (__id__ INTEGER PRIMARY KEY,  
         __next__ INTEGER, __size__ INTEGER, gallery__contents__0 INTEGER);
    

Second, using dyntype.value, any value of type image or gallery can be translated into a value of type Value.t. Save functions can be then defined with the signature:

val image_save : (image, [ `RW ]) db -> image -> unit
val gallery_save : (gallery, [ 'RW ]) db -> gallery -> unit

Finally, using Dyntype.type-of, functions to access the database are generated, with the signature:

val image_get : (image, [< `RO | `RW ]) db ->
    ?value:[`Contains of string | `Eq of string] ] ->
    ?custom:(image -> bool) ->
    image list

val gallery_get : (gallery, [< `RO | `RW ]) db ->
    ?name:[ `Eq string | `Contains string] ->
    ?date:[ `Le float | `Ge float | `Eq float | `Neq float] ->
    ?custom:(gallery -> bool) ->
    gallery list

For both types, we are generating:

  1. arguments that can be easily translated into an optimized SQL queries;
  2. a more general (and thus slow) custom query function directly written in OCaml.

On one hand, (1) is achieved by generating optional labelled arguments with the OCaml type corresponding to what Dyntype.type_of generated. This allows the programmer to specify a conjunction of type-safe constraints for his queries. For example, the field name is of type string which is associated to the constraint of type Eq of string | Contains of string. Values of this type can then be mapped to SQL equality or the LIKE operator.

On the other hand, (2) is achieved using a SQLite extension to define custom SQL functions; in our case we register an OCaml callback directly. This is relatively slow as it bypasses the query optimizer, but allows the programmer to define very complex queries.

orm's People

Contributors

avsm avatar mvalle avatar samoht avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

orm's Issues

Express a many-to-many relationship: JOINs needed?

Disclaimer: I don't really know anything about SQL and SQLite.

I was looking at orm to have a simple database for some ocsigen project, and needed to express a many-to-many relationship (I have names and tags, a name being related to many tags, and want to query the tags of a name, and the names that have a given tag). After googling a bit, it seems that the standard way to express such a relationship in SQL requires JOINs operations - which SQLite has, but orm doesn't seem to provide.

Am I missing something? Is this something that will be added in the future?

--Armael

Type problem with get and set

I'm having difficulties getting the types sorted out when passing a Orm.Db.t between functions that mix read and write functionality.
e.g.

type t = {
  mutable x : string;
  mutable y : string;
} with orm()

let get db x =
  t_get db ~x:x

let set db x y =
  let t = List.hd (get db x) in
  t.y <- y;
  t_save db t

Gives the error:
File "test.ml", line 12, characters 9-11:
Error: This expression has type (t, [ RO |RW ]) Orm.Db.t
but an expression was expected of type (t, [ RW ]) Orm.Db.t The second variant type does not allow tag(s)RO

I can't seem to constrain the get function to work only with `RW databases either.

Type issue with boolean getters

Looks like a similar issue to me. The test program,

type t = {
  x : bool;
} with orm()

let get_true db =
  t_get ~x:`True

results in:
File "test.ml", line 6, characters 11-16:
Error: This expression has type [> True ] but an expression was expected of type [<False | True ] The second variant type does not allow tag(s) True

Contains is not working for string gets

The Eq test works, but Contains doesnt seem to:

utop $ #require "orm.syntax";;
utop $ type t = { foo: int; bar: string } with orm;;
<snip scary types>
utop $ let db = t_init "test.db";;
val db : (t, [ `RW ]) Orm.Db.t = <abstr>                                                                                                                  }─utop $ t_save db { foo=1; bar="one" };;
- : unit = ()
}─utop $ t_save db { foo=2; bar="two" };;
- : unit = ()
}─utop $ t_get db;;
- : Cache_t.elt list = [{foo = 1; bar = "one"}; {foo = 2; bar = "two"}]
}─utop $ t_get ~bar:(`Contains "t") db;;
- : Cache_t.elt list = []
}─utop $ t_get ~bar:(`Contains "two") db;;
- : Cache_t.elt list = []
}─utop $ t_get ~bar:(`Eq "two") db;;
- : Cache_t.elt list = [{foo = 2; bar = "two"}]                                                                                                                       utop $ t_get ~foo:(`Ge 0) db;;
- : Cache_t.elt list = [{foo = 1; bar = "one"}; {foo = 2; bar = "two"}]
}─utop $ t_get ~foo:(`Ge 1) db;;
- : Cache_t.elt list = [{foo = 2; bar = "two"}]

Ordering in query

It would be nice to have an ordering option in queries, something like (following the image example in the documentation):

let my_pics db = gallery_get ~order_by:(`date) db

This is important because in-memory sorting after a "SELECT *" is possibly inefficient and even impossible in many cases (large datasets).

orm + cohttp

Hi,

I might be missing something huge, but as far as I can tell orm and cohttp aren't compatible with each other, even though they're both from the mirage project.
orm requires camlp4o to compile (or "with orm" doesn't work), and cohttp can't have camlp4o enabled or it's ~mode: when creating a server is an error, since mode is a keyword.

Is there no way to have both in the same project ? I'm not usually an ocaml user, so that might be me.
Bonus question, is there any kind of doc for orm ? I've been figuring it out using the tests in lib_test, but that's really not ideal.

Thanks !

Lazy getters

Another important point, now missing in this project, is the availability of some sort of "lazy getter". I have in mind something like the Enum module present in Batteries. The x_get (or x_get_enum) function should return an enumeration, so that a "SELECT *" on a large dataset wouldn't blow up the server.

This is a naive (but working) implementation for the Sqlite3 backend:

open BatStd;;

module MySqlite3 =
  struct
    include Sqlite3

    let enum (db : Sqlite3.db) (query : string) : Sqlite3.Data.t BatMap.StringMap.t BatEnum.t =
      (* stmt creation *)
      let stmt = Sqlite3.prepare db query in

      let rec prepare_tail stmt =
        match Sqlite3.prepare_tail stmt with
        | None -> ()
        | Some s -> prepare_tail s in
      prepare_tail stmt;

      (* Enum "next" *)
      let next () =
        try
          match Sqlite3.step stmt with
          | Sqlite3.Rc.ROW -> begin
              let col_names = Sqlite3.row_names stmt in
              let col_data = Sqlite3.row_data stmt in
              let len = Array.length col_names in
              let map = ref BatMap.StringMap.empty in
              for i = 0 to (len - 1) do
                map := BatMap.StringMap.add col_names.(i) col_data.(i) !map;
              done;
              !map
            end
          | Sqlite3.Rc.DONE -> begin
              Sqlite3.finalize stmt |> ignore;
              raise BatEnum.No_more_elements
            end
          | e -> begin
              failwith (Sqlite3.Rc.to_string e)
            end;
        with
        | Sqlite3.Error "Sqlite3.step called with finalized stmt" ->
            raise BatEnum.No_more_elements
      in (* let next ... *)
      BatEnum.from next
  end
;;

It returns an enumeration of maps representing records, with the keys (strings) populated with the columns name, and the values being the corresponding Sqlite3 values. You could return the "X" type, of course.

orm type_conv not working in utop

utop $ type t = { foo: int; bar: string } with orm;;
Error: Failure: "Pa_type_conv: \"orm\" is not a supported type generator. (supported generators: bin_io, variants, of_sexp, bin_read, bin_type_class, sexp_of, sexp, fields, bin_write)"

with orm-0.6.3 and dyntype-0.8.4 (latest opam)

Install fails with opam due to Unbound module Big_int

Build fails due to Big_int being unbound.

$ opam switch 
#  switch   compiler             description
→  4.12.0   ocaml-system.4.12.0  4.12.0
   default  ocaml.4.12.0         default

$ opam install orm          
The following actions will be performed:
  ∗ install orm 0.7.1

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫 
⬇ retrieved orm.0.7.1  (cached)
[ERROR] The compilation of orm.0.7.1 failed at "make".

#=== ERROR while compiling orm.0.7.1 ==========================================#
# context     2.1.0 | macos/x86_64 | ocaml-system.4.12.0 | https://opam.ocaml.org#7247859e
# path        ~/.opam/4.12.0/.opam-switch/build/orm.0.7.1
# command     ~/.opam/opam-init/hooks/sandbox.sh build make
# exit-code   2
# env-file    ~/.opam/log/orm-95495-692be4.env
# output-file ~/.opam/log/orm-95495-692be4.out
### output ###
# [...]
# Use Array.make/ArrayLabels.make instead.
# File "_none_", line 1:
# Warning 58 [no-cmx-file]: no cmx file was found in path for module Dyntype, and its interface was not compiled with -opaque
# ocamlfind ocamlopt -package sqlite3,dyntype.syntax -c -annot -for-pack Orm sql_get.ml
# File "sql_get.ml", line 76, characters 53-78:
# 76 | 		| `Big_int i -> int_like name (fun i -> Data.TEXT (Big_int.string_of_big_int i)) i
#                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^
# Error: Unbound module Big_int
# make[3]: *** [sql_get.cmi] Error 2
# make[2]: *** [native-code-library] Error 2
# make[1]: *** [all] Error 2
# make: *** [all] Error 2



<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫 
┌─ The following actions failed
│ λ build orm 0.7.1
└─ 
╶─ No changes have been performed

Signature generator needed for scary types

We can improve on:

utop $ type t = { foo: int; bar: string } with orm;;
type t = { foo : int; bar : string; }
val __env__ : Orm.Sql_backend.env = [] 
val hash_of_t : t -> int = <fun>                                                                                                                                      
module Cache_t : sig type tbl type elt = t val create : string -> (tbl, elt) Orm.Sql_cache.t end
val t_cache : (Cache_t.tbl, Cache_t.elt) Orm.Sql_cache.t =
  {Orm.Sql_cache.type_name = "t"; Orm.Sql_cache.tbl = <abstr>; Orm.Sql_cache.create = <fun>; 
   Orm.Sql_cache.to_weakid = <fun>; Orm.Sql_cache.of_weakid = <fun>; 
   Orm.Sql_cache.mem = <fun>; Orm.Sql_cache.mem_weakid = <fun>; Orm.Sql_cache.add = <fun>; 
   Orm.Sql_cache.remove = <fun>; Orm.Sql_cache.replace = <fun>; Orm.Sql_cache.dump = <fun>}
val type_of_t : Dyntype.Type.t = Dyntype.Type.Ext ("t", Dyntype.Type.Dict (`R, [("foo", `RO, Dyntype.Type.Int (Some 63)); ("bar", `RO, Dyntype.Type.String)]))
val value_of_t : id_seed:Orm.Sql_backend.state -> t -> Dyntype.Value.t = <fun>
val t_of_value : Orm.Sql_get.V.t -> t = <fun>
module ORMID_t : Orm.Sig.ID
val t_init : string -> (t, [ `RW ]) Orm.Db.t = <fun>
val t_init_read_only : string -> (t, [ `RO ]) Orm.Db.t = <fun>
val t_save : db:(t, [ `RW ]) Orm.Db.t -> Cache_t.elt -> unit = <fun>
val t_get :
  ?foo:[< `Eq of int | `Ge of int | `Geq of int | `Le of int | `Leq of int | `Neq of int ] ->
  ?bar:[< `Contains of string | `Eq of string ] -> ?custom:(t -> bool) -> ?order_by:[> `bar | `foo ] -> (t, [< `RO | `RW ]) Orm.Db.t -> Cache_t.elt list = <fun>
val t_lazy_get :
  ?foo:[< `Eq of int | `Ge of int | `Geq of int | `Le of int | `Leq of int | `Neq of int ] ->
  ?bar:[< `Contains of string | `Eq of string ] -> ?custom:(t -> bool) -> ?order_by:[> `bar | `foo ] -> (t, [< `RO | `RW ]) Orm.Db.t -> unit -> Cache_t.elt option =
  <fun>
val t_get_by_id : id:[< `Eq of ORMID_t.t ] -> (t, [< `RO | `RW ]) Orm.Db.t -> Cache_t.elt = <fun>
val t_delete : ?recursive:bool -> db:(t, [ `RW ]) Orm.Db.t -> Cache_t.elt -> unit = <fun>
val t_id : db:(t, [< `RO | `RW ]) Orm.Db.t -> Cache_t.elt -> ORMID_t.t = <fun>
module ORM_t :
  sig
    type t = t
    type id = ORMID_t.t
    type 'get_result get_params =
        ?foo:[ `Eq of int | `Ge of int | `Geq of int | `Le of int | `Leq of int | `Neq of int ] -> ?bar:[ `Contains of string | `Eq of string ] -> 'get_result
    type order_by = [ `bar | `foo ]
    val init : string -> (t, [ `RW ]) Orm.Db.t
    val init_read_only : string -> (t, [ `RO ]) Orm.Db.t
    val save : db:(t, [ `RW ]) Orm.Db.t -> t -> unit
    val get : (?custom:(t -> bool) -> ?order_by:order_by -> (t, [< `RO | `RW ]) Orm.Db.t -> t list) get_params
    val lazy_get : (?custom:(t -> bool) -> ?order_by:order_by -> (t, [< `RO | `RW ]) Orm.Db.t -> unit -> t option) get_params
    val get_by_id : id:[ `Eq of id ] -> (t, [< `RO | `RW ]) Orm.Db.t -> t
    val delete : ?recursive:bool -> db:(t, [ `RW ]) Orm.Db.t -> t -> unit
    val id : db:(t, [< `RO | `RW ]) Orm.Db.t -> t -> id

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.