scottpjones / libbson.jl Goto Github PK

View Code? Open in Web Editor NEW

9.0 4.0 16.0 104 KB

libbson bindings for the Julia programming language

License: Other

Julia 100.00%

libbson.jl's Introduction

LibBSON.jl

libbson bindings for The Julia Language

License

This software is licensed under the simplified BSD license; see the file LICENSE.md for details.

Example Usage

using LibBSON

bsonObject = BSONObject(Dict(
    "null" => nothing,
    "bool" => true,
    "int" => 42,
    "double" => 3.141,
    "string" => "Hello, Jérôme",
    "oid" => BSONOID(),
    "minkey" => :minkey,
    "maxkey" => :maxkey,
      "binaryData" => Array{UInt8}(20),
    "array" => Any[5.41, false]
    ))
println(bsonObject)
println(bsonObject["string"])
for (k, v) in bsonObject
    println("$k => $v")
end
bsonArray = BSONArray(Any[
    "one",
    Dict("key" => 6.7)
    ])
for e in bsonArray
    println(e)
end

Please refer to the unit tests in test/runtests.jl for more examples of usage.

Contributing

Contributions are welcome! Please fork on github.com and submit a pull request if you have a contribution you think is worthwhile!

libbson.jl's People

Contributors

Stargazers

Watchers

Forkers

tmlbl tkelman szalmaf rened moisespsena hatsunemiku gearsad slowbrain turtleslow jeffbezanson dehann antonioloureiro pazzo83 zalmoxis standardgalactic

libbson.jl's Issues

Add a build process under Windows

Would it be possible to add support for Windows?

I am not sure how to go about this but am willing to help.

Nested OrderedDicts raise MethodError

I'm writing command methods for Mongo.jl and came across an issue for writing collection commands:

using LibBSON, DataStructures
BSONObject(OrderedDict("a" => OrderedDict("b" => "c")))

Raises:

ERROR: MethodError: `append` has no method matching append(::LibBSON.BSONObject, ::ASCIIString, ::DataStructures.OrderedDict{ASCIIString,ASCIIString})
Closest candidates are:
  append(::LibBSON.BSONObject, ::AbstractString, ::Bool)
  append(::LibBSON.BSONObject, ::AbstractString, ::Union{Int16,Int32,Int8,UInt16,UInt32,UInt8})
  append(::LibBSON.BSONObject, ::AbstractString, ::Union{Int64,UInt64})
  ...
 in call at /Users/isaac/.julia/v0.4/LibBSON/src/BSONObject.jl:18

While

using LibBSON, DataStructures
BSONObject(OrderedDict("a" => Dict("b" => "c")))

returns

BSONObject({ "a" : { "b" : "c" } })

and

using LibBSON, DataStructures
BSONObject(OrderedDict("a" => BSONObject(OrderedDict("b" => "c"))))

returns

BSONObject({ "a" : { "b" : "c" } })

ERROR: MethodError: no method matching done(::BSONObject, ::Bool)

I am updating my code from Julia v0.6 to v0.7.

LibBSON.dict() is throwing an error

BSONObject
wrap: Ptr{Nothing} @0x00007fdfed51f5a0
owner: Core.TypeofBottom Union{}
k = 0, v = 250
ERROR: MethodError: no method matching done(::BSONObject, ::Bool)
Closest candidates are:
done(::I, ::Base.LegacyIterationCompat{I,T,S}) where {I, T, S} at essentials.jl:896
done(::AbstractString, ::Integer) at deprecated.jl:53

securitydoc = BSONObject({ "exchange" : "NSE", "ticker" : "NIFTY_50", "name" : "National Stock Exchange of India (NSE) Stock Index - Nifty 50", "_id" : { "$oid" : "59647a141078ddab805d5cf8" }, "securitytype" : "EQ", "datasources" : [ { "refreshed_at" : "2018-03-13T15:01:17.527Z", "premium" : false, "name" : "National Stock Exchange of India (NSE) Stock Index - Nifty 50", "dataset_code" : "NIFTY_50", "sourcename" : "quandl_NSE", "oldest_available_date" : "1990-07-03", "id" : 30022566, "description" : "Historical indexes for Nifty 50 (NIFTY_50), National Stock Exchange of India (NSE).", "newest_available_date" : "2018-03-13", "database_code" : "NSE", "database_id" : 33, "column_names" : [ "Date", "Open", "High", "Low", "Close", "Shares Traded", "Turnover (Rs. Cr)" ], "frequency" : "daily", "type" : "Time Series" } ], "securityid" : 54312, "country" : "IN", "detail" : { } })

get(LibBSON.dict(securitydoc), "detail", Dict{Any, Any}())

Thanks!

v0.7/v1.0 unit test failures

When I try unit tests for latest branch, there are few tests that are failing. Is master Julia 0.7/1.0 compatible?

   Testing LibBSON
 Resolving package versions...
    Status `/tmp/tmpZKQMDb/Manifest.toml`
  [b99e7846] BinaryProvider v0.5.1
  [b5cb2838] LibBSON v0.3.0+ [`~/.julia/dev/LibBSON`]
  [2a0f44e3] Base64  [`~/Downloads/julia-0.7.0/bin/../share/julia/stdlib/v0.7/Base64`]
  [ade2ca70] Dates  [`~/Downloads/julia-0.7.0/bin/../share/julia/stdlib/v0.7/Dates`]
  [8ba89e20] Distributed  [`~/Downloads/julia-0.7.0/bin/../share/julia/stdlib/v0.7/Distributed`]
  [b77e0a4c] InteractiveUtils  [`~/Downloads/julia-0.7.0/bin/../share/julia/stdlib/v0.7/InteractiveUtils`]
  [76f85450] LibGit2  [`~/Downloads/julia-0.7.0/bin/../share/julia/stdlib/v0.7/LibGit2`]
  [8f399da3] Libdl  [`~/Downloads/julia-0.7.0/bin/../share/julia/stdlib/v0.7/Libdl`]
  [37e2e46d] LinearAlgebra  [`~/Downloads/julia-0.7.0/bin/../share/julia/stdlib/v0.7/LinearAlgebra`]
  [56ddb016] Logging  [`~/Downloads/julia-0.7.0/bin/../share/julia/stdlib/v0.7/Logging`]
  [d6f4376e] Markdown  [`~/Downloads/julia-0.7.0/bin/../share/julia/stdlib/v0.7/Markdown`]
  [44cfe95a] Pkg  [`~/Downloads/julia-0.7.0/bin/../share/julia/stdlib/v0.7/Pkg`]
  [de0858da] Printf  [`~/Downloads/julia-0.7.0/bin/../share/julia/stdlib/v0.7/Printf`]
  [3fa0cd96] REPL  [`~/Downloads/julia-0.7.0/bin/../share/julia/stdlib/v0.7/REPL`]
  [9a3f8284] Random  [`~/Downloads/julia-0.7.0/bin/../share/julia/stdlib/v0.7/Random`]
  [ea8e919c] SHA  [`~/Downloads/julia-0.7.0/bin/../share/julia/stdlib/v0.7/SHA`]
  [9e88b42a] Serialization  [`~/Downloads/julia-0.7.0/bin/../share/julia/stdlib/v0.7/Serialization`]
  [6462fe0b] Sockets  [`~/Downloads/julia-0.7.0/bin/../share/julia/stdlib/v0.7/Sockets`]
  [8dfed614] Test  [`~/Downloads/julia-0.7.0/bin/../share/julia/stdlib/v0.7/Test`]
  [cf7118a7] UUIDs  [`~/Downloads/julia-0.7.0/bin/../share/julia/stdlib/v0.7/UUIDs`]
  [4ec0a83e] Unicode  [`~/Downloads/julia-0.7.0/bin/../share/julia/stdlib/v0.7/Unicode`]
Test Summary: | Pass  Total
BSONOID       |    5      5
BSONObject
  _wrap_: Ptr{Nothing} @0x000000000233f890
  _owner_: Core.TypeofBottom Union{}
k = 16, v = 119
BSONObject: Error During Test at /home/gearsad/.julia/dev/LibBSON/test/runtests.jl:39
  Test threw exception MethodError(done, (BSONObject({ "string" : "Hello, Jérôme", "anotherNull" : null, "null" : null, "regularSymbol" : "symbol", "bool" : true, "int" : 42, "minkey" : { "$minKey" : 1 }, "maxkey" : { "$maxKey" : 1 }, "binaryData" : { "$binary" : "AQIDBAUGBwgJCgsMDQ4PEBESExQ=", "$type" : "00" }, "array" : [ "hello", { "foo" : [ 56, false ] } ], "subdict" : { "key": "value" }, "double" : 0.125 }), false), 0x0000000000006c50)
  Expression: dict(bsonObject) == Dict{Any, Any}("string" => "Hello, Jérôme", "anotherNull" => nothing, "null" => nothing, "regularSymbol" => "symbol", "bool" => true, "int" => 42, "minkey" => :minkey, "maxkey" => :maxkey, "binaryData" => collect(map((t->begin
                            UInt8(t)
                        end), 1:20)), "array" => Any["hello", Dict{Any, Any}("foo" => Any[56, false])], "subdict" => Dict{Any, Any}("key" => "value"), "double" => 0.125)
  MethodError: no method matching done(::BSONObject, ::Bool)
  Closest candidates are:
    done(::I, !Matched::Base.LegacyIterationCompat{I,T,S}) where {I, T, S} at essentials.jl:896
    done(!Matched::AbstractString, ::Integer) at deprecated.jl:53
  Stacktrace:
   [1] iterate at ./essentials.jl:848 [inlined]
   [2] dict(::BSONObject) at /home/gearsad/.julia/dev/LibBSON/src/BSONObject.jl:204
   [3] macro expansion at /home/gearsad/.julia/dev/LibBSON/test/runtests.jl:39 [inlined]
   [4] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Test/src/Test.jl:1079 [inlined]
   [5] top-level scope at /home/gearsad/.julia/dev/LibBSON/test/runtests.jl:21
BSONObject
  _wrap_: Ptr{Nothing} @0x0000000003ef4280
  _owner_: Core.TypeofBottom Union{}
k = 136, v = 66
BSONObject Copy Constructor: Error During Test at /home/gearsad/.julia/dev/LibBSON/test/runtests.jl:51
  Test threw exception MethodError(done, (BSONObject({ "someText" : "hello", "someValue" : -123.45600000000000307 }), false), 0x0000000000006c50)
  Expression: dict(bsonObject) != initialDict
  MethodError: no method matching done(::BSONObject, ::Bool)
  Closest candidates are:
    done(::I, !Matched::Base.LegacyIterationCompat{I,T,S}) where {I, T, S} at essentials.jl:896
    done(!Matched::AbstractString, ::Integer) at deprecated.jl:53
  Stacktrace:
   [1] iterate at ./essentials.jl:848 [inlined]
   [2] dict(::BSONObject) at /home/gearsad/.julia/dev/LibBSON/src/BSONObject.jl:204
   [3] macro expansion at /home/gearsad/.julia/dev/LibBSON/test/runtests.jl:51 [inlined]
   [4] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Test/src/Test.jl:1079 [inlined]
   [5] macro expansion at /home/gearsad/.julia/dev/LibBSON/test/runtests.jl:46 [inlined]
   [6] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Test/src/Test.jl:1079 [inlined]
   [7] top-level scope at /home/gearsad/.julia/dev/LibBSON/test/runtests.jl:21
BSONObject
  _wrap_: Ptr{Nothing} @0x0000000003ef4280
  _owner_: Core.TypeofBottom Union{}
k = 136, v = 66
BSONObject Copy Constructor: Error During Test at /home/gearsad/.julia/dev/LibBSON/test/runtests.jl:52
  Test threw exception MethodError(done, (BSONObject({ "someText" : "hello", "someValue" : -123.45600000000000307 }), false), 0x0000000000006c50)
  Expression: dict(bsonObjectCopyBad) != initialDict
  MethodError: no method matching done(::BSONObject, ::Bool)
  Closest candidates are:
    done(::I, !Matched::Base.LegacyIterationCompat{I,T,S}) where {I, T, S} at essentials.jl:896
    done(!Matched::AbstractString, ::Integer) at deprecated.jl:53
  Stacktrace:
   [1] iterate at ./essentials.jl:848 [inlined]
   [2] dict(::BSONObject) at /home/gearsad/.julia/dev/LibBSON/src/BSONObject.jl:204
   [3] macro expansion at /home/gearsad/.julia/dev/LibBSON/test/runtests.jl:52 [inlined]
   [4] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Test/src/Test.jl:1079 [inlined]
   [5] macro expansion at /home/gearsad/.julia/dev/LibBSON/test/runtests.jl:46 [inlined]
   [6] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Test/src/Test.jl:1079 [inlined]
   [7] top-level scope at /home/gearsad/.julia/dev/LibBSON/test/runtests.jl:21
BSONObject
  _wrap_: Ptr{Nothing} @0x00007efd3baf9d30
  _owner_: Array{UInt8}((128,)) UInt8[0x03, 0x00, 0x00, 0x00, 0x19, 0x00, 0x00, 0x00, 0x19, 0x00  …  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00]
k = 56, v = 157
BSONObject Copy Constructor: Error During Test at /home/gearsad/.julia/dev/LibBSON/test/runtests.jl:53
  Test threw exception MethodError(done, (BSONObject({ "someText" : "hello" }), false), 0x0000000000006c50)
  Expression: dict(bsonObjectCopyGood) == initialDict
  MethodError: no method matching done(::BSONObject, ::Bool)
  Closest candidates are:
    done(::I, !Matched::Base.LegacyIterationCompat{I,T,S}) where {I, T, S} at essentials.jl:896
    done(!Matched::AbstractString, ::Integer) at deprecated.jl:53
  Stacktrace:
   [1] iterate at ./essentials.jl:848 [inlined]
   [2] dict(::BSONObject) at /home/gearsad/.julia/dev/LibBSON/src/BSONObject.jl:204
   [3] macro expansion at /home/gearsad/.julia/dev/LibBSON/test/runtests.jl:53 [inlined]
   [4] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Test/src/Test.jl:1079 [inlined]
   [5] macro expansion at /home/gearsad/.julia/dev/LibBSON/test/runtests.jl:46 [inlined]
   [6] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/Test/src/Test.jl:1079 [inlined]
   [7] top-level scope at /home/gearsad/.julia/dev/LibBSON/test/runtests.jl:21
┌ Warning: `Date()` is deprecated, use `Date(1)` instead.
│   caller = macro expansion at runtests.jl:51 [inlined]
└ @ Core ~/.julia/dev/LibBSON/test/runtests.jl:51
Test Summary:                      | Pass  Error  Total
BSONObject                         |   11      4     15
  BSONObject Copy Constructor      |           3      3
  BSONObject with OID              |    2             2
  BSONObject with DateTime         |    2             2
  BSONObject from JSON             |    2             2
  BSONObject containing BSONObject |    2             2
ERROR: LoadError: Some tests did not pass: 11 passed, 0 failed, 4 errored, 0 broken.

MongoDB uses milliseconds for DateTime while Julia uses seconds

Peter, great work on the LIbBSON / Mongo drivers! DateTimes didn't seem to make it into the last release, so I looked at adding the functionality and noticed that you basically have it implemented in your master branch. However, there seems to be a small inconsistency at the moment since MongoDB uses milliseconds and Julia uses seconds.

For reference see:
https://docs.mongodb.org/manual/reference/bson-types/#date
http://docs.julialang.org/en/release-0.4/stdlib/dates/#Dates.unix2datetime

I believe the simple fixes below should do the trick:

In LibBSON.jl/src/BSONIter.jl:

        elseif ty == BSON_TYPE_DATE_TIME
        ts = ccall(
            (:bson_iter_date_time, libbson),
            Int64, (Ptr{UInt8}, ),
            bsonIter._wrap_
            ) / 1000  # millisecond to second conversion
        return Dates.unix2datetime(ts)

In LibBSON.jl/src/BSONObject.jl:

function append(bsonObject::BSONObject, key::AbstractString, val::Union{Date,DateTime})
    keyCStr = bytestring(key)
    ts = typeof(val) == Date ? datetime2unix(DateTime(val)) : datetime2unix(val)
    ccall(
        (:bson_append_date_time, libbson),
        Bool, (Ptr{Void}, Ptr{UInt8}, Cint, Clong),
        bsonObject._wrap_,
        keyCStr,
        length(keyCStr),
        ts * 1000   # second to millisecond conversion
        ) || error("libBSON: overflow")

In LibBSON.jl/src/BSONArray.jl:

function append(bsonArray::BSONArray, val::Union{Date,DateTime})
    keyCStr = bytestring(string(length(bsonArray)))
    ts = typeof(val) == Date ? datetime2unix(DateTime(val)) : datetime2unix(val)
    ccall(
        (:bson_append_date_time, libbson),
        Bool, (Ptr{Void}, Ptr{UInt8}, Cint, Clong),
        bsonArray._wrap_,
        keyCStr,
        length(keyCStr),
        ts * 1000   # second to millisecond conversion
        ) || error("libBSON: overflow")
end

Turtle

Should LibBSON.BSONObject be an Associative?

julia> supertype(LibBSON.BSONObject)
Any

julia> supertype(Dict)
Associative{K,V} where V where K

How to get dates out of a BSONObject

If the following BSONObject is returned from a (Mongo) database, how to get the "d" date out of this object, at least the unix date integer?

BSONObject({ "_id" : { "$oid" : "55d8f6b1f69dbaebaacd7e58" }, "d" : { "$date" : 1438628400000 }, "f" : 0.25., "s": "hello" })

BSONOID drops references

This type allocates a julia Array, but stores it as a Ptr. The data is still reachable through the Ptr, but the Array object is not referenced and so will be freed by the GC. This is a serious memory bug.

This is a good example of why in v0.4 we're deprecating converting to Ptr; it makes it too easy to accidentally write a type like this without any warning.

Installation error in julia 0.5.0

Any idea what's causing this?

LoadError: SystemError: opening file /Users/IanB/.julia/v0.5/Homebrew/deps/usr/Library/Taps/homebrew/homebrew-core/Formula/mongo-c.rb: No such file or directory
while loading /Users/IanB/.julia/v0.5/LibBSON/deps/build.jl, in expression starting on line 27

This prints just after a @osx_only depreciation warning

Convert BSONObject to Dict

I feel like there's a very simple answer to this, but I can't figure it out. How can I convert a BSONObject into a Dict with the same structure/content?

If I'm not missing something incredibly simple, it may be worth adding it as an example.

(I'm working with Mongo.jl trying to extract a Dict, modify it and re-add it. To do that I have to re-add it as a Dict)

Keys with the same name

Apparently, I can construct a document in MongoDB like,

{
    "title" : "mytitle",
    "somekey" : {
            "someotherkey" : "somedatahere"
    },
    "somekey" : {
            "someotherkey" : "someotherdatahere"
    }
}

Now, when I query this document in Julia, using for doc in find(collection, query()) println(doc) end it returns me -

BSONObject({ "_id" : { "$oid" : "5b2c99fe3d6c3f8629771362" }, "title" : "mytitle", "somekey" : { "someotherkey" : "somedatahere" }, "somekey" : { "someotherkey" : "someotherdatahere" } })

But if ask for doc["somekey"] then it only returns me contents of first somekey.

I agree, that having keys with same name is not quite appropriate but I think it should have been either:

LibBSON had a way to access contents from both the keys directly (without iterating over the entire object and then matching the key names)
Or otherwise, LibBSON threw an error (or some warning maybe?) about duplicate keys.

Info about upcoming removal of packages in the General registry

As described in https://discourse.julialang.org/t/ann-plans-for-removing-packages-that-do-not-yet-support-1-0-from-the-general-registry/ we are planning on removing packages that do not support 1.0 from the General registry. This package has been detected to not support 1.0 and is thus slated to be removed. The removal of packages from the registry will happen approximately a month after this issue is open.

To transition to the new Pkg system using Project.toml, see https://github.com/JuliaRegistries/Registrator.jl#transitioning-from-require-to-projecttoml.
To then tag a new version of the package, see https://github.com/JuliaRegistries/Registrator.jl#via-the-github-app.

If you believe this package has erroneously been detected as not supporting 1.0 or have any other questions, don't hesitate to discuss it here or in the thread linked at the top of this post.

BSONArray uses 0-based indexing

Would it be worth implementing 1-based indexing to match other Julia data structures? Or better to leave it as is?

Migrate to 0.6

Now that 0.6 is out. :)