Evaluate options for a language-independent checkpoint/serialization format for the CLA about nupic HOT 20 CLOSED

Thrift is a PITA both conceptually and in implementation.

--Please excuse brevity, sent from phone.--
On Oct 24, 2013 5:01 PM, "Matthew Taylor" [email protected] wrote:

The current CLA model checkpoint uses the pickle module. As we move
towards multiple language support and more external model sharing, we
should define a language-independent format for serializing the CLA.

The major objectives for this would be cross-language implementation (i.e.
we don't have to create serialize functions separately for each language),
speed, checkpoint size, and ease of development and versioning.

Some options include:

• Protocol buffers
• Thrift
• Cap'n proto - a little early in development but has some nice
  properties
• Some form of binary JSON

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/333
.

from nupic.

@iandanforth Yes I have heard (and experienced) similar opinions. Frank mentioned the same in JIRA I believe. Something like MessagePack is easy to use but is harder to maintain and doesn't work as well across languages (as far as I can tell).

I am leaning towards protobufs or similar. I quite like Cap'n Proto and there is active development but previously is was still a bit raw. One potential advantage of it over PB is that it has planned support for memory mapping that could make deserialize-run-serialize operations very fast (this is the current Grok use case).

from nupic.

Here is my branch with initial Python spatial pooler test:
https://github.com/scottpurdy/nupic/tree/sp-capnp

The files are in nupic/proto.

My initial results weren't very good. I suspect there is type conversion going on. It would probably be better to do the first tests in C++ where it is more obvious what is happening. And I wasn't doing thorough profiling, just saw that the time to create the model and the time to feed a record in were longer. From a theoretical standpoint, I am fairly confident that we can make serialization/deserialization and runtime both faster though.

from nupic.

👍

from nupic.

There are currently four different options we want to measure times for: C++ and Python protocol buffers and C++ and Python Cap'n Proto buffers.

There are two ways to use these. In the simple scenario, we leave the code pretty much the same and simply create and populate the buffers when we need to serialize. The other is to actually use the buffer in memory during execution. This doesn't work for some fields like sparse matrices but it works for most everything else.

In the former case, we want to measure the times for the following operations:

• Time to create and populate the buffer
• Time to serialize the buffer to disk
• Time to deserialize the buffer into memory
• Time to create the object and copy data from the buffer into it

In the case that we use the buffer in memory during execution, we would want to measure the time it takes to run records through in addition to the times for copying/serializaing/deserialization.

Finally, when doing these timing tests it is important to run some records through the pre-serialization and post-de-serialization objects and compare the results to ensure that everything is implemented correctly (wouldn't be fair timing tests is some pieces were left out!).

from nupic.

Yeah, seen the code and it's taking the approach of "actually use the buffer in memory during execution".

from nupic.

It would probably be better to do the first tests in C++ where it is more obvious what is happening. And I wasn't doing thorough profiling, just saw that the time to create the model and the time to feed a record in were longer.

It seems that Cap’n Proto does the encoding when feeding the data, and decoding on retrieval:

Cap’n Proto gets a perfect score because there is no encoding/decoding step. The Cap’n Proto encoding is appropriate both as a data interchange format and an in-memory representation, so once your structure is built, you can simply write the bytes straight out to disk!

So, I guess it's definitely not an option to use Cap’n Proto as internal structure of SP and such, it will slow things down.

from nupic.

Captain Proto is currently C++11 only and I haven't been able to get it to link with the current nupic.core (or even Marek's C++11branch)

Performance with protocol buffers looks good: both the stages of creating the protocol buffer and serializing, as well as deserialization and loading variables back into the class fields were about 2 times faster than the current implementation. Right now my implementation doesn't use the protobuf object in memory throughout SP execution, but allocates and populates it when the save function is called.

from nupic.

I'm wondering... what kind of API should we need for language bindings?
Should it be an IDL kind of stuff to allow generic RPC --- procedural?
or should it be a memcached or SQL kind of text/binary protocol to allow generic data access -- data oriented?

It also depends on the medium to communicate upon; dynamic linking mechanism or mere network packets. We can do binding in multiple levels. Does the efficiency matter? The speed can be optimized for throughput, latency, operations per second, ..., etc.

from nupic.

When I get back from vacation, I want to have a discussion about updating to C++11. That might change the implementation of this issue significantly.

from nupic.

There's #130 taking the approach of using google protocol buffer. Just wonder if anyone has noticed and evaluate FlatBuffers which is also the successor of protocol buffer and developed by google. At first glance, it also provides "Access to serialized data without parsing/unpacking" and has miscellaneous efficiency improvements just like Cap'n proto .

from nupic.

@scottpurdy Would you call this ticket complete?

from nupic.

rhyolight commented a day ago

@scottpurdy Would you call this ticket complete?

If it's complete, what's the conclusion?

from nupic.

We don't have a decision on this yet. I think it makes sense to keep tracking here. I will create a follow up issue to track the implementation to be done after we finalize the decision.

from nupic.

Now that we have a C++11 nupic.core I am going to attempt Capn Proto again.

from nupic.

We have more motivation for this issue from #1231 which is a somewhat serious bug in NuPIC.

from nupic.

@scottpurdy Once C++11 is finished across nupic and nupic.core, the decision is to start a Cap'n Proto implementation, right?

from nupic.

@rhyolight - that is my current plan, yes

from nupic.

Shall we close this yet?

Matt Taylor
OS Community Flag-Bearer
Numenta

On Fri, Oct 3, 2014 at 10:28 AM, Scott Purdy [email protected]
wrote:

@rhyolight https://github.com/rhyolight - that is my current plan, yes

—
Reply to this email directly or view it on GitHub
#333 (comment).

from nupic.

Closing, assuming we're going with Cap'n Proto in #1336.

from nupic.