TestSPFrequency.testCategory test failure in Vagrant VM about nupic HOT 10 CLOSED

The entire output of the failure as requested by Matt Taylor:
https://gist.github.com/mtbvang/7458134

from nupic.

I ran up my own Lubuntu vagrant box and set it up using the instructions on https://github.com/numenta/nupic/wiki/Install-Nupic-on-ubuntu-13.04 and all the tests passed using code from:

commit e3719ae
Merge: 23a5225 0c36eb9
Author: Matthew Taylor [email protected]
Date: Tue Nov 12 11:00:00 2013 -0800

The code that failed with the Numenta VM was from:

commit 23a5225
Merge: 5fe9e3e 4abef3c
Author: Matthew Taylor [email protected]
Date: Fri Nov 8 11:39:45 2013 -0800

I hope this isn't a red herring caused by it being my first time environment setup.

My system is:

Lubuntu 13.10 64bit
Virtualbox 4.3.2
Vagrant 1.3.5

from nupic.

Hi,

I don't understand what you''re observing? You say all tests pass oK with e3719ae from Nov 12 (= master/HEAD), but then what does the 23a5225 from Nov 8 mean?

Or did you figure 23a5225 is the offending commit? (shouldn't be) - try git revert 23a5225

from nupic.

Breznak, on the 8 Nov I tried runniing up the Numenta VM and got the test failures. Since then I have also run up my own installation on Lubuntu using the wiki/Install-Nupic-on-ubuntu-13.04 intructions and all the test ran fine on that. I haven't gone back to try and get the Numenta VM since the test error.

from nupic.

Ah, I see. So changing the title. Glad it runs for you somehow, so it's not a hot issue. Feel free to comment on the bug if you do retest that, or just close if you dont bother anymore.

from nupic.

I'm hitting this same error. I hit it with the unit tests
$NUPIC/run_tests.sh
and with the example
python $NUPIC/examples/opf/bin/OpfRunExperiment.py $NUPIC/examples/opf/experiments/multistep/hotgym/

The common error is:
TypeError: Cannot cast array data from dtype('float32') to dtype('<m8') according to the rule 'safe'

More fully:
[vagrant@nupic-vagrant:/vagrant]$ python $NUPIC/examples/opf/bin/OpfRunExperiment.py $NUPIC/examples/opf/experiments/multistep/hotgym/ Using logging configuration file: /home/vagrant/nta/eng/conf/default/nupic-logging.conf Traceback (most recent call last): File "/vagrant/examples/opf/bin/OpfRunExperiment.py", line 51, in <module> main() File "/vagrant/examples/opf/bin/OpfRunExperiment.py", line 46, in main runExperiment(sys.argv[1:]) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/frameworks/opf/experiment_runner.py", line 147, in runExperiment model = _runExperimentImpl(opt, model) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/frameworks/opf/experiment_runner.py", line 409, in _runExperimentImpl model = ModelFactory.create(modelDescription) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/frameworks/opf/modelfactory.py", line 76, in create return modelClass(**modelConfig['modelParams']) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/frameworks/opf/clamodel.py", line 203, in __init__ anomalyParams) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/frameworks/opf/clamodel.py", line 1200, in __createCLANetwork n.initialize() File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/engine/__init__.py", line 694, in initialize engine.Network.initialize(self, *args, **kwargs) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/bindings/engine_internal.py", line 1034, in initialize return _engine_internal.Network_initialize(*args, **kwargs) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/regions/SPRegion.py", line 459, in initialize self._allocateSpatialFDR(None) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/regions/SPRegion.py", line 516, in _allocateSpatialFDR **autoArgs) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/research/FDRCSpatial2.py", line 449, in __init__ self.random)) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/research/FDRCSpatial2.py", line 1342, in _makeMasterCoincidences masterPotentialM.append(SM_01_32_32(denseM.reshape(coincRFShape))) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/bindings/math.py", line 4572, in __init__ self.fromDense(numpy.asarray(args[0])) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/bindings/math.py", line 4695, in fromDense return _math._SM_01_32_32_fromDense(*args, **kwargs) TypeError: Cannot cast array data from dtype('float32') to dtype('<m8') according to the rule 'safe'

This is with the Numenta VM from https://github.com/numenta/nupic/wiki/Running-Nupic-in-a-Virtual-Machine running on VirtualBox running on Windows.

I haven't successfully worked around the issue.

from nupic.

I did some debugging. I plan on continuing, but here's what I learned.

Here's a gdb snippet that shows where this is going very badly:

3877        if (type < NPY_NTYPES) {
(gdb) n
3878            ret = _builtin_descrs[type];
(gdb) p type
$31 = 22
(gdb) backtrace
#0  PyArray_DescrFromType (type=22)
    at numpy/core/src/multiarray/arraytypes.c.src:3878
#1  0x00007fa3d3da8abb in nta::NumpyArray::NumpyArray (this=0x7ffff08b2860,
    p=0x2e89350, dtype=6, requiredDimension=2)
    at /vagrant/lang/py/support/NumpyVector.cpp:147
#2  0x00007fa3d3c50e24 in NumpyMatrixT (args=<value optimized out>,
    kwargs=<value optimized out>) at /vagrant/lang/py/support/NumpyVector.hpp:236
#3  nta_SparseBinaryMatrix_Sl_nta_UInt32_Sc_nta_UInt32_Sg__fromDense (
    args=<value optimized out>, kwargs=<value optimized out>)
    at /tmp/ntabuild/py/bindings/math/math_pyPYTHON_wrap.cxx:6665
#4  _wrap__SM_01_32_32_fromDense (args=<value optimized out>,
    kwargs=<value optimized out>)
    at /tmp/ntabuild/py/bindings/math/math_pyPYTHON_wrap.cxx:37721

If you look around, this should never happen. This function is called with type == PyArray_NOTYPE, i.e. NPY_NOTYPE, i.e. the value in the enum that comes after NPY_NTYPES. So it should not have entered into this if (type < NPY_NTYPES) block.

Here's the enum in question.

enum NPY_TYPES {    NPY_BOOL=0,
                    NPY_BYTE, NPY_UBYTE,
                    NPY_SHORT, NPY_USHORT,
                    NPY_INT, NPY_UINT,
                    NPY_LONG, NPY_ULONG,
                    NPY_LONGLONG, NPY_ULONGLONG,
                    NPY_FLOAT, NPY_DOUBLE, NPY_LONGDOUBLE,
                    NPY_CFLOAT, NPY_CDOUBLE, NPY_CLONGDOUBLE,
                    NPY_OBJECT=17,
                    NPY_STRING, NPY_UNICODE,
                    NPY_VOID,
                    /*
                     * New 1.6 types appended, may be integrated
                     * into the above in 2.0.
                     */
                    NPY_DATETIME, NPY_TIMEDELTA, NPY_HALF,

                    NPY_NTYPES,
                    NPY_NOTYPE,
                    NPY_CHAR,      /* special flag */
                    NPY_USERDEF=256,  /* leave room for characters */

                    /* The number of types not including the new 1.6 types */
                    NPY_NTYPES_ABI_COMPATIBLE=21
};

So /vagrant/lang/py/support/NumpyVector.cpp thinks that NPY_NOTYPE is 22, eh? Well, multiarray.so thinks that 22 corresponds to NPY_TIMEDELTA. You can see how this would lead to this error message. dtype('<m8') is a DateTime datatype.

So the binaries are using different versions of an enum. I'll try to find where it's pulling the stale enum value...

from nupic.

Interesting, the issue stopped reproing after I specifically ran:
make lang/py/support/NumpyVector.o
make support
make _math/fast
from $NUPIC/build_system (after editing $NUPIC/lang/py/support/NumpyVector.cpp to mark the file as dirty). Now when I step through, it uses enum value 25 instead of 22.

I might be pinpointing this incorrectly -- I ran a few other build commands today not mentioned above. But my point is that this seems to be a subtle build-ordering issue.

from nupic.

...and now I synced to latest, and it's not reproing. Not even on a fresh VM. My previous comment is still correct -- I went from repro -> no repro without changing/syncing any code. Anyway, maybe something from the past week fixed the issue.

from nupic.

@mtbvang Can you pull the latest, rebuild and try running tests again? I'd like to see if the problem is still occurring in your environment.

from nupic.