Git Product home page Git Product logo

Comments (12)

ivmai avatar ivmai commented on July 17, 2024

Please provide stack trace
-- вторник, 22 декабря 2015г., 21:42 +03:00 от Linas Vepštas < [email protected]> :

Current git master HEAD and also tag gc7_4_2 fail with error Assertion failure: reclaim.c:314 almost immediately, on a certain complex app, when I build bdwgc with
./configure --enable-cplusplus --enable-gc-assertions --enable-large-config
The failure goes away when built without the --enable-gc-assertions flag (and things seem to work fine). I don't have any simple way of reproducing this; its a large complex app using guile, and guile is the only component that calls bdwgc directly. (Note to self: its the AnaphoraUTest that fails)
Side remark: without --enable-large-config I was hitting Too many root sets. Now it works fine(barring above). Given today's large machines, perhaps its time for a --enable-super-large-config?

Reply to this email directly or view it on GitHub .

from bdwgc.

linas avatar linas commented on July 17, 2024

Yes, of course. Silly me. It's huge, though. I'm not trimming it; perhaps the excessive complexity/stack size is part of the problem.

#0  0x00007ffff782dcc9 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff78310d8 in __GI_abort () at abort.c:89
#2  0x00007ffff14a4908 in GC_reclaim_generic (hbp=hbp@entry=0xd6e000, 
    hhdr=0xb307f0, sz=800, init=0, list=0x0, 
    count=count@entry=0x7ffff16f79a8 <GC_bytes_found>) at reclaim.c:314
#3  0x00007ffff14a4970 in GC_reclaim_small_nonempty_block (
    hbp=hbp@entry=0xd6e000, report_if_found=report_if_found@entry=0)
    at reclaim.c:340
#4  0x00007ffff14a4ef1 in GC_continue_reclaim (sz=sz@entry=50, 
    kind=kind@entry=5) at reclaim.c:680
#5  0x00007ffff1494818 in GC_allocobj (gran=gran@entry=50, kind=5)
    at alloc.c:1343
#6  0x00007ffff149a6e5 in GC_generic_malloc_inner (lb=lb@entry=744, 
    k=k@entry=5) at malloc.c:133
#7  0x00007ffff149a842 in GC_generic_malloc (lb=744, k=5) at malloc.c:182
#8  0x00007ffff18b043e in allocate_entries (size=size@entry=31, 
    kind=kind@entry=SCM_WEAK_TABLE_KIND_KEY) at ../../libguile/weak-table.c:362
#9  0x00007ffff18b04ce in make_weak_table (k=<optimized out>, 
    kind=kind@entry=SCM_WEAK_TABLE_KIND_KEY) at ../../libguile/weak-table.c:777
#10 0x00007ffff18b05cb in scm_c_make_weak_table (kind=SCM_WEAK_TABLE_KIND_KEY, 
    k=<optimized out>) at ../../libguile/weak-table.c:830
#11 scm_make_weak_key_hash_table (n=<optimized out>)
    at ../../libguile/weak-table.c:1064
#12 0x00007ffff18abfc2 in vm_regular_engine (thread=0x7014, vp=0xc71d80, 
    registers=0x6, resume=10) at ../../libguile/vm-engine.c:800
#13 0x00007ffff18ae969 in scm_call_n (proc=0xd6ce20, argv=argv@entry=0x0, 
    nargs=nargs@entry=0) at ../../libguile/vm.c:1236
#14 0x00007ffff1837c69 in scm_call_0 (proc=<optimized out>)
    at ../../libguile/eval.c:473
#15 0x00007ffff18aeb35 in scm_load_compiled_with_vm (file=file@entry=0xd6a320)
    at ../../libguile/vm.c:1475
#16 0x00007ffff1855ec5 in scm_primitive_load_path (args=<optimized out>)
    at ../../libguile/load.c:1069
#17 0x00007ffff18abfc2 in vm_regular_engine (thread=0x7014, vp=0xc71d80, 
    registers=0x6, resume=10) at ../../libguile/vm-engine.c:800
#18 0x00007ffff18ae969 in scm_call_n (proc=0xd64c50, argv=argv@entry=0x0, 
    nargs=nargs@entry=0) at ../../libguile/vm.c:1236
#19 0x00007ffff1837c69 in scm_call_0 (proc=<optimized out>)
    at ../../libguile/eval.c:473
#20 0x00007ffff18aeb35 in scm_load_compiled_with_vm (file=file@entry=0xd62560)
    at ../../libguile/vm.c:1475
#21 0x00007ffff1855ec5 in scm_primitive_load_path (args=<optimized out>)
    at ../../libguile/load.c:1069
#22 0x00007ffff18abfc2 in vm_regular_engine (thread=0x7014, vp=0xc71d80, 
    registers=0x6, resume=10) at ../../libguile/vm-engine.c:800
#23 0x00007ffff18ae969 in scm_call_n (proc=0xd5b340, argv=argv@entry=0x0, 
    nargs=nargs@entry=0) at ../../libguile/vm.c:1236
#24 0x00007ffff1837c69 in scm_call_0 (proc=<optimized out>)
    at ../../libguile/eval.c:473
#25 0x00007ffff18aeb35 in scm_load_compiled_with_vm (file=file@entry=0xd5dae0)
    at ../../libguile/vm.c:1475
#26 0x00007ffff1855ec5 in scm_primitive_load_path (args=<optimized out>)
    at ../../libguile/load.c:1069
#27 0x00007ffff18abfc2 in vm_regular_engine (thread=0x7014, vp=0xc71d80, 
    registers=0x6, resume=10) at ../../libguile/vm-engine.c:800
#28 0x00007ffff18ae969 in scm_call_n (proc=0xd19a00, argv=argv@entry=0x0, 
    nargs=nargs@entry=0) at ../../libguile/vm.c:1236
#29 0x00007ffff1837c69 in scm_call_0 (proc=<optimized out>)
    at ../../libguile/eval.c:473
#30 0x00007ffff18aeb35 in scm_load_compiled_with_vm (file=file@entry=0xd16a40)
    at ../../libguile/vm.c:1475
#31 0x00007ffff1855ec5 in scm_primitive_load_path (args=<optimized out>)
    at ../../libguile/load.c:1069
#32 0x00007ffff18abfc2 in vm_regular_engine (thread=0x7014, vp=0xc71d80, 
    registers=0x6, resume=10) at ../../libguile/vm-engine.c:800
#33 0x00007ffff18ae969 in scm_call_n (proc=0xcfa680, 
    argv=argv@entry=0x7fffffffb2b0, nargs=nargs@entry=3)
    at ../../libguile/vm.c:1236
#34 0x00007ffff1837ccf in scm_call_3 (proc=<optimized out>, 
    arg1=arg1@entry=0xd19ae0, arg2=arg2@entry=0xce7a60, arg3=arg3@entry=0x4)
    at ../../libguile/eval.c:493
#35 0x00007ffff185ab90 in scm_public_variable (module_name=0xd19ae0, 
    name=0xc8a620) at ../../libguile/modules.c:636
#36 0x00007ffff189a5c7 in init_eval_string_var_and_k_module ()
    at ../../libguile/strports.c:463
#37 0x00007ffff7bc9a90 in pthread_once ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:103
#38 0x00007ffff189a7dd in scm_eval_string_in_module (string=0xd16c40, 
    module=0x904) at ../../libguile/strports.c:479
#39 0x00007ffff1b25520 in opencog::SchemeEval::init (this=this@entry=0xaf1530)
    at /home/linas/src/novamente/src/atomspace-git/opencog/guile/SchemeEval.cc:44
#40 0x00007ffff1b255c9 in opencog::SchemeEval::c_wrap_init (p=0xaf1530)
    at /home/linas/src/novamente/src/atomspace-git/opencog/guile/SchemeEval.cc:153
#41 0x00007ffff1832e1a in c_body (d=0x7fffffffb730)
    at ../../libguile/continuations.c:429
#42 0x00007ffff18abfc2 in vm_regular_engine (thread=0x7014, vp=0xc71d80, 
    registers=0x6, resume=10) at ../../libguile/vm-engine.c:800
#43 0x00007ffff18ae969 in scm_call_n (proc=proc@entry=0xd16ca0, 
    argv=argv@entry=0x0, nargs=nargs@entry=0) at ../../libguile/vm.c:1236
#44 0x00007ffff1837c69 in scm_call_0 (proc=proc@entry=0xd16ca0)
    at ../../libguile/eval.c:473
#45 0x00007ffff18a0212 in catch (tag=tag@entry=0x404, thunk=0xd16ca0, 
    handler=0xd16c80, pre_unwind_handler=0xd16c60)
    at ../../libguile/throw.c:135
#46 0x00007ffff18a0645 in scm_catch_with_pre_unwind_handler (
    key=key@entry=0x404, thunk=<optimized out>, handler=<optimized out>, 
    pre_unwind_handler=<optimized out>) at ../../libguile/throw.c:249
#47 0x00007ffff18a06ff in scm_c_catch (tag=tag@entry=0x404, 
    body=body@entry=0x7ffff1832e10 <c_body>, 
    body_data=body_data@entry=0x7fffffffb730, 
    handler=handler@entry=0x7ffff18331a0 <c_handler>, 
    handler_data=handler_data@entry=0x7fffffffb730, 
    pre_unwind_handler=pre_unwind_handler@entry=0x7ffff1832f50 <pre_unwind_handler>, pre_unwind_handler_data=pre_unwind_handler_data@entry=0xcd24a0)
    at ../../libguile/throw.c:370
#48 0x00007ffff18334ef in scm_i_with_continuation_barrier (
    body=body@entry=0x7ffff1832e10 <c_body>, 
    body_data=body_data@entry=0x7fffffffb730, 
    handler=handler@entry=0x7ffff18331a0 <c_handler>, 
    handler_data=handler_data@entry=0x7fffffffb730, 
    pre_unwind_handler=pre_unwind_handler@entry=0x7ffff1832f50 <pre_unwind_handler>, pre_unwind_handler_data=0xcd24a0) at ../../libguile/continuations.c:367
#49 0x00007ffff1833585 in scm_c_with_continuation_barrier (
    func=<optimized out>, data=<optimized out>)
    at ../../libguile/continuations.c:463
#50 0x00007ffff189e1ac in with_guile_and_parent (
    base=base@entry=0x7fffffffb790, data=data@entry=0x7fffffffb7c0)
    at ../../libguile/threads.c:786
#51 0x00007ffff14a1012 in GC_call_with_stack_base (
    fn=fn@entry=0x7ffff189e160 <with_guile_and_parent>, 
    arg=arg@entry=0x7fffffffb7c0) at misc.c:1840
#52 0x00007ffff189e668 in scm_i_with_guile_and_parent (parent=<optimized out>, 
    data=<optimized out>, func=<optimized out>) at ../../libguile/threads.c:829
#53 scm_with_guile (func=<optimized out>, data=<optimized out>)
    at ../../libguile/threads.c:835
#54 0x00007ffff1b26a08 in opencog::SchemeEval::SchemeEval (this=0xaf1530, 
    as=0x0)
    at /home/linas/src/novamente/src/atomspace-git/opencog/guile/SchemeEval.cc:313
#55 0x00007ffff1b2807e in get_from_pool ()
    at /home/linas/src/novamente/src/atomspace-git/opencog/guile/SchemeEval.cc:1113
#56 opencog::SchemeEval::get_evaluator (as=0xa94330)
    at /home/linas/src/novamente/src/atomspace-git/opencog/guile/SchemeEval.cc:1165
#57 0x00007ffff1d49446 in opencog::eval_scheme (as=..., s=...)
    at /home/linas/src/novamente/src/atomspace-git/opencog/cython/opencog/PyScheme.cc:35
#58 0x00007ffff1d49df0 in __pyx_pf_7opencog_14scheme_wrapper_scheme_eval (
    __pyx_self=<optimized out>, __pyx_v_a=0x7ffff7f56210, 
    __pyx_v_s=<optimized out>)
    at /home/linas/src/novamente/src/atomspace-git/build/opencog/cython/opencog/scheme_wrapper.cpp:996
#59 __pyx_pw_7opencog_14scheme_wrapper_1scheme_eval (
    __pyx_self=<optimized out>, __pyx_args=<optimized out>, 
    __pyx_kwds=<optimized out>)
    at /home/linas/src/novamente/src/atomspace-git/build/opencog/cython/opencog/scheme_wrapper.cpp:956
#60 0x000000000049968d in call_function (oparg=<optimized out>, 
    pp_stack=0x7fffffffba70) at ../Python/ceval.c:4020
#61 PyEval_EvalFrameEx (
    f=f@entry=Frame 0xa94120, for file tests/nlp/anaphora/test_anaphora.py, line 31, in setUp (self=<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: 'assertListEqual', <type a...(truncated), throwflag=throwflag@entry=0) at ../Python/ceval.c:2666
#62 0x0000000000499ef2 in fast_function (nk=<optimized out>, 
    na=<optimized out>, n=1, pp_stack=0x7fffffffbbc0, 
    func=<function at remote 0x7ffff6e65a28>) at ../Python/ceval.c:4106
#63 call_function (oparg=<optimized out>, pp_stack=0x7fffffffbbc0)
    at ../Python/ceval.c:4041
#64 PyEval_EvalFrameEx (
    f=f@entry=Frame 0xa93ed0, for file /usr/lib/python2.7/unittest/case.py, line 322, in run (self=<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: 'assertListEqual', <type at...(truncated), throwflag=throwflag@entry=0) at ../Python/ceval.c:2666
#65 0x00000000004a1c9a in PyEval_EvalCodeEx (closure=<optimized out>, 
    defcount=<optimized out>, defs=0x7ffff6e9c128, kwcount=<optimized out>, 
    kws=<optimized out>, argcount=11091664, args=<optimized out>, locals=0x0, 
    globals=<optimized out>, co=<optimized out>) at ../Python/ceval.c:3252
#66 function_call.15337 (func=<optimized out>, arg=<optimized out>, 
    kw=<optimized out>) at ../Objects/funcobject.c:526
#67 0x0000000000505f96 in PyObject_Call (
    func=<function at remote 0x7ffff6e9d230>, arg=<optimized out>, 
    kw=<optimized out>) at ../Objects/abstract.c:2529
#68 0x000000000049b07a in ext_do_call (nk=<optimized out>, na=<optimized out>, 
    flags=<optimized out>, pp_stack=0x7fffffffbe00, 
    func=<function at remote 0x7ffff6e9d230>) at ../Python/ceval.c:4333
#69 PyEval_EvalFrameEx (
    f=f@entry=Frame 0x7ffff6e15960, for file /usr/lib/python2.7/unittest/case.py
, line 395, in __call__ (self=<AnaphoraUnitTester(_testMethodName='test_bfs', _r
esultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa41
50>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(strea
m=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer
=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_
buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[],
 errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, une
xpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfa
st=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at re
mote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x9068
80>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: 'assertListEqual...(truncated), throwflag=throwflag@entry=0) at ../Python/ceval.c:2705
#70 0x00000000004a1c9a in PyEval_EvalCodeEx (closure=<optimized out>, 
    defcount=<optimized out>, defs=0x0, kwcount=<optimized out>, 
    kws=<optimized out>, argcount=-153003680, args=<optimized out>, 
    locals=0x0, globals=<optimized out>, co=<optimized out>)
    at ../Python/ceval.c:3252
#71 function_call.15337 (func=func@entry=<function at remote 0x7ffff6e9d320>, 
    arg=arg@entry=(<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: 'assertListEqual', <type at remote 0x9199c0>: 'assertMultiLineEqual', <type at remote 0x918de0>: 'assertDictEqu...(truncated), kw=kw@entry=0x0) at ../Objects/funcobject.c:526
#72 0x00000000004dfe94 in PyObject_Call (kw=0x0, 
    arg=(<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: 'assertListEqual', <type at remote 0x9199c0>: 'assertMultiLineEqual', <type at remote 0x918de0>: 'assertDictEqu...(truncated), func=<function at remote 0x7ffff6e9d320>) at ../Objects/abstract.c:2529
#73 instancemethod_call.8802 (func=<function at remote 0x7ffff6e9d320>, 
    arg=(<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: 'assertListEqual', <type at remote 0x9199c0>: 'assertMultiLineEqual', <type at remote 0x918de0>: 'assertDictEqu...(truncated), kw=0x0) at ../Objects/classobject.c:2602
#74 0x0000000000505f96 in PyObject_Call (
    func=<instancemethod at remote 0x7ffff6e95500>, arg=<optimized out>, 
    kw=<optimized out>) at ../Objects/abstract.c:2529
#75 0x00000000004dddad in slot_tp_call.25849 (
    self=<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<T
extTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipp
ed=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: 'assertListEqual', <type at remote 0x9199c0>: 'assertMultiLineEqual', <type at remote 0x918de0>: 'assertDictEqua...(truncated), 
    args=(<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>,), kwds=0x0) at ../Objects/typeobject.c:5432
#76 0x0000000000499be5 in PyObject_Call (kw=0x0, 
    arg=(<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>,), 
    func=<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: 'assertListEqual', <type at remote 0x9199c0>: 'assertMultiLineEqual', <type at remote 0x918de0>: 'assertDictEqua...(truncated)) at ../Objects/abstract.c:2529
#77 do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7fffffffc360, 
    func=<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: 'assertListEqual', <type at remote 0x9199c0>: 'assertMultiLineEqual', <type at remote 0x918de0>: 'assertDictEqua...(truncated)) at ../Python/ceval.c:4238
#78 call_function (oparg=<optimized out>, pp_stack=0x7fffffffc360)
    at ../Python/ceval.c:4043
#79 PyEval_EvalFrameEx (
    f=f@entry=Frame 0x7ffff7f73c20, for file /usr/lib/python2.7/unittest/suite.p
y, line 108, in run (self=<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodName
='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remo
te 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_Writeln
Decorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, tes
tsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=
False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expec
tedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: '...(truncated), throwflag=throwflag@entry=0) at ../Python/ceval.c:2666
#80 0x00000000004a1c9a in PyEval_EvalCodeEx (closure=<optimized out>, 
    defcount=<optimized out>, defs=0x7ffff6ea2568, kwcount=<optimized out>, 
    kws=<optimized out>, argcount=-134792160, args=<optimized out>, 
    locals=0x0, globals=<optimized out>, co=<optimized out>)
    at ../Python/ceval.c:3252
#81 function_call.15337 (func=<optimized out>, arg=<optimized out>, 
    kw=<optimized out>) at ../Objects/funcobject.c:526
#82 0x0000000000505f96 in PyObject_Call (
    func=<function at remote 0x7ffff6ea35f0>, arg=<optimized out>, 
    kw=<optimized out>) at ../Objects/abstract.c:2529
#83 0x000000000049b07a in ext_do_call (nk=<optimized out>, na=<optimized out>, 
    flags=<optimized out>, pp_stack=0x7fffffffc5a0, 
    func=<function at remote 0x7ffff6ea35f0>) at ../Python/ceval.c:4333
#84 PyEval_EvalFrameEx (
    f=f@entry=Frame 0x7ffff6e155c0, for file /usr/lib/python2.7/unittest/suite.py, line 70, in __call__ (self=<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80...(truncated), throwflag=throwflag@entry=0) at ../Python/ceval.c:2705
#85 0x00000000004a1c9a in PyEval_EvalCodeEx (closure=<optimized out>, 
    defcount=<optimized out>, defs=0x0, kwcount=<optimized out>, 
    kws=<optimized out>, argcount=-153004608, args=<optimized out>, 
    locals=0x0, globals=<optimized out>, co=<optimized out>)
    at ../Python/ceval.c:3252
#86 function_call.15337 (func=func@entry=<function at remote 0x7ffff6ea3500>, 
    arg=arg@entry=(<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: 'assertListEqual', <type at remote 0x9199c0>: 'assertMultiLineEqual', <type at remote 0x918d...(truncated), kw=kw@entry=0x0) at ../Objects/funcobject.c:526
#87 0x00000000004dfe94 in PyObject_Call (kw=0x0, 
    arg=(<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodName='test_bfs', _res
ultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150
>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=
<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=N
one, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_bu
ffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], e
rrors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexp
ectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: 'assertListEqual', <type at remote 0x9199c0>: 'assertMultiLineEqual', <type at remote 0x918d...(truncated), func=<function at remote 0x7ffff6ea3500>) at ../Objects/abstract.c:2529
#88 instancemethod_call.8802 (func=<function at remote 0x7ffff6ea3500>, 
    arg=(<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: 'assertListEqual', <type at remote 0x9199c0>: 'assertMultiLineEqual', <type at remote 0x918d...(truncated), kw=0x0) at ../Objects/classobject.c:2602
#89 0x0000000000505f96 in PyObject_Call (
    func=<instancemethod at remote 0x7ffff6e954b0>, arg=<optimized out>, 
    kw=<optimized out>) at ../Objects/abstract.c:2529
#90 0x00000000004dddad in slot_tp_call.25849 (
    self=<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: 'assertListEqual', <type at remote 0x9199c0>: 'assertMultiLineEqual', <type at remote 0x918de...(truncated), 
    args=(<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>,), kwds=0x0) at ../Objects/typeobject.c:5432
#91 0x0000000000499be5 in PyObject_Call (kw=0x0, 
    arg=(<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=
True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at
 remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _or
iginal_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=Non
e, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[]
, descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuc
cesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) a
t remote 0x7ffff6e8ec10>,), 
    func=<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodName='test_bfs', _res
ultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150
>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=
<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=N
one, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_bu
ffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], e
rrors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: 'assertListEqual', <type at remote 0x9199c0>: 'assertMultiLineEqual', <type at remote 0x918de...(truncated)) at ../Objects/abstract.c:2529
#92 do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7fffffffcb00, 
    func=<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: 'assertListEqual', <type at remote 0x9199c0>: 'assertMultiLineEqual', <type at remote 0x918de...(truncated)) at ../Python/ceval.c:4238
#93 call_function (oparg=<optimized out>, pp_stack=0x7fffffffcb00)
    at ../Python/ceval.c:4043
#94 PyEval_EvalFrameEx (
    f=f@entry=Frame 0x7ffff7f73830, for file /usr/lib/python2.7/unittest/suite.py, line 108, in run (self=<TestSuite(_tests=[<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at ...(truncated), throwflag=throwflag@entry=0) at ../Python/ceval.c:2666
#95 0x00000000004a1c9a in PyEval_EvalCodeEx (closure=<optimized out>, 
    defcount=<optimized out>, defs=0x7ffff6ea2568, kwcount=<optimized out>, 
    kws=<optimized out>, argcount=-134793168, args=<optimized out>, 
    locals=0x0, globals=<optimized out>, co=<optimized out>)
    at ../Python/ceval.c:3252
#96 function_call.15337 (func=<optimized out>, arg=<optimized out>, 
    kw=<optimized out>) at ../Objects/funcobject.c:526
#97 0x0000000000505f96 in PyObject_Call (
    func=<function at remote 0x7ffff6ea35f0>, arg=<optimized out>, 
    kw=<optimized out>) at ../Objects/abstract.c:2529
#98 0x000000000049b07a in ext_do_call (nk=<optimized out>, na=<optimized out>, 
    flags=<optimized out>, pp_stack=0x7fffffffcd40, 
    func=<function at remote 0x7ffff6ea35f0>) at ../Python/ceval.c:4333
#99 PyEval_EvalFrameEx (
    f=f@entry=Frame 0x7ffff6e15220, for file /usr/lib/python2.7/unittest/suite.p
y, line 70, in __call__ (self=<TestSuite(_tests=[<TestSuite(_tests=[<AnaphoraUni
tTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_origin
al_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type...(truncated), throwflag=throwflag@entry=0) at ../Python/ceval.c:2705
#100 0x00000000004a1c9a in PyEval_EvalCodeEx (closure=<optimized out>, 
    defcount=<optimized out>, defs=0x0, kwcount=<optimized out>, 
    kws=<optimized out>, argcount=-153005536, args=<optimized out>, 
    locals=0x0, globals=<optimized out>, co=<optimized out>)
    at ../Python/ceval.c:3252
#101 function_call.15337 (func=func@entry=<function at remote 0x7ffff6ea3500>, 
    arg=arg@entry=(<TestSuite(_tests=[<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: 'assertListEqual', <type at remote 0x9199c0>: 'assertMultiLineEqual', <ty...(truncated), kw=kw@entry=0x0) at ../Objects/funcobject.c:526
#102 0x00000000004dfe94 in PyObject_Call (kw=0x0, 
    arg=(<TestSuite(_tests=[<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: 'assertListEqual', <type at remote 0x9199c0>: 'assertMultiLineEqual', <ty...(truncated), func=<function at remote 0x7ffff6ea3500>) at ../Objects/abstract.c:2529
#103 instancemethod_call.8802 (func=<function at remote 0x7ffff6ea3500>, 
    arg=(<TestSuite(_tests=[<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: 'assertListEqual', <type at remote 0x9199c0>: 'assertMultiLineEqual', <ty...(truncated), kw=0x0) at ../Objects/classobject.c:2602
#104 0x0000000000505f96 in PyObject_Call (
    func=<instancemethod at remote 0x7ffff7ec3190>, arg=<optimized out>, 
    kw=<optimized out>) at ../Objects/abstract.c:2529
#105 0x00000000004dddad in slot_tp_call.25849 (
    self=<TestSuite(_tests=[<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodNa
me='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at re
mote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: 'assertListEqual', <type at remote 0x9199c0>: 'assertMultiLineEqual', <typ...(truncated), 
    args=(<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>,), kwds=0x0) at ../Objects/typeobject.c:5432
#106 0x0000000000499be5 in PyObject_Call (kw=0x0, 
    arg=(<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>,), 
    func=<TestSuite(_tests=[<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: 'assertListEqual', <type at remote 0x9199c0>: 'assertMultiLineEqual', <typ...(truncated)) at ../Objects/abstract.c:2529
#107 do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7fffffffd2a0, 
    func=<TestSuite(_tests=[<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assertSetEqual', <type at remote 0x9192c0>: 'assertTupleEqual', <type at remote 0x906ee0>: 'assertSetEqual', <type at remote 0x918f80>: 'assertListEqual', <type at remote 0x9199c0>: 'assertMultiLineEqual', <typ...(truncated)) at ../Python/ceval.c:4238
#108 call_function (oparg=<optimized out>, pp_stack=0x7fffffffd2a0)
    at ../Python/ceval.c:4043
#109 PyEval_EvalFrameEx (
    f=f@entry=Frame 0xa47910, for file /usr/lib/python2.7/unittest/runner.py, li
ne 151, in run (self=<TextTestRunner(buffer=None, verbosity=1, descriptions=True
, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x
7ffff6e8ebd0>, failfast=None) at remote 0x7ffff6e8eb90>, test=<TestSuite(_tests=
[<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDo
Cleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=
True, skipped=[], _mirrorOutput=False, stream=<...>, testsRun=1, buffer=None, _o
riginal_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'a...(truncated), throwflag=throwflag@entry=0) at ../Python/ceval.c:2666
#110 0x0000000000499ef2 in fast_function (nk=<optimized out>, 
    na=<optimized out>, n=2, pp_stack=0x7fffffffd3f0, 
    func=<function at remote 0x7ffff6ec4398>) at ../Python/ceval.c:4106
#111 call_function (oparg=<optimized out>, pp_stack=0x7fffffffd3f0)
    at ../Python/ceval.c:4041
#112 PyEval_EvalFrameEx (
    f=f@entry=Frame 0x7ffff6eca420, for file /usr/lib/python2.7/unittest/main.py, line 232, in runTests (self=<TestProgram(buffer=None, verbosity=1, progName='test_anaphora.py', module=<module at remote 0x7ffff7f43d70>, testRunner=<type at remote 0xa53040>, defaultTest=None, catchbreak=None, testNames=None, test=<TestSuite(_tests=[<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace a...(truncated), throwflag=throwflag@entry=0) at ../Python/ceval.c:2666
#113 0x0000000000499ef2 in fast_function (nk=<optimized out>, 
    na=<optimized out>, n=1, pp_stack=0x7fffffffd540, 
    func=<function at remote 0x7ffff6ec4668>) at ../Python/ceval.c:4106
#114 call_function (oparg=<optimized out>, pp_stack=0x7fffffffd540)
    at ../Python/ceval.c:4041
#115 PyEval_EvalFrameEx (
    f=f@entry=Frame 0xa88700, for file /usr/lib/python2.7/unittest/main.py, line 95, in __init__ (self=<TestProgram(buffer=None, verbosity=1, progName='test_anaphora.py', module=<module at remote 0x7ffff7f43d70>, testRunner=<type at remote 0xa53040>, defaultTest=None, catchbreak=None, testNames=None, test=<TestSuite(_tests=[<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remot...(truncated), throwflag=throwflag@entry=0) at ../Python/ceval.c:2666
#116 0x00000000004a1c9a in PyEval_EvalCodeEx (closure=<optimized out>, 
    defcount=<optimized out>, defs=0x7ffff6eb1420, kwcount=<optimized out>, 
    kws=<optimized out>, argcount=11044608, args=<optimized out>, locals=0x0, 
    globals=<optimized out>, co=<optimized out>) at ../Python/ceval.c:3252
#117 function_call.15337 (func=func@entry=<function at remote 0x7ffff6ec4410>, 
    arg=arg@entry=(<TestProgram(buffer=None, verbosity=1, progName='test_anaphor
a.py', module=<module at remote 0x7ffff7f43d70>, testRunner=<type at remote 0xa5
3040>, defaultTest=None, catchbreak=None, testNames=None, test=<TestSuite(_tests
=[<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForD
oCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots
=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file a
t remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _o
riginal_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=No
ne, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[
], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSu
ccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) 
at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7f
fff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'asser...(truncated), kw=kw@entry=0x0) at ../Objects/funcobject.c:526
#118 0x00000000004dfe94 in PyObject_Call (kw=0x0, 
    arg=(<TestProgram(buffer=None, verbosity=1, progName='test_anaphora.py', module=<module at remote 0x7ffff7f43d70>, testRunner=<type at remote 0xa53040>, defaultTest=None, catchbreak=None, testNames=None, test=<TestSuite(_tests=[<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'asser...(truncated), func=<function at remote 0x7ffff6ec4410>) at ../Objects/abstract.c:2529
#119 instancemethod_call.8802 (func=<function at remote 0x7ffff6ec4410>, 
    arg=(<TestProgram(buffer=None, verbosity=1, progName='test_anaphora.py', module=<module at remote 0x7ffff7f43d70>, testRunner=<type at remote 0xa53040>, defaultTest=None, catchbreak=None, testNames=None, test=<TestSuite(_tests=[<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'asser...(truncated), kw=0x0) at ../Objects/classobject.c:2602
#120 0x0000000000505f96 in PyObject_Call (
    func=<instancemethod at remote 0x7ffff7f2cc80>, arg=<optimized out>, 
    kw=<optimized out>) at ../Objects/abstract.c:2529
#121 0x00000000004de41a in slot_tp_init.25831 (
    self=self@entry=<TestProgram(buffer=None, verbosity=1, progName='test_anaphora.py', module=<module at remote 0x7ffff7f43d70>, testRunner=<type at remote 0xa53040>, defaultTest=None, catchbreak=None, testNames=None, test=<TestSuite(_tests=[<TestSuite(_tests=[<AnaphoraUnitTester(_testMethodName='test_bfs', _resultForDoCleanups=<TextTestResult(_original_stdout=<file at remote 0x7ffff7fa4150>, dots=True, skipped=[], _mirrorOutput=False, stream=<_WritelnDecorator(stream=<file at remote 0x7ffff7fa41e0>) at remote 0x7ffff6e8ebd0>, testsRun=1, buffer=None, _original_stderr=<file at remote 0x7ffff7fa41e0>, showAll=False, _stdout_buffer=None, _stderr_buffer=None, _moduleSetUpFailed=False, expectedFailures=[], errors=[], descriptions=True, _previousTestClass=<type at remote 0xa52930>, unexpectedSuccesses=[], failures=[], _testRunEntered=True, shouldStop=False, failfast=None) at remote 0x7ffff6e8ec10>, atomspace=<opencog.atomspace.AtomSpace at remote 0x7ffff7f56210>, _cleanups=[], _type_equality_funcs={<type at remote 0x906880>: 'assert...(truncated), args=args@entry=(), kwds=kwds@entry=0x0)
    at ../Objects/typeobject.c:5692
#122 0x00000000005039eb in type_call.25495 (type=<optimized out>, args=(), 
    kwds=0x0) at ../Objects/typeobject.c:745
#123 0x0000000000499be5 in PyObject_Call (kw=0x0, arg=(), 
    func=<type at remote 0xa44510>) at ../Objects/abstract.c:2529
#124 do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7fffffffdad0, 
    func=<type at remote 0xa44510>) at ../Python/ceval.c:4238
#125 call_function (oparg=<optimized out>, pp_stack=0x7fffffffdad0)
    at ../Python/ceval.c:4043
#126 PyEval_EvalFrameEx (
    f=f@entry=Frame 0x7ffff7f683c0, for file tests/nlp/anaphora/test_anaphora.py, line 451, in <module> (), throwflag=throwflag@entry=0)
    at ../Python/ceval.c:2666
#127 0x00000000004a1634 in PyEval_EvalCodeEx (closure=0x0, defcount=0, 
    defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, 
    locals=<unknown at remote 0x7ffff7f68538>, 
    globals=<unknown at remote 0x9ef830>, co=0x7ffff7e86030)
    at ../Python/ceval.c:3252
#128 PyEval_EvalCode (locals=<unknown at remote 0x7ffff7f68538>, 
    globals=<unknown at remote 0x9ef830>, co=0x7ffff7e86030)
    at ../Python/ceval.c:667
#129 run_mod.42576 (mod=mod@entry=0xa18730, 
    filename=filename@entry=0x7fffffffe147 "tests/nlp/anaphora/test_anaphora.py", 
    globals=globals@entry={'scheme_eval_h': <built-in function scheme_eval_h>, 'TruthValue': <type at remote 0x7ffff6dcf960>, 'AtomSpace': <type at remote 0x7ffff6dce9a0>, 'load_scm': <built-in function load_scm>, 'get_type': <built-in function get_type>, 'HobbsAgent': <type at remote 0xa87810>, '__init__': <method-wrapper at remote 0x7ffff6ece950>, 'Handle': <type at remote 0x7ffff6dcf620>, '__package__': None, 'TestCase': <type at remote 0xa3ef70>, 'Atom': <type at remote 0x7ffff6dcf060>, 'unittest': <module at remote 0x7ffff7e7bde0>, '__builtins__': <module at remote 0x7ffff7f82b08>, '__file__': 'tests/nlp/anaphora/test_anaphora.py', '__author__': 'Hujie', 'get_type_name': <built-in function get_type_name>, '__name__': '__main__', 'types': <type at remote 0xa7bca0>, 'AnaphoraUnitTester': <type at remote 0xa52930>, 'scheme_eval': <built-in function scheme_eval>, '__DEV_MODE__': False, '__doc__': None, 'is_a': <built-in function is_a>, '__VERBOSE__': False}, 
    locals=locals@entry={'scheme_eval_h': <built-in function scheme_eval_h>, 'TruthValue': <type at remote 0x7ffff6dcf960>, 'AtomSpace': <type at remote 0x7ffff6dce9a0>, 'load_scm': <built-in function load_scm>, 'get_type': <built-in function get_type>, 'HobbsAgent': <type at remote 0xa87810>, '__init__': <method-wrapper at remote 0x7ffff6ece950>, 'Handle': <type at remote 0x7ffff6dcf620>, '__package__': None, 'TestCase': <type at remote 0xa3ef70>, 'Atom': <type at remote 0x7ffff6dcf060>, 'unittest': <module at remote 0x7ffff7e7bde0>, '__builtins__': <module at remote 0x7ffff7f82b08>, '__file__': 'tests/nlp/anaphora/test_anaphora.py', '__author__': 'Hujie', 'get_type_name': <built-in function get_type_name>, '__name__': '__main__', 'types': <type at remote 0xa7bca0>, 'AnaphoraUnitTester': <type at remote 0xa52930>, 'scheme_eval': <built-in function scheme_eval>, '__DEV_MODE__': False, '__doc__': None, 'is_a': <built-in function is_a>, '__VERBOSE__': False}, flags=flags@entry=0x7fffffffdc80, arena=arena@entry=0x95e7d0)
    at ../Python/pythonrun.c:1370
#130 0x000000000044e4a5 in PyRun_FileExFlags (fp=fp@entry=0x9ef830, 
    filename=filename@entry=0x7fffffffe147 "tests/nlp/anaphora/test_anaphora.py", start=start@entry=257, 
    globals=globals@entry={'scheme_eval_h': <built-in function scheme_eval_h>, 'TruthValue': <type at remote 0x7ffff6dcf960>, 'AtomSpace': <type at remote 0x7ffff6dce9a0>, 'load_scm': <built-in function load_scm>, 'get_type': <built-in function get_type>, 'HobbsAgent': <type at remote 0xa87810>, '__init__': <method-wrapper at remote 0x7ffff6ece950>, 'Handle': <type at remote 0x7ffff6dcf620>, '__package__': None, 'TestCase': <type at remote 0xa3ef70>, 'Atom': <type at remote 0x7ffff6dcf060>, 'unittest': <module at remote 0x7ffff7e7bde0>, '__builtins__': <module at remote 0x7ffff7f82b08>, '__file__': 'tests/nlp/anaphora/test_anaphora.py', '__author__': 'Hujie', 'get_type_name': <built-in function get_type_name>, '__name__': '__main__', 'types': <type at remote 0xa7bca0>, 'AnaphoraUnitTester': <type at remote 0xa52930>, 'scheme_eval': <built-in function scheme_eval>, '__DEV_MODE__': False, '__doc__': None, 'is_a': <built-in function is_a>, '__VERBOSE__': False}, 
    locals=locals@entry={'scheme_eval_h': <built-in function scheme_eval_h>, 'TruthValue': <type at remote 0x7ffff6dcf960>, 'AtomSpace': <type at remote 0x7ffff6dce9a0>, 'load_scm': <built-in function load_scm>, 'get_type': <built-in function get_type>, 'HobbsAgent': <type at remote 0xa87810>, '__init__': <method-wrapper at remote 0x7ffff6ece950>, 'Handle': <type at remote 0x7ffff6dcf620>, '__package__': None, 'TestCase': <type at remote 0xa3ef70>, 'Atom': <type at remote 0x7ffff6dcf060>, 'unittest': <module at remote 0x7ffff7e7bde0>, '__builtins__': <module at remote 0x7ffff7f82b08>, '__file__': 'tests/nlp/anaphora/test_anaphora.py', '__author__': 'Hujie', 'get_type_name': <built-in function get_type_name>, '__name__': '__main__', 'types': <type at remote 0xa7bca0>, 'AnaphoraUnitTester': <type at remote 0xa52930>, 'scheme_eval': <built-in function scheme_eval>, '__DEV_MODE__': False, '__doc__': None, 'is_a': <built-in function is_a>, '__VERBOSE__': False}, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffdc80)
    at ../Python/pythonrun.c:1356
#131 0x000000000044ec9f in PyRun_SimpleFileExFlags (fp=fp@entry=0x9ef830, 
    filename=<optimized out>, 
    filename@entry=0x7fffffffe147 "tests/nlp/anaphora/test_anaphora.py", 
    closeit=closeit@entry=1, flags=flags@entry=0x7fffffffdc80)
    at ../Python/pythonrun.c:948
#132 0x000000000044ed9b in PyRun_AnyFileExFlags (fp=fp@entry=0x9ef830, 
    filename=filename@entry=0x7fffffffe147 "tests/nlp/anaphora/test_anaphora.py", closeit=closeit@entry=1, flags=flags@entry=0x7fffffffdc80)
    at ../Python/pythonrun.c:752
#133 0x000000000044f904 in Py_Main (argc=<optimized out>, argv=0x7fffffffde38)
    at ../Modules/main.c:640
#134 0x00007ffff7818ec5 in __libc_start_main (main=0x44f9c2 <main>, argc=2, 
    argv=0x7fffffffde38, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7fffffffde28) at libc-start.c:287
#135 0x0000000000578c4e in _start ()

from bdwgc.

linas avatar linas commented on July 17, 2024

Oh, above was with git tag gc7_4_2 -- I can retry again with master, if you wish.

from bdwgc.

linas avatar linas commented on July 17, 2024

Stack is almost identical for master HEAD: here is the top bits only:

#0  0x00007ffff782dcc9 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff78310d8 in __GI_abort () at abort.c:89
#2  0x00007ffff14a4458 in GC_reclaim_generic (hbp=hbp@entry=0xd6e000, 
    hhdr=0xb307f0, sz=800, init=0, list=0x0, 
    count=count@entry=0x7ffff16f7a28 <GC_bytes_found>) at reclaim.c:314
#3  0x00007ffff14a44c0 in GC_reclaim_small_nonempty_block (
    hbp=hbp@entry=0xd6e000, report_if_found=report_if_found@entry=0)
    at reclaim.c:340
#4  0x00007ffff14a4a41 in GC_continue_reclaim (sz=sz@entry=50, 
    kind=kind@entry=5) at reclaim.c:680
#5  0x00007ffff1493444 in GC_allocobj (gran=gran@entry=50, kind=kind@entry=5)
    at alloc.c:1423
#6  0x00007ffff1499cd9 in GC_generic_malloc_inner (lb=lb@entry=744, 
    k=k@entry=5) at malloc.c:140
#7  0x00007ffff1499f52 in GC_generic_malloc (lb=744, k=5) at malloc.c:189
#8  0x00007ffff18b043e in allocate_entries (size=size@entry=31, 
    kind=kind@entry=SCM_WEAK_TABLE_KIND_KEY) at ../../libguile/weak-table.c:362
#9  0x00007ffff18b04ce in make_weak_table (k=<optimized out>, 
    kind=kind@entry=SCM_WEAK_TABLE_KIND_KEY) at ../../libguile/weak-table.c:777
#10 0x00007ffff18b05cb in scm_c_make_weak_table (kind=SCM_WEAK_TABLE_KIND_KEY, 
    k=<optimized out>) at ../../libguile/weak-table.c:830

from bdwgc.

ivmai avatar ivmai commented on July 17, 2024

What's the value of GC_mark_state when GC_reclaim_generic is invoked in the above stack trace? If it is non-zero then I know what's wrong (and basically how to fix it).

from bdwgc.

ivmai avatar ivmai commented on July 17, 2024

> Given today's large machines, perhaps its time for a --enable-super-large-config?
Sorry, I don't have opinion about it.

from bdwgc.

hboehm avatar hboehm commented on July 17, 2024

from bdwgc.

linas avatar linas commented on July 17, 2024

FYI, here's my "real world" setup.

My current production machine has 96GB of RAM, I plan an upgrade to 192 or 256 GB later this year. Its got 12 cores, I expect 24 cores this or next year. I'm running guile-2.1.5 from the latest git on it, and it currently reports:

(gc-stats)
((gc-time-taken . 6416202793258) (heap-size . 14162456576) (heap-free-size . 4668755968) (heap-total-allocated . 808354430640) (heap-allocated-since-gc . 4573882000) (protected-objects . 340) (gc-times . 1034))

i.e. 14GB of ram that bdwgc is currently managing. This is somewhat un-neccessary and excessive, the app is ... well, its not exactly a memleak, but it is splurging in an inelegant way, I've tried hacking to make it use less RAM but its a hard problem.

Anyway, right now, guile seems to be running fine in this configuration -- its stable. Its had a few threading and memleak bugs -- besides the above, the app uses dozens of threads or more. By contrast, when I reported this bug a year ago, things were not so stable. It is possible or likely that since guile is now behaving nicely, its not triggering code paths that make bdwgc unhappy.

BTW: the above system is running against the default bdwgc packages with ubuntu xenial 16.04 -- it seems to be this:

||/ Name              Version       Architecture  Description
+++-=================-=============-=============-=======================================
ii  libgc1c2:amd64    1:7.4.2-7.3   amd64         conservative garbage collector for C an

from bdwgc.

ivmai avatar ivmai commented on July 17, 2024

@linas

  1. Is the assertion still violated? (on master head)
  2. What is the value of GC_obj_kinds[5]? (as I understand SCM_WEAK_TABLE_KIND_KEY==5)

from bdwgc.

ivmai avatar ivmai commented on July 17, 2024

The assertion in GC_reclaim_generic looks correct. I suspect incorrect parameters are passed to GC_new_kind.

from bdwgc.

linas avatar linas commented on July 17, 2024

I'm travelling and can't (easily) test for 3-4 weeks. Reproducing this will require me to set up a container with 2-year-old versions of the system. I'll try ...

Meanwhile I'll start running bdwgc head on the semi-production machines ....

from bdwgc.

ivmai avatar ivmai commented on July 17, 2024

Good,
I'm first interested in the parameters passed to GC_new_kind[_inner] from your client code. In particular, if clear==false then adjust should be false and descr should be 0. Could you look at the source code where SCM_WEAK_TABLE_KIND_KEY created please? No hurry, of course.

from bdwgc.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.