Python-gmpy should not call mp_set_memory_functions at module load time about gmpy HOT 10 CLOSED

Depending on the version of gmpy, there are two different strategies for memory 
management. In older versions, the only benefit for the custom allocations was 
an increase of the minimum memory size which decreases the number of 
reallocations. In later versions, Python's internal memory manager is used. The 
PyMem_ versions are slightly faster. Which version of gmpy are you using?

In either case, changing the memory allocation functions will impact the entire 
GMP library so I agree that it shouldn't be done. I was planning on a bugfix 
release soon so I'll look at making the standard memory allocation functions 
the default.

Mandriva has python-gmpy 1.14 packaged.

I believe it should work if python-gmpy did use malloc/realloc/free,
as in older versions, but should defeat whatever is fixed in
sagemath by wrapping the malloc/realloc/free with the sig_off/sig_on
interfaces.

Right now, if gmpy is installed, it does not matter is MPMATH_NOGMPY
is set, and it calls mp_set_memory_functions from the gmpy shared
object, and afaik PyMem_* memory pointers cannot be used with standard
malloc/realloc/free.

If mpmath actually use gmpy in sagemath, I think there would also be
issues with sage and gmpy mp*_t object caches.

Note that actually it somehow works for most things, save for a few
very weird corruptions/crashes, and, if not setting MALLOC_CHECK_=1
it does not even print any warnings from glibc.

gmpy 1.14 uses the PyMem_ approach. Older versions just changed the minimum 
memory size and then called the libc functions. I'll change 1.15 to offer three 
options for memory management: do nothing, increase minimum memory size (like 
older versions), or use PyMem. I'll let you know when I have a test version 
available.

I've posted a test build in the Download section. I marked it deprecated since 
I haven't tested it on anything other 64-bit Linux. The link is:

http://code.google.com/p/gmpy/downloads/detail?name=gmpy-1.15-test.zip

Please give it a try and let me know if it solves the issues with Sage. It 
fixes a couple of other bugs, too.

It does not print any glibc messages besides my /usr/bin/sage
script sets MALLOC_CHECK_=1, and should be due to _PyInitGMP()
not calling mp_set_memory_functions().

But something is still happening, because the mpmath and
gmpy interaction should have been to not use gmpy, but the
sage cython code to handle integers, as the sage code does
several "short circuits" directly updating mpz_t fields,
calculating number of mp_limb_t, etc before some operations,
like converting to/from pari, ntl, pylong, etc numbers.

I mean, this:

$ GMPY_DEBUG=99 MPMATH_NOGMPY=1 sage -t -force_lib 
"devel/sage/sage/rings/integer.pyx" >& log.txt

and in log.txt I see

sage -t -force_lib "devel/sage/sage/rings/integer.pyx"      
�[?1034hinitgmpy() called...
Entering set_pympzcache
Entering set_pympqcache
gmpy_module at 0x7f6c104922b8
gmpy_module imported copy_reg OK
Pygmpy_mpz() called...
Entering Pympz_new
Pympz_new is creating a new object
Initing new not in zcache
anynum2Pympz(0x228c9b0)->0x7f6c112f2530
Pygmpy_mpz: created mpz = 0
Pympz_dealloc: 0x7f6c112f2530
Pygmpy_mpq() called...
[...]
Not placing in full zcache(100/100)
Pympz_dealloc: 0x7f6c0f7417b0
Not placing in full zcache(100/100)
Pympz_dealloc: 0x7f6c0f741790
Not placing in full zcache(100/100)
For whitespace errors, see the file /home/pcpa/.sage//tmp/integer_12676.py
     [6.6 s]

  The sagemath doctest errors I see are or should be
due to using python 2.7 and a few other components
with different versions in my rpm packages but should
be worth to paste here. Updating the python-gmpy rpm
package to use gmp-1.5-test.zip as source (setting
GMPY_DEBUG=1 besides also setting MPMATH_NOGMPY is
enough to have more than 1k lines output).

x86_64:
$ sage -t -force_lib "devel/sage/sage/rings/integer.pyx" 
sage -t -force_lib "devel/sage/sage/rings/integer.pyx"      
**********************************************************************
File "/usr/share/sage/devel/sage/sage/rings/integer.pyx", line 4653:
    sage: numpy.array(2**40).dtype
Expected:
    dtype('int64')
Got:
    dtype('object')
**********************************************************************
File "/usr/share/sage/devel/sage/sage/rings/integer.pyx", line 3046:
    sage: n = -920390823904823094890238490238484; n.__hash__()
Expected:
    6874330978542788722   
Got:
    -2623069716
**********************************************************************
File "/usr/share/sage/devel/sage/sage/rings/integer.pyx", line 3061:
    sage: hash(n)
Expected:
    -9223372036854767616      
Got:
    8192
**********************************************************************
File "/usr/share/sage/devel/sage/sage/rings/integer.pyx", line 3064:
    sage: hash(n) == hash(int(n))
Expected:
    True
Got:
    False
**********************************************************************
2 items had failures:
   1 of   9 in __main__.example_113
   3 of  13 in __main__.example_63
***Test Failed*** 4 failures.
For whitespace errors, see the file /home/pcpa/.sage//tmp/integer_12790.py
         [6.6 s]

----------------------------------------------------------------------
The following tests failed:


        sage -t -force_lib "devel/sage/sage/rings/integer.pyx"
Total time for all tests: 6.6 seconds


i586 (same computer, "linux32 chroot ..."):
$ sage -t -force_lib "devel/sage/sage/rings/integer.pyx" 
sage -t -force_lib "devel/sage/sage/rings/integer.pyx"      
**********************************************************************
File "/usr/share/sage/devel/sage/sage/rings/integer.pyx", line 4653:
    sage: numpy.array(2**40).dtype
Expected:
    dtype('int64')
Got:
    dtype('object')
**********************************************************************
1 items had failures:
   1 of   9 in __main__.example_113
***Test Failed*** 1 failures.
For whitespace errors, see the file /home/pcpa/.sage//tmp/integer_12824.py
         [13.5 s]

----------------------------------------------------------------------
The following tests failed:


        sage -t -force_lib "devel/sage/sage/rings/integer.pyx"
Total time for all tests: 13.5 seconds

  I do not know what else may be loading gmpy if
it is installed...

A couple of ideas. In gmpy.c, change the line

} options = { 0, 0, 5, 100, 128, 0 };

to

} options = { 0, 0, 5, 0, 0, 0 };

That will enable internal caching of deleted objects. It shouldn't be an issue, 
but it might.

The hash() error message imply something is working with 32-bits and and 
something else is working in 64-bit mode. Could there be an old version of gmpy 
used by something else? 

Try editing the mpmath "import gmpy" code to something like "import gmpyXXX" to 
see if that stops the importing of gmpy.

After looking at grep -r 'gmpy' /usr/lib{,64}/python2.7
output, I got it to not load gmpy with these system wide
changes:

--- /usr/lib/python2.7/site-packages/sympy/core/numbers.py~     2011-11-08 
02:28:15.000000000 -0200
+++ /usr/lib/python2.7/site-packages/sympy/core/numbers.py      2011-11-11 
13:42:23.352591670 -0200
@@ -1962,7 +1962,7 @@ except ImportError:
     pass

 try:
-    import gmpy
+    import gmpyXX

     def sympify_mpz(x):
         return Integer(long(x))
--- /usr/lib/python2.7/site-packages/sympy/polys/domains/groundtypes.py~       
2011-11-08 02:28:15.000000000 -0200
+++ /usr/lib/python2.7/site-packages/sympy/polys/domains/groundtypes.py 
2011-11-11 13:44:15.954119582 -0200
@@ -2,16 +2,16 @@

 from sympy.external import import_module

-HAS_GMPY = True
+HAS_GMPY = False

 # Versions of gmpy prior to 1.03 do not work correctly with int(largempz)
 # For example, int(gmpy.mpz(2**256)) would raise OverflowError.
 # See issue 1881.

-gmpy = import_module('gmpy', min_module_version='1.03',
-    module_version_attr='version', module_version_attr_call_args=())
+#gmpy = import_module('gmpy', min_module_version='1.03',
+#    module_version_attr='version', module_version_attr_call_args=())

-HAS_GMPY = bool(gmpy)
+#HAS_GMPY = bool(gmpy)

 from __builtin__ import (
     int     as PythonIntegerType,

so that running:

$ GMPY_DEBUG=99 MPMATH_NOGMPY=1 sage -t -force_lib 
"devel/sage/sage/rings/integer.pyx"

does not print any debug from gmpy.

Note that sympy in sagemath is sympy-0.6.4 while Mandriva have

$ rpm -q python-sympy
python-sympy-0.7.1-2-mdv2012.0.noarch

The sympy in the sagemath spkg does not use either gmpy or
mpmath, so, these issues are probably not known to upstream.

Offtopic: I recently imported a gentoo patch to force sympy to
use system mpmath instead of an internal copy, after keeping
rediff'ing for quite some time a patch to force sagemath to
use sympy's mpmath.
Ontopic: Using the same approach I used for mpmath for some
time where I added a "Conflicts: python-mpmath", now I added
a "Conflicts: python-gmpy" to the sagemath rpm spec file. I
wish it would not be required...

The hash errors still happen when not loading gmpy and should
be a bug in sagemath with python 2.7 (upstream sage uses
python 2.6) -> http://trac.sagemath.org/sage_trac/ticket/11986

For extra information, with the patches above, it not setting
MPMATH_NOGMPY=1, and still toying with install/unsintall of
the rpm I generated with gmpu-1.5-test.zip:

$ GMPY_DEBUG=99  sage -t -force_lib "devel/sage/sage/rings/integer.pyx"
sage -t -force_lib "devel/sage/sage/rings/integer.pyx"      
initgmpy() called...
Entering set_pympzcache
Entering set_pympqcache
gmpy_module at 0x7f53ae42f638
gmpy_module imported copy_reg OK
Pygmpy_mpz() called...
Entering Pympz_new
Pympz_new is creating a new object
Initing new not in zcache
anynum2Pympz(0x22b49b0)->0x7f53b53e3db0
Pygmpy_mpz: created mpz = 0
Pympz_dealloc: 0x7f53b53e3db0
Pygmpy_mpq() called...
Entering Pympq_new
Pympq_new is creating a new object
Initing new not in qcache
Initing new not in qcache, done
anynum2Pympq(0x22b49b0)->0x7f53ae435b70
Pygmpy_mpq: created mpq = 0
Pympq_dealloc: 0x7f53ae435b70
Pygmpy_mpf() called...
prec 64 size 0 toclear -2 carry 0
anynum2Pympf(0x22b49b0,0)->0x7f53ae435ba0 (53)
Pygmpy_mpf: created mpf = 0.e0 bits=53 (0)
Pympf_dealloc: 0x7f53ae435ba0
gmpy_module enable pickle OK
**********************************************************************
File "/usr/share/sage/devel/sage/sage/rings/integer.pyx", line 4653:
    sage: numpy.array(2**40).dtype
Expected:
    dtype('int64')
Got:
    dtype('object')
**********************************************************************
File "/usr/share/sage/devel/sage/sage/rings/integer.pyx", line 3046:
    sage: n = -920390823904823094890238490238484; n.__hash__()
Expected:
    6874330978542788722   
Got:
    -2623069716
**********************************************************************
File "/usr/share/sage/devel/sage/sage/rings/integer.pyx", line 3061:
    sage: hash(n)
Expected:
    -9223372036854767616      
Got:
    8192
**********************************************************************
File "/usr/share/sage/devel/sage/sage/rings/integer.pyx", line 3064:
    sage: hash(n) == hash(int(n))
Expected:
    True
Got:
    False
**********************************************************************
2 items had failures:
   1 of   9 in __main__.example_113
   3 of  13 in __main__.example_63
***Test Failed*** 4 failures.
For whitespace errors, see the file /home/pcpa/.sage//tmp/integer_17350.py
         [6.7 s]

----------------------------------------------------------------------
The following tests failed:


        sage -t -force_lib "devel/sage/sage/rings/integer.pyx"
Total time for all tests: 6.7 seconds

The hash problem still bothers me. I don't think there was a change in Python's 
hash() from 2.6 to 2.7. There is a major change in the values calculated by 
hash() in Python 3.2. Does the hash issue go away in a 32-bit build of Sage?

I don't have Sage running on any of my systems at the moment. I'll see if I can 
get it installed and produce the issue. Is this a Mandriva specific issue?

I've released 1.15. I was not able to find any differences in the has function 
with Python 2.7. I believe Sage has some functions that mutate the memory 
contents of an integer. Since gmpy caches the hash value, that may be related 
to errors in the hash values.

Sorry for the long delay in the response.

I confirm, as expected, it no longer causes memory corruption due to both 
sagemath, and later gmpy conflicting usage of mp_set_memory_functions, and 
sympy uses gmpy without problems.

The hashing issue is not related to gmpy, and the doctest failures are 
corrected when applying the patch at 
http://trac.sagemath.org/sage_trac/ticket/11986

I will submit a newer sagemath package to Mandriva buildsystem, that now 
conflicts only with "python-gmpy < 1.15" (gpmy 1.15 already packaged) as well 
as a few other corrections, including the integer hash patch above.