Comments (8)
I am aware that
nodes[1]
is just a pointer that points to the second element ofnodes
, not an array of size 1
Don't understand your question. I'm referring to the declaration PyObject *nodes[1];
being an array of indefinite size. The literal declaration of this is "nodes is an array of 1 element, of a pointer to PyObject". What I'm saying is, rather than that, it is "nodes is an array of indefinite size, of pointers to PyObject".
In other words, the assumption "NyNodeSetObject
holds the pointer to nodes[1]
" is incorrect.
This might help you: https://cdecl.org/?q=int+*nodes%5B1%5D%3B
Edit: I resized I was skipping too fast in my thoughts in my answer above. Sorry,
from guppy3.
My bad! It all makes sense! I need to brush up on my C syntax! Thank you!
from guppy3.
Hmm... so the things such as NyNodeSet is just, think of it like a python set, except instead of equality being governed by the object's defined __eq__
, it's equality based on the objects identity (a.k.a. memory address). Almost everything else from the API standpoint should be similar to that of a python set. You can see the test cases: https://github.com/zhuyifei1999/guppy3/blob/master/guppy/sets/test.py#L1327
Because nodeset contains objects, it has to keep the objects alive. This is where Py_INCREF
you mentioned comes in.
Things like IdentitySet
is more of a Python wrapper around the C NyNodeSet
that provides the lots of methods and printers and stuffs to better interact with and visualize the sets.
Since you're looking at hv_heap
, I assume you want to see all the way from how nodesets gets created, so let's assume somehow the heap contains only one object, so you did hpy().heap().theone
:
Use.heap()
:
Lines 175 to 195 in a1bd557
View.heap()
:
Lines 340 to 367 in a1bd557
View.enter
is for limiting the visibility of guppy-internal frames during reference traversal, so the important part is self.idset(self.hv.heap())
hv.heap()
is your mentioned hv_heap
:
Lines 889 to 922 in a1bd557
The important thing here is that it creates and returns a mutnodeset (a mutable nodeset) that contains everything it saw during the traversal.
Then View.idset()
is an import / alias:
Line 58 in a1bd557
Use.idset()
:
Line 396 in a1bd557
UniSet.idset()
:
Lines 2214 to 2215 in a1bd557
Here immnodeset
first creates a copy of the mutnodeset, but now it's immutable, then it does IdentitySetFamily._cons()
:
Lines 1526 to 1538 in a1bd557
If the size of the set is 1, then it gets the single element of the set via tuple(arg)[0]
, and calls the constructor of IdentitySetSingleton
with it
And IdentitySetSingleton
's constructor sets self._node
with the given element:
Lines 624 to 630 in a1bd557
So tl;dr: in hpy().heap().theone
, heap()
calls hv_heap()
to create a mutnodeset, and then IdentitySetFamily._cons()
sees that it's a set of size 1 and calls IdentitySetSingleton
with the single element it contains, and the constructor sets the _node
attribute.
Does this answer how it works?
from guppy3.
Hi @zhuyifei1999,
Thank you for your thorough explanation. This is definitely very helpful! I was able to follow the exact control flow when debugging with PyCharm.
Because nodeset contains objects, it has to keep the objects alive. This is where Py_INCREF you mentioned comes in.
This makes sense! To confirm: when adding an object by calling NyNodeSet_setobj()
:
Lines 599 to 619 in a1bd557
The core functionality that adds the object is actually this:
Line 604 in a1bd557
where it adds the object's adjusted address to the
bitset
(or nodes
, depending on whether the set is immutable or not) within the NyNodeSetObject
struct:Lines 325 to 329 in a1bd557
Am I correct?
When retrieving the object from theone
, I am assuming this method:
Lines 640 to 641 in a1bd557
somehow calls some C code to get the object from the address? I am sorry, this is where I get a bit fuzzy. I presume the following C function is triggered by some function(s) when
self._node
in the code above is called?Lines 331 to 335 in a1bd557
Thank you!
from guppy3.
where it adds the object's adjusted address to the bitset (or nodes, depending on whether the set is immutable or not) within the NyNodeSetObject struct:
Immutable one cannot be added. NyMutNodeSet_Check(v)
fails and you get ValueError: mutable nodeset required
.
When retrieving the object from theone, I am assuming this method:
somehow calls some C code to get the object from the address? I am sorry, this is where I get a bit fuzzy. I presume the following C function is triggered by some function(s) when self._node in the code above is called?
No, custom C code is completely uninvolved here.
This corresponds to
Lines 624 to 630 in a1bd557
In __init__
:
self._node = node
In _get_theone
:
return self._node
How __init__
was called with the element / node was explained above. The argument to __init__
is the object returned by theone
. Nothing fancy involved.
from guppy3.
OK. I think I see what is going on here. I believe, correct me if I am wrong, that the "trick" here is:
Lines 2214 to 2215 in a1bd557
It is when
self.immnodeset()
is called where we store actual objects into the immutable node set.I see that when a copy is made:
Lines 135 to 151 in a1bd557
NyNodeSet_iterate()
would call mutnodeset_iterate_visit()
to dereference the address from bitno
, which is stored in mutable node set's bitset
, and get the object. The object is then stored in nodes
:Lines 8 to 16 in a1bd557
I guess my last question is, what is stored in nodes[0]
since the NyNodeSetObject
holds the pointer to nodes[1]
instead of simply nodes
?
Thanks!
from guppy3.
I guess my last question is, what is stored in nodes[0] since the NyNodeSetObject holds the pointer to nodes[1] instead of simply nodes?
This you can answer with GDB:
$ find build guppy -name '*.so' -delete; pip install --global-option build --global-option --debug -e .
Obtaining file:///home/zhuyifei1999/guppy3
Installing collected packages: guppy3
Attempting uninstall: guppy3
Found existing installation: guppy3 3.1.0
Uninstalling guppy3-3.1.0:
Successfully uninstalled guppy3-3.1.0
Running setup.py develop for guppy3
Successfully installed guppy3-3.1.0
$ gdb python
GNU gdb (Gentoo 10.2 vanilla) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
[...]
(gdb) r
Starting program: /home/zhuyifei1999/guppy3/venv/bin/python
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Python 3.9.5 (default, Jun 2 2021, 13:52:24)
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import guppy.sets.setsc
>>> def a():
... b = object()
... print(b)
... c = guppy.sets.setsc.ImmNodeSet([b])
... while True: pass
...
>>> a()
<object object at 0x7ffff6bec3a0>
^C
Program received signal SIGINT, Interrupt.
0x00007ffff7c2d1cc in _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/dev-lang/python-3.9.5_p2/Python-3.9.5/Python/ceval.c:3256
3256 DISPATCH();
(gdb) py-locals
b = <object at remote 0x7ffff6bec3a0>
c = <guppy.sets.setsc.ImmNodeSet at remote 0x7ffff6ad8ae0>
(gdb) p *(NyNodeSetObject *)0x7ffff6ad8ae0
$1 = {
ob_base = {
ob_base = {
ob_refcnt = 0x1,
ob_type = 0x7ffff6abd5c0 <NyImmNodeSet_Type>
},
ob_size = 0x1
},
flags = 0x1,
_hiding_tag_ = 0x0,
u = {
bitset = <object at remote 0x7ffff6bec3a0>,
nodes = {<object at remote 0x7ffff6bec3a0>}
}
}
(gdb)
The nodes[1]
doesn't really mean it's an array of size 1, it's a C array of indefinite size. The array contains whatever objects the nodeset have. This is similar to how CPython implements tuples:
https://github.com/python/cpython/blob/769d7d0c66c5b86e2dd29b9ce67ac2daaab1bb38/Include/cpython/tupleobject.h#L5-L11
typedef struct {
PyObject_VAR_HEAD
/* ob_item contains space for 'ob_size' elements.
Items must normally not be NULL, except during construction when
the tuple is not yet visible outside the function that builds it. */
PyObject *ob_item[1];
} PyTupleObject;
from guppy3.
I am not sure if you actually answered my question. I am aware that nodes[1]
is just a pointer that points to the second element of nodes
, not an array of size 1. But why does it skip the first element? Why does bitset
not do the same (i.e., bitset[1]
)?
from guppy3.
Related Issues (20)
- Feature: replace/patch an imported class at runtime HOT 15
- Question: How to analyze guppy heap files HOT 2
- Getting text output from tool HOT 3
- Idea: Save the entire reference graph (to make profile browsers more useful)
- Feature: monitor external python process, possibly by injecting a stub? HOT 3
- pywin32<300 causes NULL pointer deference during referrer graph generation HOT 32
- Provide `guppy.__version__` HOT 1
- Use commas for big numbers HOT 5
- Add support to release aarch64 wheels HOT 2
- python.exe crashed on hpy.heap() after Import Official Dropbox SDK for Python HOT 2
- Is it possible to dump the memory snapshot for offline analysis? HOT 3
- Usage with JAX HOT 8
- TypeError: '<' not supported between instances of 'weakref' and 'weakref' HOT 4
- Heisenbug: test_RefPat.RefPatCase.test_presentation fails sometimes on Python 3.9 on Windows HOT 4
- AttributeError with guppy.heapy.UniSet.IdentitySetMulti.partition HOT 3
- Fails to build on Python 3.11 RC2: fatal error: longintrepr.h: No such file or directory HOT 13
- Profile Browser fails with AttributeError: 'bool' object has no attribute '_root' HOT 4
- Remote monitor mode not available for python >3.8 ? HOT 1
- Exception in creating hpy instance
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from guppy3.