Comments (24)
mkonstapel
commented on August 23, 2024
2
Yes, that works! 👍 Guess I'll have to get a move on with our Py3 migration ;-)
from austin.
P403n1x87
commented on August 23, 2024
1
Thanks for opening this issue. I believe that Austin will have to support embedded Python in order for this to work. The change should be quite easy. I will look into it during the next couple of weeks.
from austin.
P403n1x87
commented on August 23, 2024
1
@d0ugal I believe your error is caused by the extra security inside Docker containers. See https://github.com/benfred/py-spy#how-do-i-run-py-spy-in-docker. As recommended there, allowing the capability SYS_PTRACE by adding the --cap-add SYS_PTRACE option when spinning the container should fix the problem.
from austin.
P403n1x87
commented on August 23, 2024
1
@mkonstapel I have pushed a new branch, libpython with the changes required to support libpython on Linux. I have tested with the Apache2 server running inside a systemd Ubuntu 18.04.2 container and mod_wsgi using Python 3.6.7.
Could I kindly ask you to give the changes a try? It is enough to check out the new branch and compile with
gcc -Wall -O3 -s `ls src/*.c` -o src/austinThe binary will be created in the src folder. Use it as, e.g.,
$ sudo src/austin -p <pid>Cheers!
from austin.
mkonstapel
commented on August 23, 2024
1
I'll try, but that'll have to wait until I get back from vacation so it may be a few weeks.
from austin.
mkonstapel
commented on August 23, 2024
1
It's working! 🎉
from austin.
P403n1x87
commented on August 23, 2024
1
@mkonstapel that might not be a bad thing. You will only see numbers if the server is sitting idle doing nothing. You will see frame information when the server is actually running some code. If you can then generate some traffic you should be able some interesting activity from Austin,
from austin.
d0ugal
commented on August 23, 2024
I got the same error when trying to run Austin against a python process in a docker container. I’m not sure if that would be possible to handle but that would be very cool.
from austin.
mkonstapel
commented on August 23, 2024
Alas! It builds and runs, but I get a stream of error messages:
Failed to create thread object
Invalid thread
Failed to create thread object
Invalid thread
Failed to create thread object
Invalid thread
...
Failed to create thread object
Invalid thread
Thread 7f64f446cec0 4
Bad samples 4
Thread 7f64f446cec0 1150
Bad samples 1150
Thread 7f64f446cec0 682
Bad samples 682
Thread 7f64f446cec0 696
Bad samples 696
Thread 7f64f446cec0 784
Bad samples 784
Thread 7f64f446cec0 674
Bad samples 674
But this is Python 2 mod_wsgi. Let me try Python 3...
from austin.
P403n1x87
commented on August 23, 2024
@mkonstapel Many thanks for your precious feedback! I will try to see if I can sort Python 2 out too!
from austin.
P403n1x87
commented on August 23, 2024
@mkonstapel I just tested with mod_wsgi compiled with Python 2.7 and I have got the same results as with Python 3 for my simple WSGI test application. This is what my log shows
May 23 23:07:08 gabriele-clake austin[4836]: ============================ AUSTIN ╦̵̵̿╤─ ҉ ~ • ============================
May 23 23:07:08 gabriele-clake austin[4836]: austin version: 0.6.1-beta
May 23 23:07:08 gabriele-clake austin[4836]: Python version: 2.7.? (from shared library)
May 23 23:07:08 gabriele-clake austin[4836]: Failed to retrieve PyCodeObject
May 23 23:07:08 gabriele-clake austin[4836]: Failed to get code object for frame
May 23 23:07:08 gabriele-clake austin[4836]: Failed to get top frame for thread
May 23 23:07:08 gabriele-clake austin[4836]: Sampling interval: 100 usec
May 23 23:07:24 gabriele-clake austin[4836]: Sampling time statistics (min/avg/max) : 14/39/20969 us
May 23 23:07:24 gabriele-clake austin[4836]: Long-running sample rate: 245 samples over sampling interval/103644 (0.24 %)
May 23 23:07:24 gabriele-clake austin[4836]: Error rate: 0 invalid samples/103644 (0.00 %)
May 23 23:07:24 gabriele-clake austin[4836]: ============================================================================
Was it the same application that you tried with Python 3? Are there any chances that you could repeat the tests with Python 2, perhaps using a different sampling interval?
Cheers,
Gabriele
from austin.
mkonstapel
commented on August 23, 2024
Yes, same application. I'm getting:
May 24 13:14:24 michieldev austin[2329]: ============================ AUSTIN ╦̵̵̿╤─ ҉ ~ • ============================
May 24 13:14:24 michieldev austin[2329]: austin version: 0.6.1-beta
May 24 13:14:24 michieldev austin[2329]: Python version: 2.7.? (from shared library)
May 24 13:14:27 michieldev austin[2331]: ============================ AUSTIN ╦̵̵̿╤─ ҉ ~ • ============================
May 24 13:14:27 michieldev austin[2331]: austin version: 0.6.1-beta
May 24 13:14:27 michieldev austin[2331]: Python version: 2.7.? (from shared library)
May 24 13:14:29 michieldev austin[2331]: Sampling interval: 100 usec
May 24 13:14:29 michieldev austin[2331]: Failed to create thread object
May 24 13:14:29 michieldev austin[2331]: Invalid thread
...
May 24 13:14:30 michieldev austin[2331]: Failed to create thread object
May 24 13:14:30 michieldev austin[2331]: Invalid thread
May 24 13:14:30 michieldev austin[2331]: Failed to create thread object
May 24 13:14:30 michieldev austin[2331]: Invalid thread
May 24 13:14:30 michieldev austin[2331]: Failed to create thread object
May 24 13:14:30 michieldev austin[2331]: Invalid thread
May 24 13:14:30 michieldev austin[2331]: Sampling time statistics (min/avg/max) : 18/102/26428 us
May 24 13:14:30 michieldev austin[2331]: Long-running sample rate: 166 samples over sampling interval/1695 (9.79 %)
May 24 13:14:30 michieldev austin[2331]: Error rate: 1695 invalid samples/1695 (100.00 %)
This is Python 2.7.15rc1, with libapache2-mod-wsgi 4.5.17-1.
from austin.
P403n1x87
commented on August 23, 2024
@mkonstapel Could you please compile with the DEBUG option:
gcc -Wall -g `ls src/*.c` -o src/austin -DDEBUGand share the new log?
Cheers!
from austin.
mkonstapel
commented on August 23, 2024
Here you are!
May 24 22:16:33 michieldev austin[4373]: ============================ AUSTIN ╦̵̵̿╤─ ҉ ~ • ============================
May 24 22:16:33 michieldev austin[4373]: austin version: 0.6.1-beta
May 24 22:16:33 michieldev austin[4373]: Attaching to process with PID 3852
May 24 22:16:33 michieldev austin[4373]: Timeout: 1000ms
May 24 22:16:33 michieldev austin[4373]: HEAP bounds: 55835bd31000-55835bd73000 rw-p 00000000 00:00 0 [heap]
May 24 22:16:33 michieldev austin[4373]: BSS bounds: 7fa7a823b000-7fa7a825e000 rw-p 00000000 00:00 0
May 24 22:16:33 michieldev austin[4373]: Maximal VM address space: 0x55835a5b9000-0x7ffe163f5000
May 24 22:16:33 michieldev austin[4373]: Base @ 0x7fa7a7ce1000
May 24 22:16:33 michieldev austin[4373]: Symbol _PyThreadState_Current found @ 0x7fa7a823b6a0
May 24 22:16:33 michieldev austin[4373]: Python binary: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
May 24 22:16:33 michieldev austin[4373]: Python version: 2.7.? (from shared library)
May 24 22:16:35 michieldev austin[4373]: Unable to de-reference global symbols. Scanning the bss section...
May 24 22:16:35 michieldev austin[4373]: Found possible interpreter state @ 0x7fa7a8222f60 (offset 0x2a244c4f1f60).
May 24 22:16:35 michieldev austin[4373]: Stack trace constructed from possible interpreter state (error code: 0)
May 24 22:16:35 michieldev austin[4373]: Possible interpreter state referenced by BSS @ 0x7fa7a823b108
May 24 22:16:35 michieldev austin[4373]: Sampling interval: 100 usec
May 24 22:16:35 michieldev austin[4373]: Failed to create thread object
May 24 22:16:35 michieldev austin[4373]: Invalid thread
from austin.
P403n1x87
commented on August 23, 2024
@mkonstapel Many thanks for your cooperation so far! Those log entries look fine except maybe for one bit. Here is my log for comparison
May 25 11:25:00 gabriele-clake austin[22651]: ============================ AUSTIN ╦̵̵̿╤─ ҉ ~ • ============================
May 25 11:25:00 gabriele-clake austin[22651]: austin version: 0.6.1-beta
May 25 11:25:00 gabriele-clake austin[22651]: Attaching to process with PID 3499
May 25 11:25:00 gabriele-clake austin[22651]: Timeout: 1000ms
May 25 11:25:00 gabriele-clake austin[22651]: HEAP bounds: 55bc7b211000-55bc7b232000 rw-p 00000000 00:00 0 [heap]
May 25 11:25:00 gabriele-clake austin[22651]: BSS bounds: 7f3bb998d000-7f3bb99b0000 rw-p 00000000 00:00 0
May 25 11:25:00 gabriele-clake austin[22651]: Maximal VM address space: 0x55bc7acb1000-0x7ffe11f68000
May 25 11:25:00 gabriele-clake austin[22651]: Base @ 0x7f3bb9433000
May 25 11:25:00 gabriele-clake austin[22651]: Symbol _PyThreadState_Current found @ 0x7f3bb998d6a0
May 25 11:25:00 gabriele-clake austin[22651]: Python binary: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
May 25 11:25:00 gabriele-clake austin[22651]: Python version: 2.7.? (from shared library)
May 25 11:25:02 gabriele-clake austin[22651]: Unable to de-reference global symbols. Scanning the bss section...
May 25 11:25:02 gabriele-clake austin[22651]: Found possible interpreter state @ 0x7f3ba8000c30 (offset 0x297f2cdefc30).
May 25 11:25:02 gabriele-clake austin[22651]: Stack trace constructed from possible interpreter state (error code: 0)
May 25 11:25:02 gabriele-clake austin[22651]: Possible interpreter state referenced by BSS @ 0x7f3bb99a9fd8
May 25 11:25:02 gabriele-clake austin[22651]: Sampling interval: 10000 usec
May 25 11:27:08 gabriele-clake austin[22651]: Sampling time statistics (min/avg/max) : 26/145/35359 us
May 25 11:27:08 gabriele-clake austin[22651]: Long-running sample rate: 1 samples over sampling interval/12470 (0.01 %)
May 25 11:27:08 gabriele-clake austin[22651]: Error rate: 0 invalid samples/12470 (0.00 %)
May 25 11:27:08 gabriele-clake austin[22651]: ============================================================================
I'm also using Python 2.7.15rc1 and this can be seen from Symbol _PyThreadState_Current found @ 0x7f3bb998d6a0 which is common to both logs. The rest of the numbers depend on how memory is mapped and thus can vary. However, the difference between the address on
Possible interpreter state referenced by BSS @ 0x7f3bb99a9fd8
and the lower bound of the BSS section as logged on
May 25 11:25:00 gabriele-clake austin[22651]: BSS bounds: 7f3bb998d000-7f3bb99b0000 rw-p 00000000 00:00 0
should also be the same and this doesn't seem to be the case. This is quite unexpected and might be the cause of the problem. Just to rule out any load issues, I have stress-tested Apache with 1e5/1e6 parallel short requests and Austin didn't bulge (see the error rate in my log).
I would be very grateful if you could pull the last commit from the libpython branch, compile with debug support and test again. I have introduced a possible fix that should detect stale interpreter state instances and skip them, thus hopefully settling for the correct one in all cases.
Cheers!
from austin.
mkonstapel
commented on August 23, 2024
No change, I'm afraid:
austin version: 0.6.1-beta
Attaching to process with PID 3852
Timeout: 1000ms
HEAP bounds: 55835bd31000-55835bd73000 rw-p 00000000 00:00 0 [heap]
BSS bounds: 7fa7a823b000-7fa7a825e000 rw-p 00000000 00:00 0
Maximal VM address space: 0x55835a5b9000-0x7ffe163f5000
Base @ 0x7fa7a7ce1000
Symbol _PyThreadState_Current found @ 0x7fa7a823b6a0
Python binary: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
Python version: 2.7.? (from shared library)
Unable to de-reference global symbols. Scanning the bss section...
Found possible interpreter state @ 0x7fa7a8222f60 (offset 0x2a244c4f1f60).
Stack trace constructed from possible interpreter state (error code: 0)
Possible interpreter state referenced by BSS @ 0x7fa7a823b108 (offset 108)
Sampling interval: 100 usec
Failed to create thread object
Invalid thread
from austin.
P403n1x87
commented on August 23, 2024
@mkonstapel I'm afraid there isn't much more I can do without a minimal example to reproduce the problem. Would you be able to provide one?
Cheers!
from austin.
P403n1x87
commented on August 23, 2024
@mkonstapel Would you be able to give the devel branch a try? Many improvements have been merged since the last tests. Thanks!
from austin.
mkonstapel
commented on August 23, 2024
Cool, let me give it a spin!
Jul 1 15:24:43 michieldev austin[27731]: ============================ AUSTIN ╦̵̵̿╤─ ҉ ~ • ============================
Jul 1 15:24:43 michieldev austin[27731]: austin version: 0.6.1-beta
Jul 1 15:24:43 michieldev austin[27731]: Attaching to process with PID 27214
Jul 1 15:24:43 michieldev austin[27731]: Start up timeout: 1000ms
Jul 1 15:24:43 michieldev austin[27731]: BSS bounds: 55e355c9c000-55e355c9f000 rw-p 00000000 00:00 0
Jul 1 15:24:43 michieldev austin[27731]: HEAP bounds: 55e356c58000-55e356cae000 rw-p 00000000 00:00 0 [heap]
Jul 1 15:24:43 michieldev austin[27731]: BSS bounds: 7fe470207000-7fe47022a000 rw-p 00000000 00:00 0
Jul 1 15:24:43 michieldev austin[27731]: BSS bounds: 55e355c9c000-55e355c9f000 rw-p 00000000 00:00 0
Jul 1 15:24:43 michieldev austin[27731]: HEAP bounds: 55e356c58000-55e356cae000 rw-p 00000000 00:00 0 [heap]
Jul 1 15:24:43 michieldev austin[27731]: BSS bounds: 7fe470207000-7fe47022a000 rw-p 00000000 00:00 0
Jul 1 15:24:43 michieldev austin[27731]: Base @ (nil)
Jul 1 15:24:43 michieldev austin[27731]: Symbol _PyThreadState_Current found @ 0x7fe470207680
Jul 1 15:24:43 michieldev austin[27731]: Page size: 4096
Jul 1 15:24:43 michieldev austin[27731]: Python library: /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
Jul 1 15:24:43 michieldev austin[27731]: Maximal VM address space: 0x55e3559f8000-0x7ffff8c00000
Jul 1 15:24:43 michieldev austin[27731]: Python version: 2.7.? (from shared library)
Jul 1 15:24:43 michieldev austin[27731]: Found possible interpreter state @ 0x7fe4701eef60 (offset 0x2a0119596f60).
Jul 1 15:24:43 michieldev austin[27731]: Stack trace constructed from possible interpreter state (error code: 0)
Jul 1 15:24:43 michieldev austin[27731]: Possible interpreter state referenced by BSS @ 0x7fe4702070e8 (offset e8)
Jul 1 15:24:43 michieldev austin[27731]: Interpreter state located from BSS scan.
Jul 1 15:24:43 michieldev austin[27731]: Interpreter State de-referenced @ raddr: 0x7fe4701eef60 after 1 attempts
Jul 1 15:24:43 michieldev austin[27731]: Sampling interval: 100 usec
Jul 1 15:24:43 michieldev austin[27731]: Failed to create thread object
Jul 1 15:24:43 michieldev austin[27731]: Invalid thread
and that keeps repeating. stdout just shows
Thread 7fe46fdda300 4
Bad sample 4
Thread 7fe46fdda300 718
Bad sample 718
Thread 7fe46fdda300 3970
Bad sample 3970
...
Looks like I'll have to get you that minimal reproducing example, but it's been a wee bit busy around here...
from austin.
P403n1x87
commented on August 23, 2024
@mkonstapel thanks for your test results. It would be grand if you could provide a minimal example as I still cannot reproduce the issue with Python 2.7, Apache 2 and mod_wsgi from the Ubuntu repositories.
from austin.
P403n1x87
commented on August 23, 2024
@mkonstapel version 1.0.0 is out. It adds support for multi-process applications, so mod_wsgi is the ideal candidate for testing it. Would you have some time to give the new release a try and see if it has solved this issue?
Cheers!
from austin.
mkonstapel
commented on August 23, 2024
Oh cool, and yes, I will :)
from austin.
mkonstapel
commented on August 23, 2024
... I think, at first I got stack traces and now it's only outputting numbers:
Thread 7f6e5357d540 507
Thread 7f6e5357d540 182
Thread 7f6e5357d540 377
Thread 7f6e5357d540 170
Thread 7f6e5357d540 157
Thread 7f6e5357d540 412
Will have to do some more testing...
from austin.
P403n1x87
commented on August 23, 2024
Resolved by #39
from austin.
Related Issues (20)
- Install via apt prints (null) in help HOT 1
- Using the extension HOT 4
- PackagesNotFoundError on conda installation HOT 1
- Python version inference can segfault
- Insufficient Permissions HOT 8
- austinp only builds on systems with a patched libbfd HOT 2
- Tracing a program running inside docker container may fail HOT 8
- austin 3.5.0 regression test failure HOT 8
- Can't run austin directly with python scripts HOT 20
- Can't run Austin against pyenv python distribution HOT 3
- Austin Windows installer does not uninstall older version
- Can't trace a program running inside docker container by pid or ppid HOT 17
- No samples collected HOT 6
- Can't run script with local package import HOT 1
- Feature request: Integrate Austin with python debugging functions in vscode extension HOT 1
- Update vscode extension docs to say what to do with task output options on run HOT 1
- VS code extension reports "profiling complete" for non-existent files HOT 1
- Feature Request: Call Austin programmatically HOT 1
- Processes not recognized as Python as Rasberry pi HOT 7
- austinp hanging on profiling numpy/scipy
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from austin.