giampaolo / psutil Goto Github PK

In r84 I added UID/GID support for Linux.
The same should be done also for Windows and OS X.

Implement a method to determine the amount of time that a process has
executed in kernel/user mode.

It happens in zombie process test when accessing create_time property but
also get_cpu_times() and get_cpu_percent() methods.

======================================================================
FAIL: test_zombie_process (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\pyftpdlib\svn\psutil\test\test_psutil.py", line 236, in
test_zombie_p
rocess
    self.assertRaises(psutil.NoSuchProcess, getattr, p, "create_time")
AssertionError: NoSuchProcess not raised

----------------------------------------------------------------------

Implement a get_total_phymem() function or a TOTAL_PHYMEM constant to read
the total physical memory available.

Knowing the number of processors is needed for a reliable get_cpu_percent()
implementation (see Issue #34 ) on systems with multiple CPU cores.

Such method should not be included in the public API but only used
internally in get_cpu_percent().

Linux:

root@ubuntu:/home/user# python -m timeit -s "import psutil" -c
"psutil.pid_exists(1)"
100000 loops, best of 3: 4.5 usec per loop


Windows:

C:\>C:\python26\python.exe -m timeit -s "import psutil" -c
"psutil.pid_exists(1)"
10000 loops, best of 3: 161 usec per loop


This is due to the underlying Windows implementation using "pid in
get_all_pids()" which is much slower than Linux and OS X implementations
relying on "os.kill(pid, 0)".

What steps will reproduce the problem?  
>>> psutil.Process(0).kill()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python26\Lib\site-packages\psutil\psutil.py", line 140, in kill
    _platform_impl.kill_process(self.pid, sig)
  File "C:\Python26\Lib\site-packages\psutil\_psmswindows.py", line 21, in
wrapper
    return callable(*args, **kwargs)
  File "C:\Python26\Lib\site-packages\psutil\_psmswindows.py", line 44, in
kill_process
    return _psutil_mswindows.kill_process(pid)
WindowsError: [Error 87] Incorrect parameter


It seems we have to add another special case when killing PID 0.
Are you ok with raising InsufficientPrivileges?

What steps will reproduce the problem?  
>>> p = psutil.Process(0)
>>> p.name
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python26\Lib\site-packages\psutil\psutil.py", line 106, in name
    self.deproxy()
  File "C:\Python26\Lib\site-packages\psutil\psutil.py", line 65, in deproxy
    self._procinfo = _platform_impl.get_process_info(self._procinfo.pid)
  File "C:\Python26\Lib\site-packages\psutil\_psmswindows.py", line 16, in
get_p
rocess_info
    infoTuple = _psutil_mswindows.get_process_info(pid)
WindowsError: [Error 87] Parameter incorrect


PID 0 on Windows refers to "System Idle Process", a special process from
which we are not supposed to fetch any information.
A special case for this process should be implemented so that "name"
property is the only information we provide for it.

Implement a method to determine when a process has been started/created.
As for time.time(), it would make sense returning the time as a floating
point number expressed in seconds since the epoch, in UTC.

Here are 7 issues with the psutil/arch/osx/process_info.c file:

line 134 -- val is unused.

line 135 -- val_start is unused.

line 167 -- pid is long while mib is int.

line 171 -- should the constant 22 not be EINVAL?

line 173 -- should -2 not be ARGS_ACCESS_DENIED?

line 242 -- np is never assigned any value.

line 254 -- ERROR_B falls thru to ERROR_C.


For your consideration, attached is a modified version of this file.

psutil should support per-process memory percent utilization.

Maybe it's because I miss something but that's what I get when I try to
compile psutil on Windows by using MingGW.
Any hint?


D:\pyftpdlib\svn\psutil\trunk>C:\python25\python setup.py build -c mingw32
running build
running build_ext
building 'psutil/_psutil_mswindows' extension
C:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -D_WIN32_WINNT=0x0500
-IC:\pytho
n25\include -IC:\python25\PC -c psutil/_psutil_mswindows.c -o
build\temp.win32-2
.5\Release\psutil\_psutil_mswindows.o
psutil/_psutil_mswindows.c: In function `kill_process':
psutil/_psutil_mswindows.c:186: error: `hProcess' undeclared (first use in this
function)
psutil/_psutil_mswindows.c:186: error: (Each undeclared identifier is
reported o
nly once
psutil/_psutil_mswindows.c:186: error: for each function it appears in.)
psutil/_psutil_mswindows.c:186: error: `false' undeclared (first use in
this fun
ction)
psutil/_psutil_mswindows.c:186: error: `PID' undeclared (first use in this
funct
ion)
psutil/_psutil_mswindows.c:199: warning: passing arg 1 of `Py_BuildValue' makes
pointer from integer without a cast
psutil/_psutil_mswindows.c: In function `SetSeDebug':
psutil/_psutil_mswindows.c:237: error: initializer element is not constant
psutil/_psutil_mswindows.c:237: error: (near initialization for
`PsutilMethods[0
].ml_meth')
psutil/_psutil_mswindows.c:238: error: initializer element is not constant
psutil/_psutil_mswindows.c:238: error: (near initialization for
`PsutilMethods[0
]')
psutil/_psutil_mswindows.c:239: error: initializer element is not constant
psutil/_psutil_mswindows.c:239: error: (near initialization for
`PsutilMethods[1
].ml_meth')
psutil/_psutil_mswindows.c:240: error: initializer element is not constant
psutil/_psutil_mswindows.c:240: error: (near initialization for
`PsutilMethods[1
]')
psutil/_psutil_mswindows.c:241: error: initializer element is not constant
psutil/_psutil_mswindows.c:241: error: (near initialization for
`PsutilMethods[2
].ml_meth')
psutil/_psutil_mswindows.c:242: error: initializer element is not constant
psutil/_psutil_mswindows.c:242: error: (near initialization for
`PsutilMethods[2
]')
psutil/_psutil_mswindows.c:243: error: initializer element is not constant
psutil/_psutil_mswindows.c:243: error: (near initialization for
`PsutilMethods[3
]')
psutil/_psutil_mswindows.c:251: error: syntax error at end of input
error: command 'gcc' failed with exit status 1

On Python bug tracker there's a discussion [1] about adding a new function
to the os module to change the name of the executable as it is shown by ps
and top.

This could be useful since on a UNIX system there might be many processes
running under the name "python".
In such cases it's desirable that the process is displayed as "foo" rather
than "python foo", and this seems to be possible with prctl(), at least on
Linux.

I still don't know the implementation details implied (for example: is
prctl() available on other UNIXes?) but it would be a useful thing to have.


[1] http://bugs.python.org/issue5672 [2] http://linux.die.net/man/2/prctl

It could be useful implementing < and > Process objects comparisons based
on PID.
Example:

>>> Process(50) > Process(30)
True
>>> Process(50) < Process(30)
False

A method to calculate process CPU utilization as a percentage would be needed.

It would make sense adding an iter_process() module-level function similar
to get_process_list() but returning an iterator instead.
get_process_list() could be removed since easily replaceable by
list(iter_process()).

What steps will reproduce the problem?  
>>> psutil.Process(0).kill()
>>> psutil.Process(2408).name
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python26\Lib\site-packages\psutil\psutil.py", line 54, in __init__
    if not pid_exists(pid):
  File "C:\Python26\Lib\site-packages\psutil\psutil.py", line 149, in
pid_exists

    return _platform_impl.pid_exists(pid)
  File "C:\Python26\Lib\site-packages\psutil\_psmswindows.py", line 29, in
pid_e
xists
    return pid in _psutil_mswindows.get_pid_list()
WindowsError: [Error 87] Parametro non corretto
>>>
>>> psutil.Process(2408).name
'xchat.exe'
>>>


It seems that psutil.Process(0).kill() fails internally and that its
exception is "cached" somehow and raised when we attempt to fetch
information from another process.

In issue 33 we decided that PID 0's create_time must be equal to system uptime.
I'm opening this report to keep track of our progresses about system uptime
implementation on all platforms.

system uptime has been currently implemented on Windows and Linux platforms
as r257 .
OS X and FreeBSD implementations are still missing as well as any potential
test case.

Currently we fetch processes path and command line info by reading
/proc/PID/exe and /proc/PID/cmdline respectively but it seems they're not
available for quite a lot of processes having low pids.

In attachment is the output of psutil.py run on my Ubuntu box.

It could be helpful seeing how Linux "ps" utility gathers such information
and possibly doing the same.

psutil should provide a module-level function to return system-wide CPU
utilization as a percentage.

Per conversation with Jay this should be scheduled for 0.1.1 release.
The real effort is the Windows implementation since on OS X and Linux pwd
module already does the work for us.

What steps will reproduce the problem?  
<snippet>
import psutil

n = 0
for p in psutil.process_iter():
    try:
        n += p.get_memory_percent()
    except:
        print p.pid
        print p.name
        raise
print n
</snippet>

raises:

D:\pyftpdlib\svn\psutil>foo.py
944
csrss.exe
Traceback (most recent call last):
  File "D:\pyftpdlib\svn\psutil\foo.py", line 10, in <module>
    n += p.get_memory_percent()
  File "D:\pyftpdlib\svn\psutil\psutil\_psutil.py", line 250, in
get_memory_perc
ent
    rss = _platform_impl.get_memory_info(self.pid)[0]
  File "psutil\_psmswindows.py", line 29, in wrapper
    raise AccessDenied
psutil.error.AccessDenied


csrss.exe is owned by the SYSTEM user, I'm running the script as Administrator.

What steps will reproduce the problem?  
1. Compile psutil with -O2 enabled
2. Run the test suite or psutil.py
3. NoSuchProcess exception thrown for *any* PID, caused by sysctl returning
ESRC for every PID passed to it. 

What is the expected output?  


What do you see instead?  
Process information should be printed as it is normally. Instead
NoSuchProcess is raised, caused by ESRCH error from sysctl regardless of PID

psutil module should have module-level functions to tell total used
physical/virtual memory in bytes and as a percentage of total available.

root@ubuntu:/home/user/psutil/test# python -V
Python 2.5.2
root@ubuntu:/home/user/psutil/test# uname -a
Linux ubuntu 2.6.27-7-server #1 SMP Fri Oct 24 07:37:55 UTC 2008 i686 GNU/Linux
root@ubuntu:/home/user/psutil/test# python test_psutil.py
test_args (__main__.TestCase) ... FAIL
test_get_process_list (__main__.TestCase) ... ok
test_kill (__main__.TestCase) ... ok
test_name (__main__.TestCase) ... FAIL
test_path (__main__.TestCase) ... FAIL
test_pid (__main__.TestCase) ... ok

======================================================================
FAIL: test_args (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_psutil.py", line 54, in test_args
    self.assertEqual(psutil.Process(self.proc.pid).args, [sys.executable])
AssertionError: None != ['/usr/bin/python']

======================================================================
FAIL: test_name (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_psutil.py", line 62, in test_name
    os.path.basename(sys.executable))
AssertionError: 'python2.5' != 'python'

======================================================================
FAIL: test_path (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_psutil.py", line 47, in test_path
    self.assertEqual(psutil.Process(self.proc.pid).path, sys.executable)
AssertionError: '/usr/bin/python2.5' != '/usr/bin/python'

----------------------------------------------------------------------
Ran 6 tests in 0.436s

FAILED (failures=3)

Line 165 of function getcmdargs in file /psutil/arch/bsd/process_info.c should be

    int mib[4];

since lines 194-7 set mib[0..3] to

    mib[0] = CTL_KERN;
    mib[1] = KERN_PROC;
    mib[2] = KERN_PROC_ARGS;
    mib[3] = pid;

What steps will reproduce the problem?  
1. p = Process(1234)
2. time.sleep(5)  # time-consuming task, process dies in meantime
3. p.name 
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "psutil/psutil.py", line 69, in name
    self.deproxy()
  File "psutil/psutil.py", line 58, in deproxy
    self._procinfo = _platform_impl.get_process_info(self._procinfo.pid)
  File "psutil/_pslinux.py", line 39, in get_process_info
    f = open("/proc/&#37;s/stat" &#37;pid)
IOError: [Errno 2] No such file or directory: '/proc/1234/stat'


As discussed in Issue 12 it would make sense that NoSuchProcess is raised
when we attempt to retrieve info from a Process() instance referring to a
process which is dead in meantime. Issue 12 includes a patch which solves this 
anything we should figure out how to implement that same thing on Windows
and OS X in a reliable way as well.

As pointed out by Jay it seems acceptable that we check for process
"aliveness" only when calling get_process_info() and avoid to raise
NoSuchProcess in case the process dies later.

======================================================================
FAIL: test_get_cpu_times (__main__.LimitedUserTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_psutil.py", line 82, in test_get_cpu_times
    self.fail("expected: %s, found: %s" %(utime, user_time))
AssertionError: expected: 0.233333333333, found: 0.111669

----------------------------------------------------------------------

If I'm not mistaken Jay should have told me that the culprit is os.times()
implementation which is buggy on FreeBSD.
If so it would make sense disabling the test for FreeBSD and file a new
report on the Python bug tracker.

What steps will reproduce the problem?  
>>> psutil.pid_exists(0)
True
>>> 0 in psutil.get_pid_list()
False

A README file describing instructions on how to compile psutil on different
platforms is needed.

psutil support for Solaris OS

What steps will reproduce the problem?  
Running a continual script in the background, e.g.: 

import psutil
import sys

run_count = 0
while (run_count < 20000):
    run_count = run_count + 1
    sys.stdout.write("\r\r&#37;s" &#37; run_count)
    sys.stdout.flush()

    try:
        cmdline = psutil.Process(0).cmdline
    except Exception, e:
        print "\n"
        print "Exception for PID 0: &#37;s" &#37; e
        continue

    PID = -1
    for i in psutil.get_process_list():
        PID = i.pid
        try:
            cmdline = i.cmdline
        except psutil.NoSuchProcess:
            continue
        except Exception, e:
            print "\n"
            print "Exception for PID &#37;s: &#37;s" &#37; (PID, e)
            continue 

What is the expected output?  


What do you see instead?  
Memory usage (monitored via ps or top) climbs steadily in the background,
instead of staying constant or rising and falling as you'd expect with
garbage collecting.

_Skipping this issue number to maintain synchronization with Google Code issue 

As a new feature, it would be nice to get the process parent as a psutil class.

Example:

>>> parent = psutil.parent(os.getpid())
>>> parent.pid
11038
>>> parentPID = psutil.Process(11038).parent.pid
>>> parentPID
1

Thank you.

Almost all test cases implemented so far use a time.sleep(0.1) call after
subprocess.Popen() as a provisional workaround to let the sub process
initialize properly.
This should be avoided by using some kind of signal that let us know when
the process is properly initialized.

What steps will reproduce the problem?  
See Issue #12 for details 

What is the expected output?  


What do you see instead?  
memory leak instead of stable memory usage during long running script.

As far as I know Jay should have wrote one.
Before adding further C code, especially on Windows, a new test script
should be added in the current code base to allow us testing that any
change applied to C extensions does not introduce memory leaks.

What steps will reproduce the problem?  
1. >>> import psutil
2. >>> psutil
3. <module 'psutil' from
'/usr/local/lib/python2.4/site-packages/psutil/__init__.pyc'>
4. >>> dir(psutil)
['Process', 'ProcessInfo', '__builtins__', '__doc__', '__file__',
'__name__', '__path__', '_pslinux', 'get_process_list', 'os', 'psutil',
'sys', 'test']
5. >>> psutil.psutil
<module 'psutil.psutil' from
'/usr/local/lib/python2.4/site-packages/psutil/psutil.pyc'>
6. >>> dir(psutil.psutil)
['Process', 'ProcessInfo', '__builtins__', '__doc__', '__file__',
'__name__', '_platform_impl', '_pslinux', 'get_process_list', 'os', 'sys',
'test']



Anomalies:

- step 4: 'psutil' shouldn't be there
- step 4: '_platform_impl' is missing
- step 6: '_platform_impl' is available in psutil.psutil (?)

There is a potential memory leak in function getcmdpath of file psutil/arch/bsd/process_info.c.  A 
free(path) should be inserted before line 143, like

    if (sysctl(mib, 4, path, &size, NULL, 0) == -1) {
        //perror("sysctl");
        free(path);
        return NULL;       /* Insufficient privileges */
    }

Currently this is already working on Windows but other platforms do not
currently implement get_total_virtmem() to read the total virtual memory
available.

What steps will reproduce the problem?  
1. download the tarball
2. tar zxvf psutil-0.1.0.tar.gz
3. resulting directory is rooted at 'usr'.

A "normal" python tarball would untar into a directory named
'psutil-0.1.0', and contain a 'setup.py' that would build and install
the package.

I would suggest investigating distutils, creating a setup.py, and using
it to build the source distribution.  I just learned how myself a couple
days ago, and it's really pretty easy.

--RDM

psutil should support per-process memory utilization statistics in both raw
(e.g. bytes) numbers and % utilization.

What steps will reproduce the problem?  
>>> p = psutil.Process(0)
>>> p.cmdline
'<unknown>'

Windows "System IDLE process" can't be queried even by Process Explorer,
and that's ok, but for being consistent with other platforms cmdline should
return an empty list "[]" instead of '<unknown>'.

What steps will reproduce the problem?  
Windows:

1. Log in as a limited user (not Administrator)
2. Attempt to fetch some information from a process owned by a
high-privileged user:

>>> import psutil
>>> p = psutil.Process(1016)  # svchost.exe owned by "SYSTEM" 
>>> p.name
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python26\Lib\site-packages\psutil\psutil.py", line 106, in name
    self.deproxy()
  File "C:\Python26\Lib\site-packages\psutil\psutil.py", line 65, in deproxy
    self._procinfo = _platform_impl.get_process_info(self._procinfo.pid)
  File "C:\Python26\Lib\site-packages\psutil\_psmswindows.py", line 17, in
get_process_info
    infoTuple = _psutil_mswindows.get_process_info(pid)
WindowsError: [Error 5] Access denied
>>>



Linux:

1. Log in as a limited user
2. Attempt to kill a process owned by root (fetching process info is
permitted):

user@ubuntu:~$ whoami
user
user@ubuntu:~$ python
Python 2.4.6 (#1, Jan 26 2009, 11:40:33)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> p = psutil.Process(1)
>>> p.name
'init'
>>> p.kill()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/local/lib/python2.4/site-packages/psutil/psutil.py", line 136,
in kill
    _platform_impl.kill_process(self.pid, sig)
  File "/usr/local/lib/python2.4/site-packages/psutil/_pslinux.py", line
67, in kill_process
    os.kill(pid, sig)
OSError: [Errno 1] Operation not permitted 

What is the expected output?  


What do you see instead?  
I expect a unique psutil.InsufficientPrivilegesError exception to be raised
on all systems instead of OSError, WindowsError or anything else.
*When* exactly raising the exception (e.g. when instantiating the class or
when accessing properties) needs to be discussed.

What steps will reproduce the problem?  
>>> p = psutil.Process(3124234545)
>>> 

There's no such process with PID 3124234545 but no exception is raised when
Process() gets instantiated.
Various types of exceptions may occur later when we start to access process
properties, for example:

>>> p.name
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "psutil/psutil.py", line 69, in name
    self.deproxy()
  File "psutil/psutil.py", line 58, in deproxy
    self._procinfo = _platform_impl.get_process_info(self._procinfo.pid)
  File "psutil/_pslinux.py", line 39, in get_process_info
    f = open("/proc/&#37;s/stat" &#37;pid)
IOError: [Errno 2] No such file or directory: '/proc/3124234545/stat'


It would make sense checking if the PID exists at least in the Process
class constructor and eventually also in other parts of the code.

What steps will reproduce the problem?  
>>> import psutil
>>> psutil.Process(4390).cmdline
['/bin/login', '--', '', '', '', '', '']
>>>

What steps will reproduce the problem?  
>>> psutil.get_pid_list().count(0)
2

get_process_list() should swallow NoSuchProcess exceptions (and possibly
others) during creation of the Process objects from PIDs, since a PID may
go away while generating the Process object.
When such an event occurs get_process_list() should just go on and process
other pids.