Linux / Python3 implementation speedup

[giampaolo]

From g.rodola on February 15, 2014 04:00:11

I noticed a substantial difference between Python 2.7 and 3.4 in terms of speed on Linux.
Apparently all psutil functions are considerably slower on Python 3.4, 
sometimes by a factor of 2x!
Some examples:

giampaolo@UX32VD:~/svn/psutil$ python2.7 -m timeit -s "import psutil" 
"psutil.cpu_times(True)"
10000 loops, best of 3: 51 usec per loop
giampaolo@UX32VD:~/svn/psutil$ python3.4 -m timeit -s "import psutil" 
"psutil.cpu_times(True)"
10000 loops, best of 3: 87.9 usec per loop

giampaolo@UX32VD:~/svn/psutil$ python2.7 -m timeit -s "import psutil" 
"psutil.virtual_memory()"
10000 loops, best of 3: 29.5 usec per loop
giampaolo@UX32VD:~/svn/psutil$ python3.4 -m timeit -s "import psutil" 
"psutil.virtual_memory()"
10000 loops, best of 3: 60 usec per loop

giampaolo@UX32VD:~/svn/psutil$ python2.7 -m timeit -s "import psutil" 
"psutil.swap_memory()"
10000 loops, best of 3: 52.9 usec per loop
giampaolo@UX32VD:~/svn/psutil$ python3.4 -m timeit -s "import psutil" 
"psutil.swap_memory()"
10000 loops, best of 3: 81.4 usec per loop


This is true for both system and process functions.
The culprit is the way files are opened.
By default "open(file, 'r')" on Python 2.7 opens the file in binary mode 
whereas on Python 3.X the file is opened in text mode.
That means that every time we read a line Python will convert it in 
text/unicode by using system default encoding.
Since Linux implementation of psutil relies almost entirely on reading files in 
/proc the slowdown affects the entire API.

Original issue: http://code.google.com/p/psutil/issues/detail?id=478

[giampaolo]

From g.rodola on February 22, 2014 05:34:35

Fixed in revision f8c40e7ecbab .
Some benchmarks:

===============================================================================
function                              1.2.1       2.0.0           unit
===============================================================================
psutil.cpu_percent                      101        44.5           msec per loop
psutil.cpu_times                       59.2          41           usec per loop
psutil.cpu_times_percent                101        76.5           msec per loop
psutil.disk_io_counters                 213         228           usec per loop
psutil.disk_partitions                  103        80.1           usec per loop
psutil.disk_usage                      6.02         6.4           usec per loop
psutil.get_boot_time                     53        39.3           usec per loop
psutil.get_pid_list                     236         199           usec per loop
psutil.get_users                       50.8        52.3           usec per loop
psutil.net_io_counters                 97.1         100           usec per loop
psutil.pid_exists                      1.13        1.26           usec per loop
psutil.process_iter                    11.4        9.45           msec per loop
psutil.swap_memory                     81.6        57.7           usec per loop
psutil.virtual_memory                  57.7        44.3           usec per loop
===============================================================================

Process class methods benefit of similar speedups.
Closing this out.

Status: FixedInHG

[giampaolo]

From g.rodola on March 10, 2014 04:36:50

Closing out as fixed as 2.0.0 version is finally out.

Status: Fixed
Labels: Milestone-2.0.0