psutil.cpu_count(logical=False) fails to report correct number or even gives None #1404
Labels
Comments
|
ppc: |
|
But this works maybe, from: https://stackoverflow.com/questions/6481005/how-to-obtain-the-number-of-cpus-cores-in-linux-from-the-command-line |
|
Mmm... the problem with your /proc/cpuinfo is that it looks atypical. The current logic determines physical cores if the CPU section in /proc/cpuinfo includes Since your /proc/cpuinfo only provides a I don't know if the current parser logic could be expanded in order to try another methodology before giving up and returning
What does /proc/cpuinfo looks like when you get 64k? |
|
Why not use lscpu that is more reliable? I'll see if we can catch the crazy 64k case. |
|
I don't want to depend on external CLI tools and spawn sub processes. Other than being slower they are unreliable as their output format may change. |
|
On the other hand, parsing cat /proc/cpuinfo is apparently not portable, right? |
|
AFAICT apparently /proc/cpuinfo format changes depending on the Linux distro and CPU architecture. What distro is this? And what does |
|
For ppc: Linux version 4.4.0-128-generic (buildd@bos02-ppc64el-001) (gcc version 5.4.0 20160609 (Ubuntu/IBM 5.4.0-6ubuntu1~16.04.9) ) #154-Ubuntu SMP Fri May 25 14:13:59 UTC 2018 Just normal Ubuntu 16.04. |
|
Here is some data for: >>> import psutil
>>> psutil.cpu_count(logical=False)
65510 #slightly wrong
>>> psutil.cpu_count(logical=True)
20 #correcton Ubuntu 16.04: cpuinfo.txt lscpu |
|
AFAICT the output of /proc/cpuinfo on that distro is just wrong. The only way to solve it (on your platform) is to find an alternative to /proc/cpuinfo (which apparently is what lscpu command does). |
|
psutil doesn't return the proper value on a Raspberry Pi 2 either. Looks like it returns None For reference the Pi 2 has one CPU with 4 cores. Doing a quick strace of lscpu (which reports correct values on the Raspberry Pi), it seems to be looking at |
|
Yeah, it seems |
giampaolo
added a commit
that referenced
this issue
Mar 14, 2019
determine CPUs from /sys/devices/system/cpu/cpu[0-9]/topology/core_id in case /proc/cpuinfo does not provide this info
|
Done in b583b8b. @pseudotensor can you try this out? |
|
@giampaolo Your recent commit (b583b8b) introduces a regression for machines with more than 10 physical cores. The globbing used in |
giampaolo
added a commit
that referenced
this issue
Apr 2, 2019
|
Fixed, thanks. |
nlevitt
added a commit
to nlevitt/psutil
that referenced
this issue
Apr 9, 2019
* origin/master: (150 commits) Linux / CPU freq, fixes giampaolo#1481 improve pmap.py script reuse ps.py script in psutil.test() move get_terminal_size() in _compat.py improve ps.py script improve ps.py script move bytes2human() into psutil._common and reused it from scripts dir fix windows failure re. py 2 vs. 3 fix linux tests fix giampaolo#1474: fix formatting of psutil.tests() which mimicks 'ps aux' output give CREDITS for giampaolo#1480 remove outdated tests Fix read access violation in psutil.cpu_count(logical=False) (giampaolo#1480) update doc update doc [Win] Process IO priority constants + high priority (giampaolo#1479 / giampaolo#1476) don't fail if there are not prev failed tests fix giampaolo#1478: add make command to re-run tests failed on last run [Win] return value is not properly handled for undocumented NT* Windows APIs. (giampaolo#1477) fix error on py 2.7 where OSError doesn't always have winerror attribute update HISTORY for giampaolo#1475 properly check OSError.winerror refactor ionice() on Linux refactor ionice() on Linux ionice test refactoring giampaolo#1404: fix regression not returning CPUs > 9 give CREDITS to Daniel Beer for giampaolo#1471 Fix spurious exception when iterating processes on Solaris (giampaolo#1471) give CREDITS for giampaolo#1470 Fix corner case when /etc/mtab doesn't exist and procfs=/proc (giampaolo#1470) update DEVNOTES Typo fixed (giampaolo#1469) giampaolo#1458: implement colors on Windows update HISTORY / CREDITS, fix some C warnings Make uptime type consitent to fix boot time error. (giampaolo#1225) fix giampaolo#1463: cpu_distribution.py script is broken update issue template issue giampaolo#1404 / linux / phys CPUs count Big docfix (giampaolo#1464) Big docfix (giampaolo#1464) Make tests invariant to LANG setting (giampaolo#1462) test runner: show errors on KeyboardInterrupt test runner refactoring (avoid code duplication) Coloured tests (giampaolo#1459) pre-release [Windows] calculate USS memory by using NtQueryVirtualMemory (giampaolo#1453) fix version highlighting in docs/index (giampaolo#1455) fix version highlighting in README (giampaolo#1454) run win specific tests twice as fast test refactoring test: avoid failing at import time mention how to run tests in INSTALL guide giampaolo#1448: fix Wine support due to missing rtlIpv6AddressToStringA update HISTORY Fix giampaolo#1329: [AIX] disable some functions based on availability in libperfstat (giampaolo#1349) bump up version, fix some doc issues fix ResourceWarning pre-release fix giampaolo#1447: we weren't use @wrap_exceptions around oneshot() (doh\!) update doc + change git hook location update doc make pre-release checks/install src dist in a venv add new make command to check tar.gz sanity move doc; rephrase it a bit add issue templates for 'bug' and 'enhancement' types remove issue template commited by accident Update issue templates giampaolo#1291: (BACKWARD-INCOMPATIBLE) remove memory_maps() on OSX Restore Win-7 support on GIT master (5.5.1 was OK) (giampaolo#1446) try to fix ntext.h restore previous def fix compiler warning fix compiler warning fix compiler warning fix compiler warning take defs from PH set proper SYSTEM_PROCESS_INFORMATION struct from PH fix compilation warnings giampaolo#1398 / win / cmdline: call NtQueryInformationProcess twice, the first time to get the right buf size (ProcessHacker does this) update doc better print formatting for print scripts fix giampaolo#1442: use python3 as Makefile default appveyor: run print scripts after tests highlight top 6 slowest calls add printerr() and exit() to shared utils module add arg parser for ad script introduce a new scriptsutils.py private module shared by all internal utils + refactor print_access_speed.py script giampaolo#1291 / OSX: mark memory_maps() as deprecated and make it alwats raise AccessDenied OSX memory_maps() - add error handling add script for to benchmark API calls move access_denied script _assert_alive() refactor (linux) refactor fix NetBSD: Process.connections() may return incomplete results if using oneshot() giampaolo#1439 add win tests related to send_signal(CTRL_C_EVENT) giampaolo#1227 fix win test fix win tests fix backslash warnings refactor README a bit #fix 1438: do not return any parent() for PID 0 + update doc ...
|
physical cpu count is still incorrect on RHEL7.6 with python 3.6 and psutil 5.7.2. /proc/cpuinfo and lscpu are both correct.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
On ppc systems, this just gives None, even though (say) the system has 8 hyperthreading cores per physical core and (say) 160 total hyperthreading cores.
Even worse, on x86 systems, it randomly gives 64k as the result. A reboot changes that for the systems affected, but it eventually gets stuck that way again.
Since we were feeding that into a choice of physical cores, this led to major problems for any multi-threaded application (a thread bomb).
Perhaps psutil is not reading the data correctly in all cases, or makes assumptions about the data format that can change.
The text was updated successfully, but these errors were encountered: