Skip to content

Parallel / delayed garbage collection error using Python 3.12.10 #1707

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
manucalop opened this issue Apr 11, 2025 · 4 comments · Fixed by #1711
Closed

Parallel / delayed garbage collection error using Python 3.12.10 #1707

manucalop opened this issue Apr 11, 2025 · 4 comments · Fixed by #1711

Comments

@manucalop
Copy link

manucalop commented Apr 11, 2025 •
edited
Loading

Hi! I've found the following issue:

from math import sqrt
from joblib import Parallel, delayed

Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))

which provides the following error:

Exception ignored in: <function ResourceTracker.__del__ at 0x1049654e0>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.12/3.12.10/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/resource_tracker.py", line 77, in __del__
  File "/opt/homebrew/Cellar/python@3.12/3.12.10/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/resource_tracker.py", line 86, in _stop
  File "/opt/homebrew/Cellar/python@3.12/3.12.10/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/resource_tracker.py", line 111, in _stop_locked
ChildProcessError: [Errno 10] No child processes
Exception ignored in: <function ResourceTracker.__del__ at 0x1049054e0>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.12/3.12.10/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/resource_tracker.py", line 77, in __del__
  File "/opt/homebrew/Cellar/python@3.12/3.12.10/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/resource_tracker.py", line 86, in _stop
  File "/opt/homebrew/Cellar/python@3.12/3.12.10/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/resource_tracker.py", line 111, in _stop_locked
ChildProcessError: [Errno 10] No child processes

The error doesn't show if you set the number of jobs to 1 or if you use Python 3.12.9 or below. Joblib's version is 1.4.2. Tested on Docker too. I hope it helps!
@lesteve
Copy link
Member

lesteve commented Apr 14, 2025 •
edited
Loading

I confirm this happens with Python 3.12.10 and 3.13.3 at exit. Probably a recent bugfix in CPython, we would need to look at it in more details.

This is very likely why the CI is failing on main #1708.

@lesteve lesteve mentioned this issue Apr 14, 2025
Closed
@buchijw
Copy link
Contributor

buchijw commented Apr 23, 2025

The issue remains on Python 3.12.10, joblib 1.4.2 and the newest loky 3.5.2 (joblib/loky#451). Tested on both conda-forge and pip packages.

>>> from math import sqrt
>>> from joblib import Parallel, delayed
>>>
>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
>>> exit()
Exception ignored in: <function ResourceTracker.__del__ at 0x10a2068e0>
Traceback (most recent call last):
  File "/Users/johnwilliam/anaconda3/envs/test_joblib/lib/python3.12/multiprocessing/resource_tracker.py", line 77, in __del__
  File "/Users/johnwilliam/anaconda3/envs/test_joblib/lib/python3.12/multiprocessing/resource_tracker.py", line 86, in _stop
  File "/Users/johnwilliam/anaconda3/envs/test_joblib/lib/python3.12/multiprocessing/resource_tracker.py", line 111, in _stop_locked
ChildProcessError: [Errno 10] No child processes
Exception ignored in: <function ResourceTracker.__del__ at 0x10488a8e0>
Traceback (most recent call last):
  File "/Users/johnwilliam/anaconda3/envs/test_joblib/lib/python3.12/multiprocessing/resource_tracker.py", line 77, in __del__
  File "/Users/johnwilliam/anaconda3/envs/test_joblib/lib/python3.12/multiprocessing/resource_tracker.py", line 86, in _stop
  File "/Users/johnwilliam/anaconda3/envs/test_joblib/lib/python3.12/multiprocessing/resource_tracker.py", line 111, in _stop_locked
ChildProcessError: [Errno 10] No child processes

@tomMoral
Copy link
Contributor

This will be fixed by #1711 but we also need to handle a deprecation warning in numpy

@tomMoral tomMoral mentioned this issue Apr 23, 2025
Merged
@beatrizadan-tech
Copy link

When will the new joblib version be released with this fix?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

MTN vendor loky 3.5.2 tomMoral/joblib
5 participants
@lesteve @tomMoral @manucalop @buchijw @beatrizadan-tech