Comments (2)
trws
commented on June 14, 2024
After some digging into this, I think it worked before because the script wasn't actually run as a true "main" module. Otherwise this should have been a failure from the start. With multiprocessing used in log.py anything that runs back to that point at the top level unprotected by if __name__ == '__main__': should blow up this way. It's a change in behavior, but I'm not sure the new behavior is wrong since it's what would happen if you run the same script with python with the correct import paths:
❯ PYTHONPATH=./lib/spack:./lib/spack/external:./lib/spack/external/_vendoring python3 ./test-script.py
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/spawn.py", line 122, in spawn_main
exitcode = _main(fd, parent_sentinel)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/spawn.py", line 131, in _main
prepare(preparation_data)
File "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/spawn.py", line 246, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/spawn.py", line 297, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen runpy>", line 291, in run_path
File "<frozen runpy>", line 98, in _run_module_code
File "<frozen runpy>", line 88, in _run_code
File "/Users/scogland1/Workspace/dir.noindex/Repos/spack/test-script.py", line 6, in <module>
find()
File "/Users/scogland1/Workspace/dir.noindex/Repos/spack/lib/spack/spack/main.py", line 720, in __call__
with log_output(out, echo=True):
File "/Users/scogland1/Workspace/dir.noindex/Repos/spack/lib/spack/llnl/util/tty/log.py", line 566, in __enter__
self.process.start()
File "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/context.py", line 288, in _Popen
return Popen(process_obj)
^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 42, in _launch
prep_data = spawn.get_preparation_data(process_obj._name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/spawn.py", line 164, in get_preparation_data
_check_not_importing_main()
File "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/spawn.py", line 140, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
To fix this issue, refer to the "Safe importing of main module"
section in https://docs.python.org/3/library/multiprocessing.html
Traceback (most recent call last):
File "/Users/scogland1/Workspace/dir.noindex/Repos/spack/./test-script.py", line 6, in <module>
find()
File "/Users/scogland1/Workspace/dir.noindex/Repos/spack/lib/spack/spack/main.py", line 720, in __call__
with log_output(out, echo=True):
File "/Users/scogland1/Workspace/dir.noindex/Repos/spack/lib/spack/llnl/util/tty/log.py", line 643, in __exit__
string = self.parent_pipe.recv()
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/connection.py", line 430, in _recv_bytes
buf = self._recv(4)
^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/connection.py", line 399, in _recv
raise EOFError
EOFError
from spack.
psakievich
commented on June 14, 2024
Thanks @trws. Your diagnosis makes sense to me. It seems that just making sure all spack.main.SpackCommands are called from inside __main__ is sufficient to resolve this issue functionally.
#!/usr/bin/env spack-python
from spack.main import SpackCommand
find = SpackCommand("find")
if __name__=="__main__":
find()At a minimum we should update the documentation. It would be nice to have a way to detect this and give a better error thou.
from spack.
Related Issues (20)
- Installation issue: gperftools when `build_system=cmake` HOT 9
- Avoid storing binary blobs as test files for sz compression HOT 4
- Installation issue: openmpi (Spack v0.21 & v0.22) with compiler [email protected] HOT 6
- Spec matrix: concretization failures when matrixing MPI dependency HOT 3
- Concretizer error for Python variant (+tkinter) HOT 2
- Installation issue: HIP package sets excessive environment variables in dependent package modulefiles
- cuda: `nvcc` is unable to find `-lcupti`
- Installation issue: [email protected]%[email protected]
- Installation issue: glibc iconv incomplete by default on fedora, needs libiconv HOT 11
- Environments: root specs are not always marked explicit HOT 5
- Add an option to shorten git.[commit] version in Spec.format() HOT 3
- Installation issue: `seacas 2022-10-14:2023-05-30` with `fmt@9` when `gcc` mixed with `apple-clang` HOT 5
- Installation issue: `vtk@9%gcc14` HOT 4
- Cannot use `config:shared_linking:bind:true` anymore HOT 1
- Cannot install musl HOT 2
- Installation issue: abinit HOT 1
- Installation issue: gh
- Installation issue: kentutils HOT 2
- Spec syntax: semantic for `%` and `^` for compiler as deps
- Installation issue: [email protected] +cuda ~rocm fails to build HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spack.