Comments (4)
This is behavior that we'll change for v0.2. For now, please use the resume
flag when initializing your pipeline. e.g.
args = ImportArguments(
...
resume=True,
overwrite=True,
)
runner.pipeline(args)
from hipscat-import.
I tried overwrite=True
and resume=True
. I get two runs of Planning, followed by a warning that maybe a cluster is running already, followed by an error.
Planning : 100%|████████████████████████████████| 5/5 [00:00<00:00, 1067.80it/s]
Planning : 100%|████████████████████████████████| 5/5 [00:00<00:00, 1699.47it/s]
/stage/irsa-staff-shupe/hipscat/allwise/venv/lib/python3.11/site-packages/distributed/node.py:182: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 41397 instead
warnings.warn(
2023-08-15 13:52:26,916 - distributed.nanny - ERROR - Failed to start process
Traceback (most recent call last):
File "/stage/irsa-staff-shupe/hipscat/allwise/venv/lib/python3.11/site-packages/distributed/nanny.py", line 434, in instantiate
result = await self.process.start()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/stage/irsa-staff-shupe/hipscat/allwise/venv/lib/python3.11/site-packages/distributed/nanny.py", line 722, in start
await self.process.start()
File "/stage/irsa-staff-shupe/hipscat/allwise/venv/lib/python3.11/site-packages/distributed/process.py", line 55, in _call_and_set_future
res = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/stage/irsa-staff-shupe/hipscat/allwise/venv/lib/python3.11/site-packages/distributed/process.py", line 215, in _start
process.start()
File "/stage/irsa-staff-shupe/micromamba/envs/hipscat/lib/python3.11/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
^^^^^^^^^^^^^^^^^
File "/stage/irsa-staff-shupe/micromamba/envs/hipscat/lib/python3.11/multiprocessing/context.py", line 288, in _Popen
return Popen(process_obj)
^^^^^^^^^^^^^^^^^^
File "/stage/irsa-staff-shupe/micromamba/envs/hipscat/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/stage/irsa-staff-shupe/micromamba/envs/hipscat/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/stage/irsa-staff-shupe/micromamba/envs/hipscat/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 42, in _launch
prep_data = spawn.get_preparation_data(process_obj._name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/stage/irsa-staff-shupe/micromamba/envs/hipscat/lib/python3.11/multiprocessing/spawn.py", line 158, in get_preparation_data
_check_not_importing_main()
File "/stage/irsa-staff-shupe/micromamba/envs/hipscat/lib/python3.11/multiprocessing/spawn.py", line 138, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
from hipscat-import.
Are you using a main guard to run the pipeline? e.g.
import stuff
def main():
args = ImportArguments(
...
)
runner.pipeline(args)
if __name__ == '__main__':
main()
If this doesn't improve things, can you add the code you're using to start the pipeline?
from hipscat-import.
Using a main guard works when using resume=True
and overwrite=True
. Thanks for the tip! Would it be worth it to add to the docs in the example scripts, or (easier) in Debugging Tips?
from hipscat-import.
Related Issues (20)
- Check the total number of rows throughout the catalog import pipeline
- Add `__version__` attribute
- Automatically convert coordinate to degrees when units are available
- Complex arrow columns cause an error during the reducing stage
- Use healpix minimum distances for margin filtering
- Use index mask filtering for margin generation
- Pipeline for importing auxiliarry data sets
- Add flag for including radec columns on index catalog creation
- Improve margin_order selection algorithm HOT 1
- Catalog import: Option to run a single stage HOT 3
- Bundle a poetry.lock file with the release HOT 1
- Support default_columns in catalog_info
- High memory usage for Finishing stage
- Catalog import fails with `csv.gz` files
- Docs: add PS1 python type files from STScI
- Row count checks on final parquet, _metadata and ancillary files
- Schema check; check for correct columns and dtypes.
- Check precision during import
- Explore VOTable as import. HOT 1
- Run margin generation within the main import pipeline
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hipscat-import.