Comments (6)
savingoyal
commented on May 13, 2024
5
@visr, thanks for opening the issue -
- Windows support would be a complex undertaking (given our liberal use of multiprocessing for starters). We are definitely open to contributions and a good way to kick start the conversation would be to do a POC.
- We are updating our contributing guidelines to reflect this. Expect some progress on that front in the next couple of days.
from metaflow.
pombredanne
commented on May 13, 2024
3
@savingoyal re:
Windows support would be a complex undertaking (given our liberal use of multiprocessing for starters). We are definitely open to contributions and a good way to kick start the conversation would be to do a POC.
FWIW, all data scientists may not be running a *nix Linux or mac. Having official support for Windows would be a big plus. I wish I could coerce them though but that's a dream. (And FWIW, WSL or Cygwin are their own can of frankenstein worms)
Multiprocessing may not be the main issue IMHO but rather:
-
keeping Python 2 and 3 compatibility is going to be a major pain on Windows
-
using signals such as here https://github.com/Netflix/metaflow/blob/c2f70c73782549fdce6c418410ecd9b305199b28/metaflow/plugins/timeout_decorator.py
-
possibly some assumptions about calling some functions and file system semantics that do not apply or are not available on Windows
For 2. Windows is a pain but I have some POSIX/windows utilities that have been battle tested for timeouts here https://github.com/nexB/scancode-toolkit/blob/develop/src/scancode/interrupt.py
For 1. we did that in https://github.com/nexB/scancode-toolkit and I would never do it again which would mean to support Python 3 for Windows. Anything else is a sure way to destroy grey matter very quickly ;)
from metaflow.
pablodz
commented on May 13, 2024
1
And what about news about Windows supporting Linux subsystem?
from metaflow.
pombredanne
commented on May 13, 2024
1
My latest thinking is that Windows support is too mucho pita and with the new WSL2, running on Windows means running on Linux, which is a sane thing. I would close this once some doc is added that WSL2 must be used
from metaflow.
erowan
commented on May 13, 2024
I was hoping that a full remote k8s execution of a flow would work on windows but it doesn't. I stubbed out fcntl but hit this polling dependency. I am in a corporate setup so windows is unfortunately my client. I could develop flows locally in docker but it would be nice to be able to execute the flow in k8s from windows.
$ python 00-helloworld/helloworld.py run --with kubernetes:service_account=auser
Metaflow 2.5.2.post7+git6b07668 executing HelloFlow for user:U038178
Validating your flow...
The graph looks good!
Running pylint...
Pylint is happy!
Polling is not supported on your operating system (Windows)
2022-03-17 13:23:24.242 Workflow starting (run-id 1647523400851411):
2022-03-17 13:23:24.628 [1647523400851411/start/1 (pid 34040)] Task is starting.
2022-03-17 13:23:24.629 Workflow failed.
2022-03-17 13:23:24.630 Terminating 1 active tasks...
2022-03-17 13:23:24.631 [1647523400851411/start/1 (pid 34040)] [KILLED BY ORCHESTRATOR]
2022-03-17 13:23:24.632 [1647523400851411/start/1 (pid 34040)] [KILLED BY ORCHESTRATOR]
2022-03-17 13:23:29.000 Killing 1 remaining tasks after having waited for 5 seconds -- some tasks may not exit clean
2022-03-17 13:23:29.002 Flushing logs...
Internal error
Traceback (most recent call last):
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 209, in execute
self._launch_workers()
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 564, in _launch_workers
self._launch_worker(task)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 584, in _launch_worker
self._poll.add(fd)
AttributeError: 'NoneType' object has no attribute 'add'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\cli.py", line 1102, in main
start(auto_envvar_prefix="METAFLOW", obj=state)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 829, in __call__
return self.main(args, kwargs)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, ctx.params)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\core.py", line 610, in invoke
return callback(args, kwargs)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\cli.py", line 666, in wrapper
return func(args, kwargs)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\_vendor\click\decorators.py", line 33, in new_func
return f(get_current_context().obj, args, kwargs)
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\cli.py", line 789, in run
runtime.execute()
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 230, in execute
self._killall()
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 275, in _killall
list(self._poll_workers())
File "c:\users\u038178\dev\flows\metaflow\metaflow\metaflow\runtime.py", line 521, in _poll_workers
for event in self._poll.poll(PROGRESS_INTERVAL):
AttributeError: 'NoneType' object has no attribute 'poll'
I see Argo support might be forthcoming #50 would this be an option for starting flows in k8s from windows or is windows as a client completely out?
Also its not clear to me how the code is packaged? I see decorators for conda & image and reference to the code being tarball'd to S3. Can you explain or point me to the docs that explain the code packaging for k8s. Thanks.
from metaflow.
savingoyal
commented on May 13, 2024
@erowan Would you like to try out #992 in your windows environment? Here is the code that packages up your flow.
from metaflow.
Related Issues (20)
- Question: Access hash of a card while being populated HOT 2
- Conda environment being treated as disabled, and not appending environment to PATH.
- Metaflow crashes on AWS Batch if folder called `metaflow` is present in the working directory HOT 5
- Cardview on WSL error HOT 2
- S3 access denied even if I have full access to S3
- Certain flows failing on Argo Workflows =>3.5.0 HOT 1
- Metaflow job completion or exit handlers?
- run.finished not set when using AWS Step Functions and there's an error
- setting METAFLOW_OTEL_ENDPOINT when running in ECS fargate, not Kubernetes HOT 1
- add __repr__ methods to Parameter
- create contributing guide
- "Service token file does not exist" error when deploying flow to Argo from CI HOT 1
- argo-workflows create --only-json doesn't export the cron workflow configuration
- Using `tags` as a Parameter name breaks flow. HOT 1
- Add option to batch decorator to increase ephemeralStorage on Fargate
- `--package-suffixes` omits dotfiles HOT 1
- Is it possible to run metaflow steps in custom docker containers on local?
- Opentelemetry configuration not carrying over to Batch
- Add a priority class option for the kubernetes flow decorator HOT 1
- Reduce the number of reserved parameter names
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from metaflow.