haidra-org / ai-horde-worker Goto Github PK
View Code? Open in Web Editor NEWThis repo turns your PC into a AI Horde worker node
License: GNU Affero General Public License v3.0
ai-horde-worker's People
Contributors
Stargazers
Watchers
Forkers
tijszwinkels johndpope hugoross3000 calebzchong tazlin vadi2 xcytxs alrlc henk717 alcyon6 esuwarrior all-ki jodanjodan evguu theairaj rlm7d holynuts ryanehamil niwethp500 kinghenry14 albertahn guspuffygit wolfgangmeyers bearla efreak machado2 brok3nstrings fedmich tiagofema bizai3000 robballantyne motolies itzyounghazn redboxing amoranemix zent3600 mrseeker heisenstein0 demandcluster chaoren12345 5l1v3r1 jackrhode xilocoo g4rg luisjesch hanzrexd godlikedesigningmachine lostruins 6un9-h0-dan city-unit butterfly-afk ttmor antlersandsunbeams lavaplanetllm gonner22 teasitta cjvandyk goldenwind8 menonpg yingyingyang7878 deevee3 zackbradshawai-horde-worker's Issues
Create a rwkvstic worker
https://github.com/harrisonvanderbyl/rwkvstic
Looks like this is an agnostic library for text. It might make sense to onboard to our worker as a new worker type and connect it to the new text generation API
Maintenance mode could release vram
Sometimes when I'm putting the worker into maintenance mode it's because I want to use the graphics card for something else for a minute. Unfortunately as the worker doesn't release it's vram, one has to sometimes terminate the worker. With many models restarting the worker can take well over 30 minutes as they load into the ray object store.
It would be nice if when a worker detects it's in maintenance mode, it temporarily releases VRAM.
"ModuleNotFoundError: No module named 'gradio'" when trying to run WebUI
I'm on a Windows 11 machine. I cloned the repository. Ran "update-runtime" and after 30 minutes it's completed like this (I'm not sure if it's completed succesfully. Please confirm):
After that, I ran the bridge-webui. But it fails to run throwing this error message continuously:
Traceback (most recent call last):
File "webui.py", line 1, in <module>
import gradio as gr
ModuleNotFoundError: No module named 'gradio'
So what should I do now?
"inpainting image doesn't have an alpha channel. Aborting gen"
This error causes worker to go into maintenance.
Pruned models
It would be good if we maintain a curated collection of pruned models, as we only need to do inference in this project. Currently some downloaded models are as large as 7 GB, while a pruned, fp16 quantized, VAE merged model only takes 2 GB, which reduces VRAM usage and helps serve multiple models on mid-tier gaming GPUs.
(If multiple models use the same VAE, splitting out the VAE can additionally save 0.2 GB for each model, but I think that's negligible.)
Allow to translate interrogated captions
Allow captions to receive a language payload then pass it though free translator services before sending it back
For example https://pypi.org/project/translate-api/
BUG: img_source referenced before assignment
Latest error I noticed that caused my worker to go into maitenance mode:
File "/home/wolfgang/AI-Horde-Worker/worker/jobs/stable_diffusion.py", line 241, in start_job
if img_mask.size != img_source.size:
│ └ <property object at 0x7f37f5398630>
└ <PIL.Image.Image image mode=RGB size=448x1024 at 0x7F3457C55060>
UnboundLocalError: local variable 'img_source' referenced before assignment
ERROR | 2023-04-03 09:04:14.989259 | worker.workers.framework:check_running_job_status:141 - Job failed with exception, local variable 'img_source' referenced before assignment
Sorry if this is already fixed. If not I can try submitting a PR when I have time.
Using controlnet + pix2pix sets workers into maintenance mode
Received a request for controlnet in conjunction with pix2pix, it ended up setting my worker to maintenance.
pix2pix by itself works fine, just cant be combined with CN 😅
ERROR | 2023-03-03 00:18:10.532211 | worker.jobs.stable_diffusion:start_job:325 - Something went wrong when processing request. Please check your trace.log file for the full stack trace. Payload: {'prompt': 'Hills with some snow on top', 'height': 960, 'width': 768, 'seed': '2509572726', 'tiling': False, 'n_iter': 1, 'batch_size': 1, 'save_individual_images': False, 'save_grid': False, 'ddim_steps': 30, 'sampler_name': 'k_euler_karras', 'cfg_scale': 7.0, 'denoising_strength': 0.75, 'control_type': 'canny', 'init_img': <PIL.Image.Image image mode=RGB size=768x960 at 0x1841D0AAD70>, 'request_type': 'img2img', 'model': 'pix2pix'}
TRACE | 2023-03-03 00:18:10.534212 | worker.jobs.stable_diffusion:start_job:331 - Traceback (most recent call last):
File "C:\Users\Airic\AI-Horde-Worker\worker\jobs\stable_diffusion.py", line 319, in start_job
generator.generate(**gen_payload)
File "C:\Users\Airic\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\util\cast.py", line 29, in wrap
return amp.autocast(device_type="cuda", dtype=dtype)(no_grad()(func))(*args, **kwargs)
File "C:\Users\Airic\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\amp\autocast_mode.py", line 14, in decorate_autocast
return func(*args, **kwargs)
File "C:\Users\Airic\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "C:\Users\Airic\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\stable_diffusion\compvis.py", line 307, in generate
(self.control_net_model.control_model, "cpu"),
AttributeError: 'NoneType' object has no attribute 'control_model'
Ensure "ALL MODELS" loads SD standard models first
The first models loaded are used as fallback if there is a problem.
stable_diffusion_inpainting with no alpha channel puts worker into maintenance?
Log provided by TroyBORG:
INFO | 2023-03-03 00:59:36.314204 | worker.jobs.framework:submit_job:137 - Submitted job with id 4f53b779-c4a0-471a-86f9-8e93e6ccb56d and contributed for 13.14
DEBUG | 2023-03-03 00:59:37.927499 | nataili.stable_diffusion.compvis:generate:672 - [Low VRAM] after sampling - model.device = cpu, model.cond_stage_model.device = cpu, model.first_stage_model.device = cuda:0
DEBUG | 2023-03-03 00:59:37.927499 | nataili.stable_diffusion.compvis:generate:873 - [Low VRAM] decode - model.first_stage_model.device = cuda:0
DEBUG | 2023-03-03 00:59:38.071926 | worker.jobs.poppers:horde_pop:39 - Job pop took 2.400418 (node: kai:7002:4.1.0)
INFO | 2023-03-03 00:59:38.320071 | worker.jobs.stable_diffusion:start_job:320 - Generation finished successfully.
DEBUG | 2023-03-03 00:59:38.321049 | worker.jobs.stable_diffusion:start_job:361 - post-processing done...
DEBUG | 2023-03-03 00:59:38.323001 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG | 2023-03-03 00:59:38.325929 | worker.workers.framework:check_running_job_status:146 - Job finished successfully in 2.703s (Total Completed: 9998)
DEBUG | 2023-03-03 00:59:38.346436 | worker.workers.framework:start_job:116 - New job processing
DEBUG | 2023-03-03 00:59:38.346436 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: stable_diffusion_inpainting
DEBUG | 2023-03-03 00:59:38.347415 | worker.jobs.stable_diffusion:start_job:148 - ['stable_diffusion_inpainting', 'stable diffusion 1', 'img2img', 'txt2img']
WARNING | 2023-03-03 00:59:38.347415 | worker.jobs.stable_diffusion:start_job:157 - Model stable_diffusion_inpainting chosen for txt2img or img2img gen, switching to Liberty instead.
DEBUG | 2023-03-03 00:59:38.349017 | worker.jobs.stable_diffusion:start_job:200 - txt2img (Liberty) request with id 959f1d4e-a91d-4e1c-a057-7daa7a0f86d1 picked up. Initiating work...
INFO | 2023-03-03 00:59:38.349017 | worker.jobs.stable_diffusion:start_job:312 - Starting generation: Liberty @ 512x512 for 30 steps. Prompt length is 175 characters And it appears to contain 0 weights
DEBUG | 2023-03-03 00:59:38.751956 | nataili.stable_diffusion.compvis:generate:584 - model.device = cuda:0, model.cond_stage_model.device = cuda:0, model.first_stage_model.device = cuda:0
DEBUG | 2023-03-03 00:59:38.751956 | nataili.stable_diffusion.compvis:generate:589 - Iteration: 1/1
DEBUG | 2023-03-03 00:59:38.764650 | ldm.modules.encoders.modules:forward:301 - Token count for prompt = 49
DEBUG | 2023-03-03 00:59:38.876250 | nataili.stable_diffusion.compvis:generate:641 - [Low VRAM] before sampling - model.device = cuda:0, model.cond_stage_model.device = cpu, model.first_stage_model.device = cpu
DEBUG | 2023-03-03 00:59:38.877226 | ldm.models.diffusion.kdiffusion:sample:143 - model.device: cuda:0
DEBUG | 2023-03-03 00:59:39.081555 | worker.jobs.stable_diffusion:prepare_submit_payload:376 - R2 Upload response: <Response [200]>
DEBUG | 2023-03-03 00:59:39.082071 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG | 2023-03-03 00:59:40.025794 | worker.jobs.poppers:horde_pop:39 - Job pop took 1.625835 (node: goblin:7001:4.1.0)
DEBUG | 2023-03-03 00:59:40.712635 | worker.jobs.framework:submit_job:113 - Upload completed in 1.628548
INFO | 2023-03-03 00:59:40.713156 | worker.jobs.framework:submit_job:137 - Submitted job with id b4542659-b5a4-4a5b-a7c9-b8d2da215457 and contributed for 5.54
DEBUG | 2023-03-03 00:59:40.847607 | nataili.stable_diffusion.compvis:generate:672 - [Low VRAM] after sampling - model.device = cpu, model.cond_stage_model.device = cpu, model.first_stage_model.device = cuda:0
DEBUG | 2023-03-03 00:59:40.847607 | nataili.stable_diffusion.compvis:generate:873 - [Low VRAM] decode - model.first_stage_model.device = cuda:0
INFO | 2023-03-03 00:59:41.233947 | worker.jobs.stable_diffusion:start_job:320 - Generation finished successfully.
DEBUG | 2023-03-03 00:59:41.234925 | worker.jobs.stable_diffusion:start_job:361 - post-processing done...
DEBUG | 2023-03-03 00:59:41.235901 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG | 2023-03-03 00:59:41.239805 | worker.workers.framework:check_running_job_status:146 - Job finished successfully in 2.891s (Total Completed: 9999)
DEBUG | 2023-03-03 00:59:41.259834 | worker.workers.framework:start_job:116 - New job processing
DEBUG | 2023-03-03 00:59:41.259834 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: stable_diffusion_inpainting
DEBUG | 2023-03-03 00:59:41.259834 | worker.jobs.stable_diffusion:start_job:200 - inpainting (stable_diffusion_inpainting) request with id bf3ddfa9-f62d-4e22-90c1-e611c0da47bf picked up. Initiating work...
WARNING | 2023-03-03 00:59:41.260354 | worker.jobs.stable_diffusion:start_job:297 - inpainting image doesn't have an alpha channel. Aborting gen
DEBUG | 2023-03-03 00:59:41.261397 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG | 2023-03-03 00:59:41.261902 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG | 2023-03-03 00:59:42.038881 | worker.jobs.stable_diffusion:prepare_submit_payload:376 - R2 Upload response: <Response [200]>
DEBUG | 2023-03-03 00:59:42.038881 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG | 2023-03-03 00:59:42.460393 | worker.jobs.framework:submit_job:113 - Upload completed in 1.196615
INFO | 2023-03-03 00:59:42.460393 | worker.jobs.framework:submit_job:137 - Submitted job with id bf3ddfa9-f62d-4e22-90c1-e611c0da47bf and contributed for 0.0
DEBUG | 2023-03-03 00:59:42.963758 | worker.jobs.poppers:horde_pop:39 - Job pop took 1.680121 (node: thrr:7001:4.1.0)
DEBUG | 2023-03-03 00:59:43.211049 | worker.jobs.framework:submit_job:113 - Upload completed in 1.170281
INFO | 2023-03-03 00:59:43.211049 | worker.jobs.framework:submit_job:137 - Submitted job with id 959f1d4e-a91d-4e1c-a057-7daa7a0f86d1 and contributed for 6.5
DEBUG | 2023-03-03 00:59:43.509148 | worker.workers.framework:check_running_job_status:146 - Job finished successfully in 2.250s (Total Completed: 10000)
DEBUG | 2023-03-03 00:59:43.530443 | worker.workers.framework:start_job:116 - New job processing
DEBUG | 2023-03-03 00:59:43.530443 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: stable_diffusion_inpainting
DEBUG | 2023-03-03 00:59:43.530443 | worker.jobs.stable_diffusion:start_job:200 - inpainting (stable_diffusion_inpainting) request with id 11af04bc-8d01-4fc8-a380-b51c6e91a72a picked up. Initiating work...
WARNING | 2023-03-03 00:59:43.531421 | worker.jobs.stable_diffusion:start_job:297 - inpainting image doesn't have an alpha channel. Aborting gen
DEBUG | 2023-03-03 00:59:43.532396 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG | 2023-03-03 00:59:43.532396 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG | 2023-03-03 00:59:44.441255 | worker.jobs.framework:submit_job:113 - Upload completed in 0.906672
INFO | 2023-03-03 00:59:44.441781 | worker.jobs.framework:submit_job:137 - Submitted job with id 11af04bc-8d01-4fc8-a380-b51c6e91a72a and contributed for 0.0
DEBUG | 2023-03-03 00:59:45.847746 | worker.jobs.poppers:horde_pop:39 - Job pop took 2.294338 (node: kai:7002:4.1.0)
DEBUG | 2023-03-03 00:59:46.399698 | worker.workers.framework:check_running_job_status:146 - Job finished successfully in 2.859s (Total Completed: 10001)
DEBUG | 2023-03-03 00:59:46.419986 | worker.workers.framework:start_job:116 - New job processing
DEBUG | 2023-03-03 00:59:46.419986 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: stable_diffusion_inpainting
DEBUG | 2023-03-03 00:59:46.419986 | worker.jobs.stable_diffusion:start_job:200 - inpainting (stable_diffusion_inpainting) request with id f73281be-aa8f-4388-b04b-d1ff4a79d24a picked up. Initiating work...
WARNING | 2023-03-03 00:59:46.420967 | worker.jobs.stable_diffusion:start_job:297 - inpainting image doesn't have an alpha channel. Aborting gen
DEBUG | 2023-03-03 00:59:46.421941 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG | 2023-03-03 00:59:46.421941 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG | 2023-03-03 00:59:47.304982 | worker.jobs.framework:submit_job:113 - Upload completed in 0.880648
INFO | 2023-03-03 00:59:47.305518 | worker.jobs.framework:submit_job:137 - Submitted job with id f73281be-aa8f-4388-b04b-d1ff4a79d24a and contributed for 0.0
DEBUG | 2023-03-03 00:59:48.886971 | worker.jobs.poppers:horde_pop:39 - Job pop took 2.443463 (node: stablehorde:7001:4.1.0)
DEBUG | 2023-03-03 00:59:49.456767 | worker.workers.framework:check_running_job_status:146 - Job finished successfully in 3.032s (Total Completed: 10002)
DEBUG | 2023-03-03 00:59:49.477096 | worker.workers.framework:start_job:116 - New job processing
DEBUG | 2023-03-03 00:59:49.477096 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: stable_diffusion_inpainting
DEBUG | 2023-03-03 00:59:49.477096 | worker.jobs.stable_diffusion:start_job:200 - inpainting (stable_diffusion_inpainting) request with id 85b782b8-976f-41a2-9555-6632a031c83d picked up. Initiating work...
WARNING | 2023-03-03 00:59:49.478076 | worker.jobs.stable_diffusion:start_job:297 - inpainting image doesn't have an alpha channel. Aborting gen
DEBUG | 2023-03-03 00:59:49.479049 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG | 2023-03-03 00:59:49.479049 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG | 2023-03-03 00:59:50.471636 | worker.jobs.framework:submit_job:113 - Upload completed in 0.990169
INFO | 2023-03-03 00:59:50.471636 | worker.jobs.framework:submit_job:137 - Submitted job with id 85b782b8-976f-41a2-9555-6632a031c83d and contributed for 0.0
DEBUG | 2023-03-03 00:59:52.161064 | worker.jobs.poppers:horde_pop:39 - Job pop took 2.660226 (node: stablehorde:7001:4.1.0)
DEBUG | 2023-03-03 00:59:52.707828 | worker.workers.framework:check_running_job_status:146 - Job finished successfully in 3.235s (Total Completed: 10003)
DEBUG | 2023-03-03 00:59:52.728832 | worker.workers.framework:start_job:116 - New job processing
DEBUG | 2023-03-03 00:59:52.728832 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: stable_diffusion_inpainting
DEBUG | 2023-03-03 00:59:52.728832 | worker.jobs.stable_diffusion:start_job:200 - inpainting (stable_diffusion_inpainting) request with id d6fc432c-b6d5-4d09-84f2-3869eed066cc picked up. Initiating work...
WARNING | 2023-03-03 00:59:52.729812 | worker.jobs.stable_diffusion:start_job:297 - inpainting image doesn't have an alpha channel. Aborting gen
DEBUG | 2023-03-03 00:59:52.730786 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG | 2023-03-03 00:59:52.730786 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG | 2023-03-03 00:59:54.332901 | worker.jobs.framework:submit_job:113 - Upload completed in 1.599558
INFO | 2023-03-03 00:59:54.333429 | worker.jobs.framework:submit_job:137 - Submitted job with id d6fc432c-b6d5-4d09-84f2-3869eed066cc and contributed for 0.0
DEBUG | 2023-03-03 00:59:55.536398 | worker.jobs.poppers:horde_pop:39 - Job pop took 2.783143 (node: kai:7002:4.1.0)
WARNING | 2023-03-03 00:59:55.536922 | worker.jobs.poppers:horde_pop:70 - Maintenance mode activated because worker is dropping too many jobs.Please investigate if your performance has been impacted and consider reducing your max_power or your max_threads (403)
DEBUG | 2023-03-03 00:59:57.538836 | worker.workers.framework:check_running_job_status:146 - Job finished successfully in 4.813s (Total Completed: 10004)
DEBUG | 2023-03-03 00:59:58.393343 | worker.jobs.poppers:horde_pop:39 - Job pop took 0.830176 (node: orc:7001:4.1.0)
WARNING | 2023-03-03 00:59:58.393875 | worker.jobs.poppers:horde_pop:70 - Maintenance mode activated because worker is dropping too many jobs.Please investigate if your performance has been impacted and consider reducing your max_power or your max_threads (403)
DEBUG | 2023-03-03 01:00:02.052249 | worker.jobs.poppers:horde_pop:39 - Job pop took 1.630994 (node: kai:7002:4.1.0)
WARNING | 2023-03-03 01:00:02.052249 | worker.jobs.poppers:horde_pop:70 - Maintenance mode activated because worker is dropping too many jobs.Please investigate if your performance has been impacted and consider reducing your max_power or your max_threads (403)
DEBUG | 2023-03-03 01:00:05.051193 | worker.jobs.poppers:horde_pop:39 - Job pop took 0.973929 (node: troll:7001:4.1.0)
WARNING | 2023-03-03 01:00:05.051721 | worker.jobs.poppers:horde_pop:70 - Maintenance mode activated because worker is dropping too many jobs.Please investigate if your performance has been impacted and consider reducing your max_power or your max_threads (403)
DEBUG | 2023-03-03 01:00:08.425477 | worker.jobs.poppers:horde_pop:39 - Job pop took 1.348741 (node: kai:7002:4.1.0)
WARNING | 2023-03-03 01:00:08.425982 | worker.jobs.poppers:horde_pop:70 - Maintenance mode activated because worker is dropping too many jobs.Please investigate if your performance has been impacted and consider reducing your max_power or your max_threads (403)
DEBUG | 2023-03-03 01:00:12.596920 | worker.jobs.poppers:horde_pop:39 - Job pop took 2.144849 (node: thrr:7001:4.1.0)
WARNING | 2023-03-03 01:00:12.597438 | worker.jobs.poppers:horde_pop:70 - Maintenance mode activated because worker is dropping too many jobs.Please investigate if your performance has been impacted and consider reducing your max_power or your max_threads (403)
DEBUG | 2023-03-03 01:00:15.799604 | worker.jobs.poppers:horde_pop:39 - Job pop took 1.176818 (node: kai:7002:4.1.0)
WARNING | 2023-03-03 01:00:15.800137 | worker.jobs.poppers:horde_pop:70 - Maintenance mode activated because worker is dropping too many jobs.Please investigate if your performance has been impacted and consider reducing your max_power or your max_threads (403)
DEBUG | 2023-03-03 01:00:18.975121 | worker.jobs.poppers:horde_pop:39 - Job pop took 1.15014 (node: stablehorde:7001:4.1.0)
WARNING | 2023-03-03 01:00:18.975121 | worker.jobs.poppers:horde_pop:70 - Maintenance mode activated because worker is dropping too many jobs.Please investigate if your performance has been impacted and consider reducing your max_power or your max_threads (403)
DEBUG | 2023-03-03 01:00:22.196573 | worker.jobs.poppers:horde_pop:39 - Job pop took 1.196226 (node: gremlin:7001:4.1.0)```
Local worker model statistics
Now that logging has a machine readable output of model usage, would it be possible to add a small script to process the log and output interesting statistics?
Output stats for the entire log or for the current day perhaps. For example:
$ python model-stats.py --today
1. Deliberate 22.5% (1742)
2. Realistic Vision 19.1% (1475)
3. Liberty 16.5% (1279)
4. Dreamshaper 8.2% (635)
5. Anything Diffusion 7.5% (582)
6. HASDX 4.6% (358)
7. Dreamlike Photoreal 4.1% (321)
8. Protogen Infinity 4.1% (317)
9. ProtoGen 2.7% (207)
10. Hassanblend 2.3% (175)
11. Midjourney Diffusion 1.4% (105)
12. Zeipher Female Model 1.3% (100)
13. Anygen 1.1% (85)
14. Dreamlike Diffusion 0.9% (70)
15. Robo-Diffusion 0.9% (69)
16. stable_diffusion 0.5% (39)
17. AIO Pixel Art 0.5% (39)
18. Epic Diffusion 0.5% (35)
19. Concept Sheet 0.4% (33)
20. Microworlds 0.3% (21)
21. App Icon Diffusion 0.3% (21)
22. Dungeons and Diffusion 0.2% (17)
23. pix2pix 0.1% (8)
24. Voxel Art Diffusion 0.1% (4)
25. RPG 0.0% (2)
ViT-L/14 jobs failing due to TypeError
Worker is getting forced into maintenance mode due to repeated ViT-L/14 failures.
ERROR | 2023-03-09 18:27:31.384936 | worker.jobs.stable_diffusion:start_job:353 - Something went wrong when processing request. Please check your trace.log file for the full stack trace. Payload: {'prompt': 'Cortana, full body, face, Big ass, Big breast, anus', 'height': 512, 'width': 512, 'seed': '282081520', 'tiling': False, 'n_iter': 1, 'batch_size': 1, 'save_individual_images': False, 'save_grid': False, 'ddim_steps': 30, 'sampler_name': 'k_euler_karras', 'cfg_scale': 7.0, 'hires_fix': False, 'request_type': 'txt2img', 'model': 'ViT-L/14'}
TRACE | 2023-03-09 18:27:31.387937 | worker.jobs.stable_diffusion:start_job:359 - Traceback (most recent call last):
File "\AI-Horde-Worker\worker\jobs\stable_diffusion.py", line 347, in start_job
generator.generate(**gen_payload)
File "\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\util\cast.py", line 29, in wrap
return amp.autocast(device_type="cuda", dtype=dtype)(no_grad()(func))(*args, **kwargs)
File "\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\amp\autocast_mode.py", line 14, in decorate_autocast
return func(*args, **kwargs)
File "\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\stable_diffusion\compvis.py", line 204, in generate
with model_context as model:
File "\AI-Horde-Worker\conda\envs\windows\lib\contextlib.py", line 135, in __enter__
return next(self.gen)
File "\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\util\voodoo.py", line 109, in load_from_plasma
with open(ref, "rb") as cache:
TypeError: expected str, bytes or os.PathLike object, not CLIP
Don't try to reload the model database
In issue #45 there is an attempt to reload the model database if an unknown model is detected. This should be reverted. See PR below which does just that.
- The previous code, although harmless, is ineffective and does nothing useful.
- The nataili library currently provides no support for refreshing the model database once it's been instantiated. Calls into
compvis.init()to reload the database don't propagate back up to the "super.py" aggregate class and it starts to get messy from there.
This should be reverted until nataili provides a nice simple method call to refresh the model database.
Error caught was: No module named 'triton'
Im newbie here, just install this in mint linux = ubuntu 22.04 (Kernel Linux 5.15.0-67-generic x86_64)
/runtime.sh pip list | grep nataili > nataili 0.2.9016
Error message shown below, is it correct?
A matching Triton is not available, some optimizations will not be enabled.
Error caught was: No module named 'triton'
Error when creating image: can not identify image file
TLDR; This is because the files don't exist to download on cloudflare.
The image byte steam fails to decode sometimes because, if you print the image byte steam in these cases, it is not an image, it's a string saying,
b'<?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message></Error>'"
The attempt to download the image didn't result in image data it resulted in that XML string, so it can not be decoded. Presumably this XML is from cloudflare saying it can't find the file to download.
Display a periodic "approximate kudos per hour" worker output
Perhaps once a minute or so we could print the estimated kudos per hour the worker is generating.
[Suggestion] PNG Info -Interrogation mode
Add a new form to the interrogation worker that uses piexif to retrieve the exif tags from the source image provided
Logging Refactor
Public interface
-
logger.init() - logger.trace()
- logger.debug()
- logger.info()
- logger.warning()
- logger.success()
- logger.error()
- logger.critical()
i.e. revert to the standard python logging levels.
Other features
- Preserve the existing file and terminal formats.
- Ability to override any other log handlers that our dependencies may place
- Ability to capture stray print() statements (stderr/stdout) and redirect to logging or suppress
- Set verbosity (to the terminal)
Add method to serve the top X models from last day
Add a way to override models_to_load with serve_daily_top_models which will take as an argument an integer (disabled at -1) which will use the horde's daily top models endpoint to compile the list of models to load.
Request: OpenSYCL/DPC++ support for multiple architectures
Requesting OpenSYCL support (Or DPC++) to be able to have jobs on Intel dGPUs, iGPUs, AMD GPUs, CUDA GPUs, and more simultaneously: or working together to share resources.
Inpainting fixes for v0.2
To do list for inpainting:
- Allow multiple models to access the inpainting pipe
- split the prompt and pass the negative prompt to the pipe
"Highres fix" for Stable Horde?
Can this be implemented somehow for client and worker? (Two-stage generation when img2img is used with the same prompt for upscale to 1024p without loss of consistency)
If new models are added to the database the worker errors
Model name requested Abyss OrangeMix in bridgeData is unknown to us. Please check your configuration. Aborting!
Model name requested Anything v3 in bridgeData is unknown to us. Please check your configuration. Aborting!
Model name requested Woop-Woop Photo in bridgeData is unknown to us. Please check your configuration. Aborting!
The worker doesn't seem to reload the remote model database.
pix2pix jobs failing due to tensor size mismatch
Add generation parameters to generated images as exif
[Feature Request] I request the ability to install pip packages on a different hard drive when updating runtime
Because my storage drive is too small.
M1 Mac Support
Can my mac be an AI Horde Worker?
ControlNet causing worker to be put into maintenance?
In my trace.log, I've seen it affect the following:
Models: Anything, Anything v3, Openniji, Poison, Counterfeit
CN: canny, fakescribbles
ERROR | 2023-03-04 02:01:22.165279 | worker.jobs.stable_diffusion:start_job:329 - Something went wrong when processing request. Please check your trace.log file for the full stack trace. Payload: {'prompt': 'Cute girl, blue hair.', 'height': 1024, 'width': 640, 'seed': '808565739', 'tiling': False, 'n_iter': 1, 'batch_size': 1, 'save_individual_images': False, 'save_grid': False, 'ddim_steps': 30, 'sampler_name': 'k_euler_a_karras', 'cfg_scale': 7.5, 'control_type': 'fakescribbles', 'init_as_control': True, 'init_img': <PIL.Image.Image image mode=RGB size=683x1024 at 0x20D6C9CDC30>, 'request_type': 'img2img', 'model': 'Counterfeit'}
TRACE | 2023-03-04 02:01:22.169280 | worker.jobs.stable_diffusion:start_job:335 - Traceback (most recent call last):
File "C:\Users\Airic\AI-Horde-Worker\worker\jobs\stable_diffusion.py", line 323, in start_job
generator.generate(**gen_payload)
File "C:\Users\Airic\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\util\cast.py", line 29, in wrap
return amp.autocast(device_type="cuda", dtype=dtype)(no_grad()(func))(*args, **kwargs)
File "C:\Users\Airic\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\amp\autocast_mode.py", line 14, in decorate_autocast
return func(*args, **kwargs)
File "C:\Users\Airic\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "C:\Users\Airic\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\stable_diffusion\compvis.py", line 851, in generate
samples_ddim, _ = sampler.sample(
File "C:\Users\Airic\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "C:\Users\Airic\AI-Horde-Worker\conda\envs\windows\lib\site-packages\ldm\models\diffusion\ddim.py", line 105, in sample
samples, intermediates = self.ddim_sampling(conditioning, size,
File "C:\Users\Airic\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "C:\Users\Airic\AI-Horde-Worker\conda\envs\windows\lib\site-packages\ldm\models\diffusion\ddim.py", line 165, in ddim_sampling
outs = self.p_sample_ddim(img, cond, ts, index=index, use_original_steps=ddim_use_original_steps,
File "C:\Users\Airic\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "C:\Users\Airic\AI-Horde-Worker\conda\envs\windows\lib\site-packages\ldm\models\diffusion\ddim.py", line 213, in p_sample_ddim
model_uncond, model_t = self.model.apply_model(x_in, t_in, c_in).chunk(2)
File "C:\Users\Airic\AI-Horde-Worker\conda\envs\windows\lib\site-packages\cldm\cldm.py", line 333, in apply_model
control = self.control_model(x=x_noisy, hint=cond_hint, timesteps=t, context=cond_txt)
File "C:\Users\Airic\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\Airic\AI-Horde-Worker\conda\envs\windows\lib\site-packages\cldm\cldm.py", line 295, in forward
h += guided_hint
RuntimeError: The size of tensor a (85) must match the size of tensor b (86) at non-singleton dimension 3
WebUI Improvements
The Good
- Has all latest config values
- Opens automatically in the web browser
- Handles magic constants like, "all models", "top 5", etc
- Future proof: Does not remove config file items it doesn't understand
- Everything has labels and descriptions
- Descriptions all mention the actual underlying technical config file item name
-
--sharecommand line option to get a Gradio public URL -
--no-browsercommand line option to NOT open in browser -
--lancommand line option to allow access on the local network - Can exit the application now with Ctrl+C
- Display max_power as Max Image Size and show 512x512, 768x768, etc
- Re-implement functionality to enable/disable maintenance mode.
- Support editing config in a text editor and the webui at the same time without losing any config.
The Bad
I removed the automatic worker information display. I'll add this back later.
GUI Reorganisation
The natali library is out of date. Please run update-runtime to update your environment.
Referencing #88
Posting here because I saw two Linux users in discord having this issue
ERROR | 2023-03-13 13:39:37.060854 | main:check_dependencies:54 - The natali library is out of date. Please run update-runtime to update your environment.
$ ./runtime.sh pip list | grep nataili
nataili 0.2.9014
Output of runtime.sh python:
import importlib.metadata
import packaging.version
version = importlib.metadata.version("nataili")
packaging.version.parse(version)
<Version('0.2.9014')>
update-runtime log provided by vadi
(base) vadi@barbar:/media/vadi/SSDer/Programs/AI-H-W$ git pull
Already up to date.
(base) vadi@barbar:/media/vadi/SSDer/Programs/AI-H-W$ git status
On branch main
Your branch is up to date with 'origin/main'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: bridge-webui.sh
no changes added to commit (use "git add" and/or "git commit -a")
(base) vadi@barbar:/media/vadi/SSDer/Programs/AI-H-W$ ./update-runtime.sh
bin/micromamba
__
__ ______ ___ ____ _____ ___ / /_ ____ _
/ / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
/ /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
/ .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
/_/
pkgs/main/linux-64 No change
nvidia/linux-64 No change
nvidia/noarch No change
pkgs/r/noarch No change
pkgs/r/linux-64 No change
pkgs/main/noarch No change
conda-forge/noarch 11.5MB @ 4.3MB/s 2.8s
conda-forge/linux-64 30.2MB @ 4.6MB/s 6.9s
Transaction
Prefix: /media/vadi/SSDer/Programs/AI-H-W/conda/envs/linux
Updating specs:
- cudatoolkit==11.7.0
- git
- pip
- python==3.10
Package Version Build Channel Size
─────────────────────────────────────────────────────────────────────────────────────────
Install:
─────────────────────────────────────────────────────────────────────────────────────────
+ _libgcc_mutex 0.1 conda_forge conda-forge/linux-64 Cached
+ _openmp_mutex 4.5 2_gnu conda-forge/linux-64 Cached
+ bzip2 1.0.8 h7f98852_4 conda-forge/linux-64 Cached
+ c-ares 1.18.1 h7f98852_0 conda-forge/linux-64 Cached
+ ca-certificates 2022.12.7 ha878542_0 conda-forge/linux-64 Cached
+ cudatoolkit 11.7.0 hd8887f6_10 nvidia/linux-64 Cached
+ curl 7.88.1 hdc1c0ab_0 conda-forge/linux-64 Cached
+ expat 2.5.0 h27087fc_0 conda-forge/linux-64 Cached
+ gettext 0.21.1 h27087fc_0 conda-forge/linux-64 Cached
+ git 2.39.2 pl5321h693f4a3_0 conda-forge/linux-64 Cached
+ keyutils 1.6.1 h166bdaf_0 conda-forge/linux-64 Cached
+ krb5 1.20.1 h81ceb04_0 conda-forge/linux-64 Cached
+ ld_impl_linux-64 2.40 h41732ed_0 conda-forge/linux-64 Cached
+ libcurl 7.88.1 hdc1c0ab_0 conda-forge/linux-64 Cached
+ libedit 3.1.20191231 he28a2e2_2 conda-forge/linux-64 Cached
+ libev 4.33 h516909a_1 conda-forge/linux-64 Cached
+ libffi 3.4.2 h7f98852_5 conda-forge/linux-64 Cached
+ libgcc-ng 12.2.0 h65d4601_19 conda-forge/linux-64 Cached
+ libgomp 12.2.0 h65d4601_19 conda-forge/linux-64 Cached
+ libiconv 1.17 h166bdaf_0 conda-forge/linux-64 Cached
+ libnghttp2 1.52.0 h61bc06f_0 conda-forge/linux-64 Cached
+ libnsl 2.0.0 h7f98852_0 conda-forge/linux-64 Cached
+ libsqlite 3.40.0 h753d276_0 conda-forge/linux-64 Cached
+ libssh2 1.10.0 hf14f497_3 conda-forge/linux-64 Cached
+ libstdcxx-ng 12.2.0 h46fd767_19 conda-forge/linux-64 Cached
+ libuuid 2.32.1 h7f98852_1000 conda-forge/linux-64 Cached
+ libzlib 1.2.13 h166bdaf_4 conda-forge/linux-64 Cached
+ ncurses 6.3 h27087fc_1 conda-forge/linux-64 Cached
+ openssl 3.0.8 h0b41bf4_0 conda-forge/linux-64 Cached
+ pcre2 10.40 hc3806b6_0 conda-forge/linux-64 Cached
+ perl 5.32.1 2_h7f98852_perl5 conda-forge/linux-64 Cached
+ pip 23.0.1 pyhd8ed1ab_0 conda-forge/noarch Cached
+ python 3.10.0 h543edf9_3_cpython conda-forge/linux-64 Cached
+ readline 8.1.2 h0f457ee_0 conda-forge/linux-64 Cached
+ setuptools 67.6.0 pyhd8ed1ab_0 conda-forge/noarch Cached
+ sqlite 3.40.0 h4ff8645_0 conda-forge/linux-64 Cached
+ tk 8.6.12 h27826a3_0 conda-forge/linux-64 Cached
+ tzdata 2022g h191b570_0 conda-forge/noarch Cached
+ wheel 0.38.4 pyhd8ed1ab_0 conda-forge/noarch Cached
+ xz 5.2.6 h166bdaf_0 conda-forge/linux-64 Cached
Summary:
Install: 40 packages
Total download: 0 B
─────────────────────────────────────────────────────────────────────────────────────────
Transaction starting
Linking ca-certificates-2022.12.7-ha878542_0
Linking _libgcc_mutex-0.1-conda_forge
Linking ld_impl_linux-64-2.40-h41732ed_0
Linking libstdcxx-ng-12.2.0-h46fd767_19
Linking libgomp-12.2.0-h65d4601_19
Linking _openmp_mutex-4.5-2_gnu
Linking libgcc-ng-12.2.0-h65d4601_19
Linking libuuid-2.32.1-h7f98852_1000
Linking libev-4.33-h516909a_1
Linking c-ares-1.18.1-h7f98852_0
Linking xz-5.2.6-h166bdaf_0
Linking ncurses-6.3-h27087fc_1
Linking libnsl-2.0.0-h7f98852_0
Linking libffi-3.4.2-h7f98852_5
Linking bzip2-1.0.8-h7f98852_4
Linking gettext-0.21.1-h27087fc_0
Linking keyutils-1.6.1-h166bdaf_0
Linking openssl-3.0.8-h0b41bf4_0
Linking libzlib-1.2.13-h166bdaf_4
Linking libiconv-1.17-h166bdaf_0
Linking expat-2.5.0-h27087fc_0
Linking libedit-3.1.20191231-he28a2e2_2
Linking readline-8.1.2-h0f457ee_0
Linking perl-5.32.1-2_h7f98852_perl5
Linking libnghttp2-1.52.0-h61bc06f_0
Linking tk-8.6.12-h27826a3_0
Linking libsqlite-3.40.0-h753d276_0
Linking libssh2-1.10.0-hf14f497_3
Linking pcre2-10.40-hc3806b6_0
Linking krb5-1.20.1-h81ceb04_0
Linking sqlite-3.40.0-h4ff8645_0
Linking libcurl-7.88.1-hdc1c0ab_0
Linking curl-7.88.1-hdc1c0ab_0
Linking git-2.39.2-pl5321h693f4a3_0
Linking tzdata-2022g-h191b570_0
Linking cudatoolkit-11.7.0-hd8887f6_10
By downloading and using the CUDA Toolkit conda packages, you accept the terms and conditions of the CUDA End User License Agreement (EULA): https://docs.nvidia.com/cuda/eula/index.html
Linking python-3.10.0-h543edf9_3_cpython
Linking pip-23.0.1-pyhd8ed1ab_0
Linking setuptools-67.6.0-pyhd8ed1ab_0
Linking wheel-0.38.4-pyhd8ed1ab_0
Transaction finished
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cu117
Collecting git+https://github.com/db0/nataili.git (from -r requirements.txt (line 2))
Cloning https://github.com/db0/nataili.git to /tmp/pip-req-build-g3piofke
Running command git clone --filter=blob:none --quiet https://github.com/db0/nataili.git /tmp/pip-req-build-g3piofke
Resolved https://github.com/db0/nataili.git to commit 4750ce1ecc01708a58730ce66af6d871c1c7c406
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... done
Collecting xformers
Using cached xformers-0.0.16-cp310-cp310-manylinux2014_x86_64.whl (50.9 MB)
Collecting gradio
Using cached gradio-3.20.1-py3-none-any.whl (14.3 MB)
Collecting pyyaml
Using cached PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB)
Collecting unidecode
Using cached Unidecode-1.3.6-py3-none-any.whl (235 kB)
Collecting tqdm
Using cached tqdm-4.65.0-py3-none-any.whl (77 kB)
Collecting open-clip-torch
Using cached open_clip_torch-2.16.0-py3-none-any.whl (1.5 MB)
Collecting einops
Using cached einops-0.6.0-py3-none-any.whl (41 kB)
Collecting accelerate
Using cached accelerate-0.17.0-py3-none-any.whl (212 kB)
Collecting opencv-python-headless
Using cached opencv_python_headless-4.7.0.72-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (49.2 MB)
Collecting pydantic
Using cached pydantic-1.10.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
Collecting loguru
Using cached loguru-0.6.0-py3-none-any.whl (58 kB)
Collecting webdataset
Using cached webdataset-0.2.39-py3-none-any.whl (51 kB)
Collecting basicsr
Using cached basicsr-1.4.2-py3-none-any.whl
Collecting k-diffusion
Using cached k_diffusion-0.0.14-py3-none-any.whl (25 kB)
Collecting omegaconf
Using cached omegaconf-2.3.0-py3-none-any.whl (79 kB)
Collecting prettytable
Using cached prettytable-3.6.0-py3-none-any.whl (27 kB)
Collecting safetensors
Using cached safetensors-0.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
Collecting python-slugify
Using cached python_slugify-8.0.1-py2.py3-none-any.whl (9.7 kB)
Collecting ray
Using cached ray-2.3.0-cp310-cp310-manylinux2014_x86_64.whl (58.5 MB)
Collecting facexlib
Using cached facexlib-0.2.5-py3-none-any.whl (59 kB)
Collecting transformers
Using cached transformers-4.26.1-py3-none-any.whl (6.3 MB)
Collecting diffusers
Using cached diffusers-0.14.0-py3-none-any.whl (737 kB)
Collecting torch
Using cached https://download.pytorch.org/whl/cu117/torch-1.13.1%2Bcu117-cp310-cp310-linux_x86_64.whl (1801.8 MB)
Collecting taming-transformers-rom1504
Using cached taming_transformers_rom1504-0.0.6-py3-none-any.whl (51 kB)
Collecting gfpgan
Using cached gfpgan-1.3.8-py3-none-any.whl (52 kB)
Collecting bitsandbytes
Using cached bitsandbytes-0.37.1-py3-none-any.whl (76.3 MB)
Collecting kornia
Using cached kornia-0.6.10-py2.py3-none-any.whl (612 kB)
Collecting pytorch-lightning
Using cached pytorch_lightning-1.9.4-py3-none-any.whl (827 kB)
Collecting realesrgan
Using cached realesrgan-0.3.0-py3-none-any.whl (26 kB)
Collecting fairscale
Using cached fairscale-0.4.13-py3-none-any.whl
Collecting pyre-extensions==0.0.23
Using cached pyre_extensions-0.0.23-py3-none-any.whl (11 kB)
Collecting numpy
Using cached numpy-1.24.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
Collecting typing-extensions
Using cached typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Collecting typing-inspect
Using cached typing_inspect-0.8.0-py3-none-any.whl (8.7 kB)
Collecting aiohttp
Using cached aiohttp-3.8.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB)
Collecting orjson
Using cached orjson-3.8.7-cp310-cp310-manylinux_2_28_x86_64.whl (140 kB)
Collecting mdit-py-plugins<=0.3.3
Using cached mdit_py_plugins-0.3.3-py3-none-any.whl (50 kB)
Collecting python-multipart
Using cached python_multipart-0.0.6-py3-none-any.whl (45 kB)
Collecting httpx
Using cached httpx-0.23.3-py3-none-any.whl (71 kB)
Collecting ffmpy
Using cached ffmpy-0.3.0-py3-none-any.whl
Collecting matplotlib
Using cached matplotlib-3.7.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.6 MB)
Collecting pandas
Using cached pandas-1.5.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.1 MB)
Collecting altair>=4.2.0
Using cached altair-4.2.2-py3-none-any.whl (813 kB)
Collecting requests
Using cached requests-2.28.2-py3-none-any.whl (62 kB)
Collecting uvicorn
Using cached uvicorn-0.21.0-py3-none-any.whl (57 kB)
Collecting pydub
Using cached pydub-0.25.1-py2.py3-none-any.whl (32 kB)
Collecting markdown-it-py[linkify]>=2.0.0
Using cached markdown_it_py-2.2.0-py3-none-any.whl (84 kB)
Collecting fsspec
Using cached fsspec-2023.3.0-py3-none-any.whl (145 kB)
Collecting markupsafe
Using cached MarkupSafe-2.1.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Collecting jinja2
Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting pillow
Using cached Pillow-9.4.0-cp310-cp310-manylinux_2_28_x86_64.whl (3.4 MB)
Collecting pycryptodome
Using cached pycryptodome-3.17-cp35-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB)
Collecting fastapi
Using cached fastapi-0.94.0-py3-none-any.whl (56 kB)
Collecting aiofiles
Using cached aiofiles-23.1.0-py3-none-any.whl (14 kB)
Collecting websockets>=10.0
Using cached websockets-10.4-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (106 kB)
Collecting entrypoints
Using cached entrypoints-0.4-py3-none-any.whl (5.3 kB)
Collecting jsonschema>=3.0
Using cached jsonschema-4.17.3-py3-none-any.whl (90 kB)
Collecting toolz
Using cached toolz-0.12.0-py3-none-any.whl (55 kB)
Collecting mdurl~=0.1
Using cached mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Collecting linkify-it-py<3,>=1
Using cached linkify_it_py-2.0.0-py3-none-any.whl (19 kB)
Collecting pytz>=2020.1
Using cached pytz-2022.7.1-py2.py3-none-any.whl (499 kB)
Requirement already satisfied: python-dateutil>=2.8.1 in /home/vadi/.local/lib/python3.10/site-packages (from pandas->gradio->-r requirements.txt (line 4)) (2.8.2)
Requirement already satisfied: packaging>=20.0 in /home/vadi/.local/lib/python3.10/site-packages (from accelerate->nataili==0.2.9014->-r requirements.txt (line 2)) (23.0)
Requirement already satisfied: psutil in /home/vadi/.local/lib/python3.10/site-packages (from accelerate->nataili==0.2.9014->-r requirements.txt (line 2)) (5.9.4)
Collecting attrs>=17.3.0
Using cached attrs-22.2.0-py3-none-any.whl (60 kB)
Collecting aiosignal>=1.1.2
Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting multidict<7.0,>=4.5
Using cached multidict-6.0.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (114 kB)
Collecting frozenlist>=1.1.1
Using cached frozenlist-1.3.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (149 kB)
Collecting yarl<2.0,>=1.0
Using cached yarl-1.8.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (264 kB)
Collecting charset-normalizer<4.0,>=2.0
Using cached charset_normalizer-3.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (199 kB)
Collecting async-timeout<5.0,>=4.0.0a3
Using cached async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
Collecting addict
Using cached addict-2.4.0-py3-none-any.whl (3.8 kB)
Collecting opencv-python
Using cached opencv_python-4.7.0.72-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.8 MB)
Collecting tb-nightly
Downloading tb_nightly-2.13.0a20230313-py3-none-any.whl (5.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.6/5.6 MB 14.7 MB/s eta 0:00:00
Collecting scipy
Using cached scipy-1.10.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.4 MB)
Collecting future
Using cached future-0.18.3-py3-none-any.whl
Collecting lmdb
Using cached lmdb-1.4.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (306 kB)
Collecting yapf
Using cached yapf-0.32.0-py2.py3-none-any.whl (190 kB)
Collecting torchvision
Using cached https://download.pytorch.org/whl/cu117/torchvision-0.14.1%2Bcu117-cp310-cp310-linux_x86_64.whl (24.3 MB)
Collecting scikit-image
Using cached scikit_image-0.20.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.2 MB)
Collecting huggingface-hub>=0.10.0
Using cached huggingface_hub-0.13.1-py3-none-any.whl (199 kB)
Collecting filelock
Using cached filelock-3.9.0-py3-none-any.whl (9.7 kB)
Collecting regex!=2019.12.17
Using cached regex-2022.10.31-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (770 kB)
Collecting importlib-metadata
Using cached importlib_metadata-6.0.0-py3-none-any.whl (21 kB)
Collecting filterpy
Using cached filterpy-1.4.5-py3-none-any.whl
Collecting numba
Using cached numba-0.56.4-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.5 MB)
Collecting starlette<0.27.0,>=0.26.0
Using cached starlette-0.26.0.post1-py3-none-any.whl (66 kB)
Collecting rfc3986[idna2008]<2,>=1.3
Using cached rfc3986-1.5.0-py2.py3-none-any.whl (31 kB)
Collecting httpcore<0.17.0,>=0.15.0
Using cached httpcore-0.16.3-py3-none-any.whl (69 kB)
Collecting certifi
Using cached https://download.pytorch.org/whl/certifi-2022.12.7-py3-none-any.whl (155 kB)
Collecting sniffio
Using cached sniffio-1.3.0-py3-none-any.whl (10 kB)
Collecting clean-fid
Using cached clean_fid-0.1.35-py3-none-any.whl (26 kB)
Collecting torchdiffeq
Using cached torchdiffeq-0.2.3-py3-none-any.whl (31 kB)
Collecting jsonmerge
Using cached jsonmerge-1.9.0-py3-none-any.whl
Collecting clip-anytorch
Using cached clip_anytorch-2.5.2-py3-none-any.whl (1.4 MB)
Collecting resize-right
Using cached resize_right-0.0.2-py3-none-any.whl (8.9 kB)
Collecting wandb
Using cached wandb-0.13.11-py3-none-any.whl (2.0 MB)
Collecting torchsde
Using cached torchsde-0.2.5-py3-none-any.whl (59 kB)
Collecting fonttools>=4.22.0
Using cached fonttools-4.39.0-py3-none-any.whl (1.0 MB)
Collecting contourpy>=1.0.1
Using cached contourpy-1.0.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (300 kB)
Collecting pyparsing>=2.3.1
Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting kiwisolver>=1.0.1
Using cached kiwisolver-1.4.4-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB)
Collecting cycler>=0.10
Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting antlr4-python3-runtime==4.9.*
Using cached antlr4_python3_runtime-4.9.3-py3-none-any.whl
Collecting ftfy
Using cached ftfy-6.1.1-py3-none-any.whl (53 kB)
Collecting protobuf<4
Using cached protobuf-3.20.3-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)
Collecting sentencepiece
Using cached sentencepiece-0.1.97-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
Collecting timm
Using cached timm-0.6.12-py3-none-any.whl (549 kB)
Requirement already satisfied: wcwidth in /home/vadi/.local/lib/python3.10/site-packages (from prettytable->nataili==0.2.9014->-r requirements.txt (line 2)) (0.2.6)
Collecting text-unidecode>=1.3
Using cached text_unidecode-1.3-py2.py3-none-any.whl (78 kB)
Collecting lightning-utilities>=0.6.0.post0
Using cached lightning_utilities-0.8.0-py3-none-any.whl (20 kB)
Collecting torchmetrics>=0.7.0
Using cached torchmetrics-0.11.4-py3-none-any.whl (519 kB)
Collecting grpcio>=1.42.0
Using cached grpcio-1.51.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB)
Collecting msgpack<2.0.0,>=1.0.0
Using cached msgpack-1.0.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (316 kB)
Collecting click>=7.0
Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting virtualenv>=20.0.24
Using cached virtualenv-20.21.0-py3-none-any.whl (8.7 MB)
Collecting idna<4,>=2.5
Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
Collecting tokenizers!=0.11.3,<0.14,>=0.11.1
Using cached tokenizers-0.13.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.6 MB)
Collecting h11>=0.8
Using cached h11-0.14.0-py3-none-any.whl (58 kB)
Collecting braceexpand
Using cached braceexpand-0.1.7-py2.py3-none-any.whl (5.9 kB)
Collecting anyio<5.0,>=3.0
Using cached anyio-3.6.2-py3-none-any.whl (80 kB)
Collecting pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0
Using cached pyrsistent-0.19.3-py3-none-any.whl (57 kB)
Collecting uc-micro-py
Using cached uc_micro_py-1.0.1-py3-none-any.whl (6.2 kB)
Requirement already satisfied: six>=1.5 in /home/vadi/.local/lib/python3.10/site-packages (from python-dateutil>=2.8.1->pandas->gradio->-r requirements.txt (line 4)) (1.16.0)
Requirement already satisfied: platformdirs<4,>=2.4 in /home/vadi/.local/lib/python3.10/site-packages (from virtualenv>=20.0.24->ray->nataili==0.2.9014->-r requirements.txt (line 2)) (2.6.2)
Collecting distlib<1,>=0.3.6
Using cached distlib-0.3.6-py2.py3-none-any.whl (468 kB)
Collecting zipp>=0.5
Using cached zipp-3.15.0-py3-none-any.whl (6.8 kB)
Collecting numpy
Using cached numpy-1.23.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
Collecting llvmlite<0.40,>=0.39.0dev0
Using cached llvmlite-0.39.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.6 MB)
Requirement already satisfied: setuptools in ./conda/envs/linux/lib/python3.10/site-packages (from numba->facexlib->nataili==0.2.9014->-r requirements.txt (line 2)) (67.6.0)
Collecting imageio>=2.4.1
Using cached imageio-2.26.0-py3-none-any.whl (3.4 MB)
Collecting PyWavelets>=1.1.1
Using cached PyWavelets-1.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.8 MB)
Collecting networkx>=2.8
Using cached networkx-3.0-py3-none-any.whl (2.0 MB)
Collecting tifffile>=2019.7.26
Using cached tifffile-2023.2.28-py3-none-any.whl (216 kB)
Collecting lazy_loader>=0.1
Using cached lazy_loader-0.1-py3-none-any.whl (8.6 kB)
Collecting tensorboard-data-server<0.8.0,>=0.7.0
Using cached tensorboard_data_server-0.7.0-py3-none-manylinux2014_x86_64.whl (6.6 MB)
Collecting tensorboard-plugin-wit>=1.6.0
Using cached tensorboard_plugin_wit-1.8.1-py3-none-any.whl (781 kB)
Collecting absl-py>=0.4
Using cached absl_py-1.4.0-py3-none-any.whl (126 kB)
Collecting google-auth<3,>=1.6.3
Using cached google_auth-2.16.2-py2.py3-none-any.whl (177 kB)
Collecting markdown>=2.6.8
Using cached Markdown-3.4.1-py3-none-any.whl (93 kB)
Collecting google-auth-oauthlib<0.5,>=0.4.1
Using cached google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)
Collecting werkzeug>=1.0.1
Using cached Werkzeug-2.2.3-py3-none-any.whl (233 kB)
Requirement already satisfied: wheel>=0.26 in ./conda/envs/linux/lib/python3.10/site-packages (from tb-nightly->basicsr->nataili==0.2.9014->-r requirements.txt (line 2)) (0.38.4)
Collecting boltons>=20.2.1
Using cached boltons-23.0.0-py2.py3-none-any.whl (194 kB)
Collecting trampoline>=0.1.2
Using cached trampoline-0.1.2-py3-none-any.whl (5.2 kB)
Collecting mypy-extensions>=0.3.0
Using cached mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)
Collecting docker-pycreds>=0.4.0
Using cached docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)
Collecting setproctitle
Using cached setproctitle-1.3.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (30 kB)
Collecting appdirs>=1.4.3
Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting pathtools
Using cached pathtools-0.1.2-py3-none-any.whl
Collecting sentry-sdk>=1.0.0
Using cached sentry_sdk-1.16.0-py2.py3-none-any.whl (184 kB)
Collecting GitPython!=3.1.29,>=1.0.0
Using cached GitPython-3.1.31-py3-none-any.whl (184 kB)
Collecting gitdb<5,>=4.0.1
Using cached gitdb-4.0.10-py3-none-any.whl (62 kB)
Collecting pyasn1-modules>=0.2.1
Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting rsa<5,>=3.1.4
Using cached rsa-4.9-py3-none-any.whl (34 kB)
Collecting cachetools<6.0,>=2.0.0
Using cached cachetools-5.3.0-py3-none-any.whl (9.3 kB)
Collecting requests-oauthlib>=0.7.0
Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting smmap<6,>=3.0.1
Using cached smmap-5.0.0-py3-none-any.whl (24 kB)
Collecting pyasn1<0.5.0,>=0.4.6
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting oauthlib>=3.0.0
Using cached oauthlib-3.2.2-py3-none-any.whl (151 kB)
Building wheels for collected packages: nataili
Building wheel for nataili (pyproject.toml) ... done
Created wheel for nataili: filename=nataili-0.2.9014-py3-none-any.whl size=1184420 sha256=f9e18f5528326424883702ede28400c077e5fd688934250556ca947d42f06123
Stored in directory: /tmp/pip-ephem-wheel-cache-bbjiz186/wheels/27/a8/72/f87a0e0f8d6ee6fe691bd767e88a418cf805e4626f66533f49
Successfully built nataili
Installing collected packages: yapf, trampoline, tokenizers, text-unidecode, tensorboard-plugin-wit, sentencepiece, safetensors, rfc3986, resize-right, pytz, pydub, pyasn1, pathtools, msgpack, lmdb, ffmpy, distlib, braceexpand, boltons, bitsandbytes, appdirs, antlr4-python3-runtime, addict, zipp, websockets, urllib3, unidecode, uc-micro-py, typing-extensions, tqdm, toolz, tensorboard-data-server, sniffio, smmap, setproctitle, rsa, regex, pyyaml, python-slugify, python-multipart, pyrsistent, pyparsing, pycryptodome, pyasn1-modules, protobuf, prettytable, pillow, orjson, oauthlib, numpy, networkx, mypy-extensions, multidict, mdurl, markupsafe, markdown, loguru, llvmlite, lazy_loader, kiwisolver, idna, h11, grpcio, future, ftfy, fsspec, frozenlist, fonttools, filelock, entrypoints, einops, docker-pycreds, cycler, click, charset-normalizer, certifi, cachetools, attrs, async-timeout, aiofiles, absl-py, yarl, werkzeug, webdataset, virtualenv, uvicorn, typing-inspect, torch, tifffile, sentry-sdk, scipy, requests, PyWavelets, pydantic, pandas, opencv-python-headless, opencv-python, omegaconf, numba, markdown-it-py, linkify-it-py, lightning-utilities, jsonschema, jinja2, importlib-metadata, imageio, google-auth, gitdb, contourpy, anyio, aiosignal, torchvision, torchsde, torchmetrics, torchdiffeq, starlette, scikit-image, requests-oauthlib, ray, pyre-extensions, mdit-py-plugins, matplotlib, kornia, jsonmerge, huggingface-hub, httpcore, GitPython, fairscale, altair, aiohttp, accelerate, xformers, wandb, transformers, timm, httpx, google-auth-oauthlib, filterpy, fastapi, diffusers, clip-anytorch, clean-fid, tb-nightly, pytorch-lightning, open-clip-torch, k-diffusion, gradio, facexlib, taming-transformers-rom1504, basicsr, gfpgan, realesrgan, nataili
Successfully installed GitPython-3.1.31 PyWavelets-1.4.1 absl-py-1.4.0 accelerate-0.17.0 addict-2.4.0 aiofiles-23.1.0 aiohttp-3.8.4 aiosignal-1.3.1 altair-4.2.2 antlr4-python3-runtime-4.9.3 anyio-3.6.2 appdirs-1.4.4 async-timeout-4.0.2 attrs-22.2.0 basicsr-1.4.2 bitsandbytes-0.37.1 boltons-23.0.0 braceexpand-0.1.7 cachetools-5.3.0 certifi-2022.12.7 charset-normalizer-3.1.0 clean-fid-0.1.35 click-8.1.3 clip-anytorch-2.5.2 contourpy-1.0.7 cycler-0.11.0 diffusers-0.14.0 distlib-0.3.6 docker-pycreds-0.4.0 einops-0.6.0 entrypoints-0.4 facexlib-0.2.5 fairscale-0.4.13 fastapi-0.94.0 ffmpy-0.3.0 filelock-3.9.0 filterpy-1.4.5 fonttools-4.39.0 frozenlist-1.3.3 fsspec-2023.3.0 ftfy-6.1.1 future-0.18.3 gfpgan-1.3.8 gitdb-4.0.10 google-auth-2.16.2 google-auth-oauthlib-0.4.6 gradio-3.20.1 grpcio-1.51.3 h11-0.14.0 httpcore-0.16.3 httpx-0.23.3 huggingface-hub-0.13.1 idna-3.4 imageio-2.26.0 importlib-metadata-6.0.0 jinja2-3.1.2 jsonmerge-1.9.0 jsonschema-4.17.3 k-diffusion-0.0.14 kiwisolver-1.4.4 kornia-0.6.10 lazy_loader-0.1 lightning-utilities-0.8.0 linkify-it-py-2.0.0 llvmlite-0.39.1 lmdb-1.4.0 loguru-0.6.0 markdown-3.4.1 markdown-it-py-2.2.0 markupsafe-2.1.2 matplotlib-3.7.1 mdit-py-plugins-0.3.3 mdurl-0.1.2 msgpack-1.0.5 multidict-6.0.4 mypy-extensions-1.0.0 nataili-0.2.9014 networkx-3.0 numba-0.56.4 numpy-1.23.5 oauthlib-3.2.2 omegaconf-2.3.0 open-clip-torch-2.16.0 opencv-python-4.7.0.72 opencv-python-headless-4.7.0.72 orjson-3.8.7 pandas-1.5.3 pathtools-0.1.2 pillow-9.4.0 prettytable-3.6.0 protobuf-3.20.3 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycryptodome-3.17 pydantic-1.10.6 pydub-0.25.1 pyparsing-3.0.9 pyre-extensions-0.0.23 pyrsistent-0.19.3 python-multipart-0.0.6 python-slugify-8.0.1 pytorch-lightning-1.9.4 pytz-2022.7.1 pyyaml-6.0 ray-2.3.0 realesrgan-0.3.0 regex-2022.10.31 requests-2.28.2 requests-oauthlib-1.3.1 resize-right-0.0.2 rfc3986-1.5.0 rsa-4.9 safetensors-0.3.0 scikit-image-0.20.0 scipy-1.10.1 sentencepiece-0.1.97 sentry-sdk-1.16.0 setproctitle-1.3.2 smmap-5.0.0 sniffio-1.3.0 starlette-0.26.0.post1 taming-transformers-rom1504-0.0.6 tb-nightly-2.13.0a20230313 tensorboard-data-server-0.7.0 tensorboard-plugin-wit-1.8.1 text-unidecode-1.3 tifffile-2023.2.28 timm-0.6.12 tokenizers-0.13.2 toolz-0.12.0 torch-1.13.1+cu117 torchdiffeq-0.2.3 torchmetrics-0.11.4 torchsde-0.2.5 torchvision-0.14.1+cu117 tqdm-4.65.0 trampoline-0.1.2 transformers-4.26.1 typing-extensions-4.5.0 typing-inspect-0.8.0 uc-micro-py-1.0.1 unidecode-1.3.6 urllib3-1.26.15 uvicorn-0.21.0 virtualenv-20.21.0 wandb-0.13.11 webdataset-0.2.39 websockets-10.4 werkzeug-2.2.3 xformers-0.0.16 yapf-0.32.0 yarl-1.8.2 zipp-3.15.0
(base) vadi@barbar:/media/vadi/SSDer/Programs/AI-H-W$ ./horde-bridge.sh --disable_voodoo
INIT | Checking | xformers optimizations
A matching Triton is not available, some optimizations will not be enabled.
Error caught was: No module named 'triton'
INIT | Loaded | xformers optimizations
INIT | OS Default | Ray temp dir'
ERROR | 2023-03-13 12:39:48.305022 | __main__:check_dependencies:54 - The nataili library is out of date. Please run update-runtime to update your environment.
(base) vadi@barbar:/media/vadi/SSDer/Programs/AI-H-W$
Job already submitted error
This is pretty rare, but here's a log extract in any case:
DEBUG | 2023-02-07 00:26:55 | worker.workers.framework:start_job:120 - New job processing
DEBUG | 2023-02-07 00:26:55 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: ProtoGen
DEBUG | 2023-02-07 00:26:55 | worker.jobs.stable_diffusion:start_job:176 - img2img (ProtoGen) request with id 66080db1-fa83-41b0-a876-2f79abbb3c29 picked up. Initiating work...
INFO | 2023-02-07 00:26:55 | worker.jobs.stable_diffusion:start_job:282 - Starting generation...
DEBUG | 2023-02-07 00:26:57 | worker.jobs.poppers:horde_pop:36 - Job pop took 1.544195
INFO | 2023-02-07 00:26:57 | nataili.inference.compvis:generate:363 - [ProtoGen] (384x960) k_euler 100 steps (1/1)
INFO | 2023-02-07 00:26:57 | worker.jobs.stable_diffusion:start_job:284 - Finished generation...
DEBUG | 2023-02-07 00:26:57 | worker.jobs.stable_diffusion:start_job:324 - post-processing done...
DEBUG | 2023-02-07 00:26:57 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG | 2023-02-07 00:26:57 | worker.workers.framework:check_running_job_status:150 - Job finished successfully in 2.266s (Total Completed: 1121)
DEBUG | 2023-02-07 00:26:57 | worker.workers.framework:start_job:120 - New job processing
DEBUG | 2023-02-07 00:26:57 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: Laolei New Berry Protogen Mix
DEBUG | 2023-02-07 00:26:57 | worker.jobs.stable_diffusion:start_job:176 - txt2img (Laolei New Berry Protogen Mix) request with id c6e3d02a-a69a-4317-8457-6dc89818a9cb picked up. Initiating work...
INFO | 2023-02-07 00:26:57 | worker.jobs.stable_diffusion:start_job:282 - Starting generation...
DEBUG | 2023-02-07 00:26:58 | worker.jobs.stable_diffusion:prepare_submit_payload:339 - R2 Upload response: <Response [200]>
DEBUG | 2023-02-07 00:26:58 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG | 2023-02-07 00:26:58 | worker.jobs.framework:submit_job:113 - Upload completed in 0.321775
WARNING | 2023-02-07 00:26:58 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 1/10)
INFO | 2023-02-07 00:26:59 | nataili.inference.compvis:generate:363 - [Laolei New Berry Protogen Mix] (768x512) k_euler 30 steps (1/1)
DEBUG | 2023-02-07 00:27:00 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG | 2023-02-07 00:27:00 | worker.jobs.framework:submit_job:113 - Upload completed in 0.183467
WARNING | 2023-02-07 00:27:00 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 2/10)
DEBUG | 2023-02-07 00:27:01 | worker.jobs.poppers:horde_pop:36 - Job pop took 3.002054
DEBUG | 2023-02-07 00:27:02 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
INFO | 2023-02-07 00:27:02 | worker.jobs.stable_diffusion:start_job:284 - Finished generation...
DEBUG | 2023-02-07 00:27:02 | worker.jobs.stable_diffusion:start_job:324 - post-processing done...
DEBUG | 2023-02-07 00:27:02 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG | 2023-02-07 00:27:02 | worker.workers.framework:check_running_job_status:150 - Job finished successfully in 4.766s (Total Completed: 1122)
DEBUG | 2023-02-07 00:27:02 | worker.workers.framework:start_job:120 - New job processing
DEBUG | 2023-02-07 00:27:02 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: Laolei New Berry Protogen Mix
DEBUG | 2023-02-07 00:27:02 | worker.jobs.stable_diffusion:start_job:176 - txt2img (Laolei New Berry Protogen Mix) request with id c7345143-5cbe-4648-9efe-b29222d20af1 picked up. Initiating work...
INFO | 2023-02-07 00:27:02 | worker.jobs.stable_diffusion:start_job:282 - Starting generation...
INFO | 2023-02-07 00:27:03 | nataili.inference.compvis:generate:363 - [Laolei New Berry Protogen Mix] (768x512) k_euler 30 steps (1/1)
DEBUG | 2023-02-07 00:27:03 | worker.jobs.stable_diffusion:prepare_submit_payload:339 - R2 Upload response: <Response [200]>
DEBUG | 2023-02-07 00:27:03 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG | 2023-02-07 00:27:03 | worker.jobs.framework:submit_job:113 - Upload completed in 0.887986
WARNING | 2023-02-07 00:27:03 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 3/10)
DEBUG | 2023-02-07 00:27:03 | worker.jobs.framework:submit_job:113 - Upload completed in 0.64281
INFO | 2023-02-07 00:27:03 | worker.jobs.framework:submit_job:137 - Submitted job with id c6e3d02a-a69a-4317-8457-6dc89818a9cb and contributed for 9.44
DEBUG | 2023-02-07 00:27:05 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG | 2023-02-07 00:27:05 | worker.jobs.framework:submit_job:113 - Upload completed in 0.268584
WARNING | 2023-02-07 00:27:05 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 4/10)
INFO | 2023-02-07 00:27:06 | worker.jobs.stable_diffusion:start_job:284 - Finished generation...
DEBUG | 2023-02-07 00:27:06 | worker.jobs.stable_diffusion:start_job:324 - post-processing done...
DEBUG | 2023-02-07 00:27:06 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG | 2023-02-07 00:27:06 | worker.jobs.stable_diffusion:prepare_submit_payload:339 - R2 Upload response: <Response [200]>
DEBUG | 2023-02-07 00:27:06 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG | 2023-02-07 00:27:07 | worker.jobs.framework:submit_job:113 - Upload completed in 0.887012
INFO | 2023-02-07 00:27:07 | worker.jobs.framework:submit_job:137 - Submitted job with id c7345143-5cbe-4648-9efe-b29222d20af1 and contributed for 9.44
DEBUG | 2023-02-07 00:27:07 | worker.jobs.poppers:horde_pop:36 - Job pop took 4.869675
DEBUG | 2023-02-07 00:27:07 | worker.workers.framework:check_running_job_status:150 - Job finished successfully in 4.922s (Total Completed: 1123)
DEBUG | 2023-02-07 00:27:07 | worker.workers.framework:start_job:120 - New job processing
DEBUG | 2023-02-07 00:27:07 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: Laolei New Berry Protogen Mix
DEBUG | 2023-02-07 00:27:07 | worker.jobs.stable_diffusion:start_job:176 - txt2img (Laolei New Berry Protogen Mix) request with id c24ecf25-2a1e-4184-880d-05a7872834ec picked up. Initiating work...
INFO | 2023-02-07 00:27:07 | worker.jobs.stable_diffusion:start_job:282 - Starting generation...
DEBUG | 2023-02-07 00:27:07 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
INFO | 2023-02-07 00:27:07 | nataili.inference.compvis:generate:363 - [Laolei New Berry Protogen Mix] (768x512) k_euler 30 steps (1/1)
DEBUG | 2023-02-07 00:27:08 | worker.jobs.framework:submit_job:113 - Upload completed in 0.660988
WARNING | 2023-02-07 00:27:08 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 5/10)
DEBUG | 2023-02-07 00:27:10 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG | 2023-02-07 00:27:11 | worker.jobs.framework:submit_job:113 - Upload completed in 0.551509
WARNING | 2023-02-07 00:27:11 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 6/10)
INFO | 2023-02-07 00:27:11 | worker.jobs.stable_diffusion:start_job:284 - Finished generation...
DEBUG | 2023-02-07 00:27:11 | worker.jobs.stable_diffusion:start_job:324 - post-processing done...
DEBUG | 2023-02-07 00:27:11 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG | 2023-02-07 00:27:11 | worker.jobs.stable_diffusion:prepare_submit_payload:339 - R2 Upload response: <Response [200]>
DEBUG | 2023-02-07 00:27:11 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG | 2023-02-07 00:27:12 | worker.jobs.poppers:horde_pop:36 - Job pop took 4.343741
DEBUG | 2023-02-07 00:27:12 | worker.workers.framework:check_running_job_status:150 - Job finished successfully in 4.390s (Total Completed: 1124)
DEBUG | 2023-02-07 00:27:12 | worker.workers.framework:start_job:120 - New job processing
DEBUG | 2023-02-07 00:27:12 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: Laolei New Berry Protogen Mix
DEBUG | 2023-02-07 00:27:12 | worker.jobs.stable_diffusion:start_job:176 - txt2img (Laolei New Berry Protogen Mix) request with id 0713a22c-67e6-425d-be40-30e075807337 picked up. Initiating work...
INFO | 2023-02-07 00:27:12 | worker.jobs.stable_diffusion:start_job:282 - Starting generation...
INFO | 2023-02-07 00:27:12 | nataili.inference.compvis:generate:363 - [Laolei New Berry Protogen Mix] (768x512) k_euler 30 steps (1/1)
DEBUG | 2023-02-07 00:27:13 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG | 2023-02-07 00:27:13 | worker.jobs.framework:submit_job:113 - Upload completed in 0.287999
WARNING | 2023-02-07 00:27:13 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 7/10)
DEBUG | 2023-02-07 00:27:15 | worker.jobs.framework:submit_job:113 - Upload completed in 3.71467
INFO | 2023-02-07 00:27:15 | worker.jobs.framework:submit_job:137 - Submitted job with id c24ecf25-2a1e-4184-880d-05a7872834ec and contributed for 9.44
DEBUG | 2023-02-07 00:27:15 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
INFO | 2023-02-07 00:27:15 | worker.jobs.stable_diffusion:start_job:284 - Finished generation...
DEBUG | 2023-02-07 00:27:15 | worker.jobs.stable_diffusion:start_job:324 - post-processing done...
DEBUG | 2023-02-07 00:27:15 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG | 2023-02-07 00:27:15 | worker.jobs.stable_diffusion:prepare_submit_payload:339 - R2 Upload response: <Response [200]>
DEBUG | 2023-02-07 00:27:15 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG | 2023-02-07 00:27:16 | worker.jobs.poppers:horde_pop:36 - Job pop took 3.883048
DEBUG | 2023-02-07 00:27:16 | worker.workers.framework:check_running_job_status:150 - Job finished successfully in 3.906s (Total Completed: 1125)
DEBUG | 2023-02-07 00:27:16 | worker.workers.framework:start_job:120 - New job processing
DEBUG | 2023-02-07 00:27:16 | worker.jobs.stable_diffusion:start_job:42 - Starting job in threadpool for model: Dreamlike Photoreal
DEBUG | 2023-02-07 00:27:16 | worker.jobs.stable_diffusion:start_job:176 - txt2img (Dreamlike Photoreal) request with id 04655444-5afc-471e-93f4-519c3943796d picked up. Initiating work...
INFO | 2023-02-07 00:27:16 | worker.jobs.stable_diffusion:start_job:282 - Starting generation...
DEBUG | 2023-02-07 00:27:16 | worker.jobs.framework:submit_job:113 - Upload completed in 0.85561
WARNING | 2023-02-07 00:27:16 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 8/10)
DEBUG | 2023-02-07 00:27:17 | worker.jobs.framework:submit_job:113 - Upload completed in 1.835789
INFO | 2023-02-07 00:27:17 | worker.jobs.framework:submit_job:137 - Submitted job with id 0713a22c-67e6-425d-be40-30e075807337 and contributed for 9.44
INFO | 2023-02-07 00:27:17 | nataili.inference.compvis:generate:363 - [Dreamlike Photoreal] (512x768) k_dpmpp_sde 50 steps (1/1)
DEBUG | 2023-02-07 00:27:18 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG | 2023-02-07 00:27:18 | worker.jobs.framework:submit_job:113 - Upload completed in 0.222212
WARNING | 2023-02-07 00:27:18 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 9/10)
DEBUG | 2023-02-07 00:27:19 | worker.jobs.poppers:horde_pop:36 - Job pop took 2.975233
DEBUG | 2023-02-07 00:27:20 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG | 2023-02-07 00:27:21 | worker.jobs.framework:submit_job:113 - Upload completed in 0.611655
WARNING | 2023-02-07 00:27:21 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 10/10)
DEBUG | 2023-02-07 00:27:23 | worker.jobs.framework:submit_job:104 - posting payload with size of 0.1 kb
DEBUG | 2023-02-07 00:27:23 | worker.jobs.framework:submit_job:113 - Upload completed in 0.20465
WARNING | 2023-02-07 00:27:23 | worker.jobs.framework:submit_job:128 - During gen submit, server https://stablehorde.net/ responded with status code 400: Processing Generation with ID 66080db1-fa83-41b0-a876-2f79abbb3c29 already submitted.. Waiting for 2 seconds... (Retry 11/10)
ERROR | 2023-02-07 00:27:25 | worker.jobs.framework:submit_job:99 - Exceeded retry count 11 for job id 66080db1-fa83-41b0-a876-2f79abbb3c29. Aborting job!
INFO | 2023-02-07 00:27:27 | worker.jobs.stable_diffusion:start_job:284 - Finished generation...
DEBUG | 2023-02-07 00:27:27 | worker.jobs.stable_diffusion:start_job:324 - post-processing done...
DEBUG | 2023-02-07 00:27:27 | worker.jobs.framework:start_submit_thread:79 - Finished job in threadpool
DEBUG | 2023-02-07 00:27:27 | worker.workers.framework:check_running_job_status:150 - Job finished successfully in 11.297s (Total Completed: 1126)
Model integrity check too slow
Hashing all the models when the worker starts up is unbearably slow as the number of models installed locally increases.
This could be massively sped up by caching the results of hashing the files and only rehashing if the source file has been updated since we last hashed.
Worker configuration options could be clearer.
We have a mix of:
- bridgeData.yaml
- Environmental variables
- Command line switches
This could do with a tidy up to:
- Ensure bridgeData.yaml can be used to configure as much as possible
- Keep environmental variables and command line overrides.
- More clearly document what all the options do and how to use them.
Make webui show information about each option somehow
A popup on mouseover, or a little info button to press?
Auto-configuration
Add a feature which automatically configures the worker to sensible settings based on inspection of GPU, VRAM, RAM, storage space. Or perhaps ask the user some wizard style setup questions about their wishes. Or both!
This will help ease the frequent confusion of new worker operators of where to even start.
Depends on #78
Terminal UI required fixes
- Confirm terminal ui works on Linux.
- Fix stdout/stderr suppression on Linux
- Fix kudos per hour in alchemy worker.
- Move CPU Load over to the left. [2]
- Fix kudos per hour setting if it's configured badly and terminal ui is enabled.
- Check all input prompts respect the "always_download" config setting. (They don't [1])
- Don't read the log file, hook Loguru directly.
- Don't allow terminal ui to start if auto-download option is not selected. Unless run in parallel mode.
- Confirm works with alchemy worker
[1] They don't, but it doesn't matter as these prompts happen before the UI is initialised.
help: cuInit results in UNKNOWN_CUDA_ERROR
My computer has an RTX 2070 graphics card and everything runs fine with SDWEBUI. I am using a virtual environment with miniconda3. However, after installing AI Horde, I am getting an error. What could be the reason for this?
(horde-worker) ox2098@DAN:~/AI-Horde-Worker$ sh horde-bridge.sh
Traceback (most recent call last):
File "/home/ox2098/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py", line 247, in ensure_initialized
self.cuInit(0)
File "/home/ox2098/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py", line 320, in safe_cuda_api_call
self._check_ctypes_error(fname, retcode)
File "/home/ox2098/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py", line 388, in _check_ctypes_error
raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [34] Call to cuInit results in UNKNOWN_CUDA_ERROR
Unrecognised models doesn't give obvious feedback which model
Its come up a few times when people see the warning about unrecognised models, it's not super clear which model names are causing the problem. Could probably just do with a more explicit warning.
Terminal UI Improvements
- Add a warning when we're waiting due to not getting a job popped fast enough. [bitte]
- Allow toggling of features such as control net, etc. [bitte]
- Add "tabs" for other settings and a help tab explaining the totally non-obvious features. [jug]
- Add a (r)eset option to reset local stats/counters. [bitte]
Moved Install Folder, no longer working
Hi,
I'm on Windows 11 and I moved the install folder and now it no longer works.
I've since deleted the whole folder and gone through the installation process again but it's still not working.
When I ran update-runtime.cmd the first time everything got set up correctly, and I was able to run the horde-bridge.cmd once, however, the next time it didn't work.
Running update-runtime.cmd again gives the following error:
__
__ ______ ___ ____ _____ ___ / /_ ____ _
/ / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
/ /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
/ .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
/_/
error libmamba Non-conda folder exists at prefix
critical libmamba Aborting.
Traceback (most recent call last):
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\pip\__main__.py", line 29, in <module>
from pip._internal.cli.main import main as _main
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\pip\_internal\cli\main.py", line 9, in <module>
from pip._internal.cli.autocompletion import autocomplete
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\pip\_internal\cli\autocompletion.py", line 10, in <module>
from pip._internal.cli.main_parser import create_main_parser
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\pip\_internal\cli\main_parser.py", line 9, in <module>
from pip._internal.build_env import get_runnable_pip
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\pip\_internal\build_env.py", line 19, in <module>
from pip._internal.cli.spinners import open_spinner
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\pip\_internal\cli\spinners.py", line 9, in <module>
from pip._internal.utils.logging import get_indentation
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\pip\_internal\utils\logging.py", line 4, in <module>
import logging.handlers
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\logging\handlers.py", line 26, in <module>
import io, logging, socket, os, pickle, struct, time, re
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\socket.py", line 54, in <module>
import os, sys, io, selectors
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\selectors.py", line 12, in <module>
import select
ModuleNotFoundError: No module named 'select'
If there are no errors above everything should be correctly installed (If not, try running update_runtime.cmd as admin).
Running horde-bridge.cmd again gives the following error:
The operation completed successfully.
Traceback (most recent call last):
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\bridge_stable_diffusion.py", line 10, in <module>
from nataili.model_manager.super import ModelManager
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\model_manager\super.py", line 18, in <module>
import torch
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\__init__.py", line 877, in <module>
from torch import multiprocessing as multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\__init__.py", line 18, in <module>
from .reductions import init_reductions
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\reductions.py", line 6, in <module>
import multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\__init__.py", line 16, in <module>
from . import context
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\context.py", line 6, in <module>
from . import reduction
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\reduction.py", line 16, in <module>
import socket
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\socket.py", line 54, in <module>
import os, sys, io, selectors
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\selectors.py", line 12, in <module>
import select
ModuleNotFoundError: No module named 'select'
The operation completed successfully.
Traceback (most recent call last):
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\bridge_stable_diffusion.py", line 10, in <module>
from nataili.model_manager.super import ModelManager
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\model_manager\super.py", line 18, in <module>
import torch
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\__init__.py", line 877, in <module>
from torch import multiprocessing as multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\__init__.py", line 18, in <module>
from .reductions import init_reductions
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\reductions.py", line 6, in <module>
import multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\__init__.py", line 16, in <module>
from . import context
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\context.py", line 6, in <module>
from . import reduction
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\reduction.py", line 16, in <module>
import socket
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\socket.py", line 54, in <module>
import os, sys, io, selectors
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\selectors.py", line 12, in <module>
import select
ModuleNotFoundError: No module named 'select'
The operation completed successfully.
Traceback (most recent call last):
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\bridge_stable_diffusion.py", line 10, in <module>
from nataili.model_manager.super import ModelManager
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\model_manager\super.py", line 18, in <module>
import torch
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\__init__.py", line 877, in <module>
from torch import multiprocessing as multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\__init__.py", line 18, in <module>
from .reductions import init_reductions
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\reductions.py", line 6, in <module>
import multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\__init__.py", line 16, in <module>
from . import context
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\context.py", line 6, in <module>
from . import reduction
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\reduction.py", line 16, in <module>
import socket
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\socket.py", line 54, in <module>
import os, sys, io, selectors
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\selectors.py", line 12, in <module>
import select
ModuleNotFoundError: No module named 'select'
The operation completed successfully.
Traceback (most recent call last):
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\bridge_stable_diffusion.py", line 10, in <module>
from nataili.model_manager.super import ModelManager
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\model_manager\super.py", line 18, in <module>
import torch
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\__init__.py", line 877, in <module>
from torch import multiprocessing as multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\__init__.py", line 18, in <module>
from .reductions import init_reductions
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\reductions.py", line 6, in <module>
import multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\__init__.py", line 16, in <module>
from . import context
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\context.py", line 6, in <module>
from . import reduction
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\reduction.py", line 16, in <module>
import socket
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\socket.py", line 54, in <module>
import os, sys, io, selectors
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\selectors.py", line 12, in <module>
import select
ModuleNotFoundError: No module named 'select'
The operation completed successfully.
Traceback (most recent call last):
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\bridge_stable_diffusion.py", line 10, in <module>
from nataili.model_manager.super import ModelManager
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\model_manager\super.py", line 18, in <module>
import torch
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\__init__.py", line 877, in <module>
from torch import multiprocessing as multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\__init__.py", line 18, in <module>
from .reductions import init_reductions
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\reductions.py", line 6, in <module>
import multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\__init__.py", line 16, in <module>
from . import context
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\context.py", line 6, in <module>
from . import reduction
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\reduction.py", line 16, in <module>
import socket
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\socket.py", line 54, in <module>
import os, sys, io, selectors
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\selectors.py", line 12, in <module>
import select
ModuleNotFoundError: No module named 'select'
The operation completed successfully.
Traceback (most recent call last):
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\bridge_stable_diffusion.py", line 10, in <module>
from nataili.model_manager.super import ModelManager
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\model_manager\super.py", line 18, in <module>
import torch
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\__init__.py", line 877, in <module>
from torch import multiprocessing as multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\__init__.py", line 18, in <module>
from .reductions import init_reductions
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\reductions.py", line 6, in <module>
import multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\__init__.py", line 16, in <module>
from . import context
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\context.py", line 6, in <module>
from . import reduction
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\reduction.py", line 16, in <module>
import socket
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\socket.py", line 54, in <module>
import os, sys, io, selectors
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\selectors.py", line 12, in <module>
import select
ModuleNotFoundError: No module named 'select'
The operation completed successfully.
Traceback (most recent call last):
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\bridge_stable_diffusion.py", line 10, in <module>
from nataili.model_manager.super import ModelManager
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\model_manager\super.py", line 18, in <module>
import torch
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\__init__.py", line 877, in <module>
from torch import multiprocessing as multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\__init__.py", line 18, in <module>
from .reductions import init_reductions
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\reductions.py", line 6, in <module>
import multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\__init__.py", line 16, in <module>
from . import context
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\context.py", line 6, in <module>
from . import reduction
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\reduction.py", line 16, in <module>
import socket
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\socket.py", line 54, in <module>
import os, sys, io, selectors
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\selectors.py", line 12, in <module>
import select
ModuleNotFoundError: No module named 'select'
The operation completed successfully.
Traceback (most recent call last):
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\bridge_stable_diffusion.py", line 10, in <module>
from nataili.model_manager.super import ModelManager
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\model_manager\super.py", line 18, in <module>
import torch
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\__init__.py", line 877, in <module>
from torch import multiprocessing as multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\__init__.py", line 18, in <module>
from .reductions import init_reductions
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\reductions.py", line 6, in <module>
import multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\__init__.py", line 16, in <module>
from . import context
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\context.py", line 6, in <module>
from . import reduction
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\reduction.py", line 16, in <module>
import socket
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\socket.py", line 54, in <module>
import os, sys, io, selectors
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\selectors.py", line 12, in <module>
import select
ModuleNotFoundError: No module named 'select'
The operation completed successfully.
Traceback (most recent call last):
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\bridge_stable_diffusion.py", line 10, in <module>
from nataili.model_manager.super import ModelManager
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\model_manager\super.py", line 18, in <module>
import torch
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\__init__.py", line 877, in <module>
from torch import multiprocessing as multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\__init__.py", line 18, in <module>
from .reductions import init_reductions
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\reductions.py", line 6, in <module>
import multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\__init__.py", line 16, in <module>
from . import context
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\context.py", line 6, in <module>
from . import reduction
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\reduction.py", line 16, in <module>
import socket
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\socket.py", line 54, in <module>
import os, sys, io, selectors
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\selectors.py", line 12, in <module>
import select
ModuleNotFoundError: No module named 'select'
The operation completed successfully.
Traceback (most recent call last):
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\bridge_stable_diffusion.py", line 10, in <module>
from nataili.model_manager.super import ModelManager
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\model_manager\super.py", line 18, in <module>
import torch
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\__init__.py", line 877, in <module>
from torch import multiprocessing as multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\__init__.py", line 18, in <module>
from .reductions import init_reductions
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\reductions.py", line 6, in <module>
import multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\__init__.py", line 16, in <module>
from . import context
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\context.py", line 6, in <module>
from . import reduction
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\reduction.py", line 16, in <module>
import socket
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\socket.py", line 54, in <module>
import os, sys, io, selectors
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\selectors.py", line 12, in <module>
import select
ModuleNotFoundError: No module named 'select'
The operation completed successfully.
Traceback (most recent call last):
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\bridge_stable_diffusion.py", line 10, in <module>
from nataili.model_manager.super import ModelManager
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\nataili\model_manager\super.py", line 18, in <module>
import torch
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\__init__.py", line 877, in <module>
from torch import multiprocessing as multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\__init__.py", line 18, in <module>
from .reductions import init_reductions
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\site-packages\torch\multiprocessing\reductions.py", line 6, in <module>
import multiprocessing
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\__init__.py", line 16, in <module>
from . import context
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\context.py", line 6, in <module>
from . import reduction
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\multiprocessing\reduction.py", line 16, in <module>
import socket
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\socket.py", line 54, in <module>
import os, sys, io, selectors
File "D:\Software\Stable Diffusion\AIHorde\AI-Horde-Worker\conda\envs\windows\lib\selectors.py", line 12, in <module>
import select
ModuleNotFoundError: No module named 'select'
The operation completed successfully.
[process exited with code 3221226505 (0xc0000409)]
Any help is appreciated, thanks.
Output to terminal is not great as a worker operatior
This is not so useful:
Could we output this less messily? E.g.
Stats.log not populating for Interrogation worker
Not urgent - just thought I'd document it!
More magic constants for models to load
Following on from #62, As suggested by Raineh Daze, et al, in discord:
Things like:
- ALL ANIME
- ALL SFW
- etc
Use of folder 'nataili' causes package conflicts
While #149 was being drafted, it was discovered the folder for models, nataili/ causes certain unintended behavior with ruff. In this particular case ruff fails to navigate import sorting, failing silently locally, and causes issues during CI.
This only highlights the issue of using this folder with this project's folder/file structure, and could cause more problems in the future.
I am proposing the following high-level set of changes:
-
Code which identifies users who are using the default folder scheme (ie, users with
./asNATAILI_CACHE_DIR)- Notify the user a move is about to be made, and give them the option to abort. Emphasize that symlinks will be broken and make it abundantly clear that files will be moved.
- Move it to a folder that has no chance of conflict, such as
./.data/nataili - Notify the user of the new location.
- Set
NATAILI_CACHE_DIRto the new data folder, IENATAILI_CACHE_DIR=./.data/- Further, this may be an opportunity to do the same for
ray/.
- Further, this may be an opportunity to do the same for
- There are a couple of lines which catch a small corner case to do with this directory, they can be removed as the case it addresses could be obsoleted by this set of changes.
-
Code which identifies if the the folder
nataili/exists, but the worker is not currently configured to use it or it appears to be empty- Warn the user, but take no action
Update to Torch 2.0
Please update the torch dependency from 1.14 to 2.0 - in part because new torch will pull in newer cudnn, cublas and nvblas libraries which yield a 200% speed improvement on RTX 40-series cards.
Document YAML options
I've seen on Discord that disabling voodoo can now be done via yaml, but I don't see any documentation for doing so and the readme still refers to a cli flag.
More generally, it would be great for all possible yaml options to be documented!
Unable to start `horde-bridge.sh` due to an error
When running sh horde-bridge.sh I get
INIT | Checking | xformers optimizations
INIT | Loaded | xformers optimizations
2023-03-03 18:33:38,789 INFO worker.py:1553 -- Started a local Ray instance.
INIT | Prepared | Ray temp dir '/home/amda/AI-Horde-Worker/ray'
INIT | Downloading | Model Reference
INIT | OK | Model Reference
INFO | 2023-03-03 18:34:07.269151 | nataili.model_manager.base:init:65 - Downloaded model reference. Got 167 models.
INFO | 2023-03-03 18:34:07.271257 | nataili.model_manager.base:init:77 - Got 0 available models.
Traceback (most recent call last):
File "/home/amda/AI-Horde-Worker/bridge_stable_diffusion.py", line 66, in <module>
main()
File "/home/amda/AI-Horde-Worker/bridge_stable_diffusion.py", line 47, in main
model_manager = ModelManager(
File "/home/amda/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/nataili/model_manager/super.py", line 68, in __init__
from nataili.model_manager.diffusers import DiffusersModelManager
File "/home/amda/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/nataili/model_manager/diffusers.py", line 22, in <module>
from diffusers.pipelines import StableDiffusionDepth2ImgPipeline, StableDiffusionInpaintPipeline
ImportError: cannot import name 'StableDiffusionDepth2ImgPipeline' from 'diffusers.pipelines' (/home/amda/.local/lib/python3.10/site-packages/diffusers/pipelines/__init__.py)
This might be my error as I have not gave model checkpoint or a hugging face api key. I am not sure how I am supposed to ass these?
Model reference is retrieved every time a model is downloaded
Model reference retrieval should be cached.
Wrong credentials
I'm getting this warning:
WARNING | 2023-03-23 17:46:30.143813 | worker.jobs.poppers:horde_pop:70 - Wrong credentials to submit as this worker. (403)
Is my worker working?
I added my API key in bridgeData.yaml.
Ensure we check for bad .cache files
If .cache files become corrupted, handle this properly by discarding the cache file.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.