weijiawu / diffumask Goto Github PK
View Code? Open in Web Editor NEW[ICCV2023] DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models
[ICCV2023] DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models
the last step:
sh ./script/augmentation_VOC.sh
but i don't find the file 'augmentation_VOC.sh' in script; are you release all the codes?
Do you have any script where I can use pretrain model and generate a mask for unseen objects using prompt only?
There has been a major install error when using Cython=3.0.0 for installation.
Instead, Bychanging the comand to the following can solve the problem:
python -m pip install git+https://github.com/kodalli/pydensecrf.git
pip3 install --no-build-isolation -r requirements.txt
can you give me your code?
Thanks for your great work! There is no module voc12 when I train the affinitynet?
Traceback (most recent call last):
File "/root/anaconda3/envs/DiffuMask/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/root/anaconda3/envs/DiffuMask/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self.args, **self.kwargs)
File "/18054208921/diffumask/Stable_Diffusion/parallel_generate_VOC_Attention_AnyClass.py", line 738, in sub_processor
image, x_t = run(prompts, controller, latent=None, generator=g_cpu,out_put = os.path.join(image_path,"image{}{}.jpg".format(args.classes,image_cnt)),ldm_stable=ldm_stable)
File "/18054208921/diffumask/Stable_Diffusion/parallel_generate_VOC_Attention_AnyClass.py", line 438, in run
images_here, x_t = ptp_utils.text2image_ldm_stable(ldm_stable, prompts, controller, latent=latent, num_inference_steps=NUM_DIFFUSION_STEPS, guidance_scale=7, generator=generator, low_resource=LOW_RESOURCE)
File "/18054208921/diffumask/myenv/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
return func(*args, **kwargs)
File "/18054208921/diffumask/Stable_Diffusion/ptp_utils.py", line 175, in text2image_ldm_stable
latents = diffusion_step(model, controller, latents, context, t, guidance_scale, low_resource)
File "/18054208921/diffumask/Stable_Diffusion/ptp_utils.py", line 77, in diffusion_step
noise_pred = model.unet(latents_input, t, encoder_hidden_states=context)["sample"]
File "/18054208921/diffumask/myenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/18054208921/diffumask/myenv/lib/python3.8/site-packages/diffusers/models/unet_2d_condition.py", line 773, in forward
sample = upsample_block(
File "/18054208921/diffumask/myenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/18054208921/diffumask/myenv/lib/python3.8/site-packages/diffusers/models/unet_2d_blocks.py", line 1858, in forward
hidden_states = attn(
File "/18054208921/diffumask/myenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/18054208921/diffumask/myenv/lib/python3.8/site-packages/diffusers/models/transformer_2d.py", line 265, in forward
hidden_states = block(
File "/18054208921/diffumask/myenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/18054208921/diffumask/myenv/lib/python3.8/site-packages/diffusers/models/attention.py", line 313, in forward
attn_output = self.attn1(
File "/18054208921/diffumask/myenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/18054208921/diffumask/Stable_Diffusion/ptp_utils.py", line 221, in forward
sim = torch.einsum("b i d, b j d -> b i j", q, k) * self.scale
RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 1; 23.50 GiB total capacity; 20.56 GiB already allocated; 627.25 MiB free; 21.34 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
I run the data and mask generation part and find this problem. It seems that It is when i load the model and sudddenly use a large part of memory. I wonder if you could give me some advice and solve it. Thanks a lot. @weijiawu
When I install git+https://github.com/lucasb-eyer/pydensecrf.git, I have trouble with cython , so I found that I must downgrade to cython==0.29.36 and then install pydensecrf that okay but when i install OpenAI Clip I get likely the same error.
/tmp/pip-build-env-k7kqvt3l/overlay/lib/python3.8/site-packages/Cython/Compiler/Main.py:384: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /tmp/pip-install-ktjymnfq/pydensecrf_a587772943f6421480fe6eb089f3a216/pydensecrf/densecrf.pxd
tree = Parsing.p_module(s, pxd, full_module_name)
Error compiling Cython file:
------------------------------------------------------------
...
from eigen cimport *
^
------------------------------------------------------------
pydensecrf/densecrf.pxd:1:0: 'eigen.pxd' not found
Error compiling Cython file:
------------------------------------------------------------
...
cdef cppclass PottsCompatibility(LabelCompatibility):
PottsCompatibility(float) except +
cdef cppclass DiagonalCompatibility(LabelCompatibility):
DiagonalCompatibility(const c_VectorXf&) except +
^
------------------------------------------------------------
pydensecrf/densecrf.pxd:12:36: 'c_VectorXf' is not a type identifier
Error compiling Cython file:
------------------------------------------------------------
...
cdef cppclass DiagonalCompatibility(LabelCompatibility):
DiagonalCompatibility(const c_VectorXf&) except +
cdef cppclass MatrixCompatibility(LabelCompatibility):
MatrixCompatibility(const c_MatrixXf&) except +
^
------------------------------------------------------------
pydensecrf/densecrf.pxd:15:34: 'c_MatrixXf' is not a type identifier
Error compiling Cython file:
------------------------------------------------------------
...
cdef extern from "densecrf/include/unary.h":
cdef cppclass UnaryEnergy:
pass
cdef cppclass ConstUnaryEnergy(UnaryEnergy):
ConstUnaryEnergy(const c_MatrixXf& unary) except +
^
------------------------------------------------------------
pydensecrf/densecrf.pxd:23:31: 'c_MatrixXf' is not a type identifier
Error compiling Cython file:
------------------------------------------------------------
...
cdef cppclass ConstUnaryEnergy(UnaryEnergy):
ConstUnaryEnergy(const c_MatrixXf& unary) except +
cdef cppclass LogisticUnaryEnergy(UnaryEnergy):
LogisticUnaryEnergy(const c_MatrixXf& L, const c_MatrixXf& feature) except +
^
------------------------------------------------------------
pydensecrf/densecrf.pxd:26:34: 'c_MatrixXf' is not a type identifier
Error compiling Cython file:
------------------------------------------------------------
...
cdef cppclass ConstUnaryEnergy(UnaryEnergy):
ConstUnaryEnergy(const c_MatrixXf& unary) except +
cdef cppclass LogisticUnaryEnergy(UnaryEnergy):
LogisticUnaryEnergy(const c_MatrixXf& L, const c_MatrixXf& feature) except +
^
------------------------------------------------------------
pydensecrf/densecrf.pxd:26:55: 'c_MatrixXf' is not a type identifier
Error compiling Cython file:
------------------------------------------------------------
...
c_DenseCRF(int N, int M) except +
# Setup methods.
# TODO
#void addPairwiseEnergy(PairwisePotential *potential)
void addPairwiseEnergy(const c_MatrixXf &features, LabelCompatibility*, KernelType, NormalizationType)
^
------------------------------------------------------------
pydensecrf/densecrf.pxd:54:37: 'c_MatrixXf' is not a type identifier
Error compiling Cython file:
------------------------------------------------------------
...
# Setup methods.
# TODO
#void addPairwiseEnergy(PairwisePotential *potential)
void addPairwiseEnergy(const c_MatrixXf &features, LabelCompatibility*, KernelType, NormalizationType)
void setUnaryEnergy(UnaryEnergy *unary)
void setUnaryEnergy(const c_MatrixXf &unary)
^
------------------------------------------------------------
pydensecrf/densecrf.pxd:56:34: 'c_MatrixXf' is not a type identifier
Error compiling Cython file:
------------------------------------------------------------
...
# TODO
#void addPairwiseEnergy(PairwisePotential *potential)
void addPairwiseEnergy(const c_MatrixXf &features, LabelCompatibility*, KernelType, NormalizationType)
void setUnaryEnergy(UnaryEnergy *unary)
void setUnaryEnergy(const c_MatrixXf &unary)
void setUnaryEnergy(const c_MatrixXf &L, const c_MatrixXf &feature)
^
------------------------------------------------------------
pydensecrf/densecrf.pxd:57:34: 'c_MatrixXf' is not a type identifier
Error compiling Cython file:
------------------------------------------------------------
...
# TODO
#void addPairwiseEnergy(PairwisePotential *potential)
void addPairwiseEnergy(const c_MatrixXf &features, LabelCompatibility*, KernelType, NormalizationType)
void setUnaryEnergy(UnaryEnergy *unary)
void setUnaryEnergy(const c_MatrixXf &unary)
void setUnaryEnergy(const c_MatrixXf &L, const c_MatrixXf &feature)
^
------------------------------------------------------------
pydensecrf/densecrf.pxd:57:55: 'c_MatrixXf' is not a type identifier
Error compiling Cython file:
------------------------------------------------------------
...
void setUnaryEnergy(UnaryEnergy *unary)
void setUnaryEnergy(const c_MatrixXf &unary)
void setUnaryEnergy(const c_MatrixXf &L, const c_MatrixXf &feature)
# Inference methods.
c_MatrixXf inference(int n_iterations)
^
------------------------------------------------------------
pydensecrf/densecrf.pxd:60:8: 'c_MatrixXf' is not a type identifier
Error compiling Cython file:
------------------------------------------------------------
...
c_MatrixXf inference(int n_iterations)
# TODO: Not enabled because it would require wrapping VectorXs (note the `s`)
#c_VectorXs map(int n_iterations)
# Step-by-step inference methods.
c_MatrixXf startInference() const
^
------------------------------------------------------------
pydensecrf/densecrf.pxd:65:8: 'c_MatrixXf' is not a type identifier
Error compiling Cython file:
------------------------------------------------------------
...
# TODO: Not enabled because it would require wrapping VectorXs (note the `s`)
#c_VectorXs map(int n_iterations)
# Step-by-step inference methods.
c_MatrixXf startInference() const
void stepInference(c_MatrixXf &Q, c_MatrixXf &tmp1, c_MatrixXf &tmp2) const
^
------------------------------------------------------------
pydensecrf/densecrf.pxd:66:27: 'c_MatrixXf' is not a type identifier
Error compiling Cython file:
------------------------------------------------------------
...
# TODO: Not enabled because it would require wrapping VectorXs (note the `s`)
#c_VectorXs map(int n_iterations)
# Step-by-step inference methods.
c_MatrixXf startInference() const
void stepInference(c_MatrixXf &Q, c_MatrixXf &tmp1, c_MatrixXf &tmp2) const
^
------------------------------------------------------------
pydensecrf/densecrf.pxd:66:42: 'c_MatrixXf' is not a type identifier
Error compiling Cython file:
------------------------------------------------------------
...
# TODO: Not enabled because it would require wrapping VectorXs (note the `s`)
#c_VectorXs map(int n_iterations)
# Step-by-step inference methods.
c_MatrixXf startInference() const
void stepInference(c_MatrixXf &Q, c_MatrixXf &tmp1, c_MatrixXf &tmp2) const
^
------------------------------------------------------------
pydensecrf/densecrf.pxd:66:60: 'c_MatrixXf' is not a type identifier
Error compiling Cython file:
------------------------------------------------------------
...
# Step-by-step inference methods.
c_MatrixXf startInference() const
void stepInference(c_MatrixXf &Q, c_MatrixXf &tmp1, c_MatrixXf &tmp2) const
#double gradient( int n_iterations, const ObjectiveFunction & objective, c_VectorXf * unary_grad, c_VectorXf * lbl_cmp_grad, c_VectorXf * kernel_grad=NULL ) const;
double klDivergence(const c_MatrixXf &Q) const
^
------------------------------------------------------------
pydensecrf/densecrf.pxd:69:34: 'c_MatrixXf' is not a type identifier
warning: pydensecrf/densecrf.pyx:20:4: Unreachable code
Error compiling Cython file:
------------------------------------------------------------
...
return eigen.MatrixXf().wrap(self._this.inference(niter))
def startInference(self):
return eigen.MatrixXf().wrap(self._this.startInference()), eigen.MatrixXf(), eigen.MatrixXf()
def stepInference(self, MatrixXf Q, MatrixXf tmp1, MatrixXf tmp2):
^
------------------------------------------------------------
pydensecrf/densecrf.pyx:97:28: 'MatrixXf' is not a type identifier
Error compiling Cython file:
------------------------------------------------------------
...
return eigen.MatrixXf().wrap(self._this.inference(niter))
def startInference(self):
return eigen.MatrixXf().wrap(self._this.startInference()), eigen.MatrixXf(), eigen.MatrixXf()
def stepInference(self, MatrixXf Q, MatrixXf tmp1, MatrixXf tmp2):
^
------------------------------------------------------------
pydensecrf/densecrf.pyx:97:40: 'MatrixXf' is not a type identifier
Error compiling Cython file:
------------------------------------------------------------
...
return eigen.MatrixXf().wrap(self._this.inference(niter))
def startInference(self):
return eigen.MatrixXf().wrap(self._this.startInference()), eigen.MatrixXf(), eigen.MatrixXf()
def stepInference(self, MatrixXf Q, MatrixXf tmp1, MatrixXf tmp2):
^
------------------------------------------------------------
pydensecrf/densecrf.pyx:97:55: 'MatrixXf' is not a type identifier
Error compiling Cython file:
------------------------------------------------------------
...
return eigen.MatrixXf().wrap(self._this.startInference()), eigen.MatrixXf(), eigen.MatrixXf()
def stepInference(self, MatrixXf Q, MatrixXf tmp1, MatrixXf tmp2):
self._this.stepInference(Q.m, tmp1.m, tmp2.m)
def klDivergence(self, MatrixXf Q):
^
------------------------------------------------------------
pydensecrf/densecrf.pyx:100:27: 'MatrixXf' is not a type identifier
Error compiling Cython file:
------------------------------------------------------------
...
raise ValueError("Bad shape for pairwise energy (Need (?, {}), got {})".format(self._nvar, (features.shape[0], features.shape[1])))
self._this.addPairwiseEnergy(eigen.c_matrixXf(features), _labelcomp(compat), kernel, normalization)
def setUnary(self, Unary u):
self._this.setUnaryEnergy(u.move())
^
------------------------------------------------------------
pydensecrf/densecrf.pyx:78:33: ambiguous overloaded method
Compiling pydensecrf/eigen.pyx because it changed.
Compiling pydensecrf/densecrf.pyx because it changed.
[1/2] Cythonizing pydensecrf/densecrf.pyx
Traceback (most recent call last):
File "/home/vdkhoi/.conda/envs/DiffuMask/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/home/vdkhoi/.conda/envs/DiffuMask/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/vdkhoi/.conda/envs/DiffuMask/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
File "/tmp/pip-build-env-k7kqvt3l/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
File "/tmp/pip-build-env-k7kqvt3l/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
self.run_setup()
File "/tmp/pip-build-env-k7kqvt3l/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 507, in run_setup
super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
File "/tmp/pip-build-env-k7kqvt3l/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 341, in run_setup
exec(code, locals())
File "<string>", line 13, in <module>
File "/tmp/pip-build-env-k7kqvt3l/overlay/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 1134, in cythonize
cythonize_one(*args)
File "/tmp/pip-build-env-k7kqvt3l/overlay/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one
raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: pydensecrf/densecrf.pyx
[end of output]
I can't find the code about noise learning in your repository, may I ask if this part needs to be operated manually?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.