Labeling of trees about com.unity.perception HOT 12 CLOSED

Hi Henrik. In order for us to help you, we would need a bit more info about the issue. Can you perhaps provide some screenshots of the visualized bounding boxes, as well as how your labeling is set up for the trees?

from com.unity.perception.

OK, very interesting. We haven't played with Alpha-Clipping or Double-Sided. Could you tell us which asset store package you used so we can test ourselves?

Querying the alpha channel of textures is on our backlog but we do not yet have an ETA. In the meantime if you have some experience with shaders you may want to try playing with the two shaders used for labeling:

https://github.com/Unity-Technologies/com.unity.perception/blob/master/com.unity.perception/Runtime/GroundTruth/Resources/SemanticSegmentation.shader

https://github.com/Unity-Technologies/com.unity.perception/blob/master/com.unity.perception/Runtime/GroundTruth/Resources/InstanceSegmentation.shader

We will update this thread when we have a solution available to try.

from com.unity.perception.

I had the time to look at it myself. I changed the shader, but this was not enough. I also changed the SemanticSegmentationCrossPipelinePass so that the texture of each material is accessible by the shader. In the image you can see the result. Do you want me to contribute this somehow (it fits my application, but it definitely still has to be tested carefully)?

from com.unity.perception.

Thanks for your quick reply. Sure, here are a few pictures. Behind the tree are buildings and the sky. I use the SemanticSegmentationLabeler. Config: Trees are labeled green, buildings in the background gray. As you can see, some leaves on the tree are not getting the correct label / color assignment. They just get the color / label of the objects that are behind them and overall the correspondence between leaves and label / color is not very accurate

from com.unity.perception.

There appear to be two issues at play here.

The first is that the perception package do not currently support transparency for in ground truth rendering. This causes the leaf sections to be show up as large geometric shapes instead of individual leaves. Is it important to you to have pixel-perfect segmentation of leaves?

The second is that the geometry in the semantic segmentation does not line up with the positions of the actual leaves. My hunch is that the tree has a geometry or vertex shader which is deforming the tree. Vertex and geometry shaders are also not run when generating ground truth. What kind of tree is this? Speed tree or some other asset store package? You may find an option to disable tree movement which will get the geometry to be static and line up properly.

from com.unity.perception.

Just for clarification, the result shown in the images above is the same as in the generated segmentation images.

It could be important for the training to have pixel-perfect segmentation of leaves or at least no misclassified parts. It is an asset store package, but it just uses the HDRP/Lit-Shader and there is no tree movement at all.

I think I found the reason: The first image is without Alpha-Clipping and the second without Double-Sided. Alpha clipping explains why it does not match the shape and double-sided why some parts are not labeled at all. Without both, the labeling is exactly the same, but of course it looks very bad.

from com.unity.perception.

You can test it with any object that changes it appearance (especially the shape / contour), if you deactivate alpha-clipping or double-sided. I tested it beside other assets with trees from the Forest Environment - Dynamic Nature from NatureManufacture in the asset store. But as I said, it also happens with other assets.

from com.unity.perception.

By the way, we have a branch with experimental support for transparency here: https://github.com/Unity-Technologies/com.unity.perception/tree/seg_opacity Feel free to give feedback if you try it out!

from com.unity.perception.

SemanticSegmentationCrossPipelinePas

Hi, @HenrikWerner, I'm also interested in labeling leaves. Would you mind sharing your modification of CrossPipelinePass if possible?

from com.unity.perception.

Hi ! I'm working with trees also. Support for transparency helped here but still facing the issue with double-side. Could you share or point me how to change to make visible double-side ?

from com.unity.perception.

Hi ! any help with double-side please?

from com.unity.perception.

Hey all! The Perception team just released 🎉 Perception 1.0 🎉, a major update to the toolset! We have added support for transparency in our segmentation pass which should enable the above use-case. Let us know how it goes! Closing this issue for now.

There are a bunch of cool features to check out such as ray tracing, depth, a new output format SOLO, normals, and more! Here is the full list of changes: Perception 1.0 Changelog

from com.unity.perception.