Currently while perception exports a lot of data with the various annotation files, it doesn't export the generated image sizes.
Coco format requires the image size and the size info is vital for rescaling of the bboxes during training.
Thus, would ask that image sizes be exported somewhere (i.e. captures_000.json) for easy access.

*I wrote a coco converter utility that reads the data from the annotations to generate coco files for training, but to fill in the image size, which is required in coco format, I have to go inspect every img file to pull the sizes.

Thanks!

Hi,
Can Perception create annotations only for objects that are above a minimum size? (see picture with example)

the sim to real gap (based on IOU of holdout sets) on one of my datasets has been much higher than other sets.
After randomizing a number of items to try and reduce that, I have effectively found that the core issue is that the bounding boxes that are generated during rotations are ballooning more than needed.
That results in non-rotated examples having a similar excess buffer and accounts for the lower sim to real showing on this dataset.
Thus my question is there any options to reduce or tighten how bounding boxes are resized to account for randomized rotations?

Let me show an example - first one that is basically vertical. The bounding boxes segment up the diagnostic - the bottom of the 'text' portion runs along the bottom of the IVD box, and above the read window.
The top of the diagnostic portion goes right against the bottom of the text portion.
(these junctures will show the issue in the next photo).

Now when we add rotation, then top of the diagnostic goes way higher than it needs to, and the bottom of the text also extends much further than needed.
I've tried to highlight with the red bubble on the side to show the gap, but basically the top of the diagnostic can still be below the IVD box. Similarly the bottom of the text box doesn't need to drop all the way down to go below the read window.

Thus, we now 'teaching' the AI it should have these excess buffers and hence the lower performance on the IOU holdout.

vs the result of tuning with hand labelled real images:

Note that I have rotations in my 'real image' training and we don't see this excess buffering...thus is there a way to tune/modify the bounding boxes generated for rotated items to tighten their expansion?

I must be missing something obvious, but having spent some time on it I can't find the solution and thus posting this here for input.
Much like the prefabs that came with the tutorial, I have my prefabs moved to -90 x rotation so that their front now faces the camera during synthetic generation.
That all works as expected.
However, now I want to to rotate the camera facing objects, so that they are randomized 0-359 degrees in a clock wise type rotation.
By rights, this should be rotation along the Y axis based on the rotation gymbal that is showing.
However, I find that rotating the Y or Z axis end up moving the object the along the exact same rotation plane...and am unable to accomplish the rotation of the object in a clock-wise fashion. This seems to repeat with the tutorial objects as well so it's not a prefab issue.
I think some images will help show the problem so let me attach:
Here's my synthetic 3d model as imported. The view is from the camera direction, slightly up and to the right to show it better.

If I run a scenario like this, then all I will see is the very top of the item since it was built flat...you get this:

So, much like the tutorial prefabs I align the prefab rotation to be set with -90x. Now you see the face (albeit upside down) but that gets things going:

Now what I want to is to randomly rotate them along this (y axis) so that they are randomized as shown by the blue arrow:

However, the issue is that regardless of whether I rotate the Y axis or the Z axis...the object moves exactly the same rotation angle. (??)
Here's Y with 45:

and now Z with 45 - identical rotation:

and a bit shockingly, if I have both Y and Z with 45...it now makes effectively a 90 degree rotation to show 100% the rotations are now moving on the identical rotation plane.

And that's the issue here - how do we have a rotation randomized to rotate it like a clock hand once rotated to face the camera?
Thanks!

Can the unity perception tool export keypoints location from 3d model to 2d coordinates into a text file?
Thank you

Hello,
Is there an easy way to control the output directory? Is there a command line argument, a way to access it in script, or a config setting that I'm missing?
I have been unable to find a way yet. I'm trying to run a simple build on Windows and need to control where the output goes other than the default location: C:/Users/User/AppData/LocalLow/CompanyName/ProjectName/GUIDString. This is because the C drive has limited space.

Thanks!

I have been using the tutorial config which mimics an outdoor scenario (sunlight, open air) but now need to also do obj detection in indoor (home/office) type of lighting, esp to mimic the softer shadows vs the harsher ones from sunlight.
It seems the only way to mimic that type of ambient lighting/softer shadows is to literally have the requisite walls and interior to get the realistic light bouncing and color pickup, so wondering if there are any prebuilt setups that are pending or recommended, or do we just have to build our own?
I'm using HDRP.
Thanks!

Hey Unity team! :)

I'd like to render the virtual/game world from multiple camera perspectives at every given time.
Using 1 PerceptionCamera works (as the system was designed for), basically by:

rendering 1 camera,
then time advances,
render 1 camera,
then time advances.

What I'd like is...
render X cameras,
then time advances,
render X cameras,
then time advances.

However, using multiple PerceptionCameras in the scene causes file sharing violations (overwriting due to async/parallelism), and even when modifying PerceptionCameras to use unique file paths based on the Camera's Unity InstanceID, the saved images are often duplicated and skip over PerceptionCameras.

Support for this would allow my team to capture multiple perspectives of the exact same point in time, and would be extremely helpful for capturing more data about rare states in the game world from multiple camera angles.
I don't believe this is supported, but please correct me if I'm wrong!
I'd really love this to be implemented, and wondered if you had any thoughts (or potentially a time-frame) on implementng this?

Are there any tips or recommendations on how to avoid pixelation during rotation of the objects? I am running in HDRP and generating high res but I see a lot of jagged pixelations on my objects. The effect is reduced with higher res finished images, but seems I must have something wrong.

I don't see these in the samples objects, but not clear what the fix is. Our models are high_res and look great in normal viewing.
Here's an example of the issue:

Images generated with the SemanticSegmentationLabeler have slightly other colors than the colors which have been defined in the SemanticSegmentationLabelConfig (e.g. rgb 100,100,100 results in 32,32,32). If I explicitly create a new render texture in SemanticSegmentationLabeler.cs with RenderTextureReadWrite.sRGB given in the constructor than the generated images have the correct color as defined in the config. So Linear <-> sRGB conversions are missing.

I am newbie in unity and I wanted to used it for synthetic data. But I came into contact with strange behaviours of 2d bounding boxes, like they don't really fit well with labeled objects in some cases. What's more sometimes objects on view of perception camera are not labeled. Even if I am using GameObjectOneWayCache that's not make any difference compared to simply destroy the objects. Below images with example weird behaviour of bounding boxes.

I think it's great package for synthetic data, so if you could help I would be glad :D

Is generating depth image in the development roadmap ? Simulating depth camera is important for many applications.

I've had poor training results the past two days and after a lot of tracing backwards, have found that while bounding boxes are always the correct fit, the class labels for those boxes are incorrect at maybe ~20% of the time, and randomly incorrect...and hence the poor training results.

I've confirmed the incorrect labels are present in the captures.json and not a translation into coco issue since I wrote my own coco converter, but hoping to get some insight on how to fix as this is a core issue.
Note this is not a 'every label is consistently off', but rather randomly the labels are swapped or incorrect. The boxes always fit nicely, it's the class label that is randomly wrong.

Here's a single example:

The bottom box should be 'bottom_nhs' and diagnostic should be in the middle, but in this case as you can see it's flipped. The actual boxes are correct in terms of their fit.
The class label for the diagnostic is also incorrect though less apparent....but here's an image that shows that random mistake better:

If you look at the two left images you can see the upper left has the correct type B (note the rounded handle cutout) and bottom left is all rectangular but also labelled as type B (it's a type D).

I didn't hit this issue on smaller (< 2K ) datasets but I'm seeing this in case on a 6K dataset. Other changes from working sets are 20 total prefabs (vs 4 or 8) and I also changed from auto-labelling via asset class name to using free form labelling.
I have verified multiple times that the prefabs all have correct invisible geometry.

I'm going to reset and rebuild with 4 and then 8 prefabs and see if I can narrow down the issue here but any insight or suggestions to isolate/ fix would be appreciated.

Is there a way to fix the placement of an item on each image such that I could produce a calibrated test set (rather than a randomized set)?
Example - we want to be able to take a given prefab, place it in the center every time, and then rotate it 1 degree from 0 - 90 on an axis, and thus produce a calibrated set of 0 - 90 degree to test where our current models break...and repeat that for each axis to simulate various perspective skew and test where our models fall off in detection accuracy.
Similarly, place the object in a grid on the image of upper left, middle upper, upper right, etc. and create a calibrated test set of the item being placed all throughout a given image. testing would use that to verify the range where in the image the items must be placed before accuracy falls off.
Again the idea is to produce calibration sets for testing to use rather than making these manually..so kind of the inverse of randomizing for training. :)
I have some idea how to code this but any details would be appreciated!

When I run a scenario, I can see the labels for the bounding boxes flashing past as each image is being created (since I have display visualizations checked).
However, it goes so fast I can't get any useful info from that....just a blur of images and yellow boxes.
I thus would like to step through so that I can see each individual image, so that I can actually see the labels to inspect that things are happening as expected.

However, if I pause, then hit play, and thus step through each image, I can see the individual images, but no bounding box labels are shown on the generated image.
Is there a way to make that happen - i.e. pause and step such that the bounding box labels are displayed on each new generated image for review?

Is there a way to give the sky a label and a color for semantic segmentation? I cannot find a way to do this (using HDRP).

The example code in Step 4 of the Randomization Tutorial md file is wrong. The Perception.Randomization namespace is in UnityEngine.Experimental.

Existing:
using UnityEngine.Perception.Randomization.Configuration;
using UnityEngine.Perception.Randomization.Parameters;
using UnityEngine.Perception.Randomization.Scenarios;

Correct:
using UnityEngine.Experimental.Perception.Randomization.Configuration;
using UnityEngine.Experimental.Perception.Randomization.Parameters;
using UnityEngine.Experimental.Perception.Randomization.Scenarios;

Since I will be randomizing the positions of the object, there is a good chance any two objects will collide. How do I prevent that from happening?

Hi everyone,
I updated the Perception package from 0.6.0-preview.1 to 0.7.0-preview.2 I get the following error in the Console when I run the scene:

d3d11: failed to create internal blit vertex shader, HR=80070057

project HDRP
Unity Editor 2020.1.15f1

According to these docs (https://github.com/Unity-Technologies/com.unity.perception/blob/master/com.unity.perception/Documentation~/Randomization/Index.md), it mentions to create a new ParameterConfiguration. But where is this component?

For the objects I am detecting, we frequently have QR codes or id numbers with text on the objects. In the real life scenario, every object will have it's own unique QR code and random text id written underneath it.

Thus, I'd like to be able to randomly apply varying QR codes and hand written number as masks onto the prefab for each image.
However, by it's nature, it seems prefabs are effectively 'fixed'...thus right now I am making a broader suite of prefabs, each with it's own QR code, but that feels a bit crude esp as we get into a matrix of prefab type * qr code * id text to try and create dynamic objects to mimic closer to the flow of real incoming data for it to train on.

Thus my question - is there a way to dynamically swap out masks/materials during the image OnIterationStart() or similar so that a single prefab/model could have it's masks/materials dynamically selected and assigned to randomize it on each image?
(simple example, having 100 QR codes as masks available, and a single base model - and it randomly pulls QR code from that set and puts that one on the prefab, and it's used for that individual image...on the next image, again one of the 100 is randomly selected for the next image, etc).
Related - if it's possible to swap/randomize materials in the same manner, it would reduce the need to premake so many prefabs each one hard coded with a given result material.

Right now I'll continue on the path of 'lots of prefabs' with hard coded QR / result materials, and randomly selecting from those, but if there is a more elegant way to do this in code both for masks and materials, would appreciate any advise on how to do it!

My images, both RGB and semantic images, were being saved in the respective folder.
Now only the sematic segmentation images are being saved. I have checked the 'Save RGB to disk option' but the folder for RGB images is still empty.
Anything else I can change or adjust?
Thanks in advance.

I need to controll which objects can be labeled on perception camera.
So I tried to disable Labeling script on every object when his instance was created,
like on the code example down below

var instance = gameObjectOneWayCache.GetOrInstantiate(prefabs.Sample());
var labeling = instance.GetComponent<Labeling>();
labeling.enabled = false;

But it does not change anything.

I want to simply disable labeling on every object and then activate it on the specific ones,
which for example are inside some range to camera view.
I know how to find that specific objects, but problem is I can't disable labeling on them.

Is there any option to do this?

I realize I went into unsupported land here, but since it was a minor upgrade (.1 to .6) and I wanted to test out the MasterStack feature, I upgraded.
Anyway, the result is a bevy of null reference errors that block me from seeing the inspector tab completely, but they are all the same error pointing to the same line of code.
Not sure if there's a quick fix or if I just need to rebuild in the previous version but wanted to post out for any feedback in case this is an easy fix.

and clicking on these goes to the same code:

If I use the Perception module and the DepthGrab.cs script (Unity.Simulate.Capture), then DepthGrab will generate only black images.
Do you know the cause of this problem and how could I resolve it?

I have noticed that I am getting some degree of 'ghost labels' and wondering how to avoid that?

1 - I refer to a 'ghost label' as when I get a label showing up on the image, likely due to an object being buried within the distraction layer close to the surface but not actually visible (or at times, just barely visible). Regardless of reason, it's labelling that will detract from the AI training as there is not enough signal there for it to learn from.

2 - This issue may be caused by my need to wrap my detection objects inside a game object in order to have a local and global rotation, rather than simply the object itself...I thus have all of my desired objects laying flat in the scene at the start, and then are moved in and out based on the foreground randomizer.
Thus, is the fix that I need to do everything in code for this instead of having them already in the scene? Or is there a need to reset the state image to image?
Most images don't have it but it's enough that it's a concern.

I'm attaching an image showing what I mean. You can see that while most labels are valid, there are a few (circled in red) where there is no object readily apparent but it has a label. In some cases I can see a tiny sliver of the object peeking through the distraction layer, so it's not a totally random label but for the obj detection, there is not enough visible for it to be a legit signal to learn from and thus a label is not a valid option. (see the bottom most red circle...there is an object peeking out of the distraction layer, but not enough to be valid).

Anyway, any input on how to avoid/fix this would be appreciated to ensure that I'm training only with valid labels.

I was following the instructions given here : https://github.com/Unity-Technologies/com.unity.perception/blob/master/com.unity.perception/Documentation~/GettingStarted.md

However am stuck at Step 4: Create labeled objects, point 4 In the Project panel right click -> Perception -> Labeling Configuration

I see only the following options when i right click the Perception:

Following are the environment details:

• MacOS Catalina 10.15.5 (19F101)
• Unity 2020.1.0f1
• Perception Version 0.2.0-preview.2 - July 16, 2020

Hope I did not miss anything obvious. Would appreciate any help. Thanks!

Following the Randomization Tutorial, I do not see an "Add Randomizer" button in the Fixed Length Scenario Component. Attached is a screenshot of my component:

Unity Version: 2019.4.10f1 Personal
Perception Version: 0.4.0-preview-1
OS: MacOS Catalina

I would like to add a delay (in real-world time, not simulation time) between each iteration/capture. How do I go about this? It's a bit urgent, so any help would be appreciated. Thank you!

Hi - I'm working with the perception package and wanted to ask what is the recommended process to do 2D detection labelling for parts within a given object?
Specifically, the tutorial shows how to detect objects as single units (i.e. this is a whipping cream container, this is a tea container, etc).
For our work we need to label/detect not just the object, but also parts within an object.. - so for the whipping cream example, imagine you need to detect the whipping cream carton, but also the expiry label, the cap on it, the brand label etc.
example with the whipping cream carton:

Can you advise how to set this up? I've made singular fbx assets and not sure how to define labels for not just the object, but for internal parts on the object.

Thanks very much. Quite excited to move more and more of our dataset creation work to synthetic generation!

Here is my code:

I am facing this issue and when I click on the simulation scenario, I am unable to change any of its features because of these errors.
Thanks in advance,
Arjun

Hi,

installing both, the Unity Perception and Jobs package (both 0.5 preview versions, installed on 2020.1.10) leads to a cyclic dependency error. As a consequence, Perception is not working.

Regards, Christian

Hello,

I would like to use this tool to generate YOLO-ready labels. However, I lack the knoweledge on Unity and C# to make a script myself. Is it possible that you already have any tool available for doing this?

Thanks in advance.

The background is a little too random and unstructured when I am trying to create it. Is there any way I can order the objects that are being generated in the background?
Suppose you want to create randomized but organized sets of environments, how do you go about doing that?

Thank you in advance.

Purpose of scenario:
We want to rotate randomly an object and place a Decale Projector on the rotated object.
Therefor we want to receive the objects face normal.

We run a script which we call from our ExplorerEvent (public class ExplorerEvent : Randomizer) to execute after the RotationRandomizer.
This script is using Physics.Raycast to hit the object and get the RaycastHit.point and
RaycastHit.normal to place a Decal Projector.
The Raycast is called in average about 1000 times (per frame) to get different hits on the object because the final place also depends on other factors.

Issue:
By analyzing the scene, we concluded that there is an offset between the frames: The Decal is rotated right to the former (frame 0) rotation of the object but not to the current rotation (frame 1).

We did not get a different result either by reducing the amount of Raycasts (like not 1000 but 1) or using more frames per iteration.
Changes of parameters (like fixed timesteps) in the Time Project Settings have also had no effects.

Please see the attached images.
Here you can see that the Decal Projector in the second frame adopts the normal of the object in the first frame. This pattern runs through all the other frames.

Our guesses:

• Physics engine is not updating within the scenario iteration.
• In the scenarios can be called usual randomizers only not custom Events disguised as randomizers.

We would be grateful if you could help us.
If further information is needed to discuss the problem, we will share it with you.

[

](url)

I am able to generate a huge dataset. Thanks to you guys.
Aim: To train with the Faster RCNN model and I am successful.
Improvements:

1. When the image is generated, we had margin-left and margin-right of the image which are just background which is non-required data. Is there any way we can crop it? or focus mainly on our foreground window only?
2. Certain DNN models required fixed-size input images. (of course, we handle it while preprocessing) But it would be great if we could restrict it somehow to a certain dimension.

Is there any way to improve the labeling of the leaves of trees? Labeling and leaves do not exactly match and sometimes several leaves are not labeled at all (and then they get the label of the object behind them).

I've been closely reviewing the bboxes for our datasets and have noticed that the outermost bounding box is consistently coming up short when the object is heavily rotated.
*I'm using invisible geometry labelling. The invisible labels (sub-components) remain accurate. It's the prefab itself that is coming up short.
Here it is with no rotation:

Here it is at extreme (45 degree) rotation. If you look at the green outer box (the prefab object itself) then I've highlighted the difference of it vs the bottom label. It should be at the same width/height of the bottom box to be accurate:

Another example:

The discrepancy is reduced as rotation drops away from 45 degrees:

Any ideas on how to correct? For now I will synthetise the outer box by building it from the top and bottom boxes which so far appear to remain accurate.
Note - one idea, I'm rescaling the items randomly. Not sure if that impacts, but as noted it appears correct for non-rotated, and then degrades and peaks at 45 degrees rotation.
Thanks!

For the work I'm doing, I'd like to be able to implement two specific controls related to the amount and type of items per randomized image.
Thus, wanted to see if there are any recommendations for how to implement the following types of controls to the randomization for the detectable objects:

1 - A "one of" feature. Example - I have diagnostic test prefabs that show different results. I'd like to be able to have a "one of" selected from each test type, randomly chosen per image. Thus I may have Covid test, model A, and select to show "one of" type model A, but randomly selecting from the model A prefabs that show positive-1, positive-2/negative/invalid, etc. I only want to show a single prefab of this type, not simply have a pct chance that of each various version appearing.

2 - max or explicit quantity per image of detectable objects - in some cases I want to generate images that have only a single detectable item displayed, selected from all the available prefabs.
Another dataset might need only 2 types of items to detect shown, others 3, etc. Is there a way to script or otherwise control this (beyond the make a new project, only add one item, or two items). I'd like to work within a much larger scene and have dynamic control of this.

I get this new log message "Could not get a unique color for 255" (using RC-0.6.0-preview.1). I found that this is printed in InstanceSegmentationCrossPipelinePass.cs and caused by InstanceIdToColorMapping.cs if the instance id is 255. But why is the instance id set to 255 (that is causing this error) and how can this be solved, so that the instance id is not 255?

Thanks to the help on earlier issues, I've been able to label things with invisible geometry (adding transparent quads) such that 'sub-components' on objects are also labelled.
If I only use the generic random rotation, then labelling works as expected in the captures json.
However, in my situation I have to force the prefabs to rotated so that all items are 'facing the camera'. We can't do our object detection from the backside of the object or side of the object.
Once I force it to rotate to face the front to the camera..all my invisible geometry is lost.
I then only get labels for the singular object, and not any of the labelled sub-components.
I've spent hours trying to adjust the quads to be higher on the object, to rotate a containing game object, even rotated the quads as a check...and unfortunately nothing permits the sub-labels to show up once the object is forcibly rotated.
I guess what I'll try is to redo the geometry with the object pre-rotated, but any input or help here would be appreciated.
Summary - invisible geometry works fine if the object is freely randomized in rotation, but forcing a fixed rotation of the X (to make it face front) breaks the labelling for the sub-components.

Hey! Great work there!
I have an issue that I want to generate masks for semantic segmentation image, and output labels like COCO dataset.

I only found how to generate mask photos like above, but cannot generate a proper json file which include the mask information. Can you guys demonstrate it for me? Many thanks to you~

How do we create separate JSON files for each image?

I'm using invisible geometry with quads to label our sub-components.
I noticed that if I set the quad lighting mode to 'shadows only' then perception will no longer treat these as having labels.

I suppose that might be by design, but the reason I was trying to use 'shadows only' was to avoid the un-natural sheen that takes place when run with shadows = off.
Basically the invis geometry will reflect light at some angles showing that there is that upper layer, and not behave like the regular item.
To show - here is the labelled quad with lighting set to 'shadows only' - you can see that there is a sheen where there should be empty space due to the quad:

Now with quad set to shadows only - you have no abnormal sheen and it functions as a better 'invisible geometry' ...except there is no labelling under this lighting setting:

If there's a better way to achieve the goal of labelling while minimizing any lighting artifacts that otherwise increase the sim vs real images, please let me know.
I'll revert to shadows off since having the labelling is more important than minimizing artifacts, but wanted to post this issue for any feedback.
Thanks!

Unity outputs the synthetic data and annotations, which is great, but there are no instructions for how to turn these annotations into something usable for training an object detection or semantic segmentation model.

Hi,

Is there a configuration in perception camera to capture frames only when there's a change?
For some projects, we want to capture and save frames only if there is a change in object position or camera rotation or if there's a change in the pose of objects with labels. If there is, we can reduce the number of redundant images files and have lower C02 footprint.

Hi,

are there plans to support terrains (incl. trees/vegetation/layers) for semantic segmentation?

Regards, Christian

It looks like atm we get JSON exports from creating the synthetic datasets but will have to write a converter to push it into coco?
Most obj detection is based around Coco format so adding this would be useful.
I did see something in a thread saying you had opted against including this (not sure if that's still correct), but thought it was worth opening an express issue to track the status, even if it just gets closed as won't do. If so, are you interested in a PR to add a converter as I'll need to write this regardless at least for 2D obj detection.
I do think it would add a lot of user benefit to be able to export directly to Coco format without running a secondary utility.
Thanks!

Hi,

This project has a lot of potential but at the moment it doesn't seem to allow scripted instantiation of objects with labeling components to be included in the dataset for segmentation/bounding box calculation. If this is a limitation, it's not documented.

Example - I want to generate up to 3000 instances of an object with a dynamically chosen texture on the fly, label these as such (by modifying the labeling component to reflect the texture name) and then move them around to generate training samples. With the Perception package as-is, none of these objects appear in the segmented pngs, though they are obviously present in the rgb images.

I've tried doing my instantiation and labeling in the Awake function and moving my script to the highest priority in the script execution order. It doesn't make any difference.

I'm at a loss for a workaround here, short of manually instantiating 3000 objects in the editor!

I upgraded today to 0.7, preview 2 in order to make sure I am working with the latest codebase.
After hitting issues with my custom code from .6, I decided to just make a whole new project and rebuild from scratch, and then work on importing my earlier code after that.

However, I can't get a basic perception fixed length scenario going with the 2D object labeller as I immediately hit this error:

This is specific to the background placement randomizer. As a result the distraction wall has no textures applied and also appears to be quite blurred:

I'm using 2020 and HDP.
Thanks!