Support for multiple worlds about gz-sim HOT 13 OPEN

I am not sure if ODE was chosen just for historical reasons or indeed the is the best performing collision detector for Gazebo's needs.

If I remember correctly, it was chosen because it worked much better than DART or FCL as collision detectors. I remember FCL converted primitive shapes to meshes, which caused unexpected behaviors. And I think DART as a collision engine is missing some features. It's been a while though, so it might be worth revisiting. Also, bullet might be a viable option, but upstream dartsim needs some changes to make mesh-to-mesh collisions work.

from gz-sim.

Original comment by Addisu Z. Taddese (Bitbucket: azeey, GitHub: azeey).

Looks like there is a global cache used by the collision detector in ODE. One thing to try would be to build ode with --enable-ou,

We recently experienced this again with @xela-95 in robotology/gz-sim-yarp-plugins#153 . A workaround is to change DART's collision detector not to use ODE . At least on conda-forge, it is relatively easy to set ENABLE_OU option in CMake, I checked and it does not affect the public ABI of ode, while on apt is more complex and can't be done for stable distros. However, I am not sure if this is helpful in the case two servers are stepped by the same thread (unless a new thread is always created by the server, in that case indeed that would solve the problem).

from gz-sim.

Original comment by Michael Grey (Bitbucket: mxgrey, GitHub: mxgrey).

This might be something that needs to be addressed in the ignition-physics-dartsim plugin.

Is this happening due to multi-threading, or will it still happen even with only a single thread? If it's a multi-threading race condition issue, then maybe we just need to put a mutex in a sensible place in the ignition-physics-dartsim plugin.

from gz-sim.

Original comment by Addisu Z. Taddese (Bitbucket: azeey, GitHub: azeey).

I believe this is a race condition on global variables in ODE. I was experimenting with a solution here (914d44a), but ignition-physics-dartsim is probably a more appropriate place to put the mutex.

from gz-sim.

Original comment by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

By the way, the same problem happens if we try to spin up 2 secondary servers running physics simulation in the same test, to test distributed simulation.

from gz-sim.

Original comment by Diego Ferigo (Bitbucket: dgferigo).

Is there any update on this problem? I started testing multi-world simulations and I stumbled upon this problem right away.

If this feature is not yet completed, it would be nice at least having a warning printed when the world file is loaded and multiple worlds are found.

from gz-sim.

Original comment by Diego Ferigo (Bitbucket: dgferigo).

Thanks @azeey for the fix you proposed. I know it’s a hacky workaround, though it works reliably. Do you see any better solution that does not involve static mutexes?

from gz-sim.

Original comment by Diego Ferigo (Bitbucket: dgferigo).

Quick update. With fixed-base simple models like pendulum and cartpole the mutex workaround is effective as I reported in my last comment.

However, with complex floating-base models like a humanoid robot with meshes both for visual and collisions, a multiworld setup still segfaults.

I managed to “isolate” the problem, if we can say it. In PhysicsPrivate::Step, locking the same mutex in this scope prevents the segfault. Though, this is not a solution since it means that the real step of the world is not concurrently executed by the threads that handle the different worlds (one for each SimulationRunner). In other words, this mutex prevents stepping the worlds in parallel. I think that parallel execution within a single process is the main feature that triggers downstream users to use a multiworld setup.

Here below the stack trace (text here):

‌

from gz-sim.

Original comment by Addisu Z. Taddese (Bitbucket: azeey, GitHub: azeey).

Looks like there is a global cache used by the collision detector in ODE. One thing to try would be to build ode with --enable-ou,

from gz-sim.

Original comment by Diego Ferigo (Bitbucket: dgferigo).

Thanks for the suggestion! From my side recompiling ODE would mean that all the chain ODE → DART → ign-physics should be compiled from sources. In this moment I doubt I can dedicate enough time to set everything up and start debugging the problem (also considering that ODE code is not very straighforward to read) 😕

from gz-sim.

One thing that we could think of at some point in the future (clearly not in an already released version of Gazebo) is to switch the default collision detector in DART from ODE to something else, I am not sure if ODE was chosen just for historical reasons or indeed the is the best performing collision detector for Gazebo's needs.

from gz-sim.

remember FCL converted primitive shapes to meshes, which caused unexpected behaviors.

That seems to be still the case, see the corresponding DART issue: dartsim/dart#19 (that seems to be blocked by something, but I am not sure which FCL issue, perhaps flexible-collision-library/fcl#106 ? ).

from gz-sim.

Yes, indeed looking at bit at issues it seems that each collision detector has some issues, so there is no clearly better DART collision detector, xref: #1306 . Also in https://github.com/resibots/robot_dart it seems that the collision detector used changes from example to example.

from gz-sim.