Reset errors

[maleroy]

Hello all,

I've been having some strange behavior in my SUPERball_CPG branch (latest commit fd62e0c). Specifically with the AppSUPERballPosition app, which I call with a YAML file.

I've been trying to run multiple episodes with the same controller parameters (that are manually set), just to check that I get the same traveled path between runs, but each episode gives a different total distance.

I checked that my time counters are correctly reset to 0 between runs, as well as my controller's states. I also check that the model's initial position is always the same.

In case somebody wants to try to re-create these problems, what you should be looking for is that when you run the app (by typing once the branch is cloned ./build/dev/mleroy/SUPERballPosition/AppSUPERballPosition ./resources/YamlStructures/mleroy_test3.yaml), it will run 3 episodes. Some details about each of those runs are as follows:

• On my onSetup function, my controller parameters are always set to the same values (which will be output on the console);
• Two values are also cout-ed and correspond to states of my controller at the same timestamp (also to check that they are equal between runs);
• The CoM position is then also cout-ed at another timestamp, along with all states of the controller. Even though states are equal, positions are not;
• The polar distance traveled is then cout-ed at the end of the 30s run.
  (These are all implemented within src/dev/mleroy/SUPERballPosition/HopfControllerML.cpp/h)

I also reset the time counters and controller states between runs, and check that the model's initial position is always the same at time t=0.
(These are called in src/dev/mleroy/SUPERballPosition/AppSUPERballPosition.cpp) My guess is that the error is at some point with the reset call.

You'll see that if you call ./build/dev/mleroy/SUPERballPosition/AppSUPERballPosition ./resources/YamlStructures/mleroy_test3.yaml multiple times, you'll always get the same x1, x2 and x3 total distances traveled, but that x1!=x2!=x3.

Thanks in advance!

[maleroy]

If somebody is trying to check into this, I've tried to recreate the same issues on simpler models, specifically the three bar example from the NTRT Intro Seminar. The updated code is in commit 21abd62.

In red you'll see the model's CoM at time t=0, and in green its CoM when the actual lengths become stable (which always seem to be at time t=6.718 for the parameters chosen for these runs). Yet again the CoM are not equal between episodes...

Does anybody knows what could be causing this?

[apsabelhaus]

Hey @maleroy, I plan to look at this once I'm back from a conference this week. Will likely get a chance to look next Monday. If this is really holding you up, feel free to dig into core a bit more on your own. I'll have thoughts soon.

[maleroy]

Will do! Thanks! :)

[apsabelhaus]

Hello everyone,

Zakary Littlefield from Rutgers has explored this issue in-depth, and apparently, the underlying non-determinism comes from Bullet Physics. He's provided some work in making Bullet more deterministic. I've placed his patch online at

https://gist.github.com/apsabelhaus/719ca07f430105f6ddf3d98db7e366d8

You'll need to follow Zak's instructions below to use it.

Zak has also provided a new version of the bullet setup script, you'll need it (I think):
https://gist.github.com/apsabelhaus/925a829f441837c4cdbe7de7eaee8c9d

If someone tests this, can push to a branch, or can incorporate into master, it would be much appreciated.

Zak's original email:

Determinism in physics engines has been the bane of most of my Ph.D work, and hopefully I can help you out.

I was able to get determinism out of NTRT, but it wasn't actually NTRT's fault (mostly). It was actually the Bullet physics engine.

I am attaching a patch file (which needs to go into ./bin/setup/patches/) and the updated setup script for Bullet (for ./bin/setup/). You will then need to re-setup NTRT so that Bullet is rebuilt with the patch.