About data collection.

[qyx1121]

Hi,

I am working on collecting trajectory data for a Franka Panda robot arm, specifically in end-effector mode with 7-dimensional actions. So far, I have explored a few approaches, but I encountered some challenges:

Using the StarterSemanticActionPrimitives API:
I tried completing tasks with this API and used the DataCollectionWrapper and DataPlaybackWrapper to record trajectory data. However, this approach only supports the Fetch robot. It doesn't allow for end-effector-based control and is limited to the JointController. Additionally, the InverseKinematicsController isn't usable in this setup, which means the gripper can't effectively grab objects.

Using Keyboard Control:
I also experimented with manually controlling the robot arm using the keyboard. This generates 7-dimensional action data, but the data is discrete (e.g., [0, 0, 0, 0.5, 0, 0, 1]) and doesn't align with the continuous action space required for traditional VLA models.

Given these challenges, I would like to ask for guidance on how to collect trajectory data in continuous end-effector mode (7 dimensions). Is it possible to use the DataCollectionWrapper and DataPlaybackWrapper to record the robot's "proprioceptive" state during keyboard control and convert it into continuous end-effector actions?

[hang-yin]

Hi @qyx1121 ,

first off, by "continuous end-effector actions", do you mean a trajectory of eef poses/waypoints? If so, no matter which way you choose to control the robot, you can always record the eef poses of the robot during data collection or replay.

If you were to use a scripted policy to collect your demo, there are some other alternatives:

1. Just load your robot with a IK controller and specify your eef goal as action for each step
2. Skip our controller stack and directly control the robot with IK; an example of this is omnigibson/examples/robots/advanced/ik_example.py
3. We are in the process of incorporating cuRobo into OmniGibson. This would allow you to generate collision-free trajectories by specifying target eef poses. On our develop branch (og-develop), we already have an initial version of this working. If you are interested, you can take a look at omnigibson/examples/robots/curobo_example.py on og-develop. Note that 1. you will need to install curobo following our setup guide (would probably be easier if you install omnigibson from scratch with og-develop) 2. this only supports a few mobile manipulators at this point; we will add more support in the near future (@ChengshuLi can comment more on this).

[qyx1121]

Thanks for your reply!
I tried running omnigibson/examples/robots/curobo_example.py and selected Tiago, but it seemed that this file was missing: omnigibson/data/assets/models/tiago/tiago_description_curobo_base.yaml

[qyx1121]

BTW, Can IKSolver be used with DataCollectionWrapper?

[qyx1121]

BTW, Can IKSolver be used with DataCollectionWrapper?

I mean, when I use the IKSolver, it calculates the target joint positions and moves the arm directly to the target position. Is there a way to obtain the trajectory generated during this movement?

[Bailey-24]

hello, could you please tell me how do you use keyboard to collect data.
My script is
collect.txt.
But it stuck, and i could not control the robot.