AutoTrialGen: Automated Data Generation from Few Human Demonstrations via Trajectory Annotation and Simulation Trials

Huailiang Ma, Aiguo Song, Mutian He, Yibing Yan, and Linhu Wei

Southeast University · Shanghai Innovation Institute

Paper Video BibTeX

Abstract

Imitation learning is a powerful paradigm for teaching robots complex manipulation skills, but it is often bottlenecked by the need for large-scale, human-collected datasets. AutoTrialGen is an automated framework that generates large and diverse datasets of successful simulated demonstrations through trial-and-error from only a few human demonstrations.

The framework leverages foundation models to decompose raw human demonstrations into reusable, object-centric skill primitives, then intelligently composes these primitives in simulation using a weighted manipulability selection mechanism. Policies trained on the sim-augmented data achieve stronger data efficiency and real-world manipulation performance than existing generation pipelines.

Framework Overview

AutoTrialGen overview — AutoTrialGen starts from a few teleoperated demonstrations, decomposes them into reusable object-centric primitives, generates successful simulated trials, and trains policies for real-world deployment.

Automatic Decomposition

Vision-language models generate subtask termination discriminators and segment demonstrations into object-centric skill primitives.

Real-to-Sim Transfer

Digital twin scenes and reconstructed object assets enable randomized simulation trials with reusable skill trajectories.

Weighted Skill Composition

Candidate primitives are selected by balancing kinematic manipulability and transition efficiency, reducing singular and inefficient motions.

AutoTrialGen pipeline — The full pipeline combines VLM-assisted trajectory annotation, simulation-based data generation, and policy learning for real-world manipulation.

Demonstrations

Dataset Generation I

Dataset Generation II

Beverage Placement

Cup Hanging

Experimental Results

Experimental tasks — Six manipulation tasks are evaluated in simulation and the real world: button pressing, mouse placement, cup hanging, beverage placement, storage-box organization, and tea pouring.

Task	MimicGen	AutoTrialGen
Button pressing	80.0%	90.0%
Mouse placement	54.0%	88.0%
Cup hanging	62.0%	74.0%
Beverage placement	40.0%	64.0%
Storage box organization	42.0%	68.0%
Tea pouring	68.0%	86.0%
Average	57.7%	78.3%

Weighted manipulability selection filters unstable or inefficient candidates and selects better skill primitives.

Data scale results — Policies trained with generated data improve with scale and can substitute costly human teleoperation data.

Ablation results show that manipulability and pose proximity both contribute to stronger task performance.

Citation

@article{ma2025autotrialgen,
  title   = {AutoTrialGen: Automated Data Generation from Few Human Demonstrations via Trajectory Annotation and Simulation Trials},
  author  = {Ma, Huailiang and Song, Aiguo and He, Mutian and Yan, Yibing and Wei, Linhu},
  journal = {IEEE Robotics and Automation Letters},
  year    = {2025}
}