Interlocking brick assembly provides a representative testbed for evaluating real-world robotic manipulation capabilities, where diverse structural designs, complex inter-step dependencies, intricate mechanical interactions and tight insertion tolerances pose substantial challenges. We present BrickCraft-Duo, a modular framework for long-horizon dual-arm collaborative assembly of interlocking bricks through data-efficient skill learning and composition. BrickCraft-Duo learns reusable single- and dual-arm assembly skills from diverse demonstrations, with bilateral symmetry alignment facilitating skill sharing across symmetric arms and assembly–support role assignments. Guided by stability-aware assembly reasoning, BrickCraft-Duo composes heterogeneous skills to achieve autonomous long-horizon execution, and further integrates human-in-the-loop correction for targeted skill refinement. The resulting system achieves long-horizon success rates of at least 60% and step-level completion rates of at least 95% across five real-world assembly tasks involving partially supported configurations, with horizons of up to nine steps.
Given a target brick structure \(\mathcal{B}\), BrickCraft-Duo autonomously generates a feasible assembly plan \(\mathcal{P}\) that specifies the assembly sequence and arm assignments. Reusable skills are then instantiated and composed into a skill execution graph \(\mathcal{G}\), which captures execution dependencies within and across arms to enable coordinated long-horizon dual-arm assembly.
Two arms are better than one! For partially supported structures, single-arm assembly (UniAssembly) can cause the structure to deform or detach. BrickCraft-Duo predicts such instability through stability-aware reasoning and invokes dual-arm cooperative assembly (CoAssembly), where one arm performs insertion while the other provides support.
One Skill, One Attachment. Each assembly skill focuses on attaching one brick relative to a reference brick. The task encoding \(\boldsymbol{\tau}\) specifies their relative translation, rotation, and brick types. Segmenting and highlighting both bricks provides visual grounding, allowing the policy to focus on the local assembly geometry.
Demonstrate once, both arms learn. The left and right arms perform approximately mirrored assembly operations. We apply bilateral symmetry alignment to map demonstrations from both arms and opposite assembly–support role assignments into a common convention. This enables data and policy sharing across arms and role configurations.
Diverse Demonstrations, Reusable Skills. We collect demonstrations across diverse structural configurations, robot arms, and assembly–support role assignments to train the UniAssembly and CoAssembly skills.
The videos show demonstrations after bilateral symmetry alignment.
Targeted Corrections, Refined Skills. A human operator corrects underperforming assembly steps through SpaceMouse teleoperation. The corrective trajectories are combined with the original demonstrations to fine-tune the corresponding reusable skills.
BrickCraft-Duo composes the same set of reusable skills to autonomously assemble diverse structures through coordinated dual-arm execution.
Completion Rate: the ratio of successfully completed assembly steps to the total number of assembly steps in a long-horizon task.
Starting from a coarse pre-assembly pose, the learned assembly skills use closed-loop visual feedback to adapt to changes in the local assembly geometry without requiring precise workspace calibration. The videos below illustrate adaptation to structure and baseplate perturbations.
Between assembly steps, we move the assembled structure to a new position on the baseplate.
We repeatedly move the baseplate during assembly.