BrickCraft-Duo

Efficient Dual-Arm Skill Learning and Refinement for Compositional Long-Horizon Assembly

Jichuan Yu1, Zhenyu Xiao1, Ze Wang1, Ruixuan Liu2, Changliu Liu2, Chuxiong Hu1
1 Tsinghua University     2 Carnegie Mellon University
Abstract

Interlocking brick assembly provides a representative testbed for evaluating real-world robotic manipulation capabilities, where diverse structural designs, complex inter-step dependencies, intricate mechanical interactions and tight insertion tolerances pose substantial challenges. We present BrickCraft-Duo, a modular framework for long-horizon dual-arm collaborative assembly of interlocking bricks through data-efficient skill learning and composition. BrickCraft-Duo learns reusable single- and dual-arm assembly skills from diverse demonstrations, with bilateral symmetry alignment facilitating skill sharing across symmetric arms and assembly–support role assignments. Guided by stability-aware assembly reasoning, BrickCraft-Duo composes heterogeneous skills to achieve autonomous long-horizon execution, and further integrates human-in-the-loop correction for targeted skill refinement. The resulting system achieves long-horizon success rates of at least 60% and step-level completion rates of at least 95% across five real-world assembly tasks involving partially supported configurations, with horizons of up to nine steps.

Framework Overview

Given a target brick structure \(\mathcal{B}\), BrickCraft-Duo autonomously generates a feasible assembly plan \(\mathcal{P}\) that specifies the assembly sequence and arm assignments. Reusable skills are then instantiated and composed into a skill execution graph \(\mathcal{G}\), which captures execution dependencies within and across arms to enable coordinated long-horizon dual-arm assembly.

BrickCraft-Duo framework: a target brick structure is converted into an assembly plan and compiled into a dual-arm skill execution graph composed of Transit, Pick, UniAssembly, and CoAssembly skills

Two arms are better than one! For partially supported structures, single-arm assembly (UniAssembly) can cause the structure to deform or detach. BrickCraft-Duo predicts such instability through stability-aware reasoning and invokes dual-arm cooperative assembly (CoAssembly), where one arm performs insertion while the other provides support.

UniAssembly ×
→
CoAssembly ✓
UniAssembly ×
→
CoAssembly ✓

One Skill, One Attachment. Each assembly skill focuses on attaching one brick relative to a reference brick. The task encoding \(\boldsymbol{\tau}\) specifies their relative translation, rotation, and brick types. Segmenting and highlighting both bricks provides visual grounding, allowing the policy to focus on the local assembly geometry.

Three brick attachment examples with a task encoding containing relative translation, rotation, and the reference and attaching brick types
Raw workspace-camera view of two robot arms assembling interlocking bricks
Raw Observation
Segmentation mask identifying the attaching brick b_att in cyan and the reference brick b_ref in yellow
Task-Relevant Mask
Spatially grounded observation highlighting the attaching and reference bricks while dimming the background
Visual Grounding

Symmetry-Aligned Skill Learning

Demonstrate once, both arms learn. The left and right arms perform approximately mirrored assembly operations. We apply bilateral symmetry alignment to map demonstrations from both arms and opposite assembly–support role assignments into a common convention. This enables data and policy sharing across arms and role configurations.

Symmetry-aligned skill learning: UniAssembly aligns right-arm demonstrations to the left-arm convention; CoAssembly aligns reversed assembly-support roles through role swapping and bilateral reflection. Both shared policies are conditioned on the task encoding.

Diverse Demonstrations, Reusable Skills. We collect demonstrations across diverse structural configurations, robot arms, and assembly–support role assignments to train the UniAssembly and CoAssembly skills.

UniAssembly Dataset
CoAssembly Dataset

The videos show demonstrations after bilateral symmetry alignment.

Empirical evaluations show that bilateral symmetry alignment improves skill learning across arms and assembly–support role assignments.

UniAssembly

Left onlyRight onlyAligned L+R
UniAssembly success rates Training sets: left-arm demonstrations only, right-arm demonstrations only, and symmetry-aligned demonstrations from both arms. Left-arm success rates: 60%, 20%, 85%. Right-arm success rates: 5%, 65%, 85%. Each setting is evaluated over 20 trials. 0 20 40 60 80 100 Success rate (%) Left: Left only, 60% 60% Left: Right only, 20% 20% Left: Aligned L+R, 85% 85% Left Right: Left only, 5% 5% Right: Right only, 65% 65% Right: Aligned L+R, 85% 85% Right

CoAssembly

UnalignedSymmetry-aligned
CoAssembly success rates Success rates with unaligned and symmetry-aligned demonstrations. Left assembly, right support: 45% and 75%. Right assembly, left support: 40% and 70%. Each setting is evaluated over 20 trials. 0 20 40 60 80 100 Success rate (%) Left: Unaligned, 45% 45% Left: Symmetry-aligned, 75% 75% Left assembly Right support Right: Unaligned, 40% 40% Right: Symmetry-aligned, 70% 70% Right assembly Left support

Targeted Corrections, Refined Skills. A human operator corrects underperforming assembly steps through SpaceMouse teleoperation. The corrective trajectories are combined with the original demonstrations to fine-tune the corresponding reusable skills.

UniAssembly
CoAssembly
Autonomous Long-Horizon Assembly

BrickCraft-Duo composes the same set of reusable skills to autonomously assemble diverse structures through coordinated dual-arm execution.

16×
Snake4 assembly steps
InitialRefined
Success Rate5/5—
Completion Rate100%—
16×
Gate8 assembly steps
InitialRefined
Success Rate1/54/5
Completion Rate85.0%97.5%
16×
Toy Gun4 assembly steps
InitialRefined
Success Rate4/5—
Completion Rate95.0%—
16×
Scale9 assembly steps
InitialRefined
Success Rate0/54/5
Completion Rate75.6%97.8%
16×
Eagle9 assembly steps
InitialRefined
Success Rate0/53/5
Completion Rate73.3%95.6%

Completion Rate: the ratio of successfully completed assembly steps to the total number of assembly steps in a long-horizon task.

Adaptation to Workspace Perturbations

Starting from a coarse pre-assembly pose, the learned assembly skills use closed-loop visual feedback to adapt to changes in the local assembly geometry without requiring precise workspace calibration. The videos below illustrate adaptation to structure and baseplate perturbations.

Structure Perturbation

Between assembly steps, we move the assembled structure to a new position on the baseplate.

8×
8×

Baseplate Perturbation

We repeatedly move the baseplate during assembly.

8×
8×