Spec sheet for integration of graspgen and manipulation module with dimos for end to end grasping baseline version

[linear]

[JalajShuklaSS]

Summary

Introduce a single agent-facing pick() skill that performs end-to-end grasp execution using an internal Behavior Tree, with retries, recovery, and safe cancellation, would also try to have simulation parallely.

Final Design Decisions

• Orchestration: Behavior Tree inside pick() (no external BT runner) Similar would be for place() rather simpler.
• Module Choice: Dedicated PickPlaceModule keeping ManipulationModule focused on planning/execution
• Agent Control: High-level skills only (pick, stop, detect objects)
• Low-level Skills: Unchanged, internal-only in Agent Mode
• BT Action Semantics: Leaves follow navigation-style start -> status(poll) -> cancel (tick-driven, interruptible)
• Blueprints: Separate Agent vs Debug surfaces

Critical Rule Assumption Made:

The VLM never sequences low-level skills.
All grasp logic ( graspgen -> plan -> execute -> gripper -> lift -> recovery) is encapsulated inside pick().

Architecture

Modules:

• Perception / ObjectSceneRegistration
• GraspGenModule (Docker)
• GraspingModule (generate_grasps)
• ManipulationModule (plan / execute / cancel)
• Control Coordinator Module (whole hardware interface and control)
• PickPlaceModule (BT-based orchestration)

Blueprints:

• Agent Mode:pick, stop
• Debug Mode: full low-level access

Interfaces

Agent-facing

• pick(object_name | object_id, max_attempts, dry_run)
• stop()

Internal (BT leaves)

• generate_grasps
• plan_to_pose
• execute + get_state
• open / close

Phases

• P0: Agent vs Debug blueprints
• P1: TF / frame correctness
• P2: Pick BT (dry-run)
• P3: Execution + gripper
• P4: Retry + recovery
• P5: Hardware validation

Acceptance Criteria

• Agent request “pick object” calls only pick()
• End-to-end pick + lift succeeds on hardware with interaction only through humancli
• Automatic fallback to next grasp on failure
• stop() reliably cancels motion and opens gripper
• Structured success/failure returned

User Experince

User: Can you see the cup?
Agent: Let me detect the objects in front.....Yes I can see the cup.
User: Can you pick it up?
Agent: Success

[mustafab0]

We should separate the agentic integration and the graspgen integration with the manipulation module.

For the initial implementation we just want to have graspgen skill integrated with the manipulation module. The agentic integration would be a next step. This is a good start.

[JalajShuklaSS]

Here this is implemented check the PR: #1234