This project aims at pose-aware robotic manipulation for a category of objects. In contrast to most existing methods that must contain an explicit pose estimation, we define the object target configuration on top of semantic keypoints. In this way, the proposed pipeline can handle potentially unknown objects with substantial variations on shape and size, as shown in the demo below.

Lucas Manuelli*, Wei Gao*, Pete Florence, and Russ Tedrake, "kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation", International Symposium on Robotics Research (ISRR) 2019 [Project][Paper][Video]

The code is distributed into several modularized packages

• mrcnn_ros: The ros binding of maskrcnn-benchmark
• mankey_ros: The keypoint detection for the pipeline
• kplan_ros: The kPAM action planner
• mrcnn_integrate: The code to generate data and train the maskrcnn-benchmark instance segmentor

In our experiment setup, these packages are used as the submodules of spartan, which provides the interface to our robots. Although spartan contains private submodules and is not supported for external use, all packages above are not specified to particular environments and can run independently.

The test data, trained model and run instruction for each package are provided in its own repository. Following the manipulation pipeline, the instructions are:

• Run object detection from raw images
• Perform 3D keypoint detection given segmented objects
• kPAM action planning given 3D keypoints
• Grasp planning

After these operations, the remaining task is to apply the planned action to the grasped object. In kPAM, the action is essentially a rigid transformation on the robot end-effector. This part is platform-specific and you might implement it according to your own robot interface.

The instruction to setup the dataset is available here. The code for training the instance segmentation and keypoint detection network are available in their own repo. Please follow the instruction accordingly.