I built system which uses a Realsense camera and Interbotix px-100 arm to pick up a pen held before it. This system was built in 4 days. The first 2 weeks of the Masters of Science in Robotics program at Northwestern University are an extended orientation-hackathon, the ultimate portion of which is this challenge.
The pen was identified by its distinctive purple color, which was used to create an image mask overlaid onto the RealSense image. The mask was then dilated in order to lessen the effects of noise. The center of the largest resulting contour was then presumed to be the pen. The real-world centerpoint was identified using the RealSense’s intrinsic and depth information.
Prior to grabbing the pen, the system must be calibrated. This was done by having the robot carry the pen to several points within view of the camera. At each point, the robot-to-pen and camera-to-pen transform was recorded; by comparing these point collections, the camera-to-robot transform could be determined.
The code (in its full hacked-together mess, test and demo files and all) can be found here