The first 2 weeks of the Masters of Science in Robotics program at Northwestern University are an extended orientation-hackathon, the ultimate portion of which is to use a Realsense camera and Interbotix px-100 arm to pick up a pen held before it. This system was built in 4 days.
The pen was identified by its distinctive purple color, and its location relative to the arm determined using the RealSense’s depth-image capability. An automatic calibration routine was required to be run before grabbing could commence.
The code (in its full hacked-together mess, test and demo files and all) can be found here