Error message

Deprecated function: The each() function is deprecated. This message will be suppressed on further calls in menu_set_active_trail() (line 2405 of /home/pracsysl/public_html/includes/

CS 672 - Seminar on Algorithmic Robotics

Focus on Robot Manipulation

In artificial intelligence, an intelligent agent is an autonomous entity, which observes its environment through sensors, acts upon the environment using actuators and directs its activity towards achieving goals, i.e. it is "rational". A robot manipulator is the prototypical embodiment of an intelligent agent. Solving robot manipulation problems requires reasoning about the physical world and building upon foundational mathematical principles. At the same time, one can use as inspiration the impressive manipulation capabilities of biological organisms.

Description of manipulation from the book "Mechanics of Robotic Manipulation" by Matt Mason: Manipulation is the process of using one's hands to rearrange one's environment. There are many facets to manipulation. Manipulation is an art, since it is practiced by all of us, without any systematic or fundamental understanding of the process. Manipulation is also an engineering discipline, for there are some systematic tools for applying robotic manipulation to various problems. Finally, manipulation is also a science, since it is a process that engages our curiosity, which we can explore using scientific methods.

Motivation for Robot Manipulation

Commercially available robotic toys and vacuum cleaners inhabit our living spaces, and robotic vehicles have raced across the desert. These successes appear to foreshadow an explosion of robotic applications in our daily lives, but without advances in robot manipulation, many promising robotic applications will not be possible. Whether in a domestic setting or the workplace, we would like robots to physically alter the world through contact. Robots have long been imagined as mechanical workers, helping us in our daily life. Research on manipulation in human environments may someday lead to robots that work alongside us, extending the time an elderly person can live at home, providing physical assistance to a worker on an assembly line, or helping with household chores.

Within factories around the world, robots perform heroic feats of manipulation on a daily basis. They lift massive objects, move with blurring speed, and repeat complex performances with unerring precision. Yet outside of carefully controlled settings, even for the most sophisticated robot it would be difficult to get a glass of water. The everyday manipulation tasks we take for granted would stump the greatest robot bodies and brains in existence today. Why are robots so incompetent in unstructured, human environments?

The above motivating material for robot manipulation comes from the paper "Challenges for Robot Manipulation in Human Environments" by Kemp, Edsinger, Torres-Jara, IEEE Robotics and Automation Magazine, March 2007.

Course Description

This Computer Science seminar will cover fundamental mathematical and algorithmic techniques that are important for achieving autonomous manipulation with modern robots. The course will be split into the following components:

  • Mathematical foundations of robot manipulation: i.e., mathematics of rigid motion, forward and inverse kinematics solutions, the notions of a manipulator Jacobian, velocity kinematics and manipulability will also be discussed.
  • Motion and task planning for manipulation: i.e., computing paths for robotic arms, trajectory generation, pick-and-place manipulation, rearrangment and multi-arm manipulation, quasi-static and dynamic manipulation, considering more complex hands, etc.
  • Interaction with sensing and vision: i.e., object detection and pose estimation, manipulation under uncertainty, visual servoing foundations, etc.

Beyond lectures, the course will utilize light homework assignments and a test during the semester in order to provide an opportunity to review the course's material. Furthermore, there will be a project during the second component of the course.

Robotics Challenge

The instructor, together with the research group of Dr. Jingjin Yu, are considering to participate in a robot manipulation competition representing Rutgers University using available hardware and sensors in the Computational Biomedicine, Imaging and Modeling (CBIM) center. The hardware includes dual-arm manipulators, such as a Baxter robot by Rethink Robotics, robotic hands, such as a ReFlex hand by RightHand Robotics and RGBD sensors, such as Kinect.

The tentative plan is for the course projects to be related with the "Tidy Up My Room" challenge.

Tentative Calendar

Check Rutgers' Key Dates.

  • January 22: Introduction to course's material
  • January 29: Transformations
  • February 5: Forward Kinematics - DH Parameters
  • February 12: Outline of IK strategies - A Manipulator's Jacobian
  • February 19: Inverse Velocity and Acceleration - Analytical IK - Potential Field Methods
  • February 26: Sampling-based Planning for Manipulation -
  • March 5: Image Formation Process and Camera Calibration - Intro to Visual Servoing
  • March 19: Midterm - Visual Servoing - Object Pose Estimation
  • March 26: Paper presentations
  • April 2: Paper presentations
  • April 9: Paper presentations
  • April 16: Paper presentations
  • April 23: Paper presentations
  • April 30: Review of Material

List of candidate papers:

  • "CHOMP: Covariant Hamiltonian Optimization for Motion Planning" by Matthew Zucker, Nathan Ratliff, Anca Dragan, Mihail Pivtoraiko, Matthew Klingensmith, Christopher Dellin, J. Andrew (Drew) Bagnell and Siddhartha Srinivasa, International Journal of Robotics Research, May, 2013
  • "Motion Planning with Sequential Convex Optimization and Convex Collision Checking", by John Schulman, Yan Duan, Jonathan Ho, Alex Lee, Ibrahim Awwal, Henry Bradlow, Jia Pan, Sachin Patil, Ken Goldberg, Pieter Abbeel.
    In the International Journal of Robotics Research (IJRR), Vol. 33, No. 9, pp. 1251-1270, Aug. 2014.
  • "Single- and dual-arm motion planning with heuristic search" by Benjamin Cohen, Sachin Chitta and Maxim Likhachev, International Journal of Robotics Research (IJRR), 2013.
  • "Manipulation planning with probabilistic roadmaps" by T Siméon, J Cortés, JP. Laumond, A Sahban,. The International Journal of Robotics Research, Vol.23, N°7-8 (2004).
  • "Task Space Regions: A Framework for Pose-Constrained Manipulation Planning" by Dmitry Berenson, Siddhartha Srinivasa, and James Kuffner, International Journal of Robotics Research (IJRR), Volume 30, Number 12, pp. 1435 - 1460, October, 2011.
  • "Global manipulation planning in robot joint space with task constraints" by M Stilman, IEEE Transactions on Robotics 26 (3), 576-584.
  • "Elastic Roadmaps - Motion Generation for Autonomous Mobile Manipulation" by Yuandong Yang and Oliver Brock, Autonomous Robots 28(1):113-130, 2010.
  • "From caging to grasping" by A Rodriguez, MT Mason, S Ferry. The International Journal of Robotics Research 31 (7), 886-900, 2012
  • "A planning framework for non-prehensile manipulation under clutter and uncertainty", by MR Dogar, SS Srinivasa, Autonomous Robots 33 (3), 217-236, 2012
  • "A Framework for Optimal Grasp Contact Planning" by K Hang, JA Stork, NS Pollard, D Kragic, IEEE Robotics and Automation Letters 2 (2), 704-711, 2017.

Text and Reading Material

No specific textbook is required. Students are expected to study the papers that are recommended for each lecture. Regular attendance and active participation during the meetings is expected. Specific material related to the corresponding topics will become available via Sakai.

Deliverable & Grades

The first half of the course will focus on book-based material, on which you will be evaluated given homeworks (30% of the course grade) and a midterm (20% of the course grade). The second half of the course will focus on research papers and a project, and you will be evaluated based on the project's quality, presentation and report (30% of the course grade) as well as your in-class presentation of a research paper (20% of the course grade). There will also be 5% bonus points for active in-class participation. Furthermore, extra credit can be awarded for impressive effort and performance in the project.

The students should submit typesetted reports using LaTeX. Resources on using LaTeX are available below. You are expected to use IEEE RAS' double column format for conferences.

Submissions will be processed through Sakai. Late project report will not be accepted. Final project presentations cannot be easily rescheduled. Unless there are unprecedented circumstances, no incomplete grade will be awarded and students will be graded based on the submitted reports and presentations up to the exam week.

As a rough guide, the following rule may be used for the final grade (this is tentative):

  • A: >= 90
  • B+: 80-89
  • B: 70-79
  • C+: 60-69
  • C: 50-59
  • D: 40-49
  • F: less than 40

Academic Standards

Discussions on the class material, on project challenges and programming practices are highly encouraged. Furthermore, students should try to exchange feedback on drafts of their project reports. Nevertheless, each student needs to eventually independently develop one's proposed methodology, code the corresponding solution and execute the corresponding experiments.

You should carefully study the website of Rutgers University on Academic Integrity and the corresponding policy, as well as the corresponding policy from the department of Computer Science. Your continued enrollment in this course implies that you have read these policies, and that you subscribe to the principles stated therein.

Latex Resources

General info on what you can do with LaTeX:
Getting Started with Latex
The Not So Short Introduction to Latex
Comprehensive List of Latex Symbols
Latex for Logicians

Tex on Mac OS X

The first link describes many alternatives that are available for installing Tex on a Mac. The second link forwards to the MacTex package, one of the alternatives mentioned in the first website. MaxTex provides everythink that you need to use Latex on Mac except from a text editor. It is, however, compatible with a wide variety of popular editors (e.g., Alpha, BBEdit, Emacs, VIM, iTeXMac, TeXShop). Note that MaxTex is a large package.

Carbon Emacs has been succesfully tested with MacTex. After installing MacTex, it is possible to directly compile and view *.tex files from Carbon Emacs's UI.

Note for Mac users: You will probably have problems previewing your PDF output when using the postscript images provided by the instructor for developing the notes. Nevertheless, the PDF file can be printed properly. Prepare your document without the images and then add them. You will probably still be able to preview the intermediate .dvi output file with the "xdvi" program.

Linux (Ubuntu)
Latex on Ubuntu
Tex Live

You just have to download and install the proper packages described above (e.g., through apt-get), use your favorite editor (e.g., emacs) to prepare a *.tex file and then you compile (run at least two times: "latex filename.tex") to get the *.dvi output. You can go from dvi to postscript with the command "dvips" and you can convert postscript to pdf with the command "ps2pdf".

Latex for Windows help
MikTex (Latex for Windows)

If you follow the instructions on the first link you should be able to get it working on a Windows system.

Below you can find Windows executables (32 bit) for the following programs (follow the order when installing):