CS 598 - Topics in AI: Introduction to Robot Manipulation

About Robot Manipulation

In artificial intelligence, an intelligent agent is an autonomous entity, which observes its environment through sensors, acts upon the environment using actuators and directs its activity towards achieving goals, i.e. it is "rational". A robot manipulator is the prototypical embodiment of an intelligent agent. Solving robot manipulation problems requires reasoning about the physical world and building upon foundational mathematical principles. At the same time, one can use as inspiration the impressive manipulation capabilities of biological organisms.

Description of manipulation from the book "Mechanics of Robotic Manipulation" by Matt Mason: Manipulation is the process of using one's hands to rearrange one's environment. There are many facets to manipulation. Manipulation is an art, since it is practiced by all of us, without any systematic or fundamental understanding of the process. Manipulation is also an engineering discipline, for there are some systematic tools for applying robotic manipulation to various problems. Finally, manipulation is also a science, since it is a process that engages our curiosity, which we can explore using scientific methods.

Course Description

This graduate course will cover fundamental mathematical and algorithmic techniques that are important for achieving autonomous manipulation with modern robots. The course will be split into the following components:

  • The first component will deal with mathematical foundations of robot manipulation:
    • Mathematics of rigid motion, forward and inverse kinematics solutions.
    • The notions of a manipulator Jacobian, velocity kinematics and manipulability will also be covered.
  • The second component will deal with motion planning and vision-based challenges:
    • Discussion of motion planning methods and trajectory generation approaches for robot manipulators.
    • Visual servo control methodologies.
  • The third component will touch upon robot dynamics and decoupled trajectory planning for robot manipulators.

Beyond lectures, the course will utilize light homework assignments during the semester in order to provide an opportunity to review the course's material. Furthermore, there will be a project that will involve a robotic competition using the Baxter robot manipulator available at CBIM.

Tentative Calendar

Check Rutgers' Key Dates.

  • September 1: [1] Introduction to Manipulation & Mathematical Modeling of Robots
  • September 8 (makeup lecture): [2] Rigid Motion and Homogeneous Transformations
  • September 15: “A Gentle Introduction to ROS” by Jason O’Kane (a lecture provided by the TA Rahul Shome)
  • September 22: [3] Kinematic Chains, The Denavit-Hartenberg Convention and Forward Kinematics
  • September 29: No lecture
  • October 6: No lecture
  • October 13: [4] Inverse Kinematics
  • October 20: [6] Angular Velocity Representations, The Jacobian and Singularities, Manipulability
  • October 27: [7] C-Space Planning and Potential Fields
  • November 3: [8] Sampling-based Approaches
  • November 10: [9] Useful Computer Vision Principles for Manipulation
  • November 17: [10] Visual Servo Control
  • November 24: [11] Robot Manipulator Dynamics
  • December 1: [12] Decoupled Trajectory Planning
  • December 8: Review and Make-up lecture

During exam period: Demonstration day and Competition on Baxter system.

Reading Material

The course will primarily draw upon material from the following book:

  • "Robot Modeling and Control" by Mark Spong, Seth Hutchinson and Mathukumalli Vidyasagar

The book is recommended but not required. Students are encouraged to take notes during lectures. Regular attendance and active participation during the meetings is encouraged and expected.

Other related books on the subject are the following:

  • "Introduction to Robotics" by John J. Craig
  • "Principles of Robot Motion: Theory, Algorithms, and Implementation" by Howie Choset, Kevin M. Lynch, Seth Hutchinson, George A. Kantor, Wolfram Burgard, Lydia E. Kavraki, Sebastian Thrun
  • "Mechanics of Robotic Manipulation" by Matt Mason
  • "A Mathematical Introduction to Robotic Manipulation" by Richard M. Murray, Zexiang Li and S. Shankar Sastry


Knowledge of basic computer science principles and programming skills at a level sufficient to write a reasonably non-trivial computer program. C++, Python and potentially Matlab are useful skills. Knowledge of Linear Algebra and Calculus courses are recommended.


Students will be assigned homework problems, which will correspond to 50% of the course's grade (there will also be extra credit questions that will correspond to 10% of the final grade). Students can form pairs to work on the homework problems. The homework sets consist of practice questions which are intended to assist students in mastering the course content. They may also potentially involve programming effort.

No additional credit will be given for students that complete a homework individually.

Students will receive 10% extra credit if they typeset their report using LaTeX or 5\% extra credit if they typewrite their answers (e.g., using Word). Submit only PDF documents. For instance, if a pair was to receive a score of 62/100 and they typeset their report, then their score will be 68/100, i.e,. they receive a bonus of +10\% of 62 points. Resources on how to use LaTeX are available below.

Submissions of homework assignments will be processed through Sakai. Late submissions will not be accepted.

Tentative homework schedule:

  • Homework 1: On Mathematical Modeling of Robots, Rigid Motion and Homogeneous Transformations
  • Homework 2: On Forward and Inverse Kinematics
  • Homework 3: On Jacobians
  • Homework 4: On Planning Methods
  • Homework 5: On Vision-based approaches for Manipulation

Competition Project

A key feature of the course is a robotics competition centered around robot manipulation, which will take place during the exam period. In particular, students will work in pairs on a semester-long project implemented on the Baxter robot available at CBIM. Access to the robot will need to be coordinated with the course's TA.

The competition will be centered around a picking challenge. The robot will be placed in front of a table at a known relative configuration. On top of the table, objects of ten different, previously known types will be placed at a random arrangement (see image for potential objects: http://amazonpickingchallenge.org/meta_items.jpg). The objects will be split into two pre-specified categories: left objects and right objects. There will also be two bins placed at known locations, one placed on the left side of Baxter and another placed on the right side.

The overall task is to pick objects of the left category and place them into the left bin, while right objects need to be placed in the right bin. A time limit will be specified (e.g., 10 minutes - one restart will be provided if a problem arises for a team). The number of items placed in the correct bins within the time limit will provide points to each team. Wrongly placed objects, objects dropped on the floor or damaged will result in a penalty.

The objects will be placed on the table in a way that will be possible to grasp them given the available grippers for Baxter and its arms' reachability. The solution should use the cameras available on the robot's hands in order to detect the objects. Thus, the team must make use of computer vision methods to detect the objects. Each team also needs to develop a planning solution for controlling the arms of Baxter.

The exact point system and more details about the competition will be defined as the semester progresses. The difficulty of the project will be adapted after coordinating with the participating teams.

The use of available open-source software is not only allowed but it is encouraged. Furthermore, the instructor and the TA will be frequently forwarding mathematical and motion planning software that will be helping with the implementation of the project. The provided code comes with no guarantees and can be involved. The teams are free to use the provided software primitives or ignore them and use alternative available online, which will also be discussed. Early on in the semester an tutorial on the Robot Operating System and interacting with the Baxter robot will be provided by the TA.

Grading System

The final grade will be computed according to the following rule (this is tentative):

  • Homeworks:
    • 5 Homework assignments: 50 points (there will also be a lot of extra credit problems available - up to 10 more points)
  • Project:
    • Demonstrating a vision component (able to correctly identify the location of some objects): 10 points
    • Demonstrating a planning component (able to pick some items given their known location): 10 points
    • Demonstrating the integration component (able to combine the two and pick some items): 10 points
    • Placement in the robot competition: 10 points
  • In-class Participation: 10 points

As a rough guide, the following rule may be used for the final grade (this is tentative):

  • A: >= 90
  • B+: 80-89
  • B: 70-79
  • C+: 60-69
  • C: 50-59
  • D: 40-49
  • F: less than 40

Academic Standards

Exams are to be treated as individual efforts. Homeworks are not to be treated as collective efforts beyond the participation of the team members! Discussions are not allowed on how to solve specific questions in homeworks. Do not discuss assignments with students that are not currently taking the class.

A severe penalty will be given to any assignment which indicates collusion or cheating. The usual penalty for cheating on an assignment or an exam is failure in the course. At a minimum your participation grade will be influenced negatively. Stealing another person's listing or having another person "ghost write" an assignment will be considered cheating.

Turning in work without properly citing the sources of the content included in your work is plagiarism. All kinds of sources are included in this definition, even those downloaded from the web, in which case an operable link must be cited. Plagiarism from the web or other sources is considered cheating and has the same effects. Even with a reference, submitting an answer to a homework question, verbatim from any source and without any contribution on your part, draws zero credit.

You should carefully study the website of Rutgers University on Academic Integrity and the corresponding policy, as well as the corresponding policy from the department of Computer Science. Your continued enrollment in this course implies that you have read these policies, and that you subscribe to the principles stated therein.

Relation to the Amazon Picking Challenge

The PRACSYS research group is planning to participate for a second time in a robot manipulation competition representing Rutgers University using available hardware and sensors in the Computational Biomedicine, Imaging and Modeling (CBIM) center. The hardware includes dual-arm manipulators, such as a Baxter robot by Rethink Robotics, robotic hands, such as a ReFlex hand by RightHand Robotics and RGBD sensors, such as Kinect. The competition is organized by Amazon.com Inc. and is called the Amazon Picking Challenge. It will take place at the IEEE International Conference on Robotics and Automation (ICRA) 2016 in Stockholm this coming May. The competition will challenge entrants to build their own solutions for solving a simplified version of the general task of picking items from shelves.

More information regarding last year's competition can be found under the Amazon Picking Challenge website.

Interested students taking this course can participate in the Rutgers team during the spring semester by taking an independent study.

Latex Resources

General info on what you can do with LaTeX:
Getting Started with Latex
The Not So Short Introduction to Latex
Comprehensive List of Latex Symbols
Latex for Logicians

Tex on Mac OS X

The first link describes many alternatives that are available for installing Tex on a Mac. The second link forwards to the MacTex package, one of the alternatives mentioned in the first website. MaxTex provides everythink that you need to use Latex on Mac except from a text editor. It is, however, compatible with a wide variety of popular editors (e.g., Alpha, BBEdit, Emacs, VIM, iTeXMac, TeXShop). Note that MaxTex is a large package.

Carbon Emacs has been succesfully tested with MacTex. After installing MacTex, it is possible to directly compile and view *.tex files from Carbon Emacs's UI.

Note for Mac users: You will probably have problems previewing your PDF output when using the postscript images provided by the instructor for developing the notes. Nevertheless, the PDF file can be printed properly. Prepare your document without the images and then add them. You will probably still be able to preview the intermediate .dvi output file with the "xdvi" program.

Linux (Ubuntu)
Latex on Ubuntu
Tex Live

You just have to download and install the proper packages described above (e.g., through apt-get), use your favorite editor (e.g., emacs) to prepare a *.tex file and then you compile (run at least two times: "latex filename.tex") to get the *.dvi output. You can go from dvi to postscript with the command "dvips" and you can convert postscript to pdf with the command "ps2pdf".

Latex for Windows help
MikTex (Latex for Windows)

If you follow the instructions on the first link you should be able to get it working on a Windows system.

Below you can find Windows executables (32 bit) for the following programs (follow the order when installing):