CS 672 - Manipulation Planning: Robots Changing the World

Regarding Robot Manipulation

Commercially available robotic toys and vacuum cleaners inhabit our living spaces, and robotic vehicles have raced across the desert. These successes appear to foreshadow an explosion of robotic applications in our daily lives, but without advances in robot manipulation, many promising robotic applications will not be possible. Whether in a domestic setting or the workplace, we would like robots to physically alter the world through contact. Robots have long been imagined as mechanical workers, helping us in our daily life. Research on manipulation in human environments may someday lead to robots that work alongside us, extending the time an elderly person can live at home, providing physical assistance to a worker on an assembly line, or helping with household chores.

Within factories around the world, robots perform heroic feats of manipulation on a daily basis. They lift massive objects, move with blurring speed, and repeat complex performances with unerring precision. Yet outside of carefully controlled settings, even the most sophisticated robot would be unable to get you a glass of water. The everyday manipulation tasks we take for granted would stump the greatest robot bodies and brains in existence today. Why are robots so incompetent in unstructured, human environments?

Motivating material for robot manipulation comes from the paper "Challenges for Robot Manipulation in Human Environments" by Kemp, Edsinger, Torres-Jara, IEEE Robotics and Automation Magazine, March 2007.

Course Description

This is a presentation and project-based seminar focusing on the challenges of manipulation and grasping planning. We will review papers related to how we can plan the motion of robotic arms and hands so as to change the robot's workspace. The class will first focus on functional aspects of manipulation, such as the topology of state space representation for manipulation, motion planning in these spaces and synthesizing robust and stable grasps for dexterous hands. Moving forward, we will discuss additional requirements that arise from performing manipulation tasks in the real world and potentially in collaboration with people: machine learning and data driven approaches, coordination via understanding and expressing intent, interactive manipulation.

Students will work on a semester-long project, which should eventually lead to an implementation with real sensing data and a real robot (for instance, the Baxter dual-arm manipulator available at CBIM). Many of the lectures will be mixed student and instructor presentations. The instructor will introduce some background information so that everyone can follow the student presentation and then discussion will ensue.

Amazon Picking Challenge

The PRACSYS research group is planning to participate in a robot manipulation competition representing Rutgers University using available hardware and sensors in the Computational Biomedicine, Imaging and Modeling (CBIM) center. The hardware includes dual-arm manipulators, such as a Baxter robot by Rethink Robotics, robotic hands, such as a ReFlex hand by RightHand Robotics and RGBD sensors, such as Kinect. The competition is organized by Amazon.com Inc. and is called the Amazon Picking Challenge. It will take place at the IEEE International Conference on Robotics and Automation (ICRA) 2015 in Seattle this coming May.

The competition will challenge entrants to build their own solutions for solving a simplified version of the general task of picking items from shelves. The robots will be presented with a stationary lightly populated inventory shelf and be asked to pick a subset of the products and put them on a table. The challenge combines object recognition, pose recognition, grasp planning, compliant manipulation, motion planning, task planning, task execution, and error detection and recovery. The robots will be scored by how many items are picked in a fixed amount of time.

The course projects will be related to the Amazon Picking Challenge.

More information regarding the competition can be found under the Amazon Picking Challenge website.

A description of the capabilities of the available hardware and software for our participation in the competition can be found in a video accessible by the PRACSYS YouTube channel.

Tentative Calendar

Check Rutgers' Key Dates.

  • April 3: Chris's lecture
  • April 6 (Monday): APC meeting coordinated
  • April 10: Rohan's lecture
  • April 14: Talk by visitor
  • April 17: Colin's lecture
  • April 21: Changkyu's lecture
  • April 24: Shaojun's lecture
  • April 28: Jason's lecture
  • May 1: Kostas' lecture

Text and Reading Material

No specific textbook is required. Students are expected to study the papers that are recommended for each lecture. Regular attendance and active participation during the meetings is expected. The specific papers (or book chapters) related to the corresponding topics are available via Sakai.


Students will work on a semester-long project corresponding to 60% of the course's grade. The projects will be related to aspects of the Amazon Picking Challenge:

  1. 3D mapping, object recognition, pose recognition
  2. grasp planning, control of robotic hands, visual servoing, compliant manipulation
  3. motion planning, task planning, task execution, and error detection and recovery

While each student will work one one of the above individual objectives, teams of students will be formed. Each team will aim to address the entire Amazon Picking Challenge by integrating the efforts of its members. The objective will be for different teams to try alternative methodologies for each sub-challenge of the overall competition. Teams will typically have 3 students in them.

The remaining of the course grade will correspond to class participation and presentations of papers to the class. Many lectures will involve one or two 25 minute presentation(s) of a paper by a student. The rest of each of those lectures will correspond to discussion among the class regarding the research results and direction, as well as progress reports regarding the status of the projects. The number of paper presentations per student will depend on the number of students in the class but it is expected to be from 1 to 3. The schedule will be appropriately prepared so that all students give the same number of paper presentations (in some cases this may require students collaborating on a presentation).

Semester-Long Project

For the project, the students will work during the entire semester on research challenges related to the Amazon Picking Challenge. Each team will have to submit intermediate reports and make an intermediate project presentation. A final project presentation and demo will take place during the final week of classes or during Rutgers Day (i.e., April 25). The objective of the reports is to acquaint students with the paper writing process (useful both for academic papers, as well as for technical project reports in the industry). The final version will hopefully be close in quality to a good conference submission. The presentation should also be equivalent to a conference presentation and accompanied by a working (!) demo involving every aspect of the Amazon Picking Challenge. Each student will be graded separately for their individual contribution to the project.

The teams and individual project topics will be decided after coordinating with the instructor early on during the semester. The teams will be provided access to the existing software available related to the Amazon Picking Challenge.

The students need to submit typesetted reports using LaTeX. Resources on using LaTeX are available below. You are expected to use IEEE RAS' double column format for conferences.

The following project components and deadlines for submissions will be used to guide your work during the entire

  1. January 29: Project Proposal
    • Provide an abstract for the project
    • Then in more detail describe the general topic, provide motivation for the work and application area, emphasize the significance and the difficulty of the challenge, highlight the most relevant state-of-the-art papers, the main idea behind the approach you are considering, justify why your project will be novel
    • 2-3 pages, corresponding to the Abstract and Introduction sections of the final report
  2. February 9: Literature Search
    • Improve upon the Introduction section given better knowledge of the related literature
    • Provide a comprehensive coverage of the related literature on the topic, make sure to cover efforts from different research communities and perspectives, do not just describe related work but try to point out the similarities and differences between existing efforts, identify key contributions in the field as well as limitations of existing work, conclude with the set of open-problems given the state-of-the-art, specify which of the existing contributions your project will be building upon and which open problems it will be addressing
    • 2-3 pages, corresponding to the Related Work section of the final report, references are excluded from the page count
  3. February 23: Formal Problem Setup, Experimental Setup and Infrastructure
    • Improve upon your assessment of the related literature given initial experimentation
    • A formal problem setup section with definition, notation and a clear description of the challenge in mathematical terms. Clearly state the assumptions you will be considering in your work.
    • Work on providing the software/hardware platform that you will employ for your work, describe the experimental setup for your evaluation, demonstrate to the instructor the capability to run experiments or simulations on the proposed research topic using naive solutions (even if your project is more theoretical in nature, you need to provide some form of simulation that evaluates or confirms your analysis)
    • 2-3 pages, corresponding to the Problem Setup and the Experimental Setup sections of the final report
  4. March 9: Algorithmic Methodology and Mathematical Foundations
    • Improve upon your experimental platform given implementation of methods and evaluation
    • Describe in detail the novel algorithmic or mathematical process that you are proposing in order to address the project's challenge, start from the foundations of the solution in existing work and progress in emphasizing the new aspects of the approach, provide an initial implementation of the corresponding solution on the experimental platform that you have available and perform initial evaluation - no need to describe data at this point but you should be getting initial indications on how the method is performing
    • 3-4 pages, corresponding to the Proposed Methodology sections of the final report
  5. March 26: Analysis of Properties and Comparison Solutions
    • Improve upon the algorithmic solution given the observed experimental performance and analysis of the solution's properties
    • Evaluate the properties of the proposed methodology (e.g., in terms of metrics such as safety, completeness, optimality, computational performance and complexity, scalability, information requirements, etc.), theoretical and mathematically study of the underlying properties is highly encouraged if it makes sense for the project, describe alternative methodologies that need to be evaluated in order to show the relative performance of the proposed approach, provide initial implementations of the alternative methodologies and initial evaluations
    • 2-3 pages, corresponding to the Properties and/or Comparison Solutions sections of the final report - note that more theoretical projects will be focusing more on ``properties'', while more experimental ones will be focusing more on evaluating performance against ``comparison solutions''
  6. April 9: Experimental Evaluation
    • Improve upon the analysis of the proposed method, as well as better identify appropriate comparison points given additional study, experimentation and feedback from the instructor
    • Perform a comprehensive experimental evaluation of the proposed methodology in appropriate benchmarks that you have identified and described in detail, the evaluation will include running experiments with competitive methods and providing the corresponding graphs that show the differences between the performance of different algorithms, study experimentally the performance of the studied methods in terms of the appropriate metrics (e.g., safety, completeness, optimality, computational performance and complexity, scalability, information requirements, etc.)
    • 3-4 pages, corresponding to the Results section of the final report, even theoretical projects need to evaluate methods using some form of simulation
  7. April 23: Complete Report - Discussion of Performance, Improvement of Methodology and Revisions
    • Improve upon the experimental evaluation given a study of the resulting graphs
    • Provide a detailed discussion of the observations during the experimental evaluation, based on your conclusions, revise and improve the proposed methodology, run additional experiments and show potential improvements against alternatives or versus prior performance, study in terms of theoretical properties the effects of your revisions in the underlying algorithm, update the introductory part of the paper given observations and the potentially shifted focus of the project, summarize the contribution of the project and identify the important future steps
    • 2-3 pages, corresponding to the Discussion section of the final report, as well as revisions and extensions in the entire paper given observations, further study and changes in the underlying methods
  8. May 3: Final Report
    • Improve upon the entire paper and tighten up description. The final report will no longer need to describe
      in detail the process with which you ended up in the best performing solution (i.e., the initial approach you were considering and how you ended up improving upon it). It needs to directly inform the reader about the final contribution and observations that will be the result of your entire effort. It still important, however, to provide key comparisons with alternatives in your experimental setup and justify the algorithmic choices made in the final solution. Aim for 10 pages total excluding references.

Submissions will be processed through Sakai. Late project report will not be accepted. Final project presentations cannot be easily rescheduled. Unless there are unprecedented circumstances, no incomplete grade will be awarded and students will be graded based on the submitted reports and presentations up to the exam week.

Extra credit can be awarded for impressive effort and performance in the project. These projects should be able to contribute software to Rutgers' participation to the Amazon Picking Challenge.

Grading System

The final grade will be computed according to the following rule (this is tentative):

  • Project Proposal (relevance, significance, interest, novelty and level of difficulty of proposed ideas): 5 points
  • Literature Search (comprehensiveness, capability to classify previous work, ability to identify relations and open problems): 5 points
  • Problem and Experimental Setup (appropriateness of setup, benchmarks and capability to evaluate the correct metrics for the challenge and the methodologies): 5 points
  • Algorithmic Methodology (formal and rigorous description, appropriateness of solution, clarity of presentation, novelty of ideas): 5 points
  • Analysis of Properties (proper identification of comparison points, mathematical capabilities in analyzing proposed methods): 5 points
  • Experimental Evaluation (comprehensiveness of evaluation, study of appropriate metrics and informative presentation of data): 5 points
  • Complete Report (ability to communicate message and progress the state-of-the-art, novelty, significance, difficulty, success in objectives): 10 points
  • Final Report (efficient presentation of material): 5 points
  • Intermediate Presentation/Demo (clear explanation of objectives, methods and results, ability to maintain interest, use of visual assistance, success of demonstration - feedback from all the students will be used to grade the presentation): 10 points
  • Final Presentation/Demo (success of demonstration, clear explanation of objectives, methods and results, ability to maintain interest, use of visual assistance - feedback from all the students will be used to grade the presentation): 15 points
  • Paper Presentations during lectures: 20 points total
  • In-class participation and discussion: 10 points

As a rough guide, the following rule may be used for the final grade (this is tentative):

  • A: >= 90
  • B+: 80-89
  • B: 70-79
  • C+: 60-69
  • C: 50-59
  • D: 40-49
  • F: less than 40

Academic Standards

Discussions on the class material, on project challenges and programming practices are highly encouraged. Furthermore, students should try to exchange feedback on drafts of their project reports. Nevertheless, each student needs to eventually independently develop one's proposed methodology, code the corresponding solution and execute the corresponding experiments.

You should carefully study the website of Rutgers University on Academic Integrity and the corresponding policy, as well as the corresponding policy from the department of Computer Science. Your continued enrollment in this course implies that you have read these policies, and that you subscribe to the principles stated therein.

Latex Resources

General info on what you can do with LaTeX:
Getting Started with Latex
The Not So Short Introduction to Latex
Comprehensive List of Latex Symbols
Latex for Logicians

Tex on Mac OS X

The first link describes many alternatives that are available for installing Tex on a Mac. The second link forwards to the MacTex package, one of the alternatives mentioned in the first website. MaxTex provides everythink that you need to use Latex on Mac except from a text editor. It is, however, compatible with a wide variety of popular editors (e.g., Alpha, BBEdit, Emacs, VIM, iTeXMac, TeXShop). Note that MaxTex is a large package.

Carbon Emacs has been succesfully tested with MacTex. After installing MacTex, it is possible to directly compile and view *.tex files from Carbon Emacs's UI.

Note for Mac users: You will probably have problems previewing your PDF output when using the postscript images provided by the instructor for developing the notes. Nevertheless, the PDF file can be printed properly. Prepare your document without the images and then add them. You will probably still be able to preview the intermediate .dvi output file with the "xdvi" program.

Linux (Ubuntu)
Latex on Ubuntu
Tex Live

You just have to download and install the proper packages described above (e.g., through apt-get), use your favorite editor (e.g., emacs) to prepare a *.tex file and then you compile (run at least two times: "latex filename.tex") to get the *.dvi output. You can go from dvi to postscript with the command "dvips" and you can convert postscript to pdf with the command "ps2pdf".

Latex for Windows help
MikTex (Latex for Windows)

If you follow the instructions on the first link you should be able to get it working on a Windows system.

Below you can find Windows executables (32 bit) for the following programs (follow the order when installing):