> For the complete documentation index, see [llms.txt](https://x.ancorasir.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://x.ancorasir.com/me336/2019-spring/me336-collaborative-robot-learning.md). # \[ME336] Collaborative Robot Learning 2019 Spring ## Course Description Selective Core Course for Final Year Undergrad This course is intended for students advancing in the study of robotic engineering. The focus is on the problems of how a robot can learn to perceive the physical world well enough to act in it and make reliable plans. Subjects covered by this course include robotic collaboration, kinematics, Robot Operating Systems (ROS), robotic vision, calibration, RGB-D sensing, object recognition, artificial intelligence (AI) and deep learning (DL). Specific projects will be carried out throughout this course regarding the simulation of robot picking using fundamental kinematics and robot vision, an AI robot to play tic-tac-toe game, and a DL robot to play an arcade claw game. ## Teaching Goals * To teach students how to conduct the basic kinematic formulation of a robotic system in simulation. * To teach students how to use robotic vision, including algorithm, hardware, and software, in simulation. * To teach students how to program artificial intelligence into a robot hardware performing interactive tasks. * To teach students how to use deep learning methods to program a robot hardware to perform advanced tasks. * To reinforce students’ team skills through various team project, including problem formulation, problem solutions and written reporting of results. * To reinforce students’ visualization and hands­-on skills through project virtual prototyping and/or physical construction exercises. ## Learning Outcomes As an elective course for robotic engineering major, this course lays the foundation for students to use widely adopted Robot Operating System (ROS) to perform advanced robot control including basic mathematical formulation, hardware usage, and intelligence integration. The following learning outcomes are expected for students taking this course: * Given functional and environmental requirement, utilize concepts generation methods within a team setting to achieve a consensus for a robot design concept. * Design and develop functional robot programs from the perspectives of function, hardware, algorithm, and physical environment. * Apply basics of disciplines including mechanical engineering, electrical engineering, applied mathematics, computer science to understand the use of robots in action. * Communicate engineering decisions, justification for those decisions, designs, programming, and test results in multi-media presentation and report writing. ## Content Summary ### Robotic Engineering Project * **Project 1: Simulate picking using kinematics \[**[**Github**](https://github.com/ancorasir/BionicDL-CobotLearning-Project1)**]** * Simulate a robot arm picking objects from a shelf and placing them in a bin. * Similar to Amazon Picking Challenge, but much simplified in ROS; * Without using any image data, focus mainly on inverse kinematics for planning; * Focus on the understanding of analytical method for common robotic grasping; * To be completed using the students’ own laptop. * **Project 2: Simulate picking using robotic vision \[**[**Github**](https://github.com/ancorasir/BionicDL-CobotLearning-Project2)**]** * Simulate a robot arm processing RGB-D images to grasp objects from a bin. * Similar to industrial bin-picking scenario, but much simplified in ROS; * Process simulated images in ROS to pick up simple object autonomously; * Focus on the use of robotic vision techniques for common robotic grasping; * To be completed using the students’ own laptop. * **Project 3: Program an AI robot to play the Tic-Tac-Toe game \[**[**Github**](https://github.com/ancorasir/BionicDL-CobotLearning-Project3)**]** * Use AI to control a real robot to play tic-tac-toe with a human player. * Similar to the Alpha-Go scenario, but much simplified using a real desktop robot; * Design and implement an AI algorithm on a robot for tic-tac-toe; * Focus on the design of AI algorithms to control a real robot arm autonomously; * To be completed using a real desktop robot hardware (in progress). * **Project 4: Program a DL robot to play the arcade claw game \[**[**Github**](https://github.com/ancorasir/BionicDL-CobotLearning-Project4)**]** * Use Deep Learning to control a real robot to play the arcade claw machine game. * Similar to the DeepClaw scenario, but much simplified using a real desktop robot; * Design and implement a deep neural network to play the game for data acquisition; * Focus on the use of DL techniques on a real robot for data-driven robotics; * To be completed using a real desktop robot hardware (in progress). * **Project 5: Autonomous Robot Manipulation Competition \[Github]** * Design a competitive and autonomous picking robot * Design a robot system to autonomously play a board game competitively; * In the final class, each team will present your robot system design in details by * Writing a full paper describing your robot system in technical details; * Submit a video demo of your robot system design; * Submit a posted to present your robot system design; * Open-source your codes by uploading on GitHub; * Compete live and win! (Good luck) * To be completed using a real desktop robot hardware. * **Due to the limitation of the robotic hardware, a total of 9 students will be limited to this course.** ### How to Video Record Your Presentation Using PowerPoint * [There is a simple solution in Microsoft PowerPoint](https://support.office.com/en-us/article/Record-your-slide-show-in-PowerPoint-2016-or-2011-for-Mac-93e8b4a1-5efb-4f78-903b-62897b8bb14b), as a recording function is built-in the software. * The content of the video should revolve around your presentation, but you are welcome to be creative about how to present it. * This is a Team assignment, and the question about who to present makes no sense at all. * You need to prepare and submit a PDF version of the slides at this \[link]. ![2019 Spring course exhibition poster](https://content.gitbook.com/content/2bhlCeGuMphEBHq6PKcJ/blobs/ryOHe8GxiT4ZzsHgMBmW/assets/2019-course-poster.jpg) ![](https://content.gitbook.com/content/2bhlCeGuMphEBHq6PKcJ/blobs/7nk33cRVi9iX8v3Ks7v9/assets/2019-course-overview.png) {% file src="/files/V1OoEc1RB3SAPy4xt4Bn" %} ## Co-Requisite Courses * MEE5101 Introduction to Robotics and Automation * MEE5104 Robotic Modeling and Control ## Course Instructor & Teaching Team * Lead Instructor: Dr. Song Chaoyang, Dr. Wan Fang (Invited) * Office: Room 606, 7 Innovation Park * Teaching Assistant: Mr. Liu Xiaobo ## Grading Policy * 60%: project marking * 15% per project for the first four projects, including 10% code submission and 5% video presentation. * 10%: individual marking * 30%: final project marking, including * 10% final paper | 10% final video demo | 5% final poster | 5% code submission ## Academic Integrity This course follows the SUSTech Code of Academic Integrity. Each student in this course is expected to abide by the SUSTech Code of Academic Integrity. Any work submitted by a student in this course for academic credit will be the student’s own work. Violations of the rules (e.g., cheating, copying, non-approved collaborations) will not be tolerated. ## Recommended Textbook(s) * **Required**: * Introduction to Robotics – Mechanics and Control (4th Edition) by John Craig. \[[PDF](http://www.mech.sharif.ir/c/document_library/get_file?uuid=5a4bb247-1430-4e46-942c-d692dead831f\&groupId=14040)] * **Optional**: * Robotics, Vision & Control – Fundamental Algorithms in MATLAB® by Peter Corke. * Artificial Intelligence – A Modern Approach (3rd Edition) by Stuart Russel, and Peter Norvig. * Mastering ROS for Robotics Programming by Lentin Joseph. \[[Website](http://www.mastering-ros.com/)] * A Mathematical Introduction to Robotic Manipulation by Richard M. Murray, Zexiang Li, and S. Shankar Sastry. \[[PDF](https://www.cds.caltech.edu/~murray/books/MLS/pdf/mls94-complete.pdf)] ## Teaching Schedule * Lecture: Wednesday 10:20-12:10, Location @ #208, 2 Lychee Park * Lab: Every Friday 10:20-12:10, Location @ #607, 7 Innovation Park ## First Class * W01-C01 | [**Course Introduction**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2FUDSZBMstGNHy5HhZEY0D%2F2019Autumn-ME336-WK01-Lecture-Course-Introduction.pdf?alt=media\&token=f2e83014-9931-4f13-a3f7-bdc33848d61c) | Wed Feb 20, 1020–1210, #208, 2 Lychee Park * W01-C02 | [**Linux and ROS**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2FIheZqNUHV5nFKUzPjQlW%2F2019Autumn-ME336-WK01-Project-Programming-Basics.pdf?alt=media\&token=757439f4-6ac4-4451-b137-960b6afe852b) | Fri Feb 22, 1020–1210, #607, 7 Innovation Park ## Module 01 Robot Foundations & Simulate Picking Kinematics * W02-C03 | [**Mathematical Foundations**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2F1BwxBXthF1oBL8Weu5qo%2F2019Autumn-ME336-WK02-Lecture-Foundations-Mathematical-Foundations.pdf?alt=media\&token=9259ac83-528e-4a35-97d1-08cc47deb01d) | Wed Feb 27, 1020–1210, #208, 2 Lychee Park * W02-C04 | [**ROS Basics**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2FJ4FiU0uOR3XWh4Cf6jb3%2F2019Autumn-ME336-WK02-Project-I-SimKinePick-ROS-Basics.pdf?alt=media\&token=e569f5a6-17de-46dc-8576-747faa9323eb) | Fri Mar 01, 1020–1210, #607, 7 Innovation Park * W03-C05 | [**Kinematics & Jacobian**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2FjPeG3WElihFCeuY1nruF%2F2019Autumn-ME336-WK03-Lecture-Foundations-Kinematics-Jacobian.pdf?alt=media\&token=71f60c31-dafc-49be-a804-eeb4788dbb71) | Wed Mar 06, 1020–1210, #208, 2 Lychee Park * W03-C06 | [**ROS Simulation**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2FfVPih8r9TXzYaj2z4DpX%2F2019Autumn-ME336-WK03-Project-SimKinePick-ROS-Simulation.pdf?alt=media\&token=2205750c-5316-4dde-b7cc-a9f5b2a5a4e5) | Fri Mar 08, 1020–1210, #607, 7 Innovation Park * W04-C07 | [**Dynamics & Control**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2FF9O2nwdSSnLqbGh8qZGa%2F2019Autumn-ME336-WK04-Lecture-Foundations-Dynamics-Control.pdf?alt=media\&token=7bc6073b-585b-4d3a-8f94-83ebfcf82c3b) | Wed Mar 13, 1020–1210, #208, 2 Lychee Park * W04-C08 | [**Seminar \[Hu Chunxu\]**](https://ancorasir.com/?p=1791) **/** [**ROS Picking**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2Fhkuq56j8YSXEGYhPS9v4%2F2019Autumn-ME336-WK04-Project-SimKinePick-ROS-Picking.pdf?alt=media\&token=0691853f-62da-4d05-bc0b-170be5a33d1f) | Fri Mar 15, 1020–1210, #607, 7 Innovation Park ## Module 02 Robotic Vision & Simulate Picking with Vision * W05-C09 | [**Seminar \[Dr. Qiu Qiang\]**](https://ancorasir.com/?p=1802) | Wed Mar 20, 1020–1210, #208, 2 Lychee Park * W05-C10 | [**Camera & Images**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2FHEvOTl7Zjf12v5Jhm6BC%2F2019Autumn-ME336-WK05-Lecture-Vision-ImagesCamera-ClickSequence.pdf?alt=media\&token=d76be550-4efa-4b60-b071-96d385d8933e) **/** [**RealSense with ROS**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2FyuqlQHPWB2TlL7hgAtfJ%2F2019Autumn-ME336-WK06-Project-VisPick-Hand-Eye-Calibration.pdf?alt=media\&token=4a9add74-035b-4689-a881-1dab5467444d) | Fri Mar 22, 1020–1210, #607, 7 Innovation Park * W06-C11 | [**Calibration & Servoing**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2Ffx8rv6rugtNeA4sCZf4J%2F2019Autumn-ME336-WK06-Lecture-Vision-CalibrationServoing.pdf?alt=media\&token=1acdebcc-b59f-45d1-9e76-ba8e2352431d) | Wed Mar 27, 1020–1210, #208, 2 Lychee Park * W06-C12 | [**Seminar \[Dr. Gao Lei\]**](https://ancorasir.com/?p=1814#0) | Fri Mar 29, 1020–1210, #607, 7 Innovation Park * W07-C13 | [**Features & Filters**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2FiGLM9O1Wpm6wMoRKHhYF%2F2019Autumn-ME336-WK07-Lecture-Vision-FiltersFeatures.pdf?alt=media\&token=f2e846c7-946e-4ce5-882b-520ac1d9666c) **/** [**OpenCV & PCL**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2FcWHW0vUsXTPJHejl4PBj%2F2019Autumn-ME336-WK07-Project-VisPick-OpenCV-PCL.pdf?alt=media\&token=0c30ffd7-e987-4ab8-8a1a-7df00d28faeb) | Wed Apr 03, 1020–1210, #208, 2 Lychee Park * W07 | **Canceled due to holiday** | Fri Apr 05, 1020–1210, #607, 7 Innovation Park * Scheduled activity: Project II Simulate Picking with Vision. ## Module 03 Artificial Intelligence & Program a Tic-Tac-Toe Robot * W08-C14 | [**Rational Agents**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2FGc8pH7hM3iTfENuJFlUQ%2F2019Autumn-ME336-WK08-Lecture-AI-Rational-Agents.pdf?alt=media\&token=49d5ad51-32e0-4aa3-83e4-ddefc4dc9a39) | Wed Apr 10, 1020–1210, #208, 2 Lychee Park * W08-C15 | [**Seminar \[Liu Sicong\]**](https://ancorasir.com/?p=1832) | Fri Apr 12, 1020–1210, #607, 7 Innovation Park * Lab: [Project III Program a Tic-Tac-Toe Robot](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2FOolVfrHBMISX69lCxdFk%2F2019Autumn-ME336-WK08-Project-TicTacToe-Robot-Player.pdf?alt=media\&token=b3e89b7a-53d0-469b-be04-35b8285ac789). * W09-C16 | [**Search Problems**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2FXTk1sI2aGwzg9g98s7EA%2F2019Autumn-ME336-WK09-Lecture-AI-Search.pdf?alt=media\&token=5cecc855-9bd0-4627-8220-cd309f1aeb57) | Wed Apr 17, 1020–1210, #208, 2 Lychee Park * W09-C17 | [**Seminar \[Zhang Min\]**](https://ancorasir.com/?p=1841) | Fri Apr 19, 1020–1210, #607, 7 Innovation Park * W10-C18 | [**Adversarial Search**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2FA9oY2gBFo9MX8wNKRDrQ%2F2019Autumn-ME336-WK10-Lecture-AI-Robotic-AI.pdf?alt=media\&token=8d5f95ee-b67e-4271-a23f-f5f6882b1217) | Wed Apr 24, 1020–1210, #208, 2 Lychee Park * W10-C19 | [**Seminar \[Dr. Liang Conghui\]**](https://ancorasir.com/?p=1852) | Fri Apr 26, 1020–1210, #607, 7 Innovation Park * Lab: [Project III Program a Tic-Tac-Toe Robot](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2FBxcilPYMqSU7vLw5rXZ4%2F2019Autumn-ME336-WK10-Project-TicTacToe-Robot-Player.pdf?alt=media\&token=1c240205-5bbe-4da8-9d16-6700ac577f13). ## Module 04 Deep Learning & Program an Arcade Claw Robot * W11 | **Canceled due to holiday** | Wed May 01, 1020–1210, #208, 2 Lychee Park * W11-C20 | [**Neural Networks**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2FZ41s6W9ZrRN261AC0ZCr%2F2019Autumn-ME336-WK11-Lecture-DL-Neural-Networks.pdf?alt=media\&token=15018fd5-ac72-44e3-a044-39f407dd4c87) **/** [**Tensorflow Basics**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2F81DDdjllRYaLDrCc53km%2F2019Autumn-ME336-WK11-Project-ArcadeClaw-TensorFlow-Basics.pdf?alt=media\&token=d9e4316e-fe32-402a-a873-6fa0c9c61498) | Sun May 05, 1020–1210, #607, 7 Innovation Park * W12-C21 | [**Network Tuning**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2Ff91akLNAqJKwIdKUIUwO%2F2019Autumn-ME336-WK12-Lecture-DL-NetworkTuning-reduced.pdf?alt=media\&token=e44e2f48-3e28-4f7a-bfa7-7934ca1b5c1f) | Wed May 08, 1020–1210, #208, 2 Lychee Park * W12-C22 | [**Learning Picking**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2FrI4xLPMQN8Uayjp3rKFp%2F2019Autumn-ME336-WK12-Project-ArcadeClaw-Learning-Picking.pdf?alt=media\&token=03e274a9-d38c-4321-b242-ee3bde510d32) | Fri May 10, 1020–1210, #607, 7 Innovation Park * W13-C23 | [**Seminar \[Liu Guoqing\]**](https://ancorasir.com/?p=1849) | Wed May 15, 1020–1210, #208, 2 Lychee Park * W13-C24 | [**Robot Learning**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2FwhUmnscOnzR4YjGwPgz7%2F2019Autumn-ME336-WK13-Lecture-DL-Robot-Learning.pdf?alt=media\&token=ec30a5b8-bd32-4495-80c7-d0801ec220df) | Fri May 17, 1020–1210, #607, 7 Innovation Park ## Module 05 Special Topics & Collaborative Robot Learning * W14-C25 | [**Shareability & Reproducibility**](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F2bhlCeGuMphEBHq6PKcJ%2Fuploads%2FCcabLnCoiENRGcVjYh82%2F2019Autumn-ME336-WK14-Lecture-CoBot-Shareability-Reproducibility.pdf?alt=media\&token=37c1b495-bbcc-461f-bd89-e03610819dca) | Wed May 22, 1020–1210, #208, 2 Lychee Park * W14-C26 | [**\[Seminar\] Li Miao**](https://ancorasir.com/?p=2097) | Fri May 24, 1020–1210, #607, 7 Innovation Park * W15-C27 | **Industrial vs. Collaborative** | Wed May 29, 1020–1210, #208, 2 Lychee Park ## Final Class * W15-C28 | **Seminar \[Albert Causo] / Final Presentation** | Fri May 31, 1020–1210, #607, 7 Innovation Park ## Important Deadlines Historical submission labels below are associated with these schedule rows. The source does not separately confirm project cutoff times; these are not new submission instructions. * Fri Mar 15: Project I Submission (historical schedule entry). * Fri Apr 05: Project II Submission (historical schedule entry). * Fri Apr 26: Project III Submission (historical schedule entry). * Fri May 17: Project IV Submission (historical schedule entry). * Fri May 31: Project V Submission (historical schedule entry). ### Late Homework Policy * Each student is granted four unpenalized late days for the semester. Homework can be handed in no more than four days late and will receive a 25% penalty for each day late (excluding unpenalized late days used). Homework are due at 3 PM on the due date, and each late day extends the deadline by exactly 24 hours. All homework must be done to pass the course. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://x.ancorasir.com/me336/2019-spring/me336-collaborative-robot-learning.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.