> For the complete documentation index, see [llms.txt](https://x.ancorasir.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://x.ancorasir.com/me336/me336-collaborative-robot-learning.md).

# \[ME336] Collaborative Robot Learning

2024 Spring

<figure><img src="https://content.gitbook.com/content/MJwwmQ8eCYejhFgyqHrR/blobs/hxL2oymc8Isv3qwF3jXr/assets/2024-course-overview.png" alt="DeepClaw course overview"><figcaption></figcaption></figure>

{% columns %}
{% column %}

<figure><img src="https://content.gitbook.com/content/MJwwmQ8eCYejhFgyqHrR/blobs/f7dtTsS7RoGa5ewNAB2v/assets/2024-team-01.jpeg" alt="Team 1"><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}

<figure><img src="https://content.gitbook.com/content/MJwwmQ8eCYejhFgyqHrR/blobs/F1JjNUXPzz3iO02ty7kt/assets/2024-team-02.jpeg" alt="Team 2"><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}

<figure><img src="https://content.gitbook.com/content/MJwwmQ8eCYejhFgyqHrR/blobs/pKOi7lQhjPRGVCxfEPXp/assets/2024-team-03.jpeg" alt="Team 3"><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

<figure><img src="https://content.gitbook.com/content/MJwwmQ8eCYejhFgyqHrR/blobs/NZbxtq85jbNGw7AhNB8D/assets/2024-team-04.jpeg" alt="Team 4"><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}

<figure><img src="https://content.gitbook.com/content/MJwwmQ8eCYejhFgyqHrR/blobs/e4ZoT0ejNOyfwlxrK8Ym/assets/2024-team-05.jpeg" alt="Team 5"><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}

<figure><img src="https://content.gitbook.com/content/MJwwmQ8eCYejhFgyqHrR/blobs/bs3gwBdtjgzetIiEOIos/assets/2024-team-06.jpeg" alt="Team 6"><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

<figure><img src="https://content.gitbook.com/content/MJwwmQ8eCYejhFgyqHrR/blobs/AzPI5GHHsAQqKXvqrXgz/assets/2024-team-07.jpeg" alt="Team 7"><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}

<figure><img src="https://content.gitbook.com/content/MJwwmQ8eCYejhFgyqHrR/blobs/1VlOA7NgZ0v2GvLvRzxF/assets/2024-team-08.jpeg" alt="Team 8"><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}

<figure><img src="https://content.gitbook.com/content/MJwwmQ8eCYejhFgyqHrR/blobs/X3HRr4SfM98YuhM0inqh/assets/2024-team-09.jpeg" alt="Team 9"><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

<figure><img src="https://content.gitbook.com/content/MJwwmQ8eCYejhFgyqHrR/blobs/sRLTygPGzyYtIMhGElcC/assets/2024-guest-joonho-lee.jpg" alt="Guest lecture poster — Joonho Lee"><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}

<figure><img src="https://content.gitbook.com/content/MJwwmQ8eCYejhFgyqHrR/blobs/9iZ90FHgkM3WKrDZzorz/assets/2024-guest-wang-haokun.png" alt="Guest lecture poster — Wang Haokun"><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}

<figure><img src="https://content.gitbook.com/content/MJwwmQ8eCYejhFgyqHrR/blobs/gLAkNGhyq0M5lacZaSuD/assets/2024-guest-qiu-qiang.jpg" alt="Guest lecture poster — Qiu Qiang"><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

## Course Description

This course is intended for students advancing in the study of robotic engineering. The focus is on the problems of how a robot can learn to perceive the physical world well enough to act in it and make reliable plans. We have updated the course syllabus throughout the years. This semester, we will use the DeepClaw toolkit for basic training in machine vision and tactile learning, and we will adopt the RoboSuite toolkit and the teaching format developed by [Dr. Zhu Yuke’s team at UT Austin](https://www.cs.utexas.edu/~yukez/cs391r_fall2020/logistics.html) for simulation and learning.

## Teaching Goals

* Learn the basics of building robotic systems with vision-based machine learning and AI in the real world;
* Understand the technical challenges arising from building learning-based robotic manipulation systems;
* Get familiar with a variety of model-driven and data-driven principles and algorithms on robot learning;
* Evaluate, communicate, and apply AI-based techniques to problem-solving in robotics.

## Course Prerequisite

**Assumed Knowledge**

* Basic knowledge of data structures and algorithms, as well as practical computer programming skills.
* Proficiency in Python is required, and high-level familiarity with C/C++ is a plus.
* Familiarity with calculus, statistics, and linear algebra. Strong mathematical skills are required.
* Coursework and/or equivalent experience in Robotics, Machine Learning, and AI are preferred.
* Be passionate, patient, and fearless when working with Robotics + AI systems.

## Course Instructor & Teaching Support

* **Lead Instructor**: Dr. Song Chaoyang (<songcy@sustech.edu.cn>)
  * Office Location: Room 517, North Block, Engineering Building
  * Office Hours: 10:00\~12:00 on Mondays
* **Teaching Assistant 1**: Wu Tianyu (<12332365@mail.sustech.edu.cn>)
* **Teaching Assistant 2**: Huang Bangchao ([12332379@mail.sustech.edu.cn](mailto:11910206@mail.sustech.edu.cn))
* **Technical Support**: Sun Haoran (<11610409@mail.sustech.edu.cn>)
* **Student Assistant**: Zhang Zishang (<12012305@mail.sustech.edu.cn>)
* **Administrative Assistant**: Fu Tian (<fut@mail.sustech.edu.cn>)
  * For the lab session, we will use a set of tools developed by previous students of this class called *DeepClaw*. The teaching assistants are all previous course members, and some are the core developers of the latest version.
  * The latest version won the 1st prize in the 2022 Capstone Project, and we have already pilot-tested it for three semesters in multiple courses offered by the SUSTechDL group.
  * It is imperfect, and we hope to keep iterating the design to enhance your learning experience.
  * Thank you very much for your understanding, patience, and cooperation.

## Grading Policy

* **Attendance**: 10%
* **Project #1 on DeepClaw Assignment**: 20%
  * In-class Workshop 1/2 (10%)
* **Project #2 on Paper Review**: 30%
  * 1st/2nd/3rd Paper Review (10%)
* **Project #3 on Final Project**: 40%
  * Project Proposal Presentation (5%)
  * Project Milestone Presentations (5%)
  * Final Report and Code (20%)
  * Spotlight Presentation Video (10%)

### Important Deadlines

* **Submission Deadline**
  * Mar 10 @ 23:30: 1st Paper Review PowerPoint
  * Mar 17 @ 23:30: DeepClaw Project Report 1
  * Mar 24 @ 23:30: Project Proposal PowerPoint
  * Apr 07 @ 23:30: 2nd Paper Review PowerPoint
  * Apr 07 @ 23:30: DeepClaw Project Report 1
  * Apr 14 @ 23:30: Project Milestone PDF
  * May 12 @ 23:30: 3rd Paper Review PowerPoint
  * Jun 02 @ 23:30: Final Report and Codes
  * Jun 06 @ 23:30: Spotlight Presentation Video
* **Course Feedback Form**
  * Apr 11 @ 23:30: Mid-term Course Feedback
  * Jun 06 @ 23:30: End-of-term Course Feedback

## Academic Integrity

* This course follows the SUSTech Code of Academic Integrity. Students in this course must abide by the SUSTech Code of Academic Integrity. Any work submitted by a student in this course for academic credit will be the student’s work. Violations of the rules will not be tolerated.

## University Calendar

<figure><img src="https://content.gitbook.com/content/MJwwmQ8eCYejhFgyqHrR/blobs/79kS0eEejglFyL0OZBso/assets/2024-university-calendar.png" alt="2024 Spring university calendar"><figcaption></figcaption></figure>

## Lecture & Lab Notes

**Recommended Textbook(s)**

Since this course meets the Machine Learning credit requirement (one of the four required machine learning courses for Robotic Engineering), we will cover extensive ML-related content in the lectures. If you are keen to learn more about it, you are recommended to refer to the following learning resources:

* CS329P: Practical Machine Learning: [Book & Codes](https://d2l.ai/index.html) | [Lecture Notes](https://c.d2l.ai/stanford-cs329p/syllabus.html#)
* CS231n: Deep Learning for Computer Vision: [Link](http://cs231n.stanford.edu/)

## Teaching Schedule

<table data-header-hidden><thead><tr><th width="46.2265625">Wk</th><th>Tuesday 1400~1550
Room 210, Teaching Hall #3</th><th>Friday 1620~1810
Room 210, Teaching Hall #3</th><th>Submission Deadlines</th></tr></thead><tbody><tr><td>01</td><td>W01-C01 | <br><a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FeDPpt6WG5G4V6QgLrVpd%2F23Spring-ME336-Lec01-CourseIntro.pdf?alt=media&#x26;token=a738b35c-4cdc-4a92-acef-bbd3b488091f"><strong>L01 on Course Introduction</strong></a></td><td>W01-C02 | <br><a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FGIia8HNBlMtVnopQzGzK%2F23Spring-ME336-Lec02-CobotDesign.pdf?alt=media&#x26;token=f00180d8-d20f-4469-a66b-a6dc3879b51b"><strong>L02 on Collaborative Robots</strong></a></td><td></td></tr><tr><td>02</td><td>W02-C03 | <br><a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FLdyUjmX2sFjrTkWPdxJr%2F23Spring-ME336-Lec03-PerceptionControl.pdf?alt=media&#x26;token=ae7ab9be-f8b2-447a-8b0f-266ae2666f62"><strong>L03 on Vision-based Perception</strong></a></td><td>W02-C04 | <strong>Guest Lecture</strong><br>– By Wang Haokun</td><td></td></tr><tr><td>03</td><td>W03-C05 | <br><a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FwK16s8w7q1bL9GdHKTQI%2F23Spring-ME336-Lec04-LearningBasics.pdf?alt=media&#x26;token=14d62197-6266-4871-a835-2f7973a42824"><strong>L04 on Machine Learning</strong></a></td><td>W03-C06 | <br><a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FJjkXre2x1UQdgyeKP9ka%2F23Spring-ME336-Lec05-LearningClassification.pdf?alt=media&#x26;token=a551d623-e9a9-4ce5-b833-be6103ab230f"><strong>L05 on Learning Classification</strong></a></td><td><strong>Mar 10 @ 23:30</strong>:<br>– 1st Paper Review PowerPoint</td></tr><tr><td>04</td><td>W04-C07 | <br><strong>DeepClaw Workshop A1</strong><br>– By WuTY &#x26; HuangBC</td><td>W04-C08 | <br><strong>DeepClaw Workshop A2</strong><br>– By WuTY &#x26; HuangBC</td><td><strong>Mar 17 @ 23:30</strong>:<br>– DeepClaw Workshop Report 1</td></tr><tr><td>05</td><td>W05-C09 | <strong>1st Paper Review</strong><br><br><strong>Team materials</strong><br>– Team 2: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FVsIr9UGck4JzZTtBGASA%2FTeam2-1stPaperReview-Paper.pdf?alt=media&#x26;token=d4f74939-299e-4272-92d1-880a407b015e">Paper</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FrxAjIdEXwj69jitO276a%2FTeam2-1stPaperReview-Report.pdf?alt=media&#x26;token=25f803cb-cc13-4ac8-90e8-c041efec0159">PDF</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FlILViOi7Oq27TrOpBfGs%2FTeam2-1stPaperReview-PPT.pptx?alt=media&#x26;token=a4e35b89-fd11-4f3d-b8c6-04893b2f6e21">PPT</a><br>– Team 6: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FliExBr4X21wXrKHsK3kF%2FTeam6-1stPaperReview-Paper.pdf?alt=media&#x26;token=28eb8513-6f25-4cbb-924f-4433b7d995f9">Paper</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FNWcWCwZGIonkJirLOFgz%2FTeam6-1stPaperReview-Report.pdf?alt=media&#x26;token=c0a88e9b-490c-4b9f-8609-faa4d5dcb3c4">PDF</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FRIsxrmTGOw0ejJ0TMLrO%2FTeam6-1stPaperReview-PPT.pptx?alt=media&#x26;token=ede36245-66b1-47bd-8b1c-8a3dbbcc1515">PPT</a><br>– Team 7: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FFIzGmzZic2SsOEkcn8hl%2FTeam7-1stPaperReview-Paper.pdf?alt=media&#x26;token=19bef93e-205d-4604-9e5b-5de73c1c433c">Paper</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2Fna8Bsetp7jXmuF9zxWpy%2FTeam7-1stPaperReview-Report.pdf?alt=media&#x26;token=fdf8a85f-8a13-4b1d-ae17-1a0bf7c5b400">PDF</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FyI3lMMOTz2DrxoXnrVHN%2FTeam7-1stPaperReview-PPT.pptx?alt=media&#x26;token=6cccf6b3-ab7c-4158-813a-b24c87a8e89e">PPT</a></td><td>W05-C10 | <br><a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2Ft6RJiBvMXYekWr4LleFF%2F23Spring-ME336-Lec06-NetworkBasics.pdf?alt=media&#x26;token=b5b457d4-5335-4c40-a485-a266d2b62c80"><strong>L06 on Neural Networks</strong></a></td><td><strong>Mar 24 @ 23:30</strong>:<br>– Project Proposal PowerPoint</td></tr><tr><td>06</td><td>W06-C11 | <br><strong>Project Proposal Presentation</strong><br><br><strong>Team materials</strong><br>– Team 1: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FJDLwSK8xiA30jwDySesu%2F24Spring-ME336-Team1-ProjectProposal.pptx?alt=media&#x26;token=1f63f5d1-58b9-4f85-afe7-b8e112caacbb">PowerPoint</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FrzAzp86LNk3B34q1vcPf%2F24Spring-ME336-Team1-ProjectProposal.pdf?alt=media&#x26;token=522fb562-7682-4f7e-9b55-24cd3b58b6fc">PDF</a><br>– Team 2: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2F3F1smFuOtavzELyKOxjs%2F24Spring-ME336-Team2-ProjectProposal.pptx?alt=media&#x26;token=adb14f36-9674-4314-89d8-faa2895c5d9f">PowerPoint</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2Fw9U4gQeyS7tU3CQTj2CZ%2F24Spring-ME336-Team2-ProjectProposal.pdf?alt=media&#x26;token=ae236e4e-f75c-4b91-a3c6-577ac012b296">PDF</a><br>– Team 3: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FhWVG3NHxj7oF00eaUu9R%2F24Spring-ME336-Team3-ProjectProposal.pptx?alt=media&#x26;token=a7ad4013-6d37-4e15-aa29-b794c8e21649">PowerPoint</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2F6AQvJoaDwx74Ijy2m98n%2F24Spring-ME336-Team3-ProjectProposal.pdf?alt=media&#x26;token=0ca024ef-d4b4-49d6-b57c-595b98e11081">PDF</a><br>– Team 4: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FWoD20KO3kkQItgaNMuQ1%2F24Spring-ME336-Team4-ProjectProposal.pptx?alt=media&#x26;token=8a5f3dbe-7a7b-4e8d-a690-d75ea8ee7c07">PowerPoint</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FYxWGt4JIirGgfxtvdLnY%2F24Spring-ME336-Team4-ProjectProposal.pdf?alt=media&#x26;token=63340845-1391-4423-98f1-d447a3bcd771">PDF</a><br>– Team 5: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FIhLqWvKXPYC6FjETwlvq%2F24Spring-ME336-Team5-ProjectProposal.pptx?alt=media&#x26;token=d6af496f-2679-47d6-aba3-6d1d2cf8ce07">PowerPoint</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2Frhun7hRHnuht1bhJk2XD%2F24Spring-ME336-Team5-ProjectProposal.pdf?alt=media&#x26;token=38ebdc9e-ae13-4ca4-a12e-bf7c21555260">PDF</a><br>– Team 6: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FBRqPL32ygUcUFbee6pIb%2F24Spring-ME336-Team6-ProjectProposal.pptx?alt=media&#x26;token=4223b5a8-5902-484a-88e2-0b6388e6470c">PowerPoint</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2F6PwJ8s6w0r8VFHXIc7Gq%2F24Spring-ME336-Team6-ProjectProposal.pdf?alt=media&#x26;token=00e4601c-f613-452b-8fcd-239883f533b0">PDF</a><br>– Team 7: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2Fsy3NBHFevxxBkRnKZ8d3%2F24Spring-ME336-Team7-ProjectProposal.pptx?alt=media&#x26;token=99d1d96a-4be1-4406-bec0-fc7d7c28e6fc">PowerPoint</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FZIQ4vKbsI7YpbfWNZG2p%2F24Spring-ME336-Team7-ProjectProposal.pdf?alt=media&#x26;token=d1daaed7-94b5-451b-b0d0-f7c493ed4f7a">PDF</a><br>– Team 8: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FYjeDHw8WiXr2CE51g7TF%2F24Spring-ME336-Team8-ProjectProposal.pptx?alt=media&#x26;token=32b0bee4-58d8-4a3c-9158-0c04037b3153">PowerPoint</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FseIsRoVzXfJgjtxYNoFI%2F24Spring-ME336-Team8-ProjectProposal.pdf?alt=media&#x26;token=06bc62cd-4969-476d-8a3c-66ab81c46b22">PDF</a><br>– Team 9: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FW4LCjD14h5X4BPicwgo7%2F24Spring-ME336-Team9-ProjectProposal.pptx?alt=media&#x26;token=a4e9d9c5-e724-4a4f-a79f-89ce7a3fbdc8">PowerPoint</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FxMFnGPr0qHrNFGilW43K%2F24Spring-ME336-Team9-ProjectProposal.pdf?alt=media&#x26;token=cbe857c3-b5f7-404c-893e-0b44e0c7bdac">PDF</a></td><td>W06-C12 | <br><a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FOU6zkRuxFKqnAuafFllj%2F23Spring-ME336-Lec07-ConvNetworks.pdf?alt=media&#x26;token=a571c00e-e745-4301-8b0d-40499163e8d4"><strong>L07 on Convolutional Networks</strong></a></td><td></td></tr><tr><td>07</td><td>W07-C13 | <br><strong>DeepClaw Workshop B1</strong><br>– By WuTY &#x26; HuangBC</td><td>W07 | <del><strong>Qingming Holiday</strong></del><br>W07-C14 | <br><strong>DeepClaw Workshop B2</strong><br>– By WuTY &#x26; HuangBC</td><td><strong>Apr 07 @ 23:30</strong>:<br>– 2nd Paper Review PowerPoint<br>– DeepClaw Workshop Report 2</td></tr><tr><td>08</td><td>W08-C15 | <br><a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FcxFTBCyn4FLOGuhkfVzU%2F23Spring-ME336-Lec08-Regularization.pdf?alt=media&#x26;token=e587a825-76dc-4b06-b00f-9b954b7b056d"><strong>L08 on Regularization</strong></a></td><td>W08-C16 | <strong>2nd Paper Review</strong><br><br><strong>Team materials</strong><br>– Team 3: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FRfbH0iOprIPIaOsLzoMj%2F24Spring-ME336-Team3-2ndPaperReview-Paper.pdf?alt=media&#x26;token=807516df-97dc-4808-af25-498e3f4c3419">Paper</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2F5fc5UkTOj2o0hfdYI9vP%2F24Spring-ME336-Team3-2ndPaperReview-Report.pdf?alt=media&#x26;token=00e09c5f-6086-4797-a09e-1bdb2b4dc149">PDF</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FLKf89RVxYYO2Lz0pqsX4%2F24Spring-ME336-Team3-2ndPaperReview-PPT.pptx?alt=media&#x26;token=b1ef0a2d-cd5b-4109-a5d7-a431dd6c970c">PPT</a><br>– Team 4: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2F58GxwD7PxjOyvo2dC3lK%2F24Spring-ME336-Team4-2ndPaperReview-Paper.pdf?alt=media&#x26;token=1131c77c-bcc9-4078-b658-237e14d44829">Paper</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2Fk2EBbXqRFbhWJDU8douR%2F24Spring-ME336-Team4-2ndPaperReview-Report.pdf?alt=media&#x26;token=a9a513b2-2bce-46c2-9553-4c014aaa54a8">PDF</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FhukQbeAD3bJEkM8wwaYv%2F24Spring-ME336-Team4-2ndPaperReview-PPT.pptx?alt=media&#x26;token=efee820f-8790-46ff-96f9-ae7b3ffb191b">PPT</a><br>– Team 8: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FLnBrY9RIgw49XY8CaFD5%2F24Spring-ME336-Team8-2ndPaperReview-Paper.pdf?alt=media&#x26;token=00eddddf-6951-4adf-9a31-abfd99face6a">Paper</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FO6DXycNQ8PhKcERVwaXt%2F24Spring-ME336-Team8-2ndPaperReview-Report.pdf?alt=media&#x26;token=c9b134ba-1f4f-47ba-90da-a343c7da55a6">PDF</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2F3hrUWU5N5Pdj1PKMS2cl%2F24Spring-ME336-Team8-2ndPaperReview-PPT.pptx?alt=media&#x26;token=1f0cea5a-c52f-41d8-97ea-0cb906f9a1be">PPT</a></td><td><strong>Apr 14 @ 23:30</strong>:<br>– Project Milestone PDF</td></tr><tr><td>09</td><td>W09-C17 | <br><strong>Project Milestone Presentation</strong><br><br><strong>Team materials</strong><br>– Team 1: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FuilriFheoGjjhEidFF1g%2F24Spring-ME336-Team1ProjectMilestone.pdf?alt=media&#x26;token=0dbeae26-b00f-467a-9404-3bce1568ffd3">PDF</a><br>– Team 2: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FwyGMfCTXKvWILxLGitAH%2F24Spring-ME336-Team2ProjectMilestone.pdf?alt=media&#x26;token=a9b440c0-67c4-476d-8cf0-50dd1ae241c7">PDF</a><br>– Team 3: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FhrNgeJf8GOMCuyyyjnsN%2F24Spring-ME336-Team3ProjectMilestone.pdf?alt=media&#x26;token=a4d266f3-4c57-4fce-aa4c-346980348043">PDF</a><br>– Team 4: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FbXWWDrwxg4ggHECpPQd1%2F24Spring-ME336-Team4ProjectMilestone.pdf?alt=media&#x26;token=78a12046-5952-4fc0-a64a-25a2e600b433">PDF</a><br>– Team 5: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2Fnodl24B8qIFjugsXMIQH%2F24Spring-ME336-Team5ProjectMilestone.pdf?alt=media&#x26;token=9a5fb892-a4e5-4dd1-bbb5-ed538097ea5b">PDF</a><br>– Team 6: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FGl1bTF4kwyRqmDQC9t2x%2F24Spring-ME336-Team6ProjectMilestone.pdf?alt=media&#x26;token=b878dc84-9060-458a-a8da-8f29c12703c8">PDF</a><br>– Team 7: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2Fh9FjtsYTj6RgRWMAgITU%2F24Spring-ME336-Team7ProjectMilestone.pdf?alt=media&#x26;token=79854a0c-b5a3-4cbd-8ed3-901db77236c5">PDF</a><br>– Team 8: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FBWEG5UDy43ZSJHFmddX8%2F24Spring-ME336-Team8ProjectMilestone.pdf?alt=media&#x26;token=c36e0008-059b-46b6-aea1-f589b82d26f0">PDF</a><br>– Team 9: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2F1HU3MrW0jmiTPFH8bE3e%2F24Spring-ME336-Team9ProjectMilestone.pdf?alt=media&#x26;token=f5e8a067-9434-43ce-b838-ad17b8199ad0">PDF</a></td><td>W09-C18 | <br><a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FEYgxATgqbrfM0qqBC4GN%2F23Spring-ME336-Lec09-Optimization.pdf?alt=media&#x26;token=870fa872-9155-435c-99ac-7da601e12fa8"><strong>L09 on Optimization</strong></a></td><td></td></tr><tr><td>10</td><td>W10-C19 | <br><a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FbXbSoutRkS9KMOuLWID6%2F23Spring-ME336-Lec10-MarkovianBasics.pdf?alt=media&#x26;token=07c4a844-e230-40ca-8bed-e4a108317a13"><strong>L10 on Markovian Modeling</strong></a></td><td>W10-C20 | <br><a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FPkwPD0unaX6F6kZS4SOz%2F23Spring-ME336-Lec11-MarkovianLearning.pdf?alt=media&#x26;token=5dcb82ab-b642-4307-93ef-0153a80e3b8a"><strong>L11 on Markovian Learning</strong></a></td><td></td></tr><tr><td>11</td><td>W11-C21 | <br><a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2F2ryB0mreGba7hHynFKrD%2F23Spring-ME336-Lec12-ManipulationLearning.pdf?alt=media&#x26;token=efac0d1e-726b-4f38-8867-b0e7cfabedb2"><strong>L12 on Manipulation Learning</strong></a></td><td>W11 | <br><del><strong>National Holiday</strong></del></td><td></td></tr><tr><td>12</td><td>W12-C22 | <br><strong>DeepClaw Workshop C1</strong><br>– By WuTY &#x26; HuangBC</td><td>W12-C23 | <br><a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FY4XOXlapQL6D1GDVGurk%2F23Spring-ME336-Lec13-ChatGPT4Robotics.pdf?alt=media&#x26;token=047aff50-b35f-4df8-adca-1b810ab9c08d"><strong>L13 on ChatGPT for Robotics</strong></a></td><td><strong>May 12 @ 23:30</strong>:<br>– 3rd Paper Review PowerPoint</td></tr><tr><td>13</td><td>W13-C24 | <br><strong>3rd Paper Review</strong><br><br><strong>Team materials</strong><br>– Team 1: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FmMXJMRG2aVutTh4wlDKI%2F24Spring-ME336-Team1-3rdPaperReview-Paper.pdf?alt=media&#x26;token=0c2d2c2d-c5e6-42ce-b6f6-0f193a792784">Paper</a>| <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FVBJ54KjVuZdIQCKDl3DR%2F24Spring-ME336-Team1-3rdPaperReview-PDF.pdf?alt=media&#x26;token=f97ada0d-2ac9-4b1b-aca2-716edeb73804">PDF</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2F3iM7eSJrRudWiuodSM1X%2F24Spring-ME336-Team1-3%E3%80%81rdPaperReview-PPT.pptx?alt=media&#x26;token=584587c8-d33b-4290-b19e-0db642d6ef29">PPT</a><br>– Team 5: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FybTQhx9iJaiSBkFKk8mL%2F24Spring-Team5Paper.pdf?alt=media&#x26;token=7d2b329c-4ced-4d9d-92d4-29c3d2855297">Paper</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2Fw0nYR5pv3DnlECgtafqr%2F24Spring-ME336-Team5-3ndPaperReview-PPT.pdf?alt=media&#x26;token=8ab8c1a0-6d7a-4d24-9647-32ec00175298">PDF</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FTNyCtak9o52DE5gn8VZR%2F24Spring-ME336-Team5-3rdPaperReview-PPT.pptx?alt=media&#x26;token=fdb219c1-bcbe-405e-857f-aec9bae592e3">PPT</a><br>– Team 9: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FEJOApy7oPUOARDeH6xhq%2F24Spring-ME336-Team9-3ndPaperReview-Paper.pdf?alt=media&#x26;token=dc0a9bb7-2e04-4f4a-9eaa-04473965ca52">Paper</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FAz9gNR8l7kG8xqQZUuCj%2F24Spring-ME336-Team9-3ndPaperReview-Report.pdf?alt=media&#x26;token=23c44c4d-15d3-4c90-bacb-00c27e12e784">PDF</a> | <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FiMnydBKE8wS9RrDfJTPe%2F24Spring-ME336-Team9-3ndPaperReview-PPT.pptx?alt=media&#x26;token=591de894-5427-4e0d-8989-e11ad44b8161">PPT</a></td><td>W13-C25 | <br><strong>Final Project Preparation</strong></td><td></td></tr><tr><td>14</td><td>W14-C26 | <br><strong>Final Project Preparation</strong></td><td>W14-C27 | <strong>Guest Lecture</strong><br>– By Dr. Qiu Qiang</td><td></td></tr><tr><td>15</td><td>W15-C28 | <br><strong>Final Project Preparation</strong></td><td>W15-C29 | <br><strong>Final Project Preparation</strong></td><td><strong>Jun 02 @ 23:30</strong>:<br>– Final Report and Codes</td></tr><tr><td>16</td><td>W16-C30 | <br><strong>Feedback on Final Report</strong></td><td>W16-C31 | <br><strong>Final Spotlight Presentation</strong><br><br><strong>Team materials</strong><br>– Team 1: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FQDXNc2BilPGWBZg9YWH5%2F24Spring-ME336-Final-Team1-Report.pdf?alt=media&#x26;token=bb0dbd71-f16f-460f-93b5-7db14424729e">Report</a> | <a href="https://pan.baidu.com/s/1GRgG6E97MIPXF6aYbn-xRw?pwd=9344">Video</a> | Code<br>– Team 2: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FbESMizKgzy5tzO7NURQz%2F24Spring-ME336-Final-Team2-Report.pdf?alt=media&#x26;token=0d5ca0ca-db12-4c4e-acde-4b1ac96dbbe8">Report</a> | <a href="https://pan.baidu.com/s/1GRgG6E97MIPXF6aYbn-xRw?pwd=9344">Video</a> | Code<br>– Team 3: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FMO23Sflqp3lBeARmOiT8%2F24Spring-ME336-Final-Team3-Report.pdf?alt=media&#x26;token=dd7fc4e2-574a-4a74-a5ab-dafaa0a361bd">Report</a> | <a href="https://pan.baidu.com/s/1GRgG6E97MIPXF6aYbn-xRw?pwd=9344">Video</a> | Code<br>– Team 4: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FUODCx4gqRUz1uo3l6v5A%2F24Spring-ME336-Final-Team4-Report.pdf?alt=media&#x26;token=b1e76127-3ab0-4829-b58b-e783180fbcfc">Report</a> | <a href="https://pan.baidu.com/s/1GRgG6E97MIPXF6aYbn-xRw?pwd=9344">Video</a> | Code<br>– Team 5: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FOimXHyyFDq1OXVDB0bdq%2F24Spring-ME336-Final-Team5-Report.pdf?alt=media&#x26;token=d38e1085-d6b6-4ae9-a8de-16232c8b489d">Report</a> | <a href="https://pan.baidu.com/s/1GRgG6E97MIPXF6aYbn-xRw?pwd=9344">Video</a> | Code<br>– Team 6: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2F7RbYIeSUKMjW8TYFDx5Q%2F24Spring-ME336-Final-Team6-Report.pdf?alt=media&#x26;token=dec1d59e-b411-458e-bbab-f42f22ec7e7b">Report</a> | <a href="https://pan.baidu.com/s/1GRgG6E97MIPXF6aYbn-xRw?pwd=9344">Video</a> | Code<br>– Team 7: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FKxei5LuZBvWlFNmdQkkl%2F24Spring-ME336-Final-Team7-Report.pdf?alt=media&#x26;token=93fed225-1f27-4316-a000-bb583264436a">Report</a> | <a href="https://pan.baidu.com/s/1GRgG6E97MIPXF6aYbn-xRw?pwd=9344">Video</a> | Code<br>– Team 8: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FVQrLoiGVi7hMpqpwNxVR%2F24Spring-ME336-Final-Team8-Report.pdf?alt=media&#x26;token=0d6eba4a-6bb7-4db1-a157-8c3fbf0a2928">Report</a> | <a href="https://pan.baidu.com/s/1GRgG6E97MIPXF6aYbn-xRw?pwd=9344">Video</a> | Code<br>– Team 9: <a href="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2Fq2CLKCH4FIXt1uSC88St%2F24Spring-ME336-Final-Team9-Report.pdf?alt=media&#x26;token=1d06f53b-0d90-4ca0-b34f-9ecbe1349f80">Report</a> | <a href="https://pan.baidu.com/s/1GRgG6E97MIPXF6aYbn-xRw?pwd=9344">Video</a> | Code</td><td><strong>Jun 06 @ 23:30</strong>:<br>– Spotlight Presentation Video<br>Final project peer-evaluation QR reference</td></tr></tbody></table>

## Content Summary

**Detailed Course Pages**

* [DeepClaw Workshops](#deepclaw-workshops)
* [Paper Reviews](#paper-reviews)
* [Assignment Projects](#assignment-projects)
* [Course Project](#course-project)

***

## Student Team and Group Formulation

This course comprises four modules: Machine Learning, Deep Networks, Regularization & Optimization, and Advanced Topics. Each module will be completed within four weeks and includes the following:

* 4 Main Lecture Sessions by the Lead Instructor
* 2 DeepClaw Workshop Sessions by Teaching Assistants
* 1 Project Presentation Session by the Students
* 1 Paper Review Session by the Students

We will assign a DeepClaw toolkit to two students as a pair for the workshop session.

6 students will form a team for the paper review and project presentation sessions. **Register your team here**.

## DeepClaw Workshops

[DeepClaw](https://github.com/bionicdl-sustech/DeepClaw) is a set of tools developed by the SUSTechDL group over the past few years to support teaching and research on robot learning, where we believe robotic hardware should play a central role in bridging simulation and reality for manipulation learning problems. We originally developed it as a reconfigurable workstation for vision-based robotic picking at the SIR Group at Monash University ([FROBT 2018](https://www.frontiersin.org/articles/10.3389/frobt.2018.00086/full)). After transitioning to SUSTech, we evolved the system to include APIs to communicate with various robotic hardware. We streamlined the mechanical design using standard aluminum extrusion systems ([AIM 2020](https://arxiv.org/abs/2005.02588)). With the growing ME336 student base and various hardware integration constraints, we developed an evolved version using soft robotic tongs for more efficient data collection in grasping tasks ([FROBT 2022](https://www.frontiersin.org/articles/10.3389/frobt.2022.787291/full)). In the latest version, we further integrated tactile learning to refine the system and make it portable and low-cost ([CoRL 2021, unpublished](https://github.com/bionicdl-sustech/SoftRoboticTongs)). Recently, we published a paper describing its theoretical foundation and robotic applications (MatDes, 2024) and submitted another paper introducing its applications in teaching robot learning (ICARM2024, Submitted). The DeepClaw Project won the [2023 UNESCO-ICHEI Higher Education Digitalisation Pioneer Case Award](http://deepclaw.ancorasir.com/). Students are central to the iterative development of DeepClaw at SUSTechDL. Since Dr. Song and Dr. Wan developed the original version, many students have contributed to subsequent development. Almost all SUSTechDL students are involved to some extent. [Please refer to this page for key developers and milestones.](https://deepclaw.ancorasir.com/)

This semester, we will introduce the latest development of the DeepClaw system to our students and use it as a teaching tool to support you as you learn ME336. We have planned three workshop sessions, delivered by the Teaching Assistants, to cover topics in the following areas. We will provide one set of the DeepClaw toolkit to two or three students for practice.

* [DeepClaw Workshops A1 & A2](https://x.ancorasir.com/me336/me336-collaborative-robot-learning/deepclaw-workshops/session-1) (Related to Project Assignment #1)
* [DeepClaw Workshops B1 & B2](https://x.ancorasir.com/me336/me336-collaborative-robot-learning/deepclaw-workshops/session-2) (Related to Project Assignment #2)
* [DeepClaw Workshops C1 & C2](https://x.ancorasir.com/me336/me336-collaborative-robot-learning/deepclaw-workshops/session-3) (No Project Assignment)
* [Debugging and Solutions](https://x.ancorasir.com/me336/me336-collaborative-robot-learning/deepclaw-workshops/debugging-and-solutions)

## Paper Reviews

Starting this semester, we will introduce a paper review session where students conduct a systematic literature review on robot learning research through presentations and in-class discussions. We adapted the general format from [Dr. Zhu Yuke’s course](https://www.cs.utexas.edu/~yukez/cs391r_fall2020/logistics.html). Each student team should expect at least one presentation regarding their assigned paper reading. To ensure the quality and clarity of the presentations, we expect the students to

* read the assigned papers thoroughly and develop a strong understanding before creating the presentation slides.\
  **Template:** \[[PowerPoint](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FneS3Wjardbwr0tRWcNvW%2FTemplate-PaperReview.pptx?alt=media\&token=aec7e1cd-e500-4f8b-b886-28ec2f286a5a)]
* Email the slides to the TAs and the instructor seven days before the presentation date for feedback and revision.\
  **Deadline**: \[Link to Submission]
  * **Mar 10 @ 23:30**: 1st Paper Review PowerPoint, to be presented on Mar 12 class
  * **Apr 07 @ 23:30**: 2nd Paper Review PowerPoint, to be presented on Apr 09 class
  * **May 12 @ 23:30**: 3rd Paper Review PowerPoint, to be presented on May 14 class
* [Recommended Papers to Review](https://me336.ancorasir.com/?page_id=155)\
  You can choose from the list above or search for others that suit your interest and are related to the course topic. Consult with the lead instructor if you have any questions.

Failing to email the PowerPoint on time will incur a 20% deduction from the presentation score. Each presentation should be **20min** (± 2min). The presentations will be graded in the following aspects:

* Clarity of presentation (problem formulation, proposed method, key results);
* Review of prior work and the challenges addressed by this work;
* Analysis of the strengths and weaknesses of the research;
* Discussion of potential research extensions and applications;
* Response to in-class student questions.

After each presentation, we will do a **5-minute** Q\&A. Each class will present three papers, followed by a 20-minute open-ended discussion. The instructor will post the discussion questions through QQ the day before the class. The Paper Review presentation will be worth **30%** of the total grade. For students who present more than once (based on availability), the final presentation grade will be the **highest** score of their presentations.

**Useful Resources**

* [Tips for Giving Clear Talks](http://graphics.stanford.edu/~kayvonf/misc/cleartalktips.pdf), Kayvon Fatahalian
* [Three Tips for Giving a Great Research Talk](https://www.sciencemag.org/careers/2019/04/three-tips-giving-great-research-talk), Lewis, Gruber, Van Bavel, Somerville
* [CoRL 2019](https://sites.google.com/robot-learning.org/corl2019) talk videos ([day 1](https://www.youtube.com/watch?v=glMwF5-q51E), [day 2](https://www.youtube.com/watch?v=b7StSnt85S4), [day 3](https://www.youtube.com/watch?v=QaCuEv_7lfs)) and [RSS 2019](https://www.youtube.com/channel/UCeEbAUGjtBlzmqWO5u6VeGg) talk videos
* [Reviewing a CS Conference Paper](https://cs.uwaterloo.ca/~smann/Research/review-conference.txt), Stephen Mann
* [How to Write Good Reviews for CVPR](https://www.dropbox.com/s/725p60wcajbb8xh/How%20to%20Review%20for%20CVPR.pptx?dl=0), CVPR 2019 Program Chairs
* Example reviews from the [ICLR](https://openreview.net/group?id=ICLR.cc) and [NeurIPS](https://papers.nips.cc/book/advances-in-neural-information-processing-systems-32-2019) conferences.

## Assignment Projects

### Notes on Assignment Projects

* The assignment projects aim to provide a reproducible practice experience for all students to achieve the intended learning outcomes. In other words, it is not (and never) our intention to make it too hard to complete, but in a guided way, preferably with some interesting aspects, for you to work out the assigned projects.
  * We recommend completing the assigned projects within class. If necessary, you may extend the submission deadline to next Sunday at 23:30.

### Assignment Project 1

* Your assignment is to use the raw dataset provided [here](https://github.com/bionicdl-sustech/SoftRoboticTongs/tree/main/SupplementaryFile-A-ThreeDataset) (Wu Tianyu) to build a simple model that estimates the force and torque of the soft finger network with the highest accuracy.
  * More details are [here](https://me336.ancorasir.com/?page_id=457).
  * The input to your model should be the raw dataset, and your model’s output should be the three forces and three torques estimated.
  * The evaluation metric is mainly the prediction accuracy (%), which should be as high as possible at the lowest cost.
  * You may use any model or programming language, but you must implement the entire model independently.

### Assignment Project 2

* Your assignment is to use the DeepClaw toolkit to collect action data (motion + force/torque) for a specific manipulation task, visualize and analyze the collected data, and train a model to learn the action with the highest performance.
  * More details are [here](https://me336.ancorasir.com/?page_id=457).
  * The input to your model should be the collected raw dataset, and your model’s output should be the action (motion + force/torque) learned.
  * The evaluation metric focuses on the learned action, including motion and force/torque, and how well it represents and recreates the action when performing the task.
    * For example, suppose your chosen task is a trajectory of an English character, using the collected data. In that case, the model you trained should be able to reproduce a new path for the English character with a high success rate (%).
  * We will mainly limit the task to the following scope: manually holding the tongs with a soft object between the fingers, then moving the tongs in space while squeezing them for different purposes.
    * Move the tongs in 3D space: you can move in any way you choose, as long as your team decides on a specific geometry. Optional choices include an English letter, a Chinese character, a logo with reasonable geometry, etc.
    * Assign meanings while squeezing the tongs: one way to integrate force/torque sensing into the recorded motion data is to add extra expressions to the recorded trajectory. For example, when squeezed, the estimated changes in force and torque could represent the changing intensity, color range, trajectory size, or even on/off commands.

## Course Project

For the final project, you will work in a group of 4 students to propose a feasible project idea that uses available tools and resources and applies AI-based techniques to practical robot perception and decision-making problems. It consists of four components and is worth 40% of your total grade.

* Project Proposal Presentation (5%)
* Project Milestone Presentations (5%)
* Final Report and Code (25%)
* Spotlight Presentation Video (5%)

**Useful Resources**

* [Writing in the Sciences](https://online.stanford.edu/courses/som-ysciewrite-writing-sciences), Kristin Sainani ([Coursera](https://www.coursera.org/learn/sciwrite), [YouTube](https://www.youtube.com/channel/UC-wb-n89yM0lBiP2QltsDaA))
* [How to Write a Good CVPR Submission](https://billf.mit.edu/sites/default/files/documents/cvprPapers.pdf), Bill Freeman
* [Novelist Cormac McCarthy’s Tips on How to Write a Great Science Paper](https://www.nature.com/articles/d41586-019-02918-5), Van Savage, Pamela Yeh

A successful project topic should involve at least one, ideally both, of the two critical components: a perception component, i.e., processing raw sensory data, and a decision-making component, i.e., controlling robot actions. You can choose to use DeepClaw or RoboSuite based on your needs. For example,

* Learning vision-based robot manipulation with deep reinforcement methods;
* Self-supervised representation learning of visual and tactile data;
* Model-based object pose estimation for 6-DoF grasping from RGB-D images.

Potential projects can have the following flavors:

* **Improve an existing approach.** Select a paper you are interested in, reimplement it, and improve it using what you learned in the course.
* **Apply an algorithm to a new problem.** You will need to understand the strengths and weaknesses of an existing algorithm from research work, reimplement it, and apply it to a new problem.
* **Stress test existing approaches.** This kind of project thoroughly compares several existing approaches to a robot learning problem.
* **Design your own approach.** You develop an entirely new approach to a specific problem in these projects. The problem may even be something that has not been considered before.
* **Mix and Match approach.** For these projects, you typically combine approaches that have been developed separately to address a larger and more complex problem.
* **Join a research project.** You can join an existing Robot Learning project with UT faculty and researchers. You are expected to articulate your own contributions in your project reports (more detail below).

You will work as a team, and grades will be calibrated based on individual contribution. Your project may be related to research in another class project as long as consent is granted by instructors of both classes; however, you must clearly indicate in the project proposal, milestone, and final reports the exact portion of the project that is being counted for this course. In this case, you must prepare separate reports for each course and also submit the final report for the other course.

### Project Inspirations and Resources

To inspire ideas, you might also look at recent **robotics publications** from top-tier conferences and other resources below.

* [RSS](https://roboticsconference.org/): Robotics: Science and Systems
* [ICRA](https://www.icra2019.org/): IEEE International Conference on Robotics and Automation
* [IROS](https://www.iros2019.org/): IEEE/RSJ International Conference on Intelligent Robots and Systems
* [CORL](https://www.robot-learning.org/): Conference on Robot Learning
* [ICLR](https://iclr.cc/Conferences/2019/Schedule): International Conference on Learning Representations
* [NeurIPS](https://papers.nips.cc/book/advances-in-neural-information-processing-systems-32-2019): Neural Information Processing Systems
* [ICML](http://proceedings.mlr.press/v97/): International Conference on Machine Learning
* [Publications](https://www.cs.utexas.edu/~yukez/publications/) from the UT Robot Perception and Learning Lab

You may also look at popular **simulated environments** and **robotics datasets** listed below.

**Simulated Environments**

* [robosuite](https://robosuite.ai/): MuJoCo-based toolkit and benchmark of learning algorithms for robot manipulation
* [RoboVat](https://github.com/StanfordVL/robovat): Tabletop manipulation environments in Bullet Physics
* [OpenAI Gym](https://gym.openai.com/): MuJoCo-based environments for continuous control and robotics
* [AI2-THOR](https://ai2thor.allenai.org/): open-source interactive environments for embodied AI
* [RLBench](https://sites.google.com/view/rlbench): robot learning benchmark and learning environment built around V-REP
* [CARLA](http://carla.org/): self-driving car simulator in Unreal Engine 4
* [AirSim](https://microsoft.github.io/AirSim/): a simulator for autonomous vehicles built on Unreal Engine / Unity
* [Interactive Gibson](http://svl.stanford.edu/gibson2/): an interactive environment for learning robot manipulation and navigation
* [AI Habitat](https://aihabitat.org/): simulation platform for research in embodied artificial intelligence

**Robotics Datasets**

* [Dex-Net](https://berkeleyautomation.github.io/dex-net/): 3D synthetic object model dataset for object grasping
* [RoboTurk](http://roboturk.stanford.edu/): crowdsourced human demonstrations in simulation and real world
* [RoboNet](https://www.robonet.wiki/): video dataset for large-scale multi-robot learning
* [YCB-Video](https://rse-lab.cs.washington.edu/projects/posecnn/): RGB-D video dataset for model-based 6D pose estimation and tracking
* [nuScenes](https://www.nuscenes.org/): large-scale multimodal dataset for autonomous driving

### Project Proposal

The project proposal should be one paragraph (300-400 words). Your project proposal should describe the following:

* (20%) What problem will you investigate? Why is it interesting?
* (20%) What reading will you examine to provide context and background?
* (20%) What data will you use? If you are collecting new data, how will you do it?
* (20%) What method or algorithm are you proposing? If existing implementations exist, will you use them, and how? How do you plan to improve or modify such implementations? You don’t have to have an exact answer at this point, but you should have a general sense of how you will approach the problem you are working on.
* (20%) How will you evaluate your results? Qualitatively, what kind of results do you expect (e.g., plots or figures)? Quantitatively, what kind of analysis will you use to evaluate and/or compare your results (e.g., what performance metrics or statistical tests)?

**Submission:** \[Deadline: Mar 17 @ 23:30] | \[[Proposal Template](https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMJwwmQ8eCYejhFgyqHrR%2Fuploads%2FGDU7i9jqkrVvGiC0KhkX%2FTemplate-Project-Proposal.pptx?alt=media\&token=b381262d-89c1-4d43-a5e2-ac74e3c17547)] | \[Link for Submission] | **Only one person on your team should submit.**

### Project Milestone

Your project milestone report should be between 2-3 pages using the [RSS template](https://roboticsconference.org/docs/paper-template-latex.tar.gz) in LaTeX. The following is a suggested structure for your report:

* **Title, Author(s)**
* **Introduction:** Introduce your problem and the overall plan for approaching your problem
* **Problem Statement:** Describe your problem precisely, specifying the dataset to be used, expected results, and evaluation
* **Literature Review:** Describe important related work and its relevance to your project
* **Technical Approach:** Describe the methods you intend to apply to solve the given problem
* **Intermediate/Preliminary Results:** State and evaluate your results up to the milestone

**Submission:** \[Deadline: Apr 14 @ 23:30] | \[[Milestone Template](https://roboticsconference.org/docs/paper-template-latex.tar.gz)] | \[Link for Submission] | **Only one person on your team should submit.**

### Final Report

Your final write-up must be between 6-8 pages (8 pages max) using the RSS template, structured like a paper from a robotics conference. Please use this template so we can fairly judge all student projects without worrying about altered font sizes, margins, etc. After the class, we will post all the final reports online so you can read about each other’s work. If you do not want your write-up to be posted online, please let us know when submitting it. The following is a suggested structure for your report and the rubric we will follow when evaluating reports. You don’t necessarily have to organize your report using these sections in this order, but that would likely be a good starting point for most projects.

* **Title, Author(s)**
* **Abstract:** Briefly describe your problem, approach, and key results. Keep it to no more than 300 words.
* **Introduction (10%):** Describe the problem you are working on, why it’s important, and an overview of your results
* **Related Work (10%):** Discuss published work on your project. How is your approach similar or different from others?
* **Data (10%):** Describe the data or simulation environment you are working with for your project. What type is it? Where did it come from? How much data are you working with? How many simulation runs did you work with? Did you have to do any preprocessing, filtering, or other special treatment to use this data in your project?
* **Methods (30%):** Discuss your approach to solving the problems you set up in the introduction. Why is your approach the right thing to do? Did you consider alternative approaches? You should demonstrate that you have applied ideas and skills built up during the quarter to tackling your problem of choice. It may be helpful to include figures, diagrams, or tables to describe your method or compare it with other methods.
* **Experiments (30%):** Discuss the experiments you performed to demonstrate that your approach solves the problem. The exact experiments will vary by project. Still, you might compare with previously published methods, perform an ablation study to determine the impact of various components of your system, experiment with different hyperparameters or architectural choices, use visualization techniques to gain insight into how your model works, discuss common failure modes of your model, etc. You should include graphs, tables, or other figures to illustrate your experimental results.
* **Conclusion (5%):** Summarize your key results – what have you learned? Suggest ideas for future extensions or new applications.
* **Writing / Formatting (5%):** Is your paper clearly written and nicely formatted?
* **Supplementary Material** is not counted toward your 6-8 page limit and is submitted as a separate file. Your supplementary material might include the following:

  * Source code (if your project proposed an algorithm or code that is relevant and important for your project).
  * Cool videos, interactive visualizations, demos, etc.

  Examples of things not to put in your supplementary material:

  * The entire PyTorch/TensorFlow GitHub source code.
  * Any code that is larger than 10 MB.
  * Model checkpoints.
  * A computer virus.

**Submission:** \[Deadline: Jun 02 @ 23:30] | \[[Final Report Template](https://roboticsconference.org/docs/paper-template-latex.tar.gz)] | \[Link for Submission] | **Only one person on your team should submit.**

* **Your report PDF should list&#x20;*****all*****&#x20;authors who have contributed to your work** enough to warrant a co-authorship position. This includes people not enrolled in ME336, such as faculty/advisors who sponsored your work with funding or data, and significant mentors (e.g., Ph.D. students or postdocs who coded with you, collected data with you, or helped you draft your model on a whiteboard). All authors should be listed directly underneath the title on your PDF. Include a footnote on the first page indicating which authors are not enrolled in ME336. All co-authors should have their institutional/organizational affiliation specified below the title. If you have non-ME336 contributors, you will be asked to describe the following:
  * **Specify the involvement of non-ME336 contributors** (discussion, writing code, writing paper, etc.). For example, please see the author’s contributions to [AlphaGo (Nature 2016)](https://www.nature.com/nature/journal/v529/n7587/full/nature16961.html#author-information).
  * **Specify whether the project has been submitted to a peer-reviewed conference or journal.** Include the full name and acronym of the conference (if applicable). For example, Neural Information Processing Systems (NIPS). This only applies if you have already submitted your paper/manuscript and it is under review as of the report deadline.
* **Any code used as a base for projects must be referenced and cited in the body of the paper.** This includes assignment code, fine-tuning example code, open-source, or GitHub implementations. You can use a footnote or a full reference/bibliography entry.
* **If you are using this project for multiple classes, submit the other class PDF as well.** Remember, you may not use the same final report PDF for multiple classes.
* In summary, include all contributing authors in your PDF; include detailed non-ME336 co-author information; tell us if you submitted to a conference, cite any code you used, and submit your dual-project report.

### Spotlight Talk

You can present your awesome work to the instructor and other students in the last week of class. This resembles the spotlight talks at large AI conferences, such as CVPR, NeurIPS, RSS, and CoRL. See an example spotlight video at [RSS 2018](https://www.youtube.com/watch?v=y77FKikIIyE). Each team must submit an MP4 video of the slides, preferably at 1280×720 resolution. This will let us load all talks onto the same laptop without configuration or format issues, while allowing presenters to use whatever graphics or video tools they choose to create the presentation. Presentations should be limited to 6 minutes and 55 seconds, with the next speaker’s video starting automatically at the 7-minute mark. If your video is longer than 6:55, it will be truncated. Please see the CVPR Presenter Instructions page for more details about converting presentation slides to videos. We will send information about uploading presentation slides and videos as the deadline approaches. For each project team, the spotlight talk can be presented by one member or multiple. The spotlight is worth 5% of the total grade and will be graded with the same criteria as the in-class paper presentations.

**Submission:** \[Deadline: Jun 06 @ 23:30] | \[Link for Submission] | **Only one person on your team should submit.**


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://x.ancorasir.com/me336/me336-collaborative-robot-learning.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.