CS598: Advanced Software Testing and Debugging

Course Overview

The purpose of this course is to teach the principles and practices of software testing and debugging. We will together explore advanced testing and debugging techniques to detect, diagnose, localize, and fix software bugs for real-world software systems from various application domains. This course will not only expose students to the cutting-edge research of software testing and debugging, but will also encourage students to explore the bidirectional synergy between software testing/debugging and other research areas, such as formal methods (FM), programming languages (PL), machine learning (ML), and security. This is a research-oriented seminar course with a major course project, including topics on:

Class Organization

This is a research-oriented seminar course, which does not have any textbook. In each class, the students will participate in the discussion of two research papers related to the same topic. Note that usually the primary paper will be formally presented and discussed in details, while the optional paper will be briefly discussed (especially its differences with the primary paper). During the class, the students will be randomly chosen to answer questions related to the papers, such as:

Grading Details

There is no exam! The grading will be mainly based on the following aspects:

Tentative Schedule

Date Topics/Readings Memo
08/25 Course Introduction
    		 Presenter: Lingming Zhang (Slides)
    08/27 Program Analysis Basics (I)
      		 Presenter: Lingming Zhang (Slides)
      09/01 Program Analysis Basics (II)
        		 Presenter: Lingming Zhang (see above link)
        09/03 Formal Methods Basics
          		 Presenter: Lingming Zhang (Slides)
          09/05 Presentation Choice Submission (11:59pm) Submission Link: [Link]
          09/08 Guided Unit Test Generation Presenter: Lingming Zhang (Slides) Review Submission: [Link]
          09/10 Symbolic Execution Presenter: Lingming Zhang (Slides) Review Submission: [Link]
          09/15 Spec-based Testing Presenter: Lingming Zhang (Slides) Review Submission: [Link]
          09/17 Mutation-based Fuzzing Presenter: LT Review Submission: [Link]
          09/22 Grammar-based Fuzzing Presenter: NR Review Submission: [Link]
          09/24 Proposal Presentation Each team should prepare 4-5 slides to present your proposal in class for about 5min, including:
          • Problem: the problem you want to work on and the background
          • Proposed solution: how you want to solve it
          • Evaluation: how you will evaluate it (e.g., with what benchmarks/subjects and what metrics)
          • Initial results: what you did so far
          • Future plan: what you will do in the midterm and final reports
          09/28 Proposal Submission Each team should email Lingming a formal proposal, named as proposal.txt (together with your slides), including:
          • Problem: up to 80 words describing the problem you want to work on
          • Background: up to 80 words describing background or related work
          • Proposed solution: up to 80 words describing how you want to solve it
          • Evaluation: up to 80 words describing how you will evaluate it
          • Initial results: up to 80 words describing what you did so far
          • Future plan: up to 100 words describing what you will do in the midterm and final reports
          09/29 Learning-based Fuzzing Presenter: WW Review Submission: [Link]
          10/01 Differential Testing Presenter: YY Review Submission: [Link]
          10/06 Metamorphic Testing Presenter: YZ Review Submission: [Link]
          10/08 Runtime Checking Presenter: JC Review Submission: [Link]
          10/13 Human-assisted Bug Finding Presenter: XS Review Submission: [Link]
          10/15 Regression Testing Presenter: MS Review Submission: [Link]
          10/20 Failure Diagnosis Presenter: XT Review Submission: [Link]
          10/22 Fault Localization Presenter: NJ Review Submission: [Link]
          10/27 Search-based Program Repair Presenter: SC Review Submission: [Link]
          10/29 Semantics-based Program Repair Presenter: KC Review Submission: [Link]
          11/03 Faster Program Repair ( Merged into 11/05 class due to new school holiday schedule) Presenter: Lingming Zhang Review Submission: [Link]
          11/05 Unified Debugging Presenter: Lingming Zhang Review Submission: [Link]
          11/10 Project Midterm Presentation Each team should prepare 8-9 slides to present the midterm progress report in class for about 8min (plus 2min for Q&A), including:
          • Problem: the problem you are working on
          • Solution: how you are currently solving it
          • Evaluation: how you are evaluating it
          • Current results: what you did so far and whether you have met your midterm goal
          • Future plan: what you will do in the next month to wrap up the project
          11/12 Testing&Debugging: Flaky Tests Presenter: Wing Lam (UIUC) Review Submission: [Link]
          11/15 Project Midterm Report Submission Each team should email Lingming the slides and a report with 3-4 pages using standard ACM Article Template (more specifically using \documentclass[sigconf]{acmart} for Latex, or Interim-layout.docx for Word), including the following sections:
          • Abstract: a brief summary of all the following sections.
          • Problem (Section 1): the problem you are working on and also some necessary background information
          • Solution (Section 2): how you are solving it (including the link to your GitHub repo)
          • Evaluation Setup(Section 3): how you are evaluating it, including the benchmarks/subjects and metrics used, as well as the experimental process
          • Result Analysis (Section 4): your current experimental results
          • Future Plan (Section 5): what you will do in the next month to wrap up the project
          11/17 Testing&Debugging: DNN Models Presenter: YT Review Submission: [Link]
          11/19 Testing&Debugging: DNN Libs Presenter: Junjie Chen (Tianjin University) Review Submission: [Link]
          11/24 Fall Break
          11/26 Fall Break
          12/01 Testing&Debugging: FM Solvers Presenter: MZ Review Submission: [Link]
          12/03 Testing&Debugging: DB Systems Presenter: XL Review Submission: [Link]
          12/08 Project Final Presentation Each team should prepare 10-15 slides to present the final report in class for about 12min (plus 3min for Q&A), including:
          • Problem: the problem you are working on
          • Solution: how you are currently solving it
          • Evaluation: how you are evaluating it
          • Current results: what you did so far and whether you have met your final goal
          • [Optional] Future plan: some potential future work if you still have bandwidth after the course
          12/15 Project Final Report Submission Each team should extend the midterm report to prepare a final report with 5-6 pages using standard ACM Article Template, including the following sections:
          • Abstract: a brief summary of all the following sections, especially your main achievement.
          • Problem (Section 1): the problem you are working on and also some necessary background information
          • Solution (Section 2): how you are solving it (including the link to your GitHub repo)
          • Evaluation Setup(Section 3): how you are evaluating it, including the benchmarks/subjects and metrics used, as well as the experimental process
          • Result Analysis (Section 4): your final experimental results
          • [Optional] Future Plan (Section 5): some potential future work if you still have bandwidth after the course

          Resulting Publications

          The course projects of this course have resulted in the following research paper publications:

          Academic Integrity Policy

          The University of Illinois at Urbana-Champaign Student Code should also be considered as a part of this syllabus. Students should pay particular attention to Article 1, Part 4: Academic Integrity. Read the Code at the following URL: http://studentcode.illinois.edu/.

          Academic dishonesty may result in a failing grade. Every student is expected to review and abide by the Academic Integrity Policy: http://studentcode.illinois.edu/. Ignorance is not an excuse for any academic dishonesty. It is your responsibility to read this policy to avoid any misunderstanding. Do not hesitate to ask the instructor(s) if you are ever in doubt about what constitutes plagiarism, cheating, or any other breach of academic integrity.

          Students with Disabilities

          To obtain disability-related academic adjustments and/or auxiliary aids, students with disabilities must contact the course instructor and the as soon as possible. To insure that disability-related concerns are properly addressed from the beginning, students with disabilities who require assistance to participate in this class should contact Disability Resources and Educational Services (DRES) and see the instructor as soon as possible. If you need accommodations for any sort of disability, please speak to me after class, or make an appointment to see me, or see me during my office hours. DRES provides students with academic accommodations, access, and support services. To contact DRES you may visit 1207 S. Oak St., Champaign, call 333-4603 (V/TDD), or e-mail a message to disability@uiuc.edu. Please refer to http://www.disability.illinois.edu/.

          Family Educational Rights and Privacy Act (FERPA)

          Any student who has suppressed their directory information pursuant to Family Educational Rights and Privacy Act (FERPA) should self-identify to the instructor to ensure protection of the privacy of their attendance in this course. See http://registrar.illinois.edu/ferpa for more information on FERPA.

          Statement on Mental Health

          Diminished mental health, including significant stress, mood changes, excessive worry, substance/alcohol abuse, or problems with eating and/or sleeping can interfere with optimal academic performance, social development, and emotional wellbeing. The University of Illinois offers a variety of confidential services including individual and group counseling, crisis intervention, psychiatric services, and specialized screenings at no additional cost. If you or someone you know experiences any of the above mental health concerns, it is strongly encouraged to contact or visit any of the University’s resources provided below. Getting help is a smart and courageous thing to do -- for yourself and for those who care about you.

          Counseling Center: 217-333-3704, 610 East John Street Champaign, IL 61820

          McKinley Health Center:217-333-2700, 1109 South Lincoln Avenue, Urbana, Illinois 61801

          Please send feedbacks and questions to lingming@illinois.edu.