COSC 3P95: Lecture 9 Notes

Note

Starting in delta debugging, slide 43

Delta Debugging

Minimizing Test Cases

• Global Minimum: the smallest test case that makes a test fail
• Local Minimum: the smallest test case with respect to itself that causes a test to fail
  • removing one or more items from the failing test case would not create any other smaller test case
• n-minimal: if making between 1 and n changes from $c$ causes the test to no longer fail
• 1-minimal: if making any single change from c causes the test to no longer fail
• 2-minimal: if making any 2 changes from c causes the test to no longer fail
• k-minimal: if making any k changes from c causes the test to no longer fail
  • different than n-minimal, since n-minimal is any 1 to n, and this is exactly k changes

Examples

For the following test cases, the root cause for a failing test case is if it contains 3,4 or 8

$$T{C}_1=\{2,1,3,4\}$$

Is this a failing test case? Is it the global minimum? Is it a local minimum?

Solution

Yes it is failing, because it contains $\{3,4\}$. It is not the global minimum, because there is a smaller test case that exists that fails. It is also not a local minimum for the same reason.

$$T{C}_2=\{3,4\}$$

Is this a failing test case? Is it the global minimum? Is it a local minimum?

Solution

Yes it is failing, because it contains $\{3,4\}$. It is not the global minimum, because there is still a smaller test case that causes the test case to fail ($\{8\}$). However, it is a local minimum because removing either $\varnothing$, $3$, or $4$ does not cause the test. to fail.

$$T{C}_3=\{8\}$$

Is this a failing test case? Is it the global minimum? Is it a local minimum?

Solution

Yes it is failing, because it contains $\{8\}$. It is the global minimum, because it is the smallest possible test case that causes the test to fail. It is also a local minimum because if you remove $\varnothing$ or 8, the test will not fail.

$$T{C}_4=\{1\}$$

Is this a failing test case? Is it the global minimum? Is it a local minimum?

Solution

No, this is not a failing test case → not a global or local minimum

Quiz

A program takes a string of a’s and b’s as input. It crashes on input with an odd number of b’s and an even number of a’s. Write a crashing test case (or NONE if none exists) that is a sub-sequence of input babab and is:

• smallest
• 1-minimal, of size 3
• 2-minimal, of size 3

Remember, a failing test case is 1-minimal if, no matter what change we make, we get a passing test case. 2-minimal is the same but no matter what 2 changes we make, we get a passing test case

Solution

Smallest: b
1-minimal, of size 3: aab, aba, baa, bbb
2-minimal, of size 3: NONE

$$\{b,b,b\}$$

Is this 1-minimal? 2-minimal? 3-minimal? Why?

Solution

1-minimal: Yes, $\{b,b\}$ is a passing test case
2-minimal: No, $\{b\}$ is a failing test case
3-minimal: Yes (unsure, board says No, but makes no sense)

Naive algorithm to find 1-minimal

• iterate through each change ($\delta i$) in c, testing whether set c minus delta i fails or not
  • if every change’s removal causes. the test to stop failing, then c is 1-minimal → return c
  • if we find a change delta such that $c-\{\delta \}$ still induces failure, then we call the algorithm recursively on $c‘=c-delta$
• if we start with N elements, then perform up to $N-i$ tests on the ith iteration → this is $O(N^2)$

Improved version

• divide the set of changes (c) into 2 initially
• increase the number of subsets if we can’t make progress
• if we get lucky, search will converge quickly

Minimization algorithm

• The delta debugging algorithm searches for a **$1$-minimal test case.
• It partitions the failing configuration $c_F$ into disjoint subsets:

$$c_F={\Delta }_1\cup {\Delta }_2\cup \cdots \cup {\Delta }_n$$

where ${\Delta }_1,{\Delta }_2,\dots ,{\Delta }_n$ are pairwise disjoint.

• the complement of a partition is defined as:

$${\nabla }_i=c_F-{\Delta }_i$$

Steps

1. start with $n=2$
2. test each configuration defined by every partition ${\Delta }_i$ and its complement ${\nabla }_i$
3. if any of these test cases fails, reduce $c_F$ to that failing subset
4. otherwise, refine the partition:

$$n=n\times 2$$

Quiz

A program crashes when its input contains 42. Fill in the data in each iteration of the minimization algorithm assuming character granularity (and ignore duplicate strings).

Fill in the table:

Iteration	$n$	$\Delta$	${\Delta }_1,{\Delta }_2,...,{\Delta }_n,{\nabla }_1,{\nabla }_2,...,{\nabla }_n$
1	2	2424	24
2
3
4

Solution

Iteration	$n$	$\Delta$	${\Delta }_1,{\Delta }_2,...,{\Delta }_n,{\nabla }_1,{\nabla }_2,...,{\nabla }_n$
1	2	2424	24
2	4	2424	2, 4, 242, 224, 424, 224
3	3	242	2, 4, 24, 42, 22
4	2	42	4, 2

Note

Ended at slide 60