Connor's Notes

⬅️ Back to portfolio

COSC 3P95: Lecture 10 Notes

4 min read

Announcements

Midterm

  • up to chapter 4 (delta debugging)
  • mostly multiple choice

Assignment

  • “due Sunday” but actually due Monday (Oct 13)

Delta Debugging

Note

starting from slide 51 (recap from Tuesday), i’ll only take notes on new stuff

ALG3: third algorithm

  • slight change from the minimization algorithm
  1. first minimize the right half as a complement to the left (find smallest subset of the right half that, together with the left half, still fails):
  1. minimize the left half using the found complement as the new context (find the smallest subset of the left half that, together with M2, still fails)
  1. Return the union:

Example 1

  • consider an input of 8 numbers (1 to 8), of which 3, 5, and 7 together cause the failure. minimize the input using the third algorithm
Configuration (extra space)ResultNote
1 2 3 4 5 6 7 8❌
1 2 3 4 ✅
5 6 7 8✅ - interferenceWe need a combination of 1 2 3 4 and 5 6 7 8
1 2 3 4 5 6 ✅IDEA: keep all of 1 2 3 4 and search for the minimal subset of 5 6 7 8 that, in combination with 1 2 3 4, gives failure
1 2 3 4 7 8✅ - interferenceWe need combination of 5 6 and 7 8
1 2 3 4 5 6 7 ❌IDEA: we keep all of 1 2 3 4 5 6 and search for the minimal subset of 7 and 8 that, in combination with 1 2 3 4 5 6 gives failure
1 2 3 4 5 7 ❌
1 2 5 7 ✅
3 4 5 7 ❌
3 5 7 ❌

Example 2

  • consider crash due to <select> in input <select size=5>, with each character being an input element
ConfigurationResult
<select size=5>❌
<select ✅
size=5>✅ - interference
<select siz ✅
<select e=5>❌
<select e= ✅
<select 5>❌
<select 5 ✅
<select >❌
<sel >✅
ect >✅ - interference
<selec >✅
<sel t >✅ - interference
<select >❌
<sele t >✅
<sel ct >✅ - interference
<select >❌

Quiz: Delta Debugging

Check the statements that are true about delta debugging:

  • is fully automatic
  • finds 1-minimal instead of local minimum test case due to performance
  • finds the smallest failing subset of a failing input in polynomial time
  • may find a different sized subset of a failing input depending upon the order in which it tests different input partitions
  • is also effective at reducing non-deterministically failing inputs

Note

end of chapter 4, start of chapter 5 (instrumentation)

Software Measurements & Instrumentation & Telemetry

Why measure software?

  • decisions need evidence, not intuition
  • shows how a system behaves and improves
  • “you cannot improve what you cannot measure”

Measurement categories

  • Product measurements: properties of the code or system
  • Process measurements: how development activities are performed
  • Project measurements: cost, schedule, resource metrics

Common product metrics

  • code coverage
  • lines of code (LOC)
  • cyclomatic complexity
    • logical complexity of code structure
  • defect density
    • number of defects per size unit
  • response time, memory usage, throughput

Common process and project metrics

  • effort (person hours)
  • schedule variance
  • bug detection rate
  • test efficiency
  • productivity (output per unit of effort)

From measurement to data collection

  • we know what to measure, but how do we get the data
  • manual observation is slow and unreliable
  • needs automated, repeatable data collection
  • leads to the idea of instrumentation

Example

i did user instrumentation at Shopify (track what users do → leads to purchase or not?). not exactly used for debugging, but similar idea

Software instrumentation

  • Data collection: the process of preparing and collecting data
  • Software instrumentation: the primary method for collecting data in software systems
  • data collected while instrumenting a system can be used to analyze faults, performance issues, understand system behaviour, etc.

Note

ended slide 10

Graph View