Connor's Notes

⬅️ Back to portfolio

COSC 3P95: Week 3 Notes

6 min read

Software testing

  • Software engineering: very human-intensive process (human skills, judgment, and collaborations)
    • less now, due to ai coding tools
  • Software testing: the process used ot identify the completeness, correctness, and quality of developed software
    • the process of executing a program under positive and negative conditions by manual or automated means to check against:
      • Specification: does the system have all the features advertised/specified?
      • Functionality: do all those features actually work when you test them?
      • Performance: how well does it run under different conditions

Human-intensive process

flowchart TD
  programmer[Programmer]
  specs((Specifications))
  error((Error in thought or action))
  fault((Fault))
  test[Test data]
  program[Program]
  observable((Observable behavior))
  observed((Observed behavior))
  desired((Desired behavior))
  compare{Are these the same?}
  yes[Yes. Program behaves as desired.]
  no[No. Program does not behave as desired.\nA failure has occurred.]

  specs -->|"are used by"| programmer
  programmer -->|writes| program
  programmer -->|"possibility of"| error
  error -->|"may lead to"| fault
  program -->|"might contain"| fault

  test -->|"is input to"| program
  program -->|"produces"| observable
  observable -->|"leads to"| observed

  specs -->|"determines"| desired
  observed --> compare
  desired --> compare

  compare -->|Yes| yes
  compare -->|No| no

Goals in software testing

  1. demonstrate a given software product is matching its requirements and specifications
    • make sure the software product’s quality is very high
  2. uncover as many bugs as possible

What is a “good” test?

A “good” software test is a test that is effective, efficient, and provides maximum value with minimum effort.

  • a good test has a high probability of finding an error
  • a good test is not redundant
  • a good test should be “best of breed”
  • a good test should be neither too simple or too complex

Black-box testing

  • based on requirements and functionality, not code
    • written without knowledge of how the class being tested is implemented
  • tester may have actually seen the code before (“grey box”)
    • but does not look at it while constructing tests
  • often done from end users perspective
  • emphasis on parameters, input/output (and their validity)

Types of black-box testing

  • requirements based
    • pick one from each category
  • equivalence partitioning
    1. identify classes of inputs with same behaviour
    2. test at least one member
    3. assume the rest of the members will be the same
  • boundary value analysis
    • testing conditions near the bounds of inputs
    • likely source of programmer error (< vs <=, etc.)
  • positive/negative
    • checks both good/bad results
  • decision tables
  • state-based
    • based on object state diagrams
  • compatibility testing
    • run the software in different environments
    • e.g. open a website on different browsers/devices
      • firefox, mobile, etc.

White-box testing

  • use knowledge of the code to write the tests
  • e.g. if there is an if statement:
    • write test(s) based on if it is true
    • write test(s) based on if it is false

Types of white-box testing

  • statement testing
    • test single statements
  • branch testing
    • make sure each possible outcome from a condition is tested at least once
    • e.g. print("yes" if condition else "false")
      • test when condition == true and condition == false
  • loop testing
    • check for infinite loops, performance, loop initialization, uninitialized variables
  • path testing
    • make sure all paths are executed
  • exhaustive testing
    • test every possible path
  • selective testing
    • choose & test specific paths (the most likely for a user to encounter)

Example

void FindMean(FILE *ScoreFile)
{
    float SumOfScores = 0.0;
    int NumberOfScores = 0;
    float Mean = 0.0;
    float Score;
 
    Read(ScoreFile, Score);
 
    while (!EOF(ScoreFile)) {
        if (Score > 0.0) {
            SumOfScores = SumOfScores + Score;
            NumberOfScores++;
        }
        Read(ScoreFile, Score);
    }
 
    /* Compute the mean and print the result */
    if (NumberOfScores > 0) {
        Mean = SumOfScores / NumberOfScores;
        printf("The mean score is %f\n", Mean);
    } else {
        printf("No scores found in file\n");
    }
}

Find the paths:

  1. float SumOfScores = 0.0; int NumberOfScores = 0; float Mean = 0.0; float Score; Read(ScoreFile, Score);
  2. while (!EOF(ScoreFile)) {
  3. if (Score > 0.0) {
  4. SumOfScores = SumOfScores + Score; NumberOfScores++;
  5. }
  6. Read(ScoreFile, Score);
  7. if (NumberOfScores > 0) {
  8. Mean = SumOfScores / NumberOfScores; printf("The mean score is %f\n", Mean);
  9. printf("No scores found in file\n");

Designing the tests:

flowchart TD
    Start([Start]) --> step1([1: Initialize variables])
    step1 --> step2{"2: Covered by any data?"}
    step2 -- T --> step3{"3: Score > 0.0 ?"}
    step2 -- F --> step1
    step3 -- T --> step4([4: Add score to SumOfScores, increment count])
    step3 -- F --> step5([5: Ignore score])
    step4 --> step6([6: Read next score])
    step5 --> step6
    step6 --> step7{"7: End of file reached?"}
    step7 -- T --> step8([8: Compute mean and print result])
    step7 -- F --> step9([9: No scores found in file])
    step8 --> Exit([Exit])
    step9 --> Exit

Code coverage

  • metric which essentially means how much of the code is tested by a given test suite
    • what fraction of the code being tested is being reached by tests
  • Function coverage: which functions were called?
  • Statement coverage: which statements are covered?
  • Branch coverage: which branches were taken?
  • also: line coverage, condition coverage, basic block coverage, path coverage, etc.

Quiz: Code Coverage Metrics

// code being tested:
int foo(int x, int y) {
    int z = 0;
    if (x <= y) {
        z = x;
    } else {
        z = y;
    }
    return z;
}

Test suite: { foo(1,0) } Find statement coverage, branch coverage, and arguments for another call that increases both coverages to 100%

Mutation testing

a method of software testing where small, intentional changes (“mutations”) are made to the source code of a program, and the test suite is run to see if any of the tests detect the change

  • also called “fault-based testing” as it involves creating a fault in the program and it is a type of White-box testing

Graph View