David Ko的學習之旅

回歸測試的作法
The Minefield Myth (Part 2) – The value of regression testing
http://blogs.msdn.com/imtesty/archive/2009/01/30/the-minefield-myth-part-2-the-value-of-regression-testing.aspx

Pesticide Paradox (殺蟲劑詭論)


第一次看到Pesticide Paradox這個term, 覺得很陌生, 總覺得怎麼可能他和software testing有關, 但是我卻從來沒聽過. 後來上網查了一下, 才知道它出自於Boris Beizer的大作. 在1990, Boris Beizer在Software Testing Techniques, Second Edition一書中, 第1.7小節中提到

1.7 Pesticide Paradox and the Complexity Barrier

.......
First Law: The Pesticide Paradox - Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.

你Run你的測試越多次, 受測軟體就越有免疫力. 也就是說相同的測試個案在重複執行後, 能夠再找到bug的機率會越來越低

因為為了克服Pesticide Paradox的現象, QA 必須要不斷開立新的測試個案, 來驗證受測軟體, 以期待找出不同的bug.

同樣地, 在test automation上, 也是會有同樣的現象. 就像James Bach說的
Highly repeatable testing can actually minimize the chance of discovering all the important problems, for the same reason that stepping in someone else’s footprints minimizes the chance of being blown up by a land mine.

所以不是你把所有測試個案自動化, 世界就太平了, 最多只是這些個案的形很快, 不需要人介入, 但是也是非常的一成不變. 這是另一個test automation的迷思, 也是upper manager最誤會的地方.


Reference
1. Pesticide Paradox
http://blogs.msdn.com/nihitk/articles/185836.aspx

2. Software Testing Techniques, Second Edition, Boris Beizer
這本書應該是絕版了, 目前在Amazon好像是買不到, 我有的也是翻印的版本. 它是所有white box testing的bible. 幾乎沒有書寫的這麼詳盡, 不過也相對的很難看, 實在是太硬了. 慶幸的是我在研究所看過後, 確實幫我打下很好的基礎.

How to conduct regression tests
- by Mike Kelly
http://www.michaeldkelly.com/blog/archives/201

Regression testing strategy  - test suite maintenance
I also found some articles that they suggests we can review our test cases frequently. Then the number or correctness of test cases can controlled. It's also useful for regression test because about 30% useless test cases can be removed.

B. Retest Changed Segments
(1) Intent
    Use code change analysis to select a partial regression test suite.
(2) Context
    The available time, personnel or equipment is not sufficient for a full regression run.
    It is applicable at the class, cluster, or subsystem scope and after maintenance, for reuse, and during iterative development.
(3) Strategy
    a. Test Model
    A regression fault must be related to a new, modified, or deleted code segment.
    All tests that have reached a changed or deleted segment are selected
    Rothermel and Harrold call this approach the “graph walk technique”
    The basic code change model does not consider data flow, control flow or other dependencies arising from state-based behavior, iteration, or recursion.
    It does not explicitly consider the effects of inheritance or dynamic binding
     b. Test Procedure
     Step1 : Obtain a report from your coverage analyzer that lists code segments by test case for the baseline SUT and test suite
         Coverage Analyzer Output
         T1: B1, B3, B6, B7
         T2: B1, B4, B8
         T3: B2, B5
         T4: B1, B4, B6
     Step2: Use a version control tool to generate a report on the
changes between the baseline and the delta
          Version Control Report
          Baseline      Delta   Change
          B1            B1      same
          B2                       deleted
          B3            B3      changed
          B4            B4      same
          B5            B5      changed
          B6            B6      same
          B7                       deleted
          B8            B8      changed
          B9                       new
 
     Step 3: Use the following rules to decide the action for test cases
          Tests under same a skip
          Tests under deleted a include
          Tests under changed a include
          Tests under new a should be empty
          ==> T1, T2, and T3 need to be rerun again
(4) Entry Criteria
     The delta components pass component scope testing
     A suitable baseline test suite exists
     The test environment has been restored to the same configuration used to run the original baseline test suite
(5) Exit Criteria
     All no pass test cases reveal bugs whose presence and severity are deemed acceptable
     All remaining test cases pass
(6) Consequences
     a. Inclusiveness
     All baseline tests that can produce a different result are selected.
     b. Precision
     Retest changed code is the most precise of the white box partial regression strategies
     Baseline tests that exercise unchanged code are not selected for testing
     c. Efficiency
     The computation is on the order of the size of the baseline or test suite times the number of segments in the smaller of the baseline or delta component
     d. Generality
     Retest changed code is applicable at class scope and requires the programmer to have skills in code analysis and related tools

Regression Testing Strategy (1)
Regression Testing是測試人員心中最大的痛, 因為這時間通常要花很多(因為test case實在太多了), 而且次數也不會只有一次. 因此好的regression testing strategy是非常重要的 
我從"Testing Object-Oriented System" 中survey一些方法, 希望能對大家有幫助
A. Retest Risky Use Cases
(1) Intent
Use risk-based heuristics to select regression test suite. 
(2) Context
The available time, personnel or equipment is not sufficient for a full regression run
This pattern is applicable at any scope. It may be used to support iterative velopment, the instantiation of reusable components and maintenance 
(3) Strategy
a. Test Model: 
The goal is to skip enough non-critical, low priority, or highly stable use cases to meet or budget constraint. 
     a-1 Risk criteria
            - Suspicious use cases 
                      Are individually unstable or unproven 
                      Have not been shown to work together before or are unstable 
                      Implement complex business rules 
                      Have a complex implementation 
                      Were subject to high churn during development
           - Critical use cases 
                      Select tests for use cases that are necessary for safe, effective operation. 
                      For each use cases, try to identify the worst-case effects of failure