David Ko的學習之旅

測試自動化的時機

Test Automation ROI (Part II)
http://blogs.msdn.com/imtesty/archive/2009/09/02/test-automation-roi-part-ii.aspx
Posted by Bj Rollison
Posted in I. M. Testy

作者這裡討論到什麼時候需要去做測試自動化. 首先他認為他自己會想去做自動化的原因如下

1. Free up my time,
2. Re-verify baseline assessments (BVT/BAT, regression, acceptance test suites)
3. Increase test coverage (via increased breadth of data or state variability),
4. Accomplish tasks that can’t easily be achieved via manual testing approaches.

接著大家可以先去參考Brian Marick所寫的"When Should a Test Be Automated"
http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=ART&ObjectId=2010
是目前是作者認為整理的很完整的一篇文章. 這裡面主要提到以下幾點:

1. How much more will this test cost to automate versus running it manually?

2. What is the potential lifetime of this automated test?

3. Does the automated test have some probability of exposing new issues?
- 作者並不太認同這一點, 因為有些自動化測試無法找到新的bug,但是仍然很有價值.
- 所以作者建議把它改成: “Does this automated test have some probability of exposing new issues, providing additional information that increases confidence, or increases test coverage?”

此外作者還會問自己以下問題, 來進一步確認是否值得做自動化
1. What exactly is being evaluated?
- If the test is evaluating functional or non-functional (stress, perf, security, etc.) capabilities then automation may be worthwhile.
- But, behavioral tests such as usability tests and content testing are generally not good candidates for automated testing.

2. What is the reliability of automating this test?
- On one wants to have to constantly massage a test in order to get it to run. 

3. What are the oracles for this test and can they be automated?
- If the most effective oracle is a human reviewing the results then that test should probably not be automated using current technologies.
- 如果你想要知道什麼是test oracle, 你可以參考這裡: http://en.wikipedia.org/wiki/Oracle_(testing)