David Ko的學習之旅

Bug Number 和Test Automation Ratio

Motley says: "More test automation is always better"
http://blogs.msdn.com/progressive_development/archive/2008/08/19/motley-says-more-test-automation-is-always-better.aspx

最近逛到這個blog, 發現他的文章十分好玩, 會用一些對話來說明他要闡述的道理, 以講故事的方式來進行, 讓人看起來很有趣

在這篇文章中, 他提到兩個重點

1. 用Bug個數來評量QA的產量好不好是不正確的
(1) QA要做的是quality assurance而不是quality control, 也就是說 prevent a bug before it happens
(2) 要專案一開始就要要求QA一起加入, 在很多方面幫助整個product的品質更好. 例如:
- functional specification reviews
- contribute to user scenario development
- design reviews
- code reviews
- do private buddy testing,
- help compute metrics like code coverage
(3) 所以一個好的QA是什麼呢?作者給了以下的解釋
A good tester helps assure quality by finding issues in all aspects of the development process and helps to improve the team's processes to prevent bugs.

2. 不要太多不必要的test automation
(1) 作者認為unit testing是必要的, 而且能有更高的code coverage更好, 希望每個RD都要去落實它.
(2) 認為scenario-based的test automation不需要太多, 因為會有下列問題
a. 大部分的測試程式架構不太強健
- If the architecture of the test infrastructure is not robust, every time the product changes, major changes in the automated tests may be required.
- Testers spend more time trying to get the automation to work than adding real value to the product.

b. 大部分的受測程式的testability不高
- If the developers design for testability, as they should, this situation is less of a problem as the testers can start developing automation fairly quickly.
- If automation is an afterthought, the developers may be on to the next feature before the test team is done, which means the feature isn't "done" according to our definition.
- It is also a context switch to go back and diagnose and debug issues later as the code is not as fresh.

c. 有些功能可能下一版會修改不少
- If the infrastructure on which the current features are built is expected to change in the next release, often a rewrite of the test automation is required. 
- Should the team spend a large amount of time building automation infrastructure if it will change in the near future? Tough call.

d. 測試程式畢竟沒有人腦聰明
- An automated test cannot think like a user, even if you inject some randomness into the tests.
- Over-reliance on automation creates some nasty test holes in the product that may only get discovered after shipping.

所以作者的重點是, 不要化太多時間在維護測試程式或是對它除錯. 可能的話, 多花時間和RD討論如何進行測試, 和盡可能以user角度來評估受測系統