看ASP .NET 團隊如何進行測試(1)
Evolution of the ASP.NET Test Process (part 1)
http://testertales.blogspot.com/2009/07/evolution-of-aspnet-test-process-part-1.html
Sunday, July 26, 2009
Posted by Federico Silva Armas
作者是ASP. NET QA團隊的人, 他和我們分享了他們的測試經驗
以前舊的時代 (.NET 1.0, 1.1, 2.0)
當作者加入時, .NET 1.1正要release, 而團隊正要開始進行2.0版. 那時候QA的流程是這樣的:
1. 當QA team結束對1.1版的測試, 並且簽字畫押可以發行時, 下一版本正在順利開發中. 這時候通常有一個"stabilization"的時期, QA team 正在撰寫自動化測試, 準備一堆檢查清單(我們稱做"Exit Criteria"), 以讓他們跑在各式各樣的configurations上. 也就是說, Dev/PM正忙於做下一版的prototyping和design feature, 而QA正全力於測試目前這個release
2. QA team會根據興趣和了解程度, 將features指派給團隊成員. 我們稱為"owning" a feature. 作者一開始時, 是分配到新的GridView and Login controls, spec已經大多數完成, 這些程式已經check-in可以產生build了
3. 一開始要做的事情是先撰寫測試計劃, 這意思是定義, 你總共計畫要做的測試, 及要自動化的項目. 這計劃是十分詳盡的, 所以頁數是蠻多頁的. 我們會把這些測試資料放到內部的工具去(Maddog)
4. 測試計畫會被PM, 處理這些功能的RD, 以及相關的QA所檢視, 試圖去找出遺漏的地方或是值得加以增加的部份
5. 接下來是主要測試的階段, 會分成"ad-hoc"(任意嘗試各種各樣的scenario, 以發現缺陷) 和撰寫自動化測試兩個部份
6. 我們會紀錄是否我們已經達成測試計畫中, 所要求的自動化測試個案的數目 . 例如, 測試計畫中定義200個測試個案, 我們會報告還有多少個還沒自動化, 還需要多少時間去完成
7. 有一點要注意的是, 當時的情況RD不寫任何單元測試
8. 在某些時間點上, 團隊已經開始啟動運行. 這意味著所有測試以跑在各式各樣不同的platforms和configurations, 來找出所有可能的錯誤. 這時候QA的時間可能主要在處理以下事情: (1) 撰寫測試自動化 (2) 分析每次run失敗的原因
9. 這時候我們進入我之前提過的"stabilization" time, 這時候會建製一些環境, 來跑各種SKU, languages, X32, X64, stress和performance等等
10. 在幾個月後, 我們已經完成所有測試的自動化, 並且跑完所有的測試. 任何發現的錯誤可能被修復或試之後再修, 但都已經被記錄下來. 這時候每個功能的被畫押, 產品準備出貨. (每次beta relase都會重複同樣的流程)
這個流程所產生的問題
1. 過度自動化
- 過於將重心放在自動化上, 測試人員會花費很多的時間, 在編寫和調試大量的回歸測試, 而不是找出更多產品可能被使用的scenario
- 會不遺餘力地, 去自動化你所能想到的任何一切東西
2. 改變所帶來的巨大代價
- ASP.NET is a framework, and frameworks go through many revisions and changes before they are released. However, our QA process was not designed to accommodate this. The QA methodology consistently tried to lock down details too early.
- We spent a lot of time writing fully detailed test plans based on the spec for things that were bound to change a lot, and immediately after test plan creation we launched ourselves into an automation frenzy.
- What ended up happening is that changes in the design of a feature where disastrous, the cost of updating all test plans and all the previously written automation was very high.
3. 沒有單元測試
- The QA team was responsible for testing everything, from passing every possible value to each public method of a class to end to end integration tests. This amounted to a TON of tests written by QA.
- Since the only automated mechanism to find regressions was the QA tests it was common for the builds to be broken regularly.
4. QA team 太晚找到問題
- A side effect of our automation story was that scenarios were not verified until it came time to automate the test, which usually meant months after the feature was coded.
- All this miss placed effort meant that bugs where found later rather than sooner.
5. QA team 沒有像RD/PM一樣同時開始加入專案
- In some ways this fact set us up to be the long pole of the team. There were occasions where the Dev had completed all his/her work, but the QA team still had months and months of work to go through.
- It was also common practice for the QA members to have little input on the design of the product. By the time we became fully engaged, most of the features were already designed and implemented.
留言列表
