Microsoft如何測試ASP.NET 2.0 and Visual Web Developer
Testing ASP.NET 2.0 and Visual Web Developer
http://weblogs.asp.net/scottgu/archive/2004/10/28/249458.aspx
很多應該都有興趣Microsoft如何測試他們的產品, 尤其是他們的主流產品. 可惜的是, 市面上不太容易找到這樣的資料, 即使有也都談的很淺.
這篇(2004的文章)是我很久以前找到, Microsoft如何測試ASP.NET 2.0 and Visual Web Developer, 我想很值得大家參考. 作者是ASP .NET的manager, 我想大方向上, 應該是可靠的資訊.
1. Team的組織架構
(1) Test的成員
- Our test team is staffed by engineers who own writing test plans, developing automated tests, and building the test infrastructure required to run and analyze them.
- The job title we use to describe this role at Microsoft is SDE/T (Software Design Engineer in Test).
(2) 其他成員的組成
- All members of the test team report through a Test Manager (TM)
- Similarly all members of the development team and program management team report through a Development Manager (DM) and Group Program Manager (GPM) respectively.
- The TM, DM and GPM are peers who report to a Product Unit Manager (PUM) who runs the overall product team (note: 作者就是PUM).
(3) RD和QA的比例
- We currently have approximately 1.4 testers for every 1 developer.
2. 為什麼test team要比development team大?
(1) We take quality pretty seriously at Microsoft – hence the reason we invest the time and resources.
(2) We also have a lot of very hard requirements that necessitate a heck of a lot of careful planning and work to ensure high quality.
- different processor architectures (x86, IA-64, and x64 processor architectures),
- on 4 different major OS variations (Windows 2000, Windows XP, Windows 2003 and Longhorn)
- support design-time scenarios with 7 different Visual Studio SKUs, and be localized into 34+ languages
(3) Long maintenance period
- Making things even more challenging is the fact that Microsoft supports all software for at least 10 years after the date of its release
> which means that customers at any point during that timeframe can report a problem and request a QFE fix.
- We’ll also then do periodic service packs (SPs) rolling up these fixes during these 10 years as well.
3. What is our process for testing?
(1) 基本上分成三個重要的階段
a. We build detailed test plans that comprehensively cover all product scenarios
b. We automate the test scenarios in the test plans to eliminate the need for manual steps to test or verify functionality
c. We build and maintain infrastructure that enables us to rapidly run, analyze and report the status of these automated tests
(2) Test Plans
a. Test Plan 的內容
- Test plans are the first step, and happen as early as possible in the product cycle.
- A separate test plan will be written by a tester for each feature or feature area of the product.
- The goal with them is to comprehensively detail all of the scenarios needed to test a given feature.
- The test plan will group each of these scenarios into a test case
> where 1 test case might have up to 10 or more separately verified scenarios,
> assign a priority (P1, P2, or P3) to each test case.
b. Feature Team中對quality的運作
- The entire feature team (pm, dev, and test) will get together during a coding milestone to review the test plan and try to ensure that no scenarios are missing.
- The team will then use the test plan as the blueprint when they go to write and automate tests, and they will implement the test scenarios in the priority order defined by the plan.
c. 專案運作過程如何改進Test Plan
- During the product cycle we’ll often find new scenarios not covered by the original test plan.
- We call these missing scenarios “test holes”, and when found they’ll be added to the test plan and be automated.
- Every new bug opened during the product cycle will also be analyzed by test to ensure that it would be found by the test plan
> if not, a new test case is added to cover it.
d. Test Plan 的Sample
- Here is a pointer to a few pages from the test plan of our new GridView data control in ASP.NET 2.0: http://www.scottgu.com/blogposts/testingatmicrosoft/testplan/testplan.htm
- The full test plan for this feature is 300+ pages and involves thousands of total scenarios
- Note that some of the test cases have a number associated with them
- Look at the first AutoFormat one
> it indicates that this test case was missed during the original review of the document (meaning a test hole) and
> it has been added in response to bugs being opened (110263 is the bug number).
(3) Test Automation
a. Test automation的時機
- After testers finalize their test plans, they will start writing and automating the tests defined within them.
b. 使用的languages和tools
- We use a variety of languages to test the product, and like to have a mixture of C#, VB and some J# so as to exercise the different compilers in addition to our own product.
- Tests on my team are written using a testing framework that we’ve built internally.
- Long term we’ll use vanilla VSTS (Visual Studio Team System) infrastructure more and more, but given that they are still under active development we aren’t using it for our Whidbey release.
The teams actually building the VSTS technology, though, are themselves “dogfooding” their own work and use it for their source control and testing infrastructure (and it is definitely being designed to handle internal Microsoft team scenarios).
c. Test Case的內容和 Sample
- The test cases themselves are often relatively straight forward and not too code-heavy.
- Instead, the bulk of the work goes into the shared test libraries that are shared across test scenarios and test cases.
- Here is a pointer to an example test case written for our new WebPart personalization framework in ASP.NET 2.0: http://www.scottgu.com/blogposts/testingatmicrosoft/testcase/testcase.htm
- Note how the test case contains a number of distinct scenarios within it – each of which is verified along the way.
- This test case and the scenarios contained within it will match the test plan exactly.
- Each scenario is then using a common WebPart automation test library built by the SDE/T that enables heavy re-use of code across test cases.
d. Test Cases 和 Functional Test Scnearios的數量
- My team will have ~105,000 test cases and ~505,000 functional test scenarios covered when we ship Whidbey.
e. Coverage Testing的pass criteria, execution time, tool
- Our hope/expectation is that these will yield us ~80-90% managed code block coverage of the products when we ship.
- We use this code coverage number as a rough metric to track how well we are covering test scenarios with our functional tests.
- We also then measure “arc” coverage, which includes measuring further individual code paths within a block
- We measure both block and arc numbers regularly along the way when we do full test passes (like we are doing this week) to check whether we are on target or not.
- One really cool thing about VS 2005 is that VSTS includes support to automatically calculate code coverage for you
f. 如何提升coverage ratio
- There is always a percentage of code that cannot be easily exercised using functional tests
> common examples: catastrophic situations involving a process running out of memory, difficult to reproduce threading scenarios, etc
- We exercise these conditions using our stress lab
> where we’ll run stress tests for days/weeks on end and put a variety of weird load and usage scenarios on the servers
> for example: we have some tests that deliberately leak memory, some that AV every once in awhile, some that continually modify .config files to cause app-domain restarts under heavy load, etc
- Stress is a whole additional blog topic that I’ll try and cover at some point in the future to give it full justice. - Going forward, my team is also moving to a model where we’ll also add more fault-injection specific tests to our functional test suites to try and get coverage of these scenarios through functional runs as well.
4. Running Tests
(1) 測試管理系統: Maddog
- My team uses an internally built system we affectionately call “Maddog” to handle managing and running our tests.
- Post Whidbey my team will be looking to transition to a VSTS one, but for right now Maddog is the one we use.
- Maddog does a couple of things for my team, including:
> managing test plans
> managing test cases
> providing a build system to build
> deploy all test suites we want to execute during a given test run
> providing infrastructure to image servers to run and execute our tests
> ultimately providing a reporting system so that we can analyze failures and track the results.
(2) Testing Lab
- My team currently has 4 labs where we keep approximately 1,200 machines that Maddog helps coordinate and keep busy.
- The machines vary in size and quality – with some being custom-built towers and others being rack-mounts.
(3) How Maddog handles test execution
- We use Maddog to help coordinate and use all these machines.
- A tester can use Maddog within their office to build a query of tests to run
> selecting either a sub-node of feature areas – or doing a search for tests based on some other criteria
> then pick what hardware and OS version the tests should run on, pick what language they should be run under (Arabic, German, Japanese, etc),
> pick what ASP.NET and Visual Studio build should be installed on the machine,
> pick how many machines it should be distributed over.
- Maddog will then identify free machines in the lab
> automatically format and re-image them with the appropriate operating system
> install the right build on them
> build and deploy the tests selected onto them, and then run the tests.
- After everything is selected above, the tester can hit “go” and launch the test run. Anywhere from 30 minutes to 14 hours later it will be done and ready to be analyzed.
5. What tests are run when?
(1) Functional Test執行時機
- We run functional tests on an almost daily basis.
- We do a functional run on our shipping products every time we release a patch or QFE(Quick Fix Engineering, this is the Microsoft and Intel term for a hotfix).
- We also do a functional run anytime a big software component in Microsoft releases a GDR(General Distribution Release) (for example: a security patch to Windows).
(2) 何時Run Subset Test Cases
- With ASP.NET 2.0 and Visual Web Developer we’ll usually try and run a subset of our tests 2-3 times a week.
- This subset contains all of our P0 test cases and provides broad breadth coverage of the product (about 12% of our total test cases).
(3) 何時Run Full Test Cases
- We’ll then try and complete a full automation run every 2-3 weeks that includes all PO, P1, P2, P3 test cases.
- As we get closer to big milestone or product events(like a ZBB, Beta or RTM), we’ll do a full test pass where we’ll run everything
> including manually running those tests that aren’t automated yet
