A Low-Tech Testing Dashboard
- from James Bach, Principal Consultant
通常在測試的過程中, manager常常對於QA在做什麼不是很了解, 總覺得我們好像是黑箱作業, 讓他們無法幫助或是評量我們.
尤其, 他們常常會問
* How does the build look?
* How much testing is left?
* What are the major problems?
這些問題其實還蠻難回答, 你很難以簡單, 又不會太打擾team members的方式, 讓上面的人知道你目前大概的狀況是什麼.
在Agile的世界中, 這樣的狀況又更嚴重. 因為iteration/sprint的時間不長, 因此相對的時間更寶貴, engineer更沒太多時間來update report. 加上大家對agile的誤解是不用寫documentation. 所以manager或是其他非QA的人, 更不知道我們在幹什麼.
直到我遇到James, 我才知道原來已有人, 對這些問題試著提出一些解答.
http://www.satisfice.com/presentations/dashboard.pdf
在這個dashboard中, 它企圖做到下列事情
Report test cycle progress in a simple, structured way...
- that shows progress toward a goal...
- manages expectations...
- and inspires support...
- for an effective test process.
在這份dashboard中, 它會呈現以下五種資訊: (當然我想你可以自行客制化, 重點是他的精神)
1.Product Area
- 15-30 areas (keep it simple)
- Avoid sub-areas: they’re confusing.
- Areas should have roughly equal value.
- Areas together should be inclusive of everything reasonably testable.
- “Product areas” can include tasks or risks- but put them at the end.
- Minimize overlap between areas.
- Areas must "make sense" to your clients, or they won’t use the board
2. Test Effort
(1) 分類
None: Not testing; not planning to test.
Start: No testing yet, but expect to start soon.
Low: Regression or spot testing only; maintaining coverage.
High: Focused testing effort; increasing coverage.
Pause: Temporarily ceased testing, though area is testable.
Blocked: Can’t effectively test, due to blocking problem.
Ship: Going through final tests and signoff procedure.
(2) 規則
- Use red to denote significant problems or stoppages, as in blocked, none, or pause.
- Color ship green once the final tests are complete and everything else on that row is green.
- Use neutral color (such as black or blue, but pick only one) for others, as in start, low, or high.
3. Test Coverage
(1) 分類
0: We have no good information about this area.
1: Sanity Check: major functions & simple data.
1+: More than sanity, but many functions not tested.
2: Common Cases: all functions touched; common & critical tests executed.
2+: Some data, state, or error coverage beyond level 2.
3: Corner Cases: strong data, state, error, or stress testing.
(2) 規則
- Color green if coverage level is acceptable for ship, otherwise color black.
- Level 1 and 2 focus on functional requirements and capabilities: can this product work at all?
- Level 2 may span 50%-90% code coverage.
- Level 2+ and 3 focus on information to judge performance, reliability, compatibility, and other “ilities”: will this product work under realistic usage?
- Level 3 or 3+ implies “if there were a bad bug in this area, we would probably know about it.”
4. Quality Assessment
“We know of no problems in this area that threaten to stop ship or interrupt testing, nor do we have any definite suspicions about any.”
“We know of problems that are possible showstoppers, or we suspect that there are important problems not yet discovered.”
“We know of problems in this area that definitely stop ship or interrupt testing.”
5. Comments
(1) Definition
- Use the comment field to explain anything colored red, or any non-green quality indicator.
(2) Example:
- Problem ID numbers.
- Reasons for pausing, or delayed start.
- Nature of blocking problems.
- Why area is unstaffed.
那你要如何來使用這個dashboard
- 更新的頻率: 2-5/week, or at each build, or prior to each project meeting.
- 如何進行: Set expectation about the duration of the “Testing Clock” and how new builds reset it.
- 如何解釋內容: Be ready to justify the contents of any cell in the dashboard. The authority of the board depends upon meaningful, actionable content.
- 要不要用較High Tech的方式記錄: Sure, you can put this on the web, but will anyone actually look at it???
- from James Bach, Principal Consultant
通常在測試的過程中, manager常常對於QA在做什麼不是很了解, 總覺得我們好像是黑箱作業, 讓他們無法幫助或是評量我們.
尤其, 他們常常會問
* How does the build look?
* How much testing is left?
* What are the major problems?
這些問題其實還蠻難回答, 你很難以簡單, 又不會太打擾team members的方式, 讓上面的人知道你目前大概的狀況是什麼.
在Agile的世界中, 這樣的狀況又更嚴重. 因為iteration/sprint的時間不長, 因此相對的時間更寶貴, engineer更沒太多時間來update report. 加上大家對agile的誤解是不用寫documentation. 所以manager或是其他非QA的人, 更不知道我們在幹什麼.
直到我遇到James, 我才知道原來已有人, 對這些問題試著提出一些解答.
http://www.satisfice.com/presentations/dashboard.pdf
在這個dashboard中, 它企圖做到下列事情
Report test cycle progress in a simple, structured way...
- that shows progress toward a goal...
- manages expectations...
- and inspires support...
- for an effective test process.
在這份dashboard中, 它會呈現以下五種資訊: (當然我想你可以自行客制化, 重點是他的精神)
1.Product Area
- 15-30 areas (keep it simple)
- Avoid sub-areas: they’re confusing.
- Areas should have roughly equal value.
- Areas together should be inclusive of everything reasonably testable.
- “Product areas” can include tasks or risks- but put them at the end.
- Minimize overlap between areas.
- Areas must "make sense" to your clients, or they won’t use the board
2. Test Effort
(1) 分類
None: Not testing; not planning to test.
Start: No testing yet, but expect to start soon.
Low: Regression or spot testing only; maintaining coverage.
High: Focused testing effort; increasing coverage.
Pause: Temporarily ceased testing, though area is testable.
Blocked: Can’t effectively test, due to blocking problem.
Ship: Going through final tests and signoff procedure.
(2) 規則
- Use red to denote significant problems or stoppages, as in blocked, none, or pause.
- Color ship green once the final tests are complete and everything else on that row is green.
- Use neutral color (such as black or blue, but pick only one) for others, as in start, low, or high.
3. Test Coverage
(1) 分類
0: We have no good information about this area.
1: Sanity Check: major functions & simple data.
1+: More than sanity, but many functions not tested.
2: Common Cases: all functions touched; common & critical tests executed.
2+: Some data, state, or error coverage beyond level 2.
3: Corner Cases: strong data, state, error, or stress testing.
(2) 規則
- Color green if coverage level is acceptable for ship, otherwise color black.
- Level 1 and 2 focus on functional requirements and capabilities: can this product work at all?
- Level 2 may span 50%-90% code coverage.
- Level 2+ and 3 focus on information to judge performance, reliability, compatibility, and other “ilities”: will this product work under realistic usage?
- Level 3 or 3+ implies “if there were a bad bug in this area, we would probably know about it.”
4. Quality Assessment
“We know of no problems in this area that threaten to stop ship or interrupt testing, nor do we have any definite suspicions about any.”
“We know of problems that are possible showstoppers, or we suspect that there are important problems not yet discovered.”
“We know of problems in this area that definitely stop ship or interrupt testing.”
5. Comments
(1) Definition
- Use the comment field to explain anything colored red, or any non-green quality indicator.
(2) Example:
- Problem ID numbers.
- Reasons for pausing, or delayed start.
- Nature of blocking problems.
- Why area is unstaffed.
那你要如何來使用這個dashboard
- 更新的頻率: 2-5/week, or at each build, or prior to each project meeting.
- 如何進行: Set expectation about the duration of the “Testing Clock” and how new builds reset it.
- 如何解釋內容: Be ready to justify the contents of any cell in the dashboard. The authority of the board depends upon meaningful, actionable content.
- 要不要用較High Tech的方式記錄: Sure, you can put this on the web, but will anyone actually look at it???
全站熱搜
留言列表
