成功和失敗的Test Automation的差別
The Difference Between Test Automation Success and Failure
Elisabeth Hendrickson
Quality Tree Software, Inc.
esh@qualitytree.com
這篇文章作者在討論, 成功和失敗的Test automation差在哪裡. 首先他先定義何謂失敗, 何謂成功.
什麼是失敗的Test Automation
‧ Wasted Time
‧ Wasted Money
‧ Inaccurate Results
‧ Demoralized Team
‧ Overall Reduced Productivity
‧ Lost Opportunity
什麼是成功的Test Automation
‧ Overall Cost Savings
‧ Improved Testing
‧ Shortened Software Development Cycle
‧ Reliable Results
‧ Process in Place for Future Success
接下來從Project的 level來討論, 她所經歷過的成功和失敗的project有怎樣的差別
(1). 領導階層的特徵
Failed Project
‧ Executives expected immediate payback.
‧ QA Manager had unrealistic expectations.
‧ Automation lead (me) inexperienced in leadership and automation.
Successful Project
‧ Different executives were open to having their expectations reset.
‧ Different QA manager with more automation experience.
‧ Automation lead (me) got a clue.
(2) Project對Test Automation的目標
Failed Project
‧ Stated goal: “Automate Everything”
‧ Unstated goal: “Reduce number of testers needed.”
‧ Goals not measurable.
Successful Project
‧ Stated goal: “Save manual testers time and improve testing coverage.”
‧ Unstated goal: “Reduce test cycle time.”
‧ Goals specifically designed to be measurable.
(3) 溝通狀況
Failed Project
‧ No consistent communication about project goals and status.
‧ Inadequate communication with executives.
‧ Inadequate communication with manual testers.
Successful Project
‧ Same detailed weekly status report sent to all. Status information available online at all times.
‧ Close communication with the VP of Development and Director of QA
‧ Verbal status reports delivered in weekly QA meeting.
(4) 自動化的準備程度
Failed Project
‧ Extremely limited test documentation; most testing ad hoc.
‧ No method of tracking test results.
‧ Testers lacked a strong understanding of how to test the product.
Successful Project
‧ Written test documentation. Each test case numbered individually.
‧ Test results tracked on spreadsheets that referenced the test case number.
‧ The test group as a whole had a much better understanding of how to test the product.
(5) Automated Testing Team的心態
Failed Project
‧ “Bulldozer builders vs. ditch diggers”
‧ Automators didn’t appreciate the value of manual testing.
‧ Manual testers felt threatened.
Successful Project
‧ Service organization focused on building tools.
‧ Automators understood that automation cannot replace manual testing.
‧ Manual testers more involved in the process and therefore less threatened.
從技術的觀點, 她所經歷過的成功和失敗的project有怎樣的差別
(1) Automated Test System 的架構
Failed Project
‧ Tool best-suited to creating individual scripts, not entire systems.
‧ Tool did not support creating a reusable library of functions
‧ No support for logical layer resulted in maintenance nightmare.
Successful Project
‧ Tool specifically designed to support creation of automation systems.
‧ Tool supported & encouraged creating a reusable library (“infrastructure”).
‧ Logical layer vastly improved portability & maintainability of scripts.
(2) Script Creation 的方法
Failed Project
‧ Primarily record & playback
‧ No automatic test case or test data generation
Successful Project
‧ Primarily data driven; record & playback used as a learning tool only.
‧ Used advanced features in the automation tool to support automated test data generation.
Note:
Good Script 設計要素
‧ Tests structured with setup, action, and result
‧ Tests are not order-dependent
‧ Test data is never hard coded
‧ Results are informative
‧ Pass/Fail determination is as automated as is practical
(3) Verification的方法
Failed Project
‧ Bitmap comparisons to verify both window existence & contents of window.
Successful Project
‧ Logical window existence functions to verify window appeared.
‧ Logical comparison between expected fields in window and actual fields.
‧ Test data verification
(4) Automation Programming Practices
Failed Project
‧ Automation standards focused on file naming conventions.
‧ Extremely limited code reviews.
‧ No source control; test script management done exclusively through tool’s management module.
Successful Project
‧ Automation standards focused on what constitutes good code
‧ Both formal & informal code reviews on a regular basis.
‧ Commercial source control system used.
作者在 Automation Management所學到的事情
‧ Set realistic goals.
‧ Measure your progress toward those goals.
‧ Communicate goals and status clearly and consistently.
‧ Don’t let your management set their expectations based on vendor hype.
‧ Coordinate with manual testers.
作者在 Automation Creation所學到的事情
‧ The right architecture can make everything else fall into place.
‧ Having the right tool for the job makes a difference.
‧ Simple scripts can be more powerful than complex do-everything scripts.
‧ Automation is programming: good programming practices apply.
The Difference Between Test Automation Success and Failure
Elisabeth Hendrickson
Quality Tree Software, Inc.
esh@qualitytree.com
這篇文章作者在討論, 成功和失敗的Test automation差在哪裡. 首先他先定義何謂失敗, 何謂成功.
什麼是失敗的Test Automation
‧ Wasted Time
‧ Wasted Money
‧ Inaccurate Results
‧ Demoralized Team
‧ Overall Reduced Productivity
‧ Lost Opportunity
什麼是成功的Test Automation
‧ Overall Cost Savings
‧ Improved Testing
‧ Shortened Software Development Cycle
‧ Reliable Results
‧ Process in Place for Future Success
接下來從Project的 level來討論, 她所經歷過的成功和失敗的project有怎樣的差別
(1). 領導階層的特徵
Failed Project
‧ Executives expected immediate payback.
‧ QA Manager had unrealistic expectations.
‧ Automation lead (me) inexperienced in leadership and automation.
Successful Project
‧ Different executives were open to having their expectations reset.
‧ Different QA manager with more automation experience.
‧ Automation lead (me) got a clue.
(2) Project對Test Automation的目標
Failed Project
‧ Stated goal: “Automate Everything”
‧ Unstated goal: “Reduce number of testers needed.”
‧ Goals not measurable.
Successful Project
‧ Stated goal: “Save manual testers time and improve testing coverage.”
‧ Unstated goal: “Reduce test cycle time.”
‧ Goals specifically designed to be measurable.
(3) 溝通狀況
Failed Project
‧ No consistent communication about project goals and status.
‧ Inadequate communication with executives.
‧ Inadequate communication with manual testers.
Successful Project
‧ Same detailed weekly status report sent to all. Status information available online at all times.
‧ Close communication with the VP of Development and Director of QA
‧ Verbal status reports delivered in weekly QA meeting.
(4) 自動化的準備程度
Failed Project
‧ Extremely limited test documentation; most testing ad hoc.
‧ No method of tracking test results.
‧ Testers lacked a strong understanding of how to test the product.
Successful Project
‧ Written test documentation. Each test case numbered individually.
‧ Test results tracked on spreadsheets that referenced the test case number.
‧ The test group as a whole had a much better understanding of how to test the product.
(5) Automated Testing Team的心態
Failed Project
‧ “Bulldozer builders vs. ditch diggers”
‧ Automators didn’t appreciate the value of manual testing.
‧ Manual testers felt threatened.
Successful Project
‧ Service organization focused on building tools.
‧ Automators understood that automation cannot replace manual testing.
‧ Manual testers more involved in the process and therefore less threatened.
從技術的觀點, 她所經歷過的成功和失敗的project有怎樣的差別
(1) Automated Test System 的架構
Failed Project
‧ Tool best-suited to creating individual scripts, not entire systems.
‧ Tool did not support creating a reusable library of functions
‧ No support for logical layer resulted in maintenance nightmare.
Successful Project
‧ Tool specifically designed to support creation of automation systems.
‧ Tool supported & encouraged creating a reusable library (“infrastructure”).
‧ Logical layer vastly improved portability & maintainability of scripts.
(2) Script Creation 的方法
Failed Project
‧ Primarily record & playback
‧ No automatic test case or test data generation
Successful Project
‧ Primarily data driven; record & playback used as a learning tool only.
‧ Used advanced features in the automation tool to support automated test data generation.
Note:
Good Script 設計要素
‧ Tests structured with setup, action, and result
‧ Tests are not order-dependent
‧ Test data is never hard coded
‧ Results are informative
‧ Pass/Fail determination is as automated as is practical
(3) Verification的方法
Failed Project
‧ Bitmap comparisons to verify both window existence & contents of window.
Successful Project
‧ Logical window existence functions to verify window appeared.
‧ Logical comparison between expected fields in window and actual fields.
‧ Test data verification
(4) Automation Programming Practices
Failed Project
‧ Automation standards focused on file naming conventions.
‧ Extremely limited code reviews.
‧ No source control; test script management done exclusively through tool’s management module.
Successful Project
‧ Automation standards focused on what constitutes good code
‧ Both formal & informal code reviews on a regular basis.
‧ Commercial source control system used.
作者在 Automation Management所學到的事情
‧ Set realistic goals.
‧ Measure your progress toward those goals.
‧ Communicate goals and status clearly and consistently.
‧ Don’t let your management set their expectations based on vendor hype.
‧ Coordinate with manual testers.
作者在 Automation Creation所學到的事情
‧ The right architecture can make everything else fall into place.
‧ Having the right tool for the job makes a difference.
‧ Simple scripts can be more powerful than complex do-everything scripts.
‧ Automation is programming: good programming practices apply.
全站熱搜
留言列表
