為何大家只想做GUI Test Automation? 如何改變?
Flipping the Automated Testing Triangle: the Upshot
http://patrickwilsonwelsh.com/?p=32
在Cohn’s ideal triangle中分成三種Test Automation: Brick tests, Stick tests, and Straw tests.
1. Brick tests: unit testing
- They tend to run really fast (on the order of 10 to 100 per second) because they run entirely in memory. - - They tend to pinpoint bugs really well. Then tend to be hard to break. Then are hard to learn to write truly well.
- They are the single most important thing you can learn to automate. Without a solid suite of wellwritten
unit tests, it’s hard to find a software team is that is not basically screwed.
- Tools: xUnit, TestNG, MbUnit...
2. Stick tests: 不透過GUI來做測試, 像是end to end, integration等等
- They bypass the GUI, then tend to be less brittle. They are certainly more brittle than really
good unit tests, however.
- They do tend to be about chunks of behavior as large as a feature, more than about smaller isolated
behavior.
- They again tend to use real external resources, real-ish data.
- They again tend to be large,and to run slowly. By this we mean many minutes, as opposed to a few seconds.
- They are building the right thing, much more than building the thing right.
- Tools: Fit, xUnit, FitNesse, ZiBreve, Concordian...
3. Straw tests: GUI testing
- They tend to use the entire system as a black box, mimicking real-world behavior, talking to all of the real code and external resources and dependencies of the system.
- They tend to focus on large chunks of behavior.
- Tthey tend to be slow
- Tools: Watir, Selenium, Canoo and commercial products...
從這個列表來看, 我們應該要投資在Brick tests, 因為他最快, 開發最省力, 能夠cover的test最多, rework的狀況最少. 可是真實的狀況是什麼呢?
根據作者的調查, 絕大部分的team都投資在Straw tests.
為什麼會這樣呢?
作者認為會比較想做GUI test automation的理由如下:
- They are easier, at first, to learn to write
- Our programmers “don’t do testing,” because we have dedicated manual testers who at first seem like the logical test automators.
- We tend to fall naturally into this pattern: starting with the through-the-GUI tests.
那為什麼不做unit testing的理由如下
We tend not to start with unit tests, because they are hard to learn to do well, because
- Programmers are afraid of them, or because programmers feel that they do not have the time or permission to write them.
- Sometimes hear software professionals claim that these tests are not worth the effort, that they are not valuable enough.
- Here are some of the things a programmer must learn to do well, in order to produce high-value, low-maintenance-cost unit test suites. This is quite a bit to learn.
a. SRP, small modules, mocking/faking, dependency injection, Refactoring, legacy code rescue, TDD, OO, BDD, CI, Design Patterns
b. This is a hell of a lot of work. Unit testing is really, really, hard for most teams to learn. This is especially true when they are already being slogged about by enormous, nasty, untested legacy codebases.
那我們能做什麼去改變這樣的狀態呢?
1. Follow the pain.
- This is true for most agile transitions.
- Is it really production defects blowing up badly that you want to focus on? And are you constrained in the unit tests you can create just yet?
- Are all or most of your tests currently manual? Then it might, in fact, make sense to start with GUI test suites.
2. Early wins;low-hanging fruit.
- For such teams, GUI tests are often the lowest hanging fruit, a logical starting place (though a
dreadful ending place).
3. The Whole Team owns the transition.
- Let there not be fiat testing initiatives from above.
- No “THou Shalt Do TDD or Else” directives, nor coverage rate directives. At least, not without the buy-in and input from the entire team.
- Let the team self-organize -- testers, programmers, BAs, managers -- around where the biggest pain points are, where the low hanging fruit is, and which steps to take when, while still finding a way to meet ridiculous production schedules.
4. Pair programmers with testers/QA persons.
- The parable of the overloaded tester.
- There should be no such thing as a group of programmers blocked by testers who cannot test code fast enough.
- That’s just a version of “that’s not my job.” Here is what you get when any role or responsibility on the team says “that’s not my job.”
5. Pair programmers with testers/QA persons.
- Testing is everybody’s job.
6. Three initiatives: straw, sticks, and bricks.
- The team needs to look at these three initiatives as separate.
- We need separate plans for enough bricks, enough sticks, and just enough straw.
- We need plans for how to increase the bricks as we scale back the straw.
7. Learn to make bricks, no matter what.
- And no matter what happens, if you don’t end up with an absolutely great suite of unit tests
on the typical non-trivial project, this means thousands of tests than run in a few seconds
then you won’t ever get your automated test costs down to least TCO levels.
- And your code maintain costs, turnover costs, and customer satisfaction costs will all be higher as well.
8. Earn testers freedom to do more Exploratory Testing.
- What are the best testers really good at? They are good at sniffing out bugs where others would never think to find them.
- This is instinctual, “blink” skill (from Malcolm Gladwell’s book on deep expertise).
- Use a great automated testing strategy to buy your testers time to do more and more and more exploratory testing (ET) per iteration.
9. There is such a thing as too little courage, skill, trust, respect
Flipping the Automated Testing Triangle: the Upshot
http://patrickwilsonwelsh.com/?p=32
在Cohn’s ideal triangle中分成三種Test Automation: Brick tests, Stick tests, and Straw tests.
1. Brick tests: unit testing
- They tend to run really fast (on the order of 10 to 100 per second) because they run entirely in memory. - - They tend to pinpoint bugs really well. Then tend to be hard to break. Then are hard to learn to write truly well.
- They are the single most important thing you can learn to automate. Without a solid suite of wellwritten
unit tests, it’s hard to find a software team is that is not basically screwed.
- Tools: xUnit, TestNG, MbUnit...
2. Stick tests: 不透過GUI來做測試, 像是end to end, integration等等
- They bypass the GUI, then tend to be less brittle. They are certainly more brittle than really
good unit tests, however.
- They do tend to be about chunks of behavior as large as a feature, more than about smaller isolated
behavior.
- They again tend to use real external resources, real-ish data.
- They again tend to be large,and to run slowly. By this we mean many minutes, as opposed to a few seconds.
- They are building the right thing, much more than building the thing right.
- Tools: Fit, xUnit, FitNesse, ZiBreve, Concordian...
3. Straw tests: GUI testing
- They tend to use the entire system as a black box, mimicking real-world behavior, talking to all of the real code and external resources and dependencies of the system.
- They tend to focus on large chunks of behavior.
- Tthey tend to be slow
- Tools: Watir, Selenium, Canoo and commercial products...
從這個列表來看, 我們應該要投資在Brick tests, 因為他最快, 開發最省力, 能夠cover的test最多, rework的狀況最少. 可是真實的狀況是什麼呢?
根據作者的調查, 絕大部分的team都投資在Straw tests.
為什麼會這樣呢?
作者認為會比較想做GUI test automation的理由如下:
- They are easier, at first, to learn to write
- Our programmers “don’t do testing,” because we have dedicated manual testers who at first seem like the logical test automators.
- We tend to fall naturally into this pattern: starting with the through-the-GUI tests.
那為什麼不做unit testing的理由如下
We tend not to start with unit tests, because they are hard to learn to do well, because
- Programmers are afraid of them, or because programmers feel that they do not have the time or permission to write them.
- Sometimes hear software professionals claim that these tests are not worth the effort, that they are not valuable enough.
- Here are some of the things a programmer must learn to do well, in order to produce high-value, low-maintenance-cost unit test suites. This is quite a bit to learn.
a. SRP, small modules, mocking/faking, dependency injection, Refactoring, legacy code rescue, TDD, OO, BDD, CI, Design Patterns
b. This is a hell of a lot of work. Unit testing is really, really, hard for most teams to learn. This is especially true when they are already being slogged about by enormous, nasty, untested legacy codebases.
那我們能做什麼去改變這樣的狀態呢?
1. Follow the pain.
- This is true for most agile transitions.
- Is it really production defects blowing up badly that you want to focus on? And are you constrained in the unit tests you can create just yet?
- Are all or most of your tests currently manual? Then it might, in fact, make sense to start with GUI test suites.
2. Early wins;low-hanging fruit.
- For such teams, GUI tests are often the lowest hanging fruit, a logical starting place (though a
dreadful ending place).
3. The Whole Team owns the transition.
- Let there not be fiat testing initiatives from above.
- No “THou Shalt Do TDD or Else” directives, nor coverage rate directives. At least, not without the buy-in and input from the entire team.
- Let the team self-organize -- testers, programmers, BAs, managers -- around where the biggest pain points are, where the low hanging fruit is, and which steps to take when, while still finding a way to meet ridiculous production schedules.
4. Pair programmers with testers/QA persons.
- The parable of the overloaded tester.
- There should be no such thing as a group of programmers blocked by testers who cannot test code fast enough.
- That’s just a version of “that’s not my job.” Here is what you get when any role or responsibility on the team says “that’s not my job.”
5. Pair programmers with testers/QA persons.
- Testing is everybody’s job.
6. Three initiatives: straw, sticks, and bricks.
- The team needs to look at these three initiatives as separate.
- We need separate plans for enough bricks, enough sticks, and just enough straw.
- We need plans for how to increase the bricks as we scale back the straw.
7. Learn to make bricks, no matter what.
- And no matter what happens, if you don’t end up with an absolutely great suite of unit tests
on the typical non-trivial project, this means thousands of tests than run in a few seconds
then you won’t ever get your automated test costs down to least TCO levels.
- And your code maintain costs, turnover costs, and customer satisfaction costs will all be higher as well.
8. Earn testers freedom to do more Exploratory Testing.
- What are the best testers really good at? They are good at sniffing out bugs where others would never think to find them.
- This is instinctual, “blink” skill (from Malcolm Gladwell’s book on deep expertise).
- Use a great automated testing strategy to buy your testers time to do more and more and more exploratory testing (ET) per iteration.
9. There is such a thing as too little courage, skill, trust, respect
全站熱搜
留言列表
