How We Test Software at Microsoft目錄介紹
What's in HWTSAM
http://blogs.msdn.com/alanpa/
很多人都很關心這本書到底好不好, 作者post 這本書的目錄在他的blog上. 目錄的東西看起來還中規中矩的, 確實是在談Microsoft的經驗, 希望看到內容時也會十分滿意.
第一部份討論Microsoft中各種role: engineers and test engineers. 以及Microsoft所用的software engineering
Part I, “About Microsoft”
Chapter 1, “Software Engineering at Microsoft,”
Chapter 2, “Software Test Engineers at Microsoft”
Chapter 3, “Engineering Life Cycles”
第二部份討論Microsoft常用的各種測試方法
Part II, “About Testing”
Chapter 4, “A Practical Approach to Test Case Design”
Chapter 5, “Functional Testing Techniques”
Chapter 6, “Structural Testing Techniques”
Chapter 7, “Analyzing Risk with Code Complexity”
Chapter 8, “Model-Based Testing”
(Notes:我覺得Microsoft很喜歡這樣的測試方法, 之前好像就有Microsoft的員工出書, 有興趣的可以參考一下
Practical Model-Based Testing: A Tools Approach
http://www.amazon.com/Practical-Model-Based-Testing-Tools-Approach/dp/0123725011/ref=sr_1_1?ie=UTF8&s=books&qid=1230600127&sr=1-1
)
第三部份討論Microsoft所使用的各種工具及系統
Part III, “Test Tools and Systems”
Chapter 9, “Managing Bugs and Test Cases”
Chapter 10, “Test Automation”
Chapter 11, “Non-Functional Testing”
Chapter 12, “Other Tools”
Chapter 13, “Customer Feedback Systems”
Chapter 14, “Testing Software Plus Services”
最後一部份, 討論testing 的未來, 和Microsoft要如何去建立
Part IV, “About the Future”
Chapter 15, “Solving Tomorrow’s Problems Today”
Chapter 16, “Building the Future”
kojenchieh 發表在 痞客邦 留言(0) 人氣()
為何大家只想做GUI Test Automation? 如何改變?
Flipping the Automated Testing Triangle: the Upshot
http://patrickwilsonwelsh.com/?p=32
在Cohn’s ideal triangle中分成三種Test Automation: Brick tests, Stick tests, and Straw tests.
1. Brick tests: unit testing
- They tend to run really fast (on the order of 10 to 100 per second) because they run entirely in memory. - - They tend to pinpoint bugs really well. Then tend to be hard to break. Then are hard to learn to write truly well.
- They are the single most important thing you can learn to automate. Without a solid suite of wellwritten
unit tests, it’s hard to find a software team is that is not basically screwed.
- Tools: xUnit, TestNG, MbUnit...
2. Stick tests: 不透過GUI來做測試, 像是end to end, integration等等
- They bypass the GUI, then tend to be less brittle. They are certainly more brittle than really
good unit tests, however.
- They do tend to be about chunks of behavior as large as a feature, more than about smaller isolated
behavior.
- They again tend to use real external resources, real-ish data.
- They again tend to be large,and to run slowly. By this we mean many minutes, as opposed to a few seconds.
- They are building the right thing, much more than building the thing right.
- Tools: Fit, xUnit, FitNesse, ZiBreve, Concordian...
3. Straw tests: GUI testing
- They tend to use the entire system as a black box, mimicking real-world behavior, talking to all of the real code and external resources and dependencies of the system.
- They tend to focus on large chunks of behavior.
- Tthey tend to be slow
- Tools: Watir, Selenium, Canoo and commercial products...
從這個列表來看, 我們應該要投資在Brick tests, 因為他最快, 開發最省力, 能夠cover的test最多, rework的狀況最少. 可是真實的狀況是什麼呢?
根據作者的調查, 絕大部分的team都投資在Straw tests.
為什麼會這樣呢?
作者認為會比較想做GUI test automation的理由如下:
- They are easier, at first, to learn to write
- Our programmers “don’t do testing,” because we have dedicated manual testers who at first seem like the logical test automators.
- We tend to fall naturally into this pattern: starting with the through-the-GUI tests.
那為什麼不做unit testing的理由如下
We tend not to start with unit tests, because they are hard to learn to do well, because
- Programmers are afraid of them, or because programmers feel that they do not have the time or permission to write them.
- Sometimes hear software professionals claim that these tests are not worth the effort, that they are not valuable enough.
- Here are some of the things a programmer must learn to do well, in order to produce high-value, low-maintenance-cost unit test suites. This is quite a bit to learn.
a. SRP, small modules, mocking/faking, dependency injection, Refactoring, legacy code rescue, TDD, OO, BDD, CI, Design Patterns
b. This is a hell of a lot of work. Unit testing is really, really, hard for most teams to learn. This is especially true when they are already being slogged about by enormous, nasty, untested legacy codebases.
那我們能做什麼去改變這樣的狀態呢?
1. Follow the pain.
- This is true for most agile transitions.
- Is it really production defects blowing up badly that you want to focus on? And are you constrained in the unit tests you can create just yet?
- Are all or most of your tests currently manual? Then it might, in fact, make sense to start with GUI test suites.
2. Early wins;low-hanging fruit.
- For such teams, GUI tests are often the lowest hanging fruit, a logical starting place (though a
dreadful ending place).
3. The Whole Team owns the transition.
- Let there not be fiat testing initiatives from above.
- No “THou Shalt Do TDD or Else” directives, nor coverage rate directives. At least, not without the buy-in and input from the entire team.
- Let the team self-organize -- testers, programmers, BAs, managers -- around where the biggest pain points are, where the low hanging fruit is, and which steps to take when, while still finding a way to meet ridiculous production schedules.
4. Pair programmers with testers/QA persons.
- The parable of the overloaded tester.
- There should be no such thing as a group of programmers blocked by testers who cannot test code fast enough.
- That’s just a version of “that’s not my job.” Here is what you get when any role or responsibility on the team says “that’s not my job.”
5. Pair programmers with testers/QA persons.
- Testing is everybody’s job.
6. Three initiatives: straw, sticks, and bricks.
- The team needs to look at these three initiatives as separate.
- We need separate plans for enough bricks, enough sticks, and just enough straw.
- We need plans for how to increase the bricks as we scale back the straw.
7. Learn to make bricks, no matter what.
- And no matter what happens, if you don’t end up with an absolutely great suite of unit tests
on the typical non-trivial project, this means thousands of tests than run in a few seconds
then you won’t ever get your automated test costs down to least TCO levels.
- And your code maintain costs, turnover costs, and customer satisfaction costs will all be higher as well.
8. Earn testers freedom to do more Exploratory Testing.
- What are the best testers really good at? They are good at sniffing out bugs where others would never think to find them.
- This is instinctual, “blink” skill (from Malcolm Gladwell’s book on deep expertise).
- Use a great automated testing strategy to buy your testers time to do more and more and more exploratory testing (ET) per iteration.
9. There is such a thing as too little courage, skill, trust, respect
kojenchieh 發表在 痞客邦 留言(2) 人氣()
Test Automation Video
這裡有一些Test automation的video, 都是來自於 Google Tech Talk. Google對testing真的蠻重視的, 有些presenter 都還蠻有名的. 我想我們公司應該可以效倣他們, 來提升我們testing的水準, 不要只是光只有RD的演講, 這樣會讓人家認為我們其實不重視testing.
1. GTAC 2008: Context-Driven Test Automation - How to Build the System You Really Need
http://tw.youtube.com/watch?v=N9sm_zcpUEw
- Google Tech Talks October 24, 2008
- Presenter: Peter Schneider
2. Google London Test Automation Conference (LTAC) Openning
http://tw.youtube.com/watch?v=iahKzSoK6kY
- Google Tech Talks September 7th, 2006
- Presenters: Shannon Maher and Allen Hutchison
3. Using Test Oracles in Automation
http://tw.youtube.com/watch?v=Yusd_ogkiaQ
- Google TechTalks April 25, 2006
- Presenter: Douglas Hoffman
4. Automated Testing Patterns and Smells
http://tw.youtube.com/watch?v=Pq6LHFM4JvE
- Google Tech Talks March, 6 2008
- Presenter: Gerard Meszaros
5. Advances in Automated Software Testing Technologies
http://tw.youtube.com/watch?v=HEpSdSyU03I&feature=related
- Google Tech Talks October 23, 2008
- Presenters: Elfriede Dustin and Marcus Borch
6. The Clean Code Talks -- Unit Testing
http://tw.youtube.com/watch?v=wEhu57pih5w&feature=channel
- Google Tech Talks October, 30 2008
- Presenter: Misko Hevery
7. Using Cloud Computing to Automate Full-Scale System Tests
http://tw.youtube.com/watch?v=atyq-41Gnjc&feature=channel
- Google Tech Talks October 23, 2008
- Presenter: Marc-Elian Begin
8. Testing mobile handsets with Fitnesse
http://tw.youtube.com/watch?v=Ul78_CqvXwk
- Google Tech Talks September 8th, 2006
- Presenters: Uffe Koch & Mark Boxall
9. GTAC 2008: Automated Model-Based Testing of Web Applications
http://tw.youtube.com/watch?v=6LdsIVvxISU&feature=channel
- Google Tech Talks October 24, 2008
- Presenters: Atif Memon, Oluwaseun Akinmade
kojenchieh 發表在 痞客邦 留言(0) 人氣()
kojenchieh 發表在 痞客邦 留言(0) 人氣()
Google C++ Mocking Framework
五個月前, Google才announce C++ Testing Framwork, 現在Goolge又announce C++ Mocking Framework. 它可以support Linux, Windows 和Mac OS X. Google宣稱內部已經有超過100projects用過它, 大家的反應都還不錯, 以下是他的好處
* Simple, declarative syntax for defining mocks
* Rich set of matchers for validating function arguments
* Intuitive syntax for controlling the behavior of a mock
* Automatic verification of expectations
* Easy extensibility through new user-defined matchers and actions
以下是相關的URL:
Google C++ Mocking Framework Home Page
http://code.google.com/p/googlemock/
Documentation URL
http://code.google.com/p/googlemock/w/list
Download URL
http://code.google.com/p/googlemock/downloads/list
Google Mocking Framework Discussion Group
http://groups.google.com/group/googlemock
Google提供一個簡單的sample:
這裡有一個class ShoppingCart, 它會從server拿到tax rate.
你要測試的狀況是: it remembers to disconnect from the server even when the server has generated an error.
class TaxServer {
// Returns the tax rate of a location
// (by postal code) or -1 on error.
virtual double FetchTaxRate(
const string& postal_code) = 0;
virtual void CloseConnection() = 0;
};
這裡demo你如何用mock server去verify
class MockTaxServer : public TaxServer { // #1
MOCK_METHOD1(FetchTaxRate, double(const string&));
MOCK_METHOD0(CloseConnection, void());
};
TEST(ShoppingCartTest,
StillCallsCloseIfServerErrorOccurs) {
MockTaxServer mock_taxserver; // #2
EXPECT_CALL(mock_taxserver, FetchTaxRate(_))
.WillOnce(Return(-1)); // #3
EXPECT_CALL(mock_taxserver, CloseConnection());
ShoppingCart cart(&mock_taxserver); // #4
cart.CalculateTax(); // Calls FetchTaxRate()
// and CloseConnection().
} // #5
1. Derive the mock class from the interface. For each virtual method, count how many arguments it has, name the result n, and define it using MOCK_METHODn, whose arguments are the name and type of the method.
2. Create an instance of the mock class. It will be used where you would normally use a real object.
3. Set expectations on the mock object (How will it be used? What will it do?). For example, the first EXPECT_CALL says that FetchTaxRate() will be called and will return an error. The underscore (_) is a matcher that says the argument can be anything. Google Mock has many matchers you can use to precisely specify what the argument should be like. You can also define your own matcher or use an exact value.
4. Exercise code that uses the mock object. You'll get an error immediately if a mock method is called more times than expected or with the wrong arguments.
5. When the mock object is destroyed, it checks that all expectations on it have been satisfied.
Reference:
1. Announcing Google C++ Mocking Framework
http://googletesting.blogspot.com/2008/12/announcing-google-c-mocking-framework.html
2. Mockers of the (C++) World, Delight!
http://googletesting.blogspot.com/2008/12/mockers-of-c-world-delight.html
kojenchieh 發表在 痞客邦 留言(0) 人氣()
