目前分類：Exploratory Testing (14)
- Aug 14 Sun 2016 09:59
- May 25 Mon 2015 23:11
- May 17 Tue 2011 15:34
Exploratory Testing 相關資料
1. Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design
- Apr 26 Mon 2010 10:04
- Aug 26 Wed 2009 13:52
Exploratory Testing 新書介紹
等了很久終於有人為exploratory testing 出書了, 作者是很有名的專家, 在microsoft有很崇高的地位, 不過在2009時加入了Google, 看起來Google應該也是很重視測試才會把他挖角過去吧!!
Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design
by James A. Whittaker
- Aug 11 Tue 2009 09:41
Scripted vs Exploratory testing from a managerial perspective
July 30, 2009
Posted by Martin Jansson
- Mar 06 Fri 2009 09:31
2009 Feb 17
Pubished in Inside QA
ET(Exploratory Testing) 在agile中常常被人拿出來使用, 但是即使在非agile的狀況下, 作者也認為ET也是非常適用的. 它能幫助你找到一些你沒想到的問題.
在Agile的project中, 正常來說要執行, unit testing, 用FIT來做integration testing, 要和CI(continuous integration) 整合, 要做acceptance testing等等, 如果有正確被執行, 相信品質會有一定的水準.
但是事實上, 事情通常沒有這麼順利, 可能是那些事情沒有做, 或是即使有做, 但是那些測試項目範圍或是深度不夠. 因此tester通常需要適時執行一些ET, 來確保軟體的品質.
以下是作者列出的checking list, 來確認ET要使用的時機
1. Always use in conjunction with planned tests on high impact stories. Cover as much as you can!
2. Use when trying to reproduce system failure.
3. Use when defect clusters have been identified. This will flush out even more defects.
4. Always use when you have a good technical understanding of the system architecture. You will already be aware of what usually breaks certain systems.
1. Demonstrate a plan of action. Even a quick outline of what you aim to achieve by carrying out certain actions will give confidence in your actions.
2. Write down all tests that are performed. I use a test short hand that describes navigation/action/result in just one sentence. This enables you to create more tests further down the line.
3. Let the system risk analysis guide you to critical areas of the application. This is where exploratory testing pays off.
4. Sit near to, or with the development team to enable quick solutions to problems and questions.
5. Never rely on just doing exploratory testing.
記住: 完全沒有計畫的隨機測試, 不是exploratory Testing, 而是糟糕的測試.
- Jan 15 Thu 2009 09:44
explaining exploratory testing
January 08, 2009
Posted by James Whittaker
Published in JW on Test
公司最近有介紹Exploratory Testing 的sharing, 在會議中發現大家幾乎都不知道這是什麼, 因此希望這篇文章可以多幫忙了解一點為何要做Exploratory Testing
軟體測試之所以複雜, 是在於它有過多可能的組合. (在我認為是接近無限多):
- code paths to state
- operational environment.
這三種的情況, 前兩種的數量接近無限多, 第三種是很多, 所以combine起來後, 狀況幾乎是無法想像的多.
所以不管你採取什麼方法, Scripted Testing (也就是事先規劃) 或是 Exploratory Testing (一邊學, 一邊規劃, 和一邊測試), 要完整測試是不可能的事
然而Exploratory Testing的好處是, 讓QA能在測試過程中, 根據他所學到, 所收集到的, 然後來決定接下來要整麼調整, 怎麼進行接下來的測試. 而不是只是follow up之前規劃好的test cases 或是test plan.
你可以想像, 如果你若是要在季賽前, 要預測 Super Bowl (美國橄欖球超級杯大賽) or Premier League(英超聯賽)的贏家, 你會不會認為是一件非常難的事情.
因為你需要觀察team如何來運作, 進攻; 或是主力球員是否整季都健康, 沒有受傷; 或是教練和球員整合程度. 這些都是變因, 這些資訊會隨著賽季的展開, 你才能逐步調整你的預測, 然後會對的機會, 才會越來越高.
同理, 再軟體測試上也是一樣, Exploratory Testing也是在過程中, 一方面根據之前的歷史資料, 所學會的domain knowledge, 再加上目前狀況的需求, 不斷再調整所要進攻的方向, 已確定你所要測試的重點
所以有效的使用Exploratory Testing, 可以幫助你在面對軟體測試複雜的狀況, 能夠更有信心去deliver quality software.
- Jan 04 Sun 2009 15:38
Exploratory Testing challenged - Part I
2007 Dec 29
Published in Thinking Tester
1. Exploratory Testing (ET)的好處是什麼?
ET will extend the test coverage over traditional scripted testing, you will be able discover those bugs that are not “catchable” by scripted tests.
2. 為什麼Scripted Testing無法找到所有bugs?
- Our scripts are based on specifications which are one narrow, fallible source of information about “intended software behavior”.
- Since specifications are written in English – there could interpretations/misinterpretations. Since our specifications are fallible, so are our scripts.
- There is a human limitation to understand and interpret specifications (objectively) and design the test cases that cover the entire test space.
- So, there is good possibility that scripts will not find all bugs that potentially be discovered.
3. 為什麼test cases為什麼被執行多次後, 越來越沒有用處呢?
- The scripted approach to testing attempts to mechanize the test process by taking test ideas out of a test designer's head and putting them on paper.
- Test cases wear out as they do not have any in built mechanism in them to alter themselves to changing product environment.
- Test scripts can not think, infer, improvise, get frustrated as intelligent human testers do. Hence test scripts can not find bugs that repeatedly than a human tester.
- But exploratory testers take the view that writing down test scripts and following them tends to disrupt the intellectual processes that make testers able to find important problems quickly.
4. 那Ad Hoc Testing和Exploratory Testing是相同的東西嗎?
- Yes as Adhoc testing is well known predecessor to ET. Cem Kaner coined this term ET around early 80’s to distinguish ET and Adhoc Testing.
- Cem thought that there were lots of confusions regarding some kind of “impromptu” testing that does not rely on predefined scripts.
- Ad hoc testing normally refers to a process of improvised, impromptu bug searching. By definition, anyone can do ad hoc testing.
- The term "exploratory testing"--coined by Cem Kaner, in “Testing Computer Software” -- refers to ET as a sophisticated, thoughtful approach to ad hoc testing.
5. 聽說 ET要求要有經驗, 並且Domain knowledge要高. 那average tester可以做好ET嗎?
- I believe, ET requires skills like “questioning”, “modeling”, “critical thinking” among others.
- Domain knowledge certainly helps in ET but I do not consider it as mandatory.
- It depends upon what kind of bugs you want to discover.
- ET can be performed in controlled, small time boxed sessions with specific charters to explore a specific feature of the application.
- ET can be configured to cater to specific investigative missions.
- You could use few ET sessions to develop a software product documentation or to analyse and isolate performance test results.
當你有5000 test scripts, 你執行1到2的cycles, 你就從目前的test script再也找不出錯誤了. 你會不會覺得這系統已經沒有問題了?
或者你找到的bug都不是從這些test scripts中發現的, 你會不會覺得這test script有問題?
如果你還要在執行5 個test cycles, 你會全跑這些test script或是怎麼做呢?
Shrini沒有提到答案, 但是已經暗示我們scripted testing是不夠的. 或許你的Testing Manager或是其他的Manager 無法被說服去採去ET, 但是你自己內心深處, 一定知道scripted testing無法解決所有問題
這時候大師出馬了, Cem Kaner 說
I rarely try to persuade a test manager to switch to exploratory testing.
Instead, I recommend to them that they include some exploratory testing in their regular testing process.
Over time, a thoughtful manager will find a context- appropriate balance between scripted and exploratory tests.
這裡他提出一個經驗法則, 告訴你要如何混搭ET 和ST
In every build, even the last build:
- spend 25% of your time trying new things. Some of these will find bugs. Some will give you good ideas for reusable tests.
- spend 25% of your time implementing new tests as reusable tests.
- spend 50% of your time running old tests (including bug regressions).
這個經驗法則有個問題: 花太多時間在做test execution, learning的時間不夠多
-Learning is a very big part of exploration, and that doesn't come just from running tests and seeing results.
- It comes from finding information about the market, the platform, the customer expectations, the risks for similar products, etc.
- This learning guides initial test design (the 25% for new things) and the design of the reusable tests.
- It also provides important guidance for writing bug reports (helping you decide, and explain) whether a bug is serious and what customer reactions might be.
不過對於一般的manager來說, 這個經驗法則的好處是: it is very easy for many managers to understand and try.
- It hides the non-test-execution research that competent explorers must do -- is a serious problem.
- If you are the consultant, then as your client gains experience with exploratory work, you can raise higher level cognitive issues and the value of preparatory research gradually, as examples present themselves in the situation of your client.
- Nov 19 Wed 2008 09:30
- Nov 18 Tue 2008 22:46
Exploratory Testing是否只適用於senior testers的身上?
Is exploratory testing only for senior testers?
September 16th, 2008 in SearchSoftwareQuality
- Nov 11 Tue 2008 08:00
- Oct 17 Fri 2008 13:35
Where Does Exploratory Testing Fit?
by James Bach
1. Exploratory Testing(ET)的定義
(1)concurrent test design and test execution.
(2) Any testing is exploratory to the extent that
- Oct 16 Thu 2008 09:31
What is Exploratory Testing?
by James Bach
Exploratory testing (ET) 是agile testing中最有名的方法, 可是也是大家最陌生的方法. 因為很多人很容易把它和Ad Hoc testing想在一起, 而對它有錯誤的誤解. 所以這篇文章企圖給大家一些基本介紹, 讓你了解ET到底是什麼
1. ET和scripted testing的定義