Exploratory Testing常見的Q&A
Exploratory Testing challenged - Part I
http://shrinik.blogspot.com/2007/12/exploratory-testing-challenged-part-i.html
2007 Dec 29
Posted by Shrini Kulkarni
Published in Thinking Tester
1. Exploratory Testing (ET)的好處是什麼?
Ans:
ET will extend the test coverage over traditional scripted testing, you will be able discover those bugs that are not “catchable” by scripted tests.
2. 為什麼Scripted Testing無法找到所有bugs?
Ans:
- Our scripts are based on specifications which are one narrow, fallible source of information about “intended software behavior”.
- Since specifications are written in English – there could interpretations/misinterpretations. Since our specifications are fallible, so are our scripts.
- There is a human limitation to understand and interpret specifications (objectively) and design the test cases that cover the entire test space.
- So, there is good possibility that scripts will not find all bugs that potentially be discovered.
3. 為什麼test cases為什麼被執行多次後, 越來越沒有用處呢?
Ans:
- The scripted approach to testing attempts to mechanize the test process by taking test ideas out of a test designer's head and putting them on paper.
- Test cases wear out as they do not have any in built mechanism in them to alter themselves to changing product environment.
- Test scripts can not think, infer, improvise, get frustrated as intelligent human testers do. Hence test scripts can not find bugs that repeatedly than a human tester.
- But exploratory testers take the view that writing down test scripts and following them tends to disrupt the intellectual processes that make testers able to find important problems quickly.
4. 那Ad Hoc Testing和Exploratory Testing是相同的東西嗎?
Ans:
- Yes as Adhoc testing is well known predecessor to ET. Cem Kaner coined this term ET around early 80’s to distinguish ET and Adhoc Testing.
- Cem thought that there were lots of confusions regarding some kind of “impromptu” testing that does not rely on predefined scripts.
- Ad hoc testing normally refers to a process of improvised, impromptu bug searching. By definition, anyone can do ad hoc testing.
- The term "exploratory testing"--coined by Cem Kaner, in “Testing Computer Software” -- refers to ET as a sophisticated, thoughtful approach to ad hoc testing.
5. 聽說 ET要求要有經驗, 並且Domain knowledge要高. 那average tester可以做好ET嗎?
Ans:
- I believe, ET requires skills like “questioning”, “modeling”, “critical thinking” among others.
- Domain knowledge certainly helps in ET but I do not consider it as mandatory.
6. 那ET可以找到什麼種類的bug?
Ans:
- It depends upon what kind of bugs you want to discover.
- ET can be performed in controlled, small time boxed sessions with specific charters to explore a specific feature of the application.
- ET can be configured to cater to specific investigative missions.
- You could use few ET sessions to develop a software product documentation or to analyse and isolate performance test results.
7.
當你有5000 test scripts, 你執行1到2的cycles, 你就從目前的test script再也找不出錯誤了. 你會不會覺得這系統已經沒有問題了?
或者你找到的bug都不是從這些test scripts中發現的, 你會不會覺得這test script有問題?
如果你還要在執行5 個test cycles, 你會全跑這些test script或是怎麼做呢?
Ans:
Shrini沒有提到答案, 但是已經暗示我們scripted testing是不夠的. 或許你的Testing Manager或是其他的Manager 無法被說服去採去ET, 但是你自己內心深處, 一定知道scripted testing無法解決所有問題
這時候大師出馬了, Cem Kaner 說
I rarely try to persuade a test manager to switch to exploratory testing.
Instead, I recommend to them that they include some exploratory testing in their regular testing process.
Over time, a thoughtful manager will find a context- appropriate balance between scripted and exploratory tests.
這裡他提出一個經驗法則, 告訴你要如何混搭ET 和ST
In every build, even the last build:
- spend 25% of your time trying new things. Some of these will find bugs. Some will give you good ideas for reusable tests.
- spend 25% of your time implementing new tests as reusable tests.
- spend 50% of your time running old tests (including bug regressions).
這個經驗法則有個問題: 花太多時間在做test execution, learning的時間不夠多
-Learning is a very big part of exploration, and that doesn't come just from running tests and seeing results.
- It comes from finding information about the market, the platform, the customer expectations, the risks for similar products, etc.
- This learning guides initial test design (the 25% for new things) and the design of the reusable tests.
- It also provides important guidance for writing bug reports (helping you decide, and explain) whether a bug is serious and what customer reactions might be.
不過對於一般的manager來說, 這個經驗法則的好處是: it is very easy for many managers to understand and try.
壞處則是:
- It hides the non-test-execution research that competent explorers must do -- is a serious problem.
- If you are the consultant, then as your client gains experience with exploratory work, you can raise higher level cognitive issues and the value of preparatory research gradually, as examples present themselves in the situation of your client.
