目前分類：Exploratory Testing (22)
- May 21 Thu 2020 20:14
探索測試 (ET) 和 腳本測試 (ST) 的比較
- Jun 11 Mon 2018 11:24
[ET] 探索式測試如何整合測試自動化
測試自動化目前是開發活動中的當紅要角, 不管有沒有做, 任何人都可以來說嘴幾句, 探索式測試自然也要來跟他攀上關係, 就讓我們來談談兩者要如何合作.
測試的做法, 主要分成檢查 (checking) 和探索 (exploring) 兩種. 因為很多人的誤解, 測試自動化 (Test Automation) 往往變成是偏向檢查類別的.
- Jun 10 Sun 2018 16:22
[ET] 如何整合探索式測試到你的專案中
前面提了不少探索式測試 (exploratory testing, ET) 的原理, 相信大家已有基本的認識. 但是要如何落實到專案中, 大家還是有很多疑問. 因為每個專案特性不同, 測試流程也不同, 需要對這些狀況做客製化調整.
在探索式測試實踐之路ㄧ書中 , 作者根據他在淘寶網的經驗, 整理了四種流程, 我想應該可以適用在大多數環境. 描述如下:
- Jun 07 Thu 2018 09:36
[ET] 你不知道其實人人都有在 run 探索性測試 - 你不知道你不知道什麼
軟體測試可以分成探索的方式 (Exploratory Testing, 之後簡稱 ET) 和檢查的方式 (Scripted Testing, 之後簡稱 ST) 來進行. 所謂 ST, 就是根據測試個案的內容, 照表操課來執行. 所謂的 ET, 就是事先沒有寫好的測試個案, 在測試當下, 同時進行測試設計, 執行和學習.
但是這絕對不是二分法. 也就是說, 在專案執行過程中, 不會單純只做 ST 和 ET, 兩者通常是混合進行的. 說得更準確一點, 其實每個人在測試的過程, ET 和 ST 的動作往往是交互發生的.
是這樣嗎? 我們來看一下各種不同程度的 ET 和 ST 的組合
- Jun 06 Wed 2018 15:08
[ET] 暴徒式探索測試 - 搶劫也是可以來當測試的招
我想 Mob Programming (暴徒式編程) 大家可能聽過吧? 他是搭擋編程 (Pair Programming) 的變形版, 讓較多人一起開發, 一起學習.
或許一開始大家會覺得有點浪費時間, 其實他可以省下 review 時間, 讓大家在第一時間就知道這系統是怎麼一回事, 並且也學習到別人的開發技巧.
- Jun 05 Tue 2018 21:32
[ET] 怎樣用玩來進行測試? 探索式測試也是可以玩很爽的
探索式測試通常會以迭代為主的方式來進行, 在每次迭代中, 會測試某個 charter. 但是大家痛苦的, 就是 charter 應該怎樣找呢?
在 James Whittaker 所著的探索式軟體測試(Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design) 一書中, 便有一些 charter 的點子可以參考.
- Jun 04 Mon 2018 17:39
[ET] 何謂探索式測試? 一種強迫動腦的測試風格
- Jun 03 Sun 2018 15:21
[ET] 為什麼測試不受重視? 你老是只求有做, 如何帶來價值和尊重呢?
- Aug 14 Sun 2016 09:59
探索性測試執行之旅的回顧
公司在五月時, 請了邰曉梅老師來台灣教授探索性測試, 那次的課程收穫良多, 推翻了不少我對探索性測試的認識. 因此, 之前也花些時間, 整理了探索性測試的相關資料.
接著我也在專案中, 讓團隊成員試試看探索性測試的做法, 幾個上過課的成員, 覺得效果不錯, 值得在專案內繼續使用, 並且也值得讓剩下的成員在明年來上這門課. 因此, 趁著剛做完回顧會議, 把一些重點給記錄下來.
- May 25 Mon 2015 23:11
起手式也能制敵
在碧血劍中, 有一場劇情是這樣的, 男主角袁承志和其師兄的弟子過招, 在過程中他告訴對方華山派拳法的起手式也是能在實戰上被使用的. 小說是這樣寫:
袁承志道：「你以為起手式只是客套禮數，臨敵時無用的麼？要知咱們祖師爺創下這套拳來，沒一招不能克敵制勝。你瞧著。」
- May 17 Tue 2011 15:34
Exploratory Testing 相關資料
Exploratory Testing 相關資料
A. 書籍
1. Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design
http://www.amazon.com/Exploratory-Software-Testing-Tricks-Techniques/dp/0321636414/ref=sr_1_1?s=books&ie=UTF8&qid=1305614121&sr=1-1
- Apr 26 Mon 2010 10:04
Exploratory Testing 相關文章
Exploratory Testing 相關文章
最近看了James A Whitttaker所寫的Exploratory Software Testing, 覺得exploratory testing比我想像的還有趣, 因此收集了一些Exploratory testing相關文章, 之後好好地再回味一下
1. Exploratory Testing Explained
http://www.satisfice.com/articles/et-article.pdf
2. Session-Based Test Management,
- Aug 26 Wed 2009 13:52
Exploratory Testing 新書介紹
Exploratory Testing 新書介紹
等了很久終於有人為exploratory testing 出書了, 作者是很有名的專家, 在microsoft有很崇高的地位, 不過在2009時加入了Google, 看起來Google應該也是很重視測試才會把他挖角過去吧!!
Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design
by James A. Whittaker
http://www.amazon.com/exec/obidos/tg/detail/-/0321636414/ref=ord_cart_shr?%5Fencoding=UTF8&m=ATVPDKIKX0DER&v=glance
- Aug 11 Tue 2009 09:41
從管理階層的觀點來看Scripted和Exploratory testing
從管理階層的觀點來看Scripted和Exploratory testing
Scripted vs Exploratory testing from a managerial perspective
July 30, 2009
http://thetesteye.com/blog/2009/07/scripted-vs-exploratory-testing-from-a-managerial-perspective/
Posted by Martin Jansson
- Mar 06 Fri 2009 09:31
Exploratory Testing
Exploratory Testing
http://insidesqa.blogspot.com/2009/02/exploratory-testing.html
2009 Feb 17
Pubished in Inside QA
ET(Exploratory Testing) 在agile中常常被人拿出來使用, 但是即使在非agile的狀況下, 作者也認為ET也是非常適用的. 它能幫助你找到一些你沒想到的問題.
在Agile的project中, 正常來說要執行, unit testing, 用FIT來做integration testing, 要和CI(continuous integration) 整合, 要做acceptance testing等等, 如果有正確被執行, 相信品質會有一定的水準.
但是事實上, 事情通常沒有這麼順利, 可能是那些事情沒有做, 或是即使有做, 但是那些測試項目範圍或是深度不夠. 因此tester通常需要適時執行一些ET, 來確保軟體的品質.
以下是作者列出的checking list, 來確認ET要使用的時機
1. Always use in conjunction with planned tests on high impact stories. Cover as much as you can!
2. Use when trying to reproduce system failure.
3. Use when defect clusters have been identified. This will flush out even more defects.
4. Always use when you have a good technical understanding of the system architecture. You will already be aware of what usually breaks certain systems.
當你在執行ET時, 作者建議必須要小心以下事情
1. Demonstrate a plan of action. Even a quick outline of what you aim to achieve by carrying out certain actions will give confidence in your actions.
2. Write down all tests that are performed. I use a test short hand that describes navigation/action/result in just one sentence. This enables you to create more tests further down the line.
3. Let the system risk analysis guide you to critical areas of the application. This is where exploratory testing pays off.
4. Sit near to, or with the development team to enable quick solutions to problems and questions.
5. Never rely on just doing exploratory testing.
記住: 完全沒有計畫的隨機測試, 不是exploratory Testing, 而是糟糕的測試.
http://insidesqa.blogspot.com/2009/02/exploratory-testing.html
2009 Feb 17
Pubished in Inside QA
ET(Exploratory Testing) 在agile中常常被人拿出來使用, 但是即使在非agile的狀況下, 作者也認為ET也是非常適用的. 它能幫助你找到一些你沒想到的問題.
在Agile的project中, 正常來說要執行, unit testing, 用FIT來做integration testing, 要和CI(continuous integration) 整合, 要做acceptance testing等等, 如果有正確被執行, 相信品質會有一定的水準.
但是事實上, 事情通常沒有這麼順利, 可能是那些事情沒有做, 或是即使有做, 但是那些測試項目範圍或是深度不夠. 因此tester通常需要適時執行一些ET, 來確保軟體的品質.
以下是作者列出的checking list, 來確認ET要使用的時機
1. Always use in conjunction with planned tests on high impact stories. Cover as much as you can!
2. Use when trying to reproduce system failure.
3. Use when defect clusters have been identified. This will flush out even more defects.
4. Always use when you have a good technical understanding of the system architecture. You will already be aware of what usually breaks certain systems.
當你在執行ET時, 作者建議必須要小心以下事情
1. Demonstrate a plan of action. Even a quick outline of what you aim to achieve by carrying out certain actions will give confidence in your actions.
2. Write down all tests that are performed. I use a test short hand that describes navigation/action/result in just one sentence. This enables you to create more tests further down the line.
3. Let the system risk analysis guide you to critical areas of the application. This is where exploratory testing pays off.
4. Sit near to, or with the development team to enable quick solutions to problems and questions.
5. Never rely on just doing exploratory testing.
記住: 完全沒有計畫的隨機測試, 不是exploratory Testing, 而是糟糕的測試.
- Jan 15 Thu 2009 09:44
為什麼需要做Exploratory Testing
為什麼需要做Exploratory Testing
explaining exploratory testing
http://blogs.msdn.com/james_whittaker/archive/2009/01/08/explaining-exploratory-testing.aspx
January 08, 2009
Posted by James Whittaker
Published in JW on Test
公司最近有介紹Exploratory Testing 的sharing, 在會議中發現大家幾乎都不知道這是什麼, 因此希望這篇文章可以多幫忙了解一點為何要做Exploratory Testing
軟體測試之所以複雜, 是在於它有過多可能的組合. (在我認為是接近無限多):
- inputs
- code paths to state
- operational environment.
這三種的情況, 前兩種的數量接近無限多, 第三種是很多, 所以combine起來後, 狀況幾乎是無法想像的多.
所以不管你採取什麼方法, Scripted Testing (也就是事先規劃) 或是 Exploratory Testing (一邊學, 一邊規劃, 和一邊測試), 要完整測試是不可能的事
然而Exploratory Testing的好處是, 讓QA能在測試過程中, 根據他所學到, 所收集到的, 然後來決定接下來要整麼調整, 怎麼進行接下來的測試. 而不是只是follow up之前規劃好的test cases 或是test plan.
這是最主要比Scripted Testing(泛指事先規劃的方法)佔優勢的地方.
你可以想像, 如果你若是要在季賽前, 要預測 Super Bowl (美國橄欖球超級杯大賽) or Premier League(英超聯賽)的贏家, 你會不會認為是一件非常難的事情.
因為你需要觀察team如何來運作, 進攻; 或是主力球員是否整季都健康, 沒有受傷; 或是教練和球員整合程度. 這些都是變因, 這些資訊會隨著賽季的展開, 你才能逐步調整你的預測, 然後會對的機會, 才會越來越高.
同理, 再軟體測試上也是一樣, Exploratory Testing也是在過程中, 一方面根據之前的歷史資料, 所學會的domain knowledge, 再加上目前狀況的需求, 不斷再調整所要進攻的方向, 已確定你所要測試的重點
所以有效的使用Exploratory Testing, 可以幫助你在面對軟體測試複雜的狀況, 能夠更有信心去deliver quality software.
explaining exploratory testing
http://blogs.msdn.com/james_whittaker/archive/2009/01/08/explaining-exploratory-testing.aspx
January 08, 2009
Posted by James Whittaker
Published in JW on Test
公司最近有介紹Exploratory Testing 的sharing, 在會議中發現大家幾乎都不知道這是什麼, 因此希望這篇文章可以多幫忙了解一點為何要做Exploratory Testing
軟體測試之所以複雜, 是在於它有過多可能的組合. (在我認為是接近無限多):
- inputs
- code paths to state
- operational environment.
這三種的情況, 前兩種的數量接近無限多, 第三種是很多, 所以combine起來後, 狀況幾乎是無法想像的多.
所以不管你採取什麼方法, Scripted Testing (也就是事先規劃) 或是 Exploratory Testing (一邊學, 一邊規劃, 和一邊測試), 要完整測試是不可能的事
然而Exploratory Testing的好處是, 讓QA能在測試過程中, 根據他所學到, 所收集到的, 然後來決定接下來要整麼調整, 怎麼進行接下來的測試. 而不是只是follow up之前規劃好的test cases 或是test plan.
這是最主要比Scripted Testing(泛指事先規劃的方法)佔優勢的地方.
你可以想像, 如果你若是要在季賽前, 要預測 Super Bowl (美國橄欖球超級杯大賽) or Premier League(英超聯賽)的贏家, 你會不會認為是一件非常難的事情.
因為你需要觀察team如何來運作, 進攻; 或是主力球員是否整季都健康, 沒有受傷; 或是教練和球員整合程度. 這些都是變因, 這些資訊會隨著賽季的展開, 你才能逐步調整你的預測, 然後會對的機會, 才會越來越高.
同理, 再軟體測試上也是一樣, Exploratory Testing也是在過程中, 一方面根據之前的歷史資料, 所學會的domain knowledge, 再加上目前狀況的需求, 不斷再調整所要進攻的方向, 已確定你所要測試的重點
所以有效的使用Exploratory Testing, 可以幫助你在面對軟體測試複雜的狀況, 能夠更有信心去deliver quality software.
- Jan 04 Sun 2009 15:38
Exploratory Testing常見的Q&A
Exploratory Testing常見的Q&A
Exploratory Testing challenged - Part I
http://shrinik.blogspot.com/2007/12/exploratory-testing-challenged-part-i.html
2007 Dec 29
Published in Thinking Tester
1. Exploratory Testing (ET)的好處是什麼?
Ans:
ET will extend the test coverage over traditional scripted testing, you will be able discover those bugs that are not “catchable” by scripted tests.
2. 為什麼Scripted Testing無法找到所有bugs?
Ans:
- Our scripts are based on specifications which are one narrow, fallible source of information about “intended software behavior”.
- Since specifications are written in English – there could interpretations/misinterpretations. Since our specifications are fallible, so are our scripts.
- There is a human limitation to understand and interpret specifications (objectively) and design the test cases that cover the entire test space.
- So, there is good possibility that scripts will not find all bugs that potentially be discovered.
3. 為什麼test cases為什麼被執行多次後, 越來越沒有用處呢?
Ans:
- The scripted approach to testing attempts to mechanize the test process by taking test ideas out of a test designer's head and putting them on paper.
- Test cases wear out as they do not have any in built mechanism in them to alter themselves to changing product environment.
- Test scripts can not think, infer, improvise, get frustrated as intelligent human testers do. Hence test scripts can not find bugs that repeatedly than a human tester.
- But exploratory testers take the view that writing down test scripts and following them tends to disrupt the intellectual processes that make testers able to find important problems quickly.
4. 那Ad Hoc Testing和Exploratory Testing是相同的東西嗎?
Ans:
- Yes as Adhoc testing is well known predecessor to ET. Cem Kaner coined this term ET around early 80’s to distinguish ET and Adhoc Testing.
- Cem thought that there were lots of confusions regarding some kind of “impromptu” testing that does not rely on predefined scripts.
- Ad hoc testing normally refers to a process of improvised, impromptu bug searching. By definition, anyone can do ad hoc testing.
- The term "exploratory testing"--coined by Cem Kaner, in “Testing Computer Software” -- refers to ET as a sophisticated, thoughtful approach to ad hoc testing.
5. 聽說 ET要求要有經驗, 並且Domain knowledge要高. 那average tester可以做好ET嗎?
Ans:
- I believe, ET requires skills like “questioning”, “modeling”, “critical thinking” among others.
- Domain knowledge certainly helps in ET but I do not consider it as mandatory.
6. 那ET可以找到什麼種類的bug?
Ans:
- It depends upon what kind of bugs you want to discover.
- ET can be performed in controlled, small time boxed sessions with specific charters to explore a specific feature of the application.
- ET can be configured to cater to specific investigative missions.
- You could use few ET sessions to develop a software product documentation or to analyse and isolate performance test results.
7.
當你有5000 test scripts, 你執行1到2的cycles, 你就從目前的test script再也找不出錯誤了. 你會不會覺得這系統已經沒有問題了?
或者你找到的bug都不是從這些test scripts中發現的, 你會不會覺得這test script有問題?
如果你還要在執行5 個test cycles, 你會全跑這些test script或是怎麼做呢?
Ans:
Shrini沒有提到答案, 但是已經暗示我們scripted testing是不夠的. 或許你的Testing Manager或是其他的Manager 無法被說服去採去ET, 但是你自己內心深處, 一定知道scripted testing無法解決所有問題
這時候大師出馬了, Cem Kaner 說
I rarely try to persuade a test manager to switch to exploratory testing.
Instead, I recommend to them that they include some exploratory testing in their regular testing process.
Over time, a thoughtful manager will find a context- appropriate balance between scripted and exploratory tests.
這裡他提出一個經驗法則, 告訴你要如何混搭ET 和ST
In every build, even the last build:
- spend 25% of your time trying new things. Some of these will find bugs. Some will give you good ideas for reusable tests.
- spend 25% of your time implementing new tests as reusable tests.
- spend 50% of your time running old tests (including bug regressions).
這個經驗法則有個問題: 花太多時間在做test execution, learning的時間不夠多
-Learning is a very big part of exploration, and that doesn't come just from running tests and seeing results.
- It comes from finding information about the market, the platform, the customer expectations, the risks for similar products, etc.
- This learning guides initial test design (the 25% for new things) and the design of the reusable tests.
- It also provides important guidance for writing bug reports (helping you decide, and explain) whether a bug is serious and what customer reactions might be.
不過對於一般的manager來說, 這個經驗法則的好處是: it is very easy for many managers to understand and try.
壞處則是:
- It hides the non-test-execution research that competent explorers must do -- is a serious problem.
- If you are the consultant, then as your client gains experience with exploratory work, you can raise higher level cognitive issues and the value of preparatory research gradually, as examples present themselves in the situation of your client.
Exploratory Testing challenged - Part I
http://shrinik.blogspot.com/2007/12/exploratory-testing-challenged-part-i.html
2007 Dec 29
Published in Thinking Tester
1. Exploratory Testing (ET)的好處是什麼?
Ans:
ET will extend the test coverage over traditional scripted testing, you will be able discover those bugs that are not “catchable” by scripted tests.
2. 為什麼Scripted Testing無法找到所有bugs?
Ans:
- Our scripts are based on specifications which are one narrow, fallible source of information about “intended software behavior”.
- Since specifications are written in English – there could interpretations/misinterpretations. Since our specifications are fallible, so are our scripts.
- There is a human limitation to understand and interpret specifications (objectively) and design the test cases that cover the entire test space.
- So, there is good possibility that scripts will not find all bugs that potentially be discovered.
3. 為什麼test cases為什麼被執行多次後, 越來越沒有用處呢?
Ans:
- The scripted approach to testing attempts to mechanize the test process by taking test ideas out of a test designer's head and putting them on paper.
- Test cases wear out as they do not have any in built mechanism in them to alter themselves to changing product environment.
- Test scripts can not think, infer, improvise, get frustrated as intelligent human testers do. Hence test scripts can not find bugs that repeatedly than a human tester.
- But exploratory testers take the view that writing down test scripts and following them tends to disrupt the intellectual processes that make testers able to find important problems quickly.
4. 那Ad Hoc Testing和Exploratory Testing是相同的東西嗎?
Ans:
- Yes as Adhoc testing is well known predecessor to ET. Cem Kaner coined this term ET around early 80’s to distinguish ET and Adhoc Testing.
- Cem thought that there were lots of confusions regarding some kind of “impromptu” testing that does not rely on predefined scripts.
- Ad hoc testing normally refers to a process of improvised, impromptu bug searching. By definition, anyone can do ad hoc testing.
- The term "exploratory testing"--coined by Cem Kaner, in “Testing Computer Software” -- refers to ET as a sophisticated, thoughtful approach to ad hoc testing.
5. 聽說 ET要求要有經驗, 並且Domain knowledge要高. 那average tester可以做好ET嗎?
Ans:
- I believe, ET requires skills like “questioning”, “modeling”, “critical thinking” among others.
- Domain knowledge certainly helps in ET but I do not consider it as mandatory.
6. 那ET可以找到什麼種類的bug?
Ans:
- It depends upon what kind of bugs you want to discover.
- ET can be performed in controlled, small time boxed sessions with specific charters to explore a specific feature of the application.
- ET can be configured to cater to specific investigative missions.
- You could use few ET sessions to develop a software product documentation or to analyse and isolate performance test results.
7.
當你有5000 test scripts, 你執行1到2的cycles, 你就從目前的test script再也找不出錯誤了. 你會不會覺得這系統已經沒有問題了?
或者你找到的bug都不是從這些test scripts中發現的, 你會不會覺得這test script有問題?
如果你還要在執行5 個test cycles, 你會全跑這些test script或是怎麼做呢?
Ans:
Shrini沒有提到答案, 但是已經暗示我們scripted testing是不夠的. 或許你的Testing Manager或是其他的Manager 無法被說服去採去ET, 但是你自己內心深處, 一定知道scripted testing無法解決所有問題
這時候大師出馬了, Cem Kaner 說
I rarely try to persuade a test manager to switch to exploratory testing.
Instead, I recommend to them that they include some exploratory testing in their regular testing process.
Over time, a thoughtful manager will find a context- appropriate balance between scripted and exploratory tests.
這裡他提出一個經驗法則, 告訴你要如何混搭ET 和ST
In every build, even the last build:
- spend 25% of your time trying new things. Some of these will find bugs. Some will give you good ideas for reusable tests.
- spend 25% of your time implementing new tests as reusable tests.
- spend 50% of your time running old tests (including bug regressions).
這個經驗法則有個問題: 花太多時間在做test execution, learning的時間不夠多
-Learning is a very big part of exploration, and that doesn't come just from running tests and seeing results.
- It comes from finding information about the market, the platform, the customer expectations, the risks for similar products, etc.
- This learning guides initial test design (the 25% for new things) and the design of the reusable tests.
- It also provides important guidance for writing bug reports (helping you decide, and explain) whether a bug is serious and what customer reactions might be.
不過對於一般的manager來說, 這個經驗法則的好處是: it is very easy for many managers to understand and try.
壞處則是:
- It hides the non-test-execution research that competent explorers must do -- is a serious problem.
- If you are the consultant, then as your client gains experience with exploratory work, you can raise higher level cognitive issues and the value of preparatory research gradually, as examples present themselves in the situation of your client.
- Nov 19 Wed 2008 09:30
如何讓 Exploratory Testing是可以管理的
如何讓 Exploratory Testing是可以管理的
雖然ET並不等於unstructed或是unplanned, 但是對於test manager來說, 它還是不好管理和控制. 因此有人就發展出一套對ET的管理方法, 讓你可以掌握ET的狀況, 讓相關的stakeholder可以清楚了解我們在做什麼.
這個流程是由以下幾各單元所組成的
雖然ET並不等於unstructed或是unplanned, 但是對於test manager來說, 它還是不好管理和控制. 因此有人就發展出一套對ET的管理方法, 讓你可以掌握ET的狀況, 讓相關的stakeholder可以清楚了解我們在做什麼.
這個流程是由以下幾各單元所組成的
- Nov 18 Tue 2008 22:46
Exploratory Testing是否只適用於senior testers的身上?
Exploratory Testing是否只適用於senior testers的身上?
Is exploratory testing only for senior testers?
September 16th, 2008 in SearchSoftwareQuality
http://www.michaeldkelly.com/blog/archives/190
上一篇"Exploratory Testing的優缺點及適用時機"
http://www.wretch.cc/blog/kojenchieh/13053436#comment249421680
- Nov 11 Tue 2008 08:00
Exploratory Testing的優缺點及適用時機
Exploratory Testing的優缺點及適用時機
from Exploratory Testing: Evolution or Revolution?, SVEN SAMBAER
http://www.ctg.com/pdf/exploratory-testing.pdf
Exploratory Testing的優點
(1) 它鼓勵創造性
(2) 它增加機會去找到新的, 未知的bugs
