目前分類:測試自動化 (62)

瀏覽方式: 標題列表 簡短摘要
Can Test Automation Tools Replace The Human Testers?
http://software-testing-zone.blogspot.com/2008/10/automation-tools-replace-manual-testers.html

Oct 22th, 2008
Posted by Debasis
Published in Software Testing Zone

kojenchieh 發表在 痞客邦 留言(2) 人氣()

Agile Testing tools List
http://weblogs.asp.net/rosherove/archive/2008/10/03/agile-testing-tools-list.aspx

這裡有一堆在做unit testing, 或test automation時可以用的testing framework, 大家看看吧.
有些我能找到source, 我都已經把URL列出來. 其他的可能要你自己找找.

A.  Mock Frameworks
(1) Moq
    - Moq (pronounced "Mock-you") is the only mocking library for .NET developed from scratch to take full advantage of .NET 3.5 (i.e. Linq expression trees) and C# 3.0 features (i.e. lambda expressions) that make it the most productive, simple and refactoring-friendly mocking library available
    - http://code.google.com/p/moq/
(2) Rhino Mocks
    - A dynamic mock object framework for the .Net platform. It's purpose is to ease testing by allowing the developer to create mock implementations of custom objects and verify the interactions using unit testing.
    - http://ayende.com/projects/rhino-mocks.aspx
(3) Typemock Isolator
- The Typemock Isolator mocking framework for .NET unit testing, allows you to focus your energy on developing quality code.
- http://www.typemock.com/
(4) NMock
    - NMock is a dynamic mock object library for .NET. Mock objects make it easier to test single components - often single classes - without relying on real implementations of all of the other components.
    - http://www.nmock.org/
(5) NUnit.Mocks
    - http://www.nunit.org/index.php
    
B. Test Frameworks
(1) MSTest
- MSTest is a software unit testing framework developed by Microsoft, which integrates closely with Visual Studio. MSTest lets you create, manage and run unit tests from within the Visual Studio IDE, as well as externally, from the command line.
- http://msdn.microsoft.com/en-us/library/ms182489(VS.80).aspx
(2) NUnit
- http://www.nunit.org/index.php
(3) MbUnit
- MbUnit provides advanced unit testing support with advanced fixtures to enable developers and testers to test all aspects of their software.
- http://www.mbunit.com/
(4) XUnit
- http://www.codeplex.com/xunit
(5) Gallio
http://www.gallio.org/

C. Dependency Injection and Ioc Containers
(1) StructureMap
    - StructureMap is a Dependency Injection framework that can be used to improve the architectural qualities of an object oriented system by reducing the mechanical costs of good design techniques. StructureMap can enable looser coupling between classes and their dependencies, improve the testability of a class structure, and provide generic flexibility mechanisms. Used judiciously, StructureMap can greatly enhance the opportunities for code reuse by minimizing direct coupling between classes and configuration mechanisms.
    - http://structuremap.sourceforge.net/Default.htm
(2) Microsoft Unity
    - The Unity Application Block (Unity) is a lightweight, extensible dependency injection container.
    - http://msdn.microsoft.com/en-us/library/cc468366.aspx
(3) Castle Windsor
(4) Common IServiceLocator Framework
(5) Managed Extensibility Framework
    - The Managed Extensibility Framework (MEF) is a new library in .NET that enables greater reuse of applications and components. Using MEF, .NET applications can make the shift from being statically compiled to dynamically composed. If you are building extensible applications, extensible frameworks and application extensions, then MEF is for you.
    - http://code.msdn.microsoft.com/mef
(6) Spring.NET
    - Spring.NET is an open source application framework that makes building  enterprise .NET applications easier.
    - http://springframework.net/
(7) AutoFac
    - Autofac was designed with modern .NET features and obsessive object-orientation in mind. It will change the way you approach dependency injection in .NET.
    - http://code.google.com/p/autofac/
    
D. Database testing
(1) Use Integration testing
(2) MbUnit,NUnit,XUnit Rollback attributes
(3) TransactionScope
    - http://msdn.microsoft.com/zh-tw/library/system.transactions.transactionscope(VS.80).aspx
        
E. Web Testing
(1) Ivonna and Typemock
(2) Team System Web Test
    - The Visual Studio 2005 Team System enables Web Testing which makes it easy for testing your website without writing any code. When we say web testing, the Visual Studio 2005 Team System Web testing does not do any user interface testing; instead the tests are done against the data that is flowing through the HTTP. This data is examined and validated with various rules in place.
    - http://msdn.microsoft.com/en-us/library/ms364077(VS.80).aspx
(3) NUnitASP
    - NUnitAsp is a tool for automatically testing ASP.NET web pages. It's an extension to NUnit, a tool for test-driven development in .NET.
    - http://nunitasp.sourceforge.net/
(4) Watin
    - Inspired by Watir  development of WatiN started in December 2005 to make a similar kind of Web Application Testing possible for the .Net languages.
    - http://watin.sourceforge.net/
(5) Watir
    - Watir is an open-source library for automating web browsers. It allows you to write tests that are easy to read and maintain. It is simple and flexible.
    - http://wtr.rubyforge.org/
(6) Selenium
    - Selenium is a suite of tools to automate web app testing across many platforms.
    - http://selenium.seleniumhq.org/
    
F. UI Testing
(1) NUnitForms
    - NUnitForms is an NUnit extension for unit and acceptance testing of Windows Forms applications.
    - http://nunitforms.sourceforge.net/
(2) Project White
    - White supports all rich client applications, which are Win32, WinForm, WPF and SWT (java).
It is .NET based and hence you wouldn't have use proprietary scripting language. You can use your favourite .NET language, IDE and tools for developing tests/automation programs.
    White provides consistent object oriented API for all kinds of applications. Also it hides all the complexity of Microsoft's UIAutomation library and windows messages (on which it is based).
    - http://www.codeplex.com/white
(3) Team System UI Tests
(4) Typemock Isolator
    
G. Threading Related Testing
(1) Microsoft Chess
    - CHESS is an automated tool for finding errors in multithreaded software by systematic exploration of thread schedules.
    - http://research.microsoft.com/chess/
(2) Typemock Racer
    - Typemock Racer helps find possible deadlocks in production code by allowing the developer to write threaded tests that flesh out possible problems.
    - Racer provides the developer with a framework API which can be used in Parallel Computing tests (under NUnit for example), that allows the code to be tested for deadlocks in multithreaded code.
    - http://www.typemock.com/learn_about_typemock_racer.html
(3) Osherove.ThreadTester
    - The ThreadTester library helps create tests that use multiple threads. It’s main use is to synchronize and “block” the running test until all the threads have finished their job, or a timeout has occurred. It is designed to make the test developer's life easier with a simple API that is readable and quick to use.
    - http://weblogs.asp.net/rosherove/archive/2007/06/22/multi-threaded-unit-tests-with-osherove-threadtester.aspx
    
H. General Testing
(1) Pex
    - Automated White Box Testing for .NET
    - http://research.microsoft.com/Pex/

I. Acceptance Testing
(1) Fit & Fitnesse
- The fully integrated standalone wiki, and acceptance testing framework
- http://fitnesse.org/
(2) Watin and Watir and selenium
    a. Watin
        - Inspired by Watir  development of WatiN started in December 2005 to make a similar kind of Web Application Testing possible for the .Net languages.
        - http://watin.sourceforge.net/
    b. Watir
        - Watir is an open-source library for automating web browsers. It allows you to write tests that are easy to read and maintain. It is simple and flexible.
        - http://wtr.rubyforge.org/
    c. Selenium
        - Selenium is a suite of tools to automate web app testing across many platforms.
        - http://selenium.seleniumhq.org/


kojenchieh 發表在 痞客邦 留言(0) 人氣()

Test Code必須和Dev Code 的開發一樣嚴謹

Test Code Must Be As Solid As Dev Code
http://blogs.msdn.com/steverowe/archive/2008/08/07/test-code-must-be-as-solid-as-dev-code.aspx

Test Code要能有用, 必須要和開發Dev Code一樣, 要有相同的process和警慎的心態. 否則Test Code將會充滿bug, 無法經得起長時間使用.

最近公司有一份survey 公司和Microsoft test automation的比較. 發現到Microsoft真的很嚴謹在看待test automation:
(1) 認為auto test implementation process 和 development process一樣重要
(2) 有專門的 test automation team.
(3) 要求QA 和RD要加入
(4) test code 也有明確 development phrase
(5) testing script也需要做code review
(6) 有大量的輔助系統來管理和deploy test automation system (Bug tracking system, source control system, log praser, build deployment system, test data management system...)

當下是覺得他們是下很大的工夫, 在落實test automation的進行. 做的是比我們還好很多. 我們只是有, 但是分工還是沒有很細緻, 還是停留在土法大煉鋼的時代. 至少我認為有以下缺點
1. 只靠QA 或少部分的test developer在做開發
2. RD很少加入討論或是提供現成的工具或程式, testability的修改更是少數
3. 並沒有採用像dev process一樣的輔助工具: bug tracking syste, log parser, build deployment, source control system. 即使有用, 但是也不嚴謹. 總是自己認為是Test Code, 心態上就隨便許多
4. 沒有套用code review, design review, architecture design, unit test等這些pratices, 到Test Code身上.
5. 更不用講project management
......
反正只要能交的出來就好, 怎麼做不是重點.

後來在網路上看到"Test Code Must Be As Solid As Dev Code"這篇時, 發現 Microsoft的員工認為他們還是有進步的空間, 他們也認為他們還是沒有像dev code那樣的嚴謹. 他是列出一些Test Code常見的問題,
1. Flaky test code can make determining the source of an error difficult.  
- Tests that cannot be trusted make it hard to convince developers to fix issues.  
- No one wants to believe there is a bug in their code so a flaky test becomes an easy scapegoat.

2. Spurious failures take time to triage.  
- Test cases that fall over because they are unstable will take a lot of time to maintain.  
- This is time that cannot be spent writing new test automation or testing new corners of the product.

3. Poorly written test code will hide bugs in the product.  
- If test code crashes, any bugs  in the product after that point will be missed.  
- Similarly, poor test code may not execute as expected..  

不過比較好玩的部份, 是後來有人問了問題:
It should be done but sounds like a resource nightmare - "hey boss, we need some testers to test the test code..."
How close are you to getting this to happen at MS ? Any problems in
a) getting a budget for this ?
b) finding enough testers capable of doing it ?

我想resource 這是很多公司很多人都考慮的問題, resource不夠是無法做事的.不過我要講的是, 這也要看公司重不重視. 像我們公司RD 和QA 的比例是1:1, 對台灣很多公司來說, 是有點天方夜譚. 同樣地, 當我們公司在看Microsoft對test automation的作法時, 我們也是會覺得有點天方夜譚. 只能說, 要做到那樣不是不可能, 在於你重不重視.





kojenchieh 發表在 痞客邦 留言(0) 人氣()

要將Test Automation外包的十誡

10 Commandments for Test Automation Outsourcing
http://shrinik.blogspot.com/2008/09/10-commandments-for-automation.html

當你要將test automation外包時, 要記得考慮下面十件事情:

1. Test Automation不能解決你測試上遇到的難題
- 像是測試時間不夠, 受測程式的品質不好等等這些問題, 是無法靠它解決的. 唯一能做的還是要改進你的manual testing, 而不是去想要加強test automation

kojenchieh 發表在 痞客邦 留言(1) 人氣()

A catalogue of Test Automation Benefits ...
http://shrinik.blogspot.com/2008/06/catalogue-of-test-automation-benefits.html

有天有人問作者 test automation的好處是什麼, 因此引發他想寫篇文章在探討這個議題. 它分成真的好處, 假的好處, 以及有條件的好處, 完全鬼扯的等四種狀況來敘述.

Real: 就作者的觀點, 他認為test automation是真的可以達到這些好處. (不過我覺得有些好處好像講的好酸的樣子)
- Consistent and Accurate Test results (nearly free from human errors) – when accuracy and accurate results are important? – Numerical calculations
- Untiring and can be run for long hours without any loss of efficiency

kojenchieh 發表在 痞客邦 留言(0) 人氣()

GUI Automation真的值得嗎? Part III


A layer below the GUI
http://blogs.msdn.com/alanpa/archive/2008/09/20/a-layer-below-the-gui.aspx

接續上一篇Part II, 作者發憤圖強寫了一些例子來解釋他的本意, 這裡的說明和我在

測試程式架構的重要性: 看Commerical UI automation tool的演進

kojenchieh 發表在 痞客邦 留言(1) 人氣()

GUI Automation真的值得嗎? Part II


在post 完"GUI Automation 真的值得嗎?"後,公司同事問我為何我們公司大部分的人, 都是還在做GUI test automation. 那時候我回答的答案是
(1)   QA is not familiar with RD’s codes.

kojenchieh 發表在 痞客邦 留言(0) 人氣()

檢查這人有沒有test automation經驗的好問題

在interview 要做test developer 時, 這裡有一些問題, 可以用來檢查他的經驗如何. 我想這修問題還蠻全面性, 你有多少斤兩, 一下應該就可以看的出來.

至於作者寫的答案, 我想參考就好, 有些部份我並不完全相同, 不過可以用它來了解他問題的用意為何, 幫助也是蠻大的.

可惜這些問題大多只能問國外的candidate或是國外回來的, 國內的可能大都是回答不知道!!!

Question about Automation
by Jaanu

kojenchieh 發表在 痞客邦 留言(0) 人氣()

聽聽微軟員工說: 如何決定何時做Test Automation?


這篇文章列出automation testing和manual testing的優缺點, 並且有些簡單的評量標準, 來判斷是否要automate. 我想這應該是一個很好的起點, 讓你base這份rules去建立你的rules


kojenchieh 發表在 痞客邦 留言(0) 人氣()

Bug Number 和Test Automation Ratio


Motley says: "More test automation is always better"
http://blogs.msdn.com/progressive_development/archive/2008/08/19/motley-says-more-test-automation-is-always-better.aspx

最近逛到這個blog, 發現他的文章十分好玩, 會用一些對話來說明他要闡述的道理, 以講故事的方式來進行, 讓人看起來很有趣

在這篇文章中, 他提到兩個重點

kojenchieh 發表在 痞客邦 留言(1) 人氣()

Google C++ Testing Framework

最近有位芭樂老兄, 硬是要我去看看Google C++ Testing Framework. 他說這套軟體最近很紅, 公司內部很多員工都在討論它. 說它很厲害, 可以增加生產力. 於是小弟只好勉為其難, 上網找了一下資料, 可是內心不免小小幹糕一下, 想說是不是又是另一個銀製子彈.

它的首頁在這裡
http://code.google.com/p/googletest/

不過你可以在這裡找到快速入門的介紹
http://code.google.com/p/googletest/wiki/GoogleTestPrimer

kojenchieh 發表在 痞客邦 留言(0) 人氣()

測試自動化的策略

What's your test automation strategy?
http://blogs.msdn.com/imtesty/archive/2006/05/08/593395.aspx

通常我們都誤解test automation要越高越好, 100%自動化是最佳狀態. 但是不管做什麼事, 投資報酬率是最重要的考量. 千萬不要是為了做automation而automation. 因此每次在test automation, 策略是一件很重要的事, 要決定什麼該做, 什麼不該做, 我們的方向是什麼.

作者首先提出什麼是 test automation的strategy:

 A practical test automation strategy is one that

kojenchieh 發表在 痞客邦 留言(0) 人氣()

小心BVT的陷阱

Escape the BVT automation trap
http://blogs.msdn.com/dustin_andrews/archive/2007/11/16/escape-the-bvt-automation-trap.aspx

這是另一個微軟的員工提出對BVT的建議. 不外乎就是要確定BVT要快, 要真的有用, 能真的檢查出這各build是否ready for testing.

1. BVT automation is a tool.
Just as a hammer is a great tool for nail, BVT automation is a great tool for a specific purpose. It’s not a catch all place to dump all the automation you have ever created for your product.

kojenchieh 發表在 痞客邦 留言(1) 人氣()

Build Verification Test在做些什麼?


What Tests Belong in the BVTs?
http://blogs.msdn.com/steverowe/archive/2008/03/05/what-tests-belong-in-the-bvts.aspx

之前我提到在開測試個案時, 要分成不同的purpose之後, 在一個個針對它做處理.所以現在我最痛苦的工作就是在定義那些purpose, 要有明確的定義才能知道我們鎖定的範圍為何.

kojenchieh 發表在 痞客邦 留言(0) 人氣()

聽聽微軟員工說: Why we automate


Why we automate
http://blogs.msdn.com/imtesty/archive/2007/08/30/why-we-automate.aspx


* Increasingly complex test matrices.

kojenchieh 發表在 痞客邦 留言(0) 人氣()

SDET在做什麼?

What does an SDET do?
http://aliabdin.wordpress.com/2007/03/31/what-does-an-sdet-do/

所謂SDET 就是Software Development Engineer in Test, 這是微軟的叫法. 在我們公司叫做 Test Developer.

作者認為SDET的工作和Software Test Engineers(STE)不一樣, (STE也就是一般的軟體測試工程師, 或是叫做QA). 他並不是指有著重在測試而已, 他還有別的事情要處理. 因此這篇文章列出哪些事情是他認為SDET特別要做的

kojenchieh 發表在 痞客邦 留言(0) 人氣()

Ruby適合當test scripting language嗎?


Disadvantages of Ruby for Test Automation
http://blogs.msdn.com/imtesty/archive/2006/06/08/621755.aspx

Test Automation with Ruby: Don't drink the kool-aid
http://blogs.msdn.com/imtesty/archive/2006/08/25/720782.aspx

kojenchieh 發表在 痞客邦 留言(0) 人氣()

GUI Automation 真的值得嗎?


GUI Automation and ROI
http://blogs.msdn.com/imtesty/archive/2008/03/28/gui-automation-and-roi.aspx


kojenchieh 發表在 痞客邦 留言(0) 人氣()

測試程式架構的重要性: 看Commerical UI automation tool的演進


Mercury Business Process Testing:專注於業務需求的自動化測試
http://blog.joycode.com/oldsidney/articles/86763.aspx

當我在和同事談test automation時, 提到QA在做test automation時, 常常沒有 architecture design的觀念. 每次程式一有做修正或是需求改變時, 測試程式常常會改到翻掉.

像是在做UI測試時, 一開始的script都是直接描述UI的動作, 和UI的control 綁的很緊, 可以參照文章中第二代的自動化測試的script

kojenchieh 發表在 痞客邦 留言(1) 人氣()

有關軟體測試自動化的書籍

目前在Amazon上有關測試自動化的書不多, 大部分都有點時間了. 只有一本是今年出的, 不過要100美金, 真是買不下手. 再加上主題缺乏吸引力, 我應該不會用自己的錢買吧

1. Software Test Automation (ACM Press)
by Mark Fewster (Author), Dorothy Graham (Author)
http://www.amazon.com/Software-Test-Automation-ACM-Press/dp/0201331403/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1222000288&sr=8-1

2. Just Enough Software Test Automation (Yourdon Press Series)

kojenchieh 發表在 痞客邦 留言(0) 人氣()

Close

您尚未登入,將以訪客身份留言。亦可以上方服務帳號登入留言

請輸入暱稱 ( 最多顯示 6 個中文字元 )

請輸入標題 ( 最多顯示 9 個中文字元 )

請輸入內容 ( 最多 140 個中文字元 )

reload

請輸入左方認證碼:

看不懂,換張圖

請輸入驗證碼