目前分類:Agile Testing (41)

瀏覽方式: 標題列表 簡短摘要
在Scrum中如何進行測試?

Scrum - Two Levels of Test?

http://insidesqa.blogspot.com/2008/11/scrum-two-levels-of-test.html

23 November 2008
Posted by Peter Marshall
Published in Inside QA

kojenchieh 發表在 痞客邦 留言(0) 人氣()

Agile Testing 的策略
http://insidesqa.blogspot.com/2009/02/agile-test-strategy.html

作者過去曾經設計很heavy的testing strategy, 並且試圖讓人知道為什麼要這麼做, 以及要做些什麼事

但是他也發現, 不管先前做這些事的意圖多麼正當, 可是在執行的過程中常常無法達到那樣的效果

所以他在最近一次的project中, 他改良出一版lean testing strategy. 他認為這次是比較有用, 並且比較實務

所以就讓我們來看看, 作者改良後的testing strategy為何:


Phase: Project Set Up
    * Understand the project
    * Collect information about the project
    * Create a test knowledge repository

Phase: Planning & Analysis or Release Planning
    * Assist in the definition and scope of stories
    * Develop test plans based on planning session

Phase: Development Iterations
    * Risk analysis during the sprint/iteration planning
    * Construct acceptance tests for each story
    * Develop business functionality validation plan
    * Document and write tests for defects
    * Automate with both unit and UI tests
    * Assist in functional review/demo
    * Accept User stories

Phase: Hardening Iterations
    * Regression test
    * Business acceptance testing
    * Develop release readiness plan
    * Run performance tests

Phase: Release
    * Assist in release readiness
    * Plan test release data and tests
    * Accept the Release

作者認為根據這個strategy, 它可以deliver經得起考驗的軟體, 但是又不會有像傳統測試一樣, 需要有大量文件撰寫的負擔

不知過位看倌你覺得如何?

kojenchieh 發表在 痞客邦 留言(0) 人氣()

Scrum + Manual Testing methodology

Scrum + Manual Testing methodology (for WEB apps.)
http://www.testingreflections.com/node/view/7378

作者認為在run Scrum時, 對於testing經常會有以下兩種錯誤的作法
- to give up manual testing skills and learn automation or
- give up agile principle of quick feedback and test one iteration behind.

他認為應該在Scrum中加入一個role叫做Tester (而不是只有team, scrum master和project owner). 這樣才會讓Scrum更完整. 可是目前他並沒有看到任何研究, 是有關這方面的. 因此他會致力於這方面的研究, 如何有興趣的人可以和他一起討論.(作者的mail: ojnjars@inbox.lv)

以下是他方法的一些前提假設(Context – Assumptions)
1. Development Scrum consists of 2 week sprints

2. The software is a web based with certain data in database.
- The role of tester will include testing software in “QA environment”, while developers are still doing unit tests in their environment which is not as close to production one.

3. It is decided that software deployment in “QA environment” should follow the manual installation process the same as it will take place in production environment.
- It optionally needs smart decision if to redeploy data into database.
- The process may take up to 30 minutes and is never done this way by developers in their environment

4. The process of creating installation package (build process) may also take up to 30 minutes, so will be executed once per night (nightly build)

接下來, 他描述在一個2 weeks 的sprint中, QA的活動在做些什麼:
Day 1. Sprint Planning
During the first day of a spring – sprint planning meeting – tester participates in order to help to define acceptance criteria (by asking what if questions to product owner) and estimating testing effort for stories.
- It may appear that there are stories requiring a little development effort but a lot of testing effort.
- Story selection may be changed to disallow too much stories of such type into one sprint.
- Alternatively part of testing activities may be postponed, creating a separate story (adding what I would call “quality debt”) and adding to backlog.

Day 2. Test Case Drafting
- A tester works: one day behind” the team.
- The second day of a sprint is dedicated by tester to draft initial set of test cases for each features included in the sprint.

Day 3-8. Normal days
(1) 基本作法
- Each day starting day 3 of sprint in the morning tester installs latest nightly build (manually using installation procedure/instructions supplied by development team) to QA environment, do fast smoke tests if they fail and emergency fixes are applied by the team.
- Features developed yesterday shall be tested today by tester.
- As required tester communicates with each developer to transition the knowledge of the feature functionality.
- Tester must also update (complete) Test Cases in the process of testing as more test ideas may come out during Exploratory Testing.

(2) 如果測試無法進行
- If a feature testing requires too much effort or tester discovers too much ideas and can’t accomplish testing within the day he create new story ticket for further feature investigation.
- If the ticket is not closed by the end of a sprint it appears to be “Quality Debt” and must be announced to product owner for prioritization.
- Because developers do unit testing most of the features should pass acceptance criteria.
- However if for any reason they do not pass them in QA environment, QA may return ticket to the team.
- If acceptance criteria are met but there are more bugs around, tester reports them separately, sending the initial story to End-To-End testing.

(3) 如果bug無法處理時
- Tester also has to do wider investigation (based on project quality goals) and report any usability suggestions, performance, security or other potential treats.
- Such bugs must be assigned to Product Owner if they don’t directly break the acceptance criteria of a story.
- Bugs that (developer) team disagrees with or appears to require too much effort also get assigned to Product Owner.
- They have to be addressed during sprint retrospective meeting.
- So it appears that during sprint retrospective no feature could be demonstrated if it has an open bug against it assigned to anyone but product owner.

Day 9 – The QA day
- Because tester works one day behind, the day 8 of the sprint shall be the last day of a new feature development.
- That is the reason why day 9 can be dedicated by team to bug fixing, re-factoring and analysis of new features which might be included into next sprint i.e. next sprint preparation.

Day 10 - retrospective meeting
- In order to demonstrate a feature in retrospective meeting it should be implemented; tested and all bugs either fixed or assigned to Product Owner for clarification (otherwise feature is postponed to next sprint).
- Tester should demonstrate that features meet acceptance criteria and demonstrate the issues assigned to product owner if any.
- Product Owner should decide and prioritize those features/defects.
- The team participates in the meeting to qualify their performance and probably counter tester’s negative opinion.
- Tester also describes “Quality Debt” changes – what was not tested due to lack of time.
- Product owner may prioritize the postponed tests or even allow skipping them.

在Sprint結束後, QA還有很多事要做
- Test Cases created during sprint shall be peer-reviewed by another tester at any time later. For example by End-To-End tester if there are any.
- Ideally review should be done during or by end of the day 10 so that tester has the feedback by next planning meeting, but it may be impossible.
- In any case tester is not forced to respond to review feedback (fix test cases and run additional test if identified) until the end of sprint (so that tester commitment during sprint planning is not compromised).
- If more tests are identified during review they must be executed later and any defects discovered should be assigned to product owner (added to backlog).


不包含在Sprint中的測試
(1) Non-function的測試
- Still there are types of tests that are not directly related to functionality implemented by sprints.
- For example non-functional tests: to validate that given architecture satisfies high availability requirements.
- Actually the tester may not have enough skills to do performance tests and a separate team or person may be assigned to do performance testing based on agreed schedule.

(2) Regression Testing
- More over it is highly welcome to develop system-level automated regression tests (based on, but not necessarily repeating manual test cases) along with developer created unit tests.
- Those tests will aim for different type of regression bugs.
- This activity may also take place with a significant delay to feature implementation.
- The tester should make a smart decision to nominate the build that was especially successful (without critical bugs) to be installed to the special environment such as for performance test or test automation environment.

kojenchieh 發表在 痞客邦 留言(0) 人氣()

在Agile中如何進行測試的工作 (2)


QA and Testing in an Agile Environment
Chris, 05Nov
http://edgehopper.com/qatesting-in-an-agile-environment/

* Tim Lesher Says:

Tim認為前面那些agile testing 的作法, 在iteration最後階段會造成, QA會很忙, RD會很輕鬆. 因此為了避免學生症候群Parkinson’s Law (where the work expands to fill the remaining iteration time), Tim提出了一些修正:

1. Dev designs and codes; SQA designs tests and pokes holes in the design
2. SQA executes tests; Dev addresses issues and pokes holes in the tests
3. SQA executes acceptance tests and writes defects; Dev automates tests

    The acceptance-level and integration-level tests that SQA is running during iteration N get automated by development, and take effect as part of the automated suite beginning in interation N+1.

目前這個作法仍持續在觀察中, 但是看起來還work

===============================================

* Gunther Verheyen Says:

Cunther在每天的工作中, 加入以下四項quality loops: (記住是在每天中做, 而不是到sprint的末期才做)

Loop 1.
    - Developers work in pairs upon a test-first basis. By the way, a test-first also includes some GUI-testing
    - For example, using Selenium: is a suite of tools to automate web app testing across many platforms.

Loop 2.
    - Developers can refactor, i.e. rewrite working code to make it better, simpler, remove duplicate code etc.
    - When a test-first no longer fails, the code is checked in and gets picked up by a Continuous Integration system. The CI runs multiple times a day.
    - Logs are being checked regularly and rework is inserted into the team using a Kanban board.

Loop 3.
    - The guide deploys a stable version from the CI to perform the functional testing.
    - Functional feedback is inserted into the team using a Kanban board (items are generally too small to make it a Story).

Loop 4.
    - (if applicabel) Overnight performance tests are performed on a deployed version using scripts with ramp-up’s, think time, etc.

Cunther覺得他們這樣做了以後, 基本上系統沒有在production, 或是sprint review中, 被發現大的bugs或是problems

===============================================

* Surendra Lingareddy Says:

Agile 所帶來的好處: The tendency to natural juxtapose QA and Dev.
- The planning meeting serves to select the stories for this iteration as well as to flush their details out.
- QA and Dev both along with the business (Prod Mgr and SCMs at least) walk out with the same understanding of what the story is.

===============================================

* Patrick Curtain Says:

主要做法: QA following one iteration behind
- Focus on black box test automation and maintenance of tests affected by change.
- That puts the focus (and personnel budgeting) on TDD in the dev team, continuous integration and flowing QA effort back into the team.

===============================================

* Bob Galen Says:

      
對Chirs前面的提作法有疑問
(1)  Isn’t setting up some sort of intra-Sprint ’schedule’ sort of defeating the purpose of self-direction?
因為他認為以下事情比較重要
(1) The team buy-into the notion of getting things ‘done’ and
(2) that they need to work efficiently and effectively to maximize how much they deliver and the quality of what they deliver–so the team should be figuring this out–sort of on their own.
(3) Another part of that is significantly blending the individual roles–so that it’s the team working and not developers handing over things to testers.

所以Bob認為前面這些作法只是限制了sprint的精神, 感覺裡面還是存在著waterfall的骨子. 所以他認為是怪怪的.

Bob建議使用 kanban來幫助會比較好一點
- The whole notion seems to be to throttle the work (cards) that a team can take on to try and reinforce blended team behavior to drive work to done.
- So fewer in-process tasks/features, and heavy collaboration to get it done as quickly as possible. It seems that this leads to (the claims) to higher efficiency / effectiveness.
( notes: kanban是源自於Toyota的看板文化, agile中有使用這個觀念. 不過老實說我不是很清楚Bob指的是什麼, 在網路上我還找到他所講的, 下次找到在share給大家)

===============================================

* Chris Says:

@Bob Galen:
這是Chris對 Bob意見的回應
- This is merely a suggestion as a way for teams to think about organizing their work to maximize the effectiveness and utilization of both the QA’s and the devs.
- Teams can self-organize and self-direct, and that is a real key to agile success.
- However, having some guidance or idea of how to attack the work of an iteration isn’t necessarily a bad idea.

@Patrick Curtain:
如果使用one iteration behind的情況要小心. 根據我的經驗, 一個user story 要到被測試完成才較被做完(done), 也就是到那個時候你才算你拿到那個story point , 也就是說你要到下一個iteration才算是completed or done.

這將會嚴重影響你team的velocity. 更重要的是user story為測試完, 這代表它可能是不能shippable, 因此對你的customer來說是沒有value的.


kojenchieh 發表在 痞客邦 留言(0) 人氣()

在Agile中如何進行測試的工作 (1)


QA and Testing in an Agile Environment
Chris, 05Nov
http://edgehopper.com/qatesting-in-an-agile-environment/

當作者要adopt 2 weeks的iteration時, 他問了很多人對於這樣的事, 有怎樣的concern. 以下是他得到的主要考量:
(1) QA’s and testers on an agile team have nothing to do at the start of an iteration.
(2) We can’t keep writing new code up until the last minute of an iteration if QA is to adequately test the code. So, what do the developers do at the end of the iteration?

因此作者提出它的作法, 希望能mitigate這些問題, 並且能讓QA在agile iteration中融入的很好.
1. 第一週
Day 1
    Dev: Coding
    QA: Write Tests
Day 2
    Dev: Coding
    QA: Write Tests
Day 3
    Dev: Coding/Defect Fixes
    QA: QA/Testing
Day 4
    Dev: Coding/Defect Fixes
    QA: QA/Testing
Day 5
    Dev: Coding/Defect Fixes
    QA: QA/Testing

2. 第二週
Day 1
    Dev: Coding/Defect Fixes
    QA: QA/Testing
Day 2
    Dev: Coding/Defect Fixes
    QA: QA/Testing
Day 3
    Dev: Defect Fixes/Design/Story Development
    QA: QA/Testing
Day 4
    Dev: Defect Fixes/Design/Story Development
    QA: QA/Testing
Day 5
    Dev: Defect Fixes
    QA: Acceptance Testing

細部說明:
前兩天
Dev:
    - Get busy coding the stories in their backlog while the QA’s start writing test cases/acceptance tests.
QA:
    - Start writing test cases/acceptance tests.
    - Should also be running these tests against the existing code base.
    - This validates that the test harness is working correctly and that the new tests do not mistakenly pass without requiring any new code.
    - Obviously, the new tests should also fail for the expected reason.
    - This tests the tests themselves, in the negative: it rules out the possibility that the new tests will always pass, and therefore be worthless.
    
接下來五天
Dev
    - Continue coding and also start working on fixing any defects picked up in the testing.
    - Pretty standard days. Code-test-fix.
QA
    - Begin testing any testable code that has been completed.

第 8 天開始
Dev
    - Effectively stop writing “new” code and focus their energy on fixing all defects identified in testing.
    - If has no defect work or find themselves with down-time while waiting for testing results, they can and should be helping the product owner develop and refine the user stories that are likely to be played in the next iteration.
    - Start considering the design aspects of the immediate upcoming stories and coordinating design decisions with the project architect.
QA
    - Continue to test all of the remaining code written during the iteration.

最後一天上午
Dev
    - Focus on bashing any remaining defects and making the code bombproof.
QA
    - Spend most of their time doing some final acceptance testing

最後一天下午    
    - It’s time for your review and demo.
    - You’ve tested and fixed everything so you and your team can demo with total confidence.
    - No surprises for your team, you’ve built serious quality into your software!

作者在post完他的做法之後, 精采的來了, 一共有54 篇response. 一起來討論他們自己怎麼做agile testing. 讓各位看倌繼續欣賞下去吧, 很精采ㄡ!!

===============================================
*    Dave Rooney Says:

定義了何謂一個user story是已經done了:
- A story is “done and tested by development, tested by QA and accepted by the business”.

主要想法
- You need sufficient QA capacity to handle the pipeline of completed stories coming from the developers.
- After all 2 stories completed to 100% in an iteration delivers more business value than 8 stories completed to 75%.

對於defect的處理態度
- I have the developers, QA and the business triage them -
- 確認 Is it a technical issue or an unclear, misunderstood or missing requirement?
- Does it affect the done state of a story? If so, fix it immediately. If not, confer with the business about its priority and schedule it accordingly.

===============================================

* Basim Baassiri Says:

Agile對testing所帶來的優點
- To start early and to shorten the feedback loop

Agile對testing上所造成的難處
- From a QA perspective, code complete has a different meaning then when development code is complete.
- What if the code completed by dev is a dependency for another set of features?
- What if the code completed by dev is not testable from an acceptance test perspective?
- Testing a moving target or code base causes unnecessary churn. Here is the scenario:
      Dev completes story 1 –> QA tests it and marks as DONE
      Dev completes story 2 that affects story 1 –> QA tests it and marks as DONE
    ==> The result here is that story 1 now has to be re-tested as a result of the changes done to story 2. Story 1 is at a state of quality unknown

解法
- We tried the approach where by QA is an iteration behind.
- What I mean by behind, the build candidate at the end of the iteration is taken and testing starts.

所採取的測試流程
1. test cases definition done early in the iteration
2. Test case review (not being done but ideally solicit feedback from dev and story owners)
3. We use keyword and domain based automation techniques so test cases are written once and can be used with the automation framework
4. Once dev signals the story is complete, start testing manually
5. Code supporting features in the automation framework if required. This obviously can’t be done until the code is complete
6. Verify defects and run regression testing activity
7. Release

kojenchieh 發表在 痞客邦 留言(0) 人氣()

3 week iteration中的測試工作

Testing Tasks in a 3 week iteration
http://www.testcommon.com/content.php?cid=918

在網路上我找到, 別人如何對3 週的iteration, 安排testing的工作.

首先這是他規劃的測試工作: (這裡是6RD對1 QA)
    * 2 days in the daily scrums and administrative meetings.
    * 1 day in some form of review with developers on feature implementations, bugs, test approach, XP test pairs, etc.
    * 2 days creating, or updating the test cases (low priority, time permitting).
    * 5 days on feature testing the new/modified features and residual bug fixes.
    * 2 days of strategic long-term tasks ie performance tests, automating regressions, environment build-up/clean-up (the lowest priority until critical to the functional testing)
    * 3 days regression testing (not complete regression, only a partial, evolving regression subject to features changed)

這是他執行後實際的工作狀況:
    * 1-2 days on the combination of scrums and design/developer discussions
    * <1 day on combination of test planning, test case maintenance and development of strategic test tasks
    * 3 days investigating field support issues and rumors of bugs
    * 1 day rebuilding/recovering the test environment for particular feature tests
    * 1 day on regression testing the final build(s)
    * 8 days or more on feature and bug testing

這裡是另一個iterative的pratice, 它是一家startup 公司的例子. (這裡是3RD對1QA) 基本上, 這家公司的feature 管理很混亂, 並沒有適當的document給QA, 或是和QA提醒一些implementation的改變.

所以他的規劃如下:
    * <1 day in all combinations of scrum, discussions, test planning, test case and environment maintenance
    * 2 days (and ever increasing) investigations of field reported issues
    * 1 day regression testing the prior iteration after it completed
    * 11 days on feature and bug fix verification and exploratory/ad-hoc testing

看完他的規劃後, 我很好奇幾點:
1. 它如何有這麼長的測試時間? 我的意思是RD如何這麼快deliver東西出來, 我很好奇他們是如何做開發設計的
2. 還是說他所謂的測試, 不只包含test case 的execution, 可能還有一些inpsection, static analysis?
3. 或者是說他其實包含對前一 iteration的功能持續在做測試?
4. 或者是說RD 的unit testing 他也算進去?

不知各位看倌, 你們的經驗或是作法是什麼?

kojenchieh 發表在 痞客邦 留言(0) 人氣()

A Low-Tech Testing Dashboard
- from James Bach, Principal Consultant

通常在測試的過程中, manager常常對於QA在做什麼不是很了解, 總覺得我們好像是黑箱作業, 讓他們無法幫助或是評量我們.

尤其, 他們常常會問
* How does the build look?
* How much testing is left?
* What are the major problems?

這些問題其實還蠻難回答, 你很難以簡單, 又不會太打擾team members的方式, 讓上面的人知道你目前大概的狀況是什麼.

在Agile的世界中,  這樣的狀況又更嚴重. 因為iteration/sprint的時間不長, 因此相對的時間更寶貴, engineer更沒太多時間來update report. 加上大家對agile的誤解是不用寫documentation. 所以manager或是其他非QA的人, 更不知道我們在幹什麼.

直到我遇到James, 我才知道原來已有人, 對這些問題試著提出一些解答.
http://www.satisfice.com/presentations/dashboard.pdf

在這個dashboard中, 它企圖做到下列事情
Report test cycle progress in a simple, structured way...
- that shows progress toward a goal...
- manages expectations...
- and inspires support...
- for an effective test process.



在這份dashboard中, 它會呈現以下五種資訊: (當然我想你可以自行客制化, 重點是他的精神)
1.Product Area
- 15-30 areas (keep it simple)
- Avoid sub-areas: they’re confusing.
- Areas should have roughly equal value.
- Areas together should be inclusive of everything reasonably testable.
- “Product areas” can include tasks or risks- but put them at the end.
- Minimize overlap between areas.
- Areas must "make sense" to your clients, or they won’t use the board

2. Test Effort
(1) 分類
None: Not testing; not planning to test.
Start: No testing yet, but expect to start soon.
Low: Regression or spot testing only; maintaining coverage.
High: Focused testing effort; increasing coverage.
Pause: Temporarily ceased testing, though area is testable.
Blocked: Can’t effectively test, due to blocking problem.
Ship: Going through final tests and signoff procedure.
(2) 規則
- Use red to denote significant problems or stoppages, as in blocked, none, or pause.
- Color ship green once the final tests are complete and everything else on that row is green.
- Use neutral color (such as black or blue, but pick only one) for others, as in start, low, or high.

3. Test Coverage
(1) 分類
0: We have no good information about this area.
1: Sanity Check: major functions & simple data.
1+: More than sanity, but many functions not tested.
2: Common Cases: all functions touched; common & critical tests executed.
2+: Some data, state, or error coverage beyond level 2.
3: Corner Cases: strong data, state, error, or stress testing.
(2) 規則
- Color green if coverage level is acceptable for ship, otherwise color black.
- Level 1 and 2 focus on functional requirements and capabilities: can this product work at all?
- Level 2 may span 50%-90% code coverage.
- Level 2+ and 3 focus on information to judge performance, reliability, compatibility, and other “ilities”: will this product work under realistic usage?
- Level 3 or 3+ implies “if there were a bad bug in this area, we would probably know about it.”

4. Quality Assessment
“We know of no problems in this area that threaten to stop ship or interrupt testing, nor do we have any definite suspicions about any.”
“We know of problems that are possible showstoppers, or we suspect that there are important problems not yet discovered.”
“We know of problems in this area that definitely stop ship or interrupt testing.”

5. Comments
(1) Definition
- Use the comment field to explain anything colored red, or any non-green quality indicator.
(2) Example:
- Problem ID numbers.
- Reasons for pausing, or delayed start.
- Nature of blocking problems.
- Why area is unstaffed.

那你要如何來使用這個dashboard
- 更新的頻率: 2-5/week, or at each build, or prior to each project meeting.
- 如何進行: Set expectation about the duration of the “Testing Clock” and how new builds reset it.
- 如何解釋內容: Be ready to justify the contents of any cell in the dashboard. The authority of the board depends upon meaningful, actionable content.
- 要不要用較High Tech的方式記錄: Sure, you can put this on the web, but will anyone actually look at it???


kojenchieh 發表在 痞客邦 留言(0) 人氣()

Agile Testing要成功的7個關鍵要素

Teaching Values vs. Process
http://lisacrispin.com/wordpress/?p=82

最近這個project要開始run agile. 因此小弟不遺餘力在加強agile的knowledge. 這裡"Agile Testing"一書的作者提出了, agile testing要成功的7個關鍵要素, 請大家享用

1. Use the whole-team approach.
- The whole development team must commit to producing high-quality software that delivers maximum business value.
    
2. Adopt an agile testing mind-set.
- We have a chapter in the book with “Ten Principles for Agile Testers”, and explain that an agile testing attitude is proactive, creative, open to new ideas, and willing to take on any task.
    
3. Automate regression testing.
- Without automated regression tests, the team will fail in the long run, because there won’t be time to do adequate exploratory testing, much less to do all the manual regression tests as the product grows.
- This is really a practice, but its root is in the value of feedback.

4. And feedback is our next success factor
- Provide and Obtain Feedback.
- Feedback is a core agile value, and testers are in a prime position to help provide the feedback that keeps the team on course.

5. Build a foundation of core practices.
- Again, these are practices such as continuous integration and working incrementally, but they are rooted in core agile values such as simplicity, communication and feedback.

6. Collaborate with customers
- Another core agile value.

7. Look at the big picture
- Another strength of testers.
- Even code produced test-first may cause unintended results in other parts of the system.
- Testers help the team evaluate how each story fits into the larger system.


kojenchieh 發表在 痞客邦 留言(0) 人氣()

Agile Testing 的十個神話

Ten myths of agile testing
http://www.aphids.com/agiletesting/2008/09/ten_myths_of_agile_testing.html

如果你的公司也採用agile development, 你可能也會面臨到和George Wilson (他是AIG Computer Services的Business Group Manager)相同的挑戰.  

Wilson提到 :

"Agile projects are an excellent opportunity for QA to take leadership of the [testing] processes,"

與其在developers主導時坐在後面, Wilson建議testers是可以主導一些事情:

"Who else is better placed to bridge the gap between users and developers, understand … what is required, how it can be achieved and how it can be assured prior to deployment?"

要做到這些 QA team需要更了解agile 是在做什麼, 所以Wilson列出了下面事實, 讓你可以看清agile testing的實質是什麼, 要如何做才是正確的

1. 你只需要Unit Test - TDD 測試已經足夠
- Test Driven Development is a good start, but for those who think it's all there is, "for the vast majority of commercial developments this simply isn’t true.
- Even strong proponents of agile development recognize the need for their armory to include a range of testing techniques...including white box, black box, regression testing, stress testing and UAT," said Wilson.
- The most effective agile projects will include investigative (black box) testing efforts that support (rather than compete) with white box testing.
- "Good investigative testing will reveal problems that the developer missed before they get too far downstream," said Wilson.

2. 你可以再利用 Unit Tests去建立Regression Test Suite
- Has anyone ever told you that conventional downstream testing is unnecessary because every line of application code has a corresponding test case?
- "Some TDD proponents ... suggest that by reassembling unit tests, everything from User Acceptance Tests to Regression Tests can be performed," said Wilson.
- While this may sound feasible, Wilson says it isn't realistic because the granularity and objectives of white box unit tests developed in TDD serve a different purpose than downstream black box testing.
- "While the overall objective of a unit test is to prove that the code will do what is expected, the aim of regression testing is to ensure that no unexpected effects result from the application code being changed. These two objectives are not synonymous.

3. 我們不再需要 Tesers或是 Automation Tools
- Not true.
- Testers perform a "different and equally valid role from their development colleagues," says Wilson, and should therefore be have an equal place at the project table.
- "It is [also] widely recognized that traditional test automation tools have failed to live up to the hype of the vendors." Perhaps vendors (no offense, Original) should play down the hype.

4. Unit Tests 可以排除要Manual Testing的需求
- Few would disagree that manual testing is repetitive, expensive, boring and error-prone.
- While TDD can reduce the amount of manual effort needed for functional testing, it does not remove the need for further manual or automated black box testing.
- "By automatically capturing the tester’s process and documenting their keystrokes and mouse clicks, the tester will have more time for the interesting, value-add activities, such as testing complex scenarios that would be difficult or impossible to justify automating.
- Though manual testing is a time-consuming (and therefore expensive) way to find errors, the costs of not finding them are often much higher," said Wilson.

5. User Acceptance Testing 不再是需要的
- Acceptance testing is often positioned as working with the customer to resolve incorrect requirements rather than correcting functionality that does not map to what the user needs.
- And maybe it's actually both. "When the user initially defines their requirements, they do it based on their initial expectations. When they see the system 'in the flesh,' they will invariably come up with different or additional requirements," he says.
- While the agile process might reduce the frequency of this happening, Wilson believes it is unreasonable to expect the approach to resolve them entirely, so there should be no expectation that user acceptance testing will be obviated.
- "This is especially true when it comes to the business user signing off approval of the user interface, since they may have envisaged something different from the developer."

6. Automation 是不可能的
- "Automation in the early stages of an agile project is usually very tough, but as the system grows and evolves, some aspects settle," says Wilson, simplifying the deployment of automation that can cope with changes.
- "To begin with, almost all testing will be manual, but this testing effort and design can be beneficial later if captured and reused. Knowing the right time to automate is tough, so using technology that proactively supports early manual testing but provides a path to evolve this into automation is key."

7. Developers 有足夠和適當的Ttesting Skills
- "If testing was easy, everybody would do it and we’d deliver perfect code every time," says Wilson. Just as writers benefit from proofreading by eyes other than their own, a testing team independent of developers can better see the big picture and can validate not just the functionality, but also the quality of the deliverable.
- "While developers tend towards proving the system functions as required, a good tester will be detached enough to ask, 'What happens if...?' ”

8. Unit Test可以涵蓋100% Design Spec
- Regardless of the development method, requirements must be known before code is developed.
- “While TDD ‘done well’ may identify a fair percentage of the design specification, there are still concerns about gaps between the unit tests,” says Wilson.
- He asserts that there are other approaches that are equally viable, and that the argument that you need TDD to prove the requirements are captured accurately isn’t substantiated by history.
- “Defining test cases to check the accuracy and succinctness of the requirements is nothing new. The V-model, for example, is a valid approach to understanding testing requirements, and by implication, functional requirements. Like TDD, the V-model and most other models fall down when the practitioner’s approach is rigid, while development processes are fluid. Software engineering is not like mechanical engineering, and trying to force conformity is wasted effort,” Wilson says.
- Whichever approach a team chooses, each user requirement should still be challenged by asking how it should be tested.
- “The important factor here is that test cases are challenged before they are committed to code, otherwise you’ll be doing a lot of recoding and calling it refactoring.”
- When the requirements are refined through collaboration, he adds, “the developer receives a [better] specification that is less likely to change because it has been openly appraised from several people’s perspectives.”

9. TDD 可適用於每個專案
- As the size of the project increases, so too does the time required to run the tests.
- Wilson says this can be addressed by partitioning the project, the tests or both. “Either route results in tests that run at different frequencies depending on their relevance to the current coding effort.”
- This approach, he says, introduces the need for test planning and execution management.
- “To achieve this efficiently, in addition to white box testing, you need to be thinking about…

10. Developers 和 Testers 就像油和水一樣格格不入
- There has long been an “us and them” tension between developers and testers, but it doesn’t have to be contentious.
- “This is usually a healthy symbiotic relationship which, when working correctly, provides a mutually beneficial relationship between the two groups, resulting in a higher quality deliverable for the customer," says Wilson.
- To help keep a handle on tensions, Wilson suggests focusing discussions on the following:
    * On meeting business objectives and NOT on who owns which parts of the process.
    * Involving testers in the requirements gathering phase and the TDD process.
    * Maximizing reuse of test assets created during the development phase.
    * The role of the traditional tester in TDD, and how to acquire new skills and enable them to adapt.
    * How developers and testers can find mutual benefit in exploiting new advances in software development and testing tools.

kojenchieh 發表在 痞客邦 留言(0) 人氣()

Agile Testing Video

我在Youtube 找到一些有關Agile testing 的videos, Enjoy it!!

(1) Agile Testing
- http://tw.youtube.com/watch?v=bqrOnIECCSg
- Speaker: Elisabeth Hendrickson
- 這是長達一小時的agile testing presentation, 這個speaker非常有名, 之前我有介紹過top 40 testing blogs, 這個speaker排行第15. 不過這個演講有點歷史在December 9, 2005, 在Google Tech Talks講的. 但是我想應該還是值得去聽一下
- Speaker's Web Site: http://testobsessed.com





(2) Is agile affecting testing?

- http://tw.youtube.com/watch?v=zhITAFaITCA&feature=related
- 這是採訪一些人, 分享有關agile對testing影響的想法. 這裡你會看到些有名的testing expert, 像是James Bach, Jonathan Kohl and Scott Barber. 其中Jonathan Kohl還蠻帥的.

kojenchieh 發表在 痞客邦 留言(0) 人氣()

在Agile Team中RD和QA的比例為何?

Tester-Developer Ratios on Agile Teams
http://lisacrispin.blogspot.com/2008/12/tester-developer-ratios-on-agile-teams.html

上次提到微軟對RD:QA比例的看法, 這次我們來看在agile team中, 比例應該又是多少?

作者提到這需要看情況, 那要看什麼情況呢? 以下是他考慮的因素:
1. Are the developers doing a good job of TDD? Does the team use acceptance tests to drive development, also?
2. How good is the automated regression test coverage?
3. How good is the communication between developers and customers?
4. Is the application testing-intensive, or does testing go pretty fast compared to coding?
5. Do the developers help with testing activities such as automating acceptance tests?

作者舉了三個例子
Example 1
(1) Project的背景
- My current team works with a financial services web app that is quite testing-intensive.
- The functionality is complex, and since we are dealing with peoples' money, we have to get it right.
- For some stories, testing takes more time than coding.
(2) RD所做的事情
- Our programmers are quite experienced in TDD and our regression test coverage is excellent.
- While our developers and customers sit close to one another and communicate well, and the developers have a high degree of domain knowledge, our stories are often quite complex.
- The developers take on a lot of testing tasks, including automating FitNesse tests and often writing the FitNesse test cases themselves.
(3) QA所做的事情
- A lot of tester time is devoted to working with customers to get examples of desired behavior and sort out technical challenges and other questions.
(4) RD:QA
- We have five programmers and two testers, and even so, the developers often have to pitch in extra on testing.

Example 2
(1) Project的背景
We were developing a non-critical internal application that would be used by a few expert users, so it wasn't the end of the world if the functionality wasn't perfect.
While the team did a good job of TDD, and we had good communication with customers despite the fact that we were a distributed team, this was clearly not enough.
(2) RD所做的事情
One or two developers or business analysts wore a tester "hat" each iteration and performed testing activities.
(3) QA所做的事情
I worked hard to transfer my testing knowledge to everyone on the team, so that our unit tests covered more test cases, and we could automate at least some of the functional regression tests.
(4) RD:QA
20 more more developers (subdivided into smaller teams but all working on one project) and I was the only official tester.

Example 3
(1) Project的背景
At first, the programmers could not get a handle on TDD or even unit test automation after coding, and I was buried.
I asked our coach for help. He helped us figure out how to get traction on unit testing, by providing training and time to learn.
Once the team mastered TDD, and pitched in on functional test automation, I was able to handle the other testing activities and we all worked well together.
(2) RD:QA
8 programmers and me, the tester.

由這些例子看起來, 如果RD可以大量apply TDD, unit testing, 也就整個開發活動是testing-intensive, 則可以讓QA專注於大量use case, or business scenarios.  這樣QA就不用佔有很高的比例.

我想這個結論, 和上次Microsoft QA manager的想法大致雷同. 若是RD能做更多基礎測試, 則系統會更穩定, regression testing能更快速, 也就是要讓RD確保do the thing right.

這樣QA就會有空確保do the right thing, 確認user想要的scenarios, 或是真正的use cases能夠正確運作. 而不是讓QA把時間大量花在do the thing right的測試上, 這樣就沒有空去照顧到do the right thing的部份, 這地方才是QA真正價值所在.

所以當一個team真正apply agile時, RD會做大量的test automation去確認do the thing right的部份. 這時候, 若是QA無法提升do the right thing這部份的能力, 小心啊, 你到時會被認為是多餘的....

kojenchieh 發表在 痞客邦 留言(1) 人氣()

Agile Testing Book

公司很多同事對這本是期待已久, 這是一本約576頁的巨作. 小弟終於在作者的blog中看到他終於寫完了, 應該2009/1月出版, 可能拿到手大約是過完農曆年後吧.

Agile Testing: A Practical Guide for Testers and Agile Teams (Addison-Wesley Signature Series) (Paperback)
by Lisa Crispin (Author), Janet Gregory (Author)
http://www.amazon.com/Agile-Testing-Practical-Guide-Testers/dp/0321534468/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1220381570&sr=1-1

這裡是作者的blog.
http://lisacrispin.blogspot.com/

這裡是這本書的網站, 有一些章節可以先downalod試讀, 並且也有整本書章節的mind map.
http://www.agiletester.ca/
http://www.agiletester.ca/Download.html

這是這本書的目錄, 若是你想看更多細節, 你可以看mind map中有寫
Part I Introduction
Ch1 What is Agile Testing, Anyway?
Ch2 Ten Principles for Agile Testers
- What's an Agile tester?
- Agile testing mind-set
- Applying agile principles and values
- Adding value

Part II Organizational Challengages
Ch3 Cultural Challengages
Ch4 Team Logistics
Ch5 Transitioning Traditional Processes

Part III Types of Agile  Testing
Ch6 The Purpose of Testing - Intro to Quadrants
Ch7 Technology-facing/support team
Ch8 Business-facing/support team
Ch9 Toolkit for business facing/suport team
Ch10 Business facing/critique product
Ch11 Technique facing/critique product
Ch12 Quadrant Summary - testing on a real agile project

Part IV Automation
Ch14 Why Automation, What Holds Us Back
Ch15 Developing an Automation Strategy

Part V An Interation in the life of a Tester
Ch16 Tester Activities in Release/Theme Planning
Ch17 Hit the Ground Running
Ch18 Interation Kickoff
Ch19 Coding and Testing
Ch20 Successful Delivery

Part VI Summary
Ch21 Key Success Factors

kojenchieh 發表在 痞客邦 留言(0) 人氣()



How can I get tests automated with back-to-back, short iterations? I don't have time

Agile最大的特色, 就是會有很多iteration發生. 但是這件事情, 帶給QA很大的負擔. 因為每次你將會有很多regression test需要執行, 而且你要執行的範圍, 不是只有這次新增的功能, 往往可能是要包含以前的features.

可是呢? 每次iteration的時間都很有限, 最多只有讓你有時間測完這次的功能, 你不太可能有時間去保全部的test cases, 去確認所有的功能是否正常. 所以每次只好欺騙自己, 之前的部分沒有動到, 所以應該部會有問題. 真的是這樣嗎? 我想你應該知道那是有點在騙自己的感覺.

那有沒有方法可以解決呢? 以下是這篇文章建議如何adopt test atuomation到你各個層面, 期待藉由高automation ratio, 來讓你每個interation的測試能夠cover的層面較廣, 讓你會比較安心

Agile Testing Realities
http://lisacrispin.blogspot.com/2008/12/agile-testing-realities.html

1. The whole development team, not only the testers, needs to tackle test automation.
2. Start by implementing a continuous integration and build process so you have a way to run your tests automatically, and get quick feedback from them.
3. Unit tests have the best ROI, so start test automation efforts there.
4. For legacy code, try a lightweight automation tool and do simple smoke tests that don't have a lot of logic in them and are easy to maintain. You'll be surprised how many regression bugs they find.
5. Cut down the scope of work your development team commits to in each iteration so you make sure there is time to finish all test automation tasks. Don't put them off until the next iteration - that's the road to perdition.
6 Repeat this mantra, write it on your story board or whiteboard: No story is done until it's tested! This includes automating regression tests for it too!

其實agile的方法是配套產生的, 你既然要iteration, 就會面臨這樣的測試問題, 所以他們才會提倡test driven, 以及一些高度test automation的工作, 這樣才能真正確保每個iteration的品質.

不過往往我們在執行時, 常常只是adopt一部分的practices, 自然而然地就會遇到很多問題. 就像我們公司, 很多team就只是要求要iteration, 可是相關RD or QA要配合的動作沒做, 自然就會run的很辛苦. 怪不得每次一adopt某個新的methodology, 就黑掉一個methodology. 慎之, 戒之啊!!

kojenchieh 發表在 痞客邦 留言(2) 人氣()

Becoming a More Agile Tester
http://www.jrothman.com/weblog/JR.Agiletester.pdf

Posted by Johanna Rothman

 這是另外一篇網路上討論agile testing的文章, enjoy it

kojenchieh 發表在 痞客邦 留言(0) 人氣()

Agile Testing - Nine Principles and Six Concrete Practices for Testing on Agile Teams

- Elisabeth Hendrickson

http://testobsessed.com/2008/08/11/agile-testing-overview-redux/

這是一篇介紹agile testing的簡報. 她告訴你agile testing基本上的不同是什麼. 其中令我最驚訝的地方, 是簡報中的插圖, 畫的十分生動傳神. 把agile testing mindset的轉變給highlight出來. 我想它是一篇讓你很快可以上手, 了解agile testing是什麼的文章


kojenchieh 發表在 痞客邦 留言(0) 人氣()

這些是我目前找到有關agile testing的相關書籍, 看起來真的不多, 所以大部分的時候還是要找一些paper or industry reports來看看

A. Agile Testing
1. Lessons Learned in Software Testing 
by Cem Kaner, James Bach, Bret Pettichord
http://www.amazon.com/Lessons-Learned-Software-Testing-Kaner/dp/0471081124/ref=sr_1_9?ie=UTF8&s=books&qid=1220664539&sr=1-9

kojenchieh 發表在 痞客邦 留言(0) 人氣()

Part IV QA Paradigm

1. QA Paradigm #1: Quality Assurance is Testing
(1) Most QA people are actually employed as testers
(2) “Did you QA this?”
(3) “Independent testing is better testing”

kojenchieh 發表在 痞客邦 留言(0) 人氣()

Part III Agile Activities and Reality Check
(這裡談到在agile activities的agile testing的關係, 在這些activities中testing 要如何運作和apply)

1. Test-First Programming
(1) Developers write unit tests before coding.
   - Motivates coding

kojenchieh 發表在 痞客邦 留言(0) 人氣()

Agile Testing Introduction

我survey了Bret Pettichord幾篇介紹agile testing的簡報. 我想應該可以幫大家快速入門

Content
Agile/Agile methodologies
Agile Testing
Agile Activities and Reality Check

kojenchieh 發表在 痞客邦 留言(0) 人氣()

4. Functional Acceptance Testing
(1) Story-based (aka use-case based)
   - In the main books on XP, including Crispin's book on testing in XP, story-based tests are described as the primary vehicle for functional acceptance testing.

(2) Typical tests are fairly simple
   a. Happy path:

kojenchieh 發表在 痞客邦 留言(0) 人氣()

«12 3
Close

您尚未登入,將以訪客身份留言。亦可以上方服務帳號登入留言

請輸入暱稱 ( 最多顯示 6 個中文字元 )

請輸入標題 ( 最多顯示 9 個中文字元 )

請輸入內容 ( 最多 140 個中文字元 )

reload

請輸入左方認證碼:

看不懂,換張圖

請輸入驗證碼