Compatibility Testing
最近在測試時遭遇到一個問題:
當使用者在執行我們的產品, 同時也在執行其他軟體, 像是ftp, 聽音樂, 上網, 燒片子, 用MS Office, MSN, facebook...等等, 可能會造成系統緩慢或是不正常.
我們的QA大部分時間都在執行funcational testing, 也就是確認我們系統的功能是否正確, 因此在他們執行過程中不會去執行其他軟體, 因此也就沒有遇到這種狀況.
或許有些QA會想說要做相容性測試(compatibility testing), 但是大多只是把一些常用的軟體裝上去, 簡單看一下我們的受測程式是否大致正常, 這樣就結束了. 所以可想而知應該是抓不到什麼問題.
在wiki中有提到相容性測試要做哪些事情:
kojenchieh 發表在 痞客邦 留言(0) 人氣()
kojenchieh 發表在 痞客邦 留言(0) 人氣()
昨天老闆和我們分享一個想法: 除錯只能靠Debug Log嗎?
他說到他來公司已經不少年, 可是對於公司的一件事情還是不是很習慣. 那就是engineer要解決一個bug時, 總是要求說要提供debug log給他, 否則他們就無法解這個問題.
對他來說, 這事件很不可思議的事. 因為解決問題的方法很多, 並不是只有看debug log才能解決, 而且這可能是你不肯動腦的藉口, 只想讓人把線所放到你面前. 更糟的事, 這線索可能是不完整或是不正確的.
kojenchieh 發表在 痞客邦 留言(8) 人氣()
開發人員常去的前 100名的blog
Top 100 Blogs for Developers (Q1 2009)
http://www.noop.nl/2009/03/top-100-blogs-for-developers-q1-2009.html
March 16, 2009
Posted by Jurgen Appelo
kojenchieh 發表在 痞客邦 留言(0) 人氣()
我在幫你忙啊!!我真的在幫你啊!!
I'm helping you. I'm helping you.
http://www.questioningsoftware.com/2009/01/im-helping-you-im-helping-you.html
January 5, 2009
Posted by Ben Simo
kojenchieh 發表在 痞客邦 留言(0) 人氣()
100% 測試涵蓋度真的是100%嗎?
100% Test Coverage?
http://www.infoq.com/news/2007/05/100_test_coverage
May 29, 2007
Posted by Amr Elssamadisy
Published in Info Queue
kojenchieh 發表在 痞客邦 留言(0) 人氣()
如果你找不到root cause時要怎麼辦?
where you don’t know the root cause
http://www.quicktestingtips.com/tips/2009/05/reporting-issues-where-you-dont-know-the-root-cause/
當你要report一個bug時, 若是你並不知道root cause, 請確認你會把以下資訊放到bug tracking system中:
# screen shots
# log files (local and server if you can get them)
kojenchieh 發表在 痞客邦 留言(2) 人氣()
整合測試(Integation Testing)和單元測試(Unit Testing)的差別在哪裡?
What type of designs are needed to write integration test cases?
http://www.michaeldkelly.com/blog/archives/265
April 26th, 2009
Posted by Mike Kelly
Published in Mike Kelly’s blog
kojenchieh 發表在 痞客邦 留言(0) 人氣()
如何面對 "God! 你又沒測到" 這個問題
Why Why Why Why Why?
"Why did you miss that?"
http://blogs.msdn.com/micahel/archive/2008/06/04/WhyWhyWhyWhyWhy.aspx
June 04, 2008
Posted by micahel
kojenchieh 發表在 痞客邦 留言(0) 人氣()
為何你會找不到Bug?
The creative tester
http://xndev.blogspot.com/2009/04/creative-tester.html
April 02, 2009
Posted by Matthew
Published in Creative Chaos
kojenchieh 發表在 痞客邦 留言(0) 人氣()
kojenchieh 發表在 痞客邦 留言(0) 人氣()
找很多的Bugs重要, 還是找重要的Bugs重要?
Test Smarter: The Top 3 Testing Myths
http://worksoftinc.blogspot.com/2008/08/this-is-test-blog.html
August 26, 2008
Posted by Linda Hayes
kojenchieh 發表在 痞客邦 留言(0) 人氣()
一個好的QA要具備怎樣的能力
雖然最近經濟不景氣, 老闆還是要我們盡量能找一些優秀的員工進來, 因此特別詢問了一下, 我們是如何找的QA. 剛好最近正在看 "How We Test Software at Microsoft", 書中提到好的QA應該要有這些特質. 雖然是老生常談, 但是重要的還是就是這些東西. 需知道有時候越是基本的東西, 通常是越難練的.
1. Analytical Problem Solving
- problem decomposition and RCA are key to driving quality upstream
2. Customer-Focused Innovation
kojenchieh 發表在 痞客邦 留言(0) 人氣()
Testers在微軟的發展史
在How We Test Software At Microsoft一書中, 有提到在微軟中, 過去有兩個有關tester的職務: STE(Software Test Engineer 和SDE/T(Software Development Engineer in Test).
這兩個職稱常常讓人家覺得很困擾, 因為不知道他們的差別到底是什麼. 在有些team中, SDET是指使用testing tool的engineer, 可是在有些team中, 是指做test automation的人.
在那個年代, 作者認為大約的工作區分如下:
1. SDE/T
- Develop test harness for test execution
kojenchieh 發表在 痞客邦 留言(0) 人氣()
幫助做Cross Browser Compatibility Test的工具
7 Fresh and Simple Ways to Test Cross-Browser Compatibility
http://freelancefolder.com/7-fresh-and-simple-ways-to-test-cross-browser-compatibility/
February 25, 2009
Posted by Mason Hipp
Published in freelancefolder
kojenchieh 發表在 痞客邦 留言(2) 人氣()
QA該做轉變了
最近有人提到test cases太多run不完, 找到bug太多RD解不完, automation rate太少, 高層總是schedule導向, 因此導致整個QA的工作很忙, 士氣也不高.
我想這些都是QA常見的問題, 不過我想換個角度來思考這些問題:
1. Test cases太多run不完
kojenchieh 發表在 痞客邦 留言(5) 人氣()
二十個最佳測試實務
20 Best Testing Practices
http://jaanujeeva.blogspot.com/2009/01/20-best-testing-practices.html
根據作者的經驗, 整理出一些他認為best testing practices:
1) 學習去徹底地分析你的測試結果
- Do not ignore the test result.
- The final test result may be ‘pass’ or ‘fail’ but troubleshooting the root cause of ‘fail’ will lead you to the solution of the problem.
- Testers will be respected if they not only log the bugs but also provide solutions.
2) 在每次測試中, 學習去涵蓋更多受測範圍
- Though 100 percent test coverage might not be possible still you can always try to reach near it.
3) 將你的受測軟體分解成更小的部份, 並且最大化其涵蓋程度
- Write test cases on such individual unit modules. Also if possible break these modules into smaller parts.
* E.g: Let’s assume you have divided your website application in modules and ‘accepting user information’ is one of the modules.
- You can break this ‘User information’ screen into smaller parts for writing test cases:
* Parts like UI testing, security testing, functional testing of the ‘User information’ form etc.
- Apply all form field type and size tests, negative and validation tests on input fields and write all such test cases for maximum coverage.
4) 在撰寫測試個案, 先考慮測試所想要的功能, 也就是先確認規格上的功能
- Then write test cases for invalid conditions.
- This will cover expected as well unexpected behavior of application under test.
5) 要認為你一定找得到bug
- Start testing the application by intend of finding bugs/errors.
- Don’t think beforehand that there will not be any bugs in the application.
- If you test the application by intention of finding bugs you will definitely succeed to find those subtle bugs also.
6) 根據需求分析和設計階段本身來開力測試個案
- This way you can ensure all the requirements are testable.
7) 確認你的測試個案能在RD撰寫程式前就能拿到
- Don’t keep your test cases with you waiting to get final application release for testing, thinking that you can log more bugs.
- Let developers analyze your test cases thoroughly to develop quality application. This will also save the re-work time.
8) 如果可能的話, 確認和分類哪些測試個案可供回歸測試使用
- This will ensure quick and effective manual regression testing.
9) 需要徹底測試, 受測軟體是否有能在適當的時間內反應
- Performance testing is the critical part of many applications.
- In manual testing this is mostly ignored part by testers due to lack of required performance testing large data volume.
- Find out ways to test your application for performance.
- If not possible to create test data manually then write some basic scripts to create test data for performance test or ask developers to write one for you.
10) RD不應該測試他們自己寫的code
- Basic unit testing of developed application should be enough for developers to release the application for testers.
- But you (testers) should not force developers to release the product for testing. Let them take their own time.
- Everyone from lead to manger know when the module/update is released for testing and they can estimate the testing time accordingly.
- This is a typical situation in agile project environment.
11) 應該不只只測試需求上面的東西而已
- Test application for what it is not supposed to do.
12) 在做回歸測試時, 需要根據bug grpah來決定測試方向
- Bug graph - number of bugs found against time for different modules
- This module-wise bug graph can be useful to predict the most probable bug part of the application.
13) 在測試的時候記錄下新的要處理項目和想法
- Keep a text file open while testing an application.
- Note down the testing progress, observations in it.
- Use these notepad observations while preparing final test release report.
- This good habit will help you to provide the complete unambiguous test report and release details.
14) 有時候你會因為特需需求而對受測軟體做修改
- This is required step in development or testing environment to avoid execution of live transaction processing like in banking projects.
- Note down all such code changes done for testing purpose and at the time of final release make sure you have removed all these changes from final client side deployment file resources.
15) 讓RD 遠離測試環境
- This is required step to detect any configuration changes missing in release or deployment document.
- Some times developers do some system or application configuration changes but forget to mention those in deployment steps.
- If developers don’t have access to testing environment they will not do any such changes accidentally on test environment and these missing things can be captured at the right place.
16) 讓QA在需求和設計階段時加入, 是非常有幫助的作法
- These way testers can get knowledge of application dependability resulting in detailed test coverage.
- If you are not being asked to be part of this development cycle then make request to your lead or manager to involve your testing team in all decision making processes or meetings.
17) QA team 需要和組織內其他團隊, 分享好的testing practices和experience
18) 需要多花時間和RD對話, 以去了解你的受測軟體
- Whenever possible make face-to-face communication for resolving disputes quickly and to avoid any misunderstandings.
- But also when you understand the requirement or resolve any dispute - make sure to communicate the same over written communication ways like emails.
- Do not keep any thing verbal.
19) 不要用完的所有時間只做high priority testing tasks.
- Prioritize your testing work from high to low priority and plan your work accordingly.
- Analyze all associated risks to prioritize your work.
20) 要能撰寫簡潔, 描述清楚的, 讓人不會搞混的Bug Report
- Do not only provide the bug symptoms but also provide the effect of the bug and all possible solutions.
不要忘記, 測試是一個非常creative 和 challenging的工作. 當你在處理這樣的挑戰時, 你的經驗和技術非常有關係的.
kojenchieh 發表在 痞客邦 留言(0) 人氣()
Bug Bash的執行經驗談
How to run a bug bash
http://www.scottberkun.com/blog/2008/how-to-run-a-bug-bash/
作者認為bug bash, 是軟體開發產業中不能說的秘密. 在正式的軟體工程課程上, 你從來沒有看過它.
如果有專門的QA, 或是你能夠把TDD執行的很好, 或許你會使用到bug bash的機率會比較小. 但是大部分的組織, 不會有這樣的事發生. 所以作者認為bug bash是經常被使用的技巧, 來維持產品的品質.
雖然bug bash有許多好處, 但是若是你執行方法不正確, 那並不會產生好的結果, 會變成只是一個多餘的工作. 所以作者將會告訴我們, 要如何進行才會讓bug bash的效果發揮到最好.
以下是作者在執行bug bash的致勝秘訣
1. 不要造成大家恐慌
- If you say “Tomorrow! Everyone find bugs! Aaaah!” You are creating a panic and look like an idiot.
- You should know a week or more in advance that the bug counts are soft, or the database needs scrubbing, and line up leads and key players to support the effort.
2. 凍結目前的source codes
- You can not do a bug bash on a moving target: you invalidate repro cases and bug findings.
- Pick a build, freeze it, and make sure no one, NO ONE touches the live codebase during the bugbash.
- This should go without saying, but you never know.
- If it’s your first team wide bugbash make sure the entire programming team understands this basic rule.
3. 告訴大家怎樣才是好的bug reports
- Remind everyone crappy bug reports create extra work.
- Provide two bug report examples: one good, one bad.
In the good example show well written description, clear repro steps, and a search for duplicate bugs.
In the bad, show incomprehensible descriptions, impossible repro steps, etc.
- If you don’t provide examples, don’t expect people to magically know what you’re looking for.
- Finding 1000 crappy bugs that need to be heavily cleaned up is a waste of everyone’s time.
4. 告訴大家要測試哪些地方或是要找出哪類型的bug
- Saying “find bugs” is a shot in the dark.
- It shows you have no clue what’s going on in your project.
- Think through what the weakest areas are, or what types of bugs you are most afraid of, and designate them the primary goals of the bug bash.
- Or offer bonus points (e.g. bugs in area 6 are worth x2) for people who find the specific type of bug most valuable to you.
5. 要確認大家在這段時間能配合作bug bash
- A bug bash should be an entire team activity and a half-day is the perfect amount of time.
- Everyone should be working on the same goal: getting good data into the bug database, and getting that database in shape.
- If it’s voluntary, or only half the team is asked to do it, the bash will fail.
- People will smell you’re not serious about the effort, and will contribute accordingly.
- Get permission to reschedule all team meetings for that afternoon to later in the week, and send out a new meeting invite to the team for the entire bug bash time slot.
- Include details (see below) on where bug bash HQ is, what the prizes are, etc.
6. 需要取得大頭們的支持
- With the support of leaders it sets the tone of how important the activity is, and eliminates the BS excuses people find not to participate (”If Fred, our best programmer is doing it, I should be doing it too”).
- Find the key players on your team, either key leaders or the star programmers, and get them to help promote and contribute.
7. 找一個地方大家可以集中進行bug bash
- Finding bugs can be a social activity: have a bug bash headquarters.
- Grab a conference room, order pizza and beer, and invite people with laptops to hang out and find bugs together.
- This invites people to help each other find repro-cases, share knowledge and bug database tricks, makes keeping a scoreboard easy, and makes the bug bash a proper morale event.
- A case of beer and few pizzas costs $60. Well worth it.
8. 記錄執行成績並提供獎賞
- Geeks are competitive. Use this to your advantage.
- Any bug database allows queries for open bugs by date: Get this up on a website or hallway monitor and show it in real time.
- Buy some nerf weapons, dinner gift certificates, or even some X-box video games, and have them visible at HQ - give them away as prizes, or set up a betting pool: $10 per person, and the winner gets the pot.
- You can get fancy and have special prizes for most twisted bug, the bug least likely to ever get fixed, etc.
9. 建立可相互競爭的小組
- If you are totally poor, use ego prizes.
- Have the designers challenge the programmers, or the marketing team challenge the management team.
- Throw down: “I’ll bet the whole marketing team dinner at Ruth Chris’ my 3 reports can find more bugs than your whole team can”.
- If you don’t have cash, bet embarrassment: loser shaves their heads, has to dress in costume the next day, has to wash the opponents cars, etc.
- Get two sets of people who have some built in animosity or rivalry, especially if it’s well known, to openly challenge each other.
- This rivalry will draw more people in, if only to follow along.
- Do this once and you’ll have a tradition to build on for the next bug bash.
kojenchieh 發表在 痞客邦 留言(0) 人氣()
GUI Testing的檢查清單
很多QA只會做functional testing, 除了這些之外, 其他種類的測試便不知道要如何進行. 尤其在GUI方面, 因為缺乏資料, 即使有心想做的人也不知道要如何進行. 再加上無人重視, 更不容易會將這項工作納入處理.
這篇是我在網路上, 找到對GUi的檢查清單. 我想這是一份很好的入門指引, 讓你快速瞭解到, 專業的 GUI應該要注意到哪些事情. 你可以把這份清單當做起始點, 然後來建立你自己的知識庫.
Enjoy it!!
GUI Checklist
http://jaanujeeva.blogspot.com/2008/12/gui-checklist.html
1. USER INTERFACE
1.1 COLORS
1.1.1 Are hyperlink colors standard?
1.1.2 Are the field backgrounds the correct color?
1.1.3 Are the field prompts the correct color?
1.1.4 Are the screen and field colors adjusted correctly for non-editable mode?
1.1.5 Does the site use (approximately) standard link colors?
1.1.6 Are all the buttons are in standard format and size?
1.1.7 Is the general screen background the correct color?
1.1.8 Is the page background (color) distraction free?
1.2 CONTENT
1.2.1 All fonts to be the same
1.2.2 Are all the screen prompts specified in the correct screen font?
1.2.3 Does content remain if you need to go back to a previous page, or if you move forward to another new page?
1.2.4 Is all text properly aligned?
1.2.5 Is the text in all fields specified in the correct screen font?
1.2.6 Is all the heading are left aligned
1.2.7 Does the first letter of the second word appears in lowercase? Eg:
1.3 IMAGES
1.3.1 Are all graphics properly aligned?
1.3.2 Are graphics being used the most efficient use of file size?
1.3.3 Are graphics optimized for quick downloads?
1.3.4 Assure that command buttons are all of similar size and shape, and same font & font size.
1.3.5 Banner style & size & display exact same as existing windows
1.3.6 Does text wrap properly around pictures/graphics?
1.3.7 Is it visually consistent even without graphics?
1.4 INSTRUCTIONS
1.4.1 Is all the error message text spelt correctly on this screen?
1.4.2 Is all the micro-help text(i.e tool tip) spelt correctly on this screen?
1.4.3 Microhelp text(i.e tool tip) for every enabled field & button
1.4.4 Progress messages on load of tabbed(active screens) screens
1.5 NAVIGATION
1.5.1 Are all disabled fields avoided in the TAB sequence?
1.5.2 Are all read-only fields avoided in the TAB sequence?
1.5.3 Can all screens accessible via buttons on this screen be accessed correctly?
1.5.4 Does a scrollbar appear if required?
1.5.5 Does the Tab Order specified on the screen go in sequence from Top Left to bottom right? This is the default unless otherwise specified.
1.5.6 Is there a link to home on every single page?
1.5.7 On open of tab focus will be on first editable field
1.5.8 When an error message occurs does the focus return to the field in error when the user cancels it?
1.6 USABILITY
1.6.1 Are all the field prompts spelt correctly?
1.6.2 Are fonts too large or too small to read?
1.6.3 Are names in command button & option box names are not abbreviations.
1.6.4 Assure that option boxes, option buttons, and command buttons are logically grouped together in clearly demarcated areas “Group Box”
1.6.5 Can the typical user run the system without frustration?
1.6.6 Do pages print legibly without cutting off text?
1.6.7 Does the site convey a clear sense of its intended audience?
1.6.8 Does the site have a consistent, clearly recognizable “look-&-feel”?
1.6.9 Does User cab Login Member Area with both UserName/Email ID ?
1.6.10 Does the site look good on 640 x 480, 600×800 etc.?
1.6.11 Does the system provide or facilitate customer service? i.e. responsive, helpful, accurate?
1.6.12 Is all terminology understandable for all of the site’s intended users?
Performance & Security Testing Checklist
2. PERFORMANCE
2.1 LOAD
2.1.1 Many users requesting a certain page at the same time or using the site simultaneously
2.1.2 Increase the number of users and keep the data constant
2.1.3 Does the home page load quickly? within 8 seconds
2.1.4 Is load time appropriate to content, even on a slow dial-in connection?
2.1.5 Can the site sustain long periods of usage by multiple users?
2.1.6 Can the site sustain long periods of continuous usage by 1 user?
2.1.7 Is page loading performance acceptable over modems of different speeds?
2.1.8 Does the system meet its goals for response time, throughput, and availability?
2.1.9 Have you defined standards for response time (i.e. all screens should paint within 10 seconds)?
2.1.10 Does the system operate in the same way across different computer and network configurations, platforms and environments, with different mixes of other applications?
2.2 VOLUME
2.2.1 Increase the data by having constant users
2.2.2 Will the site allow for large orders without locking out inventory if the transaction is invalid?
2.2.3 Can the site sustain large transactions without crashing?
2.3 STRESS
2.3.1 Increase both number of users and the data
2.3.2 Performance of memory, CPU, file handling etc.
2.3.3 Error in software, hardware, memory errors (leakage, overwrite or pointers)
2.3.4 Is the application or certain features going to be used only during certain periods of time or will it be used continuously 24 hours a day 7 days a week? Test that the application is able to perform during those conditions. Will downtime be allowed or is that out of the question?
2.3.5 Verify that the application is able to meet the requirements and does not run out of memory or disk space.
2.4 SECURITY
2.4.1 Is confidentiality/user privacy protected?
2.4.2 Does the site prompt for user name and password?
2.4.3 Are there Digital Certificates, both at server and client?
2.4.4 Have you verified where encryption begins and ends?
2.4.5 Are concurrent log-ons permitted?
2.4.6 Does the application include time-outs due to inactivity?
2.4.7 Is bookmarking disabled on secure pages?
2.4.8 Does the key/lock display on status bar for insecure/secure pages?
2.4.9 Is Right Click, View, Source disabled?
2.4.10 Are you prevented from doing direct searches by editing content in the URL?
2.4.11 If using Digital Certificates, test the browser Cache by enrolling for the Certificate and completing all of the required security information. After completing the application and installation of the certificate.
kojenchieh 發表在 痞客邦 留言(0) 人氣()
撰寫好的Bug Report之技巧與秘訣
How to write a good bug report? Tips and Tricks
http://jaanujeeva.blogspot.com/2009/01/how-to-write-good-bug-report-tips-and.html
最近老闆發現某個product, 有20%~30%的bug被reject. 中間的原因可能很多, 但是也突顯了bug report 的重要性. 如果你能產生有效的bug report, 能準確提供問題所在之處, 這樣才能讓所有人都達到win win的地步.
這裡找到一篇bug report 的經驗談, 大家享用吧!!
1. 為什麼要產生好的bug report
- 如果你的bug report能有效率的指出問題所在, RD越有機會能更解決它
- Cem Kaner.曾說“The point of writing problem report(bug report) is to get bugs fixed”
- 如果QA沒有辦法產生出正確的bug report, RD最後有可能說這bug是沒有問題的, 因而reject它. 這將會導致QA的士氣其受損, 甚至防衛心理或對立心理而升高
2. 好的Bug Report應該具備有怎樣的特質呢?
(1) Having clearly specified bug number:
- Always assign a unique number to each bug report.
- This will help to identify the bug record.
- Note the number and brief description of each bug you reported.
(2) Reproducible:
- If your bug is not reproducible it will never get fixed.
- You should clearly mention the steps to reproduce the bug.
- Do not assume or skip any reproducing step.
- Step by step described bug problem is easy to reproduce and fix.
(3) Be Specific:
- Do not write a essay about the problem.
- Be Specific and to the point.
- Try to summarize the problem in minimum words yet in effective way.
- Do not combine multiple problems even they seem to be similar.
- Write different reports for each problem.
一些能幫助你寫好bug report的小技巧
(1) Report the problem immediately:
- If you found any bug while testing, do not wait to write detail bug report later. Instead write the bug report immediately.
- This will ensure a good and reproducible bug report.
-If you decide to write the bug report later on then chances are high to miss the important steps in your report.
(2) Reproduce the bug three times before writing bug report:
- Your bug should be reproducible.
- Make sure your steps are robust enough to reproduce the bug without any ambiguity.
- If your bug is not reproducible every time you can still file a bug mentioning the periodic nature of the bug.
(3) Test the same bug occurrence on other similar module:
- Sometimes developer use same code for different similar modules.
- So chances are high that bug in one module can occur in other similar modules as well.
- Even you can try to find more severe version of the bug you found.
(4) Write a good bug summary:
- Bug summary will help developers to quickly analyze the bug nature.
- Poor quality report will unnecessarily increase the development and testing time.
- Communicate well through your bug report summary. Keep in mind bug summary is used as a reference to search the bug in bug inventory.
(5) Read bug report before hitting Submit button:
- Read all sentences, wording, steps used in bug report.
- See if any sentence is creating ambiguity that can lead to misinterpretation.
- Misleading words or sentences should be avoided in order to have a clear bug report.
(6) Do not use Abusive language:
- It’s nice that you did a good work and found a bug but do not use this credit for criticizing developer or to attack any individual.
- Conclusion: No doubt that your bug report should be a high quality document.
- Focus on writing good bug reports, spend some time on this task because this is main communication point between tester, developer and manager.
- Mangers should make aware to their team that writing a good bug report is primary responsibility of any tester.
- Your efforts towards writing good bug report will not only save company resources but also create a good relationship between you and developers.
kojenchieh 發表在 痞客邦 留言(0) 人氣()
