效能測試中有哪些需要注意的階段
Performance Testing Core Principles: CCD IS EARI
http://www.testingreflections.com/node/view/5448
21/05/2007
Posted by sbarber
Published in TestingReflection
目前分類：Performance Testing (38)
- Jun 08 Mon 2009 09:33
效能測試中有哪些需要注意的階段
- Jun 02 Tue 2009 09:19
在Agile專案中如何進行效能工程
在Agile專案中如何進行效能工程
Performance Engineering in an Agile Project
http://www.infoq.com/news/2009/03/agile-performance-engineering
Mar 25, 2009
Posted by Vikas Hazrati
Published i InfoQ
- Mar 23 Mon 2009 09:45
在面試Automation 和Load testing相關經驗時可以問的問題
在面試Automation 和Load testing相關經驗時可以問的問題
Automation Q & A
http://thiyagarajan.wordpress.com/automation-q-a/
Posted by Thiyagarajan Veluchamy
Published in Thiyagarajan Veluchamy's BLog
- Dec 20 Sat 2008 19:06
The 10 Commandments of Load Testing
The 10 Commandments of Load Testing
http://www.myloadtest.com/ten-commandments-of-load-testing/
以下是你在做load testing, 要小心的十件事情
1. 你需要知道test toll是如何運作的
- The worst performance testers I have met were always more concerned about whether they could get their scripts to run, rather than whether the tests they were running were realistic.
- Read the documentation, practice, spend some time figuring out what all the settings do, then relate how your scripts are running back to how real users exercise your application.
2. 你需要收集真實的在運作或使用的資料
- Garbage in, garbage out. If your transaction volumes are wrong, then your load test is wrong.
3. 你需要有可測試的requirements.
- Non-functional requirements (especially load and performance-related requirements) are usually an afterthought for many projects.
- This shouldn’t stop you from trying to gather the requirements you need for your tests.
- The business approach of “let us know how fast it is, and we will let you know if that’s okay” isn’t good enough.
- Get some numbers. The numbers can change in the future (maybe call them “targets” or “guidelines” rather than “requirements”), but you need something to test against before you start.
4. 你應該要撰寫測試計劃
- Even if you already know what you’re going to be doing, other people would probably like to know too - they might even be able to help.
- A a signed-off test plan has saved many a tester from the wrath of project management.
5. 你應該要測試最差的狀況
- Don’t test with transactions from an average day, test for the busiest day your business has ever had.
- Add a margin for growth.
- Testing failover? A server doesn’t fall over at midnight when no one is using your application (would we care in this situation anyway?), it falls over in the middle of the day when lots of real people are using it.
6. 你需要監控你的測試環境
- Monitoring your servers allows you to more easily figure out where the problem is.
- You can also make neat observations like “response times for the new version of the application are the identical to the previous version, but CPU utilisation on the servers has increase by 10%”
- When I say “monitor your servers”, this includes your load generators.
7. 必須要對你的測試環境做change control
- The final thing you tested should be what is deployed into Production - same application version, same system configuration.
- It’s easy to lose track of what you are actually testing against if people are making uncontrolled changes to your environment, or if people are making tuning changes without tracking what they are changing.
- Keep a list of changes that are made…even if you are in a hurry; and always make sure you know what you are testing against.
8. 你應該要使用 defect tracking tool.
- An untracked defect is a little like a tree that fall in the forest when no-one is around - no-one cares.
- Raising defects lets everyone know there is a problem
- It also provides a neat repository to keep track of all the things that have been tried to fix the problem.
9. 在submit defect之前, 你應該先確認那不是你的問題或是工具的問題
- “Oops, my bad!” is a great way to lose credibility with the people who are going to be fixing your defects.
- If you don’t have credibility, you are going to have to work much harder to convince people that the problem you are seeing is due to a fault with the system rather than a fault with your test scripts.
- Don’t be so afraid of making a mistake that you test “around” errors (like people who see HTTP 500 errors under load and “solve” the problem by changing their scripts to put less load on the system).
10. 你應該要傳遞你的知識給相關的人
- Write a Test Summary Report and let management know what you found (and fixed) during testing, make some PowerPoint slides, hold a meeting.
- Let the Production monitoring group know which metrics are useful to monitor, let them re-use your LoadRunner scripts for Production monitoring with BAC.
- Leave some documentation for future testers; don’t make them gather requirements and transaction volumes again, or re-write all your scripts because they don’t understand them.
- Retain your test results until you are sure that no-one is going to ever ask about the results of that test you ran all those months ago.
http://www.myloadtest.com/ten-commandments-of-load-testing/
以下是你在做load testing, 要小心的十件事情
1. 你需要知道test toll是如何運作的
- The worst performance testers I have met were always more concerned about whether they could get their scripts to run, rather than whether the tests they were running were realistic.
- Read the documentation, practice, spend some time figuring out what all the settings do, then relate how your scripts are running back to how real users exercise your application.
2. 你需要收集真實的在運作或使用的資料
- Garbage in, garbage out. If your transaction volumes are wrong, then your load test is wrong.
3. 你需要有可測試的requirements.
- Non-functional requirements (especially load and performance-related requirements) are usually an afterthought for many projects.
- This shouldn’t stop you from trying to gather the requirements you need for your tests.
- The business approach of “let us know how fast it is, and we will let you know if that’s okay” isn’t good enough.
- Get some numbers. The numbers can change in the future (maybe call them “targets” or “guidelines” rather than “requirements”), but you need something to test against before you start.
4. 你應該要撰寫測試計劃
- Even if you already know what you’re going to be doing, other people would probably like to know too - they might even be able to help.
- A a signed-off test plan has saved many a tester from the wrath of project management.
5. 你應該要測試最差的狀況
- Don’t test with transactions from an average day, test for the busiest day your business has ever had.
- Add a margin for growth.
- Testing failover? A server doesn’t fall over at midnight when no one is using your application (would we care in this situation anyway?), it falls over in the middle of the day when lots of real people are using it.
6. 你需要監控你的測試環境
- Monitoring your servers allows you to more easily figure out where the problem is.
- You can also make neat observations like “response times for the new version of the application are the identical to the previous version, but CPU utilisation on the servers has increase by 10%”
- When I say “monitor your servers”, this includes your load generators.
7. 必須要對你的測試環境做change control
- The final thing you tested should be what is deployed into Production - same application version, same system configuration.
- It’s easy to lose track of what you are actually testing against if people are making uncontrolled changes to your environment, or if people are making tuning changes without tracking what they are changing.
- Keep a list of changes that are made…even if you are in a hurry; and always make sure you know what you are testing against.
8. 你應該要使用 defect tracking tool.
- An untracked defect is a little like a tree that fall in the forest when no-one is around - no-one cares.
- Raising defects lets everyone know there is a problem
- It also provides a neat repository to keep track of all the things that have been tried to fix the problem.
9. 在submit defect之前, 你應該先確認那不是你的問題或是工具的問題
- “Oops, my bad!” is a great way to lose credibility with the people who are going to be fixing your defects.
- If you don’t have credibility, you are going to have to work much harder to convince people that the problem you are seeing is due to a fault with the system rather than a fault with your test scripts.
- Don’t be so afraid of making a mistake that you test “around” errors (like people who see HTTP 500 errors under load and “solve” the problem by changing their scripts to put less load on the system).
10. 你應該要傳遞你的知識給相關的人
- Write a Test Summary Report and let management know what you found (and fixed) during testing, make some PowerPoint slides, hold a meeting.
- Let the Production monitoring group know which metrics are useful to monitor, let them re-use your LoadRunner scripts for Production monitoring with BAC.
- Leave some documentation for future testers; don’t make them gather requirements and transaction volumes again, or re-write all your scripts because they don’t understand them.
- Retain your test results until you are sure that no-one is going to ever ask about the results of that test you ran all those months ago.
- Nov 07 Fri 2008 09:35
Performance Techniques, Building Codes, and Approach
Performance Techniques, Building Codes, and Approach
http://blogs.msdn.com/jmeier/archive/2008/01/14/performance-techniques-building-codes-and-approach.aspx
一般人在做performance enhancement時, 常常不知道要怎麼下手. 因為學校裡不會教這些東西, 上班之後也沒人重視這些事情. 等到問題發生時, 往往不知道要如何下手. 作者整理了一些資料, 告訴你在面對不同狀況或不同問題時, 你可以去哪裡找資料. 它所提到範圍還真的很廣, 我想還蠻有幫助的.
(1) Performance Modeling
http://www.guidanceshare.com/wiki/Performance_Modeling
(2) Performance Design Inspection
- Nov 06 Thu 2008 09:32
Principles of Effective Reporting
Principles of Effective Reporting
from Performance Testing Guidance for Web Applications
Chapter 16 – Performance Test Reporting Fundamentals
產生一份好的rerpot, 是performance testing重要的事情. 因為即使你前面做的再好, 若是在report上失敗了, 大家還是無法容易recognize 你的功勞, 所謂佛要金裝, 人要衣裝. 若是沒無法適時包裝你的功勞, 很多時候會容易讓自己frustrated
這本書的作者介紹了一些在做report時所要注意的事情, 我想很值得大家去看看.
‧Report early, report often
- Oct 31 Fri 2008 09:33
Benchmarking
Benchmarking
from Performance Testing Guidance for Web Applications
在建立完performance baseline之後, 接下來要做的事情便是benchmarking. 什麼是benchmarking呢? Benchmarking就是一個在比較目前系統的速度, 和baseline或是業界標準速度的程序
如果你有baseline, 你就要必須確認你的執行條件, 和所要收集的資料, 要和baseline相同. 這樣你們之間才可以比較, 比較的結果才會有意義.
在做Benchmarking時, 以下幾點是你要考慮的
- Oct 27 Mon 2008 09:30
Baselines Creation
Baselines Creation
在做Performacne Testing時, 有一件很重要的事就是找出baseline. 有了baseline, 才會有比較的基礎. 否則你將無法判斷, 目前turning或是修改的結果是好是壞, 必須要有相對比較的東西
那找出baseline的流程是在做什麼呢, 定義如下:
- the process of running a set of tests
- to capture performance metric data
- for the purpose of evaluating the effectiveness of subsequent performance-improving changes to the system or application.
- Oct 15 Wed 2008 13:17
Visual Studio 2005 Team Edition for Software Testers
Visual Studio 2005 Team Edition for Software Testers
最近在上課時, 有人問我VS 2005 for Software tester到底好不好用, 再加上RD demo了如何用Load Test + Python去測試. 因此花了一點時間, 收集了網路上有關VS 2005 for software tester相關的文章. 看起來大部分還是微軟自家的人!!
1.邱英瑞(Jacky)
http://blogs.msdn.com/jchiou/archive/2008/03/31/visual-studio-team-test.aspx
- 如何錄製一個 Web 測試
- Web 測試之如何將 Web Server 參數化
- Oct 04 Sat 2008 08:04
做Performance Test 時一定要懂點統計
做Performance Test 時一定要懂點統計
Performance Testing Guidance for Web Applications
Chapter 15 Key Mathematic Principles for Performance Testers
- Oct 03 Fri 2008 11:17
如何產生好的performance test report
如何產生好的performance test report
Performance Testing Guidance for Web Applications
Chapter 16 – Performance Test Reporting Fundamentals
在教同事performance testing時, 我和他們說了一件事: 或許你無法記住我們教的流程, 或許你無法很早開始進行測試, 或許你無法自動化, 但是你一定要記得如何產生好的report.
我看到很多同事在交report時, 整份report都是數字或是大量的圖表, 整份report只有他看的懂的東西, 而且也沒有任何summary 或是結論. 我想結果一定沒有人欣賞你的努力, 因為你自己也不被自己的東西感動.
這裡是書上所寫的基本準則
- Sep 30 Tue 2008 14:14
Performance Testing的迷思: 放下去跑就可以了
Performance Testing的迷思: 放下去跑就可以了
每當有engineer要去做performance testing時, 有些upper managers總是會有個錯覺, 認為這個engineer 可以同時再接一些別的事情. 因為他們會說, "不是放下去run就好了嗎? 其他時間就可以來做別的事". 他的意思好像是說performance test 的工作就是丟下跑就是了.
一開始聽到這樣的說法時, 還無法很快說出哪裡有問題. 剛好最近在上performance testing時, 看到書裡面有描述在做performance testing的一些準備工作
Patterns of Performance and Operability,
Chapter 7 Test Preparation and Execution
1. Script Development: If you are using load testing software, you will need to develop scripts that implement the load scenarios contained in your test plan.
- Sep 25 Thu 2008 13:35
Non-functional requirement v.s functional requirement
Non-functional requirement v.s functional requirement
上次在公司內部教育訓練時, 發現很多員工不 知道什麼是non-functional requirment. 我想這個現象在台灣非常普遍, 因為學校通常沒有把軟體工程視為必修, 再加上學生也不會選修它. (都會寫軟體了, 還修什麼軟體工程).
我雖然在課堂上解釋了一下, 但是我還是上網找了一些資料
在Wiki中, 定義是這樣的:
non-functional requirements are requirements which specify criteria that can be used to judge the operation of a system, rather than specific behaviors
跟他相反的functional requirement其定義是:
- Sep 23 Tue 2008 09:39
如何找software performance tester
如何找software performance tester
“…5 or 6 bullet points for a performance tester…”
by Mike Kelly
http://www.michaeldkelly.com/blog/archives/71
在網路上有人提出在interview performance tester時, 要問哪些問題來確認其專業知識是否過關.
基本上每個面向都涵蓋到了, 不過我也很懷疑真的會有人可以面面俱到嗎? 尤其在台灣, item1 滿足並且有意願就已經快要可以進來了
- Sep 11 Thu 2008 23:48
PerfTestPlus: 一個介紹Performance Test超專業的網站
PerfTestPlus
這是目前我在網路上找到介紹Performance Testing最詳盡的網站
http://www.perftestplus.com/index.htm
曾經寫過一些系列文章, 像是
User Experience, not Metrics Series (共13篇)
Beyond Performance Testing Series (共14篇)
- Sep 10 Wed 2008 22:40
微軟Performance Testing 的步驟
微軟有出一本書叫
Improving .NET Application Performance and Scalability
這本書是在描述微軟如何做Performance engineering. 其中有一章節是在描述如何做performance testing. 我節錄了一小段讓大家聞香一下. 有興趣的人可至微軟網站download這本書. 不過有一千多頁, 要花不少時間才能看完.
Step 1: Identify key scenarios
(1) Scenarios are anticipated user paths that generally incorporate multiple application activities
- Sep 09 Tue 2008 06:59
The definition of Load/Stress/Performance Test
The definition of Load/Stress/Performance Test
Most people don't know how to distinguish between load, stress and performance testing. Therefore I do a little survey to confirm the definition of load, stress and performance testing. I hope it's helpful to let everyone can understand them well.
1. The Art of Software Testing, Glenford J. Myers
A. Volume Testing
Volume testing is subjecting the program to heavy volumes of data.
- Sep 07 Sun 2008 17:40
Performance Testing相關的書籍
Performance Testing的書其實也不多, 但至少比agile testing多一點. 而且大多是Microsoft所寫的書. 軟體大廠是真的有他厲害的地方, 雖然你還覺得Microsoft的Bug不少, 但是他在各方面還是下了不少功夫, 有許多地方還是值得我們借鏡
1. Integrated Approach to Web Performance Testing: A Practitioner's Guide
by B. M. Subraya
http://www.amazon.com/Integrated-Approach-Performance-Testing-Practitioners/dp/1591407850/ref=sr_1_2?ie=UTF8&s=books&qid=1220702000&sr=1-2
