The 10 Commandments of Load Testing
http://www.myloadtest.com/ten-commandments-of-load-testing/
以下是你在做load testing, 要小心的十件事情
1. 你需要知道test toll是如何運作的
- The worst performance testers I have met were always more concerned about whether they could get their scripts to run, rather than whether the tests they were running were realistic.
- Read the documentation, practice, spend some time figuring out what all the settings do, then relate how your scripts are running back to how real users exercise your application.
2. 你需要收集真實的在運作或使用的資料
- Garbage in, garbage out. If your transaction volumes are wrong, then your load test is wrong.
3. 你需要有可測試的requirements.
- Non-functional requirements (especially load and performance-related requirements) are usually an afterthought for many projects.
- This shouldn’t stop you from trying to gather the requirements you need for your tests.
- The business approach of “let us know how fast it is, and we will let you know if that’s okay” isn’t good enough.
- Get some numbers. The numbers can change in the future (maybe call them “targets” or “guidelines” rather than “requirements”), but you need something to test against before you start.
4. 你應該要撰寫測試計劃
- Even if you already know what you’re going to be doing, other people would probably like to know too - they might even be able to help.
- A a signed-off test plan has saved many a tester from the wrath of project management.
5. 你應該要測試最差的狀況
- Don’t test with transactions from an average day, test for the busiest day your business has ever had.
- Add a margin for growth.
- Testing failover? A server doesn’t fall over at midnight when no one is using your application (would we care in this situation anyway?), it falls over in the middle of the day when lots of real people are using it.
6. 你需要監控你的測試環境
- Monitoring your servers allows you to more easily figure out where the problem is.
- You can also make neat observations like “response times for the new version of the application are the identical to the previous version, but CPU utilisation on the servers has increase by 10%”
- When I say “monitor your servers”, this includes your load generators.
7. 必須要對你的測試環境做change control
- The final thing you tested should be what is deployed into Production - same application version, same system configuration.
- It’s easy to lose track of what you are actually testing against if people are making uncontrolled changes to your environment, or if people are making tuning changes without tracking what they are changing.
- Keep a list of changes that are made…even if you are in a hurry; and always make sure you know what you are testing against.
8. 你應該要使用 defect tracking tool.
- An untracked defect is a little like a tree that fall in the forest when no-one is around - no-one cares.
- Raising defects lets everyone know there is a problem
- It also provides a neat repository to keep track of all the things that have been tried to fix the problem.
9. 在submit defect之前, 你應該先確認那不是你的問題或是工具的問題
- “Oops, my bad!” is a great way to lose credibility with the people who are going to be fixing your defects.
- If you don’t have credibility, you are going to have to work much harder to convince people that the problem you are seeing is due to a fault with the system rather than a fault with your test scripts.
- Don’t be so afraid of making a mistake that you test “around” errors (like people who see HTTP 500 errors under load and “solve” the problem by changing their scripts to put less load on the system).
10. 你應該要傳遞你的知識給相關的人
- Write a Test Summary Report and let management know what you found (and fixed) during testing, make some PowerPoint slides, hold a meeting.
- Let the Production monitoring group know which metrics are useful to monitor, let them re-use your LoadRunner scripts for Production monitoring with BAC.
- Leave some documentation for future testers; don’t make them gather requirements and transaction volumes again, or re-write all your scripts because they don’t understand them.
- Retain your test results until you are sure that no-one is going to ever ask about the results of that test you ran all those months ago.
http://www.myloadtest.com/ten-commandments-of-load-testing/
以下是你在做load testing, 要小心的十件事情
1. 你需要知道test toll是如何運作的
- The worst performance testers I have met were always more concerned about whether they could get their scripts to run, rather than whether the tests they were running were realistic.
- Read the documentation, practice, spend some time figuring out what all the settings do, then relate how your scripts are running back to how real users exercise your application.
2. 你需要收集真實的在運作或使用的資料
- Garbage in, garbage out. If your transaction volumes are wrong, then your load test is wrong.
3. 你需要有可測試的requirements.
- Non-functional requirements (especially load and performance-related requirements) are usually an afterthought for many projects.
- This shouldn’t stop you from trying to gather the requirements you need for your tests.
- The business approach of “let us know how fast it is, and we will let you know if that’s okay” isn’t good enough.
- Get some numbers. The numbers can change in the future (maybe call them “targets” or “guidelines” rather than “requirements”), but you need something to test against before you start.
4. 你應該要撰寫測試計劃
- Even if you already know what you’re going to be doing, other people would probably like to know too - they might even be able to help.
- A a signed-off test plan has saved many a tester from the wrath of project management.
5. 你應該要測試最差的狀況
- Don’t test with transactions from an average day, test for the busiest day your business has ever had.
- Add a margin for growth.
- Testing failover? A server doesn’t fall over at midnight when no one is using your application (would we care in this situation anyway?), it falls over in the middle of the day when lots of real people are using it.
6. 你需要監控你的測試環境
- Monitoring your servers allows you to more easily figure out where the problem is.
- You can also make neat observations like “response times for the new version of the application are the identical to the previous version, but CPU utilisation on the servers has increase by 10%”
- When I say “monitor your servers”, this includes your load generators.
7. 必須要對你的測試環境做change control
- The final thing you tested should be what is deployed into Production - same application version, same system configuration.
- It’s easy to lose track of what you are actually testing against if people are making uncontrolled changes to your environment, or if people are making tuning changes without tracking what they are changing.
- Keep a list of changes that are made…even if you are in a hurry; and always make sure you know what you are testing against.
8. 你應該要使用 defect tracking tool.
- An untracked defect is a little like a tree that fall in the forest when no-one is around - no-one cares.
- Raising defects lets everyone know there is a problem
- It also provides a neat repository to keep track of all the things that have been tried to fix the problem.
9. 在submit defect之前, 你應該先確認那不是你的問題或是工具的問題
- “Oops, my bad!” is a great way to lose credibility with the people who are going to be fixing your defects.
- If you don’t have credibility, you are going to have to work much harder to convince people that the problem you are seeing is due to a fault with the system rather than a fault with your test scripts.
- Don’t be so afraid of making a mistake that you test “around” errors (like people who see HTTP 500 errors under load and “solve” the problem by changing their scripts to put less load on the system).
10. 你應該要傳遞你的知識給相關的人
- Write a Test Summary Report and let management know what you found (and fixed) during testing, make some PowerPoint slides, hold a meeting.
- Let the Production monitoring group know which metrics are useful to monitor, let them re-use your LoadRunner scripts for Production monitoring with BAC.
- Leave some documentation for future testers; don’t make them gather requirements and transaction volumes again, or re-write all your scripts because they don’t understand them.
- Retain your test results until you are sure that no-one is going to ever ask about the results of that test you ran all those months ago.
