David Ko的學習之旅

如何產生好的performance test report
Performance Testing Guidance for Web Applications
Chapter 16 – Performance Test Reporting Fundamentals

Performance Testing的迷思: 放下去跑就可以了
每當有engineer要去做performance testing時, 有些upper managers總是會有個錯覺, 認為這個engineer 可以同時再接一些別的事情. 因為他們會說, "不是放下去run就好了嗎? 其他時間就可以來做別的事". 他的意思好像是說performance test 的工作就是丟下跑就是了.
一開始聽到這樣的說法時, 還無法很快說出哪裡有問題. 剛好最近在上performance testing時, 看到書裡面有描述在做performance testing的一些準備工作

Non-functional requirement v.s functional requirement
上次在公司內部教育訓練時, 發現很多員工不 知道什麼是non-functional requirment. 我想這個現象在台灣非常普遍, 因為學校通常沒有把軟體工程視為必修, 再加上學生也不會選修它. (都會寫軟體了, 還修什麼軟體工程).
我雖然在課堂上解釋了一下, 但是我還是上網找了一些資料

如何找software performance tester
“…5 or 6 bullet points for a performance tester…”
by Mike Kelly
http://www.michaeldkelly.com/blog/archives/71

PerfTestPlus
這是目前我在網路上找到介紹Performance Testing最詳盡的網站
http://www.perftestplus.com/index.htm

微軟有出一本書叫 
Improving .NET Application Performance and Scalability
這本書是在描述微軟如何做Performance engineering. 其中有一章節是在描述如何做performance testing. 我節錄了一小段讓大家聞香一下. 有興趣的人可至微軟網站download這本書. 不過有一千多頁, 要花不少時間才能看完.
Step 1: Identify key scenarios
(1) Scenarios are anticipated user paths that generally incorporate multiple application activities
(2) Scenarios are those for which
    - Have specific performance goals
    - Have a significant  performance impact (common executed or resource intensive)
(3) Example
    - Log on to the application
    - Browse a product catalog
    - Search for a specific product
    - Add items to the shopping cart
    - Validate credit card details and place an order
   
Step 2: Identify Workload
(1) Identity the performance characteristics or workload associated with each of the defined scenarios
(2) For each scenarios you must identify the following:
    - Number of users:
      * the total number of concurrent and simultaneous users who access the application in a given time frame
    - Rate of requests
      * The requests received from the concurrent load of users per unit time
    - Pattern of requests
      * A given load of concurrent users may be performing different tasks using the application
      * Pattern of requests identify the average load of users, and rate of requests for a given functionality of an application

The definition of Load/Stress/Performance Test
Most people don't know how to distinguish between load, stress and performance testing. Therefore I do a little survey to confirm the definition of load, stress and performance testing. I hope it's helpful to let everyone can understand them well.
 
1. The Art of Software Testing, Glenford J. Myers
A. Volume Testing
Volume testing is subjecting the program to heavy volumes of data.
For instance, compiler would be fed an absurdly large source program to compile.
The purpose of volume testing is to show that the program cannot handle the volume of data specified in its objectives.
 
B. Stress Testing
Stress testing involves subjecting the program to heavy loads or stresses. This should not be confused with volume testing, a heavy stress is a peak volume of data encountered over a short span of time.
Am analogy is an appraisal of a typist. A volume test is the determination of whether the typist can cope with a draft of a large report; a stress test is the determination of whether the typist can type at a rate of 50 words per minute.
 
C. Performance Testing
The purpose of performance testing is to demonstrate that the program does not meet its objectives, test cases must be devised that attempt to show that the program does not satisfy its performance objectives.
 
=========================================================================
2. WiKi
A. Load Testing
Load testing is the the process of running a number of clients simultaneously to test a client-server system and measure response times
Load testing generally refers to the practice of modeling the expected usage of a software program by simulating multiple users accessing the program's services concurrently. As such, this testing is most relevant for multi-user systems, often one built using a client/server model, such as web servers.
When the load placed on the system is raised beyond normal usage patterns, in order to test the system's response at unusually high or peak loads, it is known as Stress testing. The load is usually so great that error conditions are the expected result, although there is a gray area between the two domains and no clear boundary exists when an activity ceases to be a load test and becomes a stress test
 
B. Stress Testing
Stress testing is a form of testing that is used to determine the stability of a given system or entity. It involves testing beyond normal operational capacity, often to a breaking
 
C. Performance Testing
In software engineering, performance testing is testing that is performed to determine how fast some aspect of a system performs under a particular workload.
Performance testing can serve different purposes. It can demonstrate that the system meets performance criteria.
It can compare two systems to find which performs better. Or it can measure what parts of the system or workload cause the system to perform badly. In the diagnostic case, software engineers use tools such as profilers to measure what parts of a device or software contribute most to the poor performance or to establish throughput levels (and thresholds) for maintained acceptable response time. It is critical to the cost performance of a new system that performance test efforts begin at the inception of the development project and extend through to deployment. The later a performance defect is detected, the higher the cost of remediation. This is true in the case of functional testing, but even more so with performance testing, due to the end-to-end nature of its scope.
 
=========================================================================
3. Software Testing and Continuous Quality Improvement, William E. Lewis
A. Volume Testing
The objective of volume testing is to subject the system to heavy volumes of data to find if it can handle the volume of data.
Stress testing subjects the system to heavy loads or stresses in term of rates, such as throughputs over a short time period.
Volume testing is data oriented and its purpose it to show that the system can handle the volume of data specified in its objectives
 
B. Stress Testing
The objective of stress testing is to investigate the behavior of the system under conditions that overload its resources. Of particular interest is the impact that this has on the system processing time.
Stress testing is boundary testing. For example, test with the max number of terminals active and then add more terminals than specified in the requirements under different limit combinations
Stress testing studies the system's response to peak bursts of activity in short period of time and attempts to find defects in a system.
It is often confused with volume testing, in which the system's capability of handling large amounts of data is the objective.
 
C. Performance Testing
The objective of performance testing is to measure the system against predefined objectives. The required performance levels are compared against the actual performance keveks and discrepancues are documented.
 
=========================================================================
4. Improving .NET Application Performance and Scalability
A. Volume Testing
Use load testing to verify application behavior under normal and peak load conditions.
You incrementally increase the load from normal to peak load to see how your application performs with varying load conditions.
You continue to increase the load until you cross the threshold limit for your performance objectives.
 
B. Stress Testing
Use stress testing to evaluate your application's behavior  when it's pushed beyond the normal or peal load conditions
Stress test your application by subjecting it to very high loads that are beyond the capacity of the application, while denying it the resources required to process that load.

Performance Testing的書其實也不多, 但至少比agile testing多一點. 而且大多是Microsoft所寫的書. 軟體大廠是真的有他厲害的地方, 雖然你還覺得Microsoft的Bug不少, 但是他在各方面還是下了不少功夫, 有許多地方還是值得我們借鏡
1. Integrated Approach to Web Performance Testing: A Practitioner's Guide
by B. M. Subraya
http://www.amazon.com/Integrated-Approach-Performance-Testing-Practitioners/dp/1591407850/ref=sr_1_2?ie=UTF8&s=books&qid=1220702000&sr=1-2