David Ko的學習之旅

要多快才是夠快

Part 3: How Fast is Fast Enough?
http://www.perftestplus.com/resources/BPT3.pdf

Posted by Scott Barber

接著上次那篇"可接受的應用程式反應時間 v.s. 業界標準"(http://www.wretch.cc/blog/kojenchieh/15275061), 我又找出同一個作者所寫的: "How Fast is Fast Enough?", 希望來更清楚一步了解, 怎樣解決到底快不快的問題

同樣在這篇文章中, 作者依舊說快不快這件事, 是要根據情況而定, 並且再次強調沒有所謂的業界標準.

他引用他朋友Joe Strazzere所說的話:
“There are no industry standards.

You must analyze your site in terms of who the customers are, what their needs are, where they are located, what their equipment and connection speed might be, etc., etc.

I suspect 1.5 seconds would be a rather short interval for many situations.

Do you really require that quick of a response?”

也就是說, 不同狀況有不同答案. 那你說這不是廢話嗎? 讓我門一起來看看作者怎麼接下去分析

1. 考慮會影響你對校能期望的因素
讓我們先討論, 那些主要因素, 會影響我們思考"快"到底是什麼? 作者認為有以下三點

A. 使用者的心理(User psychology)
- 這是最容易被忽略的因素, 有時候系統已經很慢了, 但是使用者預期系統應該會很慢, 所以相對就不會苛求它, 導致我們忽略了其實系統在這狀況下其實是很慢的
- 例如: 在看summary的資料, 或是在建立report時, 系統可能會提醒使用者這會需要花數分鐘到數小時, 因此使用者自然會去做其他事情, 或是自然覺得曼是很合理的. 很容易不去深究這樣長的時間可能是不合理的
- 開發人員最容易有這樣的心理, 會認為產生報表或是某些複雜計算, 本來就需要較多的時間. 可以卻沒有再進一步去思考, 到底是真的要用這麼多時間, 還是程式方面其實是可以改善的

B. System considerations
- 這是最容易考慮的因素
- 若是要跑的快一點, 自然會要有好的硬體配備, 否則自然是跑不動的
- 通常會考慮以下事情:
  # system hardware
  # network and/or Internet bandwidth of the system
  # geographical replication
  # software architecture

C. Usage considerations
- 和User psychology很類似, 但不一樣. Usage consideration是要考慮系統如何被使用. 不同用法的系統, 其速度被要求的不同
- 所謂"快", 是跟其他系統比.內部使用要輸入大量資料的系統, 可能就要求要比網站購物系統快. 每天工作要用的系統, 就會要求要比公佈服委會資訊的系統快.
- 也就是說, 知道系統會被怎麼使用,才能知道預期的使用者對系統速度容忍的程度為何

2. 如何收集有關效能的需求
當你知道大家如何來看待系統的速度後, 接下來如何將這些期待轉換成系統效能的需求規格. 這裡有三個方向是需求的來源, 需要加以考量:

A. User expections
- 使用者只關心end-to-end response time, 她們並不在呼同時間有多少使用者在使用
- 當從舊系統換成新系統時, 使用者第一個可以回答的是他對系統效能的感覺. 因為新功能可能他無法這麼快會用, 但是只要他原先的功能用的不順, 一定會馬上反應.
- 若是你問使用者系統有多快, 或是要多快, 她們無法回答你精準到秒的地步, 最多跟你說他感覺到fast, typical, or slow.\
- 作者自己也研究過許多調查報告, 發現它們的調查並不完全可靠. 所以他只能告訴你他工作經驗得到的答案. 但是千萬不要把它當標準
  # no delay or fast – under 3 seconds
  # typical – 3 to 5 seconds
  # slow – 5 to 8 seconds
  # frustrating – 8 to 15 seconds
  # unacceptable – more than 15 seconds
- 若是高度互動式系統, 可能要比上述快20%, 如果是static or recreational sites, 可能比上述慢25%都還可以接受
- 如果有些工作真的需要做很久, 若系統能顯示提示, 告知目前已做了多少, 花了多少時間, 或是還需多少時間. 都會讓使用者可以接受, 而不至於覺得太慢

B. Resource limitations
- Time, money, people, hardware, networks, and software這些都會影響系統的速度
- 知道這些限制, 才能知道你怎樣設計最務實

C. Stakeholder expections
- 這最容易得到, 老闆說了算
- 有時候老闆會說依照業界標準, 可是通常沒有業界標準. 若是沒有, 老闆通常會叫你儘可能的fast和scalable, 直到他們發現這會很多時間和精力才能達成. 簡單說, 他們要最快的系統, 但是要花最少的代價.
- 我們的責任是幫助stakeholder去決定, 量化, 以及管理效能的期待.
  # User expectations very rarely change
  # Resource limitations are generally fairly static throughout a performance testing/engineering effort
  # Stakeholder expectations, however, are likely to change when decisions have to be made about trade-offs.
 

3. 決定和紀錄效能的需求

一但你收集到上述的資訊後, 你要做的就是產生出一份meaningful, quantifiable,and testable requirements. 這不是很容易的事, 並且它是一個iterative的過程. 要不斷給相關人我們所知道的東西, 讓他們能盡快給我們feedback, 好讓我們能持續改進.

作者認為效能是由三個東西所組成的
A. Speed Requirement
- Remember that we want to focus on end user response time
- 作者曾對速度的需求做分類, 然後在將系統內的功能或頁面歸納到這些分類中.
  # normal pages – typical to fast
  # reports – under a minute
  # exception activities (list) – fast to very fast
  # query execution – under 30 seconds
  # nightly backup batch process – under an hour
- 最後再對這些分類訂出想要的requirement 和goal

B. Scalability Requirement
- Scalability requirements are the "how much" and "how many" questions that go with the "how fast" of the speed requirements
- 可以在"User Experience, Not Metrics, Part 4: Modeling Groups of Users."找到更多參考資料
- 以下是一些例子
  # peak expected hourly usage – 500 users
  # peak expected sustained usage – 300 users
  # maximum percentage of users expected to execute reports in any one hour – 75%
  # maximum percentage of users expected to execute queries in any one hour – 75%
  # maximum number of rows to be replicated during nightly backup – 150,000
- 另外一個主題是 user abandonment, 在"Part 4: Accounting for User Abandonment"有更詳細的介紹. 它也會對效能有很大的影響.

C. Stability Requirement
- Stability通常所包含的範圍很廣,主要是在處理"What will the system do if . . . ?"的事情
- 不過這裡我們只處理usage stability, 並不管data recovery, fail-over, or disaster recovery
- 以下是一些例子
  # System returns to expected performance within five minutes after the occurrence of an extreme
usage condition, with no human interaction.
  # System displays a message to users informing them of unexpected high traffic volume and
requests they return at a later time.
  # System automatically recovers with no human interaction after a reboot/power down.
  # System limits the total number of users to a number less than that expected to cause significant performance degradation.
 

4. Composite Requirements
通常效能測試的需求是單獨出現的, 無法分開來看. 因此作者認為這是很常見的, 需要把他們組合在一起. 以下是他所舉的一些例子
A. Composite Requirements
1). The system exhibits not more than a 5-second response time for normal pages and meets all exception requirements, via intranet, 95% of the time under an extended 300-hourly-user load (in accordance with the user community model) with less than 5% user abandonment.
2). The system exhibits not more than a 5-second response time for normal pages and meets all exception requirements, via intranet, 90% of the time under a 500-hourly-user load (in accordance with the user community model) with less than 10% user abandonment.
3). All exception pages exhibit not more than a 3-second response time 95% of the time, with no user abandonment, under the conditions in items 1 and 2 above.
4). All reports exhibit not more than a 60-second response time 95% of the time, with no user abandonment, under the conditions in items 1 and 2 above.
5). All reports exhibit not more than a 60-second response time 90% of the time, with less than 5% user abandonment, under the 75% report load condition identified in our scalability requirements.
6). All queries exhibit not more than a 30-second response time 95% of the time, with no user abandonment, under the conditions in items 1 and 2 above. 

B.Composite Goals
1). The system exhibits not more than a 3-second response time for normal pages and meets all exception requirements, via intranet, 95% of the time under a 500-hourly-user load (in accordance with the user community model) with less than 5% user abandonment.
2). All exception pages exhibit not more than a 2-second response time 95% of the time, with no user abandonment, under the conditions in item 1 above.
3). All reports exhibit not more than a 60-second response time 95% of the time, with no user abandonment, under the conditions in items 1 and 2 above.
4). All reports exhibit not more than a 30-second response time 95% of the time, with no user abandonment, under the conditions in item 1 above.
5). All reports exhibit not more than a 30-second response time 90% of the time, with less than 5% user abandonment, under the 75% report load condition identified in our scalability requirements.
6). All queries exhibit not more than a 15-second response time 95% of the time, with no user abandonment, under the conditions in item 1 above.