GUI Automation真的值得嗎? Part II
在post 完"GUI Automation 真的值得嗎?"後,公司同事問我為何我們公司大部分的人, 都是還在做GUI test automation. 那時候我回答的答案是
(1) QA is not familiar with RD’s codes.
(2) QA does not have strong coding skill, so he is not good at handle this kind of thing
(3) RD has no enough time to provide testability to let QA to do testing from internalmodule.
(4) ETS or company always promotes GUI testing tool because it is the most maturetesting tool.
(5) QAis not good at create homegrown testing tool or framework, so he just uses the tools company suggests.
後來我在網路上找到相關的討論, 覺得還不錯, 大家可以看看不同人的論點是什麼.
(本來想把他們大意翻譯一下, 後來我放棄了, 請自行看吧)
GUI schmooey
- alanpa
http://blogs.msdn.com/alanpa/archive/2008/09/18/gui-schmooey.aspx
有人問作者有關GUI automation test的問題:
(1) 推薦給不會coding QA用的GUI automation tool
(2) 對於用 MVC架構的系統上, 會花多少時間在GUI automation 上?
作者的答案很勁爆
(1) 他測試UI都是用手動測試, 如果用tool, TOI會不高
(2) 他建議要做test automation, 最好by pass GUI, 直接由底層來做test automation
簡單來說, 作者認為
For 95% of all software applications, automating the GUI is a waste of time.
頭先作者還打算寫99%, 後來他自己沒種才改成95%. 他說他並不是反對test automation, 但是他認為automation要做在UI 下面的system才有意義. 並且他也不是反對做GUI testing, 只是覺得用手動作就可以.
可是之後還是引起眾人的討論, 讓我們來看看大家怎麼說:
==================================================
Shrini 說
這個敘述的確聳動, 但是對於 outsourced的狀況, 他還是成立嗎?
因為大部分outsourced的狀況下, 大部分的test automation 都是發生在GUI layer. 他們叫做GUI regression automation. 這是他們做automation最容易可以做到, 並且你一說automation, 他們就立即想到 QTP 或是其他COTS 的工具.
那為什麼 GUI層以下的automation無法在業界這麼popular?
Shrini認為理由有三
1. Automation Engineers (called "wrongly" so) are erstwhile black box testers with only "tool knowledge". Less than 5% of this crowd is programming knowledge. Some of these are even mediocre testers.
2. Applications are legacy and are in maintenance mode. Application devleopment is outsourced to another vendor hence really there is a WALL between GUI automation and development team
3. Whole concept of automation under the hood - likes of TDD, xUnit frameworks etc is alien to IT testing/automation world. Very few have now started to look beyond GUI COTS automation tools due to cost aspects.
==================================================
Shane MacLaughlin 說
Shane認為業界大部分的狀況是, tester 是無法access到 GUI 層以下的東西. 或許經由GUI 層來做test automation是比較沒有效率的, 但是它卻是最務實的方法. 而且藉由一些好的framework, 可能還會運作的不錯. 所以Shane認為95%這件事是非常違背事實
當然啦, 如果說能在design stage, 就可以規劃及執行automation test, 這一定是會比在GUI層做test automation好. 在這點上, Shane也是完全同意alanpa的說法.
Ideally, I'd like to have a recorder that recorded at interface object level rather than GUI level, and have pestered a couple of the COTs vendors to allow hooks into their apps to do this.
For many graphically intensive domains, such as CAD, GIS, etc... the user interface is still the best place to generate the test cases. Not much response so far...
==================================================
Brent Paine說
Monday, September 22, 2008 9:04 AM by Brent Paine
I tend to agree with Shane on this one.
The whole 95% theory would work great if we all agreed that an application is the sum of all it's individual parts. However, I don't think it leaves space for some important elements of testing.
As someone who worked for a year and a half on a single project, and then be asked to automate it, I might look at your statement and say, "GREAT!" and go off on my merry way and automate the application without issue. That's because my domain expertise will be sufficient enough to tell me when what I'm doing is stupid.
For someone who doesn't have adequate domain experience, the problem that presents itself is they will begin attempting to automate things that would never interact, their automation flow would be completely irrelevant.
So as far as I see it, the 95% line is probably going to end up being the smartest thing anybody has ever said, to some, and the stupidist thing anyone has ever said to others.
One point I do agree with you on, though, is that I find it very difficult to justify automating GUI to the point of verifying that buttons are in the right place or the font is correct, etc. However, I think that the GUI is a great vessel for someone who has little or no experience on the application to use their QA knowledge to estimate how the program should flow.
==================================================
alanpa 回應以上人的質疑
Monday, September 22, 2008 10:34 AM by alanpa
I stand by my statement. IN fact, I shuddered a bit when Shane used the word "pragmatic" to use fragile tools.
I have written automated gui tests for entire applications using some of these frameworks, and they all break down in future revs of the application, or when the underlying application changes.
If your application isn't designed well (well designed applications never put the logic in the GUI - if you're stuck with one of these applications, you're going to have bad quality regardless of whether you write automated tests or not, so why bother?
==================================================
Shane MacLaughlin說
(這個回應值得看, 妳可以檢視你在做UI testing framework時, 是否有考慮到這些東西)
Monday, September 22, 2008 11:07 AM by Shane MacLaughlin
So if you're presented with an application that has been written without an object level automation interface, you deem it badly designed, and hence not a candidate for automation? Certainly not in my experience.
Like yourself, I have written automation frameworks to drive testing through a GUI, which years later are still running just fine. One of the principal purposes of the framework in this instance is to abstract the GUI to logical object style interfaces, such that the tests are not fragile. It might not be as robust in most cases as using a COM interface, but that doesn't render it either unworkable or a waste of time. You simply have to be aware of, and cater for, the pitfalls.
(這裡是重要的論述, 如果你的UI testing framework必須要夠彈性, 例如Shane寫的能abstract GUI to logical object style interface, 我想或許GUI Tes automation是可以做的. 但是問問你自己, 你的UI testing framework有這麼有彈性嗎?)
It is also worth noting that object based interfaces can change from release to release, breaking object based testing. It is bad programming, sure, but what is the point of testing if not to pick up on bad programming. Similarly, the GUI shouldn't contain any logic, or work differently from other interfaces, but it isn't always the case. Good designs can be badly implemented.
Don't get me wrong, testing through a programmatic or object based interface is way superior if its available. If it isn't, IMO, GUI is a viable alternative in many cases.
==================================================
alanpa 說
Monday, September 22, 2008 11:20 AM by alanpa
To me, part of good testing is having sufficient early influence in the product to ensure I never had a poorly designed application plopped on my lap. However, I realize that for many testers, this isn't a possible solution (see my post on the Test Test for more information).
The point I didn't make strong enough is that whole, of course, it's possible to drive an application purely through GUI automation, this is one are of automation where I worry that the time investment isn't worth the effort vs. just performing manual testing on the GUI in the first place.
==================================================
Brent Paine說
(這個回應也值得看, 一個好的UI testing framework要注意什麼. As I said before: architecture design)
Monday, September 22, 2008 11:25 AM by Brent Paine
I don't think that Shane is referring to record and playback tools. I don't think that any of these tools are all that useful, for the reason you state above. In general, the application will always break.
What Shane does mention is having a solid framework. I can completely agree with this. I actually designed the majority of my most recent project at the GUI layer. However, I used my experience in automation to avoid the pitfalls that often plague automation projects. In the end, we've managed to create a project that has actually just successfully bridged two separate versions of the software.
This automation suite was built entirely through the GUI, using manual test cases, with the exception that we understood that things happening at this layer were likely to change. We used a record and playback tool, but one with a robust scripting engine also. In the end we probably used the record and playback 1% of the time. Maybe for identifying objects, maybe to see how the scripting engine wants us to express something.
In the end, we had a substantial suite which included 3 separate layers, the global framework, the project framework, and the test cases. Built in a heirachal system, the global framework was designed to be portable to other projects. It can be used across both Windows and Java platforms (which was the requirement of our organization) and also acts as a communication medium with the automation tool. The project framework utilizes the global framework, but also implements simple elements related specifically to the project itself. At the test cases layer, I expect to see function calls over tons of scripting. When I do see scripting at the test cases layer, I question it.
In the end, the project has survived the transition from version 1 to version 2 of the software, even though each test case is designed right off the GUI elements.
So, maybe I misunderstand what you mean by testing at the GUI level, but in my experience, if you don't plan ahead then you will ultimately fail anyway. Sometimes, even the best laid plans will fail. However, I think that a poor plan will fail regardless of where you decide to put your chips.
==================================================
TheUIGuy說
(終於有人出來幫alanpa解釋, 可是之後還是被修理)
Monday, September 22, 2008 12:02 PM by TheUIGuy
I am having this discussion at work right now, even after 7 years of only having manual smoke testing. The pattern is this: java or OOP engineers don't write the proper unit tests for something, and the customer sees it. Then we scramble to figure out what went wrong, and bolster the tests - meanwhile, the UI area (presentation layer) gets the "radioactive fallout" from these discussions and is forced to consider automated testing - when all the while it is not a paradigm that the UI ever even considered, or for that matter, would ever need to do.
Then we politely explain that the UI never introduces bugs that affect the customer in any meaningful way, and if so, occur with much less regularity - where most of them are caught during smoke testing. So, of the average 1% of bugs that come from the UI side, we need to integrate a slow, murky, painful automated testing framework? To reduce .07% of bugs to what? .02%.
Because we all have eyeballs and fingers. This is the best way to test your UI. It is now, and it always will be.
Not to do this begs the question: do you care about your users? Do you care about usability? Because automated tests, as error-prone and time-consuming as they are, do not care, and could never cover the use cases that fingers and eyeballs can.
The major issue people fail to get over is that there *ALWAYS* will be bugs.. the question is, how can we minimize those bugs?
Relying on a testing framework, where the engineer has to "guess" what may or may not be testable, and is often wrong, is like saying that testing is simply not important enough to be given the same level of human interaction that coding did in the first place.
So, its all this work, for very little benefit. Thats the viewpoint from the UI side. Java engineers really do need this, because their environments are homogenous and have many different consumers touching it all day long. If a build fails, it fails. They need unit testing and smoke testing.
The ui can benefit from change control procedures, modularization, tested frameworks like mooTools or jQuery, and a solid smoke testing environment.
==================================================
Brent Paine說
Monday, September 22, 2008 1:23 PM by Brent Paine
I totally agree with you UIGuy. There are certainly limits to what I think should be tested. We haven't, and won't, automate 100% of our application, for the exact reason that you mention.
What I'm trying to illustrate, possibly poorly, is that if we had modules X, Y, and Z, they might all function properly as standalone modules. We might test every possible combination on each of these modules, individually. However, even though we know what they are meant to do, without a GUI we're never really sure how these modules will interact. That is, of course, unless we are allowed source-level access to the modules. Then it becomes an organizational question of what level of access to automation staffers have?
At the same time, it could be a matter of semantics. If we're talking about automating a GUI to verify the GUI layout, button location, etc., that's one thing. If we're talking about doing all of our automation within the layers underneath the GUI, then I think that is also lacking. Then there are X-Factors (I don't know why I always capitalize that) like how much of the app are you planning on automating? Are you being asked to automate test cases or automate a module? Do you have access to teh application source code? Are you treated as a developer or a QA person who is acting like a developer ( :P )?
So is Alan wrong? Meh, probably not. I think that the number might be misleading to a lot of people only because I think that there are fewer companies who would be ok with you trudging through their source code than those who don't mind. Not to mention, I would expect that most companies would not allow juniors or even many seniors to work at this low a level. That's why they buy the tools to work on top of those layers. For many, the GUI layer will often be their only access to any low-level items for automation. Now, this might be my own assumption. Maybe I'm wrong. In all my time, though, I haven't granted someone full source-level access, even if I had it myself.
留言列表
