Fictional scenario: Trek–Lance Armstrong’s bicycle sponsor–is behind schedule and over-budget on creating a new cycle. They need to find a way to get their product out the door, find it now, and find it cheap. Now, imagine that they threw my grandmother on their bike, had her drive it around the block, and declared it fully tested and ready for mass-production. Would you be satisfied? If it found 300 grandmothers and had them drive around the block twice, would that satisfy you? How about if they used 300 average Joes? Would that satisfy Lance Armstrong? Would he have full confidence in his ride for twenty-one days and over 3,500 km in the tour? I doubt it. That bike wouldn’t even make it out of the warehouse, let alone to the starting line. That bike would not earn respect until it was rigorously tested in a scenario that at least simulates its intended use. So why do so many fail to put their web applications through the same trials?
Money? It will cost more money to fix it after launch than it will to test it during development, identify the issues early, and get them fixed before the product goes out the door.
Time? Well, time is money, so see above.
Experience? There are a lot of good, quality testers out there. If my mechanic doesn’t properly fix my car, I’ll take my car to a different mechanic.
I’m curious about the thoughts of everyone out there.
We’ve all been there. You’re cruising down I-5, minding your own business, when the big SUV next to you decides that it is time to change lanes, and that it wants to occupy the space you are currently claiming for yourself. Or there’s the motor home doing 61 in the left lane passing the tractor-trailer in the right lane that’s doing 60. There’s the guy that passes everyone at the construction zone, repeatedly ignoring “Lane Ends. Merge Left,” and expects that someone will let him in when pylons encroach, and he will force the issue if they don’t.
All of this traffic congestion, incompetence, and utter disregard for fellow citizens pollutes your driving experience. You work day is stressful enough. Why must you go through it on the way home?
Your web applications have to go through the same thing: A poorly-coded neighbor with the memory leak, that just keeps taking and taking from the available RAM, until there is nothing left for your app, just like that SUV; An application that didn’t get properly optimized, and hogs all of your available bandwidth, slowing down your application like that 40-foot RV; An evil report that thinks it is superior, and locks the entire database from outside access until it is finished generating that 400-page PDF.
When you are testing the performance of your application, make sure that the environment you are about to stuff it in to is up to par. No matter how pristine your application looks in Staging, it is only going to be as good as the environment that you launch it to. If you ignore the big picture, and your application succumbs to the web environment pollution, your application will be to blame. No matter how mediocre the environment is without your application, your superiors or clients will still say “the environment works just fine without your app.”
Build a testing environment that mimics production, and that includes any other applications or components that you will be sharing resources with. Create some generic scripts that will generate traffic against these neighbors and execute tests against your application. This will help identify any integration issues between you and your environment, and help eliminate any surprises when you launch.
The environment is supposed to work just fine with your app, too.
In the QA forums I frequent, there are often questions about how to properly load test when you don’t have access to production or an identically built environment. Most companies won’t spring the cash to build an environment that is identical to production; generally, testing environments are made up of hand-me-down servers that used to be in production. Of course, there is also the cost of test suite licensing to produce a productional load, and the near impossibility of mimicking real production traffic.
Though a production clone would be ideal, a watered down environment can be sufficient, and in some ways better. Bottlenecks are achieved faster, without having to push through 50 Mbps of data. Additionally, a “lesser” environment will be more sensitive to changes; your transaction may take 0.5 seconds on production-grade servers, and a defect that doubles it to 1.0 seconds is hardly noticeable, but on a lesser environment where that transaction takes 6.0 seconds, doubling it to twelve throws up red flags.
For a watered-down environment, try to lessen the horsepower of your system while matching the architecture. If your productional environment is eight web servers that are all quad 3.2 Ghz Xeons running Windows Server 2003 Web Edition, and all load balanced through a hardware load balancer, you can bring it down to two web servers with less horsepower–perhaps dual 700Mhz P3s–but the servers should still run Windows Server 2003 Web Edition and be balanced with a hardware balancer. Do not drop below two web servers because you will still want a load balanced environment, and do not switch to Windows 2000 or use Microsoft’s NLB (Network Load Balancing). If your production web environment uses Windows 2000 and NLB, obviously use that technology in your testing environment; do not switch to Windows 2003 or a hardware load balancer.
Additionally, try to reduce equally throughout your environment. Don’t drop your web servers from Pentium 4s to Pentium 3s while dropping your database servers from Pentium 4s to an old 486 desktop. Equal reductions maintain your continuity, and in the end, your sanity. Unequal reductions introduce new problems that don’t exist in production, but will still happily waste your time and money. A major bottleneck might exist on your web servers, but the defect could be masked because you were database-bound by using that old 486.
The idea behind this is that many bugs can be introduced by a specific revision of your OS (Think of the problems from Windows XP SP2), from your style of network infrastructure, the version of your graphics driver, etc. You want as many common points as possible between your testing and production environments to eliminate any surprises when you launch your application. Ideally, your testing environment is an exact replica of your production environment, but unless you are making desktop applications, it is only a fantasy, so just try to get as close as you can. Use the same OS version, including the same service pack and the same installed hot fixes. Use the same driver versions, and configure the same settings on your web server software. You are trying to create a miniature version of your production environment, like a model car or a ship in a bottle. Pay attention to the details and you will be okay. To your application, the environments should be exactly the same; one is just a little snug.
For love of all things QA, before you launch a new application, test production!
“What? That’s stupid! Why would I want to perform a load test production and risk an outage? That impacts my SLAs. I can’t impact my SLAs!”
Remember the number one rule of quality control: if you don’t find it, your customers will.
When you are about to launch a brand new application into your production environment, test that application against production. However, this only applies for new applications. New applications will introduce new, additional load on the environment, while existing, revised applications already have added that load to the system. Essentially, with an existing application, you already know how well the production environment can handle the additional demand generated by the application’s audience. New applications have not yet generated that load, and production has yet to prove itself.
There is no hard evidence that production can take the additional demand. Maybe your production load balancer can only handle another 5 MB/s, and your new application will demand another 7. Perhaps it is one of the switches, instead. Or for my recent life, maybe it is your ISP. You will not know until you test it, until you measure it, and “if you didn’t measure it, you didn’t do it.” With a past project, my company created an intranet application for our client, and our application just happened to be hosted off-site. The off-site environment was green, and wasn’t hosting anything else, so our client had no issue with us testing this environment fully since it was going to be production, but wasn’t yet. The hosting company and their ISP rated the environment at 45 Mbps (That’s megabits–lower-case ‘b’), and based on the clients traffic expectations, we needed about 30. It is a good thing we tested the site because we found an issue with the load balancer at about 15 Mbps, a problem with server memory when it was processing enough transactions to produce 20 Mbps, a problem with the database switches when we were generating 22 Mbps, and–this one is the kicker–a bandwidth ceiling at 28. Though all of the routers, switches, balancers, and servers were performing well, we couldn’t get more than 28 Mbps to the web servers. It turns out that the ISP didn’t ever expect anyone to use that 45 Mbps rating, and never tested to make sure they could handle it.
“If you didn’t measure it, you didn’t do it.”
Through two months of midnight through 0600 testing, we upgraded the load balancer, added more memory, put in gigabit switches, had the ISP tweak their infrastructure, pushed through all of the data we needed, and successfully proved that the off-site environment and our new application could handle the load. But, the environment still wasn’t fully tested. Our client used everyone’s favorite single-signon, SiteMinder. However, they wouldn’t let us test the application while integrating their productional SiteMinder policy servers. We could only use staging, and when the staging servers couldn’t handle the load, “that’s okay because it’s staging.” But no matter how much we advocated, we couldn’t test production. We might impact the environment and the SLAs. So, we launched without testing it, and guess what happened? The policy servers failed, and they severely impacted their SLAs.
And to think, we could have tested that at 1:00 AM on a Saturday, and they even if we fried the policy servers, they would have had all weekend to fix it. And most importantly, we would have identified it before the end-user did. But what really cooked their goose was the difference between productional load and performance testing load: performance tests can be stopped. It is a lot harder to fix a jet engine at 30,000 ft.
The moral of the story: when launching a new application, always test production. Always.
It’s not all about Internet Explorer any more. Yet, I am surprised at the number of web houses still coding specifically to IE. Much to my dismay, even my own company does it. Though we have a little bit of an excuse—our client only supports IE in their organization, and the app is internal—it still bothers me that we are abandoning everyone else.
New figures released a week ago place IE’s market share at 89%. That means more than 1 in 10 users are not using IE. (Read the Article) By coding specific to Microsoft, you are abandoning 11% of your potential users. That is astonishing and disturbing.
Pay particular attention to Firefox. Its user-base is growing exponentially, and doubling every 9 months. I’m a fan of the application. It is much easier to use than IE, and much more solid. I’ve converted all of my friends and almost all of my family. I even have my in-laws using Firefox. (Get Firefox)
As the IE behemoth continues to fall, you and your organization should be paying more and more attention to standards and multiple-browser testing. Check that your HTML is compliant, and test your sites in at least IE and Firefox, if not others. Don’t force your users to use a particular browser; chances are that if they can, they will just go somewhere else for their information.
When testing .Net web application forms that use postback, it is always a good idea to leave the form and come back. Postback is when a page refreshes or submits to itself; generally, identified by the pre- and post-submit URL being the same page. Often times, the status of the form fields is saved in the .Net ViewState after a submit, rather than retrieved from the database. You might have checked the “Display me” checkbox and clicked submit. The “cached” version from the ViewState says that this control should be checked, so when the page reloads, it is. However, the value may have not been saved to the database, so when the value is loaded from the DB, the box is not checked, but you would not have known since the ViewState version was used. When testing, to make sure you are getting the actual values and not the “cached” counterparts, make sure you leave the page and come back.
My favorite part of my job is most definitely that I get paid to break things.
As a kid I had a playroom that was filled with legos. Legoland lived on two 8′x4′ sheets of plywood, and covered them both with roads, hospitals, race tracks, and restaurants. But it would be impossible to recall all of the horrible, terrible tragedies that happened to Legoland. Every weekend there was a new disaster: a high-speed police chase that would end with the perpetrator crashing into the gas station, and the ensuing explosion would level every building within 4 base plates; a tornado that blew the truck stop clear over to the other side of the Cantina; an earthquake–perfect when Legoland exists on two sheets of plywood–would split the town in half. Every weekend Legoland would get completely destroyed, leaving just an assorted pile of legos strewn across sixty-four square feet of what once was a happy little town. I would spend the next week reconstructing each building in true make-believe fashion, construction vehicles and all, just to repeat it all again come Saturday. The perpetrator was always the same guy, too, in the red helmet and the little blue dune buggy. You’d think that the Legolanders would revoke his driving privileges after the twenty-sixth time.
“Find something you love doing, and find someone that will pay you to do it.”
I break things. And, they pay me for it.
Screen Hunter 4.0 Free - www.wisdom-soft.com
Screen Capture Tool
Cost: Free
Quite possible the most essential task for any tester is taking a snapshot of the current screen to give their developer a visual representation of the logged error. The classic Windows hotkey, [Alt] + [PrtScn], will take a screen capture of the entire active window. However, sometimes the text on a link is spelled wrong, a button uses the wrong icon, or an error message displays in the wrong style; in these scenarios an entire screen grab is overkill and often confusing. Yet there are few things that a tester can do about that short of opening up MS Paint or Macromedia Fireworks and cropping the image, completely wasting valuable time and causing pointed comments from the Project Manager about diddling in Photoshop.
Screen Hunter 4.0 Free allows you to capture the important pixels quickly and effortlessly. Tap F6 (The default hotkey, but it can be modified), and your cursor changes to a cross-hair. Click-drag a box around whatever you want to capture, and it’s done. Instantly cropped screen capture for your bug-tracking pleasure.
The developers will be happier, too.
|