So your wonderful little creation is finished, and it does exactly what it was designed to do. But, have you prevented it from doing what it’s not supposed to do?
Enter the forgotten art of negative testing. This is the safeguard from user error, malicious attacks, and blatant developer oversight. Negative testing is taking your calculator application and trying to add “Hello” and “Goodnight”. Negative testing is trying to supply an invalid email address–.anything@something.q–into your mailing list form. Negative testing is trying to cause a buffer overflow on your lead-developer’s computer because you were able to sneak in a script injection.
The key word here is “try.”
If everyone has done their job, you will get nowhere. Unfortunately, rarely is this job done right. In 3 minutes I could considerably alter my best friend’s blog, and he doesn’t even know it. In 10 minutes I could corrupt the online database of a Fortune 500’s web site–both company and URL to remain anonymous. And, what scares me the most, in 20 minutes I could download the entire database of a certain benefits company, including the complete identity–SSN included–of a few thousand people.
For years, I have been paid to break things as much as build them. When that calculator finally adds 2 and 2 correctly, don’t be satisfied. Try to add “Hello” and “Goodnight”. Will it give you a neatly handled error message informing you that it couldn’t complete the procedure, or did it return a fatal exception and die a miserable death because it expected a Double and you gave it a String? Optimally, it shouldn’t allow you to even type characters into the input area unless you are working in hex; even then, only A-F.
If instructions tell you to do one thing, enter the opposite. If you see a value in the URL, change it. If a field asks for an integer between 0 and 5, try 0, 2, 5, -1, 9, 3.5, and “Q”, and see how it handles “unexpected inputs.” If a querystring is “?UserID=6″, change the 6 to a 7, to see if you get information on User 7, and try invalid items like 3.5 and “Q” to see if it fails on unexpected inputs. If a client-side cookie has a value of “User”, try changing it to “Admin” or “Administrator” and see if your access-level is increased.
Find the weaknesses, find the holes, and find the bugs so that they can get fixed. You are the demolition man. You get paid to blow things up. Do it. Do it with purpose. Pretend you are a hacker trying to get into the system. Pretend you are a teenager-hacker-wannabe trying to screw with the system. Pretend you are a grandma that doesn’t know what to do with the system. Do all of the things that you aren’t supposed to do to the application and do them on purpose, because if by ignorance or intelligence, your users will find what was missed.
For my needs, the biggest hole in Mercury LoadRunner is its lack of page size monitoring. LoadRunner can monitor anything else imaginable, including transaction counts, transaction times, errors, and all Windows Performance Monitor metrics. However, monitoring page size, download times, and HTTP Return codes are only available through programming.
The following function will monitor the page size of all responses, logging an error if it exceeds you specified limit, as well as track all values on the user-defined graphs.
si_page_size_limit(int PageLimit, char* PageName, char *PageURL, long TransactionID){
//
// Page Size Limit Monitor
// Author: Jay Harris, http://www.cptloadtest.com, (c) 2004 Jason Harris
// License: This work is licensed under a
// Creative Commons Attribution 3.0 United States License.
// http://creativecommons.org/licenses/by/3.0/us/
//
// Created: 10-Aug-2004
// Last Modified: 10-May-2005, Jay Harris
//
// Description:
// Logs an error to the log, pass or fail, including the applicable status, if logging is enabled.
// Plots page size datapoint to User Defined graph.
//
// Inputs:
// int PageLimit Maximum page size allowed, in bytes
// char* PageName Name of the page, such as the Title. For identification in logs.
// char* PageURL URL of the page. For reference in logs. FOr identification in logs.
// long TransactionID Transaction ID for the current request.
// Note: Transaction must be explicitly opened via lr_start_transaction_instance.
// Note: TransactionID is returned by lr_start_transaction_instance.
//
int iPageSize = web_get_int_property(HTTP_INFO_DOWNLOAD_SIZE);
char DataPointName[1024] = “Response Size [”;
strcat(DataPointName, PageName);
strcat(DataPointName, “]”);
if (PageLimit < iPageSize) {
lr_continue_on_error(1);
lr_debug_message(LR_MSG_CLASS_BRIEF_LOG | LR_MSG_CLASS_EXTENDED_LOG,
“Page Size Check FAILED - %s [%s] exceeds specified page size limit of %d (Total: %d)”,
PageName,PageURL,PageLimit,iPageSize);
lr_continue_on_error(0);
} else {
lr_debug_message(LR_MSG_CLASS_BRIEF_LOG | LR_MSG_CLASS_EXTENDED_LOG,
“Page Size Check PASSED - %s [%s] meets specified page size limit of %d (Total: %d)”,
PageName,PageURL,PageLimit,iPageSize);
}
if (lr_get_trans_instance_status(TransactionID) == LR_PASS) {
lr_user_data_point_instance_ex(DataPointName,iPageSize,TransactionID,DP_FLAGS_EXTENDED_LOG);
}
return 0;
}
All too often, we forget about usability. We get so caught up in fixing functionality and enhancing performance that we forget about the most important part: how easy is it to use this thing we have just created. Sure. That new super-gigantic Humvee will go in any direction you want, can climb a 21-inch vertical, and can pull small houses with ease, but who cares? It drools at the sight of a gas station, it is impossible to parallel park, and most importantly, how do you expect grandma to climb in and out of that thing?
What would grandma do?
Imagine that poor old lady trying to raise her little leg up onto the running board, and then pull herself up into the cab with those little arms. She’s a GRANDMA! She isn’t 20 anymore. Or 50, for that matter. We forget about grandma in our software testing, too. How would she use that application you just made? How would she react to that detailed error message your creation just spit out?
My poor father; he just got his first computer, and I’ve been trying to teach him how to do instant messaging. He knows enough about Windows XP to be familiar with the big ‘X’ in upper-right corner. Click it and everything goes away. But, Trillian is different. In the upper-left of the contact list is an upside-down triangle that minimizes the window to the system tray. Right below that is a little small ‘x’. Unfortunately, that ‘x’ removes your contacts from within your Trillian window. You have to play around in your ‘View’ menu to get the list to come back, again. However, my father doesn’t know upside-down triangles, and he certainly doesn’t know about the ‘View’ menu, yet. He just knows the ‘x’. So that’s what he does. He clicks the little ‘x’. And every time he does, his contacts go *poof*, and he has to call me to help him get his contacts back. My grandmother would do the same thing. I don’t think Trillian did any usability testing on that feature.
What would grandma do? You know that she’s going to want to click the ‘x’, no matter what, because the ‘x’ is what she knows, just like my father. So why not make the upside-down triangle an ‘x’? It can still minimize to the system tray. The ‘x’ isn’t a cast-in-stone rule that the application must quit all-together. If you don’t believe me, try the ‘x’ on your MSN Messenger window. It minimizes to the system tray. Why did Microsoft brake their own tradition? I bet what they really did was a little usability testing, and discovered that new users always want to click the ‘x’. To new users, the ‘x’ is a big “CLICK HERE” sign to make that window go away. They don’t care if it closes; new users just want it to go away. And if she were still around, Violet–my grandmother–would always be a new user when it came to computers. Just make the window go away. Like clearing the dishes after dinner: it didn’t matter if you threw the plate out, just get it off the table.
So, we have this problem. Now, what do we do about it? Ask yourself:What would grandma do? “Fatal error: Userdata insert failed. Connection to database unavailable. \\jedimaster\yoda\greenlightsaber\sqlserver2000 not found.” If she saw that, what would grandma do? Stare blankly at the computer? “Unable to save your contact information. Please try again later.” Grandma can understand that. So, think of your grandma when you test that new application. Think of your grandma when you write your error messages. Think of grandma when you draw pretty graphics or design a button icon. Your program will be much more friendly, and much easier to use. Even grandma could use it. If you need help remembering, put a picture of grandma on your desk at work, right next to your monitor. And if you don’t have a grandma, substitute that sweet old lady down the street that bought all of your raffle tickets when you were 12 and baked you cookies because you were such a good little kid.
Remember grandma.
What would grandma do? She’d tell you that she’s proud of you, because that’s what grandmas do.
Imagine a world where no one had a name. Instead of John Doe, you would know “the 185lb, long black haired, 5-foot-11 guy that lives on the corner of 43rd and 5th.” What happened if that guy got a crew cut? To his close friends, he would now be “the 185lb, short black haired, 5-foot-11 guy that lives on the corner of 43rd and 5th.” But not-so-close acquaintances, ex-girlfriends, the florist, cabbie, and the IRS would all know him as his former name. Few would know this new guy. There would be confusion when one group tries to talk to the other about this guy. What happened if he gained a few pounds, too, and moved over to 54th? No one would know who he was, anymore.
That’s why we have names. They are a constant in a dynamic life. They are the dependable value that gives the world security when all else changes. If John dyed his hair blue, moved to Phoenix, and got a sunburn, people would still know who he is. He’s John Doe.
ID attributes do for web objects what names do for humanity. In a world where static web sites are the ones your 12-year-old makes on Geocities, automated QA tools need a little help. Do you have a link to your favorite news site? Today, your automated tool can find that link to <a href=”http://www.cnn.com”>CNN.com</a>, but tomorrow’s <a href=”http://www.msnbc.com”>MSNBC</a> link is lost; the tool is still looking for CNN. So, give the link an ID. That’s its John Doe. The tool can find your link whether it is <a id=”favNewsLink” href=”http://www.cnn.com”>, <a id=”favNewsLink” href=”http://www.msnbc.com”>, or any other link that floats your boat. All it has to do is look for the name. favNewsLink. John Doe.
Be nice to your automated tools: give your web objects a name. Rename your ‘Comments’ link to ‘Feedback’? John Doe. Change your image of your dog to an image of your cat? John Doe. Multilingual site with translated text? John Doe. It will help you automate your QA process, and you will spend less time retooling your automation scripts and more time downing that Corona. Just don’t forget the lime.
Happy Cinco de Mayo.
Yes,
yes. I know. I should have done this long ago. People keep harassing me
to put my thoughts out on some odd web site, essentially exposing
myself to the world. I’m not an exhibitionist. I don’t like doing this.
But, I will try it. Maybe I will like green eggs and ham.
|