It was a quiet Friday Morning in the United States.  Or, I mean, Friday afternoon in India and Singapore.  Later in New Zealand, and in Europe ...

The point is, there were a lot of people playing. Seventeen teams registered four different continents for three hours of aggressive testing, followed by a weekend where teams rotated through performance testing.

Now, the winners in each category.

It’s here!  It’s here!  If you are reading this, then the Test Competition is finally here!

No, you don’t need to have registered in advance to play, just to win the big prizes and have your name in lights.

Either way, here’re the rules of play.

The class where I learned method-R performance tuning also taught me something else, something I call the consultant's gambit. This is that story.

On March 11th, 2013, registration for the test competition closed.

We have sixteen teams registered, from India to Europe, Southern Asia, and the United States. We have a team that has people from two continents.

And that's just the folks who gave me a street address.

One of our contributors, Pete Walen, recently put out a blog post on leadership and power.  It is a good post; if you enjoy reading long-form articles that are more conversation than bullet-points, I suspect you'll get a kick out of it.

Pete's article did a lot for me, but most importantly, it got me to think, to finally publish that piece on power that had been niggling in the back of my mind for  nearly a year.

Instead of a rambling conversation, I'd like to start out with an assertion, then defend it, then suggest a thing or two to do about it.

My claim is very simple: Most of what you've been taught about Power is probably wrong.

If you've been following me talk about Method-R, you might wonder how I learned it.  In my case, I learned it the "right way", the "official way":  My employer brought in a consultant to teach a three-day class in performance tuning.  Method-R was the method he taught.

It also got me very, very interested in code profiling, which is a way to measure performance at a much more detailed level -- not just is it fast enough, but, instead, how long does each piece take?

Let's consider, for a moment, the poor IT manager, told by end users that the application is "too slow."

I've been that guy. You may be that person some day ... you may be that person now. Either way, it is safe to sat that those in that position would prefer to be out of it.

The manager wants to fix the problem, so he goes downstairs to the DBAs and tells them. The DBA's do some research and find out that the memory percentage is way high, so the database is going to disk (a lot), and disk is slow.They suggest buying a great deal more memory , which is the cheap and easy solution. 

In most cases, the solution is a lot more complex, and involves tuning, news boxes, creating new indexes, maybe upgrading the network switch. After two days, two weeks, two months or two quarters, the DBAs say they are done, and show some dashboards like the one above, with all the data trending in the right direction. Average CPU use is down, so is memory, disk I/O requests are down, response times are down - everything looks good.

Allow me to ask, just for the sake of argument. What happened to average customer response time?

We don't know.

It is this problem, our love affair with data that is often irrelevant, that Method-R solves. 

It hasn’t been easy to organize our first test competition. We’ve had to look for collaboration tools, make sure they scale, plan the scoring, find a way to record the bugs, figure out what to test ...

We have also had five registered teams, and, in that time, a lot of similar questions. Today, I will take a stab at the answers.

1.   What is it about (in a nutshell)?

Every week on "The Apprentice" may be different, but we know one thing:  Donald Trump is going to walk in, ask questions, fire someone, and drive on.

You might want to work like "The Donald" on the show, but if you measure software performance, you probably feel more like a contestant on the show. After all, you start with a problem too broad and vague to adequately solve in the timefame, are involved too late to impact quality, and assigned too little staff, software, and hardware resources to do the job right.

If you've ever felt like that, well, I have too, and I survived. Today I'm going to offer a way out.

It has been two weeks since we announced the details and how to register for the NRGGlobal test competition.  Since that time, we've had three teams register.  

No, I'm not worried, here's why: At the same time I am organizing this, I am also co-chair of the test and quality track of the Agile Conference. That conference had almost no submissions, I was near despair, until the submission deadline of February first approached. Now I have dozens of submissions to evaluate, which puts me in a different kind of despair. :-)

No, seriously, it is alright, we have a review team that is slicing up the reviews right now.  I will be fine.

It is called the basketball exercise, or sometimes inattention blindness, and it demonstrates a real problem in many kinds of automated testing.

What is this thing, you ask?  It can be hard to explain, but I can show you in about a minute.  Look, there's video:

Today I’ll cover the story of project number two, "just a little code" to integrate two systems.  But first, let’s talk about my motivations. 

About the Killer App Series

"The Killer App" is not a work of fiction, or a work of historical fiction. It is not a made-for-TV movie, "inspired by" true events but with the drama cranked up a notch or two; it is history as I recall it.  

While it has been a pleasure to get this off my chest, I have no axe to grind.  The players have moved on.  Enough time and space now separates us that even the “bad guys”, as much as there were bad guys, would probably not recognize themselves in the writing.

Nor is this a case study, all wrapped up in a bow with a key learning takeaway.

The Killer app is ... a story.  

My story, about a time in my life before I was a writer or consultant, back when Matt was an employee, and, sometimes, felt more than a little trapped by his situation.  The story points out that there is more to life than process -- that often, it is the sticky blobs of flesh called humans that can make a difference.  I didn’t always make the right choices. Back then I was more than a bit scared of conflict.  But if a person or two can learn something from the story, and do it with a smile on their face ... I hope you’d agree with me to call that success.

Now it’s time for round two.

The Next Killer App

At the end of the killer app, our hero had transferred off the project, to go work on project number two.  The transfer was no brilliant duck-and-groove out of the matrix. 

The reality was that I failed a code review, so they transferred me to fix the code.

At least, I tried to fail the code review. 

You see, the code came to me late in December, between Christmas and New Year's, with a bow on the end that said it "had to go into production by the end of the year."

The code itself pushed data from one system into another and provided

Last week I posted The Killer App, the story of a real software project with a cliff-hanger, pick-a-path ending.  It's time to close the loop. "The Killer App" was a project that was not just late, but hopeless; a claims processing system that could not process a single claim end-to-end.  When I suggested that we should get both technical teams in a room, the PM insisted that we communicate on paper only.  After all, without a paper trail, the vendor would do whatever they would do, and claim they "heard it in the room." The project was a train wreck.  The software didn't work.  It wouldn't run.  If, by some miracle, we got the system to process a single claim, there was no way the vendor really understood all the combinations. For the first time in the history of the company, it was about to experience a multi-million dollar failure. It's a...

Over the past two weeks I have spent a great deal of time doing research, gathering resources, and judging interest in an on-line test competition in software testing.  My conclusion is kind of like Nike.

Let's do it!

 It's time to talk about details.

 What: The first NRGGlobal Online Test Competition will be April 19, 2013, starting at 10:00AM Eastern with a blog post.  The blog post will describe what software to test, how to get it, how to submit bugs, and how to submit a final test report.

We called it the Killer App, and, for the insurance industry, it was magic.

At least, it was supposed to be.

Forget waiting two weeks to get an explanation of benefits from the insurance company, two more to get a bill, then two hours on the phone to figure it out.  We were going to take care of the whole thing while you were at the doctors office.  To do that, we were going to adjudicate (that’s a fancy word for ‘process’) the claim in real time, so the customer could pay direct with an HSA Debit card -- and know he was paying the right amount.   That meant taking the claim from the doctor, adjusting to the negotiated fee schedule, figuring who owed what, and reporting that back so the customer could pay.

After that, if the customer used cash and had a flex spending account, we would know they qualified to spend out of their FSA, and cut them a check back, from the account, automatically.

Whew.

Like I said, it was the killer app.  I was working for the insurance company, and we hired a vendor to build us a system, to talk to the bank to calculate the balance on the claim.  My job was to write the software to pull from the claims database, compare to the amount in the Flex Spending Account, generate the flat file, and transmit it to the print vendor to make checks.

At least, that’s what I was supposed to do.

You see, the project was nine months late when I started.  The data tables I was supposed to pull from were not set up, nor did I have any written requirements.  All I had was conversations with a supervisor.

It was time for me to talk to people.

Imagine for a moment two companies have an identical problem - the website is slow.  Perhaps this is a huge problem; the company bet it's future on a new product, that customers sign up for on the web, and performance is bad enough that people are abandoning the site and going to the competition.  In a few more months, the competition will 'own' the market, and the percentage of customers that are even worth fighting over will be a small percentage of the total - more like 9% than the 90% of market share hoped for in the initial offering.

Like I said, there are two companies.  Both appoint a point man on performance improvement and give him the speech that the future of the company is in his hands.  Both companies find a war-room, an easel to write on, some markers and stickynotes.  Management asks for daily reports, and steps away. Knowing the PMI playbook backwards and forwards, plus a half-dozen other techniques, both newly appointed 'performance project managers' are competent, smart people, and they have a plan:  Figure out what component(s) are the bottleneck, and speed them up.  It's not rocket science, right?

Then differences show up.

Last week I put out the Black Swan In The Enterprise, which argued that most uptime planning comes from the ability to predict the future -- and sometimes, the future is wildly different that you would have predicted.  Things fall apart; the centre does not hold -- and they fall apart very differently than what the team planned for up front.  If that's the case, what's the company to do, and how can Operating Systems promise those 99.995% uptimes and take themselves seriously?

System Effects

The incredibly positive uptimes we see in sales and marketing material all come from the same place -- looking at one component of the project in isolation.  For example, let's look at that Amazon Outage again.  The severs lost power.  This has nothing to do with the operating system, it's not the fault of the operating system!  You can almost hear the executives arguing "look, man, I said 99.995% uptime, but you can't get that if you yank the power cord out."

Or yank the internet connection.

Or a bridge or router fails.

Or a sysadmin makes a mistake an mis-configures a router table.

Or a sysadmin bingos a firewall rule, and suddenly your webserver is prevented to talking to the internet.

Or the loadbalancer gets bingoes, pointing to an old server that is no longer there.

Or ...

The problem isn't an individual component -- it is all the components, acting as a system.   Just like the old cliche, the delivery chain of software is only as strong as its weakest link.  My suggestion on what to do with this is more than a little bit counter-intuitive.

When the staff at NRGGlobal asked me to start blogging for them, they asked what we had in common, and what they could do for the test community.

Over the past few years, I've gotten to know the test community pretty well, and it seems to me there are really two test communities.

First there's the folks I see a few times at a year at conferences, who work in big cities and have a reasonably large amount of personal influence and career mobility.

Then there is everybody else, the folks who say "that must be nice", with maybe a touch of honest disappointment in the words.

Well, everybody else, this one is for you.

Recently, I was invited to observe a colleague’s monthly stress test exercise. There is a large room filled with mangers and technicians, experts in various aspects of the system, general network specialists, application specialists, customer support representatives and a collection of people with backgrounds in various aspects of testing. Once a month, they get a group of people together and observe how the systems behave under massive load levels. They make observations and notes and examine what possible causes for various behaviors can be. Then they compare the results with what happens in their test environment.   Questions and Answers This caught me by surprise so I asked the question: “You don’t run this exercise in your test environment? You do this live?”The person I was sitting next to seemed baffled at the idea. “Of course we can’t run this exercise in the test environment. It is far too small and...

Amazon’s Service level agreement for it’s cloud computer service is 99.95% uptime for a service region (that’s a data center) during a one year period. That means that the site will be down five one-hundredth’s of a percent of the time. That means that EC2 should be down for less than four and a half hours per year.  Being  tester, I set out to evaluate this claim, starting with a google search for “Amazon Ec2 Down 2012.”   I found a half-dozen issues in the twenty-minute to half hour range plus a Seven Hour Outage in April.   Instead of tearing into Amazon, I’d like to be fair.  They did a lot right.  They have a great deal of technical depth, they have multiple redundant switches, internet connections, backup power, and servers.  They have technical depth, 24/7 monitoring, and keep most outages to twenty minutes by acting swiftly.  When they have...

A few years ago, one of the companies I worked with created a "Production Application Support Team", or PAST.  These were the sysadmins and programmers intended as the next line of support after the helpdesk, who could triage some problems and perhaps actually fix them. The team had a problem - vague complaints that the website was "slow" for login.  Yet anytime a leader got the phone call, they would give the site a try, and see reasonable response times.  PAST had a motivated programmer on it called Doug and a free copy of a functional test tool. They gave the tool to Doug and asked him to figure it out. Doug was a sharp guy, and he had an idea.  He used the tool to record the login operation and write the timing of the full page download to a database, then set it to login every five minutes, all...

A few years ago, I worked a performance project.  You probably know the type:  Customers were upset, executives were upset, technical staff were asking for specific direction (and not getting much) ... nobody was happy. The code was already in production, and there was no obvious ‘roll it back to the previous version that is just fine’ trick for us to pull.  It was pretty bad. There are worse projects.  When the technical staff starts pointing fingers at each other, then you know nothing is going to get done.  At least we had hope. Then came the fateful day.  The CEO was on the conference call, and he said, very sincerely, “We’ve got problems with the website ... we really need some more testing.” At this, the director of product quality perked up. “I have to politely disagree.  Testing will tell us if the product is slow or not.  We already...

If you've done any performance testing, or even seen a demo, you are probably familiar with the “big line graph” that is created as the test runs. A Typical Response Time Graph The typical scenario is simple:  Have the load tool start with one simultaneous user running a script.  Then add a new user, say, every thirty seconds plus a small random pause.  The script consists of a set of pre-defined operations - login, search, click on some item, and logout. As more users are added we expect to see response times get higher.  We might see response times go through the roof at some magic number as the system hits a bottleneck, or, perhaps it goes up gracefully.  With some cloud-based systems, ideally, performance stays within a range over time. It’s a simple way to do testing.  It is straightforward, even easy to implement. And it has a pile of problems. Let’s...

By Matthew Heusser Hello; I am Matt Heusser, a risk management consultant and sometimes writer.  You can learn more about me at xndev.com. The folks at NRG Global have asked me to start blogging for them because their place in the community aligns with mine.  We are both interested in minimizing software quality risk by inquiry, assessment, and evaluation. Given the time of year, I thought it might be appropriate to start ... with a story. A Performance Story It was more than a few years ago. I was a bright, young software engineer, working for a large insurance company.  Y2K was finally over, the internet crash had begun, companies were sinking millions into ERP upgrades ... all the while under pressure to “do more with less.” How we ever thought we would do more while executing on a multi-year, hundreds-of-person ERP conversion was beyond me.  One thing that did come...

Many of our clients have enterprise applications that are accessed 24 hours a day, seven days a week.  Their applications may be accessed remotely from all over the world.  In such cases, total users could number in the thousands. When configuring a test plan for our clients we want to establish a realistic number of concurrent users, in other words, the number of users actually in the system, actively using the application at a given time. Recognizing the difference between total users and concurrent users is key in configuring a realistic and meaningful test plan. Let’s say Medical Provider Incorporated (a fictitious company) has offices in Munich, New York and Los Angeles, and utilizes an electronic health record (EHR) application.  Each office has 2,000 employees.  First, determine the number of employees that actually access the EHR application under test.  For our example, we’ll say 1,000 in each office.  Of the 1,000, let’s say...

If you are starting or thinking of starting your Eclipsys 5.5 upgrade, you are probably wondering what approach to take when testing the upgrade and what issues others have uncovered during testing. Eclipsys 5.5 features a suite of modules wrapped by a new gateway user interface to allow for faster work and better flow and integration of data access. This intuitive new gateway interface allows users to access one or two or many modules at one time. This major change in Eclipsys introduces many new questions about the performance of the application. Some are: Will it require same amount of memory/CPU resources as previous version? Will the current security settings be intact or will it be modified after the upgrade? Will accessing multiple modules from the Gateway cause any integration problems? Will the database and MSMQ storage requirements increase? How about using Eclipsys through a portal like Citrix? How about session endurance or printing reports? And many more… Testing and finding...