Quality Assurance in Software (It CAN be done!)

By Larry Laux
President and Founder

I spent an hour today with the Metrix person in charge of Quality Assurance in our products. It was a most pleasant, and interesting, conversation. I’ll share some of it with you. One of the basic points of understanding is: all software has bugs. Ours, Microsoft’s, SAP’s, the video game you picked up for your kids – all software. Some bugs are known, some are not. And some that are known get reclassified as ‘features’(that’s a topic for a separate essay). So given that, what can and should be done?

Our guru walked me through the Metrix QA framework. At a very high level, he explained that automated testing, where pre-formatted inputs of data and simulated keystrokes are fed in to the system via a testing program, can reasonably exercise 75 to 80% of the code. The output of these exercises are compared (also automated) to a pre-built expected outcome reference file. When they match, life is good; the programs are functioning as designed. When they don’t match, something is wrong. An error exception report is generated and routed to, among others, the QA manager for further investigation.

So far, so good.

Manual test scripts are another piece of the plan. They are step-by-step instructions that are taken by a person at a device or workstation, using the system. They can visually observe the reaction of the system as the keys are hit, and record results. This type of testing is great for catching items that aren’t bugs per se, but may make the system annoying or difficult to use. An example – when a user hits backspace on a field on a web page, and the page fully resets and clears all of the values already entered. This is annoying (has it happened to you?) but the type of thing that is not easily built in to automated test scripts.

These manual test scripts should cover 20 to 25% of the system, making sure that each function type (add, delete, inquire, report etc.) is represented, as well as key functions.

A third component is called scenario testing. This may be driven by a particular client need. For example, we know that Metrix client ‘X’ uses the Repair Center module in a certain un-typical way. We can have them provide us with a scenario, or set of testing scripts, which can be performed prior to any release or upgrade patch application to their environment. This technique would test 10 - 15% of the overall system, typically. (Percentages add up to more than 100% because of overlap; some items get tested in multiple ways.)

Scenario testing is also useful for exercising integrations.

Finally, perhaps the most ‘fun’ testing is ad-hoc. This is usually done by two types of individual: 1) a very, very knowledgeable person who knows the system well and thus knows where the fragile parts are; and 2) a very novice user that might put funny characters in a date field, because they aren’t sure what the software is looking for. Both of these testers look at their tasks as a game as they try to ‘out-smart’ the programs (and programmers). This process covers 10 – 15% of the system, and is looked at as a statistical proof step in the quality assurance process.

Our conversation covered many other points but I thought these items were interesting enough in general to share. Does your group engage in a QA process that is similar to Metrix’? If you have feedback to share, please leave a comment, thanks.

The Discipline of Project Justification - Part 2

Second in a multi-Part Series
By Tom DeVroy
VP of Sales

In the last post I discussed a methodology for justifying a capital project to management. My proposition here is that there is a process involved and some discipline required for a successful conclusion. In this post I'd like to address the first requirement of project justification.

The Pain Statement
The first step is defining the problem you are trying to address. If you can't articulate the problem you definitely can't make a case for the money to fix what (apparently) isn't broke. There isn't any reason to buy a system, or acquire new hardware, if you can't clearly indicate the pain caused by the current status quo. The definition of your problem is what I like to call the "pain statement". The bigger the pain statement is, the easier it will be to ask for the financial means to fix it.

For example, if your business consistently missed customer expectations in customer service, this would drive down customer satisfaction and cause you to miss out on new business opportunities for product sales. This pain statement, although intuitive at this point can probably be backed by financial justification later down the road. Pain statements don't always have to be so complete though.

Another example could be something along these lines: Our current system is old, it runs on hardware no longer supported and much of the technology is aging. It is forcing us to not change anything. First, this makes us very vulnerable to catastrophic failure, and secondly, this forces us to remain in a status quo for operating procedures and business processes. Our infrastructure needs updating or we run the risk of not being able to support our own environment. Once again, the pain is intuitive and it opens the door for further justification down the road.

The nature of the problem probably will dictate who will fund the solution to the problem. My first example is an operational issue. The solution to the problem probably comes out of a departmental budget within Service Operations. The second problem is likely an IT problem that should be funded from the IT department, although both problems affect the business directly.

The Hypothesis
After creating a pain statement a hypothesis should be developed for solving the problem. The hypothesis defines a possible outcome and perhaps defines a potential solution. Although the person that develops the hypothesis should be careful not to be too forthcoming with a solution. This might imply that the author has an agenda and has pre-determined an outcome before adequate research has been performed.

If we go back to the first example, a proposed hypothesis might be that we look for an integration strategy or solution that combines our contract SLA's, our customer master, and our problem tracking system. We can then have visibility between the customer contract commitment, their currently reported problem, and what levels of service we owe them to resolve their problem. Notice that I don't suggest a "new" system, although such a suggestion might be the final solution.

Further into the evaluation, you likely will be asked for one or more alternatives that address a solution to the problem, to financially justify it, and to create a coherent business case. We'll save these topics for future posts. Next post, establishing the business case.

View Part 1 Here
View Part 3 Here
View Part 4 Here
View Part 5 Here

Mobile + Technician + Linked to Information

By Larry Laux
President and Founder

In 1991 a national service manager with Du-Pont medical products approached us with a classic service problem. His service engineers were responsible for both break/fix service and for preventive maintenance service on their products. He was wrestling with this issue - if an engineer is sent to St. Michael’s Hospital on a repair call, wouldn’t it be great if he could also easily check to see if any preventative maintenance was due? If one was due, he could perform that work right then, while he was on site already for the repair call. Saving even a few additional trips per week would have a big cost benefit.

It was this scenario that lead to Mobile Techlink being born. It seems almost laughable in retrospect, but the state of the art in portable computing at the time was the HP 110. That diskless, clam-shell PC had 640 Kilobytes (Kilobytes, not Megabytes) of RAM. In that space we had to store the data, the operating system and the Mobile Techlink application. Can you say ‘efficient programming’?

While developing Mobile Techlink, data synchronization was one of the most difficult problems we had to deal with. It took years to work out such questions as "Who has the right to update what data?" and "What if the same data element is updated by two different people on two different computers at the same time?"

Since there was no wireless or broadband to speak of back then, connection was via phone lines. Some of the engineers used acoustic couplers (you are officially old if you remember them). Even though the data was, generally, only updated once a day, the benefit of current data available when the engineer needed it was huge.

Some of the technology has improved - handhelds are much more robust and laptops (with much more RAM than 640K) are standard issue. In addition, wireless and broadband have become cost-effective and widely available, aiding the access to data while in the field. Some technologies have arrived that weren’t possible then - signature capture is one example.

But all of this technology has not made the Field Service Engineer as connected as I would have expected. I think part of the reason is the devices themselves - is a cell phone a reasonable form factor? Is a full keypad needed? (I for one am glad to be past the Graffiti® stage.) Will there ever be a truly usable tablet device?

Mea Culpa, software must also take some of the blame. It still has room for improvements on features, speed, configurability and usability (and we’ve re-written Mobile Techlink four times in the last 15 years to tackle many of these). I think we are much better than we were, but still not quite where we need to be (but - shameless plug - we are pretty proud of our most recent release of Mobile Techlink).

And ten years from now, will the equipment be contacting the engineers devices directly? I’m not sure I’m mentally prepared for my washing machine to speak up and tell me it has logged a service call for itself and the tech will be here Tuesday at 8:15. Although, now that I type it, if it could send me an e-mail instead of talking...