Testing Commercial Off- the- shelf (COTS) Softwareby. W. Rice. Commercial. Off- the- Shelf (COTS) software is becoming an ever- increasing part of. IT strategy for building and delivering systems. A. common perception held by many people is that since a vendor developed.

However, people are learning that as they buy and. COTS- based systems, the test activities are not necessarily. In this. article, we will explore the challenges and solution strategies for. COTS- based applications.

We will also see a process for testing. COTS- based applications. Case. Insurance Company plans to deploy a new system to allow its 1,2.

The Graduate School of Operational and Information Sciences includes Graduate Resident Programs consisting of 15 technical curricula and awards Master of.

Instead of writing its. The solution appears to be. In addition, the insurance company does not have. There are. however, some considerations that the company realized after they made.

New. versions of the application will be released each year. An. annual maintenance fee will be required for vendor support. The. interfaces between the contact management software and the office suite. There. are some computers being used by agents that are too old or too new for.

Each. agent has an existing client contact database of about 1,0. In. planning the purchase and deployment of the application, the project.

Short for commercial off-the-shelf, an adjective that describes software or hardware products that are ready-made and available for sale to the general public. Its easy to write about what the government or other people should do with our/their money. It’s harder to come up with a course of action that I can undertake on. Need of User Acceptance Testing. Developers and functional testers are technical people who validate the software against the functional specifications.

Initial feedback from the 1. Management felt that over. The. deployment plan was to have all agents download and install the new. Instructions were posted on the company. A help line was. established to provide support to the agents. On deployment weekend.

About 2. 0% of the agents had problems installing the. Many agents (about 7.

In addition, all of the agents found that the new. Fortunately, many of the agents kept their old. After. four weeks, the company decided to implement another product, but this. All agents. were trained using a web- based training course before the new. The second deployment was a huge success. In the. first project, the results were: A. A. loss of credibility for the project team and the IT department.

A. loss of sales as agents could not use the system to follow- up with. In the. second deployment,The. Agents. were trained to avoid confusion in how to use the product. Testing. was more complete, which gave a higher level of confidence in deploying. In. comparing the deployments, the company learned that: Application. End- users. must understand how to use the product.

The. product must work with other products and on a wide variety of. Although. the vendor tested the product, the customer has responsibility to test. A. product needs to be validated to work with an organization's business. I have seen in acquiring and deploying COTS products. From. this example, we can see the need for testing, but what are the issues. COTS testing and how do we solve them? Unique. Challenges of Testing COTS- based Applications.

Challenge. #1 - COTS is a Black Box. The. customer has no access to source code in COTS products. This forces. testers to adopt an external, black- box, test approach. Although. black- box testing is certainly not foreign to testers, it limits the. This is very troublesome. Functional. testing is redundant by its very nature. From the purely external.

In addition, functional tests miss conditions that are. The bottom line is that in functional. In COTS applications. Solution. Strategies: Avoid complex combinations of tests and the idea of.

The initial tendency of. COTS applications is to start defining tests based on. This is a. slippery slope which can lead to many test scenarios, some meaningful. Challenge. #2 - Lack of Functional and Technical Requirements.

The. message that testing should be based on testable requirements has been. Requirements- based testing has been taught so much, however. Testing from the. COTS product. The. In COTS you may have defined user needs, but you do not have the. In fact, the developer of the software may not.

For the. customer, this means you have to look elsewhere for test cases, such as: Exploring. Business. also a good degree of professional judgment required in designing. Finding test cases is one thing. Finding the. right test cases and understanding the software's behavior is something. Solution. Strategy: Design. The features. you test and the features another customer may test could be very. Consider. the 8. 0/2.

Challenge. #3 - The Level of Quality is Unknown. The COTS. product will have defects, you just don't know where or how many there. For many software vendors, the primary defect metric. I know that sounds rather cynical, but once. Software vendors are in business to make a. Although perfection is a noble goal and (largely) bug- free. It would be nice, however, to at least see.

Many times, known defects are. This. aspect of COTS is where management may have the most unrealistic. A savvy manager will admit the product they have. That same manager, however, will. A. related issue is that the overall level of product quality may actually.

On occasion, some vendors change. Solution. not assume any level of product quality without at least a preliminary. A common strategy is not to be an early customer of a new. It's often wise to wait and see what other users are saying. With today's trade press, there are plenty of forums. Beta. testers are also a good source of early information about a release. An. example of this was when some beta testers noticed that Microsoft. Auto Tune 5 Serial Cracker.

Java Virtual Machine in the Windows XP beta. It's. difficult, if not improbable for a customer to see what methods a.

That's a real. problem, especially when one considers that the quality of software is. Here are. some things you might like to know, but probably will not be able to. How. experienced are the developers?

Which. phases of testing were performed? Which. types of testing were performed? Are. test tools used? Are. defects tracked?

How. do developers collaborate on projects? How. are product features conceived and conveyed to developers? What. type of development methodology used? Is. there any level of customer or user input to the development and. Solution. Strategies: This.

In fact, all vendors require their. Occasionally, you will. Independent. assessments may help, but like any kind of audit or review, people know. Therefore, you may think you are getting.

Challenge. #5 - Compatibility Issues. Software. vendors, especially those in the PC- based arena, have a huge challenge.

When you also. consider peripherals, drivers, and many other variables, the task of. Perhaps the most reasonable goal.

The job. of validating software compatibility is up to the customer to be. With the widely diverse environments. Another. wrinkle is that a product that is compatible in one release may not. Even with. upgrade a product version, you need a plan that addresses: When. Which. users will have their products upgraded. Hardware. and other upgrades that may be needed. Data. conversions that may be needed.

Contingency. plans in case the upgrade is not successful. Solution. Strategies: Test. Have. an upgrade plan in place to avoid incompatibility between users of the. Challenge. #6 - Uncertain Upgrade Schedules and Quality.

When you. select a COTS product for an application solution, the decision is. Although the current. The problem is that upgrade schedules fluctuate greatly, are. When it. comes to future product quality, vendor reputation carries a lot of. Also, past performance of the product is often an indicator of.

This should be a motivator for vendors to maintain. Solution. Strategies: Keep. This may include attending. Find out as much as you can about planned releases and: don't. Challenge. #7 - Varying Levels of Vendor Support. Vendor. support is often high on the list of acquisition criteria. However, how. can you know for sure your assessment is correct?

The perception of. Most people judge the quality. In COTS. applications you are dealing with a different support framework as. When you call technical. Fortune 1. 00. customer vs. One. perspective is to view a new release as a new version of the same basic. In this view, the functions are basically the same, and the.