Identifying the Challenges You Face in Testing Web Applications

Step 1 - Complete the self-assessment questionnaire based on the levels of "None", "Low", "Moderate", and "High".

Step 2 - Add the totals for each assessment category.

Step 3 - Based on your scores, determine the adequacy of each area.

The Key Areas of Focus

This assessment focuses on four key components of the testing effort: People, tools, process and environment. Therefore, the questionnaire will be structured around these four key areas. Each key assessment area contains five questions that can be answered as "None", "Low", "Moderate", and "High." Simply place the appropriate number (0,1,3,or 5) at the level you can best determine your level to be.

You need to be aware that assessments such as this are very subjective. The rankings are in the eye of the assessor and may vary from person to person. Some people have found it helpful to have the assessment performed by several people representing different areas of the organization. The results can be combined to determine an overall set of scores.

The Scoring Legend

The levels of adequacy as represented by assessment are relative to your world. What you see as constituting a "Low" ranking might be "Moderate" for me. Much depends on levels of process rigor and your relative risk. For example, if large sums of money are going to be processed through this application or web site, you would want to set a high bar for achieving a "High" score.

As a general rule, here are some scoring legends you can use as a starting point for your assessment. You can make these more rigorous or lenient as you see fit.

None � There is no capability in this aspect of the project.

Low � There is limited evidence that this aspect of the project is being addressed.

Moderate � There is adequate progress being made in this area, but more could be done.

High � There is an area of mastery for the organization.

The Questions

A Sample Assessment

Joe Smith, a QA manager for the XYZ company, is performing this assessment for his company. The QA organization at XYZ performs no testing, but is responsible for ensuring that the three test teams are performing the test process as defined in the XYZ software process manual. The QA function is also responsible for measuring the results from the projects at XYZ.

The XYZ company is just starting to develop an e-business application that will hopefully be a major source of income for the company. No formal risk assessment has been performed, but some of the senior managers feel that a poor implementation could seriously jeprodize the business. In the past, senior management at XYZ has overridden QA recommendations concerning software defects and have implemented projects with known problems. In the past ten years, there have been three successive QA groups formed and disbanded at XYZ company. The stated reasons for disbanding the previous QA organizations have been that they were delaying the progress of delivering systems and were not able to get along with the development teams at XYZ. It is interesting to note that most of the IT projects at XYZ over the past twelve years have experienced a 50% schedule overrun on average, whether or not a QA or test group was in place.

There are some defined processes for software development at XYZ, but none for web development or web testing. Likewise, there are no standards for web development. People in the IT group at XYZ are not hostile toward processes. In the few experimental web projects at XYZ, some of the problems were related to poor version control, poor usability and lack of compatibility between browsers and platforms.

XYZ has also experimented with automated test tools, but has had limited success with them. XYZ has purchased three different automated test tools over the past seven years, but have not been able to deploy them to test more than ten percent of their test cases. Management is skeptical of buying any more test tools, based on their past poor performance. One of the observations from an outside consulting group is that the reason people did not use the tools owned by XYZ is that 1) people were never trained in how to use the tools and 2) the tools were an optional part of the testing process.

The bright spot in XYZ's web development efforts is that they have invested in a dedicated web server for testing. The test server is under the physical control of the test organization. There is a person responsible for controlling the test environment and there is a process for checking items into the test server environment. There is also a process to monitor the target production web server and to mirror the production server in the test environment. Test data is created to be reused and restored at any point during or after a test. The weak spot in the test environment is the integration of automated test tools. There is an automated test tool installed, but it is not the current release of the tool and few people know how to use it or have any motivation to use it.

Based on the above information, we could score the assessment as follows:

How to Make Sense of Your Score

Add together the sub-total columns for each aspect of the assessment: People, Tools, Processes and Environment. This will be a number between 0 and 25.

As in scoring the assessment, you must also take care to consider the subjective nature of interpreting the results. Here are some general guidelines for interpreting the assessment results:

0 - 5 indicates little or no readiness in an area.

6 - 12 indicates little to marginal readiness in an area.

13 - 20 indicates a moderate level of readiness in an area.

21 - 25 indicates a very high level of readiness in an area.

After you have tabulated your score and determined the level of readiness in each area, you are ready to interpret the results as they apply to your project. You may want to only use the numbers as a means to conveying the assessment results, or you may want to show them graphically. I like to use a Kiviat or radar chart to graphically display the results.

In the above example, the lowest area of readiness is the people aspect, with a score of 4. The highest level is that of environment with a score of 21. Other low areas include process and tools.

Showing these results graphically looks like this.

This assessment indicates that the organization has a good grip on the test environment, but is struggling with people, tools and processes.

How to Apply the Results

Performing an assessment like this is like getting a physical examination. After you receive the information from the examination, you have a choice to either act upon or ignore the information. The value of the activity depends on the degree you act upon the revealed needs.

In the web testing needs assessment, the low scores indicate gaps in the testing effort. You can take corrective action to remedy the shortcomings, or you can ignore the recommendations and take the risks of missing some important test considerations.

In taking corrective action, you can use the assessment information to focus on the immediate pain, show reasons for changing existing practices, or as a benchmark for determining the level of future progress.

Focus on the Pain

If you are currently experiencing problems in web testing that are inhibiting your success, the assessment can show you where to focus your improvement efforts. This is the most obvious application of the assessment results, but not necessarily the easiest to perform. Most people know where the problems are, but sometimes management is not aware of the problems until they see them documented.

Use the Assessment as a Tool for Change

Now that you know where the trouble spots are, the challenge is to initiate positive change to correct them. Some problems can be corrected quickly, while others may require extensive planning and cultural change.

A helpful way to structure an action plan for change is to identify improvements that can be implemented in the short term (0 - 2 months), mid term (2 - 6 months) and long term (6 - 8 months). By attacking the easier, short-term tasks first, you can show quick successes and gain momentum while you are working on the improvement tasks that will take longer.

Use the Assessment as a Benchmark

In any improvement effort you need a way to measure your progress. The assessment results give you a way to look back a year from now to see how much you have improved. This is a process of performing the assessment a second time (or third, fourth, etc.) and comparing your scores to previous ones. Using the assessment as a benchmark also serves a very helpful purpose by placing an expectation of a future follow-up examination.

Yes, the world of web-based development is changing at a rapid pace. If you have solid components (people, tools, process and environment) in place, you will be better prepared to adapt when changes are required.

Conclusion

There is much about testing web-based applications that is different from testing in other environments. There is also much that remains the same in testing web-based applications. The trick is to keep what works in testing and adapt it to web-based applications where adaptation makes sense. This book is designed to be a guide in applying new and proven techniques for testing web-based applications by providing processes and examples for your own application.

The one thing that web-based testers agree on is that testing must be performed very quickly to keep up with the rapid pace of web development. Using ad-hoc processes, tools and environments takes more time than stable environments because you keep re-inventing the wheel. By building a solid test infrastructure designed for flexibility, you are positioned to react to changes quickly.

An effective way to determine where to start in building a good testing infrastructure is to perform a self-assessment for web testing. Such an assessment can serve multiple purposes and form the basis of a action plan for improvement.