Final report of the RAID Quality Improvement Team

Adriana Christopher Rocklyn Clarke Gordie Noseworthy
Andy Oakland Harold Pakulat Ron Parker

Naomi Schmidt

April 12, 1993

Abstract

The task set before the “Raid” Quality Improvement Team (“QIT”) was to “reduce the number of unaddressed bug reports and software suggestions in the IBM mainframe and administrative VAX environments.” After substantial investigation, we found no evidence of a systemic problem of this type appropriate for the 7-Step Structured Problem Solving Process.

In this report, we reflect on our experiences as a QIT, and offer some recommendations for next steps on the part of Operations and Systems, where the perception of a problem lies.

The Raid QIT requests permission to disband.

Introduction

The “Raid” Quality Improvement Team (QIT) was formed by the Information System (IS) Directors in November of 1992, as one of three QIT’s in a pilot program to determine the applicability of Total Quality Management (TQM) in the MIT environment.

Raid contained the following members, chosen to present a cross-section of departments within IS.

• ASD: Adriana Christopher (leader), Ron Parker

• DCNS: Andy Oakland

• O&S: Rocklyn Clarke, Gordie Noseworthy, Harold Pakulat

• Sponsor: Dan Weir

• Facilitator: Naomi Schmidt

The original charter statement of the group was to “reduce the number of unaddressed bug reports and software suggestions in the IBM mainframe and administrative VAX environments.” The full charter is enclosed as an appendix.

We intended to follow TQM’s “Seven-Step” process. Ultimately, the team was only able to complete the first two Steps, for reasons which will be described below.

Selecting a Theme

Unfortunately, there was no single clear process described in the charter, so the team attempted to select one. The QIT had specifically been instructed not to address help desk coordination and account management problems, so this was not an option.

We spent a number of meetings describing all the processes that members are aware of that fit within the charter, such as the current Athena bug-tracking process and Datacomm’s problem tracking system. However, there was no specific process in place for handling bug reports and software suggestions in the IBM and VAX environments, which caused some floundering for the QIT. ^{1}

We decided to go in search of a process that had a systemic problem, which we would do by interviewing clients. We selected as a working theme, “Reduce the time between a problem report and problem closure,” where problem reports included any complaints or suggestions made by mainframe users. (This includes any problems that fall between the cracks, because time→ ∞ in those cases.)

Data Collection and Analysis

The next Step was to gather data. We decided to interview principal users of the IBM mainframe, excluding the administrative clients on VAX mainframes in order to maintain focus and keep the project to a manageable size. As VAX clients all had their own system administrators who were responsible for responding to problem reports, it didn’t make sense to analyze VAX problem reports as though they were part of a single centralized system.

1TQM specializes in improving repetitive processes. Without a process already in place, the Seven Steps have nothing to work on.

We assembled a list of 41 key users of the IBM mainframe, beginning with an existing contact list of client managers and programmers responsible for production applications, and augmenting it with other people known to be frequent users. We took care to include representatives from a wide-ranging collection of administrative offices, including Alumni, the CAO, Personnel, Physical Plant, the Registrar’s office, Admissions, Resource Development, Telecommunications, and the Treasurer’s office.

We also made certain to include managers, programmers, and lay users in our list of interviewees. We decided to ask the following questions:

• How often do you report mainframe problems/suggestions/concerns?

• Whom do you contact and how?

• Are you kept informed of progress (to your satisfaction)?

• On a scale of 1 to 5, how happy are you with response time?

• How well do you think the problem was addressed?

After a preliminary survey of eleven users showed that they were generally satisfied with mainframe service, we added a more focussed question:

• Do you feel a problem report of yours has ever “fallen through the cracks?”

Graphs of the resulting aggregate data are enclosed at the end of this report.

There may be some response biases in the answers, causing some interviewees to be less critical of response time because they know the service providers. However, we actually noticed bias in the other direction, e.g. former O&S employees tended to be more critical. After completing the remaining interviews, we came to the following conclusions.

• IBM mainframe clients seem to be generally satisfied with the level of support they receive, and there do not appear to be any systemic flaws in the way their problem reports are handled.

• At least four different team members reported that some of their interviewees spontaneously commented that IBM mainframe support had recently improved, with six months to one year ago being the turning point.

There are, obviously, some users who are not completely satisfied, (i.e., not all users reported ‘5’s for response time). We don’t mean to trivialize this information. But this dissatisfaction seems to be indicative of unique incidents and particular users, and is not evidence of any systemic problem in a recurring process.

With no systemic problem for us to investigate, there was no root cause for us to search out. Many of the assertions made in the original charter were not supported by the data we had gathered, and we were unable to continue with the Seven Steps.

Search for Additional Information

This left us puzzled. The QIT had obviously been formed in response to a perception that the handling of problem reports was severely flawed, but our extensive interviews with clients showed that they had no such perception. Where had the perception of failure come from?

We decided to step outside of the official framework and seek additional information. We would interview the O&S director and managers, to see how they viewed the current system for handling bug reports and software suggestions. In addition to asking them the same six questions we had asked the clients, we would make the interviews open-ended to ensure that the managers had a full opportunity to voice their beliefs. The persons to be interviewed were Susan Minai-Azary, Karen Fortoul, Kevin Gedrich, Dave Lambert, and Roger Roach.

To ensure consistency between the interviews, given the small sample size, all interviews would be carried out in person by the QIT leader.

Results of Additional Interviews

These additional interviews revealed the reason for the QIT’s formation. The perceptions of the O&S managers contrasted sharply with the perceptions of other clients. They were strongly dissatisfied with the handling of internally reported problems and suggestions, and at least one felt that reports frequently “fell through the cracks.”

The following general findings were evident from the additional interviews:

• Many O&S managers are dissatisfied with handling of internally reported problems and suggestions.

• Several managers mentioned the need for a common problem tracking system.

• Managers felt that external customers could be effected by the lack of internal tracking procedures.

• Outstanding problems need to be reviewed on a regular basis; this is not currently being done.

• Different groups within O&S have conflicting priorities, often leading to a lack of cooperation.

There clearly are difficulties with problem report handling within O&S. While this is not generally evident to external clients, it generates a great deal of dissatisfaction among the O&S managers. Although this is beyond the scope of the tasks of a QIT as defined by strict TQM procedures, we have the following recommendations which may alleviate the problems described by the O&S managers:

• There is a need for a common problem resolution process in O&S to track both internal and external problems and suggestions.

• The team to put together such a process should consist of those persons whose work would be effected by it.

• As part of its investigation, the team should look at processes and tracking systems that have been tried in the past, and try to understand the reasons for their success or failure.

• Supporting the process may require the acquisition of a software problem tracking package.

• Once this process is in place and has been used for some time, it may be amenable to TQM’s Seven Steps for continuous improvement.

• There appears to be a need for better communication and cooperation between the various groups of O&S. Suggesting specific ways to address this problem would be better done by the O&S managers, and not by this QIT.

Reflections on the Process

Although this QIT was not able to complete the Seven Steps, we hope that the lessons learned from this experience will be valuable to those considering setting up other QIT’s. ^{2}

The most important lesson we wish to pass on is that QITs must never be formed casually.

Given the level of time and effort required of team members, preliminary data showing the existance of a problem needs to be collected and documented before a a QIT is formed and sent to “fix” the problem. We must be careful to distinguish beliefs that a problem may exist from hard data showing that it does. Similarly, anecdotes of one or a few historical events are far different from evidence pointing to a currently recurring problem, and should not be the sole basis for establishing a QIT.

In order to clearly set the team’s direction, directors and managers must be willing to provide, in an accessible way, the information they used in deciding to charter the QIT.

QIT’s are formed to improve ongoing processes. There must be a clearly defined process in place for the QIT to address, preferably one which produces data on a fairly frequent schedule.

Members of a QIT should be users of the process they are improving, and should feel that improving the process will enable them to do their own jobs better or make their own work easier. First-hand experience with the process will enable them to better zero in on the part of it that needs improvement.

A Final Note

One very positive result of this experience is that all group members feel that they have learned a lot about other areas of IS. Working on this interdepartmental team has brought us into contact with people and departments with whom we do not normally interact, and these contacts are likely to be beneficial in the future.

2Chapter 3 of Scholtes’ The TEAM Handbook, titled “Setting the Stage for a Successful Project,” is especially useful reading for those involved in chartering QIT’s. Of particular interest is the “Project Selection Checklist,” which sets out criteria for successful projects. The process the RAID team was to investigate does not meet many of these criteria.