\documentstyle[guidemacros]{report}
\stdsizes

%include the below line for tty output.  Reduce to "-2" if excluding
%change bars.
%\addtolength{\textwidth}{-2in}


%\newcommand{\draft}[1]{{\huge #1}}

\title{Final report of the RAID Quality Improvement Team}
\author{Adriana Christopher \and
Rocklyn Clarke \and
Gordie Noseworthy \and
Andy Oakland \and
Harold Pakulat \and
Ron Parker \and
Naomi Schmidt}
\date{April 12, 1993}
\maketitle

\begin{document}



\begin{abstract}
The task set before the ``Raid'' Quality Improvement Team (``QIT'') was to
``reduce the number of unaddressed bug reports and software
suggestions in the IBM mainframe and administrative VAX environments.''
After substantial investigation,
we found no evidence of a systemic problem of this type
appropriate for the 7-Step Structured Problem Solving Process.

In this report, we reflect on our experiences as a QIT,
and offer some recommendations for next steps on the part of
Operations and Systems, where the perception of a problem lies.

The Raid QIT requests permission to disband.
\end{abstract}


\section{Introduction}

The ``Raid'' Quality Improvement Team (QIT) was formed by the Information
System (IS) Directors in November of 1992, as one of three QIT's in
a pilot program to determine the applicability of Total Quality
Management (TQM) in the MIT environment.

Raid contained the following members, chosen to present a cross-section
of departments within IS.  

\begin {itemize}
\item    ASD:  Adriana Christopher (leader), Ron Parker
\item    DCNS: Andy Oakland
\item    O\&S:  Rocklyn Clarke, Gordie Noseworthy, Harold Pakulat
\item    Sponsor:  Dan Weir
\item    Facilitator:  Naomi Schmidt
\end {itemize}

The original charter statement of the group was to
``reduce the number of unaddressed bug reports and software
suggestions in the IBM mainframe and administrative VAX 
environments.''  The full charter is enclosed as an appendix.

We intended to follow TQM's ``Seven-Step'' process.  Ultimately, the
team was only able to complete the first two Steps, for reasons
which will be described below.

\section {Selecting a Theme}

Unfortunately, there
was no single clear process described in the charter, so the team attempted
to select one.  
The QIT had specifically been instructed not to address help desk
coordination and account management problems, so this was not an option.

We spent a number of meetings describing all the processes
that members are aware of that fit within the charter, such as
the current Athena bug-tracking process and
Datacomm's problem tracking system.
However, there was no specific process in place for handling
bug reports and software suggestions in the IBM and VAX environments,
which caused some floundering for the QIT.
\footnote{TQM specializes in improving repetitive processes.  Without 
a process already in place, the Seven Steps have nothing to work on.}

We decided to go in search of a process that had a systemic problem,
which we would do by interviewing clients.
We selected as a working theme, ``Reduce the time between a problem report
and problem closure,'' where problem reports included any complaints
or suggestions made by mainframe users.
(This includes any problems that fall between
the cracks, because time$\rightarrow\infty$ in those cases.)

\section {Data Collection and Analysis}

The next Step was to gather data.
We decided to interview principal users of the IBM mainframe, excluding
the administrative clients on VAX mainframes in order to maintain focus
and keep the project to a manageable size.  As VAX clients all had their
own system administrators who were responsible for responding to
problem reports, it didn't make sense to analyze VAX problem reports
as though they were part of a single centralized system.

We assembled a list of 41 key users of the IBM mainframe,
beginning with an existing contact list of client managers and
programmers responsible for production applications,
and augmenting it
with other people known to be frequent users.  We took care to include
representatives from a wide-ranging collection of administrative offices, 
including
Alumni,
the CAO,
Personnel,
Physical Plant,
the Registrar's office,
Admissions,
Resource Development,
Telecommunications, and
the Treasurer's office.

We also made certain to include managers, programmers, and lay users
in our list of interviewees.  We decided to ask the following questions:

\begin{itemize}
\item How often do you report mainframe problems/suggestions/concerns?
\item Whom do you contact and how?
\item Are you kept informed of progress (to your satisfaction)?
\item On a scale of 1 to 5, how happy are you with response time?
\item How well do you think the problem was addressed?
\end{itemize}

After a preliminary survey of eleven users showed that they were generally
satisfied with mainframe service, we added a more focussed question:

\begin{itemize}
\item Do you feel a problem report of yours has ever ``fallen
through the cracks?''
\end{itemize}

Graphs of the resulting aggregate data are enclosed at the end of this report.

There may be some response biases in the answers, causing some
interviewees to be less critical of response time because they know the
service providers.  However, we actually noticed bias in the
other direction, e.g. former O\&S employees tended to be more critical.
After completing the remaining interviews, we came to the following
conclusions. 

\begin{itemize}
\item	IBM mainframe clients seem to be
	generally satisfied with the level of support they receive, and there
	do not appear to be any systemic flaws in the way their
	problem reports are handled.
\item
	At least four different team members reported that some of their
	interviewees spontaneously commented that IBM mainframe support had 
	recently improved, with six months to one year ago being the
	turning point.  
\end{itemize}

There are, obviously, some users who are not completely satisfied, (i.e.,
not all users reported `5's for response time).  We don't mean to trivialize
this information.  But this dissatisfaction seems to be indicative of
unique incidents and particular users, and is {\it not} evidence of any
systemic problem in a recurring process.

With no systemic problem for us to
investigate, there was no root cause for us to search out.
Many of the assertions made in the original charter were not supported 
by the data we had gathered, and we were unable to continue with the
Seven Steps.

\section {Search for Additional Information}

This left us puzzled.  The QIT had obviously been formed in response
to a perception that the handling of problem reports was severely flawed,
but our extensive interviews with clients showed that they had no
such perception.  Where had the perception of failure come from?

We decided to step outside of the official framework and seek additional
information.
We would interview the O\&S director and managers, to see how
{\it they} viewed the current system for handling bug reports and
software suggestions.  In addition to asking them the same six
questions we had asked the clients, we would make the interviews
open-ended to ensure that the managers had a full opportunity
to voice their beliefs.  The persons to be interviewed were Susan Minai-Azary,
Karen Fortoul, Kevin Gedrich, Dave Lambert, and Roger Roach.

To ensure consistency between the interviews, given the small sample
size, all interviews would be carried out in person by the
QIT leader.

\section {Results of Additional Interviews}

These additional interviews revealed the reason for the QIT's formation.
The perceptions of the O\&S managers contrasted sharply with the
perceptions of other clients.  They were strongly dissatisfied with
the handling of internally reported problems and suggestions, and
at least one felt that reports {\it frequently} ``fell through the cracks.''

The following general findings were evident from the additional interviews:

\begin{itemize}
\item Many O\&S managers are dissatisfied with handling of internally
reported problems and suggestions.
\item Several managers mentioned the need for a common problem
tracking system.
\item Managers felt that external customers could be effected by the lack
of internal tracking procedures.
\item Outstanding problems need to be reviewed on a regular basis; 
this is not currently being done.
\item Different groups within O\&S have conflicting priorities, often
leading to a lack of cooperation.
\end{itemize}

There clearly are difficulties with problem report handling within O\&S.
While this is not generally evident to external clients, it generates
a great deal of dissatisfaction among the O\&S managers.
Although this is beyond the scope of the tasks of a QIT as defined
by strict TQM procedures, we have the following recommendations which may
alleviate the problems described by the O\&S managers:

\begin{itemize}
\item There is a need for a common problem resolution process in O\&S to
track both internal and external problems and suggestions.
\item The team to put together such a process should consist of those
persons whose work would be effected by it.
\item As part of its investigation, the team should look at 
processes and tracking systems
that have been tried in the past, and try to understand the reasons for
their success or failure.
\item Supporting the process may require the acquisition of a
software problem tracking package.
\item {\it Once this process is in place and has been used for some time,}
it {\it may} be amenable to TQM's Seven Steps for continuous improvement.
\item There appears to be a need for better communication and cooperation
between the various groups of O\&S.  Suggesting specific ways to address
this problem would be better done by the O\&S managers, and not by this QIT.
\end{itemize}

\section {Reflections on the Process}

Although this QIT was not able to complete the Seven Steps, we hope that
the lessons learned from this experience will be valuable to those
considering setting up other QIT's. 
\footnote{Chapter 3 of Scholtes' {\it The TEAM Handbook}, titled
``Setting the Stage for a Successful Project,'' is especially useful
reading for those involved in chartering QIT's.  Of particular interest
is the ``Project Selection Checklist,'' which sets out criteria for
successful projects.  The process the RAID team was to investigate
does not meet many of these criteria.}

The most important lesson we wish to pass on is that QITs must {\it never}
be formed casually.

Given the level of time and effort required of team members,
preliminary data showing the existance of a problem needs to be collected
and documented before a a QIT is formed and sent to ``fix'' the problem.
We must be careful to distinguish beliefs that a problem {\it may} exist
from hard data showing that it {\it does}.  Similarly,
anecdotes of one or a few historical events are far different from evidence
pointing to a currently recurring problem, and should not be the sole
basis for establishing a QIT.

In order to clearly set the team's direction, 
directors and managers must be willing to provide, in an accessible way,
the information they used in deciding to charter the QIT.

QIT's are formed to improve ongoing processes.  There must be a 
clearly defined process in place for the QIT to address,
preferably one which produces data on a fairly frequent schedule.

Members of a QIT should be users of the process they are improving, and
should feel that improving the process will enable
them to do their own jobs better or make their own work easier.
First-hand experience with the process will enable them to better zero in on
the part of it that needs improvement. 

\section {A Final Note}
One very positive result of this experience is that all group members feel
that they have learned a lot about other areas of IS.  Working on this
interdepartmental team has brought us into contact with people and
departments with whom we do not normally interact, and these contacts
are likely to be beneficial in the future.
\end{document}
