This research provides an ethnographic style of analysis, with
inherent benefits and limitations.  Although the problems are bounded,
the research still provides value within its scope.  The research
answers some questions and poses more; it suggestions caution for
software manufacturers and managers who make and use customizable
software, it highlights phenomena that traditional research models and
methodologies miss and suggests directions for future research.

This section addresses several limitations of this research.

@b[Relevance of studying these groups]

Ethnographic styles of research are commonly questioned with regard to
the generalizability of the groups being studied.  This is a relevant
concern if the goal is to take a small sample from a large population
and make generalizations about the large group from the small.
However, that is not the intent of this research.  One of the
important assumptions underlying this work is that context is an
important component in human interaction with technology.  Thus, much
of the behavior that we see is context-dependent and may differ
greatly from one organization to the next.  Instead of concentrating
only on the 'average' or most common behavior and extracting the rest
as 'noise' or random variation, this style of research explicitly
looks for variation and examines it.  However, this is not to say that
generalizations cannot exist at all.  Certain patterns of behavior may
be detected, such as the presence of translators for example, which
may occur in other organizations.  If so, future studies may choose 
to investigate this particular pattern in greater detail or attempt 
to determine which kinds of environments encourage the patterns.

Another question to address is why study these particular groups?  In
some sense, each is quite different from a 'typical' business
organization (if such a thing exists).  Both are lead users of an
advanced technology and have created software that has influenced the
entire software industry, particularly at the user interface level.
This software has resulted in multi-million dollar investments by
other corporations and has influenced or created internationally
recognized software standards.  @foot[For example, the X Window System
created at Athena, was made into an international software standard in
the shortest amount of time in history.  The Open Software Foundation
was started as a $100 million joint investment between Digital,
Hewlett Packard, IBM and other major computer vendors.  Even its
competitors, Sun and AT&T, support it.  Digital alone has invested
over a billion dollars in creating a DEC version of the X Window
system, which is by far the largest software effort ever undertaken at
Digital and perhaps in any company.  The X Window System is now
available on wide variety of workstations and personal computers.]

As the first active users of the new technology, they provide the
first opportunity to study the effectiveness of the technology in the
field as opposed to the laboratory.  They can help us understand how
people will react to technology that will be in wide-spread use in
five to ten years.

Both organizations value and have been successful at innovation.  They
each have on-site technical experts who can incorporate changes based
on feedback from users.  Staff members are used to trying new software
and some (although not all) groups within both organizations actively
value innovation and strive to create novel and effective uses of the
technology. To the extent that they use the software in innovative
ways, it is useful to study what they do.  To the extent that they are
@i[not] innovative, it is important to begin to understand why.  For
example, if staff members at Project Athena have problems with the X
Window System, then it is quite likely that organizations without the
resources of Project Athena are likely to also experience problems.

Although these groups have a number of unique characteristics that
make them interesting to study, they also share many characteristics
with other organizations.  Both employ a large number of people with
jobs that are similar to those in any corporation, including
secretaries, non-technical managers, financial administrators, etc.
The majority of the staff in both organizations do not program for a
living.  The staff of Project Athena are similar to the staff of an
MIS department in a large corporation, with a number of programmers
and a majority of non-programmers who handle administration.  The
Athena staff provide a service for over 10,000 demanding users and
face very real deadlines.  They must make the kinds of decisions and
trade-offs about the use of the technology that members of any service
organization must make.

@b[Rate of change in the technology]

Another issue is the rate of change in the technology experienced by
the members of staff.  Both organizations studied experience higher
than normal change, although the changes are controlled and released
at regular intervals.  In the Lens study, major releases occur every
four to eight months.  At Athena, major releases occur three times a
year, at the beginning of each semester.  In organizations with
centrally-managed time-sharing systems, such change may occur once a
year or once every two years.  In places in which users use personal
computers, particularly those they've purchased for their own use,
such change may occur once every five or more years, as the hardware
becomes obsolete and is replaced by new hardware.  However, change
does occur.  The reactions of the Athena staff, particularly those who
are not programmers, may represent one form of reaction to such
change.  (This group is likely to react @i[better] to change than
other organizations, given the technical talent locally available,
however, they still actively resist change.)

@b[Limitations of the data collected]

Although I have been able to collect a variety of different kinds of
data, there are additional forms that would be very interesting to
study.  For example, in the Lens study, I had access to a complete
record of all changes in Lens rules for over two years.  In the Athena
study, the data was collected in the form of snapshots which provide a
very good record of change for people who do not change files often
and a less effective record of changes for people who do.  (Each
snapshot shows the date of the most recent file and the date of
previous file change.  In situations in which the snapshots are taken
when there is an overlap between at least one of the files, it is
possible to get a complete record.  However, if several changes are
made, the intermediate versions are lost.)

Another problem is that I do not have a record of the dates of the
communications by translators, nor do I have a complete record.  The
important result from this study is that the translators exist and
have an effect on the organization.  A longitudinal study that
measures both on-line and off-line communication would be necessary to
accurately trace the influence of these people on the organizaton over
a period of time.

Another difficulty is that users forget their "micro-level" decisions
about when to customize.  Thus, in the example of my own customization
use at the begining of Chapter @ref[individual], I might not have
remembered the problem or why I didn't fix it, if I hadn't just
experienced the phenomenon.  Thus, it is important to recognize that
many decisions about when to customize, adjust, learn, or innovate are
made fleetingly, on the spur of the moment, very much influenced by
the immediate factors in the users environment.  This matches
Suchman's concept of "situated action", in which people may derive a
post-hoc analysis of what they do, but that in the actual course of
action, their behavior is much more affected by the immediate
environment around them.  This is not to say that interviews are 
not useful.  However it is important to recognize the limitations.
Given the richness of the data and the range of possible issues 
to study, the interviews provide valuable insights about promising 
phenomena and patterns to study further.

@b[Co-adaptive phenomena]

Co-adaptive phenomena may occur in many places, under many conditions.
It is not my intent to question whether or not co-adaptive phenomena
exist.  With a little reflection, one can think of numerous examples.
@foot[I had a discussion with Vik Vyssotsky, Director of Digital's
Cambridge Research Lab, who immediately came up with an example.  He
described an example of the process he goes through when buying a new
bicycle.  He first finds one that is basically the right size with the
appropriate features.  Then, he begins to ride it.  He realizes that
several things are not adjusted quite right, makes those changes, and
rides again.  Over time, the adjustments become smaller and
eventually, the bicycle is adjusted properly.  If someone borrows it,
the adjustments must be made again.]  What is important is whether or
not the characteristics of co-adaptive phenomena are present in the
interaction between people and technology and whether or not this view
provides insights and causes us to adjust our theories and research
methodologies to accommodate it.

In some situations, the effect of the technology on the users 
may overwhelm the effect of the users on the use of technology.
In other cases, the reverse may be true.  To the extent that those 
are the phenomena of interest, it may not be necessary to take the 
co-adaptive perspective into account.  However, I will argue that 
many studies that do not take this into account are missing something 
important.  For these people, this work is particularly useful.

@newpage

============================================================

Many people here "grew up" with X windows -- some are students who
became staff.  Others are staff who have been here for years.  They
really use the system that way.  People in other organizations will
tend to have the same phenomenon of having early experiences with
computers dictate how they use it -- so people who convert from a
single window environment to a multi-tasking environment may not
fundamentally change their work styles.

(Note:  I may be a particularly parallel sort of person, but I 
also find that I perform certain tasks, such as writing, in a 
highly parallel way.  Having access to a single window when writing 
drives me nuts.  I really want to be able to take notes, think about 
the global picture, edit parts of the text, make figures, work on 
the bibliography and refer to other relevant items on the system.
Although at one level, it's one task:  writing a paper, at another 
level, it's a multi-faceted task and I want to be able to act on 
things as I think of them, not be forced into a linear mode.  When 
I first wrote papers, I did them with a type-writer -- and that was 
a pretty linear process -- although I always had a pad of paper at
my side for notes and ideas as they occurred to me.)



==================================

@b[The study of transition states]

One of the difficulties of research of this sort is that the 
phenomena of interest are fundamentally transitory.  Holding 
one variable fixed and studying another will not provide the 
information necessary to study this phenomenon.  This problem 
has been observed in other fields, such as experimental 
Psychology.  Sidman (1960) is optimistic that solutions will 
be found -- but thirty years later, they have not.

@begin[center]
@begin[i]

The illusion that learning and other behavioral transitions
are continuous processes, a view fostered by 
the almost exclusive utilization of group averages and 
inadequate experimental control, remains almost uncallenged, in 
spite of a few otustanding demonstrations that discontinuous 
change is often to be expected.

When such difficulties are faced squarely, we may expect that the
study of behavioral tranistions will take its palce as a solid
scientific endeavor.  The task is a diffiuclt one, and demands the
painstaking type of exprimental labor that must accompany any
unexplored problem.  I have no way of predicting where such research
will lead, but it is certainly possible to point out thte kind of
initial steps that will have to be taken.  The first requirement will
bea reorientation in expreimental rationale.  The student must no
longer formulate his problem in general terms.  He will not be
studying learning, or even more generally, behavioraltransitions.  he
must first select a specific example of a transition state, and even
further, he must select a specific aspect of that transition for
detailed study.  At this stage of the game he must study the slected property 
of the transition as a phenomenon of interest in its own right, 
and not as an example of a more general class.  Generalizations 
will come in time, after the properties of a number of 
individual transition states have been studied....It is unlikely that 
the resulting science will even remotely resemble what passes today 
as the study of learning.


...As a general rule, whenever a problem seems as ill-defined as the one 
we have been discussing, the root of the trouble will usually be found to 
be lack of information.  Until the basic propoerties of a beavioral process
are made known, little can be accomplished in the way of answering 
more subtle questions.

In fact -- and this brings us back to methodological problems -- the
one large generalization that appears possible is that any
non-instantaneous behavioral transition is a product of special
conditions.  It is highly likely that curvature in a learning curve is
simply an indirect reflection of the interaction of processes other
than those under direct experimental control.  In most experiments on
learning, the subject is actually learning more than he is being
taught.  That is to say, more is being learning than the experimenter
is deliberately trying to teach.  Just as curvature may result from
the averaging of of a number of discontinuous curves ina group of
individuals, it may also result from the combination of a number of
discontinuous curves for individual responses within a single subject.

...Behavioral transitions are always a function not only of the new
variables that produce the transition but also of the variables that
have been maintaining the behavior up to that point.  Transition
states cannot be studied in isolation from their history.

@end[i]
@end[center]
