\documentstyle[times,fullpage,rcsid,changebar]{article}

\rcs$Header: /afs/gza.com/product/qa/standards/unit-test/RCS/requirements.tex,v 1.20 1993/12/01 15:59:11 jik Exp $

\title{Unit Test Procedures\\
for the Cambridge Security Business Unit\\
of OpenVision Technologies, Inc.\footnote{\rcsHeader}}
\author{Jonathan Kamens}

\begin{document}

\maketitle

\section{Introduction}

This document outlines the procedures for unit testing of product
modules within the Cambridge Security Business Unit of OpenVision
Technologies, Inc.

{\bf Unit testing}, sometimes called {\bf component testing}, is
``testing conducted to verify the implementation of the design for one
software element (for example, unit, module) or a collection of
software elements'' (ANSI/IEEE Std 1012-1986).  Libraries, user
interfaces and application programs are all examples of modules that
require unit testing.  Furthermore, a module that requires unit
testing may itself be composed of sub-modules which should also be
subject to unit testing, before the entire module is unit-tested as a
whole (it might be said, then, that unit-testing of a module composed
of unit-tested sub-modules is equivalent to integration testing of
that module).

\section{Test categories}
\label{test-areas}

A module's unit test should strive for completeness in the following
categories (listed from most to least desirable):
\begin{enumerate}
\item {\bf Design conformance verification.} Show that the implementation
	agrees with the design.
\label{test-cats:design}
\item {\bf User interface testing.} For modules which have a user
	interface, verify that it functions as expected and as
	specified in the design.
\item {\bf Memory error detection.} Verify that memory is allocated,
	deallocated and referenced correctly.
\item {\bf Boundary error detection.} Verify that boundaries in the code
	are handled correctly.
\label{test-cats:boundary}
\item {\bf Stress testing.} Where applicable, verify that the module
	functions correctly under heavy-load conditions.
\label{test-cats:stress}
\item {\bf Code coverage.} Exercise as much of the source code as possible.
	Ideally, over the course of the unit test, every line in the
	module will execute at least once.
\item {\bf Code complexity evaluation.} Confirm that the source code is
	readable, predictable, and easy to understand.
\end{enumerate}
The unit test is not expected to fully test all of these categories.
The following sections describe each category in more detail.

For the December, 1993 deadline, developers are expected to emphasize
categories \ref{test-cats:design} through \ref{test-cats:boundary}, as
well as category \ref{test-cats:stress} if implied by the module's
requirements.


\subsection{Design conformance verification}

Design conformance verification involves reviewing the design of the
module and verifying that the implementation conforms to it in all
respects (or noting where it fails to conform).

To some extent, the other unit testing categories are subsets of
design conformance verification, because a design has, ``There
shouldn't be any bugs,'' as an axiomatic requirement, and the other
unit testing categories are intended to discover bugs so that they can
be fixed and/or to verify that a module has no (known) bugs.

However, the separate listing of design conformance verification here
refers not to the aspects of it embodied by the other unit testing
categories, but rather to the aspect of it which involves explicit
comparison of the module design to its implementation.

This comparison is usually done manually, i.e., a developer steps
through the design document for the module and shows how each element
of the design is met by the implementation.  However, such a manual
design conformance verification is usually assisted by automatic
tests, created to show how the implementation agrees with specific
aspects of the design.

Design conformance is the most desirable component of any unit test,
because it confirms that ``what the user sees'' is correct.  When
available resources are not sufficient to allow all test categories to
be fully explored, design conformance should be given the highest
priority.

\subsection{User interface testing}

User interface testing includes, but is not limited to, verifying the
following:
\begin{itemize}
\item Correct function arguments produce expected results.
\item Incorrect/invalid function arguments produce expected results.
\item Applications accept the correct command-line arguments and interpret
	those arguments correctly.  Applications do not accept
	command-line arguments that are not in the design.
\item GUIs have the interface specified in design.
\end{itemize}

Furthermore, a module that is an API library also has a user interface
-- the function prototypes and possible return values of the functions
in the library.  Their conformance to the design should be verified.

\subsection{Memory error detection}

Memory errors fall into several categories, including (but not limited
to):
\begin{itemize}
\item Dereferencing uninitialized pointers.
\item Dereferencing null pointers.
\item Failing to deallocate dynamically allocated memory (most often,
	memory allocated by {\em malloc}(3) or {\em calloc}(3) which
	should be deallocated with {\em free}(3)) when it is no longer
	needed.
\item Dereferencing a pointer to dynamically allocated memory that has
	already been deallocated.
\end{itemize}

Several tools exist for detecting these errors.  They include: Purify,
Purify2, CodeCenter, TestCenter, and the {\tt malloclib} library.  One
or more of these tools should be used to detect memory errors in the
module.  Memory error detection is also a high priority.

\subsection{Boundary error detection}

Boundary errors include, but are not limited to:
\begin{itemize}
\item Incorrect ranges in loop statements (e.g., $i \leq 10$ when $i <
10$ was intended) in code.
\item Failure to detect out-of-bounds input (e.g., a user enters $32$
as the day of the month on which a task should run).
\end{itemize}

A unit test should check for boundary errors.

\subsection{Stress testing}

Stress testing means verifying that a module performs correctly when
subjected to peak bursts of activity.  Some modules have requirements
specifying how much activity they must support without failure.  In
that case, stress testing is meant to ensure that the module does not
fail at or below that activity level.  Furthermore, stress testing is
meant to ensure that when a module does fail because of activity
overload, it does so in a graceful manner.

For example, a server which accepts client TCP/IP connections might
have requirements in its design stating how many simulateneous clients
it must support.  Stress testing should verify that it can support
that many clients, and that when the number of current clients is at
its maximum, attempts to connect by additional clients will not lead
to crashing or other incorrect behavior.

As another example, consider an X program which accepts mouse and
keyboard input.  The program's behavior should be acceptable when
it receives a large number of mouse and keyboard events in a short
amount of time.

When activity-level requirements are placed on a module, then stress
testing of that module to verify that it meets those requirements is a
high priority.  Otherwise, stress testing, where applicable, is
desirable, but not a high priority.

\subsection{Code coverage}
\label{code-coverage}
It is unlikely that unit testing at CSBU will be able to achieve 100\%
code coverage in the short term.  However, a unit test should make an
attempt to test as much of the code as possible, and the test design
document (see below) should document the percentage of code exercised
by the unit test.  Coverage of at least 50\% of the code in a module
is a high priority.

Tools for automatic code-coverage analysis include GCT and TestCenter
(see below).  Where possible, such a tool should be used to determine
the code coverage percentage; otherwise, the percentage will have to
be determined by hand (by counting the lines of code exercised by the
unit test).

If complete code coverage is not possible, then a possible alternative
is {\em common branch coverage,} which involves choosing the most
commonly executed alternative of each branch in the code (e.g., {\tt if}
and {\tt case} statements) and ensuring that the unit test exercises that
branch.

\subsection{Code complexity evaluation}

Automated tools exist for evaluating the complexity of code.  However,
it is unlikely that CSBU will be acquiring such tools in the short
term, and we do not have the manpower resources (or, to some extent,
the expertise) for manual code complexity analysis.  Therefore, at
this time, code complexity evaluation is a low priority.

On the other hand, it should be obvious that developers should
endeavor to write clear, readable code, so that when CSBU does start
formally evaluating code complexity, existing code will fall within
acceptable complexity levels.

\section{Deliverables}

A unit test associated with a module has the following deliverables:
\begin{description}
\item[Design.] (also called the ``unit test plan'') A document
	describing the strategy behind the unit test.  This document should
	show how the unit test meets the requirements outlined above, and
	the extent to which it meets them.

	Whenever possible, the design should be submitted to the
	quality assurance group and approved by them before
	the unit test is implemented.
\item[Instructions.] Instructions for executing the unit test and for
	verifying its success.  These instructions should be detailed
	enough that they can be followed by someone other than the
	developer.
\item[Implementation.] The source code, data files, and other
	materials needed to run the unit test.
\item[Results.] A summary of the results of the unit test, especially
	of any anomalies detected.
\end{description}

These deliverables evolve over time.  For example:
\begin{itemize}
\item When a defect is located, a unit test is added to duplicate the
	defect and to verify that it is not reintroduced after it is
	fixed.
\item Some unit test categories may be neglected in early testing because
	of insufficient resources for full unit testing.  When this
	omission is rectified, the unit test deliverables are updated.
\end{itemize}

The files associated with a unit test should be in a subdirectory
called {\tt unit-test} of the directory in which the module's files
are located.  If the module is spread across multiple directories,
then there can be multiple {\tt unit-test} directories as well.

The following sections describe the deliverables in more detail.

\subsection{Design}

The unit test design should contain a section for each of the unit
test categories listed in Section \ref{test-areas}.  If a unit test
category is not applicable to the module being tested, or if the
category is being temporarily ignored due to insufficient resources,
this should be stated explicitly.

The design should explain the techniques and tools used for testing.
It should explain the structure of the unit test and justify that
structure.  It should explain {\em why,} as opposed to the test
instructions, which should explain {\em how.}

The design document should be detailed enough that a developer other
than the initial implementor of the unit test will be able to update
the unit test and/or add new tests easily after reading it.

\subsection{Instructions}

The instructions for running the unit test should be independent of
the design document.  They should be detailed enough that they stand
alone, yet they should also be as brief as possible.  The instructions
should not justify the various components of the test; they should
just explain how to run them.  They should also list the various files
that are part of the unit test implementation.

The instructions should be located in a file called {\tt INSTRUCTIONS}
in the {\tt unit-test} directory.  If the instructions are not
formatted as a plain ASCII file, then they may reside in some other
file, with the {\tt INSTRUCTIONS} file explaining how to access them.

\subsection{Implementation}

The test implementation is the source code, Makefiles, data files,
etc. that are necessary to run the test.

If the unit test implementation is spread across multiple directories,
then (if possible) there should be a ``master'' {\tt unit-test}
directory, from which other tests can be executed with a single
command (e.g., ``{\tt make unit-test}'').

Each {\tt unit-test} directory should have a Makefile in it with a
``unit-test'' rule that performs all available non-interactive,
automated unit testing.  Normal test results should appear as stdout
output, while errors and anomalies should appear on stderr.  This rule
will be used to do a regular (e.g., nightly), automatic ``{\tt make
unit-test}'' run through the source tree.\footnote{Actually, we
probably won't be doing such nightly unit testing before the December
release, but this requirement is included here to make it easier for
us to do such testing in the future.}

Furthermore, each unit test Makefile should have a ``unit-test-root''
rule, for test preparation steps which must be run as root on the
machine on which the tests are being run, before the ``unit-test''
rule is invoked.  This rule can assume read access to the build tree
and write access to the local machine.  If there are no preparation
tests that need to be run as root, the Makefile should contain an
empty ``unit-test-root'' rule.

\subsection{Results}

The results report of a unit test should be composed of two parts -- a
manually composed summary of the results of the test, and the output
of the test programs.  Some unit tests may not have any written
summary at all, while others may not have any program output.

Defects and other anomalies detected by the test should stand out in
the results document.  The report should show obviously whether or not
(or how well) the module passed the unit test in each category.

\subsection{When the unit test is finished}

When a developer has implemented and tested a module, the unit test
design and results report should be submitted to the QA group.  They
will accept the unit test or ask for changes and/or additions to it.

When modifications to a module (e.g., bug fixes) are required after
the unit test has been submitted to, and approved by, the QA group,
the complete unit test should be rerun by the developer in order to
verify that the modifications have not introduced new defects.  If the
unit test changes as a result of the modifications (e.g., if a new
test is added to check for the bug fixed by the modifications), then
the design and results should be resubmitted to the QA group, with the
changes highlighted.

\section{Testing tools}

This section documents the tools available at CSBU for unit testing.
Furthermore, tool-use requirements for the December deliverable are
presented in Section \ref{tool-requirements}.

\subsection{CodeCenter -- interactive program debugging}
\label{codecenter}

CodeCenter, formerly ``saber'', is an interactive debugger and C
interpreter which can run either C code or object code.

To use CodeCenter, add these directories to your search path:
\begin{quotation}
\noindent{\tt /afs/gza.com/packages/CenterLine/bin\\
/afs/gza.com/packages/CenterLine/sparc-sunos4/bin}
\end{quotation}
and this directory to your MANPATH:
\begin{quotation}
\noindent{\tt /afs/gza.com/packages/CenterLine/man}
\end{quotation}
The program can be invoked as {\tt codecenter}.  There is a
``codecenter'' man page.

CodeCenter can also be used non-interactively.  The CodeCenter
variables {\tt batch\_load} and {\tt batch\_run} can be used to
control whether CodeCenter should prompt interactively when warnings
or errors occur, or should just display them and continue running.
Developers are encouraged to include non-interactive CodeCenter runs
in their unit tests, so that any run-time errors that only CodeCenter
(i.e., none of the other tools mentioned in this section) can detect
will be found and fixed.  See the ``options'' manual page inside
CodeCenter for more information about {\tt batch\_load} and {\tt
batch\_run}.

\subsection{dbmalloc -- debugging malloc library}
\label{dbmalloc}

Dbmalloc is a library that can be used to detect programming errors
related to dynamic memory allocation.  There is a man page for it
(``dbmalloc'').  To use it, put {\tt -I/usr/local/debug\_include} at
the front of your include directory list when compiling, include
{\tt <malloc.h>} in your source files, and include {\tt -ldbmalloc}
when linking.

\subsection{DejaGnu -- test driver}

DejaGnu is a framework, based on Expect and Tcl, for running tests.
It provides a consistent environment for tests and a consistent output
format for test reports.  However, it is somewhat complicated and the
documentation (which can be found in the ``DEJAGNU'' binder in the
shelf outside jik's office; there's also a ``dejagnu'' man page) is
inadequate.

Developers who need to write tests which call TTY programs and verify
that their output is correct, or which run a single TTY program, feed
it input, and verify that the output in response to each input is
correct, might find DejaGnu a useful framework in which to implement
their tests.  However, developers doing such tests are not required to
use DejaGnu before the December deliverable.

\subsection{GCT -- Generic Coverage Tool}
\label{gct}

GCT measures branch, loop and relational operator coverage.  When used
with a unit test, it will produce figures for how much of the module
is exercised and detailed information about particular branches, loops
and relational operators that are not sufficiently exercised.

Most developers should be able to use GCT enough to satisfy these
requirements after reading the GCT tutorial, a short document which
can be found in the binder labelled ``GCT'' outside jik's office.
That binder also contains a user's manual and several other small
GCT-related documents which will probably not be needed by most
developers.  The documentation in the binder can also be found in {\tt
/afs/gza.com/software/gct/build/doc/postscript}.  Furthermore, there
are man pages for all GCT commands.

\subsection{mprof -- memory profiling with stack traces}

To use mprof, you link your program against {\tt -lc\_mp} when
compiling.  When your program runs, it will produce a {\tt mprof.data}
file in the current directory.  This file can be turned by the mprof
command into a report showing memory leaks (including stack traces
showing where unfreed memory was allocated) and writes outside the
bounds of allocated data.

See also the ``mprof'' man page.

\subsection{Purify -- locate memory errors}
\label{purify}

Purify detects memory errors related to dynamic memory allocation,
usage of uninitialized variables, and usage of no-longer-valid
pointers.

To use it, add this to your search path:
\begin{quotation}
\noindent{\tt /afs/gza.com/packages/purify}
\end{quotation}
and this to your MANPATH:
\begin{quotation}
\noindent{\tt /afs/gza.com/packages/purify/man}
\end{quotation}
See the
``purify'' man page for more information.

\subsection{TestCenter -- find memory errors and measure code
	efficiency and coverage}
\label{testcenter}

TestCenter shows the amount of time spent on each line of instrumented
source code.  Furthermore, it detects memory errors similar to what
Purify detects, and does rudimentary code coverage, measuring which
lines of code are executed and which are not.

To use it, modify your search path and MANPATH as described for
CodeCenter (Section \ref{codecenter}), and replace {\tt cc} with {\tt
proof cc} (or {\tt gcc} with {\tt proof gcc}, etc.) when compiling.
When the resulting executable is run, the TestCenter GUI is created.

See also the ``testcenter'' man page.

\subsection{XRunner -- GUI tester}

We hope to acquire XRunner, to be used to test our admin GUI, in the
near future.  However, it has not yet been acquired, so nothing more
can be said about it right now.

\subsection{Tool use requirements}
\label{tool-requirements}

Developers must use at least one of dbmalloc, mprof, Purify and
TestCenter to detect memory errors during unit testing.  Since CSBU
currently can legally allow only one developer to use Purify, it
should be avoided until we obtain more licenses.

Developers must use at least one of GCT and TestCenter to measure code
coverage during unit tests and to detect areas where the unit test
might need to be improved.  TestCenter measures code coverage simply
by measuring which lines of code are executed and which are not; GCT,
on the other hand, measures conditional, loop and relational operator
coverage, which is probably more useful (although there's no spiffy
GUI for it).  Furthermore, developers are encouraged to use GCT
for its loop boundary coverage checking, even if they use TestCenter
to obtain simple code-coverage statistics.

\section{General guidelines}

\subsection{Automate testing}

Testing should be automated as much as possible.  If the test requires
that a sequence of commands be run, then put those commands into a
script (or a Makefile) so that the person running the test does not
have to type them all by hand.  Automating the test makes it easier to
perform and more reliable.

\subsection{Reuse utilities}

If, in the process of implementing a unit test, a developer writes a
utility which might be useful to other developers when they do their
tests, the utility should be ``packaged'' so that others can use it
(i.e., it should be cleaned up and isolated enough that it can be
copied into another test directory and used there), and a man page
should be prepared for it.

\subsection{Try not to recompile code for testing}

Whenever possible, the unit test of a module should not require that
any code in the module be recompiled.  Obviously, this may not always
be possible; for example, memory error detection may require
recompiling with Purify.

\subsection{Module isolation is not necessary}

For the December, 1993 deadline, unit tests are not required to be
completely isolated from other modules when such isolation is
infeasible.

For example, a module which depends on the Kerberos libraries may
assume that they function correctly, and the module's unit test may
link against them.  It is not necessary to write stubs to replace the
real Kerberos library functions during the unit test.

This implies that some module unit tests may uncover defects in other
modules.  Such defects should obviously be fixed.

\subsection{Testing during coding is a good idea}

Rather than implementing a unit test all at once when the module is
completely implemented, it is often a good idea to do some unit testing
during implementation.

Such incremental testing often takes the form of ``bottom-up
testing'', when simple sub-modules (e.g., utility functions) in a
module are tested before they are used in other, more complicated
functions.

Often, tests written during module implementation can be used as part
of the overall unit test for the entire module.

\end{document}
