intro,
methods,
objects,
contracts,
formal,
summary,
Q/A,
literature
Presently, we have no generally accepted framework for testing
object-oriented systems.
However, it seems likely that we can to some extent reuse
the insights and methods coming from
traditional testing practice.
Further, it seems that we may gain great benefits from
adopting a contract based design discipline.
In the following, we will study what influence the architectural
structure of object-oriented systems
has on the practice of testing.
In particular, we will look at ways in which to test
that the actual behavior of an object conforms to our expectations.
Levels of testing
Adopting an object-oriented approach will
generally have a significant influence
on the (architectural) structure of the program.
Consequently, there will be a somewhat different
distinction between levels of testing
in comparison with a functional approach.
The difference arises from the fact that
in an object-oriented system the algorithm
is distributed over a number of classes,
involving multiple methods, whereas in a functional
decomposition the components directly
reflect the structure of the algorithm.
Another difference comes from the fact
that the notion of module
in an object-oriented system encompasses both
the concept of a class and the concept of a
cluster, which is to be understood as
a collection of (cooperating) classes.
See slide 4-levels.
Levels of testing
algorithms -- methods
class -- interaction between methods and instance variables
When testing a system, a collection of objects, or an individual
object,
the effect that an error may not always be visible
should be taken into account.
It may be the case that erroneous code is simply not
executed, or that the error is
executed but without any effect on the
results of the computation
(as was the case for the instance of class A discussed
previously).
A further distinction must be made between errors
that do have an effect on the computation,
but nevertheless result in a legal (although erroneous)
state, and errors that leave the computation in
an illegal state.
To understand what this means, however, we need
to delineate more precisely the notion of state.
Object behavior
To test the behavior of an object
it is necessary to have some knowledge of
the internal structure of the object,
that is the state the object may be in at successive
moments of the computation.
For example, a counter object may be regarded as having two
states, an initial state zero and a state
in which the instance variable is greater than zero.
On the other hand, for a bounded counter,
bounded by max, three states must be distinguished:
an initial state zero, a state characterized by
(where n is the instance variable of
the bounded counter), and a state max that represents
the terminal state of the counter, unless
it can be reset to zero.
Although many more states could have been distinguished,
it suffices to consider only three states, since
all the states (strictly) between zero and max may
regarded as being equivalent.
Since the actual parameters of a method
may influence the transition from one object state
to another object state, the values of these parameters
must also be taken into account,
in a similar way as when testing the extremum
input values of a function.
See slide 4-methods.
The actual testing may occur with reference to a transition
matrix displaying the effect of each method invocation.
Inspecting a transition matrix based on the
internal state of the (instance variables of)
the object may seem to be in contradiction with
the principle of encapsulation encouraged
in the chapter on design.
However, providing
a means to observe the state of an object is different
from allowing clients unrestricted access to
its instance variables.
As an example, consider the transition matrices
for a counter and a bounded counter
displayed in slide 4-matrix.
Two states are distinguished for the
counter, respectively for the state
and for the state ,
where we assume that the counter has an instance variable
n to keep the actual count.
For the bounded counter an additional state is
added to allow for the possibility that .
Checking the behavior of these (admittedly very simple)
objects may take place by a sequence of
method calls followed by a check to determine
whether the expected state
changes have taken place.
For example, when incrementing a counter initialized
to zero we must observe a state change from $(1).
The important cases to test are the borderline cases.
For instance, what happens when we decrement a newly created
counter?
With regard to the definition of the counter,
as expressed by the pre- and post-conditions
given in the transition matrix, this operation
must be considered illegal
since it will lead to an inconsistent state.
What to do in such cases depends upon the policy
taken when designing the object.
When what [Meyer88] calls a defensive programming
approach is followed, calling the method will be allowed
but the illegal state change will not occur.
When following the (preferred) method of {\em programming
by contract} the method call results in a failure
due to the violation of a pre-condition,
since the user did not conform to the protocol
specified in the contract.
We will consider this issue further when discussing
runtime consistency checking in section consistency.
Identity transitions
Obviously, for other than very simple objects
the number of states and the transitions to test
for may become quite unwieldy.
Hence, a state transition matrix enumerating all
the interesting states in general seems not to be a practical
solution.
A better solution lies in looking for
sequences of method calls that have an identical
begin and end state.
In slide 4-identity, some of the identity transition sequences
for the counter are given,
but obviously there are many more.
One of the interesting features of identity transitions
is that they may easily be checked by an automated test
tool.
A tool employing identity transitions is discussed in
[Smith90].
The tool generates arbitrarily many sequences
of method calls resulting in an identity
transition,
and also generates the code to test these
sequences,
that is whether they actually leave the
state of the object unaffected.
The idea of identity transitions ultimately
derives from the axiomatic characterization
of invariance properties of abstract data types.
For example, when specifying
the behavior of a stack algebraically, one of the axioms
will be of the form ,
expressing that first pushing an element on the stack
and then popping it results in an identical stack.
(See section ADT-algebra for a more detailed discussion of
abstract data types.)
In contrast, we know that this property does
not hold for a queue, unless the queue involved is the empty
queue.
The advantage of the method of testing for
identity transitions is that we need not
explicitly specify the individual states
and state transitions associated with each method.
However, to use automated testing tools,
the method requires that we are able to specify by what
rules sequences of method calls resulting in
identity transitions may be constructed.
Moreover, we cannot be sure that we have
tested all relevant properties of the object,
unless we can prove this from its formal
specification.
Most difficult to detect, however, are
errors that result
from not complying to some (implicitly stated) protocol related to multiple objects.
For an example, think of the model-view
protocol outlined in section 3-mvc.
When the initialization of the model-view pairs
is not properly done, for instance when a view
is not initialized with a model,
an error will occur when updating the value of the
model.
Such requirements are hard if not impossible
to specify by means of merely client/server contracts,
since possibly multiple objects are involved along with
a sequence of method invocations.
We will look at formal methods providing support for these issues in section
formal-coop.
Another tool for testing sequences of method
invocations is described in [Doong90].
The approach relies
on an algebraic specification of the
properties of the object, and seems to be suitable
primarily for testing associativity
and commutativity properties of methods.
Runtime consistency checking
Debugging is a hopelessly time-consuming and unrewarding
activity.
Unless the testing process is guided by clearly
specified criteria on what to test for,
testing in the sense of looking for errors
must be considered as ordinary debugging,
that is running the system to see what will happen.
Client/server contracts, as introduced
in section contracts as a method for design,
do offer such guidelines in that they enable the programmer
to specify precisely the restrictions characterizing
the legal states of the object,
as well as the conditions that must be satisfied
in order for legal state transitions to occur.
See slide 4-contracts.
The Eiffel language is the first (object-oriented)
language in which assertions were explicitly
introduced as a means to develop software
and to monitor the runtime consistency of a system.
Contracts as supported by Eiffel were primarily
influenced by notions concerning the construction
of correct programs.
The unique contribution of [Meyer88] consists
of showing that these notions may be employed operationally
by specifying the pragmatic meaning of pre- and post-conditions
defining the behavior of methods.
To use assertions operationally, however, the
assertion language must be restricted to
side-effect free
boolean expressions
in the language being used.
Combined with a bottom-up approach to development,
the notion of contracts gives rise to the following
guidelines for testing.
Post-conditions and invariance assertions should
primarily be checked during development.
When sufficient confidence is gained in the reliability
of the object definitions, checking these assertions
may be omitted in favor of efficiency.
However, pre-conditions must be checked
when delivering the system to ensure that
the user complies with the protocol specified by the contract.
When delivering the system, it is a matter of contractual
agreement between the deliverer and user
whether pre- and/or post-conditions will be enabled.
The safest option is to enable them both,
since the violation of a pre-condition
may be caused by an undetected violated post-condition.
In addition, the method of testing for identity transitions
may be used to cover higher level invariants,
involving multiple objects.
To check whether the conditions with
respect to complex interaction protocols are
satisfied, explicit consistency checks need to be inserted
by the programmer. See also section global-invariants.