A framework for testing object-oriented programs

Presently, we have no generally accepted framework for testing object-oriented systems. However, it seems likely that we can to some extent reuse the insights and methods coming from traditional testing practice. Further, it seems that we may gain great benefits from adopting a contract based design discipline. In the following, we will study what influence the architectural structure of object-oriented systems has on the practice of testing. In particular, we will look at ways in which to test that the actual behavior of an object conforms to our expectations.

Levels of testing

Adopting an object-oriented approach will generally have a significant influence on the (architectural) structure of the program. Consequently, there will be a somewhat different distinction between levels of testing in comparison with a functional approach. The difference arises from the fact that in an object-oriented system the algorithm is distributed over a number of classes, involving multiple methods, whereas in a functional decomposition the components directly reflect the structure of the algorithm. Another difference comes from the fact that the notion of module in an object-oriented system encompasses both the concept of a class and the concept of a cluster, which is to be understood as a collection of (cooperating) classes. See slide 4-levels.

Levels of testing

Influence of errors


slide: Levels of testing

When testing a system, a collection of objects, or an individual object, the effect that an error may not always be visible should be taken into account. It may be the case that erroneous code is simply not executed, or that the error is executed but without any effect on the results of the computation (as was the case for the instance of class A discussed previously). A further distinction must be made between errors that do have an effect on the computation, but nevertheless result in a legal (although erroneous) state, and errors that leave the computation in an illegal state. To understand what this means, however, we need to delineate more precisely the notion of state.

Testing the behavior of objects

To test the behavior of an object it is necessary to have some knowledge of the internal structure of the object, that is the state the object may be in at successive moments of the computation. For example, a counter object may be regarded as having two states, an initial state zero and a state in which the instance variable is greater than zero. On the other hand, for a bounded counter, bounded by max, three states must be distinguished: an initial state zero, a state characterized by $0 < n < max (where n is the instance variable of the bounded counter), and a state max that represents the terminal state of the counter, unless it can be reset to zero. Although many more states could have been distinguished, it suffices to consider only three states, since all the states (strictly) between zero and max may regarded as being equivalent. Since the actual parameters of a method may influence the transition from one object state to another object state, the values of these parameters must also be taken into account, in a similar way as when testing the extremum input values of a function. See slide 4-methods.

Object test methods -- state transitions

  • equivalence classes -- distinct \c{object} states
  • extrema testing -- includes parameters \c{of methods}

Errors

-- wrong result, illegal state change
  • within object -- invariance
  • involving multiple objects -- interaction protocols

slide: Object test methods

The actual testing may occur with reference to a transition matrix displaying the effect of each method invocation. Inspecting a transition matrix based on the internal state of the (instance variables of) the object may seem to be in contradiction with the principle of encapsulation encouraged in the chapter on design. However, providing a means to observe the state of an object is different from allowing clients unrestricted access to its instance variables. As an example, consider the transition matrices for a counter and a bounded counter displayed in slide 4-matrix. Two states are distinguished for the counter, respectively $(1)
for the state n = 0 and $(2) for the state n > 0, where we assume that the counter has an instance variable n to keep the actual count. For the bounded counter an additional state is added to allow for the possibility that n = max. Checking the behavior of these (admittedly very simple) objects may take place by a sequence of method calls followed by a check to determine whether the expected state changes have taken place.

Transition matrix

-- counter
slide: Transition matrix

For example, when incrementing a counter initialized to zero we must observe a state change from $(1)
to $(2). The important cases to test are the borderline cases. For instance, what happens when we decrement a newly created counter? With regard to the definition of the counter, as expressed by the pre- and post-conditions given in the transition matrix, this operation must be considered illegal since it will lead to an inconsistent state. What to do in such cases depends upon the policy taken when designing the object. When what  [Meyer88] calls a defensive programming approach is followed, calling the method will be allowed but the illegal state change will not occur. When following the (preferred) method of {\em programming by contract} the method call results in a failure due to the violation of a pre-condition, since the user did not conform to the protocol specified in the contract. We will consider this issue further when discussing runtime consistency checking in section consistency.

Identity transitions

Obviously, for other than very simple objects the number of states and the transitions to test for may become quite unwieldy. Hence, a state transition matrix enumerating all the interesting states in general seems not to be a practical solution. A better solution lies in looking for sequences of method calls that have an identical begin and end state. In slide 4-identity, some of the identity transition sequences for the counter are given, but obviously there are many more. One of the interesting features of identity transitions is that they may easily be checked by an automated test tool.

Identity transitions

  counter c; int n1, n2;
  n1 = c.value(); c.inc(1); c.dec(1); n2 = c.value();
  n1 = c.value(); c.inc(1); c.inc(2); c.dec(3); n2 = c.value();
  

Abstract data types

  • stack -- pop( push(s,x) ) = s
  • queue -- remove( insert(q,x) ) != q

Interaction protocols

  • tests all interesting \c{interaction} sequences

slide: Identity transitions and interaction protocols

A tool employing identity transitions is discussed in  [Smith90]. The tool generates arbitrarily many sequences of method calls resulting in an identity transition, and also generates the code to test these sequences, that is whether they actually leave the state of the object unaffected. The idea of identity transitions ultimately derives from the axiomatic characterization of invariance properties of abstract data types. For example, when specifying the behavior of a stack algebraically, one of the axioms will be of the form
pop(push(s,x)) = s, expressing that first pushing an element on the stack and then popping it results in an identical stack. (See section ADT-algebra for a more detailed discussion of abstract data types.) In contrast, we know that this property does not hold for a queue, unless the queue involved is the empty queue. The advantage of the method of testing for identity transitions is that we need not explicitly specify the individual states and state transitions associated with each method. However, to use automated testing tools, the method requires that we are able to specify by what rules sequences of method calls resulting in identity transitions may be constructed. Moreover, we cannot be sure that we have tested all relevant properties of the object, unless we can prove this from its formal specification. Most difficult to detect, however, are errors that result from not complying to some (implicitly stated) protocol related to multiple objects. For an example, think of the model-view protocol outlined in section 3-mvc. When the initialization of the model-view pairs is not properly done, for instance when a view is not initialized with a model, an error will occur when updating the value of the model. Such requirements are hard if not impossible to specify by means of merely client/server contracts, since possibly multiple objects are involved along with a sequence of method invocations. We will look at formal methods providing support for these issues in section formal-coop. Another tool for testing sequences of method invocations is described in  [Doong90]. The approach relies on an algebraic specification of the properties of the object, and seems to be suitable primarily for testing associativity and commutativity properties of methods.

Runtime consistency checking

Debugging is a hopelessly time-consuming and unrewarding activity. Unless the testing process is guided by clearly specified criteria on what to test for, testing in the sense of looking for errors must be considered as ordinary debugging, that is running the system to see what will happen. Client/server contracts, as introduced in section contracts as a method for design, do offer such guidelines in that they enable the programmer to specify precisely the restrictions characterizing the legal states of the object, as well as the conditions that must be satisfied in order for legal state transitions to occur. See slide 4-contracts.

Assertions

-- side-effect free

contracts

  • require -- test on delivery
  • promise -- test during development

Object invariance

-- exceptions
  • invariant -- verify when needed

Global properties

-- requirements
  • interaction \c{protocols} -- formal specification

slide: Runtime consistency checking

The Eiffel language is the first (object-oriented) language in which assertions were explicitly introduced as a means to develop software and to monitor the runtime consistency of a system. Contracts as supported by Eiffel were primarily influenced by notions concerning the construction of correct programs. The unique contribution of  [Meyer88] consists of showing that these notions may be employed operationally by specifying the pragmatic meaning of pre- and post-conditions defining the behavior of methods. To use assertions operationally, however, the assertion language must be restricted to side-effect free boolean expressions in the language being used. Combined with a bottom-up approach to development, the notion of contracts gives rise to the following guidelines for testing. Post-conditions and invariance assertions should primarily be checked during development. When sufficient confidence is gained in the reliability of the object definitions, checking these assertions may be omitted in favor of efficiency. However, pre-conditions must be checked when delivering the system to ensure that the user complies with the protocol specified by the contract. When delivering the system, it is a matter of contractual agreement between the deliverer and user whether pre- and/or post-conditions will be enabled. The safest option is to enable them both, since the violation of a pre-condition may be caused by an undetected violated post-condition. In addition, the method of testing for identity transitions may be used to cover higher level invariants, involving multiple objects. To check whether the conditions with respect to complex interaction protocols are satisfied, explicit consistency checks need to be inserted by the programmer. See also section global-invariants.

Example -- robust programming

As an example of how assertions may be applied to characterize the possible states of an object and to guard its runtime consistency, consider the doubly-bounded counter in slide 4-robust.
  class ctr { 
\fbox{doubly-bounded \c{counter}
int n, lb, ub; public: ctr(int l, int, u) : n(0), lb(l), ub(u) { promise( invariant() ); } void inc() { require( n < ub ); n++; promise( invariant() ); } void dec() { require( lb < n ); n--; promise( invariant() ); } int value() { return n; } protected: bool invariant() { return lb <= n && n <= ub; } };

slide: Robust programming

The counter has both a lower and upper bound that are set when constructing the object. Both the functions inc and dec have pre-conditions, respectively stating that incrementing the counter is legal only when its value is less than its upper bound and, similarly, that decrementing a counter may be done only when its value is greater than its lower bound. This characterization is clearly equivalent to a characterization as given by the transition matrix for a bounded counter. The implementation of the counter is robust, since it guards clients against possible misuse. The advantage of using assertions, apart from providing checks to test legal usage, is that they explicitly state the requirements imposed on the user.

Example -- binary tree

As a slightly less academic example (due to Meyer, 1992b), consider the implementation of a binary tree, consisting of nodes that are kept in a certain order. See slide 4-inv-1.
  template 
\fbox{binary tree}
class tree { public: tree( tree* p, T& n ) : parent(p) { left = right = 0; node = n; } void insert( tree* t ) { require( t != 0 ); insert_node( t ); promise( invariant() ); } virtual bool invariant() { return ( left == 0 || left->parent == this ) && ( right == 0 || right->parent == this ); } protected: tree *left, *right, *parent; void insert_node(tree* t);
\c{// does the real work}
T& node; };

slide: Checking invariants

How a node is actually inserted is not important; the only requirement imposed is simply that the inserted node does exist. However, we must guarantee that, in whatever way the node is inserted, the (ordered) structure of the tree is preserved. This requirement is expressed in the invariant, which states that whenever a child does exist, it points to the current object as its parent. Now, when it comes to testing, we may wish to check more thoroughly whether the ordered structure of the tree is indeed preserved when inserting a node. This may be done in a non-intrusive way by refining the tree class as in slide 4-inv-2.
  template  
\c{\fbox{test version}
class sortedtree : public tree { public: sortedtree( tree* p, T& n ):tree(p,n){} protected: bool invariant() {return sorted()&&tree::invariant();} int sorted();
check for order

};

slide: Test version

Assume that we have defined a function
sorted() to check whether the tree has the right order. Because the original tree invariant has been defined as a virtual function, we may rely on the dynamic binding mechanism to check the strengthened invariant when inserting a node. Thus, without much trouble, we have created a more robust version of the tree that may be used during testing and later be replaced by the original version.

Static testing

The test methods just described, whether organized around transition matrices or contracts, both involve the execution of the program and looking for errors. Either way, this is a laborious and time-consuming task. To avoid dynamic testing, several methods of program validation have been proposed that do not require the program to run. These methods may be referred to as static testing. One of the oldest, and perhaps most fruitful, methods is simply careful reading.  [Fiedler] reports that in an experimental setting most errors were detected by carefully reading the relevant program text. The most explicit proponent of this method is undoubtly  [Knuth92], who has proposed (and demonstrated) the discipline of literate programming. Although tools supporting literate programming in C++ are available (see section pde), no environment is as yet available that fully supports literate programming in an object-oriented style. See slide 4-static.

Methods for static testing

  • careful reading -- most successful (?)
  • code inspection -- looking for errors
  • walkthrough -- simulation game
  • correctness proof -- rigorous, but complex

slide: Static testing

Another method of static testing, based on similar assumptions, is groupwise code inspection. This method may profitably be used by teams of programmers. Although the attention is primarily directed towards the detection of errors, the method has proved to be beneficial for improving the code. An additional advantage is that it provides a background for discussing terminological issues, programming practice and opportunities for code reuse. More directed towards operational issues is the method of walkthroughs. A walkthrough is similar to the simulation game proposed for CRC cards (see section CRC). The idea is that by simulating a computation, while reading the relevant parts of the code, errors will come to light. As for code inspection, walkthroughs are best performed in a group. When employing groupwise code inspection it is advisable to have a small group of about four people, with a chairman to organize the meeting and with the author of the code as a silent observer. Each participant should receive the code and documentation a few days ahead, as well as a checklist with commonly occurring errors. As a goal, the actual meeting should result in a list of faults detected in the code. A quite different method of static testing, which in contrast to the previous methods is not often used in practice, is to provide correctness proofs for the relevant parts of the program. Proving a program correct is by far the most rigorous of all methods discussed, but unfortunately quite complex and demanding with respect to the formal skills of the programmer. However, actually proving a program correct seems to be far more difficult than only annotating the program with appropriate invariants and pre- and post-conditions. I believe that the notion of contracts provides a valuable means both to reason about the program and to check (dynamically) for the runtime consistency of a system, even without detailed correctness proofs. Not as a means of static testing but as a way of increasing our belief in the reliability of software, it may be advisable to take recourse to bottom-up development. According to  [Meyer88], an object-oriented approach lends itself extremely well to bottom-up development. Instead of trying to grasp the functionality of a system as a whole, small well-understood building blocks may be constructed (preferably documented by contracts) which may be used for increasingly complex abstractions.