Testing object behavior

Instructor's Guide


intro, methods, objects, contracts, formal, summary, Q/A, literature
Presently, we have no generally accepted framework for testing object-oriented systems. However, it seems likely that we can to some extent reuse the insights and methods coming from traditional testing practice. Further, it seems that we may gain great benefits from adopting a contract based design discipline. In the following, we will study what influence the architectural structure of object-oriented systems has on the practice of testing. In particular, we will look at ways in which to test that the actual behavior of an object conforms to our expectations.

Levels of testing

Adopting an object-oriented approach will generally have a significant influence on the (architectural) structure of the program. Consequently, there will be a somewhat different distinction between levels of testing in comparison with a functional approach. The difference arises from the fact that in an object-oriented system the algorithm is distributed over a number of classes, involving multiple methods, whereas in a functional decomposition the components directly reflect the structure of the algorithm. Another difference comes from the fact that the notion of module in an object-oriented system encompasses both the concept of a class and the concept of a cluster, which is to be understood as a collection of (cooperating) classes. See slide 4-levels.

Levels of testing


slide: Levels of testing

When testing a system, a collection of objects, or an individual object, the effect that an error may not always be visible should be taken into account. It may be the case that erroneous code is simply not executed, or that the error is executed but without any effect on the results of the computation (as was the case for the instance of class A discussed previously). A further distinction must be made between errors that do have an effect on the computation, but nevertheless result in a legal (although erroneous) state, and errors that leave the computation in an illegal state. To understand what this means, however, we need to delineate more precisely the notion of state.

Object behavior

To test the behavior of an object it is necessary to have some knowledge of the internal structure of the object, that is the state the object may be in at successive moments of the computation. For example, a counter object may be regarded as having two states, an initial state zero and a state in which the instance variable is greater than zero. On the other hand, for a bounded counter, bounded by max, three states must be distinguished: an initial state zero, a state characterized by 0 < n < max (where n is the instance variable of the bounded counter), and a state max that represents the terminal state of the counter, unless it can be reset to zero. Although many more states could have been distinguished, it suffices to consider only three states, since all the states (strictly) between zero and max may regarded as being equivalent. Since the actual parameters of a method may influence the transition from one object state to another object state, the values of these parameters must also be taken into account, in a similar way as when testing the extremum input values of a function. See slide 4-methods.

Object test methods -- state transitions

Errors

-- wrong result, illegal state change
slide: Object test methods

The actual testing may occur with reference to a transition matrix displaying the effect of each method invocation. Inspecting a transition matrix based on the internal state of the (instance variables of) the object may seem to be in contradiction with the principle of encapsulation encouraged in the chapter on design. However, providing a means to observe the state of an object is different from allowing clients unrestricted access to its instance variables. As an example, consider the transition matrices for a counter and a bounded counter displayed in slide 4-matrix. Two states are distinguished for the counter, respectively (1) for the state n = 0 and (2) for the state n > 0 , where we assume that the counter has an instance variable n to keep the actual count. For the bounded counter an additional state is added to allow for the possibility that n = max . Checking the behavior of these (admittedly very simple) objects may take place by a sequence of method calls followed by a check to determine whether the expected state changes have taken place.

Transition matrix -- counter

Transition matrix -- bounded counter


slide: Transition matrix

For example, when incrementing a counter initialized to zero we must observe a state change from $(1) to $(2). The important cases to test are the borderline cases. For instance, what happens when we decrement a newly created counter? With regard to the definition of the counter, as expressed by the pre- and post-conditions given in the transition matrix, this operation must be considered illegal since it will lead to an inconsistent state. What to do in such cases depends upon the policy taken when designing the object. When what  [Meyer88] calls a defensive programming approach is followed, calling the method will be allowed but the illegal state change will not occur. When following the (preferred) method of {\em programming by contract} the method call results in a failure due to the violation of a pre-condition, since the user did not conform to the protocol specified in the contract. We will consider this issue further when discussing runtime consistency checking in section consistency.

Identity transitions

Obviously, for other than very simple objects the number of states and the transitions to test for may become quite unwieldy. Hence, a state transition matrix enumerating all the interesting states in general seems not to be a practical solution. A better solution lies in looking for sequences of method calls that have an identical begin and end state. In slide 4-identity, some of the identity transition sequences for the counter are given, but obviously there are many more. One of the interesting features of identity transitions is that they may easily be checked by an automated test tool.

Identity transitions

  counter c; int n1, n2;
  n1 = c.value(); c.inc(1); c.dec(1); n2 = c.value();
  n1 = c.value(); c.inc(1); c.inc(2); c.dec(3); n2 = c.value();
  

Abstract data types

  • stack -- pop( push(s,x) ) = s
  • queue -- remove( insert(q,x) ) != q

Interaction protocols

  • tests all interesting interaction sequences

slide: Identity transitions and interaction protocols

A tool employing identity transitions is discussed in  [Smith90]. The tool generates arbitrarily many sequences of method calls resulting in an identity transition, and also generates the code to test these sequences, that is whether they actually leave the state of the object unaffected. The idea of identity transitions ultimately derives from the axiomatic characterization of invariance properties of abstract data types. For example, when specifying the behavior of a stack algebraically, one of the axioms will be of the form pop(push(s,x)) = s, expressing that first pushing an element on the stack and then popping it results in an identical stack. (See section ADT-algebra for a more detailed discussion of abstract data types.) In contrast, we know that this property does not hold for a queue, unless the queue involved is the empty queue. The advantage of the method of testing for identity transitions is that we need not explicitly specify the individual states and state transitions associated with each method. However, to use automated testing tools, the method requires that we are able to specify by what rules sequences of method calls resulting in identity transitions may be constructed. Moreover, we cannot be sure that we have tested all relevant properties of the object, unless we can prove this from its formal specification. Most difficult to detect, however, are errors that result from not complying to some (implicitly stated) protocol related to multiple objects. For an example, think of the model-view protocol outlined in section 3-mvc. When the initialization of the model-view pairs is not properly done, for instance when a view is not initialized with a model, an error will occur when updating the value of the model. Such requirements are hard if not impossible to specify by means of merely client/server contracts, since possibly multiple objects are involved along with a sequence of method invocations. We will look at formal methods providing support for these issues in section formal-coop. Another tool for testing sequences of method invocations is described in  [Doong90]. The approach relies on an algebraic specification of the properties of the object, and seems to be suitable primarily for testing associativity and commutativity properties of methods.

Runtime consistency checking

Debugging is a hopelessly time-consuming and unrewarding activity. Unless the testing process is guided by clearly specified criteria on what to test for, testing in the sense of looking for errors must be considered as ordinary debugging, that is running the system to see what will happen. Client/server contracts, as introduced in section contracts as a method for design, do offer such guidelines in that they enable the programmer to specify precisely the restrictions characterizing the legal states of the object, as well as the conditions that must be satisfied in order for legal state transitions to occur. See slide 4-contracts.

Assertions -- side-effect free

contracts


  • require -- test on delivery
  • promise -- test during development

Object invariance -- exceptions

  • invariant -- verify when needed

Global properties

-- requirements
  • interaction protocols -- formal specification

slide: Runtime consistency checking

The Eiffel language is the first (object-oriented) language in which assertions were explicitly introduced as a means to develop software and to monitor the runtime consistency of a system. Contracts as supported by Eiffel were primarily influenced by notions concerning the construction of correct programs. The unique contribution of  [Meyer88] consists of showing that these notions may be employed operationally by specifying the pragmatic meaning of pre- and post-conditions defining the behavior of methods. To use assertions operationally, however, the assertion language must be restricted to side-effect free boolean expressions in the language being used. Combined with a bottom-up approach to development, the notion of contracts gives rise to the following guidelines for testing. Post-conditions and invariance assertions should primarily be checked during development. When sufficient confidence is gained in the reliability of the object definitions, checking these assertions may be omitted in favor of efficiency. However, pre-conditions must be checked when delivering the system to ensure that the user complies with the protocol specified by the contract. When delivering the system, it is a matter of contractual agreement between the deliverer and user whether pre- and/or post-conditions will be enabled. The safest option is to enable them both, since the violation of a pre-condition may be caused by an undetected violated post-condition. In addition, the method of testing for identity transitions may be used to cover higher level invariants, involving multiple objects. To check whether the conditions with respect to complex interaction protocols are satisfied, explicit consistency checks need to be inserted by the programmer. See also section global-invariants.