Parallelism and distribution

\label{des/per/concept} One of the reasons for employing parallelism in a program is the need for efficiency. Preferably, the parallelism remains implicit, which means that the compiler takes care of how the concurrent execution of the program will take place. However, realizing implicit parallelism is in practice quite difficult and in most cases the programmer will have to indicate explicitly how to execute the program concurrently. Another, equally valid motivation for employing parallelism is that it better fits the conceptual structure of the problem to be solved. As an example, to synchronize the behavior of objects it may be worthwhile to allow objects to have activity of their own. C.f.  [Pe89] and  [Bro86]. \parindex{geographical distribution} As an additional aspect, the processes that are involved in a concurrent computation may be geographically distributed, either because they need resources residing on distant processors or to attain an increase in execution speed. See  [CDJ84]. Programming languages that allow the computation to be spread over a number of distinct processors connected by a network are called {\em distributed programming languages}.

Computation models

\parindex{computation models} \disindex{computation models} When classifying distributed programming languages we can distinguish between three distinct underlying computation models. The most basic of these is that of communicating sequential processes, first presented in the influential paper of  [Ho78]. Object-based concurrent languages may be regarded as extending this basic model, by generalizing communication and providing additional features for synchronization and protection. Finally, the model underlying concurrent logic programming languages is perhaps the most flexible of these, since it allows to mimick the two previous ones. The languages that we will refer to in this section are listed in the table below. .so tl Distributed programming languages may differ in what is employed as the unit of parallelism, the way they deal with communication and how partial failures are handled. In  [Ba89] an overview is given of a number of distributed programming languages, discussing the choices made with regard to these dimensions.

Parallelism

\disindex{unit of parallelism} There seems to be abundant choice in what to take as the unit of parallelism, to name a few: processes (CSP), tasks (Ada), active objects (POOL), multi-threaded objects (Emerald), clauses (Concurrent Prolog and Parlog), or even statements (Occam). With respect to the granularity of computation, we encounter Concurrent Prolog and Parlog on the side of the spectrum of languages supporting small-grain parallelism and Ada or POOL on the other side, supporting large-grain parallelism. Large-grain parallelism means that, proportionally, the amount of computation significantly exceeds the time spent communicating with other processes. Another important issue is whether a language supports the allocation of processes to processors. Allocation is supported for instance by POOL and Occam.

Communication

\disindex{communication} Another decision that must be made concerns the way communication is dealt with. As alternatives we encounter data sharing and message passing. We mention Linda as an interesting example of data sharing.\ftn{ The apparent contradiction between distribution and data sharing is resolved by making a distinction between physical data sharing and logical data sharing. Obviously, we mean the latter here. Logical data sharing provides the programmer with the illusion of common data by hiding the physical distribution of the data. } Also Concurrent Prolog and Parlog, utilizing shared logical variables as the medium of communication, deserve to be classified among the distributed languages. Choosing for message passing we may employ point to point connections (CSP), channels (Occam, Delta Prolog) or broadcasting. Communication may to a certain extent be non-deterministic. For example, both the select statement of Ada and the guarded Horn clauses of Concurrent Prolog and Parlog result in a choice for a particular communication, ignoring alternatives.

Failures

\disindex{failures} \disindex{partial failure} As an additional feature, some of the languages mentioned in  [Ba89] handle partial failure by offering exceptions, atomic sections or recovery mechanisms. Failure may be due to, for example, hardware errors or the violation of integrity constraints. We wish to remark that such failures are rather different from the failure encountered in a language such as Prolog. Failure in Prolog is one of the possible outcomes of a computation; it may even be used to generate all the solutions to a particular goal. \nop{

Computation models

\parindex{computation models} \disindex{computation models} When classifying the languages mentioned we can distinguish between three computation models underlying distributed programming. The most basic of these is that of communicating sequential processes, first presented in the influential paper  [Ho78]. Object based concurrent languages may be regarded as extending this basic model, by generalizing communication and providing additional features for synchronization and protection. Finally, the model underlying concurrent logic programming languages is perhaps the most flexible of these, since it allows to mimick the two previous ones. }

Communicating sequential processes

\label{des/per:CSP} \parindex{sequential processes} The basic model of a distributed programming language is that of a group of sequential processes that run in parallel and communicate by message passing. By a sequential process we mean a process with a single thread of control. C.f.  [We87].

Processes

\parindex{processes} The prime example of a language supporting communicating sequential processes is CSP.  [Ho78]. In CSP we may create a fixed number of parallel processes by using a parallel operator. Each process consists of a name and a body, that is a sequence of statements. Communication between processes is achieved by using send and receive statements. As an example, consider the program [] [ p1 :: p2!3 || p2 :: p1?n ] where process p1 is about to execute the statement p2!3, sending the value 3 to process p2 and p2 is about to execute p1?n, to receive a value from p1 that is assigned to the (integer) variable n. The proposal in  [Ho78] also provides for pattern matching in communication. \parindex{guarded command} Also, a guarded command is offered, that allows to select a particular alternative dependent on the possibility of a communication. Due to its synchronous nature, communication in CSP is said to subsume synchronization mechanisms such as semaphores, events, conditional critical regions and monitors. C.f.  [AS83].

Channels

\parindex{channels} Occam is a language that embodies a number of the features of CSP.  [In84]. A noticeable difference with CSP is that communication statements do not address processes, but that instead communication takes place over channels. Occam is the machine language of the transputer. Transputers may be connected into a network. The language provides a mechanism for mapping symbolic channels to the actual hardware channels implementing the network. In contrast to CSP, Occam also provides facilities for mapping processes on processing units. \nop{ Despite the fact that Occam is a low level language, lacking data types, recursive procedures and modules, it is used in a number of applications.  [Jo85]. } \parindex{efficiency} \parindex{grain of parallelism} Perhaps the major advantage of such a language is that it is efficiently implementable, giving the programmer full control over the hardware resources. C.f.  [Ba89]. From the point of view of program design, however, the necessity of such control may be considered a disadvantage. From this perspective, languages with inherent parallelism seem more suitable. As alternatives to languages supporting the basic model we have: object oriented languages, that support concurrently executing active objects; functional languages, that allow parallel evaluation due to the absence of side-effects; and logic programming languages, that enable to work in parallel on parts of the and/or proof tree. \nop{ With the possible exception of the latter, these languages usually display a more fine grained parallelism. }

Object-based concurrency

\label{des/per:obj} \parindex{object based concurrency} Conceptually, objects are independent entities and the paradigm of method call by message passing allows in a natural way for concurrency. \nop{ The notion of an object based language covers a wide range of languages, including Ada, POOL, Emerald, Smalltalk and C++. An object may be characterized as an entity that has a collection of operations and a state that remembers the effect of operations.\ftn{ In accordance with the literature, we speak of object based languages and reserve the phrase object oriented language for the languages offering inheritance as an additional mechanism. See  [We87] for a detailed discussion of the dimensions of object based language design. } C.f.  [We87]. Apart from providing a construct to group operations, and a facility for defining an abstract interface to a collection of data, an object provides an additional protection mechanism since access and modification of the data it encapsulates is allowed only through the use of the operations defined for the object, the so-called methods. } However, even when considering method calls as (synchronous) message passing, object based languages may fit well in a sequential model of computation, assuming that an object is passive except when answering a method call. Extending the sequential object model to include parallelism may be achieved simply by allowing an object to be active on its own account, that is when not answering a message. As alternative ways to obtain parallelism, we mention the possibility to employ asynchronous communication as encountered in the Actor languages  [He77],  [Ag86]; or to add processes as an orthogonal concept to the language. A drawback of the last solution however is the need to provide extra facilities for synchronization and mutual exclusion. See also  [GR88]. Active objects seem in this respect to be a much more natural solution, since such protection is already offered by the method interface, assuming that only one method is answered at a time. C.f.  [Am89b].

Active objects

\parindex{active objects} The notion of active objects, that may be created dynamically, has been adopted by the language POOL. For a more extensive description of POOL see section \ref{impl/comp:lang}. Each object may have own activity, called the body of the object, that is started as soon as the object is created. The own activity of the object is interrupted to answer a method call when a so-called answer statement is encountered. The answer statement introduces a certain degree of non-determinism since, although a number of method calls may be considered acceptable, only one of these will be chosen. The communication model of method calls in POOL has been derived from the rendez-vous as encountered in Ada. See below. The rendez-vous, as an interaction between processes, has a two-way nature. It generalizes in this respect the primitives provided by for example CSP, that allow only one-directional point-to-point communication. In the terminology of POOL, the rendez-vous model is based on three concepts: a method declaration, which is like the declaration of a procedure having the right to access the private data of an object; a method call, which is like a procedure call but with an object as an additional parameter; and an answer statement, to interrupt the own activity of an object and to state the willingness to accept a particular method call.\ftn{ In the context of Ada one speaks of respectively an entry declaration, an entry call and an accept statement. } Answer statements allow to suspend the acceptance of a method call, dependent on the state of the object.\ftn{ Even stronger acceptance conditions may be imposed in Concurrent C that allows to inspect the actual parameters of a call to determine acceptance. }

The rendez-vous in Ada

\parindex{rendez-vous} We will explain the rendez-vous concept in somewhat more detail by looking at some simple Ada program fragments, taken from  [Perrott87]. \parindex{Ada -- task} In Ada a process is called a task. As an example of the specification of a task, consider the declaration of a (single-element) buffer. \yprog{task}{
   task buffer is
     deposit( c : in character );
     remove( c : out character );
   end buffer
  
} The declaration specifies that a buffer allows two operations, namely an operation to deposit a character and an operation to remove a character. An implementation of the buffer is given by the following definition \yprog{body}{
   task body buffer is
       ch : character;
   begin
      loop
         accept deposit( c : in character)  do   ch := c   end
         accept remove( c : out character)  do   c := ch   end
      end loop
   end buffer
  
} The body of a buffer specifies the own activity of a buffer, which is given by the succession of two accept statements, to accept subsequently a deposit call and a remove call, repeated indefinitely. To illustrate the use of the buffer, we assume the existence of a producer task in which the statement buffer.deposit(c); for a character c, occurs and a consumer task in which a statement buffer.remove(c); occurs. The first rendez-vous then takes place between the producer and the buffer when the buffer accepts the call for deposit. After that, the buffer has no other choice then to accept a remove call, which must come from the consumer task. It is important to note that the rendez-vous in Ada is of a synchronous nature. Only when the remote procedure call is completed may both tasks resume their activity. From the implementation of the body of the buffer task, we can infer that the buffer actually is a one-element buffer. However, the implementation may be changed (without affecting the task specification) by using for example an array of characters. In that case we may wish to use a more sophisticated protocol for accepting a call, a protocol that allows to take into account the number of elements the buffer contains. Ada offers a so-called select statement and a construct enabling the conditional acceptance of a call by which such a protocol can be implemented. The non-determinism allowed by these constructs is local, since it is solely dependent on the internal state of the task.\ftn{ In constrast, CSP supports global non-determinism by enabling a programmer to impose conditions with respect to the environment of the process, for instance to check whether a communication event may occur. }

Multiple threads

\parindex{multiple threads} As another object-based distributed language, we wish to mention Emerald. Just as POOL, Emerald offers the possibility to create active objects dynamically. An important difference between POOL and Emerald however is that Emerald allows multiple threads of control: one object can be active answering a number of method calls. Moreover, the processes created for answering a method call run in parallel with the process executing the own activity of the object. A monitor construct is provided to enable synchronization and protection when accessing local data shared by processes active for the object.

Allocation

\disindex{allocation} Object-based concurrency is most suitable for large-grain parallelism. Large-grain parallelism results in processes of considerable size. To the extent that these processes may run independently, speed-up can be obtained by allocating these processes to distinct processors. The language POOL enables the programmer to locate a newly created object on a particular processor by so-called pragmas, listing the set of processors from which the system may choose. In addition to a facility for mapping objects and processes to processors, Emerald supports the migration of objects and processes by allowing them to move, or to be moved, from one processor to another.

Concurrent logic programming

\label{des/rel:CP} \parindex{concurrent logic programming} \disindex{concurrent logic programming} The model underlying concurrent logic programming forms a rather radical departure from the two previous models in that communication is essentially effected through shared logical variables. Parallelism is inherent in the computation model of logic programming languages, because of their declarative nature. \parindex{{\em or} parallelism} \parindex{{\em and} parallelism} Basically, two kinds of parallelism can be distinguished: and-parallelism that is due to the parallel evaluation of the atoms in a compound goal, and or-parallelism that arises from trying multiple clauses simultaneously for finding a solution to a goal atom. Although there are a number of attempts at implementing parallel Prolog this way, the two major representatives of concurrent logic programming, Concurrent Prolog and Parlog, have based their approach on the additional assumption of committed choice non-determinism and restricted unification.

Committed choice

\parindex{committed choice} \parindex{guarded Horn clauses} Unlimited or-parallelism, required to find all solutions to a goal, may result in an uncontrollable amount of processes. To restrict or-parallelism, guarded Horn clauses were introduced. A guarded Horn clause is a clause of the form [] A:- G1,...,G_n | B1,...,B_m. where A is the head of the clause, G1,...,G_n the guard goals and B1,...,B_m the actual body of the clause. When a goal atom is evaluated, all clauses of which the head unifies with the atom are selected and the guards of these clauses are evaluated in parallel. The first clause of which the guard is evaluated successfully is committed to. The alternative solutions to the goal atom, embodied in the competing clauses, are thrown away. Since only one clause is chosen, backtracking over alternative solutions is impossible, once the commitment to that particular clause is made. What is allowed as a guard influences the expressiveness of the language in a significant degree, and for that matter the difficulty of implementing it. See  [Sh89] for an extensive discussion of this topic.

Restricted unification

\parindex{restricted unification} Unrestricted and-parallelism, that is the parallel evaluation of the atoms in a compound goal, may result in incompatible bindings of the logical variables involved. To handle this problem, both Concurrent Prolog and Parlog require to indicate which atom acts as the producer of a binding to a variable and which atoms are merely consuming the binding. Concurrent Prolog uses annotations to indicate the variables that must be bound to a term to enable the evaluation of the atom in which they occur to proceed. Parlog, on the other hand, uses mode declarations, indicating the input/output behavior of the arguments of a predicate.

Objects

\disindex{objects in concurrent logic programming} Concurrent logic programming languages offer a very versatile mechanism for implementing distributed systems. C.f.  [Sh89]. In particular these languages allow to implement active objects with state variables in a very elegant way. This is achieved by defining clauses for objects according to the scheme presented below. \oprog{object}{
   obj(State, [Message|Messages]) :-
  		\{\it handle\} Message,
  	        \{\it update\} State \{\it to \} State',
  		obj(State',Messages).
  
} An object is implemented as a tail-recursive process, that receives messages and updates its state if necessary. C.f.  [ST83]. As an example, consider the clauses implementing a counter in Concurrent Prolog. \oprog{ctr}{
   ctr(N,[inc()|T]):- N1 = N + 1, ctr(N1,T).
   ctr(N,[value(N)|T]) :- ctr(N,T).
  
} The first argument of ctr represents the state of the object, that is passed as an argument to the recursive call, appropriately modified if necessary. The second argument represents the stream of incoming messages, with the tail unspecified to await later binding. Concurrent logic programming languages offer fine-grained parallelism. As an additional feature for dynamically mapping computations to processes  [Sh84] proposes a turtle notation for executing Concurrent Prolog programs on a grid of processors. See also section \ref{des/ext/alloc}.

Extensions

The primary advantage of using a concurrent logic programming language for implementing distributed systems is the declarative nature of these languages, allowing a logical interpretation of a program. This property is preserved when implementing objects in the way shown. To overcome the syntactical complexity of this approach, two languages combining logic programming and object oriented programming have been proposed, Vulcan and Polka, that preserve the declarative semantics of their underlying logic programming languages.\ftn{ These languages will be discussed in section Vulcan. } The drawback of this approach, however, is that the restrictions imposed by the requirement of efficiency -- committed choice and restricted unification -- do not allow for the occurrence of backtracking.