Parallelism and distribution
\label{des/per/concept}
One of the reasons for employing parallelism in a program is
the need for efficiency.
Preferably, the parallelism remains implicit,
which means that the compiler takes care of how
the concurrent execution of the program will take
place.
However, realizing implicit parallelism is in practice quite difficult
and in most cases the programmer will have to indicate explicitly how
to execute the program concurrently.
Another, equally valid motivation for employing parallelism
is that it better fits the conceptual structure of the problem
to be solved.
As an example, to synchronize the behavior of objects it may be worthwhile
to allow objects to have activity of their own. C.f. [Pe89] and [Bro86].
\parindex{geographical distribution}
As an additional aspect, the processes that are involved in a concurrent computation may be
geographically distributed,
either because they need resources residing on distant processors or to
attain an increase in execution speed. See [CDJ84].
Programming languages that allow the computation to be spread over a number of distinct
processors connected by a network are called {\em distributed
programming languages}.
Computation models
\parindex{computation models}
\disindex{computation models}
When classifying distributed programming languages we can distinguish between
three distinct underlying computation models.
The most basic of these is that of communicating sequential processes,
first presented in the influential paper of [Ho78].
Object-based concurrent languages may be regarded as
extending this basic model, by
generalizing communication and
providing additional features for synchronization
and protection.
Finally, the model underlying concurrent logic programming languages
is perhaps the most flexible of these,
since it allows to mimick the two
previous ones.
The languages that we will refer to in this section
are listed in the table below.
.so tl
Distributed programming languages may differ in what
is employed as the unit of parallelism,
the way they deal with communication and how partial failures
are handled.
In [Ba89] an overview is given of a number of distributed programming languages,
discussing the choices made with regard to these dimensions.
Parallelism
\disindex{unit of parallelism}
There seems to be abundant choice in what to take as the unit
of parallelism, to name a few: processes (CSP), tasks (Ada),
active objects (POOL), multi-threaded objects (Emerald),
clauses (Concurrent Prolog and Parlog), or even statements (Occam).
With respect to the granularity of computation, we encounter
Concurrent Prolog and Parlog on the side of the spectrum of
languages supporting small-grain parallelism
and Ada or POOL on the other side,
supporting large-grain parallelism.
Large-grain parallelism means that, proportionally, the amount of
computation significantly exceeds the time spent communicating
with other processes.
Another important issue is whether a language supports the allocation of processes
to processors.
Allocation is supported for instance by POOL and Occam.
Communication
\disindex{communication}
Another decision that must be made concerns the way
communication is dealt with.
As alternatives we encounter data sharing and message passing.
We mention Linda as an interesting example
of data sharing.\ftn{
The apparent contradiction between distribution and data sharing is resolved by making a distinction
between physical data sharing and logical
data sharing. Obviously, we mean the latter here.
Logical data sharing provides the programmer with the illusion
of common data by hiding the physical distribution of the data.
}
Also Concurrent Prolog and Parlog,
utilizing shared logical variables as the medium of communication,
deserve to be classified among the distributed languages.
Choosing for message passing we may employ point to point
connections (CSP), channels (Occam, Delta Prolog) or broadcasting.
Communication may to a certain extent be non-deterministic.
For example, both the select statement of Ada
and the guarded Horn clauses of Concurrent Prolog and Parlog
result in a choice for a particular communication,
ignoring alternatives.
Failures
\disindex{failures}
\disindex{partial failure}
As an additional feature, some of the languages mentioned
in [Ba89] handle partial failure by offering
exceptions, atomic sections or recovery mechanisms.
Failure may be due to, for example,
hardware errors or the violation of integrity constraints.
We wish to remark that such failures are rather different
from the failure encountered in a language such as Prolog.
Failure in Prolog is one of the possible outcomes of a computation;
it may even be used to generate all the solutions to
a particular goal.
\nop{
Computation models
\parindex{computation models}
\disindex{computation models}
When classifying the languages mentioned we can distinguish between
three computation models
underlying distributed programming.
The most basic of these is that of communicating sequential processes,
first presented in the influential paper [Ho78].
Object based concurrent languages may be regarded as
extending this basic model, by
generalizing communication and
providing additional features for synchronization
and protection.
Finally, the model underlying concurrent logic programming languages
is perhaps the most flexible of these,
since it allows to mimick the two
previous ones.
}
Communicating sequential processes
\label{des/per:CSP}
\parindex{sequential processes}
The basic model of a distributed programming language is that
of a group of sequential processes that run in parallel
and communicate by message passing.
By a sequential process we mean a process with a single thread
of control. C.f. [We87].
Processes
\parindex{processes}
The prime example of a language supporting communicating
sequential processes is CSP. [Ho78].
In CSP we may create a fixed number of parallel processes
by using a parallel operator.
Each process consists of a name and a body, that is a sequence
of statements.
Communication between processes is achieved by using send
and receive statements.
As an example, consider the program
[]
where process p1 is about to execute the statement ,
sending the value 3 to process p2 and p2 is about to execute
, to receive a value from p1 that is assigned to
the (integer) variable n.
The proposal in [Ho78] also provides for pattern matching
in communication.
\parindex{guarded command}
Also, a guarded command is offered, that allows to select a particular alternative
dependent on the possibility of a communication.
Due to its synchronous nature, communication in CSP
is said to subsume synchronization mechanisms such as
semaphores, events, conditional critical regions
and monitors. C.f. [AS83].
Channels
\parindex{channels}
Occam is a language that embodies a number of the features
of CSP. [In84].
A noticeable difference with CSP is that communication statements
do not address processes, but that instead communication
takes place over channels.
Occam is the machine language of the transputer.
Transputers may be connected into a network.
The language provides a mechanism for mapping symbolic channels to
the actual hardware channels implementing the network.
In contrast to CSP, Occam also provides facilities for mapping
processes on processing units.
\nop{
Despite the fact that Occam is a low level language,
lacking data types, recursive procedures and modules,
it is used in a number of applications.
[Jo85].
}
\parindex{efficiency}
\parindex{grain of parallelism}
Perhaps the major advantage of such a language is
that it is efficiently implementable, giving
the programmer full control over the hardware resources. C.f. [Ba89].
From the point of view of program design, however,
the necessity of such control may be considered a disadvantage.
From this perspective, languages with inherent
parallelism seem more suitable.
As alternatives to languages supporting the basic model
we have:
object oriented languages, that support concurrently executing
active objects;
functional languages, that allow parallel evaluation
due to the absence of side-effects; and
logic programming languages, that enable to work in parallel
on parts of the and/or proof tree.
\nop{
With the possible exception of the latter, these languages usually
display a more fine grained parallelism.
}
Object-based concurrency
\label{des/per:obj}
\parindex{object based concurrency}
Conceptually, objects are independent entities
and the paradigm of method call by message passing
allows in a natural way for concurrency.
\nop{
The notion of an object based language covers a wide range
of languages, including Ada, POOL, Emerald, Smalltalk and C++.
An object may be characterized as an entity that has a collection of
operations and a state that remembers the effect of
operations.\ftn{
In accordance with the literature, we speak of object based languages
and reserve the phrase object oriented language for
the languages offering inheritance as an additional mechanism.
See [We87] for a detailed discussion of the
dimensions of object based language design.
}
C.f. [We87].
Apart from providing a construct to group operations,
and a facility for defining an abstract interface
to a collection of data, an object
provides an additional protection mechanism since
access and modification of the data
it encapsulates is allowed only through the use of the operations
defined for the object, the so-called methods.
}
However, even when considering method calls as (synchronous) message passing,
object based languages may fit well in a sequential model
of computation,
assuming that an object is passive except when answering a method call.
Extending the sequential object model to include parallelism
may be achieved simply by allowing an object to be active
on its own account, that is when not answering a message.
As alternative ways to obtain parallelism,
we mention the possibility to employ
asynchronous communication as
encountered in the Actor languages [He77], [Ag86]; or
to add processes as an orthogonal concept to
the language.
A drawback of the last solution however is the need to
provide extra facilities for synchronization and mutual
exclusion. See also [GR88].
Active objects seem in this respect to be a much
more natural solution,
since such protection is already offered by the method interface,
assuming that only one method is answered at a time.
C.f. [Am89b].
Active objects
\parindex{active objects}
The notion of active objects, that may be created dynamically,
has been adopted by the language POOL.
For a more extensive description of POOL
see section \ref{impl/comp:lang}.
Each object may have own activity, called the body of
the object, that is started as soon as the object is created.
The own activity of the object is interrupted
to answer a method call when a so-called answer statement
is encountered.
The answer statement introduces a certain degree of
non-determinism since, although a number of method calls may be
considered acceptable, only one of these will be chosen.
The communication model of method calls in POOL has been derived
from the rendez-vous as encountered in Ada. See below.
The rendez-vous, as an interaction between processes,
has a two-way nature.
It generalizes in this respect the primitives provided by for example CSP,
that allow only one-directional point-to-point communication.
In the terminology of POOL, the rendez-vous model is based
on three concepts:
a method declaration, which is like the declaration of a
procedure having the right to access the private data
of an object;
a method call, which is like a procedure call but with an object
as an additional parameter;
and an answer statement, to interrupt the
own activity of an object and to state the willingness
to accept a particular method call.\ftn{
In the context of Ada one speaks of respectively an
entry declaration, an entry call and an
accept statement.
}
Answer statements allow to suspend the acceptance of a method call,
dependent on the state of the object.\ftn{
Even stronger acceptance conditions may be imposed in Concurrent C
that allows to inspect the actual parameters of a call
to determine acceptance.
}
The rendez-vous in Ada
\parindex{rendez-vous}
We will explain the rendez-vous concept in somewhat more detail
by looking at some simple Ada program fragments, taken from [Perrott87].
\parindex{Ada -- task}
In Ada a process is called a task.
As an example of the specification of a task, consider
the declaration of a (single-element) buffer.
\yprog{task}{
task buffer is
deposit( c : in character );
remove( c : out character );
end buffer
}
The declaration specifies that a buffer allows two operations,
namely an operation to deposit a character and an operation to remove
a character.
An implementation of the buffer is given by the following definition
\yprog{body}{
task body buffer is
ch : character;
begin
loop
accept deposit( c : in character) do ch := c end
accept remove( c : out character) do c := ch end
end loop
end buffer
}
The body of a buffer specifies the own activity of a buffer,
which is given by the succession of two accept statements,
to accept subsequently a deposit call and a remove call,
repeated indefinitely.
To illustrate the use of the buffer, we assume the existence of a
producer task in which the statement
buffer.deposit(c);
for a character c, occurs and a consumer task in which
a statement
buffer.remove(c);
occurs.
The first rendez-vous then takes place between the producer
and the buffer when the buffer accepts the call for deposit.
After that, the buffer has no other choice then to accept a remove
call, which must come from the consumer task.
It is important to note that
the rendez-vous in Ada is of a synchronous nature.
Only when the remote procedure call is completed may both tasks
resume their activity.
From the implementation of the body of the
buffer task, we can infer that the buffer actually is a one-element
buffer.
However, the implementation may be changed (without affecting the task specification)
by using for example an array of characters.
In that case we may wish to use a more sophisticated protocol
for accepting a call, a protocol that allows to take into account the number
of elements the buffer contains.
Ada offers a so-called select statement and a construct enabling
the conditional acceptance of a call by which such a protocol can be implemented.
The non-determinism allowed by these constructs is local,
since it is solely dependent on the internal state of the task.\ftn{
In constrast, CSP supports global non-determinism
by enabling a programmer to impose conditions
with respect to the environment of the process,
for instance to check whether a communication event may occur.
}
Multiple threads
\parindex{multiple threads}
As another object-based distributed language,
we wish to mention Emerald.
Just as POOL,
Emerald offers the possibility to create active objects
dynamically.
An important difference between POOL and Emerald however is
that Emerald allows multiple threads of control:
one object can be active answering a number of method calls.
Moreover, the processes created for answering a method call
run in parallel with the process executing
the own activity of the object.
A monitor construct is provided to enable synchronization
and protection when accessing local data shared by processes
active for the object.
Allocation
\disindex{allocation}
Object-based concurrency is most suitable for large-grain parallelism.
Large-grain parallelism results in processes of considerable size.
To the extent that these processes may run independently,
speed-up can be obtained by allocating these processes
to distinct processors.
The language POOL enables the programmer
to locate a newly created object
on a particular
processor by so-called pragmas, listing the set of processors
from which the system may choose.
In addition to a facility for mapping objects and
processes to processors, Emerald supports the migration of objects
and processes by allowing them to move,
or to be moved, from one processor to another.
Concurrent logic programming
\label{des/rel:CP}
\parindex{concurrent logic programming}
\disindex{concurrent logic programming}
The model underlying concurrent logic programming
forms a rather radical departure from the two previous models
in that communication is essentially effected through
shared logical variables.
Parallelism is inherent in the computation model
of logic programming languages, because of their declarative nature.
\parindex{{\em or} parallelism}
\parindex{{\em and} parallelism}
Basically, two kinds of parallelism can be distinguished:
and-parallelism that is due to the parallel evaluation of the atoms
in a compound goal,
and or-parallelism that arises from trying multiple clauses simultaneously
for finding a solution to a goal atom.
Although there are a number of attempts at implementing
parallel Prolog this way,
the two major representatives of concurrent logic programming,
Concurrent Prolog and Parlog, have based their approach
on the additional assumption of committed choice non-determinism
and restricted unification.
Committed choice
\parindex{committed choice}
\parindex{guarded Horn clauses}
Unlimited or-parallelism, required to find all solutions to a goal,
may result in an uncontrollable amount of processes.
To restrict or-parallelism, guarded Horn clauses were introduced.
A guarded Horn clause is a clause of the form
[]
where A is the head of the clause, the guard
goals and the actual body of the clause.
When a goal atom is evaluated, all clauses of which
the head unifies with the atom are selected and the
guards of these clauses are evaluated in parallel.
The first clause of which the guard is evaluated successfully
is committed to.
The alternative solutions to the goal atom,
embodied in the competing clauses, are thrown away.
Since only one clause is chosen, backtracking over alternative solutions
is impossible,
once the commitment to that particular clause is made.
What is allowed as a guard
influences the expressiveness of the language in a significant degree,
and for that matter the difficulty of implementing it.
See [Sh89] for an extensive discussion of this topic.
Restricted unification
\parindex{restricted unification}
Unrestricted and-parallelism,
that is the parallel evaluation of the atoms in a compound goal,
may result in incompatible bindings of the logical variables
involved.
To handle this problem,
both Concurrent Prolog and Parlog require to indicate
which atom acts as the producer of a binding to a variable and
which atoms are merely consuming the binding.
Concurrent Prolog uses annotations to indicate the variables
that must be bound to a term to enable the evaluation of
the atom in which they occur to proceed.
Parlog, on the other hand, uses mode declarations,
indicating the input/output behavior of the arguments of
a predicate.
Objects
\disindex{objects in concurrent logic programming}
Concurrent logic programming languages offer
a very versatile mechanism for implementing distributed
systems. C.f. [Sh89].
In particular these languages allow to implement active objects with
state variables in a very elegant way.
This is achieved by defining clauses for objects
according to the scheme presented below.
\oprog{object}{
obj(State, [Message|Messages]) :-
\{\it handle\} Message,
\{\it update\} State \{\it to \} State',
obj(State',Messages).
}
An object is implemented as a tail-recursive process,
that receives messages and updates its state if necessary.
C.f. [ST83].
As an example, consider the clauses implementing
a counter in Concurrent Prolog.
\oprog{ctr}{
ctr(N,[inc()T]):- N1 = N + 1, ctr(N1,T).
ctr(N,[value(N)T]) :- ctr(N,T).
}
The first argument of ctr represents the state of the object,
that is passed as an argument to the recursive call,
appropriately modified if necessary.
The second argument represents the stream of incoming messages,
with the tail unspecified to await later binding.
Concurrent logic programming languages offer fine-grained parallelism.
As an additional feature for dynamically mapping
computations to processes [Sh84] proposes a turtle notation
for executing Concurrent Prolog programs on a grid of processors.
See also section \ref{des/ext/alloc}.
Extensions
The primary advantage of using a concurrent logic programming
language for implementing distributed systems is the declarative nature
of these languages,
allowing a logical interpretation of a program.
This property is preserved when implementing objects
in the way shown.
To overcome the syntactical complexity of this approach,
two languages combining logic programming
and object oriented programming have been
proposed, Vulcan and Polka, that preserve the declarative
semantics of their underlying logic
programming languages.\ftn{
These languages will be discussed in section [Vulcan].
}
The drawback of this approach, however, is that the restrictions
imposed by the requirement of efficiency -- committed choice and restricted unification --
do not allow for the occurrence of backtracking.