topical media & game development
1
networked multimedia
For the end user there should not be much difference
between a stand-alone media presentation and
a networked media presentation.
But what goes on
behind the scenes
will be totally different.
In this section, we will study, or rather have a glance at,
the issues that play a role in realizing effective
multimedia presentations.
These issues concern the management of resources
by the underlying network infrastructure,
but may also concern authoring to the extent that
the choice of which media objects to present may affect
the demands on resources.
To begin, let's try to establish, following [Networked],
in what sense networked multimedia applications might
differ from other network applications:
networked multimedia
- real-time transmission of continuous media information (audio, video)
- substantial volumes of data (despite compression)
- distribution-oriented -- e.g. audio/video broadcast
Naturally, the extent to which network resource demands
are made depends heavily on the application at hand.
But as an example, you might think of the retransmission
of television news items on demand, as nowadays
provided via both cable and DSL.
For any network to satisfy such demands,
a number of criteria must be met,
that may be summarized as:
throughput, in terms of bitrates and burstiness;
transmission delay, including signal propagation time;
delay variation, also known as jitter; and
error rate, that is data alteration and loss.
network criteria
- throughput -- bitrates, burstiness
- transmission delay -- including signal propagation time
- delay variation -- jitter
- error rate -- data alteration, loss
For a detailed discussion of criteria,
consult [Networked], or any other book on
networks and distributed systems.
With respect to distribution-oriented multimedia,
that is audio and video broadcasts, two
additional criteria play a role, in particular:
multicasting and broadcasting capabilities and
document caching.
- multicasting and broadcasting capabilities
- document caching
Especially caching strategies are of utmost importance
if large volumes of data need to be (re)transmitted.
Now, how do we guarantee that our (networked)
multimedia presentations will come across with the
right quality, that is free of annoying jitter,
without loss or distortion,
without long periods of waiting.
For this, the somewhat magical notion
of Quality of Service has been invented.
Quoting [Networked]:
Quality of Service
Quality of Service is a concept
based on the statement that not all applications need
the same performance from the network over which they run.
Thus, applications may indicate their specific requirements
to the network, before they actually start transmitting
information data.
Quality of Service (QoS) is one of these notions
that gets delegated to the other parties, all the time.
For example, in the MPEG-4 standard proposal interfaces
are provided to determine QoS
parameters, but the actual realization of it
is left to the network providers.
According to [Networked] it is not entirely clear
how QoS requirements should be interpreted.
We have the following options:
we might consider them as hard requirements,
or alternatively as
guidance for optimizing internal resources,
or even more simply as criteria for the acceptance of a request.
QoS requirements
- hard requirements
- guidance for optimizing internal resources
- criteria for acceptance
At present, one thing is certain.
The current web does not offer Quality of Service.
And what is worse, presentation formats
(such as for example flash) do not cope well
with the variability of resources.
More specifically, you may get quite different results
when you switch to another display platform
2
virtual objects
Ideally, it should not make any difference to the author
at what display platform a presentation is viewed,
nor should the author have to worry about
low-quality or ill-functioning networks.
In practice, however, it seems not to be realistic
to hide all this variability from the author and
delegate it entirely to the 'lower layers'
as in the MPEG-4 proposal.
Both in the SMIL and RM3D standards, provisions
are made for the author to provide a range
of options from which one will be chosen,
dependent on for example availability,
platform characteristics, and network capabilities.
A formal characterization of such an approach
is given in [MMDBMS], by defining virtual objects.
virtual objects
-
where
- -- mutually exclusive conditions
- -- queries
- -- objects
In general, a virtual object is a media object
that consists of multiple objects,
that may be obtained by executing a query,
having mutually exclusive conditions to determine
which object will be selected.
Actually, the requirement that the conditions are
mutually exclusive is overly strict.
A more pragmatic approach would be to regard
the objects as an ordered sequence, from which
the first eligible one will be chosen,
that is provided that its associated conditions
are satisfied.
As an example, you may look at the
Universal Media proposal from the Web3D Consortium,
that allows for providing multiple URNs or URLs,
of which the first one that is available is chosen.
In this way, for instance, a texture may be loaded
from the local hard disk, or if it is not
available there from some site that replicates
the Universal Media textures.
3
networked virtual environments
It does seem to be an exageration to declare
networked virtual environments
to be the ultimate challenge for networked multimedia,
considering that such environments may contain all
types of (streaming) media,
including video and 3D graphics,
in addition to rich interaction facilities.
(if you have no idea what I am talking about,
just think of, for example, Quake or DOOM, and read on.)
To be somewhat more precise, we may list a number
of essential characteristics of networked
virtual environments,
taken from [VE]:
networked virtual environments
- shared sense of space -- room, building, terrain
- shared sense of presence -- avatar (body and motion)
- shared sense of time -- real-time interaction and behavior
In addition, networked virtual environments offer
- a way to communicate -- by gesture, voice or text
- a way to share ... -- interaction through objects
Dependent on the visual realism, resolution and
interaction modes such an environment may be more
or less 'immersive'.
In a truly immersive environment, for
example one with a haptic interface and force feedback,
interaction through objects may become even threathening.
In desktop VEs, sharing may be limited to
the shoot-em-up type of interaction,
that is in effect the exchange of bullets.
Networked virtual environments have a relatively long
history.
An early example is SIMNET (dating from 1984),
a distributed command and control simulation developed
for the US Department of Defense, [VE].
Although commercial multi-user virtual communities,
such as the blaxxun Community server,
may also be ranked under networked virtual
environments, the volume of data exchange
needed for maintaining an up-to-date state
is far less for those environments than for game-like
simulation environments from the military tradition.
Consider, as an example, a command and control strategy
game which contains
a variety of vehicles, each of which must send
out a so-called Protocol Data Unit (PDU),
to update the other participants as to their
actual location and speed.
When the delivery of PDUs is delayed
(due to for example geographic dispersion,
the number of participants, and the size of the PDU),
other strategies, such as dead reckoning,
must be used to perform collision detection
and determine possible hits.
To conclude,
let's establish what challenges networked virtual
environments offers with respect to software design
and network performance.
challenges
- network bandwidth -- limited resource
- heterogeneity -- multiple platforms
- distributed interaction -- network delays
- resource management -- real-time interaction and shared objects
- failure management -- stop, ..., degradation
- scalability -- wrt. number of participants
Now it would be too easy to delegate this all back
to the network provider.
Simply requiring more bandwidth would not solve
the scalability problem and even though adding bandwidth
might allow for adding another hundred of entities,
smart updates and caching is probably needed
to cope with large numbers of participants.
The distinguishing feature of networked virtual
environments, in this respect, is the need to
manage dynamic shared state
to allow for real-time interaction between
the participants.
Failing to do so would result in poor performance
which would cause immersion, if present at all,
to be lost immediately.
4
example(s) -- unreal
Unreal Tournament is a highly popular multiplayer game.
The storyline is simple, but effective:
It's the year 2362. The most anticipated Tournament ever is about to take place, dwarfing the spectacle and drama of previous events. The finest competitors ever assembled prepare to lay waste to their opponents and claim the Tournament Trophy for themselves.
There a a number of roles you can associate with:
the corrupt,
thunder crash,
iron guard,
juggernauts,
iron skull,
sun blade,
super nova,
black legion,
fire storm,
hellions,
bloof fist,
goliath
An interesting feature of the Unreal Tournament games
is that they can be andapted and even be
re-programmed
by the users themselves, has has been done for example for the
Mission Rehearsal Exercise discussed in section 9.2.
scripting:
www.gamedev.net/reference/list.asp?categoryid=76
5
research directions -- architectural patterns
Facing the task of developing a multimedia information system,
there are many options.
Currently, the web seems to be the dominant infrastructure
upon which to build a multimedia system.
Now, assuming that we chose the web as our vehicle,
how should we approach building such a system or,
in other words,
what architectural patterns can we deploy to build
an actual multimedia information system?
As you undoubtly know,
the web is a document system that makes a clear distinction
between servers that deliver documents
and clients that display documents.
See [OO], section 12.1.
At the server-side you are free to do almost anything, as long as
the document is delivered in the proper format.
At the client-side, we have a generic document viewer
that is suitable for HTML with images and sound.
Dependent on the actual browser, a number of other
formats may be allowed.
However, in general, extensions with additional formats
are realized by so-called plugins
that are loaded by the browser to enable a particular format,
such as shockwave, flash or VRML.
Nowadays, there is an overwhelming number of formats including,
apart from the formats mentioned, audio and video formats
as well as a number of XML-based formats as for example SMIL and SVG.
For each of these formats the user (client) has to
download a plugin.
An alternative to plugins (at the client-side) is provided
by Java applets.
For Java applets the user does not need to download any
code, since the Java platform takes care of downloading
the necessary classes.
However, since applets may be of arbitrary complexity,
downloading the classes needed by an application may take
prohibitively long.
The actual situation at the client-side may be even more complex.
In many cases a media format does not only require
a plugin, but also an applet.
The plugin and applet can communicate with eachother through
a mechanism (introduced by Netscape under the name LiveConnect)
which allows for exchanging messages using the built-in DOM
(Document Object Model) of the browser.
In addition, the plugin and applet may be controlled
through Javascript (or VBscript).
A little dazzling at first perhaps,
but usually not to difficult to deal with in practice.
Despite the fact that the web provides a general infrasructure
for both (multimedia) servers and clients,
it might be worthwhile to explore other options,
at the client-side as well as the server-side.
In the following, we will look briefly at:
- the Java Media Framework, and
- the DLP+X3D platform
as examples of, respectively,
a framework for creating dedicated multimedia applications
at the client-side
and a framework for developing intelligent multimedia systems,
with client-side (rich media 3D) components as well as
additional server-side (agent) components.
Java Media Framework
The Java platform offers rich means to create (distributed) systems.
Also included are powerful GUI libraries (in particular, Swing),
3D libraries (Java3D) and libraries that allow
the use and manipulation of images, audio and video
(the Java Media Framework).
Or, in the words of the SUN web site:
java Media Framework
The JavaTM Media APIs meet the increasing demand for multimedia in the enterprise by providing a unified, non-proprietary, platform-neutral solution. This set of APIs supports the integration of audio and video clips, animated presentations, 2D fonts, graphics, and images, as well as speech input/output and 3D models. By providing standard players and integrating these supporting technologies, the Java Media APIs enable developers to produce and distribute compelling, media-rich content.
However, although Java was once
introduced as the dial tone of the Internet
(see [OO], section 6.3),
due to security restrictions on applets it is not always possible
to deploy media-rich applets, without taking recourse
to the Java plugin to circumvent these restrictions.
DLP+X3D
In our DLP+X3D platform, that is introduced
in section [7-3] and described in more detail
in appendix [platform],
we adopted a different approach by assuming the availability
of a generic X3D/VRML plugin with
a Java-based External Authoring Interface (EAI).
In addition, we deploy a high-level ditributed logic programming
language (DLP) to control the content and behavior
of the plugin.
Moreover, DLP may also be used for creating dedicated (intelligent)
servers to allow for multi-user applications.
The DLP language is Java-based and is loaded using an applet.
(The DLP jar file is of medium size, about 800 K,
and does not require the download of any additional code.)
Dua, again, to the security restrictions on applets,
additional DLP servers must reside on the site
from where the applet was downloaded.
Our plugin, which is currently the blaxxun VRML plugin, allows for incorporating a fairly
large number of rich media formats,
including (real) audio and (real) video.,
thus allowing for an integrated presentation environment
where rich media can be displayed in 3D space in a unified
manner.
A disadvantage of such a unified presentation format, however,
is that additional authoring effort is required
to realize the integration of the various formats.
(C) Æliens
04/09/2009
You may not copy or print any of this material without explicit permission of the author or the publisher.
In case of other copyright issues, contact the author.
