Unfortunately, the phrase representation
is also somewhat vague.
To be more precise,
we must distinguish between a visual information space
(for presentation),
a logical information space (in which we can reason
about abstract information objects) and
a physical information space
(where our concrete multimedia objects are stored).
Summarizing we have:
- physical information space -- images, animations, video, voice, ...
- logical information space -- abstract database objects
- presentational information space -- to present information to the user
Our visual information space, our presentation space,
as you may prefer to call it, might reflect the logical
information space in a symbolic manner by using diagrams,
icons, text and possibly visualzations, or,
going one step further, it may also mimic
the logical information space by using virtual
reality, as discussed in chapter 8.
Now we can give a more precise definition of the
notion of information space, in particular
logical information spaces:
a logical information space is a multidimensional
space where each point represents an object from the
physical information space (read database).
First of all, observe that when we speak of dimensions
we might also speak of attributes
that can take either continuous,
numerical, discrete or logical values.
So, concretely, these attributes may be directly or
indirectly related to information stored in the database,
and hence we can give a more precise definition of
the notion of (multimedia) information objects,
queries and cues (in the logical information space):
- information object -- a point in the (logical) information space
- query -- an arbitrary region in this information space
- clue -- a region with directional information, to facilitate browsing
The notion of clue is actually quite interesting,
since both examples and queries may be regarded
as clues, that facilitate browsing through the
contents of an information space.
As an example,
just think of the situation that, when looking for another notebook,
you want something
that is similar to the the thing you've previously
seen, but that has an additional video output slot
that may be connected to your TV.
Also, clues are needed to allow for query by example.
In this case you need to help the user to define
a query in the logical information space,
so that the system can construct an optimal query to search for
the desired object(s) in the physical information space.
When we regard the information retrieval problem
to be
the construction of the optimal query with respect
to the examples and clues presented by the user,
then we may characterize the optimal query as
the one that will retrieve the largest number of relevant database objects within the smallest possible region in the (logical) information space.
extensions
Given the stratification, that is levels or layers,
of information systems discussed above, we can think
of improvements or extensions on each level.
At the physical layer, for example
networked multimedia, in a client/server architecture, see 6.3.
At the logical layer, as an
information hyper space,consisting of chunks and hyperlinks,
as explained in section 2.2.
And at the presentation layer
a virtual reality interface, representing for example the physical location of student records, somewhere at a virtual campus [x],
as further explored in chapter 8.
Each of these improvements or extensions can be regarded as
a technological or scientific adventure in it's own right.
...
3
example(s) -- e-flux
Do you recognize this?
When we visit a contemporary art exhibition, we find ourselves before the works, which are often quite difficult to interpret, and we observe them without understanding the process that generated them. Between a chopped-up cow immersed in formaldehyde and a replica of the Pope blindsided by a meteorite, it's legitimate to ask questions.
To provide a counter-force
the exhibiton
Project Room challenges the usual exhibition routine and decides to not exhibit executed art works but rather offers ten self-interviewing videos by as many artists, who speak openly about a piece they are working on, or a visionary project they want to realize, or about their creative process.
In other words, this is about works of art with no physical
manifestion.
It is an interesting issue whether this would still
count as a work of art.
And for multimedia, is there multimedia
without a physical manifestation,
with sensorily impressing the user/client.
Do you remember the children story, the New Clothes
of the Emperor?
research directions -- universal interchange
Technology changes rapidly.
Just think about the development of the PC in the last two
decades of the previous century.
And applications change rapidly too.
At the time of writing the web does barely exist for ten years.
Information spaces, on the other hand, from a sufficiently
abstract perspective at least, should be rather stable
over time.
So the question is, how can we encode information
content in an application-independent way?
As a remark, application-independence implies technology-independence.
The answer is, simply, XML.
The next question then should be, what is XML and why is
it more suitable for encoding information then
any of the other formats, such as for example relational tables.
The first question is not so difficult.
There are many sources from where an answer may be obtained.
Perhaps too many.
A good place to start is the XML FAQ (Frequently Asked Questions)
at the Web Consortium site:
www.w3.org/XML/1999/XML-in-10-points
XML is a set of rules (you may also think of them as guidelines or conventions) for designing text formats that let you structure your data.
More specifically, XML may be characterized as follows:
XML in 10 points
- XML is for structuring data
- XML looks a bit like HTML
- XML is text, but isn't meant to be read
- XML is verbose by design
- XML is a family of technologies
- XML is new, but not that new
- XML leads HTML to XHTML
- XML is the basis for RDF and the Semantic Web
- XML is license-free, platform-independent and well-supported
Perhaps not all of these points make sense to you at this stage.
So let me first indicate that XML has in fact quite a long history.
XML is the successor of SGML (the Structured Generalized Markup Language)
that was developed in the 1980s to encode documents
(such as airplane manuals) in an application-independent manner.
SGML is not a language itself, but a descritpion of
how to create a content description language, using tags
and attributes (as in HTML).
In fact, HTML is an application of SGML, using tags with
attributes both for formatting and hyperlinks.
In other words, SGML is a meta language.
And so is XML.
Since everything got messy on the web, XML was proposed
(as a subset of SGML)
to make a clear distinction between content and presentation.
Presentation aspects should be taken care of by stylesheets
(see below)
whereas the content was to be described using and XML-based
language.
Now, why is XML a suitable format for encoding data?
That question is a bit harder to answer.
One of the reasons to use XML might be that it comes with
a powerful set of related technologies (including facilities
to write stylesheets):
related technologies
- Xlink -- hyperlinks
- XPointer -- anchors and fragments
- XSL -- advanced stylesheets
- XSLT -- transformation language
- DOM -- object model for application programmer interface
- schemas -- to specify the structure of XML documents
These technologies (that are, by the way, still in development)
provide the support needed by applications
to do something useful with the XML-encoded
information.
By itself, XML does not provide anything but a way
to encode data in a meaningful manner.
Meaning, however, comes by virtue of applications
that make use of the (well-structured) data.
In summary, XML and its related technologies provide the means
to
XML
- separate data from presentation
- transmit data between applications
Actually, the fact that XML was useful
also for arbitrary data interchange
became fully apperent when XML was available.
To get an impression of what XML is used
for nowadays, look at www.xml.org.
This leaves us with the question of why XML is to be preferred
over other candidate technolgies,
such as relational databases and SQL.
According to [Kay (2001)],
the answer to that question is simply that XML provides a richer data
structure to encode information.
In the multimedia domain we see that XML is widely adopted
as an encoding format, see section [3-2].
For an example you might want to have a look at
MusicXML,
an interchange format for notation, analysis, retrieval, and performance applications, that is able to deal with
common Western musical notation as used from the 17th century onwards.
In appendix [XML] we will explore
how XML might be useful for your own multimedia application
by treating some simple examples.
...
4
hypermedia
Given an information space we may turn it into
an information hyperspace, that is, following [Chang and Costabile (1997)],
information hyperspace
the logical information space may further be structured in a logical information hyperspace, where the clues become hyperlinks that provide directional information, and the information space can be navigated by the user following directional clues.
In other words,
information is chunked, and each chunk is illustrated
or made accessible by an example (hypernode).
Now, what exactly does information hyperspace mean?
To answer this question, let's briefly look at the history of
hypertext and hypermedia.
history
- 1945 -- Vannevar Bush (Memex) -- as we may think, [ Bush (1995)]
- 1963 -- Douglas Engelbart (Augment) -- boosting the human intellect [Engelbart (1963)]
- 1980 -- Ted Nelson (Xanadu) -- everything is intertwinkled, [Nelson (1980)]
Vannevar Bush' seminal paper As we may think
may be regarded as the origin of what is known as
hypertext with which, even if you don't know the phrase,
every one of you is familiar, since it is (albeit in a rather
simple way) realized in the web.
The phrase hypertext was invented by Ted Nelson
(not patented, as far as I know),
who looked for a less constraining way
to organize information then was common in
the educational system he grew up with.
But before that, Douglas Engelbarth, who
incidently invented the mouse, developed the Augment
system to, as he said, boost the human intellect.
What for, you may ask.
Let me quote the series of flashes
that Engelbarth went through, according to Dust or Magic [Klabbers (2006)]:
- flash 1: we are in trouble (human mankind)
- flash 2: we need to boost mankind's ability to deal with complex urgent problems
- flash 3: aha, graphic vision surges forth of me ...
- flash 4: hypermedia -- to augment the human intellect
- flash 5: augment (multimedia) workstation -- portal into an information space
classification of hypermedia
Perhaps it is good to know that Vannevar Bush
wrote his article when working for an information agency
in the second world war period.
From that perspective, we can easily see that hypermedia
(combining hypertext and multimedia)
were thought of an instruments of intelligence.
Basically, hypermedia systems must be able to deal with:
hypermedia systems
- components -- text, graphics, audio, video
- links -- relations between components
- presentation -- structured display
Far from being a definition, this characterization
gives some insight in what functionality hypermedia systems
must support.
Recall that dealing with complex information is what
hypermedia is all about.
Is this a natural way to deal with information?
Just think about how you are taught to deal with
information and how you actually go about with it.
Speaking about Ted Nelson, [Klabbers (2006)] observed that
he realized that this intertwingularity was
totally at odds with the education system he spent so long in
and had been so uncomfortable with.
Quoting Ted Nelson himself from his book Literary Machines:
A curriculum promotes a false simplification of any subject,
cutting the subject's many interconnections and leaving
a skeleton of sequence which is only a charicature
of its richness and intrinsic fascination.
Judge for yourself.
Would you prefer to have an 'immersive' course in
multimedia rather than a more or less ordered
collection of abstractions?
True enough, the visions of the pioneers of hypermedia
where overwhelming.
Nevertheless, the concept of hypermedia,
that is non-linear media with
machine-supported links, or 'text' as a network,
found an application in a large variety of systems, see [McKnight et al. (1991)].
classification of hypermedia systems
- macro-literary systems -- publishing, reading, criticism
- problem exploration tools -- authoring, outlining, programming
- browsing systems -- teaching, references, information
- general hypermedia technology -- authoring, browsing, collaboration
- embedded hypermedia -- CASE, decision support, catalogs
An example of a hypermedia system that has
extensively been used in education, for
example biology and chemistry classes,
is the Brown University Intermedia
system of which supports so-called information webs,
consisting of documents and links,
that could both be retrieved by specifying attribute,
allowing in this way for respectively both filtered content and
conditional navigation.
An interesting aspect of this system is that the user
may create maps, that is structures containing
documents and links, which form a personalized
version of the web of information for a specific user,
superimposed on the information space offered by the system.
...
5
Dexter Hypertext Reference Model
After many years of developing ideas and exploring
implementations, one group of experts in the
field came together and developed what is commonly
known as the Dexter Hypertext Reference Model,
named after the location, actually a pub, where the meetings were held.
The Dexter model offers an abstract description
of hypertext.
It made a distinction between components, anchors within components
and links between components, attached to anchors.
The model was meant as a reference standard against
which existing and future hypertext systems could
be compared.
Components have the following attributes:
component
- content -- text, graphics, video, program
- attributes -- semantic description
- anchors -- (bi-directional) links to other documents
- presentation -- display characteristics
The Dexter Hypertext Model has been criticised
from the beginning.
Among others, because compound documents,
that is documents having subcomponents,
where not adequately dealt with.
And also because it did not accomodate
multimedia (such as video) content very well.
In practice, however, the Dexter model has proven
to be even somewhat overambitious in some respects.
For example, the web does (currently) not support
bi-directional links in a straightforward manner.
Amsterdam Hypermedia Model
When looking for alternatives, a Dutch multimedia research
group at CWI proposed to extend the Dexter model
with their own multimedia model (CMIF), an extension for
which they coined the name Amsterdam Hypermedia Model.
Let's look at the (CMIF) multimedia model first:
(CMIF) multimedia model
- data block -- atomic component
- channel -- abstract output device
- synchronization arc -- specifying timing constraints
- event -- actual presentation
What strikes as an immediate difference with respect to
the hypertext model is the availability of channels,
that allow for presenting information simultaneously,
and so-called synchronization arcs,
that allow the author to specify timing constraints.
Also, events are introduced in the model to deal with
user interactions.
With respect to authoring, the model
supports a declarative approach to specifying sequential
and parallel compounds, that is in what order specific things
must be presented and what may ocuur simultaneously.
Again, channels may be employed to offer a choice in the presentation,
for example a dutch or english account of a trip in Amsterdam,
dependent on the preferences of the (human) viewer.
The Amsterdam Hypermedia Model (AHM) extends the Dexter Hypertext
Reference Model in a rather straigthforward way with channels
and synchronization arcs.
Amsterdam Hypermedia Model
- contents -- data block
- attributes -- semantic information
- anchors -- (id, value)
- presentation -- channel, duration, ...
Obviously, the difference between Dexter and AHM is primarily
the more precise definition of presentation characteristics,
by introducing channels as in the (CMIF) multimedia model.
Another (major) difference lies in the
characterization of compounds.
Each compound has one or more children, or subcomponents.
Subcomponents may act as the source or destination
of synchronization arcs.
Each component obtains a start-time, that may result
from parallel or sequential composition
and synchronisation arcs.
Another interesting concept introduced by
the Amsterdam Hypermedia Model is the notion of context.
What happens when you click on a link?
Does everything change or are only some parts affected?
Then, when you return, does your video fragment start anew
or does it take up where you left it?
Such and other issues are clarified in the
Amsterdam Hypermedia Model, of which we have omitted many details here.
It is perhaps interesting to know that the
Amsterdam Hypermedia Model has served as a reference for the
SMIL standard discussed in section 3.2.
If you want to know more about
the Amsterdam Hypermedia Model, you may consult
[Ossenbruggen (2001)] or [Hardman et al. (1994)].
...
6
example(s) -- hush
In the hush
we explore a variety of hypermedia applications.
In fact already in 1994 we developed a SGML-based
browser with applets in Tcl/Tk.
Somehow, we did a lot with music with optimistic titels such as
Bringing music to the Web, [Ossenbruggen & Eliens (1994)] and more pessimistic ones such
as Jamming (on) the Web, [Eliens et al. (1997)].
The acronym hush stands for hyper utility shell.
Many of the projects with hush were student projects,
in which we studied operational support for hypermedia applications.
Although we used SGML for markup, we did not have
any specific document model, as in [CMIF].
An overview and rationale of hush is given in [Eliens (2000)].
A significant part of the hush software is being
reused in the ViP system, that is discussed in section
4.3, albeit with an entirely different
presentation technology.
research directions -- computational models
Today, hypermedia functionality is to some extent embedded
in almost all applications.
However, to realize the full potential of hypermedia,
and in effect the networked multimedia computer,
there are still many (research) issues to be resolved.
To get an impression of the issues involved,
have a look at the famous seven hypermedia research
issues formulated by Halasz.
research issues
- search and query -- for better access
- composition -- for imposing structure
- virtual structures -- on top of existing structures
- computation -- for flexibility and interaction
- versioning -- to store modification histories
- collaborative work -- sharing objects with multiple users
- extensibility and tailorability -- to adapt to individual preferences
See [Ossenbruggen (2001)], section 2.3 for a more extensive description.
Although the research issues listed above were formulated
quite early in the history of hypermedia,
as a reflection on the requirements for second-generation
hypermedia systems, they remain valid even today.
Without going into any detail with respect to the individual
research issues, I rather wish to pose the grand encompassing
research issue for the networked multimedia computer:
What is the proper computational model underlying
hypermedia or, more generally, for applications
that exploit the networked multimedia computer
in its full potential?
Some directions that are relevant to this issue will be given in
section [3-3] which deals with
the multimedia semantic web.
multimedia authoring
It is tempting to identify a presentation with
the information space it presents.
This is what users often do, and perhaps should do.
When that happens, the presentation is effective.
But you must remember that the actual presentation is
just one of the many possible ways to engage a user
in exploring an information space.
Making the choice of what to present to the user
is what we understand by (multimedia) authoring.
Authoring is what we will discuss in this section.
Not by giving detailed guidelines on how to produce
a presentation (although you may look at the online assignment
for some hints in this respect), but rather by collecting
wisdom from a variety of sources.
...
7
visualization
Let's start with our explorations by looking at the
problem of visualisation with a quote
from David Gelernter, taken from [Shneiderman (1997)]:
visualization
Grasping the whole is a gigantic theme,
intellectual history's most important.
Ant vision is humanity's usual fate;
but seeing the whole is every thinking person's aspiration.
David Gelernter, Mirror Worlds 1992
Now, consider, there are many ways in which the underlying
information space may be structured,
or speaking as a computer scientist, what data types
may be used to represent the (abstract) information.
data types
- 1-D linear data -- text, source code, word index
- 2-D map data -- floor plan, office layout
- 3-D world -- molecules, schematics, ...
- temporal data -- 1 D (start, finish)
- multi-dimensional data -- n-dimensional (information) space
- tree data -- hierarchical
- network data -- graph structure
The visualisation problem then is to
find a suitable way to present these structures to
the user.
Basicall, following [Shneiderman (1997)], there are two paradigms
to present this information:
- interactive -- overview first, zoom and filter, then details on demand
- storytelling -- as a paradigm for information presentation
Storytelling may be very compelling, and
does not force the user to interact.
On the other hand, storytelling may lead to
information consumerism alike to television enslavement.
An interaction paradigm that combines 'storytelling' with
opportunities for interaction, as for example
in the blendo approach discussed in section 3.2,
would seem to be most favorable.
Interaction then may result in either changing the direction
of the story, or in the display of additional information
or even transactions with a third party
(for example to buy some goodies).
persuasive technology
Whatever your target audience, whatever your medium, whatever your message, you have to be convincing if not compelling.
In the tradition of rethorics,
which is the ancient craft of convincing others,
a new line of research has arisen under the name
of persuasive technology.
In the words of my collegue, Claire Dormann, persuasion is:
persuasion
- a communication process in which the communicator seeks to elicit a desired response from his receiver
- a conscious attempt by one individual to change the attitudes, beliefs or behaviours of another individual or group individual through the transmission of some messages.
In other words,
the purpose of persuasion is to accomplish one of the following goals: to induce
the audience to take some action, to educate the audience (persuade them to accept to accept information or data), or to provide the audience with an experience.
In the area of multimedia, one may think of many
applications.
Quoting Claire Dormann,
in interactive media, the field of application of
persuasive technology ranges from E-commerce,
social marketing (like an anti-AIDS campaign) to museum exhibits.
Also E-commerce provides an obvious example.
To convince people to buy more, more persuasive messages and technologies are developed through the use of humorous and emotional communication, agents (such as price finders) or 3D representations of products and shops.
For health campaigns (or any campaign of your choice) one can imagine
3D information spaces with agents presenting different point of views
and where users are given different roles to play.
In a museum you might want to highlight key points through innovative
and fun interactive exhibits.
Although the subject of persuasive technology
is far less technology-oriented than the name suggests,
multimedia (in a broad sense)
form an excellent platform to explore persuasion.
As concerns multimedia authoring, set yourself a goal,
do the assignment, explore your capabilities,
convey that message,
and make the best of it.
(re)mediation
What can you hope to achieve when working with the new media?
Think about it.
Are the new media really new?
Does anyone want to produce something that nobody has
ever seen or heard before?
Probably not.
But it takes some philosophy to get that sufficiently clear.
In [Bolter and Grusin (2000)], the new media are analyzed from the
perspective of remediation, that is the mutual influence
of media on eachother in a historical perspective.
In any medium, according to [Bolter and Grusin (2000)],
there are two forces at work:
(re)mediation
- immediacy -- a tendency towards transparent immersion, and
- hypermediacy -- the presence of referential context
Put in other words, immediacy occurs when the medium itself is
forgotten, so to speak, as is (ideally) the case in realistic painting,
dramatic movies, and (perhaps in its most extreme form)
in virtual reality.
Hypermediacy may be observed when either the medium
itself becomes the subject of our attention as in some
genres of modern painting, experimental literature
and film making, or when there is an explicit reference
to other related sources of information or areas of experience,
as in conceptual art, many web sites, and also in CNN news,
where apart from live reports of ongoing action,
running banners with a variety of information keep the
viewers up to date of other news facts.
Now, the notion of remediation comes into play
when we observe that every medium draws on the history
of other media, or even its own history, to achieve
a proper level of immediacy, or 'natural immersion'.
For example, Hollywoord movies are only realistic to the
extent that we understand the dramatic intent of cuts,
close-ups and storylines, as they have been developed
by the industry during the development of the medium.
As another example, the realism of virtual reality
can only be understood when we appreciate linear
perspective (which arose out of realistic Renaissance
painting) and dynamic scenes from a first
person perspective (for which we have been prepared
by action movies and TV).
Even if you may argue about the examples,
let it be clear that each (new) medium refers,
at least implicitly, to another medium,
or to itself in a previous historic phase.
So, what does this mean for new media, like TV or virtual reality?
Let's start with virtual reality.
[Bolter and Grusin (2000)] comment on a statement of Arthur C. Clarke
Virtual Reality won't merely replace TV. It will eat it alive.
by saying that
... he is right in the sense that virtual reality remediates
television (and film) by the strategy of incorporation.
This strategy does not mean that virtual reality can obliterate
the earlier visual point-of-view technologies,
rather it ensures that these technologies remain as least
as reference points by which the immediacy of virtual
reality is measured.
So, they observe "paradoxically, then, remediation is as important for
the logic of transparency as it is for hypermediacy".
Following [Bolter and Grusin (2000)], we can characterize the notions
of immediacy and hypermediacy somewhat more precisely.
immediacy
- epistemological: transparency, the absence of mediation
- psychological: the medium has disappeared, presence, immersion
hypermediacy
- epistemological: opacity, presence of the medium and mediation
- psychological: experience of the medium is an experience of the real
Now, sharpen your philosophical teeth at the following statement,
taken from [Bolter and Grusin (2000)], p. 224:
Convergence is the mutual remediation of at least
three important technologies -- telephone, televison
and computer -- each of which is a hybrid of technical,
social and economic practice,
and each of which offers its own path to immediacy.
The telephone offers the immediacy of voice or the interchange of voices in real-time.
Television is a point-of-view technology that promises immediacy through its insistent real-time monitoring of the world.
The computer's promise of immediacy comes through the combination of three-dimensional graphics, automatic (programmed) action, and an interactivity that television can not match.
As they come together, each of these is trying to absorb the others and promote its own version of immediacy.
Let us once more come back to virtual reality
and its possible relevance in our information age, [Bolter and Grusin (2000)], p. 225::
in the claim that new media should not be merely archival
but immersive,
the rhetoric of virtual reality finally enters in,
with its promise of the immediacy of experience
through transparency. .
So, with respect to the new media, we may indeed conclude:
what is in fact new is the particular way
in which each innovation rearranges and reconstitutes
the meaning of earlier elements.
What is new about media is therefore also old and familiar:
that they promise the new by remediating what has gone before.
The true novelty would be a new medium that did not refer to the other media
at all.
For our culture, such mediation without remediation seems
to be impossible.
...
8
example(s) -- mobius
Rurger van Dijk, a former student of mine,
has implemented an interactive story in flash.
The story is a romance,
told with images displaying scenes from
the life of the players, a young man and a young women.
The user can choose perspectives, either the
man's or woman's, and watch the story from that point of view.
The story is both non-linear and circular.
The scenes can be connected in various way, and order
is not compulsory.
research directions -- narrative structure
Where do we go from here?
What is the multimedia computer, if not a new medium?
To close this section on multimedia authoring,
let us reconsider in what way the networked
multimedia computer differs from other media,
by taking up the theme of convergence again.
The networked multimedia computer seems to remediate
all other media.
Or, in the words of [Murray (1997)]:
convergence
(p. 27) ... merging previously disparate technologies of communication and representation into a single medium.
The networked computer acts like a telephone in offering one-to-one real-time communication, like a television in broadcasting moving pictures, like an auditorium in bringing groups together for lectures and discussion, like a library in offering vast amounts of textual information for reference, like a museum in its ordered presentation of visual information, like a billboard, a radio, a gameboard and even like a manuscript in its revival of scrolling text.
In [Murray (1997)], an analysis is given of a great variety
of computer entertainment applications,
varying from shoot-em-up games to collaborative interactive role
playing.
[Murray (1997)] identifies four essential properties that
make these applications stand out against the entertainment offered
by other media, which include books and TV.
Two key properties determine the interactive nature
of computer entertainment applications:
interactive
- procedural -- 'programmed media' ...
- participatory -- offering agency
All applications examined in [Murray (1997)] may be regarded
as 'programmed media', for which interactivity is determined
by 'procedural rules'.
With agency is meant that the user can make active choices
and thus influence the course of affairs, or at least
determine the sequence in which the material is experienced.
...
9
Another common characteristic of the applications examined
is what [Murray (1997)] calls immersiveness.
Immersiveness is determined by two other key properties:
immersive
- spatial -- explorable in (state) space
- encyclopedic -- with (partial) information closure
All applications are based on some spatial metaphor.
Actually, many games operate in 'levels' that can be
accessedonly after demonstrating a certain degree
of mastery.
Networked computer applications allow for incorporating an almost
unlimited amount of information.
Some of the information might be open-ended, with storylines
that remain unfinished.
Closure, then, is achieved simply by exhaustive exploration
or diminishing attention.
multimedia authoring
Coming back to the question what the 'new medium',
that is the networked multimedia computer, has to offer
from the perspective of multimedia authoring,
two aspects come to the foreground:
multimedia authoring
- narrative format
- procedural authorship
The narrative format is incredibly rich,
offering all possibilities of the multimedia
computer, including 3D graphics, real-time sound,
text. In short, everything up to virtual reality.
But perhaps the most distinguishing feature
of the new medium is that true authorship
requires both artistic capabilities
as well as an awareness of the computational
power of the medium.
That is to say, authorship also means
to formulate generic computational rules for telling a story
while allowing for interactive interventions by the user.
Or, as phrased in [Murray (1997)],
the new cyberbard must create prototypical
stories and formulaic characters that, in some way,
lead their own life and tell their stories
following their innate (read: programmed) rules.
In section 8.3
and appendix C, we will present
a framework that may be used as a testbed
for developing programmed narrative structures
with embodied agents as the main characters.
development(s) -- mashup semantic(s)
The old media have a hard time to catch up with the new media.
While TV still may be considered a mass-medium, it seems to be loosing ground to
online games and, indeed, youtube.com.
In a panel of experts, gathered to discuss the notion of
crossmedia,
all agreed that the development(s) commonly referred to as web 2.0 are here to stay:
web 2.0
video sharing /
online gaming /
social networking
Not only do these application areas appeal to the user(s),
but moreover they seem to be fruitful from an entrepeneurial
perspective as well. In other words, there is money in it!
The spirit of the shift of culture that characterizes these developments is
well expressed in the following poem/rap from a local group, called daft punk:
daft punk -- technologic (cn / jp)
Buy it, use it, break it, fix it.
Trash it, change it, melt -- upgrade it.
Change it, point it, zoom it, press it.
Snap it, work it, quick -- erase it.
Write it, out it, paste it, save it.
Load it, check it, quick -- rewrite it.
Plug it, play it, burn it, rip it.
Drag and drop it, zip -- unzip it.
Look it, fill it, curl it, find it.
View it, coat it, jam -- unlock it.
Surf it, scroll it, pose it, click it.
Cross it, crack it, twitch -- update it.
Name it, rate it, tune it, print it.
Scan it, send it, fax -- rename it.
Touch it, bring it. Pay it, watch it.
Turn it, leave it, stop -- format it.
From a more objective perspective, we may observe that information has become a commodity,
that is easily re-used, or put together in different combinations, for different purposes.
In an extremely well-readible
article,
entitled: What Is Web 2.0 --
Design Patterns and Business Models for the Next Generation of Software,
Tim O'Reilly, ponders on the question(s), what makes these things work, and why are they profitable?
When we look at many of these new applications or mashups, for example those using google maps,
that these are:
mashup(s)
- substituting a single pragmatism for ideal design
- using light weight programming models
In other words, where the original visions of hypertext and hypermedia
suffered from megalomaniac ambitions such as boosting the human intellect,
many mashups simply provide a useful service or entertaining content.
And in the same vein, where software engineering principles dominated the
early hypermedia systems, the new mashups are often no more than a simple hack,
exploiting existing services in a clever way. With great effect!
O'Reilly also sketches the shift that characterizes the underlying economic model
of these development(s), that is the growth of the original web into the web 2.0,
and beyond:
web 2.0 design pattern(s)
- web 1.0 -- the web as platform
- web 2.0 -- architecture of participation
- web 3.0 -- data is the (intel) inside
The gist of these characterizations should be clear, service-oriented, and with
a clear eye to the data that makes service(s) worthwhile, and profitable.
In a study, investigating how to use web services to enhance Second Life, [Eliens et al. (2007c)],
we wrote:
by now the phrase web 2.0 as well as applications representing it, such as Flickr and YouTube,
are well established, and enjoyed by a wide community.
Each day new items are added to the growing list of mashups,
and the number of web services that constitute the building blocks of mashups also shows a steady growth.
Mashups seem to be the easy way to start up a company, since the technology is relatively easy and,
making use of appropriate services, initial investment costs can be low. Cf. [Amazon].
What web 2.0 stands for, from a technical perspective, is
succinctly expressed in
Dorai's:
Learnlog:
XML Is The Fabric Of Web 2.0 Applications
- the client side is AJAX (Asynchronous Javascript and XML)
- the server application typically exposes data through XML
- the interaction model is web services
- mashups combine multiple webservices to create new types of applications
And eventhough many alternative representations, such as
JSON
(Javascript Object Notation) are increasingly being used,
all in all XML may be regarded as the interlingua of the Web 2.0.
Before taking a closer look at the communication protocol(s) underlying web 2.0 and
de-construct the tight link of AJAX to HTML in-page formatting, it is worthwhile, following [Amazon],
to give an overview
of a selected number of services, that may be used to create mashups:
service(s)
Although mashups featuring google maps seem to be the dominant mashup type,
other services such as offered by del.ici.us, Flickr and BBC might prove to be more worthwhile
for 'serious' applications.
For example, for developing e-commerce applications
Amazon
offers services for
product operations, such as item search and similarity lookup,
remote shopping carts, to create and manage purchase collections,
customer content, to access information contributed by customers,
and third party listings, to find related resellers.
It is important to note that many of these services, as for example the shoppong cart services,
may be used independently of the commercial offerings of Amazon!
Most of the service providers and services mentioned above are accessible using a choice of
protocols, including WSDL, SOAP, XML-RPC and the REST protocol.
The REST protocol seems to be most widespread and as we will discuss in the next section, it seems
to be tho most appropriate protocol in Second Life.
REST stands for Representational State Transfer.
In essence, the REST protocol uses the url as a command-line for stateless RPC invocations,
which allows for services to be executed by typing in the
address box of a web browser.
A great tutorial about the REST protocol can be found in
Joe Gregorio's column
column: The Restful Web.
As fully explained in [Visser and Eliens (2000)], the phrases representation, state and transfer,
respectively, stand for:
REST
- representation -- encoding in a particular format
- state -- data encapsulated in an object
- transfer -- using HTTP methods
In practice, the use of REST means that the state associated with a resource or service must be
managed by the client.
Together with mechanisms such as content-negotiation and URL-rewriting, REST provides a simple,
yet powerful method to invoke services using HTTP requests.
A common misunderstanding is that AJAX is intimately tied to web browsers and in-page HTML formatting.
This misunderstanding is due to the fact that AJAX is often used to improve the
user experience of web pages bij emulating RIA (Rich Internet Applications)
using DHTML and CSS.
However, the real meaning of AJAX in our view is that AJAX allows for
asynchronous client-controlled server requests, that are executed without an immediate visible
effect for the user.
The web 2.0 offers a lively arena for consumers and developers alike,
with a multitude of blogs discussing the future of the web.
For example,
in Dion Hinchcliffe rebuttal of
Jeffrey Zeldman's Web 3.0 – Web 1.0 = Web 2.0 blog, entitled
Is Web 2.0 Entering "The Trough of Disillusionment"?
it is suggested that
our services could even be more powerful by creating
semantic mashups.
Although the notion of sematic web technology is widely known and accepted,
we include for reference
a characterization of Nova Spivack
quoted from Dan Farber and Larry Dignan's
blog Web 2.0 isn’t dead, but Web 3.0 is bubbling up:
The Semantic Web is a set of technologies which are designed to enable a particular vision
for the future of the Web – a future in which all knowledge exists on the Web in a format
that software applications can understand and reason about. By making knowledge more accessible
to software, software will essentially become able to understand knowledge, think about knowledge,
and create new knowledge. In other words, software will be able to be
more intelligent, not as intelligent as humans perhaps, but more intelligent than say,
your word processor is today.
But even in the semantic web community the discussion whether to go for folksonomies
or formal ontologies rages, [Folk],
and it is not clear at this stage what will prove to be more powerful, HTML-scraping,
tags, microformats, or full ontologies.
Instead of joining this perhaps endless discussion, let us explore what is involved
in incorporating web services in Second Life, and how to realize meaningful mashups
in 3D virtual environments.
Nevertheless, to conclude this brief overview of web services and mashups I wish to give
another quote from Dorai's Learnlog, this time from Jon Udell, in his blog on his move to Microsoft:
the most powerful mashups don't just mix code and data, they mix cultures.
which provides a challenge that trancends all issues of mere technological correctness.
using web services in Second Life
Second Life offers an advanced scripting language with a C-like syntax
and an extensive library of built-in functionality.
Although is has support for objects, LSL (the Linden Scripting Language) is not
object-oriented. Cf. [Eliens (2000)].
Scripts in Second Life are server-based, that is all scripts are executed at the server,
to allow sharing between visitors.
Characteristic for LSL are the notions of state and eventhandler, which
react to events in the environments.
Among the built-in functions there are functions to connect to a (web) server, and obtain
a response, in particular (with reference to their wiki page):
built-in(s)
Other functions to connect to the world include sensors,
for example to detect the presence of (visitors') avatars, and chat and instant
messaging functions to communicate with other avatars using scripts.
In addition, LSL offers functions to control the behavior and appearance of objects,
including functions to make objects react to physical laws,
to apply force to objects, to activate objects attached to an avatar
(as for example the phantom Mario sprites mentioned earlier),
and functions to animate textures, that can be used to present slide shows in Second Life.
On the Mashable
Social Networking News site a brief
overview is given of the use of web services in Second Life, entitled
Second Life + Web 2.0 = Virtual World Mashups.
To access Second Life from outside-in (that is from a web browser), so-called slurls may be used,
for example to reach VU @ Second Life,
and all slurls listed in
del.icio.us under
slurlmarker may be used,
also to activate in-world teleporting using scraping techniques.
As remarked in the
hackdiary
by Matt Biddulph, Second Life (currently)
lacks the ability to parse XML or JSON,
so the best way to incorporate web services is to set up a web server with adequate
resources.
As Matt Biddulph indicates, to access flickr photographs
for a particular user (avatar), a web server may contain the following resources:
resource(s)
- /seen?user=SomeAvatar -- records the presence of SomeAvatar
- /touched?user=SomeAvatar -- invokes flickr API with users tag
- /set_tag?user=SomeAvatar&tag=FavoriteTag -- records SomeAvatar's favourite tag
For example,
in response to a 'touch' event, invoking touch results in consulting the database for the user's tag
and asking the Flickr API for a random photo with that tag.
It then returns a string containing the url for a particular photograph.
LSL functions used in this application include sensors, to check for presence,
listen functions, to respond to spoken commands,
and touch events, for the physical interface.
In addition to supporting
strings and lists, LSL provides a perl-like split function to convert a string into a list
of strings, thus allowing for processing multiple items in response to a server request.
Another example of using web services in Second Life is writing
blogs
from within Second Life using the
BlogHUD
developed by Koz Farina who also is reported to have found a flash hack
that allows for
reading RSS feeds.
As explained by Koz Farina:
flash/quicktime in SL
Quicktime supports Flash, but only up to Flash version 5. We're up to version 9 on that now!
Luckily, I have been dabbling with Flash since the early days,
so already knew how to do this 'the old way'... So,
Flash is doing all the work. No LSL at all... I heart feeds.
Did I say 'I heart feeds?
The RSS display uses the ability to stream Quicktime video in Second Life, and again the
mashup is not created in Second Life but by appropriate server support.
In a similar vein we may incorporate live
streaming video,
for example by using
WireCast
to capture and organize live camera input, possibly together the screen output of
other applications such as powerpoint, which must then be sent to a
streaming server supporting Quicktime, such as Apple's
Darwin,
which may then be accessed from Second Life to texture a display object.
Finally, as
another Web 2.0 to Web 3D phenomenon, announced in
New World Notes,
we may mention the used of
Twitter messages, that allow residents to send and receive message about
ongoing activities.
A similar service is reported to exist for jaiku messages.
Referring to section 7.4 for a more detailed discussion, we may observe
that there is no meaning in merely putting things together.
Without mechanisms of personalization and recommendation we would simply be flooded
by data and information, in a way that even search would not be able to cope with.
Context, narratives and personalized presentation(s), notions from the past,
reappear as keywords for the future of the web 2.0 and beyond.
...
10
questions
concepts
technology
projects & further reading
As a project, I suggest the development of a virtual tour
in a city, museum or other interesting locatoion.
You may further explore
the implementation of traversal within a context,
taking into account the history of navigation
when backtracking to a particular point,
issues in
hyperlinking and interaction in multimedia applications, and
computational support for narratives.
For further reading I advice you to take a look
at the history of hypermedia and the web,
using online material from the
W3C, or the history of
media as accounted for in [Briggs and Burke (2001)] and [Bolter and Grusin (2000)].
- book covers -- [Weishar (1998)], [Eco (1994)], [Burger (1981)], [Kunst], [Betsky (2004)]
- Federico Campanale -- Oxygen, fragments from video installation, 2004
- Vasarely -- [Diehl 1973].
- Vasarely -- [Diehl 1973].
- Vasarely -- [Diehl 1973].
- Federico Campanale -- Oxygen, more fragments.
- student work -- from introduction multimedia 2000.
- Rutger van Dijk -- mobius, interactive story, opening screen,
see section 2.3.
- edgecodes -- screenshots, see section 2.3
- signs -- people, [ van Rooijen (2003)], p. 244, 245.
The work of Vasarely has served as an example
for many contemporary digital artists.
It is playful, mat may be characterized
also as formalist.
The highly aesthetic video work of Federico Campanale
who, as he told me was strongly influenced by vasarely
in his early years, shows a similar combination
of formalism and playfulness.
The interactive story by Rutger van Dijk has a rather different
atmosphere, it is highly romantic, with slick graphics.
The musea sites are included to point to the existence
of (an increasing number) of virtual tours.
(C) Æliens
23/08/2009
You may not copy or print any of this material without explicit permission of the author or the publisher.
In case of other copyright issues, contact the author.
