mixed media

mixed media


A framework for mixed media

emotive dialogs, rich media and virtual environments

Intelligent Multimedia Group

Anton Eliens Claire Dormann Zhisheng Huang Cees Visser

Vrije Universiteit, Amsterdam

Some phenomena in the media are older than is generally recognized. As observed in  [History], some of the conventions of the 20th century comic books draw directly or indirectly on an even longer visual tradition: speech balloons can be found in the eighteenth century prints, which are in turn an adaptation of the 'text scrolls' coming from the mouths of the Virgin and other figures in medieval religious art.

In other words, mixed media have their origin in the medieval era, where both visual material and textual material were part of the rethorics of institutionalized religion.

structure

Desktop VR is an excellent medium for presenting information, for example in class, in particular when rich media or 3D content is involved.

At VU, we have been using presentational VR for quite some time, in a course on Web3D technology and also in our introduction multimedia course.

Recently we included dialogs using speech balloons (and possibly avatars) to display the text commenting on a particular presentation.

Our approach is clearly reminiscent to the notorious Agneta & Frida characters developed in the Persona project.

In contrast with the Personas project, our (dialog) comments are part of a presentation which of itself has a definite narrative structure, in opposition to the 'random' navigation that occurs by browsing 'information spaces'.

The characters and dialog text may be used to enliven the material.

In this way, the students' engagement with the material may be increased,  [Engagement].

Clearly, there is a tension between engagement and immersion.

In one of her talks, Kristina Höök observed that some users get really fed up with the comments delivered by Agneta and Frieda. Nevertheless, it also appeared that annoyance and irritation increased the emotional involvement with the task.

For our presentations, we may ask how mixed media may help in increasing the emotional involvement of the audience or, phrased differently, how dialogs may lead to emotional enhancement of the material,  [Enhancement].

An important difference with the Personas project is that our platform supports the actual merge of dialogs and the humanoid characters that deliver them in a unified presentation format, that is a rich media 3D graphics format based on X3D/VRML.

As a consequence, the tension between immersion and engagement may be partially resolved, since the characters delivering the dialog may be placed in their 'natural' context, that is a virtual environment

levels of complexity


level 0: basic material

level 1 -- combined:

level 2 -- with avatars:

level 3 -- with attributes

level 4 -- with context:

dialog



  <phrase right="how~are~you"/>
  <phrase left="fine~thank~you"/>
  <phrase right="what do~you think~of studying ..."/>
  ...
  <phrase left="So,~what~are you?"/>
  <phrase right="an ~agent" style="[a(e)=1]"/>
  <phrase left="I always~wanted to be~an agent" style="[a(e)=1]"/>

style parameters



  <phrase right="red" style="[p=(0.5,0,0),persona=0,balloon=0]"/>
  <phrase left="cutie" style="[p=(-0.5,0,0),persona=0,balloon=0]"/>
  <gesture right=1 style=default/>
  <gesture left=1 style=default/>

DLP+X3D


We have described a framework for mixed media that allows for the superposition of (text) dialogs delivered by humanoid avatars and/or speech balloons, on arbitrary rich media objects and virtual environments.

We have looked at the design space of mixed media presentations, by discussing a number of representative examples, each illustrating a particular level of complexity. Also authoring issues were discussed, and an indication was given of the style parameters needed to develop effective presentations.

We have further described the implementation platform used to realize the mixed media presentations and explored what new applications and extensions are feasible.

slides in XML



  <document>
  ...
  <slide id="1">
  <text>
  <line>What about the slide format?</line>
  <break/>
  <line>yeh, what about it"?</line>
  </text>
  <vrml>Sphere { radius 0.5 }</vrml>
  </slide>
  ...
  <slide id="2">
  <vrml>Sphere { radius 0.5 }</vrml>
  </slide>
  ...
  </document>

protos


observer



  :- object observer : [actions].
  var slide = anonymous, level = 0, projector = nil.
  
  observer(X) :- 
     projector := X,
     repeat,
       accept( id, level, update, touched),
     fail.
  
  id(V) :-  slide := V.
  level(V) :- level := V.
  touched(V) :- projector<-touched(V).
  update(V) :- act(V,slide,level).
  :- end_object observer.