virtual environments
learning objectives
After reading this chapter you should be able to
characterize the notion of virtual context,
discuss the issue of information retrieval in
virtual environments,
explain what is meant about intelligent multimedia
and discuss the potential role of intelligent
agents in multimedia applications.
From a user perspective, virtual environments offer the most advanced interface to multimedia information systems. Virtual environments involve the use of (high resolution) 3D graphics, intuitive interaction facilities and possibly support for multiple users.
In this chapter, we will explore the use of (desktop) virtual environments as an interface to (multimedia) information systems. We will discuss a number of prototype implementations illustrating, respectively, how paintings can be related to their context, how navigation may be seen as a suitable answer to a query, and how we can define intelligent agents that can interact with the information space. Take good notice, the use of virtual environments as an interface to information systems represents a major challenge for future research!
Dam Square, Amsterdam
- it is a 'real world' environment
- it has 700 years of (recorded) history
- it has a fair amount of historical buildings
- buildings and street life change over time
how can we give access to the 'Dam square' information space
virtual context
- VR model of Dam square
- selection of related paintings fromRoyalMuseum
- viewpoint adjustment, to match painting
- (transparent) overlay of paintings over buildings
augmented virtual reality
- give user sense of geographic placement of buildings
- show how multiple objects in a museum relate to eachother
- show what paintings convey about their subject, and how
problems
- organised guided tours
- account for buildings that no longer exist
- change temporal context
- allow user queries as input
VRML
- declarative means for defining geometry and appearance
- prototype abstraction mechanism
- powerful event model
- relatively strong programmatic capabilities
virtual archeology
- variety of archeological sites
- various paths through individual site
- reconstruction of 'lost' elements
- 'discovery' of new material
- glossary -- general background knowledge
RIF
Retrieval of Information in Virtual Worlds using Feature Detectors
On the left is the 2D map of the third floor of CWI, on the right the model generated from it.
the map
CWI
the model
the query
C O F F E E
navigation
explore
discover
what are we searching for?
what are we searching for?
types of information
- viewpoints
- areas of interest
- objects
- persons
- text
availability
- static -- always
- shared -- users
- dynamic -- runtime
- temporal -- events
- hidden -- actions
scanning the scenegraph
- annotations
- node types
- textual content
- materials
- textures
- geometry
choose a metaphor
get a viewpoint
- viewpoints
- areas of interest
- objects and persons
answer the query
- route planning
- viewpoint transformation
walking
assumptions
- explicit annotation
- map for route planning
- keyword matching
requirements
- database -- annotations & map
- 3D (pseudo-immersive) interface
the system
relaxing the assumptions
- annotation -- incremental and/or automatic
- (explicit) maps -- based on user navigation
- (keyword) matching -- text retrieval
...
- navigation by query is feasible and may help users to find locations and objects
- determining suitable navigation routes without an explicitly defined map is hard
future work
- shift in focus -- intelligent agents
- DLP + VRML -- distributed logic programming
Web Agent Support Program
no database, no walking
3D GUI
Wishful thinking about the widespread adoption of
three-dimensional
interfaces has not helped spawn winning applications.
Success stories with
three-dimensional games do not translate into broad
acceptance of
head-tracking immersive virtual reality. To accelerate
adoption of advanced
interfaces, designers must understand their appeal and
performance benefits
as well as honestly identify their deficits. We need to
separate out the
features that make 3D useful and understand how they help
overcome the
challenges of dis-orientation during navigation and
distraction from
occlusion.
Ben Shneiderman
Does spatial memory improve with 3D layouts? Is
it true that 3D
is more natural and easier to learn? Careful empirical
studies clarify why
modest aspects of 3D, such as shading for buttons and
overlapping of windows
are helpful, but 3D bar charts and directory structures are
not. 3D
sometimes pays off for medical imagery, chemical molecules,
and
architecture, but has yet to prove beneficial for performance
measures in
shopping or operating systems.
Ben Shneiderman
WASP
Web Agent Support Program
DLP
Distributed Logic Programming
RIF + WASP
- distributed logic programming -- uniform platform
- agent technology -- subsumes multi-user server
multi-user soccer game
- multiple (human) users -- may join during the game
- multiple agents -- to participate in the game (e.g. as goalkeeper)
- reactivity -- players (users and agents) have to react quickly
- cooperation/competition -- requires 'intelligent' communication
- dynamic behavior -- sufficiently complex 3D scenes,
including the dynamic behavior of the ball
control points
- get/set -- position, rotation, viewpoint
agents in virtual environments
- virtual environments with embedded autonomous agents
- virtual environments supported by ACL communication
Living Worlds
- scene -- geometrically bounded, continuously navigable
- world -- collection of (linked) scenes
Shared Object
- pilot -- instance that will be replicated
- drone -- instance that replicates pilot
- pilot agents -- control state of a shared object
- drone agents -- replicate the state of a shared object
- object agents -- controls a single shared object (like the soccerball) pilot at server, drone at client
- controls users' avatar pilot at user side, drone at server or clients
- autonomous agents -- like football player, with own avatar pilot at server, drone at clients
programming platform
- VRML EAI support
- distributed communication capabilities (TCP/IP)
- multiple threads of control -- for multiple shared objects
- declarative language -- for agent support
taxonomy of agents
- 2D/3D -- to distinguish between text-based and avatar embodied agents
- client/server -- to indicate where agents reside
- single/multi -- as a measure of complexity
PAMELA
Personal Assistent for Multimedia Electronic Archives
- autonomous and on-demand search capabilities
- (user and system) modifiablepreferences
- multimedia presentation facilities
H-Anim
- control points -- joints, limbs and facial features
presentation agent
Given any collection of results, PAMELA could
design some spatial layout and select suitableobject types,
including for example color-based relevance cues,
to present the results in a scene.
PAMELA could then navigate you through the scene,
indicating the possible relevance of particular results.
persuasion games
- single avatar persuasive argumentation
- multiple avatar dialog games
PAMELA
Persuasive Agent with Multimedia Enlightened Arguments
A variety of applications may benefit from deploying
embodied conversational agents, either in the form
of animated humanoid avatars or, more simply, as a 'talking head'.
An interesting example is provided by
Signing Avatar,
a system that allows for translating arbitrary text in both spoken
language and sign language for the deaf, presented by animated
humanoid avatars.
Here the use of animated avatars is essential to communicate
with a particular group of users, using the sign language
for the deaf.
STEP
- convenience -- for non-professional authors
- compositional semantics -- combining operations
- re-definability -- for high-level specification of actions
- parametrization -- for the adaptation of actions
- interaction -- with a (virtual) environment
DLP+X3D
The DLP+X3D platform provides together with the STEP
scripting language
the computational facilities for defining semantically meaningful
behaviors and allows for a rich presentational
environment,
in particular 3D virtual environments that may include
streaming video, text and speech.
evaluation criteria
conversational agents in VR
- presentational VR, instructional VR, educational VR
applications
- information agents, presentation agents
system perspective
- range of agent categories, open standards
user perspective
- naturalness -- contextual & emotional
initial target(s)
- build initial (throwaway) prototype
- explore content creation technology
- create tutorial(s) for content contribution
- analyse technological requirements
|
|
|
| (a) outside view | (b) inside view |
The history of Second Life is extensively descibed in the official Second Life guide, [SecondLife].
Beginning 2004, almost out of the blue,
Second Life
appeared with
a high adoption and low churn rate, now counting, March 2007,
over 4 million inhabitants.
Considering the cost of ownership of land, which easily amounts to
200 euro per month rent after an initial investment of 1500 euro for a
single piece of land measuring 65,536 square meters,
the adoption of Second Life by individuals as well as companies such as ABN-AMRO,
Philips and institutions such as Harvard is surprising.
What is the secret of the success of Second Life?, we asked in
[VUSL], and we immediately confessed:
We don't know!
But in comparison to other platforms for immersive worlds,
including MMORPGs such as
World of Warcraft
and
Everquest,
Second Life seems to offer an optimal combination of
avatar modification options,
gesture animations, in-game construction tools,
and facilities for communication and social networking, such as chatting and instant messaging.
Incorporating elements of community formation, commonly denoted
as Web 2.0, and exemplified in
MySpace,
YouTube and
Flickr,
the immersive appearance, perhaps also the built-in physics
and the inclusion of elementary economic principles,
seem to be the prime distinguishing factors
responsible for the success of Second Life.
In addition, the possibility of recording collaborative enacted stories, [Stories],
using built-in machinima certainly
also contributes
to its appeal.
What has been characterized as a shift of culture,
from a media consumer culture to a participatory culture, [Participatory],
where users also actively contribute content, (was) for our
institution one of the decisive reasons to create a presence
in Second Life, to build a virtual platform that may embody
our so-called community of learners,
where both staff and students cooperate in contributing content,
content related to our sciences, that is.
The first idea that comes to mind, naturally, is to
use Second Life to offer courses online.
But, although we did have plans to give lectures (college)
on law, probably including the enactment of a particular case,
we did consider this approach as rather naive, and frankly I see
no reason to include what may be considered an outdated
paradigm of learning in our virtual campus, where there
might be more appealing alternatives.
Similarly, using the virtual laboratory for experiments
might not be the best way to offer courses, although,
again, we do intend to provide a model of a living cell,
allowing students to study the structure, functionality and behavior
of organic cells in virtual space.
active learning
- experience the world in new ways
- form new affiliations
- preparate for future learning
concepts
technology
projects & further reading
As a project, I suggest the implementation
of storytelling in virtual environments,
with (possibly) an embodied agent as the narrator.
You may further explore or evaluate
the role of agents in multimedia applications
and virtual environments.
For further reading in (real) VR, I advice [VR],
and for gaining an understanding in story telling
and applications you may try to get hold of the proceedins,
of
TIDSE 2003, and
TIDSE 2004.
- another series of dutch light.
- virtual context -- Dam Square, Amsterdam, see 8.1.
- VU Campus in VRML -- student project.
- CWI 3th floor, floormap and model, see 8.2..
- query -- on 3th floor of CWI.
- navigation -- on 3th floor of CWI.
- soccer game -- image from WASP project,
see section 8.3.
- digital beauties -- taken from [Beauties].
- digital beauties -- taken from [Beauties].
- VU @ Second Life -- screenshots.
- signs -- sports, [Signs], p. 276, 277.
