learning objectives
After reading this chapter you should be able to
mention some basic rules of digital content creation,
discuss what criteria your portfolio should meet,
describe how you would approach the design of a logo,
explain the notion of user-centered design,
and characterize the issues that play a role
in dveloping multimedia for theatre.
Whether your ambition is to become a professional designer or not,
also for students of information science
and computer science, a course in visual design is a must, I think.
In this chapter, we will treat various aspects of digital content
creation.
The first section discusses how to approach visual design
and gives a number of basic design assignments,
that can be used to get experience with visual design.
Section 2 discusses the issue of workflow and tools,
and investigates how design fits in with the process of
developing multimedia applications.
In the final section, I will elaborate on a theatre
project I was involved in, for which I had to develop an augmented
reality application.
...
1
visual design
When you are trained as a visual artist, as I once was (in the pred-digital era),
you must do many basic exercises with form and color,
learn the skills of drawing and painting, including sill lifes and portraits.
And, to graduate you must make an exhibition of your work over the years.
In the digital era, things have changed.
The skills that you need to learn, which include the use of tools for
digital content creation, as well as the artistic goals, I would say.
What has not changed, however, is the need for basic exercises and the presentation of your
work, that is the creation of a portfolio.
In this section, I will give an outline of the course visual design,
that I started in februari 2005.
This outline is meant to give you a general idea of how to approach design,
and to give you some hints on how to acquire the skills needed to act as a designer.
If you have the opportunity to take a course in visual design, then don't hesitate
and do it!
...
2
perspective(s)
The overall goal of the visual design course is to establish
some basic aesthetic awareness,
by providing suitable exercises and assignments.
In addition, the student is supposed to become familiar with the craft of design,
which necessarily, but not exclusively, involves the use of tools and techniques.
To accomodate for the various interests and backgrounds of the students,
we distinguish between several tracks or perspectives,
as summarized below:
track(s) -- perspective
- styling -- concept and presentation
- digital content -- material, animation
- tech track -- special effects
To illustrate the various perspectives, I invited guest speakers who showed their work
and talked about their approach.
A selection of their works is hown in this book.
The process of design is very complex, ranging from
conceptual explorations and
sketching up to the stage of finalizing delivery.
It is also very personal.
However, as each of the speakers testified to,
a significant portion of the time and effort goes into negotiating
with the client.
Whether it involves taking photographs for an advertisement
or setting up a campaign, it takes a lot of
back and forth to get an idea of what the client wants.
In the agency of one of the speakers Mark Veldhuijzen van Zanten, they have created
roles to help eachother come up with the right ideas.
www.178aardigeontwerpers.nl
- e-motionist -- make emotion, rhythm and movement flow together
- chaoticus -- who sees chaos within order
- formologist -- who approaches the fabrication of forms as an art
- infonaut -- who moves in the twilight zone of information and meaning
- transformator -- who transforms images and concepts into new matter
Although slightly ironic, and in practice not so clear-cut,
these roles give you an idea what cognitive modes are involved
in bringing an original concept to a stage of finalization.
Coming up with an idea and sketching require a more reflective
cognitive mode, whereas finalizing a design requires a more
experience-related cognitive mode.
It is important to choose the right tools to work with,
dependent on which phase you're in, for example paper and pencil when you
are still sketching and Photoshop or maya when you put your ideas
into production.
...
3
deliverable(s)
However interesting the process may be,
design is not about process but about product(s).
Such a product may be, dependent on what you ar good at,
one of the following:
products
- web site -- e.g. conference, campaign (browse)
- 2D/3D animation -- promotion/ad (temporal sequence)
- virtual space -- game/infotainment (navigate)
- ebook -- story (sequential experience)
As indicated between brackets, each of the products
favors a particular mode of interaction.
Although interaction is not an aspect of visual design as such,
it is an important aspect to take into account.
In section 9.2, we will look in more detail at the issue of interaction
and usability in general.
One easily overlooked issue in a design project,
is the creation of a portfolio.
There may easily be some confusion here with regard to what should be
considered the product of design, one of the items in the list above,
or your portfolio.
The answer is simple. Both!
As a record of the process of design, the portfolio is itself a product of design.
portfolio -- design as a product
- concept(s)
- sketches & explorations
- finalized products
- evaluation & reflection
What criteria should a portfolio meet?
Well, nowadays it is not only common to have your portfolio on the web,
I have been told that you cannot do without it.
So, first of all, your portfolio should be web-friendly.
And your work should not be too many clicks away!
And, secondly, it should give sufficient insight in what you have to offer,
so that a potential client can decide whether it is worth the effort to contact you.
In our visual design course, I require that the portfolio contains
a description of the concept(s) underlying the design,
sketches and explorations as well as the finalized products.
It is also required, after all it is an academic course,
to provide an evaluation and some reflection in the form of an essay
on a topic such as 2D versus 3D aesthetics, animation techniques,
or (more theoretical), elements of style or theories of creativity.
www.jaapstahlie.com
In my perception a portfolio is about the
past and I feel much more related to the
present especially in my work as a
photographer. To me the relation with
the present and the subject/assignment directs
my creativity, the experiences over the
past draw my skills. My challenge is to be
truly inspirred, to be present in the present.
However, as illustrated by the motivation Jaap Stahlie gives
with his portfolio (above), it is perhaps wise not to overdo it!
There are basic exercises, obligatory for all students,
and a final assignment, where you have a choice
between three productions, each with a different supervisor.
In addition, as explained in the guidelines, all students must write an essay,
and give a presentation in class.
For deadlines, see the schedule.
There will be periodic checks on the status
of your work.
Each year there will be recommended themes.
assignment(s)
In the visual design course there are basic exercises, obligatory for all students,
and a final assignment, where the student has a choice
between three productions.
basic exercises
- develop a logo
- create a sign
- design a collage
- write a story
For the final assignment, there is a choice between the following assignments:
developing a house style,
creating a non-linear visual story, and
designing a suitable game environment
For this assignment) the students are allowed to work in groups.
However, the contribution of each individual must be reflected
in his/her own portfolio.
...
4
regulation(s)
The first requirement when working in an area such as visual design
is that you acquire sufficient self-discipline to find
the challenge in the assignments and to complete the tasks involved.
Since a department of computer science is not the natural habitat
for a course in visual design, I have laid down some strict rules:
rules
- be present -- 2 omissions max.
- be in time -- hard deadlines
- be online -- have your portfolio available
- be creative -- don't steal without a reason/mentioning
- be smart -- there is no 2nd chance
These rules may well apply when you work,
after you graduated, as an individual designer/developer
or as a member of a team in some agency.
To finish this section, I may remark that
design is an interesting field, full of implicit
(not always so obvious) wisdom
and apparent paradoxes.
Whatever you do, deliver! Silence is lethal.
And, as another item of colloquial wisdom, be authentic, but only if possible!
...
5
research directions -- on creativity
One of the assignments in the visual design course
is an essay.
As one of the recommended topics we have a reflection on
theories of creativity.
As such it is not a training in creativity.
This section contains some random thoughts on the processes and products of
design, and ends with the provoking statement
there is no theory of creativity.
You may, however, try to find some counter-arguments,
for example in the line of [Hewlett and Selfridge-Field (1998)].
multimediocrity
Multimedia is a promising technology, and (nowadays) affordable.
So we see that multimedia
(which includes 3D-graphics, video and sound)
is increasingly being used, also in information visualisation.
But what is it good for?
To quote [Klabbers (2006)]:
multimedia's promise is terribly generalized,
it simply lets you do anything.
As with any new technology, the early multimedia
productions (in particular CDROM and CD-I)
were not optimal with respect to (aesthetic) quality.
To quote [Klabbers (2006)], again:
shovelware -- multimediocrity
... far from making a killing, it looked as if the big boys ...
had killed the industry by glutting the market with
inferior products.
Perhaps the industry in the late eighties did not
have the right business model.
But, then again, what are the chances of multimedia
in our time.
One more quote from [Klabbers (2006)]:
if multimedia is comparable to print then yes,
we'd be crazy to expect it to mature in a mere ten years.
eliminating complexity
So now, in the new millenium, we are (sadder and wiser)
in a position to approach the effective deployment
of multimedia afresh.
What we look for is aesthetic quality.
How do we find it?
Easy enough, just be authentic.
"Learning how to not fool ourselves is, I'm sorry to say, something that we haven't specifically included in any particular course that I know of. We just hope you've caught it by osmosis."
Richard Feynman
Authentic in creating multimedia means, apart from
not fooling yourselves, that you must become aware of the
message orinformation you want to convey
and learn to master the technology to
a sufficient degree.
But beware, an effective multimedia presentation
is not the same as scientific argumentation:
the media equation
We regularly exploit the media equation for enjoyment by the willing suspension of our critical faculties. Theatre is the projection of a story through the window of a stage, and typically the audience gets immersed in the story as if it was real.
These quotes, as well as the following one have been taken from
an online essay on eliminating complexity
which provides an argument against inessential gadgets
and spurious complexity and bells and whistles
in whatever you can think of,
including user interfaces and scientific theories.
Back to the subject,
what does master the technology to a sufficient degree
mean?
Just remember that what you do is a form of engineering.
engineering
"engineering is the art of moulding materials we do not wholly understand ... in such a way that the community at large has no reason to suspect the extent of our ignorance."
A. R. Dykes.
In other words, learn the tool(s) that you are using
to a degree that you master the basics and easily
cut through its apparent magic.
theories of creativity
Producing multimedia, in whatever form, has an element
of craftsmanship.
But, given the need for aesthetic quality,
whatever way you approach it, there is an element of creativity.
That means, you're in for a challenge.
And, to quote [Klabbers (2006)],
The best thing is to empower yourself.
But before you can do that, you need to understand
what you are doing -- which is a surprisingly novel thing to do.
Now it is tempting to look for a set of guidelines and rules
that give you a key to creativity.
So let me be straight with you:
there is no theory of creativity
On the other hand, there are techniques for producing ideas.
And some recommend a sequence of steps, such as:
steps
browse, explore;
chew it over;
incubation, let it rest;
illumination (YES);
verification,does it work?
And in addition, still following [Klabbers (2006)],
there are some general rules:
general rules
- if you aim to please everybody, you will please nobody
- constraints come with the territory, you must learn to love them
- emotional charge is the key to success
Now, if you'd ask me, I would say, just
make your virtual hands dirty.
...
6
designing the user experience
In a time in which so much information is available
as in ours, we may make statements like
postmodern design
... postmodern design is of a highly reflective
nature ... appropriating design of the past ...
in other words, sampling is allowed but no plagianarism
One interpretation might be that it is impossible to be original.
But another interpretation might be that it is undesirable to be
(totally) original.
In other words, as discussed in section 2.3,
it is necessary that your design contains references to not only
some real situation but also to other work, so that it can be understood
and experienced by the user/spectator/client whatever you like to call
the people that look at your work.
As observed in [Sherman and Craig (2003)], designing for multimedia does not take into
account only the technological or aesthetic issues, but also
constraints on what people can perceive and what the experiential context
is in which the work is presented, which may be re-phrased more plainly as what
expectations the user has.
game design
Let us consider how these observations affect one of the
project assignments for our visual design course.
Also for game design,
there are several options, dependent on the track the student
is most comfortable with.
game design
- style -- develop concept, plot and visual assets for a game of choice
- content -- develop environments, models and animations for a game of choice
- effects -- develop models, textures and special effects (shaders) for a game of choice
To explain, style may be considered to involve the whole
gamut of concepts, plot and genre, as well
as the visual assets or props of the game, those things
by which the game differentiates itself from other games.
Since the reader of this book will
probably be more familiar with games than the author, there is no
need to expand on these issues.
Content is concerned with the actual game environment,
including the models and animations.
Finally, effects, to simplify things, is everything else,
those things that are visual but does not belong to the story line
or game environment.
Games, perhaps more than any other multimedia application,
are appealing, not because they are useful, although they might be,
but because the user gets emotionally involved, not to say addicted.
Now, following [Norman (2004)],
did you ever wonder why cheap wine tastes better in fancy glasses?
Exactly, because cheap glasses do not give us the same emotion.
It is, indeed, a matter of style!
Obviously, games are played for fun.
As applications, games may be classified as seductive,
which is, see section 2.3, stronger than persuasive.
[Norman (2004)] distinguishes between four categories of pleasure.
seduction
- physio-pleasure -- of the body
- socio-pleasure -- by interaction with others
- psycho-pleasure -- due to use of the product
- ideo-pleasure -- reflecting on the experience
In other words, games are seductive, or fun to play,
because they arouse any combination of pleasure
from the categories above.
Which combination depends on the kind or genre of game.
Quoted from [Norman (2004)], but originally from [Wolf]
we can list, not exhaustively, the following genres of video game:
genre(s)
Abstract, Adaptation, Adventure, Artificial Life, Board Games, Capturing, Card Games, Catching,
Chase, Collecting, Combat, Demo, Diagnostic, Dodging, Driving, Educational, Escape,
Fighting, Flying, Gambling, Interactive Movie, Management Simulation,
Maze, Obstacle Course, Pencil-and-Paper Games, Pinball,
Platform, Programming Games, Puzzle, Quiz, Racing, Role-Playing,
Rhythm and Dance, Shoot Em Up, Simulation, Sports, Strategy,
Table-Top Games, Target, Text Adventure, Training Simulation, and Utility.
When you develop a game it is good to reflect on what genre your game
belongs to, because that will directly affect the user's expectation
when playing the game, due to the conventions and rules that exist
within a particular genre.
For video games, which can be
characterized as a mixture of interactive fiction with entertainment,
interaction is evidently another distinguishing factor in determining
the success of the game.
Corresponding to the kind of pleasure a user may experience,
[Norman (2004)] distuinguishes between three levels of design:
levels of design
- visceral -- what appeals to our intuition (affordance)
- behavioral -- is all about use (performance)
- reflective -- its all about message, culture and meaning
Of these, the latter two should be rather obvious, although we
will elaborate on the notion of usability later on.
But what does affordance mean, and how is it related to
our intuition.
...
7
affordance -- ecology of behavior
The notion
of affordance
has a long history.
According to Don Norman,
the word "affordance" was invented by the perceptual psychologist J. J. Gibson,
to refer to the properties of the world that 'invite' actions, as for example
a char invites one to sit.
Originally, however, the notion of affordance dates back to the beginning of the
20th century, when it was used in phenomenologist philosophy to describe how
the world around us presents itself as meaningful.
Affordance, in other words, is a concept that explains why it seems natural for
us to behave in a particular way, either because it is innate, as the reflex
to close one's eyes by sudden exposure to light, or because we have learned that
behavior, as for example surfing the web by clicking on links.
In game or product design, thinking about 'affordance' may help
us to find the most natural way to perform certain actions.
Natural is in this context perhaps not the best phrase.
What we must take into account is what is perceived as an affordance,
and how actions fit in with what we may
call an exology of behavior (with the system).
How does this diversion in abstract philosophy help us design better games?
To answer this question, I'd like to recount my visit
at the Virtual Humans Workshop,
held in october 2004 at the Institute of Creative Technologies,
in Los Angeles.
Of interest in particular is the ICT Games Project:
ICT Games Project
The goal of the ICT games project is to develop immersive, interactive,
real time training simulations to help the Army create a new generation of
decision-making and leadership-development tools.
As further explained on the website:
with the cooperation of the U.S. Army Research, Development and Engineering Command,
Simulation Technology Center (RDECOM STC),
Training and Doctrine Command (TRADOC), and commercial game development companies,
ICT is creating two training simulations that are intended to have the same holding power
and repeat value as mainstream entertainment software.
The two training applications developed by ICT are:
- Mission Rehearsal Exercise -- to solve a potential conflict after a car accident
- Language Training Simulation -- to learn how to contact local leaders in arabic
The mission rehearsal exercise is situated in former Yugoslavia. The trainee
is confronted with the situation after a car accident in which a boy got injured.
The mother of the boy is furious, and a potentially hostile crowd is waiting.
An army convoy is on its way to a nearby airport, and needs to pass the crossing
where the accident took place. The trainee must decide what to do and give appropriate
orders, knowing that the wrong decision may lead to serious trouble.
The language training simulation is situated in the Middle-East,
and is meant to teach the trainee not only some basic arabic but also proper ways of conduct,
in conformance with local customs to gain confidence.
Both applications are highly realistic, with impressive graphics.
A lot of attention has been devoted to creating the models and environments.
Both simulations are implemented using the Unreal game engine.
The both support speech input.
The challenge in both simulation games was to come up with a natural
way to indicate to the trainee what options for actions were available.
Natural means, in this context, that it should fit within the simulation or game
environment.
Obviously, a menu or a row of pushbuttons does not fit naturally within such an
environment, and would break what we have previously, in section 2.3
called 'immersion'.
I was invited at ICT for the Virtual Humans Workshop
because of my involvement with embodied conversational agents (ECAs),
as discussed in section 8.3.
The topic of the workshop was, among others,
to investigate whether the notion of affordance could help
in analyzing and evaluating the interaction of a user/trainee
with the simulation game.
These were the questions we tackled:
Virtual Humans Workshop
-
Is it more appropriate to construct a frame of analysis that encompasses both user and ECA
in a single interaction graph?
-
Is it fitting to think in terms of a fixed graph that the user comes to recognize,
or is the graph itself a dynamic structure?
-
Is it even appropriate to focus on "affordances to act," or is it more fitting to consider cues
that influence the mental interpretations that lead to
action (e.g., affordances of control, affordances of valence of potential outcomes, etc.)?
How does this relate to intrinsic motivation?
This workshop was a follow-up on a seminar in Dagstuhl
on Evaluating Embodied Conversational Agents,
where we discussed the tipics of interaction and affordance in a special interest group.
In the research directions, I will further discuss an evaluation study that
we did on agent-supported navigation in a virtual environment.
Back to our question, how can affordance help us in designing a game?
In the mission rehearsal exercise, described above,
it would be much more easy to have a menu with all available options listed.
However, such amenu would defeat the purpose of the simulation,
since such menus will not likely occur in real life.
Immersion is, in other words, necessary to maintain the emotional involvement
with the application, and affordance is the key to immersion.
But, although it sounds like an answer, it does rather lead to another question,
how can we define the usability of a game?
...
8
usability and fun
In interaction design there is a clear, yet unresolved, tension between
usability and fun.
Usability is, despite the many disputes, a well-defined notion:
usability (ISO DIS 9241-11)
... the effectiveness, efficiency and satisfaction with which
specified users can achieve particular goals in particular environments ...
This is the ISO DIS 9241-11 definition, cited from [Faulkner (2000)].
In sectiona 10.3 we will further investigate usability
as a means to evaluate systems from an interaction perspective.
Now, I wish to focus on why artefacts or games might be appealing
even if these same aspects may compromise usability in the traditional interpretation.
In describing a fancy juice squeezer, designed by Philip Starck [Norman (2004)] observes,
following [KS], that is:
emotional involvement
- entices by diverting attention -- unlike the common
- delivers surprising novelty -- not identifiable to its function
- goes beyond obvious needs and expectations -- it becomes something else
- creates an instinctive response -- curiosity and confusion
The phrase satisfaction in the definition of usability above
seems somewhat meagre to explain the emotional involvement with games,
and even inappropriate as one realizes that, in the mission rehearsal exercise,
frustration might actually be beneficial for the learning experience.
...
9
example(s) -- visual sensations
The dutch
visual sensations festival is
an annual contest for VJs.
In 2005, in cooperation with the festival, a parallel seminar
seminar was held discussing the topic of the history of VJ-ing,
a aplenary discussion of the relation between club-VJs
and the established art circuit.
In addition there were two guest speakers,
Geert Mul and Micha Klein,
both visual artists
who also have a ten-years experience as VJ.
...
10
Above is another work of Geert Mul, in cooperation with
DJ Speedy J.
It was shown a dance event in cooperation with
Rotterdam Maritime Museum.
On the right, the cranes are swinging on the rhythm
of the music.
The portfolio of Geert Mul
starts with a quote from
[Arnheim (1957)]:
form and content
Very often people assume that "form" as a concept is the opposite of something called "content". This assumption implies that a poem or a musical piece or a film is like a jug. An external shape, the jug, contains something that could just as easily be held in a cup or pail. Under this assumption, form becomes less important than whatever it is presumed to contain.
We do not accept this assumption. If form is the total system, which the viewer attributes to the film, there is no inside or outside. Every component functions as part of the overall pattern that is perceived. Thus we shall treat as formal elements many things that some people consider content. From our standpoint, subject matter and abstract ideas all enter into the total system of the artwork ( .... )
I totally agree with this.
And perhaps this is why I have a preference
for artworks that are slightly out of the main stream of
tradional art.
research directions -- engaging with fictional characters
What do you need to evaluate your game or multimedia application?
There are many ways to gain insight in how your system is being used,
see section 10.3.
But if you want to establish functional properties of a multimedia application,
for example the effectiveness of using an agent in navigating a virtual environment,
in a scientifically more rigorous way, you need to have:
experimental validation
- a theory -- in our case: PEFiC
- a test scenario -- for example, memory tasks in a digital dossier
- the technology -- to realize applications
In this section, I will briefly describe our efforts in experimentally
validating the use of ECAs in virtual environments.
As technology, we use our intelligent multimedia technolgy,
described in sectiona 8.3 and appendix E.
So what must be explained is the theory we adopt and the test scenarios we use.
PEFiC is a theory developed by Johan Hoorn and Elly Konijn,
to explain Perceiving and Experiencing Fictional Characters, see [PEFIC].
The PEFiC theory
may serve as the basis for the experimental evaluation of user responses to embodied agents.
In summary, PEFiC distinguishes between three phases, encoding, comparison and response,
in analyzing the user's behaviour towards an agent.
Encoding involves positioning the agent (or fictitious character)
on the dimensions of ethics (good vs bad),
aesthetics (beauty vs ugliness) and epistemics (realistic vs unrealistic).
Comparison entails establishing personal relevance and valence towards the agent.
Response, finally, determines the tendency to approach or avoid the character,
in other words involvement versus distance.
In general, having a virtual environment, there is, for developing test
scenarios, a choice between:
validation scenario(s)
- navigation -- pure interactivity
- guided tours -- using some narrative structure
- agent-mediated -- navigation and guided tours
For our application, a virtual environment
of an artist's atelier,
we have three experimental conditions,
navigation without an agent, navigation with a realistic agent
and navigation with a cartoon-like (unrealistic) agent.
To ensure that these conditions can be compared,
the acrtual information encountered when using the application
is in all conditions the same.
The independent variable in our experiment, the degree of realism of the agent,
corresponds with the epistemic and to some extent the aesthetic
dimension of appraisal in the PEFiC theory.
As dependent variables we have, among others,
user satisfaction, believability, that is estimated usefulness of the agent,
and also the extent to which the relevant information is retained.
...
11
The application is a digital dossier for the Dutch artist
Marinus Boezem.
The spatial metaphor we used for the dossier is the artist's atelier.
We created a virtual environment containing a display of the artworks, in 3D,
a file cabinet with textual information,
a workbench for inspecting the artist's material,
and a video projector, with which the user can display a video-recorded
interview with the artist.
The actual task to be performed by the user is to learn
what constraints do apply to the installation of one
of the artworks, Stone and Feather:
Stone and Feather
- feather: 70 cm, from ostrich, curved
- stone: 13.5 cm, white marble
- position: alignment with pedestal, no glue
- environment: 50 lux of light max.
The items mentioned in this list must be reproduced by
the user in a subsequent memory test,
and in another experiment the user must be able
to choose the right materials and reconstruct the artwork.
Our assumption in designing this test scenario
was that the gestural nature of positioning the artwork
will be favorable for the condition with a gesturing agent,
whereas believability will be positively affected
by the degree of realism of the agent.
...
12
multimedia augmented theatre
In june 2003, I was asked to advise on the use
of VR in a theatre production of the Odyssee.
Lacking experience in this field,
I accepted the invitation to participate with some reluctance,
since at the time I didn't have any clue what the VR
for the theatre production should look like.
Nevertheless, I took the invitation as a challenge
and started looking for appropriate hardware,
bothering collegues for information on mixed reality
art productions, and downloading code to explore
software technologies.
Many hurdles ware to be taken.
We had to deal with organizational issues,
such as finding the money for financing the actual production
(which unfortunately never came through),
finding the right people (students, in our case) to select material and
contribute to the code; aesthetic issues, in particular
to determine which approach to take to reach at an effective solution;
and not in the least technical issues, to realize the
production on a sufficiently efficient low-cost platform.
...
13
context -- the Odyssee theatre production
The Odyssee.
theatre production was initiated by Ground Control,
as a successor of previously succesful theatrical spectacles,
including an open air performance of Faust.
In effect, two performances of the Odyssee were planned,
an out-door (external) version, involving real ships
at the shore of a lake, and an in-door (internal) version,
to be played in temporarily empty office buildings.
The in-door version is meant to give
a more psychological rendering of the Odyssee [Entanaclaz (2003)],
where the travels of Ulysses are experienced by the audience
as a confrontation with themselves.
Our contribution was asked for the in-door version,
to enhance the experience of the audience with
additional VR.
The Odyssee is a wellknown account of the travels of Ulysses
leaving Troje, in 24 episodes ending in his return to Ithaca and
his reunion with Penelope.
The actual theatre production takes 12 parts which are played
in 12 successive rooms through which the audience,
subdivided in small groups,
is guided one room after another for about five minutes per room.
Our initial idea was to add information in the form of text and
images, to direct the interpretation of the audience
towards a particular perspective.
In that beginning stage, somewhat optimistically, we
planned to offer multiple perspectives to each
participant, in an individualized manner,
dependent on the actual focus of attention of the individual participant.
initial ideas -- VR and augmented reality:
Our first problem was to find suitable hardware,
that is see-through goggles.
Searching the Internet gave us the name of a relatively nearby
company, Cyber Mind NL,
that specialized in entertainment VR solutions.
Both price-wise and in terms of functionality semi-transparent
see-through glasses appeared to be no option,
so instead we chose for simple LCD-projection goggles
with a (head-mounted) low-resolution camera.
This solution also meant that we did not need
expensive head orientation tracking equipment,
since we could, in principle, determine focus using
captured image analysis solutions such as provided by the AR Toolkit.
Moreover, captured video feed ensured the
continuity and reactiveness needed for
a true (first-person perspective) VR experience.
Augmented or mixed reality
is an interesting area of research
with many potential applications.
However, in the course of the project we dropped our ambition
to develop personalized presentations using
image analysis, since we felt that the technology
for doing this in a mixed reality theatre setting is simply not ripe,
and instead we concentrated on using the captured
video feed as the driver for text and image presentation.
In addition, we developed image manipulation techniques
to transform the (projection of the) captured video,
to obtain more implicit effects, as to avoid the explicit
semantic overload resulting from the exclusive use of text
and images.
technological constraints -- the DirectX platform:
After a few experiments with the AR Toolkit, it soon appeared
that the frame rate would not be sufficient, on the type
of machines our budget would allow for.
Moreover, reading the AR Toolkit mailing list,
marker tracking in a theatrical context
seemed to be more or less unfeasible.
So, we shifted focus to the
DirectX SDK 9,
both for video capture and projection in 3D.
The DirectX toolkit is a surprisingly functional, and very rich
technology for multimedia applications,
supporting streamed video, including live capture,
3D object rendering and precise synchronisation
between multimedia content-related events.
At that time, and still at the time of writing,
our own intelligent multimedia technology
was no option, since it does not allow for
using live video capture and is also lacking in down-to-the-millisecond
synchronisation.
After exploring texture mapping images copied
from the incoming captured video stream,
we decided to use the
VMR-9 video mixing renderer
introduced in DirectX 9,
that allows for allocating 3D objects as its rendering surface,
thus avoiding the overhead of
explicit copies taken from a video processing stream
running in a separate thread. See section 4.3.
Although flexible and efficient,
DirectX is a low-level toolkit,
which means that we had to create our own
facilities for processing a scenegraph,
world and viewpoint transformations, and, even more importantly,
structuring our mixed reality presentations in time.
...
14
structuring time -- maintaining 'see-through' aesthetics
One of the problems we encountered in discussing what
we conveniently may call the VR with the producer of
the Odyssee theatre performance was the high expectancy
people have of VR, no doubt inspired by movies
as the Matrix and the like.
In mixed reality applications, manipulating persons,
warps in space, and basically any intensive image
analysis or image manipulation is simply not possible
in real time.
Moreover, there is a disturbing tendency
with the layman to strive for semantic overload
by overlaying the scene with multiple images and lines of text,
thus obscuring the reality captured by the camera
and literally blocking the participants view
and awareness of the scene.
Basically, as a guideline, we tend to strive for
70% visibility of the scene,
20% image or projection transformations
and only 10% of information in the form of text and images.
The total duration of our presentation is only 2 minutes,
or 118 seconds to be precise.
We made a subdivision in 4 scenes, with transitions
inbetween, hierarchically ordered in a tree-like structure.
Initially, we abstracted from the actual duration,
by taking only the fraction of the time passed
(in relation to the total duration) as an indication
for which scene to display.
However, when the development reached its final
stages, we introduced actual durations that allowed
us to time the sequence of scenes to the tenth
of a second.
In addition, we used multiple layers of presentation,
roughly subdivided in background captured image,
the transformed captured image projected on 3D objects,
and, finally, pictures and text.
These layers are rendered on top of eachother,
triggered in a time-based fashion, semi-independent
of one another.
The frame rate varies between 20 and 30, dependent
on the number of images simultaneously used
for texturing.
Our final mixed reality theatre application may be considered
a prototype,
awaiting to be put to the test by the audience.
...
15
lessons learned -- our explorations revisited:
Altogether, the development of the mixed reality
theatre application has been quite an experience,
in multiple ways.
Not in the least it has been (and still is) a challenge
to explain the possibilities of mixed reality
applications to the layman, that do not take
the abstractions we use in our daily academic
life for granted.
To be frank, it also has opened my eyes to what some
consider 'politically incorrect' technology,
in other words Microsoft DirectX,
originally developed as game technology,
and no doubt a rich toolbox for real life multimedia
applications.
Reinventing the wheel is not as simple as it seems.
Nevertheless, developing scenegraph processing facilities
and the appropriate timing mechanisms for controlling
the mixed reality presentation was, apart from
being a rekindling of basic skills, a learnful experience.
Finally, even before putting the application
to the test,
considering the aesthetics of mixed reality
theatre productions,
it may be called an eye-opener to realize
how important reality is,
and how meaningless explicit semantics
(in the form of text and images) may become.
Rather, our explorations were an incentive
to further explore more implicit (graphical)
modifications of the captured scene to convey meaning.
...
16
example(s) -- pizza boy
Did you like to go to a theatre play, when you were sixteen?
Peter van Kessel, one of the speakers in our visual design
develeped a game pizza boy to overcome
the resistance of secondary school students to theatre.
The game, developed by Peter's agency
Headland,
introduced elements of the theatre play in a game setting.
The player must deliver pizzas, riding a scooter, and
finds him/herself in a situations where he/she
must enact a role or perform some actions, related to the theatre
play.
Visiting the theatre, there is a MIDI-enabled scooter
with which the students can play the game, in preparation
for the actual play (image on the right in the figure above).
Peter reported that the game did indeed help
to overcome the scepticism these yougsters had with theatre.
The game was implemented by
the Little Chicken Game Company,
with
Virtools.
It was available for download at the website of
the governmental department that initiated
the development of the game,
but became such a huge success, also with
other people than the original target group,
that it had to be taken offline.
research directions -- computational art
I started studying computer music in 1979.
For about four years, I worked on a system for real-time sound synthesis
and algorithmic composition.
Some years later, I wrote an essay that I presented in 1986 at the First
Symposium for Electronic Arts, in Utrecht.
The essay was the published in Leonardo, [Eliens (1988)].
Now, almost 20 years later, I reread the abstract, which I OCRed from the original hardcopy:
Computational Art
The author conducts a simple thought experiment investigating the existence
and scope of 'computational art': the utilization of the computer in the visual arts and music.
In the experiment he sets the task of constructing an artifact that is capable of
producing works of art. Since it appears that the artifact needs at least the capability of
imagination, he queries the nature of images and imagery and argues that imagination
is strongly intentional. Next he introduces the concept of notational systems,
since they seem to govern the artistic activity of (not exclusively) machines.
Confronted with the question of whether we are able to develop a computational analogue for taste,
he finds that notational systems prove to be necessary for mediating the method of production
of an artwork and the appraisal of its artistic value.
Furthermore, the author shows that there are certain epistemological limits to the creativity of
an imaginative device.
Although the outcome of this hypothetical construction task clearly
denies the possibility of an autonomously creative artifact,
there seems to be no reason to worry about the opportunities for computational art:
the computer appears to be a unique tool in exploring the possibilities of artistic production,
guided by artists.
Today, despite the immense increase in computational power
and the overwhelming amount of multimedia peripherals, the computer is still not able
to produce art autonomously.
But as a tool it is about to take over the entire industry,
ranging from traditional publishing to film production and distribution.
And, perhaps more tangible, as a tool for the creation of media content
it is becoming ever better!
development(s) -- dialectic(s) of awareness
There is an overload of information.
Usually we don't pay much attention to this.
Increasingly, we rely on search (engines) to find the information, when needed.
But often our search is a bit disappointing,
simply because there are too many results.
If only there was more focus in search.
To assist you in proving focussed search, using google,
that limits search to your site, look at the following fragement of code.
google(s)
<form action="http://www.google.com/search?" method="GET">
<input name="q" value="+site:www.cs.vu.nl/~eliens/media ">
<input type="text" name="q" size=40>
<input type="submit" value=" ">
</form>
Adapt the url, and put it in your web page, and, voila,
your site is search-enabled.
On a deeper level, we may wonder why we are so impatient,
in a hurry to search for information,
and often not taking the time to properly digest it.
In [Eliens & Vyas (2007)] we wrote:
In the course of our field study for the PANORAMA system,
we tried to establish what relation users would have
to the system, not only in the way they interact
with it, but also in terms of what role the system plays
in their lives, and when and how they would be aware of
the systema.
Due to the intrinsic properties of the PANORAMA
system, as a system meant to support social awareness
in a work environment, we could not assume
direct focussed attention.
Instead, we must take
the various forms of awareness or attention into account.
Our thoughts in this direction were triggered
by a lecture of Linda Stone (former vice-president
of Microsoft) at the Crossmedia Week
September 2006 in Amsterdam,
entitled Attention -- the Real Aphrodisiac.
In that lecture Linda Stone made a distinction between
applications popular before 1985,
applications which were in general meant for
self-improvement, for example language-learning,
applications that were popular between 1985 and 2005,
applications that she characterized as supporting
continuous partial awareness,
such as email and news-feeds, and applications
of the period thereafter, from now into the future,
which may be characterized as applications
that allow the user to be creative, take part in a community,
and are in other words more focussed and less dependent
on the external environment.
Admittedly, it takes a few more steps to formulate
a theory of the dialectics of awareness.
However, with the function of the PANORAMA
system in mind, we may make, following [Benjamin (1936)],
some interesting distinctions between
the experience of art and architecture.
Where art is usually experienced in a delimited
time span, and is similarly delimited in space,
that is the position of the observer,
architecture is everywhere and always there.
As a consequence, art receives focussed attention
and may be appreciated with reflective distance,
whereas architecture is often not perceived
consciously, but merely present and subject to
an almost sub-conscious sensibility,
which is only brought to the focus of attention
when it is either aesthetisized, for example
when taking photographs, or when something
surprising is sensed, for example in the change
of skyline in New York.
As argued in [Hallnäss and Redström (2002)],
many of the new interactive systems,
whether in the category of ambient media,
ubiquitous computing or calm technology,
will fall somewhere inbetween the spectrum spanned
by art and architecture, or more likely even
alternate between the forms of awareness associated
with respectively art and architecture.
In designing the new interactive systems and games, we need to be explicitly concerned with
the actual phases of awareness that occur, simply because it
is not clear what role these systems play in our life.
When introducing a new system or artefact, we may distinguish
between the following phases:
phases of awareness
- initiation -- appeal to curiosity
- promotion -- raising interest
- progression -- prolonged involvement
As designers we must ask ourselves the following questions.
How do we appeal to the users' curiosity, so that our system is noticed?
How do we get a more sustained interest?
How de we get the user to interact with or contribute to the system?
And, how do we obtain prolonged involvement, and avoid boredom?
These questions are not simple to answer, and require also an understanding of the
actual context in which the system is deployed as well as an understanding
of the level of (aesthetic) literacy of the user(s).
Aesthetic awareness is common to us all, [Saw (1971)].
Having an understanding of aesthetic awareness,
can we isolate the relevant design parameters and formulate
rules of composition that may help us in developing
interactive applications?
According to our philosophical credo, [Eliens (1979)], no!
However, the history of art clearly shows the impact
of discoveries, such as the discovery of perspective,
as well as conventions in the interpretation of
art, as for example in the iconic representation
of narrative context in 17th century Dutch painting.
Moreover, the analysis of the visual culture of mass media may also
give us better understanding of the implied meaning
of compositional structures.
The notion of perspective, described in [Alberti (1435)],
is an interesting notion in itself,
since it describes both the organisation of the image
as well as the optimal point of view of the viewer.
The normal perspective as we know it is the central
perspective.
However, there are variants of perspective that force the
viewer in an abnormal point of view,
as for example with anamorphisms.
Perspective had an enormous impact on (western) art
and visual culture.
It defines our notion of naturalist realism, and allowed
for the development of the panorama as
a mass medium of the 19th century, [Grau (2003)].
Art that deviated from central perspective, such
as cubism or art from other cultures, was often
considered naive.
Photography and its pre-cursors had a great impact
on the perfection of perspectivist naturalism,
and what is called photorealism
became the touchstone of perfection for early
computer graphics, [Bolter and Grusin (2000)].
Apart from perspective, other conventions regulate
the composition of the 2D image,
in particular, following [Kress and van Leeuwen (1996)],
the information value related to where
an object is placed in the image,
and the salience of the object,
determined by its relative size, being foreground or background,
and visual contrast.
Also framing is used to emphasize meaning,
as for example in the close-up in a movie shot.
In analysing a large collection of image material,
[Kress and van Leeuwen (1996)], somewhat surprisingly found
that lef/right positioning usually meant
given versus new,
top/bottom positioning ideal versus real,
and centre/margin positioning important
versus marginal.
It is doubtful whether these meaning relationships
hold in all cultures, but as a visual convention
it is apparently well-rooted in western visual culture.
...
17
questions
concepts
technology
projects & further reading
As a project, you may develop a dialog engine
for non-linear interactive story telling
or a collage generator, that produces
artworks from a collection of images.
You may further explore the various presentation platforms,
and assess the tradeoffs with respect to the support they offer
for authoring.
For further reading, I suggest to study
interaction design patterns.
It is also worthwhile to get some books on modern art,
to gain some knowledge about art and design.
- street logos -- images from [Manco (2004)].
- photograph of oilpaint box.
- Mark Veldhuijzen van Zanten -- the six roles
in their agency.
- Mark Veldhuijzen van Zanten -- to design for the salon,
periodic lounge evenings in musea and art institutes.
- Geert Mul -- interactive multimedia installation.
- Geert Mul -- multimedia installation in dutch consulate in India.
- website of Institute of Creative Technologies, showing scenes from
Mission rehearsal Exercise (MRE).
- street logos -- more images from [Manco (2004)].
- website for Visual Sensations, a yearly VJ contest in the Netherlands,
developed by the agency of mark Veldhuijzen van Zanten.
- Geert Mul -- Harbour Sound & Vision, 1999
- screenshots from virtual atelier of Marinus Boezem.
- left: don't spit, a chines poster against spitting
during the SARS period, taken from dutch newspaper;
right: filmteckarna, [Wiedermann (2004)].
- sketches -- from filmteckarna, [Wiedermann (2004)].
- sketches -- from filmteckarna, [Wiedermann (2004)].
- sketches -- from filmteckarna, [Wiedermann (2004)].
- game -- pizza boy, developed by
Headland, see
9.3.
- signs -- health and safety, [ van Rooijen (2003)], p. 258, 259
The artwork for this chapter is meant
to emphasize context.
The street logos opening this chapter,
as well as the work
of Mark veldhuijzen van Zanten and Geert Mul,
must be experienced in a context
to fully appreciate their meaning.
Also for the MRE
application, it is the context,
in this case the stress and anxiety of a war situation,
that determines the impact.
The photorealistic graphic style of MRE,
wellknown by the trainees from other games,
is meant
to strengthen the experience of immersion.
Notice that the street logos assume an almost iconic character.
(C) Æliens
23/08/2009
You may not copy or print any of this material without explicit permission of the author or the publisher.
In case of other copyright issues, contact the author.
