preface -- topical media

topical media & game development

This book provides a concise and comprehensive introduction to multimedia. It arose out of the need for material with a strong academic component, that is material related to scientific research.

Indeed, studying multimedia is not only fun. Compare it with obtaining a driver license. Before you are allowed to drive on the highway, you have to take a theory exam. So why not take such an exam before entering the multimedia circus.

Don't complain, and take the exam. After all it makes you aware of the rules governing the (broadband) digital highway.

themes and variations


An underlying thought that motivated the writing of this book is that somehow the gap between authoring and retrieval should be bridged.

In other words, either by developing the technology for extracting features or attributes from multimedia objects, or by applying content annotation for such objects, multimedia information retrieval should be considered as a necessary asset to make a multimedia web an effective information repository.

what do we have to learn?

digital convergence

...

broadband communication

...

multimedia information retrieval

assignment

Annotated Tour in Amsterdam

multimedia

intended audience(s)


multimedia course


part(s)


This book is the result of developing the course notes for an introduction to multimedia for first year Computer Science and Information Science students. Hence, first of all, I like to thank the students that had to endure all the rough drafts of this material, and perhaps not to forget my experiment(s) with the presentation format of it.

references

  1.  [Convergence],  [Entertainment],  [Experience].
  2.  [Spaces],  [Hypermedia],  [Magic].
  3.  [Codecs],  [MPEG-4],  [Web].
  4.  [Game],  [Animation],  [Cg]
  5.  [MMDBMS],  [IR].
  6.  [MMDBMS],  [Meldex],  [ACOI].
  7.  [MMDBMS],  [Networked],
  8.  [Networked],  [Query],  [Community].
  9.  [Effects],  [Remediation],
  10.  [Chapman1],  [Chapman2],  [Magic],
  11.  [Ultimate],  [VirtualArt],
  12.  [HalfReal],  [Film],  [Cinema],  [Semiotics].

more than the art of turning base metals into gold, alchemy is a system of cosmic symbolism

perfect solutions

digital convergence


...



perspectives -- digital convergence


...



the artwork

  1. alchemy -- an illustration from a book about alchemy, from which also the quote is taken, the quote is explained in the afterthoughts.
  2. signs -- ancient chemical symbols,  [Signs], p. 171, 172.
  3. photographs -- Jaap Stahlie, from portrait series.

1

digital culture

life is becoming digital

digital convergence

learning objectives

After reading this chapter you should be able to define the notion of multimedia, recount the history of digital entertainment, explain the concept of digital convergence, discuss the future of cyberspace, and speculate about the commercial viability of mobile multimedia.

We live in the digital era,  [Negroponte (1995)]. We are surrounding ourselves with gadgets and we are consuming immense amounts of information, that is increasingly being delivered to us via the Internet. We play games, and we still watch (too much) television.

Some of us watch televion on our PCs, and may be even looking forward to watch television on their mobile phone. This is multimedia. For others, the PC is still a programmable machine. Being able to program it might earn you a living. Understanding multimedia, however, might even provide you with a better living.

In this chapter, we study what trends may currently be observed in the creation and delivery of multimedia information, and we explore what impact the digital revolution may have from a commercial perspective.

...



levels of meaning


cultural convergence


The cultural convergence of art, science, and technology provides ample opportunity for artists to challenge the very notion of how art is produced and to call into question its subject matter and its function in society.

standardization and uniformity


  1. Telematic media were incorporated very quickly in the globalization strategies of transnational corporations and their political administrators and they became increasingly dependent on existing power structures.
  2. At the other end of the scale, there were individuals, or comparatively small groups, who projected great hopes onto these networks as a testing ground for cultural, artistic and political models that would give greater prominence and weight to divergence and plurality.

Scientific American (november 2000)


The barriers between TV, movies, music, videogames and the Internet are crumbling.

Audiences are fetting new creative options.

Here is what entertainment could become if the technological and legal hurdles can be cleared ...

Underlying the importance of entertainment in the era of digital convergence is the premisse governing an entertainment economy, which may be stated as

there is no business without show business

evolution of digital entertainment


  • 1953: Winky Dink (CBS) -- interactive television, drawing exercise
  • 1972: Pong (Atari) -- ping-pong on computer screen
  • 1977: Adventure -- text-based interactive fiction
  • 1983: Dragon's Liar -- laser-disc technology 3D game
  • 1989: SimCity -- interactive simulation game
  • 1989: Back to the Future -- the Ride
  • 1993: Doom -- 3D action game
  • 1995: The Spot -- interactive web-based soap opera (Webisodic)
  • 1999: IMAX3D -- back to Atlantis (Las Vegas)
  • 2000: Big Brother -- TV + around the clock Web watch + voting
  • 2001: FE Sites -- fun enhanced web sites

experience is fundamental to human life


The desire to share experiences will be the motivating factor in the development of exciting multimedia technology in the foreseeable future.

communication technology


  • oral -- communicate symbolic experiences
  • writing -- record symbolic experiences
  • paper -- portability
  • print -- mass distribution
  • telegraph -- remote narrow communication
  • telephone -- remote analog communication
  • radio -- analog broadcasting of sound
  • television -- analog A/V broadcasting
  • recording media -- analog recording
  • digital processing -- machine enhancement
  • internet -- multimedia communication

the medium was the message when only one medium could be used to communicate messages.

... cyberspace is a construct in terms of an electronic system.

cyberspace


television, video cassettes, video tape-recorder/players, video games, and personal computers all form an encompassing electronic system whose various forms interface to constitute an alternative and absolute world that uniquely incorporates the spectator/user in a spatially decentered, weakly temporalized and quasi-disembodied state.

virtual reality


virtual reality (is) when and where the computer disappears and you become the 'ghost in the machine' ...

history

the receiver at the RCA Pavillon was way ahead of its time, it was a combination of television - radio - recorder - playback - facsimile - projector ...

digital convergence

the union of audio, video and data communication into a single source, received on a single device, delivered by a single connection

subsidiary convergences


  • content -- audio, video, data
  • platform -- PC, TV, internet, game machine
  • distribution -- how it gets to your platform

convergence


  • content -- 2D/3D graphics, data, video, audio
  • distribution -- broadcast, wireless, DVD, internet, satelite, cable
  • platform -- PC, television, game machine, wireless data pad, mobile phone

acronyms


  • HDTV -- high definition television
  • SDTV -- standard definition television
  • ITV -- interactive television

a killer d-TV appliance ...

  • personal television -- TiVo, Replay-TV (MPEG-2 cache)
  • game machine -- Sony PS 2/3, X-Box

TV or PC


The roadblock to the Entertainment PC could be the PC itself. Even a cheap TV doesn't crash or freeze. The best computers still do.

distribution


  • telephone network -- from 0.5 - 2 Mbps to 60 Mpbs (2.5km)
  • broadcast TV -- 6 MHz / 19 Mbps (4 channels MPEG HDTV)
  • cable TV -- hybrid fiber-optic coaxial cable 6 Mbps
  • fixed wireless -- 2 Mbps (radiotowers + rooftop antenna), phones/handhelds
  • satellite -- downloads to 100kbps, modem for uploads ...

digital convergence


what will we do with convergence once we have it?

emergence

we will watch

Google Earth


media as materials


each medium of communication tended to create a dangerous monopoly of knowledge

technological determinism


technological determinism was not the answer, ... more attempts were to be made to provide answers about the social consequences of television than had ever been asked about radio.

information


Information became a major concern anywhere during the late 1960 and 1970s where there was simultaneous talk both of 'lack of information' and 'information saturation'.

 [Briggs and Burke (2001)], p. 555

Peter Greven 23/3/2001 (Volkskrant)


new media sucks

--

people like new technology.

they don't like new media.

www.tvmeetstheweb.com


streaming media (audio and video), interactive gaming, virtual reality and 3D animation, interactive TV programming, interactive advertising, video on-demand, webcasting and multimedia

strategic questions


  • how quickly will wireless connectivity speeds improve?
  • what is the demand for services that deliver music and video to wireless devices?
  • how can suppliers of multimedia services monetize demand for wireless access?
  • how much will it cost to stream multimedia content to wireless devices now and in 2006?
  • are consumers willing to compromise quality for lower cost?

the players


Alltel, AT&T Wireless, AtomShockwave, Cingular Wireless, Clear Channel, HitHive, Ifilm, Infinity, KDDI, Liquid Audio, LMIV, Mannesmann, MP3.com, MTV, NetCom, Myplay, Nortel Networks, NTT DoCoMo, Omnitel, Sprint, Telefonica, Telstra, Vitaminic, Verizon Wireless, Virgin Megastores, Vodafone, Voicestream.

functions of media


information, education, entertainment

medium


television is a medium 'because it is neither rare nor well done'

information society


the new term 'information society' gave form to a cluster of hitherto more loosely related aspects of communication -- knowledge, news, literature, entertainment, all exchanged through different media and different media materials -- paper, ink, canvas, paint, celluloid, cinema, radio, television and computers.

From the 1960s onwards, all messages, public and private, verbal and visual, began to be considered as 'data', information that could be transmitted, collected, recorded, whatever their point of origin, most effective through electronic technology.

communication


  • what -- content
  • who -- control
  • whom -- audience (how many)

第二生命


Second Life seems to be overtaking the world. In the whole range of cummunity-building platforms, Second Life stands out as an immersive 3D world with an almost stunning adoption, by both individuals, companies and institutions, followed attentively by the Press. Not entirely without an understanding of the value of press coverage, the VU University Amsterdam decided to create presence in Second Life, by creating a virtual campus, to realize a (virtual) community of learners,  [Eliens et al. (2007)]. And, indeed, we succeeded in being the first university in The Netherlands with presence in Second Life and, as hoped, this was covered in the 8 o'clock nation-wide TV news.

success factors (1/2)


  • convergence of social networking and content creation
  • immersive networked 3D environment
  • inclusion of elementary economic principles

success factors (2/2)


  • don't miss the boat effect
  • free and easy accessible 3D design tool set
  • adoption by big companies like IBM, Reebok, ...
  • marketing of Second Life by Linden Lab (?)
  • the promise to make (real) money (?)

reference model


  • rules -- construct and communicate!
  • outcome -- a second world
  • value -- virtual and real (monetary) rewards
  • effort -- requires elementary skills
  • attachment -- a virtual identity
  • consequences -- transfer to first life

critical theory


attempt(s) to link the arts, literature, media studies, politics, sociology, antropology, philosophy and technology in an interdisciplinary search for relevant concepts and frameworks with which to understand the current world.

contemporary perception(s)


... in the contemporary world, the perceptual task has changed, in both leisure and work, to monitor data displays, ready for events.

californian dream(s)


... the new faith has emerged from a bizarre fusion of the cultural bohemianism of San Francisco with the high-tech industries of Silocon Valley...

and, to further de-construct the digital utopianism:

... the californian ideology promiscuously combines the freewheeling spirit of the hippies and the entrepeneurial zeal of of the yuppies

digital class


... the shadow side of the digital class's freedom and individuality is a lack of connection ... and an unrealized acceptance of work as the main life value.

cyberfeminism(s)


... empowerment of women in the field of new electronic media can only result from the demystification of technology and the appropriation of access to (these) tools.

projects & further reading

As a project, consider the development of a Java-based mobile game using J2ME, see  [Morrison (2005)], or a web-based game using Visual Basic .NET, see  [Santos Lobao and Hatton (2003)].

You may further explore multiplatform game development, and find arguments to choose for either Java-based or managed code based implementations.

For further reading, I advice to have a look at the special issues of the Scientific American,  [American], and the CACM on the next 1000 years of computing,  [CACM (2001)], and, for getting an idea where this all leads to, Schneidermann's Leonardo's laptop,  [Shneiderman (2003)]. For Second Life, see  [Rymaszweski et al. (2007)].

the artwork

  1. photographs of art works by Marina Abramovic, Art must be beautiful, Blue period, Dissolution, Dozing consciousness, In between, with (pending) permission from Montevideo. See also section 10.2.
  2. medium, according to the Visual Thesaurus.
  3. fMRI Research on Virtual Reality Analgesia, see section 1.1.
  4. television and communication, according to the Visual Thesaurus.
  5. TV Today, exhibition at Montevideo, februari 2005.
  6. visible world -- taken from  [Rosenblum and Macedonia (2002)], see section 1.2.
  7. personal event database and personal gadgets, from Freeband project.
  8. Thomas Lips 1975, Thomas Lips 1993, from Marina Abramovic, with permission from Montevideo.
  9. scanlines from Woody Vasulka, 197x, with permission from the artist.
  10. VU @ SecondLife, taken from  [Eliens et al. (2007)].
  11. signs -- people,  [ van Rooijen (2003)], p. 254, 256.

everything must be intertwinkled

hypermedia information spaces

learning objectives

After reading this chapter you should be able to define information spaces in a precise manner, position the hypertextual capabilities of the web in a historical perspective, explain the difference between multimedia and hypermedia, and argue why computational support for narrative structure in multimedia applications is desirable.

However entertaining it might be presented to you, underlying every multimedia presentation there is an information space. That is to say, irrespective of the medium, there is a message. And being confronted with a message, we might want to inquire for more information.

In this chapter, we will define the notion of information space more precisely.

We will extend this definition to include information hyperspaces, by looking at the history of hypertext and hypermedia.

Finally, we will discuss visualisation as a means to present (abstract) information in a more intuitive way, and we will reflect on what is involved in creating compelling multimedia.

Current day multimedia information systems distinguish themselves from older day information systems not only by what information they contain, that includes multimedia objects such as images and sounds, but also by a much more extensive repertoire of query mechanisms, visual interfaces and rich presentation facilities. See  [Chang and Costabile (1997)].

S.K. Chang and M.F. Costabile -- Visual Interfaces to Multimedia Databases


The Handbook of Multimedia Information Management


multimedia information systems

  • storage technology -- multimedia databases
  • wideband communication -- distribution accross networks
  • parallel computing -- voice, image and video processing
  • graphic co-processors -- visual information with high image quality

multimedia applications

geographical information systems, office automation, distance learning, health care, computer aided design, scientific visualization, and information visualization.

multimedia databases

  • the size of data,
  • synchronization issues,
  • query mechanisms, and
  • real time processing.

an information space is a representation of the information stored in a system or database that is used to present that information to a user.

we must distinguish between a visual information space (for presentation), a logical information space (in which we can reason about abstract information objects) and a physical information space (where our concrete multimedia objects are stored).

  • physical information space -- images, animations, video, voice, ...
  • logical information space -- abstract database objects
  • presentational information space -- to present information to the user

a logical information space is a multidimensional space where each point represents an object from the physical information space (read database).

  • information object -- a point in the (logical) information space
  • query -- an arbitrary region in this information space
  • clue -- a region with directional information, to facilitate browsing

www.w3.org/XML/1999/XML-in-10-points


XML is a set of rules (you may also think of them as guidelines or conventions) for designing text formats that let you structure your data.

XML in 10 points


  1. XML is for structuring data
  2. XML looks a bit like HTML
  3. XML is text, but isn't meant to be read
  4. XML is verbose by design
  5. XML is a family of technologies
  6. XML is new, but not that new
  7. XML leads HTML to XHTML
  8. XML is the basis for RDF and the Semantic Web
  9. XML is license-free, platform-independent and well-supported

related technologies


  • Xlink -- hyperlinks
  • XPointer -- anchors and fragments
  • XSL -- advanced stylesheets
  • XSLT -- transformation language
  • DOM -- object model for application programmer interface
  • schemas -- to specify the structure of XML documents

XML


  • separate data from presentation
  • transmit data between applications

information hyperspace


the logical information space may further be structured in a logical information hyperspace, where the clues become hyperlinks that provide directional information, and the information space can be navigated by the user following directional clues.

history


  • 1945 -- Vannevar Bush (Memex) -- as we may think,  [ Bush (1995)]
  • 1963 -- Douglas Engelbart (Augment) -- boosting the human intellect  [Engelbart (1963)]
  • 1980 -- Ted Nelson (Xanadu) -- everything is intertwinkled,  [Nelson (1980)]

  • flash 1: we are in trouble (human mankind)
  • flash 2: we need to boost mankind's ability to deal with complex urgent problems
  • flash 3: aha, graphic vision surges forth of me ...
  • flash 4: hypermedia -- to augment the human intellect
  • flash 5: augment (multimedia) workstation -- portal into an information space

hypermedia systems

  • components -- text, graphics, audio, video
  • links -- relations between components
  • presentation -- structured display

A curriculum promotes a false simplification of any subject, cutting the subject's many interconnections and leaving a skeleton of sequence which is only a charicature of its richness and intrinsic fascination.

classification of hypermedia systems

  • macro-literary systems -- publishing, reading, criticism
  • problem exploration tools -- authoring, outlining, programming
  • browsing systems -- teaching, references, information
  • general hypermedia technology -- authoring, browsing, collaboration
  • embedded hypermedia -- CASE, decision support, catalogs

component

  • content -- text, graphics, video, program
  • attributes -- semantic description
  • anchors -- (bi-directional) links to other documents
  • presentation -- display characteristics

(CMIF) multimedia model

  • data block -- atomic component
  • channel -- abstract output device
  • synchronization arc -- specifying timing constraints
  • event -- actual presentation

Amsterdam Hypermedia Model

  • contents -- data block
  • attributes -- semantic information
  • anchors -- (id, value)
  • presentation -- channel, duration, ...

research issues


  • search and query -- for better access
  • composition -- for imposing structure
  • virtual structures -- on top of existing structures
  • computation -- for flexibility and interaction
  • versioning -- to store modification histories
  • collaborative work -- sharing objects with multiple users
  • extensibility and tailorability -- to adapt to individual preferences

visualization


Grasping the whole is a gigantic theme, intellectual history's most important.

Ant vision is humanity's usual fate; but seeing the whole is every thinking person's aspiration.

David Gelernter, Mirror Worlds 1992


data types


  • 1-D linear data -- text, source code, word index
  • 2-D map data -- floor plan, office layout
  • 3-D world -- molecules, schematics, ...
  • temporal data -- 1 D (start, finish)
  • multi-dimensional data -- n-dimensional (information) space
  • tree data -- hierarchical
  • network data -- graph structure

  • interactive -- overview first, zoom and filter, then details on demand
  • storytelling -- as a paradigm for information presentation

Whatever your target audience, whatever your medium, whatever your message, you have to be convincing if not compelling.

persuasion

  • a communication process in which the communicator seeks to elicit a desired response from his receiver
  • a conscious attempt by one individual to change the attitudes, beliefs or behaviours of another individual or group individual through the transmission of some messages.

(re)mediation

  • immediacy -- a tendency towards transparent immersion, and
  • hypermediacy -- the presence of referential context

Virtual Reality won't merely replace TV. It will eat it alive.

immediacy

  • epistemological: transparency, the absence of mediation
  • psychological: the medium has disappeared, presence, immersion

hypermediacy

  • epistemological: opacity, presence of the medium and mediation
  • psychological: experience of the medium is an experience of the real

Convergence is the mutual remediation of at least three important technologies -- telephone, televison and computer -- each of which is a hybrid of technical, social and economic practice, and each of which offers its own path to immediacy.

The telephone offers the immediacy of voice or the interchange of voices in real-time.

Television is a point-of-view technology that promises immediacy through its insistent real-time monitoring of the world.

The computer's promise of immediacy comes through the combination of three-dimensional graphics, automatic (programmed) action, and an interactivity that television can not match.

As they come together, each of these is trying to absorb the others and promote its own version of immediacy.

convergence

(p. 27) ... merging previously disparate technologies of communication and representation into a single medium.

The networked computer acts like a telephone in offering one-to-one real-time communication, like a television in broadcasting moving pictures, like an auditorium in bringing groups together for lectures and discussion, like a library in offering vast amounts of textual information for reference, like a museum in its ordered presentation of visual information, like a billboard, a radio, a gameboard and even like a manuscript in its revival of scrolling text.

interactive


  • procedural -- 'programmed media' ...
  • participatory -- offering agency

immersive


  • spatial -- explorable in (state) space
  • encyclopedic -- with (partial) information closure

multimedia authoring


  • narrative format
  • procedural authorship

web 2.0


video sharing / online gaming / social networking

daft punk -- technologic (cn / jp)



  Buy it, use it, break it, fix it.
  Trash it, change it, melt -- upgrade it.
  Change it, point it, zoom it, press it.
  Snap it, work it, quick -- erase it.
  Write it, out it, paste it, save it.
  Load it, check it, quick -- rewrite it.
  Plug it, play it, burn it, rip it.
  Drag and drop it, zip -- unzip it.
  Look it, fill it, curl it, find it.
  View it, coat it, jam -- unlock it.
  Surf it, scroll it, pose it, click it.
  Cross it, crack it, twitch -- update it.
  Name it, rate it, tune it, print it.
  Scan it, send it, fax -- rename it. 
  Touch it, bring it. Pay it, watch it.
  Turn it, leave it, stop -- format it.
  

mashup(s)


  • substituting a single pragmatism for ideal design
  • using light weight programming models

web 2.0 design pattern(s)


  • web 1.0 -- the web as platform
  • web 2.0 -- architecture of participation
  • web 3.0 -- data is the (intel) inside

Learnlog: XML Is The Fabric Of Web 2.0 Applications

  • the client side is AJAX (Asynchronous Javascript and XML)
  • the server application typically exposes data through XML
  • the interaction model is web services
  • mashups combine multiple webservices to create new types of applications

REST


  • representation -- encoding in a particular format
  • state -- data encapsulated in an object
  • transfer -- using HTTP methods

the most powerful mashups don't just mix code and data, they mix cultures.

which provides a challenge that trancends all issues of mere technological correctness.

built-in(s)


resource(s)


  • /seen?user=SomeAvatar -- records the presence of SomeAvatar
  • /touched?user=SomeAvatar -- invokes flickr API with users tag
  • /set_tag?user=SomeAvatar&tag=FavoriteTag -- records SomeAvatar's favourite tag

flash/quicktime in SL


Quicktime supports Flash, but only up to Flash version 5. We're up to version 9 on that now! Luckily, I have been dabbling with Flash since the early days, so already knew how to do this 'the old way'... So, Flash is doing all the work. No LSL at all... I heart feeds. Did I say 'I heart feeds?

Referring to section 7.4 for a more detailed discussion, we may observe that there is no meaning in merely putting things together. Without mechanisms of personalization and recommendation we would simply be flooded by data and information, in a way that even search would not be able to cope with. Context, narratives and personalized presentation(s), notions from the past, reappear as keywords for the future of the web 2.0 and beyond.

projects & further reading

As a project, I suggest the development of a virtual tour in a city, museum or other interesting locatoion.

You may further explore the implementation of traversal within a context, taking into account the history of navigation when backtracking to a particular point, issues in hyperlinking and interaction in multimedia applications, and computational support for narratives.

For further reading I advice you to take a look at the history of hypermedia and the web, using online material from the W3C, or the history of media as accounted for in  [Briggs and Burke (2001)] and  [Bolter and Grusin (2000)].

the artwork

  1. book covers --  [Weishar (1998)],  [Eco (1994)],  [Burger (1981)],  [Kunst],  [Betsky (2004)]
  2. Federico Campanale -- Oxygen, fragments from video installation, 2004
  3. Vasarely --  [Diehl 1973].
  4. Vasarely --  [Diehl 1973].
  5. Vasarely --  [Diehl 1973].
  6. Federico Campanale -- Oxygen, more fragments.
  7. student work -- from introduction multimedia 2000.
  8. Rutger van Dijk -- mobius, interactive story, opening screen, see section 2.3.
  9. edgecodes -- screenshots, see section 2.3
  10. signs -- people,  [ van Rooijen (2003)], p. 244, 245.

if you linger for a long time in one place you'd almost think there must be something there

wittgenstein

delivery & presentation


perspectives -- delivery & presentation


  • technical -- codec selection
  • political -- market vs. consortium
  • sociological -- digital services
  • legal -- copyright protection
  • scientific -- experience design
  • computer science -- computational support
  • futuristic -- global & personal information
  • commercial -- WMV, Quicktime, RealONE

the artwork

  1. logo -- a drawing by Soutine, it is (almost) my personal logo, and also decorates the cover of  [Eliens (2000)].
  2. signs -- property marks,  [ van Rooijen (2003)], p. 76, 77.
  3. photographs -- Jaap Stahlie, commissioned work.

without compression delivery is virtually impossible

codecs and standards

learning objectives

After reading this chapter you should be able to demonstrate the necessity of compression, to discuss criteria for the selection of codecs and mention some of the alternatives, to characterize the MPEG-4 and SMIL standards, to explain the difference between MPEG-4 and MPEG-2, and to speculate about the feasibility of a semantic multimedia web.

Without compression and decompression, digital information delivery would be virtually impossible. In this chapter we will take a more detailed look at compression and decompression. It contains the information that you may possibly need to decide on a suitable compression and decompression scheme (codec) for your future multimedia productions.

We will also discuss the standards that may govern the future (multimedia) Web, including MPEG-4, SMIL and RM3D. We will explore to what extent these standards allow us to realize the optimal multimedia platform, that is one that embodies digital convergence in its full potential. Finally, we will investigate how these ideas may ultimately lead to a (multimedia) semantic web.

compression is the key to effective delivery

mediauncompressedcompressed
voice 8k samples/sec, 8 bits/sample64 kbps2-4 kbps
slow motion video 10fps 176x120 8 bits5.07 Mbps8-16 kbps
audio conference 8k samples/sec 8bits64 kbps16-64 kbps
video conference 15 fps 352x240 8bits30.4 Mbps64-768 kbps
audio (stereo) 44.1 k samples/s 16 bits1.5 Mbps128k-1.5Mbps
video 15 fps 352x240 15 fps 8 bits30.4 Mbps384 kbps
video (CDROM) 30 fps 352x240 8 bits60.8 Mbps1.5-4 Mbps
video (broadcast) 30 fps 720x480 8 bits248.8 Mbps3-8 Mbps
HDTV 59.9 fps 1280x720 8 bits1.3 Gbps20 Mbps

(phone: 56 Kb/s, ISDN: 64-128 Kb/s, cable: 0.5-1 Mb/s, DSL: 0.5-2 Mb/s)

images, video and audio are amenable to compression

statistical redundancy in signal


  • spatial correlation -- neighbour samples in single frame
  • temporal correlation -- between segments (frames)

irrelevant information


  • from perceptual point of view

B. Vasudev & W. Li, Memory management: Codecs


codec = (en)coder + decoder



  signal  -> source coder   ->  channel coder    (encoding)
  
  signal  <- source decoder <-  channel decoder  (decoding)
  

codec design problem


From a systems design viewpoint, one can restate the codec design problem as a bit rate minimization problem, meeting (among others) constraints concerning:

  • specified levels of signal quality,
  • implementation complexity, and
  • communication delay (start coding -- end decoding).

tradeoffs

  • resilience to transmission errors
  • degradations in decoder output -- lossless or lossy
  • data representation -- browsing & inspection
  • data modalities -- audio & video.
  • transcoding to other formats -- interoperability
  • coding efficiency -- compression ratio
  • coder complexity -- processor and memory requirements
  • signal quality -- bit error probability, signal/noise ratio

  • pixel-based -- MPEG-1, MPEG-2, H3.20, H3.24
  • object-based -- MPEG-4

MPEG-1 video compression uses both intra-frame analysis, for the compression of individual frames (which are like images), as well as. inter-frame analysis, to detect redundant blocks or invariants between frames.

frames


  • I: intra-frames -- independent images
  • P: computed from closest frame using DCT (or from P frame)
  • B: computed from two closest P or I frames

GigaPort


  • optical network technologies - models for network architecture, optical network components and light path provisioning.
  • high performance routing and switching - new routing technologies and transport protocols, with a focus on scalability and stability robustness when using data-intensive applications with a high bandwidth demand.
  • management and monitoring - incident response in hybrid networks (IP and optical combined) and technologies for network performance monitoring, measuring and reporting.
  • grids and access - models, interfaces and protocols for user access to network and grid facilities.
  • test methodology - effective testing methods and designing tests for new technologies and network components.

system spatial resolution frame rate mbps
NTSC704 x 480 30 243 mbps
PAL/SECAM 720 x 576 25 249 mbps

item streaming downloaded
bandwidth equal to the display rate may be arbitrarily small
disk storage none the entire file must be stored
startup delay almost none equal to the download time
resolution depends on available bandwidth depends on available disk storage

formats


Quicktime, introduced by Apple, early 1990s, for local viewing; RealVideo, streaming video from RealNetworks; and Windows Media, a proprietary encoding scheme fromMicrosoft.

Examples of these formats, encoded for various bitrates are available at Video at VU.

standards


  • XML -- eXtensible Markup Language (SGML)
  • MPEG-4 -- coding audio-visual information
  • SMIL -- Synchronized Multimedia Integration Language
  • RM3D -- (Web3D) Rich Media 3D (extensions of X3D/VRML)

"Perhaps the most immediate need for MPEG-4 is defensive. It supplies tools with which to create uniform (and top-quality) audio and video encoders on the Internet, preempting what may become an unmanageable tangle of proprietary formats."

MPEG-4


a toolbox of advanced compression algorithms for audiovisual information

scalability

  • bitrate -- switching to lower bitrates
  • bandwidth -- dynamically discard data
  • encoder and decoder complexity -- signal quality

audiovisual information


  • still images, video, audio, text
  • (synthetic) talking heads and synthesized speech
  • synthetic graphics and 3D scenes
  • streamed data applied to media objects
  • user interaction -- e.g. changes of viewpoint

example


Imagine, a talking figure standing next to a desk and a projection screen, explaining the contents of a video that is being projected on the screen, pointing at a globe that stands on the desk. The user that is watching that scene decides to change from viewpoint to get a better look at the globe ...

media objects


  • media objects -- units of aural, visual or audiovisual content
  • composition -- to create compound media objects (audiovisual scene)
  • transport -- multiplex and synchronize data associated with media objects
  • interaction -- feedback from users' interaction with audiovisual scene

composition


  • placing media objects anywhere in a given coordinate system
  • applying transforms to change the appearance of a media object
  • applying streamed data to media objects
  • modifying the users viewpoint

transport


The data stream (Elementary Streams) that result from the coding process can be transmitted or stored separately and need to be composed so as to create the actual multimedia presentation at the receivers side.

scenegraph


  • BIFS (Binary Format for Scenes) -- describes spatio-temporal arrangements of (media) objects in the scene
  • OD (Object Descriptor) -- defines the relationship between the elementary streams associated with an object
  • event routing -- to handle user interaction

DMIF


Delivery Multimedia Integration Framework

benefits


  • end-users -- interactive media accross all platforms and networks
  • providers -- transparent information for transport optimization
  • authors -- reusable content, protection and flexibility

managing intellectual property

XMT


  • XMT contains a subset of X3D
  • SMIL is mapped (incompletely) to XMT

SMIL


TV-like multimedia presentations

parallel and sequential


Authoring a SMIL presentation comes down, basically, to

name media components for text, images,audio and video with URLs, and to schedule their presentation either in parallel or in sequence.

presentation characteristics


  • The presentation is composed from several components that are accessible via URL's, e.g. files stored on a Web server.
  • The components have different media types, such as audio, video, image or text. The begin and end times of different components are specified relative to events in other media components. For example, in a slide show, a particular slide is displayed when the narrator in the audio starts talking about it.
  • Familiar looking control buttons such as stop, fast-forward and rewind allow the user to interrupt the presentation and to move forwards or backwards to another point in the presentation.
  • Additional functions are "random access", i.e. the presentation can be started anywhere, and "slow motion", i.e. the presentation is played slower than at its original speed.
  • The user can follow hyperlinks embedded in the presentation.

applications


  • Photos taken with a digital camera can be coordinated with a commentary
  • Training courses can be devised integrating voice and images.
  • A Web site showing the items for sale, might show photos of the product range in turn on the screen, coupled with a voice talking about each as it appears.
  • Slide presentations on the Web written in HTML might be timed so that bullet points come up in sequence at specified time intervals, changing color as they become the focus of attention.
  • On-screen controls might be used to stop and start music.

example



   <par>
      <a href="#Story"> <img src="button1.jpg"/> </a>
      <a href="#Weather"> <img src="button2.jpg"/></a>
       <excl>
           <par id="Story" begin="0s">
             <video src="video1.mpg"/>
             <text src="captions.html"/>
           </par>
  
           <par id="Weather">
             <img src="weather.jpg"/>
             <audio src="weather-rpt.mp3"/>
           </par>
       </excl>
   </par>
  

history


Experience from both the CD-ROM community and from the Web multimedia community suggested that it would be beneficial to adopt a declarative format for expressing media synchronization on the Web as an alternative and complementary approach to scripting languages.

Following a workshop in October 1996, W3C established a first working group on synchronized multimedia in March 1997. This group focused on the design of a declarative language and the work gave rise to SMIL 1.0 becoming a W3C Recommendation in June 1998.

SMIL 2.0 Modules


  • The Animation Modules
  • The Content Control Modules
  • The Layout Modules
  • The Linking Modules
  • The Media Object Modules
  • The Metainformation Module
  • The Structure Module
  • The Timing and Synchronization Module
  • The Time Manipulations Module
  • The Transition Effects Module

module-based reuse


  • SMIL modules could be used to provide lightweight multimedia functionality on mobile phones, and to integrate timing into profiles such as the WAP forum's WML language, or XHTML Basic.
  • SMIL timing, content control, and media objects could be used to coordinate broadcast and Web content in an enhanced-TV application.
  • SMIL Animation is being used to integrate animation into W3C's Scalable Vector Graphics language (SVG).
  • Several SMIL modules are being considered as part of a textual representation for MPEG4.

www.web3d.org


  • VRML 1.0 -- static 3D worlds
  • VRML 2.0 or VRML97 -- dynamic behaviors
  • VRML200x -- extensions
  • X3D -- XML syntax
  • RM3D -- Rich Media in 3D

groups.yahoo.com/group/rm3d/


The Web3D Rich Media Working Group was formed to develop a Rich Media standard format (RM3D) for use in next-generation media devices. It is a highly active group with participants from a broad range of companies including 3Dlabs, ATI, Eyematic, OpenWorlds, Out of the Blue Design, Shout Interactive, Sony, Uma, and others.

RM3D


The Web3D Consortium initiative is fueled by a clear need for a standard high performance Rich Media format. Bringing together content creators with successful graphics hardware and software experts to define RM3D will ensure that the new standard addresses authoring and delivery of a new breed of interactive applications.

requirements


  • rich media -- audio, video, images, 2D & 3D graphics (with support for temporal behavior, streaming and synchronisation)
  • applicability -- specific application areas, as determined by commercial needs and experience of working group members

  • interoperability -- VRML97, X3D, MPEG-4, XML (DOM access)

  • object model -- common model for representation of objects and capabilities
  • extensibility -- integration of new objects (defined in Java or C++), scripting capabilities and declarative content

  • high-quality realtime rendering -- realtime interactive media experiences
  • platform adaptability -- query function for programmatic behavior selection
  • predictable behavior -- well-defined order of execution
  • high precision number systems -- greater than single-precision IEEE floating point numbers
  • minimal size -- download and memory footprint

SMIL is closer to the author and RM3D is closer to the implementer.

working draft


Since there are three vastly different proposals for this section (time model), the original <RM3D> 97 text is kept. Once the issues concerning time-dependent nodes are resolved, this section can be modified appropriately.

time model


  • MPEG-4 -- spring metaphor
  • SMIL -- cascading time
  • RM3D/VRML -- event routing

MPEG-4 -- spring metaphor


  • duration -- minimal, maximal, optimal

SMIL -- cascading time


  • time container -- speed, accelerate, decelerate, reverse, synchronize


  <seq speed="2.0">
     <video src="movie1.mpg" dur="10s"/>
     <video src="movie2.mpg" dur="10s"/>
     <img src="img1.jpg" begin="2s" dur="10s">
                 <animateMotion from="-100,0" to="0,0" dur="10s"/>
     </img>
     <video src="movie4.mpg" dur="10s"/>
  </seq>
  

RM3D/VRML -- event routing


  • TimeSensor -- isActive, start, end, cycleTime, fraction, loop

web content

  • 1st generation -- hand-coded HTML pages
  • 2nd generation -- templates with content and style
  • 3rd generation -- rich markup with metadata (XML)

structure to the meaningful content of web pages,

meta data


Metadata is data about data. Specifically, the term refers to data used to identify, describe, or locate information resources, whether these resources are physical or electronic. While structured metadata processed by computers is relatively new, the basic concept of metadata has been used for many years in helping manage and use large collections of information. Library card catalogs are a familiar example of such metadata.

Dublin Core example



  <rdf:RDF
      xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
      xmlns:dc="http://purl.org/dc/elements/1.1/"
      xmlns:dcterms="http://purl.org/dc/terms/">
      <rdf:Description rdf:about="http://www.dlib.org/dlib/may98/miller/05miller.html">
        <dc:title>An Introduction to the Resource Description Framework</dc:title>
        <dc:creator>Eric J. Miller</dc:creator>
        <dc:description>The Resource Description Framework (RDF) is an
         infrastructure that enables the encoding, exchange and reuse of
         structured metadata. rdf is an application of xml that imposes needed
         structural constraints to provide unambiguous methods of expressing
         semantics. rdf additionally provides a means for publishing both
         human-readable and machine-processable vocabularies designed to
         encourage the reuse and extension of metadata semantics among
         disparate information communities. the structural constraints rdf
         imposes to support the consistent encoding and exchange of
         standardized metadata provides for the interchangeability of separate
         packages of metadata defined by different resource description
         communities. </dc:description>
        <dc:publisher>Corporation for National Research Initiatives</dc:publisher>
        <dc:subject>
          <rdf:Bag>
            <rdf:li>machine-readable catalog record formats</rdf:li>
            <rdf:li>applications of computer file organization and
             access methods</rdf:li>
          </rdf:Bag>
        </dc:subject>
        <dc:rights>Copyright © 1998 Eric Miller</dc:rights>
        <dc:type>Electronic Document</dc:type>
        <dc:format>text/html</dc:format>
        <dc:language>en</dc:language>
        <dcterms:isPartOf rdf:resource="http://www.dlib.org/dlib/may98/05contents.html"/>
      </rdf:Description>
  </rdf:RDF>
  

Dublin Core


  • title -- name given to the resource
  • creator -- entity primarily responsible for making the content of the resource
  • subject -- topic of the content of the resource
  • description -- an account of the content of the resource
  • publisher -- entity responsible for making the resource available
  • contributor -- entity responsible for making contributions to the content of the resource
  • date -- date of an event in the lifecycle of the resource
  • type -- nature or genre of the content of the resource
  • format -- physical or digital manifestation of the resource
  • identifier -- unambiguous reference to the resource within a given context
  • source -- reference to a resource from which the present resource is derived
  • language -- language of the intellectual content of the resource
  • relation -- reference to a related resource
  • coverage -- extent or scope of the content of the resource
  • rights -- information about rights held in and over the resource

information repository


The Web is becoming a universal repository of human knowledge and culture, which has allowed unprecedented sharing of ideas and information in a scale never seen before.

browsing & navigation


To satisfy his information need, the user might navigate the hyperspace of web links searching for information of interest. However, since the hyperspace is vast and almost unknown, such a navigation task is usually inefficient.

information agent


  • gather information
  • filter and select

presentation agent


  • access information
  • find suitable mode of presentation

PERsonal and SOcial NAvigation through information spaceS

PERSONAS


investigating a new approach to navigation through information spaces, based on a personalised and social navigational paradigm.

Agneta & Frieda


The AGNETA & FRIDA system seeks to integrate web-browsing and narrative into a joint mode. Below the browser window (on the desktop) are placed two female characters, sitting in their livingroom chairs, watching the browser during the session (more or less like watching television). Agneta and Frida (mother and daughter) physically react, comment, make ironic remarks about and develop stories around the information presented in the browser (primarily to each other), but are also sensitive to what the navigator is doing and possible malfunctions of the browser or server.

Agneta & Frieda


In this way they seek to attach emotional, comical or anecdotal connotations to the information and happenings in the browsing session. Through an activity slider, the navigator can decide on how active she wants the characters to be, depending on the purpose of the browsing session (serious information seeking, wayfinding, exploration or entertainment browsing).

game as social system


actorsrule(s)resource(s)
players eventsgame space
rolesevaluationsituation
goalsfacilitator(s)context

criteria


  • relevance -- what is our message?
  • identity -- who are we?
  • impact -- why would anybody be interested?

climate star


  • climate strategies -- (1) emission reduction, (2) adaptation
  • climate systems -- (3) feedback monitoring, (4) investment in research, (5) climate response
  • energy and CO2 -- (6) investment in efficiency, (7) investment in green technology, (8) governement rules
  • regional development -- (9) campain for awareness, (10) securing food and water
  • adaptation measures -- (11) public space, (12) water management, (13) use of natural resources
  • international relations -- (14) CO2 emission trade, (15) European negotiations, (16) international convenants

simulation parameters


  • people -- how is the policy judged by the people?
  • profit -- what is the influence on the (national) economy?
  • planet -- what are the effects for the environment?

game elements


  1. game cycle -- turns in subsequent rounds (G)
  2. simulation(s) -- based on (world) climate model (W)
  3. exploration -- by means of interactive video (E)

argument(s)


  • topic-centered -- common beliefs, use of logic, examples
  • viewer-centered -- patriotisms, religious or romantic sentimentality
  • speaker-centered -- the makers are well-informed, sincere and trusthworthy

projects & further reading

As a project, you may think of implementing for example JPEG compression, following  [Li and Drew (2004)], or a SMIL-based application for cultural heritage.

You may further explore the technical issues on authoring DV material, using any of the Adobe, mentioned in appendix E. or compare

For further reading I advice you to take a look at the respective specifications of MPEG-4 and SMIL, and compare the functionality of MPEG-4 and SMIL-based presentation environments. An invaluable book dealing with the many technical aspects of compression and standards in  [Li and Drew (2004)].

the artwork

  1. costume designs -- photographed from Die Russchische Avantgarde und die Buhne 1890-1930
  2. theatre scene design, also from (above)
  3. dance Erica Russel,  [Wiedermann (2004)]
  4. MPEG-4 -- bits rates, from  [Koenen (2000)].
  5. MPEG-4 -- scene positioning, from  [Koenen (2000)].
  6. MPEG-4 -- up and downstream data, from  [Koenen (2000)].
  7. MPEG-4 -- left: scene graph; right: sprites, from  [Koenen (2000)].
  8. MPEG-4 -- syntax, from  [Koenen (2000)].
  9. MIT Media Lab web site.
  10. student work -- multimedia authoring I, dutch windmill.
  11. student work -- multimedia authoring I, Schröder house.
  12. student work -- multimedia authoring I, train station.
  13. animation -- Joan Gratch, from  [Wiedermann (2004)].
  14. animation -- Joan Gratch, from  [Wiedermann (2004)].
  15. animation -- Joan Gratch, from  [Wiedermann (2004)].
  16. animation -- Joan Gratch, from  [Wiedermann (2004)].
  17. Agneta and Frieda example.
  18. diagram (Clima Futura) game elements
  19. signs -- people,  [ van Rooijen (2003)], p. 246, 247.

.. my history might well be your future ...

ted nelson

perspectives -- multimedia information retrieval


  • application(s) -- digital dossier
  • psychological -- focus
  • experimental -- user interaction
  • algorithmic -- (information) access
  • system -- unified presentation space
  • presentation -- embodied agents
  • search -- semantic annotation
  • commercial -- future systems

the artwork

  1. kata -- japanese martial arts picture.
  2. signs -- japanese coats of arms,  [ van Rooijen (2003)], p. 140, 141.
  3. photographs -- Jaap Stahlie, two early experiments (left, and right)

information retrieval is usually an afterthought

information retrieval

learning objectives

After reading this chapter you should be able to describe scenarios for information retrieval, to explain how content analysis for images can be done, to characterize similarity metrics, to define the notions of recall and precision, and to give an example of frequence tables, as used in text search.

Searching for information on the web is cumbersome. Given our experiences today, we may not even want to think about searching for multimedia information on the (multimedia) web.

Nevertheless, in this chapter we will briefly sketch one of the possible scenarios indicating the need for multimedia search. In fact, once we have the ability to search for multimedia information, many scenarios could be thought of.

As a start, we will look at two media types, images and documents. We will study search for images, because it teaches us important lessons about content analysis of media objects and what we may consider as being similar. Perhaps surprisingly, we will study text documents because, due to our familiarity with this media type, text documents allow us to determine what we may understand by effective search.

Amsterdam Drugport


Amsterdam is an international centre of traffic and trade. It is renowned for its culture and liberal attitude, and attracts tourists from various ages, including young tourists that are attracted by the availability of soft drugs. Soft drugs may be obtained at so-called coffeeshops, and the possession of limited amounts of soft drugs is being tolerated by the authories.

The European Community, however, has expressed their concern that Amsterdam is the centre of an international criminal drug operation. Combining national and international police units, a team is formed to start an exhaustive investigation, under the code name Amsterdam Drugport.

information

  • video surveillance -- monitoring
  • telephone wiretaps -- audio recording
  • photography -- archive
  • documents -- investigations
  • transactions -- structured data
  • geographic information -- locations, routes

media types


  • images -- photos
  • video -- surveillance
  • audio -- interviews, phone tracks
  • documents -- forensic, reports
  • handwriting -- notes
  • structured data -- transactions

retrieval


  • image query -- all images with this person
  • audio query -- identity of speaker
  • text query -- all transactions with BANK Inc.
  • video query -- all segments with victim
  • complex queries -- convicted murderers with BANK transactions
  • heterogeneous queries -- photograph + murderer + transaction
  • complex heterogeneous queries -- in contact with + murderer + transaction

information retrieval


Information retrieval, according to  [Baeza-Yates and Ribeiro-Neto (1999)], deals with the representation, storage, organisation of, and access to information items.

To see what is involved, imagine that we have a (user) query like:

find me the pages containg information on ...

information retrieval models


  • boolean or set-theoretic models
  • vector or algebraic models
  • probabilistic models

vector models


  • attribute term weighting scheme improves performance
  • partial matching strategy allows retrieval of approximate material
  • metric distance allows for sorting according to degree of similarity

image query


  • obtaining descriptive information
  • establishing similarity

content-based description


  • objects in image
  • shape descriptor -- shape/region of object
  • property description -- cells in image

shape


  • bounding box -- (XLB,XUB,YLB,YUB)

property


  • property -- name=value

example



  shape descriptor: XLB=10; XUB=60; YLB=3; YUB=50   (rectangle)  
  property descriptor: pixel(14,7): R=5; G=1; B=3 
  

definitions


  • image grid: (m * n) cells of equal size
  • cell property: (Name, Value, Method)

example



  property: (bwcolor,{b,w},bwalgo) 
  

similarity-based retrieval


How do we determine whether the content of a segment (of a segmented image) is similar to another image (or set of images)?

solutions

  • metric approach -- distance between two image objects
  • transformation approach -- relative to specification

metric approach


distance d:X->[0,1] is distance measure if:


           d(x,y) = d(y,x)
  	 d(x,y) <= d(x,z) + d(z,y)
  	 d(x,x) = 0
  

pixel properties


  • objects with pixel properties p_1,...,p_n
  • pixels: (x,y,v1,...,v_n)
  • object contains w x h (n+2)-tuples

complexity


a set of points in k-dimensional space for k = n + 2

feature extraction


  • maps object into s-dimensional space

transformation approach


Given two objects o1 and o2, the level of dissimilarity is proportional to the (minimum) cost of transforming object o1 into object o2 or vice versa

transformation operators



    to_1,...,to_r  -- translation, rotation, scaling
  

cost


  • cost(TS) = %S_{i=1}^{r} cost(to_{i})

distance


  • d(o,o') = min { cost(TS) | TS in TSeq(o,o') }

advantages


  • user-defined similarity -- choice of transformation operators
  • user-defined cost-function

operations



   rotate(image-id,dir,angle)
   segment(image-id, predicate)
   edit(image-id, edit-op)
  

image repository


  • storage -- unsegmented images
  • description -- limited set of features
  • index -- feature-based index
  • retrieval -- distance between feature vectors

mission


Our goal is to study aspects of the deployment and architecture of virtual environments as an interface to (intelligent) multimedia information systems ...

query


  • document database + string matching

problems


  • synonymy -- topic T does not occur literally in document D
  • polysemy -- some words may have many meanings

effective search


  • precision -- how many answers are correct
  • recall -- how many of the right documents are returned

precision and recall



  precision = ( returned and relevant ) / returned  
  recall = ( returned and relevant ) / relevant 
  

anomalies


  • return all documents: perfect recall, low precision
  • return 'nothing': 'perfect' precision, low recall

example


term/documentd0d1d2
snacks100
drinks103
rock-roll011

complextity


compare term frequencies per document -- O(M*N)

reduction


  • stop list -- irrelevant words
  • word stems -- reduce different words to relevant part

user-oriented measures


  • coverage ratio -- fraction of known documents
  • novelty ratio -- fraction of new (relevant) documents
  • relative recall -- fraction of expected documents
  • recall effort -- fraction of examined documents

Aesthetics


  • intentions -- motives of the artist
  • expression -- where form takes over
  • representation -- the relation of art to reality

projects & further reading

As a project, you may implement simple image analysis algorithms that, for example, extract a color histogram, or detect the presence of a horizon-like edge.

You may further explore scenarios for information retrieval in the cultural heritage domain. and compare this with other applications of multimedia information retrieval, for example monitoring in hospitals.

For further reading I suggest to make yourself familiar with common techniques in information retrieval as described in  [Baeza-Yates and Ribeiro-Neto (1999)], and perhaps devote some time to studying image analisis,  [Gonzales and Wintz (1987)].

the artwork

  1. artworks -- ..., Miro, Dali, photographed from Kunstsammlung Nordrhein-Westfalen, see artwork 2.
  2. left Miro from  [Kunst], right: Karel Appel
  3. match of the day (1) -- Geert Mul
  4. match of the day (2) -- Geert Mul
  5. match of the day (3) -- Geert Mul
  6. mario ware -- taken from gammo/veronica.
  7. baten kaitos -- eternal ways and the lost ocean, taken from gammo/veronica.
  8. idem.
  9. PANORAMA -- screenshots from field test.
  10. signs -- people,  [ van Rooijen (2003)], p. 252, 253.

video annotation requires a logical approach to story telling

content annotation

learning objectives

After reading this chapter you should be able to explain the difference between content and meta information, to mention relevant content parameters for audio, to characterize the requirements for video libraries, to define an annotation logic for video, and to discuss feature extraction in samples of musical material.

Current technology does not allow us to extract information automatically from arbitrary media objects. In these cases, at least for the time being, we need to assist search by annotating content with what is commonly referred to as meta-information.

In this chapter, we will look at two more media types, in particular audio and video. Studying audio, we will learn how we may combine feature extraction and meta-information to define a data model that allows for search. Studying video, on the other hand, will indicate the complexity of devising a knowledge representation scheme that captures the content of video fragments.

Concluding this chapter, we will discuss an architecture for feature extraction for arbitrary media objects.

audio databases


  • audio signals -- compression, discrete representation
  • musical patterns -- similarity-based retrieval

audio data model


  • meta-data -- describing content
  • features -- using feature extraction

example



   singers -- (Opera,Role,Person)
   score -- ...
   transcript -- ...
  

signal-based content


  • audio data -- %F(x) over time x
  • wave -- period T, frequency f = 1/T
  • velocity -- v = w/T = w * f , with w wavelength
  • amplitude -- a

windowing


  • break signal up in small windows of time

feature extraction


  • intensity -- watts/m^2
  • loudness -- in decibels
  • pitch -- from frequency and amplitude
  • brightness -- amount of distortion

video annotation


  • what are the interesting aspects?
  • how do we represent this information?

video content



  video v, frame f 
  f has associated objects and activities 
  objects and activities have properties
  

property


  property: name = value 
  

object schema


   (fd,fi) -- frame-dependent and frame-independent properties 
  

object instance: (oid,os,ip)

example


frameobjectsframe-dependent properties
1Janehas(briefcase), at(path)
-housedoor(closed)
-briefcase
2Janehas(briefcase), at(door)
-Dennisat(door)
-housedoor(open)
-briefcase

frame-independent properties


objectframe-independent propertiesvalue
Janeage35
height170cm
houseaddress...
colorbrown
briefcasecolorblack
size40 x 31

activity

  • activity name -- id
  • statements -- role = v

example


   { giver : Person, receiver : Person, item : Object } 
   giver = Jane, receiver = Dennis, object = briefcase 
  

video libraries



  which videos are in the library 
  what constitutes the content of each video
  what is the location of a particular video
  

query language for video libraries


  • segment retrievals -- exchange of briefcase
  • object retrievals -- all people in v:[s,e]
  • activity retrieval -- all activities in v:[s,e]
  • property-based -- find all videos with object oid

VideoSQL



  SELECT -- v:[s,e] 
  FROM -- video:<source><V> 
  WHERE -- term IN funcall 
  

example



  SELECT  vid:[s,e]
  FROM video:VidLib
  WHERE (vid,s,e) IN VideoWithObject(Dennis) AND
  	object IN ObjectsInVideo(vid,s,e) AND
  	object != Dennis AND
  	typeof(object) = Person
  

To improve library access, the Informedia Digital Video Library uses automatic processing to derive descriptors for video. A new extension to the video processing extracts geographic references from these descriptors.

The operational library interface shows the geographic entities addressed in a story, highlighting the regions discussed in the video through a map display synchronized with the video display.

The map can also serve as a query mechanism, allowing users to search the terabyte library for stories taking place in a selected area of interest.

questions


  • what -- content-related
  • when -- position on time-continuum
  • where -- geographic location

More recently, it has been recognized that the process of spatialization -- where a spatial map-like structure is applied to data where no inherent or obvious one does exist -- can provide an interpretable structure to other types of data.

atlas of cyberspace


We present a wide range of spatializations that have employed a variety of graphical techniques and visual metaphors so as to provide striking and powerful images that extend from two dimension 'maps' to three-dimensional immersive landscapes.

feature grammar



  
  detector song; ## to get the filename
  detector lyrics; ## extracts lyrics
  detector melody; ## extracts melody
  detector check;  ## to walk the tree
  
  atom str name;
  atom str text;
  atom str note;  
  
  midi: song;
  
  song: file lyrics melody check;
  
  file: name;
  
  lyrics: text*;
  melody: note*;
  
  


  event('twinkle',2,time=384, note_on:[chan=2,pitch=72,vol=111]).
  event('twinkle',2,time=768, note_off:[chan=2,pitch=72,vol=100]).
  

melody detector



  int melodyDetector(tree *pt, list *tks ){
  char buf[1024]; char* _result;
  void* q = _query;
  int idq = 0; 
  
    idq = query_eval(q,"X:melody(X)");
    while ((_result = query_result(q,idq)) ) {
           putAtom(tks,"note",_result);
           }
    return SUCCESS;
  } 
  

prediction techniques


  • social-based -- dependent on (group) rating of item(s)
  • information-based -- dependent on features of item(s)
  • hybrid methods -- combining predictors

definition(s)


  • rating -- a value representing a user's interest
  • recommendation -- item(s) that might be of interest to the user
  • regret -- a function to measure the accuracy of recommendations

guided tour(s)


  • automated (viewpoint) navigation in virtual space,
  • an animation explaining, for example, the construction of an artwork, or
  • the (narrative) presentation of a sequence of concept nodes.

projects & further reading

As a project, think of implementing musical similarity matching, or developing an application retrieving video fragments using a simple annotation logic.

You may further explore the construction of media repositories, and finding a balance between automatic indexing, content search and meta information.

For further reading I advice you to google recent research on video analysis, and the online material on search engines.

the artwork

  1. works from  [Weishar (1998)]
  2. faces -- from www.alterfin.org, an interesting site with many surprising interactive toys in flash, javascript and html.
  3. mouth -- Annika Karlson Rixon, entitled A slight Acquaintance, taken from a theme article about the body in art and science, the Volkskrant, 24/03/05.
  4. story -- page from the comic book version of City of Glass,  [Auster (2004)], drawn in an almost tradional style.
  5. story -- frame from  [Auster (2004)].
  6. story -- frame from  [Auster (2004)].
  7. story -- frame from  [Auster (2004)].
  8. white on white -- typographical joke.
  9. modern art -- city of light (1968-69), Mario Merz, taken from  [Hummelen and Sill\'e (1999)].
  10. modern art -- Marocco (1972), Krijn Griezen, taken from  [Hummelen and Sill\'e (1999)].
  11. modern art -- Indestructable Object (1958), Man Ray, Blue, Green, Red I (1964-65), Ellsworth Kelly, Great American Nude (1960), T. Wesselman, taken from  [Hummelen and Sill\'e (1999)].
  12. signs -- sports,  [ van Rooijen (2003)], p. 272, 273.

effective retrieval requires visual interfaces

information system architecture

learning objectives

After reading this chapter you should be able to dicuss the considerations that play a role in developing a multimedia information system, characterize an abstract multimedia data format, give examples of multimedia content queries, define the notion of virtual resources, and discuss the requirements for networked virtual environments.

From a system development perspective, a multimedia information system may be considered as a multimedia database, providing storage and retrieval facilities for media objects. Yet, rather than a solution this presents us with a problem, since there are many options to provide such storage facilities and equally many to support retrieval.

In this chapter, we will study the architectural issues involved in developing multimedia information systems, and we will introduce the notion of media abstraction to provide for a uniform approach to arbitrary media objects.

Finally, we will discuss the additional problems that networked multimedia confront us with.

issues


  • multimedia storage and retrieval -- homegrown, third-party and legacy sources
  • information architecture -- common format, native format, hybrid
  • media abstraction -- unified indexes, query relaxation

content organisation


  • autonomy -- index per media type
  • uniformity -- unified index
  • hybrid -- media indexes + unified index

Principle of Uniformity


... from a semantical point of view the content of a multimedia source is independent of the source itself, so we may use statements as meta data to provide a description of media objects.

  • from a semantical point of view the content of a multimedia source is independent of the source itself.
  • use statements as meta data
  • md(o) -- metadata associated with media object o

tradeoffs


  • metadata can be stored using standard relational and OO structures
  • manipulating metadata is easy
  • feature extraction is (!) straightforward

    is it?


software architecture


  • a database of media object, supporting
  • operations on media objects, and offering
  • logical views on media objects

information retrieval cycle


  1. specification of the user's information need
  2. translation into query operations
  3. search and retrieval of media objects
  4. ranking according to likelihood or relevance
  5. presentation of results and user feedback
  6. resulting in a possibly modified query

  • despite high interactivity, access is difficult;
  • quick response is and will remain important!

media abstraction


  • state -- smallest chunk of media data
  • feature -- any object in a state
  • attributes -- characteristics of objects
  • feature extraction map -- to identify content
  • relations -- to capture state-dependent information
  • (inter)relations between 'states' or chunks

example -- image database



  states: { pic1.gif,...,picn.gif } 
  features: names of people 
  extraction: find people in pictures 
  relations: left-of, ... 
  

example -- video database



  states:  set of frames 
  features:  persons and objects
  extraction:  gives features per frame 
  relations:  frame-dependent and frame-independent information
  inter-state relation:  specifies sequences of frames
  

simple multimedia database


  • a finite set M of media abstractions

structured multimedia database


  • equivalence relations --to deal with synonymy
  • partial ordering -- to deal with inheritance
  • query relaxation -- to please the user

SMDS -- functions



  Type: object  |->  type 
  ObjectWithFeatures:  f |-> { o |  object o contains  f }  
  ObjectWithFeaturesAndAttributes:  (f,a,v) |-> { o |  o contains f with  a=v }  
  FeaturesInObject:  o |-> { f | o  contains  f }  
  FeaturesAndAttributesInObject:  o |-> { (f,a,v) | o  contains  f  with  a=v }  
  

SMDS-SQL



SELECT -- media entities
  • m -- if m is not a continuous media object
  • m:[i,j] -- m is continuous, i,j integers (segments)
  • m.a -- m is media entity, a is attribute

FROM

  • <media><source><M>

WHERE

  • term IN funcall

example



    SELECT M
    FROM   smds source1 M
    WHERE  Type(M) = Image AND
  	 M IN ObjectWithFeature("Dennis") AND
  	 M IN ObjectWithFeature("Jane") AND
  	 left("Jane","Dennis",M)
  

hybrid representations: HM-SQL


  • express queries in specialized language
  • perform operations (joins) between SMDS and non-SMDS data

differences


  • function calls are annotated with media source
  • queries to non-SMDS data may be embedded

example HM-SQL



   SELECT M
   FROM smds video1, videodb video2
   WHERE M IN smds:ObjectWithFeature("Dennis") AND
         M IN videodb:VideoWithObject("Dennis")
  

digital libraries


Digital libraries are constructed -- collected and organized -- by a community of users. Their functional capabilities support the information needs and users of this community. Digital libraries are an extension, enhancement and integration of a variety of information institutions as physicalplaces where resources are selected, collected, organized, preserved and accessed in support of a user community.

... federated structures that provide humans both intellectual and physical access to the huge and growing worldwide networks of information encoded in multimedia digital formats.

digital libraries (5S)


  • streams: (content) -- from text to multimedia content
  • structures: (data) -- from database to hypertext networks
  • spaces: (information) -- from vector space to virtual reality
  • scenarios: (procedures) -- from service to stories
  • societies: (stakeholders) -- from authors to libraries


   D-Lib Forum -- www.dlib.org
   Informedia -- www.informedia.cs.cmu.edu
  

networked multimedia


  • real-time transmission of continuous media information (audio, video)
  • substantial volumes of data (despite compression)
  • distribution-oriented -- e.g. audio/video broadcast

network criteria


  • throughput -- bitrates, burstiness
  • transmission delay -- including signal propagation time
  • delay variation -- jitter
  • error rate -- data alteration, loss

  • multicasting and broadcasting capabilities
  • document caching

Quality of Service


Quality of Service is a concept based on the statement that not all applications need the same performance from the network over which they run. Thus, applications may indicate their specific requirements to the network, before they actually start transmitting information data.

QoS requirements


  • hard requirements
  • guidance for optimizing internal resources
  • criteria for acceptance

virtual objects

  • VO = { (O_i,Q_i,C_i) | 1 <= i <= k }

where

  • C_1,...,C_k -- mutually exclusive conditions
  • Q_1,...,Q_k -- queries
  • O_1,...,O_k -- objects

networked virtual environments


  • shared sense of space -- room, building, terrain
  • shared sense of presence -- avatar (body and motion)
  • shared sense of time -- real-time interaction and behavior

  • a way to communicate -- by gesture, voice or text
  • a way to share ... -- interaction through objects

challenges


  • network bandwidth -- limited resource
  • heterogeneity -- multiple platforms
  • distributed interaction -- network delays
  • resource management -- real-time interaction and shared objects
  • failure management -- stop, ..., degradation
  • scalability -- wrt. number of participants

manage dynamic shared state

  • the Java Media Framework, and
  • the DLP+X3D platform

java Media Framework


The JavaTM Media APIs meet the increasing demand for multimedia in the enterprise by providing a unified, non-proprietary, platform-neutral solution. This set of APIs supports the integration of audio and video clips, animated presentations, 2D fonts, graphics, and images, as well as speech input/output and 3D models. By providing standard players and integrating these supporting technologies, the Java Media APIs enable developers to produce and distribute compelling, media-rich content.

recommender economy


  • cross sale -- users who bought A also bought B
  • up sale -- if you buy A and B together ...

recommender model



  U = user
  I = item
  B = behavior
  R = recommendation
  F = feature
  
  • observations -- U \* I \* B
  • recommendations -- U \* I

  B = [ time = 20sec, rating = r ]
  F = [ artist = rembrandt, topic = portrait ]
  R = [ artist(rembrandt) = r, topic(portrait) = r ]
  

  A = [  p_{1}, p_2 , ... ]
  where p_{k} = [ f_1 = v_1, f_2 = v_2, ... ]
  
with as an example

   A_{nightwatch} = [ artist=rembrandt, topic=group ]
   A_{guernica} = [ artist=picasso, topic=group ]
  

distance metric



       d(x,y) = d(y,x)
       d(x,y) <= d(x,z) + d(z,y)
       d(x,x) = 0
  

dimension(s)


  • positive vs negative
  • individual vs community/collaborative
  • feature-based vs item-based

interpretation(s)


  • neutral interpretation -- use d(s_{n}, a_{k}) < d(s_{n}, s_{n+1} )
  • positive interpretation -- increase w(feature(a_{k}))
  • negative interpretation -- decrease w(feature(s_{n+1}))

projects & further reading

As a project, you may implement a multi-player game in which you may exchange pictures and videos, for example pictures and videos of celebrities.

Further you may explore the development of a data format for text, images and video with appropriate presentation parameters, including postioning on the screen and intermediate transitions.

For further reading you may study information system architecture patterns, nd explore the technical issues of constructing server based advanced multimedia applications in  [Li and Drew (2004)].

the artwork

  1. examples of dutch design, from  [Betsky (2004)].
  2. idem.
  3. screenshots -- from splinter cell: chaos theory, taken from Veronica/Gammo, a television program about games.
  4. screenshots -- respectively Sekken 5, Sims 2, and Super Monkey Ball, taken from insidegamer.nl.
  5. screenshots -- from Unreal Tournament, see section 7.3.
  6. idem.
  7. idem.
  8. resonance -- exhibition and performances, Montevideo, april 2005.
  9. CHIP -- property diagram connecting users.
  10. signs -- sports,  [ van Rooijen (2003)], p. 274, 275.

a journey of a thousand miles begins with the first step

chinese/japanese proverb

perspectives -- multimedia applications


  • technical -- algorithmic effects
  • sociological -- stakeholders and teamwork
  • tool selection -- Maya vd 3DSMAX
  • political -- negotiating support
  • scientific -- experience design
  • computer science -- tools and technologies
  • artistic -- portfolio as a design product

the artwork

  1. walking figure -- sculpture by Alberto Giacometti,  [Hohl (1971)].
  2. signs -- meteorological symbols,  [ van Rooijen (2003)], p. 214, 215.
  3. photographs -- Jaap Stahlie, commissioned work, using traditional non-digital techniques.

augmented virtuality acts as an intelligent looking glass

virtual environments

learning objectives

After reading this chapter you should be able to characterize the notion of virtual context, discuss the issue of information retrieval in virtual environments, explain what is meant about intelligent multimedia and discuss the potential role of intelligent agents in multimedia applications.

From a user perspective, virtual environments offer the most advanced interface to multimedia information systems. Virtual environments involve the use of (high resolution) 3D graphics, intuitive interaction facilities and possibly support for multiple users.

In this chapter, we will explore the use of (desktop) virtual environments as an interface to (multimedia) information systems. We will discuss a number of prototype implementations illustrating, respectively, how paintings can be related to their context, how navigation may be seen as a suitable answer to a query, and how we can define intelligent agents that can interact with the information space. Take good notice, the use of virtual environments as an interface to information systems represents a major challenge for future research!

Dam Square, Amsterdam


  • it is a 'real world' environment
  • it has 700 years of (recorded) history
  • it has a fair amount of historical buildings
  • buildings and street life change over time

how can we give access to the 'Dam square' information space

virtual context


  • VR model of Dam square
  • selection of related paintings fromRoyalMuseum
  • viewpoint adjustment, to match painting
  • (transparent) overlay of paintings over buildings

augmented virtual reality


  • give user sense of geographic placement of buildings
  • show how multiple objects in a museum relate to eachother
  • show what paintings convey about their subject, and how

problems


  • organised guided tours
  • account for buildings that no longer exist
  • change temporal context
  • allow user queries as input

VRML


  • declarative means for defining geometry and appearance
  • prototype abstraction mechanism
  • powerful event model
  • relatively strong programmatic capabilities

virtual archeology


  • variety of archeological sites
  • various paths through individual site
  • reconstruction of 'lost' elements
  • 'discovery' of new material
  • glossary -- general background knowledge

RIF


Retrieval of Information in Virtual Worlds using Feature Detectors

the map CWI

the model

the query
C O F F E E

navigation
explore discover

explore and discover

what are we searching for?

what are we searching for?

types of information

  • viewpoints
  • areas of interest
  • objects
  • persons
  • text

availability

  • static -- always
  • shared -- users
  • dynamic -- runtime
  • temporal -- events
  • hidden -- actions

scanning the scenegraph

  • annotations
  • node types
  • textual content
  • materials
  • textures
  • geometry

choose a metaphor

choose a metaphor

get a viewpoint

  • viewpoints
  • areas of interest
  • objects and persons

answer the query

  • route planning
  • viewpoint transformation

walking

walking

assumptions

  • explicit annotation
  • map for route planning
  • keyword matching

requirements

  • database -- annotations & map
  • 3D (pseudo-immersive) interface

the system

relaxing the assumptions

  • annotation -- incremental and/or automatic
  • (explicit) maps -- based on user navigation
  • (keyword) matching -- text retrieval

...

  • navigation by query is feasible and may help users to find locations and objects
  • determining suitable navigation routes without an explicitly defined map is hard

future work

  • shift in focus -- intelligent agents
  • DLP + VRML -- distributed logic programming

Web Agent Support Program


DEMO

no database, no walking

3D GUI


Wishful thinking about the widespread adoption of three-dimensional interfaces has not helped spawn winning applications. Success stories with three-dimensional games do not translate into broad acceptance of head-tracking immersive virtual reality. To accelerate adoption of advanced interfaces, designers must understand their appeal and performance benefits as well as honestly identify their deficits. We need to separate out the features that make 3D useful and understand how they help overcome the challenges of dis-orientation during navigation and distraction from occlusion.

Ben Shneiderman

Does spatial memory improve with 3D layouts? Is it true that 3D is more natural and easier to learn? Careful empirical studies clarify why modest aspects of 3D, such as shading for buttons and overlapping of windows are helpful, but 3D bar charts and directory structures are not. 3D sometimes pays off for medical imagery, chemical molecules, and architecture, but has yet to prove beneficial for performance measures in shopping or operating systems.

Ben Shneiderman

WASP


Web Agent Support Program

DLP


Distributed Logic Programming

RIF + WASP


  • distributed logic programming -- uniform platform
  • agent technology -- subsumes multi-user server

multi-user soccer game


  • multiple (human) users -- may join during the game
  • multiple agents -- to participate in the game (e.g. as goalkeeper)
  • reactivity -- players (users and agents) have to react quickly
  • cooperation/competition -- requires 'intelligent' communication
  • dynamic behavior -- sufficiently complex 3D scenes, including the dynamic behavior of the ball

control points


  • get/set -- position, rotation, viewpoint

agents in virtual environments


  • virtual environments with embedded autonomous agents
  • virtual environments supported by ACL communication

Living Worlds


  • scene -- geometrically bounded, continuously navigable
  • world -- collection of (linked) scenes

Shared Object


  • pilot -- instance that will be replicated
  • drone -- instance that replicates pilot

  • pilot agents -- control state of a shared object
  • drone agents -- replicate the state of a shared object

  • object agents -- controls a single shared object (like the soccerball) pilot at server, drone at client
  • controls users' avatar pilot at user side, drone at server or clients
  • autonomous agents -- like football player, with own avatar pilot at server, drone at clients

programming platform


  • VRML EAI support
  • distributed communication capabilities (TCP/IP)
  • multiple threads of control -- for multiple shared objects
  • declarative language -- for agent support

taxonomy of agents


  • 2D/3D -- to distinguish between text-based and avatar embodied agents
  • client/server -- to indicate where agents reside
  • single/multi -- as a measure of complexity

PAMELA


Personal Assistent for Multimedia Electronic Archives

  • autonomous and on-demand search capabilities
  • (user and system) modifiablepreferences
  • multimedia presentation facilities

H-Anim


  • control points -- joints, limbs and facial features

presentation agent


Given any collection of results, PAMELA could design some spatial layout and select suitableobject types, including for example color-based relevance cues, to present the results in a scene. PAMELA could then navigate you through the scene, indicating the possible relevance of particular results.

persuasion games


  • single avatar persuasive argumentation
  • multiple avatar dialog games

PAMELA


Persuasive Agent with Multimedia Enlightened Arguments

A variety of applications may benefit from deploying embodied conversational agents, either in the form of animated humanoid avatars or, more simply, as a 'talking head'. An interesting example is provided by Signing Avatar, a system that allows for translating arbitrary text in both spoken language and sign language for the deaf, presented by animated humanoid avatars.

Here the use of animated avatars is essential to communicate with a particular group of users, using the sign language for the deaf.

STEP


  • convenience -- for non-professional authors
  • compositional semantics -- combining operations
  • re-definability -- for high-level specification of actions
  • parametrization -- for the adaptation of actions
  • interaction -- with a (virtual) environment

DLP+X3D


The DLP+X3D platform provides together with the STEP scripting language the computational facilities for defining semantically meaningful behaviors and allows for a rich presentational environment, in particular 3D virtual environments that may include streaming video, text and speech.

evaluation criteria

  • effective communication

conversational agents in VR


  • presentational VR, instructional VR, educational VR

applications


  • information agents, presentation agents

system perspective


  • range of agent categories, open standards

user perspective


  • naturalness -- contextual & emotional

initial target(s)


  • build initial (throwaway) prototype
  • explore content creation technology
  • create tutorial(s) for content contribution
  • analyse technological requirements

The history of Second Life is extensively descibed in the official Second Life guide,  [Rymaszweski et al. (2007)]. Beginning 2004, almost out of the blue, Second Life appeared with a high adoption and low churn rate, now counting, March 2007, over 4 million inhabitants. Considering the cost of ownership of land, which easily amounts to 200 euro per month rent after an initial investment of 1500 euro for a single piece of land measuring 65,536 square meters, the adoption of Second Life by individuals as well as companies such as ABN-AMRO, Philips and institutions such as Harvard is surprising.

What is the secret of the success of Second Life?, we asked in  [Eliens et al. (2007)], and we immediately confessed: We don't know! But in comparison to other platforms for immersive worlds, including MMORPGs such as World of Warcraft and Everquest, Second Life seems to offer an optimal combination of avatar modification options, gesture animations, in-game construction tools, and facilities for communication and social networking, such as chatting and instant messaging. Incorporating elements of community formation, commonly denoted as Web 2.0, and exemplified in MySpace, YouTube and Flickr, the immersive appearance, perhaps also the built-in physics and the inclusion of elementary economic principles, seem to be the prime distinguishing factors responsible for the success of Second Life. In addition, the possibility of recording collaborative enacted stories,  [Stories], using built-in machinima certainly also contributes to its appeal.

What has been characterized as a shift of culture, from a media consumer culture to a participatory culture,  [Jenkins (2006)], where users also actively contribute content, (was) for our institution one of the decisive reasons to create a presence in Second Life, to build a virtual platform that may embody our so-called community of learners, where both staff and students cooperate in contributing content, content related to our sciences, that is.

The first idea that comes to mind, naturally, is to use Second Life to offer courses online. But, although we did have plans to give lectures (college) on law, probably including the enactment of a particular case, we did consider this approach as rather naive, and frankly I see no reason to include what may be considered an outdated paradigm of learning in our virtual campus, where there might be more appealing alternatives. Similarly, using the virtual laboratory for experiments might not be the best way to offer courses, although, again, we do intend to provide a model of a living cell, allowing students to study the structure, functionality and behavior of organic cells in virtual space.

active learning


  • experience the world in new ways
  • form new affiliations
  • preparate for future learning

projects & further reading

As a project, I suggest the implementation of storytelling in virtual environments, with (possibly) an embodied agent as the narrator.

You may further explore or evaluate the role of agents in multimedia applications and virtual environments.

For further reading in (real) VR, I advice  [Sherman and Craig (2003)], and for gaining an understanding in story telling and applications you may try to get hold of the proceedins, of TIDSE 2003, and TIDSE 2004.

the artwork

  1. another series of dutch light.
  2. virtual context -- Dam Square, Amsterdam, see 8.1.
  3. VU Campus in VRML -- student project.
  4. CWI 3th floor, floormap and model, see 8.2..
  5. query -- on 3th floor of CWI.
  6. navigation -- on 3th floor of CWI.
  7. soccer game -- image from WASP project, see section 8.3.
  8. digital beauties -- taken from  [Wiedermann (2002)].
  9. digital beauties -- taken from  [Wiedermann (2002)].
  10. VU @ Second Life -- screenshots.
  11. signs -- sports,  [ van Rooijen (2003)], p. 276, 277.

man is a playful animal

johan huizinga

perspectives -- game development


  • artistic -- plot, narrative, style
  • technical -- choice of game engine (SDK)
  • sociological -- sharing within game communities
  • tool selection -- supporting the workflow
  • commercial -- success factors
  • management -- teamwork

the artwork

  1. manuscript -- used as a desktop by my favorite student.
  2. signs -- abstract symbols,  [ van Rooijen (2003)], p. 214, 215.
  3. photographs -- Jaap Stahlie, commissioned work.

media equation(s) 1/4


We regularly exploit the media equation for enjoyment by the willing suspension of our critical faculties. Theatre is the projection of a story through the window of a stage, and typically the audience gets immersed in the story as if it was real.

multimedia equation(s) 2/4


multimedia = presentation + context

multimedia equation(s) 3/4


  • context = convergence + information + architecture

where

multimedia equation(s) 4/4


  • convergence = data +platform + distribution
  • information = storage and retrieval
  • architecture = compression + components + connectivity

perfect solutions


Much more than the art of turning base metals into gold, alchemy is a system of cosmic symbolism. The alchemist learns how to create within a sealed vessel a Model of the Universe in which the opposing complementary forces of Male and Female, Earth and Air, Fire and Water attain the perfect synthesis of which gold is the emblem.

multimedia engineering


"engineering is the art of moulding materials we do not wholly understand ... in such a way that the community at large has no reason to suspect the extent of our ignorance."

multimedia courses


  • multimedia authoring -- Web3D/VRML
  • intelligent multimedia technology -- Virtual Environments
  • visual design -- digital content creation
  • multimedia casus -- digital dossier(s)

topical


  1. designed for involving local application (as an anesthetic),
  2. relating to, or arranged by, topics,
  3. referring to the topics of the day or place.

content(s) / list(s) / online(s) / web3d / resource(s) / tool(s) / technology / ? / zaal

resource(s) -- design

resource(s) -- game(s)

content(s) / list(s) / online(s) / web3d / resource(s) / tool(s) / technology / ? / zaal

imaging and graphics

3D modeling

Alias Wavefront Maya

Discreet 3D Studio Max

content(s) / list(s) / online(s) / web3d / resource(s) / tool(s) / technology / ? / zaal

DirectX SDK 9

Wild Tangent

Virtools Software Suite

OpenML

open source technology

outline



     title -- indicating the topic
     name  -- to tell who you are
     abstract -- giving the 'message' of your efforts
     introduction --  clarifying the approach and structure
     background -- explaining the context of the subject
     sections -- to elaborate on the subject
     related work -- characterizing related approaches
     conclusion(s) -- summarizing the main point(s)
     references -- listing the literature you consulted
     appendices (optional) -- providing extra information
  

perspective(s)


  • review/background -- sketch perspectives, history, viewpoints
  • case study -- analyse assumptions, gather empirical data, and explain!
  • technical analysis -- technology-oriented, work out the details
  • formal study -- clarify in a formal manner, conceptualize and formalize
  • tutorial -- explain for the laymen, but do it very good

authoring convergence standards retrieval
review/background -+++++
case study ++++
technical analysis -++++++
formal study --++-
tutorial ---?-

title(s) / web3d


select:

references

[Role] Adams J. (2002),
Programming Role Playing Games with DirectX, Premier Press
[Animation] Adams J. (2003),
Advanced Animation with DirectX, Premier Press
[Perspective] Alberti L.B. (1435),
On painting and on sculpture, Phaidon, edited by C. Grayson, 1972
[Multiplayer] Alexander T., ed. (2005),
Massively Multiplayer Game Development 2, Charles River Media
[Cyberspace] Anders P. (1999),
Envisioning Cyberspace -- Designing 3D Electronic Spaces, McGraw-Hill
[Angel] Angel E. (1997),
Interactive Computer Graphics -- A top-down approach with OpenGL, Addison-Wesley
[Film] Arnheim R. (1957),
Film as Art, The University of California Press
[Collada] Arnaud R, and Barnes M.C. (2006),
Collada -- sailing the gulf of 3D digital content creation, A.K. Peters Ltd.
[Enhancement] Astleneir H. (2000),
Designing emotionally sound instruction: the FEASP approach, Instructional Science 28, pp. 169-198
[Framework] Atanasova T., Nern H.J., Dziech A. (2007),
Framework Approach for Search and Meta-Data Handling of AV Objects in Digital TV Cycles, Workshop on Digital Television, Proc. EUROMEDIA 2007, Delft, Netherlands
[Glass] Auster P. (2004),
City of Glass, Faber and Faber, adaptation by Karasik P. and Mazzucchelli D.
[PAR] Badler N., Bindiginavale R., Bourne J., Palmer M., Shi J., Schuler W. (1998),
A Parameterized Action Representation for Virtual Human Agents, Workshop on Embodied Conversational Characters, WECC98, Lake Tahoe, CA,Oct 12-15, 1998.
[IR] Baeza-Yates R. and Ribeiro-Neto B. (1999),
Modern Information Retrieval, Addison-Wesley, 1999
[Query] Ballegooij A. van and Eliens A. (2001),
Navigation by Query in Virtual Worlds, Web3D 2001 Conference, Paderborn, Germany, 19-22 Feb 2001
[Archeology] Barcelo S.A., Forte M., Sanders D.H., eds. (2000),
Virtual Reality in Archeology, Bar International Series 843, 2000
[Strategy] Barron T. (2003),
Strategy Game Programming with DirectX 9.0, Wordware Publishing
[Impasto] Baxter W., Wendt J., and Lin M. (2004),
IMPaSTo: A Realistic, Interactive Model for Paint, Proc. of NPAR 2004, The 3rd Int. Symp. on Non-Photorealistic Animation and Rendering. June 7-9 2004, Annecy, France. pp. 45--56.
[Reproduction] Benjamin W. (1936),
The Work of Art in the Age of Mechanical Reproduction, Online Archive -- http://www.marxists.org
[Web] Berners-Lee T., Hendler J., Lassila O. (2001),
The semantic web, Scientific American, may 2001, pp. 28-37
[GamePatterns] Björk S. and Holopainen J. (2005),
Patterns in Game Design, Charles River Media
[Flat] Betsky A. with Eeuwens A. (2004),
False Flat -- Why Dutch Design is so Good, Phaidon
[Vox] Bocconi S. (2006),
Vox Populi: generating video documentation from semantically annotated data repositories, Ph.D. Thesis SIKS 2006-27
[Remediation] Bolter J.D and Grusin R. (2000),
Remediation -- Understanding New Media, MIT Press
[Monet] Boncz P.A. and Kersten M.L. (1995),
Monet -- an impressionist sketch of an advanced database system, In Proc. BIWIT95
[Interactivation] Bongers B. (2006),
Interactivation -- Towards an e-cology of people, our environment, and the arts, Ph.D. Thesis, Vrije Universiteit
[FilmTheory] Bordwell D. and Thomson K. (2003),
Film art, an introduction, McGraw-Hill, 7th edn.
[History] Briggs A. and Burke P. (2001),
A social history of the media -- from Gutenberg to the Internet , Polity Press
[Environments] Broll W. (1996),
VRML and the Web: A basis for Multi-user Virtual Environments on the Internet, In Proceedings of WebNet96, H. Maurer (ed.), AACE, Charlottesville, VA (1996), 51-56.
[Angelic] Broll W., Shäfer L., Höllerer T., Bowman D. (2001),
Interface with Angels: the future of VR and AR interfaces, IEEE Computer Graphics, November/December 2001, pp. 14-17
[Education] Bruner J.S. (1972),
Relevance of Education, Penguin Education
[Avantgarde] Burger P. (1981),
Theorie de Avantgarde, Edition Suhrkamp
[Bush] Bush V. (1945),
As we may think, Atlantic Monthly, July 1945
[Next] CACM 44:3 (2001),
The next 1000 years, Communications of the ACM, March 2001, 44:3
[ Capin T.K., Pandzic I.S., Magnenat-Thalmann N., Thalman D. (1999),
Avatars in Networked Virtual Environments, Wiley
[Annotated] Carey R. and Bell G. (1997),
The Annotated VRML 2.0 Reference Manual, Addison-Wesley
[Prediction] Cesa-Bianchi N. and Lugosi G. (2006),
Prediction, Learning, and Games, Cambridge University Press
[Spaces] Chang S.C. and Costabile M.F. (1997),
Visual Interfaces to Multimedia Databases, In [_Handbook]
[Chapman1] Chapman N. and Chapman J. (2004a),
Digital Multimedia, Wiley, 2nd edn.
[Chapman2] Chapman N. and Chapman J. (2004b),
Digital Media Tools, Wiley, 2nd edn.
[Beauty] Cheng F. (2006),
Five meditations on beauty (in french), Editions Albin Michel
[Video] Christel, M., Olligschlaeger, A., Huang, C. (2000),
Interactive maps for digital video, IEEE Multimedia 7(1), pp. 60-67
[Collaborative] Churchill E.F., Snowdon D.N. and Munro A.J., eds. (2001),
Collaborative Virtual Environments -- Digital Places and Spaces for Interaction, Springer
[Modeling] Conger D. (2004),
Physics Modeling, Thomson
[Hypertext] Conklin J. (1987),
Hypertext: An Introduction and Survey, IEEE Computer 20(9), pp. 17-41
[Design] Danaher S. (2004),
Digital 3D Design, Thomson
[DTV] Daskalova H. and Atanasova T. (2007),
Web Services and Tools for their Composition considering Aspects of Digital TV Workflow, Workshop on Digital Television, Proc. EUROMEDIA 2007, Delft, Netherlands
[Entertainment] Davenport G.(2000),
Your own virtual story world, Scientific American, november 2000, pp. 61-64
[Survey] Davison A. (1993),
A survey of Logic Programming-based Object-Oriented Languages, In Research Directions on Concurrent Object-Oriented programming, G. Agha, P. Wegner, A. Yonezawa (eds.), MIT Press (1993)
[Scripts] Davison A. (2001),
Enhancing VRML97 Scripting, Euromedia\'2001, Valencia, Spain, April 18-20. available from: http://fivedots.coe.psu.ac.th/~ad
[Logic] Davison A. (2001b),
Logic Programming Languages for the Internet, Chapter in Computational Logic: From Logic Programming into the Future, Antonis Kakas, Fariba Sadri (eds.), Springer Verlag (2001). available from: http://fivedots.coe.psu.ac.th/~ad
[Pragmatics] Dewey J. (1931),
Art as Experience, Ref ???
[Vasarely] Diehl G. (1973),
Vasarely, Crown Publishers Inc.
[Distributed] Diehl S. (2001),
Distributed Virtual Worlds -- Foundations and Implementation Techniques using VRML, Java and Corba, Springer
[Reactions] Dijkstra K., Zwaan R. Graesser A.and J. Magliano (1994),
Character and reader emotions in literary texts, Poetics 23, pp. 139-157
[Dix] Dix A., Finlay J., Abowd G., Beale R. (1998),
Human-Computer Interaction (1998), Prentice Hall, 2nd edn.
[Maps] Dodge M. and Kitchin R. (2000),
Mapping Cyberspace, Routledge
[Atlas] Dodge M. and Kitchin R. (2002),
Atlas of Cyberspace, Addison-Wesley
[Humour] Dormann C., Barr P. and Biddle R. (2007),
Humour Theory and Videogames: Laughter in the Slaughter, to appear in XXX
[Virtual] Earnshaw R. and Vince J., eds. (2002),
Intelligent Agents for Mobile and Virtual Media, Springer
[Engine] Eberly D.H. (2001),
3D Game Engine Design, Morgan Kaufmann Publishers
[Physics] Eberly D.H. (2004),
Game Physics, Morgan Kaufmann Publishers
[Eco] Eco U. (1994),
Six walks in the fictional woods, Harvard University Press
[Creativity] Eliens A. (1979),
Creativity: reflection and involvement, Ninth Int Conf of Aesthetics, Dubrovnic, August 1979
[Art] Eliens A. (1988),
Computational Art, First Int Symposium on Electronic Art, Leonardo Supplementary Issue, Pergamon Press 1988, pp. 21-26
[DLP] Eliens A. (1992),
DLP -- A language for Distributed Logic Programming, Wiley
[Workshop] Eliens A. (1996),
workshop: Logic Programming and the Web, \\ http://www.cs.vu.nl/~eliens/online/workshops/www6
[Jamming] Eliens A., Welie M. van, Ossenbruggen J. van and Schonhage S.P.C (1997) ,
Jamming (on) the Web, In: Proceedings of the 6th International World Wide Web Conference --- Everone, Everything Connected, O\'Reilly and Associates, Inc. April 1997, pp. 419-426
[Simulate] Eliens A., Niessink F., Schonhage S.P.C., van Ossenbruggen J.R., Nash P. (1996),
Support for BPR -- simulation, hypermedia and the Web, Proceedings Euromedia\'96, Euromedia, London 1996
[OO] Eliens A. (2000),
Principles of Object-Oriented Software Development, Addison-Wesley Longman, 2nd edn.
[IMVU] Eliens A. (2002),
intelligent multimedia @ VU, http://www.cs.vu.nl/~eliens/research
[Platform] Eliens A., Huang Z., and Visser C. (2002),
A platform for Embodied Conversational Agents based on Distributed Logic Programming, AAMAS Workshop -- Embodied conversational agents - let\'s specify and evaluate them!, Bologna 17/7/2002
[TIDSE] Eliens A., Dormann C., Huang Z. and Visser C. (2003),
A framework for mixed media -- emotive dialogs, rich media and virtual environments, Proc. TIDSE03, 1st Int. Conf. on Technologies for Interactive Digital Storytelling and Entertainment, Göobel S. Braun N.,n Spierling U., Dechau J. and Diener H. (eds>), Fraunhofer IRB Verlag, Darmstadt Germany, March 24-26, 2003
[Slides] Eliens A., Huang Z., Visser C. (2005),
Presentational VR -- What is the secret of the slides?, in preparation
[Navigate] Eliens A., van Riel C., Wang Y. (2006),
Navigating media-rich information spaces using concept graphs -- the abramovic dossier, accepted for: International Conference on Multidisciplinay Information Sciences and Technologies (InSciT2006) October, 25-28th 2006, Merida, Spain www.instac.es/inscit2006
[Odyssee] Eliens A. (2006b),
Odyssee -- explorations in mixed reality theatre, accepted for: GAME'ON-NA'2006, September 19-21, 2006 - Naval Postgraduate School, Monterey, USA
[VULife] Eliens A., S.V. Bhikharie (2006),
game @ VU -- developing a masterclass for high-school students using the Half-life 2 SDK, GAME'ON-NA'2006, September 19-21, 2006 - Naval Postgraduate School, Monterey, USA
[Serious] Eliens A. and Chang T. (2007),
Let's be serious -- ICT is not a (simple) game, In Proc. FUBUTEC 2007, April 2007, Delft
[VUSL] Eliens A. Feldberg F., Konijn E., Compter E. (2007) ,
VU @ Second Life -- creating a (virtual) community of learners, In Proc. EUROMEDIA 2007, Delft, Netherlands
[Recommend] Eliens A. and Wang Y. (2007),
Expert advice and regret for serial recommenders, In Proc. EUROMEDIA 2007, Delft, Netherlands
[Climate] Eliens A., van de Watering M., Huurdeman H. Bhikharie W., Lemmers H., Vellinga P. (2007b),
Clima Futura @ VU -- communicating (unconvenient) science, submitted to GAME ON 2007
[Mashups] Eliens A. Feldberg F., Konijn E., Compter E. (2007c) ,
Mashups in Second Life @ VU, submitted to GAME ON 2007
[NewPanorama] Eliens A. and Vyas V. (2007) ,
Panorama -- explorations in the aesthetics of social awareness, submitted to GAME ON 2007
[Aesthetics] Eliens A., Vyas V. and van der Veer G. (2007d) ,
A theoretical foundation for the aesthetics of interaction and awareness -- making sense of the senses, in preparation
[Beginning] Engel W.F. (2003),
Beginning Direct3D Programming, Premier Press
[Shader1] Engel W.F., ed. (2004a),
Shader X2, Introduction & Tutorials with DirectX 9, Wordware Publishing
[Shader2] Engel W.F., ed. (2004b),
Shader X2 -- Shader Programming, Tips & Tricks with DirectX 9, Wordware Publishing
[Shader3] Engel W.F., ed. (2005),
Shader X3, Advanced Rendering with DirectX and OpenGL, Charles River Media
[Engelbart] Engelbart D. (1963),
A Conceptual Framework for the Augmentation of Man\'s Intellect, Vistas in Information Handling 1(9)
[Managing] England E. and Finney A. (1999),
Managing Multimedia -- project Management for Interactive Media, Addison-Wesley
[Metamorphose] Entabaclaz A. (2003),
Les metamorphoses d\'Ulysse -- reecritures de l\'Odyssee, Editions Flammarion, Paris
[ Ericson C. (2005),
Real-time Collision Detection, Morgan Kaufmann Publishers
[Black] Fan J., Ries E. and Tenitchi C. (1996),
Black Art if JAVA Game Programming, Waite Group Press
[Faulkner] Faulkner X. (2000),
Usability Engineering, MacMillan Press
[Audio] Fay T.M. with Selfon S. and Fay T.J. (2004),
DirectX Audio Exposed -- Interactive Audio Development, Wordware Publishing Inc.
[Cg] Fernando R. and Kilgard M.J. (2003),
The Cg Tutorial -- The Definitive Guide to Programming Real-Time Graphics, Addison-Wesley
[Future] Fischetti M. (2000),
The future of digital entertainment, Scientific American, november 2000, pp. 31-33
[Networked] Fluckiger R. (1995),
Understanding networked multimedia -- applications and technology, Prentice Hall, 1995
[Persuasive] Fogg B.J. (2003),
Persuasive Technology -- Using Computers to Change What We Think and Do, Morgan Kaufmann Publishers
[Graphics] Foley J.D. and van Dam A. (1982),
Fundamentals of Interactive Computer Graphics, Addison-Wesley
[Convergence] Forman P. and Saint John R.W. (2000),
Digital Convergence, Scientific American, november 2000, pp. 34-40
[Witz] Freud S. (1958),
Der Witz und seine Beziehung zum Unbewussten, Fischer Verlag
[Dolores] Fuhr, N., Gövert, N., Rölleke, Th. (1998),
DOLORES: A System for Logic-Based Retrieval of Multimedia Objects, In: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 257-265. ACM, New York
[Ludic] Gaver W. (2002),
Designing for Homo Ludens, i3 Magazine
[VideoGame] Gee J.P. (2003),
What video games have to teach us about learning and literacy, Palgrave Macmillan
[Neuromancer] Gibson W. (1986),
Neuromancer,
[Image] Gonzales R.C. and Wintz P. (1987),
Digital Image Processing, Addison-Wesley, 2nd edn.
[VirtualArt] Grau O. (2003),
Virtual Art -- From Illusion to Immersion, The MIT Press
[Handbook] Grosky W., Jain R., Mehrotra R., eds. (1997),
The Handbook of Multimedia Information Management, Prentice Hall, 1997
[Presence] Hallnäss L. and Redström J. (2002),
From Use to Presence: On the Expressions and Aesthetics of Everyday Computational Things, ACM Trans. on Computer-Human Interactions, Vol. 9, No 2, pp. 106-124
[AHM] Hardman L., Bulterman D., van Rossum G. (1994),
The Amsterdam Hypermedia Model: Adding Time and Context to the Dexter Model, CACM 37(2), pp. 50-62, febr 1994
[DL] Harel D. (1984),
Dynamic Logic, In: Handbook of Philosophical Logic, Vol. II, D. Reidel Publishing Company, 1984, pp. 497-604
[Worlds] Hartman J. and Wernecke J. (1996),
The VRML 2.0 Handbook -- Building Moving Worlds on the Web, Addison-Wesley
[Cinema] Hawkins B. (2005),
Real-time cinematography for games, Charles River Media
[Phenomenology] Hegel G.W.F. (1807),
Phänomenologie des Geistes, Felix Meiner Verlag, ed. 1952
[Sein] Heidegger M. (1927),
Sein und Zeit, Niemeyer Tubingen, 1993
[Similarity] Hewlett W.B. and Selfridge-Field E., eds. (1998) ,
Melodic Similarity -- Concepts, Procedures and Applications, MIT Press
[Interactive] Hildebrand M., Eliens A., Huang Z. and Visser C. (2003),
Interactive Agents Learning their Environment, Proc. Intelligent Virtual Agents 2003, Irsee, September 15-17, 2003 J.G. Carbonell and J.Siekmann (eds.), LNAI 2792, Springer, pp. 13-17
[Hix] Hix D. and Hartson R. (1993),
Developing User Interfaces, Wiley
[Mathematics] Hoggar S.G. (1992),
Mathematics for Computer Graphics, Cambridge University Press
[Giacometti] Hohl H. (1971),
Giacometti, Abrams Publishers
[AWC] Hoorn J., Eliens A., Huang Z., van Vugt H.C., Konijn E.A., Visser C.T.,
Agents with character: Evaluation of empathic agents in digital dossiers, Emphatic Agents, AAMAS 2004 New York 19 July - 23 July, 2004
[Taxonomy] Huang Z., Eliens A., van Ballegooij A., De Bra P. (2000),
A Taxonomy of Web Agents, IEEE Proceedings of the First International Workshop on Web Agent Systems and Applications (WASA \'2000), 2000.
[Architecture] Huang Z., Eliens A., and De Bra P. (2001),
An Architecture for Web Agents, Proceedings of the Conference EUROMEDIA 2001, 2001.
[ Huang Z., Eliens A., Visser C. (2001),
Programmability of Intelligent Agent Avatars, Proceedings of the Agent\'01 Workshop on Embodied Agents, June 2001, Montreal, Canada
[Community] Huang Z., Eliens A., Visser C. (2002),
3D Agent-based Virtual Communities, In: Proc. Int. Web3D Symposium, Wagner W. and Beitler M.( eds), ACM Press, pp. 137-144
[Intelligent] Huang Z., Eliens A., Visser C. (2005),
Intelligent Multimedia Technology: An Approach to Combine Agent Technologies with Multimedia, in preparation
[STEP] Huang, Z., Eliens, A., and Visser, C.,
STEP: a Scripting Language for Embodied Agents, in: Helmut Prendinger and Mitsuru Ishizuka (eds.), Life-like Characters, Tools, Affective Functions and Applications, Springer-Verlag, 2003.
[Reach] Huang, Z., Eliens, A., and Visser, C. (2003d),
"Is it within my reach?" -- an agents perspective , Proc. Intelligent Virtual Agents 2003, Irsee, September 15-17, 2003, J.G. Carbonell and J.Siekmann (eds.), LNAI 2792, Springer, pp. 150-158
[STEP-face] Huang Z., Eliens A., and Visser C. (2004),
Facial Animation in STEP, AAMAS 2004 New York 19 July - 23 July, 2004
[STEP-book] Huang, Z., Eliens, A., and Visser, C. (2003c),
STEP: a Scripting Language for Embodied Agents, in: Helmut Prendinger and Mitsuru Ishizuka (eds.), Life-like Characters, Tools, Affective Functions and Applications, Springer-Verlag, (to appear).
[XSTEP] Huang Z., Eliens A., Visser C. (2003b),
XSTEP: A Markup Language for Embodied Agents, Proc. CASA03, The 16th Int. Conf. on Computer Animation and Social Agents
[STEPIMP] Huang Z., Eliens A., Visser C. (2003a),
Implementation of a scripting language for VRML/X3D-based embodied agents, Proc. Web3D 2003 Symposium, Saint Malo France, S. Spencer (ed.) ACM Press, pp. 91-100
[STEP-imp] Huang Z., Eliens A., Visser C. (2003a),
Implementation of a scripting language for VRML/X3D-based embodied agents, Proc. Web3D 2003 Symposium, Saint Malo France, S. Spencer (ed.) ACM Press, pp. 91-100
[STEP-workshop] Huang Z., Eliens A., Visser C. (2002b),
STEP -- a scripting language for Embodied Agents, PRICAI-02 Workshop -- Lifelike Animated Agents: Tools, Affective Functions, and Applications, Tokyo, 19/8/2002
[Magic] Hughes B. (2000),
Dust or Magic -- Secrets of Successful Multimedia Design, Addison-Wesley, 2000
[Modern] Hummelen IJ. and Sill\'e D. (1999),
Modern Art: Who cares?, Foundation for the Conservation of Modern Art and the Netherlands Institute for Cultural Heritage
[Humdrum] Huron D. (1997) ,
Humdrum and Kern: Selective Feature Encoding, In [_Beyond], pp. 375-400
[Experience] Jain R. (2000),
Digital Experience, [_1000], pp. 38-40
[Java3D] Javasoft (2001),
Java3D API, http://java.sun.com/products/java-media/3D
[Participatory] Jenkins H. (2006),
Confronting the Challenges of Participatory Culture: Media Education for the 21th Century, White Paper, MIT MediaLab
[EducationalVR] Johnson A., Moher T., Cho Y-J., Lin Y.J., Haas D., and Kim J. (2002),
Augmenting Elementary School Education with VR, IEEE Computer Graphics and Applications, March/April
[Studio] Johnson B., Skibo C., Young M. (2003),
Inside Microsoft Visual Studio .NET, Microsoft Press
[HalfReal] Juul J. (2005),
Half Real -- Video Games between Real Rules and Fictional Worlds, MIT Press
[Science] Kalawsky R.S. (1993),
The Science of Virtual Reality and Virtual Reality Environments, Addison-Wesley
[Kritik] Kant E. (1781),
Kritik der reinen Vernunft, Felix Meiner Verlag, ed. 1976
[Urteil] Kant E. (1781),
Kritik der Urteilskraft, Reclam, Stuttgart, 1976
[XSLT] Kay M. (2001),
XSLT Programmer\'s Reference, Wrox Press
[Economy] Kassel S., Schumann C-A. and Tittman C. (2007),
Intelligent Advertisement for E-Commerce, In Proc. EUROMEDIA 2007, Delft, Netherlands
[ACOI] Kersten M. L.,Nes M., Windhouwer M.A. (1998),
A feature database for multimedia objects, CWI Report INS-R9807, July 1998
[ Kientzle T. (1998),
A programmer Guide to Sound, Addison-Wesley
[Numerical] Kimmel R. (2004),
Numerical Geometry of Images -- Theory, Algorithms, and Applications, Springer,
[Magic] Klabbers J.H.G. (2006),
The Magic Circle: principles of Gaming and Simulation, Sense Publishers
[Systems] Koegel Buford J.F., ed. (1994),
Multimedia Systems, Addison-Wesley, 1994
[Time] Koenen R. (1999),
MPEG-4 - Multimedia for our time, IEEE Spectrum, Vol. 36, No. 2,February 1999, pp. 26-33.
[MPEG-4] Koenen R. (ed.) (2000),
Coding of moving pictures and audio, ISO/ITEC JTC1/SC29/WG11 N3747
[Fundamental] Körner S. (1973),
Fundamental Questions of Philosophy, Penguin Books
[Semiotics] Kress G. and van Leeuwen T. (1996),
Reading Images: The Grammar of Visual Design, Routledge
[Samurai] Krikke J. (2003),
Samurai Romanesque, and the battle for mobile cyberspace, IEEE Computer, Jan/Febr 2003
[Quaternions] Kuipers J.B. (1999),
Quaternions and Rotation Sequences -- A Primer with Applications to Orbits, Aerospace, and Virtual Reality, Princeton University Press
[ACL] Labrou L., Finin T. and Peng Y. (1999),
The current landscape of Agent Communication Languages, IEEE Intelligent Systems, Vol. 14, no. 2, 1999
[Tricks] LaMothe A. (2002),
Tricks of the Windows Game Programming Gurus, Sams, 2nd edn.,
[GameMathematics] Lengyel E. (2004),
Mathematics for 3D Game Programming & Computer Graphics , Charles River Media
[Fundamentals] Li Z-N. and Drew M.S. (2004),
Fundamentals of Multimedia, Prentice Hall, Pearson
[Game] Luna F. (2003),
Introduction to 3D Game Programming with DirectX 9.0, Wordware Publishing Inc.
[Street] Manco T. (2004),
Street logos, Thames & Hudson
[GPU] McCool M. and Du Toit S. (2004),
Metaprogramming GPUs with Sh, A K Peters, Ltd.
[Effects] McCuskey M. (2002),
Special Effects Game Programming with DirectX, Premier Press
[Hypertext] McKnight C., Dillon A. and Richardson J. (1991),
Hypertext in Context, Cambridge University Press
[Meldex] McNab R.J., Smith L.A., Bainbridge D. and Witten I.M. (1997),
The New Zealand Digital Library MELody inDEX, D-Lib Magazine, May 1997
[Policy] McQuail D. and Siune K. (1989),
Media Policy -- Convergence, Concentration & Commerce, Euromedia Research Group, Sage Publications
[MIDI] Messick P. (1998),
Maximum MIDI -- Music Applications in C++, Manning
[Grammar] Moortgat, M. (1996),
Categorial Type Logics, Handbook of Logic and Language, J. van Benthem and A. ter Meulen (eds.), Elsevier Amsterdam (1996), pp. 93-177
[Categorial] Moortgat, M. (2002),
Categorial Grammar and Formal Semantics, Encyclopedia of Cognitive Science, Nature Publishing Group, Macmillan Publishers Ltd.
[Compare] Mongeau M. and Sankoff D. (1990),
Comparison of musical sequences, Computers and the Humanities 24, pp. 161-175, 1990
[JavaGame] Morrison M. (2005),
Beginning Mobile Phone Game Programming, Sams Publishing
[Mozart] Mozart W.A. (1781),
Twinkle,twinkle little star (Oh! vous dirai-je, mamam), K300, 1781-82
[Social] Munro A., Höök, K. and Benyon D.R. (1999),
Footprints in the Snow, In: Social Navigation of Information Space, Springer
[Hamlet] Murray J. (1997),
Hamlet on the Holodeck -- The future of narrative in Cyberspace , Free Press
[Digital] Negroponte N. (1995),
Being Digital, New Riders
[Nelson] Nelson T. (1980),
Literary Machines, Mindfull Press
[VisualComputing] Nielsen F. (2005),
Visual Computing: Geometry, Graphics, And Vision, Charles River Media
[Nielsen] Nielsen J. and Mack R.L., eds. (1994) ,
Usability inspection methods, Wiley
[Norman] Norman D.A. (2004),
Emotional Design, Basic Books
[Privacy] Oard D.W., Leuski A. and Stubblebine S. (2006),
Protecting the Privacy of Observable Behavior in Distributed Recommender Systems, In: Proc. SIGIR 2003
[AMICO] Obrenovic Z. and Gasevic D. (2007),
Open-Source Software: All you do is put it together, IEEE Software, Vol. 24, No. 5, Sept/Oct 2007
[AMICOCALC] Obrenovic Z. and Eliens A. (2007),
End-User Development with Heterogeneous Software Services -- Using AMICO:CALC within Spreadsheets, submitted to: ICSE 2008
[Selection] Pachet F., Roy P. and Cazaly D. (1999),
A combinatorial approach to content-based music selection, Proc. IEEE Int. Conf. on Multimedia Systems, Firenze
[Music] Ossenbruggen J. van and Eliens A. (1994),
Music in Time-based Hypermedia, Proc. European Conference on Hypermedia Technology 1994, pp. 224-227
[Hypermedia] Ossenbruggen J. van (2001),
Processing Structured Hypermedia -- A matter of style, Ph.D. Thesis, Free University, 2001
[CWI] Ossenbruggen J. van, Geurts J., Cornelissen F., Rutledge L., Hardman L. (2001),
Towards Second and Third Generation Web-Based Multimedia, In Proc. of The Tenth International World Wide Web Conference pp. 479-488, May 1-5, 2001, Hong Kong
[Nympheas] Patin S. (2006),
Regards sur les Nympheas (in french), Editions de la reunion des musees nationaux.
[Video] Pesce M. (2003),
Programming Microsoft DirectShow for digital video and television, Microsoft Press
[Affective] Picard R.W. (1998),
Affective Computing, MIT Press
[Preece] Preece J., Rogers Y., Sharp H., Benyon D., Holland S., Carey T. (1994),
Human-Computer Interaction, Addison-Wesley
[Design] Purvis A.W. and Le Coutre M.F. (2003),
Graphic Design 20th Century, BIS Publishers
[Human] Ratner P. (2003),
3-D Human Modeling and Animation, Wiley, 2nd edn.
[Guide] Riel C. van, Eliens A., Wang Y (2006),
Exploration and guidance in media-rich information spaces: the implementation and realization of guided tours in digital dossiers, accepted for: International Conference on Multidisciplinay Information Sciences and Technologies (InSciT2006) October, 25-28th 2006, Merida, Spain www.instac.es/inscit2006
[ConceptMaps] Riel C. van, Wang Y. and Eliens A. (2006b) ,
Concept map as visual interface in 3D Digital Dossiers: implementation and realization of nthe Music Dossier, CMC2006, Costa Rica, Sept 5-8 2006
[Signs] Rooijen P. van, ed. (2003),
Signs & Symbols, Pepin Press
[World] Rosenblum L. and Macedonia M., eds. (2002),
Projects in VR -- From Urban Terrain Models to Visible Cities, IEEE Computer Graphics and Applications, 22:4, pp. 10-15
[Tangible] Rosenblum L. and Macedonia M., eds. (2005),
Projects in VR -- Tangible Photorealistic Virtual Museum, IEEE Computer Graphics and Applications, 25:1, pp. 15-17
[Context] Rutledge L., van Ballegooij A., Eliens A. (2000),
Virtual Context - relating paintings to their subject, Culture Track of WWW9 in Amsterdam, The Netherlands, May 16th, 2000
[Reusable] Ruttkay M., Huang Z. and Eliens A. (2003),
Reusable gestures for interactive web agents, Proc. Intelligent Virtual Agents 2003, Irsee, September 15-17, 2003 J.G. Carbonell and J.Siekmann (eds.), LNAI 2792, Springer, pp. 80-87
[Conductor] Ruttkay Z., Huang Z. and Eliens A.,
The Conductor: Gestures for Embodied Agents with Logic Programming, in: Recent Advances in Constraints K.R. Apt, F. Fages, F. Rossi, P. Szeredi and J. Vancza (eds.) LNAI 3010, Springer 2004
[SecondLife] Rymaszewski M., Au W.J., Wallace M., Winters C., Ondrejka C., Batstone-Cunningham B. (2007),
Second Life -- the official guide, Wiley
[Algorithms] Sanchez-Crespo Dalmau D. (2004),
Core Techniques and Algorithms in Game Programming , New Riders
[3DEffects] Santiago D. (2005),
Creating 3D Effects for Film, TV and Games, Thomson
[NET] Santos Lobao A. and Hatton E. (2003),
.NET Game Programming with DirectX 9.0, The Authors Press
[Imagination] Sartre J.P. (1936),
L'imagination, Presse Universitaire de France, ed. 8, 1971
[Aesthetics] Saw R.L. (1971),
Aesthetics -- An Introduction, Anchor Books, New York
[Erziehung] Schiller F. (1977),
Uber die ästhetische erziehung des Menschen, Reclam, Stuttgart
[Features] Schmidt, A.R. Windhouwer M.A., Kersten M.L. (1999),
Indexing real-world data using semi-structured documents, CWI Report INS-R9902
[ Schneider P.J. and Eberly D.H. (2003),
Geometric Tools for Computer Graphics, Morgan Kaufmann Publishers
[Gadgets] Schonhage S.P.C., van Ballegooij A. and Eliens A. (2000),
3D Gadgets for Business Process Visualization: a case study, Proc. Int. Web3D/VRML Conference - 2000, Monterey CA, Febr 2000, ACM Press pp. 131-138
[GameDesign] Schuytema P. (2007),
Game Design: A Practical Approach, Charles River Media
[Beyond] Selfridge-Field E., ed. (1997),
Beyond MIDI -- The Handbook of Musical Codes , MIT Press 1997
[Concepts] Selfridge-Field E. (1998),
Conceptual and Representational Issues in Melodic Comparison, In [_Similarity], pp. 3-64
[Hybrid] Setten M. van (2005),
Supporting People in Finding Information -- Hybrid recommender Systems and Goal-based Structuring, Ph.D. Thesis, Telematica Institute Netherlands
[VR] Sherman W.R. and Craig A.B. (2003),
Understanding Virtual Reality -- Interface, Application and Design, Morgan Kaufmann Publishers
[Ultimate] Sherrod A. (2006),
Ultimate Game Programming with DirectX, Charles River Media
[User] Shneidermann B. (1997),
Designing the user interface -- strategies for effective human-computer interaction, Addison-Wesley (3rd edn)
[Laptop] Shneiderman B (2003),
Leonardo's Laptop, Mit Press
[PanoramaWeb] Si Yin and Eliens A. (2007),
PANORAMA -- A Rich VRML Application Platform For Online Gaming, Workshop Web 3D Games, Web3D Symposium 07, Perugia Italy, April 15-18
[VE] Singhal S. and Zyda M. (1999),
Networked Virtual Environments, Addison-Wesley, 1999
[Terrain] Snook G. (2003),
Real-time 3D terrain engines using C++ and DirectX 9, Charles Riven Media
[Computer] Spalter A.M. (1999),
The Computer in the Visual Arts, Addison-Wesley
[Snowcrash] Stephenson N. (1992),
Snowcrash,
[ShaderArt] St-Laurent S. (2004),
Shaders for Game Programmers and Artists, Thomson
[MMDBMS] Subrahmanian V.S. (1998),
Principles of Multimedia Databases, Morgan Kaufmann
[DEEP] Sullivan A. (2005),
3-Deep, IEEE Spectrum 42-4, April 2005, pp. 22-27
[Jinni] Tarau P. (1999),
Jinni: Intelligent Mobile Agent Programming at the Intersection of Java and Prolog, Proc. of PAAM\'99, London, UK, April, see also http://www.binnetcorp.com/Jinni
[Meter] Temperley D. and Sleator D. (1999) ,
Modeling Meter and Harmony: A Preference-Rule Approach, Computer Music Journal 23(1), 1999, pp. 10-27
[CodeArt] Trogemann G. & Viehoff J. (2004),
CodeArt -- Eine elementare Einfhrung in die Programmierung als kunstlerische Praktik, Springer
[Codecs] Vasudev B. and Li W. (1997),
Memory management: Codecs, In [_Handbook], pp. 237-278
[Making] Vaughan T. (1998),
Multimedia -- Making It Work, Osborne/McGraw-Hill, 4th edn
[Essential] Vince J. (1998),
Essential Virtual Reality, Springer
[Web] Visser C. and Eliens A. (2000),
A High-Level Symbolic Language for Distributed Web Programming, Internet Computing 2000, June 26-29, Las Vegas
[Meaning] Vyas, D., van der Veer, G. C. (2006),
Experience as Meaning: Some Underlying Concepts and Implications for Design. , Proceedings of 13th European Conference on Cognitive Ergonomics. ACM Press: NY, (2006), 81-91.
[Affordance] Vyas, D., Chisalita, C., van der Veer, G. C. (2006),
Affordance in Interaction. , Proc. of 13th European Conference on Cognitive Ergonomics. ACM Press: NY, (2006), 92-99.
[SMIL] W3C SMIL Working Group (2001),
Synchronized Multimedia Integration Language (SMIL 2.0), W3C Recommendation 07 August 2001
[Web3D-2002] Wagner W. and Beitler M. (eds), 2002,
Proc. Int. Web3D Symposium, ACM Press
[Advance] Walsh P. (2003),
Advanced 3D Programming with DirectX 9.0, Wordware Publishing Inc.
[Present] Wang Y., Eliens A., van Riel C. (2006),
Content-oriented presentation and personalized interface of cultural heritage in digital dossiers, accepted for: International Conference on Multidisciplinay Information Sciences and Technologies (InSciT2006) October, 25-28th 2006, Merida, Spain www.instac.es/inscit2006
[Games] Watt A. and Policarpo F. (2001),
3D Games -- Real-time Rendering and Software Technology, Addison-Wesley
[Web3D] Web3D Consortium (2001),
VRML97 Standard, \\ http://www.web3d.org/fs_specifications.htm
[Design] Weishar P. (1998),
Digital Space -- Designing Virtual Environments, McGraw-Hill
[Chatting] Welie M. and and Eliens A. (1996),
Chatting on the Web, Proc. ERCIM W4G Workshop on CSCW and the Web, 7-9 febr 1996 GMD St Augustin, Germany, 1996
[Euterpe] Welie M. van, G.C. van der Veer G.C., and Eliens A. (1998) ,
Euterpe - Tool support for analyzing cooperative environments, Proceedings of the Ninth European Conference on Cognitive Ergonomics , August 24-26, 1998, University of Limerick, Ireland, pp. 25-30
[Beauties] Wiedermann J., ed. (2002),
Digital Beauties, Taschen Verlag
[Animovie] Wiedermann J., ed. (2004),
Animation now, Taschen Verlag
[InformationArts] Wilson S. (2002),
Information Arts -- Intersections of Art, Science and Technology, The MIT Press
[Flow] Yellowlees D. and Hargadon A. (2000),
The pleasure principle: immersion, engagement, flow, Proc. of the 11th ACM Conf. on Hypertext and Hypermedia, May 2000,San Antonio, pp. 153-160
[DeepTime] Zielinski S. (2006),
Deep Time of the Media -- Towards an archaeology of Hearing and Seeing by Technical Means, The MIT Press
[AI] Zimmermann D.,
Modeling Musical Structures, Aims, Limitations and the Artist\'s Involvement, Proc. Constraints techniques for artistic applications, Workshop at ECAI\'98 25th August, 1998, Brighton, UK, ftp://ftp.csl.sony.fr/pub/pachet/workshopEcai
\begin{theindex} \item Adams (2003), \hyperpage{x}, \hyperpage{111} \item Alberti (1435), \hyperpage{223} \item Anders (1999), \hyperpage{195} \item Arnaud \& Barnes (2006), \hyperpage{110} \item Arnheim (1957), \hyperpage{x}, \hyperpage{213}, \hyperpage{249}, \hyperpage{291, 292}, \hyperpage{299} \item Atanasova et al. (2007), \hyperpage{171} \item Auster (2004), \hyperpage{155} \indexspace \item Baeza-Yates and Ribeiro-Neto (1999), \hyperpage{x}, \hyperpage{79}, \hyperpage{117, 118}, \hyperpage{128}, \hyperpage{131}, \hyperpage{159}, \hyperpage{162, 163} \item Ballegooij and Eliëns (2001), \hyperpage{x}, \hyperpage{154} \item Baxter et al. (2004), \hyperpage{96} \item Benjamin (1936), \hyperpage{222} \item Betsky (2004), \hyperpage{50}, \hyperpage{175} \item Bj\"{o}rk \& Holopainen (2005), \hyperpage{282}, \hyperpage{287, 288}, \hyperpage{305} \item Bocconi (2006), \hyperpage{86} \item Bolter and Grusin (2000), \hyperpage{x}, \hyperpage{40, 41}, \hyperpage{50}, \hyperpage{195}, \hyperpage{199}, \hyperpage{223}, \hyperpage{249}, \hyperpage{305} \item Briggs and Burke (2001), \hyperpage{9}, \hyperpage{14}, \hyperpage{19, 20}, \hyperpage{50} \item Broll (1996), \hyperpage{329} \item Broll et. al (2001), \hyperpage{80} \item Bruner (1972), \hyperpage{4}, \hyperpage{295}, \hyperpage{310} \item Burger (1981), \hyperpage{50}, \hyperpage{130}, \hyperpage{250} \item Bush (1995), \hyperpage{33} \indexspace \item CACM (2001), \hyperpage{25} \item Cesa-Bianchi and Lugosi (2006), \hyperpage{151} \item Chang and Costabile (1997), \hyperpage{x}, \hyperpage{28}, \hyperpage{33} \item Chapman and Chapman (2004a), \hyperpage{x} \item Chapman and Chapman (2004b), \hyperpage{x} \indexspace \item Daskalova \& Atanasova (2007), \hyperpage{170} \item Davenport (2000), \hyperpage{x}, \hyperpage{5} \item Davison (2001), \hyperpage{329} \item Dewey (1931), \hyperpage{291}, \hyperpage{302, 303} \item Diehl 1973, \hyperpage{50} \item Dodge and Kitchin (2000), \hyperpage{152} \item Dodge and Kitchin (2002), \hyperpage{144} \item Dormann et al. (2007), \hyperpage{301} \indexspace \item Eco (1994), \hyperpage{50} \item Eliëns (1979), \hyperpage{223}, \hyperpage{302} \item Eliëns (1988), \hyperpage{220}, \hyperpage{308} \item Eliëns (1992), \hyperpage{323} \item Eliëns (2000), \hyperpage{37}, \hyperpage{48}, \hyperpage{54}, \hyperpage{79}, \hyperpage{99}, \hyperpage{148}, \hyperpage{168, 169}, \hyperpage{174}, \hyperpage{182}, \hyperpage{240}, \hyperpage{324} \item Eliëns \& Chang (2007), \hyperpage{22}, \hyperpage{82}, \hyperpage{250}, \hyperpage{263}, \hyperpage{287} \item Eliëns \& Vyas (2007), \hyperpage{129}, \hyperpage{221} \item Eliëns \& Wang (2007), \hyperpage{151}, \hyperpage{171} \item Eliëns and Bhikharie (2006), \hyperpage{109}, \hyperpage{263} \item Eliëns et al. (2002), \hyperpage{323} \item Eliëns et al. (2007), \hyperpage{21}, \hyperpage{23}, \hyperpage{25}, \hyperpage{108}, \hyperpage{195, 196} \item Eliëns et al. (2007b), \hyperpage{82}, \hyperpage{109}, \hyperpage{282} \item Eliëns et al. (2007c), \hyperpage{20, 21}, \hyperpage{45}, \hyperpage{251} \item Eliëns {\em et al.} (1997), \hyperpage{37} \item Engel (2004a), \hyperpage{96} \item Engel (2004b), \hyperpage{96} \item Engel (2005), \hyperpage{96} \item Engelbart (1963), \hyperpage{33} \item Entanaclaz (2003), \hyperpage{217} \indexspace \item Faulkner (2000), \hyperpage{212} \item Fay et al. (2004), \hyperpage{111} \item Fernando and Kilgard (2003), \hyperpage{x}, \hyperpage{90, 91}, \hyperpage{96} \item Fluckiger (1995), \hyperpage{x}, \hyperpage{164, 165} \item Forman and Saint John (2000), \hyperpage{x} \item Freud (1958), \hyperpage{302, 303} \item Fuhr et al. (1998), \hyperpage{119} \indexspace \item Gee (2003), \hyperpage{83}, \hyperpage{197}, \hyperpage{284} \item Gonzales and Wintz (1987), \hyperpage{131} \item Grau (2003), \hyperpage{x}, \hyperpage{4}, \hyperpage{22}, \hyperpage{129}, \hyperpage{223}, \hyperpage{276, 277}, \hyperpage{279, 280}, \hyperpage{299, 300}, \hyperpage{303} \indexspace \item Halln\"{a}ss and Redstr\"{o}m (2002), \hyperpage{222}, \hyperpage{290, 291}, \hyperpage{304} \item Hardman et al. (1994), \hyperpage{37} \item Harel (1984), \hyperpage{193} \item Hawkins (2005), \hyperpage{x} \item Hegel (1807), \hyperpage{302, 303} \item Heidegger (1927), \hyperpage{291} \item Hewlett and Selfridge-field (1998), \hyperpage{135}, \hyperpage{206} \item Hohl (1971), \hyperpage{178} \item Huang et al. (2000), \hyperpage{191} \item Huang et al. (2002), \hyperpage{x}, \hyperpage{190} \item Hummelen and Sill\'e (1999), \hyperpage{148}, \hyperpage{155, 156} \item Huron (1997), \hyperpage{138} \indexspace \item Jain (2000), \hyperpage{x}, \hyperpage{6} \item Jenkins (2006), \hyperpage{86}, \hyperpage{196} \item Johnson et al. (2002), \hyperpage{193} \item Juul (2005), \hyperpage{x}, \hyperpage{22}, \hyperpage{250}, \hyperpage{286} \indexspace \item K\"{o}rner (1973), \hyperpage{130} \item Kant (1781), \hyperpage{130}, \hyperpage{302, 303} \item Kassel et al. (2007), \hyperpage{170} \item Kay (2001), \hyperpage{32}, \hyperpage{322} \item Kersten et al. (1998), \hyperpage{x}, \hyperpage{145} \item Klabbers (2006), \hyperpage{x}, \hyperpage{33, 34}, \hyperpage{82}, \hyperpage{206, 207}, \hyperpage{284} \item Koenen (1999), \hyperpage{62} \item Koenen (2000), \hyperpage{x}, \hyperpage{61}, \hyperpage{63}, \hyperpage{87} \item Kress and van Leeuwen (1996), \hyperpage{x}, \hyperpage{223}, \hyperpage{249, 250}, \hyperpage{258}, \hyperpage{274--278}, \hyperpage{293}, \hyperpage{296--298}, \hyperpage{302, 303} \item Krikke (2003), \hyperpage{262} \indexspace \item Li and Drew (2004), \hyperpage{87}, \hyperpage{175} \item Luna (2003), \hyperpage{x}, \hyperpage{111}, \hyperpage{284} \indexspace \item Manco (2004), \hyperpage{224} \item McCuskey (2002), \hyperpage{x}, \hyperpage{252} \item McKnight et al. (1991), \hyperpage{34} \item McNab et al. (1997), \hyperpage{x}, \hyperpage{137, 138}, \hyperpage{149} \item Mongeau and Sankoff (1990), \hyperpage{135}, \hyperpage{137} \item Morrison (2005), \hyperpage{25} \item Mozart (1787), \hyperpage{135}, \hyperpage{149} \item Murray (1997), \hyperpage{42--44} \indexspace \item Negroponte (1995), \hyperpage{3} \item Nelson (1980), \hyperpage{33} \item Norman (2004), \hyperpage{209}, \hyperpage{212} \indexspace \item Oard et al. (2006), \hyperpage{171, 172} \item Obrenovic \& Eliëns (2007), \hyperpage{271} \item Obrenovic \& Gasevic (2007), \hyperpage{272} \item Ossenbruggen (2001), \hyperpage{x}, \hyperpage{37, 38} \item Ossenbruggen \& Eliëns (1994), \hyperpage{37} \item Ossenbruggen et. al. (2001), \hyperpage{75} \indexspace \item Pesce (2003), \hyperpage{99}, \hyperpage{143} \item Picard (1998), \hyperpage{81} \item Preece et al. (1994), \hyperpage{233}, \hyperpage{239} \indexspace \item Rosenblum and Macedonia (2002), \hyperpage{13}, \hyperpage{25} \item Rosenblum and Macedonia (2005), \hyperpage{232}, \hyperpage{252} \item Rymaszweski et al. (2007), \hyperpage{25}, \hyperpage{195} \indexspace \item Sanchez-Crespo Dalmau (2004), \hyperpage{261, 262} \item Santos Lobao and Hatton (2003), \hyperpage{25} \item Sartre (1936), \hyperpage{304} \item Saw (1971), \hyperpage{130}, \hyperpage{223}, \hyperpage{304} \item Schiller (1977), \hyperpage{303} \item Schmidt et al. (1999), \hyperpage{146}, \hyperpage{174} \item Schuytema (2007), \hyperpage{259} \item Selfridge (1997), \hyperpage{150} \item Selfridge (1998), \hyperpage{135, 136} \item Sherman and Craig (2003), \hyperpage{198}, \hyperpage{208} \item Sherrod (2006), \hyperpage{x}, \hyperpage{258} \item Shneiderman (1997), \hyperpage{38, 39}, \hyperpage{174} \item Shneiderman (2003), \hyperpage{25} \item Si \& Eliëns (2007), \hyperpage{129}, \hyperpage{251}, \hyperpage{253} \item Singhal and Zyda (1999), \hyperpage{166, 167} \item St-Laurent (2004), \hyperpage{96} \item Subrahmanian (1998), \hyperpage{x}, \hyperpage{115, 116}, \hyperpage{119, 120}, \hyperpage{122--127}, \hyperpage{133, 134}, \hyperpage{139--142}, \hyperpage{158}, \hyperpage{160, 161}, \hyperpage{165} \item Sullivan (2005), \hyperpage{101}, \hyperpage{112} \indexspace \item Tarau (1999), \hyperpage{329} \item Temperley and Sleator (1999), \hyperpage{138} \item Trogemann \& Viehoff (2004), \hyperpage{107} \indexspace \item van Rooijen (2003), \hyperpage{2}, \hyperpage{25}, \hyperpage{50}, \hyperpage{54}, \hyperpage{87}, \hyperpage{112}, \hyperpage{114}, \hyperpage{132}, \hyperpage{156}, \hyperpage{176}, \hyperpage{178}, \hyperpage{199}, \hyperpage{225}, \hyperpage{253}, \hyperpage{256}, \hyperpage{284}, \hyperpage{306} \item van Setten (2005), \hyperpage{151}, \hyperpage{174} \item Vasudev and Li (1997), \hyperpage{x}, \hyperpage{56--58} \item Visser and Eliëns (2000), \hyperpage{x}, \hyperpage{46} \item Vyas and van der Veer (2006) , \hyperpage{290} \indexspace \item W3C (2001), \hyperpage{61} \item Weishar (1998), \hyperpage{50}, \hyperpage{155} \item Welie et al. (1998), \hyperpage{239}, \hyperpage{253} \item Wiedermann (2002), \hyperpage{199}, \hyperpage{284} \item Wiedermann (2004), \hyperpage{87}, \hyperpage{225} \item Wilson (2002), \hyperpage{4}, \hyperpage{23, 24}, \hyperpage{279--281} \indexspace \item Zielinski (2006), \hyperpage{x}, \hyperpage{4, 5}, \hyperpage{22}, \hyperpage{129}, \hyperpage{270}, \hyperpage{280} \item Zimmerman (1998), \hyperpage{146} \end{theindex} \mbox{}\newpage

topical media & game development

\hspace{-0.35cm}

\vspace{0.0cm} This book provides a concise and comprehensive introduction to multimedia. It arose out of the need for material with a strong academic compontent, that is (simply) material related to scientific research. Indeed, studying multimedia is not (only) fun. Compare it with obtaining a driver license. Before you are allowed to drive on the higway, you have to take a theory exam. So why not take such an exam before entering the multimedia circus. Don't complain, and take the exam. After all it makes you aware of the rules governing the (broadband) digital highway. The book and accompanying material is available at www.cs.vu.nl/~eliens/media

\hspace{3cm}

\parbox{5cm}{© \hspace{0.2cm}Æliens, Amsterdam (2002)}