This book provides a concise and comprehensive introduction to multimedia. It arose out of the need for material with a strong academic component, that is material related to scientific research.
Indeed, studying multimedia is not only fun. Compare it with obtaining a driver license. Before you are allowed to drive on the highway, you have to take a theory exam. So why not take such an exam before entering the multimedia circus.
Don't complain, and take the exam. After all it makes you aware of the rules governing the (broadband) digital highway.
themes and variations
So, who is this book meant for?
It is meant for the student or reader who is looking for
a quick introduction to the main topics in multimedia.
The twelve chapters provide a concise
overview of the themes and trends in current multimedia practice
and research.
The themes and variations addressed in this book may be summarized
as follows.
themes and variations
- digital convergence -- all for one, one for all
- broadband communication
-- entertainment
- multimedia information retrieval -- as an afterthought?
- multimedia and game application(s) -- from design to development
To explain in somewhat more detail, digital convergence
may be characterized as the coming together of data
(including audio, video and information)
in a possible multitude of platforms,
to which these data are delivered by a variety
of (broadband) communication channels.
In fact, the increasingly powerful communication infrastructure
due to the popularity of the Internet and the World Wide Web,
leads to an almost universally accessible multimedia (information)
repository, for which (unfortunately) the notion of
(multimedia) information retrieval seems to have occurred
only as an afterthought.
Digital content design is only one step in the process
of multimedia application development.
Important issues in multimedia application development are,
apart from project maanagement, data representation, navigation,
presentation and usability.
An underlying thought that motivated the writing of this book is that somehow the gap between authoring and retrieval should be bridged.
In other words, either by developing the technology for extracting features or attributes from multimedia objects, or by applying content annotation for such objects, multimedia information retrieval should be considered as a necessary asset to make a multimedia web an effective information repository.
In multimedia applications, such as the digital dossier we introduce in chapter 10, the data representation must accomodate meta-information, to support effective navigation and search.
Another line of thought, that became more clear during the writing of the book
is concerned with the aesthetics of (interactive) applications.
You will find more on this in chapters 11 and 12, that deal with game development.
what do you need to learn
When taking up multimedia as a subject of study,
you may ask yourself what you need to know and learn about it.
In general, what this book presents is
what do we have to learn?
- concepts,
- facts,
- history,
- applications
- standards
- technology
- multimedia and games
Let me be frank with you.
There is too much information to be digested in a first course.
Nevertheless, after studying this book you will have an introduction
to multimedia that should be viable for the rest
of your (academic) career.
Now, don't hesitate, put yourself to the test
and check which phrases and acronyms you are familiar with
in the lists given for the subjects of
digital convergence,
broadband communication and
information retrieval.
digital convergence
- concepts -- digital revolution
- facts -- from the entertainment industry
- history -- from Pong to Big Brother
- applications -- infotainment
- standards -- MPEG, RM3D, SMIL
- technology -- TV, PC, DVD
...
How did you succeed thus far?
If you did well, try the second round and test yourself in
what detail you have have knowledge about technologies
mentioned.
broadband communication
- concepts -- Quality of Service
- facts -- compression is needed
- history -- the internet
- applications -- entertainment and communication
- standards -- HTTP, TCP/IP, RTP
- technology -- cable, (X)DSL
...
Finally, check to what extent you master the vocabulary
of multimedia information retrieval.
multimedia information retrieval
- concepts -- features, precision, recall
- facts -- the problem is utterly complex
- history -- from text to multimedia
- applications -- digital libraries
- standards <-- distance metrics
- technology -- indexing & algorithms
If you are working online, you may click back to the text in the book
that explains these notions.
Just to make sure whether your impression of familiarity was justified.
assignment(s)
I strongly believe that practical work is necessary,
also for academics,
to get a good grasp on multimedia and game development.
Even if your interest is purely intellectual,
it pays off to make your virtual hands dirty
and indulge in making a compelling presentation.
As an assignment, consider making a presentation that offers an
Annotated Tour in Amsterdam
Amsterdam is the place where I live, and where our
students take their courses.
You may find it more convenient or natural to
replace Amsterdam with a location of your choice.
Online, you will find an elaborated version of the assignment,
including an extended description, a working plan,
deliverables and hints.
In essence though, the intent of the assignment is
to make a compelling, not to say artistic, presentation,
and to explore the realm of multimedia rethorics.
As a tool you may choose, for example,
Flash or the flex 2 SDK, which is freely available.
examination
Despite the fact that some consider the practical aspects
of multimedia to be exclusively relevant,
the intellectual aspects of multimedia should not be ignored.
Consider the following question, which is directly
related to one of the themes underlying this book,
that is the complementarity of authoring and retrieval:
This question can only be answered when the student
has a sufficient level of experience, insight and
knowledge of the field, and is able to relate theory and practice.
Each chapter contains a brief list of questions
that may be used as a checklist,
to see if you have sufficient knowledge of
a particular area.
These questions may also be used to prepare exams!
The questions are meant to test for insight,
that is the ability to discuss a somewhat broader theme,
and knowledge of concepts
and technology,
covering definitions, applications, historical facts, as well
as the technological infrastructure enabling the deployment
of multimedia applications.
In addition to the regular material, the book also contains
a number of examples and sections indicating research directions. These sections are not meant to be part of the exam, but might provide the student
with suggestions for projects or further research.
Moreover, both the discussions in the research directions
and the material in the appendices presents a vision on what
multimedia should be.
In effect, I have a strong preference for a programmatic approach
to (intelligent) multimedia, as outlined in appendix E.
Nevertheless, the bulk of the (regular) material is relevent also
for readers with a rather different opinion on
what consitutes the essence of multimedia.
how to use this book
The intended audience for this book is
intended audience(s)
The course notes were explicitly written for first year
Computer Science and Information Science students.
The Information Science students are expected to choose
the specialisation Multimedia and Culture, a curriculum
provided by the department of Mathematics and Computer Science
of the Faculty of Sciences of the VU University Amsterdam.
The course has a practical part and a theoretical part,
which in combination takes 2-4 weeks, full time study.
The book covers the theoretical part.
The online version gives a skeleton assignment that
may adapted by the one responsible for the course.
The online version contains all the material needed for
giving a multimedia course, that is
multimedia course
- presentations for all chapters, including the preface in dynamic HTML slides
- presentable versions of the MPEG-4 standard, and other relevant material
- possible exam questions, with back links into the text for quick learning and review
- seven sample lectures, with additional explanation for the instructor
One additional remark may be made. This is (so to speak)
'a book with an attitude'. It is slightly authorative
and directive towards the students, telling them to learn
the facts and 'do the exam'.
Some students take refuge to learning the 'keywords and phrases'.
They are even helped in this respect, since the text uses
a 'graphic' layout to emphasize important points, and
to allow for a quick recognition of chunks of relevant material.
the artwork
Although a book about multimedia does not need to be a multimedia artefact
itself, it seemed better to include illustrations, to avoid the impression of
a 'dry' book. Since I did not want to include any redundant diagrams or pictures,
I decided to use a personal selection from the history of visual design, games,
computer art and video art, not only to spice up the book but also to
give the reader a collection of interesting samples.
Each chapter starts with illustrations setting the visual theme
of the chapter.
All other illustrations are, in one way or another, related to
the examples or the text of that chapter.
Brief comments about the artwork, and an explanation of the visual theme,
can be found at the end of each chapter.
about the author
At some point you may wonder whether the author is qualified or authorized
to write about a particular subject and, in this particular case, to publish a
book about such an elusive notion as multimedia.
Let me give you some personal history.
Way back in the seventies, I did a degree in painting at the Gerrit Rietveld Academy
in Amsterdam.
At the same time, I did a master's philosophy, where I graduated in the field
of aesthetics on a comparative study on theories of imagination and creativity,
reading writers such as Kant, Husserl and Sartre.
Then I got an interest in computer music, after listening to a concert of Xenakis
in Paris, and started to work on a PDP-15 (with 4K of memory) at the Institute
of Sonology in Utrecht.
Leaving all philosophy and traditional art behind, I learned programming,
studied AI and theoretical computer science.
Some eight years later, I obtained my Ph.D. in computer science and started
my academic career.
After working in software engineering, and in particular object-oriented
software development, I was asked, at the end of the millenium,
to set up a collection of multimedia courses,
since by then multimedia was coming in vogue as an academic subject.
These courses, which include the introduction multimedia,
Web3D authoring, intelligent virtual environments, a multimedia casus,
and recently also visual design, are reflected
in this book.
about the book
What started as a (not so) gentle introduction to multimedia, has grown into a rich
(at times somewhat idiosyncratic)
collection of topical material about multimedia and game development.
Borrowing a phrase from the politics of the seventies, at some point, apparently,
the professional became personal, and the personal professional.
Nevertheless,
the book may still be read as an introduction.
It is written in a concise and compact manner, supported by the slides
format, which allows for presentation of the material in class,
and is illustrated by a variety of images, taken from the arts, design,
and multimedia and game projects.
The book consists of the following parts:
part(s)
- part i -- digital convergence
- part ii -- delivery & presentation
- part iii -- multimedia information retrieval
- part iv -- applications & tools
- part v -- game development
Actually, as will become apparent when reading, the book is the result of a series of
revisions.
Since I started writing the book, the vision of digital convergence has become a reality,
game research has become a respectable academic discipline, and attention has
shifted from new media to cross media.
Moreover, the role of media in our society is subject to change as well.
With the Web 2.0, our information society is no loner passive, but part of
a participatory culture in which users contribute and ultimately control content.
And these developments are not likely to stop or slow down.
As concerns the history of this book, after writing chapters 1-8 (minus 4, which was
added later), I extended the manuscript with with chapters 4, 9 and 10.
After a thorough revision, I included the artwork, and then decided to subdivide the book in parts,
adding chapter 11 and 12 in part v, on game development.
The latest revision is the inclusion of another section with each chapter, discussing the
latest development(s).
Not visible in the printed book are the numerous technical examples, including
ajax, flex and wiki experiments, that may help the reader in his/her exploration(s).
Over the last year(s), it has become evident for me
that the only viable method of learning multimedia is exploratory development.
That is not to say that the theoretical material is superfluous.
On the contrary.
But only in the actual context of developing meaningful applications,
meaningful for the individual that is,
does the necessity of theory become an obvious truth.
acknowledgements
This book is the result of developing
the course notes for an introduction to multimedia
for first year Computer Science and Information Science students.
Hence, first of all, I like to thank the students that
had to endure all the rough drafts of this material,
and perhaps not to forget my experiment(s) with the presentation format
of it.
Further I like to thank Harrie van der Lubbe and Sander
Lammers for developing the manual for Director
and their support in developing the practical assignment.
Also, I like to thank Martin Kersten from CWI for
allowing me to join his Multimedia Database Systems research group
as a guest for a period of about two years, and Alex van Ballegooij
for his active involvement in the RIF project and
his coding effort for the slide PROTOs, used
to produce the presentation slides for this book and described
in appendix B.
Also from CWI, I like to thank Lloyd Ruttledge, Lynda Hardman
and Jacco van Ossenbruggen, for their effort
in thinking about the multimedia course in its initial stages,
and Lloyd and Jacco for their involvement in some of
the practical work, and Jacco in particular for
his knowledge of hypermedia systems that he shared with me
during the period that he was my Ph.D. student.
From CWI, I like to thank Zsofi Ruttkay for her general
interest in 'my projects'.
From the VU, I like to thank Andy Tanenbaum
for allowing me to use his material on digital video,
Gerrit van der Veer
for taking the initiative for Multimedia and Culture,
Zhisheng Huang for his excellent contributions to the WASP
project,
Johan Hoorn for our spirited cooperation,
and
Claire Dormann, for our discussions on
the direction the Multimedia and Culture
curriculum should take, and for sharing
her thoughts
on persuasive technology with me.
I also like to thank Tatja Scholte and IJsbrand Hummelen from ICN
(Netherlands Institute for Cultural Heritage)
for their contributions to the multimedia casus,
and Gaby Weijers and Bart Rutten from Montivideo for their
cooperation and all the video material they so generously provided.
Thanks are due to Mark Veldhuijzen van Zanten,
Jaap Stahlie, Peter van Kessel, Federico Campanale
and Katelijne Arts for providing me with material.
Special thanks goes to the student members of the 2004 autumn
multimedia casus group, for their collective work
on the abramovic dossier,
and to Rutger van Dijk for rekindling my interest in C++/DirectX
programming by his youthful enthousiasm.
I also should not forget two students from the Computer Science master Multimedia,
who were exceptional in their dedication and skills, Winoe Bhikharie, who acted (among others) as the
manager in the development of the VU-Life game described in 11.2,
and Marco Bouterse, who excelled in both DirectX and Half-life shader programming.
Also I need to mention Marek van de Watering, a student from Multimedia and Culture,
who often surprised me by his sincerity and sensitivity, and Hugo Huurdeman, who convinced me of
the viability of interactive (flash) video and who proved to
be an extremely dedicated and creative worker.
No doubt, I owe thanks to Gaynor Redvers-Mullon who made a serious attempt to encourage me
to get the best out of this manuscript,
even though at some point I decided to do it in my own way.
Further,
I also wish to thank Dhaval Vyas, at that time PhD. student at VU, for his interest in my
projects and for sharing his
thoughts on the panorama
project, znd Zeljko Obrenovic, postdoc at CWI, for his enthusiasm, his technical
skills and his willingness to assist me with my teaching.
I thank
Pier Vellinga, who I googled as the spin-doctor of climate change,
for involving me in Clima Futura, and not in the least for the
pleasant cameraderie we enjoyed during the effort to obtain
the dutch national prize for academic communication, and
Frans Feldberg, for involving me in the VU @ Second Life effort,
an equally refreshing interdisciplinary partnership,
with which we got into the 8 o'clock nation TV news,
as planned!
of course, I should not forget the guys from the communication department of the VU,
Egon Compter and Merlijn Draaisma, who perhaps somewhat overestimated my expertise,
in their enthusiasm to involve me in their projects.
Last but not least, I owe Yiwen Wang (ηδΎζ) my deepest gratitude
for providing the motivation to continue the work, and to strive
for beauty and pleasure, in a serious way.
Finally, I must mention that I owe much insight
and material to (among others) the following
books and articles: [Subrahmanian (1998)], [Forman and Saint John (2000)], [Chang and Costabile (1997)],
[Ossenbruggen (2001)], [Vasudev and Li (1997)], [Klabbers (2006)], [Grau (2003)], [Kress and van Leeuwen (1996)], and not to forget
[Zielinski (2006)].
As in any intellectual endeavor, intellectual ancestry can
hardly be praised enough.
So let me briefly indicate, for each chapter, some of the sources
that provided me with inspiration, insight and material:
references
- [Forman and Saint John (2000)], [Davenport (2000)], [Jain (2000)].
- [Chang and Costabile (1997)], [Ossenbruggen (2001)], [Klabbers (2006)].
- [Vasudev and Li (1997)], [Koenen (2000)], [Visser and Eliens (2000)].
- [Luna (2003)], [Adams (2003)], [Fernando and Kilgard (2003)]
- [Subrahmanian (1998)], [Baeza-Yates and Ribeiro-Neto (1999)].
- [Subrahmanian (1998)], [McNab et al. (1997)], [Kersten et al. (1998)].
- [Subrahmanian (1998)], [Fluckiger (1995)],
- [Fluckiger (1995)], [Ballegooij and Eliens (2001)], [Huang et al. (2002)].
- [McCuskey (2002)], [Bolter and Grusin (2000)],
- [Chapman and Chapman (2004a)], [Chapman and Chapman (2004b)], [Klabbers (2006)],
- [Sherrod (2006)], [Grau (2003)],
- [Juul (2005)], [Arnheim (1957)], [Hawkins (2005)], [Kress and van Leeuwen (1996)].
The material in sections 4.3, 7.1, 7.3, chapter 8,
sections 9.3 and 10.2, and section 11.2 reflect
my own research efforts.
The other material has all been diligently collected
from (among others) the sources mentioned.
(C) Æliens
04/09/2009
You may not copy or print any of this material without explicit permission of the author or the publisher.
In case of other copyright issues, contact the author.
