topical media & game development
[] readme course(s) preface I 1 2 II 3 4 III 5 6 7 IV 8 9 10 V 11 12 afterthought(s) appendix reference(s) example(s) resource(s) _

talk show tell print

print / present

10

application development

learn the craft, break through the magic of engineering

learning objectives

After reading this chapter you should be able to discuss the multimedia development process, to indicate the need for information system support in the cultural heritage domain, to characterize the notion of digital dossier, to provide solutions for navigating complex information spaces, and to discuss the data representation issues involved.

As you gather from reading this book, the field of multimedia is widely divergent. However, when you develop a multimedia application, you will find that all topics treated so far will become relevant. There will be a need to mix multiple media formats. You will have to find suitable codecs for your video. You will be asked whether search is possible. And, not the least important, you will have to balance navigation and presentation.

This chapter is based on the work we, that is my students, have been doing in the domain of cultural heritage. In the first section, we will introduce the notion of digital dossier and outline our general approach. We will then in section 2 look at some examples, and describe how we deploy concept graphs as a universal navigation tool for complex information spaces. Finally, in section 3, we will explore the options for presenting multimedia material and discuss the design issues as well as the technical issues that have arisen in the course of our work.

...

1

PDF

multimedia casus

You can learn a great deal about technology, but there is no meaning to that unless the technology is applied to produce something worthwhile. In this final chapter, the outline of a multimedia casus will be presented, that is a course in which students face the challenge of creating a veritable (intelligent) multimedia information system.

In the studyguide, the course is described as follows.

multimedia casus

The assignment in the multimedia casus is to develop a virtual environment for some cultural or governemental institute or company.

The practicum takes the form a stage, in which external supervision plays an important role.

In the multimedia casus, techniques learned in previous courses (see the afterthoughts) will be applied to create the application.

At the start of the course the actual assignment will be determined.

Examples of possible assignments are: the development of a virtual exposition hall for the Dutch Royal Museum of the Arts, a virtual city square, which gives information about both the present and the past, a virtual shop, with online buying facilities, or an online broker, which offers facilities for inspecting houses.

In effect, the availability of a representative of a cultural institute, industry, or governmental department is crucial, otherwise the assignment might easily degrade to the type of toy assignments so common in academia. Now, what is the challenge in such an assignment?

augmented information

In the research directions of section 8.1 the notion of augmented virtuality was introduced to clarify the duality between information and presentation. More in particular, it was argued that the use of VR makes no sense unless there is some added value, that is by using the rich presentation and interaction facilities that come with this technology.

In an abstract fashion, we may rephrase the assignment as follows:

Given an information space, create a VR that resolves the duality between information and presentation, using intelligent multimedia technology. The VR must offer access to all relevant information entities, organized in a suitable spatial layout, and must allow for presentations from a variety of perspectives, making full use of graphical and rich media facilities.

Below, we will see how this may work out for a concrete assignment.

...
front page of the INCCA website

2

project assignment -- present a complex information space

 Art is an interesting and complex phenomenon. No art, no culture! Hence, the preservation of collections of artworks is of crucial importance. The ICN (Netherlands Institute for Cultural Heritage) is a government-funded institute for the preservation of (dutch) cultural heritage. ICN gives advice, organises courses, does research, etcetera.

ICN is actively involved in the preservation of modern art, being project leader for INCCA (International Network for the Conservation of Contemporary Art), in the person of Tatja Scholte.

INCCA


In 1999, a group of eleven international modern art museums and related institutions applied to the European Commission (Raphael Programme) under the umbrella International Network for the Conservation of Contemporary Art (INCCA). The INCCA project was accepted and work started in January 2000 led by the organiser, the ICN (Netherlands Institute for Cultural Heritage) and the co-organiser, Tate, London.

The objectives of INCCA are phrased as follows.

objectives


INCCA's most important set of objectives, which are closely interlinked, focuses on the building of a website with underlying databases that will facilitate the exchange of professional knowledge and information. Furthermore, INCCA partners are involved in a collective effort to gather information directly from artists.

The INCCA web site contains a wealth of information about contemporary artists, as well as links to virtual collections of the works of a variety of artists, as for example Mondriaan. The way the virtual Mondriaan collection is presented is interesting in itself. It is a running display with iconic representations of his paintings. The speed of the display varies with the user's mouse movement, and at any time the user may select a painting to obtain more information about it. This particular site suggests where our intelligent multimedia approach may fit in.

Returning to the INCCA project once more, as its mission statement we read:

mission


INCCA's guiding mission is to collect, share and preserve knowledge needed for the conservation of modern and contemporary art.

By now, the outlines of our assignment should become clear. Our information space is information about modern and contemporary artists, in the form of digital representations of there work, photographs, audio recordings from interviews and written text. The project assignment is to organize (part of) this material in a virtual environment and to include interaction facilities that highlight particular aspects of this information.

At this stage it would be too ambitious to cover all the material in the INCCA database, so we should restrict ourselves to one or more smaller case studies. The challenge, obviously, is to create presentations with a solid narrative structure and to augment the presented material in a suitable manner, using intelligent multimedia technology. What is suitable, is part of the challenge!

project management -- roles

Can the challenge, stated above, be met? Well, there are many ways the project may loose its focus, or fail alltogether. Students should be aware of the fact that the challenge is real and that failure would bring about shame.

Since there are no golden rules for project management, the students themselves are responsible for keeping the project on track. In other words, project management is part of the experience. Here is a checklist.

checklist


  • roles -- create a team
  • project goal -- develop a vision
  • production -- construct the assets
  • quality assesment -- test and control
  • delivery -- present and archive
  • manage -- all along
  • document -- track project's history
The rule of the supervisor should be minimal, as a critical third party. The students work as a group, and they should take responsibility as a group, including the management of the project, assigning roles, and keeping track of progress. In such an approach intervision (students supervise one another) is a necessary mechanism in judging the final result of the project.

judgement


  • group -- (2) effort, 5 (product), 3 (documentation)
  • individual -- (4) responsibility, (3) productivity, (3) quality
On a scale of 0-10, both the group result and the individual efforts may be assigned a mark with proper weights, as indicated above. In addition, target deliverables should be defined to assure that the project meets its deadlines and to inspect the nature and quality of the students' work.

deliverables


  • group -- project plan, design, project report, product
  • individual -- detailed weekly account of activities
Dependent on the time available a schedule should be defined indicating when the deliverables should be ... delivered.

schedule

  1. project organisation
  2. project definition
  3. planning and design
  4. construction and development
  5. integration and delivery
  6. presentation and archiving
Is this a realistic setup? It should be. Besides, it is not the supervisor's responsibility, is it? It is first of all the responsibility of the students themselves!

peer reviews -- to control group dynamics

 Whether you are a student or responsible for supervising projects, this account of how our multimedia casus is organized should give you some indications of what you may encounter in a team project. But apart from the organizational issues, you should be aware of the group dynamics, that is the individual relations and clusters of persons that emerge during the process of development. In general it takes some time before the various roles are established, that is who takes leadership, who takes notes during meetings, and who does most of the technical work. Also, sometimes some of the more creative members of the tem are overshadowed by some of the more outspoken ones. Some people simply need to learn to assert themselves!

For a supervisor, it is often quite difficult to assess the contribution of each individual. I remedy this by having both informal peer reviews, in the group, as well as formal peer reviews, where marks must be given for responsibility, productivity and quality of the work. The order in which the students to be reviewed are presented to the reviewer is random, to avoid any bias due to presentation order.

I started using peer reviews about ten years ago, then informally, because I noticed that students could be much more direct in their criticisms that I dared to be. Recently, following suggestions from my collegue Johan Hoorn, who is an experimental psychologist, we formalized the procedure and introduced peer reviews for other courses as well, including the visual design course.

...
physical control cubehead-mounted display

3

example(s) -- tangible virtual museum

 Don't touch that! Keep your hands off! This is what you often hear parents shouting at their children in a museum or gallery exhibit. More often, however, precious artifacts, ceramics, porcelain or bronzes, are stored away in glass showcases, precluding any kind of physical interaction, and many times a proper look as well.

To remedy this situation, researchers from the Academia Sinica and the National Cheng Kung University have developed a tangible photorealistic virtual museum, a system for real-time interaction with photorealistic museum artifacts, which allows for an immersive experience using tangible interfaces, in the form of a 3D control cube (image left),  [Rosenblum and Macedonia (2005)]. The display is a kiosk-like system showing a panoramic view of the exhibit, augmented with a collection of perspective photographs for each object. The user may examine any of the objects by using a handheld control cube (PCC) to control size and rotation of the object.

As indicated, the system is not 3D mesh-based but image-based, which allows for high resolutions on mid-range platforms, which would not be feasible according to the authors, when using 3D modelling techniques.

research directions -- metaphors and interaction style

Given a problem statement as the one above, to present information about contemporary artists, how would you proceed? You might start by asking potential users, or stakeholders, how they would like the system to be. The answer you will get this way is likely to be disappointing. They will probably tell you that it must be like something they already know. So it might be better to rely on your own intuition and find a creative solution by choosing a fitting metaphor.

Let me give an example. In creating the digital dossier, a notion that will be explained in the next section, for the artist Marinus Boezem, as presented in the research directions of section 9.2, we choose the artist's atelier as a metaphor, and we used the spatial layout of the atelier as an organizational principle for presenting the information. In this, indeed very naturalistic, approach, we used pedestals to present the artworks, a file cabinet to present the textual information and a video projector to present the video recorded interview with the artist. The extent to which the virtual atelier does represent the artist's atelier faithfully is not important, in this context. What is important is whether the spatial metaphor did function as a valid organizational principle for presenting the information.

Instead of arguing whether this is the case or not, or whether the graphics chosen were right, etcetera, I would rather like to refer you to the literature, so that you can investigate the issues involved yourself.

In  [Preece et al. (1994)], it is observed that interface metaphors act as conceptual models to support particular tasks. For office tasks, for example, we have the wellknown desktop metaphor.  [Preece et al. (1994)] lists a number of such metaphors, for a variety of application domains:

application areametaphorfamiliar knowledge
operating environmentdesktopoffice tasks
spreadsheetsledger sheetcolumnar table
object-oriented environmentphysical worldreal world
hypertextnotecardsorganization of text
learning environmenttraveltours, guides, movement
file storagepilescategorizing
multimedia environmentsroomsspatial structures
cooperative workmulti-agentstravel agents, servants
In the most right column it is indicated why the metaphors should work, assuming real world situations that we are familiar with.

In some cases it is necessary to speak of a composite metaphor. For example, scrollbars are not easily to be found on your natural desktop. Form a cognitive perspective then, we may speak of multiple mental models.

When we look at what interaction styles are supported from a more technical perspective, we have following  [Preece et al. (1994)], the following options:

interaction styles


  • command entry
  • menus and navigation
  • forms fills and spreadsheets
  • natural language dialog
  • direct manipulation
However, each of these interaction styles may somehow be incorporated in the representation that we adopt for our metaphor.

2D vs 3D

 Surprisingly, each year that I start with another multimedia casus group, there is a discussion whether the application should be in 2D, using traditional web technology or flash, or 3D, using VRML or any other suitable 3D technology. My answer to the students objections, which can partly be explained by the fact that they fear the complexity of 3D, is flatly that anything that can be done in 2D can be done in 3D. But looking at the list of interaction styles above, I am tempted to add that a 3D representation allows for a more rich repertoire of interaction styles, such as spatial navigation. It would be interesting to investigate to what extent the interaction styles used in game playing can be incorporated in 'more serious' applications.

digital dossier(s)

After a first round of the multimedia casus, in which the students produced an application giving an overview of the INCCA information archive, the participants, but only incidental information about the artists and their artworks, we decided to focus on case studies of individual artists, and we introduced the notion of digital dossier:

digital dossier

Create a VR that realizes a digital dossier for a work of a particular artist. A digital dossier represents the information that is available for a particular work of art, or a collection of works, of a particular artist. The digital dossier should be multimedia-enhanced, that is include photographs, audio and other multimedia material in a compelling manner.

Like a medical dossier, the digital dossier was meant to give the information about the artist and the works of art readily at hand, so that it could effectively be used for the task of conservation and the re-installation of the artworks.

Since we were in doubt whether the phrase dossier actually existed in the English language, we looked it up in a dictionary:

Webster New World Dictionary


  • dossier (dos-si-er) [ Fr < dos (back); so named because labeled on the back ] a collection of documents concerning a particular person or matter
  • archive -- 1) a place where public records are kept ... 2) the records, material itself ...
We chose for the phrase digital dossier, and not for archive or library, to stress that our focus lies on presentational aspects. Although issues of data representation and content management are clearly important, our primary interest was with issues of presentation and navigation.

...
main nodeinterviews

4

the abramovic dossier

 For the 2004 autumn group, we decided to take the work of Marina Abramovic, a serbian-dutch artist who became wellknown in the seventies with performances with her partner Ulay, and has since then produced numerous installations, videos and performances with what I would like to call 'high existential impact'. The directive with which the students where set to work was, quoting Ted Nelson:

everything must be highly intertwinkled

Since virtual museums are by now a common phenomenon, and the virtual atelier for Marinus Boezem may be considered to be just a variant of this, the 2004 autumn group decided to explore alternative ways of presentation and navigation.

As material for the abramovic dossier there was an interview with Marina Abramovic from ICN, made in cooperation with the Dutch Foundation for the Visual Arts, and a great collection of videos from Montevideo. In addition, a transcription of the contents of the interview made by Michela Negrini, a student of media art at the University of Amsterdam, who also provided an interpretation as well as a categorization of the works of art. Given the material and the categories along which this material was classified, the students decided to explore the use of concept graphs as an instrument for navigating the information space.

navigation -- concept graphs

The reader has already encountered concept graphs in chapter 1, when the notions of multimedia, medium, television and communication were explained by indicating their relations to other concepts.

Concept-relation graphs are a familiar tool in linguistics and have also been used for a long time in Artificial Intelligence to describe the semantic relationships in complex domains. As a navigation instrument it is, to my knowledge only used in a kanji learning tool and the Visual Thesaurus. The Visual Thesaurus allows also for invoking Google image or document search from any of the elements of the concept graph.

...
presentation of video clips from Marina Abramovic

5

After the initial idea was there, one of the students of the group, Olaf van Zon, an AI student, managed to get a first version of a 3D concept graph working in VRML. This prototype implementation demonstrated the potential of the concept graph as a navigation instrument in the abramovic dossier.

presentation -- gadgets

The original idea of presenting information, that is the actual interview, the videos and images of the works of art, as well as the textual information, was to use rooms, where the information could be projected on the walls. The room metaphor, which obviously stems from the virtual museum approach, did however not seem appropriate since it conflicted with the concept graph used for navigation. After some discussion, information rooms were abandoned in favor of information gadgets, that could be expanded from and collapsed into the concept graph.

In the original abramovic dossier, the presentation gadget consists of three panes that can simultaneously show a video of the work, the interview, that is the fragment in which Abramovic speaks about that particular work, and the textual information related to the work and the interview. However, it appeared that in some cases there was not enough information, because the work was not spoken about in the interview, and in other cases there was too much information, for example multiple recordings or text documents. It was then decided to extend the presentation gadget with lists of alternative material that the user could select from and direct to one of the panes for further inspection.

To enable the user to focus on one of the panes, for example to get a better view of the video material a zoom in/out button was provided. All these enhancements, however, did complicate the interaction, as became clear when the abramovic dossier was presented at Montevideo.

In the course of the project, another interesting presentation feature was added, namely the reconstruction of one of the video installations in 3D, incidentally demonstrating the advantages of using 3D.

reconstruction -- recreating the installation

In discussing the abramovic dossier with Bart Rutten from Montevideo, who provided us with all the video material, another project was mentioned which was concerned with 3D-recordings/models of existing installations. Having full confidence in the technical capabilities of my students, I promised to show that such a reconstruction of an installation would naturally fit within our approach.

...
Reconstruction of Terra della Dea Madre in VRML.

6

The installation for which the reconstruction was made is Terra dea degli madre, and installation with two chairs and a television, which was exhibited in the Stedelijk Museum of Amsterdam, in 1986. As a starting point, we took a video produced at the time of the exhibition, which shows the installation in an exposition room in the Stedelijk Museum, and which contains, apart from comments from Abramovic, also the video shown on the televison in the installation.

At this point, we can only speculate how useful such a reconstruction can be as a tool for the conservator responsible for the re-installation, to play around with the presentation parameters, the positioning in space, the overall size, light and ambient effects.

style issues -- how to improve the dossier

The abramovic dossier does also provide a facility for search, as well as online help. However, as already mentioned, when demonstrating the application to the interested parties, that is ICN and Montevideo, a number of issues came along, that I will here summarize as a list of questions:

style issues


  • what icons should be used to identify the elements of the concept graph?
  • what categories and relationships are most appropriate?
  • how should the information be displayed, simultaneously or more focussed?
  • how do we allow the user to choose between multiple information items?
  • how do we avoid visually disturbing elements?
Obviously, although the abramovic dossier was very positively received, these issues must be dealt with to make it a success. Having a first prototype, we need to rethink our application, not only with regard to its style of presentation, but as we will discuss in section 10.3, also in terms of its underlying data representation.

...
no lighthalf lightfull light

7

example(s) -- conservator studio

 Ever thought of becoming a conservator? Seattle Artmuseum's Conservator Studio gives you the opportunity to explore this career options:

Explore four paintings from the Mexican Modernism exhibition through the eyes of a conservator (what's a conservator? you'll find that out too!). You'll have a new perspective on the paintings as well as how they are handled and prepared for display.

The illustrations above show what occurs when manipulating transmitted light on the painting Self-Protrait with Braid, oil on canvas, from the Mexican painter Frida Kahlo. As explained in the accompanying text: when a light is shone through this painting one can see that the hair and the flesh areas are painted with thin layers of paint.

These series of images are part of an interactive flash application developed by the Seattle Artmuseum to engage the general audience in the conservation of art, and to arouse an interest in art in general. The application allows the user to experiment with the various techniques used for the analysis and conservation of oil paintings.

research directions -- establishing usability

In the March 2005 volume of CACM, an assessment is given of the current state of user-centered design practice. User-centered design is, quoting  [UCD], a multi-disciplinary design approach based on an active involvement of users to improve the understanding of user and task-requirements, iterative design and evaluation. In the article, which is based on a survey among user-centered design practitioners, user-centered design is claimed to have been beneficial for, among others, customer satisfaction and enhanced ease of use. Other measures mentioned are mostly relevant for e-commerce applications, which, as the authors observe, have greatly bolstered the the appeal of usability and user-centered design, as users can take their business elsewhere with just one mouse click.

In our case, the competition is fortunately less threatening. Nevertheless, usability issues such as legibility of text, ease in navigation and adequate task support are equally relevant. As a first step after completing the abramovic dossier, we have developed a test-plan and a sample task, and (the students) executed two test-sessions with participants from ICN and Montevideo, who where asked to work with the system thinking aloud. The test-sessions were recorded on video, and the participants were requested to complete a questionnaire.

In  [UCD], a list of approaches is given, which were reported to have been used by the respondents of the survey:

user-centered design methods


field studies, user requirement analysis, iterative design, usability evaluation, task analysis, focus groups, formal/heuristic analysis, user interviews, prototype (without user testing), surveys, informal expert review, card sorting, participatory design

The three most frequently used methods in this list are, respectively, iterative design, usability evaluation and task analysis. These three methods were also considered to be important by the respondents. Frequently used, but not considered to be as important, were informal expert reviews. And less frequently used, but considered important, were field studies. This distinction can, according to  [UCD], attributed to cost-benefit trade-offs, since clearly field studies are much more costly.

Usability evaluation looks, according to  [Preece et al. (1994)] to issues such as:

usability evaluation


  • learnability -- time and effort to reach level of performance
  • throughput -- the amount of work done
  • flexibility -- accomodating changes in the task
  • attitude -- of users to the system
To conclude this section, let's take a closer look at task analysis.

task analysis

 Task analysis may be characterized as the decomposition of a task into subtasks or steps, to arrive at a sufficiently detailed description of the task and its relation to the environment.

In  [Welie et al. (1998)], a description is given of what might be understood as the task world ontology, the concepts and relations that play a role in performing a task analysis. The main concepts figuring in the task world ontology are, following  [Welie et al. (1998)]:

task world ontology


  • task -- activity performed by an agent to reach a certain goal
  • goal -- a desired state in the task world or system
  • role -- a meaningful collection of tasks
  • object -- refers to a physical or non-physical entity
  • agent -- an entity that is considered active
  • event -- a change in the state of the task world
As indicated in the diagram above, these concepts are related in various ways. Example relations include uses, triggers, plays, performed_by, has, etcetera.

...

8

Creating a task model based on this, or a similar, ontology may help us understand what a user needs to accomplish and how this may be supported by an information system. As such, creating a task model should be considered to be an essential ingredient of the software engineering life cycle,  [Eliens (2000)].

representation & interaction

In re-thinking the abramovic dossier, we first needed to re-establish what are our goals in developing this application and what are our primary data sources. The goal, first of all, is to support conservators in their task of preserving contemporary art, and to assist them with the re-installation of such artworks.

...
Our primary data sources are meta-information, coming from the INCCA initiative, and video-recorded artist interviews, which were initiated by ICN as a means to record information about contemporary art that would otherwise be lost. In addition we provide media-material, including images and video, that may be regarded as recordings of the works of art, as well as the textual interpretations and classifications that exist, or may be constructed from this material.

At this point, I may remark that one of the pitfalls in creating a dossier is to get trapped in the visually salient features of the dossier, the presentation of the artworks themselves, and forget about the primary focus of the dossier, to make all information accessible in an adequate manner. For many cultural heritage applications, which aim to present art to the layman, presenting the artwork is the primary focus, and giving access to the information context generally comes second.

For our next generation of digital dossiers, we decided to take the following steps:

next generation dossier(s)


  1. adaptation of representation to Dublin Core (+ annotation needed for presentation)
  2. XML-based content management, with php forms (extending what we have now)
  3. there should also be a possibility to present the information and material in a 'plain' web format
  4. as well as in (a new version of) 3D dossiers
  5. we should think about the proper presentation parameters.
Dublin Core is the standard used in the INCCA initiative, to record meta-information about existing information sources. See section 3.3 for a description of the Dublin Core element set and the Resource Description Framework (RDF) on which it is based.

For the abramovic dossier, a collection of record-like structures was developed, together with a simple content-management tool, written in PHP. This content-management system must be adapted to be compatible with the Dublin Core-based resource descriptions.

Further, we decided that, along with the 3D presentation of the dossier, it would be worthwhile to develop a conversion tool that produces standard web-technology based presentations as well. This approach allows us to assess the usability merits of the 3D dossiers in a comparative way.

Finally, as I indicated before, an important issue that must be resolved concerns the proper presentation parameters. What do we present to the user? And how do we allow the user to interact with the material presented?

content management and data representation

For developing the abramovic dossier, we have a fixed number of record-like structures:

structures


  • Video -- to display video fragment, including interviews
  • Picture -- to present pictures of the artwork
  • Artwork -- contains all information connected to a work of art
  • TextItem -- to present text, from the interview or any other source
  • MaterialItem -- to present information about material(s) used
  • GroupNode -- to combine nodes in the concpet graph
  • Information -- acts as the outer container for all nodes
All these structures support a set of common attributes, including shortName, longName, ID, connectedNodesIDs, and description. In addition the Video, Picture and Picture have fields allowing to show a preview image. And the Video, Picture and TextItem, also have a url field giving access to the actual information item.

The Information and GroupNode structures are used for creating the top-levels of the concept graph, whereas the other structures, such as the Video and TextItem give access to for example a fragment of an interview and its transcription.

Below an example is given of the data underlying the concept graph of the abramovic dossier:

concept graph



  Information {
   informationNodes [
    GroupNode {
  	ID "MAIN"
  	shortName "Main"
  	longName "Main"
  	urlModel "models/conceptGraph/main/modelMain.wrl"
  	description [ "Central information node" ]
  	connectedNodesIDs [ "ARTWORKS", "KEYWORDS", 
  	                         "INTERVIEWS", "REPORT" ]
  	}
    GroupNode {
  	shortName "Artworks"
  	longName "Artworks"
  	description [ "Node that connects to all the artworks" ]
  	ID "ARTWORKS"
  	connectedNodesIDs [ "MAIN", "TRANSITORY", 
  	              "ULAY", "VIDEOINSTALLATION", "ABRAMOVIC" ]
  	urlModel "models/conceptGraph/artworks/artworksGroup.wrl"
  	}
  	## ...
   ]
  }
The Information node collects all available nodes, and takes care of connecting the individual nodes, based on the information specified for each node.

As an example of an Artwork node, that is an element of the list of nodes in the Information node, look at: 


  Artwork {
    shortName "Terra degla Dea Madre"
    longName "Terra degla Dea Madre"
    description ["15:40 min, colour, sound."]
    ID "AV24"
    connectedNodesIDs ["VIDEOINSTALLATION", "DTV24", 
               "TTV24", "PV24", "CV24", "VV24", "G0"]
    urlPreviewImage "images/previewImages/AV24.jpg"
    widthPreviewImage 479
    heightPreviewImage 349
  }
This node is connected to many other nodes, giving access to the information items that belong to it, such as the video clips of the interview, shown below. 

  Video {
    ID "CV24"
    shortName "Interview clip Terra degla Dea Madre"
    longName "Interview clip showing Terra degla Dea Madre"
    url "interviewclips/interview_terra_degla.avi"
    width 320
    height 360
    urlPreviewImage "images/previewImages/interview_terra_degla.jpg"
    widthPreviewImage 320
    heightPreviewImage 240
    description [""]
    connectedNodesIDs ["CLIP", "AV24"]
  }
In the url field of this declaration, the actual video file is indicated, which should be displayed at a resolution of 320x360, as specified in the width and height fields.

And finally, as an example of a TextItem, consider: 


  TextItem {
    shortName "Instruction"
    longName "Green Dragon Lying instructions for the public."
    description ["Text explaining the way the public has to interact with the artwork."]
    ID "ITO05"
    connectedNodesIDs ["AO05", "INTERACTION"]
    url "text/AO05_instruction.txt"
  }
For constructing the abramovic dossier, Tim Verweij developed the content management tool, that allows the user to browse and edit existing nodes, and to insert new nodes into the graph.

integration with the Dublin Core

 The Dublin Core is a general resource description formalism, that allows for specifying resources in a variety of domains. See section 3.3. For INCCA the Dublin Core was chosen, not because it is the most suitable formalism, but because it may serve as the least common denominator, and agreement on anything else simply seemed to be impossible. As a reminder, the Dublin Core provides the following elements:

Dublin Core


  • title -- name given to the resource
  • creator -- entity primarily responsible for making the content of the resource
  • subject -- topic of the content of the resource
  • description -- an account of the content of the resource
  • publisher -- entity responsible for making the resource available
  • contributor -- entity responsible for making contributions to the content of the resource
  • date -- date of an event in the lifecycle of the resource
  • type -- nature or genre of the content of the resource
  • format -- physical or digital manifestation of the resource
  • identifier -- unambiguous reference to the resource within a given context
  • source -- reference to a resource from which the present resource is derived
  • language -- language of the intellectual content of the resource
  • relation -- reference to a related resource
  • coverage -- extent or scope of the content of the resource
  • rights -- information about rights held in and over the resource
Descriptions of items in the digital dossier should incorporate these elements, together with the attributes needed for the insertion of items in the concept graph and the presentation parameters, that are necessary for displaying the (media) material. Technically, the namespaces supported by RDF does allow for merging these different types of annotations. However, the challenge here is to derive the presentation attributes automatically, and to come up with a reasonable default for inserting these items in the concept graph.

...
location of Tower of Babel project

9

intelligent guidance -- I-GUARD

 Although digital archives or digital libraries are by no means a new phenomenon, our concept of digital dossiers contains a number of innovative elements. A digital dossier provides a unified information and presentation space. In this sense it differs significantly from a digital archive with a traditional web interface, where navigation and presentation are distinct. Digital dossiers allow to a much greater extent for an immersive experience of the information related to works of art. As such it is reminiscient to explorations in virtual archeology, our to our notion of virtual context, presented in section 8.1.

Working out the issues indicated above, that is the integration with the Dublin Core and providing suitable content management, is a matter of diligent software engineering. But what can we further do to support the construction of digital dossiers and improve the usability of such dossiers? And what are the scientific issues, worth to be investigated?

To indicate the research issues, let me first expand the cope of our project and re-define the goal of our research:

I-GUARD


Contemporary art is an intrinsic part of our cultural heritage. Installations, performances, video and other forms of media art, as for example web art, have the interest of a small group of adherents, but are in comparison with more traditional art forms, far more difficult to present to a general audience. Another problem presents itself, due to the type of materials used and the context-specific aspects of these art forms, in the conservation of the works.

In our research we address the issue of providing access to these contemporary art forms from a wide variety of perspectives, ranging from the interested layman to the expert that has to deal with archiving, conserving and the possible re-installation of the art works.

The acronym I-GUARD stands for Intelligent Guidance in Artist's Digital Dossiers, and refers to a project the aim of which is to arrive at a general framework for artist's digital dossiers, that provide intelligent guidance to both the expert user, responsible for the future re-installation of the work(s), and the interested layman, that wishes to get acquainted with a particular work or collection of works. In general, there are two techniques that we can apply to provide such guidance:

intelligent guidance


  • filtering the information space according to the user's perspective, and
  • intelligent agents, that (pro) actively aid the user in searching the information space.
Filtering the information space may be done by using techniques from formal linguistics to restrict the concept graph that defines the navigation structure, that is by stating assumptions with respect to the relevance of particular (linguistic) categories or elements from a user's perspective. Intelligent agents is an approach stemming from artificial intelligence which allows for providing guidance in a variety of ways, possibly even in an embodied form using a face or humanoid figure to give suggestions to the user on what interactions to perform. With the latter type of guidance we have already experimented in the Marinus Boezem dossier, as described in section 9.2. So let's look at what natural language technology has to offer.

natural language:

 Having a concept graph as a generic navigation device, it still remains a problem how to fill the concept graph with meaningful content, and how to indicate meaningful relations between the concepts and aspects covered by the nodes of the concept graph. In the abramovic dossier this was done by hand, based on information derived from a transcripion of an interview with the artist. (provided to us by ICN). Interviews with artists is one of the means ICN deploys to gain knowledge needed for the conservation of contemporary artworks. Such interviews provide a rich source of textual information, that includes both general viewpoints on the artist's oeuvre as well as specific constraints that adhere to the (re) installation of the work(s) of art.

What we should strive for is to derive both structure and content of the concept graph for a particular dossier (semi) automatically. Using a basic lexicon of terms and phrases related to contemporary art we should be able to generate a representation of the textual information that may serve as a basis for constructing the concept graph. This representation must contain an enumeration of the concepts, the relation between occurrences of concepts, as well as a reference to the work(s) of art to which the concepts apply.

Natural language processing technology may not only serve for the static analysis of the material, when the digital dossier is created, but also dynamically when the dossier is being used, to aid the user in finding relevant information. Research issues here are, on the one hand, the interpretation of user input (that is, loosely structured natural language), and on the other hand, filtering the concept graph representing the information space in such a way that it adequately reflects the user's interest or perspective.

In summary, from a research perspective, digital dossier(s) concern the following issues:

digital dossier(s)


  • representation of information of one or multiple works of art,
  • presentation of that information in a rich media presentation environment,
  • intelligent navigation and interaction, and
  • support for interaction with loosely-structured natural language.
And to conclude, digital dossiers will on the one hand contribute to making contemporary art forms accessible to a larger audience and on the other hand are explicitly meant to support the complex task of the conservation and re-installation of works of art in an effective manner.

...
outside view of Tower of Babel project

10

example(s) -- Tower of Babel

 In the Tower of Babel project, shown above, multimedia material was projected from within buildings, on the windows, to the outside. Local citizens in a neighbourhood in Amsterdam where approached to submit material that expressed their emotions of daily life, with the question what moves you. The text and photograms could be submitted either by email or SMS. Also workshops were held, during which particpants could develop material. This material was then edited and prepared for projection, using 40 carroussel dia-projectors, taking about 2000 images, and six beamers projecting images and video. Also sound material, that was collected in the same manner, was being used during the projection.

...
Inside view of Tower of Babel project.

11

The centre of the location, a somewhat impoverished neighbourhood near the centre of Amsterdam, is a building dating from 1926, originally an antroposofic temple, that once served as a cinema for avant-garde movies, and is now being used as a library. The buildings surrounding it are, if not split up into apartments, being used as a local youth centre, a city archive and another library.

research directions -- media art

 In a recent symposium on the preservation of contemporary media art, a number of institutions presented their projects, ranging from more technical topics, such as the conservation of videotapes and the mass storage of digital material to the conceptual issues in capturing new media, the variety of media formats and the need to record and maintain meta data about the artworks and related information.

To get an idea what the phrase media art encompasses, have a look at the circumscription given in the WikiPedia: new media art is a generic term used to describe art related to, or created with, technology invented or made widely available since the mid-20th Century, including technology stemming from telecommunications, mass media and digital modes of delivery the artworks Below, the disciplines that belong to this form of art are listed, together with their entries in the WikiPedia, in an abbreviated form:

(new) media art


  • audio art -- no definition available
  • computer art -- any art in which computers played a role in production or display of the artwork.
  • digital art -- art created on a computer in digital (that is, binary) form.
  • electronic art -- entry to game producer, should be Leonardo.
  • generative art -- art or design generated, composed, or constructed through computer software algorithms, or similar mathematical or mechanical autonomous processes
  • hacktivism -- the writing of code, or otherwise manipulating bits, to promote political ideology
  • interactive art -- a piece of art that involves the spectator in some way.
  • internet art -- art or, more precisely, cultural production which uses the Internet as its primary medium and, more importantly, its subject.
  • performance art -- art where the actions of an individual or a group at a particular place and in a particular time, constitute the work.
  • robotic art -- page does not exist
  • software art -- is an intersection of two almost non-overlapping realms: software and art.
  • video art -- is a subset of artistic works which relies on "moving pictures" and is comprised of video and/or audio data.
  • video game art -- involves the use of a computer game for the creation of a digital artwork.
By the nature of the WikiPedia, to which every user can contribute entries, this list nor the defining entries are by any means authorative. Nevertheless, it does provide an overview and may serve as a starting point for further research.

development(s) -- hybrid multimedia

 In chapter 1, we introduced the notion of digital convergence to explain the occurrence of the great variety of elements of multimedia applications, from a technical perspective. From an aesthetic perspective, this great variety of elements may easily lead to chaos, unrelatedness or divergence, where meaning gets lost in a multitude of perspective(s). To cut a long deliberation short, for simplicity, let's assume that meaning lies in the context, the story or narrative structure.

For 2D images,  [Kress and van Leeuwen (1996)] identify narrative elements, that is relations between objects in the image that suggest a story, such as a diagonal line from a person to a door, or a relation of an object to the viewer, such as a gaze towards the viewer, a technique that has been used only since late renaissance painting.

More than paintings or 2D images, film is the medium for conveying narrative structures. The art of storytelling in film has been perfected in such a way that Hollywood films may seem more real than life. However, as emphasized in  [Bolter and Grusin (2000)], this is not due to any inherent form of naturalism, but to the fact that we have got accustomed to the conventions applied, that is the techniques of cutting, montage, camera movements, close-ups, etcetera. In a highly recommended book,  [Arnheim (1957)], Rudolf Arnheim gives an extensive analysis of the principles of montage and film technique, and he explains why film is such an effective medium:

frame(s) of reference


It is one of the most important formal qualities of film that every object that is reproduced appears simultaneously in two entirely different frames of reference, namely the two-dimensional and the three-dimensional, and that as one identical object it fulfills two different functions in the two contexts.

Due to the subtle play between these two frames of reference film may be considered an art form, and as such perhaps the dominant art form of the 20th century. As a mass medium, film may be characterized by what Arnheim, following Benjamin, called the aesthetics of shock, replacing reflective distance with immersive thrill. As an art form, however, it is the dominant paradigm for aesthetic awareness, lacking however still one dimension, interactive dynamics.

As observed in  [Bolter and Grusin (2000)], interaction is what distinguishes video games from film. Current day technology allows for high-resolution photorealist graphics, that make video games or virtual applications almost indistinguishable from film. Virtual reality technology as applied in video games adds arbitrary choice of perspective, as exemplified in first-person shooters or fly-overs, as well as an arbitrary mix of the imaginary and real, as in CG movies, in an interactive fashion.

Now, should we take the aesthetics of interactive video games as the standard for interactive applications? Not necessarily, since the naturalism strived for in most games may at best be characterized as naive realism, mostly photorealism. As observed in  [Kress and van Leeuwen (1996)], realism is a social construct, and hence the program for developing an aesthetics for interactive applications should perhaps include the development of appropriate realisms. Again with an eye to the history of art, where we have for example impressionism, cubism, expressionism, as a guideline in the design of interactive systems, it might be even better to look for appropriate interaction-isms, styles of developing interactive systems and games from a particular perspective. Not excluding provocative perspectives! Cf.  [Burger (1981)].

Where an arbitrary interactive system may differ from a game played for entertainment is obviously the actual outcome, the value attributed to that in the real world, and probably the effort required and the possible consequences. You would not like to run the risk to die a virtual death when answering your email, would you? However, when interactive systems replace task-bound functionality with fun, the difference becomes less clear.

As we indicate in  [Eliens & Chang (2007)], one element not sufficiently captured by a classic game model, as introduced in  [Juul (2005)], is the narrative aspect of the game play. To quote  [Juul (2005)]:

rules vs fiction


Game fiction is ambiguous, optional and imagined by the player in uncontrollable and unpredictable ways, but the emphasis on fictional worlds may be the strongest innovation of the video game.

We may observe that many games already have a strong relation to reality in what narrative context they supply, or else in the realities of the media industry, in particular Hollywood. For serious interactive systems, we may assume an even stronger and in some sense more straightforward relation with reality, by the use of media content that is relevant for the life of the individual.

All these aspects of playing games are clearly relevant for the new interactive systems, which appeal more to play than task-oriented behavior. For example rules may be used to describe the visual characteristics of a system (e.g. the display of images as a flow in a particle system), outcome may be regarded as the benefits of the system (e.g. social awareness), value may include the risks of the system (e.g. a transgression of privacy), efforts is important when asking for contributions from the user (e.g. as image material to be displayed in the system), attachment may result when the system is installed (e.g. when people look forward to find new information), and finally consequences must be considered when a system is installed and used (e.g. interaction between people may actually change when they get to know eachother, for better or worse).

Given the large variety of games, including first person shooters, role-playing games, strategy games and decision-making simulation games, we can distinguish between a range of degrees of interaction, direct interaction, on the one hand, as for example in first person shooter and indirect interaction, on the other hand, as for example in simulation games, or role-playing games where the individual actions may contribute to a plot such that the effects will become visible at a later time. Where in game playing the variety of interaction modes seems to be well understood within each community of game players, for the development of more general interactive systems we will have to think seriously whether the target user will be able to learn the various modes of interaction, either by explicit instruction or during play. And as designers we must be concerned with the rules of interaction as well as issues of visualisation and interaction mappings, that is in other words which affordances the application offers for a particular group of users.

dynamic contribution(s)

Another potential source of confusion lies in where the material comes from. Not in the sense of network transport or local storage of the platform of delivery, as discussed in the context of convergence of delivery in section 1.2, but in terms of authorship, which in our participatory culture, where users contribute content may result in a great variety of forms and formats. To develop multimedia applications and games that accomodate contributive authorship by a community of users is the great challenge for the next era (period).

In  [Eliens et al. (2007c)] we wrote: We explored the use of AJAX and web services in an X3D/VRML implementation of PANORAMA, a system meant to support social awareness in a work environment. As explained in section 5.4, PANORAMA represents casual encounters in the work environment and displays self reflections, that is postcards and other item contributed by employees, at a large display in a central public space.

...

12

The figure above, taken from  [Si & Eliens (2007)], illustrates the architecture of an AJAX-based web implementation of PANORAMA, which includes facilities for game playing as occasional battle(s), using a PHP server and the google GWT toolkit to allow users to contribute their image material, video's and whatever else.

...

13

questions

10. application development

concepts


technology


projects & further reading

As a project, develop a data format for text, images and video in XML, and implement stylesheets in XSLT to convert the format for display, for example in HTML frames or using SMIL.

You may further explore the formulation of criteria for selecting software and tool support for developing multimedia applications.

For further reading I suggest, apart from the manuals and learning materials that come with your tools, to study example projects and in particular the workflow, that is the dependencies between stages in the production, as for example explained in  [McCuskey (2002)].

the artwork

  1. website of Montevideo Collection Catalogue. To avoid being parochial here, I should also mention similar institutes abroad, such as Electronic Arts Intermix from New York, USA, and LUX, from London, UK.
  2. website of INCCA.
  3. tangible virtual museum -- from  [Rosenblum and Macedonia (2005)], see section 10.1.
  4. digital dossier -- concept graph for abramovic dossier, see section 10.2.
  5. digital dossier -- presentation gadget in abramovic dossier, with video of Relation in Time, with Ulay.
  6. digital dossier -- installation Terra dea degli madre, as 3D model.
  7. conservator's studio -- Self-Portrait with Braid, see section 10.2
  8. diagram -- task world ontology,  [Welie et al. (1998)].
  9. tower of babel -- location where the event took place, see below.
  10. tower of babel -- projection of tower of babel project, see section 10.3, submitted by Katelijne Arts. The project is a concept of Katelijne Arts, Tineke Goemans, Franka van de Goor, Leidi Haaijer en Bert Vogels.
  11. tower of babel -- a view from the inside of the building.
  12. PANORAMA architecture -- from  [Si & Eliens (2007)].
  13. signs -- sports,  [ van Rooijen (2003)], p. 278, 279.
The artwork for this chapter is selected to emphasize variety and experiment. The collection of Montevideo contains a great number of works from the early history of video art, including the works of Nam June Paik and Bill Viola. Yet, despite the experimental flavor of these works, contemporary media art shows a strong sense of context, experience and communication, as demonstrated for example in the tower of babel project. The issues of preservation we dealt with in this chapter, may now, to conclude this chapter, be summarized as: how do we preserve the context of experience of contemporary media art?
[] readme course(s) preface I 1 2 II 3 4 III 5 6 7 IV 8 9 10 V 11 12 afterthought(s) appendix reference(s) example(s) resource(s) _

(C) Æliens 23/08/2009

You may not copy or print any of this material without explicit permission of the author or the publisher. In case of other copyright issues, contact the author.