<s5>Æliens</s5> -- slide(s)

learning objectives

After reading this chapter you should be able to dicuss the considerations that play a role in developing a multimedia information system, characterize an abstract multimedia data format, give examples of multimedia content queries, define the notion of virtual resources, and discuss the requirements for networked virtual environments.

From a system development perspective, a multimedia information system may be considered as a multimedia database, providing storage and retrieval facilities for media objects. Yet, rather than a solution this presents us with a problem, since there are many options to provide such storage facilities and equally many to support retrieval.

In this chapter, we will study the architectural issues involved in developing multimedia information systems, and we will introduce the notion of media abstraction to provide for a uniform approach to arbitrary media objects.

Finally, we will discuss the additional problems that networked multimedia confront us with.

...

issues

multimedia storage and retrieval -- homegrown, third-party and legacy sources
information architecture -- common format, native format, hybrid
media abstraction -- unified indexes, query relaxation

content organisation

autonomy -- index per media type
uniformity -- unified index
hybrid -- media indexes + unified index

Principle of Uniformity

... from a semantical point of view the content of a multimedia source is independent of the source itself, so we may use statements as meta data to provide a description of media objects.

from a semantical point of view the content of a multimedia source is independent of the source itself.

use statements as meta data

$md(o)$ -- metadata associated with media object o

tradeoffs

metadata can be stored using standard relational and OO structures
manipulating metadata is easy
feature extraction is (!) straightforward
is it?

software architecture

a database of media object, supporting
operations on media objects, and offering
logical views on media objects

information retrieval cycle

specification of the user's information need
translation into query operations
search and retrieval of media objects
ranking according to likelihood or relevance
presentation of results and user feedback
resulting in a possibly modified query

despite high interactivity, access is difficult;
quick response is and will remain important!

...

media abstraction

state -- smallest chunk of media data
feature -- any object in a state
attributes -- characteristics of objects
feature extraction map -- to identify content
relations -- to capture state-dependent information
(inter)relations between 'states' or chunks

example -- image database


  states: { pic1.gif,...,picn.gif } 
  features: names of people 
  extraction: find people in pictures 
  relations: left-of, ...

example -- video database


  states:  set of frames 
  features:  persons and objects
  extraction:  gives features per frame 
  relations:  frame-dependent and frame-independent information
  inter-state relation:  specifies sequences of frames

simple multimedia database

a finite set $M$ of media abstractions

structured multimedia database

equivalence relations --to deal with synonymy
partial ordering -- to deal with inheritance
query relaxation -- to please the user

...

SMDS -- functions


  Type: object  $|->$  type 
  ObjectWithFeatures:  $f |-> { o |$  object o contains  $f }$  
  ObjectWithFeaturesAndAttributes:  $(f,a,v) |-> { o |$  o contains f with  $a=v }$  
  FeaturesInObject:  $o |-> { f | o$  contains  $f }$  
  FeaturesAndAttributesInObject:  $o |-> { (f,a,v) | o$  contains  $f$  with  $a=v }$

SMDS-SQL

SELECT -- media entities

m -- if m is not a continuous media object
$m:[i,j]$ -- m is continuous, $i,j$ integers (segments)
$m.a$ -- m is media entity, a is attribute

FROM

<media><source><M>

WHERE

term IN funcall

example


    SELECT M
    FROM   smds source1 M
    WHERE  Type(M) = Image AND
  	 M IN ObjectWithFeature("Dennis") AND
  	 M IN ObjectWithFeature("Jane") AND
  	 left("Jane","Dennis",M)

hybrid representations: HM-SQL

express queries in specialized language
perform operations (joins) between SMDS and non-SMDS data

differences

function calls are annotated with media source
queries to non-SMDS data may be embedded

example HM-SQL


   SELECT M
   FROM smds video1, videodb video2
   WHERE M IN smds:ObjectWithFeature("Dennis") AND
         M IN videodb:VideoWithObject("Dennis")

digital libraries

Digital libraries are constructed -- collected and organized -- by a community of users. Their functional capabilities support the information needs and users of this community. Digital libraries are an extension, enhancement and integration of a variety of information institutions as physicalplaces where resources are selected, collected, organized, preserved and accessed in support of a user community.

... federated structures that provide humans both intellectual and physical access to the huge and growing worldwide networks of information encoded in multimedia digital formats.

digital libraries (5S)

streams: (content) -- from text to multimedia content
structures: (data) -- from database to hypertext networks
spaces: (information) -- from vector space to virtual reality
scenarios: (procedures) -- from service to stories
societies: (stakeholders) -- from authors to libraries


   D-Lib Forum -- www.dlib.org
   Informedia -- www.informedia.cs.cmu.edu

...

networked multimedia

real-time transmission of continuous media information (audio, video)
substantial volumes of data (despite compression)
distribution-oriented -- e.g. audio/video broadcast

network criteria

throughput -- bitrates, burstiness
transmission delay -- including signal propagation time
delay variation -- jitter
error rate -- data alteration, loss

multicasting and broadcasting capabilities
document caching

Quality of Service

Quality of Service is a concept based on the statement that not all applications need the same performance from the network over which they run. Thus, applications may indicate their specific requirements to the network, before they actually start transmitting information data.

QoS requirements

hard requirements
guidance for optimizing internal resources
criteria for acceptance

...

virtual objects

$VO = { (O_i,Q_i,C_i) | 1 <= i <= k }$

where

$C_1,...,C_k$ -- mutually exclusive conditions
$Q_1,...,Q_k$ -- queries
$O_1,...,O_k$ -- objects

...

networked virtual environments

shared sense of space -- room, building, terrain
shared sense of presence -- avatar (body and motion)
shared sense of time -- real-time interaction and behavior

a way to communicate -- by gesture, voice or text
a way to share ... -- interaction through objects

challenges

network bandwidth -- limited resource
heterogeneity -- multiple platforms
distributed interaction -- network delays
resource management -- real-time interaction and shared objects
failure management -- stop, ..., degradation
scalability -- wrt. number of participants

manage dynamic shared state

...

the Java Media Framework, and
the DLP+X3D platform

java Media Framework

The JavaTM Media APIs meet the increasing demand for multimedia in the enterprise by providing a unified, non-proprietary, platform-neutral solution. This set of APIs supports the integration of audio and video clips, animated presentations, 2D fonts, graphics, and images, as well as speech input/output and 3D models. By providing standard players and integrating these supporting technologies, the Java Media APIs enable developers to produce and distribute compelling, media-rich content.

recommender economy

cross sale -- users who bought A also bought B
up sale -- if you buy A and B together ...

recommender model


  U = user
  I = item
  B = behavior
  R = recommendation
  F = feature

observations -- U \* I \* B
recommendations -- U \* I


  B = [ time = 20sec, rating = r ]
  F = [ artist = rembrandt, topic = portrait ]
  R = [ artist(rembrandt) = r, topic(portrait) = r ]


  A = [  p_{1}, p_2 , ... ]
  where p_{k} = [ f_1 = v_1, f_2 = v_2, ... ]

with as an example


   A_{nightwatch} = [ artist=rembrandt, topic=group ]
   A_{guernica} = [ artist=picasso, topic=group ]

...

users, artworks and properties

distance metric


       d(x,y) = d(y,x)
       d(x,y) <= d(x,z) + d(z,y)
       d(x,x) = 0

Æliens

2009

7

information system architecture

information system architecture

...

...

...

...

...

virtual objects

...

...

...

...

...

7. information system architecture

projects & further reading

the artwork

Æliens

2009

information system architecture

virtual objects

document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('@tmp-7.html#slide-'); } document.write('decision-model-privacy'); if (!slidemode) document.write('.html');document.write('>');

document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('@tmp-7.html#slide-'); } document.write('decision-model-space'); if (!slidemode) document.write('.html');document.write('>');

document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('@tmp-7.html#slide-'); } document.write('decision-behavior'); if (!slidemode) document.write('.html');document.write('>');

document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('@tmp-7.html#slide-'); } document.write('decision-chip-collection'); if (!slidemode) document.write('.html');document.write('>');

document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('@tmp-7.html#slide-'); } document.write('description-distance'); if (!slidemode) document.write('.html');document.write('>');

document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('@tmp-7.html#slide-'); } document.write('decision-dimensions'); if (!slidemode) document.write('.html');document.write('>');

document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('@tmp-7.html#slide-'); } document.write('decision-interpr'); if (!slidemode) document.write('.html');document.write('>');

projects & further reading