introduction multimedia

[] readme preface 1 2 3 4 5 6 7 appendix checklist powerpoint resources director

media abstractions

Let's have a closer look at media abstractions. How can we capture the characterization of a variety of media types in one common media abstraction. A definition of such a media abstraction is proposedin [MMDBMS]. Leaving the formal details aside, a media abstraction has the following components:

media abstraction

state -- smallest chunk of media data
feature -- any object in a state
attributes -- characteristics of objects
feature extraction map -- to identify content
relations -- to capture state-dependent information
(inter)relations between 'states' or chunks

Now, that characterization is sufficiently abstract, and you may wonder how on earth to apply this to an actual media database.

However, before giving some examples, we must note that the feature extraction map does not need to provide information about the content of a chunk of media data automatically. It may well be a hand-coded annotation.

Our first example is an image database.

example -- image database


  states: { pic1.gif,...,picn.gif } 
  features: names of people 
  extraction: find people in pictures 
  relations: left-of, ...

In an image database it does not make much sense to speak about relations between 'states' or chunks of media data, that is the images. %5

For our next example though, video databases, it does make sense to speak about such relations, since it allows us to talk about scenes as sequences of frames.

example -- video database


  states:  set of frames 
  features:  persons and objects
  extraction:  gives features per frame 
  relations:  frame-dependent and frame-independent information
  inter-state relation:  specifies sequences of frames

Now, with this definition of media abstractions, we can define a simple multimedia database, simply as

simple multimedia database

a finite set $M$ of media abstractions

But, following [MMDBMS], we can do better than that. In order to deal with the problems of synonymy and inheritance, we can define a structured multimedia database that supports:

structured multimedia database

equivalence relations --to deal with synonymy
partial ordering -- to deal with inheritance
query relaxation -- to please the user

Recall that we have discussed the relation between a 'house of prayer' and 'church' as an example of synonymy in section 4.3. As an example of inheritance we may think of the relation between 'church' and 'cathedral'. Naturally, every cathedral is a church. But the reverse does not necessarily hold. Having this information about possible equivalence and inheritance relationships, we can relax queries in order to obtain better results. For example, when a user asks for cathedral in a particular region, we could even notify the user of the fact that although there are no cathedrals there, there are a number of churches that may be of interest. (For a mathemathical characterization of structured multimedia databases, study [MMDBMS].)

query languages

Having media abstractions, what would a query language for such a database look like? Again, following [MMDBMS], wemay extend SQL with special functions as indicated below:

SMDS -- functions


  Type: object  $|->$  type 
  ObjectWithFeatures:  $f |-> { o |$  object o contains  $f }$  
  ObjectWithFeaturesAndAttributes:  $(f,a,v) |-> { o |$  o contains f with  $a=v }$  
  FeaturesInObject:  $o |-> { f | o$  contains  $f }$  
  FeaturesAndAttributesInObject:  $o |-> { (f,a,v) | o$  contains  $f$  with  $a=v }$

Having such functionswe can characterize an extension of SQL, which has been dubbed SMDS-SQL in [MMDBMS], as follows.

SMDS-SQL

SELECT -- media entities

m -- if m is not a continuous media object
$m:[i,j]$ -- m is continuous, $i,j$ integers (segments)
$m.a$ -- m is media entity, a is attribute

FROM

<media><source><M>

WHERE

term IN funcall

As an example, look at the following SMDS-SQL snippet.

example


    SELECT M
    FROM   smds source1 M
    WHERE  Type(M) = Image AND
  	 M IN ObjectWithFeature("Dennis") AND
  	 M IN ObjectWithFeature("Jane") AND
  	 left("Jane","Dennis",M)

Note that M is a relation in the image database media abstraction, which contains one or more images that depict Jane to the left of Dennis. Now, did they exchange the briefcase, or did they not?

When we do not have a uniform representation, but a hybrid representation for our multimedia data instead, we need to be able to: express queries in specialized language, and to perform operations (joins) between SMDS and non-SMDS data.

hybrid representations: HM-SQL

express queries in specialized language
perform operations (joins) between SMDS and non-SMDS data

Our variant of SQL, dubbed HM-SQL, differs from SMDS-SQL in two respects: function calls are annotated with media source, and queries to non-SMDS data may be embedded.

differences

function calls are annotated with media source
queries to non-SMDS data may be embedded

As a final example, look at the following snippet:

example HM-SQL


   SELECT M
   FROM smds video1, videodb video2
   WHERE M IN smds:ObjectWithFeature("Dennis") AND
         M IN videodb:VideoWithObject("Dennis")

In this example, we are collecting all video fragments with Dennis in it, irrespective of where that fragment comes from, an (smds) database or another (video) database.

research directions -- digital libraries

Where media abstractions, as discussed above, aremeant to be technical abstractions needed for uniform access to media items, we need quite a different set of abstraction to cope with one of the major applications of multimedia information storage and retrieval: digital libraries.

According to [IR], digital libraries will need a long time to evolve, not only because there are many technical hurdles to be overcome, but also because effective digital libraries are dependent on an active community of users:

digital libraries

Digital libraries are constructed -- collected and organized -- by a community of users. Their functional capabilities support the information needs and users of this community. Digital libraries are an extension, enhancement and integration of a variety of information institutions as physicalplaces where resources are selected, collected, organized, preserved and accessed in support of a user community.

The occurrence of digital libraries on the web is partly a response to advances in technology, and partly due to an increased appreciation of the facilities the internet can provide. From a development perspective, digital libraries may be regarded as:

... federated structures that provide humans both intellectual and physical access to the huge and growing worldwide networks of information encoded in multimedia digital formats.

Early research in digital libraries has focussed on the digitization of existing material, for the preservation of our cultural heritage, as well as on architectural issues for the 'electronic preservation', so to speak, of digital libraries themselves, to make them "immune to degradation and technological obsolescence", [IR].

To bring order in the variety of research issues related to digital libraries, [IR] introduces a set of abstractions that is know as the 5S model:

digital libraries (5S)

streams: (content) -- from text to multimedia content
structures: (data) -- from database to hypertext networks
spaces: (information) -- from vector space to virtual reality
scenarios: (procedures) -- from service to stories
societies: (stakeholders) -- from authors to libraries

These abstractions act as "a framework for providing theoretical and practical unification of digital libraries". More concretely, observe that the framework encompasses three technical notions (streams, structures and spaces; which correspond more or less with data, content and information) and two notions related to the social context of digital libraries (scenarios and societies; which range over possible uses and users, respectively).

For further research you may look at the following resources:


   D-Lib Forum -- http://www.dlib.org
   Informedia -- http://www.informedia.cs.cmu.edu

The D-Lib Forum site gives access to a variety of resources, including a magazine with background articles as well as a test-suite that may help you in developing digital library technology. The Informedia site provides an example of a digital library project, with research on, among others, video content analysis, summarization and in-context result presentation.

[] readme preface 1 2 3 4 5 6 7 appendix checklist powerpoint resources director

eliens@cs.vu.nl

draft version 1 (16/5/2003)