part iii. multimedia information retrieval
.. my history might well be your future ...
ted nelson
|
|
|
|
|
|
|
|
|
2
reading directives
perspectives
perspectives -- multimedia information retrieval
- application(s) -- digital dossier
- psychological -- focus
- experimental -- user interaction
- algorithmic -- (information) access
- system -- unified presentation space
- presentation -- embodied agents
- search -- semantic annotation
- commercial -- future systems
essay topics
For further study you may want to look at algorithms for analyzing content, annotation schemes for pareticular application domains, or the presentation issues mentioned before. Possible essay titles are:
- searching the web -- searching for images, video and sound
- finding a tune -- mobile music search services
|
|
|
|
3
the artwork
- kata -- japanese martial arts picture.
- signs -- japanese coats of arms,
[ Signs] , p. 140, 141. - photographs -- Jaap Stahlie, two early experiments (left, and right)
5
information retrieval
information retrieval is usually an afterthought
learning objectives
After reading this chapter you should be able to describe scenarios for information retrieval, to explain how content analysis for images can be done, to characterize similarity metrics, to define the notions of recall and precision, and to give an example of frequence tables, as used in text search.
Searching for information on the web is cumbersome. Given our experiences today, we may not even want to think about searching for multimedia information on the (multimedia) web.
Nevertheless, in this chapter we will briefly sketch one of the possible scenarios indicating the need for multimedia search. In fact, once we have the ability to search for multimedia information, many scenarios could be thought of.
As a start, we will look at two media types, images and documents. We will study search for images, because it teaches us important lessons about content analysis of media objects and what we may consider as being similar. Perhaps surprisingly, we will study text documents because, due to our familiarity with this media type, text documents allow us to determine what we may understand by effective search.
|
|
|
|
|
|
1
|
|
|
|
|
|
|
|
|
10
questions
5. information retrieval
concepts
- How would you approach content-based description of images?
- What is the difference between a metric approach and the transformational approach to establishing similarity between images?
- What problems may occur when searching in text or document databases?
technology
- Give a definition of: shape descriptor and property descriptor. Give an example of each.
- How would you define edit distance?
- Characterize the notions precision and recall.
- Give an example (with explanation) of a frequency table.
projects & further reading
As a project, you may implement simple
image analysis algorithms that, for example, extract
a color histogram, or detect the presence of
a horizon-like edge.
video annotation requires a logical approach to story telling
learning objectives
After reading this chapter you should be able to
explain the difference between content and meta information,
to mention relevant content parameters for audio,
to characterize the requirements for video libraries,
to define an annotation logic for video,
and to discuss feature extraction in samples of musical material.
Current technology does not allow us to extract information
automatically from arbitrary media objects.
In these cases, at least for the time being,
we need to assist search by annotating content
with what is commonly referred to as meta-information.
In this chapter, we will look at two more media types,
in particular audio and video.
Studying audio, we will learn how we may combine
feature extraction and meta-information to define a
data model that allows for search.
Studying video, on the other hand,
will indicate the complexity of devising a
knowledge representation scheme that captures
the content of video fragments.
Concluding this chapter, we will discuss an architecture
for feature extraction for arbitrary media objects.
1 13
concepts technology
As a project, think of implementing musical similarity matching,
or developing an application retrieving video fragments
using a simple annotation logic.
effective retrieval requires visual interfaces
learning objectives
After reading this chapter you should be able to
dicuss the considerations that play a role in
developing a multimedia information system,
characterize an abstract multimedia data format,
give examples of multimedia content queries,
define the notion of virtual resources,
and discuss the requirements for networked virtual
environments.
From a system development perspective, a multimedia
information system may be considered as a multimedia
database,
providing storage and retrieval facilities for media objects.
Yet, rather than a solution this presents us with a
problem,
since there are many options to provide such storage facilities
and equally many to support retrieval.
In this chapter, we will study the architectural issues
involved in developing multimedia information systems,
and we will introduce the notion of media abstraction
to provide for a uniform approach to arbitrary media objects.
Finally, we will discuss the additional problems
that networked multimedia confront us with.
1 10
concepts technology
As a project, you may implement a multi-player
game in which you may exchange pictures and videos,
for example pictures and videos of celebrities.
the artwork
The art opening this chapter belongs to
the tradition of 20th century art.
It is playful, experimental,
with strong existential implications,
and it shows an amazing variety of styles.
6
content annotation
questions
6. content annotation
projects & further reading
the artwork
Opening this chapter are examples of
design of the 20th century,
posters to announce a public event like
a theatre play, a world fair, or a festival.
In comparison to the art works
of the previous chapter,
these designs are more strongly expressive
and more simple and clear in their message.
Yet, they also show a wide variety of styles
and rethorics to attract the attention of the audience.
Both the faces and the mouth are examples of
using body parts in contemporary art.
The page of the comic book version of City of Glass,
illustrates how the 'logic' of a story can be visualised.
As an exercise, try to annoyaye the sequence of frames
from the City of Glass can be described
using the annotation logic you learned in this chapter.
The modern art examples should interesting by themselves.
7
information system architecture
questions
7. information system architecture
projects & further reading
the artwork
Opening this chapter are examples of dutch design,
taken from the book False Flat,
with the somewhat arrogant subtitle
why is dutch design so good?.
It is often noted that dutch design is original,
functional and free from false traditionalism.
Well, judge for yourself.
(C) Æliens 18/6/2009
You may not copy or print any of this material without explicit permission of the author or the publisher. In case of other copyright issues, contact the author.