Integrating user-generated metadata into video collections
Michiel Hildebrand
,
VU University Amsterdam
http://www.flickr.com/photos/fransdewit/4586821787
Analysis of search behavior at Netherlands Institute for Sound & Vision*
People predominantly order video fragments
broadcast (33%)
stories (17%)
fragments (49%)
Finding fragments takes much longer than finding broadcasts
stories (2x)
fragment (3x)
Vocabulary mismatch between searchers and cataloguers
35% of clicked results are
not
found by title or thesaurus term
*
Search Behavior of Media Professionals at an Audiovisual Archive: A Transaction Log Analysis
B. Huurnink, L. Hollink, W. van den Heuvel, and M. de Rijke.
http://www.flickr.com/photos/wonderlane/2450438944/
Metadata to support within video search?
multimedia content analysis
(computer vision)
user-generated metadata
(crowdsourcing)
http://www.flickr.com/photos/wonderlane/2450438944/
... this could also enable
Recommendation of video fragments
Links between video fragments
Links to background information
CROWDSOURCING
Crowdsourcing blends open innovation concepts with top-down, traditional management structures so that crowdsourcing organizations can effectively tap the collective intelligence of online communities for specific purposes
http://dbrabham.wordpress.com/crowdsourcing/
don't underestimate the Web
The Web as facilitator for big impact
enables grassroots movement
spreads across domains
allows contribution and participation
Pillowfight @ de Dam 03-04-2010: www.thehospages.com
CAPTCHA
± 200,000,000 captchas entered daily
reCAPTCHA
reuse to improve Optical Character Recognition (OCR)
Indirectly use human intelligence for digitisation
e.g. Google books, NY times
reCAPTCHA: Human-Based Character Recognition via Web Security Measures
Luis von Ahn, Benjamin Maurer, Colin McMillen, David Abraham and Manuel Blum
Dutch National Archive on Flickr the Commons
After two years:
1273 photos
13,000 tags
> 3 million pageviews
http://www.flickr.com/photos/nationaalarchief/5097599312/
don't overestimate the crowd
organisational structure required
goals, rules, policies, ...
quality control
motivation for users
Pillowfight @ de Dam 03-04-2010: www.thehospages.com
Quality of tags?
Consensus among users
Games with a purpose: score points for matching tags
http://www.gwap.com/
Wikipedia quality assurance
Information quality work organization in wikipedia
Besiki Stvilia et al.
Different roles and processes:
editors themselves evaluate quality of an article
directly modify, delete, or change the article
(content)-community evaluates performance of Wikipedia editors
select quality assurance agents (administrators)
(software)-community builds and maintains coordination artefacts
discussion pages, history logs, etc.
Wikipedia edit history
Wikipedia edit history
Revision history of article on Chocolate
Talk Before You Type: Coordination in Wikipedia
Fernanda B. Viégas et al.
Two considerations for crowdsourcing
Web factor
to facilitate big impact
Crowd control
to assure goals and quality
CROWDSOURCING
VIDEO ANNOTATION
Crowdsourcing at
Sound & Vision
Motivation:
Collect time-based annotations
Bridging the ‘semantic gap’
Engagement with the public
http://www.flickr.com/photos/mariannedewit/4583461285/
Web factor
Engagement with the public
44,362 pageviews
555 registered players (and thousands of anonymous players!)
Time-based annotations
> 600 videos
> 420,000 tags
46,792 unique tags
Bridging the ‘semantic gap’
user perspective
Waida? statistics from November 2009
Farmer seeks wife
BBC News
crowd control
Engagement with the public
>> constant effort required
Time-based annotations
Each tag is assigned to a single timepoint
a frame?
a shot?
a scene?
Bridging the ‘semantic gap’
Automatically matched to vocabularies
46,792 unique tags
7,372 matched tags
B&G vocabulary (GTAA)
2,517 (5.3%)
822 (11%)
Cornetto WordNet
10,757 (23%)
3,858 (52%)
Judged by a cataloguer
Genre influences tags specificity and usefulness
INTEGRATION INTO ARCHIVE?
crowdsourced video tagging
crowd
tags
time-based descriptions
user perspective (subjective)
community consensus
http://www.flickr.com/photos/flatbushgardener/2678195717/
in-house controlled annotation
professionals
concepts
descriptions of video/shot
expert perspective (objective)
cataloguers experience
http://www.flickr.com/photos/8430129@N06/2791957026/
crowdsourced video tagging
crowd
tags
time-based descriptions
user perspective (subjective)
community consensus
http://www.flickr.com/photos/flatbushgardener/2678195717/
?????
bridge communities
link tags to concepts
descriptions of fragments
combine perspectives
quality assurance by moderation
in-house controlled annotation
professionals
concepts
descriptions of video/shot
expert perspective (objective)
cataloguers experience
http://www.flickr.com/photos/8430129@N06/2791957026/
http://www.flickr.com/photos/10110263@N03/4608152854
TAG GARDENING
Edit your own tags
Roles and processes
Tagger:
edits own tags, e.g. after a game
correct spelling
delete tags
merge tags
...
Gardener:
moderate tags of multiple taggers to create fragment annotations
select and flag tags
link tags to fragments
link tags to concepts
...
combine community consensus with cataloguers/domain experience
Garden tags
... another role and process?
Landlord:
create high quality annotations of video scenes
select video scenes
supported by video analysis
describe event: who, what, where, when
annotation selection algorithm
combine community consensus with cataloguers/domain experience and video analysis
Garden scenes
m.hildebrand@few.vu.nl
