topical media & game development
[] readme course(s) preface I 1 2 II 3 4 III 5 6 7 IV 8 9 10 V 11 12 afterthought(s) appendix reference(s) example(s) resource(s) _

talk show tell print

@VR

[] readme proto extern template effect object library avatar ml reference

AVID -- Agents and Virtual environments In DLP

PDF

1) NWO/GBE grant request: AVID

1a) Project: Agents and Virtual Environments in DLP
The proposal is online at: www.cs.vu.nl/~eliens/research/avid.html

1b) Project Acronym: AVID

1c) Principal Investigator: Dr. A. Eliëns
Address: Dr. A. Eliëns, Faculty of Sciences, Div. of Mathematics and Computer Science, De Boelelaan 1081, 1081 HV Amsterdam, email: eliens@cs.vu.nl

1d) Online Information:

online

The online version provides preliminary demonstrators, as well as additional online references and updates or corrections.

2) Summary:

The AVID research project aims for a novel integration of multimedia presentation formats and intelligent agent technology. In particular AVID, which stands for Agents and Virtual environments In DLP, aims at a tight integration of the object models underlying the various technologies involved, in a uniform platform based on the (object-oriented) distributed logic programming language DLP.

The language DLP is implemented on top of Java and currently allows for writing stand-alone applications as well as programs that run in a Web browser either as applet or as an extension of VRML.

Our approach may be positioned against the background of intelligent multimedia research. Where multimedia research is primarily concerned with authoring, retrieval and presentation of multimedia information, intelligent agent research focusses on user profiling, knowledge representation, and inter-agent communication. Merging these two areas of interest should result in the technology needed to develop versatile, flexible multimedia information systems for the presentation of increasingly larger collections of data with suitable support for user tasks, guidance and navigation. In our research we focus on networked virtual environments that are accessible through a Web browser.

A somewhat futuristic approach, which clearly indicates the need to go beyond traditional WIMP (Windows and Menus) interfaces, is sketched in [Angelic], which presents under the rather poetic label of angelic guidance a vision of how intelligent agent technology could be merged with virtual actors in rich media 3D environments to assist the user in a variety of (possibly related) tasks.

The vision expressed there is clearly futuristic since there does not consist any convincing integration of the technologies needed to realize this vision. Our opinion is that, given the programmatic models in current Web3D technology, such an integration is hard, if not impossible, to realize unless a dramatic shift in paradigms is effected. To be more specific, current Web3D technologies such as VRML (Virtual Reality Modeling Language) and Java3D offer rich facilities for creating static 3D content as well as a programmatic model for defining dynamic behavioral properties, using either built-in primitives, in-file script nodes (in the case of VRML), or user-defined objects written in Java. See [Web3D] and [Java3D]. Yet, despite the rich functionality offered, we believe that the programmability of rich media 3D should significantly be improved to utilize desktop virtual reality as an interface to intelligent multimedia systems effectively.

To remedy the noted deficiency, we propose a tight integration of agent object models and the object models underlying Web3D, in the framework of an object-oriented logic programming language. Moreover, we may extend the agent communication language used for inter-agent communication to realize shared objects and events in multi-user virtual environments, due to the distributed nature of our logic programming language DLP.

To realize the envisaged integration, we must develop sufficiently rich representational and presentational models of agents, based on respectively the BDI cognitive model and a behavioral model of virtual actors. Also, we need to develop the syntax and semantics for an agent communication language that can cope with both inter-agent communication and the communications involved in realizing shared objects and events.

Finally, our research efforts in the AVID project should result in a framework for developing multi-user, multi-agent applications in rich media Web3D environments, based on the distributed logic programming language DLP.

3) Classification:

Following NOAG-i 2001-2005, the research proposed is related to the following areas:

NOAG-i 2001-2005

In the online version links are provided to information about these areas (in Dutch) taken from NOAG-i 1997. As an estimate in percentages, the contribution of the respective fields might be summarized as: 50 (IS), 30 (MM), 20 (SE).

4) Composition of the Research Team:

nameexpertiseaffiliationhours/week
Dr. A. EliensmultimediaVU/IMSE8 (coordination)
Dr. Z. HuangagentsVU/AI8 (WASP/RIF)
Prof. dr. J.C. van VlietarchitectureVU/IMSE2 (promotor)
Prof. dr. J. Treurintelligent systemsVU/AI2 (promotor)
Drs. C. VisserDLPVU 18 (programmer)
vacatureAI/CSVU32-36 (OIO)

The research will be executed within the intelligent multimedia group at VU, under the supervision of Dr. A. Eliëns and dr. Z. Huang. Drs. C. Visser will provide programming support during the full four years of the project, of which two years will be covered by the requested funding (see 10).

5) Research School: SIKS

 The partners are members of SIKS (the Dutch Research School for Information and Knowledge Systems, www.siks.nl).

6) Description of the Proposed Research

research

The AVID project aims at a unified framework for agents and virtual environments, built on the distributed logic programming language DLP.

Before stating our main research goals more explicitly a brief characterization will be given of the work that led to this proposal. At the end of this section, it will be sketched how this work fits in with our educational activities and (intelligent) multimedia research.

background

The AVID project is a continuation of research done in two NWO projects:

In the online version, links are given to more detailed information about these projects.

Although the WASP proposal was written more than three years before the RIF proposal, dr. Z. Huang started as a post-doc on the WASP project when the RIF project ran already for more than six months. Since we then felt that the WASP project proposal was slightly outdated, we made an effort to merge the WASP and RIF projects, by focussing on agents in 3D virtual environments which resulted in a paper presenting a taxonomy of Web agents, [Taxonomy]. However, after about a year the continuity of the RIF project was endangered due to a mutation of personell at CWI. So, NWO was asked for permission to utilize the RIF funds for prolonging the WASP project. This was granted, provided that the research goals of the RIF project were sufficiently covered within WASP.

In the RIF project we used the blaxxun Community Server and VRML to realize information retrieval and delivery im multi-user 3D environments. See [Navigation]. Agents were then conceived as extensions on the server-side using blaxxun's native agents enriched with embedded logic. However, at the same time that the RIF project funds were transferred to WASP, the first prototype of DLP in Java became available, and we decided, somewhat radically, to drop the more low-level blaxxun technology in favor of a unified approach in DLP. Thus, we extended DLP with primitives for manipulating VRML worlds, using the Java External Authoring Interface that is part of VRML. The DLP+VRML framework proved to be surprisingly effective, as testified by the following references:

DLP+VRML

  • architecture for agents in virtual environments [Architecture]
  • intelligent avatars in 3D virtual worlds [Avatars]
  • demonstrator: multi-user soccer game [Community]
Now, the language DLP itself has quite a long history, [DLP]. It has also been described in [OO]. As a language, DLP offers an object-oriented extension of (traditional, Edinburgh-style) Prolog, with multi-threaded objects, non-logical instance or state variables, communication by rendex-vous and (distributed) backtracking. After preliminary prototypes in C++, we focussed on an implementation in Java, to be able to use DLP for Web programming, [Web]. The first Java implementation became available relatively late, but just in time to create the DLP+VRML extension when needed.

the research

 Given the background of our research, we can state our research goal quite simply:

problem

  • starting point: Java-based DLP+VRML framework
  • end point: fully integrated agent and virtual environment framework in DLP
So, what needs to be done is to develop more fully the concepts and models that appeared to be fruitful in our initial explorations of deploying the DLP+VRML framework for the realization of agents in 3D (multi-user) environments.

As a first characterization of how to achieve this we may indicate the following steps:

steps

  1. realization of agent taxonomy, as a hierarchy of agent objects;
  2. tight integration of DLP with Web3D (VRML), that is an integration of the object model of VRML in DLP to allow for the dynamic creation of (shared) Web3D content;
  3. realization of extensible Agent Communication Language (ACL) in DLP;
  4. application of ACL to realize multi-user virtual environments in DLP+VRML;
  5. target applications.
Below, we will indicate in more detail the research topics that need to be addressed to take these steps.

Target applications will result from other research in our group in the area of persuasive technology as for example [Dialogs], from cooperation with other research group at CWI as in [Context], and from student projects. As outlined below both VRML and DLP+VRML technology is part of the students' curriculum and there have already been two master thesis projects in the area of virtual environments. Target application areas may include, but are not limited to, e-commerce, edutainment , agent-aided 3D design and social marketing.

With regard to the integration of DLP and VRML, let me explain that the approach we have chosen is one that allows for treating an arbitrary VRML world as a black box, with handles to which events can be sent or from which events can be received. Events are the primary mechanism of scenegraph-based 3D formats, such as VRML and Java3D, for controlling dynamic, behavioral properties. A tight integration does not mean giving up this clean separation, which allows for using existing VRML worlds, but instead to allow also for the dynamic creation of 3D content from within DLP. Also, the manipulation of existing or dynamically created 3D content, as for example the animation of an avatar (the humanoid presentation of a user or agent) should be made more easy by offering suitable primitives in DLP. For a further explanation of our approach and how it is related to other approaches, see the paragraph on related work below.

research topics

To summarize, our research goals may be stated more succinctly as follows:

research goals

  • suitably rich model of Web Agents and corresponding realization (WAF)
  • sufficiently expressive Agent Communication Language, to be used for interaction between agents, as well as for the realization of shared objects and events (MU-ACL)
  • framework for developing multi-user, multi-agent applications in Rich Media and Web3D (RM3D-DLP).
To realize our research goals, we have to address a variety of topics.

topics

  • representation and presentation model of agents
  • ACL syntax and semantics
  • multi-user virtual environments
  • applications

representation and presentation model of agents

 Our agent model is based on an extension of the BDI cognitive model with sensors and effectors, that allow agents to perceive events that occur in the virtual world as well as to operate on the virtual world by sending events. In this way agents can not only control the 3D world they live in, but also their own presentational characteristics, that is their appearance and attributes they possess. In other words, we need to augment the agent model with characteristics of a virtual actor for effectively deploying the agent in 3D virtual environments.

ACL syntax and semantics

 In our multi-user soccer prototype [Community], the agents (footballers and goalkeepers) did not communicate amongst themselves. Rather they took notice of the state of the game, that is their own position and the position of the ball, the opposite goal, and their fellow players. However, to allow for multiple users to join the game, we used an invisible agent that communicates the game information to the (invisible) agents of other users, so that these agents may update the actual appearance of the game as presented to the user. In this way we realized a shared environment using communication between agents based on the model of agents that we also used for the visible (footballing) agents. Clearly, we must more fully develop the syntax and semantics of such an ACL to allow for inter-agent communication as well as shared objects and events in a more generic fashion. See [ACL] for an overview of Agent Communication Languages. We also need to investigate the proper architecture to realize a distributed multi-agent system. In the current prototype, we used mediator DLP objects on the server, which were needed to deal with the security restrictions imposed by the Java-based architecture of our DLP+VRML framework.

multi-user virtual environments

Current systems for creating multi-user virtual environments are rather complex, and require a variety of authoring tools and programming languages, ranging over C, Java, Javascript, and proprietary scripting languages. Our explorations indicate that we can develop a unified framework for dealing with the various aspects of rich media 3D environments, which include 3D models (static and dynanic), intelligent agent avatars (WAF), shared objecs and events (MU-ACL).

For an early account of what the architecture of multi-user environments in VRML look like, see [Environments].

applications

As indicated before, there is a large variety of potential applications for RM3D-DLP technology, including information retrieval and delivery, presentation and navigation agents, user assistance. Assuming that such applications require intensive knowledge programming, they may clearly benefit from our approach.

related work

 Our work is related to or even shows resemblance with a number of other projects. We will discuss:

related work

  • [Logic] -- logic programming for the internet
  • [Scripts] -- merging VRML with logic scripts
  • [Jinni] -- client-side logic with VRML
  • [DIVE] -- distributed interactive virtual environment
  • [Parlevink] -- agents in VRML worlds
  • [Alice] -- 3D interactive programming environment

An overview of logic programming languages for the Internet is given in Andrew Davison's [Logic]. Due to its late arrival on the Web scene, DLP is not mentioned in this overview. It was, however, mentioned in a previous overview of the same author [Survey], in the category of object-oriented extensions. For your information, the principal investigator organized a workshop at WWW6 in 1997 entitled Logic Programming for the Web, [Workshop].

There have been a number of attempts to improve on the scripting capabilities of VRML. One example of interest, from our perspective, is the extension proposed in [Script], which allows for logic programming in VRML script nodes. This approach affects the VRML files themselves, and hence the VRML97 standard. As a drawback, existing VRML content cannot be (re)used without modification. More importntly, however, such an approach does not allow for migrating to newer or other scenegraph-based formats such as X3D or Java3D.

In effect, Jinni (developed by Paul Tarau) was a source of inspiration for our first DLP+VRML prototype, [Jinni]. Nevertheless, our DLP language is semantically more suited to deal with the complexities of a more tight coupling of VRML and logic programming. In particular, multi-threaded objects in DLP allow for receiving events in an asynchronous manner, which is impossible to do in ordinary sequential Prolog. Distributed applications in Jinni are possible using a Linda-style shared tuple space. In our approach we rely on rendez-vous communication and server-side mediator objects. As the soccer prototype shows, it is rather easy to create a shared repository of state information using agent communication primitives in DLP.

The DIVE project at the Swedish Institute of Computer Science is an example of a complex heterogeneous system for developing multi-user virtual communities, [DIVE]. It is offers support for a variety of graphic formats, including VRML, and allows for scripting behaviors and interactions using Tcl. DIVE applications and activities include virtual battlefields, spatial models of interaction, virtual agents, real-world robot control and multi-modal interaction. DIVE is supported on a variety of platforms. The first DIVE version appeared in 1991. Our approach clearly builds on later developments, in particular Java, to achieve cross-platform portability. In contrast to the heterogeneous nature of the DIVE platform, we provide a unified language framework as a programmatic interface for the creation of 3D content and behavioral properties.

The Dutch Parlevink project, at the University of Twente, has developed a virtual theater, which demonstrates that agents may usefully be employed in (VRML-based) virtual environments, [Parlevink] The focus of the Parlevink project, however, is more on cognitive models and natural language processing. In contrast, our goal is to improve virtual environment technology itself, by providing a unified framework for developing agents and virtual environments.

Finally, we cannot conclude this brief overview of related work without mentioning Alice, a 3D interactive programming environment for virtual worlds, developed at Carnegie-Mellon University, [Alice]. One interesting feature of Alice is that it allows for programming generic behavioral properties in the object-oriented scripting language Python. Nevertheless, we believe that due to the knowledge-intensive nature of the programming tasks required for developing real applications in rich media virtual environments, a logic-based language such as DLP will in the long end be much more effective.

embedding in education: focus on multimedia

focus on multimedia

The research will have a clear impact on the educational activities for the specialisation multimedia, in particular the Multimedia Authoring II course, as indicated below. It is to be expected that student projects will deploy the technology developed within the AVID project.

In the first year students start with a general Introduction to Multimedia. This course centers around three themes: the convergence between media, platforms and delivery technology, the availability of broadband communication and its impact on the development of standards such as MPEG-4, and multimedia information retrieval as an essential ingredient of the growing multimedia information repository on the Web.

There are two follow-up courses, which are given in respectively the second and third year:

courses

The first of these courses deals with the technology for creating 3D scenes and worlds, whereas the second is more focused on providing intelligent services in virtual environments. Students use the DLP+VRML framework for their assignment in the second course. See [Intelligent].

embedding in research: (intelligent) multimedia

SIKS report (2001)

Over the past six years, our research efforts have focussed on developing models and software architectures for multimedia and hypermedia applications. (A full version of this description, including publications, is available online in the SIKS report (2001)).

This research has resulted in two Ph.D. theses:

  • 10/4/2001 -- Structured hypermedia -- a matter of style
  • 8/5/2001 -- Diva: Architectural Perspectives on Information Visualization
For both theses, prof. dr.J.C. van Vliet was promotor, and the principal investigator had daily supervision and acted as co-promotor.

The hypermedia work was done in collaboration with dr. L .Hardman and dr L. Rutledge from the CWI Multimedia group. This cooperation resulted in the formalization of the Amsterdam Hypermedia Model, an extension of the Dexter Hypertext Reference Model. The hypermedia project also resulted in a software framework for developing web-based hypermedia applications, the hush library and its music and video extensions.

The visualisation project concerned the use of animations and visualisation to display business process simulation results in a hypermedia context. During the project the focus shifted towards visualisation, in particular business visualisation. Also, explorations were done to investigate interactive visualisation in 3D.

Our current research efforts are directed towards developing a high-level platform for rich media 3D virtual environments. Our goal is to study aspects of the deployment and architecture of virtual environments as an interface to multimedia information systems.

intelligent multimedia

The intelligent multimedia research theme may be regarded as continuing the hypermedia and visualisation projects described before. Our efforts are directed towards realizing the technology needed for developing intelligent multimedia applications. In particular, we aim for developing demonstrators in the area of persuasive technology, and thus explore design methods and patterns for the realization of emotionally charged virtual worlds. This work is being done in cooperation with dr. C. Dormann and dr. Z. Ruttkay from CWI.

cooperation

 The list below summarizes the cooperations that are due to either the position of the research group within the department or contacts resulting from previous research, as sketched above.

cooperation

  • multi-agent research group at VU/AI (dr. C. Jonker)
  • HCI at VU/IMSE (dr. G. van der Veer)
  • persuasive technology VU/IMSE (dr. C. Dormann)
  • Facial and Avatar Animation Group at CWI & Epictoid (dr. Z.Ruttkay)
  • Multimedia Authoring Group at CWI (dr. J. van Ossenbruggen & dr. L. Rutledge)

7) Work Programme

 With reference the our discussion of the steps that must be taken to realize our research goals and the topics that need to be addressed we suffice by giving a brief summary of the deliverables that we expect to produce during the four years of the project.

deliverables:

  • year 1: realization of Web Agent Framework (WAF)
  • year 2: realization of ACL for multi-user environments (MU-ACL)
  • year 3: integration of WAF and MU-ACL
  • year 4: thesis

8) Instrumentation

Not relevant.

9) Literature

The online version has links to online copies of most of the cited references.
Angelic
Wolfgang Broll, Leonie Shäfer, Tobias Höllerer, Doug Bowman, Interface with Angels: the future of VR and AR interfaces, IEEE Computer Graphics, November/December 2001, pp. 14-17
Web3D
Web3D Consortium, VRML97 Standard,
http://www.web3d.org/fs_specifications.htm
Java3D
Javasoft, Java3D API, java.sun.com/products/java-media/3D
Taxonomy
Zhisheng Huang, Anton Eliëns, Alex van Ballegooij, Paul De Bra, A Taxonomy of Web Agents, IEEE Proceedings of the First International Workshop on Web Agent Systems and Applications (WASA '2000), 2000.
Navigation
A. van Ballegooij and A. Eliëns, Navigation by Query in Virtual Worlds, Web3D 2001 Conference, Paderborn, Germany, 19-22 Feb 2001
Architecture
Zhisheng Huang, Anton Eliëns, and Paul De Bra, An Architecture for Web Agents, Proceedings of the Conference EUROMEDIA 2001, 2001. @K Avatars Zhisheng Huang, Anton Eliëns, and Cees Visser, Programmability of Intelligent Agent Avatars, Proceedings of the Agent'01 Workshop on Embodied Agents, June 2001, Montreal, Canada
Community
Zhisheng Huang, Anton Eliëns, and Cees Visser, 3D Agent-based Virtual Communities, submitted to Web3D 2002
DLP
A. Eliëns, DLP -- A language for Distributed Logic Programming, Wiley, 1992
OO
A. Eliëns , Principles of Object-Oriented Software Development (2nd edn), Addison-Wesley (2000)
Web
Cees Visser, Anton Eliëns, A High-Level Symbolic Language for Distributed Web Programming, Internet Computing 2000, June 26-29, Las Vegas
Dialogs
C. Dormann & A. Eliëns, Exploring the design space for emotive dialogues, submitted to: Third Int Conf on Emotion and Design
Context
Lloyd Rutledge, Alex van Ballegooij and Anton Eliëns, Virtual Context - relating paintings to their subject, Culture Track of WWW9 in Amsterdam, The Netherlands, Tuesday, May 16th, 2000
ACL
Yannis Labrou, Tim Finin and Yung Peng, The current landscape of Agent Communication Languages, IEEE Intelligent Systems, Vol. 14, no. 2, 1999
Environments
Wolfgang Broll, VRML and the Web: A basis for Multi-user Virtual Environments on the Internet, In Proceedings of WebNet96, World Conference of the Web Society (San Francisco, CA, Oct. 16-19, 1996), H. Maurer (ed.), AACE, Charlottesville, VA (1996), 51-56.
Logic
A. Davison, Logic Programming Languages for the Internet, Chapter in Computational Logic: From Logic Programming into the Future, Antonis Kakas, Fariba Sadri (eds.), Springer Verlag (2001). available from: fivedots.coe.psu.ac.th/~ad
Scripts
A. Davison, Enhancing VRML97 Scripting, Euromedia'2001, Valencia, Spain, April 18-20. available from: fivedots.coe.psu.ac.th/~ad
Jinni
P. Tarau, Jinni: Intelligent Mobile Agent Programming at the Intersection of Java and Prolog, Proc. of PAAM'99, London, UK, April, see also www.binnetcorp.com/Jinni
DIVE
Distributed Interactive Virtual Environment, Interactive Colloborative Environments, SICS, www.sics.se/dive
Parlevink
University of Twente, Language Engineering Group,
parlevink.cs.utwente.nl
Alice
Carnegie-Mellon University, 3D Programming Environment, www.alice.org
Intelligent
Zhisheng Huang, Anton Eliëns, Cees Visser, Intelligent Multimedia Technology: An Approach to Combine Agent Technologies with Multimedia, in preparation
Survey
A. Davison, A survey of Logic Programming-based Object-Oriented Languages, In Research Directions on Concurrent Object-Oriented programming, G. Agha, P. Wegner, A. Yonezawa (eds.), MIT Press (1993)
Workshop
A. Eliëns (chair), Logic Programming and the Web,
www.cs.vu.nl/~eliens/online/workshops/www6

10) Requested Budget

personellperiodeuro
OIO4 year129.879
programmer2 year107.248
241.665

Remarks:

  • the amount of euro 129.879 includes the benchfee for travel expenses and other support.
  • the VU will provide an additional 2 years of programmer support.
[] readme course(s) preface I 1 2 II 3 4 III 5 6 7 IV 8 9 10 V 11 12 afterthought(s) appendix reference(s) example(s) resource(s) _

(C) Æliens 04/09/2009

You may not copy or print any of this material without explicit permission of the author or the publisher. In case of other copyright issues, contact the author.