The V6 Engine

Bernard Lang and François Rouaix
INRIA
March 14, 1996

Submitted to the WWW5 Workshop: Programming the Web - in search for APIs

Abstract : WWW browsers may be seen as the composition of a multimedia interface, and a set of internet communications facilities. We propose (like some other authors) to extend the communications/processing functionalities by means of filters placed on the communication lines. The originality of this proposal lies in an architecture to efficiently compose filters, and in a simple use of the HTTP protocol, without extensions, to control the configuration of the set of filters, as well as the configuration of services embedded in each filter. We list a number of applications that show that existing communication protocols are sufficient for a very wide range of browser enhancement, with little need for other APIs.

Introduction

The client-server model has been most successful in its World-Wide-Web (WWW) application to internet. A major drawback of this success is that, since Web browsers are fast becoming the unique/universal tool for accessing all services on internet, they are also becoming huge and unwieldy general purpose pieces of software.

One reason for this success is that for a large number of applications, the browser paradigm has proved a rich, effective, and friendly man-machine interface, easily understood by the average user, which tends to make it a unique standard interface whenever possible.

This is so true that some developers have started using Web browsers and the hypermedia facilities they offer to produce cheaply friendly man-machine interfaces for non-Web applications. The development of some distributed applications has lead naturally to this conclusion: these applications are installed as CGI applications on some server, which mediates HTML interfaces with browsers, using forms for user input [ Harvest, SATAN ].

This cannot usually be done for an application running on the user workstation (or personal computer) since that station will not in general run a server. However, with very minimal addition to a browser, it is possible to use it as an interface to CGI-like programs run on the local machine. This feature is available on the MMM navigator. MMM allows the user to specify a "local binaries directory", say /home/WWW/local/bin such that any URI of the form file://localhost/home/WWW/local/bin/foo?...arguments... will be considered as a call to a program foo in that directory. The text output of the program (usually HTML source) is then displayed by the browser, spiced with links to documentation or to further requests. This approach was used to implement very cheaply CD-Punch, an interactive debugger (still a prototype) for ISO-9660 file systems, with full online documentation.

Our conclusion is that Web browsers have 2 aspects:

they are sophisticated, standardized man machine interfaces, which can actually be used for a variety of purposes, not necessarily related to internet communications or even distributed applications,
and the major application they were originally created for, and are still used for, is specifically accessing data or (CGI) programs through internet in a friendly multi-media way.

Coming back to our original concern that browsers are becoming unacceptably large, the obvious conclusion is they should be structured into smaller more manageable modules. Indeed this trend is already visible since much of the growth of browser capabilities is achieved by linking independent extensions through (yet to be) standardized APIs. However this still maintains a fairly monolithic runtime structure revolving around the chosen browser.

Another issue to be considered is that we should expect browsers with various capabilities to appear on the market, and that, most likely, users will have several browsers available, more or less convenient, depending on the type of site or data they want to access. In such a context, it should be made easy to switch between browsers to observe a given site. This will in particular require that much of the communication data used by one browser be made easily available to others.

As was already hinted by our conclusions above, the easiest solution to achieve this is to separate in browsers the display functionality from the internet communication functionality. This will at the same time allow for a much greater flexibility in configuring internet navigation systems while simplifying the architecture, and reduce the browser size problem by breaking it into two fully distinct components with more specific purposes. Hopefully, this architecture will allows software producers to concentrate their efforts either on multimedia interface capabilities, or on internet communication capabilities, and thus will let the users build their configurations from the best products of each world. The separation of concerns may also clarify the definition of APIs.

We present below a browsing environment/architecture which aims at separating Display from Web functionalities. In order to stay compatible with the existing technology, our design assumes that the multimedia interface capabilities are provided, with no change, by the already existing browsers, though we suggest some minimal change to make the architecture more powerful. We then discuss a variety of applications that are allowed by this architecture.

The web communication management and enhancement is realized by a proxy like program that acts as a filter between the client and the server. Such use of filters has been proposed by several people, usually for specific communication or information management enhancements [ WebFilter, Fast Forward, Annotation support ]. This approach has been systematized in the collection of services developed by the OSF WAIBA group [ Stream transducers ], who is also proposing a scheme to organize the cooperation of these filters [ Filter integration ]. The latter proposal by the OSFF WAIBA group is closest to our in functionality though different in realization. OSF proposes to organize each functionality as an independent transmission agent on the HTTP communication path, while we suggest modularly grouping functionalities into larger entities, around a support backbone, the V6 engine, which is the object of this paper.

The first version of the V6 Engine, that implements the ideas presented in this paper, has been programmed in the CSL version of the CAML functional programming language, which in particular offers threads to program the parallelism needed for such applications. It has been so far populated with a few simple components, and tested with both the Netscape and MMM browsers, on the Linux and SUN-OS operating systems.

Architecture

The basic idea of the architecture is to deport net related facilities of the browser to an independent program, the V6 engine, which in first approximation is much like a proxy, i.e. it mediates invisibly the communication between the browser (client) and the servers. However, unlike usual proxies that are supposed to be essentially tranparent communication optimizers, the V6 engine has the following features:

it is personal, i.e. it is intended to be accessible only by a single user, though not necessarily a single browser or application, and it can be accessed from several hosts;
it is not tranparent, in the sense that it can alter in many ways both the queries from the client, and the replies from the servers;
it can be accessed by its owner, via the http protocol, to be dynamically reconfigured or extended;
it may or may not run on the same machine, or in the same environment as the browser(s) or other application programs that access it.

Actually, though we consider that a primary use of V6 will be as a functionality extender for personal browsers (and it was originally thought of for this purpose), it can also, like proxies, be used to implement shared functionalities for a group of users. In this latter case, the owner is the V6 engine administrator. Whether the tuning of the code will have to depend on the intended number of users will be a topic for experimentation. However, assigning a instance of V6 to a single user will simplify and better ensure the dynamic control of configuration and throughput. Furthermore, it is not clear to us at this point whether there is a significant number of applications requiring otherwise.

[ V6 architecture diagram ]

Though V6 appears like a proxy, and is declared as such to your browser, it can actually be attached to, and accessed through several ports . The ports may differs by the protocol used (it needs not be HTTP, though that is the only protocol currently implemented), but also by the way the information will be processed, which depends on the sequence of filters attached to the port.

A V6 HTTP port may be declared as HTTP proxy to any standard web browser. The V6 user configures his instance of V6 by attaching to this port a sequence of query filters . Each query that enters the port is processed by each query filter in turn. The filter may leave it unchanged or alter it in any way it sees fit, including changing header lines, and/or modifying the requested URL. At the same time it processes the query and passes the result to the next filter, the query filter creates () a reply filter , which may depend on the query. The reply filters are stacked together and will be used to process the answers to the query.

After the last query filter has been processed, the query is passed to the V6 Distributor which, depending on the URL requested, will pass it to one of several V6 services (their type and number is of course configurable and extensible). A virtual host name (we could have several), say "service.v6", is assigned to the instance of V6 being used, that should not conflict with any host name on the network. If the URL passed to the distributor start with the V6 address "http://service.v6/", then the request is to be processed locally by one of the V6 service modules. Otherwise it is transmitted via the HTTP client service if is is a HTTP request, or via the Proxies service if it is another protocol (FTP, Gopher, NNTP, ...) since the (current) implementation of V6 will not handle directly these protocols, though it can process queries/replies using them encoded in HTTP. The HTTP client can either run through the usual shared proxies, or access directly HTTP servers. The Proxies service passes the request to appropriate proxies that can handle the non-HTTP protocols used to communicate with servers.

If the URL is for a local processing, say for example "http://service.v6/cache/", then the distributor checks whether a service is attached to this URL. If yes, the query is transmitted to this service, which produces the reply. Otherwise, an error reply is produced by the distributor.

Replies to a query, whether coming from the network or from a service module, are passed to the stack of reply filters that was created when processing the corresponding query. Again the filters may modify the header of the reply or the reply itself in any way they see fit. After the last filter, the possibly transformed reply is passed by the port to the browser, or any other application program, that emitted the original query.

Actually the internal architecture is somewhat more complex, in order to deal with some technical problems, for example the control of the amount of parallelism, or the limitation of the buffering between reply filters. But the V6 users need not be aware of this. Actually, in most cases, they need not be aware of much of the above description, unless they want to write their own filters. The casual user will simply use the service module Services to include preprogrammed components on her V6 backbone.

Local URLs, and virtual hosts

There are actually several ways of adressing V6 services. One is by means of its virtual host name like "http://service.v6/" used above. The other is by using the URL for entry ports, for example "http://cervoise.inria.fr:1515/" if the port used to communicate with V6 is 1515 on host "cervoise.inria.fr". This URL is tranformed into a local one when the query passes the port.

More generally, one can include filters that can change an URL into another one, either local or global. This may be applied in many ways. One is the caching service described below.

Building a body around V6.

V6 is only an engine that serves as an infra structure to run a variety of modules, filters and services, that control the client-server communications. To illustrate the power and flexibility of the architecture, we describe a few applications already implemented.

The service Services

When V6 is activated for the first time, it comes with a minimal configuration that includes essentially the service Services at the URL "http://service.v6/services/". As with the Harvest system, the user can access at any time this URL with her favorite browser, and receives a page (actually several pages linked together) that gives her the current configuration, and allows her to install (or remove) new filters, and new services with associated URLs.

From this page, the user can access specific services (they can of course be accessed directly with their URLs) in order to check or modify their configuration.

The V6 distributor keeps a list of virtual URLs attached to the various services. When a new service is added dynamically, it is registered with the distributor for the URLs it serves. The distributor is also informed whenever the configuration of a service is changed.

The services: Files and CGI

Two standard services are Files and CGI which perform essentially the usual function of a server, i.e. to map URLs to files or executable programs of the local host. In this context, CGI programs are local programs that may be controlled through a browser interface.

User authentication

Modules of the V6 engine can manage and store confidential information, such as site passwords, cached pages of confidential data, etc. They may also give access to private files that the owner has remapped to private URLs. And, of course, they control the configuration of V6 itself.

It is therefore essential to make sure that only the rightful owner can access confidential information or services. The basic proxy authentication proposed in HTTP 1.1 has been implemented in an Authentication filter (alternative such filters could be made available, based on different authentication mecanisms). If the request fails the authentication check, the query is changed into a local query addressed to the Denial service, which will produce a HTML reply denying the access, and possibly asking the user to authenticate herself (with a form).

A more lenient approach, though harder to manage, is to allow access to V6 services that are not security risks to unauthenticated users, by checking the requested URLs. Different forms of access and authentication may be attached to different ports.

Caching service

A cache has been implemented by combining a filter and a cache service that runs the cache memory. When the query reaches the cache filter, it checks whether the requested URL is in the cache. If there is a hit, the URL is changed to "http://service.v6/cache/get?document-index" so that the query is passed to the Cache service rather than being forwarded outside. The cache then answers with its copy of the requested page. Otherwise, the query is transmitted normally.

In case of hit, the cache filter produces a reply filter which does nothing. Otherwise, the reply filter produced will make a copy of the reply in the cache memory.

Of course, this scheme allows to manage several caches with different policies. In particular it is possible to have specialized caches, that will be chosen depending on the URLs being addressed, using PICS ratings for example. One can also cache documents that are often not cached by browsers, in particular those accessed with other net protocols or services.

Other applications

Developing all the possible applications is beyond the scope of this position paper. We will briefly sketch a few of the areas we are considering:

Reprocessing of retrieved documents to adapt them to the formats accepted by the browser, or to eliminate unwanted data as in the WebFilter filter and PrivNet's Fast Forward software, or to add within the document useful information such as the URL and the date of retrieval.
Switch between browsers with different capabilities.
Preprocessing of queries to improve communications (e.g. remapping of an URL to that of a closer mirror site), or to control access, for example using the PICS protocol.
Prefetching of pages.
Security control: encryption of communications and authentication. mechanism.
Enhancement of HTML or HTTP protocols, in agreement with the server (possibly through CGI extentions on the server side). Management of user states with cookie.
Printing pages with other formatting/printing programs than the one supplied with the browser.
Applets interpretation: currently we plan to accept applets in CAML bytecode, as in the MMM navigator. Of course, such applets can be used to add new filters and services.
etc.

V6 APIs and suggested extensions to existing browsers

The whole purpose of this work is to increase the modularity of web browsing tool, and to enhance existing ones without changing them. This is a rather simple approach to the API problem, when it can be worked out with existing communication protocols.

Though the extension modules (filters and services) are currently written in CAML, a system interface will be provided to allow filters written in any language. New services can already be implemented in any language with the CGI protocol. The communication protocol for filters is just a pipe mechanism for HTTP headers and data.

With respect to existing browser, one feature that would be useful is an API to create new buttons, or new items in a menu, that would be used to activate processors on the page being viewed. Even better, from our point of view would be the possibility to request (or later delete) a new button via the HTTP protocol (as part of a document header, for example), and to attach to it a URL which would be accessed with the current page URL as query string when the button is depressed.

Discussion and conclusion

The salient aspects of this proposal are:

use of HTTP based filters to enhance web services independently of browsers and servers. This seems an unescapable conclusion, apparently reached independently by several people. Filters can also be developed for other protocols, the point being that no other API is necessary.
use of browsers as GUI for configuring services, through CGI like connections to, possibly virtual, URLs.
uniform threaded support structure, to connect filters without much communication overhead.
sharing of basic services within the common structure.

This architecture is somewhat different from that proposed by the OSF WAIBA group. Their decision to use of separate filters, though it leaves the components independent, raises many communication problems related mainly to connectivity consistency and communication overhead. These problems do not arise (or are lessened) within V6 support structure, while the modularity (of Strand-like fillings [ Filter Integration ]) is retained. The separation of filters and services actually increases the modularity, since services can be both shared by different filters and reconfigured dynamically. Typically security of access to confidential data in the filters has to be checked only once, when accessing the V6 engine, rather than every time a filter is entered. If there is a need to separate filters, for example to run them on distinct machines, or share some of them within a group of users, it is possible to connect several V6 engines as one would simple filters: the V6 engine may be seen as an efficient filter composition device, but does not preclude the OSF WAIBA approach. The communication with filters for configuration purposes is done exclusively with virtual URLs, without modifying the HTTP protocol or the pages returned by servers, even when filtering other protocols than HTTP. This discussion applies to call-return communication protocols like HTTP. Browser extensions using a different communication paradigm have not been considered and may require a different approach.

An essential point in this presentation is that net related functions may be "physically" separated from multi-media display functions, thus avoiding to mix concerns related to two distinct technologies, and allowing developpers to concentrate their effort on one problem only. Furthermore, one can avoid any need for new APIs other than the existing protocols, though slight enhancement of browsers might help.

A possible conclusion of this work is that, where management and processing of distributed structures is concerned, protocols for distributed communications should be preferred to APIs, though this should be further confirmed by more extensive experimentation with V6 or similar systems. Of course this is probably not true for the integration of multi-media facilities, which may be another reason to keep separate these two types of concern.

References

Annotation support: Pan-Browser Support for Annotations and Other Meta-Information on the World Wide Web, by Matthew A. Schickler, Murray S. Mazer, and Charles Brooks, Fifth International WWW Conference Proceedings, Paris, France, May 1996, http://www.osf.org/www/waiba/papers/grpannot/grpannot.html.
CAML: The Caml Light system, documentation and user's guide, by Xavier Leroy <Xavier.Leroy@inria.fr> http://pauillac.inria.fr/caml/man-caml/index.html, Release 0.71, January 1996.
CSL: The Caml Special Light system, documentation and user's guide, by Xavier Leroy <Xavier.Leroy@inria.fr> http://pauillac.inria.fr/csl/htmlman/ Release 1.13, 1995.
Cookie: Persistent Client Side State HTTP Cookies, Preliminary Specification, by Netscape Communications Corporation, http://www.netscape.com/newsref/std/cookie_spec.html,1996.
Fast Forward Internet Fast Forward, by PrivNet http://www.privnet.com/index2.html, commercial presentation.
Filter Integration: Dynamic Integration of HTTP Stream Transforming Services, by Misha Davidson, Charles Brooks, and Murray S. Mazer, OSF, January 1996. http://www.osf.org/www/waiba/papers/switching/paper.html,
Harvest Harvest User's Manual,Version 1.1, by Darren R. Hardy and Michael F. Schwartz, U. Colorado, Boulder, February 1995 , Last Revised on February 15, 1995. http://siros.lf.net:81/Harvest/doc/user-manual/
HTTP 1.1: Hypertext Transfer Protocol -- HTTP/1.1, INTERNET-DRAFT , by HTTP Working Group: Roy T. Fielding, Henrik Frystyk Nielsen, and Tim Berners-Lee, http://w3c1.inria.fr/pub/WWW/Protocols/HTTP/1.1/spec January 19, 1996, Expires May 22, 1996.
Linux Linux Operating system, http://www.linux.org/
MMM: A Web navigator with applets in Caml, by Fran�ois Rouaix <Francois.Rouaix@inria.fr>, http://pauillac.inria.fr/~rouaix/mmm/papers/mmm.ps.gz, To appear in Proc. of the 5th WWW Conference, Paris, May 1996.
PICS(Platform for Internet Content Selection): Rating Services and Rating Systems (and Their Machine Readable Descriptions) by Jim Miller <jmiller@w3.org>, Paul Resnick <presnick@research.att.com>, and David Singer <singer@almaden.ibm.com>, Revision 1, November 1995. http://www.w3.org/pub/WWW/PICS/services.html
RFC1630: Universal Resource Identifiers in WWW, by Network Working Group: Tim Berners-Lee, ftp://ds.internic.net/rfc/rfc1630.txt, June 1994
SATAN <http://www.fish.com/~zen/satan/>, Dan Farmer <zen@fish.com> and Wietse Venema <wietse@wzv.win.tue.nl>; see The SATAN User Interface and
Improving the Security of Your Site by Breaking Into it, comp.security.unix, December 1993.
Stream transducers: Application-Specific Proxy Servers as HTTP Stream Transducers, by Charles Brooks, Murray S. Mazer, Scott Meeks and Jim Miller, Fourth International WWW Conference Proceedings, pp. 539-548,Boston, MA December 11-14, 1995. http://www.w3.org/pub/Conferences/WWW4/Papers/56/.
SUN-OS: see Sun Microsystems, http://www.sun.com/.
WebFilter Filtering the Web using WebFilter, by Axel Boldt <boldt@math.ucsb.edu> http://math-www.uni-paderborn.de/~axel/NoShit/, March 1996.

Bernard.Lang@inria.fr, Francois.Rouaix@inria.fr
INRIA
B.P. 105
78153 Le Chesnay CEDEX
France
tel: +33 (1) 3963 5644 and 3963 5629

Last modified: