EP0916117A2 - Virtual environment data browser - Google Patents

Abstract

A data browser is described for accessing data defining virtual environments via the Internet/World Wide Web. Where the data defining objects to appear within the virtual environment is in the form of discrete geometrical components, with each object being arranged as a hierarchy linking the components and with transformations (T0, T1, T2, T3) specifying the translation of vertex locations for a child component to the coordinate system of its parent, the browser maintains a record of previously calculated transformation products (T0. T1). Each time the browser updates its generated representation of the virtual environment it checks which components have moved, and hence which transformation multiplications must be updated, and otherwise recalls the result of the previous calculation.

Description

DESCRIPTION

VIRTUAL ENVIRONMENT DATA BROWSER

The present invention relates to apparatus for accessing, from a remote source, data defining a graphical representation of a virtual environment, with the user being enabled to select a viewpoint within the virtual environment and the apparatus having means for rendering a display of the virtual environment from that viewpoint.

A description of a service providing a virtual environment (or cyber-space) accessible by remote users is given in European patent application EP-A-0 697 613 (Sony Corp.). The system described includes a server providing data defining a virtual reality space, and user terminals connected to the server via a high-speed communications network (using optical fibres or the like). In operation, the server maintains data for a number of virtual environments and supports many differing terminal types by the use of conversion objects between information objects and user objects: the conversion objects provide individually tailored translation for communications back and forth between each type of terminal and each configuration of virtual environment supported.

A particular benefit arises when the format of data storage and representation for virtual environments is at least partially standardised, allowing for greater interchange between systems of different manufacturers as well as a reduction in the necessity for individually tailored translation utilities. With at least partial standardisation, the necessary configuration of a browser for accessing such data, whether in a hardware or software implementation or a mixture of the two, becomes simpler.

A notable example of standardisation in the field of data defining virtual environments is the so-called Virtual Reality Modelling Language (VRML) as described, for example, in the VRML standard, version 2.0, issued as ISO/IEC WD14772 on 4th August 1996. VRML is a file format for describing interactive three-dimensional objects and worlds to be experienced on the Internet/World Wide Web and it is generally analogous to the way HTML (HyperText Markup - Language) is used to describe documents for transmission over the Internet. A number of examples of VRML browsers are discussed in "Building VRML Worlds" by E. Tittel et al, published by McGraw Hill 1997, ISBN 0-07- 882233-5, at pages 26 to 31. The functionalities of the different browsers depend to a large extent on their target host system and the likely uses of the same, as well as whether the browsers are purpose-built/written or whether they are modifications of earlier utilities. In order to simplify the specification of the shapes of objects to be rendered within a virtual environment, it is a known technique to restrict such objects to constructions from a limited set of "building blocks" such as rods, cones and cubes, which may be simply specified in relatively few parameters. The construction of the objects from components is defined by a hierarchy of 5 parent and child nodes or components, where each child may have children of its own. Relative motion and scaling of these components is defined by a transformation (such as a matrix multiplication) with compound movement of components of an object being represented by a series of sequentially applied transformations. As will be recognised, for complex objects having several o hierarchical levels, the necessary sequences of transformation applications may provide an excessive computational load to the host system.

It is an object of the present invention to provide a user operable means for browsing data defining a virtual environment and rendering a view of the 5 same, having a reduced computational loading through improved handling of hierarchical data structures.

In accordance with the present invention there is provided a data processing apparatus configured as a virtual environment data browser, the apparatus comprising a processor coupled with at least one memory device o and data network interface means capable of receiving data defining a virtual environment and objects therein when coupled to a data network including a source of such data, the apparatus further comprising user-operable input means and rendering means configured to periodically generate an image - based on said data defining the virtual environment and objects and from a viewpoint at a location and with an orientation in said virtual environment determined at least partially by input from said user-operable input means, wherein objects within the virtual environment are formed from a plurality of geometrical components, each defined according to its own coordinate system, with the components in an object being arranged in a hierarchy of parent and child components with each child being linked to its parent by a transformation specifying the conversion of the child component parameters to the parental coordinate system; characterised in that the apparatus is arranged to maintain a record, for each image generation, of the result of each transformation application, to determine which transformations have been altered as a result of movement between parent and child components and apply the altered transformation to the parameters of the child component, otherwise to use the stored result of the previous application of the transformation.

By maintaining a record of those transformations which have been altered and which have not, repetition of the process of applying the transformation to convert a child component parameters to the parental coordinate system can be avoided unless actually required. The technique is particularly beneficial where the hierarchy has several levels (grandparent, parent, child, grandchild etc) with the check for alteration of the transformation (and usage of stored results where possible) being carried out down the hierarchy, that is to say towards the most junior components. For example, where it is found that only a grandchild component has moved, only the transformation between child and grandchild need be effected with the results of the child's conversion to the grandparent coordinate system (via that of the parent) simply being recalled from storage. In a preferred embodiment, the generated virtual environment hosting the compound objects forms the upper level of all hierarchical object groupings: in such a case, a further transformation is suitably applied, during each update of the virtual environment, to the most senior component of the or each hierarchy to convert the transformed component parameters of the components of the hierarchy to the further coordinate system specified for the- virtual environment as a whole.

As will be described with reference to example hereinafter, the means for effecting the conversion of child coordinate parameters to a parental coordinate system may suitably comprise calculation means operable to effect matrix multiplication, which calculation means may be effected in software or hardware, or a combination of both. These means for effecting the conversion of child coordinate parameters to a parental coordinate system will preferably also include multiplication means operable to effect scaling of child coordinate parameters, such that components need only be specified in a single size (to save storage space) and scaled as required.

The record of transformation application is suitably held in the said at least one memory device (optionally linked to the result of the previous application of the transform) as an identifier for each transformation with an associated flag, which flag is set on the alteration of the transform and reset if the transform is unchanged following a subsequent update of the generated virtual environment. Alternately, a separate memory (for example a high speed cache) may hold the ongoing record of which transformations have been altered, and hence require recalculation, and which have not.

The invention will now be described by way of example only, with reference to the accompanying drawings, in which:

Figure 1 is a block schematic diagram of a data processing system suitable to embody the present invention;

Figure 2 represents the component functions of a browser, as hosted by the system of Figure 1 , and embodying the invention;

Figure 3 shows side and elevational views of an object, formed from four discrete components, for display in a virtual environment; Figure 4 illustrates a hierarchical node structure linking the components of Figure 3; and Figure 5 is a flowchart representing the periodic virtual environment refresh routine implemented by the browser of Figure 2.

Figure 1 represents a data processing system, such as a personal computer, which acts as host for a software utility which configures the system as a browser for data defining a virtual environment. The system comprises a central processing unit (CPU) 10 coupled via an address and data bus 12 to random-access (FvAM) and read-only (ROM) memory devices 14, 16. The capacity of these memory devices may be augmented by providing the system with means to read from additional memory devices, such as a CD-ROM (not shown).

Also coupled to the CPU 10 via bus 12 are first and second user input devices 18, 20 which may suitably comprise a keyboard and a cursor control and selection device such as a mouse or trackball. Audio output from the system is via one or more speakers 22 driven by an audio processing stage 24; in addition to providing amplification, the audio processing stage is preferably also configured to provide a signal processing capability under the control of the CPU 10 to allow the addition of sound treatments such as echo to existing audio data. Video output from the system is presented via display 26 driven by display driver stage 28 under control of the CPU 10: display 26 may be a flat screen showing a single image of the virtual environment from a selected viewpoint, or it may be a more complex device such as a binocular head- mounted display or a multiple interlaced view autostereoscopic screen.

A further source of data for the system is via online link to remote sites, for example via the Internet, to which end the system is provided with a network interface 30 coupled to the bus 12. The precise construction of the interface is not an essential feature of the present invention, although it will be recognised that the interface configuration will depend on the type of data network to which the system is to be coupled: for example, where the system is for use by a private home user, the data link is likely to be a telephone connection to a local service provider. In such a case, the interface 30 will suitably incorporate a modem. For other types of data link, such as an ISDN connection, the interface will be configured accordingly.

Turning now to Figure 2, the inter-relationship of a number of functions assembled to configure the hardware of Figure 1 as a browser is illustrated. The functions in the example illustrated are particularly suited for browsing virtual environments defined in accordance with the VRML standard, version 2.0, issued as ISO/IEC WD14772 on 4th August 1996, although it will be understood that the present invention is not restricted by, or limited to, conformance with this standard. The browser is based around a scene manager 40 coupled with respective stores for functions 42, assets 44, and a hierarchy of scene nodes 52, as will be described in greater detail below. In terms of the hardware of Figure 1 , the function, asset and hierarchy stores will generally be held in RAM 14, although some of the assets (for example standardised texture maps for application to the surfaces of the scene components) may be provided via offline storage such as CD-ROM. The browser implementation program is suitably held in non-volatile ROM 16, optionally linked to a conventional boot-up utility for the system. Also coupled with the scene manager are four interfaces (IF.1-IF.4) 46-49 and a clock 50. The first of the four interfaces 46 corresponds in part to interface 30 of Figure 1 in that it represents the source of data received via the Internet: as represented by dashed line 54, the received data may be passed on from the interface to other (not shown) destinations, such as a further browser for HTML content in the received data. The first interface is the general arrival point for the data defining the virtual environment and, as such, it may also receive data from more local sources such as storage in the system RAM (14; Fig.1) or from a CD-ROM.

From the first interface, the data passes to a parsing stage 56 which divides it into data defining or relating to assets (which data is then passed to store 44) and data defining or relating to the run-time node hierarchy (which data is then passed to store 52). In operation, it will generally be the case that only the data required (in terms of texture, coordinates etc) immediately or in the short term will actually be loaded up by the browser. Related information not immediately required is instead held in the form of an address for its current- storage location. In the case of information called up via the Internet, the address held by the browser will suitably be a URL (uniform resource locator) so that the browser can simply call up the data, via the first interface IF.1 , when the need for it becomes imminent; for more local information, an address in RAM 14 or ROM 16 is held.

The second interface 47 is an external API (application programming interface) comprising a set of interfacing subroutines enabling multiple-user extensions to the system, such as to simultaneously support two or more users having respective viewpoint locations and orientations within a common environment. It will be recognised that the particular form and specification of the subroutines will vary according to factors such as the number of users supported simultaneously, what form of user-operable input device or devices each is provided with, and so forth: the interfacing subroutines have no direct bearing on the operation of the present invention and will not be further described.

The third interface 48 is for operational segments used to program behaviour in a scene: these segments are referred to in VRML as script nodes. The script nodes contain a program module which, in response to a change or user action within a scene, effect a change somewhere else in the scene depending on the contents of the program module. The program modules may suitably contain Java ® language segments (Java is the object-oriented platform-independent and general-purpose programming environment developed by Sun Microsystems Inc). The third interface 48 provides a path to (and in some instances from) a Java interpreter which converts the Java segments from their platform-independent form to a form recognised by the host system.

The fourth interface 49 handles the passing of signals from (and in some instances to) the user input devices.

The functions (functional modules) held in store 42 include the following: Audio manager: this controls the operation of the audio processor (24; Fig.1) where provided, or the handling of audio data processing in the host ^~ systems CPU.

MPEG decoder: software decoding of MPEG-compliant compressed image data.

Node run-time implementations: an example of these are the interpolators provided for animations, where instead of requiring specification of a location for each component at each instant, starting and finishing locations (keyframes) and times to travel between them are specified with the system providing a mechanism to derive (via interpolation) the intermediate positions, as required for compliance with VRML specifications.

Collision detection; for determining relative locations of components within the virtual environment in order to trigger so-called contact events and thereby avoid anomalous features such as permitting characters to walk 5 through supposedly solid walls.

Three-dimensional picking: following rendering of the virtual environment, picking is the VRML term for the process for generating an appropriate response to user selection (for example point-and-select using a mouse) of artifacts displayed in the environment. o - Viewpoint control: handling changes to the viewpoint location and/or orientation from which the image of the virtual environment is to be generated, in response to program instruction or user input. The movement may be an instantaneous "jump" from one predetermined viewpoint to another, or it may comprise the apparent movement relative to the virtual environment of the 5 viewpoint itself, realised by interpolating between two predetermined viewpoints. As will be understood, the viewpoint control function will have different levels of complexity depending on whether it is handling a single viewpoint for generation of a planar two-dimensional image of the environment, or a binocular pair of viewpoints to allow the generation of three-dimensional o images.

Routing mechanism: the procedure for propagating events through the node hierarchy. As an example, version 2.0 of the VRML specification has a specific communications protocol between nodes. So-called "event-out" fields - on one node are "routed" to "event-in" fields of another. An operational realisation of this would be a mouse click starting an animation. The routing 5 for this node communication would be that the event-out of a touch-sensor node goes to the event-in of a timer node, the timer is connected to an interpolator node, and the interpolator connects to the transform node of the VRML object to be animated.

The assets within store 44 are the basic building blocks and operational ι o features from which the virtual environment is built up. These include:

Three-dimensional geometry: defining the configuration and relative location of the polygons to be assembled for the virtual environment. It should be noted that each node (or screen component) will generally be defined in terms of its own co-ordinate system, which coordinates are converted by matrix 15 multiplication back through a hierarchy of component interconnections to convert all components to a common set of "world" coordinates.

Textures: surface detail for mapping onto polygon surfaces during rendering. The textures may be stored at variable levels of resolution where storage space permits and/or the system may be provided with means for 20 varying the resolution, such as interpolators operable to generate intermediate resolution texture values from a pair of pre-stored textures of different resolutions.

Audio data: audio segments, sound effects etc. to accompany particular scenes or events within the virtual environment. 25 - Java ® code: the segments of code within script nodes which are called to handle various processes or actions within the rendered scene.

User interaction data: for example, viewpoint control modes, identification of different user input devices and so forth.

MPEG data; video image data with accompanying data identifying 30 compression, stored motion vectors etc.

The further store for run-time node structures 52 contains the VRML scene graph together with routes to allow dynamic behaviour. The scene graph is a hierarchical file specifying the connections between the various geometrical - components making up an object within the virtual environment: the arrangement is a parent-child hierarchy with, for example, arms and legs being children of a body and hands and feet being respectively children of arms and legs. By way of example, Figure 3 shows a simple representation of a fan to be generated as a moving object within a virtual environment which fan is made up of four components, namely the base 60, body 62, blade 64 and a button 66. As shown by Figure 4, these components are arranged in the hierarchy with the body as a child of the base and the blade and button as separate children of the body.

It should be noted that each of the components of Figure 3 is originally specified according to its own coordinate system rather than in terms of the coordinate system of the virtual environment. This simplifies initial storage as, for example, a cylindrical component such as the body 62 may be defined in a coordinate system having an axis concurrent with the major axis of the cylinder with the only individual parameters required being the length and radius.

As shown in Figure 4, between parent and child nodes of the scene graph is a transformation T (suitably a matrix multiplication) which, when effected, translates the component parameters of the child to the coordinate system of the parent. In terms of the example of Figure 3, transformation T1 is that between the body 62 and base 60; transformation T2 is that between the blade 64 and body 62; and transformation T3 is that between the button 66 and body 62. It will also be noted that there is a further transformation TO between the base and a node identified as "world": transformation TO converts the parameters of the base and, through the matrix multiplication, all other components of the object to the coordinate system of the virtual environment. In addition to handling modification of component parameters between coordinate systems, the transformations also handle scaling of components relative to one another so that components such as spheres need not be specified for every size in which they are required but only once with different transformation scaling factors being used for each different sized spherical - component required.

A particular problem which the present invention seeks to overcome is the excessive computational load caused by multiple level hierarchies requiring a large number of matrix multiplications before a distal child component is scaled and converted to the coordinate system of the virtual environment. In the case of the example, each time the virtual environment is updated, the following compound transformations are required for the object of Figure 3: (T0.T1).T2 x (blade vertices)

(T0.T1).T3 x (button vertices)

(T0).T1 x (body vertices)

TO x (base vertices)

With the object of Figure 3 being a fan, a likely scenario is that the base, body and button components will generally remain stationary from one rendition to the next whilst the blade location will be updated on each rendition by modification of transformation T2 such as to give the appearance of the blade rotating relative to the body over a series of frames. In terms of the above necessary transformations it will be seen that this requires the calculation of (T0.T1).T2 x (blade vertices) for every refresh of the virtual environment.

Having recognised that this is wasteful in terms of processing capability, the browser embodying the present invention is arranged to store (in FiAM 14;

Fig.1) the result of a previous transformation multiplication such as (T0.T1) and also to flag, for each update of the virtual environment, which transformations have been modified and which remain as at the previous rendition. By checking the flags, the representation of the fan can simply call on the previously derived and stored result of (T0.T1) for multiplication with the updated transformation T2 rather than having to recalculate T0.T1 to repeatedly get the same result. Whilst the technique will not produce a saving if more or less all of the components are moving relative to each other at most or all refreshes of the virtual environment, this is not a particularly common scenario. The applicants having found, for many applications, the number of parent and child ^~ combinations for which the transformation does not need recalculation is generally sufficient to free a useful amount of processor capability which may then be applied to increase frame rates or handle greater complexities of scene. Reverting to the example of Figure 3, the savings that are possible are illustrated if the blade 64 is considered not as a unitary component, but as an assembly comprising the hub and four attached vanes. This is actually a construction of just two components, as the vane need only be specified once in terms of its own coordinate system, with four different matrices applied to the same stored vane to position it in equally spaced positions relative to the hub: furthermore, assuming the vanes to be equally sized, there is a further saving from performing the scaling operation just once on the stored vane before applying the four transformations. With the vane coordinates being converted to the hub coordinate system, the conversion matrices will remain constant with their flags showing "no change" on each update of the scene graph and consequently stored values of the result of the application of the conversion matrices for the vanes will be used rather than a recalculation being made.

The flowchart of Figure 5 illustrates the refresh routine implemented by the browser. In simplified form, the routine comprises the steps of getting any input (A), evaluating the effect of that input (B), updating the database (C), and output of the results (D). In detail, the routine starts by checking the time (from CLK 50) at step 100 to determine whether it is time for the refresh to commence. If so, this is followed by checking at step 102 whether there has been any input from the keys of a keyboard (18; Fig.1) and at step 104 whether there has been any input from a cursor control and selection device such as a mouse (20; Fig.1). The evaluation stage begins at step 106 with a determination as to whether any selection has been made by the cursor control. Next, at step 108, the user's position within the virtual environment is used to evaluate whether any trigger nodes (touch sensors or time sensors) have been activated - that is to say whether the user has clicked on a touch sensor or a time sensor has reached its start time.

Having evaluated the effects or otherwise of the input, the updating of the ^~ database C begins with routing, at steps 110, and 1 12. At step 1 10 all those nodes having fields invalidated by the activation of a trigger node are notified of that invalidation. At step 112, all of the invalidated nodes are given respective new values: note that this procedure must be performed "bottom up" in relation to the hierarchy to ensure that all new values are propagated. The result of this is that a node having received notification that it has an invalid field requests a new value via the routing from its input node. This input node may, in turn, have to request a new value before it can calculate the new value that has been requested from it, and so forth up the hierarchy.

At step 114, any changes to the user's position in terms of viewpoint location and/or orientation within the virtual environment are calculated, and any changes to the presented audio (such as the triggering of new audio 5 strands) are checked and the audio model updated at 116. The final stage of the updating is to update the scene graph (the node hierarchy) which involves firstly updating the transformation matrices between parent and child nodes to reflect any relative movement of the respective components (step 118) - including the setting of flags to show that changes have been made - followed o by updating the bounding boxes around the compound components at step 120.

The final stage of the refresh procedure (the output of results D) begins by sending the updated transformation matrices to the renderer at step 122 together with the updated user viewpoint at step 124. Following the image 5 rendering (step 126) the procedure reverts to step 100 until it is again time to run the refresh loop.

Although defined principally in terms of a software browser implementation, the skilled reader will be well aware than many of the above- described functional features could equally well be implemented in hardware. o From reading the present disclosure, other modifications will be apparent to persons skilled in the art. Such modifications may involve other features which are already known in the design, manufacture and use of image processing and/or data network access apparatus and devices and component^" parts thereof and which may be used instead of or in addition to features already described herein. Although claims have been formulated in this application to particular combinations of features, it should be understood that the scope of the disclosure of the present application also includes any novel feature or novel combination of features disclosed herein either explicitly or implicitly or any generalisation thereof, whether or not it relates to the same invention as presently claimed in any claim and whether or not it mitigates any or all of the same technical problems as does the present invention. The applicants hereby give notice that new claims may be formulated to such features and/or combinations of such features during the prosecution of the present application or of any further application derived therefrom.

Claims

1 . Data processing apparatus configured as a virtual environment data browser, the apparatus comprising a processor coupled with at least one memory device and data network interface means capable of receiving data defining a virtual environment and objects therein when coupled to a data network including a source of such data, the apparatus further comprising user- operable input means and rendering means configured to periodically generate an image based on said data defining the virtual environment and objects and from a viewpoint at a location and with an orientation in said virtual environment determined at least partially by input from said user-operable input means, wherein objects within the virtual environment are formed from a plurality of geometrical components, each defined according to its own coordinate system, with the components in an object being arranged in a hierarchy of parent and child components with each child being linked to its parent by a transformation specifying the conversion of the child component parameters to the parental coordinate system; characterised in that the apparatus is arranged to maintain a record, for each image generation, of the result of each transformation application, to determine which transformations have been altered as a result of movement between parent and child components and apply the altered transformation to the parameters of the child component, otherwise to use the stored result of the previous application of the transformation.

2. Apparatus as claimed in Claim 1 , wherein the means for effecting the conversion of child coordinate parameters to a parental coordinate system comprises calculation means operable to effect matrix multiplication.

3. Apparatus as claimed in Claim 1 or Claim 2, wherein the means for effecting the conversion of child coordinate parameters to a parental coordinate system includes multiplication means operable to effect scaling of child coordinate parameters.

4. Apparatus as claimed in Claim 1 , wherein the record of transformation application is held in said at least one memory device as an identifier for each transformation with an associated flag, which flag is set on the alteration of the transform and reset if the transform is unchanged following a subsequent update of the generated virtual environment.

5. Apparatus as claimed in Claim 1 , wherein a further transformation is applied, during each update of the virtual environment, to the most senior component of the or each hierarchy to convert the transformed component parameters of the components of the hierarchy to a further coordinate system specified for the virtual environment as a whole.