WO2001048698A1

WO2001048698A1 - Streaming virtual reality

Info

Publication number: WO2001048698A1
Application number: PCT/US2000/034822
Authority: WO
Inventors: Morgan S. Mcguire
Original assignee: Curl Corporation
Priority date: 1999-12-29
Filing date: 2000-12-20
Publication date: 2001-07-05
Also published as: AU2111701A

Abstract

A system and method are described for processing three-dimensional data by loading the three-dimensional data for a first region into memory, the data being based on a first coordinate system. Additional three-dimensional data for an adjacent second region is also loaded into memory, the additional data being based on a second coordinate system. The data is rendered using both the three-dimensional data for the first region and the three-dimensional data for the adjacent second region. The three-dimensional data may contain a region, a portal and a delegator. When the three-dimensional data for an adjacent second region is loaded from a second memory the portal points to the delegator which includes a pointer to the three-dimensional data for the adjacent second region. Each region may have portals linking the region to adjacent regions. The regions may be stored in separate memories and managed by delegators. The delegator is capable of addressing regions and transforming coordinate systems. Regions in computer memory are processed by a garbage collector, the garbage collector works in conjunction with the delegator, to effectively manage computer memory used to store very large virtual reality worlds.

Description

STREAMING VIRTUAL REALITY

BACKGROUND OF THE INVENTION

Virtual reality ("VR") is a simulated three-dimensional ("3-D") environment that a user (viewer) can experience and manipulate as if it were physical. The user sees the environment on display screens, possibly mounted in a special pair of goggles. The user can interact with environment using conventional computer input devices such as a keyboard and mouse or by using special input devices, such as gloves or suits fitted with motion sensors used to detect the user's actions.

VR environments can be defined using Virtual Reality Modeling Language ("VRML"). VRML is a scene description language for creating 3-D interactive Web graphics similar to those found in some video games, allowing the user to "move around" within a graphic image and interact with objects. VRML, a subset of Silicon Graphics' Inventor File Format (ASCII), was created by Mark Pesce and Tony Parisi in 1994. VRML files can be created in a text editor, although CAD packages, modeling and animation packages, and VRML authoring software are the tools preferred by most VRML authors. VRML files reside on an HTTP server; links to these files can be embedded in HTML documents, or users can access the VRML files directly. To view VRML Web pages, users need a VRML-enabled browser, such as WebSpace from Silicon Graphics, or a VRML plug-in for Internet Explorer or Netscape Navigator.

Examples of existing VR systems include the Quake game by id Software, Inc. and the Unreal game by Epic Games, Inc., these systems can segment their VR environments and allow users to load parts of those environments for display. Other uses of VR include the treatment of phobias, for example simulating height in a virtual environment to allow people with a fear of height to experience the sensation of height in a safe and controlled environment. Remote medical diagnosis, guided tours and house walk-throughs have also been implemented using virtual reality. Viewing a VR environment typically requires a user to download the entire file representing the environment prior to rendering any images on the display screen. This presents a problem as the 3-D data used to simulate the virtual reality environment can be very large. To render a realistic VR environment, geometric and image data must be captured along with texture and lighting data. The limited capacity of communications links, computer memory and computer processing impose severe constraints on current VR environment developers. Developers are limited to rather small, finite, worlds to model, and transitions between worlds are undesirably visible to the user and cumbersome from both visual and storage points of view.

SUMMARY OF THE INVENTION

Creating a visually seamless VR environment capable of modeling very large, or infinite worlds is desirable to enhance the VR experience. Streaming virtual reality data, possibly from many sources, and managing it on a computer makes seamless VR environments of infinite size possible. Streaming virtual reality provides for the processing of three-dimensional data (representing a three- dimensional space) in a computer system. A first three-dimensional data corresponding to a first region of the space, the first region being based upon a first coordinate system is accessed. Additional three-dimensional data corresponding to a second region of the space is also accessed, the second region being based upon a second coordinate system, the first coordinate system and the second coordinate system being different. The first three-dimensional data is then transformed into the second coordinate system or the second three-dimensional data is transformed into the first coordinate system. At least a portion of the first three-dimensional data and at least a portion of the second three-dimensional data, each portion viewable from a viewpoint is then rendered and displayed together. In a common display frame, the portion of the three-dimensional data visible to the user is rendered from both the first region and the adjacent regions. The processing of the three-dimensional data visible to the user includes a portal pointing to its adjacent regions. In one preferred embodiment, the three-dimensional data for adjacent regions is stored separately and the portal points to a delegator which points to the adjacent regions.

The delegator includes a coordinate transform function used to transform various coordinate systems of three-dimensional image data into a form for efficient processing by the present invention. Additionally, the delegator contains a region pointer which points to an adjacent region and a region reference which points to an adjacent region when the adjacent region is stored separately. The region pointer is typically a memory address of the form native to the computer hosting the process of the present invention. The sector reference is used to address sectors stored outside of computer memory and is composed of two parts; an area name and a region name. Examples of area names include a Uniform Resource Locator ("URL") for referencing three-dimensional image data stored on World Wide Web servers connected to the Internet.

In another embodiment, the area name is a file system name capable of addressing the three-dimensional image data stored on a computer disk or CD-ROM. Additionally, a viewpoint within a sector may be included in the sector reference. Sector references can be stored to form a bookmark which can later be used to return to a specific place within the three-dimensional image.

A garbage collection technique is used in order to maintain three- dimensional image data on a computer without overwhelming its memory capabilities. Three-dimensional data for adjacent sectors least likely to be seen by the viewer is removed from computer memory to make room for new three- dimensional image data. Determining what three-dimensional image data is least likely to be seen by the viewer can be based upon a measurement of distance from the user's current position to the three-dimensional data for adjacent sectors.

The display of three-dimensional image data can be made more efficient by rendering only the portion of the three-dimensional data visible to a user. Additionally, information about the portion of the three-dimensional data visible to a user can be used to direct the loading of additional three-dimensional data, thus eliminating the need to download data that has a low probability of being seen, or is impossible to see. BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features and advantages of the invention will be apparent from the following more particular description of preferred embodiments of the invention, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention.

Fig. 1 is a block diagram of a computer configured according to an embodiment of the present invention. Fig. 2 is an illustration of a VR scene divided into sectors and portals.

Fig. 3 is a diagram showing the relationship between sectors and portals of the VR scene. Fig. 4 is an illustration of a VR scene divided into areas, sectors and portals. Fig. 5 is a diagram showing the relationship between areas, sectors and portals of the VR scene.

Fig. 6 is a diagram showing the relationship between the objects in a VR streaming hierarchy. Fig. 7 is a block diagram showing the computer memory arrangement of the objects in a VR streaming hierarchy. Fig. 8 is a flowchart showing an overview of the Streaming Virtual Reality process. Fig. 9 is a flowchart showing details of the Sector Reference Lookup process. Fig. 10 is a flowchart showing details of the Sector Rendering process. Fig. 11 is a flowchart showing details of the Portal Rendering process.

DETAILED DESCRIPTION OF THE INVENTION

Fig. 1 is a block diagram of a computer configured according to an embodiment of the present invention. Client computer 100 is a general-purpose computer capable of hosting computer graphic applications, such as those capable of displaying virtual reality environments. Client computer 100 is connected to general-purpose input devices, such as mouse 102 and keyboard 104. Additionally, client computer 100 may be connected to special-purpose input devices, such as gloves or suits fitted with motion sensors (not shown) used to detect the user's actions. Display screen 106 is used to display images to a user of the virtual reality system, display screen 106 can be a conventional Cathode-Ray Tube, Light Emitting Diode or Liquid Crystal Display screen. Additionally, special-purpose displays, such as goggles with internally-mounted display screens (not shown) may be used to display VR data. The VR data is organized in a 3D dataset and stored on 3D dataset storage 108. 3D datasets contain image data and geometry data. The geometry data and/or the image data can be compressed data. Texture and lighting data are often used to enhance VR environments, this data can be stored as part of the 3D dataset or maintained separately. 3D dataset storage 108 is connected to client computer 100 via interface 110. 3D dataset storage 108 may be a file stored on a Web-server connected to the Internet, in that case interface 110 would be an Internet link. 3D dataset storage 108 may be a file stored on a local server or on the client computer 100 itself, in that case interface 110 would be local area network connection or bus connection, respectively. Streaming virtual reality segments worlds into areas which can be stored in

3D datasets and addressed by name or pointer (when loaded in computer memory). Management of the multiple areas used to create a VR environment is performed by delegators. Delegators act as proxies for parts of the NR environment not yet loaded in computer memory. Based upon the visual focal point of a viewer, Visual Surface Determination ("VSD") techniques are used to identify which parts (sectors) of the loaded areas are visible to the viewer and only downloads the areas from the 3D dataset storage containing those sectors. The VSD techniques are also used to determine the sectors most likely to be seen next, and pre-fetches those sectors by downloading the areas in which they are contained. Since 3D datasets can be large, and computer memory is limited, any significant interaction with a VR environment could quickly overwhelm the computer memory. Garbage collection algorithms can be used to remove the areas containing the sectors least likely to be viewed, thus freeing enough computer memory to create a visually seamless VR environment capable of modeling very large, or infinite worlds.

Visual surface determination and its inverse, hidden surface removal, are computer graphics tasks important to realistic rendering of three-dimensional scenes. These tasks involve exact determination of which surfaces in a three-dimensional scene are visible from a given viewpoint. Any inaccuracy in the determination of surfaces will result in artifacts, such as surfaces farther away piercing near objects, or solid objects appearing to be hollow or inside out. These artifacts destroy the user ' s perception of a consistent, solid virtual world.

Performing visible surface computation early in the rendering process can result in performance improvements because rendering work need only be done for surfaces that are visible to the user. Unfortunately, most visible surface computation methods are time-intensive. Given the complete geometry for a world as a set of n objects (like polygons), a naive algorithm will take on the order of n² computations to determine which surfaces are visible. This is because each of the objects must be compared against all of the others to determine if it is occluded from the viewer's perspective.

The class of visibility computation algorithms known as portal algorithms group nearby objects together and require no globally recursive structure. Each scene is divided into adjacent convex cells called sectors. The faces of these convex cells are called portals. Because the convex cells are adjacent they must be polyhedral. The faces of convex polyhedra are always convex polygons, so it is implicit that the portals must all be convex polygons. The portals are "single-sided" polygons, with surface normals pointing inward towards the inside of the sectors to which they belong. The directionality of a portal surface is conventionally determined by the direction that its vertices circle. Where two sectors meet they have identically shaped, but oppositely oriented, portals. Each portal maintains a pointer to the sector it bounds and to the adjacent sector, if one exists. Some of the portals have surface data describing how they are to be rendered, these make up the solid surfaces in a scene; others lack surface data and are invisible to the viewer. Some portals are solid and cannot be walked through by the user, others are not solid and can be walked through. Portals that are opaque cannot be seen through; portals that are not opaque visually connect sectors, and thus may be seen through. Opaque portals are usually solid and non-opaque portals are usually not solid, but the two properties may be set independently. This technique of polygon rendering is one of many that can be applied to streaming virtual reality.

Fig. 2 is an illustration of a VR scene divided into sectors and portals. This scene is a room with a large extrusion (at the top right-hand side of the illustration) and an object 204. SI, S2 and S3 are sectors. SI is a sector with three portals: PI, Plb and Pic. Portals Plb and Plc have no adjacent sectors, portal PI has an adjacent Sector, S2. Sector S2 also has three portals: P2, P2b and P2c. Portal P2c has no adjacent sector; portal P2 is adjacent to sector SI; and portal P2b is adjacent to sector S3. Sector S3 has four portals: P3, P3b, P3c and P3d. Portals P3b, P3c and P3d have no adjacent sectors; portal P3 is adjacent to sector S2. Portals PI, P2, P2b and P3 contain no surface data because they are transparent, but do have pointers to adjacent sectors. Note that each portal with a adjacent sector also has a back- to-back portal associated with the adjacent sector. Each sector has a set of unique portals and each portal is single-sided.

Rendering a scene described by sector and portal data structures is straightforward and efficient. Each portal in the viewer's 200 sector is rendered sequentially. Because the sector is convex, the portal surfaces cannot occlude each other, so order is irrelevant for visibility correctness. When a portal is encountered that maintains a pointer to an adjacent sector, the portion of the displayed image that lies beyond the visual boundary is intersected with that portal's projected shape, and rendering proceeds recursively to the adjacent sector. In this way, very few surfaces are ever transformed or rendered that are not actually visible to the viewer.

The viewpoint 200 of the user determines the sight-lines 202 of the scene. The set of surfaces that will be rendered by the portal algorithm are indicated by the hashed lines along portals P2c, P3d, P3c and P3b. One way to picture this rendering is to envisage that viewpoint 200 is a flashlight, with its beam defined by sight-lines 202. The visible surface set would be the surfaces indicated by the hashed lines along portals P2c, P3d, P3c and P3b and object 204. Transparent portal line within the "beam" (i.e., PI, P2, P2b and P3) need not be rendered.

Fig. 3 is a diagram showing the relationship between sectors and portals of the VR scene in Fig. 2. Only portals with adjacent sectors are diagramed. From a viewpoint 200 in sector SI, portal PI points to sector S2, and sector S2 points to adjacent sector S3 through portal P2b. Sector S3 points through portal P3 to sector S2 and sector S2 points through portal P2 to sector S 1.

A world (scene) is a large, potentially infinite, three-dimensional space filled with complex geometry, such as a large city or an entire planet. A world can be rigorously defined as the complete visual space described by recursively following all portal references outward from a single sector. A user within a world defines an orientation or viewpoint 200 from which the three-dimensional world can be viewed. An area is a small part of a world that is conceptually complete and has relatively simple and usually approximate convex geometry. Traditional portal systems lack a concept of area, although some simulate areas by creating multiple worlds and laboriously loading and unloading them when a user moves from one location to another, this technique is insufficient for providing the streaming virtual reality of the present invention. Worlds are divided into areas based upon designers' judgement; there are no strict technical requirements. The geometry for a single area is stored as source code in a single computer file, on a single server, disk or CD- ROM. When an area is loaded into the memory of a client computer and run, it creates a table that maps sector names to sector pointers. Sectors may also be grouped into regions, which may or may not correspond to areas. A region includes one or more sectors sharing a common coordinate system for rendering and display. Regions provide a convenient way of managing sectors with the same coordinate space while avoiding the precision overflow problem associated with defining each point of a large world with a single (global) coordinate system.

Precision overflow in the presentation of a virtual world can occur when a single coordinate system is used to render and display very large worlds or when viewing objects within a world which are located visually far away from each other. Overflow occurs when too many bits may be required to define the location of high resolution image points across a large space.

Some existing three-dimensional modeling systems (e.g., 3D Studio Max produced by Kinetix®, a business unit of Autodesk®, Inc., 111 Mclnnis Parkway, San Rafael, California 94903, USA) feature separate object spaces and world spaces for modeling three-dimensional environments. Each object in the model has an object (local) coordinate system, typically centered in the middle of the object, while the entire model has a single world (global) coordinate system. Because each object must be designed (or transformed) into the common global coordinate system for rendering and display, these systems would suffer from precision overflow problems if extended to a very large space, such as that envisioned with the present invention.

Other prior art systems use a single, global coordinate system for rendering and display without offering a transformation among objects. These systems also suffer from the precision overflow problem and additionally restrict the system designer to a single coordinate system for all objects, because no transforms are performed.

Still other prior art systems have dealt with the precision overflow problem by permitting different coordinate systems to be used in adjacent sectors, but such systems do not allow the adjacent sectors to be incorporated in a common scene. The present invention uses different coordinate systems and transforms these coordinate systems upon rendering and display to prevent precision overflow. With the present invention, by relying on different coordinate systems as the viewpoint moves from sector to sector, no single coordinate system is required to define every position in the large global space and the precision overflow problem is avoided. By providing for local transforms from sector to sector and by adopting new coordinate systems as the viewpoint moves through the environment, the precision overflow problem is avoided, while allowing rendering and displaying of adjacent sectors. This technique allows very large virtual environments to be rendered and displayed without a loss of precision by adopting different coordinate systems (allowing the center of the coordinate system to stay "close" to the viewpoint) as the viewpoint moves through the three dimensional space. Fig. 4 is an illustration of a VR scene divided into areas, sectors and portals. The scene in Fig. 4 is identical to the scene in Fig. 3 with the exception that the scene is divided into two separate areas Area 1 and Area 2. Area 1 encompasses sector SI and its portals. Area 2 encompasses sector S2 and S3 and their portals. Fig. 5 is a diagram showing the relationship between the areas, sectors and portals of the VR scene in Fig. 4. Note that Area 1 contains sector SI, portal PI and delegator dl . Area 2 contains sectors S2 and S3 with portals P2, P2b and P3. The portal PI and P2 point to sectors stored in separate areas and thus use a delegator for addressing. A delegator contains a reference to a sector as an area name and a sector name, combined these two components produce a sector reference. Additionally, a viewpoint within a sector may be included in the sector reference. The exact format of sector references is not relevant; the key concept is that each area is addressed by a string that can be used to get a three-dimensional input stream. A sector reference is used to download an area that may be stored on a Web server, disk file, CD-ROM or the like.

Once the system downloads the area into client computer 100 memory, the delegator maintains a "weak" memory pointer to the sector referenced by the sector reference. A "weak" memory pointer is one which allows removal of the area containing the sector pointed to, while maintaining the ability to seamlessly reload the area when it is needed again. The combination of a sector reference and a "weak" pointer in the delegator provide for a robust three-dimensional data processing system capable of streaming virtual reality data for worlds divided into area and possibly stored separately on different servers, files, CD-ROMs or the like. Fig. 6 is a diagram showing the relationship between the objects in a VR streaming hierarchy. A world (or scene) 250 is a large, potentially infinite, three- dimensional space filled with complex geometry, such as a large city or an entire planet. A world 200 can be rigorously defined as the complete visual space described by recursively following all portal references outward from a single sector. An area 252 is a small part of a world that is conceptually complete and has relatively simple and usually approximate convex geometry. Area divisions may be made to facilitate the maintenance and development of worlds by the virtual reality developer. Each area is defined to have a coordinate system, the coordinate systems are transformed as a viewer moves through the portals in an area. This transformation in effect synchronizes the display with the area and provides continued high-resolution viewing even as the viewer moves further and further into the virtual world. Areas also facilitate storage of virtual worlds across separate (possibly networked) storage devices, this provides for more efficient management of very large virtual worlds. Each area is divided into sectors 254, which are adjacent convex cells. The faces of the sectors are called portals 256. Portals 256 are "single-sided" polygons, with surface normals pointing in towards the inside of the sectors to which they belong. The directionality of a portal surface is conventionally determined by the direction that its vertices circle. Where two sectors 254 meet they have identically shaped, but oppositely oriented, portals 256. Each portal 256 maintains a pointer to the sector 254 it bounds and to the adjacent sector 254, if one exists. Using other rendering techniques, portals can be closed areas, with surface normals pointing in towards the inside of sectors to which they belong.

Fig. 7 is a block diagram showing a possible computer memory arrangement of the objects in a VR streaming hierarchy. In the preferred embodiment an area 300 is represented as a collection of linked sector data structures 310, portal data structures 320 and delegator data structures 330. Each sector data structure 310 contains a sector name 312 and an array of portal pointers 314. Each portal data structure 320 contains a portal name 322, a sector pointer 324, a next-sector pointer 326 and image data 328 (e.g., surface and shape information). The sector pointer 324 associates a portal with the sector it is attached to, and next-sector pointer 326 defines the sector adjacent to the portal.

The delegator 330 contains a delegator name 332, a sector reference (not shown) and a weak (sector) pointer 334. When a sector adjacent to a given sector is stored in a separate area, the portal used to view the adjacent sector points to a delegator instead of the sector itself. The delegator acts as a proxy for the separately stored sector, allowing the system to process three-dimensional data as if the separately stored sector were in memory. In the example of Fig. 7, delegator dl would store a sector reference of "/http/www. curl.com/3D/area2. curl" and delegator d2 would store a sector reference of "/http/www. curl.com/3D/areal .curl". In this example the areas are stored on Web servers and addressable using the HTTP file protocol. Once the separately stored sector is loaded into memory the delegator data structure 330 is set to hold a weak pointer 334 to the now in memory sector. A weak pointer is one which allows for the removal of the sector pointed to in the event that memory becomes overused and some data structures must be removed (i.e., garbage collected or reclaimed). In the event that garbage collection removes the area containing a sector of interest, that sector can be reloaded using the sector reference stored in the delegator data structure. A strong pointer, by contrast, will prevent the memory for the area pointed to from being reclaimed.

Fig. 8 is a flowchart showing an overview of the Streaming Virtual Reality process. The viewer enters a virtual reality world at step 400 by specifying the sector reference of the initial sector from which to being the virtual reality interaction. In step 402 the system accepts this initial sector reference and in step 404 looks for the area containing the initial sector reference in memory, loading it if the area is not loaded already. Once loaded the sector is rendered in step 406. Rendering a sector involves transforming sector and portal geometry as well as culling and projecting the sector onto a display screen. Next, the portals are rendered in step 408, each portal with no adjacent sector is drawn.

Portals that point to adjacent sectors have transformations performed and each sector is recursively rendered. Portals pointing to delegators must go have the area that the delegator points to loaded into memory. The address of that memory is saved in the delegator data structure 330 and then the adjacent sectors are processed as portals pointing to adjacent sectors. The viewer's movement within the virtual reality world is tracked in step 410. If the viewer exited the virtual reality world (step 412) the Streaming Virtual Reality process ends at step 416, otherwise the viewer's current sector is determined in step 414 and processing continues at step 404. When the Streaming Virtual Reality process completes the viewer is presented with a three-dimensional virtual reality world ready for interaction. Fig. 9 is a flowchart showing details of the Sector Reference Lookup process (404). The Sector Reference Lookup process (404) starts at step 420, it accepts a sector reference (e.g., "www.curl. com/3 D/areal.curl#Sl") at step 422 then at step 424 parses the sector reference into an area name (e.g.,"www. curl.com/3D/areal .curl") and a sector name (e.g., "SI"). Using the area name the Sector Reference Lookup process (404) attempts to locate the area in the world cache (step 426). The world cache is located on the client computer 100 and stores the contents of frequently accessed areas. If the requested area is not present in the world cache, a request is made to retrieve it from the 3D dataset storage at step 428. The world cache is then evaluated, at step 430, to determine if there is enough available memory to load the desired area into the world cache for processing. If there is not enough available memory in the world cache a garbage collection process (step 432) is performed on the areas currently in the world cache. The desired area is then loaded at step 434. Once the area is loaded in the world cache the system locates, at step 436, the sector name specified. The sector is then located in step 438 and the sector pointer is returned in step 440 where the Sector Reference Lookup process ends. Caching is not limited to the client computer 100, caching can be done at any intermediary computer (e.g., a network server or Internet Service Provider) between the three-dimensional data source and the client computer 100. Fig. 10 is a flowchart showing details of the Sector Rendering process (406).

To render the viewer's viewpoint, the system begins, at step 460, a recursive algorithm starting with the sector in which the viewer is currently positioned. The sector currently being operated on is called the current sector. The viewpoint defines, and is always located in, the current sector. The geometry data for the current sector must be present on the client computer at the time the sector is to be rendered. When a sector is rendered, the geometry of the current sector is transformed (step 462). Next, the sector's portals are transformed (step 464) and pushed on the transformation stack so that the sector orientation matches the viewer's viewpoint. The transformation stack stores the transformation frames created as the viewer moves from one sector into another. Transformation frames contain scale and orientation information used to synchronize the viewer to a sector. These transformations prevent the problem of losing resolution as the viewer moves far way from an initial sector. Since dimensional coordinates are often stored using floating point numbers the predefined mantissia and exponent lengths can become unable to accurately represent the coordinates needed for high resolution graphic display as the viewer moves farther and father away from an initial sector. The transformations re-synchronize the viewer to the coordinate system of each sector entered. All portals behind the viewer (after the transform is complete) are removed from the scene in step 466 and not processed because they are not visible to the viewer. Removal of these hidden surfaces enhances performance within the virtual reality environment. Using perspective transformation, portals are projected from 3D to 2D in step 468 by drawing the portal surfaces. Portals that are facing opposite the display screen are removed from processing (step 470) because that are not visible, likewise portals that are completely outside the clipping region (the portion of the displayed image that lies beyond the visual boundary) are similarly removed in step 470. The Sector Rendering process ends at step 472.

Fig. 11 is a flowchart showing details of the Portal Rendering process (408). The Portal Rendering process (408) starts at step 480 and, in step 482, obtains the next sector pointer from the portal data structure 320. If the next sector pointer is null (step 484), the portal itself is drawn at step 502 and the Portal Rendering process (408) for that portal is complete. If the next sector pointer is not null, step 486 determines if the next sector points to a delegator (i.e., the next sector is stored separately). If it does, the sector reference stored in the delegator data structure 330 is used in step 488 by the Sector Reference Lookup process (404) to load the area containing the area pointed to by the next sector pointer. Once loaded, the delegator' s sector pointer (weak pointer) is updated in step 490 and transformations are computed (step 492) between the current sector and the next sector's geometry such that the next sector is spacially adjacent to the current sector. Whether the next sector had to be loaded or already resided in memory, the transformations are pushed onto the transformations stack at step 494. At step 496 the intersection of the clipping region and the portal's two-dimensional bounds are then determined. The next sector is then recursively rendered at step 498 using the Sector Rendering process (406). The transformation stack is then popped (step 500) and at step 502 the surface of the portal is drawn using the texture and lighting data specified in the portal data structure 320. If the texture and lighting data is not loaded on the client computer it will be retrieved from its storage location. The Portal Rendering process (408) ends at step 504.

Since retrieval of image data (e.g., texture and lighting data) can be time consuming, a pre-fetch technique can be implemented using a separate thread of execution to fetch texture and light data from a storage location in parallel with the rendering process. This texture and light data pre-fetch thread speculatively fetches data likely to be required for rendering in the near future. One pre-fetch strategy is to pre-fetch all sectors, but this can be costly as not all sectors are likely to be seen by the viewer. A more effective pre-fetch strategy is to fetch all data referenced by the next n adjacent sectors, where n is a small number (e.g., 1-10) and determined using the amount of memory available in the client computer 100. An even more effective pre-fetch strategy can take into account the viewer's orientation (e.g., only pre-fetch adjacent sectors in front of the viewer) or other attributes (e.g., a viewer might be more likely to proceed on a level surface as opposed to climbing uphill).

The limited capacity of memory in client computer 100 may be reached when areas that are stored separately are loaded onto the client computer 100 (areas are stored in the world cache which resides in memory). In that case, one or more areas must be removed from the world cache in order to make room for the area being loaded. This process of reclaiming memory for a new area is know as garbage collection. The correct time to remove areas from memory on the client computer 100 is when memory must be allocated for additional areas, for efficiency it is best to remove areas that are least likely to be used again. For three-dimensional virtual reality data this is equivalent to removing areas from memory that correspond to images "least likely to be seen again". This determination can be made by measuring the distance from the viewer's current sector to a sector targeted for removal by the garbage collection algorithm. The measurement of distance can be derived using a Most Recently Used tracking algorithm, a spacial distance algorithm, a number of chained sectors distance algorithm, or the like. Other techniques can base garbage collection on attributes of the currently rendered data, such as which portals are now closed. Additionally, the time to load a cached sector can be used in determining whether or not that sector should be garbage collected. Note that because all the sectors in a given area maintain strong (in memory) pointers to each other, whole areas are garbage collected atomically.

The methods and systems of the present invention provide the advantage of avoiding the transfer of texture and lighting data that are never seen by the viewer, thus reducing the amount of data transfer bandwidth and inproving performance. In one embodiment of the present invention a virtual reality world can be constructed from existing three-dimensional data stored as part of the World Wide Web on the Internet. For example, a designer may be assigned the task of creating a tour of sites in Washington, D.C. The designer may want to include popular tourist sites (e.g., The United States Patent and Trademark Office, The Smithsonian Air and Space Museum and The Capital). Some of these sites may already have existing virtual reality tours of their own. Using the present invention the designer can stitch together existing virtual reality worlds as well as incorporating newly created scenes to produce a true virtual reality environment on the Internet. Because streaming virtual reality can accommodate worlds using different coordinate systems and worlds of very large size (possibly stored on separate computers) a seamless virtual reality environment can be created.

While this invention has been particularly shown and described with references to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims.

Claims

CLAΓMSWhat is claimed is:

1. A method of processing three-dimensional data in a computer system, the three-dimensional data representing a three-dimensional space, the method comprising: accessing first three-dimensional data corresponding to a first region of the space, the first region being based upon a first coordinate system; accessing second three-dimensional data corresponding to a second region of the space, the second region being based upon a second coordinate system, the first coordinate system and the second coordinate system being different; transforming the first three-dimensional data into the second coordinate system or transforming the second three-dimensional data into the first coordinate system; and rendering and displaying together at least a portion of the first three- dimensional data and at least a portion of the second three-dimensional data, each portion viewable from a viewpoint.

2. The method of Claim 1 wherein each region comprises one sector.

3. The method of Claim 1 wherein the rendering and displaying together relies on the first coordinate system and transforms the second three-dimensional data into the first coordinate system.

4. The method of Claim 1 wherein the rendering and displaying together relies on the second coordinate system as the viewpoint moves from the first region to the second region.

5. The method of Claim 1 wherein the three-dimensional data corresponding to regions least likely to be seen from the viewpoint are removed from the memory.

6. The method of Claim 5 wherein the regions least likely to be seen from the viewpoint are determined based upon a measurement of distance from the viewpoint to the regions.

7. The method of Claim 1 wherein the accessing second three-dimensional data is responsive to a change in the viewpoint.

8. The method of Claim 7 wherein: the first three-dimensional data includes the first region, a portal associated with the first region and a delegator to which the portal points, the delegator including a pointer to second three-dimensional data corresponding to a second region in a second memory; the second three-dimensional data corresponding to the second region of the three-dimensional space is loaded from the second memory into the first memory; and the pointer to second three-dimensional data is updated to point to the second region in the first memory.

9. A method of processing three-dimensional data in a computer system, the three-dimensional data representing a three-dimensional space, the method comprising: accessing first three-dimensional data corresponding to a first region of the three-dimensional space from a first memory, the first three- dimensional data including the first region, a portal associated with the first region and a delegator to which the portal points, the delegator including a pointer to second three-dimensional data corresponding to a second region in a second memory; loading the second three-dimensional data corresponding to the second region of the three-dimensional space from the second memory into the first memory; updating the pointer to second three-dimensional data to point to the second region in the first memory; and rendering and displaying together at least a portion of the first three- dimensional data and at least a portion of the second three-dimensional data, each portion viewable from a viewpoint.

10. The method of Claim 9 wherein each region comprises one sector.

11. The method of Claim 9 wherein the delegator includes: a coordinate transform function.

12. The method of Claim 11 wherein the delegator includes: a region pointer which points to the second region in the first memory; and a region reference which points to the second region when the second region is stored in the second memory.

13. The method of Claim 12 wherein the region reference is composed of two parts; an area name and a region name.

14. The method of Claim 13 wherein the area name is a Uniform Resource Locator.

15. The method of Claim 13 wherein the area name is a file system name.

16. The method of Claim 12 wherein the region reference is stored to form a bookmark.

17. A method of providing a streaming virtual reality environment in a computer system, the method comprising: receiving a stream of three-dimensional data, the three-dimensional data representing a three-dimensional space, different sections of the stream being based upon different coordinate systems; and rendering and displaying together the different portions of the stream, each portion viewable from a viewpoint.

18. The method of Claim 17 wherein the different sections of the stream are received from different locations on a computer network.

19. The method of Claim 17 wherein the three-dimensional data includes a first region, a second region, a portal and a delegator, the delegator including a pointer to the second region.

20. The method of Claim 19 wherein each region comprises one sector.

21. A computer system for processing three-dimensional data, the three- dimensional data representing a three-dimensional space, comprising: first three-dimensional data corresponding to a first region of the space, the first region being based upon a first coordinate system; second three-dimensional data corresponding to a second region of the space, the second region being based upon a second coordinate system, the first coordinate system and the second coordinate system being different; and a processor rendering and displaying together at least a portion of the first three-dimensional data and at least a portion of the second three- dimensional data, each portion viewable from a viewpoint.

22. The computer system of Claim 21 wherein each region comprises one sector.

23. The computer system of Claim 21 wherein the processor relies on the first coordinate system and transforms the second three-dimensional data into the first coordinate system.

24. The computer system of Claim 21 wherein the processor relies on the second coordinate system as the viewpoint moves from the first region to the second region.

25. The computer system of Claim 21 wherein the three-dimensional data corresponding to regions least likely to be seen from the viewpoint are removed from the memory.

26. The computer system of Claim 25 wherein the regions least likely to be seen from the viewpoint are determined based upon a measurement of distance from the viewpoint to the regions.

27. The computer system of Claim 21 wherein the accessing second three- dimensional data is responsive to a change in the viewpoint.

28. The computer system of Claim 27 wherein: the first three-dimensional data includes the first region, a portal associated with the first region and a delegator to which the portal points, the delegator including a pointer to second three-dimensional data corresponding to a second region in a second memory; the second three-dimensional data corresponding to the second region of the three-dimensional space is loaded from the second memory into the first memory; and the pointer to second three-dimensional data is updated to point to the second region in the first memory.

29. A computer system for processing three-dimensional data in a computer system, the three-dimensional data representing a three-dimensional space, comprising: first three-dimensional data corresponding to a first sector of the space from a first memory, the three-dimensional data corresponding to the first sector including a sector, a portal and a delegator to which the portal in the three-dimensional data corresponding to the first sector points, the delegator including a pointer to the three-dimensional data corresponding to a second sector in a second memory; second three-dimensional data corresponding to a second sector of the space from a second memory, including a pointer in the delegator to the second sector in the first memory; and a processor rendering and displaying together at least a portion of the first three-dimensional data and at least a portion of the second three- dimensional data, each portion viewable from a viewpoint.

30. The computer system of Claim 29 wherein each region comprises one sector.

31. The computer system of Claim 29 wherein the delegator includes: a coordinate transform function.

32. The computer system of Claim 31 wherein the delegator further includes: a region pointer which points to the second region; and a region reference which points to the second region when the second region is stored in the second memory.

33. The computer system of Claim 32 wherein the region reference is composed of two parts; an area name and a region name.

34. The computer system of Claim 33 wherein the area name is a Uniform Resource Locator.

35. The computer system of Claim 33 wherein the area name is a file system name.

36. The computer system of Claim 32 wherein the region reference is stored to form a bookmark.

37. A computer system for providing a streaming virtual reality environment, comprising: a stream of three-dimensional data, the three-dimensional data representing a three-dimensional space, different portions of the stream being based upon different coordinate systems; and a processor rendering and displaying the different portions of the stream, each portion viewable from a viewpoint.

38. The computer system of Claim 37 wherein the different portions of the stream are received from different locations on a computer network.

39. The computer system of Claim 37 wherein the three-dimensional data includes a first region, a second region, a portal and a delegator, the delegator including a pointer to the second region.

40. The computer system of Claim 39 wherein each region comprises one sector.

41. A computer system for processing three-dimensional data, the three- dimensional data representing a three-dimensional space, comprising: means for accessing first three-dimensional data corresponding to a first region of the space, the first region being based upon a first coordinate system; means for accessing second three-dimensional data corresponding to a second region of the space, the second region being based upon a second coordinate system, the first coordinate system and the second coordinate system being different; and means for rendering and displaying together at least a portion of the first three-dimensional data and at least a portion of the second three- dimensional data, each portion viewable from a viewpoint.

42. A computer system for processing three-dimensional data in a computer system, the three-dimensional data representing a three-dimensional space, comprising: means for accessing first three-dimensional data corresponding to a first region of the space from a first memory, the three-dimensional data corresponding to the first region including a region, a portal and a delegator to which the portal in the three-dimensional data corresponding to the first region points, the delegator including a pointer to the three-dimensional data corresponding to a second region in a second memory; means for accessing second three-dimensional data corresponding to a second region of the space from a second memory, including a pointer in the delegator to the second region in the first memory; and means for rendering and displaying together at least a portion of the first three-dimensional data and at least a portion of the second three- dimensional data, each portion viewable from a viewpoint.

43. A computer system for providing a streaming virtual reality environment, comprising: means for receiving a stream of three-dimensional data, the three- dimensional data representing a three-dimensional space, different portions of the stream being based upon different coordinate systems; and means for rendering and displaying together the different portions of the stream, each portion viewable from a viewpoint.

44. A computer program product comprising: a computer usable medium for processing three-dimensional data; a set of computer instructions embodied on the computer usable medium, including instructions to: access first three-dimensional data corresponding to a first region of the space, the first region being based upon a first coordinate system; access second three-dimensional data corresponding to a second region of the space, the second region being based upon a second coordinate system, the first coordinate system and the second coordinate system being different; and render and display together at least a portion of the first three- dimensional data and at least a portion of the second three-dimensional data, each portion viewable from a viewpoint.

45. A computer program product comprising: a computer usable medium for processing three-dimensional data; a set of computer instructions embodied on the computer usable medium, including instructions to: access first three-dimensional data corresponding to a first region of the space from a first memory, the three-dimensional data corresponding to the first region including a region, a portal and a delegator to which the portal in the three-dimensional data corresponding to the first region points, the delegator including a pointer to the three-dimensional data corresponding to a second region in a second memory; load the second three-dimensional data corresponding to a second region of the space from a second memory, including a pointer in the delegator to the second region in the first memory; and render and display together at least a portion of the first three- dimensional data and at least a portion of the second three-dimensional data, each portion viewable from a viewpoint.

46. A computer program product comprising: a computer usable medium for providing a streaming virtual reality environment; a set of computer instructions embodied on the computer usable medium, including instructions to: receive a stream of three-dimensional data, the three-dimensional data representing a three-dimensional space, different portions of the stream being based upon different coordinate systems; and render and display together the different portions of the stream, each portion viewable from a viewpoint.

47. A propagated signal carried on an electromagnetic waveform, the signal comprising a set of computer program instructions to: access first three-dimensional data corresponding to a first region of the space, the first region being based upon a first coordinate system; access second three-dimensional data corresponding to a second region of the space, the second region being based upon a second coordinate system, the first coordinate system and the second coordinate system being different; and render and display together at least a portion of the first three- dimensional data and at least a portion of the second three-dimensional data, each portion viewable from a viewpoint.

48. A propagated signal carried on an electromagnetic waveform, the signal comprising a set of computer program instructions to: access first three-dimensional data corresponding to a first region of the space from a first memory, the three-dimensional data corresponding to the first region including a region, a portal and a delegator to which the portal in the three-dimensional data corresponding to the first region points, the delegator including a pointer to the three-dimensional data corresponding to a second region in a second memory; load the second three-dimensional data corresponding to a second region of the space from a second memory, including a pointer in the delegator to the second region in the first memory; and render and display together at least a portion of the first three- dimensional data and at least a portion of the second three-dimensional data, each portion viewable from a viewpoint.

49. A propagated signal carried on an electromagnetic waveform, the signal comprising a set of computer program instructions to: receive a stream of three-dimensional data, the three-dimensional data representing a three-dimensional space, different portions of the stream being based upon different coordinate systems; and render and display together the different portions of the stream, each portion viewable from a viewpoint.

50. A method of processing three-dimensional data in a computer system, the three-dimensional data representing a three-dimensional space, the method comprising: defining, through the three-dimensional space, individual coordinate systems; reading and displaying data viewable from a viewpoint based upon a coordinate system local to the viewpoint; moving the viewpoint through the three-dimensional space and adopting a second coordinate system local to the moved viewpoint; rendering and displaying data viewable from the moved viewpoint based upon the second coordinate system local to the moved viewpoint.

51. The method of Claim 50 wherein: the three-dimensional data is received as a stream, different sections of the stream being based upon different coordinate systems; and rendering and displaying together the different portions of the stream, each portion viewable from a viewpoint.

52. The method of Claim 51 wherein the three-dimensional data corresponding to regions least likely to be seen from the viewpoint are removed from memory.

53. The method of Claim 50 wherein the three-dimensional data comprises: a first three-dimensional data corresponding to a first region in a first memory; a portal associated with the first region; and a delegator to which the portal points, the delegator including a pointer to a second three-dimensional data corresponding to a second region in a second memory.