WO2007048197A1

WO2007048197A1 - Systems for providing a 3d image

Info

Publication number: WO2007048197A1
Application number: PCT/AU2006/001602
Authority: WO
Inventors: Steven John Scheding
Original assignee: The University Of Sydney
Priority date: 2005-10-26
Filing date: 2006-10-26
Publication date: 2007-05-03

Abstract

Systems, methods, apparatuses, computer program products are described for providing a 3D image. In one system for providing a 3D image, the system comprises a camera array (414) for generating at least one image of a multi- dimensional scene (412), the camera array (414) comprising a master camera and at least one slave camera; a monitor (416) coupled to the camera array (414) for transmitting the generated image; and a physical lens array (418) having a rectangular configuration related to the display of the monitor (416) to transmit the rendered camera views and provide the 3D image (420). An input device (422) may be provided for user interaction.

Description

SYSTEMS FOR PROVIDING A 3D IMAGE

FIELD OF THE INVENTION

The present invention relates generally to three-dimensional (3D) image visualization of spatial data and in particular to 3D imaging systems and methods.

BACKGROUND

Many applications exist where environmental data must be gathered, stored (and perhaps transformed algorithmically), and then transmitted to a human user. Such applications include Geospatial Information Systems (GIS) used by the

Mining, Agriculture and Defence industries, for example, medical imaging, and x- ray inspection systems to name a few. A shortcoming of such applications is that the resultant data is frequently transmitted to a user as a projection onto a two dimensional space. Because the user sees a representation on a computer screen or television, the visualization is inherently limited to two dimensions (2D).

Because the real world is three dimensional (3D), a human may find interacting with a 2D representation of a 3D (or higher dimensional) space to be difficult and frustrating. This is an ergonomic issue caused by users' familiarity with the everyday world. Further, this is not a recent problem, as evidenced by various different methods that have attempted to provide a synthetic 3D image for human viewers. Such methods may be thought of as arising from the Virtual Reality (VR) movement of the early 1980s (see Okoshi, T., Three dimensional imaging techniques, Academic Press, London, 1976), including the use of VR helmets, shutter glasses, blue / red glasses, polaroid glasses, CAVE systems, lenticular arrays, and lens arrays.

The foregoing methods have several disadvantages for use in communicating spatial data to a user or a group of users. A number of the methods disadvantageously require a user to wear glasses or headsets. The methods may use custom built hardware, i.e., not readily available equipment, or custom software, or both. Still further, several methods may only be used by a single user at any one time. Lippman, G., "La Photographie integrale", Comtes Rendus, Academie des Sciences, Vol. 146, 1908, pp. 446 - 451) describes using microlens arrays to create, record on photographic film, and replay 3D images (referred to as Integral Imaging). The technique resembles holography, except that the technique uses natural light and reproduces true color optical models. Existing computers may be used to implement Integral Videography, which constructs color, optical video holograms. To capture a light field, a microlens array 312 is placed in the aperture of a conventional camera (e.g., video, CCD, and still cameras) 314, as shown by the system 300 of Fig. 3. The camera 314 captures the scene 310 through the lens array 312. Use of the array 312 is functionally equivalent to having multiple small cameras, each of which records the scene from a slightly different location. A monitor 316 (using a photo, television or computer monitor as the light source) coupled to the camera 314 retransmits this captured image (or images in the case of video) through another lens array 318. This causes the light field captured by the camera 314 to be reassembled as a reconstructed scene 320 in the space in front of the transmission device 316. This gives a viewer 322 the illusion 320 of seeing the original scene 310 in a full three dimensions.

Integral Videography allows the illusion of 3D to be shared by many participants and is limited only by display size and resolution. A viewer 322 does not need any special viewing device and there is no 'sweet spot', so the viewer may move around the display device 316, 318. The field of view is limited however, so that anyone outside of the field of view does not see the illusion of 3D.

Research into Integral Videography is aimed at reproducing video captured with a television camera, for television displays. However, this is not particularly useful for the above applications for which 'synthetic', or computer generated, images need to be displayed.

Such systems that aim at computer-generated integral video have several disadvantages. The systems are typically based on ray tracing techniques, which are slow. The systems also tend to model the entire integral photography setup by modeling the lens array and a single camera. Such systems are not capable of real-time holographic video.

U.S. Patent No. 6,496,183 issued to Bar-Nahum on 17 December 2002, is assigned to Koninklijke Philips Electronics N.V., and is entitled, "Filter for transforming 3D data in a hardware accelerated rendering architecture". This patent describes a computerized method and apparatus for generating stereoscopic three-dimensional (3D) graphics using a stereoscopic filter that transforms the 3D data. The stereoscopic filter is a software component loaded by the operating system, such as Microsoft Windows, that intercepts function calls intended for a 3D hardware acceleration driver, via an API, and generates viewpoint data for the 3D graphic object. The filter generates left-eye and right-eye viewpoint data and outputs the generated viewpoint data for storage in a frame buffer of a display driver. That is, every 3D object is separated into two instances or viewpoints, i.e., a left viewpoint and a right viewpoint. In a system that renders the 3D scene into a projection of a point in space, the stereoscopic filter viewer model representation replaces the rendering request from the 3D rendering engine by two different rendering requests to the 3D acceleration driver. A digtal-to-analaog converter (DAC) outputs analog RGB signals to a stereoscopic display, which creates a stereoscopic effect in the viewer's mind, given that the two frames are correctly displaced to match the distance between the eyes. The stereoscopic display is described as a head-mounted display that has a separate monitor for each eye, where the RGB signal is modulated to create an effect of simulated depth vision. A projection system is described as an alternative display that projects two views, where the viewer uses special glasses that blocks one view and passes the other view delivering a different view to each eye.

The disadvantages of such stereo systems are that such systems usually require custom hardware/headset and force a user's eyes to a single vergence, which frequently causes nausea. -A-

SUMMARY

In accordance with an aspect of the invention, there is provided a method of providing a 3D image. At least one image of a multi-dimensional scene is generated using a camera array. The multi-dimensional scene comprises an internal representation of a synthetic multi-dimensional scene. The camera array comprises a master camera and at least one slave camera. An input device may be provided for user interaction. The generated image is transmitted using a monitor. The rendered camera views are transmitted and the 3D image is provided using a physical lens array having a rectangular configuration related to the display of the monitor.

The camera array may comprise hardware cameras or computer- implemented virtual cameras.

The lens array may comprise at least two Fresnel lenses, plano-convex lenses, convex-convex lenses, a microlens array, a compound lens arrangement, or a lenticular lens array.

The monitor may be a flat-screen, cathode ray tube (CRT) device, an LCD display, and a plasma display.

In accordance with another aspect of the invention, there is provided a system for providing a 3D image. The system comprises: a camera array for generating at least one image of a multi-dimensional scene, the multi-dimensional scene comprising an internal representation of a synthetic multi-dimensional scene, the cameras comprising a master camera and at least one slave camera; an input device for user interaction; a monitor coupled to the camera array for transmitting the generated image; and a physical lens array having a rectangular configuration related to the display of the monitor to transmit the rendered camera views and provide the 3D image.

In accordance with still another aspect of the invention, there is provided a system for providing a 3D image. The system comprises: a virtual camera array for rendering an image of a synthetic multi-dimensional scene, the camera array comprising a master camera and at least one slave camera; a monitor coupled to the virtual camera array for transmitting the rendered image; and a physical lens array having a rectangular configuration related to the shape of the monitor to provide the 3D image

The camera array may comprise hardware cameras or computer- implemented virtual cameras. The virtual camera array may be implemented using software.

The system may further comprise a computer-generated, integral video module, the module comprising the virtual camera array and the internal representation of the synthetic multi-dimensional scene.

In accordance with a further aspect of the invention, there is provided a method of providing a 3D image. The method comprises the steps of: rendering an image of a synthetic multi-dimensional scene using a virtual camera array; providing an internal representation of the synthetic multi-dimensional scene; transmitting the rendered image using a monitor; and providing the 3D image using a physical lens array having a configuration related to the display of the monitor.

In accordance with still a further aspect of the invention, there is provided a method of producing a 3D image from computer-generated data. The method comprises the steps of: providing a master camera of a virtual camera array at a position; translating, rotating, or both, at least one slave camera of the virtual camera array, the position of each slave camera being relative to the master camera; rendering camera views of the computer-generated data representing a 3D scene using the virtual camera array; transmitting the rendered camera views using a monitor coupled to the virtual camera array; and reconstructing the 3D image dependent upon the rendered camera views using a physical lens array having a configuration related to the display of the monitor. The method may further comprise the step of initializing the master and slave cameras.

The method may further comprise the step of creating or loading an initial geometry. The method may further comprise the step of creating a render window.

The method may further comprise the step of creating Tenderers. The method may further comprise the step of creating and initializing the master camera and at least one slave camera. The method may further comprise the step of rendering initial camera views. The method may further comprise the step of interactively modifying the computer- generated data representing the 3D scene in response to an input device.

The lens array may comprise at least two Fresnel lenses, plano-convex lenses, convex-convex lenses, a microlens array, a compound lens arrangement, or a lenticular lens array. In accordance with yet another aspect of the invention, there is provided an apparatus for producing a 3D image from computer-generated data. The apparatus comprises: a module for providing a master camera of a virtual camera array at a position; a module for translating, rotating, or both, at least one slave camera of the virtual camera array, the position of each slave camera being relative to the master camera; a module for rendering camera views of the computer- generated data representing a 3D scene using the virtual camera array; a monitor coupled to the virtual camera array transmitting the rendered camera views; and a physical lens array for reconstructing the 3D image dependent upon the rendered camera views of the computer-generated data representing the 3D scene.

In accordance with a further aspect of the invention, there is provided a computer program product comprising a computer readable medium with a computer program recorded thereon for producing a 3D image from computer- generated data. The computer program product comprises: a computer program code module for providing a master camera of a virtual camera array at a position; a computer program code module for translating, rotating, or both, at least one slave camera of the virtual camera array, the position of each slave camera being relative to the master camera; a computer program code module for rendering camera views of the computer-generated data representing a 3D scene using the virtual camera array; a computer program code module for transmitting the rendered camera views using a monitor coupled to the virtual camera array; and a computer program code module for reconstructing the 3D image dependent upon the rendered camera views using a lens array having a configuration related to the display of the monitor.

In accordance with a still further aspect of the invention, there is provided a system for producing a 3D image from computer-generated data. The system comprises: an interface for communicating data; a storage unit for storing computer instructions and the data; a processing unit coupled to the interface and the storage unit; a monitor coupled to the virtual camera array to transmit the rendered camera views and the interface; and a physical lens array to reconstruct the 3D image dependent upon the rendered camera views of the computer- generated data representing the 3D scene. The processing unit is programmed with program code to: provide a master camera of a virtual camera array at a position; translate, rotate, or both, at least one slave camera of the virtual camera array, the position of each slave camera being relative to the master camera; render camera views of the computer-generated data representing a 3D scene using the virtual camera array; and transmit using the interface the rendered camera views of the computer-generated data representing the 3D scene; and

In each of the foregoing aspects, the 3D image may be a 3D autostereoscopic image. The 3D image may be selectively rendered as a real or a virtual image. The method may, further comprise the step of, for each camera, rendering an image of the 3D scene into a render window at a location referenced to the camera array to render the 3D image as a virtual image. The method may further comprise the steps of: for each camera, rendering an image of the 3D scene; and placing the rendered image into a render window referenced to the camera array to render a real image as follows: if the image is taken by the master camera, leaving the image as is; and if the image is taken by one camera above the master camera, rendering the image to a location one place below the master camera; and if an image is taken from the 2nd to the right, 3rd above the master camera, the image is rendered to the position 2 places to the left and 3 below the master camera's reference point.

These and other aspects of the invention are set forth in greater detail hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are described hereinafter with reference to the drawings, in which:

Fig. 1 is a normal, three-dimensional (3D) image of a cow, which is provided as an example and is computer generated; Fig. 2 is a multiple component 3D image of the cow of Fig. 1, which has been processed in accordance with an embodiment of the invention; Fig. 3 is a block diagram of an Integral Videography system; Fig. 4 is a block diagram of a system for producing 3D autostereoscopic images in accordance with an embodiment of the invention; Fig. 5 is a plan view of a physical lens array with which the system of

Fig. 4 may be practiced;

Fig. 6 is a flow diagram illustrating a process of initializing the autostereoscopic system;

Fig. 7 is a flow diagram of a process of event flows in accordance with the embodiment of the invention; Fig. 8 is a block diagram of a general-purpose computer system with which embodiments of the invention may be practiced;

Fig. 9 illustrates the relationship between renderers and a render window in accordance with embodiments of the invention; Figs. 1OA and 1OB are plan and top views of a lenticular lens array with which the system of Fig. 4 may be practiced;

Fig. 11 is a block diagram of a system for producing 3D autostereoscopic images in accordance with another embodiment of the invention;

Fig. 12 is a flow diagram illustrating a method of providing a autostereoscopic image in accordance with an embodiment of the invention; and

Figs. 13 A and 13B are real and virtual images of the earth (Antarctica in view) in accordance with another embodiment of the invention.

DETAILED DESCRIPTION Systems and methods for providing a 3D image are described hereinafter.

Also, methods, apparatuses, computer program products, and systems producing a 3D image are described. In the following description, numerous specific details, including particular computer systems, lens configurations, interactive input devices, and the like are set forth. However, from this disclosure, it will be apparent to those skilled in the art that modifications and/or substitutions may be made without departing from the scope and spirit of the invention. In other circumstances, specific details may be omitted so as not to obscure the invention.

A module, and in particular its functionality, can be implemented in either hardware or software. In the software sense, a module is a process, program, or portion thereof that usually performs a particular function or related functions.

Such software may be implemented in C, C++, JAVA, JAVA BEANS, Fortran, or a combination thereof, for example, but may be implemented in any of a number of other programming languages/systems, or combinations thereof. In the hardware sense, a module is a functional hardware unit designed for use with other components or modules. For example, a module may be implemented using discrete electronic components, or it may form at least a portion of an entire electronic circuit such as a Field Programmable Gate Arrays (FPGA), Application Specific Integrated Circuit (ASIC), and the like. A physical implementation may also comprise configuration data for a FPGA, or a layout for an ASIC, for example. Still further, the description of a physical implementation may be in EDIF netlisting language, structural VHDL, structural Verilog, or the like.

Numerous other possibilities exist. Those skilled in the art will appreciate that the system may also be implemented as a combination of hardware and software modules.

Some portions of the following description are presented in terms of algorithms and representations of operations on data within a computer system or other device capable of performing computations, e.g. a Personal Digital Assistant (PDA). Such algorithmic descriptions and representations may be used by those skilled in the art to convey the substance of their work to others skilled in the art. An algorithm is a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic, or electromagnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. These signals may be referred to as bits, values, elements, symbols, characters, terms, numbers, or the like. The above and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to such quantities. Unless specifically stated otherwise, and as apparent from the following, discussions utilizing terms such as "creating", "computing", "loading", "initializing", "rendering", "getting", "translating", "rotating" "copying", "performing", or the like, refer to the action and processes of a computer system, or a similar electronic device. Such a system or device manipulates and transforms data represented as physical quantities within the registers and memories of the computer system into other data similarly represented as physical quantities within the computer system registers, memories, or another form of storage, transmission or display devices. Apparatuses and systems for performing the operations of the methods are also described. Such an apparatus may be specifically constructed for the required purpose, e.g., providing a 3D image. Alternatively, the apparatus may comprise a general-purpose computer or another computing device, which may be selectively activated or reconfigured by a computer program read by the computer. The algorithms presented herein are not inherently related to any particular computer or other apparatus; various general-purpose machines may be used with programs.

The embodiments of the invention also relate to a computer program(s) or software, in which method steps may be put into effect by computer code. The computer program is not intended to be limited to any particular programming language, operating environment, and implementation thereof. A variety of programming languages, operating systems, and coding thereof may be used. Moreover, the computer program is not intended to be limited to any particular control flow. There are many other variants of the computer program, which can use different control flows without departing the scope and spirit of the invention. Furthermore, one or more of the steps of the computer program may be performed in parallel rather than sequentially.

The computer program may be stored on any computer readable medium. The computer readable medium may comprise storage devices, such as magnetic media disks, CD-ROMs, DVDs, flash RAM devices, memory chips, memory cards, magnetic tape, other storage devices and media suitable for interfacing with and being read by a general-purpose computer, and combinations thereof. The computer readable medium may also include a hard- wired medium, such as a local area network or the Internet, or wireless medium, such as an IEEE 802.11 wireless network, a GSM mobile telephone system, PCS, and GPS. The computer program when loaded and executed on such a general-purpose computer effectively results in an apparatus that implements the method steps of the embodiments.

The methods of the embodiments comprise particular control flows. However, different control flows can be practice without departing from the scope and spirit of the invention. L Introduction

The embodiments of the invention provide an autostereoscopic video display that may be interactive, based on the principles of Integral Videography. This involves an applied real-time framework for creating and displaying totally or partially synthetic autostereoscopic video. The embodiments of the invention may allow a user or users to interact with completely synthetic objects.

Systems in accordance with embodiments of the invention model the lens array as an array of cameras. In one embodiment, the array of cameras is a virtual camera array implemented in software. In another embodiment, the array of cameras is an arrangement of physical cameras. In the software implementation, the virtual camera array when coupled with modern graphics accelerators allows a significant increase in speed. The system may implement autostereoscopic video (e.g., frame rates greater than 24 frames per second).

In the virtual camera implementation, the system is able to produce true color autostereoscopic video from computer-generated (or modified) data in realtime. This effectively communicates spatial information, both for the scientific community, and also the general population. In contrast, most existing technologies for communicating spatial data are inherently two dimensional (i.e. computer monitors, televisions and still images) and are therefore ill suited to presenting 3D information to a user, who is used to a 3D world. The embodiments of the invention provide synthetic autostereoscopic video in real time and may allow the user to have direct, physical interaction with the resultant autostereoscopic projection in a way that is natural for a human user.

Further, the areas of application for the embodiments of the invention are extensive, with applications useful for scientific visualization problems, industry projects, and even at the consumer level, for example, by providing new levels of immersion in computer games. Another application is autostereoscopic video conferencing, which may be achieved with little change to the existing infrastructure. The resulting ability for spatial information to be communicated widely, both amongst individuals and within groups, affects the way in which people can interact with such information. By being a natural and intuitive way of interacting with complex 3D data, the embodiments of the invention may improve productivity in numerous applications. For example, in the underground mining industry, the remote control of vehicles (i.e., teleoperation) is widely considered to be a significant goal, both for increasing productivity by having a single operator control multiple vehicles and for safety since the operator is remote from the vehicle. However, live video from the vehicle is insufficient for an operator to exert satisfactory control over the machine, as depth cues in the 2D video stream that the operator normally uses are missing. Using the system according to the embodiments of the invention overcomes this limitation. The significant economic potential (e.g., gains in productivity and savings due to increased safety) exists in such applications.

The following description first sets forth the software implementation of the system, i.e., using a virtual camera array, followed by details of the hardware implementation of the camera array.

Generally speaking, the embodiments of the invention involve a method 1200 of providing a 3D image, as shown in Fig. 12. Processing commences in step 1210. In step 1212, at least one image of a multi-dimensional scene is generated using a camera array. The multi-dimensional scene comprises an internal representation of a synthetic multi-dimensional scene. The cameras comprise a master camera and at least one slave camera. In step 1214, an input device may be provided for user interaction. In step 1216, the generated image is transmitted using a monitor. In step 1218, the rendered camera views are transmitted and the 3D image is provided using a physical lens array having a rectangular configuration related to the display of the monitor. In step 1220, processing terminates. The camera array may comprise either hardware cameras or computer-implemented virtual cameras. The lens array may comprise at least two Fresnel lenses, plano-convex lenses, convex-convex lenses, a microlens array, a compound lens arrangement, or a lenticular lens array. The monitor may be a flat-screen, cathode ray tube (CRT) device, an LCD display, or a plasma display. II. System Architecture

Fig. 4 shows diagrammatically a system 400 in accordance with an embodiment of the invention. The system 400 comprises a computer-generated, integral-video module 410, a display device 416, and a physical lens array 418. The computer-generated, integral-video module 410 comprises a virtual camera array 414. The module 410 also comprises an internal representation of a 3D scene 412. The internal representations 412 used may be any of those standard within the computer graphics and visualizations fields. Fig. 1 is an image 100 of such a computer-generated scene, which in this example is a 3D representation of a cow. The virtual camera array 414 captures images of the internal representation 412 of the 3D scene and provides output from the module 410 to the monitor 416. Fig. 2 depicts the captured image 200 generated by the virtual camera array 414 for the image 100 of Fig. 1.

The monitor 416 retransmits the captured image, or images in the case of video, through the lens array 418 to produce the reconstructed scene 420 that the viewer(s) observe. This is the three-dimensional, autostereoscopic image that is produced. The monitor 416 may be implemented using a true or near true, flat- screen light source, including a television or computer monitor. The monitor 418 may also be implemented using an LCD or Plasma screen, such as a TFT LCD screen available with many laptop and notebook computers, as a separate monitor for a computer, or as a television display.

The "notional" light field captured by the virtual camera array 414 is reassembled as the reconstructed scene 420 in the space in front of, or behind, the monitor 416. A 3D image in front is known as a 'real' image, while one behind the screen is known as a 'virtual' image. The terms "front" and "behind" are relative to the lens array. This is typically close enough to the monitor screen that there is no noticeable difference. This gives the viewer 424 the illusion 420 of seeing the internal representation of the 3D scene 412 in a full three dimensions.

The system 400 may also comprise an input device 422 coupled to the computer-generated, integral-video module 410. The input device 422 may be a haptic interaction device, such as a force-feedback device. Other devices may be practiced including those that are tactile in nature. The input device 422 may be configured to interact with the camera array 414, or the internal representation of the scene 412, or both; this may be software configurable. In one embodiment, the input device 422 allows the viewer 424 to interact with the internal representation of the 3D scene 412. The input device 422 may pick and/or move an object(s), and/or animate a scene. The input device 422 may provide the viewer 424 with the illusion that the autostereoscopic object 420 has real physical properties. In another embodiment, the input device 422 may cause the cameras to move, for example. Such input devices 422 are available off the shelf. However, any mechanism that is capable of generating an event may be used as an input device. The mechanism may be a mouse, touchpad, touch screen, keyboard, or similar input device, or even algorithmically generated events, e.g., events that may be used for animations or flythroughs, etc.

The computer- generated, integral-video module 410 may be implemented using a general-purpose computer system, such as a desktop computer, a workstation, or a portable computer, such as a laptop or notebook computer.

In this embodiment of the invention, the computer system generates the synthetic, integral video. Also, the external feedback loop from the input device 422 provides feedback to the module 410. This allows the internal representation 412 to be modified as a result of viewer interaction. In another embodiment, the cameras may be moved. The entire process takes place in real-time. "Real-time" video involves anything capable of producing more than 24 images per second (i.e., film speed). However, variations as to the number of images per second may be practiced without departing from the scope and spirit of the invention. To achieve real-time autostereoscopic video using synthetic data, modern computer-graphics hardware is exploited. Such computer-graphics hardware typically uses the triangle as its basic geometric primitive. If the lens array comprises square-packed lenses (rather than hexagonal, which is commonly used), the graphics card may be used to quickly simulate the "virtual lens array" of the virtual camera array 410, because each lens is equivalent to a conventional camera. If hexagonal lenses are used, ray-tracing techniques may be used. In this embodiment of the invention, square-packed lens arrays are used. This technique may be simply integrated with existing software, such as Geospatial Information Systems, CAD programs, and the like. This may be done simply by providing a new graphics "driver" for the software to use. The original software may therefore be used, without any changes.

By using parallel computers, the display may be sped up linearly with the number of computers used. Alternatively, parallel computers may be used to drive multiple monitors, for a autostereoscopic 'video wall'. III. Physical lens array The physical lens array 418 may be a microlens array made using conventional molding techniques. A negative blank may be milled using a high precision CNC mill and then polished. An optically clear polyurethane resin is poured into the mould to create the positive final product for the lens array 418. This process may be repeated several times to generate arrays for the monitor in varying sizes and lens spacings. However, this technology may easily be adapted to injection molding techniques, allowing mass production of inexpensive microlens arrays capable of being used with a variety of computer monitors, televisions, and cameras.

Fig. 5 is a plan view of a physical lens array 500, with which embodiments of the invention may be practiced. The microlens array comprises at least two lenses. In this embodiment, the array 500 comprises 6 lens arrays 512, each comprising 4X4 Fresnel lenses, arranged in two rows by three columns. Each lens array 512 has dimensions of 77 mm X 77 mm in this example. While the microlens array 500 is described as comprising 6 arrays, a single unitary array 500 may be practiced.

In addition to Fresnel lenses, the lenses may be plano-convex lenses, convex-convex lenses, or a compound lens arrangement. Still further, the lens array 418 may be a lenticular lens array 1000, such as the one shown in Figs. 1OA and 1OB. A lenticular lens array 1000 comprises a number of lenticules arranged on a sheet. A single lenticule lens element is characterized by curvature along one axis only. The array 1000 comprises repeated, identical elongated lenses arranged horizontally across the lens array 1000. The structure of the lenticular lens allows the viewer to see different images with each eye and therefore to see 3D images. Further, the physical lens array may have a rectangular configuration related to the shape the display of the monitor. The compound lens arrangement may be like one found in a camera.

IV. Autostereoscopic Process

Figs. 6 and 7 are flow diagrams illustrating the initialization and operating processes 600 and 700, respectively, for the computer-generated, integral video module 410 of Fig. 4. In Fig. 6, processing commences in step 602, where system parameters are setup. The system parameters may comprise the number of lenses (rows and columns), and the relative spacing between the cameras in the virtual lens array (these must match the physical lens arrays 418), for example. In step 604, an initial geometry is created or loaded. The initial geometry comprises the objects in the scene to be rendered. These objects are usually stored in files on disk (in various formats) and must be loaded into memory. Objects may also be created algorithmically. This step may also comprise the creation of any lighting and/or any special shading instructions. The initial geometry may also refer to both the scene (all the objects, lighting, any special effects, etc.) and the position and orientation of the cameras with respect to the scene. In step 606, a render window is created. Fig. 9 illustrates a render window 900, which is the window on the computer desktop into which the images are rendered. In Fig. 9, the image of the cow 100 of Fig. 1 is depicted. The render window 900 may be full screen or only part of the screen (or be multiple screens combined). The render window 900 may have a border and a title, or no border.

In step 608, a renderer(s) is created. The renderers 910 of Fig. 9 are 2D windows into which the 3D geometry is drawn. For this application, there may be one renderer 910 per 'virtual camera'. In step 610, a "master" camera is created and initialized in the virtual camera array 414. In step 610, "slave" cameras are created and initialized in the virtual camera array 414. The master camera controls operation of the slave cameras. The slave cameras are offset to the master window according to the parameters set in the block "Setup System Parameters". There is a parameter 'Delta' that specifies the distance between camera centers. The cameras may be assumed to lie on a plane, although this does not have to be the case. Therefore, the 'virtual cameras' all lie on the same plane (set by the master) and are regularly positioned according to the parameter 'Delta'. More complex arrangements are possible however. In step 614, the initial camera views are rendered. Once the steps 602-614 are completed, the initialization process 600 terminates. Processing then continues at the event operating process for rendering views of Fig. 7. In step 702 of Fig. 7, the master camera settings are obtained ("get" in step

702). In step 704, for each slave camera, the slave camera views are translated and/or rotated to new positions. The new positions are relative to the master camera. In step 706, for each slave camera, any other parameters are copied from the master camera, if necessary. In step 708, the camera views are rendered. Once the views are rendered, the resulting image is transmitted from the computer- generated, integral video module 410 to the monitor 416, for transmission to the viewer 424 via the lens array 418. This produces the reconstructed, synthetic scene 420 seen by the viewer 424. V. Another System Architecture Fig. 11 shows diagrammatically another system 1100 in accordance with a further embodiment of the invention. The system 1100 comprises a camera array 1114, a display device 1116, and a physical lens array 1118. In place of the internal representation of a 3D scene 412 of Fig. 4, there is a real 3D scene 1112. The camera array 1114 captures images of the 3D scene 1112 and provides output to the monitor 1116.

The video output from the camera array 1114 may be captured by a computer(s), using any of a number of technologies well known to those skilled in the art. For example, if the cameras are digital and produce Digital Video (DV), the digital video output from the cameras may be provided to the computer using a Firewire interface (IEEE 1394). Alternatively, if the cameras are analog, the computer may use a video capture card to capture the analog video and produce digital video for processing by the computer. Software may be used with the computer to generate a single image comprising the individual camera images. This image is then provided to the monitor screen.

In another embodiment, the video output from the camera array may be fed to a custom signal processing board that combines the elemental images into a single video output. The single video output can then either be connected straight to a video monitor, or captured on computer for possible further processing. In still another embodiment, each camera may capture an image from behind a lens array. These systems may be formed into a single larger array. The above methods can then be used to create a single video stream.

The monitor 1116 retransmits the captured image, or images in the case of video, through the lens array 1118 to produce the reconstructed scene 1120 that the viewer(s) 1124 observe. Again, this is the three-dimensional, autostereoscopic image that is produced. The monitor 1116 may be of the type of monitors 416 discussed above.

The light field captured by the camera array 1114 is reassembled as the reconstructed scene 1120 in the space in front of, or behind, the monitor 1116. Again, this gives the viewer 1124 the illusion 1120 of seeing the 3D scene 1112 in a full three dimensions. As with the system 400 of Fig. 4, this embodiment may also comprise an input device 1122, such as any of the mechanisms described with reference to the device 422. The input device 1122 is used to interact with the camera array. Optionally, the input device may be used to interact with the 3D scene or object, for example, if such an object able to be moved responsive to user input. A computer system may be used to generate the synthetic, integral video, dependent upon the camera configuration (as noted above). If feedback exists in the configuration, the feedback may be used to control the position and orientation of the camera array. For example, the array may be mounted on a pan-tilt platform (e.g., those used for security cameras), and a joy stick may be used to change the position and orientation of the platform. Other variations of this arrangement may be practiced without departing from the scope and spirit of the invention. Further entire process takes place in real-time, as described above. VI. Rendering Real and Virtual Images

Unlike stereoscopic systems, the embodiments of the invention are capable of more general application, for example, expensive stereoscopic headgear is not required. The embodiments of the invention do not have to provide separate left- and right-eye views, but instead provide a light field given the use of a physical lens array in front of the monitor. The method set forth hereinafter provides for the generation of real and virtual images, in essentially the same number of steps as the foregoing embodiments. Fig. 13A illustrates a rendered real image 1300 (i.e., in front of the screen), and Fig. 13B illustrates the corresponding image rendered for a virtual (behind the screen) image of the planet. The source image geometry is a sphere. Table 1 hereinafter sets forth an algorithm for generating the images that can be applied directly to hardware systems, as well as software systems:

TABLE 1

VII. Computer Implementation

The methods according to the embodiments of the invention may be practiced using one or more general-purpose computer systems, handheld devices, and other suitable computing devices, in which the processes described with reference to Figs. 1, 2, and 4-11 may be implemented as software, such as an application program executing within the computer system or a handheld device. In particular, instructions in the software that are carried out by the computer effect the steps in the method of providing an autostereoscopic image are effected, at least in part. Software may include one or more computer programs, including application programs, an operating system, procedures, rules, data structures, and data. The instructions may be formed as one or more code modules, each for performing one or more particular tasks. The software may be stored in a computer readable medium, comprising one or more of the storage devices described below, for example. The computer system loads the software from the computer readable medium and then executes the software. Fig. 8 depicts an example of a computer system 800 with which the embodiments of the invention may be practiced. A computer readable medium having such software recorded on the medium is a computer program product. The use of the computer program product in the computer system may effect an advantageous apparatus for providing an autostereoscopic image in accordance with the embodiments of the invention. Fig. 8 illustrates the computer system 800 in block diagram form, coupled to a network. An operator may use the keyboard 830 and/or a pointing device such as the mouse 832 (or touchpad, for example) to provide input to the computer 850. The computer system 800 may have any of a number of output devices, including line printers, laser printers, plotters, and other reproduction devices connected to the computer. The computer system 800 can be connected to one or more other computers via a communication interface 864 using an appropriate communication channel 840 such as a modem communications path, router, or the like. The computer network 820 may comprise a local area network (LAN), a wide area network (WAN), an Intranet, and/or the Internet, for example. The computer 850 may comprise a processing unit 866 (e.g., one or more central processing units) 866, memory 870 which may comprise random access memory (RAM), read-only memory (ROM), or a combination of the two, input/output (IO) interfaces 872, a graphics interface 860, and one or more storage devices 862. The storage device(s) 862 may comprise one or more of the following: a floppy disc, a hard disc drive, a magneto-optical disc drive, CD- ROM, DVD, a data card or memory stick, flash RAM device, magnetic tape or any other of a number of non-volatile storage devices well known to those skilled in the art. While the storage device is shown directly connected to the bus in Fig. 8, such a storage device may be connected through any suitable interface, such as a parallel port, serial port, USB interface, a Firewire interface, a wireless interface, a PCMCIA slot, or the like. For the purposes of this description, a storage unit may comprise one or more of the memory 870 and the storage devices 862 (as indicated by a dashed box surrounding these elements in Fig. 8). Each of the components of the computer 850 is typically connected to one or more of the other devices via one or more buses 880, depicted generally in Fig. 8, that in turn comprise data, address, and control buses. While a single bus 880 is depicted in Fig. 8, it will be well understood by those skilled in the art that a computer or other electronic computing device, such as a PDA, may have several buses including one or more of a processor bus, a memory bus, a graphics card bus, and a peripheral bus. Suitable bridges may be utilized to interface communications between such buses. While a system using a CPU has been described, it will be appreciated by those skilled in the art that other processing units capable of processing data and carrying out operations may be used instead without departing from the scope and spirit of the invention. The computer system 800 is simply provided for illustrative purposes, and other configurations can be employed without departing from the scope and spirit of the invention. Computers with which the embodiment can be practiced comprise IBM-PC/ ATs or compatibles, laptop/notebook computers, one of the Macintosh (TM) family of PCs, Sun Sparcstation (TM), a PDA, a workstation or the like. The foregoing are merely examples of the types of devices with which the embodiments of the invention may be practiced. Typically, the processes of the embodiments, described hereinafter, are resident as software or a program recorded on a hard disk drive as the computer readable medium, and read and controlled using the processor. Intermediate storage of the program and intermediate data and any data fetched from the network may be accomplished using the semiconductor memory.

In some instances, the program may be supplied encoded on a CD-ROM or a floppy disk, or alternatively could be read from a network via a modem device connected to the computer, for example. Still further, the software can also be loaded into the computer system from other computer readable medium comprising magnetic tape, a ROM or integrated circuit, a magneto-optical disk, a radio or infra-red transmission channel between the computer and another device, a computer readable card such as a PCMCIA card, and the Internet and Intranets comprising email transmissions and information recorded on websites and the like. The foregoing is merely an example of relevant computer readable mediums. Other computer readable mediums may be practiced without departing from the scope and spirit of the invention.

The embodiments of the invention enhance and evolve data representation techniques to suit the visualization process involved in integral videography, develop a real-time framework for creating and displaying totally or partially synthetic autostereoscopic video, and allow a viewer to interact with computer generated holographic objects in real-time. The embodiments assist a user to be more productive in applications where the user is dealing with spatial data, by providing a more intuitive and easy-to-use interface.

A small number of embodiments of the invention regarding systems and methods for providing an autostereoscopic image have been described. Also, methods, apparatuses, computer program products, and systems producing an autostereoscopic image from computer-generated data have been described. In the light of the foregoing, it will be apparent to those skilled in the art in the light of this disclosure that various modifications and/or substitutions may be made without departing from the scope and spirit of the invention.

Claims

CLAIMSThe claims defining the invention are as follows:

1 . A method of providing a 3D image, said method comprising the steps of: generating at least one image of a multi-dimensional scene using a camera array, said multi-dimensional scene comprising an internal representation of a synthetic multi-dimensional scene, said camera array comprising a master camera and at least one slave camera; providing an input device for user interaction; transmitting said generated image using a monitor; and transmitting said rendered camera views and providing said 3D image using a physical lens array having a rectangular configuration related to the display of said monitor.

2. The method according to claim 1, wherein said camera array is selected from the group consisting of a plurality of hardware cameras and a plurality of computer-implemented virtual cameras.

3. The method according to claim 1, wherein said lens array is selected from the group consisting of: at least two Fresnel lenses; plano-convex lenses; convex-convex lenses; a microlens array; a compound lens arrangement; and a lenticular lens array.

4. The method according to claim 1, wherein said monitor is selected from the group consisting of flat-screen, cathode ray tube (CRT) device, an LCD display, and a plasma display.

5. A system for providing a 3D image, said system comprising: a camera array for generating at least one image of a multi-dimensional scene, said multi-dimensional scene comprising an internal representation of a synthetic multi-dimensional scene, said camera array comprising a master camera and at least one slave camera; an input device for user interaction; a monitor coupled to said camera array for transmitting said generated image; and a physical lens array having a rectangular configuration related to the display of said monitor to transmit said rendered camera views and provide said 3D image.

6. A system for providing a 3D image, said system comprising: a virtual camera array for rendering an image of a synthetic multidimensional scene, said camera array comprising a master camera and at least one slave camera; a monitor coupled to said virtual camera array for transmitting said rendered image; and a physical lens array having a rectangular configuration related to the shape of said monitor to provide said 3D image.

7. The system according to claim 5 or 6, wherein said camera array is selected from the group consisting of a plurality of hardware cameras, and a plurality of computer-implemented virtual cameras.

8. The system according to claim 7, wherein said virtual camera array is implemented using software.

9. The system according to claim 5, or 6, wherein said monitor is selected from the group consisting of flat-screen, cathode ray tube (CRT) device, an LCD display, and a plasma display.

10. The system according to claim 5 or 6, wherein said lens array is selected from the group consisting of: at least two Fresnel lenses; plano-convex lenses; convex-convex lenses; a microlens array; a compound lens arrangement; and a lenticular lens array.

11. The system according to claim 5 or 6, further comprising a computer-generated, integral video module, said module comprising said virtual camera array and said internal representation of said synthetic multi-dimensional scene.

12. The system according to claim 7, further comprising a mechanism that is capable of generating an event.

13. The system according to claim 12, wherein said mechanism is selected from the group consisting of a keyboard, a mouse, a haptic interaction device, and an algorithm for generating events.

14. The system according to claim 12, further comprising a computer- generated, integral video module, said module comprising said virtual camera array and an internal representation of said synthetic multi-dimensional scene, said mechanism coupled to said computer-generated, integral video module.

15. The system according to claim 11 or 14, wherein said computer- generated, integral video module comprises: a module that moves a master camera of said virtual camera array; and a module that translates, rotates, or both, at least one slave camera of said virtual camera array.

16. The system according to claim 15, wherein said computer- generated, integral video module comprises a module that renders camera views based on said master and at least one slave cameras to provide said rendered image of said synthetic 3D scene.

17. A method of providing a 3D image, said method comprising the steps of: rendering an image of a synthetic multi-dimensional scene using a virtual camera array; providing an internal representation of said synthetic multi-dimensional scene; transmitting said rendered image using a monitor; and providing said 3D image using a physical lens array having a configuration related to the display of said monitor.

18. The method according to claim 17, wherein said virtual camera array comprises a master camera and at least one slave camera.

19. The method according to claim 17, wherein said virtual camera array is implemented using software.

20. The method according to claim 17, wherein said monitor is selected from the group consisting of flat-screen, cathode ray tube (CRT) device, an LCD display, and a plasma display.

21. The method according to claim 18, wherein said lens array is selected from the group consisting of: at least two Fresnel lenses; plano-convex lenses; convex-convex lenses; a microlens array; a compound lens arrangement; and a lenticular lens array.

22. The method according to claim 17, wherein a computer-generated, integral video module comprises said virtual camera array and said internal representation of said synthetic multi-dimensional scene.

23. The method according to claim 17, further comprising the step of providing a mechanism that is capable of generating an event.

24. The method according to claim 23, wherein said mechanism is selected from the group consisting of a keyboard, a mouse, a haptic interaction device, and an algorithm for generating events.

25. The method according to claim 24, wherein a computer-generated, integral video module comprises said virtual camera array and an internal representation of said synthetic multi-dimensional scene, said mechanism coupled to said computer-generated, integral video module.

26. The method according to claim 22 or 25, further comprising the steps of: moving a master camera of said virtual camera array; and translating, rotating, or both, at least one slave camera of said virtual camera array.

27. The method according to claim 26, further comprising the step of rendering camera views based on said master and at least one slave cameras to provide said rendered image of said synthetic 3D scene.

28. A method of producing a 3D image from computer-generated data, said method comprising the steps of: providing a master camera of a virtual camera array at a position; translating, rotating, or both, at least one slave camera of said virtual camera array, the position of each slave camera being relative to said master camera; rendering camera views of said computer-generated data representing a 3D scene using said virtual camera array; transmitting said rendered camera views using a monitor coupled to said virtual camera array; and reconstructing said 3D image dependent upon said rendered camera views using a physical lens array having a configuration related to the display of said monitor.

29. The method according to claim 28, further comprising the step of initializing said master and slave cameras.

30. The method according to claim 28, further comprising the step of creating or loading an initial geometry.

31. The method according to claim 28, further comprising the step of creating a render window.

32. The method according to claim 31, further comprising the step of creating renderers.

33. The method according to claim 32, further comprising the step of creating and initializing said master camera and at least one slave camera.

34. The method according to claim 33, further comprising the step of rendering initial camera views.

35. The method according to claim 28, further comprising the step of interactively modifying said computer-generated data representing said 3D scene in response to an input device.

36. The method according to claim 28, wherein said lens array is selected from the group consisting of: at least two Fresnel lenses; plano-convex lenses; convex-convex lenses; a microlens array; a compound lens arrangement; and a lenticular lens array.

37. An apparatus for producing a 3D image from computer-generated data, said apparatus comprising: means for providing a master camera of a virtual camera array at a position; means for translating, rotating, or both, at least one slave camera of said virtual camera array, the position of each slave camera being relative to said master camera; means for rendering camera views of said computer-generated data representing a 3D scene using said virtual camera array; a monitor coupled to said virtual camera array transmitting said rendered camera views; and a physical lens array for reconstructing said 3D image dependent upon said rendered camera views of said computer-generated data representing said 3D scene.

38. The apparatus according to claim 37, further comprising means for initializing said master and slave cameras.

39. The apparatus according to claim 37, further comprising means for creating or loading an initial geometry.

40. The apparatus according to claim 39, further comprising means for creating a render window.

41. The apparatus according to claim 40, further comprising means for creating Tenderers.

42. The apparatus according to claim 41, further comprising means for creating and initializing said master camera and said at least one slave camera.

43. The apparatus according to claim 42, further comprising means for rendering initial camera views.

44. The apparatus according to claim 37, further comprising means for interactively modifying said computer-generated data representing said 3D scene in response to an input device.

45. The apparatus according to claim 37, wherein said lens array is selected from the group consisting of: at least two Fresnel lenses; plano-convex lenses; convex-convex lenses; a microlens array; a compound lens arrangement; and a lenticular lens array.

46. A computer program product comprising a computer readable medium with a computer program recorded thereon for producing a 3D image from computer-generated data, said computer program product comprising: computer program code means for providing a master camera of a virtual camera array at a position; computer program code means for translating, rotating, or both, at least one slave camera of said virtual camera array, the position of each slave camera being relative to said master camera; computer program code means for rendering camera views of said computer-generated data representing a 3D scene using said virtual camera array; computer program code means for transmitting said rendered camera views using a monitor coupled to said virtual camera array; and computer program code means for reconstructing said 3D image dependent upon said rendered camera views using a lens array having a configuration related to the display of said monitor.

47. The computer program product according to claim 46, further comprising computer program code means for initializing said master and slave cameras.

48. The computer program product according to claim 46, further comprising computer program code means for creating or loading an initial geometry.

49. The computer program product according to claim 46, further comprising computer program code means for creating a render window.

50. The computer program product according to claim 49, further comprising computer program code means for creating renderers.

51. The computer program product according to claim 50, further comprising computer program code means for creating and initializing said master camera and said at least one slave camera.

52. The computer program product according to claim 51 , further comprising computer program code means for rendering initial camera views.

53. The computer program product according to claim 46, further comprising computer program code means for interactively modifying said computer-generated data representing said 3D scene in response to an input device.

54. The computer program product according to claim 46, wherein said lens array is selected from the group consisting of: at least two Fresnel lenses; plano-convex lenses; convex-convex lenses; a microlens array; a compound lens arrangement; and a lenticular lens array.

55. A system for producing a 3D image from computer-generated data, said system comprising: an interface for communicating data; a storage unit for storing computer instructions and said data; a processing unit coupled to said interface and said storage unit, said processing unit being programmed with program code to: provide a master camera of a virtual camera array at a position; translate, rotate, or both, at least one slave camera of said virtual camera array, the position of each slave camera being relative to said master camera; render camera views of said computer-generated data representing a 3D scene using said virtual camera array; and transmit using said interface said rendered camera views of said computer-generated data representing said 3D scene; and a monitor coupled to said virtual camera array to transmit said rendered camera views and said interface; and a physical lens array to reconstruct said 3D image dependent upon said rendered camera views of said computer-generated data representing said 3D scene.

56. The system according to claim 55, wherein said processing unit is programmed to initialize said master and slave cameras.

57. The system according to claim 55, wherein said processing unit is programmed to create or load an initial geometry.

58. The system according to claim 55, wherein said processing unit is programmed to create a render window.

59. The system according to claim 58, wherein said processing unit is programmed to create renderers.

60. The system according to claim 59, wherein said processing unit is programmed to create and initialize said master camera and said at least one slave camera.

61. The system according to claim 60, wherein said processing unit is programmed to render initial camera views.

62. The system according to claim 55, further comprising an input device to interactively modify said computer-generated data representing said 3D scene.

63. The system according to claim 55, wherein said lens array is selected from the group consisting of: at least two Fresnel lenses; plano-convex lenses; convex-convex lenses; a microlens array; a compound lens arrangement; and a lenticular lens array.

64. The method according to any one of claims 1-4 and 17-36, wherein said 3D image is a 3D autostereoscopic image.

65. The system according to any one of claims 5-16 and 55-63, wherein said 3D image is a 3D autostereoscopic image.

66. The apparatus according to any one of claims 37-45, wherein said 3D image is a 3D autostereoscopic image.

67. The computer program product according to claim 46-54, wherein said 3D image is a 3D autostereoscopic image.

68. The method according to any one of claims 1-4 and 17-36, wherein said 3D image is selectively rendered as a real or a virtual image.

69. The method according to claim 68, further comprising the step of, for each camera, rendering an image of said 3D scene into a render window at a location referenced to said camera array to render said 3D image as a virtual image.

70. The method according to claim 68, further comprising the steps of: for each camera, rendering an image of said 3D scene; and placing said rendered image into a render window referenced to said camera array to render a real image as follows: if said image is taken by said master camera, leaving said image as is; and if said image is taken by one camera above said master camera, rendering said image to a location one place below said master camera; and if an image is taken from the 2nd to the right, 3rd above said master camera, said image is rendered to the position 2 places to the left and 3 below the master camera's reference point.

71. The system according to any one of claims 5-16 and 55-63, wherein said 3D image is selectively rendered as a real or a virtual image.

72. The system according to claim 71, further comprising means for, for each camera, rendering an image of said 3D scene into a render window at a location referenced to said camera array to render said 3D image as a virtual image.

73. The system according to claim 71, further comprising: means for, for each camera, rendering an image of said 3D scene; and means for placing said rendered image into a render window referenced to said camera array to render a real image as follows: if said image is taken by said master camera, leaving said image as is; and if said image is taken by one camera above said master camera, rendering said image to a location one place below said master camera; and if an image is taken from the 2nd to the right, 3rd above said master camera, said image is rendered to the position 2 places to the left and 3 below the master camera's reference point.