WO2010139984A1

WO2010139984A1 - Device and method of display

Info

Publication number: WO2010139984A1
Application number: PCT/GB2010/050899
Authority: WO
Inventors: Anthony William Godar
Original assignee: Sony Computer Entertainment Europe Limited
Priority date: 2009-06-03
Filing date: 2010-05-28
Publication date: 2010-12-09
Also published as: GB0909591D0; GB2470759A

Abstract

An entertainment device comprises a display generator operable to output a stereoscopic image for display, a first program operable to generate a monoscopic image of a virtual environment, a memory arranged to store depth information associated with the generation of the monoscopic image of the virtual environment, and an image processor; and the image processor is arranged in operation to generate a stereoscopic image for display comprising a pair of monoscopic images showing different respective viewpoints, in which at least one of the pair of monoscopic images comprises a new monoscopic image generated by the image processor, and the image processor is arranged to generate the at least one new monoscopic image by displacing elements of the monoscopic image generated by the first program dependent upon a determined respective viewpoint of the at least one new monoscopic image and the depth information associated with the monoscopic image generated by the first program.

Description

DEVICE AND METHOD OF DISPLAY

The present invention relates to a device and method of display.

Traditional video games, and particularly videogames for use on personal computers and videogame consoles, generate a rendered image of a virtual environment for display on a conventional television or computer monitor.

However, it is anticipated that next generation displays may have a 3D display capability, providing a sensation of depth for suitable media displayed on them. Unfortunately, with the exception of some specialist video games custom written to work with various experimental 3D displays, the vast majority of existing videogame titles will not be able to fully utilise this 3D capability.

The present invention seeks to mitigate the above problem.

In a first aspect of the present invention, an entertainment device is provided as recited in claim 1.

In another aspect of the present invention, a method of display is provided as recited in claim 6. Advantageously the above aspects of the present invention enable a companion image processing application (whether part of an operating system, part of a graphics card's hardware, or a combination thereof), to take a conventionally generated videogame image and generate a stereoscopic version thereof without the need to modify the source code of the videogame itself, thereby enabling such videogames to be properly displayed on 3D display systems.

Further respective aspects and features of the invention are defined in the appended claims.

Embodiments of the present invention will now be described by way of example with reference to the accompanying drawings, in which:

Figure 1 is a schematic diagram of an entertainment device;

Figure 2 is a schematic diagram of a cell processor;

Figure 3 is a schematic diagram of a video graphics processor;

Figure 4A is a schematic diagram of a stereoscopic movie camera and image pair; Figure 4B is a schematic diagram of a superposed stereoscopic image pair; Figure 5 is a flow diagram of a method of display in accordance with an embodiment of the present invention; and

Figure 6 is a schematic diagram of a stereoscopically arranged virtual camera and second viewpoint and corresponding nionoscopic images.

A device and method of display are disclosed. In the following description, a number of specific details are presented in order to provide a thorough understanding of the embodiments of the present invention. It will be apparent, however, to a person skilled in the art that these specific details need not be employed to practise the present invention. Conversely, specific details known to the person skilled in the art are omitted for the purposes of clarity where appropriate.

In an example embodiment of the present invention, an entertainment device such as a Sony ® Playstation 3 ® (or 'PS3' ®) device runs a conventional video game program that renders a single image for a default point of view within the game's virtual environment. In addition the entertainment device runs a companion image processing application (for example, as part of the operating system, or built into modified hardware such as a video card) which takes low-level information about the virtual depth of pixels within the rendered image (widely known as z-buffer information) and uses it in conjunction with the rendered single image to generate a 3D image.

In one instance, the companion application generates a second image at a second viewpoint by shifting pixels of the rendered single image according to a degree of parallax consistent with their associated depth information and the second viewpoint. Moreover, by preventing some or all so-called 'z-culling' (wherein occluded pixels are discarded before the full rendering process is complete), the companion application is also able to fill in any resulting blank pixels that occur in this process by using pixels that were effectively obscured within the original rendered image.

Alternatively or in addition, blank pixels may be filled in using texture copying or interpolation, or by access to the original texture data.

The resulting second image can be used in conjunction with the original single image as a stereoscopic image pair suitable to drive a 3D stereoscopic display.

Alternatively, where a 3D stereoscopic display is not available, the companion application may take the z-buffer data and generate a chromostereoscopic image from it (i.e. a single image coloured so as to give an illusion of depth if a viewer wears ChromaDepth ® glasses), and then modify this image using brightness information from the original rendered image to generate a chromostereoscopic version of the image that retains the textures of the original rendered image. Thus the companion application generates a 3D image (chromostereoscopic or conventional stereoscopic) from the original, conventionally generated image together with the z-buffer information, hi this way conventional video games can run without modification on an entertainment device, whilst the entertainment device outputs a 3D version of the videogame's images.

Figure 1 schematically illustrates the overall system architecture of the Sony® Playstation 3® entertainment device. A system unit 10 is provided, with various peripheral devices connectable to the system unit.

The system unit 10 comprises: a Cell processor 100; a Rambus® dynamic random access memory (XDRAM) unit 500; a Reality Synthesiser graphics unit 200 with a dedicated video random access memory (VRAM) unit 250; and an I/O bridge 700.

The system unit 10 also comprises a BIu Ray® Disk BD-ROM® optical disk reader 430 for reading from a disk 440 and a removable slot-in hard disk drive (HDD) 400, accessible through the I/O bridge 700. Optionally the system unit also comprises a memory card reader 450 for reading compact flash memory cards, Memory Stick® memory cards and the like, which is similarly accessible through the I/O bridge 700.

The I/O bridge 700 also connects to four Universal Serial Bus (USB) 2.0 ports 710; a gigabit Ethernet port 720; an IEEE 802.11b/g wireless network (Wi-Fi) port 730; and a Bluetooth® wireless link port 740 capable of supporting up to seven Bluetooth connections. In operation the I/O bridge 700 handles all wireless, USB and Ethernet data, including data from one or more game controllers 751. For example when a user is playing a game, the I/O bridge 700 receives data from the game controller 751 via a Bluetooth link and directs it to the Cell processor 100, which updates the current state of the game accordingly.

The wireless, USB and Ethernet ports also provide connectivity for other peripheral devices in addition to game controllers 751, such as: a remote control 752; a keyboard 753; a mouse 754; a portable entertainment device 755 such as a Sony Playstation Portable® entertainment device; a video camera such as an EyeToy® video camera 756; and a microphone headset 757. Such peripheral devices may therefore in principle be connected to the system unit 10 wirelessly; for example the portable entertainment device 755 may communicate via a Wi-Fi ad-hoc connection, whilst the microphone headset 757 may communicate via a Bluetooth link.

The provision of these interfaces means that the Playstation 3 device is also potentially compatible with other peripheral devices such as digital video recorders (DVRs), set-top boxes, digital cameras, portable media players, Voice over IP telephones, mobile telephones, printers and scanners.

In addition, a legacy memory card reader 410 may be connected to the system unit via a USB port 710, enabling the reading of memory cards 420 of the kind used by the Playstation® or Playstation 2® devices. In the present embodiment, the game controller 751 is operable to communicate wirelessly with the system unit 10 via the Bluetooth link. However, the game controller 751 can instead be connected to a USB port, thereby also providing power by which to charge the battery of the game controller 751. In addition to one or more analogue joysticks and conventional control buttons, the game controller is sensitive to motion in 6 degrees of freedom, corresponding to translation and rotation in each axis. Consequently gestures and movements by the user of the game controller may be translated as inputs to a game in addition to or instead of conventional button or joystick commands. Optionally, other wirelessly enabled peripheral devices such as the Playstation Portable device may be used as a controller. In the case of the Playstation Portable device, additional game or control information (for example, control instructions or number of lives) may be provided on the screen of the device. Other alternative or supplementary control devices may also be used, such as a dance mat (not shown), a light gun (not shown), a steering wheel and pedals (not shown) or bespoke controllers, such as a single or several large buttons for a rapid-response quiz game (also not shown). The remote control 752 is also operable to communicate wirelessly with the system unit 10 via a Bluetooth link. The remote control 752 comprises controls suitable for the operation of the BIu Ray Disk BD-ROM reader 430 and for the navigation of disk content.

The BIu Ray Disk BD-ROM reader 430 is operable to read CD-ROMs compatible with the Playstation and PlayStation 2 devices, in addition to conventional pre-recorded and recordable CDs, and so-called Super Audio CDs. The reader 430 is also operable to read DVD-ROMs compatible with the Playstation 2 and PlayStation 3 devices, in addition to conventional pre-recorded and recordable DVDs. The reader 430 is further operable to read BD-ROMs compatible with the Playstation 3 device, as well as conventional pre-recorded and recordable Blu-Ray Disks. The system unit 10 is operable to supply audio and video, either generated or decoded by the Playstation 3 device via the Reality Synthesiser graphics unit 200, through audio and video connectors to a display and sound output device 300 such as a monitor or television set having a display 305 and one or more loudspeakers 310. In embodiments of the present invention, the display 305 is an autostereoscopic display or other suitable 3D display. The audio connectors 210 may include conventional analogue and digital outputs whilst the video connectors 220 may variously include component video, S-video, composite video and one or more High Definition Multimedia Interface (HDMI) outputs. Consequently, video output may be in formats such as PAL or NTSC, or in 72Op, 1080i or 1080p high definition. Audio processing (generation, decoding and so on) is performed by the Cell processor

100. The Playstation 3 device's operating system supports Dolby® 5.1 surround sound, Dolby® Theatre Surround (DTS), and the decoding of 7.1 surround sound from Blu-Ray® disks.

In the present embodiment, the video camera 756 comprises a single charge coupled device (CCD), an LED indicator, and hardware-based real-time data compression and encoding apparatus so that compressed video data may be transmitted in an appropriate format such as an intra-image based MPEG (motion picture expert group) standard for decoding by the system unit 10. The camera LED indicator is arranged to illuminate in response to appropriate control data from the system unit 10, for example to signify adverse lighting conditions. Embodiments of the video camera 756 may variously connect to the system unit 10 via a USB, Bluetooth or Wi-Fi communication port. Embodiments of the video camera may include one or more associated microphones and also be capable of transmitting audio data. In embodiments of the video camera, the CCD may have a resolution suitable for high-definition video capture. In use, images captured by the video camera may for example be incorporated within a game or interpreted as game control inputs.

In general, in order for successful data communication to occur with a peripheral device such as a video camera or remote control via one of the communication ports of the system unit 10, an appropriate piece of software such as a device driver should be provided. Device driver technology is well-known and will not be described in detail here, except to say that the skilled man will be aware that a device driver or similar software interface may be required in the present embodiment described.

Referring now to Figure 2, the Cell processor 100 has an architecture comprising four basic components: external input and output structures comprising a memory controller 160 and a dual bus interface controller 170A,B; a main processor referred to as the Power Processing Element 150; eight co-processors referred to as Synergistic Processing Elements (SPEs) 11 OA-H; and a circular data bus connecting the above components referred to as the Element Interconnect Bus 180. The total floating point performance of the Cell processor is 218 GFLOPS, compared with the 6.2 GFLOPs of the Playstation 2 device's Emotion Engine. The Power Processing Element (PPE) 150 is based upon a two-way simultaneous multithreading Power 970 compliant PowerPC core (PPU) 155 running with an internal clock of 3.2 GHz. It comprises a 512 kB level 2 (L2) cache and a 32 kB level 1 (Ll) cache. The PPE 150 is capable of eight single position operations per clock cycle, translating to 25.6 GFLOPs at 3.2 GHz. The primary role of the PPE 150 is to act as a controller for the Synergistic Processing Elements 11 OA-H, which handle most of the computational workload. In operation the PPE 150 maintains a job queue, scheduling jobs for the Synergistic Processing Elements 11 OA-H and monitoring their progress. Consequently each Synergistic Processing Element 11 OA-H runs a kernel whose role is to fetch a job, execute it and synchronise with the PPE 150. Each Synergistic Processing Element (SPE) 11 OA-H comprises a respective

Synergistic Processing Unit (SPU) 120 A-H, and a respective Memory Flow Controller (MFC) 140 A-H comprising in turn a respective Dynamic Memory Access Controller (DMAC) 142A- H, a respective Memory Management Unit (MMU) 144 A-H and a bus interface (not shown). Each SPU 120A-H is a RISC processor clocked at 3.2 GHz and comprising 256 kB local RAM 130A-H, expandable in principle to 4 GB. Each SPE gives a theoretical 25.6 GFLOPS of single precision performance. An SPU can operate on 4 single precision floating point members, 4 32-bit numbers, 8 16-bit integers, or 16 8-bit integers in a single clock cycle. In the same clock cycle it can also perform a memory operation. The SPU 120 A-H does not directly access the system memory XDRAM 500; the 64-bit addresses formed by the SPU 120 A-H are passed to the MFC 140 A-H which instructs its DMA controller 142 A-H to access memory via the Element Interconnect Bus 180 and the memory controller 160.

The Element Interconnect Bus (EIB) 180 is a logically circular communication bus internal to the Cell processor 100 which connects the above processor elements, namely the PPE 150, the memory controller 160, the dual bus interface 17OA₅B and the 8 SPEs 110A-H, totalling 12 participants. Participants can simultaneously read and write to the bus at a rate of 8 bytes per clock cycle. As noted previously, each SPE 11 OA-H comprises a DMAC 142 A-H for scheduling longer read or write sequences. The EIB comprises four channels, two each in clockwise and anti-clockwise directions. Consequently for twelve participants, the longest step-wise data-flow between any two participants is six steps in the appropriate direction. The theoretical peak instantaneous EIB bandwidth for 12 slots is therefore 96B per clock, in the event of full utilisation through arbitration between participants. This equates to a theoretical peak bandwidth of 307.2 GB/s (gigabytes per second) at a clock rate of 3.2GHz.

The memory controller 160 comprises an XDRAM interface 162, developed by Rambus Incorporated. The memory controller interfaces with the Rambus XDRAM 500 with a theoretical peak bandwidth of 25.6 GB/s.

The dual bus interface 170 A₅B comprises a Rambus FlexIO® system interface 172A₃B- The interface is organised into 12 channels each being 8 bits wide, with five paths being inbound and seven outbound. This provides a theoretical peak bandwidth of 62.4 GB/s (36.4 GB/s outbound, 26 GB/s inbound) between the Cell processor and the I/O Bridge 700 via controller 170A and the Reality Simulator graphics unit 200 via controller 170B.

Data sent by the Cell processor 100 to the Reality Simulator graphics unit 200 will typically comprise display lists, being a sequence of commands to draw vertices, apply textures to polygons, specify lighting conditions, and so on. Referring now to Figure 3, the Reality Simulator graphics (RSX) unit 200 is a video accelerator based upon the NVidia® G70/71 architecture that processes and renders lists of commands produced by the Cell processor 100. In embodiments of the present invention, the RSX 200 acts as a display generator to output a stereoscopic image for display, for example by outputting left and right images of a stereoscopic pair in succession. The RSX unit 200 comprises a host interface 202 operable to communicate with the bus interface controller 170B of the Cell processor 100; a vertex pipeline 204 (VP) comprising eight vertex shaders 205; a pixel pipeline 206 (PP) comprising 24 pixel shaders 207; a render pipeline 208 (RP) comprising eight render output units (ROPs) 209; a memory interface 210; and a video converter 212 for generating a video output. The RSX 200 is complemented by 256 MB double data rate (DDR) video RAM (VRAM) 250, clocked at 600MHz and operable to interface with the RSX 200 at a theoretical peak bandwidth of 25.6 GB/s. In operation, the VRAM 250 maintains a frame buffer 214 and a texture buffer 216. The texture buffer 216 provides textures to the pixel shaders 207, whilst the frame buffer 214 stores results of the processing pipelines. The RSX can also access the main memory 500 via the EIB 180, for example to load textures into the VRAM 250.

The vertex pipeline 204 primarily processes deformations and transformations of vertices defining polygons within the image to be rendered.

The pixel pipeline 206 primarily processes the application of colour, textures and lighting to these polygons, including any pixel transparency, generating red, green, blue and alpha (transparency) values for each processed pixel. Texture mapping may simply apply a graphic image to a surface, or may include bump-mapping (in which the notional direction of a surface is perturbed in accordance with texture values to create highlights and shade in the lighting model) or displacement mapping (in which the applied texture additionally perturbs vertex positions to generate a deformed surface consistent with the texture).

The render pipeline 208 performs depth comparisons between pixels to determine which should be rendered in the final image. Optionally, if the intervening pixel process will not affect depth values (for example in the absence of transparency or displacement mapping) then the render pipeline and vertex pipeline 204 can communicate depth information between them, thereby enabling the removal of occluded elements prior to pixel processing, and so improving overall rendering efficiency. In addition, the render pipeline 208 also applies subsequent effects such as full-screen anti-aliasing over the resulting image.

Both the vertex shaders 205 and pixel shaders 207 are based on the shader model 3.0 standard. Up to 136 shader operations can be performed per clock cycle, with the combined pipeline therefore capable of 74.8 billion shader operations per second, outputting up to 840 million vertices and 10 billion pixels per second. The total floating point performance of the RSX 200 is 1.8 TFLOPS.

Typically, the RSX 200 operates in close collaboration with the Cell processor 100; for example, when displaying an explosion, or weather effects such as rain or snow, a large number of particles must be tracked, updated and rendered within the scene, hi this case, the PPU 155 of the Cell processor may schedule one or more SPEs 110A-H to compute the trajectories of respective batches of particles. Meanwhile, the RSX 200 accesses any texture data (e.g. snowflakes) not currently held in the video RAM 250 from the main system memory 500 via the element interconnect bus 180, the memory controller 160 and a bus interface controller 170B. The or each SPE 11 OA-H outputs its computed particle properties (typically coordinates and normals, indicating position and attitude) directly to the video RAM 250; the DMA controller 142 A-H of the or each SPE 11 OA-H addresses the video RAM 250 via the bus interface controller 170B. Thus in effect the assigned SPEs become part of the video processing pipeline for the duration of the task. In general, the PPU 155 can assign tasks in this fashion to six of the eight SPEs available; one SPE is reserved for the operating system, whilst one SPE is effectively disabled. The disabling of one SPE provides a greater level of tolerance during fabrication of the Cell processor, as it allows for one SPE to fail the fabrication process. Alternatively if all eight SPEs are functional, then the eighth SPE provides scope for redundancy in the event of subsequent failure by one of the other SPEs during the life of the Cell processor.

The PPU 155 can assign tasks to SPEs in several ways. For example, SPEs may be chained together to handle each step in a complex operation, such as accessing a DVD, video and audio decoding, and error masking, with each step being assigned to a separate SPE. Alternatively or in addition, two or more SPEs may be assigned to operate on input data in parallel, as in the particle animation example above.

Software instructions implemented by the Cell processor 100 and/or the RSX 200 may be supplied at manufacture and stored on the HDD 400, and/or may be supplied on a data carrier or storage medium such as an optical disk or solid state memory, or via a transmission medium such as a wired or wireless network or internet connection, or via combinations of these.

The software supplied at manufacture comprises system firmware and the Playstation 3 device's operating system (OS). In operation, the OS provides a user interface enabling a user to select from a variety of functions, including playing a game, listening to music, viewing photographs, or viewing a video. The interface takes the form of a so-called cross media-bar (XMB), with categories of function arranged horizontally. The user navigates by moving through the function icons (representing the functions) horizontally using the game controller 751, remote control 752 or other suitable control device so as to highlight a desired function icon, at which point options pertaining to that function appear as a vertically scrollable list of option icons centred on that function icon, which may be navigated in analogous fashion. However, if a game, audio or movie disk 440 is inserted into the BD-ROM optical disk reader 430, the Playstation 3 device may select appropriate options automatically (for example, by commencing the game), or may provide relevant options (for example, to select between playing an audio disk or compressing its content to the HDD 400).

In addition, the OS provides an on-line capability, including a web browser, an interface with an on-line store from which additional game content, demonstration games (demos) and other media may be downloaded, and a friends management capability, providing on-line communication with other Playstation 3 device users nominated by the user of the current device; for example, by text, audio or video depending on the peripheral devices available. The on-line capability also provides for on-line communication, content download and content purchase during play of a suitably configured game, and for updating the firmware and OS of the Playstation 3 device itself. It will be appreciated that the term "on- line" does not imply the physical presence of wires, as the term can also apply to wireless connections of various types.

In an embodiment of the present invention, the PS3 typically runs an operating system and a concurrent gaming application. During play, the gaming application will generate and maintain a virtual gaming environment, and compute the position of objects within that environment according to the current game state. The gaming application then generates a display list which, as noted previously, comprises the instructions required by the RSX 200 to render a current image depicting an appropriate view of the gaming environment, for output to a display device (e.g. a television screen) for the user to see.

A conventional gaming application will only generate a display list for rendering one image; that is to say, an image for a single screen at a single viewpoint, which may be referred to as a monoscopic image. The gaming application can thus be thought of as a first program, operable to generate a monoscopic image of a virtual environment (via the hardware and operating system of the PS3).

Consequently it would appear necessary to modify the gaming application to generate a stereoscopic image; that is to say, either a pair of images each depicting a different respective viewpoint of the same scene for use with a stereoscopic display, or a modified single image that causes a stereoscopic effect upon use of suitable glasses, such as a chromostereoscopic image viewable with ChromaDepth ® glasses.

This is undesirable as it requires that the developer of the game must spend time modifying the game and distributing the modified version to users that require it. For older games, the developer may no longer exist or may not wish to undertake the associated costs of modification. As a result such games may not be able to take advantage of new 3D-display technologies.

Consequently, in an embodiment of the present invention the operating system or a program resident in software, firmware or hardware on the RSX (or a combination of these) operates in parallel with the gaming application without substantially impinging upon the gaming application's own operation as the companion image processing application (image processor) that generates a stereoscopic image for the game without modification to the gaming application's released executable code.

It will be understood that stereoscopic display systems generally operate by displaying two images that have different respective viewpoints, typically corresponding to the different respective viewpoints of a person's left and right eyes. By way of example, referring now to Figures 4A and 4B, in conventional stereoscopic image generation a stereoscopic movie camera 1010 comprises left and right monoscopic cameras 101 OL and 101 OR that together generate a pair of images whose viewpoints are separated by a known distance δ equal to average eye separation. In Figure 4A, both monoscopic cameras 101 OL, 101 OR of the stereoscopic camera are looking at a set of objects P,Q, R, S and T at successively greater distances from the cameras, and two additional objects N and O (assumed for the purposes of explanation to be positioned above the other objects, i.e. above of the plane of the drawing). As can be seen in the resulting pair of left and right images 1012 and 1014 respectively generated by the left and right monoscopic cameras 101 OL, 101OR, the different image viewpoints result in a different image of the objects from each monoscopic camera. In Figure 4B, an image 1020 superposing the stereoscopic pair of images 1012 and 1014 illustrates that the displacement between the objects within the image pair 1012 and 1014 (illustrated by brackets between respective instances of each object) is inversely related to the distance of the object from the stereoscopic movie camera as illustrated in Figure 4 A.

Subsequently, the stereoscopic pair of images is displayed via a display mechanism (such as polarised projection and glasses with polarised filters, alternate frame sequencing and glasses with switchably occluded lenses, or lenticular lensing on an autostereoscopic display screen) that delivers a respective one of the stereoscopic pair of images 1012, 1014 to a respective eye of the viewer, and the object displacement between the images delivered to each eye causes an illusion of depth in the viewed content.

Thus, by analogy, in an embodiment of the present invention if the video game application generates a first image that may be treated as one image of a stereoscopic image pair, then the companion application (the operating system, RSX or a combination) generates a second image to complete the stereoscopic image pair.

Hence in an embodiment of the present invention, the companion image processing application is arranged in operation to generate a stereoscopic image for display comprising a pair of images showing different respective viewpoints, the first of the pair of images comprising the monoscopic image generated by the game application, and the other of the pair of images comprising a new, second monoscopic image generated by the image processing application according to a determined second respective viewpoint different to the viewpoint of the first image. To do this, the companion image processing application computes an appropriate displacement of image elements (e.g. pixels) in this second image from their positions in the first image. In an embodiment of the present invention, the companion image processing application thus generates the second monoscopic image by displacing elements of the first monoscopic image generated by the game application, the displacement of each element being dependent upon a determined second respective viewpoint and the depth information associated with the monoscopic image generated by the first program.

Determination of the second respective viewpoint may be made by calculating a lateral distance from the virtual camera used to generate the first image, thereby locating a position where a second virtual camera could be positioned (in a manner analogous to the arrangement of monoscopic cameras 101OL and 101 OR in Figure 4A) and determining this position to be the second respective viewpoint.

Referring again to Figure 3, as noted previously herein the vertex pipeline 204, pixel pipeline 206 and render pipeline 208 of the RSX 200 process polygons and textures to generate a final rendered image. In the process, depth information about pixels contributing to the image is stored in frame buffer 214 of VRAM 250. This depth information is commonly known as the z-buffer (where 'z' refers to the depth axis with respect to the plane of the image). This frame buffer thus acts as a memory to store depth information associated with the generation (rendering) of the first monoscopic image of the virtual environment by the gaming application.

As was noted previously in relation to the example of Figures 4A and 4B, the displacement of objects is related to their distance (i.e. depth) from a stereoscopic camera, and this remains true whether the camera is real or virtual. Thus given the original image produced by the gaming application and a z-buffer depth or distance value for each pixel in that image, it is possible to calculate the displacement of each pixel in the gaming application image required to generate a second image of the desired stereoscopic image pair.

Referring now to Figure 6, a virtual scene corresponding to that illustrated in Figure 4A is shown. In Figure 6 the virtual monoscopic camera of the game application is for example a right hand camera 2010R (but alternatively could equally be a left hand camera). This camera views a set of virtual objects P, Q, R, S, T, N and O similarly arranged to those described with respect to Figure 4 A, and generates a first monoscopic image 2014. This image is therefore the one conventionally generated by the game application. A determined second viewpoint 2010L is set at a distance δ from the virtual monoscopic camera. As described with respect to Figures 4A and 4B previously, this second viewpoint would result in a second monoscopic image 2012 in which the positions of the virtual objects are displaced with respect to the first monoscopic image 2014 as a function of their distance from the two viewpoints.

Therefore in an embodiment of the present invention, the image elements of the first monoscopic image corresponding to virtual objects P,Q, R, S, T, N and O are displaced by an amount consistent with the change in viewpoint between the virtual camera of the game application and the determined second viewpoint, thereby creating a stereoscopic image pair, using the distance, or depth, information contained in the z-buffer (represented in Figure 6 by example z-buffer values ZR and Zp for objects R and P respectively; it will be appreciated that each element has its own z-buffer value).

In practice the depth values in the z-buffer are often non-linear (providing higher depth resolution for near- field objects) and the relationship between distance and displacement is also non-linear. However, these relationships are each known in their respective arts and can be accounted for when calculating the required pixel displacements.

Meanwhile background elements of the virtual scene 2020 at a large distance from the virtual camera may not experience any displacement, or the calculated displacement may fall below a predetermined threshold and is not implemented in the second monoscopic image 2012.

Thus in general, the displacement of pixels in the far field (a large virtual distance) will be relatively small, whilst the displacement of pixels in the near field (a small virtual distance) will be relatively large. In either case, there is scope for the displacement of one pixel of the original, first image to be greater than that of a neighbouring pixel, effectively resulting in a gap between displaced pixels in the second image.

Where the calculated displacements result in a sub-pixel gap, then interpolation of the original pixels in their displaced positions may be used to generate new pixels in the second image.

Alternatively, the displacement (and hence the gap) may not be reproduced in the second image at all if it falls below a threshold gap size. Such a threshold may be empirically determined, for example based on viewing quality tests, but will typically correspond to a sub-pixel gap. For larger gaps (for example between the edge of a near- field object that has a large displacement and a more distant background object that has a small displacement), it may become necessary to fill in the resulting gap in the second image.

As was noted previously herein, optionally the render pipeline 208 can co-operate with the vertex pipeline 204 to remove occluded elements of the in-game scene prior to pixel processing. Such removal is known as z-culling.

Thus to provide a mechanism for filling in gaps in the second image, in an embodiment of the present invention this z-culling is inhibited by the companion image processing application. Consequently pixel processing is applied to all the elements contributing to the in-game scene by the RSX 200 before the visible elements are used to render the final game image. As a result if gaps arise in the second image, the companion image processing application can access the previously unused pixel data corresponding to unseen elements now revealed by the displacement of a near field object in the second image, and fill in the gap using that pixel data. In this case, the z-buffer and any associated memory in the RSX thus also operates as an image element memory storing image element information associated with the generation of the monoscopic game image and corresponding to image elements that were not included in the output monoscopic game image, and where a gap occurs between image elements in the second image, the image processing application can fill the gap with respective image elements obtained from the image element memory.

In a refinement of the above embodiment, where the companion application has access to the distance information during the process of generating the original game image (for example where it is part of the hardware or firmware of a suitably modified RSX or other graphics card), then displacements for the second image can be calculated at this stage and a selective z-culling of those elements that will also not be required by the second image may still be performed, as this can result in improved performance by reducing the workload in the pixel pipeline 206.

Notably in either event the above processes remain transparent to the game application. In the event that a game application explicitly controls or otherwise attempts to cause z-culling, in an embodiment of the present invention this can be intercepted and suppressed by the companion application. Optionally other commands from the game application may be suppressed or replaced as appropriate, and similarly incompatible instructions in the display list can be omitted or substituted by the companion application as required. Alternatively or in addition to the above use of previously unused pixel data, other techniques may be used to fill in gaps within the second image, including as applicable copying image elements from another part of the first image, extrapolating image elements from another part of the first image (e.g. texture patterns), interpolating image elements between other parts of the first image, and/or accessing source textures to fill in missing pixels (i.e. generating image elements from texture data).

The result is a second image that appears to be a second viewpoint of the same scene as found in the original image generated by the video game. As such the two images represent a stereoscopic pair of images suitable for display using a stereoscopic display, for example of the types mentioned previously herein.

Thus the companion application generates a second image using the pixel data of the first image together with depth information about those pixels that is held in memory, with the first and second images together forming a stereoscopic pair. Notably, therefore, the videogame itself is not required to render a second view of the environment to generate a stereoscopic image (and indeed the companion application itself does not render a whole second image either, instead using image elements from the rendered first image).

In an alternative embodiment of the present invention, rather than generate only a second image for use in conjunction with the original image generated by the game, two new images are generated with different viewpoints on either side of the viewpoint represented by the original game image, and are used to form a stereoscopic image pair that does not comprise the original game image at all. In other words, the generated stereoscopic image comprises a pair of images showing respective viewpoints different to each other and also to the original monoscopic image rendered by the game application.

This provides the advantage that the combined stereoscopic viewpoint remains centred on the original viewpoint of the image generated by the videogame, albeit at the cost of approximately doubling the additional processing required. In this case, left and right displacements corresponding to smaller changes in viewpoint (roughly half of that for the single additional image) are calculated, and two new images are generated from the original game image using the techniques described previously herein. Thus whilst typically the generated stereoscopic image comprises the monoscopic image generated by the game application and a new monoscopic image generated by the image processor, it is possible that both images are newly generated by the image processor in this fashion. Thus more generally whilst the game application still generates a first monoscopic image of a virtual environment, the generation of a stereoscopic image may involve generating one or two new monoscopic images (i.e. at least one monoscopic image) with different respective viewpoints to the original image generated by the game application.

In a further alternative embodiment, where a conventional stereoscopic display (based upon the provision of two images with different respective viewpoints) is not available, a stereoscopic effect may still be achieved using conventional colour displays by use of chromostereoscopy.

Chromostereoscopy is a technique that takes advantage of the fact that different colours are refracted at different angles in a prism (e.g. a superchromatic prism) or other suitable refractive medium. It will therefore be appreciated that by applying different colours to objects at respective depths within an image, refraction by lenses for the left and right eye can provide depth-dependent displacement of the objects in a manner similar to conventional stereoscopy (as described previously with reference to Figures 4A and 4B) using just a single false-colour image.

Therefore, in an embodiment of the present invention, the companion application generates a chromostereoscopic colour map based upon the values in the z-buffer that correspond to the pixels in the image generated by the game, and then replaces the colour in the game image with the colour from this colour map. Thus more generally, the companion application substitutes chromostereoscopic colours for those found in the original game image.

The result is a chromostereoscopic version of the image generated by the game, which provides a 3D effect on a conventional colour display for a viewer wearing appropriately refracting glasses, such as those produced by ChromaDepth ®. Optionally this effect may be switchable, for example by using the system menu button on the controller 751, and/or may be automatic based upon analysis of the game image; for example many games use a monochrome palette for night vision effects or other altered states of play, and optionally the companion application may only apply a chromostereoscopic colour map to the game when such instances are detected, providing the user with an enhanced experience for certain aspects of a game whilst retaining the original colour palette of the game at other times.

Referring now to Figure 5, a method of display comprises: in a first step (slO), generating a first monoscopic image of a virtual environment using a first program; in a second step (s20), storing depth information associated with the step of generating the first monoscopic image of the virtual environment; and in a third step (s30), generating a stereoscopic image for display based upon the generated first monoscopic image and the stored depth information.

It will be appreciated that the implementation of the above steps may substantially overlap in practice.

It will also be appreciated from the description above that in an embodiment of the present invention the third step comprises the substep of generating a stereoscopic image for display that comprises a pair of images showing different respective viewpoints, in which the first of the pair of images comprises the generated first monoscopic image, and the second of the pair of images is generated by displacing elements of the generated first monoscopic image dependent upon a determined respective viewpoint of the second image and the depth information associated with the generated first monoscopic image.

It will also be apparent to a person skilled in the art that variations in the above method corresponding to operation of the various embodiments of the apparatus as described and claimed herein are considered within the scope of the present invention, including but not limited to: the generation of the stereoscopic image comprising generating a second image based upon the generated first monoscopic image and the stored depth information, and showing a respective viewpoint different to the generated first monoscopic image, the stereoscopic image thus comprising the generated first monoscopic image and the second image; the generation of the stereoscopic image comprising generating a pair of images each based upon the generated first monoscopic image and the stored depth information, and showing respective viewpoints different to each other and to the generated first monoscopic image; storing image element information associated with the step of generating the first monoscopic image of the virtual environment and corresponding to image elements that were not included in the generated first monoscopic image, and where a gap occurs between image elements in a new monoscopic image, filling the gap with respective image elements obtained from the stored image element information; alternatively or in addition, filling the gap with one or more of copied image elements from another part of the monoscopic image rendered by the first program, extrapolated image elements from another part of the monoscopic image rendered by the first program, image elements interpolated between other parts of the monoscopic image rendered by the first program, and image elements generated from texture data; and the generation of the stereoscopic image comprising generating a chromostereoscopic image by substituting chromostereoscopic colours for those in the generated first monoscopic image in accordance with stored depth information associated with the step of generating the first monoscopic image.

Finally, it will be appreciated that the methods disclosed herein may be carried out on conventional hardware suitably adapted as applicable by software instruction or by the inclusion or substitution of dedicated hardware.

Thus the required adaptation to existing parts of a conventional equivalent device may be implemented in the form of a computer program product or similar object of manufacture comprising processor implementable instructions stored on a data carrier such as a floppy disk, optical disk, hard disk, PROM, RAM, flash memory or any combination of these or other storage media, or transmitted via data signals on a network such as an Ethernet, a wireless network, the Internet, or any combination of these of other networks, or realised in hardware as an ASIC (application specific integrated circuit) or an FPGA (field programmable gate array) or other configurable circuit suitable to use in adapting the conventional equivalent device.

Claims

1. An entertainment device, comprising a display generator operable to output a stereoscopic image for display; a first program operable to generate a monoscopic image of a virtual environment; a memory arranged to store depth information associated with the generation of the monoscopic image of the virtual environment; and an image processor; and in which the image processor is arranged in operation to generate a stereoscopic image for display comprising a pair of monoscopic images showing different respective viewpoints, in which at least one of the pair of monoscopic images comprises a new monoscopic image generated by the image processor; and the image processor is arranged to generate the at least one new monoscopic image by displacing elements of the monoscopic image generated by the first program dependent upon a determined respective viewpoint of the at least one new monoscopic image and the depth information associated with the monoscopic image generated by the first program.

2. An entertainment device according to claim 1, in which the generated stereoscopic image comprises the monoscopic image generated by the first program and a new monoscopic image generated by the image processor.

3. An entertainment device according to claim 1, in which the generated stereoscopic image comprises a pair of images showing respective viewpoints different to each other and to the monoscopic image generated by the first program.

4. An entertainment device according to any one of claims 1 to 3, in which the entertainment device comprises: an image element memory arranged to store image element information associated with the operation of generating the monoscopic image of the virtual environment by the first program and corresponding to image elements that were not included in the monoscopic image rendered by the first program; and where a gap occurs between image elements in a new monoscopic image, the image processor is arranged to fill the gap with respective image elements obtained from the image element memory.

5. An entertainment device according to any one of claims 1 to 3, in which where a gap occurs between image elements in the new monoscopic image, the image processor is arranged in operation to fill the gap with image elements obtained by one or more selected from the list consisting of: i. copying image elements from another part of the monoscopic image generated by the first program; ii. extrapolating image elements from another part of the monoscopic image generated by the first program; iii. interpolating image elements between other parts of the monoscopic image generated by the first program; and iv. generating image elements from texture data.

6. A method of display comprising the steps of: generating a first monoscopic image of a virtual environment using a first program; storing depth information associated with the step of generating the first monoscopic image of the virtual environment; and generating a stereoscopic image for display that comprises a pair of monoscopic images showing different respective viewpoints, in which at least one of the pair of monoscopic images is newly generated by displacing elements of the first monoscopic image generated using the first program, dependent upon a determined respective viewpoint of the at least one new monoscopic image and the depth information associated with the first monoscopic image generated using the first program.

7. A method of display according to claim 6, in which the stereoscopic image comprises the first monoscopic image generated using the first program and one monoscopic image newly generated by displacing elements of the first monoscopic image generated using the first program, dependent upon a determined respective viewpoint and the depth information associated with the first monoscopic image generated using the first program.

8. A method of display according to claim 6, in which the step of generating a stereoscopic image comprises generating a pair of images each based upon the generated first monoscopic image and the stored depth information, and showing respective viewpoints different to each other and to the generated first monoscopic image.

9. A method of display according to any one of claims 6 to 8, comprising the step of: storing image element information associated with the step of generating the first monoscopic image of the virtual environment and corresponding to image elements that were not included in the generated first monoscopic image, and where a gap occurs between image elements in a new monoscopic image, the step of generating a stereoscopic image comprises: filling the gap with respective image elements obtained from the stored image element information.

10. A method of display according to any one of claims 6 to 8, in which where a gap occurs between image elements in a new monoscopic image, the method comprises the step of filling the gap with image elements obtained by one or more selected from the list consisting of: i. copying image elements from another part of the first monoscopic image; ii. extrapolating image elements from another part of the first monoscopic image; iii. interpolating image elements between other parts of the first monoscopic image; and iv. generating image elements from texture data.

11. A computer program for implementing the steps of any one of claims 6 to 10.