JP4703635B2 - Stereoscopic image generation method, apparatus thereof, and stereoscopic image display apparatus - Google Patents

Description

  The present invention relates to a stereoscopic image display technique configured by combining a flat panel display of a type in which one pixel is composed of a plurality of subpixels and an optical element group.

  In recent years, research and development have been conducted on autostereoscopic image display devices that allow a user to view stereoscopic images without wearing special glasses or other devices. There are various types of mounting methods for this device. A method for displaying a stereoscopic image integrating multi-directional information on a display and forming a stereoscopic image via an optical element, etc. -There are aerial image methods. The binocular and multi-view types have restrictions on the user's viewpoint position, but the aerial image method does not assume a specific viewpoint position, but artificially configures the light state where an image exists in space There are few restrictions on the viewpoint position.

  Multi-view and aerial image methods with a large number of viewpoints require multi-directional information to generate a stereoscopic image, so when performing real-time rendering in a stereoscopic display application that interacts with user input In the stereoscopic image generating device with low processing capability, there is a problem that the update speed of the display content is slow.

  In Patent Document 1, regarding a binocular stereoscopic image display device, a stereoscopic image for a content background and a character is generated in advance as a sprite, and the processing load for generating a stereoscopic image is reduced by superimposing them in real time. Realizes content where characters move interactively while mitigating. This method is also applicable to the multi-view type and the integral photography (IP) method, which is one of the aerial image methods.

JP 2003-67782 A

  In the case where a sprite method is applied as a stereoscopic sprite in which alpha information is added to a stereoscopic image in order to operate a stereoscopic display application that performs an interaction with a user input at a sufficient speed as in Patent Document 1 In order to remove unnecessary areas when superimposing a plurality of sprites, it is common to use a 4-channel (R, G, B, alpha) image format in which transparency information is added to RGB 3-channel image information. Is.

  However, in stereoscopic display devices such as binocular, multi-view, and aerial image systems, when assigning display pixel groups to multi-directional information display, aspects such as the definition and color representation of the displayed stereoscopic video In order to improve the image quality, information in a different direction may be assigned to each sub-pixel in addition to using a display with very high definition (pixel density).

  At this time, if transparency information is added to the stereoscopic image and handled as a 4-channel (R, G, B, alpha) sprite, information such as an unintended boundary line is displayed on the superimposed stereoscopic image. I found a problem that might be done.

  In the present invention, in order to solve the above-described problem, when displaying a stereoscopic image by a sprite method, when assigning information of different light directions for each sub-pixel on the display, alpha information is prepared for each color channel. In addition to the three color channels of RGB, alpha information is also held in three channels, and a data format of a total of six channels is used as a sprite for a stereoscopic image.

  That is, in order to solve the above-described problem, as one preferable embodiment, there is provided a stereoscopic image generation method in an image generation apparatus having a control unit and a storage unit, and the storage unit displays an image three-dimensionally. As a stereoscopic image, luminance information of a plurality of color channels and a plurality of transparency information for each of the plurality of color channels are stored, and the control unit stores according to control information for displaying a stereoscopic image. A stereoscopic image generation method for performing superimposition processing using luminance information and transparency information of a plurality of color channels stored in the unit is provided.

  In another preferred embodiment, the stereoscopic image generating device generates a stereoscopic image, and includes a control unit and a storage unit, and the storage unit is a stereoscopic image for displaying the image stereoscopically. Are stored as luminance information of three color channels of red (R), green (G), and blue (B), and three transparency information corresponding to each color channel, and the control unit displays a stereoscopic image. A stereoscopic image generating apparatus that performs superimposition processing using luminance information and transparency information for each color channel stored in a storage unit according to control information for the purpose is provided.

  Furthermore, as a preferable stereoscopic image display device, a format in which one pixel is formed by three sub-pixels of RGB for displaying the stereoscopic image generation method described above or the stereoscopic image generated by the stereoscopic image generation device. A stereoscopic image display device comprising a flat panel display and a deflection optical element group is provided.

  With the configuration of the present invention, in a stereoscopic display application that interacts with a user input, it is possible to realize sub-pixel unit spatial resolution and improve the quality of stereoscopic video while reducing the load of real-time processing by the sprite method. It becomes possible.

    Prior to the description of embodiments of the present invention with reference to the drawings, an integral photography (IP) type stereoscopic image display apparatus, an IP image generation method, and an IP image sprite are the premise of the present invention. (IP sprite) will be described with reference to the drawings.

  FIG. 1 is a schematic diagram of an integral photography (IP) type stereoscopic image display apparatus which is one of the aerial image types. On the front surface of a flat panel display (FPD) 1, a lens array 2 in which a large number of small convex lenses for controlling the direction of light rays is arranged. A plurality of pixel groups 3 exist under each convex lens, and the light state in the front space of the FPD 1 is controlled by using the light emitted from each pixel to be emitted radially by the deflection of the convex lens. Here, by displaying the stereoscopic image (IP image) output from the stereoscopic image generating device 6 on the FPD 1, the light state when the three-dimensional object 4 exists in the space is configured in a pseudo manner, The user 5 can perceive as if the three-dimensional object 4 actually exists.

  A method of generating an IP image by computer graphics (CG) when the number of light beam directions in the horizontal direction is schematically set to 4 will be described with reference to FIGS. This IP image can also be generated from a photographed image.

  A CG model of the three-dimensional object 4 is arranged in the virtual space 7 and a surface 8 corresponding to the screen of the FPD 1 is defined. Further, virtual viewpoints V1 to V4 are set at positions corresponding to the extended lines of light rays that spread radially through the lens array 2.

  The FPD 1 is a display of a system in which one pixel is composed of three RGB sub-pixels such as P1 to P3. The lens 10 is an enlarged view of one of the lens arrays 2, and the IP image 11 shows a portion displayed on the subpixels P1 to P12 below the lens 10 in the IP image displayed on the FPD1. Is. Light rays emitted from the subpixels P1 to P12 of the FPD 1 spread radially through the lens 10 as light rays L1 to L12.

  The IP image 11 is generated using images 21 to 24 rendered from the virtual viewpoints V1 to V4 with respect to the CG model of the three-dimensional object 4. In FIG. 3, the values of the subpixels P1 to P3 of the IP image 11 copy the luminance values P1 to P3 of RGB of the pixels of the image 21 rendered from the virtual viewpoint V1 corresponding to the extension lines of the light rays L1 to L3. . Similarly, the subpixels P4 to P6 of the IP image 11 are values of P4 to P6 of the image 22 rendered from the virtual viewpoint V2 corresponding to the extension line of the light rays L4 to L6, and the subpixels P7 to LP9 of the IP image 11 are The values of P7 to P9 of the image 23 rendered from the virtual viewpoint V3 corresponding to the extension line of the rays L7 to L9, the subpixels P10 to P12 of the IP image 11 are the virtual viewpoint V4 corresponding to the extension line of the rays L10 to L12. The values are P10 to P12 of the rendered image 24. The same processing is performed on all the lenses of the lens array 2 using the images 21 to 24 to generate one IP image.

  When the IP image 11 generated in this way is displayed on the FPD 1 and the observer 5 moves the viewpoint position in the direction of the light rays L1 to L12, the virtual viewpoints V1 to V1 are sequentially added together with the light emitted from other lenses. Video with parallax rendered in V4 can be perceived. Further, when the right eye and the left eye are at positions where light rays corresponding to two different viewpoints of the virtual viewpoints V1 to V4 are incident, the binocular parallax causes the depth direction as if the image of the three-dimensional object 4 actually exists. A three-dimensional feeling can be perceived.

  In the spatial image type stereoscopic image display device, the higher the density of light rays emitted radially from the FPD 1 through the lens 10, the clearer the stereoscopic image, and the more the number of light ray directions (the number of virtual viewpoints for sampling). As the parallax changes more smoothly, the image quality of the stereoscopic video is improved. In the case of FIG. 3, the number of rays in the horizontal direction is 12, and the number of ray directions is 4.

  Therefore, in FIG. 4, the number of virtual viewpoints is increased to V1 to V12, and only the luminance value of any one of RGB from sub-pixels P1 to P12 is used as the subpixel by using the images 21 to 25 rendered at each virtual viewpoint. The number of ray directions is increased to 12 by extracting and copying in units. In this way, by independently assigning information on different light directions to the sub-pixels, the number of light directions can be increased by a factor of 3, and the quality of the stereoscopic image can be improved.

  FIG. 5 is an IP image generation processing flow. As the number of viewpoints increases as in the example of FIG. 4, the load on the generation process of the images 21 to 25 (step S <b> 102) and the synthesis process on the IP image 11 (step S <b> 104) increases. In the case of interactive content whose video changes in real time, it is difficult to perform steps S100 to S104 for each frame.

  Therefore, the sprite method that has been conventionally used in two-dimensional image animation and games is used for stereoscopic images. The sprite method is a technique for reducing display image generation processing by superimposing and synthesizing a group of images including transparency information (referred to as sprites) prepared in advance at appropriate positions on the background.

  In FIG. 6, images 30 to 32 are sprites composed of objects such as characters (white portions) and unnecessary ground regions (black portions). The image 36 is a scene background. The composite image 34 is generated by superimposing the images 30 to 32 on the background image 36. At this time, in order to show an unnecessary ground area of the images 30 to 32, transparency information is held by using an alpha channel or by determining a color to be subjected to a transparent process, and the area is below. Display the image transparently. Not only the background image 36 and the images 30 to 32 but also sprites can be overlapped. In addition, by preparing a series of motion animations as a plurality of sprites and sequentially superimposing them on the background image 36, it is possible to display not only the movement of the object position but also various motion animations.

  Further, the alpha information possessed by the sprite is not limited to a binary value indicating transparency and non-transmission, but can take an intermediate value indicating translucency. This not only removes unnecessary background areas in the sprite, but also smoothes the edge blending of the object boundary, realizes a translucent texture such as glass and water, You can add effects such as dropping.

  As a first embodiment of the present invention, an embodiment of an image generation method using the sprite method, an apparatus thereof, and a stereoscopic image display apparatus will be described with reference to FIGS.

  In the case of a stereoscopic image such as an IP image, by adding transparency information, image generation processing can be reduced using a sprite method. FIG. 7 is a diagram for explaining the processing of the first embodiment relating to the stereoscopic image generation processing using the sprite method, and is a generation processing flow of the IP image format sprite (IP sprite).

  7 differs from the IP image generation processing of FIG. 5 in that transparency information is generated in step S114 and an IP image for transparency is generated in step S118 and integrated with an RGB IP image in step S120. It is possible to generate a composite IP image including a plurality of objects by superimposing IP sprites generated in advance or IP sprites and IP images having no alpha information. The IP sprite does not have to have an image size corresponding to the entire screen of the FPD 1 and may be an image size obtained by cutting out only an area including an object. Therefore, the IP sprite in the stereoscopic image generating apparatus according to the present embodiment described later is used. It is also possible to save the memory capacity of the storage unit for storing.

  In the case of a sprite that is not a stereoscopic image, the overlapping position can be freely adjusted in units of pixels. However, when IP sprites are overlapped, a lens is selected according to the position of the subpixel in the image. Since the direction in which the light beam is emitted via is uniquely determined, the subpixels corresponding to the same direction must be overlapped, and the position of the overlap is limited.

  In order to operate a stereoscopic display application using the sprite method, IP sprites of all necessary motion animations are generated in advance in steps S110 to S120 in FIG. 7 for all objects included in the content. deep. Next, the application is started, an IP sprite indicating a reaction to the input from the user 5 is called in real time, a composite IP image is generated by superimposing it on an appropriate position on the background, and displayed on the FPD 1 is repeated. Since this superposition processing can be performed at high speed, real-time processing of the application is reduced.

  In this embodiment, the alpha image is generated for each color channel in step S114 for generating an alpha image representing the alpha of the RGB image.

  Generally, the sprite data format is generally an image format such as PNG (Portable Network Graphics) or TGA (Truevision Graphics Adapter) having one alpha channel indicating transparency for the three color channels of RGB. A pixel has four values, R, G, B, and alpha. However, in the case of the IP sprite in this embodiment, the subpixels of the FPD1 are handled as the minimum unit, and the superposition is also performed in units of subpixels, and the same number of alpha channels as the color channels are held. Hereinafter, the reason and the effect will be described with reference to FIGS.

  FIG. 8 shows an example of four pixels when the sprites in the format of 3 color channels and 1 alpha channel are overlapped by the general sprite method described above. In this case, each sprite can be handled in a 4-channel image format. The values of the three color channels are displayed on three adjacent subpixels on the FPD1. The sprite 82 includes a color channel 61 and an alpha channel 62, and the sprite 84 includes a color channel 63 and an alpha channel 66. In the alpha channels 62 and 66, white represents non-transparent and hatched lines represent transparent. In the sprite 82, the areas of the subpixels P1 to P6 are areas to be displayed, and P7 to P12 via the boundary line 80 are unnecessary areas, indicating that the corresponding alpha information A3 and A4 is transparent. The color channel 68 as a result of superimposing the sprites 82 and 84 sandwiches the boundary 80 and the information of the sprites 82 and 84 is superposed. In this way, in a general sprite method, superposition is performed in units of pixels. In the case of FIG. 8, information such as an unintended boundary line is not displayed on the stereoscopic image that has been subjected to the overlay process.

  Next, FIG. 9 illustrates the case of superimposing three color channels and one alpha channel sprite as in FIG. As the data of the color channels 60 and 64, IP images to which different light directions are assigned in units of subpixels are used. At this time, the boundary 80 between the object area to be displayed and the unnecessary area in the IP sprite 86 may exist between P6 (B) and the subpixel P5 (G) of the same pixel. However, in FIG. 9, when the value of the color channel 60 is superimposed on the color channel 64 using the same alpha information A2 for P4 to P6, the resulting P6 of the color channel 68 becomes the color channel 60. P6, and information such as an unintended boundary line is displayed on the stereoscopic video.

  Therefore, in the present embodiment, the problem that occurs peculiarly in the IP sprite as described above is that an alpha image is generated for each color channel in S114 of FIG. 7, and the IP sprites 90 and 92 as shown in FIG. This is solved by having three alpha channels 70 and 72. Thereby, the superposition can be performed in units of subpixels, and P6 of the color channel 64 can be displayed on P6 of the color channel 74 as a result.

  Note that the 6-channel IP sprite format with 3 color channels and 3 alpha channels may handle two 3-channel images as one set, or define and use a 6-channel image format. May be. Moreover, although the data amount increases by using 6 channels, some image compression may be applied because it includes a lot of redundant information. Further, the number of color channels and alpha channels is not limited to three.

  Next, with reference to FIG. 11 and FIG. 12, details of a generation apparatus and processing flow for generating a composite IP image in real time, to which the present embodiment is applied, in a stereoscopic display application that interacts with a user input. explain.

  FIG. 11 is a configuration diagram of the stereoscopic image generating / displaying device of the present embodiment. The stereoscopic image display device 40 is a device including a stereoscopic image generation device 6 and a stereoscopic image display unit 44, and is assumed to be an embedded device with a display, a personal computer (PC), or the like. The stereoscopic image generation device 6 includes a control unit 46 and a storage unit 48 each including a normal central processing unit (CPU), and a synthesized IP image is processed by the control unit in accordance with a signal from the input device 42. Is output to the stereoscopic image display unit 44. The input device 42 is various input devices such as a sensor, a pointing device, a keyboard, and a button, and may be included in the stereoscopic image display device 40 or may be externally connected by some communication means (not shown). good.

  The control unit 46 is a part on which various stereoscopic display applications operate, and corresponds to an input information processing unit 50 that processes a signal from the input device 42 and an appropriate next motion animation depending on the input information and the current situation. An image selection unit 51 that selects an IP sprite, an overlay processing unit 52 that overlays the IP sprite selected by the image selection unit 51 for each color channel of each pixel, and a composite IP image obtained as a result of the overlay Is output to the stereoscopic image display unit 44. These are usually composed of programs and applications executed by the CPU described above.

  In addition, the storage unit 48 of the stereoscopic image generation device 6 includes an object coordinate storage unit 54 that stores the position in the screen where the object included in the content is located, and rules for transition from the input information and the current state to the next state. Animation transition rule storage unit 55, IP sprite storage unit 56 that stores the IP sprites generated in advance in steps S 110 to S 120 of FIG. 7, and a synthesized IP image that stores an IP image synthesized by the overlay process The storage unit 59 is included.

  The IP sprite storage unit 56 includes an RGB image storage unit 57 and an alpha image storage unit 58 that store the values of the 3-channel color channel and the 3-channel alpha channel information as shown in FIG. The memory areas of the RGB image storage unit 57 and the alpha image storage unit 58 may be independent of each other, or may be grouped for each IP sprite. All IP sprites are assigned identifiers (IDs) indicating which frames of which animations of which objects are sprites.

  The stereoscopic image display unit 44 includes a light beam direction control unit 60 and the FPD 1. The light beam direction controller 60 may use various optical elements such as the lens array 2 shown in FIG. 1 or a lenticular lens that realizes parallax only in one direction. The FPD 1 uses a display such as a liquid crystal display, an organic EL display, or a plasma display, in which pixels are composed of three sub-pixels of RGB.

  Next, FIG. 12 is a processing flow of the stereoscopic display application that operates in the stereoscopic image generation apparatus 6 of FIG. 11 in the present embodiment. When the application is started, the image selection unit 51 inquires of the animation transition rule storage unit 55 about the number of objects in the content, the ID of the IP sprite of each object in the initial state, and the coordinates of the display position on the screen (step). S200). The coordinates of the display position include not only the two-dimensional coordinates corresponding to the plane of the FPD 1 but also the coordinates in the depth direction of the screen, which serve as an index for the order of superposition.

  Next, the input information processing unit 50 detects an input from the input device 42 (step S202), and the image selection unit 51 stores the next in the animation transition rule storage unit 55 based on the input information and the current state and coordinates of each object. The ID and coordinates of the IP sprite corresponding to the above operation are inquired (steps S204 and S206). Next, the overlay processing unit 52 reads the background IP image from the IP sprite storage unit 56 into the composite IP image storage unit 59 as an initial value of the composite IP image (step S210). The background IP image is an IP image generated for an object existing on the back side of the screen that does not change in response to a user input, and is displayed on the rearmost surface of the screen, and therefore does not have alpha channel information.

  Next, it is checked whether or not the overlay processing has been performed on all objects existing in the scene (step S212). If there is an unprocessed object, the composite IP image at that time is read from the composite IP image storage unit 59. (Step S216), the IP image (RGB) and IP image (alpha) of the IP sprite of the object which is designated by the IP image selection unit 52 and is the innermost coordinate position among the unprocessed objects are converted into RGB images. The storage unit 57 and the alpha image storage unit 58 are sequentially read (step S218), and, as described above, the overlay processing unit 52 refers to the value of the IP image (alpha) for each color channel to obtain the IP image (RGB). The image is superimposed on the synthesized IP image and stored in the synthesized IP image storage unit 59 (step S220).

  When the superimposition process (S216 to S220) is completed for all objects, the final synthesized IP image is output from the image output unit 53 to the stereoscopic image display unit 44 (step S214). Once displayed, the process returns to step S202 and is repeated, whereby a stereoscopic image that operates interactively in response to an input from the user 5 can be displayed.

  Since the processing time of the superimposition processing (S216 to S220) is very high compared to the case where the processing (steps S100 to S104) for generating the IP image by rendering 3D model data from multiple viewpoints is performed in real time, It is possible to prevent a delay in updating the display contents in response to user input. Also, smooth motion parallax is realized by superimposing in units of sub-pixels, and the image quality of stereoscopic images is improved.

  The present invention described in detail above enables the operation of interactive content using a high-light-density stereoscopic image having sub-pixel level resolution even in a stereoscopic image generating apparatus with low processing performance. As industrial applications, high-quality and interactive stereoscopic images can be displayed for various uses such as entertainment applications such as games, applications on mobile terminals, car navigation systems, touch panels, etc., thereby improving the sense of reality and usability.

It is a schematic diagram of an IP system. It is a figure which shows the CG model and virtual viewpoint in virtual space. It is a figure which shows the production | generation method of an IP image in case it has a light ray direction of a pixel unit. It is a figure which shows the production | generation method of an IP image in case it has a light beam direction of a sub pixel unit. It is a flowchart which shows the production | generation process of IP image. It is a schematic diagram of a sprite system. It is a flowchart which shows the production | generation process of IP sprite. It is a flowchart explaining the production | generation process of a sprite. It is a figure which shows the superimposition of the IP sprite in the case of having transparency of 1 channel. FIG. 3 is a diagram illustrating overlaying of IP sprites having three-channel transparency according to the first embodiment. It is a block diagram which shows the structure of the stereo image production | generation apparatus in Example 1, and a stereoscopic vision image display apparatus. 4 is a flowchart illustrating a superimposing process of stereoscopic images in interactive content according to the first exemplary embodiment.

Explanation of symbols

1. Flat panel display (FPD)
2 ... Lens array 6 ... Stereoscopic image generation device 10 ... Lens 40 ... Stereoscopic image display device 44 ... Stereoscopic image display unit 46 ... Control unit 48 ... Storage unit 56 ... IP sprite storage unit 57 ... RGB information storage unit 58 ... Alpha information storage unit.

Claims (10)

A stereoscopic image generation method in an image generation apparatus having a control unit and a storage unit,
The storage unit stores brightness information of a plurality of color channels and a plurality of transparency information for each of the plurality of color channels as a stereoscopic image for stereoscopically displaying an image,
The control unit performs an overlay process using the luminance information and the transparency information of the color channel stored in the storage unit according to control information for displaying the stereoscopic image.
A stereoscopic image generation method characterized by that. The stereoscopic image generation method according to claim 1,
The plurality of color channels are three channels of red (R), green (G), and blue (B).
A stereoscopic image generation method characterized by that. The stereoscopic image generation method according to claim 1,
The luminance information and the transparency information of the plurality of color channels are integral photography format sprites (IP sprites).
A stereoscopic image generation method characterized by that. The stereoscopic image generation method according to claim 1,
The luminance information and the transparency information of the plurality of color channels correspond to images from a plurality of different viewpoints of the object,
A stereoscopic image generation method characterized by that. A stereoscopic image generation device that generates a stereoscopic image,
A control unit and a storage unit;
The storage unit is a stereoscopic image for displaying an image three-dimensionally, luminance information of three color channels of red (R) green (G) blue (B), and three corresponding to each of the color channels Remembers transparency information,
The control unit performs an overlay process using the luminance information and the transparency information for each of the color channels stored in the storage unit in accordance with control information for displaying the stereoscopic image.
A stereoscopic image generating apparatus characterized by that. The stereoscopic image generating device according to claim 5,
The luminance information and the transparency information of the three color channels are integral photography format sprites (IP sprites).
A stereoscopic image generating apparatus characterized by that. The stereoscopic image generation device according to claim 6,
The controller is
An image selection unit that selects the IP sprite from the storage unit based on input information;
An overlay processor that superimposes the IP sprites selected by the image selector for each color channel to obtain a composite IP image;
An image output unit for outputting the synthesized IP image obtained by the overlay processing unit;
A stereoscopic image generating apparatus characterized by comprising: A stereoscopic image display device for displaying a stereoscopic image,
A control unit and a storage unit;
In order to display an image three-dimensionally, the storage unit holds luminance information of three color channels of red (R), green (G), and blue (B) and transparency information of three channels corresponding to each channel. ,
The controller is
In accordance with control information of an application for displaying a stereoscopic image, a stereoscopic image generation device that performs superimposition processing using the luminance information and transparency information for each color channel and generates the stereoscopic image;
A stereoscopic image display unit for displaying the stereoscopic image generated by the stereoscopic image generation device;
A stereoscopic image display device comprising: The stereoscopic image display device according to claim 8,
In the stereoscopic image generation device, the luminance information and the transparency information of the three color channels are integral photography sprites (IP sprites).
A stereoscopic image display device characterized by that. The stereoscopic image display device according to claim 9,
The stereoscopic image display unit
A flat panel display of a format in which one pixel is composed of three RGB sub-pixels for displaying the generated stereoscopic image;
Consisting of a deflection optical element group,
A stereoscopic image display device characterized by that.

Images