JP2006287592A - Image generating device, electronic equipment, and image generation method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a new stereoscopic view system capable of attaining both a natural stereoscopic effect and a high resolution and to achieve the acceleration of generating a stereoscopic image by the new stereoscopic view system. <P>SOLUTION: A visual line V and a viewpoint for each pixel are set for each pixel PE of a display surface 22, and a locus in the direction of a viewpoint for each pixel is calculated from the the visual line V and the viewpoint for each pixel CM to the pixel PE. Intersections between the locus in the direction of the viewpoint for each pixel and a depth information model are determined, the intersection that is the closest to an observer among intersections between the locus in the direction of the viewpoint for each pixel and the depth information model is acquired as a target intersection, and the color information of a dot corresponding to the target intersection of a plane view image is determined to be the color information of the pixel. In addition, at determining an intersection. a search range for performing intersection determination is limited on the basis of the maximum value and the minimum value of a depth value of the depth information model to thereby reduce a processing load of the intersection determination, then, the acceleration of the intersection determination is achieved. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention generates a stereoscopic image in a three-dimensional virtual space to be displayed on a stereoscopic video display device that includes a pixel panel in which pixels are arranged and an optical element group that provides directivity to each pixel of the pixel panel. The present invention relates to an image generation device and the like.

As conventionally known, optical element groups such as flat panel displays such as LCDs and lens arrays (for example, lenticular lens arrays and eyelet lens arrays) and barrier arrays (for example, parallax barrier arrays and pinhole arrays) By combining these, a stereoscopic video display device can be created. As the method, a multi-view method (including two eyes), a super multi-view method, an IP (Integral Photography) method, a light beam reproduction method, and the like are known, and disclosed in Non-Patent Document 1 and Non-Patent Document 2, for example. ing.
Yasuhiro Takagi, “Display of 3D Objects Combining with 64 Eyes 3D Color Display”, 3D Image Conference 2002 Proceedings, 3D Image Conference 2002 Executive Committee, July 4, 2002, p. 85-88 Onishi Akihiro, Takeda Tsutomu, Taniguchi Hideyuki, Kobayashi Tetsuro, “Three-Dimensional Video Display by Ray Reproduction”, Three-dimensional Image Conference 2001 Proceedings, Three-Dimensional Image Conference 2001 Executive Committee, July 4, 2001, p. 173-176

  In the multi-view method, as shown in FIG. 51, light rays emitted from each pixel on the display surface on which a stereoscopic image is displayed and given directivity by an optical element group (in the figure, a lenticular lens array) It is designed to gather at a plurality of set viewpoint positions (four in the figure). The resolution depends on the pitch of the optical element, and the number of viewpoints (views) depends on the ratio between the pixel pitch and the pitch of the optical element. For this reason, the multi-view method with a small number of viewpoints has a drawback that a natural stereoscopic effect cannot be obtained because the number of viewpoints is small although the resolution at each viewpoint is relatively high.

  Therefore, the super multi-view system shown in FIG. 52 improves the multi-view system so that a natural stereoscopic effect can be obtained by setting very many viewpoints inside the parallax between the left and right eyes. . However, the super multi-view system has a disadvantage that the resolution is remarkably reduced as a result of increasing the number of viewpoints, and it is necessary to use a very high resolution pixel panel in order to obtain a satisfactory resolution. That is, the resolution and the number of viewpoints have a trade-off relationship.

  In the multi-view method and the super-multi-view method, drawing is performed (generating an image) from each of a plurality of assumed viewpoint positions. For this reason, it is necessary to design the optical element so that the pitch of the optical element and the pixel pitch exactly match.

  In the IP (Integral Photography) method and the ray reproduction method, as shown in FIGS. 53 and 54, rays emitted from each pixel and given directivity by the optical element group are collected at the sampled point group of the object. The stereoscopic view is realized by observing this from a further distant viewpoint. FIG. 53 shows the case of the IP system, and FIG. 54 shows the case of the light beam reproduction method.

  The resolution depends on the number of sampling points of the object, and the number of lines of sight at each sampling point depends on the number of rays collected at the sampling point. That is, the smaller the number of sampling points, the more light rays can be collected at each sampling point, that is, a natural three-dimensional effect can be reproduced. However, since the sampling points are small, the resolution is lowered. Further, if the number of sampling points is increased in order to increase the resolution, only a small number of light beams can be collected at each sampling point, and natural stereoscopic vision becomes impossible.

  In particular, in the IP method, as shown in FIG. 53, the position at which a natural sense of distance can be observed is limited to the image plane parallel to the drawing plane (stereoscopic image display plane), and the object at other positions can be observed. A sense of distance is observed unnaturally. On the other hand, the light beam reproduction method can form an image at a free distance as compared with the IP method.

In addition, IP systems include those using a lens array and those using a pinhole array. As shown in FIG. 53, when a lens array is used, there is a gap between the drawing surface and the imaging surface. Depends on the focal length of the lens. That is, as shown in FIG. 55, if the distance between the focal plane of the lens and the display plane is A, the distance between the focal plane of the lens and the imaging plane is B, and the focal distance of the lens is F, As is known, since there is a relationship represented by the following expression, two or more imaging positions (distances from the drawing surface) cannot be set at the same time.
(1 / A) + (1 / B) = (1 / F)

  On the other hand, as shown in FIG. 54, the light beam reproduction method uses a pinhole array instead of a lens array, so that two or more imaging distances can be set simultaneously (two in the figure). Since the pinhole array is used, there is a drawback that the screen is dark and the image is like a sequence of dots.

  Further, in the IP method and the light beam reproducing method, since it is necessary to collect a very large number of light beams in principle, sampling points are generally sparse, that is, those having low resolution. That is, in order to obtain a satisfactory resolution, it is necessary to use a pixel panel with a very high resolution, as in the super multi-view system. That is, although the scale is different from the multi-view method and the super multi-view method, it can be said that the resolution and the number of viewpoints have a trade-off relationship.

  In the IP method and the light beam reproduction method, the position and direction of the line of sight (viewpoint) are determined based on the positional relationship between each imaging position and each optical element or the positional relationship between each imaging position and each pixel. Specifically, when the optical element group is prepared first and the line of sight is determined by the positional relationship between the imaging position and each optical element, the pixel arrangement must be determined according to the optical element group, and When the pixel panel is prepared in advance and the line of sight is determined by the positional relationship between each imaging position and each pixel, the arrangement of the optical element group must be determined in accordance with the pixel panel. In any case, it is necessary to design to match the pitch of the optical elements with the pitch of the pixels.

  In any method, in the conventional stereoscopic video display device, it is necessary to match the optical element pitch of the optical element group and the pixel pitch of the pixel panel, and either pitch of the optical element group or the pixel panel is adjusted to the other. The decision is mainly based on the cost relationship between the two.

  For example, a stereoscopic image printed on a combination of a printed material (such as paper or plastic card) on which a stereoscopic image is printed and an optical element such as a lenticular lens array, which has been known for a long time, is printed. An optical element group (such as a lens array or a barrier array) is attached to the printed surface of the printed material, and the reflected light of each dot of the stereoscopic image printed on the printed material is given directivity by the optical element group. As a result, stereoscopic vision is realized. In the case of such a printed product, it is easy to change the arrangement of the dots of the stereoscopic image to be printed, and therefore the pitch of the dots of the stereoscopic image to be printed is determined according to the optical element group. In addition, when a computer display or the like is used as a pixel panel, a dedicated optical element group (for example, a lenticular lens array) that matches the pixel pitch is designed and manufactured.

  The present invention has been made in view of the above-mentioned problems, and a first object of the present invention is to overcome the trade-off relationship between the number of viewpoints and the video resolution as in the conventional stereoscopic vision system, Providing a new stereoscopic method that can achieve both stereoscopic effect and high resolution, and eliminate the restriction that one of the pixel panel and the optical element group must be designed exclusively for the other. It is. The second purpose is to minimize a reduction in the processing speed of generating a stereoscopic image by a new stereoscopic method, and to use a depth image, so that a general CG tool or a photograph for planar viewing is used. Therefore, it is possible to generate a simple stereoscopic image.

  Means for solving the above problems will be described below, but the stereoscopic viewing method of the present invention is considered to be a development of the conventional multi-view and super-multi-view methods. Compared with the multi-view type, the superiority from them is described.

  Since the amount of information that can be displayed on the pixel panel of the stereoscopic video display device is constant, it has been conventionally assumed that the resolution and the number of viewpoints have a trade-off relationship. That is, in order to achieve both a natural stereoscopic effect and a high resolution, it is necessary to introduce elements other than the resolution and the number of viewpoints. Therefore, in the present invention, an element of image accuracy is introduced.

  A conventional stereoscopic method generates a stereoscopic image in consideration of an image (image) viewed at each viewpoint. That is, first, an image (hereinafter referred to as “individual viewpoint image”) that is a video of each viewpoint is generated, and a stereoscopic image is generated by interleaving the generated individual viewpoint image. Since a stereoscopic image is generated based on an image that is a video viewed at each viewpoint, the video at each viewpoint is naturally an accurate image.

  When the stereoscopic image generated by the present invention is stereoscopically viewed, the video viewed at each viewpoint is slightly inaccurate compared to the conventional video. However, according to the principle described in detail in the following embodiments, the stereoscopic image generated by the present invention has sufficient clarity to be visually recognized as an image.

  On the other hand, it is possible to improve both the number of viewpoints and the video resolution at each viewpoint, which could not be realized by the conventional method according to the present invention, and the increase in processing load accompanying the increase in the number of viewpoints. The point that it was possible to prevent the generation speed of the visual image from being lowered will be described below according to each invention.

In order to solve the above problem, the first invention is:
A pixel panel (for example, the pixel panel 20 in FIG. 2) on which the pixels are arranged, and an optical element group (for example, the lens plate 30 in FIG. 2) that gives directivity to the light emitted from each pixel of the pixel panel. The viewing angle with respect to the one optical element at the assumed observation position is the viewing angle λ with respect to the optical element, the viewing angle with respect to one pixel to which directivity is given by the one optical element is the viewing angle σ with respect to the pixel, and the stereoscopic image drawing area in the pixel panel When the viewing angle with respect to (for example, the display surface 22 in FIG. 2) is a pair-drawing region viewing angle Φ, stereoscopic viewing in which λ: σ = n: m (where n is a natural number and m is a natural number less than Φ / λ) is not established. A stereoscopic image in a three-dimensional virtual space displayed on a video display device (for example, the stereoscopic video display device 200A in FIG. 2 or the stereoscopic video display device 200 in FIG. 26) is displayed from the given viewpoint in the three-dimensional virtual space. It is a planar view image A quasi-image (for example, a planar view image of FIG. 29) and depth value information of each dot of the reference image with the viewpoint position as a reference (for example, a depth image of FIG. 30 and a depth information model of FIG. 31B) An image generation device (for example, the stereoscopic image generation device 1 in FIG. 26) generated based on
Regarding the color information of each pixel in the stereoscopic image drawing region, 1) the direction of the light beam passing through the representative point of the pixel and the representative point of the optical element that gives directivity to the emitted light beam of the pixel (for example, in the embodiment) Of the representative ray PR) and a direction corresponding to the direction of the ray in the three-dimensional virtual space based on the viewpoint position in the three-dimensional virtual space (for example, the pixel-specific viewpoint CM in the embodiment). Calculation step (for example, steps S15 and S17 in FIG. 32) and 2) Based on the direction obtained in the direction calculation step and the depth value information, the closest object in the direction in the three-dimensional virtual space A dot determination step (for example, steps S35 to S61 in FIG. 33) for determining dots representing color information from the reference image is executed, and before the dot determination step is performed. The color information of the reference image dots by the color information of the pixel, an image generating apparatus for generating a stereoscopic image.

The eleventh invention is
A pixel panel (for example, the pixel panel 20 in FIG. 2) on which the pixels are arranged, and an optical element group (for example, the lens plate 30 in FIG. 2) that gives directivity to the light emitted from each pixel of the pixel panel. The viewing angle with respect to the one optical element at the assumed observation position is the viewing angle λ with respect to the optical element, the viewing angle with respect to one pixel to which directivity is given by the one optical element is the viewing angle σ with respect to the pixel, and the stereoscopic image drawing area in the pixel panel When the viewing angle with respect to the display area viewing angle Φ (for example, the display surface 22 in FIG. 2) is Φ: σ = n: m (where n is a natural number and m is a natural number less than Φ / λ) A stereoscopic image in a three-dimensional virtual space displayed on a video display device (for example, the stereoscopic video display device 200A in FIG. 2 or the stereoscopic video display device 200 in FIG. 26) is displayed from the given viewpoint in the three-dimensional virtual space. It is a planar view image A quasi-image (for example, a planar view image of FIG. 29) and depth value information of each dot of the reference image with the viewpoint position as a reference (for example, a depth image of FIG. 30 and a depth information model of FIG. 31B) An image generation method generated based on
Regarding the color information of each pixel in the stereoscopic image drawing region, 1) the direction of the light beam passing through the representative point of the pixel and the representative point of the optical element that gives directivity to the emitted light beam of the pixel (for example, in the embodiment) Of the representative ray PR) and a direction corresponding to the direction of the ray in the three-dimensional virtual space based on the viewpoint position in the three-dimensional virtual space (for example, the pixel-specific viewpoint CM in the embodiment). Calculation step (for example, steps S15 and S17 in FIG. 32) and 2) Based on the direction obtained in the direction calculation step and the depth value information, the closest object in the direction in the three-dimensional virtual space A dot determination step (for example, steps S35 to S61 in FIG. 33) for determining dots representing color information from the reference image, and the dot determination step determines the dot The color information of the dots of the quasi image by the color information of the pixel is an image generating method for generating a stereoscopic image.

The sixteenth invention is
A pixel panel (for example, the pixel panel 20 in FIG. 2) on which the pixels are arranged, and an optical element group (for example, the lens plate 30 in FIG. 2) that gives directivity to the light emitted from each pixel of the pixel panel. The viewing angle with respect to the one optical element at the assumed observation position is the viewing angle λ with respect to the optical element, the viewing angle with respect to one pixel to which directivity is given by the one optical element is the viewing angle σ with respect to the pixel, and the stereoscopic image drawing area in the pixel panel When the viewing angle with respect to (for example, the display surface 22 in FIG. 2) is a pair-drawing region viewing angle Φ, stereoscopic viewing in which λ: σ = n: m (where n is a natural number and m is a natural number less than Φ / λ) is not established. A stereoscopic image in a three-dimensional virtual space displayed on a video display device (for example, the stereoscopic video display device 200A in FIG. 2 or the stereoscopic video display device 200 in FIG. 26) is displayed from the given viewpoint in the three-dimensional virtual space. It is a planar view image A quasi-image (for example, a planar view image of FIG. 29) and depth value information of each dot of the reference image with the viewpoint position as a reference (for example, a depth image of FIG. 30 and a depth information model of FIG. 31B) A program (for example, the stereoscopic image generating apparatus 1 in FIG. 26) to generate based on
Regarding the color information of each pixel in the stereoscopic image drawing region, 1) the direction of the light beam passing through the representative point of the pixel and the representative point of the optical element that gives directivity to the emitted light beam of the pixel (for example, in the embodiment) Of the representative ray PR) and a direction corresponding to the direction of the ray in the three-dimensional virtual space based on the viewpoint position in the three-dimensional virtual space (for example, the pixel-specific viewpoint CM in the embodiment). Calculation step (for example, steps S15 and S17 in FIG. 32) and 2) Based on the direction obtained in the direction calculation step and the depth value information, the closest object in the direction in the three-dimensional virtual space A dot determination step (for example, steps S35 to S61 in FIG. 33) for determining dots representing color information from the reference image is executed, and the dot determination step is performed. The color information of the dot of the serial reference image by the color information of the pixel, as a means for generating a stereoscopic image, a program for causing a computer to function.

  According to the first, eleventh, or sixteenth invention, a stereoscopic image to be displayed on a so-called “pitch mismatched” stereoscopic video display device is generated as follows. That is, the color information of each pixel in the stereoscopic image drawing area in the pixel panel of the stereoscopic video display device passes through the representative point of the pixel and the representative point of the optical element that gives directivity to the emitted light of the pixel. The three-dimensional virtual space is generated by rendering processing based on the direction of the ray to be generated.

  Therefore, when the generated stereoscopic image is displayed on the stereoscopic video display device, each pixel of the pixel panel represents the color information of the three-dimensional virtual space based on the direction of the light beam of the pixel. Therefore, it can be said that there are as many viewpoints as the number of light rays, that is, as many viewpoints as there are pixels. Therefore, the viewer's eyes must be positioned at the assumed viewpoint position as in the conventional multi-view method, and the observer is positioned within a certain area as in the super multi-view method. Even if there is an eye at any position, it is possible to achieve a good stereoscopic view.

  On the other hand, the light beam of each pixel of the pixel panel is in a separate direction for each pixel. However, when the direction of the light ray is traced, the light ray passing through the observer's left eye and the vicinity of the position of the left eye is in a substantially uniform direction. Therefore, the image visually recognized by the observer is visually accurate with certain clarity although it is not accurate color information viewed from the position of the eyes. Since the resolution of the image at the position of the pixel surface is about the width of one optical element, it does not lead to a significant resolution degradation as in the super multi-view type.

  Further, the stereoscopic image is generated based on a reference image that is a planar image obtained by viewing the three-dimensional virtual space from a given viewpoint, and depth value information of each dot of the reference image with the viewpoint position as a reference. be able to. Therefore, for example, an image created with an existing high-quality CG tool can be easily three-dimensionalized without using a dedicated renderer, and a highly versatile image generation apparatus can be provided. In addition, when creating a stereoscopic image from a real image, it is possible to shoot from one viewpoint position, so that a stereoscopic image can be easily generated.

  As described above, when a stereoscopic image is generated from the planar image from one viewpoint and the depth value information corresponding thereto, there is no information from the viewpoint that wraps around to the left and right. In such a case, the image may become unnatural. However, when the assumed observation range is not so wide, that is, when the left and right binocular parallax is only inside or near the outside, the influence is small and a stereoscopic image having no practical problem can be obtained.

The second invention is
A pixel panel (for example, the pixel panel 20 in FIG. 2) on which the pixels are arranged, and an optical element group (for example, the lens plate 30 in FIG. 2) that gives directivity to the light emitted from each pixel of the pixel panel. The viewing angle with respect to the one optical element at the assumed observation position is the viewing angle λ with respect to the optical element, the viewing angle with respect to one pixel to which directivity is given by the one optical element is the viewing angle σ with respect to the pixel, and the stereoscopic image drawing area in the pixel panel When the viewing angle with respect to the display area viewing angle Φ (for example, the display surface 22 in FIG. 2) is Φ: σ = n: m (where n is a natural number and m is a natural number less than Φ / λ) A stereoscopic image in a three-dimensional virtual space displayed on a video display device (for example, the stereoscopic video display device 200A in FIG. 2 or the stereoscopic video display device 200 in FIG. 26) is displayed from the given viewpoint in the three-dimensional virtual space. It is a planar view image A quasi-image (for example, a planar view image of FIG. 29) and depth value information of each dot of the reference image with the viewpoint position as a reference (for example, a depth image of FIG. 30 and a depth information model of FIG. 31B) An image generation device (for example, the stereoscopic image generation device 1 in FIG. 26) generated based on
Regarding the color information of each pixel in the stereoscopic image drawing region, 1) the direction of the light beam passing through the representative point of the pixel and the representative point of the optical element that gives directivity to the emitted light beam of the pixel (for example, in the embodiment) Of the representative ray PR) and a direction corresponding to the direction of the ray in the three-dimensional virtual space based on the viewpoint position in the three-dimensional virtual space (for example, the pixel-specific viewpoint CM in the embodiment). Calculation step (for example, steps S15 and S17 in FIG. 32) and 2) The reference for drawing the locus when the locus of the direction obtained in the direction calculation step is projected and drawn on the reference image. A dot group detection step (for example, step S33 in FIG. 33) for detecting a dot group between a dot having a maximum depth value and a dot having a minimum depth value among the dots in the image based on the depth value information; 3) Based on the direction obtained in the direction calculation step and the depth value of each dot of the dot group detected in the dot group detection step, the most in the direction in the three-dimensional virtual space from the dot group A dot determination step (for example, steps S35 to S61 in FIG. 33) for determining a dot representing the color information of the near object, and the dot color information determined in the dot determination step is used as the color of the pixel. It is an image generation device that generates a stereoscopic image by using information.

The twelfth invention
A pixel panel (for example, the pixel panel 20 in FIG. 2) on which the pixels are arranged, and an optical element group (for example, the lens plate 30 in FIG. 2) that gives directivity to the light emitted from each pixel of the pixel panel. The viewing angle with respect to the one optical element at the assumed observation position is the viewing angle λ with respect to the optical element, the viewing angle with respect to one pixel to which directivity is given by the one optical element is the viewing angle σ with respect to the pixel, and the stereoscopic image drawing area in the pixel panel When the viewing angle with respect to (for example, the display surface 22 in FIG. 2) is a pair-drawing region viewing angle Φ, stereoscopic viewing in which λ: σ = n: m (where n is a natural number and m is a natural number less than Φ / λ) is not established. A stereoscopic image in a three-dimensional virtual space displayed on a video display device (for example, the stereoscopic video display device 200A in FIG. 2 or the stereoscopic video display device 200 in FIG. 26) is displayed from the given viewpoint in the three-dimensional virtual space. It is a planar view image A quasi-image (for example, a planar view image of FIG. 29) and depth value information of each dot of the reference image with the viewpoint position as a reference (for example, a depth image of FIG. 30 and a depth information model of FIG. 31B) An image generation method generated based on
Regarding the color information of each pixel in the stereoscopic image drawing region, 1) the direction of the light beam passing through the representative point of the pixel and the representative point of the optical element that gives directivity to the emitted light beam of the pixel (for example, in the embodiment) Of the representative ray PR) and a direction corresponding to the direction of the ray in the three-dimensional virtual space based on the viewpoint position in the three-dimensional virtual space (for example, the pixel-specific viewpoint CM in the embodiment). Calculation step (for example, steps S15 and S17 in FIG. 32) and 2) When the locus of the direction obtained in the direction calculation step is projected on the reference image and drawn, the reference is drawn. A dot group detection step (for example, step S33 in FIG. 33) for detecting a dot group between a dot having a maximum depth value and a dot having a minimum depth value among the dots in the image based on the depth value information; 3) Based on the direction obtained in the direction calculation step and the depth value of each dot of the dot group detected in the dot group detection step, the most in the direction in the three-dimensional virtual space from the dot group A dot determination step (for example, steps S35 to S61 in FIG. 33) for determining a dot representing the color information of the near object, and the dot color information determined in the dot determination step is used as the color information of the pixel. This is an image generation method for generating a stereoscopic image.

The seventeenth invention
A pixel panel (for example, the pixel panel 20 in FIG. 2) on which the pixels are arranged, and an optical element group (for example, the lens plate 30 in FIG. 2) that gives directivity to the light emitted from each pixel of the pixel panel. The viewing angle with respect to the one optical element at the assumed observation position is the viewing angle λ with respect to the optical element, the viewing angle with respect to one pixel to which directivity is given by the one optical element is the viewing angle σ with respect to the pixel, and the stereoscopic image drawing area in the pixel panel When the viewing angle with respect to (for example, the display surface 22 in FIG. 2) is a pair-drawing region viewing angle Φ, stereoscopic viewing in which λ: σ = n: m (where n is a natural number and m is a natural number less than Φ / λ) is not established. A stereoscopic image in a three-dimensional virtual space displayed on a video display device (for example, the stereoscopic video display device 200A in FIG. 2 or the stereoscopic video display device 200 in FIG. 26) is displayed from the given viewpoint in the three-dimensional virtual space. It is a planar view image A quasi-image (for example, a planar view image of FIG. 29) and depth value information of each dot of the reference image with the viewpoint position as a reference (for example, a depth image of FIG. 30 and a depth information model of FIG. 31B) A program for causing a computer (for example, the stereoscopic image generating apparatus 1 in FIG. 26) to generate a program based on
Regarding the color information of each pixel in the stereoscopic image drawing region, 1) the direction of the light beam passing through the representative point of the pixel and the representative point of the optical element that gives directivity to the emitted light beam of the pixel (for example, in the embodiment) Of the representative ray PR) and a direction corresponding to the direction of the ray in the three-dimensional virtual space based on the viewpoint position in the three-dimensional virtual space (for example, the pixel-specific viewpoint CM in the embodiment). Calculation step (for example, steps S15 and S17 in FIG. 32) and 2) When the locus of the direction obtained in the direction calculation step is projected on the reference image and drawn, the reference is drawn. A dot group detection step (for example, step S33 in FIG. 33) for detecting a dot group between a dot having a maximum depth value and a dot having a minimum depth value among the dots in the image based on the depth value information; 3) Based on the direction obtained in the direction calculation step and the depth value of each dot of the dot group detected in the dot group detection step, the most in the direction in the three-dimensional virtual space from the dot group A dot determination step (for example, steps S35 to S61 in FIG. 33) for determining a dot representing the color information of the near object, and the dot color information determined in the dot determination step is used as the color of the pixel. By using the information, it is a program for causing a computer to function as means for generating a stereoscopic image.

  According to the second, twelfth, or seventeenth invention, unlike the conventional multi-view system, the observer's eyes have to be positioned at the assumed viewpoint position, as in the super multi-view system. In addition, as long as the observer is located within a certain region, stereoscopic viewing can be satisfactorily performed regardless of where the eyes are. In addition, although the image visually recognized by the observer is not accurate color information viewed from the position of the eyes, the image is visually recognized with a certain degree of clarity. Therefore, the resolution of the image at the position of the pixel surface is the optical element 1. However, the resolution is not significantly reduced as in the super multi-view type. Furthermore, such a stereoscopic image can be generated without reducing its generation speed.

  That is, since each pixel in the stereoscopic image drawing area represents the color information of the three-dimensional virtual space based on the direction of the light ray of the pixel, there are as many viewpoints as the number of light rays. It can be said that there are as many perspectives. Therefore, since the number of viewpoints increases as the number of viewpoints increases, there is a concern that the generation speed of the stereoscopic image is reduced as compared with the generation speed of the conventional stereoscopic image. Therefore, from the dots in the reference image, a dot group between the dot with the largest depth value and the dot with the smallest depth value is detected based on the depth value information, and from the detected dot group, in the three-dimensional virtual space The dot representing the color information of the foremost object in the above direction is determined. Thereby, since it is only necessary to make a determination regarding the dot group between the dot having the largest depth value and the dot having the smallest depth value, the calculation step can be omitted and a stereoscopic image can be generated at high speed.

The third invention is
A stereoscopic panel including a pixel panel (for example, the pixel panel 20 in FIG. 2) on which pixels are arranged, and an optical element group (for example, the lens plate 30 in FIG. 2) that gives directivity to the light emitted from each pixel of the pixel panel. A stereoscopic image in a three-dimensional virtual space to be displayed on a visual image display device (for example, the stereoscopic image display device 200A in FIG. 2 or the stereoscopic image display device 200 in FIG. 26) is displayed on the three-dimensional virtual space in a given viewpoint. A reference image (for example, the planar view image of FIG. 29) that is a planar view viewed from the viewpoint, and depth value information (for example, the depth image of FIG. 30, FIG. 31) of each dot of the reference image relative to the viewpoint position. (B) depth information model) and an image generation device (for example, the stereoscopic image generation device 1 in FIG. 26),
Regarding the color information of each pixel in the stereoscopic image drawing region, 1) the direction of the light beam passing through the optical element that gives directivity to the pixel and the emitted light beam of the pixel (for example, the representative light beam PR in the embodiment) and And a direction calculating step for obtaining a direction corresponding to the direction of the light beam in the three-dimensional virtual space based on the viewpoint position in the three-dimensional virtual space (for example, the pixel-specific viewpoint CM in the embodiment) (for example, FIG. 32 of steps S15, S17) and 2) among the dots in the reference image that will draw the locus when the locus in the direction obtained in the direction calculating step is projected onto the reference image and drawn. A dot group detection step (for example, step S33 in FIG. 33) for detecting a dot group between a dot having a maximum depth value and a dot having a minimum depth value based on the depth value information; and 3) calculating the direction. Based on the direction obtained in the step and the depth value of each dot of the dot group detected in the dot group detection step, the color of the nearest object in the direction in the three-dimensional virtual space from the dot group A dot determination step (for example, steps S35 to S61 in FIG. 33) for determining a dot representing information is executed, and the color information of the dot determined in the dot determination step is used as the color information of the pixel. An image generation apparatus that generates a stereoscopic image.

The thirteenth invention is
A stereoscopic panel including a pixel panel (for example, the pixel panel 20 in FIG. 2) on which pixels are arranged, and an optical element group (for example, the lens plate 30 in FIG. 2) that gives directivity to the light emitted from each pixel of the pixel panel. A stereoscopic image in a three-dimensional virtual space to be displayed on a visual image display device (for example, the stereoscopic image display device 200A in FIG. 2 or the stereoscopic image display device 200 in FIG. 26) is displayed on the three-dimensional virtual space in a given viewpoint. A reference image (for example, the planar view image of FIG. 29) that is a planar view viewed from the viewpoint, and depth value information (for example, the depth image of FIG. 30, FIG. 31) of each dot of the reference image relative to the viewpoint position. (B) a depth information model) and an image generation method that generates the
Regarding the color information of each pixel in the stereoscopic image drawing region, 1) the direction of the light beam passing through the optical element that gives directivity to the pixel and the emitted light beam of the pixel (for example, the representative light beam PR in the embodiment) and And a direction calculating step for obtaining a direction corresponding to the direction of the light beam in the three-dimensional virtual space based on the viewpoint position in the three-dimensional virtual space (for example, the pixel-specific viewpoint CM in the embodiment) (for example, FIG. 32 of steps S15, S17) and 2) among the dots in the reference image that will draw the locus when the locus in the direction obtained in the direction calculating step is projected onto the reference image and drawn. A dot group detection step (for example, step S33 in FIG. 33) for detecting a dot group between a dot having a maximum depth value and a dot having a minimum depth value based on the depth value information; and 3) calculating the direction. Based on the direction obtained in the step and the depth value of each dot of the dot group detected in the dot group detection step, the color of the nearest object in the direction in the three-dimensional virtual space from the dot group A dot determination step (for example, steps S35 to S61 in FIG. 33) for determining a dot representing information, and the color information of the dot determined in the dot determination step is used as the color information of the pixel. This is an image generation method for generating a stereoscopic image.

The eighteenth invention
A stereoscopic panel including a pixel panel (for example, the pixel panel 20 in FIG. 2) on which pixels are arranged, and an optical element group (for example, the lens plate 30 in FIG. 2) that gives directivity to the light emitted from each pixel of the pixel panel. A stereoscopic image in a three-dimensional virtual space to be displayed on a visual image display device (for example, the stereoscopic image display device 200A in FIG. 2 or the stereoscopic image display device 200 in FIG. 26) is displayed on the three-dimensional virtual space in a given viewpoint. A reference image (for example, the planar view image of FIG. 29) that is a planar view viewed from the viewpoint, and depth value information (for example, the depth image of FIG. 30, FIG. 31) of each dot of the reference image relative to the viewpoint position. (B) a depth information model) and a program for causing a computer (for example, the stereoscopic image generation apparatus 1 in FIG. 26) to generate the program,
Regarding the color information of each pixel in the stereoscopic image drawing region, 1) the direction of the light beam passing through the optical element that gives directivity to the pixel and the emitted light beam of the pixel (for example, the representative light beam PR in the embodiment) and And a direction calculating step for obtaining a direction corresponding to the direction of the light beam in the three-dimensional virtual space based on the viewpoint position in the three-dimensional virtual space (for example, the pixel-specific viewpoint CM in the embodiment) (for example, FIG. 32 of steps S15, S17) and 2) among the dots in the reference image that will draw the locus when the locus in the direction obtained in the direction calculating step is projected onto the reference image and drawn. A dot group detection step (for example, step S33 in FIG. 33) for detecting a dot group between a dot having a maximum depth value and a dot having a minimum depth value based on the depth value information; and 3) calculating the direction. Based on the direction obtained in the step and the depth value of each dot of the dot group detected in the dot group detection step, the color of the nearest object in the direction in the three-dimensional virtual space from the dot group By executing a dot determination step (for example, steps S35 to S61 in FIG. 33) for determining dots representing information, the color information of the dots determined in the dot determination step is used as the color information of the pixel. A program for causing a computer to function as means for generating a stereoscopic image.

  According to the third, thirteenth, or eighteenth invention, a reference image that is a planar image when the three-dimensional virtual space is viewed from a given viewpoint, and a depth value of each dot of the reference image with the viewpoint position as a reference When generating a stereoscopic image based on the information, the dot group detection step detects the dot group between the dot with the largest depth value and the dot with the smallest depth value based on the depth value information, and the detected dot group The dot determination step is determined from among the above. Thereby, the calculation process in the dot determination process can be omitted, and a stereoscopic image can be generated at high speed.

A fourth invention is the image generating device of the third invention,
The dot determination step is based on a) the depth value of each dot of the dot group detected in the dot group detection step, and b) the viewpoint position when the direction obtained in the direction calculation step is a trajectory. The depth value of the position corresponding to the dot on the locus is compared with the depth value of b) in order from the dot on the near side, and based on whether the context of the two depth values to be compared is switched. The image generating apparatus is a step of determining a dot representing color information of the closest object in the direction in the three-dimensional virtual space.

A fourteenth aspect of the invention is an image generation method of the thirteenth aspect of the invention,
The dot determination step is based on a) the depth value of each dot of the dot group detected in the dot group detection step, and b) the viewpoint position when the direction obtained in the direction calculation step is a trajectory. The depth value of the position corresponding to the dot on the locus is compared with the depth value of b) in order from the dot on the near side, and based on whether the context of the two depth values to be compared is switched. The image generating method is a step of determining a dot representing the color information of the closest object in the direction in the three-dimensional virtual space.

The nineteenth invention is the program of the eighteenth invention,
The dot determination step is based on a) the depth value of each dot of the dot group detected in the dot group detection step, and b) the viewpoint position when the direction obtained in the direction calculation step is a trajectory. The depth value of the position corresponding to the dot on the locus is compared with the depth value of b) in order from the dot on the near side, and based on whether the context of the two depth values to be compared is switched. A program for determining a dot representing color information of the closest object in the direction in the three-dimensional virtual space.

  According to the fourth, fourteenth, or nineteenth invention, a) the depth value of each dot of the dot group detected in the dot group detection step, and b) the direction obtained in the direction calculation step as a locus. Whether the depth value of b) is compared with the depth value of the position on the locus on the basis of the viewpoint position in order from the dot on the near side, and the context of the two depth values to be compared is switched. Based on whether or not, the dot representing the color information of the closest object in the direction in the three-dimensional virtual space is determined. That is, since the dot representing the color information of the closest object is determined in order from the dot with the depth value of the dot group in order, the target dot determination can be reached quickly. In addition, because the dot representing the color information of the nearest object is determined based on the change in the context of the depth value to be compared, the dot can be determined with a simple calculation and the calculation process can be performed at high speed. Can be

The fifth invention is the image generating device of the fourth invention,
In the dot determination step, when the contexts of the two depth values to be compared are not interchanged, the depth value of b) is a dot adjacent to the opposite side of the dot group of the end dot on the farthest side, The image generating apparatus is a step of determining as a dot representing color information of the closest object in the direction in the three-dimensional virtual space.

A fifteenth aspect of the invention is the image generation method of the fourteenth aspect of the invention,
In the dot determination step, when the contexts of the two depth values to be compared are not interchanged, the depth value of b) is a dot adjacent to the opposite side of the dot group of the end dot on the farthest side, The image generation method is a step of determining as a dot representing the color information of the foremost object in the direction in the three-dimensional virtual space.

The twentieth invention is the program of the nineteenth invention,
In the dot determination step, when the contexts of the two depth values to be compared are not interchanged, the depth value of b) is a dot adjacent to the opposite side of the dot group of the end dot on the farthest side, The program is a step of determining as a dot representing the color information of the closest object in the direction in the three-dimensional virtual space.

  According to the fifth, fifteenth, or twentieth invention, when the order of the two depth values to be compared is not interchanged, the depth group in b) is the dot group of the dot at the end on the farthest side. A dot adjacent to the opposite side is determined as a dot representing the color information of the closest object in the direction in the three-dimensional virtual space. That is, if the context of the two depth values to be compared is not interchanged, the depth value in b) is the distance between the dot at the end on the farthest side and the dot adjacent to this dot on the side opposite to the dot group. Thus, since the order of the two depth values to be compared changes, the dot adjacent to this dot on the side opposite to the dot group can be determined as the dot representing the color information of the closest object. Thus, even when the dot group detected in the dot group detection step does not include the dot representing the color information of the foremost object, the color information of the foremost object is easily represented. It is possible to quickly determine a stereoscopic image by determining the dot that is present.

  As a sixth aspect of the invention, the optical element group of the image generation apparatus according to any one of the first to fifth aspects may be a lenticular lens array, an eyelet lens array, a parallax barrier array, or a pinhole array. .

As a seventh invention, the optical element group of the image generation apparatus according to any one of the first to fifth inventions is a lenticular lens array in which a plurality of lenses are connected as an optical element, or an eyelet lens array,
The optical element group is installed between the pixel panel and an observer so that the distance between the display surface of the pixel panel and the principal point plane of the optical element group matches the focal length of the lens. It is also good.

As an eighth invention, the optical element group of the image generation apparatus according to any one of the first to fifth is a lenticular lens array in which a plurality of lenses are connected as an optical element or an eyelet lens array,
In this optical element group, the distance between the display surface of the pixel panel and the principal point plane of the optical element group is D, the focal length of the lens is F, the pitch of the pixels in the horizontal scanning direction is S, and the lens Is set between the pixel panel and the viewer so that (LS) .multidot.F / L.ltoreq.D.ltoreq. (L + S) .multidot.F / L. It is also good.

  As a ninth invention, the optical element group of the image generating apparatus according to any one of the sixth to eighth inventions is arranged so that an optical element arrangement direction is oblique with respect to a pixel arrangement direction of the pixel panel. It's also good.

The tenth invention is
A pixel panel in which pixels are arranged;
An optical element group that gives directivity to the emitted light of each pixel of the pixel panel;
An image generating device according to any of the first to ninth inventions for generating a stereoscopic image of a three-dimensional virtual space to be displayed on the pixel panel;
Is an electronic device.

  According to the tenth invention, an electronic apparatus having the same function and effect as any one of the first to ninth inventions can be realized.

  By degrading the accuracy of the image to such an extent that it does not hinder stereoscopic vision, the resolution of the image can be maintained at the width of one optical element, although there is a viewpoint corresponding to the total number of pixels of the pixel panel. A new stereoscopic method can be realized, and the restriction that one of the pixel panel and the optical element group must be designed exclusively for the other can be eliminated. In addition, the processing speed of generating a stereoscopic image by a new stereoscopic method can be increased.

  Preferred embodiments of the present invention will be described below with reference to the drawings. In each drawing, hatching is not drawn in order to clearly show the direction of light rays. In the following, a case where a stereoscopic image to be displayed on a stereoscopic video display device using a lenticular lens array as an optical element group is generated will be described, but the application of the present invention is not limited to this.

[Generation principle of stereoscopic images]
Stereo image generation methods include (I) a method of creating an accurate 3D model at the same time as rendering, (II) a method of creating a stereoscopic image from a planar image and a depth image from one viewpoint, III) There is a method of creating an image for two eyes or multiple eyes and interpolating an intermediate viewpoint from them. In the present embodiment, a case where a stereoscopic image is generated based on the methods (I) and (II) will be described below as an example.

First, a case where a stereoscopic image is generated based on the method (I) will be described.
FIG. 1 is a diagram showing an outline of stereoscopic image generation in this embodiment, and shows a vertical sectional view with respect to the display surface 22. As shown in the figure, in this embodiment, for each pixel PE on the display surface 22, (1) a representative point of the pixel PE (for example, the center of the pixel PE) and a lens (optical element) corresponding to the pixel PE (2) The color information of the object in the line-of-sight direction of the determined line-of-sight V is used as the color information of the pixel PE ( Rendering) to generate a stereoscopic image.

(1) Determination of the line of sight V The line of sight V is a configuration parameter of a stereoscopic video display device that displays a stereoscopic image (as will be described later, the relative arrangement relationship between the pixel panel and the lens plate, The pixel pitch, the lens pitch of the lens plate, the focal length, and the like) and an assumed observer position (hereinafter referred to as “assumed observation position”). Specifically, for each pixel PE, a lens (optical element) corresponding to the pixel PE is determined based on the configuration parameter of the stereoscopic video display device and the assumed observation position, and the representative point of the pixel PE is determined. A ray (representative ray) after passing through the principal point of the lens corresponding to the pixel PE is calculated. Then, the line of sight having the same position as that of the representative light ray PR and having the direction reversed is determined as the line of sight V of the pixel. The assumed observation position is the position of the observer's viewpoint relative to the display surface of the stereoscopic video display device.

  Here, the stereoscopic video display apparatus handled in this embodiment will be described. In the present embodiment, a stereoscopic image to be displayed on a lenticular stereoscopic image display device is generated. A lenticular stereoscopic image display device is a stereoscopic image display device that uses a lenticular lens array as an optical element group, and the lenticular lens array is mounted at a certain distance from the display surface of a flat panel display such as a flat liquid crystal display. The display device allows the observer to recognize stereoscopic vision by viewing (observing) the image displayed on the display surface via the lenticular lens array.

  FIG. 2 is a diagram showing a schematic structure of a vertical lenticular stereoscopic image display apparatus 200A. FIG. 4A shows a cross-sectional view in the horizontal direction (horizontal scanning direction) with respect to the display surface of the stereoscopic video display apparatus 200A, and FIG. 4B shows a plan view seen from the observer side.

  According to the figure, the stereoscopic video display apparatus 200A is mainly configured to include a backlight 10, a pixel panel 20, and a lens plate 30. The backlight 10, the pixel panel 20, and the lens plate 30 are plate-like bodies and are arranged in parallel to each other.

  The backlight 10 emits light, which passes through the pixel panel 20 and the lens plate 30 and travels out of the stereoscopic video display device 200A. That is, the observer sees an image displayed on the pixel panel 20 through the lens plate 30.

  The pixel panel 20 is a display capable of color display in which pixels (pixels) are arranged in a fixed arrangement, and any type can be used as long as it can be stereoscopically combined with the lens plate 30. For example, there are a color filter type liquid crystal display, a plasma display, an inorganic EL display, an organic EL display, and the like. Even if no color filter is used, elements that emit light in a single color such as red (R), green (G), and blue (B) are arranged like an organic EL display or LED display in which single-color light emitting elements are arranged. Even a display that has been used is applicable. Further, it may be a monochrome display in which pixels emitting light of the same color are arranged, and further, a display having pixels of colors other than R (red), G (green), and B (blue). It doesn't matter. Further, the pixel arrangement may be a delta arrangement or other arrangements as long as the coordinates of the representative points of the pixels can be obtained as well as the lattice shape.

  One surface of the lens plate 30 is a concavo-convex surface formed by connecting a microlens (hereinafter simply referred to as a “lens”) 32 that is a semi-cylindrical cross section (saddle-shaped) or an optical element optically equivalent thereto. There is a lenticular lens array whose other surface is substantially planar. Each lens 32 of the lens plate 30 functions to give directivity to light rays (emitted light rays) emitted from the respective pixels PE of the display surface 22.

The lens plate 30 has a flat surface facing the display surface 22 of the pixel panel 20, and the distance G between the principal point surface of the lens plate 30 and the display surface 22 substantially matches the focal length F of each lens 32. Are arranged to be. Note that the distance G does not completely coincide with the focal length F and may have some error. For example, FIG. 49A shows a state in which G = F, and when viewed from a specific direction, one pixel PE extends over the lens 32 and is observed. Further, even when the distance G is slightly away from the focal distance F, that is, in the state shown in FIGS. 5B and 5C, when viewed from a specific direction, one pixel PE is observed to spread over the entire lens 32. . However, when the distance G is further away from the focal distance F, the lens 32 also displays the adjacent pixel PE, so that the stereoscopic image quality is impaired. That is, if the pixel pitch length in the sub-pixel unit of the pixel panel 20 is S and the lens pitch length of the lens plate 30 is L, the distance G is arranged so as to satisfy the following expression (1). Therefore, it is possible to realize a stereoscopic view with better image quality than in other cases.
(LS) · F / L ≦ G ≦ (L + S) · F / L (1)

In the IP system, the distance G is longer than the focal length F in order to form an image at a constant distance C. That is, the following equation is established. In this respect, the system in this embodiment is different in principle from the IP system.
1 / G + 1 / C = 1 / F, that is, G = (C · F) / (C−F)> F
In the light beam reproduction method, since an image is formed at a plurality of distances, a lens cannot be used as an optical element, and a pinhole is used. In this respect, this method is also different in principle from the light beam reproduction method.

  By disposing the display panel 20 and the lens plate 30 in this way, the focal point of each lens 32 is located at one point on the display surface 22 of the pixel panel 20, and the pixel PE where the focal point is located appears enlarged by the lens 32. It will be. In addition, when it can be considered that it is optically substantially equivalent, you may arrange | position so that the uneven surface of the lens plate 30 may oppose the display surface 22 of the pixel panel 20. FIG.

  Further, as shown in FIG. 2B, the lens plate 30 has a principal dotted line 36 (a set of principal points) of each lens 32. Since the microlens of the lenticular lens plate has a cylindrical cross section (a bowl shape), the principal point Are arranged in such a manner that the direction of the pixel line 20 coincides with the vertical pixel arrangement direction (vertical scanning direction) of the pixel panel 20. In FIG. 2B, a line 32a indicates an end portion of each lens 32 of the lens plate 30.

By the way, the conventional lenticular stereoscopic image display device is designed so that the lens pitch of the lens plate matches the pixel pitch of the pixel panel (hereinafter simply referred to as “pitch match”). That is, in the case of the n-eye system, the following expression (2) is established.
L = n · S (2)

However, in the present embodiment, the lens plate 30 is designed such that the lens pitch does not match the pixel pitch of the pixel panel 20 (hereinafter simply referred to as “pitch does not match”). That is, the following equation (3) is not satisfied.
L = n · S (3)
However, n is a natural number.

By the way, when an observer actually views a stereoscopic image displayed on the stereoscopic video display device, the viewpoint of the observer is located at a finite distance from the display surface 22. That is, as shown in FIG. 3, the line-of-sight direction of the observer's viewpoint differs depending on the location of the display surface 22, and therefore the correspondence between the lens 32 and the pixel PE is shifted. That is, the substantial lens pitch L ′ is given by the following equation (4). The figure shows a cross-sectional view in the transverse direction with respect to the display surface of the stereoscopic video display device of the present embodiment.
L ′ = L × (D + F) / D (4)
However, D is the distance between an observer's viewpoint and a display surface.

Therefore, strictly speaking, it can be said that the following equation (5) is satisfied in the “pitch match” state, and not satisfied is the “pitch mismatch” state.
L ′ = n · S (5)

In addition, “pitch matches / does not match” refers to a viewing angle (versus-pixel viewing angle) σ with respect to one pixel PE viewed from the viewpoint of an actual (or assumed) observer, and an emission ray of the pixel PE. It is also expressed by a viewing angle (a viewing angle with respect to the lens) λ with respect to one lens 32 that imparts a characteristic. The anti-pixel viewing angle σ is given by the equation (6a), and the anti-lens viewing angle λ is given by the following equation (6b).
tanσ = S / (D + F) (6a)
tanλ = L / D (6b)

If the following expression (7) is satisfied, the “pitch is matched” state is satisfied, and if not satisfied, the “pitch is not matched” state.
λ = n · σ (7)
However, n is a natural number.

  That is, Expression (7) is satisfied when m times the viewing angle λ of the lens matches n times viewing angle σ of the pixel, that is, m times the lens pitch L ′ matches n times the pixel pitch S. In this case, it can be said that the pitch is “matched”.

The condition of Expression (7) is a condition in the case of not considering the necessity of making the distance between the viewpoints coincide with the binocular distance of the human in the conventional multi-view or super-multi-view stereoscopic view. In order to make the distance between the viewpoints coincide with the binocular distance of the human, as shown in FIG. 50, the distance between the observation distance D, the binocular distance E, the pixel pitch S, and the focal distance F of the lens. The following equation (8) needs to be satisfied.
E / D = S / F (8)
That is, in the case of a multi-lens system in which the distance between the respective viewpoints is matched with the human binocular distance, it is necessary to satisfy the expressions (7) and (8) at the same time. Therefore, an accurate lens design is necessary for the conventional multi-lens system.

  As described above, in the conventional multi-view system, the same viewpoint is repeatedly generated for every certain length in the horizontal direction. In addition, in these multi-view systems, images (individual viewpoint images) based on n preset viewpoints (individual viewpoints) are generated, and these images are rearranged (interleaved) according to the viewpoint repetition pattern. Thus, a stereoscopic image is generated.

However, the image generation method of the present embodiment is characterized in that a good stereoscopic image can be obtained only when the same viewpoint is not repeated as described above. Here, it can be said that the condition that the same viewpoint repeats in the horizontal direction is that the relationship of the following expression (9) is established between the above-described anti-pixel viewing angle σ and the anti-lens viewing angle λ.
λ: σ = n: m (9)
However, n and m are natural numbers.

  That is, when Expression (9) is not satisfied, “the same viewpoint is not repeated”, that is, “the pitch is not matched. However, the repetition of the same viewpoint is at least within the image display area of the stereoscopic image. Therefore, the condition that “the repetition of the same viewpoint does not occur”, that is, “the pitch does not match” is “the natural number where m is less than Φ / λ” in Equation (9). That is. However, Φ is a viewing angle (versus display area viewing angle) with respect to an area (stereoscopic image display area) in which a stereoscopic image on the display surface 22 is viewed from the viewpoint of an actual (or assumed) observer.

  However, the present embodiment uses a stereoscopic video display device that does not match the pitch (that is, equations (7) and (8) do not hold), and generates a stereoscopic image to be displayed on this stereoscopic video display device. Shall. In other words, in the conventional lenticular stereoscopic image display device, it is necessary to design the lens pitch and the pixel pitch to be in order to enable stereoscopic viewing. Enables stereoscopic viewing in a video display device. Therefore, it is not necessary to manufacture a lenticular lens plate that matches the pixel pitch of each display, and the ready-made lenticular lens plate can be applied to various displays, so that the cost of the lens plate can be greatly reduced. . It is also possible to select the most suitable one from a plurality of selectable lens plates. Furthermore, when the oblique lenticular method is adopted, the oblique angle θ can be set freely. Specifically, moire and color fringes can be reduced by adjusting the oblique arrangement angle θ of the lens plate with respect to the pixel panel without manufacturing a new lens.

  Next, a method for determining the line of sight V of each pixel PE on the display surface 22 will be described. Before that, the coordinate system of the display surface 22 is defined as shown in FIG. That is, the direction along the horizontal scanning direction (lateral direction) of the display surface 22 is the x-axis direction, the direction along the vertical scanning direction (longitudinal direction) is the y-axis direction, and is perpendicular to the viewer side from the display surface 22. The direction to go is the z-axis positive direction.

  First, the most basic method of determining the line of sight V in this method will be described. In this method, the observer's line of sight is “front” of the display surface 22 (the position where the observation line of sight passing through the center O of the display surface 22 is perpendicular to the display surface 22) and “infinity”. In other words, the non-stereoscopic image drawing method corresponds to “parallel projection”. Here, a method of determining the line of sight V when the stereoscopic video display apparatus is each of the above-described vertical / oblique lenticular methods will be described. In the following, a method for determining the line of sight V for one pixel PE will be described, but it is obvious that other pixels PE can be similarly determined.

  A method of determining the line of sight V when the stereoscopic video display device is of the vertical lenticular method will be described with reference to FIG. FIG. 5 is a schematic three-view diagram of a vertical lenticular stereoscopic image display apparatus 200A. FIG. 5A shows a cross-sectional view (cross-sectional view in the horizontal scanning direction) parallel to the xz plane. b) shows a cross-sectional view (cross-sectional view in the vertical scanning direction) parallel to the yz plane, and FIG. 5C shows an xy plan view.

  First, a lens 32 corresponding to a pixel (hereinafter referred to as “target pixel”) PE that determines the line of sight V is determined. In FIG. 5A, each lens 32 of the lens plate 30 is projected in parallel on the display surface 22 of the pixel panel 20 (that is, by a straight line perpendicular to the display surface 22 passing through the end portion 32a of each lens 32). The display surface 22 is divided into projection areas of the lenses 32. The lens 32 in the projection region to which the representative point of the target pixel PE (here, the center of the pixel) belongs is set as the lens 32 corresponding to the target pixel PE. However, FIG. 4A is a cross-sectional view passing through a representative point of the target pixel PE.

  In FIG. 5A, the display surface 22 is divided into a projection area 26-1 of the lens 32-1, a projection area 26-2 of the lens 32-2, and so on. Since the representative point of the target pixel PE belongs to the projection area 26-1, the lens 32 corresponding to the target pixel PE is the lens 32-1.

  Next, a light ray PR (hereinafter referred to as “representative ray”) PR after passing through the representative point of the target pixel PE and the principal point of the lens 32 corresponding to the target pixel PE is calculated, and the position is the same as that of the representative ray PR. The line of sight whose direction has been reversed is set as the line of sight V of the target pixel PE. Specifically, a point where the y coordinate is equal to the y coordinate of the representative point of the target pixel PE is calculated from the main dotted line 36 of the lens 32-1 corresponding to the target pixel PE, and this is set as the representative main point 36a. Then, a representative ray PR after passing through the representative point of the target pixel PE and the representative principal point 36a is calculated, and the line of sight having the same position and the opposite direction as the representative ray PR corresponds to the target pixel PE. The line of sight is V. Here, for the sake of simplicity, the direction from the representative point of the target pixel PE toward the representative principal point of the lens 32 corresponding to the target pixel PE is defined as the direction of the representative light ray PR.

  The method described above assumes that the observer's viewpoint is at infinity, but when observing an actual stereoscopic image, the observer's viewpoint is not at infinity. When observing from a short distance, you may feel a sense of incongruity in perspective. However, since the method of determining each line of sight V is simple, there is an advantage that the calculation load can be reduced.

  Next, a method for determining the line of sight V that enables more natural stereoscopic viewing than the above-described method will be described. This method is a drawing method that assumes the observer's viewpoint at a fixed position, and corresponds to “perspective projection” in terms of a non-stereoscopic image drawing method.

  Here, as shown in FIG. 6, the assumed observation position 40 is set to “front” with respect to the display surface 22 of the stereoscopic video display device. The “front” assumed observation position 40 is a position where an observation line-of-sight direction passing through the center O of the display surface 22 is perpendicular to the display surface 22. Hereinafter, the distance D between the assumed observation position 40 and the display surface 22 is referred to as “assumed observation distance D”. Then, the case where the method of determining the line of sight V when the assumed observation position 40 is “front position and a fixed position at a finite distance” is applied to a vertical lenticular stereoscopic image display device will be described.

  FIG. 7 is a partial schematic perspective view of a vertical lenticular stereoscopic image display apparatus 200A. 8 is a schematic three-view diagram of the stereoscopic video display device 200A. FIG. 8A is a cross-sectional view (cross-section in the horizontal scanning direction) at the position AA ′ parallel to the xz plane of FIG. 8 (b) is a cross-sectional view (vertical cross-sectional view) at a position BB ′ parallel to the yz plane of FIG. 7, and FIG. 8 (c) is an xy diagram. A plan view is shown. The lens plate 30 and the pixel panel 20 are arranged in parallel with a focal length F of each lens 32 of the lens plate 30.

  First, the lens 32 corresponding to the target pixel PE is determined. Specifically, in FIG. 8A, each lens 32 of the lens plate 30 is projected from the assumed observation position 40 onto the display surface 22 of the pixel panel 20 (that is, the end of each lens 32 from the assumed observation position 40). The display surface 22 is divided into projection areas of each lens 32 (by a straight line going to). Then, the corresponding lens 32 is determined depending on which projection region the representative point of the target pixel PE belongs to. However, FIG. 4A is a cross-sectional view passing through a representative point of the target pixel PE.

  In FIG. 6A, the display surface 22 includes a projection area 26-7 of the lens 32-7, a projection area 26-8 of the lens 32-8, a projection area 26-9 of the lens 32-9,.・ It is divided into Since the representative point of the target pixel PE belongs to the projection area 26-7, the lens 32 corresponding to the target pixel PE is the lens 32-7.

  Next, a representative ray after passing through the representative point of the target pixel PE and the principal point of the lens 32 corresponding to the target pixel PE is calculated, and a line of sight with the same position as the calculated representative ray but in the opposite direction is calculated. The line of sight V of the target pixel is assumed. Specifically, in FIG. 8B, a straight line LN1 connecting the representative point of the target pixel PE and the assumed observation position 40 and the principal point surface of the lens plate 30 (a surface including the principal point of each lens 32). The y-coordinate of the intersection with 35), which is a plane parallel to 22 is calculated. The calculated y coordinate is assumed to be “y1”. However, FIG. 4B is a cross-sectional view passing through the representative point of the target pixel PE. Next, a point whose y coordinate is “y1” is calculated from the main dotted line 36 of the lens 32-1 corresponding to the target pixel PE, and this is set as a representative main point 36c. Then, a representative ray PR after passing through the representative point of the target pixel PE and the representative principal point 36c is calculated, and the line of sight having the same position and the opposite direction to the representative ray PR corresponds to the target pixel PE. The line of sight is V.

  The method for determining the line of sight V of each pixel PE on the display surface 22 when the assumed observation positions are “front and infinity” and “front and finite distance fixed positions” has been described above. In the above, light rays incident on each lens 32 are not refracted (that is, the direction from the representative point of the target pixel PE toward the representative principal point of the lens 32 corresponding to the target pixel PE matches the direction of the representative light ray PR). Strictly speaking, as shown in FIG. 9, the representative ray PR connects the representative point of the target pixel PE and the representative principal point of the lens 32 corresponding to the target pixel PE, as shown in FIG. The y coordinate position slightly deviates from the straight line and does not match. Therefore, it is more preferable that the line of sight V of each pixel PE is accurately obtained by calculating and correcting this shift.

  In addition, as described with reference to FIGS. 7 and 8, the determination of the lens 32 for each pixel PE is performed by setting the assumed observation position 40 as “a fixed position at a finite distance”, so that the degree of perspective (perspective) is affected. In addition to becoming natural, the effect of widening the observation range around the assumed observation position 40 can be obtained.

  In this way, after determining the line of sight V corresponding to each pixel, as shown in FIG. 10, a pixel-specific viewpoint CM corresponding to the virtual camera is set for each pixel PE based on the determined line of sight V. Here, the pixel-specific viewpoint CM corresponding to the pixel PE is set for the pixel PE, but the pixel-specific viewpoint CM is not set in particular, and a drawing range in the z direction common to the line of sight V for all the pixels PE is set. However, drawing may be performed for each line of sight V.

  FIG. 10 is a diagram for explaining the setting of the pixel-specific viewpoint CM, and shows a partial horizontal cross-sectional view of the display surface 22. As shown in the figure, the pixel-specific viewpoints CM (CM1, CM2,...) Of each pixel PE (PE1, PE2,...) Have a line of sight V (V1, V2,. Set so that Further, the distance between each pixel-specific viewpoint CM and the display surface 22 is set so as to be located on the same plane parallel to the display surface 22, for example, as shown in FIG.

  In the figure, the visual lines V of the pixels PE1, PE2,... Are visual lines V1, V2,. Accordingly, the pixel-specific viewpoint CM of the pixel PE1 has the line of sight V1 as the pixel-specific viewpoint CM1 in the line-of-sight direction. In the pixel-specific viewpoint CM of the pixel PE2, the line of sight V2 is the pixel-specific viewpoint CM2 in the line-of-sight direction. Further, similarly, for the pixels PE3, PE4,..., The viewpoints CM for each pixel have the line of sight V3, V4,. Become.

(2) Rendering After setting the pixel-specific viewpoint CM of each pixel PE, a stereoscopic image is generated by rendering a three-dimensional virtual space based on the set pixel-specific viewpoint CM. Specifically, for each pixel PE, color information (RGB value, α value, etc.) of the object space in the line-of-sight direction of the pixel-specific viewpoint CM corresponding to the pixel PE is calculated, and the calculated color information is calculated for the pixel PE. A stereoscopic image is generated by using color information.

  FIG. 11 is a diagram for explaining the calculation of the color information, and shows a partial cross-sectional view of the display surface 22 in the horizontal direction. As shown in the figure, for each pixel PE on the display surface 22, the color information of the object space in the line-of-sight direction of the corresponding pixel-specific viewpoint CM is calculated, and the calculated color information is used as the color information of the pixel PE. The color information calculation method is realized by, for example, a so-called ray-tracing method that is determined based on light rays along the line-of-sight direction from the pixel-specific viewpoint CM.

  In the figure, the pixel-specific viewpoints of the pixels PE1, PE2,... Are pixel-specific viewpoints CM1, CM2,. Therefore, the color information of the pixel PE1 is color information of the object space in the line-of-sight direction of the pixel-by-pixel viewpoint CM1, and the color information of the pixel PE2 is color information of the object space in the line-of-sight direction of the pixel-by-pixel viewpoint CM2. Further, for each of the pixels PE3, PE4,..., The color information of the object space in the line-of-sight direction of the corresponding pixel-specific viewpoints CM3, CM4,. , Color information.

  As described above, in this embodiment, for each pixel PE on the display surface, (1) the line of sight V is determined, and (2) the color information of the line of sight of the determined line of sight V is used as the color information of the pixel PE (rendering). To generate a stereoscopic image.

  When the image generated in this way is displayed as a stereoscopic image on the stereoscopic video display apparatus of the present embodiment, the video visually recognized by the observer is slightly less accurate than the conventional stereoscopic video. Become a statue.

  FIG. 12 is a diagram for explaining that the stereoscopic image of the present embodiment is slightly inaccurate, and shows a partial cross-sectional view of the display surface 22 in the horizontal direction. In this figure, when the stereoscopic video display device is viewed from the observer's right eye EY1, the pixel PE1 can be seen through the lens 32-1, the pixel PE2 can be seen through the lens 32-2, and the lens 32-3 can be seen. Pixel PE3 is visible.

  By the way, the color information of the pixel PE1 is color information of the object space in the line-of-sight direction of the pixel-specific viewpoint CM1, the color information of the pixel PE2 is color information of the object space in the line-of-sight direction of the pixel-specific viewpoint CM2, The color information of the pixel PE3 is color information of the object space in the viewing direction of the pixel-specific viewpoint CM3. That is, since the right eye EY1 does not match the pixel-specific viewpoints CM1, CM2, and CM3, the color information of each pixel PE recognized by the observer is not accurate color information viewed from the position.

  However, the positions of the pixel-specific viewpoints CM1, CM2, and CM3 are in the vicinity of the right eye EY1, and the line-of-sight directions of the pixels PE1, PE2 are determined by the right eye EY1 through the lenses 32-1, 32-2, and 32-3. , PE3 viewing direction is slightly shifted. For this reason, the image (color information) visually recognized by the observer's right eye EY1 is not an accurate image (color information) viewed from the position, but is visually recognized with a certain degree of clarity.

  Further, according to the present embodiment, the number of viewpoints (views) is extremely large, and natural stereoscopic vision is possible. This will be described with reference to FIGS. 13 and 14 compared with the conventional multi-view stereoscopic vision.

  FIG. 13 is a diagram showing an outline (image) of stereoscopic viewing of a conventional multi-view system, and shows a case of a trinocular system. As shown in the upper side of the figure, in the conventional three-eye stereoscopic view, three individual viewpoints 1, 2, 3 are set at an appropriate distance in the object space, and each of the individual viewpoints 1, 2, 3 is set. The individual viewpoint images 1, 2, and 3 of the object space viewed from the above are generated. Then, a stereoscopic image is generated by interleaving these three individual viewpoint images 1, 2, and 3. In the figure, the number of each pixel of the stereoscopic image represents the number of the corresponding individual viewpoint image (individual viewpoint). Further, the position and the line-of-sight direction of each pixel viewpoint CM are rough because they are schematic diagrams (image diagrams), and are not accurate.

  Then, as shown in the lower side of the figure, when the generated stereoscopic image is displayed on a conventional three-lens stereoscopic image display device and viewed from each of the appropriate viewing positions 1, 2, and 3, the suitable viewing position 1 The individual viewpoint image 1 can be seen, the individual viewpoint image 2 can be seen at the appropriate viewing position 2, and the individual viewpoint image 3 can be seen at the suitable viewing position 3. More specifically, the individual viewpoint image 1 can be seen in the suitable viewing range 1 that is substantially centered on the suitable viewing position 1, and the individual viewpoint image 2 can be seen in the suitable viewing range 2 that is substantially centered on the suitable viewing position 2. The image 3 can be seen in the individual viewpoint appropriate viewing range 3 centering on 3. However, in the same figure, the appropriate viewing range is a schematic diagram (image diagram), and is therefore an approximate one and is not accurate.

  That is, when the observer OB views the stereoscopic image at a position where the right eye EY1 substantially matches the appropriate viewing position 2 and the left eye EY2 substantially matches the appropriate viewing position 1, the right eye EY1 sees the individual viewpoint image 2, and the left eye In EY2, the stereoscopic video is recognized when the individual viewpoint image 1 is seen. That is, this corresponds to a state in which the object space is viewed with the right eye EY1 as the individual viewpoint 2 and the left eye EY2 as the individual viewpoint 1.

  Further, when the position of the observer OB moves to the right with respect to the stereoscopic image, when the right eye EY1 or the left eye EY2 passes through the boundary portion of the appropriate viewing range, an image that can be seen with the right eye EY1 or the left eye EY2 suddenly appears. Switch to. Specifically, for example, when the right eye EY1 passes through the boundary portion between the suitable viewing range 2 and the suitable viewing range 3, the image seen by the right eye EY1 is switched from the individual viewpoint image 2 to the individual viewpoint image 3. In addition, when the left eye EY2 passes through the boundary portion between the appropriate viewing range 1 and the appropriate viewing range 2, the image seen by the left eye EY2 is switched from the individual viewpoint image 1 to the individual image 2.

  This is because in conventional multi-view stereoscopic vision, each individual viewpoint image viewed from n individual viewpoints is interleaved to generate a stereoscopic image, which is designed to match the pitch. This is because the stereoscopic view is realized by displaying the image on a stereoscopic image display device. That is, in the conventional stereoscopic video display device, the stereoscopic image is separated into the individual viewpoint images by the lenticular lens plate.

  FIG. 14 is a diagram showing an outline (image) of stereoscopic vision of the present embodiment. In the present embodiment, as described above, a pixel-specific viewpoint CM is set for each pixel PE, and the color information of the object space in the line-of-sight direction of each pixel-specific viewpoint CM is used as the color information of the corresponding pixel PE. An image is generated. That is, as shown in the upper side of the figure, pixel-specific viewpoints CM1, CM2,... Equal to the number of pixels are set, and the color information of the respective line-of-sight directions of the set pixel-specific viewpoints CM1, CM2,. A stereoscopic image is generated as color information of the pixels PE1, PE2,. In the figure, the number of each pixel PE of the stereoscopic image represents the number of the corresponding pixel viewpoint CM.

  The stereoscopic image generated in this way is displayed on the stereoscopic video display device 200A of the present embodiment shown in FIG. 2, for example, and the observer OB views the stereoscopic image at the position shown on the lower side of the figure. Then, the image A composed of the pixels PE1, PE2, PE3,... Can be seen in the left eye EY2, and the image B composed of the pixels PE11, PE12, PE13,. appear. That is, the left eye EY2 is a viewpoint group including pixel-specific viewpoints CM1, CM2,..., CM10, and the right eye EY1 is a viewpoint group including pixel-specific viewpoints CM11, CM12,. It corresponds to the state of watching.

  Then, when the position of the observer OB moves slightly to the right with respect to the stereoscopic image, an image A in which an image seen by the left eye EY2 of the observer is replaced with a pixel PE that is a part of the image A is replaced with an adjacent pixel PE. The image that changes to 'and appears to the right eye EY1 changes to an image B' in which some of the pixels PE of the pixel B are replaced with the adjacent pixels PE.

  Thus, in this embodiment, when the position (observation position) of the observer who views the stereoscopic image changes, the images seen in each of the right eye EY1 and the left eye EY2 change little by little with this change. Specifically, the image changes to a pixel in which some pixels are replaced by neighboring pixels. Therefore, the recognized image changes little by little as the images seen by each of the right eye EY1 and left eye EY2 of the observer OB change little by little.

  For this reason, for example, an image seen at the boundary portion of the appropriate viewing range suddenly changes like the conventional multi-view stereoscopic image shown in FIG. 13 (that is, the recognized stereoscopic image suddenly changes). ), A natural stereoscopic image that changes little by little as the observation position changes can be realized, and the clarity of the image visually recognized by the observer is maintained above a certain level.

  Note that, as described above, the images of the right eye EY1 and the left eye FY2 that are visible to the observer OB are images that are slightly inaccurate from the actual images. However, the line-of-sight direction in which each eye views each pixel is substantially along the line-of-sight direction of the pixel-specific viewpoint CM of the pixel, as shown in the lower side of FIG. That is, the line-of-sight direction in which the left eye EY2 views each pixel PE1, PE2,... Of the image A is that of the pixel-specific viewpoints CM1, CM2,. The direction is substantially along the line-of-sight direction. Similarly, for the right eye EY1, the line-of-sight direction for viewing the pixels PE11, PE12,... Of the image B is the pixel-specific viewpoints CM11, CM12,.・ ・ The direction is almost along the line of sight. For this reason, the image visually recognized by the observer has clarity that can be visually recognized as an image, although it is slightly inaccurate. Further, as described above, even if the position of the observer changes, the clarity of the visually recognized image is maintained at a certain level or higher.

  In addition, the stereoscopic image recognized by the observer in the present embodiment can have the same resolution as the conventional multi-view stereoscopic image. For example, in the stereoscopic video display apparatus 200A shown in FIG. 2, the lens pitch L is 3 to 4 times the pixel pitch S in subpixel units. Therefore, in such a stereoscopic video display device 200, a resolution of about 1/3 to 1/4 of the resolution of the pixel panel 20, that is, a resolution comparable to that of a conventional 3-4 eye stereoscopic video is obtained. become.

  As described above, in the stereoscopic vision according to the present embodiment, although the accuracy of the recognized stereoscopic video is slightly lacking, it has the same resolution as the conventional multi-view stereoscopic video and has a viewpoint ( It is possible to realize a natural stereoscopic video image with a large number of views.

  The method for generating a stereoscopic image according to the present embodiment has been described above based on the method (I). Next, a method for generating a stereoscopic image of this embodiment from a planar image from one viewpoint and its depth image based on the method (II) will be described. Here, method (II) is a) When an image created with an existing high-quality CG tool can be easily three-dimensionalized without using a dedicated renderer. B) When a stereoscopic image is created from a photographed image. C) Depth image creation can be easily created when drawing with a CG tool, d) Depth image corresponding to a real image, A range image camera is required, and its accuracy and resolution are not sufficient at present, but its research is being actively pursued because it can be expected to improve in the future.

  For example, as described in Japanese Patent Application Laid-Open No. 2000-78611, when a binocular stereoscopic image is generated from a planar image and a depth image, the planar image is respectively converted based on the depth value of the depth image. By shifting in the horizontal direction, a parallax image for the left eye and a parallax image for the right eye are generated, and the generated parallax image is interleaved to generate a stereoscopic image. The depth image here is image data representing the depth value at the corresponding position of the planar view image in black and white gradation. If the depth value for each pixel is known, the position in the parallax image where each pixel of the planar image is converted can be uniquely determined from the size of the screen, the observation distance of the observer, and the distance between both eyes. .

  For example, as shown in FIG. 15, first, a position P1 where a line extending perpendicularly from the dot position P0 of the planar view image in the direction of the observer's line of sight and the depth value intersect is set as a position where the pixel is to be displayed. decide. Next, when generating a parallax image for the left eye, a point where the extension line connecting the left-eye viewpoint camera corresponding to the assumed observation position 40 and the position P1 intersects the display surface is set as the pixel movement position P2 for the left eye. . On the other hand, when generating a parallax image for the right eye, a point where the extension line of the line connecting the right eye viewpoint camera corresponding to the assumed observation position 40 and the position P1 intersects the display surface is the pixel movement position P3 for the right eye. And As described above, for each pixel in the planar view image, a position to be displayed is determined based on the depth value, and the movement position of the pixel is determined based on the position and the positions of the left and right viewpoint cameras. Thus, a parallax image for the left eye and a parallax image for the right eye are generated. Then, a binocular stereoscopic image can be generated by interleaving the generated two parallax images.

  On the other hand, in this embodiment, for each pixel on the display surface, (1) a representative point of the pixel (for example, the center of the pixel) and a representative of an optical element (for example, a lens) that gives directivity to the emitted light of the pixel. A direction corresponding to the direction of the light beam in the three-dimensional virtual space is determined based on the direction of the light beam passing through the point and the viewpoint position in the three-dimensional virtual space, and (2) the determined direction and depth image Based on the above, the dot representing the color information of the nearest object in the direction in the three-dimensional virtual space is determined from the planar view image, and the color information of the determined reference image dot is the color of the pixel A stereoscopic image is generated by using the information. Since the method for determining the direction corresponding to the direction of the light beam in (1) is substantially the same as the method (I) described above, the following description is mainly based on (2) the determined direction and the depth image. A method for determining the color information will be described in detail.

First, the outline of the principle will be described.
For example, as shown in FIG. 16, when attention is paid to the pixel at the position P12 on the display surface, the line of sight V and the pixel-specific viewpoint CM11 for the pixel are determined in the same manner as in the method (I). Next, the position at which the trajectory in the direction corresponding to the determined line of sight V (the line passing through the position P12 and the pixel-specific viewpoint CM in the figure, hereinafter referred to as “pixel-specific line-of-sight direction trajectory”) intersects the depth value. P11 is determined as a position where the pixel is to be displayed. Then, the position of the intersection P10 between the line extending perpendicularly to the display surface from the position P11 and the display surface is acquired, and the color information of the dot of the planar image corresponding to the acquired position is obtained from the pixel at the position P12. Obtained as color information.

  Next, a case where there are a plurality of intersections between the per-pixel gaze direction locus and the depth value will be described. For example, when attention is paid to the pixel at the position P24 on the display surface, the pixel-specific line-of-sight direction locus and the depth value intersect at the intersection point P21, the intersection point P22, and the intersection point P23. In this case, an intersection point closest to the observer, that is, an intersection point P21 closest to the individual viewpoint CM22 is determined as a position where the pixel is to be displayed. Then, the position of the intersection P20 between the line extending perpendicularly to the display surface from the intersection P21 and the display surface is acquired, and the color information of the dot of the planar image corresponding to the acquired position is obtained from the pixel at the position P24. Obtained as color information.

  In this way, the line of sight V and the pixel-specific viewpoint CM for each pixel on the display surface are determined, and the pixel-specific line-of-sight direction locus is determined from the direction of the line of sight V and the pixel and pixel-specific viewpoint CM. Among the points where the depth value intersects, the position of the intersection closest to the pixel-specific viewpoint CM is determined as the position where the pixel is to be displayed. Then, the position of the intersection of the line extending perpendicularly to the display surface from the determined position and the display surface is acquired, and the color information of the dot of the planar image corresponding to the acquired position is the color information of the pixel. By doing so, a stereoscopic image can be generated.

Next, a method for generating a stereoscopic image based on a specific planar image and its depth image will be described. In the following, for the sake of simplicity of explanation, a case where the planar view image is a gray scale image (monochrome image) will be described as an example. FIG. 17A is a depth image in which the depth value of the corresponding position of the planar image is represented by black and white gradation, and FIG. 17D is a standard planar image. FIG. 6B is a depth information model created based on the depth value of the depth image, and is obtained by converting the Z value in a certain scan line (for example, Y = y ₀ ) into xy coordinates. This depth information model indicates that the greater the Z value, the closer to the observer, and one dot corresponds to one pixel. The depth image is an image created by a parallel projection method assuming that the observer's viewpoint is at infinity.

FIG. 4C is a diagram simply showing a vertical cross section with respect to the display surface 22, and the intersections X1, X2 of the representative rays PR1, PR2 emitted from the representative points of the pixels PE1, PE2 and the depth information model. It is a figure which shows the correspondence with the planar view image shown to the same figure (a). As shown in FIG. 6C, the intersection point is obtained from the representative ray PR1 emitted from the pixel PE1 (that is, the ray in the direction opposite to the line of sight V and equivalent to the line-of-sight line direction trajectory) and the depth information model. X1 (z ₁ , x ₁ ) is determined. The color information of the dot of the planar view image corresponding to the position of this intersection X1 is acquired. That is, the position coordinate (x ₁ , y ₀ ) of the corresponding planar view image is determined from the intersection X1 (z ₁ , x ₁ ), and the color information acquired from the dot at this position coordinate is used as the color information of the pixel PE1. Ask. Similarly, the position coordinates (x ₂ , y ₀ ) of the corresponding planar view image are determined from the intersection X2 (z ₂ , x ₂ ) between the representative ray PR2 emitted from the pixel PE2 and the depth information model. Then, the color information acquired from the dot at the position coordinates (x ₂ , y ₀ ) of the planar view image is obtained as the color information of the pixel PE2.

  Furthermore, a method for obtaining color information when there are a plurality of intersections between the per-pixel line-of-sight direction trajectory and the depth value of the depth information model will be described in detail. FIG. 18 is a diagram schematically showing a vertical section with respect to the display surface 22, and shows a depth information model in a scan line corresponding to the vertical section. Also, the black circle (●) on the display surface 22 represents the representative point of each pixel, and the x coordinate of each representative point is the same as the x coordinate of the depth value. That is, one pixel corresponds to one pixel. In this method, the direction of the representative ray emitted from the representative point of each pixel is calculated from the positional relationship between the lens and the pixel, and this is used as the direction of the line of sight V. In addition, description of the lens is abbreviate | omitted in the figure.

  FIG. 19 is a diagram showing representative rays PR1 to PR3 corresponding to the pixels PE1 to PE3 by focusing on the pixels PE1, PE2, and PE3 among the pixels shown in FIG. The representative rays PR1 to PR3 show three typical patterns. That is, the representative ray PR1 is a pattern that is emitted from the pixel PE1 in an obliquely upward left direction, the representative ray PR2 is a pattern that is emitted from the pixel PE2 in an obliquely upward right direction, and the representative ray PR3 is vertically upward from the pixel PE3. The pattern which injects in the direction is shown. First, a description will be given of how to obtain the color information of the pixel PE1 when the pattern in which the representative light beam is emitted obliquely upward to the left, that is, when the inclination of the trajectory direction for each pixel is negative.

  FIG. 20 is a diagram illustrating an intersection between V1 and the depth information model, where V1 is the line-of-sight line direction locus for each pixel. In addition, the lens and the pixel are omitted. First, intersection determination between the line-of-sight line direction trajectory V1 for each pixel and all line segments of the depth information model is performed to obtain each intersection (five intersections in the figure). Then, the intersection having the largest depth value (Z value) among all the intersections (that is, the closest intersection from the pixel-specific viewpoint CM, hereinafter referred to as “target intersection”) is acquired as an intersection for obtaining color information. . Then, similarly to the method described with reference to FIGS. 17C and 17D, the color information of the dot corresponding to the position of the target intersection among the dots of the planar image is acquired, and this is used as the color information of the pixel PE1. To do. In the figure, the intersection X1 is the target intersection.

  Similarly, when obtaining the color information of the pixel PE2, a crossing determination is performed with respect to the line-of-sight line direction locus V2 for the pixel PE2 and all line segments of the depth information model, and the Z value is the largest among all obtained intersections. Get the target intersection. And the color information of the dot corresponding to the position of the object intersection among the dots of the planar view image is acquired, and this is used as the color information of the pixel PE2. Further, when obtaining the color information of the pixel PE3, since the line of sight V is perpendicular to the pixel PE3, the color information of the dot corresponding to the position of the pixel PE3 is obtained from the dots of the planar view image, and this is obtained. The color information of the pixel PE3 is used.

  As described above, when obtaining the color information of each pixel, it is necessary to perform intersection determination between the viewpoint direction trajectory for each pixel and all line segments of the depth information model, and the color information of the three-dimensional virtual space is obtained. There was concern that it would take time to ask. In other words, in this embodiment, each pixel of the pixel panel represents the color information of the three-dimensional virtual space based on the direction of the light beam for each pixel, and therefore has as many viewpoints as the number of light beams. It can be said that there are as many viewpoints as there are. Therefore, there has been a problem that a great deal of time is required for calculation processing for each pixel as the number of viewpoints increases. When a moving image is generated in real time and displayed on a stereoscopic video apparatus, it is necessary to generate a stereoscopic image for each frame. Therefore, when the time restriction is considerably large and a stereoscopic image cannot be generated in units of frames, a situation in which a moving image cannot be displayed on the stereoscopic video display device (so-called processing failure) may occur. In view of this, the present inventors have invented a method of performing the process of calculating the target intersection in a short time from the intersection determination between the line-of-sight line direction trajectory for each pixel and the line segment of the depth information model.

Hereinafter, a method for calculating the target intersection in a shorter time will be described.
First, as shown in FIG. 21, the maximum value Zmax and the minimum value Zmin of the depth value (Z value) of the depth information model in a certain scan line are obtained. Subsequently, as shown in FIG. 22, intersections Qmax (z _max , x _max ), Qmin (z _min ) between a plane parallel to the display plane passing through Zmax and Zmin and V1 which is a pixel-specific line-of-sight direction locus for the pixel PE1. , X _min ). Then, with x _max ≦ x ≦ x _min as a search range, intersection determination between the line-of-sight line direction locus V1 for each pixel and the line segment of the depth information model is performed. In the figure, since the slope of V1 is negative, x _max <x _min .

Thereby, the intersection of the line-of-sight line direction locus V1 for each pixel and the line segment group of the depth information model is limited to a range of x _max ≦ x ≦ x _min . That is, since the intersection determination in the ranges of x _max > x and x _min <x can be omitted, the calculation processing speed can be improved.

Further, a method for shortening the calculation processing time will be described. First, the method of intersection determination for the pixel PE1 will be described with reference to FIG. When the line-of-sight direction trajectory for the pixel PE1 is V1 = f (x) and the slope of V1 is negative, the pixel is close to Qmax (x = _xmax ), that is, at the left end of the search range in the figure. The intersection determination is performed in order from the pixel.

For example, as shown in FIG. 23, the positions of the respective pixels PE included in the range of x _max ≦ x ≦ x _min are arranged in order from the left end, x ₁ , x ₂ , x ₃ , x ₄ , x ₅ , x ₆ , x ₇ In this case, first, it obtains the leftmost pixel position x _1, obtains the Z value of V1 at the pixel position x ₁ as Za. Here, the pixel position x _1, the pixel PE1 (i.e., the pixel position x ₄₎ because it is in a position moved three pixels left, obtained as Za = -3 (dz / dx) . Here, dz / dx is the increment of the Z value of V1 when the x coordinate increases in the positive direction by a length corresponding to one pixel, and is obtained simultaneously when obtaining the line-of-sight direction for each pixel. be able to. Further, the Zd with reference to Z values of depth model information at the pixel position x _1, compares the magnitude of Za and Zd. When the magnitude relationship between Za and Zd is the first time Zd ≧ Za, a position where a pixel position x ₁ is determined.

  That is, at a stage where V1 and each line segment of the depth information model have never intersected, Za> Zd, and if V1 and any line segment of the depth information model intersect once, before and after that The magnitudes of Za and Zd are interchanged to satisfy Zd ≧ Za. Further, since the depth value corresponds to one pixel, it is not possible to intersect V1 more than once between pixels. Therefore, by simply comparing the magnitudes of Za and Zd in order from the Qmax side, it is possible to quickly find the intersection where the Z value is the largest. Further, the obtained position does not necessarily match the target intersection of V1 and the line segment of the depth information model. This is because the intersection of V1 and the line segment of the depth information model does not coincide with the representative point of the pixel except for Za = Zd. Therefore, the position to be obtained is made to coincide with the representative point of the pixel accurately, and the position where the pixel value adjacent to the right side of the intersection is obtained is set as the position where the Z value becomes the largest.

On the other hand, when it is Za> Zd, the position of the next pixel PE, in the figure to calculate the Za at the pixel position x _2. In this case, since the pixel is shifted to the right by one, Za at the pixel position x ₂ can be obtained by adding (dz / dx) to Za at the pixel position x ₁ . Then, with reference to the Z value of the depth information model in the pixel position x _2, compares the Za and Zd. In the example of the figure, the pixel position x _2, since the Zd ≧ Za, seek position is the pixel position x _2, and the ends the calculation of the intersection determination at this time.

Next, an intersection determination method for the pixel PE2 will be described with reference to FIG. First, the pixel-line-of-sight direction trajectory for the pixel PE2 and V2 = f (x), V2 and intersection Qmax of the horizontal lines of the maximum and minimum values of Z values of the depth image model _{(z max, x max),} Qmin ( z _min , x _min ) are obtained. Then, as in this case, when the slope of V2 is positive, intersection determination is performed in order from a position close to Qmax (x = _xmax ), that is, from the right end of the search range in the figure.

For example, as shown in FIG. 24, the positions of the pixels PE included in the range of x _min ≦ x ≦ x _max are set to x ₃ , x ₄ , x _{5 in} order from the left end. Then, first, it obtains the pixel position x ₅ rightmost obtains the Z value of V2 at the pixel position x ₅ as Za. Here, the pixel position x _5, since in the pixel PE2 (i.e., the pixel position x ₄₎ has moved from one pixel right position, a Za = (dz / dx). Further, the Zd with reference to Z values of depth model information at the pixel position x _5, compares the magnitude of Za and Zd. In the figure, because it is Za> Zd, the position of the next pixel PE2, calculates Za at the pixel position x ₄ in the figure, with reference to the Z value of the depth information model in the pixel position x _4, Za and Zd Compare The comparison between Za and Zd is repeated until Zd ≧ Za or until the search range x _max ≧ x ≧ x _min is exceeded.

Here, when the search range x _max ≧ x ≧ x _min is exceeded while Za> Zd, the pixel position on the left side of the search range is determined. In other words, when in the search range does not change the magnitude of Za and Zd, the search range of x _min side of the boundary section (in the figure, the x ₃ x ₂ sections) in the V2, the depth information model line And will definitely intersect. Therefore, a V2, adjacent the intersection of the line segment of the depth information model, and most Z value increases pixel, the pixel position of the left side of the search range, and a position for determining the pixel position x ₂ in the drawing.

Incidentally, if it exceeds still search range Za> Zd, the boundary interval x _max side of the search range (in the drawing, interval x ₆ from x ₅₎ in, and V2, and the segment depth information model Never cross. This is because, when V2 and the line segment of the depth information model intersect at the boundary section on the _xmax side, the line segment of the depth information model in the section will always exceed the maximum value Zmax of the depth value. . That is, when such an intersection occurs, there is a contradiction in the position of the intersection point Qmax between V2 and the maximum horizontal line of the depth information model. Accordingly, when the search range is exceeded while Za> Zd, it can be determined that V2 and the line segment of the depth information model do not intersect in the boundary section on the x _max side of the search range.

Similarly, in the intersection determination pixel PE1, if it exceeds still search range of Za> Zd, (in FIG. 23, the interval from x ₇ x ₈₎ boundary interval x _min side of the search range in a V1, depth The line segment of the information model always intersects. Therefore, such a case, the pixel position to the right of the search range, and a position for determining the pixel position x ₈ in the figure.

Further, a method of intersection determination for the pixel PE3 will be described. In this case, it is the same as the normal intersection determination in the case where the inclination of the pixel-specific gaze direction locus is zero. That is, as shown in FIG. 25, if the pixel-specific line-of-sight direction locus is V3, there is one intersection between V3 and the depth information model. Accordingly, the pixel position corresponding to the intersection point, and a position for determining the pixel position x ₅ in the drawing. In this case, it is not necessary to calculate the intersection determination as in the case where the inclination of the line of sight V is not zero.

  As described above, when the intersection determination between the line-of-sight line direction trajectory for the pixel PE and the depth information model is performed, the search range for performing the intersection determination is determined based on the maximum value and the minimum value of the depth value of the depth information model. To do. Further, when the inclination of the trajectory direction for each pixel is negative, Za is obtained in order from the leftmost pixel position that is most troublesome for the pixel-specific gaze among the pixels included in the search range, and is compared with the corresponding Zd. . On the other hand, when the inclination of the pixel-specific line-of-sight direction locus is positive, Za is obtained in order from the rightmost pixel position located most laborious for the pixel-specific line-of-sight among the pixels included in the search range, and compared with the corresponding Zd. When the comparison result is Zd ≧ Za, the pixel position is the position to be obtained, and color information of the pixel corresponding to the pixel position is acquired from the planar view image, and this is used as the color information of the pixel PE. be able to.

  Thereby, the intersection determination can be performed by a simple calculation such as calculation of Za corresponding to each pixel position included in the search range and comparison between Za and Zd. Specifically, by sequentially performing the intersection determination for each pixel from the Qmax side, Za can be calculated only by addition processing (for example, addition of dz / dx), and the calculation processing can be speeded up. . Further, by sequentially performing the intersection determination from the Qmax side, the subsequent search can be terminated when the condition is satisfied, so that the intersection where the Z value becomes the maximum can be searched quickly. Thereby, the number of calculations can be reduced and the target intersection can be searched efficiently.

[Stereoscopic image generation device]
Next, a stereoscopic image generation device based on the above-described principle will be described. Such a stereoscopic image generation apparatus generates a stereoscopic image that realizes stereoscopic viewing of a moving image.

  FIG. 26 is a block diagram illustrating a configuration of the stereoscopic image generation apparatus 1 according to the present embodiment. As shown in the figure, the stereoscopic image generation apparatus 1 includes an input unit 100, a stereoscopic video display device 200, a processing unit 300, and a storage unit 400.

  The input unit 100 receives an operation instruction from a user and outputs an operation signal corresponding to the operation to the processing unit 300. This function is realized by an input device such as a button switch, lever, joystick, dial, mouse, trackball, keyboard, tablet, touch panel, or various sensors.

  The stereoscopic video display device 200 is a display device that displays the stereoscopic image generated by the stereoscopic image generation unit 320 and allows the observer to recognize the stereoscopic video. In the present embodiment, for example, it is realized by the vertical lenticular stereoscopic image display device shown in FIG.

  The processing unit 300 performs various arithmetic processes such as control of the entire stereoscopic image generation apparatus 1 and image generation. This function is realized by, for example, an arithmetic device such as a CPU (CISC type, RISC type), ASIC (gate array, etc.) or a control program thereof. In particular, in the present embodiment, the processing unit 300 includes an object space setting unit 310 that sets an object space that is a three-dimensional virtual space, and a stereoscopic view that generates a stereoscopic image of the object space set by the object space setting unit 310. An image generation unit 320.

  The stereoscopic image generation unit 320 includes a pixel-specific viewpoint setting unit 322, a rendering unit 324, and a color information setting unit 326, and a stereoscopic image generation program 410 and a color information setting processing program stored in the storage unit 400. By executing the processing according to 420, a stereoscopic image of the object space set by the object space setting unit 310 is generated, and the generated stereoscopic image is displayed on the stereoscopic video display apparatus 200.

  The pixel-specific viewpoint setting unit 322 sets the pixel-specific viewpoint CM in the object space with reference to the display device data 430 and the assumed observation position data 440. Specifically, for each pixel PE on the display surface 22 of the stereoscopic video display device 200, the corresponding lens 32 is determined with reference to the display device data 430 and the assumed observation position data 440. Then, the representative ray PR after passing through the representative point of the pixel PE and the principal point (specifically, the representative principal point) of the lens 32 corresponding to the pixel PE is calculated. The same line of sight with the direction reversed is set as the line of sight V of the pixel PE. At this time, the lens 32 and the line of sight V corresponding to the pixel PE are determined by a method according to the stereoscopic video display device 200. This is performed as described with reference to FIGS.

  The pixel-specific viewpoint setting unit 322 sets a pixel-specific viewpoint CM with the calculated line of sight V as the line-of-sight direction for each pixel PE. Further, the position of the pixel-specific viewpoint CM is set based on the setting reference position determined by the stereoscopic image generation unit 320. Specifically, for example, as illustrated in FIG. 11, each pixel viewpoint CM is set on the same plane parallel to the display surface 22.

  Here, the display device data 430 is data of configuration parameters of the stereoscopic video display device 200. FIG. 27 shows an example of the data configuration of the display device data 430. According to the figure, the display device data 430 includes a pixel pitch (431) of the pixel panel 20 constituting the stereoscopic video display device 200, a lens pitch (432) and a focal length (433) of the lens plate 30, and pixels. The arrangement angle (434) of the lens plate 30 with respect to the panel 20 is stored.

  The arrangement angle (434) stores the value of the angle θ formed by the pixel pitch direction of the pixel panel 20 and the lens pitch direction of the lens plate 30. That is, the arrangement angle (434) is data indicating whether the stereoscopic image display apparatus 200 is of a vertical / oblique lenticular method. In the case of the vertical lenticular method, θ = 0 °, and the oblique lenticular method. In some cases, θ is not 0 °. The display device data 430 is stored in advance as fixed data, but may be set according to a user input from the input unit 100 as described later.

  The assumed observation position data 440 is data of the assumed observation position 40. Specifically, the display surface of the pixel panel 20 of the stereoscopic video display device 200 and the assumed observer's viewpoint (assumed observation position). The value of the assumed observation distance D between 40 is stored. The assumed observation position data 440 is stored in advance as fixed data, but may be set by user input from the input unit 100. By making it possible to set the assumed observation position data 440 by user input, it is possible to easily cope with a case where it is desired to change the assumed viewpoint position. Further, the observation position may be automatically fed back (input) using a head tracking device as an input device.

  The pixel-specific viewpoint CM data of each pixel PE set by the pixel-specific viewpoint setting unit 322 is stored in the pixel-specific viewpoint data 450. FIG. 28 shows an example of the data configuration of the pixel-specific viewpoint data 450. According to the figure, the pixel-specific viewpoint data 450 stores a set pixel-specific viewpoint (452) in association with each pixel (451) of the pixel panel 20 of the stereoscopic video display apparatus 200. In the viewpoint by pixel (452), data of the line of sight V is stored as data representing the corresponding viewpoint by pixel CM.

  The rendering unit 324 sets one virtual camera in the object space set by the object space setting unit 310, performs rendering by the Z buffer method based on the viewpoint of the virtual camera, and performs planar view image data 462 and depth image data 464 is generated.

  The color information setting unit 326 executes color information setting processing according to the color information setting processing program 420 and sets color information of a stereoscopic image in the three-dimensional virtual space. Specifically, for each pixel PE, the pixel PE and the pixel-specific viewpoint (data of the line of sight V) with respect to the pixel PE are acquired from the pixel-specific viewpoint data 450 to calculate the pixel-specific line-of-sight direction locus, and this pixel-specific line-of-sight is obtained. Of the intersections between the direction trajectory and the depth information model 470 in the scan line corresponding to the position of the pixel PE, a target intersection having the maximum Z value is determined. Then, the pixel position corresponding to the determined position of the target intersection is acquired, and the color information of the dot corresponding to the pixel position acquired from the planar view image is obtained as the color information of the pixel PE. The obtained color information of each pixel PE is written in the corresponding position of the stereoscopic image data 480 by the color information setting unit 326.

  The storage unit 400 stores a system program, data, and the like for causing the processing unit 300 to control the stereoscopic image generation apparatus 1 in an integrated manner, and is used as a work area of the processing unit 300. The executed calculation results, input data input from the input unit 100, and the like are temporarily stored. This function is realized by, for example, various IC memories, a hard disk, a floppy (registered trademark) disk, a CD-ROM, a DVD, an MO, a RAM, a VRAM, and the like.

  In particular, in the present embodiment, the storage unit 400 assumes a stereoscopic image generation program 410 for causing the processing unit 300 to function as the stereoscopic image generation unit 320, a color information setting processing program 420, display device data 430, and the like. Observation position data 440, pixel-specific viewpoint data 450, rendering data 460, depth information model 470, and stereoscopic image data 480 are stored.

  The stereoscopic image generation processing program 410 is a program for causing the processing unit 300 to function as the stereoscopic image generation unit 320. The color information setting processing program 420 is a program for causing the stereoscopic image generation unit 322 to function as the color information setting unit 326.

  The rendering data 460 stores planar image data 462 generated by the rendering unit 324 and depth image data 464. FIG. 29 shows an example of the planar image data 462, and FIG. 30 shows an example of the depth image data 464 corresponding to the planar image data 462 of FIG. The planar image data 462 is image data in which color information (for example, RGB value, α value) is stored, and the depth image data is an image that represents the depth value of the corresponding position of the planar image in monochrome gradation. It is data. The planar view image data 462 is a kind of reference image, and the depth image 464 is a kind of depth value information.

  The depth information model 470 stores the depth value of the depth image data for each scan line. FIG. 31 shows an example of the depth information model. FIG. 31B is a model diagram in which the depth value in the scan line 11 of the depth image data shown in FIG. In FIG. 31 (a), the larger the Z value (brighter), the closer to the observer. The depth information model 470 is a type of depth value information.

  The stereoscopic image data 480 stores image data for one frame (specifically, color information of each pixel) generated by the stereoscopic image generation unit 320.

[Process flow]
Next, the process flow will be described.
FIG. 32 is a flowchart showing a flow of stereoscopic image generation processing in the present embodiment. This process is a process for generating and displaying a stereoscopic image for each frame, that is, realizing a real-time moving image stereoscopic view. The stereoscopic image generation unit 320 stores a stereoscopic image generation processing program 410 in the storage unit 400. It is realized by executing. The planar view image and the depth image are gray scale images, and the pixel referred to below corresponds to one pixel.

  According to the figure, in the stereoscopic image generation process, the stereoscopic image generation unit 320 first determines a reference position (setting reference position) that serves as a reference for the setting position of the pixel-specific viewpoint CM (step S11). Next, by executing the process of loop A for each pixel PE on the display surface 22, the pixel-specific viewpoint CM of each pixel PE is set in the object space.

  In the loop A, the pixel-specific viewpoint setting unit 322 determines a lens 32 corresponding to a pixel PE that is a processing target (hereinafter referred to as “the pixel”) (step S13). Next, a ray (representative ray) PR after passing through the representative point of the pixel PE and the principal point (representative principal point) of the lens 32 corresponding to the pixel PE is calculated, and the position is the same as the representative ray PR. The line of sight whose direction has been reversed is set as the line of sight V (step S15). Then, a pixel-by-pixel viewpoint CM in which the direction of the line of sight V is the line-of-sight direction is set (step S17). Loop A is executed in this way.

  When the process of Loop A is performed for all the pixels PE on the display surface 22, the loop A is terminated. When the loop A ends, the stereoscopic image generation unit 320 subsequently executes the process of loop B for each frame.

  In the loop B, first, the object space is set by the object space setting unit 310 (step S19). Subsequently, a virtual camera is set in the set object space (step S21), and the rendering unit 324 performs rendering by the Z buffer method based on the set virtual camera, and the planar view image data 462 and the depth image data 464. Is generated (step S23). Thereafter, based on the generated depth image data 464, a depth information model 470 obtained by converting the depth value for each scan line into the zx coordinate is generated (step S25).

  Subsequently, the color information setting unit 326 executes a color information setting process (see FIG. 33) for setting the color information of the pixel PE to be processed (the pixel). When the color information setting process for all the pixels PE on the display surface 22 ends, the stereoscopic image generation unit 320 converts the image data for one frame stored in the stereoscopic image data 480 into the stereoscopic image. Is displayed on the stereoscopic video display device 200 (step S23).

  In this manner, the stereoscopic view of the moving image is realized by repeatedly executing the process of Loop B for each frame. Then, for example, when a stereoscopic image generation end instruction is input by inputting a stereoscopic image generation end instruction from the input unit 100, the stereoscopic image generation unit 320 ends the loop B, and in this embodiment The stereoscopic image generation process ends. In addition, when it takes time to generate an image or when it is desired to reuse the generated moving image later, the output destination may be a storage medium such as a hard disk instead of the stereoscopic video display device 200. . In addition, when the processing time is sufficient, output may be performed on both the stereoscopic video display device 200 and the storage medium.

  Next, the flow of color information setting processing will be described with reference to FIG. As shown in FIG. 33, in executing the color information setting process, the color information setting unit 326 sets the maximum value Zmax and the minimum value Zmin of the depth value of the depth information model 470 for each scan line (step S31). . Then, a loop process is started for each pixel PE on the display surface 22.

  First, the color information setting unit 326 determines whether or not the inclination of the pixel-specific gaze direction locus Vd with respect to the pixel is vertical (step S33). When the inclination of the pixel-specific line-of-sight direction locus Vd is vertical (step S33; YES), that is, in the same manner as PE3 in FIG. 25, the color information of the dot in the planar view image corresponding to the xy coordinates of the pixel position is obtained. The color information of the pixel is set (step S37).

When the inclination of the pixel-specific gaze direction locus V _d with respect to the pixel is not vertical (step S33; NO), the pixel-specific gaze is based on the gaze direction of the pixel PE to be processed (the pixel) and the pixel-specific viewpoint CM. The direction locus V _d = f (x _d ) is calculated, and the intersection points Qmax (z _max , x _max ), Qmin between V _d = f (x _d ) and a plane parallel to the display plane passing through Z _max and Z _min (z _min , x _min ) are respectively acquired (step S35). Next, it is determined whether or not the slope is negative, that is, whether or not (dz / dx) <0 (step S39). If the slope is negative (step S39; YES), the same as the example of the pixel PE1 described in FIG. 23, first, nearest pixel positions in x = x _max in the search range, i.e., x _max ≦ Zd A pixel position x _d satisfying (z _d , x _d ) <x _max +1 pix is acquired (step S41). Then, Za corresponding to the acquired pixel position _xd is calculated (step S43), and Zd and Za are compared with reference to Zd corresponding to the pixel position _xd (step S45).

Here, when Za ≧ Zd (step S45; NO), the color information of the dot of the planar image corresponding to the xy coordinates of the pixel position _xd is set as the color information of the pixel (step S51). On the other hand, when Zd> Za is satisfied (step S45; YES), the next pixel position is acquired (x _d ← x _{d + 1} ), and Za corresponding to the pixel position x _d is calculated (Za ← Za +). dz / dx) (step S47). Further, it is determined whether or not the pixel position x _d is greater than x _min of Qmin (step S49). When the pixel position x _d exceeds x _min (step S49; YES), that is, when the pixel position x _d exceeds the search range, the dot of the planar view image corresponding to the xy coordinates of the pixel position x _d is determined. The color information is used as the color information of the pixel (step S51).

On the other hand, when the pixel position x _d does not exceed x _min of Qmin (step S49; NO), i.e., when the pixel position x _d is within the search range, the process proceeds to step S45, and the above processing is repeatedly And search for a coordinate position _xd where Za ≧ Zd.

Returning to step S39, if the slope of the pixel-sight direction trajectory V _d is positive; are the same as examples of (step S39 NO), the pixel PE2 described in FIG 24, first, in the search range x = x _A pixel position closest to _max , that is, a pixel position x _d that satisfies x _max ≧ Zd (z _d , x _d )> x _max −1 pix is acquired (step S53). Then, Za corresponding to the acquired pixel position _xd is calculated (step S55), and Zd and Za are compared with reference to Zd corresponding to the pixel position _xd (step S57).

Here, when Za ≧ Zd (step S57; NO), the color information of the dot of the planar image corresponding to the xy coordinates of the pixel position _xd is set as the color information of the pixel (step S51). On the other hand, when Zd> Za is satisfied (step S57; YES), the next pixel position is acquired (x _d ← x _d−1 ), and Za corresponding to the pixel position x _d is calculated (Za ← Za−). dz / dx) (step S59). Further, it is determined whether or not the pixel position x _d is greater than x _min of Qmin (step S61). When the pixel position x _d exceeds x _min (step S61; YES), that is, when the pixel position x _d exceeds the search range, the dot of the planar image corresponding to the xy coordinate of the pixel position x _d is changed. The color information is used as the color information of the pixel (step S51).

On the other hand, when the pixel position x _d does not exceed x _min of Qmin (step S61; NO), i.e., when the pixel position x _d is within the search range, the process proceeds to step S45, and the above processing is repeatedly And search for a coordinate position _xd where Za ≧ Zd. As described above, the color information of the stereoscopic image for one frame can be set by repeatedly executing the loop process for each pixel.

[Action / Effect]
As described above, according to the present embodiment, the pixel-specific viewpoint CM is set for each pixel PE on the display surface 22, and the color information of the object space in the line-of-sight direction of the set pixel-specific viewpoint CM is used as the color information of the pixel PE. By doing so, a stereoscopic image is generated.

  Accordingly, since the viewpoints for each pixel as many as the number of pixels PE are set, that is, as many viewpoints (views) as the number of pixels PE exist, an assumed observation position (as in conventional multi-view stereoscopic viewing) ( It is not necessary that the eyes have to be positioned at the individual viewpoint), and stereoscopic viewing is possible at any position within a certain area as in the conventional super multi-view system.

  Further, the pixel-specific viewpoint CM of each pixel PE determines the lens 32 corresponding to the pixel PE based on the assumed observation position 40, and represents the representative point of the pixel PE and the principal point of the lens 32 corresponding to the pixel PE. It is determined that the reverse direction of the light beam (representative light beam) PR after passing through is the line-of-sight direction. Therefore, the line-of-sight direction of the pixel-specific viewpoint CM corresponding to each pixel PE visually recognized by the observer through the lens 32 is substantially along the observer's line-of-sight direction. The color information) is not an accurate image (color information) viewed from the position, but is clear enough to be sufficiently visually recognized.

  Further, since the resolution of the image is a resolution that can be separated by the lens plate 30, the resolution is comparable to that of the conventional multi-view stereoscopic view, and the remarkably degraded resolution as in the conventional super multi-view method is not observed. Does not occur.

  Furthermore, the present embodiment generates a stereoscopic image to be displayed on a stereoscopic video display device that does not match the pitch. In other words, stereoscopic viewing is possible even with a stereoscopic video display device that does not match the pitch. Therefore, there is no need to manufacture a lenticular lens plate having the same pitch for each display, and a stereoscopic image display device can be manufactured by applying one lenticular lens plate to another display having a different pixel pitch. The cost for manufacturing the device can be greatly reduced.

  Further, in the present embodiment, when a stereoscopic image is generated from the planar image and the depth image, the intersection of the pixel-specific line-of-sight direction locus corresponding to the pixel PE on the display surface 22 and the depth information model obtained from the depth image. High speed judgment is realized. In other words, when the intersection determination between the depth-of-depth information model and the per-pixel gaze direction trajectory based on the line-of-sight V and the per-pixel viewpoint with respect to the pixel PE is performed, the depth value (Z value) of the depth information model is Thus, by limiting the search range for performing the intersection determination, it is possible to reduce the processing load of the intersection determination and realize the color information setting process at high speed.

  When the inclination of the line-of-sight line direction trajectory is negative, Za is sequentially obtained from the leftmost pixel position located closest to the pixel-specific viewpoint among the pixels included in the search range, and is compared with the corresponding Zd. . On the other hand, when the inclination of the line-of-sight line direction trajectory is positive, Za is determined in order from the rightmost pixel position located closest to the pixel-specific viewpoint among the pixels included in the search range, and compared with the corresponding Zd. When the comparison result is Zd ≧ Za, the pixel position is the position to be obtained, and color information of the dot corresponding to the pixel position is acquired from the planar view image, and this is used as the color information of the pixel PE. be able to.

  Thereby, the intersection determination can be performed by a simple calculation such as calculation of Za corresponding to each pixel position included in the search range and comparison between Za and Zd. Specifically, by sequentially performing the intersection determination for each pixel from the Qmax side, Za can be calculated only by addition processing (for example, addition of dz / dx), and the calculation processing can be speeded up. . Further, by sequentially performing the intersection determination from the Qmax side, it is possible to quickly search for an intersection where the Z value becomes the largest. Thereby, the number of calculations can be minimized and the target intersection can be searched efficiently. As a result, redundant processing can be greatly reduced, and a stereoscopic image can be generated at high speed.

[Modification]
The application of the present invention is not limited to the above-described embodiment, and can be changed as appropriate without departing from the spirit of the present invention. For example, the following modification is mentioned.

(A) Color Image In the above-described embodiment, as an example of generating a stereoscopic image from a planar image and its depth image, the case where the planar image is a grayscale image has been described. However, it is not limited to this. For example, the planar image may be a color image having RGB values (the luminance values of the three primary colors of red, green, and blue). In this case, for each subpixel, the line of sight V is determined to set a pixel-specific viewpoint, Intersection determination between the line-of-sight line direction trajectory for each subpixel and the depth information model is performed. Then, the target intersection for each subpixel is acquired, and the color information of the subpixel corresponding to the target intersection from the planar view image is used as the color information of the subpixel. Therefore, in this case, the depth image data preferably has a depth value for each subpixel. Of course, a stereoscopic image can be generated (although the image quality is slightly reduced) even if each subpixel has a format in which depth data is shared for each pixel to which the subpixel belongs.

(B) Oblique Lenticular Stereoscopic Video Display Device In the above-described embodiment, the case of a stereoscopic video display device using a vertical lenticular method as the optical element group has been described. You may apply to an apparatus.

  FIG. 34 is a diagram showing a schematic structure of an oblique lenticular stereoscopic display device 200B. FIG. 4A shows a cross-sectional view in the horizontal direction (horizontal scanning direction) with respect to the display surface of the stereoscopic video display apparatus 200B, and FIG. 4B shows a plan view seen from the observer side.

  In the oblique lenticular stereoscopic image display device, the direction of the principal dotted line 36 of the lens plate 30 (the end portion 32a of the lens 32) is arranged obliquely with respect to the pixel arrangement direction of the pixel panel 20, and thus the lens plate 30 is provided. It is known to disperse moire generated in an image that is visually recognized when the pixel panel 20 is viewed through the pixel panel 20.

  As shown in the drawing, the stereoscopic video display device 200B is mainly composed of a backlight 10 that is a plate-like body and arranged in parallel to each other, and the pixels, like the vertical lenticular stereoscopic video display device 200A described above. A panel 20 and a lens plate 30 are provided.

In the oblique lenticular stereoscopic image display device 200B, the lens plate 30 has a pixel array direction (vertical scanning direction) in which the direction of the main dotted line 36 is the vertical direction of the pixel panel 20, as shown in FIG. With respect to the angle θ. Accordingly, the lens pitch (lens width along the pixel pitch direction of the pixel panel 20) M in the cross-sectional view shown in FIG.
M = L / cosθ (10)

The lens plate 30 is arranged so that the lens pitch M and the pixel pitch S of the pixel panel 20 in the cross-sectional view shown in FIG. That is, the following equation (11) is not satisfied.
M = n · S (11)
However, n is a natural number.

  Next, a method of determining the line of sight V of each pixel PE on the display surface 22 in the oblique lenticular method will be described with reference to FIGS. FIG. 35 is a partial schematic perspective view of an oblique lenticular stereoscopic image display apparatus 200B. FIG. 36 is a schematic three-view diagram of the stereoscopic video display device 200B. FIG. 36 (a) is a cross-sectional view (transverse cross-sectional view) taken along the line CC ′ parallel to the xz plane of FIG. 36 (b) shows a cross-sectional view (longitudinal cross-sectional view) at a DD ′ position parallel to the yz plane of FIG. 35, and FIG. 36 (c) shows an xy plane. The figure is shown. The pixel panel 20 and the lens plate 30 are arranged in parallel with a focal length F of each lens 32 of the lens plate 30.

  First, in FIG. 36B, the y coordinate of the intersection point between the straight line LN2 connecting the representative point of the target pixel PE and the assumed observation position 40 and the principal point surface 35 of the lens plate 30 is calculated. The calculated y coordinate is assumed to be “y2”. However, FIG. 4B is a cross-sectional view passing through the representative point of the target pixel PE.

  Next, in FIG. 36A, each lens 32 is projected onto the display surface 22 from the assumed observation position 40, and the display surface 22 is divided into projection areas of the respective lenses 32. Then, the corresponding lens 32 is determined depending on which projection region the representative point of the target pixel PE belongs to. However, FIG. 6A is a cross-sectional view in which the y coordinate is “y2” calculated previously.

  In FIG. 6A, the display surface 22 includes a projection area 26-10 of the lens 32-10, a projection area 26-11 of the lens 32-11, a projection lens 32-12 of the lens 32-12,.・ It is divided into Since the representative point of the target pixel PE belongs to the projection area 26-10, the lens 32 corresponding to the target pixel PE is the lens 32-10.

  Subsequently, a point whose y coordinate is “y2” is calculated from the main dotted line 36 of the lens 32 corresponding to the target pixel PE, and this is set as a representative main point 36d. Then, the representative ray PR after passing through the representative point of the target pixel PE and the representative principal point 36d is calculated, and the line of sight V having the same position and the opposite direction as the representative ray PR is the line of sight V of the target pixel PE. And Then, by performing intersection determination with the depth information model based on the determined line of sight V, color information for each pixel is set, and stereoscopic image data is generated.

(C) Lens plate In the above-described embodiment, the case of a stereoscopic video display device using a lenticular lens array as the optical element group has been described. A stereoscopic video display device using (C-2) a stereoscopic video display device using a parallax barrier array, or (C-3) a stereoscopic video display device using a pinhole array. Also good. In this case, the line of sight V of each pixel PE on the display surface 22 is determined as follows.

(C-1) Eye-eye lens array The eye-eye lens array, as shown in FIG. 37, is a lens array (lens plate) in which lattice-like unit lenses are continuously arranged (connected) vertically and horizontally. That is. In the stereoscopic image display device using the eyelid lens array, the eyelet lens array is arranged so that the lateral connection direction of the unit lenses is parallel to the pixel pitch direction (horizontal scanning direction) of the pixel panel 20. Is done. At this time, the eyelet lens array is designed so that the lens pitch of the unit lens and the pixel pitch of the pixel panel 20 do not match. That is, if the lens pitch of the unit lens is L, Expression (9) does not hold.

  FIG. 38 is a diagram for explaining a method of determining the line of sight V in the stereoscopic video display apparatus 200C using the eyelet lens array, and shows a schematic three-view diagram of the stereoscopic video display apparatus 200C. FIG. 4A shows a horizontal scanning direction sectional view parallel to the xz plane passing through the representative point of the target pixel PE, and FIG. 4B shows the yz plane passing through the representative point of the target pixel PE. A parallel sectional view in the vertical scanning direction is shown, and FIG. 8C shows an xy plan view. The assumed observation position 40 is “front”.

  First, as shown in FIGS. 2A and 2B, from the assumed observation position 40, each unit lens 62 of the lens plate 60 realized by the eyelet lens array is projected onto the display surface 22 of the pixel panel 20. (Ie, by a straight line from the assumed observation position 40 toward the end of each unit lens 62), the display surface 22 is divided into projection areas of each unit lens 62. The unit lens 62-1 in the projection area to which the representative point of the target pixel PE belongs is set as a unit lens corresponding to the target pixel PE. Then, the light beam (representative light beam) PR after passing through the representative point of the target pixel PE and the principal point (center of the unit lens) of the unit lens 62-1 corresponding to the target pixel PE has the same position and reverse direction. The line of sight that is set as the line of sight V of the target pixel PE.

  In FIG. 39, the lens plate 60 is arranged so that the lateral connecting direction of the unit lenses 62 is parallel to the pixel pitch direction of the pixel panel 20. As shown in FIG. The horizontal connecting direction of the eyelet lens array and the pixel pitch direction of the pixel panel 20 may be arranged obliquely so as to form an angle θ.

  Further, the unit lens constituting the eyelet lens may be a triangle or a polygon such as a hexagon shown in FIG. 40 instead of the lattice shape (square). In any case, the method of determining the line of sight V of each pixel PE on the display surface 22 is the same as that shown in FIG.

(C-2) Parallax Barrier Array As shown in FIG. 41, a parallax barrier array has a slit-shaped barrier opening (optical element) for transmitting light to a light shielding plate (barrier) that blocks light. Many are provided at equal intervals. That is, it is a barrier array in which unit parallax barriers having slit-like barrier openings are connected, and directivity is given to light rays (emitted light rays) emitted from each pixel PE of the pixel panel 20 by the barrier openings. As a result, it is possible to recognize a stereoscopic image in the same manner as a stereoscopic image display device using a lenticular lens array. In the figure, a black or gray portion is a barrier portion (light-shielding portion). At this time, the parallax barrier array is arranged so that the barrier opening is parallel to the vertical scanning direction of the pixel panel 20. Further, the parallax barrier array is designed so that the above-described formula (9) is not satisfied (that is, the pitch does not match) when the unit parallax barrier pitch is L.

  Therefore, in this case, the line of sight V of each pixel PE can be determined in the same manner as in the lenticular lens array in the above-described embodiment. Specifically, when the parallax barrier array is arranged so that the barrier opening is parallel to the vertical scanning direction of the pixel panel 20, the vertical lenticular stereoscopic image display device 200 in the above-described embodiment is used. Equivalent to. Further, when the parallax barrier array is arranged obliquely so that the barrier opening forms an angle θ with respect to the vertical scanning direction of the pixel panel 20, it corresponds to the oblique lenticular stereoscopic image display device 200B.

  Further, the parallax barrier array may be an oblique parallax barrier in which barrier openings are formed obliquely as shown in FIG. When such an oblique parallax barrier is used, it corresponds to the case where the above-described parallax barrier array (see FIG. 41) is arranged obliquely with respect to the pixel panel 20.

  In addition, as shown in FIG. 43, the parallax barrier array may be a stepped parallax barrier array in which a plurality of unit parallax barriers connected in the horizontal direction are arranged in a staircase pattern with a predetermined amount shifted in the vertical direction. good. In the stereoscopic video display device using such a staircase parallax barrier, the staircase parallax barrier is arranged so that the barrier opening is parallel to the vertical scanning direction of the pixel panel 20.

(C-3) Pinhole Array As shown in FIG. 44, the pinhole array is provided with a large number of hole-shaped pinholes (optical elements) that allow light to pass through a blocking plate (barrier) that blocks light. It is what was done. That is, it is a barrier array in which lattice-shaped unit pinhole barriers having pinholes are continuously arranged (connected) vertically and horizontally, and directivity is given to the emitted light of each pixel PE of the pixel panel 20 by the pinhole. . In the figure, the gray portion is a light shielding portion (barrier portion). At this time, the pinhole array is arranged so that the horizontal direction of the unit pinhole barrier is parallel to the pixel pitch direction (horizontal scanning direction) of the pixel panel 20. Further, the pinhole array is designed so that equation (9) does not hold when the pitch of the unit pinhole barrier is L.

  In the stereoscopic video display apparatus using the pinhole array, the line of sight V of each pixel PE of the pixel panel 20 is determined as follows. That is, each unit pinhole barrier of the pinhole array is projected onto the display surface 22 of the pixel panel 20 from the assumed observation position 40, and the display surface 22 is divided into projection regions of each unit pinhole barrier. A unit pinhole barrier in the projection area to which the representative point of each pixel PE belongs is set as a unit pinhole barrier corresponding to the pixel PE. For each pixel PE, the line of sight is the same as the representative point of the pixel PE and the light ray (representative ray) after passing through the pinhole of the unit pinhole barrier corresponding to the pixel PE, and the direction is reversed. The line of sight V of the pixel PE is assumed.

  Further, the unit pinhole barrier constituting the pinhole array may be a triangle or a polygon such as a hexagon shown in FIG. 45 instead of the lattice shape (square). In any case, the method of determining the line of sight V of each pixel PE is the same as in the case of the lattice shape described above.

  The parallax barrier array or pinhole array usually does not clearly indicate the boundary line indicating the unit barrier. However, the slit or pinhole closest to any point on the barrier array is obtained, and the slit or pinhole is obtained. A boundary line for each unit barrier can be set by dividing the barrier plate into regions by holes.

(D) The stereoscopic video display device is a separate device. In the above-described embodiment, the stereoscopic video display device 200 has been described as a part of the stereoscopic image generation device 1, but the stereoscopic video display device 200 is used. It is good also as comprising as another apparatus and outputting the produced | generated stereoscopic vision image to the connected stereoscopic vision video display apparatus 200, and displaying it.

  FIG. 46 is a block diagram illustrating a configuration of the stereoscopic image generation device 3 in which the stereoscopic video display device is a separate device. As shown in the figure, the stereoscopic image generation device 3 includes an input unit 100, a processing unit 300, and a storage unit 400. In the stereoscopic image generation device 3, it is necessary to set the display device data 420 according to the connected stereoscopic video display device 200.

  Therefore, when generating an image, the stereoscopic image generation unit 320 firstly includes, for example, the model number of the stereoscopic video display device 200 input from the input unit 100 by the user (specifically, the pixel image is arranged on the pixel panel 20 or the upper surface thereof) Appropriate data is selected from each of the pixel panel data 490 and the lens plate data 510 stored in the storage unit 400 based on the model number of the lens plate 30 being set, and set as the display device data 420.

  The pixel panel data 490 is a data table that stores physical parameters of the pixel panel 20 that can be used in the stereoscopic video display device 200. FIG. 47A shows an example of the data configuration of the pixel panel data 490. According to FIG. 5A, the pixel panel data 490 corresponds to the model number (491) and the pixel pitch (492) for each type (type) of the pixel panel 20 that can be used in the stereoscopic video display device 200. It is attached and stored.

  The lens plate data 510 is a data table that stores physical parameters of the lens plate 30 that can be used in the stereoscopic video display device 200. FIG. 47B shows an example of the data configuration of the lens plate data 510. As shown in FIG. 5B, the lens plate data 510 includes, for each type (type) of the lens plate 30 that can be used in the stereoscopic video display device 200, a model number (511), a lens pitch (512), The focal distance (513) and the arrangement angle (514) are stored in association with each other.

  The input unit 100 may be a separate device from the stereoscopic image generation device. For example, as shown in FIG. 48, the input unit 100 and the stereoscopic video display device 200 are separate from the stereoscopic video generation device. That is, the stereoscopic image generation apparatus 5 includes a processing unit 300 and a storage unit 400. As such a configuration, for example, a case where a stereoscopic image is desired to be displayed on a display of a communication terminal device such as a mobile phone can be considered. That is, the lens pitch L, the arrangement angle θ of the optical element group (lens plate 30) with respect to the display (pixel panel 20), the focal point from the external device 7 such as a communication terminal device including the input unit 100 and the stereoscopic video display device 200. Information relating to the lens such as the distance F, information relating to the display such as the pixel pitch S, and information relating to the observer such as the assumed observation position 40 are transmitted to the image generating device 5. Then, the image generation device 5 generates a stereoscopic image based on the received information and transmits the generated stereoscopic image to the external device 7 or stores and stores the generated image as a moving image. Transmit to device 7.

(E) Printed Product In the above-described embodiment, the case of generating a stereoscopic image to be displayed on a stereoscopic video display device including the pixel panel 20 such as a flat panel display has been described. The present invention can be similarly applied to a printed material using a printed material such as paper or a plastic card on which a stereoscopic image is printed. In such a printed product, an optical element group (lens array, pinhole array, etc.) is attached to the printing surface of the printed material on which the stereoscopic image is printed, and reflection of each dot of the printed image is performed. Stereopsis is realized by the directivity of the light beams provided by the optical element group.

  When generating an image to be printed on such a stereoscopic print processed product, the conventional method sets a certain number of viewpoints as in the case of a conventional multi-view stereoscopic image display device, and An image to be printed is generated by generating an image (per-pixel viewpoint) for each viewpoint and rearranging (interleaving) the images according to a certain pattern.

  In the printed product, the resolution (that is, the position and size of each dot of the image) of the stereoscopic image to be generated / printed can be freely set / changed according to the ready-made optical element. Further, by increasing the resolution of the stereoscopic image to be generated / printed, the resolution of the visually recognized stereoscopic video image can be made sufficiently large even when there are relatively many viewpoints. However, in that case, there is a problem that the data size of the stereoscopic image becomes large in proportion to the resolution.

  When the present embodiment is applied to such a printed product, each dot of the stereoscopic image to be printed is treated as each pixel PE of the pixel panel 20 so that the stereoscopic view is similar to the above-described embodiment. An image can be generated. Specifically, if the size of the print area of the stereoscopic image on the printed product and the resolution of the stereoscopic image to be printed are determined, the position and size of each dot of the stereoscopic image to be printed are unique. It is decided. For this reason, each dot of the stereoscopic image can be handled in the same manner as each pixel PE of the display panel 20 in the above-described embodiment.

  That is, for example, when using a lenticular lens array as the optical element group, first, a lens (optical element) corresponding to each dot of the stereoscopic image is determined. Next, for each dot, a ray (representative ray PR) after passing through the representative point (for example, the center) of the dot and the principal point of the lens corresponding to the dot is calculated, and the position of the ray is the same. A line of sight in the opposite direction is determined as the line of sight V of the dot. Thereafter, for each dot, the intersection determination between the line-of-sight line direction trajectory and the depth information model is performed to obtain the target intersection, and the color information of the pixel corresponding to the target intersection from the planar view image is used as the color information of the dot. Thus, a stereoscopic image is generated.

  Further, when the printed product uses a lens array as the optical element group, the pitch between each dot of the printed stereoscopic image is S, and the distance between the print surface and the principal point surface of the lens array is Is configured so that the distance G satisfies the above equation (1), it is possible to realize stereoscopic viewing with better image quality.

  According to the drawing method of the present embodiment, a stereoscopic image to be printed on a stereoscopic print processed product having the above-described effects is generated with a resolution lower than that of the conventional method, that is, with an image data size smaller than that of the conventional method. can do.

(F) Depth Value Information In this embodiment, a case has been described in which a depth information model is generated from a depth image as depth value information, and color information of a stereoscopic image is calculated based on the generated depth information model. It is not limited. For example, only the depth information model 470 may be stored without storing the depth image data 464 in the storage unit 400. In this embodiment, an example in which a stereoscopic image is generated based on a depth image generated by a parallel projection method has been described. However, the present invention is not limited thereto, and a stereoscopic image is generated based on a depth image generated by a perspective projection method. The structure to produce | generate may be sufficient.

(G) When obtaining the pixel value of the stereoscopic image from a plurality of pixel values in the planar image In the embodiment described above, the pixel value of the stereoscopic image is obtained from the single pixel value of the planar image. Alternatively, it may be obtained from an average of a plurality of pixel values or a weighted average. For example, employed in FIG. 23, when determining the values of the pixels PE, in the embodiment described above, the closest to the intersection position of the x ₂ (X1 in FIG. 20), the values of the pixels of the corresponding plan view image thereto but was actually intersection is positioned between the x ₁ and x _2. In this case, in the plan view image, the average value of the values of two pixels corresponding to the position of x ₁ and x _2, may determine the value of the pixel PE. In addition, the value of the pixel PE may be determined by a weighted average according to the distance in the x direction from the both to the intersection in consideration of whether the intersection is closer to x1 or x2 instead of a simple average. When these processes are performed, the image quality is improved, but the calculation amount naturally increases. In practice, it is preferable to select a method for obtaining the pixel color in consideration of the required image quality and processing speed.

The schematic diagram of the stereoscopic vision image generation in an embodiment. 1 is a schematic configuration diagram of a vertical lenticular stereoscopic image display device. Explanatory drawing of "pitch does not fit / does not fit". Explanatory drawing of the coordinate system setting with respect to a display surface. Explanatory drawing of the gaze determination in the assumption observing position "front and infinity" in the vertical lenticular system stereoscopic image display device. The figure which shows the state whose assumption observation position is a "front". Explanatory drawing of the gaze determination in the vertical lenticular system stereoscopic vision video display apparatus in case the assumption observation position is "the fixed position of a finite distance." Explanatory drawing of the gaze determination in the vertical lenticular system stereoscopic vision video display apparatus in case the assumption observation position is "the fixed position of a finite distance." Explanatory drawing of the refractive action by a lens plate. Explanatory drawing of the viewpoint setting according to pixel. Explanatory drawing of color information calculation of a pixel. Explanatory drawing that the color information of the visually recognized pixel is slightly inaccurate. The schematic diagram of the stereoscopic view of the conventional multi-view method (n eye type). The schematic diagram of the stereoscopic view in this embodiment. Explanatory drawing which produces | generates a binocular stereoscopic vision image from a planar view image and a depth image. Explanatory drawing of color information calculation of a pixel. Explanatory drawing of the color information calculation of the pixel based on a planar view image and a depth image. Explanatory drawing of the relationship between a display surface and a depth information model. Explanatory drawing of the relationship between a display surface, each pixel, and depth information model. Explanatory drawing of the intersection of a depth information model and a gaze direction locus according to pixel. Explanatory drawing of the maximum value and minimum value of a depth information model and a depth value. Explanatory drawing of the intersection of the gaze direction locus | trajectory classified by pixel, and the maximum value and minimum value of depth value. Explanatory drawing of the intersection determination of a depth information model and a gaze direction locus according to pixel. Explanatory drawing of the intersection determination of a depth information model and a gaze direction locus according to pixel. Explanatory drawing of the intersection determination of a depth information model and a gaze direction locus according to pixel. 1 is a configuration diagram of a stereoscopic image generation apparatus. The data structural example of display apparatus data. The data structural example of viewpoint data classified by pixel. The figure which shows an example of planar view image data. The figure which shows an example of depth image data. The figure which shows an example of a depth information model. The flowchart which shows the flow of a stereoscopic vision image generation process. 6 is a flowchart showing a flow of color information setting processing. 1 is a schematic configuration diagram of an oblique lenticular stereoscopic image display device. FIG. FIG. 4 is an explanatory diagram of line-of-sight determination when the assumed observation position is “a fixed position at a finite distance” in an oblique lenticular stereoscopic image display device. FIG. 4 is an explanatory diagram of line-of-sight determination when the assumed observation position is “a fixed position at a finite distance” in an oblique lenticular stereoscopic image display device. The top view of an eye lens array of an eyelid. Explanatory drawing of the gaze determination in the stereoscopic vision video display apparatus using a moth-eye lens array. The top view at the time of arranging the eye lens array of an eyelid diagonally. The top view of the eye lens array of the eyelid which made the unit lens hexagon. The top view of a parallax barrier. The top view of a diagonal parallax barrier. The top view of a staircase parallax barrier. The top view of a pinhole barrier. The top view of the pinhole barrier which made the unit pinhole barrier hexagonal. The block diagram of the stereoscopic image production | generation apparatus which used the stereoscopic vision video display apparatus as another apparatus. 6 is a data configuration example of pixel panel data and lens plate data. The block diagram of the stereoscopic image production | generation apparatus which used the input part and the stereoscopic vision video display apparatus as another apparatus (external device). Explanatory drawing of the distance between a suitable pixel panel and a lens board. Explanatory drawing at the time of considering the binocular distance. The conceptual diagram of multi-view type stereoscopic vision. The conceptual diagram of super multi-view type stereoscopic vision. FIG. 3 is a conceptual diagram of IP system stereoscopic vision. The conceptual diagram of the stereoscopic vision of the ray reproduction method. Explanatory drawing of the distance between a display surface and an image formation surface being dependent on the focal distance of a lens.

Explanation of symbols

1, 3, 5 Stereoscopic image generation device 100 Input unit 300 Processing unit 310 Object space setting unit 320 Stereoscopic image generation unit 322 Per-pixel viewpoint setting unit 324 Rendering unit 324 Color information setting unit 400 Storage unit 410 Stereoscopic image generation program 420 Color information setting processing program 430 Display device data 440 Assumed observation position data 450 Pixel-specific viewpoint data 460 Rendering data 462 Plane view image data 464 Depth image data 470 Depth information model 480 Stereoscopic image data 200 Stereoscopic image display device 10 Backlight 20 pixel panel 22 display surface EL pixel 30 lens plate (lenticular lens plate)
32 Lens (micro lens)
40 Assumed observation position PE Pixel V Line-of-sight CM Per-pixel viewpoint

Claims (20)

A pixel panel on which pixels are arranged, and an optical element group that gives directivity to the emitted light of each pixel of the pixel panel, and the viewing angle with respect to one optical element at a predetermined assumed observation position is the viewing angle λ with respect to the optical element. When the viewing angle with respect to one pixel to which directivity is given by the optical element is a counter-pixel viewing angle σ, and the viewing angle with respect to the stereoscopic image rendering region in the pixel panel is the pair-drawing region viewing angle Φ, λ: σ = n: m ( Where n is a natural number, m is a natural number less than Φ / λ), and a stereoscopic image of the three-dimensional virtual space displayed on the stereoscopic image display device is a plane obtained by viewing the three-dimensional virtual space from a given viewpoint. An image generation device that generates a reference image that is a visual image and depth value information of each dot of the reference image that is based on the viewpoint position,
Regarding the color information of each pixel in the stereoscopic image drawing area, 1) the representative point of the pixel, the direction of the light beam passing through the representative point of the optical element that gives directivity to the emitted light beam of the pixel, and the three-dimensional virtual A direction calculating step for obtaining a direction corresponding to the direction of the light beam in the three-dimensional virtual space based on the viewpoint position in space; and 2) based on the direction obtained in the direction calculating step and the depth value information. , A dot determination step of determining, from the reference image, a dot representing color information of the closest object in the direction in the three-dimensional virtual space, and the reference determined in the dot determination step An image generation apparatus that generates a stereoscopic image by using color information of a dot of an image as color information of the pixel. A pixel panel on which pixels are arranged, and an optical element group that gives directivity to the emitted light of each pixel of the pixel panel, and the viewing angle with respect to one optical element at a predetermined assumed observation position is the viewing angle λ with respect to the optical element. When the viewing angle for one pixel to which directivity is given by the optical element is a counter-pixel viewing angle σ and the viewing angle for a stereoscopic image rendering area in the pixel panel is a pair-drawing area viewing angle Φ, λ: σ = n: m ( Where n is a natural number, m is a natural number less than Φ / λ), and a stereoscopic image of the three-dimensional virtual space displayed on the stereoscopic image display device is a plane obtained by viewing the three-dimensional virtual space from a given viewpoint. An image generation device that generates a reference image that is a visual image and depth value information of each dot of the reference image that is based on the viewpoint position,
Regarding the color information of each pixel in the stereoscopic image drawing area, 1) the representative point of the pixel, the direction of the light beam passing through the representative point of the optical element that gives directivity to the emitted light beam of the pixel, and the three-dimensional virtual A direction calculating step for obtaining a direction corresponding to the direction of the light beam in the three-dimensional virtual space based on the viewpoint position in the space; and 2) a trajectory of the direction obtained in the direction calculating step on the reference image. Dots for detecting a dot group between a dot having a maximum depth value and a dot having a minimum depth value based on the depth value information among the dots in the reference image that will draw the locus when projected and drawn A group detection step, and 3) based on the direction obtained in the direction calculation step and the depth value of each dot of the dot group detected in the dot group detection step, from the dot group, in the three-dimensional virtual space of A dot determination step of determining a dot representing the color information of the closest object in the direction, and using the color information of the dot determined in the dot determination step as the color information of the pixel, An image generation device that generates a visual image. A stereoscopic image in a three-dimensional virtual space to be displayed on a stereoscopic video display device comprising a pixel panel in which pixels are arranged and an optical element group that directs the emitted light of each pixel of the pixel panel. An image generation device that generates a reference image that is a planar image when the virtual space is viewed from a given viewpoint, and depth value information of each dot of the reference image that is based on the viewpoint position,
Regarding the color information of each pixel in the stereoscopic image drawing region, 1) the direction of the light beam passing through the pixel and the optical element that gives directivity to the emitted light beam of the pixel, and the viewpoint position in the three-dimensional virtual space A direction calculating step for obtaining a direction corresponding to the direction of the light beam in the three-dimensional virtual space, and 2) projecting and drawing a locus of the direction obtained in the direction calculating step on the reference image A dot group detection step of detecting a dot group between a dot having a maximum depth value and a dot having a minimum depth value among the dots in the reference image that will draw the locus on the basis of the depth value information; ) Based on the direction obtained in the direction calculation step and the depth value of each dot of the dot group detected in the dot group detection step, from the dot group, in the direction in the three-dimensional virtual space A dot determination step of determining a dot representing color information of an object in front, and using the color information of the dot determined in the dot determination step as the color information of the pixel, An image generation device to generate.   The dot determination step is based on a) the depth value of each dot of the dot group detected in the dot group detection step, and b) the viewpoint position when the direction obtained in the direction calculation step is a trajectory. The depth value in b) is compared with the depth value of the position corresponding to the dot on the locus in order from the dot on the near side, and based on whether the context of the two depth values to be compared is switched. The image generating apparatus according to claim 3, wherein the image generating apparatus is a step of determining a dot representing color information of a foremost object in the direction in the three-dimensional virtual space.   In the dot determination step, when the contexts of the two depth values to be compared are not interchanged, the depth value of b) is a dot adjacent to the opposite side of the dot group of the end dot on the farthest side, The image generating apparatus according to claim 4, wherein the image generating apparatus is a step of determining as a dot representing color information of the closest object in the direction in the three-dimensional virtual space. An image generation apparatus according to any one of claims 1 to 5,
The optical element group is an image generation device that is a lenticular lens array, a moth-eye lens array, a parallax barrier array, or a pinhole array. An image generation apparatus according to any one of claims 1 to 5,
The optical element group is a lenticular lens array or an eyelet lens array in which a plurality of lenses are connected as optical elements,
The optical element group is installed between the pixel panel and an observer so that the distance between the display surface of the pixel panel and the principal point plane of the optical element group matches the focal length of the lens. Image generating device. An image generation apparatus according to any one of claims 1 to 5,
The optical element group is a lenticular lens array or an eyelet lens array in which a plurality of lenses are connected as optical elements,
The optical element group includes a distance D between a display surface of the pixel panel and a principal point surface of the optical element group, a focal length of the lens F, a pitch of the pixel in a horizontal scanning direction S, and the lens Is set between the pixel panel and the viewer so that (LS) · F / L ≦ D ≦ (L + S) · F / L is satisfied, where L is the horizontal scanning direction pitch. Yes,
An image generation apparatus characterized by that. The image generation apparatus according to any one of claims 6 to 8,
The image generation apparatus, wherein the optical element group is arranged so that an optical element arrangement direction is oblique with respect to a pixel arrangement direction of the pixel panel. A pixel panel in which pixels are arranged;
An optical element group that gives directivity to the emitted light of each pixel of the pixel panel;
The image generation device according to any one of claims 1 to 9, which generates a stereoscopic image of a three-dimensional virtual space to be displayed on the pixel panel;
Electronic equipment comprising. A pixel panel on which pixels are arranged, and an optical element group that gives directivity to the emitted light of each pixel of the pixel panel, and the viewing angle with respect to one optical element at a predetermined assumed observation position is the viewing angle λ with respect to the optical element. When the viewing angle with respect to one pixel to which directivity is given by the optical element is a counter-pixel viewing angle σ, and the viewing angle with respect to the stereoscopic image rendering region in the pixel panel is the pair-drawing region viewing angle Φ, λ: σ = n: m ( Where n is a natural number, m is a natural number less than Φ / λ), and a stereoscopic image of the three-dimensional virtual space displayed on the stereoscopic image display device is a plane obtained by viewing the three-dimensional virtual space from a given viewpoint. An image generation method for generating based on a reference image that is a visual image and depth value information of each dot of the reference image with respect to the viewpoint position,
Regarding the color information of each pixel in the stereoscopic image drawing region, 1) the representative point of the pixel, the direction of the light beam passing through the representative point of the optical element that gives directivity to the emitted light beam of the pixel, and the three-dimensional virtual A direction calculating step for obtaining a direction corresponding to the direction of the light beam in the three-dimensional virtual space based on the viewpoint position in space; and 2) based on the direction obtained in the direction calculating step and the depth value information. A dot determination step of determining, from the reference image, a dot representing color information of the foremost object in the direction in the three-dimensional virtual space, and the reference image determined in the dot determination step Generation method for generating a stereoscopic image by using the color information of the pixel as the color information of the pixel. A pixel panel on which pixels are arranged, and an optical element group that gives directivity to the emitted light of each pixel of the pixel panel, and the viewing angle with respect to one optical element at a predetermined assumed observation position is the viewing angle λ with respect to the optical element. When the viewing angle for one pixel to which directivity is given by the optical element is a counter-pixel viewing angle σ and the viewing angle for a stereoscopic image rendering area in the pixel panel is a pair-drawing area viewing angle Φ, λ: σ = n: m ( Where n is a natural number, m is a natural number less than Φ / λ), and a stereoscopic image of the three-dimensional virtual space displayed on the stereoscopic image display device is a plane obtained by viewing the three-dimensional virtual space from a given viewpoint. An image generation method for generating based on a reference image that is a visual image and depth value information of each dot of the reference image with respect to the viewpoint position,
Regarding the color information of each pixel in the stereoscopic image drawing area, 1) the representative point of the pixel, the direction of the light beam passing through the representative point of the optical element that gives directivity to the emitted light beam of the pixel, and the three-dimensional virtual A direction calculating step for obtaining a direction corresponding to the direction of the light beam in the three-dimensional virtual space based on the viewpoint position in the space; and 2) a trajectory of the direction obtained in the direction calculating step on the reference image. Dots for detecting a dot group between a dot having a maximum depth value and a dot having a minimum depth value based on the depth value information among the dots in the reference image that will draw the locus when projected and drawn A group detection step, and 3) based on the direction obtained in the direction calculation step and the depth value of each dot of the dot group detected in the dot group detection step, from the dot group, in the three-dimensional virtual space of A dot determination step of determining a dot representing the color information of the closest object in the direction, and using the color information of the dot determined in the dot determination step as the color information of the pixel, An image generation method for generating an image. A stereoscopic image in a three-dimensional virtual space to be displayed on a stereoscopic video display device comprising a pixel panel in which pixels are arranged and an optical element group that directs the emitted light of each pixel of the pixel panel. An image generation method for generating based on a reference image that is a planar image when the virtual space is viewed from a given viewpoint, and depth value information of each dot of the reference image based on the viewpoint position,
Regarding the color information of each pixel in the stereoscopic image drawing region, 1) the direction of the light beam passing through the pixel and the optical element that gives directivity to the emitted light beam of the pixel, and the viewpoint position in the three-dimensional virtual space A direction calculating step for obtaining a direction corresponding to the direction of the light beam in the three-dimensional virtual space, and 2) projecting and drawing a locus of the direction obtained in the direction calculating step on the reference image A dot group detection step of detecting a dot group between a dot having a maximum depth value and a dot having a minimum depth value among the dots in the reference image that will draw the locus on the basis of the depth value information; ) Based on the direction obtained in the direction calculation step and the depth value of each dot of the dot group detected in the dot group detection step, from the dot group, in the direction in the three-dimensional virtual space And a dot determination step for determining a dot representing the color information of the object in the foreground, and generating a stereoscopic image by using the color information of the dot determined in the dot determination step as the color information of the pixel. Image generation method.   The dot determination step is based on a) the depth value of each dot of the dot group detected in the dot group detection step, and b) the viewpoint position when the direction obtained in the direction calculation step is a trajectory. The depth value in b) is compared with the depth value of the position corresponding to the dot on the locus in order from the dot on the near side, and based on whether the context of the two depth values to be compared is switched. The image generation method according to claim 13, wherein the image generation method is a step of determining a dot representing the color information of the closest object in the direction in the three-dimensional virtual space.   In the dot determination step, when the contexts of the two depth values to be compared are not interchanged, the depth value of b) is a dot adjacent to the opposite side of the dot group of the end dot on the farthest side, The image generation method according to claim 14, wherein the image generation method is a step of determining the dot representing the color information of the closest object in the direction in the three-dimensional virtual space. A pixel panel on which pixels are arranged, and an optical element group that gives directivity to the emitted light of each pixel of the pixel panel, and the viewing angle with respect to one optical element at a predetermined assumed observation position is the viewing angle λ with respect to the optical element. When the viewing angle for one pixel to which directivity is given by the optical element is a counter-pixel viewing angle σ and the viewing angle for a stereoscopic image rendering area in the pixel panel is a pair-drawing area viewing angle Φ, λ: σ = n: m ( Where n is a natural number, m is a natural number less than Φ / λ), and a stereoscopic image of the three-dimensional virtual space displayed on the stereoscopic image display device is a plane obtained by viewing the three-dimensional virtual space from a given viewpoint. A program for causing a computer to generate a reference image that is a visual image and depth value information of each dot of the reference image with respect to the viewpoint position,
Regarding the color information of each pixel in the stereoscopic image drawing area, 1) the representative point of the pixel, the direction of the light beam passing through the representative point of the optical element that gives directivity to the emitted light beam of the pixel, and the three-dimensional virtual A direction calculating step for obtaining a direction corresponding to the direction of the light beam in the three-dimensional virtual space based on the viewpoint position in space; and 2) based on the direction obtained in the direction calculating step and the depth value information. And a dot determination step of determining, from the reference image, a dot representing color information of the foremost object in the direction in the three-dimensional virtual space, and the reference determined in the dot determination step A program for causing the computer to function as means for generating a stereoscopic image by using color information of a dot of an image as color information of the pixel. A pixel panel on which pixels are arranged, and an optical element group that gives directivity to the emitted light of each pixel of the pixel panel, and the viewing angle with respect to one optical element at a predetermined assumed observation position is the viewing angle λ with respect to the optical element. When the viewing angle for one pixel to which directivity is given by the optical element is a counter-pixel viewing angle σ and the viewing angle for a stereoscopic image rendering area in the pixel panel is a pair-drawing area viewing angle Φ, λ: σ = n: m ( Where n is a natural number, m is a natural number less than Φ / λ), and a stereoscopic image of the three-dimensional virtual space displayed on the stereoscopic image display device is a plane obtained by viewing the three-dimensional virtual space from a given viewpoint. A program for causing a computer to generate a reference image that is a visual image and depth value information of each dot of the reference image with respect to the viewpoint position,
Regarding the color information of each pixel in the stereoscopic image drawing area, 1) the representative point of the pixel, the direction of the light beam passing through the representative point of the optical element that gives directivity to the emitted light beam of the pixel, and the three-dimensional virtual A direction calculating step for obtaining a direction corresponding to the direction of the light beam in the three-dimensional virtual space based on the viewpoint position in the space; and 2) a trajectory of the direction obtained in the direction calculating step on the reference image. Dots for detecting a dot group between a dot having a maximum depth value and a dot having a minimum depth value based on the depth value information among the dots in the reference image that will draw the locus when projected and drawn A group detection step, and 3) based on the direction obtained in the direction calculation step and the depth value of each dot of the dot group detected in the dot group detection step, from the dot group, in the three-dimensional virtual space of A dot determination step of determining a dot representing the color information of the foremost object in the direction, and using the color information of the dot determined in the dot determination step as the color information of the pixel, A program for causing the computer to function as means for generating a visual image. A stereoscopic image in a three-dimensional virtual space to be displayed on a stereoscopic video display device comprising a pixel panel in which pixels are arranged and an optical element group that directs the emitted light of each pixel of the pixel panel. A program for causing a computer to generate a reference image that is a planar image when the virtual space is viewed from a given viewpoint, and depth value information of each dot of the reference image with respect to the viewpoint position,
Regarding the color information of each pixel in the stereoscopic image drawing region, 1) the direction of the light beam passing through the pixel and the optical element that gives directivity to the emitted light beam of the pixel, and the viewpoint position in the three-dimensional virtual space A direction calculating step for obtaining a direction corresponding to the direction of the light beam in the three-dimensional virtual space, and 2) projecting and drawing a locus of the direction obtained in the direction calculating step on the reference image A dot group detection step of detecting a dot group between a dot having a maximum depth value and a dot having a minimum depth value among the dots in the reference image that will draw the locus on the basis of the depth value information; ) Based on the direction obtained in the direction calculation step and the depth value of each dot of the dot group detected in the dot group detection step, from the dot group, in the direction in the three-dimensional virtual space A dot determination step of determining a dot representing color information of the object in front, and using the color information of the dot determined in the dot determination step as the color information of the pixel, A program for causing the computer to function as means for generating.   The dot determination step is based on a) the depth value of each dot of the dot group detected in the dot group detection step, and b) the viewpoint position when the direction obtained in the direction calculation step is a trajectory. The depth value in b) is compared with the depth value of the position corresponding to the dot on the locus in order from the dot on the near side, and based on whether the context of the two depth values to be compared is switched. 19. The program according to claim 18, wherein the program is a step of determining a dot representing color information of the closest object in the direction in the three-dimensional virtual space.   In the dot determination step, when the contexts of the two depth values to be compared are not interchanged, the depth value of b) is a dot adjacent to the opposite side of the dot group of the end dot on the farthest side, The program according to claim 19, wherein the program is a step of determining as a dot representing color information of the foremost object in the direction in the three-dimensional virtual space.