JP4856534B2 - Image generating apparatus, program, and information storage medium - Google Patents

Description

  The present invention provides a stereoscopic image in a three-dimensional virtual space to be displayed on a stereoscopic video display device that includes a pixel panel in which pixels are arranged and an optical element group that provides directivity to the light emitted from each pixel of the pixel panel. The present invention relates to an image generation apparatus and the like to be generated.

As conventionally known, optical element groups such as flat panel displays such as LCDs and lens arrays (for example, lenticular lens arrays and eyelet lens arrays) and barrier arrays (for example, parallax barrier arrays and pinhole arrays) By combining these, a stereoscopic video display device can be created. As the method, a multi-view method (including two eyes), a super multi-view method, an IP (Integral Photography) method, a light beam reproduction method, and the like are known, and disclosed in Non-Patent Document 1 and Non-Patent Document 2, for example. ing.
Yasuhiro Takagi, “Display of 3D Objects Combining with 64 Eyes 3D Color Display”, 3D Image Conference 2002 Proceedings, 3D Image Conference 2002 Executive Committee, July 4, 2002, p. 85-88 Onishi Akihiro, Takeda Tsutomu, Taniguchi Hideyuki, Kobayashi Tetsuro, “Three-Dimensional Video Display Using Ray Reconstruction”, 3D Image Conference 2001 Proceedings, 3D Image Conference 2001 Executive Committee, July 4, 2001, p. 173-176

  In the multi-view method, as shown in FIG. 51, light rays emitted from each pixel on the display surface on which a stereoscopic image is displayed and given directivity by an optical element group (in the figure, a lenticular lens array) It is designed to gather at a plurality of set viewpoint positions (four in the figure). The resolution depends on the pitch of the optical element, and the number of viewpoints (views) depends on the ratio between the pixel pitch and the pitch of the optical element. For this reason, the multi-view method with a small number of viewpoints has a drawback that a natural stereoscopic effect cannot be obtained because the number of viewpoints is small although the resolution at each viewpoint is relatively high.

  Therefore, the super multi-view system shown in FIG. 52 improves the multi-view system so that a natural stereoscopic effect can be obtained by setting very many viewpoints inside the parallax between the left and right eyes. . However, the super multi-view system has a disadvantage that the resolution is remarkably reduced as a result of increasing the number of viewpoints, and it is necessary to use a very high resolution pixel panel in order to obtain a satisfactory resolution. That is, the resolution and the number of viewpoints have a trade-off relationship.

  In the multi-view method and the super-multi-view method, drawing is performed (generating an image) from each of a plurality of assumed viewpoint positions. For this reason, it is necessary to design the optical element so that the pitch of the optical element and the pixel pitch exactly match.

  In the IP (Integral Photography) method and the ray reproduction method, as shown in FIGS. 53 and 54, rays emitted from each pixel and given directivity by the optical element group are collected at the sampled point group of the object. The stereoscopic view is realized by observing this from a further distant viewpoint. FIG. 53 shows the case of the IP system, and FIG. 54 shows the case of the light beam reproduction method.

  The resolution depends on the number of sampling points of the object, and the number of lines of sight at each sampling point depends on the number of rays collected at the sampling point. That is, the smaller the number of sampling points, the more rays can be collected at each sampling point. That is, it is possible to reproduce a natural three-dimensional effect, but the resolution is lowered because there are few sampling points. Further, if the number of sampling points is increased in order to increase the resolution, only a small number of light beams can be collected at each sampling point, and natural stereoscopic vision becomes impossible.

  In particular, in the IP method, as shown in FIG. 53, the position at which a natural sense of distance can be observed is limited to the image plane parallel to the drawing plane (stereoscopic image display plane), and the object at other positions can be observed. A sense of distance is observed unnaturally. On the other hand, the light beam reproduction method can form an image at a free distance as compared with the IP method.

In addition, IP systems include those using a lens array and those using a pinhole array. As shown in FIG. 53, when a lens array is used, there is a gap between the drawing surface and the imaging surface. Depends on the focal length of the lens. That is, as shown in FIG. 55, if the distance between the focal plane of the lens and the display plane is A, the distance between the focal plane of the lens and the imaging plane is B, and the focal distance of the lens is F, As is known, since there is a relationship represented by the following expression, two or more imaging positions (distances from the drawing surface) cannot be set at the same time.
(1 / A) + (1 / B) = (1 / F)

  On the other hand, as shown in FIG. 54, the light beam reproduction method uses a pinhole array instead of a lens array, so that two or more imaging distances can be set simultaneously (two in the figure). Since the pinhole array is used, there is a drawback that the screen is dark and the image is like a sequence of dots.

  Further, in the IP method and the light beam reproducing method, since it is necessary to collect a very large number of light beams in principle, sampling points are generally sparse, that is, those having low resolution. That is, in order to obtain a satisfactory resolution, it is necessary to use a pixel panel with a very high resolution, as in the super multi-view system. That is, although the scale is different from the multi-view method and the super multi-view method, it can be said that the resolution and the number of viewpoints have a trade-off relationship.

  In the IP method and the light beam reproduction method, the position and direction of the line of sight (viewpoint) are determined based on the positional relationship between each imaging position and each optical element or the positional relationship between each imaging position and each pixel. Specifically, when the optical element group is prepared first and the line of sight is determined by the positional relationship between the imaging position and each optical element, the pixel arrangement must be determined according to the optical element group, and When the pixel panel is prepared in advance and the line of sight is determined by the positional relationship between each imaging position and each pixel, the arrangement of the optical element group must be determined in accordance with the pixel panel. In any case, it is necessary to design to match the pitch of the optical elements with the pitch of the pixels.

  In any method, in the conventional stereoscopic video display device, it is necessary to match the optical element pitch of the optical element group and the pixel pitch of the pixel panel, and either pitch of the optical element group or the pixel panel is adjusted to the other. The decision is mainly based on the cost relationship between the two.

  For example, a stereoscopic image printed on a combination of a printed material (such as paper or plastic card) on which a stereoscopic image is printed and an optical element such as a lenticular lens array, which has been known for a long time, is printed. An optical element group (such as a lens array or a barrier array) is attached to the printed surface of the printed material, and the reflected light of each dot of the stereoscopic image printed on the printed material is given directivity by the optical element group. As a result, stereoscopic vision is realized. In the case of such a printed product, it is easy to change the arrangement of the dots of the stereoscopic image to be printed, and therefore the pitch of the dots of the stereoscopic image to be printed is determined according to the optical element group. In addition, when a computer display or the like is used as a pixel panel, a dedicated optical element group (for example, a lenticular lens array) that matches the pixel pitch is designed and manufactured.

  The present invention has been made in view of the above-mentioned problems, and a first object of the present invention is to overcome the trade-off relationship between the number of viewpoints and the video resolution as in the conventional stereoscopic vision system, Providing a new stereoscopic method that can achieve both stereoscopic effect and high resolution, and eliminate the restriction that one of the pixel panel and the optical element group must be designed exclusively for the other. It is. The second purpose is to minimize a reduction in the processing speed of generating a stereoscopic image by a new stereoscopic method, and to use a depth image, so that a general CG tool or a photograph for planar viewing is used. Therefore, it is possible to generate a simple stereoscopic image.

  Means for solving the above problems will be described below, but the stereoscopic viewing method of the present invention is considered to be a development of the conventional multi-view and super-multi-view methods. Compared with the multi-view type, the superiority from them is described.

  Since the amount of information that can be displayed on the pixel panel of the stereoscopic video display device is constant, it has been conventionally assumed that the resolution and the number of viewpoints have a trade-off relationship. That is, in order to achieve both a natural stereoscopic effect and a high resolution, it is necessary to introduce elements other than the resolution and the number of viewpoints. Therefore, in the present invention, an element of image accuracy is introduced.

  A conventional stereoscopic method generates a stereoscopic image in consideration of an image (image) viewed at each viewpoint. That is, first, an image (hereinafter referred to as “individual viewpoint image”) that is a video of each viewpoint is generated, and a stereoscopic image is generated by interleaving the generated individual viewpoint image. Since a stereoscopic image is generated based on an image that is a video viewed at each viewpoint, the video at each viewpoint is naturally an accurate image.

  When the stereoscopic image generated by the present invention is stereoscopically viewed, the video viewed at each viewpoint is slightly inaccurate compared to the conventional video. However, according to the principle described in detail in the following embodiments, the stereoscopic image generated by the present invention has sufficient clarity to be visually recognized as an image.

  Stereoscopic images from the point that it was possible to improve both the number of viewpoints and the video resolution at each viewpoint, which could not be realized by the conventional method according to the present invention, and the increased processing load accompanying the increase in the number of viewpoints. Hereinafter, the point that it was possible to prevent the generation rate of the liquid crystal from being lowered will be described along each invention.

The first invention for solving the above-described problems is
A stereoscopic panel including a pixel panel (for example, the pixel panel 20 in FIG. 2) on which pixels are arranged, and an optical element group (for example, the lens plate 30 in FIG. 2) that gives directivity to the light emitted from each pixel of the pixel panel. A stereoscopic image in a three-dimensional virtual space to be displayed on a visual image display device (for example, the stereoscopic image display device 200A in FIG. 2 or the stereoscopic image display device 200 in FIG. 23) is displayed on the three-dimensional virtual space in a given viewpoint. And a depth value information (for example, depth image data of FIG. 27) of each dot of the reference image with the viewpoint position as a reference. 472) based on the image generation device (for example, the stereoscopic image generation device 1 in FIG. 23),
Regarding the color information of each pixel of the pixel panel, 1) the direction of a light beam (for example, the representative light beam PR) that passes through the optical element that gives directivity to the pixel and the light beam emitted from the pixel, and the color information in the three-dimensional virtual space A direction calculating step (for example, step S39 in FIG. 31) for obtaining a direction corresponding to the direction of the light ray in the three-dimensional virtual space based on the viewpoint position (for example, the pixel-specific viewpoint CM), and 2) the direction calculation. A dot representing the color information of the object closest to the viewpoint in the direction in the three-dimensional virtual space is selected from the reference image by a predetermined convergence calculation process based on the direction obtained in the step and the depth value information. Dot determination step (for example, steps S39 to S49 in FIG. 31) is executed, and the reference image determined in the dot determination step is deleted. DOO color information by the color information of the pixel, an image generating apparatus for generating a stereoscopic image.

Further, the eighth invention is
A stereoscopic panel including a pixel panel (for example, the pixel panel 20 in FIG. 2) on which pixels are arranged, and an optical element group (for example, the lens plate 30 in FIG. 2) that gives directivity to the light emitted from each pixel of the pixel panel. A stereoscopic image in a three-dimensional virtual space to be displayed on a visual image display device (for example, the stereoscopic image display device 200A in FIG. 2 or the stereoscopic image display device 200 in FIG. 23) is displayed on the three-dimensional virtual space in a given viewpoint. And a depth value information (for example, depth image data of FIG. 27) of each dot of the reference image with the viewpoint position as a reference. 472), and a program (for example, the stereoscopic image generation program 410 in FIG. 23) for causing the computer to generate the program.
Regarding the color information of each pixel of the pixel panel, 1) the direction of a light beam (for example, the representative light beam PR) that passes through the optical element that gives directivity to the pixel and the light beam emitted from the pixel, and the color information in the three-dimensional virtual space A direction calculating step (for example, step S39 in FIG. 31) for obtaining a direction corresponding to the direction of the light ray in the three-dimensional virtual space based on the viewpoint position (for example, the pixel-specific viewpoint CM), and 2) the direction calculation. A dot representing the color information of the object closest to the viewpoint in the direction in the three-dimensional virtual space is selected from the reference image by a predetermined convergence calculation process based on the direction obtained in the step and the depth value information. Dot determination step (for example, steps S39 to S49 in FIG. 31) is executed by the computer, and is determined by the dot determination step. The color information of the dots of the reference image by the color information of the pixel, a program to be executed by the computer to generate a stereoscopic image.

  According to the first or eighth invention, for the color information of each pixel of the pixel panel, 1) the direction of the light passing through the pixel and the optical element that gives directivity to the emitted light of the pixel, and the three-dimensional virtual Based on the viewpoint position in the space, a direction corresponding to the direction of the light beam in the three-dimensional virtual space is obtained, and by a predetermined convergence calculation process based on the obtained direction and the depth value information, A dot representing the color information of the object closest to the viewpoint in the direction is determined from the reference image. Then, by using the color information of the determined dot of the reference image as the color information of the pixel, a stereoscopic image is generated.

  Therefore, when the generated stereoscopic image is displayed on the stereoscopic video display device, each pixel of the pixel panel represents the color information of the three-dimensional virtual space based on the direction of the light beam of the pixel. Therefore, it can be said that there are as many viewpoints as the number of light rays, that is, as many viewpoints as there are pixels. Therefore, the viewer's eyes must be positioned at the assumed viewpoint position as in the conventional multi-view method, and the observer is positioned within a certain area as in the super multi-view method. Even if there is an eye at any position, it is possible to achieve a good stereoscopic view.

  On the other hand, the light beam of each pixel of the pixel panel is in a separate direction for each pixel. However, when the direction of the light ray is traced, the light ray passing through the observer's left eye and the vicinity of the position of the left eye is in a substantially uniform direction. Therefore, the image visually recognized by the observer is visually accurate with certain clarity although it is not accurate color information viewed from the position of the eyes. Since the resolution of the image at the position of the pixel surface is about the width of one optical element, it does not lead to a significant resolution degradation as in the super multi-view type.

  Further, the stereoscopic image is generated based on a reference image that is a planar image obtained by viewing the three-dimensional virtual space from a given viewpoint, and depth value information of each dot of the reference image with the viewpoint position as a reference. be able to. Therefore, for example, an image created with an existing high-quality CG tool can be easily three-dimensionalized without using a dedicated renderer, and a highly versatile image generation apparatus can be provided. In addition, when creating a stereoscopic image from a real image, it is possible to shoot from one viewpoint position, so that a stereoscopic image can be easily generated.

  In addition, since the dots of the reference image for setting the color information of each pixel are determined by a predetermined convergence calculation process, when there are a plurality of objects in a three-dimensional virtual space, the intersection determination between the light ray and each object is performed. Need not be sequentially performed, and a stereoscopic image can be generated at high speed.

  As described above, when a stereoscopic image is generated from the planar image from one viewpoint and the depth value information corresponding thereto, there is no information from the viewpoint that wraps around to the left and right. In such a case, the image may become unnatural. However, when the assumed observation range is not so wide, that is, when the left and right binocular parallax is only inside or near the outside, the influence is small and a stereoscopic image having no practical problem can be obtained.

The second invention is the image generating device of the first invention,
The dot determining step includes a depth differential information calculating step (for example, steps S51 and S57 in FIG. 32) for calculating depth differential information that is a differential value of the depth value of the dots constituting the reference image, and the direction calculating step On the basis of the direction value obtained in step 3, the depth value information, and the depth derivative information calculated in the depth derivative information calculation step, by a predetermined convergence calculation process by Newton's method, the most in the direction in the three-dimensional virtual space An image generating apparatus characterized in that it is a step of determining a dot representing color information of an object closer to the viewpoint from the reference image.

The ninth invention is a program according to the eighth invention,
The dot determining step includes a depth differential information calculating step (for example, steps S51 and S57 in FIG. 32) for calculating depth differential information that is a differential value of the depth value of the dots constituting the reference image, and the direction calculating step On the basis of the direction value obtained in step 3, the depth value information, and the depth derivative information calculated in the depth derivative information calculation step, by a predetermined convergence calculation process by Newton's method, the most in the direction in the three-dimensional virtual space The program is a step of determining a dot representing color information of an object closer to the viewpoint from the reference image.

  According to the second or ninth aspect, in the dot determination step, the depth differential information of the dots constituting the reference image is calculated. Then, based on the direction obtained in the direction calculation step, the depth value information, and the calculated depth derivative information, a predetermined convergence calculation process by Newton's method is performed, and the most in the direction in the three-dimensional virtual space. A dot representing color information of an object closer to the viewpoint is determined from the reference image.

  Although Newton's method is an algorithm whose convergence to the solution depends on the initial value, it is characterized by a quadratic convergence, so that a solution can be obtained at high speed. Therefore, since the dot representing the color information of the object closest to the viewpoint can be determined in a short time from the reference image, it is possible to increase the speed of stereoscopic image generation.

A third invention is the image generation device of the first invention,
Before the determination process by the dot determination step, a depth differential information calculation step (for example, step S27 in FIG. 31) for calculating depth differential information that is a differential value of the depth value of each dot of the reference image is executed.
In the dot determination step, a predetermined convergence calculation by the Newton method is performed using the direction obtained in the direction calculation step and the depth derivative information calculated in the depth derivative information calculation step in addition to the depth value information. A dot representing color information of an object closest to the viewpoint in the direction in the three-dimensional virtual space is determined from the reference image by processing.
An image generating apparatus characterized by the above.

The 10th invention is the program of the 8th invention,
Before the determination process by the dot determination step, a depth differential information calculation step (for example, step S27 in FIG. 31) for calculating depth differential information that is a differential value of the depth value of each dot of the reference image is executed on the computer. Let
In the dot determination step, a predetermined convergence calculation by the Newton method is performed using the direction obtained in the direction calculation step and the depth derivative information calculated in the depth derivative information calculation step in addition to the depth value information. A dot representing color information of an object closest to the viewpoint in the direction in the three-dimensional virtual space is determined from the reference image by processing.
The program for causing the computer to execute.

  According to the third or tenth invention, the depth differential information of each dot of the reference image is calculated before the determination process in the dot determination step. In the dot determination step, in addition to the direction obtained in the direction calculation step and the depth value information, a predetermined convergence calculation process is performed by Newton's method using the depth derivative information calculated in advance, and a three-dimensional virtual A dot representing the color information of the object closest to the viewpoint in the direction in the space is determined from the reference image.

  By calculating the depth differential information for all the pixels in advance before performing the dot determination process, it is not necessary to sequentially calculate the depth differential information in the convergence calculation process by the Newton method.

The fourth invention is the image generation device of the second or third invention,
The depth derivative information calculating step calculates the difference between the depth value of the dot and the depth value of an adjacent dot adjacent to the dot for each dot of the reference image to obtain a derivative value, thereby calculating the depth derivative information. An image generation apparatus characterized by being a step of calculating.

The eleventh invention is the program of the ninth or tenth invention,
The depth derivative information calculating step calculates the difference between the depth value of the dot and the depth value of an adjacent dot adjacent to the dot for each dot of the reference image to obtain a derivative value, thereby calculating the depth derivative information. A program characterized by being a step of calculating.

  According to the fourth or eleventh invention, for each dot of the reference image, the difference between the depth value of the dot and the depth value of the adjacent dot adjacent to the dot is obtained as a differential value, so that the depth differentiation Information is calculated.

  In addition, although it is the focusing calculation process of each invention above, the initial value in performing the focusing calculation may be as follows. That is, as a fifth invention, the dot determination step of the image generation apparatus according to any one of the first to fourth inventions is performed by setting the dot position closest to the viewpoint in the direction obtained in the direction calculation step to the initial position. As a step of performing the predetermined focusing calculation process. As a twelfth invention, in the program according to any one of the eighth to eleventh inventions, the dot determination step may be performed with the dot position closest to the viewpoint in the direction obtained in the direction calculation step as an initial position. You may comprise as a step which performs the said predetermined | prescribed focusing calculation process.

  Here, “the dot position closest to the viewpoint” is the position of the intersection of a plane parallel to the pixel surface and passing through the highest point of the depth model and the direction obtained in the direction calculation step.

A sixth invention is the image generating device according to any one of the first to fifth inventions,
The stereoscopic image display device has a viewing angle with respect to one optical element at a predetermined assumed observation position as a viewing angle with respect to the optical element λ, a viewing angle with respect to one pixel to which directivity is given by the one optical element as a viewing angle with respect to pixel σ, and Λ: σ = n: m (where n is a natural number and m is a natural number less than Φ / λ) is not established when the viewing angle with respect to the stereoscopic image drawing region in the pixel panel is a paired drawing region viewing angle Φ. Is an image generation apparatus.

The seventh invention is the image generation device according to any one of the first to sixth inventions,
The stereoscopic video display device has a horizontal width of one optical element as L, a horizontal width of one pixel provided with directivity by the one optical element as S, and a stereoscopic image drawing area in the pixel panel. Is an image generation device characterized in that L: S = o: p (where o is a natural number and p is a natural number less than R / L) is not established.

  According to the sixth or seventh aspect of the invention, stereoscopic image generation is realized when the lens pitch of the lens and the pixel pitch of the display device do not match (so-called “pitch does not match”). Therefore, it is possible to eliminate the restriction that one of the pixel panel and the optical element group must be designed exclusively for the other.

  The thirteenth invention is a computer-readable information storage medium (for example, the storage unit 400 in FIG. 23) storing the program of any one of the eighth to twelfth inventions.

  According to the thirteenth aspect, an information storage medium that achieves the same operational effects as any one of the eighth to twelfth aspects is realized.

  According to the present invention, the resolution of an image is reduced to the width of one optical element, while there is a viewpoint corresponding to the total number of pixels of the pixel panel by degrading the accuracy of the image to the extent that does not hinder stereoscopic vision. It is possible to realize a new stereoscopic method that can be maintained at a certain level, and to eliminate the restriction that one of the pixel panel and the optical element group must be designed exclusively for the other. In addition, the processing speed of generating a stereoscopic image by a new stereoscopic method can be increased.

  Preferred embodiments of the present invention will be described below with reference to the drawings. In each drawing, hatching is not drawn in order to clearly show the direction of light rays. In the following, a case where a stereoscopic image to be displayed on a stereoscopic video display device using a lenticular lens array as an optical element group is generated will be described, but the application of the present invention is not limited to this.

1. Principle of generating a stereoscopic image There are three methods for generating a stereoscopic image: (I) a method of rendering an accurate three-dimensional model at the same time, and (II) a stereoscopic image from a perspective image and a depth image from one viewpoint. There are a method of creating, (III) a method of creating images for two eyes or multiple eyes, and interpolating the images of intermediate viewpoints from them. In the present embodiment, a case where a stereoscopic image is generated based on the methods (I) and (II) will be described as an example.

First, a case where a stereoscopic image is generated based on the method (I) will be described.
FIG. 1 is a diagram showing an outline of stereoscopic image generation in this embodiment, and shows a vertical sectional view with respect to the display surface 22. As shown in the figure, in this embodiment, for each pixel PE on the display surface 22, (1) a representative point of the pixel PE (for example, the center of the pixel PE) and a lens (optical element) corresponding to the pixel PE (2) The color information of the object in the line-of-sight direction of the determined line-of-sight V is used as the color information of the pixel PE ( Rendering) to generate a stereoscopic image.

(1) Determination of the line of sight V The line of sight V is a configuration parameter of a stereoscopic video display device that displays a stereoscopic image (as will be described later, the relative arrangement relationship between the pixel panel and the lens plate, The pixel pitch, the lens pitch of the lens plate, the focal length, and the like) and an assumed observer position (hereinafter referred to as “assumed observation position”). Specifically, for each pixel PE, a lens (optical element) corresponding to the pixel PE is determined based on the configuration parameter of the stereoscopic video display device and the assumed observation position, and the representative point of the pixel PE is determined. A ray (representative ray) after passing through the principal point of the lens corresponding to the pixel PE is calculated. Then, the line of sight having the same position as that of the representative light ray PR and having the direction reversed is determined as the line of sight V of the pixel. The assumed observation position is the position of the observer's viewpoint relative to the display surface of the stereoscopic video display device.

  Here, the stereoscopic video display apparatus handled in this embodiment will be described. In the present embodiment, a stereoscopic image to be displayed on a lenticular stereoscopic image display device is generated. A lenticular stereoscopic image display device is a stereoscopic image display device that uses a lenticular lens array as an optical element group, and the lenticular lens array is mounted at a certain distance from the display surface of a flat panel display such as a flat liquid crystal display. The display device allows the observer to recognize stereoscopic vision by viewing (observing) the image displayed on the display surface via the lenticular lens array.

  FIG. 2 is a diagram showing a schematic structure of a vertical lenticular stereoscopic image display apparatus 200A. FIG. 4A shows a cross-sectional view in the horizontal direction (horizontal scanning direction) with respect to the display surface of the stereoscopic video display apparatus 200A, and FIG. 4B shows a plan view seen from the observer side.

  According to the figure, the stereoscopic video display apparatus 200A is mainly configured to include a backlight 10, a pixel panel 20, and a lens plate 30. The backlight 10, the pixel panel 20, and the lens plate 30 are plate-like bodies and are arranged in parallel to each other.

  The backlight 10 emits light, which passes through the pixel panel 20 and the lens plate 30 and travels out of the stereoscopic video display device 200A. That is, the observer sees an image displayed on the pixel panel 20 through the lens plate 30.

  The pixel panel 20 is a display capable of color display in which pixels (pixels) are arranged in a fixed arrangement, and any type can be used as long as it can be stereoscopically combined with the lens plate 30. For example, there are a color filter type liquid crystal display, a plasma display, an inorganic EL display, an organic EL display, and the like. Even if no color filter is used, elements that emit light in a single color such as red (R), green (G), and blue (B) are arranged like an organic EL display or LED display in which single-color light emitting elements are arranged. Even a display that has been used is applicable. Further, it may be a monochrome display in which pixels emitting light of the same color are arranged, and further, a display having pixels of colors other than R (red), G (green), and B (blue). It doesn't matter. Further, the pixel arrangement may be a delta arrangement or other arrangements as long as the coordinates of the representative points of the pixels can be obtained as well as the lattice shape.

  One surface of the lens plate 30 is a concavo-convex surface formed by connecting a microlens (hereinafter simply referred to as a “lens”) 32 that is a semi-cylindrical cross section (saddle-shaped) or an optical element optically equivalent thereto. There is a lenticular lens array whose other surface is substantially planar. Each lens 32 of the lens plate 30 functions to give directivity to light rays (emitted light rays) emitted from the respective pixels PE of the display surface 22.

The lens plate 30 has a flat surface facing the display surface 22 of the pixel panel 20, and the distance G between the principal point surface of the lens plate 30 and the display surface 22 substantially matches the focal length F of each lens 32. Are arranged to be. Note that the distance G does not completely coincide with the focal length F and may have some error. For example, FIG. 49A shows a state in which G = F, and when viewed from a specific direction, one pixel PE extends over the lens 32 and is observed. Further, even when the distance G is slightly away from the focal distance F, that is, in the state shown in FIGS. 5B and 5C, when viewed from a specific direction, one pixel PE is observed to spread over the entire lens 32. . However, when the distance G is further away from the focal distance F, the lens 32 also displays the adjacent pixel PE, so that the stereoscopic image quality is impaired. That is, if the pixel pitch length in the sub-pixel unit of the pixel panel 20 is S and the lens pitch length of the lens plate 30 is L, the distance G is arranged so as to satisfy the following expression (1). Therefore, it is possible to realize a stereoscopic view with better image quality than in other cases.
(LS) · F / L ≦ G ≦ (L + S) · F / L (1)

In the IP system, the distance G is longer than the focal length F in order to form an image at a constant distance C. That is, the following equation is established. In this respect, the system in this embodiment is different in principle from the IP system.
1 / G + 1 / C = 1 / F, that is, G = (C · F) / (C−F)> F
In the light beam reproduction method, since an image is formed at a plurality of distances, a lens cannot be used as an optical element, and a pinhole is used. In this respect, this method is also different in principle from the light beam reproduction method.

  By disposing the display panel 20 and the lens plate 30 in this way, the focal point of each lens 32 is located at one point on the display surface 22 of the pixel panel 20, and the pixel PE where the focal point is located appears enlarged by the lens 32. It will be. In addition, when it can be considered that it is optically substantially equivalent, you may arrange | position so that the uneven surface of the lens plate 30 may oppose the display surface 22 of the pixel panel 20. FIG.

  Further, as shown in FIG. 2B, the lens plate 30 has a principal dotted line 36 (a set of principal points) of each lens 32. Since the microlens of the lenticular lens plate has a cylindrical cross section (a bowl shape), the principal point Are arranged in such a manner that the direction of the pixel line 20 coincides with the vertical pixel arrangement direction (vertical scanning direction) of the pixel panel 20. In FIG. 2B, a line 32a indicates an end portion of each lens 32 of the lens plate 30.

By the way, the conventional lenticular stereoscopic image display device is designed so that the lens pitch of the lens plate matches the pixel pitch of the pixel panel (hereinafter simply referred to as “pitch match”). That is, in the case of the n-eye system, the following expression (2) is established.
L = n · S (2)

However, in the present embodiment, the lens plate 30 is designed such that the lens pitch does not match the pixel pitch of the pixel panel 20 (hereinafter simply referred to as “pitch does not match”). That is, the following equation (3) is not satisfied.
L = n · S (3)
However, n is a natural number.

By the way, when an observer actually views a stereoscopic image displayed on the stereoscopic video display device, the viewpoint of the observer is located at a finite distance from the display surface 22. That is, as shown in FIG. 3, the line-of-sight direction of the observer's viewpoint differs depending on the location of the display surface 22, and therefore the correspondence between the lens 32 and the pixel PE is shifted. That is, the substantial lens pitch L _E is given by the following equation (4). The figure shows a cross-sectional view in the transverse direction with respect to the display surface of the stereoscopic video display device of the present embodiment.
L _E = L × (D + F) / D (4)
However, D is the distance between an observer's viewpoint and a display surface.

Therefore, strictly speaking, it can be said that the following equation (5) is satisfied in the “pitch match” state, and not satisfied is the “pitch mismatch” state.
L _E = n · S (5)

In addition, “pitch matches / does not match” refers to a viewing angle (versus-pixel viewing angle) σ with respect to one pixel PE viewed from the viewpoint of an actual (or assumed) observer, and an emission ray of the pixel PE. It is also expressed by a viewing angle (a viewing angle with respect to the lens) λ with respect to one lens 32 that imparts a characteristic. The anti-pixel viewing angle σ is given by the equation (6a), and the anti-lens viewing angle λ is given by the following equation (6b).
tan σ = S / (D + F) (6a)
tan λ = L / D (6b)

If the following expression (7) is satisfied, the “pitch is matched” state is satisfied, and if not satisfied, the “pitch is not matched” state.
λ = n · σ (7)
However, n is a natural number.

That is, the equation (7) is satisfied, m times the pair lens viewing angle λ is equal to n times the pair pixel viewing angle sigma, that is, when the m times the lens pitch L _E matches the n multiple of the pixel pitch S In this case, it can be said that the pitch is “matched”.

The condition of Expression (7) is a condition in the case of not considering the necessity of making the distance between the viewpoints coincide with the binocular distance of the human in the conventional multi-view or super-multi-view stereoscopic view. In order to make the distance between the viewpoints coincide with the binocular distance of the human, as shown in FIG. 50, the distance between the observation distance D, the binocular distance E, the pixel pitch S, and the focal distance F of the lens. The following equation (8) needs to be satisfied.
E / D = S / F (8)
That is, in the case of a multi-lens system in which the distance between the respective viewpoints is matched with the human binocular distance, it is necessary to satisfy the expressions (7) and (8) at the same time. Therefore, an accurate lens design is necessary for the conventional multi-lens system.

  As described above, in the conventional multi-view system, the same viewpoint is repeatedly generated for every certain length in the horizontal direction. In addition, in these multi-view systems, images (individual viewpoint images) based on n preset viewpoints (individual viewpoints) are generated, and these images are rearranged (interleaved) according to the viewpoint repetition pattern. Thus, a stereoscopic image is generated.

However, the image generation method of the present embodiment is characterized in that a good stereoscopic image can be obtained only when the same viewpoint is not repeated as described above. Here, it can be said that the condition that the same viewpoint repeats in the horizontal direction is that the relationship of the following expression (9) is established between the above-described anti-pixel viewing angle σ and the anti-lens viewing angle λ.
λ: σ = n: m (9)
However, n and m are natural numbers.

  That is, when Expression (9) is not satisfied, “the same viewpoint is not repeated”, that is, “the pitch is not matched. However, the repetition of the same viewpoint is at least within the image display area of the stereoscopic image. Therefore, the condition that “the repetition of the same viewpoint does not occur”, that is, “the pitch does not match” is “the natural number where m is less than Φ / λ” in Equation (9). That is. However, Φ is a viewing angle (versus display area viewing angle) with respect to an area (stereoscopic image display area) in which a stereoscopic image on the display surface 22 is viewed from the viewpoint of an actual (or assumed) observer.

  However, the present embodiment uses a stereoscopic video display device that does not match the pitch (that is, equations (7) and (8) do not hold), and generates a stereoscopic image to be displayed on this stereoscopic video display device. Shall. In other words, in the conventional lenticular stereoscopic image display device, it is necessary to design the lens pitch and the pixel pitch to be in order to enable stereoscopic viewing. Enables stereoscopic viewing in a video display device. Therefore, it is not necessary to manufacture a lenticular lens plate that matches the pixel pitch of each display, and the ready-made lenticular lens plate can be applied to various displays, so that the cost of the lens plate can be greatly reduced. . It is also possible to select the most suitable one from a plurality of selectable lens plates. Furthermore, when the oblique lenticular method is adopted, the oblique angle θ can be set freely. Specifically, moire and color fringes can be reduced by adjusting the oblique arrangement angle θ of the lens plate with respect to the pixel panel without manufacturing a new lens.

  Next, a method for determining the line of sight V of each pixel PE on the display surface 22 will be described. Before that, the coordinate system of the display surface 22 is defined as shown in FIG. That is, the direction along the horizontal scanning direction (lateral direction) of the display surface 22 is the x-axis direction, the direction along the vertical scanning direction (longitudinal direction) is the y-axis direction, and is perpendicular to the viewer side from the display surface 22. The direction to go is the z-axis positive direction.

  First, the most basic method of determining the line of sight V in this method will be described. In this method, the observer's line of sight is “front” of the display surface 22 (the position where the observation line of sight passing through the center O of the display surface 22 is perpendicular to the display surface 22) and “infinity”. In other words, the non-stereoscopic image drawing method corresponds to “parallel projection”. Here, a method of determining the line of sight V when the stereoscopic video display apparatus is each of the above-described vertical / oblique lenticular methods will be described. In the following, a method for determining the line of sight V for one pixel PE will be described, but it is obvious that other pixels PE can be similarly determined.

  A method of determining the line of sight V when the stereoscopic video display device is of the vertical lenticular method will be described with reference to FIG. FIG. 5 is a schematic three-view diagram of a vertical lenticular stereoscopic image display apparatus 200A. FIG. 5A shows a cross-sectional view (cross-sectional view in the horizontal scanning direction) parallel to the xz plane. b) shows a cross-sectional view (cross-sectional view in the vertical scanning direction) parallel to the yz plane, and FIG. 5C shows an xy plan view.

  First, a lens 32 corresponding to a pixel (hereinafter referred to as “target pixel”) PE that determines the line of sight V is determined. In FIG. 5A, each lens 32 of the lens plate 30 is projected in parallel on the display surface 22 of the pixel panel 20 (that is, by a straight line perpendicular to the display surface 22 passing through the end portion 32a of each lens 32). The display surface 22 is divided into projection areas of the lenses 32. The lens 32 in the projection region to which the representative point of the target pixel PE (here, the center of the pixel) belongs is set as the lens 32 corresponding to the target pixel PE. However, FIG. 4A is a cross-sectional view passing through a representative point of the target pixel PE.

  In FIG. 5A, the display surface 22 is divided into a projection area 26-1 of the lens 32-1, a projection area 26-2 of the lens 32-2, and so on. Since the representative point of the target pixel PE belongs to the projection area 26-1, the lens 32 corresponding to the target pixel PE is the lens 32-1.

  Next, a light ray PR (hereinafter referred to as “representative ray”) PR after passing through the representative point of the target pixel PE and the principal point of the lens 32 corresponding to the target pixel PE is calculated, and the position is the same as that of the representative ray PR. The line of sight whose direction has been reversed is set as the line of sight V of the target pixel PE. Specifically, a point where the y coordinate is equal to the y coordinate of the representative point of the target pixel PE is calculated from the main dotted line 36 of the lens 32-1 corresponding to the target pixel PE, and this is set as the representative main point 36a. Then, a representative ray PR after passing through the representative point of the target pixel PE and the representative principal point 36a is calculated, and the line of sight having the same position and the opposite direction as the representative ray PR corresponds to the target pixel PE. The line of sight is V. Here, for the sake of simplicity, the direction from the representative point of the target pixel PE toward the representative principal point of the lens 32 corresponding to the target pixel PE is defined as the direction of the representative light ray PR.

  The method described above assumes that the observer's viewpoint is at infinity, but when observing an actual stereoscopic image, the observer's viewpoint is not at infinity. When observing from a short distance, you may feel a sense of incongruity in perspective. However, since the method of determining each line of sight V is simple, there is an advantage that the calculation load can be reduced.

  Next, a method for determining the line of sight V that enables more natural stereoscopic viewing than the above-described method will be described. This method is a drawing method that assumes the observer's viewpoint at a fixed position, and corresponds to “perspective projection” in terms of a non-stereoscopic image drawing method.

  Here, as shown in FIG. 6, the assumed observation position 40 is set to “front” with respect to the display surface 22 of the stereoscopic video display device. The “front” assumed observation position 40 is a position where an observation line-of-sight direction passing through the center O of the display surface 22 is perpendicular to the display surface 22. Hereinafter, the distance D between the assumed observation position 40 and the display surface 22 is referred to as “assumed observation distance D”. Then, the case where the method of determining the line of sight V when the assumed observation position 40 is “front position and a fixed position at a finite distance” is applied to a vertical lenticular stereoscopic image display device will be described.

  FIG. 7 is a partial schematic perspective view of a vertical lenticular stereoscopic image display apparatus 200A. FIG. 8 is a schematic three-view diagram of the stereoscopic video display device 200A. FIG. 8A is a cross-sectional view (cross-sectional view in the horizontal scanning direction) at the AA position parallel to the xz plane of FIG. 8 (b) is a cross-sectional view (vertical cross-sectional view) at a BB position parallel to the yz plane of FIG. 7, and FIG. 8 (c) is an xy plan view. Is shown. The lens plate 30 and the pixel panel 20 are arranged in parallel with a focal length F of each lens 32 of the lens plate 30.

  First, the lens 32 corresponding to the target pixel PE is determined. Specifically, in FIG. 8A, each lens 32 of the lens plate 30 is projected from the assumed observation position 40 onto the display surface 22 of the pixel panel 20 (that is, the end of each lens 32 from the assumed observation position 40). The display surface 22 is divided into projection areas of each lens 32 (by a straight line going to). Then, the corresponding lens 32 is determined depending on which projection region the representative point of the target pixel PE belongs to. However, FIG. 4A is a cross-sectional view passing through a representative point of the target pixel PE.

  In FIG. 6A, the display surface 22 includes a projection area 26-7 of the lens 32-7, a projection area 26-8 of the lens 32-8, a projection area 26-9 of the lens 32-9,.・ It is divided into Since the representative point of the target pixel PE belongs to the projection area 26-7, the lens 32 corresponding to the target pixel PE is the lens 32-7.

  Next, a representative ray after passing through the representative point of the target pixel PE and the principal point of the lens 32 corresponding to the target pixel PE is calculated. The line of sight V of the target pixel is assumed. Specifically, in FIG. 8B, a straight line LN1 connecting the representative point of the target pixel PE and the assumed observation position 40 and the principal point surface of the lens plate 30 (a surface including the principal point of each lens 32). The y-coordinate of the intersection with 35), which is a plane parallel to 22 is calculated. The calculated y coordinate is assumed to be “y1”. However, FIG. 4B is a cross-sectional view passing through the representative point of the target pixel PE. Next, a point whose y coordinate is “y1” is calculated from the main dotted line 36 of the lens 32-1 corresponding to the target pixel PE, and this is set as a representative main point 36c. Then, a representative ray PR after passing through the representative point of the target pixel PE and the representative principal point 36c is calculated, and the line of sight having the same position and the opposite direction to the representative ray PR corresponds to the target pixel PE. The line of sight is V.

  The method for determining the line of sight V of each pixel PE on the display surface 22 when the assumed observation positions are “front and infinity” and “front and finite distance fixed positions” has been described above. In the above, light rays incident on each lens 32 are not refracted (that is, the direction from the representative point of the target pixel PE toward the representative principal point of the lens 32 corresponding to the target pixel PE matches the direction of the representative light ray PR). Strictly speaking, as shown in FIG. 9, the representative ray PR connects the representative point of the target pixel PE and the representative principal point of the lens 32 corresponding to the target pixel PE, as shown in FIG. The y coordinate position slightly deviates from the straight line and does not match. Therefore, it is more preferable that the line of sight V of each pixel PE is accurately obtained by calculating and correcting this shift.

  In addition, as described with reference to FIGS. 7 and 8, the determination of the lens 32 for each pixel PE is performed by setting the assumed observation position 40 as “a fixed position at a finite distance”, so that the degree of perspective (perspective) is affected. In addition to becoming natural, the effect of widening the observation range around the assumed observation position 40 can be obtained.

  In this way, after determining the line of sight V corresponding to each pixel, as shown in FIG. 10, a pixel-specific viewpoint CM corresponding to the virtual camera is set for each pixel PE based on the determined line of sight V. Here, the pixel-specific viewpoint CM corresponding to the pixel PE is set for the pixel PE, but the pixel-specific viewpoint CM is not set in particular, and a drawing range in the z direction common to the line of sight V for all the pixels PE is set. However, drawing may be performed for each line of sight V.

  FIG. 10 is a diagram for explaining the setting of the pixel-specific viewpoint CM, and shows a partial horizontal cross-sectional view of the display surface 22. As shown in the figure, the pixel-specific viewpoints CM (CM1, CM2,...) Of each pixel PE (PE1, PE2,...) Have a line of sight V (V1, V2,. Set so that Further, the distance between each pixel-specific viewpoint CM and the display surface 22 is set so as to be located on the same plane parallel to the display surface 22, for example, as shown in FIG.

  In the figure, the visual lines V of the pixels PE1, PE2,... Are visual lines V1, V2,. Accordingly, the pixel-specific viewpoint CM of the pixel PE1 has the line of sight V1 as the pixel-specific viewpoint CM1 in the line-of-sight direction. In the pixel-specific viewpoint CM of the pixel PE2, the line of sight V2 is the pixel-specific viewpoint CM2 in the line-of-sight direction. Further, similarly, for the pixels PE3, PE4,..., The viewpoints CM for each pixel have the line of sight V3, V4,. Become.

(2) Rendering After setting the pixel-specific viewpoint CM of each pixel PE, a stereoscopic image is generated by rendering a three-dimensional virtual space based on the set pixel-specific viewpoint CM. Specifically, for each pixel PE, color information (RGB value, α value, etc.) of the object space in the line-of-sight direction of the pixel-specific viewpoint CM corresponding to the pixel PE is calculated, and the calculated color information is calculated for the pixel PE. A stereoscopic image is generated by using color information.

  FIG. 11 is a diagram for explaining the calculation of the color information, and shows a partial cross-sectional view of the display surface 22 in the horizontal direction. As shown in the figure, for each pixel PE on the display surface 22, the color information of the object space in the line-of-sight direction of the corresponding pixel-specific viewpoint CM is calculated, and the calculated color information is used as the color information of the pixel PE. The color information calculation method is realized by, for example, a so-called ray-tracing method that is determined based on light rays along the line-of-sight direction from the pixel-specific viewpoint CM.

  In the figure, the pixel-specific viewpoints of the pixels PE1, PE2,... Are pixel-specific viewpoints CM1, CM2,. Therefore, the color information of the pixel PE1 is color information of the object space in the line-of-sight direction of the pixel-by-pixel viewpoint CM1, and the color information of the pixel PE2 is color information of the object space in the line-of-sight direction of the pixel-by-pixel viewpoint CM2. Further, for each of the pixels PE3, PE4,..., The color information of the object space in the line-of-sight direction of the corresponding pixel-specific viewpoints CM3, CM4,. , Color information.

  As described above, in this embodiment, for each pixel PE on the display surface, (1) the line of sight V is determined, and (2) the color information of the line of sight of the determined line of sight V is used as the color information of the pixel PE (rendering). To generate a stereoscopic image.

  When the image generated in this way is displayed as a stereoscopic image on the stereoscopic video display apparatus of the present embodiment, the video visually recognized by the observer is slightly less accurate than the conventional stereoscopic video. Become a statue.

  FIG. 12 is a diagram for explaining that the stereoscopic image of the present embodiment is slightly inaccurate, and shows a partial cross-sectional view of the display surface 22 in the horizontal direction. In this figure, when the stereoscopic video display device is viewed from the observer's right eye EY1, the pixel PE1 can be seen through the lens 32-1, the pixel PE2 can be seen through the lens 32-2, and the lens 32-3 can be seen. Pixel PE3 is visible.

  By the way, the color information of the pixel PE1 is color information of the object space in the line-of-sight direction of the pixel-specific viewpoint CM1, the color information of the pixel PE2 is color information of the object space in the line-of-sight direction of the pixel-specific viewpoint CM2, The color information of the pixel PE3 is color information of the object space in the viewing direction of the pixel-specific viewpoint CM3. That is, since the right eye EY1 does not match the pixel-specific viewpoints CM1, CM2, and CM3, the color information of each pixel PE recognized by the observer is not accurate color information viewed from the position.

  However, the positions of the pixel-specific viewpoints CM1, CM2, and CM3 are in the vicinity of the right eye EY1, and the line-of-sight directions of the pixels PE1, PE2 are determined by the right eye EY1 through the lenses 32-1, 32-2, and 32-3. , PE3 viewing direction is slightly shifted. For this reason, the image (color information) visually recognized by the observer's right eye EY1 is not an accurate image (color information) viewed from the position, but is visually recognized with a certain degree of clarity.

  Further, according to the present embodiment, the number of viewpoints (views) is extremely large, and natural stereoscopic vision is possible. This will be described with reference to FIGS. 13 and 14 compared with the conventional multi-view stereoscopic vision.

  FIG. 13 is a diagram showing an outline (image) of stereoscopic viewing of a conventional multi-view system, and shows a case of a trinocular system. As shown in the upper side of the figure, in the conventional three-eye stereoscopic view, three individual viewpoints 1, 2, 3 are set at an appropriate distance in the object space, and each of the individual viewpoints 1, 2, 3 is set. The individual viewpoint images 1, 2, and 3 of the object space viewed from the above are generated. Then, a stereoscopic image is generated by interleaving these three individual viewpoint images 1, 2, and 3. In the figure, the number of each pixel of the stereoscopic image represents the number of the corresponding individual viewpoint image (individual viewpoint). Further, the position and the line-of-sight direction of each pixel viewpoint CM are rough because they are schematic diagrams (image diagrams), and are not accurate.

  Then, as shown in the lower side of the figure, when the generated stereoscopic image is displayed on a conventional three-lens stereoscopic image display device and viewed from each of the appropriate viewing positions 1, 2, and 3, the suitable viewing position 1 The individual viewpoint image 1 can be seen, the individual viewpoint image 2 can be seen at the appropriate viewing position 2, and the individual viewpoint image 3 can be seen at the suitable viewing position 3. More specifically, the individual viewpoint image 1 can be seen in the suitable viewing range 1 that is substantially centered on the suitable viewing position 1, and the individual viewpoint image 2 can be seen in the suitable viewing range 2 that is substantially centered on the suitable viewing position 2. The image 3 can be seen in the individual viewpoint appropriate viewing range 3 centering on 3. However, in the same figure, the appropriate viewing range is a schematic diagram (image diagram), and is therefore an approximate one and is not accurate.

  That is, when the observer OB views the stereoscopic image at a position where the right eye EY1 substantially matches the appropriate viewing position 2 and the left eye EY2 substantially matches the appropriate viewing position 1, the right eye EY1 sees the individual viewpoint image 2, and the left eye In EY2, the stereoscopic video is recognized when the individual viewpoint image 1 is seen. That is, this corresponds to a state in which the object space is viewed with the right eye EY1 as the individual viewpoint 2 and the left eye EY2 as the individual viewpoint 1.

  Further, when the position of the observer OB moves to the right with respect to the stereoscopic image, when the right eye EY1 or the left eye EY2 passes through the boundary portion of the appropriate viewing range, an image that can be seen with the right eye EY1 or the left eye EY2 suddenly appears. Switch to. Specifically, for example, when the right eye EY1 passes through the boundary portion between the suitable viewing range 2 and the suitable viewing range 3, the image seen by the right eye EY1 is switched from the individual viewpoint image 2 to the individual viewpoint image 3. In addition, when the left eye EY2 passes through the boundary portion between the appropriate viewing range 1 and the appropriate viewing range 2, the image seen by the left eye EY2 is switched from the individual viewpoint image 1 to the individual image 2.

  This is because in conventional multi-view stereoscopic vision, each individual viewpoint image viewed from n individual viewpoints is interleaved to generate a stereoscopic image, which is designed to match the pitch. This is because the stereoscopic view is realized by displaying the image on a stereoscopic image display device. That is, in the conventional stereoscopic video display device, the stereoscopic image is separated into the individual viewpoint images by the lenticular lens plate.

  FIG. 14 is a diagram showing an outline (image) of stereoscopic vision of the present embodiment. In the present embodiment, as described above, a pixel-specific viewpoint CM is set for each pixel PE, and the color information of the object space in the line-of-sight direction of each pixel-specific viewpoint CM is used as the color information of the corresponding pixel PE. An image is generated. That is, as shown in the upper side of the figure, pixel-specific viewpoints CM1, CM2,... Equal to the number of pixels are set, and the color information of the respective line-of-sight directions of the set pixel-specific viewpoints CM1, CM2,. A stereoscopic image is generated as color information of the pixels PE1, PE2,. In the figure, the number of each pixel PE of the stereoscopic image represents the number of the corresponding pixel viewpoint CM.

  The stereoscopic image generated in this way is displayed on the stereoscopic video display device 200A of the present embodiment shown in FIG. 2, for example, and the observer OB views the stereoscopic image at the position shown on the lower side of the figure. Then, the image A composed of the pixels PE1, PE2, PE3,... Can be seen in the left eye EY2, and the image B composed of the pixels PE11, PE12, PE13,. appear. That is, the left eye EY2 is a viewpoint group including pixel-specific viewpoints CM1, CM2,..., CM10, and the right eye EY1 is a viewpoint group including pixel-specific viewpoints CM11, CM12,. It corresponds to the state of watching.

Then, when the position of the observer OB moves slightly to the right with respect to the stereoscopic image, an image A in which an image seen by the left eye EY2 of the observer is replaced with a pixel PE that is a part of the image A is replaced with an adjacent pixel PE ₂ and the image seen by the right eye EY1 changes to an image B _{2 in} which a part of the pixels PE of the pixel B is replaced with the adjacent pixel PE.

  Thus, in this embodiment, when the position (observation position) of the observer who views the stereoscopic image changes, the images seen in each of the right eye EY1 and the left eye EY2 change little by little with this change. Specifically, the image changes to a pixel in which some pixels are replaced by neighboring pixels. Therefore, the image that is recognized by the observer OB in the right eye EY1 and the left eye EY2 changes little by little, and the recognized video changes little by little.

  For this reason, for example, an image seen at the boundary portion of the appropriate viewing range suddenly changes like the conventional multi-view stereoscopic image shown in FIG. 13 (that is, the recognized stereoscopic image suddenly changes). ), A natural stereoscopic image that changes little by little as the observation position changes can be realized, and the clarity of the image visually recognized by the observer is maintained above a certain level.

  Note that, as described above, the images of the right eye EY1 and the left eye FY2 that are visible to the observer OB are images that are slightly inaccurate from the actual images. However, the line-of-sight direction in which each eye views each pixel is substantially along the line-of-sight direction of the pixel-specific viewpoint CM of the pixel, as shown in the lower side of FIG. That is, the line-of-sight direction in which the left eye EY2 views each pixel PE1, PE2,... Of the image A is that of the pixel-specific viewpoints CM1, CM2,. The direction is substantially along the line-of-sight direction. Similarly, for the right eye EY1, the line-of-sight direction for viewing the pixels PE11, PE12,... Of the image B is the pixel-specific viewpoints CM11, CM12,.・ ・ The direction is almost along the line of sight. For this reason, the image visually recognized by the observer has clarity that can be visually recognized as an image, although it is slightly inaccurate. Further, as described above, even if the position of the observer changes, the clarity of the visually recognized image is maintained at a certain level or higher.

  In addition, the stereoscopic image recognized by the observer in the present embodiment can have the same resolution as the conventional multi-view stereoscopic image. For example, in the stereoscopic video display apparatus 200A shown in FIG. 2, the lens pitch L is 3 to 4 times the pixel pitch S in subpixel units. Therefore, in such a stereoscopic video display device 200, a resolution of about 1/3 to 1/4 of the resolution of the pixel panel 20, that is, a resolution comparable to that of a conventional 3-4 eye stereoscopic video is obtained. become.

  As described above, in the stereoscopic vision according to the present embodiment, although the accuracy of the recognized stereoscopic video is slightly lacking, it has the same resolution as the conventional multi-view stereoscopic video and has a viewpoint ( It is possible to realize a natural stereoscopic video image with a large number of views.

  The method for generating a stereoscopic image according to the present embodiment has been described above based on the method (I). Next, a method for generating a stereoscopic image of the present embodiment from a planar image from one viewpoint and its depth image based on the method (II) will be described. Here, method (II) is a) When an image created with an existing high-quality CG tool can be easily three-dimensionalized without using a dedicated renderer; b) When a stereoscopic image is created from a real image C) Depth image creation can be easily performed when drawing with a CG tool, d) When obtaining a depth image corresponding to a real image, A range image camera is required, and its accuracy and resolution are not sufficient at present, but its research is being actively pursued because it can be expected to improve in the future.

  For example, as described in Japanese Patent Application Laid-Open No. 2000-78611, when a binocular stereoscopic image is generated from a planar image and a depth image, the planar image is respectively converted based on the depth value of the depth image. By shifting in the horizontal direction, a parallax image for the left eye and a parallax image for the right eye are generated, and the generated parallax image is interleaved to generate a stereoscopic image. The depth image here is image data representing the depth value at the corresponding position of the planar view image in black and white gradation. If the depth value for each pixel is known, the position in the parallax image where each pixel of the planar image is converted can be uniquely determined from the size of the screen, the observation distance of the observer, and the distance between both eyes. .

  For example, as shown in FIG. 15, first, a position P1 where a line extending perpendicularly from the dot position P0 of the planar view image in the direction of the observer's line of sight and the depth value intersect is set as a position where the pixel is to be displayed. decide. Next, when generating a parallax image for the left eye, a point where the extension line connecting the left-eye viewpoint camera corresponding to the assumed observation position 40 and the position P1 intersects the display surface is set as the pixel movement position P2 for the left eye. . On the other hand, when generating a parallax image for the right eye, a point where the extension line of the line connecting the right eye viewpoint camera corresponding to the assumed observation position 40 and the position P1 intersects the display surface is the pixel movement position P3 for the right eye. And As described above, for each pixel in the planar view image, a position to be displayed is determined based on the depth value, and the movement position of the pixel is determined based on the position and the positions of the left and right viewpoint cameras. Thus, a parallax image for the left eye and a parallax image for the right eye are generated. Then, a binocular stereoscopic image can be generated by interleaving the generated two parallax images.

  On the other hand, in this embodiment, for each pixel on the display surface, (1) a representative point of the pixel (for example, the center of the pixel) and a representative of an optical element (for example, a lens) that gives directivity to the emitted light of the pixel. A direction corresponding to the direction of the light beam in the three-dimensional virtual space is determined based on the direction of the light beam passing through the point and the viewpoint position in the three-dimensional virtual space, and (2) the determined direction and depth image Based on the above, the dot representing the color information of the nearest object in the direction in the three-dimensional virtual space is determined from the planar view image, and the color information of the determined reference image dot is the color of the pixel A stereoscopic image is generated by using the information. Since the method for determining the direction corresponding to the direction of the light beam in (1) is substantially the same as the method (I) described above, the following description is mainly based on (2) the determined direction and the depth image. A method for determining the color information will be described in detail.

First, the outline of the principle will be described.
For example, as shown in FIG. 16, when attention is paid to the pixel at the position P12 on the display surface, the line of sight V and the pixel-specific viewpoint CM11 for the pixel are determined in the same manner as in the method (I). Next, the position where the trajectory in the direction corresponding to the determined line of sight V (the line passing through the position P12 and the pixel-specific viewpoint CM in the figure, hereinafter referred to as “pixel-specific line-of-sight direction trajectory”) intersects the depth value. P11 is determined as a position where the pixel is to be displayed. Then, the position of the intersection P10 between the line extending perpendicularly to the display surface from the position P11 and the display surface is acquired, and the color information of the dot of the planar image corresponding to the acquired position is obtained from the pixel at the position P12. Obtained as color information.

  Next, a case where there are a plurality of intersections between the per-pixel gaze direction locus and the depth value will be described. For example, when paying attention to the pixel at the position P24 on the display surface, the pixel-specific line-of-sight direction locus and the depth value intersect at the intersection point P21, the intersection point P22, and the intersection point P23. In this case, an intersection point closest to the observer, that is, an intersection point P21 closest to the individual viewpoint CM22 is determined as a position where the pixel is to be displayed. Then, the position of the intersection P20 between the line extending perpendicularly to the display surface from the intersection P21 and the display surface is acquired, and the color information of the dot of the planar image corresponding to the acquired position is obtained from the pixel at the position P24. Obtained as color information.

  In this way, the line of sight V and the pixel-specific viewpoint CM for each pixel on the display surface are determined, and the pixel-specific line-of-sight direction locus is determined from the direction of the line of sight V and the pixel and pixel-specific viewpoint CM. Among the points where the depth value intersects, the position of the intersection closest to the pixel-specific viewpoint CM is determined as the position where the pixel is to be displayed. Then, the position of the intersection of the line extending perpendicularly to the display surface from the determined position and the display surface is acquired, and the color information of the dot of the planar image corresponding to the acquired position is the color information of the pixel. By doing so, a stereoscopic image can be generated.

Next, a method for generating a stereoscopic image based on a specific planar image and its depth image will be described. In the following, for the sake of simplicity of explanation, a case where the planar view image is a gray scale image (monochrome image) will be described as an example. FIG. 17A is a depth image in which the depth value of the corresponding position of the planar image is represented by black and white gradation, and FIG. 17D is a standard planar image. FIG. 6B is a depth information model generated based on the depth value of the depth image, and is obtained by converting the Z value in a certain scan line (for example, Y = y ₀ ) into xz coordinates. This depth information model indicates that the greater the Z value, the closer to the observer, and one dot corresponds to one pixel. The depth image is an image created by a parallel projection method assuming that the observer's viewpoint is at infinity.

FIG. 4C is a diagram simply showing a vertical cross section with respect to the display surface 22, and the intersections X1, X2 of the representative rays PR1, PR2 emitted from the representative points of the pixels PE1, PE2 and the depth information model. It is a figure which shows the correspondence with the planar view image shown to the same figure (d). As shown in FIG. 6C, the intersection point is obtained from the representative ray PR1 emitted from the pixel PE1 (that is, the ray in the direction opposite to the line of sight V and equivalent to the line-of-sight line direction trajectory) and the depth information model. X1 (z ₁ , x ₁ ) is determined. The color information of the dot of the planar view image corresponding to the position of this intersection X1 is acquired. That is, the position coordinate (x ₁ , y ₀ ) of the corresponding planar image is determined from the intersection point X1 (z ₁ , x ₁ ), and the color information acquired from the dot at this position coordinate is used as the color information of the pixel PE1. Ask. Similarly, the position coordinates (x ₂ , y ₀ ) of the corresponding planar view image are determined from the intersection X2 (z ₂ , x ₂ ) between the representative ray PR2 emitted from the pixel PE2 and the depth information model. Then, the color information acquired from the dot at the position coordinates (x ₂ , y ₀ ) of the planar view image is obtained as the color information of the pixel PE2.

  Furthermore, a method for obtaining color information when there are a plurality of intersections between the per-pixel line-of-sight direction trajectory and the depth value of the depth information model will be described in detail. FIG. 18 is a diagram schematically showing a vertical section with respect to the display surface 22, and shows a depth information model in a scan line corresponding to the vertical section. Further, the black circle (●) on the display surface 22 indicates the representative point of each pixel, and the x coordinate of each representative point is the same as the x coordinate of the depth value. That is, one pixel corresponds to one pixel. In this method, the direction of the representative ray emitted from the representative point of each pixel is calculated from the positional relationship between the lens and the pixel, and this is used as the direction of the line of sight V. In addition, description of the lens is abbreviate | omitted in the figure.

  FIG. 19 is a diagram showing representative rays PR1 to PR3 corresponding to the pixels PE1 to PE3 by focusing on the pixels PE1, PE2, and PE3 among the pixels shown in FIG. The representative rays PR1 to PR3 show three typical patterns. That is, the representative ray PR1 is a pattern that is emitted from the pixel PE1 in the diagonally upward left direction, the representative ray PR2 is a pattern that is emitted from the pixel PE2 in the diagonally upward right direction, and the representative ray PR3 is vertically upward from the pixel PE3. The pattern which injects in the direction is shown. First, a description will be given of how to obtain the color information of the pixel PE1 when the pattern in which the representative light beam is emitted obliquely upward to the left, that is, when the inclination of the trajectory direction for each pixel is negative.

  FIG. 20 is a diagram illustrating an intersection between V1 and the depth information model, where V1 is the line-of-sight line direction locus for each pixel. In addition, the lens and the pixel are omitted. First, intersection determination between the line-of-sight line direction trajectory V1 for each pixel and all line segments of the depth information model is performed to obtain each intersection (three intersections in the figure). Then, the intersection having the largest depth value (Z value) among all the intersections (that is, the closest intersection from the pixel-specific viewpoint CM, hereinafter referred to as “target intersection”) is acquired as an intersection for obtaining color information. . Then, similarly to the method described with reference to FIGS. 17C and 17D, the color information of the dot corresponding to the position of the target intersection among the dots of the planar image is acquired, and this is used as the color information of the pixel PE1. To do. In the figure, the intersection X1 is the target intersection.

  Similarly, when obtaining the color information of the pixel PE2, a crossing determination is performed with respect to the line-of-sight line direction locus V2 for the pixel PE2 and all line segments of the depth information model, and the Z value is the largest among all obtained intersections. Get the target intersection. And the color information of the dot corresponding to the position of the object intersection among the dots of the planar view image is acquired, and this is used as the color information of the pixel PE2. Further, when obtaining the color information of the pixel PE3, since the line of sight V is perpendicular to the pixel PE3, the color information of the dot corresponding to the position of the pixel PE3 is obtained from the dots of the planar view image, and this is obtained. The color information of the pixel PE3 is used.

  As described above, when obtaining the color information of each pixel, it is necessary to perform the intersection determination between the line-of-sight line direction trajectory for each pixel and all the line segments of the depth information model, and the color information of the three-dimensional virtual space is obtained. There was concern that it would take time to ask. In other words, in this embodiment, each pixel of the pixel panel represents the color information of the three-dimensional virtual space based on the direction of the light beam for each pixel, and therefore has as many viewpoints as the number of light beams. It can be said that there are as many viewpoints as there are. Therefore, as the number of viewpoints increases, there is a problem that much time is required for calculation processing for each pixel.

  When a moving image is generated in real time and displayed on a stereoscopic video apparatus, it is necessary to generate a stereoscopic image for each frame. Therefore, when time constraints are extremely large and a stereoscopic image cannot be generated in units of frames, a situation in which a moving image cannot be displayed on the stereoscopic video display device (so-called processing failure) may occur. In view of this, the present inventors have invented a method for obtaining a target intersection point in a short time by a convergence calculation process based on a pixel-specific line-of-sight direction locus for each pixel and depth value information.

Hereinafter, a method for obtaining the target intersection will be described.
Now, the depth value (Z value) of the depth information model in a certain scan line is represented by the function z = f _z (x), and the pixel-specific gaze direction locus in one pixel is the function z = ax + b (a and b are Constant). However, “x” indicates the x coordinate (pixel position) of the pixel of the pixel panel 20.

For simplicity, it is assumed that the function z = f _z (x) and the function z = ax + b intersect only at one point. An example of the graph of the function z = f _z (x) and the function z = ax + b in this case is shown in FIG.

Here, if f (x) = f _z (x) − (ax + b), the pixel position x _m corresponding to the intersection of the function z = f _z (x) and the function z = ax + b is expressed by the equation f (x ) = 0, it can be obtained as a solution, resulting in a problem of finding a solution of f (x) = 0 by numerical calculation.

  Here, it is generally not easy to find a solution of f (x) = 0. However, it is possible to obtain an approximate solution based on a known method. Therefore, in this embodiment, an approximate solution with f (x) = 0 is calculated using a method called Newton's method.

FIG. 22 is a diagram showing the principle of approximate solution calculation by the Newton method.
First, an arbitrary pixel position x = x ₀ is set as an initial value, and a point where a tangent line at a point (x ₀ , f (x ₀ )) on the function f (x) intersects the x axis is calculated as x ₁ . .

Next, a point at which the tangent line at the point (x ₁ , f (x ₁ )) on the function f (x) intersects the x axis is calculated as x ₂ . Thereafter, when x ₃ , x ₄ , x ₅ ,... Are calculated by repeating the same procedure, x _n finally converges to x _m as a solution as long as the initial value is appropriate. Here, the update formula of x _n is formulated as the following formula (10).
x _{n + 1} = x _n −f (x _n ) / f ′ (x _n ) (10)

In the Newton method, the calculation is generally terminated when the condition of the following equation (11) is satisfied.
| (X _{n + 1} −x _n ) / x _n | <ε (11)
Here, ε is a constant that determines the calculation accuracy.

As apparent from the equation (10), when updating x _n , it is necessary to obtain f ′ (x _n ). Here, since f ′ (x _n ) = f _z ′ (x _n ) −a and the value of “a” that is the inclination of the pixel-specific line-of-sight direction locus is fixed, f _z ′ (x _n ) is If you ask for it.

In this case, f _z ′ (x _n ) may be calculated sequentially, but the differential value of the depth value (hereinafter referred to as “depth differential value”) is calculated in advance for all the dots in the planar view image. It's convenient if you keep it.

If the function z = f _z (x) that gives the depth value is given as an equation, the depth differential value can be calculated from the differential function _z ′ = f _z ′ (x) obtained by differentiating the function. it can. However, in a general image, the function z = f _z (x) that gives the depth value cannot be mathematically expressed, so it is difficult to obtain a function that gives the depth differential value. Therefore, for each dot, the difference between the depth value of the dot and the depth value of the adjacent dot adjacent to the dot is calculated as a depth difference value, which is used instead of the depth differential value.

  As described above, in the present embodiment, the target intersection between the pixel-specific line-of-sight direction locus and the depth value of each pixel is obtained using the Newton method. Newton's method is an algorithm whose solution convergence depends on the initial value, but it is characterized by quadratic convergence. Therefore, if the initial value is appropriate, the solution can be obtained at high speed. . Therefore, since the target intersection can be obtained in a short time, it is possible to increase the speed of stereoscopic image generation.

2. Stereoscopic Image Generation Device Next, a stereoscopic image generation device based on the above-described principle will be described. Such a stereoscopic image generation apparatus generates a stereoscopic image that realizes stereoscopic viewing of a moving image.

2-1. Configuration FIG. 23 is a block diagram illustrating a configuration of the stereoscopic image generating apparatus 1 according to the present embodiment.
The stereoscopic image generation device 1 includes an input unit 100, a stereoscopic video display device 200, a processing unit 300, and a storage unit 400.

  The input unit 100 receives an operation instruction from a user and outputs an operation signal corresponding to the operation to the processing unit 300. This function is realized by an input device such as a button switch, lever, joystick, dial, mouse, trackball, keyboard, tablet, touch panel, or various sensors.

  The stereoscopic video display device 200 is a display device that displays the stereoscopic image generated by the stereoscopic image generation unit 320 and allows the observer to recognize the stereoscopic video. In the present embodiment, for example, it is realized by the vertical lenticular stereoscopic image display device shown in FIG.

  The processing unit 300 performs various arithmetic processes such as control of the entire stereoscopic image generation apparatus 1 and image generation. This function is realized by, for example, an arithmetic device such as a CPU (CISC type, RISC type), ASIC (gate array, etc.) or a control program thereof. In particular, in the present embodiment, the processing unit 300 includes an object space setting unit 310 that sets an object space that is a three-dimensional virtual space, and a stereoscopic view that generates a stereoscopic image of the object space set by the object space setting unit 310. An image generation unit 320.

  The stereoscopic image generation unit 320 includes a pixel-specific viewpoint setting unit 322, a rendering unit 324, a model generation unit 325, and a color information setting unit 326, and the stereoscopic image generation program 410 stored in the storage unit 400. By executing the processing according to the above, a stereoscopic image of the object space set by the object space setting unit 310 is generated, and the generated stereoscopic image is displayed on the stereoscopic video display device 200.

  The pixel-specific viewpoint setting unit 322 sets the pixel-specific viewpoint CM in the object space with reference to the display device data 430 and the assumed observation position data 440. Specifically, for each pixel PE on the display surface 22 of the stereoscopic video display device 200, the corresponding lens 32 is determined with reference to the display device data 430 and the assumed observation position data 440. Then, the representative ray PR after passing through the representative point of the pixel PE and the principal point (specifically, the representative principal point) of the lens 32 corresponding to the pixel PE is calculated. The same line of sight with the direction reversed is set as the line of sight V of the pixel PE. At this time, the lens 32 and the line of sight V corresponding to the pixel PE are determined by a method according to the stereoscopic video display device 200. That is, the process is performed as described with reference to FIGS.

  The pixel-specific viewpoint setting unit 322 sets a pixel-specific viewpoint CM with the calculated line of sight V as the line-of-sight direction for each pixel PE. Further, the position of the pixel-specific viewpoint CM is set based on the setting reference position determined by the stereoscopic image generation unit 320. Specifically, for example, as illustrated in FIG. 11, each pixel viewpoint CM is set on the same plane parallel to the display surface 22.

  Here, the display device data 430 is data of configuration parameters of the stereoscopic video display device 200. FIG. 24 shows an example of the data configuration of the display device data 430. The display device data 430 includes a pixel pitch 431 of the pixel panel 20 constituting the stereoscopic video display device 200, a lens pitch 432 and a focal length 433 of the lens plate 30, an arrangement angle 434 and an arrangement of the lens plate 30 with respect to the pixel panel 20. The reference position 435 is stored.

  The arrangement angle 434 stores the value of the angle θ formed by the pixel pitch direction of the pixel panel 20 and the lens pitch direction of the lens plate 30. That is, the arrangement angle 434 is data indicating whether the stereoscopic image display device 200 is a vertical / oblique lenticular method. In the case of the vertical lenticular method, θ = 0 °, and in the case of the oblique lenticular method. Is other than θ = 0 °. The arrangement reference position 435 is the arrangement position of the lens plate 30 and stores the magnitude dx of the displacement of the lens plate 30 with respect to the horizontal direction from the reference position of the pixel panel 20. The display device data 430 is stored in advance as fixed data, but may be set according to a user input from the input unit 100.

  The assumed observation position data 440 is data of the assumed observation position 40. Specifically, the display surface of the pixel panel 20 of the stereoscopic video display device 200 and the assumed observer's viewpoint (assumed observation position). The value of the assumed observation distance D between 40 is stored. Alternatively, the position coordinates of the assumed observer viewpoint (assumed observation position) 40 may be stored. The assumed observation position data 440 is stored in advance as fixed data, but may be set by user input from the input unit 100. By making it possible to set the assumed observation position data 440 by user input, it is possible to easily cope with a case where it is desired to change the assumed viewpoint position. Further, the observation position may be automatically fed back (input) using a head tracking device as an input device.

  The pixel-specific viewpoint CM data of each pixel PE set by the pixel-specific viewpoint setting unit 322 is stored in the pixel-specific viewpoint data 450. FIG. 25 shows an example of the data configuration of the pixel-specific viewpoint data 450. The pixel-specific viewpoint data 450 stores the pixel 451 of the pixel panel 20 of the stereoscopic video display device 200 and the pixel-specific viewpoint vector 452 in association with each other. In the per-pixel viewpoint vector 452, the data of the line of sight V as data representing the corresponding per-pixel viewpoint CM is stored as a normalized vector. In actual calculation, the data of the line of sight V is used after being converted into a function z = ax + b of the pixel-specific line-of-sight direction locus. Therefore, the values of a and b may be calculated in advance and stored in the pixel-specific viewpoint data 450.

  The rendering unit 324 sets one virtual camera in the object space set by the object space setting unit 310, performs rendering by the Z buffer method or the like based on the viewpoint of the virtual camera, and generates rendering data 460.

  Based on the rendering data 460 generated by the rendering unit 324, the model generation unit 325 includes a depth information model 470 that is information on the depth value of each dot in the planar view image, and a depth differential information model that is information on the depth differential value. 480. If rendering is performed by the Z buffer method, the depth value of each dot remains in the Z buffer for hidden surface removal. Therefore, the depth information model 470 may be obtained using the value of this Z buffer.

  The color information setting unit 326 executes color information setting processing according to the color information setting program 420, and sets the color information of the stereoscopic image in the three-dimensional virtual space. Specifically, for each pixel PE, the pixel PE and the pixel-specific viewpoint (data of the line of sight V) with respect to the pixel PE are acquired from the pixel-specific viewpoint data 450 to calculate the pixel-specific line-of-sight direction locus, and this pixel-specific line-of-sight is obtained. Among intersection points between the direction locus and the depth information model 470 in the scan line corresponding to the position of the pixel PE, a target intersection point having the maximum Z value is calculated by a convergence calculation process. Then, the pixel position corresponding to the calculated position of the target intersection is acquired, and the color information of the dot of the planar view image corresponding to the pixel position is obtained as the color information of the pixel PE. The obtained color information of each pixel PE is written in the corresponding position of the stereoscopic image data 500 by the color information setting unit 326.

  The storage unit 400 stores a system program, data, and the like for causing the processing unit 300 to control the stereoscopic image generating apparatus 1 in an integrated manner, and is used as a work area of the processing unit 300. The executed calculation results, input data input from the input unit 100, and the like are temporarily stored. This function is realized by, for example, various IC memories, a hard disk, a floppy (registered trademark) disk, a CD-ROM, a DVD, an MO, a RAM, a VRAM, and the like.

  In particular, in the present embodiment, the storage unit 400 includes a stereoscopic image generation program 410 for causing the processing unit 300 to function as the stereoscopic image generation unit 320, display device data 430, assumed observation position data 440, and pixel-by-pixel. The viewpoint data 450, the rendering data 460, the depth information model 470, the depth differential information model 480, the pixel-corresponding dot data 490, and the stereoscopic image data 500 are stored. In addition, the stereoscopic image generation program 410 includes a color information setting program 420 for causing the stereoscopic image generation unit 320 to function as the color information setting unit 326 as a subroutine.

  The rendering data 460 is data generated by the rendering unit 324 and includes planar view image data 462. FIG. 26 shows an example of the planar image data 462. The planar image data 462 is image data in which color information (for example, RGB value, α value) is stored, and is a kind of reference image.

  The depth information model 470 is data generated by the model generation unit 325, and includes depth image data 472 in which the depth value of each dot of the planar view image is stored for each scan line. FIG. 27 shows depth image data 472 for the planar view image data 462 of FIG. FIG. 27B is a model diagram in which the depth value in the scan line AA of the depth image shown in FIG. In FIG. 9A, the larger the Z value (brighter), the closer to the observer.

  The depth differential information model 480 is data generated by the model generation unit 325, and includes depth differential image data 482 that stores the depth differential value of each dot of the planar view image for each scan line. FIG. 28 shows depth differential image data 482 with respect to the planar view image data 462 of FIG. FIG. 28B is a model diagram in which the depth differential value in the scan line BB of the depth differential image shown in FIG.

  The pixel-corresponding dot data 490 is data in which dots of a planar image for setting color information are stored for each pixel PE, and an example of the data configuration is shown in FIG. The pixel-specific corresponding dot data 490 stores the pixel 491 of the pixel panel 20 of the stereoscopic video display device 200 and the corresponding dot 492 that is a dot of the planar image corresponding to the pixel.

  The stereoscopic image data 500 stores image data for one frame (specifically, color information of each pixel) generated by the stereoscopic image generation unit 320.

2-2. Process Flow Next, the process flow will be described.
FIG. 30 is a flowchart showing a flow of stereoscopic image generation processing in the present embodiment. This process is a process of generating and displaying a stereoscopic image for each frame, that is, realizing a real-time moving image stereoscopic view. The stereoscopic image generation unit 320 loads the stereoscopic image generation program 410 in the storage unit 400. It is realized by executing.

  First, the stereoscopic image generation unit 320 determines a reference position (set reference position) that serves as a reference for the set position of the pixel-specific viewpoint CM (step S11). Next, the stereoscopic image generation unit 320 sets the viewpoint CM for each pixel PE in the object space by executing the process of loop A for each pixel PE on the display surface 22.

  In the loop A, the pixel-specific viewpoint setting unit 322 refers to the display device data 430 and the assumed observation position data 440 to select the lens 32 corresponding to the pixel PE that is the processing target (hereinafter, “the pixel”). Determine (step S13).

  Next, the pixel-specific viewpoint setting unit 322 outputs a ray (representative ray) PR after passing through the representative point of the pixel PE and the principal point (representative principal point) of the lens 32 corresponding to the pixel PE and the observation position. The vector which is calculated and has the same position as that of the representative ray PR and whose direction is reversed is set as the line of sight V (step S15). Then, the pixel-specific viewpoint setting unit 322 sets a pixel-specific viewpoint CM in which the direction of the line of sight V is the line-of-sight direction (step S17), and stores it as pixel-specific viewpoint data 450. Loop A is executed in this way.

  When the loop A process is performed on all the pixels PE on the display surface 22, the stereoscopic image generation unit 320 subsequently executes the loop B process for each frame.

  In the loop B, first, the object space setting unit 310 sets an object space (step S19), and sets a virtual camera in the set object space (step S21). Then, the rendering unit 324 performs rendering by the Z buffer method based on the set virtual camera, and generates rendering data 460 (step S23).

  Next, the model generation unit 325 generates a depth information model 470 that is information on the depth value of each dot of the planar view image based on the generated rendering data 460 (step S25).

Further, the model generation unit 325 generates a depth differential information model 480 that is information on the depth differential value of each dot of the planar view image based on the depth information model 470 generated in step S25 (step S27). Specifically, for each dot, the model generation unit 325 calculates a difference between the depth value of the dot and the depth value of an adjacent dot adjacent to the dot to obtain a depth difference value, and obtains the depth differential value. As a result, the depth differential information model 480 is generated. When the function z = f _z (x) for giving the depth value is given in advance as a mathematical expression, the model generation unit 325 determines each dot from the differential function _z ′ = f _z ′ (x) obtained by differentiating the function. The depth differential information model 480 is generated by calculating the depth differential value of.

  Subsequently, the color information setting unit 326 executes a color information setting process (see FIG. 31) for setting the color information of the pixel PE (the pixel) to be processed (step S29). When the color information setting process for all the pixels PE on the display surface 22 ends, the stereoscopic image generation unit 320 converts the image data for one frame stored in the stereoscopic image data 500 into the stereoscopic image. Is displayed on the stereoscopic video display device 200 (step S31).

  In this manner, the stereoscopic view of the moving image is realized by repeatedly executing the process of Loop B for each frame. Then, for example, when a stereoscopic image generation end instruction is input by inputting a stereoscopic image generation end instruction from the input unit 100, the stereoscopic image generation unit 320 ends the loop B, and in this embodiment The stereoscopic image generation process ends. In addition, when it takes time to generate an image or when it is desired to reuse the generated moving image later, the output destination may be a storage medium such as a hard disk instead of the stereoscopic video display device 200. . In addition, when the processing time is sufficient, output may be performed on both the stereoscopic video display device 200 and the storage medium.

Next, a flow of color information setting processing will be described with reference to FIG.
First, the color information setting unit 326 executes the process of loop C for each pixel PE on the display surface 22.

  In the loop C, the color information setting unit 326 determines whether the inclination of the pixel-specific gaze direction locus with respect to the pixel is vertical (step S33). When the inclination of the trajectory direction trajectory for each pixel is vertical (step S33; Yes), that is, when the same as PE3 in FIG. 19, the color information setting unit 326 is a planar view image corresponding to the xy coordinates of the pixel position. The corresponding dot 492 of the pixel is stored in the corresponding dot data 490 for each pixel, and the color information of the dot is set as the color information of the pixel (step S35).

On the other hand, when the inclination of the pixel-gaze direction path for the pixel is not perpendicular (step S33; No), the color information setting unit 326 obtains the x-coordinate of the pixel as the pixel position x ₀ (step S37). Then, the color information setting unit 326 calculates a per-pixel gaze direction locus z = ax + b based on the gaze direction of the pixel and the per-pixel viewpoint CM, and corresponds to the pixel position x ₀ acquired from the depth information model 470. Difference between the depth value “f _z (x ₀ )” of the dot in the planar image and the depth value “ax ₀ + b” of the dot in the planar image corresponding to the pixel position x ₀ acquired from the pixel-specific gaze direction locus _{"f (x 0) = f z} (x 0) - (ax 0 + b) " it is referred to as Z _a (step S39). However, in this case, since the initial value x ₀ is the x coordinate of the pixel, z = ax + b is a straight line passing through the pixel, and thus “ax ₀ + b = 0”. That is, “f (x ₀ ) = f _z (x ₀ )” may be Z _a .

The color information setting unit 326 also displays the depth differential value “f _z ′ (x ₀ )” of the dot of the planar view image corresponding to the pixel position x ₀ acquired from the depth differential information model 480 and the per-pixel gaze direction locus. The difference “f ′ (x ₀ ) = f _z ′ (x ₀ ) −a” from the slope “a” is set as Z _d (step S41). Then, the color information setting unit 326 calculates x _d by “x _d = x ₀ −Z _a / Z _d ” (step S43), and then executes the process of loop D a predetermined number of times.

Here, as the initial value when performing convergence calculation processing by the Newton method is described as providing a pixel position x ₀ of the pixel, the value may be changed as appropriate. For example, a plane parallel to the pixel plane and passing through the highest point of the depth model and the intersection of the pixel-specific gaze direction locus of the pixel may be obtained, and the x coordinate may be used as the initial position of the search. By doing in this way, when there are a plurality of intersections between the depth information model and the pixel-specific gaze direction locus, there is a high possibility that the nearest intersection can be acquired, and a more accurate stereoscopic image can be obtained.
Further, the processing of the loop D may not be executed a predetermined number of times but may be executed until the condition of Expression (11) is satisfied.

In loop D, the color information setting unit 326 uses the depth value “f _z (x _d )” of the dot of the planar view image corresponding to the pixel position x _d acquired from the depth information model 470 and the gaze direction locus for each pixel. The difference “f (x _d ) = f _z (x _d ) − (ax _d + b)” from the depth value “ax _d + b” of the dot in the planar image corresponding to the acquired pixel position x _d is Z _a To do. Further, the color information setting unit 326 displays the depth differential value “f _z ′ (x _d )” of the dot of the planar image corresponding to the pixel position x _d acquired from the depth differential information model 480 and the per-pixel gaze direction locus. The difference “f ′ (x _d ) = f _z ′ (x _d ) −a” from the slope “a” of Z is _defined as Z _d (step S45).

Then, the color information setting unit 326 updates x _d according to “x _d ← x _d −Z _a / Z _d ” (step S47). Loop D is executed in this way.

When the processing of the loop D is completed, the color information setting unit 326 stores the dot of the planar view image corresponding to the xy coordinate of the pixel position _{xd in} the corresponding dot data 490 for each pixel as the corresponding dot 492 of the pixel, and the dot Is set as the color information of the pixel (step S49), and the process proceeds to the next pixel.

  As described above, the color information of the stereoscopic image for one frame can be set by repeatedly executing the loop process for each pixel. These processes may be performed in parallel for each pixel or every several pixels.

2-3. As described above, according to the present embodiment, the pixel-specific viewpoint CM is set for each pixel PE on the display surface 22, and the color information of the object space in the line-of-sight direction of the set pixel-specific viewpoint CM is obtained. A stereoscopic image is generated by using color information.

  Accordingly, since the viewpoints for each pixel as many as the number of pixels PE are set, that is, as many viewpoints (views) as the number of pixels PE exist, an assumed observation position (as in conventional multi-view stereoscopic viewing) ( It is not necessary that the eyes have to be positioned at the individual viewpoint), and stereoscopic viewing is possible at any position within a certain area as in the conventional super multi-view system.

  Further, the pixel-specific viewpoint CM of each pixel PE determines the lens 32 corresponding to the pixel PE based on the assumed observation position 40, and represents the representative point of the pixel PE and the principal point of the lens 32 corresponding to the pixel PE. It is determined that the reverse direction of the light beam (representative light beam) PR after passing through is the line-of-sight direction. Therefore, the line-of-sight direction of the pixel-specific viewpoint CM corresponding to each pixel PE visually recognized by the observer through the lens 32 is substantially along the observer's line-of-sight direction. The color information) is not an accurate image (color information) viewed from the position, but is clear enough to be sufficiently visually recognized.

  Further, since the resolution of the image is a resolution that can be separated by the lens plate 30, the resolution is comparable to that of the conventional multi-view stereoscopic view, and the remarkably degraded resolution as in the conventional super multi-view method is not observed. Does not occur.

  Furthermore, the present embodiment generates a stereoscopic image to be displayed on a stereoscopic video display device that does not match the pitch. In other words, stereoscopic viewing is possible even with a stereoscopic video display device that does not match the pitch. Therefore, there is no need to manufacture a lenticular lens plate having the same pitch for each display, and a stereoscopic image display device can be manufactured by applying one lenticular lens plate to another display having a different pixel pitch. The cost for manufacturing the device can be greatly reduced.

  In the present embodiment, convergence calculation processing by Newton's method is performed when obtaining a target intersection of each pixel's line-of-sight direction trajectory and depth value, but since Newton's method is an algorithm having quadratic convergence, The target intersection can be obtained in a short time. Accordingly, it is possible to realize a high-speed stereoscopic image generation.

  Furthermore, since the depth differential value is calculated and stored in advance for all the dots of the planar view image, it is not necessary to sequentially calculate the depth differential value in the Newton method calculation.

  Further, many recent GPUs (Graphics Processing Units) are equipped with a function capable of parallel processing in units of several pixels. Also in the method of the present embodiment, if these GPUs are used, the color information setting processing of FIG. 31 can be parallelized and drawing can be performed at high speed.

  In this case, it is desirable that the processing for each pixel is completed in about the same time. This is because a pixel that has completed processing earlier must wait for a pixel that takes the longest processing time.

  However, in the present embodiment, the processing time for each pixel can be made equal by setting the number of repetitions of the Newton method to the same number or almost the same number for all pixels. That is, it can be said that it is suitable when performing parallel processing for each pixel.

3. Modifications The application of the present invention is not limited to the above-described embodiment, and can be appropriately changed without departing from the spirit of the present invention. For example, the following modification is mentioned.

3-1. Calculation of Depth Derivative Value In the present embodiment, the depth differential value of all the dots in the planar view image is calculated in advance and the depth differential information model is generated before performing the color information setting process. However, instead of generating the depth differential information model in advance, the depth differential value may be calculated sequentially when performing the convergence calculation process by the Newton method in the color information setting process.

FIG. 32 is a flowchart showing the flow of color information setting processing in this case. However, the same steps as those in the color information setting process of FIG. 31 are denoted by the same reference numerals.
In the color information setting process in FIG. 32, after acquiring the Z _a in step S39, based on the depth information model 470, and the depth value of the dot of the plan view image corresponding to the pixel position x _0, adjacent adjacent to the dots A difference Z _s from the depth value of the dot is calculated (step S51).

Then, after calculating x _d by “x _d = x ₀ −Z _a / (Z _s −a)” (step S53), the processing of loop E is executed a predetermined number of times.
In the loop E, first, the depth value “f _z (x _d )” of the dot in the planar image corresponding to the pixel position x _d acquired from the depth information model 470 and the pixel position acquired from the per-pixel gaze direction locus. the difference between the depth value of the dot of the plan view image corresponding to the x _d "ax _d + b", _{"f (x d) = f z} (x d) - (ax d + b) " is referred to as Z _a (step S55) .

Next, based on the depth information model 470, a difference Z _s between the depth value of the dot of the planar image corresponding to the pixel position _xd and the depth value of the adjacent dot adjacent to the dot is calculated (step S57). Then, x _d is updated according to “x _d ← x _d −Z _a / (Z _s −a)” (step S59). Loop E is executed in this way.

3-2. Convergence Calculation Processing In the present embodiment, the target intersection point has been described as being obtained by convergence calculation processing by Newton's method, but the target intersection point may be obtained by convergence calculation processing by the scissors method, bisection method, secant method, or the like. In this case, since the depth differential value is not necessary in the convergence calculation process, it is not necessary to generate the depth differential information model.

3-3. Color Image In the above-described embodiment, as an example of generating a stereoscopic image from a planar image and its depth image, the case where the planar image is a grayscale image has been described. However, the present invention can be applied. However, this is not a limitation. For example, the planar image may be a color image having RGB values (the luminance values of the three primary colors of red, green, and blue). In this case, for each subpixel, the line of sight V is determined to set a pixel-specific viewpoint, Intersection determination between the line-of-sight line direction trajectory for each subpixel and the depth information model is performed. Then, the target intersection for each subpixel is acquired, and the color information of the subpixel corresponding to the target intersection from the planar view image is used as the color information of the subpixel. Therefore, in this case, the depth image data preferably has a depth value for each subpixel. Of course, a stereoscopic image can be generated (although the image quality is slightly reduced) even if each subpixel has a format in which depth data is shared for each pixel to which the subpixel belongs.

3-4. Oblique Lenticular Stereoscopic Video Display Device In the above-described embodiment, the case of a stereoscopic video display device using a vertical lenticular method as the optical element group has been described, but this is applied to a diagonal lenticular stereoscopic image display device. You may do it.

  FIG. 33 is a diagram illustrating a schematic structure of an oblique lenticular stereoscopic image display device 200B. FIG. 4A shows a cross-sectional view in the horizontal direction (horizontal scanning direction) with respect to the display surface of the stereoscopic video display apparatus 200B, and FIG. 4B shows a plan view seen from the observer side.

  In the oblique lenticular stereoscopic image display device, the direction of the principal dotted line 36 of the lens plate 30 (the end portion 32a of the lens 32) is arranged obliquely with respect to the pixel arrangement direction of the pixel panel 20, and thus the lens plate 30 is provided. It is known to disperse moire generated in an image that is visually recognized when the pixel panel 20 is viewed through the pixel panel 20.

  As shown in the drawing, the stereoscopic video display device 200B is mainly composed of a backlight 10 that is a plate-like body and arranged in parallel to each other, and the pixels, like the vertical lenticular stereoscopic video display device 200A described above. A panel 20 and a lens plate 30 are provided.

In the oblique lenticular stereoscopic image display device 200B, the lens plate 30 has a pixel array direction (vertical scanning direction) in which the direction of the main dotted line 36 is the vertical direction of the pixel panel 20, as shown in FIG. With respect to the angle θ. Accordingly, the lens pitch (lens width along the pixel pitch direction of the pixel panel 20) M in the cross-sectional view shown in FIG.
M = L / cos θ (12)

The lens plate 30 is arranged so that the lens pitch M and the pixel pitch S of the pixel panel 20 in the cross-sectional view shown in FIG. That is, the following equation (13) does not hold.
M = n · S (13)
However, n is a natural number.

  Next, a method for determining the line of sight V of each pixel PE on the display surface 22 in the oblique lenticular method will be described with reference to FIGS. FIG. 34 is a partial schematic perspective view of an oblique lenticular stereoscopic image display apparatus 200B. FIG. 35 is a schematic three-view diagram of the stereoscopic video display device 200B. FIG. 35A is a cross-sectional view (transverse cross-sectional view) at the CC position parallel to the xz plane of FIG. 35 (b) shows a cross-sectional view (longitudinal cross-sectional view) at a DD position parallel to the yz plane of FIG. 34, and FIG. 35 (c) shows an xy plan view. Show. The pixel panel 20 and the lens plate 30 are arranged in parallel with a focal length F of each lens 32 of the lens plate 30.

  First, in FIG. 35B, the y coordinate of the intersection point between the straight line LN2 connecting the representative point of the target pixel PE and the assumed observation position 40 and the principal point surface 35 of the lens plate 30 is calculated. The calculated y coordinate is assumed to be “y2”. However, FIG. 4B is a cross-sectional view passing through the representative point of the target pixel PE.

  Next, in FIG. 35A, each lens 32 is projected onto the display surface 22 from the assumed observation position 40, and the display surface 22 is divided into projection areas of each lens 32. Then, the corresponding lens 32 is determined depending on which projection region the representative point of the target pixel PE belongs to. However, FIG. 6A is a cross-sectional view in which the y coordinate is “y2” calculated previously.

  In FIG. 6A, the display surface 22 includes a projection area 26-10 of the lens 32-10, a projection area 26-11 of the lens 32-11, a projection lens 32-12 of the lens 32-12,.・ It is divided into Since the representative point of the target pixel PE belongs to the projection area 26-10, the lens 32 corresponding to the target pixel PE is the lens 32-10.

  Subsequently, a point whose y coordinate is “y2” is calculated from the main dotted line 36 of the lens 32 corresponding to the target pixel PE, and this is set as a representative main point 36d. Then, the representative ray PR after passing through the representative point of the target pixel PE and the representative principal point 36d is calculated, and the line of sight V having the same position and the opposite direction as the representative ray PR is the line of sight V of the target pixel PE. And Then, by performing intersection determination with the depth information model based on the determined line of sight V, color information for each pixel is set, and stereoscopic image data is generated.

3-5. Lens Plate In the above-described embodiment, the case of a stereoscopic video display device using a lenticular lens array as an optical element group has been described, but this is performed using a (3-5-1) eyelet lens array. It may be a stereoscopic video display device, (3-5-2) a stereoscopic video display device using a parallax barrier array, or (3-5-3) a stereoscopic video image using a pinhole array. A display device may be used. In this case, the line of sight V of each pixel PE on the display surface 22 is determined as follows.

3-5-1. Spider-eye lens array The spider-eye lens array is a lens array (lens plate) in which lattice-shaped unit lenses are continuously arranged (connected) vertically and horizontally as shown in FIG. In the stereoscopic image display device using the eyelid lens array, the eyelet lens array is arranged so that the lateral connection direction of the unit lenses is parallel to the pixel pitch direction (horizontal scanning direction) of the pixel panel 20. Is done. At this time, the eyelet lens array is designed so that the lens pitch of the unit lens and the pixel pitch of the pixel panel 20 do not match. That is, if the lens pitch of the unit lens is L, Expression (9) does not hold.

  FIG. 37 is a diagram for explaining a method of determining the line of sight V in the stereoscopic video display device 200C using the eyelet lens array, and shows a schematic three-view diagram of the stereoscopic video display device 200C. FIG. 4A shows a horizontal scanning direction sectional view parallel to the xz plane passing through the representative point of the target pixel PE, and FIG. 4B shows the yz plane passing through the representative point of the target pixel PE. A parallel sectional view in the vertical scanning direction is shown, and FIG. 8C shows an xy plan view. The assumed observation position 40 is “front”.

  First, as shown in FIGS. 2A and 2B, from the assumed observation position 40, each unit lens 62 of the lens plate 60 realized by the eyelet lens array is projected onto the display surface 22 of the pixel panel 20. (Ie, by a straight line from the assumed observation position 40 toward the end of each unit lens 62), the display surface 22 is divided into projection areas of each unit lens 62. The unit lens 62-1 in the projection area to which the representative point of the target pixel PE belongs is set as a unit lens corresponding to the target pixel PE. Then, the light beam (representative light beam) PR after passing through the representative point of the target pixel PE and the principal point (center of the unit lens) of the unit lens 62-1 corresponding to the target pixel PE has the same position and reverse direction. The line of sight that is set as the line of sight V of the target pixel PE.

  In FIG. 38, the lens plate 60 is arranged so that the lateral connection direction of the unit lenses 62 is parallel to the pixel pitch direction of the pixel panel 20, as shown in FIG. The horizontal connecting direction of the eyelet lens array and the pixel pitch direction of the pixel panel 20 may be arranged obliquely so as to form an angle θ.

  Further, the unit lens constituting the eyelet lens may be a triangle or a polygon such as a hexagon shown in FIG. 39 instead of the lattice shape (square). In any case, the method of determining the line of sight V of each pixel PE on the display surface 22 is the same as that shown in FIG.

3-5-2. As shown in FIG. 40, the parallax barrier array is provided with a large number of slit-like barrier openings (optical elements) at equal intervals for transmitting light to a light shielding plate (barrier) that shields light. It is what was done. That is, it is a barrier array in which unit parallax barriers having slit-like barrier openings are connected, and directivity is given to light rays (emitted light rays) emitted from each pixel PE of the pixel panel 20 by the barrier openings. As a result, it is possible to recognize a stereoscopic image in the same manner as a stereoscopic image display device using a lenticular lens array. In the figure, a black or gray portion is a barrier portion (light-shielding portion). At this time, the parallax barrier array is arranged so that the barrier opening is parallel to the vertical scanning direction of the pixel panel 20. Further, the parallax barrier array is designed so that the above-described formula (9) is not satisfied (that is, the pitch does not match) when the unit parallax barrier pitch is L.

  Therefore, in this case, the line of sight V of each pixel PE can be determined in the same manner as in the lenticular lens array in the above-described embodiment. Specifically, when the parallax barrier array is arranged so that the barrier opening is parallel to the vertical scanning direction of the pixel panel 20, the vertical lenticular stereoscopic image display device 200 in the above-described embodiment is used. Equivalent to. Further, when the parallax barrier array is arranged obliquely so that the barrier opening forms an angle θ with respect to the vertical scanning direction of the pixel panel 20, it corresponds to the oblique lenticular stereoscopic image display device 200B.

  Also, the parallax barrier array may be an oblique parallax barrier in which barrier openings are formed obliquely as shown in FIG. When such an oblique parallax barrier is used, it corresponds to the case where the above-described parallax barrier array (see FIG. 40) is arranged obliquely with respect to the pixel panel 20.

  Further, as shown in FIG. 42, the parallax barrier array may be a staircase parallax barrier array in which a plurality of unit parallax barriers connected in the horizontal direction are arranged in a staircase pattern with a predetermined amount shifted in the vertical direction. good. In the stereoscopic video display device using such a staircase parallax barrier, the staircase parallax barrier is arranged so that the barrier opening is parallel to the vertical scanning direction of the pixel panel 20.

3-5-3. As shown in FIG. 43, the pinhole array has a large number of hole-shaped pinholes (optical elements) that allow light to pass through a blocking plate (barrier) that blocks light. . That is, it is a barrier array in which lattice-shaped unit pinhole barriers having pinholes are continuously arranged (connected) vertically and horizontally, and directivity is given to the emitted light of each pixel PE of the pixel panel 20 by the pinhole. . In the figure, the gray portion is a light shielding portion (barrier portion). At this time, the pinhole array is arranged so that the horizontal direction of the unit pinhole barrier is parallel to the pixel pitch direction (horizontal scanning direction) of the pixel panel 20. Further, the pinhole array is designed so that equation (9) does not hold when the pitch of the unit pinhole barrier is L.

  In the stereoscopic video display apparatus using the pinhole array, the line of sight V of each pixel PE of the pixel panel 20 is determined as follows. That is, each unit pinhole barrier of the pinhole array is projected onto the display surface 22 of the pixel panel 20 from the assumed observation position 40, and the display surface 22 is divided into projection regions of each unit pinhole barrier. A unit pinhole barrier in the projection area to which the representative point of each pixel PE belongs is set as a unit pinhole barrier corresponding to the pixel PE. For each pixel PE, the line of sight is the same as the representative point of the pixel PE and the light ray (representative ray) after passing through the pinhole of the unit pinhole barrier corresponding to the pixel PE, and the direction is reversed. The line of sight V of the pixel PE is assumed.

  In addition, the unit pinhole barrier constituting the pinhole array may be a triangle or a polygon such as a hexagon shown in FIG. In any case, the method of determining the line of sight V of each pixel PE is the same as in the case of the lattice shape described above.

  The parallax barrier array or pinhole array usually does not clearly indicate the boundary line indicating the unit barrier. However, the slit or pinhole closest to any point on the barrier array is obtained, and the slit or pinhole is obtained. A boundary line for each unit barrier can be set by dividing the barrier plate into regions by holes.

3-6. In the above-described embodiment, the stereoscopic video display device 200 has been described as a part of the stereoscopic image generation device 1, but the stereoscopic video display device 200 is a separate device. The configured and generated stereoscopic image may be output and displayed on the connected stereoscopic video display device 200.

  FIG. 45 is a block diagram illustrating a configuration of a stereoscopic image generation device 3 in which the stereoscopic video display device is a separate device. The stereoscopic image generation device 3 includes an input unit 100, a processing unit 300, and a storage unit 400. In the stereoscopic image generation device 3, it is necessary to set the display device data 430 according to the connected stereoscopic video display device 200.

  Therefore, when generating an image, the stereoscopic image generation unit 320 firstly includes, for example, the model number of the stereoscopic video display device 200 input from the input unit 100 by the user (specifically, the pixel image is arranged on the pixel panel 20 or the upper surface thereof). Appropriate data is selected from each of the pixel panel data 510 and the lens plate data 520 stored in the storage unit 400 based on the model number of the lens plate 30 that is stored, and set as display device data 430.

  The pixel panel data 510 is a data table that stores physical parameters of the pixel panel 20 that can be used in the stereoscopic video display device 200. FIG. 46 shows an example of the data configuration of the pixel panel data 510. The pixel panel data 510 stores a model number 511 and a pixel pitch 512 in association with each type (type) of the pixel panel 20 that can be used in the stereoscopic video display device 200.

  The lens plate data 520 is a data table storing physical parameters of the lens plate 30 that can be used in the stereoscopic video display device 200. FIG. 47 shows an example of the data configuration of the lens plate data 520. The lens plate data 520 includes, for each type (type) of the lens plate 30 that can be used in the stereoscopic video display device 200, the model number 521, the lens pitch 522, the focal length 523, the arrangement angle 524, and the arrangement reference position. 525 are stored in association with each other.

  The input unit 100 may be a separate device from the stereoscopic image generation device. For example, as shown in FIG. 48, the input unit 100 and the stereoscopic video display device 200 are separate from the stereoscopic video generation device. That is, the stereoscopic image generation apparatus 5 includes a processing unit 300 and a storage unit 400. As such a configuration, for example, a case where a stereoscopic image is desired to be displayed on a display of a communication terminal device such as a mobile phone can be considered. That is, the lens pitch L, the arrangement angle θ of the optical element group (lens plate 30) with respect to the display (pixel panel 20), the focal point from the external device 7 such as a communication terminal device including the input unit 100 and the stereoscopic video display device 200. Information relating to the lens such as the distance F, information relating to the display such as the pixel pitch S, and information relating to the observer such as the assumed observation position 40 are transmitted to the stereoscopic image generating device 5. Then, the stereoscopic image generation device 5 generates a stereoscopic image based on the received information, transmits the generated stereoscopic image to the external device 7, or stores and stores the generated image, To the external device 7.

3-7. Storage of Image Data Image data may be stored in advance in a storage medium or the like. Specifically, the planar image data 462 and the depth image data 472 are created in advance and stored in a storage medium or the like. Then, these data are read from the storage medium or the like for each frame to obtain depth differential image data 482, and a stereoscopic image is generated. Further, the depth differential image data 482 may be created in advance and stored in a storage medium or the like.

  Further, when the planar view image data 462 and the depth image data 472 are stored, they may be stored in a compressed state as moving image data. In this case, the plane image data 462 and the depth image data 472 are reproduced by reading out and expanding the moving image data for each frame. Then, the depth differential image data 482 is obtained and a stereoscopic image is generated. The depth differential image data 482 may also be created in advance and stored in a storage medium or the like in a compressed state as moving image data.

  By adopting such a configuration, the stereoscopic image data 462 and the depth image data 472 that have been created and stored in advance are diverted to a variety of displays having different pixel panel and lens plate configurations. A visual image can be generated.

3-8. In addition, in the above-described embodiment, the case of generating a stereoscopic image to be displayed on a stereoscopic video display device including the pixel panel 20 such as a flat panel display has been described. The present invention can be similarly applied to a printed product using a printed material such as paper or a plastic card on which an image is printed. In such a printed product, an optical element group (lens array, pinhole array, etc.) is attached to the printing surface of the printed material on which the stereoscopic image is printed, and reflection of each dot of the printed image is performed. Stereopsis is realized by the directivity of the light beams provided by the optical element group.

  When generating an image to be printed on such a stereoscopic print processed product, the conventional method sets a certain number of viewpoints as in the case of a conventional multi-view stereoscopic image display device, and An image to be printed is generated by generating an image (per-pixel viewpoint) for each viewpoint and rearranging (interleaving) the images according to a certain pattern.

  In the printed product, the resolution (that is, the position and size of each dot of the image) of the stereoscopic image to be generated / printed can be freely set / changed according to the ready-made optical element. Further, by increasing the resolution of the stereoscopic image to be generated / printed, the resolution of the visually recognized stereoscopic video image can be made sufficiently large even when there are relatively many viewpoints. However, in that case, there is a problem that the data size of the stereoscopic image becomes large in proportion to the resolution.

  When the present embodiment is applied to such a printed product, each dot of the stereoscopic image to be printed is treated as each pixel PE of the pixel panel 20 so that the stereoscopic view is similar to the above-described embodiment. An image can be generated. Specifically, if the size of the print area of the stereoscopic image on the printed product and the resolution of the stereoscopic image to be printed are determined, the position and size of each dot of the stereoscopic image to be printed are unique. It is decided. For this reason, each dot of the stereoscopic image can be handled in the same manner as each pixel PE of the display panel 20 in the above-described embodiment.

  That is, for example, when using a lenticular lens array as the optical element group, first, a lens (optical element) corresponding to each dot of the stereoscopic image is determined. Next, for each dot, a ray (representative ray PR) after passing through the representative point (for example, the center) of the dot and the principal point of the lens corresponding to the dot is calculated, and the position of the ray is the same. A line of sight in the opposite direction is determined as the line of sight V of the dot. Thereafter, for each dot, a target intersection is obtained by a convergence calculation process based on the pixel-specific line-of-sight direction locus and the depth information model, and color information of the image dot corresponding to the target intersection is obtained from the planar view image. By doing so, a stereoscopic image is generated.

  Further, when the printed product uses a lens array as the optical element group, the pitch between each dot of the printed stereoscopic image is S, and the distance between the print surface and the principal point surface of the lens array is Is configured so that the distance G satisfies the above equation (1), it is possible to realize stereoscopic viewing with better image quality.

  According to the drawing method of the present embodiment, a stereoscopic image to be printed on a stereoscopic print processed product having the above-described effects is generated with a resolution lower than that of the conventional method, that is, with an image data size smaller than that of the conventional method. can do.

3-9. Projection Method In this embodiment, an example in which a stereoscopic image is generated based on a planar image and a depth image generated by a parallel projection method has been described. However, the present invention is not limited thereto, and a planar image generated by a perspective projection method is described. And the structure which produces | generates a stereoscopic vision image based on a depth image may be sufficient. In this case, a stereoscopic image is obtained by performing projective transformation of the line-of-sight vector for each pixel using the same projective transformation matrix used for generating a planar image and a depth image, as in the case of the parallel projection method described above. Can be generated.

3-10. Application to multi-view type In this embodiment, stereoscopic image generation by a method in which the pitch of the lens and the pixel does not match has been described. However, a method in which the pitch of the lens and the pixel is matched, that is, in multi-view type or super multi-view type Of course, it may be applied to stereoscopic image generation. Recently, even in viewpoint-based stereoscopic viewing methods such as multi-view and super-multi-view, there is a tendency that the number of set viewpoints is increasing. By using, high-speed drawing can be expected. Also in the IP method, high-speed drawing can be expected by applying the above-described stereoscopic image generation method.

The schematic diagram of the stereoscopic vision image generation in an embodiment. 1 is a schematic configuration diagram of a vertical lenticular stereoscopic image display device. Explanatory drawing of "pitch does not fit / does not fit". Explanatory drawing of the coordinate system setting with respect to a display surface. Explanatory drawing of the gaze determination in the assumption observing position "front and infinity" in the vertical lenticular system stereoscopic image display device. The figure which shows the state whose assumption observation position is a "front". Explanatory drawing of the gaze determination in the vertical lenticular system stereoscopic vision video display apparatus in case the assumption observation position is "the fixed position of a finite distance." Explanatory drawing of the gaze determination in the vertical lenticular system stereoscopic vision video display apparatus in case the assumption observation position is "the fixed position of a finite distance." Explanatory drawing of the refractive action by a lens plate. Explanatory drawing of the viewpoint setting according to pixel. Explanatory drawing of color information calculation of a pixel. Explanatory drawing that the color information of the visually recognized pixel is slightly inaccurate. The schematic diagram of the stereoscopic view of the conventional multi-view method (n eye type). The schematic diagram of the stereoscopic view in this embodiment. Explanatory drawing which produces | generates a binocular stereoscopic vision image from a planar view image and a depth image. Explanatory drawing of color information calculation of a pixel. Explanatory drawing of the color information calculation of the pixel based on a planar view image and a depth image. Explanatory drawing of the relationship between a display surface and a depth information model. Explanatory drawing of the relationship between a display surface, each pixel, and depth information model. Explanatory drawing of the intersection of a depth information model and a gaze direction locus according to pixel. Explanatory drawing of the intersection of a depth information model and a gaze direction locus according to pixel. Explanatory drawing of object intersection calculation by Newton method. 1 is a configuration diagram of a stereoscopic image generation apparatus. The data structural example of display apparatus data. The data structural example of viewpoint data classified by pixel. The figure which shows an example of planar view image data. The figure which shows an example of depth image data. The figure which shows an example of depth differential image data. The figure which shows an example of the dot data according to pixel. The flowchart which shows the flow of a stereoscopic vision image generation process. 6 is a flowchart showing a flow of color information setting processing. The flowchart which shows the flow of the color information setting process in a modification. 1 is a schematic configuration diagram of an oblique lenticular stereoscopic image display device. FIG. FIG. 4 is an explanatory diagram of line-of-sight determination when the assumed observation position is “a fixed position at a finite distance” in an oblique lenticular stereoscopic image display device. FIG. 4 is an explanatory diagram of line-of-sight determination when the assumed observation position is “a fixed position at a finite distance” in an oblique lenticular stereoscopic image display device. The top view of an eye lens array of an eyelid. Explanatory drawing of the gaze determination in the stereoscopic vision video display apparatus using a moth-eye lens array. The top view at the time of arranging the eye lens array of an eyelid diagonally. The top view of the eye lens array of the eyelid which made the unit lens hexagon. The top view of a parallax barrier. The top view of a diagonal parallax barrier. The top view of a staircase parallax barrier. The top view of a pinhole barrier. The top view of the pinhole barrier which made the unit pinhole barrier hexagonal. The block diagram of the stereoscopic image production | generation apparatus which used the stereoscopic vision video display apparatus as another apparatus. The figure which shows an example of pixel panel data. The figure which shows an example of lens plate data. The block diagram of the stereoscopic image production | generation apparatus which used the input part and the stereoscopic vision video display apparatus as another apparatus (external device). Explanatory drawing of the distance between a suitable pixel panel and a lens board. Explanatory drawing at the time of considering the binocular distance. The conceptual diagram of multi-view type stereoscopic vision. The conceptual diagram of super multi-view type stereoscopic vision. FIG. 3 is a conceptual diagram of IP system stereoscopic vision. The conceptual diagram of the stereoscopic vision of the ray reproduction method. Explanatory drawing of the distance between a display surface and an image formation surface being dependent on the focal distance of a lens.

Explanation of symbols

1, 3, 5 Stereoscopic image generation device 100 Input unit 300 Processing unit 310 Object space setting unit 320 Stereoscopic image generation unit 322 Per-pixel viewpoint setting unit 324 Rendering unit 325 Model generation unit 326 Color information setting unit 400 Storage unit 410 Stereo Visual image generation program 420 Color information setting program 430 Display device data 440 Assumed observation position data 450 Per-pixel viewpoint data 460 Rendering data 462 Plane view image data 470 Depth information model 472 Depth image data 480 Depth differential information model 482 Depth differential image data 490 Corresponding dot data for each pixel 500 Stereoscopic image data 200 Stereoscopic image display device 10 Backlight 20 Pixel panel 22 Display surface 30 Lens plate (lenticular lens plate)
32 Lens (micro lens)
40 Assumed observation position PE Pixel V Line-of-sight CM Per-pixel viewpoint

Claims (11)

A stereoscopic image in a three-dimensional virtual space to be displayed on a stereoscopic video display device comprising a pixel panel in which pixels are arranged and an optical element group that directs the emitted light of each pixel of the pixel panel. An image generation device that generates a reference image that is a planar image when the virtual space is viewed from a given viewpoint, and depth value information of each dot of the reference image that is based on the viewpoint position,
Regarding the color information of each pixel of the pixel panel, 1) based on the direction of the light beam passing through the optical element that directs the pixel and the emitted light beam of the pixel, and the viewpoint position in the three-dimensional virtual space, A direction calculation step for obtaining a direction corresponding to the direction of the light beam in the three-dimensional virtual space; and 2) a predetermined convergence calculation process based on the direction obtained in the direction calculation step and the depth value information. A dot determination step of determining, from the reference image, a dot representing color information of an object closest to the viewpoint in the direction in the original virtual space, and the dot of the reference image determined by the dot determination step By generating the stereoscopic image by using the color information as the color information of the pixel ,
The dot determination step includes a depth differential information calculation step of calculating depth differential information that is a differential value of a depth value of dots constituting the reference image, and the direction obtained in the direction calculation step and the depth value information And a dot representing the color information of the object closest to the viewpoint in the direction in the three-dimensional virtual space by a predetermined convergence calculation process based on the Newton method based on the depth derivative information calculated in the depth derivative information calculating step. An image generating apparatus comprising the step of determining from among the reference images . A stereoscopic image in a three-dimensional virtual space to be displayed on a stereoscopic video display device comprising a pixel panel in which pixels are arranged and an optical element group that directs the emitted light of each pixel of the pixel panel. An image generation device that generates a reference image that is a planar image when the virtual space is viewed from a given viewpoint, and depth value information of each dot of the reference image that is based on the viewpoint position,
Regarding the color information of each pixel of the pixel panel, 1) based on the direction of the light beam passing through the optical element that directs the pixel and the emitted light beam of the pixel, and the viewpoint position in the three-dimensional virtual space, A direction calculation step for obtaining a direction corresponding to the direction of the light beam in the three-dimensional virtual space; and 2) a predetermined convergence calculation process based on the direction obtained in the direction calculation step and the depth value information. A dot determination step of determining, from the reference image, a dot representing color information of an object closest to the viewpoint in the direction in the original virtual space, and the dot of the reference image determined by the dot determination step By generating the stereoscopic image by using the color information as the color information of the pixel ,
Before the determination process by the dot determination step, execute a depth differential information calculation step of calculating depth differential information that is a differential value of the depth value of each dot of the reference image,
In the dot determination step, a predetermined convergence calculation by the Newton method is performed using the direction obtained in the direction calculation step and the depth derivative information calculated in the depth derivative information calculation step in addition to the depth value information. A dot representing color information of an object closest to the viewpoint in the direction in the three-dimensional virtual space is determined from the reference image by processing.
An image generation apparatus characterized by that . The depth derivative information calculating step calculates the difference between the depth value of the dot and the depth value of an adjacent dot adjacent to the dot for each dot of the reference image to obtain a derivative value, thereby calculating the depth derivative information. the image generating apparatus according to claim 1 or 2, characterized in that the step of calculating the. The dot determination step, the image generation according to any one of claim 1 to 3 for the predetermined convergence calculation process most dot position of the viewpoint closer as the initial position in a direction determined by the direction calculation step apparatus. The stereoscopic image display apparatus has a viewing angle with respect to one optical element at a predetermined assumed observation position as a viewing angle with respect to the optical element λ, a viewing angle with respect to one pixel given directivity by the one optical element as a viewing angle with respect to pixel σ, and Λ: σ = n: m (where n is a natural number and m is a natural number less than Φ / λ) is not established when the viewing angle with respect to the stereoscopic image drawing region in the pixel panel is a paired drawing region viewing angle Φ. The image generation device according to any one of claims 1 to 4 . The stereoscopic video display device has a horizontal width of one optical element as L, a horizontal width of one pixel provided with directivity by the one optical element as S, and a stereoscopic image drawing area in the pixel panel. when the horizontal width is set to R in, L: S = o: p ( where, o is a natural number, p is a natural number less than R / L) to any one of claims 1-5, characterized in that is not established The image generation apparatus according to item. A stereoscopic image in a three-dimensional virtual space to be displayed on a stereoscopic video display device comprising a pixel panel in which pixels are arranged and an optical element group that directs the emitted light of each pixel of the pixel panel. An image generation device that generates a reference image that is a planar image when the virtual space is viewed from a given viewpoint, and depth value information of each dot of the reference image that is based on the viewpoint position,
The stereoscopic image display device has a viewing angle with respect to one optical element at a predetermined assumed observation position as a viewing angle with respect to the optical element λ, a viewing angle with respect to one pixel to which directivity is given by the one optical element as a viewing angle with respect to pixel σ, and A display device in which λ: σ = n: m (where n is a natural number and m is a natural number less than Φ / λ) is not established when the viewing angle with respect to the stereoscopic image drawing region in the pixel panel is a paired drawing region viewing angle Φ. Yes,
Regarding the color information of each pixel of the pixel panel, 1) based on the direction of the light beam passing through the optical element that directs the pixel and the emitted light beam of the pixel, and the viewpoint position in the three-dimensional virtual space, A direction calculation step for obtaining a direction corresponding to the direction of the light beam in the three-dimensional virtual space; and 2) a predetermined convergence calculation process based on the direction obtained in the direction calculation step and the depth value information. A dot determination step of determining, from the reference image, a dot representing color information of an object closest to the viewpoint in the direction in the original virtual space, and the dot of the reference image determined by the dot determination step An image generation apparatus that generates a stereoscopic image by using color information as color information of the pixel. A stereoscopic image in a three-dimensional virtual space to be displayed on a stereoscopic video display device comprising a pixel panel in which pixels are arranged and an optical element group that directs the emitted light of each pixel of the pixel panel. An image generation device that generates a reference image that is a planar image when the virtual space is viewed from a given viewpoint, and depth value information of each dot of the reference image that is based on the viewpoint position,
The stereoscopic video display device has a horizontal width of one optical element as L, a horizontal width of one pixel provided with directivity by the one optical element as S, and a stereoscopic image drawing area in the pixel panel. Is a display device in which L: S = o: p (where o is a natural number and p is a natural number less than R / L) is not established.
Regarding the color information of each pixel of the pixel panel, 1) based on the direction of the light beam passing through the optical element that directs the pixel and the emitted light beam of the pixel, and the viewpoint position in the three-dimensional virtual space, A direction calculation step for obtaining a direction corresponding to the direction of the light beam in the three-dimensional virtual space; and 2) a predetermined convergence calculation process based on the direction obtained in the direction calculation step and the depth value information. A dot determination step of determining, from the reference image, a dot representing color information of an object closest to the viewpoint in the direction in the original virtual space, and the dot of the reference image determined by the dot determination step An image generation apparatus that generates a stereoscopic image by using color information as color information of the pixel. A stereoscopic image in a three-dimensional virtual space to be displayed on a stereoscopic video display device comprising a pixel panel in which pixels are arranged and an optical element group that directs the emitted light of each pixel of the pixel panel. A program for causing a computer to generate a reference image, which is a planar image when a virtual space is viewed from a given viewpoint, and depth value information of each dot of the reference image with respect to the viewpoint position. ,
Regarding the color information of each pixel of the pixel panel, 1) based on the direction of the light beam passing through the optical element that directs the pixel and the emitted light beam of the pixel, and the viewpoint position in the three-dimensional virtual space, A direction calculation step for obtaining a direction corresponding to the direction of the light beam in the three-dimensional virtual space; and 2) a predetermined convergence calculation process based on the direction obtained in the direction calculation step and the depth value information. The reference image determined by the dot determination step is executed by causing the computer to execute a dot determination step of determining, from the reference image, a dot representing color information of an object closest to the viewpoint in the direction in the original virtual space. the color information of the dots by the color information of the pixel, causes the computer to perform to produce a stereoscopic image
The dot determination step includes a depth differential information calculation step of calculating depth differential information that is a differential value of a depth value of dots constituting the reference image, and the direction obtained in the direction calculation step and the depth value information And a dot representing the color information of the object closest to the viewpoint in the direction in the three-dimensional virtual space by a predetermined convergence calculation process based on the Newton method based on the depth derivative information calculated in the depth derivative information calculating step. A program characterized in that it is a step of judging from the reference image . A stereoscopic image in a three-dimensional virtual space to be displayed on a stereoscopic video display device comprising a pixel panel in which pixels are arranged and an optical element group that directs the emitted light of each pixel of the pixel panel. A program for causing a computer to generate a reference image, which is a planar image when a virtual space is viewed from a given viewpoint, and depth value information of each dot of the reference image with respect to the viewpoint position. ,
Regarding the color information of each pixel of the pixel panel, 1) based on the direction of the light beam passing through the optical element that directs the pixel and the emitted light beam of the pixel, and the viewpoint position in the three-dimensional virtual space, A direction calculation step for obtaining a direction corresponding to the direction of the light beam in the three-dimensional virtual space; and 2) a predetermined convergence calculation process based on the direction obtained in the direction calculation step and the depth value information. The reference image determined by the dot determination step is executed by causing the computer to execute a dot determination step of determining, from the reference image, a dot representing color information of an object closest to the viewpoint in the direction in the original virtual space. the color information of the dots by the color information of the pixel, causes the computer to perform to produce a stereoscopic image
Before the determination process by the dot determination step, the computer executes a depth differential information calculation step for calculating depth differential information that is a differential value of the depth value of each dot of the reference image,
In the dot determination step, a predetermined convergence calculation by the Newton method is performed using the direction obtained in the direction calculation step and the depth derivative information calculated in the depth derivative information calculation step in addition to the depth value information. A dot representing color information of an object closest to the viewpoint in the direction in the three-dimensional virtual space is determined from the reference image by processing.
Program for causing the computer to execute . A computer-readable information storage medium storing the program according to claim 9 or 10 .