JP5993261B2 - Depth range calculation device and program thereof - Google Patents

Description

  The present invention relates to a depth range calculation apparatus that calculates a parallax range and a depth range of a stereoscopic image of a subject from a group of element images obtained by photographing the subject with a stereoscopic image capturing device using an integral photography system, and a program thereof.

  Conventionally, as one of the stereoscopic image display methods capable of freely viewing stereoscopic images from an arbitrary viewpoint, integral photography (Integral Photography) using a group of convex lenses or pinholes arranged in a planar shape is possible. Hereinafter, the IP) method is known.

  Hereinafter, normal stereoscopic image capturing and stereoscopic image display based on the IP method will be described with reference to FIGS. 11 and 12. As shown in FIG. 11, the stereoscopic image capturing apparatus 910 includes a lens group 912 in which convex lenses are arranged on the same plane, and an imaging plate 913. FIG. 11 illustrates a subject 911, a photographing direction 914 of the stereoscopic image photographing device 910, and an element image 915 of the subject 911 formed by the lens group 912. When the subject 911 is viewed from the shooting direction 914, the cylinder is in front of the prism.

  The stereoscopic image photographing device 910 photographs a subject 911 through a lens group 912. Then, the same number of element images 915 of the subject 911 as the convex lenses constituting the lens group 912 are photographed on the imaging plate 913.

  As shown in FIG. 12, the stereoscopic image display device 920 includes a lens group 922 in which convex lenses are arranged on the same plane, and a display element 923. FIG. 12 illustrates a stereoscopic image 921, an observation direction 924 of the observer 926, an element image 925, and an observer 926. The display element 923 displays an element image 925 corresponding to the element image 915 photographed by the imaging plate 913 of the stereoscopic image photographing device 910.

  As a result, as shown in FIG. 12, the stereoscopic image 921 is generated such that the distance from the display element 923 is equal to the distance between the subject 911 and the imaging plate 913 in FIG. At this time, in the stereoscopic image 921 corresponding to the subject 911, when viewed from the observation direction 924, the prism is in front of the cylinder. That is, in normal stereoscopic image shooting based on the IP method, a reverse-view image (stereoscopic image 921) with a depth inverted as compared to the subject 911 in FIG. 11 is generated.

11 and 12, it has been described that the optical element array is the lens group 912 in which minute convex lenses are arranged. However, the optical element array may be an aperture array (spatial filter) in which minute pinholes are arranged. Good.
Further, although the display element 923 has been described as displaying the element image 925 photographed by the photographing plate 113, an image generated by a computer (not shown) may be displayed.

  In view of this, an invention for solving the problem of the reverse viewing image has been proposed (for example, Patent Document 1). The invention described in Patent Document 1 performs arithmetic processing on the information acquired by the stereoscopic image capturing device 910 in FIG. 11 and inputs the information after the arithmetic processing to the stereoscopic image display device 920 in FIG. A stereoscopic image 921 having a correct depth is generated.

  Hereinafter, with reference to FIGS. 13 and 14, an image depth conversion device 930 described in Patent Document 1 will be described. As shown in FIG. 13, the image depth conversion device 930 receives the element image 931 captured by the stereoscopic image capturing device 910 of FIG. 11, and this element image 931 is virtually formed through the first virtual lens array 932. The obtained stereoscopic image 933 is obtained by arithmetic processing. Then, the image depth conversion device 930 obtains an element image 935 in which the stereoscopic image 933 is virtually formed through the second virtual lens array 934 by arithmetic processing.

  As shown in FIG. 14, the stereoscopic image display device 940 has the same configuration as that of the stereoscopic image display device 920 in FIG. 12, and an element image 941 generated by the image depth conversion device 930 in FIG. That is, the element image 935) of FIG. 13 is displayed. As a result, when the stereoscopic image 943 corresponding to the element image 941 is viewed from the observation direction 944, the cylinder is in front of the prism, and the depth is equivalent to the subject 911 in FIG.

JP 2007-114483 A

  However, in the related art, there is a problem that it is impossible to grasp to what extent the stereoscopic image has parallax and depth before the stereoscopic image is actually displayed on the stereoscopic display device. That is, the depth range and the parallax range when the observer views the stereoscopic image cannot be grasped until the stereoscopic image is actually displayed. If the depth range and the parallax range can be grasped in advance, it is possible to effectively produce and edit stereoscopic images.

  Accordingly, an object of the present invention is to solve the above-described problems and to provide a depth range calculation device and a program thereof that can calculate a depth range and a parallax range of a stereoscopic image of a subject by the IP method.

  In view of the above-described problems, the depth range calculation device according to the first invention of the present application is based on a parallax range and depth of a stereoscopic image of a subject from an elemental image group in which the stereoscopic imaging device using the integral photography method captures the subject. A depth range calculating apparatus for calculating a range, comprising: a planar light intensity distribution calculating means; a volume light intensity distribution storing means; a parallax range calculating means; and a depth range calculating means.

  According to such a configuration, the depth range calculation device receives the elemental image group captured by the stereoscopic image capturing device by the planar light intensity distribution calculating unit, and displays the elemental image group for the elemental image group. By performing wave optical computation via a virtual element optical system group in which virtual element optical systems having the same pitch and focal length of the element optical system of the apparatus are arranged two-dimensionally, the distance from the virtual element optical system group differs in advance. A plane light intensity distribution that is a light intensity distribution for each set distance plane is calculated.

  In addition, the depth range calculation device uses the volume light intensity distribution storage means to stack the volume light in the three-dimensional space so that the plane light intensity distribution for each distance plane calculated by the plane light intensity distribution calculation means is stacked in the depth direction. Store as intensity distribution. Here, the volume light intensity distribution corresponds to a three-dimensional shape model of a three-dimensional image of a subject in a three-dimensional space (three-dimensional space) composed of a plurality of distance planes at an arbitrary distance from the virtual element optical system group.

  In addition, the depth range calculation device uses a parallax range calculation unit to set a position at which the volume light intensity distribution is zero in the depth direction at a viewer position preset with reference to the virtual element optical system group in the vicinity of the stereoscopic image of the subject. It calculates as a point and a far point, respectively, and calculates the difference in the amount of angular parallax between the near point and the far point as a parallax range. Here, when there are a plurality of positions where the volumetric light intensity distribution is zero in a preset range in the depth direction, the farthest position is taken as the far point, and the nearest position is taken as the near point.

  Thus, since both the volume light intensity distribution and the observer position are based on the virtual element optical system group, the position of the stereoscopic image of the subject and the observer position can be described in the same coordinate system. For this reason, the parallax range calculation means can calculate the parallax range by describing the positional relationship between the observer and the stereoscopic image of the subject in the same coordinate system.

  In addition, the depth range calculation device calculates the distance between the observer position and the near point and the distance between the observer position and the far point by the depth range calculation unit, and calculates the difference between the two distances as the depth range. To do. That is, the depth range calculation unit can calculate the depth range by describing the positional relationship between the observer and the stereoscopic image of the subject in the same coordinate system, as in the parallax range calculation unit.

  In the depth range calculation apparatus according to the second invention of the present application, the plane light intensity distribution calculation means distributes the element image group captured by the stereoscopic image capturing apparatus for each element image based on the pitch of the virtual element optical system. First light wave calculation for calculating the light wave incident on the virtual element optical system by propagating the light wave of the element image by the focal length of the virtual element optical system for each element image distributed by the distribution means and the distribution means A phase shift means for shifting the light wave for each element image incident on the virtual element optical system calculated by the first light wave calculation means by the phase of the virtual element optical system, and the phase shift means shifts the phase. The second light wave calculating means for calculating the light wave of the element image for each distance plane by propagating the obtained light wave to each distance plane, and the element calculated by the second light wave calculating means for each distance plane. The light waves of the image, characterized in that it comprises a coupling means for a plane light intensity distribution bound only element image group content, the.

  According to such a configuration, the depth range calculation apparatus propagates the light wave of the element image from the virtual display element, which is a virtual display element included in the stereoscopic image display apparatus, to each distance plane via the virtual element optical system, The plane light intensity distribution can be accurately obtained.

  The depth range calculation apparatus according to the third aspect of the present invention further includes delay means for delaying the element image group captured by the stereoscopic image capturing apparatus by a preset delay time and outputting the delay element image group, The light intensity distribution calculating means calculates the planar light intensity distribution from the element image group and the delayed element image group delayed by the delay means, and the volume light intensity distribution storage means is calculated from the element image group and the delayed element image group. The plane light intensity distribution is stored as a volume light intensity distribution, and the parallax range calculation means calculates the parallax range from the volume light intensity distributions of the element image group and the delay element image group, and the depth range calculation means Depth ranges are calculated from the volumetric light intensity distributions of the element image group and the delay element image group, respectively, and the amount of change in the parallax range between the element image group and the delay element image group calculated by the parallax range calculation unit is viewed. A parallax change amount calculating unit that calculates the amount of change, and a depth change amount calculating unit that calculates the amount of change in the depth range between the element image group and the delay element image group calculated by the depth range calculating unit as the depth change amount. It is characterized by that.

  According to such a configuration, the depth range calculation apparatus can obtain a parallax change amount that indicates the change amount of the parallax range within the delay time and a depth change amount that indicates the change amount of the depth range within the delay time. .

Further, the depth range calculation apparatus according to the fourth invention of the present application determines whether or not the parallax range calculated by the parallax range calculation means exceeds a preset parallax range threshold, and when the parallax range exceeds the parallax range threshold A parallax range warning unit that warns the depth range, and a depth range that determines whether the depth range calculated by the depth range calculation unit exceeds a preset depth range threshold and warns when the depth range exceeds the depth range threshold And a warning means.
According to such a configuration, the depth range calculation apparatus can warn when the parallax range or the depth range exceeds the allowable limit of the observer.

Further, the depth range calculation apparatus according to the fifth invention of the present application determines whether or not the parallax change amount calculated by the parallax change amount calculating means exceeds a preset parallax change amount threshold, and the parallax change amount is the parallax change amount. A parallax change amount warning unit that warns when the amount threshold value is exceeded, and whether or not the depth change amount calculated by the depth change amount calculation unit exceeds a preset depth change amount threshold value. It further comprises depth change amount warning means for warning when the change amount threshold is exceeded.
According to such a configuration, the depth range calculation device can warn when the parallax range or the depth range changes beyond the allowable limit of the observer.

  The depth range calculation apparatus according to the first aspect of the present invention causes hardware resources such as a CPU, a memory, and a hard disk included in a computer to function as a planar light intensity distribution calculation unit, a parallax range calculation unit, and a depth range calculation unit. It can also be realized by a depth range calculation program. This program may be distributed through a communication line, or may be distributed by writing in a recording medium such as a CD-ROM or a flash memory.

According to the present invention, the following excellent effects can be obtained.
According to the first invention of the present application, the range of the volume light intensity distribution represents a three-dimensional shape model of the stereoscopic image of the subject, and both the volume light intensity distribution and the observer position are based on the virtual element optical system group. Thus, according to the first invention of the present application, since the positional relationship between the observer and the stereoscopic image of the subject can be described in the same coordinate system, the volume light intensity distribution stored in the volume light intensity distribution storage means can be used according to the IP method. The depth range and the parallax range of the stereoscopic image of the subject can be calculated.

According to the second invention of the present application, since the planar light intensity distribution in each distance plane can be accurately obtained, the accuracy of the parallax range and the depth range can be improved.
According to the third aspect of the present invention, since the parallax change amount and the depth change amount within the delay time can be obtained, it is possible to more effectively produce and edit a stereoscopic video.

According to the fourth invention of the present application, since a warning can be given when the parallax range or depth range exceeds the allowable limit of the observer, the situation in which a stereoscopic video that is difficult for the observer to display is reduced, and the quality of the stereoscopic video is improved. Can be improved.
According to the fifth invention of the present application, since a warning can be given when the parallax range or the depth range changes beyond the allowable limit of the observer, the situation in which a stereoscopic image that is difficult to be viewed by the observer is reduced can be reduced. Can improve the quality.

It is a block diagram which shows the structure of the depth information generation apparatus which concerns on 1st Embodiment of this invention. It is a block diagram which shows the structure of the planar light intensity distribution calculation means of FIG. FIG. 3 is a diagram illustrating light propagation of an element image and a range in which the element image expands in the planar light intensity distribution calculating unit of FIG. It is a figure explaining the planar light intensity distribution which the planar light intensity distribution calculation means of FIG. 2 computed. It is a figure explaining the volume light intensity distribution which the volume light intensity distribution memory | storage means of FIG. 1 memorize | stores. It is a figure explaining calculation of the parallax range and depth range in the depth information generation apparatus of FIG. It is a flowchart which shows operation | movement of the depth information generation apparatus of FIG. It is a block diagram which shows the structure of the depth information generation apparatus which concerns on 2nd Embodiment of this invention. It is a block diagram which shows the structure of the depth information generation apparatus which concerns on 3rd Embodiment of this invention. It is a block diagram which shows the structure of the depth information generation apparatus which concerns on 4th Embodiment of this invention. It is a figure explaining imaging | photography of a to-be-photographed object in the conventional IP system. It is a figure explaining the display of a three-dimensional image in the conventional IP system. It is a figure explaining cancellation of depth inversion in the conventional IP system. It is a figure explaining the display of the element image group which the depth inversion canceled in the conventional IP system.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate. In each embodiment, means having the same function are denoted by the same reference numerals and description thereof is omitted.

(First embodiment)
[Configuration of depth information generation apparatus]
With reference to FIG. 1, the structure of the depth information generation apparatus (depth range calculation apparatus) 1 which concerns on 1st Embodiment of this invention is demonstrated.
The depth information generation device 1 calculates a parallax range and a depth range of a stereoscopic image of a subject from a group of element images obtained by photographing a subject with the stereoscopic image photographing device 910 using the integral photography method, and the parallax range and the depth range. Is output as depth information. For this reason, as shown in FIG. 1, the depth information generation apparatus 1 includes a planar light intensity distribution calculation unit 10, a volume light intensity distribution storage unit 20, a parallax range calculation unit 30, and a depth range calculation unit 40. .

Further, the depth information generation apparatus 1 receives optical element information and an element image group captured by the stereoscopic image capturing apparatus 910. At this time, the depth information generating apparatus 1 generates a video signal equivalent to the element image group by a computer (not shown) instead of the element image group actually captured by the stereoscopic image capturing apparatus 910, and the image signal is It may be entered.
When this element image group is displayed on a stereoscopic image display device (not shown), a stereoscopic image having a correct depth is generated.

The above-described stereoscopic image display device displays a stereoscopic image, and includes an element optical system group in which element optical systems are arranged on a two-dimensional plane.
The element optical system is, for example, an element optical element made of an element lens such as a convex lens, or an element pinhole (spatial filter) made of a minute aperture. In this case, the element optical system group is an element optical element group in which element optical elements are arranged on a two-dimensional plane, or an element pinhole group in which element pinholes are arranged on a two-dimensional plane. The element optical system group is arranged at a position separated from the element image group by the focal length.

The optical element information described above is information indicating the characteristics and arrangement of the element optical system (element lens or element pinhole) included in the stereoscopic image display device.
For example, when an element lens is assumed as the virtual element optical system, the optical element information is preset with the pitch of the element lens (reference numeral p _{1 in} FIG. 3), the focal length, and the aperture width.
For example, when an element pinhole is assumed as a virtual element optical system, the pitch of the element pinhole and the opening width are preset in the optical element information.

  Note that the depth information generation apparatus 1 does not actually include an element optical system, but an element optical system (element lens or element pinhole) or element optical system group (element lens group or element) included in the stereoscopic image display apparatus. Virtually calculate the pinhole group). For this reason, they are called a virtual element optical system (virtual element lens or virtual element pinhole) and a virtual element optical system group (virtual element lens group or virtual element pinhole group).

The planar light intensity distribution calculating means 10 performs a wave optical calculation via the virtual element optical system group on the input element image group, so that the distance from the virtual element optical system group is set in advance. A plane light intensity distribution, which is a light intensity distribution for each plane, is calculated. Therefore, the planar light intensity distribution calculating means 10 includes a distributing means 11, an element image converting means 13 for element images, and a combining means 15 as shown in FIG.
In the following description, the element image group is composed of element images from −Mth to Mth, and the element image conversion means 13 (13 _−M ,..., 13 _{0 ,.} .. 13 _M ) is provided.

The distribution unit 11 distributes the element image group input from the stereoscopic image capturing device 910 for each element image based on the input optical element information. Here, the distribution unit 11 distributes (divides) the input element image group with the position information (pitch of the virtual element optical system) set in the input optical element information. The distribution unit 11 associates the light wave (gs _{, m} ( _{xs, m} , ys _{, m} )) of the mth element image of the distributed element image group with the mth element image in advance. Is output to the element image conversion means 13 (-M ≦ m ≦ M).
Here, the light wave is a complex number representing the amplitude and phase when the light of the element image group is treated as a wave.

  The element image conversion means 13 converts the light wave of the element image input from the distribution means 11 into a light wave when passing through the virtual element optical system set in the input optical element information, and displays the image of the virtual element optical system. The light wave corresponding to the corner is added. Therefore, the element image conversion means 13 includes a light wave calculation means (first light wave calculation means) 13a, a phase shift means 13b, and a light wave calculation means (second light wave calculation means) 13c.

The light wave calculation means 13a calculates the light wave incident on the virtual element optical system by propagating the light wave of the element image for the element image distributed by the distribution means 11 by the focal length of the virtual element optical system. is there. That is, the light wave calculation means 13a calculates the light wave incident on the virtual element optical system by approximating the light wave of the element image with Fresnel approximation. More specifically, the light wave calculation means 13a uses a general Fresnel approximation as a signal corresponding to a light wave reaching the m-th virtual element optical system (aperture) of the virtual element optical system group, and The light wave (R _{i, m} (x _{o, m} , y _{o, m} )) for each element image is calculated according to equation (1). Thereafter, the light wave calculating means 13a outputs the light wave (R _{i, m} (x _{o, m} , _{yo, m} )) for each element image to the phase shift means 13b.

Here, x _{s, m} , x _{o, m} are the x coordinate when the center of the m-th element image in the entire input image (element image group) is the origin, and the m-th virtual of the virtual element optical system group. This is the x coordinate when the optical axis center of the element optical system is the origin. Y _{s, m} , _{yo, m} are the y coordinate when the center of the m-th element image in the entire input image (element image group) is the origin, and the m-th virtual element of the virtual element optical system group. This is the y coordinate when the optical axis center of the optical system is the origin. F ₁ is the focal length of the virtual element optical system (element lens), and k is the wave number 2π / λ. Also, λ is a wavelength and is set in advance as a parameter. The element image (m) that is the integration range of the expression (1) is an area of the mth element image, and is represented by the pitch of the virtual element optical system.

The phase shift means 13b shifts the light wave for each element image input from the light wave calculation means 13a by the phase of the virtual element optical system. In other words, the phase shift means 13b converts the light wave (R _{i, m} ) for each element image incident on the virtual element optical system by a phase corresponding to the virtual element optical system as shown in the following equation (2). By shifting (x _{o, m} , y _{o, m} )), a signal (R _{o, m} (x _{o, m} , y _{o, m} )) corresponding to the light wave emitted from the virtual element optical system is calculated. . Thereafter, the phase shift means 13b outputs this light wave (R _{o, m} (x _{o, m} , y _{o, m} )) to the light wave calculation means 13c.

The light wave calculating unit 13c calculates the light wave of the element image for each distance plane by propagating the light wave input from the phase shift unit 13b to each distance plane. Here, the light wave calculation means 13c is preset with a plurality of distance planes at arbitrary distances so that the distance from the virtual element optical system is different. The light wave calculating means 13c sets the light wave (R _{o, m} (x _{o, m} , y _{o, m} )) emitted from the virtual element optical system to an arbitrary distance from the virtual element optical system by approximating it with Fresnel. The light intensity distribution in the obtained distance plane is calculated. More specifically, the light wave calculation unit 13c uses a general Fresnel approximation as a signal corresponding to the light wave that reaches each distance plane, and uses the following equation (3) to calculate the light wave (R _d ) for each element image. _{, M} (x _{d, m} , y _{d, m} )). Thereafter, the light wave calculation means 13 c outputs this light wave (R _{d, m} (x _{d, m} , y _{d, m} )) to the coupling means 15.

Here, _{xd, m} is an x coordinate on the distance plane when the optical axis center of the m-th virtual element optical system (aperture) of the virtual element optical system group is the origin. Y _{d, m} is the y coordinate on the distance plane when the optical axis center of the m-th virtual element optical system of the virtual element optical system group is the origin. L is the distance from the virtual element optical system group to the distance plane. Further, the integration range of Expression (3) is equivalent to the range w in which the element image expands.

With reference to FIG. 3, the propagation of the light wave of the element image and the range in which the element image expands will be described (see FIG. 2 as appropriate).
First, the light wave calculation unit 13 a calculates a light wave from the element image m (virtual display element 220 of the stereoscopic image display device) to the incident surface of the virtual element optical system 210. Next, the phase shift unit 13 b calculates a light wave from the incident surface to the exit surface of the virtual element optical system 210, that is, a light wave propagating through the virtual element optical system 210. Then, the light wave calculation unit 13 c calculates a light wave from the emission surface of the virtual element optical system 210 to the distance plane t.

In this case, the width of the area of the m-th element image in the element image group (that is, the pitch p ₁ of the virtual element optical system 210) is k _1, and the distance from the element image group to the virtual element optical system group 200 (that is, Assuming that the focal length of the virtual element optical system 210 is d ₁ , the range w in which the m-th element image expands in the element image group is expressed by the following equation (4).

  In FIG. 3, only one distance plane t is shown for simplicity of explanation, but actually, a plurality of distance planes t are set at different distances L. At this time, the number of distance planes t is larger, and the shorter the distance between them, the more preferable the light wave of the element image incident on each distance plane t is calculated more accurately.

Returning to FIG. 2, the description of the configuration of the planar light intensity distribution calculating means 10 will be continued.
The combining unit 15 combines the light wave (R _{d, m} (x _{d, m} , y _{d, m} )) of the element image input from the light wave calculating unit 13c for each distance plane by the amount corresponding to the element image group. The light intensity distribution is used. In other words, the coupling means 15 reaches the distance plane set at an arbitrary distance from the virtual element optical system and is emitted from the virtual element optical system (R _{d, m} (x _{d, m} , y _{d, m} )). Are added within the area of this distance plane. More specifically, the coupling means 15 uses the following equation (5) as a signal corresponding to the light wave that reaches each distance plane, and the light wave (R _p (x _p , y _p )) at each distance plane. calculate.

Here, x _{p, m} is the x coordinate when the origin is the center of the distance plane set at an arbitrary distance from the virtual element optical system group. Moreover, yp _{, m} is the y coordinate when the origin is the center of the distance plane set at an arbitrary distance from the virtual element optical system group.

  And the coupling means 15 calculates the light intensity distribution in each distance plane by calculating the square of the light wave reaching the distance plane set at an arbitrary distance from the virtual element optical system group by the following equation (6). (Planar light intensity distribution) is obtained.

  As described above, the planar light intensity distribution calculating unit 10 calculates the planar light intensity distribution for each distance plane set at an arbitrary distance from the virtual element optical system group, and the planar light intensity distribution for each distance plane is calculated as the volume light intensity. Write to the distribution storage means 20. Then, the planar light intensity distribution calculating unit 10 outputs a depth information generation command (not shown) to the parallax range calculating unit 30 and the depth range calculating unit 40 after the writing of the planar light intensity distribution is completed for all the distance planes. .

Referring to FIG. 4, will be described plane light intensity distribution t _2.
In FIG. 4, the x-axis is the horizontal direction, the y-axis is the vertical direction, and the z-axis is the depth direction. Consider a case where two objects, a cylindrical body and a triangular pyramid, are taken as subjects, and the bottom surfaces of these cylindrical bodies and the triangular pyramid are photographed toward the z axis. In this case, the plane light intensity distribution t _2, the light intensity distribution range in which the light intensity is greater than zero would represent a three-dimensional image β stereoscopic image α and triangular pyramid of the cylindrical body as viewed from the z-axis.
In FIG. 4, in the planar light intensity distribution t ₂ , the light intensity distribution range in which the light intensity exceeds zero is illustrated by dots, the stereoscopic image α is illustrated by fine dots, and the stereoscopic image β is illustrated by coarse dots.

Returning to FIG. 1, the description of the configuration of the depth information generation device 1 will be continued.
The volume light intensity distribution storage unit 20 is a storage device such as a memory or a hard disk that stores the plane light intensity distribution for each distance plane written by the plane light intensity distribution calculation unit 10 as a volume light intensity distribution in a three-dimensional space. .

Referring to FIG. 5, a description will be given volume light intensity distribution V _2.
This volume light intensity distribution V ₂ is obtained by stacking planar light intensity distributions t ₂₁ , t ₂₂ ,..., T _2n in the depth direction (z axis) as shown in FIG. Number of distance planes). Therefore, the volume light intensity distribution V _2, in the three-dimensional space composed of a plurality of distances plane at an arbitrary distance from the virtual element optical group, stereoscopic images alpha, corresponding to the three-dimensional shape model of the beta.

Returning to FIG. 1, the description of the configuration of the depth information generation device 1 will be continued.
The parallax range calculation means 30 calculates the position where the volume light intensity distribution becomes zero in the depth direction as the near point and the far point of the stereoscopic image of the subject at the observer position set in advance with reference to the virtual element optical system group. The difference in the amount of angular parallax between the near point and the far point is calculated as the parallax range.
The depth range calculation means 40 calculates the distance between the observer position and the near point, and the distance between the observer position and the far point, and calculates the difference between these distances as the depth range.
The parallax range calculation unit 30 and the depth range calculation unit 40, when there are a plurality of positions where the volume light intensity distribution is zero in a preset range in the depth direction, the farthest position among them is the far point. And the nearest position is calculated as a near point.

<Calculation of parallax range and depth range>
The parallax range by the parallax range calculation unit 30 and the calculation of the depth range by the depth range calculation unit 40 will be described with reference to FIG.
In FIG. 6, it is assumed that the observer position a is set in advance to the positions of the right eye and left eye of the observer with respect to the virtual element optical system group.
Further, the boundary between the stereoscopic image of the subject and the stereoscopic space, that is, the boundary position where the volumetric light intensity becomes zero in the z-axis direction is the near point n of the stereoscopic image and the far point f of the stereoscopic image, respectively. Specifically, the near point n of the stereoscopic image is a boundary position on the near side (side closer to the observer position a) among the boundary positions where the volume light intensity becomes zero. On the other hand, the far point f of the stereoscopic image is a boundary position on the back side (the side far from the observer position a) among the boundary positions where the volume light intensity becomes zero. When there are a plurality of positions where the volumetric light intensity distribution is zero in a preset range in the depth direction, the farthest side is the far point and the foremost side is the near point.

Here, since both the volume light intensity distribution V ₂ and the observer position a to the reference virtual element optical group, stereoscopic images alpha, the position and the observer position a beta, it can be described in the same coordinate system. For this reason, for example, a distance D _d between the image point d of the stereoscopic image β and the observer position a is obtained. This distance _Dd is the depth of the image point d. The parallax image point d is a distance D _d, and a preset distance between the left and right eyes can be determined by the principle of triangulation. Accordingly, as with the image point d, the parallax range and the depth range at the near point n and the far point f of the stereoscopic image can be calculated by obtaining the distance D _n and the distance D _f between the near point n and the far point f of the stereoscopic image. Can do.

First, the parallax range calculation unit 30 calculates the near point n of the stereoscopic image and the far point f of the stereoscopic image based on the boundary between the stereoscopic image of the subject and the stereoscopic space. In addition, when the depth information generation command is input from the planar light intensity distribution calculating unit 10, the parallax range calculating unit 30 reads the volume light intensity distribution V ₂ stored in the volume light intensity distribution storage unit 20. Also, the parallax range calculation unit 30 calculates the distance D _n between the near point n of the observer position a and the stereoscopic image, and calculates the distance D _f between the far point f of the observer position a and stereoscopic image. Then, the parallax range calculation unit 30 calculates the angular parallax amount θ _n at the near point n of the stereoscopic image and the angular parallax amount θ _f at the far point f of the stereoscopic image. Further, the parallax range calculation unit 30 calculates a difference (θ _n −θ _f ) between these angular parallax amounts θ _n and θ _f as a parallax range, and outputs the parallax range to the depth range calculation unit 40.

Next, the depth range calculation means 40 calculates the near point n of the stereoscopic image and the far point f of the stereoscopic image based on the boundary between the stereoscopic image of the subject and the stereoscopic space. Further, when a depth information generation command is input from the planar light intensity distribution calculating unit 10, the depth range calculating unit 40 reads the volume light intensity distribution V ₂ stored in the volume light intensity distribution storage unit 20. Further, the depth range calculation unit 40 calculates the distance D _n between the near point n of the observer position a and the stereoscopic image, and calculates the distance D _f between the far point f of the observer position a and stereoscopic image. Then, the depth range calculation means 40 calculates a difference (D _f −D _n ) between the distances D _f and D _n as a depth range. Thereafter, the depth range calculation unit 40 outputs the calculated depth range and the parallax range input from the parallax range calculation unit 30 as depth information.
The observer position can be set at an arbitrary position regardless of the inside or outside of the volume light intensity distribution (three-dimensional space).

[Operation of depth information generator]
With reference to FIG. 7, the operation of the depth information generating apparatus 1 in FIG. 1 will be described (see FIG. 1 as appropriate).
The depth information generation device 1 performs a wave optical calculation via the virtual element optical system group on the input element image group by the plane light intensity distribution calculation unit 10, thereby obtaining a light intensity distribution for each distance plane. A certain planar light intensity distribution is calculated (step S1).

The depth information generation apparatus 1 writes the planar light intensity distribution for each distance plane in the volume light intensity distribution storage unit 20 as the volume light intensity distribution in the three-dimensional space by the plane light intensity distribution calculation unit 10 (step S2).
The depth information generating device 1 calculates the difference in the amount of angular parallax between the near point and the far point of the stereoscopic image of the subject as the parallax range at the observer position by the parallax range calculating unit 30 (step S3).
The depth information generation device 1 calculates the distance between the observer position and the near point of the stereoscopic image of the subject and the distance between the observer position and the far point of the stereoscopic image of the subject by the depth range calculation unit 40, respectively. The difference between these distances is calculated as a depth range (step S4).

  As described above, the depth information generation device 1 according to the first embodiment of the present invention accurately obtains the planar light intensity distribution in each distance plane by wave optical calculation, and stacks the planar light intensity distribution in the depth direction. The volume light intensity distribution is stored. Then, the depth information generation device 1 shows the parallax range and the depth range when the observer views the stereoscopic image of the subject from the stored volumetric light intensity distribution before actually displaying the stereoscopic image on the stereoscopic display device. Can be calculated with high accuracy.

  In addition, although 1st Embodiment demonstrated the example using Fresnel approximation like Formula (1) and Formula (3), this invention is not limited to this. For example, the depth information generation apparatus 1 may calculate the light wave using integration based on the Huygens-Fresnel principle or integration based on the Francofa approximation.

(Second Embodiment)
With reference to FIG. 8, a difference from the first embodiment will be described regarding the configuration of the depth information generating apparatus 1B according to the second embodiment of the present invention.
This depth information generation device 1B is different from the first embodiment in that a warning is issued when the calculated parallax range and depth range exceed predetermined thresholds.

  For this reason, as shown in FIG. 8, the depth information generating apparatus 1B includes a planar light intensity distribution calculating unit 10, a volume light intensity distribution storing unit 20, a parallax range calculating unit 30, a depth range calculating unit 40, and a parallax. A range warning unit 50 and a depth range warning unit 60 are provided.

The planar light intensity distribution calculating means 10 and the volume light intensity distribution storing means 20 are the same as the means in FIG.
1 except that the parallax range calculation unit 30 outputs the parallax range to the parallax range warning unit 50 and the depth range calculation unit 40 outputs the depth range to the depth range warning unit 60. Is omitted.

The parallax range warning unit 50 determines whether or not the parallax range input from the parallax range calculation unit 30 exceeds the parallax range threshold, and warns when the parallax range exceeds the parallax range threshold. Here, the parallax range warning unit 50 may add a warning message of the parallax range to the video signal without any particular limitation on the warning method. Further, the parallax range warning unit 50 may transmit this warning message to a preset mail address or may sound a predetermined warning sound.
The parallax range threshold is set in advance as an arbitrary value, for example, indicating that the parallax range has exceeded the allowable limit of the observer.

The depth range warning unit 60 determines whether or not the depth range input from the depth range calculation unit 40 exceeds the depth range threshold, and warns when the depth range exceeds the depth range threshold. Here, the parallax range warning unit 50 gives a warning by the same warning method as the parallax range warning unit 50.
The depth range threshold is set in advance as an arbitrary value, for example, indicating that the depth range has exceeded the allowable limit of the observer.

  As described above, the depth information generating apparatus 1B according to the second embodiment of the present invention warns when the parallax range or the depth range exceeds the allowable limit of the observer in addition to the same effects as the first embodiment. Therefore, it is possible to improve the quality of the stereoscopic video by reducing the situation in which the stereoscopic video that is difficult for the observer to view is displayed.

(Third embodiment)
With reference to FIG. 9, the difference from the first embodiment will be described regarding the configuration of the depth information generating apparatus 1C according to the third embodiment of the present invention.
The depth information generation device 1C delays the input element image group by a delay time, and a parallax change amount indicating a change amount of the parallax range within the delay time and a depth range change amount within the delay time. The point which calculates | requires the depth variation | change_quantity which shows is different from 1st Embodiment.

For this reason, as shown in FIG. 9, the depth information generating apparatus 1C includes planar light intensity distribution calculating means 10 ₁ , 10 ₂ , volume light intensity distribution storing means 20 ₁ , 20 ₂ , and parallax range calculating means 30 ₁ , comprises a 30 _2, the depth range calculating means ₄₀ 1, 40 _2, the delay means 70, the depth information storing means ₈₀ 1, 80 _2, the parallax variation calculating means 90, and a depth variation amount calculating means 100.

In the following description, the time at which the element image group from the stereoscopic imaging apparatus 910 is input as T _1, the time after the delay processing in the delay means 70 to be described later and T _2. Accordingly, the delay time (T ₂ −T ₁ ) and the element image group delayed by the delay time (T ₂ −T ₁ ) is referred to as a delay element image group.

The plane light intensity distribution calculating means 10 _1, volume light intensity distribution storage unit 20 _1, the parallax range calculation unit 30 ₁ and the depth range calculation unit 40 _1, except that handle element images at time T _1, the means of FIG. 1 The description is omitted because it is the same as.
The planar light intensity distribution calculating means 10 _2, volume light intensity distribution storage unit 20 _2, the disparity range calculation unit 30 ₂ and the depth range calculation unit 40 _2, except that handle delay element images at time _{T 2,} in FIG. 1 Since it is the same as each means, description is abbreviate | omitted.

Delay means 70, the stereoscopic image photographing apparatus 910 delay the input element image group from (T 2 _{-T 1)} by delaying, and outputs the delay element images the plane light intensity distribution calculating means 10 ₂ is there. The delay means 70 is composed of, for example, a frame memory (not shown) that stores element image groups and a control means (not shown) that controls the frame memory.
This delay time (T ₂ −T ₁ ) is set in advance as an arbitrary value, for example, indicating a minute time.

Depth information storing section 80 ₁ is a storage device such as a memory and a hard disk for storing the depth information of the time T ₁ which is input from the depth range calculating means 40 ₁ (disparity range and depth range).
Depth information storing section ₈₀₂ is a storage device such as a memory and a hard disk for storing the depth information of the time T _2, which is input from the depth range calculating means 40 ₂ (disparity range and depth range).

Parallax change amount calculation unit 90 reads the parallax range theta ₁ at time T _1, which is stored in the depth information storing means 80 _1, and a parallax range theta ₂ of time T _2, which is stored in the depth information storing means 80 ₂ The change amount of the parallax ranges θ ₁ and θ ₂ is calculated as the parallax change amount θ _dif . That is, the parallax change amount calculation unit 90 divides the difference (θ ₂ −θ ₁ ) between the parallax ranges θ ₁ and θ ₂ by the delay time (T ₂ −T ₁ ) as shown in the following equation (7). Thus, the parallax change amount θ _dif is calculated. Thereafter, the parallax change amount calculating means 90 outputs the parallax change amount θ _dif .
θ _dif = (θ ₂ −θ ₁ ) / (T ₂ −T ₁ ) (7)

Depth variation amount calculating means 100 reads the depth range D ₁ of the time T _1, which is stored in the depth information storing means 80 _1, and a depth range D ₂ of time T _2, which is stored in the depth information storing means 80 ₂ The change amount of the depth ranges D ₁ and D ₂ is calculated as the depth change amount D _dif . That is, the depth change amount calculation unit 100 divides the difference in the depth range (D ₂ −D ₁ ) by the delay time (T ₂ −T ₁ ) as shown in the following equation (8), thereby changing the depth change. The quantity D _dif is calculated. Thereafter, the depth change amount calculation means 100 outputs the depth change amount _Ddif .
D _dif = (D ₂ −D ₁ ) / (T ₂ −T ₁ ) (8)

  As described above, the depth information generating apparatus 1C according to the third embodiment of the present invention can obtain the parallax change amount and the depth change amount within the delay time in addition to the same effect as the first embodiment. Video production and editing can be performed more effectively.

(Fourth embodiment)
With reference to FIG. 10, the difference from the third embodiment will be described regarding the configuration of the depth information generating apparatus 1D according to the fourth embodiment of the present invention.
This depth information generation device 1D is different from the third embodiment in that a warning is issued when the calculated parallax change amount and depth change amount exceed a predetermined threshold.

For this reason, as shown in FIG. 10, the depth information generating apparatus 1D includes a planar light intensity distribution calculating unit 10 ₁ , 10 ₂ , a volume light intensity distribution storing unit 20 ₁ , 20 _2, and a parallax range calculating unit 30 ₁ , 30 _2, the depth range calculating means ₄₀ 1, 40 _2, the delay means 70, the depth information storing means ₈₀ 1, 80 _2, the parallax variation calculating means 90, the depth change amount calculation unit 100, the parallax variation Warning means 110 and depth change amount warning means 120 are provided.

The planar light intensity distribution calculating means 10 ₁ , 10 ₂ , volume light intensity distribution storing means 20 ₁ , 20 ₂ , parallax range calculating means 30 ₁ , 30 ₂ , depth range calculating means 40 ₁ , 40 ₂ , delay means 70, depth The information storage means 80 ₁ and 80 ₂ are the same as the respective means in FIG.
9 except that the parallax change amount calculating unit 90 outputs the parallax change amount to the parallax change amount warning unit 110, and the depth change amount calculating unit 100 outputs the depth change amount to the depth change amount warning unit 120. Therefore, explanation is omitted.

The parallax change amount warning unit 110 determines whether or not the parallax change amount input from the parallax change amount calculation unit 90 exceeds a parallax change amount threshold, and warns when the parallax change amount exceeds the parallax change amount threshold. It is. Here, the parallax change amount warning means 110 gives a warning by the same warning method as the parallax range warning means 50 of FIG.
This parallax change amount threshold value is set in advance as an arbitrary value, for example, indicating that the parallax change amount exceeds the allowable limit of the observer.

The depth change amount warning means 120 determines whether or not the depth change amount input from the depth change amount calculation means 100 exceeds the depth change amount threshold value, and warns when the depth change amount exceeds the depth change amount threshold value. It is. Here, the depth change amount warning means 120 gives a warning by the same warning method as the parallax range warning means 50 of FIG.
This depth change amount threshold value is set in advance as an arbitrary value, for example, indicating that the depth change amount exceeds the allowable limit of the observer.

  As described above, the depth information generating apparatus 1D according to the fourth embodiment of the present invention has the same effect as the third embodiment, and the parallax range and the depth range change beyond the allowable limit of the observer. In such a case, it is possible to reduce the situation in which a stereoscopic video that is difficult for an observer to view is displayed, and improve the quality of the stereoscopic video.

  In each embodiment, the parallax range, the depth range, the parallax change amount, and the depth change amount are calculated using the volume light intensity distribution, but the present invention is not limited to this. For example, the present invention may calculate a parallax distribution, a temporal change in the depth of a stereoscopic video for a specific subject, a parallax change amount and a depth change amount at a plurality of times from the volumetric light intensity distribution.

1,1B, 1C, 1D Depth information generation device (depth range calculation device)
10, 10 ₁ , 10 ₂ plane light intensity distribution calculating means 11 distributing means 13 element image converting means 13a light wave calculating means (first light wave calculating means)
13b Phase shift means 13c Light wave calculation means (second light wave calculation means)
15 coupling means 20, 20 ₁ , 20 ₂ volume light intensity distribution storage means 30, 30 ₁ , 30 ₂ parallax range calculation means 40, 40 ₁ , 40 ₂ depth range calculation means 50 parallax range warning means 60 depth range warning means 70 delay Means 80 ₁ , 80 ₂ Depth information storage means 90 Parallax change amount calculation means 100 Depth change amount calculation means 110 Parallax change amount warning means 120 Depth change amount warning means

Claims (6)

A depth range calculation device that calculates a parallax range and a depth range of a stereoscopic image of a subject from a group of element images obtained by photographing a subject with a stereoscopic image photographing device using an integral photography system,
A virtual element optical system having the same pitch and focal length of an element optical system of a stereoscopic image display apparatus that inputs an element image group captured by the stereoscopic image capturing apparatus and displays the element image group with respect to the element image group. A planar light intensity distribution that is a light intensity distribution for each preset distance plane that is different in distance from the virtual element optical system group by performing wave optical calculation via the virtual element optical system group arranged in two dimensions. Planar light intensity distribution calculating means for calculating
A volume light intensity distribution storage means for storing a plane light intensity distribution for each distance plane calculated by the plane light intensity distribution calculation means as a volume light intensity distribution in a three-dimensional space;
At the observer position preset with reference to the virtual element optical system group, the positions where the volume light intensity distribution becomes zero in the depth direction are respectively calculated as the near point and the far point of the stereoscopic image of the subject, A parallax range calculating means for calculating a difference in the amount of angular parallax between the point and the far point as the parallax range;
A depth range calculation means for calculating a distance between the observer position and the near point, and a distance between the observer position and the far point, and calculating a difference between the two distances as the depth range;
A depth range calculation apparatus comprising: The planar light intensity distribution calculating means includes
Distributing means for distributing the element image group captured by the stereoscopic image capturing device for each element image based on the pitch of the virtual element optical system;
First light wave calculating means for calculating the light wave incident on the virtual element optical system by propagating the light wave of the element image for the focal length of the virtual element optical system for each element image distributed by the distributing means. When,
Phase shift means for shifting the light wave of each element image incident on the virtual element optical system calculated by the first light wave calculation means by the phase of the virtual element optical system;
A second light wave calculation means for calculating the light wave of the element image for each distance plane by propagating the light wave phase-shifted by the phase shift means to each of the distance planes;
Combining means for combining the light waves of the element images calculated by the second light wave calculating means for the distance planes by the element image group to form the planar light intensity distribution;
The depth range calculation apparatus according to claim 1, comprising: A delay unit that delays the element image group captured by the stereoscopic image capturing apparatus by a preset delay time and outputs the delay element image group;
The planar light intensity distribution calculating means calculates the planar light intensity distribution from the element image group and the delayed element image group delayed by the delay means,
The volume light intensity distribution storage means stores the planar light intensity distribution calculated from the element image group and the delay element image group as the volume light intensity distribution,
The parallax range calculation means calculates the parallax range from the volume light intensity distribution of the element image group and the delay element image group,
The depth range calculation means calculates the depth range from the volume light intensity distribution of the element image group and the delay element image group,
Parallax change amount calculating means for calculating a change amount of the parallax range between the element image group and the delay element image group calculated by the parallax range calculating means as a parallax change amount;
A depth change amount calculating means for calculating a change amount of the depth range between the element image group and the delay element image group calculated by the depth range calculating means as a depth change amount;
The depth range calculation apparatus according to claim 1, further comprising: A parallax range warning unit that determines whether or not the parallax range calculated by the parallax range calculation unit exceeds a preset parallax range threshold and warns when the parallax range exceeds the parallax range threshold;
A depth range warning unit that determines whether or not the depth range calculated by the depth range calculation unit exceeds a preset depth range threshold, and warns when the depth range exceeds the depth range threshold;
The depth range calculation apparatus according to claim 1, further comprising: A parallax change amount that determines whether the parallax change amount calculated by the parallax change amount calculation unit exceeds a preset parallax change amount threshold and warns when the parallax change amount exceeds the parallax change amount threshold Warning means;
Depth change amount that determines whether or not the depth change amount calculated by the depth change amount calculation unit exceeds a preset depth change amount threshold value and warns when the depth change amount exceeds the depth change amount threshold value Warning means;
The depth range calculation apparatus according to claim 3, further comprising: In order to calculate a parallax range and a depth range of a stereoscopic image of the subject from a group of element images in which the stereoscopic image capturing device using the integral photography system has captured the subject, a computer including volumetric light intensity distribution storage means,
A virtual element optical system having the same pitch and focal length of an element optical system of a stereoscopic image display apparatus that inputs an element image group captured by the stereoscopic image capturing apparatus and displays the element image group with respect to the element image group. A planar light intensity distribution that is a light intensity distribution for each preset distance plane that is different in distance from the virtual element optical system group by performing wave optical calculation via the virtual element optical system group arranged in two dimensions. A plane light intensity distribution calculating means for writing the calculated plane light intensity distribution into the volume light intensity distribution storage means as a volume light intensity distribution in a three-dimensional space;
At the observer position preset with reference to the virtual element optical system group, the positions where the volume light intensity distribution becomes zero in the depth direction are respectively calculated as the near point and the far point of the stereoscopic image of the subject, A parallax range calculating means for calculating a difference in the amount of angular parallax between the point and the far point as the parallax range;
A depth range calculating means for calculating a distance between the observer position and the near point, and a distance between the observer position and the far point, and calculating a difference between the two distances as the depth range;
Depth range calculation program to function as

Images