JP4875680B2 - 3D information integration apparatus and 3D information integration program - Google Patents

Description

  The present invention acquires a plurality of three-dimensional information (group of element images) composed of a plurality of element images by an integral photography (hereinafter referred to as IP) method, and a ratio of sizes for each of the acquired plurality of three-dimensional information. The present invention relates to a three-dimensional information integration apparatus and a three-dimensional information integration program for performing depth control and integrating them to generate one piece of three-dimensional information.

  In general, among the methods for acquiring three-dimensional information of a subject through a lens array or a spatial filter, an IP method is known as a method for capturing and displaying a stereoscopic image using a minute optical element array.

  Here, normal stereoscopic image shooting based on the IP method will be described with reference to FIGS. 16 (a) and 16 (b). The stereoscopic image capturing apparatus illustrated in FIG. 16A captures an object 111 from a capturing direction 114 indicated by an arrow through a lens group 112 configured by, for example, a convex lens. Here, the photographing direction 114 is a direction in which the stereoscopic image photographing device photographs the subject 111 arranged in front of the lens group 112 (leftward in FIG. 16A). At this time, on the imaging plate 113 behind the lens group 112 (right side in FIG. 16A), the same number of images of the subject 111, for example, the image 115 is formed as the number of convex lenses constituting the lens group 112. . Here, the imaging plate 113 is an information acquisition device configured to include a plurality of imaging elements arranged on a substrate. Each image sensor is, for example, a CCD (Charge Coupled Device) image sensor.

  FIG. 16B is a diagram showing a stereoscopic image display device including a lens group 122 in which convex lenses are arranged in one plane and a display element 123, a stereoscopic image 121, an observation direction 124, and an image 125 of the lens group. . Here, the display element 123 displays an image 125 corresponding to the image 115 photographed by the photographing plate 113 of the stereoscopic image photographing device shown in FIG. The display element 123 is composed of an information display device such as a liquid crystal panel. As a result of the display from the stereoscopic image display device, a stereoscopic image 121 is generated at the same place as the place where the subject 111 exists, as shown in FIG. However, the stereoscopic image 121 is generated as a reverse view image. That is, as shown in FIG. 16A, when viewed from the shooting direction 114, the cylinder of the subject 111 is present in front of the prism. However, as shown in FIG. 16B, in the stereoscopic image 121 corresponding to the subject 111, a prism is generated in front of the cylinder as viewed from the observation direction 124.

  In FIGS. 16A and 16B, the lens group is displayed as a minute optical element array, and the operation is described on the assumption that the three-dimensional information of the subject is acquired and displayed using this lens group. As a minute optical element array, a minute aperture array (spatial filter) may be used.

  In FIG. 16A, the size (height) of the prism of the subject 111 is xc, the distance from the lens group 112 to the prism of the subject 111 is zc, and the distance from the lens group 112 to the surface on which the image 115 is photographed. Let dc be the pitch of the convex lens that constitutes the lens group 112, and pc be the size (height) of the image of the subject that is generated by the convex lens that constitutes the lens group 112. In FIG. 16B, the size (height) of the prism of the stereoscopic image 121 is xr, the distance from the lens group 122 to the prism of the stereoscopic image 121 is zr, and the surface on which the image 125 is displayed from the lens group 122. , The convex lens pitch constituting the lens group 122 is pr, and the size (height) of each image displayed on the display element 123 is kr. In this case, the relationship between xc and xr is expressed by Equation (101) by Non-Patent Document 1.

ここで、

here,

  From the relationship of the above-described formula (102), the pitch pc of the convex lenses constituting the lens group 112 of the stereoscopic image capturing apparatus shown in FIG. 16A and the lens group of the stereoscopic image display apparatus shown in FIG. By making the pitch pr of the convex lenses constituting 122 different from each other, it is derived that the ratio xr / xc of the size of the stereoscopic image 121 with respect to the subject 111 can be controlled. Similarly, the ratio xr / xc of the size of the stereoscopic image 121 to the subject 111 can be controlled by changing the size kc of the photographed subject image 115 and the size kr of the displayed image 125. Is guided.

  Conventionally, in order to solve the problem of generating a reverse-view image in which the depth is inverted compared to the subject, the information acquired by the stereoscopic image capturing device shown in FIG. A stereoscopic image depth conversion device that inputs the information after the input to the stereoscopic image display device shown in FIG. 16B and finally generates a stereoscopic image with the correct depth is disclosed (see Patent Document 1).

  A stereoscopic image depth conversion device disclosed in Patent Document 1 will be described with reference to FIGS. 17 (a) and 17 (b). FIG. 17A shows the image 131 acquired in FIG. 16A, the first virtual lens array 132, the virtually generated stereoscopic image 133, the second virtual lens array 134, and the second. It is a figure which shows the image 135 (image of the three-dimensional image 133 produced | generated virtually) produced | generated by the virtual lens array 134 of FIG. Here, the image 135 is not optically generated, but is generated by calculation processing of the stereoscopic image depth conversion device.

FIG. 17B shows an image 141 generated by the arithmetic processing in FIG. 17A (that is, the image 135 in FIG. 16A), a lens group 142 in which convex lenses are arranged in one plane, a stereoscopic image 143, and an observation. FIG. As shown in FIG. 17B, for example, as a result of the operation of the stereoscopic image display apparatus of FIG. 16B, the cylinder is observed in front of the prism as viewed from the observation direction 144. The stereoscopic image 143 is equivalent to the depth relationship of the subject in FIG.
JP 2007-114483 A J. Opt. Soc. Am. A, Vol. 21, pp. 951-958, 2004

By the way, in recent years, in order to further enhance the production effect, establishment of a technique for integrating a plurality of three-dimensional information and generating one three-dimensional information is desired.
However, although the conventional technology can perform the size ratio and depth control processing of a single three-dimensional space, after performing the size ratio and depth control processing for each of a plurality of three-dimensional information, It is not possible to generate one three-dimensional information by integrating. In addition, when a plurality of three-dimensional information is integrated to generate one three-dimensional information, when the three-dimensional information is displayed as a stereoscopic image, the plurality of three-dimensional information overlap each other in the depth direction. It is also an important problem to prevent the so-called phantom phenomenon from occurring in the part.

  In view of such circumstances, the present invention performs a size ratio and depth control processing for each of a plurality of three-dimensional information without causing a so-called phantom phenomenon in a portion where the plurality of three-dimensional information overlap each other. An object of the present invention is to provide a three-dimensional information integration apparatus and a three-dimensional information integration program capable of integrating these and generating one piece of three-dimensional information.

  The present invention was created to achieve the above object, and first, the three-dimensional information integration device according to claim 1 is acquired by an integral photography method in order to display a subject as a stereoscopic image. A three-dimensional information integration device for generating a single three-dimensional information by integrating a plurality of three-dimensional information composed of a plurality of element images, comprising: an element image separation means, a pixel storage means, and a stereoscopic image depth control means And integrated three-dimensional information storage means, pixel value assigned position storage means, and output means.

  According to such a configuration, the three-dimensional information integration device exists in a desired depth range from each element image for each of a plurality of element image groups based on the operation signal input from the outside by the element image separation unit. A pixel corresponding to the subject area to be processed and a pixel corresponding to the other area are separated. In the three-dimensional information integration device, the element image separation unit outputs and stores the processing target element image group including the pixels corresponding to the subject area to the pixel storage unit.

Further, the three-dimensional information integration apparatus is a three-dimensional information integration unit that is specified by the stereoscopic image depth control unit for each of the subjects associated with each of the plurality of processing target element image groups stored in the pixel storage unit. Are read from the pixel storage means in order from the processing target element image group associated with the subject arranged in the forefront in the integrated three-dimensional information specified by a value indicating an arbitrary depth position in the image, and the read processing target the elements images, processing is controlled to be any value that the size ratio of the reproduced image relative to size of the object is set in advance, and performs a process of controlling the depth according to the value that indicates the depth position processing A completed element image group is generated.

  Then, the 3D information integration apparatus stores the processed element image group generated by the stereoscopic image depth control unit by the integrated 3D information storage unit. Then, the three-dimensional information integration apparatus stores the position where the pixel value is assigned to the integrated three-dimensional information storage means, which is input from the stereoscopic image depth control means, by the pixel value assigned position storage means.

  Here, the three-dimensional information integration apparatus refers to the pixel value assigned position storage unit by the stereoscopic image depth control unit for the processing target element image group that is the second and subsequent integration target processing units, and Of the pixels to which the image group is to be assigned, only the pixels to which no information has been assigned yet in the integrated three-dimensional information storage means are assigned pixel values whose size ratio and depth are controlled to the pixels of the integrated three-dimensional information storage means.

  According to this, the size ratio and depth of the plurality of three-dimensional information can be arbitrarily controlled by the operator and integrated. Further, in the integrated 3D information, the size ratio and the depth control process are performed in order from the element image group of the subject arranged at the foremost side, the pixel values are assigned to the integrated 3D information storage means, and a plurality of pixels are arranged in the depth direction. Since the pixel value is not assigned, when the integrated 3D information is displayed as a stereoscopic image, only the subject arranged in front is displayed for the portion where a plurality of subjects overlap. Thereby, it is possible to prevent a so-called phantom phenomenon from occurring in a portion where a plurality of subjects overlap each other.

  Also, the 3D information integration apparatus outputs the integrated 3D information stored in the integrated 3D information storage means to the outside by the output means after the size ratio and depth control for all the processing target element image groups is completed. Output. For example, a display device or a storage medium corresponds to the output destination.

The three-dimensional information integration device according to claim 2 is the stereoscopic image display device according to claim 1, wherein the stereoscopic image depth control means distributes the light wave of the processing target element image group for each element image. And the second virtual aperture group in which the light wave of the element image distributed by the distribution means passes through the first virtual aperture group and the pitch of the aperture is different from that of the first virtual aperture group. First element image conversion means for converting light waves propagating to the first element image conversion means, and light waves of the respective element images converted by the first element image conversion means for the second virtual aperture group by the distributed number. An addition means for adding on the entrance surface, a redistribution means for redistributing the light wave added by the addition means for each element image, and a light wave of the element image distributed by the redistribution means The distance from the virtual aperture group to the image formation A second element image converting means for converting into a propagated light wave, and a combination for generating the processed element image group by combining the redistributed number of light waves converted by the second element image converting means. Means for controlling the size ratio of the processing target element image group, the first virtual opening group and the second virtual opening group set to arbitrary values in advance. And the pitch ratio Φ.
Further, the 3D information integration apparatus according to claim 2 is the stereoscopic image display apparatus according to claim 1 or 2, wherein the size of all the processing target element image groups in the stereoscopic image depth control means is the same. After completion of the ratio and depth control processing, the integrated three-dimensional information storage means further comprises pixel value assigning means for assigning pixel values to pixels to which pixel values have not yet been assigned based on operation signals input from the outside. Features. Thereby, pixel values can be assigned to all the pixels of the integrated three-dimensional information storage means.

According to a fourth aspect of the present invention, there is provided a three-dimensional information integration program for integrating a plurality of three-dimensional information composed of a plurality of element images to generate a single piece of three-dimensional information. And a three-dimensional image depth control unit, an integrated three-dimensional information storage unit, a pixel value assigned position storage unit, and an output unit.

The present invention has the following excellent effects. According to the invention described in claim 1, 2 and claim 4, it is possible to produce an integrated three-dimensional information that integrates a plurality of three-dimensional information. Further, since the pixel value is not assigned to the position where the pixel value has already been assigned in the integrated three-dimensional information storage means, when the integrated three-dimensional information is displayed as a stereoscopic image, a plurality of the pixel values are not assigned. It is possible to generate integrated three-dimensional information in which a so-called phantom phenomenon does not occur in a portion where subjects overlap each other. In addition, the size ratio and depth of a plurality of subjects can be arbitrarily controlled by the operator and integrated. For this reason, various variations of stereoscopic images can be generated, and the effect of rendering the images can be enhanced. Furthermore, according to the present invention, integrated three-dimensional information obtained by integrating a plurality of three-dimensional information can be generated by calculation, so that the configuration of the apparatus can be simplified. According to the invention described in claim 3 , in addition to the effect of the invention described in claim 1, it is possible to assign a pixel value to a pixel to which the integrated 3D information is not allocated in the integrated 3D information storage means. it can.

Hereinafter, the present embodiment will be described in detail with reference to the drawings as appropriate.
[Outline]
First, prior to detailed description of the three-dimensional information integration apparatus according to the present embodiment, an outline of the present invention will be briefly described with reference to FIG. FIG. 1 is a diagram for explaining the outline of the present invention, for explaining how a plurality of three-dimensional information (element image group) is integrated to generate one three-dimensional information (element image group). It is explanatory drawing.

As shown in FIG. 1, element image group 30 ₁ and the element image group 30 _2, the film was shot inverted image of the reduced object corresponding to the size and number of pinholes or convex lens (element image) It will be. In this embodiment, the element image group 30 ₁ is obtained by photographing an object and a background consisting of a cylinder and triangular pyramid, elemental image group 30 ₂ is obtained by photographing an object and a background consisting of spherical and triangular pyramid is there. Three-dimensional information integration device of the present invention, a portion of the element images 30 _1, for example, a subject area, a part of the elemental image group 30 _2, for example, for each of the subject region, the magnitude ratio and the depth control processing After performing, it integrates and produces | generates integrated three-dimensional information as shown in the rightmost figure of FIG.

  Further, the three-dimensional information integration device of the present invention provides the ratio of the size of each of the plurality of element image groups, regardless of the size and depth of the object in the element image group obtained by photographing the subject by the IP method. It is possible to integrate after the depth is arbitrarily controlled by the operator.

[Configuration of 3D information integration device]
Hereinafter, the configuration of the three-dimensional information integration device 1 will be described with reference to FIG. FIG. 2 is a block diagram showing the configuration of the three-dimensional information integration apparatus according to this embodiment.
As shown in FIG. 2, the three-dimensional information integration device 1 according to this embodiment includes an element image separation unit 2, a pixel storage unit 3, a stereoscopic image depth control unit 4, an integrated three-dimensional information storage unit 5, A pixel value assigned position storage means 6, a pixel value assignment means 7, and an output means 8 are provided.

  The element image separation means 2 receives a plurality of three-dimensional information (element image groups) photographed by the IP method based on an operation signal input from the outside, and each of the plurality of element image groups is obtained from each element image. The pixel corresponding to the subject area existing in the desired depth range and the pixel corresponding to the other area are separated.

  The element image separation means 2 corresponds to, for example, one that uses color / luminance components of a video signal and one that uses depth information of a subject. A chroma key technique can be used as a method of using the color component of the video signal. This is a video having a specific color from individual element images based on an operation signal input from the outside by generating a three-dimensional information by placing a subject in front of a background of a specific color (key signal). Only the signal is separated as the key signal, and only the video signal other than the key signal is extracted. According to this, it is possible to separate the subject area (video signal other than the key signal) existing in the desired depth range and the other area (key signal).

Further, in the method using the depth information of the subject, the depth information of the subject is calculated from each element image, and the subject region existing in the desired depth range is separated from the other region using the calculated depth information. be able to.
When a desired depth range is input by the operator via the input unit for each of the plurality of element image groups, the element image separation unit 2 determines the depth from each element image based on the calculated depth information. By extracting the pixels existing in the range, the subject region existing in the desired depth range and the other region (background region) can be separated. If a plurality of subjects are overlapped, all the plurality of subjects are extracted, or only the foremost subject is extracted. Pixels corresponding to a subject existing in a desired depth range separated by the element image separation unit 2 are output to the pixel storage unit 3 and stored as a processing target element image group.

  Here, the operation signal input from the outside means an input of a predetermined operation signal to the three-dimensional information integration apparatus 1 by the operator via the input unit 20 in the present embodiment. The input means 20 corresponds to, for example, a computer keyboard, and the input of a predetermined operation signal corresponds to, for example, an operation of a computer keyboard. For example, in the method using the depth information of the subject, inputting a numerical value of the depth range with a computer keyboard is applicable.

  The pixel storage unit 3 stores a plurality of processing target element image groups composed of pixels corresponding to a subject area existing in a desired depth range input from the element image separation unit 2. The pixel storage means 3 is, for example, a hard disk, an optical memory, a magnetic disk or the like having a sufficient capacity for storing an image.

  The three-dimensional image depth control means 4 starts from the pixel storage means 3 in order from the processing target element image group that is specified by the operation signal input from the outside and that is associated with the subject that is arranged in the forefront in the integrated three-dimensional information. The read and the size ratio and depth control of the read processing target element image group are performed.

Here, the operation signal input from the outside corresponds to, for example, a numerical value representing the depth position of the subject in the integrated three-dimensional information when the input unit 20 is a computer keyboard.
For example, when the operator inputs an arbitrary numerical value such as a depth value “0” or a depth value “51”, the depth position of the subject in the integrated 3D information is determined.

  The control of the ratio of the size of the pixel corresponding to the subject and the depth in the stereoscopic image depth control unit 4 can be performed using the size and depth of the subject to be processed. Information on the size and depth of the subject to be processed may be obtained by acquiring in advance information on the size and depth of the subject when the three-dimensional information is generated. Further, as shown in Japanese Patent Laid-Open No. 2002-51358, depth information of a subject calculated from an element image group may be used.

As the stereoscopic image depth control means 4, for example, a stereoscopic image depth conversion device as described in JP 2007-114483 A can be used.
Next, the configuration of the stereoscopic image depth control means 4 according to the present embodiment will be described in detail with reference to FIG. 3 and FIG. 4 as appropriate. FIG. 3 is a block diagram illustrating the configuration of the stereoscopic image depth control unit according to the present embodiment, and FIG. 4 illustrates a method for controlling the depth of the stereoscopic image by calculation using the stereoscopic image depth control unit according to the present embodiment. It is a schematic diagram for doing.

The element image group input to the stereoscopic image depth control means 4 is a video signal captured by a CCD or the like in a stereoscopic image capturing device (not shown), or a video equivalent to a CCD generated by a computer. Signal. The stereoscopic image depth control means 4 according to the present embodiment performs the depth control of the stereoscopic image by the optical system as shown in FIG.

  As shown in FIG. 3, the stereoscopic image depth control unit 4 includes a distribution unit 41, an element image conversion unit 42 for element images, a depth conversion unit 43, an element image reproduction unit 44, and an addition unit 45. .

The distribution unit 41 divides the input element image group into element image units. Here, the distribution unit 41 outputs the light wave (gs _{, m} (x _s) of the m-th element image of the processing target element image group displayed on the display surface of the first element image surface 11a in the input element image group. _{, M 1} , y _{s, m 2} )) are output to the element image conversion means 42 previously associated with the mth element image. Here, the light wave is a complex number representing the amplitude and phase when the pixel value is handled as a wave of light.

  Element image conversion means (processing object element image light wave conversion means) 42 converts the light wave of the element image into a light wave when passing through an element lens (virtual pinhole or lens) having a predetermined focal length. is there. Here, the element image conversion means 42 includes a light wave calculation means 42a and a phase shift means 42b.

The light wave calculating means 42a calculates the light wave incident on the element lens (virtual pinhole or lens) by Fresnel approximation of the light wave of the element image. Lightwave outputted from the light wave meter Sante stage 42a is output to the phase shift means 42b.

The phase shift unit 42b shifts the phase by the phase corresponding to the virtual element lens from the light wave (R _{i, m} (x _{o, m} , y _{o, m} )) of the element image input from the light wave calculation unit 42a. The calculated light wave is calculated.
The light wave (R _{o, m} (x _{o, m} , y _{o, m} )) for each element image is output to the depth conversion means 43.

The depth conversion unit 43 converts the light wave converted by the element image conversion unit 42 into a light wave when passing through a virtual depth control lens L (convex lens) having a predetermined focal length. Here, the depth conversion means 43 includes a phase shift means 43a and a redistribution means 43b.
The phase shift unit 43a calculates a light wave obtained by shifting the phase of the light wave converted by the element image conversion unit 42 by a phase corresponding to a virtual convex lens.

The redistribution unit 43b uses the light wave (R _{d, m} (x _{o, m} , y _{o, m} )) for each element image input from the phase shift unit 43a, and the apertures of the second virtual aperture group 12b. The light wave incident on the (element lens) is calculated.
The light wave (R _{p, n, m} (x _{p, n} , y _{p, n} )) for each opening of the second virtual opening group 12 b is output to the element image reproduction means 44.

  The element image reproduction means (processed element image light wave conversion means) 44 is a light wave when the light wave converted by the depth conversion means 43 passes through an element lens (virtual pinhole or lens) having a predetermined focal length. It is to convert to. Here, the element image reproduction means 44 includes a phase shift means 44a and a light wave calculation means 44b.

The phase shift means 44a shifts the phase from the light wave (R _{p, n, m} (x _{p, n} , y _{p, n} )) converted by the depth conversion means 43 by the phase corresponding to the virtual element lens. The light wave is calculated.

  The light wave calculating means 44b calculates the light wave that is emitted from the element lens (virtual pinhole or lens) and reaches the imaging surface of the second element image surface 12a by approximating the light wave of each element lens by Fresnel approximation. It is.

  The adding means 45 calculates the sum of the light wave power from the light wave for each element image output from the element image reproducing means 44, thereby obtaining a video signal subjected to depth conversion processing, that is, a second element image group ( A video signal to be a processed element image group) is generated. Further, the adding means 45 refers to the pixel value assigned position storage means 6 when calculating the total sum of the lightwave power, and for the pixels corresponding to the positions to which the pixel values are assigned in the integrated three-dimensional information storage means 5. , Do not calculate.

The stereoscopic image depth control means 4 configured as described above can generate an image (processed element image group) in which the depth of a reproduced image displayed as a stereoscopic image is changed by calculation.
Further, the stereoscopic image depth control means 4 outputs the size ratio and the pixel value of the processed element image group subjected to the depth control processing to the integrated three-dimensional information storage means 5 for storage.

  Further, the stereoscopic image depth control means 4 outputs the information on the position to which the pixel value is assigned in the integrated three-dimensional information storage means 5 to the pixel value assigned position storage means 6 for storage. For example, the stereoscopic image depth control means 4 assigns “1” to the pixel of the pixel value assigned position storage means 6 corresponding to the pixel to which the pixel value is assigned in the integrated three-dimensional information storage means 5, thereby integrating the three-dimensional information storage. "0" is assigned to the pixel in the pixel value assigned position storage means corresponding to the pixel to which no pixel value is assigned in the means 5.

  Returning to FIG. 2, the integrated three-dimensional information storage means 5 stores the pixels whose size ratio and depth are controlled, which are input from the stereoscopic image depth control means 4. The integrated 3D information storage means 5 has a capacity capable of storing all processed element image groups input from the stereoscopic image depth control means 4.

  The pixel value assigned position storage means 6 stores the position where the pixel value is assigned to the integrated three-dimensional information storage means 5 input from the stereoscopic image depth control means 4. The pixel value assigned position storage means 6 has a capacity capable of storing all the positions where the pixel values are assigned to the integrated three-dimensional information storage means 5.

  The pixel value allocating unit 7 determines that the pixel value is not yet stored in the integrated three-dimensional information storage unit 5 based on the operation signal input from the outside after the size ratio and depth control processing for all the processing target element image groups. Pixel values are assigned to pixels that are not assigned. The output unit 8 outputs the integrated 3D information stored in the integrated 3D information storage unit 5 to the outside. The output destination is, for example, a display device or a storage medium. Note that the pixel value assigning means 7 may not be provided. In this case, no pixel value is assigned to the pixel to which the integrated 3D information is not assigned in the integrated 3D information storage means 5 (the video signal is set to 0).

[Operation of 3D information integration device]
Hereinafter, the operation of the three-dimensional information integration device 1 according to the present embodiment will be described with reference to FIGS. 5 to 8 and FIGS. 2 to 4 as appropriate. Specifically, the operation of the three-dimensional information integration device 1 when integrating subjects arranged in different three-dimensional spaces will be described.
FIG. 5 to be referred to is a conceptual diagram for explaining the operation of the element image separation unit of the three-dimensional information integration apparatus according to the present embodiment, and FIG. 6 illustrates the processing order of the processing target element image group stored in the pixel storage unit. FIGS. 7A to 7E are conceptual diagrams for explaining a state of being determined. FIG. 7A to FIG. 7E are pixel values stored in the integrated three-dimensional information storage unit by the stereoscopic image depth control unit of the three-dimensional information integration apparatus according to the present embodiment. (F) to (h) are conceptual diagrams for explaining a state in which information on positions where pixel values are assigned to the integrated three-dimensional information storage means is assigned to the pixel value assigned position storage means. FIG. 8 is a flowchart showing the operation of the three-dimensional information integration apparatus according to this embodiment. As shown in FIGS. 7A and 7F, it is assumed that the integrated three-dimensional information storage unit 5 and the pixel value assigned position storage unit 6 are in an initial state.

As shown in FIG. 2 and FIG. 5, when a plurality of element image groups 30 are input from the outside, the three-dimensional information integration device 1 uses a plurality of element image separation means 2 based on operation signals input from the outside. For each of the element image groups 30 (element image group 30 ₁ to element image group 30 _n ), a subject area existing in a desired depth range and an area existing in other ranges are separated from each element image.

For example, as shown in FIG. 5, the elemental image group 30 _3, and a triangle present in the depth range specified by the operation signal, and the object region composed of rectangles arranged in the rear overlapping part and a triangle, otherwise The region existing in the range (the background region represented in light gray in FIG. 5) is separated. In the element image group 30 _n , a circular subject region existing in the depth range specified by the operation signal and a region existing in the other range (a triangular subject region and a background region represented by dark gray) are displayed. To separate.
Then, the element image separating unit 2 converts the plurality of processing target element image groups 31 (processing target element image group 31 ₁ to processing target element image group 31 _n ) thus separated from the plurality of element image groups 30 respectively. The data is output to the pixel storage means 3 and stored sequentially (step S1).

  As shown in FIGS. 2 and 6, the three-dimensional information integration device 1 includes a plurality of pieces of information stored in the pixel storage unit 3 according to the order specified by the operation signal input from the outside by the stereoscopic image depth control unit 4. The processing target element image group 31 is read out. The order here is the order in which the plurality of subjects are arranged in the depth direction in the integrated three-dimensional information arbitrarily determined by the operator, and the stereoscopic image depth control means 4 includes the plurality of processing target element image groups 31. Among them, the processing target element image group associated with the subject arranged closest in the integrated 3D information is sequentially read from the pixel storage unit 3 (step S2).

For example, in the present embodiment, as illustrated in the left diagram of FIG. 6, the processing target element image group 31 ₁ to the processing target element image group 31 _n are stored by the pixel storage unit 3. , In the order specified by the operation signal, for example, as shown in the right diagram of FIG. 6 in this embodiment, the processing target element image group 31 ₃ , the processing target element image group 31 _n , the processing target element image group 31 _1. read in the order of the processing target element image group 31 _2. Note that the processing order of the plurality of processing target element image groups may be determined at once for all the processing target element image groups, or each time processing for one processing target element image group ends, the next processing is performed. A processing target element image group to be performed may be determined.

  Next, the three-dimensional information integration device 1 reads the position where the pixel value has already been assigned in the integrated three-dimensional information storage means 5 from the pixel value assigned position storage means 6 by the stereoscopic image depth control means 4 (Step 3). S3).

Then, the three-dimensional image depth control means 4 performs the ratio and depth control processing size of the processing target element image group 31 ₃ assigns a pixel value after the control in the integrated three-dimensional information storage unit 5 (step S4). However, the three-dimensional image depth control means 4 as a result of referring to the pixel values assigned position storage means 6, to the pixel value obtained after the ratio of the size of the processing target element image group 31 ₃ and the depth control processing is allocated If already pixel values of the other processed element images in the pixel are allocated, except for the pixel locations corresponding to the assigned pixel, and generates the processed element image group 32 _3.

That is, the stereoscopic image depth control means 4 operates as follows. Stereoscopic image depth control means 4, by the distribution means 41, light waves of the first element image plane 11a processing is displayed on the display surface target element image group 31 ₃ of the m-th element image in an inputted video signal (g _{s, m} (x _{s, m} , y _{s, m} )) is output to the element image conversion means 42 previously associated with the m-th element image.

The stereoscopic image depth control unit 4 performs a general Fresnel approximation as a signal corresponding to a light wave reaching the mth opening (element lens or the like) of the first virtual aperture group 11b by the light wave calculation unit 42a. Then, the light wave (R _{i, m} (x _{o, m} , _{yo, m} )) for each element image is calculated by the following equation (2).

Here, x _{s, m} and x _{o, m} are the x coordinate of the center of the m-th element image in the entire image (processing target element image group), and the m-th opening of the first virtual opening group 11b. X coordinate from the center of the optical axis of the part. Y _{s, m} , _{yo, m} are the y-coordinate of the center of the m-th element image in the entire image (processing target element image group), and the m-th opening of the first virtual opening group 11b. Y coordinate from the optical axis center. F ₁ is the focal length of the aperture (element lens) of the first virtual aperture group 11b, and k is the wave number 2π / λ (λ is the wavelength).
The light wave (R _{i, m} (x _{o, m} , y _{o, m} )) for each element image is output to the phase shift means 42b.

The stereoscopic image depth control unit 4 uses the phase shift unit 42b to generate light waves (R _{i, m} (x _{o, m} , y _{o, m} ) corresponding to the phase corresponding to the element lens, as shown in the following expression (3). ) Is calculated _, a signal (light wave R _{o, m} (x _{o, m} , y _{o, m} )) corresponding to the light wave emitted from the element lens is calculated.

The light wave (R _{o, m} (x _{o, m} , y _{o, m} )) for each element image is output to the depth conversion means 43.
The stereoscopic image depth control unit 4 causes the phase shift unit 43a of the depth conversion unit 43 to generate light waves (R _{o, m} (x _o, x), as much as the phase corresponding to the depth control lens L, as shown in the following equation (4) _{. m} , _{yo, m} )) is shifted to calculate a signal (light wave _{Rd, m} ( _{xo, m} , _{yo, m} )) corresponding to the light wave emitted from the depth control lens L.

Here, x ₀ and y ₀ are distances from the optical center of the depth control lens L. If the pitch of the apertures (element lenses) of the first virtual aperture group 11b is P, the relationship of x _o = x _{o, m} + mP, y _o = y _{o, m} + mP is established.
The light wave (R _{d, m} (x _{o, m} , y _{o, m} )) for each element image calculated by the phase shift means 43a is output to the redistribution means 43b.

The light wave (R _{d, m} (x _{o, m} , y _{o, m} )) calculated by the phase shift means 43a not only reaches the mth element lens but also reaches the surrounding element lens. Become. Therefore, the redistribution unit 43b causes the light wave (R _{d, m} (x _{o, m} , y _{o, m} )) input from the phase shift unit 43a to the second virtual aperture group 12b. The light wave (R _{p, n, m} (x _{p, n} , y _{p, n} )) that has reached the nth element lens, which is the result of being distributed to the aperture (element lens), is obtained using a general Fresnel approximation. Thus, calculation is performed according to the following equation (5).

Here, x _{p, n} and y _{p, n} are the x coordinate and the y coordinate from the optical center of the aperture (element lens) of the nth second virtual aperture group 12b, respectively. L _d is the distance between the depth control lens L and the second virtual aperture group 12b.
The light wave (R _{p, n, m} (x _{p, n} , y _{p, n} )) for each opening of the second virtual opening group 12 b is output to the element image reproduction means 44.

The stereoscopic image depth control means 4 is obtained from the light wave (R _{p, n, m} (x _{p, n} , y _{p, n} )) converted by the depth conversion means 43 by the phase shift means 44a of the element image reproduction means 44. A light wave whose phase is shifted by a phase corresponding to a virtual element lens is calculated. That is, the phase shift means 44a shifts the light wave (R _{p, n, m} (x _{p, n} , y _{p, n} )) by the phase corresponding to the element lens, as shown in the following formula (6). Thus, a signal (light wave R _{r, n, m} (x _{o, m} , _{yo, m} )) corresponding to the light wave emitted from the element lens is calculated.

Here, f ₂ is the focal length of the opening of the second virtual opening group 12b (element lens). The light wave (R _{r, n, m} (x _{p, n} , y _{p, n} )) for each element lens is output to the light wave calculation means 44b.

The stereoscopic image depth control means 4 is emitted from the n-th element lens by the light wave calculation means 44b of the element image reproduction means 44 according to the following equation (7) and reaches the imaging surface of the second element image surface 12a. The light wave (R _{e, n, m} (x _{e, n} , y _{e, n} )) is calculated.

Here, x _{e, n} and y _{e, n} are the x coordinate and y coordinate from the center of the element image, respectively. The light wave (R _{e, n, m} (x _{e, n} , y _{e, n} )) calculated by the light wave calculating unit 44 b is output to the adding unit 45.

By the way, the power of the light wave (R _{e, n, m} (x _{e, n} , y _{e, n} )) reaching the imaging surface of the second element image surface 12a output from the element image reproduction means 44 is the light power. It can be expressed by the square of the amplitude. Further, the light wave emitted as the element image of the processing target element image group 31 ₃ are the incoherent (not wavelength or phase is constant), the light wave phase can be considered to be uncorrelated.

Therefore, the stereoscopic image depth control means 4, the addition means 45, the total power of the light wave emitted as the element image of the processing target element image group 31 ₃ (-M~M), the following equation (8) by calculating to obtain the power of one element image to be processed element image group 31 ₃ (equation (8), the n-th element image), and outputs a video signal that is proportional to its power.

In the present embodiment, referring to the pixel value assigned position storage means 6, as shown in FIG. 7 (f), the pixel values are not assigned to all the pixels of the integrated three-dimensional information storage means 5 at this time. since it can be seen is, the adding means 45, for all pixels of the processing target element image group 31 _3, obtaining the total power of the light wave. Thus, since the processed element image group 32 ₃ as shown in FIG. 7 (b) is generated, the stereoscopic image depth control means 4 outputs to the integrated three-dimensional information storage unit 5 to be stored therein.

In this way, three-dimensional information integrating device 1, the stereoscopic image depth control means 4, as if executing the arithmetic processing as if they were optically displayed to retake the light of the processing target element image group 31 ₃ . Further, since the light wave passes through two virtual aperture groups (the first virtual aperture group 11b and the second virtual aperture group 12b) having different aperture pitches, it is changed from the size of the subject. In addition, it is possible to generate a processed element image group that displays a reproduction image having a free size.

In the three-dimensional information integration device 1, as shown in FIG. 7G, pixel values are assigned to the integrated three-dimensional information storage unit 5 by the stereoscopic image depth control unit 4 to the pixel value assigned position storage unit 6. position outputs information (processed element image group 32 ₃ is assigned a position) to be stored therein (step S6).

In addition, when there is a pixel corresponding to a position to which a pixel value has already been assigned in the integrated three-dimensional information storage unit 5 in the processing target element image group after the size ratio and depth control currently being processed. The processed element image group is generated by excluding the pixel. A case _where the processing of the processing target element image group 31 _n is performed will be described as an example. Referring to the pixel value assigned position storage means 6, as shown in FIG. 7 (g), the integrated three-dimensional information storage unit 5, it can be seen that already processed element image group 32 ₃ assigned. For this reason, the pixel at the position corresponding to the pixel to which the processed element image group 32 ₃ is assigned is excluded from the processing target element image group 31 _n subjected to the size ratio and depth control by the adding unit 45. Find the total power of light waves. In this way, as shown in FIG. 7 (c), integrated three-dimensional information storage unit 5 in the processed element image group 32 ₃ except a corresponding pixel position is assigned the processed element image group 32 _n Is generated.

As shown in FIG. 7C, the three-dimensional information integration device 1 outputs the generated processed element image group 32 _n to the integrated three-dimensional information storage unit 5 by the stereoscopic image depth control unit 4 and stores it. In FIG. 7 (c), the order of clarity, the processed element image group 32 ₃ already assigned are displayed in broken lines. Further, the three-dimensional information integration apparatus 1 assigns pixel values to the integrated three-dimensional information storage means 5 by the stereoscopic image depth control means 4 as shown in FIG. Information on the assigned position (position to which the processed element image group 32 _n is assigned) is output and stored.

Further, the three-dimensional information integration device 1 assigns pixel values to the integrated three-dimensional information storage means 5 by the stereoscopic image depth control means 4 as shown in FIG. was positioned Print information (processed element image group 32 ₃ is assigned a position) to be stored therein (step S6).

  In this way, steps S2 to S6 are repeated until the processing for all the processing target element image groups is completed.

  When the processing is completed for all the processing target element image groups (Yes in step S7), the stereoscopic image depth control unit 4 outputs a control signal that notifies the pixel value allocation unit 7 of the end of the processing.

  When the control signal is input from the stereoscopic image depth control unit 4, the pixel value allocation unit 7 reads out the pixel to which the pixel value is allocated in the integrated three-dimensional information storage unit 5 from the pixel value allocated position storage unit 6 ( Step S8).

Then, based on the operation signal input from the outside, as shown in FIG. 7D, pixel values are assigned to pixels to which no pixel value is assigned to the integrated three-dimensional information storage means 5 (step S9). For example, when the operator constructs a three-dimensional space serving as a monochromatic background, desired monochromatic information is allocated to pixels to which no information is allocated. If the pixel value assigning unit 7 is not provided, no pixel value is assigned to a pixel to which no pixel value is assigned to the integrated three-dimensional information storage unit 5. In this way, integrated three-dimensional information is generated as shown in FIG.
Then, the integrated 3D information storage unit 5 outputs the integrated 3D information to the output unit 8.

The output means 8 outputs the integrated 3D information input from the integrated 3D information storage means 5 to the outside (step S10).
The above is a series of operations of the three-dimensional information integration apparatus 1.

  The present invention is not limited to this configuration. For example, in the present embodiment, the depth information of the subject is calculated from the group of element images. However, the present invention is not limited to this, and when three-dimensional information is generated by a computer, the size and depth position information of the subject is acquired in advance. If the 3D information is generated from a video signal photographed by a CCD or the like, the subject depth information may be acquired by irradiating the subject with modulated infrared light or the like. Of course.

Further, for example, a stereoscopic image depth control unit 4B described below may be used as the stereoscopic image depth control unit instead of the stereoscopic image depth control unit 4 described above.
[Outline of configuration of stereoscopic image depth control means]
Another configuration example of the stereoscopic image depth control means will be described with reference to FIG. FIG. 9 is a block configuration diagram showing another configuration example of the stereoscopic image depth control means applied to the three-dimensional information integration device of the present invention.

  The stereoscopic image depth control means 4B converts light waves of a processing target element image group (hereinafter also referred to as “first element image group”) that is a reproduced image in which the unevenness of the subject is reversed when displayed as a stereoscopic image. , A processed element image group (hereinafter also referred to as a “second element image group”) that is a reproduced image in which the image size is changed in the same manner as the unevenness of the subject when displayed as a stereoscopic image. is there. Note that the stereoscopic image depth control means 4B is a pixel storage means in order from the processing target element image group that is specified by the operation signal input from the outside and that is associated with the subject that is arranged closest to the integrated 3D information. 3 is read and processed. The first element image group input to the stereoscopic image depth control unit 4B is a video signal captured by an imaging element such as a CCD in a stereoscopic image capturing device (not shown). The second element image group generated by the stereoscopic image depth control unit 4B is output to a stereoscopic image display device (not shown).

  As shown in FIG. 9, the stereoscopic image depth control means 4B is combined with a distribution means 41B, a first element image conversion means 42B, an addition means 43B, a redistribution means 44B, and a second element image conversion means 45B. Means 46B. The stereoscopic image depth control means 4B performs the depth control of the stereoscopic image by the optical system as shown in FIG.

  The distribution unit 41B distributes the light wave of the first element image group for each element image. Here, the distributed element image is identified by m (−M ≦ m ≦ M). The first element image conversion means 42B functions for each (2M + 1) distributed element images. The distributed element image m is then added by the adding means 43B and distributed again by the redistributing means 44B. At this time, the redistributed element images are identified by n (−N ≦ n ≦ N). The second element image conversion means 45B functions for each (2N + 1) element images that have been redistributed. The redistributed element images n are then combined by the combining unit 46B to generate a processed element image group (second element image group).

[Outline of virtual aperture group assumed in calculation processing of stereoscopic image depth control means]
Next, an outline of a virtual aperture group assumed in the calculation processing of the stereoscopic image depth control unit of the present embodiment will be described with reference to FIG. 10 (see FIG. 9 as appropriate). FIG. 10 is an explanatory diagram schematically showing an example of a virtual aperture group assumed in the calculation processing of the stereoscopic image depth control means. The first element image conversion means 42B shown in FIG. 9 passes the light wave of the element image distributed by the distribution means 41B through the first virtual aperture group 11Bb (see FIG. 10) and the second virtual image. It converts into the light wave which propagates to aperture group 12Bb (refer FIG. 10). Here, the light wave is a complex number representing the amplitude and phase when a video signal is handled as a wave of light. In addition, the elements (openings) constituting the aperture group mean optical elements such as lenses and spatial filters such as pinholes. Hereinafter, the aperture is represented by a convex lens (element lens) as shown in FIG. Further, as shown in FIG. 10, the first virtual opening group 11Bb and the second virtual opening group 12Bb are separated by a predetermined distance L. If the first virtual aperture group 11Bb is a virtual lens array as shown in FIG. 10, the predetermined distance L can be the focal length of the element lens. Note that the first and second virtual aperture groups 11Bb and 12Bb do not actually exist and are assumed for the stereoscopic image depth control means 4B to perform arithmetic processing.

  In FIG. 10, the light waves of the first element image group are emitted from the first element image surface 11Ba, and the planar first and second virtual opening groups 11Bb and 12Bb whose thickness direction is shown in FIG. Passes from the left to the right and enters the second element image plane 12Ba. K1 represents the area of the mth element image in the first element image group, and d1 represents the distance from the first element image group (first element image surface 11Ba) to the first virtual opening group 11Bb. Show. Similarly, k2 indicates the area of the nth element image in the second element image group, and d2 is the distance from the second virtual opening group 12Bb to the second element image group (second element image plane 12Ba). Indicates.

  In FIG. 10, p1 indicates the pitch of the element lenses that form the first virtual aperture group 11Bb, and p2 indicates the pitch of the element lenses that form the second virtual aperture group 12Bb. In the present embodiment, the pitch ratio Φ represented by the following formula (9) is set as Φ ≠ 1. That is, the pitches of the element lenses (openings) of the first and second virtual aperture groups 11Bb and 12Bb are different. In FIG. 10, the state of Φ <1 is shown, but Φ> 1 may be set.

[Detailed Configuration of Stereoscopic Image Depth Control Unit]
The distribution unit 41B divides the input video signal (light wave of the first element image group) into element images. The input video signal (the light wave emitted from the first element image surface 11Ba) corresponds to a bundle of light waves of each element image constituting the first element image group, and is denoted as a first element image group t1. I will do it. The distribution means 41B preliminarily applies the light wave (gs, m (xs, m, ys, m)) of the m-th element image of the first element image group t1 in the input video signal to the m-th element image. It outputs to the 1st element image conversion means 42B matched. Here, xs, m indicates the x coordinate when the center of the mth element image in the entire input image (first element image group) is the origin. Similarly, ys, m represents the y coordinate when the center of the mth element image in the entire input image (first element image group) is the origin.

As shown in FIG. 9, the first element image conversion unit 42B includes a light wave calculation unit 42Ba, a phase shift unit 42Bb, and a light wave calculation unit 42Bc.
The light wave calculating means 42Ba calculates light waves incident on the element lenses constituting the first virtual aperture group 11Bb (see FIG. 10) from the light waves of the element image based on Fresnel approximation. That is, the light wave calculating means 42Ba uses the general Fresnel approximation as a signal corresponding to the light wave reaching the mth element lens of the first virtual aperture group 11Bb, and uses the following equation (10) to calculate the element The light wave (Ri, m (x0, m, yo, m)) for each image is calculated.

  Here, x o, m is the x coordinate when the optical axis center of the m-th element lens of the first virtual aperture group 11Bb is the origin. Similarly, yo, m is the y coordinate when the optical axis center of the mth aperture of the element lens group of the first virtual aperture group 11Bb is the origin. F1 indicates the focal length of the element lens of the first virtual aperture group 11Bb. K is the wave number 2π / λ (λ is the wavelength of the light wave). The light wave (Ri, m (x0, m, y0, m)) for each element image is output to the phase shift means 42Bb.

  The phase shift means 42Bb converts the phase from the light wave (Ri, m (x0, m, y0, m)) of the input element image by a phase corresponding to the element lens of the first virtual aperture group 11Bb. The shifted light wave is calculated. That is, the phase shift means 42Bb corresponds to the element lens of the first virtual aperture group 11Bb by applying the light wave (Ri, m (x0, m, y0, m)) as shown in the following equation (11). By shifting the phase by the phase, the signal corresponding to the light wave emitted from the element lens of the first virtual aperture group 11Bb (Ro, m (xo, m, yo, m)) is calculated. This light wave (Ro, m (xo, m, yo, m)) is output to the light wave calculation means 42Bc.

  The light wave calculation means 42Bc is a light wave emitted from the element lens of the first virtual aperture group 11Bb, that is, a light wave (Ro, m (xo, m, yo, m) of the element image inputted from the phase shift means 42Bb. ) Is Fresnel approximated to calculate the light wave incident on the element lens of the second virtual aperture group 12Bb (see FIG. 10). That is, the light wave calculation means 42Bc uses a general Fresnel approximation as a signal corresponding to the light wave reaching the element lens of the second virtual aperture group 12Bb, and uses the following formula (12) for each element image. The light wave (Rd, m (xd, m, yd, m)) is calculated. The light wave (Rd, m (xd, m, yd, m)) for each element image is output to the adding means 43B.

  In Expression (12), xd, m is the incident surface of the second virtual aperture group 12Bb when the optical axis center of the mth element lens of the first virtual aperture group 11Bb is the origin. x coordinate. Similarly, yd, m is the y coordinate on the incident surface of the second virtual aperture group 12Bb when the optical axis center of the m-th element lens of the first virtual aperture group 11Bb is the origin. Show. L is the distance between the first virtual opening group 11Bb and the second virtual opening group 12Bb. Further, the range in which the integral calculation is performed is set to be equivalent to the range in which the m-th element image (element image (m)) expands. An example of the range w in which the mth element image in the first element image group extends is shown in FIG. In this case, a range w in which the m-th element image in the first element image group extends is expressed by Expression (13). Here, k1 represents the area of the mth element image in the first element image group, and d1 represents the distance from the processing target element image group (first element image surface 11Ba) to the first virtual opening group 11Bb. Show.

  The adding means 43B adds the light waves of the respective element images converted by the first element image converting means 42B by the number distributed by the distributing means 41B on the incident surface of the second virtual aperture group 12Bb. It is. The adding means 43B adds light waves corresponding to the field angles of the element lenses of the first virtual aperture group 11Bb. The adding means 43B emits a light wave (Rd, m (xd, m,) which is emitted from the element lens of the first virtual aperture group 11Bb and reaches the entrance surface of the element lens of the second virtual aperture group 12Bb. yd, m)) is added within the region of the second virtual aperture group 12Bb. That is, as a signal corresponding to the light wave on the incident surface of the second virtual aperture group 12Bb, the light wave (Rp (xp) on the incident surface of the second virtual aperture group 12Bb is obtained by the following equation (14). , Yp)).

  Here, xp is the x coordinate when the center of the second virtual aperture group 12Bb is the origin, and similarly, yp is the case where the center of the second virtual aperture group 12Bb is the origin. Y coordinate. This light wave (Rp (xp, yp)) is output to the redistribution means 44B.

  The redistribution unit 44B redistributes the light waves added by the addition unit 43B for each element image constituting the second element image group. The redistribution unit 44B divides the light wave (Rp (xp, yp)) input from the addition unit 43B into element image units. As a light wave for each element image obtained by redistributing the light wave (Rp (xp, yp)), a light wave corresponding to the nth element lens constituting the second virtual aperture group 12Bb is represented by (Rp, n (xp, n, yp, n)). Here, the redistribution means 14 converts the light wave (Rp, n (xp, n, yp, n)) input to each element lens constituting the second virtual aperture group 12Bb into the second virtual Output to the second element image conversion means 45B associated in advance with the n-th element lens that crosses the open aperture group 12Bb.

  The second element image conversion unit 45B converts the light wave of the element image distributed by the redistribution unit 44B into a light wave that has propagated through the second virtual aperture group 12Bb until the image is formed. It is. The second element image conversion means 45B redistributes the light wave input from the redistribution means 44B to the aperture region of the element lens (focal length f2) of the second virtual aperture group 12Bb, and then the element lens. It is converted into a light wave that passes through (focal length f2), and the light wave is further propagated by the focal length f2 of the element lens. The second element image conversion unit 45B functions for every (2N + 1) element images, for example, to propagate a bundle of light waves of each element image. The second element image group is composed of a bundle of light waves of the element images that propagate. This is expressed as a second element image group t2. That is, the second element image conversion unit 45B calculates the second element image group t2. As shown in FIG. 9, the second element image conversion unit 45B includes a phase shift unit 45Ba and a light wave calculation unit 45Bb.

  The phase shift unit 45Ba converts the phase from the light wave (Rp, n (xp, n, yp, n)) input from the redistribution unit 44B to a phase corresponding to the element lens of the second virtual aperture group 12Bb. The light wave shifted by the amount is calculated. In other words, the phase shift means 45Ba corresponds to the element lens of the second virtual aperture group 12Bb by applying the light wave (Rp, n (xp, n, yp, n)) as shown in the following equation (15). By shifting the phase by the phase, the signal (Rr, n (xp, n, yp, n)) corresponding to the light wave emitted from the element lens of the second virtual aperture group 12Bb is calculated.

  Here, xp, n is an x coordinate when the optical axis center of the nth element lens of the second virtual aperture group 12Bb is the origin. Similarly, yp, n is the y coordinate when the center of the optical axis of the nth element lens of the second virtual aperture group 12Bb is the origin. F2 is a focal length of the element lens of the second virtual aperture group 12Bb. The light wave (Rr, n (xp, n, yp, n)) for each element image is output to the light wave calculating means 45Bb.

  The light wave calculating means 45Bb emits the light wave for each element lens of the second virtual aperture group 12Bb from the element lens of the second virtual aperture group 12Bb by performing Fresnel approximation, and the second element image plane The light wave reaching 12Ba (see FIG. 10) is calculated. That is, the light wave calculation means 45Bb calculates the light wave (Re, n) emitted from the nth element lens of the second virtual aperture group 12Bb and reaching the second element image plane 12Ba by the following equation (16). (Xe, n, ye, n)) is calculated.

  Here, xe, n is an x coordinate from the center of the nth element image in the entire output image (second element image group). Similarly, ye, n is the y coordinate from the center of the nth element image in the entire output image. The light wave (Re, n (xe, n, ye, n)) calculated by the light wave calculating means 45Bb is output to the coupling means 46B.

  The combining means 46B generates the second element image group by combining the light waves converted by the second element image converting means 45B by the number redistributed by the redistributing means 44B. This combining means 46B calculates the sum of the powers of the light waves from the light waves for each element image output from the second element image conversion means 45B, thereby obtaining a video signal in which the size of the stereoscopic image is converted, that is, Then, a video signal to be the second element image group t2 is generated. The second element image group t2 obtained by the combining means 46B is output to a stereoscopic image display device (not shown).

  Here, the power of the light wave (Re, n (xe, n, ye, n)) reaching the second element image plane 12Ba output from the second element image conversion means 45B is the square of the amplitude of the light. Can be represented. Moreover, since the light wave emitted as each element image of the first element image group t1 is incoherent (the wavelength and phase are not constant), the phase of the light wave can be regarded as uncorrelated. Therefore, as shown in the following equation (17), the combining unit 46B uses the light wave of each element image that reaches the second element image plane 12Ba (for example, if it is the n-th element image, the light wave is (Re, n (Xe, n, ye, n)) is calculated, and the sum thereof is calculated to obtain a video signal of the entire second element image group t2.

At this time, the stereoscopic image depth control unit 4B acquires information on the pixels to which pixel values have already been assigned in the integrated three-dimensional information storage unit 5 from the pixel value assigned position storage unit 6, and performs the current processing. When there is no pixel corresponding to the position to which the pixel value is already assigned in the integrated three-dimensional information storage unit 5 in the processing target element image group after the depth ratio and depth control, the first element image group is combined by the combining unit 46B. For all the pixels of the processing target element image group 31 ₃ shown in FIG. As a result, the second element image group (processed element image group 32 ₃ shown in FIG. 7) is generated, so that the stereoscopic image depth control means 4B outputs to and stores the integrated three-dimensional information storage means 5.

On the other hand, when there is a pixel corresponding to a position to which a pixel value has already been assigned in the integrated three-dimensional information storage unit 5 in the processing target element image group currently being processed, the combination unit 4 6 B The sum of the power of the light wave excluding the pixels is obtained.

Stereoscopic image depth control means 4B is a pixel value assigned position storage unit 6, the integrated three-dimensional information storage unit 5 position where a pixel value is assigned to the (e.g., the processed element image group 32 ₃ assigned position) Output and store information. Since the operation of the other configuration is the same as that of the above-described three-dimensional information integration apparatus 1, description thereof is omitted.

[Reproduction image size ratio control]
The stereoscopic image depth control means 4B has the configuration described in detail as described above, and by setting in advance the pitch ratio Φ represented by the above equation (9), the size of the reproduced image can be adjusted. The ratio can be controlled to a desired value. If the processing of the stereoscopic image depth control unit 4B is not performed, that is, if the first and second virtual aperture groups 11Bb and 12Bb are not used, the subject size xc (see FIG. 16A). ) And the size of the stereoscopic image to be reproduced xr (see FIG. 16B) is as in the above-described equation (101). When the stereoscopic image depth control means 4B is not used in this way, if α = γ in the above equation (101), the subject size xc (see FIG. 16A) and the size of the reproduced stereoscopic image The relationship with the length xr (see FIG. 16B) is expressed by the following equation (18).

  That is, in FIG. 9, when the stereoscopic image depth control means 4B is ignored, the element lens pitch pc (see FIG. 16A) is equal to the element lens pitch pr (see FIG. 16B). Is equal to the size xc of the subject (see FIG. 16A) and the size xr of the stereoscopic image to be reproduced (see FIG. 16B). On the other hand, when the stereoscopic image depth control means 4B is used, the relationship between the subject size xc (see FIG. 16A) and the reproduced stereoscopic image size xr (see FIG. 16B). Is represented by the following equation (19).

  That is, in FIG. 9, even if the element lens pitch pc (see FIG. 16A) and the element lens pitch pr (see FIG. 16B) are equal, the first virtual aperture group By controlling the pitch p1 (see FIG. 10) of the element lens of 11Bb and the pitch p2 (see FIG. 10) of the element lens of the second virtual aperture group 12Bb, the size xc of the subject (see FIG. 16). It is possible to change the size xr (see FIG. 16B) of the reproduced stereoscopic image with respect to (a).

  For convenience of explanation, the case where α and γ are equal in the above-described equation (101) has been described in the above-described equations (18) and (19). However, in the present invention, the above-described equation (101) is described. ) May be α ≠ γ. Similarly, the case where the pitch pc of the element lens is equal to the pitch pr of the element lens is described, but pc ≠ pr may be used.

  According to such a stereoscopic image depth control means 4B, the first element image group t1 is processed by assuming that the light wave of each element image m has passed through the first and second virtual aperture groups 11Bb and 12Bb. Thus, the second element image that reproduces an image in which the size of the image is changed to a desired size is the same as the unevenness of the subject without depending on the optical device used for photographing and displaying the subject. It can be converted into the group t2. Therefore, since the size of the stereoscopic image can be changed by the arithmetic processing in this way, it is not necessary to change the optical device used for photographing and displaying such as a lens as in the conventional case. In addition, according to the stereoscopic image depth control unit 4B, an image equivalent to a case where light emitted from an image as an element image group obtained by imaging a subject as a plurality of element images can be generated by arithmetic processing. . Furthermore, when displaying a reproduced image whose size with respect to the subject is arbitrarily controlled, it is possible to enhance the effect of rendering a stereoscopic image.

  The stereoscopic image depth control means 4B can be realized by an arithmetic circuit, and is generally provided with a central processing unit (CPU), a main storage device (RAM: Random Access Memory), and the like. It is also possible to realize a simple computer as a program that functions as each of the above-described means.

  Each of the stereoscopic image depth control means 4 and 4B controls the depth of a stereoscopic image using a wave optical technique, but is not limited to this, and a geometrical optical technique may be used. good. Hereinafter, the stereoscopic image depth control unit 4C that controls the depth of the stereoscopic image by a geometric optical method will be described.

FIG. 12 shows a block configuration diagram showing another configuration example of the stereoscopic image depth control means applied to the three-dimensional information integration device of the present invention, and a schematic diagram schematically showing a configuration example of the element image group imaging device. FIG.
The stereoscopic image depth control unit 4C is configured to store information on the preset distance g1 between the element lens group 51 of the element image group imaging device 50 and the imaging surface of the imaging unit 52, and the pitch p1 of the adjacent two element lenses L1 and L1. The processed element image group (hereinafter, also referred to as “first element image group”) of the subject can be displayed at a position of an arbitrary depth as a processed three-dimensional image having a correct depth positional relationship ( Hereinafter, it is also referred to as “second element image group”). In addition,
Although not shown, an element image separation unit 2 (see FIG. 2) and a pixel storage unit 3 (see FIG. 2) are arranged between the element image group imaging device 50 and the stereoscopic image depth control unit 4C.
The stereoscopic image depth control means 4C starts from the pixel storage means 3 in order from the processing target element image group that is specified by the operation signal input from the outside and that is associated with the subject that is arranged in the forefront in the integrated 3D information. Read and process. Here, the stereoscopic image depth control unit 4 </ b> C includes a second element image generation unit 47.

The second element image generation unit 47 converts the first element image group (for example, the processing target element image group 31 ₃ shown in FIG. 6) captured by the element image group imaging device 50 to a position of an arbitrary depth. This is converted to the second element image group (processing object element image group 32 ₃ shown in FIG. 7) that can display a stereoscopic image with a correct positional relationship. The second element image generation unit 47 includes a 3D information allocation unit 47a and a 3D information interpolation unit 47b.

Here, the information of the luminance value of each pixel of the first element image group 31 ₃ taken by the element image group imaging device 50, and it is stored in the pixel memory means 3 (see FIG. 2), the The two-element image generation unit 47 reads out information on the luminance value of each pixel from the pixel storage unit 3 (see FIG. 2).

Referring now to FIG. 13 (appropriately with reference to Figure 12), the second element image generating means 47 will be described how to convert the first element image group 31 _3. FIG. 13 is an explanatory diagram for explaining a conversion method of the first element image group of the depth conversion means.

As shown in FIG. 13, the second element image generating means 47, the first and the element image plane 11Ca, element lens group 5 1 and element lens L1 for displaying a first component image group 31 ₃ virtually pitch p1 A first virtual aperture group 11Cb in which first virtual apertures L1 ′, which are the same virtual plurality of virtual element lenses, are arranged on the same plane, and the first virtual aperture group 11Cb A second virtual aperture L2 ′, which is a virtual element lens that forms images I, I,... Of a stereoscopic image (not shown) virtually reproduced by the first image, is arranged on the same plane. and 2 of the virtual aperture group 12Cb, (by calculation) virtually the second elemental image surface 12Ca of the second element image group 32 ₃ made of the image I is imaged virtually set. The second element image generating means 47, based on the first luminance value of each pixel of the element images 31 ₃ taken by the element image group imaging device 50 (pixel value), the first elemental image surface 11Ca forming light from the first elemental image group 31 ₃ which is virtually displayed and via the first virtual aperture group 11Cb and a second virtual aperture group 12Cb second elemental image surface 12Ca to by calculating the brightness values of the pixels on the second element image group 32 ₃ generated in the image, it was decided to produce a second component image group 32 _3.

  Here, the distance g1 between the first element image plane 11Ca and the first virtual aperture group 11Cb is the imaging plane (not shown) of the element lens group 51 and the imaging means 52 of the element image group imaging apparatus 50. Is equal to the distance g1. The first virtual opening L1 'and the second virtual opening L2' are parallel to the optical axis. On the other hand, the first virtual opening L1 ′ and the second virtual opening L2 ′ may have the same or different pitches p1 and p2, and the first element image plane 11Ca and the first virtual opening L2 ′ may be different from each other. The distance g1 between the virtual aperture group 11Cb and the distance g2 between the second virtual aperture group 12Cb and the second element image plane 12Ca may be equal or different.

FIG. 13 shows an example in which the second virtual opening L2 ′ is arranged at the position of the subject S in FIG. opening group 12Cb and the second element image plane 12Ca, a first virtual virtual light of the light from the first element image group 31 ₃ of the first element image surface 11Ca emitted from aperture group 11Cb It can be set at an arbitrary position on the road. Then, by the second element image generation means 47, the second virtual aperture group 12Cb and the second element image set on the virtual optical path of the light emitted from the first virtual aperture group 11Cb. Depending on the distance from the surface 12Ca to the position corresponding to the position of the subject S, a stereoscopic image (not shown) displayed by the stereoscopic image display device (not shown) from the stereoscopic image display device (not shown). The distance in the depth direction is determined. As described above, the second element image generation unit 47 sets the second virtual aperture group 12Cb and the second element image surface 12Ca at positions corresponding to positions in the depth direction for displaying a stereoscopic image. in, it is possible to generate the second element image group 32 ₃ capable of displaying a stereoscopic image on the depth direction position.

Furthermore, the second element image generating means 47, the light from the first elemental image plane first element image group 31 ₃ 11Ca emitted from the first virtual aperture group 11cb, a second virtual In order to generate the second element image group 32 ₃ that is virtually imaged on the second element image surface 12Ca by the aperture group 12Cb, the second element image group 32 ₃ is the element image group imaging device 50 of FIG. The image is taken from a direction opposite to the image taking direction. As a result, the second element image generation means 47 can display a stereoscopic image (not shown) having a correct positional relationship in depth compared to the subject S by a stereoscopic image display device (not shown).

Returning to FIG. 12, the description will be continued. Three-dimensional information assigning means (assignment means) 47a has a luminance value of each pixel of the first element image group 31 ₃ taken by the element image group imaging device 50, the light from the pixel, a first virtual it is an intensity value of the pixel on the second element image group 32 ₃ which forms the second element image plane 12Ca through the opening group 11Cb and a second virtual aperture group 12Cb.

Referring now to FIG. 14, the three-dimensional information allocation means 47a methods described for assigning luminance values of the pixels of each of the first element image group 31 ₃ to the pixels on the second element image group 323. FIG. 14 is an explanatory diagram for explaining a method in which the three-dimensional information assigning unit assigns the luminance value of each pixel of the first element image group to the pixel on the second element image group.

As shown in FIG. 14, the three-dimensional information allocating unit 47a includes a first element image plane 11Ca, a first virtual opening group 11Cb, a second virtual opening group 12Cb, and a second element image. set the surface 12Ca virtually, and the first element image group 31 ₃ contains the image _Ia 1, Ia ₂ subjects Sa. The virtual light from the pixel image Ia ₁ of the first element image group 31 ₃ passes through the center of the first virtual aperture L1 '(hereinafter, referred to as virtual ray) is a second virtual aperture when passing through the center of L2 'is three-dimensional information assigning means 47a is in the virtual ray and the intersection second element pixels of the image group 32 ₃ which corresponds to the second element image plane 12Ca, the first element image group Assign a luminance value of ₃ 13 pixels.

On the other hand, if the pixel image Ia ₂ of the first element image group 31 ₃ first virtual aperture L1 'virtual ray passing through the center of, the second virtual aperture L2' does not pass through the center of the The three-dimensional information assigning means 47a is a second element image group 32 corresponding to the intersection of a line passing through the center of the second virtual opening L2 ′ and the second element image plane 12Ca parallel to the virtual ray. the _third pixel, assigning a brightness value of the pixels of the first element image group 31 _3.

Returning to FIG. 12, the description will be continued. Stereo information interpolating unit 47b is calculated brightness value of the pixels of the stereoscopic information allocation means 47a second element image group 32 ₃ unassigned luminance values by interpolating, based on the luminance value of the pixels near the pixel To do.

Referring now to FIG. 15, based on the luminance value of the stereoscopic information interpolating unit 47b is in the vicinity of the pixel, a method for calculating by interpolation luminance values of the pixels of the second element image group 32 _3. FIG. 15 is an explanatory diagram for explaining a method by which the three-dimensional information allocating unit allocates the luminance value of each pixel of the first element image group to the pixel on the second element image group. Virtually set to the first element image plane, the first virtual aperture group, the second virtual aperture group, the second element image plane, and the second element image group. The schematic diagram schematically showing the included image, (b) is an enlarged view of the portion indicated by A in (a), the image virtually included in the second element image group, and the image of the image It is a schematic diagram which shows typically the pixel calculated | required by interpolating from the information of a pixel.

As shown in FIG. 15 (a), the first element image group 31 _3, and contains the image Ia, Ib of different subjects (not shown). Then, the three-dimensional information assigning means 47a, corresponding to the pixels of the second pixel element image group 32 _3, the image Ib of the first element image group 31 ₃ corresponding to the pixels of the image Ia of the first element image group 31 ₃ to a pixel of the second element image group 32 _3, and that each luminance value of the pixels of the corresponding first element image group 31 ₃ is assigned. Further, as shown in FIG. 15B, a luminance value is assigned to the pixel Pc between the pixels Pa and Pb of the images Ia and Ib (see FIG. 15A) by the three-dimensional information assigning unit 47a. Suppose there is nothing.

Then, the three-dimensional information interpolating unit 47b, based on the luminance values of the pixels Pa, Pb of the upper and lower pixel Pc, and the luminance value _{m c} of the pixel Pc, calculated by the following equation (20). _Here, m a, _{m b} is the pixel Pa, a luminance value of Pb, a, b is the distance to the pixel Pa, Pb of the pixel Pc.

  Here, although the case where the three-dimensional information interpolation unit 47b interpolates using the one-dimensional interpolation equation shown in Equation (20) based on the luminance values of the pixels Pa and Pb above and below the pixel Pc has been described, Interpolation may be performed based on the luminance value of a pixel (not shown) that is vertically or horizontally or diagonally with respect to the pixel Pc to be interpolated. At this time, the three-dimensional information interpolating means 47b is, for example, Seiko Inoue et al., “Practical image processing learned in C language”, Ohmsha, 1999, p. 149 can be interpolated using a two-dimensionally expanded interpolation formula.

  Further, the three-dimensional information interpolation means 47b is described in Kenichi Kido, “Introduction to Digital Signal Processing”, Maruzen, 1985, p. A sampling function that generates a continuous data function from a discrete data string as described in 31-35, and information on discrete luminance values for each pixel is used as continuous information. It is also possible to obtain the luminance value of the pixel Pc.

Such second element image group 32 ₃ generated by the, by the stereoscopic image depth control means 4C, is output to the integrated three-dimensional information storage unit 5. In addition, the stereoscopic image depth control unit 4C outputs the position of the pixel to which the pixel value is assigned in the integrated three-dimensional information storage unit 5 to the pixel value assigned position storage unit 6. In this way, the same processing is repeated until the processing for all the processing target element image groups is completed. Note that the 3D image depth control unit 4C refers to the pixel value assigned position storage unit 6 for the second and subsequent processing target element image groups, and sets the processing target element image group currently being processed. When there is a pixel corresponding to a position to which a pixel value has already been assigned in the integrated 3D information storage means 5, the 3D information assigning means 47a and the 3D information interpolation means 47b perform the calculation excluding the pixel.

  Then, after finishing the processing for all the processing target element image groups, the stereoscopic image depth control means 4C outputs a control signal notifying the end of the processing to the pixel value assignment means 7. If the pixel value assigning means 7 is not provided, the signal is not output.

  According to the stereoscopic image depth control means 4C as described above, according to the position of the second virtual opening group 12Cb with respect to the first element image group and the first virtual opening group 11Cb, Since the distance in the depth direction of the stereoscopic image is determined, a second element image group that can display the stereoscopic image at a desired position in the depth direction is generated by adjusting the position of the second virtual aperture group 12Cb. can do.

  The stereoscopic image depth control means 4C can be realized by an arithmetic circuit, and is generally provided with a central processing unit (CPU), a main storage device (RAM), and the like. It is also possible to realize a simple computer as a program that functions as each of the above-described means.

  As described above, according to the three-dimensional information integration device 1, one three-dimensional information (integrated three-dimensional information) obtained by integrating a plurality of three-dimensional information can be generated. Further, it is possible to integrate a plurality of pieces of three-dimensional information in a state of being controlled to an arbitrary ratio and depth of the operator.

  Furthermore, according to the three-dimensional information integration device 1, a plurality of pixel values are not assigned to the pixels of the integrated three-dimensional information storage means 5, that is, a subject that is disposed in the forefront in a portion where a plurality of subjects overlap each other. Therefore, when the integrated 3D information is displayed as a stereoscopic image, a so-called phantom phenomenon can be prevented from occurring in a portion where a plurality of subjects overlap each other.

  In addition, according to the three-dimensional information integration device 1, the processed element image group except for the pixels assigned to the position to which the pixel value has already been assigned in the integrated three-dimensional information storage unit 5 in the processing target element image group. Therefore, the calculation amount can be reduced and the processing speed can be improved.

  In the above-described embodiment, the method of integrating subjects in different three-dimensional spaces has been described. However, when subjects in the same three-dimensional space are integrated, size ratio and depth control are not performed. May be.

  The three-dimensional information integration device 1 described above can realize each of the above-described units by an arithmetic circuit, a central processing unit (CPU), a main storage device (RAM), and the like. It is also possible to realize a general computer provided with the above by operating it with a program (three-dimensional information integration program) that functions as each means described above. As described above, the present invention has an effect that the apparatus can be miniaturized by performing the control processing of the ratio of the size of the reproduced image and the depth not by the optical system but by calculation.

It is a figure for demonstrating the outline of this invention, and is explanatory drawing for demonstrating a mode that several 3D information (element image group) is integrated and one 3D information (element image group) is produced | generated. . It is a block diagram which shows the structure of the three-dimensional information integration apparatus which concerns on this embodiment. It is a block block diagram which shows the structure of the stereo image depth control means in the three-dimensional information integration apparatus which concerns on this embodiment. It is a schematic diagram for demonstrating the method of controlling the depth of a stereo image by a calculation by the stereo image depth control means of the three-dimensional information integration apparatus which concerns on this embodiment. It is a conceptual diagram for demonstrating operation | movement of the element image separation means of the three-dimensional information integration apparatus which concerns on this embodiment. It is a conceptual diagram for demonstrating a mode that the process order of the process target element image group memorize | stored in the pixel memory | storage means is determined. (A) to (e) are states in which pixel values are assigned to the integrated 3D information storage unit by the stereoscopic image depth control unit of the 3D information integration apparatus according to the present embodiment, and (f) to (h) are: It is a conceptual diagram for demonstrating a mode that the information of the position where the pixel value was allocated to the integrated three-dimensional information storage means is assigned to the pixel value assigned position storage means. It is a flowchart which shows operation | movement of the three-dimensional information integration apparatus which concerns on this embodiment. It is a block block diagram which shows the other structural example of the stereo image depth control means applied to the three-dimensional information integration apparatus of this invention. It is explanatory drawing which shows typically an example of the virtual opening group assumed by the arithmetic processing of a stereo image depth control means. It is explanatory drawing which shows typically the example of the integration range used by the arithmetic processing of a stereo image depth control means. It is a block block diagram which shows the other structural example of the stereo image depth control means applied to the three-dimensional information integration apparatus of this invention. It is explanatory drawing for demonstrating the conversion method of the 1st element image group of a depth conversion means. It is explanatory drawing for demonstrating the method in which a three-dimensional information allocation means allocates the luminance value of each pixel of a 1st element image group to the pixel on a 2nd element image group. It is explanatory drawing for demonstrating the method for a three-dimensional information allocation means to allocate the luminance value of each pixel of a 1st element image group to the pixel on a 2nd element image group, (a) is set virtually. A first element image plane, a first virtual aperture group, a second virtual aperture group, a second element image plane, and an image virtually included in the processed element image group (B) is an enlarged view of the portion indicated by A in (a), and an image virtually included in the second element image group, and pixel information of the image. It is a schematic diagram which shows typically the pixel calculated | required by interpolation. It is explanatory drawing which shows typically imaging | photography and reproduction | regeneration of the stereo image by the conventional IP system, (a) has shown the stereo image imaging device, (b) has each shown the stereo image display apparatus. It is explanatory drawing which shows typically the method of avoiding the conventional reverse vision, (a) is the stereo image depth control means which exchanges the information acquired by imaging | photography, (b) displays the converted image. A stereoscopic image display device is shown.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 3D information integration apparatus 2 Element image separation means 3 Pixel storage means 4 Stereo image depth control means 41 Distribution means 42 Element image conversion means 43 Depth conversion means 44 Element image reproduction means 45 Addition means 5 Integrated 3D information storage means 6 Pixel Value assigned position storage means 7 Unprocessed pixel assignment means 8 Output means

Claims (4)

Integral photography system acquires multiple 3D information that is an element image group consisting of multiple element images, performs size ratio and depth control for each of the acquired 3D information, and integrates them. A three-dimensional information integration device for generating one piece of three-dimensional information,
Based on an operation signal input from the outside, for each of a plurality of element image groups, a pixel corresponding to a subject area existing in a desired depth range and a pixel corresponding to another area are separated from each element image. Element image separation means;
Pixel storage means for storing each processing target element image group composed of pixels corresponding to a subject area existing in the desired depth range separated by the element image separation means;
By a value indicating the depth position in the integrated three-dimensional information specified for each of the subjects associated with each of the plurality of processing target element image groups stored in the pixel storage unit by the operation signal input from the outside Read from the pixel storage unit in order from the processing target element image group associated with the subject that is specified in the three-dimensional information after integration , and the size of the subject for the read processing target element image group. A processed element image group is generated by performing processing for controlling the ratio of the size of the reproduced image to the height to an arbitrary value set in advance and processing for controlling the depth according to the value indicating the depth position. Stereoscopic image depth control means;
An integrated three-dimensional information storage means for storing the processed element images generated by the stereoscopic image depth control means,
Pixel value assigned position storage means for storing the position where the processed element image group is assigned to the integrated three-dimensional information storage means;
Output means for outputting the integrated three-dimensional information stored in the integrated three-dimensional information storage means to the outside,
The stereoscopic image depth control means includes:
For the processing target element image group to be integrated after the second, refer to the pixel value assigned position storage means, and the integrated three-dimensional information storage among the areas to which the processing target element image group should be assigned. A three-dimensional information integration apparatus, wherein only a pixel to which no pixel value has been assigned yet is assigned to a pixel of the integrated three-dimensional information storage means, with a pixel value whose size ratio and depth are controlled. The stereoscopic image depth control means includes:
Distributing means for distributing the light wave of the processing target element image group for each element image;
The light wave of the element image distributed by this distribution means passes through the first virtual aperture group and propagates to the second virtual aperture group having a pitch different from that of the first virtual aperture group. First element image conversion means for converting into a light wave to
Adding means for adding the light waves of the respective element images converted by the first element image conversion means by the distributed number at the entrance surface of the second virtual aperture group;
Redistribution means for redistributing the light waves added by the addition means for each element image;
A second element image conversion means for converting the light wave of the element image distributed by the redistribution means into a light wave propagated by a distance until it passes through the second virtual aperture group and forms an image;
A combining unit that generates the processed element image group by combining the light waves converted by the second element image converting unit by the redistributed number;
The ratio Φ of the pitch between the first virtual aperture group and the second virtual aperture group, which is set to an arbitrary value in advance, is a process for controlling the ratio of the sizes of the processing target element image group. The three-dimensional information integration apparatus according to claim 1, wherein the three-dimensional information integration apparatus is performed according to After completion of the ratio of the sizes of all the processing target element image groups in the stereoscopic image depth control means and the depth control processing, pixels are still stored in the integrated 3D information storage means based on the operation signal input from the outside. The three-dimensional information integration device according to claim 1 or 2 , further comprising pixel value assignment means for assigning a pixel value to a pixel to which no value is assigned. Integral photography system acquires multiple 3D information that is an element image group consisting of multiple element images, performs size ratio and depth control for each of the acquired 3D information, and integrates them. Computer to generate a single 3D information
Based on an operation signal input from the outside, for each of a plurality of element image groups, a pixel corresponding to a subject area existing in a desired depth range and a pixel corresponding to another area are separated from each element image. Element image separation means for storing a processing target element image group composed of pixels corresponding to a subject area existing in the desired depth range in a pixel storage means,
Indicates an arbitrary depth position in the integrated three-dimensional information specified for each of the subjects associated with each of the plurality of processing target element image groups stored in the pixel storage unit by an operation signal input from the outside The processing target element image group associated with the subject arranged in the forefront in the integrated three-dimensional information identified by the value is read from the pixel storage unit in order, and the read processing target element image group is subject to the subject. A processed element image generated by performing processing for controlling the ratio of the size of the reproduced image to the size of the image to an arbitrary value set in advance, and processing for controlling the depth according to the value indicating the depth position And storing the processed element image group in the integrated three-dimensional information storage means. Position stereoscopic image depth control means for storing the pixel values assigned position storing means,
Output means for outputting the integrated three-dimensional information stored in the integrated three-dimensional information storage means to the outside;
A three-dimensional information integration program characterized by functioning as

Images