JP5852465B2 - Spatial information interpolation apparatus and program thereof - Google Patents

Description

  The present invention relates to a technique for acquiring information on a space where a subject exists through a lens array or a spatial filter for a three-dimensional image such as an IP, and in particular, interpolation of the missing spatial information when the spatial information is missing. The present invention relates to a spatial information interpolation apparatus and a program thereof.

  Conventionally, among techniques for acquiring information on a space where a subject exists through a lens array or a spatial filter, an imaging optical element array in which a plurality of minute optical elements or a plurality of element pinholes are arranged in a two-dimensional manner is used. An integral photography (IP) system is known as an apparatus for capturing and displaying a stereoscopic image.

  A normal spatial information acquisition apparatus based on the IP system will be described with reference to FIGS. 16 and 17. 16 and 17 are conceptual diagrams of the device when viewed downward in the vertical direction. First, as shown in FIG. 16, the spatial information acquisition device 110 includes a lens group 112 in which convex lenses are arranged in a single plane and an imaging plate 113. The lenses constituting the lens group 112 are also arranged in the direction perpendicular to the paper surface (the height direction of the lens group 112), but in FIG. 16, the lens group 112 is shown in a horizontal row (one-dimensional). Yes. The number of lenses is an example. Further, the imaging plate 113 is plate-shaped and has a spread in a direction perpendicular to the paper surface, and has a thickness along the paper surface, but is shown by a straight line parallel to the row of lens groups 112 in FIG.

  In this spatial information acquisition device 110, the subject 111 is photographed through the lens group 112 from the photographing direction 114 indicated by the arrow. On the imaging plate 113, the same number of images of the subject 111, for example, the image 115, are taken as many as the convex lenses constituting the lens group 112. An image acquired by each convex lens is referred to as an element image (imaging element image), and a collection of images acquired by a plurality of convex lenses is referred to as an element image group (imaging element image group). In FIG. 16, an image of the subject is schematically shown at a position where the subject forms an image on the imaging plate 113 by the lens group 112. Pixel information such as luminance is recorded in pixels (element pixels) constituting an image of a subject formed on the imaging plate 113 in accordance with the position of the pixel. Note that when the video signals of the photographing element images acquired by the individual convex lenses are integrated and presented by the IP method, a stereoscopic image corresponding to the subject 111 can be reconstructed.

  An example of the spatial information display device 120 that displays the spatial information acquired by the spatial information acquisition device 110 shown in FIG. 16 is shown in FIG. Here, the spatial information indicates pixel information such as luminance of the image when the space where the subject exists is displayed as an image. As shown in FIG. 17, the spatial information display device 120 includes a lens group 122 in which convex lenses are arranged in a single plane and a display element 123. In FIG. 17, the lens group 122 is shown in a horizontal row (one-dimensional). Further, the display element 123 has a display surface extending in a direction perpendicular to the paper surface and has a thickness along the paper surface. However, in FIG. 17, the display element 123 is indicated by a straight line parallel to the row of lens groups 122.

  The display element 123 displays an image 125 corresponding to the image 115 photographed by the imaging plate 113 of the spatial information acquisition device 110 in FIG. As a result, as shown in FIG. 17, the observer H can observe a three-dimensional image generated at the same position as the place where the subject exists. However, as shown in FIG. 16, when viewed from the shooting direction 114 indicated by the arrow, the cylinder of the subject 111 exists in front of the prism, but as shown in FIG. 17, a stereoscopic image corresponding to the subject 111 is obtained. In 121, a prism is generated in front of the cylinder as viewed from the observation direction 124 indicated by the arrow. Therefore, in this case, a reverse view image in which the depth is inverted compared to the subject is generated. In a reverse-view image, a portion that is actually convex in the subject appears to be concave. Therefore, for example, the depth of the image acquired by the spatial information acquisition device 110 is inverted in advance so that the unevenness of the presented stereoscopic image becomes normal, and then displayed on the display element 123 of the spatial information display device 120. It is also generally done.

  In FIGS. 16 and 17, the operation of acquiring and displaying information on the space in which the subject exists using a small optical element array (lens group) as the optical element array has been described. However, a small pinhole is used as the optical element array. An array may be used, or a spatial filter having a pinhole array may be used.

In a normal spatial information acquisition device 110 based on the IP method, the correspondence between the size of a subject and the size of a stereoscopic image to be reproduced is determined depending on the device, for example. Here, as shown in FIG. 16, the size of the prism 111 of the subject 111 is x _c , the distance from the lens group 112 to the prism of the subject 111 is z _c , and the distance from the lens group 112 to the surface on which the image 115 is taken. D _c , the pitch of the convex lens constituting the lens group 112 is p _c , and the size of the subject image generated by the convex lens constituting the lens group 112 is k _c .

In FIG. 16, x _c (+) is the length in the x direction (here, the height direction, that is, the direction perpendicular to the paper surface), and indicates a positive value based on the bottom surface of the prism of the subject. z _c (−) is the length in the z direction (the shooting direction and the observation direction, that is, the horizontal direction in FIG. 16), and indicates a negative value based on the arrangement surface of the lens group 112. d _c (+) is the length in the z direction and indicates a positive value based on the arrangement surface of the lens group 112.

As shown in FIG. 17, the size of the prism of the stereoscopic image 121 is x _r , the distance from the lens group 122 to the prism of the stereoscopic image 121 is z _r , and the distance from the lens group 122 to the surface on which the image 125 is displayed. distance d _r, the convex lens pitch p _r of the lens group 122, the size of each image displayed on the display device 123 and k _r.

In FIG. 17, x _r (+) is a length in the x direction (a direction perpendicular to the paper surface), and indicates a positive value based on the bottom surface of the prism that is the image of the subject. z _r (+) is the length in the z direction (horizontal direction, that is, the horizontal direction in FIG. 17), and indicates a positive value based on the arrangement surface of the lens group 122. d _c (−) is the length in the z direction and represents a negative value based on the arrangement surface of the lens group 122.

  The subscript c of each parameter shown in FIG. 16 indicates that it relates to the side where the spatial information is acquired from the subject, and the subscript r of each parameter shown in FIG. Represents the relationship to the construction side). Hereinafter, in this specification, each parameter is described by omitting the positive and negative signs (+) and (−).

In this case, the relationship between x _c shown in FIG. 16 and x _r shown in FIG. 17 is expressed by Equation (1) by Non-Patent Document 1.

The parameter α in the formula (1), as represented by the following formula (2a), the ratio of the convex lens pitch p _r presented side convex lens pitch of the acquisition side of the spatial information for the p _c (enlargement ratio or reduction ratio ). The parameter γ in the equation (1) represents the ratio of the image size k _r on the presentation side to the image size k _c on the spatial information acquisition side, as represented by the following equation (2c). Here, the parameter β is defined as a ratio including the sign of the distance d _r (−) on the presentation side with respect to the distance d _c (+) on the spatial information acquisition side by the following equation (2b).

Further, the relationship between z _c shown in FIG. 16 and z _r shown in FIG. 17 is expressed by Equation (3) by Non-Patent Document 1. The right side of Expression (3) is represented by parameters (z _c , d _c ) related to the space information acquisition side and parameters α, β, γ indicating the ratio, and the left side of Expression (3) is the space. It is represented only by the parameter z _r on the information presentation side.

J. Opt. Soc. Am. A, 2004, vol.21, p.951-958

  In the integral photography method, in order to acquire spatial information with high spatial resolution, a multi-pixel high-definition image sensor is required as the imaging plate 113 (see FIG. 16) of the spatial information acquisition device 110. Therefore, for example, like the spatial information acquisition device 130 illustrated in FIG. 18, the imaging element array 132 is configured using a plurality of imaging elements 131, and the spatial information can be acquired by the imaging element array 132 and the lens group 112. Conceivable. In addition, although the image pick-up element which comprises the image pick-up element array 132 is also arranged in the direction perpendicular | vertical to a paper surface, in FIG. 18, the image pick-up element array 132 is shown with the image pick-up element 131a, 131b, 131c which made the column of one pixel. ing. In this case, the video signal of the imaging element image group acquired by the imaging element 131a, the video signal of the imaging element image group acquired by the imaging element 131b, and the video signal of the imaging element image group acquired by the imaging element 131c are integrated. When presented in the IP system, a stereoscopic image corresponding to the subject 111 can be reconstructed.

  As described with reference to FIGS. 16 and 17, by using the lens group, the image sensor, and the display element, it is possible to acquire and display information on the space where the subject exists. Then, as described with reference to FIG. 18, it is possible to acquire information on a space in which a subject exists with high spatial resolution by using an imaging element array including a plurality of imaging elements.

  However, as shown in FIG. 18, when the image sensor array 132 is used, for example, a gap 133 is generated between the pixels of the adjacent image sensor 131a and the image sensor 131b. Therefore, when information on a space where a subject exists is acquired using the image sensor array 132 in which such a gap 133 is generated, the acquired space information is lost. That is, if the gap 133 is not generated, it is impossible to acquire spatial information that should have been originally acquired.

  The present invention has been made in view of the above problems, and it is an object of the present invention to provide a spatial information interpolating apparatus capable of interpolating a lack of spatial information acquired by an image sensor array and a program thereof. To do.

  The present invention has been made to solve the above-mentioned problems. First, in the spatial information interpolating device according to claim 1, a plurality of element optical elements or a plurality of element pinholes are arranged two-dimensionally. Spatial information missing in the pixel information of the imaging element image group obtained as spatial information of the space where the subject exists by imaging using the imaging optical element array and the imaging element array composed of a plurality of imaging elements Information interpolating apparatus for interpolating the image, and comprising storage means, imaging optical element array arrangement information acquisition means, imaging element array arrangement information acquisition means, integration means, and interpolation element image group generation means .

  According to such a configuration, the spatial information interpolating apparatus acquires the arrangement information of each optical element constituting the imaging optical element array from the outside by the imaging optical element array arrangement information acquisition unit and stores it in the storage unit. Then, the spatial information interpolating apparatus acquires the arrangement information of each imaging element constituting the imaging element array from the outside by the imaging element array arrangement information acquisition means and stores it in the storage means. Then, the spatial information interpolating device corresponds to each image sensor with pixel information of the imaging element image group acquired as a video signal obtained by imaging the subject in each image sensor constituting the image sensor array by the integration unit. Receiving, arranging the pixel information of each received imaging element image group in accordance with the position specified by the arrangement information of the imaging element acquired in advance, and including the gap between the imaging elements. Pixel information of the integrated photographing element image group is generated by integrating the pixel information. Here, when a gap is generated between adjacent image sensors in the image sensor array, this is reflected, and a lack occurs in the integrated photographing element image group. Then, the spatial information interpolating apparatus is a position specified by the interpolated element image group generation means based on the integrated photographing element image group arranged based on the arrangement information of the image sensor and the arrangement information of the optical element acquired in advance. First and second virtual optical element arrays configured by arranging virtual optical elements in a two-dimensional manner and a virtual imaging device having virtual pixels arranged are spaced apart from each other in a virtual space. The pixel information of the interpolation element image group that is arranged and associated with the pixel information of the integrated photographing element image group is generated on the virtual imaging element by geometrical optical calculation in the virtual space. Here, since the integrated photographing element image group used by the interpolation element image group generation means already reflects the position of the gap between the adjacent image pickup elements, the interpolation element image group generation means is used between the adjacent image pickup elements. The pixel information of the interpolated element image group reflecting the position of the gap can be generated. Then, the interpolation element image group generation means correctly associates the pixel information of the imaging element image group in the interpolation element image group with respect to the pixels for which spatial information is missing in the imaging element image group before integration. Interpolation processing is performed using the current pixel. Therefore, the spatial information interpolation apparatus can interpolate the lack of spatial information acquired by the image sensor array.

  The spatial information interpolating apparatus according to claim 2 is the spatial information interpolating apparatus according to claim 1, wherein the integrating means includes integrated memory securing means, spatial information allocating means, provisional luminance information, and identification code allocation. Means.

  According to such a configuration, the spatial information interpolating device has a storage area in which the pixel information of the imaging element image group acquired by each imaging element constituting the imaging element array can be expanded in the storage means by the integrated memory securing unit. The pixel information of the element image group is expanded corresponding to the gap according to the distance information of the gap between the adjacent image pickup elements, which is secured as an integrated memory and included in the arrangement information of the image pickup elements constituting the image pickup element array. Possible storage areas are included in the integrated memory. Then, the spatial information interpolating apparatus allocates and writes, as spatial information, the pixel information corresponding to the pixels of the imaging element image group acquired by each imaging element constituting the imaging element array to the integrated memory by the spatial information allocation unit. . Then, the spatial information interpolation device temporarily stores predetermined luminance information in a memory area to which pixel information corresponding to the pixels of the imaging element image group is not allocated in the integrated memory by the temporary luminance information and identification code assigning unit. While assigning and writing as pixel information, an identification code for identifying that the pixel information is provisional is assigned. In the spatial information interpolating apparatus, the interpolation element image group generation means refers to the information in the integrated memory, and the identification code is assigned to the pixels of the integrated photographing element image group detected by the geometric optical calculation. If so, the interpolation processing is performed assuming that the pixel is a pixel for which spatial information is missing before integration. Therefore, in the spatial information interpolating apparatus, the interpolation element image group generation means calculates the pixels allocated as spatial information and the unallocated pixels in the integrated memory in the arithmetic processing for detecting the pixels of the integrated photographing element image group. Similarly, it is possible to perform interpolation processing for pixels for which spatial information is missing due to the identification code without using the temporary luminance value assigned to the integrated memory.

  The spatial information interpolation apparatus according to claim 3 is the spatial information interpolation apparatus according to claim 2, wherein the interpolation element image group generation means includes an interpolation element image memory securing means, a corresponding pixel detection and allocation means, And interpolation processing means.

  According to this configuration, the spatial information interpolating device has a storage area having the same size as that of the integrated memory, and a storage area for storing pixel information of the pixels on the virtual imaging device by the interpolation element image memory securing unit. The storage means secures an interpolation element image memory. Then, the spatial information interpolating device corresponds to an arbitrary pixel on the virtual image sensor to which pixel information is allocated to a predetermined position of the interpolation element image memory by the corresponding pixel detection and allocation unit, and performs geometric optical calculation. For the detected pixels of the integrated photographing element image group, the position on the integrated memory is detected, and the pixel information stored at the position is assigned and written to the interpolation element image memory as pixel information of the integrated photographing element image group. When the identification code is stored at the position detected on the integrated memory, the identification code is assigned to the predetermined position in the interpolation element image memory. The spatial information interpolating apparatus refers to the interpolation element image memory after the pixel information of the integrated photographing element image group stored in the integrated memory is written into the interpolation element image memory by the interpolation processing unit, The interpolation processing is performed on the pixels to which the identification code is assigned.

  The spatial information interpolating device according to claim 4 is the spatial information interpolating device according to claim 2 or 3, wherein the provisional luminance information and the identification code allocating means are the spatial information allocating means in the integrated memory. As a piece of information indicating the provisional pixel information and the identification code in a memory area to which the pixel information has not been assigned, the luminance values of the pixels of the imaging element image group acquired by the imaging element are determined in advance. The assigned minimum luminance value or maximum luminance value was assigned.

  According to such a configuration, the spatial information interpolating device can reduce the necessary storage capacity as compared with the case where the temporary pixel information and the identification code are handled as separate information.

  The spatial information interpolation apparatus according to claim 5 is the spatial information interpolation apparatus according to any one of claims 1 to 4, wherein the interpolation element image group generation unit further includes a depth correction unit. It was decided.

  According to such a configuration, the spatial information interpolation device performs the interpolation processing in the interpolation element image group corresponding to the pixels in which spatial information is missing in the imaging element image group before integration by the depth correction unit. Later, the spatial information of the interpolation element image group is corrected so that the depth of the stereoscopic image when the spatial information of the interpolation element image group generated on the virtual image sensor is presented is equal to the depth of the subject. Therefore, when presenting spatial information, it is possible to present a natural stereoscopic image that reproduces the same depth as the subject.

  In addition, the spatial information interpolation program according to claim 6 is an imaging element array including an imaging optical element array in which a plurality of element optical elements or a plurality of element pinholes are two-dimensionally arranged, and a plurality of imaging elements. In order to interpolate the missing spatial information in the pixel information of the imaging element image group obtained as the spatial information of the space where the subject exists by imaging using an imaging optical element array The spatial information interpolation program is to function as an arrangement information acquisition unit, an image sensor array arrangement information acquisition unit, an integration unit, and an interpolation element image group generation unit.

  According to this configuration, the spatial information interpolation program acquires the arrangement information of each optical element constituting the imaging optical element array from the outside by the imaging optical element array arrangement information acquisition unit, and stores it in the storage unit. Then, the spatial information interpolation program acquires from the outside the arrangement information of each imaging element constituting the imaging element array by the imaging element array arrangement information acquisition means, and stores it in the storage means. Then, the spatial information interpolation program corresponds to each image sensor with pixel information of a photographing element image group acquired as a video signal obtained by photographing the subject in each image sensor constituting the image sensor array by the integration unit. Receiving, arranging the pixel information of each received imaging element image group in accordance with the position specified by the arrangement information of the imaging element acquired in advance, and including the gap between the imaging elements. Pixel information of the integrated photographing element image group is generated by integrating the pixel information. Then, the spatial information interpolation program is a position specified by the interpolated element image group generation means based on the integrated photographing element image group arranged based on the arrangement information of the image sensor and the arrangement information of the optical element acquired in advance. First and second virtual optical element arrays configured by arranging virtual optical elements in a two-dimensional manner and a virtual imaging device having virtual pixels arranged are spaced apart from each other in a virtual space. The pixel information of the interpolation element image group that is arranged and associated with the pixel information of the integrated photographing element image group is generated on the virtual imaging element by geometrical optical calculation in the virtual space. The interpolation element image group generation means correctly associates pixel information of the imaging element image group in the interpolation element image group with respect to pixels in which spatial information is missing in the imaging element image group before integration. Interpolation processing is performed using pixels.

The present invention has the following effects.
According to the first aspect of the present invention, when the spatial information interpolation apparatus acquires spatial information by a plurality of image sensors in the integral photography system, the spatial information missing is caused by a gap between adjacent image sensors. An interpolated element image group can be generated in an interpolated state.

According to the second aspect of the invention, the spatial information interpolating device distinguishes between the pixels of the photographing element image group acquired by the image sensor and the pixels in which the spatial information is missing, assigns them to the integrated memory, and writes them. Therefore, the pixel to be interpolated can be easily detected.
According to the third aspect of the present invention, the spatial information interpolation apparatus can easily detect the pixel lacking the spatial information and perform the interpolation process by referring to the information in the integrated memory.

  According to the invention described in claim 4, since the spatial information interpolating device allocates the temporary pixel information and the identification code to the integrated memory as one piece of information, the required storage capacity is compared with the case where the information is handled as separate information. Can be reduced.

  According to the fifth aspect of the present invention, the spatial information interpolation device corrects the spatial information of the interpolation element image group so as to be equal to the depth of the subject. Therefore, when presenting the spatial information, the same depth as the subject is obtained. It is possible to present a natural three-dimensional image that reproduces.

  According to the sixth aspect of the present invention, the computer in which the spatial information interpolation program is installed can achieve the same effects as the spatial information interpolation device by realizing each function based on this program.

It is a block diagram which shows typically the structure of the spatial information interpolation apparatus which concerns on 1st Embodiment of this invention. It is a block diagram which shows typically the structure of the integration means with which the spatial information interpolation apparatus which concerns on 1st Embodiment of this invention is provided. FIGS. 3A and 3B are conceptual diagrams schematically showing the operation of the integrated memory securing unit shown in FIG. 2, wherein FIG. 3A shows an image sensor array, and FIG. 3B shows an integrated memory corresponding to FIG. It is a block diagram which shows typically the structure of the interpolation element image group production | generation means with which the spatial information interpolation apparatus which concerns on 1st Embodiment of this invention is provided. It is a conceptual diagram which shows typically operation | movement of the corresponding pixel detection and allocation means shown in FIG. It is a conceptual diagram which shows typically the effect of a corresponding pixel detection and allocation means shown in FIG. FIG. 5 is a conceptual diagram schematically showing an intermediate image generated between virtual lens arrays in the process of calculation processing by the corresponding pixel detection and assignment unit shown in FIG. 4. FIG. 5 is a conceptual diagram schematically showing a reproduced image generated from information in an interpolation element image memory after the interpolation processing is executed by the interpolation processing means shown in FIG. 4. It is a block diagram which shows typically the structure of the interpolation element image group production | generation means with which the spatial information interpolation apparatus which concerns on 2nd Embodiment of this invention is provided. It is a conceptual diagram which shows typically operation | movement of the 2nd corresponding | compatible pixel detection and allocation means shown in FIG. FIG. 10 is a conceptual diagram schematically showing an intermediate image generated between virtual lens arrays in the process of calculation processing by the second corresponding pixel detection and assignment unit shown in FIG. 9. FIG. 10 is a conceptual diagram schematically showing a reproduced image presented by information in a second interpolation element image memory allocated by the second corresponding pixel detection and allocation unit shown in FIG. 9. FIG. 10 is a conceptual diagram schematically showing an operation of generating information in a third interpolation element image memory for inverting the depth of a reproduced image in the inversion processing information generating means shown in FIG. 9. FIG. 10 is a conceptual diagram schematically showing a reproduced image presented by information in a third interpolation element image memory generated by the process of the inversion processing information generation unit shown in FIG. 9. FIG. 10 is a conceptual diagram schematically showing a reproduced image presented by information obtained by horizontally inverting the information in the third interpolation element image memory by the inversion processing information generation unit shown in FIG. 9. It is a conceptual diagram which shows typically the example of arrangement | positioning of the conventional spatial information acquisition apparatus. It is a conceptual diagram which shows typically the example of arrangement | positioning of the conventional spatial information presentation apparatus. It is a conceptual diagram which shows typically an example of the spatial information acquisition apparatus using the conventional several image pick-up element.

  Hereinafter, embodiments for implementing a spatial information interpolation apparatus of the present invention will be described in detail with reference to the drawings. Note that the size and positional relationship of members and the like shown in the drawings may be exaggerated for clarity of explanation.

(First embodiment)
[Outline of Spatial Information Interpolator]
The spatial information interpolation device 2 according to the first embodiment shown in FIG. 1 is missing in the pixel information of the photographing element image group obtained as spatial information of the space where the subject exists by photographing using the spatial information acquisition device 1. It interpolates spatial information.

The spatial information acquisition apparatus 1 includes an imaging element array 140 (FIG. 3) including an imaging optical element array in which a plurality of minute optical elements or a plurality of minute pinholes are two-dimensionally arranged, and a plurality of imaging elements 40. (See (a)).
In the following, the imaging optical element array is an imaging optical element array similar to the lens group 112 in which convex lenses are arranged in a single plane as shown in FIG. 16, and the first and second virtual lens arrays 31, FIG. It is assumed that it has a size equivalent to 32. Further, the number of imaging elements is not particularly limited as long as it is plural, but in the following description, it is assumed that the number is four as an example.

  As shown in a front view in FIG. 3A, the imaging element array 140 includes, for example, four rectangular imaging elements 40 in an arrangement of 2 rows and 2 columns. The two image sensors 40A and 40B in the first row and the two image sensors 40C and 40D in the second row are arranged on a substrate (not shown) with a gap 41 therebetween. The two image pickup devices 40A and 40C in the first row and the two image pickup devices 40B and 40D in the second row are arranged on a substrate (not shown) with a gap 42 therebetween. Note that the gap 41 indicates the distance between the pixels of the image sensor 40 in the first row and the pixels of the image sensor 40 in the second row, and the gap 42 indicates the pixels of the image sensor 40 in the first column and 2 The space | interval between the pixels of the image sensor 40 in the column is shown.

  The spatial information acquisition apparatus 1 acquires spatial information of a space where a subject exists by photographing the subject using an integral photography (IP) method. The spatial information acquired by the spatial information acquisition device 1 is a video signal of a photographing element image group captured by a conventional spatial information acquisition device 130 using a plurality of imaging elements as shown in FIG. Note that the image sensor array 140 shown in FIG. 3A can acquire spatial information of the photographed subject at the locations where the image sensors 40A, 40B, 40C, and 40D are provided, but the locations of the gaps 41 and 42. Then, it is not possible to acquire the spatial information of the subject.

  The spatial information interpolating device 2 receives from the spatial information acquiring device 1 video signals of imaging element image groups respectively acquired by the imaging elements 40A, 40B, 40C, and 40D of the imaging element array 140. Note that the spatial information interpolation device 2 may use the video signal of the photographing element image group input from the outside as it is for processing, or store the input video signal of the photographing element image group in the spatial information interpolation device 2. In addition, it may be read from the inside of the apparatus at the time of processing.

[Configuration of Spatial Information Interpolator]
The spatial information interpolation apparatus 2 includes, for example, a computer including a calculation unit such as a CPU (Central Processing Unit), a storage unit such as a memory and a hard disk, and an interface that transmits and receives various types of information to and from the outside. It comprises a program installed in a computer, and comprises storage means 21 and processing means 22 as shown in FIG.

  The storage unit 21 stores control programs and various information stored in advance before processing by the processing unit 22, data acquired in advance for preparation of interpolation processing by the processing of the processing unit 22, interpolation processing, etc. The calculation result of the main processing is stored in a general hard disk or memory. In the storage means 21, storage areas for an integrated memory 51a, an interpolation element image memory 61a and the like which will be described later are secured. The stored contents may be stored in the same storage unit, or may be distributed and stored in other storage units.

  As shown in FIG. 1, the processing unit 22 includes an imaging optical element array arrangement information acquisition unit 3, an imaging element array arrangement information acquisition unit 4, an integration unit 5, and an interpolation element image group generation unit 6. Yes.

<Imaging optical element array arrangement information acquisition means>
The imaging optical element array arrangement information acquisition means (imaging optical element array arrangement information acquisition means) 3 acquires the arrangement information of individual element optical elements constituting the imaging optical element array (for example, the lens group 112: see FIG. 16) from the outside. And stored in the storage means 21. The imaging optical element array arrangement information acquisition unit 3 acquires the arrangement information of the element optical elements in advance so that the interpolation element image group generation unit 6 can perform an interpolation process or the like.

  Arrangement information of individual element optical elements constituting the imaging optical element array is input from an input device 11 such as a mouse, a keyboard, or a disk drive device, and is written in the storage unit 21 by the imaging optical element array arrangement information acquisition unit 3. It is. When the interpolation element image group generation unit 6 performs an interpolation process or the like, the imaging optical element array arrangement information acquisition unit 3 receives from the storage unit 21 arrangement information of individual element optical elements constituting the imaging optical element array. Is output to the interpolation element image group generation means 6.

  In the present embodiment, an optical element array (for example, the lens group 122: FIG. 17) used in a display device (for example, the spatial information display device 120: see FIG. 17) assumed to display the spatial information acquired by the spatial information acquisition device 1. 17) is, for example, the same as the imaging optical element array. Such an optical element array is referred to as a display lens array 122A (see FIG. 8). The imaging optical element array arrangement information acquisition unit 3 acquires information on the pitch, focal length, and aperture width (lens diameter) of the convex lens (element optical element) in the display lens array 122A assumed in this display device.

  In addition, when the optical element array used in the assumed display device is configured by two-dimensionally arranging the element pinholes, the pitch of the element pinholes, the opening width (pinhole diameter), the element pins Information on the distance from the hall to the imaging surface is acquired.

<Image sensor array arrangement information acquisition means>
The imaging element array arrangement information acquisition unit 4 acquires the arrangement information of each imaging element 40 constituting the imaging element array 140 from the outside and stores it in the storage unit 21. The imaging element array arrangement information acquisition unit 4 acquires the arrangement information of each imaging element 40 constituting the imaging element array 140 in advance so that the integration unit 5 and the interpolation element image group generation unit 6 can perform processing. .

  Arrangement information of each image pickup element 40 constituting the image pickup element array 140 is input from the input device 11 such as a mouse, a keyboard, and a disk drive device, and is written in the storage unit 21 by the image pickup element array arrangement information acquisition unit 4. Further, when the integration unit 5 and the interpolation element image group generation unit 6 perform processing, the image sensor array arrangement information acquisition unit 4 receives the arrangement information of each image sensor 40 constituting the image sensor array 140 from the storage unit 21. Is output to the integration unit 5 and the interpolation element image group generation unit 6. The image sensor array arrangement information acquisition unit 4 acquires position information of individual image sensors 40 constituting the image sensor array 140 and distance information of the gaps 41 and 42 between pixels of adjacent image sensors.

<Integration means>
The integration unit 5 accepts pixel information of a photographing element image group acquired as a video signal obtained by photographing a subject in each of the imaging elements constituting the imaging element array, corresponding to each imaging element, and each received photographing element image. The pixel information of the groups is aligned with the position specified by the pre-acquired image sensor arrangement information, and the pixel information of each imaging element image group including the gap between the imaging elements is integrated to integrate the group imaging element image group. Pixel information is generated.
In the present embodiment, the integration unit 5 captures image element images acquired by the individual image sensors 40A, 40B, 40C, and 40D constituting the image sensor array 140 for the image element groups input from the spatial information acquisition device 1. Integrate groups.

  As shown in FIG. 2, the integrating unit 5 includes an integrated memory securing unit 51, a spatial information allocating unit 52, and provisional luminance information and identification code allocating unit 53.

≪Integrated memory securing means≫
The integrated memory securing unit 51 stores a storage area in which all of the pixel information of the imaging element image group acquired by each imaging element 40 constituting the imaging element array 140 can be developed in the integrated memory 51a (FIG. 3B). ))). At this time, the integrated memory securing unit 51 also obtains information on the portions of the gaps 41 and 42 according to the distance information of the gaps 41 and 42 between the pixels of the adjacent image pickup devices acquired by the image pickup device array arrangement information acquisition unit 4. Allocate a memory area that can be stored. Details of FIG. 3B will be described later.

≪Spatial information allocation means≫
The spatial information allocating means 52 allocates and writes pixel information corresponding to the pixels of the imaging element image group acquired by each imaging element 40 constituting the imaging element array 140 to the integrated memory 51a as spatial information.
The spatial information assigned to the integrated memory 51a includes pixel position information of the photographing element image group and pixel information such as luminance. Note that when the luminance of a pixel is represented by, for example, 8-bit information, any value from 0 to 255 is included as the spatial information of the pixel.

≪Temporary luminance information and identification code assigning means≫
The temporary luminance information and identification code assigning means 53 assigns and writes predetermined luminance information as temporary pixel information to a memory area to which pixel information corresponding to the pixels of the photographing element image group is not assigned in the integrated memory 51a. An identification code for identifying that the pixel information is provisional is assigned.

  Here, the luminance assigned as temporary pixel information may be an arbitrary value, for example, a fixed value or a random number. That is, when the luminance of the pixel is represented by, for example, 8-bit information, it takes any value from 0 to 255. Also, the identification code may be an arbitrary value. For example, when represented by 1-bit information, F = 1 is set when the condition determination of assigning an identification code is satisfied, and F = 0 is set otherwise. Alternatively, the set value of F may be switched.

≪Specific example of processing≫
Here, as a specific example of the processing by the integration means 5, refer to FIG. 3A and FIG. 3B for the correspondence between the arrangement of the image sensor array 140 and the information assigned to and written in the integrated memory 51a ( This will be described with reference to FIG. FIG. 3A is a schematic diagram of the arrangement of the image pickup devices 40 constituting the image pickup device array 140. FIG. 3B shows the storage area of the integrated memory 51a reserved by the integrated memory reservation unit 51. It is a conceptual diagram showing typically.

  As shown in FIG. 3B, the integrated memory 51a corresponds to the arrangement of the image pickup devices 40A, 40B, 40C, and 40D constituting the image pickup device array 140, for example, four memories in an arrangement of 2 rows and 2 columns. Regions 50A, 50B, 50C, and 50D are provided. Between the two storage areas 50A and 50B in the first row and the two storage areas 50C and 50D in the second row, a storage area 50E is provided corresponding to the gap 41 in FIG. Between the two storage areas 50A, 50C in the first column and the two storage areas 50B, 50D in the second column, a storage area 50F is provided corresponding to the gap 42 in FIG.

The spatial information assigning means 52 assigns and writes the pixel information of the photographing element image group acquired by the image sensor 40A shown in FIG. 3A to the storage area 50A shown in FIG. 3B. Similarly, the image sensors 40B to 40D are assigned and written to the storage areas 50B to 50D, respectively.
The provisional luminance information and identification code assigning means 53 assigns provisional luminance values and identification codes to the storage areas 50E and 50F as spatial information of portions corresponding to the gaps 41 and 42 of the image sensor array 140. Thereby, the pixel information of the integrated photographing element image group is generated.
In the example of FIG. 3B, the temporary luminance value and the identification code are assigned to the intersection of the storage areas 50E and 50F, assuming that one of the storage areas 50E and 50F is present, and is not assigned twice.

With reference to FIG. 1, description of the structure of the spatial information interpolation apparatus 2 is continued.
[Outline of Interpolating Element Image Group Generation Means]
Interpolation element image group generation means 6 is based on the integrated photographing element image group generated by the integration means 5, and the pixel information of the interpolation element image group associated with the pixel information of the integrated photographing element image group is virtually The calculation processing is performed on the virtual imaging device 400 (see FIG. 6) in which the pixels are arranged.
The interpolation element image group generation means 6 uses the arrangement information of the convex lenses (element optical elements) in the display lens array 122A (see FIG. 8) acquired in advance by the imaging optical element array arrangement information acquisition means 3 for this calculation process. The integrated photographing element image group arranged on the basis is used. In addition, the interpolation element image group generation unit 6 is located at a position specified by the arrangement information of the image sensor 40 of the image sensor array 140 (see FIG. 3A) acquired in advance by the image sensor array arrangement information acquisition unit 4. The first and second virtual lens arrays 31 and 32 (see FIG. 5) configured by two-dimensionally arranging virtual optical elements are used.

  The interpolation element image group generation means 6 includes the integrated photographing element image group, the first and second virtual lens arrays 31 and 32 (see FIG. 5), and the virtual image sensor 400 (see FIG. 6) on the virtual space. The pixel information of the interpolated element image group is generated by geometrical optical calculation in the virtual space. The description of the geometric optical calculation will be described later. The interpolation element image group generation means 6 uses a pixel in which the pixel information of the imaging element image group is correctly associated in the interpolation element image group for the pixel in which the spatial information is missing in the imaging element image group before integration. To perform interpolation processing.

[Outline of processing procedure by interpolation element image group generation means]
As shown in FIG. 6, the interpolating element image group generating means 6 performs integrated processing of generating the pixel information of the interpolating element image group, and the first and second virtual lens arrays 31 and 32, as shown in FIG. And the virtual image sensor 400 are assumed to be arranged at predetermined relative positions. The description of FIG. 6 will be described later.
The interpolation element image group generation means 6 is an optical path that reaches the integrated photographing element image group via the first and second virtual lens arrays 31 and 32 with reference to a light beam emitted from an arbitrary pixel on the virtual image sensor 400. A specific pixel (corresponding to a pixel or a gap of the image sensor array 140) of the integrated photographing element image group is detected by the geometric optical calculation.

The interpolation element image group generation unit 6 assigns the pixel information of the detected pixels to the pixel that is the starting point of the light ray and records it on the virtual image sensor 400, thereby associating with the pixel information of the integrated shooting element image group. Pixel information of the interpolation element image group is generated on the virtual image sensor 400.
However, the interpolation element image group generation means 6 is configured so that the pixels of the integrated photographing element image group detected corresponding to the light rays emitted from the pixels on the virtual image sensor 400 have the spatial information in the photographing element image group before integration. If the pixel is missing, it is not assigned on the virtual image sensor 400. In this case, the interpolation element image group generation means 6 uses the pixel information assigned to the pixels in which the pixel information of the photographing element image group is correctly associated in the interpolation element image group, and the pixel information generated by the interpolation process. Is assigned to the pixel that is the starting point of the ray.

[Configuration example of interpolation element image group generation means]
A configuration example of the interpolation element image group generation unit 6 will be described with reference to FIG.
As illustrated in FIG. 4, the interpolation element image group generation unit 6 includes an interpolation element image memory securing unit 61, a corresponding pixel detection and allocation unit 62, and an interpolation processing unit 63.

<Interpolating element image memory securing means>
The interpolation element image memory securing means 61 is a storage area having the same size as the integrated memory 51a (see FIG. 3B), and stores the pixel information of the pixels on the virtual image sensor 400 (see FIG. 6). The area is secured in the storage means 21 as an interpolation element image memory (not shown). Hereinafter, in order to distinguish from the integrated memory 51a, it is expressed as an interpolation element image memory 61a.

<Corresponding pixel detection and allocation means>
≪Outline of corresponding pixel detection and allocation means≫
Corresponding pixel detection and allocation means 62 detects the position on the integrated memory 51a corresponding to the interpolation element image memory 61a, and the pixel information of the integrated element image group stored at the position of the integrated memory 51a is stored in the interpolation element image memory. This is assigned to 61a and written.

<< Operation by corresponding pixel detection and allocation means >>
Here, the operation by the geometric optical calculation performed by the corresponding pixel detection and assignment unit 62 will be described with reference to FIG. FIG. 5 is a conceptual diagram schematically showing a state in which the first virtual lens array 31 and the second virtual lens array 32 are arranged in a virtual three-dimensional space. This virtual three-dimensional space corresponds to the x direction (direction perpendicular to the paper surface), the y direction (width direction along the paper surface), and the z direction (depth direction along the paper surface). The first and second virtual lens arrays 31 and 32 shown in FIG. 5 have a size equivalent to an imaging optical element array (not shown) of the spatial information acquisition apparatus 1.

The information of the integrated memory 51a is shown in FIG. 5 in association with the first virtual lens array 31. In FIG. 5, the position (x ₁ y ₁ plane) in the z direction (depth direction along the paper surface) where the information of the integrated memory 51 a is arranged in a virtual three-dimensional space is specified and shown. The information in the integrated memory 51a becomes an integrated photographing element image group when all of the information is presented. Therefore, the information in the integrated memory 51a means pixel information of the integrated photographing element image group. Further, the pixel information of the integrated photographing element image group represents pixel information (position and luminance) corresponding to the element pixels constituting the element image of the integrated photographing element image group.

Further, information of the interpolation element image memory 61a is shown in FIG. 5 in association with the second virtual lens array 32. In FIG. 5, the position (x ₂ y ₂ plane) in the z direction (depth direction along the paper surface) where the information of the interpolation element image memory 61a is arranged in a virtual three-dimensional space is shown. The information in the interpolation element image memory 61a means information recorded on the virtual image sensor 400 in the processing of the corresponding pixel detection and allocation unit 62. The information in the interpolation element image memory 61a is updated during the processing of the corresponding pixel detection and assignment unit 62, and the pixel information of the integrated photographing element image group is stored in the interpolation element image memory 61a when the processing is completed. It will be the information transferred to the right place in the area. In the first embodiment, since the interpolation element image memory 61a is updated until the interpolation processing unit 63 finishes the process, the pixel information of the interpolation element image group is generated while being updated until this point.

In the z direction (depth direction along the paper surface) in FIG. 5, the distance d ₁ from the information in the integrated memory 51 a to the first virtual lens array 31 and the information from the second virtual lens array 32 to the information in the interpolation element image memory 61 a. The distance d ₂ corresponds to the focal length of an optical element constituting an imaging optical element array (not shown) of the spatial information acquisition device 1. Further, the distance L from the first virtual lens array 31 to the second virtual lens array 32 can be arbitrarily set. Note that L (+) in FIG. 5 indicates a positive value based on the arrangement surface of the first virtual lens array 31.

As information in the interpolation element image memory 61a, for the information specified by the position (h ₂ , v ₂ ) in FIG. 5 (here, information of the point P ₂ (h ₂ , v ₂ )), the integrated memory 51a The procedure for assigning the information is as follows (a1) to (a6).

The (a1) point P ₂ (h _2, v ₂₎ as the starting point, in the most short distance from the point P ₂ (h _2, v _2), to detect the optical element g ₂ constituting the second virtual lens array 32 , the point _{P 2 (h 2, v 2} ) and the determined and set by the arithmetic processor straight h ₂ passing through the principal point P ₂₀ of the optical element g _2.
(A2) The intersection point P ₁₁ between the straight line h ₂ and the first virtual lens array 31 is detected.
(A3) The optical element g ₁ constituting the first virtual lens array 31 that is closest to the intersection point P ₁₁ is detected, and the principal point P ₁₀ of the optical element g ₁ is detected.
(A4) A straight line h ₁ passing through the principal point P ₁₀ of the optical element g ₁ and having the same inclination as the straight line h ₂ is obtained and set by calculation processing.
(A5) Information (here, the point P ₁ (h ₁ , v ₁ ) specified by the position (h ₁ , v ₁ ) where the straight line h ₁ and the x ₁ y ₁ plane (information in the integrated memory 51a) intersect. ) Is detected. Of the information in the integrated memory 51a, the information on the point P ₁ (h ₁ , v ₁ ) is assigned and written as information P ₂ (h ₂ , v ₂ ) in the interpolation element image memory 61a.
(A6) However, when an identification code is assigned to the point P ₁ (h ₁ , v ₁ ) on the integrated memory 51a together with the temporary luminance, the information P of the interpolation element image memory 61a is combined with the temporary luminance and the identification code. ₂ Assign to (h ₂ , v ₂ ) and write.

With reference to FIG. 4, the description of the configuration example of the interpolation element image group generation means 6 will be continued.
<Interpolation processing means>
The interpolation processing means 63 is assigned an identification code with reference to the interpolation element image memory 61a after the pixel information of the integrated photographing element image group stored in the integration memory 51a is assigned to the interpolation element image memory 61a. Interpolation processing is performed on the pixels.
For example, as an interpolation process, there is a method of performing interpolation by linear interpolation from pixel information around a pixel to which an identification code is assigned, but other high-order interpolation methods may be applied.

  When the interpolation processing unit 63 ends the process, the update of the interpolation element image memory 61a is also ended. Therefore, the update of the pixel information of the interpolation element image group is completed at this time. By using the generated pixel information of the interpolated element image group and presenting it on a normal spatial information display device 120 (see FIG. 17), it corresponds to the subject photographed by the spatial information acquisition device 1 (see FIG. 1). A stereoscopic image can be reconstructed. Further, the generated pixel information of the interpolation element image group may be output to the output device 12 (see FIG. 1). The output device 12 includes a display device such as an LCD (Liquid Crystal Display).

[Details of processing by spatial information interpolator]
Here, the arithmetic processing by the interpolation element image group generation means 6 of the spatial information interpolation device 2 will be described in detail.
FIG. 6 is a conceptual diagram schematically showing the arrangement of the virtual three-dimensional space shown in FIG. 5 in two dimensions, and corresponding to the arrangement of the image sensor and the arrangement of the memory space. The same reference numerals are given to the same components as those in FIG.

  In FIG. 6, for example, the information of the integrated memory 51 a is stored in a storage area (memory space) assigned corresponding to the arrangement (real space) of the image sensors that constitute the image sensor array 140. In FIG. 6, the image sensor that constitutes the image sensor array 140 is indicated by a thick solid line, and a part of the thick solid line is deleted to indicate that there is a gap in the image sensor array 140. The number of image sensors and the number of lenses in the lens array are examples. The storage area of the memory space corresponding to the pixel is indicated by a rectangle along the thick solid line indicating the image sensor, that is, along the pixel.

  In FIG. 6, the information in the interpolation element image memory 61a is a storage area (memory space) allocated corresponding to the arrangement as if the virtual imaging device 400 does not actually exist but exists in the real space. Stored in In FIG. 6, the virtual image sensor 400 is indicated by a thick solid line. The storage area of the memory space corresponding to the pixel is indicated by a rectangle along the thick solid line indicating the virtual image sensor 400, that is, along the pixel.

As shown in FIG. 6, the corresponding pixel detection and allocation unit 62 assigns pixel information to a predetermined position (for example, (h ₂ , v ₂ ) in FIG. 6) as a reference in the interpolation element image memory 61a. Corresponding to the upper arbitrary pixel, the position (for example, (h ₁ , v ₁ ) in FIG. 6) of the integrated photographing element image group detected by the geometric optical calculation is detected on the integrated memory 51a. Corresponding pixel detection and allocation means 62 stores pixel information (points P ₁ (h ₁ , v ₁ )) stored at a position (for example, (h ₁ , v ₁ ) in FIG. 6) detected on the integrated memory 51a. Information) is assigned to the interpolation element image memory 61a as pixel information of the integrated photographing element image group.

When a gap is generated between adjacent image sensors in the image sensor array 140, in the integrated memory 51a, in the storage area of the memory space corresponding to the pixels along the pixels that are missing from the gap and do not actually exist. A range of information is missing continuously. In FIG. 6, the missing information corresponding to each successive pixel in the storage area is shown as filled in the positions e ₁ , e ₂ , and e ₃ as an example. Note that the omission is not limited to three pixels.

As a comparative example of the present invention, for example, when interpolating the missing e ₂ information shown in FIG. 6 on the integrated memory 51a, the information on e ₁ and e ₃ around e ₂ is also missing. For this reason, the accuracy of the interpolation process deteriorates.

On the other hand, on the interpolation element image memory 61a of the spatial information interpolating apparatus 2 according to the embodiment of the present invention, information continuously missing due to the arithmetic processing of the corresponding pixel detection and allocation means 62 is shown in FIG. In this manner, the storage areas on the interpolation element image memory 61a are assigned to dispersed positions (for example, positions represented by e _c1 , e _c2 and e _c3 in FIG. 6). Therefore, on the interpolation element image memory 61a, there is information correctly assigned around the missing information. Therefore, the missing information can be interpolated with high accuracy by interpolation processing.

  For the above reason, the interpolation processing unit 63 of the interpolation element image group generation unit 6 uses the information on the interpolation element image memory 61a to perform the interpolation processing on the pixels to which the identification code is assigned.

Subsequently, the position of the three-dimensional image when the spatial information is presented by the conventional spatial information presentation device using the information on the interpolation element image memory 61a after the interpolation processing by the interpolation processing means 63 is shown in FIGS. Explanation will be made with reference to FIG.
7 is a conceptual diagram corresponding to the conceptual diagram of FIG. 6. The same components as those in FIG. 6 are denoted by the same reference numerals, and detailed description thereof is omitted as appropriate. In FIG. 7, for example, in order to represent information in the integrated memory 51 a, the image sensor array is indicated by a straight line, and the integrated element image group is indicated along the straight line. Here, the gap is ignored. Further, in FIG. 7, in order to represent information on the interpolation element image memory 61 a, the virtual imaging element is shown by a straight line, and the interpolation element image group is shown along this straight line.
FIG. 7 is a conceptual diagram schematically showing an intermediate image generated between the first and second virtual lens arrays 31 and 32 in the process of calculation processing by the corresponding pixel detection and assignment unit 62.

This virtual intermediate image corresponds to a stereoscopic image generated when the spatial information acquisition device 1 in FIG. 1 captures the subject 111 shown in FIG. When information (integrated photographing element image group) in the integrated memory 51a is imaged by the first virtual lens array 31, a virtual intermediate image is generated. Then, this virtual intermediate image is changed by the second virtual lens array, converted so as to be re-photographed, and recorded on the virtual image sensor 400 as information on the interpolation element image memory 61a. At this time, if the distance from the subject 111 to the imaging optical element array (not shown) of the spatial information acquisition device 1 in FIG. 1 is z _c (see FIG. 16), the distance z _c2 from the first virtual lens array 31 to the intermediate image. Is represented by the following equation (4).

FIG. 8 is a conceptual diagram schematically showing a reproduced image generated from information on the interpolation element image memory 61a after the interpolation processing unit 63 executes the interpolation processing. In FIG. 8, the display element of the spatial information display device including the display lens array 122A is simplified by a straight line parallel to the column of the display lens array 122A, but this display element spreads in a direction perpendicular to the paper surface. And a thickness along the plane of the paper. When information on the interpolation element image memory 61a after the interpolation processing by the interpolation processing means 63, that is, an interpolation element image group is displayed on this display element, a reproduced image is generated as shown in FIG. Here, the display lens array 122A, when the distance to the face interpolator image group is displayed on the display device and d _r, the distance z _r to the reproduced image from the display lens array 122A is in FIGS. 7 and 8 From the geometric relationship, it can be expressed by the following equation (5).

As described above, the spatial information interpolating apparatus 2 according to the first embodiment has the space generated by the gap between the adjacent image sensors 40 when acquiring spatial information by the plurality of image sensors 40 in the integral photography method. An interpolated element image group can be generated in a state where the lack of information is interpolated.
In addition, the spatial information interpolation device 2 associates the pixel information of the integrated photographing element image group on the virtual imaging device 400 by the geometric optical calculation process using the optical path via the first and second virtual lens arrays 31 and 32. A pixel in which pixel information is correctly associated on the virtual image sensor 400 can be used for the interpolation process.

(Second Embodiment)
The spatial information interpolating apparatus according to the second embodiment of the present invention differs from the first embodiment in the function of the interpolation element image group generation means 6, so the same components as those in FIG. Therefore, the description is omitted. FIG. 9 is a block diagram schematically showing the configuration of the interpolation element image group generation means provided in the spatial information interpolation apparatus according to the second embodiment of the present invention.
Since the interpolation element image group generation unit 6A shown in FIG. 9 is different from the configuration of FIG. 4 in that a depth correction unit 64 is added, only the difference will be described below.

  The depth correction unit 64 generates the image on the virtual image sensor 400 (see FIG. 6) after executing the interpolation processing in the interpolation element image group corresponding to the pixel in which the spatial information is missing in the imaging element image group before integration. When the spatial information of the interpolated element image group is presented, the spatial information of the interpolated element image group is set so that the depth of the stereoscopic image (reproduced image: see FIG. 8) is equal to the depth of the subject 111 (see FIG. 16). It is to correct.

As shown in Equation (5), the distance z _r from the display lens array 122A to the reproduced image is the distance from the subject 111 in FIG. 16 to the imaging optical element array (not shown) of the spatial information acquisition device 1 in FIG. It is different from z _c . Even if the distance is different, it may be preferable as one effect when presenting an image. For example, scale conversion is not a problem when using an image effect in which a subject is enlarged through a concave mirror or a convex mirror.
In the second embodiment, it is assumed that a video effect without subject scale conversion is performed.

Therefore, the depth correction unit 64 stores the distance z _c from the subject to the imaging optical element array and the distance z _r from the display lens array 122A to the reproduced image on the interpolation element image memory 61a. Update information.
For this purpose, as shown in FIG. 9, the depth correction unit 64 includes a second interpolation element image memory securing unit 641, a second corresponding pixel detection and allocation unit 642, and an inversion processing information generation unit 643. .

<Second interpolation element image memory securing means>
The second interpolation element image memory securing means 641 has a function similar to that of the interpolation element image memory securing means 61, and a second interpolation element image memory (not shown) having an area of the same size as the interpolation element image memory 61a. Secure. Hereinafter, the secured area is referred to as a second interpolation element image memory 71a.

  The second corresponding pixel detection and allocation unit 642 detects a position on the interpolation element image memory 61a corresponding to the second interpolation element image memory 71a, and stores an interpolation element image group stored at the position in the interpolation element image memory 61a. Is assigned to the second interpolation element image memory 71a. That is, the second corresponding pixel detection and allocation unit 642 has the same function as the corresponding pixel detection and allocation unit 62, but between the information in the interpolation element image memory 61a and the information in the second interpolation element image memory 71a. The difference is that the corresponding pixels are detected and assigned.

  The operation by geometric optical calculation performed by the second corresponding pixel detection and assignment unit 642 will be described with reference to FIG. FIG. 10 is a conceptual diagram schematically showing a state in which the third virtual lens array 33 and the fourth virtual lens array 34 are arranged in a virtual three-dimensional space. The way of viewing FIG. 10 is the same as that of FIG. The information in the interpolation element image memory 61a is shown in FIG. 10 in association with the third virtual lens array 33. Further, information of the second interpolation element image memory 71a is shown in FIG. 10 in association with the fourth virtual lens array 34.

  In FIG. 10, a symbol indicating a distance, a position, a coordinate axis, and the like illustrated by adding 3 to the subscript related to the third virtual lens array 33 is a subscript related to the first virtual lens array 31 in FIG. 5. Symbols similar to symbols representing distances, positions, coordinate axes, and the like shown in FIG. In FIG. 10, a symbol indicating a distance, a position, a coordinate axis, and the like illustrated by adding 4 to the subscript related to the fourth virtual lens array 34 is a subscript related to the second virtual lens array 32 in FIG. 5. A symbol similar to the symbol representing the distance, position, coordinate axis, etc. shown in FIG. Therefore, detailed description is omitted.

  As shown in FIG. 10, the second corresponding pixel detection and allocation unit 642 uses the information in the interpolation element image memory 61a and the information in the interpolation element image memory 61a using the third virtual lens array 33 and the fourth virtual lens array 34. Are assigned to the second interpolation element image memory 71a.

As information of the second interpolation element image memory 71a, interpolation is performed on information specified by the position (h ₄ , v ₄ ) in FIG. 10 (here, information of the point P ₄ (h ₄ , v ₄ )). The procedure for assigning information in the element image memory 61a is as follows (b1) to (b5).

(B1) point P ₄ and (h _4, v ₄₎ as the starting point, in the most short distance from the point P ₄ (h _4, v _4), to detect the optical element g ₄ constituting a fourth virtual lens array 34 , the point _{P 4 (h 4, v 4} ) and determined and set by the arithmetic processor straight h ₄ passing through the principal point P ₄₀ of the optical element g _4.
(B2) The intersection point P ₃₁ between the straight line h ₄ and the third virtual lens array 33 is detected.
(B3) The optical element g ₃ constituting the third virtual lens array 33 that is closest to the intersection point P ₃₁ is detected, and the principal point P ₃₀ of the optical element g ₃ is detected.
(B4) A straight line h ₃ passing through the principal point P ₃₀ of the optical element g ₃ and having the same inclination as the straight line h ₄ is obtained and set by calculation processing.
(B5) Information (here, the point P ₃ (h ₃ , h ₃ , v ₃ )) specified by the position (h ₃ , v ₃ ) where the straight line h ₃ and the x ₃ y ₃ plane (information in the interpolation element image memory 61a) intersect. v ₃ that information)) is detected. Then, of the information in the interpolation element image memory 61a, the information on the point P ₃ (h ₃ , v ₃ ) is assigned and written as information P ₄ (h ₄ , v ₄ ) in the second interpolation element image memory 71a.

  Subsequently, using the information on the second interpolation element image memory 71a after the processing by the second corresponding pixel detection and assignment unit 642, the stereoscopic image when the spatial information is presented by the conventional spatial information presentation device is used. The position will be described with reference to FIGS. 11 and 12.

  FIG. 11 is a conceptual diagram in the same format as the conceptual diagram of FIG. In FIG. 11, for example, in order to represent information in the interpolation element image memory 61 a, the virtual imaging element is indicated by a straight line, and the interpolation element image group after the interpolation processing is indicated along the straight line. In FIG. 11, in order to represent the information on the second interpolation element image memory 71a, the virtual image sensor is indicated by a straight line, and the updated interpolation element image group is indicated along the straight line. That is, FIG. 11 is a conceptual diagram schematically showing the intermediate image 2 generated between the third and fourth virtual lens arrays 33 and 34 in the process of the arithmetic processing by the second corresponding pixel detection and assignment unit 642. The virtual intermediate image 2 is an image in which the depth direction (horizontal direction in FIG. 11) is inverted and the left and right directions (vertical direction in FIG. 11) are inverted, compared to the virtual intermediate image of FIG. It has become.

At this time, as shown in FIGS. 10 and 11, the distance from the information in the interpolation element image memory 61a to the third virtual lens array 33 is d ₃ , and the information from the fourth virtual lens array 34 to the second interpolation element image memory 71a. Is d ₄ , the distance z _r2 from the third virtual lens array 33 to the intermediate image 2 is expressed by the following equation (6).

D _r of formula (6), as shown in FIG. 8, the display lens array 122A, shows the distance d _r to the surface on which an image is displayed on the display device. Also, z _r in equation (6) is expressed by equation (5), and as shown in FIG. 8, when information on the interpolation element image memory 61a after executing the interpolation processing is used, the display lens The distance from the array 122A to the reproduced image to be generated is shown. On the other hand, FIG. 12 corresponds to FIG. 8 and is different in that information in the second interpolation element image memory 71a is presented. That is, FIG. 12 is a conceptual diagram schematically showing a reproduced image presented by the information in the second interpolation element image memory 71a assigned by the second corresponding pixel detection and assignment means 642. Also in FIG. 12, the distance from the display lens array 122A, to the surface on which an image is displayed on the display device is d _r.

In FIG. 12, the distance z _r3 from the display lens array 122A to the reproduced image can be expressed by the following equation (7) from the geometrical relationship of FIGS.

Here, the distance d ₃ from the information of the interpolation element image memory 61a shown in FIG. 11 to the third virtual lens array 33, a distance d _r from the information of the second interpolator image memory 71a shown in FIG. 12 to the display lens array When the relationship between the distance d ₄ from the fourth virtual lens array 34 shown in FIG. 11 to the information of the second interpolator image memory 71a, it is set to _{d 3 = d 4 = -d r} , the formula (7 ) distance z _r3 represented by will be expressed by the following equation (8).

Here, the distance z _{r on} the right side of the equation (8) is, as shown in the equation (5), the distance d ₁ (the distance from the information in the integrated memory 51a to the first virtual lens array) and the distance d ₂ ( The distance from the second virtual lens array 32 to the information in the interpolation element image memory 61a). Therefore, if these distances d ₁ and d ₂ are set to be equal distances, the distance z _r is represented by the expression (9) from the expression (5).

The equation (8) is related to the distance L ₂ (the distance between the third and fourth virtual lens arrays 33 and 34), while the equation (9) is the distance L (first and second). The distance between the virtual lens arrays 31 and 32). Therefore, if these distances L ₂ and L are set to be equal distances, the distance z _r3 is expressed by the expression (10) from the expressions (8) and (9).

Here, the distance z _c is a distance (see FIG. 16) from the subject 111 to an imaging optical element array (not shown) of the spatial information acquisition device 1 in FIG. Therefore, in FIG. 12, the distance z _r3 from the display lens array 122A to the reproduced image is equal to the distance from the subject to the imaging optical element array. However, the distance z _c is a negative value (see FIG. 16), and the distance z _r3 is a positive value (see FIG. 12). Therefore, the reproduced image shown in FIG. 12 is a reverse-view image in which the depth is inverted compared to the subject. That is, an image having the same depth (for example, concave shape) as that of the subject viewed from the left at the time of shooting is not the depth of the subject (for example, convex shape) viewed from the right through the imaging optical element array in FIG. In FIG. 12, it is observed as a reproduced image viewed from the right through the display lens array 122A.

Returning to FIG. 9, the description of the configuration of the depth correction means 64 will be continued.
<Inversion processing information generation means>
The inversion processing information generation unit 643 performs arithmetic processing for inverting the individual element images constituting the element image group that is a reproduction image in which the unevenness of the subject is inverted when displayed as a stereoscopic image, in a point-symmetric manner. A group of element images that are the same reconstructed image as the projections and depressions are generated as inversion processing information. Since this inversion processing information generating means 643 is conventionally known, it will be described briefly.

FIG. 13 is a conceptual diagram schematically showing the operation of generating information in the third interpolation element image memory for inverting the depth of the reproduced image in the inversion processing information generating means 643.
The inversion processing information generating unit 643 has the same function as the second interpolation element image memory securing unit 641 and has a third interpolation element image memory (not shown) having the same size as the second interpolation element image memory 71a. ). Hereinafter, the secured area is referred to as a third interpolation element image memory 72a. As shown in FIG. 13, the fifth virtual lens array 35 shown in FIG. 13 performs inversion processing for each element image (for each virtual lens) on the information in the second interpolation element image memory 71a, and performs third interpolation. Information in the element image memory 72a is generated.

  Subsequently, using the information on the third interpolation element image memory 72a after the processing through the fifth virtual lens array 35 is executed by the inversion processing information generation means 643, the spatial information is presented by the conventional spatial information presentation device. The position of the stereoscopic image in this case will be described with reference to FIG. FIG. 14 corresponds to FIG. 12 and is different in that information in the third interpolation element image memory 72a is presented.

In FIG. 14, the distance z _r3 from the display lens array 122A to the reproduced image is equal to the distance z _c from the subject to the imaging optical element array, and the distance z _r3 is also a negative value as in the distance z _c. (Z _r3 = z _c ) Therefore, the reproduced image shown in FIG. 14 is the same reproduced image as the unevenness of the subject.

<Modification of Second Embodiment>
The reproduced image in FIG. 14 is different in the left-right relationship when compared with the subject 111 (see FIG. 16). Here, the left and right indicate the left and right when the direction in which the observer observes the reproduced image from the observation direction indicated by the white arrow is the front in FIG. 14 (the upper in FIG. 14 represents the right and left right). . Specifically, the cylinder of the subject viewed from the right through the imaging optical element array in FIG. 16 at the time of shooting is in front of the right side, but when viewed from the right through the display lens array 122A in FIG. The cylinder will be observed on the left side. Even if the left-right relationship when compared with the subject is different, it may be preferable as one effect when presenting an image. For example, if a special method of photographing a subject reflected in a mirror is employed, a reproduced image equivalent to the subject is generated when presented.
In the modification of the second embodiment, it is assumed that the calculation process is performed so that a reproduced image equivalent to the subject is generated.

  The inversion processing information generation means 643 performs processing to invert the entire information in the third interpolation element image memory 72a to the left and right in order to correct the element image information that displays the reproduced image of FIG. By using the information on the third interpolation element image memory 72a after executing the process of inverting the entire information of the third interpolation element image memory 72a to the left and right by the inversion processing information generation means 643, the conventional spatial information presentation is performed. FIG. 15 shows the position of the stereoscopic image when the spatial information is presented by the apparatus. As shown in FIG. 15, a reproduction image equivalent to the subject is generated as a result of the processing by the inversion processing information generation means 643.

  As mentioned above, although each embodiment was described, this invention is not limited to these, In the range which does not change the meaning, it can implement variously. For example, the provisional luminance information and identification code assigning means 53 has been described as assigning provisional pixel information and identification code to a memory area to which pixel information is not assigned in the integrated memory 51a. Can be assigned as one piece of information.

  In this case, a predetermined minimum luminance value or maximum luminance value is assigned among the luminance values of the pixels of the imaging element image group acquired by the imaging element 40 constituting the imaging element array 140 of the spatial information acquisition apparatus 1. For example, when the luminance of a pixel is represented by 8-bit information, it takes any value from 0 to 255. However, for a pixel lacking spatial information, 255 or 0 is set as temporary pixel information (luminance). ). Thereby, in the integrated memory 51a, 255 or 0 can be treated as temporary pixel information (luminance) and an identification code, and other pixel information can be identified as having information acquired by the image sensor 40. . As a result, the spatial information interpolating device can reduce the required storage capacity as compared with the case where the temporary pixel information and the identification code are handled as separate information.

  In the processing procedure (a6) of the corresponding pixel detection and assignment unit 62, when the identification code is assigned together with the temporary luminance, the temporary luminance and the identification code are assigned to the information in the interpolation element image memory 61a together. However, the present invention is not limited to this. For example, when the temporary pixel information (luminance) and the identification code are different information, the temporary luminance value assigned to the integrated memory 51a may not be assigned to the interpolation element image memory 61a.

  Moreover, in each embodiment, although the optical element which comprises the 1st-5th virtual optical element array was demonstrated as an element lens, you may substitute to a pinhole. In this case, the distance corresponding to the focal length of the element lens can be an arbitrary distance.

  In addition, the spatial information interpolating device can realize each of the above means by an arithmetic circuit, and includes a central processing unit (CPU), a main storage device (RAM: Random Access Memory), and the like. It can also be realized by operating a general computer by a program (spatial information interpolation program) that functions as each means described above. This program (interpolation program) can be provided via a communication line, or can be written on a recording medium such as a CD-ROM and distributed.

DESCRIPTION OF SYMBOLS 1 Spatial information acquisition apparatus 2 Spatial information interpolation apparatus 3 Imaging optical element array arrangement information acquisition means (Imaging optical element array arrangement information acquisition means)
4 Image sensor array arrangement information acquisition means 5 Integration means 6, 6A Interpolation element image group generation means 11 Input device 12 Output device 21 Storage means 22 Processing means 31 First virtual lens array (first virtual optical element array)
32 Second virtual lens array (second virtual optical element array)
33 Third virtual lens array (third virtual optical element array)
34 Fourth virtual lens array (fourth virtual optical element array)
35 Fifth virtual lens array (fifth virtual optical element array)
40A, 40B, 40C, 40D, 40 Image sensor 41 Gap 51 Integrated memory securing means 51a Integrated memory 52 Spatial information allocation means 53 Temporary luminance information and identification code allocation means 61 Interpolation element image memory securing means 61a Interpolation element image memory 62 Corresponding pixel Detection and allocation means 63 Interpolation processing means 64 Depth correction means 71a Second interpolation element image memory 72a Third interpolation element image memory 122A Display lens array 140 Image sensor array 641 Second interpolation element image memory securing means 642 Second corresponding pixel detection and Assigning means 643 Inversion processing information generating means

Claims (6)

A space in which a subject is present by photographing using an imaging optical element array in which a plurality of element optical elements or a plurality of element pinholes are two-dimensionally arranged, and an imaging element array composed of a plurality of imaging elements A spatial information interpolating device for interpolating spatial information missing in pixel information of a photographing element image group obtained as information,
Storage means;
Imaging optical element array arrangement information acquisition means for acquiring arrangement information of each optical element constituting the imaging optical element array from outside and storing it in the storage means;
Image sensor array arrangement information acquisition means for acquiring arrangement information of each image sensor constituting the image sensor array from the outside and storing it in the storage means;
The pixel information of the imaging element image group acquired as a video signal obtained by imaging the subject in each imaging element constituting the imaging element array is received corresponding to each imaging element, and each of the received imaging element image groups is received. The pixel information is arranged in accordance with the position specified by the arrangement information of the image pickup element acquired in advance, and the pixel information of each of the image pickup element image groups including the gap between the image pickup elements is integrated to obtain the integrated image pickup element image group. An integration means for generating pixel information;
The integrated photographing element image group arranged based on the arrangement information of the image sensor, and virtual optical elements are arranged in a two-dimensional manner at a position specified by the optical element arrangement information acquired in advance. The first and second virtual optical element arrays and the virtual image sensor on which virtual pixels are arranged are arranged separately from each other in the virtual space, and the integrated photographing is performed by geometric optical calculation in the virtual space. Interpolating element image group generation means for generating pixel information of the interpolation element image group associated with the pixel information of the element image group on the virtual imaging device,
The interpolation element image group generation means includes
For pixels in which spatial information is missing in the imaging element image group before integration, interpolation processing is performed using pixels in the interpolation element image group that are correctly associated with pixel information of the imaging element image group. Spatial information interpolation device characterized by the above. The integration means includes
A storage area where the pixel information of the imaging element image group acquired by each imaging element constituting the imaging element array can be developed as an integrated memory in the storage means, and arrangement information of the imaging elements constituting the imaging element array Integrated memory securing means for including, in the integrated memory, a storage area in which pixel information of an elemental image group can be developed corresponding to the gap, according to the distance information of the gap between the adjacent image sensors included in
Spatial information allocating means for allocating and writing pixel information corresponding to the pixels of the photographic element image group acquired by each imaging element constituting the imaging element array to the integrated memory as spatial information;
Predetermined luminance information is allocated and written as temporary pixel information in a memory area to which pixel information corresponding to pixels of the imaging element image group is not allocated in the integrated memory, and the pixel information is temporary Provisional luminance information and identification code assigning means for assigning an identification code for identifying
The interpolation element image group generation means includes
When the identification code is assigned to the pixel of the integrated photographing element image group detected by the geometric optical calculation with reference to the information of the integrated memory, the pixel has lost spatial information before the integration. The spatial information interpolation apparatus according to claim 1, wherein the interpolation processing is performed on the assumption that the pixel is a pixel. The interpolation element image group generation means includes
Interpolation element image memory securing means that secures a storage area having the same size as the integrated memory and storing pixel information of pixels on the virtual image sensor as an interpolation element image memory in the storage means;
Corresponding to an arbitrary pixel on the virtual image sensor to which pixel information is assigned to a predetermined position of the interpolation element image memory, pixels of the integrated photographing element image group detected by the geometric optical calculation are stored on the integrated memory. A position is detected, and pixel information stored at the position is assigned and written to the interpolation element image memory as pixel information of the integrated photographing element image group, and the identification code is stored at the position detected on the integrated memory. Corresponding pixel detection and assignment means for assigning the identification code to the predetermined position of the interpolation element image memory,
After the pixel information of the integrated photographing element image group stored in the integrated memory is written in the interpolation element image memory, the interpolation processing is performed on the pixels to which the identification code is assigned with reference to the interpolation element image memory. Interpolation processing means for performing
The spatial information interpolating apparatus according to claim 2, comprising: The temporary luminance information and the identification code assigning means are
In the integrated memory, as a piece of information indicating the provisional pixel information and the identification code in a memory area to which the pixel information is not allocated by the spatial information allocation unit, 4. The spatial information interpolation apparatus according to claim 2, wherein a predetermined minimum luminance value or maximum luminance value is assigned among the luminance values of the pixels. The interpolation element image group generation means includes
The space of the interpolation element image group generated on the virtual imaging device after executing the interpolation processing in the interpolation element image group corresponding to the pixel in which spatial information is missing in the imaging element image group before integration 5. The apparatus according to claim 1, further comprising a depth correction unit that corrects spatial information of the interpolation element image group so that a depth of a stereoscopic image when information is presented is equal to a depth of the subject. The spatial information interpolation device according to any one of the above. A space in which a subject is present by photographing using an imaging optical element array in which a plurality of element optical elements or a plurality of element pinholes are two-dimensionally arranged, and an imaging element array composed of a plurality of imaging elements In order to interpolate the missing spatial information in the pixel information of the imaging element image group obtained as information, a computer having storage means,
Imaging optical element array arrangement information acquisition means for acquiring arrangement information of each optical element constituting the imaging optical element array from outside and storing it in the storage means;
Image pickup device array arrangement information acquisition means for acquiring arrangement information of each image pickup element constituting the image pickup element array from the outside and storing it in the storage means;
The pixel information of the imaging element image group acquired as a video signal obtained by imaging the subject in each imaging element constituting the imaging element array is received corresponding to each imaging element, and each of the received imaging element image groups is received. The pixel information is arranged in accordance with the position specified by the arrangement information of the image pickup element acquired in advance, and the pixel information of each of the image pickup element image groups including the gap between the image pickup elements is integrated to obtain the integrated image pickup element image group. Integration means for generating pixel information;
The integrated photographing element image group arranged based on the arrangement information of the image sensor, and virtual optical elements are arranged in a two-dimensional manner at a position specified by the optical element arrangement information acquired in advance. The first and second virtual optical element arrays and the virtual image sensor on which virtual pixels are arranged are arranged separately from each other in the virtual space, and the integrated photographing is performed by geometric optical calculation in the virtual space. Interpolation element image group generation means for generating pixel information of an interpolation element image group associated with pixel information of the element image group on the virtual imaging device;
Spatial information interpolation program for functioning as
The interpolation element image group generation means includes
For pixels in which spatial information is missing in the imaging element image group before integration, interpolation processing is performed using pixels in the interpolation element image group that are correctly associated with pixel information of the imaging element image group. Spatial information interpolation program characterized by

Images