JP2005069936A - Three-dimensional image forming method, and method for deriving distance from three-dimensional object - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide the image of a three-dimensional object with no image blurring using a compound-eye camera. <P>SOLUTION: The compound-eye camera focuses a plurality of reduced images on light receiving element through a micro lens array where a plurality of micro lenses are arrayed. A plurality of reduced images are obtained by photographing (S1) with the compound-eye camera, in such condition as a three-dimensional object is disposed in the range from a front side distance b<SB>f</SB>that satisfies an expression (X1) in the front direction of the micro lens array to a rear side distance b<SB>b</SB>acquired from an expression (X2). A shift amount is acquired (S2) regarding deviation in relative position between a plurality of reduced images assuming a set distance b<SB>r</SB>obtained with an expression (X3) as a distance from the micro lens array to the three-dimensional object. The plurality of reduced images are disposed on the same region based on the acquired shift amount, to constitute a single image of the three-dimensional object (S3). <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  According to the present invention, a stereoscopic object is photographed by a compound eye camera that forms a plurality of reduced images on a light receiving element through a microlens array in which a plurality of microlenses are arranged, and the image of the stereoscopic object is obtained from the plurality of reduced images. The present invention relates to a method for constructing a three-dimensional image, and a method for deriving a distance of a three-dimensional object to obtain a distance from the microlens array to each part of the three-dimensional object based on the image of the three-dimensional object composed of the plurality of reduced images. .

Recently, in a wide range of fields such as manufacturing fields such as various inspection processes in semiconductor manufacturing, medical fields such as endoscopes, and security fields such as shape recognition (recognition of human face shape, etc.) in entrance / exit management. , A technology for obtaining a clear stereoscopic image (three-dimensional image) without image blur for a three-dimensional object having a three-dimensional shape, information on a three-dimensional shape (distance information from an imaging device (camera) to each part of the three-dimensional object, etc. , Hereinafter referred to as 3D information). Further, particularly in the medical field and the security field, there is a demand for downsizing of an image pickup apparatus and that a three-dimensional object at a short distance can be picked up due to problems in use environment.
Conventionally, as a technique for acquiring a stereoscopic image and stereoscopic information, images are taken while changing the distance between a single camera (imaging device) and a stereoscopic target (imaging target), and images at each location of a plurality of images obtained thereby are obtained. A method of acquiring a stereoscopic image and information by analyzing a blur or the like, or capturing a stereoscopic object with a plurality of cameras arranged at different positions, and analyzing disparity information of the plurality of images obtained thereby to generate a stereoscopic image And multi-view methods for obtaining three-dimensional information (three-dimensional shape) are known.
However, since the former method requires an actuator that changes the distance between the camera and the three-dimensional object and its control device, the device becomes larger and complicated, and the time points at which the obtained images are captured are different. There is a problem that it cannot be applied when the three-dimensional object moves.
In the latter multi-view method, it is necessary to dispose a plurality of cameras, and to provide control means for performing synchronous control of these cameras, resulting in a problem that the apparatus becomes large and complicated.

On the other hand, in the above two methods, a monocular camera (monocular imaging device) composed of a light receiving element array and a pair of optical systems is used, but in recent years an imaging device that can be reduced in size and thickness. As a (image input device), a compound eye camera (compound eye image input device) that mimics the compound eye structure found in insects and the like has been developed.
The compound-eye camera forms a plurality of reduced images (object reduced images) on a light receiving element through a microlens array in which a plurality of microlenses are arranged.
FIG. 11 is a diagram schematically showing an optical system of a monocular camera ((a) monocular optical system) and an optical system of a compound eye camera ((b) compound eye optical system).
As can be seen by comparing (a) and (b) of FIG. 11, in the compound-eye camera, the focal length of each of the microlenses 80 (indicated by f / N in the figure) is very small. Compared to a monocular camera that can obtain an image with the same brightness, it is possible to make the device configuration extremely small and thin.
In the compound-eye camera, each of the plurality of reduced images obtained by forming the image on the light receiving element 81 for each micro lens 80 having a small aperture is an image having a low resolution. A single high-definition image can be constructed by performing a process of correcting (shifting) the image and arranging it in one area.
Here, as a technique for obtaining one high-definition image from a plurality of reduced images, for example, an image reconstruction method based on an arithmetic mean method or a pseudo inverse matrix method shown in Non-Patent Document 1, As shown in the figure, the shift amount is calculated by correlation calculation between a plurality of reduced images (reduced object images), and a plurality of reduced images are rearranged on the same region based on the shift amount, thereby obtaining a high-definition image. An image reconstruction method for obtaining the above has been proposed.

By the way, in an imaging optical system, there is a depth of field as an evaluation measure for obtaining an image having no image blur (herein, representing so-called out-of-focus) (or allowing image blur). This depth of field is expressed as the depth width (depth viewed from the camera) of the stereoscopic object that can obtain an image without image blur (or image blur is allowed) when the stereoscopic object is photographed. The The depth of field is detailed in Non-Patent Document 2 and the like.
This depth of field will be described with reference to FIG.
In FIG. 12, the object plane (front surface of the imaging target) is O, and the focal plane of the lens 82 is O ′. Here, in the image obtained through the lens 82, it is assumed that an image blur having an allowable circle of confusion circle diameter ε on the focal plane O ′ is allowed. At this time, if the image planes before and after the farthest from the focal plane O ′ when the light collected by the lens 82 falls within the range of the allowable circle of confusion (diameter ε) are O1 ′ and O2 ′, The image of the object in the range from the object side position O1 to O2 that forms an image within the range of O1 ′ to O2 ′ can be acquired without blur as an image that is imaged on the focal plane O ′. In this case refers to the distance from the O1 to the O2 and depth of field, furthermore, the distance a ₁ to the rear depth of field from the O1 to said O, the distance a ₂ from the O to the O2 front This is called depth of field.
From the imaging formula. a ₁ and a ₂ are expressed by the following equations (1) and (2), respectively. However, s, s ₁ and s ₂ represent the distances from the lens 82 to the object planes O, O 1 and O 2, f and d represent the focal length and aperture of the lens 82, and F = f / d.
Here, by setting the S _h which is obtained a distance s to the object by the following formula (3), (1) In the formula and _{(2), a 1 = ∞, a 2} ≒ S h / 2 , and the endless All objects within the distance from far to S _h / 2 fall within the depth of field. In general, _Sh is called the hyperfocal distance.
As can be seen from this equation (3), the smaller the focal length f and the aperture d of the lens, the shorter the hyperfocal length, that is, the closer the object is within the depth of field. Accordingly, in general, when a compound eye camera in which the focal length f and the aperture diameter d of each lens (microlens) is much smaller than that of a monocular camera is used, an object (very short distance) is used compared to the case of using a monocular camera. An image with no image blur (the reduced image) can be obtained with respect to the imaging target.
Japanese Patent Laid-Open No. 2003-141529 J. Tanida, T. Kumagai, K. Yamada, S. Miyatake, K. Ishida, T. Morimoto, N. Kondo, D. Miyazaki, and Y. Ichioka, "Thin Observation module by bound optics (TOMBO): concept and experimental verification, "Appl. Opt. 40, 1806-1813 (2001) Tomohashi Takahashi, "Lens Design", (Tokai University Press, 1994)

However, if a conventional multi-lens camera that can be reduced in size and thickness is used for shooting and image composition under any conditions and methods, an image or a stereoscopic image that is not blurred about a stereoscopic target (three-dimensional imaging target). There was a problem that it was not clear whether information could be obtained.
For example, when a compound eye camera is used, a reduced image without blur can be obtained even for an object (imaging target) at a short distance as described above, but in order to obtain a high-resolution (high-definition) stereoscopic image. In this case, it is necessary to perform processing for correcting a shift of a plurality of reduced images and shifting the reduced images to one area. The amount of shift varies depending on the setting value of the distance to the imaging target. Therefore, even if a stereoscopic target (stereoscopic imaging target) is in the depth of field, the setting value of the distance from the camera to the stereoscopic target is appropriate. If a reduced image is not arranged by obtaining the shift amount after setting to the above, lattice-like noise (an example of image blur) due to the misalignment of the plurality of reduced images (incorrect calculation of the shift amount) occurs. After all, it is impossible to construct a high-definition 3D image without image blur.
Accordingly, the present invention has been made in view of the above circumstances, and an object of the present invention is to provide a stereoscopic image constructing method and a compound eye camera for obtaining a stereoscopic target image without image blur using a compound eye camera. The object of the present invention is to provide a method for deriving a distance to a three-dimensional object to obtain a distance (three-dimensional information) to the three-dimensional object.

In order to achieve the above object, according to a first aspect of the present invention, a stereoscopic object is photographed by a compound eye camera that forms a plurality of reduced images on a light receiving element through a microlens array in which a plurality of microlenses are arranged. In the stereoscopic image forming method for forming the stereoscopic target image from the reduced image, the back side distance b _b obtained from the following formula (X2) from the front side distance b _f satisfying the following formula (X1) in the front direction of the microlens array. for the plurality of reduced image in which the stereoscopic object is obtained by photographing by the compound-eye camera positioned state in the range of up to the three-dimensional set distance b _r which is obtained by the following (X3) equation from the microlens array A first shift amount calculating step of obtaining a shift amount relating to a shift in relative position between the plurality of reduced images as being a distance to an object; A first reduced image arrangement step of arranging the plurality of reduced images on the same region based on the shift amount obtained in the shift amount calculation step to form a single stereoscopic object image. This is a method for constructing a stereoscopic image.
Where D is the thickness in the depth direction of the three-dimensional object viewed from the microlens array side, d and f are the aperture and focal length of each microlens, N is the number of arrays in the one-dimensional direction of the microlens, and ν is The number of pixels in the one-dimensional direction of each of the reduced images on the light receiving element, r represents a coefficient whose value is about 0.1.
According to such a method, when the thickness in the depth direction of the three-dimensional object to be imaged is known and the distance to the three-dimensional object can be set to a desired distance, there is no image blur due to one imaging. A stereoscopic target image can be obtained. In addition, since the aperture and focal length of the microlens are very small, it is possible to construct an image without image blur for a stereoscopic object at a shorter distance than when a monocular camera is used. Setting the coefficient r to about 0.1 was experimentally obtained as a condition for obtaining a stereoscopic target image without image blur.

According to a second aspect of the present invention, a stereoscopic object is photographed by a compound eye camera that forms a plurality of reduced images on a light receiving element through a microlens array in which a plurality of microlenses are arranged. In the stereoscopic image constructing method for constructing the target image, the microlens array has the above-mentioned formulas (X4) and (X5) in the front direction within the range from the front side distance b _f to the back side distance b _b. for the plurality of reduced image obtained by photographing by the compound-eye camera in a state where the three-dimensional object is located, is the distance of the set distance b _r which is obtained by the following (X6) equation from the microlens array to said solid target Obtained by a second shift amount calculating step for obtaining a shift amount relating to a relative position shift between the plurality of reduced images, and the second shift amount calculating step. And a second reduced image arrangement step of arranging a plurality of reduced images on the same region based on the shift amount to form a single image of the three-dimensional object. This is an image construction method.
Where d and f are the aperture and focal length of each of the microlenses, N is the number of arrays in the one-dimensional direction of the microlenses, ν is the number of pixels in the one-dimensional direction of each of the reduced images on the light receiving element, and r is A coefficient whose value is about 0.1, y represents a predetermined parameter common to each equation.
Even with such a method, the same effects as those of the first invention can be obtained.

According to a third aspect of the present invention, a three-dimensional object is imaged by a compound eye camera that forms a plurality of reduced images on a light receiving element through a microlens array in which a plurality of microlenses are arranged, and the three-dimensional object is obtained from the plurality of reduced images. In the method for constructing a stereoscopic image that constitutes an image of an object, the set distance is determined from the microlens array for each of a plurality of predetermined set distances for the plurality of reduced images obtained by photographing the stereoscopic object by the compound eye camera. A third shift amount calculation step for obtaining a shift amount relating to a shift in relative position between the plurality of reduced images as a distance to the three-dimensional object, and the shift amount obtained in the third shift amount calculation step. A plurality of reduced images based on the same region, and a third reduced image arrangement step of forming a single stereoscopic target image for each set distance; From the image of a plurality of element regions obtained by dividing the entire region of each component image obtained by the reduced image arrangement step, there is no image blur or small image blur for each of the corresponding component regions between the component images. The first element region image selecting step for selecting the element region image and the element region image selected by the first element region image selecting step are combined to form a single three-dimensional object image. And a method for constructing a three-dimensional image.
According to this method, a stereoscopic image is configured for each of the plurality of set distances assuming a distance to the stereoscopic object, with respect to the plurality of reduced images obtained by one imaging, and from the plurality of constituent images, The image of the element area without image blur (or small) is selected and combined for each element area, and a stereoscopic image without image blur (or small) can be obtained for all areas. Therefore, when the distance to the three-dimensional object cannot be arbitrarily set, or when the thickness of the three-dimensional object is in the depth direction, an image blur occurs in the configuration image configured based on the shift amount calculation using one set distance. Even when the maximum depth length is not exceeded, a stereoscopic image free from (or small) image blur can be obtained for the entire region.
In addition, since the aperture and focal length of the microlens are very small, it is possible to obtain an image without image blur for a stereoscopic object at a shorter distance than when a monocular camera is used.
In this case, the set distance b _r (i) (i = 1, 2,..., N) is set to a distance obtained by the following equation (X7) or a distance whose setting interval is narrower than the distance obtained by the following equation (X7). Is preferred.
Here, b _f (1) is a predetermined initial value, and r is a coefficient whose value is about 0.1.
The (X7) expression the set distance between b _r the distance to the top rear surface of the three-dimensional object image blur does not occur when the _{(i) b b (i)} , the set distance b _r (i + The setting is made so that the distance b _f (i + 1) to the forefront of the three-dimensional object that does not cause image blur when 1) matches (b _b (i) = b _f (i + 1)). The distance b _r (i) is obtained.
Therefore, if the set distance b _r (i) is set according to the equation (X7), the number of times of the reduced image placement process in the third reduced image placement step (= the number n of the set distances for a certain depth length). ) In the smallest number of times, and in the first element area image selection step, an image for each element area without image blur (or small) can be selected with certainty.
Of course, if the set distance is set to a distance whose set interval is narrower than the distance obtained by the equation (X7), the image for each element area without image blur (or smaller) in the first element area image selection step can be ensured. Can be selected. However, in this case, since the number of times of reduction image placement processing (= n) in the third reduced image placement step increases with respect to a certain length in the depth direction, the calculation load increases accordingly.

According to a fourth aspect of the present invention, a three-dimensional object is photographed by a compound eye camera that forms a plurality of reduced images on a light receiving element through a microlens array in which a plurality of microlenses are arranged, and is configured from the plurality of reduced images. A method of deriving a distance of a three-dimensional object that obtains a distance from the microlens array to each part of the three-dimensional object based on the image of the three-dimensional object, the plurality of the plurality of objects obtained by photographing the three-dimensional object with the compound eye camera For each of a plurality of reduced images, a third shift amount is obtained for each of a plurality of predetermined set distances, wherein the set distance is a distance from the microlens array to the three-dimensional object, and a shift amount relating to a relative position shift between the plurality of reduced images is obtained. Based on the shift amount obtained in the shift amount calculation step and the third shift amount calculation step, the plurality of reduced images are arranged on the same area before the shift amount calculation step. A third reduced image arrangement step that constitutes a single three-dimensional object image for each set distance, and a plurality of elements in which all regions of each component image obtained by the third reduced image arrangement step are divided Obtained from an image blur information derivation step for obtaining image blur information relating to the presence or absence of image blur or the degree of image blur for each element region corresponding to each of the component images, and the image blur information derivation step. A distance deriving step for obtaining a distance from the microlens array for each part of the three-dimensional object corresponding to each of the element regions based on the image blur information and the set distance corresponding to the image blur information. This is a method for deriving the distance of a featured solid object.
For example, the distance deriving step includes the element region having no image blur or small image blur for each of the element regions corresponding between the constituent images based on the image blur information obtained by the image blur information deriving step. A second element area image selecting step for selecting the image of the element, and corresponding to each of the element areas based on the set distance corresponding to the image of the element area selected in the second element area image selecting step The distance obtained from the microlens array for each part of the three-dimensional object to be obtained is obtained.
According to this method, a stereoscopic image is configured for each of the plurality of set distances assuming a distance to the stereoscopic object, with respect to the plurality of reduced images obtained by one imaging, and from the plurality of constituent images, The image blur information can be obtained for each element region. Here, from the image blur information, the part of the three-dimensional object corresponding to the element region determined to have no image blur (or small) is the set distance corresponding to the image of the element region (that is, the element This is within a predetermined range before and after the set distance when the image of the area is obtained.
Therefore, for the image of the element area determined to have no (or small) image blur or the smallest degree of image blur, for example, the set distance corresponding to the image is the part of the three-dimensional object corresponding to the element area. If the distance is determined as the distance, the distance between the respective portions of the three-dimensional object can be determined with the range where the image blur before and after the set distance is not generated as the maximum error range.
In addition, for example, in the case where there are a plurality of images of the element region selected as having no image blur (or small) for each element region, the maximum value and the minimum value are set for the plurality of set distances corresponding to the images. A value (distance) obtained by obtaining a value or an average value, or by obtaining a weighted average value (including a simple average) with a predetermined weight between the minimum set distance and the maximum set distance. It is also conceivable to set the distance of each part.

According to the first aspect of the invention, the compound eye camera is used when the depth of the stereoscopic object to be imaged is known and the distance to the stereoscopic object (imaging object) can be set to a desired distance. Thus, it is possible to obtain a stereoscopic target image without image blur by one imaging. In addition, it is possible to construct an image without image blur for a three-dimensional object at a shorter distance than when a monocular camera is used.
According to the third aspect of the present invention, a stereoscopic image is formed for each of a plurality of set distances assuming a distance to a stereoscopic object with respect to a plurality of reduced images obtained by one imaging using a compound eye camera. From the plurality of component images, the image of the element area without (or small) image blur is selected and combined for each element area, and a stereoscopic image without image blur (or small) is obtained for all areas. it can. Therefore, when the distance to the three-dimensional object cannot be arbitrarily set, or when the thickness of the three-dimensional object is in the depth direction, an image blur occurs in the configuration image configured based on the shift amount calculation using one set distance. Even when the maximum depth length is not exceeded, a stereoscopic image free from (or small) image blur can be obtained for the entire region.
In addition, since the aperture and focal length of the microlens are very small, it is possible to obtain an image without image blur for a stereoscopic object at a shorter distance than when a monocular camera is used.
Further, in this case, if the set distance b _r (i) is set according to the equation (X7), the number of reduced image placement processes in the third reduced image placement step (= the set distance for a certain depth length). In the first element region image selection step, it is possible to reliably select an image for each element region without image blur (or small). Further, if the set distance is set to a distance whose set interval is narrower than the distance obtained by the equation (X7), the image for each element area that is surely free from (or smaller in) image blur in the first element area image selection step. Can be selected.
According to the fourth aspect of the present invention, a stereoscopic image is constructed for each of the plurality of set distances assuming a distance to the stereoscopic object, with respect to the plurality of reduced images obtained by one imaging. The image blur information can be obtained for each element region from the component image, and from this image blur information, a range in which image blur before and after the set distance does not occur is set as a maximum error range, and each part of the three-dimensional object is determined. The distance (three-dimensional information) can be obtained.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings so that the present invention can be understood. The following embodiment is an example embodying the present invention, and does not limit the technical scope of the present invention.
FIG. 1 is a flowchart showing an example of the procedure of the stereoscopic image forming method according to the first invention, FIG. 2 is an example of a reduced image obtained by photographing with a compound eye camera, and FIG. FIG. 4 is a diagram schematically showing an observation area by a compound eye camera when the distance and the set distance used for generating the composition image are different. FIG. 4 is a diagram illustrating the set distance and the thickness of the three-dimensional object in the three-dimensional image construction method according to the first invention. FIG. 5 is a diagram schematically showing the parallax between the microlenses in the compound-eye camera, and FIG. 6 schematically shows how a plurality of reduced images are arranged in the same region. FIG. 7 is an example of a stereoscopic image obtained by the stereoscopic image construction method according to the first invention, FIG. 8 is a flowchart showing an example of the procedure of the stereoscopic image construction method according to the third invention, and FIG. 3D image according to invention 3 FIG. 10 is a flowchart schematically showing an example of the procedure of the method for deriving the distance of a three-dimensional object according to the fourth invention, and FIG. 11 is an optical system of a monocular camera and a compound eye type. FIG. 12 is a schematic diagram of the optical system of the camera, and FIG. 12 is a diagram for explaining the depth of field.
Hereinafter, the steps of the method for constructing the stereoscopic image and the method for deriving the distance of the stereoscopic object according to the embodiment of the present invention will be described step by step, but the calculation of the formulas shown in each step and the arrangement of the images (to the memory area of the image data) And the like are executed while a predetermined program is executed by a computer and the processing result is stored in a storage means such as a hard disk.

(First invention)
First, an example of the procedure of the stereoscopic image construction method according to the first invention will be described with reference to the flowchart of FIG. In the following, S1, S2, and so on indicate process procedure (step) numbers.
<S1: Shooting process>
First, under a predetermined photographing condition X, a plurality of reductions are performed on a light receiving element (the light receiving element 81 shown in FIG. 11) through a microlens array in which a plurality of microlenses (the microlens 80 shown in FIG. 11) are arranged. A three-dimensional object (three-dimensional object) that is an object to be imaged is imaged by a compound eye camera that forms an image (S1).
In the photographing condition X, the three-dimensional object is arranged within a range from a front side distance b _f satisfying the following formula (X1) in the front direction of the microlens array to a back side distance b _b obtained by the following formula (X2). Shooting with the compound-eye camera is performed.
Where D is the thickness in the depth direction of the three-dimensional object viewed from the microlens array side, d and f are the aperture and focal length of each microlens, N is the number of arrays in the one-dimensional direction of the microlens, and ν is The number of pixels in the one-dimensional direction of each of the reduced images on the light receiving element, r represents a coefficient whose value is about 0.1.
As a result, a plurality of reduced images (object reduced images) formed on the light receiving element through the microlens array are obtained. The data of the plurality of reduced images obtained by the actual photographing is transferred from the compound-eye camera to a predetermined computer and stored in a storage unit of the computer.
FIG. 2A shows a case where a dial “A”, which is a planar object, is photographed by the compound eye camera having a microlens array in which 11 × 11 (vertical × horizontal) microlenses are arranged. FIG. 2B shows an example of the plurality of reduced images (121 reduced images) formed on the light receiving element and converted into image information, and FIG. 2B shows an example of the reduced image alone. In FIG. 2, an image of a planar object is shown for convenience, but the same applies to a case where a three-dimensional object (three-dimensional object) is photographed. As shown in FIG. 2B, each of the reduced images is a coarse image with a low resolution.

Here, the derivation process of the equations (X1) and (X2) will be described.
In order to obtain a high-definition image from the plurality of reduced images obtained by photographing a three-dimensional object with a compound eye camera, the relative position shift (shift amount) between the plurality of reduced images is described later. It is necessary to arrange (rearrange) in the same area while correcting (shifting). Hereinafter, an image obtained by shifting the plurality of reduced images and arranging them in the same area is referred to as a component image. Here, the shift amount of each reduced image can be obtained by giving a distance from the microlens array to the surface of the three-dimensional object as will be described later (hereinafter, this distance is referred to as a set distance). When there is a difference between the distance and the actual distance to the surface of the three-dimensional object, image blurring occurs in the component image.

FIG. 3 shows the observation by the compound-eye camera when the distance b _f to the forefront of the three-dimensional object that is the imaging target (hereinafter referred to as the forefront of the object) is different from the set distance b _r used for generating the component image. This is a schematic representation of the region. For simplicity of explanation, FIG. 3 will be described in one dimension.
3, the microlenses each diameter d, a focal length f, the when the distance to the foreground and b _f, the relationship between the focal length and magnification of the object the reduction optical system, the object foreground observation region The length in the one-dimensional direction is b _f × d / f (region W1 in the figure). In the object reduction optical system, a plurality of pieces of pixel information are observed at regular intervals within the width d on the light receiving element as image information of the region W1.
If the number of microlens arrays in the one-dimensional direction (the number of the microlenses) is N (N = 3 in FIG. 3), the compound-eye camera (compound-eye optical system) is calculated from the parallax amount of each microlens. ) In the one-dimensional direction of the entire observation region (region W0 surrounded by a rectangle in the figure) at the position of the distance b _f is d × (N−1) + b _f × d / f. Since this total observation region W0 varies depending on the distance to the forefront of the object, when there is a difference between the distance b _f to the forefront of the object and the set distance b _r used when generating the component image, There is a difference between the component image and the actual image of the object plane.
For example, as shown in FIG. 3, when the set distance b _r is larger than the distance b _f to the forefront of the object, the configuration surface corresponding to the entire region of the configuration image (surface located at the set distance b _r ) The length of the entire observation region (region Wr surrounded by a rectangle in the figure) is wider than the entire observation region W0 on the forefront of the object. For this reason, the black point Pb on the forefront of the object, which is the end point of the region observed by one microlens in FIG. 3, is correctly outlined on the entire observation region Wr on the component surface. Although it should be projected on the single circle point Pw, it is configured as if it is on the double circle point Pg. When the distance (deviation) between the two points Pg and Pw is large, grid-like noise (an example of image blur) is generated in the configuration image. A displacement between the component surface and the actual object surface (hereinafter referred to as a positional displacement) corresponds to the extreme end point (corresponding to the points Pb and Pg) of the region observed by each of the microlenses. Becomes the most prominent.

Hereinafter, the positional shift is analyzed from the observation area of each pixel in the reduced image.
The aperture and focal length of each of the microlenses is d and f, and the number of pixels in each one-dimensional direction of the reduced image on the light receiving element is ν (the number of pixels per width d which is the same as the aperture of the microlens in the light receiving element). ), Where N is the number of arrays in the one-dimensional direction of the micro lens, and the coordinate of one end point (the left end in FIG. 3) of the entire observation region Wr on the component surface is 0, the other of the entire observation regions Wr on the component surface The coordinate of the end point (right end in FIG. 3) can be set to (ν + N−1), and the coordinate y of the point Pg on the component surface (that is, the coordinate of the most prominent point of the displacement) is It is represented by the formula (4).
This equation (4) can be transformed into the following equation (5).
Here, the distance to the surface of the solid object, even if different from the set distance b _r to be used for generating the configuration image, a predetermined ratio to the size of one pixel in the direction of the coordinate y r ( 0 <r <1, or less, where r is an allowable coefficient) (within a difference), there is almost no difference in the obtained pixel information, and image blur is not recognized (no image blur or small) Suppose that it is a range.
At this time, when the component image is generated using the set distance b _r , the front side distance b _f that is the distance on the most front side (side closest to the microlens array) where image blur is not recognized is the same. the rear side distance b _b is the distance of the most backmost side image blur is not recognized (most the microlens farther from the array), the (5) respectively in the expression of y (y + r), a (y-r) It is expressed by the following equations (6) and (7) obtained by substitution.
Thus, (6) and (7) above for a plurality of reduced image obtained by the imaging condition satisfying, by performing generation of the constituent images by using the set distance b _r which is obtained by (5), the image The component image without blur can be obtained. Here, y can be regarded as a common parameter in the equations (5) to (7).

In the expressions (5) to (7), when b _f , b _b , and b _r are regarded as variables, if any of these variables is given (if known), the value of the parameter y is obtained. The remaining variables are obtained by substituting the value of y into any one of the equations (5) to (7).
For example, _let us consider a case where a thickness D in the depth direction (depth direction viewed from the microlens array) of the three-dimensional object for which an image without image blur is to be acquired is given, and the front side distance b _f can be arbitrarily set. The thickness D here does not mean the thickness in the depth direction of the object placed as the three-dimensional object, but the depth of the portion that can be imaged by the compound eye camera (the portion that can be seen from the compound eye camera side). It means the thickness in the direction.
In this case, it is necessary to satisfy D ≦ b _b −b _f in order to obtain the component image without image blur. Here, the following equation (8) can be derived by arranging the above equations (6) and (7) as D ≦ b _b −b _f .
By substituting this equation (8) into equation (5), the above equation (X1) is obtained as a conditional expression to be satisfied by the front side distance b _f .
On the other hand, if the front side distance b _f satisfying the equation (X1) is determined, the equation (6) is transformed into “y =” and substituted for the equations (5) and (7). Thus, the set distance b _r and the back side distance b _b can be obtained.
In this way, the formula obtained by substituting the formula (6) into the formula (7) is the formula (X2).
Accordingly, the compound-eye camera under the photographing condition X in which the three-dimensional object is arranged within the range from the front side distance b _f satisfying the formula (X1) to the back side distance b _b determined by the formula (X2). For the plurality of reduced images obtained by the above, the expression (6) transformed into the form of “y =” is substituted into the expression (5) (expression (X3) described later). If the component image is generated using the determined set distance br, the component image without image blur can be obtained.

FIG. 4 shows that the forefront of the three-dimensional object having a thickness D in the depth direction is the smallest distance among the front surface side distances b _f (the front surface side distance b _f when the left side in the equation (X1) is the right side). It is an example of the graph showing the relationship between the front side distance _bf and the said thickness D when arrange _{| positioning} . In FIG. 4, f = 1.3 mm, d = 0.5 mm, N = 5, and ν = 80, and these values are values that can be satisfactorily constructed by the current device technology that constitutes the compound-eye camera. Note that r = 0.1.
As shown in FIG. 4, when the front side distance b _f is 400 mm, the thickness D in the depth direction where image blur does not occur is about 75 mm. For example, in the security field, the shape of a human face is recognized. However, it is shown that a three-dimensional image of an uneven face can be constructed without image blur.

<S2: First shift amount calculation step>
Next, the plurality of reduced image obtained by imaging using the compound-eye camera under the imaging conditions X, the set distance b _r which is obtained by the (X3) equation from the microlens array to said solid target The shift amount relating to the shift of the relative position between the plurality of reduced images is obtained (S2). The processing in this step is also performed by the computer executing a predetermined program, and the processing result (the shift amount) is stored in the storage means. Here, the expression (X3) is an expression obtained by substituting the expression (6) transformed into the form of “y =” into the expression (5). Therefore, the image capturing conditions of the plurality of reduced image obtained under X, it is possible to obtain the set distance b By using _r, the constituent images without image blur obtained by (X3) equation.
FIG. 5 is a diagram schematically showing parallax between the microlenses in the compound-eye camera.
As shown in FIG. 5, when observing an object separated by a distance b, the width W1 of the observation region by one microlens is db / f considering the magnification of the microlens. In addition, a shift in the relative position of d (= the diameter of the microlens) occurs between the observation regions of the adjacent microlenses due to parallax. The shift d due to the parallax on the object plane is reduced by f / b times in the reduced image.
In the first shift amount calculating step, the relative distance between adjacent reduced images is determined by assuming that the set distance b _f obtained by the equation (X3) is a distance from the microlens array to the three-dimensional object. Ask. Then, (df / b _f ) is a relative position shift between the adjacent reduced images.
Further, when arranging the plurality of reduced images in a predetermined same region (two-dimensional region) based on the amount of deviation of the relative value, each pixel constituting the plurality of reduced images is placed on the same region. The shift amount of each pixel is obtained for each reduced image as a correction amount to be arranged (coordinate conversion) at a position (pixel) corresponding to the position on the object plane. Here, using any one of the reduced images (for example, the reduced image at the center position) as a reference, the shift amount for correcting the shift of the relative position of the other reduced image with respect to the reference reduced image is set as each reduced image. For each two-dimensional direction of each pixel.

<S3: First Reduced Image Placement Step>
Next, the plurality of reduced images are arranged on the same region (predetermined image memory region) based on the shift amount obtained in S2 (the first shift amount calculation step), and the single three-dimensional object (S3). The processing in this step is also performed by the computer executing a predetermined program, and the processing result (image data arranged in the image memory area (data of the constituent image)) is stored in the storage means.
FIG. 6 schematically shows how the plurality of reduced images are arranged on the same region. Q0, Q1, and Q2 shown in FIG. 6 represent the reduced images each having 3 × 3 pixels formed on the light receiving element, and M1 is a region where a single three-dimensional object image is formed. The number of pixels is the same as that of the entire light receiving element.
In this reduced image arrangement step, first, the pixels of the reduced image (indicated by Q0 in FIG. 6) used as a reference in S2 are arranged as indicated by arrows in accordance with the scale of the region M1.
Subsequently, the other pixels of the reduced image (represented by Q1 and Q2 in FIG. 6) are arranged in the region M1 based on the shift amount calculated in S2.
As a result, a single high-resolution image (the constituent image) is arranged on the area M1 with the relative position shift between the reduced images corrected (corrected).
This component image is obtained under the photographing condition X based on the equations (X1) and (X2) using the allowable coefficient r (≈0.1) experimentally confirmed as a range where no image blur occurs. the resulting plurality of reduced image, wherein (X3) arranged in said plurality of reduced images from the microlens array set distance b _r which is obtained based on the shift amount calculated as a distance to the three-dimensional object by formula Therefore, the constituent image is a stereoscopic image without image blur.

Here, since the component image obtained in S3 is an image in which the pixels of the reduced images are arranged in the same area M1, the deleted pixel is dependent on the distance between the object and the microlens array. May be included.
In such a case, interpolation may be performed using pixel values around the deleted pixel.
For example, if it is assumed that the pixel value changes linearly in the horizontal or vertical direction in the case of a deleted pixel, if there are N _del consecutive deleted pixels, the i-th deleted pixel is interpolated. The pixel value P _del (i) can be calculated by applying the following equation (9).
Here, P1 and P2 are pixel values of pixels adjacent to the deleted pixel.
Then, the expression (11) is applied to the target deletion pixel in the horizontal and vertical directions, and the average value of the obtained values is set as the pixel value for the deletion pixel.
Thereby, the deletion pixel contained in the said composition image obtained by S3 is interpolated, and the said finer composition image (stereoscopic image) can be obtained.
Furthermore, sharpening processing may be further performed on the component image after interpolation.
For example, by emphasizing high frequency components using a known high-frequency emphasis filter or the like, edge enhancement is performed to sharpen the component image.

FIG. 7 is an example of a stereoscopic image (the configuration image) obtained through the above-described steps S1 to S3.
The three-dimensional object is a 100 mm square die (however, the black point of the die is indicated only by a point “5”), and one vertex of the die (four vertexes forming the corner of the “5” surface). One of them is disposed obliquely with respect to the microlens array in the compound-eye camera so as to be the forefront (shortest distance) (the distance b _f = 350 mm). R = 0.1. Thereby, the thickness (distance) in the depth direction of the die visible from the microlens array side is 120 mm. The radius of the black spot of the die is 27 mm.
As shown in FIG. 7, at the short distance of the front surface side distance b _f = 350 mm, a clear stereoscopic image (the component image) without image blur is obtained over a deep depth range of 120 mm in the depth direction. Yes.
As described above, according to the first aspect of the present invention, the thickness in the depth direction of the three-dimensional object that is the imaging target is known and the distance to the three-dimensional object can be set to a desired distance once. A three-dimensional object image free from image blur can be obtained by the imaging. In addition, since the aperture and focal length of the microlens are very small, it is possible to construct an image without image blur for a stereoscopic object at a shorter distance than when a monocular camera is used. Note that setting the coefficient r to about 0.1 is experimentally obtained as a condition for obtaining a stereoscopic target image without image blur.

(Second invention)
Not limited to the method shown for the first invention, as described above, the plurality of reduced images obtained under the photographing conditions satisfying the expressions (6) and (7) can be expressed by the expression (5) (the above by performing the production of the constituent images by using the set distance b _r which is obtained by corresponding) to X6 formula, it is possible to obtain the configuration image without image blur.
That, wherein in the front direction of the microlens array (6) of (the equivalent to X4 formula) and (7) (corresponding to the X5 equation) from the front side distance b _f respectively satisfying to the back side distance b _b A set distance obtained by the equation (5) (corresponding to the equation (X6)) for the plurality of reduced images obtained by one imaging with the compound-eye camera in a state where the three-dimensional object is arranged within the range. Assuming that _br is a distance from the microlens array to the three-dimensional object, a shift amount relating to a relative position shift between the plurality of reduced images is obtained by the same process as the first shift amount calculation step (the first shift amount). 2), the plurality of reduced images are arranged on the same region by the same process as the first reduced image arrangement step based on the shift amount obtained in the step. One standing By configuring the image of the object, it is possible to obtain an image blur without the structure image (a stereoscopic image).

As described above, if any of the variables b _f , b _b , and b _r is given (if known) in the equations (5) to (7), the value of the parameter y is obtained. By substituting the values into any of the equations (5) to (7), the remaining variables are obtained.
As a photographing condition satisfying the expressions (6) and (7), for example, a case where the front side distance b _f is given can be considered.
In this case, the parameter y is obtained by substituting the given front side distance b _f into the equation (6). If y is y1, the three-dimensional object is within the range from the given front side distance b _f to the back side distance b _b obtained by substituting y1 into the equation (7) in front of the microlens array. What is necessary is just to arrange and shoot the target.
By generating the configuration image distance calculated by substituting y1 in the equation (5) as the set distance b _r, it is possible to obtain the configuration image without image blur. The same applies when the back side distance b _b is given or when the set distance b _r is given.

(Third invention)
The above-described stereoscopic image construction method according to the first invention is effective when the thickness in the depth direction of the stereoscopic object to be imaged is known and the distance to the stereoscopic object can be set to a desired distance. the when the distance to the foreground is unclear or solid object, the three-dimensional object in the depth direction of the thickness of the image to the constituent images constructed on the basis of the shift amount calculation using one of the set distance b _r When the maximum depth length at which blur does not occur ((b _b −b _f ) when the above formulas (5) to (7) are satisfied) is exceeded, an image area in which image blur occurs is prevented. Absent.
Therefore, a stereoscopic image forming method according to the third aspect of the present invention that can obtain a stereoscopic image that does not cause image blur even in such a case will be described below.
FIG. 8 is a flowchart showing an example of the procedure of the stereoscopic image forming method according to the third invention.
<S11: Shooting process>
First, a three-dimensional object (three-dimensional object) that is an object to be photographed is photographed by a compound eye camera that forms a plurality of reduced images on a light receiving element through a microlens array in which a plurality of microlenses are arranged (S11). The photographing conditions are not particularly limited, but the stereoscopic object is arranged at least in a range that can be imaged by the compound-eye camera. The data of the plurality of reduced images obtained by the actual photographing is transferred from the compound-eye camera to a predetermined computer and stored in a storage unit of the computer.
Here, photographing with the compound-eye camera is performed in a state where the three-dimensional object is arranged at an arbitrary position (distance) in the front direction of the microlens array.
As a result, a plurality of reduced images (object reduced images) are formed on the light receiving element through the microlens array.

<S12: Third Shift Amount Calculation Step>
Next, with respect to the plurality of reduced images obtained by photographing in the second photographing step (S11), the set distance is a distance from the microlens array to the three-dimensional object for each of a plurality of predetermined set distances. Assuming that there is a shift amount regarding the relative position shift between the plurality of reduced images, the shift amount is obtained (S12). The processing in this step is also performed by the computer executing a predetermined program, and the processing result (the shift amount) is stored in the storage means.
Here, the set distance b _r (i) (i = 1, 2,..., N) is set to a distance obtained by the following equation (X7).
However, b _f (1) is a predetermined initial value, r is the allowable coefficient, and its value is about 0.1.
This equation (X7) is based on the above equations (5) to (7), and the distance b _b to the rearmost surface of the three-dimensional object where no image blur occurs when the set distance is b _r (i). (i) coincides with the distance b _f (i + 1) to the forefront of the three-dimensional object where no image blur occurs when the set distance is b _r (i + 1) (b _b (i ) = B _f (i + 1)) to obtain the set distance b _r (i).
Therefore, if the set distance b _r (i) is set according to the formula (X7), the number of times of the reduced image arrangement process in the third reduced image arrangement step described later (= the number of the set distances for a certain depth length) It is possible to select an image for each of the element regions with the smallest number of times n) and with no (or small) image blurring in a first element region image selection step described later. That is, according to the formula (X7), the number of times of the reduced image arrangement process can be set to the minimum necessary number in a third reduced image arrangement step described later.
Of course, if the set distance is set to a distance whose set interval is narrower than the distance obtained by the equation (X7), it is ensured that there is no image blur (or small) for each element region in the first element region image selection step described later. An image can be selected. However, in this case, the number of times of reduction image placement processing (= n) in the third reduced image placement step increases with respect to a certain length in the depth direction, and the calculation load increases accordingly.
As the initial value b _f (1), the closest distance to the microlens array within a range that can be imaged with no blur by the compound-eye camera, or the forefront of the three-dimensional object is the most conceivable range. The distance when approaching the microlens array may be considered.
The calculation method of the shift amount for each of the set distances b _r (i) is the same as the first shift amount calculation step.

<S13: Third Reduced Image Placement Step>
Next, on the basis of the shift amount obtained in S12 (the third shift amount calculating step), the plurality of reduced images are arranged on the same region (a predetermined image memory region) and are set for each set distance. A single image of the three-dimensional object is constructed (S13). The processing in this step is also performed by the computer executing a predetermined program, and the processing result (image data arranged in the image memory area (data of the constituent image)) is stored in the storage means.
In this step, the same processing as in the first reduced image arrangement step is performed for each of the plurality of set distances b _r (i). As a result, the number of the component images is generated by the number n of the set distances b _r (i). Here, with respect to the generated component image, the above-described deletion image interpolation / sharpening is performed as necessary.

<S14: First Element Region Image Selection Step>
Next, image blurring is performed for each element region corresponding to each of the component images from an image of a plurality of element regions obtained by dividing the entire region of each component image obtained in the third reduced image arrangement step. An image of the element region that is absent or has a small image blur is selected (S14). The processing in this step is also performed by the computer executing a predetermined program, and the processing result (for example, the number of the constituent image selected for each element region (that is, the number of the set distance br (i)) Etc.) is stored in the storage means.
FIG. 9 schematically shows the division of the component images.
In this step, as shown in FIG. 9A, when the number of pixels in the entire area of the component image is J × J (vertical × horizontal), the entire area is, for example, shown in FIG. ) To be divided into element regions of j × j pixels (1 <j <J / 2).
Then, the degree of image blur (image blur information) for each element region corresponding to each of the component images from the images of the component regions of the plurality of component images obtained by the third reduced image arrangement step. (An example of an image blur information deriving step described later) is selected, and an image with the smallest degree of image blur is selected as an image with no (or small) image blur.
Here, as the value of j is smaller (closer to 1), the image in the local area can be analyzed, so that the image quality of the final result can be improved, while the number of element areas increases (the number of computations is (J ² / j ² ) times), the calculation load of image processing increases.
Various methods are conceivable as a method for obtaining the degree of image blur in the element region.
For example, it is conceivable to apply autofocus processing in a conventional camera and use the contrast of each image in the element region as an evaluation index for the degree of image blur. In this case, the image having the maximum contrast is selected as an image without image blur (or small).
It is also conceivable to obtain a difference in image data for the same (corresponding) element regions of the i-th and i + 1-th component images and evaluate the presence / absence of image blur based on the difference. This utilizes the characteristic that the change (difference) in the value of the image data becomes large when the image data is out of focus and the image data is out of focus. For example, if there is no image blur in the image data in the h-th component image and image blur in other image data for a certain element region, the difference between the (h−1) th and h-th, and The h-th and (h + 1) -th differences are larger than other differences.
Accordingly, for example, for each element region, two of the obtained (n−1) differences having the maximum absolute value are selected, and the number of the image data used for calculating the two differences (the above example) In the case of (h-1) th, hth, (h + 1) th), the image of the number other than the maximum number and the minimum number (in the above example, the hth) is not blurred (or It may be possible to select as a small image.
In addition to these, it is possible to detect the presence or absence of lattice-like noise that occurs when the set distance br (i) is different from the actual distance to the object surface, and the contrast, the difference, and the lattice-like noise. It is also conceivable that an index combining a plurality of these factors is used as an index for image blur evaluation. These methods are merely examples.

<S15: Image combining step>
Next, the images of the element regions selected in the first element region image selection step are combined to form a single three-dimensional object image (S15). The processing in this step is also performed by the computer executing a predetermined program, and the processing result (the combined data of the single three-dimensional object image) is stored in the storage means.
As a result, the image of the element area without image blur (or small) can be combined to obtain a stereoscopic image without image blur (or small) for the entire area. Therefore, when the distance to the three-dimensional object cannot be arbitrarily set, or when the thickness of the three-dimensional object is in the depth direction, an image blur occurs in the configuration image configured based on the shift amount calculation using one set distance. Even when the maximum depth length is not exceeded, a stereoscopic image free from (or small) image blur can be obtained for the entire region.
In addition, since the aperture and focal length of the microlens are very small, it is possible to obtain an image without image blur for a stereoscopic object at a shorter distance than when a monocular camera is used.

(Fourth invention)
The first invention and the third invention are for obtaining a stereoscopic image free from image blur. However, by applying the third invention, the stereoscopic image to be imaged is obtained from the microlens array. It is possible to obtain the distance to each target part.
Hereinafter, an example of the procedure of the method for deriving the distance of the three-dimensional object according to the fourth invention will be described using the flowchart of FIG.
Since the steps S21 to S24 in the method for deriving the distance of the three-dimensional object are the same as the steps S11 to S14 shown in FIG. 8, the description thereof is omitted here. Here, the first element region image selection step in FIG. 8 corresponds to the second element region image selection step in FIG. Further, the process (step) for obtaining the degree of image blur (an example of the image blur information) of the element region described in the first element region image selection step is an example of the image blur information deriving step.

<S25: Distance derivation process>
After the processing of S21 to S24 (that is, the processing of S11 to S14), using (based on) the set distance b _r (i) corresponding to the image of the element region selected for each of the element regions in S24. , The distance from the microlens array for each part of the three-dimensional object corresponding to each of the element regions is obtained.
The part of the three-dimensional object corresponding to the element area determined as having no image blur (or small) in S24 (second element area image selection step) is the set distance b _r corresponding to the image of the element area. (i) (that is, within the predetermined range (b _f to _b b) before and after the set distance b _r (i) when the image of the element region is obtained).
Therefore, in this step, the set distance b _r (i) corresponding to the image of the element area determined to have no image blur (or small) is obtained as the distance of the part of the three-dimensional object corresponding to the element area. .
As a result, the distance of each part of the three-dimensional object can be obtained with the range (b _f to _{b b} ) in which no image blur occurs before and after the set distance b _r (i) as the maximum error range.
In addition, for example, for each of the element regions, a plurality of images with a small degree of image blur are selected, and the maximum value and minimum value are set for the set distances (plurality) corresponding to the plurality of images selected for the element region. A value (distance) obtained by obtaining a value or an average value, or by obtaining a weighted average value (including a simple average) with a predetermined weight between the minimum set distance and the maximum set distance. It is also conceivable to set the distance of each part of (other examples of the distance deriving step) and the like.
According to the fourth aspect of the invention, distance information to each part of the three-dimensional object can be obtained from the plurality of reduced images obtained by one imaging.

  The present invention can be used for various image processing apparatuses.

The flowchart showing an example of the procedure of the stereo image structure method which concerns on 1st invention. An example of a reduced image obtained by photographing with a compound eye camera. The figure which represented typically the observation area | region by the compound eye type camera in case the distance to the surface of a solid object and the setting distance used for the production | generation of a structure image differ. An example of the graph showing the relationship between the front side distance and the thickness of a three-dimensional object in the three-dimensional image structure method which concerns on 1st invention. The figure which represented typically about the parallax between the microlenses in the said compound-eye camera. The figure which represented typically a mode that a some reduced image was arrange | positioned on the same area | region. An example of a stereoscopic image obtained by the stereoscopic image construction method according to the first invention. The flowchart showing an example of the procedure of the stereo image structure method which concerns on 3rd invention. The figure which represented typically about the division | segmentation of the structure image in the stereo image structure method which concerns on 3rd invention. The flowchart showing an example of the procedure of the distance derivation | leading-out method of the solid object which concerns on 4th invention. The schematic diagram of the optical system of a monocular camera, and the optical system of a compound eye camera. The figure for demonstrating the depth of field.

Explanation of symbols

Q0, Q1, Q2 ... Reduced image M1 ... Number of regions S1, S2,...

Claims (6)

A stereoscopic image in which a stereoscopic object is photographed by a compound-eye camera that forms a plurality of reduced images on a light receiving element through a microlens array in which a plurality of microlenses are arranged, and the stereoscopic target image is configured from the plurality of reduced images In the configuration method,
The compound eye type in a state where the three-dimensional object is arranged in a range from a front side distance b _f satisfying the following formula (X1) in the front direction of the microlens array to a back side distance b _b obtained by the following formula (X2). for the plurality of reduced image obtained by the shooting by the camera, the following (X3) from the microlens array set distance b _r which is obtained by expression of the relative position between the plurality of reduced image as a distance to the three-dimensional object A first shift amount calculating step for obtaining a shift amount related to the shift;
A first reduced image arranging step of arranging a plurality of reduced images on the same region based on the shift amount obtained in the first shift amount calculating step to form a single stereoscopic target image; ,
A method for constructing a three-dimensional image, comprising:
Where D is the thickness in the depth direction of the three-dimensional object viewed from the microlens array side, d and f are the aperture and focal length of each microlens, N is the number of arrays in the one-dimensional direction of the microlens, and ν is The number of pixels in the one-dimensional direction of each of the reduced images on the light receiving element, r represents a coefficient whose value is about 0.1. A stereoscopic image in which a stereoscopic object is photographed by a compound-eye camera that forms a plurality of reduced images on a light receiving element through a microlens array in which a plurality of microlenses are arranged, and the stereoscopic target image is configured from the plurality of reduced images In the configuration method,
The compound-eye camera in a state in which the three-dimensional object is arranged within a range from a front side distance b _f to a back side distance b _b that respectively satisfy the following expressions (X4) and (X5) in the front direction of the microlens array: for the plurality of reduced image obtained by photographing by the deviation of the relative position between the plurality of reduced images as the distance the set distance b _r which is obtained by the following (X6) equation from the microlens array to said solid target A second shift amount calculating step for obtaining a shift amount with respect to
A second reduced image arrangement step of arranging the plurality of reduced images on the same region based on the shift amount obtained in the second shift amount calculation step to form a single three-dimensional object image; ,
A method for constructing a three-dimensional image, comprising:
Where d and f are the aperture and focal length of each of the microlenses, N is the number of arrays in the one-dimensional direction of the microlenses, ν is the number of pixels in the one-dimensional direction of each of the reduced images on the light receiving element, and r is A coefficient whose value is about 0.1, y represents a predetermined parameter common to each equation. A three-dimensional image in which a three-dimensional object is imaged by a compound eye camera that forms a plurality of reduced images on a light receiving element through a microlens array in which a plurality of microlenses are arranged, and the image of the three-dimensional object is configured from the plurality of reduced images In the configuration method,
For the plurality of reduced images obtained by photographing the stereoscopic object by the compound eye camera, the set distances are assumed to be distances from the microlens array to the stereoscopic object for each of a plurality of predetermined set distances. A third shift amount calculating step for obtaining a shift amount relating to a relative position shift between the reduced images;
A plurality of reduced images are arranged on the same region based on the shift amount obtained in the third shift amount calculating step, and a single three-dimensional object image is configured for each set distance. A reduced image placement process;
There is no image blur or image for each of the corresponding element regions among the plurality of element regions obtained by dividing the entire region of each component image obtained by the third reduced image arrangement step. A first element region image selection step of selecting an image of the element region having a small blur;
An image combination step of combining the images of the element regions selected in the first element region image selection step to form a single image of the three-dimensional object;
A method for constructing a three-dimensional image, comprising: The set distance b _r (i) (i = 1, 2,..., N) is set to a distance obtained by the following equation (X7) or a distance whose setting interval is narrower than a distance obtained by the following equation (X7). Item 3. The stereoscopic image construction method according to Item 2.
Here, b _f (1) is a predetermined initial value, and r is a coefficient whose value is about 0.1. A three-dimensional object is photographed by a compound eye camera that forms a plurality of reduced images on a light receiving element through a microlens array in which a plurality of microlenses are arranged, and based on the image of the three-dimensional object composed of the plurality of reduced images A method for deriving a distance of a three-dimensional object to obtain a distance from the microlens array to each part of the three-dimensional object,
For the plurality of reduced images obtained by photographing the stereoscopic object by the compound eye camera, the set distances are assumed to be distances from the microlens array to the stereoscopic object for each of a plurality of predetermined set distances. A third shift amount calculating step for obtaining a shift amount relating to a relative position shift between the reduced images;
A plurality of reduced images are arranged on the same region based on the shift amount obtained in the third shift amount calculating step, and a single three-dimensional object image is configured for each set distance. A reduced image placement process;
The presence / absence of image blur or an image for each element region corresponding to each component image from images of a plurality of element regions obtained by dividing all the regions of each component image obtained by the third reduced image arrangement step An image blur information deriving step for obtaining image blur information related to the degree of blur;
A distance derivation for obtaining a distance from the microlens array for each part of the three-dimensional object corresponding to each of the element regions based on the image blur information obtained in the image blur information deriving step and the set distance corresponding to the image blur information. Process,
A method for deriving a distance of a three-dimensional object. The distance deriving step includes
A second image that has no image blur or a small image blur is selected for each element region corresponding to each component image based on the image blur information obtained in the image blur information deriving step. For each part of the three-dimensional object corresponding to each of the element regions based on the set distance corresponding to the image of the element region selected in the second element region image selection step The method of deriving a distance of a three-dimensional object according to claim 5, wherein the distance from the microlens array is obtained.