JP2014022806A - Image pickup device and image pickup device control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image pickup device and image pickup device control method which can photograph an image in a short photographing time while at the same time realizing it with a simple configuration at low cost, thereby making it possible to synthesize a bright and clear image having an increased depth of field.SOLUTION: An image pickup device of the present invention has at least a first camera and a second camera for photographing an image. The image pickup device comprises: a subject position detection unit for detecting the subject position; a focus adjustment unit for determining the photographing focus position of the first camera and the photographing focus position of the second camera according to the subject position detected by the subject position detection unit; a parallax calculation unit for calculating parallax information from the image photographed at the photographing focus position of the first camera and the image photographed at the photographing focus position of the second camera; and an image synthesis unit for synthesizing the image of the first camera and the image of the second camera on the basis of the parallax information.

Description

  The present invention relates to an imaging apparatus that shoots a subject by changing a focus position with a plurality of cameras and synthesizes an image having a deep depth of field, and an imaging apparatus control method.

  Image sensors have become smaller in size, and are now mounted on highly mobile devices such as many mobile phones and tablet terminals. Along with that, humans began to use the camera in a wide range of scenes, but there were many complaints due to the camera's defocusing. For example, when shooting a landscape and a person at the same time, focusing on a person in the foreground will blur the background, and focusing on a distant view will blur the person, making it difficult to capture an image in which the scene and the person are both in focus. It was. For this reason, an image with a deep depth of field that is in focus (in focus) in a wide range from a foreground to a distant view has been demanded.

  However, in a general camera, when the focal point is determined to be one point from the characteristic of the lens, blurring occurs in a region far from the in-focus position, and thus it is difficult to acquire an image having a deep depth of field. Therefore, conventionally, many methods for synthesizing an image having a deep depth of field using a plurality of images having different in-focus positions have been proposed.

  In Patent Document 1, a blur function is estimated from a plurality of images with different in-focus positions, and after adding a filter process using a coefficient determined based on each estimated blur function, the depth of field is increased. A method for synthesizing deep images has been proposed.

  Further, in Patent Document 2, light passing through a lens system from a subject is divided into a plurality of lights by a half mirror and incident on a plurality of CCD (Charge Coupled Device Image Sensor) imaging elements arranged so as to be shifted along the optical axis. A method has been proposed in which a plurality of images with different in-focus positions are recorded, pixels with high sharpness are extracted from the plurality of images, and an image with a deep depth of field is synthesized at high speed.

Japanese Patent Laid-Open No. 2000-251600 JP-A-10-257369

  However, since the apparatus described in Patent Document 1 captures a plurality of images with a monocular camera, the capturing time becomes long, and if the subject moves during the capturing, the positions of the subjects are shifted between the captured images, and the images are combined. There was a problem that the image quality deteriorated.

  In addition, the apparatus described in Patent Document 2 can acquire a composite image at high speed, but requires a camera configuration using a plurality of CCDs and using a half mirror or the like, which is expensive and uses a half mirror. There was also a problem that the amount of light was reduced.

  The present invention has been made in view of such circumstances, and can be realized with a simple configuration at a low cost, and at the same time, by capturing an image with a short shooting time, a bright and clear image having a deep depth of field can be obtained. An imaging device and an imaging device control method that can be combined are provided.

  According to an aspect of the present invention, an imaging apparatus including at least a first camera and a second camera that capture an image is provided. An imaging apparatus according to the present invention includes a subject position detection unit that detects a subject position, and a shooting focus position of the first camera and the second camera according to the subject position detected by the subject position detection unit. A parallax from a focus adjustment unit that determines a shooting focus position of the first camera, an image shot at the shooting focus position of the first camera, and an image shot at the shooting focus position of the second camera A parallax calculation unit that calculates information; and an image synthesis unit that synthesizes the image of the first camera and the image of the second camera based on the parallax information. The focus adjustment unit is configured to determine a shooting focus position and the number of shooting times of the first camera and the second camera based on the subject position.

  According to another aspect of the present invention, there is provided an imaging apparatus control method in an imaging apparatus including at least a first camera and a second camera that capture an image. The imaging apparatus control method of the present invention includes a subject position detection step for detecting a subject position, and a focus position of the first camera and the second position according to the subject position detected by the subject position detection step. A focus adjustment step for determining a shooting focus position of the first camera; an image shot at the shooting focus position of the first camera; and an image shot at the shooting focus position of the second camera; A parallax calculating step for calculating parallax information from the image, and an image synthesizing step for synthesizing the image of the first camera and the image of the second camera based on the parallax information. Based on the position, the photographing focus positions and the number of photographing times of the first camera and the second camera are determined.

  Further, the present invention may be a program for causing an information processing apparatus to execute the above-described imaging device control method, or a computer-readable recording medium for recording the program. Furthermore, the program may be acquired by a transmission medium such as the Internet.

  According to the present invention, it is possible to capture an image necessary for acquiring an image with a deep depth of field in a short time by changing the in-focus position using two cameras.

It is a block diagram which shows the structural example of the image processing apparatus in 1st Embodiment. 3 is a flowchart illustrating a configuration example of an imaging apparatus according to the first embodiment. It is explanatory drawing which shows typically the operation | movement in a to-be-photographed object position detection part. It is a schematic diagram for demonstrating the depth of field. It is explanatory drawing which shows typically the determination method of an imaging | photography focus position. It is explanatory drawing which shows typically the synthetic | combination method in an image synthetic | combination part. It is a block diagram which shows the structural example of the image processing apparatus in 2nd Embodiment. It is a flowchart which shows the structural example of the imaging device in 2nd Embodiment. It is explanatory drawing which shows typically the operation | movement in a to-be-photographed object position detection part. It is a figure which shows the relationship between a focus position and a depth of field. It is a figure which shows the example regarding the case where the frequency | count of imaging | photography is an even number. It is a figure which shows the example regarding the case where the frequency | count of imaging | photography is an odd number. It is a figure which shows the example of a synthesis | combination of an integrated parallax map. It is the figure which represented the relationship between a focus position and depth of field to the parallax value.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The attached drawings show specific embodiments and implementation examples based on the principle of the present invention, but these are for understanding the present invention and are not intended to limit the present invention. Not used.

<First Embodiment>
Hereinafter, the imaging apparatus according to the first embodiment will be specifically described with reference to the drawings. FIG. 1 is a configuration example of an imaging apparatus according to this embodiment.

  The imaging apparatus 100 includes a first camera 101, a second camera 102, a subject position detection unit 103, a parameter storage unit 104, a focus adjustment unit 105, a position adjustment unit 106, a parallax calculation unit 107, and an image composition. Part 108.

  Each of the first camera 101 and the second camera 102 includes an imaging optical system including a lens, and a sensor (not shown) that receives light passing through the lens with an imaging element and converts it into a digital signal. In addition, the cameras 101 and 102 are arranged so that their optical axes are substantially parallel, and the lens position can be moved to a position that focuses on a subject at an arbitrary distance, and can be individually controlled.

  The imaging apparatus 100 uses the subject position detection unit 103 to acquire an image from the second camera 102 and obtains a focus position from the camera to the closest subject. Then, the focus adjustment unit 105 reads a parameter related to the focus position control of the camera from the parameter storage unit 104 based on the focus position detected by the subject position detection unit 103, and uses the read parameter for the shooting focus position. Is calculated. The parameters relating to the focus position control will be described later.

  The imaging apparatus 100 performs shooting by moving the cameras 101 and 102 to the shooting focus position according to the focus position calculated by the focus adjustment unit 105. The alignment unit 106 reads alignment parameters from the parameter storage unit 104 with respect to the images of the first camera 101 and the second camera 102 captured by the control of the focus adjustment unit 105, and performs alignment.

  The parallax calculation unit 107 calculates a parallax map from the images of the first camera 101 and the second camera 102 after the alignment processing by the alignment unit 106. Then, the image synthesis unit 108 creates a mask using the calculated parallax map and synthesizes an image with a deep depth of field. Here, the parallax is a shift amount of a position where the same subject is photographed between photographed images with different viewpoints. The parallax map is information that summarizes parallaxes calculated in all or some of the captured images.

  FIG. 2 is a flowchart illustrating a control method of the imaging apparatus according to the first embodiment. Hereinafter, details of the imaging apparatus control method will be described with reference to the flowchart of FIG.

  First, in step S <b> 10, the subject position detection unit 103 detects the foreground subject position by the second camera 102. At this time, the second camera 102 searches for the in-focus position from the shortest shooting distance toward infinity. The infinity refers to a distance that does not require focus adjustment after that distance in the camera optical system, and the infinity distance refers to a distance corresponding to a lens position that focuses on infinity. In addition, the shortest shooting distance is a distance from the camera corresponding to the lens position that is closest to the lens in the focusing range of the camera lens. Hereinafter, the infinite distance is called INF, and the shortest shooting distance is called MOD.

As shown in FIG. 3, the second camera 102 sets the focus position search start point to MOD, and searches for the focus position in the INF direction from MOD. In this example, a contrast detection method is used for searching for a focus position. The contrast detection method is a method of acquiring an image while moving a lens, calculating a contrast value in each image, and determining a lens position where an image with a high contrast value is taken as a focus position. First time the contrast is increased, i.e. the second camera 102 is completed at the time of moving the lens to a position D _A focus of the subject A. Here, the contrast detection range is the entire image, and the entire image is divided into a plurality of regions, and the contrast value is calculated to detect the in-focus position. Whether the contrast has increased can be determined by a method of determining a threshold value in advance and determining whether the contrast value is equal to or higher than the threshold value. For example, the determination is made when at least one of the plurality of areas exceeds the threshold. When the in-focus position is detected, the subject position detection unit 103 determines the in-focus position of the second camera 102 as the foreground subject position.

  In this example, the case of using the focus position detection by the contrast detection method has been described, but focus detection means such as a phase difference detection method used in autofocus and a hybrid method using both contrast and phase difference are used. It may be used. As described above, by detecting the position of the subject in advance, it is possible to prevent shooting in a range where the subject does not exist at the time of image shooting, and to synthesize an image having a deep depth of field in the range where the subject exists. It becomes possible.

  Next, in step S <b> 11 shown in FIG. 2, the focus adjustment unit 105 reads parameters from the parameter storage unit 104 according to the subject position detected by the subject position detection unit 103, and calculates and determines the shooting focus position. The parameter storage unit 104 stores parameters for obtaining the shooting focus position. The parameters stored here include, for example, information indicating the correspondence between the lens position and the in-focus position, the focal length of the lens, the F value of the camera, the allowable confusion circle diameter of the lens, the blur tolerance (from how many pixel deviations It is the tolerance of whether to treat as blur. In the following description, the correspondence relationship between the lens position and the focus position is stored in the parameter storage unit 104, and the focus adjustment unit 105 determines the lens position when the subject position is detected by the subject position detection unit 103 as the focus position. Convert to and control. Also, when adjusting the focus, control is performed after conversion from the in-focus position to the lens position.

FIG. 4 is a schematic diagram for explaining the depth of field, and the horizontal axis is the distance from the second camera 102. As shown in FIG. 4, the forward depth of field and L _1, when the L ₂ and rear depth of field, depth of field L _f of the camera is shown as equation (1). Here, the front depth of field refers to a range that appears to be in focus on the front side from the focus position, and the rear depth of field appears to be focused on the rear side from the focus position. Refers to a range.
L _f = L ₁ + L ₂ (1)

Further, the focal length of the lens is f, the F value of the camera is F, the allowable circle of confusion diameter of the lens is δ, the tolerance of blur (the tolerance of how many pixel shifts are treated as blur) is p, and the distance from the camera is If it is set to l, the relationship of Formula (2) and (3) will respectively hold. Therefore, Expression (4) can be obtained from Expression (1), Expression (2), and Expression (3).
L ₁ = δpFl ² / (f ² + δpFl) (2)
L ₂ = δpFl ² / (f ² −δpFl) (3)
L _f = δpFl ² / (f ² + δpFl) + δpFl ² / (f ² −δpFl) (4)

FIG. 5 is an explanatory diagram schematically showing a method for determining the photographing focus position. As shown in FIG. 5, the in-focus position f _A which foreground object position D _A detected by the object position detection unit 103 is included in the forward depth of field, determined using the above equation. Using equation (2), first, the initial value of the distance l from the camera as the foreground subject position D _A, will change the toward the distance l from the camera in the INF direction every fixed value, for each distance The forward depth of field is calculated. Foreground object position D _A and the forward depth of field is determined as the focusing position f _A distance became closest. The determined focus position f _A and photographic focal position of the second camera 102.

Next, using the expression (3), the rear depth of field is obtained from the determined shooting focus position f _{A of} the second camera 102. Then, using equation (2), the imaging focus position f _A of obtaining the backward depth of field as the initial value, from the camera from the rear depth of field of the taking-focus position f _A toward the INF position The distance l is changed for every fixed value, and the forward depth of field when each distance is set to the in-focus position is calculated. The focus position forward depth of field becomes closest to the rear depth of field and f _B photographing focus position f _A decide focus position f _B. The determined focus position f _B and imaging focus position of the first camera 101. By determining the shooting focus position according to the above policy, two images including a deep depth of field can be obtained. Focusing in this embodiment, although the focus position forward depth of field becomes closest to the rear depth of field and f _B photographing focus position f _A is determined as f _B, which necessarily closest The position need not be determined as f _B, and if at least a part of the front depth of field of f _B is included in the rear depth of field of the shooting focus position f _A , the focus position is determined as f _B. You may decide. This is because the depth of field of a composite image, which will be described later, becomes the deepest by being closest, but if at least a portion is included, the depth of field of the composite image is deeper than a monocular image.

Next, in step S12, the focus adjustment unit 105 moves the in-focus positions of the cameras 101 and 102 to the two calculated in-focus positions f _A and f _B , and performs imaging while focusing. That is, in the present embodiment, the focus adjustment unit 105 performs imaging at the two calculated imaging focus positions f _A and f _B , so the number of imaging is two.

  Next, in step S <b> 13, the alignment unit 106 reads parameters for alignment from the parameter storage unit 104. The alignment parameter is a parameter for correcting a difference in angle of view (image size) due to a change in the focus position. Since the angle of view becomes smaller as the in-focus position is closer to the INF side, the captured image is enlarged, and it becomes difficult to calculate parallax and synthesize an omnifocal image in subsequent steps. When the first and second cameras 101 and 102 are arranged in parallel as in the present embodiment and only the difference in the angle of view (image size) needs to be corrected, the parameter storage unit 104 is focused. An enlargement ratio between images photographed at different positions is held, and the alignment unit 106 reads it as a parameter.

Next, in step S < _b > 14, the alignment unit 106 performs image enlargement / reduction processing on the images captured at the in-focus positions f _A and f _B using the read parameters.

Next, in step S <b> 15, the parallax calculation unit 107 calculates a parallax map from the image that has been registered by the registration unit 106. The parallax calculation unit 107 performs block matching using the image of the first camera 101 taken as a reference image and the image of the second camera 102 taken at different in-focus positions f _A and f _B as a reference image, and the degree of correlation between both pixel blocks And a parallax map is calculated based on the degree of correlation. The correlation degree can be obtained, for example, by a correlation value calculation by the SAD (Sum of Absolute Difference) method. By detecting a position where the correlation degree is highest on the reference image in each pixel of the standard image, the parallax can be obtained. It can be calculated. In addition to SAD, a parallax map may be obtained using another degree of correlation such as SSD (Sum of Squared Difference) or ZNCC (Zero-mean Normalized Cross-Correlation).

Next, in step S <b> 16, the image composition unit 108 creates a mask at the in-focus position of the first camera 101 using the parallax map obtained by the parallax calculation unit 107, and performs image synthesis based on the mask. For image synthesis, a mask is created by dividing the parallax map into two parallax ranges, and a part of the image of the second camera 102 corresponding to the mask range is pasted to the image shot by the first camera 101. Perform synthesis. In order to divide the parallax range, it is necessary to have correspondence between the distance and the parallax. The distance-to-parallax value conversion is expressed by the following equation (5), where d is the parallax value, f is the focal length of the camera, B is the base length of the camera, δ is the allowable circle of confusion circle diameter of the lens, and D is the distance.
d = f · B / δ · D (5)

  Using equation (5), a parallax value corresponding to the in-focus position and the depth of field can be obtained. The parallax range is divided using Equation (5). For example, when the range of the parallax value corresponding to the depth of field at the in-focus position of the first camera 101 is 25 to 72, the range of the parallax value is 72 or less. And the parallax value is 73 or more, and the range where the parallax value is 73 or more is used as a mask for the in-focus position of the second camera 102.

FIG. 6 is an explanatory diagram schematically showing a composition method in the image composition unit. As shown in FIG. 6, the image synthesis is performed by canceling the parallax value from the image 202 captured by the second camera 102 using the created mask, and then extracting the mask 203 and capturing the image by the first camera 101. This is performed by replacing the corresponding pixel 201. Thereby, the image 204 with a deep depth of field can be synthesized.
Through the above processing, processing from mask generation to image composition is performed.

  Note that in the configuration of the present embodiment, an image with a deep depth of field can be synthesized by two cameras 101 and 102 at the same time, so that it can also handle moving images. In addition, since a special device such as a half mirror is not used, a sufficient amount of light can be secured, and a general camera can be used, so that the cost is low.

In this embodiment, the subject position detection unit 103 detects the foreground subject position using the second camera 102. However, the use of the second camera 102 is not limited, and which camera is used. Also good.
Although an example in which two cameras are used is shown here, three or more cameras may be used. Also, even when shooting at different in-focus positions using a single camera, the in-focus position is determined so that the subject position is considered and the depth of field calculated based on the amount of blur overlaps in advance. Thus, an image for generating an omnifocal image effectively can be acquired with a small number of times.
In this embodiment, an example is shown in which the two cameras 101 and 102 are used to capture images at the two calculated in-focus positions f _A and f _B. However, three or more cameras have different in-focus positions. You may make it image | photograph, and the frequency | count of imaging | photography is determined according to the imaging | photography focusing position calculated.

Second Embodiment
Hereinafter, the imaging apparatus according to the second embodiment will be specifically described with reference to the drawings. FIG. 7 is a configuration example of the imaging apparatus in the present embodiment.

  The imaging apparatus 300 includes a first camera 301, a second camera 302, a subject position detection unit 303, a parameter storage unit 304, a focus adjustment unit 305, a position adjustment unit 306, a parallax calculation unit 307, and parallax integration. A unit 308 and an image composition unit 309.

  Each of the first camera 301 and the second camera 302 includes an imaging optical system including a lens, and a sensor (not shown) that receives light passing through the lens with an imaging device and converts the light into a digital signal. The cameras 301 and 302 are arranged so that their optical axes are parallel to each other, and the lens position can be moved to a position for focusing on a subject at an arbitrary distance and can be individually controlled.

  The imaging apparatus 300 acquires images from the first camera 301 and the second camera 302 by the subject position detection unit 303, and focuses the in-focus position from each camera 301, 302 to the closest subject and the farthest subject. The subject position is determined by detecting the position.

  The focus adjustment unit 305 refers to the table information regarding the relationship between the in-focus position and the depth of field from the parameter storage unit 304 based on the subject position detected by the subject position detection unit 303, and determines the in-focus position and the number of times of shooting. Determine parameters such as. The table information will be described later.

  The imaging apparatus 300 performs imaging by moving the cameras 301 and 302 to the imaging focus position according to the determined parameters. The alignment unit 306 reads alignment parameters from the parameter storage unit 304 for the image captured by the focus adjustment unit 305 and performs alignment.

  The parallax calculation unit 307 calculates a parallax map that is parallax information from the images of the first camera 301 and the second camera 302 after the alignment processing by the alignment unit 306. The parallax map refers to an image for which parallax is obtained for all pixels.

  The parallax integration unit 308 synthesizes an integrated parallax map described later from the plurality of parallax maps given from the parallax calculation unit 307. The image synthesis unit 309 creates a mask using the integrated parallax map and synthesizes the omnifocal image. Here, the omnifocal image is a clear image having a deep depth of field in which all the subjects are in focus. A clear and focused image means that there is no blur larger than the preset blur tolerance.

  FIG. 8 is a flowchart illustrating a control method of the imaging apparatus according to the second embodiment. Hereinafter, the details of the imaging apparatus control method will be described with reference to the flowchart of FIG.

First, in step S20, the subject position detection unit 303 determines the subject position. FIG. 9 is an explanatory diagram schematically showing the operation of the subject position detection unit 303. The horizontal axis in FIG. 9 is the distance from the lens surfaces of the cameras 301 and 302.
The first camera 301 searches from the INF toward MOD, while the second camera 302 searches for the in-focus position from MOD toward INF.

As shown in FIG. 9, the search start point of the focus position of the first camera 301 is set to INF, and the focus position is searched in the MOD direction. In the second camera 302, the in-focus position search start point is set to MOD, and the in-focus position is searched from MOD to INF. In this example, a contrast detection method is used for searching for a focus position. First time the contrast is increased, i.e., the first camera 301, when the user moves the lens to the focus position of D _B with the subject B, the second camera 302, the focus position of the D _A which is subject A The process ends when the lens is moved to. Here, the contrast detection range differs between the first camera 301 and the second camera 302. In the case of the first camera 301 that detects from the MOD, the central area in the captured image is the contrast detection range, and in the case of the second camera 302 that detects from the INF, the peripheral area is the contrast detection range. This is because the foreground subject has a strong tendency to be the center of the image when the camera is photographed, and the distant view subject has a strong tendency to become the peripheral portion of the image. For example, when a person and a landscape are photographed at the same time, the photograph is often photographed with a composition in which the person is the center of the image and the landscape is the background surface. Whether the contrast has increased can be determined by a method of determining a threshold value in advance and determining whether the contrast value is equal to or higher than the threshold value. If the focus position has been detected, respectively, to determine the focus position D _A of the first camera 301 distant object position, the focus position D _B of the second camera 302 as a foreground subject position.
Here, the contrast detection ranges of the first camera 301 and the second camera 302 are the center region and the peripheral region, respectively. However, the detection range is the entire image, or the user can set the detection range. It is not limited to the range described here.

In this example, the case of using the focus position detection by the contrast detection method has been described, but focus detection means such as a phase difference detection method used in autofocus and a hybrid method using both contrast and phase difference are used. It may be used. Also, the subject position can be quickly determined by detecting the in-focus position of one of the two cameras and detecting the in-focus position of the far side with the other camera. Things will be possible.
As described above, by detecting the foreground subject position and the foreground subject position in advance, it is possible to prevent shooting in a range where no subject exists during image shooting, and to synthesize an omnifocal image with a small number of shooting times. Is possible.

  Next, in step S <b> 21 shown in FIG. 8, the focus adjustment unit 305 refers to the table information for obtaining the shooting focus position and the number of shootings from the parameter storage unit 304 according to the subject position determined by the subject position detection unit 303. Then, the photographing focus position and the number of photographing are determined. The parameter storage unit 304 stores information indicating the relationship between the lens position and the in-focus position, the in-focus position corresponding to the subject position, table information for obtaining the number of times of imaging, and alignment parameters. The focus adjustment unit 305 refers to this table information and determines the number of shooting times and the shooting focus position. The number of times of photographing represents the total number of times of photographing with the first camera 301 and the second camera 302, and the photographing focus position represents the focus position corresponding to the number of times of actual photographing obtained from the table information based on the subject position. . The interval between the in-focus positions to be photographed is set to be smaller on the MOD side and larger on the INF side. This is because consideration is given to the characteristics of a lens in which the depth of field is shallower toward the MOD side and the depth of field is deeper toward the INF side. In the following description, the correspondence between the lens position and the focus position is stored in the parameter storage unit 304, and the focus adjustment unit 305 focuses the lens position when the subject position is detected by the subject position detection unit 303. Convert to position and control. Also, when adjusting the focus, control is performed after conversion from the in-focus position to the lens position.

  The table information is created in consideration of the depth of field based on Expression (4) described in the first embodiment. When the focal length of the lens is 4.50 mm, the F value is 3.5, the allowable confusion circle diameter of the lens is 1.4 μm, and the blur tolerance is 2 pixels, the table information shown in FIG. 10 is obtained. Here, each in-focus position is an in-focus position that includes the entire range from MOD to INF at each in-focus position when the MOD is 270 mm and the INF is 5000 mm, 300 mm, 350 mm, 450 mm, 600 mm, 1000 mm, Six points of 2066 mm were set. Note that the inclusion here indicates that the depth of field at each in-focus position is set to overlap, and the entire range from MOD to INF is included. FIG. 10 shows that when the in-focus position is set to 300 mm, the range from 270 mm of the front depth of field to 351 mm of the rear depth of field (depth of field) is in focus. Yes. The table information of FIG. 10 is stored in advance in the parameter storage unit 304, and the number of times of photographing and the photographing focus position are obtained from this table information.

360mm the foreground object position _{D B} shown in FIG. 9, when the distant object position _{D A} and 900 mm, 360mm of foreground object position from FIG. 10 is included in 421mm from 299mm is the depth of field of 350 mm, the photographic Of the in-focus positions included in the depth of field, 350 mm is the smallest, so the in-focus position is determined to be 350 mm. Further, 900 mm of the far-field subject position is included in the depth of field of 1000 mm from 674 mm to 1938 mm, and 1000 mm is the maximum among the focus positions included in the depth of field, so the shooting focus position is determined to be 1000 mm. Is done. Therefore, in order to obtain a depth of field that includes all the distances from 360 mm to 900 mm in the synthesized omnifocal image, the shooting focus positions are 350 mm, 450 mm, 600 mm, and 1000 mm, and the number of shootings is 4 times. In FIG. 10, there are distances included in the depth of field at a plurality of in-focus positions, and in this case, either focus position may be selected. A focus position is selected, and if it is a distant subject position, a focus position included in the rear depth of field is selected. In this way, an omnifocal image can be taken with a smaller number of times.

Furthermore, 460 mm the foreground object position _{D B,} when the distant object position _{D A} and 900 mm, 460 mm in the foreground object position from FIG. 10 is included in 575mm from 370mm is the depth of field of 450 mm, the depth of field Since 450 mm is the smallest among the in-focus positions included in, the photographing in-focus position is determined to be 450 mm. Further, 900 mm of the far-field subject position is included in the depth of field of 1000 mm from 674 mm to 1938 mm, and 1000 mm is the maximum among the focus positions included in the depth of field, so the shooting focus position is determined to be 1000 mm. Is done. Therefore, in order to obtain a depth of field that encompasses all the distances from 460 mm to 900 mm in the combined omnifocal image, the shooting focus positions are 450 mm, 600 mm, and 1000 mm, and the number of shootings is 3 times. It becomes.
According to the above processing, the shooting focus position and the number of shootings are determined with reference to the table information.

Next, in step S22, the focus adjustment unit 305 changes the focus position of each of the cameras 301 and 302 according to the determined shooting focus position and the number of shooting times, and performs shooting.
As described above, considering the preset blur tolerance, the focus position is set so that the depth of field of each focus position is included at all distances, and the shooting focus position and the number of shooting times are determined. By doing so, it is possible to synthesize an omnifocal image with the smallest possible number of shootings.

  FIG. 11 is a diagram schematically showing a specific example regarding the setting of the photographing focus position and the number of photographing times. When the foreground subject position obtained by the subject position detection unit 303 is 360 mm and the foreground subject position is 900 mm, the required number of times of shooting is four times as shown in FIG. As shown in FIG. 11, first, the second camera 302 is photographed with the in-focus position set at 350 mm and the first camera 301 at the in-focus position 450 mm. Next, the second camera 302 is moved to the in-focus position 600 mm, and the first camera 301 is moved to the in-focus position 1000 mm to take an image.

  FIG. 12 is a diagram showing another example regarding the setting of the shooting focus position and the number of shootings. As shown in FIG. 12, when the foreground object position of the first camera 301 is 460 mm and the distant object position of the second camera 302 is 900 mm, the required number of times of shooting is three. Therefore, as shown in FIG. 12, first, the second camera 302 is photographed with the in-focus position set to 450 mm, and the first camera 301 with the in-focus position 600 mm. Next, only the second camera 302 is moved to the in-focus position 1000 mm for shooting.

  In addition, when taking an omnifocal image, there is a method to shoot with the focus position on the foreground subject side and the far view subject side with each camera, but the focus position on the near view subject side and the focus position on the far view subject side are also available. Depending on the distance between them, the depth of field at each in-focus position may vary greatly. In such a case, the degree of blur in each captured image is greatly different, and corresponding points between the left and right images cannot be taken, making it difficult to calculate parallax. Here, the movement of the in-focus position of the first camera 301 and the second camera 302 is directed from the foreground subject side to the distant subject side, and images are taken alternately according to the depth of field. By doing so, it is possible to shorten the time required for focusing adjustment while maintaining the accuracy of the parallax map described later.

  Next, in step S <b> 23, the alignment unit 306 reads alignment parameters from the parameter storage unit 304. The alignment parameters include internal parameters and external parameters of the camera, and a process called rectification that performs parallelization and size adjustment of images according to the optical axes of the first camera 301 and the second camera 302 is performed. Is to do. As the in-focus position is closer to the INF side, the angle of view becomes smaller and the captured image is enlarged, which makes it difficult to calculate parallax and synthesize an omnifocal image in subsequent steps. Therefore, camera parameters are calculated by performing camera calibration at each in-focus position (in this example, the in-focus positions are set as 300 mm, 350 mm, 450 mm, 600 mm, 1000 mm, and 2066 mm), and stored in the parameter storage unit 304. Keep it.

Next, in step S24, the alignment unit 306 performs rectification processing on the image captured at each in-focus position based on the parameters read in step S23, and determines the size and position of the subject appearing in the image. Adjust the position to be corrected.
In this example, the rectification process is performed using the camera parameters. However, when the cameras are arranged in parallel and only the difference in the angle of view (image size) needs to be corrected. Alternatively, the enlargement ratio between images taken by changing the in-focus position may be held in the parameter storage unit 304, and the image may be enlarged or reduced by the alignment unit 306 using this as a parameter.

  Next, in step S <b> 25, the parallax calculation unit 307 calculates a parallax map from the image that has been registered by the registration unit 306. The parallax calculation unit 307 determines that the image of the first camera 301 is the reference image and the image of the second camera 302 when the number of times the first camera 301 and the second camera 302 are captured at different in-focus positions is an even number. Is used as a reference image to obtain a correlation between both pixel blocks, and a disparity map is calculated based on the correlation. For example, as shown in FIG. 11, when shooting is performed four times with the focus position being 350 mm, 450 mm, 600 mm, and 1000 mm, the first camera 301 is aligned with the image shot with the focus position being 350 mm. A parallax map is obtained between images shot with a focal position of 450 mm, and then a parallax between an image shot with a focus position of 600 mm by the second camera 302 and an image shot with the focus position of 450 mm by the first camera 301. Ask for a map. And a parallax map is calculated | required between the image image | photographed with the focus position set to 1000 mm with the 1st camera 301, and the image image | photographed with the focus position set to 600 mm with the 2nd camera 302. FIG. Here, a total of three parallax maps can be obtained.

Further, when the number of times of shooting is an odd number, the parallax calculation unit 307 performs block matching using the image of the second camera 302 as a reference image and the image of the first camera 301 as a reference image, and obtains the degree of correlation between both pixel blocks. A parallax map is calculated based on the degree of correlation.
For example, as shown in FIG. 12, assuming that the in-focus position is 450 mm, 600 mm, and 1000 mm, and the in-focus position is 450 mm, the image captured by the second camera 302 and the in-focus position is 450 mm. A parallax map is obtained between images shot at a position of 600 mm, and then a parallax map between an image shot by the first camera 301 at a focus position of 600 mm and an image shot by the second camera 302 at a focus position of 1000 mm Ask for. Here, a total of two parallax maps can be obtained.
Here, the reference image refers to an image shot by a camera including an image at the shooting focus position on the distant side, out of the two images when calculating the parallax map. The reference image changes. This is because the image is synthesized based on an image taken with the in-focus position on the far-field subject side in an all-focus synthesis process described later.

  Similar to the first embodiment, the degree of correlation can be obtained by calculating a correlation value by a SAD (Sum of Absolute Difference) method, and the position where the degree of correlation is highest on the reference image in each pixel of the standard image is determined. By detecting, the parallax can be calculated. In addition to SAD, a parallax map may be obtained using another degree of correlation such as SSD (Sum of Squared Difference) or ZNCC (Zero-mean Normalized Cross-Correlation).

  Next, in step S <b> 26, the parallax integration unit 308 uses the parallax map at each in-focus position obtained by the parallax calculation unit 307 to perform one integrated parallax map (hereinafter referred to as an integrated parallax map). ).

  The integrated parallax map is a map obtained by obtaining a histogram from the parallax map at each in-focus position and comparing the number of parallax values at the pixel of interest to determine whether the pixel has the correct parallax value. It is.

  FIG. 13 is an explanatory diagram in the case of synthesizing an integrated parallax map from three parallax maps. As described above, when shooting is performed four times with the shooting focus positions being 350 mm, 450 mm, 600 mm, and 1000 mm, three parallax maps are obtained. A part of the parallax map between the images taken with the in-focus positions set to 350 mm and 450 mm is the parallax map 400. Also, a part of the parallax map between images taken with the in-focus position being 450 mm and 600 mm is a parallax map 401. In addition, a part of the parallax map between images taken with the focus position set to 600 mm and 1000 mm is a parallax map 402. The histograms corresponding to the parallax maps 400, 401, and 402 are a histogram 404, a histogram 405, and a histogram 406, respectively.

  The integrated parallax map 403 is synthesized using these parallax maps 400, 401, and 402. Disparity values are obtained in blocks of 3 × 3 pixels, which are parts 400, 401, and 402 of each disparity map. When the three parallax maps 400, 401, and 402 have the same parallax value, this parallax value is adopted as the parallax value of the integrated parallax map 403 regardless of the histograms 404, 405, and 406. Next, when the parallax values are different in the three parallax maps 400, 401, and 402, the parallax value of the integrated parallax map 403 is regarded as having high reliability based on the histograms 404, 405, and 406 as the parallax value of the integrated parallax map 403. adopt.

  Here, the parallax values 120, 140, and 50 are obtained for the circled pixels, but when the histogram 404, the histogram 405, and the histogram 406 corresponding to each parallax are compared, the parallax values 120 in the histogram 404 are compared. Since the number is the largest, it is adopted as the parallax value of the integrated parallax map 403. Further, the same processing is performed for the pixels surrounded by double circles, and is employed as the parallax value of the integrated parallax map 403 because the number of parallax values 10 in the histogram 406 is the largest.

  In addition, when the parallax value is not obtained for all the parallax maps in the pixel of interest, the average value of the 8 neighboring parallax values (excluding the parallax values not obtained) in the integrated parallax map 403 is used as the parallax value of the integrated parallax map 403. Adopt as.

In the histogram of the parallax map obtained from images taken at different in-focus positions, the closer to the in-focus position, the more the parallax value corresponding to the in-focus position tends to increase. On the other hand, as the distance from the in-focus position increases, the other parallax values decrease. For such a reason, an appropriate parallax value can be selected by comparing parallax values using a histogram.
In this example, the parallax map is synthesized from the parallax histogram, but since the in-focus position is known from FIG. 10, the parallax value corresponding to the in-focus position is obtained, and each synthesizing map is synthesized. An integrated parallax map may be obtained by dividing the synthesis range of parallax maps based on the depth of field at the focal position.

Next, in step S27, the image synthesis unit 309 creates a mask at each in-focus position using the integrated parallax map 403 obtained by the parallax integration unit 308, and synthesizes an omnifocal image based on the mask.
In the synthesis of the omnifocal image, the parallax map is divided into parallax ranges corresponding to the number of times of photographing, and the images photographed at the respective in-focus positions are bonded together. In order to divide the parallax range, it is necessary that correspondence between the distance and the parallax is obtained. Therefore, a correspondence table between the parallax and the distance is obtained using the equation (5) shown in the first embodiment. As described above, since the focal length of the lens is 4.50 mm and the allowable circle of confusion diameter of the lens is 1.4 μm, if the baseline length of the first camera 301 and the second camera 302 is 15 mm, FIG. As shown, the relationship between the in-focus position and the depth of field can be expressed as a parallax value.
The mask at each in-focus position is created by extracting the parallax range from the integrated parallax map 403 based on the parallax values using FIG. Note that the processing method and the base image differ depending on whether the number of times of shooting is an even number or an odd number. Here, the base image is an image shot at the shooting focus position on the far-field subject position side, and an omnifocal image is synthesized based on this image.

  When the number of times of shooting is an even number, the image of the first camera 301 on the distant subject position side is set as the base image of the omnifocal image. The parallax range is divided using FIG. 14 mainly based on the in-focus position and the depth of field captured by the first camera 301. As shown in FIG. 11, when the imaging focus positions are 350 mm, 450 mm, 600 mm, and 1000 mm, the parallax values at the focus positions are 138, 107, 80, and 48, respectively, from FIG. The base image is an image captured by the first camera 301 with the in-focus position set at 1000 mm. Here, first, a parallax range at a position of 1000 mm serving as a base image is determined as 72 to 25 using FIG. Next, a parallax range at a focusing position of 450 mm, which is a camera image for capturing a base image, is determined as 130 to 84. Then, the remaining parallax range is allocated to the parallax ranges of the focusing positions 350 mm and 600 mm taken by the second camera 302. Thereby, the parallax range at the focusing position 300 mm is 161 to 131, and the parallax range at the focusing position 600 mm is 83 to 73.

  In accordance with the above determination, a parallax range with a parallax value of 84 to 130 taken at the 450 mm position photographed by the first camera 301 is extracted from the integrated parallax map 403 to create a mask at the in-focus position 450 mm. In addition, the parallax range of parallax values 131 to 161 at the 350 mm position and the parallax range of parallax values 73 to 83 at the 600 mm position captured by the second camera 302 are extracted from the integrated parallax map, respectively, and masks at the focusing positions of 350 mm and 600 mm, respectively. Create

  When the number of times of photographing is an odd number, the image of the second camera 302 on the distant subject position side is set as the base image of the omnifocal image. The parallax range is divided using FIG. 14 mainly based on the focus position and the depth of field captured by the second camera. As shown in FIG. 12, when the photographing focus positions are 450 mm, 600 mm, and 1000 mm, the parallax values at that time are 107, 80, and 48 in this order from FIG. The base image is an image photographed by the second camera 302 at a focusing position of 1000 mm. Again, the parallax range at each in-focus position is determined in the same flow as described above. In accordance with this determination, a parallax range having a parallax value of 73 to 83 at the 600 mm position taken by the first camera 301 is extracted from the integrated parallax map, and a mask at the in-focus position 600 mm is created. Further, a parallax range of parallax values 84 to 130 at a 450 mm position photographed by the second camera 302 is extracted from the integrated parallax map, and a mask at a focusing position of 450 mm is created.

  The omnifocal image synthesis is performed by extracting a mask range from an image photographed at each in-focus position using the created mask and pasting it on the base image. In addition, omnifocal image composition does not require parallax cancellation because the image captured with the same camera as the base image has no parallax, and the corresponding pixel of the base image can be replaced as it is, but it is captured with a camera different from the base image. Since the processed image has parallax, it is necessary to cancel the parallax value and replace the corresponding pixel of the base image.

  Below, a specific example is given and demonstrated about the omnifocal image composition in case the frequency | count of imaging | photography is an even number. As shown in FIG. 11, when the number of times of photographing is four, an image photographed at the in-focus position of 1000 mm by the first camera 301 is used as a base image, and a pixel of the base image with respect to an area overlapping the mask at the in-focus position of 450 mm. The value is replaced with the pixel value of the image captured at the focusing position of 450 mm. In addition, after canceling the parallax value for the region overlapping the mask at the focusing position 350 mm and 600 mm, the pixel value at the focusing position 1000 mm is changed to the pixel value of the image taken with the focusing position being 350 mm and 600 mm, respectively. Replace. An omnifocal image can be synthesized by performing the above processing.

  Below, a specific example is given and demonstrated about the omnifocal image composition when the frequency | count of imaging | photography is an odd number. When the number of times of photographing is three as shown in FIG. 12, an image photographed at the in-focus position of 1000 mm by the second camera 302 is used as a base image, and the pixel of the base image with respect to an area overlapping the mask at the in-focus position of 450 mm. The value is replaced with the pixel value of the image taken as the in-focus position 450 mm. Further, after canceling the parallax value for the area overlapping the mask at the focus position 600 mm, the pixel value at the focus position 1000 mm is replaced with the pixel value of the image taken with the focus position 600 mm. By performing the above processing, processing from mask generation to omnifocal image synthesis is performed.

  In the mask generation, the parallax range at each in-focus position is determined from the parallax value of the depth of field with reference to the camera image capturing the base image, and the remaining range is determined from one camera image. The reason for this is that if a mask with a wider range is created from an image taken with the same camera with no parallax with respect to the base image, the pixel value of the camera image that is affected by the parallax will be replaced during the synthesis process. This is because the amount is reduced and the breakdown of the composite image is reduced as a whole. In addition, the base image is taken as the in-focus position on the far-sighted subject position side, and the captured image at the in-focus position on the farthest far side tends to occupy a larger proportion of the image in the distant view than in the close-up view. Based on this, the amount of replacement of the pixel value of the camera image having the influence of parallax during image synthesis is reduced, and the failure of the composite image can be reduced.

In step S20, when the first camera 301 and the second camera 302 are focused on the same subject surface (D _A = D _B in FIG. 3), the focusing position of the first camera 301 is set to D _A. The captured image may be output as an omnifocal image.

In the present embodiment, the focal length, the shortest shooting distance MOD, and the like, which are the specifications of the camera, are shown by specific numerical values, but these values are not limited to the values shown here, and various values may be used depending on the design. Can take a value.
In addition, here, the subject position detection unit 303 limits the subject to a certain position, but this process is not necessarily performed. As described above, this allows you to optimize the number of shots by limiting the focus position for shooting, but for example, if you know in advance that the subject is distributed over a wide range, or start shooting immediately For example, if the foreground subject position is set to MOD and the foreground subject position is set to INF, shooting may be performed over the entire range.

  In the present embodiment, the in-focus photographing position and the number of photographing times are determined with reference to the table. However, instead of the table, the calculation formula shown in the equation (4) is held in the parameter storage unit 304 and detected. This can also be realized by determining the in-focus shooting position and the number of times of shooting each time according to the near-field subject position and the far-field subject position. In addition, although the blur tolerance is set to 2 pixels, the present invention is not limited to this value. The smaller the tolerance is, the more reliably the parallax is generated, and the generated omnifocal image is clearer. However, when the tolerance is increased, the interval between the in-focus positions is widened so that the number of photographing can be reduced. When the subject moves, the amount of blur due to the movement of the subject increases when shooting takes a long time. Therefore, a clearer omnifocal image may be acquired by increasing the tolerance and reducing the number of shootings. Therefore, it may be changed as appropriate according to the situation.

  In addition, here, the integrated parallax map 403 is created, and a mask is created to synthesize an omnifocal image. However, a plurality of parallax maps corresponding to each in-focus position can be used to accurately detect each distance. It is also possible to extract pixels from an image photographed at each in-focus position for a parallax range for which a particular parallax is obtained, and combine them with a base image to synthesize an omnifocal image.

  As described above, according to the present embodiment, by detecting the foreground subject position and the foreground subject position in advance, it is possible to prevent shooting in a range where no subject exists at the time of image shooting. It becomes possible to synthesize omnifocal images. In addition, by determining the shooting focus position and the number of times of shooting with reference to the information in the table storing the relationship between the focus position and the depth of field, the omnifocal image is synthesized with as few times as possible. be able to.

  The present invention is not construed as being limited by the above-described embodiments, and various modifications are possible within the scope of the matters described in the claims, and are included in the technical scope of the present invention. .

  The imaging apparatus of the present invention is not limited to an imaging apparatus such as a digital camera, and can be applied to an information processing apparatus incorporating an imaging function. For example, the present invention can also be applied to an information processing apparatus (for example, a computer, a mobile terminal, a tablet terminal, a smartphone, or the like) that includes at least a first camera and a second camera.

  Further, various functions of the imaging apparatus according to the present invention may be realized by software program codes. The program code may be temporarily stored in a storage medium during the processing, and the present invention may be realized by an arithmetic device such as a CPU reading and executing the program code as necessary. The storage medium refers to a storage device such as a portable medium such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, a memory built in a computer system, and a hard disk.

  In addition, a part or all of the imaging device in the above-described embodiment may be realized by a hardware configuration, and typically may be realized as an LSI that is an integrated circuit. Each functional block of the imaging apparatus may be individually formed into a chip, or a part or all of them may be integrated into a chip. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to progress in semiconductor technology, an integrated circuit based on the technology can also be used.

DESCRIPTION OF SYMBOLS 100 Imaging device 101 1st camera 102 2nd camera 103 Subject position detection part 104 Parameter storage part 105 Focus adjustment part 106 Position adjustment part 107 Parallax calculation part 108 Image composition part 300 Imaging apparatus 301 1st camera 302 2nd camera 303 Subject position Detection unit 304 Parameter storage unit 305 Focus adjustment unit 306 Position adjustment unit 307 Parallax calculation unit 308 Parallax integration unit 309 Image composition unit

Claims (9)

An imaging device comprising at least a first camera and a second camera for taking an image,
A subject position detector for detecting the subject position;
A focus adjusting unit that determines a shooting focus position of the first camera and a shooting focus position of the second camera according to the subject position detected by the subject position detection unit;
A parallax calculation unit that calculates parallax information from an image shot at the shooting focus position of the first camera and an image shot at the shooting focus position of the second camera;
An image combining unit that combines the image of the first camera and the image of the second camera based on the parallax information;
The focus adjustment unit determines an imaging focus position and the number of imaging times of the first camera and the second camera based on the subject position. 2. The imaging according to claim 1, wherein the focus adjustment unit determines in-focus shooting positions of the first camera and the second camera so that depths of field of the respective cameras overlap each other. apparatus. The subject position detection unit detects a foreground subject position and a foreground subject position by each of the first camera and the second camera;
The focus adjustment unit is configured such that a distance from the foreground subject position to the distant subject position is included in at least one depth of field of an image captured by the first camera and the second camera. The imaging apparatus according to claim 1, wherein the imaging focus positions and the number of imaging times of the first camera and the second camera are determined.   The focus adjustment unit may determine that the shooting focus position of the first camera and the shooting focus position of the second camera are alternated when the number of shootings is 3 or more. Item 4. The imaging device according to any one of Items 1 to 3. The parallax calculation unit
From a plurality of images photographed by the first camera and the second camera based on the photographing focus position and the number of photographing times, a plurality of images of the first camera and a second camera Select a combination
The imaging apparatus according to claim 1, wherein the parallax information is calculated for each of the plurality of combinations.   6. The imaging apparatus according to claim 1, wherein the image composition unit uses a base image used for composition as an image at a focus position on a distant subject side. The parallax calculation unit
The imaging apparatus according to claim 6, wherein a reference image for calculating parallax is an image captured by a camera that captured the base image. A parallax integration unit that integrates one piece of parallax information from a plurality of pieces of parallax information calculated by the parallax calculation unit;
The parallax integration unit obtains a histogram of parallax information from each of the plurality of parallax information, and integrates one piece of parallax information based on a distribution of the histogram. Imaging device. An imaging control method in an imaging apparatus including at least a first camera and a second camera for capturing an image,
A subject position detection step for detecting the subject position;
A focus adjustment step for determining a shooting focus position of the first camera and a shooting focus position of the second camera according to the subject position detected by the subject position detection step;
A parallax calculation step of calculating parallax information from an image shot at the shooting focus position of the first camera and an image shot at the shooting focus position of the second camera;
An image combining step of combining the image of the first camera and the image of the second camera based on the parallax information;
In the focus adjustment step, an imaging focus position and the number of times of imaging of the first camera and the second camera are determined based on the subject position.

Images