JP2022173069A - Image processing apparatus and method, and imaging apparatus and method for controlling the same, program, and storage medium - Google Patents

Abstract

To accurately acquire a distance image of the entire photographic scenes including a moving subject.SOLUTION: An image processing apparatus has: acquisition means that acquires a plurality of different viewpoint images obtained by photographing a single scene from different positions and at least one pair of pupil-divided parallax images having parallax; first creation means that creates a first distance image from the pair of parallax images; second creation means that creates a second distance image from the plurality of different viewpoint images; and integration means that integrates the first distance image and the second distance image to create an integrated distance image.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing apparatus and method, an imaging apparatus and its control method, a program, and a storage medium, and more particularly to technology for generating a distance image from an image obtained from an imaging apparatus.

2. Description of the Related Art Conventionally, there is known a technique of using a pair of pupil-divided images to obtain a distance image indicating the distance to each subject included in the images (Patent Document 1). In this technology, in addition to a stationary subject (hereinafter referred to as a "still subject"), a moving subject (hereinafter referred to as a "moving subject"), which is the main subject in many shooting scenes, is included. There is an advantage that a depth image of a scene can be obtained in one shot.

Compound-eye stereo cameras have long been used as distance-measuring cameras. A compound-eye stereo camera uses a plurality of cameras whose positional relationships are known, and calculates the distance to an object from the parallax of a commonly captured object and the baseline length between the cameras. However, in a compound eye camera with a short base line length, the parallax of an image with respect to a subject at a long distance is small, so the distance measurement accuracy is low.

On the other hand, in a monocular stereo camera, a single camera takes a plurality of images with different positions and postures, and calculates the relative distance to the subject from the parallax of the same subject imaged in those images. A monocular stereo camera can take a large baseline length, so it can accurately calculate the relative distance even for a distant subject, and using a large number of images from tens to hundreds increases the accuracy of distance measurement. be able to.

However, since the monocular stereo camera estimates the distance to the object and the position and orientation of the camera at the same time, the distance measurement value is not uniquely determined, and the obtained distance measurement value is a relative distance value rather than an absolute distance value.

Here, the relative distance value is a dimensionless quantity, and is defined, for example, as a ratio of the distance to a subject of interest when the distance to a specific subject is set to 1. On the other hand, the absolute distance value has a dimension of length, and is a value such as 3 [m].

Therefore, in a distance measuring device equipped with both a monocular stereo camera and a compound eye stereo camera, there is known a technique for converting relative distance values obtained by the monocular stereo camera into absolute distance values using information obtained by the compound eye stereo camera. ing. For example, in Patent Document 2, the absolute distance value of a subject is obtained by a compound eye stereo camera, the absolute position and orientation of the camera are calculated therefrom, and the absolute distance values of all subjects are obtained.

Japanese Patent No. 5192096 JP 2011-112507 A

Furukawa, Y. and Ponce, J., 2010. "Accurate, dense, and robust multiview stereopsis", IEEE Transactions Actions Pattern Analysis and Machine Intelligence, 32(8), pp.1362-1376. Abdullah Abuolaim and Michael S. Brown,"Defocus Deblurring Using Dual-Pixel Data",ECCV2020 Liyuan Pan, Shah Chowdhury, Richard Hartley, Miaomiao Liu, Hongguang Zhang, Hongdong Li,”Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration”,arxiv(CVPR2021) Abhijith Punnappurath, Abdullah Abuolaim, Mahmoud Afifi and Michael S. Brown,"Modeling Defocus-Disparity in Dual-Pixel Sensors",ICCP2020

However, the above-described conventional technology disclosed in Patent Document 1 has the following problems. That is, since the base line length between the main pixel and the sub-pixel for acquiring a pair of pupil-divided images is short, among the subjects in the image, there is no object at a short distance from the imaging device and from the in-focus distance. It is not possible to accurately calculate the distance to a subject outside the predetermined distance range. On the other hand, if the pupil is widened and the baseline length is lengthened in order to measure the distance to the subject with the configuration of Patent Document 1 with even a little accuracy, the depth of field becomes shallow, and the distance can be calculated this time. It narrows the range. Thus, when the depth range of the shooting scene is wide, it is difficult to obtain an accurate range image.

Further, according to the method described in Patent Document 2, when the absolute distance of the subject acquired by the compound-eye stereo camera is inaccurate, the absolute position and orientation of the camera cannot be calculated accurately. As a result, there is a problem that the absolute distance values of all subjects also become inaccurate.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and it is an object of the present invention to accurately acquire a range image of an entire scene including a moving subject.

Another object of the present invention is to accurately convert a relative distance value obtained by a monocular camera into an absolute distance value.

To achieve the above object, the image processing apparatus of the present invention includes acquisition means for acquiring a plurality of different viewpoint images of the same scene photographed from different positions, and at least a pair of parallax images having parallax due to pupil division. a first generation means for generating a first distance image from the pair of parallax images; a second generation means for generating a second distance image from the plurality of different viewpoint images; An image processing apparatus, comprising integration means for integrating the distance image and the second distance image to generate an integrated distance image.

According to the present invention, it is possible to accurately acquire a range image of an entire shooting scene including a moving subject.

Also, it is possible to accurately convert the relative distance value acquired by the monocular camera into the absolute distance value.

1 is a block diagram showing a schematic configuration of an imaging device according to a first embodiment of the present invention; FIG. FIG. 4 is a diagram for explaining an imaging unit according to the first embodiment; 4 is a flow chart showing the procedure of photographing and integrated range image generation in the first embodiment. 4 is a flowchart of single-shot distance image generation processing according to the first embodiment; FIG. 5 is a diagram for explaining a method of converting a defocus amount into a distance value; FIG. 4 is a diagram for explaining a method of generating a multi-view range image; FIG. 4 is a diagram showing an example of integration of a single-shot distance image and a multi-view distance image according to the first embodiment; FIG. 5 is a diagram showing another example of integration of a single-shot distance image and a multi-view distance image according to the first embodiment; 4A and 4B are diagrams for explaining examples of variations of imaging timings according to the first embodiment; FIG. 4A and 4B are diagrams for explaining an example of changing imaging conditions according to the first embodiment; FIG. FIG. 2 is a block diagram showing a schematic configuration of an image processing apparatus according to a second embodiment; FIG. 9A and 9B are diagrams for explaining selection of shots for single-shot distance image generation during post-processing in the second embodiment; FIG. The figure which shows the outline of the production|generation procedure of the integrated range image at the time of integrating a several single shot range image in a modification. FIG. 11 is a block diagram showing a schematic configuration of an image processing apparatus according to a third embodiment; FIG. 10 is a flowchart showing the procedure of photographing and integrated range image generation in the third embodiment; FIG. 11 is a diagram showing the input/output relationship of a network used for defocus recovery in the third embodiment; The figure which shows the outline|summary of the procedure from imaging|photography to integrated range image generation in 3rd Embodiment. FIG. 11 is a diagram showing an overview of procedures from multi-frame acquisition to generation of an integrated range image in moving image shooting according to the third embodiment; FIG. 11 is a block diagram showing the configuration of an imaging device according to a fourth embodiment of the present invention; FIG. 11 is an external front view of an imaging device according to a fourth embodiment; FIG. 11 is an external rear view of an imaging device according to a fourth embodiment; 10 is a flow chart showing the operation of distance measurement processing in the fourth embodiment; 21 is a flowchart showing detailed processing in S401 and S402 of FIG. 20; The figure which shows the mode selection screen of a display part. The figure which shows the display during distance measurement of a display part. The figure which shows the mode of feature point tracking with a monocular stereo camera. 21 is a flowchart showing detailed processing in S403 and S404 of FIG. 20; The figure which shows the mode of the window matching in a compound-eye stereo camera. FIG. 2 is a diagram showing the structure of a split-pupil imaging device and the principle of distance calculation;

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

<First embodiment>
FIG. 1 is a block diagram showing the schematic configuration of an imaging device 100 according to the first embodiment of the invention, showing only the components necessary for explaining the invention. The image capturing apparatus 100 includes an image capturing unit 101 including an optical system 1011 and an image sensor 1012, a memory 102, a viewing image generating unit 103, a single shooting range image generating unit 104, a multi-view range image generating unit 105, and a range image integrating unit 106. .

FIG. 2 is a diagram for explaining the imaging unit 101. As shown in FIG.
In FIG. 2A, an optical system 1011 is composed of a plurality of lenses, mirrors, and the like, and forms an image of reflected light from the subject 10 on the light receiving surface of an imaging device 1012 . In addition, in FIG. 2A, the optical system 1011 is represented by one lens. 202 is the optical axis of the optical system 1011 . The imaging device 1012 receives the optical image of the subject 10 formed by the optical system 1011, converts it into an electrical signal, and outputs the electrical signal.

As shown in FIG. 2B, the imaging device 1012 is covered with so-called Bayer array color filters 222 of red (R), green (G), and blue (B) arranged on the xy plane. A number of separated pixels 210 are juxtaposed. As a result, each pixel of the image sensor 1012 is given spectral characteristics according to the wavelength band transmitted by the color filter 222, and outputs signals mainly for red, green, and blue light.

FIG. 2C is a cross-sectional view showing the configuration of each pixel, which includes a microlens 211, a color filter 222, photoelectric conversion units 210a and 210b, and a waveguide 213. FIG. The substrate 224 is made of a material, such as silicon (Si), that absorbs light in the wavelength band that the color filter 222 transmits, and the photoelectric conversion units 210a and 210b are formed in at least a part of the internal region by ion implantation or the like. be done. Also, each pixel has wiring (not shown).

In the photoelectric conversion units 210a and 210b, a light beam 232a that has passed through a first pupil region 231a and a light beam 232b that have passed through a second pupil region 231b, which are different pupil regions of the exit pupil 230 of the optical system 1011, respectively. Incident. Thereby, the pupil-divided first signal and second signal can be obtained from each pixel 210 . From each pixel 210, the first signal and the second signal, which are the signals of the photoelectric conversion units 210a and 210b, may be read out independently. Alternatively, after reading the first signal, a signal obtained by adding the first signal and the second signal may be read, and the second signal may be obtained by subtracting the first signal from the added signal.

FIG. 2D is a diagram showing the arrangement of the photoelectric conversion units 210a and 210b corresponding to each microlens 211 on the image pickup device 1012 as seen from the incident direction of the optical axis. This is an example of when Note that the present invention is not limited to this, and pixels divided in the horizontal direction and pixels divided in the vertical direction may be arranged in a mixed manner. By arranging them in such a manner, it is possible to accurately obtain a defocus amount, which will be described later, not only for subjects whose luminance distribution changes in the horizontal direction, but also for subjects whose luminance distribution changes in the vertical direction. In addition, three or more photoelectric conversion units may be configured for each microlens 211, and FIG. 210c-210f.

The first signal and the second signal obtained from the photoelectric conversion units 210a and 210b are sent to the arithmetic processing unit 204 included in the imaging unit 101 and converted into electronic information. When the signal acquired by photoelectric conversion is analog, the arithmetic processing unit 204 performs noise removal by correlated double sampling (CDS), exposure control by gain increase by auto gain control (AGC), black level correction, A/ Basic processing such as D conversion is performed to obtain an image signal converted into a digital signal. These arithmetic processes are generally called AFE (analog front-end) processes because they mainly preprocess analog signals. Also, when used in combination with a digital output sensor, it is sometimes called DFE (digital front end) processing.

Then, an A image that collects the first signals output from the plurality of pixels 210 of the image sensor 1012 and a B image that collects the second signals output from the plurality of pixels 210 are generated. Since the A image and the B image are images having parallax with each other, they are hereinafter referred to as a "parallax image", and the A image and the B image are collectively referred to as a "parallax image pair".

The arithmetic processing unit 204 also performs Bayer array interpolation and the like when the imaging element 1012 is a color sensor. Further, in order to improve the quality of the parallax image and the quality of the later-described viewing image output together with the parallax image pair, filtering processing such as low-pass and high-pass filtering and sharpening processing may be performed. Further, various processing such as gradation correction including dynamic range expansion such as HDR (high dynamic range) processing, and color tone correction such as WB (white balance) correction may be performed. Note that the processing of the arithmetic processing unit 204 tends to be integrated with the image sensor 1012 at the chip level or unit level, so it is not shown in FIG.

In this way, by forming a plurality of photoelectric conversion units under each of the plurality of microlenses 211 formed on the light receiving surface of the image pickup device 1012, the plurality of photoelectric conversion units are arranged in different pupil regions of the optical system 1011. receives the subject light flux that has passed through the As a result, even if the optical system 1011 has one aperture, it is possible to obtain a pair of parallax images in one shot.
The generated parallax image pair is temporarily stored in the memory 102 .

Returning to FIG. 1, the viewing image generation unit 103 receives the parallax image pair obtained by one shooting stored in the memory 102, and adds the first signal and the second signal for each pixel. one image is generated. That is, an image for viewing (hereinafter referred to as a “viewing image”) is generated for an image formed by the light flux that has passed through the entire pupil region of the optical system 1011 .

Note that when the detection of the focus state using the parallax image and the generation of the distance image are not performed, the viewing image is generated by the arithmetic processing unit 204 integrated with the image sensor 1012 at the chip level, or the Alternatively, the first signal and the second signal may be added and then read out. In that case, it is possible to contribute to saving the transmission band and shortening the time required for reading. Also, when there is no need for a viewing image that is finally paired with a range image, the viewing image generator 103 may not explicitly exist and may be included in the multi-view range image generator 105 .

A pair of parallax images obtained by one shooting stored in the memory 102 is transmitted to the single shooting range image generation unit 104 . At that time, conversion to a luminance image may be performed. Then, correspondence between parallax images of the input parallax image pair is performed, and camera parameters such as the focal length and aperture value defined from the zoom state of the optical system 1011 and image sensor information such as the pixel pitch of the image sensor 1012 are included. A distance image is generated based on the imaging information. Hereinafter, the distance image generated by the single-shot distance image generation unit 104 will be referred to as a "single-shot distance image".

In the multi-view range image generation unit 105, the same scene is continuously shot multiple times from different positions, that is, a plurality of parallax image pairs obtained by multi-shots are converted into one image for each shot in the viewing image generation unit 103. The obtained viewing image is used as an input, and a plurality of viewing images (images from different viewpoints) are used to generate a distance image. The distance image generated by the multi-view distance image generation unit 105 is hereinafter referred to as a "multi-view distance image". In the case of a camera whose imaging device moves, parallax occurs between viewing images obtained by performing multi-shots obtained in time series. A multi-view range image can be calculated from the parallax between images).

A distance image integration unit 106 integrates the single-shot distance image and the multi-view distance image to generate an integrated distance image.

Next, with reference to the flowchart of FIG. 3, the procedure of photographing and integrated range image generation in this embodiment will be described.
In S101, shooting for single-view distance image generation (single shooting) and shooting for multi-view distance image generation (continuous shooting) are performed. Note that the shooting order will be described later with reference to FIG.

式（１）において、Ｓ（ｒ）は像シフト量ｒにおける２つの像の間の相関度を示す相関値、ｉは画素番号、ｒは２つの像の相対的な像シフト量である。ｐ及びｑは、相関値Ｓ（ｒ）の算出に用いる対象画素範囲を示している。相関値Ｓ（ｒ）の極小値を与える像シフト量ｒを求めることで像ズレ量を算出することができる。 In S102, distance measurement processing is performed using the pair of parallax images obtained by single shooting, and a single shooting distance image is calculated. FIG. 4 is a flow chart of the single-shot distance image generation processing performed in S102. In S201 of FIG. 4, an image displacement amount, which is a relative positional displacement amount between parallax images, is calculated. A known method can be used to calculate the amount of image shift. For example, using the following equation (1), the correlation value is calculated from the signal data A(i) and B(i) of the A image and the B image.

In equation (1), S(r) is a correlation value indicating the degree of correlation between two images at the image shift amount r, i is the pixel number, and r is the relative image shift amount of the two images. p and q indicate the target pixel range used to calculate the correlation value S(r). The image shift amount can be calculated by obtaining the image shift amount r that gives the minimum value of the correlation value S(r).

Note that the method for calculating the image shift amount is not limited to the method described above, and other known methods may be used.

Next, in S202, a defocus amount is calculated from the image shift amount calculated in S201. An image of the subject 10 is formed on the imaging element 1012 via the optical system 1011 . Note that the example shown in FIG. 2A described above shows a defocused state in which the luminous flux that has passed through the exit pupil 230 is focused on the imaging plane 207 . This defocus state is a state in which the imaging surface 207 and the imaging surface (light receiving surface) of the imaging device 1012 do not match and are deviated in the direction of the optical axis 202. and the imaging plane 207 .

Here, an example of a conversion method from a defocus amount to a distance value will be described using the simplified optical layout diagram of the imaging apparatus shown in FIG.

FIG. 5 shows light rays 232 in a state in which the image of the object 10 is defocused with respect to the image sensor 1012, 202 is the optical axis, 208 is the aperture stop, 205 is the front principal point, 206 is the rear principal point, and 207 is 1 shows the imaging plane. In addition, d is the amount of image shift, W is the base length, D is the distance between the image sensor 1012 and the exit pupil 230, Z is the distance between the front principal point 205 of the optical system 1011 and the subject 10, and L is the image captured by the image sensor 1012. A distance ΔL between the surface and the rear principal point 206 is a defocus amount.

式（２）は、比例係数Ｋを用いて、式（３）のように簡略化して書くことができる。 In the imaging apparatus of this embodiment, the distance of the subject 10 is detected based on the defocus amount ΔL. An image shift amount d representing a relative position shift between an A image based on the first signal obtained from the photoelectric conversion unit 210a of each pixel 210 and a B image based on the second signal obtained from the photoelectric conversion unit 210b. and the defocus amount .DELTA.L have the relationship shown in Equation (2).

Equation (2) can be simplified using a proportionality factor K to be written as Equation (3).

ΔL≈K・d (3)
A coefficient for converting an image shift amount into a defocus amount is hereinafter referred to as a “conversion coefficient”. The conversion factor is, for example, the proportionality factor K or the baseline length W shown in Equation (3). Henceforth, the correction of the base length W is equivalent to the correction of the conversion factor. Note that the method of calculating the defocus amount is not limited to the method of this embodiment, and other known methods may be used.

入力した複数の視差画像間、例えばＡ像とＢ像間の例えば全ての画素についてデフォーカス量を求めることで、デフォーカスマップを算出することができる。デフォーカスマップを式（４）の関係を用いて変換することで、対応する単写距離画像を算出することができる。 Furthermore, the conversion from the defocus amount to the subject distance may be performed using the following formula (4) showing the image forming relationship between the optical system 1011 and the imaging device 1012 . Alternatively, the image shift amount may be directly converted into the subject distance using a conversion coefficient. Note that f is the focal length in Equation (4).

A defocus map can be calculated by obtaining the defocus amount for all pixels, for example, between a plurality of input parallax images, for example, between the A image and the B image. By converting the defocus map using the relationship of Expression (4), the corresponding single-shot distance image can be calculated.

According to the procedure of distance calculation processing as described above, in a split-pupil imaging system, a single-shot distance image can be calculated from a pair of parallax images obtained in one shot.

Returning to FIG. 3, in the next step S103, the viewing image generation unit 103 inputs the viewing image corresponding to the light passing through the entire pupil region, which is generated from the pair of parallax images obtained for each shot of continuous shooting. To generate a multi-view distance image from a plurality of viewing images. It should be noted that, on the premise that shooting is being carried out while moving by hand or the like, a three-dimensional scene is reconstructed from a plurality of viewing images with overlapping fields of view, and in the process a range image for each shot is generated.

The relative position and orientation between shots of the imaging device 100 during continuous shooting can be determined by orientation sensors such as a gyro sensor, an acceleration sensor, and a tilt sensor, which are standardly attached to the imaging device 100 in recent years, and recent standards for imaging devices. It can be obtained by well-known camera posture estimation using shake detection in the anti-shake function, image vector, and the like. Since the method of obtaining the position and orientation is known, the description is omitted here.

If changes in the position and orientation of the imaging device 100 between shots in continuous shooting are known, estimating the distance of the subject in the image due to the epipolar geometric constraint as shown in FIG. 6A is a simple one-dimensional search problem. can be replaced with In addition, when it is possible to track the points of a subject in the same space between a plurality of viewing images over a plurality of frames, this epipolar search is performed as a multi-baseline search for finding the same distance between a plurality of different sets of viewing images. can be replaced with By using this, it is possible to obtain the distance with high accuracy. For example, by adding one shot as shown in FIG. 6(b), distance image estimation for the reference viewpoint C1 in the first shot can be performed not only between shots at viewpoint C2 but also between shots at viewpoint C3. becomes.

By increasing the number of related shots in this way, it becomes possible to perform multiple searches for the depth value of each point in the image of the shot set as the reference image. This enables robust and highly accurate distance value estimation. On the other hand, the moving subject is not calculated in the multi-view range image because the epipolar geometry is not established between the correspondences between shots.

There are various known correspondences between shots. For example, a method called PMVS, etc., in which a patch-based correspondence is performed and a distance value is calculated in consideration of the normal direction (Non-Patent Document 1), or a virtual depth plane called plane sweep is set for each shot. It includes a method of matching by back projection from . Generating a multi-view range image is also called a multi-view stereo method or the like. With the above-described method, a multi-view range image corresponding to the viewing image of the shot selected as the reference image can be obtained from the viewing image group, which is an aggregated image of the light images from all the pupil areas continuously photographed.

In S104, the single-view range image and the multi-view range image are integrated to generate a highly accurate integrated range image for the entire depth direction of the scene.

As described above, the single-view distance image includes distance information on the still subject area and the moving subject area, but the multi-view distance image includes only distance information on the still subject area. This is because the matching of the moving subject areas does not satisfy the epipolar geometry when generating the multi-view range image. Therefore, in integrating distance images according to the present embodiment, only the distance values of the moving subject area from the single-shot distance image are used, and the distance information of overlapping still subject areas is acquired from the multi-view distance image and integrated.

When the single-shot distance image and the multi-view distance image are superimposed, the area where both distance information exists is the static subject area, the distance information exists only in the single-shot distance image, and the uncalculated area or distance information exists in the multi-view distance image. A region with a low reliability even if there is a , can be regarded as a moving subject region. Therefore, by obtaining and integrating the distance information of the still subject area from the multi-view distance image and the distance information of the moving subject area from the single-shot distance image, an integrated distance image with high accuracy of distance information is obtained.

FIG. 7 shows an example of integrating a single-view distance image and a multi-view distance image. 7A is a viewing image corresponding to a single-shot range image, FIG. 7B is a single-shot range image, FIG. 7C is a corresponding multi-view range image, and FIG. 7D is an integrated range image. In the single-shot distance image of FIG. 7(b), there is distance information of the human area, which is a moving subject, but in the multi-view distance image of FIG. It is white because it does not exist. Therefore, the distance information of the moving subject area is obtained from the single-shot distance image of FIG. 7B, and the distance information of the overlapping still subject area is obtained from the multi-view distance image of FIG. to generate an integrated range image of

Further, as described above, as a characteristic of the pupil division imaging system, the distance to an object that is not at a short distance from the imaging device and is outside the predetermined distance range from the in-focus distance is determined by the baseline length of the pupil division optical system. Since it is short, there is a problem that it cannot be calculated accurately. To solve this problem, based on the distance information of the single-shooting distance image, the distance information of the neighboring area and the area within the predetermined distance range of the in-focus distance among the moving subject area and the still subject area is acquired from the single-shooting distance image. However, the integrated range image may be generated by acquiring the range information of other still subject areas from the multi-view range image.

FIG. 8 shows an example of determining from where to obtain the distance value of the integrated distance image based on the distance value of the single shot distance image. u0 is the focus distance, v1 and v2 are predetermined distance ranges from the focus distance u0, v1 is the front focus side, and v2 is the rear focus side. Also, u1 is a distance (threshold value) indicating a neighboring area from the imaging device 100 . Let u be the distance value at each position in the range image,
u≦u1 (5)
-v1≤u-u0≤v2 (6)
The distance value (distance information) is obtained from the single-view range image for the area that satisfies at least one of the formulas (5) and (6), and from the multi-view range image for the other areas. As a result, it is possible to acquire a range image with higher precision for the entire scene in the depth direction than only the single-shot range image.

Also, in the multi-view distance image, the distance information of the moving subject area is omitted as an uncalculated area, and the distance information of the still object area other than that is overlapped with the same area of the single-shot distance image, but in the distance image, The boundary between the moving subject area and the still subject area tends to be blurred. Therefore, moving subject area detection using a viewing image as an input may be used together with area boundaries used to extract distance information from each distance image. For example, a CNN network may be used in which a face or a human body is detected and its area is cut out, or prior knowledge is also used to improve the accuracy of cutting out a moving subject area through learning or the like.

Note that the distance information (pixels) forming the distance image is not necessarily limited to the subject distance (distance value), and may be an image shift amount or defocus amount before conversion to the subject distance. Furthermore, the reciprocal of the subject distance (reverse distance value) may be used as the distance information (pixels) forming the distance image.

Also, in the single-shot and continuous-shot shooting in S101, single-shot shooting and continuous-shot shooting can be performed in various variations. That is, the shooting for obtaining the single-shot distance image may be performed as one shot of continuous shooting, or may be performed as single-shot shooting. Also, the continuous shooting may be performed before or after the single shooting for obtaining the single shooting range image. Further, when the amount of movement of the imaging apparatus 100 during continuous shooting is small and the base line length cannot be obtained, shooting may be performed later.

FIG. 9 shows an example of variation of shooting timing. FIG. 9A shows an example in which continuous shooting is first performed as pre-shooting to acquire a plurality of images, and then single shooting is performed to obtain a single shooting distance image. In FIG. 9, SW1 represents a state in which the shutter of the imaging apparatus 100 is half-pressed, and SW2 represents a state in which the shutter is fully-pressed. Pre-shooting is a shooting method that utilizes a circular buffer, and it is a general function that is standardly installed not only in devices dedicated to shooting such as cameras, but also in smartphone camera functions.

In this photographing method, an image before the shutter full-press SW2 is temporarily stored in advance, and when the shutter full-press SW2 is pressed, the user is allowed to select a true best shot, or continuous shooting for a certain period of time is performed for post-processing. save the number. This pre-captured image is obtained as an input for generating a multi-view range image. At this time, since the pre-photographed image is used only for generating the multi-view distance image, the signals of the photoelectric conversion units 210a and 210b may be added for each pixel at the time of photographing, and the image data capacity may be reduced and saved as a viewing image. good. Images obtained when the shutter is fully pressed SW2 are stored as a parallax image pair and used to generate a single-shot distance image. can be In that case, a multi-view range image is generated using this image as a reference.

FIG. 9(b) shows an example in which continuous shooting is performed after shooting for obtaining a single shooting distance image. This is a photographing method in which continuous photographing is performed for a certain period of time even after the shutter switch SW2 of the image pickup apparatus 100 is fully depressed, and the images are stored. The best shot of a moving subject is photographed when the shutter is fully pressed SW2. Then, even after the shutter is released, shooting is continued to obtain an image for generating a multi-view distance image. Since the post-shooting image after releasing the shutter is used only for generating the multi-view distance image, it may be added for each pixel at the time of shooting as in the case of pre-shooting, and saved as a viewing image with a reduced storage capacity. . However, the image obtained when the shutter is fully pressed SW2 is stored as a pair of parallax images and used to generate a single-shot distance image. Note that this shot may also be added and used to generate the multi-view range image. In that case, a multi-view range image is generated using this image as a reference.

FIG. 9C shows a case where pre-photographing and post-photographing are performed before and after the shutter is fully depressed SW2. Images obtained when the shutter is not fully pressed SW2 are obtained from the circular buffer. Since these images are used only for generating a multi-view distance image, they may be added for each pixel at the time of shooting and stored as a viewing image with reduced storage capacity. An image obtained when the shutter is fully pressed SW2 is stored as a pair of parallax images and used to generate a single-shot distance image. Also good. In that case, a multi-view range image is generated using this image as a reference.

In addition, when the photographer intentionally selects in advance the best shot that he/she wants to obtain as a set of the viewing image and the distance image by operating the shutter of the imaging device, etc., before the distance image is generated, the imaging condition is set for single-shot distance image generation. , and may be explicitly switched for multi-view range image generation. For example, as shown in FIG. 10, in continuous shooting for multi-view range image generation, the Av (aperture) value is set large, the aperture is narrowed down, and pan-focusing is performed so that feature points can be easily captured in the entire depth direction of the scene. . On the other hand, it is conceivable to change the Av value so as to be smaller only when the shutter is fully pressed SW2 so that the pupil is opened and the distance measurement in the single-shot photographing can be accurately performed near the in-focus distance. Following the Av value, the Tv value and the ISO value are appropriately set according to a program table or the like.

Furthermore, since only the distance value is actually integrated finally, there is no need to align the Ev (exposure) value, which is the total value of the Av value, Tv value, and ISO. Therefore, as shown in FIG. 10B, the Ev value of the photographing conditions may be greatly changed between imaging for single-view distance image generation and multi-view distance image generation. FIG. 10(b) is an example in which different Ev values are used for single-view range image generation and multi-view range image generation. For example, if such photographing conditions are adopted for photographing in a dark place, the scenes in which the technique of the present invention can be used are expanded.

For example, in continuous shooting, it is desired to perform pan-focus shooting, but also to suppress motion blur. On the other hand, in single-shot shooting, the Ev value can be increased by increasing the Av value. and Even if the images are captured in this way, the captured images themselves are not integrated, so problems are less likely to occur. When an image obtained by single-shot shooting is used to generate a multi-view distance image, the gradation scale of the pixel value of the Ev value difference image is corrected and adjusted. Again, this is less likely to affect the final integrated range image because it is only indirectly related.

As described above, according to the first embodiment, by integrating the single-shot distance image and the multi-view distance image so as to compensate for their shortcomings, the distance image of the entire shooting scene including the moving subject can be obtained. It can be obtained with high accuracy.

Although the integrated distance is described as "image", the format of the output information may not be an image. For example, a 2.5-dimensional stereoscopic image obtained by projecting a distance image onto a three-dimensional space according to the distance value, point clouds or volume data in different storage formats, or stereoscopic data converted into mesh information may be used.

<Second embodiment>
Next, a second embodiment of the invention will be described. FIG. 11 is a block diagram showing an image processing device 200 according to the second embodiment.

When the image input unit 201 obtains a parallax image pair captured by an external imaging device (not shown) and a viewing image obtained by adding and converting a parallax image pair for each pixel in the imaging device, the viewing image is obtained. is input, and the input parallax image pair and viewing image are stored in the memory 102 . Then, the input parallax image pair and viewing image are processed as described in the first embodiment to obtain an integrated range image. Since the configuration other than the image input unit 201 is the same as that shown in FIG. 1, the same reference numerals are given and the description is omitted here.

As shown in FIG. 12A, when parallax image pairs for all shots are input, a parallax image pair used for generating a single-view distance image and a viewing image serving as a reference when generating a multi-view distance image and can be selected. Therefore, from a plurality of parallax image pairs obtained by continuous shooting, a parallax image pair to be used for generating a single shot distance image and a reference viewing image are selected after the fact or by a predetermined algorithm. For example, it may be set in advance so that the parallax image pair obtained in the last shot of all shots is selected. Since such a case is assumed to occur particularly in video shooting, images obtained from shots corresponding to each frame or shots at intervals of several frames are always acquired as parallax image pairs during video shooting.

Alternatively, as shown in FIG. 12B, the single-view distance image and the multi-view distance image may be generated by sequentially moving the parallax image pair to be selected. By doing so, it is possible to obtain an integrated range image of the entire scene with high accuracy even for a wide-depth scene by integrating the single-view range image and the multi-view range image for all pairs of parallax images. can be done.

As described above, according to the second embodiment, an image processing apparatus can obtain a highly accurate range image using a parallax image pair obtained from an imaging apparatus.

<Modification>
In the description of the first and second embodiments above, it is assumed that the shooting of the parallax image pair for generating the single-shot distance image is a typical single-shot shooting. However, when the subject moves slowly, a plurality of pairs of parallax images are captured, and the single-shot distance image generation unit 104 generates single-shot distance images and integrates them to improve the quality of the single-shot distance image. After that, it may be integrated with the multi-view range image.

With reference to FIG. 13, a flow of generating single-shot distance images from a plurality of pairs of parallax images, integrating them, enhancing the quality of the single-shot distance images, and then integrating them with the multi-view distance image will be described. For example, if the position or posture of the imaging device changes between shots, the viewpoint of the single-shot distance image is changed three-dimensionally, and the single-shot distance image is converted to a single-shot distance image that matches the coordinates at the time of the reference shot. Integrate.

For example, in FIG. 13, of the three parallax image pairs, the central parallax image pair is set as a reference, and the single shooting distance images obtained from the remaining parallax image pairs are used as the shooting position or the shooting position of the parallax image pair. Change the viewpoint according to the posture. Then, the two single-shot distance images after viewpoint conversion and the reference single-shot distance image are integrated, and the integrated single-shot distance image is integrated with the multi-view distance image to obtain a final integrated distance image.

As described above, according to the modified example, it is possible to obtain a more accurate distance image.

<Third Embodiment>
Next, a third embodiment of the invention will be described.
In the above-described first and second embodiments, the single shooting distance image generated from the pair of parallax images obtained in one shot with the split-pupil imaging system and the multiple distance images obtained by performing multi-shots in time series. A case of generating an integrated range image by integrating the multi-view range image generated from the viewing image has been described. However, if a pair of parallax images captured by a split-pupil imaging system is used as an input and a single shooting range image is to be accurately obtained, it is necessary to widen the pupil and lengthen the baseline length. On the other hand, when the pupil is widened, the depth of field becomes shallower, and subjects outside the predetermined distance range from the in-focus position become blurred, making it difficult to associate the viewing images. Therefore, it becomes impossible to accurately calculate the distance to the subject outside the predetermined distance range of the in-focus position using the viewing image.

Thus, in principle, there is a limit to the shooting scenes and shooting conditions under which both highly accurate single-view distance images and multi-view distance images can be obtained. For example, under dark environments, selectable shooting conditions become more severe, and it becomes more difficult to accurately obtain both single-shot range images and multi-view range images. This is because in dark places, it is necessary to keep the shutter speed short in order to prevent motion blur, but since the amount of light is insufficient, it is necessary to open the pupil of the optical system and decrease the F-number to increase the amount of light. be. As described above, when the pupil of the optical system is opened, the depth of field becomes shallow, so the depth range in which the multi-view distance image can be generated with high accuracy is narrowed.

Therefore, in the third embodiment, even in multi-shot shooting other than the main shooting in which the parallax image pair is acquired by the pupil division imaging system, the shooting is performed so that the parallax image pair can be acquired. Then, using the parallax image pair of each shot, the defocusing of the image area outside the predetermined distance range from the in-focus position is restored to generate a viewing image with an expanded depth of field, which is obtained by multi-shot photography. To associate a plurality of viewing images with no defocus. In addition, by integrating the single-shooting distance image and the multi-view distance image to generate an integrated distance image, the accuracy of the entire scene, including the distance from the in-focus position to the subject outside the predetermined distance range, can be improved. to obtain a high integrated range image of .

FIG. 14 is a block diagram showing an image processing device 300 according to the third embodiment. In FIG. 14, the same reference numerals are given to the same components as those of the image processing apparatus 200 described in the second embodiment with reference to FIG. 11, and the description thereof will be omitted.

The viewing image generation unit 303 according to the third embodiment performs a process of generating a viewing image as it is from an input parallax image pair and a defocus recovery process for the input parallax image pair in the same manner as the viewing image generation unit 103 . A process of generating a viewing image can be performed after execution. That is, defocus blur recovery may be applied to all of the plurality of viewing images input to the multi-view range image generation unit 105, or may be applied to only some of them. Therefore, the images to be input to the viewing image generation unit 303 may all be parallax image pairs, or the parallax image pairs and the first signal and the second signal are added for each pixel before being output. It is also possible to mix the displayed viewing image.

Next, the procedure of photographing and integrated range image generation in the third embodiment will be described with reference to the flowchart of FIG. 15 .

In S301, a pair of parallax images used for generating a single shooting range image and recovering out-of-focus blur is continuously photographed a plurality of times (multi-shot). When generating a single shooting distance image and recovering the out-of-focus blur for all images obtained by performing multi-shots, a set of parallax images ( obtained as a parallax image pair). If the generation of single-shot distance images and recovery of defocus blur are not performed for some images obtained by performing multi-shot, the generated viewing images may be mixed. If a plurality of images are to be captured by video shooting instead of multi-shot, the control is simpler to acquire parallax image pairs in every frame.

In S302, a shot used as a reference for generating an integrated range image is selected from a plurality of images obtained by performing multi-shot. In the case of continuous shooting of still images, selection may be made in advance using the GUI of the imaging device at the time of shooting. Alternatively, a reference shot may be selected using a GUI (not shown) of the image processing apparatus 300 . In the case of moving image shooting, the reference frame for generating the integrated range image is sequentially moved in the time direction.

In S303, a pair of parallax images corresponding to the number of viewpoints under the microlens 211 photographed in each shot, or a pair of parallax images consisting of a part of them, is input, and one viewing image subjected to defocus recovery is generated.

Defocus recovery processing for the parallax image pair of each shot may be realized by deconvolution processing or MAP estimation processing performed by calculating blinds or range images and estimating a defocus kernel. Alternatively, it may be realized by a deep learning process that replaces it. When implementing defocus blur recovery processing for a parallax image pair of each shot by deep learning processing, implementation by end-to-end processing using an encoder-decoder structure is first considered. Alternatively, a network corresponding to deconvolution processing performed by estimating a conventional non-blind range image or focus blur kernel is constructed and realized. Each process will be described below.

First, an implementation method by end-to-end processing using an encoder-decoder structure will be described. FIG. 16 is a diagram showing the input/output relationship of the network used for the defocus recovery process. FIG. 16(a) is an input/output example of a network that performs end-to-end defocus recovery. Regarding this approach, a system for realizing a plurality of networks has been proposed in the form of Non-Patent Document 2. Note that any approach may be used as long as a parallax image pair (A image, B image) of each shot is input and a defocus recovery image is obtained. For the encoder-decoder structure network, the network is learned using all-in-focus images taken with the aperture of the imaging device narrowed down as training data. The network is trained using the difference between the output image and the ground truth all-in-focus image as the loss. When the learning is completed, the pair of parallax images of each shot is used as an input, and a single viewing image that has undergone defocus recovery is obtained.

Next, an example of an implementation method for constructing a network corresponding to deconvolution processing performed by estimating a range image or a focus blur kernel will be described. FIG. 16(b) shows an example of a network. A network receives a parallax image pair of each shot as an input and calculates a temporary distance image or a temporary inverse depth image with the distance value d set to 1/d, and a parallax image pair and this temporary distance image or temporary inverse depth image. It consists of a network that outputs a defocused image and a refined range image or an inverse depth image as input. The provisional distance image or the provisional inverse depth image may be a map of parameters representing a focus blur kernel or an image thereof. Note that FIG. 16B illustrates a case where there are two parallax images (A image and B image) for each shot. The distance calculation network and the defocus recovery network can be learned independently. The network is learned by using the correct image of the distance value corresponding to the parallax image pair and the omnifocal image captured by narrowing down the aperture of the imaging device as training data. This learning is performed by performing backpropagation using distance value error, pixel value error, edge definition, etc. as an error function. As such a network, the one disclosed in Non-Patent Document 3 has been proposed. Details of the network, details of learning data, examples of error functions, etc. can be realized by referring to Non-Patent Document 3.

Although the distance calculation network and the defocus recovery network were illustrated with reference to the network configuration of Non-Patent Document 3, the present invention is not limited to the specific network configurations illustrated. Each can be replaced by other deep learning networks or classical methods. For example, the distance calculation network may be a deep learning network for estimating the model parameters of the focus blur kernel disclosed in Non-Patent Document 4, as shown in FIG. 16(c). Also, the defocus recovery network can replace the shift-variant deconvolution process of the classical process.

Non-Patent Document 4 assumes a model of a focus blur kernel, finds the parameters of the focus blur kernel for each angle of view for the input parallax image pair of each shot, creates a parallax map, and outputs it as a substitute for the distance image. is.

In S304, a single-shot distance image is generated, but since this process is the same as the process in S102 of FIG. 3 of the first embodiment, description thereof will be omitted.

In S305, a multi-view distance image is generated using a plurality of viewing images acquired by multi-shot, which include viewing images with or without defocus recovery in S303. Note that the method of generating the multi-view range image is the same as the process in S103 described in the first embodiment, so description thereof will be omitted.

In S306, the single-view range image and the multi-view range image are integrated to generate a highly accurate integrated range image for the entire depth direction of the scene. Note that the method of generating the integrated range image is the same as the processing in S104 described in the first embodiment, so description thereof will be omitted.

Next, with reference to FIG. 17, the procedure from photographing to integrated range image generation will be outlined. A case of acquiring all images as a pair of parallax images in multi-shot will be described. For example, assuming that a still camera is used for photographing, in S301, the photographer may fully depress the shutter to intentionally photograph. After that, in S302, a reference parallax image pair (reference shot) is selected from a plurality of parallax image pairs obtained by performing multi-shot. Note that the parallax image pair to be selected may be the same as the parallax image pair of the shot used to generate the single-shot distance image. In S303, defocus blur recovery processing is performed on the parallax image pair for generating the viewing image used to generate the multi-view range image.

In S304, the selected parallax image pair is used to generate a single-shot distance image. Also, in S305, a multi-view range image is generated from a plurality of viewing images obtained by performing defocus recovery processing. In S306, the single-shot range image and the multi-view range image are integrated to obtain an integrated range image. The obtained integrated distance image is a distance image with higher distance accuracy than the single-shooting distance image for the entire scene including the area outside the predetermined distance range from the in-focus position. Also, unlike the multi-view range image, a range value can be obtained even if a moving object is included.

FIG. 18 is an explanatory diagram for a case where it is desired to generate an integrated range image for all shots obtained by multi-shots, or for a case where an integrated range image for all frames in moving image shooting is desired to be generated. In this case, by sequentially changing the selection of the reference shots in S302, integrated range images corresponding to all shots are generated.

As described above, according to the third embodiment, it is possible to obtain highly accurate range images in various shooting scenes.

<Fourth Embodiment>
Next, a fourth embodiment of the invention will be described.
FIG. 19A is a block diagram showing the configuration of the imaging device according to the fourth embodiment of the present invention. In FIG. 19A , imaging device 400 includes imaging unit 401 , imaging unit 402 , arithmetic unit 403 , storage unit 404 , shutter button 405 , operation unit 406 , display unit 407 , and control unit 408 .

FIG. 19B is an external front view of the imaging device 400 of this embodiment. The imaging device 400 has two imaging units, an imaging unit 401 and an imaging unit 402 . The imaging unit 401 includes a lens 401a and an imaging device 401b. The imaging unit 402 includes a lens 402a and an imaging device 402b. Lenses 401a and 402a converge the light reflected by the subject and form optical images on imaging elements 401b and 402b. The imaging devices 401b and 402b convert optical images into electrical signals and output video data.

The imaging unit 401 and the imaging unit 402 are arranged so that they can photograph a common subject and obtain images with a mutual parallax. When the user of the imaging apparatus 400 presses the shutter button 405, the imaging units 401 and 402 perform compound-eye stereo imaging. It is assumed that the distance D between the optical axis of the imaging unit 401 and the optical axis of the imaging unit 402 is known.

FIG. 19C is an external rear view of the imaging device 400 of this embodiment. An operation unit 406 is pressed to set imaging conditions and to instruct the start and end of the distance measurement mode. A display unit 407 is, for example, a liquid crystal display, and displays the composition at the time of shooting and various setting items. When the display unit 407 is a touch panel, the shutter button 405 and the operation unit 406 can also be used, and the user can perform shooting and settings by touching the display unit 407 . In this case, hardware parts such as the shutter button 405 and the operation unit 406 are not required, and the user can easily operate the camera, and the large display unit 407 can be installed for easy viewing.

The control unit 408 controls shooting conditions when the imaging unit 401 and the imaging unit 402 perform shooting. For example, it controls the aperture diameter of the diaphragm of the optical system, the shutter speed, the ISO sensitivity, and the like. The calculation unit 403 performs development processing on the images captured by the imaging unit 401 and the imaging unit 402, and calculates the subject distance.

FIG. 20 is a flowchart showing the operation of distance measurement processing in the imaging device 400 of this embodiment. When the user instructs the start of the distance measurement mode in S401, the operation of this flow chart is started.

FIG. 21(a)A is a flowchart showing detailed processing in S401. S401 consists of the processing of S4011 and S4012.

In S4011, the user selects the distance measurement mode. FIG. 22 shows the state of the display section 407 when the distance measurement mode is selected. The user causes the display unit 407 to display the display 501 by operating the operation unit 406 . A display 501 shows items of shooting mode, and the user operates the operation unit 406 to select the distance measurement mode. In S4012, the control unit 408 of the imaging device 400 starts continuous shooting.

Returning to FIG. 20, in S402, the relative distance is acquired by the monocular stereo camera. Either the imaging unit 401 or the imaging unit 402 may be used for photographing with a monocular stereo camera. In this embodiment, it is assumed that the imaging unit 401 is used for imaging. The continuous shooting may be performed with close frame intervals such as moving image shooting, or may be performed with loose frame intervals such as continuous shooting of still images.

FIG. 21B is a flowchart showing detailed processing in S402. S402 consists of the processing of S4021 to S4027.

S4021 is a loop of frames captured by the monocular stereo camera. Acquisition of the relative distance by the monocular stereo camera first acquires the three-dimensional relative position of the object, and uses it to convert the relative distance value from the position where the user wants to acquire the absolute distance value. Techniques such as SfM (Structure from Motion) and SLAM (Simultaneous Localization And Mapping) can be used to acquire the three-dimensional relative position of the subject. Alternatively, after calculating the position and orientation of the imaging device 400 by SfM or SLAM, a method called MVS (Multi View Stereo) can be used in which the calculated position and orientation are used to acquire a dense three-dimensional relative position. In the following description, it is assumed that a technique such as SfM or SLAM that simultaneously obtains the position and orientation of the imaging device 400 and the three-dimensional relative position of the subject is used.

FIG. 23 is a diagram showing the state of the display unit 407 during relative distance measurement. A display 601 displays in real time the subject whose relative distance is to be measured. By taking a picture while viewing this image, the user can measure the relative distance including the object for which the distance is to be measured. A notification 602 is a display for notifying the user that relative distance measurement is being performed by monocular stereo shooting, and is displayed superimposed on the display 601 . The notification 602 can also be given by an alarm sound, an audio guide, etc. In that case, the display 601 can be displayed without loss.

In S4022, the control unit 408 extracts feature points from the image of the current captured frame. Typical methods for extracting feature points include SIFT (Scale Invariant Feature Transform), SURF (Speeded-Up Robust Features), and FAST (Features from Accelerated Segment Test), but other methods may also be used.

In S4023, the control unit 408 determines whether or not the current frame is the first frame.

In S4024, the control unit 408 associates the feature points extracted in S4022 in the previous frame with the feature points extracted in S4022 in the current frame. When the frame interval is sparse, it is necessary that the common portion of the object to be distance-measured is sufficiently captured in the images between different frames. This is because if there are few common parts, the feature points cannot be associated, and the calculation for obtaining the relative distance stops.

FIG. 24 is a diagram showing how feature points are associated during relative distance measurement by a monocular stereo camera. A position 701 and a position 702 indicate positions of the imaging device 400 at different times. An image 703 and an image 704 represent images captured by the imaging unit 401 at positions 701 and 702, respectively. A feature point 705 is a feature point extracted from the image 703 and correctly associated with the feature point 706 in the image 704 . On the other hand, since the feature point 707 is not extracted at the correct position in the image 704 , it is incorrectly associated as the feature point 708 in the image 704 .

Now, if the reliability of the relative distance value by the monocular stereo camera is defined as the matching accuracy of the feature point, the feature point 707 is a feature point with poor matching accuracy, and the reliability of the relative distance value is relatively low. Reasons for such a decrease in reliability include the fact that the subject has little texture and that the image is blurred at a position in the image 704 corresponding to the feature point 707 .

In S4025, the control unit 408 calculates the position and orientation of the imaging device 400 and the three-dimensional relative position of the subject.

In S4026, the control unit 408 calculates the reliability of the relative distance value. The matching accuracy of the feature points associated in S4024 is calculated, and the higher the matching accuracy, the higher the reliability. Algorithms such as RANSAC (Random Sample Consensus) can be used to calculate matching accuracy. RANSAC calculates how much the motion of a feature point of interest deviates from the position and orientation of the imaging device 400 and the average value of the motion of a large number of feature points on the image, and determines the matching accuracy of the feature points.

In S4027, the control unit 408 determines whether or not the user has instructed the compound-eye stereo camera shooting. If the user has not given an instruction, the process proceeds to the next frame. Here, the instruction to shoot with the compound-eye stereo camera means that the user presses the shutter button 405 halfway, for example, to enter the shooting operation for calculating the absolute distance value.

In S403, the control unit 408 captures images with the compound eye stereo camera, that is, both the imaging unit 401 and the imaging unit 402, and acquires the absolute distance value.

FIG. 25(a) is a flowchart showing detailed processing of S403. S403 consists of the processing of S4031 to S4039.

In S4031, the control unit 408 focuses (focuses on) a specific subject. A specific subject is, for example, an object whose distance the user particularly wants to measure. As a method of focusing, for example, there is a method of focusing by an autofocus function that operates when the user half-presses the shutter button 405 . Focusing may be manual. Also, both the imaging unit 401 and the imaging unit 402 focus on the same subject.

In S4032, the absolute distance and its reliability by the compound-eye stereo camera are calculated. In absolute distance calculation by a compound eye stereo camera, a technique such as stereo matching is used for images captured by the imaging units 401 and 402 respectively.

FIG. 26 is a diagram showing how stereo matching is performed during absolute distance measurement by a compound-eye stereo camera. Assume that the images captured by the imaging units 401 and 402 when the user performs focusing at the position 801 in S4031 are images 802 and 803, respectively. Using an image 802 as a reference image and an image 803 as a reference image, windows (regions) 804 and 805 are set in the images 802 and 803, respectively. The window 804 is fixed, the window 805 is moved, and the correlation calculation is performed on the images within the window. The absolute distance to the subject captured in the window 804 is calculated from the parallax value and the baseline length based on the distance D between the optical axis of the imaging unit 401 and the optical axis of the imaging unit 402, where the amount of shift on the image of the corresponding position is parallax. calculate. At this time, the reliability for the calculated absolute distance is calculated. For example, the position of the window 804 may be changed in various ways, and the highest correlation value obtained by the window matching calculation for each of them may be regarded as having the highest reliability. Also, as described later, the reliability may be represented by the defocus amount with respect to the subject. Although the method of window matching has been described here, the method is not limited to this, and feature points may be extracted and associated as in relative distance calculation using a monocular stereo camera.

In S4033, the control unit 408 determines whether there is a substantially identical subject with high reliability in both the relative distance value obtained by the monocular stereo camera and the absolute distance value obtained by the compound eye stereo camera. If there is a substantially identical subject with high reliability, the process proceeds to S4037; otherwise, the process proceeds to S4034.

In S4034, the feature point near the focus position is used to convert the relative distance acquired by the monocular stereo camera into an absolute distance. When calculating the reliability of relative distance calculation and absolute distance calculation, it is not always the case that there is a subject near the focus with high reliability for both. good.

Next, the reliability of the absolute distance acquired by the compound-eye stereo camera is defined using the defocus amount. For example, the reciprocal of the magnitude of the defocus amount is used as the reliability, and the smaller the defocus amount, the higher the reliability. This allows the window 804 to be set at different positions to determine the position with the highest confidence. If a window 804 including candidate points that can be the feature point X among the feature points acquired by the monocular stereo camera is set, the reliability can be obtained from the defocus amount. In shooting with a compound-eye stereo camera, if the absolute distance value of a subject whose defocus amount is within the depth of field is highly reliable, the subject included in the window 804 is within the depth of field. becomes a candidate point for feature point X.

In S4035, the control unit 408 sets the aperture small so that the candidate point of the feature point X is within the depth of field. By this processing, the candidate point of the feature point X can be photographed without being blurred, and the absolute distance value can be obtained with high accuracy. When converting the relative distance value to the absolute distance value in S4034, it is not always possible to use a feature point with high reliability. good. Here, it is not necessary to set the size of the aperture, and the focus may be readjusted to the position where the candidate point of the feature point X is within the depth of field while the size of the aperture remains unchanged. By doing this, even if the shooting scene is too dark to reduce the aperture, the candidate points of the feature point X can be shot without being blurred.

In S4036, the control unit 408 calculates the reliability of the absolute distance value for the feature point X candidate point.

In S4037, the feature point X is determined from the candidate points of the feature point X. FIG. The determination method is, for example, the position where the total ranking of the reliability calculated by the monocular stereo camera and the reliability obtained by the compound eye stereo camera is highest.

In S4038, the user presses the shutter button 405 to take an image.

In S4039, the absolute distance value of the feature point X is obtained from the two images taken by the compound-eye stereo camera. The method of obtaining the absolute distance value is the same as described above, but when calculating by window matching, a window containing the feature point X and as small as possible should be set. In the case of calculation by feature point matching, the feature points X may be matched between two images as they are. From S4039, the process returns to S404 in FIG.

In S404, the relative distance value acquired by the monocular stereo camera is converted into an absolute distance value. FIG. 25(b) is a flowchart showing detailed processing of S404. S404 consists of the processing of S4041 and S4042.

In S4041, using the relative distance value and the absolute distance value for the feature point X, a conversion formula (conversion relation) for converting the relative distance value to the absolute distance value for all subjects is calculated. For example, when the relative distance value of the feature point X is zr and the absolute distance value is Za[m], Z=(Za/zr)×z. where z is the relative distance value of an object and Z is the absolute distance value relative to it.

In S4042, the relative distance value obtained by the monocular stereo camera is converted into an absolute distance value using a conversion formula. Here, if the conversion range is limited to the range of the composition photographed in S4038, the absolute distance value of the composition desired by the user can be obtained with high accuracy.

By using the method of this embodiment, it is possible to determine the conversion formula from the relative distance value to the absolute distance value at the position of the image where the reliability of the relative distance value is high and the reliability of the absolute distance value is high. . Then, by converting other images using the conversion formula, highly accurate absolute distance values can be obtained for all subjects.

<Fifth Embodiment>
Next, a fifth embodiment of the invention will be described.
In the above-described fourth embodiment, the imaging device is configured as a twin-lens stereo camera. Combining them into one simplifies the configuration.

FIG. 27 is a diagram showing the structure of a split-pupil imaging device and the principle of distance calculation. FIG. 27(a) shows a state in which the subject is in focus, and FIG. 27(b) shows a state in which the subject is located in front of the focus position.

The imaging device 901 has a split-pupil structure, and the interior of a pixel 903 is divided into two sub-pixels 904 and 905 . Of the light reflected by the subject, one light flux passes through the end of the imaging optical system 902 and is received by the sub-pixels 904, and the other light flux passes through the opposite end of the imaging optical system 902. Light is received by sub-pixels 905 . Images 906 and 908 are images generated from light received by subpixel 904 , and images 907 and 909 are images generated from light received by subpixel 905 .

As shown in FIG. 27A, the light from the subject at the focus position is received by the sub-pixels 904 and 905 in the same pixel, so there is a parallax between the subjects in the images 906 and 907. do not have. On the other hand, as shown in FIG. 27B, light from an out-of-focus subject is received by sub-pixels 904 and 905 of different pixels, so that it appears in images 908 and 909. Parallax occurs in the subject. An absolute distance can be calculated from this parallax.

When a split-pupil imaging device is used, the compound eyes share a lens, so the size of the aperture determines the base length of the compound eye stereo camera. For this reason, if the aperture is reduced in order to keep the candidate point of the feature point X within the depth of field, the base line length will be shortened, making it impossible to obtain a highly accurate absolute distance value. In this case, the reliability of the absolute distance value obtained by the split-pupil imaging device may be determined by considering both the magnitude of the defocus and the baseline length. Further, as described above, the aperture may be fixed and the focus may be reset to a position where the reliability of the candidate for the feature point X increases.

In the above explanation, we explained how to calculate a highly accurate absolute distance value by matching the relative distance value obtained with the monocular stereo camera with the absolute distance value obtained with the compound eye stereo camera. may

For example, when it is desired to obtain an absolute distance value of a scene including a dynamic subject, a monocular stereo camera cannot obtain a relative distance value of the dynamic subject. In this case, the absolute distance value of the dynamic object obtained by the compound-eye stereo camera may be used as it is.

Since the monocular stereo camera calculates the corresponding points of the image using the images between the frames taken at different times, it may not be possible to find the corresponding points in the case of the image including a dynamic subject. Also, it may not be possible to distinguish between the movement of the dynamic subject and the change in the camera's position and orientation. On the other hand, since the compound-eye stereo camera can match the photographing times, it is possible to treat a dynamic subject between two images in the same manner as a static subject, and to calculate an absolute distance value. Using this, when creating an absolute distance image represented by gradation, the distance obtained by converting the relative distance value obtained by the monocular stereo camera to the absolute distance value with the feature point X extracted from the stationary object use an image. Then, the absolute distance value obtained by the compound-eye stereo camera is used for the dynamic subject. Determination of whether or not the subject is dynamic may be performed using machine learning or the like, and the subject is not in the relative distance image obtained by the monocular stereo camera, but in the absolute distance image obtained by the compound eye stereo camera. A dynamic subject may be used, or both may be combined.

<Other embodiments>
The present invention may be applied to a system composed of a plurality of devices or to an apparatus composed of a single device.

Further, the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device executes the program. It can also be realized by a process of reading and executing. It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

100: imaging device, 101: imaging unit, 102: memory, 103, 303: viewing image generation unit, 104: single-shot distance image generation unit, 105: multi-view distance image generation unit, 106: distance image integration unit, 1011: Optical system 1012: Imaging device 204: Arithmetic processing unit 400: Imaging device 401, 402: Imaging unit 403: Arithmetic unit 404: Storage unit 405: Shutter button 406: Operation unit 407: Display unit , 408: control unit

Claims (37)

acquisition means for acquiring a plurality of different viewpoint images of the same scene photographed from different positions and at least a pair of parallax image pairs having parallax due to pupil division;
a first generation means for generating a first distance image from the pair of parallax images;
a second generation means for generating a second distance image from the plurality of different-viewpoint images;
Integrating means for integrating the first range image and the second range image to generate an integrated range image. The integration means supplements the distance information of the area where the subject is moving among the distance information forming the second distance image with the distance information of the same area of the first distance image, thereby 2. The image processing apparatus according to claim 1, wherein the first distance image and the second distance image are integrated. The integrating means may combine distance information constituting the first distance image with an area in which the distance information indicates a distance shorter than a predetermined distance and a distance within a predetermined range from the in-focus distance. Integrating the first distance image and the second distance image by complementing the distance information of the area showing and the area excluding the area with the distance information of the same area of the second distance image 2. The image processing apparatus according to claim 1. 4. The image processing according to any one of claims 1 to 3, wherein said second generating means generates said second distance image based on parallax between said different viewpoint images. Device. 5. The image processing apparatus according to claim 4, wherein said second generating means generates said second range image using epipolar geometry from said different viewpoint images. the acquisition means inputs a plurality of parallax image pairs obtained by photographing the same scene from different positions;
6. The image processing apparatus according to any one of claims 1 to 5, further comprising third generating means for generating said plurality of different-viewpoint images from said plurality of parallax image pairs. The third generating means uses one of the plurality of parallax image pairs as a reference parallax image pair, and adds the plurality of parallax image pairs excluding the reference parallax image pair for each pixel to generate the plurality of different pairs. 7. The image processing apparatus according to claim 6, wherein a viewpoint image is generated. The third generating means leaves one of the plurality of parallax image pairs as a reference parallax image pair, and adds the plurality of parallax image pairs including the reference parallax image pair for each pixel, thereby obtaining the 7. The image processing apparatus according to claim 6, wherein a plurality of images of different viewpoints are generated. The third generating means generates a plurality of different-viewpoint images while sequentially changing the reference parallax image pair among the plurality of parallax image pairs,
The integration means integrates a first distance image generated from the reference parallax image pair sequentially shifted and a second distance image generated from the corresponding plurality of different viewpoint images, and a plurality of the The image processing device according to claim 7 or 8, wherein an integrated range image is generated. The third generating means performs defocus recovery processing on the plurality of parallax image pairs, and generates the plurality of different-viewpoint images from the plurality of parallax image pairs subjected to the defocus recovery processing. 10. The image processing apparatus according to any one of claims 6 to 9. 11. The method according to claim 10, wherein the focus blur recovery processing includes deconvolution processing or MAP estimation processing performed by estimating a focus blur kernel, and deep learning processing by end-to-end processing using an encoder-decoder structure. image processing device. the acquisition means inputs a plurality of parallax image pairs obtained by photographing the same scene from different positions;
The first generating means generates the first distance images for each of a plurality of predetermined parallax image pairs among the plurality of parallax image pairs, and generates the first distance images among the plurality of generated first distance images. Using one of the first distance images as a reference, the other first distance images are integrated by viewpoint conversion,
6. The image processing apparatus according to claim 1, wherein said integrating means integrates said integrated first distance image and said second distance image. An image processing device according to any one of claims 1 to 12;
An imaging apparatus, comprising: imaging means for imaging the at least one pair of parallax images as at least a part of the acquisition means. 14. The imaging apparatus according to claim 13, wherein said imaging means captures said at least one pair of parallax images after capturing said plurality of images of different viewpoints. 14. The imaging apparatus according to claim 13, wherein said imaging means captures said plurality of different-viewpoint images after capturing said at least one pair of parallax images. 14. The imaging apparatus according to claim 13, wherein said imaging means captures said plurality of different-viewpoint images before and after imaging said at least one pair of parallax images. 17. The imaging according to any one of claims 13 to 16, wherein the imaging means narrows down the aperture when shooting the plurality of images of different viewpoints than when shooting the at least one pair of parallax images. Device. 17. The photographing means sets photographing conditions respectively when photographing the at least one pair of parallax images and when photographing the plurality of images of different viewpoints. The imaging device according to . an obtaining step in which the obtaining means obtains a plurality of different viewpoint images of the same scene photographed from different positions and at least one pair of parallax images having parallax due to pupil division;
a first generating step in which a first generating means generates a first distance image from the pair of parallax images;
a second generating step in which a second generating means generates a second distance image from the plurality of different-viewpoint images;
An image processing method, comprising: an integrating step of integrating the first range image and the second range image to generate an integrated range image. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 12. A computer-readable storage medium storing the program according to claim 20. at least two imaging units with a known distance between the optical axes;
Based on a plurality of first images having parallax with each other taken at approximately the same time using the imaging unit, a plurality of absolute distance values to the subject and their reliability are calculated, and the imaging unit is used to calculate calculating means for calculating a plurality of relative distance values to the subject and their reliability based on a plurality of second images having parallax with each other taken at different times;
The calculating means uses, of the plurality of absolute distance values and the plurality of relative distance values, an absolute distance value and a relative distance value that have a relatively high degree of reliability and that correspond to substantially the same region of the subject. , and obtains a conversion relationship between an absolute distance value and a relative distance value. 23. The imaging apparatus according to claim 22, wherein said calculating means calculates said plurality of absolute distance values using window matching. 24. The imaging apparatus according to claim 23, wherein said calculation means determines the reliability of said absolute distance value or the reliability of said relative distance value based on accuracy of window matching. 23. The imaging apparatus according to claim 22, wherein said calculation means determines the reliability of said absolute distance value or the reliability of said relative distance value based on accuracy of matching of feature points. 26. The imaging apparatus according to any one of claims 22 to 25, wherein said at least two imaging units are composed of pupil-divided pixels of an imaging device. 27. The imaging apparatus according to any one of claims 22 to 26, wherein said calculator calculates the reliability of said absolute distance value based on the magnitude of defocus amount of said imaging unit. 28. The imaging apparatus according to claim 27, wherein said calculation means performs calculation so that the reliability of absolute distance values within the depth of field of said imaging unit is relatively high. The calculating means calculates the relative distance value before photographing the plurality of first images, and uses the reliability of the relative distance value to increase the reliability of the absolute distance value. 29. The imaging apparatus according to any one of claims 22 to 28, wherein the imaging conditions for one image are determined. 30. An imaging apparatus according to claim 29, wherein said imaging condition is an aperture value of said imaging unit. 30. An imaging apparatus according to claim 29, wherein said imaging condition is a focus position of said imaging unit. The calculation means generates a first absolute distance image using the absolute distance value for the subject, converts the relative distance value to an absolute distance value for the other subject using the conversion relationship, and converts the relative distance value to the absolute distance value. 32. The imaging apparatus according to any one of claims 22 to 31, wherein two absolute distance images are generated, and the first absolute distance image and the second absolute distance image are integrated. 33. The imaging apparatus according to any one of claims 22 to 32, wherein said calculation means converts a relative distance value of another object into an absolute distance value using said conversion relationship. 34. A method according to any one of claims 22 to 33, wherein said calculating means calculates said absolute distance value based on a plurality of said first images and an interval between said optical axes. imaging device. A method of controlling an imaging device comprising at least two imaging units with a known distance between the optical axes, comprising:
Based on a plurality of first images having parallax with each other taken at approximately the same time using the imaging unit, a plurality of absolute distance values to the subject and their reliability are calculated, and the imaging unit is used to calculate a calculating step of calculating a plurality of relative distance values to the subject and their reliability based on a plurality of second images having parallax with each other taken at different times;
In the calculating step, of the plurality of absolute distance values and the plurality of relative distance values, an absolute distance value and a relative distance value having a relatively high degree of reliability and corresponding to substantially the same region of the subject are used. A control method for an imaging device, comprising: obtaining a conversion relationship between an absolute distance value and a relative distance value. A program for causing a computer to execute the control method according to claim 35. A computer-readable storage medium storing a program for causing a computer to execute the control method according to claim 35.

Images