JP2024005590A - Image processing device, imaging device, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide image processing technology with which it is possible to acquire depth information regarding the depth direction of a subject with greater accuracy.

SOLUTION: An imaging element 101c of an imaging device 100 includes a plurality of photoelectric conversion units that respectively perform photoelectric conversion on light having passed through different pupil portion regions in an imaging optical system. The imaging device 100 is capable of generating an image corresponding to each of the photoelectric conversion units and acquiring a plurality of viewpoint images. The imaging device 100 changes the opening amount of a diaphragm 101b and captures the image of a subject, then performs processing regarding the difference between a viewpoint image captured with a first opening amount and a viewpoint image captured with a second opening amount that is smaller than the first opening amount, so as to generate a difference image. The imaging device 100 is capable of searching for a corresponding point of images using a plurality of images that are free of blurs and largely misaligned and acquiring depth information concerning the subject in the images.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present invention relates to a technique for acquiring depth information of a subject in the depth direction.

When the image sensor included in the imaging device has a distance measurement function, it is possible to obtain an ornamental image and distance information to a subject. Three-dimensional (hereinafter referred to as "3D") data generated based on distance information can be used as display content for VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality).

Distance information can be obtained by capturing the light passing through different pupil partial regions of the imaging optical system using two types of pixel sections and calculating the amount of image shift from the images obtained by the respective pixel sections. One of the two types of pixel portions will be referred to as an A pixel, and the other will be referred to as a B pixel. An image generated from the outputs of a plurality of A pixels is called an A image, and an image generated from the outputs of a plurality of B pixels is called a B image. For example, focus detection and focus adjustment control of the imaging optical system can be performed based on the correlation calculation result between the A image and the B image. Furthermore, an ornamental image can be generated based on the added image obtained by adding the A image and the B image. Patent Document 1 discloses a method of searching for an appropriate direction of image shift by changing the aperture size (opening amount) of an aperture in order to calculate the amount of image shift of images with blur in images A and B. Disclosed.

JP 2019-200348 Publication

With conventional technology, when searching for the direction of image shift when there is large blur in the image of a subject located outside the focus range, it is difficult to detect the amount of image shift regardless of which direction the image shift occurs. It is difficult to increase accuracy.
An object of the present invention is to provide an image processing technique that can obtain depth information with higher accuracy.

An image processing device according to an embodiment of the present invention is an image processing device capable of acquiring depth information in the depth direction of a subject in an image from a plurality of viewpoint images having different viewpoints, and which an acquisition unit that acquires a plurality of images based on light passing through each region; and a control that changes the aperture amount of the aperture of the imaging optical system, and uses a plurality of images captured with different aperture amounts of the aperture; and control means for controlling the acquisition of the depth information. The acquisition means is configured to acquire first and second viewpoint images captured with a first aperture, and third and fourth viewpoint images captured with a second aperture smaller than the first aperture. The control means generates a first difference image from the first viewpoint image and the third viewpoint image, and generates a second difference image from the second viewpoint image and the fourth viewpoint image. Control is performed to generate an image and obtain the depth information using one or more of the first and second difference images.

According to the present invention, it is possible to provide an image processing technique that allows depth information to be acquired with higher accuracy.

1 is a diagram showing a configuration example of an imaging device according to an embodiment. FIG. 1 is a diagram schematically showing the appearance of an imaging device according to the present embodiment. FIG. 2 is an explanatory diagram of a pupil division type image sensor. FIG. 3 is a diagram showing how a pixel section receives light from a subject. FIG. 3 is a diagram showing how a pixel section receives light from an out-of-focus object. FIG. 7 is a diagram illustrating an example of an image when corresponding locations cannot be calculated with high accuracy. FIG. 7 is a diagram showing another example of how a pixel section receives light from an out-of-focus object. FIG. 3 is a diagram schematically showing a photographing state in this embodiment. It is a figure showing an example of an image in this embodiment. FIG. 3 is a diagram schematically showing how vignetting occurs. It is a flowchart explaining the processing of this embodiment. It is a figure showing the example of a screen of the display part concerning this embodiment. FIG. 3 is a diagram schematically showing a plurality of regions and windows in an image. FIG. 7 is a diagram showing the relationship between the aperture and the luminous flux when the aperture is made small.

Embodiments of the present invention will be described in detail below with reference to the drawings. FIG. 1 is a block diagram showing a configuration example of an imaging device as an application example of an image processing device. The imaging device 100 can acquire depth information based on parallax images. A parallax image is composed of a plurality of images from different viewpoints (hereinafter referred to as viewpoint images). Depth information is information representing the depth of the subject in the depth direction.

The imaging device 100 includes an imaging section 101, a calculation section 102, a storage section 103, a shutter button 104, an operation section 105, a display section 106, and a control section 107. The imaging unit 101 includes a lens 101a, an aperture 101b, and an image sensor 101c. The lens 101a and the aperture 101b constitute an imaging optical system (imaging optical system). Light from the subject passes through the imaging optical system and is then received by the imaging element 101c. The image sensor 101c performs photoelectric conversion on the optical image and outputs an electrical signal to the calculation unit 102 and the control unit 107.

The calculation unit 102 performs development processing on the captured image, and stores the data after the development processing in the storage unit 103. The storage unit 103 includes, for example, a removable storage medium (such as an SD card) or a storage medium provided inside the imaging device 100.

The shutter button 104 is an operation member that is pressed by the user of the imaging apparatus 100 to issue a shooting instruction, and outputs an operation signal to the control unit 107 . The operation unit 105 includes an operation member that can be used when a user issues an operation instruction to the imaging apparatus 100, and outputs a signal of the operation instruction to the control unit 107.

The display unit 106 includes a display device such as a liquid crystal display. The display unit 106 displays captured images, GUI (graphic user interface) images, and the like on the screen according to control commands from the control unit 107.

The control unit 107 includes, for example, a CPU (central processing unit), and controls the entire imaging device. A control unit 107 performs various interface processing and controls the calculation unit.

Although this embodiment shows an example of a configuration in which the imaging optical system is integrated into the main body of the imaging device, it is also possible to apply it to a configuration in which a lens unit having an imaging optical system can be attached to the main body of the imaging device. It is.

FIG. 2 is a diagram schematically showing the appearance of the imaging device 100. FIG. 2A shows the appearance of the imaging device 100 when viewed from the front. The lens 101a collects light from the subject and generates an optical image on the image sensor 101c. The image sensor 101c outputs an electrical signal by photoelectrically converting the optical image. The light collection range in the imaging optical system is determined by the controller 107 controlling the aperture 101b. In other words, the light that has passed through the condensing range becomes the light that reaches the image sensor 101c. Photographing is started when the user presses the shutter button 104. The data of the photographed image is developed by the calculation unit 102 and then stored in the storage unit 103.

FIG. 2(B) shows the appearance of the imaging device 100 when viewed from the back. A part of the operation unit 105 provided on the back side of the imaging device 100 is shown. For example, the user uses the operation unit 105 when setting photographing conditions or instructing to start or end the high-precision ranging mode. The display unit 106 displays the composition at the time of shooting, and also displays items, video search results, etc. at the time of various settings. Further, in the case of a display device having a touch panel, the functions of the shutter button 104 and the operation unit 105 can be shared by the function of the display unit 106. The user can perform shooting and setting operations by touching the screen of the display unit 106. Since there is no need to equip hardware components such as the shutter button 104 and the operation section 105, this is suitable for increasing the screen size of the display section 106 and improving operability.

FIG. 3A is a schematic diagram showing the internal structure of a pixel portion in the split-pupil image sensor 101c. A pixel section 108 having one microlens 111 among a large number of microlenses arranged over the light-receiving surface of the image sensor 101c is shown. The pixel section 108 includes a sub-pixel 109 and a sub-pixel 110.

Among the light emitted from the subject, the light that has passed through different pupil partial regions in the imaging optical system is captured by the sub-pixel 109 and the sub-pixel 110, and the distance can be calculated from the amount of shift of the subject image in each image. . Hereinafter, the sub-pixel 109 will be referred to as an A-pixel, and the sub-pixel 110 will be referred to as a B-pixel.

FIG. 3B shows a pixel array of the image sensor 101c. A pixels and B pixels are arranged alternately over the entire imaging surface. The A pixel and the B pixel can each receive light that has passed through different pupil partial regions in the imaging optical system. That is, it is possible to obtain an A image (first viewpoint image) based on the output of a plurality of A pixels and a B image (second viewpoint image) based on the output of a plurality of B pixels. The images of the subject in the A image and the B image will be referred to as the A image and the B image, respectively.

The pixel unit 108 is designed so that the A pixel and the B pixel receive the light that has passed through the shared microlens 111. Whether light is received by the A pixel or the B pixel is determined by the angle of incidence 112. In FIG. 3A, regarding the sign of the incident angle 112, with the optical axis direction as a reference, the angle sign of incident light from the right direction is defined as plus, and the angle sign of incident light from the left direction is defined as minus. The A pixel and the B pixel have different angular characteristics of light reception sensitivity with respect to incident light, and can only receive light that has passed through different pupil partial regions.

FIG. 3C is a graph illustrating the light receiving sensitivity of the A pixel and the B pixel with respect to the incident angle of light. The horizontal axis represents the incident angle, and the vertical axis represents the light receiving sensitivity. A solid line light-receiving sensitivity curve 113 represents the light-receiving sensitivity of the A pixel. The light receiving sensitivity is relatively high in the range where the incident angle is positive (angle range of incident light from the right direction), and the light receiving sensitivity is relatively high in the range where the incident angle is negative (angle range of incident light from the left direction). low. On the other hand, a broken line light-receiving sensitivity curve 114 represents the light-receiving sensitivity of the B pixel. The light-receiving sensitivity is relatively high in a negative incident angle range, and the light-receiving sensitivity is relatively low in a positive incident angle range. By arranging a light absorbing member at an appropriate position within the waveguide inside the pixel portion 108, such a light reception sensitivity curve can be obtained.

The imaging device 100 converts light that passes through different pupil partial regions in the imaging optical system into image signals using a plurality of photoelectric conversion units included in the imaging element 101c. It is possible to obtain a parallax image having an image shift according to the distance to the subject, as if the images were captured by two cameras having different viewpoints. Corresponding locations can be detected by window matching processing of images A and B for the subject, and distance information to the subject can be calculated based on the amount of image shift on the image. In this embodiment, a process of acquiring distance information from an imaging device to a subject will be described as an example of depth information. Other depth information includes an image shift amount map and a defocus amount map. The image shift amount map can be calculated from a plurality of viewpoint images, and the defocus amount map can be calculated by multiplying the image shift amount by a predetermined conversion coefficient.

FIG. 4A shows how light from an object is received by the A pixel and the B pixel, respectively. Of the light emitted from the subject, a light beam 115 passing through the first area (right side area) of the lens 101a and a light beam 116 passing through the second area (left side area) of the lens 101a are shown. Regarding the light flux 115, since the light enters the pixel portion 108 at a positive incident angle, the A pixel has high light receiving sensitivity, but the B pixel has low light receiving sensitivity (the light receiving sensitivity curve 113 in FIG. 3(C), 114). On the contrary, regarding the light beam 116, since the light enters the pixel portion 108 at a negative incident angle, the B pixel has high light receiving sensitivity, but the A pixel has low light receiving sensitivity. The A image generated from the output of the A pixel and the B image generated from the B pixel output are images based on light fluxes that have passed through different areas of the lens 101a, respectively.

FIG. 4A shows how the A pixel and the B pixel receive light from a focused subject (focused subject). Since the light beams 115 and 116 from the focused object are converged on the image sensor 101c, substantially no image shift occurs.

Next, the relationship between image shift and blur between the A image and the B image, which is the basic principle of the present invention, will be explained. Photographing an out-of-focus subject will be described with reference to FIG. 4(B). In this case, the light flux 115 and the light flux 116 are focused at a position shifted from the image sensor 101c.

FIG. 4(B) shows how the A pixel and the B pixel receive light from a subject (unfocused subject) located at a position closer to the lens 101a than the in-focus position in FIG. 4(A). The light beams 115 and 116 from the unfocused object are received by the image sensor 101c before being condensed. Therefore, the light-receiving pixel (A pixel) for the light beam 115 and the light-receiving pixel (B pixel) for the light beam 116 are separated from each other, so that image shift 201 occurs. The magnitude of the image shift 201 (image shift amount) increases as the shift amount from the in-focus position where the subject is focused increases. Processing is performed to obtain the amount of image shift and convert it into a distance to the subject. However, for a subject that has a large amount of deviation from the in-focus position, it is difficult to accurately calculate the amount of image deviation.

FIG. 5 shows how the A pixel and the B pixel receive light from an out-of-focus subject located at a position closer to the lens 101a than in FIG. 4(B). In the case of FIG. 4(B), since neither the light flux 115 nor the light flux 116 is focused on the image sensor, blurring occurs in images A and B, but since the amount of blur is small, the amount of image shift is calculated. It is possible to do so. On the other hand, in the case of FIG. 5, large blurring occurs between the A image and the B image. Since the amount of blur 301 and the amount of blur 302 are large, it becomes difficult to search for corresponding locations between the A image and the B image with high precision by window matching processing or the like. For example, the image shift amount 303 is calculated by setting the position of the image based on the light beam passing through the center side of the lens 101a among the light beams 115 and 116 as corresponding points. The image shift amount 303 has a relatively small value. Further, the image shift amount 304 is calculated by setting the positions of the images based on the light beams that pass through a position away from the center of the lens 101a among the light beams 115 and 116 as corresponding positions. The image shift amount 304 becomes a large value, and the corresponding location between the A image and the B image cannot be determined with high accuracy.

FIG. 6 is a diagram illustrating an example of a case where the corresponding locations cannot be calculated with high accuracy between the A image and the B image. An A image 401 is shown on the upper side, and a B image 402 is shown on the lower side. The A image 401 and the B image 402 show the subject in a blurred state, and the amount of image shift 303 between the A image 401 and the B image 402 is shown.

FIG. 7 shows how the A pixel and the B pixel receive light from an out-of-focus subject located further from the lens 101a than in FIG. 4(B). The light flux 115 and the light flux 116 are reversed in the left-right direction and reach the image sensor 101c. Although the direction of the image shift between the A image and the B image is opposite to that in the case of FIG. 5, the accuracy of searching for the corresponding location between the A image and the B image is reduced, as explained above. .

Therefore, the image processing apparatus of this embodiment generates the A image or the B image using only the light beams that pass through a position away from the center of the lens 101a regarding the light beams 115 and 116. Search accuracy can be improved by searching for corresponding locations between images using images with little blur and a large amount of image shift. The image processing method of this embodiment will be described below.

In this embodiment, two images are taken by changing the aperture amount of the aperture 101b, and an A image and a B image are obtained respectively. A image data and B image data obtained when the imaging device 100 shoots an image with the aperture 101b set to a large aperture are expressed as A _{large aperture} and B _{aperture large} , respectively. A image data and B image data obtained when the image capturing apparatus 100 shoots an image with the aperture 101b set to a small aperture are expressed as A _{small aperture} and B _{aperture small} , respectively. The difference between two A image data is expressed as A _difference , and the difference between two B image data is expressed as B _difference . The difference image data A _difference and B _difference can be obtained from equation (1).
The control unit 107 uses the A _difference and the B _difference generated by (Equation 1) to search for a corresponding part of the image, and calculates the amount of image shift. This will be explained in detail with reference to FIG.

FIG. 8 is a diagram schematically showing a shooting state by the imaging device 100. The state of the first photograph 601 is shown on the upper left side, and the state of the second photograph 602 is shown on the upper right side. In the first photographing 601, photographing is performed with the aperture 101b set to the first aperture amount, and _{large aperture} A and _{large aperture} B are obtained. In the second photographing 602, the aperture 101b is set to the second aperture amount and photographing is performed. The second opening amount is smaller than the first opening amount. In the second photographing 602, A _{small aperture} and B _{aperture small} are acquired.

The luminous flux 115 and the luminous flux 116 will be explained by comparing the first photographing 601 and the second photographing 602. Compared to the first photograph 601, in the second photograph 602, the light rays passing through a position away from the center of the lens 101a are partially blocked by the aperture. The third photograph 603 schematically represents the optical state when an A image and a B image equivalent to the A _difference and the B _difference are acquired. Although the first imaging 601 and the second imaging 602 are imaging actually performed by the imaging apparatus 100, it should be noted that the third imaging 603 is not an actual imaging (virtual imaging).

In the third photographing 603, of the light beams 115 and 116 from the first photographing 601, only the light beams passing through a position away from the center of the lens 101a contribute to the generation of the A _difference and the B _difference , respectively. I understand. The amount of blur 604 of the A image and the B image is small, and the amount of image shift 605 is large. Therefore, the amount of image shift can be calculated with higher precision.

FIG. 9 shows an example of images A and B when corresponding locations can be calculated with high accuracy. The A image 701 and the B image 702 are images generated from the A _difference and the B _difference, respectively, and are clearer than the A image 401 and the B image 402 in FIG. The amount of image shift 703 between the A image 701 and the B image 702 is shown. Both the A image 701 and the B image 702 are images generated only from light rays passing through a position far from the center of the lens 101a, and the image shift amount 703 is larger than the image shift amount 303. Therefore, the corresponding locations in the image of the subject can be calculated with higher precision.

In FIG. 8, when shooting multiple times, it is preferable to match shooting conditions other than the aperture amount (F number) such as shutter speed and ISO sensitivity, but images A generated by light rays passing near the center of the lens 101a are different from each other. , B images may have the same conditions. For example, for the first and second shots with a faster shutter speed, exposure control is performed to compensate for the brightness of the image by, for example, setting to increase the ISO sensitivity.

By the way, although one lens is illustrated as the lens 101a to simplify the explanation, an actual imaging optical system is often composed of a plurality of lenses. In this case, so-called vignetting of light rays occurs, in which the range of the light flux is determined by the lens 101a, with respect to light emitted from the subject that passes through a position away from the central axis of the lens 101a.

FIG. 10 is a diagram schematically showing how vignetting occurs. FIG. 10 shows a configuration example having a first lens 101a_1 on the front side and a second lens 101a_2 on the rear side when photographing a subject located at a position off the central axis of the lens 101a. The range of the luminous flux 115 is determined by the aperture 101b, whereas the range of the luminous flux 116 is determined by the outer edge portion 801 of the second lens 101a_2. At this time, even if the aperture 101b is changed and the image is photographed, the B image does not change. When the difference between the images obtained by changing the aperture 101b is calculated, the A image becomes clear, but the B image disappears. In this case, since it is not possible to search for a corresponding part of the image using the A _difference and B _difference , the control unit 107 searches for a corresponding part of the image using the A _difference and B _{large aperture} or the A _difference and B _{small aperture} . Execute.

FIG. 11 is a flowchart illustrating the processing of this embodiment. The processes from S901 to S910 are realized by the CPU of the control unit 107 executing a program. In S901, the process starts when the user instructs to start the high-precision distance acquisition mode.

FIG. 12A is a diagram showing an interface related to an instruction to start the high-precision distance acquisition mode, and shows a display example on the display unit 106 of the imaging device 100. The display unit 106 displays an operation display area 1001 on the screen. The user instructs the start of the high-precision distance acquisition mode by operating the operation unit 105 and selecting "start" in the operation display area 1001. When the display unit 106 includes a touch panel, the user selects a portion of the operation display area 1001 where “Start” is displayed by touching the area with a finger or finger. When the process of the high-precision distance acquisition mode is started, a display process is executed to notify the user of precautions when taking a photograph.

FIG. 12(B) shows an example of notification of precautions when photographing. The display unit 106 presents a notification display 1101 to the user, and urges the user to be careful about restraining the movement of the camera so that it does not move during shooting. This warning is to prevent the image from moving due to camera shake or the like during the shooting, since it is necessary to change the aperture of the diaphragm 101b and take two or more shots. Methods for alerting the user include a method of outputting audio using a speaker and a method of presenting a tactile sensation using a tactile device. Note that if the imaging device 100 has a function of correcting image blur caused by hand shake or the like, image blur correction is performed by setting that function to ON, so that restrictions on the user when taking pictures are relaxed. Furthermore, if the photographing operation of changing the aperture of the diaphragm 101b can be performed at high speed, image movement due to camera shake or the like is alleviated.

In S902 of FIG. 11, the imaging apparatus 100 performs imaging using the aperture amount of the aperture 101b as the first aperture amount. The data of the A image and the B image obtained as a result are stored in the storage unit 103 as A _{large aperture} and B _{large aperture} , respectively. Next, in S903, the imaging apparatus 100 performs imaging by setting the aperture amount of the aperture 101b to the second aperture amount, which is smaller than the first aperture amount. The data of the A image and the B image obtained as a result are stored in the storage unit 103 as A _{small aperture} and B _{small aperture} , respectively. For example, a great effect can often be obtained by the imaging device 100 shooting with the aperture 101b wide open in S902, and shooting with the aperture 101b stopped down one step further than the wide open aperture in S903. The reason for this is that the amount of image shift can be increased by using a light beam that passes through a position as far away as possible from the center of the lens 101a. After S903, the process advances to S904.

In S904, the control unit 107 calculates the difference between the image data acquired in S902 and S903 based on (Equation 1), and generates an A _difference and a B _difference . What should be noted here is how to deal with the case where the images taken in S902 and S903 move due to the user's camera shake or the movement of the subject. In that case, the control unit 107 compares the A _{large aperture} and the A _{small aperture} , or the B _{aperture large} and the B _{aperture small} , calculates the movement of the image by finding a corresponding part of the image, and adjusts the image according to the amount of image movement. Perform processing to shift the . For example, assume that the corresponding location at _{small aperture} A has moved to the right by 10 pixels with respect to the image at _{large aperture} A. The control unit 107 changes the A _{small aperture} to image data shifted to the left by 10 pixels. By deleting the 10 pixels on the right side of the _{large A aperture} for the 10 pixel width reduction in the small A _aperture , the sizes of both images can be made the same. At this time, the sizes of the generated A _difference and B _difference are both 10 pixels smaller in width.

In S905, the control unit 107 starts window loop processing. Although this embodiment will be described as performing window matching processing by searching for corresponding locations between images A and B, feature point matching or the like may also be used. Next, in S906, the control unit 107 determines a combination of the A image and the B image. A specific example will be explained with reference to FIG.

FIG. 13 is a diagram schematically showing a plurality of regions and windows within an image. Image 1201 is either A image or B image, and indicates a window of interest 1202 within image 1201. The position of the window of interest 1202 is changed in loop processing. The image 1201 is divided into three parts in the horizontal direction, and shows a first area 1203 including the center, a second area 1204 located on the left side, and a third area 1205 located on the right side. In the first region 1203, since the subject is located near the center of the lens 101a, the range of both the light flux 115 and the light flux 116 of the light emitted from the subject is determined by the aperture 101b. Therefore, the A _difference is used as the A image, and the B _difference is used as the B image. On the other hand, in the second region 1204, since the subject is away from the position corresponding to the center of the lens 101a, vignetting occurs in the light beam 116 due to the lens 101a. In this case, the A _difference is used as the A image, and the B _{large aperture} or B _{small aperture} is used as the B image. On the contrary, in the third region 1205, vignetting occurs in the light beam 115 due to the lens 101a. In this case, A _{large aperture} or A _{small aperture} is used as the A image _{, and} B _difference is used as the B image. In the example of FIG. 13, since the window of interest 1202 is within the first region 1203, the control unit 107 selects the A _difference as the A image and the B _difference as the B image to execute window matching processing. The width distribution of the regions 1203, 1204, and 1205 can be determined from the relationship between the light flux and vignetting caused by the lens 101a when designing the imaging device 100. After S906 in FIG. 11, the process advances to S907.

In S907, the control unit 107 performs window matching processing on the windows of interest in the A image and B image determined in S906, and searches for corresponding locations between the images. Next, in S908, the control unit 107 calculates the amount of image shift between the A image and the B image from the corresponding location searched in S907. After S908, the process advances to S909.

In S909, the control unit 107 calculates the distance from the amount of image shift calculated in S908. As a method for calculating the distance from the amount of image shift, there is a method using, for example, a reference table. The storage unit 103 includes a reference table having data representing the relationship between the amount of image shift and the distance based on the focal length of the lens 101a, the F value of the aperture 101b, the subject distance, the focusing distance, and the position of the subject on the image. memory is retained. For example, the focal length of the lens 101a is 50 mm, the F value of the aperture 101b is 1.2 and 1.4, the subject distance is 1.5 m, the focusing distance is 1.0 m, and the position of the subject on the image is within the area 1203 ( Figure 13). At this time, the amount of image shift between the A _difference and the B _difference is 5 pixels from the reference table. In other words, 1.5 m is obtained as the subject distance corresponding to 5 pixels. Note that values of parameters that are not in the reference table can be calculated from values close to the reference table data by known processing (interpolation processing, extrapolation processing, etc.).

The process advances from S909 to S910, and the control unit 107 determines whether processing has been completed for all windows. If it is determined that the processing has been completed for all windows, the loop processing is exited and the series of processing is ended. If there is an unprocessed window, the process moves to S906 and continues processing.

[First modified embodiment]
In this modified embodiment, noise countermeasure processing will be described. In the above description, an example was given in which photographing is performed twice by changing the aperture amount of the aperture 101b, and one image each of A _{large aperture} , A _{small aperture} , B _{aperture large} , and B _{aperture small} is obtained. The smaller the change in the aperture of the diaphragm, the more narrow the images created by the light beams 115 and 116 that have passed through a position away from the center of the lens 101a can be left in the A _difference and the B _difference , respectively. However, if the change in the aperture of the diaphragm is too small, image data with almost no difference in brightness will be subtracted from each other. As a result, the difference image (A _difference or B _difference ) becomes a dark image, which may make it difficult to search for a corresponding location in the image where noise is noticeable.

Therefore, the imaging device 100 performs imaging two or more times for each aperture amount of the aperture 101b, and adds the plurality of image data. This makes it possible to generate bright differential image data A _difference and B _difference and reduce noise. The image data taken the i-th time are respectively expressed as A _{large aperture, i} , A _{small aperture, i} , B _{large aperture, i} , and B _{small aperture, i} . The A _difference and the B _difference can be calculated from (Equation 2).
The variable N in (Equation 2) represents the number of times of photographing for each aperture amount of the aperture 101b. In other words, a total of 2×N images will be taken. However, if the value of N is too large, it will take time to photograph, and the movement of the image during that time may become large. Therefore, it is preferable to photograph a stationary subject with the imaging device 100 fixed on a tripod or the like.

This modified embodiment has the effect that the corresponding portions of the images can be searched with higher accuracy because the noise related to the generated images of the A _difference and the B _difference is reduced.

[Second modified embodiment]
In this modified embodiment, an example will be shown in which the aperture amount of the aperture is changed three or more times and the image is photographed. In the embodiment described above, the case where vignetting occurs in one of the light beams 115 and 116 has been described. An example was given in which a normally captured image is used for the image corresponding to the light flux where vignetting has occurred, and a difference image is used for the image corresponding to the light flux where vignetting is not caused, in order to search for the corresponding part of the image. . The imaging device of this modified embodiment performs imaging by changing the aperture amount of the aperture 101b three or more times. The control unit 107 generates difference images for both the A image and the B image, and then searches for corresponding locations in the images. For example, the aperture amount of the aperture 101b is expressed as large aperture, medium aperture, and small aperture, respectively. As explained with reference to FIG. 10, the range of the luminous flux 115 changes by changing from a large aperture to a medium aperture, but the luminous flux 116 does not change.

FIG. 14 shows the relationship between the aperture 101b and the luminous flux when the aperture amount of the aperture 101b is made small. The aperture amount of the aperture 101b is small enough to limit the range of the light beam 116, and there is no vignetting of the light beam. On the other hand, since the light beam 115 is completely blocked by the aperture 101b, image A is not generated. Therefore, as shown in Equation 3, differential image data is generated for image A by determining the image difference between large aperture and medium aperture. For the B image, difference image data is generated by determining the difference between images by a combination of a large aperture and a small aperture, or a combination of a medium aperture and a small aperture.
For example, A _difference is generated from the difference between A image data acquired by shooting at F value 1.2 and F value 1.4, and A B _difference is generated from the difference in B image data.

By performing the above processing, it is possible to obtain a clearer B image than when using a normally captured image as the image corresponding to the B image, so the search for the corresponding part of the image can be performed with higher precision. I can do it. It is possible to calculate the object distance with high accuracy, and by applying it to 3D mapping of the object, it is possible to generate more detailed 3D objects. Furthermore, by applying the above processing to the autofocus function of the imaging device, it is possible to more accurately focus on the subject.

[Third modified embodiment]
In the above description, an example is described in which a difference image based on an image photographed by changing the aperture amount of the diaphragm 101b is used for at least one of the A image and the B image among all images captured in the image. I mentioned it. In this modified embodiment, an example will be shown in which normal processing (processing using a viewpoint image instead of a difference image) is performed on a subject near the in-focus position or a dark difference image.

For objects near the in-focus position, there is little blurring even in normally photographed images, so there may be no need to use a method using differential images. Therefore, the imaging device of this modified embodiment uses the A image and the B image for the first subject whose depth information is within a predetermined range (for example, near the in-focus position). Furthermore, for a second subject whose depth information is different from that of the first subject, at least one of the A _difference and the B _difference is used, similar to the embodiment described above. This makes it possible to simplify the calculation of the amount of image shift.

Furthermore, if the image of _difference A or _difference B becomes a dark image containing a lot of noise, calculation of the amount of image shift may become unstable. Therefore, it is effective to calculate the amount of image shift using the A image and the B image. In other words, when the control unit 107 determines that the A _difference or the B _difference is less than the threshold, it calculates the amount of image shift using the A image and the B image, and when the controller 107 determines that the A _difference or the B _difference is greater than or equal to the threshold. , the image shift amount is calculated using at least one of the A _difference and the B _difference .

When detecting the amount of image shift between images, the larger the distance between two light beams on the pupil of the lens, the larger the amount of image shift, and the higher the accuracy of calculating the distance to the subject. It is preferable to capture the image and generate the A image and the B image. To achieve this, you can increase the aperture of the lens aperture, but opening the aperture reduces the depth of field, so images of objects located outside the in-focus range will be blurred. As a result, it is necessary to search for a corresponding location between the blurred images A and B, making it difficult to detect the amount of image shift.

In the embodiment described above, by changing the aperture amount of the aperture and photographing a plurality of times to generate a difference image, corresponding points between images are searched for using images with less blur and a large amount of image shift. Thereby, it is possible to perform more accurate distance measurement (acquisition of depth information) for a subject located at a distance away from the in-focus position. Although an example has been shown in which each sub-pixel divided into two corresponding to each microlens in the pixel section has a photoelectric conversion section, the present invention is directed to an embodiment in which the pixel section has a photoelectric conversion section divided into three or more sections. It is possible to apply to

[Other embodiments]
The present invention provides a system or device with a program that implements one or more functions of the embodiments described above via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Embodiments of the present disclosure include the following configurations, methods, and programs.
(Configuration 1)
An image processing device capable of acquiring depth information in the depth direction of a subject in an image from a plurality of viewpoint images having different viewpoints, the image processing device comprising:
acquisition means for acquiring a plurality of images based on light passing through different pupil partial regions in the imaging optical system;
a control unit that performs control to change the aperture amount of the aperture of the imaging optical system, and performs control to obtain the depth information using a plurality of images captured with different aperture amounts of the aperture,
The acquisition means is configured to acquire first and second viewpoint images captured with a first aperture, and third and fourth viewpoint images captured with a second aperture smaller than the first aperture. get
The control means generates a first difference image from the first viewpoint image and the third viewpoint image, and generates a second difference image from the second viewpoint image and the fourth viewpoint image. , an image processing device that controls acquiring the depth information using one or more of the first and second difference images.
(Configuration 2)
The image processing according to configuration 1, wherein the control means performs control to acquire the depth information using the first and second difference images in a first region including a central part of the image. Device.
(Configuration 3)
The control means may control the first difference image and the second or fourth viewpoint image, or the second difference image and the first or fourth viewpoint image in a second region different from the first region. The image processing device according to configuration 2, wherein the image processing device performs control for acquiring the depth information using a viewpoint image of No. 3.
(Configuration 4)
The control means controls imaging two or more times each time the aperture amount is changed,
The acquisition means is configured to acquire a plurality of the first and second viewpoint images captured with the first aperture amount and a plurality of third and fourth viewpoint images captured with the second aperture amount. The image processing device according to any one of configurations 1 to 3, characterized in that the image processing device acquires images.
(Configuration 5)
The control means generates the first difference image based on the difference between the plurality of first viewpoint images and the plurality of third viewpoint images, and generates the first difference image based on the difference between the plurality of second viewpoint images and the plurality of third viewpoint images. The image processing device according to configuration 4, wherein the image processing device performs control to generate the second difference image based on the difference between the second difference image and the second difference image.
(Configuration 6)
The control means uses a combination of the aperture amounts to obtain the depth information when generating the first or second difference image using a plurality of images captured with different aperture amounts of the diaphragm. The image processing device according to any one of configurations 1 to 3, characterized in that the image processing device changes an image.
(Configuration 7)
The acquisition means further acquires fifth and sixth viewpoint images captured with a third aperture smaller than the second aperture,
The control means performs control to generate the second difference image from the second viewpoint image and the sixth viewpoint image, or from the fourth viewpoint image and the sixth viewpoint image. The image processing device according to configuration 6.
(Configuration 8)
The control means controls acquiring the depth information for the first subject using the first and second viewpoint images or the third and fourth viewpoint images, and From configuration 1, characterized in that control is performed to acquire the depth information of a second subject whose depth information is different from that of the subject using one or more of the first and second difference images. 7. The image processing device according to any one of 7.
(Configuration 9)
When the data of the first or second difference image is less than a threshold, the control means controls the depth information using the first and second viewpoint images or the third and fourth viewpoint images. control to acquire the depth information, and if data of the first or second difference image is equal to or greater than a threshold, control to acquire the depth information using one or more of the first and second difference images; 8. The image processing device according to any one of configurations 1 to 7, characterized in that:
(Configuration 10)
The configuration is characterized in that the acquisition means acquires the plurality of viewpoint images corresponding to each of the photoelectric conversion units using an image sensor including a plurality of microlenses and a plurality of photoelectric conversion units corresponding to each microlens. 10. The image processing device according to any one of 1 to 9.
(Configuration 11)
The image processing device according to any one of configurations 1 to 10,
An imaging device including an imaging element.
(Configuration 12)
The pixel section of the image sensor includes a microlens and a plurality of photoelectric conversion sections corresponding to the microlens,
The imaging device according to configuration 11, wherein the plurality of viewpoint images are acquired from images corresponding to each of the photoelectric conversion units.
(Configuration 13)
The imaging device according to configuration 11 or configuration 12, further comprising a notification unit that provides notification for suppressing movement of the imaging device when a mode for acquiring the depth information is set.
(Method 1)
An image processing method executed by an image processing device capable of acquiring depth information in a depth direction of a subject in an image from a plurality of viewpoint images having different viewpoints, the method comprising:
an acquisition step of acquiring a plurality of images based on light respectively passing through different pupil partial regions in the imaging optical system;
a control step of controlling to change the aperture of the aperture of the imaging optical system and controlling to acquire the depth information using a plurality of images captured with different apertures of the aperture;
In the acquisition step, first and second viewpoint images are captured with a first aperture, and third and fourth viewpoint images are captured with a second aperture smaller than the first aperture. is obtained,
In the control step, a first difference image is generated from the first viewpoint image and the third viewpoint image, and a second difference image is generated from the second viewpoint image and the fourth viewpoint image. , control is performed to obtain the depth information using one or more of the first and second difference images.
(program)
A program that causes a computer to execute each step described in Method 1.

100 Imaging device 101 Imaging section 101b Aperture 102 Arithmetic section 107 Control section

Claims (15)

An image processing device capable of acquiring depth information in the depth direction of a subject in an image from a plurality of viewpoint images having different viewpoints, the image processing device comprising:
acquisition means for acquiring a plurality of images based on light passing through different pupil partial regions in the imaging optical system;
a control unit that performs control to change the aperture amount of the aperture of the imaging optical system, and performs control to obtain the depth information using a plurality of images captured with different aperture amounts of the aperture,
The acquisition means is configured to acquire first and second viewpoint images captured with a first aperture, and third and fourth viewpoint images captured with a second aperture smaller than the first aperture. and
The control means generates a first difference image from the first viewpoint image and the third viewpoint image, and generates a second difference image from the second viewpoint image and the fourth viewpoint image. , an image processing device that controls acquiring the depth information using one or more of the first and second difference images. The image according to claim 1, wherein the control means performs control to acquire the depth information using the first and second difference images in a first region including the center of the image. Processing equipment. The control means may control the first difference image and the second or fourth viewpoint image, or the second difference image and the first or fourth viewpoint image in a second region different from the first region. The image processing device according to claim 2, wherein control is performed to obtain the depth information using three viewpoint images. The control means controls imaging two or more times each time the aperture amount is changed,
The acquisition means is configured to acquire a plurality of the first and second viewpoint images captured with the first aperture amount and a plurality of third and fourth viewpoint images captured with the second aperture amount. The image processing device according to claim 1, wherein the image processing device acquires an image. The control means generates the first difference image based on the difference between the plurality of first viewpoint images and the plurality of third viewpoint images, and generates the first difference image based on the difference between the plurality of second viewpoint images and the plurality of third viewpoint images. The image processing device according to claim 4, wherein the image processing device performs control to generate the second difference image based on a difference from a fourth viewpoint image. The control means uses a combination of the aperture amounts to obtain the depth information when generating the first or second difference image using a plurality of images captured with different aperture amounts of the diaphragm. The image processing device according to claim 1, wherein the image processing device changes an image. The acquisition means further acquires fifth and sixth viewpoint images captured with a third aperture smaller than the second aperture,
The control means performs control to generate the second difference image from the second viewpoint image and the sixth viewpoint image, or from the fourth viewpoint image and the sixth viewpoint image. The image processing device according to claim 6. The control means controls acquiring the depth information for the first subject using the first and second viewpoint images or the third and fourth viewpoint images, and 2. Control is performed to obtain depth information of a second object having different depth information from the object using one or more of the first and second difference images. The image processing device described in . When the data of the first or second difference image is less than a threshold, the control means controls the depth information using the first and second viewpoint images or the third and fourth viewpoint images. control to acquire the depth information, and if data of the first or second difference image is equal to or greater than a threshold, control to acquire the depth information using one or more of the first and second difference images; The image processing device according to claim 1 , wherein the image processing device performs the following steps. A claim characterized in that the acquisition means acquires the plurality of viewpoint images corresponding to each of the photoelectric conversion units using an image sensor including a plurality of microlenses and a plurality of photoelectric conversion units corresponding to each microlens. The image processing device according to item 1. An image processing device according to any one of claims 1 to 10,
An imaging device including an imaging element. The pixel section of the image sensor includes a microlens and a plurality of photoelectric conversion sections corresponding to the microlens,
The imaging device according to claim 11, wherein the plurality of viewpoint images are acquired from images corresponding to each of the photoelectric conversion units. The imaging device according to claim 11, further comprising a notification unit that provides notification for suppressing movement of the imaging device when a mode for acquiring the depth information is set. An image processing method executed by an image processing device capable of acquiring depth information in a depth direction of a subject in an image from a plurality of viewpoint images having different viewpoints, the method comprising:
an acquisition step of acquiring a plurality of images based on light respectively passing through different pupil partial regions in the imaging optical system;
a control step of controlling to change the aperture of the aperture of the imaging optical system and controlling to acquire the depth information using a plurality of images captured with different apertures of the aperture;
In the acquisition step, first and second viewpoint images are captured with a first aperture, and third and fourth viewpoint images are captured with a second aperture smaller than the first aperture. is obtained,
In the control step, a first difference image is generated from the first viewpoint image and the third viewpoint image, and a second difference image is generated from the second viewpoint image and the fourth viewpoint image. , control is performed to obtain the depth information using one or more of the first and second difference images. A program that causes a computer to execute each step according to claim 14.

Images