JP2018007205A - Image processing apparatus, imaging apparatus, control method for image processing apparatus, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of generating a picked-up image in which noise is reduced, and depth information corresponding to the picked-up image.SOLUTION: An imaging apparatus 100 comprises: an imaging device 11 including multiple pixel groups, each of pixels in the pixel group including multiple photoelectric conversion elements; a system control part 31 performing a first read operation for reading signals from partial photoelectric conversion elements from among the multiple photoelectric conversion elements as a viewpoint image and a second read operation for mixing the signals from the multiple photoelectric conversion elements and reading the mixed signal as a first composite image with respect to the multiple pixel groups; and an image processing part 13 for generating depth information corresponding to the first composite image. The image processing part 13 generates the depth information using the viewpoint image that the system control part 31 has read through the first read operation, and depth information corresponding to the first composite image that the system control part 31 obtains by performing the second read operation without performing the first read operation, is generated by using the depth information that is generated by using the viewpoint image.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image processing apparatus, an imaging apparatus, an image processing apparatus control method, and a program for acquiring information related to a defocus distribution.

  There has been proposed an imaging apparatus that can divide an exit pupil of a photographing lens into a plurality of areas and simultaneously photograph a plurality of viewpoint images corresponding to the divided pupil areas. Patent Document 1 discloses an imaging apparatus using a two-dimensional imaging element in which one microlens and a plurality of divided photoelectric conversion units are formed for one pixel. The divided photoelectric conversion unit is configured to receive a pupil partial region which is divided into pupils through one microlens and has different exit pupils of the photographing lens. A plurality of viewpoint images corresponding to the divided pupil partial regions can be generated from the respective signals received by these photoelectric conversion units. Patent Document 2 discloses that a captured image is generated by adding all signals received by divided photoelectric conversion units. Further, Patent Document 3 discloses an imaging device that creates a defocus map based on left-right parallax or vertical parallax from the output of a plurality of viewpoint images obtained by pupil division.

US Pat. No. 4,410,804 JP 2001-083407 A JP 2009-165115 A

  When reading from a pupil-divided photoelectric conversion unit, when reading a combined image that combines (mixed) viewpoint images and multiple viewpoint images to generate depth information, compared to reading a combined image without reading the viewpoint image In some cases, the amount of noise in the composite image increases.

  In view of the above problems, an object of the present invention is to provide an image processing apparatus capable of generating a captured image with reduced noise and depth information corresponding to the captured image.

  In order to solve the above-described problem, an imaging device of the present invention includes a plurality of pixel groups, each pixel of the pixel group including a plurality of photoelectric conversion elements, and a part of the plurality of photoelectric conversion elements. A plurality of first readout operations for reading out signals from the photoelectric conversion elements as viewpoint images, and a second readout operation for reading out signals from the plurality of photoelectric conversion elements as a first composite image. Read control means for the pixel group, and generation means for generating depth information corresponding to the first composite image, wherein the read control means performs the first read operation. The depth information is generated using the viewpoint image read out by the step, and the read control unit corresponds to the first composite image obtained by performing the second read operation without performing the first read operation. Degree information is generated using the depth information is generated by using the viewpoint images.

  ADVANTAGE OF THE INVENTION According to this invention, the image processing apparatus which can produce | generate the depth information corresponding to the captured image which reduced noise, and the captured image can be provided.

It is a figure which shows the structural example of an imaging device. It is a figure which shows the structural example of an imaging part. It is a flowchart which shows operation | movement of an imaging device. It is a flowchart which shows the production | generation process of the defocus map for still images.

  FIG. 1 is a diagram illustrating a configuration example of an imaging apparatus according to an embodiment of the present invention. The imaging apparatus 100 includes a photographing lens 10, an imaging element 11, an A / D converter 12 that converts an analog signal output from the imaging element 11 into a digital signal, and an image processing unit 13. The imaging apparatus 100 is a digital camera, a video camera, or the like, but is not limited thereto. The imaging element 11 is an imaging unit that includes a plurality of pixel groups, and each pixel of the pixel group includes a plurality of photoelectric conversion elements that convert an optical image into an electrical signal, and can simultaneously acquire a plurality of viewpoint images. The image processing unit 13 performs predetermined demosaic processing, white balance adjustment, color interpolation, reduction / enlargement, filtering, and the like on the data from the A / D converter 12 or the data from the memory control unit 20. Further, various image processing such as defocus map generation and motion vector calculation of a plurality of images is performed.

  The imaging apparatus 100 further includes a memory control unit 20, a D / A converter 21, an image display unit 22, an image display memory 23, and a memory 24. The memory control unit 20 controls the A / D converter 12 and the image processing unit 13. The data output from the A / D converter 12 is sent to the image display memory 23 via the image processing unit 13 and the memory control unit 20, or the data of the A / D converter 12 is sent directly to the memory control unit 20. Or it is written in the memory 24. The image display unit 22 is an image display unit having a TFT LCD or the like, and the display image image data written in the image display memory 23 is displayed by the image display unit 22 via the D / A converter 21. The live view function can be realized by sequentially displaying the captured image data using the image display unit 22.

  The memory 24 is a memory for storing captured still images and moving images, and has a sufficient storage capacity for storing a predetermined number of still images and moving images for a predetermined time. Further, the memory 24 stores one or more OTF data 110 for generating a recovery filter. This makes it possible to write a large amount of images to the memory 24 at high speed even in continuous shooting where a plurality of still images are continuously captured. The memory 24 can also be used as a work area for the system control unit 31.

  The imaging apparatus 100 further includes a recording unit 30, a system control unit 31, SW32, SW33, and a mode dial 40. The recording unit 30 includes a recording medium. The system control unit 31 is a system control unit that controls the entire imaging apparatus 100. SW32 is a shutter switch, which is turned on when a shutter button (not shown) is being operated (half-pressed), and AF (auto focus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, EF (flash pre-flash). ) Instruct the start of operations such as processing. SW33 is a shutter switch that is turned ON when a shutter button (not shown) is pressed (fully pressed). The SW 33 is turned ON to instruct the start of a series of processes from reading a signal from the image sensor 11 to a recording process for writing image data to the recording unit 30. In a series of processing, a signal is read from the image sensor 11, development processing using calculation in the image processing unit 13 or the memory control unit 20 via the A / D converter 12 and the memory control unit 20, and an image from the memory 24. Data reading and compression processing are performed. Then, a recording process for writing image data into the recording unit 30 is performed.

  FIG. 2 shows a conceptual diagram of the correspondence relationship between the image sensor of this embodiment and pupil division. FIG. 2A is a schematic diagram illustrating a configuration example of a pixel array of the image sensor 11. In FIG. 2A, the direction orthogonal to the paper surface is defined as the z direction, and the near side is defined as the + z direction. The first direction orthogonal to the z direction is defined as the x direction, and the right direction in FIG. 2A is defined as the + x direction. Further, the second direction orthogonal to the z direction is defined as the y direction, and the upward direction in FIG. 2A is defined as the + y direction. FIG. 2A shows a schematic diagram in which a certain pixel 200 is enlarged. The pixel 200 includes a microlens 201 and pupil division pixels 202A and 203B that are a pair of photoelectric conversion elements. In the image sensor 11, a plurality of pixels 200 are regularly arranged in a two-dimensional array. Signals are read as viewpoint images from some of the plurality of photoelectric conversion elements. In the present embodiment, it is assumed that an A image and a B image are read out as a pair of images from pupil division pixels 202A and 203B, which are regularly arranged readout units. When a combined image obtained by combining (mixing) a viewpoint image and a plurality of viewpoint images to read depth information such as a defocus map is read in this order, the combined image AB image is read without reading the viewpoint image A image (B image). In some cases, the amount of noise in the composite image is larger than in the case of reading. This may be because the Null reading timing for reading a component for removing noise caused by dark current or the like is separated from the composite image accumulation period. Therefore, in this embodiment, as will be described later, the noise deterioration of the AB image is reduced by not reading the A image when reading the still image synthetic image AB image.

  FIG. 2B is a schematic diagram illustrating the relationship between the pupil of the imaging optical system and the photoelectric conversion unit of the imaging element. The image sensor 11 is disposed in the vicinity of the imaging surface of the photographing lens 10, and the light flux from the subject passes through the exit pupil 800 of the photographing lens 10 and enters each corresponding pixel. The size of the exit pupil 800 varies depending on the size of the stop and the size of the lens frame that holds the lens. The pupil partial areas 801A and 802B are substantially conjugated with each other by the microlens and the light receiving surfaces of the pupil divided pixels 202A and 203B divided into two in the x direction. Therefore, the light flux that has passed through each pupil partial region is received by each pupil division pixel that is in a conjugate relationship. When the number of pupil divisions in the horizontal direction is M and the number of pupil divisions in the vertical direction is N, the exit pupil 800 of the photographing lens 10 is divided into different pupil partial regions with the number of pupil divisions Np = M × N. When the aperture value of the taking lens 10 is F, the effective aperture value of the pupil partial area is approximately (√ (M × N)) × F. The pupil region 803 is a pupil region (entire aperture region) that can receive light in the entire pixel 200 when all the photoelectric conversion units divided into M × N are combined. For example, when the number of pupil divisions is M = 2 and N = 1, the effective aperture value of the pupil partial area is √2 times F. That is, one pupil-divided image has a depth of field that is one step deeper than that of the full aperture image, and a dark image is obtained.

  With the above configuration, a pair of light beams that pass through different regions in the pupil of the photographing lens 10 in FIG. 1 are formed as a pair of optical images, and a viewpoint image that is a pair of images is obtained. By adding (combining) all signals for each pixel of a pair of viewpoint images, an image having a resolution of the effective number of pixels can be generated. Here, an image obtained by mixing (adding) the A image and the B image before reading them from the image sensor 11 and A / D converting (first synthesized image) is referred to as an AB image.

3 and 4 are flowcharts showing processing of the imaging apparatus 100 of the present embodiment. The operation of the imaging apparatus 100 will be described with reference to FIGS. 3 and 4.
When the process is started, the system control unit 31 initializes flags and control variables, and initializes the image display of the image display unit 22 to an OFF state or the like.
In step S301, the system control unit 31 determines the setting position of the mode dial 40. If the mode dial 40 is set to power OFF, the process proceeds to S302. In S302, the system control unit 31 changes the display of each display unit to an end state, records necessary parameters, setting values, and setting modes including flags, control variables, and the like in a non-volatile memory. Process. If the mode dial 40 is set to a mode other than the shooting mode, the process proceeds to S303. In S303, the system control unit 31 executes processing according to the selected mode, and returns to S301 when the processing is completed. If the mode dial 40 is set to the shooting mode, the process proceeds to S304.

In step S304, the system control unit 31 reads the A and B images that are viewpoint images and the A + B image that is a composite image for display (first reading operation). Specifically, the image sensor 11 is exposed according to photometric data stored in the internal memory or the memory, and after a predetermined time has elapsed according to the photometric data, the charge signals of the A and B images are read from the image sensor 11 respectively. . Then, the A and B image data are stored in the memory 24 via the A / D converter 12, the image processing unit 13, the memory control unit 20, or directly from the A / D converter 12 via the memory control unit 20. Write. Further, the image processing unit 13 generates an A + B image by adding pixel values for each target pixel position (x, y) of the A image and the B image, as shown in Expression (1), and the generated A + B image is Write to memory 24.
A + B (x, y) = A (x, y) + B (x, y) (1)

In step S305, the image processing unit 13 calculates the defocus amount for each pixel position of interest using each signal of the A image and the B image, which are pupil-divided images acquired by the shooting in step S304, and performs defocusing. A focus map (DEF1, depth information) is generated. The defocus amount is information related to the depth distribution of the subject and represents the value of the defocus map data. Details of the defocus amount calculation are shown below.
The signal sequence of the A image at the target pixel position is expressed as E (1) to E (m), and the signal sequence of the B image is expressed as F (1) to F (m). The two signal sequences are expressed using Equation (2) while relatively shifting the B image signal sequences F (1) to F (m) with respect to the A image signal sequences E (1) to E (m). The correlation amount C (k) is calculated for the gap amount k.
C (k) = Σ | E (n) −F (n + k) | (2)
In Equation (2), Σ operation means an operation for calculating the sum for n. The range taken by n and n + k in the Σ operation is limited to a range of 1 to m. The shift amount k takes an integer value and represents a relative shift amount with the detection pitch of a pair of data as a unit.

Among the calculation results of Expression (2), the correlation amount C (k) is minimized at the shift amount (shift amount k) in which the correlation between the pair of signal sequences is high. In the following, k when the discrete correlation amount C (k) is minimum is denoted as kj. The shift amount x that gives the minimum value C (x) with respect to the continuous correlation amount is calculated using the three-point interpolation processing according to the following equations (3) to (5).
x = kj + D / SLOP (3)
D = {C (kj−1) −C (kj + 1)} / 2 (4)
SLOP = MAX {C (kj + 1) -C (kj), C (kj-1) -C (kj)}
... (5)

The defocus amount (denoted as DEF) can be obtained from equation (6) from the shift amount x calculated by equation (3).
DEF = KX · PY · x (6)
KX shown in Expression (6) is a conversion coefficient determined by the size of the opening angle of the center of gravity of the light beam passing through the pair of pupil regions. PY is a detection pitch. When the area to be referred to is an overexposure area or an underexposure area, the term | E (n) −F (n + k) | For this reason, kj that minimizes the correlation amount C (k) cannot be calculated correctly. As a result, the calculated defocus amount is not an accurate value. Defocus map data representing the spatial distribution of the defocus amount calculated as described above (distribution on the two-dimensional plane of the captured image) is recorded in the memory 24.

In step S306, the image processing unit 13 acquires a display image image. Specifically, first, the system control unit 31 reads the A + B image written in the memory 24 and the defocus map data via the memory control unit 20. The image processing unit 13 sequentially performs correction processing using color processing and a defocus map on the read A + B image to generate a display image image, and writes the display image image data in the memory 24.
In step S307, the system control unit 31 reads the display image image data from the memory 24, and transfers the display image data to the image display memory 23 via the memory control unit 20. Then, the display image data written in the image display memory 23 is displayed in live view on the image display unit 22.

In step S308, the system control unit 31 determines whether the still image shooting SW 33 is not pressed (whether the switch is fully pressed). If not, the process returns to S301. If SW33 is pressed, the process proceeds to S309.
In S309, the system control unit 31 reads an AB image for a still image (for a captured image) (second read operation). Specifically, according to the photometric data stored in the internal memory or the memory 24, a shutter having an aperture function is opened according to the aperture value, and if necessary, a flash is emitted to expose the image sensor 11. According to the photometric data, the image sensor 11 waits for the exposure to end, closes the shutter, and reads the AB image signal from the image sensor 11. Then, an AB image for a captured image is stored in the memory 24 via the A / D converter 12, the image processing unit 13, the memory control unit 20, or directly from the A / D converter 12 via the memory control unit 20. Write data.

In step S310, a defocus map (DEF2) for the AB image is generated to perform development processing such as color processing of the AB image acquired in step S309 and correction processing using the defocus map. Details of the processing for generating a defocus map for a captured image in step S310 will be described later with reference to FIG.
In step S311, the image processing unit 13 performs development processing on the AB image using a defocus map (DEF2) for a captured image, performs image compression processing, and generates a captured image (still image). Specifically, first, the system control unit 31 reads the defocus map generated in step S310 from the memory 24. The image processing unit 13 performs development processing such as color processing and correction processing using the read defocus map, and performs image compression processing according to the set mode to generate a captured image. Then, the system control unit 31 executes a recording process for writing captured image data to the recording unit 30.

FIG. 4 is a flowchart showing detailed processing for generating a defocus map for a captured image in step S310 in FIG. The captured image is generated without acquiring the viewpoint image (without acquiring the signals of the A image and the B image), and a defocus map corresponding to the captured image having only the signal of the AB image is stepped. It cannot be generated from the A and B image signals as in S305. Processing for generating a defocus map corresponding to a captured image even when signals A and B are not acquired during shooting will be described with reference to FIG.
In step S401, the system control unit 31 stores the A + B image for the display image recorded in step S304 and the defocus map (depth information; hereinafter referred to as DEF1) generated from the A image and the B image in S305. Read from. The signal and the defocus map read at this time are acquired before the still image shooting time and are the one immediately before the still image shooting time.
In step S402, the system control unit 31 reads the still image AB image recorded in step S309 from the memory 24.

  In step S403, the image processing unit 13 calculates the motion vectors of the A + B image for the display image and the AB image for the captured image. The motion vector calculates the sum of absolute differences by block matching for each of a plurality of blocks obtained by dividing two images into blocks. The smaller the difference absolute value sum, the higher the block matching correlation. A shift amount that minimizes the sum of absolute differences in block matching is defined as a motion vector (Vx, Vy). In this embodiment, the motion vector is calculated by block matching, but may be calculated by an optical flow.

In step S404, the image processing unit 13 performs interpolation based on the motion vector (Vx, Vy) calculated for DEF1, and generates a defocus map (hereinafter referred to as DEF2) for the captured image. The generation of DEF2 may be performed by moving a pixel corresponding to the motion vector with respect to each pixel of DEF1, as shown in the following equation (8).
DEF2 (x, y) = DEF1 (x−Vx, y−Vy) (8)

In step S405, the image processing unit 13 performs a shaping process on DEF2 generated in step S404. The image processing unit 13 performs a bilateral filter process on the defocus map using the AB image for the captured image as a shaping image, and the system control unit 31 records the shaped defocus map in the memory 24. Details of the bilateral filter processing will be described below. If the filter result at the target pixel position p is denoted by Jp, this is expressed by the following equation (9).
Jp = (1 / Kp) ΣI1q · f (| p−q |) · g (| I2p−I2q |)
... (9)

The meaning of each symbol in Formula (9) is as follows.
q: peripheral pixel position Ω: integration target region centered on the target pixel position p Σ: integration in the range of q∈Ω I1q: defocus map signal value at the peripheral pixel position q f (| p−q |): target pixel position Gaussian function centered on p I2p: image signal value for shaping at target pixel position p I2q: image signal value for shaping at peripheral pixel position q g (| I2p−I2q |): centering on image signal value I2p for shaping Gapsian function Kp: normalization coefficient, integrated value of f · g weights.

When the difference between the signal value I2p at the target pixel position p and the signal value I2q at the peripheral pixel position q is small, that is, when the pixel value of the target pixel and the pixel value of the peripheral pixel are close in the shaping image, The f · g weight (smoothing weight) increases.
By the shaping process in S405, the contour of the subject in the defocus map matches the correct contour. The defocus map shaping process not only has the effect of matching the distance signal value with the correct contour, but is also effective in correcting the distortion of the defocus map that occurs during frame interpolation.

  In the present embodiment, a pair of images, A and B images, are read from the image sensor 11 and A / B images obtained by performing A / D conversion on the A and B images as display images are obtained. However, the present invention is not limited to this, and for example, the AB image and the A image may be read from the image sensor 11 and the B image may be acquired therefrom. Also in this case, compared to the case of a captured image that acquires only an AB image, the amount of noise increases because two images, the AB image and the A image, are read, and therefore, the defocus map corresponding to the captured image and the captured image with less noise The present invention for generating can be applied.

  In the present embodiment, the defocus map has been described. However, the present invention is not limited to this, and any information (depth information) representing various depths generated from the viewpoint image may be used. The depth information is, for example, distribution information (parallax map, image shift map) of the amount of parallax of the pupil-divided image, distribution information that indicates the relative positional relationship of the subject in the depth direction, and the like. The depth information is used in correction processing performed when a captured image is generated in step S311. The correction process includes, for example, refocusing, background blurring, area determination, and the like, but is not limited thereto.

  As described above, according to the present embodiment, a captured image can be generated from an AB image with less noise that is added before an A image and a B image are read from the image sensor 11. Also, a defocus map corresponding to the captured image can be generated by interpolation from the defocus map for live view display. Therefore, it is possible to generate a defocus map without acquiring a plurality of noisy viewpoint images in the captured image and perform correction using the defocus map for the captured image with less noise.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

13 Image Processing Unit 31 System Control Unit 100 Imaging Device

Claims (10)

An image sensor including a plurality of pixel groups, each pixel of the pixel group including a plurality of photoelectric conversion elements;
A first readout operation for reading out signals from some of the plurality of photoelectric conversion elements as a viewpoint image, and mixing the signals from the plurality of photoelectric conversion elements as a first composite image A read control means for performing a second read operation for reading a plurality of pixel groups;
Generating means for generating depth information corresponding to the first composite image,
The generation means generates the depth information using the viewpoint image read by the read control means by the first read operation,
The depth information corresponding to the first composite image obtained by performing the second read operation without performing the first read operation by the read control unit is the depth information generated using the viewpoint image. An imaging apparatus characterized by being generated by using.   The imaging apparatus according to claim 1, wherein the depth information is distribution information that indicates a relative positional relationship of the subject in the depth direction. Synthesizing means for synthesizing the viewpoint images read out by the first reading operation to generate a second synthesized image;
The generation means detects a motion vector from the first composite image and the second composite image, and generates depth information corresponding to the first composite image using the viewpoint image and the motion vector. The imaging device according to claim 1, wherein:   The second synthesized image used for detecting the motion vector is generated by synthesizing the viewpoint image read out in the first readout operation immediately before the second readout operation for reading out the first synthesized image. The imaging device according to claim 3, wherein the imaging device is a second synthesized image.   5. The generation unit according to claim 1, wherein the generation unit performs shaping processing on depth information generated using the viewpoint image and the motion vector, and generates depth information corresponding to the first composite image. The imaging device according to any one of the above.   The imaging apparatus according to claim 5, wherein the shaping process is a bilateral filter process.   The imaging apparatus according to claim 1, further comprising a correction unit that corrects an image based on the depth information. A first readout operation for reading out signals from some of the plurality of photoelectric conversion elements as viewpoint images, and a second for reading out signals from the plurality of photoelectric conversion elements as a combined image. Reading control means for performing the reading operation for a plurality of pixel groups;
Generating means for generating depth information corresponding to the composite image,
The generation means generates the depth information using the viewpoint image read by the read control means by the first read operation,
Depth information corresponding to a composite image obtained by performing the second read operation without performing the first read operation by the read control unit is generated using depth information generated using the viewpoint image. An image processing apparatus. A control method for an imaging apparatus that includes a plurality of pixel groups, and each pixel of the pixel group includes an imaging element including a plurality of photoelectric conversion elements,
A first reading step for reading a signal from some of the plurality of photoelectric conversion elements as a viewpoint image;
A second readout step for mixing signals from the plurality of photoelectric conversion elements to read out as a composite image;
Generating depth information corresponding to the composite image,
In the generating step, the depth information is generated using the viewpoint image read in the first reading step,
Depth information corresponding to a composite image obtained by performing the second readout step without performing the first readout step is generated using depth information generated using the viewpoint image. Control method. A program that causes a computer to function as each unit included in the imaging apparatus according to any one of claims 1 to 7.

Images