JP7234057B2 - Image processing method, image processing device, imaging device, lens device, program, storage medium, and image processing system - Google Patents

Description

The present invention relates to an image processing method for correcting blur due to defocus in an image captured by splitting the pupil of an optical system, and obtaining an image with a good blur.

In Patent Document 1, a pupil of an optical system is divided into a plurality of divisions, and a plurality of parallax images obtained by observing an object space from each divided pupil are captured, and weights are adjusted when synthesizing the plurality of parallax images. A method for controlling the shape of defocus blur (defocus blur) is disclosed.

JP 2016-220016 A

However, the method disclosed in Patent Document 1 adjusts the weight of each divided pupil and synthesizes a plurality of parallax images, so it is not possible to reproduce the defocus blur corresponding to the pupil larger than the pupil of the optical system. . In other words, this method cannot fill the lack of defocus blur due to vignetting. In addition, noise increases when weights for synthesizing a plurality of parallax images become uneven. In addition, since the annular pattern of defocus blur caused by two-line blur and the aspherical lens included in the optical system has a fine structure, in order to reduce their effects, the pupil of the optical system should be finely divided. There is a need. In this case, a decrease in spatial resolution or an increase in noise occurs in each parallax image.

Accordingly, the present invention provides an image processing method, an image processing apparatus, an imaging apparatus, a lens apparatus, a program, and a storage medium capable of correcting blur due to defocusing of an image and obtaining an image with good blur. for the purpose.

An image processing method as one aspect of the present invention includes a first image obtained by imaging a subject through a first pupil in an optical system, and the first pupil in the optical system. obtaining a second image obtained by imaging the subject through different second pupils; and inputting the first and second images to a multi-layered neural network. and generating an output image in which blur due to defocusing has been corrected .

An image processing apparatus as another aspect of the present invention comprises a first image obtained by imaging a subject through a first pupil in an optical system, and the first pupil in the optical system; a second image obtained by imaging the subject through different second pupils; and an obtaining means for obtaining the first and second images to a multilayer neural network and generating means for generating an output image in which blur due to defocus is shaped by inputting.

An image pickup device as another aspect of the present invention includes an image pickup device that photoelectrically converts an optical image formed by an optical system, and the image processing device.

A lens device according to another aspect of the present invention is a lens device that is detachable from an imaging device, has an optical system, and storage means for storing information about weights to be input to a multi-layered neural network. The imaging device captures a first image obtained by imaging a subject through a first pupil in the optical system and a second pupil different from the first pupil in the optical system. an acquisition means for acquiring a second image obtained by imaging the subject through an image sensor; and generating means for generating an output image in which defocus blur has been corrected .

A program as another aspect of the present invention causes a computer to execute the image processing method.

A storage medium as another aspect of the present invention stores the program.

An image processing system as another aspect of the present invention is an image processing system having a first processing device and a second processing device, wherein the first processing device is configured to convert a first pupil in an optical system into a a first image obtained by imaging a subject through the optical system; and a second image obtained by imaging the subject via a second pupil different from the first pupil in the optical system and transmitting means for transmitting a request for image processing using a second image to the second processing device, wherein the second processing device receives the request sent from the first processing device A receiving means for receiving the request, an obtaining means for obtaining the first and second images , and inputting the first and second images to a multi-layered neural network to reduce blur due to defocus. and generating means for generating a shaped output image .

Other objects and features of the invention are illustrated in the following examples.

According to the present invention, there is provided an image processing method, an image processing apparatus, an imaging apparatus, a lens apparatus, a program, and a storage medium capable of correcting blur due to image defocus and obtaining an image with good blur. can do.

FIG. 4 is a diagram showing a network structure for generating a blurred image in Example 1; 1 is a block diagram of an imaging device in Example 1. FIG. 1 is an external view of an imaging device in Example 1. FIG. 4 is an explanatory diagram of an imaging unit in Example 1. FIG. 6 is a flow chart showing generation processing of a blurred image in Embodiment 1. FIG. 4A and 4B are explanatory diagrams of pupil division at each image height and azimuth in Example 1. FIG. FIG. 10 is a diagram showing point spread intensity distributions before and after blur shaping at a defocus distance in each example. 4 is a flow chart relating to weight learning in each embodiment. FIG. 11 is a block diagram of an image processing system in Example 2; FIG. 11 is an external view of an image processing system in Example 2; FIG. 10 is a configuration diagram of an imaging element in Example 2; 10 is a flow chart showing generation processing of a blurred image in Example 2. FIG. FIG. 10 is a diagram showing the relationship between a split pupil, image height, and vignetting in Example 2; FIG. 10 is a diagram showing a network structure for generating a blurred image in Example 2; FIG. 10 is a diagram showing another network structure for generating a blurred image in Example 2; It is explanatory drawing of the optical system provided with the mirror lens in each Example.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and overlapping descriptions are omitted.

First, the gist of the present invention will be described prior to specific description of each embodiment. The present invention uses deep learning to shape defocus blur in an image and obtain an image with good blur. Shaping of defocus blur means loss of defocus blur due to vignetting, multiple blur due to separation of defocus blur peaks (e.g., double-line blur), and defocus blur caused by cutting traces of aspherical lens molds. It refers to suppressing the ring pattern etc.

Shaping of defocus blur also includes suppressing a phenomenon in which defocus blur of a mirror lens becomes ring-shaped. FIG. 16 is a configuration diagram of an optical system 10 having a mirror lens. As shown in FIG. 16, the optical system 10 has a mirror lens configured with a primary mirror M1 and a secondary mirror M2. The ring-shaped defocus blur is caused by the secondary mirror M2 blocking the pupil of the mirror lens (optical system 10). In FIG. 16, L1 to L4 denote lenses, and L4 is a cemented lens. Further, IP is an image plane, which corresponds to a position where the imaging element is arranged. By shaping, the defocus blur is changed into a desired shape for the user (for example, a flat circular shape, a Gaussian distribution function, or the like), thereby realizing a good defocus blur.

In order to achieve highly accurate shaping of defocus blur, in the present invention, a first image captured by a pupil (first pupil) of an optical system and a part of the pupil (second pupil different from the first pupil) are used. The second image captured in the pupil of the eye) is input to deep learning. Since the first image and the second image have different pupil sizes, the magnitude of defocus blur when shifted from the in-focus distance is different. Therefore, when only one of the first image and the second image is input, the defocus blur in the image and the structure of the subject can be distinguished. As a result, defocus blur can be shaped with high accuracy by deep learning.

First, an imaging apparatus according to a first embodiment of the present invention will be described with reference to FIGS. 2 and 3. FIG. FIG. 2 is a block diagram of the imaging device 100. As shown in FIG. FIG. 3 is an external view of the imaging device 100. As shown in FIG. Note that the imaging apparatus 100 of the present embodiment includes a camera body and a lens device integrated with the camera body, but is not limited to this. The present invention can also be applied to an imaging system that includes a camera body (imaging device body) and a lens device (interchangeable lens) detachable from the camera body. First, an outline of each unit of the imaging apparatus 100 will be described, and details thereof will be described later.

As shown in FIG. 2, the imaging device 100 has an imaging unit 101 that acquires an image of an object space as an image (captured image). The imaging unit 101 has an optical system (imaging optical system) 101a that collects incident light from a subject space, and an imaging device 101b that has a plurality of pixels. The imaging device 101b is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor.

FIG. 4 is an explanatory diagram of the imaging unit 101. As shown in FIG. FIG. 4A shows a cross-sectional view of the imaging unit 101, and the dashed-dotted line represents the axial light flux. FIG. 4B is a top view of the imaging device 101b. The image pickup device 101b has a microlens array 122 and a plurality of pixels 121 . The microlens array 122 is arranged at a position conjugate with the object plane 120 via the optical system 101a. As shown in FIG. 4B, the microlenses 122 (only the microlenses 122a are shown and the parts after 122b are omitted) that constitute the microlens array 122 are divided into a plurality of pixels 121 (only the pixels 121a are shown and the parts after 121b are omitted). Here, when specifying a plurality of parts collectively, only a number is attached, and when indicating one of them, a number and a symbol such as a are attached.

Each of the plurality of pixels 121 has a first photoelectric conversion unit 123 and a second photoelectric conversion unit 124 that photoelectrically convert an optical image formed via the optical system 101a. As a result, for example, the light incident on the pixel 121a is received separately by the first photoelectric conversion unit 123a and the second photoelectric conversion unit 124a depending on the incident angle (the first photoelectric conversion unit 123a and the second photoelectric conversion unit 124a receive light incident at different incident angles). The incident angle of light is determined by which position of the pupil in the optical system 101a the light passes through. Therefore, the pupil of the optical system 101a is divided into two partial pupils by the two photoelectric converters, and the two photoelectric converters in one pixel receive information obtained by observing the object space from different viewpoints (pupil positions). get. In the present embodiment, the direction of division of the pupil is the horizontal direction, but it is not limited to this, and other directions such as the vertical direction and the oblique direction may be used.

The image pickup device 101b converts the signal (second image, A image) acquired by the first photoelectric conversion unit 123, this signal (A image), and the signal (third image) acquired by the second photoelectric conversion unit 124 to each other. image, B image) and an addition signal (first image, A+B image) is output. As described above, in this embodiment, the first image and the second image are images obtained by simultaneously imaging the subject space via the optical system 101a. Also, in this embodiment, the first image and the second image are images captured by the same image sensor 101b.

The A image and the A+B image are temporarily stored in the storage unit 103 . The image processing unit (image processing apparatus) 102 has an information acquisition unit (acquisition unit) 102a and an image generation unit (generation unit) 102b, and performs the image processing method (blur shaping process for shaping defocus blur) of the present embodiment. to run. At this time, the information acquisition unit 102a acquires the A image and the A+B image from the storage unit (storage means) 103, and the image generation unit 102b generates a defocused blur corrected image from the A image and the A+B image. . The blur-shaped image of this embodiment reduces the influence of at least one of vignetting, multi-blurring due to peak separation of the point image intensity distribution, ring pattern, or blocking of the pupil of the optical system for defocus blur. This is an image with The details of this processing will be described later.

The generated deblurred image is saved in the recording medium 105 . When the user issues an instruction regarding the display of the captured image, the stored defocused image is read and displayed on the display unit 104 . Note that the A image and the A+B image already stored in the recording medium 105 may be read out and the image processing unit 102 may generate a blurred image. The above series of controls are performed by the system controller 106 .

Next, blur shaping processing (blur shaping image generation) for shaping defocus blur executed by the image processing unit 102 will be described with reference to FIG. FIG. 5 is a flow chart showing processing for generating a blurred image. Each step in FIG. 5 is executed by the image processing unit 102 based on commands from the system controller 106 .

First, in step S<b>101 , the image processing unit 102 (information acquisition unit 102 a ) acquires the A+B image (first image) 201 and the A image (second image) 202 temporarily stored in the storage unit 103 . An A image 202 is an image obtained by capturing an object space based on a light flux passing through a partial pupil (second pupil) that is a part of the pupil of the optical system 101a. The A+B image 201 is an image obtained by imaging the subject space based on the light flux passing through the pupil (first pupil) of the optical system 101a. In this embodiment, the second pupil is included in the first pupil and is part of the first pupil. The A+B image and the A image differ in magnitude of defocus blur, and the second pupil is smaller than the first pupil, so the defocus blur of the A image is smaller than that of the A+B image. According to this embodiment, by using both the A+B image and the A image, it is possible to distinguish between the defocus blur in the image and the structure of the object. In other words, if there is a blurred area with no high-frequency information in the image, is it blurred because this area is defocused, or is it in focus but the subject has no high-frequency information? Therefore, it is possible to distinguish whether it looks blurred. In addition, by using the configuration of the imaging unit 101 shown in FIG. 4, it is possible to simultaneously capture the A+B image and the A image with mutually different pupil sizes, thereby avoiding deviation between the images due to movement of the subject. can be done.

In the A+B image, there is a lack of defocus blur due to vignetting of the optical system 101a, multiple blur due to separation of defocus blur peaks, and a defocus blur ring pattern caused by the aspherical lens included in the optical system 101a. It has occurred. Next, with reference to FIGS. 6 and 7, these will be described.

FIG. 6 is an explanatory diagram of pupil division at each image height and azimuth. FIG. 6 shows an A+B image, with the split pupil at the x's image height and azimuth plotted next to the x's. Since the defocus blur has substantially the same shape as the pupil except for inversion, the defocus blur has a shape that is not circular at an off-axis image height where vignetting occurs. A dashed line in FIG. 6 is a dividing line (dividing straight line) of the pupil, and the second pupil corresponding to the A image is a partial pupil on the right side of the dashed line. For this reason, the defocus blur of the A image also has a shape that is not circular.

Next, multi-bokeh will be described with reference to FIG. 7(A). FIG. 7A is a diagram showing a point spread (PSF) before blur shaping at a defocus distance. In FIG. 7A, the horizontal axis indicates spatial coordinates (position) and the vertical axis indicates intensity. This point also applies to FIGS. 7B to 7D, which will be described later. As shown in FIG. 7A, the bilinear blur, which is an example of the multiple blur, has a PSF with separated peaks. When the PSF at the defocus distance has a shape as shown in FIG. 7A, the subject, which is originally a single line, appears to be double blurred when defocused. Similarly, if the PSF peak is separated into three peaks, ie, the center and both ends, it will appear triple blurred. For this reason, the influence of separation of PSF peaks on defocus blur is called multiple blur.

Next, the annular pattern will be described with reference to FIG. 7(B). FIG. 7B is a diagram showing a point spread (PSF) before blur shaping at a defocus distance. As shown in FIG. 7B, the annular pattern is realized by the PSF having a vibration component. Such a vibration component is mainly caused by uneven cutting of the mold used when manufacturing the aspherical lens included in the optical system 101a. These undesired shapes of defocus blur can be shaped by the blur shaping process described below.

Subsequently, in step S102, the image generation unit 102b generates a defocused blur corrected image using a multi-layered neural network. The multi-layer neural network is fed with an A+B image (first image) and an A image (second image). The blurred image is an image obtained by changing the shape of the blur in the defocus area with respect to the image (A+B image) captured by the entire pupil of the optical system 101a. At this time, the focused subject does not change between the blurred image and the A+B image. In this embodiment, a convolutional neural network (CNN: Convolutional Neural Network) is used as the multilayer neural network. However, the present invention is not limited to this, and other methods such as GAN (Generative Adversarial Network) may be used.

Here, with reference to FIG. 1, the process of generating the blur-corrected image 213 by CNN will be described in detail. FIG. 1 is a diagram showing a network structure for generating a deblurred image. A CNN has multiple convolutional layers. In this embodiment, the input image 201 is an image in which an A+B image (first image) and an A image (second image) are concatenated in the channel direction. If the first image and the second image each have multiple color channels, the resulting image will have twice as many channels. An input image 201 is convolved with a plurality of filters and bias summed in a first convolutional layer 202 . Here, the filter coefficients are called weights (weight information). The filter and bias values in each layer are determined by prior learning to shape the undesired defocus blur into a good shape, details of which are given below.
A first feature map 203 summarizes the results calculated for each filter. The first feature map 203 is input to a second convolutional layer 204, which is similarly convolved with new filters and bias summed. This is repeated, and the result obtained by inputting the (N−1)th feature map 211 to the Nth convolution layer 212 is the blurred image 213 . Here, N is a natural number of 3 or more. Generally, a CNN having three or more convolutional layers is said to correspond to deep learning. Each convolutional layer performs nonlinear transformation using an activation function in addition to convolution. Examples of activation functions include a sigmoid function and ReLU (Rectified Linear Unit). In this embodiment, ReLU represented by the following formula (1) is used.

In Equation (1), x represents a feature map, and max represents a MAX function that outputs the maximum value among arguments. However, the last Nth convolutional layer does not need to perform a nonlinear transform.

In this embodiment, preprocessing is performed on each of the A+B image and the A image, and the preprocessed A+B image and the A image are input to the CNN. The pre-processing is the division of the A+B image and the A image, and control (inversion processing) regarding the inversion of the divided image or filter. That is, the image generation unit 102b sets reference points (optical axis or vicinity of the optical axis) of the A+B image and the A image parallel to the axis on which the second pupil is symmetrical with respect to the object point on the optical axis. A straight line passing through divides each of the A+B image and the A image. The image generation unit 102b also performs preprocessing (reversal processing) for controlling reversal of the divided A+B image and A image or weight information. By generating a blurred image based on the reversed A+B image and the A image (or the reversed weight information), the amount of weight information can be reduced. This will be explained with reference to FIG.

As shown in FIG. 6, in this embodiment, when one of the upper and lower sides of the A+B image (or the A image) is inverted about the dashed-dotted line, the image overlaps the other pupil division and is line symmetrical. For this reason, the defocus blur is also symmetrical with respect to the dashed line. Therefore, if weight information for correcting blur is stored for one of the areas above and below the dashed-dotted line, a blurred image can be estimated for the other area by inverting the image or the weight information.

Here, inversion includes the case of reversing the reference order when taking the product of the image and the weight information. By learning the weight information for only one of the upper and lower sides, the width of the defocus blur to be shaped by the CNN is limited, and highly accurate blur shaping can be achieved with a smaller network. In this embodiment, the A+B image and the A image are vertically divided into two, and the upper half or the lower half of the A+B image and the A image is used as the input image 201 . By individually processing the upper and lower divided images by CNN and combining the output blur-corrected images, it is possible to obtain an image in which the blur has been corrected for the entire image.

In this embodiment, since the pupil is divided in the horizontal direction, the axis of symmetry is a horizontal straight line. If the pupil is divided in the vertical direction, the axis of symmetry will also be a vertical straight line. A more general expression of this is as follows. The axis along which the relationship of the divided pupils is linearly symmetrical with respect to the entire image passes through the optical axis and is parallel to the axis along which the second pupil is linearly symmetrical on the optical axis. For the A+B image and the A image divided by this axis, if weight information is held only for one of the divided areas, the other can be reversed to perform defocus blur shaping processing with the same weight information. can.

In the generated blur-corrected image, the spread of defocus blur is greater than or equal to the spread of defocus blur of the A+B image. When the lack of defocus blur due to vignetting is corrected, the spread of the defocus blur becomes larger than in the A+B image. When the multi-bokeh or ring pattern is shaped, the spread of the defocused blur is the same as the A+B image. According to the present embodiment, it is possible to shape the defocus blur of an image and generate an image with good blur.

Next, with reference to FIG. 8, learning of weight information used in a multilayer neural network (CNN in this embodiment) will be described. FIG. 8 is a flowchart regarding learning of weight information. In this embodiment, learning is performed in advance by an image processing apparatus other than the imaging apparatus 100 , and the result (a plurality of pieces of weight information) is stored in the storage unit 103 . However, the present invention is not limited to this, and a part that performs learning may exist within the imaging device 100 .

First, in step S201, the image processing apparatus acquires a plurality of learning pairs. A learning pair is an A+B image and an A image as CNN input images, and an image (correct image) to be obtained as a CNN output image (deblurring image). The object to be corrected by the CNN changes depending on the relationship between the input image of the learning pair and the correct image.

Here, a method for generating learning pairs will be described. First, input images (A+B image and A image) and source data from which a correct image is generated are prepared. The source data are 3D models or 2D images with spectral intensities over a wide range of spatial frequencies. A three-dimensional model can be generated by CG (computer graphics) or the like. A two-dimensional image may be either CG or a photographed image. The A image and the B image can be generated by simulation (imaging simulation) in which source data is captured by the imaging unit 101 . In the imaging simulation, a defocus blur that occurs in the imaging unit 101 is given. The A+B image is obtained by adding the generated A and B images. A correct image is generated by giving a good defocus blur different from that of the optical system 101a. Examples of good defocus blur include, for example, the flat circular blur shown in FIG. 7(C) and the Gaussian distribution function shown in FIG. 7(D). Alternatively, a PSF that is a weighted average of multiple Gaussian distribution functions with different variances may be used. When the source data is a three-dimensional model, defocus blur corresponding to the distance of each object from the in-focus distance is given.

When the source image is a two-dimensional image, imaging simulation is performed by arranging the two-dimensional image at various defocus distances, and a plurality of learning pairs corresponding to them are created. However, at this time, a learning pair in which the two-dimensional image is at the in-focus distance is also created. It is preferable that the defocus blur shaping does not change with respect to the subject at the in-focus distance. Unless the CNN learns that there is no change for the learning pairs at the in-focus distance, it is impossible to predict what results the CNN will output for the in-focus object. Therefore, it is also necessary to create a learning pair in which the two-dimensional image is at the in-focus distance. In this embodiment, the A+B image and the A image are divided into two in the vertical direction and the blur shaping is performed. For this reason, the defocus blur given to the learning pair may be limited to one that occurs only in either the top or bottom of the image. Alternatively, noise generated by the image sensor 101b may be added to the input image, and the correct image may be an image without noise. By training with this training pair, the CNN will also perform denoising at the same time as deblurring.

Preferably, the source data has luminance in a width exceeding the dynamic range of the image sensor 101b. This is because even in actual subjects, when an image is captured by the imaging apparatus 101 under a specific exposure condition, there are subjects whose brightness does not fall within the saturation value. A correct image is generated by giving good defocus blur to the source data and clipping the signal at the luminance saturation value of the image sensor 101b. The training image is generated by adding defocus blur generated by the imaging unit 101 and clipping it by the luminance saturation value. There are two major problems that occur in clipping based on luminance saturation values. The first is that the shape of the defocus blur changes. For example, the shape of the defocus blur of a point light source should match the PSF at the defocus position, but it becomes a different shape because it is clipped at the luminance saturation value. The second is the appearance of false edges. Since false edges have high frequencies, it is difficult to determine whether they are focused objects or defocused blurs with saturated brightness. However, the first problem is that by using learning pairs generated from source data that exceeds the dynamic range of the image sensor 101b, the multi-layered neural network learns changes in defocus blur due to clipping of the luminance saturation value. so it can be resolved. Furthermore, by using two images with different pupils as inputs to the neural network, it becomes easy to determine whether the false edge is a focused object or a defocused blur, and the second problem can be solved.

Subsequently, in step S202 of FIG. 8, the image processing apparatus performs learning from a plurality of learning pairs to generate weight information. For learning, the same network structure as that for generating the deblurred image in step S102 is used. In this embodiment, the A+B image and the A image are input to the network structure shown in FIG. 1, and the error between the output result (estimated blur corrected image) and the correct image is calculated. In order to minimize this error, the error back propagation method or the like is used to update and optimize the filter (weight information) and bias used in each layer. The initial values of the filter and bias are arbitrary and can be determined, for example, from random numbers. Alternatively, pre-training such as Auto Encoder that pre-learns initial values for each layer may be performed.

A method of inputting all learning pairs into a network structure and updating learning information using all the information is called batch learning. However, this learning method increases the computational load as the number of learning pairs increases. Conversely, a learning method that uses only one learning pair for updating learning information and uses a different learning pair for each update is called online learning. This method has the advantage that the amount of calculation does not increase even if the number of learning pairs increases, but instead it is greatly affected by noise that exists in one learning pair. Therefore, it is preferable to learn using the mini-batch method, which is located between these two methods. The mini-batch method extracts a small number of all training pairs and uses them to update the learning information. In the next update, we will extract and use a different fractional training pair. By repeating this, the shortcomings of batch learning and online learning can be reduced.

An image to be handled when weight information is learned and a blur-corrected image is generated may be either a RAW image or a developed image. If the A+B and A images are encoded, then decode before training and generation. If the image used for learning and the input image for generating the deblurred image are different in gamma correction or gamma value, it is preferable to process the input image to match the image for learning. In addition, it is preferable to normalize the signal values of the A+B image and the A image (and the correct image during learning) before inputting them to the neural network. Without normalization, if the number of bits differs between learning and blur-corrected image generation, the blur-corrected image cannot be estimated correctly. Also, since the scale changes depending on the number of bits, optimization during learning may affect convergence. For normalization, the maximum value (brightness saturation value) that the signal can actually take is used. For example, even if the A+B image is stored in 16 bits, the luminance saturation value may be 12 bits. Also, it is preferable to subtract the value of optical black when normalizing. As a result, the range of signals that the image can actually take can be brought closer to 0 to 1. Specifically, it is preferable to standardize according to the following formula (2).

In equation (2), s is the signal of the A+B image (or A image or correct image), s _OB is the signal value of optical black (minimum signal value that the image can take), s _satu is the luminance saturation value of the signal, s _nor indicates a normalized signal.

According to this embodiment, it is possible to provide an image processing method, an image processing apparatus, an imaging apparatus, and a lens apparatus capable of shaping the defocus blur of an image and generating an image having a favorable blur. can.

Next, an image processing system according to Embodiment 2 of the present invention will be described. In this embodiment, an image processing device for estimating a blurred image, an imaging device for acquiring a captured image, and a server for learning exist separately.

The image processing system in this embodiment will be described with reference to FIGS. 9 and 10. FIG. FIG. 9 is a block diagram of the image processing system 300. As shown in FIG. FIG. 10 is an external view of the image processing system 300. As shown in FIG. As shown in FIGS. 9 and 10, the image processing system 300 includes an imaging device 301, an image processing device 302, a server 308, a display device 311, a recording medium 312, and an output device 313. FIG.

The basic configuration of the imaging device 301 is the same as that of the imaging device 100 shown in FIG. 2 except for an image processing unit that generates a blurred image and an imaging unit. Note that the lens device (optical system) of the imaging apparatus 301 of this embodiment can be replaced. The imaging element of the imaging device 301 is configured as shown in FIG. FIG. 11 is a configuration diagram of an imaging element in this embodiment. In FIG. 11, dashed lines indicate microlenses. Four photoelectric conversion units 321, 322, 323, and 324 (a, b and onwards are omitted) are provided in each of the pixels 320 (a, b onwards are omitted), and the pupil of the optical system is divided into four 2×2 pixels. split. The images acquired by the photoelectric conversion units 321 to 324 are assumed to be A image, B image, C image, and D image in order, and the addition result thereof is assumed to be ABCD image. Two images, an ABCD image (first image) and an A image (second image), are output from the imaging device as captured images.

When the imaging device 301 and the image processing device 302 are connected, the ABCD image and the A image are stored in the storage unit 303 . The image processing device 302 generates a blurred image from the ABCD image and the A image by the information acquisition unit 304 , the image generation unit 305 and the depth estimation unit 306 . At this time, the image processing apparatus 302 accesses the server 308 via the network 307 and reads weight information used for generation. The weight information is pre-learned by the learning unit 310 and stored in the storage unit 309 . The weight information is individually learned according to the type of lens, the F-number, the defocus blur shape after shaping, etc., and there is a plurality of weight information.

The image processing apparatus 302 acquires the weight information to be used in the storage unit 303 by user's selection instruction or automatic selection determined from the input ABCD image, and generates a blurred image. The blurred image is output to at least one of display device 311 , recording medium 312 and output device 313 . The display device 311 is, for example, a liquid crystal display, a projector, or the like. The user can work while confirming the image being processed through the display device 311 . The recording medium 312 is, for example, a semiconductor memory, a hard disk, a server on a network, or the like. The output device 313 is a printer or the like. The image processing apparatus 302 has a function of performing development processing and other image processing as necessary. Also, in this embodiment, the weight information may be held in a memory unit in the lens device connected to the imaging device 301 and called upon defocus blur shaping.

Next, with reference to FIG. 12, the process of generating a blurred image performed by the image processing apparatus 302 will be described. FIG. 12 is a flowchart showing processing for generating a blurred image. Each step in FIG. 12 is mainly executed by the image processing device 302 (information acquisition unit 304, image generation unit 305, depth estimation unit 306).

First, in step S301, the information acquisition unit 304 acquires the ABCD image and the A image. In this example, the first image is the ABCD image and the second image is the A image. However, the first image does not have to be an image corresponding to the entire pupil of the optical system, and may be an image obtained by adding at least two of the A image, B image, C image, and D image.

Subsequently, in step S302, the information acquisition unit 304 determines defocus blur shaping conditions. When determining the shaping conditions, it is recommended to select factors that suppress the effects of blur shaping from among multiple factors that change the defocus blur into an undesirable shape (loss of defocus blur, multiple blurs, annular patterns, etc.). include. Alternatively, determining the shaping conditions includes specifying a defocus blur shape target (such as a flat intensity PSF or Gaussian distribution function) to be varied by blur shaping. Note that the shaping conditions can be manually determined by the user, or may be automatically determined from the image acquired in step S301.

Regarding the automatic determination of shaping conditions, an example is described below. ABCD images (or A images) store the type of lens used for imaging as metadata. By specifying the type of lens used for imaging, it is possible to know the magnitude of vignetting and the presence or absence of double blurring and annular patterns. Therefore, it is possible to determine a factor (such as lack of defocus blur) that suppresses the influence of blur shaping based on the type of lens used to capture the ABCD image. Also, if the focal length at the time of photographing is stored in the metadata, the shaping conditions can be determined based on the focal length information. Since vignetting tends to increase with a wide-angle lens, when the focal length is smaller than a certain threshold, the shaping condition is determined so as to suppress the loss of defocus blur by blur shaping. Also, the defocus blur shape target may be determined, for example, based on the luminance value of the ABCD image as follows. A PSF having a flat intensity as shown in FIG. 7(C) is used for the ABCD image with saturated brightness, and a Gaussian distribution function as shown in FIG. 7(D) is used for other regions. Alternatively, the information of the shooting scene determined at the time of shooting may be read from the metadata, and if the shooting scene is a night scene, a PSF with a flat intensity may be used, otherwise a Gaussian distribution function may be used.

Subsequently, in step S303, the information acquisition unit 304 acquires weight information corresponding to the defocus blur shaping condition determined in step S302 from a plurality of pieces of weight information. A method of learning a plurality of pieces of weight information will be described later.

Subsequently, in step S304, the depth estimation unit 306 calculates a depth map from the ABCD image and the A image. Parallax between images can be used to calculate the depth map, or DFD (Depth From Defocus) based on the difference in magnitude of defocus blur may be used. Instead of the depth map, a parallax map representing the amount of parallax deviation between the ABCD image and the A image may be calculated. Note that step S304 may be executed at any time between steps S301 and S307.

Subsequently, in step S305, the image generation unit 305 performs processing for matching the brightness of the ABCD image and the A image. The A image is a darker image because the pupil is smaller than that of the ABCD image. Also, since vignetting occurs at image heights other than those on the optical axis, the ratio of brightness (light amount ratio) between the ABCD image and the A image changes depending on the image height and azimuth. This will be explained with reference to FIG.

FIG. 13 is a diagram showing the relationship between split pupil, image height, and vignetting. FIG. 13A shows a pupil on the optical axis of the optical system of the imaging device 301. FIG. Broken lines in FIG. 13 represent division lines of the pupil divided by the four photoelectric conversion units. FIG. 13(B) shows the pupil at an image height different from that in FIG. 13(A). In FIG. 13A, the light amounts of the four split pupils are uniform, but in FIG. 13B, vignetting causes a deviation in the light amount ratio between the two. FIG. 13(C) is the same image height as FIG. 13(B) (position at the same distance from the optical axis in the plane perpendicular to the optical axis) and azimuth (the optical axis is rotated in the plane perpendicular to the optical axis). and the azimuth angles) are different. Also in this case, the light amount ratio of the partial pupil changes. Therefore, when the ABCD image and the A image are input to a multi-layered neural network, which will be described later, the brightness of the two images varies due to the image height and azimuth in the image, which may reduce the accuracy of blur generation. be. Therefore, in this embodiment, it is preferable to perform preprocessing for matching the brightness of the ABCD image and the A image. In this embodiment, the brightness of the A image is adjusted to that of the ABCD image, but the opposite is also possible. Also, for each image height and azimuth, a first integrated value and a second integrated value obtained by integrating the transmittance distributions of the first pupil and the second pupil are obtained and used for brightness adjustment. good too. For each image height and azimuth pixel of the first image, multiply the corresponding first integral by the inverse, and for each image height and azimuth pixel of the second image, multiply the corresponding second integral The brightness can also be adjusted by multiplying the reciprocal of the value.

Two examples of methods for matching the brightness of two images are given below. The first is the light amount ratio between the first pupil (entire pupil of the optical system) and the second pupil (partial pupil corresponding to the A image) (the ratio of the transmittance distribution between the first pupil and the second pupil). This is a method of matching the brightness based on . A light amount ratio of the first pupil to the second pupil (the ratio of the transmittance distribution of the first pupil to the transmittance distribution of the second pupil) is stored in a storage unit for each image height and azimuth pixel of the A image. The product is read from 303 and the brightness is matched with the ABCD image. The light amount ratio is a value of 1 or more, and has different values depending on the image height and the azimuth.

The second method is to use local average pixel values of the ABCD image and the A image. Although the ABCD image and the A image have different aberrations and noises and have parallax, the same subject is captured, so the ratio of the average pixel values in the partial area roughly corresponds to the above-described light amount ratio. For this reason, for example, a smoothing filter is applied to the ABCD image and the A image to obtain the average pixel value for each pixel, and from the ratio of the average pixel values of the pixels at the same position, the light amount ratio at this position is obtained. can match. However, when calculating the average pixel value, if a pixel with saturated luminance is included, the value may deviate from the light amount ratio. For this reason, in this embodiment, it is preferable to calculate the average pixel value by excluding pixels whose luminance is saturated. If the luminance saturation area is large and the average pixel value at that position cannot be obtained, interpolation can be performed from the light quantity ratios calculated in the periphery to calculate the light quantity ratio corresponding to that position. The size of the partial region is preferably determined based on the size of the blur and the baseline length (the length between the centroid positions) of the first pupil and the second pupil. Note that step S305 may be executed at any time between steps S301 and S307.

Subsequently, in step S306 of FIG. 12, the image generation unit 305 determines the input regions of the ABCD image and the A image to be input to the multilayer neural network. Although the entire image may be input to the neural network, it is not necessary to input the non-defocused area (focused area) because the defocus blur is shaped in step S307, which will be described later. The calculation load can be reduced by using only the area of the image excluding the in-focus area as the input area of the neural network. The edge distribution of the ABCD image (or A image) is used to determine the in-focus area. The edge distribution is obtained, for example, by wavelet transforming the ABCD image. An edge (edge region) corresponds to a region in which a high-frequency component has a certain intensity or more. Since an edge exists in the focus area, an area that does not include the edge is used as an input area. Also, instead of the edge distribution, the input area may be determined based on the depth map (or parallax map) calculated in step S304. In this case, the input area is any area other than the area whose depth matches the focal distance in the metadata of the ABCD image. In addition, since the imaging device 301 is configured so that the parallax becomes zero at the in-focus distance, in the case of the parallax map, the region with the parallax larger than the threshold is set as the input region. Note that step S306 may be executed at any time between steps S301 and S307.

Subsequently, in step S307, the image generation unit 305 generates a blurred image. At this time, the image generation unit 305 extracts an input area from the ABCD image and the A image, and inputs the extracted input area to the multilayer neural network as an input image. The image generator 305 also adds a depth map (or parallax map) corresponding to the input area as an input image.

In this embodiment, the image generator 305 uses the network structure shown in FIG. FIG. 14 is a diagram showing a network structure for generating a blurred image in this embodiment. In FIG. 14, the input image 401 may be the entire input area or a portion (divided area) obtained by dividing the input area. An input image 401 is an image in which an ABCD image, an A image, and a depth map (or parallax map) are connected in the channel direction. In FIG. 14, CN is a convolution layer, DS is a downsampling layer that downsamples the sampling rate of the input feature map, DC is a deconvolution layer, and US is an upsampling layer that upsamples the feature map. The downsampling rate of the downsampling layer and the upsampling rate of the upsampling layer have an inverse relationship. The filters used in each convolutional and deconvolutional layer are determined based on the weight information.

Skip connections 412, 413 combine feature maps output from discontinuous layers. The sum of each element of the feature map may be taken, or they may be connected in the channel direction. Multiple downsampling layers and skip connections generate multiple feature maps with different resolutions (feature maps downsampled with different sampling rates). As a result, both local features and more global features can be calculated without increasing the filter size of the convolution layer. Since the defocus blur increases according to the deviation from the in-focus distance and the F-number, it is preferable to calculate global features. Skip connections also play a role in improving convergence during filter learning. The skip connection 411 sums the ABCD image 401a of the input image 401 and the output of the multilayer neural network. As a result, a blurred image 402 is obtained. When the input image 401 is obtained by dividing the input area, the blur corrected image 402 is calculated for the entire input area. A blur-corrected image for the entire image is generated by combining the blur-corrected image calculated by the neural network and the ABCD image other than the input region.

In this embodiment, the numbers of convolution layers, deconvolution layers, downsampling layers, upsampling layers, and skip connections are not limited to the numbers shown in FIG. Also, the network structure is not limited to the structure shown in FIG. For example, a network structure such as that shown in FIG. 15 may be used. FIG. 15 is a diagram showing another network structure for generating a blurred image in this embodiment.

The network structure of FIG. 15 is divided into multiple stages 500, 510, 520, each with a different resolution. Stage 520 has a resolution of (1/m) ² (where m is a positive integer) that of the ABCD image. The input image 521 is an ABCD image, an A image, and a depth map whose resolution is down-sampled to (1/m) ² times. Note that the skip connection is the same as in FIG. A deblurred image 522 at the resolution in stage 520 is upsampled by a factor of m in an upsampling layer 592 and input to stage 510 at 1/m times the original resolution.

An input image 511 in stage 510 is data obtained by concatenating an ABCD image with a resolution of 1/m, an A image, a depth map, and an upsampling result of a blurred image 522 . Similarly, deblurred image 512 of stage 510 is upsampled by a factor of m in upsampling layer 591 and input to stage 500 .

Stage 500 has the same scale resolution as the original ABCD image. An input image 501 is data in which an ABCD image, an A image, a depth map, and an upsampling result of a deblurred image 512 are concatenated. A deblurred image 502 calculated in stage 500 is output as the final deblurred image. In this embodiment, the number of stages is not limited to 3, and the network structure within the stages is not limited to the structure shown in FIG. Also, the weight of the filter in each stage may be common (the same filter may be used regardless of the stage). As a result, the data volume of weight information can be reduced.

Learning of the weight information executed by the learning unit 310 is performed according to the flowchart of FIG. 8 as in the first embodiment. In this example, aberration (affecting multiple blurring), vignetting, and the presence or absence of an aspherical lens differ depending on the type of lens. Create learning pairs and learn weight information. In the present embodiment, an example of one second image was given, but a plurality of second images (for example, three images of A image, C image, and D image) may be used.

According to this embodiment, it is possible to provide an image processing system capable of shaping the defocus blur of an image and generating an image with a favorable blur.

(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

Further, the present invention provides, for example, a first processing device (imaging device, smartphone, user terminal such as a PC) that requests image processing, and a second processing device that substantially performs the image processing of the present invention in response to the request. It can also be implemented as an image processing system configured by a processing device (server). For example, the information acquisition unit 304, the image generation unit 305, and the depth estimation unit 306 in the image processing system 300 of the second embodiment can be provided in the server 308 as the second processing device. Also, the image processing device 302 as the first processing device can be configured to request the server 308 to perform image processing using the first image and the second image. In this case, the first processing device (user terminal) has transmission means for transmitting a request for image processing to the second processing device (server), and the second processing device (server) It has receiving means for receiving a request sent from a processing device (user terminal).

In this case, the first processing device may transmit the first image and the second image together with the image processing request to the second processing device. However, the second processing device may acquire the first image and the second image stored in a location (external storage device) other than the first processing device in response to a request from the first processing device. good. Further, after the second processing device performs blur shaping processing on the first image and the second image, the second processing device may transmit the blurred images to the first processing device. . By configuring the image processing system in this way, it is possible to perform the processing by the image generation unit, which requires relatively heavy processing load, on the second processing device side, and it is possible to reduce the burden on the user.

According to each embodiment, an image processing method, an image processing device, an imaging device, a lens device, a program, and a storage medium capable of correcting blur due to defocusing of an image and obtaining an image with good blurring are provided. can provide.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist.

102 image processing unit (image processing device)
102a information acquisition unit (acquisition means)
102b image generator (generating means)

Claims (24)

A first image obtained by imaging a subject through a first pupil in an optical system, and the subject through a second pupil different from the first pupil in the optical system obtaining a second image obtained by imaging the body ;
and inputting the first and second images to a multi-layer neural network to generate an output image in which blur due to defocus has been corrected . The output image is reduced in the influence of at least one of vignetting, multi-blurring due to peak separation of point image intensity distribution, ring pattern, and pupil shielding of the optical system with respect to the blurring due to the defocusing. 2. The image processing method according to claim 1, wherein the image is an image that has been distorted. 3. The image processing method according to claim 1, wherein the spread of blur due to defocusing of said output image is greater than the spread of blur due to defocusing of said first image. 4. The image processing method according to claim 1, wherein a transmittance distribution of said first pupil is different from a transmittance distribution of said second pupil. 5. The image processing method according to claim 1, wherein said second pupil is a part of said first pupil. 6. The method according to any one of claims 1 to 5, wherein the first and second images are images obtained by simultaneously capturing images of the subject through the optical system. Image processing method. 7. The image processing method according to any one of claims 1 to 6, wherein the first and second images are images obtained by imaging using the same imaging device. The imaging element has a plurality of pixels each including first and second photoelectric conversion units that receive light with different incident angles,
8. The method according to claim 7, wherein the first photoelectric conversion section receives light from the first pupil, and the second photoelectric conversion section receives light from the second pupil. Image processing method. further comprising performing a process of matching the brightness of the first and second images ;
9. The method according to any one of claims 1 to 8 , wherein said step of generating said output image is performed based on said first and second images on which said brightness matching process has been performed. image processing method . 10. The image processing method according to claim 9 , wherein said brightness matching process is executed based on information on transmittance distributions of said first and second pupils. 10. The image processing method according to claim 9 , wherein said brightness matching process is performed based on average pixel values calculated for respective partial areas of said first and second images. Inversion processing is performed on the first and second images divided by a straight line passing through the respective reference points of the first and second images, parallel to the axis of line symmetry of the second pupil . further comprising a step of
12. An image according to any one of claims 1 to 11, wherein said step of generating said output image is performed based on said first and second images on which said inverting process has been performed. Processing method. further comprising calculating a parallax map or depth map corresponding to the subject based on the first and second images;
13. The image processing method according to any one of claims 1 to 12 , wherein the step of generating the output image is performed based on the parallax map or the depth map. 14. The image processing method according to claim 13, wherein generating the output image includes inputting the parallax map or the depth map to the multilayer neural network. The step of generating the output image includes inputting an input image configured by connecting the first image, the second image, and the parallax map or the depth map in a channel direction to the multilayer neural network. 15. The image processing method according to claim 14, further comprising the step of: The step of generating the output image is based on a distribution of edges in the first image or the second image, or the parallax map or the depth map calculated based on the first and second images. 14. The image processing method according to claim 13, further comprising the step of determining respective input regions of said first and second images to be input to said multi-layered neural network. The step of generating the output image includes:
calculating a plurality of feature maps down-sampled at different sampling rates using the multi-layered neural network;
and generating the output image based on the plurality of feature maps. A first image obtained by imaging a subject through a first pupil in an optical system, and the subject through a second pupil different from the first pupil in the optical system an acquisition means for acquiring a second image obtained by imaging the body ;
and generating means for inputting the first and second images to a multi-layered neural network to generate an output image in which blur due to defocus has been corrected . an imaging device that photoelectrically converts an optical image formed by an optical system;
An imaging apparatus comprising: the image processing apparatus according to claim 18 . The imaging element has a plurality of pixels,
each of the plurality of pixels has a plurality of photoelectric conversion units,
each of the pixels receives light incident on each of the plurality of photoelectric conversion units at different incident angles to generate a plurality of signals;
The imaging element is configured to produce the first image corresponding to an added signal obtained by adding the plurality of signals, and the first image corresponding to an added signal obtained by adding one of the plurality of signals or a part of the plurality of signals. 20. The imaging device according to claim 19, wherein the image of 2 is output. A lens device detachable from an imaging device,
an optical system;
a storage means for storing information about weights input to the multi-layered neural network;
The imaging device is
A first image obtained by imaging a subject through a first pupil in the optical system and the subject through a second pupil different from the first pupil in the optical system an acquisition means for acquiring a second image obtained by imaging a subject ;
generating means for generating an output image in which blur due to defocus has been corrected by inputting information about the first and second images and the weights to a multi-layered neural network. Device. A program for causing a computer to execute the image processing method according to any one of claims 1 to 17. 23. A storage medium storing the program according to claim 22. An image processing system having a first processing device and a second processing device,
The first processing device provides a first image obtained by imaging a subject through a first pupil in an optical system and a second image obtained by imaging a subject through a first pupil in the optical system and a second a transmission means for transmitting a request for image processing using a second image obtained by imaging the subject through the pupil of the second processing device,
The second processing device is
receiving means for receiving the request transmitted from the first processing device;
acquisition means for acquiring the first and second images ;
generating means for inputting the first and second images to a multi-layered neural network to generate an output image in which blur due to defocus has been corrected ;
An image processing system comprising:

Images