JP7129229B2 - Image processing method, image processing device, imaging device, program, and storage medium - Google Patents

Description

The present invention relates to an image processing method.

In general, the optical performance of an imaging device is evaluated by the imaging performance of an in-focus object, but depending on the application, the appearance of an out-of-focus image (bokeh image) may be an important evaluation index for the optical performance of the imaging device. be. In particular, in imaging devices such as digital still cameras, video cameras, and TV cameras, there is a tendency to attach importance to how an out-of-focus image appears.

Patent Document 1 discloses an optical system having an apodization filter in the vicinity of the diaphragm. Generally, in a wide-angle to medium-telephoto imaging optical system, a sagittal halo of an off-axis light beam causes intensity unevenness of an out-of-focus image at the periphery of the screen. Apodization filters are effective in removing such sagittal halos.

JP 2016-145862 A

Y. LeCun, et al. , "Gradient-based Learning Applied to Document Recognition", Proc. of The IEEE, 1998. G. E. Hinton, et al. , "A fast learning algorithm for deep belief nets", Neural Comput. 2006 Jul; 18(7): 1527-54. I. J. Goodfellow, et al. , "Maxout networks", arXiv preprint arXiv: 1302.4389 (2013). G. E. Hinton & R. R. Salakhutdinov (2006-07-28). "Reducing the Dimensionality of Data with Neural Networks", Science 313(5786): 504-507. N. Srivastava, et al. , "Dropout: A simple way to prevent neural networks from overfitting", The Journal of Machine Learning Research, 15(1): 1929-1958, 2014. A. Krizhevsky, "Learning Multiple Layers of Features from Tiny Images", 2009, https://www. cs. toronto. edu/~kriz/learning-features-2009-TR. pdf

However, the optical system with the apodization filter disclosed in Patent Document 1 cannot correct the shape of an out-of-focus image caused by vignetting such as ring blur caused by a cadadioptric lens (reflective telephoto lens).

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an image processing method, an image processing apparatus, an imaging apparatus, a program, and a storage medium capable of estimating or correcting deformation of an out-of-focus image from a captured image with high accuracy. .

An image processing method according to one aspect of the present invention includes the steps of acquiring at least a partial region of an input image, and acquiring learning information learned in advance regarding deformation of an out-of-focus image due to the effects of vignetting or aberration. and a step of estimating or correcting the deformation of the out-of-focus image included in the region using the learning information, wherein the step of estimating or correcting includes N being an integer of 2 or more and n being from 1 to n is an integer up to N, the region is subjected to n-th linear transformation by each of the plurality of linear functions based on the learning information and n-th nonlinear transformation by a nonlinear function, where n is from 1 to N. a step of generating intermediate data by sequentially executing up to and performing an N+1th linear transformation on the intermediate data by at least one linear function based on the learning information.

An image processing apparatus according to another aspect of the present invention includes a storage unit that stores learning information about deformation of an out-of-focus image due to the influence of vignetting or aberration; and an image processing unit for estimating or correcting deformation of an out-of- focus image in the region, wherein N is an integer of 2 or more and n is an integer from 1 to N, for the region generating intermediate data by sequentially executing the n-th linear transformation by each of the plurality of linear functions based on the learning information and the n-th nonlinear transformation by the nonlinear function until n is from 1 to N, An N+1th linear transformation is performed on the intermediate data by at least one linear function based on the learning information.

An imaging device as another aspect of the present invention includes an imaging unit that acquires an image of a subject space as an input image, and the image processing device.

A program as another aspect of the present invention causes a computer to execute the image processing method.

A storage medium as another aspect of the present invention stores the program.

Other objects and features of the invention are described in the following embodiments.

According to the present invention, it is possible to provide an image processing method, an image processing device, an imaging device, a program, and a storage medium that can estimate or correct deformation of an out-of-focus image from a captured image with high accuracy.

FIG. 4 is a diagram showing a network structure for correcting out-of-focus images in Examples 1 and 2; 1 is a block diagram of an imaging device in Examples 1 and 3; FIG. 1 is an external view of an imaging device in Examples 1 and 3; FIG. 5 is a flowchart showing correction processing for an out-of-focus image in Example 1. FIG. 4 is a flow chart showing learning of learning information in Example 1. FIG. FIG. 11 is a block diagram of an image processing system in Example 2; FIG. 11 is an external view of an image processing system in Example 2; 10 is a flow chart showing correction processing for an out-of-focus image in Example 2. FIG. 14 is a flow chart showing a process of estimating deformation of an out-of-focus image in Example 3. FIG. FIG. 10 is a diagram showing a network structure for estimating deformation of an out-of-focus image in Example 3; 11 is a flow chart showing learning of learning information in Example 3. FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and overlapping descriptions are omitted.

The gist of the present invention will be described before going into specific description of each embodiment. In the present invention, deep learning is used to estimate or correct the deformation of the out-of-focus image (the shape of the deformed out-of-focus image) from the captured image. In general, the imaging optical system can be applied to photographing a stereoscopic object. Here, a "stereoscopic subject" is a subject that consists of a plurality of parts with different distances in the optical axis direction, and in particular, a subject that has a point that is more than the depth of field away from the focal plane of the imaging optical system at the time of photographing. is. At this time, an out-of-focus image is formed on the imaging plane, and when the diameter of the out-of-focus image is larger than about 1 to 2% of the image circle radius of the imaging optical system, it becomes recognizable as an out-of-focus image. . Here, the "image circle" is a circle on which a light beam passing through the effective diameter of the lens forms an image.

When the optical system of this embodiment is used as an imaging optical system for a digital still camera or a video camera, the imaging plane is the imaging plane of a semiconductor imaging element (photoelectric conversion element) such as a CCD sensor or CMOS sensor. Note that the aforementioned image circle radius may be the maximum image height of the imaging surface of the imaging device. Also, in a general imaging optical system, vignetting is observed in the off-axis light flux. "Vignetting" means vignetting of a portion of the luminous flux and is also called vignetting. An out-of-focus image, free of vignetting effects, will have a shape that reflects the shape of the aperture, typically circular. Due to vignetting, in a general imaging optical system, the shape is such that a circle is missing off-axis, and in a reflective telephoto lens in which the central portion of the aperture is blocked, it becomes a ring-shaped shape (ring blur).

According to each embodiment, using images with different states of out-of-focus images, the deformation of the out-of-focus images (the shape of the deformed out-of-focus images) can be estimated with high accuracy by learning the correspondence relationship through deep learning. or can be corrected. That is, according to each of the embodiments, it is possible to highly accurately estimate the shape of an out-of-focus image deformed by the photographing conditions such as the optical system from the ideal shape of the out-of-focus image. Further, according to each embodiment, it is possible to correct the shape of the deformed out-of-focus image to the shape of the ideal out-of-focus image with high precision (to bring the shape closer to the ideal shape of the out-of-focus image). Here, the ideal shape of the out-of-focus image is the shape of the out-of-focus image obtained using an optical system that is free from the effects of vignetting, aberration, etc. of the optical system, and is, for example, a circular shape. On the other hand, the shape of the out-of-focus image deformed due to vignetting or aberration of the optical system is, for example, a ring shape.

First, an imaging apparatus according to a first embodiment of the present invention will be described with reference to FIGS. 2 and 3. FIG. FIG. 2 is a block diagram of the imaging device 100. As shown in FIG. FIG. 3 is an external view of the imaging device 100. As shown in FIG. In this embodiment, the imaging apparatus 100 executes an image processing method to correct deformation of the out-of-focus image (shape of the out-of-focus image deformed due to vignetting, aberration, etc. of the optical system) by deep learning.

The imaging device 100 has an imaging unit 101 that acquires an image of a subject space as a captured image (input image). The imaging unit 101 has an imaging optical system 101a that collects light incident from a subject space, and an imaging element 101b that has a plurality of pixels. The imaging device 101b is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor.

The image processing unit 102 corrects an out-of-focus image (deformation of an out-of-focus image) for a captured image (input image). The image processing unit 102 has a learning unit 102a and a correction unit 102b. The storage unit 103 stores learning information (learning information learned in advance regarding deformation of the out-of-focus image) used when the image processing unit 102 corrects the out-of-focus image (deformation of the out-of-focus image). The details of this processing will be described later. An output image such as an image corrected for deformation of an out-of-focus image by the image processing unit 102 is displayed on a display unit 104 such as a liquid crystal display or stored in a recording medium 105 . However, the captured image may be stored in the recording medium 105 and the out-of-focus image correction process may be performed at an arbitrary timing. The captured image may be a moving image as well as a still image. In this case, out-of-focus image correction processing is performed for each frame. The above series of controls are performed by the system controller 106 .

Next, correction processing of an out-of-focus image by the image processing unit 102 will be described with reference to FIG. The image processing unit 102 uses learning information that has been learned in advance when correcting an out-of-focus image, and the details of this learning will be described later. FIG. 4 is a flow chart showing an out-of-focus image correction process. Each step in FIG. 4 is mainly executed by the correction unit 102b of the image processing unit 102. FIG.

First, in step S101, the image processing unit 102 (correction unit 102b) acquires a captured image (input image) in which an out-of-focus image is deformed and learning information. Here, the deformation of the out-of-focus image means a state in which the shape and luminance distribution are different from those of the ideal out-of-focus image. The learning information is information learned in advance for associating a captured image with an image corrected for an out-of-focus image (deformation of the out-of-focus image). Subsequently, in step S102, the correction unit 102b acquires at least a partial area (partial area) of the captured image from the captured image. Correction processing for an out-of-focus image is performed in units of this area (partial area) (each partial area).

Subsequently, in step S103, the correction unit 102b generates a corrected partial area, which is a partial area in which the out-of-focus image is corrected, from the partial area using the learning information. Now, with reference to FIG. 1, the out-of-focus image correction processing will be described in detail. FIG. 1 shows the network structure of CNN (Convolutional Neural Network), which is one of deep learning. However, as deep learning, methods other than CNN, for example, DBN (Deep Belief Network) may be used. Details of CNN and DBN are described in Non-Patent Document 1 and Non-Patent Document 2, respectively.

A CNN has a multi-layered structure, and linear transformation and non-linear transformation using learning information are executed in each layer. When n is an integer from 1 to N, the n-th layer is called the n-th layer, and the linear transformation and the non-linear transformation in the n-th layer are called the n-th linear transformation and the n-th non-linear transformation, respectively. However, N is an integer of 2 or more. Concerning the partial region 201, convolution (first linear transformation by a plurality of linear functions) with each of the plurality of filters 202 is performed in the first layer. A transformation (first nonlinear transformation) is then performed using a nonlinear function called the Activation Function. In FIG. 1 the activation function is indicated as AF. A plurality of partial regions 201 are drawn because the input image (captured image) has a plurality of channels. In this embodiment, the partial area has three channels of RGB (Red, Green, Blue). However, the number of channels is not limited to this. As will be described later, when a plurality of captured images are input, the partial area may have the number of channels corresponding to the input captured images. Also, even if the partial area has a plurality of channels, each channel may be individually input to the CNN.

A plurality of filters 202 exist. The correction unit 102b individually calculates the convolution between each of the plurality of filters 202 and the partial region 201. FIG. The coefficients of filter 202 are determined based on the learning information. The learning information may be the coefficients (filter coefficients) of the filter 202 themselves, or the coefficients obtained by fitting the filter 202 with a predetermined function. The number of channels in each filter 202 matches the number of sub-regions 201 . When the number of channels in the partial area 201 is two or more, the filter becomes a three-dimensional filter (the third dimension represents the number of channels). Alternatively, a constant (possibly negative) determined from learning information may be added to the result of convolution.

Examples of the activation function f(x) include the following equations (1) to (3).

Equation (1) is called a sigmoid function, Equation (2) is a hyperbolic tangent function, and Equation (3) is called a ReLU (Rectified Linear Unit). max in equation (3) represents a MAX function that outputs the maximum value of the arguments. The activation functions f(x) shown in equations (1)-(3) are all monotonically increasing functions. Also, Maxout may be used as the activation function. Maxout is a MAX function that outputs the maximum signal value for each pixel in a plurality of images output from the n-th linear transformation. Details of Maxout are described in Non-Patent Document 3.

In FIG. 1 , the partial area subjected to the first linear transformation and the first nonlinear transformation is called a first transformed partial area 203 . Each channel component of the first transform sub-region 203 is generated from the convolution of the sub-region 201 with each of the plurality of filters 202 . Therefore, the number of channels in the first conversion partial area 203 is the same as the number of filters 202 .

In the second layer, the first transformation partial region 203 is subjected to convolution (second linear transformation) with a plurality of filters 204 determined from learning information in the same manner as in the first layer, and nonlinear transformation using an activation function. (second nonlinear transformation). The filter 204 used in the second layer is generally not the same as the filter 202 used in the first layer. The size and number of filters 204 also need not match the filters 204 . However, the number of channels of the filter 204 and the number of channels of the first conversion partial area 203 match each other. The correcting unit 102b acquires the intermediate data 210 by repeating similar calculations up to the Nth layer (executing the nth linear transformation and the nth nonlinear transformation (n=1 to N)).

Finally, in the N+1th layer, by adding a constant to the convolution of the intermediate data 210 and each of the plurality of filters 211 (N+1th linear transformation), the corrected partial area 212 in which the out-of-focus image is corrected is obtained. be. The filter 211 and the constants used here are also each determined based on the learning information. The number of channels of the correction partial area 212 is the same as that of the partial area 201 . Therefore, the number of filters 211 is also the same as the number of channels of the partial area 201 . The components of each channel of the correction partial area 212 are obtained from operations including convolution between the intermediate data 210 and each of the filters 211 (there may be one filter 211). Note that the sizes of the partial area 201 and the size of the correction partial area 212 do not have to match each other. Since there is no data outside the partial area 201 during the convolution, the size of the convolution result will be reduced if the calculation is performed only in the area where the data exists. However, the size can be maintained by setting a periodic boundary condition or the like.

The reason why deep learning can exhibit high performance is that high nonlinearity can be obtained by executing nonlinear transformation many times with a multi-layered structure. If the activation function responsible for the nonlinear transformation does not exist and the network is composed only of linear transformations, no matter how many layers there are, there is an equivalent single-layer linear transformation. There is no Deep learning is said to be more likely to achieve high performance because the more layers it has, the more strongly nonlinear it can be obtained. In general, the case of having at least three layers is called deep learning.

Subsequently, in step S104 in FIG. 4, the correction unit 102b determines whether or not the out-of-focus image correction has been completed for all of the predetermined regions (partial regions) of the captured image. If the correction partial areas 212 have been generated for all of the predetermined areas, the process proceeds to step S105. On the other hand, if there remains an area (partial area) for which the out-of-focus image correction has not been completed, the process returns to step S102, and the correction unit 102b acquires the partial area for which the out-of-focus image has not yet been corrected from the captured image. .

In step S105, the correction unit 102b outputs an image (corrected image) in which the out-of-focus image is corrected. An image in which the out-of-focus image is corrected is generated by synthesizing the plurality of generated correction partial areas 212 . However, when the partial area is the entire captured image, the correction unit 102b outputs the corrected partial area 212 as it is as an image in which the out-of-focus image is corrected. Through the above processing, an image in which an out-of-focus image has been corrected (an out-of-focus image having an ideal shape (for example, a circular shape)) can be acquired.

In the present embodiment, a case has been described in which both the captured image (input image) and the out-of-focus image corrected image (output image) are one. However, this embodiment is not limited to this. For example, configure a CNN network so that multiple captured images (multiple input images) can be input and multiple output images in which the out-of-focus images of each of the multiple captured images are corrected can be acquired at once. You may Alternatively, a configuration may be adopted in which a plurality of photographed images are input and one out-of-focus image is corrected to be acquired. When inputting a plurality of captured images, it is preferable to use a plurality of images with different aperture values (F values) and different focus positions. When the aperture value or focus position changes, the size, shape, brightness distribution, etc. of the out-of-focus image in the subject differ. Accuracy can be improved. In addition, it is preferable to use an image having a plurality of color channels as the captured image to be input. In this embodiment, the case of correcting the out-of-focus image to an ideal shape (circular shape) has been described, but the shape of the out-of-focus image after correction is not limited to this. For example, it is possible to correct to a shape desired by the user, such as a star shape or a heart shape.

Next, learning of learning information in this embodiment will be described with reference to FIG. FIG. 5 is a flowchart showing learning of learning information. Each step in FIG. 5 is mainly performed by the learning unit 102 a of the image processing unit 102 . However, the present embodiment is not limited to this, and learning of learning information is performed by a learning unit provided in a device (computing device) separate from the imaging device 100 before correction of an out-of-focus image. may In this embodiment, a case where the learning unit 102a of the imaging device 100 learns learning information will be described.

First, in step S201, the learning unit 102a acquires at least a pair of learning images. A pair of training images differ in the state of deformation of the out-of-focus image (the shape and luminance distribution of the out-of-focus image deformed according to the shooting conditions such as vignetting and aberration of the optical system) and the same subject is present. It is an image. An image in which deformation of an out-of-focus image occurs corresponds one-to-one with an image in which deformation of an out-of-focus image does not occur (an image containing an ideal out-of-focus image), or a single out-of-focus image There may be a plurality of images for which no deformation has occurred. In the latter case, the images with deformed out-of-focus images are a plurality of images with different sizes, shapes, or luminance distributions of the out-of-focus images.

A simulation or a photographed image may be used as a method of preparing a learning image. When the simulation is performed, an image in which the out-of-focus image is not deformed may be generated by performing imaging simulation in consideration of the effects of vignetting and aberration on the image in which the out-of-focus image is not deformed. On the other hand, when a photographed image is used, images obtained by photographing the same subject under different deformation conditions of the out-of-focus image may be used. For example, if the deformation of an out-of-focus image is due to vignetting, the deformation is affected by aperture value, focus position, and the like. Alternatively, by photographing a known object such as a point light source (an image in which the out-of-focus image is not deformed) under different shooting conditions such as the optical system, training images with different deformations of the out-of-focus image can be obtained. good.

Alternatively, a training image may be prepared by estimating an image in which an out-of-focus image is deformed using a technique other than deep learning to estimate an image in which the out-of-focus image is corrected. Note that the training images preferably include images containing various out-of-focus images (various deformed shapes of out-of-focus images). This is because an out-of-focus image cannot be corrected with high accuracy for an image that is deformed in a way that is not included in the training images. In addition, the learning image may include only a pair of images with different deformations of the out-of-focus image, but preferably includes a plurality of images for the reason described above.

Subsequently, in step S202, the learning unit 102a acquires a plurality of learning pairs from the learning images acquired in step S201. A learning pair consists of a learning partial area (learning area) and a learning correction partial area. The learning correction partial area is obtained from the image in which the out-of-focus image is deformed, and its size is the same as the partial area of the photographed image obtained in step S102. The training sub-region is obtained from an image in which no deformation of the out-of-focus image has occurred, and the center of the training sub-region is at the same position in the image as the center of the training correction sub-region. Its size is the same as the corrected partial area generated in step S103. As described above, pairs of learning partial areas and learning correction partial areas (learning pairs) do not need to correspond one-to-one. One learning correction partial area and a plurality of learning partial areas may be paired (grouped).

Subsequently, in step S203, the correction unit 102a acquires (generates) learning information from a plurality of learning pairs (learning partial areas and learning correction partial areas) through learning. Training uses the same network structure that corrects out-of-focus images. In this embodiment, a learning correction partial area is input to the network structure shown in FIG. 1, and the error between the output result and the learning partial area is calculated. In order to minimize this error, for example, using the error backpropagation method, etc., the coefficients of the multiple filters used in the 1st to N+1 layers and the constants to be added (learning information) are updated and optimized. do. The initial values of the coefficients and constants of each filter can be arbitrarily set, and are determined from random numbers, for example. Alternatively, pre-training such as Auto Encoder that pre-learns initial values for each layer may be performed. Details of the Auto Encoder are described in Non-Patent Document 4.

A method of inputting all learning pairs into a network structure and updating learning information using all the information is called batch learning. However, this learning method has a huge computational load as the number of learning pairs increases. Conversely, a learning method in which only one learning pair is used to update learning information and a different learning pair is used for each update is called online learning. Although this method does not increase the amount of calculation even if the number of learning pairs increases, it is greatly affected by noise that exists in one learning pair. Therefore, it is preferable to learn using the mini-batch method, which is located between these two methods. The mini-batch method extracts a small number of pairs from all learning pairs and uses them to update learning information. In the next update, we will extract and use a different fractional training pair. By repeating this process, the disadvantages of batch learning and online learning can be reduced, making it easier to obtain a high correction effect.

Subsequently, in step S204, the correction unit 102a outputs learned information. In this embodiment, the learning information is stored in the storage unit 103 . Through the above processing, learning information for correcting an out-of-focus image can be learned with high precision.

Also, in addition to the above processing, a device for improving the performance of the CNN may be used together. For example, in order to improve robustness, pooling, which is dropout or downsampling, may be performed in each layer of the network. Alternatively, in order to improve the learning accuracy, ZCA whitening, which normalizes the average value of the pixels of the learning image to 0 and the variance to 1 to eliminate the redundancy of adjacent pixels, may be used together. Details of dropout and ZCA whitening are described in Non-Patent Documents 5 and 6, respectively.

According to this embodiment, it is possible to provide an imaging apparatus capable of correcting an out-of-focus image (deformation of an out-of-focus image) from a captured image with high accuracy.

Next, an image processing system according to Embodiment 2 of the present invention will be described. In the image processing system of the embodiment, an image processing device that corrects an out-of-focus image (deformation of the out-of-focus image), an imaging device that acquires a captured image, and a server that performs learning are provided separately. Further, in this embodiment, the learning information to be used is switched by determining the size of the out-of-focus image (out-of-focus image region). By individually learning and using the learning information used for the correction processing of the out-of-focus image according to the size of the out-of-focus image area, the out-of-focus image can be corrected with higher accuracy.

The image processing system in this embodiment will be described with reference to FIGS. 6 and 7. FIG. FIG. 6 is a block diagram of the image processing system 200. As shown in FIG. FIG. 7 is an external view of the image processing system 200. As shown in FIG. As shown in FIGS. 6 and 7, the image processing system 200 includes an imaging device 300, an image processing device 301, a server 305, a display device 308, a recording medium 309, and an output device 310. FIG.

The basic configuration of the image pickup apparatus 300 is the same as that of the image pickup apparatus 100 described with reference to FIG. 2, except for an image processing unit related to correction of out-of-focus images and learning of learning information. A captured image (input image) captured using the imaging device 300 is stored in the storage unit 302 provided in the image processing device 301 . The image processing apparatus 301 is connected to a network 304 by wire or wirelessly, and can access a server 305 via the network 304 . The server 305 has a learning unit 307 that learns learning information for correcting an out-of-focus image from a captured image, and a storage unit 306 that stores the learning information. A correction unit 303 (image processing unit) provided in the image processing apparatus 301 acquires learning information from the storage unit 306 of the server 305 via the network 304 and corrects an out-of-focus image of the captured image. An output image such as an image whose out-of-focus image has been corrected by the correction unit 303 is output to at least one of the display device 308 , the recording medium 309 , and the output device 310 . The display device 308 is, for example, a liquid crystal display or a projector. The user can work while checking the image in the middle of processing via the display device 308 . A recording medium 309 is, for example, a semiconductor memory, a hard disk, or a server on a network. The output device 310 is, for example, a printer. The image processing apparatus 301 may have a function of performing development processing and other image processing as necessary.

Next, correction processing for an out-of-focus image will be described with reference to FIG. FIG. 8 is a flow chart showing an out-of-focus image correction process. Each step in FIG. 8 is mainly executed by the correction unit 303 (image processing unit) of the image processing apparatus 301 .

First, in step S<b>301 , the correction unit 303 acquires a captured image (input image) from the storage unit 302 . Subsequently, in step S302, the correction unit 303 determines the size of the out-of-focus image area included in the captured image. The size of an out-of-focus image area is the size of an area in which an out-of-focus image is generated that varies according to the amount of defocus on the image plane and the aperture value. In this embodiment, these amounts are converted to the number of pixels by dividing them by the pixel size. Three examples of how to determine the size of the out-of-focus image area are given below.

A first example is a method using a plurality of images with different focus positions. Since the size of the out-of-focus image area changes according to the amount of defocus on the image plane, images with different sizes of the out-of-focus image area can be obtained by photographing at different focal positions. Further, by selecting an image with the highest contrast for each partial area from a plurality of images, a deep focus image in which the entire image is in focus can be obtained. By comparing the pan-focus image with the photographed image to be corrected, an out-of-focus image area can be detected as an area in which the brightness is greatly changed due to defocus. Further, since it is possible to obtain the correlation between the blur amount and the distance information for each partial area, the subject distance can be calculated for each partial area by a method generally known as DFD (Depth from Defocus). can be done. Since the amount of defocus on the image plane can be estimated based on the subject distance, the size of the out-of-focus image area can be estimated based on the subject distance and the aperture value obtained by the above method.

A second example is a method using a plurality of images with different aperture values. Since the size of the out-of-focus image area changes according to the aperture value, images with different sizes of the out-of-focus image area can be obtained by photographing with different aperture values. Further, by selecting an image with the highest contrast for each partial area from a plurality of images, a deep focus image in which the entire image is in focus can be obtained. By comparing the pan-focus image with the photographed image to be corrected, an out-of-focus image area can be detected as an area in which the brightness is greatly changed due to defocus. Further, since it is possible to obtain the correlation between the blur amount and the distance information for each partial area, the subject distance can be calculated for each partial area by a method generally known as DFD (Depth from Defocus). can be done. Since the amount of defocus on the image plane can be estimated based on the subject distance, the size of the out-of-focus image area can be estimated based on the subject distance and the aperture value obtained by the above method.

A third example is a method using subject distance (subject distance information). Since the amount of defocus on the image plane can be estimated based on the subject distance, the size of the out-of-focus image area can be estimated based on the subject distance and the aperture value by a method described later. The object distance information can be acquired according to the stereo principle, for example, by configuring the imaging device 300 with a multi-view configuration. In addition, object distance information can be acquired by separately providing a distance measuring unit such as an existing TOF device.

Since the size of the out-of-focus image area may change depending on the position of the image, the size is determined for multiple positions in the captured image, and the learning information is switched according to the position of the partial area. good too.

Subsequently, in step S303, the correction unit 303 determines the network structure and learning information to be used, and the size of the partial region based on the size of the out-of-focus image region determined in step S302. In this embodiment, the corrector 303 corrects the out-of-focus image using the CNN shown in FIG. If the out-of-focus image area is larger than the size of the partial area, the information of the subject to be corrected is lost, and thus the out-of-focus image cannot be corrected with high accuracy. In addition, since CNN corrects out-of-focus images by convolving filters in each layer, if the total range affected by these filters is smaller than the out-of-focus image area of the captured image, the out-of-focus image can be correctly corrected. Can not do it. For example, if the total number of layers is 2, the filter size of the first layer is 5×5, and the filter size of the second layer is 3×3, then the range that can be used to correct the out-of-focus image of a pixel is 7×7 pixels centered on . Therefore, in this network structure, the out-of-focus image cannot be corrected with high accuracy unless the size of the out-of-focus image area is 7 pixels or less. Therefore, the size of the out-of-focus image area is preferably used to determine the range (determined by the size of the filters in each layer) used to correct the out-of-focus image of a pixel.

More specifically, it is preferable to determine the filter size of each layer so as to satisfy the following conditional expression (4).

In conditional expression (4), d is the size of the out-of-focus image area with respect to the pixels of the captured image (input image), and represents the length of the out-of-focus image area in the one-dimensional direction by the number of pixels. N+1 is the total number of layers. s _m (m=1 to N+1) is the size of the filter used in the mth linear transform (the one-dimensional size of the filter in each of the mth linear transforms (m=1 to N+1)). If a mixture of multiple size filters is used in the _m -th linear transform, sm is the maximum filter size. The upper limit of conditional expression (4) indicates that the range that can be used for correcting an out-of-focus image of a certain pixel is greater than or equal to the size of the out-of-focus image area. On the other hand, theoretically, the lower limit of conditional expression (4) cannot be exceeded.

The network structure includes not only the size of filters used in each layer, but also the number of filters used in one layer and the number of layers. In order to correct the out-of-focus image with high accuracy, it is necessary to increase the number of layers and the number of filters as the out-of-focus image area increases. The learning information is learned for each size of the out-of-focus image area, and among them, the learning information corresponding to the size of the out-of-focus image area included in the captured image is used. This makes it possible to correct an out-of-focus image with higher accuracy. The details of learning will be described later.

Subsequently, in step S304, the correction unit 303 acquires a partial area (at least a partial area of the captured image) from the captured image. Subsequently, in step S305, the correction unit 303 generates a correction partial area based on the learning information determined in step S303. Subsequently, in step S306, the correction unit 303 determines whether or not the out-of-focus image correction processing has been completed for all predetermined regions (partial regions) of the captured image. If the out-of-focus image correction has been completed for the entire predetermined area, the process proceeds to step S307. On the other hand, if there remains an area (partial area) for which the out-of-focus image correction has not been completed, the process returns to step S304, and the correction unit 303 selects a partial area (new partial area) for which the out-of-focus image has not yet been corrected. is obtained from the captured image. In step S307, the correction unit 303 outputs an image in which the out-of-focus image has been corrected.

Note that if the size of the out-of-focus image area varies greatly depending on the position in the captured image, the correction unit 303 preferably executes step S304 before steps S302 and S303. At this time, in steps S302 and S303, the correction unit 303 acquires the size of the out-of-focus image area for the local area of the captured image, and acquires the corresponding learning information.

Next, learning of learning information performed by the learning unit 307 of the server 305 will be described. In this embodiment, the learning unit 307 learns different learning information according to the size of the out-of-focus image area. The learning method is basically the same as the method described in the first embodiment with reference to FIG. 5, but the preprocessing differs depending on how the learning images are prepared.

First, a case will be described in which learning images with different deformations of out-of-focus images (shapes of deformed out-of-focus images) are generated by simulation. In this case, by setting the size of the out-of-focus image area, an image in which the out-of-focus image is not deformed is generated from an image in which the out-of-focus image is not deformed, and a pair of learning images are obtained. The learning unit 307 performs steps S201 to S204 on the acquired training images, and then repeats the same procedure for different sizes of out-of-focus image regions.

Next, a case will be described in which learning images are generated by obtaining an image in which an out-of-focus image is not deformed from an image in which an out-of-focus image is deformed. In this case, since the size of the out-of-focus image area is obtained when obtaining an image in which deformation of the out-of-focus image does not occur, the training images are divided into a plurality of groups. Grouping can be performed on an image-by-image basis. If the size of the out-of-focus image area changes in the image in which the deformation of one out-of-focus image occurs, the image may be divided and grouped. Since each group includes out-of-focus image regions having similar sizes, steps S201 to S204 are executed for each group to generate learning information.

According to this embodiment, it is possible to provide an image processing system capable of correcting an out-of-focus image (deformation of an out-of-focus image) from a captured image with high accuracy.

Next, an imaging apparatus according to Embodiment 3 of the present invention will be described. The imaging apparatus of this embodiment estimates the deformation of the out-of-focus image (the shape of the deformed out-of-focus image) from the captured image (input image). The configuration and appearance of the imaging apparatus according to this embodiment are the same as those of the imaging apparatus 100 described in Embodiment 1 with reference to FIGS. However, in the imaging apparatus of this embodiment, an estimation unit is provided as an image processing unit instead of the correction unit 102b.

Processing for estimating deformation of an out-of-focus image from a captured image (input image) will be described with reference to FIG. FIG. 9 is a flow chart showing a process of estimating deformation of an out-of-focus image. Each step in FIG. 9 is mainly executed by the estimation unit of the image processing unit 102 .

First, in step S401, the image processing unit 102 (estimating unit) acquires a captured image (input image) and learning information. Learning of learning information will be described later. Through this learning, learning information is obtained that links the partial area of the captured image and the deformation of the out-of-focus image occurring in the partial area. Subsequently, in step S402, the estimation unit acquires a partial area for estimating deformation of the out-of-focus image from the captured image. In this embodiment, the estimating unit determines whether or not a change in an out-of-focus image has occurred in a pixel included in the partial area (for example, the central pixel), and also includes information on peripheral pixels (pixels included in the partial area). Estimate using

Subsequently, in step S403, the estimation unit estimates deformation of the out-of-focus image based on the learning information acquired in step S401. In this embodiment, the estimator uses the network structure shown in FIG. 10 to perform the estimation. FIG. 10 is a diagram showing a network structure for estimating deformation of an out-of-focus image in this embodiment. In FIG. 10, the processes up to the generation of the intermediate data 410 are the same as the processes described in the first embodiment with reference to FIG. 1, so description thereof will be omitted. That is, the partial area 401, the filter 402, the first transformation partial area 403, the filter 404, and the intermediate data 410 in FIG. and intermediate data 210 .

In this embodiment, the full connection 411 is executed as the (N+1)th linear transformation in the (N+1)th layer. The full connection 411 linearly combines all signals of the input intermediate data 410 . At this time, coefficients applied to each signal and constants to be added are determined by learning information. In addition, there are multiple types of coefficients and constants, and linear combinations are calculated for each of the coefficients and constants, and multiple results are output. A plurality of values output by the full connection 411 are transformed by the activation function (N+1th nonlinear transformation) and input to the softmax 412 . Softmax 412 computes the softmax function expressed in Equation (5) below.

In equation (5), vector x is a column vector whose components are a plurality of values output by the N+1th nonlinear transformation, and vector w is a column vector whose components are coefficients determined from learning information. The T on the right shoulder of the vector w represents transposition. From equation (5), it is possible to obtain the probability of which of the distributions 413a to 413d (413e and after) the deformation of the out-of-focus image occurring in the partial area 401 belongs to. Here, the distribution indicates whether or not deformation of the out-of-focus image exists in the partial area. Each deformation is shown. K in Equation (5) is the total number of distributions, and j and k are indices indicating distribution numbers.

Subsequently, in step S404 of FIG. 9, the estimation unit determines whether or not the estimation of the deformation of the out-of-focus image has been completed for all of the predetermined regions (partial regions) of the captured image. If the estimation has been completed for all of the predetermined regions, the process proceeds to step S405. On the other hand, if there remains an area (partial area) for which the estimation has not been completed, the process returns to step S402, and the estimating unit selects a partial area (new partial area) for which the deformation of the out-of-focus image has not yet been estimated. Get from

In step S405, the estimation unit outputs an estimation result of the deformation of the out-of-focus image in each partial area within the predetermined area. The estimated deformation of the out-of-focus image can be used to analyze the shooting conditions of the optical system (imaging optical system) in which the deformation of the out-of-focus image occurs, and to correct the out-of-focus image from the captured image. can. Techniques other than deep learning may be used to correct out-of-focus images. With the above processing, the deformation of the out-of-focus image can be estimated with high accuracy from the photographed image in which the out-of-focus image is deformed.

Next, generation of learning information in this embodiment will be described with reference to FIG. FIG. 11 is a flowchart showing learning of learning information. Each step in FIG. 11 is mainly performed by the learning unit 102 a of the image processing unit 102 . However, the present embodiment is not limited to this, and learning of learning information may be performed by a learning unit provided in a device (arithmetic device) different from the imaging device 100 . In this embodiment, as in the first embodiment, a case where the learning unit 102a of the imaging device 100 learns the learning information will be described.

First, in step S501, the learning unit 102a acquires a learning image. In the present embodiment, an image obtained by applying deformation of an out-of-focus image due to vignetting using a simulation to an image having no deformation of the out-of-focus image is used as a learning image. The training image may be singular or plural. However, to accurately estimate the deformation of out-of-focus images of various sizes and shapes, the training images should include the deformations of the out-of-focus images due to different vignetting.

Subsequently, in step S502, the learning unit 102a acquires a plurality of learning pairs from the learning images acquired in step S501. In this embodiment, a learning pair is a partial area (learning deformed partial area) of a learning image (an image in which an out-of-focus image is deformed) and information about the distribution acting on this partial area. Information about distribution is a number indicating a specific distribution among a plurality of deformation distributions. The plurality of distributions are distributions 413a to 413d (413e and subsequent ones are omitted) shown in FIG.

Subsequently, in step S503, the learning unit 102a generates learning information based on the learning pair (information on deformation distribution and deformation partial area for learning). The network structure of FIG. 10 is used for generating learning information. Subsequently, in step S504, the learning unit 102a outputs the generated learning information. In this embodiment, similarly to the second embodiment, learning information may be prepared for each size of the out-of-focus image area.

According to this embodiment, it is possible to provide an imaging apparatus capable of estimating deformation of an out-of-focus image with high accuracy from a captured image.

(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

According to each embodiment, it is possible to provide an image processing method, an image processing device, an imaging device, a program, and a storage medium capable of estimating or correcting deformation of an out-of-focus image from a captured image with high precision. .

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist.

102 image processing unit 102b, 303 correction unit (image processing unit)
103, 302 Storage unit 301 Image processing device

Claims (17)

obtaining at least a partial region of an input image;
obtaining pre-learned learning information regarding deformation of the out-of-focus image due to vignetting or aberration effects ;
using the learned information to estimate or correct deformation of out-of-focus images contained in the region;
In the estimating or correcting step, when N is an integer of 2 or more and n is an integer from 1 to N,
intermediate data by sequentially performing n-th linear transformation by each of a plurality of linear functions based on the learning information and n-th nonlinear transformation by a nonlinear function on the region until n is from 1 to N; and
an image processing method, comprising: performing an N+1-th linear transformation on the intermediate data using at least one linear function based on the learning information. further comprising obtaining a size of an out-of-focus image region in the input image where the out-of-focus image deformation occurs;
2. The image processing method of claim 1, wherein the size of the region or the learning information is determined based on the size of the out-of-focus image region. 3. The image processing method according to claim 2, wherein the size of the out-of-focus image area is calculated using an image with an aperture value or focus position different from that of the input image. the input image includes an out-of-focus image deformed from an ideal out-of-focus image shape;
In the estimating or correcting step, the deformation of the out-of-focus image included in the input image is estimated, or the shape of the out-of-focus image included in the input image is approximated to the ideal shape of the out-of-focus image. 4. The image processing method according to any one of claims 1 to 3, wherein the correction is performed as follows. The ideal out-of-focus image is an out-of-focus image acquired without the effects of vignetting or aberrations of the optical system ,
5. The image processing method according to claim 4, wherein said input image includes an out-of-focus image deformed from said ideal out-of-focus image due to the effect of said vignetting or said aberration. 6. The method according to any one of claims 1 to 5, wherein each of the n-th linear transformations (n=1 to N) includes convolution with each of a plurality of filters based on the learning information. Image processing method. 7. The image processing method according to claim 6, wherein said (N+1)th linear transformation includes convolution with a filter based on said learning information. further comprising obtaining a size of an out-of-focus image region in the input image where the out-of-focus image deformation occurs;
8. The method of claim 7, wherein the size of the filter in each of the nth linear transformation (n=1 to N) and the N+1th linear transformation is determined based on the size of the out-of-focus image area. The described image processing method. further comprising obtaining a size of an out-of-focus image region in the input image where the out-of-focus image deformation occurs;
d is the size of the out-of-focus image area for pixels of the input image, and s _m (m= 1 to N+1),

9. The image processing method according to claim 7, wherein the following conditional expression is satisfied. 10. The learning information according to any one of claims 1 to 9, wherein the learning information is information learned using at least a pair of learning images in which the deformation state of the out-of- focus image is different and the same subject is present. 2. The image processing method according to item 1. 11. The image processing method according to claim 10, wherein the pair of training images includes an image in which an out-of-focus image is deformed and an image in which an out-of- focus image is not deformed. 12. The image according to claim 11, wherein the images in which deformation of the out-of- focus image does not occur among the learning images are images obtained by photographing the same subject at different aperture values or different focus positions. Processing method. 13. The image processing method according to any one of claims 10 to 12, wherein the learning image is an image generated by simulation. a storage unit for storing learned information about deformation of the out-of-focus image due to vignetting or aberration effects ;
an image processing unit that estimates or corrects deformation of the out-of- focus image in at least a partial region of the input image using the learning information;
In the image processing unit, when N is an integer of 2 or more and n is an integer from 1 to N,
intermediate data by sequentially performing n-th linear transformation by each of a plurality of linear functions based on the learning information and n-th nonlinear transformation by a nonlinear function on the region until n is from 1 to N; to generate
An image processing apparatus, wherein the intermediate data is subjected to an (N+1)th linear transformation using at least one linear function based on the learning information. an imaging unit that acquires an image of an object space as an input image;
An imaging apparatus comprising: the image processing apparatus according to claim 14 . A program for causing a computer to execute the image processing method according to any one of claims 1 to 13. 17. A storage medium storing the program according to claim 16.

Images