JP7309520B2 - Image processing method, image processing device, imaging device, program, storage medium, image processing system, and learned model manufacturing method - Google Patents

Description

The present invention relates to an image processing method for processing a plurality of input images using a convolutional neural network for a plurality of input images.

In recent years, image processing techniques using convolutional neural networks (CNN) have begun to be used in order to improve the image quality of images. CNN is a learning-type image processing technology that convolves a filter generated by learning with an input image, adds a bias generated by learning, and then repeats nonlinear operations to convert to a desired output image. is. This learning is performed using a learning image consisting of a set of an input learning image and an output learning image. Simply put, learning involves preparing a large number of input learning images corresponding to input images and output learning images corresponding to output images (for example, about tens of thousands of images), and using these learning images as input/output images. It is learning to relate.

For example, Non-Patent Document 1 discloses a CNN that acquires an input learning image with a smartphone camera and an output learning image with a digital single-lens camera, and converts the smartphone camera image, which is the input image, into digital single-lens camera image quality. . As a result, it is possible to obtain an image with image quality close to that of a digital single-lens camera, which is a large imaging device, with a small smartphone camera.

Andrey Ignaov, Nikolay Kobyshev, Radu Timofte, Kenneth Vanhoey, Luc Van Gool, "DSLR-Quality Photos on Mobile Devices with Deep Convolutional Networks" , arXiv:1704.02470v2, United States, 2017

However, in the method using CNN disclosed in Non-Patent Document 1, since only the image acquired by one imaging unit is used as the input image, the effect of restoring high-frequency components and reducing noise is insufficient. In general, the amount of noise in a captured image depends on the pixel size of the image sensor. Therefore, it is possible to obtain an image with a lower noise amount as the pixel size is larger. On the other hand, the smaller the pixel size, the higher the reproducibility of the high frequency components of the subject. That is, the reproducibility of high-frequency components and the reduction of the amount of noise are in a trade-off relationship, and it is difficult to improve the performance of both by using only the image acquired by one imaging unit.

Accordingly, the present invention provides an image processing method, an image processing apparatus, an imaging apparatus, a program, a storage medium, an image processing system, and a trained model capable of reducing noise while restoring high-frequency components of a captured image. The purpose is to provide a method.

An image processing method as one aspect of the present invention includes the step of acquiring a first image of a subject , and the amount of noise is greater than that of the first image, and the high frequency component of the subject is greater than that of the first image. inputting the first image and the second image to a neural network ; using the neural network to detect higher frequency components of the subject than the first image and generating a third image that contains more and has a lower amount of noise than the second image.

An image processing apparatus as another aspect of the present invention includes a first image of a subject , and a second image having a larger amount of noise than the first image and containing more high-frequency components of the subject than the first image. and an acquisition unit that acquires an image of the and a calculator for generating a third image with less noise than the second image.

An imaging device as another aspect of the present invention comprises the image processing device, a first imaging unit that captures the first image , and a second imaging unit that captures the second image. have

A program as another aspect of the present invention causes a computer to execute the image processing method.

A storage medium as another aspect of the present invention stores the program.

An image processing system as another aspect of the present invention is an image processing system having a first device and a second device, wherein the first device comprises a first image of a subject and the first transmitting means for transmitting a request to cause the second device to execute image processing using a second image that has a larger amount of noise than the image and contains more high-frequency components of the subject than the first image ; wherein the second device comprises: receiving means for receiving the request transmitted by the transmitting means; obtaining means for obtaining the first image and the second image; inputting the second image to a neural network, and using the neural network to generate a third image containing more high-frequency components of the subject than the first image and having less noise than the second image; and a calculation unit for generating.

According to another aspect of the present invention, there is provided a method for producing a trained model, comprising: acquiring a first image of a subject ; acquiring a second image containing many high-frequency components; and acquiring a third image containing more high-frequency components of the subject than the first image and having less noise than the second image. and learning parameters of a neural network using the first image, the second image, and the third image.

Other objects and features of the invention are illustrated in the following examples.

According to the present invention, an image processing method, an image processing device, an imaging device, a program, a storage medium, an image processing system, and a trained model capable of reducing noise while restoring high-frequency components of a captured image. A manufacturing method can be provided.

1 is an external view of an imaging device in Example 1. FIG. 1 is a block diagram of an imaging device in Example 1. FIG. FIG. 5 is an explanatory diagram of a network structure for correcting an image in Examples 1 and 2; 5 is a flow chart showing image correction processing in Examples 1 and 2. FIG. 5 is a flowchart showing learning of learning information in Examples 1 and 2; 4 is an image processing result in Example 1. FIG. 4 is a numerical calculation result of image processing in Example 1. FIG. FIG. 11 is a block diagram of an image processing system in Example 2; FIG. 11 is an external view of an image processing system in Example 2;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and overlapping descriptions are omitted.

Before going into the specific description of the embodiments, the gist of the present invention will be described. In the present invention, a convolutional neural network (CNN), which is one of deep learning, is used to reduce noise while restoring high-frequency components for images captured by a plurality of different imaging units. . Each embodiment described below is based on a low-resolution image with a low noise amount (first image) and a high-resolution image with a high noise amount (second image). A small number of high-quality images (third images) are generated.

First, an imaging apparatus according to a first embodiment of the present invention will be described with reference to FIGS. 1 and 2. FIG. 1A and 1B are external views of the imaging device 1, FIG. 1A is a bird's-eye view, and FIG. 1B is a front view. FIG. 2 is a block diagram of the imaging device 1. As shown in FIG. In this embodiment, the imaging apparatus 1 executes an image processing method to restore high-frequency components to the captured image while reducing the amount of noise.

The imaging apparatus 1 has a main imaging unit 100 used for imaging a subject with a wide angle of view, and a sub imaging unit 110 used for imaging a subject with a narrow angle of view. The main imaging unit 100 is configured with an imaging optical system 101 and an imaging element (first imaging element) 102 . The imaging optical system 101 includes one or more lenses, an aperture 101A and a focus lens 101F (focusing mechanism), and forms an image of light from a subject (not shown) on the imaging device 102 . Further, the imaging optical system 101 may be a variable magnification optical system in which one or more lenses provided therein are driven to change the focal length. In FIG. 1, the imaging optical system 101 is configured as a part of the imaging device 1 (integrated with the imaging device 1), but it is replaceable (detachable from the imaging device main body) like a single-lens reflex camera. a) It may be an imaging optical system.

The imaging device 102 is a solid-state imaging device such as a CMOS sensor or a CCD sensor, and photoelectrically converts an optical image (object image) formed via the imaging optical system 101 to output an analog electrical signal (image signal). Mechanical driving of the diaphragm 101A and the focus lens 101F in the imaging optical system 101 is performed by the imaging control unit 40 (focus control unit 41) according to control instructions from the system controller 30. FIG. The imaging control unit 40 controls the aperture diameter of the diaphragm 101A according to the set diaphragm value (F number, F number). The focus control unit 41 performs focus adjustment by controlling the position of the focus lens 101F according to the subject distance.

The A/D converter 10 converts an analog electrical signal generated by photoelectric conversion of the imaging element 102 into a digital signal and outputs the digital signal to the image processing unit 20 . The image processing unit 20 performs so-called development processing such as pixel interpolation processing, luminance signal processing, and color signal processing on the digital signal output from the A/D converter 10 to generate an image. An image generated by the image processing section 20 is recorded in an image recording medium 60 such as a semiconductor memory or an optical disk. Also, the image generated by the image processing section 20 may be displayed on the display section 70 . The information input unit 50 inputs various information according to the user's operation. An example of various information is imaging conditions at the time of image acquisition, specifically, the F value of the main imaging unit 100, ISO sensitivity, and the like.

The sub imaging unit 110 has an imaging optical system 111 and an imaging element (second imaging element) 112 . The imaging optical system 111 is a single focus imaging optical system that forms an image of light from a subject (not shown) on the imaging element 112 . The imaging optical system 111 is an optical system with a narrower angle of view (telephoto) than the imaging optical system 101 . The imaging optical system 111 also has a focus lens 111F. An analog electrical signal (image signal) generated by photoelectric conversion of the image sensor 112 is treated in the same manner as an analog electrical signal (image signal) generated by the image sensor 102, and the image processing unit 20 outputs the signal from the image sensor 112. An image is generated based on the obtained image signal. An image generated by the image processing section 20 can be displayed on the display section 70 in the same manner as the main imaging section 100 . Here, the sub-imaging unit 110 may be detachable from the imaging device 1, and the sub-imaging unit suitable for the main imaging unit 100 is selected from among the plurality of sub-imaging units 110 and attached to the imaging device 1. good too.

The sub-imaging unit 110 is a telephoto imaging unit that captures a relatively narrow angle of view with respect to the main imaging unit 100 . In addition, compared to the image sensor 102 provided in the main imaging unit 100, the image sensor 112 provided in the sub imaging unit 110 has a smaller imaging area in which the pixels constituting the image sensor are arranged, and the pixel size (pixel pitch) is also small. That is, the image (first input image) acquired by the main imaging unit 100 is a wide-angle image with a small amount of noise, and the image (second input image) acquired by the sub imaging unit 110 is relatively The image is a telephoto image with a large amount of noise.

Here, the amount of noise in this embodiment will be described. The amount of noise σ0 contained in the image is obtained by measuring or estimating from the image. When the noise is uniform white Gaussian noise in real space and frequency space, the noise included in the input image can be estimated from MAD (Median Absolute Deviation) as shown in Equation (1) below.

MAD=median(|wHH1−median(wHH1)|) (1)
The MAD is obtained using the median (central value) of the wavelet coefficients wHH1 in the HH1 subband image obtained by wavelet transforming the input image. At this time, the standard deviation of the noise component can be estimated from the relationship between the standard deviation (noise amount σ0) and the MAD expressed by the following equation (2).

σ0=MAD/0.6745 (2)
In the wavelet transform of an image, wavelet transform is applied to the horizontal direction of the image to decompose it into low-frequency components and high-frequency components. do the conversion. By wavelet transform, the image is divided into four parts and frequency-decomposed into four sub-band images having different frequency bands. At this time, the subband image of the upper left low frequency band component (scaling coefficient) is called LL1, and the subband image of the lower right high frequency band component (wavelet coefficient) is called HH1. The upper right (HL1) and lower left (LH1) subband images are obtained by taking the high frequency band components in the horizontal direction and extracting the low frequency band components in the vertical direction, and by taking the low frequency band components in the horizontal direction. High-frequency band components are taken out in the vertical direction.

Also, the number of pixels of the imaging device 112 provided in the sub-imaging unit 110 and the imaging device 102 provided in the main imaging unit 100 are the same. That is, in the same subject area (within the angle of view), the image acquired by the sub imaging section 110 is an image with relatively higher resolution (higher reproducibility of high frequency components) than that of the main imaging section 100 . It can also be said that the image acquired by the sub-imaging unit 110 contains more high-frequency components of the subject than the image acquired by the main imaging unit 100 .

The image processing unit 20 uses the input image to perform high-frequency component restoration processing and noise reduction processing (collectively referred to as correction processing). The image processing section 20 has a learning section 21 and a correction section 22 . The correction unit 22 has an acquisition unit 22a and a calculation unit 22b. Further, the image processing unit 20 calls and uses the learning information stored in the memory (storage unit) 80 when executing the correction processing. Details of the correction process will be described later.

An output image such as a corrected image is displayed on a display unit 70 such as a liquid crystal display or saved in an image recording medium 60 . However, the captured image may be stored in the image recording medium 60 and corrected at any timing. Also, the captured image may be a moving image, and in this case, each frame is corrected. The above series of controls are performed by the system controller 30 .

Next, high-frequency component restoration processing and noise reduction processing (image correction processing) performed by the image processing unit 20 will be described with reference to FIG. FIG. 4 is a flowchart showing high-frequency component restoration processing and noise reduction processing. Each step in FIG. 4 is mainly executed by the image processing section 20 (correction section 22 ) based on commands from the system controller 30 . Note that learning information that has been learned in advance is used in restoration processing and noise reduction processing for high-frequency components, but the details of learning will be described later.

First, in step S101, the image processing unit 20 (correction unit 22) generates a first input image with a low resolution and a small amount of noise, a second input image with a high resolution and a large amount of noise (two input images), and Get learning information. The learning information is information learned in advance by the learning unit 21 in order to associate the two captured images with the image in which the high-frequency components are restored and the noise amount is reduced.

Subsequently, in step S102, the correction unit 22 (acquisition unit 22a) acquires partial images from the two input images. That is, the correction unit 22 (acquisition unit 22a) converts the first input image into a first image based on a first partial region that is a part of the first input image, and the second input image into a second image. Obtaining a second image based on a second subregion that is part of the second input image. High-frequency component restoration processing and noise amount reduction processing are performed in units of partial regions (first partial region, second partial region) (each partial region). In this embodiment, the first partial area and the second partial area are areas corresponding to the first input image and the second input image, respectively, that is, areas corresponding to the same subject area. Note that the first partial area and the second partial area may be all of the first input image and the second input image, respectively.

Subsequently, in step S103, the correction unit 22 uses the learning information, the first partial region, and the second partial region to restore the high-frequency component and reduce the noise amount (correction processing). Generate a correction partial area. Here, details of the correction process will be described with reference to FIG. FIG. 3 shows the network structure of CNN (Convolutional Neural Network), which is one of deep learning.

A CNN has a multi-layered structure, and linear transformation and non-linear transformation using learning information are executed in each layer. When n is an integer from 1 to N, the n-th layer is called the n-th layer, and the linear transformation and the non-linear transformation in the n-th layer are called the n-th linear transformation and the n-th non-linear transformation, respectively. However, N is an integer of 2 or more. Concerning the partial region 201, convolution (first linear transformation by a plurality of linear functions) with each of the plurality of filters 202 is performed in the first layer. A transformation (first nonlinear transformation) is then performed using a nonlinear function called the Activation Function. In FIG. 3 the activation function is indicated as AF. A plurality of partial regions 201 are drawn because the input image (captured image) has a plurality of channels. In this embodiment, each partial area has three channels of RGB (Red, Green, Blue). However, the number of channels is not limited to this. Also, even if the partial area has a plurality of channels, each channel may be individually input to the CNN.

A plurality of filters 202 exist. The correction unit 22 individually calculates the convolution of each of the plurality of filters 202 and the partial region 201 . The coefficients of filter 202 are determined based on the learning information. The learning information may be the coefficients (filter coefficients) of the filter 202 themselves, or the coefficients obtained by fitting the filter 202 with a predetermined function. The number of channels in each filter 202 matches the number of sub-regions 201 . When the number of channels in the partial area 201 is two or more, the filter becomes a three-dimensional filter (the third dimension represents the number of channels). Alternatively, a constant (possibly negative) determined from learning information may be added to the result of convolution.

Examples of the activation function f(x) include the following equations (1) to (3).

Equation (1) is called a sigmoid function, Equation (2) is called a hyperbolic tangent function, and Equation (3) is called a ReLU (Rectified Linear Unit). max in equation (3) represents a MAX function that outputs the maximum value of the arguments. The activation functions f(x) shown in equations (1)-(3) are all monotonically increasing functions. Also, Maxout may be used as the activation function. Maxout is a MAX function that outputs the maximum signal value for each pixel in a plurality of images output from the n-th linear transformation.

In FIG. 3 , the partial area subjected to the first linear transformation and the first nonlinear transformation is called a first transformed partial area 203 . Each channel component of the first transform sub-region 203 is generated from the convolution of the sub-region 201 with each of the plurality of filters 202 . Therefore, the number of channels in the first conversion partial area 203 is the same as the number of filters 202 .

In the second layer, the first transformation partial region 203 is subjected to convolution (second linear transformation) with a plurality of filters 204 determined from learning information in the same manner as in the first layer, and nonlinear transformation using an activation function. (second nonlinear transformation). The filter 204 used in the second layer is generally not the same as the filter 202 used in the first layer. The size and number of filters 204 also need not match the filters 204 . However, the number of channels of the filter 204 and the number of channels of the first conversion partial area 203 match each other. The correction unit 22 acquires the intermediate data 210 by repeating similar calculations up to the Nth layer (executing the nth linear transformation and the nth nonlinear transformation (n=1 to N)).

Finally, in the N+1-th layer, a corrected partial region 212 is obtained by adding a constant to the convolution of the intermediate data 210 and each of the plurality of filters 211 (N+1-th linear transformation). The filter 211 and the constants used here are also each determined based on the learning information. The number of channels of the correction partial area 212 is the same as that of the partial area 201 . Therefore, the number of filters 211 is also the same as the number of channels of the partial area 201 . The components of each channel of the correction partial area 212 are obtained from operations including convolution between the intermediate data 210 and each of the filters 211 (there may be one filter 211). Note that the sizes of the partial area 201 and the size of the correction partial area 212 do not have to match each other. Since there is no data outside the partial area 201 during convolution, the size of the convolution result will be reduced if the calculation is performed only in the area where the data exists. However, the size can be maintained by setting a periodic boundary condition or the like.

The reason why deep learning can exhibit high performance is that high nonlinearity can be obtained by executing nonlinear transformation many times with a multi-layered structure. If the activation function responsible for the nonlinear transformation does not exist and the network is composed only of linear transformations, no matter how many layers there are, there is an equivalent single-layer linear transformation. There is no Deep learning is said to be more likely to achieve high performance because the more layers it has, the more strongly nonlinear it can be obtained. In general, the case of having at least three layers is called deep learning.

Subsequently, in step S104 in FIG. 4, the correction unit 22 performs high-frequency component restoration processing and noise amount reduction processing (correction processing) on all predetermined regions of the first input image and the second input image. A determination is made as to whether or not the processing (that is, the generation of the correction partial area) has been completed. If correction processing has not been completed for all of the predetermined regions, the process returns to step S102, and the correction unit 22 acquires partial regions that have not yet been corrected from the captured image. On the other hand, if the correction process has been completed for all of the predetermined regions (if corrected partial regions have been generated for all of the predetermined regions), the process proceeds to step S105.

In step S105, the correction unit 22 (calculation unit 22b) outputs an image (corrected image) that has been subjected to high-frequency component restoration processing and noise amount reduction processing. A corrected image is generated by synthesizing the generated corrected partial areas. However, if the partial area is the entire captured image (input image), the corrected partial area is the image that has undergone high-frequency component restoration processing and noise amount reduction processing.

By the above processing, it is possible to obtain an image in which the high frequency components of the photographed image are restored and the amount of noise is reduced. In this embodiment, only the same object regions of the first input image and the second input image are corrected. That is, an image with an angle of view equal to or smaller than that of the second input image is generated as the output image. In other words, high-quality telephoto shooting with an angle of view equivalent to that of the sub-imaging unit 110 is performed using images captured by two imaging units, the main imaging unit 100 having a large imaging unit size and the sub-imaging unit 110 having a small imaging unit size. Images can be output. In order to obtain a high-quality telephoto image by normal shooting, the imaging optical system 101 of the main imaging unit 100 needs to be a telephoto lens with an angle of view equivalent to that of the imaging optical system 111 of the sub imaging unit 110. A telephoto lens corresponding to an imaging device having a large imaging area is large in size. According to this embodiment, the main imaging unit 100 capable of imaging with a small amount of noise and the small sub-imaging unit 110 capable of telephoto imaging are provided, and the image processing method described above is executed, thereby reducing the device size. It is possible to output a high-quality telephoto image while maintaining it.

Next, with reference to FIG. 5, learning of learning information (method of manufacturing a learned model) in this embodiment will be described. FIG. 5 is a flow chart showing learning of learning information. Each step in FIG. 5 is mainly performed by the learning unit 21 of the imaging device 1 (image processing unit 20). However, the present embodiment is not limited to this, and the learning of the learning information can be performed by a device (arithmetic device) other than the imaging device 1, as long as it is before the restoration of the high-frequency component and the reduction of the noise amount. It may be done in the provided study section. In this embodiment, a case where the learning unit 21 of the imaging device 1 learns learning information will be described.

First, in step S201, the learning unit 21 acquires at least one set of learning images. The set of learning images is a plurality of images in which the same subject exists, and is a first input learning image that is a wide-angle image with a small amount of noise, a second input learning image that is a telephoto image and has a large amount of noise, and Includes output training images that are telephoto images and have a low amount of noise. The first input learning image and the second input learning image may have a one-to-one correspondence with the output learning image, or a plurality of images may exist for one output learning image. In the latter case, the first input learning image and the second input learning image are a plurality of images with different noise amounts and the like.

As a method of preparing learning images, simulations and photographed images can be used. When performing the simulation, an input learning image may be generated by performing an imaging simulation on the output learning image in consideration of the image quality deterioration factor of the imaging unit. When a photographed image is used, an image of the same subject photographed under the same conditions by the main imaging section 100 and the sub-imaging section 110 of the imaging apparatus 1 may be used. Note that the learning images preferably include subjects having various characteristics. This is because images having features not included in the learning images cannot be corrected with high accuracy.

Subsequently, in step S202, the learning unit 21 acquires a plurality of learning pairs from the learning images acquired in step S201. A learning pair consists of a learning partial area (learning area) and a learning correction partial area. The learning correction partial area is obtained from the first input learning image and the second input learning image, and has the same size as the partial area of the captured image obtained in step S102. A training sub-region is obtained from the output training image, and the center of the training sub-region is at the same position in the image as the center of the training correction sub-region. Its size is the same as the corrected partial area generated in step S103. As described above, pairs of learning partial areas and learning correction partial areas (learning pairs) do not need to correspond one-to-one. One learning correction partial area and a plurality of learning partial areas may be paired (grouped).

Subsequently, in step S203, the learning unit 21 acquires (generates) learning information from a plurality of learning pairs (learning partial areas and learning correction partial areas) through learning. Training uses the same network structure that performs high-frequency restoration and noise reduction. In this embodiment, a learning correction partial area is input to the network structure shown in FIG. 3, and an error between the output result and the learning partial area is calculated. In order to minimize this error, parameters (learning information) such as the coefficients of the multiple filters used in the 1st to N+1 layers and constants to be added are updated using, for example, the error backpropagation method. optimized for The initial values of the coefficients and constants of each filter can be arbitrarily set, and are determined from random numbers, for example. Alternatively, pre-training such as Auto Encoder that pre-learns initial values for each layer may be performed.

A method of inputting all learning pairs into a network structure and updating learning information using all the information is called batch learning. However, this learning method has a huge computational load as the number of learning pairs increases. On the other hand, a learning method that uses only one learning pair for updating learning information and uses a different learning pair for each update is called online learning. Although this method does not increase the amount of calculation even if the number of learning pairs increases, it is greatly affected by noise that exists in one learning pair. Therefore, it is preferable to learn using the mini-batch method, which is located between these two methods. The mini-batch method extracts a small number of pairs from all learning pairs and uses them to update learning information. In the next update, we will extract and use a different fractional training pair. By repeating this process, the disadvantages of batch learning and online learning can be reduced, making it easier to obtain a high correction effect.

Subsequently, in step S204, the learning unit 21 outputs the learned learning information. In this embodiment, the learning information is stored in memory 80 . Through the above processing, it is possible to learn learning information for restoring high-frequency components and reducing the amount of noise. That is, it is possible to manufacture a trained model for restoring high-frequency components and reducing the amount of noise.

Also, in addition to the above processing, a device for improving the performance of the CNN may be used together. For example, in order to improve robustness, pooling, which is dropout or downsampling, may be performed in each layer of the network. Alternatively, in order to improve the learning accuracy, ZCA whitening, which normalizes the average value of the pixels of the learning image to 0 and the variance to 1 to eliminate the redundancy of adjacent pixels, may be used together.

FIG. 6 shows the result of image processing in this embodiment, where FIG. 6(a) is the first image, FIG. 6(b) is the second image, and FIG. 6(c) is obtained by the image processing in this embodiment. An output image and FIG. 6(d) show a correct image, respectively. All the images in FIGS. 6(a) to 6(d) are monochrome images of 256×256 pixels, and the pixel values are normalized to be in the range of [0 1]. Note that all the images are actual images.

FIG. 7 shows numerical calculation results of the image processing in this embodiment, showing the first image (main image), the second image (sub-image), and the output image obtained by the image processing in this embodiment. The image quality is represented by the image quality evaluation index SSIM. Note that SSIM takes a value between 0 and 1, and the closer to 1, the more similar it is to the correct image. From FIG. 7, it can be seen that the value of the image quality evaluation index SSIM of the output image obtained by the image processing of this embodiment is closer to 1 than each of the first image and the second image. Therefore, it is possible to quantitatively convert a wide-angle image with a small amount of noise and a telephoto image with a large amount of noise into a telephoto image in which high-frequency components are restored and the amount of noise is reduced using CNN.

According to this embodiment, it is possible to provide an imaging apparatus capable of executing high-frequency component restoration processing and noise amount reduction processing of a captured image.

Next, Embodiment 2 in which the image processing method of the present invention is applied to an image processing system will be described with reference to FIGS. 8 and 9. FIG. In the present embodiment, an image processing device that corrects a captured image, an imaging device that acquires the captured image, and a server that performs learning exist separately. Also, in this embodiment, the learning information to be used is switched by determining the type of imaging device used for shooting. By individually preparing learning information for each combination of imaging devices used for imaging, more highly accurate image correction becomes possible.

FIG. 8 is a block diagram of the image processing system 200. As shown in FIG. FIG. 9 is an external view of the image processing system 200. As shown in FIG. As shown in FIGS. 8 and 9, the image processing system 200 includes a plurality of imaging devices 300, an image processing device 301, a server 305, a display device 308, a recording medium 309, and an output device 310. .

The imaging device 300 includes a plurality of imaging devices 300a, 300b, . . . , 300n. In this embodiment, for example, among the plurality of imaging devices 300, a general single-lens reflex camera can be used as the main imaging section described in the first embodiment, and a compact camera can be used as the sub-imaging section. Also, a small camera mounted on a smartphone or the like may be used as the sub-imaging unit. In addition, in this embodiment, the plurality of imaging devices is not limited to the case of using two imaging devices, and it is also possible to use three or more imaging devices.

Captured images (input images) captured using a plurality of imaging devices 300 a to 300 n are stored in a storage unit 302 provided in the image processing device 301 . The image processing apparatus 301 is connected to a network 304 by wire or wirelessly, and can access a server 305 via the network 304 . The server 305 has a learning unit 307 that learns learning information for reducing the amount of noise while restoring high-frequency components of the captured image, and a storage unit 306 that stores the learning information. A correction unit 303 (image processing unit) provided in the image processing apparatus 301 acquires learning information from the storage unit 306 of the server 305 via the network 304 and restores the high frequency components of the captured image while reducing the amount of noise. to generate the output image. The generated output image is output to at least one of display device 308 , recording medium 309 and output device 310 . The display device 308 is, for example, a liquid crystal display or a projector. The user can work while confirming the image being processed through the display device 308 . A recording medium 309 is, for example, a semiconductor memory, a hard disk, or a server on a network. The output device 310 is, for example, a printer. The image processing apparatus 301 may have a function of performing development processing and other image processing as necessary. Restoration of high-frequency components, noise reduction processing, and learning of learning information are the same as those in the first embodiment, so description thereof will be omitted.

Thus, in each embodiment, the image processing method includes the steps of acquiring a first image, and acquiring a second image that has a larger amount of noise than the first image and contains many high-frequency components of the subject ( S102). In the image processing method, the first image and the second image are input to a neural network to generate a third image containing more high-frequency components of the subject than the first image and having less noise than the second image. (S103 to S105).

Preferably, the first image and the second image are at least part of the first input image and the second input image, respectively. More preferably, the first input image has a wider angle of view than the second input image. More preferably, the angle of view of the third image is less than or equal to the angle of view of the second input image.

Preferably, the second image is an image obtained by imaging using the image sensor 112 having a smaller pixel pitch than the image sensor 102 used to acquire the first image. Moreover, preferably, the second image is an image obtained by imaging using the imaging device 112 having a smaller imaging area than the imaging device 102 used to obtain the first image. Also preferably, the neural network has at least one convolutional layer.

Preferably, the image processing method has a step of acquiring learning information about a pretrained neural network. The step of generating the third image includes the following first step and second step, where N is an integer of 2 or more and n is an integer from 1 to N. A first step performs n-th linear transformation by each of a plurality of linear functions based on learning information and n-th nonlinear transformation by a nonlinear function on the first image and the second image, where n is 1. to N until intermediate data is generated. A second step is to perform an N+1th linear transformation on the intermediate data by at least one linear function based on the learning information. More preferably, the learning information is a plurality of learning images in which the same subject exists, and is information learned using a first input learning image, a second input learning image, and an output learning image. be. Here, the second input learning image is an image that has a larger amount of noise than the first input learning image and contains many high-frequency components of the subject. The output learning image is an image that contains more high-frequency components of the subject than the first input learning image and has less noise than the second image. More preferably, at least one of the first input learning image and the second input learning image is an image generated by simulation.

(Other examples)
The present invention provides a first device (user terminal such as an imaging device, a smartphone, a PC, etc.) that makes a request related to image processing and substantively controls the entire system, and a second device that performs image processing according to the request. It can also be implemented as an image processing system configured by devices (servers, etc.). For example, the correction unit 303 in the image processing system 200 of the second embodiment is provided on the server 305 side as the second device, and the image processing device 301 as the first device transmits the first image and the second image to the server 305 . may be configured to request the execution of image processing using the image. In this case, the first device (user terminal) has a transmission means for transmitting a request for image processing to the second device (server), and the second device (server) sends the request to the first device (user terminal). ) for receiving a request sent from the

In this case, the first processing device may transmit the first image and the second image to the second device together with the image processing request. However, the second device may acquire the first image and the second image stored in a location (external storage device) other than the first device in response to a request from the first device. Also, after image processing is performed on the first image and the second image by the second device, the second device may transmit the output image to the first device. By configuring the image processing system in this way, it is possible to perform the processing by the correction unit, which has a relatively heavy processing load, on the second device side, and it is possible to reduce the burden on the user terminal side.

Further, the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads the program. It can also be realized by executing processing. It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

According to each embodiment, an image processing method, an image processing apparatus, an imaging apparatus, a program, a storage medium, an image processing system, and a learned model that can reduce noise while restoring high-frequency components of a captured image. can provide a manufacturing method of

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist.

20 Image processing unit (image processing device)
22a acquisition unit 22b calculation unit

Claims (21)

obtaining a first image of the subject ;
Acquiring a second image that has more noise than the first image and contains more high-frequency components of the subject than the first image ;
inputting the first image and the second image into a neural network;
using the neural network to generate a third image containing more high-frequency components of the subject than the first image and having less noise than the second image. Image processing method. 2. The image processing method according to claim 1, wherein said first image and said second image are at least parts of a first input image and a second input image, respectively. 3. The image processing method according to claim 2, wherein said first input image has a wider angle of view than said second input image. 4. The image processing method according to claim 3, wherein the angle of view of said third image is equal to or less than the angle of view of said second input image. 1. The second image is an image obtained by imaging using an imaging device having a pixel pitch smaller than that of the imaging device used to acquire the first image. 5. The image processing method according to any one of 4. 1. The second image is an image obtained by imaging using an imaging device having an imaging area smaller than that of the imaging device used to obtain the first image. 6. The image processing method according to any one of 5. 7. The image processing method according to claim 1, wherein said neural network has at least one convolutional layer. further comprising acquiring learning information about the pre-trained neural network;
In the step of generating the third image, when N is an integer of 2 or more and n is an integer from 1 to N,
n-th linear transformation by each of a plurality of linear functions based on the learning information and n-th nonlinear transformation by a nonlinear function with respect to the first image and the second image, where n is from 1 to N a step of generating intermediate data by sequentially executing until
8. The image processing according to any one of claims 1 to 7, further comprising the step of executing an (N+1)th linear transformation on the intermediate data using at least one linear function based on the learning information. Method. The learning information includes a plurality of learning images in which the same subject is present, and a first input learning image and a second input learning image having a larger amount of noise than the first input learning image and containing a large amount of high-frequency components of the subject. Information learned using two input learning images and an output learning image containing more high-frequency components of the subject than the first input learning image and having less noise than the second input learning image 9. The image processing method according to claim 8, wherein: 10. The image processing method according to claim 9, wherein at least one of said first input learning image and said second input learning image is an image generated by simulation. obtaining a first partial image from the first image;
obtaining a second partial image from the second image;
further comprising obtaining a third partial image by inputting the first partial image and the second partial image into the neural network;
11. The image processing method according to claim 1, wherein said third image is generated by synthesizing said third partial images. a first image of a subject , and an acquisition unit that acquires a second image that has a larger amount of noise than the first image and contains more high-frequency components of the subject than the first image ;
The first image and the second image are input to a neural network, and the neural network contains more high-frequency components of the subject than the first image and has a noise amount that is greater than that of the second image. and a calculator for generating a small number of third images. 13. The image processing apparatus according to claim 12 , further comprising a storage unit for storing learning information related to the neural network learned in advance. The acquisition unit acquires a first partial image from the first image, acquires a second partial image from the second image,
The calculation unit acquires a third partial image by inputting the first partial image and the second partial image into the neural network, and synthesizes the third partial image to obtain the third partial image. 14. The image processing apparatus according to claim 12, wherein three images are generated. An image processing device according to any one of claims 12 to 14;
a first imaging unit that captures the first image ;
and a second imaging unit that captures the second image . A program for causing a computer to execute the image processing method according to any one of claims 1 to 11 . 17. A storage medium storing the program according to claim 16 . An image processing system having a first device and a second device,
The first device is
The second device performs image processing using a first image of a subject and a second image that has a larger amount of noise than the first image and contains more high-frequency components of the subject than the first image. a sending means for sending a request to cause the
The second device is
receiving means for receiving the request transmitted by the transmitting means;
acquisition means for acquiring the first image and the second image;
The first image and the second image are input to a neural network, and the neural network contains more high-frequency components of the subject than the first image and has a noise amount that is greater than that of the second image. and a calculator for generating fewer third images. the acquisition means acquires a first partial image from the first image and acquires a second partial image from the second image;
The calculation unit acquires a third partial image by inputting the first partial image and the second partial image into the neural network, and synthesizes the third partial image to obtain the third partial image. 19. The image processing system of claim 18, wherein three images are generated. obtaining a first image of the subject ;
Acquiring a second image that has more noise than the first image and contains more high-frequency components of the subject than the first image ;
Acquiring a third image that contains more high-frequency components of the subject than the first image and has less noise than the second image;
and learning parameters of a neural network using the first image, the second image, and the third image. obtaining a first partial image from the first image;
obtaining a second partial image from the second image;
further comprising obtaining a third partial image by inputting the first partial image and the second partial image into the neural network;
21. The method of manufacturing a trained model according to claim 20, wherein said third image is generated by synthesizing said third partial images.