JP7246900B2 - Image processing device, image processing system, imaging device, image processing method, program, and storage medium - Google Patents

Description

The present invention relates to an image processing apparatus that uses a convolutional neural network (CNN) to measure the intensity of disturbance with high accuracy.

Conventionally, there has been proposed a method of recovering image deterioration caused by disturbance (atmospheric fluctuation) by image processing. In Non-Patent Document 1, image degradation caused by disturbance is recovered by correcting the positional deviation of each frame of a moving image due to disturbance, correcting blurring that differs depending on the location of each frame, and then removing the blurring by blind deconvolution. A method for doing so is disclosed.

Xiang Zhu, Peyman Milanfar, "Removing atmospheric turbulence via space-invariant deconvolution", IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 1, 2016, pp. 157-170 Xia-Jiao Mao, Chunhua Shen, Yu-Bin Yang, "Image restoration using convolutional auto-encoders with symmetric skip connections", arXiv:1606.08921, 2016 Ｘａｖｉｅｒ Ｇｌｏｒｏｔ、Ｙｏｓｈｕａ Ｂｅｎｇｉｏ、「Ｕｎｄｅｒｓｔａｎｄｉｎｇ ｔｈｅ ｄｉｆｆｉｃｕｌｔｙ ｏｆ ｔｒａｉｎｉｎｇ ｄｅｅｐ ｆｅｅｄｆｏｒｗａｒｄ ｎｅｕｒａｌ ｎｅｔｗｏｒｋｓ」、Ｐｒｏｃｅｅｄｉｎｇｓ ｏｆ ｔｈｅ １３ｔｈ Ｉｎｔｅｒｎａｔｉｏｎａｌ Ｃｏｎｆｅｒｅｎｃｅ ｏｎ Ａｒｔｉｆｉｃｉａｌ Ｉｎｔｅｌｌｉｇｅｎｃｅ ａｎｄ Ｓｔａｔｉｓｔｉｃｓ、２０１０、ｐｐ． 249-256

However, although the method disclosed in Non-Patent Document 1 can remove atmospheric fluctuations, it is not possible to measure the intensity of atmospheric fluctuations, which is the degree of image deterioration caused by atmospheric fluctuations. Estimation of deformation vectors of non-rigid body registration from atmospheric fluctuation videos can be considered as intensity measurement of disturbances. Deformation vectors cannot be estimated.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an image processing device, an image processing system, an imaging device, an image processing method, a program, and a storage medium that can measure the intensity of disturbance with high accuracy.

An image processing apparatus as one aspect of the present invention includes an image acquisition unit that acquires a plurality of temporally different images degraded by a disturbance, and a learning process using a plurality of image groups obtained based on the known intensity of the disturbance. a parameter acquiring unit for acquiring network parameters of the neural network generated by; and generating a plurality of normalized images by subtracting an average image of the plurality of images from each of the plurality of images, the network parameters and a measurement unit that measures the intensity of the disturbance from the plurality of normalized images using the neural network having

An image processing system as another aspect of the present invention is an image processing system comprising the image processing device and a client device connected to the image processing device via a network, wherein the client device comprises the An image output unit for outputting a plurality of temporally different images degraded by the disturbance to the image processing device, and the image processing device further includes a disturbance intensity output unit for outputting the intensity of the disturbance to the client device. .

An imaging device as another aspect of the present invention has an imaging element and the image processing device.

An image processing method as another aspect of the present invention uses an image acquisition step of acquiring a plurality of temporally different images degraded by a disturbance, and a plurality of groups of images obtained based on a known intensity of the disturbance. a parameter acquisition step of acquiring network parameters of a neural network generated by learning ; and generating a plurality of normalized images by subtracting an average image of the plurality of images from each of the plurality of images, the network measuring the intensity of the disturbance from the normalized images using the neural network with parameters.

A program as another aspect of the present invention causes a computer to execute the image processing method.

A storage medium as another aspect of the present invention stores the program.

Other objects and features of the invention are described in the following embodiments.

According to the present invention, it is possible to provide an image processing device, an image processing system, an imaging device, an image processing method, a program, and a storage medium capable of measuring the intensity of disturbance with high accuracy.

1 is a block diagram of an image processing apparatus in this embodiment; FIG. 1 is a configuration diagram of an image processing system in this embodiment; FIG. 1 is a configuration diagram of an imaging device according to this embodiment; FIG. 1 is a block diagram of an image processing system in this embodiment; FIG. 4 is a flow chart of an image processing method according to the present embodiment; 1 is a diagram showing a network structure in Example 1; FIG. 4 is a diagram showing numerical calculation results in Example 1. FIG. FIG. 10 is a diagram showing a network structure in Example 2; FIG. 10 is a diagram qualitatively showing numerical calculation results in Example 2; FIG. 10 is a diagram quantitatively showing numerical calculation results in Example 2;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

First, the disturbance will be explained. A captured image is degraded by turbulence of a medium existing between an imaging device and a subject. For example, when photographing under the scorching sun or photographing a distant subject, the photographed image deteriorates due to atmospheric fluctuations (turbulence). Further, for example, when photographing an object on the bottom of the water from above, the photographed image deteriorates due to fluctuations (turbulence) in the water.

Degradation of captured images due to medium disturbance is caused by changes in the refractive index of the medium depending on location and time. For this reason, generally obtained captured images have different degrees of deterioration depending on the location. This is because the thickness, temperature distribution, flow, etc. of the medium existing between the imaging device and the subject differ from place to place, and as a result, the refractive index of the medium differs from place to place. For the same reason, the degree of deterioration of the captured image differs from time to time.

Positional deviation correction (registration) of each frame of a moving image due to disturbance (atmospheric fluctuation) disclosed in Non-Patent Document 1 is performed using non-rigid registration. Here, the non-rigid registration is a simple atmospheric fluctuation (correction) model often used in the field of image processing. Briefly, first, rough image control points are set in the original image to be degraded by atmospheric fluctuations. Next, a vector representing the amount of deformation to be applied at each control point (deformation vector) is randomly determined using normal random numbers to deform the control point. Simply put, when the variance of the normal random numbers of the deformation vectors is large, the deterioration of the resulting image due to atmospheric fluctuations becomes large.

Next, from the deformed control points, the amount of deformation due to atmospheric fluctuations to be applied to each pixel of the original image is determined according to the following equation (1).

In equation (1), Δx is the amount of deformation applied to each pixel of the original image, p is the amount of deformation applied to the control point (deformation vector), A(x) is the matrix for transforming p into Δx, and ε _x and ε _y are These are the distances between the control points in the x and y directions, respectively. Also, (x _c , y _c ) are the coordinates of an arbitrary control point, and (x _i , y _i ) are the coordinates of the i-th pixel of the original image. Formula (1) is simply a formula for adding smooth deformation to each pixel of the original image along the deformed control points.

Finally, the pixel value of each deformed pixel is determined by interpolation from the original image to generate an atmospheric fluctuation image. This model is also called B-Spline. In addition, this model can be applied to image degradation caused by arbitrary disturbances regardless of the type of medium (air or water).

In the case of positional deviation correction, on the contrary, a deformation vector is estimated by iterative calculation from an atmospheric fluctuation image and a reference image, and the deformation applied to the atmospheric fluctuation image is corrected from the obtained deformation vector. Here, the reference image is a reference image that can be regarded as having no image deterioration due to atmospheric fluctuations, and is given by, for example, averaging a plurality of atmospheric fluctuation images. The details are disclosed in Non-Patent Document 1.

When creating an atmospheric fluctuation moving image, the above-described process of adding atmospheric fluctuation may be performed for each frame of the original moving image to which the atmospheric fluctuation is to be added. In this case, since there is no correlation between the atmospheric fluctuations applied to each frame, the obtained atmospheric fluctuation moving image is different from reality (atmospheric fluctuation real moving image). However, the obtained atmospheric fluctuation animation is qualitatively similar to the real atmospheric fluctuation animation. Also, the method disclosed in Non-Patent Document 1 can satisfactorily recover image deterioration due to atmospheric fluctuations. Therefore, since the B-Spline-based disturbance model is considered to be close to reality, it is also used in the present invention to create a training video (training image group) degraded by disturbance for CNN learning, which will be described later.

To correct the blur that differs depending on the location of each frame, select the sharpest one (with the largest pixel value variance) for a certain period of time (frame) in the region of interest, apply this to the entire image, and It is also done by joining. This process is called lucky imaging. Blind deconvolution is performed by simultaneously estimating both a PSF (Point Spread Function) representing blur due to atmospheric fluctuations and an image from which atmospheric fluctuations are removed.

Next, the image processing apparatus according to this embodiment will be described with reference to FIG. FIG. 1 is a block diagram of an image processing apparatus 100. As shown in FIG. The image processing apparatus 100 includes an image acquisition section 101 , a parameter acquisition section 102 , a measurement section 103 and a correction section 104 . The image acquisition unit 101 acquires a plurality of images (input images, moving images) captured by an imaging device. The imaging device is capable of acquiring digital moving image data, and is, for example, a digital video camera or a digital camera.

Motion picture frames are generally degraded. For example, in the case of a digital camera, deterioration factors include blur caused by the imaging optical system (imaging optical system) and optical low-pass filter, noise caused by the image sensor, demosaicing errors, and noise caused by data compression. . It is desirable that these moving image deterioration processes are known. This is because a large number of training image groups (training videos) required for learning of the CNN, which will be described later, can be generated by numerical calculation. The format of moving image data is not limited as long as it is computer-readable digital data, and examples thereof include AVI (Audio Video Interleave) and MPEG (Moving Picture Experts Group). Also, in the present embodiment, the moving image may be either color or monochrome, but for the sake of simplicity, the following description assumes a monochrome moving image.

In addition, the image acquisition unit 101 uses the optical conditions (focal length, aperture value, etc.) of the optical system (imaging optical system) used for photographing, the pixel pitch of the image sensor used for photographing, as the photographing conditions of the plurality of input images. , or to get the frame rate. This is to match the learning conditions of the CNN, which will be described later, with the shooting conditions (conditions for acquiring the input image).

The parameter acquisition unit 102 acquires learned network parameters. Note that the network parameters include filters and biases, which are CNN parameters described later. A CNN is simply an operation using learned parameters, and is composed of, for example, a PC (Personal Computer), a workstation, an FPGA (Field Programmable Gate Array), or a server. Therefore, the parameter acquisition unit 102 is configured by, for example, an HDD (Hard Disk Drive) of a PC. Alternatively, the parameter acquisition unit 102 may acquire a storage medium storing network parameters via an interface device such as a CD-ROM drive or a USB interface. In this case, the parameter acquisition unit 102 is configured to include an interface device.

A learned network parameter is a CNN network parameter that constitutes a measuring unit 103 and a correcting unit 104, which will be described later, and is generated in advance by learning. Alternatively, a condition (imaging condition) under which the input image provided from the image acquisition unit 101 is acquired and a network parameter learning condition that are close to each other may be selected and used as the parameter to be acquired. Here, the learning conditions are imaging conditions (optical conditions of the optical system, pixel pitch, frame rate, etc.) when numerically generating (or acquiring) a group of training images used for learning of the CNN, which will be described later.

Next, CNN will be briefly described. CNN is a learning-type image processing technology that repeats non-linear operations after convolution of a filter generated by training or learning with respect to an image. After convolving the filter with respect to the image, the image obtained by non-linear operation is called a feature map. Learning is performed using training images or data sets consisting of pairs of input and output images. In simple terms, learning is the process of generating, from a set of training images, filter values that can convert an input image into a corresponding output image with high accuracy. Details will be described later.

Also, if the image has RGB color channels or is composed of multiple images (moving image), or if the feature map is composed of multiple images, the filter used for convolution will be multi-channel accordingly. have. That is, the convolution filter is represented by a four-dimensional array in which the number of channels is added to the vertical and horizontal size and the number of images. Further, the process of performing non-linear calculation after convolving an image (or feature map) with a filter is expressed in units of layers. For example, it is called an m-th layer feature map or an n-th layer filter. For example, a CNN that repeats filter convolution and nonlinear operation three times is called a three-layer network. This processing can be formulated as in the following equation (2).

In equation (2), _Wn is the n-th layer filter, _bn is the n-th layer bias, f is the non-linear operator, _Xn is the n-th layer feature map, and * is the convolution operator. Note that (l) in the right shoulder represents the l-th filter or feature map. Filters and biases are generated by learning, which will be described later, and are collectively called network parameters. A sigmoid function and ReLU (Rectified Linear Unit) are often used as nonlinear operations.

Next, learning of CNN will be described. CNN learning is generally performed by the following equation (3) for training images (training image group) consisting of a set of input training images (e.g. degraded images) and corresponding output training images (e.g. sharp correct images). It is done by minimizing the objective function represented.

In equation (3), L is a loss function that measures the error between the correct answer and its estimate. Y _i is the i th output training image and X _i is the i th input training image. F is a function collectively representing the operations (Formula 2) performed in each layer of the CNN. θ is the network parameters (filter and bias). Also, |Z|| ₂ is the L2 norm, which is simply the square root of the sum of the squares of the elements of vector Z.

Input/output images having a known correspondence relationship are used as training images. For example, there is a sharp output image and an input image that is degraded by adding blur due to an optical system. Also, when the output of the CNN is not an image but a scalar (value), the loss function should be similarly defined and the network parameters determined. In that case, the training images are the input images and the corresponding output values. A CNN that outputs a scalar is a special type called a fully connected neural network, and details of which will be described later.

In formula (3), n is the total number of training images used for learning, but since the total number of training images is generally large (up to tens of thousands of images), in the stochastic gradient descent method (SGD), A part of the training image is randomly selected and used for learning. This makes it possible to reduce the computational load in learning using many training images.

Also, various methods such as the momentum method, the AdaGrad method, the AdaDelta method, and the Adam method are known as methods for minimizing (=optimizing) the objective function. However, there is currently no selection guideline for optimization methods in learning. Therefore, basically any method can be used, but it is known that the learning time differs due to the difference in convergence between optimization methods.

Using the network parameters and CNN learned by the above procedure, image processing can be performed to convert, for example, a degraded image into a sharp image with high accuracy. This image processing is also called deep learning.

The measuring unit 103 measures and outputs the intensity of the disturbance from the input image (a plurality of images) using the learned network parameters and the CNN. The measurement unit 103 is the above-described CNN, and includes, for example, a PC, a workstation, an FPGA, and a server, but is not limited to these, and may be a computer capable of realizing the above-described CNN calculation. . The intensity of the disturbance is a scalar representing the intensity of the disturbance of the input image, specifically, the temporal or spatial distribution of pixel values of the input image. Here, the degree of dispersion is a statistic including variance and standard deviation. For example, when the B-Spline described above is used as the disturbance model, the variance of the deformation vector described above may be used as the strength of the disturbance. Thus, one of the features of the present invention is that the intensity of the disturbance in the input image is represented by a scalar, that is, the temporal or spatial distribution of pixel values.

As mentioned above, the output of a CNN is an image. Therefore, in order to output the scalar intensity of the disturbance, a fully-connected neural network that converts the image, which is the output of the CNN, to a scalar should be added to the output part. A fully-connected neural network can be formulated as in Equation (4) below.

In equation (4), _Xn is a vector representing the n-th layer feature map, and _Wn is a matrix representing a weight to be added to each element of _Xn-1 . Therefore, it is necessary to convert the image output by the CNN into a vector and then input it to the fully-connected neural network. For example, a 50×50 pixel image is transformed into a 2500-dimensional vector. Also, the image size that can be input to the fully-connected neural network is defined by the size of the fully-connected neural network. Therefore, it is necessary to adjust the size of the input image to the CNN so that an output image that can be input to the fully-connected neural network can be obtained.

Note that even a CNN added with a fully-connected neural network can be learned by the method described above. This is because, historically, fully-connected neural networks were first studied, and CNNs were later studied as a derivation thereof, but details thereof will be omitted. The measurement unit 103 is precisely configured with a CNN and a fully-connected neural network added to its output part, but is simply referred to as "the CNN of the measurement unit 103".

In addition, training images (training image group) composed of pairs of input training images and their disturbance intensities are used for the learning of the CNN network parameters by the measuring unit 103 . However, it is generally difficult to obtain degraded input training images with known disturbance strengths. Therefore, it may be generated numerically using the disturbance model B-Spline described above, for example. In this case, the variance of the deformation vector described above can be used as the intensity of the disturbance. For example, the network parameters may be a plurality of temporally different first image groups (first moving images) and a plurality of second image groups degraded by a known disturbance strength for the first image groups. It is generated by learning using a training image group (training video) consisting of a set of (second video). Multiple images that are "temporally different" include multiple images acquired at different times.

In addition, by using input training images including moving objects as training images, it is possible to measure the intensity of disturbances robust (highly accurate) to moving objects. Similarly, if the motion picture deterioration process of the image acquisition unit 101 is known, an input training image including the deterioration is generated by numerical calculation and used as the training image. can be measured.

(standardization)
The measurement unit 103 normalizes the input image or the input training image. The purpose of this is to prevent the measurement result from being influenced by the absolute value of the pixel value of the input image. Normalization is performed, for example, by generating an average image of multiple input images and subtracting the average image from each of the multiple input images. This normalization method can be formulated as the following equation (5).

In equation (5), I _i is the i-th input image, and m is the number of input images. Also, a bar (-) above I _i indicates normalization.

Standardization can also be performed, for example, by generating a difference image between two temporally adjacent images from a plurality of input images. This normalization method can be formulated as in Equation (6) below.

The symbols used in formula (6) have the same meanings as in formula (5) above. The normalization method can be appropriately selected according to the definition of the intensity of the disturbance, the input image, the measurement accuracy of the intensity of the disturbance, etc. In this embodiment, basically the normalization method given by Equation (5) is used. . Thus, in this embodiment, preferably, the intensity of the disturbance is measured using the normalized input image. As a result, it is possible to eliminate the influence of the absolute values of the pixel values of the input image on the measurement results, so that highly accurate measurement results can be obtained.

(input image size adjustment)
The number of vertical and horizontal pixels of the image input to the measurement unit 103 is determined by a fully connected neural network added to the output part of the network. For this reason, it is necessary to adjust the number of vertical and horizontal pixels of each image by trimming, interpolating, or thinning the input image and the input training image.

Also, the number of images (number of frames) to be input to the measurement unit 103 is determined according to the number of frames of the input training images. Therefore, the input image needs to be temporally interpolated or thinned out before being input to the measurement unit 103 . For example, when the input images are acquired at a high frame rate, the input images thinned out so as to match the frame rate of the input training images are input to the measurement unit 103 . This is to prevent the measurement result from being affected by the difference in frame rate between the input image and the input training image. Also, the condition for acquiring the input image may be acquired from the image acquisition unit 101, and the number of frames of the input training image may be acquired from the parameter acquisition unit 102, respectively.

Thus, in this embodiment, the intensity of disturbance is preferably measured using an input image whose image size (especially frame rate) has been adjusted. As a result, it is possible to eliminate the influence of the difference in frame rate between the input image and the input training image on the measurement result, so it is possible to obtain a highly accurate measurement result.

(measurement at multiple locations)
The measuring unit 103 may measure the intensity of the disturbance at multiple locations in the input image and determine the final intensity of the disturbance. Here, multiple locations in the input image means multiple locations in the space and time (vertical, horizontal, frame) of the input image. More specifically, an image having the number of vertical and horizontal pixels and the number of frames is extracted from a certain location in the input image in the spatial and temporal directions by the method described above, and is input to the measurement unit 103 as an input image. Measure the intensity of the disturbance.

Alternatively, the final disturbance intensity may be obtained by, for example, averaging the intensity of the disturbance measured at a plurality of locations in the input image. Alternatively, an intermediate value, minimum value, maximum value, or mode value may be obtained from the disturbance intensities measured at a plurality of locations, and used as the final disturbance intensity. This is to prevent the measurement result from being affected by the portion of the input image that is locally greatly degraded due to disturbance.

Thus, in this embodiment, preferably, the disturbance strength is measured at a plurality of locations in the input image to determine the final disturbance strength. As a result, highly accurate measurement results can be obtained even if there is a localized portion of the input image that is significantly deteriorated due to disturbance.

The correction unit 104 corrects deterioration of the input image due to the disturbance based on the measured strength of the disturbance. Here, although the correction method is not limited, it is preferable to use the above-described CNN from the viewpoint of accuracy of image processing. Therefore, the correction unit 104 is exemplified as a CNN below. The correction unit 104 is the above-described CNN and includes, for example, a PC, a workstation, an FPGA, and a server, but is not limited to these, and may be any computer capable of realizing the above-described CNN calculations. The correction unit 104 performs correction processing using learned network parameters provided by the parameter acquisition unit 102 . Also, for the learning of the network parameters of the CNN by the correcting unit 104, a training image is used that is a pair of an output training image and an input training image obtained by adding deterioration of known disturbance intensity to the output training image. However, it is generally difficult to obtain such training images. For this reason, training images may be generated numerically using, for example, the aforementioned disturbance model B-Spline. In this case, the variance of the deformation vector described above can be used as the intensity of the disturbance.

In addition, by using input/output training images including moving objects as training images, it is possible to correct disturbances robust to moving objects. Similarly, if the moving image degradation process of the image acquisition unit 101 is known, an input training image including the degradation is generated by numerical calculation and used as the training image, thereby making it possible to correct disturbance robustly against degradation.

(Network parameter selection)
The correction unit 104 selects the learned network parameters provided by the parameter acquisition unit 102 based on the measured intensity of the disturbance, and performs correction processing. This is because highly accurate disturbance correction is performed by using network parameters learned from training images having the same disturbance intensity as the input image. For example, network parameters learned from training images that have been degraded by the measured intensity of the disturbance and the intensity of the nearest disturbance may be selected and used in the correction process. In this manner, highly accurate disturbance correction can be performed by selecting learned network parameters and performing correction processing based on the measured disturbance intensity.

(number of frames)
The correction unit 104 determines the number of input images (the number of frames) based on the measured intensity of the disturbance. This is because if the disturbance is large, a large number of input images are required for correction, and conversely, if the disturbance is small, a large number of input images are not required for correction. When the correction unit 104 performs CNN network parameter learning, the number of input training images is adjusted according to the disturbance intensity applied to the training images. Specifically, if the disturbance is large, the number of input training images is increased, and if the disturbance is small, the number of input training images is decreased for learning. Thus, by determining the number of input images (the number of frames) based on the measured disturbance intensity, it is possible to determine data necessary for correction.

An output image as a result of image processing obtained by the correction unit 104 can be stored in a storage unit (not shown) provided inside the image processing apparatus 100 . The output image may also be displayed on a display unit (not shown) provided outside the image processing apparatus 100 . Alternatively, the output image may be stored in an external storage medium (not shown) of the image processing apparatus 100 via an interface device such as a CD-ROM drive (not shown) or a USB interface. A description of wiring and wireless communication for exchanging information (data) among the image acquiring unit 101, the parameter acquiring unit 102, the measuring unit 103, and the correcting unit 104 will be omitted.

The functions of the image processing apparatus 100 may be realized on the computer by causing the computer to execute a program describing the functions of the image acquisition unit 101, the parameter acquisition unit 102, the measurement unit 103, and the correction unit 104. Similarly, a program describing the function of at least one of the measuring unit 103 and the correcting unit 104 may be implemented as an electronic circuit on VLSI to implement a part of the functions of the image processing apparatus 100 .

Next, referring to FIG. 2, the image processing system according to this embodiment will be described. FIG. 2 is a configuration diagram of the image processing system 200. As shown in FIG. The image processing system 200 includes an image processing device 100 a and an imaging device (digital camera) 201 . The imaging device 201 has an imaging optical system and an imaging device, and acquires a captured image. A photographed image acquired by the imaging device 201 is output to the image processing device 100a. The image processing device 100a has a PC and a display. The PC has an image acquisition section 101 , a parameter acquisition section 102 , a measurement section 103 and a correction section 104 . The display displays an output image as a result of image processing.

Next, the imaging device according to this embodiment will be described with reference to FIG. FIG. 3 is a configuration diagram of the imaging device 300. As shown in FIG. The imaging device 300 is configured including a camera body 301 and a lens device (interchangeable lens) 302 . The camera body 301 has an image sensor 303 , an image processing engine (image processing device) 304 and a monitor 305 . The image processing engine 304 has an image acquisition unit 101 , a parameter acquisition unit 102 , a measurement unit 103 and a correction unit 104 . A monitor 305 displays an output image as a result of image processing.

Next, another image processing system according to this embodiment will be described with reference to FIG. FIG. 4 is a block diagram of an image processing system 400. As shown in FIG. The image processing system 400 has a client device 401 and a server device 402 connected to the client device 401 via a network 403 . The client device 401 has an image output unit 404 . The image output unit 404 outputs to the server device 402 a plurality of temporally different images degraded by the disturbance. The server device 402 has a parameter acquisition section 405 , a measurement section 406 and a disturbance intensity output section 407 . The parameter acquisition unit 405 acquires learned network parameters. A measurement unit 406 measures the intensity of disturbance from a plurality of images using network parameters and a neural network. A disturbance intensity output unit 407 outputs the intensity of the disturbance to the client device 401 . Also, the client device 401 or the server device 402 may have a correction unit (not shown) that corrects a plurality of images based on the intensity of the disturbance.

Next, an image processing method according to this embodiment will be described with reference to FIG. FIG. 5 is a flow chart of the image processing method, showing the program flow when the image processing method is implemented by VLSI or the like. Each step in FIG. 5 is executed by the image acquisition unit 101, the parameter acquisition unit 102, the measurement unit 103, and the correction unit 104 of the image processing apparatus 100, for example.

First, in step S501, the image acquisition unit 101 acquires a plurality of temporally different images (input images, moving images) degraded by disturbance. Subsequently, in step S502, the parameter acquisition unit 102 acquires learned network parameters. Subsequently, in step S503, the measurement unit 103 measures the intensity of disturbance from a plurality of images using network parameters and a neural network. Finally, in step S504, the correction unit 104 corrects the multiple images based on the intensity of the disturbance.

Next, each embodiment will be described in detail.

First, Embodiment 1 of the present invention will be described with reference to FIGS. 6 and 7. FIG. In this embodiment, a program describing the functions of the image processing apparatus 100 is used to explain the results of numerical calculations in which the disturbance intensity of an input image degraded by a known disturbance intensity is measured.

FIG. 6 is a diagram showing a network structure (CNN of measurement unit 103) 600 in this embodiment. In FIG. 6, conv indicates a convolutional layer, and deconv indicates an inverse (transposed) convolutional layer. Also, the number strings on each layer represent the vertical and horizontal size of the filter, the number of channels, and the number of sheets. For example, "3×3×1×8" in FIG. 6 indicates that convolution or inverse (transposition) convolution is performed using filters with a size of 3×3, 1 channel, and 8 filters. Inverse (transposed) convolution is a kind of convolution, and is simply the inverse process of convolution. Details are disclosed in Non-Patent Document 2, for example. In addition, the + mark in the circle in FIG. 6 represents the sum of each element of the feature map.

Also, the FC at the output of the network structure 600 indicates a fully connected network. The numbers above the fully-connected network represent the input and output sizes to the fully-connected network. For example, "2500×2500" in FIG. 6 indicates that a 2500-dimensional vector is input and a 2500-dimensional vector is output. More specifically, a 50×50 pixel image output by the CNN is converted into a 2500-dimensional vector and input to the fully-connected network. As described above, the image size that can be input to the fully-connected network is fixed. The input image size is determined accordingly. In this embodiment, the input image size is 50×50 pixels and 11 frames.

It should be noted that the network structure 600 shown in FIG. 6 is only an example, and the present invention is not limited to this. The training images consist of pairs of input training images with known disturbance strengths and their disturbance strengths. Note that the input training image size is 50×50 pixels and 11 frames in accordance with the network structure 600 of the measurement unit 103 . Also, the input training images are numerically generated using the disturbance model B-Spline described above. At that time, the variance of the deformation vector described above is used as the disturbance intensity. Also, input training images containing moving objects are used as training images. This is to demonstrate that the present invention enables robust measurement of disturbance intensity for moving objects, as described above.

The input image uses an image that can be considered to have been acquired under the same conditions (optical system optical conditions, image sensor pixel pitch, and frame rate) as the training input image. Therefore, the frame rate of the input image is not adjusted. The input image size is 400×400 pixels and 40 frames. From there, 11 frames of 50×50 pixels are randomly extracted temporally and spatially at 20 locations, and the average value of the calculated disturbance strength is used as the final disturbance strength. Also, an input image containing a moving object (a car) was used.

Normalization of input images and input training images uses the method given by Equation (5). That is, normalization is performed by generating an average image of multiple input (training) images and subtracting it from each of the multiple input (training) images. All images are monochrome images, and the pixel values are normalized so that they fall within the range of [0 1].

Learning is SGD (see Non-Patent Document 2) using Adam's method as an optimization method. The parameters for Adam's method are α=10 ⁻⁴ , β ₁ =0.9, β ₂ =0.999, and ε=10 ⁻⁸ . Also, SGD is used by randomly selecting 128 images from the total number of 76,800 training images. The number of iterations of learning is 18×10 ⁴ times (300 epochs). For the initial values of network parameters (filters and biases), Xavier (see Non-Patent Document 3) is used in all layers.

Note that the size, the number of frames, and the normalization method of the input image and the input training image to be input to the measurement unit 103 are examples, and the present invention is not limited thereto. Also, the definition of the disturbance intensity output from the measurement unit 103 and the final calculation method of the disturbance intensity are examples, and the present invention is not limited to this.

As shown in FIG. 6, a network structure (neural network) 600 has a main section 601, an input section 602, a transform section 603, and an output section 604. FIG. The main unit 601 converts a plurality of images into first features 611a, 611b, 611c using first network parameters and a first convolutional neural network (CNN) of at least two layers. The input unit 602 converts a plurality of images into second features 612a, 612b, 612c using the second network parameters and the second CNN. The transformation unit 603 adds the first feature amount and the second feature amount to generate third feature amounts 613a, 613b, and 613c, and uses the third network parameter and the third CNN to generate the third feature amount. 3 is converted into a fourth feature quantity 614 . The output unit 604 uses the fourth network parameters and the fully-connected neural network to output the intensity of the disturbance from the fourth feature amount.

FIG. 7 is a diagram showing numerical calculation results (disturbance intensity measurement results) in this embodiment. In FIG. 7, the horizontal axis is the disturbance intensity applied to the input image (true disturbance intensity), and the vertical axis is the disturbance intensity measured from the input image. Note that the error bars in the graph represent the standard deviation of the 20 measured disturbance intensities randomly extracted from the input image. From this, it can be seen that there is a high correlation with the disturbance intensity applied to the input image, and robust disturbance intensity measurement can be performed for moving objects.

Next, Embodiment 2 of the present invention will be described with reference to FIGS. 8 to 10. FIG. In this embodiment, a program describing the functions of the image processing apparatus 100 is used to measure the disturbance intensity of an input image having an unknown disturbance intensity, and then numerical calculation results for correcting the disturbance will be described. Note that the CNN of the measurement unit 103 is the same as that of the first embodiment, so the description thereof will be omitted.

FIG. 8 is a diagram showing a network structure (CNN of correction unit 104) 800 in this embodiment. Since the basic configuration of the network structure 800 in FIG. 8 is the same as the network structure 600 of the measurement unit 103 described in the first embodiment except that there is no fully connected network in the output portion, detailed description thereof will be omitted.

The network structure 800 has a main part 801 , an input part 802 and an output part 803 . A main unit 801 converts a plurality of images into fifth feature amounts 811a, 811b, and 811c using a learned fifth network parameter and a fifth convolutional neural network (CNN) having at least two layers. . The input unit 802 converts a plurality of images into sixth feature amounts 812a, 812b, and 812c using the sixth learned network parameter and the sixth CNN. The output unit 803 adds the fifth feature amount and the sixth feature amount to generate the seventh feature amounts 813a, 813b, and 813c, and outputs the learned seventh network parameter and the seventh CNN. is used to transform the seventh feature quantity into an output image. Note that the network structure 800 shown in FIG. 8 is only an example, and the present invention is not limited to this.

The training images consist of a set of input training images degraded by a known disturbance strength to the output training images. Note that the input training images are numerically generated from the output training images using the disturbance model B-Spline described above. At that time, the variance of the deformation vector described above is used as the disturbance intensity. Also, because the network structure 800 does not have a fully connected network at its output, training images of any size can be used. In this embodiment, as in the first embodiment, the input/output training image size is 50×50 pixels. Also, the number of input training images (the number of frames) can be determined according to the disturbance intensity. In this embodiment, for the sake of simplicity, the number of frames is 11 regardless of the disturbance intensity.

As the input image, an image that can be regarded as having been acquired under the same conditions as the training input image (optical conditions of the optical system, pixel pitch of the image sensor, frame rate) is used. For this reason, no adjustments have been made to the frame rate. The input image size is 400×400 pixels and 80 frames. Also, the output image size is the same as the input image size. Also, all the images are monochrome images, and the pixel values are normalized so that they fall within the range of [0 1].

Learning is SGD using Adam's method as an optimization method, as described above. The parameters for Adam's method are α=10 ⁻⁴ , β ₁ =0.9, β ₂ =0.999, ε=10 ⁻⁸ . SGD is used by randomly selecting 128 training images from a total of 76,800 training images. The number of learning iterations is 18×10 ⁴ times (300 epochs). Initial values of network parameters (filters and biases) use Xavier (see Non-Patent Document 3) in all layers. Note that the size and the number of frames of the input image and the input training image to be input to the measurement unit 103 are examples, and the present invention is not limited to this. Also, the size and the number of frames of the output image and the output training image output from the measurement unit 103 are examples, and the present invention is not limited to this.

FIG. 9 is a diagram qualitatively showing numerical calculation results in this embodiment, showing disturbance correction results. FIG. 9(a) shows one frame of an image (input image) degraded by the disturbance, and FIG. 9(b) shows one corresponding frame of the disturbance-corrected (output image). For ease of understanding, a diagram in which one cross-section of each figure is stacked in the time direction is also shown below each figure. From this, it can be qualitatively understood that the disturbance is appropriately corrected because the fluctuation of the cross-sectional view is moderate.

FIG. 10 is a diagram quantitatively showing the results of numerical calculations in this embodiment, in which the disturbance intensity of an image (input image) degraded by disturbance and an image corrected for disturbance (output image) is measured according to this embodiment. The results are shown. The method described in Example 1 is used for measuring the disturbance intensity. Since the disturbance intensity of the output image is smaller than that of the input image, it can be quantitatively understood that the disturbance is appropriately corrected.

(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

According to each embodiment, it is possible to provide an image processing device, an image processing system, an imaging device, an image processing method, a program, and a storage medium capable of measuring the intensity of disturbance with high precision.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist.

100 image processing device 101 image acquisition unit 102 parameter acquisition unit 103 measurement unit

Claims (15)

an image acquisition unit that acquires a plurality of temporally different images that are degraded by the disturbance;
A parameter acquisition unit that acquires network parameters of a neural network generated by learning using a plurality of image groups obtained based on known disturbance intensities;
generating a plurality of normalized images by subtracting an average image of the plurality of images from each of the plurality of images ; and a measurement unit that measures the intensity of the disturbance from the image of the image processing apparatus. 2. The image processing apparatus according to claim 1, wherein the intensity of the disturbance is temporal or spatial dispersion of pixel values. The plurality of images are images acquired by photographing using an optical system and an imaging device,
3. The image processing apparatus according to claim 1 , wherein the parameter acquisition unit selects the network parameters to be acquired based on shooting conditions in the shooting . 4. The image processing apparatus according to claim 3, wherein the photographing condition is an optical condition of the optical system, a pixel pitch of the imaging device, or a frame rate. The measurement unit generates a plurality of images whose sizes are adjusted based on the frame rate and the learning conditions of the neural network, and measures the intensity of the disturbance from the adjusted images. 5. The image processing apparatus according to claim 4. The neural network is
a main unit that transforms the plurality of images into a first feature using a first convolutional neural network having first network parameters ;
an input unit that transforms the plurality of images into a second feature using a second convolutional neural network having second network parameters ;
A third feature is generated by adding the first feature and the second feature, and the third feature is obtained using a third convolutional neural network having a third network parameter. a conversion unit that converts the quantity into a fourth feature quantity;
an output unit that outputs the intensity of the disturbance from the fourth feature using a fully connected network having a fourth network parameter ;
6. The image processing apparatus according to claim 1, wherein said network parameters include said first to fourth network parameters . 7. The image processing apparatus according to any one of claims 1 to 6 , wherein the measurement unit measures the intensity of the disturbance at a plurality of locations in the plurality of images. 3. The intensity of the known disturbance is a normal random number variance of a deformation amount given by random numbers to the control points in the image , and is generated using a B-Spline-based disturbance model. 8. The image processing device according to any one of 7 . 9. The image processing apparatus according to any one of claims 1 to 8 , further comprising a correction unit that corrects the plurality of images based on the intensity of the disturbance. 9. The image processing apparatus according to any one of claims 1 to 8, further comprising a correction unit that selects network parameters for correcting the plurality of images based on the intensity of the disturbance. An image processing system comprising the image processing device according to any one of claims 1 to 10 and a client device connected to the image processing device via a network,
The client device has an image output unit that outputs a plurality of temporally different images degraded by the disturbance to the image processing device,
The image processing system , wherein the image processing device further includes a disturbance intensity output unit that outputs the intensity of the disturbance to the client device . an imaging device;
An imaging apparatus comprising the image processing apparatus according to any one of claims 1 to 10 . an image acquisition step of acquiring a plurality of temporally different images degraded by the disturbance;
A parameter acquisition step of acquiring network parameters of a neural network generated by learning using a plurality of image groups obtained based on known disturbance intensities;
generating a plurality of normalized images by subtracting an average image of the plurality of images from each of the plurality of images ; and a measuring step of measuring the intensity of the disturbance from the image of the image. A program that causes a computer to execute the image processing method according to claim 13 . 15. A storage medium storing the program according to claim 14 .

Images