JP6886887B2 - Error calculator and its program - Google Patents

Description

The present invention relates to an error computer and its program, and more particularly to an error computer and its program for calculating errors in learning a neural network.

Conventionally, a plurality of automatic coloring techniques for monochrome images converted into digital data have been developed. Monochrome images converted to digital data have few clues for color information, which is an image feature amount. Therefore, it is said that the difficulty level is higher than that of colorizing an analog image recorded on a physical medium such as a film. For example, a method of converting monochrome data into color data is known (see Patent Document 1). This method assumes a specific object recorded in monochrome data and calculates a color distribution model from this specific object. Then, the color information is estimated from the calculated color distribution model. In this method, it is assumed that the object to be colorized is a specific object, so if the assumed object and the object in the monochrome image are different, it is difficult to convert the monochrome image into a natural color image. ..

In order to solve such a problem, in recent years, a plurality of methods for estimating color information that make the object to be colorized more general by using so-called machine learning technology have been proposed (Non-Patent Document 1 and Non-Patent Document 2). reference). The color information estimation method using these machine learning techniques is premised on preparing a huge amount of color images showing various objects. Then, machine learning is performed based on such an enormous amount of color images to generate a color information estimator. At this time, for example, a huge amount of color images prepared in advance are input to a machine learning device composed of a so-called neural network or the like, and the correspondence between the monochrome image and the color information is learned. With the color information estimator generated in this way, color information can be estimated accurately from various monochrome images. This makes it possible to convert a monochrome image converted into digital data into a natural color image.

Japanese Unexamined Patent Publication No. 2016-146529

Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa., "Let there be Color !: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification," ACM Transaction on Graphics (Proc. Of) SIGGRAPH), 35 (4): 110, 2016. Richard Zhang, Phillip Isola, and Alexei A. Efros. “Colorful Image Colorization.” In ECCV 2016.

However, in the prior art, an error based on a value obtained by independently comparing the estimated color information (for example, image feature amount) and the true color information (for example, image feature amount) for each pixel. To calculate. Therefore, for example, with respect to color information, learning proceeds only in a direction in which the error for each pixel becomes smaller, and color uniformity between a certain pixel and its adjacent pixels is not considered at all. Therefore, in the estimated color information, the region that should be estimated to be the same color is likely to be spotted. For example, a landscape in the sky without clouds should generally be uniformly blue, but the prior art has the problem of causing unnatural color unevenness that is incorrect to the human eye.

The present invention has been made in view of the above problems, and is more correct from the human eye with respect to the error between the estimated value and the true value calculated when learning the parameters of the neural network. It is an object of the present invention to provide an error calculator capable of calculating an error that can output an image and a program thereof.

In order to solve the above problems, the error calculator according to the present invention is an error calculator that calculates the error between the first image feature amount which is the estimated color information and the second image feature amount which is the true color information. Therefore, it was decided to provide a feature amount map creating means, a first error calculating means, and an error synthesizing means.

According to such a configuration, the error calculator uses the feature amount map creating means to characterize the relationship of a plurality of pixels in the image from the first image feature amount and the second image feature amount by a predetermined calculation. The quantity is extracted to create a first feature map and a second feature map, respectively.
Then, the error calculator calculates the error between the feature amount maps based on the error of the pixel value between the pixels corresponding to the first feature amount map and the second feature amount map by the first error calculation means. To do.
Then, the error calculator is an error between the image feature amounts calculated based on the error of the pixel value between the pixels corresponding to the first image feature amount and the second image feature amount by the error synthesizing means. Is input, and the error between the image feature quantities and the error between the feature quantity maps are added to generate a composite error.

The present invention can also be realized by an error calculation program for causing the computer to function as the error calculator.

The present invention has the following excellent effects.
According to the error computer according to the present invention, an error between two feature maps reflecting the relationship between a plurality of pixels is calculated, and each pixel is independently compared and obtained based on the calculated value. The combined error can be obtained by adding the error between the feature maps to the error.
Therefore, if the learner learns the parameters of the neural network so as to minimize this synthesis error, it is possible to learn the relationship between a certain pixel and its adjacent pixel. Therefore, the image based on the image feature amount estimated by the learner that has performed the learning using this synthesis error becomes a more correct image when viewed from the human eye.
Therefore, by using the error computer according to the present invention for learning in a neural network that outputs color information, it is possible to generate a color image in which the occurrence of unnatural color unevenness is reduced.

It is a block diagram which shows typically the learning apparatus which includes the error calculator which concerns on embodiment of this invention. It is explanatory drawing which shows typically the creation of the feature amount map by the error computer which concerns on embodiment of this invention, (a) shows an example of an input image, (b) and (c) show an example of a filter. There is. It is explanatory drawing which shows typically the method of making the feature amount map of the same size as an input image, (a) shows an example of an input image, and (b) shows an extended example of an input image. It is a block diagram which shows typically the automatic coloring apparatus which includes the color information magnifier which uses the error calculator which concerns on embodiment of this invention for learning. It is a block diagram which shows typically the learning flow of the low-resolution color information estimator shown in FIG. It is a block diagram which shows typically the learning flow of the color information magnifier shown in FIG. It is a block diagram which shows typically an example of the color information magnifier shown in FIG. It is explanatory drawing which shows another example of the color information magnifier shown in FIG. 4 schematically.

Hereinafter, the error computer according to the embodiment of the present invention will be described with reference to the drawings.
The learning device S shown in FIG. 1 includes a learning device 60 and an error calculator 40.
The learning device S calculated the error between the estimated color information of the first image feature amount 401 and the true color information of the second image feature amount 402 by the error calculator 40, and the error calculator 40 calculated the error. The learner 60 learns the parameters for constructing the neural network so as to minimize the error.
Here, the learner 60 is composed of a neural network that outputs an image feature amount described in detail below. In the following description, the image feature amount means, for example, an amount representing a color space such as brightness, chromaticity, and saturation, and for example, an average value, a dispersion, and a convolution integral value extracted from the amount representing the color space. Etc. are included. Further, the set of image feature amounts for each pixel means, for example, a monochrome image (monochrome information) or color information. Further, the image feature amount may be handled by a matrix in which elements are arranged in the height direction and the width direction (vertical and horizontal), or may be handled by a one-dimensional multivariable vector.

Further, here, the monochrome image specifically refers to an image composed of only the luminance channels in the color space (V channel in the HSV color space, L channel in the Lab color space, etc.). When the pixel information is luminance, the pixel value (luminance value) has a value of 0 to 255 when represented by 8-bit information. The monochrome information, which is the image feature amount of this monochrome image, is represented by, for example, a luminance distribution. In this specification, this monochrome information is used in the same meaning as a monochrome image. Further, the color information can be, for example, an image feature amount for two channels other than the luminance channel.

In the following examples, the learner 60 will be described as being configured by a neural network that outputs color information as the image feature amount as an example. The learner 60 is used to create, for example, a color information estimator that estimates color information from a monochrome image and a color information magnifier that estimates high-resolution color information from low-resolution color information. The learner 60 has parameters (parameter group) for constructing a neural network inside, and outputs an estimated value according to the internal parameters to the input as the learning data. The learner 60 adjusts the output value by changing this internal parameter.

The error calculator 40 calculates, for example, the error between the estimated color information for learning in which the learner 60 estimates the color information by the neural network from the color information (image feature amount) and the true color information. As shown in FIG. 1, the error calculator 40 includes a feature amount map creating means 41, a first error calculating means 42, and an error synthesizing means 43.
The error calculator 40 extracts a feature amount that characterizes the relationship between a plurality of pixels in the image from the first image feature amount 401 and the second image feature amount 402 by a predetermined calculation, and uses the first feature amount map 403 and the first feature amount map 403. A second feature map 404 is created, an error between the feature maps based on a pixel value error between the pixels corresponding to the first feature map 403 and the second feature map 404 is calculated, and a second feature map is calculated. The error between the image feature amounts calculated based on the error of the pixel value between the pixels corresponding to the image feature amount 401 of 1 and the image feature amount 402 of the second image feature amount 402 is input, and the error between the image feature amounts and the error between the image feature amounts are obtained. The error between the feature maps and the error are added to generate a composite error.
The error calculator 40 includes a second error calculating means 51 and a minimizing means 52 as a configuration similar to that of the conventional error calculator. The error calculator 40 is composed of, for example, a general computer, and includes a computing device such as a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and an HDD (Hard Disk Drive). It has an input / output interface.

(Second error calculation means 51)
The second error calculating means 51 calculates an error between these two image feature amounts by using the first image feature amount 401 and the second image feature amount 402. Here, the first image feature amount 401 is estimated color information that the learner 60 outputs according to an internal parameter with respect to the input value input as the training data. The first image feature amount 401 is, for example, a high-resolution image feature estimated by the learner 60 from low-resolution color information (image feature amount) input to the learner 60 as training data for learning. Color information (estimated color information).

Further, the second image feature amount 402 is, for example, high-resolution color information (true color information) of the correct answer prepared as training data of the learning device 60.
The second error calculating means 51 calculates an error between color information (image feature amount) based on the error of the pixel value between pixels corresponding to the first image feature amount 401 and the second image feature amount 402. It is output to the error synthesis means 43.

As the error between the color information, for example, the mean square error similar to the conventional method or the cross entropy is used. As a specific example, the loss function (Loss) defined by the following equation (1) represents the mean square error similar to the conventional method.

Here, H is the number of pixels in the vertical direction of the image to be input as color information, and W is the number of pixels in the horizontal direction of the image to be input as color information. C is the number of estimated color information channels, and usually C = 2.
y _{h, w, c} are estimated color information in the channel c of the pixels located at the coordinates (w, h) on the image.
y ^gt _{h, w, c} are the true color information values in the channel c of the pixels located at the coordinates (w, h) on the image. In general, H × W × C in the above equation (1) is replaced with N, and h, w, and c are collectively replaced with i, which is often expressed as the following equation (2).

(Feature quantity map creation means 41)
The feature amount map creating means 41 extracts a feature amount that characterizes the relationship between a plurality of pixels in the image from the first image feature amount 401 and the second image feature amount 402, which are color information, by a predetermined calculation. The feature amount map 403 of 1 and the feature amount map 404 of 2 are created respectively. Here, the feature amount that characterizes the relationship between a plurality of pixels is a feature that characterizes the relationship between a plurality of pixels in color information (image feature amount), and is not a feature that is independently obtained from a single pixel in color information. .. Further, the pixel relationship is represented by, for example, the difference between two pixel values to be the target of the relationship. The pixel of interest and its peripheral pixels may be separated or adjacent to each other.

Spatial filtering may be used to extract features that characterize the relationship between a plurality of adjacent pixels. In the present embodiment, the feature amount map creating means 41 performs an operation of performing a filter process on the first image feature amount 401 or the second image feature amount 402 which is color information by using a predetermined filter. Then, the feature amount that characterizes the relationship between the plurality of adjacent pixels is extracted to create the first feature amount map 403 or the second feature amount map 404.

Here, the filter is not particularly limited in purpose and function as long as it can be used for spatial filtering, and for example, a contour extraction filter, a noise removal filter, a smoothing filter, a moving average filter, a non-linear filter such as a median filter, or the like. May be used.

In the present embodiment, as an example, the feature amount map creating means 41 uses an edge filter that detects an edge included in an image based on the first image feature amount 401 or the second image feature amount 402 which is color information. The edge map will be described as being created as the first feature map 403 or the second feature map 404. The edge map is a map (gradient map) in which the relationship with peripheral pixels is represented by a gradient of pixel values.

As the edge filter, for example, a first-order differential filter such as a Sobel filter or a Prewitt filter can be used. Further, the present invention is not limited to the first-order differential filter, and a second-order differential filter such as a Laplacian filter may be used.
Further, for example, a filter having a size of 3 × 3 in consideration of 8 peripheral pixels may be used, or a filter having a size of 5 × 5 in consideration of 24 peripheral pixels may be used.
Further, the shape of the filter is not limited to a square shape, and may be, for example, a shape that considers four peripheral pixels adjacent to each other in the vertical and horizontal directions.
In the following, as an example, a gradient map (edge map) will be created using a Sobel filter.

The feature amount map creating means 41 creates a first feature amount map 403 (gradient map) from the first image feature amount 401 (estimated color information), and outputs the first feature amount map 403 (gradient map) to the first error calculating means 42.
The feature amount map creating means 41 creates a second feature amount map 404 (gradient map) from the second image feature amount 402 (true color information), and outputs the second feature amount map 404 (gradient map) to the first error calculating means 42.
The first feature amount map 403 is, for example, a gradient map for estimated high-resolution color information (hereinafter, referred to as a gradient map for estimated color information).
The second feature map 404 is, for example, a gradient map for high-resolution color information of the prepared correct answer (hereinafter, referred to as a gradient map for true color information).
The details of creating the gradient map by the feature map creating means 41 will be described later.

(First error calculation means 42)
The first error calculating means 42 calculates the error between the gradient maps (between the feature maps) based on the error of the pixel value between the pixels corresponding to the first feature map 403 and the second feature map 404. It is a thing. The first error calculating means 42 calculates the error between the gradient maps using the gradient map for the estimated color information and the gradient map for the true color information, and outputs the error to the error combining means 43. Although the input information of the first error calculation means 42 is different from that of the second error calculation means 51, the same method as that of the second error calculation means 51 can be applied to the error calculation method.

(Error synthesis means 43)
The error synthesizing means 43 includes an error between the color information (between the image feature amounts) between the first image feature amount 401 and the second image feature amount 402, and the first feature amount map 403 and the second feature amount map. The error between the gradient maps (between the feature maps) and the 404 is added to generate a composite error 405.
The error synthesizing means 43 receives an error between the estimated color information and the true color information from the second error calculating means 51.
The error synthesizing means 43 receives an error between the gradient map for the estimated color information and the gradient map for the true color information from the first error calculating means 42.
The error synthesizing means 43 calculates the sum of the error between the color information acquired from the second error calculating means 51 and the error between the gradient maps acquired from the first error calculating means 42 as the synthesizing error 405. The error combining means 43 outputs the calculated synthesis error 405 to the minimizing means 52.

(Minimizing means 52)
The minimizing means 52 sequentially inputs a set of the first image feature amount 401 and the second image feature amount 402, and the compositing error 405 is reduced by a predetermined calculation according to the set of the input image feature amounts. The parameters of the learner 60 are adjusted, and the adjusted parameters are supplied to the learner 60 as update parameters 406 (update parameters).

The minimizing means 52 adjusts the parameters of the learner 60 so that the synthesis error 405 is reduced by using an optimization method based on an error gradient such as SGD. Since SGD is described in the following references, the description thereof will be omitted.
(Reference) L. Bottou., "Stochastic Gradient Descent Tricks.," Neural Networks: Tricks of the Trade: Springer, 2012.

Here, the problem that unnatural color unevenness occurs in the prior art will be described.
For example, in a cloudless sky landscape image (color-estimated image), a row of pixel regions consisting of 10 pixels is assumed. Then, the correct pixel value is set to, for example, "B", the estimated pixel value for each of the first to fifth pixels in this row is "B + 10", and the estimated pixel value for each of the sixth to tenth pixels is "B". Is "B-10".
At this time, the conventional error calculator performs an error calculation for comparing each pixel independently. In this example, since the difference (absolute value) from the correct pixel value is all "10" for the pixel values in this row, the conventional error computer may determine that the error is minimized. is there. Therefore, using the estimator created by such learning, the colors estimated for the 1st to 5th pixels in this row and the 6th to 10th pixels in this row can be obtained. On the other hand, the estimated color will be different.
On the other hand, the error calculator 40 of the present embodiment also calculates the difference between the adjacent pixels as the relationship between the plurality of pixels, so that the fifth pixel value and the sixth pixel value in this row are calculated. It can be expected that it is detected that there is a large gap of 20 between and, and it is determined that the error is larger than when all the pixel values in this row are "B + 10".
That is, in the prior art, the error calculator 40 uses the color of the pixel of interest and its peripheral pixels to display an unnatural color unevenness such as a mottled landscape in the sky without clouds that should be uniformly blue. It is possible to calculate an error that can output an image in which the uniformity of the image is preserved, that is, an image that is more correct from the human eye.

Hereinafter, the description of the error computer 40 will be continued using a mathematical formula.
In the present embodiment, the minimization means 52 updates the parameters of the learner 60 by using, for example, the loss function (Loss) defined by the following equation (3).

As the error used for learning (composite error 405), the error calculator 40 uses an error calculated from a plurality of pixels in addition to the mean square error and cross entropy similar to those in the conventional method. Equation (3) shows an example in which the mean square error similar to the conventional method and the mean square error in the gradient map as the error calculated from a plurality of pixels are used.

Specifically, the first term of the equation (3) represents the mean square error similar to that of the conventional method. The first term of this equation (3) corresponds to the processing of the second error calculating means 51 described above. In the conventional method, the parameters of the learner are updated so as to minimize the first term of the equation (3).

On the other hand, the second term of the equation (3) represents the mean square error in the gradient map created by the feature map creating means 41 as an error calculated from a plurality of pixels. The second term of this equation (3) corresponds to the processing of the first error calculating means 42 described above. Further, the sum of the first term and the second term of the equation (3) corresponds to the processing of the error synthesis means 43 described above.

In the second term of the equation (3), α is a weighting coefficient of the error calculated from a plurality of pixels. Although it depends on what kind of learning device is made, it is desirable that α is a small value of about 0.1.
H ^grad is the vertical size (number of pixels) of ^{the gradient map, and W grad} is the horizontal size (number of pixels) of the gradient map.
K represents the number of filters used to calculate the gradient map.

Further, in the second term of the equation (3), g _{h, w, c, k} are values in the channel c of the pixels located at the coordinates (w, h) on the gradient map created from the estimated color information (hereinafter, , The value of the gradient map for the estimated color information).
g ^gt _{h, w, c, k} are the values in the channel c of the pixels located at the coordinates (w, h) on the gradient map created from the true color information (hereinafter, the gradient map for the true color information). It is called a value).
Of these, the value of the gradient map for the estimated color information is calculated by the feature map creating means 41, for example, based on the following equation (4).

In equation (4), M _k is the vertical size of _{the k-th filter, and N k} is the horizontal size of the k-th filter.
s (1 ≦ s ≦ M _k ) is an identifier of each coefficient arranged in the vertical direction of the k-th filter, and the upper end of the column is represented by 1.
t (1 ≦ t ≦ N _k ) is an identifier of each coefficient arranged in the horizontal direction of the kth filter, and the left end of the row is represented by 1.
ω _{s, t, k} are filter coefficients, which are the coefficients arranged in the s row and t column of the kth filter.

Similarly, the value of the gradient map for the true color information is calculated by the feature map creating means 41, for example, based on the following equation (5).

These equations (4) and (5) correspond to the processing of the feature amount map creating means 41.
When the feature map creating means 41 uses a Sobel filter that detects vertical edges and horizontal edges to create a gradient map, the following conditions are set.
(conditions)
Number of filters K = 2
k = Vertical size for the first filter M _{k = 1} = 3
k = Horizontal size for the first filter N _{k = 1} = 3
k = Vertical size for the second filter M _{k = 2} = 3
k = Horizontal size for the second filter N _{k = 2} = 3
When k = the first filter is a filter that detects vertical edges, that is, when it detects a horizontal difference (gradient), the filter coefficient ω _{k = 1} is represented by FIG. 2 (b). To.
If the k = second filter is a filter that detects horizontal edges, that is, if it detects a vertical difference (gradient), the filter coefficient ω _{k = 2} is represented by FIG. 2 (c). To.

The feature amount map creating means 41 illustrates, for example, when there is a pixel value as illustrated in FIG. 2A as the first image feature amount 401 which is the estimated color information to be input, FIG. 2B is illustrated. A 3x3 area is scanned using a filter that detects vertical edges. For example, when the pixel in the second row and the second column is the pixel of interest in FIG. 2A, the pixel value in the region of 3 × 3 centered on the pixel of interest and the filter coefficient ω _{k = 1} are convolved. ) Is 20.
Similarly, the calculation result when the pixels in the second row and the third column are the pixels of interest is 20. The value of this filter (edge filter) increases as there are edges.
On the other hand, when this filter is applied to an area where all the pixel values of the 3 × 3 area are the same, the value becomes 0. That is, the calculation results when the pixels in the third row and the second column and the pixels in the third row and the third column are the pixels of interest are both 0.

Similarly, when the feature amount map creating means 41 has a pixel value as illustrated in FIG. 2A as the first image feature amount 401 which is the estimated color information to be input, FIG. 2C A 3 × 3 region is scanned using a filter (filter coefficient ω _{k = 2) that detects horizontal edges as illustrated in.} The feature map creating means 41 adds the calculation result obtained at this time and the calculation result when the filter for detecting the vertical edge (filter coefficient ω _{k = 1} ) is used to create an edge map. Create (gradient map).

By applying the same error calculation as the second error calculating means 51 to the gradient map created in this way, the first error calculating means 42 can calculate the error between the gradient maps. The first error calculation means 42 and the second error calculation means 51 may use other error functions such as cross entropy instead of the mean square error in the error calculation formula. Further, in the calculation of the second term of the formula (3) performed by the second error calculating means 51, it is not necessary to be exactly the same formula as the second term of the formula (3), and between the values related to a plurality of pixels. It does not matter as long as it performs a predetermined calculation with.

Further, the size of the edge map (H ^grad × W ^grad ) created in the specific example described above is smaller than the size of the input image (H × W) by two pixels in the vertical and horizontal directions. To explain the filter size in general, the size of the edge map is H ^grad = H − (M _k -1) and W ^grad = W − (N _k -1). However, the size of the edge map does not necessarily have to be smaller than the size of the input image, and an edge map having the same size as the input image may be created as follows.

For example, by temporarily _{expanding the size of the input image to the size of (H + (M k} -1)) × (W + (N _k -1)), an edge map having the same size as the input image can be obtained. .. In order to expand to the above size, the pixels to be interpolated may be generated while being arranged along the outer circumference of the original image. Here, as a method of determining the value of the pixel temporarily generated for size expansion, for example, a method of giving an average pixel value of the original image to all the generated pixels or a method of generating is generated. A method of assigning the value of the pixel of the original image closest to each pixel can be adopted.

Of these, when the method of assigning the average pixel value is used, for a 3 × 3 image as shown in FIG. 3A, for all the pixels generated so as to fill the periphery of the original image. , The pixel value "45" may be given.
Further, when the method of assigning the value of the closest pixel is used, for example, for a 3 × 3 image as shown in FIG. 3A, each pixel generated so as to fill the periphery of the original image is used. 3 (b) may be given the values of the pixels shown by hatching.

The error computer 40 according to the present embodiment is used for learning performed by using an error obtained from a plurality of pixels in addition to an error obtained by independently comparing each pixel, which has been conventionally used, as an error used for optimization. Used. As a result, the learning device S can perform learning in consideration of the relationship between pixels.
Therefore, by using the error calculator 40 for learning in a neural network that outputs color information, for example, the learning device S looks like a gradient of pixel values between the color information of the training data and the estimated color information. It is possible to learn so that the relationship with the peripheral pixels is the same. Thereby, for example, a highly accurate color estimator and a color information magnifier can be created from the learning device 60 of the learning device S. As a result, the color estimator and the color information magnifier can generate a color image in which the occurrence of unnatural color unevenness is reduced.
With the above technique, even a high-resolution monochrome image such as a 4K / 8K image can be naturally colored with less color unevenness.

Although the embodiments of the present invention have been described above, the present invention is not limited to this, and can be carried out without changing the gist thereof. For example, although described as the error calculator 40, it can also be regarded as an error calculator written in a general-purpose or special computer language so as to enable processing of the configuration of this device.

Further, in the above-described embodiment, the error calculator 40 is provided with the second error calculating means 51 and the minimizing means 52 as the same configuration as the conventional error calculator, but the second error The calculation means 51 and the minimization means 52 may be separate from the error calculator. In this case, for example, the second error calculating means 51 may be provided in the front stage of the error calculator, or the minimizing means 52 may be provided in the rear stage of the error calculator.

Further, the learning device S may configure the color information estimation learning device by the error calculator 40 and, for example, the learning device 60 prepared for creating the estimator for estimating the color information.
Further, the learning device S may configure the color information expansion estimation learning device by the error calculator 40 and, for example, the learning device 60 prepared for creating the estimation device that expands the color information.
Further, the learning device S is provided with a super-resolution image by an error calculator 40 and a learning device 60 prepared for creating an estimator for estimating a high-resolution image such as a super-resolution image from a low-resolution image, for example. An estimation learning device may be configured.

Hereinafter, the color information magnifier that uses the error computer 40 according to the embodiment of the present invention for learning will be described in detail.
The automatic coloring device 1 shown in FIG. 4 automatically colors a monochrome image by estimating color information from the monochrome image, and includes a color information magnifier 10. As shown in FIG. 4, the automatic coloring device 1 mainly includes a color information estimator 3 and an information synthesizer 9.
The automatic coloring device 1 is composed of, for example, a general computer, and includes an arithmetic unit such as a GPU (Graphics Processing Units), a ROM, a RAM, an HDD, a general image memory, and an input / output interface. ..

The color information estimator 3 generates a low-resolution monochrome image 103 and a low-resolution color information 105 from the input high-resolution monochrome image 101, and estimates the high-resolution color information 107 using these information. ..
The high-resolution monochrome image 101 is a first-resolution monochrome image. The high-resolution monochrome image 101 is, for example, a monochrome image digitized by scanning from a past black-and-white film or photograph.
The low-resolution monochrome image 103 is a monochrome image having a second resolution lower than the first resolution.
The low resolution color information 105 is the color information of the second resolution.
The high-resolution color information 107 is the color information of the first resolution.

The value of the first resolution (value of high resolution) is not particularly limited as long as it is larger than the value of the second resolution (value of low resolution). For example, the size of the image of the second resolution may be 256 × 256 pixels, and the size of the image of the first resolution may be 512 × 512 pixels. Further, for example, the size of the image of the second resolution may be 480 × 270 pixels, and the size of the image of the first resolution may be 4K (3840 × 2160). Further, the size of the first resolution image may be 8K (7680 × 4320).

As shown in FIG. 4, the color information estimator 3 includes a reducer 5, a low-resolution color information estimator 7, and a color information magnifier 10.

The reducer 5 generates a low-resolution monochrome image 103 by performing a process of reducing the input high-resolution monochrome image 101. Here, reduction means reducing the resolution, that is, reducing the number of pixels. When the reduction ratio in the reduction is, for example, 0.5, the number of pixels in the horizontal direction and the vertical direction of the reduced image is 1/2 of the number of pixels in the horizontal direction and the vertical direction of the original image, respectively. The reducer 5 outputs the generated low-resolution monochrome image 103 to the low-resolution color information estimator 7.

The low-resolution color information estimator 7 uses low-resolution color information (image feature amount) from the low-resolution monochrome image 103 generated by the reducer 5 by using a parameter group predetermined by learning for estimation. To extract. As a result, the low-resolution color information estimator 7 estimates the low-resolution color information 105. The learning flow for creating the low-resolution color information estimator 7 is the same as that of the conventional technique, but a brief description will be described later.

The color information magnifier 10 expands the image size by inputting the low-resolution color information 105 estimated by the low-resolution color information estimator 7 and the high-resolution monochrome image 101 input by bypassing the reducer 5. The process of estimating the color information (high-resolution color information 107) is performed. The color information magnifier 10 enlarges the low-resolution color information 105 by using the high-resolution monochrome image 101 (monochrome information). Then, the color information magnifier 10 outputs the estimated high-resolution color information 107 to the information synthesizer 9.

The information synthesizer 9 synthesizes the high-resolution color information 107 estimated by the color information estimator 3 and the high-resolution monochrome image 101 to create a high-resolution color image 109. The information synthesizer 9 simply combines monochrome information of one channel (hereinafter, may be referred to as 1ch) and color information of two channels (2ch) to generate a color image.

Next, the learning flow of the color information magnifier 10 will be described in comparison with the learning flow of the low-resolution color information estimator 7.

First, the learning flow of the low-resolution color information estimator 7 will be described with reference to FIG.
The low-resolution color information estimator 7 is generated from a learning device prepared in advance by the following procedure. This learner inputs a monochrome image and estimates and outputs color information by performing a predetermined calculation process. In FIG. 5, the learner is represented as a low-resolution color information estimator 7 in a state where learning has been completed. Then, a large amount of color images for learning are prepared, and the following steps S1 to S4 are repeated a sufficient number of times. This learner learns this parameter, and by setting the parameter appropriately, an accurate color information estimator can be created.

(Step S1)
A low-resolution color image 202 is prepared as a color image for learning, and the low-resolution monochrome image 203 and the true color information 204 are separated.
Here, the low-resolution monochrome image 203 is a low-resolution learning monochrome image.
Further, the true color information 204 is correct color information having the same size as the low-resolution learning monochrome image, and is used for error calculation with the estimated color information.

(Step S2)
Next, the learner (low-resolution color information estimator 7) inputs the low-resolution monochrome image 203, and outputs the low-resolution color information 205 as the color information of the estimation result using the current parameters.

(Step S3)
Next, the conventional error calculator 8 calculates the error between the low resolution color information 205 (estimated color information) and the true color information 204. As this error, the mean square error of each pixel value or the like is used.

(Step S4)
Further, the conventional error computer 8 uses a learning device (low resolution color information estimator 7) so that the error is reduced by using an optimization method based on an error gradient such as SGD from the calculated error. ) Is adjusted, and the adjusted parameter is output to the learner.
The conventional error calculator 8 is composed of, for example, the second error calculating means 51 and the minimizing means 52 shown in FIG.

Next, the learning flow of the color information magnifier 10 will be described with reference to FIG.
The color information magnifier 10 is generated from a learning device prepared in advance by the following procedure. This learner inputs the high-resolution monochrome image 301 and the low-resolution color information 305, and estimates and outputs the high-resolution color information 307 by performing a predetermined calculation process. In FIG. 6, the learning device is represented as a color information magnifier 10 in a state where learning has been completed. Then, a large amount of color images for learning are prepared, and the following steps S10 to S14 are repeated a sufficient number of times. This learner learns this parameter, and by setting the parameter appropriately, an accurate color information magnifier can be created.

(Step S10)
A high-resolution color image 309 is prepared as a color image for learning, and the high-resolution color image 309 is simply reduced by the reducer 5 to obtain low-resolution color information 305.
Here, as the high-resolution color image 309, a colorized version of an old black-and-white film is also used. In this case, for example, digital data obtained by manually coloring a monochrome image digitized by scanning from a past black-and-white film or photograph is used. Further, in order to prepare a large amount of high-resolution color images 309 for learning, a new color image such as 4K taken in color may be used in addition to the old black-and-white film.

(Step S11)
Next, the high-resolution color image 309 is separated into a high-resolution monochrome image 301 and a high-resolution color information (true color information) 304.
Here, the high-resolution monochrome image 301 is a high-resolution learning monochrome image.
Further, the high-resolution color information 304 is correct color information having the same size as the high-resolution learning monochrome image, and is used for error calculation with the estimated high-resolution color information.

(Step S12)
Next, the learning device (color information magnifier 10) inputs the high-resolution monochrome image 301, and outputs the high-resolution color information 307 as the color information of the estimation result using the current parameters.

(Step S13)
Next, the error calculator 40 according to the embodiment of the present invention calculates an error (composite error) between the high-resolution color information 307 (estimated color information) and the high-resolution color information (true color information) 304. As this error, a combined error 405 (see FIG. 1), which is a combination of the above-mentioned error between color information and the error between gradient maps, is used.

(Step S14)
Further, the error calculator 40 uses a learning device (color information expander) so that the error is reduced from the calculated error (composite error 405) by using an optimization method based on an error gradient such as SGD. The parameter of 10) is adjusted, and the adjusted parameter is output to the learner. The error computer 40 is added at the time of learning, but is disconnected after learning.

Next, the detailed configuration of the color information magnifier 10 will be described with reference to FIG. 7.
As shown in FIG. 7, the color information magnifier 10 includes a size enlarging means 21, a synthesizing means 22a, and a high-resolution color information estimating means 23. Although the color information magnifier 10 of FIG. 7 is shown in a form including the feature extraction means 31, 32, 33, for example, all the feature extraction means may be omitted. In the following, the feature extraction means may be referred to as the first feature extraction means 31, the second feature extraction means 32, and the third feature extraction means 33 for convenience.

The color information magnifier 10 can be configured by, for example, a neural network. Further, the neural network may be, for example, a CNN (Convolutional Neural Network). In CNN, a Convolution layer (convolutional layer) or a Deconvolutional layer (deconvolutional layer or Transposed Convolutional layer) is used as a hidden layer. Therefore, when CNN is adopted, each component can be mounted by the color information magnifier 10 by using the Convolution layer or the Deconvolution layer, and can be calculated at high speed by using the GPU.

The size enlargement means 21 generates a high-resolution image feature amount by performing a process of enlarging the input low-resolution image feature amount. Here, the low-resolution image feature amount means, for example, low-resolution color information 105. As shown in FIG. 7, when the color information magnifier 10 includes the second feature extraction means 32, the image feature amount extracted from the low resolution color information 105 by the second feature extraction means 32 has a low resolution. It becomes the image feature amount of. The size enlargement means 21 outputs the generated high-resolution image feature amount to the synthesis means 22a.

As the size enlargement means 21, for example, a Deconvolution layer (an image enlargement layer using a neural network) may be used. Further, the parameters used in a general image enlargement algorithm may be fixedly used. As a general image enlargement algorithm, for example, the nearest neighbor interpolation method or the Bilinear interpolation method may be used.

The synthesizing means 22a synthesizes, for example, the input high-resolution monochrome image 101 and the high-resolution image feature amount generated by the size expanding means 21. As shown in FIG. 7, when the color information magnifier 10 includes the first feature extracting means 31, the compositing means 22a includes an image feature amount extracted from the high-resolution monochrome image 101 and a size expanding means. The high-resolution image feature amount generated by 21 is combined. The synthesizing means 22a outputs the synthesized high-resolution image feature amount to the high-resolution color information estimating means 23.
The synthesizing means 22a simply synthesizes 1ch monochrome information and 2ch color information having the same size as the monochrome information to generate a high-resolution image feature amount. For the synthesis means 22a, for example, the Convolution layer of the neural network may be used.

The high-resolution color information estimation means 23 extracts an image feature amount from a high-resolution image feature amount synthesized by the synthesis means 22a using a predetermined parameter group by learning for estimating high-resolution color information. However, the high-resolution color information 107 is estimated. Here, the learning means learning for creating the color information magnifier 10.

The high-resolution color information 107 corresponds to the enlarged color information of the low-resolution color information 105, and has a resolution corresponding to the high-resolution monochrome image 101. The high-resolution color information 107 is color information for each channel in the color space, and refers, for example, an image feature amount for two channels other than the luminance channel.

The high-resolution color information estimation means 23 uses image features for a plurality of (3 or more) output channels corresponding to a plurality of (3 or more) outputs from the previous stage, and an image for two channels in the color space. Convert to feature quantity and estimate color information.
For the high-resolution color information estimation means 23, for example, the Convolution layer of the neural network may be used. In addition, there may be a plurality of Convolution layers (hidden layers). That is, the Convolution may be repeated continuously.
The number of output channels from the previous stage of the high-resolution color information estimation means 23 can be set to a desired value. For example, the number of output channels from the synthesis means 22a may be 3 channels or more.

As shown in FIG. 7, the color information magnifier 10 includes at least one feature extraction means of the first feature extraction means 31, the second feature extraction means 32, and the third feature extraction means 33. May be good.

The first feature extraction means 31 extracts a high-resolution image feature amount from the high-resolution monochrome image 101 using a parameter group predetermined by learning, and uses the extracted high-resolution image feature amount as the synthesis means 22a. It is the one to output. The learning refers to learning for creating the color information magnifier 10. The first feature extraction means 31 converts 1ch monochrome information input to the first feature extraction means 31 into a high-resolution image feature amount for each output channel of the first feature extraction means 31.

The second feature extraction means 32 extracts a low-resolution image feature amount from the low-resolution color information 105 using a parameter group predetermined by learning, and size-enlarges the extracted low-resolution image feature amount 21. It is output to. The second feature extraction means 32 converts the 2ch color information input to the second feature extraction means 32 into a low-resolution image feature amount for each output channel of the second feature extraction means 32.

The third feature extraction means 33 extracts a high-resolution image feature amount from the high-resolution image feature amount generated by the synthesis means 22a using a parameter group predetermined by learning, and extracts the high-resolution image feature amount. The image feature amount is output to the high-resolution color information estimation means 23. The third feature extraction means 33 increases the amount of image features for a plurality of output channels (for example, 3 channels) corresponding to the plurality of outputs from the synthesis means 22a for each output channel of the third feature extraction means 33. Convert each to the image feature amount of the resolution. The number of output channels of the third feature extraction means 33 is set to, for example, 64ch, 128ch, 256ch, or the like.

For each feature extraction means 31 to 33, for example, a Convolution layer of a neural network may be used. In addition, there may be a plurality of Convolution layers (hidden layers). The number of output channels from each feature extraction means can be set to a desired value. In this specification, the image feature amount obtained by convolving the image feature amount input to the feature extraction means or the like for each output channel is referred to as a feature obtained from the input. Further, in the present specification, the feature extraction is defined as converting the input image feature amount by applying convolution to the input information consisting of a plurality of channels to the feature extraction means or the like.

In FIG. 7, the third feature extraction means 33 is shown separately from the high-resolution color information estimation means 23, but the high-resolution color information estimation means 23 may include the third feature extraction means 33 inside. The third feature extraction means 33 is a parameter group for outputting the image feature amounts for the two channels of the color space before the high-resolution color information estimation means 23 extracts the image feature amount for each channel in the color space. Uses different parameter groups to generate high-resolution image features for a plurality of channels (for example, 64 channels) from the high-resolution image features generated by the processing of the size enlargement means 21 and the compositing means 22a.

According to the color information magnifier 10, since the high-resolution monochrome image 101 (monochrome information) is explicitly used, it is possible to reduce the blurring of the estimated color information and magnify the low-resolution color information 105 with high accuracy. The color information magnifier 10 can be incorporated into a color information estimator 3 that estimates color information used when automatically coloring a high-resolution monochrome image 101 such as 4K or 8K. Further, the color information estimator 3 can improve the accuracy of estimating the color information used when automatically coloring the high-resolution monochrome image 101.

Further, digital data of a high-resolution monochrome image can be obtained by scanning, for example, from a physical film, but conventional coloring techniques cannot directly color such a high-resolution monochrome image. On the other hand, the automatic coloring device 1 provided with the color information estimator 3 can enable natural coloring of a high-resolution monochrome image such as 4K.

Further, for example, even in a situation where there is data of a monochrome image scanned from a photograph or a physical film, but the photograph or the physical film disappears and only the data remains, the automatic coloring device 1 provided with the color information estimator 3 Can estimate the color information at that time and color the monochrome image.

Further, for example, in a color-photographed image from which the low-resolution color information 105 is derived, a region having a clear boundary on the monochrome information channel (luminance channel in the color space) is a color information channel (for example, a luminance channel). In many cases, the boundary is clear even on the other 2 channels). Here, the boundary is a part represented by a line such as an outline of an object (a boundary between an object and its background).
Therefore, when the low-resolution color information 105 is enlarged by using the high-resolution monochrome image 101 like the color information magnifier 10, the boundary becomes clear especially on the high-resolution monochrome information channel (high-resolution monochrome image 101). It has the effect of reducing the blurring of color information in the area.

Although the above-mentioned color information has been described as two channels other than the luminance channel in the color space, it is possible to handle other channels. As an example, all three channels in the RGB color space may be used as color information.

Further, the format of the color information input to the color information magnifier and the color information estimator and the format of the output color information do not have to match. As an example, when the L channel in the Lab color space is input to the color information magnifier 10 as the high resolution monochrome image 101 and the ab channel in the Lab color space is input as the low resolution color information 105, the high resolution color information 107 It is also possible to output RGB channels in the RGB color space.

Further, the color information magnifier 10 is not limited to learning by a neural network, and can be configured by using other machine learning techniques. The error calculator 40 may be used not only for learning the color information magnifier 10 but also for learning the low-resolution color information estimator 7.

In order to confirm the performance of the error computer according to the embodiment, an experiment was conducted in which the color information was expanded by the color information expander using the error computer 40 for learning. FIG. 8 is an explanatory diagram schematically showing the color information magnifier used in the experiment. As shown in FIG. 8, the color information magnifier used in the experiment includes a size enlarging means 21, a synthesizing means 22a, a third feature extracting means 33, and a high-resolution color information estimating means 23. .. In this color information magnifier, the same components as those of the color information magnifier 10 shown in FIG. 7 are designated by the same reference numerals, and the description thereof will be omitted.

The compositing means 22a synthesizes the high-resolution monochrome image 101 and the high-resolution image information generated by the size-enlarging means 21 to generate high-resolution composite image information.

The high-resolution monochrome image 101 is 1ch monochrome information (image feature amount) corresponding to the L channel in the Lab color space. In FIG. 8, it is schematically shown as one image.
Further, in the experiment, it was assumed that the high-resolution monochrome image 101 was an image of 960 × 540 pixels. When the pixel value in the high-resolution monochrome image 101 is expressed by a vector, it is generally expressed by the following equation (6). _{The vector x 1} represented by the equation (6) has 518,400 components as in the number of pixels of the high-resolution monochrome image 101.

The low-resolution color information 105 is 2ch color information (image feature amount) corresponding to the ab channel in the Lab color space. In FIG. 8, it is schematically shown as two small images.
Further, in the experiment, it was assumed that the resolution of the low resolution color information 105 was 480 × 270 pixels. Then, in the experiment, the enlargement ratio by the size enlargement means 21 was set to 2 (double in the vertical direction x 2 times in the horizontal direction). In FIG. 8, it is schematically shown as two enlarged images.

When the pixel values in the enlarged 2ch color information are expressed by vectors, they are generally expressed by the following equations (7) and (8). Each of the vectors x ₂ and x ₃ has the same number of components _{as the vector x 1} represented by the above equation (6).

The synthesizing means 22a takes each vector x ₁ , x ₂ , and x ₃ as inputs, and arranges the vector components corresponding to each pixel to obtain 3ch information. In FIG. 8, it is schematically shown as three images. At this point, for example, a memory corresponding to the feature amount for each of 3 × 960 × 540 pixels is required.

The third feature extraction means 33 is composed of a neural network that performs convolution. In this experiment, 20 Convolution layers were constructed.
Further, in each Convolution layer, N features are extracted as outputs. That is, the number of output channels is N. In this experiment, Nch = 64ch.
In FIG. 8, only three Convolution layers are shown, and the others are omitted. Further, only 12 channels out of 64 channels are shown as Nch, and the others are omitted.

The first layer (first time) of the Convolution layer has 3 channels (3 channels in the color space), and each of the 64 output channels for the first layer is represented by the following equation (9). Convolution was done.

In equation (9), ω _i is a weight vector. The weight vector ω _i is a learning parameter determined by error calculation in _{which ω i} is updated by using an error during learning in this color information expander. The weight vector ω _i is a one-dimensional multivariable vector and has the same number of components as the number of pixels of the input high-resolution monochrome image 101. b is a bias. _{Note that x 1} , x ₂ , and x ₃ corresponding to i = 1, 2, and 3 are defined by equations (6) to (8).
At this point, for example, a memory corresponding to the feature amount for each of 64 × 960 × 540 pixels is required.

The second layer (second time) of the Convolution layer has 64 channels of input channels (64 channels in the output for the first layer in the previous stage), and for each of the 64 output channels for the second layer, the following equation ( The convolution represented by 10) was performed.

Equation (10) is expressed in the same format as Equation (9). Incidentally, x ₁ ~x ₆₄ corresponding to the i = 1 to 64 indicates the respective information 64 channels at the output of the first layer of the preceding stage, as in equation (6) to (8) Since it can be defined, the details are omitted.

Similarly, in the 3rd to 19th layers (3rd to 19th times) of the Convolution layer, the input channels are 64 channels (64 channels in the output for the previous layer), and each of the 64 output channels is described above. The convolution represented by the equation (10) was performed. Also in 3 to 19-layer, x ₁ ~x ₆₄ corresponding to the i = 1 to 64 are likewise shows an image feature amount for 64 channels at the output of their previous layer.

The high-resolution color information estimation means 23 is also composed of a Convolution layer. The high-resolution color information estimation means 23 extracts features corresponding to two channels in the color space as outputs. That is, the output channel is 2ch.
This Convolution layer (high-resolution color information estimation means 23) has 64 channels of input channels (64 channels in the output of the previous layer), and each of the two channels in the color space is represented by the above equation (10). The convolution to be done was done.

They are different from each from the omega _i in equation (10) and omega _i in the equation (9). _{Also, ω i} is different for each output channel. _{Further, different weight vectors ω i} were used for the 20 Convolution layers described above.

In the experiment, all 1282 convolutions (= 64 + 64 × 19 + 2) were performed as an example under the same conditions as below, while changing the _{weight vector ω i.}
Kernel: 3
Padding: 1
Stride: 1

Therefore, the total number of each component of the weight vector used in the experiment is the number of results obtained by calculating the following equation (11).
3 × 3 × (3 × 64 + 64 × 64 × 19 + 64 × 2)… Equation (11)
The total number of bias terms is 1282, which is the same as the number of convolutions. The sum of these is the total number of parameters.
That is, in the color information magnifier used in the experiment, the number of parameter groups determined in advance by learning is 703296 + 1282 = 704578.

As shown in FIG. 4, the high-resolution color information 107 obtained by the above processing is combined with the high-resolution monochrome image 101 which is the original image to create a high-resolution color image 109 (hereinafter, Example 1). ..
Further, the color information enlarged by the method of the prior art was combined with the high-resolution monochrome image 101 which is the original image to create a high-resolution color image (hereinafter, Comparative Example 1).
In Example 1, it was confirmed visually that the color blur was reduced as compared with Comparative Example 1.
Further, it was confirmed that the error that can be calculated by the equation (1) was reduced by about 17% from 7.66 (Comparative Example 1) to 6.53 (Example 1) on average.

The error calculator according to the present embodiment can be used for learning when creating a color information magnifier and a color information estimator.

1 Automatic coloring device 3 Color information estimator 5 Reducer 7 Low resolution color information estimator 9 Information synthesizer 10 Color information magnifier 21 Size enlargement means 22a Synthesis means 23 High resolution color information estimator 31 to 33 Feature extraction means 40 Error Computer 41 Feature map creation means 42 First error calculation means 43 Error synthesis means 51 Second error calculation means 52 Minimization means 60 Estimator S Learning device

Claims (7)

An error calculator that calculates the error between the first image feature amount, which is the estimated color information, and the second image feature amount, which is the true color information.
From the first image feature amount and the second image feature amount, the feature amount that characterizes the relationship of a plurality of pixels in the image is extracted by a predetermined calculation, and the first feature amount map and the second feature amount map Feature map creation means to create each
A first error calculating means for calculating an error between feature amount maps based on an error of pixel values between pixels corresponding to the first feature amount map and the second feature amount map, and a first error calculating means.
An error between the image feature amounts calculated based on the error of the pixel value between the pixels corresponding to the first image feature amount and the second image feature amount is input, and the error between the image feature amounts and the error between the image feature amounts are input. An error calculator comprising an error compositing means for generating a compositing error by adding the error between the feature quantity maps. The feature amount map creating means uses a predetermined filter to perform an operation of filtering the first image feature amount or the second image feature amount, thereby relating a plurality of adjacent pixels. The error calculator according to claim 1, wherein a feature amount that characterizes the property is extracted to create the first feature amount map or the second feature amount map. The feature amount map creating means maps the edge map to the first feature amount map by using an edge filter that detects an edge included in the image based on the first image feature amount or the second image feature amount. Alternatively, the error calculator according to claim 2, which is created as the second feature amount map. Second error calculation that calculates the error between the image feature amount based on the error of the pixel value between the pixels corresponding to the first image feature amount and the second image feature amount and outputs it to the error synthesizing means. The error calculator according to any one of claims 1 to 3, further comprising means. The first image feature amount is estimated color information that the learner outputs according to internal parameters with respect to the input value input as the training data.
The error calculator according to claim 4, wherein the second image feature amount is true color information prepared as training data of the learning device. A set of the first image feature amount and the second image feature amount is sequentially input to the feature amount map creating means and the second error calculating means, and according to the set of the input image feature amounts, The error calculator according to claim 5, further comprising an error minimizing means for adjusting the parameters of the learning device so that the synthesis error is reduced by a predetermined calculation and supplying the adjusted parameters as parameters for updating to the learning device. .. An error calculation program for operating a computer as an error calculator according to any one of claims 1 to 6.

Images