JP6902425B2 - Color information magnifiers and color information estimators, and their programs - Google Patents

Description

The present invention relates to a color information magnifier that magnifies color information to be added to a monochrome image, a color information estimator that estimates color information to be added to a monochrome image, and a program thereof.

In recent years, a plurality of automatic coloring technologies for converting monochrome images into digital data and colorizing them have been developed. This digital data has almost no clue of color information which is an image feature amount for converting this monochrome image into a color image. Therefore, the colorization of this digital data is more difficult than the colorization of so-called analog images recorded on a physical medium such as a film. For example, a method of converting monochrome data into color data is known (see Patent Document 1). This method assumes a specific object recorded in monochrome data and calculates a color distribution model from this specific object. Then, the color information is estimated from the calculated color distribution model. Since this method targets this specific object to be colored, there is a problem that it is difficult to make a monochrome image into a natural color image when the object assumed in advance and this specific object are different. ..

On the other hand, in recent years, a method of estimating color information for making the selection of a colorization target in a black-and-white image more general and colorizing by using a so-called machine learning technique has been proposed (Non-Patent Document 1, Non-Patent Document 1, See Non-Patent Document 2). However, the method of estimating color information using such machine learning technology is premised on preparing a huge amount of color images in which various objects are captured. Then, in this color information estimation method, a huge amount of color images are input to a learning device composed of, for example, a neural network, when learning to create a color information estimator. Then, according to the color information estimator created by this learning, the correspondence relationship between the monochrome image and the color information corresponding to the monochrome image is learned by the machine learning technique, and based on the correspondence relationship with the learned color information. , Color information can be estimated more accurately than before for various monochrome images given as inputs, and a natural color image can be generated by this.

Japanese Unexamined Patent Publication No. 2016-146529

Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa., "Let there be Color !: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification," ACM Transaction on Graphics (Proc. Of) SIGGRAPH), 35 (4): 110, 2016. Richard Zhang, Phillip Isola, and Alexei A. Efros. “Colorful Image Colorization.” In ECCV 2016.

However, the existing automatic coloring technology based on the machine learning technology described above mainly targets low-resolution images having a resolution of about 256 to 512 pixels in length and width, and naturally creates high-resolution monochrome images such as 4K images and 8K images. There was no technology that made it possible to color the image.

In the conventional plurality of techniques as described above, for a high-resolution monochrome image, the monochrome image that is the source of color imaging is compressed to a low resolution, and then low-resolution color information is estimated, and the input image is input. Expand to size. Further, these techniques then synthesize the enlarged color information with the original monochrome image, and make the combined color image correspond to the input high-resolution monochrome image. However, this conventional method has a problem that the estimated color image is blurred. In particular, there is a problem that when the enlargement ratio is increased, the color information is also greatly blurred.

The present invention has been made in view of the above problems, and is a color information magnifier and a color information estimator capable of reducing blurring in color information estimated from a high-resolution monochrome image, and a color information estimator thereof. The purpose is to provide a program.

In order to solve the above problems, the color information magnifier according to the first aspect of the present invention is the color information magnifier estimated from the monochrome information which is the image feature amount of the monochrome image of the first resolution and the monochrome information of the first resolution. A color information expander that inputs color information, which is an image feature amount of a second resolution lower than the first resolution, expands the image size of the color information by a predetermined arithmetic process, and outputs it as high-resolution color information. Therefore, it was decided to provide a size enlargement means, a synthesis means, and a high-resolution color information estimation means.

According to such a configuration, the color information magnifier is a low-resolution image feature amount that is either one of the second-resolution color information or the image feature amount extracted from the second-resolution color information by the size enlargement means. Generate high-resolution image features from.
Then, the color information magnifier is a high-resolution image feature amount extracted from the first-resolution monochrome information or the first-resolution monochrome information by the compositing means, and a high-resolution image generated by the size-enlarging means. Combine with features.
Then, the color information magnifier is a parameter determined in advance by learning for estimating for each channel of the color space from the high-resolution image feature amount synthesized by the high-resolution color information estimation means by the synthesis means. The high-resolution color information is estimated by extracting the image feature amount using the group.

Further, the color information magnifier according to the second aspect of the present invention is more than the monochrome information which is the image feature amount of the monochrome image of the first resolution and the first resolution estimated from the monochrome information of the first resolution. It is a color information magnifier that inputs color information, which is a low second resolution image feature amount, expands the image size of the color information by a predetermined arithmetic process, and outputs it as high resolution color information. It was decided to include an extraction means, a synthesis means, a size enlargement means, and a high-resolution color information estimation means.

According to such a configuration, the color information magnifier extracts a low-resolution image feature amount from the monochrome information of the first resolution by using a feature extraction means using a parameter group predetermined by learning for estimation. To do.
Then, the color information magnifier is a low-resolution image feature amount extracted from the second-resolution color information or the second-resolution color information by the compositing means, and a low-resolution image extracted by the feature extraction means. The feature amount and the feature amount are synthesized.
Then, the color information magnifier generates a high-resolution image feature amount from the low-resolution image feature amount synthesized by the compositing means by the size enlargement means.
Then, the color information magnifier is determined in advance by the high-resolution color information estimation means by learning to perform estimation for each channel in the color space from the high-resolution image feature amount generated by the size enlargement means. The high-resolution color information is estimated by extracting the image feature amount using the parameter group.

Further, in order to solve the above-mentioned problems, the color information estimator according to the first aspect of the present invention performs the color information magnifier and the process of reducing the monochrome information of the first resolution to achieve the second resolution. A low-resolution image feature amount is extracted from the reducer that generates monochrome information and the second-resolution monochrome information generated by the reducer using a parameter group that is predetermined by learning for estimation. The second resolution color information estimator for estimating the color information of the second resolution is provided, and the color information magnifier comprises the second resolution color information estimated by the low resolution color information estimator. It was decided to perform a process of estimating the high-resolution color information by using the monochrome information of the first resolution input by bypassing the reducer.

According to such a configuration, the color information estimator according to the first aspect of the present invention includes the color information of the second resolution estimated from the monochrome image of the first resolution through the reduction device and the low resolution color information estimator. , A monochrome image of the first resolution can be given as an input of the color information magnifier.

Further, in order to solve the above-mentioned problems, the color information estimator according to the second aspect of the present invention is determined in advance by learning to perform estimation from the color information magnifier and the monochrome information of the first resolution. A low-resolution color information estimator that estimates the second-resolution color information by extracting a low-resolution image feature amount using the parameter group is provided, and the color information magnifier provides the low-resolution color information. The high-resolution color information is estimated using the second-resolution color information estimated by the estimator and the first-resolution monochrome information input by bypassing the low-resolution color information estimator. It was decided to perform processing.

According to such a configuration, the color information estimator according to the second aspect of the present invention has the second resolution color information estimated by the low resolution color information estimator from the first resolution monochrome image and the first resolution. A monochrome image can be given as an input to the color information magnifier.

The present invention can also be realized by a color information enlargement program for making a computer function as the color information magnifier.
The present invention can also be realized by a color information estimation program for making a computer function as the color information estimator.

The present invention has the following excellent effects.
According to the color information magnifier according to the present invention, it is possible to reduce the blurring of the enlarged color information in the estimation process for enlarging the low-resolution color information estimated from the high-resolution monochrome image.
Further, according to the color information estimator according to the present invention, it is possible to reduce the blurring of the color information estimated from the input high-resolution monochrome image.

It is a block diagram which shows typically the structure of the automatic coloring apparatus which includes the color information estimator which concerns on 1st Embodiment of this invention. It is a block diagram which shows typically the structure of the color information magnifier which concerns on 1st Embodiment of this invention. It is a block diagram which shows typically the learning flow of the conventional low-resolution color information estimator. It is a block diagram which shows typically the learning flow of the color information expander which concerns on 1st Embodiment of this invention. It is a block diagram which shows typically the structure of the color information magnifier which concerns on 2nd Embodiment of this invention. It is a block diagram which shows typically the structure of the color information magnifier which concerns on 3rd Embodiment of this invention. It is a block diagram which shows typically the structure of the color information magnifier which concerns on 4th Embodiment of this invention. It is a block diagram which shows typically the structure of the color information estimator which concerns on 2nd Embodiment of this invention. It is a block diagram which shows typically the structure of the color information estimator which concerns on 3rd Embodiment of this invention. It is explanatory drawing which shows typically the color information magnifier used in an experiment.

Hereinafter, the color information magnifier and the color information estimator according to the embodiment of the present invention will be described with reference to the drawings.

[Automatic coloring device]
FIG. 1 is a block diagram schematically showing a configuration of an automatic coloring device including a color information estimator according to a first embodiment of the present invention.
The automatic coloring device 1 automatically colors a monochrome image by estimating color information from the monochrome image, and as shown in FIG. 1, mainly includes a color information estimator 3 and an information synthesizer 9. , Is equipped.
The automatic coloring device 1 is composed of, for example, a general computer, and includes a computing device such as a GPU (Graphics Processing Units), a ROM (Read Only Memory), a RAM (Random Access Memory), an HDD (Hard Disk Drive), and a general device. It has a typical image memory and an input / output interface.

The color information estimator 3 generates a low-resolution monochrome image 103 and a low-resolution color information 105 from the input high-resolution monochrome image 101, and estimates the high-resolution color information 107 using these information. ..
The high-resolution monochrome image 101 is a first-resolution monochrome image. The high-resolution monochrome image 101 is, for example, a monochrome image digitized by scanning from a past black-and-white film or photograph.
The low-resolution monochrome image 103 is a monochrome image having a second resolution lower than the first resolution.
The low resolution color information 105 is the color information of the second resolution.
The high-resolution color information 107 is the color information of the first resolution.

Here, the monochrome image is specifically an image composed of only the luminance channels in the color space (V channel in the HSV color space, L channel in the Lab color space, etc.). When the pixel information is luminance, the pixel value (luminance value) has a value of 0 to 255 when represented by 8-bit information. Monochrome information, which is an image feature amount of a monochrome image, is represented by, for example, a luminance distribution. In this specification, this monochrome information is used in the same meaning as a monochrome image.
Further, the color information can be, for example, an image feature amount for two channels other than the luminance channel. Here, the image feature amount is an amount representing a color space such as brightness, chromaticity, and saturation, for example. Further, the image feature amount may be, for example, an average value, a variance, a convolution integral value, or the like extracted from an amount representing a color space. The set of image features for each pixel is, for example, a monochrome image (monochrome information) or color information. Further, the image feature amount may be handled by a matrix in which elements are arranged in the height direction and the width direction (vertical and horizontal), or may be handled by a one-dimensional multivariable vector.

The value of the first resolution (value of high resolution) is not particularly limited as long as it is larger than the value of the second resolution (value of low resolution). For example, the size of the image of the second resolution may be 256 × 256 pixels, and the size of the image of the first resolution may be 512 × 512 pixels. Further, for example, the size of the image of the second resolution may be 480 × 270 pixels, and the size of the image of the first resolution may be 4K (3840 × 2160). Further, the size of the first resolution image may be 8K (7680 × 4320).

(First Embodiment of Color Information Estimator)
As shown in FIG. 1, the color information estimator 3 of the first embodiment includes a reducer 5, a low-resolution color information estimator 7, and a color information magnifier 10.
The color information magnifier 10 generates a high-resolution image feature amount from a low-resolution image feature amount which is either the low-resolution color information 105 or the image feature amount extracted from the low-resolution color information 105, and generates a high-resolution image feature amount to obtain a high-resolution monochrome image. The high-resolution image feature amount extracted from 101 or the high-resolution monochrome image 101 is combined with the high-resolution image feature amount generated by the size enlargement means 2, and the combined high-resolution image feature amount is used as the color space. For each channel of, the image feature amount is extracted using a parameter group determined in advance by learning for estimation.
As a result, the high-resolution color information 107 is estimated.

The reducer 5 generates a low-resolution monochrome image 103 by performing a process of reducing the input high-resolution monochrome image 101. Here, reduction means reducing the resolution, that is, reducing the number of pixels. When the reduction ratio in the reduction is, for example, 0.5, the number of pixels in the horizontal direction and the vertical direction of the reduced image is 1/2 of the number of pixels in the horizontal direction and the vertical direction of the original image, respectively. The reducer 5 outputs the generated low-resolution monochrome image 103 to the low-resolution color information estimator 7.

The low-resolution color information estimator 7 uses low-resolution color information (image feature amount) from the low-resolution monochrome image 103 generated by the reducer 5 by using a parameter group predetermined by learning for estimation. To extract. As a result, the low-resolution color information estimator 7 estimates the low-resolution color information 105. The learning flow for creating the low-resolution color information estimator 7 is the same as that of the conventional technique, but a brief description will be described later. Further, as a method for estimating color information, for example, a conventionally known method described in Non-Patent Document 1 can be used. The low-resolution color information estimator 7 inputs the luminance channel and outputs the estimated color information of two channels, similarly to the conventionally known color information estimator. Then, the low-resolution color information estimator 7 outputs the estimated low-resolution color information 105 to the color information magnifier 10.

The color information magnifier 10 expands the image size by inputting the low-resolution color information 105 estimated by the low-resolution color information estimator 7 and the high-resolution monochrome image 101 input by bypassing the reducer 5. The process of estimating the obtained color information (high-resolution color information 107) is performed. The color information magnifier 10 enlarges the low-resolution color information 105 by using the high-resolution monochrome image 101 (monochrome information). Then, the color information magnifier 10 outputs the estimated high-resolution color information 107 to the information synthesizer 9.

The information synthesizer 9 synthesizes the high-resolution color information 107 estimated by the color information estimator 3 and the high-resolution monochrome image 101 to create a high-resolution color image 109. The information synthesizer 9 simply combines monochrome information of one channel (hereinafter, may be referred to as 1ch) and color information of two channels (2ch) to generate a color image.

(Details of color information magnifier)
FIG. 2 is a block diagram schematically showing the configuration of the color information magnifier according to the first embodiment of the present invention. As shown in FIG. 2, the color information magnifier 10 includes a size enlarging means 21, a synthesizing means 22a, and a high-resolution color information estimating means 23. Although the color information magnifier 10 of FIG. 2 is shown in a form including the feature extraction means 31, 32, 33, for example, all the feature extraction means may be omitted. In the following, the feature extraction means may be referred to as the first feature extraction means 31, the second feature extraction means 32, and the third feature extraction means 33 for convenience.

The color information magnifier 10 can be configured by, for example, a neural network. Further, the neural network may be, for example, a CNN (Convolutional Neural Network). In CNN, a Convolution layer (convolutional layer) or a Deconvolutional layer (deconvolutional layer or Transposed Convolutional layer) is used as a hidden layer. Therefore, when CNN is adopted, each component of the color information magnifier 10 can be mounted by using the Convolution layer or the Deconvolution layer, and can be calculated at high speed by using the GPU.

The size enlargement means 21 generates a high-resolution image feature amount by performing a process of enlarging the input low-resolution image feature amount. Here, the low-resolution image feature amount means, for example, low-resolution color information 105. As shown in FIG. 2, when the color information magnifier 10 includes the second feature extraction means 32, the image feature amount extracted from the low resolution color information 105 by the second feature extraction means 32 has a low resolution. It becomes the image feature amount of. The size expanding means 21 outputs the generated high-resolution image feature amount to the compositing means 22a.

As the size enlargement means 21, for example, a Deconvolution layer (an image enlargement layer using a neural network) may be used. Further, the parameters used in a general image enlargement algorithm may be fixedly used. As a general image enlargement algorithm, for example, the nearest neighbor interpolation method or the Bilinear interpolation method may be used.

The synthesizing means 22a synthesizes, for example, the input high-resolution monochrome image 101 and the high-resolution image feature amount generated by the size expanding means 21. As shown in FIG. 2, when the color information magnifier 10 includes the first feature extraction means 31, the compositing means 22a includes an image feature amount extracted from the high-resolution monochrome image 101 and a size enlargement means. The high-resolution image features generated by 21 are combined. The synthesizing means 22a outputs the synthesized high-resolution image feature amount to the high-resolution color information estimating means 23.
The synthesizing means 22a simply synthesizes 1ch monochrome information and 2ch color information having the same size as the monochrome information to generate a high-resolution image feature amount. For the synthesis means 22a, for example, the Convolution layer of the neural network may be used.

The high-resolution color information estimation means 23 extracts an image feature amount from a high-resolution image feature amount synthesized by the synthesis means 22a using a predetermined parameter group by learning for estimating high-resolution color information. However, the high-resolution color information 107 is estimated.
Here, the learning means learning for creating the color information magnifier 10. Specifically, the accuracy is achieved by appropriately setting the internal parameters (parameter group) of the high-resolution color information estimation means 23 and the like by learning to create the color information magnifier 10 including the high-resolution color information estimation means 23. As a good estimator, the high-resolution color information estimation means 23 can be created. The flow of learning for creating the color information magnifier 10 will be described later.

The high-resolution color information 107 corresponds to the enlarged color information of the low-resolution color information 105, and has a resolution corresponding to the high-resolution monochrome image 101. The high-resolution color information 107 is color information for each channel in the color space, and refers, for example, an image feature amount for two channels other than the luminance channel.

The high-resolution color information estimation means 23 uses image features for a plurality of (3 or more) output channels corresponding to a plurality of (3 or more) outputs from the previous stage, and an image for two channels in the color space. Convert to feature quantity and estimate color information.
For the high-resolution color information estimation means 23, for example, the Convolution layer of the neural network may be used. In addition, there may be a plurality of Convolution layers (hidden layers). That is, the Convolution may be repeated continuously.
The number of output channels from the previous stage of the high-resolution color information estimation means 23 can be set to a desired value. For example, the number of output channels from the synthesis means 22a may be 3 channels or more.

As shown in FIG. 2, the color information magnifier 10 includes at least one feature extraction means of the first feature extraction means 31, the second feature extraction means 32, and the third feature extraction means 33. May be good.

The first feature extraction means 31 extracts a high-resolution image feature amount from the high-resolution monochrome image 101 using a parameter group predetermined by learning, and uses the extracted high-resolution image feature amount as the synthesis means 22a. It is the one to output. The learning refers to learning for creating the color information magnifier 10. The first feature extraction means 31 converts 1ch monochrome information input to the first feature extraction means 31 into a high-resolution image feature amount for each output channel of the first feature extraction means 31.

The second feature extraction means 32 extracts a low-resolution image feature amount from the low-resolution color information 105 using a parameter group predetermined by learning, and size-enlarges the extracted low-resolution image feature amount 21. It is output to. The second feature extraction means 32 converts the 2ch color information input to the second feature extraction means 32 into a low-resolution image feature amount for each output channel of the second feature extraction means 32.

The third feature extraction means 33 extracts a high-resolution image feature amount from the high-resolution image feature amount generated by the synthesis means 22a using a parameter group predetermined by learning, and extracts the high-resolution image feature amount. The image feature amount is output to the high-resolution color information estimation means 23. The third feature extraction means 33 increases the amount of image features for a plurality of output channels (for example, 3 channels) corresponding to the plurality of outputs from the synthesis means 22a for each output channel of the third feature extraction means 33. Convert each to the image feature amount of the resolution. The number of output channels of the third feature extraction means 33 is set to, for example, 64ch, 128ch, 256ch, or the like.

For each feature extraction means 31 to 33, for example, a Convolution layer of a neural network may be used. In addition, there may be a plurality of Convolution layers (hidden layers). The number of output channels from each feature extraction means can be set to a desired value. In this specification, the image feature amount obtained by convolving the image feature amount input to the feature extraction means or the like for each output channel is referred to as a feature obtained from the input. Further, in the present specification, the feature extraction is defined as converting the input image feature amount by applying convolution to the input information consisting of a plurality of channels to the feature extraction means or the like.

In FIG. 2, the third feature extraction means 33 is shown separately from the high-resolution color information estimation means 23, but the high-resolution color information estimation means 23 may include the third feature extraction means 33 inside. The third feature extraction means 33 is a parameter group for outputting the image feature amounts for the two channels of the color space before the high-resolution color information estimation means 23 extracts the image feature amount for each channel in the color space. Uses different parameter groups to generate high-resolution image features for a plurality of channels (for example, 64 channels) from the high-resolution image features generated by the processing of the size enlargement means 21 and the compositing means 22a.

(Learning flow of low resolution color information estimator)
Next, the learning flow of the low-resolution color information estimator 7 will be described with reference to FIG. The learning flow of the low-resolution color information estimator 7 is the same as the learning flow of the conventional color information estimator, and thus will be briefly described.
The low-resolution color information estimator 7 is generated from a learning device prepared in advance by the following procedure. This learner inputs a monochrome image and estimates and outputs color information by performing a predetermined calculation process. This learner (in FIG. 3, represented as a low-resolution color information estimator 7 in a state where learning has been completed) has an internal parameter (parameter group), and by changing this parameter, the learner can be used. Adjust the output. Then, a large amount of color images for learning are prepared, and the following steps S1 to S4 are repeated a sufficient number of times. This learner learns this parameter, and by setting the parameter appropriately, an accurate color information estimator can be created.

(Step S1)
A low-resolution color image 202 is prepared as a color image for learning, and the low-resolution monochrome image 203 and the true color information 204 are separated.
Here, the low-resolution monochrome image 203 is a low-resolution learning monochrome image.
Further, the true color information 204 is correct color information having the same size as the low-resolution learning monochrome image, and is used for error calculation with the estimated color information.

(Step S2)
Next, the learner (low-resolution color information estimator 7) inputs the low-resolution monochrome image 203, and outputs the low-resolution color information 205 as the color information of the estimation result using the current parameters.

(Step S3)
Next, the error calculator 40 calculates the error between the low resolution color information 205 (estimated color information) and the true color information 204. As this error, the mean square error of each pixel value or the like is used.

(Step S4)
Further, the error computer 40 uses an optimization method based on an error gradient such as SGD from the calculated error, so that the error is reduced by the learning device (low resolution color information estimator 7). Adjust the parameters and output the adjusted parameters to the learner. Since SGD is described in the following references, the description thereof will be omitted.
(Reference) L. Bottou., "Stochastic Gradient Descent Tricks.," Neural Networks: Tricks of the Trade: Springer, 2012.

The parameters appropriately set by the above learning are the parameters used when the low-resolution color information estimator 7 shown in FIG. 1 extracts an image feature amount from the low-resolution monochrome image 103 and estimates the low-resolution color information 105. It refers to a group. That is, the parameter group used when estimating the low-resolution color information 105 is the same as the low-resolution color information estimated by a predetermined calculation from the low-resolution learning monochrome image input to the learner and the learning monochrome image. It is determined by learning the correspondence between the correct color information of the size and.

(Learning flow of color information magnifier)
Next, the learning flow of the color information magnifier 10 will be described with reference to FIG.
The color information magnifier 10 is generated from a learning device prepared in advance by the following procedure. This learner inputs the high-resolution monochrome image 301 and the low-resolution color information 305, and estimates and outputs the high-resolution color information 307 by performing a predetermined calculation process. This learner (indicated as the color information magnifier 10 in the state where learning is completed in FIG. 4) has an internal parameter (parameter group), and by changing this parameter, the output from the learner can be output. adjust. Then, a large amount of color images for learning are prepared, and the following steps S10 to S14 are repeated a sufficient number of times. This learner learns this parameter, and by setting the parameter appropriately, an accurate color information magnifier can be created.

(Step S10)
A high-resolution color image 309 is prepared as a color image for learning, and the high-resolution color image 309 is simply reduced by the reducer 5 to obtain low-resolution color information 305.
Here, as the high-resolution color image 309, a colorized version of an old black-and-white film is also used. In this case, for example, a black-and-white image digitized by scanning from a past black-and-white film or a photograph is manually colored as digital data. Further, in order to prepare a large amount of high-resolution color images 309 for learning, a new color image such as 4K taken in color may be used in addition to the old black-and-white film.

(Step S11)
Next, the high-resolution color image 309 is separated into a high-resolution monochrome image 301 and a high-resolution color information (true color information) 304.
Here, the high-resolution monochrome image 301 is a high-resolution learning monochrome image.
Further, the high-resolution color information 304 is correct color information having the same size as the high-resolution learning monochrome image, and is used for error calculation with the estimated high-resolution color information.

(Step S12)
Next, the learning device (color information magnifier 10) inputs the high-resolution monochrome image 301, and outputs the high-resolution color information 307 as the color information of the estimation result using the current parameters.

(Step S13)
Next, the error calculator 40 calculates the error between the high-resolution color information 307 (estimated color information) and the high-resolution color information (true color information) 304. As this error, the mean square error of each pixel value, the cross entropy, or the like, which is the same as the above method, is used.

(Step S14)
Further, the error calculator 40 uses an optimization method based on an error gradient such as SGD from the calculated error to reduce the error so that the parameter of the learning device (color information magnifier 10) is set. Adjust and output the adjusted parameters to the learner. The error computer 40 is added at the time of learning, but is disconnected after learning.

The parameters appropriately set by the above learning refer to a group of parameters used by the color information magnifier 10 shown in FIG. 2 when estimating the high-resolution color information 107. For example, the high-resolution color information estimation means 23 is also used when extracting the image feature amount for each channel of the color space from the high-resolution image feature amount generated by the synthesis means 22a.
The high-resolution image feature amount generated by the compositing means 22a includes information derived from the information (monochrome information) of the high-resolution monochrome image 101 and the low-resolution color information 105.
That is, the parameter group used when estimating the high-resolution color information 107 is an enlarged height estimated by a predetermined calculation from the low-resolution learning color information and the high-resolution learning monochrome image input to the learner, respectively. It is determined by learning the correspondence between the color information of the resolution and the correct color information of the same size as the monochrome image for learning.

When the color information magnifier 10 is made by learning, for example, if it is desired to make the color information magnifier 10 having the configuration shown in FIG. 2, the color information magnifier 10 having the same configuration as that shown in FIG. 2 is used for learning. Further, when making a color information magnifier by learning, if it is desired to make a color information magnifier that omits at least one feature extraction means, a color information magnifier that omits the feature extraction means may be used for learning.

According to the color information magnifier 10 according to the present embodiment, since the high-resolution monochrome image 101 (monochrome information) is explicitly used, the estimated color information blurring is reduced and the low-resolution color information 105 is accurate. Can be expanded well. The color information magnifier 10 can be incorporated into a color information estimator 3 that estimates color information used when automatically coloring a high-resolution monochrome image 101 such as 4K or 8K. Further, the color information estimator 3 according to the present embodiment can improve the accuracy of estimating the color information used when automatically coloring the high-resolution monochrome image 101.

Further, digital data of a high-resolution monochrome image can be obtained by scanning, for example, from a physical film, but conventional coloring techniques cannot directly color such a high-resolution monochrome image. On the other hand, the automatic coloring device 1 provided with the color information estimator 3 can enable natural coloring of a high-resolution monochrome image such as 4K.

Further, for example, even in a situation where there is data of a monochrome image scanned from a photograph or a physical film, but the photograph or the physical film disappears and only the data remains, the automatic coloring device 1 provided with the color information estimator 3 Can estimate the color information at that time and color the monochrome image.

Further, for example, in a color-photographed image from which the low-resolution color information 105 is derived, a region having a clear boundary on the monochrome information channel (luminance channel in the color space) is a color information channel (for example, a luminance channel). In many cases, the boundary is clear even on the other 2 channels). Here, the boundary is a part represented by a line such as an outline of an object (a boundary between an object and its background).
Therefore, when the low-resolution color information 105 is enlarged by using the high-resolution monochrome image 101 like the color information magnifier 10, the boundary becomes clear especially on the high-resolution monochrome information channel (high-resolution monochrome image 101). It has the effect of reducing the blurring of color information in the area.

(Second Embodiment of Color Information Enlarger)
Next, a second embodiment of the color information magnifier will be described with reference to FIG. 5 (see FIG. 2 as appropriate). The color information magnifier 10A shown in FIG. 5 is different from the color information magnifier 10 shown in FIG. 2 in that the compositing means 22b is provided in front of the size enlarging means 21. In the color information magnifier 10A, the same components as those of the color information magnifier 10 shown in FIG. 2 are designated by the same reference numerals, and the description thereof will be omitted.

The color information expander 10A includes a feature extraction means 34, a synthesis means 22b, a size enlargement means 21, and a high-resolution color information estimation means 23. Although the color information magnifier 10A of FIG. 5 is shown in a form including the feature extraction means 35 and 36, for example, the feature extraction means 35 and 36 may be omitted. In the following, for convenience, the feature extraction means may be referred to as the second feature extraction means 35 and the third feature extraction means 36. The color information magnifier 10A can be configured by, for example, a neural network.

The feature extraction means 34 extracts a low-resolution image feature amount from the high-resolution monochrome image 101 using a parameter group predetermined by learning for estimation, and synthesizes the extracted low-resolution image feature amount. It is output to 22b.

The synthesizing means 22b, for example, synthesizes the low-resolution color information 105 and the low-resolution image feature amount extracted by the feature extraction means 34 to generate a low-resolution image feature amount. As shown in FIG. 5, when the color information magnifier 10A includes the second feature extraction means 35, the synthesis means 22b includes a low-resolution image feature amount extracted from the low-resolution color information 105 and a low-resolution image feature amount. The low-resolution image feature amount extracted by the feature extraction means 34 is combined. The compositing means 22b outputs the synthesized low-resolution image feature amount to the size enlarging means 21.
The compositing means 22b simply synthesizes 2ch low-resolution color information and 1ch monochrome information having the same size as the low-resolution color information to generate a low-resolution image feature amount. For the synthesis means 22b, for example, the Convolution layer of the neural network may be used.

In the present embodiment, the size enlargement means 21 outputs the high resolution image feature amount generated by the size enlargement means 21 to, for example, the high resolution color information estimation means 23.
In the present embodiment, the high-resolution color information estimation means 23 estimates the high-resolution color information 107 from the high-resolution image feature amount generated by the size enlargement means 21.

As shown in FIG. 5, the color information magnifier 10A further includes at least one feature extraction means of the second feature extraction means 35 and the third feature extraction means 36, in addition to the feature extraction means 34. You may prepare.

The second feature extraction means 35 extracts a low-resolution image feature amount from the low-resolution color information 105 using a parameter group predetermined by learning, and uses the extracted low-resolution image feature amount in the synthesis means 22b. It is the one to output. The second feature extraction means 35 is the same as the second feature extraction means 32 shown in FIG. 2, except for the output destination of the extracted low-resolution image feature amount.

The third feature extraction means 36 extracts a high-resolution image feature amount from the high-resolution image feature amount generated by the size enlargement means 21 using a parameter group predetermined by learning, and extracts the high-resolution image feature amount. The image feature amount of the above is output to the high resolution color information estimation means 23. The third feature extraction means 36 is the same as the third feature extraction means 33 shown in FIG. 2, except for an input destination for receiving a high-resolution image feature amount.

In FIG. 5, the third feature extraction means 36 is shown separately from the high-resolution color information estimation means 23, but the high-resolution color information estimation means 23 may include the third feature extraction means 36 inside. For example, the Convolution layer of the neural network may be used for each of the feature extraction means 34 to 36.

Since the learning flow of the color information magnifier 10A is the same as the learning flow of the color information magnifier 10, the description thereof will be omitted. When making a color information magnifier by learning, if it is desired to make the color information magnifier 10A having the configuration shown in FIG. 5, the color information magnifier 10A having the configuration shown in FIG. 5 may be used for learning. Further, when making a color information magnifier by learning, if it is desired to make a color information magnifier that omits at least one feature extraction means, a color information magnifier that omits the feature extraction means may be used for learning.

According to the color information magnifier 10A according to the second embodiment, it is estimated because the high resolution monochrome image 101 (monochrome information) is explicitly used as in the color information magnifier 10 according to the first embodiment. The blurring of color information can be reduced, and the low-resolution color information 105 can be enlarged with high accuracy.

(Third Embodiment of the color information magnifier)
Next, a third embodiment of the color information magnifier will be described with reference to FIG. 6 (see FIGS. 2 and 5 as appropriate). In the color information magnifier 10B, the same components as those of the color information magnifiers 10 and 10A are designated by the same reference numerals, and the description thereof will be omitted.

The color information expander 10B includes a feature extraction means 34, a synthesis means 22b, a size enlargement means 21, a synthesis means 22a, and a high-resolution color information estimation means 23. The color information magnifier 10B of FIG. 6 is shown in a form including a first feature extraction means 31, a second feature extraction means 35, and a third feature extraction means 33. The extraction means 31, 35, 33 may be omitted. The color information magnifier 10B can be configured by, for example, a neural network.

Since the color information magnifier 10B shown in FIG. 6 is a mixture of the color information magnifiers 10 and 10A and provided with the compositing means 22b and 22a before and after the size enlarging means 21, further description thereof will be omitted. To do. Since the learning flow of the color information magnifier 10B is the same as the learning flow of the color information magnifier 10, the description thereof will be omitted.

According to the color information magnifier 10B according to the third embodiment, it is estimated because the high resolution monochrome image 101 (monochrome information) is explicitly used as in the color information magnifier 10 according to the first embodiment. The blurring of color information can be reduced, and the low-resolution color information 105 can be magnified with high accuracy.

(Fourth Embodiment of Color Information Enlarger)
Next, a fourth embodiment of the color information magnifier will be described with reference to FIG. 7 (see FIGS. 1 and 2 as appropriate). In the color information magnifier 10C, the same reference numerals are given to the same configurations as those of the color information magnifier 10, and the description thereof will be omitted.
Here, the size of the image of the first resolution is N (= 3840 × 2160 pixels), and the size of the image of the second resolution is N / 8 (= 480 × 270 pixels). That is, it is assumed that the high-resolution monochrome image 101 input to the color information magnifier 10C is a monochrome image having a resolution of N. Further, it is assumed that the low resolution color information 105 input to the color information magnifier 10C is color information having a resolution of N / 8.

The color information magnifier 10C is a recursive color information magnifier using a structure in which the color information magnifiers 10 are recursively connected. Here, it is assumed that the color information magnifier 10 has an enlargement ratio of 2, and the three color information magnifiers 10 are recursively connected.

The color information magnifier 10C includes a size reducing means 50.
The size reduction means 50 recursively reduces the high-resolution monochrome image 101 at a predetermined reduction ratio, so that the monochrome image 101 has a plurality of levels of resolution that is smaller than the first resolution and larger than the second resolution. Generate an image.

The size reduction means 50 reduces the high-resolution monochrome image 101 (monochrome image with resolution = N) to generate the monochrome image 111. The monochrome image 111 is a monochrome image having a resolution of N / 2.
Further, the size reducing means 50 reduces the monochrome image 111 (monochrome image having a resolution = N / 2) to generate the monochrome image 121. The monochrome image 121 is a monochrome image having a resolution of N / 4. Further, the monochrome image 121 is a monochrome image having the minimum level resolution generated by the size reduction means 50 in this case.

The color information magnifier 10 that inputs the low-resolution color information 105 input to the color information magnifier 10C and the monochrome image 121 of the minimum level resolution generated by the size reduction means 50 finally has a color. As the estimated color information for each channel in the space, the low-resolution color information 105 is expanded to output the color information 127. Here, since the color information magnifier 10 has an enlargement ratio of 2, the color information 127 is color information having a resolution of N / 4.

The color information magnifier 10 that inputs the color information 127 and the monochrome image 111 generated by the size reduction means 50 finally expands the color information 127 as the estimated color information for each channel in the color space. The color information 117 is output. Here, since the color information magnifier 10 has an enlargement ratio of 2, the color information 117 is color information having a resolution of N / 2.

The color information magnifier 10 that inputs the color information 117 and the high-resolution monochrome image 101 input to the color information magnifier 10C finally obtains color information as estimated color information for each channel in the color space. The 117 outputs the enlarged high-resolution color information 107. Here, since the color information magnifier 10 has an enlargement ratio of 2, the high-resolution color information 107 is color information having a resolution of N. Thus, the low-resolution color information 105 which is input to the color information expander 10C are enlarged in 2 ^triple (= 8 times), so as to be outputted as a high-resolution color information 107.

As described above, the color information magnifier 10C uses the low-resolution color information 105 and the monochrome image 121 with the lowest resolution to be generated as initial values, and obtains the estimated color information and a resolution larger than the color information. The process of estimating the high-resolution color information 107 having the first resolution is performed by recursively performing the process of estimating the color information having the same resolution as the monochrome image from the possessed monochrome image.

Since the color information magnifier 10C is a recursive color information magnifier using a structure in which the color information magnifiers 10 are recursively connected, the color information magnifier 10 having an enlargement ratio of M (for example, M = 2) is used. By making one, you can make a magnifier that is a multiple of M.
For example the final magnification when making the learning color information expander 10C for two ^3-fold (= 8 times) may be used color information expander 10 to the magnification doubled learning. This can have the effect of reducing the number of parameters to be determined to about one-third when trying to achieve eight-fold magnification without using a recursive color information magnifier.
Therefore, according to the color information magnifier 10C, when the resolution of the low-resolution color information 105 estimated in advance is N / 8 (480 × 270 pixels), the resolution is expanded to N (= 3840 × 2160 pixels). High-resolution color information 107 can be easily acquired.

The color information magnifier 10C may recursively connect the color information magnifiers 10A and 10B instead of the color information magnifier 10. Further, the final enlargement ratio by the color information magnifier 10C is not limited to 8 times, but may be 4 times, 16 times, or the like.

Further, in the color information magnifier 10C, for example, instead of connecting the three color information magnifiers 10, one color information magnifier 10 is operated at different timings so that the three color information magnifiers 10 work. You may do so.
Similarly, in the color information magnifier 10C, for example, instead of connecting the two size reduction means 50, one size reduction means 50 is operated at different timings so that the two size reduction means 50 work. You may.

[Second Embodiment of Color Information Estimator]
Next, the color information estimator according to the second embodiment of the present invention will be described with reference to FIG. 8 (see FIG. 1 as appropriate). As shown in FIG. 8, the color information estimator 3B includes a low-resolution color information estimator 7B and a color information magnifier 10D.

The low-resolution color information estimator 7B estimates the low-resolution color information 105 in the same manner as the low-resolution color information estimator 7 shown in FIG. The low-resolution color information estimator 7B outputs the estimated low-resolution color information 105 to the color information magnifier 10D. This low-resolution color information estimator 7B differs from the low-resolution color information estimator 7 shown in FIG. 1 in that it uses a high-resolution monochrome image 101 as an input, but since it is a conventionally known color information estimator, it is used. The above description will be omitted.

The color information magnifier 10D uses the low-resolution color information 105 estimated by the low-resolution color information estimator 7B and the high-resolution monochrome image 101 input by bypassing the low-resolution color information estimator 7B. The process of estimating the high-resolution color information 107 is performed. As shown in FIG. 8, the color information expander 10D includes a size enlargement means 21, a synthesis means 22a, a third feature extraction means 33, and a high-resolution color information estimation means 23. In the color information magnifier 10D, the same components as those of the color information magnifier 10 shown in FIG. 2 are designated by the same reference numerals, and the description thereof will be omitted.

The synthesizing means 22a synthesizes the same monochrome image as the high-resolution monochrome image 101 input to the low-resolution color information estimator 7B and the high-resolution image feature amount generated by the size-enlarging means 21. Here, the same monochrome image as the high-resolution monochrome image 101 input to the low-resolution color information estimator 7B is input to the synthesizing means 22a of the color information magnifier 10D via the bypass path 401. As a result, the color information magnifier 10D can directly use the high-resolution monochrome image 101 (monochrome information) when creating the high-resolution color information 107 from the low-resolution color information 105.

Assuming that the color information estimator of the comparative example in which the high-resolution monochrome image 101 is not input from the bypass path 401 is assumed, it is possible to provide the color information expansion function even in such a comparative example. it is conceivable that. The reason is that the information possessed by the high-resolution monochrome image 101 is deformed in the process of passing through the low-resolution color information estimator 7B, but theoretically, it is transmitted to the color information magnifier 10D even without bypass. Because there is.
On the other hand, the color information estimator 3B according to the second embodiment learns a portion corresponding to the color information magnifier 10D due to the presence of the input of the high-resolution monochrome image 101 from the bypass path 401. In addition, high-resolution monochrome information has a stronger influence than such comparative examples. Therefore, the color information estimator 3B expands the color information so that there is no blur in the region where the boundary is clear on the high-resolution monochrome information channel (high-resolution information of the brightness channel in the color space). It has been experimentally found that learning is promoted more than the above-mentioned comparative example.

[Third Embodiment of the color information estimator]
Next, the color information estimator according to the third embodiment of the present invention will be described with reference to FIG. 9 (see FIGS. 7 and 8 as appropriate). As shown in FIG. 9, the color information estimator 3C includes a low-resolution color information estimator 7B, a color information magnifier 10E, and a size reducing means 50. Here, it is assumed that the high-resolution monochrome image 101 input to the color information estimator 3C is a monochrome image having a resolution of N (= 3840 × 2160 pixels). Further, it is assumed that the color information output by the low resolution color information estimator 7B is the color information of resolution = N / 8.

In the color information estimator 3C, the same components as those of the color information magnifier 10C of FIG. 7 are designated by the same reference numerals, and the description thereof will be omitted. The size reduction means 50 generates a monochrome image 111 and the monochrome image 121 in the same manner as the size reduction means 50 in the color information magnifier 10C of FIG. 7.

As shown in FIG. 9, the color information expander 10E includes size enlargement means 21,21b, 21c, synthesis means 22a, 22b, 22c, a third feature extraction means 33, and high-resolution color information estimation means 23. , Is equipped. In the color information magnifier 10E, the same reference numerals are given to the same configurations as the color information magnifier 10D shown in FIG. 8, and the description thereof will be omitted.

The size enlargement means 21, 21b, and 21c have the same function, and here, the input low-resolution image feature amount is increased by performing a process of magnifying the input low-resolution image feature amount by, for example, twice by a general image enlargement algorithm. Generate image features with resolution.
Each of the compositing means 22a, 22b, and 22c has the same function, and here, each image feature amount input from a different route is simply combined to generate an image feature amount in which the number of output channels is increased. ..

The color information estimator 3C having such a configuration is provided with the size reduction means 50, so that the enlargement ratio for enlarging the estimated low-resolution color information can be set to a relatively large value.
Specifically, in the color information estimator 3C, the size expanding means 21c expands the 2ch color information (resolution = N / 8) output by the low resolution color information estimator 7B, thereby expanding the 2ch color information (resolution). = N / 4) is generated.
Then, the synthesizing means 22c synthesizes the monochrome image 121 (resolution = N / 4) and the color information (resolution = N / 4) generated by the size expanding means 21c, so that the 3ch image feature amount (resolution = N / 4) is combined. Generate resolution = N / 4). Here, the compositing means 22c inputs the monochrome image 121 (resolution = N / 4) to the compositing means 22c via the bypass path 403.

Then, the size expanding means 21b enlarges the 3ch image feature amount (resolution = N / 4) generated by the synthesizing means 22c to generate the 3ch image feature amount (resolution = N / 2). Then, the synthesizing means 22b synthesizes the monochrome image 111 (resolution = N / 2) and the 3ch image feature amount (resolution = N / 2) generated by the size expanding means 21b to form a 4ch image feature amount. (Resolution = N / 2) is generated. Here, the size enlargement means 21b inputs the monochrome image 111 (resolution = N / 2) to the synthesis means 22b via the bypass path 402.

Then, the size expanding means 21 enlarges the 4ch image feature amount (resolution = N / 2) generated by the compositing means 22b to generate the 4ch image feature amount (resolution = N).
Then, the synthesizing means 22a synthesizes the high-resolution monochrome image 101 (resolution = N) and the 4ch image feature amount (resolution = N) generated by the size expanding means 21, and the 5ch image feature amount (resolution = N). ) Is generated. Here, the size enlargement means 21 inputs the high-resolution monochrome image 101 (resolution = N) to the synthesis means 22a via the bypass path 401.

Then, the third feature extraction means 33 converts the 5ch image feature amount (resolution = N) generated by the synthesis means 22a into, for example, 64ch image feature amount (resolution = N).
Finally, the high-resolution color information estimation means 23 converts, for example, an image feature amount (resolution = N) of 64 channels into color information (resolution = N) of two channels in the color space. As a result, high-resolution color information 107 is generated.

Although each embodiment of the present invention has been described above, the present invention is not limited to these and can be carried out without changing the gist thereof. For example, the color information in the present invention is set to two channels other than the luminance channel in the color space, but other channels can be handled. As an example, all three channels in the RGB color space may be used as color information.

Further, the format of the color information input to the color information magnifier and the color information estimator and the format of the output color information do not have to match. As an example, when the L channel in the Lab color space is input to the color information magnifier 10 as the high resolution monochrome image 101 and the ab channel in the Lab color space is input as the low resolution color information 105, the high resolution color information 107 It is also possible to output RGB channels in the RGB color space.

Further, instead of configuring all the components of the color information magnifier 10 with a neural network, parameters used in a general image magnifying algorithm such as Bilinear interpolation method are fixedly used for the size enlarging means 21, and other configurations are used. The elements may be composed of a neural network. In this case, it is experimentally found that it is better than the case where all the components of the color information magnifier 10 are configured by the neural network.
Further, the color information magnifier 10 is not limited to learning by a neural network, and can be configured by using other machine learning techniques.

Further, in each of the above-described embodiments, the color information magnifiers 10, 10A to 10D have been described, but they are regarded as a color information magnifying program described in a general-purpose or special computer language so as to enable processing of the configuration of each device. It is also possible.
Further, in each of the above-described embodiments, the color information estimators 3 and 3B have been described, but they can also be regarded as a color information estimation program written in a general-purpose or special computer language so as to enable processing of the configuration of each device. It is possible.

An experiment was conducted to confirm the performance of the color information magnifier according to the embodiment. FIG. 10 is an explanatory diagram schematically showing the color information magnifier used in the experiment. As shown in FIG. 10, the color information magnifier used in the experiment has the same configuration as the color information magnifier 10D shown in FIG.

The high-resolution monochrome image 101 is 1ch monochrome information (image feature amount) corresponding to the L channel in the Lab color space. In FIG. 10, it is schematically shown as one image.
Further, in the experiment, it was assumed that the high-resolution monochrome image 101 was an image of 960 × 540 pixels. When the pixel value in the high-resolution monochrome image 101 is expressed by a vector, it is generally expressed by the following equation (1). _{The vector x 1} represented by the equation (1) has 518,400 components as in the number of pixels of the high-resolution monochrome image 101.

The low-resolution color information 105 is 2ch color information (image feature amount) corresponding to the ab channel in the Lab color space. In FIG. 10, it is schematically shown as two small images.
Further, in the experiment, it was assumed that the resolution of the low resolution color information 105 was 480 × 270 pixels. Then, in the experiment, the enlargement ratio by the size enlargement means 21 was set to 2 (double in the vertical direction x 2 times in the horizontal direction). In FIG. 10, it is schematically shown as two enlarged images.

When the pixel values in the enlarged 2ch color information are expressed by vectors, they are generally expressed by the following equations (2) and (3). Each of the vectors x ₂ and x ₃ has the same number of components _{as the vector x 1} represented by the above equation (1).

The synthesizing means 22a takes each vector x ₁ , x ₂ , and x ₃ as inputs, and arranges the vector components corresponding to each pixel to obtain 3ch information. In FIG. 10, it is schematically shown as three images. At this point, for example, a memory corresponding to the feature amount for each of 3 × 960 × 540 pixels is required.

The third feature extraction means 33 is composed of a neural network that performs convolution. In this experiment, 20 Convolution layers were constructed.
Further, in each Convolution layer, N features are extracted as outputs. That is, the number of output channels is N. In this experiment, Nch = 64ch.
In FIG. 10, only three Convolution layers are shown, and the others are omitted. Further, only 12 channels out of 64 channels are shown as Nch, and the others are omitted.

The first layer (first time) of the Convolution layer has 3 channels (3 channels in the color space), and each of the 64 output channels for the first layer is represented by the following equation (4). Convolution was done.

In equation (4), ω _i is a weight vector. The weight vector ω _i is a learning parameter determined by error calculation in _{which ω i} is updated by using an error during learning in this color information expander. The weight vector ω _i is a one-dimensional multivariable vector and has the same number of components as the number of pixels of the input high-resolution monochrome image 101. b is a bias. _{Note that x 1} , x ₂ , and x ₃ corresponding to i = 1, 2, and 3 are defined by equations (1) to (3).
At this point, for example, a memory corresponding to the feature amount for each of 64 × 960 × 540 pixels is required.

The second layer (second time) of the Convolution layer has 64 channels of input channels (64 channels in the output for the first layer in the previous stage), and for each of the 64 output channels for the second layer, the following equation ( The convolution represented by 5) was performed.

Equation (5) is expressed in the same format as Equation (4). Incidentally, x ₁ ~x ₆₄ corresponding to the i = 1 to 64 indicates the respective information 64 channels at the output of the first layer of the preceding stage, as in equation (1) to (3) Since it can be defined, the details are omitted.

Similarly, in the 3rd to 19th layers (3rd to 19th times) of the Convolution layer, the input channels are 64 channels (64 channels in the output for the previous layer), and each of the 64 output channels is described above. The convolution represented by the equation (5) was performed. Also in 3 to 19-layer, x ₁ ~x ₆₄ corresponding to the i = 1 to 64 are likewise shows an image feature amount for 64 channels at the output of their previous layer.

The high-resolution color information estimation means 23 is also composed of a Convolution layer. The high-resolution color information estimation means 23 extracts features corresponding to two channels in the color space as outputs. That is, the output channel is 2ch.
This Convolution layer (high-resolution color information estimation means 23) has 64 channels of input channels (64 channels in the output of the previous layer), and each of the two channels in the color space is represented by the above equation (5). The convolution to be done was done.

They are different from each from the omega _i in omega _i the formula in equation (4) described above (5). _{Also, ω i} is different for each output channel. _{Further, different weight vectors ω i} were used for the 20 Convolution layers described above.

In the experiment, all 1282 convolutions (= 64 + 64 × 19 + 2) were performed as an example under the same conditions as below, while changing the _{weight vector ω i.}
Kernel: 3
Padding: 1
Stride: 1

Therefore, the total number of each component of the weight vector used in the experiment is the number of results obtained by calculating the following equation (6).
3 × 3 × (3 × 64 + 64 × 64 × 19 + 64 × 2) ・ ・ ・ Equation (6)
The total number of bias terms is 1282, which is the same as the number of convolutions. The sum of these is the total number of parameters.
That is, in the color information magnifier used in the experiment, the number of parameter groups determined in advance by learning is 703296 + 1282 = 704578.

As shown in FIG. 1, the high-resolution color information 107 obtained by the above processing is combined with the high-resolution monochrome image 101 which is the original image to create a high-resolution color image 109 (hereinafter, Example 1). ..
Further, the color information enlarged by the method of the prior art was combined with the high-resolution monochrome image 101 which is the original image to create a high-resolution color image (hereinafter, Comparative Example 1).
In Example 1, it was confirmed visually that the color blur was reduced as compared with Comparative Example 1.
Further, when applied to 110 randomly selected images, PSNR (Peak Signal-to-Noise Ratio), which is an average value of a scale indicating the degree of deterioration with respect to the original image, ranges from 37.66 (comparative example) to 41. It was confirmed that the improvement was made to 35 (Example).

When the same experiment is performed using a larger 4K image as the high-resolution monochrome image 101, the number of features increases dramatically, so that a hardware resource having a larger memory area is required. is there.
Further, when the first feature extraction means 31 and the second feature extraction means 32 shown by the broken lines in FIG. 10 are added to have the same configuration as the color information magnifier 10 of FIG. 2, more parameters are determined. There is a need.

The color information magnifier according to this embodiment can be used for automatic coloring of 4K monochrome video data and the like.

1 Automatic coloring device 3,3B, 3C color information estimator 5 reducer 7,7B low resolution color information estimator 9 information synthesizer 10,10A, 10B, 10C, 10D, 10E color information magnifier 21,21b, 21c size Enlargement means 22a, 22b, 22c Synthesis means 23 High-resolution color information estimation means 31-36 Feature extraction means 40 Error calculator 50 Size reduction means

Claims (10)

Monochrome information, which is an image feature amount of a monochrome image of the first resolution, and color information, which is an image feature amount of a second resolution lower than the first resolution estimated from the monochrome information of the first resolution, are input. This is a color information magnifier that enlarges the image size of the color information by a predetermined arithmetic process and outputs it as high-resolution color information.
A size enlargement means for generating a high-resolution image feature from a low-resolution image feature that is either one of the second-resolution color information or the image feature extracted from the second-resolution color information.
A compositing means for synthesizing a high-resolution image feature amount extracted from the first-resolution monochrome information or the first-resolution monochrome information and a high-resolution image feature amount generated by the size enlargement means.
The high-resolution color is extracted from the high-resolution image feature amount synthesized by the compositing means using a parameter group predetermined by learning for estimation for each channel in the color space. A color information magnifier including a high-resolution color information estimation means for estimating information. Monochrome information, which is an image feature amount of a monochrome image of the first resolution, and color information, which is an image feature amount of a second resolution lower than the first resolution estimated from the monochrome information of the first resolution, are input. This is a color information magnifier that enlarges the image size of the color information by a predetermined arithmetic process and outputs it as high-resolution color information.
A feature extraction means for extracting a low-resolution image feature amount from the first-resolution monochrome information using a parameter group predetermined by learning for estimation.
A compositing means for synthesizing a low-resolution image feature amount extracted from the second-resolution color information or the second-resolution color information and a low-resolution image feature amount extracted by the feature extraction means.
A size enlargement means for generating a high-resolution image feature amount from a low-resolution image feature amount synthesized by the compositing means, and a size enlargement means.
The high-resolution image feature amount is extracted from the high-resolution image feature amount generated by the size enlargement means using a parameter group predetermined by learning for estimation for each channel in the color space. A color information magnifier including a high-resolution color information estimation means for estimating color information. The high-resolution color information estimation means
To estimate from the high-resolution image features before extracting the image features for each channel in the color space from the high-resolution image features generated by the processing of the compositing means and the size enlargement means. It is provided with a feature extraction means for extracting a higher resolution image feature amount using a parameter group determined in advance by learning.
The color information magnifier according to claim 1 or 2, wherein the image feature amount is extracted for each channel of the color space from the high-resolution image feature amount extracted by the feature extraction means. A high-resolution image feature amount is extracted from the first-resolution monochrome information using a parameter group predetermined by learning for estimation, and the extracted high-resolution image feature amount is output to the synthesis means. The first feature extraction means and
A low-resolution image feature is extracted from the second-resolution color information using a parameter group predetermined by learning for estimation, and the extracted low-resolution image feature is output to the size enlargement means. Second feature extraction means to be
The color information magnifier according to claim 1, further comprising at least one feature extraction means. A low-resolution image feature is extracted from the second-resolution color information using a parameter group predetermined by learning for estimation, and the extracted low-resolution image feature is output to the synthesis means. The color information magnifier according to claim 2, further comprising a second feature extraction means. By recursively performing a process of reducing the monochrome information of the first resolution at a predetermined reduction ratio, monochrome information of a plurality of levels having a resolution smaller than the first resolution and larger than the second resolution can be obtained. Equipped with a size reduction means to generate
With the color information of the second resolution and the monochrome information of the minimum level resolution generated as initial values, the same resolution as the monochrome information is obtained from the estimated color information and the monochrome information having a resolution larger than the color information. The color according to any one of claims 1 to 5, wherein the process of estimating the high-resolution color information having the first resolution is performed by recursively performing the process of estimating the color information having the first resolution. Information expander. The color information magnifier according to any one of claims 1 to 6.
A reduction device that generates the monochrome information of the second resolution by performing a process of reducing the monochrome information of the first resolution.
The color information of the second resolution is obtained by extracting a low-resolution image feature amount from the monochrome information of the second resolution generated by the reducer using a parameter group predetermined by learning for estimation. Equipped with a low resolution color information estimator to estimate
The color information magnifier uses the second resolution color information estimated by the low resolution color information estimator and the first resolution monochrome information input by bypassing the reducer. A color information estimator that performs processing for estimating the high-resolution color information. The color information magnifier according to any one of claims 1 to 6.
Low-resolution color information that estimates the second-resolution color information by extracting low-resolution image features from the first-resolution monochrome information using a parameter group predetermined by learning for estimation. Equipped with an estimator,
The color information magnifier includes the second resolution color information estimated by the low resolution color information estimator, the first resolution monochrome information input by bypassing the low resolution color information estimator, and the like. A color information estimator that performs processing for estimating the high-resolution color information using the above. A color information enlargement program for causing a computer to function as the color information magnifier according to any one of claims 1 to 6. A color information estimation program for operating a computer as the color information estimator according to claim 7.

Images