JP2019117558A - Color information estimation model generating device, image colorization device and programs for the same - Google Patents

Abstract

To provide a color information estimation model generating device for generating a color information estimation model for estimating color information from a monochrome image.SOLUTION: A color information estimation model generating device 1 comprises: information separation means 10 for separating a color image into color information and monochrome information; boundary learning data generation means 12 for selecting a pixel from a boundary having a larger change in color than a predetermined threshold in the separated color information at random, and generating boundary information indicative of a position thereof; and model learning means 13 for learning a neural network, using the monochrome information and the boundary information as input and the color information separated in the information separation means 10 as correct data, and generating a color information estimation model M.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a color information estimation model generation device for coloring monochrome images, an image colorization device, and programs therefor.

  In recent years, methods have been proposed for estimating color information in a monochrome image (black and white image) by using a so-called machine learning technology (see Non-Patent Document 1 and Non-Patent Document 2). The method of estimating color information using such a machine learning technique is premised on preparing a vast amount of color images in which various objects are captured.

  In this color information estimation method, when learning an estimation model for estimating color information, for example, an enormous number of color images are input to a model configured by a neural network or the like. Then, in this estimation method, the correspondence between the monochrome image and the color information corresponding to the monochrome image is learned as a parameter of an estimation model by machine learning technology. Then, this estimation method estimates color information with respect to the monochrome image given as input from the correspondence between the monochrome image and the color information using the learned estimation model, and generates a color image.

  However, such a color information estimation method using only monochrome images as input learns an estimation model so as to output an average color of an object included in learning data. Therefore, this estimation method estimates similar colors for objects with many color variations such as clothes, and can not generate a color image of natural colors.

Therefore, conventionally, there has been proposed a method using not only monochrome images but also color designation information by the user as input to the estimation model of color information (see Non-Patent Document 3). This method learns that the object is colored with the specified color by giving the information indicating the color desired to be colored by the user for the object on the monochrome image, and the monochrome image is colored. Generate an image.
By this method, even for an object with many color variations, such as clothes, it is possible to perform coloring desired by the user. Further, this method can generate a more accurate color image from a monochrome image when the correct color is known in advance by historical fact or the like.

Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. “Let there be Color! SIGGRAPH), 35 (4): 110, 2016. Richard Zhang, Phillip Isola, and Alexei A. Efros. “Colorful Image Colorization.” In ECCV 2016. Z. Zhang, J. Zhu, P. Isola, X. Geng, A. S. Lin, T. Yu, A. A. A. A. A. A. A. A. A. A. A. A. A. A. A. A. A. A. A. A. A. A. A. A. A. A. A. A. E. f.

The method of coloring a monochrome image using color designation information according to the user's instruction described above can not directly control to which region of the image the color designated by the user is to be colored. Therefore, this method has a problem that an area different from the user's intention may be colored with a designated color.
For example, even if an instruction to color a certain clothes on a monochrome image in red is given, the trees appearing adjacent to the clothes may be colored in red. Also in this case, although it is possible to specify the correct color (for example, brown) for the tree, it is necessary to specify coloring for the entire image, which requires much time and effort.

  Also, the conventional method has a problem that even if colors are specified for many objects on the image, it may not be possible to color correctly around the boundaries of the objects. In particular, it is difficult to color different colors for each area in an area where the change in luminance value is small, which is an area of substantially the same color on a monochrome image.

  The present invention has been made in view of the above problems, and is a color information estimation model generating apparatus capable of separating colors at the boundary of an area on an image when coloring a monochrome image. An object of the present invention is to provide a coloring apparatus and a program for the same.

  In order to solve the above problem, a color information estimation model generation apparatus according to the present invention is a color information estimation model generation apparatus for generating a color information estimation model for estimating color information from a monochrome image, which comprises: , Boundary learning data generation means, and model learning means.

In such a configuration, the color information estimation model generation device separates the color image as learning data into monochrome information as lightness component and color information as color components other than the lightness component by the information separating means. The monochrome information separated from the color image becomes learning data of a monochrome image for learning a color information estimation model.
Then, the color information estimation model generation apparatus randomly selects a pixel from the boundary where the change in color is larger than a predetermined threshold in the color information separated by the information separation unit by the boundary learning data generation unit, and the position thereof Generate boundary information to indicate. This boundary information is learning data for the user to indicate the position to be the boundary on the monochrome image. In addition, the reason for randomly selecting the pixel to be the boundary is to make the color information estimation model learn that the user designates a part of the boundary.

The color information estimation model generation apparatus is configured by a neural network by inputting monochrome information and boundary information by the model learning means, and learning a neural network using the color information separated by the information separation means as correct data. Generate a color information estimation model.
Thus, the color information estimation model generation device generates a color information estimation model capable of estimating color information from monochrome information and boundary information.

  Further, in order to solve the above problem, the image coloring apparatus according to the present invention estimates color information from a monochrome image using a color information estimation model generated by the color information estimation model generating apparatus to generate a color image. The image colorization apparatus includes: a boundary instruction unit; a boundary information generation unit; a color information estimation unit; and a color image synthesis unit.

In such a configuration, the image colorization apparatus inputs the position of the boundary in the monochrome image to be colored as the boundary instruction by the boundary instruction means. This allows the user to explicitly indicate the boundary even if the luminance on the monochrome image is substantially the same.
Then, the image colorization apparatus generates boundary information in which the information indicating the boundary is set at the pixel position designated by the boundary instruction, corresponding to the pixel position of the monochrome image, by the boundary information generation means.

Then, the image colorization apparatus estimates color information from the monochrome image, which is monochrome information, and the boundary information using the color information estimation model by the color information estimation means.
Then, the image colorization apparatus generates a color image by combining the color information estimated by the color information estimation means with the input monochrome image by the color image combining means.

The present invention can also be realized by a color information estimation model generation program for causing a computer to function as the color information estimation model generation device.
The present invention can also be realized by an image colorization program for causing a computer to function as the image colorization device.

The present invention exhibits the following excellent effects.
According to the color information estimation model generation apparatus of the present invention, it is possible to learn a color boundary with respect to a color image, and to generate a color information estimation model in which a different color is learned according to the boundary.
Further, according to the image colorization apparatus according to the present invention, it is possible to generate a more accurate color image from a monochrome image by a simple operation in which the user explicitly gives color boundaries on the monochrome image.

It is a schematic diagram for explaining an outline of the present invention. It is a block diagram showing composition of a color information presumption model generation device concerning an embodiment of the present invention. It is explanatory drawing for demonstrating the data structure of the color designation | designated learning data (color designation | designated information) which a color | collar designation | designated learning data production | generation means produces | generates. It is a figure which shows an example of the data of color designation information. It is a figure which shows an example of the boundary learning data which a boundary learning data production | generation means produces | generates, Comprising: (a) shows the example of data after selecting a boundary at random after boundary detection. FIG. 5 is a structural diagram of a neural network showing an example of a color information estimation model. It is a flowchart which shows operation | movement of the color information estimation model production | generation apparatus which concerns on embodiment of this invention. FIG. 1 is a block diagram showing a configuration of an image coloring apparatus according to an embodiment of the present invention. It is a flowchart which shows operation | movement of the image coloring apparatus based on embodiment of this invention. In the image colorization apparatus according to the embodiment of the present invention, it is a view schematically showing how a color image to be generated changes in accordance with a color specification and a boundary specification, wherein (a) is a color specification specification , (B) indicates color designation indication, (b) indicates color designation indication, (c) indicates boundary designation (left), (d) indicates color designation, (b) indicates boundary designation (right) Show.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
<Overview of the Invention>
First, the outline of the present invention will be described with reference to FIG. FIG. 1 is a schematic diagram for explaining an overview of the present invention.
As shown in FIG. 1, the color information estimation model generation apparatus 1 according to the present invention generates a color information estimation model M. The image colorization device 2 according to the present invention estimates color information of a monochrome image Pm using the color information estimation model M generated by the color information estimation model generation device 1 to generate a color image Pc. is there.

  The color information estimation model M is a neural network that inputs a monochrome image Pm and information indicating a color designation instruction Ic and a boundary instruction Ib that are instructions from the user, and outputs color information of the monochrome image Pm.

The monochrome image Pm is a digital image having information of lightness components scanned from a black and white photograph, a film or the like.
The color specification instruction Ic is an instruction to specify a color at a certain position on the image. For example, when the color of the correct answer of a certain object is known in advance for the monochrome image Pm, the color specification instruction Ic specifies the position of the object and the color.
The boundary instruction Ib is an instruction for specifying a position indicating a color boundary. For example, the boundary designation Ib designates the boundary between a person and the background, the boundary between overlapping objects, and the like.
The color information estimation model M learns to apply the designated color to the area including the position designated by the color designation instruction Ic, and further, the color is separated at the boundary designated by the boundary designation Ib. Was learned to

The image colorization apparatus 2 estimates color information so that the area including the position designated by the color designation instruction Ic is colored with the designated color by the color information estimation model M. The color information is information of color components (first color component, second color component) obtained by removing the lightness component of the color space from the color image.
Further, the image colorization apparatus 2 estimates color information such that colors are classified by the color information estimation model M at the boundary indicated by the boundary instruction Ib.
Thus, the present invention can generate a more accurate color image Pc by estimating color information for the monochrome image Pm.
Hereinafter, the color information estimation model generation device 1 and the image colorization device 2 according to the present invention will be described in more detail.

<Color Information Estimation Model Generation Device: Configuration>
First, the configuration of the color information estimation model generation device 1 will be described with reference to FIG. FIG. 2 is a block diagram showing the configuration of the color information estimation model generation apparatus 1.
As shown in FIG. 2, the color information estimation model generation apparatus 1 includes an information separation unit 10, a color designation learning data generation unit 11, a boundary learning data generation unit 12, a model learning unit 13, and a model storage unit 14. And.

The color information estimation model generation device 1 generates a color information estimation model M from a plurality of color images Po.
The color image Po is learning data for learning the color information estimation model M, and is a color image prepared in advance in a large amount (for example, 2,000,000 sheets or more).
The color image Po is an image represented in a predetermined color space with the same image size (vertical H pixels × horizontal W pixels) as the monochrome image Pm (see FIG. 1) to be colored. Here, it is assumed that the color image Po is an image represented in Lab color space.

The information separating means 10 separates the input color image Po into monochrome information and color information.
Specifically, the information separation unit 10 includes a color image Po configured by three channels of Lab color space, an L channel (brightness component) which is monochrome information, and color information (color components other than the brightness component). It is separated into a and b channels (first color component, second color component).
The information separation unit 10 outputs the separated monochrome information (L channel) to the model learning unit 13.
Further, the information separation unit 10 outputs the separated color information (a, b channels) to the color specification learning data generation unit 11, the boundary learning data generation unit 12 and the model learning unit 13.

  The color specification learning data generation means 11 generates learning data (color specification learning data) of a color specification instruction Ic (see FIG. 1) which instructs the user to indicate the position on the monochrome image and the color corresponding to the position. is there. The color specification learning data generation unit 11 includes an instruction information addition unit 110 and a random color specification setting unit 111.

The instruction information adding means 110 adds information (instruction information) indicating the presence or absence of a color specification instruction to each element (pixel) of data of color information (channels a and b).
As shown in FIG. 3, when the size of the image to be subjected to color estimation is H (the number of vertical pixels) × W (the number of horizontal pixels), the instruction information adding unit 110 adds color information to channels Ch1 and Ch2. A new channel Ch3 is added to generate color designation information DIc having a data structure of H × W × 3 channels.

The channel Ch1 and the channel Ch2 are color information (a and b channels of Lab color space) separated by the information separation unit 10. The channel Ch1 is composed of the value of a, and the channel Ch2 is composed of the value of b.
The channel Ch3 is information (instruction information) indicating the presence or absence of a color designation instruction for a pixel. For example, “1” is set to each element of the channel Ch3 when the color designation is instructed at the position (pixel position) of the corresponding element, and “0” is set when the color designation is not instructed. Here, the instruction information adding unit 110 sets, as an initial value, an indication (for example, “1”) of an instruction for color specification to all elements of the channel Ch3 (instruction information).

The color designation information DIc thus initialized becomes color designation learning data (complete color designation learning data) in which color designation is designated for all pixel positions.
The instruction information adding unit 110 outputs data (color designation information) obtained by adding the instruction information (Ch 3) to the color information (Ch 1 and Ch 2) to the random color designation setting unit 111.

The random color designation setting unit 111 sets information for instructing color designation at a randomly selected pixel position in the color designation information generated by the instruction information addition unit 110.
Specifically, the random color specification setting unit 111 specifies the number of pixels for which color specification is designated (color specification), with a value randomly selected in the range from “0” to the total number of pixels (vertical H pixels × horizontal W pixels). The number of pixels) The random color specification setting unit 111 may calculate the number of color specification pixels, for example, as the number according to the geometric distribution of the predetermined probability.
Then, the random color designation setting unit 111 randomly selects pixel positions (color designation pixel positions) corresponding to the number of color designation pixels on the image (H × W). Note that the random color specification setting unit 111 may calculate the color specification pixel position according to a predetermined average, a two-dimensional normal distribution of variance, or a uniform distribution.

  Here, since the color designation instruction (for example, “1”) is set to all elements (pixels) of the instruction information (Ch 3; FIG. 3) as the initial value, the random color designation setting unit 111 “0” is set in the element position of the instruction information (Ch3) corresponding to the pixel position other than the specified pixel position. In addition, the random color specification setting unit 111 sets the value of “a” at the pixel position of color information (a, b channels [Ch1, Ch2]) corresponding to the element position where “0” is set in the color specification specification information (Ch3). The values of and b are set to a predetermined fixed value ("0").

For example, as shown in FIG. 4, the random color specification setting means 111 sets coordinates (3, 1), (1, 3), (3, 3) which are (x, y) coordinates on the two-dimensional data of the channel Ch3. , (2, 4) are assumed to be color designation pixel positions randomly selected, and “0” is set to the other coordinates. In that case, the random color specification setting unit 111 sets “0” to the element value of the channels Ch1 and Ch2 corresponding to the element position where “0” is set in the channel Ch3.
As a result, the color designation learning data generation unit 11 can obtain valid color information (a, b channels [Ch1, Ch2) only for pixels for which “1” is set at the element position randomly set in the designation information (Ch3). It is possible to generate color designation learning data having a value of Ch2]).
The color designation learning data generation means 11 outputs the generated color designation learning data to the model learning means 13.

  The boundary learning data generation unit 12 generates learning data (boundary learning data) of the boundary instruction Ib (see FIG. 1) which indicates the boundary of the color on the monochrome image by the user. The boundary learning data generation unit 12 includes a boundary detection unit 120 and a random boundary selection unit 121.

The boundary detection means 120 detects, from the color information (channels a and b), whether or not the pixel is a pixel serving as a color boundary for each pixel.
When the image size is H (the number of vertical pixels) × W (the number of horizontal pixels), the boundary detection unit 120 generates boundary information for one channel of H × W × 1. The boundary information is composed of values (for example, “1” for the boundary and “0” for the boundary) indicating whether or not the boundary is detected as the boundary for each element (pixel).

The boundary detection means 120 detects the boundary where color changes greatly in each of the a channel and the b channel, and at least one of the channels is an element of the pixel position detected as a boundary as “1” as a boundary in both channels. Boundary information is generated by setting “0” to the element of the pixel position not detected.
The boundary detection means 120 detects a boundary using a general edge detection algorithm (for example, the Canny algorithm) to detect a pixel whose difference in value with surrounding pixels is larger than a predetermined threshold value as a boundary pixel. do it.
The boundary detection unit 120 outputs boundary information indicating the pixel position of the boundary to the random boundary selection unit 121.

The random boundary selection unit 121 randomly selects an element set as a boundary in the boundary information generated by the boundary detection unit 120.
Specifically, the random boundary selection unit 121 specifies the number of pixels (border indication pixel number) which designates as a boundary a value randomly selected in a range from “0” to the number of pixels set as the boundary (the number of boundary pixels). And). The random boundary selection unit 121 may calculate the number of boundary designation pixels as, for example, the number according to the geometric distribution of the predetermined probability.
Then, the random boundary selection unit 121 randomly selects pixels corresponding to the number of boundary designation pixels from the pixels set as the boundary in the boundary information. The random boundary selection unit 121 sets “0” at pixel positions other than the selected boundary designation pixel.

For example, as shown in FIG. 5 (a), when “1” is set in the pixel position of the boundary after the boundary detection, the random boundary selection means 121 is “1” as shown in FIG. 5 (b). Is set, and “0” is set to the other pixel position leaving only “1” of the pixel position randomly selected from the pixel. In FIG. 5, dotted lines indicate boundaries on the color image.
By this, the boundary learning data generation unit 12 can generate boundary learning data that indicates the boundary at a part of pixel positions of the boundary detected from the color information.
The boundary learning data generation means 12 outputs the generated boundary learning data to the model learning means 13.

The model learning means 13 learns a color information estimation model for estimating color information. The model learning unit 13 generates monochrome learning information (L channel) of one channel separated by the information separating unit 10, color designation learning data of three channels generated by the color designation learning data generation unit 11, and boundary learning data generation. The boundary learning data of one channel generated by the means 12 is input as learning data.
Then, the model learning unit 13 sets the color information (channels a and b) of the two channels separated by the information separating unit 10 as correct data, and eliminates the error with the output of the color information estimation model M (error is predetermined The color information estimation model M (specifically, its model parameters) is learned using an error back propagation method so as to be less than or equal to
The model learning means 13 writes and stores the learned color information estimation model M in the model storage means 14.

Here, a configuration example of the color information estimation model M learned by the model learning unit 13 will be described with reference to FIG.
As shown in FIG. 6, the color information estimation model M can be configured by a neural network including a convolutional layer.

  As shown in FIG. 6, the color information estimation model M inputs, as an input layer I, a monochrome image of one channel (1 ch), color designation information of three channels, and boundary information of one channel.

Also, the color information estimation model M has a plurality of convolution layers (C1 to C6) as the hidden layer (intermediate layer) H.
The color information estimation model M is a channel defined in advance by a plurality of kernels (convolution filters) in the convolution layer C1 and having a monochrome image (1 ch) in the same image size (vertical H pixels × horizontal W pixels) as the monochrome image. Convolve into a number N (eg, N = 64) of data.

Further, the color information estimation model M convolutes the color designation information (3 ch) into data of the number N of channels of H × W pixels in the convolution layer C 2 by a plurality of kernels. In addition, the color information estimation model M convolutes the boundary information (1 ch) into data of the number N of channels of H × W pixels in the convolution layer C3 by a plurality of kernels.
Then, the color information estimation model M adds the information (monochrome image, color designation information, boundary information) convoluted to Nch by the adder S. It is also possible to use concatenated information of 3 × N channels without using the adder S.

Then, the color information estimation model M performs convolution using a plurality of kernels sequentially by the convolution layers C4 and C5.
Finally, the color information estimation model M convolves Nch information into 2ch information by the convolution layer C6.
The color information estimation model M outputs information of two channels as an output layer O as channels a and b which are color information.
The model learning unit 13 learns kernels and weights of the convolution layers as model parameters.

  The structure of the neural network of the color information estimation model M is not limited to this embodiment. For example, the number of convolution layers and the number of parameters (kernel size, number of channels) may be changed, or layers other than convolution layers (for example, all bonding layers, normalization layers) may be added.

Returning to FIG. 2, the configuration of the color information estimation model generation device 1 will be described.
The model storage unit 14 stores the color information estimation model M learned by the model learning unit 13. The model storage means 14 can be configured by a general storage medium such as a semiconductor memory. The color information estimation model M after learning is used in the image colorization device 2 (see FIG. 1).
The color information estimation model generation apparatus 1 can be operated by a color information estimation model generation program for causing a computer to function as each of the above-described units.

As described above, by configuring the color information estimation model generation device 1, the color information estimation model generation device 1 receives the monochrome image, the color designation information, and the boundary information as input, and estimates the color information estimation model M. Can be generated by learning.
Also, when generating the color information estimation model M, the color information estimation model generation device 1 generates learning data from the color image Po without preparing a monochrome image, color designation information, and boundary information as learning data. Can.

<Color Information Estimation Model Generation Device: Operation>
Next, the operation of the color information estimation model generation apparatus 1 will be described with reference to FIG. 7 (for the configuration, refer to FIG. 2 as needed). FIG. 7 is a flowchart showing the operation of the color information estimation model generation apparatus 1.
In step S1, the information separation unit 10 separates the input color image into the a and b channels (two channels) as color information and the L channel (one channel) as monochrome information. The monochrome information separated from the color image is a monochrome image, and directly becomes learning data (monochrome image learning data) for learning the color information estimation model M.

  In step S2, the instruction information adding unit 110 of the color specification learning data generation unit 11 generates data of one channel in which the instruction information having a color specification instruction (for example, "1") is set to all elements (all pixels). , And added to the color information (a, b channels) separated in step S1. Thus, the instruction information adding unit 110 generates color specification information of three channels.

  In step S3, the random color designation setting unit 111 of the color designation learning data generation unit 11 sets "0" to an element (pixel) randomly selected in the color designation information of the three channels generated in step S2. As a result, the color specification learning data generation means 11 can only use the a and b pixels effective for the pixels of the color information (channels a and b) corresponding to the element position at which “1” is set in the instruction information on one channel. Generate color specification learning data with values.

In step S4, the boundary detection unit 120 of the boundary learning data generation unit 12 detects a boundary of a color whose color changes greatly in at least one of the a channel and the b channel which are the color information separated in step S1.
In step S5, the boundary detection unit 120 sets data of one channel (boundary information) in which “1” is set to the element at the pixel position detected as the boundary and “0” is set to the element at the pixel position not detected as the boundary. Generate

  In step S6, the random boundary selection means 121 of the boundary learning data generation means 12 sets “0” to the element selected at random among the elements for which “1” is set as the boundary information generated in step S5. Do. By this, the boundary learning data generation unit 12 generates boundary learning data indicating a boundary at a part of pixel positions of the boundary detected from the color information.

In step S7, the model learning means 13 inputs the monochrome image learning data separated in step S1, the color designation learning data generated in step S3, and the boundary learning data generated in step S6, and is the correct data. The color information estimation model M is learned so as to output the color information (a, b channels) separated in S1.
In step S8, the model learning unit 13 determines the end of learning based on whether input of a predetermined color image has ended or the amount of change in estimation error due to learning has become equal to or less than a threshold.

Here, if the learning is not completed (No in step S8), the color information estimation model generation device 1 returns to step S1, inputs another color image, and continues the learning.
On the other hand, when learning is completed (Yes in step S8), in step S9, the model learning means 13 writes the learned color information estimation model M in the model storage means 14 and stores it.
By the above operation, the color information estimation model generation apparatus 1 can generate a color information estimation model for estimating color information by learning from a color image only, using a monochrome image, color designation information, and boundary information as input.

<Image colorization device: Configuration>
Next, the configuration of the image colorization device 2 will be described with reference to FIG. FIG. 8 is a block diagram showing the configuration of the image colorization device 2.
As shown in FIG. 8, the image colorization apparatus 2 includes an image input unit 20, a color designation instruction unit 21, a color designation information generation unit 22, a boundary instruction unit 23, a boundary information generation unit 24, and a model storage. A means 25, a color information estimating means 26 and a color information combining means 27 are provided.

  The image input means 20 is for inputting a monochrome image Pm to be colored. The image input unit 20 outputs the input monochrome image Pm to the color information estimation unit 26 and the color information synthesis unit 27 as monochrome information (monochrome image) of L channel (1 channel) of Lab space.

  The color designating instruction unit 21 is for inputting a position and a color at which the user wishes to designate a color on the monochrome image Pm as a color designating instruction Ic. The color designation instruction unit 21 may display the monochrome image Pm on a display device (not shown) and input the color designation instruction through an input unit such as a mouse and a keyboard.

The position of the color designation instruction Ic is a pixel position on the monochrome image Pm. A plurality of pixel positions may be provided.
The color of the color designation instruction Ic is a color corresponding to the position of the color designation instruction Ic, and is the values (ab values) of the a and b channels in the Lab space. The color of the color designation instruction Ic does not have to be the values of the a and b channels, and may be, for example, a color of another color space such as an RGB value. In that case, the color specification instructing unit 21 may perform color space conversion to obtain the values of the a and b channels in the Lab space.
The color designation instruction unit 21 outputs the designated position and color (ab value) to the color designation information generation unit 22.

  The color designating instruction unit 21 may input an instruction indicating that the color designation is not performed for all the positions. In that case, the color designation instruction unit 21 outputs an instruction indicating that color designation is not performed to the color designation information generation unit 22.

The color designation information generation unit 22 generates color designation information as an input of the color information estimation model M from the color designation instruction Ic.
The color designation information generation unit 22 designates a color having a data structure of H × W × 3 when the size of the monochrome image Pm is H (vertical pixel number) × W (horizontal pixel number) size. Generate information. This color designation information is the same as the data structure described in FIG.

The color designating information generating means 22 sets the designated a value to the element at the designated position of the channel Ch1 in the data structure shown in FIG. Set 0 ”).
In addition, the color designation information generation means 22 sets the designated b value to the element at the designated position of the channel Ch2, and sets a predetermined fixed value ("0") to the other elements of the channel Ch2.

Further, the color designation information generation means 22 sets a value ("1") indicating that there is a color designation instruction in the element at the designated position of the channel Ch3, and designates the color in the other elements of the channel Ch3. Set a value (“0”) indicating that there is no indication of.
When color designation information generation means 22 receives from color designation indication means 21 an instruction indicating that color designation is not performed for all positions, “0” is set for all elements of three channels. Just do it.
The color designation information generation means 22 outputs the generated color designation information of the three channels to the color information estimation means 26.

  The boundary designating means 23 is for inputting, as a boundary designating Ib, a position on the monochrome image Pm where the user wishes to designate as a boundary. The boundary instructing unit 23 may display the monochrome image Pm on a display device (not shown) and input the boundary instruction through an input unit such as a mouse.

The boundary designation Ib is a pixel position indicating a boundary on the monochrome image Pm. This pixel position may be one point or plural points. For example, if the designated position is two points, a straight line connecting those points is the boundary. Also, if the instructed position is three or more points, a curve connecting those points is the boundary. Further, boundary designation Ib may be a combination of these.
The boundary instructing unit 23 outputs the instructed position to the boundary information generating unit 24.
The boundary instructing unit 23 may input an instruction indicating that no boundary is to be specified. In that case, the boundary instructing unit 23 outputs, to the boundary information generating unit 24, an instruction indicating that the boundary specification is not performed.

The boundary information generation means 24 generates boundary information to be input of the color information estimation model M from the boundary instruction Ib.
When the size of the monochrome image Pm is H (the number of vertical pixels) × W (the number of horizontal pixels), the boundary information generation unit 24 has boundary information having a data structure of H × W × 1 channel. Generate
The boundary information generation unit 24 sets a value (“1”) indicating that the pixel position is a boundary to an element corresponding to the pixel position specified by the boundary instruction Ib, and does not indicate to other elements as a boundary. Set a value (“0”) indicating.

When two points are indicated by the boundary indication Ib, the boundary information generation means 24 sets a value (“1”) indicating an edge as an element on a straight line connecting the indicated two points. . Also, when three or more points are indicated by the boundary indication Ib, the boundary information generation means 24 approximates a curve connecting the indicated three or more points with, for example, a spline curve, and the element on the curve is a boundary Set a value (“1”) to indicate that.
The boundary information generation unit 24 may set “0” to all elements of the boundary information of one channel when an instruction indicating that the boundary specification is not performed is input from the boundary instruction unit 23.
The boundary information generation unit 24 outputs the generated boundary information of one channel to the color information estimation unit 26.

The model storage unit 25 stores the color information estimation model M generated by the color information estimation model generation apparatus 1 described with reference to FIG. The model storage means 25 can be configured by a general storage medium such as a semiconductor memory.
The color information estimation model M is referred to by the color information estimation means 26.

The color information estimation means 26 inputs a monochrome image, color designation information and boundary information, and estimates color information (channels a and b) using the color information estimation model M.
The color information estimation unit 26 receives a monochrome image of one channel from the image input unit 20. Further, the color information estimation unit 26 receives color designation information of three channels from the color designation information generation unit 22. Further, the color information estimation unit 26 receives the boundary information of one channel from the boundary information generation unit 24.

The color information estimation means 26 uses the color information estimation model M shown in FIG. 6 to input monochrome image, color designation information and boundary information to the input layer I, and output color information of two channels output from the output layer O (A, b channels) are generated by calculation.
The color information estimation means 26 outputs the estimated color information (a, b channels) to the color information combining means 27.

The color information combining unit 27 sets the monochrome image input from the image input unit 20 as the L channel of the Lab color space, and combines the color information (channels a and b) estimated by the color information estimating unit 26.
The color information combining unit 27 outputs data of three channels of L channel, a channel and b channel as a color image Pc of Lab color space having a data structure of three channels.
The image coloring apparatus 2 can be operated by an image coloring program for causing a computer to function as each of the above-described units.

  By configuring the image colorization apparatus 2 as described above, the image colorization apparatus 2 estimates color information for the monochrome image by the user specifying the color and instructing the boundary, and the color image is estimated. Can be generated.

<Image colorization device: Operation>
Next, the operation of the image coloring apparatus 2 will be described with reference to FIG. 9 (for the configuration, refer to FIG. 8 as needed). FIG. 9 is a flowchart showing the operation of the image colorization device 2.
In step S10, the image input unit 20 inputs a monochrome image to be colored. The monochrome image becomes an input to the color information estimation model M and also becomes an L channel of the Lab color space which constitutes a color image to be finally generated.

In step S11, the color designating instruction unit 21 inputs, as a color designating instruction, a position and a color at which the user desires to designate the color on the monochrome image.
In step S12, the color designation information generation unit 22 generates color designation information of three channels serving as an input of the color information estimation model M from the color designation instruction input in step S11. The color designation information generation unit 22 determines that the color designation is designated at the pixel position where the color designation is designated, at the pixel position at which the color designation is designated, the a value and the b value of the Lab color space which is the designated color information. Color designation information is generated by setting the indicated value ("1") to the corresponding elements (pixels) of three channels. The color designation information generation unit 22 sets a value (“0”) indicating that color designation is instructed for all elements corresponding to pixel positions for which color designation is not instructed.

In step S13, the boundary instructing unit 23 inputs, as a boundary instruction, a position where the user wishes to specify as a boundary on the monochrome image.
In step S14, the boundary information generation unit 24 generates boundary information of one channel, which is an input of the color information estimation model M, from the boundary instruction input in step S13. The boundary information generation means 24 sets the element corresponding to the pixel position specified by the boundary instruction Ib to a value indicating that it is a boundary (“1”) and a value indicating that other elements are not a boundary (“0”) Boundary information is generated by setting).

  In step S15, the color information estimation unit 26 uses the color information estimation model M to obtain color information from the monochrome image input in step S10, the color designation information generated in step S12, and the boundary information generated in step S14. Estimate (a, b channels).

In step S16, the color information combining unit 27 generates the color image by setting the monochrome image input in step S10 as the L channel of the Lab color space and combining the color information (a, b channels) estimated in step S15. Do.
By the above operation, the image colorization device 2 generates an accurate color image by inputting a color designation and a boundary instruction from the user using the color information estimation model generated by the color information estimation model generation device 1 can do.

<Example of color image generation of image colorization device>
Next, with reference to FIG. 10, how the color image generated by the image colorization device 2 changes in accordance with the color designation and the boundary designation will be schematically described.
FIG. 10 shows the colored state of the color image when the color designation instruction and the boundary instruction are respectively given. In FIG. 10, the figure on the left side of the arrow (here, a bowl) indicates a monochrome image, and the figure on the right of the arrow (a bowl) schematically indicates a color image Pc (Pc1 to Pc4) in which the monochrome image is colored. In FIG. 10, the difference in color is indicated by the difference in hatching. In addition, the band-shaped area Z at the center of the teacup is an area in which the change in luminance value is small.

FIG. 10A shows the case where neither the color designation instruction nor the boundary instruction is present.
In this case, since the change in luminance value of the band-shaped area Z at the center of the teacup is small, the band-shaped area Z is colored in substantially the same color as in the color image Pc1.
FIG. 10B shows the case where there is a color designation instruction and no boundary designation. Also, here, the color A is specified on the left side of the band-like region, and the color B is specified on the right side.
In this case, as in the color image Pc2, the band-shaped area Z is colored with the designated colors A and B. However, the boundary between the color A and the color B is a blurred color because the change in the luminance value of the strip area Z is small.

FIG. 10C shows the case where the color designation instruction and the boundary instruction are both present, and the boundary C is indicated to the left.
In this case, as in the color image Pc3, in the band-shaped area Z, the designated colors A and B are divided and colored at the designated left boundary.
FIG. 10D shows the case where the color designation instruction and the boundary instruction are both present, and the boundary D is indicated to the right.
In this case, as in the color image Pc4, in the band-shaped area Z, the designated colors A and B are separately colored at the designated right boundary.

  Here, an example in which the boundary is indicated by a straight line is shown. This boundary may be designated by a point. For example, when the boundary is designated by a point in FIGS. 10C and 10D, the color image generated is as shown by a color image Pc2 in FIG. 10B. The boundary between color A and color B is a blurred color, but the boundary moves to a designated point.

<Modification>
As mentioned above, although embodiment of this invention was described, this invention is not limited to this embodiment.
(Modification 1)
Here, as shown in FIG. 6, the color information estimation model M is a model for estimating color information (a, b channels) with three inputs of a monochrome image, color designation information, and boundary information.
However, color designation information is not essential in the present invention.
In that case, the color designation learning data generation means 11 is omitted from the color information estimation model generation device 1 shown in FIG. 2, and the model learning means 13 inputs the color information estimation model M into two, monochrome image and boundary information. You should learn. Further, the color designation instruction means 21 and the color designation information generation means 22 are omitted from the image colorization device 2 shown in FIG. 8, and the color information estimation means 26 receives color information (monochrome image and boundary information) by two inputs. a, b channels) may be estimated.

  Of course, the color information estimation model M may be a model that estimates color information by adding other information besides monochrome images, color designation information, and boundary information. For example, as input of the color information estimation model M, edge data of a monochrome image, a genre (animation, drama, news, etc.) or the like may be used.

  By inputting edge data of a monochrome image, the color information estimation model M becomes a model capable of accurately classifying colors by the edge of the monochrome image. Further, by using the genre as an input, the color information estimation model M becomes a model capable of estimating the hue specific to the genre.

For example, when edge data of a monochrome image is used as the input of the color information estimation model M, the color information estimation model generation apparatus 1 shown in FIG. 2 may be provided with edge detection means for detecting an edge. Then, the edge detection means may detect an edge from the L channel which is monochrome information separated by the information separation means 10, and output edge data (edge image) to the model learning means 13 as information of one channel.
In addition, the image color conversion device 2 shown in FIG. 8 may be provided with edge detection means. Then, the edge detection means may input the monochrome image input by the image input means 20 to detect an edge, and output edge data (edge image) to the color information estimation means 26 as information of one channel.

  Also, for example, when the genre of monochrome image is input to the color information estimation model M, when the type of genre to be handled is Nj in the color information estimation model generation device 1 and the image colorization device 2, data for Nj channels A genre information generation means for generating (genre information) may be provided, and the genre information may be output to the model learning means 13 and the color information estimation means 26. In this genre information, one channel corresponds to one genre. And, the genre information generation means has a value (for example, “1”) indicating that the genre is set to only the channel corresponding to the corresponding genre, and a value indicating that the genre is not set to the other channels. By setting (for example, “0”), genre information is generated. By this, even when a plurality of genres correspond to a monochrome image, it can be used as an input of the color information estimation model M.

(Modification 2)
Further, here, as shown in FIG. 6, the color information estimation model M is individually convoluted into three inputs of a monochrome image (1 channel), color designation information (3 channels), and boundary information (1 channel). After that, the model was added (or connected).
However, the color information estimation model M may connect these three inputs and perform convolution on data of 5 channels. Of course, it is not necessary to connect all three inputs, and arbitrary two of them may be connected, folded, and then each added (or connected). For example, convolution is performed by dividing into one channel of a monochrome image and four channels in which color designation information and boundary information are connected.

(Modification 3)
Also, in this case, the color image is an image of the Lab color space. However, the color image is not limited to an image in the Lab color space, and may be an image in another color space (for example, an RGB color space).

For example, when color information estimation model M is generated using color images in the RGB color space as learning data in color information estimation model generation apparatus 1 shown in FIG. It should be provided. Then, the color space conversion unit may convert a color image in the RGB color space into a color image in the Lab color space, and output the color image to the information separation unit 10.
Further, in the image colorization apparatus 2 shown in FIG. 8, the color information conversion means may be provided downstream of the color information combining means 27 for converting a color image in the Lab color space into a color image in the RGB color space.

DESCRIPTION OF SYMBOLS 1 color information estimation model generation apparatus 10 information separation means 11 color designation learning data generation means 110 instruction information addition means 111 random color specification setting means 12 boundary learning data generation means 120 boundary detection means 121 random boundary selection means 13 model learning means 14 model Storage means 2 image colorization device 20 image input means 21 color designation instruction means 22 color designation information generation means 23 boundary designation means 24 boundary information generation means 25 model storage means 26 color information estimation means 27 color information synthesis means

Claims (7)

A color information estimation model generation apparatus for generating a color information estimation model for estimating color information from a monochrome image, comprising:
Information separation means for separating a color image which is learning data into monochrome information which is a lightness component and color information which is a color component other than the lightness component;
Boundary learning data generation means for randomly selecting pixels from the boundaries where color change is larger than a predetermined threshold in color information separated by the information separation means, and generating boundary information indicating the position;
Model learning means for generating the color information estimation model by inputting the monochrome information and the boundary information and learning a neural network using the color information separated by the information separation means as correct data;
An apparatus for generating a color information estimation model, comprising: The information separation unit separates monochrome information of the color image as a lightness component of a color space, and separates color information of the color image as a first color component and a second color component of the color space.
The boundary learning data generation means
Boundary means for detecting color boundaries in each of the first color component and the second color component;
The boundary information is generated randomly by setting information indicating a boundary at a pixel position randomly selected from a pixel position detected as a boundary at least one of the first color component and the second color component. Boundary selection means,
The color information estimation model generation apparatus according to claim 1, comprising: In the color information separated by the information separating means, the instruction information in which the information indicating the presence or absence of the instruction for specifying the color is randomly set for each pixel is added to the color information, and the pixel of the instruction information without the instruction for the color specification And color specification learning data generation means for generating color specification information in which a predetermined fixed value is set to the pixel of the color information corresponding to
The color information estimation model generating apparatus according to claim 1 or 2, wherein the model learning means learns the neural network by inputting the monochrome information, the boundary information, and the color designation information. . An image colorization apparatus for estimating color information from a monochrome image using the color information estimation model generated by the color information estimation model generation apparatus according to claim 1 or 2, and generating a color image,
Boundary designation means for inputting the position of the boundary in the monochrome image as the boundary designation;
Boundary information generation means for generating boundary information in which information indicating a boundary is set at the pixel position instructed by the boundary instruction corresponding to the pixel position of the monochrome image;
Color information estimation means for estimating color information from the monochrome image as monochrome information and the boundary information using the color information estimation model;
Color image combining means for generating the color image by combining the color information estimated by the color information estimation means with the monochrome image;
An image colorization apparatus comprising: An image colorization apparatus for estimating color information from a monochrome image using the color information estimation model generated by the color information estimation model generation apparatus according to claim 3 to generate a color image,
Boundary designation means for inputting the position of the boundary in the monochrome image as the boundary designation;
Boundary information generation means for generating boundary information in which information indicating a boundary is set at the pixel position instructed by the boundary instruction corresponding to the pixel position of the monochrome image;
A color designation instruction input unit that inputs, as a color designation instruction, a position and a color for performing color designation in the monochrome image;
Color designation information generation means for generating color designation information in which instruction information indicating presence / absence of the color designation instruction is set corresponding to a pixel position of the monochrome image, and color information for which a color designated color is set;
Color information estimation means for estimating color information from the monochrome image which is monochrome information, the boundary information, and the color designation information using the color information estimation model;
Color image combining means for generating the color image by combining the color information estimated by the color information estimation means with the monochrome image;
An image colorization apparatus comprising:   A color information estimation model generation program for causing a computer to function as the color information estimation model generation apparatus according to any one of claims 1 to 3.   An image colorization program for causing a computer to function as the image colorization device according to claim 4 or 5.

Images