JP2019128889A - Image information converter and program therefor - Google Patents

Abstract

To provide an image information converter that can accurately perform image conversion.SOLUTION: An image information converter is configured by connecting multi-scale converters 10, and the multi-scale converters 10 each include: a feature quantity creation unit 20 that creates, from image information with m (m is an integer of 1 or more) resolutions, a feature quantity for one resolution predetermined through convolution using a learned parameter; an image information creation unit 30 that creates, from the feature quantity created by the feature quantity creation unit, image information with n (n is an integer of 1 or more) resolutions using the learned parameter; and an image composition unit 40 that composites image information having the same resolution, of input image information, on the image information created by the image information creation unit.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image information converter for converting image information into target image information by a neural network and a program thereof.

In recent years, machine learning techniques called deep learning and deep neural networks (DNN) have been actively researched and developed.
DNN is a converter (network) that converts complex numerical information by inputting one or more values and combining a large number of simple units called neurons that output one value. Each neuron has a parameter that can be changed inside, and by adjusting the parameter appropriately, it is possible to construct a transducer according to the purpose. For example, a color converter or the like that inputs a pixel value of a monochrome image and outputs a pixel value of a color image. This parameter adjustment is called learning, and this learning is generally performed by a method called an error back propagation method.

  The error back-propagation method makes possible, as much as possible, the error of the output of the converter whose network structure is defined in advance (for example, the difference between the pixel value of the color image output by the color converter and the pixel value of the correct color image given by human). This is a method of updating the parameters so as to be small. Since the parameters obtained by this method are not optimal solutions but local solutions, how the network structure is determined greatly affects the final performance of the converter. This is because, in general, the more complex the combination of neurons, the higher the possibility that better conversion performance can be obtained, but instead, it becomes difficult to learn the parameter that brings the local solution closer to the optimal solution.

  Therefore, a convolution network in which a large number of neuron structures called convolution layers are stacked is often used in recent years. In general, in the convolution layer, neurons are arranged in a three-dimensional (vertical × horizontal × channel) arrangement like an image, and the neuron which is the Nth layer is the neuron of the (N−1) th layer. The number of parameters to be learned can be reduced because it is connected only to neurons that are close to each other in spatial position. By layering the structure of this convolution layer, a converter with high performance can be configured even with a small number of parameters.

For example, it has been reported that a highly accurate converter can be constructed by a network (FCN: Fully Convolutional Networks) shown in FIG. 15 as a network structure for inputting an image and outputting a converted image (Non-patent Document 1). reference).
The FCN shown in FIG. 15 extracts features by gradually reducing the spatial size of the image from the shallow layer to the deep layer by convolution in the convolution layer (Conv) C, and then from the deep layer It has a structure that extracts a region in an image by gradually expanding toward a shallow layer. Here, the convolution layer C having a smaller image spatial size is referred to as a deep layer.

Further, for example, as another network structure for inputting an image and outputting a converted image, it has been reported that a more accurate converter can be constructed by the network (U-Net) shown in FIG. Patent Document 2).
Similar to FCN, U-Net shown in FIG. 16 is a network in which the spatial size of an image is gradually reduced from a shallow layer to a deep layer by convolution in convolution layer C, and then gradually enlarged. . However, U-Net is convolution layer before being reduced (e.g., C ₁₎ the output of skipping intermediate convolution layer, convolution layer intended for spatial size of the same image directly behind Transfer to (for example, C ₂ ) (skip connection). As a result, the local features (eg, edge features of the input image, etc.) are directly transmitted backward without deterioration.

Evan Shelhamer, Jonathan Long, and Trevor Darrell, “Fully Convolutional Networks for Semantic Segmentation”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 39 Issue 4, April 2017, pp640-651. Olaf Ronneberger, Philipp Fischer, Thomas Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation”, Medical Image Computing and Computer-Assisted Intervention -MICCAI 2015, pp234-241.

In the conventional FCN network structure, since the spatial size of the convolution layer is gradually reduced toward the deep layer, local features of the image (such as edge position information) become ambiguous and the detail of the output image is There is a problem that accuracy falls.
On the other hand, the conventional U-Net can transmit backward gradually from the spatial size feature to the small feature of the convolution layer by the skip connection. However, global features (such as shape) of the image are not sufficiently extracted as large spatial features of the image that U-Net transmits in shallow layers. Therefore, U-Net can not extract effective features in the shallow layer, and there is room for further improvement in order to improve the precision of the output image.

  The present invention has been made in view of the problems as described above, and by associating global features with local features of an image, the global features and the local features are transmitted in a well-balanced manner, and the image is accurately produced. It is an object of the present invention to provide a neural network image information converter capable of converting information and a program thereof.

  In order to solve the above problems, an image information converter according to the present invention inputs image information of m (m is an integer of 1 or more) resolutions, and images of n (n is an integer of 1 or more) resolutions. An image information converter of a convolutional neural network in which a plurality of multi-scale converters for converting information into information are connected from the input side to the output side, the multi-scale converter comprising: a feature amount generation unit; And an image composition unit.

In such a configuration, the image information converter uses the feature amount generation unit of the multi-scale converter to extract one of the features for one resolution predetermined by convolution operation using learned parameters from image information of m resolutions. Generate quantity. Then, the image information converter generates image information of n resolutions by a convolution operation using learned parameters from the feature amounts generated by the feature amount generating unit by the image information generating unit of the multi-scale converter. To do.
The image information converter connects multiple multi-scale converters so that the feature value generation unit and the image information generation unit repeatedly extract feature values at different resolutions by convolution and distribute them to different scales. Run. As a result, the image information converter can be configured to more accurately extract complex features of image information by combining features of different scales.

Further, the image information converter combines the image information having the same resolution among the input image information with the image information generated by the image information generation unit by the image synthesis unit of the multi-scale converter.
By providing the image synthesizing unit in the multi-scale converter, the image information converter further synthesizes the image information for which the convolution operation is not performed on the image information sequentially output by the multi-scale converter to the subsequent multi-scale converter. And transmit it to the next stage. This enables the image information converter to transmit information that may be lost due to spatial reduction by the convolution operation to the subsequent stage.

  The present invention can also be realized by an image information conversion program for causing a computer to function as the image information converter.

The present invention has the following excellent effects.
According to the image information converter of the present invention, extraction of feature quantities at a plurality of resolutions is sequentially repeated by a plurality of multi-scale converters by convolution operation of image information, and image information not to be subjected to convolution operation and convolution operation Can be directly synthesized with the image information that has been
As a result, the image information converter according to the present invention can transmit local features with reduced ambiguity together with global features, and can suppress deterioration in the accuracy of details in output image information.

It is a block diagram which shows the example of the whole structure of the image information converter which concerns on 1st Embodiment of this invention. It is an explanatory view for explaining an outline of a multi-scale converter of an image information converter concerning a 1st embodiment of the present invention. It is a block block diagram which shows the structural example of the multiscale converter of the image information converter which concerns on 1st Embodiment of this invention. FIG. 6 is an explanatory diagram for describing the operation content of an individual feature calculation unit and a combined feature calculation unit of the multi-scale converter of FIG. 3; It is explanatory drawing for demonstrating the operation | movement content of the characteristic synthetic | combination part of the multiscale converter of FIG. It is explanatory drawing for demonstrating the operation | movement content of the separate information distribution part of the multiscale converter of FIG. It is a block block diagram which shows the structural example of the multiscale converter of 1 input 2 output. It is a block block diagram which shows the structural example of the multi-scale converter of 2 inputs 3 outputs. It is a block block diagram which shows the structural example of the multi-scale converter of 3 inputs 2 outputs. It is a block block diagram which shows the structural example of the multi-scale converter of 2 inputs 1 output. It is a block block diagram which shows the structure of the modification of a multi scale converter. It is a flowchart which shows operation | movement of the image information converter which concerns on 1st Embodiment of this invention. It is a block diagram which shows the example of the whole structure of the image information converter which concerns on 2nd Embodiment of this invention. It is a block block diagram which shows the structure of the colorization apparatus to which the image information converter which concerns on embodiment of this invention is applied. It is a figure which shows the network structure of the conventional image information converter (FCN). It is a figure which shows the network structure of the conventional image information converter (U-Net).

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
First Embodiment
<Configuration of image information converter>
First, the configuration of the image information converter 1 according to the first embodiment of the present invention will be described with reference to FIG.

The image information converter 1 converts image information into information to be converted by a previously learned neural network.
The image information to be input is image data having a spatial structure, and is a monochrome image of one channel, a color image of three channels (RGB), or the like. The information to be converted to be output is image data having a spatial structure, high-dimensional numerical information similar to the image, or the like.
For example, the image information converter 1 can be configured as a converter that receives pixel values of a black and white image and outputs pixel values (RGB) of a color image. Also, for example, the image information converter 1 is configured as a converter that inputs pixel values of a black and white image and pixel values (RGB) of a color image and outputs information obtained by classifying (dividing an area of) a subject area in the image. can do.

Here, the dimension of input image information is assumed to be resolution (full resolution) of vertical H [pixel] and horizontal W [pixel], and the number of channels is C ₁ . Further, the dimension of the image information after conversion is also vertically H, and lateral W, the image information of a channel number C _1.
1, the image information converter 1 toward the output side from the input side, a plurality of multi-scale converter 10 (10 1, _... 10 ₁₅₎ and by connecting in association with input and output configuration To do.

A multi-scale converter 10 (MSNB: Multi-Scale Neural Block) inputs image information of m (m is an integer of 1 or more) resolutions, and n (n is an integer of 1 or more) by convolution operation of a neural network. ) To convert image information of individual resolutions. The multiscale converter is hereinafter referred to as MSNB.
The MSNBs 10 ₁ to 10 ₇ from the input side to the predetermined intermediate MSNB 10 ₈ have a configuration in which the number (type) of resolution of the image information to be output is increased stepwise. In addition, for the MSNBs 10 ₉ to 10 ₁₅ from the predetermined MSNB 10 ₈ to the output side, the number (type) of resolutions of the image information to be output is gradually reduced.
In addition, as shown in FIG. 1, MSNBs 10 (for example, MSNBs 10 ₅ , 10 _6, etc.) having the same input and output may be connected successively. Furthermore, the intermediate MSNB 10 does not have to be located strictly between the plurality of MSNBs 10 (the number of MSNBs 10 in the former stage is the same as the number of MSNBs 10 in the latter stage).

Furthermore, the MSNB 10 has a path for transmitting the feature of the image information not accompanied by the reduction to the subsequent stage by combining the input image information with the image information generated by the convolution operation without performing the convolution operation. In the example of FIG. 1, R ₁ is a path for transmitting full resolution image information to the subsequent stage, R ₂ is a path for transmitting 1/2 resolution image information to the subsequent stage, and R ₃ is a path for transmitting 1/4 resolution image information to the subsequent stage. R ₄ is a path for transmitting image information of 1/8 resolution to the subsequent stage.
It should be noted that parameters such as filter weights used for the convolution calculation of the MSNB 10 constituting the image information converter 1 are preliminarily calculated from the image information before conversion and the image information which is known correct answer information after conversion, etc. To learn.

As described above, the image information converter 1 combines and transmits the image information generated at the stepwise resolutions having different scales by the convolution calculation and the image information not to be subjected to the convolution calculation by the plurality of MSNBs 10.
As a result, the image information converter 1 generates image information (globally extracted) in which features generated by performing convolution on non-convoluted image information (local features with reduced ambiguity) are generated in MSNB 10 units. Can be transmitted to the subsequent stage.

Hereinafter, the MSNB 10 that is the basic configuration of the image information converter 1 will be described. Note that the number of inputs and outputs of the MSNB 10 is not necessarily the same from the input side to the output side. Therefore, here, the outline and configuration of the MSNB 10 (for example, MSNB 10 ₅ ) having three inputs and three outputs including all the basic configurations will be described as an example.

<Overview of Multi-scale Converter (MSNB)>
First, the outline of MSNB10 ₅ (10) will be described with reference to FIG. As shown in FIG. 2, MSNB10 ₅ is for converting the image information input system 3 (input 1-3) image information output system is three is (output 1-3). MSNB10 ₅ as input 1, the vertical H [pixel], enter the spatial size, the image information of the full resolution of the number of channels _{C 1 [H × W × C} 1] of horizontal W [pixel]. Further, MSNB10 ₅ as input 2, the vertical H / 2 [pixel], horizontal W / spatial size of 2 [pixel], ½ resolution channel number _{C 2} [H / 2 × W / 2 × _{C 2} Enter the image information of]. Further, MSNB10 ₅ as an input 3, the vertical H / 4 [pixels], the lateral W / 4 spatial size, 1/4 resolution channel number _{C 3 [H / 4 × W} / 4 × C 3 of [pixel] ] Image information is input. The outputs 1 to 3 are image information having the same spatial size as the inputs 1 to 3. However, the number of channels does not have to be the same for input and output.

MSNB10 ₅ is the dimension (the number of channels, spatial size) of each image information inputted by the input 1-3 aligned by the convolution operation (Co1). Here, MSNB10 ₅ aligns the dimension to smallest input 3 of the image information of the spatial size. Then, MSNB10 ₅ combines the feature amount after convolution (Co1) (Su1), extracts the feature amount by convolution (Co2). Thus, MSNB10 ₅ extracts one resolution (1/4 resolution) worth of features from the three image information.

Then, MSNB10 ₅ from one resolution (1/4 resolution) partial feature quantity, the convolution operation (Co2), into a number of channels of the output 1-3, the expansion process Ex1, space of the output 1 and 2 Convert to the desired size.
Then, MSNB10 _5, the output 1-3 of dimensions (the number of channels, spatial size) and the image information converted into the image information is not performed convolution inputted by the path Sk1 (skip connection) synthesized (Su2) and output.
Thus, MSNB10 _5, even large picture information spatial size effectively with the image information whose feature amount has been extracted, can be transmitted to the subsequent stage.

<Configuration of Multi-scale Converter (MSNB)>
Next, the configuration of MSNB10 ₅ (10) will be described with reference to FIG.
As shown in FIG. 3, comprises MSNB10 ₅ includes a feature amount generating unit 20, an image information generating unit 30, an image combining unit 40, a.

The feature amount generation unit 20 generates a feature amount for one predetermined resolution by convolution operation from image information having a plurality of resolutions (here, “3”).
The feature quantity generation unit 20 includes an individual feature calculation unit 21 (21 ₁ , 21 ₂ , 21 ₃ ) for each resolution, a feature combination unit 22, and a combined feature calculation unit 23.

The individual feature calculation unit 21 calculates feature amounts from image information by a convolution operation. Note that the image information input to the MSNB ₁₀ other than the input-side MSNB 10 (MSNB ₁₀ 1 in FIG. 1) is a feature amount (image information) output from the preceding MSNB 10.

Individual characteristic calculation unit 21 ₁ receives the image information of the full resolution _{[H × W × C 1]} , to calculate the characteristic amount by the convolution operation using the learned parameters.
Individual characteristic calculation unit 21 _2, 1/2 inputs the image information of the resolution [H / 2 × W / 2 × C 2], calculates the feature amount by a convolution operation using the learned parameters.
Individual characteristic calculation unit 21 _3, 1/4 receives the image information of the resolution [H / 4 × W / 4 × C 3], to compute the characteristic quantity by the convolution operation using the learned parameters.

The individual feature calculation unit 21 (21 ₁ , 21 ₂ , 21 ₃ ), for example, as shown in FIG. 4, kernel (3) (3 × 3 convolution filter), padding (1), by performing the convolution operation stride (stride) "1", from the image information _{D 1,} to produce a feature amount _{D 2.} The number of channels after convolution may be any number, but for example, is the same as the input image information. The value (weight) of the convolution filter is a parameter obtained by learning of the neural network.
The individual feature calculation unit 21 (21 ₁ , 21 ₂ , 21 ₃ ) outputs the calculated feature amount to the feature combination unit 22.

The feature synthesis unit 22 synthesizes the feature amounts calculated by the individual feature calculation unit 21 (21 ₁ , 21 ₂ , 21 ₃ ). The feature synthesis unit 22 aligns the dimensions (number of channels, spatial size) of the feature amounts calculated by the individual feature calculation units 21 ₁ , 21 ₂ , and 21 ₃ to a predetermined resolution, and adds them for each element. To generate feature quantities. Here, it is preferable that the predetermined resolution here be the minimum resolution (here, 1⁄4 resolution) in the scale on the input side and the output side of the MSNB 10. In addition, the feature combining unit 22 may align and connect the dimensions of the feature amounts calculated by the individual feature calculation units 21 ₁ , 21 ₂ , and 21 ₃ .
The feature synthesis unit 22 outputs the synthesized feature quantity to the synthesis feature calculation unit 23.

Here, an example of the processing content of the feature synthesis unit 22 will be described with reference to FIG.
As shown in FIG. 5, the feature synthesis unit 22 is configured to use kernel “3” (3 × 3 convolution filter), padding “1”, stride “4”, channel “channel” By performing a convolution operation using C _{3 ′} ′, a feature amount D ₂ of the same dimension (number of channels, spatial size) as the minimum resolution (here, 1/4 resolution) is generated from the full resolution image information D ₁ To do.
Also, the feature composition unit 22 performs the convolution operation on the kernel “3”, padding “1”, stride “2”, and channel “C ₃ ” to obtain the minimum resolution from the half resolution image information D _3. A feature quantity D ₄ having the same dimension as (here, ¼ resolution) is generated.
The feature synthesis unit 22 performs a convolution operation with the kernel “3”, the padding “1”, the stride “1”, and the channel “C ₃ ” to obtain the minimum resolution from the 1/4 resolution image information D _5. (here, 1/4 resolution) generates a feature amount D ₆ of the same dimensions.

Then, the feature synthesis unit 22 adds (or concatenates) the feature quantity D ₂ , the feature quantity D _4, and the feature quantity D ₆ of the same dimension for each element to generate the feature quantity D ₇ .
The value of the convolution filter used for the convolution calculation of the image information of each resolution is a parameter obtained by learning of the neural network.

In the case where the feature synthesizing unit 22 generates the feature quantities by linking them, the spatial sizes of the individual feature quantities (here, the feature quantities D ₂ , D ₄ , and D ₆ ) may be equalized, and the number of channels is equalized. There is no need. In that case, the feature synthesis unit 22 may perform pooling by a pooling layer that simply calculates a maximum value or an average value, instead of convolution by a convolution layer as shown in FIG. For example, when the memory capacity of a computer that learns a neural network is limited, it is more appropriate to use a pooling layer that does not require learning.

In the example shown in FIG. 5, wherein the combining unit 22, the image information D ₁ of the full resolution, the kernel "4", the maximum pooling stride "4" (Max Pooling) or average pooling (Average, which Pooling) By performing this, a feature amount D ₂ having a 1/4 resolution [H / 4 × W / 4 × C ₁ ] is generated. Similarly, the feature combining unit 22 performs maximum pooling or average pooling of the kernel “2” and the stride “2” on the image information D _{3 of half} resolution to obtain 1/4 resolution [H / 4]. A feature amount D ₄ of × W / 4 × C ₂ ] is generated.
Then, the feature synthesizer 22 concatenates the generated feature quantity D ₂ and feature quantity D ₄ with the feature quantity D _{6 of} ¼ resolution, thereby [H / 4 × W / 4 × (C ₁ + C ₂ + C ₃ )] feature amount D ₇ is generated.
Returning to FIG. 3, continued explanation of the structure of MSNB10 _5.

The combined feature calculation unit 23 extracts combined feature amounts by performing a convolution operation on the feature amounts combined by the feature combining unit 22. The combined feature calculation unit 23 performs the same operation as the individual feature calculation unit 21 and extracts the combined feature amount by the same convolution operation as the example described in FIG. 4. Note that the value of the convolution filter is a parameter obtained by learning of the neural network.
The composite feature calculation unit 23 outputs the calculated composite feature amount to the image information generation unit 30.

The image information generation unit 30 performs a plurality of (here, “3”) scale image information (features) according to the resolution of the output from the feature amounts (combined feature amounts) generated by the feature amount generation unit 20 by convolution. Amount).
The image information generation unit 30 includes an individual information distribution unit 31 and an individual feature calculation unit 32 (32 ₁ , 32 ₂ , 32 ₃ ) for each resolution with different scales.

The individual information distribution unit 31 distributes the combined feature amount calculated by the combined feature calculation unit 23 to image information of dimensions (number of channels, spatial size) corresponding to the resolution of the output system.
The individual information distribution unit 31 performs a convolution operation to make the number of channels of the combined feature amount the same as the number of channels of image information to be output, and performs an enlargement process to make the spatial size the same. . The individual information distribution unit 31 may perform a deconvolution operation to generate image information of dimensions (number of channels, spatial size) according to the resolution of the output from the combined feature amount. Good.
The individual information distribution unit 31 outputs the image information classified by resolution to the individual feature calculation unit 32 (32 ₁ , 32 ₂ , 32 ₃ ) corresponding to each resolution.

Here, with reference to FIG. 6, an example of the processing content of the individual information distribution unit 31 will be described.
As shown in FIG. 6, the individual information distribution unit 31 performs the convolution operation on the kernel “3”, padding “1”, stride “2”, and channel “C ₃ ” so that the spatial size is H / 4. in × W / 4, 1/4 resolution image data of the channel number _{C 3} (feature quantity) from _{D 1,} with the spatial size of the same (H / 4 × W / 4 ), an image in which the number of channels and _{C 1} It generates information _{D 2.}
Then, the individual information distribution unit 31 enlarges the spatial size of the image information D ₂ by 4 in vertical and horizontal directions in channel units, so that the full-resolution image information with a spatial size of H × W and the number of channels C ₁ to generate a D _3. In addition, what is necessary is just to use general methods, such as bi-linear expansion and nearest neighbor expansion, for the expansion which the separate information distribution part 31 performs.

Further, the individual information allocating unit 31 performs convolution calculation with the kernel “3”, the padding “1”, the stride “1”, and the channel “C ₃ ”, so that the image information (feature amount) D of ¼ resolution is obtained. ₁ , image information D ₄ having the same spatial size (H / 4 × W / 4) and the number of channels as C ₂ is generated.
Then, the individual information sorting unit 31, to enlarge the spatial size of the image information D ₄ doubled vertically and horizontally in units of channel, spatially size H / 2 × W / 2, 1 channel number C ₂ / 2 to generate image data _{D 5} resolution.

Also, the individual information distribution unit 31 performs a convolution operation on kernel “3”, padding “1”, stride “1”, and channel “C ₃ ” to obtain image information (feature amount) D of 1⁄4 resolution. ₁ , image information D ₆ having the same spatial size (H / 4 × W / 4) and the number of channels as C ₃ is generated. Note that the image information D _6, since 1/4 resolution and spatial size of the output is the same and does not expand.
Note that the value of the convolution filter used for the convolution calculation of the image information of each resolution is a parameter obtained by learning of the neural network.
Thus, the individual information sorting unit 31 can generate image information of a dimension that matches the resolution to be output from one composite feature amount.
Returning to FIG. 3, continued explanation of the structure of MSNB10 _5.

The individual feature calculation unit 32 calculates a feature amount by performing a convolution operation on the resolution-based image information generated by the individual information distribution unit 31. The individual feature calculation unit 32 is the same operation as the individual feature calculation unit 21 and extracts the feature amount by the same convolution operation as the example described in FIG. 4. Note that the value of the convolution filter is a parameter obtained by learning of the neural network.
The individual feature calculation unit 32 outputs the calculated feature amount to the image composition unit 40.

Image synthesizing unit 40, individual characteristic calculation unit ₃₂ (32 _1, 32 2, 32 ₃₎ is calculated resolution another feature quantity (image information), the same resolution in the image information input to MSNB10 ₅ Image information is synthesized.
The image combining unit 40 includes a plurality of skip combining units 41 (41 ₁ , 41 ₂ , 41 ₃ ) according to the resolution of the image information (feature amount) to be output.

Skip combining unit 41, the image information generation unit 30 (individual characteristic calculation unit 32) resolution-specific image information generated by, MSNB10 is inputted to _5, MSNB10 not done convolution operation in the _fifth image information of the same resolution Is synthesized.

Skip combining unit 41 ₁ includes a full-resolution image information generated by the convolution operation individual characteristic calculation unit 32 ₁ (features), and image information of the full resolution input to MSNB10 ₅ (features), the element Each addition (or concatenation), and output to the latter stage.
Skip combining unit 41 _2, 1/2 resolution image information generated by the convolution operation individual characteristic calculation unit 32 ₂ (feature quantity), 1/2 resolution image information input to MSNB10 ₅ (features) Are added (or concatenated) for each element and output to the subsequent stage.
Skip combining unit 41 _3, 1/4 resolution image information generated by the convolution operation individual characteristic calculation unit 32 ₃ (feature quantity), ¼ resolution image information input to MSNB10 ₅ (features) Are added (or concatenated) for each element and output to the subsequent stage.

As described above, by configuring the MSNB 10 ₅ (10), the MSNB 10 becomes a model in which the feature quantities of the image information of each resolution are learned as complex features using feature quantities of other resolutions. In addition, the MSNB 10 can transmit the feature lost due to the spatial reduction while being held by the skip connection.
Although the configuration of the MSNB 10 ₅ including all the basic configurations has been described above, the MSNB ₁₀ other than the MSNB 10 ₅ may be configured by increasing or decreasing each basic configuration.
Hereinafter, representative configurations will be described.

<Other Configurations of Multi-scale Converter (MSNB)>
(1 input 2 output MSNB)
The configuration of the asymmetric MSNB 10 ₁ (10) for converting image information of one resolution into image information of two resolutions will be described with reference to FIG.
As shown in FIG. 7, the 1-input 2-output MSNB _{10 1} omits the individual feature calculation units 21 ₂ , 21 ₃ , 32 ₃ and the skip synthesis units 41 ₂ , 41 ₃ from the MSNB 10 ₅ described in FIG. Can be configured.

Here, characterized synthesizing unit 22, from the output side of the minimum resolution (here, 1/2 resolution) feature amounts of full resolution calculated by the individual characteristic calculation unit 21 ₁ and the same dimension (the number of channels, spatial Feature quantities of size) are generated by convolution operation. For example, the feature synthesizing unit 22 performs a convolution operation on the kernel “3”, padding “1”, stride “2”, and channel “C ₂ ” to obtain half-resolution feature amounts from full-resolution feature amounts. Is generated.
Note that, since there is one individual feature calculation unit 21, the feature synthesis unit 22 does not perform synthesis, and outputs the generated 1/2 resolution feature quantity as a synthesis feature quantity to the synthesis feature calculation unit 23. Other configurations are the same as the configuration of MSNB10 ₅ described in FIG. 3, the description thereof is omitted.

(MSNB with 2 inputs and 3 outputs)
FIG. 8 shows a configuration example of the asymmetric MSNB 10 ₄ (10) that converts image information of two resolutions into image information of three resolutions.
As shown in FIG. 8, 2 inputs 3 outputs MSNB10 ₄ can be from MSNB10 ₅ described in FIG. 3, it is constructed by omitting individual characteristic calculation unit 21 ₃ and the skip combining unit 41 _3.

(MSNB with 3 inputs and 2 outputs)
FIG. 9 shows an exemplary configuration of an asymmetric MSNB 10 ₁₂ (10) that converts image information of three resolutions into image information of two resolutions.
As shown in FIG. 9, MSNB10 ₁₂ of the three-input two-output may be from MSNB10 ₅ described in FIG. 3, it is constructed by omitting individual characteristic calculation unit 32 ₃ and the skip combining unit 41 _3.

(MSNB with 2 inputs and 1 output)
FIG. 10 shows an exemplary configuration of an asymmetric MSNB 10 ₁₅ (10) that converts image information of two resolutions into image information of one resolution.
As shown in FIG. 10, the 2-input 1-output MSNB 10 ₁₅ omits the individual feature calculation units 21 ₃ , 32 ₂ , 32 ₃ and the skip synthesis units 41 ₂ , 41 ₃ from the MSNB 10 ₅ described in FIG. Can be configured.
The configuration example of the MSNB 10 has been described above. However, when the number of inputs and outputs is larger than that of the illustrated MSNB 10, the individual feature calculation units 21 and 32 and the skip synthesis unit 41 may be configured according to the number of inputs and outputs. That's fine.

Further, in the MSNB 10, the feature quantity generation unit 20 does not necessarily need to provide the individual feature calculation unit 21 in all resolution paths. For example, in the case of MSNBs 10 with the same number of inputs / outputs or with a smaller number of outputs than the number of inputs (for example, 10 ₅ , 10 ₁₂ etc. in FIG. Other than the individual feature calculation unit 21 that extracts the feature quantity with the minimum resolution may be omitted.
For example, the MSNB10 ₅ described in FIG. 3, may be configured as MSNB10B ₅ shown in FIG. 11. Note that the MSNB 10B ₅ shown in FIG. 11 omits the feature synthesis unit 22 and the synthesis feature calculation unit 23 due to the omission of the individual feature calculation units 21 ₁ and 21 ₂ .

By connecting the MSNB 10 described above from the input side to the output side of the image information, the image information converter 1 performs the convolution by the skip operation when transmitting the global features to the subsequent stage. Not local features can be transmitted to a later stage.
As a result, the image information converter 1 can suppress the ambiguity of local features such as edge position information, and can improve the accuracy of details of the output image.
In addition, the image information converter 1 can operate the computer which abbreviate | omitted illustration with the program for functioning as a neural model network which connected MSNB10.

<Operation of image information converter>
Next, the operation of the image information converter 1 according to the first embodiment of the present invention will be described with reference to FIG. 12 (for the configuration, refer to FIG. 1 and FIG. 3 as needed).
In step S <b> 1, the individual feature calculation unit 21 of the MSNB 10 calculates the feature amount by the convolution calculation for the number of input systems.
In step S2, the feature synthesizing unit 22 of the MSNB 10 converts the feature quantities for the number of input systems calculated in step S1 by convolution operation so as to make dimensions (number of channels, spatial size) uniform. At this time, the feature synthesis unit 22 sets the dimension to be the same as the minimum resolution that is the output of the MSNB 10.

In step S3, the feature synthesis unit 22 of the MSNB 10 further synthesizes the feature quantities having the same dimensions (number of channels and spatial size) converted in step S2 by adding or concatenating them for each element.
In step S4, the combined feature calculation unit 23 of the MSNB 10 generates a combined feature amount by performing a convolution operation on the feature amount combined in step S3.

In step S5, the individual information allocating unit 31 of the MSNB 10 performs the convolution operation according to the number of channels of the output system and distributes the combined feature value generated in step S4 for each output system.
In step S6, the individual information distribution unit 31 of the MSNB 10 further expands each image information allocated to the output system in step S5 in accordance with the spatial size of the output system.

  The individual information distribution unit 31 deconvolves (deconvolutes) the combined feature quantities generated in step S4 in steps S5 and S6 into the dimensions (number of channels, spatial size) of the output system. ) Operation may be performed.

  In step S7, the skip composition unit 41 of the MSNB 10 synthesizes the image information of the input system of the same system with the image information of the output system generated in step S6. As a result, the image information input to the MSNB 10 is synthesized as it is with the image information of each resolution subjected to the convolution calculation of the output system.

If the MSNB 10 is connected to the subsequent stage in step S8 (Yes), the image information converter 1 returns to step S1 and repeats the operations from step S1 to S7 in the subsequent MSNB 10.
On the other hand, when the MSNB 10 is not connected to the subsequent stage in step S8 (No), the image information converter 1 ends the operation.

By the above operation, the image information converter 1 repeats extraction (composition) and distribution of feature quantities at resolutions with different scales by convolution of image information, and an image subjected to convolution with image information not to be subjected to convolution. Sequentially synthesize with information.
As a result, the image information converter 1 can extract feature quantities with high accuracy in which global features are associated with local features, and can operate as a neural network with high conversion accuracy.

Second Embodiment
Next, an image information converter 1B according to a second embodiment of the present invention will be described with reference to FIG.
The image information converter 1 described with reference to FIG. 1 is configured such that the number (type) of resolutions of image information to be output is gradually reduced for the MSNB ₁₀ from the intermediate MSNB 108 to the output side determined in advance. Reduction in the number of the resolution may be achieved by combining the output from the intermediate MSNB10 _8.

  As shown in FIG. 13, the image information converter 1 B increases the number (types) of resolutions of image information to be output stepwise from the input side to the output side, It is configured by associating and connecting. Furthermore, the image information converter 1B includes a scale integration unit 50. The MSNB 10 has the same configuration as the image information converter 1 described in FIG. The output of MSNB 10 at the last stage is 2 or more.

The scale integration unit 50 integrates a plurality of pieces of image information (features) generated by a plurality of MSNBs 10 into one piece of image information.
As illustrated in FIG. 13, the scale integration unit 50 includes a scale conversion unit 51, a synthesis unit 52, and a feature calculation unit 53.

The scale conversion unit 51 performs scale conversion of image information of other resolutions except for the maximum resolution (full resolution) of the MSNB 10 (10 ₈ ) at the last stage to the maximum resolution. Here, the scale conversion unit 51 includes three scale conversion units 51 (51 ₁ , 51 ₂ , 51 ₃ ) according to 1/2 resolution, 1/4 resolution, and 1/8 resolution.
The scale conversion unit 51 performs enlargement processing in order to align the resolution to the full resolution space size. For the enlargement in the scale converter 51, a general method such as bilinear enlargement or nearest neighbor enlargement may be used.

Scale conversion unit 51 _1, 1/2-resolution vertical doubling image information, by expanding doubled horizontal, into a full-resolution image information, and outputs to the combining unit 52.
Scaling portion 51 _2, the vertical quadruple image information of 1/4-resolution, by expanding laterally fourfold, into image information of full resolution, and outputs to the combining unit 52.
Scaling portion 51 _3, 1/8-resolution vertical 8 times the image information, by expanding laterally 8 times, into a full-resolution image information, and outputs to the combining unit 52.

The synthesizing unit 52 synthesizes image information (feature amount) transmitted through a plurality of routes.
The combining unit 52 combines the full-resolution image information output from the MSNB 10 (10 ₈ ) at the last stage and the image information scaled to full resolution by the scale conversion unit 51 (51 ₁ 51 ₂ 51 ₃ ). Synthesize. The combining process of the combining unit 52 can be performed, for example, by connecting all full resolution image information.
The combining unit 52 outputs the combined image information to the feature calculation unit 53.

  Note that the composition processing of the composition unit 52 may add all the full-resolution image information for each element. In that case, it is necessary to make the number of channels uniform in all full-resolution image information. Specifically, the scale conversion unit 51 may perform a convolution operation to equalize the number of channels before expanding to full-resolution image information. Of course, the scale conversion unit 51 may perform a deconvolution operation so that the input image information has dimensions (number of channels, spatial size) of full resolution.

The feature calculation unit 53 converts the image information (feature amount) synthesized by the synthesis unit 52 into image information of a conversion target dimension.
The feature calculation unit 53 converts the input image information into image information to be converted by performing a convolution operation. The value of the convolution filter is a parameter obtained by learning of the neural network.

By configuring the image information converter 1B as described above, the image information converter 1B has a high-precision feature value in which global features are associated with local features in the same manner as the image information converter 1. And can operate as a neural network with high conversion accuracy.
The image information converter 1B can operate a computer (not shown) with a program for functioning as a neural model network configured of a plurality of MSNBs 10 and a scale integration unit 50.

As mentioned above, although image information converter 1 and 1B concerning the embodiment of the present invention was explained, the present invention is not limited to these embodiments.
For example, other arithmetic processing, for example, a normalization layer may be provided before and after MSNB 10 or in a transfer path inside MSNB 10 to normalize the overall numerical value of the image information.

  Furthermore, here, the image information converter 1 connects 15 MSNBs 10 and the image information converter 1 B connects 8 MSNBs 10, and converts image information with four scale resolutions (full resolution to 1/8 resolution). As an example. However, these numbers are not limited to this embodiment, and depending on the resolution of the image to be converted, several tens to several hundreds of MSNBs 10 may be connected, and the number of scales is two, three or five or more. It doesn't matter.

Here, the image information converters 1 and 1B are connected in series in the image information transmission path of the MSNB 10. However, the image information converters 1 and B may be configured to connect the MSNBs 10 in parallel. For example, in the image information converter 1 of FIG. 1, the output of the MSNB 101 is output to two MSNBs 10, and the MSNBs ₁₀ are connected in series and then connected to the MSNB 10 at the last stage (e.g., 10 ₁₅ in FIG. 1). It is good.

Here, the image information converters 1 and 1B input one piece of image information and output one piece of converted image information. However, input / output is not limited to one. For example, the input may be a black and white image and a genre to which the black and white image belongs (for example, sports, animation, etc.). In this case, the genre is, for example, a value indicating that one channel is associated with one channel of the same spatial size as the black and white image and only the channel corresponding to the corresponding genre is set (for example, “1 A value (for example, “0”) may be set to indicate that no genre is set for the other channels.
Also, for example, the output may be a color image of three channels and a probability distribution of colors corresponding to the pixels of the color image (for example, probability distributions of colors quantized to x class [for x channels]). .

<Example of application of image information converter>
Next, application examples of the image information converters 1 and 1B according to the embodiment of the present invention will be described.
FIG. 14 is a configuration diagram showing an example in which the image information converters 1 and B are configured as a colorization apparatus. A colorizing apparatus 100 shown in FIG. 14 converts a vertical H [pixel], horizontal W [pixel], 1 channel monochrome image BW into a vertical H [pixel], horizontal W [pixel], 3 channel (RGB) color image. This is converted to CL.
As shown in FIG. 14, the colorization apparatus 100 includes an information input unit 110, an information conversion unit 120, and an information output unit 130.

The information input unit 110 inputs a monochrome image BW to be converted from the outside. Note that the information input unit 110 may input a monochrome moving image in units of frames.
The information input unit 110 outputs the input monochrome image BW to the information conversion unit 120.

  The information conversion means 120 converts the black and white image which is the image information input by the information input means 110 with a learning model learned in advance. The information conversion unit 120 converts the monochrome image BW into a color image CL using the image information converter 1 or the image information converter 1B as a learning model learned in advance. The information conversion means 120 outputs the converted color image CL to the information output means 130.

The information output unit 130 outputs the color image CL, which is the image information converted by the information conversion unit 120, to the outside. For example, the information output unit 130 stores the color image CL in a storage device (not shown).
By this, the colorization apparatus 100 can generate a color image with high accuracy, such as division of edge color, by the image information converters 1 and 1B.

  The application of the image information converters 1 and 1B can be variously applied in addition to the colorization of black and white images. For example, it is possible to configure as an area dividing device that inputs a color image of 3 channels and outputs area information of 1 channel obtained by dividing an area of a subject included in the image.

1,1B Image information converter 10 Multiscale converter (MSNB)
DESCRIPTION OF SYMBOLS 20 Feature-value production | generation part 21 Individual feature calculation part 22 Feature composition part 23 Synthesis | combination feature calculation part 30 Image information generation part 31 Individual information distribution part 32 Individual feature calculation part 40 Image composition part 41 Skip composition part 50 Scale integration part 51 Scale conversion Unit 52 Combining Unit 53 Feature Calculation Unit 100 Coloring Device 110 Information Input Unit 120 Information Conversion Unit 130 Information Output Unit

Claims (7)

A plurality of multi-scale converters for inputting image information of m (m is an integer of 1 or more) resolutions and converting them into image information of n (n is an integer of 1 or more) resolutions An image information converter of a convolutional neural network connected toward
The multi-scale converter is
A feature amount generation unit that generates a feature amount for one resolution determined in advance by convolution operation using learned parameters from the image information of the m resolutions;
An image information generation unit that generates image information of the n resolutions by performing a convolution operation using learned parameters from the feature amounts generated by the feature amount generation unit;
An image synthesis unit that synthesizes image information having the same resolution among the image information input to the multi-scale converter with respect to the image information generated by the image information generation unit;
An image information converter characterized by comprising:   The feature value generation unit of the multi-scale converter in which m is 2 or more performs convolution operation on the image information of the m resolutions to the image information of the smallest resolution among the input and output of the multi-scale converter. The image information converter according to claim 1, wherein the feature amount is generated by adding or concatenating calculation results.   The image information generation unit of the multi-scale converter in which the n is 2 or more performs n convolution operations on the feature amount, and expands a calculation result according to the n resolutions. The image information converter according to claim 1 or 2, wherein the image information of the n resolutions is generated.   The image information generation unit of the multiscale converter in which the n is 2 or more performs n deconvolution operations on the feature amount to generate image information of the n resolutions. The image information converter according to any one of claims 1 to 3. Gradually increasing the number of resolutions of the image information output by the multi-scale converter from the input side to a predetermined intermediate multi-scale converter;
The method according to any one of claims 1 to 4, wherein the number of resolutions of the image information output from the multiscale converter from the intermediate multiscale converter to the output side is gradually reduced. The image information converter according to any one of the items. It further includes a scale integration unit that integrates the output of the multi-scale converter at the last stage,
The multi-scale converter is
A feature amount generation unit that generates a feature amount for one resolution predetermined by convolution calculation from the image information of m resolutions;
An image information generation unit that generates the image information of the n resolutions by a convolution operation from the feature amounts generated by the feature amount generation unit;
And an image combining unit configured to combine the image information generated at the image information generation unit with the image information having the same resolution among the input image information,
The scale integration unit
A scale conversion unit for equalizing resolutions of image information of a plurality of resolutions generated by the last stage multi-scale converter;
A combining unit that combines a plurality of pieces of image information with the same resolution in the scale conversion unit;
A feature calculation unit that generates image information after conversion by convolution operation from the image information synthesized by the synthesis unit;
The image information converter according to any one of claims 1 to 4, further comprising:   The image information conversion program for functioning a computer as an image information converter as described in any one of Claims 1-6.

Images