JP2020038572A - Image learning program, image learning method, image recognition program, image recognition method, creation program for learning data set, creation method for learning data set, learning data set, and image recognition device - Google Patents

Abstract

To propose an image learning program and the like that can improve efficiency in learning with teacher images.SOLUTION: An image learning program causes an image recognition device to execute learning by using a learning data set including learning images and teacher images. The image recognition device comprises a first image recognition unit and a second image recognition unit. The first image recognition unit performs image segmentation of the learning images. The second image recognition unit performs more elaborate image segmentation than that of the first image recognition unit. The image learning program causes to execute a first step of inputting the learning image in the first image recognition unit to acquire a first output image, a second step of inputting the learning image and first output image to the second image recognition unit to acquire a second output image, a third step of acquiring a second error of the second output image with respect to the teacher image, and a fourth step of correcting an error in the second image recognition unit on the basis of the second error.SELECTED DRAWING: Figure 9

Description

  The present invention relates to an image learning program, an image learning method, an image recognition program, an image recognition method, a learning data set generation program, a learning data set generation method, a learning data set, and an image recognition device.

  As an image recognition technology, Semantic Segmentation using a Fully Convolutional Network (FCN: full-layer convolution network) is known (for example, see Non-Patent Document 1). In semantic segmentation, a digital image is classified into pixels (class inference). That is, in the semantic segmentation, a class is inferred for each pixel of the digital image, and as a result of the inference, a class is labeled for each pixel to divide a region of the digital image.

Zhao, Hengshuang, et al. "Pyramid scene parsing network." IEEE Conf. On Computer Vision and Pattern Recognition (CVPR). 2017

  Here, in the semantic segmentation, the accuracy of image recognition is increased by performing deep learning using a learning data set. The learning data set includes a learning image serving as a learning target image, and a region-divided teacher image serving as an answer to the learning image. The teacher image used for the semantic segmentation is generated by the annotation work. However, since the workload of the annotation work is high, the annotation cost is high. Further, in order to accurately perform image recognition on various scenes, it is necessary to prepare a large number of teacher images used for semantic segmentation, and the annotation cost further increases.

  The present invention provides an image learning program, an image learning method, an image recognition program, an image recognition method, a learning data set generating program, a learning data set generating method, a learning data set, and a learning method that can improve the learning efficiency of a teacher image. It is an object to provide an image recognition device.

  An image learning program according to one aspect is an image learning program executed by an image recognition device that performs image segmentation, wherein a learning data set used for learning of the image recognition device is a learning target set of the image recognition device. The image recognition device includes a first image recognition unit that performs image segmentation of the learning image, and a first image recognition unit that performs image segmentation of the learning image. A second image recognizing unit that performs image segmentation of the learning image so as to be more finely divided into regions, and the learning image is input to the first image recognizing unit, and the first image recognition is performed. A first step of performing image segmentation of the learning image by a unit to obtain a first output image; A second step of inputting to the second image recognizing unit and performing image segmentation of the learning image by the second image recognizing unit using the first output image to obtain a second output image And a third step of obtaining a second error of the second output image with respect to the teacher image; and correcting the image segmentation process by the second image recognition unit based on the second error. And performing the fourth step.

  An image learning method according to one aspect is an image learning method performed by an image recognition device that performs image segmentation, wherein a learning data set used for learning of the image recognition device includes a learning data set of a learning target of the image recognition device. A learning image to be an image, and a teacher image corresponding to the learning image, wherein the image recognition device includes a first image recognition unit that performs image segmentation of the learning image, and a first image recognition unit. A second image recognizing unit that performs image segmentation of the learning image so that the region is finely divided, and inputs the learning image to the first image recognizing unit. A first step of performing image segmentation of the learning image to obtain a first output image, and performing the second image recognition on the learning image and the first output image. A second step of performing image segmentation of the learning image by the second image recognition unit using the first output image to obtain a second output image; and A third step of obtaining a second error of the second output image; and a fourth step of correcting an image segmentation process by the second image recognition unit based on the second error. Including.

  An image recognition program according to one aspect is an image recognition program executed by an image recognition device that performs image segmentation of an input image that has been input, wherein the image recognition device performs image segmentation of the input image. A first image recognition unit, and a second image recognition unit that performs image segmentation of the input image so as to be more finely divided into regions than the first image recognition unit. An eighth step of inputting the input image to the image recognition unit, and performing an image segmentation of the input image by the first image recognition unit to obtain a first output image; and the input image and the first output image. Is input to the second image recognizing unit, and the second image recognizing unit performs image segmentation of the input image using the first output image. To execute a ninth step of obtaining a second output image.

  An image recognition method according to one aspect is an image recognition method performed by an image recognition device that performs image segmentation of an input image that has been input, wherein the image recognition device performs first image segmentation that performs image segmentation of the input image. An image recognizing unit, and a second image recognizing unit that performs image segmentation of the input image so that the area is more finely divided than the first image recognizing unit. An eighth step of inputting the input image to the image recognizing unit, performing an image segmentation of the input image by the first image recognizing unit, and obtaining a first output image; Is input to the second image recognizing unit, and the second image recognizing unit performs image segmentation of the input image using the first output image, thereby obtaining a second output image. Including a ninth step of Tokusuru, the.

  A program for generating a learning data set according to one aspect is executed by an image recognition device that performs image segmentation of an input image that has been input, and generates a learning data set that is used in the image recognition device. The image recognition device may further include: a first image recognition unit that performs image segmentation of the input image; and an image segmentation of the input image that is more minutely divided than the first image recognition unit. A second image recognizing unit for performing, wherein the learning data set includes a first learning data set for learning by the first image recognizing unit, and a learning data set for learning by the second image recognizing unit. A second learning data set, wherein the first learning data set includes a first learning image and a first teacher image corresponding to the first learning image. Wherein the second learning data set includes a second learning image and a second teacher image corresponding to the second learning image, and the second teacher image includes The first learning image and the first teacher image are input to the second image recognition unit, and the second learning image and the first teacher image are input to the second image recognition unit. An eleventh step of performing image segmentation of the first learning image using the first teacher image by an image recognition unit to obtain a second output image, and converting the first learning image to the second learning image. And the second output image is obtained as the second teacher image, and a second learning data set including the second learning image and the second teacher image is generated. The twelfth step is executed.

  A method of generating a learning data set according to one aspect is performed by an image recognition device that performs image segmentation of an input image that has been input, and generates a learning data set that generates a learning data set used in the image recognition device. A method, comprising: a first image recognition unit for performing image segmentation of the input image; and an image segmentation of the input image so as to perform a finer region division than the first image recognition unit. And a second image recognizing unit for performing the learning. The learning data set includes a first learning data set for the first image recognizing unit to learn, and a learning data set for the second image recognizing unit to learn. A second learning data set, wherein the first learning data set includes a first learning image, and a first teacher image corresponding to the first learning image. The second learning data set includes a second learning image and a second teacher image corresponding to the second learning image, and the second teacher image is included in the first teacher image. The first learning image and the first teacher image are input to the second image recognizing unit, and the first learning image and the first teacher image are input to the second image recognizing unit. An eleventh step of performing image segmentation of the first learning image using a first teacher image to obtain a second output image, and obtaining the first learning image as the second learning image And a twelfth step of obtaining the second output image as the second teacher image and generating a second learning data set including the second learning image and the second teacher image. ,including.

  An image recognition device according to one aspect includes a first image recognition unit that performs image segmentation of an input image, and performs an image segmentation of the input image so as to perform a finer region division than the first image recognition unit. A second image recognizing unit for performing, when the input image is input, the first image recognizing unit performs image segmentation of the input image to generate a first output image, and generates the first output image. The first output image is output to the second image recognition unit, and the second image recognition unit receives the first image when the input image and the first output image are input. The image segmentation of the input image is performed by the second image recognition unit using the output image of (i), and a second output image is output.

FIG. 1 is a diagram illustrating an outline of an image recognition device according to the embodiment. FIG. 2 is a diagram illustrating an outline of an image recognition unit of the image recognition device according to the embodiment. FIG. 3 is a diagram illustrating an example of an input image input to the image recognition device. FIG. 4 is a diagram illustrating an example of an output image output from the image recognition device. FIG. 5 is a diagram illustrating an example of the first learning data set. FIG. 6 is a diagram illustrating an example of the second learning data set. FIG. 7 is a diagram illustrating an example of a process related to image learning of the image recognition device. FIG. 8 is a diagram illustrating an example of a process related to image learning of the image recognition device. FIG. 9 is a diagram illustrating an example of a process related to image learning of the image recognition device. FIG. 10 is a diagram illustrating an example of a process related to image recognition of the image recognition device. FIG. 11 is a diagram illustrating an example of a process regarding image recognition of the image recognition device. FIG. 12 is a diagram illustrating an example of a process regarding image recognition of the image recognition device. FIG. 13 is a diagram illustrating an example of a process regarding generation of a learning data set by the image recognition device.

  An embodiment according to the present application will be described in detail with reference to the drawings. In the following description, similar components may be denoted by the same reference numerals. Further, duplicate description may be omitted. In addition, description and illustration of matters that are not closely related in describing the embodiment according to the present application may be omitted.

(Embodiment)
FIG. 1 is a diagram illustrating an outline of an image recognition device according to the embodiment. The image recognition device 1 recognizes an object included in an input image I to be input, and outputs a recognition result as an output image O. The image recognition device 1 receives a captured image captured by an imaging device such as a camera as an input image I. The image recognition device 1 performs image segmentation on an input image I. Image segmentation refers to labeling a class with respect to a divided image region of a digital image, and is also referred to as class inference (class classification). That is, the image segmentation is to determine which class the predetermined image area obtained by dividing the digital image belongs to, and to attach an identifier (label) for identifying the class indicated by the image area. The image recognition device 1 outputs an image obtained by performing image segmentation (class inference) on the input image I as an output image O.

  The image recognition device 1 is provided, for example, in an in-vehicle recognition camera of a car. The in-vehicle recognition camera captures the running state of the vehicle at a predetermined frame rate in real time, and inputs the captured image to the image recognition device 1. The image recognition device 1 acquires a captured image input at a predetermined frame rate as an input image I. The image recognition device 1 classifies the objects included in the input image I into classes, and outputs the classified images as output images O at a predetermined frame rate. Note that the image recognition device 1 is not limited to being mounted on a vehicle-mounted recognition camera, and may be provided in another device.

  First, the input image I will be described with reference to FIG. FIG. 3 is a diagram illustrating an example of an input image I input to the image recognition device 1. The input image I is a digital image including a plurality of pixels. The input image I is, for example, an image generated by an imaging device provided in an imaging device such as a camera and having a resolution according to the number of pixels of the imaging device. That is, the input image I is an original high resolution original image that has not been subjected to upsampling processing for increasing the number of pixels of the image or downsampling processing for decreasing the number of pixels of the image.

  Next, the output image O will be described with reference to FIG. FIG. 4 is a diagram illustrating an example of an output image O output from the image recognition device 1. The output image O is divided into regions for each class. The class includes, for example, an object included in the input image I, and is a person, a car, a road, a building, and the like. The output image O is classified into classes by each object on a pixel-by-pixel basis, and the classes classified on a pixel-by-pixel basis are labeled to classify the image into regions. FIG. 4 illustrates, for example, an image region Oa classified into a person class, an image region Ob classified into a car class, and an image region Oc classified into a road class. Note that the output image O in FIG. 4 is an example, and there is no particular limitation on this class classification. The output image O has the same resolution as the input image I.

  Referring to FIG. 1 again, the image recognition device 1 will be described. The image recognition device 1 includes a control unit 5, a storage unit 6, and an image recognition unit 7.

  The storage unit 6 stores programs and data. Further, the storage unit 6 may be used as a work area for temporarily storing a processing result of the control unit 5. The storage unit 6 may include an arbitrary storage device such as a semiconductor storage device and a magnetic storage device. Further, the storage unit 6 may include a plurality of types of storage devices. The storage unit 6 may include a combination of a portable storage medium such as a memory card and a storage medium reading device.

  The storage unit 6 includes, as programs, an image learning program P1, an image recognition program P2, and a learning data set generation program P3. The image learning program P1 is a program for causing the image recognition unit 7 to perform learning. The image recognition program P2 is a program for causing the image recognition unit 7 to perform image recognition. The learning data set generation program P3 is a program for generating a learning data set used for learning of the image recognition unit 7. Further, the storage unit 6 includes various images and a learning data set as data. The various images are an input image I input to the image recognition device 1, an output image O output from the image recognition device 1, and the like. The learning data set is data used for learning of the image recognition unit 7.

  The control unit 5 controls the operation of the image recognition device 1 comprehensively to realize various functions. The control unit 5 includes, for example, an integrated circuit such as a CPU (Central Processing Unit). Specifically, the control unit 5 executes commands included in the program stored in the storage unit 6 and controls the image recognition unit 7 and the like to realize various functions.

  The control unit 5 executes the learning of the image recognition unit 7 using the learning data set, for example, by executing the image learning program P1. Further, the control unit 5 causes the image recognition unit 7 to execute the image recognition of the input image I by executing the image recognition program P2, for example. Further, the control unit 5 causes the image recognition unit 7 to generate a learning data set by executing the generation program P3.

  Next, the image recognition unit 7 will be described with reference to FIG. FIG. 2 is a diagram illustrating an outline of an image recognition unit of the image recognition device according to the embodiment. The image recognition unit 7 includes an integrated circuit such as a GPU (Graphics Processing Unit). The image recognition section 7 includes a first image recognition section 11 and a second image recognition section 12. When the input image I is input, the image recognition unit 7 inputs the input image I to the first image recognition unit 11 and the second image recognition unit 12, respectively.

  The first image recognition unit 11 executes a task for finding the position of an object included in the input image I. The first image recognition unit 11 performs, for example, image segmentation using a bounding box. The bounding box is a rectangular image area surrounding an object included in the input image I. When the input image I is input, the first image recognition unit 11 extracts a bounding box surrounding the object from the input image I, and outputs an image classified into classes for each bounding box to a first output. Output as image O1. That is, the first image recognizing unit 11 performs region segmentation on the input image I roughly (roughly) as compared with the second image recognizing unit 12 described later, and includes information on the position of the object. A first output image O1 is output.

  The first image recognition unit 11 performs image segmentation using a neural network including a convolutional layer such as a CNN (Convolution Neural Network) or an FCN (Fully Convolutional Network) (hereinafter, also simply referred to as a network). The first image recognition unit 11 has an encoder 22 and a decoder 23.

  The encoder 22 performs an encoding process on the input image I. The encoding process is a process of executing a downsampling (also referred to as pooling) for lowering the resolution of the feature map while generating a feature map (Feature Map) in which the feature amount of the input image I is extracted. Specifically, in the encoding processing, processing is performed on the input image I in the convolution layer and the pooling layer. In the convolutional layer, a kernel (filter) for extracting a feature amount of the input image I is moved at a predetermined stride in the input image I. Then, in the convolution layer, a convolution calculation for extracting a feature amount of the input image I is performed based on the weight of the convolution layer, and a feature map from which the feature amount is extracted by the convolution calculation is generated. The generated feature maps are generated in a number corresponding to the number of channels of the kernel. The pooling layer reduces the feature map from which the feature amount has been extracted, and generates a feature map having a low resolution. In the encoding process, a process in the convolutional layer and a process in the pooling layer are repeatedly performed a plurality of times to generate a feature map having a down-sampled feature amount.

  The decoder 23 performs a decoding process on the encoded feature map. The decoding process is a process of executing upsampling (also referred to as ampling) for increasing the resolution of the feature map. Specifically, the decoding process is performed on the feature map in the deconvolution layer and the amplifying layer. In the amplifying layer, a low-resolution feature map including a feature amount is enlarged to generate a high-resolution feature map. In the deconvolution layer, a deconvolution calculation for restoring the feature amount included in the feature map is executed based on the weight of the deconvolution layer, and a feature map in which the feature amount is restored by this calculation is generated. In the decoding process, a first output image O1 that is an up-sampled and region-divided image is generated by repeatedly performing the processing in the amplifying layer and the processing in the deconvolution layer a plurality of times. The first output image O1 is up-sampled until it has the same resolution as the input image I input to the image recognition unit 7.

  As described above, the first image recognition unit 11 performs the encoding process and the decoding process on the input image I, and performs the class inference (class classification) on a pixel-by-pixel basis, whereby the image segmentation of the input image I is performed. I do. Then, the first image recognition unit 11 outputs, as a first output image O1, an image obtained by dividing the input image I into regions for each class.

  Note that the first image recognition unit 11 has been described as applied to image segmentation using a bounding box, but is not particularly limited. The first image recognition unit 11 may execute, for example, image segmentation using a different network as long as it can execute a task for obtaining the position of an object included in the input image I. In addition, the first image recognition unit 11 has executed the encoding process and the decoding process. However, if the task for obtaining the position of the object included in the input image I can be executed, the pooling layer included in the encoding process includes The ampling layer included in the decoding process may be omitted.

  Further, as will be described in detail later, the first image recognition unit 11 learns using a large amount of the first teacher image T1 at the time of learning. For this reason, since the first image recognition unit 11 can learn various fluctuation factors with respect to the input image I, the first image recognition unit 11 executes image recognition while ensuring robustness. For example, the robustness of an in-vehicle recognition camera means various fluctuations including illuminance fluctuations such as backlight and dark places, climate fluctuations, lens fluctuations such as raindrops, mud and scratches, and running space fluctuations such as snow and road surface reflection. Resistance to fluctuation factors.

  The output side of the first image recognition unit 11 is connected to the input side of the second image recognition unit 12. For this reason, the first image recognition unit 11 inputs the first output image O1 to the second image recognition unit 12. In addition, the first image recognition unit 11 outputs the first output image O1 to the outside as an intermediate image.

  The second image recognizing unit 12 executes a task of performing the area division of the input image I more precisely than the first image recognizing unit 11. The second image recognition unit 12 performs image segmentation using, for example, semantic segmentation. In the semantic segmentation, class inference is performed for each pixel of the input image I, and as a result of the inference, a class is labeled for each pixel to perform region division of the input image I. The input image I and the first output image O1 are input to the second image recognition unit 12. When the input image I and the first output image O1 are input, the second image recognizing unit 12 uses the first output image O1 to convert an image that has been classified into classes for each pixel of the input image I, Output as the second output image O2. That is, the second image recognizing unit 12 uses the first output image O1 as a hint to divide the input image I more densely than the first image recognizing unit 11, and to perform the second An output image O2 is output.

  The second image recognition unit 12 performs image segmentation using a neural network including a convolutional layer such as a CNN (Convolution Neural Network) or an FCN (Fully Convolutional Network) (hereinafter, simply referred to as a network). In addition, the second image recognition unit 12 performs a feature amount extraction process of extracting a feature amount of the input image I. Further, the second image recognition unit 12 performs a fusion process for integrating the first output image O1 and the image on which the feature amount extraction process is performed, and performs class inference on a pixel-by-pixel basis.

  The feature amount extraction process is a process of extracting the feature amount of the input image I in a plurality of convolution layers, and is substantially the same as the processing of the convolution layer in the encoder 22. Further, the feature amount extraction processing is processing in which the pooling layer is omitted. In the convolutional layer, a convolution calculation for extracting a feature amount of the input image I is executed based on the weight of the convolutional layer, and a feature map from which the feature amount is extracted by this calculation is generated. In the feature extraction process, a convolution calculation is performed on the input image I a plurality of times to generate a feature map.

  In the fusion process, the first output image O1 is used as a hint to merge the feature maps on which the feature amount extraction process is performed, and to perform class inference, thereby generating an image divided into regions for each class. To generate a second output image O2 having the same resolution as that of.

  As described above, the second image recognition unit 12 performs the feature amount extraction processing and the fusion processing on the input image I, and performs class inference (class classification) on a pixel-by-pixel basis using the first output image O1 as a hint. ) To perform the image segmentation of the input image I. In addition, the second image recognition unit 12 outputs the input image I that has been subjected to the image segmentation as a second output image O2.

  The second image recognition unit 12 has been described as applied to image segmentation using semantic segmentation, but is not particularly limited. If the second image recognizing unit 12 can execute a task of performing the area division of the input image I more precisely than the first image recognizing unit 11, for example, it executes image segmentation using a different network. It may be something.

  As will be described later in detail, the second image recognition unit 12 learns at the time of learning using the first output image O1 and the high-resolution second teacher image that is finely divided into regions. I have. For this reason, since the second image recognition unit 12 can perform fine area division on the input image I, the second image recognition unit 12 performs image recognition while ensuring image recognizability (recognition accuracy).

  The second image recognition unit 12 outputs the second output image O2 to the outside. In addition, the second image recognition unit 12 can output the first output image O1 used when generating the second output image O2 in association with the second output image O2.

  As described above, since the first image recognition unit 11 does not need to perform a fine area division as compared with the second image recognition unit 12, it is a task with low difficulty and low calculation load. Further, since the second image recognition unit 12 performs the area division of the input image I using the first output image O1, the difficulty level is higher than the semantic segmentation in which the image segmentation is performed only from the input image I. Task with low computational load.

  Next, learning of the image recognition device 1 will be described. The learning of the image recognition device 1 uses a learning data set. The learning data set includes a learning image, which is an image to be learned, and a teacher image corresponding to the learning image. The learning image is a digital image, like the input image. The teacher image is an image that is an answer that has been subjected to image segmentation corresponding to the learning image, that is, an image obtained by region segmentation. The teacher image is an image generated by the annotation work.

  FIG. 5 is a diagram illustrating an example of the first learning data set. FIG. 6 is a diagram illustrating an example of the second learning data set. The learning data set includes a first learning data set D1 used for learning of the first image recognizing unit 11 and a second learning data set D2 used for learning of the second image recognizing unit 12.

  As shown in FIG. 5, the first learning data set D1 includes a first learning image G1 and a first teacher image T1. The first learning image G1 is an image to be learned by the first image recognition unit 11, and is a digital image, like the input image. The first teacher image T1 is an image that is divided into regions for each class using a bounding box. The first teacher image T1 illustrated in FIG. 5 includes, for example, a rectangular image region T1a classified into a person class and a rectangular image region T1b classified into a car class.

  The second learning data set D2 includes a second learning image G2 and a second teacher image T2. The second learning image G2 is an image to be learned by the second image recognition unit 12, and is a digital image like the input image and the first learning image G1. In FIGS. 5 and 6, the first learning image G1 and the second learning image G2 are the same image for the sake of simplicity, but may be different images. The second teacher image T2 is an image that is divided into regions by pixel in units of classes. In the second teacher image T2 shown in FIG. 6, for example, an image region T2a classified into a person class, an image region T2b classified into a car class, and an image region T2c classified into a road class are included. Contains.

  The first teacher image T1 is an image that has been subjected to image segmentation that results in coarser area division than the second teacher image T2. In other words, the second teacher image T2 is an image that has been subjected to image segmentation that results in a finer area division than the first teacher image T1. The first teacher image T1 and the second teacher image T2 are images generated by the annotation work. Specifically, the first teacher image T1 is generated by an annotation operation of classifying the object included in the first learning image G1 by surrounding the object with a bounding box. The second teacher image T2 is generated by an annotation work of performing class classification on a pixel-by-pixel basis in the first learning image G1. Therefore, the first teacher image T1 has a lower work load and a lower annotation cost than the second teacher image T2. In other words, the second teacher image T2 has a higher work load and a higher annotation cost than the first teacher image T1.

  Further, the first teacher image T1 prepared for learning by the image recognition device 1 is larger than the second teacher image T2. In other words, the amount of the prepared second teacher image T2 is smaller than that of the first teacher image T1. That is, a large amount of the first learning data set D1 including the first teacher image T1 with low annotation cost is prepared, and the learning of the first image recognition unit 11 is performed. Further, the second image recognition unit 12 learns by preparing a small amount of the second learning data set D2 including the second teacher image T2 having a high annotation cost.

  Next, with reference to FIG. 7 to FIG. 9, a process related to learning of the image recognition device 1 using the first learning data set D1 and the second learning data set D2 will be described. 7 to 9 are diagrams illustrating an example of processing related to image learning of the image recognition device. In the learning of the image recognition device 1, the learning of the second image recognition unit 12 is performed after the learning of the first image recognition unit 11 is performed.

  With reference to FIG. 7, a process of performing learning of the first image recognition unit 11 using the first learning data set D1 will be described. In the process of performing learning of the first image recognition unit 11, the first learning image G1 is input to the first image recognition unit 11, and the first image recognition unit 11 performs image segmentation of the first learning image G1. Then, the step of acquiring the first output image O1 (fifth step) is performed.

  Specifically, the first learning image G1 of the first learning data set D1 is input to the first image recognition unit 11 of the image recognition device 1 (Step S1). When the first learning image G1 is input, the first image recognition unit 11 executes an encoding process on the first learning image G1 using the first learning image G1 as an input image (Step S2). . The first image recognition unit 11 generates a low-resolution feature map including the down-sampled feature amounts by executing the encoding process. The first image recognition unit 11 executes a decoding process on the feature map including the down-sampled low-resolution feature amount (step S3). The first image recognizing unit 11 performs a decoding process to perform upsampling while restoring a feature map including a feature amount, thereby obtaining the same resolution as that of the first learning image G1. Then, the first image recognition unit 11 executes a class inference for dividing the image into regions on a pixel-by-pixel basis for each class (step S4). The first image recognition unit 11 obtains a first output image O1 as a result of the class inference (Step S5).

  Next, in the learning process of the first image recognition unit 11, a step of acquiring a first error of the first output image O1 with respect to the first teacher image T1 (Step S6: sixth step) is executed. I do.

  Specifically, in step S6, when the first image recognition unit 11 obtains the first output image O1, it obtains the first teacher image T1 of the first learning data set D1. The first image recognition unit 11 sets an error amount between the first teacher image T1 and the first output image O1 as a first error based on the obtained first teacher image T1 and the first output image O1. calculate. The error amount is calculated by performing an error calculation using the Cross Entropy function.

  Then, in the process of learning the first image recognition unit 11, a step (seventh step) of correcting the image segmentation by the first image recognition unit 11 based on the first error is executed.

  Specifically, when the first image recognition unit 11 obtains the first error, the first image recognition unit 11 corrects the error in the network by the error backpropagation method based on the error amount, so that the convolutional layer and the deconvolutional layer of the network can be corrected. The weight is learned and the network is updated (step S7). The first image recognition unit 11 ends the learning using the first learning data set D1 by executing step S7. Then, the first image recognition unit 11 repeatedly executes steps S1 to S7 according to the number of the first learning data sets D1.

  Next, with reference to FIGS. 8 and 9, a process of learning the second image recognition unit 12 using the second learning data set D2 will be described. In the process of learning the second image recognition unit 12, the first image recognition unit 11 has already learned, and the first output image O1 output from the first image recognition unit 11 is used. In the process of learning the second image recognizing unit 12, the second learning image G2 is input to the first image recognizing unit 11, and the first image recognizing unit 11 performs image segmentation of the second learning image G2. Then, the step of obtaining the first output image O1 (first step) is performed.

  Specifically, as shown in FIG. 8, the second learning image G2 of the second learning data set D2 is input to the first image recognition unit 11 of the image recognition device 1 (Step S11). When the second learning image G2 is input, the first image recognition unit 11 performs an encoding process on the second learning image G2 using the second learning image G2 as an input image (Step S12). . The first image recognition unit 11 generates a low-resolution feature map including the down-sampled feature amounts by executing the encoding process. The first image recognition unit 11 performs a decoding process on the feature map including the down-sampled feature amount (Step S13). The first image recognition unit 11 performs up-sampling while restoring a low-resolution feature map including a feature amount by executing a decoding process to obtain the same resolution as that of the second learning image G2. Then, the first image recognition unit 11 executes a class inference for dividing the image into regions on a pixel-by-pixel basis for each class (step S14). The first image recognition unit 11 obtains a first output image O1 as a result of the class inference (Step S15).

  Next, in the process of performing learning of the second image recognition unit 12, the second learning image G2 and the first output image O1 are input to the second image recognition unit 12, and the first output image O1 is processed. The second image recognizing unit 12 performs image segmentation of the second learning image G2 by using the second image recognition unit 12 to obtain a second output image O2 (second step).

  Specifically, as shown in FIG. 9, the second learning image G2 of the second learning data set D2 is input to the second image recognition unit 12 of the image recognition device 1 (Step S21). When the second learning image G2 is input, the second image recognition unit 12 executes a feature amount extraction process on the second learning image G2 using the second learning image G2 as an input image (Step S10). S22). The second image recognition unit 12 generates a feature map including a feature amount by executing a feature amount extraction process. In addition, the second image recognition unit 12 performs a fusion process on the feature map including the feature amount (Step S23). The second image recognizing unit 12 executes the fusion process to restore the feature map on which the feature amount extraction process is performed using the first output image O1 as a hint. Then, the second image recognizing unit 12 executes class inference that divides the area for each class on a pixel basis from the feature map (step S24). The second image recognition unit 12 acquires a second output image O2 as a result of the class inference (Step S25).

  Next, in the learning process of the second image recognition unit 12, a step of acquiring a second error of the second output image O2 with respect to the second teacher image T2 (Step S26: a third step) is executed. I do.

  Specifically, in step S26, when acquiring the second output image O2, the second image recognition unit 12 acquires the second teacher image T2 of the second learning data set D2. The second image recognizing unit 12 sets an error amount between the second teacher image T2 and the second output image O2 as a second error based on the acquired second teacher image T2 and the second output image O2. calculate. The error amount is calculated by performing an error calculation using the Cross Entropy function.

  Then, in the process of learning by the second image recognition unit 12, a step (fourth step) of correcting the image segmentation by the second image recognition unit 12 based on the second error is executed.

  Specifically, when the second image recognition unit 12 acquires the second error, the second image recognition unit 12 learns the weight of the convolutional layer of the network so that the error in the network is corrected by the error backpropagation method based on the error amount. The network is updated (step S27). Here, in step S27, in the learning based on the second error, the learning of the first image recognition unit 11 is interrupted while the learning of the second image recognition unit 12 is performed. That is, while the second error is backpropagated to the second image recognition unit 12, it is not backpropagated to the first image recognition unit 11. For this reason, in step S27, while the network in the second image recognition unit 12 is corrected for errors, the network in the first image recognition unit 11 is not corrected for errors. The second image recognition unit 12 ends the learning using the second learning data set D2 by executing step S27. Then, the second image recognition unit 12 repeatedly executes steps S21 to S27 according to the number of the second learning data sets D2.

  As described above, in the learning of the image recognition device 1, the first image recognition unit 11 is trained by using a large amount of the first learning data set D1 with low annotation cost. In the learning of the image recognition device 1, the second image recognition unit 12 is trained by using a small amount of the second learning data set D2 having a high annotation cost.

  Next, with reference to FIGS. 10 and 11, image recognition by the learned image recognition device 1 will be described. FIG. 10 and FIG. 11 are diagrams illustrating an example of processing related to image recognition of the image recognition device. In a process related to image recognition of the image recognition device 1, the input image I is input to the first image recognition unit 11, and the first image recognition unit 11 performs image segmentation of the input image I, and outputs the first output image O1. Is executed (eighth step).

  Specifically, as shown in FIG. 9, the input image I is input to the image recognition device 1 (Step S31). When the input image I is input, the first image recognition unit 11 performs an encoding process on the input image I (Step S32). The first image recognition unit 11 generates a low-resolution feature map including the down-sampled feature amounts by executing the encoding process. The first image recognition unit 11 performs a decoding process on the feature map including the down-sampled feature amount (Step S33). The first image recognition unit 11 performs upsampling while restoring a low-resolution feature map including a feature amount by executing a decoding process, and sets the same resolution as the input image I. Then, the first image recognition unit 11 executes a class inference for dividing the image into regions on a pixel-by-pixel basis for each class (step S34). The first image recognition unit 11 obtains a first output image O1 as a result of the class inference (Step S35).

  Next, in the process related to image recognition of the image recognition device 1, the input image I and the first output image O1 are input to the second image recognition unit 12, and the second image is input using the first output image O1. A step (a ninth step) of performing the image segmentation of the input image I by the recognition unit 12 to obtain the second output image O2 is executed.

  Specifically, as shown in FIG. 11, the input image I is input to the second image recognition unit 12 of the image recognition device 1 (Step S41). When the input image I is input, the second image recognition unit 12 performs a feature amount extraction process on the input image I (Step S42). The second image recognizing unit 12 generates a feature map including the feature amount from the input image I by executing the feature amount extracting process. In addition, the second image recognition unit 12 performs a fusion process on the feature map including the feature amount (Step S43). The second image recognizing unit 12 executes the fusion process to restore the feature map on which the feature amount extraction process is performed using the first output image O1 as a hint. Then, the second image recognition unit 12 executes a class inference that divides the area for each class on a pixel-by-pixel basis from the feature map (step S44). The second image recognition unit 12 acquires the second output image O2 as a result of the class inference (Step S45).

  As described above, in the image recognition of the image recognition device 1, the first image recognition unit 11 performs the image recognition while ensuring the robustness. In addition, the first image recognition unit 11 does not need to perform more precise area division than the second image recognition unit 12, and thus performs image recognition using a task with a low calculation load. Then, in the image recognition of the image recognition device 1, the second image recognition unit 12 performs the fine area division, so that the image recognition is performed while ensuring the image recognizability. In addition, the second image recognition unit 12 performs the area division of the input image I by using the first output image O1, and thus performs the image recognition using a task with a low calculation load.

  In addition, the processing illustrated in FIG. 12 is performed as processing related to image recognition of the image recognition device 1. FIG. 12 is a diagram illustrating an example of a process regarding image recognition of the image recognition device. In the process shown in FIG. 12, a step (tenth step) of acquiring the first output image O1 acquired by the image recognition and the second output image O2 corresponding to the first output image O1 in association with each other is executed. I do.

  More specifically, as shown in FIG. 12, the first image recognition unit 11 acquires the first output image O1 as an intermediate image (Step S51). Further, the second image recognition unit 12 acquires a second output image O2 corresponding to the first output image O1 (Step S52). The image recognition device 1 acquires the first output image O1 and the second output image O2 in association with each other (Step S53).

  Then, the obtained first output image O1 and second output image O2 are used when evaluating or analyzing image recognition by the image recognition device 1. For example, when there is a defect such as erroneous recognition in image recognition by the image recognition device 1, by comparing the first output image O1 and the second output image O2, an abnormality in the first image recognition unit 11 is detected. It is possible to estimate whether there is an abnormality in the second image recognition unit 12. That is, when there is an erroneous recognition in the second output image O2, if there is no erroneous recognition in the first output image O1, it can be estimated that there is an abnormality in the second image recognition unit 12. On the other hand, if the second output image O2 has an erroneous recognition, and if the first output image O1 has an erroneous recognition, it can be estimated that the first image recognition unit 11 has an abnormality.

  Next, generation of a learning data set by the learned image recognition device 1 will be described with reference to FIG. FIG. 13 is a diagram illustrating an example of a process regarding generation of a learning data set by the image recognition device. In the process regarding the generation of the learning data set of the image recognition device 1, the second learning data set D2 is generated using the first learning data set D1 that has already been prepared. In the processing related to the generation of the learning data set, the first learning image G1 and the first teacher image T1 are input to the second image recognition unit 12, and the first teacher image T1 is generated by the second image recognition unit 12. A step (eleventh step) of performing image segmentation of the first learning image G1 to obtain a second output image O2 is performed.

  Specifically, as shown in FIG. 13, the first learning image G1 is input to the second image recognition unit 12 of the image recognition device 1 (Step S61). When the first learning image G1 is input, the second image recognition unit 12 executes a feature amount extraction process on the first learning image G1 (Step S62). The second image recognition unit 12 generates a feature map including a feature amount from the first learning image G1 by executing a feature amount extraction process. In addition, the second image recognition unit 12 performs a fusion process on the feature map including the feature amount (Step S63). The second image recognition unit 12 executes the fusion process to restore the feature map on which the feature amount extraction process is performed using the first teacher image T1 as a hint. Then, the second image recognizing unit 12 executes class inference for dividing the area for each class on a pixel-by-pixel basis from the feature map (step S64). The second image recognition unit 12 acquires a second output image O2 as a result of the class inference (Step S65).

  Next, in the process relating to the generation of the learning data set, the image recognition device 1 acquires the first learning image G1 as the second learning image G2. In addition, the image recognition device 1 acquires the second output image O2 as the second teacher image T2. Then, the image recognition device 1 generates a second learning data set D2 including the second learning image G2 and the second teacher image T2 (Step S66: twelfth step). After performing Step S66, the image recognition device 1 ends the generation of the second learning data set D2. Then, the image recognition device 1 generates a plurality of second learning data sets D2 by repeatedly executing steps S61 to S66 a plurality of times while using different first learning data sets D1. Note that a step of selecting a usable second learning data set D2 from the plurality of generated second learning data sets D2 may be added.

  As described above, learning of the image recognition device 1 according to the embodiment can be divided into learning of the first image recognition unit 11 and learning of the second image recognition unit 12. In the learning of the first image recognition unit 11, learning can be performed using a large amount of the first learning data set D1 with low annotation cost. In the learning of the second image recognition unit 12, learning can be performed using a small amount of the second learning data set D2 with high annotation cost. For this reason, the learning of the image recognition device 1 requires only a small amount of the second teacher image T2 having a high annotation cost, so that the annotation cost can be reduced. Thereby, in the learning of the image recognition device 1, the learning efficiency by the teacher images T1 and T2 can be improved.

  In the learning of the first image recognition unit 11, since learning can be performed using a large amount of the first learning data set D1, image recognition with high robustness can be learned. In the learning of the second image recognizing unit 12, since fine image segmentation can be performed, image recognition with high recognition accuracy can be learned. Therefore, in the learning of the image recognition device 1 according to the embodiment, it is possible to learn image recognition with high robustness and high recognition accuracy.

  In the learning of the first image recognition unit 11, since learning can be performed using the first learning data set D1, highly accurate learning suitable for the first image recognition unit 11 can be performed. . Similarly, in the learning of the second image recognition unit 12, since learning can be performed using the second learning data set D2, accurate learning suitable for the second image recognition unit 12 can be performed. it can.

  In the learning of the second image recognition unit 12, since the acquired second error is not propagated to the first image recognition unit 11, the first error is learned by the learning of the second image recognition unit 12. The influence on the image recognition unit 11 can be eliminated.

  Further, the image recognition of the image recognition device 1 according to the embodiment can be divided into image recognition by the first image recognition unit 11 and image recognition by the second image recognition unit 12. In the image recognition of the first image recognition unit 11, image recognition with high robustness can be performed. Further, in the image recognition of the second image recognition unit 12, image recognition with high recognition accuracy can be performed. Therefore, in the image recognition of the image recognition device 1 according to the embodiment, image recognition with high robustness and high recognition accuracy can be performed.

  Further, in the image recognition of the first image recognition unit 11, by executing the task for obtaining the position of the object included in the input image I, the task can be executed as a task with a low calculation load. Also, in the image recognition of the second image recognition unit 12, by executing the fine image segmentation of the input image I using the first output image O1 as a hint, it can be executed as a task with a low calculation load. it can. For this reason, the image recognition device 1 according to the embodiment can perform image recognition at high speed because the calculation load is low.

  In the image recognition performed by the image recognition device 1, the first output image O1 and the second output image O2 can be acquired in association with each other. Therefore, in the evaluation or analysis of the image recognition of the image recognition device 1, it is possible to estimate the abnormality of the first image recognition unit 11 and the second image recognition unit 12.

  In the image recognition device 1, the second learning data set D2 can be generated using the first learning data set D1. For this reason, in the image recognition device 1, since the second learning data set D2 can be automatically generated, the annotation cost for generating the second learning data set D2 can be reduced.

  Note that, in the second image recognition unit 12 of the embodiment, the number of channels of the kernel and the number of channels of the feature map input to the convolutional layer may be smaller than those of the first image recognition unit 11. . The product of the number of channels of the kernel and the number of channels of the feature map input to the convolutional layer is the expressiveness of image recognition. By making the expressive power of the second image recognition unit 12 lower than that of the first image recognition unit 11, the calculation load of the second image recognition unit 12 can be reduced. For this reason, in the image recognition device 1 according to the embodiment, the calculation load can be further reduced, and the image recognition can be performed faster.

Reference Signs List 1 image recognition device 5 control unit 6 storage unit 7 image recognition unit 11 first image recognition unit 12 second image recognition unit 22 encoder 23 decoder I input image O output image P1 image learning program P2 image recognition program P3 learning data set Generation program D1 first learning data set D2 second learning data set

Claims (13)

An image learning program executed by an image recognition device that performs image segmentation,
A learning data set used for learning of the image recognition device includes:
A learning image to be an image to be learned by the image recognition device,
And a teacher image corresponding to the learning image,
The image recognition device,
A first image recognition unit that performs image segmentation of the learning image;
A second image recognition unit that performs image segmentation of the learning image so as to be more densely divided than the first image recognition unit,
A first step of inputting the learning image to the first image recognizing unit, performing image segmentation of the learning image by the first image recognizing unit, and acquiring a first output image;
Inputting the learning image and the first output image to the second image recognition unit, performing image segmentation of the learning image by the second image recognition unit using the first output image, A second step of obtaining a second output image;
A third step of obtaining a second error of the second output image with respect to the teacher image;
A fourth step of correcting an image segmentation process by the second image recognition unit based on the second error;
An image learning program that lets you execute The learning data set includes a first learning data set for learning by the first image recognition unit, and a second learning data set for learning by the second image recognition unit,
The first learning data set includes a first learning image and a first teacher image corresponding to the first learning image,
The second learning data set includes a second learning image and a second teacher image corresponding to the second learning image,
The second teacher image has a smaller number of images than the first teacher image, and is an image that is more precisely divided into regions than the first teacher image,
Before performing the first step, the first learning image is input to the first image recognition unit, and the first image recognition unit performs image segmentation of the first learning image, A fifth step of acquiring the first output image;
A sixth step of obtaining a first error of the first output image with respect to the first teacher image;
Correcting the image segmentation by the first image recognition unit based on the first error.
In the first step, the second learning image is input to the first image recognition unit to obtain the first output image,
In the second step, the second learning image and the first output image are input to the second image recognition unit to obtain the second output image,
The computer-readable storage medium according to claim 1, wherein in the third step, the second error of the second output image with respect to the second teacher image is obtained.   The image learning program according to claim 1, wherein, in the fourth step, correction of image segmentation processing by the first image recognition unit based on the second error is blocked. An image learning method performed by an image recognition device that performs image segmentation,
A learning data set used for learning of the image recognition device includes:
A learning image to be an image to be learned by the image recognition device,
And a teacher image corresponding to the learning image,
The image recognition device,
A first image recognition unit that performs image segmentation of the learning image;
A second image recognition unit that performs image segmentation of the learning image so as to be more densely divided than the first image recognition unit,
A first step of inputting the learning image to the first image recognizing unit, performing image segmentation of the learning image by the first image recognizing unit, and acquiring a first output image;
Inputting the learning image and the first output image to the second image recognition unit, performing image segmentation of the learning image by the second image recognition unit using the first output image, A second step of obtaining a second output image;
A third step of obtaining a second error of the second output image with respect to the teacher image;
A fourth step of correcting an image segmentation process by the second image recognition unit based on the second error;
An image learning method including: An image recognition program executed by an image recognition device that performs image segmentation of an input image that has been input,
The image recognition device,
A first image recognition unit that performs image segmentation of the input image;
A second image recognizing unit that performs image segmentation of the input image so as to be more densely divided than the first image recognizing unit,
An eighth step of inputting the input image to the first image recognition unit, performing image segmentation of the input image by the first image recognition unit, and acquiring a first output image;
Inputting the input image and the first output image to the second image recognition unit, performing image segmentation of the input image by the second image recognition unit using the first output image, A ninth step of obtaining a second output image;
Image recognition program that executes   The image recognition program according to claim 5, further comprising: executing a tenth step of associating the acquired first output image with the second output image corresponding to the first output image. An image recognition method performed by an image recognition device that performs image segmentation of an input image that has been input,
The image recognition device,
A first image recognition unit that performs image segmentation of the input image;
A second image recognizing unit that performs image segmentation of the input image so as to be more densely divided than the first image recognizing unit,
An eighth step of inputting the input image to the first image recognition unit, performing image segmentation of the input image by the first image recognition unit, and acquiring a first output image;
Inputting the input image and the first output image to the second image recognition unit, performing image segmentation of the input image using the first output image by the second image recognition unit, A ninth step of obtaining a second output image;
An image recognition method including: A learning data set generation program that is executed by an image recognition device that performs image segmentation of an input image that has been input, and generates a learning data set used in the image recognition device.
The image recognition device,
A first image recognition unit that performs image segmentation of the input image;
A second image recognizing unit that performs image segmentation of the input image so as to be more densely divided than the first image recognizing unit,
The learning data set includes a first learning data set for learning by the first image recognition unit, and a second learning data set for learning by the second image recognition unit,
The first learning data set includes a first learning image and a first teacher image corresponding to the first learning image,
The second learning data set includes a second learning image and a second teacher image corresponding to the second learning image,
The second teacher image is an image that is more precisely divided into regions than the first teacher image,
The first learning image and the first teacher image are input to the second image recognition unit, and the second image recognition unit uses the first teacher image to generate the first learning image. An eleventh step of performing image segmentation to obtain a second output image;
Acquiring the first learning image as the second learning image, acquiring the second output image as the second teacher image, and combining the second learning image with the second teacher image. A twelfth step of generating a second training data set comprising:
A program for generating a training data set to execute. A method for generating a learning data set, which is executed by an image recognition device that performs image segmentation of an input image that has been input and generates a learning data set used in the image recognition device,
The image recognition device,
A first image recognition unit that performs image segmentation of the input image;
A second image recognizing unit that performs image segmentation of the input image so as to be more densely divided than the first image recognizing unit,
The learning data set includes a first learning data set for learning by the first image recognition unit, and a second learning data set for learning by the second image recognition unit,
The first learning data set includes a first learning image and a first teacher image corresponding to the first learning image,
The second learning data set includes a second learning image and a second teacher image corresponding to the second learning image,
The second teacher image is an image that is more precisely divided into regions than the first teacher image,
The first learning image and the first teacher image are input to the second image recognition unit, and the second image recognition unit uses the first teacher image to generate the first learning image. An eleventh step of performing image segmentation to obtain a second output image;
Acquiring the first learning image as the second learning image, acquiring the second output image as the second teacher image, and combining the second learning image with the second teacher image. A twelfth step of generating a second training data set comprising:
A method for generating a training data set containing   A learning data set generation program according to claim 8, or a learning data set generated by the learning data set generation method according to claim 9. A first image recognition unit that performs image segmentation of the input image;
A second image recognizing unit that performs image segmentation of the input image so as to be more densely divided than the first image recognizing unit,
When the input image is input, the first image recognition unit performs image segmentation of the input image to generate a first output image, and generates the generated first output image as the second output image. Output to the image recognition unit,
The second image recognition unit, when the input image and the first output image are input, performs image segmentation of the input image by the second image recognition unit using the first output image. An image recognition device that outputs a second output image.   The image recognition device according to claim 11, wherein the first image recognition unit performs image segmentation using a bounding box.   The image recognition device according to claim 11, wherein the second image recognition unit performs image segmentation by semantic segmentation.

Images