JP2021068141A - Region dividing device, dividing method, and dividing program - Google Patents

Abstract

To solve a problem in that division results easily vary due to diversification of learning data of a classifier, etc., in region division in which data groups distributed in a space are classified into a plurality of classes and the space is divided into label regions.SOLUTION: As a classifier that outputs a label region for an interest class group after performing a classification process as a result of input of a data group and the interest class group, model storage means 42 stores a learned model in which learning is performed using a learning data group and a learning interest class group given in the forms of a correct class thereof and a subset of the correct class. Interest class setting means 54 sets an interest class group for the data group in a plurality of ways. For each of the plurality of ways of setting the interest class group, region dividing means 55 obtains a label region by means of the classifier, and, among region division results of a space based on the label region, selects, as a region division result for the data group, one satisfying a predetermined condition for a degree of matching between the label region composing the region division result and the space.SELECTED DRAWING: Figure 6

Description

The present invention relates to a technique for classifying a data group such as an image into a class such as a subject and dividing the data group into label areas.

For the purpose of automatically recognizing the scene captured in the image, the image is divided into regions for each of the plurality of objects captured in the image and regions for each of the plurality of parts, and the images are captured in each region. Technology for recognizing objects and parts has been researched and developed. Hereinafter, the object or part being photographed is referred to as a subject. Area division that involves recognition of the subject is called semantic segmentation or the like.

In particular, in recent years, techniques for performing the above division and recognition based on learning have been actively studied. That is, for example, in Non-Patent Document 1 below, a large number of learning images to which a class representing a subject is given for each pixel of a region divided in advance for each subject are prepared, and a computer is made to machine-learn these learning images. It is stated that. The information given in advance is called an annotation or the like. If an arbitrary image is input to the trained model generated by this learning, a class for each pixel is output for the input image. That is, the input image is divided into a class-labeled area (label area) for each subject.

Further, in recent years, a data set consisting of learning images and annotations has been made public and available. Basically, the more diverse the trained model is, the more accurate the region division can be performed. Therefore, it is desirable that the scale of the data set used for training is large.

“Fully Convolutional Networks for Semantic Segmentation”, Jonathan Long, Evan Shelhamer, and Trevor Darrell (Proceedings of the IEEE conference on computer vision and pattern recognition, 2015)

However, there is a problem that the area division result tends to fluctuate due to the diversity of learning data and the mixture of annotations with different assignment criteria. In addition, the mixture of annotations with different assignment criteria has also been a cause of deterioration in learning accuracy.

For example, using an image of a black carpet and an image of similar asphalt for learning, not only to properly divide the area of the floor with the black carpet from the area of the floor, but also part or all of it on the road. In some cases, the area may be accidentally divided. This is an example in which the domain division result tends to fluctuate due to the diversity of learning.

Further, for example, when an image of a baseball field is input, the turf area in the image may be divided as a grass area, or the turf area in the image may be divided as a part of the game field area. In some cases. This is an example in which the area division result tends to fluctuate due to the mixture of annotations with different assignment criteria. For example, in a publicly available dataset, one of the learning images of a baseball stadium has a label representing "grass" in the turf area and a label representing "soil" in the soil area. However, in another learning image of a baseball stadium, different grant criteria are mixed, such as a label representing "amusement park" is given to the combined area of turf and soil. There is. In other words, both grass and the amusement park are correct answers for the turf area. Therefore, fluctuations due to differences in input images are likely to occur.

On the other side, the existence of multiple correct answers, such as the turf domain example, makes learning difficult to converge. Therefore, the mixture of annotations with different assignment criteria is also a factor in lowering the learning accuracy.

The above problem may occur not only in a two-dimensional image but also in spatiotemporal data formed from a time-series image, three-dimensional data such as a point cloud, and the like.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a region division technique capable of suppressing fluctuations in the region division result.

(1) The area dividing device according to the present invention is a device that performs a classification process for classifying a data group distributed in a predetermined space into a plurality of classes and divides the space into label areas identified by the classes. As a classifier in which the data group and the attention class group are input, the classification processing for the data group is performed, and the label area for the attention class group is output, the training data group and the learning data group are obtained in advance. A model storage means that stores a given correct answer class and a trained model that has been trained using the learning attention class group given by a subset of the correct answer class, and the attention to the data group. For each of the attention class setting means for setting a plurality of class groups and the plurality of settings of the attention class group by the attention class setting means, the label area is obtained by the classifier, and the area of the space based on the label area is obtained. Among the division results, there is a region division means for selecting a division result that satisfies a predetermined condition regarding the degree of matching between the label region and the space constituting the region division result as the region division result for the data group.

(2) In the region dividing device according to (1) above, the attention class setting means sequentially adds the supplementary class to the attention class group to set a new attention class group, thereby sequentially setting the attention class. Is set in a plurality of ways, and the area dividing means refers to the data group in which the size of the label area output by the classifier for the plurality of types of the attention class group is equal to or larger than a predetermined reference. It can be configured to be selected as the area division result.

(3) In the region dividing device according to (2) above, the trained model is used as an estimator for estimating the supplementary class by inputting the data group and the attention class group, and further for the learning data group. It is possible to configure the learning by using the correct answer of the supplementary class of.

(4) The area division method according to the present invention is a method of classifying a data group distributed in a space into a plurality of classes and dividing the space into label areas identified by the classes. It is given in advance to the training data group and the learning data group as a classifier in which the group and the attention class group are input, the classification processing for the data group is performed, and the label area for the attention class group is output. A step of preparing a trained model in which training is performed using a correct class and a learning attention class group given by a subset of the correct answer class, and a plurality of the attention class groups for the data group are set. The label area is obtained by the classifier for each of the attention class setting step and the plurality of settings of the attention class group in the attention class setting step, and among the area division results of the space based on the label area, the said It has a region division step of selecting a data group that satisfies a predetermined condition regarding the degree of matching between the label region and the space that constitutes the region division result as the region division result.

(5) The area division program according to the present invention is a program in which a computer performs a processing of classifying a data group distributed in a space into a plurality of classes and dividing the space into label areas identified by the classes. The computer is used as a classifier that inputs the data group and the attention class group, performs the classification process for the data group, and outputs the label area for the attention class group. A model storage means that stores a trained model that has been trained using a class of correct answers given in advance to the data group for learning and a group of attention classes for learning given by a subset of the classes of correct answers. The label area is obtained by the classifier for each of the attention class setting means for setting a plurality of the attention class groups for the data group and the plurality of settings of the attention class group by the attention class setting means. Among the area division results of the space based on the label area, those satisfying a predetermined condition regarding the degree of matching between the label area and the space constituting the area division result are selected as the area division result for the data group. It functions as an area division means.

According to the present invention, it is possible to suppress fluctuations in the result of region division.

It is a block diagram which shows the schematic structure of the image processing system which concerns on embodiment of this invention. It is a schematic functional block diagram of the classifier in the 1st Embodiment of this invention. It is a schematic diagram explaining the generation process of a synthetic feature amount. It is a schematic functional block diagram as a learning apparatus of the image processing system which concerns on 1st Embodiment. It is a schematic flow chart of the operation at the time of learning of the image processing system which concerns on 1st Embodiment. It is a schematic functional block diagram as the area division apparatus of the image processing system which concerns on 1st Embodiment. It is a schematic flow chart of the operation in the area division processing of the image processing system which concerns on 1st Embodiment. It is a schematic diagram for demonstrating the processing example of the area division processing of the image processing system which concerns on 1st Embodiment. It is a schematic flow chart of the operation in the area division processing of the image processing system which concerns on 2nd Embodiment.

Hereinafter, embodiments of the present invention (hereinafter referred to as embodiments) will be described with reference to the drawings.

<< First Embodiment >>
The present embodiment is an image processing system 1 in which a photographing unit and a display unit are connected to a computer, and the image processing system 1 operates as an area dividing device and a learning device thereof.

The area dividing device according to the present invention performs a classification process for classifying target data distributed in a predetermined space into a plurality of classes and divides the space into label areas identified by the classes, and is an example in the present embodiment. The area division device shown as, divides the image obtained by capturing the monitoring space into areas. That is, in the present embodiment, the target data to be classified is the pixels constituting the two-dimensional image, and the divided space is a two-dimensional space having a finite size corresponding to the image. Further, the area dividing device includes a classifier that performs classification processing and an estimator for the classes included in the target data. The learning device learns the classifier and estimator used in the area dividing device. In the present embodiment, the classifier is configured by sharing a part of the classifier, and hereinafter, the term classifier is basically used in a broad sense including the classifier. That is, hereinafter, unless otherwise specified, the classifier means an integrated configuration of the above-mentioned classifier in a narrow sense and an estimator.

[Configuration of image processing system 1]
FIG. 1 is a block diagram showing a schematic configuration of an image processing system 1. The image processing system 1 includes a photographing unit 2, a communication unit 3, a storage unit 4, an image processing unit 5, and a display unit 6.

The photographing unit 2 is a camera that acquires an image that is a collection of target data, and is a surveillance camera in the present embodiment. The photographing unit 2 is connected to the image processing unit 5 via the communication unit 3, photographs the monitoring space at predetermined time intervals to generate an image, and sequentially inputs the generated images to the image processing unit 5. For example, the photographing unit 2 is installed on an indoor wall which is a monitoring space with a predetermined fixed field of view overlooking the monitoring space, and photographs the monitoring space with a frame period of 1 second to generate a color image. The photographing unit 2 may generate a monochrome image instead of the color image.

The communication unit 3 is a communication circuit, one end of which is connected to the image processing unit 5 and the other end of which is connected to the photographing unit 2 and the display unit 6. The communication unit 3 acquires an image from the photographing unit 2 and inputs it to the image processing unit 5. Further, the communication unit 3 inputs the classification result into the class and the division result into the label area from the image processing unit 5 and outputs the result to the display unit 6.

The photographing unit 2, the communication unit 3, the storage unit 4, the image processing unit 5, and the display unit 6 are appropriately connected in a form according to the installation location of each unit. For example, when the photographing unit 2, the communication unit 3, and the image processing unit 5 are installed remotely, the photographing unit 2 and the communication unit 3 can be connected by an internet line. Further, the communication unit 3 and the image processing unit 5 can be connected by a bus. In addition, a LAN (Local Area Network), various cables, or the like can be used as the connection means.

The storage unit 4 is a memory device such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and stores various programs and various data. For example, the storage unit 4 stores learning data and information of a classifier that is a learned model, and inputs and outputs such information to and from the image processing unit 5. That is, information used for learning the classifier, information required for the classification process, information generated in the process of the process, and the like are input / output between the storage unit 4 and the image processing unit 5.

The image processing unit 5 is composed of arithmetic units such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an MCU (Micro Control Unit), and a GPU (Graphics Processing Unit). The image processing unit 5 operates as various processing means / control means by reading a program from the storage unit 4 and executing the program, reads various data from the storage unit 4 as necessary, and stores the generated data in the storage unit 4. Remember. For example, the image processing unit 5 learns and generates a classifier, and stores the generated classifier in the storage unit 4 via the communication unit 3. Further, the image processing unit 5 classifies the pixels constituting the image from the photographing unit 2 into classes using a classifier, and divides the image.

The display unit 6 is a liquid crystal display, an organic EL (Electro-Luminescence) display, or the like, and displays a classification result or a division result input from the image processing unit 5 via the communication unit 3.

[Classifier configuration]
FIG. 2 is a schematic functional block diagram of the above-mentioned broadly defined classifier. In the explanation of the configuration of this classifier, for the convenience of distinguishing between a classifier in a narrow sense and an estimator, a classifier in a narrow sense is simply described as a classifier, and a classifier in a broad sense is described as a classifier / estimator. The classifier / estimator inputs the image and the class of interest information, and the classifier classifies each pixel of the image and outputs the result, while the estimator estimates the supplementary class. The result is output. The attention class information is information that specifies the attention class group.

Here, the class of interest is a class for which a label area is obtained by a classifier, and is basically composed of one or a plurality of classes. That is, the attention class is set among the plurality of classes predetermined as the classification target, and the set having one or more classes set as the attention class as elements is the attention class group. Then, the classifier identifies the area corresponding to the attention class in the image to be divided into areas as the label area, while the area corresponding to the class other than the attention class in the image is identified as the label area of the specific class. Not performed. As a result, the area that does not correspond to the class of interest is treated as, for example, the area of another class.

The supplementary class is a class to be added as a new attention class to the current attention class group. In other words, a new attention class group is set by adding a supplementary class to the current attention class group. By updating the attention class group by adding a supplementary class, basically, the label area corresponding to the attention class group is enlarged and approaches the entire area of the image, but the estimator is a label corresponding to the new attention class group. Estimate a supplementary class in which the region preferably approaches the entire image. The supplementary class is, for example, one of the classes other than the attention class, and by adding it to the attention class group, the size of the area of the other classes that are not labeled areas can be reduced most. it can.

In the present embodiment, the classifier / estimator is composed of a multi-layered network as used in deep learning, and can be modeled by, for example, a convolutional neural network (CNN). The network constituting the classification / estimator of the present embodiment includes a feature amount extraction unit 400, a feature amount information compression unit 401, a feature amount synthesis unit 402, a class classification unit 403, and a supplementary class estimation unit 404. Of these, the feature amount extraction unit 400, the feature amount information compression unit 401, and the feature amount synthesizer unit 402 are shared by the classifier and the estimator, and the configuration in which the class classification unit 403 is connected to the shared part forms the classifier. On the other hand, a configuration in which the supplementary class estimation unit 404 is connected to the shared portion forms an estimator.

In the configuration of the classifier, the feature amount extraction unit 400, the feature amount synthesis unit 402, and the class classification unit 403 have a network structure composed of a plurality of layers connected in series, and this part is referred to as a classifier main unit. Similarly, in the configuration of the estimator, the feature amount extraction unit 400, the feature amount synthesis unit 402, and the supplementary class estimation unit 404 have a network structure composed of a plurality of layers connected in series, and this part is called an estimator main unit. I will decide.

The feature amount extraction unit 400, the classification unit 403, and the supplementary class estimation unit 404 are composed of a convolution layer, an activation function, a pooling layer, and the like. For example, the main part of the classifier repeatedly performs a process of obtaining a feature map that convolves the features of neighboring pixels, aggregates the relationship with surrounding pixels, and further performs a process of identifying a class for the pixels of the original image. .. In the present embodiment, the network structure of the classifier main part and the estimator main part is divided into two parts, a front part and a back part of the feature amount synthesis part 402, by inserting the feature amount synthesis part 402 in the middle of the network structure. The front part of these two parts is the feature amount extraction unit 400 shared by the classifier main part and the estimator main part, while the rear part is the classification unit 403 in the classifier main part. In the main part of the estimator, there is a supplementary class estimator 404.

The feature amount extraction unit 400 inputs an image and calculates the feature amount from the image. The feature amount calculation performed by the feature amount extraction unit 400 can be performed up to a layer in the middle of the feature amount map generated in a plurality of layers, and the processing performed by the class classification unit 403 and the supplementary class estimation unit 404 is relevant. It may include the generation of feature maps after the middle layer.

The class classification unit 403 classifies the pixels based on the composite feature amount generated by the feature amount synthesis unit 402, and performs a process of dividing the image into regions. In the area division, the label area corresponding to the class of interest is output. That is, the class classification unit 403 classifies the class of each pixel, and at that time, outputs the class for the pixel classified in the attention class, and thereby, in the image, the label area consisting of the pixel group belonging to the class. Is obtained and the image is divided into label areas. Specifically, adjacent pixels classified into the same class form one section of the label area of the class. On the other hand, the class classification unit 403 outputs the portion of the image that is not classified into the class of interest as, for example, the "other class" as described above.

The supplementary class estimation unit 404 performs a process of estimating the supplementary class based on the composite feature amount generated by the feature amount synthesis unit 402. For example, the supplementary class estimation unit 404 has the largest area among the classes not included in the attention class group but included in the image based on the composite feature quantity created by the feature quantity synthesis unit 402. Estimate the class as a supplementary class. Further, the supplementary class estimation unit 404 may be configured to output a vector in which a score that can be a attention class is stored for each class and select a supplementary class based on the score. If the group of interest includes all the classes included in the image, the estimation result of "no supplementary class" is returned.

The attention class information compression unit 401 is composed of a fully connected layer and the like, obtains attention class information in a low-dimensional representation, and outputs the information to the feature amount synthesis unit 402. In other words, the class of interest information is set based on what is shown in the image and the scene, but the number of classes that appear in the input image is sufficiently smaller than the number of all classes that the classifier can classify. In addition, since indoor classes are difficult to be included in outdoor images, and walls and floors are likely to be included at the same time indoors, the class of interest information can be represented by relatively low-dimensional information. The attention class information compression unit 401 can perform this low-dimensional conversion processing. For example, the attention class information compression unit 401 inputs attention class information represented by a number of variables corresponding to all the predefined classes, dimensionally compresses the information, and represents attention class information with fewer variables. Convert to and output.

The feature amount synthesis unit 402 synthesizes the feature amount extracted by the feature amount extraction unit 400 with the attention class information compressed by the attention class information compression unit 401 to generate a composite feature amount, and generates a composite feature amount, and classifies the feature amount 403. And input to the supplementary class estimation unit 404.

FIG. 3 is a schematic diagram illustrating the process of generating the composite feature amount. FIG. 3 schematically shows the data in the classifier / estimator shown in FIG. 2, and on the left side of the figure, the feature amount extraction unit 400, the feature amount synthesis unit 402, and the feature amount synthesis unit 402 of FIG. Corresponding to the arrangement of the class classification unit 403, the image 100 input to the classifier, the composite feature amount 110 generated by the feature amount synthesis unit 402, and the class classification result 140 output from the classifier are arranged. Further, on the right side of the figure, the input node 120 of the attention class information compression unit 401, the attention class information 121 input to the node, and the output node 130 of the attention class information compression unit 401 are shown.

Regarding the data of the main part of the classifier arranged on the left side of FIG. 3, the x-axis is taken in the width direction and the y-axis is taken in the height direction of the image 100, and the dimension corresponding to the feature channel is represented by the c-axis. The size of the image 100 is _{H I} pixel _{W I} pixels in the y direction in the x-direction. The feature amount map generated by the feature amount extraction unit 400 W _F pixels in the x direction, the magnitude of H _F pixels in the y-direction, c the direction of the size, i.e. the number of channels is the C channel. Incidentally, the feature maps of x, y-direction size does not match the general size of the image 100, becomes _normal, W _F <W _I, and _H F <H _I.

The attention class information 121 illustrated in FIG. 3 is information indicating whether or not each class is a attention class for each of the predetermined N classes. For example, all classes targeted by the classifier are set as the N class.

Specifically, the attention class information 121 is an N-dimensional vector in which the attention class has a value of "1" and is not the attention class, and therefore the class to be output by replacing it with another class in the classification result is represented by "0". The attention class information 121 shows a specific example thereof, and is generated for an image taken indoors. For example, the class of "people" and "floor" is included in the image, so it is regarded as a class of interest, and "1" is set for the corresponding element in the vector, while the class of "road", for example, is included in the image. Since it is not possible, it is not a class of interest, and "0" is set for the corresponding element.

The input node 120 of the attention class information compression unit 401 has a one-to-one correspondence with the elements of the attention class information 121, and the number thereof is N, while the number D of the output nodes 130 is less than N. The attention class information compression unit 401 dimensionally compresses the attention class information 121 input to the input node 120, and outputs the compressed attention class information from the output node 130. That is, the attention class information 121 is compressed from the N-dimensional vector to the D-dimensional vector. Incidentally, FIG. 3 shows a configuration in which the input node 120 and the output node 130 are fully connected as the attention class information compression unit 401.

The feature amount synthesizing unit 402 inputs the attention class information compressed from the output node 130 of the attention class information compression unit 401, synthesizes the feature amount class information with the feature amount map input from the feature amount extraction unit 400, and combines the feature amount synthesis unit 402 with the feature amount map input from the feature amount extraction unit 400. Generates a synthetic feature amount 110. The composite feature amount 110 is obtained by concatenating the attention class information represented by the D-dimensional vector to each of the C-dimensional feature amount vectors specified by the set of x-coordinate and y-coordinate in the feature amount map before synthesis. It has the same width and height as the feature map before synthesis, and has a structure in which the number of channels is (C + D). For example, the 1st to Cth channels of the composite feature amount 110 are the feature amount map before synthesis, and the 1st to 1st D of the output node 130 of the attention class information compression unit 401 are set to the (C + 1) to the (C + D) channels. The output value of the node is set.

In the present embodiment, common attention class information is set for each (x, y) coordinate, so that the structure of the composite feature amount 110 duplicates each of the D elements of the attention class information in the x and y directions. expanding the size of the same W _F × H _F pixels and an output of the feature extraction unit 400, a structure obtained by laminating it to the feature amount map before combining. That is, for example, the feature quantities of the first to C channels may differ depending on the coordinates (x, y), whereas in the present embodiment, all the coordinates of the (C + 1) to (C + D) channels are all coordinates. A common value is set for (x, y).

Hereinafter, regarding the configuration of the image processing system 1, first, the configuration and operation as a learning device will be described, and then the configuration and operation as a region dividing device will be described.

[Configuration as a learning device]
FIG. 4 is a schematic functional block diagram of the image processing system 1 according to the first embodiment as a learning device, in which the storage unit 4 functions as the learning data storage means 40 and the learning model storage means 41, and the image processing unit 5 functions as a correct answer label replacement means 50, a learning attention class generating means 51, a learning supplementary class generating means 52, and a learning means 53.

The learning data storage means 40 stores a large number of images that are learning target data and a class of correct answers given in advance for the images. The learning image and the correct answer class corresponding to each of the images are stored in the learning data storage means 40 in advance prior to the learning process.

The learning model storage means 41 stores a learning model for the classifier. Along with the learning process by the learning means 53, the learning model stored in the learning model storage means 41 is updated. Then, when the learning is completed, the learning model storage means 41 stores the learned model of the classifier and functions as the model storage means 42 described later. As described above, in the present embodiment, the classifier is composed of, for example, a network modeled by CNN, and the learning model storage means 41 includes the filter coefficients and the network structure of the filters constituting the network such as CNN. Store information as a classifier.

The learning means 53 learns the learning model stored in the learning model storage means 41. In the learning, the learning image and the learning attention class information are input to each of the feature amount extraction unit 400 and the attention class information compression unit 401 in the learning model of the classifier, and the correct answer of the classification result of the output of the class classification unit 403. The learning model is updated and learned based on the error with respect to and the error with respect to the correct answer of the supplementary class obtained in the output of the supplementary class estimation unit 404. By the way, the error for the correct answer of the output in the learning of the classifier is a value obtained by integrating the above errors of the classification unit 403 and the supplementary class estimation unit 404 by addition or the like, and the learning can be controlled based on the integrated error. it can.

In the learning means 53, a learning image used for the learning, learning attention class information, and correct answers for each of the classification and the supplementary class are input. Of these, the learning image is input from the learning data storage means 40 to the learning means 53. Further, the learning attention class generating means 51, the correct answer label replacing means 50, and the learning supplementary class generating means 52 input the learning attention class information, the correct answer of the classification, and the correct answer of the supplementary class into the learning means 53, respectively.

The correct answer label replacement means 50 reads out the correct answer class stored in the learning data storage means 40, and performs replacement processing for the correct answer class. In the replacement process, it is assumed that the correct answer class, that is, a part of the class existing in the learning image does not exist. For example, the class to be replaced can be randomly set in the correct class corresponding to each learning image. At this time, instead of randomly replacing each class with a certain probability, a random number of 0 or more and less than or equal to the number of classes included in the correct answer label is generated, and as many classes as the number of the random numbers are randomly selected and replaced. Therefore, it is preferable that the number of classes to be replaced is evenly distributed. Alternatively, instead of randomly replacing, the number of replacements of each class may be counted, and the class with the smaller number of replacements may be preferentially selected and replaced among the correct answer classes corresponding to each learning image.

The process of the correct label replacement means 50 will be described using an image showing a person, a floor, a wall, and a window as an example. Class information can be represented by a vector consisting of elements that have a one-to-one correspondence with each of all classes. This vector will be called a class vector. As for the correct answer class of each pixel, assuming that the number of all classes defined in advance as the classification target is N, the value "1" is set for the element corresponding to the class and the value "0" is set for the other elements. It can be represented by a dimensional class vector. For example, the correct class of pixels showing a person is represented by an N-dimensional vector in which the element of the human class is "1" and the other elements are "0", and the correct class of pixels on the floor is the floor. The elements of the class of are expressed as "1" and the other elements are expressed as "0", and the correct class of the pixel of the wall or window is also expressed in the same manner. The correct label replacement means 50 inputs an N-dimensional vector representing the correct class for each pixel, and outputs an N + 1-dimensional class vector obtained by adding elements corresponding to other classes to the vector for each pixel. If the correct label replacement means 50 does not pay attention to other than the floor, that is, the person, the wall, or the window, the person, the wall, or the window in the N + 1 dimensional class vector of the pixel including the person, the wall, or the window. The element of the class of is replaced with "0", and the element corresponding to the other class is set to "1". Further, "0" is set for the elements corresponding to other classes in the N + 1 dimensional class vector of the pixel including the floor.

By the replacement process, the original correct label area based on the correct answer class before replacement is obtained for the attention class, and the correct answer label area replaced with other classes except for the attention class is obtained, and this replaced correct answer label area is the correct answer label. The replacement means 50 gives the learning means 53 as the correct answer for the classification.

The learning attention class generating means 51 inputs the replaced correct answer label area from the correct answer label replacing means 50, and generates learning attention class information based on the input correct answer label area. The learning attention class generating means 51 sets the class corresponding to the replaced correct label area, that is, the correct answer class remaining after the replacement process by the correct label replacement means 50 as the learning attention class group, and represents the learning attention class group. Generate class information. Incidentally, as learning attention class information, one N-dimensional class vector is generated for each replaced correct label image. For example, the learning attention class information is represented by a class vector in which the element corresponding to the attention class has a value of "1" and the other elements have a value of "0".

The learning supplementary class generating means 52 reads the original correct answer label area from the learning data storage means 40, and the learning attention class information is input from the learning attention class generating means 51, and the correct answer of the supplementary class is input based on them. Generate a supplementary class for learning. By comparing the correct answer class before replacement found from the original correct answer label area with the attention class indicated by the learning attention class information, the correct answer class is replaced by the correct answer label replacement means 50 and is not included in the attention class. You can see the class that is supposed to be. The learning supplementary class generating means 52 inputs the class having the largest area as the correct answer label area among the classes into the learning means 53 as the learning supplementary class. If there is no supplementary class, the learning supplementary class generating means 52 outputs to that effect. The output of the learning supplementary class generation means 52 can be an N + 1 dimension class vector, and the element of the N + 1th dimension can be a supplementary class no flag.

[Operation as a learning device]
The image processing system 1 performs an operation of learning the classifier prior to the operation of dividing the input image into regions. The learning of this classifier will be described below. The learning of the classifier in the image processing system 1 uses the learning image and the learning attention class information, the replaced correct answer label area which is the correct answer data of the class classification, and the learning supplementary class which is the correct answer data of the supplementary class. Based on the integrated error described above, the parameters of the training model are iteratively updated until the error converges using a known optimization method such as the backpropagation error method. By this learning, the classifier (narrow sense) capable of classifying to find the label area corresponding to the attention class and the area of "other classes" not included in the label area corresponding to the attention class in the area division result are reduced. It is possible to train an estimator that estimates a supplementary class. Further, the learning of the classifier includes the operation of learning the attention class information compression unit 401 using the learning attention class information in addition to the learning of the feature amount extraction unit 400, the class classification unit 403, and the supplementary class estimation unit 404. ..

FIG. 5 is a schematic flow chart of the operation of the image processing system 1 during learning.

When the start of the learning operation is instructed, the image processing unit 5 reads the initial setting values of the parameters of the learning model of the classifier from the learning model storage means 41 (step S1), and the learning operation for the model (steps S2 to S10). To start.

The image processing unit 5 acquires the learning image and the correct label of the image from the learning data storage means 40 (step S2). The image processing unit 5 functions as the correct label replacement means 50, randomly selects a class included in the correct label, and replaces it with the “other class” (step S3).

The image processing unit 5 functions as a learning attention class generating means 51, and generates learning attention class information based on the label generated by the correct label replacement means 50 (step S4). For example, the replaced correct label area generated by the correct label replacing means 50 may be composed of N + 1 classes including the “other class”, but the learning attention class generating means 51 is N excluding the “other class”. Output the class vector of the class.

Next, the image processing unit 5 functions as a learning supplementary class generating means 52, and is not included in the learning attention class information based on the original correct answer label and the learning attention class information, but the original correct answer. Among the classes included in the label, the class having the largest area in the correct answer label is set as a supplementary class for learning (step S5). At this time, if no class is replaced by the correct answer label replacement means 50 and all the classes included in the original correct answer label are included in the learning attention class information, a special class that there is no learning supplementary class. To set. That is, the learning supplementary class generating means 52 generates a class vector having N + 1 elements, and outputs a vector in which the element corresponding to the class to be added to the attention class has a value of “1” and the other elements have a value of “0”.

The image processing unit 5 functions as a learning means 53, and updates the parameters of the learning model based on the learning image, the replaced correct answer label area, the learning attention class information, and the learning supplementary class. First, the learning means 53 inputs a learning image and learning attention class information into the learning model, and performs area division and estimation of the supplementary class according to the parameters at the time of input (step S6). After that, the obtained area division result is compared with the correct label to obtain an error (step S7), and further, the error between the estimated supplementary class and the learning supplementary class is obtained (step S8). The learning means 53 updates the parameters of the learning model by a stochastic gradient descent method or the like so that these errors become small (step S9).

In the learning operation, the image processing system 1 repeats the processes of steps S2 to S9 while changing the learning data until the error converges (when “NO” in step S10), and when the error satisfies a predetermined convergence condition (when the error satisfies a predetermined convergence condition (in the case of “NO” in step S10)). In the case of "YES" in step S10), the learning model storage means 41 stores the parameters of the learned model (that is, the classifier), and ends the learning operation (step S11).

The classifier (in a narrow sense) generated by the above learning inputs pixels (data) by inputting attention class information (attention class group) indicating that the classification result is limited to the attention class together with an image (data group). It will be possible to suppress the classification to other than the attention class. In learning, since the class of interest is a subset of the correct class, high-precision class classification that suppresses fluctuations due to the diversity of learning data (for example, eliminates the room for misclassifying floors as roads) is possible. High-precision class classification that can and / or suppress fluctuations due to mixing even if the learning data is a mixture of annotations with different assignment criteria (for example, eliminate the room for classifying turf as a playground and classify it as grass) Will be possible. In addition, learning tends to converge. Therefore, the classifier can perform highly accurate class classification (region division) that suppresses fluctuations due to the diversity of learning data and the mixture of assignment criteria.

In addition, the estimator generated by the above learning can estimate the class in the input image (data group) that is not in the input attention class group, that is, the supplementary class to be added to the attention class group with high accuracy. It becomes.

[Configuration as an area division device]
FIG. 6 is a schematic functional block diagram of the image processing system 1 according to the first embodiment as an area dividing device, in which the storage unit 4 functions as the model storage means 42 and the image processing unit 5 is the attention class setting means 54. And functions as the area dividing means 55. Further, the communication unit 3 cooperates with the image processing unit 5 to function as the image input means 30 and the area information output means 31. Here, since the main processing in the attention class setting means 54 and the area dividing means 55 is performed by using the classifier, the attention class setting means 54 and the area dividing means 55 are designated as the classifier 56 in FIG. 6 for convenience. It is shown in the figure.

The model storage means 42 stores the classifier generated by learning. In the present embodiment, the model storage means 42 is the same as the learning model storage means 41 described above as the configuration of the learning device, and the classifier is the learned model described above.

The image input means 30 sequentially acquires images from the photographing unit 2 and inputs them to the classifier 56.

The area dividing means 55 inputs an image (input image) from the image input means 30, inputs attention class information from the attention class setting means 54, performs class classification processing for each pixel of the input image, and uses the result as a result. Output the label area of the attention class obtained based on. Specifically, the input image and the attention class information are input to the feature amount extraction unit 400 and the attention class information compression unit 401 of the classifier, respectively, and are transferred to the label area based on the class classification result output from the class classification unit 403. The division result is obtained. The area dividing means 55 obtains a label area by a classifier for each of a plurality of settings of the attention class group by the attention class setting means 54, and corresponds to the label area and the input image among the plurality of settings of the attention class group. A label area that satisfies a predetermined condition for the degree of matching with the two-dimensional space is selected as the area division result for the input image.

The attention class setting means 54 sets a plurality of types of attention class groups in which an image is input from the image input means 30 and is input to the area division means 55. In the present embodiment, the attention class setting means 54 sequentially sets a plurality of types of attention class groups by adding a supplementary class to the attention class group and setting a new attention class group. The attention class setting means 54 repeats the sequential setting until the class classification process in the area dividing means 55 satisfies a predetermined condition. The set attention class group is given to the region dividing means 55 as attention class information. Supplemental classes are estimated using a classifier. Specifically, the input image and the attention class information are input to the feature amount extraction unit 400 and the attention class information compression unit 401 of the classifier, respectively, and the estimation result of the supplementary class is obtained from the supplementary class estimation unit 404.

The area information output means 31 outputs the label area obtained by the area dividing means 55 to the display unit 6. For example, the area information output means 31 generates a color-coded image for each label area and outputs the image to the display unit 6.

[Operation as an area division device]
FIG. 7 is a schematic flow chart relating to the operation of the image processing system 1 in the area division processing.

When the image processing system 1 starts the area division processing, the photographing unit 2 sequentially outputs the images captured in the monitoring space at predetermined time intervals. The image processing unit 5 cooperates with the communication unit 3 to repeat the operation shown in the flow chart of FIG. 7 every time an image is received from the photographing unit 2.

In this operation, the communication unit 3 first functions as an image input means 30, and when an image is received, the image is input to the image processing unit 5 (step S20).

The image processing unit 5 inputs the input image (input image) to the feature amount extraction unit 400 of the classifier, and calculates the feature amount of the input image (step S21). In the subsequent processing, the area division processing is performed a plurality of times for one input image while changing the class of interest, and the feature amount calculated here is repeatedly used at that time. In this way, the calculation amount of the image processing unit 5 can be reduced by reusing the feature amount instead of calculating it each time the area division processing is performed.

On the other hand, the initial value of the attention class group set by the attention class setting means 54 is input to the attention class information compression unit 401 of the classifier (step S22). The attention class setting means 54 sets, for example, an attention class group consisting of only one class as the initial value. The attention class setting means 54 sets, for example, P ways (1 ≦ P) of attention class groups. Here, P is predetermined. For example, P = 3. For example, each of the upper P classes of the supplementary class obtained in the output of the supplementary class estimation unit 404 when nothing is set in the attention class can be set as the initial attention class group. Alternatively, the area is divided by setting each of the N all classes as one attention class group, and among the N kinds of attention class groups, the P class is initially focused on in ascending order of the area of the "other class" in the area division result. It can also be a class group.

The attention class setting means 54 and the area division means 55 share a classifier integrated with the estimator, and the update of the attention class group and the area division process are repeated so that a suitable area division result can be obtained (steps S23 to S29). ).

The attention class setting means 54 searches for a suitable attention class group while holding a plurality of attention class groups so that the estimation result of the supplementary class may contain some errors.

The supplementary class is calculated by the supplementary class estimation unit 404. Here, it is assumed that the supplementary class estimation unit 404 outputs as supplementary class information a vector in which a score representing the supplementary class-likeness is stored for each class. The attention class setting means 54 sequentially inputs the held P-like attention class groups into the classifier and obtains supplementary class information for the attention class group. Then, each of the Q (1 ≦ Q <N) classes having the highest score in the supplementary class information is selected as the supplementary class for the attention class group (step S23). Here, Q is predetermined. For example, Q = 3.

The attention class setting means 54 sequentially selects one from the supplementary classes according to Q in step S23 and adds it to the current attention class group to create a trial attention class group (step S24), and the image processing unit 5 is created. A series of processes of inputting the trial attention class group into the classifier and performing the area division processing (step S25) are repeated until the processing for Q supplementary classes for the current attention class group is completed (step S26). In the case of "NO"). In addition, the supplementary class estimation process may be performed together with the area division process in step S25, and the estimation result can be used as the result of step S23 for the attention class group to be updated in the later step S28.

When the processing is completed for all the Q supplementary classes (in the case of "YES" in step S26), the trial attention class group is created according to Q for one of the current attention class groups.

The image processing unit 5 sequentially performs the processes of steps S24 to S26 for the current attention class group according to P (in the case of "NO" in step S27). When the processing is completed for all of the current attention class groups according to P (when “YES” in step S27), the trial attention class group and the region division result thereof are obtained according to P × Q. Of the trial attention class groups, the attention class setting means 54 replaces the attention class group held in P ways with the first to Pth ranks in ascending order of the area of the “other class” in the area division result. By doing so, the attention class group is updated (step S28). With this update, the attention class group according to P will be the current attention class group plus supplementary classes. Further, in this process, the label area corresponding to the class of interest basically expands and approaches the entire area of the image.

The image processing unit 5 completes the operations of steps S23 to S28 of generating P × Q trial attention class groups to which one supplementary class is added and selecting P attention class groups from them. Is repeated until is satisfied (in the case of "NO" in step S29).

The end condition is about the degree of matching between the label area corresponding to the class of interest and the two-dimensional space corresponding to the input image. For example, there is no change or a predetermined value due to the repetition of the index indicating the degree. If it is as follows, it is judged to be finished. Specifically, the end conditions are that the area of the "other class" in the image does not decrease, that the decrease is less than the predetermined value, and that "no supplementary class" is held in all the attention class groups that are held. It can be assumed that a class already included in the attention class group is estimated as a supplementary class, and a new attention class group cannot be obtained. Further, for example, if the index indicating the degree of matching exceeds or falls below a predetermined standard, it is determined to end. Specifically, the end conditions are that the area of the label area of the "other class" is less than or equal to the predetermined ratio of the area of the input image, and that the number of classes already included in the attention class group exceeds the preset maximum number. Can be. Alternatively, the termination condition may satisfy any one of two or more of the above conditions.

When the end condition is satisfied (when "YES" in step S29), the image processing unit 5 determines that the search for the attention class group has been completed, and among the P-like attention class groups obtained as a result. The final result is selected that satisfies the predetermined conditions for the above-mentioned degree of consistency. In the present embodiment, the label area corresponding to the P-like attention class group having the smallest area of the “other class” is output by the area information output means 31 as the area division result of the input image (step). S30).

FIG. 8 is a schematic diagram for explaining a processing example of the area division processing of the image processing system 1. The image 200 of FIG. 8A shows an input image, in which the wall 201, the window 202, the person 203, and the floor 204 on which the black carpet is laid are photographed.

The image 210 of FIG. 8B represents a label area obtained by the prior art with respect to the input image 200. On the other hand, the image 220 of FIG. 8C shows the label area obtained by the image processing system 1 of the present embodiment for the input image 200, and the images 220a to 220c show the process of the area division processing when obtaining the label area. Represents.

According to the processing results of the prior art shown in FIG. 8B, the areas where the wall 201, the window 202, and the person 203 were photographed are correctly the label area 211 of the wall class, the label area 212 of the window class, and the person class, respectively. Although it is divided as the label area 213, the area where the floor 204 is photographed is divided into the label area 214 which is correctly divided as the floor class and the label area 215 which is mistakenly divided as the road class. ..

In the processing example shown in FIG. 8C, the retention number P of the above-mentioned attention class group is set to 1 for the sake of simplicity. In the processing process, the image 220a of the class classification unit 403 of the classifier when a class vector in which the class of "floor" is set to the value "1" and the other classes are set to the value "0" is input as the attention class information. The output is represented, and the label area 224 of the floor class is obtained as the label area, and the other areas are “other classes” indicated by diagonal lines. The supplementary class estimation unit 404 sets the one class with the highest score of the supplementary class information as the supplementary class, so that the supplementary class is set as the "wall" at this stage, and the "floor" and "wall" classes are set to the value "1". "New attention class information is set.

The image 220b shows the output of the class classification unit 403 when the attention class information is input, and the label area 224 of the floor class and the label area 221 of the wall class are obtained as the label areas, and the other areas are obtained. Is the "other class" indicated by the diagonal line. At this time, the supplementary class estimation unit 404 estimates the supplementary class as a “window”, and sets new attention class information in which the “floor”, “wall”, and “window” classes have a value of “1”.

The image 220c shows the output of the class classification unit 403 when the attention class information is input, and as the label area, the label area 224 of the floor class, the label area 221 of the wall class, and the label area 222 of the window class are shown. Is obtained, and the other areas are "other classes" indicated by diagonal lines. At this time, the supplementary class estimation unit 404 sets the supplementary class to "person", and sets new attention class information in which the classes of "floor", "wall", "window", and "person" have a value of "1". ..

The image 220 shows the output of the class classification unit 403 when the attention class information is input, and the input image 200 shows the label area 221 of the wall class, the label area 222 of the window class, and the label area 223 of the person class. And divided into floor class label areas 224. At this stage, the end conditions such as the disappearance of the "other class" area and the setting of "no addition" in the output of the supplementary class estimation unit 404 are satisfied, and the area division process ends.

In this process, by giving the class that can appear in the input image with the attention class information, for example, when the floor is the attention class, the label area 215 of the road having the image feature similar to the floor is set in the image 210. The part of the floor is easily guided to the floor correctly, and misclassification is suppressed.

<< Second Embodiment >>
The image processing system 1B according to the second embodiment of the present invention has the same configuration as that of FIG. 1 of the first embodiment, and is a region dividing device that performs region dividing processing on an input image as in the first embodiment. , And its learning device. Hereinafter, with respect to the second embodiment, the same components as those of the first embodiment are designated by the same reference numerals, and the description in the first embodiment is incorporated. In order to avoid confusion with the constituent elements of the above, a reference numeral having "B" added after the reference numeral of the first embodiment will be used, and the differences from the first embodiment will be mainly described.

The basic difference between the image processing system 1B and the image processing system 1 of the first embodiment is that the classifier is not in the broad sense defined in the first embodiment but in the narrow sense, and is a supplementary class estimator. The point is that it does not require. That is, the classifier of the first embodiment is a combination of a classifier in a narrow sense for classifying and an estimator for estimating a supplementary class in a broad sense, but the classifier in the second embodiment is used. Does not include the function of the estimator. Specifically, the classifier of the second embodiment has a configuration in which the supplementary class estimation unit 404 is omitted in FIG.

Corresponding to the configuration of this classifier, the image processing system 1B as the learning device of the classifier does not require the supplementary class for learning which is the correct answer data for the supplementary class which is the output of the estimator. Specifically, the image processing system 1B has a configuration in which the learning supplementary class generation means 52 is omitted in FIG. 4 as a learning device.

The attention class setting means 54B in the image processing system 1B sets a plurality of attention class groups. In the first embodiment, a plurality of attention class groups are sequentially set by inputting certain attention class information into the classifier and updating the attention class information using the supplementary class obtained in the output. On the other hand, in the present embodiment, a plurality of types of attention class groups are set regardless of the form of sequential update. Specifically, the attention class setting means 54B sets all N classes as attention classes one by one, and sets N ways of attention class groups including the one attention class.

The area dividing means 55B in the image processing system 1B obtains a label area by a classifier for each of a plurality of settings of the attention class group by the attention class setting means 54B, and is a two-dimensional space corresponding to the input image based on the label area. Is divided into areas. Then, among the plurality of types of area division results obtained by the area division process, those that satisfy a predetermined condition regarding the degree of matching between the label area constituting the area division result and the two-dimensional space corresponding to the input image are input. Select as the area division result for the image.

FIG. 9 is a schematic flow chart relating to the operation of the image processing system 1B in the area division processing.

When the image processing system 1B starts the area division processing, the photographing unit 2 sequentially outputs the images captured in the monitoring space at predetermined time intervals. The image processing unit 5 cooperates with the communication unit 3 to repeat the operation shown in the flow chart of FIG. 9 every time an image is received from the photographing unit 2.

In this operation, the communication unit 3 first functions as an image input means 30, and when an image is received, the image is input to the image processing unit 5 (step S40).

The image processing unit 5 inputs the input image (input image) to the feature amount extraction unit 400 of the classifier, and calculates the feature amount of the input image (step S41). Similar to the first embodiment, the feature amount calculated here is repeatedly used in a plurality of area division processes in which the attention class for one input image is changed, whereby the calculation amount of the image processing unit 5 is reduced. To.

The attention class setting means 54B generates the attention class information in which only one class is set as the attention class group by the class vector, and inputs this to the attention class information compression unit 401 (step S42). The area dividing means 55B performs a class classification process by a classifier using the feature amount calculated in step S41, and obtains a label area for the class of interest set in step S42 (step S43).

The image processing unit 5 repeats steps S42 and S43 for all N classes (when “NO” in step S44). As a result, the attention class setting means 54B sets N kinds of attention class groups, and the area dividing means 55B obtains a label area with a classifier for each of the N ways of setting the attention class group.

When the image processing unit 5 finishes the area division processing in which each of the classes is set as the group of interest (in the case of "YES" in step S44), it is determined that there is only the "other class" part of the processing result. The one corresponding to the setting of the attention class group is deleted (step S45).

The area division result left in step S45 has a label area for the class of interest. The area division means 55B combines the remaining area division results to create a combination of label areas so that there is as little overlap between the label areas and the entire image can be filled. This process can be performed by, for example, the following method.

The image processing unit 5 holds each area division result left in step S45 as the original area division result (original result), and creates a duplicate of each area division result to search the area division result (search result). ) (Step S46).

The processing of steps S47 to S53 is a loop processing that is repeated until the end condition is satisfied. The loop processing of steps S47 to S52 performed therein is basically for a plurality of search results to be created, and the search results are sequentially set and executed as processing targets in the order of, for example, the identification number of the class. Further, the loop processing of steps S47 to S49 performed therein is basically for a plurality of original results.

The loop processing for the original result is repeated by the image processing unit 5 selecting the original results one by one in order from the one having the largest area of the label area of the attention class included in them (step S47). The image processing unit 5 calculates an index indicating the degree of overlap between the selected label area and the label area included in one search result set as the processing target (step S48). Here, IoU (Intersection over Union) is used as the index. IoU is given by IoU = I / U, where I is the area of the overlapping part (Intersection) of the two regions and U is the area of the sum region (Union) of the two regions. The closer they are, the lower the degree of overlap between the two regions.

If the image processing unit 5 has an original result that is larger than the threshold value T set in advance by the IoU and has not calculated the IoU from the search result to be processed, the image processing unit 5 performs steps S47 and S48 for the other original results. Repeat (in the case of "NO" in step S49). On the other hand, if the IoU is equal to or less than the threshold value T, or if the IoU has been calculated for all the original results (when “YES” in step S49), the process proceeds to step S50.

In step S50, the image processing unit 5 examines whether or not there is an original result whose IoU with the search result set as the processing target is equal to or less than the threshold value T, and if it exists (YES in step S50). ”), The label area of the attention class included in the original result is merged with the label area included in the search result (step S51), while if it does not exist (in step S50,“ step S50 ”. In the case of "NO"), step S51 is omitted. In step S51, the image processing unit 5 updates the label area of the search result with the merged result. In the portion where the label areas overlap, the label area of the search result is given priority and left.

The image processing unit 5 repeats the processing of steps S47 to S51 by changing the search result of the processing target (in the case of "NO" in step S52). When all the search results have been processed (when “YES” in step S52), the end determination is performed (step S53). If the predetermined end condition is not satisfied in the end determination (when "NO" in step S53), the processes of steps S47 to S52 are repeated. Here, the label area in the search result is expanded by the merge process in step S51, and by repeating the processes in steps S47 to S52, the area of the "other class" other than the label area in the search result is basically gradually gradually increased. Decreases to.

The end condition in the end determination in step S53 is about the degree of matching between the label area constituting the area division result obtained as the search classification result and the two-dimensional space corresponding to the input image, for example. If there is no change due to the repetition of the index indicating the degree or if it is less than a predetermined value, it is judged to be finished. Specifically, the end condition is that the label area of the original classification result that can be added disappears, that the area of "other class" in the search classification result does not decrease, and that the decrease is less than the predetermined value. Can be. Further, for example, if the index indicating the degree of matching exceeds or falls below a predetermined standard, it is determined to end. Specifically, the end condition is that the number of classes included in the label area of the search classification result exceeds the preset maximum number, and the area of the label area of the "other class" is a predetermined ratio of the area of the input image. It can be as follows. Alternatively, the termination condition may satisfy any one of two or more of the above conditions.

When the end condition is satisfied, the image processing unit 5 ends the search process (when “YES” in step S53), and sets the label area in the search result in which the “other class” area is the smallest. , The area information output means 31 outputs as the area division result of the input image (step S54).

<< Third Embodiment >>
The image processing system 1C according to the third embodiment of the present invention has the same configuration as that of FIG. 1 of the first embodiment, and is a region dividing device that performs region dividing processing on an input image as in the first embodiment. , And its learning device. Hereinafter, with respect to the third embodiment, the same components as those in the first embodiment are designated by the same reference numerals, and the description in the first embodiment is incorporated. In order to avoid confusion with the constituent elements of the above, a reference numeral having "C" added after the reference numeral of the first embodiment will be used, and the differences from the first embodiment will be mainly described.

Similar to the second embodiment, the classifier of the image processing system 1C does not include the supplementary class estimator, that is, it is the above-mentioned narrowly defined classifier, and does not have the supplementary class estimator 404, and corresponds to this. As for the configuration of the learning device, the learning supplementary class generating means 52 is not required as in the second embodiment.

In the first and second embodiments, the classification unit 403 could be configured to output the portion of the image that is not classified as the class of interest as an "other class" without specifying a specific class. However, the class classification unit 403 of the present embodiment outputs a label area for classes other than the class of interest.

The class of interest information in the present embodiment has a character as bias information given in order to give a bias to the classification process. Specifically, the class information of interest to be input to the classifier is defined by an N-dimensional class vector, and the value "1" is set for the class element that is likely to appear in the class classification result, and the class element that is difficult to appear in the class classification result. Set the value to "0".

In the learning operation of the classifier, the original correct answer label area is input to the learning means 53 instead of the replaced correct answer label area, and the learning attention class generating means 51 sets the correct answer class for the learning image as the learning attention class. It is input to the learning means 53 as information. Therefore, the correct label replacement means 50 is not required in the configuration of the learning device of the present embodiment.

The attention class setting means 54C in the image processing system 1C determines a supplementary class based on the output of the class classification unit 403, adds the supplementary class to the current attention class group, and sets a new attention class group. That is, the attention class setting means 54C is different from the first embodiment in that the supplementary class is determined without using the supplementary class estimation unit 404, but on the other hand, like the attention class setting means 54 of the first embodiment. By adding a supplementary class to the attention class group and setting a new attention class group, a plurality of attention class groups are sequentially set.

Specifically, the classes that are said to appear in the input image in the output of the class classification unit 403 and are not included in the current attention class group are set as supplementary class candidates, and the supplementary class is selected from among them. select. For example, the attention class setting means 54C selects the class having the largest label area area among the supplementary class candidates as the supplementary class, and updates the attention class group. In the attention class information after the update, the value "1" is set for the elements corresponding to the attention class group and the supplementary class before the update, and the other classes, that is, the supplementary class candidates that are not selected as the supplementary class. The value "0" is set for the element of the class that does not appear in the image.

The attention class setting means 54C sets, for example, a class vector in which a value “0” is set for all N classes as an initial value of attention class information, and sequentially adds supplementary classes based on the classification result. repeat. Then, the image processing unit 5 outputs the label area obtained by the output of the class classification unit 403 when the end condition is satisfied by the area information output means 31 as the area division result of the input image.

As the end condition, a condition regarding the degree of matching between the label area corresponding to the class of interest and the two-dimensional space corresponding to the input image can be set. For example, if there is no change due to repetition of the index indicating the degree, or if it is equal to or less than a predetermined value, it is determined to end. Specifically, the end condition can be that the attention class information is not updated, the area of the label area corresponding to the attention class group does not increase, or the increase is equal to or less than a predetermined value. .. Further, for example, if the index indicating the degree of matching exceeds or falls below a predetermined standard, it is determined to end. Specifically, the end condition is that the area of the label area corresponding to the attention class group is equal to or more than a predetermined ratio of the area of the input image, and the area of the label area not corresponding to the attention class group is the area of the input image. It can be said that the ratio is less than the predetermined ratio. Further, the end condition can be, for example, that the number of repetitions of the classification process exceeds a preset maximum number of times. Alternatively, the termination condition may satisfy any one of two or more of the above conditions.

[Modification example]
(1) In each of the above embodiments, an example in which the data group is a two-dimensional image is shown, but the present invention is not limited to this example. For example, the data group can be a time series of two-dimensional images. In that case, space is space-time and data is pixels. Further, for example, the data group can be a distance image, the space can be a two-dimensional space, and the data can be a pixel (distance value). In that case, the photographing unit 2 becomes a distance image sensor. Further, for example, the data group can be a three-dimensional measurement data such as a point cloud, the space can be a three-dimensional space, and the data can be a measurement point. In that case, a three-dimensional measuring instrument is used instead of the photographing unit 2.

(2) In the first embodiment and the modified example, the learning model of the classifier in the narrow sense and the learning model of the estimator share an example in which the feature amount extraction unit 400 is shared, but both learning models are common parts. It may be a separate model that does not have. In that case, the classifier and the estimator may be trained by common training data or may be trained by separate training data.

(3) In each of the above embodiments and each modification, the attention class information compression unit 401 is generated by simultaneous parallel learning with the feature amount extraction unit 400 and the class classification unit 403. Instead of this, the attention class information compression unit 401 may be separately generated by prior principal component analysis or the like based on the appearance tendency of the class of the training data.

(4) In each of the above-described embodiments and modifications, an example of dimensionally compressing the attention class information by the attention class information compression unit 401 of the classifier has been described. However, instead of using the attention class information compression unit 401, the input attention class information as it is may be combined with the image feature amount from the feature amount extraction unit 400 by the feature amount synthesis unit 402.

According to the area division device / method / program explained above, by inputting the attention class information (attention class group) together with the image (data group), the variety of learning data and the mixing criteria of the learning data can be mixed. Since the area is divided by a classifier (narrowly defined) learned so that the classification can be performed with high accuracy by suppressing the fluctuation caused by the fluctuation, the area can be divided with high accuracy by suppressing the fluctuation.

In particular, set multiple types of attention class groups for one data group, perform region division, and select the region division result in which the degree of matching of the label area with respect to the space in which the data group is distributed satisfies a predetermined condition. As a result, the accuracy of the attention class group is increased, so that highly accurate region division with suppressed fluctuation can be reliably executed.

Further, as illustrated in the first embodiment and its modified example, by adding a supplementary class to the class of interest and repeating the region division until the condition is satisfied, the reliable region division with suppressed fluctuation is ensured. Execution becomes possible. In other words, the smaller the subset of the attention class group, the greater the effect of suppressing fluctuations due to the mixture of diversity and grant criteria. Therefore, the region division should be performed by gradually changing the attention class group from a small subset to a larger subset. Therefore, it is possible to reliably execute highly accurate region division with suppressed fluctuations.

Further, as illustrated in the first embodiment and its modified example, the supplementary class is estimated by a learned estimator capable of estimating the supplementary class to be added in order to bring the attention class group closer to the correct answer. By repeating the region division while doing so, the accuracy of the class of interest can be further increased.

1 image processing system, 2 photographing unit, 3 communication unit, 4 storage unit, 5 image processing unit, 6 display unit, 40 learning data storage means, 41 learning model storage means, 42 model storage means, 50 correct label replacement means, 51 Learning attention class generating means, 52 Learning supplementary class generating means, 53 Learning means, 54 Focusing class setting means, 55 Region dividing means, 56 Classifier, 400 Feature extraction unit, 401 Feature class information compression unit, 402 Features Quantitative synthesis unit, 403 class classification unit, 404 supplementary class estimation unit.

Claims (5)

An area division device that performs classification processing to classify a data group distributed in a predetermined space into a plurality of classes and divides the space into label areas identified by the classes.
As a classifier in which the data group and the attention class group are input, the classification processing for the data group is performed, and the label area for the attention class group is output, the training data group and the learning data group are obtained in advance. A model storage means that stores a given correct answer class and a trained model trained using the learning attention class group given by a subset of the correct answer classes.
An attention class setting means for setting a plurality of attention class groups for the data group, and
The label area is obtained by the classifier for each of a plurality of settings of the attention class group by the attention class setting means, and among the area division results of the space based on the label area, the area division result is configured. An area division means for selecting a data group that satisfies a predetermined condition for the degree of consistency between the label area and the space as an area division result for the data group.
A region dividing device characterized by having. The attention class setting means sequentially sets a plurality of attention classes by adding a supplementary class to the attention class group and setting a new attention class group.
The area division means selects, as the area division result for the data group, the label area output by the classifier for the plurality of types of the attention class group whose size is equal to or larger than a predetermined reference. The area dividing device according to claim 1, wherein the area division device is characterized by the above. In the trained model, the training is performed by inputting the data group and the attention class group as an estimator for estimating the supplementary class and further using the correct answer of the supplementary class for the training data group. Being,
2. The area dividing device according to claim 2. A region division method in which a data group distributed in a space is classified into a plurality of classes and the space is divided into label areas identified by the classes.
As a classifier in which the data group and the attention class group are input, the classification processing for the data group is performed, and the label area for the attention class group is output, the training data group and the learning data group are obtained in advance. A step of preparing a trained model that has been trained using a given correct answer class and a group of attention classes for learning given by a subset of the correct answer classes.
A attention class setting step for setting a plurality of attention class groups for the data group, and
For each of the plurality of settings of the attention class group in the attention class setting step, the label area is obtained by the classifier, and among the area division results of the space based on the label area, the area division result is configured. An area division step of selecting a data group that satisfies a predetermined condition for the degree of matching between the label area and the space as an area division result for the data group.
A region division method characterized by having. A program in which a computer performs a classification process for classifying a data group distributed in a space into a plurality of classes and divides the space into label areas identified by the classes.
The computer
As a classifier in which the data group and the attention class group are input, the classification processing for the data group is performed, and the label area for the attention class group is output, the learning data group and the learning data group are obtained in advance. A model storage means that stores a given correct answer class and a trained model trained using the learning attention class group given by a subset of the correct answer classes.
An attention class setting means for setting a plurality of attention class groups for the data group, and an attention class setting means.
The label area is obtained by the classifier for each of a plurality of settings of the attention class group by the attention class setting means, and among the area division results of the space based on the label area, the area division result is configured. An area division means for selecting a data group that satisfies a predetermined condition for the degree of consistency between the label area and the space as an area division result for the data group.
An area division program characterized by functioning as.

Images