JP7386006B2 - Region division device, region division method, region division program, learning device, learning method, and learning program - Google Patents

Description

The present invention relates to a technique for classifying a data group such as an image by class such as a subject and dividing the data group into label regions, and a technique for performing learning related to the classification.

For the purpose of automatically recognizing the scene captured in the image, the image is divided into regions for each of multiple objects captured in the image and regions for each of multiple parts, and the images captured in each region are divided into regions for each of multiple objects captured in the image. Technology for recognizing objects and parts has been researched and developed. Hereinafter, the object or part being photographed will be referred to as a subject. Region segmentation accompanied by object recognition is called semantic segmentation.

In particular, in recent years, techniques for performing the above segmentation and recognition based on learning have been actively researched. That is, for example, in the following non-patent document 1, a large number of learning images are prepared in which a class representing the object is assigned to each pixel of a region divided in advance for each object, and a computer is caused to perform machine learning on these learning images. It is stated that. Information given in advance is called an annotation. If an arbitrary image is input to the trained model generated by this learning, a class for each pixel is output for the input image. That is, the input image is divided into regions (label regions) labeled by class for each subject.

Furthermore, in recent years, datasets consisting of training images and annotations have been made public and available for use. Basically, trained models that have undergone a variety of training can perform more accurate region segmentation, so it is desirable that the size of the dataset used for learning be larger.

“Fully Convolutional Networks for Semantic Segmentation”, Jonathan Long, Evan Shelhamer, and Trevor Darrell (Proceedings of the IEEE conference on computer vision and pattern recognition, 2015)

However, there is a problem in that the region segmentation results tend to fluctuate due to the diversity of training data and the mixture of annotations with different assignment criteria. In addition, the mixture of annotations with different assignment criteria was a cause of decreased learning accuracy.

For example, if you use an image of a black carpet and an image of asphalt similar to it for learning, you will not only be able to correctly divide a black carpeted floor area into a floor area, but also divide part or all of it into a road. There may also be cases where the area is erroneously divided. This is an example where the region segmentation results tend to fluctuate due to the diversity of learning.

For example, if an image of a baseball field is input, the grass area in the image may be divided into grass areas, or the grass area in the image may be divided as part of the playground area. In some cases. This is an example in which the result of region division tends to fluctuate due to a mixture of annotations with different assignment criteria. For example, in a publicly available dataset, in one of the training images taken of a baseball field, the grass area is labeled with the word "grass" and the dirt region is labeled with the word "soil." However, in another training image taken of a baseball field, different labeling criteria are mixed, such as a label indicating "playground" being assigned to the combined area of grass and soil. There is. In other words, for the grass area, both grass and playground are correct answers. Therefore, fluctuations due to differences in input images are likely to occur.

On the other hand, the existence of multiple correct answers, such as in the grass area example, makes it difficult for learning to converge. Therefore, the mixture of annotations with different assignment criteria is a factor in reducing learning accuracy.

Note that the above problem may occur not only in two-dimensional images but also in spatio-temporal data formed from time-series images, three-dimensional data such as point clouds, and the like.

The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide a region division technique that can suppress fluctuations in the result of region division. Another object of the present invention is to provide a learning technique that can prevent a decrease in learning accuracy even if annotations with different attachment criteria are mixed in learning data used for learning region division processing.

(1) A region dividing device according to the present invention is a device that performs classification processing to classify a data group distributed in a space into a plurality of classes, and divides the space into labeled regions identified by the classes, As a classifier that receives a group and bias information for biasing the classification process and performs the classification process on the data group, it uses a learning data group and a correct answer given in advance for the learning data group. a classifier storage means that stores a trained model that has been trained using the learning bias information that is the bias information derived from the class and the correct class; and region dividing means for inputting the bias information to the classifier and determining the label region based on the output class classification result.

(2) In the region dividing apparatus according to (1) above, bias input means inputs the bias information having elements in one-to-one correspondence with each of the predefined classes; and the bias information inputted from the bias input means. The configuration may further include bias information compression means for dimensionally compressing the bias information and subjecting it to the classification process.

(3) In the region segmentation device according to (2) above, the classifier includes a feature synthesis unit that generates a composite feature that combines the dimensionally compressed bias information and the feature of the data group; A class classification unit that performs the classification process based on the composite feature amount may be configured.

(4) In the region segmentation device described in (1) to (3) above, the bias information input to the classifier specifies a class that is more likely to appear or a class that is less likely to appear in the classification result. be able to.

(5) In the region segmentation device according to (4) above, the bias information input to the classifier can further specify the degree of ease or difficulty of the class appearing in the class classification result. .

(6) The learning device according to the present invention is a device for learning a classifier that performs classification processing for classifying a data group distributed in space into a plurality of classes, wherein the classifier uses the data group and the classification processing. a learning model storage means for storing a learning model inputted with bias information for imparting bias to the data group and outputting a class classification result for the data group; learning data storage means storing a correct answer class and learning bias information that is the bias information derived from the correct answer class; and a learning data storage means storing the learning data group and the learning bias information in the learning model. and learning means for performing learning for updating the learning model based on an error of the input and output class classification results with respect to the correct answer.

(7) In the learning device according to (6) above, the learning bias information has elements that correspond one-to-one with each of the predefined classes for each of the learning data groups, and the learning generating the learning bias information that specifies the correct class given to the data group as a class that makes it more likely to appear in the class classification result, and specifies classes other than the correct answer class as classes that make it difficult to appear in the class classification result; The configuration may further include learning bias generation means.

(8) The region dividing method according to the present invention is a method of performing classification processing to classify a data group distributed in a space into a plurality of classes, and dividing the space into labeled regions identified by the classes, the method comprising: As a classifier that receives a group and bias information for imparting bias to the classification process and performs the classification process on the data group, a learning data group and a group of data given in advance for the learning data group are used. a step of preparing a trained model that has been trained using a correct class and learning bias information that is the bias information derived from the correct class; and a region dividing step of inputting bias information to the classifier and determining the label region based on the output class classification result.

(9) The learning method according to the present invention is a method for learning a classifier that performs classification processing for classifying a data group distributed in space into a plurality of classes, wherein the classifier includes the data group, the classification A step of preparing a learning model that is input with bias information for biasing the processing and outputs a class classification result for the data group, and a step of preparing a learning data group and a correct answer given in advance for the learning data group. and a step of preparing learning bias information that is the bias information derived from the correct class; and inputting the learning data group and the learning bias information to the learning model, and preparing the output class. and a learning step of performing learning to update the learning model based on an error of the classification result with respect to the correct answer.

(10) The area division program according to the present invention is a program that causes a computer to perform a classification process of classifying a data group distributed in a space into a plurality of classes and divide the space into label areas identified by the classes. The computer is used as a classifier that receives the data group and bias information for biasing the classification process and performs the classification process on the data group. Stores a trained model that has been trained using a correct class given in advance to a group of training data and learning bias information that is the bias information derived from the correct class. It functions as a classifier storage means, and an area dividing means that inputs the data group and the bias information for the data group into the classifier and obtains the label area based on the output class classification result.

(11) The learning program according to the present invention is a program that causes a computer to perform a process of learning a classifier that performs a classification process of classifying a data group distributed in space into a plurality of classes, learning model storage means for storing a learning model that receives the data group and bias information for biasing the classification process and outputs a class classification result for the data group; a learning data group; a learning data storage means that stores a correct answer class given in advance for the learning data group and learning bias information that is the bias information derived from the correct answer class; It functions as a learning means that inputs the learning data group and the learning bias information and performs learning to update the learning model based on the error of the output class classification result with respect to the correct answer.

According to the present invention, it is possible to suppress fluctuations in region division results. Further, according to the present invention, it is possible to prevent a decrease in learning accuracy even if annotations with different attachment criteria are mixed in learning data used for learning region division processing.

FIG. 1 is a block diagram showing a schematic configuration of an area dividing device according to an embodiment of the present invention. FIG. 1 is a schematic functional block diagram of a region dividing apparatus according to an embodiment of the present invention when performing segmentation. FIG. 2 is a schematic functional block diagram of a classifier used in the region segmentation device according to the embodiment of the present invention. FIG. 1 is a schematic functional block diagram of a region segmentation device according to an embodiment of the present invention as a classifier learning device. FIG. 3 is a schematic flow diagram regarding the operation of the region dividing apparatus according to the embodiment of the present invention in region dividing processing. FIG. 2 is a schematic diagram illustrating a process of generating a composite feature amount. FIG. 3 is a schematic diagram for explaining a processing example of region division processing by the region division device according to the embodiment of the present invention. FIG. 3 is a schematic flow diagram regarding the operation of the region dividing device in learning processing according to the embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A region dividing apparatus 1 that is an embodiment (hereinafter referred to as an embodiment) of the present invention will be described below based on the drawings. The region dividing device according to the present invention performs a classification process to classify a data group distributed in space into a plurality of classes, and divides the space into labeled regions identified by classes, and is shown as an example in this embodiment. The region dividing device 1 divides an image of a surveillance space into regions. That is, in this embodiment, the data group to be classified is a two-dimensional image, the data constituting it is a pixel, and the space to be divided is a two-dimensional space corresponding to the image.

The region dividing device 1 includes a classifier that performs the above classification process. Furthermore, the region segmentation device 1 includes a learning device that learns the classifier.

[Configuration of area dividing device 1]
FIG. 1 is a block diagram showing a general configuration of an area dividing device 1. As shown in FIG. The area dividing device 1 includes a photographing section 2, a communication section 3, a storage section 4, an image processing section 5, a display section 6, and an operation input section 7.

The photographing unit 2 is a camera that acquires images as a data group to be classified, and in this embodiment is a surveillance camera. The photographing section 2 is connected to the image processing section 5 via the communication section 3, generates images by photographing the monitoring space at predetermined time intervals, and sequentially inputs the generated images to the image processing section 5. For example, the imaging unit 2 is installed in a corner of a room that is a monitoring space with a predetermined fixed field of view overlooking the monitoring space, and photographs the monitoring space at a frame period of 1 second to generate a color image. Note that the photographing unit 2 may generate a monochrome image instead of a color image.

The communication section 3 is a communication circuit, one end of which is connected to the image processing section 5, and the other end connected to the photographing section 2, the display section 6, and the operation input section 7. The communication unit 3 acquires images from the photographing unit 2 and inputs them to the image processing unit 5 , and also acquires user instructions and the like from the operation input unit 7 and inputs them to the image processing unit 5 . Further, the communication unit 3 receives the results of classification into classes and the results of segmentation into label regions from the image processing unit 5 and outputs them to the display unit 6 .

The photographing section 2, the communication section 3, the storage section 4, the image processing section 5, the display section 6, and the operation input section 7 are connected as appropriate depending on the installation location of each section. For example, when the photographing section 2, the communication section 3, and the image processing section 5 are installed remotely, the photographing section 2 and the communication section 3 can be connected via an Internet line. Further, the communication section 3 and the image processing section 5 may be connected via a bus. In addition, a LAN (Local Area Network), various cables, etc. can be used as the connection means.

The storage unit 4 is a memory device such as a ROM (Read Only Memory) or a RAM (Random Access Memory), and stores various programs and various data. For example, the storage unit 4 stores learning data and information on a classifier that is a trained model, and inputs and outputs this information to and from the image processing unit 5. That is, information used for learning the classifier, information necessary for classification processing, information generated in the process of the processing, etc. are input and output between the storage section 4 and the image processing section 5.

The image processing unit 5 is composed of arithmetic devices such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an MCU (Micro Control Unit), and a GPU (Graphics Processing Unit). The image processing unit 5 operates as various processing means and control means by reading and executing programs from the storage unit 4, and reads various data from the storage unit 4 and stores generated data in the storage unit 4 as necessary. Make me remember. For example, the image processing unit 5 learns and generates a classifier, and stores the generated classifier in the storage unit 4 via the communication unit 3. The image processing unit 5 also performs segmentation of the captured image using a classifier.

The display unit 6 is a liquid crystal display, an organic EL (Electro-Luminescence) display, or the like, and displays the results of classification processing and segmentation input from the image processing unit 5 via the communication unit 3.

The operation input section 7 is an input device for the image processing section 5, and is composed of a keyboard, a mouse, and the like.

The region dividing device 1 is a device that classifies each pixel constituting an image using a classifier and divides the image into label regions, and also functions as a learning device that performs a learning operation to construct the classifier. have Hereinafter, regarding the configuration of the region dividing device 1, first, the configuration related to segmentation processing, that is, the configuration as a region dividing device will be explained, and then the configuration as a learning device will be explained.

[Configuration as area dividing device]
FIG. 2 is a schematic functional block diagram of the region dividing device 1 when performing segmentation, in which the storage section 4 functions as the classifier storage means 40 and the image processing section 5 functions as the region dividing means 50. Further, the communication section 3 cooperates with the image processing section 5 and functions as an image input means 30 and an area information output means 31. The operation input section 7 and the communication section 3 function as a bias input means 70.

The image input means 30 sequentially acquires images photographed by the photographing section 2 and inputs them to the area division means 50.

The bias input means 70 passes bias information (input bias information) inputted by the user by operating the operation input section 7 to the area division means 50. Bias information is information given to bias the classification process. In particular, the input bias information is bias information expressed in a format that is understandable to the user. The input bias information can be a vector (an N-dimensional vector, where N is the total number of classes) having elements in one-to-one correspondence with each of all predefined classes. For example, the user sets input bias information in which elements of a class that is desired to appear more easily in the classification results are set to a value of "1", and elements of a class that is desired to appear less likely to appear in the classification results are set to a value of "0". Note that the bias input means 70 may also be a means for reading and inputting a file in which bias information is recorded in a format understandable by the user.

The classifier storage means 40 stores classifiers generated through learning. In this embodiment, the classifier is configured with a multilayer network used in deep learning, and can be modeled using, for example, a convolutional neural network (CNN). The classifier storage means 40 stores information including filter coefficients and network structure of filters constituting a network such as CNN as a classifier.

The area dividing means 50 uses the classifier stored in the classifier storage means 40 to calculate the image (input image) input from the image input means 30 and the input bias from the bias input means 70 for the input image. Using information as input, a classification process is performed to estimate which of a plurality of predefined classes the pixel belongs to. Then, the region dividing means 50 obtains a label region based on the class classification result output from the classifier.

The area information output means 31 outputs the label area obtained by the area division means 50 to the display unit 6. For example, the area information output means 31 generates a color-coded image for each label area and outputs it to the display unit 6.

FIG. 3 is a schematic functional block diagram of the classifier. The classifier receives the image and bias information, classifies each pixel of the image, and outputs the result.

As already mentioned, bias information is information given to bias the classification process, for example, it expresses information about classes that will appear in the image and classes that will not appear. . By providing bias information as input to the classifier, it is possible to control the classes included in the segmentation result.

The network that constitutes the classifier of this embodiment includes a feature extraction section 400, a bias information compression section 401, a feature synthesis section 402, and a class classification section 403. Among these, the feature extraction unit 400, the feature synthesis unit 402, and the class classification unit 403 have a network structure consisting of a plurality of layers connected in series, and these parts receive an image as input and output a class classification result. Hereinafter, this part will be referred to as the network main part.

The feature extraction unit 400 and the class classification unit 403 are composed of a convolution layer, an activation function, a pooling layer, etc., and the main part of the network performs, for example, a process to obtain a feature map convoluted with the features of neighboring pixels. This process is repeated to consolidate relationships with surrounding pixels, and further processing is performed to identify the class of the pixels in the original image. In this embodiment, the network main section has a feature amount synthesis section 402 inserted in the middle thereof, and is divided into two parts, one before and after the feature amount synthesis section 402. These two parts are a feature extraction unit 400 and a class classification unit 403.The feature extraction unit 400 receives an image and calculates a feature from the image, while the class classification unit 403 is a feature synthesis unit. 402 performs a process of classifying pixels into classes and dividing the image into regions. However, the feature amount calculation performed by the feature amount extraction unit 400 may be performed up to an intermediate layer of the feature amount map generated in multiple layers, and the processing performed by the class classification unit 403 may be performed on features from the intermediate layer onward. The method may include generating a quantity map.

The bias information compression unit 401 is composed of a fully connected layer and the like, and obtains bias information in a low-dimensional representation and outputs it to the feature amount synthesis unit 402. In other words, bias information is set based on what appears in the image and the scene, but the number of classes appearing in the input image must be sufficiently smaller than the total number of classes that can be classified by the classifier. Bias information can be expressed as relatively low-dimensional information because of the co-occurrence, for example, indoor classes are unlikely to be included in outdoor images, and walls and floors are likely to be included at the same time in indoor images. The bias information compression unit 401 performs this dimension reduction conversion process. For example, the bias information compression unit 401 receives bias information expressed by a number of variables corresponding to all predefined classes, compresses the dimensions of this information, and converts it into bias information expressed by fewer variables. and output it.

The input bias information from the bias input means 70 is dimensionally compressed by the bias information compression unit 401, thereby setting the input bias information in a format that is understandable to the user and converting it into a format that can be efficiently used by a computer. can be used for region segmentation. Therefore, it becomes possible to easily and efficiently perform control to suppress fluctuations in the region division results.

The feature quantity synthesis unit 402 synthesizes the bias information compressed by the bias information compression unit 401 with the feature quantity extracted by the feature quantity extraction unit 400 to generate a composite feature quantity, and inputs the synthesized feature quantity to the class classification unit 403. do.

[Configuration as a learning device]
FIG. 4 is a schematic functional block diagram of the region segmentation device 1 as a learning device for learning a classifier, in which the storage section 4 functions as a learning data storage means 41 and a learning model storage means 42, and the image processing section 5 functions as the learning bias generation means 52 and the learning means 53.

The learning data storage means 41 stores a large number of images as a learning data group, a correct class given in advance for the image, and learning bias information that is bias information derived from the correct class. . The learning images and the correct classes corresponding to the images are stored in advance in the learning data storage means 41 prior to the learning process. On the other hand, the learning bias information is generated by the learning bias generation means 52 and stored in the learning data storage means 41.

The learning bias generation means 52 generates learning bias information from the correct class corresponding to each learning image stored in the learning data storage means 41, and stores it in the learning data storage means 41. . The learning bias information has the same format as the input bias information. That is, the learning bias generation means 52 generates learning bias information having elements that correspond one-to-one with each of all predefined classes for each learning image (learning data group). generates learning bias information that specifies the correct class given to the class as a class that is likely to appear in the classification results, and that specifies classes other than the correct answer as classes that are difficult to appear in the classification results.

By providing this training bias information for learning, a classifier can be trained to easily and efficiently control the input bias information to set it in a format that is understandable to the user and suppress fluctuations in the region segmentation results. becomes possible.

The learning means 53 inputs the learning image, the correct class, and the learning bias information, and performs learning to update the learning model based on the error of the output class classification result with respect to the correct answer.

The learning model storage means 42 stores the learning model for the above-mentioned classifier. Along with the learning process by the learning means 53, the learning model stored in the learning model storage means 42 is updated. When the learning is completed, the learning model storage means 42 stores the learned model of the classifier and functions as the classifier storage means 40.

[Operation of area dividing device 1]
Next, the operation of the region dividing device 1 will be explained separately into region dividing processing and learning processing.

[Operation in area division processing]
FIG. 5 is a schematic flow diagram regarding the operation of the region dividing apparatus 1 in region dividing processing.

When the area dividing device 1 starts the area dividing process, the photographing unit 2 sequentially outputs images taken of the monitoring space at predetermined intervals. The image processing section 5 cooperates with the communication section 3 and repeats the operation shown in the flowchart of FIG. 5 every time it receives an image from the photographing section 2.

The communication unit 3 functions as an image input unit 30, and upon receiving an image, inputs the image to the image processing unit 5 (step S100).

The image processing unit 5 sets bias information (input bias information) for operating the segmentation result on the input image (input image). For example, users can decide which classes they want included or excluded from the segmentation results, and use them as bias information. In this case, the image processing section 5 displays the input image on the display section 6, and the user inputs input bias information for the input image from the operation input section 7. The operation input unit 7 functions as a bias input unit 70, and inputs input bias information to the area division unit 50 of the image processing unit 5 (step S101). Furthermore, if the scene in the captured image, such as outdoors or indoors, is known and the classes included in the image can be limited, input bias information can be determined based on that. In this case, the input bias information is input and set in advance to the region dividing means 50, for example, at the start of the region dividing process.

When the region dividing means 50 receives the input image and input bias information, it divides the image into regions using the classifier read from the classifier storage means 40. The input bias information in step S100 is compressed by the bias information compression unit 401 of the classifier (step S102), while the input image in step S100 is input to the feature extraction unit 400 of the classifier; Feature amounts are calculated from the input image (step S103).

The feature synthesis unit 402 of the classifier synthesizes the input bias information output from the bias information compression unit 401 with the feature output from the feature extraction unit 400 to generate a composite feature (step S104). .

FIG. 6 is a schematic diagram illustrating the process of generating a composite feature amount. FIG. 6 schematically represents the data in the classifier shown in FIG. 403, the image 100 input to the classifier, the synthesized feature amount 110 generated by the feature amount synthesis unit 402, and the class classification result 140 output from the classifier are arranged. Further, on the right side of the figure, the input node 120 of the bias information compression section 401, the bias information 121 input to the node, and the output node 130 of the bias information compression section 401 are shown.

Regarding the data of the network main part arranged on the left side of FIG. 6, the x axis is in the width direction of the image 100, the y axis is in the height direction, and the c axis is the dimension corresponding to the channel of the feature amount. The size of the image 100 is W _I pixels in the x direction and H _I pixels in the y direction. The feature map generated by the feature extraction unit 400 has a size of _WF pixels in the x direction and _HF pixels in the y direction, and a size in the c direction, that is, the number of channels is C channels. Incidentally, the size of the feature map in the x and y directions generally does not match the size of the image 100, and usually W _F <W _I and H _F <H _I.

The bias information 121 illustrated in FIG. 6 is information as to whether or not each of N predetermined classes is likely to be included in an image. For example, all classes to be classified by the classifier are set as the N classes.

Specifically, the bias information 121 is for indoor use, and is an N-dimensional vector in which classes that will appear indoors are represented by a value of "1" and classes that will not appear indoors are represented by a value of "0". be. The bias information 121 shows a specific example, and assumes that classes of objects that may exist indoors, such as "person" and "floor", are included in the image, and the corresponding element in the vector is set to "1". ” is set, and on the other hand, objects that do not exist indoors, for example, the class “road” are not included in the image, and the corresponding element is set to “0”.

A bias that specifies classes that are more likely to appear in the classification results by changing the value of the element corresponding to the class that will appear indoors to "1" based on bias information where the value of all elements is "0" It can be said to be information 121. In addition, if you change the value of the element corresponding to the class that will not appear indoors to "0" based on the bias information where the value of all elements is "1", you can specify the class that will be less likely to appear in the classification results. This can be said to be the bias information 121.

The input nodes 120 of the bias information compression unit 401 have a one-to-one correspondence with the elements of the bias information 121, and the number thereof is N, while the number D of the output nodes 130 is less than N. The bias information compression unit 401 performs dimension compression on the bias information 121 input to the input node 120 and outputs compressed bias information from the output node 130. That is, the bias information 121 is compressed from an N-dimensional vector to a D-dimensional vector. Incidentally, FIG. 6 shows a configuration in which the input node 120 and the output node 130 are fully coupled as the bias information compression unit 401.

The feature quantity synthesis unit 402 receives compressed bias information from the output node 130 of the bias information compression unit 401, synthesizes the bias information with the feature quantity map input from the feature quantity extraction unit 400, and generates a synthesized feature quantity. 110 is generated. The synthesized feature quantity 110 is obtained by concatenating bias information expressed by a D-dimensional vector to each C-dimensional feature vector specified by a pair of x and y coordinates in the feature quantity map before synthesis. It has the same width and height as the previous feature map, and has a structure in which the number of channels is (C+D) channels. For example, the first to Cth channels of the composite feature quantity 110 are feature quantity maps before composition, and the (C+1) to (C+D)th channels are the first to D nodes of the output node 130 of the bias information compression unit 401. The output value of is set.

In this embodiment, since the bias information for each (x, y) coordinate is common, the structure of the composite feature amount 110 is such that each of the D elements of bias information is duplicated in the x and y directions, and the feature amount extraction unit 400 It has a structure in which it is enlarged to the same size of W _F ×H _F pixels as the output of , and is layered on the feature map before synthesis. In other words, for example, while the feature amounts of the first to Cth channels may differ depending on the coordinates (x, y), in this embodiment, the feature amounts of the first to Cth channels may differ depending on the coordinates (x, y), whereas in this embodiment, the feature amounts of the first to Cth channels include all coordinates. A common value is set for (x, y).

The class classification unit 403 classifies each pixel of the input image 100 based on the composite feature amount 110, and outputs the class classification result 140 (step S105). That is, the class classification result 140 consists of the classification result for each pixel of the input image 100. For example, each pixel is associated with N values corresponding to the number of classes to be classified. In this case, as shown in FIG. 6, the class classification result 140 is data of W _I pixels in the x direction, H _I pixels in the y direction, and N channels in the c direction. The channels of the class classification result 140 have a one-to-one correspondence with the N classes. For example, each channel of each pixel is given a larger value as the probability that the pixel belongs to the class corresponding to the channel is higher. It will be done. The area dividing means 50 can classify the pixel at the coordinates (x, y) of the input image 100 into a class corresponding to the channel that outputs the maximum value at the coordinates (x, y) of the class classification result 140, for example. can. By classifying each pixel of the input image 100, the input image 100 is divided into regions and label regions are defined, and the region division means 50 outputs the obtained label region information to the region information output means 31 (step S106 ).

After outputting the label area information for the input image in step S100, the area dividing device 1 returns to step S100 and repeats the above-described processes of steps S100 to S106 for the next input image.

FIG. 7 is a schematic diagram for explaining a processing example of the region division process of the region division device 1. An image 200 in FIG. 7A shows an input image, and the input image 200 includes a wall 201, a window 202, a person 203, and a floor 204 covered with a black carpet.

Label areas obtained for the input image 200 are images 210 and 220 in FIGS. 7(b) and 7(c). An image 210 in FIG. 7(b) represents a label area obtained by the conventional technique, and an image 220 in FIG. 7(c) represents a label area obtained by the area dividing apparatus 1 of this embodiment.

In the processing results of the prior art shown in FIG. 7B, the areas where the wall 201, window 202, and person 203 are photographed are correctly labeled as the label area 211 of the wall class, the label area 212 of the window class, and the label area of the person class, respectively. Although it is divided into a label area 213, the area where the floor 204 was photographed is divided into a label area 214, which is correctly divided into a floor class, and a label area 215, which is incorrectly divided into a road class. .

On the other hand, FIG. 7C shows a processing result obtained by inputting the indoor bias information 121 illustrated in FIG. 6 together with the input image 200 to the region dividing apparatus 1 of this embodiment. In the example of the input bias information 121, the classes "person" and "floor" have the value "1", but the class "road" is set to the value "0", and by using this input bias information 121, , the road classes are suppressed in the classification process. As a result, in FIG. 7(c), the areas where the wall 201, window 202, and person 203 are photographed are correctly labeled as the label area 211 of the wall class, the label area 212 of the window class, and the label area 213 of the person class, respectively. By dividing and further suppressing the road class, the area where the floor 204 is photographed is also correctly divided into floor classes.

In other words, in the example shown in Fig. 7, input bias information is set to make it difficult for road classes that are assumed not to appear in the room (the space to be measured) to appear in the input image of the room (a group of data distributed in the space). ) is divided into regions to suppress misclassification of roads into classes. Therefore, while using a classifier that has undergone various learnings including road classes, it is possible to suppress fluctuations in misclassifying floors as roads.

In this way, according to the region segmentation device of the present invention, it is possible to perform highly accurate region segmentation while suppressing fluctuations while using classifiers that have undergone various types of learning. Note that the ability to use classifiers that have undergone various types of learning means that there is an advantage that there is no need to prepare a classifier specialized for each space to be measured. Incidentally, a class for which the value of input bias information is set to 0 does not appear in the classification result at all, but is suppressed to the last, so if the class has a high possibility of being the class, it may appear in the classification result. In this respect, there is an advantage in being able to use classifiers that have undergone various types of learning.

Furthermore, an example of a classifier that performs learning using the above-mentioned learning data in which annotations of the playground and grass classes are mixed for grass will be described. For example, in the case of an input image that is input from the imaging unit 2 installed in a baseball field, since the entire field of view is the baseball field, it is desirable to classify grass into the grass class. Since we want to obtain information on facilities such as baseball fields in input images that include baseball fields, it is desirable to classify baseball fields that include grass into the game field class. In this case, by setting the input bias information for the former input image to make the grass class more likely to appear and to make the game room class less likely to appear, the input bias information for the latter input image is set to make the grass class less likely to appear. In addition, by setting the classes of the game hall so that they appear easily, desired area division results can be obtained for each class.

As described above, according to the region segmentation device of the present invention, it is possible to perform highly accurate region segmentation that suppresses fluctuations while using a classifier that has been trained using training data containing a mixture of different assignment criteria. Become. Note that it is possible to use a classifier that has been trained using training data with a mixture of different assignment criteria, which means that it is possible to use a classifier that uses training data that has been re-created for each space to be measured using assignment criteria that are appropriate for that space. This means that you have the advantage of not having to prepare anything.

[Operation in learning process]
Prior to the operation of dividing an input image into regions, the region dividing device 1 performs an operation of learning a classifier. The learning of this classifier will be explained below. The learning of the classifier in the region segmentation device 1 uses a learning image, a correct class that is correct data for the corresponding region segmentation, and bias information (learning bias information) created from the correct class. Based on the error between the classification result of the classifier's learning model and the correct data, the parameters of the learning model are repeatedly updated using known optimization methods such as error backpropagation until the error converges. do. Through this learning, a classifier that can control biasing of classification processing can be trained. Further, the learning of the classifier includes, in addition to the learning of the feature extraction unit 400 and the class classification unit 403, the operation of learning the bias information compression unit 401 using the learning bias information.

FIG. 8 is a schematic flow diagram regarding the operation of the region dividing device 1 in the learning process.

In the learning process, a learning image, a correct class, and learning bias information are used as learning data. Therefore, when the start of the learning operation is instructed, the image processing section 5 functions as the learning bias generation means 52 and generates learning bias information for each learning image stored in the learning data storage means 41. Specifically, the learning bias generation means 52 generates learning bias information from the correct class stored in the learning data storage means 41 in association with the learning image, and applies this to the learning image. The data are stored in the learning data storage means 41 in association with each other (step S200).

The learning bias information has a format that matches the input bias information 121 described above, and in this embodiment is an N-dimensional vector consisting of elements corresponding to N classes. The vector is expressed as {B _i } (1≦i≦N), and when the correct class gives a set L of classes included in the corresponding learning image, as an example, the vector of learning bias information The value of each element B _i can be set depending on whether the class corresponding to the element is included in the set L or not. That is, in this example, if all the classes to be classified by the classifier are N classes, and the i-th (1≦i≦N) class is represented by C _i , the learning bias generation means ₅₂ The element B _i of the vector of learning bias information corresponding to is set by the following equation.

When the learning data is prepared by generating the learning bias information in step S200, the image processing unit 5 functions as the learning means 53 and reads out the learning model of the classifier from the learning model storage means 42 (step S201). Note that the parameters of the learning model at this stage are initial values.

Next, the learning means 53 reads out learning data consisting of a learning image, a correct class, and a set of learning bias information from the learning data storage means 41 (step S202), and performs processing for updating the learning model ( Steps S203 to S207) are performed. Note that the learning data read out in step S202 is not the entire set of learning data stored in the learning data storage means 41 but a part of the set, and the learning means 53 sequentially reads out the learning data part by part and creates a learning model. Repeat the process of updating. In this embodiment, multiple sets of learning data are read out in step S202. For example, a set of learning data corresponding to 10 learning images is read out.

The learning means 53 sequentially sets the read learning data one set at a time as a processing target (step S203), inputs the learning image to be processed and its learning bias information into the learning model, and sets the learning data to be processed one by one (step S203). Each pixel of the image is classified (step S204). In step S204, the learning bias information is compressed using the parameters of the bias information compression unit 401 at that time, and the feature amount of the learning image is compressed using the parameters of the feature amount extraction unit 400 at that time. Calculated. Other than that, the process in step S204 is basically the same as steps S102 to S106 in FIG. A composite feature quantity is created from the feature quantity extracted by the extraction unit 400, and a class classification unit 403 classifies the class to which each pixel belongs. Then, by arranging the obtained classes of each pixel in the coordinate system of the learning image, it is possible to obtain a result in which the learning image is divided into regions.

The processes in steps S203 and S204 are repeated for all the learning data read out in step S202 (if "NO" in step S205).

After completing the processing for all learning data (in the case of "YES" in step S205), the learning means 53 compares the label region obtained as a result of region division with the label region based on the correct class. , calculates the error of the classification result (step S206), and updates the learning model based on the error (step S207). For example, in step S207, the learning means 53 updates the parameters of the feature amount extraction section 400, class classification section 403, and bias information compression section 401 using an error backpropagation method or the like.

The learning means 53 repeats the processes of steps S202 to S208 if the predetermined repetition end condition is not satisfied ("NO" in step S208). For example, the iteration termination condition is that either the error determined in step S206 converges or the number of iterations reaches a predetermined upper limit number of times.

If the iteration end condition is satisfied ("YES" in step S208), the learning means 53 stores the learning model updated in step S207 as a trained model in the learning model storage means 42 (step S209). ). Specifically, each parameter updated in step S207 is saved. This completes the learning process, and as described above, the learning model storage means 42 becomes the classifier storage means 40, and the learned model is used as a classifier in the area division process of the area division apparatus 1.

The learning means 53 of this embodiment generates learning bias information for each learning image and performs learning. The meaning of this will be explained using the above-mentioned example of learning data in which annotations of the playground and grass classes are mixed for grass. In the conventional technology that does not have the concept of learning bias information, the learning image cannot be used for both the learning image in which the class of playground is assigned to the grass and the learning image in which the class of grass is assigned to the lawn. Because it was allowed to classify the grass pixels in both the playground class and the grass class, there was a training image where the error for the correct class did not become small, and learning did not converge. There were cases where learning accuracy decreased. On the other hand, the learning means 53 of the present embodiment sets the grass class to be difficult to appear and the game field class to be easy to appear, corresponding to the learning image in which the grass is assigned the playground class. By generating and using the learning bias information, the grass pixels in the learning image are restricted from being classified into the grass class, and are guided to be classified into the correct class, which is the game hall class. At the same time, by generating and using learning bias information that sets the class of the playground to be less likely to appear and the class of grass to be more likely to appear, corresponding to the learning image in which the grass class is assigned to the grass. To restrict the classification of grass pixels in a learning image into the playground class and guide them to classify them into the grass class, which is the correct class.

Therefore, according to the learning device of the present invention, the learning means 53 generates learning bias information for each learning image and performs learning, so that classification of each learning image into a class other than the correct class can be restricted. Therefore, learning can be easily converged. Therefore, the learning accuracy of the classifier can be improved even when using learning data that includes a mixture of different assignment criteria.

[Modified example]
(1) In the above embodiment, two values, 1 and 0, are used as the bias information 121 that specifies a class that is likely to appear in the classification result or a class that is difficult to appear in the class classification result. Although an example is shown in which the two states of "no" are alternatively set, the bias information may be expressed using three or more values.

For example, the bias information can specify the degree to which a class appears more easily or less easily in the classification results. The degree can be expressed using a continuous value of 0 to 1, for example. Further, the value set for each class of bias information as the degree can be determined using, for example, the proportion of the area of the class in the image. Furthermore, in the process of segmenting time-series images, bias information can be created with reference to the processing results at the previous time. Furthermore, for example, the assumed prior probability of the class may be used as the value set for each class in the bias information.

(2) In the above embodiment and modification, the bias information specifies a common condition for the entire image. On the other hand, the classifier may be configured to provide different bias information to each of a plurality of regions set in an image, and specify different conditions for each region using the plurality of bias information. This makes it possible to perform region segmentation, for example, by applying a bias that makes it easier to see the sky class in the upper part of the image, and adding a bias that makes it easier to see the ground class in the lower part of the image.

(3) In the above embodiment, an example was described in which the bias information compression unit 401 is used in the classifier to compress the dimensionality of input bias information. However, without using the bias information compression section, the bias information of the above embodiment and each modification is combined with the image feature from the feature extraction section 400 in the feature amount synthesis section 402 in the input state. It's okay.

(4) In the learning process of the classifier in the above embodiment, in addition to the learning of the feature quantity extraction section 400 and the class classification section 403, the learning of the bias information compression section 401 is performed simultaneously and in parallel. In contrast, it is possible to prepare a bias information compression means (bias information compression means) in advance using principal component analysis or the like based on the appearance tendency of classes in the learning data, and use this as the bias information compression unit 401. . In this case, the bias information compression section 401 does not need to learn when the feature extraction section 400 and the class classification section 403 learn.

Further, when the bias information compression means is prepared in advance, the bias information compression means may be connected between the bias input means 70 and the area division means 50 without providing the bias information compression unit 401 in the classifier. You can also do it. In this case, the region division means 50 inputs the bias information from the bias information compression means to the classifier, and the bias information is combined with the feature amount from the feature amount extraction section 400 in the feature amount synthesis section 402 of the classifier.

(5) In the embodiment and modification described above, the feature quantity synthesis unit 402 performs synthesis by linking the bias information to the feature quantity from the feature quantity extraction unit 400. In another embodiment, the feature quantity synthesis unit 402 can perform synthesis by calculating the product of the feature quantity from the feature quantity extraction unit 400 and the bias information as a composite feature quantity. In that case, the bias information compression section 401 or bias information compression means compresses the bias information from the bias input means 70 into C dimensions, which is equal to the number of channels C of the feature amount from the feature amount extraction section 400.

(6) In the above embodiment and each modification, the classifier has a multilayer network structure, but the structure is not limited thereto. For example, the feature extracting unit 400 may extract a HOG (Histogram of Oriented Gradients) feature, a color histogram, or the like from an image, or may be a combination of these.

(7) In the above embodiment and each modified example, an example was shown in which the data group is a two-dimensional image, but the invention is not limited to this example. For example, the data group can be a time series of two-dimensional images. In that case, space is space-time and data is pixels. Further, for example, the data group may be a distance image, the space may be a two-dimensional space, and the data may be pixels (distance values). Note that in that case, the imaging unit 2 becomes a distance image sensor. Further, for example, three-dimensional measurement data such as a point cloud, the space can be a three-dimensional space, and the data can be a measurement point. Note that in that case, a three-dimensional measuring instrument is used instead of the imaging section 2.

Reference Signs List 1 area dividing device, 2 photographing unit, 3 communication unit, 4 storage unit, 5 image processing unit, 6 display unit, 7 operation input unit, 30 image input means, 31 area information output means, 40 classifier storage means, 41 learning data storage means, 42 learning model storage means, 50 region division means, 52 learning bias generation means, 53 learning means, 70 bias input means, 100 image, 110 synthetic feature quantity, 120 input node, 121 bias information, 130 output Node, 140 Class classification result, 400 Feature extraction unit, 401 Bias information compression unit, 402 Feature synthesis unit, 403 Class classification unit.

Claims (11)

An area dividing device that performs classification processing to classify a data group distributed in a space into a plurality of classes, and divides the space into labeled areas identified by the classes,
As a classifier that receives the data group and bias information for biasing the classification process and performs the classification process on the data group, a learning data group and information provided in advance for the learning data group are used. a classifier storage means that stores a trained model that has been trained using a correct class and learning bias information that is the bias information derived from the correct class;
region dividing means for inputting the data group and the bias information for the data group into the classifier and calculating the label region based on the output class classification result;
An area dividing device comprising: The area dividing device according to claim 1,
Bias input means for inputting the bias information having elements in one-to-one correspondence with each of the predefined classes;
bias information compression means for dimensionally compressing the bias information from the bias input means and subjecting it to the classification process;
An area dividing device further comprising: The area dividing device according to claim 2,
The classifier is
a feature amount synthesis unit that generates a composite feature amount that combines the dimensionally compressed bias information and the feature amount of the data group;
a class classification unit that performs the classification process based on the composite feature amount;
An area dividing device comprising: The area dividing device according to any one of claims 1 to 3,
The area segmentation device is characterized in that the bias information input to the classifier specifies a class that is more likely to appear or a class that is less likely to appear in the class classification result. The area dividing device according to claim 4,
The area segmentation device is characterized in that the bias information input to the classifier further specifies a degree of ease or difficulty of appearance of the class in the class classification result. A learning device that trains a classifier that performs a classification process for classifying each of a plurality of data included in a data group obtained by measuring the space into a plurality of classes for each space to be measured , comprising :
The classifier stores a learning model that receives the data group and bias information for biasing the classification process and outputs a class classification result for each of the plurality of data included in the data group. a learning model storage means for
The bias information is derived from a correct class given in advance to each of a learning data group and a plurality of learning data included in the learning data group, and a correct answer class of each of the plurality of learning data. learning data storage means storing learning bias information;
learning means for inputting the learning data group and the learning bias information into the learning model, and performing learning to update the learning model based on an error of the output class classification result with respect to the correct answer;
A learning device characterized by having. The learning device according to claim 6,
For each of the training data groups, the learning bias information has elements that correspond one-to-one with each of the predefined classes, and the correct class given to the learning data group is assigned to the class. The method further comprises learning bias generation means for generating the learning bias information that is designated as a class that is likely to appear in the classification results and that is designated as a class that makes it difficult for classes other than the correct answer to appear in the classification results. A learning device. An area dividing method that performs a classification process to classify a data group distributed in a space into a plurality of classes, and divides the space into labeled areas identified by the classes, the method comprising:
As a classifier that receives the data group and bias information for biasing the classification process and performs the classification process on the data group, a learning data group and information provided in advance for the learning data group are used. preparing a trained model that has been trained using the correct class and learning bias information that is the bias information derived from the correct class;
a region dividing step of inputting the data group and the bias information for the data group to the classifier and calculating the label region based on the output class classification result;
A region dividing method characterized by having the following. A learning method for training a classifier that performs classification processing for classifying each of a plurality of data included in a data group obtained by measuring the space into a plurality of classes for each space to be measured, the method comprising:
As the classifier, a learning model is prepared that receives the data group and bias information for biasing the classification process and outputs a class classification result for each of the plurality of data included in the data group. the step of
The bias information is derived from a correct class given in advance to each of a learning data group and a plurality of learning data included in the learning data group, and a correct answer class of each of the plurality of learning data. a step of preparing learning bias information;
a learning step of inputting the learning data group and the learning bias information to the learning model and performing learning to update the learning model based on the error of the output class classification result with respect to the correct answer;
A learning method characterized by having the following. A program that causes a computer to perform a classification process to classify a data group distributed in a space into a plurality of classes, and to divide the space into label areas identified by the classes, the program comprising:
the computer,
As a classifier that receives the data group and bias information for biasing the classification process and performs the classification process on the data group, a learning data group and information provided in advance for the learning data group are used. a classifier storage means that stores a trained model that has been trained using a correct class and learning bias information that is the bias information derived from the correct class;
region dividing means for inputting the data group and the bias information for the data group into the classifier and calculating the label region based on the output class classification result;
An area division program characterized by functioning as a. A program that causes a computer to perform processing to train a classifier that performs classification processing to classify each of multiple data included in a data group obtained by measuring the space into multiple classes for each space to be measured. hand,
the computer,
The classifier stores a learning model that receives the data group and bias information for biasing the classification process and outputs a class classification result for each of the plurality of data included in the data group. learning model storage means for
The bias information is derived from a correct class given in advance to each of a learning data group and a plurality of learning data included in the learning data group, and a correct answer class of each of the plurality of learning data. learning data storage means storing learning bias information; and
learning means that inputs the learning data group and the learning bias information into the learning model, and performs learning to update the learning model based on the error of the output class classification result with respect to the correct answer;
A learning program that is characterized by functioning as a.

Images