WO2021019748A1

WO2021019748A1 - Classification method, classification program, classification device, learning method, learning program, and learning device

Info

Publication number: WO2021019748A1
Application number: PCT/JP2019/030085
Authority: WO
Inventors: 泰斗横田; 彼方鈴木
Original assignee: 富士通株式会社
Priority date: 2019-07-31
Filing date: 2019-07-31
Publication date: 2021-02-04

Abstract

A classification device (10) obtains a feature vector from each of a plurality of intermediate layers of a neural network model, which has an input layer, the plural of intermediate layers, and an output layer, when data of a captured image of one or more objects has been input into the input layer. When the obtained feature vectors have been input into respectively corresponding classifiers, the classification device (10) updates each classifier on the basis of the classification results. The classification device (10) determines the weights of each classifier on the basis of the result of a comparison between the output result of each classifier and answer data. The classification device (10) classifies an object represented by the image data into one of one or more prescribed classes on the basis of the output result and weights of each classifier.

Description

Classification method, classification program, classification device, learning method, learning program and learning device

The present invention relates to a classification method, a classification program, a classification device, a learning method, a learning program, and a learning device.

DNN (Deep Neural Network) used in deep learning has a structure in which a large number of intermediate layers are overlapped between inputs and outputs. Characteristic information (intermediate feature vector) of the input data appears in the intermediate layer.

In the case where only a small amount of data (teacher data) with a correct answer label for training the learner can be prepared, it is known that learning can be performed with a small amount of teacher data by using the intermediate feature vector of another DNN as input data. Has been done. For example, in the field of picking robots, research is performed to classify objects with a small amount of teacher data by using the intermediate feature vector of DNN that detects objects in camera images as input data of other learners. Is underway.

International Publication No. 2017/187516 Japanese Unexamined Patent Publication No. 2008-250990

However, the above technology has a problem that the accuracy of the classifier that classifies objects into classes may decrease. For example, in a DNN having a plurality of intermediate layers, the likelihood of appearance of features suitable for a classifier may differ for each intermediate layer. Therefore, in the learning of the classifier, if the influence of the intermediate feature vector in which the feature suitable for the classifier does not appear becomes large, the accuracy of the classifier may decrease.

One aspect is to improve the accuracy of the classifier.

In one embodiment, the computer inputs the data of an image of one or more objects into the input layer of a neural network model having an input layer, a plurality of intermediate layers, and an output layer. The process of acquiring the feature vector from each of the above is executed. The computer updates each classifier based on the classification result when the acquired feature vector is input to the classifier corresponding to each of the feature vectors. The computer determines the weight of each classifier based on the comparison result between the output result of each classifier and the correct answer data. The computer executes a process of classifying the object reflected in the image data into one of a predetermined class based on the output result of the classifier and the weight.

On one side, the accuracy of the classifier can be improved.

FIG. 1 is a diagram for explaining the configuration of a classification system. FIG. 2 is a block diagram showing a configuration example of the classification device. FIG. 3 is a diagram showing an example of the shape of an object. FIG. 4 is a diagram for explaining the class. FIG. 5 is a diagram for explaining the configuration of the model. FIG. 6 is a diagram showing an example of the output of the classifier. FIG. 7 is a diagram showing an example of the weighted output of the classifier. FIG. 8 is a flowchart showing the flow of the learning process. FIG. 9 is a flowchart showing the flow of the classification process. FIG. 10 is a diagram for explaining the difference between the intermediate layers. FIG. 11 is a diagram illustrating a hardware configuration example.

Hereinafter, examples of the classification method, the classification program, the classification device, the learning method, the learning program, and the learning device according to the present invention will be described in detail based on the drawings. The present invention is not limited to this embodiment. In addition, each embodiment can be appropriately combined within a consistent range.

[Functional configuration]
The configuration of the classification system 1 including the classification device according to the embodiment will be described with reference to FIG. FIG. 1 is a diagram for explaining the configuration of a classification system. As shown in FIG. 1, the classification system 1 includes a picking system 20 and a classification device 10.

The picking system 20 includes a picking robot 21, a camera 22, and a tray 23. The camera 22 shoots the tray 23 from above and transmits the data of the shot image to the classification device 10. The picking robot 21 is controlled based on the output of the image data of the classification device 10 and the like, and grips the object on the tray 23.

The classification device 10 detects the position of an object in the image using a model, and further classifies the class of the detected object. The model includes a detector that detects the position of the object in the image and a classifier that classifies the class of the object. The classification device 10 also functions as a learning device for learning the model.

The functional configuration of the classification device 10 will be described with reference to FIG. FIG. 2 is a block diagram showing a configuration example of the classification device. As shown in FIG. 2, the classification device 10 includes an input / output unit 11, a storage unit 12, and a control unit 13.

The input / output unit 11 is an interface for inputting / outputting data to / from the picking system 20 and communicating data with other devices. For example, the input / output unit 11 may input / output data to / from an input device such as a keyboard or mouse, an output device such as a display or speaker, or an external storage device such as a USB memory. Further, for example, the input / output unit 11 is a NIC (Network Interface Card), and data may be communicated via the Internet.

The storage unit 12 is an example of a storage device that stores data, a program executed by the control unit 13, and the like, such as a hard disk and a memory. The storage unit 12 stores the learning data information 121, the weight information 122, the detector information 123, and the classifier information 124.

The learning data information 121 is image data taken by the camera 22 of the picking system 20. The learning data information 121 may be the image data itself, or may be the image data to which a predetermined process is applied. Further, it is assumed that the position of the object to be classified in the image included in the learning data information 121 and the class to which the object belongs are known. That is, the learning data information 121 is learning data with a correct answer label for learning the detector and the classifier.

The weight information 122 is a weight determined by the determination unit 132. The method of determining the weight by the determination unit 132 will be described later. The weight information 122 is stored when training the model, and is referred to and used when classifying unknown data.

The detector information 123 is a parameter for constructing a detector. For example, the detector may be an SSD (Single Shot Multibox Detector). SSD is a model for detecting the position of an object using DNN (Deep Neural Network). In this case, the detector information 123 is the DNN weight, bias, and the like. Further, the detector information 123 is updated by learning.

The classifier information 124 is a parameter for constructing a classifier. For example, the classifier is SVM (Support Vector Machine). In this case, the classifier information 124 is a vector of SVM. Further, the classifier information 124 is updated by learning. Also, in the examples, there are as many classifiers as there are intermediate layers of detectors. Therefore, the classifier information 124 is a parameter of each of the plurality of classifiers.

In the control unit 13, for example, a program stored in an internal storage device is executed with RAM as a work area by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), or the like. Is realized by. Further, the control unit 13 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array), for example. The control unit 13 has an acquisition unit 131, a determination unit 132, a classification unit 133, and an update unit 134.

Here, in the embodiment, it is assumed that the detector used in the classification device 10 is a neural network model having an input layer, an intermediate layer, and an output layer, and is an SSD that detects the position of an object in an image.

Further, here, it is assumed that the picking system 20 targets objects having four shapes of a cylinder, a star, a T-shape, and an L-shape as shown in FIG. FIG. 3 is a diagram showing an example of the shape of an object. Further, as shown in FIG. 4, the classification device 10 classifies the objects into classes corresponding to each shape. FIG. 4 is a diagram for explaining the class.

Further, it is assumed that, for example, 2,000 image data (for example, 500 images for each shape) are prepared as learning data for the detector. Further, as learning data for the classifier, it is assumed that, for example, 80 image data (for example, 20 images for each shape) are prepared. Specifically, each image is, for example, a color image of 300 × 300 pix in which one object is arranged at random positions. The correct label is represented by the coordinates (x, y) indicating the position of the object, the grip width (w), and the inclination (θ).

[Learning of detector]
First, the update unit 134 learns the detector by backpropagating the error of the DNN using the learning data for the detector, and updates the detector information 123. At this time, the update unit 134 uses the data of the above-mentioned 2,000 images and performs learning with the number of epochs set to 200, for example.

[Learning of classifier]
The acquisition unit 131 is a feature vector from each of the plurality of intermediate layers when the data of the image in which one or more objects are captured is input to the input layer of the neural network model having the input layer, the plurality of intermediate layers, and the output layer. To get.

As shown in FIG. 5, the detector has a plurality of intermediate layers. FIG. 5 is a diagram for explaining the configuration of the model. For example, the detector is a convolutional neural network such as SSD. If the detector is an SSD, there are, for example, 23 intermediate layers. In addition, there is a classifier corresponding to each intermediate layer. The acquisition unit 131 acquires the dimension of the feature vector of each intermediate layer after compressing it by PCA (principal component analysis) or the like.

The determination unit 132 inputs the feature vector acquired by the acquisition unit 131 into each corresponding classifier and obtains an output result. Then, the determination unit 132 determines the weight of each classifier based on the comparison result between the output result of each classifier and the correct answer data.

Further, before the weight is determined by the determination unit 132, the classification unit 133 classifies the object reflected in the image data into one of a predetermined class based on the output result and the weight of the classifier. Then, the update unit 134 updates each classifier based on the classification result when the input is input to the classifier corresponding to each of the feature vectors. Further, when each classifier is an SVM, the number of trainings may be one, and the training of each classifier may be performed in parallel.

Here, the determination of the weight by the determination unit 132 will be described. FIG. 6 is a diagram showing an example of the output of the classifier. As shown in FIG. 6, each classifier calculates a score for each class and outputs the class with the highest score as the classification result class. In addition, the score calculated by each classifier for the correct class is used as the accuracy.

In the example of FIG. 6, it is assumed that the learning data belongs to class B. Here, as shown in FIG. 6, the No. 1 classifier calculates the class A score as, for example, "0.70", the class B score as, for example, "0.25", and the class C score, for example. It is calculated as "0.05". At this time, the classification result by the No. 1 classifier is class A and does not match the correct answer. The accuracy of the No. 1 classifier is "0.25".

On the other hand, the No. 2 classifier calculates the class A score as, for example, "0.10", the class B score as, for example, "0.80", and the class C score as, for example, "0.10". .. At this time, the classification result by the No. 2 classifier is class B, which matches the correct answer. The accuracy of the No. 2 classifier is "0.80".

In the example of FIG. 6, the classification result by 3 out of 5 classifiers is incorrect. Therefore, if the classification result of the randomly selected intermediate layer is adopted as the final classification result, there is a 60% probability that the answer will be incorrect.

Therefore, the determination unit 132 determines the weight of each classifier so that the higher the accuracy of the classifier, the larger the weight. For example, the determination unit 132 determines for each classifier with the square of accuracy as a weight. Further, the determination unit 132 stores the determined weight as the weight information 122 in the storage unit 12.

In this case, the classification unit 133 first calculates the weighted output for each class for each classifier as in Eq. (1). Then, the classification unit 133 calculates the combined output of each class as in Eq. (2).
Weighted output = score calculated by the classifier x square of accuracy ... (1)
Combined output = total weighted output of each class / total weighted output of all classes ... (2)

FIG. 7 is a diagram showing an example of the weighted output of the classifier. The classification unit 133 can calculate the combined output as shown in FIG. 7 based on the weight determined by the determination unit 132. In this case, since the combined output of the class B is the largest, the classification unit 133 finally classifies the training data into the class B.

[Processing flow]
The flow of the learning process of the Example will be described with reference to FIG. FIG. 8 is a flowchart showing the flow of the learning process. As shown in FIG. 8, first, the classification device 10 learns the detector using the learning data for the detector (step S101). Here, it is assumed that the detector is a DNN having an input layer, an intermediate layer, and an output layer.

Next, the classification device 10 inputs the training data for the classifier into the trained detector, and acquires feature vectors from each of the plurality of intermediate layers (step S102). Here, the classification device 10 inputs the acquired feature vector into the classifier corresponding to each intermediate layer, and learns each classifier (step S103). Then, the classification device 10 determines the weight of each classifier based on the classification result of the classifier (step S104).

The flow of the classification process of the examples will be described with reference to FIG. FIG. 9 is a flowchart showing the flow of the classification process. As shown in FIG. 9, first, the classification device 10 inputs the image data to the trained detector and acquires the feature vector from each of the plurality of intermediate layers (step S201).

Next, the classification device 10 inputs the acquired feature vector into the corresponding classifier (step S202). Then, the classification device 10 outputs the classification result based on the combined output calculated by weighting the classification result of the classifier (step S203).

[effect]
As described above, the classification device 10 is a plurality of intermediate layers when data of an image showing one or more objects is input to the input layer of the neural network model having an input layer, a plurality of intermediate layers, and an output layer. Get the feature vector from each. The classification device 10 updates each classifier based on the classification result when the acquired feature vector is input to the classifier corresponding to each of the feature vectors. The classification device 10 determines the weight of each classifier based on the comparison result between the output result of each classifier and the correct answer data. The classification device 10 classifies the objects reflected in the image data into one of a predetermined class based on the output result and the weight of the classifier.

Depending on the nature of the data to be classified, the feature vector acquired from which intermediate layer may be suitable for classification. For example, as shown in FIG. 10, in a DNN that performs image recognition, features of a fine region (narrow region) tend to appear in the intermediate layer close to the input layer, and an abstract (wide region) is likely to appear in the intermediate layer close to the output layer. ) Features tend to appear.

On the other hand, the classification device 10 can add weights according to accuracy to each classifier corresponding to a plurality of intermediate layers. Therefore, the classifier 10 can improve the accuracy of the classifier.

The classification device 10 compresses and acquires the dimension of the feature vector. As a result, the classification device 10 can reduce the amount of calculation.

The classifier 10 determines the weight of each classifier so that the higher the accuracy of the classifier, the larger the weight. As a result, the classification device 10 can increase the influence of the classifier having a particularly high accuracy and improve the final accuracy while utilizing the information output from each classifier.

The classification device 10 acquires a feature vector from the intermediate layer of the convolutional neural network. As a result, the classification device 10 can improve the accuracy of the classifier for image recognition in general.

In the above embodiment, the classification device 10 has been described as performing both the learning process and the classification process. On the other hand, the classification device 10 may receive information on the model trained by the learning device and perform only the classification process. Further, in the above embodiment, the classifier is assumed to be an SVM. On the other hand, the classifier is not limited to SVN, and may be DNN, for example.

In addition, the method of the example can be applied not only to the classification problem but also to the regression problem. In this case, the classification device 10 calculates a weighted average weighted by the accuracy itself for the score of each class of each classifier.

[system]
Information including processing procedures, control procedures, specific names, and various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. In addition, the specific examples, distributions, numerical values, etc. described in the examples are merely examples and can be arbitrarily changed.

Further, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific forms of distribution and integration of each device are not limited to those shown in the figure. That is, all or a part thereof can be functionally or physically distributed / integrated in an arbitrary unit according to various loads, usage conditions, and the like. Further, each processing function performed by each device may be realized by a CPU and a program analyzed and executed by the CPU, or may be realized as hardware by wired logic.

[hardware]
FIG. 11 is a diagram illustrating a hardware configuration example. As shown in FIG. 11, the classification device 10 includes a communication interface 10a, an HDD (Hard Disk Drive) 10b, a memory 10c, and a processor 10d. Further, the parts shown in FIG. 11 are connected to each other by a bus or the like.

The communication interface 10a is a network interface card or the like, and communicates with other servers. The HDD 10b stores a program or DB that operates the functions shown in FIG.

The processor 10d reads a program that executes the same processing as each processing unit shown in FIG. 2 from the HDD 10b or the like and expands the program into the memory 10c to operate the process that executes each function described in FIG. 2 or the like. It is a wear circuit. That is, this process executes the same function as each processing unit of the classification device 10. Specifically, the processor 10d reads a program having the same functions as the acquisition unit 131, the determination unit 132, the classification unit 133, and the update unit 134 from the HDD 10b or the like. Then, the processor 10d executes a process of executing the same processing as the acquisition unit 131, the determination unit 132, the classification unit 133, the update unit 134, and the like.

In this way, the classification device 10 operates as an information processing device that executes a learning method by reading and executing a program. Further, the classification device 10 can realize the same function as that of the above-described embodiment by reading the program from the recording medium by the medium reading device and executing the read program. The program referred to in the other examples is not limited to being executed by the classification device 10. For example, the present invention can be similarly applied when another computer or server executes a program, or when they execute a program in cooperation with each other.

This program can be distributed via networks such as the Internet. In addition, this program is recorded on a computer-readable recording medium such as a hard disk, flexible disk (FD), CD-ROM, MO (Magneto-Optical disk), DVD (Digital Versatile Disc), and is recorded by the computer from the recording medium. It can be executed by being read.

1 Classification system 10 Classification device 11 Input / output unit 12 Storage unit 13 Control unit 20 Picking system 21 Picking robot 22 Camera 23 Tray 121 Learning data information 122 Weight information 123 Detector information 124 Classifier information 131 Acquisition unit 132 Decision unit 133 Classification unit 134 Update

Claims

A feature vector is acquired from each of the plurality of intermediate layers when data of an image showing one or more objects is input to the input layer of a neural network model having an input layer, a plurality of intermediate layers, and an output layer. ,
Each classifier is updated based on the classification result when the acquired feature vector is input to the classifier corresponding to each of the feature vectors.
Determine the weight of each classifier based on the comparison result between the output result of each classifier and the correct answer data.
A classification method characterized in that a computer executes a process of classifying an object shown in image data into one of a predetermined class based on the output result of the classifier and the weight.
The classification method according to claim 1, wherein the acquisition process is acquired by compressing the dimension of the feature vector.
The classification method according to claim 1 or 2, wherein the determination process determines the weight of each classifier so that the higher the accuracy of the classifier, the greater the weight.
The classification method according to claim 1, wherein the acquisition process is to acquire the feature vector from an intermediate layer of a convolutional neural network.
A feature vector is acquired from each of the plurality of intermediate layers when data of an image showing one or more objects is input to the input layer of a neural network model having an input layer, a plurality of intermediate layers, and an output layer. ,
Each classifier is updated based on the classification result when the acquired feature vector is input to the classifier corresponding to each of the feature vectors.
Determine the weight of each classifier based on the comparison result between the output result of each classifier and the correct answer data.
A classification program characterized in that a computer executes a process of classifying an object reflected in image data into one of a predetermined class based on the output result of the classifier and the weight.
A feature vector is acquired from each of the plurality of intermediate layers when data of an image showing one or more objects is input to the input layer of a neural network model having an input layer, a plurality of intermediate layers, and an output layer. Acquisition department and
An update unit that updates each classifier based on the classification result when the feature vector acquired by the acquisition unit is input to the classifier corresponding to each of the feature vectors.
A decision unit that determines the weight of each classifier based on the comparison result between the output result of each classifier and the correct answer data,
Based on the output result of the classifier and the weight, the classification unit that classifies the objects in the image data into one of the predetermined classes,
A classification device characterized by having.
A feature vector is acquired from each of the plurality of intermediate layers when data of an image showing one or more objects is input to the input layer of a neural network model having an input layer, a plurality of intermediate layers, and an output layer. ,
Each classifier is updated based on the classification result when the acquired feature vector is input to the classifier corresponding to each of the feature vectors.
A learning method characterized in that a computer executes a process of determining the weight of each classifier based on the comparison result between the output result of each classifier and the correct answer data.
A feature vector is acquired from each of the plurality of intermediate layers when data of an image showing one or more objects is input to the input layer of a neural network model having an input layer, a plurality of intermediate layers, and an output layer. ,
Each classifier is updated based on the classification result when the acquired feature vector is input to the classifier corresponding to each of the feature vectors.
A learning program characterized in that a computer executes a process of determining the weight of each classifier based on the comparison result between the output result of each classifier and the correct answer data.
A feature vector is acquired from each of the plurality of intermediate layers when data of an image showing one or more objects is input to the input layer of a neural network model having an input layer, a plurality of intermediate layers, and an output layer. Acquisition department and
An update unit that updates each classifier based on the classification result when the feature vector acquired by the acquisition unit is input to the classifier corresponding to each of the feature vectors.
A decision unit that determines the weight of each classifier based on the comparison result between the output result of each classifier and the correct answer data,
A learning device characterized by having.