JP6875058B2 - Programs, devices and methods for estimating context using multiple recognition engines - Google Patents

Description

The present invention relates to a technique for estimating a context using a plurality of recognition engines.

In recent years, the use of deep learning has dramatically improved the recognition accuracy in object recognition and human behavior recognition.
For example, there is a technique for inputting a specific data set and comparing candidates for machine learning algorithms (see, for example, Patent Document 1). According to this technique, the evaluations of machine learning models can be automatically compared by aggregating the performance results for each machine learning model.

As a recognition engine for video data, for example, there is an object recognition technique for detecting an object reflected in an RGB image (see, for example, Non-Patent Document 1).
There is also a moving object recognition technique for detecting the movement of an object from a moving feature (optical flow) (see, for example, Non-Patent Document 2). For example, according to Two-stream ConvNets, using CNN (Spatial stream ConvNet) in the spatial direction and CNN (Temporal stream ConvNet) in the time series direction, the appearance features of objects and backgrounds in the image and the horizontal direction of optical flow. Both components and motion features in the series of vertical components are extracted. By integrating both of these features, behavior is recognized with high accuracy.
Further, there is also a technique for recognizing a person's behavior from a three-dimensional image (see, for example, Non-Patent Document 3).
Further, there is also a technique of recognizing a person's behavior by extracting skeleton information of a person's joint and its cooperation portion (see, for example, Non-Patent Document 4).

As another application field, there is also a technique of executing reinforcement learning of a hierarchical learning model according to the autonomous motion of the robot (see, for example, Non-Patent Document 5). There is also a technique for selecting an expert network, which is a machine learning network used by a gate network, according to input data (see, for example, Non-Patent Document 6).

FIG. 1 is a system configuration diagram having a recognition device.

According to the system of FIG. 1, the recognition device 1 functions as a server connected to the Internet. The recognition device 1 has a recognition engine in which a learning model is constructed in advance based on teacher data. When the recognition engine recognizes a person's behavior, the teacher data is a pre-association between the video data in which the person's behavior is reflected and the action target (context).

Each terminal 2 is equipped with a camera, and transmits video data of a person's behavior to the recognition device 1. The terminal 2 is a smartphone or mobile terminal owned by each user and connects to an access network such as a mobile phone network or a wireless LAN.
Of course, the terminal 2 is not limited to a smartphone or the like, and may be, for example, a Web camera installed in the house. Further, the video data captured by the Web camera may be recorded on the SD card, and the recorded video data may be input to the recognition device 1.

Specifically, for example, the user is asked to take a picture of his / her behavior with the camera of his / her smartphone. The smartphone transmits the video data to the recognition device 1. The recognition device 1 estimates a person's behavior from the video data, and uses the estimation result in various applications.
It should be noted that each function of the recognition device 1 may be incorporated in the terminal 2.

JP-A-2017-004509

Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun: Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, [online], [Search January 24, 2018], Internet <URL: https: / /arxiv.org/pdf/1506.01497.pdf ＞ Karen Simonyan and Andrew Zisserman, “Two-Stream Convolutional Networks for Action Recognition in Videos,” in NIPS 2014, [online], [Search January 24, 2018], Internet <URL: https://arxiv.org/ abs / 1406.2199.pdf ＞ Hernandez Ruiz, Alejandro & Porzi, Lorenzo & Rota Bul ?, Samuel & Moreno-Noguer, Francesc: 3D CNNs on Distance Matrices for Human Action Recognition, [online], [Search January 24, 2018], Internet <URL: https://www.researchgate.net/publication/320543521_3D_CNNs_on_Distance_Matrices_for_Human_Action_Recognition> Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh: Realtime Multi-Person 2D Pose Optimization using Part Affinity Fields., [Online], [Search January 28, 2018], Internet <https: // arxiv. org / pdf / 1611.08050.pdf ＞ Yasudake Takahashi, Minoru Asada, "Acquisition of actions by hierarchical construction of multiple learning mechanisms", [online], [Search on January 24, 2018], Internet <URL: http: //www.er.ams. eng.osaka-u.ac.jp/Paper/1999/Takahashi99c.pdf ＞ Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton and Jeff Dean: OUTRAGEOUSLY LARGE NEURAL NETWORKS: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER, [online], [January 24, 2018 Search ], Internet <URL: https://openreview.net/pdf?id=B1ckMDqlg> OpenPose, [online], [Search January 24, 2018], Internet <URL: https://github.com/CMU-Perceptual-Computing-Lab/openpose> "I tried OpenPose, which can detect bones from videos and photos", [online], [Searched on January 24, 2018], Internet <URL: http://hackist.jp/?p=8285> "OpenPose has been upgraded so that you can try 3d pose estimation", [online], [Search on January 24, 2018], Internet <URL: http://izm-11.hatenablog.com / entry / 2017/08/01/140945 ＞

According to the techniques of Non-Patent Documents 1 to 4, it is necessary to determine in advance the learning model that will have the highest recognition accuracy. Therefore, depending on the input data, a non-optimal learning model may be selected as a result.
According to the technique of Patent Document 1, in order to compare machine learning models, it is necessary to input input data to all machine learning models. The more machine learning models you have, the more server resources you need.
According to the technique of Non-Patent Document 5, the reinforcement learning of the hierarchical learning model is executed, and the context is not recognized by using a plurality of recognition engines.
According to the technique of Non-Patent Document 6, if the learning of the expert who selects the network is insufficient, the score may not reach the user's desired score.

All of the above-mentioned conventional techniques are based on the premise that the recognition engine in which the learning model is optimally constructed is used, and the recognition engine of the optimum learning model is determined in advance according to the input data. Need to keep.
On the other hand, the inventors of the present invention may be able to improve the recognition accuracy of the context (recognition result) by automatically selecting one or more optimum recognition engines according to the input data. I thought.

Therefore, an object of the present invention is to provide a program, an apparatus and a method capable of improving the recognition accuracy of a context by automatically selecting one or more optimum recognition engines according to input data. ..

According to the present invention, in a recognition program that uses a plurality of recognition engines to make a computer function to estimate a context from input data.
A selection engine that selects a recognition engine for the input data to be estimated using the learning model learned from the teacher data that associates the input data with the identifier of the recognition engine, and outputs the input data to the selected recognition engine. When,
A recognition engine whose recognition score calculated by the recognition engine is equal to or higher than the recognition threshold is determined for the input data, and the computer functions as a recognition score judgment means for feeding back the identifier of the recognition engine to the selection engine.
The selection engine is characterized in that the computer functions to relearn the learning model by the teacher data in which the input data is associated with the feedback identifier of the recognition engine.

According to other embodiments in the recognition program of the present invention.
The recognition engine is based on a classification that calculates a recognition score for each class.
It is also preferred that the recognition engine causes the computer to function as a recognition score to calculate any statistic of the highest, lowest, average or added values in a plurality of scores in a plurality of classes.

According to other embodiments in the recognition program of the present invention.
It is also preferable that the recognition score determining means causes the computer to function so as to feed back only the identifier of the recognition engine whose processing time required for the input data is equal to or less than the predetermined threshold time to the selection engine.

According to other embodiments in the recognition program of the present invention.
The selection engine is based on a classification that calculates a selection score for each class.
It is also preferable that the selection engine causes the computer to function to output the input data to the recognition engine in which the selection score for the input data to be estimated is equal to or higher than the first selection threshold.

According to other embodiments in the recognition program of the present invention.
It is also preferable that the selection engine causes the computer to further output the input data to the recognition engine whose selection score is less than the first selection threshold and greater than or equal to the second selection threshold.

According to other embodiments in the recognition program of the present invention.
When the difference between the selection score of one recognition engine that is equal to or higher than the first selection threshold and the selection score of the other recognition engine that is less than the first selection threshold is equal to or less than a predetermined difference, the selection engine is used. It is also preferable to make the computer function to output the input data to the other recognition engine.

According to other embodiments in the recognition program of the present invention.
The input data is video data,
Multiple recognition engines are different from each other
Object recognition engine based on RGB images,
Motion recognition engine based on optical flow and / or
It is also preferable to make the computer function as one of the human joint recognition engines based on skeleton information.

According to the present invention, in a recognition device that estimates a context from input data using a plurality of recognition engines.
A selection engine that selects a recognition engine for the input data to be estimated using the learning model learned from the teacher data that associates the input data with the identifier of the recognition engine, and outputs the input data to the selected recognition engine. When,
It has a recognition score determination means that determines a recognition engine whose recognition score calculated by the recognition engine is equal to or higher than the recognition threshold for the input data, and feeds back the identifier of the recognition engine to the selection engine.
The selection engine is characterized in that the learning model is retrained by the teacher data in which the input data and the feedback identifier of the recognition engine are associated with each other.

According to the present invention, in a method of recognizing a device that estimates a context from input data using a plurality of recognition engines.
The device is
Using the learning model learned by the teacher data that associates the input data with the identifier of the recognition engine, the recognition engine for the input data to be estimated is selected, and the input data is output to the selected recognition engine. Steps and
The second step of determining the recognition engine whose recognition score calculated by the recognition engine for the input data is equal to or higher than the recognition threshold, and
It is characterized in that the third step of re-learning the learning model is executed by the teacher data in which the input data is associated with the identifier of the recognition engine determined to be true by the second step.

According to the program, apparatus and method of the present invention, the recognition accuracy of the context can be improved by automatically selecting one or more optimum recognition engines according to the input data.

It is a system block diagram which has a recognition device. It is a functional block diagram of the recognition device in this invention. This is a specific first processing flow in the present invention. This is a specific second processing flow in the present invention. This is a specific third processing flow in the present invention. This is a specific fourth processing flow in the present invention. This is a specific fifth processing flow for video data. It is a flowchart based on FIG.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 2 is a functional configuration diagram of the recognition device according to the present invention.

The recognition device 1 estimates a context (for example, an object, a moving object, a human behavior, etc.) from input data by using a plurality of recognition engines.
According to FIG. 2, the recognition device 1 has a selection engine 11, a plurality of recognition engines 12 (first recognition engine 121, second recognition engine 122), and a recognition score determination unit 13. These functional components can be realized by executing a program that makes the computer mounted on the device function. Further, the processing flow of these functional components can be understood as a method of recognizing the device for the input data.

[Selection engine 11]
The selection engine 11 is a machine learning engine that is based on the classification and assigns a class (identifier of the recognition engine 12) to the input data to be estimated. The selection engine 11 is a learning model constructed in advance based on the teacher data in which the input data and the identifier of the recognition engine are associated with each other.
<Teacher data>
Input data <-> Recognition engine identifier The selection engine 11 specifically calculates a score (recognition accuracy) for each recognition engine (class). Generally, the one recognition engine with the highest score is selected as the estimation result. However, according to the present invention, the number of recognition engines is not limited to one, and may be plural. The embodiment in the selection method of the selection engine 11 will be described later with reference to FIGS. 3 to 6.
Then, the selection engine 11 selects the recognition engine 12 for the input data to be estimated by using the learning model, and outputs the input data to the selected recognition engine 12.

The selection engine 11 of the present invention does not need to build a complete learning model in advance, and relearns by feedback from the recognition score determination unit 13 described later. "Re-learning" means that the training model is further trained by using the input data and the feedback recognition engine identifier as teacher data.

[Recognition engine 12]
The recognition engine 12 selected by the selection engine 11 inputs input data from the selection engine 11. The recognition engine 12 is also based on the classification, and calculates the recognition score (recognition accuracy) for each class (estimable context). Generally, one context with the highest recognition score is output as the estimation result.

According to the present invention, there are a plurality of recognition engines 12 of different types. For example, different types of recognition engines are combined, such as an engine that mainly recognizes an object, an engine that mainly recognizes a rough action, and an engine that mainly recognizes a fine action. Each recognition engine builds a learning model in advance based on different teacher data depending on the type.

According to FIG. 2, it has two recognition engines (a first recognition engine 121 and a second recognition engine 122). The recognition score calculated by the recognition engine 12 may be a "statistical value" of any of the highest value, the lowest value, the average value, or the added value in the plurality of recognition scores in the plurality of contexts.
Then, each recognition engine 12 outputs the recognition score calculated for each context to the recognition score determination unit 13.

[Recognition score determination unit 13]
The recognition score determination unit 13 determines whether or not the recognition score calculated for each context of each recognition engine 12 with respect to the input data is equal to or greater than the “recognition threshold”.
Here, when it is determined to be true (recognition score ≥ recognition threshold value), the identifier of the recognition engine 12 is fed back to the selection engine 11. On the other hand, the selection engine 11 relearns the learning model as teacher data in which the input data and the identifier of the recognition engine are associated with each other.
Further, among the recognition scores calculated by each recognition engine 12, the context that is equal to or higher than the recognition threshold is output to the application as an estimation result.
The recognition threshold can be arbitrarily set by the operator.

As a result, the selection engine 11 relearns the learning model based on the feedback from the recognition score determination unit 13, and then selects the most suitable recognition engine 12 for the input data to be estimated. Will be.

FIG. 3 is a specific first processing flow in the present invention.

According to FIG. 3, it is assumed that the selection engine 11 calculates the selection score as follows for each recognition engine for the input data to be estimated.
[Recognition engine ID] [Selection score]
S1-> 0.7
S2-> 0.6
<Selection engine 11> * First selection threshold = 0.6
Here, the selection engine 11 outputs the input data to both the recognition engines 121 and 122 whose selection score is equal to or higher than the first selection threshold value (for example, 0.6).
The first selection threshold value can be arbitrarily set by the operator.

Next, the first recognition engine 121 and the second recognition engine 122 output the recognition score for each context with respect to the input data to the recognition score determination unit 13. Here, the "highest value" in a plurality of recognition scores in a plurality of contexts is used as a statistical value.
<First recognition engine 121> (Context): (Recognition score)
c11: 0.5
c12: 0.2
c13: 0.1
* Maximum value (statistical value) = 0.5
<Second recognition engine 122> (Context): (Recognition score)
c21: 0.7
c22: 0.3
c23: 0.3
* Maximum value (statistical value) = 0.7
<Recognition score judgment unit 13> * Recognition threshold = 0.6

The recognition score determination unit 13 determines whether or not the recognition score is equal to or higher than the recognition threshold value (0.6). Here, since only the second recognition engine 122 has a recognition score of 0.6 or more, the identifier (ID: 122) of the second recognition engine 122 is fed back to the selection engine 11.
As a result, the selection engine 11 relearns the learning model with the teacher data in which the input data is associated with the feedback identifier of the second recognition engine 122.
Although it has been described that the statistical value is the maximum value according to FIG. 3, it may be the minimum value, the average value, or the added value.

FIG. 4 is a specific second processing flow in the present invention.

According to FIG. 4, it is assumed that the selection engine 11 calculates the selection score for each recognition engine for the input data to be estimated as follows, as compared with FIG.
[Recognition engine ID] [Selection score]
S1-> 0.7
S2-> 0.6
<Selection engine 11> * First selection threshold = 0.7
Here, the selection engine 11 outputs the input data only to the first recognition engine 121 whose selection score is equal to or higher than the first selection threshold value (for example, 0.7). In this case, the input data is not output to the second recognition engine 122.

Next, the first recognition engine 121 outputs the recognition score for each context with respect to the input data to the recognition score determination unit 13. Here, too, the "highest value" in the multiple recognition scores of the context is used as the statistical value.
<First recognition engine 121> (Context): (Recognition score)
c11: 0.5
c12: 0.2
c13: 0.1
* Maximum value (statistical value) = 0.5

Then, the recognition score determination unit 13 feeds back the identifier (ID: 121) of the first recognition engine 121 whose recognition score is equal to or higher than the recognition threshold value (for example, 0.5) to the selection engine 11.
As a result, the selection engine 11 further relearns the learning model as teacher data in which the input data is associated with the identifier of the first recognition engine 121.

FIG. 5 is a specific third processing flow in the present invention.

According to FIG. 5, similarly to FIG. 3, the recognition score calculated by the recognition engine 12 is calculated by using the highest value in a plurality of scores in a plurality of contexts as a statistical value.
<First recognition engine 121> (Context): ( Recognition score)
c11: 0.6
c12: 0.2
c13: 0.1
* Maximum value (statistical value) = 0.6
* Processing time = 100ms
<Second recognition engine 122> (Context): ( Recognition score)
c21: 0.7
c22: 0.3
c23: 0.3
* Maximum value (statistical value) = 0.7
* Processing time = 500ms
<Recognition score judgment unit 13> * Recognition threshold = 0.6
* Predetermined threshold time = 200ms

The recognition score determination unit 13 determines whether or not the recognition score is equal to or higher than the recognition threshold value (0.6). Here, the recognition scores of both the recognition engines 121 and 122 are 0.6 or higher.
Further, the recognition score determination unit 13 determines whether or not the processing time required for the input data is equal to or less than the predetermined threshold time (200 ms). Here, the processing time of the second recognition engine 122 is 500 ms, which is false. In this case, the recognition score determination unit 13 feeds back only the identifier (ID: 121) of the first recognition engine 121 to the selection engine 11.
As a result, the selection engine 11 relearns the learning model with the teacher data in which the input data is associated with the identifier of the first recognition engine 121.
As described above, it is preferable from the viewpoint of processing resources to relearn the learning model of the selection engine 11 based not only on the recognition score but also on the "processing time" of the recognition engine.

FIG. 6 is a specific fourth processing flow in the present invention.

According to FIG. 6, learning omission is taken into consideration in the learning model of the selection engine 11. That is, even if the recognition engine 12 has a selection score less than the first selection threshold value, it is possible that the recognition engine 12 to be selected could not be selected because the learning model of the selection engine 11 was incompletely trained. There is sex. In that case, it is determined whether or not the recognition score of the recognition engine 12 is used for re-learning of the learning model of the selection engine 11.
According to FIG. 6, two embodiments are described.

<First Embodiment>
According to the selection engine 11 described above, the first recognition engine 121 having a selection score equal to or higher than the first selection threshold value (for example, 0.6) is selected.
On the other hand, the selection engine 11 also selects the second recognition engine 122 whose selection score is less than the first selection threshold (for example, 0.6) and equal to or more than the second selection threshold (for example, 0.5). ..
Then, the selection engine 11 outputs the input data to both the selected recognition engines 121 and 122.
Next, the recognition engines 121 and 122 output the recognition score for each context for the input data to the recognition score determination unit 13.
Then, the recognition score determination unit 13 feeds back the identifiers of the recognition engines 121 and 122 whose recognition score is equal to or higher than the recognition threshold value (for example, 0.6) to the selection engine 11.
As a result, the selection engine 11 relearns the learning model as teacher data in which the input data is associated with the identifiers of the recognition engines 121 and 122.

<Second embodiment>
According to the selection engine 11 described above, the first recognition engine 121 having a selection score equal to or higher than the first selection threshold value (for example, 0.6) is selected.
On the other hand, the selection engine 11 further has a selection score of the selected first recognition engine 121 (for example, 0.7) and a selection score of the second recognition engine 122 that is less than the first selection threshold (for example, 0.7). It is determined whether or not the difference from 0.5) is equal to or less than a predetermined difference (for example, 0.2). If determined to be true, the selection engine 11 also selects a second recognition engine 122 whose selection score is less than the first selection threshold (eg, 0.6).
Then, the selection engine 11 outputs the input data to both the selected recognition engines 121 and 122.
Next, the recognition engines 121 and 122 output the recognition score for each context for the input data to the recognition score determination unit 13.
Then, the recognition score determination unit 13 feeds back the identifiers of the recognition engines 121 and 122 whose recognition score is equal to or higher than the recognition threshold value (for example, 0.6) to the selection engine 11.
As a result, the selection engine 11 relearns the learning model as teacher data in which the input data is associated with the identifiers of the recognition engines 121 and 122.

According to FIGS. 3 to 6 described above, the recognition scores calculated by all the recognition engines 12 are determined by one recognition threshold value. On the other hand, as another embodiment, it may be determined by a different recognition threshold value for each recognition engine 12.

If the recognition scores calculated by all the recognition engines 12 do not reach the recognition threshold, they may be recognized separately or by a specific recognition engine, or the input data may include no recognition engine. As the associated teacher data, the learning model of the selection engine 11 may be relearned.

FIG. 7 is a specific fifth processing flow for video data.
FIG. 8 is a flowchart based on FIG. 7.

It is assumed that the recognition device 1 inputs video data in which a person's behavior is reflected as input data and estimates a behavior recognition result (context).
According to FIGS. 7 and 8, it has three recognition engines that are different from each other.
(1) Object recognition engine based on RGB images (2) Motion recognition engine based on optical flow (3) Joint recognition engine for people based on skeleton information Each of these recognition engines has action results in a large amount of video data in which a person is reflected. A learning model is generated in advance from the associated teacher data. In object recognition, moving object recognition, and joint recognition, even when the same video data is recognized, the context as an action result may be different.

(1) An object recognition engine based on RGB recognition specifically estimates an object (object) to be reflected in a captured image by using a neural network such as CNN (Convolutional Neural Network).
For example, when an object is reflected in the video data such as "cup", "smartphone", "television", and "building", the object is recognized with high accuracy.

(2) The moving object recognition engine based on the optical flow extracts the points where the same feature points are moving between frames, and expresses the movement of the object in the captured image by a "vector".
For example, when the movement of a person is reflected in the video data such as "grasping", "shaking", "punching", and "kicking", the moving object is recognized with high accuracy.

(3) A human joint recognition engine based on skeleton information specifically extracts feature points of human joints using a skeleton model such as OpenPose (registered trademark) (for example, Non-Patent Document 7). ~ 9). OpenPose is software that can detect multiple human body / hand / face key points in real time from images, and is published by GitHub. For example, 15 key points can be detected in the entire human body shown in the captured image.
For example, when the movement of a person is reflected in the video data based on the angle and position of the joint of the person, such as "drinking", "eating", "running", and "folding", the movement of the joint of the person is recognized with high accuracy. ..

Regarding human behavior recognition, in general, motion recognition and joint recognition have higher recognition accuracy than object recognition. Further, in the case of motion recognition of a person's body, joint recognition has higher recognition accuracy than motion recognition.

According to FIGS. 7 and 8, the processing is as follows.
(S10) The recognition device 1 inputs "video data".
(S11) According to FIG. 7, it is assumed that the selection engine 11 has selected all the recognition engines 12. In this case, the selection engine 11 outputs video data to each recognition engine 12.
(S12) Each recognition engine 12 outputs the following context and recognition score.
<RGB recognition engine 121> (Context): (Score)
Cup: 0.7
Smartphone: 0.4
TV: 0.1
* Maximum value (statistical value) = 0.7
<Optical flow recognition engine 122> (Context): (Score)
Gripping: 0.4
Shake: 0.2
Punch: 0.1
* Maximum value (statistical value) = 0.4
<Skeleton recognition engine 123> (Context): (Score)
Drink: 0.6
Eat: 0.2
Run: 0.0
* Maximum value (statistical value) = 0.6
(S13) The recognition score determination unit 13 determines whether or not the recognition score is equal to or higher than the recognition threshold value (0.6). Here, the recognition engines 121 and 123 have a recognition score of 0.6 or more.
Further, among the recognition scores calculated by the recognition engines 121 and 123, the contents "cup" and "drink" that are equal to or higher than the recognition threshold are output to the application as estimation results.
(S14) The identifiers (ID: 121, 123) of the recognition engines 121 and 123 are fed back to the selection engine 11.
As a result, the selection engine 11 relearns the learning model with the teacher data in which the input data is associated with the feedback recognition engines 121 and 123 identifiers.

As another embodiment, the recognition engine of the present invention is not limited to one based on video data, may be character recognition, or recognizes only a specific object (for example, a type of flower). May be good.

As described in detail above, according to the program, apparatus and method of the present invention, the recognition accuracy of the context is improved by automatically selecting one or more optimum recognition engines according to the input data. Can be done.
According to the present invention, the recognition engine for the input data does not need to be determined in advance because it is switched by the selection engine having the learning model.
In particular, according to the present invention, the learning model of the selection engine can be relearned not only in the learning stage but also in the operation stage.

With respect to the various embodiments of the present invention described above, various changes, modifications and omissions within the scope of the technical idea and viewpoint of the present invention can be easily made by those skilled in the art. The above explanation is just an example and does not attempt to restrict anything. The present invention is limited only to the scope of claims and their equivalents.

1 Recognition device 11 Selection engine 12 Recognition engine 121 First recognition engine 122 Second recognition engine 13 Recognition score judgment unit 2 Terminal

Claims (9)

In a recognition program that uses multiple recognition engines to make a computer function to infer context from input data.
A selection engine that selects a recognition engine for the input data to be estimated using the learning model learned from the teacher data that associates the input data with the identifier of the recognition engine, and outputs the input data to the selected recognition engine. When,
A recognition engine whose recognition score calculated by the recognition engine is equal to or higher than the recognition threshold for the input data is determined, and the computer functions as a recognition score determination means for feeding back the identifier of the recognition engine to the selection engine.
The selection engine is a recognition program characterized in that a computer functions to relearn the learning model by means of teacher data in which the input data and the feedback identifier of the recognition engine are associated with each other. The recognition engine is based on a classification that calculates a recognition score for each class.
The claim is characterized in that the recognition engine operates a computer to calculate a statistical value of any of a maximum value, a minimum value, an average value, or an addition value in a plurality of scores of a plurality of classes as the recognition score. Item 1. The recognition program according to item 1. The recognition score determining means is characterized in that the computer functions so that only the identifier of the recognition engine whose processing time required for the input data is equal to or less than a predetermined threshold time is fed back to the selection engine. The recognition program according to claim 1 or 2. The selection engine is based on a classification that calculates a selection score for each class.
The selection engine according to claims 1 to 3, wherein the selection engine causes a computer to output the input data to a recognition engine in which the selection score for the input data to be estimated is equal to or higher than the first selection threshold. The recognition program according to any one of the items. 4. The selection engine is characterized in that the computer is further operated to output the input data to the recognition engine whose selection score is less than the first selection threshold and equal to or more than the second selection threshold. The recognition program described in. When the difference between the selection score of one recognition engine that is equal to or greater than the first selection threshold and the selection score of the other recognition engine that is less than the first selection threshold is equal to or less than a predetermined difference. The recognition program according to claim 4, further comprising a computer functioning to output the input data to the other recognition engine. The input data is video data and
The plurality of recognition engines are different from each other and
Object recognition engine based on RGB images,
Motion recognition engine based on optical flow and / or
The recognition program according to any one of claims 1 to 6, wherein the computer functions so as to be one of the joint recognition engines of a person based on skeleton information. In a recognition device that estimates context from input data using multiple recognition engines
A selection engine that selects a recognition engine for the input data to be estimated using the learning model learned from the teacher data that associates the input data with the identifier of the recognition engine, and outputs the input data to the selected recognition engine. When,
It has a recognition score determination means that determines a recognition engine whose recognition score calculated by the recognition engine is equal to or higher than the recognition threshold for the input data, and feeds back the identifier of the recognition engine to the selection engine.
The selection engine is a recognition device characterized in that the learning model is relearned by the teacher data in which the input data and the feedback identifier of the recognition engine are associated with each other. In the recognition method of the device that estimates the context from the input data using multiple recognition engines,
The device is
Using the learning model learned by the teacher data that associates the input data with the identifier of the recognition engine, the recognition engine for the input data to be estimated is selected, and the input data is output to the selected recognition engine. Steps and
The second step of determining the recognition engine whose recognition score calculated by the recognition engine for the input data is equal to or higher than the recognition threshold, and
A device characterized in that a third step of re-learning the learning model is executed by teacher data in which the input data is associated with the identifier of the recognition engine determined to be true by the second step. Recognition method.

Images