WO2020049636A1

WO2020049636A1 - Identification system, model presentation method, and model presentation program

Info

Publication number: WO2020049636A1
Application number: PCT/JP2018/032761
Authority: WO
Inventors: 芙美代鷹野; 竹中　崇; 誠也柴田; 浩明井上; 高橋　勝彦; 哲夫井下
Original assignee: 日本電気株式会社
Priority date: 2018-09-04
Filing date: 2018-09-04
Publication date: 2020-03-12
Also published as: JP6981553B2; JPWO2020049636A1

Abstract

When a model learned by a first identification system is received from the first identification system, a model update means 705 updates a model which was learned by the first identification system and stored in a second model storage means 704 to the model received from the first identification system. A learning means 701 relearns the model using teaching data, which include a label for data determined on the basis of an identification result derived by a second identification means 706 and said data, and updates the model stored in a first model storage means 702 to the relearned model. A model transmission means 707 transmits the model learned by the learning means 701 to one or more of second identification system determined in advance.

Description

Identification system, model providing method, and model providing program

The present invention relates to an identification system for identifying an object represented by data by applying the data to the model, and a model providing method and a model providing program for the identification system to provide a model to another identification system.

例 An example of a general identification system will be described below. A general identification system learns a model in advance by machine learning using, as teacher data, a set of an image obtained by photographing with a camera included in the identification system and a label representing an object appearing in the image. . Then, the general identification system identifies an object shown in the image by applying an image newly obtained by photographing by the camera to the model.

Such a general identification system is used for the purpose of detecting a suspicious vehicle or a suspicious person, preventing crime, etc., detecting a user using a white stick or a wheelchair, and using a white stick or a wheelchair. It is used for the purpose of support such as guiding people.

Here, the identification system for identifying an object appearing in an image has been described as an example, but an identification system for identifying an object represented by voice data is also conceivable as a general identification system. Hereinafter, an identification system for identifying an object appearing in an image will be described as an example.

In addition, Patent Document 1 describes an image recognition method that avoids prolonged additional learning due to a difference in an imaging environment. The image recognition method described in Patent Document 1 is an image recognition method in a camera system including a plurality of camera devices. In the image recognition method described in Patent Literature 1, a first image and first imaging environment information are acquired from a first camera device. Then, using a parameter table that manages imaging environment information indicating each imaging environment when each camera device has previously captured an image and each recognition control parameter indicating each detector function corresponding to each imaging environment. And a first recognition control parameter indicating a first detector function corresponding to the same or similar imaging environment as the first imaging environment indicated in the first imaging environment information. Then, the first image obtained from the first camera device is recognized using the first detector function indicated by the first recognition control parameter.

特許 Further, Patent Document 2 discloses an image monitoring device. The image monitoring device described in Patent Literature 2 normalizes a face area using the detected facial feature points and performs matching with a person dictionary.

JP 2016-15116 A JP 2007-300185 A

複数 It is conceivable that a plurality of the above-mentioned general identification systems are provided, and cameras of each identification system are installed in various places.

Here, there is a case where a bias occurs in how an object is captured in an image obtained by one camera by shooting. For example, it is assumed that one camera has many opportunities to photograph a car traveling from right to left as viewed from the camera, but has few opportunities to photograph a car traveling in the opposite direction. In this case, many images of a car traveling in the right-to-left direction are obtained, but only a few images of a car traveling in the opposite direction are obtained. Then, the teacher data includes many images of vehicles traveling in the right-to-left direction, and includes only a few images of vehicles traveling in the opposite direction. As a result, when an image of a car traveling from right to left is applied to a model obtained by machine learning using teacher data, the identification system identifies the car with high accuracy. When an image of an automobile traveling in the direction is applied to the model, the accuracy of identifying the automobile decreases.

において In each of the identification systems located in various places, it is preferable that the identification accuracy of the model for identifying the object represented by the data is improved.

Therefore, the present invention provides an identification system that can re-learn its own model so as to improve the identification accuracy of its own model, and can also contribute to improving the identification accuracy of models of other identification systems, It is an object to provide a model providing method and a model providing program.

An identification system according to the present invention includes: learning means for learning a model for identifying an object represented by data using teacher data; first model storage means for storing a model learned by the learning means; First identifying means for identifying an object represented by data using a model learned by the first and second model storages for storing individual models learned by a plurality of predetermined first identification systems, respectively. Means, when a model learned by the first identification system is received from the first identification system, the model learned by the first identification system stored in the second model storage means is Model updating means for updating to a model received from the first identification system; and, in a predetermined case, for each model stored in the second model storage means, A second identification means for identifying an object represented by the data identified by the first identification means; a learning means comprising: a label for data determined based on an identification result derived by the second identification means; Is re-learned using the teacher data including the following, the model stored in the first model storage unit is updated to the re-learned model, and the model learned by the learning unit is set to a predetermined one. And a model transmitting means for transmitting to one or more second identification systems.

Further, in the model providing method according to the present invention, a model for identifying an object represented by data is learned using teacher data, the model is stored in the first model storage unit, and the model is stored in the first model storage unit. Executing a first identification process for identifying an object represented by data using the model that has been performed, and storing individual models learned by a plurality of predetermined first identification systems in a second model storage means. When a model learned by the first identification system is received from the first identification system, the model learned by the first identification system stored in the second model storage means is stored in the second model storage unit. The model is updated to the model received from the first identification system, and in a predetermined case, for each model stored in the second model storage means, the data to be identified in the first identification process is determined. Executing a second identification process for identifying an object represented by the data, and re-learning the model using a label for data determined based on the identification result derived in the second identification process and teacher data including the data. Then, the model stored in the first model storage unit is updated to the relearned model, and the model stored in the first model storage unit is replaced with one or more predetermined second identifications. The data is transmitted to the system.

In addition, the model providing program according to the present invention is a learning process for causing a computer to learn a model for identifying an object represented by data using teacher data, and to store the model in a first model storage unit. A first identification process for identifying an object represented by data using a model stored in the model storage means, and individual models learned by a plurality of predetermined first identification systems are respectively assigned to a second model. Processing to be stored in the storage means; when a model learned by the first identification system is received from the first identification system, the model is learned by the first identification system stored in the second model storage means. Model updating processing for updating the model that has been updated to the model received from the first identification system, and in a predetermined case, the individual model stored in the second model storage means. For each model, a second identification process for identifying an object represented by data to be identified in the first identification process, a label for data determined based on an identification result derived in the second identification process, and the data A re-learning process for re-learning the model using the included teacher data and updating the model stored in the first model storage unit to the re-learned model, and stored in the first model storage unit A model transmission process for transmitting the model to one or more predetermined second identification systems is performed.

According to the present invention, it is possible to re-learn the own model so as to improve the identification accuracy of the own model, and to contribute to the improvement of the identification accuracy of the model of another identification system.

It is a schematic diagram showing a situation where a plurality of identification systems of the present invention are provided. It is explanatory drawing which shows the example of a 1st identification system and a 2nd identification system. FIG. 1 is a block diagram illustrating a configuration example of an identification system 100 according to an embodiment of the present invention. It is a schematic diagram which shows the example of an internally generated model and an externally generated model. It is a schematic diagram which shows the example of the screen which a determination part displays on a display apparatus in the 1st determination method. It is a schematic diagram which shows the example of the screen which a determination part displays on a display apparatus in the 3rd determination method. It is a schematic diagram which shows an example of the screen which an area | region correction GUI display control part displays on a display apparatus. It is a schematic diagram which shows the other example of the screen which an area | region correction GUI display control part displays on a display apparatus. It is a schematic diagram which shows the example of the screen which a display control part displays. FIG. 4 is an explanatory diagram illustrating a specific example of a first calculation method. FIG. 9 is an explanatory diagram illustrating a specific example of a second calculation method. 11 is a flowchart illustrating an example of processing progress from the time when the camera captures an image to the time when a second identification unit performs an identification process on an image. It is a flowchart which shows the example of a process progress when re-learning a model and transmitting the model to a 2nd identification system. It is a flowchart which shows the example of a process progress of the 3rd model transmission mode. It is a flowchart which shows the example of a process progress of the 4th model transmission mode. FIG. 11 is a schematic block diagram illustrating a configuration example of a computer included in an identification system according to an embodiment of the present invention or a modified example thereof. It is a block diagram showing the outline of the identification system of the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a schematic diagram showing a situation where a plurality of identification systems of the present invention are provided. FIG. 1 illustrates a case where six identification systems 100 are provided at various locations, but the number of identification systems 100 provided at various locations is not particularly limited. In the present embodiment, a description will be given assuming that the plurality of identification systems 100 have the same configuration.

Each identification system 100 can communicate, for example, via a communication network.

Each of the identification systems 100 includes a data collection unit (a data collection unit 101 shown in FIG. 3 described later). The data collection units (not shown in FIG. 1; see FIG. 3 described later) of each identification system 100 are installed in various places where data is collected. The data collection unit collects data at a location where the data collection unit is installed. For example, the data collection unit collects image and audio data at the installation location. The data collection unit is realized by a camera or a microphone. For example, the data collection unit may collect images by photographing a monitoring place. Further, for example, audio data may be collected by recording at an installation location.

Each individual identification system 100 includes a computer separate from the data collection unit, and the computer identifies an object represented by data (image, audio data, and the like).

(4) Each identification system 100 learns a model using data collected by the data collection unit as teacher data. This model is a model for identifying an object represented by data.

各 Then, each identification system 100 provides a model, and re-learns its own model by using a model provided by another identification system 100.

Here, when focusing on one identification system 100, another identification system 100 that transmits a model to the identification system 100 of interest is predetermined. Another identification system 100 that transmits a model to the identification system 100 of interest is referred to as a first identification system. In the present embodiment, a description will be given assuming that a plurality of first identification systems are determined for the identification system 100 of interest. It can be said that the identification system 100 of interest receives the model from the first identification system.

Also, another identification system 100 that is a transmission destination when the identification system 100 of interest transmits the model is predetermined. Another identification system 100 that is a transmission destination when the identification system 100 of interest transmits the model is referred to as a second identification system. In the present embodiment, a description will be given assuming that one or more second identification systems are determined for the identification system 100 of interest. The identification system 100 of interest can be said to provide a model to the second identification system.

The first identification system and the second identification system are predetermined for each individual identification system 100. For example, an administrator who manages each identification system 100 may determine a first identification system and a second identification system for each individual identification system 100 in advance.

Each identification system 100 has the same configuration, but the first identification system and the second identification system are individually determined for each identification system 100.

FIG. 2 is an explanatory diagram showing an example of the first identification system and the second identification system. Here, attention is focused on the identification system 100a shown in FIG. In FIG. 2, arrows indicate the direction in which the model is sent. In the example shown in FIG. 2, regarding the identification system 100a of interest, the identification systems 100b and 100c are defined as first identification systems, and the

identification systems

100b, 100c and 100d are defined as second identification systems. Regarding the

identification systems

100b, 100c, and 100d, a first identification system and a second identification system are individually defined.

FIG. 3 is a block diagram showing a configuration example of the identification system 100 according to the embodiment of the present invention. The identification system 100 includes a data collection unit 101 and a computer 102. The data collection unit 101 and the computer 102 are communicably connected by wire or wirelessly. In the following description, a case where the data collection unit 101 is a camera will be described as an example, and the data collection unit 101 will be referred to as a camera 101. The camera 101 performs shooting from the installation location of the camera 101. The installation location of the camera 101 and the installation location of the computer 102 may be different.

The computer 102 includes a learning unit 103, a first model storage unit 104, a data acquisition unit 105, a first identification unit 106, a determination unit 107, an area correction GUI (Graphical User Interface) display control unit 108, , Region extraction unit 109, second model storage unit 110, model update unit 121, second identification unit 111, display control unit 112, attribute data storage unit 113, integration unit 114, display device 115, a mouse 116, a result storage unit 117, and a model transmission unit 122.

The learning unit 103 learns a model by machine learning using the image obtained by the camera 101 as the teacher data. Hereinafter, a case where the learning unit 103 learns a model by deep learning will be described as an example. The teacher data is, for example, a set of a set of an image obtained by photographing with the camera 101 and a label indicating an object appearing in the image. The label may be determined by the operator of the identification system 100. In learning, a model is learned (generated) using such a set of sets as teacher data.

{Circle around (4)} When a predetermined image and a label for the image are specified, the learning unit 103 adds a set of the image and the label to the teacher data, and re-learns the model by deep learning. The predetermined image is an image determined by the determination unit 107 described later to cause the second identification unit 111 to execute the identification processing. Note that, in the present embodiment, an example will be described in which a region in which an object is extracted is extracted from an image determined as described above, and a pair of an image of the extracted region and a label is added to teacher data. I do.

モデル A model generated by the learning unit 103 through learning may be hereinafter referred to as an internally generated model. In addition, as described later, the second model storage unit 110 stores a model similarly generated by the first identification system predetermined for the identification system 100 shown in FIG. Hereinafter, a model generated by the first identification system may be referred to as an externally generated model to distinguish it from an internally generated model. Note that a model updating unit 121 described below receives the model from the first identification system, and stores the model in the second model storage unit 110 as an externally generated model.

The internally generated model and the externally generated model are models for identifying an object appearing in a given new image. Hereinafter, in both the internally generated model and the externally generated model, the objects appearing in the image are “car”, “motorcycle”, “bus”, and “background (that is, car, motorcycle and bus are not shown)”. The description will be made assuming that the model is a model for determining which one. When learning such a model, the operator determines one of “automobile”, “motorcycle”, “bus”, and “background” as a label to be paired with an image in the teacher data for each image. In addition, even when the externally generated model is generated by the first identification system, the operator of the first identification system may use the “automobile”, “motorcycle”, “bus”, One of the “background” is determined for each image.

In the present embodiment, a case where the first identification unit 106 determines whether the object shown in the image is “car”, “motorcycle”, “bus”, or “background” using a model is described. Although described as an example, the target determined using the model is not limited to “automobile”, “motorcycle”, “bus”, and “background”. The operator may prepare teacher data according to the purpose of the identification processing, and let the learning unit 103 learn the model using the teacher data. It should be noted that the objects to be determined using the model (in this example, “automobile”, “motorcycle”, “bus”, and “background”) are common to each identification system 100.

The learning unit 103 causes the first model storage unit 104 to store the internally generated model generated by the deep learning. The first model storage unit 104 is a storage device that stores an internally generated model.

FIG. 4 is a schematic diagram illustrating an example of an internally generated model and an externally generated model. Assuming that the number of pixels of an image applied to the model is n, the image can be represented as a vector (X1, X2,..., Xn) ^T having each pixel value of n pixels as an element. . For example, X1 represents the pixel value of the first pixel in the image. The same applies to X2 to Xn. Here, T means transposition. The model has a plurality of layers and includes a plurality of coefficients for each layer. In the example shown in FIG. 4, the first layer includes coefficients a1 to am, and the second layer includes coefficients b1 to bj. The individual elements X1 to Xn of the vector representing the image are associated with the respective coefficients a1 to am of the first layer. In FIG. 4, this association is represented by a line. Further, each coefficient of a certain layer is associated with each coefficient of the next layer. In FIG. 4, this association is also represented by a line. A weight is determined between the associated elements. For example, weights are determined for the associated a1 and b1, the associated a1 and b2, and the like.

The learning unit 103 determines the number of layers, the number of coefficients included in each layer, the value of each coefficient in each layer, and the value of the weight between associated elements by performing deep learning using the teacher data. . Determining these values corresponds to generating an internally generated model.

If the teacher data is different, the number of layers, the number of coefficients included in each layer, the value of each coefficient in each layer, and the value of the weight between associated elements change. Accordingly, the internally generated model and the externally generated model can be represented as shown in the format illustrated in FIG. 4, but the number of layers, the number of coefficients included in each layer, the value of each coefficient in each layer, the , Etc., differ between the internally generated model and the externally generated model. In the present embodiment, the second model storage unit 110 stores each of the externally generated models learned by the plurality of first identification systems. Each of the externally generated models is also generated based on different teacher data in different identification systems 100, and thus the number of layers and the like are different for each externally generated model.

The data acquisition unit 105 acquires a new image obtained by the camera 101 by shooting from the camera 101. The data acquisition unit 105 is an interface for receiving an image from the camera 101.

When the data acquisition unit 105 acquires a new image from the camera 101, the first identification unit 106 applies the image to an internally generated model stored in the first model storage unit 104, thereby Identify the object represented by the image. In this example, the first identification unit 106 applies the image to the internally generated model to determine whether the object shown in the image is “car”, “motorcycle”, or “bus”. Alternatively, it is determined whether only the “background” is captured.

When an image is obtained, a vector (X1, X2,..., Xn) ^T representing the image is determined. The first identification unit 106 calculates “automobile” by using the vector (X1, X2,..., Xn) ^T , each coefficient of each layer included in the internally generated model, and each weight included in the model. , “Motorcycle”, “bus”, and “background” are calculated. Then, the first identification unit 106 determines the label with the highest reliability among “car”, “motorcycle”, “bus”, and “background” as a label indicating an object appearing in the image. . For example, as a result of the first identification unit 106 applying a vector representing an image to the model, the reliability of “car”, “motorcycle”, “bus”, and “background” is “0.6”, “0”, .2 "," 0.1 ", and" 0.1 "are obtained. In this case, the first identification unit 106 identifies the object appearing in the image as an “automobile” having the highest reliability “0.6”. When the determined label is other than “background”, the first identifying unit 106 represents a rectangular area surrounding the object (“car”, “motorcycle” or “bus”) shown in the image as the image. The determination is made by an operation using the vector and the internally generated model. The fact that the determined label is “background” means that it is determined that the object is not shown in the image. In this case, the first identification unit 106 determines that the object Does not determine the rectangular area surrounding.

{Circle around (1)} The first identification unit 106 associates the image subjected to the identification processing, the label corresponding to the identification result, and the reliability corresponding to the label, and stores the image in the result storage unit 117. For example, as in the above example, it is assumed that the first identification unit 106 determines that the object shown in the image is “car” having the highest reliability “0.6”. In this case, the first identification unit 106 causes the result storage unit 117 to store the image, the label “car”, and the reliability “0.6” in association with each other. The result storage unit 117 is a storage device that stores identification results and the like. However, the result storage unit 117 additionally stores information indicating a rectangular area in the image, as described later.

The second model storage unit 110 is a storage device that stores a model different from the internally generated model (the model generated by the learning unit 103). More specifically, the second model storage unit 110 stores individual models (externally generated models) learned by a plurality of first identification systems predetermined for the identification system 100 shown in FIG. Remember. Each model stored in the second model storage unit 110 is represented in the same format as the model schematically shown in FIG.

The model updating unit 121 receives a model from each of a plurality of first identification systems predetermined for the identification system 100 shown in FIG. 3, and uses the model as an externally generated model in the second model storage unit 110. To memorize.

Each first identification system includes a model transmission unit 122 described later, similarly to the identification system 100 shown in FIG. Then, the model transmitting unit 122 of each first identification system transmits the model learned by the first identification system to the identification system 100 shown in FIG. 3 as appropriate.

The model updating unit 121 receives the model transmitted by the first identification system. It is assumed that the first identification system transmits a model to the identification system 100 shown in FIG. 3 for the first time. In that case, the second model storage unit 110 has not yet stored the model learned by the first identification system. At this time, the model updating unit 121 stores the received model as an externally generated model in the second model storage unit 110 in association with the information indicating the first identification system that is the transmission source. It is also assumed that the first identification system has previously transmitted a model to the identification system 100 shown in FIG. In that case, the second model storage unit 110 has already stored the model learned by the first identification system. At this time, the model updating unit 121 updates a model already stored in the second model storage unit 110 as a model learned by the first identification system to a newly received model.

In the following description, the second model storage unit 110 stores all the individual models (externally generated models) learned by a plurality of first identification systems predetermined for the identification system 100 shown in FIG. It is assumed that it has been stored. In this state, when receiving the model from the first identification system, the model updating unit 121 updates the model stored in the second model storage unit 110 to the received model.

The second identification unit 111 applies a predetermined image among the images identified by the first identification unit 106 to the externally generated model stored in the second model storage unit 110, An object in the predetermined image is identified. The second identification unit 111 performs this processing for each externally generated model. The second identification unit 111 calculates the reliability of “automobile”, “motorcycle”, “bus”, and “background” by applying a predetermined image to the externally generated model. Then, the second identification unit 111 determines a label with the highest reliability among “automobile”, “motorcycle”, “bus”, and “background” as a label indicating an object appearing in the image. .

In addition, the predetermined image of the images identified by the first identification unit 106 is a predetermined image among the images identified by the first identification unit 106. This is an image determined to cause the image processing unit 111 to execute the identification processing.

The determination unit 107 determines an image that causes the second identification unit 111 to execute the identification process, from among the images that have been identified by the first identification unit 106. Hereinafter, three types of determination methods will be exemplified as a method in which the determination unit 107 determines an image to be subjected to the identification processing by the second identification unit 111 among the images identified by the first identification unit 106. Will be explained. The determination unit 107 may employ only one of the following three types of determination methods. Alternatively, the determination unit 107 may employ a plurality of determination methods among the following three types of determination methods. In this case, if the determining unit 107 determines to cause the second identifying unit 111 to execute the identifying process on a certain image by any one of the plurality of determining methods, the determining unit 107 Thus, it is determined that the second identification unit 111 executes the identification processing.

[First determination method]
The first determination method is that, when the label determined by the first identification unit 106 as a label representing an object appearing in an image is incorrect, the determination unit 107 determines the second This is a method of determining that the identification unit 111 performs the identification processing. That is, this is a method in which the determination unit 107 determines that the second identification unit 111 performs the identification process on the image that is incorrectly identified by the first identification unit 106. Whether or not the label determined by the first identification unit 106 is incorrect may be determined by, for example, an operator of the identification system 100. Hereinafter, this case will be described as an example. When the first identification unit 106 determines a label for an image, the determination unit 107 provides a GUI for allowing the operator to input the image, the label determined for the image, and whether the label is correct. (In this example, two buttons are used.) Is displayed on the display device 115. FIG. 5 is a schematic diagram illustrating an example of a screen displayed on the display device 115 by the determination unit 107 in the first determination method.

When the first identification unit 106 determines a label for an image, the determination unit 107 determines whether the image 301 that is the identification target of the first identification unit 106 and the first identification unit 106, as illustrated in FIG. A screen showing the determined label 302 (“motorcycle” in the example shown in FIG. 5) and the first button 304 and the second button 305 is displayed on the display device 115. The first button 304 is a button for inputting that the label for the image is correct. Clicking on the first button 304 means that information indicating that the label for the image is correct has been input by the operator. I do. The second button 305 is a button for inputting that the label for the image is incorrect. When the second button 305 is clicked, information indicating that the label for the image is incorrect is displayed by the operator. Means input from In the example shown in FIG. 5, the image 301 shows the self-disembarkation, but “Motorcycle” is displayed as the label determined by the first identification unit 106. Therefore, the operator clicks the second button 305 using the mouse 116. In the example illustrated in FIG. 5, if “automobile” is displayed as the label determined by the first identification unit 106, the operator clicks the first button 304.

When the second button 305 is clicked on the screen illustrated in FIG. 5, the determining unit 107 determines that the label determined by the first identifying unit 106 is incorrect, and the first identifying unit 106 identifies the label. It is determined that the second identification unit 111 performs the identification processing on the target image 301.

When the first button 304 is clicked, the determination unit 107 determines that the second identification unit 111 does not perform the identification processing on the image 301 that is the identification target of the first identification unit 106. I do.

[Second determination method]
In a second determination method, when the reliability corresponding to the label determined for the image is equal to or less than a predetermined threshold, the determination unit 107 identifies the image to the second identification unit 111. This is a method of determining to execute the processing.

That is, when the reliability corresponding to the label determined for the image by the first identification unit 106 is equal to or smaller than the threshold, the determination unit 107 causes the second identification unit 111 to execute the identification process on the image. Is determined. If the reliability corresponding to the label determined by the first identification unit 106 for the image exceeds the threshold, the determination unit 107 causes the second identification unit 111 to perform identification processing on the image. Is determined not to be executed. The threshold value is, for example, “0.5”, but may be a value other than “0.5”.

In a second determination method, the determination unit 107 determines whether or not to cause the second identification unit 111 to execute identification processing on an image by comparing the reliability derived by the first identification unit 106 with a threshold. decide. Therefore, it is not necessary to display the screen illustrated in FIG. 5 in the second determination method.

[Third determination method]
In the third determination method, although the label determined for the image by the first identification unit 106 is “background”, “automobile”, “motorcycle”, or “bus” is reflected in the image. In this case, the determination unit 107 determines that the second identification unit 111 performs the identification process on the image. In other words, although the third determination method determines that the image does not include any of “car”, “motorcycle”, and “bus” in the image, , "Motorcycle" or "bus", the determination unit 107 determines that the second identification unit 111 performs the identification process on the image. When the specified label is “background”, the operator of the identification system 100 determines whether or not “car” or the like is included in the image.

In the third method, when “background” is determined as a label for an image, the determination unit 107 sets a screen representing the image, the label “background”, and the first button 304 and the second button 305 described above. Is displayed on the display device 115. FIG. 6 is a schematic diagram illustrating an example of a screen displayed on the display device 115 by the determination unit 107 in the third determination method.

When the first identification unit 106 determines “background” as a label for the image, the determination unit 107 determines, as illustrated in FIG. 6, the image 301 that is the identification target of the first identification unit 106 and a label A screen representing 302 and first button 304 and second button 305 is displayed on display device 115. On the screen displayed by the third determination method, “background” is displayed as the label 302. The first button 304 and the second button 305 are the same as the first button 304 and the second button 305 shown in FIG.

In the example shown in FIG. 6, although the label determined by the first identification unit 106 for the image 301 is “background (cars, motorcycles, and buses are not shown)”, the image 301 includes The car is shown. Therefore, the operator clicks the second button 305 using the mouse 116. If none of the car, motorcycle, and bus is shown in the image 301, the operator clicks the first button 304.

When the second button 305 is clicked on the screen illustrated in FIG. 6, the determination unit 107 specifies the label “background”, but the image includes any of “car”, “motorcycle”, and “bus”. It is determined that the image is captured, and it is determined that the second identification unit 111 performs the identification process on the image.

When the first button 304 is clicked on the screen illustrated in FIG. 6, the determination unit 107 does not show any of “car”, “motorcycle”, and “bus” in the image, and the label “ It is determined that the “background” is correct, and it is determined that the second identifying unit 111 does not execute the identification process on the image.

Next, the area correction GUI display control unit 108 will be described. As described above, when the label defined for the image is other than “background”, the first identification unit 106 surrounds the object (“car”, “motorcycle” or “bus”) shown in the image. Determine the rectangular area. The area correction GUI display control unit 108 displays the image determined by the determination unit 107 to cause the second identification unit 111 to execute the identification processing on the display device 115 together with the rectangular area, and further corrects the rectangular area. Is displayed on the display device 115. However, since the label “background” is defined for the image determined by the above-described third method, the rectangular area is not determined. In this case, the area correction GUI display control unit 108 does not display the rectangular area.

FIG. 7 is a schematic diagram showing an example of a screen displayed on the display device 115 by the area correction GUI display control unit 108. A rectangular area 309 illustrated in FIG. 7 is a rectangular area defined by the first identification unit 106 as an area surrounding “car” in the image 301. Further, the area correction GUI display control unit 108 includes a determination button 307 and a correction button 308 in the screen. The confirm button 307 is a button for the operator to instruct to confirm the displayed rectangular area. The correction button 308 is a button for the operator to instruct to accept the correction of the rectangular area 309.

In the example shown in FIG. 7, the rectangular area 309 is appropriate as a rectangular area surrounding “car” in the image 301. When the operator makes such a determination, the operator clicks the OK button 307. When the confirm button 307 is clicked, the area extracting unit 109 determines the rectangular area 309 in the image 301 at that time.

FIG. 8 is a schematic diagram showing another example of a screen displayed on the display device 115 by the area correction GUI display control unit 108. In the example illustrated in FIG. 8, the rectangular area 309 is not appropriate as a rectangular area surrounding “car” in the image 301. In this case, the area correction GUI display control unit 108 accepts an appropriate rectangular area as a rectangular area surrounding “automobile” in accordance with the operation of the operator. When the inappropriate rectangular area 309 illustrated in FIG. 8 is displayed, the operator clicks the correction button 308. After the correction button 308 is clicked, the area correction GUI display control unit 108 receives the correction of the positions of the vertices and sides of the rectangular area 309 in accordance with the operation performed by the operator using the mouse 116. The operator can correct the position of the vertices and sides to correct the rectangular area 309 to an appropriate position and size as illustrated in FIG. The area correction GUI display control unit 108 receives such correction. The operator clicks the OK button 307 after correcting the rectangular area 309 to an appropriate position or size surrounding the object (“automobile” in this example) shown in the image 301. As described above, when the confirm button 307 is clicked, the region extracting unit 109 decides the rectangular region 309 in the image 301 at that time. In this example, the area extracting unit 109 determines the rectangular area 309 after the correction.

{Circle around (2)} As described above, since the label “background” is defined for the image determined by the above-described third method, the rectangular area is not determined. In this case, the area correction GUI display control unit 108 does not display the rectangular area 309 on the screen illustrated in FIG. In this case, when the operator clicks the correction button 308, the area correction GUI display control unit 108 displays a rectangular area 309 at an arbitrary size in an arbitrary place of the image 301, and responds to an operation performed by the operator using the mouse 116. Then, the correction of the positions of the vertices and sides of the rectangular area 309 is accepted. The operator corrects the displayed rectangular area 309 to an appropriate position and size surrounding the object shown in the image 301, and then clicks the OK button 307. When the confirm button 307 is clicked, the area extracting unit 109 determines the rectangular area 309 in the image 301 at that time.

As described above, when the confirm button 307 is clicked, the area extracting unit 109 determines the rectangular area 309 in the image 301 at that time. Then, the area extracting unit 109 extracts the determined rectangular area from the image. This rectangular area is an area surrounding the object shown in the image. The region extraction unit 109 represents the determined rectangular region in association with the image stored in the result storage unit 117, the label as the identification result by the first identification unit 106, and the reliability corresponding to the label. Information is also stored in the result storage unit 117. The information representing the rectangular area is, for example, the coordinates of each vertex of the rectangular area.

The second identification unit 111 identifies an object appearing in the image of the rectangular area with respect to the image of the rectangular area extracted by the area extracting unit 109. The second identification unit 111 executes this process for each of the externally generated models stored in the second model storage unit 110.

The second identification unit 111 calculates the reliability of “automobile”, “motorcycle”, “bus”, and “background” by applying the extracted image of the rectangular area to the externally generated model. Then, the second identification unit 111 determines a label with the highest reliability among “automobile”, “motorcycle”, “bus”, and “background” as a label indicating an object appearing in the image. . In addition, the second identification unit 111 has already stored in the result storage unit 117 the reliability calculated for each label, the label indicating the object appearing in the image, and the reliability corresponding to the label. The result is stored in the result storage unit 117 in association with the image. The second identification unit 111 executes this process for each externally generated model. Hereinafter, for the sake of simplicity, there are two first identification systems defined for the identification system 100 shown in FIG. 3 and the externally generated models stored in the second model storage unit 110 The description will be made on the assumption that the number is two. Then, one of the two externally generated models is represented by a symbol “A”, and the other is represented by a symbol “B”.

In this case, the result storage unit 117 stores the image, the label determined by the first identification unit 106 performing the identification processing on the image, the reliability corresponding to the label, and the Information indicating a rectangular area is stored. Further, in association with the information, the reliability of each label obtained by the second identification unit 111 applying the image of the rectangular area to the externally generated model A, and the label having the highest reliability and the label The corresponding reliability, the reliability for each label obtained by applying the image of the rectangular area to the externally generated model B by the second identification unit 111, and the label having the highest reliability and the reliability corresponding to the label. The degree is also stored in the result storage unit 117.

The result storage unit 117 stores the set of information as described above.

However, for an image for which the determining unit 107 has not determined that the second identifying unit 111 performs the identifying process, the image and a label determined by the first identifying unit 106 performing the identifying process on the image. And the reliability corresponding to the label are stored in the result storage unit 117, and information indicating a rectangular area in the image is not stored.

The display control unit 112 reads one set of information from the information stored in the result storage unit 117, and displays the image, the label derived by the first identification unit 106, the reliability corresponding to the label, The display unit 115 displays on the display device 115 a screen including the label derived for each externally generated model by the second identification unit 111 and the reliability corresponding to the label.

FIG. 9 is a schematic diagram showing an example of a screen displayed by the display control unit 112. The display control unit 112 corresponds to the label derived by the first identification unit 106 and the reliability 501 corresponding to the label, and corresponds to the label derived by the second identification unit 111 using the externally generated model A and the label. A screen in which the reliability 502 and the label derived by the second identification unit 111 using the externally generated model B and the reliability 503 corresponding to the label are superimposed on the image 301 is displayed on the display device 115. In the example illustrated in FIG. 9, the display control unit 112 also displays the determined rectangular area 309 so as to be superimposed on the image 301. In this example, the case where the number of the externally generated models stored in the second model storage unit 110 is two is illustrated, but the number of the externally generated models may be three or more.

(4) The display control unit 112 further displays a check box 504, a re-learning button 505, and

screen switching buttons

506 and 507 on this screen.

The check box 504 is a GUI for designating whether or not to include the image 301 displayed on the screen (more specifically, the image of the rectangular area 309 extracted from the image 301) in the teacher data. When the check box 504 is checked, it means that the image of the rectangular area 309 extracted from the image 301 is included in the teacher data. If the check box 504 is not checked, it means that the image 301 is not included in the teacher data. Note that the display control unit 112 may display the check box 504 in a state where the check box is checked in advance, according to the reliability derived using the externally generated model. For example, in a set of a label and a reliability derived using an externally generated model, if there is one or more pairs whose reliability is greater than a threshold (for example, “0.5”), the display control unit 112 The check box 504 may be displayed in a checked state. The operator can check or uncheck the check box 504 by clicking the check box 504 with the mouse 116. The operator may determine whether to include the image of the rectangular area 309 extracted from the image 301 in the teacher data by referring to the image 301 and the label and reliability derived for each externally generated model. Then, based on the determination, the operator may determine whether to check the check box 504.

The

screen switching buttons

506 and 507 are buttons for switching to a screen displaying a different image. For example, when the screen switching button 506 is clicked, the display control unit 112 switches to a screen similar to the screen illustrated in FIG. 9 including an image preceding the image 301 in chronological order. Further, for example, when the screen switching button 507 is clicked, the display control unit 112 switches to a screen similar to the image illustrated in FIG. 9 that includes an image later than the image 301 in chronological order. The operator may determine whether or not to check the check box 504 on each of the switched screens.

The re-learning button 505 is a button for the operator to instruct the identification system 100 to re-learn the internally generated model. When the re-learning button 505 is clicked, the integration unit 114 specifies a label for each screen image in which the check box 504 is checked. In the following description, for the sake of simplicity, the case where the check box 504 is checked only on the screen illustrated in FIG. 9 will be described as an example. In this case, the integration unit 114 specifies the label of the image 301 illustrated in FIG.

Hereinafter, the attribute data storage unit 113 will be described first before the process of the integration unit 114 specifying the label of one image is described. The attribute data storage unit 113 stores data (attribute data) indicating the attributes of the camera 101 connected to the computer 102 (computer 102 shown in FIG. 3) including the attribute data storage unit 113, and stores the data in the second model storage unit 110. This is a storage device for storing the attribute data of the camera 101 of each identification system 100 (that is, each first identification system) that has generated each of the stored externally generated models. The attribute data of the camera 101 of the identification system 100 (first identification system) that has generated an externally generated model is referred to as attribute data corresponding to the externally generated model.

The attributes of the camera 101 include an attribute of the camera 101 itself, an attribute depending on an environment in which the camera 101 is installed, and the like. The value of each attribute is represented by a numerical value. The value of each attribute may be determined in advance by the administrator of each identification system 100 according to the setting of the camera 101 and the installation environment. The attribute data is represented by a vector having such attribute values (numerical values) as elements.

The attribute data of the camera 101 includes at least “angle of view of the camera 101”, “whether the camera 101 is installed indoors or outdoors”, “photographing target of the camera 101”, and “ It includes the value of at least some of the attributes “moving direction”. Further, which attribute value is represented by a vector as attribute data is common to all the identification systems 100. Regarding which attribute value is a vector element, Common to the identification system 100. The numerical value of each element of the vector may be different for each identification system 100.

Since the “angle of view of the camera 101” is represented by a numerical value, the administrator may determine the numerical value representing the angle of view as a vector element.

Regarding the attribute “whether the camera 101 is installed indoors or outdoors”, for example, when the camera 101 is installed indoors, the value of this attribute is set to “0” and the camera 101 Is installed outdoors, the value of this attribute may be set to “1”.

In addition, regarding the attribute of “the object to be captured by the camera 101”, for example, when the camera 101 is installed to capture an image of a vehicle (for example, when the camera 101 is installed toward a road), this attribute The value is set to “0”. When the camera 101 is installed so as to photograph a pedestrian (for example, when the camera 101 is installed facing a sidewalk), the value of this attribute is set to “1”. When the camera 101 is installed so as to photograph both the vehicle and the pedestrian (for example, when the camera 101 is installed toward a road where both the vehicle and the pedestrian pass), the value of this attribute is set. Is set to “0.5”.

As for the attribute “moving direction of the photographing target of the camera 101”, a reference axis based on the main axis direction of the camera 101 and the like is determined, and the angle between the reference axis and the main moving direction of the photographing target is defined as the value of this attribute It may be determined as

Also, attribute values other than the above may be included in the attribute data. For example, values such as “the height of the installation location of the camera 101”, “depression angle of the camera 101”, and “resolution of the camera 101” may be included in the attribute data. Since “the height of the installation location of the camera 101”, “depression angle of the camera 101”, and “resolution of the camera 101” are all represented by numerical values, these numerical values may be determined as vector elements.

The attribute data storage unit 113 stores the attribute data (vector) of the camera 101 connected to the computer 102 (computer 102 shown in FIG. 3) including the attribute data storage unit 113. This attribute data is referred to as reference attribute data. Further, the attribute data storage unit 113 stores the attribute data of the camera 101 of each first identification system that has generated each externally generated model stored in the second model storage unit 110. In the present embodiment, the second model storage unit 110 stores the externally generated model A and the externally generated model B. Therefore, the attribute data storage unit 113 stores, in addition to the reference attribute data, attribute data corresponding to the externally generated model A (described as attribute data A) and attribute data corresponding to the externally generated model B (attribute data B and Note) is also stored. The attribute data A is attribute data of the camera 101 of the first identification system that has generated the externally generated model A. Similarly, the attribute data B is attribute data of the camera 101 of the first identification system that has generated the externally generated model B.

The administrator who manages each identification system 100 may store the attribute data of the camera 101 in FIG. 3 in the attribute data storage unit 113 as reference attribute data. Further, the administrator sets the attribute data of the camera 101 of each of the two first identification systems defined for the identification system 100 shown in FIG. 3 in the attribute data storage unit 113 as attribute data A and attribute data B. What is necessary is just to memorize.

The integration unit 114 determines the reliability of each label (in the present embodiment, “car”, “motorcycle”, “bus”, “bus”) derived by the second identification unit 111 for each externally generated model for the image. The reliability of each "background" is integrated for each label, and the label of the image is specified based on the integration result.

At this time, the integration unit 114 includes the reference attribute data (that is, the attribute data of the camera 101 of the identification system 100 including the integration unit 114) and the plurality of first identification systems that have generated the externally generated model A and the externally generated model B. Is calculated for each of the first identification systems. In the present embodiment, the integration unit 114 calculates the similarity between the reference attribute data and the attribute data A and the similarity between the reference attribute data and the attribute data B, respectively. The similarity between the reference attribute data and the attribute data A is referred to as a similarity corresponding to the externally generated model A. The similarity between the reference attribute data and the attribute data B is referred to as a similarity corresponding to the externally generated model B.

Attribute data is represented by a vector. When calculating the similarity between two pieces of attribute data (vectors), the integration unit 114 may calculate the reciprocal of the distance between the two vectors as the similarity.

The integration unit 114 weights the reliability of each label derived for each externally generated model when the label is integrated for each label, with the similarity corresponding to the externally generated model, and integrates them. The integrating unit 114 may specify the label having the highest reliability integration result as the image label.

(4) An operation for integrating the reliability of each label derived for each externally generated model for each label will be specifically described. Two calculation methods will be described as a calculation method in which the integration unit 114 integrates the reliability. Here, a case will be described in which the reliability derived for each externally generated model is integrated for one label. The integration unit 114 may perform the same operation on other labels and integrate the reliability derived for each externally generated model.

[First operation method]
First, a first calculation method for integrating reliability will be described. Let Li be the reliability of the focused label obtained using the i-th externally generated model. Also, the similarity calculated for the i-th externally generated model (the similarity between the reference attribute data and the attribute data corresponding to the i-th externally generated model) is Wi. The number of externally generated models stored in the second model storage unit 110 is N. In this case, the integrating unit 114 may integrate the reliability of the label of interest by calculating the following expression (1).

That is, the integrating unit 114 may calculate the product of Li and Wi for each externally generated model, and use the average value of the product as the integrated result of the reliability of the label of interest. The integration unit 114 performs the same operation for other labels. Then, the integration unit 114 specifies the label with the highest integration result as the label of the image.

FIG. 10 is an explanatory diagram showing a specific example of the first calculation method. It is assumed that there are two externally generated models A and B. The reliability of “car”, “motorcycle”, “bus”, and “background” derived using the externally generated model A is “0.1”, “0.7”, “0.1”, “ 0.1 ". It is also assumed that the similarity calculated for the externally generated model A is “0.9”. The integrating unit 114 calculates a result obtained by multiplying the degree of similarity by “0.9” for each reliability. As a result, the multiplication results (product) of “0.09”, “0.63”, “0.09”, “0.09” for “car”, “motorcycle”, “bus”, and “background” respectively. Is obtained.

Also, the reliability of “car”, “motorcycle”, “bus” and “background” derived using the externally generated model B are “0.1”, “0.6”, and “0.2”, respectively. , "0.1". It is also assumed that the similarity calculated for the externally generated model B is “0.8”. The integrating unit 114 calculates a result of multiplying the degree of similarity “0.8” for each of the above degrees of reliability. As a result, the multiplication results (product) of “0.08”, “0.48”, “0.16”, “0.08” for “car”, “motorcycle”, “bus”, and “background” respectively. Is obtained.

The integration unit 114 calculates an average value of the multiplication results (products) obtained for each of the “car”, “motorcycle”, “bus”, and “background”. The average values calculated for each of "car", "motorcycle", "bus" and "background" are "0.085", "0.555", "0.125", and "0.085". Therefore, the integrating unit 114 specifies “motorcycle” having the highest average value (integrated result) as the image label.

[Second calculation method]
Next, a second calculation method for integrating reliability will be described. As in the case described above, the reliability of the label of interest obtained using the i-th externally generated model is Li. Also, the similarity calculated for the i-th externally generated model (the similarity between the reference attribute data and the attribute data corresponding to the i-th externally generated model) is Wi. Also, the sum of the individual similarities calculated for the individual externally generated models is Wt. The number of externally generated models stored in the second model storage unit 110 is N. The integrating unit 114 may calculate Wt by the calculation of the following equation (2).

In this case, the integrating unit 114 may integrate the reliability of the label of interest by calculating the following expression (3).

That is, the integration unit 114 calculates, for each externally generated model, the ratio of the similarity corresponding to the externally generated model to the sum of the similarities, and uses the calculation result of the ratio as a weight to determine the reliability of the label of interest. The weighted sum may be calculated, and the calculation result may be used as the integrated result of the reliability of the label of interest. The integration unit 114 performs the same operation for other labels. Then, the integration unit 114 specifies the label with the highest integration result as the label of the image.

FIG. 11 is an explanatory diagram showing a specific example of the second calculation method. It is assumed that there are two externally generated models A and B. The reliability of “car”, “motorcycle”, “bus”, and “background” derived using the externally generated model A is “0.1”, “0.7”, “0.1”, “ 0.1 ". The reliability of “car”, “motorcycle”, “bus” and “background” derived using the externally generated model B is “0.1”, “0.6”, “0.2”, “ 0.1 ". It is assumed that the similarity calculated for the externally generated model A is “0.9” and the similarity calculated for the externally generated model B is “0.8”. In this case, the sum of the similarities is 0.9 + 0.8 = 1.7. Therefore, the ratio of the similarity “0.9” corresponding to the externally generated model A to the total similarity “1.7” is “0.9 / 1.7”. The ratio of the similarity “0.8” corresponding to the externally generated model B to the total similarity “1.7” is “0.8 / 1.7”. The integrating unit 114 calculates a weighted sum of reliability for each label, using “0.9 / 1.7” and “0.8 / 1.7” as weights, and uses the calculation result as the reliability of the label. And the integration result. Then, the integration results of “car”, “motorcycle”, “bus”, and “background” are “0.0999”, “0.6528”, “0.1470”, and “0.0999”. Therefore, the integration unit 114 specifies the “motorcycle” having the highest integration result as the image label.

Each of the first and second calculation methods is a calculation in which the reliability of the label derived for each externally generated model is weighted by the similarity corresponding to the externally generated model and integrated. .

When the integrating unit 114 specifies the label of the image based on the integration result of the reliability of each label, the learning unit 103 extracts a fixed rectangular area in the image, and integrates the rectangular area image with the image of the rectangular area. The pair with the label specified by the unit 114 is included in the existing teacher data. Then, the learning unit 103 uses the teacher data to re-learn the internally generated model by deep learning. Further, the learning unit 103 updates the existing internally generated model stored in the first model storage unit 104 to a new internally generated model generated by re-learning.

Next, the model transmitting unit 122 will be described. The model transmitting unit 122 transmits the model learned by the learning unit 103 to a predetermined second identification system. The number of the second identification system to which the model is transmitted may be one or plural. Note that the second identification system stores the received model as an externally generated model.

As a mode in which the model transmission unit 122 transmits a model to the second identification system, four model transmission modes will be described as examples.

[First Model Transmission Mode]
In the first model transmission mode, when the learning unit 103 re-learns a model (internally generated model), the model transmission unit 122 transmits the model to the second identification system. In the first model transmission mode, each time the model is re-learned by the learning unit 103, the model transmission unit 122 transmits a model newly obtained by the re-learning to the second identification system. Therefore, in the first model transmission mode, the model transmission unit 122 can transmit the latest model to the second identification system.

[Second Model Transmission Mode]
In the second model transmission mode, the model transmission unit 122 periodically transmits the model stored in the first model storage unit 104 to the second identification system. That is, the model transmission unit 122 stores the model stored in the first model storage unit 104 in the first model storage unit 104 again after a certain period of time has elapsed after transmitting the model to the second identification system. And transmitting the model to the second identification system. Even if the model stored in the first model storage unit 104 has been updated a plurality of times during this fixed period, the model transmission unit 122 will continue to operate the first model storage at the time when the fixed period has elapsed since the previous model transmission. The model stored in the storage unit 104 is transmitted to the second identification system. If the model stored in the first model storage unit 104 has not been updated during this fixed period, the model transmission unit 122 retransmits the same model as the previously transmitted model. Become.

[Third Model Transmission Mode]
In the third model transmission mode, when the learning unit 103 re-learns a model (internally generated model), the model transmission unit 122 determines whether to transmit the model to the second identification system.

The image to be specified by the integrating unit 114 as a label is determined by the determining unit 107 of the images identified by the first identifying unit 106 by the first determining method, the second determining method, or the second determining method. It is an image determined to cause the second identification unit 111 to execute the identification process by the determination method of No. 3. That is, the image that the integrating unit 114 specifies as a label is an image that has been erroneously identified by the first identification unit 106 or has not obtained a reliability higher than the threshold. Before the learning unit 103 re-learns the model, the integration unit 114 specifies the label of such an image by using the reliability of each label obtained for each externally generated model. In the third model transmission mode, when the learning unit 103 re-learns a model (internally generated model), the first identification unit 106 applies the image to the re-learned model, thereby obtaining the image. Is derived again. The model transmitting unit 122 uses the model re-learned by the first identifying unit 106 to again derive the identification result (label) of the image and the label specified by the integrating unit 114. Then, it is determined that the relearned model is to be transmitted to the second identification system, and the relearned model is transmitted to the second identification system. On the other hand, when the two labels do not match, the model transmission unit 122 determines that the relearned model is not transmitted to the second identification system, and does not transmit the model to the second identification system.

Each time the learning unit 103 re-learns the model, the model transmission unit 122 determines whether to transmit the model to the second identification system as described above. To the second identification system.

The integration unit 114 specifies the label of the image by using the reliability of each label obtained for each externally generated model. Therefore, it is considered that the accuracy of the label specified by the integration unit 114 is high even if the image has been erroneously identified by the first identification unit 106 or has not obtained a reliability higher than the threshold. Therefore, the fact that the identification result (label) of the image derived again by using the model re-learned by the first identification unit 106 and the label specified by the integration unit 114 coincides with each other means that It can be said that the identification accuracy of the model obtained by learning is higher than the identification accuracy of the model before relearning. Also, it can be said that a correct classification result was not obtained in the model before the re-learning, but a correct classification result was obtained in the model obtained by the re-learning. Therefore, in the third model transmission mode, when the identification accuracy of the model obtained by the relearning is higher than the identification accuracy of the model before the relearning, the model obtained by the relearning is transmitted to the second identification system. It can be said that the transmission mode is used. Further, in the third model transmission mode, if the model before re-learning cannot obtain a correct identification result, but the model obtained by re-learning can obtain a correct identification result, It can also be said that the model obtained by learning is transmitted to the second identification system.

[Fourth Model Transmission Mode]
Also in the fourth model transmission mode, when the learning unit 103 re-learns a model (internally generated model), the model transmission unit 122 determines whether the model transmission unit 122 transmits the model to the second identification system. Is determined.

In the fourth model transmission mode, when the model is relearned, the model transmission is performed within a predetermined time based on the accuracy rate of the identification result (label) derived by the first identification unit 106 using the model. The unit 122 determines whether to send the model to the second identification system. When the accuracy rate is equal to or greater than a predetermined threshold, the model transmitting unit 122 determines that the relearned model is to be transmitted to the second identification system, and transmits the model to the second identification system. On the other hand, when the accuracy rate is less than the threshold, the model transmitting unit 122 determines that the relearned model is not transmitted to the second identification system, and does not transmit the model to the second identification system.

An example of calculating the accuracy rate of the identification result (label) within a predetermined time will be described. Here, a case will be described as an example where the determination unit 107 determines an image for which the second identification unit 111 performs the identification process by the above-described “first determination method”. When the first identification unit 106 determines a label for an image, the determination unit 107 displays a screen illustrated in FIG. 5 on the display device 115. Therefore, it can be said that the number of times the determination unit 107 has displayed the screen illustrated in FIG. 5 is the number of times the first identification unit 106 has identified an image. Then, on the displayed screen (see FIG. 5), the number of times the first button 304 is clicked can be said to be the number of times that the label derived by the first identification unit 106 is correct. When the model is re-learned, the model transmission unit 122 calculates the ratio of the number of times the first button 304 is clicked to the number of times the determination unit 107 displays the screen illustrated in FIG. 5 within a predetermined time, The result may be the correct answer rate. Then, the model transmission unit 122 may determine whether to transmit the re-learned model to the second identification system as described above by comparing the accuracy rate with the threshold.

In the present embodiment, the model updating unit 121 and the model transmitting unit 122 are realized by, for example, a CPU (Central Processing Unit) of the computer 102 operating according to the model providing program and a communication interface of the computer 102. For example, the CPU may read the model providing program from a program recording medium such as a program storage device of the computer 102 and operate as the model updating unit 121 and the model transmitting unit 122 using the communication interface according to the model providing program.

In addition, the learning unit 103, the first identification unit 106, the determination unit 107, the area correction GUI display control unit 108, the area extraction unit 109, the second identification unit 111, the display control unit 112, and the integration unit 114 This is realized by the CPU of the computer 102 that operates according to the provided program. For example, the CPU reads the model providing program from a program recording medium such as a program storage device of the computer 102, and according to the model providing program, the learning unit 103, the first identification unit 106, the determination unit 107, the area correction GUI display control unit 108 , The region extraction unit 109, the second identification unit 111, the display control unit 112, and the integration unit 114.

The first model storage unit 104, the second model storage unit 110, the attribute data storage unit 113, and the result storage unit 117 are realized by a storage device included in the computer 102.

Next, the process of the embodiment of the present invention will be described. FIG. 12 is a flowchart illustrating an example of processing progress from the time when the camera 101 captures an image to the time when the second identification unit 111 performs an identification process on an image. The detailed description of the operation already described is omitted.

Note that it is assumed that the learning unit 103 has previously learned the internally generated model by deep learning, and has stored the internally generated model in the first model storage unit 104.

The model updating unit 121 receives a model from each of a plurality of first identification systems predetermined for the identification system 100 shown in FIG. 3, and stores the model as an externally generated model in a second model storage. It is assumed that the information is stored in the unit 110. That is, it is assumed that the second model storage unit 110 stores the models learned by the individual first identification systems as externally generated models. Note that, when a new model is received from the first identification system, the model updating unit 121 stores the model already stored in the second model storage unit 110 as a model learned by the first identification system. May be updated to the newly received model.

First, the camera 101 obtains an image by photographing at the installation location of the camera 101 (step S1). The camera 101 transmits the image to the computer 102.

The first identification unit 106 of the computer 102 receives the image via the data acquisition unit 105. Then, the first identification unit 106 identifies the object shown in the image by applying the image to the internally generated model (Step S2). In step S2, the first identification unit 106 derives a label representing the object appearing in the image and the reliability of the label. The first identification unit 106 stores the image in the result storage unit 117 in association with the derived label and reliability. When the specified label is not “background”, the first identification unit 106 determines a rectangular area surrounding the object shown in the image.

Next, the determination unit 107 determines whether or not to cause the second identification unit 111 to execute an identification process on the image identified by the first identification unit 106 in step S2 (step S3). When it is determined that the second identification unit 111 does not execute the identification processing (No in step S3), the processing after step S1 is repeated.

(4) When it is determined that the second identification unit 111 performs the identification process (Yes in step S3), the area correction GUI display control unit 108 displays an image on the display device 115. For example, the area correction GUI display control unit 108 displays the screens illustrated in FIGS. 7 and 8 on the display device 115. Then, the region extracting unit 109 determines a rectangular region surrounding the object shown in the image in accordance with the operation of the operator on the screen, and extracts the rectangular region from the image (step S4).

Next, the second identification unit 111 identifies an object appearing in the image of the rectangular area extracted in step S4 for each of the externally generated models stored in the second model storage unit 110 in advance. (Step S5). The second identification unit 111 derives the reliability of each label (“car”, “motorcycle”, “bus”, and “background”) for each externally generated model. Then, the result storage unit 117 stores the reliability of each label derived for each externally generated model. In addition, the second identification unit 111 also causes the result storage unit 117 to store, for each externally generated model, a pair of the label having the highest reliability and the reliability corresponding to the label. The label with the highest reliability indicates an object determined to be present in the image.

(4) After step S5, the processes after step S1 are repeated.

FIG. 13 is a flowchart showing an example of processing progress when a model (internally generated model) is re-learned and the model is transmitted to the second identification system. In the following description, a detailed description of the operation already described is omitted. In FIG. 13, the above-described “first model transmission mode” will be described as an example.

The display control unit 112 determines the label derived by the first identification unit 106 and the reliability corresponding to the label, and the label derived by the second identification unit 111 for each externally generated model and the reliability corresponding to each label. A screen in which the degree is superimposed on the image is displayed on the display device 115 (step S11). At this time, the display control unit 112 includes a check box 504, a re-learning button 505, and

screen switching buttons

506 and 507 in this screen. In step S11, the display control unit 112 displays, for example, a screen illustrated in FIG.

The operator checks the screen illustrated in FIG. 9 and determines whether or not to include the displayed image 301 (more specifically, the image of the rectangular area 309 determined in the image 301) in the teacher data. . By checking the check box 504, the operator specifies that the displayed image 301 is to be included in the teacher data. That is, the image displayed on the screen with the check box 504 checked is the image specified as the image to be included in the teacher data. After specifying the image to be included in the teacher data, the operator clicks the re-learning button 505.

When the operator clicks the re-learning button 505, the integration unit 114 calculates the similarity between the reference attribute data and each attribute data corresponding to each externally generated model (step S12). As described above, the attribute data is represented by a vector. When calculating the similarity between two pieces of attribute data (vectors), the integration unit 114 may calculate the reciprocal of the distance between the two vectors as the similarity.

Next, the integration unit 114 integrates the reliability of the labels derived for each of the externally generated models using the similarities calculated in step S12. The integrating unit 114 performs this process for each label, and identifies the label with the highest reliability integration result as the label for the image to be included in the teacher data (step S13).

(4) When a plurality of images to be included in the teacher data are specified by the operator, the integrating unit 114 executes the process of step S13 for each of the images.

Next, the learning unit 103 extracts a determined rectangular area in the image to be included in the teacher data, and includes a set of the image of the rectangular area and the label specified by the integration unit 114 in the existing data. Then, using the teacher data, the learning unit 103 re-learns the internally generated model by deep learning, and stores the internally generated model obtained by the re-learning in the first model storage unit 104 (step S14). The learning unit 103 updates the existing internally generated model stored in the first model storage unit 104 to a new internally generated model generated by re-learning.

Thereafter, when the first identification unit 106 identifies an object appearing in the image, the first identification unit 106 uses a new internally generated model generated by relearning.

(4) After step S14, the model transmitting unit 122 transmits the model (internally generated model) relearned in step S14 to the second identification system (step S15). Each identification system 100 has a similar configuration. Upon receiving the model transmitted in step S15, the model updating unit 121 of the second identification system updates the model stored in the second model storage unit 110 in the second identification system to the received model. I do. Therefore, the model transmitted by the model transmitting unit 122 to the second identification system in step S15 is stored as an externally generated model in the second identification system.

In FIG. 13, the “first model transmission mode” has been described as an example. When the model transmitting unit 122 transmits the model to the second identification system in the “second model transmitting mode” described above, the computer 102 may end the process in step S14. Then, the model transmission unit 122 may periodically transmit the model stored in the first model storage unit 104 to the second identification system, separately from the processing in steps S11 to S14.

FIG. 14 is a flowchart showing an example of the processing progress of the above-mentioned “third model transmission mode”. The processing up to step S14 is the same as that in the flowchart shown in FIG. 13, and a description thereof will be omitted.

After step S14, the first identification unit 106 uses the label of the image (see FIG. 9) displayed in the screen with the check box 504 checked, using the internally generated model relearned in step S14. (Step S21). The image displayed on the screen with the check box 504 checked is an image specified by the operator to be included in the teacher data.

Next, the model transmission unit 122 determines whether or not the label derived in step S21 matches the label identified by the integration unit 114 in step S13 (see FIG. 13) for the same image (step S21). S22).

If the two labels match (Yes in step S22), the model transmitting unit 122 determines that the model (internally generated model) retrained in step S14 (see FIG. 13) is to be transmitted to the second identification system, The model is transmitted to the second identification system (Step S23).

If the two labels do not match (No in step S22), the model transmitting unit 122 determines that the model re-learned in step S14 is not to be transmitted to the second identification system, and ends the process.

FIG. 15 is a flowchart showing an example of the processing progress of the “fourth model transmission mode” described above. The processing up to step S14 is the same as that in the flowchart shown in FIG. 13, and a description thereof will be omitted.

の後 After step S14, the model transmission unit 122 calculates the correct answer rate of the image identification result when the re-learned model (internally generated model) is used within a predetermined time (step S31). Since the calculation example of the correct answer rate has already been described, the description is omitted here.

Next, the model transmitting unit 122 determines whether or not the correct answer rate calculated in step S31 is equal to or greater than a predetermined threshold (step S32).

When the correct answer rate is equal to or higher than the threshold (Yes in step S32), the model transmitting unit 122 determines that the model (internally generated model) retrained in step S14 (see FIG. 13) is to be transmitted to the second identification system. , And transmits the model to the second identification system (step S33).

If the accuracy rate is less than the threshold (No in step S32), the model transmitting unit 122 determines that the model re-learned in step S14 is not transmitted to the second identification system, and ends the process.

According to the present embodiment, the determination unit 107 determines that the first identification unit 106 performs the identification process by at least one of the first determination method, the second determination method, and the third determination method described above. It is determined whether or not to cause the second identification unit 111 to execute the identification processing on the target image. Therefore, the image for which the identification processing is performed by the second identification unit 111 is an image in which the label determined by the first identification unit 106 is incorrect, and the reliability corresponding to the label determined for the image is a threshold. The following image or an image in which an object (“automobile”, “motorcycle” or “bus”) is captured despite the label determined by the first identification unit 106 being “background” is there. In the present embodiment, such an image is generated by using a model different from the internally generated model (more specifically, a model generated by a first identification system predetermined for the identification system 100 of interest. (Externally generated model)), a pair of a label specified based on the result of applying the image and the image is added to the existing teacher data, and the learning unit 103 re-learns the internally generated model. Therefore, the identification accuracy of the internally generated model can be improved.

Then, the model transmitting unit 122 transmits the model learned by the learning unit 103 to, for example, the first model transmitting mode, the second model transmitting mode, the third model transmitting mode, or the fourth model transmitting mode. In a mode or the like, the data is transmitted to a predetermined second identification system. In the second identification system, the model updating unit 121 stores the received model in the second model storage unit 110 as an externally generated model. Therefore, each identification system 100 can relearn its own model so as to improve the identification accuracy of its own model, and can contribute to the improvement of the identification accuracy of the models of other identification systems 100. it can.

Next, various modifications of the embodiment of the present invention will be described.

In the above embodiment, in response to an operator's operation on the screens illustrated in FIGS. 7 and 8, the region extracting unit 109 determines a rectangular region surrounding the object shown in the image, and extracts the rectangular region from the image. I do. Then, the second identification unit 111 identifies an object appearing in the extracted image of the rectangular area for each externally generated model. The second identification unit 111 performs a process of identifying an object appearing in the image not on the extracted image of the rectangular area but on the entire one image that has been processed by the first identification unit 106. You may. In this case, the identification system 100 (see FIG. 3) does not have to include the area correction GUI display control unit 108 and the area extraction unit 109. And the identification system 100 does not need to perform step S4 (see FIG. 12). The second identification unit 111 only needs to identify an object in the image in step 5 with respect to one entire image processed by the first identification unit 106.

{Circle around (1)} The learning unit 103 only needs to include the entire set of one image and the label specified by the integration unit 114 in existing teacher data, and re-learn the internally generated model using the teacher data.

The learning unit 103 may re-learn an internally generated model by deep learning using a set of an image and a label specified by the integration unit 114 and an existing internally generated model as teacher data.

{Circle around (2)} The second model storage unit 110 may store one externally generated model. In this case, the learning unit 103 includes the pair of the image and the label with the highest reliability derived by the second identification unit 111 in the existing teacher data, and uses the teacher data to generate the internally generated model. Should be re-learned.

FIG. 16 is a schematic block diagram illustrating a configuration example of a computer 102 included in an identification system 100 according to an embodiment of the present invention or a modified example thereof. In FIG. 16, a computer is represented by a reference numeral “1000”. The computer 1000 includes an interface between a CPU 1001, a main storage device 1002, an auxiliary storage device 1003, an interface 1004, a display device 1005, an input device 1006, a communication interface 1007, and a data collection unit 101 (for example, a camera). 1008.

The operation of the computer included in the identification system 100 is stored in the auxiliary storage device 1003 in the form of a model providing program. The CPU 1001 reads the model providing program from the auxiliary storage device 1003 and expands the program on the main storage device 1002. Then, the CPU 1001 executes the processing of the computer 102 (see FIG. 3) in the above-described embodiment and its modifications according to the model providing program.

The auxiliary storage device 1003 is an example of a non-transitory tangible medium. Other examples of non-transitory tangible media include a magnetic disk, a magneto-optical disk, a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory) connected via the interface 1004, A semiconductor memory and the like are included. When the program is distributed to the computer 1000 via a communication line, the computer 1000 that has received the program may load the program into the main storage device 1002 and execute the above processing.

The program may be for realizing a part of the processing of the computer 102 shown in the embodiment and its modified example. Furthermore, the program may be a difference program that implements the above-described processing in combination with another program already stored in the auxiliary storage device 1003.

{Some or all of the components may be realized by a general-purpose or dedicated circuit (processor), a processor, or a combination thereof. These may be configured by a single chip, or may be configured by a plurality of chips connected via a bus. Some or all of the components may be realized by a combination of the above-described circuit and the like and a program.

When a part or all of each component is realized by a plurality of information processing devices, circuits, and the like, the plurality of information processing devices, circuits, and the like may be centrally arranged or may be distributed. For example, the information processing device, the circuit, and the like may be realized as a form in which each is connected via a communication network, such as a client and server system or a cloud computing system.

Next, the outline of the present invention will be described. FIG. 17 is a block diagram showing an outline of the identification system of the present invention. The identification system according to the present invention includes a learning unit 701, a first model storage unit 702, a first identification unit 703, a second model storage unit 704, a model update unit 705, and a second identification unit 706. And a model transmitting unit 707.

The learning unit 701 (for example, the learning unit 103) learns a model for identifying an object represented by data (for example, an image) using teacher data.

The first model storage unit 702 (first model storage unit 104) stores the model learned by the learning unit 701.

The first identification unit 703 (for example, the first identification unit 106) identifies an object represented by data using the model learned by the learning unit 701.

The second model storage unit 704 (for example, the second model storage unit 110) stores the individual models learned by the plurality of first identification systems determined in advance.

When the model updating unit 705 (for example, the model updating unit 121) receives a model learned by the first identification system from the first identification system, the model updating unit 705 stores the model stored in the second model storage unit 704. The model learned by the first identification system is updated to the model received from the first identification system.

In a predetermined case, the second identification unit 706 (for example, the second identification unit 111) determines whether the first identification unit 703 is an identification target for each model stored in the second model storage unit 704. The object represented by the set data is identified.

The learning means 701 re-learns the model using the label for the data determined based on the identification result derived by the second identification means 706 and the teacher data including the data, and stores the model in the first model storage means 702. Update the stored model with the retrained model.

The model transmitting unit 707 (for example, the model transmitting unit 122) transmits the model learned by the learning unit 701 to one or a plurality of predetermined second identification systems.

With such a configuration, the identification system of the present invention can re-learn its own model so as to improve the identification accuracy of its own model, and also contributes to the improvement of the identification accuracy of the models of other identification systems. can do.

The model transmitting means 707 may be configured to transmit the model to the second identification system when the learning means 701 re-learns the model.

The model transmitting unit 707 may periodically transmit the model stored in the first model storing unit 702 to the second identification system.

The second identification unit 706 integrates the identification results derived for each model stored in the second model storage unit 704 to identify the label for the data to be identified by the first identification unit 703. The learning unit 701 re-learns the model using the teacher data including the label specified by the integrating unit and the data, and the model transmitting unit 707 performs When the identification result of the data derived by the first identifying means 703 using the relearned model matches the label for the data identified by the integrating means, the relearned model is assigned to the second learning means. It may be configured to transmit to the identification system.

When the learning unit 701 re-learns the model, the model transmission unit 707 determines that the accuracy rate of the identification result derived by the first identification unit 703 using the model within a predetermined time period is equal to or greater than a predetermined threshold value. , The model may be transmitted to the second identification system.

Although the present invention has been described with reference to the exemplary embodiments, the present invention is not limited to the above exemplary embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

Industrial applicability

The present invention is suitably applied to an identification system that identifies an object represented by data by applying the data to a model.

REFERENCE SIGNS LIST 100 identification system 101 data collection unit 102 computer 103 learning unit 104 first model storage unit 105 data acquisition unit 106 first identification unit 107 determination unit 108 region correction GUI display control unit 109 region extraction unit 110 second model storage unit 111 second identification unit 112 display control unit 113 attribute data storage unit 114 integration unit 115 display device 116 mouse 117 result storage unit 121 model update unit 122 model transmission unit

Claims

Learning means for learning a model for identifying an object represented by the data using the teacher data;
First model storage means for storing the model learned by the learning means;
A first identification unit that identifies an object represented by data using the model learned by the learning unit;
Second model storage means for storing individual models learned by a plurality of predetermined first identification systems, respectively;
When a model learned by the first identification system is received from the first identification system, the model learned by the first identification system stored in the second model storage means is received. Model updating means for updating to a model received from the first identification system;
A second identification unit that identifies an object represented by the data that has been identified by the first identification unit for each individual model stored in the second model storage unit in a predetermined case. ,
The learning means,
Using a label for the data determined based on the identification result derived by the second identification unit and teacher data including the data, a model is re-learned and stored in the first model storage unit. Update the model to the retrained model,
An identification system comprising: a model transmission unit that transmits a model learned by the learning unit to one or more predetermined second identification systems.
The model transmission means
The identification system according to claim 1, wherein when the learning unit re-learns the model, the learning unit transmits the model to the second identification system.
The model transmission means
The identification system according to claim 1, wherein the model stored in the first model storage unit is periodically transmitted to the second identification system.
An integrating means for specifying a label for data identified by the first identifying means by integrating the identification results derived for each model stored in the second model storing means by the second identifying means; With
The learning means is
Using the label specified by the integrating means and teacher data including the data, re-learning the model,
The model transmission means
When the identification result of the data derived using the model re-learned by the first identification unit matches the label for the data specified by the integration unit, the re-learned model is The identification system according to claim 1, wherein the identification system is transmitted to a second identification system.
The model transmission means
When the learning unit re-learns the model, if the accuracy rate of the identification result derived by the first identification unit using the model within a predetermined time is equal to or greater than a predetermined threshold, the model is The identification system according to claim 1, wherein the identification system is transmitted to a second identification system.
Learning a model for identifying an object represented by the data using the teacher data, storing the model in first model storage means,
Executing a first identification process for identifying an object represented by data using the model stored in the first model storage means;
Individual models learned by a plurality of predetermined first identification systems are stored in second model storage means, respectively.
When a model learned by the first identification system is received from the first identification system, the model learned by the first identification system stored in the second model storage means is received. Update to the model received from the first identification system,
In a predetermined case, for each individual model stored in the second model storage means, a second identification process for identifying an object represented by the data to be identified in the first identification process is executed. ,
Using a label for the data determined based on the identification result derived in the second identification process and teacher data including the data, the model is re-learned and stored in the first model storage means. Update the model to the retrained model,
A model providing method, comprising: transmitting a model stored in the first model storage unit to one or more predetermined second identification systems.
On the computer,
A learning process of learning a model for identifying an object represented by the data using teacher data, and storing the model in a first model storage unit;
A first identification process for identifying an object represented by data using the model stored in the first model storage means;
A process of storing the individual models learned by the predetermined plurality of first identification systems in the second model storage means,
When a model learned by the first identification system is received from the first identification system, the model learned by the first identification system stored in the second model storage means is received. A model update process for updating the model received from the first identification system,
In a predetermined case, for each model stored in the second model storage means, a second identification process for identifying an object represented by the data set as an identification target in the first identification process;
Using a label for the data determined based on the identification result derived in the second identification process and teacher data including the data, a model is re-learned and stored in the first model storage unit. A re-learning process that updates the model to the re-learned model, and
A model providing program for executing a model transmission process of transmitting a model stored in the first model storage unit to one or more predetermined second identification systems.