JP6981553B2 - Identification system, model provision method and model provision program - Google Patents

Description

The present invention relates to an identification system that identifies an object represented by the data by applying the data to a model, and a model providing method and a model providing program that the identification system provides a model to another identification system.

An example of a general identification system will be described below. In a general identification system, a model is learned in advance by machine learning using a set of an image obtained by a camera equipped with the identification system and a label representing an object in the image as teacher data. .. Then, the general identification system identifies an object in the image by applying the image newly obtained by the camera to the model.

Such a general identification system is used for the purpose of detecting a suspicious vehicle or a suspicious person and preventing crimes, or detecting a user of a white cane or a wheelchair, and using a white cane or a wheelchair. It is used for the purpose of support such as guiding people.

Here, an identification system for identifying an object in an image has been described as an example, but as a general identification system, an identification system for identifying an object represented by voice data can also be considered. Hereinafter, an identification system for identifying an object in an image will be described as an example.

In addition, Patent Document 1 describes an image recognition method for avoiding a long period of additional learning due to a difference in the imaging environment. The image recognition method described in Patent Document 1 is an image recognition method in a camera system including a plurality of camera devices. Then, in the image recognition method described in Patent Document 1, the first image and the first imaging environment information are acquired from the first camera device. Then, using a parameter table that manages the imaging environment information indicating each imaging environment when each camera device has captured an image in the past and each recognition control parameter indicating each detector function corresponding to each imaging environment. , The first recognition control parameter indicating the first detector function corresponding to the same or similar imaging environment as the first imaging environment shown in the first imaging environment information is selected. Then, the first image acquired from the first camera device is recognized by using the first detector function indicated by the first recognition control parameter.

Further, Patent Document 2 describes an image monitoring device. The image monitoring device described in Patent Document 2 normalizes the face region using the detected facial feature points and collates it with a person's dictionary.

Japanese Unexamined Patent Publication No. 2016-15116 Japanese Unexamined Patent Publication No. 2007-300185

It is conceivable that a plurality of the above-mentioned general identification systems will be installed and cameras of each identification system will be installed in various places.

Here, there may be a bias in the way an object appears in an image obtained by shooting with one camera. For example, suppose that one camera has many opportunities to shoot a car traveling from the right side to the left side when viewed from the camera, but has few opportunities to shoot a car traveling in the opposite direction. In this case, many images of a car traveling from the right side to the left side can be obtained, but only a few images of a car traveling in the opposite direction can be obtained. Then, the teacher data includes many images of a car traveling from the right side to the left side, and only a few images of a car traveling in the opposite direction. As a result, when an image of a car traveling from right to left is applied to a model obtained by machine learning using teacher data, the identification system identifies the car with high accuracy, but vice versa. When an image of a car traveling in a direction is applied to the model, the identification accuracy of the car is low.

It is preferable that the identification accuracy of the model for identifying the object represented by the data is improved in each identification system arranged in each place.

Therefore, the present invention is an identification system that can relearn its own model so as to improve the identification accuracy of its own model and can contribute to the improvement of the identification accuracy of the models of other identification systems. The purpose is to provide a model provision method and a model provision program.

The identification system according to the present invention has a learning means for learning a model for identifying an object represented by data by using teacher data, a first model storage means for storing a model learned by the learning means, and a learning means. A first discriminating means for identifying the object represented by the data and a second model storage for storing individual models trained by a plurality of predetermined first discriminating systems using the model trained by. When the means and the model learned by the first identification system are received from the first identification system, the model learned by the first identification system stored in the second model storage means is displayed. The model update means for updating to the model received from the first identification system and, in a predetermined case, the first identification means for each individual model stored in the second model storage means. A second discriminating means for identifying the object represented by the data is provided, and the learning means uses the teacher data including the label for the data determined based on the discriminating result derived by the second discriminating means and the data. The model is retrained, the model stored in the first model storage means is updated to the retrained model, and the model learned by the learning means is replaced with one or more predetermined second identification systems. It is characterized by providing a model transmission means for transmitting to.

Further, in the model providing method according to the present invention, a model for identifying an object represented by the data is learned by using the teacher data, the model is stored in the first model storage means, and the model is stored in the first model storage means. Using the model, the first identification process for identifying the object represented by the data is executed, and the individual models trained by the plurality of predetermined first identification systems are stored in the second model storage means. When a model learned by the first identification system is received from the first identification system, the model learned by the first identification system stored in the second model storage means is stored in the second model storage means. , Update to the model received from the first identification system, and in a predetermined case, the data to be identified in the first identification process is represented for each individual model stored in the second model storage means. A second identification process for identifying an object is executed, and the model is retrained using the teacher data including the label for the data determined based on the identification result derived in the second identification process and the data, and the second. The model stored in the model storage means is updated to the relearned model, and the model stored in the first model storage means is transmitted to one or a plurality of predetermined second identification systems. It is characterized by doing.

Further, the model providing program according to the present invention is a learning process of learning a model for identifying an object represented by data by a computer using teacher data and storing the model in a first model storage means. Using the model stored in the model storage means, the first identification process for identifying the object represented by the data, and the individual models learned by the plurality of predetermined first identification systems are each used as the second model. Processing to be stored in the storage means, when a model learned by the first identification system is received from the first identification system, it is learned by the first identification system stored in the second model storage means. A model update process for updating a model to a model received from the first identification system, and, in a predetermined case, identification by the first identification process for each individual model stored in the second model storage means. The model is retrained using the second identification process for identifying the object represented by the target data, the label for the data determined based on the identification result derived in the second identification process, and the teacher data including the data. Then, the re-learning process for updating the model stored in the first model storage means to the re-learned model, and one or more predetermined models stored in the first model storage means. It is characterized in that the model transmission process to be transmitted to the second identification system of the above is executed.

According to the present invention, it is possible to relearn its own model so as to improve the identification accuracy of its own model, and it is possible to contribute to the improvement of the identification accuracy of the model of another identification system.

It is a schematic diagram which shows the situation which a plurality of identification systems of this invention are provided. It is explanatory drawing which shows the example of the 1st identification system and the 2nd identification system. It is a block diagram which shows the structural example of the identification system 100 of embodiment of this invention. It is a schematic diagram which shows the example of an internally generated model and an externally generated model. It is a schematic diagram which shows the example of the screen which the determination part displays on the display apparatus in the 1st determination method. It is a schematic diagram which shows the example of the screen which the determination part displays on the display apparatus in the 3rd determination method. It is a schematic diagram which shows an example of the screen which the area correction GUI display control unit displays on a display device. It is a schematic diagram which shows the other example of the screen which the area correction GUI display control unit displays on the display apparatus. It is a schematic diagram which shows the example of the screen displayed by the display control unit. It is explanatory drawing which shows the specific example of the 1st calculation method. It is explanatory drawing which shows the specific example of the 2nd calculation method. It is a flowchart which shows the example of the processing progress from the camera taking a picture to the second identification part performing the identification process for an image. It is a flowchart which shows the example of the processing progress at the time of re-learning a model, and transmitting the model to a second identification system. It is a flowchart which shows the example of the processing progress of the 3rd model transmission mode. It is a flowchart which shows the example of the processing progress of the 4th model transmission mode. It is a schematic block diagram which shows the configuration example of the computer provided in the identification system in the embodiment of this invention and the variation | embodiment of this invention. It is a block diagram which shows the outline of the identification system of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a schematic diagram showing a situation in which a plurality of identification systems of the present invention are provided. FIG. 1 illustrates a case where six identification systems 100 are provided in various places, but the number of identification systems 100 provided in each place is not particularly limited. In this embodiment, it is assumed that the plurality of identification systems 100 have the same configuration.

Each identification system 100 can communicate, for example, via a communication network.

Each of the individual identification systems 100 includes a data collection unit (data collection unit 101 shown in FIG. 3 described later). The data collection unit of each identification system 100 (not shown in FIG. 1, see FIG. 3 described later) is installed in each place where data is collected. The data collection unit collects data at the location where the data collection unit is installed. For example, the data collection unit collects image and audio data at the installation location. The data collection unit is realized by a camera or a microphone. For example, the data collection unit may collect images by photographing the monitoring location. Further, for example, audio data may be collected by recording at the installation location.

The individual identification system 100 includes a computer separate from the data collection unit, and the computer identifies an object represented by data (image, audio data, etc.).

Further, the individual identification system 100 learns the model using the data collected by the data collection unit as teacher data. This model is a model for identifying the object represented by the data.

Then, each identification system 100 provides a model to each other, and relearns its own model by using the model provided by another identification system 100.

Here, when focusing on one identification system 100, another identification system 100 that transmits a model to the identification system 100 of interest is predetermined. Another identification system 100 that transmits a model to the identification system 100 of interest is referred to as a first identification system. In this embodiment, it is assumed that a plurality of first identification systems are defined for the identification system 100 of interest. It can be said that the identification system 100 of interest receives the model provided by the first identification system.

Further, another identification system 100 as a transmission destination when the identification system 100 of interest transmits a model is predetermined. Another identification system 100, which is a transmission destination when the identification system 100 of interest transmits a model, is referred to as a second identification system. In the present embodiment, it is assumed that one or a plurality of second identification systems are defined for the identification system 100 of interest. It can be said that the identification system 100 of interest provides a model for the second identification system.

The first identification system and the second identification system are predetermined for each individual identification system 100. For example, the administrator who manages each identification system 100 may predetermine a first identification system and a second identification system for each identification system 100.

Each identification system 100 has the same configuration, but the first identification system and the second identification system are individually defined for each identification system 100.

FIG. 2 is an explanatory diagram showing an example of a first identification system and a second identification system. Here, attention will be paid to the identification system 100a shown in FIG. Also, in FIG. 2, the arrows indicate the direction in which the model is sent. In the example shown in FIG. 2, regarding the identification system 100a of interest, the identification systems 100b and 100c are defined as the first identification system, and the identification systems 100b, 100c and 100d are defined as the second identification system. The first identification system and the second identification system are individually defined for the identification systems 100b, 100c, and 100d, respectively.

FIG. 3 is a block diagram showing a configuration example of the identification system 100 according to the embodiment of the present invention. The identification system 100 includes a data collection unit 101 and a computer 102. The data collection unit 101 and the computer 102 are connected so as to be able to communicate with each other by wire or wirelessly. In the following description, a case where the data collection unit 101 is a camera will be described as an example, and the data collection unit 101 will be referred to as a camera 101. The camera 101 shoots from the installation location of the camera 101. The installation location of the camera 101 and the installation location of the computer 102 may be different.

The computer 102 includes a learning unit 103, a first model storage unit 104, a data acquisition unit 105, a first identification unit 106, a determination unit 107, and an area correction GUI (Graphical User Interface) display control unit 108. , Area extraction unit 109, second model storage unit 110, model update unit 121, second identification unit 111, display control unit 112, attribute data storage unit 113, integration unit 114, and display device. It includes 115, a mouse 116, a result storage unit 117, and a model transmission unit 122.

The learning unit 103 learns the model by machine learning using the image obtained by the camera 101 as teacher data. Hereinafter, a case where the learning unit 103 learns a model by deep learning will be described as an example. The teacher data is, for example, a set of a set of an image obtained by shooting by the camera 101 and a label indicating an object shown in the image. The label may be determined by the operator of the identification system 100. In learning, a model is trained (generated) using a set of such sets as teacher data.

Further, when a predetermined image and a label for the image are specified, the learning unit 103 adds the set of the image and the label to the teacher data, and relearns the model by deep learning. The predetermined image is an image determined by the determination unit 107, which will be described later, to cause the second identification unit 111 to execute the identification process. In the present embodiment, an example is described in which a region in which an object is captured is extracted from the image determined in this way, and a pair of an image of the extracted region and a label is added to the teacher data. do.

The model generated by the learning unit 103 by learning may be hereinafter referred to as an internally generated model. Further, as will be described later, the second model storage unit 110 stores a model similarly generated by the first identification system predetermined for the identification system 100 shown in FIG. Hereinafter, in order to distinguish from the internally generated model, the model generated by the first identification system may be referred to as an externally generated model. The model update unit 121, which will be described later, receives the model from the first identification system and stores the model in the second model storage unit 110 as an externally generated model.

The internally generated model and the externally generated model are models for identifying an object in a given new image. Hereinafter, in both the internally generated model and the externally generated model, the objects shown in the image are "car", "motorcycle", "bus", and "background (that is, the car, motorcycle, and bus are not shown)". It will be described as a model for determining which one. When learning such a model, the operator determines one of "automobile", "motorcycle", "bus", and "background" as a label paired with the image in the teacher data for each image. Also, when the externally generated model is generated by the first identification system, the operator of the first identification system can use "car", "motorcycle", "bus", etc. as labels to be paired with the image in the teacher data. One of the "backgrounds" is determined for each image.

In the present embodiment, the first identification unit 106 uses a model to determine whether the object in the image is a "car", a "motorcycle", a "bus", or a "background". As an example, the object determined by using the model is not limited to "car", "motorcycle", "bus", and "background". The operator may prepare teacher data according to the purpose of the identification process and have the learning unit 103 learn the model using the teacher data. The objects determined using the model (in this example, "automobile", "motorcycle", "bus", and "background") are common to each identification system 100.

The learning unit 103 stores the internally generated model generated by deep learning in the first model storage unit 104. The first model storage unit 104 is a storage device that stores an internally generated model.

FIG. 4 is a schematic diagram showing an example of an internally generated model and an externally generated model. Assuming that the number of pixels of the image applied to the model is n, the image can be expressed as ^{a vector (X1, X2, ..., Xn) T having each pixel value of n pixels as an element.} .. For example, X1 represents the pixel value of the first pixel in the image. The same applies to X2 to Xn. Further, here, T means transposition. The model has multiple layers and contains multiple coefficients for each layer. In the example shown in FIG. 4, the first layer contains the coefficients a1 to am, and the second layer contains the coefficients b1 to bj. The individual elements X1 to Xn of the vector representing the image are associated with the coefficients a1 to am of the first layer. In FIG. 4, this association is represented by a line. Also, each coefficient of one layer is associated with each coefficient of the next layer. In FIG. 4, this association is also represented by a line. Weights are defined between the associated elements. For example, weights are set for the associated a1 and b1, the associated a1 and b2, and the like, respectively.

The learning unit 103 determines the number of layers, the number of coefficients contained in each layer, the value of each coefficient in each layer, and the value of the weight between the associated elements by performing deep learning using the teacher data. .. The determination of these values corresponds to the generation of an internally generated model.

Different teacher data will vary in the number of layers, the number of coefficients contained in each layer, the value of the individual coefficients in each layer, and the value of the weights between the associated elements. Therefore, the internally generated model and the externally generated model can be represented as in the format illustrated in FIG. 4, but the number of layers, the number of coefficients contained in each layer, the value of each coefficient in each layer, and between the associated elements. The values of the weights of are different between the internally generated model and the externally generated model. Further, in the present embodiment, the second model storage unit 110 stores each externally generated model learned by the plurality of first identification systems. Since each externally generated model is also generated based on different teacher data in different identification systems 100, the number of layers and the like are different for each externally generated model.

The data acquisition unit 105 acquires a new image acquired by the camera 101 from the camera 101. The data acquisition unit 105 is an interface for receiving an image from the camera 101.

When the data acquisition unit 105 acquires a new image from the camera 101, the first identification unit 106 applies the image to the internally generated model stored in the first model storage unit 104. Identify the object represented by the image. In this example, the first identification unit 106 applies an image to the internally generated model to determine whether the object in the image is a "car", a "motorcycle", or a "bus". Or, it is determined whether only the "background" is shown.

When an image is obtained, the vector (X1, X2, ..., Xn) ^T representing the image is determined. The first identification unit 106 is calculated by using the vector (X1, X2, ..., Xn) ^T , each coefficient of each layer included in the internally generated model, and each weight included in the model. , "Motorcycle", "Bus", "Background" to calculate the reliability. Then, the first identification unit 106 defines the label having the highest reliability among the "automobile", "motorcycle", "bus", and "background" as a label indicating the object shown in the image. .. For example, as a result of applying the vector representing the image to the model by the first identification unit 106, the reliability of "automobile", "motorcycle", "bus", and "background" is "0.6" and "0", respectively. It is assumed that ".2", "0.1", and "0.1" are obtained. In this case, the first identification unit 106 identifies the object shown in the image as the "automobile" having the highest reliability "0.6". When the defined label is other than the "background", the first identification unit 106 represents an image of a rectangular area surrounding an object ("automobile", "motorcycle", or "bus") shown in the image. Judgment is made by an operation using a vector and an internally generated model. The fact that the defined label is the "background" means that it is determined that the object is not shown in the image. In this case, the first identification unit 106 is the object shown in the image. Do not determine the rectangular area surrounding.

The first identification unit 106 stores the image targeted for the identification process, the label corresponding to the identification result, and the reliability corresponding to the label in association with each other in the result storage unit 117. For example, as in the above example, it is assumed that the first identification unit 106 determines that the object shown in the image is an "automobile" having the highest reliability "0.6". In this case, the first identification unit 106 associates the image with the label “automobile” and the reliability “0.6” and stores the image in the result storage unit 117. The result storage unit 117 is a storage device that stores identification results and the like. However, as will be described later, the result storage unit 117 additionally stores information or the like indicating a rectangular area in the image.

The second model storage unit 110 is a storage device that stores a model different from the internally generated model (model generated by the learning unit 103). More specifically, the second model storage unit 110 stores individual models (externally generated models) trained by a plurality of first identification systems predetermined for the identification system 100 shown in FIG. Remember. Each of the individual models stored in the second model storage unit 110 is represented in the same format as the model schematically shown in FIG.

The model update unit 121 receives a model from each of a plurality of first identification systems predetermined for the identification system 100 shown in FIG. 3, and uses the model as an externally generated model in the second model storage unit 110. To memorize.

Each first identification system includes a model transmission unit 122, which will be described later, similar to the identification system 100 shown in FIG. Then, the model transmission unit 122 of each first identification system transmits the model learned by the first identification system to the identification system 100 shown in FIG. 3 as appropriate.

The model update unit 121 receives the model transmitted by the first identification system. It is assumed that the first identification system transmits the model to the identification system 100 shown in FIG. 3 for the first time. In that case, the second model storage unit 110 has not yet stored the model learned by the first identification system. At this time, the model update unit 121 stores the received model as an externally generated model in the second model storage unit 110 in association with the information indicating the first identification system as the transmission source. It is also assumed that the first identification system has previously transmitted the model to the identification system 100 shown in FIG. In that case, the second model storage unit 110 already stores the model learned by the first identification system. At this time, the model update unit 121 updates the model already stored in the second model storage unit 110 as the model learned by the first identification system to the newly received model.

In the following description, the second model storage unit 110 includes all the individual models (externally generated models) trained by the plurality of first identification systems predetermined for the identification system 100 shown in FIG. It is assumed that it has been memorized. In this state, when the model update unit 121 receives the model from the first identification system, the model update unit 121 updates the model stored in the second model storage unit 110 to the received model.

The second identification unit 111 applies a predetermined image among the images identified by the first identification unit 106 to the externally generated model stored in the second model storage unit 110. Identify the object in the given image. The second identification unit 111 executes this process for each externally generated model. The second identification unit 111 calculates the reliability of the "automobile", the "motorcycle", the "bus", and the "background" by applying the predetermined image to the externally generated model. Then, the second identification unit 111 defines the label having the highest reliability among the "automobile", "motorcycle", "bus", and "background" as a label indicating the object shown in the image. ..

Further, the predetermined image of each image identified by the first identification unit 106 is that the determination unit 107 of each image identified by the first identification unit 106 is the second identification unit. This is an image determined to cause 111 to execute the identification process.

The determination unit 107 determines an image to be identified by the second identification unit 111 among the images targeted by the first identification unit 106. Hereinafter, three types of determination methods are exemplified as a method in which the determination unit 107 determines an image to be subjected to the identification process by the second identification unit 111 among the images targeted by the first identification unit 106. I will explain. The determination unit 107 may adopt only one of the three determination methods shown below. Alternatively, the determination unit 107 may adopt a plurality of determination methods among the three types of determination methods shown below. In this case, if the determination unit 107 decides to cause the second identification unit 111 to execute the identification process on a certain image by any one of the plurality of determination methods, the image is subjected to the identification process. It is confirmed that the second identification unit 111 is to execute the identification process.

[First determination method]
In the first determination method, when the label defined by the first identification unit 106 as the label representing the object in the image is incorrect, the determination unit 107 makes a second determination with respect to the image. This is a method of determining that the identification unit 111 is to execute the identification process. That is, it is a method in which the determination unit 107 determines that the second identification unit 111 executes the identification process for the image erroneously identified by the first identification unit 106. Whether or not the label defined by the first identification unit 106 is erroneous may be determined, for example, by the operator of the identification system 100. Hereinafter, this case will be described as an example. When the first identification unit 106 determines the label for the image, the determination unit 107 determines the image, the label specified for the image, and the GUI for the operator to input whether the label is correct or not. A screen indicating (in this example, two buttons are used) is displayed on the display device 115. FIG. 5 is a schematic diagram showing an example of a screen displayed on the display device 115 by the determination unit 107 in the first determination method.

When the first identification unit 106 determines the label for the image, the determination unit 107 is determined by the image 301 targeted by the first identification unit 106 and the first identification unit 106, as illustrated in FIG. A screen showing the defined label 302 (“motorcycle” in the example shown in FIG. 5) and the first button 304 and the second button 305 is displayed on the display device 115. The first button 304 is a button for inputting that the label for the image is correct, and clicking the first button 304 means that the operator has input information that the label for the image is correct. do. Further, the second button 305 is a button for inputting that the label for the image is incorrect, and the fact that the second button 305 is clicked means that the information indicating that the label for the image is incorrect is the operator. Means that it was entered from. In the example shown in FIG. 5, the self-disembarking vehicle is shown in the image 301, but "motorcycle" is displayed as the label defined by the first identification unit 106. Therefore, the operator clicks the second button 305 using the mouse 116. In the example shown in FIG. 5, if "automobile" is displayed as the label defined by the first identification unit 106, the operator clicks the first button 304.

When the second button 305 is clicked on the screen illustrated in FIG. 5, the determination unit 107 determines that the label defined by the first identification unit 106 is incorrect, and the first identification unit 106 identifies. It is determined that the second identification unit 111 executes the identification process for the target image 301.

When the first button 304 is clicked, the determination unit 107 determines that the second identification unit 111 does not execute the identification process for the image 301 targeted by the first identification unit 106. do.

[Second determination method]
In the second determination method, when the reliability corresponding to the label defined for the image is equal to or less than the predetermined threshold value, the determination unit 107 identifies the image to the second identification unit 111. It is a method of deciding to execute the process.

That is, when the reliability corresponding to the label defined for the image by the first identification unit 106 is equal to or less than the threshold value, the determination unit 107 causes the second identification unit 111 to execute the identification process for the image. To decide. Further, when the reliability corresponding to the label defined for the image by the first identification unit 106 exceeds the threshold value, the determination unit 107 performs identification processing on the image by the second identification unit 111. Decide not to execute. The threshold value is, for example, "0.5", but may be a value other than "0.5".

In the second determination method, the determination unit 107 determines whether or not the second identification unit 111 causes the second identification unit 111 to execute the identification process on the image by comparing the reliability derived by the first identification unit 106 with the threshold value. decide. Therefore, in the second determination method, it is not necessary to display the screen illustrated in FIG.

[Third determination method]
In the third determination method, although the label defined for the image by the first identification unit 106 is the "background", the image shows "car", "motorcycle" or "bus". In this case, the determination unit 107 determines that the image is to be subjected to the identification process by the second identification unit 111. In other words, in the third determination method, even though the first identification unit 106 determines that none of the "car", "motorcycle" and "bus" is shown in the image, the "car" is shown in the image. , "Motorcycle" or "bus", the determination unit 107 determines that the second identification unit 111 is to perform the identification process on the image. When the specified label is the "background", the operator of the identification system 100 determines whether or not an "automobile" or the like is shown in the image.

In the third method, when a "background" is defined as a label for an image, the determination unit 107 displays the image, the label "background", and a screen showing the first button 304 and the second button 305 described above. Is displayed on the display device 115. FIG. 6 is a schematic diagram showing an example of a screen displayed on the display device 115 by the determination unit 107 in the third determination method.

When the first identification unit 106 defines a "background" as a label for the image, the determination unit 107 has the image 301 and the label as the identification target by the first identification unit 106, as illustrated in FIG. A screen showing the 302, the first button 304, and the second button 305 is displayed on the display device 115. On the screen displayed by the third determination method, the "background" is displayed as the label 302. The first button 304 and the second button 305 are the same as the first button 304 and the second button 305 shown in FIG. 5, and the description thereof will be omitted.

In the example shown in FIG. 6, although the label defined for the image 301 by the first identification unit 106 is "background (cars, motorcycles and buses are not shown)", the image 301 shows the image 301. The car is in the picture. Therefore, the operator clicks the second button 305 using the mouse 116. If the image 301 does not show any of the automobile, motorcycle, and bus, the operator clicks the first button 304.

In the screen illustrated in FIG. 6, when the second button 305 is clicked, the determination unit 107 specifies the label "background", but the image is any of "car", "motorcycle", and "bus". It is determined that the image is captured, and it is determined that the second identification unit 111 is to execute the identification process on the image.

When the first button 304 is clicked on the screen illustrated in FIG. 6, the determination unit 107 does not show any of "car", "motorcycle", and "bus" in the image, and the label " It is determined that the "background" is correct, and it is determined not to cause the second identification unit 111 to execute the identification process on the image.

Next, the area correction GUI display control unit 108 will be described. As described above, the first identification unit 106 surrounds the object (“car”, “motorcycle” or “bus”) shown in the image when the label defined for the image is other than the “background”. Determine the rectangular area. The area correction GUI display control unit 108 displays an image determined by the determination unit 107 to cause the second identification unit 111 to execute the identification process on the display device 115 together with the rectangular area, and further corrects the rectangular area. A screen showing the GUI for this is displayed on the display device 115. However, since the label "background" is defined for the image determined by the above-mentioned third method, the rectangular area is not determined. In this case, the area correction GUI display control unit 108 does not display the rectangular area.

FIG. 7 is a schematic diagram showing an example of a screen displayed on the display device 115 by the area correction GUI display control unit 108. The rectangular area 309 shown in FIG. 7 is a rectangular area defined by the first identification unit 106 as an area surrounding the “automobile” in the image 301. Further, the area correction GUI display control unit 108 includes a confirmation button 307 and a correction button 308 in the screen. The confirmation button 307 is a button for the operator to instruct to confirm the displayed rectangular area. The correction button 308 is a button for the operator to instruct to accept the correction of the rectangular area 309.

In the example shown in FIG. 7, the rectangular region 309 is suitable as a rectangular region surrounding the “automobile” in the image 301. When the operator determines in this way, he / she clicks the confirmation button 307. When the confirmation button 307 is clicked, the area extraction unit 109 confirms the rectangular area 309 in the image 301 at that time.

FIG. 8 is a schematic diagram showing another example of the screen displayed on the display device 115 by the area correction GUI display control unit 108. In the example shown in FIG. 8, the rectangular region 309 is not suitable as a rectangular region surrounding the “automobile” in the image 301. In this case, the area correction GUI display control unit 108 receives an appropriate rectangular area as a rectangular area surrounding the "automobile" according to the operation of the operator. When the inappropriate rectangular area 309 illustrated in FIG. 8 is displayed, the operator clicks the correction button 308. After the correction button 308 is clicked, the area correction GUI display control unit 108 accepts correction of the positions of the vertices and sides of the rectangular area 309 in response to the operation by the operator using the mouse 116. The operator can modify the rectangular area 309 to an appropriate position and size as illustrated in FIG. 7 by modifying the positions of the vertices and sides. Area modification GUI display control unit 108 accepts such modification. The operator corrects the rectangular area 309 to an appropriate position and size surrounding the object (“automobile” in this example) shown in the image 301, and then clicks the confirm button 307. As described above, when the confirmation button 307 is clicked, the area extraction unit 109 confirms the rectangular area 309 in the image 301 at that time. In this example, the region extraction unit 109 determines the modified rectangular region 309.

Further, as already described, since the label "background" is defined for the image determined by the above-mentioned third method, the rectangular area is not determined. In this case, the area correction GUI display control unit 108 does not display the rectangular area 309 on the screen illustrated in FIG. 7. In this case, when the operator clicks the correction button 308, the area correction GUI display control unit 108 displays a rectangular area 309 of an arbitrary size at an arbitrary location of the image 301, and responds to an operation by the operator using the mouse 116. Therefore, the correction of the positions of the vertices and sides of the rectangular area 309 is accepted. The operator may modify the displayed rectangular area 309 to an appropriate position and size surrounding the object shown in the image 301, and then click the confirm button 307. When the confirmation button 307 is clicked, the area extraction unit 109 confirms the rectangular area 309 in the image 301 at that time.

As described above, when the confirmation button 307 is clicked, the area extraction unit 109 confirms the rectangular area 309 in the image 301 at that time. Then, the area extraction unit 109 extracts the determined rectangular area from the image. This rectangular area is an area surrounding an object in the image. The area extraction unit 109 represents a determined rectangular area corresponding to the image stored in the result storage unit 117, the label that is the identification result by the first identification unit 106, and the reliability corresponding to the label. Information is also stored in the result storage unit 117. The information representing the rectangular area is, for example, the coordinates of each vertex of the rectangular area.

The second identification unit 111 identifies an object appearing in the image of the rectangular region with respect to the image of the rectangular region extracted by the region extraction unit 109. The second identification unit 111 executes this process for each externally generated model stored in the second model storage unit 110.

The second identification unit 111 calculates the reliability of the "automobile", "motorcycle", "bus", and "background" by applying the image of the extracted rectangular region to the externally generated model. Then, the second identification unit 111 defines the label having the highest reliability among the "automobile", "motorcycle", "bus", and "background" as a label indicating the object shown in the image. .. Further, the second identification unit 111 has already stored in the result storage unit 117 the reliability obtained for each label, the label indicating the object shown in the image, and the reliability corresponding to the label. It is stored in the result storage unit 117 in association with the image. The second identification unit 111 executes this process for each externally generated model. Hereinafter, for the sake of simplicity, there are two first identification systems defined for the identification system 100 shown in FIG. 3, and an externally generated model stored in the second model storage unit 110. It is assumed that the number of is two. Then, one of the two externally generated models is represented by the reference numeral “A”, and the other is represented by the reference numeral “B”.

In this case, the result storage unit 117 contains an image, a label determined by the first identification unit 106 by performing identification processing on the image, a reliability corresponding to the label, and a determined image. Information representing a rectangular area is stored. Further, in association with such information, the reliability of each label obtained by applying the image of the rectangular region to the externally generated model A by the second identification unit 111, and the label having the highest reliability and the label thereof. The corresponding reliability, the reliability for each label obtained by applying the image of the rectangular region to the externally generated model B by the second identification unit 111, and the reliability corresponding to the most reliable label and the label. The degree is also stored in the result storage unit 117.

The result storage unit 117 stores a set of information as described above.

However, for an image that the determination unit 107 has not determined to cause the second identification unit 111 to execute the identification process, the image and the label determined by the first identification unit 106 by performing the identification process on the image. The reliability corresponding to the label is stored in the result storage unit 117, and the information representing the rectangular area in the image is not stored.

The display control unit 112 reads out a set of information from the information stored in the result storage unit 117, the image, the label derived by the first identification unit 106, the reliability corresponding to the label, and the second. A screen including the label derived for each externally generated model by the identification unit 111 of 2 and the reliability corresponding to the label is displayed on the display device 115.

FIG. 9 is a schematic diagram showing an example of a screen displayed by the display control unit 112. The display control unit 112 corresponds to the label derived by the first identification unit 106 and the reliability 501 corresponding to the label, and the label derived by the second identification unit 111 using the externally generated model A and the label thereof. A screen in which the reliability 502, the label derived by the second identification unit 111 using the externally generated model B, and the reliability 503 corresponding to the label are superimposed on the image 301 is displayed on the display device 115. In the example shown in FIG. 9, the display control unit 112 also superimposes and displays the fixed rectangular region 309 on the image 301. In this example, the case where the number of externally generated models stored in the second model storage unit 110 is two is illustrated, but the number of externally generated models may be three or more.

Further, the display control unit 112 displays the check box 504, the relearning button 505, and the screen switching buttons 506 and 507 in this screen.

The check box 504 is a GUI for designating whether or not to include the image 301 (more specifically, the image of the rectangular region 309 extracted from the image 301) displayed in the screen in the teacher data. When the check box 504 is checked, it means that the image of the rectangular region 309 extracted from the image 301 is included in the teacher data. If the check box 504 is unchecked, it means that the image 301 is not included in the teacher data. The display control unit 112 may display the check box 504 in a pre-checked state according to the reliability derived using the externally generated model. For example, in the label and reliability set derived using the externally generated model, if there is one or more sets whose reliability is larger than the threshold value (for example, “0.5”), the display control unit 112 may use the display control unit 112 in advance. The check box 504 may be displayed in the checked state. The operator can check or uncheck the check box 504 by clicking the check box 504 with the mouse 116. The operator may determine whether or not to include the image of the rectangular region 309 extracted from the image 301 in the teacher data by referring to the image 301 and the label and the reliability derived for each externally generated model. Then, the operator may decide whether or not to check the check box 504 based on the judgment.

The screen switching buttons 506 and 507 are buttons for switching to a screen displaying different images. For example, when the screen switching button 506 is clicked, the display control unit 112 switches to a screen similar to the screen shown in FIG. 9, which includes images before the image 301 in chronological order. Further, for example, when the screen switching button 507 is clicked, the display control unit 112 switches to a screen similar to the image shown in FIG. 9, which includes images after the image 301 in chronological order. The operator may decide whether or not to check the check box 504 on each switched screen.

The relearning button 505 is a button for the operator to instruct the identification system 100 to relearn the internally generated model. When the relearning button 505 is clicked, the integration unit 114 specifies a label for each image of the screen in which the check box 504 is checked. In the following description, for the sake of simplicity, the case where the check box 504 is checked will be described as an example only on the screen illustrated in FIG. 9. In this case, the integration unit 114 identifies the label of the image 301 illustrated in FIG.

Hereinafter, before the process of specifying the label of one image by the integration unit 114 will be described, the attribute data storage unit 113 will be described first. The attribute data storage unit 113 stores data (attribute data) indicating the attributes of the camera 101 connected to the computer 102 (computer 102 shown in FIG. 3) including the attribute data storage unit 113, and the second model storage unit 110. It is a storage device that stores the attribute data of the camera 101 of each identification system 100 (that is, each first identification system) that generated each stored externally generated model. The attribute data of the camera 101 of the identification system 100 (first identification system) that generated a certain externally generated model is referred to as attribute data corresponding to the externally generated model.

Examples of the attributes of the camera 101 include attributes of the camera 101 itself, attributes depending on the environment in which the camera 101 is installed, and the like. The value of each attribute is expressed numerically. Further, the value of each attribute may be determined in advance by the administrator of each identification system 100 according to the setting of the camera 101 and the installation environment. Attribute data is represented by a vector whose elements are the values (numerical values) of such attributes.

The attribute data of the camera 101 is at least "the angle of view of the camera 101", "whether the camera 101 is installed indoors or outdoors", "the shooting target of the camera 101", and "the shooting target of the camera 101". Includes the value of at least some of the "movement direction" attributes. Further, which attribute value is used as an element in the attribute data represented by the vector is common to all the identification systems 100, and all the attribute values are the element of the vector. It is common to the identification system 100. The numerical value of each element of the vector may be different for each identification system 100.

Since the "angle of view of the camera 101" is represented by a numerical value, the administrator may determine a numerical value representing the angle of view as an element of the vector.

Regarding the attribute "whether the camera 101 is installed indoors or outdoors", for example, when the camera 101 is installed indoors, the value of this attribute is set to "0" and the camera 101 is set. If is installed outdoors, the value of this attribute may be set to "1".

Further, regarding the attribute of "shooting target of the camera 101", for example, when the camera 101 is installed so as to shoot a vehicle (for example, when the camera 101 is installed toward the roadway), this attribute Set the value to "0". Further, when the camera 101 is installed so as to photograph a pedestrian (for example, when the camera 101 is installed toward the sidewalk), the value of this attribute is set to "1". Also, if the camera 101 is installed to capture both the vehicle and the pedestrian (for example, if the camera 101 is installed toward the road through which both the vehicle and the pedestrian pass), the value of this attribute. Is set to "0.5".

Regarding the attribute "moving direction of the shooting target of the camera 101", a reference axis based on the main axis direction of the camera 101 and the like is determined, and the angle formed by the reference axis and the main moving direction of the shooting target is the value of this attribute. It should be determined as.

Further, the value of the attribute other than the above may be included in the attribute data. For example, values such as "height of the installation location of the camera 101", "depression angle of the camera 101", and "resolution of the camera 101" may be included in the attribute data. Since the "height of the installation location of the camera 101", the "depression angle of the camera 101", and the "resolution of the camera 101" are all represented by numerical values, these numerical values may be defined as vector elements.

The attribute data storage unit 113 stores the attribute data (vector) of the camera 101 connected to the computer 102 (computer 102 shown in FIG. 3) including the attribute data storage unit 113. This attribute data is referred to as reference attribute data. Further, the attribute data storage unit 113 stores the attribute data of the camera 101 of each first identification system that generated each externally generated model stored in the second model storage unit 110. In the present embodiment, the second model storage unit 110 stores the externally generated model A and the externally generated model B. Therefore, in the attribute data storage unit 113, in addition to the reference attribute data, the attribute data corresponding to the externally generated model A (referred to as attribute data A) and the attribute data corresponding to the externally generated model B (with the attribute data B). Also remember.). The attribute data A is the attribute data of the camera 101 of the first identification system that generated the externally generated model A. Similarly, the attribute data B is the attribute data of the camera 101 of the first identification system that generated the externally generated model B.

The administrator who manages each identification system 100 may store the attribute data of the camera 101 in FIG. 3 as reference attribute data in the attribute data storage unit 113. Further, the administrator uses the attribute data of the camera 101 of each of the two first identification systems defined for the identification system 100 shown in FIG. 3 as the attribute data A and the attribute data B in the attribute data storage unit 113. Just memorize it.

In the integration unit 114, the reliability of each label derived by the second identification unit 111 for each externally generated model for the image (in the present embodiment, "automobile", "motorcycle", "bus", "bus", " The "background" (reliability of each) is integrated for each label, and the label of the image is specified based on the integrated result.

At this time, the integrated unit 114 has the reference attribute data (that is, the attribute data of the camera 101 of the identification system 100 including the integrated unit 114) and the plurality of first identification systems that have generated the externally generated model A and the externally generated model B. The degree of similarity with the attribute data of the camera 101 is calculated for each first identification system. In the present embodiment, the integration unit 114 calculates the degree of similarity between the reference attribute data and the attribute data A and the degree of similarity between the reference attribute data and the attribute data B, respectively. The degree of similarity between the reference attribute data and the attribute data A is referred to as the degree of similarity corresponding to the externally generated model A. Further, the degree of similarity between the reference attribute data and the attribute data B is referred to as the degree of similarity corresponding to the externally generated model B.

Attribute data is represented by a vector. When the integration unit 114 calculates the similarity between two attribute data (vectors), the reciprocal of the distance between the two vectors may be calculated as the similarity.

The integration unit 114 weights the reliability of each label derived for each externally generated model with the similarity corresponding to the externally generated model when integrating each label, and integrates the labels. The integration unit 114 may specify the label with the highest reliability integration result as the image label.

The operation of integrating the reliability of each label derived for each externally generated model for each label will be specifically described. Two calculation methods will be described as a calculation method in which the integration unit 114 integrates the reliability. Here, a case where the reliability derived for each externally generated model is integrated for one label will be described. The integration unit 114 may perform the same calculation for other labels to integrate the reliability derived for each externally generated model.

[First calculation method]
First, a first calculation method for integrating reliability will be described. Let Li be the reliability of the label of interest obtained by using the i-th external generative model. Further, the degree of similarity calculated for the i-th externally generated model (the degree of similarity between the reference attribute data and the attribute data corresponding to the i-th externally generated model) is defined as Wi. Further, the number of externally generated models stored in the second model storage unit 110 is N. In this case, the integration unit 114 may integrate the reliability of the label of interest by the calculation of the following equation (1).

That is, the integration unit 114 may calculate the product of Li and Wi for each externally generated model, and use the average value of the product as the integration result of the reliability of the label of interest. The integration unit 114 performs the same calculation for other labels. Then, the integration unit 114 identifies the label with the highest integration result as the label of the image.

FIG. 10 is an explanatory diagram showing a specific example of the first calculation method. Suppose there are two externally generated models A and B. The reliability of "automobile", "motorcycle", "bus" and "background" derived using the externally generated model A is "0.1", "0.7", "0.1", "", respectively. It is assumed to be 0.1 ". Further, it is assumed that the similarity calculated for the externally generated model A is "0.9". The integration unit 114 calculates the result of multiplying each of the above reliabilitys by the similarity degree "0.9". As a result, the multiplication results (product) of "0.09", "0.63", "0.09", and "0.09" for each of "automobile", "motorcycle", "bus", and "background". Is obtained.

In addition, the reliability of "automobile", "motorcycle", "bus" and "background" derived using the externally generated model B is "0.1", "0.6" and "0.2", respectively. , "0.1". Further, it is assumed that the similarity calculated for the externally generated model B is "0.8". The integration unit 114 calculates the result of multiplying each of the above reliabilitys by the similarity degree "0.8". As a result, the multiplication results (product) of "0.08", "0.48", "0.16", and "0.08" for each of "automobile", "motorcycle", "bus", and "background". Is obtained.

The integration unit 114 calculates the average value of the multiplication results (products) obtained for each of the "automobile", "motorcycle", "bus" and "background". The average values calculated for each of "automobile", "motorcycle", "bus" and "background" are "0.085", "0.555", "0.125" and "0.085". Therefore, the integration unit 114 identifies the "motorcycle" having the highest average value (integration result) as the label of the image.

[Second calculation method]
Next, a second calculation method for integrating reliability will be described. As in the above case, the reliability of the label of interest obtained by using the i-th externally generated model is Li. Further, the degree of similarity calculated for the i-th externally generated model (the degree of similarity between the reference attribute data and the attribute data corresponding to the i-th externally generated model) is defined as Wi. Further, the sum of the individual similarity calculated for each externally generated model is defined as Wt. Further, the number of externally generated models stored in the second model storage unit 110 is N. The integration unit 114 may calculate Wt by the calculation of the following equation (2).

In this case, the integration unit 114 may integrate the reliability of the label of interest by the calculation of the following equation (3).

That is, the integration unit 114 calculates the ratio of the similarity corresponding to the externally generated model to the total sum of the similarity for each externally generated model, and uses the calculation result of the ratio as a weight to determine the reliability of the label of interest. The weighted sum may be calculated, and the calculation result may be used as the integration result of the reliability of the label of interest. The integration unit 114 performs the same calculation for other labels. Then, the integration unit 114 identifies the label with the highest integration result as the label of the image.

FIG. 11 is an explanatory diagram showing a specific example of the second calculation method. Suppose there are two externally generated models A and B. The reliability of "automobile", "motorcycle", "bus" and "background" derived using the externally generated model A is "0.1", "0.7", "0.1", "", respectively. It is assumed to be 0.1 ". The reliability of "automobile", "motorcycle", "bus" and "background" derived using the externally generated model B is "0.1", "0.6", "0.2", "", respectively. It is assumed to be 0.1 ". It is assumed that the similarity calculated for the externally generated model A is "0.9" and the calculated similarity for the externally generated model B is "0.8". In this case, the sum of the similarity is 0.9 + 0.8 = 1.7. Therefore, the ratio of the similarity "0.9" corresponding to the externally generated model A to the total similarity "1.7" is "0.9 / 1.7". The ratio of the similarity "0.8" corresponding to the externally generated model B to the total similarity "1.7" is "0.8 / 1.7". The integration unit 114 calculates the weighted sum of the reliability for each label with "0.9 / 1.7" and "0.8 / 1.7" as weights, and the calculation result is the reliability of the label. The result of the integration. Then, the integration results of "automobile", "motorcycle", "bus" and "background" are "0.0999", "0.6528", "0.1470" and "0.0999" respectively. Therefore, the integration unit 114 identifies the "motorcycle" with the highest integration result as the label of the image.

It can be said that both the first calculation method and the second calculation method are operations in which the reliability of the label derived for each externally generated model is weighted and integrated by the similarity corresponding to the externally generated model. ..

When the integration unit 114 identifies the label of the image based on the integration result of the reliability of each label, the learning unit 103 extracts a rectangular region determined in the image and integrates it with the image of the rectangular region. The pair with the label identified by part 114 is included in the existing teacher data. Then, the learning unit 103 relearns the internally generated model by deep learning using the teacher data. Further, the learning unit 103 updates the existing internally generated model stored in the first model storage unit 104 with a new internally generated model generated by re-learning.

Next, the model transmission unit 122 will be described. The model transmission unit 122 transmits the model learned by the learning unit 103 to a predetermined second identification system. The number of the second identification systems to which the model is transmitted may be one or a plurality. The second identification system stores the received model as an externally generated model.

As a mode in which the model transmission unit 122 transmits the model to the second identification system, four model transmission modes will be illustrated and described.

[First model transmission mode]
In the first model transmission mode, the model transmission unit 122 transmits the model to the second identification system when the learning unit 103 relearns the model (internally generated model). In the first model transmission mode, each time the model is relearned by the learning unit 103, the model transmission unit 122 transmits the model newly obtained by the relearning to the second identification system. Therefore, in the first model transmission mode, the model transmission unit 122 can transmit the latest model to the second identification system.

[Second model transmission mode]
In the second model transmission mode, the model transmission unit 122 periodically transmits the model stored in the first model storage unit 104 to the second identification system. That is, the model transmission unit 122 stores the model stored in the first model storage unit 104 in the first model storage unit 104 again when a certain period of time has elapsed after transmitting the model to the second identification system. It repeats sending the model to the second identification system. Even if the model stored in the first model storage unit 104 is updated a plurality of times in this fixed period, the model transmission unit 122 will be the first model when a certain period has elapsed from the time of the previous model transmission. The model stored in the storage unit 104 is transmitted to the second identification system. Further, if the model stored in the first model storage unit 104 is not updated in this fixed period, the model transmission unit 122 decides to transmit the same model as the previously transmitted model again. Become.

[Third model transmission mode]
In the third model transmission mode, when the learning unit 103 relearns the model (internally generated model), the model transmission unit 122 determines whether or not to transmit the model to the second identification system.

Among the images to be identified by the first identification unit 106, the image to be identified by the integration unit 114 is the image to which the determination unit 107 is the above-mentioned first determination method, second determination method, or second. It is an image determined to cause the second identification unit 111 to execute the identification process by the determination method of 3. That is, the image to be specified by the integration unit 114 as the label is an image that has been erroneously identified by the first identification unit 106 or has a reliability higher than the threshold value. Before the learning unit 103 retrains the model, the integration unit 114 identifies such image labels by utilizing the reliability of each label obtained for each externally generated model. In the third model transmission mode, when the learning unit 103 relearns the model (internally generated model), the first identification unit 106 applies the image to the relearned model to obtain the image. The identification result (label) of is derived again. When the model transmission unit 122 matches the identification result (label) of the image derived again by the model retrained by the first identification unit 106 with the label specified by the integration unit 114. First, it is determined that the retrained model is transmitted to the second identification system, and the retrained model is transmitted to the second identification system. On the other hand, if the two labels do not match, the model transmission unit 122 determines that the retrained model is not transmitted to the second identification system, and does not transmit the model to the second identification system.

Each time the learning unit 103 relearns the model, the model transmission unit 122 determines whether or not to transmit the model to the second identification system, and if it is determined to transmit the model, the model is transmitted. Is sent to the second identification system.

The integration unit 114 identifies the label of the image by using the reliability of each label obtained for each externally generated model. Therefore, it is considered that the accuracy of the label specified by the integration unit 114 is high even if the image is erroneously identified by the first identification unit 106 or the reliability larger than the threshold value cannot be obtained. Therefore, it is reminded that the identification result (label) of the image derived again by the first identification unit 106 using the retrained model and the label specified by the integration unit 114 match. It can be said that the identification accuracy of the model obtained by training is improved compared to the identification accuracy of the model before re-learning. It can also be said that the model before re-learning did not give the correct discrimination result, but the model obtained by the re-learning now gives the correct discrimination result. Therefore, in the third model transmission mode, when the identification accuracy of the model obtained by re-learning is higher than the identification accuracy of the model before re-learning, the model obtained by re-learning is used as the second identification system. It can be said that it is a mode of transmission. Further, in the third model transmission mode, when the correct discrimination result is not obtained in the model before re-learning, but the correct discrimination result can be obtained in the model obtained by re-learning, it is re-learned. It can also be said that the model obtained by learning is transmitted to the second identification system.

[Fourth model transmission mode]
Also in the fourth model transmission mode, when the learning unit 103 relearns the model (internally generated model), the model transmission unit 122 transmits the model to the second identification system. Is determined.

In the fourth model transmission mode, when the model is retrained, the model is transmitted based on the correct answer rate of the identification result (label) derived by the first identification unit 106 using the model within a predetermined time. Unit 122 determines whether to send the model to the second identification system. When the correct answer rate is equal to or higher than a predetermined threshold value, the model transmission unit 122 determines that the relearned model is to be transmitted to the second identification system, and transmits the model to the second identification system. On the other hand, when the correct answer rate is less than the threshold value, the model transmission unit 122 determines that the relearned model is not transmitted to the second identification system, and does not transmit the model to the second identification system.

An example of calculating the correct answer rate of the identification result (label) within a predetermined time will be described. Here, a case where the determination unit 107 determines an image to be executed by the second identification unit 111 by the above-mentioned "first determination method" will be described as an example. When the first identification unit 106 determines the label for the image, the determination unit 107 displays the screen illustrated in FIG. 5 on the display device 115. Therefore, it can be said that the number of times that the determination unit 107 displays the screen illustrated in FIG. 5 is the number of times that the first identification unit 106 identifies the image. Then, it can be said that the number of times the first button 304 is clicked on the displayed screen (see FIG. 5) is the number of times that the label derived by the first identification unit 106 is correct. The model transmission unit 122 calculates the ratio of the number of times the first button 304 is clicked to the number of times the determination unit 107 displays the screen illustrated in FIG. 5 within a predetermined time when the model is retrained. The result may be the correct answer rate. Then, the model transmission unit 122 may determine whether or not to transmit the relearned model to the second identification system by comparing the correct answer rate with the threshold value.

In the present embodiment, the model update unit 121 and the model transmission unit 122 are realized by, for example, a CPU (Central Processing Unit) of a computer 102 operating according to a model providing program and a communication interface of the computer 102. For example, the CPU may read the model providing program from a program recording medium such as a program storage device of the computer 102, and operate as the model updating unit 121 and the model transmitting unit 122 by using the communication interface according to the model providing program.

Further, the learning unit 103, the first identification unit 106, the determination unit 107, the area correction GUI display control unit 108, the area extraction unit 109, the second identification unit 111, the display control unit 112, and the integration unit 114 are, for example, models. It is realized by the CPU of the computer 102 that operates according to the provided program. For example, the CPU reads the model providing program from a program recording medium such as a program storage device of the computer 102, and according to the model providing program, the learning unit 103, the first identification unit 106, the determination unit 107, and the area correction GUI display control unit 108. , The area extraction unit 109, the second identification unit 111, the display control unit 112, and the integration unit 114 may operate.

Further, the first model storage unit 104, the second model storage unit 110, the attribute data storage unit 113, and the result storage unit 117 are realized by a storage device included in the computer 102.

Next, the processing process of the embodiment of the present invention will be described. FIG. 12 is a flowchart showing an example of the processing process from when the camera 101 takes a picture to when the second identification unit 111 performs the identification process for the image. The detailed description of the operation already described will be omitted.

It is assumed that the learning unit 103 learns the internally generated model by deep learning in advance and stores the internally generated model in the first model storage unit 104.

Further, the model update unit 121 receives a model from each of a plurality of first identification systems predetermined for the identification system 100 shown in FIG. 3, and stores the model as an externally generated model in the second model. It is assumed that it is stored in the unit 110. That is, it is assumed that the second model storage unit 110 stores the model learned by each first identification system as an externally generated model. When a new model is received from the first identification system, the model update unit 121 already stores the model in the second model storage unit 110 as a model learned by the first identification system. Should be updated to the newly received model.

First, the camera 101 obtains an image by taking a picture at the installation location of the camera 101 (step S1). The camera 101 transmits the image to the computer 102.

The first identification unit 106 of the computer 102 receives the image via the data acquisition unit 105. Then, the first identification unit 106 identifies the object appearing in the image by applying the image to the internally generated model (step S2). In step S2, the first identification unit 106 derives a label representing an object shown in the image and the reliability of the label. The first identification unit 106 stores the image in the result storage unit 117 in association with the derived label and the reliability. Further, when the specified label is not the "background", the first identification unit 106 determines a rectangular area surrounding the object in the image.

Next, the determination unit 107 determines whether or not the second identification unit 111 is to execute the identification process for the image targeted by the first identification unit 106 in step S2 (step S3). If it is determined not to cause the second identification unit 111 to execute the identification process (No in step S3), the processes after step S1 are repeated.

When it is determined that the second identification unit 111 is to execute the identification process (Yes in step S3), the area correction GUI display control unit 108 displays the image on the display device 115. For example, the area correction GUI display control unit 108 displays the screen illustrated in FIGS. 7 and 8 on the display device 115. Then, the area extraction unit 109 determines a rectangular area surrounding the object in the image according to the operator's operation on the screen, and extracts the rectangular area from the image (step S4).

Next, the second identification unit 111 identifies the object shown in the image of the rectangular region extracted in step S4 for each externally generated model stored in advance in the second model storage unit 110. (Step S5). The second identification unit 111 derives the reliability of each label (“automobile”, “motorcycle”, “bus” and “background”) for each externally generated model. Then, the reliability of each label derived for each externally generated model is stored in the result storage unit 117. Further, the second identification unit 111 also stores the label with the highest reliability and the set of reliability corresponding to the label in the result storage unit 117 for each externally generated model. The label with the highest reliability represents an object that is determined to appear in the image.

After step S5, the processing after step S1 is repeated.

FIG. 13 is a flowchart showing an example of the processing progress when the model (internally generated model) is relearned and the model is transmitted to the second identification system. Also in the following description, detailed description of the operation already described will be omitted. In FIG. 13, the above-mentioned "first model transmission mode" will be described as an example.

The display control unit 112 has the label derived from the first identification unit 106 and the reliability corresponding to the label, and the label derived from each externally generated model by the second identification unit 111 and the reliability corresponding to each label. The screen superimposed on the image is displayed on the display device 115 (step S11). At this time, the display control unit 112 includes a check box 504, a relearning button 505, and screen switching buttons 506 and 507 in this screen. In step S11, the display control unit 112 displays, for example, the screen illustrated in FIG. 9.

The operator confirms the screen illustrated in FIG. 9 and determines whether or not to include the displayed image 301 (more specifically, the image of the rectangular region 309 determined in the image 301) in the teacher data. .. The operator specifies that the displayed image 301 is included in the teacher data by checking the check box 504. That is, the image displayed on the screen in which the check box 504 is checked is an image designated as an image to be included in the teacher data. Further, the operator clicks the re-learning button 505 after designating the image to be included in the teacher data.

When the relearning button 505 is clicked by the operator, the integration unit 114 calculates the similarity between the reference attribute data and the individual attribute data corresponding to each externally generated model (step S12). As described above, the attribute data is represented by a vector. When the integration unit 114 calculates the similarity between two attribute data (vectors), the reciprocal of the distance between the two vectors may be calculated as the similarity.

Next, the integration unit 114 integrates the reliability of the labels derived for each externally generated model using the similarity calculated in step S12. The integration unit 114 performs this process for each label, and identifies the label having the highest reliability integration result as the label for the image to be included in the teacher data (step S13).

When a plurality of images to be included in the teacher data are specified by the operator, the integration unit 114 executes the process of step S13 for each image.

Next, the learning unit 103 extracts a rectangular region determined in the image to be included in the teacher data, and includes a set of the image of the rectangular region and the label specified by the integration unit 114 in the existing data. Then, the learning unit 103 relearns the internally generated model by deep learning using the teacher data, and stores the internally generated model obtained by the relearning in the first model storage unit 104 (step S14). The learning unit 103 updates the existing internally generated model stored in the first model storage unit 104 with a new internally generated model generated by re-learning.

After that, when the first identification unit 106 identifies an object in the image, a new internally generated model generated by re-learning is used.

Following step S14, the model transmission unit 122 transmits the model (internally generated model) retrained in step S14 to the second identification system (step S15). Each identification system 100 has a similar configuration. Upon receiving the model transmitted in step S15, the model update unit 121 of the second identification system updates the model stored in the second model storage unit 110 in the second identification system to the received model. do. Therefore, the model transmitted by the model transmission unit 122 to the second identification system in step S15 is stored as an externally generated model in the second identification system.

In FIG. 13, the "first model transmission mode" has been described as an example. When the model transmission unit 122 transmits the model to the second identification system in the above-mentioned "second model transmission mode", the computer 102 may end the process in step S14. Then, apart from the processing of steps S11 to S14, the model transmission unit 122 may periodically transmit the model stored in the first model storage unit 104 to the second identification system.

FIG. 14 is a flowchart showing an example of the processing progress of the above-mentioned “third model transmission mode”. The processing up to step S14 is the same as the flowchart shown in FIG. 13, and the description thereof will be omitted.

After step S14, the first identification unit 106 uses the internally generated model retrained in step S14 to label the image (see FIG. 9) displayed in the screen in which the check box 504 is checked. (Step S21). The image displayed in the screen in which the check box 504 is checked is an image specified by the operator to be included in the teacher data.

Next, the model transmission unit 122 determines whether or not the label derived in step S21 matches the label specified by the integration unit 114 in step S13 (see FIG. 13) for the same image (step). S22).

When the two labels match (Yes in step S22), the model transmitter 122 determines that the model (internally generated model) retrained in step S14 (see FIG. 13) is transmitted to the second identification system. The model is transmitted to the second identification system (step S23).

If the two labels do not match (No in step S22), the model transmission unit 122 determines that the model retrained in step S14 will not be transmitted to the second identification system, and ends the process.

FIG. 15 is a flowchart showing an example of the processing progress of the above-mentioned “fourth model transmission mode”. The processing up to step S14 is the same as the flowchart shown in FIG. 13, and the description thereof will be omitted.

After step S14, the model transmission unit 122 calculates the correct answer rate of the image identification result when the relearned model (internally generated model) is used within a predetermined time (step S31). Since an example of calculating the correct answer rate has already been described, the description thereof will be omitted here.

Next, the model transmission unit 122 determines whether or not the correct answer rate calculated in step S31 is equal to or greater than a predetermined threshold value (step S32).

When the correct answer rate is equal to or higher than the threshold value (Yes in step S32), the model transmission unit 122 determines that the model (internally generated model) retrained in step S14 (see FIG. 13) is transmitted to the second identification system. , The model is transmitted to the second identification system (step S33).

If the correct answer rate is less than the threshold value (No in step S32), the model transmission unit 122 determines that the model relearned in step S14 is not transmitted to the second identification system, and ends the process.

According to the present embodiment, the determination unit 107 is a method of at least one of the above-mentioned first determination method, second determination method, and third determination method, and the first identification unit 106 performs identification processing. It is determined whether or not the second identification unit 111 is to execute the identification process for the target image. Therefore, in the image for which the identification process is performed by the second identification unit 111, the reliability corresponding to the image in which the label defined by the first identification unit 106 is incorrect and the label defined for the image is the threshold value. In the image below, or in an image showing an object ("automobile", "motorcycle" or "bus") even though the label defined by the first identification unit 106 is the "background". be. In the present embodiment, such an image is converted into a model different from the internally generated model (more specifically, a model generated by a first identification system predetermined for the identification system 100 of interest). (Externally generated model)) The pair of the label specified based on the result applied to the image and the image thereof is added to the existing teacher data, and the learning unit 103 relearns the internally generated model. Therefore, the identification accuracy of the internally generated model can be improved.

Then, the model transmission unit 122 transfers the model learned by the learning unit 103 to, for example, the above-mentioned first model transmission mode, second model transmission mode, third model transmission mode, or fourth model transmission mode. It is transmitted to a predetermined second identification system according to an aspect or the like. In the second identification system, the model update unit 121 stores the received model in the second model storage unit 110 as an externally generated model. Therefore, each identification system 100 can relearn its own model so as to improve the identification accuracy of its own model, and can contribute to the improvement of the identification accuracy of the model of another identification system 100. can.

Next, various modifications of the embodiment of the present invention will be described.

In the above embodiment, the area extraction unit 109 determines a rectangular area surrounding the object shown in the image and extracts the rectangular area from the image in response to the operator's operation on the screens exemplified in FIGS. 7 and 8. do. Then, the second identification unit 111 identifies the object reflected in the image of the extracted rectangular region for each externally generated model. The second identification unit 111 performs a process of identifying an object appearing in the image by targeting the entire image targeted by the first identification unit 106, not the image of the extracted rectangular region. You may. In this case, the identification system 100 (see FIG. 3) may not include the area correction GUI display control unit 108 and the area extraction unit 109. Then, the identification system 100 does not have to execute step S4 (see FIG. 12). In step 5, the second identification unit 111 may identify an object in the image by targeting the entire image processed by the first identification unit 106.

Further, the learning unit 103 may include the set of the entire image and the label specified by the integration unit 114 in the existing teacher data, and relearn the internally generated model using the teacher data.

Further, the learning unit 103 may relearn the internally generated model by deep learning using the pair of the image and the label specified by the integration unit 114 and the existing internally generated model as teacher data.

Further, the second model storage unit 110 may store one externally generated model. In this case, the learning unit 103 includes the pair of the image and the label with the highest reliability derived by the second identification unit 111 in the existing teacher data, and uses the teacher data to generate an internally generated model. Should be relearned.

FIG. 16 is a schematic block diagram showing a configuration example of a computer 102 included in the identification system 100 in the embodiment of the present invention and its modifications. In FIG. 16, the computer is represented by the reference numeral “1000”. The computer 1000 is an interface between a CPU 1001, a main storage device 1002, an auxiliary storage device 1003, an interface 1004, a display device 1005, an input device 1006, a communication interface 1007, and a data collection unit 101 (for example, a camera). It is equipped with 1008.

The operation of the computer included in the identification system 100 is stored in the auxiliary storage device 1003 in the form of a model-provided program. The CPU 1001 reads the model providing program from the auxiliary storage device 1003 and deploys it to the main storage device 1002. Then, the CPU 1001 executes the processing of the computer 102 (see FIG. 3) in the above-described embodiment and its modification according to the model providing program.

Auxiliary storage 1003 is an example of a non-temporary tangible medium. Other examples of non-temporary tangible media include magnetic disks, optical magnetic disks, CD-ROMs (Compact Disk Read Only Memory), DVD-ROMs (Digital Versatile Disk Read Only Memory), which are connected via interface 1004. Examples include semiconductor memory. Further, when this program is distributed to the computer 1000 by a communication line, the distributed computer 1000 may expand the program to the main storage device 1002 and execute the above processing.

Further, the program may be for realizing a part of the processing of the computer 102 shown in the embodiment and the modification thereof. Further, the program may be a difference program that realizes the above-mentioned processing in combination with another program already stored in the auxiliary storage device 1003.

Further, a part or all of each component may be realized by a general-purpose or dedicated circuitry, a processor, or a combination thereof. These may be composed of a single chip or may be composed of a plurality of chips connected via a bus. A part or all of each component may be realized by the combination of the circuit or the like and the program described above.

When a part or all of each component is realized by a plurality of information processing devices and circuits, the plurality of information processing devices and circuits may be centrally arranged or distributed. For example, the information processing device, the circuit, and the like may be realized as a form in which each is connected via a communication network, such as a client-and-server system and a cloud computing system.

Next, the outline of the present invention will be described. FIG. 17 is a block diagram showing an outline of the identification system of the present invention. The identification system of the present invention includes a learning means 701, a first model storage means 702, a first identification means 703, a second model storage means 704, a model update means 705, and a second identification means 706. And model transmission means 707.

The learning means 701 (for example, the learning unit 103) learns a model for identifying an object represented by the data (for example, an image) by using the teacher data.

The first model storage means 702 (first model storage unit 104) stores the model learned by the learning means 701.

The first identification means 703 (for example, the first identification unit 106) identifies the object represented by the data by using the model learned by the learning means 701.

The second model storage means 704 (for example, the second model storage unit 110) stores each individual model trained by a plurality of predetermined identification systems.

When the model update means 705 (for example, the model update unit 121) receives the model learned by the first identification system from the first identification system, it is stored in the second model storage means 704. The model trained in the first identification system is updated with the model received from the first identification system.

In the second identification means 706 (for example, the second identification unit 111), the first identification means 703 is an identification target for each model stored in the second model storage means 704 in a predetermined case. Identify the object represented by the data.

The learning means 701 relearns the model using the teacher data including the label for the data determined based on the discrimination result derived by the second discriminating means 706 and the data, and uses the learning means 701 as the first model storage means 702. Update the stored model to the retrained model.

The model transmitting means 707 (for example, the model transmitting unit 122) transmits the model learned by the learning means 701 to one or a plurality of predetermined second identification systems.

With such a configuration, the identification system of the present invention can relearn its own model so as to improve the identification accuracy of its own model, and also contributes to the improvement of the identification accuracy of the models of other identification systems. can do.

The model transmitting means 707 may be configured to transmit the model to the second identification system when the learning means 701 relearns the model.

The model transmission means 707 may be configured to periodically transmit the model stored in the first model storage means 702 to the second identification system.

By integrating the identification results derived by the second identification means 706 for each individual model stored in the second model storage means 704, the label for the data targeted by the first identification means 703 is specified. The learning means 701 retrains the model using the teacher data including the label specified by the integrating means and the data, and the model transmitting means 707 includes the integrating means (for example, the integrating unit 114). If the identification result of the data derived by the first discriminating means 703 using the retrained model matches the label for the data identified by the integrating means, the retrained model is referred to as the second. It may be configured to be transmitted to the identification system.

When the model transmitting means 707 relearns the model by the learning means 701, the correct answer rate of the discrimination result derived by the first discriminating means 703 using the model within a predetermined time is equal to or higher than a predetermined threshold value. If this is the case, the model may be transmitted to the second identification system.

Although the invention of the present application has been described above with reference to the embodiments, the invention of the present application is not limited to the above-described embodiment. Various changes that can be understood by those skilled in the art can be made within the scope of the present invention in terms of the configuration and details of the present invention.

Possibility of industrial use

The present invention is suitably applied to an identification system that identifies an object represented by the data by applying the data to a model.

100 Identification system 101 Data collection unit 102 Computer 103 Learning unit 104 First model storage unit 105 Data acquisition unit 106 First identification unit 107 Determination unit 108 Area correction GUI display control unit 109 Area extraction unit 110 Second model storage unit 111 Second identification unit 112 Display control unit 113 Attribute data storage unit 114 Integration unit 115 Display device 116 Mouse 117 Result storage unit 121 Model update unit 122 Model transmission unit

Claims (7)

A learning method for learning a model for identifying an object represented by data using teacher data,
A first model storage means for storing the model learned by the learning means,
Using the model learned by the learning means, a first identification means for identifying an object represented by data, and
A second model storage means for storing individual models trained by a plurality of predetermined identification systems, and a second model storage means.
When the model learned by the first identification system is received from the first identification system, the model learned by the first identification system stored in the second model storage means is referred to as described above. Model update means for updating to the model received from the first identification system,
In a predetermined case, each model stored in the second model storage means is provided with a second identification means for identifying an object represented by the data represented by the data as an identification target by the first identification means. ,
The learning means is
The model is relearned using the teacher data including the label for the data determined based on the identification result derived by the second identification means and the data, and is stored in the first model storage means. Update the model to the retrained model and
A discrimination system comprising: a model transmission means for transmitting a model learned by the learning means to one or a plurality of predetermined identification systems. The model transmission method is
The identification system according to claim 1, wherein when the learning means relearns the model, the model is transmitted to the second identification system. The model transmission method is
The identification system according to claim 1, wherein the model stored in the first model storage means is periodically transmitted to the second identification system. By integrating the identification results derived for each model stored in the second model storage means by the second identification means, the integration means for specifying the label for the data to be identified by the first identification means. Equipped with
The learning method is
The model was retrained using the teacher data including the label identified by the integration means and the data.
The model transmission method is
When the identification result of the data derived by the first identification means using the retrained model matches the label for the data identified by the integration means, the retrained model is used. The identification system according to claim 1, which is transmitted to the second identification system. The model transmission method is
When the learning means relearns the model, the model is used when the correct answer rate of the discrimination result derived by the first discrimination means using the model is equal to or more than a predetermined threshold value within a predetermined time. The identification system according to claim 1, which is transmitted to the second identification system. A model for identifying an object represented by the data is learned using the teacher data, and the model is stored in the first model storage means.
Using the model stored in the first model storage means, a first identification process for identifying an object represented by data is executed.
Each of the individual models trained by the plurality of predetermined identification systems is stored in the second model storage means.
When the model learned by the first identification system is received from the first identification system, the model learned by the first identification system stored in the second model storage means is referred to as described above. Update to the model received from the first identification system,
In a predetermined case, for each model stored in the second model storage means, a second identification process for identifying an object represented by the data as an identification target in the first identification process is executed. ,
The model is relearned using the teacher data including the label for the data determined based on the identification result derived in the second identification process and the data, and is stored in the first model storage means. Update the model to the retrained model and
A model providing method comprising transmitting a model stored in the first model storage means to one or a plurality of predetermined second identification systems. On the computer
A learning process in which a model for identifying an object represented by data is learned using teacher data, and the model is stored in a first model storage means.
A first identification process for identifying an object represented by data by using the model stored in the first model storage means.
A process of storing individual models learned by a plurality of predetermined identification systems in a second model storage means.
When the model learned by the first identification system is received from the first identification system, the model learned by the first identification system stored in the second model storage means is referred to as described above. Model update process to update to the model received from the first identification system,
A second identification process for identifying an object represented by the data as an identification target in the first identification process for each model stored in the second model storage means in a predetermined case.
The model is relearned using the teacher data including the label for the data determined based on the identification result derived in the second identification process and the data, and is stored in the first model storage means. Retraining process to update the model to the retrained model, and
A model providing program for executing a model transmission process for transmitting a model stored in the first model storage means to one or a plurality of predetermined second identification systems.

Images