JP7018408B2 - Image search device and teacher data extraction method - Google Patents

Description

The present invention relates to an image search device that performs image search using machine learning technology and a teacher data extraction method.

In recent years, the utilization of machine learning technology represented by deep learning has been progressing in the technical fields such as image recognition and image classification. In image recognition and image classification by machine learning, the machine learning model itself can learn and classify based on the input data, instead of the developer designing and programming the algorithm as in the past. Specifically, when multiple pairs of image data and its correct classification result (correct answer tag) are input, the machine learning model itself can output the classification result correctly for the input image data in the machine learning model. Adjust the parameters of. The data of the pair of the input image data and the classification result is called teacher data (learning data), and the machine learning using the teacher data is called supervised machine learning.

In supervised machine learning, a large amount of teacher data is input to the machine learning model and trained to improve the accuracy of the machine learning model. In general, the more teacher data used for learning, the better the accuracy of the machine learning model and the more accurate classification results can be obtained.
However, since it is necessary to prepare a large amount of teacher data in supervised machine learning, the preparation work of teacher data is to collect image data and associate the correct classification result (correct answer tag) with each collected image data. There is a problem that it becomes enormous and costly. In addition, it is difficult to obtain a 100% correct answer rate even for a machine learning model that has started actual operation once, so it is desirable to perform additional learning to improve accuracy. However, in the day-to-day operation of the system, collecting teacher data for additional learning and adding correct answer tags have problems such as an increase in system downtime and an increase in maintenance cost in daily work.

In the additional learning of Patent Document 1, the first feature vector, which is the feature vector of the image data with the correct answer label to which the correct answer label (classification result, the correct answer tag) prepared in advance, and the image with the correct answer label are attached. The distance information from the second feature vector, which is the feature vector of the unlabeled image (unlabeled image) generated by division, is obtained. Based on the distance information, unlabeled image data to be presented to the user is selected, labeled by the user, and used for additional learning as a second correct labeled image.

By doing so, it becomes possible to generate a new learning image from the first image with the correct answer label, which has different contents from the original image with the correct answer label. For example, an unlabeled image that is least similar to the original first correct labeled image can be selected as the image to be presented to the user. As a result, it is described that it is possible to increase the improvement rate of the accuracy of the classification model per image for learning and improve the learning efficiency.

Japanese Unexamined Patent Publication No. 2013-125322

The target of additional learning varies depending on the user's environment. Patent Document 1 does not mention a technique for efficiently collecting necessary data from an image database in which a large amount of images are stored in order to perform appropriate machine learning suitable for the operating environment of each machine learning model. .. Further, even if it is possible to collect images for learning, the problem that classification (correct tagging) is required and cost is not solved has not been solved.

The present invention has been made in view of such a background, and provides an image search device and a teacher data extraction method that can reduce the cost of collecting and creating teacher data and facilitate additional learning. Is the subject.

In order to solve the above problems, the image search device of the present invention has a feature extraction unit that extracts a feature amount from an acquired image using a machine learning model, and a feature extraction unit that searches the image using the feature amount and searches for a search result. Based on the correlation between the feature amount and the classification item, the output image search unit, the classification item acquisition unit that classifies the individual images of the search result and acquires the classification item indicating the given classification result, and the classification item acquisition unit. , A teacher data extraction unit for extracting an image to be teacher data for additional learning of the machine learning model.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to provide an image search device and a teacher data extraction method that can reduce the cost of collecting and creating teacher data and facilitate additional learning.

It is a figure which shows the whole structure of the image search system including the image search apparatus which concerns on this embodiment. It is a block diagram of the image search screen displayed on the terminal of the image search apparatus which concerns on this embodiment. It is a block diagram of the functional block of the image search apparatus which concerns on this embodiment. It is a figure for demonstrating the operation of the image acquisition part, the person detection part, and the feature extraction part which concerns on this embodiment. It is a figure for demonstrating the data structure of the feature amount table included in the feature amount database which concerns on this embodiment. It is a figure which shows the data structure of the search condition table included in the search result database which concerns on this embodiment. It is a figure which shows the data structure of the search result table included in the search result database which concerns on this embodiment. It is a figure which shows the data structure of the teacher data extraction condition table which concerns on this embodiment. It is a graph for demonstrating the classification item of the extraction target data, the threshold value A, the threshold value B, and the necessity of additional learning according to this embodiment. It is a figure which shows the data structure of the teacher data extraction result table which concerns on this embodiment. It is a flowchart of the teacher data extraction process executed by the teacher data extraction unit which concerns on this embodiment. It is a block diagram of the teacher data extraction condition setting screen displayed on the terminal of the image search apparatus which concerns on the modification of this embodiment. It is a graph which showed the correlation between the feature amount and the classification item which concerns on the modified example of this embodiment. It is a graph which showed the correlation between the feature amount and the classification item which concerns on the modified example of this embodiment.

Hereinafter, an image search device according to an embodiment (embodiment) for carrying out the present invention will be described. More specifically, an image search device in a system for searching an image of a person who meets a predetermined condition from an image taken by a camera installed in a facility such as a shopping mall or an office building will be described. The image search device extracts image data to be teacher data for additional learning from the accumulated image data. In addition, this system is supposed to be used for searching for lost children in the facility, and for searching for the person who was involved in the problem that occurred in the facility based on the information obtained by reporting. , Not limited to these applications, it can be applied to a wide range of applications.

≪Overall configuration of image search system≫
FIG. 1 is a diagram showing an overall configuration of an image search system 10 including an image search device 100 according to the present embodiment. The image search system 10 includes an image search device 100, a camera 310, a network 330 that transfers images captured by the camera 310 to the image search device 100, an additional learning device 300, and a terminal 320.

The camera 310 is a camera installed in the facility, and transmits images taken via the network 330 to the image search device 100. The terminal 320 is a terminal used by a person (hereinafter, also referred to as a user) who monitors the inside of the facility using the image search system 10. The operation method and the display screen of the user's terminal 320 will be described with reference to FIG. 2 described later. The image search device 100 extracts a person who matches the characteristics (search conditions) of the person input from the terminal 320 from the image of the camera 310 and outputs the person to the terminal 320.
The person who monitors the inside of the facility using the image search system 10 is described as a user, whereas the machine learning model 114 (see FIG. 3 described later) of the image search device 100 is maintained to manage additional learning. Write a person as an administrator. The administrator may also use the terminal 320.

The additional learning device 300 executes additional learning to generate a new machine learning model 420 that replaces the machine learning model 114 used when extracting the characteristics of a person. The additional learning device 300 acquires additional learning data 410 (teacher data) from the image search device 100, executes additional learning, and generates a new machine learning model 420.

≪Image search screen of image search device≫
FIG. 2 is a configuration diagram of an image search screen 500 displayed on the terminal 320 of the image search device 100 according to the present embodiment. In the present embodiment, the user operates the image search device 100 using a Web browser. The image search screen 500 includes a search set area 510, a search condition area 520, and a search result area 530.

The user sets the characteristics (search conditions) of the person to be searched (the person to be searched) in the search condition area 520. Specifically, the color of the head of the person (head color) is selected from the list box 521 and set. In this embodiment, the head color is selected from black, gray and white. Similarly, the color of the upper body clothing and the color of the lower body clothing are selected from black, gray and blue. When the condition clear button 522 is pressed, the selected head color, upper body clothing color, and lower body clothing color are cleared, and nothing is selected. When the search button 523 is pressed, an image of a person who meets the set conditions is searched and displayed in the search result area 530.

The search identification information 511 of the search set area 510 is the identification information assigned to the search conditions and the search results, and is automatically allocated by the image search device 100. When the save button 512 is pressed, the search conditions set in the search condition area 520 and the search results displayed in the search result area 530 are saved in the image search device 100. When the user inputs the search identification information 511 and presses the load button 513, the search conditions stored corresponding to the search identification information 511 are displayed in the search condition area 520, and the search results are displayed in the search result area 530. To.

The search result area 530 is an area in which the search result 531 is displayed in descending order of the degree of matching with the search condition (also referred to as the average similarity). In FIG. 2, six search results are displayed. The individual search result 531 consists of three areas: an image confirmation area 532, an average similarity bar 533, and a classification item setting area 534.
In the image confirmation area 532, a sample image of a video including a person is displayed. The image identification information allocated by the image search device 100 is displayed on the sample image. In addition, the area where a person is detected is displayed surrounded by a rectangle. The image confirmation area 532 may include the shooting time, the camera that shot the image, and the identification information of the area where the camera is installed.
The average similarity bar 533 displays the average value of the similarity (feature amount described later) of the head color, the color of the upper body clothes, and the color of the lower body clothes in the form of a bar graph. The more the bar (hatched rectangle) extends to the right (the larger the area), the higher the average similarity, 0 at the left end and 100 at the right end.

The classification item setting area 534 is a result of the user determining whether or not a person surrounded by a rectangle in the image (hereinafter, also referred to as a person in the image, or simply referred to as an image) corresponds to a person to be searched (hereinafter, classification). This is the area for inputting items). If the user determines that the image is the person to be searched, he / she selects the corresponding person. If the user determines that the image is not the person to be searched, he / she selects non-applicability. If the user withholds the judgment, he / she selects the withholding. If none of applicable, pending, and non-applicable is selected, the user does not select anything and is uncategorized.

The meaning of the hold is assumed to be usage such as the image is unclear and cannot be judged, it cannot be specified whether it is the person to be searched, the judgment is held until it is visually confirmed, and the user wants to mark the judgment result so as not to be forgotten. is doing.
In this embodiment, the terms applicable and non-applicable are used to explain, but in the sense that the search target is applicable or non-applicable, the target person / non-target person, confirmation / confirmation not required, caution / attention not required, etc. Other terms may be used.

In the display filter 536, it is possible to set whether or not to display the applicable, pending, non-applicable, and uncategorized search results in the search result area 530. In FIG. 2, applicable, pending, non-applicable, and uncategorized are all selected, and all search results are displayed in the search result area 530. The user can classify the image into applicable, pending, non-applicable, and unclassified classification items, and select which classification item to display by the display filter. The user can reduce the number of displayed search results or display only the images to be confirmed, which makes it easy to compare people in detail between the images, and to make a judgment or determine the person to be searched.

The image identification information of the search result in the upper left of the search result area 530 is "I3483", the average similarity is about 90% as shown in the average similarity bar, and the user has selected the corresponding. .. The image in the image confirmation area contains two people, but the person that is the search result is the person surrounded by the rectangle on the right side.
A scroll bar 535 is provided on the right side of the search result area 530. The user can browse the search result 531 that is not displayed by operating the knob (knob) of the scroll bar 535 and the arrows (arrows) at the top and bottom.

≪Overall configuration of image search device≫
FIG. 3 is a block diagram of a functional block of the image search device 100 according to the present embodiment. The image search device 100 is a computer, and operates a CPU (Central Processing Unit) that operates as a functional unit such as a feature extraction unit 113 and an image search unit 115 described later, a database, temporary processing data, and a CPU as functional units. It includes a storage unit (hard disk, SSD (Solid State Drive), etc.) for storing programs, a communication unit, and the like, but is not shown.

≪Image capture part, person detection part, feature extraction part≫
The image capture unit 111 acquires the image captured by the camera 310 and outputs each frame image of the image to the person detection unit 112.
The person detection unit 112 detects a person in a frame image (simply referred to as an image or image data), and together with the detected area, shooting time information, shooting camera identification information, and image identification information, an image database (FIG. 3). Then, it is stored in the image DB (Database)) 120. Further, the person detection unit 112 outputs an image of the region where the person is detected to the feature extraction unit 113 together with the image identification information.
There is Faster-RCNN as an existing person detection technology, and a person detection algorithm using DNN (Deep Neural Networks) is described in US9858496 (B2) and the like.

The feature extraction unit 113 extracts a plurality of feature quantities from an image of a person's region using a machine learning model 114 and stores them in a feature quantity database 130 (see FIG. 5 described later). A DNN that extracts a plurality of feature quantities is called a multi-label type DNN. For example, Japanese Patent Publication No. 2018-503161 discloses a DNN technology that analyzes a plurality of feature quantities and outputs the detection result. .. The machine learning model 114 is not limited to DNN, and may be a machine learning model of another machine learning technique such as SVM (Support Vector Machine).

FIG. 4 is a diagram for explaining the operation of the image acquisition unit 111, the person detection unit 112, and the feature extraction unit 113 according to the present embodiment. With reference to FIG. 4, the data output by the image acquisition unit 111, the person detection unit 112, and the feature extraction unit 113 will be supplementarily described.
The image 431 is a frame image of the image output by the image capturing unit 111, and includes three people. The images 432 to 434 are images of a region in which a person is detected in the image 431 output by the person detection unit 112, and are images obtained by cutting out each of the three people included in the image 431.

The feature amount data 435 is the data output by the feature extraction unit 113, and is the data obtained by analyzing the feature amounts of the images 432 to 434 by the machine learning model 114. The machine learning model 114 is a multi-label DNN that extracts a plurality of features. The features are black in the head color, gray in the head color, white in the head color, black in the upper body clothes, and gray in the upper body clothes. There are nine degrees: the color of the upper body clothes is blue, the color of the lower body clothes is black, the color of the lower body clothes is gray, and the color of the lower body clothes is blue. Yes, it is represented by a 9-dimensional vector. Each feature amount is normalized and output so that the maximum is 100 and the minimum is 0. For example, the degree (feature amount) that the head color of the image 432 whose image identification information is "I0014" is black is 80.

≪Feature database≫
FIG. 5 is a diagram for explaining the data structure of the feature amount table 131 included in the feature amount database 130 according to the present embodiment. The feature amount database 130 is composed of one or more feature amount tables 131. The feature table 131 includes feature table identification information 132, machine learning model version 133, and one or more image records. The feature amount table identification information 132 is the identification information of the feature amount table 131. The machine learning model version 133 is a version of the machine learning model 114 when the feature amount included in the feature amount table 131 is calculated.

The image record has image identification information (described as an image ID in FIG. 5) 134, a feature amount 135 to the extent that the head color is black, a feature amount 136 to the degree that the head color is gray, and a white head color. The characteristic amount of a certain degree is 137, the characteristic amount of the degree that the color of the upper body clothes is black 138, the characteristic amount of the degree that the color of the upper body clothes is gray 139, and the characteristic amount of the degree that the color of the upper body clothes is blue. 140, the characteristic amount 141 of the degree that the color of the lower body clothes is black, the characteristic amount 142 of the degree that the color of the lower body clothes is gray, and the characteristic amount 143 of the degree that the color of the lower body clothes is blue. ..

The feature amount table 131 exists for each machine learning model 114. That is, when the machine learning model 114 is updated to the new machine learning model 420 (see FIG. 1), it is switched to the new feature amount table. Therefore, the feature amount table identification information 132 and the machine learning model version 133 have a one-to-one correspondence. Image records are added to the same feature amount table 131 during the period when the feature extraction unit 113 is extracting the feature amount using the same machine learning model 114.
The feature amount table 131 may be divided by day, by camera, or by facility area. In this case, the feature amount table identification information 132 and the machine learning model version 133 have an N to 1 correspondence.

≪Image search section: Operation during search≫
Returning to the description of FIG. The Web server 116 receives an instruction from the user of the terminal 320, and transmits the result of the instructed process to the terminal 320. When the user sets the search condition in the search condition area 520 (see FIG. 2) and presses the search button 523 to instruct the search, the Web server 116 receives the search condition and outputs the search condition to the image search unit 115. The image search unit 115 stores the search conditions in the search condition table 151 (see FIG. 6 described later) of the search result database 150.

Subsequently, the image search unit 115 searches the image records in the feature amount table 131 for records in which the average value (mean similarity) of the feature amounts included in the specified search conditions is equal to or greater than a predetermined value. , It is stored in the search result table 164 of the search result database 150 (see FIG. 7 described later). Subsequently, the image search unit 115 sorts the search results in descending order of average similarity. Further, the image search unit 115 acquires the image data corresponding to each record of the sorted result from the image database 120, and searches for the data combined with the image data, the average value, and the display data of the classification item setting area 534. It is output to the Web server 116 as a search result for the condition. The Web server 116 transmits the search result to the terminal 320. The Web browser of the terminal 320 displays the received search result in the search result area 530 (see FIG. 2).

≪Search result database: Search condition table≫
FIG. 6 is a diagram showing a data structure of the search condition table 151 included in the search result database 150 according to the present embodiment. The search result database 150 includes a search condition table 151 and a search result table 164 (see FIG. 7 described later).

The search condition table 151 is, for example, tabular data, one record (row) indicates one search condition, search identification information (described as a search ID in FIG. 6) 152, and the head of the search condition. Flags 153 to 155 indicating whether the color includes black, gray, and white, flags 156 to 158 indicating whether the color of the upper body clothes include black, gray, and blue in the search conditions, and the lower body in the search conditions. The attributes of flags 159 to 161 indicating whether or not the color of clothes includes black, gray, and blue, feature amount table identification information 162, and machine learning model version 163 (described as MLM (Machine Learning Model) version in FIG. 6) are set. include.

Flags 153 to 161 are "1" if they are included in the search condition, and "0" if they are not included. For example, the record in which the search identification information 152 is "S018" indicates a search condition in which the head color is gray, the upper body clothes color is blue, and the lower body clothes color is black. The feature amount table identification information 162 and the machine learning model version 163 correspond to the feature amount table identification information 132 and the machine learning model version 133 of the feature amount table 131 (see FIG. 5), respectively, and the feature amount table 131 to be searched and the feature amount table 131 and the machine learning model version 133. The version of the machine learning model 114 when the feature quantity is calculated is shown.

≪Search result database: Search result table≫
FIG. 7 is a diagram showing a data structure of the search result table 164 included in the search result database 150 according to the present embodiment. The search result table 164 is, for example, tabular data, and is composed of search result records which are search results. The search result record includes the search identification information 165 corresponding to the search identification information 152 of the search condition table 151, the image identification information 166 corresponding to the image identification information 134 of the feature amount table 131, and the feature amounts 135 to 143 of the feature amount table 131. It includes the attributes of the machine learning model version 178 corresponding to the corresponding feature quantities 167 to 175, the average similarity 176, the classification item 177, and the machine learning model version 163 of the search condition table 151, respectively.

The classification item 177 is an attribute for storing the classification item (see the classification item setting area 534 in FIG. 2) for each search result set by the user, and is "1" if the classification item is applicable and "1" if the classification item is on hold. 2 ”,“ 3 ”if not applicable, and“ 0 ”if not classified. In the initial state before the user sets the classification item, the classification item 177 is set to "0" because the classification item is not set.

The search result record in which the image identification information 166 is "I0014" in the search result table 164 will be described. The search identification information 165 of this search result record is "S018", and indicates that the search identification information 152 in the search condition table 151 is the result of being searched by the search condition which is a record of "S018". There is. The feature amounts 167 to 175 correspond to the feature amounts 135 to 143 of the record in which the image identification information 134 in the feature amount table 131 is "I0014", respectively.
The search condition for the record in which the search identification information 152 is "S018" is that the head color is gray, the upper body clothes color is blue, and the lower body clothes color is black. The corresponding feature quantities are 20, 30, and 30, respectively, and the average similarity 176 is 27.

≪Category item registration unit: Operation when searching and saving≫
Returning to the description of FIG. When the user inputs a classification item in the classification item setting area 534 (see FIG. 2) and presses the save button 512, the Web browser of the terminal 320 displays the classification item (applicable, hold, non-applicable, or applicable) for each search result. (Uncategorized) is transmitted to the Web server 116, and the Web server 116 outputs the classified items to the classified item registration unit 117. The classification item registration unit 117 stores the classification item, which is the determination result by the user corresponding to the search result, in the classification item 177 of the search result table 164 of the search result database 150.

≪Image search section: Operation when loading search results≫
When the user inputs the search identification information 511 (see FIG. 2) and presses the load button 513, the Web browser of the terminal 320 transmits the search identification information to the Web server 116, and the Web server 116 transmits the search identification information. Output to the image search unit 115. The image search unit 115 acquires a record of the search condition in which the search identification information 152 matches the search identification information output by the Web server 116 in the records of the search condition table 151 of the search result database 150.

Further, the image search unit 115 acquires a search result record in which the search identification information 165 matches the search identification information output by the Web server 116 in the search records of the search result table 164 of the search result database 150. The image search unit 115 outputs the search condition record and the search result record to the Web server 116, and the Web server 116 transmits the record to the terminal 320. The Web browser of the terminal 320 displays the received search condition record in the search condition area 520 (see FIG. 2) and the search result record in the search result area 530.

≪Teacher data extraction result database≫
Before explaining the operation of the teacher data extraction unit 118, the teacher data extraction result database 180 will be described. The teacher data extraction result database 180 includes a teacher data extraction condition table 181 (see FIG. 8 described later) and a teacher data extraction result table 191 (see FIG. 10 described later).

≪Teacher data extraction result database: Teacher data extraction condition table≫
FIG. 8 is a diagram showing a data structure of the teacher data extraction condition table 181 according to the present embodiment. The teacher data extraction condition table 181 is, for example, tabular data, one record (row) indicates one extraction condition, extraction condition identification information 182, feature amount 183, positive / non-flag 184, threshold A185, Includes threshold B186, additional learning requirement 187, number of collected data 188 and machine learning model version 189. The extraction condition is a search condition when searching for a search result record corresponding to the image data in the search result table 164 (see FIG. 7). The machine learning model version 189 shows the machine learning model version 178 of the search result record of the search result table 164 to be extracted.

The extraction condition identification information 182 is the identification information of the extraction condition.
The feature amount 183 indicates a feature amount that is a key to the extraction conditions, and is a feature amount in which the head color is black, gray, and white, a feature amount in which the color of the upper body clothes is black, gray, and blue, and a feature amount of the lower body clothes. It is one of the feature quantities whose color is black, gray, or blue.
The correct / non-flag 184 is a correct answer, an incorrect answer, or both. The correct answer is the data that regards the search result in which the classification item 177 is set as the correct answer, and the incorrect answer is the data in which the search result in which the classification item 177 is set as the non-applicable is regarded as the incorrect answer. Refers to both correct and incorrect data.

The additional learning necessity 187 indicates whether the additional learning extracts the required data, the additional learning extracts the negative data, or the additional learning extracts both the required and negative data. Additional learning is required when the judgment result (classification item) of the user is applicable and the value of the feature amount shown in the feature amount 183 of the extraction target data is equal to or less than the threshold value B186, or the judgment result of the user. Is not applicable, and the value of the feature amount shown in the feature amount 183 of the extraction target data is the threshold value A185 or more. Further, "No additional learning" means that the judgment result of the user is applicable and the value of the feature amount shown in the feature amount 183 of the extraction target data is the threshold value A185 or more, or the judgment result of the user is not. This is the case where the value of the feature amount shown in the feature amount 183 of the extraction target data is the threshold value B186 or less. If the feature amount of the extraction target data is between the threshold value A185 and the threshold value B186, or if the classification item is reserved or unclassified, it is not necessary or not.
The necessity of additional learning 187 will be described again with reference to FIG. 9 described later.

The number of collected data 188 is the number of search result records in the search result table 164 (see FIG. 7) that match the extraction conditions, which is desirable as teacher data for additional learning.
In the record in which the extraction condition identification information 182 is "SC01", the key feature amount 183 is "head color is black", the positive / negative flag 184 is both a correct answer and an incorrect answer, and the threshold value A185 is 80. , The threshold value B186 is set to 30, and the extraction conditions that require additional learning are shown.

FIG. 9 is a graph 450 for explaining the positive / negative flags 184, the threshold value A185, the threshold value B186, and the necessity of additional learning 187 of the extraction target data (search result record) according to the present embodiment. The vertical axis of the graph 450 is the feature amount, and the horizontal axis is the classification item. The dotted horizontal line 455 is the threshold value shown in the threshold value A185, and the dotted horizontal line 456 is the threshold value shown in the threshold value B186.

The extraction target data that is determined to be applicable by the user and whose feature amount is the threshold value A185 or more is plotted in the area 451. The extraction target data that is determined to be applicable by the user and whose feature amount is the threshold value B186 or less is plotted in the area 452. The extraction target data that is determined to be non-applicable by the user and whose feature amount is the threshold value A185 or more is plotted in the area 454. The extraction target data that is determined to be non-applicable by the user and whose feature amount is the threshold value B186 or less is plotted in the area 453.

The extraction target data that requires additional learning is an extraction target plotted in region 452 or region 454 in which the feature amount and the classification item (applicable or not applicable) are different (there is an inverse correlation between the feature amount and the classification item). It is data. The extraction target data for which additional learning is rejected is the extraction target data plotted in the region 451 or the region 453 in which the feature amount and the classification item match (there is a correlation between the feature amount and the classification item).

The extraction target data for which the correct / non-flag is the correct answer is the data to which the classification item corresponds, and is the extraction target data plotted between the area 451 or the area 452, or between the area 451 and the area 452. The extraction target data that is an incorrect answer is data for which the classification item is not applicable, and is the extraction target data plotted between the area 453, the area 454, or the area 453 and the area 454.

≪Teacher data extraction result database: Teacher data extraction result table≫
FIG. 10 is a diagram showing a data structure of the teacher data extraction result table 191 according to the present embodiment. The teacher data extraction result table 191 is, for example, tabular data, and one record (row) is a search result table 164 (FIG.) extracted according to any of the extraction conditions of the teacher data extraction condition table 181. 7) Search result record (image data) is shown, teacher data identification information 192, feature amount 193, classification item 194, additional learning required 195, image identification information 196, similarity 197 and machine learning model version 198 attributes. including.

The teacher data identification information 192 is identification information of image data extracted as teacher data.
The feature amount 193 corresponds to the feature amount 183 in the search condition of the teacher data extraction condition table 181 and indicates which feature amount is the key.
The classification item 194 has a classification item which is a judgment result of the user with respect to the image data extracted as the teacher data, and is applicable or not applicable.

The necessity of additional learning 195 indicates the necessity of additional learning for the image data extracted as the teacher data.
The image identification information 196 is the identification information of the image data extracted as the teacher data, and corresponds to the image identification information 166 (see FIG. 7).
The similarity 197 is a value of the feature amount shown in the feature amount 193 of the image data.
Machine learning model version 198 shows machine learning model version 178 of image data extracted as teacher data.

The records in which the teacher data identification information 192 is "LD01" and "LD02" are the data in which the extraction condition identification information matches the extraction condition of "SC01". In the image data shown in the record in which the teacher data identification information 192 is "LD01", it is determined that the feature amount of the degree to which the head color is black is 15 and the feature amount value is the threshold value B186 or less. On the other hand, it is image data that is judged to be applicable by the user and requires additional learning. The image data shown in the record in which the teacher data identification information 192 is "LD02" is determined to have a feature amount of 85 to the extent that the head color is black and a feature amount value of the threshold value A185 or more. On the other hand, it is image data that is judged not to be applicable by the user and requires additional learning.

≪Teacher data extraction department≫
Returning to the description of FIG. When the administrator instructs the terminal 320 to extract the teacher data, the Web server 116 notifies the teacher data extraction unit 118. The teacher data extraction unit 118 extracts a search result record that matches any of the search conditions in the teacher data extraction condition table 181 from the search result records (image data) of the search result table 164 (see FIG. 7). , The teacher data extraction result table 191 is generated. The extracted teacher data is stored in the image database 120 as additional learning data and is transmitted to the additional learning device 300 together with the image data corresponding to the image identification information 196 (see FIG. 1). The details of the extraction process will be described with reference to FIG. 11 described later.

≪Machine learning model update department≫
Returning to the description of FIG. The machine learning model update unit 119 receives the new machine learning model 420 (see FIG. 1) generated by the additional learning device 300, and replaces the machine learning model 114 of the feature extraction unit 113 (the machine learning model 114 is replaced with the new machine learning model 420). Update with).

≪Teacher data extraction process≫
FIG. 11 is a flowchart of the teacher data extraction process executed by the teacher data extraction unit 118 according to the present embodiment. The details of the teacher data extraction process will be described with reference to FIG.
In step S101, the teacher data extraction unit 118 initializes the teacher data extraction result table 191 and sets the number of records to 0.
In step S102, the teacher data extraction unit 118 repeats steps S103 to S109 for the extraction conditions indicated by each record in the teacher data extraction condition table 181. In the following, the extraction condition indicated by the record selected in step S102 will be referred to as the present extraction condition.

In step S103, the teacher data extraction unit 118 repeats steps S104 to S108 for each search result record (image data) included in the latest search result table 164 (see FIG. 7). In the following, the image data indicated by the search result record selected in step S103 will be referred to as extraction target data.

In step S104, the teacher data extraction unit 118 determines whether the machine learning model version 178 of the data to be extracted matches the machine learning model version 189 of the present extraction condition, and if so, proceeds to step S105 (step S104 → Y). If they do not match (step S104 → N), the process proceeds to step S109.

In step S105, the teacher data extraction unit 118 determines whether the extraction target data matches the positive / negative flag 184 of the present extraction condition, and if it matches (step S105 → Y), proceeds to step S106, and if it does not match (step S105). → N) Proceed to step S109.
When the correct / non-flag 184 matches with the correct answer, it means that the classification item 177 (see FIG. 7) of the extraction target data is applicable. If the correct / non-flag 184 matches with an incorrect answer, it means that the classification item 177 of the extraction target data is not applicable. When the positive / negative flags 184 match in both cases, it means that the classification item 177 of the extraction target data is applicable or not applicable.

In step S106, the teacher data extraction unit 118 determines the necessity of additional determination from the correlation between the characteristics of the extraction target data and the classification items. Specifically, if the value of the feature amount is the threshold value A185 or more and the classification item is applicable, the teacher data extraction unit 118 determines that there is a correlation and that additional learning is unnecessary (additional learning is not necessary). If the value of the feature amount is the threshold value A185 or more and the classification item is not applicable, the teacher data extraction unit 118 determines that there is an inverse correlation and additional learning is required. If the value of the feature amount is equal to or less than the threshold value B186 and the classification item is applicable, the teacher data extraction unit 118 determines that there is an inverse correlation and additional learning is required. If the value of the feature amount is equal to or less than the threshold value B186 and the classification item is not applicable, the teacher data extraction unit 118 determines that there is a correlation and no additional learning is required.

In step S107, the teacher data extraction unit 118 determines whether the extraction target data matches the additional learning necessity 187 of the present extraction condition, and if it matches, proceeds to step S108 (step S107 → Y), and if it does not match (step S107). S107 → N) Proceed to step S109. The fact that the extraction target data matches the additional learning necessity 187 of the present extraction condition means that the necessity (necessary or unnecessary) of the additional learning determined in step S106 is the condition (necessary, no need) shown in the additional learning necessity 187 of the present extraction condition. , Both (either necessary or not) are met.

In step S108, the teacher data extraction unit 118 adds the extraction target data to the teacher data extraction result table 191. To add, a record is added to the teacher data extraction result table 191, new identification information is stored in the teacher data identification information 192, and the feature amount 183 of this extraction condition is stored in the feature amount 193, and the classification item. The classification item 177 of the extraction target data is stored in 194, the determination result of the addition necessity determined in step S106 is stored in the additional learning necessity 195, and the image identification information 166 of the extraction target data is stored in the image identification information 196. Is stored, the feature amount of the extraction target data corresponding to the feature amount 183 of this extraction condition is stored in the similarity 197, and the machine learning model version 178 of the extraction target data is stored in the machine learning model version 198. ..

In step S109, the teacher data extraction unit 118 proceeds to step S110 if steps S104 to S108 are executed for all the search result records included in the latest search result table 164. If there is a remainder, steps S104 to S108 are executed with the next search result record as the extraction target data.
In step S110, the teacher data extraction unit 118 ends the teacher data extraction process when steps S103 to S109 are executed for all the extraction conditions included in the teacher data extraction condition table 181. If there is a remainder, steps S103 to S109 are executed with the next extraction condition as the main extraction condition.

≪Characteristics of teacher data extraction processing≫
The image data to be extracted from the teacher data is data obtained by adding the result (classification item) of determining whether or not the user corresponds to the person to be searched to the result of searching by setting the conditions of the person to be searched. Is. Since the classification items have already been added, it can be used as teacher data as teacher data without adding a correct answer tag again. For example, the data in which the teacher data identification information 192 shown in FIG. 10 is “LD01” can be used for additional learning as the data corresponding to the feature amount 193 that the head color is black. It is not necessary to collect data or add a new correct answer tag as in the prior art, and it is possible to reduce the cost of collecting teacher data and attaching a correct answer tag.

The extracted teacher data is output to the outside of the image search device 100. By streamlining the work of assigning correct answer tags in this way, it is possible to shorten the period of additional learning. In addition, additional learning may be executed inside the image search device 100.

Additional learning of deep learning has already been disclosed in many literatures, and various learning is possible with machine learning frameworks such as Caffe. The effects of this embodiment are not limited to a specific framework. In particular, when it is desired to intensively perform additional learning of features that are highly necessary for additional learning, teacher data is extracted from the collected data with additional learning required or not 187 required (features and classification items are inversely correlated). It is possible to efficiently proceed with additional learning by learning. Further, when it is desired to improve the accuracy by general learning, additional learning can be efficiently advanced by extracting and learning the teacher data with both the positive and negative flags 184 from the collected data.

<< Modification 1: Machine learning model version >>
The image search device 100 refers to the machine learning model versions 178 and 198 assigned to the search result table 164 (see FIG. 7) and the teacher data extraction result table 191 (see FIG. 10), and refers to the old version of the machine learning model 114. A warning may be sent to a user who loads and refers to the search result table 164 including the feature amount according to the above, or an administrator who extracts teacher data. This prevents the extraction of teacher data from different versions of search result records and enables consistent additional learning.

<< Modification 2: Teacher data extraction conditions >>
In the above-described embodiment, the extraction conditions stored in the teacher data extraction condition table 181 (see FIG. 8) have been described as set conditions. The administrator of the image search device 100 may be able to change the extraction conditions. By doing so, for example, instead of extracting teacher data for all features, additional learning is focused on features that are considered to have low accuracy (for example, head color and upper body clothes are gray). Is possible. Alternatively, the number of teacher data can be adjusted by adjusting the positive / non-flag 184, the threshold value A185, and the threshold value B186.

FIG. 12 is a configuration diagram of a teacher data extraction condition setting screen 600 displayed on the terminal 320 of the image search device 100 according to the modified example of the present embodiment. The teacher data extraction condition setting screen 600 includes an extraction set area 610 and an extraction condition area 620.
The administrator sets the content to be reflected in the teacher data extraction condition table 181 (see FIG. 8) in the extraction condition area 620. Specifically, the values set in the feature amount 183, the positive / negative flag 184, the threshold value A185, the threshold value B186, and the additional learning necessity 187 of the teacher data extraction condition table 181 are set in the list box 621, the list box 622, the text box 623, and the text, respectively. Set to box 624 and list box 625. For example, the correct / non-flag 184 is selected from "correct answer", "incorrect answer", and "both" in the list box 622.

The extraction condition identification information 611 of the extraction set area 610 corresponds to the extraction condition identification information 182 of the teacher data extraction condition table 181. When the administrator inputs the extraction condition identification information 611 and presses the load button 613, the extraction condition identification information 182 in the record of the teacher data extraction condition table 181 matches the extraction condition identification information 611 input by the administrator. The contents of the record are displayed in the extraction condition area 620.

When the administrator inputs the extraction condition identification information 611 and presses the save button 612, the extraction condition identification information 182 in the record of the teacher data extraction condition table 181 matches the extraction condition identification information 611 input by the administrator. The contents set in the extraction condition area 620 are reflected in the record. If there is no record that matches the extraction condition identification information 611 input by the administrator, the record is added to the teacher data extraction condition table 181 and the contents set in the extraction condition area 620 are reflected.

≪Variation example 3: Necessity of additional learning≫
By visualizing the correlation between the feature amount and the classification item, it is possible to determine which feature amount requires additional learning. 13 and 14 are graphs 470 and 480 showing the correlation between the feature amount and the classification item according to the modified example of the present embodiment. Graph 470 is a graph in which search result records (see FIG. 7) are plotted under a search condition in which a certain feature amount, for example, the color of the upper body clothes is blue. Further, the graph 480 is a graph in which search result records are plotted under a search condition that a certain feature amount, for example, the color of the upper body clothes is black. “R = .8” and “r = −.8” in the upper right of the graph indicate the correlation coefficient.

In graph 470, there are many search result records plotted in the area 471 where the classification item is applicable and the feature amount is large, and the area 472 where the classification item is not applicable and the feature amount is small, and there is a correlation between the feature amount and the classification item. Therefore, it can be said that the need for additional learning about the feature amount that "the color of the clothes of the upper body is blue" is relatively low.
On the other hand, in Graph 480, there are many search result records plotted in the area 481 where the classification item is applicable and the feature amount is small, and the area 482 where the classification item is not applicable and the feature amount is large, and the inverse correlation between the feature amount and the classification item is large. be. Therefore, it can be said that the need for additional learning about the feature amount that "the color of the upper body clothes is black" is relatively high.

The image search device 100 may include a graph generation unit that presents a graph showing the correlation between the feature amount and the classification item as shown in the graphs 470 and 480 to the administrator. The administrator can preferentially perform additional learning for features with low correlation (inverse correlation (negative correlation) or weak correlation) on the graph.
Further, the graph generation unit may present the graph to the administrator and at the same time calculate and display the correlation coefficient. By doing so, it becomes easier for the administrator to make a decision on additional learning.
Further, in addition to the correlation coefficient, the graph generation unit may calculate and display an average value and a standard deviation for the feature quantities of the corresponding and non-applicable search result records. The administrator preferentially additionally learns the feature quantities with low applicable mean value, high non-applicable average value, and large applicable or non-applicable standard deviation, and to obtain the correlation coefficient, average value, and standard deviation. It is possible to judge the feature amount to be additionally learned by referring to it.

Regarding the teacher data used for additional learning, in addition to setting the extraction conditions on the teacher data extraction condition setting screen 600 (see FIG. 12) and extracting, the search result record specified by the administrator on the graph is extracted as teacher data. You may do so. The image search device 100 may, for example, extract a search result record in an area designated by the administrator on the graph and use it as teacher data.

≪Variation example 4: Timing of teacher data extraction≫
In the above-described embodiment, the image search device 100 extracts teacher data for additional learning when instructed by the administrator. The image search device 100 may be extracted at a predetermined cycle, for example, and output to the additional learning device 300. Further, the image search device 100 may periodically repeat the extraction and output the extraction result to the additional learning device 300 when the number of collected data is 188 or more (see FIG. 8).

The teacher data may be extracted every time the user sets the classification item in the classification item setting area 534. Specifically, every time the user sets a classification item, the Web browser on the terminal 320 transmits the image identification information and the classification item to the Web server 116. The Web server 116 outputs the image identification information and the classification item to the classification item registration unit 117 and the teacher data extraction unit 118.
The classification item registration unit 117 is a record in the search result table 164 (see FIG. 7) of the search result database 150, and the classification item 177 of the search result record corresponding to the image identification information is output by the Web server 116. Change to an item.

The teacher data extraction unit 118 is a record in the search result table 164, and the search result record corresponding to the image identification information matches any of the extraction conditions in the teacher data extraction condition table 181 (see FIG. 8). If so, the search result record is stored in the teacher data extraction result table 191 (see FIG. 10). When the user changes the classification item, the classification item 177 is changed, or when the change does not meet any of the extraction conditions of the teacher data extraction condition table 181 (see FIG. 8), the record is deleted. do.
By doing so, the teacher data can be immediately extracted according to the classification item set by the user. Further, when the number of extraction results reaches the number of collected data 188, the teacher data is immediately extracted and can be output to the additional learning device 300 (see FIG. 1). As a result, system downtime is reduced, maintenance costs in daily work are reduced, and machine learning model 114 is updated faster.

In addition, the image search device 100 periodically calculates the correlation coefficient, the average value, and the standard deviation of the feature and the classification item in the search result record, and outputs the correlation coefficient, the average value, and the standard deviation to the additional learning device 300 when a predetermined condition is satisfied. You may do so. For example, the image search device 100 may output when the correlation coefficient is smaller than a predetermined value.
By doing so, the image search device 100 can perform additional learning as needed without the instruction of the administrator. Alternatively, the image search device 100 can encourage the administrator to perform additional learning.

≪Variation example 5: Judgment target data and features≫
The machine learning model 114 in the above-described embodiment extracts and classifies the feature quantities of the head color, the color of the upper body clothes, and the color of the lower body clothes from the image data of the person. The colors do not have to be limited to black, gray, white and blue. Not limited to colors, items worn such as glasses and hats, items carried such as bags and smartphones, and height may be extracted and classified by feature quantity. By corresponding to various feature quantities for such a person, the accuracy of the search performed by the image search device 100 can be improved.

Further, the target (input data) of the machine learning model 114 in the above-described embodiment is an image of a person, but the object is not limited to this, and may be an article. Further, the search is not limited to the image search, and may be, for example, a document search. In a device that searches for documents by type or category, the teacher data of the machine learning model may be extracted using the type or category as a feature quantity. The machine learning model is not limited to deep learning, and may be a machine learning model such as SVM.

≪Variation example 6: Search condition≫
The search conditions for a person in the above-described embodiment are the color of the head, the color of the upper body clothes, and the color of the lower body clothes. In addition to this, conditions such as the shooting time of the camera 310, the shooting camera 310, and the shooting area may be added so that the search can be performed.
Alternatively, as the search condition, image data of the person to be searched may be used instead of the color of the head, the color of the clothes of the upper body, and the color of the clothes of the lower body. The image search unit 115 extracts the head color, the color of the upper body clothes, and the color of the lower body clothes from the image data that is the search condition, and the feature amount table 131 of the feature amount database 130 (FIG. 5) is searched. By doing so, the user can search if there is an image of the target person without specifying the color of the head, the color of the clothes of the upper body, and the color of the clothes of the lower body.

≪Variation example 7: Display order of search results≫
In the above-described embodiment, the search result 531 is displayed in the search result area 530 in descending order of the degree of matching with the search condition (average similarity). It may be displayed in order based on other information such as the camera, its installation area, and the shooting time.

≪Other variants≫
The present invention is not limited to the above-described embodiment, and can be modified without departing from the spirit of the present invention. The image search device 100 has executed the image acquisition, the search of the search target person, and the extraction of the teacher data on one computer, but it may be executed on a plurality of computers. Further, the user interface is a Web browser on the terminal 320, but the user interface is not limited to this.

Although some embodiments of the present invention have been described above, these embodiments are merely examples and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and further, various modifications such as omission and substitution can be made without departing from the gist of the present invention. Further, the order of processing may be changed or processing may be performed in parallel.
For example, the feature amount table 131 (see FIG. 5), the search result table 164 (see FIG. 7), and the teacher data extraction result table 191 (see FIG. 10) include image identification information 134, 166, 196, but instead. The image itself may be included. Further, the image search unit 115, the classification item registration unit 117, and the teacher data extraction unit 118 may be used as one functional unit. Alternatively, steps S104 and S105 may be replaced.
These embodiments and variations thereof are included in the scope and gist of the invention described in the present specification and the like, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

100 Image search device 111 Image capture unit 112 Person detection unit 113 Feature extraction unit 114 Machine learning model 115 Image search unit 117 Classification item registration unit (classification item acquisition unit)
118 Teacher data extraction unit 119 Machine learning model update unit 300 Additional learning device 510 Search set area 520 Search condition area 530 Search result area 531 Search result 532 Image confirmation area 533 Average similarity bar 534 Classification item setting area

Claims (12)

A feature extractor that extracts features from the acquired image using a machine learning model,
An image search unit that searches for the image using the feature amount and outputs the search result,
A classification item acquisition unit that classifies individual images of the search results and acquires classification items indicating the assigned classification results.
An image search device including a teacher data extraction unit that extracts an image as teacher data for additional learning of the machine learning model based on the correlation between the feature amount and the classification item. The image search unit acquires the feature of the target target, collates the feature of the target with the feature amount extracted from the image, searches the image, and outputs the search result.
The image according to claim 1, wherein the classification item includes a hit indicating that the target object is included in the image and a non-applicability indicating that the target object is not included in the image. Search device. The image search device according to claim 1 or 2, wherein the teacher data extraction unit outputs any of a classification item and a feature amount of the image together with the extracted image to the outside of the image search device. The claim is characterized in that the teacher data extraction unit extracts either an image in which the feature amount and the classification item are inversely correlated, or an image in which the feature amount and the classification item are correlated. The image search device according to any one of 1 to 3. In the teacher data extraction unit, an image in which the feature amount is lower than a predetermined value and the classification item is applicable, an image in which the feature amount is higher than the predetermined value and the classification item is not applicable, and the feature amount is lower than the predetermined value. The image search device according to claim 4, wherein any of an image to which the classification item is not applicable and an image whose feature amount is higher than a predetermined value and the classification item is applicable is extracted. The image search apparatus according to claim 1, further comprising a graph generation unit that generates a graph in which the image is plotted with the feature amount and the classification item as axes. The graph generator is characterized in that it calculates and displays one of the correlation coefficient of the feature amount and the classification item of the image plotted on the graph and the average and the standard deviation of the feature amount for each classification item. The image search device according to claim 6. The teacher data extraction unit classifies the images when a predetermined time has elapsed from the previous extraction, when the number of extracted images satisfies a predetermined condition, and when the correlation between the feature amount of the image and the classification item satisfies a predetermined condition. When the average of the feature amounts for each item satisfies a predetermined condition, the image to be the teacher data is extracted at any timing when the standard deviation of the feature amount for each classification item of the image satisfies the predetermined condition. The image search device according to claim 1, wherein the image to be the teacher data is extracted and output to the outside of the image search device, and the condition is notified that the condition is satisfied. The machine learning model is given a version that changes when the machine learning model is updated as a result of the additional learning.
The teacher data extraction unit extracts an image as teacher data based on the correlation between the feature amount extracted by the current version of the machine learning model and the classification item searched and assigned based on the feature amount. The image search device according to claim 1, wherein the image search device is characterized by the above. The teacher data extraction unit instructs to extract an image to be teacher data based on the correlation with the classification items searched and assigned based on the features extracted by the machine learning model of a version different from the current version. The image search device according to claim 9, further comprising notifying a warning when the image is searched. The image is an image including a person.
The feature amount is any one of the color of the head of the person, the color of the clothes of the upper body, the color of the clothes of the lower body, the feature amount related to what the person carries, and the feature amount related to what the person wears. The image search device according to claim 1, wherein the image search device is characterized by the above. It is a teacher data extraction method for image search devices.
Steps to extract features from the acquired image using a machine learning model,
A step of searching for the image using the feature amount and outputting a search result,
A step of classifying each image of the search result and acquiring a classification item indicating the given classification result, and
A teacher data extraction method characterized by executing a step of extracting an image as teacher data for additional learning of the machine learning model based on the correlation between the feature amount and the classification item.

Images