JP7357551B2 - Image judgment system - Google Patents

Description

The present invention relates to an image determination system.

For example, Patent Document 1 discloses a system that self-improves a learning model by sequentially relearning when an image judgment system does not make a correct judgment. Further, a system that presents optimal content to the user by converting each user's hobbies and preferences into feature quantities and performing machine learning is disclosed in, for example, Patent Document 2.

The image judgment system disclosed in Patent Document 1 evaluates a learning model that estimates the probability of each type represented by an image, and if the accuracy does not meet the standard, it assigns a correct label and retrains the learning model to improve accuracy. We will improve the model to a higher one.

In addition, in Patent Document 2, the reactions of each user to images, that is, the results of actions such as commenting, saying "like", or ignoring (passing), are converted into features, and the results are calculated using linear regression. By making predictions using machine learning methods such as logistic regression, boosting trees, or weighted decision trees, it is possible to form close user network groups and to customize and present images that match the user's hobbies and tastes.

JP2019-152948A Special table 2016-510441 publication

However, the techniques disclosed in Patent Documents 1 and 2 mentioned above have a mechanism for mass customizing (hereinafter abbreviated as MC) image determination results for each user, and a multi-stage image determination mechanism to improve accuracy. No consideration is given to a mechanism for making more accurate judgments by escalating to a judgment mechanism with higher precision or expertise as needed. Furthermore, no consideration is given to a mechanism that allows the user to make a comprehensive judgment by adding the learning maturity level of the judgment mechanism to the method of presenting the judgment result to the user.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an image determination system that can perform optimal image determination for each user.

In order to solve the above problems, an image determination system according to one aspect of the present invention includes a terminal, and a common server and a specialized server that can communicate with the terminal.
The terminal includes an imaging unit that captures an image of a subject and outputs an image, a storage unit that stores a lightweight learning model, an image judgment unit that uses the lightweight learning model to determine the classification of the subject based on the image, and an image processing unit that outputs an image. a display unit that presents the classification of the subject determined by the image determination unit; an input unit that accepts corrections or additional input to the presented classification of the subject; It has a data collection unit that collects and stores it in a storage unit as a lightweight MC teacher data set, and also sends classification corrections or additional inputs to a specialized server.
The common server includes a common storage unit that stores a common learning model and a common teacher data set, a common data collection unit that generates a common teacher data set using images sent from a terminal, and a common data collection unit that uses the common teacher data set. The common learning model generation unit generates a common learning model, further generates a lightweight learning model by reducing the weight of the common learning model, and transmits this lightweight learning model to a terminal.
The specialized server includes a specialized storage section in which specialized learning models and specialized teacher datasets are stored, a specialized data collection section that generates specialized teacher datasets using classification corrections or additional inputs sent from the terminal, and a specialized teacher dataset. It has a specialized learning model generating section that generates a specialized learning model using a data set, and a specialized image determining section that determines a subject using a specialized learning model based on the image when it is difficult to make a determination by the image determining section.

According to the present invention, it is possible to realize an image determination system that can perform optimal image determination for each user.

1 is a functional block diagram showing a functional configuration of an image determination system according to Example 1. FIG. 2 is a flowchart illustrating an example of a process for generating an initial learning model and a mobile lightweight MC learning model of the image determination system according to the first embodiment. 2 is a flowchart illustrating an example of a mobile lightweight MC learning model generation process of the image determination system according to the first embodiment. 7 is a flowchart illustrating another example of the mobile lightweight MC learning model generation process of the image determination system according to the first embodiment. 3 is a flowchart illustrating an example of sequential learning process processing of the image determination system according to the first embodiment. 3 is a flowchart illustrating an example of sequential learning process processing of the image determination system according to the first embodiment. FIG. 3 is a diagram illustrating an example for explaining the relationship between learning models of the image determination system according to the first embodiment and an image of a dictionary to be referred to. 7 is a flowchart illustrating an example of MC learning model generation processing of the image determination system according to the first embodiment. 7 is a flowchart illustrating an example of MC learning model generation processing of the image determination system according to the first embodiment. 7 is a flowchart illustrating an example of MC learning model generation processing of the image determination system according to the first embodiment. 7 is a flowchart illustrating an example of sequential learning process processing of the image determination system according to the second embodiment. 7 is a flowchart illustrating an example of a maturity level feedback process of the image determination system according to the second embodiment. 7 is a flowchart illustrating another example of the maturity level feedback process of the image determination system according to the second embodiment. 7 is a diagram illustrating an example of a screen displayed on a mobile terminal in the image determination system according to Example 2. FIG. 7 is a diagram showing another example of a screen displayed on a mobile terminal in the image determination system according to the second embodiment. FIG. 7 is a diagram showing another example of a screen displayed on a mobile terminal in the image determination system according to the second embodiment. FIG. FIG. 7 is a diagram illustrating an example of a usage form of an image determination system according to a third embodiment. 12 is a flowchart illustrating an example of sampling response correction method processing of the image determination system according to the fourth embodiment. 13 is a flowchart illustrating an example of MC learning model correction method processing of the image determination system according to the fourth embodiment. 12 is a flowchart illustrating an example of a learning model update/relearning/maintenance process processing of the image determination system according to the fifth embodiment. 13 is a flowchart illustrating an example of data ratio decreasing relearning method processing of the image determination system according to the fifth embodiment. 12 is a flowchart illustrating an example of data upper limit fixed relearning method processing of the image determination system according to the fifth embodiment.

Embodiments of the present invention will be described below with reference to the drawings. The embodiments described below do not limit the claimed invention, and all of the elements and combinations thereof described in the embodiments are not necessarily essential to the solution of the invention. Not exclusively.

In the drawings explaining the embodiments, parts having the same functions are denoted by the same reference numerals, and repeated explanations thereof will be omitted.

Further, in the following description, an expression such as "xxx data" may be used as an example of information, but the information may have any data structure. That is, "xxx data" can be referred to as "xxx table" to indicate that the information is independent of data structure. Furthermore, "xxx data" is sometimes simply referred to as "xxx". In the following description, the configuration of each piece of information is merely an example, and the information may be held separately or may be combined and held.

FIG. 1 is a functional block diagram showing the functional configuration of an image determination system according to a first embodiment.

The image determination system S of this embodiment includes a plurality of mobile terminals 100, a common server 200, and a single or multiple specialized servers 300. The mobile terminal 100 is carried and used by a plurality of users (USER-1 to USER-X in the illustrated example) of the image determination system S of this embodiment. In the following explanation, the mobile terminal 100 of USER-X will be mainly explained.

The mobile terminal 100 and the common server 200 are capable of mutually transmitting and receiving information through each other's communication units 160 and 230. Similarly, the common server 200 and the specialized server 300 are capable of mutually transmitting and receiving information through their communication units 230 and 330. There is no particular limitation on the means of communication between the mobile terminal 100 and the common server 200, but preferably a portion thereof is configured by wireless communication means. Further, the communication means may be at least one or both of a so-called LAN (Local Area Network) and a WAN (Wide Area Network) typified by the Internet and a mobile communication network.

The mobile terminal 100, the common server 200, and the specialized server 300 are all devices capable of processing various types of information, such as an information processing device such as a computer. The information processing device includes at least an arithmetic element and a storage medium. In particular, the mobile terminal 100 is preferably a smartphone, a tablet terminal, or the like that can be carried by the user.

The arithmetic element is, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an FPGA (Field-Programmable Gate Array). The storage medium includes, for example, a magnetic storage medium such as an HDD (Hard Disk Drive), a semiconductor storage medium such as a RAM (Random Access Memory), a ROM (Read Only Memory), or an SSD (Solid State Drive). Furthermore, a combination of an optical disk such as a DVD (Digital Versatile Disk) and an optical disk drive is also used as a storage medium. In addition, known storage media such as magnetic tape media can also be used as storage media.

Programs such as firmware are stored in the storage medium. When the mobile terminal 100, the common server 200, and the specialized server 300 start operating (for example, when the power is turned on), a program such as firmware is read from this storage medium and executed, thereby controlling the entire mobile terminal 100 and the like. In addition to programs, the storage medium also stores data necessary for the operation of the mobile terminal 100 and the like.

Note that the common server 200 and the specialized server 300 of this embodiment may each be configured as a so-called cloud in which information processing apparatuses are configured to be able to communicate via a communication network. In particular, specialized servers are not limited to servers prepared by the company, but also public network AI (APIs (application program interfaces) are released for the purpose of providing artificial intelligence functions and are allowed to be used by others). Including servers.

Mobile terminal 100 includes a control section 110, a storage section 120, an imaging section 130, an input section 140, a display section 150, and a communication section 160.

The control unit 110 is mainly composed of arithmetic elements and controls the entire mobile terminal 100. In addition, firmware (not shown) stored in the storage unit 120 is executed when the mobile terminal 100 is started, thereby performing functions as various functional units. As such functional units, the control unit 110 of this embodiment includes a mobile image determination unit 111, an imaging control unit 112, a data transmission/reception unit 113, and a data collection unit 114.

Based on the mobile lightweight learning model 121 and the mobile lightweight MC learning model 122 stored in the storage unit 120, which will be described later, the mobile image determination unit 111 uses artificial intelligence (including machine learning and deep learning) techniques to The subject imaged by the imaging unit 130 is determined. Since the subject determination method performed by the mobile image determination unit 111 is well known, detailed explanation thereof will be omitted here.

The imaging control section 112 controls the imaging section 130. The data transmission/reception unit 113 controls transmission and reception of data via the communication unit 160.

The data collection unit 114 transmits the image output by the imaging unit 130 to the common server 200. Furthermore, the data collection unit 114 adds the subject classification candidate accepted by the input unit 140 or its correction or additional input information as a label to the image information, and stores and accumulates it in the storage unit 120 as a mobile MC teacher data set 123. do.

The storage unit 120 is mainly composed of a storage medium, and in addition to programs such as the firmware described above, data necessary for the operation of the mobile terminal 100 is temporarily or constantly stored.

The storage unit 120 stores a mobile lightweight learning model 121, a mobile lightweight MC learning model 122, and a mobile MC teacher data set 123. These mobile lightweight learning models 121 and the like have data structures suitable for use in determining subjects using artificial intelligence techniques. Since the data structure is well known, a description thereof will be omitted here.

The image capturing unit 130 captures an image of a subject around the mobile terminal 100 using an image capturing element such as a CMOS, and outputs the image as an image including the subject. The input unit 140 has a touch panel or the like, and accepts various operation inputs from the user. The display unit 150 has a liquid crystal display or the like, and displays a display screen on the display surface based on commands from the control unit 110. The communication unit 160 controls transmission and reception of various data between the common server 200 and the specialized server 300.

Next, the common server 200 includes a common control section 210, a common storage section 220, and a communication section 230.

The common control unit 210 is mainly composed of arithmetic elements, and controls the entire common server 200. In addition, unillustrated firmware stored in the common storage unit 220 is executed when the common server 200 is started, thereby performing functions as various functional units. As such functional units, the common control unit 210 of this embodiment includes a common image determination unit 211, a common learning model generation unit 212, a common data transmission/reception unit 213, and a common data collection unit 214.

The common image determination unit 211 determines the classification of the subject based on the image transmitted from the mobile terminal 100. The common learning model generation unit 212 generates a common learning model 221 using the common teacher data set 222 stored in the common storage unit 220, and further reduces the weight of the common learning model 221 to create a mobile lightweight learning model 121. This mobile lightweight learning model 121 is transmitted to the mobile terminal 100.

Further, the common learning model generation unit 212 generates an MC learning model using the MC teacher data set 224 (X) for USER-X stored in the common storage unit 220 alone or the data set combined with the common teacher data set 222. 223(X), further reduces the weight of the MC learning model 223(X) to generate a lightweight mobile MC learning model 122(X), and transmits this lightweight mobile MC learning model 122(X) to the mobile terminal 100. Send to.

Since the methods of generating the common learning model 221 and the like are well known, explanations other than the method of reducing the weight of the common learning model 221 to generate the mobile lightweight learning model 121 (details will be described later) will be omitted.

The common data transmission/reception unit 213 controls transmission and reception of data via the communication unit 230.

The common data collection unit 214 adds the image transmitted from the mobile terminal 100 and the subject classification candidate accepted by the input unit 140 as a label to the image information, and adds and stores it in the common teacher data set 222. Furthermore, if USER-X does not accept the subject classification candidate displayed on the display unit 150, the common data collection unit 214 adds USER-X's correction or additional input information to the image information as a label, and It is added to and stored in the mobile MC teacher data set 123 in the storage unit 120 of the terminal 100. At this time, if the number of accumulated data exceeds N, the mobile terminal 100 collectively transmits the mobile MC teacher data set 123 to the common server 200 and adds it as an MC teacher data set 224 (X). accumulate.

The common storage unit 220 is mainly composed of a storage medium, and in addition to programs such as the firmware described above, data necessary for the operation of the common server 200 is temporarily or regularly stored.

The common storage unit 220 stores a common learning model 221, a common teacher data set 222, an MC learning model 223, and an MC teacher data set 224. These common learning models 221 and the like have data structures suitable for use in determining subjects using artificial intelligence techniques. Since the data structure is well known, a description thereof will be omitted here.

The communication unit 230 controls transmission and reception of various data between the mobile terminal 100 and the specialized server 300.

Next, the specialized server 300 includes a specialized control section 310, a specialized storage section 320, and a communication section 330.

The specialized control unit 310 is mainly composed of arithmetic elements, and controls the entire specialized server 300. Furthermore, unillustrated firmware stored in the specialized storage section 320 is executed when the specialized server 300 is started up, thereby performing functions as various functional sections. As such functional units, the specialized control unit 310 of this embodiment includes a specialized image determination unit 311, a specialized learning model generation unit 312, a specialized data transmission/reception unit 313, and a specialized data collection unit 314.

The specialized image determination unit 311 transmits information from the mobile terminal 100 when a more detailed specialized determination is required than the determination by the mobile image determination unit 111 of the mobile terminal 100 or the common image determination unit 211 of the common server 200. Based on the image, the subject is determined using the specialized learning model 321 stored in the specialized storage section 320.

The specialized learning model generation unit 312 generates a specialized learning model 321 using the specialized teacher data set 322 stored in the specialized storage unit 320. Since the method of generating the specialized learning model 321 and the like is well known, a description thereof will be omitted here.

The specialized data transmission/reception section 313 controls transmission and reception of data via the communication section 330.

The specialized data collection unit 314 uses the correction or additional input of the subject classification transmitted from the mobile terminal 100 to add to and accumulate in the specialized teacher data set 322.

The communication unit 330 controls transmission and reception of various data between the mobile terminal 100 and the common server 200.

Note that details of each function execution unit and data shown in FIG. 1 will be described later.

Next, the image determination operation by the image determination system S of this embodiment will be described with reference to FIGS. 2 to 7D. First, FIG. 2 is a flowchart illustrating an example of a process for generating an initial learning model and a mobile lightweight MC learning model of the image determination system S according to the first embodiment.

For initial learning, a plurality of users use the imaging unit 130 of the mobile terminal 100 to image a subject for building a learning model, output the image, have each user input classification category information as a label, and output the image to the mobile terminal 100. The data transmitting/receiving unit 113 of 100 adds the label to the image output from the imaging unit 130 and transmits the image to the common server 200 . The common data collection unit 214 of the common server 200 collects the set of images and labels transmitted from the mobile terminal 100, and adds and stores them as the common teacher data set 222 (step S200). The generated common teacher data set 222 is stored in the common storage unit 220.

If sufficient data cannot be collected by collecting data using a mobile terminal, or if no data can be collected at all, collect data by other means. For example, a system developer may independently photograph and collect learning data using human tactics, cut out video images, or mechanically collect multiple image contents on the Internet using bots or other means. Using an annotation tool or the like, images for teachers are cut out and labeled, and added and stored as a common teacher data set 222 (step S200).

The common learning model generation unit 212 of the common server 200 generates the common learning model 221 using the common teacher data set 222 (step S201). The generated common learning model 221 is also stored in the common storage unit 220.

Next, the common learning model generation unit 212 reduces the weight of the generated common learning model 221 to generate a mobile lightweight learning model 121 (step S202). The generated mobile lightweight learning model 121 is transmitted to the mobile terminal 100 by the common data transmitting/receiving unit 213, and the data transmitting/receiving unit 113 of the mobile terminal 100 stores the mobile lightweight learning model 121 in the storage unit 120.

Distillation, pruning, quantization, etc. are known methods for reducing the weight of learning models. Distillation is a deep, multi-layered neural network that uses a model trained using sufficient data as a teacher model, and its output (final/intermediate) results as ground truth data to train a student model and create shallower nodes. This allows high accuracy to be obtained even with a small number of metrological models. Furthermore, pruning is a method of cutting off the network by setting weakly related parts of the neural network to 0, thereby making the hierarchy shallower/diluting, thereby saving the required calculation amount and memory amount. Quantization is the process of shifting the floating point weights of a trained network to either ±1 to make calculations easier.

By reducing the weight of the common learning model 221 using such a known weight reduction method to generate the mobile lightweight learning model 121, and determining the subject on the mobile terminal 100 using this mobile lightweight learning model 121, processing is performed. The subject determination process can be performed even by the mobile terminal 100 with limited capabilities and memory.

Next, the labels of the teacher data set collected in step S200 are organized by classification category, and if a hierarchical structure can be found in the categories, the teacher data set is classified into more detailed subcategories than standard common categories. If so, it is deleted from the common teacher data set, sent to the specialized data collection unit 314 of the specialized server 300, and added and stored in the specialized teacher data set 322 (step S203). The generated specialized teacher data set 322 is stored in the specialized storage section 320.

The specialized learning model generation unit 312 of the specialized server 300 generates the specialized learning model 321 based on the specialized teacher data set 322 (step S204). The generated specialized learning model 321 is also stored in the specialized storage section 320.

On the other hand, if USER-X does not accept the subject classification candidate displayed on the display unit 150, USER-X's correction or additional input information is added to the image information as a label, and the image information is stored in the storage unit 120 of the mobile terminal 100. is added to and stored in the mobile MC teacher data set 123 (step S205). At this time, if the number of accumulated data exceeds N (N is a natural number) (YES in step S206), the mobile terminal 100 collectively transmits the mobile MC teacher data set 123 to the common server 200, It is added and stored as the MC teacher data set 224(X) (step S207). The generated MC teacher data set 224 is stored in the common storage unit 220.

The common learning model generation unit 212 of the common server 200 generates the MC learning model 223 (X) using the MC teacher data set 224 (X) for USER-X alone or the data set combined with the common teacher data set 222. At the same time, this MC learning model 223(X) is reduced in weight to generate a mobile lightweight MC learning model 122, and this mobile lightweight MC learning model 122 is transmitted to the mobile terminal 100 of USER-X (step S208). Details of step S108 will be described later with reference to FIGS. 3 and 4.

Thereafter, the mobile image determination unit 111 of the mobile terminal 100 determines the subject using the mobile lightweight learning model 121 and the mobile lightweight MC learning model 122 (step S209).

Therefore, for users who require a detailed classification judgment of a subject, by also using the mobile lightweight MC learning model 122 that performs MC to judge the subject, it is possible to perform a judgment process customized for each user. .

FIG. 3 is a flowchart illustrating an example of the lightweight MC learning model generation process of the image determination system S according to the first embodiment. The flowchart shown in FIG. 3 shows details of step S208 in the flowchart shown in FIG.

First, the common learning model generation unit 212 generates the MC learning model 223 using the MC teacher data set 224 as training data (step S300).

Next, the common learning model generation unit 212 performs ensemble learning of the MC learning model 223 and the common learning model 221 (step S301). Here, ensemble learning is a method of deriving a final answer by combining multiple learning models (using a method such as majority voting).

Then, the common learning model generation unit 212 generates the mobile lightweight MC learning model 122 using a learning model lightweighting method based on the learning model obtained by ensemble learning (step S302).

In the flowchart shown in FIG. 3, after the mobile lightweight MC learning model 122 is generated by the learning model lightweighting method, ensemble learning is performed with the mobile lightweight learning model 121 on the mobile terminal 100, thereby finally creating the mobile lightweight MC learning model 122. It is also possible to generate a lightweight MC learning model 122.

FIG. 4 is a flowchart illustrating another example of the lightweight MC learning model generation process of the image determination system S according to the first embodiment. The flowchart shown in FIG. 4 also shows details of step S208 in the flowchart shown in FIG.

First, the common learning model generation unit 212 generates the MC learning model 223 using the MC teacher data set 224 and the common teacher data set 222 (step S400).

If the MC teacher data set is extremely smaller than the common teacher data set (for example, 1/100), the data in the MC teacher data set is augmented (step S401).

Then, the common learning model generation unit 212 generates the mobile lightweight MC learning model 122 based on the MC learning model 223 using a learning model lightweighting method (step S402).

Next, FIGS. 5A and 5B are flowcharts illustrating an example of the sequential learning process processing of the image determination system S according to the first embodiment. The flowcharts shown in FIGS. 5A and 5B are executed after the initial learning model generation process shown in FIG. 2 is performed.

First, the mobile image determination unit 111 uses the mobile lightweight learning model 121 to perform subject determination processing on an image captured by the user (step S500). As a result of the determination process, the classification of the subject in the image (for example, "dog" or "cat") is obtained along with its probability ("accuracy").

Then, it is determined whether or not the sum of the respective accuracies is greater than or equal to x% (0≦x≦100, for example, x=80) for the judgment results with the highest accuracy (n is a natural number). (Step S501). If the determination is affirmative, the program moves to step S402, and displays the classification of the subject that is the top n candidate on the display unit 150 of the mobile terminal 100 together with the accuracy. On the other hand, if the determination is negative, the program moves to step S505.

Next to step S502, the mobile image determination unit 111 determines whether the classification accuracy of the top category, which has the highest accuracy, is z% (0≦z≦100) among the candidates displayed on the display unit 150 of the mobile terminal 100. , for example, z=50), and furthermore, it is determined whether there is a subcategory (subcategory) that is further subdivided from this classification (step S503). Here, the subcategory=detailed classification is, for example, if the classification is "dog", it is a dog breed that belongs to the classification "dog", such as "Shiba Inu" or "Chihuahua". Then, if the determination is affirmative, the program moves to step S511, and if the determination is negative, the program moves to step S504.

That is, the mobile image determination unit 111 waits for the user's classification designation input via the input unit 140, and when the classification designation input is made, it determines whether the user's classification designation input is among the candidates displayed in step S502. is determined (step S504). If the determination is affirmative, the program moves to step S510, and if the determination is negative, the program moves to step S505.

On the other hand, in step S505, the data transmitting/receiving unit 113 of the mobile terminal 100 transmits the image captured by the user to the common server 200. The common control unit 210 of the common server 200 judges the image using the common learning model 221, and transmits the result to the mobile terminal 100 (step S506). Next, the mobile image determination unit 111 subtracts y (for example, 5%) from x, which is the determination criterion in step S501, assuming that the accuracy does not increase even with the result of the image determination in step S406 ( Step S507).

Then, the mobile image determination unit 111 determines whether x, which is the total accuracy of the determination results in the top n ranks, is less than a% (0≦a≦100, for example, a=50). (Step S508). If the determination is affirmative, the program moves to step S509, and if the determination is negative, the program moves to step S501.

In step S509, the mobile image determination unit 111 prompts the user to input a correct classification for the image. Then, in step S410, the data collection unit 114 of the mobile terminal 100 temporarily stores the user's selection in the mobile MC teacher data set 123, and when a certain number of selections have been accumulated, the data collection unit 114 of the mobile terminal 100 stores the user's selection in the common server 200 as the MC teacher data set 224. Send.

In FIG. 5B, the specialized image determination unit 311 of the specialized server 300 performs image determination using the specialized learning model 321, and transmits the result to the mobile terminal 100 (step S511). Next, it is determined whether or not the sum of the probabilities of the top n determination results is greater than or equal to x% (step S512). If the determination is affirmative, the program moves to step S513, and displays the classification of the subject, which is the top n candidate, along with the accuracy on the display unit 150 of the mobile terminal 100. On the other hand, if the determination is negative, the program moves to step S515.

In step S513, the mobile image determination unit 111 waits for the user's classification designation input via the input unit 140, and when the classification designation input is made, whether the user's classification designation input is among the candidates displayed in step S502 or not. It is determined whether or not. If the determination is affirmative, the program moves to step S516, and if the determination is negative, the program moves to step S515. In step S515, the mobile image determination unit 111 of the mobile terminal 100 displays the image determination result of the common server 200 obtained in step S506.

Then, in step S516, the data collection unit 114 of the mobile terminal 100 transmits the user's selection input to the common server 200 as the MC teacher data set 224.

FIG. 6 is a diagram for explaining the MC learning model generation process of the image judgment system S according to the first embodiment, and FIGS. 7A to 7C are the MC learning model generation processes of the image judgment system S according to the first embodiment. It is a flowchart which shows an example.

In these figures, USER-X images a subject with the imaging unit 130 of the mobile terminal 100, and (1) performs image judgment using the mobile lightweight learning model 121 (common) (step S700). The candidates based on the image determination are displayed up to the top T in accordance with the screen size of the display unit 150 of the mobile terminal 100.

Next, the data collection unit 114 determines whether the answer selected by USER-X is among the top n candidates and has a probability of a% or more (step S701). If the candidate is among the top n candidates and the accuracy is a% or more (YES in step S701), the data collection unit 114 determines that there is no need for MC, and uses the selected candidate (label ) and sends it to the common server 200, and the common server 200 adds and stores this data in the common teacher data set 222 (step S702).

On the other hand, if the determination in step S701 is negative, the data collection unit 114 determines whether the answer selected by USER-X is among the top n candidates, but the accuracy is less than a% (step S703). . If the determination is affirmative (yes in step S503), it is assumed that a candidate different from the one recommended by the mobile lightweight learning model 121 (common) has been selected, and the data collection unit 114 uses information specific to the USER-X, That is, it is determined that MC is necessary, and the selected candidate (label) is added to the subject image data and added and stored in the mobile MC teacher data set 123 (X) (step S705).

On the other hand, if the candidate is not in the top n candidates but matches a candidate displayed in the top T candidates (NO in step S703), the data collection unit 114 uses the mobile lightweight learning model 121 (common) It is assumed that a candidate different from the one recommended by the USER-X has been selected, and it is determined that MC is necessary, and the selected candidate (label) is added to the subject image data, and the mobile version is It is added to and stored in the MC teacher data set 123(X) (YES in step S504).

Furthermore, if it is not in the top T candidates and you select others (NO in step S704), redo the image judgment using the (2) mobile lightweight MC learning model 122(X) of USER-X (step S706) ). The candidates based on the image determination are displayed up to the top T in accordance with the screen size of the display unit 150 of the mobile terminal 100.

At that time, if there are duplicate candidates within the top T rank determined in step S700 in the determination results, those candidates are eliminated and only the newly determined candidates are used to rank up to the Max top T rank. Re-present as other pull-down candidates.

Next, the data collection unit 114 determines whether the answer selected by USER-X is among the top m candidates and has a probability of b% or more (step S707). If the candidate is among the top m candidates and the accuracy is b% or higher (YES in step S707), the data collection unit 114 has determined that (2) the mobile lightweight MC learning model 122(X) is suitable. The selected candidate (label) is added to the image of the subject and added to and stored in the mobile MC teacher data set 123 (X) (step S708).

On the other hand, if the determination in step S707 is negative, the data collection unit 114 determines whether or not the answer selected by USER-X is among the top m candidates, but the accuracy is less than b% (step S709). . If the determination is affirmative (YES in step S709), it is considered that the data collection unit 114 has selected a candidate different from the one recommended by the mobile lightweight MC learning model 122(X), but the - It is determined that there is no change in the judgment specific to X, that is, that MC is necessary, and the selected candidate (label) is added to the subject image data, and the mobile MC teacher data set 123 (X ) (step S711).

If the subject image is not in the top m candidates but matches a candidate displayed in the top T positions (YES in step S710), it is considered that this is unique to the mobile lightweight MC learning model 122(X), and the subject image is The selected candidate (label) is added to the data and added to and stored in the mobile MC teacher data set 123 (X) (step S711).

If the subject image is not in the top T candidates (NO in step S710) and you select Others, the data collection unit 114 sends the subject image to the common server 200, and (3) performs image judgment using the common learning model 221. Try again (step S712). The candidates based on the image determination are displayed up to the top T in accordance with the screen size of the display unit 150 of the mobile terminal 100.

At this time, if there are duplicate candidates in the top T ranks determined in step S700 or candidates in the top T ranks determined in step S706 in the determination results, those candidates are eliminated. Only the newly determined items are re-presented as other pull-down candidates up to the maximum T rank.

Next, the data collection unit 114 determines whether the answer selected by USER-X is among the top candidates or whether the accuracy is c% or higher (step S713). If the candidate is in the top L and the accuracy is c% or higher (YES in step S713), it is determined that (3) the common learning model 221 has made an appropriate determination, and the data collection unit 114 The selected candidate (label) is added to the image of the subject and added to and stored in the common teacher data set 222 (step S714).

If the determination in step S713 is negative and the answer selected by USER-X is among the top L candidates, but the accuracy is less than c% (YES in step S715), the data collection unit 114 uses the common learning model It is assumed that a candidate different from the one recommended by 221 has been selected, and it is determined that this is a determination specific to the USER-X, that is, that MC is necessary, and the selected candidate (label) is added to the subject image data. , is added to and stored in the MC teacher data set 224(X) of USER-X (step S717).

If the candidate is not in the top L candidates but matches a candidate displayed in the T rank (YES in step S716), the data collection unit 114 uses the MC learning model 223(X) specific to USER-X. Considering this, the selected candidate (label) is added to the subject image data and added and stored in the MC teacher data set 224 (X) of USER-X (step S717).

If it is not in the top T candidates (NO in step S716) and USER-X selects Others, the data collection unit 114 uses (4) free input to classify the classification (label) that matches the subject image into the mobile An input is made through the input unit 140 of the terminal 100 (step S718).

The input data is compared with the synonym dictionary (synonym DB) 680 shown in FIG. 6, and if there is a registration as a synonym, the data collection unit 114 replaces it with a representative notation to eliminate fluctuations in the notation. prevent (step S719).

Next, the data collection unit 114 checks whether the input classification (label) corresponds to a classification category registered in the subcategory dictionary (thesaurus DB) 690 (step S720).

If the answer entered by USER-X matches an existing category, (1) Mobile lightweight learning model (common), (2) Mobile lightweight MC learning model (for USER-X), (3) Common learning Since this was a new subject image that could not be predicted by any of the learning models, the data collection unit 114 added it to the MC teacher dataset of USER-X as MC target information specific to USER-X. Accumulate (step S721).

If the input classification (label) does not correspond to any classification category registered in the subcategory dictionary (thesaurus DB) 690 (NO in step S720), then the data collection unit 114 determines whether the input classification (label) corresponds to the subcategory. Check if something is there. (Step S722).

If the answer input by USER-X matches an existing subcategory (YES in step S722), (1) mobile lightweight learning model (common), (2) mobile lightweight MC learning model (for USER-X), (3) Although this is a new subject image that could not be predicted by any of the common learning models, since it corresponds to detailed classification knowledge, it is added and stored in the specialized teacher data set 322 (step S723).

If the input classification (label) does not correspond to any classification subcategory registered in the subcategory dictionary (thesaurus DB) 690 (NO in step S722), a new category that does not exist in any existing category or subcategory is added. Since the classification category has been input, the data collection unit 114 adds the label as a new category to the subcategory dictionary (thesaurus DB) 690, adds the input classification (label) to the subject image data, It is added to and stored in the MC teacher data set 224(X) of USER-X (step S723).

In FIG. 7C, candidates (labels) selected for the subject image data in (1) mobile lightweight learning model (common), (2) mobile lightweight MC learning model (for USER-X), and (3) common learning model. (steps S702, S705, S708, S711, S714, S717), and (4) when a free input classification (label) exists in an existing category. (Step S721), the data collection unit 114 checks whether the category has a classification subcategory registered in the subcategory dictionary (thesaurus DB) 690 (step S725).

If there is a subcategory (YES in step S725), the data collection unit 114 inquires of USER-X whether subcategorization (detailing) is desired (step S726).

If USER-X desires detailing (YES in S727), the data collection unit 114 sends (escalates) the subject image to the specialized server 300, (5) judges the image using the specialized learning model 321, The top T candidates in terms of accuracy detailed into subcategories are displayed on the display unit 150 of the mobile terminal 100 (step S728).

Next, the data collection unit 114 determines whether the answer selected by USER-X is among the top k candidates and has a probability of d% or more (step S729). If it is in the top k candidates and the accuracy is d% or more (YES in step S729), it is determined that (5) the specialized learning model 321 has made an appropriate determination, and it is selected as the image of the subject. The selected candidates (labels) are added and stored in the specialized teacher data set 322 (S730).

In addition, for USER-X, in a case like the subject, it is unique to the USER-X to subcategory, that is, it is a judgment that requires MC, and it is selected for the image of the subject. The candidate (label) is added and also added to and stored in the MC teacher data set 224 (X) of USER-X (step S731).

On the other hand, if the answer selected by USER-X is among the top k candidates but the accuracy is less than d% (YES in step S732), a candidate different from the one recommended by the professional learning model 321 is selected. It is determined that this is a determination specific to the USER-X, that is, that MC is necessary, and the selected candidate (label) is added to the subject image data, and the MC teacher data set 224 ( X) (step S737).

Furthermore, if the candidate is not among the top k candidates but matches the candidate displayed within the T rank (YES in step S733), it is assumed that a candidate different from the one recommended by the professional learning model 321 has been selected. , determines that it is a determination specific to the USER-X, that is, that MC is necessary, adds the selected candidate (label) to the subject image data, and adds it to the MC teacher data set 224 (X) of the USER-X. Add and accumulate (step S737).

If the answer selected by USER-X is not among the top T candidates (NO in step S733), ask the user to select Other and refer to the subcategory dictionary (thesaurus DB) 690 shown in the example of FIG. , all subcategory candidates for the category are presented, including those that do not fit on one screen, by scrolling or other methods, and the USER-X is allowed to freely select from among them. (Step S734)

If USER-X's options are not among them, USER-X is asked to freely input detailed subcategories by key input (step S735). In this case, use a combo box type input method (both pulldown and key input are possible).

The data input in this way is compared with the synonym dictionary (synonym DB) 680 shown in FIG. 6, and if there is a registration as a synonym, it is replaced with a representative notation to prevent fluctuations in the notation. (Step S736).

The candidates selected by USER-X and the subcategories entered freely by key are added to the subcategory dictionary (thesaurus DB) 690 as new records. Note that formal updating of the thesaurus DB 690 may require approval from the administrator. After this, it is determined that MC is necessary, the input candidate (label) is added to the image of the subject, and it is added and stored in the mobile MC teacher data set 123 (X) of USER-X (step S737 ). Since this data is likely to be new and specific to USER-X, it is not added to the professional teacher data set 322 at this point.

Therefore, according to the present embodiment, the data collection unit 114 of the mobile terminal 100 transmits the modification or additional input of the classification to the specialized server 300, and the specialized server 300 receives the modification or additional input transmitted from the mobile terminal 100. a specialized data collection unit 314 that generates a specialized teacher data set 322 using the specialized teacher data set 322; a specialized learning model generation unit 312 that generates a specialized learning model 321 using the specialized teacher dataset 322; and a mobile image determination unit 111 of the mobile terminal 100. If it is difficult to make a determination based on the image, the specialized image determination unit 311 uses a specialized learning model 321 to determine the subject based on the image.

Therefore, according to the present embodiment, detailed classifications (subcategories) can be set in the classifications (categories) of objects and detailed determinations can be made, so that image similarity determination work can be performed more precisely.

Further, according to the present embodiment, the common data collection unit 214 of the common server 200 generates the MC teacher data set 224 using the correction or additional input sent from the mobile terminal 100, and generates the common learning model of the common server 200. The generation unit 212 generates an MC learning model 223 using the MC teacher data set 224, further reduces the weight of the MC learning model 223 to generate a mobile lightweight MC learning model 122, and generates a mobile lightweight MC learning model 122. is transmitted to the mobile terminal 100, and the mobile image determination unit 111 of the mobile terminal 100 determines the subject using the mobile lightweight learning model 121 and the mobile lightweight MC learning model 122.

Therefore, according to the present embodiment, image judgment can be performed using the mobile lightweight MC learning model 122 that has been MC'd for each individual user using the mobile terminal 100, so that detailed classifications preferred by the user can be judged. can be performed accurately.

In this embodiment, a group of terminals and servers that make up the artificial intelligence form a multi-stage structure (mobile terminal 100, common server 200, specialized server 300), and escalation is performed according to the desired accuracy (mobile terminal 100 ( local processing) → common server 200 (on-premises) → specialized server 30 (on-premises or network AI)).

If this specialized server 300 (on-premises) does not provide an answer, it is possible to inquire of multiple network AIs behind it (AIs provided by the company or other companies based on Internet search engines) and receive the majority opinion as the answer. In the future, it is expected that network AI with more advanced knowledge will be provided free of charge (with some charges). The more advanced network AI is, the fewer services it provides, and conversely, escalation transactions come from all over the world, resulting in a well-balanced system (AI-WiKi). It is also possible to include such a network AI system and call it a specialized server 300.

In the following description, in the second embodiment, the image determination operation is mainly performed by the common server 200. However, as in the above-mentioned first embodiment, the image determination operation is mainly performed by the mobile terminal 100. is also possible.

The difference between the image determination system S of this embodiment and the image determination system S of the first embodiment lies in the functional configurations of the common server 200 and the specialized server 300 and the operation of the mobile terminal 100.

That is, the common control unit 210 of the common server 200 has a common image determination unit 211 and a common maturity level calculation unit 215, and the specialized control unit 310 of the specialized server 300 has a specialized maturity level calculation unit 315.

The common maturity level calculation unit 215 or the specialized maturity level calculation unit 315 is a calculation mechanism for quantifying an index representing the depth of learning (training) experience of each learned model. The timing of generating the common learning model 221, the mobile lightweight learning model (common) 121, and the specialized learning model 321, which are generated by reducing the weight of the common learning model 221, using the common teacher data set 222 and the specialized teacher data set 322 that have been compiled. The maturity level is calculated for each classification.

In addition, the common maturity level calculation unit 215 or the specialized maturity level calculation unit 315 calculates the mobile MC teacher data of each USER-X collected from the mobile terminal 100 etc. while the image judgment system S is operating (operating). Using data added and accumulated in the set 123 (X), data added and accumulated in the common teacher data set 222, data added and accumulated in the MC teacher data set 224 (X), and the specialized teacher data set 322. In addition, or in addition, by relearning, the common learning model 221, the mobile lightweight learning model (common) 121 that is generated by reducing its weight, the specialized learning model 321, the MC learning model (X) 223 (X), the mobile It operates at the timing of generating a new or renewed version of the lightweight MC learning model 122(X), and calculates the maturity level for each classification.

A procedure for calculating the maturity level by the common maturity level calculation unit 215 or the specialized maturity level calculation unit 315 will be described.

First, a method for calculating the learning data amount factor for maturity level will be explained. Regarding the category or subcategory (hereinafter referred to as class) c of the subject image, when the number of training data is n, the learning data amount factor function vol is a function of the number of training data n, and its value range is [0, 1] It is. As an example, a learning data amount factor function vol is defined as follows using an arctangent function.

Note that the parameter N is a standard learning data amount and is an integer of 1 or more, and the parameter a is a coefficient of an arctangent function and is a positive real value. This is just an example, but it is assumed that learning with half the amount of standard training data will grow to approximately 50% maturity, and depending on the value of a, the rise of the maturity curve will be gradual (for example, 0<a<1.0 ), exhibiting a sudden rise (for example, a>1.0). In other words, if there is insufficient training data, the level of maturity is low (approximately similar to saying that a certain amount of training data is required for human babies to understand/recognize things). Even if the number is large enough, it will not reach 100%, and will only approach the asymptote of maturity = 100% (even if you are an expert (doctor) in that field, even if you learn a very large amount of training data, it will not reach 100%). (an approximation that understanding/cognition is not possible).

Next, a method for calculating the learning data similarity factor at the learning maturity level will be explained. The learning data similarity factor function sim regarding class c is a function of training data X={x ₁ , x ₂ , . . . , x _n } belonging to class c, and its range is [0, 1]. This function takes the minimum value when the data have the lowest similarity, and the maximum value when the data have the highest similarity.

ただし、

なお、類似度関数ＳＩＭ（ｘ_１，ｘ_２）は、データペアｘ_１，ｘ_２の類似度が最高のとき１、最低のとき０となる値域［０，１］の関数である。 As an example, the learning data similarity factor function sim is defined as follows using the similarity function SIM. b is a coefficient of the similarity function and is in the range [0, 1].

however,

Note that the similarity function SIM(x ₁ , x ₂ ) is a function in the range [0, 1] that is 1 when the similarity of the data pair x ₁ , x ₂ is the highest and 0 when it is the lowest.

The learning maturity level is a function of the learning data amount factor and the learning data similarity factor. Generally, it can be expressed by the following formula.

である。本実施例では、共通成熟度算出部２１５は単純学習成熟度を算出する。 Among the learning maturity levels, the one defined by the learning data amount factor and the learning data similarity factor is called the simple learning maturity level. That is,

It is. In this embodiment, the common maturity level calculation unit 215 calculates the simple learning maturity level.

Then, the display unit of the mobile terminal 100 displays the image, the subject classification determined by the common image determination unit 211, and the maturity level calculated by the common maturity level calculation unit 215 or the specialized maturity level calculation unit 315. Presented along with the accuracy determined by the learning model. The presentation method can be to display the maturity level as a percentage as a numerical value, but it is also possible to display the maturity level for class C of the learning model, for example, from infant level to elementary school student, adult, and doctoral level, with illustrations to make it easier to understand. Possible things to do. (Figure 13)

FIG. 8 is a flowchart illustrating an example (normal time) of the sequential learning process processing of the image determination system S according to the second embodiment.

First, the user uses the imaging unit 130 of the mobile terminal 100 to capture an image of a subject (step S800).

The image judgment system S sequentially escalates as necessary based on each learning model (mobile lightweight learning model 121 → mobile lightweight MC learning model 122 → common learning model 221 → MC learning model 223 → specialized learning model 321). while determining the image (step S801).

The classification of the subject, which is the result of the determination process, is displayed on the display unit 150 of the mobile terminal 100 together with the accuracy of the classification. (Step S802). Then, USER-X determines whether there is a correct category among the displayed candidates (step S803). If the determination is affirmative, the program moves to step S804, and if the determination is negative, the program moves to step S807.

In step S804, a correct category is selected from the candidates displayed on the display unit 150.

Next, it is determined whether the learning maturity level of the selected class c is greater than or equal to θ and the accuracy is less than or equal to κ (step S805). If the determination is affirmative, the post-maturation feedback process 1 is executed in step S806. After that, the program moves to step S809.

On the other hand, in step S807, USER-X inputs the correct category, and in step S808, the post-maturation feedback process 2 is executed.

Next, the common server 200 stores the image data captured by the mobile terminal 100 and the category selected/input by USER-X in the common storage unit 220 (step S809). The common learning model generation unit 212 of the common server 200 then generates the common learning model 221 based on the data accumulated in step S809 (step S810).

Post-maturation feedback process 1, which is step S806 in FIG. 8, will be described with reference to FIG. 9.

First, in step S900, if the situation in which the maturity level of the class c is high (more than θ) but the certainty is low (less than κ) is extremely rare and does not occur very often, Since there is no need to correct the learning maturity function, the number of times such an event occurs is counted up. That is, 1 is added to the value of CNT(c) where the initial value CNT(c)=0.

Next, the value of CNT(c) exceeds a certain value M (for example, M may be a natural number such as 5 or 10 cases, or a value calculated as m% of the standard learning amount Nc). It is determined whether or not (step S901). If M is exceeded (YES in step S901), it is determined that this situation is not a rare case (an exceptional case), and the process proceeds to steps S903 to S905 for adjusting the learning maturity function. If it is less than or equal to M, the process ends (does nothing).

In step S902, 0 is assigned to CNT(c) to initialize it.

To change the learning proficiency function, the standard learning data Nc is increased (for example, by 2 times) in step S903, the coefficient a of vol is decreased (for example, by 1/2) in step S904, and the coefficient b of sim is decreased in step S905 ( For example, 1/2), the learning proficiency level is recalculated. Any one of these three steps may be performed, two may be selected and performed, or all three steps may be performed at the same time.

Generally, Nc is increased in S903. If it is determined that the learning curve has risen too quickly, S904 is executed to smooth the curve. If it is felt that the maturity level itself deviates from the image illustration, S905 is executed to reduce the absolute value of the maturity level itself. The mechanism that automatically determines which case applies is a mechanism that automatically grows through trial and error, such as trying S904 if increasing Nc in S903 several times has no effect. Just incorporate it.

Next, the post-maturation feedback process 2, which is step S808 in FIG. 8, will be described with reference to FIG. 10.

Since the class c input by USER-X is not a candidate selected by the image determination system s, it is necessary to check whether the learning maturity level of the class c is registered. Since the learning maturity level is calculated each time a learning model is generated, the learning maturity level of the corresponding learning model is referred to. (Step S1000)

Next, it is determined whether the learning maturity level has been registered and is greater than or equal to a certain value θ (step S1001). If it is θ or more (YES in step S1001), it means that the accuracy is low even though it is sufficiently mature (if the accuracy is high, it should be a selection candidate), so the process advances to step S1002. Perform post-maturation feedback process 1 to modify the learning maturity function. If it is less than θ, it is determined that learning has not progressed yet for the class c, and nothing is done.

According to this embodiment configured in this way, the common server 200 includes a common image determining unit 211 that determines the classification of the subject based on the image transmitted from the mobile terminal 100, and a common learning model 221 for each classification. The display unit 150 of the mobile terminal 100 displays the image, the classification of the subject determined by the common image determination unit 211, and the common maturity level calculation unit 215 that calculates the maturity level indicating the amount and diversity. The maturity level calculated by the maturity level calculation unit 215 is presented.

Therefore, according to the present embodiment, it is possible to determine whether the inference of subject image determination using the common learning model 221 is correct.

That is, in this embodiment, the data maturity level for each class is calculated based on the amount of training data and the diversity of the training data, and is presented to the user. This allows the user to know whether the common learning model 221 is making a correct (or incorrect) inference by chance, or is making a correct (or incorrect) inference based on sufficient learning. Based on the data maturity level and inference accuracy (accuracy), the user can judge the applicability of AI to the classification task.

As an example, as shown in FIG. 11, in the initial common learning model 221 (learning model A), the accuracy of the classification "bird" which is the correct answer in the judgment for the image taken of the subject is 0.10, and the maturity level is 0.03. However, in the common learning model 221 (learning model B) in which learning has progressed since then, the accuracy of the correct classification "bird" has improved to 0.50, and the maturity level has also improved to 0.50. Therefore, it can be seen that correct inferences are being made as learning progresses.

On the other hand, as shown in FIG. 12, in the initial common learning model 221 (learning model A), the accuracy of the classification "bird" which is the correct answer in the judgment for the image taken of the subject is 0.10, and the maturity level is 0.03. However, in the common learning model 221 (learning model B) in which learning has progressed since then, the accuracy of the correct classification "bird" remains at 0.10 and has not progressed, while the maturity level has improved to 0.50. Therefore, it can be seen that correct inference cannot be made due to the progress of learning. In this case, inference using a learning model is likely to be a difficult task.

As shown in Figure 13, in this example, the learning maturity level (depth of knowledge) of the artificial intelligence is visually presented for each learning object, and the extent to which humans should refer to the judgment results of the artificial intelligence is shown. By providing an indicator of whether something is good or not, it can contribute to people's final judgment.

In other words, just like human advisors and consultants, artificial intelligence has its strengths and weaknesses. Presenting only accuracy does not express the presence or absence of confidence. By showing the level of maturity as an indicator of how much learning has been done, the user can understand whether the artificial intelligence is trustworthy or not, and then ask a human to make the final decision.

Next, with reference to FIG. 14, an application to which the first embodiment and the second embodiment are applied will be described.

The application shown in FIG. 14 is an evaluation of a restaurant menu using an undercover investigator. For example, thousands of undercover investigators are asked to evaluate the menus of various restaurants for three months.

First, he registers as an undercover investigator by looking at job postings on the Internet. Next, go to your favorite restaurant, order your favorite menu, and take a photo of the menu that is brought to you. The image determination system S determines the menu (classification) based on the photographed image. The undercover investigator refers to the determination result of the image determination system S, selects and registers a menu. Thereafter, the undercover investigator eats, inputs an evaluation, and sends it to the server (common server 200/specialized server 300 or other server). Evaluation items include, for example, menu name, photo, price, appearance evaluation, taste, satisfaction (quantity), and personal impressions.

The undercover investigator will then be paid a survey fee (including meal allowance) based on the amount and information provided for the predetermined evaluation.

In this embodiment, the criteria for determining whether to use certain knowledge as the common teacher data set 222 or as the MC teacher data sets 123 and 224 (for mobile) will be explained. The judgment standard of the image judgment system S of this embodiment is hereinafter referred to as a sampling response correction method.

Taking the application shown in Figure 14 as an example, if a certain undercover investigator A, who has a wealth of knowledge about ramen, selects "Other" and inputs a new category as the correct answer, the MC teacher data set of this undercover investigator A and consider it as candidate data to be incorporated into the common teacher dataset.

At a certain timing, a request is made to 2n other undercover investigators (for example, undercover investigators B and C) to investigate the same question (for example, an image of "miso pork bone ramen"), and a different human answer is prompted.

If more than n people give the same answer as Masked Investigator A ("Miso Tonkotsu Ramen") out of the answers given by different masked investigators, the common teacher will Upgrade to data. On the other hand, if less than n people give a different answer from the answer given by another undercover investigator (just ``ramen''), then the answer of undercover investigator A will be transferred to the mobile phone for undercover investigator A. Treated as MC training data.

Embodiment 4 shows an example of determining whether the MC teacher data set for each user is unique to the user or whether it can be converted into common knowledge. FIG. 15 is a flowchart illustrating an example of the "sampling answer correction method" process in which answers of other users are aggregated to determine whether it is possible to convert them into common knowledge in the fourth embodiment.

First, if the answer of a certain USER-X is not among the answers of the mobile lightweight MC learning model 122 or the answers of the mobile lightweight learning model 121 ("other" is selected), the data collection unit 114 The inquiry is made to the MC learning model 223(X) and the common learning model 221 on the MC learning model 200, but if there is no answer there either (``Others'' is selected), the correct answer data is input by free input (step S1500). The input teacher data is temporarily stored in the mobile MC teacher data set 123(X) by the data collection unit 114.

Next, when a certain amount of data has been accumulated in the mobile MC teacher data set, a process for finding data that can be converted into common knowledge (confirmation based on answers from other people) is started (step S1501). That is, the common data collection unit 214 uses data sampled from the mobile MC teacher data set 123 (X) of USER-X to share the same problem with 2n (n = an integer greater than or equal to 1) users other than USER-X. The information is distributed from the server 200 to the mobile terminal 100, and responses are collected (step S1502).

Then, the common data collection unit 214 determines whether or not the answers of n or more people match the answers of USER-X (step S1503), and if the determination is affirmative (YES in step S1503), the mobile MC teacher The MC data stored in the data set 123(X) is deleted, upgraded and stored in the common teacher data set 222 (step S1504).

FIG. 16 is a flowchart illustrating an example of the "MC learning model correction method" process.

Once a certain amount of data in the mobile MC teacher data set has been accumulated, a process is started to automatically find data that can be converted into common knowledge from among the data (step S1600). That is, the common data collection unit 214 collects data sampled from the mobile MC teacher data set 123 (X) of the mobile terminal 100 owned by USER-X, and collects data sampled from the mobile MC teacher data set 123 (X) of the mobile terminal 100 owned by USER-X to 2n (n = 1 or more) other than the mobile terminal 100 of this USER-X. The same problem is solved by the mobile lightweight MC learning model 122 (an integer of ) and the MC learning model 223 on the common server 200, and the judgment results are totaled (step S1601).

Then, the common data collection unit 214 determines whether or not the answers of n or more models match (step S1602), and if the determination is affirmative (YES in step S1602), the mobile MC teacher data set 123 (X) The MC data stored in the MC data set 222 is deleted, and the MC data is upgraded and stored in the common teacher data set 222 (step S1603).

In this embodiment, one example of the learning model update/relearning/maintenance process will be shown. If a sufficient amount of data has been collected in the common teacher dataset 222 at the time of creating the common learning model 221, maintenance of the common learning model 221 is usually not necessary, but there is chronological nature (change over time) in the data. In this case, the common learning model 221 may become obsolete and its accuracy may decrease. To prevent this, incorporate a process to retrain (maintain the learning model) using the latest data. That is, the results determined in daily operations (results as determined by the learning model or results manually corrected by the user) are added to the common teacher data set 222.

Taking the application shown in Figure 14 as an example, if 3,000 undercover investigators (part-time workers) upload images and menu names of restaurants, for example, 500 people will be working per day, and each person will have 1 to 3 If you upload 1,000 pieces of data per day (2 pieces of data on average), 1,000 pieces of data will be collected per day. After 100 days of operation, 100,000 pieces of data will be accumulated, but due to CPU (GPU) and memory/execution time limitations in generating the learning model, it will not be possible to process all the data. Therefore, an upper limit on the number of data for creating (re-creating) a common learning model is set (for example, 10,000 items).

If the number of data items in the common teacher dataset 222 at the time of creating the mobile lightweight learning model (common) 121 is n items (for example, 10,000 items), the common learning model generation unit 212 generates data newly added during daily operation. Wait until the number of data items exceeds a certain number m items (for example, 10%: 1,000 items) (YES in step S1700), and at the time the number of data items exceeds a certain number m items (for example, 10%: 1,000 items). It is determined whether the time (t1) exceeds the time (t2) during which m pieces of data are newly accumulated in daily operations (step S1701).

If the determination is affirmative (YES in step S1701), the common learning model generation unit 212 thins out n+m pieces of original data to make the number of data such that t1<t2, and runs the relearning process. (Step S1702). Details of the relearning process will be described later with reference to FIGS. 18 and 19. On the other hand, if the determination is negative (NO in step S1701), the common learning model generation unit 212 runs the relearning process without thinning out the n pieces of original data (step S1703).

18 and 19 are flowcharts illustrating an example of a learning model update process, that is, a relearning process, of the image determination system S according to the fifth embodiment. The flowchart shown in FIG. 18 is a method for preparing teacher data for relearning based on the appearance frequency of each category of the common teacher data set 222, and the flowchart shown in FIG. This method prepares as much training data for relearning as possible.

In FIG. 18, it is predicted that the appearance frequencies of new data will be distributed from those with higher frequency to those with lower frequency according to a normal distribution or the like. If the data is added to the common teacher data set 222 with this frequency, minority data (low frequency of appearance) will not be utilized for learning. To avoid this, samples are gradually reduced so that about half of the labels that appear frequently and almost all of the data that appears less frequently (long tail) are added to the training data.

First, when the common learning model generation unit 212 accumulates a certain amount of daily operation data (for example, 1,000 items), if the determined result is the same as the result predicted by the common learning model 221 (for example, 950 items) and are arranged in descending order of appearance frequency for each category (step S1800).

Next, the common learning model generation unit 212 determines the percentage of data to be excluded from sampling so that the data in the category with the highest frequency of appearance becomes a certain percentage (for example, half) and the data in the category with the lowest frequency becomes 0%. Sample data to be incorporated into the common teacher data set 222 is randomly extracted while decreasing the ratio in descending order of frequency (eg, reduced to 450 out of 950) (step S1801).

Further, the common learning model generation unit 212 uses the sampling response correction method of the third embodiment or the sample response correction method of the third embodiment for the results determined in daily operation that are different from the common learning model 221 (for example, 50 cases). 4, it is determined whether or not to be incorporated into the common teacher dataset 222, and the remaining data (for example, 30 items) are extracted as sample data to be incorporated into the complete common teacher dataset 222 (step S1802). ).

Then, the common learning model generation unit 212 sets the upper limit of the population of the common teacher dataset 222 within the allowable range of calculation time as n items (for example, 10,000 items), and If adding the data (for example, 450 + 30 = 480) exceeds the upper limit n, the excess data is randomly deleted from the original common teacher dataset 222, new data is added (replaced), The relearning process is repeated to create a new common learning model 221 (step S1803).

In FIG. 19, it is predicted that the appearance frequencies of new data will be distributed from those with higher frequency to those with lower frequency according to a normal distribution or the like. On the other hand, the common teacher data set 222 used in the common learning model 221 is often limited to a certain number for each class so as not to be dragged down by classes with a large amount of data. In this case, it is better to cut the added data by a certain number. Data that is already small (long tail) is added as is so that it can be accumulated gradually.

First, when the common learning model generation unit 212 accumulates a certain amount of daily operation data (for example, 1,000 items), if the determined result is the same as the result predicted by the common learning model 221 (for example, 950 items) and are arranged in descending order of appearance frequency for each category (step S1900).

Next, the common learning model generation unit 212 randomly selects a certain number (for example, 20 cases) of data of categories whose appearance frequency is a certain number (for example, 20 cases) or more, and generates sample data to be incorporated into the common teacher data set 222. Extract. A small number of data (long tail data) that is less than a certain number is used as sample data (for example, reduced to 450 out of 950) (step S1901).

Furthermore, the common learning model generation unit 212 uses the sampling answer correction method of the third embodiment or the implementation Using the MC learning model correction method in Example 4, it is determined whether or not to be incorporated into the common teacher dataset 222, and the remaining data (for example, 30 items) are extracted as sample data to be incorporated into the complete common teacher dataset 222 (step S1902).

Then, the common learning model generation unit 212 sets the upper limit of the population of the common teacher data set 222 to n (for example, 10,000) within the allowable range of calculation time, and adds a new number to the common teacher data set 222. If the data added to (for example, 450 items + 30 items, 480 items) exceeds the upper limit of n items, the excess amount of data is randomly deleted from the original common teacher dataset 222, and new data is added ( (replacement), the relearning process is performed, and a new common learning model is created (step S1903).

Note that the configurations of the above-described embodiments are explained in detail in order to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to having all of the configurations described. Further, a part of the configuration of each embodiment can be added to, deleted from, or replaced with other configurations.

Further, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be partially or entirely realized in hardware by designing, for example, an integrated circuit. Further, the present invention can also be realized by software program codes that realize the functions of the embodiments. In this case, a storage medium on which a program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored on the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the embodiments described above, and the program code itself and the storage medium storing it constitute the present invention. Examples of storage media for supplying such program codes include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs (Solid State Drives), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, A non-volatile memory card, ROM, etc. are used.

Further, the program code for realizing the functions described in this embodiment can be implemented in a wide range of program or script languages such as assembler, C/C++, Perl, Shell, PHP, and Java (registered trademark).

Furthermore, by distributing the software program code that realizes the functions of the embodiment via a network, it can be stored in a storage means such as a computer's hard disk or memory, or a storage medium such as a CD-RW or CD-R. Alternatively, a processor included in the computer may read and execute the program code stored in the storage means or the storage medium.

In the above-described embodiments, the control lines and information lines are those considered necessary for explanation, and not all control lines and information lines are necessarily shown in the product. All configurations may be interconnected.

S... Image determination system, 100... Mobile terminal, 110... Control unit, 111... Image determination unit for mobile, 112... Imaging control unit, 113... Data transmission/reception unit, 114... Data collection unit, 120... Storage unit, 121... Mobile 122...Lightweight MC learning model for mobile, 123...MC teacher data set for mobile, 130...imaging section, 140...input section, 150...display section, 160, 230, 330...communication section, 200...common Server, 210...Common control unit, 211...Common image determination unit, 212...Common learning model generation unit, 213...Common data transmission/reception unit, 214...Common data collection unit, 215...Common maturity level calculation unit, 220...Common storage unit , 221... Common learning model, 222... Common teacher data set, 223... MC learning model, 224... MC teacher data set, 300... Specialized server, 310... Specialized control unit, 311... Specialized image determination unit, 312... Specialized learning model Generation unit, 313... Specialist data transmission/reception unit, 314... Specialist data collection unit, 315... Specialist maturity calculation unit, 320... Specialist storage unit, 321... Specialist learning model, 322... Specialist teacher data set

Claims (8)

An image determination system having a terminal, a common server and a specialized server that can communicate with the terminal,
The terminal is
an imaging unit that images a subject and outputs an image;
a storage unit storing a lightweight learning model;
an image determination unit that determines the classification of the subject using the lightweight learning model based on the image;
a display unit that presents the image and the classification of the subject determined by the image determination unit;
an input unit that accepts correction or additional input of the classification of the presented subject;
Sending the image to the common server, collecting corrections or additional inputs to the classification and storing it in the storage unit as a lightweight mass customization (hereinafter abbreviated as MC) teacher data set, and correcting or adding the classification. a data collection unit that sends input to the specialized server;
has
The common server is
a common storage unit storing a common learning model and a common teacher dataset;
a common data collection unit that generates the common teacher data set using the image transmitted from the terminal;
a common learning model generation unit that generates the common learning model using the common teacher data set, further generates the lightweight learning model by reducing the weight of the common learning model, and transmits the lightweight learning model to the terminal; has
The specialized server is
a specialized storage unit in which a specialized learning model and a specialized teacher data set are stored;
a specialized data collection unit that generates the specialized teacher data set using the modification of the classification or additional input sent from the terminal;
a specialized learning model generation unit that generates the specialized learning model using the specialized teacher data set;
An image determination system comprising: a specialized image determination section that determines the subject using the specialized learning model based on the image when the image determination section has difficulty in determining the subject. the data collection unit transmits corrections or additional inputs to the classification to the common server;
The common data collection unit generates an MC teacher data set using the correction or additional input of the classification transmitted from the terminal,
The common learning model generation unit generates an MC learning model using the MC teacher data set, further reduces the weight of the MC learning model to generate a lightweight MC learning model, and transmits this lightweight MC learning model to the terminal. send,
The image determination system according to claim 1, wherein the image determination unit determines the subject using the lightweight learning model and the lightweight MC learning model. The common server is
a common image determination unit that determines the classification of the subject based on the image transmitted from the terminal;
a maturity level calculation unit that calculates a maturity level indicating the amount and similarity of the common learning model for each of the classifications;
The display unit presents the image, the classification of the subject determined by the common image determination unit, and the maturity level calculated by the maturity level calculation unit. image judgment system. There are multiple terminals,
The MC teacher data set, the MC learning model, and the lightweight MC learning model are created for each terminal,
The common learning model generation unit transmits the image to which the additional input of the classification has been transmitted from the specific terminal to at least one other terminal, receives correction or additional input of the classification, and 3. The image determination system according to claim 2, wherein the image determination system determines whether or not to modify the common teacher data set based on a modification of the classification or an additional input transmitted from the system. There are multiple terminals,
The MC teacher data set, the MC learning model, and the lightweight MC learning model are created for each terminal,
The common learning model generation unit determines the image to which the additional input of the classification has been transmitted from the specific terminal using the MC learning model for each of at least one other terminal, and based on the determination result. The image determination system according to claim 2, further comprising determining whether or not to modify the common teacher data set. The common learning model generation unit recreates the common learning model when the number of data added to the common teacher data set based on the image transmitted from the terminal exceeds a predetermined number; If the time required to recreate the learning model exceeds the time required to add the predetermined number of data to the common teacher dataset, the time required to recreate the common learning model is to add the data to the common teacher dataset. The number of data items in the common teacher data set, which is the source for creating the common learning model, is adjusted so that the time required to re-create the common learning model does not exceed the predetermined number. 2. The image determination system according to claim 1, wherein if the time for adding the predetermined number of data to the common teacher data set is not exceeded, the number of data items in the common teacher data set is appropriately increased. The common learning model generation unit generates data of the classification that has the same result as predicted by the common learning model, among the data of the classification added to the common teacher data set based on the image transmitted from the terminal. are arranged in descending order of frequency of appearance, a predetermined proportion of data of the classification with the highest frequency of appearance is extracted, and the data of the classification with the lowest frequency of appearance is sorted from the data of the classification with the highest frequency of appearance so as not to extract the data of the classification with the lowest frequency of appearance. 2. The image determination system according to claim 1, wherein the common learning model is re-created by extracting the data by gradually decreasing the extraction ratio of the data. The common learning model generation unit generates data of the classification that has the same result as predicted by the common learning model, among the data of the classification added to the common teacher data set based on the image transmitted from the terminal. are arranged in descending order of appearance frequency, data of the above classifications whose appearance frequency is above a certain number are randomly extracted by the above certain number, and data of the above classifications whose appearance frequency is less than a certain number are extracted as is and used in the common learning model. The image determination system according to claim 1, wherein the image determination system performs re-creation.

Images