Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N20/00—Machine learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks

Definitions

• This disclosure relates to techniques for correcting semantic segmentation results.
• AI artificial intelligence
• SS Semantic segmentation
• Patent Document 1 It is important to evaluate the uncertainty of the prediction results, which are the output of a machine learning model, along with the accuracy of the prediction.
• the method described in Patent Document 1 is known as a method for quantifying uncertainty.
• Patent Document 1 describes the repeated application of Monte Carlo dropout to evaluate uncertainty.
• This disclosure relates to a technique for correcting semantic segmentation results based on uncertainty in the results.
• a representative example of the invention disclosed in this application is as follows: That is, a computer system comprising a processor, a storage device connected to the processor, and an input/output interface connected to the processor, and holding definition information of a prediction model that makes predictions for class classification that determines the class to which a pixel of an image belongs from among a plurality of classes, the prediction model being a machine learning model that calculates the probability that each pixel of the image belongs to each class, the processor receiving an input of an input image, transforming the prediction model, making the prediction for the input image using the transformed prediction model, and saving the prediction results, multiple times, and calculating the probability distribution of each class for each pixel of the input image based on the multiple prediction results.
• a distribution is calculated, and for each pixel of the input image, the probability distribution of each class is statistically processed to assign an uncertainty level to the class classification result of the pixel of the input image, and for each pixel of the input image, the class to which the pixel belongs is determined based on the probability distribution of each class, a base image representing the class to which each pixel of the input image belongs is generated, an uncertainty map representing the uncertainty level of each pixel of the input image is generated, pixels of the input image that require correction of the class classification result are selected based on the uncertainty map, and the base image is corrected by correcting the class classification result of the selected pixels of the input image using multiple prediction results.
• an image representing the results of class classification can be automatically corrected based on the uncertainty of the results of the class classification.
• FIG. 1 illustrates an example of the functional configuration of a computer system according to a first embodiment.
• FIG. 1 is a diagram illustrating an example of a hardware configuration of a computer that configures a computer system according to a first embodiment.
• FIG. 10 is a diagram illustrating the cooperation of functions in the learning phase of the computer system of the first embodiment.
• FIG. 2 is a diagram illustrating an example of a data structure of a teacher data DB according to the first embodiment.
• FIG. 2 is a diagram illustrating an example of a data structure of a model DB according to the first embodiment.
• FIG. 10 is a diagram illustrating an example of a data structure of a prediction result DB according to the first embodiment.
• FIG. 1 illustrates an example of the functional configuration of a computer system according to a first embodiment.
• FIG. 1 is a diagram illustrating an example of a hardware configuration of a computer that configures a computer system according to a first embodiment.
• FIG. 10 is a diagram illustrating the cooperation of
• FIG. 10 is a diagram illustrating an example of a data structure of a prediction result DB according to the first embodiment.
• FIG. 2 is a diagram illustrating an example of a data structure of an uncertainty assessment DB according to the first embodiment.
• FIG. 2 is a diagram illustrating an example of a data structure of an uncertainty assessment DB according to the first embodiment.
• FIG. 1 illustrates the characteristics of prediction in semantic segmentation.
• FIG. 1 illustrates the characteristics of prediction in semantic segmentation.
• FIG. 1 illustrates the characteristics of prediction in semantic segmentation.
• 10 is a flowchart illustrating an example of a prediction process that is executed in a learning phase by the computer system according to the first embodiment.
• FIG. 10 is a diagram illustrating an example of a probability distribution of classes calculated by the computer system according to the first embodiment.
• FIG. 10 is a flowchart illustrating an example of an uncertainty level determination process executed by the computer system of the first embodiment.
• 10 is a flowchart illustrating an example of an uncertainty level determination process executed by the computer system of the first embodiment.
• 10 is a flowchart illustrating an example of an uncertainty level determination process executed by the computer system of the first embodiment.
• FIG. 10 is a diagram illustrating an example of the correlation between the uncertainty index and the accuracy rate in the first embodiment.
• FIG. 10 is a diagram illustrating an example of the correlation between the uncertainty index and the accuracy rate in the first embodiment.
• FIG. 10 is a diagram illustrating an example of a method for calculating an evaluation index in the first embodiment.
• FIG. 10 is a diagram illustrating an example of boundary area processing executed by the computer system of the first embodiment.
• FIG. 10 is a diagram illustrating an example of boundary area processing executed by the computer system of the first embodiment.
• FIG. 10 is a diagram illustrating an example of boundary area processing executed by the computer system of the first embodiment.
• FIG. 2 is a diagram illustrating an example of a data structure of data generated by the computer system of the first embodiment.
• FIG. 2 is a diagram illustrating an example of a data structure of data generated by the computer system of the first embodiment.
• 10 is a flowchart illustrating an example of an uncertainty map generation process executed by the computer system according to the first embodiment.
• FIG. 10 is a diagram illustrating the cooperation of functions in a prediction phase of the computer system according to the first embodiment.
• FIG. 10 is a flowchart illustrating an example of a prediction process that is executed in a prediction phase by the computer system according to the first embodiment.
• 10 is a flowchart illustrating an example of a class classification process executed by the computer system of the first embodiment.
• 10 is a flowchart illustrating an example of a class classification process executed by the computer system of the first embodiment.
• 10 is a flowchart illustrating an example of a class classification process executed by the computer system of the first embodiment.
• FIG. 10 is a diagram illustrating an example of an image output in the classification process according to the first embodiment.
• FIG. 10 is a diagram illustrating an example of an image output in the classification process according to the first embodiment.
• FIG. 10 is a diagram illustrating an example of an image output in the classification process according to the first embodiment.
• FIG. 10 is a diagram illustrating an example of an image output in the classification process according to the first embodiment.
• FIG. 10 is a diagram illustrating an example of an image output in the classification process according to the first embodiment.
• FIG. 10 is a diagram illustrating an example of an image output in the classification process according to the first embodiment.
• FIG. 10 is a diagram illustrating an example of an image output in the classification process according to the first embodiment.
• FIG. 2 is a diagram illustrating an example of a GUI presented by the computer system of the first embodiment.
• FIG. 2 is a diagram illustrating an example of a GUI presented by the computer system of the first embodiment.
• FIG. 2 is a diagram illustrating an example of a GUI presented by the computer system of the first embodiment.
• 10 is a flowchart illustrating an example of a class classification process executed in a prediction phase by a computer system according to a second embodiment.
• FIG. 10 is a diagram showing an example of an uncertainty map according to the third embodiment.
• FIG. 1 is a diagram illustrating an example of the functional configuration of a computer system according to a first embodiment.
• FIG. 2 is a diagram illustrating an example of the hardware configuration of a computer that constitutes the computer system according to a first embodiment.
• the computer system 100 is composed of a computer 200 as shown in FIG. 2.
• the computer 200 includes a processor 201, a main memory device 202, a secondary memory device 203, and a network interface 204.
• the computer 200 may also include input devices such as a mouse and keyboard, and an output device such as a display.
• Computer system 100 has a learning unit 101, a prediction unit 102, an accuracy rate calculation unit 103, an uncertainty index calculation unit 104, an uncertainty level determination unit 105, and an input/output unit 106. Furthermore, computer system 100 stores a teacher data DB 110, a model DB 111, a prediction result DB 112, and an uncertainty assessment DB 113.
• the training data DB110 is a database that stores training data consisting of input data to be input into the machine learning model and correct prediction data.
• the input data is an image
• the correct prediction data is an array of classes to which the pixels belong.
• the model DB111 is a database that stores information that defines the machine learning model.
• the prediction result DB112 is a database that stores the results of predictions made using the machine learning model.
• the uncertainty assessment DB113 is a database that stores assessment information regarding the uncertainty of the class classification results based on the prediction results.
• each functional unit of the computer system 100 multiple functional units may be combined into a single functional unit, or a single functional unit may be divided into multiple functional units for each function.
• the databases held by the computer system 100 may be integrated into a single database, or a single database may be divided into multiple databases depending on the purpose of data management.
• Figure 3 is a diagram showing the collaboration of functions in the learning phase of the computer system 100 in Example 1.
• the learning unit 101 acquires the training data stored in the training data DB 110 as training data and performs the training process of a machine learning model to achieve semantic segmentation.
• the machine learning model is a model that calculates the probability that a pixel belongs to each class, and is, for example, a CNN (Convolutional Neural Network), SegNet, U-Net, or PSPNet.
• the machine learning method is, for example, deep learning. Note that the present invention is not limited to the type of machine learning model. Also, the present invention is not limited to the learning method of the machine learning model.
• the learning unit 101 stores the learning results in the model DB 111.
• the prediction unit 102 obtains training data stored in the training data DB 110 as validation data, and makes predictions using a machine learning model defined by information stored in the model DB 111. In this embodiment, an array of probabilities that pixels in the image belong to each class is output as the prediction result.
• the prediction unit 102 stores the prediction result in the prediction result DB 112.
• the prediction unit 102 uses the Monte Carlo dropout method to generate a machine learning model (hereinafter referred to as an MCD model) in which some of the multiple nodes that make up the machine learning model are dropped out, and predicts validation data using the MCD model. Predictions are performed multiple times for one piece of validation data using different MCD models.
• an MCD model machine learning model
• the accuracy rate calculation unit 103 uses the prediction results and the correct answer data included in the verification data to calculate the accuracy rate for the input data and the accuracy rate for each class.
• the accuracy rate calculation unit 103 stores the processing results in the prediction result DB 112.
• the uncertainty index calculation unit 104 calculates multiple uncertainty indexes using the prediction results.
• the uncertainty index calculation unit 104 stores the processing results in the uncertainty assessment DB 113. The calculated uncertainty indexes will be described later.
• the uncertainty level determination unit 105 uses multiple uncertainty indices to assign an uncertainty level to the class classification results of each pixel based on the prediction results for one piece of verification data.
• the uncertainty level determination unit 105 stores the processing results in the uncertainty assessment DB 113.
• the input/output unit 106 accepts input for the computer system 100 and outputs various data.
• Figure 4 is a diagram showing an example of the data structure of the teacher data DB 110 in Example 1.
• the teacher data DB 110 stores a table 400 for managing teacher data.
• the table 400 stores entries including a data ID 401, input data 402, and correct answer data 403. One entry corresponds to one piece of teacher data.
• Data ID 401 is a field that stores the ID of the training data.
• Input data 402 is a field that stores the input data to be input to the machine learning model.
• an array of pixel values of the pixels that make up the image is stored as input data.
• Correct answer data 403 is a field that stores correct answer data that represents the correct answer of the prediction.
• an array of values that represents the class to which the pixel belongs is stored as correct answer data.
• Figure 4 shows a two-dimensional array that indicates whether the pixel belongs to the first class or the second class.
• the data format of the teacher data DB110 is not limited to table format.
• Figure 5 is a diagram showing an example of the data structure of the model DB 111 in Example 1.
• Model DB 111 stores table 500, which stores information that defines a machine learning model.
• table 500 exists for one machine learning model.
• Table 500 stores entries including layer ID 501, layer type 502, and parameters 503.
• Layer ID 501 is a field that stores the ID of a layer of the machine learning model.
• Layer type 502 is a field that stores the type of layer. Layer types include input layer, convolutional layer, pooling layer, fully connected layer, and output layer.
• Parameter 503 is a field that stores the parameters of the layer corresponding to layer ID 501. Layer parameters are, for example, weights.
• model DB111 is not limited to table format.
• FIGS. 6A and 6B are diagrams showing an example of the data structure of the prediction result DB 112 in Example 1.
• Prediction result DB112 stores table 600 and table 610.
• Table 600 is a table for managing prediction results.
• Table 600 stores entries including a prediction result ID 601, a data ID 602, and a prediction result 603. There is one entry for each prediction result.
• Prediction result ID 601 is a field that stores the ID of the prediction result.
• Data ID 602 is a field that stores the ID of the input data used for prediction. In the learning phase, the ID of the verification data is stored, and in the prediction phase, the ID of the input data itself is stored.
• Prediction result 603 is a field that stores the prediction result. In this embodiment, an array of the probability that a pixel belongs to each class is stored as the prediction result. In Figure 6A, an array of the probability that a pixel belongs to the first class and the second class is stored. The sum of the probabilities of each class for a pixel is 1.
• Table 610 is a table for managing the accuracy rate of prediction results.
• Table 610 stores entries including a prediction result ID 611 and an accuracy rate 612. There is one entry for each prediction result.
• Prediction result ID 611 is the same field as prediction result ID 601.
• Accuracy rate 612 is a field that stores the accuracy rate.
• the accuracy rate for each pixel, the accuracy rate for the entire input data, the accuracy rate for each class, etc. are stored.
• the accuracy rate for the entire input data can be calculated, for example, as the average accuracy rate for each pixel.
• the accuracy rate for a class can be calculated, for example, as the average accuracy rate for pixels assigned the same class in the accuracy data.
• tables 600 and 610 may also be managed in a data format other than the table format.
• FIGS. 7A and 7B are diagrams showing an example of the data structure of the uncertainty assessment DB 113 in Example 1.
• Uncertainty assessment DB113 stores tables 700 and 710.
• Table 700 is a table for managing uncertainty indices used to evaluate the uncertainty of pixel class classification results based on predictions for input data.
• Table 700 stores entries including a data ID 701, a first uncertainty index 702, and a second uncertainty index 703. There is one entry for each piece of input data.
• Data ID 701 is the same field as data ID 602.
• First uncertainty index 702 and second uncertainty index 703 are fields that store uncertainty indices. Details of each uncertainty index will be described later.
• Table 710 is a table for managing the uncertainty level assigned to input data.
• Table 710 stores entries including a data ID 711 and an uncertainty level 712. There is one entry for each piece of input data.
• Data ID 711 is the same field as Data ID 602.
• Uncertainty level 712 is a field that stores the uncertainty level. In this embodiment, an array of pixel uncertainty levels is stored.
• tables 700 and 710 may also be managed in a data format other than the table format.
• Figures 8A, 8B, and 8C are diagrams that explain the characteristics of prediction in semantic segmentation.
• Figures 8A, 8B, and 8C show the distribution of probabilities that a pixel belongs to a class.
• semantic segmentation which classifies pixels into three classes, as an example.
• FIG. 9 is a flowchart illustrating an example of a prediction process performed by the computer system 100 of the first embodiment in the learning phase.
• FIG. 10 is a diagram illustrating an example of a probability distribution of classes calculated by the computer system 100 of the first embodiment.
• the prediction unit 102 obtains definition information for the machine learning model from the model DB 111 (step S101).
• the prediction unit 102 starts loop processing of the verification data (step S102). Specifically, the prediction unit 102 obtains one piece of verification data from the training data DB 110.
• the prediction unit 102 performs prediction of the verification data using the MCD model (step S104).
• the prediction unit 102 stores the prediction results in table 600 of the prediction result DB 112.
• the prediction unit 102 instructs the accuracy rate calculation unit 103 to calculate the accuracy rate.
• This instruction includes the ID of the prediction result.
• the accuracy rate calculation unit 103 compares the prediction result with the correct data included in the verification data and calculates the accuracy rate (step S105).
• the accuracy rate calculation unit 103 stores the calculation result in table 610 of the prediction result DB 112. Thereafter, the accuracy rate calculation unit 103 notifies the prediction unit 102 that the processing is complete.
• the prediction unit 102 predicts whether the number of times the prediction has been executed is less than a predetermined number (step S106). If the number of times the prediction has been executed is less than the predetermined number, the prediction unit 102 returns to step S103.
• the prediction unit 102 instructs the uncertainty index calculation unit 104 to calculate an uncertainty index.
• This instruction includes the ID of the verification data.
• the uncertainty index calculation unit 104 calculates the probability distribution of each class for each pixel (step S107). Specifically, the uncertainty index calculation unit 104 obtains the prediction results of the verification data from table 600 and calculates the probability distribution of each class for each pixel as shown in FIG. 10.
• the uncertainty index calculation unit 104 calculates a first uncertainty index for each pixel based on the probability distribution of each class (step S108). Specifically, the following processing is performed.
• the uncertainty index calculation unit 104 selects a pixel.
• the uncertainty index calculation unit 104 calculates the most frequent value of the probability distribution of each class for the selected pixel.
• the uncertainty index calculation unit 104 calculates the variance of the probability distribution of the class with the largest mode as the first uncertainty index.
• the uncertainty index calculation unit 104 determines whether processing has been completed for all pixels. If processing has not been completed for all pixels, the uncertainty index calculation unit 104 returns to S108-1. If processing has been completed for all pixels, the uncertainty index calculation unit 104 stores the calculation results in table 700 of the uncertainty assessment DB 113.
• the uncertainty index calculation unit 104 calculates a second uncertainty index for each pixel based on the probability distribution of each class (step S109). Specifically, the following processing is performed.
• the uncertainty index calculation unit 104 selects a pixel.
• the uncertainty index calculation unit 104 calculates the most frequent value of the probability distribution of each class for the selected pixel.
• the uncertainty index calculation unit 104 ranks the classes in descending order of mode.
• the uncertainty index calculation unit 104 calculates the sum of the most frequent values of the classes ranked third and above as the second uncertainty index.
• the uncertainty index calculation unit 104 determines whether processing has been completed for all pixels. If processing has not been completed for all pixels, the uncertainty index calculation unit 104 returns to S109-1. If processing has been completed for all pixels, the uncertainty index calculation unit 104 stores the calculation results in table 700 of the uncertainty assessment DB 113.
• the uncertainty index calculation unit 104 After calculating the second uncertainty index, the uncertainty index calculation unit 104 notifies the prediction unit 102 that processing is complete. The prediction unit 102 determines whether processing has been completed for all verification data (step S110).
• the prediction unit 102 If processing has not been completed for all verification data, the prediction unit 102 returns to step S102. If processing has been completed for all verification data, the prediction unit 102 ends the prediction process.
• 11A, 11B, and 11C are flowcharts illustrating an example of uncertainty level determination processing executed by the computer system 100 of Example 1.
• FIGS. 12A and 12B are diagrams illustrating an example of the correlation between the uncertainty index and the accuracy rate in Example 1.
• FIG. 13 is a diagram illustrating an example of a method for calculating an evaluation index in Example 1.
• FIGS. 14A, 14B, and 14C are diagrams illustrating an example of boundary area processing executed by the computer system 100 of Example 1.
• FIGS. 15A and 15B are diagrams illustrating an example of the data structure of data generated by the computer system 100 of Example 1.
• the uncertainty level determination unit 105 performs the processing described below on one piece of verification data.
• the uncertainty level determination unit 105 analyzes the correlation between the first uncertainty index and the accuracy rate for the verification data (step S201). Specifically, the following processing is performed:
• the uncertainty level determination unit 105 obtains the accuracy rate for each pixel from table 610, and also obtains the first uncertainty index for each pixel from table 700.
• the uncertainty level determination unit 105 analyzes the correlation between the accuracy rate and the first uncertainty index. For example, the analysis results shown in FIG. 12A are obtained.
• Graph 1201 shows the correlation between the accuracy rate and the first uncertainty index.
• the dotted bar graph shows the number of correct pixels for the first uncertainty index.
• the white bar graph shows the number of incorrect pixels for the first uncertainty index.
• the uncertainty level determination unit 105 calculates a threshold value Th1 based on the correlation between the first uncertainty index and the accuracy rate (step S202).
• the uncertainty level determination unit 105 uses correlation to calculate the uncertainty index at which the accuracy rate becomes a user-specified value as a threshold Th1.
• the threshold Th1 is 0.15.
• the uncertainty level determination unit 105 analyzes the correlation between the second uncertainty index and the accuracy rate for the verification data (step S203). Specifically, the following processing is performed:
• the uncertainty level determination unit 105 obtains the accuracy rate for each pixel from table 610, and also obtains the second uncertainty index for each pixel from table 700.
• the uncertainty level determination unit 105 analyzes the correlation between the accuracy rate and the second uncertainty index. For example, the analysis results shown in Figure 12B are obtained.
• Graph 1202 shows the correlation between the accuracy rate and the second uncertainty index.
• the dotted bar graph shows the number of correct pixels for the second uncertainty index.
• the white bar graph shows the number of incorrect pixels for the second uncertainty index.
• the uncertainty level determination unit 105 calculates the threshold value Th2 based on the correlation between the second uncertainty index and the accuracy rate (step S204). Specifically, the following processing is performed.
• the uncertainty level determination unit 105 calculates the change in the accuracy rate relative to the change in the second uncertainty index (the gradient of the decrease in the accuracy rate).
• the uncertainty level determination unit 105 calculates the maximum value of the second uncertainty index in the range where the gradient of decline in the accuracy rate is greatest as the threshold value Th2.
• the threshold value Th2 is 0.2.
• the uncertainty level determination unit 105 calculates, for each pixel, the difference ( ⁇ C) between the most frequent value (first value) of the class with the largest mode and the most frequent value (second value) of the class with the second largest mode (step S205).
• the uncertainty level determination unit 105 extracts, for each pixel, the combination of the class with the highest mode value and the class with the second highest mode value as the boundary type (step S206).
• the boundary type extracted is Class A/Class B.
• the boundary may also be extracted by targeting only pixels that have a probability distribution relationship such as that shown in Figure 8C.
• the uncertainty level determination unit 105 starts a loop process of boundary types (step S207). Specifically, the uncertainty level determination unit 105 selects one boundary type.
• the uncertainty level determination unit 105 sets an initial value for the variable k (step S208).
• the initial value is, for example, 0.1. Note that the initial value can be set arbitrarily.
• the uncertainty level determination unit 105 starts loop processing of pixels (step S209). Specifically, the uncertainty level determination unit 105 selects one pixel of the verification data.
• the uncertainty level determination unit 105 determines whether the ⁇ C of the pixel is smaller than the variable k (step S210). If the ⁇ C of each pixel in the prediction result is greater than or equal to the variable k, the uncertainty level determination unit 105 proceeds to step S212.
• the uncertainty level determination unit 105 changes the pixel output to the class with the second most frequent value (step S211). The uncertainty level determination unit 105 then proceeds to step S212.
• the output pixel class is determined to be the class with the largest mode, but for pixels where ⁇ C is smaller than the variable k, the mode is changed to the second most frequent class.
• Figure 14A is an image (classification result) generated by outputting the class with the largest mode.
• Figure 14B when ⁇ C for pixels in the black area (boundary) is smaller than 0.1, the output is changed to the class with the second most frequent mode, resulting in an image like that shown in Figure 14C.
• the dotted line indicates the boundary before the change.
• step S212 the uncertainty level determination unit 105 determines whether processing has been completed for all pixels (step S212). If processing has not been completed for all pixels, the uncertainty level determination unit 105 returns to step S209.
• the uncertainty level determination unit 105 calculates the accuracy rate by comparing the image generated from the output of each pixel with the image generated based on the correct data (step S213).
• the uncertainty level determination unit 105 records the processing results in table 1500 as shown in FIG. 15A.
• Table 1500 is stored in the uncertainty assessment DB 113.
• Table 1500 stores entries including boundary type 1501, variable k 1502, and accuracy rate 1503.
• step S212 the uncertainty level determination unit 105 adds an entry to table 1500, sets the selected boundary type to boundary type 1501 of the entry, sets the current value of variable k to variable k 1502, and sets the calculated accuracy rate to accuracy rate 1503.
• the uncertainty level determination unit 105 determines whether the variable k is less than 0.9 (step S213).
• 0.9 is the maximum value of the variable k. Note that the maximum value can be set arbitrarily.
• the uncertainty level determination unit 105 updates the variable k (step S215) and then returns to step S209. For example, it adds 0.1 to the value of the variable k.
• the uncertainty level determination unit 105 refers to table 1510 and determines the value of the variable k with the highest accuracy rate as the boundary type threshold Th3 (step S213).
• the uncertainty level determination unit 105 records the processing results in table 1510 as shown in FIG. 15B.
• Table 1510 is stored in the uncertainty assessment DB 113.
• Table 1510 stores entries including boundary type 1511, ratio 1512, and threshold value 1513.
• the uncertainty level determination unit 105 adds an entry to table 1510, sets the selected boundary type to boundary type 1511 of the entry, and sets the value of variable k with the highest accuracy rate to threshold value 1513.
• the uncertainty level determination unit 105 determines whether processing has been completed for all boundary types (step S217). If processing has not been completed for all boundary types, the uncertainty level determination unit 105 returns to step S207.
• the uncertainty level determination unit 105 calculates the proportion of each boundary type (step S218).
• the uncertainty level determination unit 105 counts the number of pixels that output the second most frequent class for each boundary type.
• the uncertainty level determination unit 105 divides the number of pixels of each boundary type by the total number of pixels of each boundary type.
• the uncertainty level determination unit 105 sets the calculation result as the ratio 1512 for each entry in the table 1510.
• the uncertainty level determination unit 105 starts loop processing of pixels (step S219). Specifically, the uncertainty level determination unit 105 selects one pixel of the verification data.
• the uncertainty level determination unit 105 determines whether the first uncertainty index of the pixel is smaller than threshold Th1 (step S220).
• the uncertainty level determination unit 105 assigns an uncertainty level of "0" (step S221). Specifically, the uncertainty level determination unit 105 sets the uncertainty level in table 710. The uncertainty level determination unit 105 then proceeds to step S228. An uncertainty level of "0" indicates that the class classification result is reliable.
• the uncertainty level determination unit 105 predicts whether the second uncertainty index of the pixel is greater than threshold Th2 (step S222).
• the uncertainty level determination unit 105 assigns an uncertainty level of "1" (step S223). Specifically, the uncertainty level determination unit 105 sets the uncertainty level in table 710. The uncertainty level determination unit 105 then proceeds to step S228. An uncertainty level of "1" indicates that the class classification is unreliable.
• the uncertainty level determination unit 105 calculates ⁇ C based on the probability distribution of each class of the pixel (step S224).
• the uncertainty level determination unit 105 identifies the type of boundary based on the probability distribution of each pixel class (step S225). Specifically, the uncertainty level determination unit 105 identifies the pair of the class with the highest mode and the class with the second highest mode as the type of boundary.
• the uncertainty level determination unit 105 determines whether ⁇ C is smaller than the threshold value Th3 for the identified boundary type (step S226).
• the uncertainty level determination unit 105 assigns an uncertainty level of "2" (step S227). Specifically, the uncertainty level determination unit 105 sets the uncertainty level in table 710. Then, the uncertainty level determination unit 105 proceeds to step S228. An uncertainty level of "2" indicates that the class cannot be determined because the pixel is located on the boundary between areas of different classes.
• step S228, the uncertainty level determination unit 105 determines whether processing has been completed for all pixels (step S228). If processing has not been completed for all pixels, the uncertainty level determination unit 105 returns to step S219. If processing has been completed for all pixels, the uncertainty level determination unit 105 ends the uncertainty level determination process.
• the uncertainty level determination unit 105 may set, for example, "-1" to pixels that have not been assigned any uncertainty level.
• FIG. 16 is a flowchart illustrating an example of the uncertainty map generation process executed by the computer system 100 of Example 1.
• the input/output unit 106 accepts the selection of verification data and uncertainty level from the user (step S301).
• the input/output unit 106 generates an uncertainty map that displays the uncertainty level of the selected verification data (step S302). Specifically, the following processing is performed:
• the input/output unit 106 references table 710 in the uncertainty assessment DB 113 and searches for an entry corresponding to the selected verification data.
• the input/output unit 106 identifies pixels that have been assigned the specified uncertainty level based on the searched entries.
• the input/output unit 106 generates an image of the same size as the input data (image) included in the verification data, and sets RGB values corresponding to the uncertainty level for identified pixels in the image. This image is the uncertainty map. Note that the uncertainty map may also be generated by superimposing the generated image with an image of class classification based on the input data or prediction results.
• the input/output unit 106 outputs the uncertainty map (step S303), and then terminates the uncertainty map generation process.
• Uncertainty levels can be assigned to the class classification results for input data contained in the training data, and the uncertainty levels can be presented as an image (uncertainty map). Users can refer to the uncertainty map to correct the correct data. By performing machine learning using the corrected training data, it is possible to improve the accuracy of machine learning model predictions and reduce uncertainty.
• Figure 17 is a diagram showing the coordination of functions in the prediction phase of the computer system 100 of Example 1.
• the prediction unit 102 acquires input data to be predicted and performs prediction using a machine learning model defined by information stored in the model DB 111.
• the prediction unit 102 stores the prediction results in the prediction result DB 112.
• the uncertainty index calculation unit 104 calculates multiple uncertainty indexes using the prediction results.
• the uncertainty index calculation unit 104 stores the processing results in the uncertainty assessment DB 113.
• the uncertainty level determination unit 105 uses multiple uncertainty indices to assign an uncertainty level to the class classification results of each pixel based on the prediction results of the input data to be predicted.
• the uncertainty level determination unit 105 stores the processing results in the uncertainty assessment DB 113.
• the input/output unit 106 accepts inputs to the computer system 100 and outputs various data from the computer system 100. For example, the input/output unit 106 accepts input extraction conditions, searches the prediction result DB 112 for prediction results that satisfy the extraction conditions, and displays an image using the prediction results. At this time, the input/output unit 106 displays the input data and an uncertainty map. The input/output unit 106 also accepts an image selected by the user, and, based on the image, modifies an image that represents the class classification results of each pixel based on the initial prediction results, and displays the modified image.
• FIG. 18 is a flowchart illustrating an example of prediction processing executed by the computer system 100 of Example 1 in the prediction phase.
• the prediction unit 102 obtains definition information for the machine learning model from the model DB 111 (step S401).
• the prediction unit 102 starts a prediction loop process (step S402).
• the process of step S402 is the same as the process of step S103.
• the prediction unit 102 uses the MCD model to perform prediction of the input data (image) to be predicted (step S403).
• the processing in step S403 is the same as the processing in step S104.
• the prediction unit 102 determines whether the number of times the prediction has been performed is less than a predetermined number (step S404).
• the processing in step S404 is the same as the processing in step S106.
• the prediction unit 102 instructs the uncertainty index calculation unit 104 to calculate an uncertainty index.
• This instruction includes the ID of the input data.
• the uncertainty index calculation unit 104 calculates the probability distribution of each class for each pixel (step S405).
• the processing in step S405 is the same as the processing in step S107.
• the uncertainty index calculation unit 104 calculates a first uncertainty index for each pixel based on the probability distribution of each class (step S406).
• the processing of step S406 is the same as the processing of step S108.
• the uncertainty index calculation unit 104 calculates a second uncertainty index for each pixel based on the probability distribution of each class (step S407).
• the processing of step S407 is the same as the processing of step S109.
• the threshold used to assign the uncertainty level is the threshold determined in the learning phase. For example, the average value of the thresholds for each validation data is used.
• FIGS. 19A, 19B, and 19C are flowcharts illustrating an example of class classification processing executed by the computer system 100 of Example 1.
• FIG. 20A, 20B, 20C, 20D, 20E, and 20F are diagrams illustrating an example of images output in the class classification processing of Example 1.
• the input/output unit 106 accepts input data and a selection of uncertainty level from the user (step S501).
• the input/output unit 106 obtains the probability distribution of the class of each pixel in the input data, and also obtains information on the uncertainty level of the input data from table 710 in the uncertainty assessment DB 113 (step S502).
• the input/output unit 106 generates a base image that displays the results of the class classification based on the probability distribution of the class for each pixel in the input data (step S503). Specifically, the input/output unit 106 generates an image (base image) that outputs the class with the largest mode of the probability distribution for each pixel. For example, a base image like the one shown in Figure 20A is generated.
• Figure 20A is an example of an image classified into three classes. The shaded area is a group of pixels classified into class C, the black area is a group of pixels classified into class B, and the white area is a group of pixels classified into class A.
• the input/output unit 106 generates an uncertainty map that displays the uncertainty level of the selected input data (step S504).
• the processing of step S504 is the same as the processing of step S302.
• uncertainty maps such as those shown in Figures 20B and 20C are generated.
• Figure 20B is an uncertainty map when uncertainty level "1" is selected
• Figure 20C is an uncertainty map when uncertainty level "2" is selected.
• the area of pixels to which uncertainty level "1" is assigned is referred to as a doubtful area
• the area of pixels to which uncertainty level "2" is assigned is referred to as a boundary area.
• the input/output unit 106 generates an uncertainty map that allows each uncertainty level to be distinguished.
• the input/output unit 106 outputs the base image and the uncertainty map (step S505). After that, the input/output unit 106 transitions to a wait state (step S506) and waits for input from the user.
• the user refers to the base image and uncertainty map to determine whether or not the base image needs to be modified. If the user determines that the questionable region needs to be modified, the user inputs a first modification request to the computer system 100. If the user determines that the boundary region needs to be modified, the user inputs a second modification request to the computer system 100. If the user determines that no modification is necessary, the user inputs an end request to the computer system 100.
• the input/output unit 106 When the input/output unit 106 receives input from the user, it determines whether the input is a first revision request (step S507).
• the input/output unit 106 determines whether the input is a second revision request (step S508).
• the input/output unit 106 terminates the classification process.
• the correction process for the suspicious area is initiated.
• the input/output unit 106 presents a screen for inputting correction conditions and accepts the correction conditions from the user (step S509).
• the correction conditions are the number of suspicious areas and candidate images to be targeted.
• the input/output unit 106 searches the prediction result DB 112 for prediction results based on the correction conditions, and displays candidate images based on the prediction results (step S510). Specifically, the following processing is performed.
• the input/output unit 106 identifies the class with the largest area in the suspicious region of the base image. In other words, the class with the largest number of pixels is identified.
• the input/output unit 106 references table 600 in the prediction result DB 112 and counts the number of pixels in the suspicious region that have the highest probability value for the identified class. The input/output unit 106 obtains a specified number of prediction results in descending order of the number of pixels.
• the input/output unit 106 generates and displays candidate images based on the obtained prediction results. Specifically, the input/output unit 106 generates candidate images so as to output the class with the highest probability value for each pixel. For example, a candidate image like the one shown in Figure 20D is generated.
• the user selects the candidate image that they believe to be the correct one from the candidate images.
• the input/output unit 106 accepts the candidate image selection from the user (step S511).
• the input/output unit 106 modifies and displays the base image based on the selected candidate image (step S512).
• the input/output unit 106 then returns to step S506. Specifically, the input/output unit 106 changes the class to which the pixel included in the suspicious region of the base image belongs to the class with the highest probability value of the suspicious region of the selected candidate image. For example, if the candidate image shown in Figure 20D is selected for a base image such as that shown in Figure 20A, the base image is modified as shown in Figure 20E.
• the boundary area modification process begins.
• the input/output unit 106 identifies the boundary types in the base image and calculates the proportions of each boundary type (step S513).
• the process of calculating the boundary proportions is the same as the process in step S218.
• the input/output unit 106 obtains the threshold value Th3 for each boundary type from table 1510 of the uncertainty assessment DB 113 (step S514). The input/output unit 106 then presents an interface for selecting the target boundary region.
• step S516 When the input/output unit 106 receives a selection of a target boundary area from the user (step S515), it modifies and displays the display of the boundary area (step S516). The input/output unit 106 then enters a wait state (step S517). The following process is executed in step S516.
• the input/output unit 106 identifies the boundary type of the selected boundary area.
• the type of boundary area can be identified based on the probability distribution of the classes of pixels included in the boundary area.
• the input/output unit 106 selects one pixel included in the selected boundary area.
• the input/output unit 106 determines whether the ⁇ C of the selected pixel is smaller than the threshold value Th3 of the identified boundary type. If the ⁇ C of the selected pixel is smaller than the threshold value Th3 of the identified boundary type, the input/output unit 106 changes the output of the pixel to the class with the second most frequent value. Note that the class may also be changed to the third or subsequent most frequent value.
• the input/output unit 106 determines whether processing has been completed for all pixels included in the selected boundary region. If processing has not been completed, the input/output unit 106 returns to S516-2.
• the input/output unit 106 modifies the base image based on the processing results for each pixel included in the selected boundary area, and generates a candidate image.
• the base image is modified as shown in Figure 20F.
• the user refers to the corrected base image and determines whether or not the boundary area needs to be corrected. If it is determined that the boundary area needs to be corrected, the user inputs a correction instruction including a new threshold value Th3. If it is determined that the boundary area does not need to be corrected, the user inputs a completion instruction.
• the input/output unit 106 When the input/output unit 106 receives input from the user, it determines whether the input is a correction instruction (step S518).
• step S516 If the received input is a correction instruction, the input/output unit 106 returns to step S516. If the received input is a completion instruction, the input/output unit 106 returns to step S506.
• FIGS. 21, 22, and 23 are diagrams showing examples of GUIs presented by the computer system 100 of Example 1.
• GUI 2100 is presented by input/output unit 106.
• GUI 2100 includes a setting area 2101 and a display area 2102.
• the settings area 2101 is an area for making various settings required for processing.
• the settings area 2101 includes input fields 2110 and 2111, a selection field 2112, and an operation button 2114.
• Input field 2110 is a field for entering the machine learning model to be used. For example, the name of table 400 that defines the machine learning model is entered here.
• Input field 2111 is a field for entering input data. For example, the name of the input data is entered here.
• Selection field 2113 is a field for selecting the uncertainty level to display on the uncertainty map. Selection field 2113 displays check boxes for selecting the uncertainty level to display.
• Operation button 2114 is an operation button for instructing the execution of class classification processing.
• the display area 2102 is an area for displaying the processing results.
• the display area 2102 includes a display field 2120 and operation buttons 2121, 2122, and 2123.
• Display field 2120 displays input data 2131, base image 2132, and uncertainty map 2133.
• Uncertainty map 2133 is displayed as an image with RGB values set according to the uncertainty level.
• Operation button 2121 is an operation button for inputting a first revision request.
• Operation button 2122 is an operation button for inputting a second revision request.
• Operation button 2123 is an operation button for inputting a termination request.
• the GUI 2200 includes a setting area 2201 and a display area 2202.
• the setting area 2201 is an area for setting correction conditions.
• the setting area 2201 includes a selection field 2210, an input field 2211, operation buttons 2212, and an extraction condition input field 2213.
• the selection field 2210 is a field for selecting the target suspicious area.
• the input field 2211 is a field for inputting the number of candidate images.
• the operation button 2212 is an operation button for instructing the generation of candidate images.
• the extraction condition input field 2213 is a field for inputting the extraction conditions for candidate images.
• Display area 2202 is an area that displays the modification results of the candidate image and the base image.
• Display area 2202 includes display fields 2220 and 2222, and operation buttons 2221 and 2223.
• Display field 2220 is a field that displays candidate images.
• Operation button 2221 is an operation button for instructing correction of the base image. The user selects a candidate image from display field 2220 and operates operation button 2221.
• Display field 2222 is a field that displays the correction results of the base image.
• Operation button 2223 is an operation button for instructing completion of correction of the questionable area.
• the GUI 2300 includes a setting area 2301 and a display area 2302.
• the setting area 2301 includes a selection field 2310, an input field 2311, and an operation button 2312.
• the selection field 2310 is a field for selecting the target boundary region.
• the input field 2311 is a field for inputting the threshold value Th3. Note that the input field 2311 may be left blank. If the input field 2311 is blank, the threshold value Th3 obtained from the uncertainty assessment DB 113 will be used.
• the operation button 2312 is an operation button for instructing correction of the base image.
• the display area 2302 is an area where the results of modifying the base image are displayed.
• the display area 2302 includes a display field 2320 and operation buttons 2321.
• Display field 2320 is a field that displays the results of the base image correction.
• Operation button 2321 is an operation button for instructing completion of the boundary area correction.
• the computer system 100 of Example 1 assigns an uncertainty level to the class classification results of each pixel based on the prediction results of the input data and displays an uncertainty map. By referring to the uncertainty map, the user can determine whether or not the base image needs to be modified. Furthermore, when the computer system 100 receives a modification instruction from the user, it can automatically modify the base image using the uncertainty map. This reduces the effort required to modify the class classification results based on the prediction results.
• the median and mean of the probability distribution may be used instead.
• classification process may also be performed during the learning phase. This reduces the effort required to correct the correct answer data in the training data.
• Example 2 the computer system 100 automatically modifies the base image based on the uncertainty level. Below, Example 2 will be explained, focusing on the differences from Example 1.
• the configuration of the computer system 100 in Example 2 is the same as that in Example 1.
• the data structures of the various databases in Example 2 are the same as those in Example 1.
• the processing performed by the computer system 100 in Example 2 in the learning phase is the same as that in Example 1.
• Example 2 The processing performed by the computer system 100 in the prediction phase in Example 2 differs in part from Example 1. Specifically, the class classification processing is partially different. The other processing is the same as Example 1.
• FIG. 24 is a flowchart illustrating an example of the class classification process performed by the computer system 100 of Example 2 in the prediction phase.
• steps S601 to S604 is the same as the processing from steps S501 to S504.
• the input/output unit 106 modifies the base image based on the uncertainty map (step S605). For example, the following processing is performed:
• the input/output unit 106 identifies the class with the largest area (number of pixels) in the classification results based on the prediction results. For example, in the case of the base image in Figure 20A, class A (white area) is identified as the class with the largest area.
• the input/output unit 106 searches the prediction result DB 112 for the prediction result in which the identified class is the largest number of output pixels in the suspicious region.
• the input/output unit 106 corrects the suspicious region in the base image based on the output of the suspicious region in the searched prediction result.
• the correction method is the same as in step S510.
• the input/output unit 106 modifies the base image by changing the output of pixels in each boundary region where ⁇ C is smaller than the boundary type threshold Th3 to the class with the second most frequent value. Note that the class with the third most frequent value or higher may also be changed.
• the input/output unit 106 calculates the accuracy rate for each class in the learning phase. For example, the input/output unit 106 calculates the average accuracy rate for each class based on the accuracy rate for each class in each entry of table 610.
• the input/output unit 106 displays a message recommending that the user check the corrected base image. For example, if a new class area is added to the questionable area and the accuracy rate of that class is lower than the threshold, a warning message is displayed.
• the input/output unit 106 displays the base image and the modified base image (step S606), and the prediction process ends.
• Example 2 no input work by the user is required, which further reduces the effort required to correct the results of class classification based on prediction results.
• Example 3 the method for displaying the uncertainty map is different. Below, Example 3 will be explained, focusing on the differences from Example 1.
• Example 3 The configuration of the computer system 100 in Example 3 is the same as in Example 1.
• the data structures of the various databases in Example 3 are the same as in Example 1.
• the processes other than the uncertainty map generation process performed by the computer system 100 in the learning phase and prediction phase in Example 3 are the same as in Example 1.
• Example 3 the uncertainty map generation process is partially different. Specifically, the processing content of step S302 differs from Example 1.
• the input/output unit 106 divides the input data into multiple partial images. The number of divisions is set in advance.
• the input/output unit 106 selects one uncertainty level from the uncertainty levels specified by the user.
• the input/output unit 106 selects a partial image.
• the input/output unit 106 calculates the area ratio of pixels assigned the selected uncertainty level. Specifically, it divides the number of pixels assigned the uncertainty level by the total number of pixels in the partial image.
• the input/output unit 106 predicts whether the area ratio is greater than a threshold value Th4.
• Threshold value Th4 is a preset threshold value. If the area ratio is equal to or less than threshold value Th4, the input/output unit 106 proceeds to S302-7.
• the input/output unit 106 determines whether processing has been completed for all partial images. If processing has not been completed for all partial images, the input/output unit 106 returns to S302-3.
• the input/output unit 106 determines whether processing has been completed for all uncertainty levels specified by the user. If processing has not been completed for all uncertainty levels specified by the user, the input/output unit 106 returns to S302-2.
• Figure 25 shows an example of an uncertainty map for Example 3.
• Example 3 by displaying only the partial image with a large number of pixels assigned the specified uncertainty level, the effort required for correction can be reduced.
• an uncertainty map may be output regardless of the area ratio.
• the present invention can also provide machine learning models that use data other than images, such as text and spectra, as input.
• the above-mentioned configurations, functions, processing units, processing means, etc. may be implemented in part or in whole in hardware, for example by designing them as integrated circuits.
• the present invention can also be realized by software program code that realizes the functions of the embodiments.
• a storage medium on which the program code is recorded is provided to a computer, and a processor in the computer reads the program code stored on the storage medium.
• the program code read from the storage medium itself realizes the functions of the above-mentioned embodiments, and the program code itself and the storage medium on which it is stored constitute the present invention.
• Examples of storage media for supplying such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs (Solid State Drives), optical disks, magneto-optical disks, CD-Rs, magnetic tape, non-volatile memory cards, and ROMs.
• program code that realizes the functions described in this embodiment can be implemented in a wide range of programming or scripting languages, such as assembler, C/C++, Perl, Shell, PHP, Python, and Java.
• the software program code that realizes the functions of the embodiments may be distributed over a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the computer's processor may read and execute the program code stored in the storage means or storage medium.
• a storage means such as a computer's hard disk or memory
• a storage medium such as a CD-RW or CD-R
• control lines and information lines shown are those considered necessary for explanation, and not all control lines or information lines in the product are necessarily shown. All components may be interconnected.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Software Systems (AREA)
• General Physics & Mathematics (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Physics & Mathematics (AREA)
• Evolutionary Computation (AREA)
• Computing Systems (AREA)
• Medical Informatics (AREA)
• Artificial Intelligence (AREA)
• General Health & Medical Sciences (AREA)
• Databases & Information Systems (AREA)
• Health & Medical Sciences (AREA)
• Multimedia (AREA)
• Data Mining & Analysis (AREA)
• General Engineering & Computer Science (AREA)
• Mathematical Physics (AREA)
• Image Analysis (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=96772671

Family Applications (1)

Country Status (2)

Citations (2)

• 2024
  • 2024-02-13 JP JP2024019750A patent/JP2025123944A/ja active Pending
  • 2024-10-28 WO PCT/JP2024/038411 patent/WO2025173318A1/ja active Pending

Patent Citations (2)

Also Published As

Similar Documents

Legal Events