JP7096360B2 - Mini-batch learning device and its operation program and operation method - Google Patents

Description

The techniques of the present disclosure relate to a mini-batch learning device and its operating program and operating method.

Semantic segmentation is known, in which a plurality of classes in an image are discriminated on a pixel-by-pixel basis. Semantic segmentation is realized by a machine learning model (hereinafter, simply a model) such as a U-shaped convolutional neural network (U-Net; U-Shaped Neural Network).

In order to improve the discrimination accuracy of the model, it is necessary to give the model learning data to learn and update the model. The learning data is composed of a learning input image and an annotation image in which a class in the learning input image is manually specified. In JP-A-2017-107386, one learning input image that is the source of the annotation image is extracted from a plurality of learning input images.

There is a method called mini-batch learning for learning. In mini-batch learning, mini-batch data is given to the model as training data. The mini-batch data is a part of a plurality of divided images (for example, 10,000 divided images divided by a frame having a size of 1/100 of the original image) obtained by dividing the training input image and the annotation image (for example). 100 sheets). Multiple sets (for example, 100 sets) of mini-batch data are generated, and each set is sequentially given to the model.

Here, consider the case where there is a class bias in the learning input image and the annotation image. For example, the input image for learning is an image showing the state of cell culture with a phase-contrast microscope, and class 1 is classified as differentiated cells, class 2 is classified as undifferentiated cells, class 3 is classified as a medium, and class 4 is classified as dead cells. It is an image. When the area ratio of each class in the input image for learning and the annotation image is 38% of differentiated cells, 2% of undifferentiated cells, 40% of medium, and 20% of dead cells, and the area ratio of undifferentiated cells is relatively low. Is.

If the training input image and the annotation image are biased in class in this way, there is a high possibility that the mini-batch data composed of the training input image and the annotation image will also be biased in class. If there is a class bias in the mini-batch data, learning is performed without taking into account the rare classes with a relatively low area ratio. As a result, a model with low discrimination accuracy of the rare class is created.

In JP-A-2017-107386, as described above, one learning input image that is the source of the annotation image is extracted from the plurality of learning input images. However, with this method, if there is a class bias in all of the plurality of learning input images, a model with low discrimination accuracy of the rare class will be created in the end. Therefore, the method described in JP-A-2017-107386 cannot solve the problem that a model having a low discrimination accuracy of a rare class is created.

It is an object of the present disclosure technique to provide a mini-batch learning device capable of suppressing a decrease in discrimination accuracy of a class of a machine learning model for performing semantic segmentation, and an operation program and operation method thereof.

In order to achieve the above object, the mini-batch learning device of the present disclosure provides mini-batch data to a machine learning model for performing semantic segmentation that discriminates a plurality of classes in an image on a pixel-by-pixel basis, and trains the mini-batch. A calculation unit that calculates the ratio of the first area of each of a plurality of classes to the area of the entire annotation image among the learning input image and the annotation image that are the learning devices and are the source of the mini-batch data, and the first area. A specific part that identifies a rare class whose ratio is lower than the first set value, and a generation part that generates mini-batch data from a learning input image and an annotation image, and the second area ratio of the rare class in the mini-batch data is A generation unit for generating mini-batch data having a second set value or more larger than the first area ratio calculated by the calculation unit is provided.

It is preferable that the generation unit is provided with a reception unit that receives a selection instruction as to whether or not to perform a process of generating mini-batch data in which the second area ratio is equal to or greater than the second set value.

The generation unit generates a plurality of mini-batch data according to a certain rule, and among a plurality of mini-batch data generated according to a certain rule, the mini-batch data in which the second area ratio is equal to or more than the second set value is generated. , Preferably selected for use in learning.

The generator can detect rare class uneven distribution areas and non-uneven distribution areas in the annotation image, and increase the number of cuts of the image that is the source of the mini-batch data of the uneven distribution areas to be larger than the number of cuts of the non-uneven distribution areas. preferable.

The operation program of the mini-batch learning device of the present disclosure is an operation program of a mini-batch learning device that gives mini-batch data to a machine learning model for performing semantic segmentation that discriminates a plurality of classes in an image on a pixel-by-pixel basis. The calculation unit that calculates the first area ratio of each of the plurality of classes to the area of the entire annotation image among the learning input image and the annotation image that are the sources of the mini-batch data, and the first area ratio are It is a specific part that specifies a rare class lower than the first set value, and a generation part that generates mini-batch data from the input image for learning and the annotation image, and the second area ratio of the rare class in the mini-batch data is the calculation part. The computer is made to function as a generation unit for generating mini-batch data having a second set value or more larger than the first area ratio calculated in.

The method of operating the mini-batch learning device of the present disclosure is a method of operating a mini-batch learning device in which mini-batch data is given to a machine learning model for performing semantic segmentation in which a plurality of classes in an image are discriminated on a pixel-by-pixel basis. The calculation step for calculating the first area ratio of each of the plurality of classes to the area of the entire annotation image among the training input image and the annotation image which are the sources of the mini-batch data, and the first area ratio are It is a specific step to specify a rare class lower than the first set value, and a generation step to generate mini-batch data from a training input image and an annotation image, and the second area ratio of the rare class in the mini-batch data is a calculation step. A generation step for generating mini-batch data having a second set value or more larger than the first area ratio calculated in the above is provided.

According to the technique of the present disclosure, it is possible to provide a mini-batch learning device capable of suppressing a decrease in discrimination accuracy of a class of a machine learning model for performing semantic segmentation, and an operation program and operation method thereof.

It is a figure which shows the outline of the mini-batch learning apparatus and its processing. It is a figure which shows the outline of an operation apparatus and its processing. It is a figure which shows the image, FIG. 3A shows an input image for learning, and FIG. 3B shows an annotation image. It is a figure which shows the state which generates the input image for division learning from the input image for learning. It is a figure which shows the state which the division annotation image is generated from the annotation image. It is a figure which shows that a part of the input image for division learning constitutes the input image group for division learning. It is a figure which shows that the division annotation image group is formed by a part of a plurality of division annotation images. It is a block diagram which shows the computer which comprises the mini-batch learning apparatus. It is a block diagram which shows the processing part of the CPU of the mini-batch learning apparatus. It is a figure which shows the specific example of the processing of a calculation part and a specific part. It is a figure which shows the specific example of the processing of a generation part. It is a flowchart which shows the processing procedure of a mini-batch learning apparatus. It is a figure which shows the 2nd Embodiment which asks whether or not the generation part may perform the process of generating the mini-batch data whose 2nd area ratio is equal to or more than the 2nd set value. It is a figure which shows the 3rd Embodiment which selects the mini-batch data whose 2nd area ratio is not more than the 2nd set value among a plurality of mini-batch data generated according to a certain rule for learning. It is a figure which shows the 4th Embodiment which makes the number of cuts of the image which is the source of the mini-batch data of a rare class uneven distribution area in an annotation image larger than the number of cuts of a non-unevenly distributed area.

[First Embodiment]
In FIG. 1, the mini-batch learning device 2 uses mini-batch data 11 for the model 10 in order to improve the discrimination accuracy of the model 10 for performing semantic segmentation for discriminating a plurality of classes in an input image on a pixel-by-pixel basis. Have them do the mini-batch learning that they had. The mini-batch learning device 2 is, for example, a desktop personal computer. The model 10 is, for example, U-Net.

The class may be paraphrased as the type of the object reflected in the input image. In short, the semantic segmentation discriminates the class of the object reflected in the input image and its contour, and the model 10 outputs the discriminant result as an output image. For example, if the input image shows three objects, a cup, a book, and a mobile phone, the output image ideally distinguishes the cup, book, and mobile phone as classes, and faithfully outlines these objects. The contour line traced will be drawn on each object.

The discrimination accuracy of the class of the model 10 is enhanced by giving the model 10 learning data to train the model 10 and updating the model 10. The learning data is composed of a set of a learning input image to be input to the model 10 and an annotation image in which the class in the learning input image is manually specified. The annotation image is an image for matching the answer with the learning output image output from the model 10 according to the learning input image, and is compared with the learning output image. The higher the discrimination accuracy of the class of the model 10, the smaller the difference between the annotation image and the learning output image.

In the mini-batch learning device 2, as described above, the mini-batch data 11 is used as the learning data. The mini-batch data 11 is composed of a division learning input image group 12 and a division annotation image group 13.

In the mini-batch learning, the input image group 12 for division learning is given to the model 10. As a result, the learning output image is output from the model 10 for each of the divided learning input images 20S (see FIG. 4) of the divided learning input image group 12. The learning output image group 14 which is a set of learning output images output from the model 10 in this way is compared with the divided annotation image group 13, and the discrimination accuracy of the class of the model 10 is evaluated. Then, the model 10 is updated according to the evaluation result of the discrimination accuracy of this class. The mini-batch learning device 2 inputs the input image group 12 for division learning to the model 10 and outputs the output image group 14 for learning from the model 10, evaluates the discrimination accuracy of the class of the model 10, and updates the model 10. Is performed while substituting the mini-batch data 11, and the process is repeated until the discrimination accuracy of the class of the model 10 reaches a desired level.

As shown in FIG. 2, the model 10 whose class discrimination accuracy has been raised to a desired level as described above is incorporated into the operation device 15 as a trained machine learning model (hereinafter referred to as a trained model) 10T. The trained model 10T is given an input image 16 in which the class of the reflected object and its contour have not yet been determined. The trained model 10T discriminates the class of the object reflected in the input image 16 and its contour, and outputs the output image 17 as the discriminating result. Similar to the mini-batch learning device 2, the operation device 15 is, for example, a desktop personal computer, and the input image 16 and the output image 17 are displayed side by side on the display. The operation device 15 may be a device different from the mini-batch learning device 2 or the same device as the mini-batch learning device 2. Further, even after the trained model 10T is incorporated in the operation device 15, the trained model 10T may be fed with the mini-batch data 11 for training.

As shown in FIG. 3A, the learning input image 20 is, in this example, a single image showing the state of cell culture with a phase-contrast microscope. Differentiated cells, undifferentiated cells, a medium, and dead cells are shown as objects in the learning input image 20. In the annotation image 21 in this case, as shown in FIG. 3B, the differentiated cells of class 1, the undifferentiated cells of class 2, the medium of class 3, and the dead cells of class 4 are each manually designated. The input image 16 given to the trained model 10T is also an image showing the state of cell culture with a phase-contrast microscope, like the training input image 20.

As shown in FIG. 4, the divided learning input image 20S is a region surrounded by a rectangular frame 25 that is sequentially moved by DX in the horizontal direction and by DY in the vertical direction in the learning input image 20. , It is cut out each time. The lateral movement amount DX of the frame 25 is, for example, ½ of the lateral size of the frame 25. Similarly, the vertical movement amount DY of the frame 25 is, for example, ½ of the vertical size of the frame 25. The frame 25 is, for example, 1/50 of the size of the learning input image 20. In this case, there are a total of 10,000 input images 20S for division learning, 20S_1 to 20S_10000.

Similarly, as shown in FIG. 5, the divided annotation image 21S covers a region in the annotation image 21 surrounded by a rectangular frame 25 that is sequentially moved by DX in the horizontal direction and by DY in the vertical direction. , It is cut out each time. There are a total of 10,000 divided annotation images 21S, 21S_1 to 21S_10000. In the following, it is assumed that the learning input image 20 and the annotation image 21 are already prepared in the mini-batch learning device 2, and the division learning input image 20S and the division annotation image 21S are already generated.

As shown in FIG. 6, the divided learning input image group 12 is a part of the plurality of divided learning input images 20S generated as shown in FIG. 4 (for example, 10,000 divided learning input images). It is composed of 100 sheets out of 20S). Similarly, as shown in FIG. 7, the divided annotation image group 13 is a part of the plurality of divided annotation images 21S generated as shown in FIG. 5 (for example, among 10,000 divided annotation images 21S). 100 sheets). The divided learning input image 20S constituting the divided learning input image group 12 and the divided annotation image 21S constituting the divided annotation image group 13 have the same area cut out by the frame 25.

In FIG. 8, the computer constituting the mini-batch learning device 2 includes a storage device 30, a memory 31, a CPU (Central Processing Unit) 32, a communication unit 33, a display 34, and an input device 35. These are interconnected via the data bus 36.

The storage device 30 is a hard disk drive built in the computer constituting the mini-batch learning device 2 or connected via a cable or a network. Alternatively, the storage device 30 is a disk array in which a plurality of hard disk drives are connected. The storage device 30 stores control programs such as an operating system, various application programs, and various data associated with these programs. A solid state drive may be used instead of the hard disk drive.

The memory 31 is a work memory for the CPU 32 to execute a process. The CPU 32 comprehensively controls each part of the computer by loading the program stored in the storage device 30 into the memory 31 and executing the processing according to the program.

The communication unit 33 is a network interface that controls transmission of various information via a network such as a WAN (Wide Area Network) such as the Internet or a public communication network. The display 34 displays various screens. Various screens are equipped with operation functions by GUI (Graphical User Interface). The computer constituting the mini-batch learning device 2 receives input of an operation instruction from the input device 35 through various screens. The input device 35 is a keyboard, a mouse, a touch panel, or the like.

In FIG. 9, the storage device 30 stores a learning input image 20, an annotation image 21, a division learning input image 20S, a division annotation image 21S, and a model 10. Further, the storage device 30 stores an operation program 40 as an application program. The operation program 40 is an application program for operating the computer as the mini-batch learning device 2. That is, the operation program 40 is an example of the "operation program of the mini-batch learning device" according to the technique of the present disclosure.

When the operation program 40 is started, the CPU 32 of the computer constituting the mini-batch learning device 2 cooperates with the memory 31 and the like, and the calculation unit 50, the specific unit 51, the generation unit 52, the learning unit 53, the evaluation unit 54, And functions as an update unit 55.

The calculation unit 50 calculates the first area ratio of each of the plurality of classes to the area of the entire annotation image 21. More specifically, the calculation unit 50 reads the annotation image 21 from the storage device 30. Then, the number of pixels in the area manually specified in the annotation image 21 is added for each class. Next, the first area ratio is calculated by dividing the added number of pixels by the total number of pixels of the annotation image 21. For example, when the total number of pixels of the region designated as the differentiated cell of class 1 is 10,000 and the total number of pixels is 50,000, the first area ratio of the differentiated cell of class 1 is (10000/50000) ×. 100 = 20%. The calculation unit 50 outputs the calculated first area ratio to the specific unit 51.

The specific unit 51 identifies a rare class in which the first area ratio is lower than the first set value. The specific unit 51 outputs the specified rare class to the generation unit 52.

As shown in FIGS. 6 and 7, the generation unit 52 has the divided learning input image 20S and the divided annotation image 21S generated from the learning input image 20 and the annotation image 21 as shown in FIGS. 4 and 5. By selecting a part of it, mini-batch data 11 is generated. The generation unit 52 generates a plurality of sets (for example, 100 sets) of mini-batch data 11. When the rare class is specified in the specific unit 51, the generation unit 52 devises a selection method of the input image 20S for division learning and the division annotation image 21S, so that the second area ratio is larger than the first area ratio. The mini-batch data 11 which is equal to or larger than the second set value is generated. On the other hand, when the rare class is not specified in the specific unit 51, the generation unit 52 generates the mini-batch data 11 without the above-mentioned restrictions. The generation unit 52 outputs the generated mini-batch data 11 to the learning unit 53 and the evaluation unit 54.

Here, the second area ratio is the area ratio of the rare class in one set of mini-batch data 11. Further, the device for selecting the divided learning input image 20S and the divided annotation image 21S when the rare class is specified in the specific unit 51 is, for example, an input image for divided learning in which an object of the rare class is relatively large. For example, 20S and the divided annotation image 21S are preferentially selected. A method of increasing the choices of the divided learning input image 20S and the divided annotation image 21S for setting the second area ratio of the rare class of the mini-batch data 11 to the second set value or more may be executed. Specifically, the input image 20S for division learning and the division annotation image 21S in which a rare class object appears relatively large are subjected to image processing such as trimming, left-right inversion, and rotation to be tailored to another image, and the mini-batch data 11 As a new option. Such a technique is called data augmentation.

The learning unit 53 gives the model 10 the input image group 12 for division learning of the mini-batch data 11 from the generation unit 52 and trains the model 10. As a result, the learning output image group 14 output from the model 10 is output to the evaluation unit 54 by the learning unit 53.

The evaluation unit 54 compares the divided annotation image group 13 of the mini-batch data 11 from the generation unit 52 with the learning output image group 14 from the learning unit 53, and evaluates the discrimination accuracy of the class of the model 10. The evaluation unit 54 outputs the evaluation result to the update unit 55.

The evaluation unit 54 evaluates the discrimination accuracy of the class of the model 10 by using the loss function. The loss function is a function indicating the degree of difference between the divided annotation image group 13 and the learning output image group 14. The closer the calculated value of the loss function is to 0, the higher the discrimination accuracy of the class of the model 10.

The update unit 55 updates the model 10 according to the evaluation result from the evaluation unit 54. More specifically, the update unit 55 changes the values of various parameters of the model 10 by a stochastic gradient descent method or the like accompanied by a learning coefficient. The learning coefficient indicates the range of change in the values of various parameters of the model 10. That is, the larger the learning coefficient is, the larger the range of change in the values of various parameters is, and the greater the degree of update of the model 10.

10 and 11 show specific examples of processing of each of the calculation unit 50, the specific unit 51, and the generation unit 52. First, in FIG. 10, the calculation unit 50 calculates the first area ratio of each class as shown in Table 60. In FIG. 10, the first area ratio of the differentiated cells of class 1 is 38%, the first area ratio of undifferentiated cells of class 2 is 2%, the first area ratio of the medium of class 3 is 40%, and the death of class 4. The case where the first area ratio of the cell is calculated as 20% is illustrated.

The specific unit 51 identifies a rare class in which the first area ratio is lower than the first set value. FIG. 10 illustrates a case where undifferentiated cells of class 2 having a first area ratio of 2%, which is lower than the first set value, are specified as a rare class because the first set value is 5% or less. .. In addition, although the case where only one rare class is specified is illustrated here, when there are a plurality of classes whose first area ratio is lower than the first set value, naturally, a plurality of classes are regarded as rare classes. Be identified.

Subsequently, in FIG. 11, as shown in Table 61, the generation unit 52 has mini-batch data in which the second area ratio of the rare class is equal to or greater than the second set value that is larger than the first area ratio calculated by the calculation unit 50. 11 is generated. In FIG. 11, since the second set value is 25% or more, the second area ratio of the undifferentiated cells of class 2 which is a rare class is set to 25% in each mini-batch data 11. In addition, the second area ratio of other classes other than the rare class 2 undifferentiated cells is uniformly 25%. The first set value shown in FIG. 10 and the second set value shown in FIG. 11 are merely examples. The second set value may be at least larger than the first area ratio of the rare class, and may be larger than 2% in the above example. Further, since there is no particular restriction on the second area ratio of the classes other than the rare class, it is not necessary to set the value uniformly to 25% as described above.

Next, the operation of the above configuration will be described with reference to the flowchart shown in FIG. First, the operation program 40 is activated, and as shown in FIG. 9, the CPU 32 of the computer constituting the mini-batch learning device 2 functions as each processing unit 50 to 55.

As shown in Table 60 of FIG. 10, the calculation unit 50 calculates the first area ratio of each class (step ST100, calculation step). Subsequently, as also shown in FIG. 10, in the specific unit 51, a rare class having a first area ratio lower than the first set value is specified (step ST110, specific step).

When the rare class is specified in the specific unit 51 (YES in step ST120), as shown in Table 61 of FIG. 11, the mini in which the second area ratio of the rare class is equal to or larger than the second set value by the generation unit 52. Batch data 11 is generated (step ST130, generation step).

The case where the rare class is specified in the specific unit 51 is the case where the learning input image 20 and the annotation image 21 have a bias in the class. If the learning input image 20 and the annotation image 21 have a class bias, if the mini-batch data 11 is generated without any restrictions, there is a high possibility that the mini-batch data 11 will also have a class bias. Then, as a result, the model 10 having low discrimination accuracy of the rare class is completed.

However, in the present embodiment, as described above, when the rare class is specified in the specific unit 51, the generation unit 52 generates the mini-batch data 11 in which the second area ratio of the rare class is equal to or higher than the second set value. is doing. By doing so, even if the learning input image 20 and the annotation image 21 have a class bias, the mini-batch data 11 does not have a class bias. Therefore, it is possible to avoid a situation in which the model 10 having a low discrimination accuracy of the rare class is completed, and it is possible to suppress a decrease in the discrimination accuracy of the class of the model 10.

On the other hand, when the rare class is not specified in the specific unit 51, the mini-batch data 11 is generated in the generation unit 52 without any particular limitation (step ST140, generation step).

In the learning unit 53, the input image group 12 for division learning of the mini-batch data 11 from the generation unit 52 is given to the model 10 and learning is performed (step ST150). Then, the learning output image group 14 output from the model 10 and the divided annotation image group 13 of the mini-batch data 11 from the generation unit 52 are compared in the evaluation unit 54, and the discrimination accuracy of the class of the model 10 is compared. Is evaluated (step ST160).

When it is determined that the discrimination accuracy of the class of the model 10 has reached a desired level based on the evaluation result by the evaluation unit 54 (YES in ST170), the mini-batch learning is terminated. On the other hand, when it is determined that the discrimination accuracy of the class of the model 10 has not reached a desired level (NO in step ST170), the model 10 is updated by the update unit 55 (step ST180). Then, the process is returned to step ST150, another set of mini-batch data 11 is given to the model 10, and the subsequent steps are repeated.

[Second Embodiment]
In the second embodiment shown in FIG. 13, the generation unit 52 is asked whether or not to perform a process of generating the mini-batch data 11 in which the second area ratio is equal to or larger than the second set value.

In FIG. 13, the CPU of the mini-batch learning device of the second embodiment functions as a reception unit 65 in addition to the processing units 50 to 55 of the first embodiment. When the rare class is specified in the specific unit 52, the reception unit 65 selects whether or not to have the generation unit 52 perform a process of generating the mini-batch data 11 in which the second area ratio is equal to or larger than the second set value. Accept instructions.

In the second embodiment, when the rare class is specified in the specific unit 52, the inquiry screen 66 is displayed on the display 34. On the inquiry screen 66, a message 67 indicating that the rare class has been specified and whether or not to generate mini-batch data 11 having a second area ratio equal to or greater than the second set value 67, a yes button 68, No button 69 is displayed. The reception unit 65 receives the selection instruction of the yes button 68 and the no button 69 as a selection instruction as to whether or not to perform the process of generating the mini-batch data 11 in which the second area ratio is equal to or larger than the second set value. When the Yes button 68 is selected, the generation unit 52 performs a process of generating the mini-batch data 11 in which the second area ratio is equal to or larger than the second set value. On the other hand, when the No button 69 is selected, the generation unit 52 does not perform the process of generating the mini-batch data 11 in which the second area ratio is equal to or larger than the second set value.

When generating the annotation image, the class is specified manually, so the class may be specified incorrectly. In addition, although the model 10 was designated as a class at the beginning of development, some classes may become less important as the development progresses. In such a case, although the rare class is specified in the specific unit 52, it may not be necessary to generate the mini-batch data 11 in which the second area ratio is equal to or larger than the second set value.

Therefore, in the second embodiment, the reception unit 65 receives an instruction to select whether or not to cause the generation unit 52 to perform a process of generating the mini-batch data 11 in which the second area ratio is equal to or larger than the second set value. There is. Therefore, although the rare class is specified in the specific unit 52, it is possible to deal with the case where it is not necessary to generate the mini-batch data 11 in which the second area ratio is equal to or larger than the second set value.

[Third Embodiment]
In the third embodiment shown in FIG. 14, a plurality of mini-batch data 11 are generated according to a certain rule. Then, among the plurality of mini-batch data 11 generated according to a certain rule, the mini-batch data 11 whose second area ratio is equal to or higher than the second set value is selected for learning.

In FIG. 14, as shown in FIGS. 4 and 5, the generation unit 75 of the third embodiment moves the frame 25 according to a certain rule (moves sequentially by DX in the horizontal direction and by DY in the vertical direction). Then, the input image 20S for division learning and the division annotation image 21S are generated. Further, the generation unit 75 generates the division learning input image group 12 and the division annotation image group 13 from the division learning input image 20S and the division annotation image 21S according to a certain rule. In the first embodiment, by devising the selection method of the input image 20S for division learning and the division annotation image 21S, the mini-batch data 11 in which the second area ratio is equal to or more than the second set value is generated. In the third embodiment, the mini-batch data 11 is generated according to a certain rule for the time being without devising such a selection method.

The generation unit 75 selects the mini-batch data 11 whose second area ratio is equal to or higher than the second set value among the plurality of mini-batch data 11 generated according to a certain rule in this way for learning.

Table 76 shows the second area ratio of each class of the plurality of mini-batch data 11 generated by the generation unit 75 according to a certain rule. Here, as in FIG. 10 and the like, a case where undifferentiated cells of class 2 are identified as a rare class is illustrated. Further, the case where the second set value is 25% or more as in the first embodiment is exemplified. In this case, it is No. 1 that the second area ratio of the undifferentiated cells of class 2 which is a rare class is equal to or more than the second set value. It is the mini-batch data 11 of 2. Therefore, as shown in Table 77, the generation unit 75 is No. The mini-batch data 11 of 2 is selected as the mini-batch data 11 to be given to the learning unit 53.

As described above, in the third embodiment, the generation unit 75 generates a plurality of mini-batch data 11 according to a certain rule, and the second area of the plurality of mini-batch data 11 generated according to a certain rule. The mini-batch data 11 having a ratio equal to or higher than the second set value is selected for use in training. Therefore, it is possible to save the trouble of generating the mini-batch data 11 in which the second area ratio is equal to or larger than the second set value by devising the selection method of the input image 20S for division learning and the division annotation image 21S.

[Fourth Embodiment]
In the fourth embodiment shown in FIG. 15, a rare class uneven distribution region and a non-uneven distribution region in the annotation image 21 are detected. Then, the number of cuts of the image that is the source of the mini-batch data 11 in the unevenly distributed area is made larger than the number of cuts in the non-unevenly distributed area. Here, the image that is the source of the mini-batch data 11 is the divided annotation image 21S.

In FIG. 15, the generation unit of the fourth embodiment detects the rare class uneven distribution region 80 and the non-uneven distribution region 81 in the annotation image 21. As a method for detecting the uneven distribution region 80, first, the annotation image 21 is divided into a plurality of regions, and the area ratio of the rare class in each region is calculated. Subsequently, the average AVE and the standard deviation σ of the calculated area ratio of each region are obtained. Then, a region where the area ratio of the rare class exceeds, for example, AVE + 3σ is detected as the uneven distribution region 80.

The generation unit increases the number of cuts of the divided annotation image 21S of the uneven distribution region 80 detected as described above to be larger than the number of cuts of the non-uneven distribution region 81. In FIG. 15, the movement amount of the frame 25 shown in FIGS. 4 and 5 is reduced by making the movement amounts DX_A and DY_A of the uneven distribution area 80 smaller than the movement amounts DX_B and DY_B of the non-uneven distribution region 81. The number of cuts of the divided annotation image 21S in the above is larger than the number of cuts of the non-unevenly distributed region 81.

As described above, in the fourth embodiment, the generation unit detects the rare class uneven distribution region 80 and the non-uneven distribution region 81 in the annotation image 21, and cuts out the image that is the source of the mini-batch data 11 of the uneven distribution region 80. The number is larger than the number of cuts of the non-unevenly distributed region 81. Therefore, the mini-batch data 11 in which the second area ratio is equal to or larger than the second set value can be easily generated.

In each of the above embodiments, the input image 16 and the learning input image 20 exemplify an image of the state of cell culture projected by a phase-contrast microscope, and the class includes differentiated cells, a medium, and the like, but the present invention is not limited thereto. For example, an MRI (Magnetic Resonance Imaging) image may be used as an input image 16 and a learning input image 20, and an organ such as a liver or a kidney may be used as a class.

The model 10 is not limited to U-Net, and may be another convolutional neural network, for example, SegNet.

The hardware configuration of the computer constituting the mini-batch learning device 2 can be modified in various ways. For example, the mini-batch learning device 2 can be configured by a plurality of computers separated as hardware for the purpose of improving processing power and reliability. Specifically, the functions of the calculation unit 50 and the specific unit 51, the functions of the generation unit 52 and the learning unit 53, and the functions of the evaluation unit 54 and the update unit 55 are distributed to three computers. In this case, the mini-batch learning device 2 is configured by three computers.

As described above, the hardware configuration of the computer can be appropriately changed according to the required performance such as processing power, safety, and reliability. Furthermore, not only hardware but also application programs such as the operation program 40 can be duplicated or distributed and stored in multiple storage devices for the purpose of ensuring safety and reliability. be.

In each of the above embodiments, for example, a processing unit that executes various processes such as a calculation unit 50, a specific unit 51, a generation unit 52, 75, a learning unit 53, an evaluation unit 54, an update unit 55, and a reception unit 65. As the hardware structure of, various processors (Processors) shown below can be used. For various processors, as described above, in addition to the CPU 32, which is a general-purpose processor that executes software (operation program 40) and functions as various processing units, after manufacturing an FPGA (Field Programmable Gate Array) or the like. Dedicated processor with a circuit configuration designed specifically for executing specific processing such as Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed, and ASIC (Application Specific Integrated Circuit). Includes electrical circuits and the like.

One processing unit may be composed of one of these various processors, or may be a combination of two or more processors of the same type or different types (for example, a combination of a plurality of FPGAs and / or a CPU). It may be configured in combination with FPGA). Further, a plurality of processing units may be configured by one processor.

As an example of configuring a plurality of processing units with one processor, first, one processor is configured by a combination of one or more CPUs and software, as represented by a computer such as a client and a server. There is a form in which the processor functions as a plurality of processing units. Secondly, as typified by System On Chip (SoC), there is a form of using a processor that realizes the functions of the entire system including a plurality of processing units with one IC (Integrated Circuit) chip. be. As described above, the various processing units are configured by using one or more of the above-mentioned various processors as a hardware-like structure.

Further, as the hardware structure of these various processors, more specifically, an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined can be used.

From the above description, the invention described in the following Appendix 1 can be grasped.

[Appendix 1]
It is a mini-batch learning device that gives mini-batch data to a machine learning model for performing semantic segmentation that discriminates multiple classes in an image on a pixel-by-pixel basis.
A calculation processor that calculates the first area ratio of each of the plurality of classes to the area of the entire annotation image among the training input image and the annotation image that are the sources of the mini-batch data.
A specific processor that identifies a rare class whose first area ratio is lower than the first set value, and
It is a generation processor that generates the mini-batch data from the input image for learning and the annotation image, and the second area ratio of the rare class in the mini-batch data is larger than the first area ratio calculated by the calculation processor. A generation processor that generates the mini-batch data that is greater than or equal to the large second set value, and
A mini-batch learning device equipped with.

The technique of the present disclosure can also appropriately combine the various embodiments described above with various modifications. Further, it is of course not limited to each of the above embodiments, and various configurations can be adopted as long as they do not deviate from the gist. Further, the technique of the present disclosure extends to a storage medium for storing the program non-temporarily in addition to the program.

The contents described above and the contents shown in the illustration are detailed explanations of the parts related to the technique of the present disclosure, and are merely an example of the technique of the present disclosure. For example, the description of the configuration, function, action, and effect described above is an example of the configuration, function, action, and effect of a portion of the art of the present disclosure. Therefore, unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-mentioned description contents and illustration contents within the range not deviating from the gist of the technique of the present disclosure. Needless to say. In addition, in order to avoid complications and facilitate understanding of the parts relating to the technique of the present disclosure, the contents described above and the contents shown above require special explanation in order to enable the implementation of the technique of the present disclosure. The explanation about the common technical knowledge that is not used is omitted.

As used herein, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, B alone, or a combination of A and B. Further, in the present specification, when three or more matters are connected and expressed by "and / or", the same concept as "A and / or B" is applied.

All documents, patent applications and technical standards described herein are to the same extent as if it were specifically and individually stated that the individual documents, patent applications and technical standards are incorporated by reference. Incorporated by reference in the book.

Claims (6)

It is a mini-batch learning device that gives mini-batch data to a machine learning model for performing semantic segmentation that discriminates multiple classes in an image on a pixel-by-pixel basis.
A calculation unit that calculates the first area ratio of each of the plurality of classes to the area of the entire annotation image among the learning input image and the annotation image that are the sources of the mini-batch data.
A specific part that identifies a rare class in which the first area ratio is lower than the first set value, and
It is a generation unit that generates the mini-batch data from the input image for learning and the annotation image, and the second area ratio of the rare class in the mini-batch data is larger than the first area ratio calculated by the calculation unit. A generator that generates the mini-batch data that is greater than or equal to the large second set value, and
A mini-batch learning device equipped with. The mini-batch learning according to claim 1, wherein the generation unit includes a reception unit that receives a selection instruction as to whether or not to perform a process of generating the mini-batch data in which the second area ratio is equal to or larger than the second set value. Device. The generation unit generates a plurality of the mini-batch data according to a certain rule, and the second area ratio is equal to or more than the second set value among the plurality of the mini-batch data generated according to the certain rule. The mini-batch learning device according to claim 1 or 2, wherein the mini-batch data is selected for use in the learning. The generation unit detects an uneven distribution region and a non-uneven distribution region of the rare class in the annotation image, and cuts out the image that is the source of the mini-batch data in the uneven distribution region. The mini-batch learning device according to any one of claims 1 to 3, wherein the number is larger than the number. It is an operation program of a mini-batch learning device that gives learning by giving mini-batch data to a machine learning model for performing semantic segmentation that discriminates multiple classes in an image on a pixel-by-pixel basis.
A calculation unit that calculates the first area ratio of each of the plurality of classes to the area of the entire annotation image among the learning input image and the annotation image that are the sources of the mini-batch data.
A specific part that identifies a rare class in which the first area ratio is lower than the first set value, and
It is a generation unit that generates the mini-batch data from the input image for learning and the annotation image, and the second area ratio of the rare class in the mini-batch data is larger than the first area ratio calculated by the calculation unit. As a generator that generates the mini-batch data that is greater than or equal to the large second set value,
An operation program for a mini-batch learning device that makes a computer work. It is an operation method of a mini-batch learning device that gives learning by giving mini-batch data to a machine learning model for performing semantic segmentation that discriminates multiple classes in an image on a pixel-by-pixel basis.
A calculation step for calculating the first area ratio of each of the plurality of classes to the area of the entire annotation image among the training input image and the annotation image that are the sources of the mini-batch data.
A specific step for identifying a rare class in which the first area ratio is lower than the first set value, and
In the generation step of generating the mini-batch data from the learning input image and the annotation image, the second area ratio of the rare class in the mini-batch data is larger than the first area ratio calculated in the calculation step. A generation step for generating the mini-batch data that is greater than or equal to the large second set value, and
How to operate a mini-batch learning device equipped with.

Images