JP7167355B2 - LEARNING APPARATUS, LEARNING APPARATUS OPERATING METHOD, LEARNING APPARATUS OPERATING PROGRAM - Google Patents

Description

The technology of the present disclosure relates to a learning device, a learning device operating method, and a learning device operating program.

Semantic segmentation is known that distinguishes between multiple classes in an input image on a pixel-by-pixel basis. A class is the type of object appearing in the input image. A machine learning model (hereinafter simply referred to as a model) that implements semantic segmentation includes a U-shaped convolutional neural network (U-Net).

In Japanese Patent Publication No. 2010-518486, a cell image obtained by photographing a plurality of cells in culture is given to a model as an input image, and an output image in which the cells reflected in the cell image are classified as a class can be output from the model. Have been described. In JP 2010-518486 A, the output image is analyzed to count the number of cells.

In Japanese Patent Application Laid-Open No. 2010-518486, the model only performs class discrimination, and does not count the number of cells based on the output image. In other words, class discrimination by the model and counting of the number of cells are performed separately. For this reason, for example, although 20 cells could be visually confirmed in the output image, the number of cells was counted as 16 in the counting results, resulting in discrepancies between the class discrimination results and the number counting results. something happened.

Therefore, it is conceivable to use a single model for class discrimination and counting. However, depending on the model learning method, there is a possibility that the class discrimination accuracy and the number counting accuracy will be relatively low.

The technology of the present disclosure provides a learning device, an operation method of the learning device, and a learning device that can obtain a machine learning model with relatively high accuracy in class discrimination in an input image and in counting the number of objects appearing in the input image. The purpose is to provide an operating program for

In order to achieve the above object, the learning device of the present disclosure includes a segmentation part that performs semantic segmentation in which a class, which is the type of object appearing in an input image, is determined on a pixel-by-pixel basis; A first acquisition unit that acquires a basic machine learning model including a counting part that counts the number of the first First learning for learning the segmentation part using the learning data, second learning data that is a set of the input image for learning, and the result of counting the number of objects appearing in the input image for learning, and the counting part using the second learning data A learning unit that performs second learning to learn and sets the basic machine learning model as a learned machine learning model, and a control unit that controls the operation of the learning unit. is larger than the learning amount of the second learning, and in the latter learning stage, the learning amount of the second learning is made larger than the learning amount of the first learning.

Preferably, the learning unit evaluates the class discrimination accuracy of the segmentation part using the first loss function, and evaluates the counting accuracy of the number of objects of the counting part using the second loss function.

Preferably, the control unit changes the learning amount of the first learning and the learning amount of the second learning according to a preset change schedule.

The change schedule sets the learning amount of the second learning to 0, causes the learning unit to perform only the first learning first, sets the segmentation part to the learned segmentation part, then sets the learning amount of the first learning to 0, and performs the learning It is preferable that the first learning and the second learning are performed in two stages, that is, the part is made to perform only the second learning and the counting part is set as the learned counting part.

The content of the change schedule is preferably such that the learning amount of the first learning is gradually decreased and the learning amount of the second learning is gradually increased.

In the change schedule, the learning amount of the first learning is temporarily increased in the process of gradually decreasing the learning amount of the first learning, and the learning amount of the first learning is temporarily increased in the process of gradually increasing the learning amount of the second learning. It is preferable that the content is such that the amount is decreased according to the timing of increasing the amount.

The learning unit evaluates the overall accuracy including the discrimination accuracy and the counting accuracy by the weighted sum of the first loss function and the second loss function, and the control unit evaluates the first weighting factor of the first loss function, and It is preferable to change the learning amount of the first learning and the learning amount of the second learning by changing the second weighting factor of the second loss function.

It is preferable to have a reception unit that receives designation of the change schedule by the user.

The second loss function is the absolute value of the difference between the learning counting result, which is the counting result of the counting part of the number of objects appearing in the learning input image, and the counting result of the number of objects included in the second learning data, It is preferable to evaluate counting accuracy based on a value obtained by dividing the number of objects included in the second learning data by the result.

The input image is preferably a cell image obtained by photographing a plurality of cells in culture.

The operation method of the learning device of the present disclosure includes a segmentation part that performs semantic segmentation for classifying the class, which is the type of object appearing in the input image, on a pixel-by-pixel basis, and counting the number of objects whose class is determined by the semantic segmentation. using the first learning data, which is a set of an acquisition step of acquiring a basic machine learning model including a counting part to be performed, and an input image for learning and an annotation image in which a class is specified for the input image for learning , first learning for learning the segmentation part, and second learning for learning the counting part using second learning data that is a set of the input image for learning and the result of counting the number of objects appearing in the input image for learning. and in the learning step in which the basic machine learning model is a trained machine learning model, and in the first half of the learning stage, the learning amount of the first learning is made larger than the learning amount of the second learning, and in the latter half of the learning stage comprises a control step of making the learning amount of the second learning larger than the learning amount of the first learning.

The operation program of the learning device of the present disclosure includes a segmentation part that performs semantic segmentation for classifying classes, which are the types of objects appearing in an input image, on a pixel-by-pixel basis, and counting the number of objects whose classes are determined by semantic segmentation. Using first learning data that is a set of an acquisition unit that acquires a basic machine learning model including a counting part to be performed, an input image for learning, and an annotation image in which a class is specified for the input image for learning , first learning for learning the segmentation part, and second learning for learning the counting part using second learning data that is a set of the input image for learning and the result of counting the number of objects appearing in the input image for learning. and a learning unit that uses the basic machine learning model as a learned machine learning model, and a control unit that controls the operation of the learning unit. The computer is made to function as a control unit that makes the learning amount larger than the learning amount, and makes the learning amount of the second learning larger than the learning amount of the first learning in the latter learning stage.

According to the technology of the present disclosure, a learning device capable of obtaining a machine learning model with relatively high class discrimination accuracy in an input image and a relatively high counting accuracy of the number of objects appearing in the input image, an operation method of the learning device, An operating program for the learning device can be provided.

1 illustrates a machine learning system; FIG. It is a figure which shows the outline|summary of the process in a machine-learning system. FIG. 10 is a diagram showing an input image for learning and an annotation image; It is a figure which shows an annotation image and a counting result. 1 is a block diagram showing a computer that constitutes a learning device; FIG. 3 is a block diagram showing a processing section of a CPU of the learning device; FIG. It is a figure which shows a model group. It is a figure which shows a basic model. It is a figure which shows a change schedule. FIG. 4 is a diagram showing an outline of processing of a learning unit in a first stage; FIG. 10 is a diagram showing details of processing of the learning unit in the first stage; FIG. 10 is a diagram showing an outline of processing of a learning unit in a second stage; FIG. 10 is a diagram showing details of processing of the learning unit in the second stage; FIG. 14A shows the state of the first learning in the first stage, and FIG. 14B shows the state of the second learning in the second stage. FIG. 10 is a diagram showing a second loss function; 3 is a block diagram showing a processing unit of a CPU of the operation device; FIG. It is a figure which shows an evaluation result display screen. 4 is a flow chart showing a processing procedure of the learning device; FIG. 10 is a diagram showing weighted sums; It is a figure which shows the change schedule of 2nd Embodiment. 21A and 21B show the first weighting factor and the second weighting factor, respectively. It is a figure which shows the detail of the process of the learning part of 2nd Embodiment. 4 is a table showing an original model and an output model in each learning stage; FIG. 24A shows the state of the first learning in the first stage, and FIG. 24B shows the states of the first learning and the second learning in the second stage. FIG. 25A shows the first learning and second learning in the third stage, and FIG. 25B shows the first learning and second learning in the fourth stage. FIG. 26A shows the state of the first learning and the second learning in the fifth stage, and FIG. 26B shows the state of the second learning in the sixth stage. 9 is a flow chart showing a processing procedure of the learning device of the second embodiment; It is a figure which shows another example of the change schedule of 2nd Embodiment. It is a figure which shows another example of the change schedule of 2nd Embodiment. It is a figure which shows the change schedule of 3rd Embodiment. 31A and 31B are graphs showing the change schedule shown in FIG. 30, wherein FIG. 31A shows the first weighting factor and FIG. 31B shows the second weighting factor; It is a figure which shows another example of the change schedule of 3rd Embodiment. 33A and 33B are graphs showing the change schedule shown in FIG. 32, wherein FIG. 33A shows the first weighting factor and FIG. 33B shows the second weighting factor; It is a figure which shows a change schedule specification screen. 35A and 35B show a first weighting factor and a second weighting factor, respectively. FIG. 36B is a graphical representation of another example change schedule, FIG. 36A showing the first weighting factor and FIG. 36B the second weighting factor;

[First embodiment]
In FIG. 1, the machine learning system 2 is a model that performs semantic segmentation in which a class, which is the type of object appearing in an input image, is determined on a pixel-by-pixel basis, and a model that counts the number of objects whose class is determined by the semantic segmentation. system used. A machine learning system 2 includes a learning device 10 and an operation device 11 . The learning device 10 and the operation device 11 are, for example, desktop personal computers. Learning device 10 and operation device 11 are connected via network 12 so as to be able to communicate with each other. The network 12 is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet or a public communication network.

In FIG. 2, learning device 10 performs first learning and second learning on basic model 15, and sets basic model 15 as trained model 15T. The underlying model 15 includes a segmentation part 16 that performs semantic segmentation and a counting part 17 that performs counting of the number of objects. In the first learning, the segmentation portion 16 is learned using the first learning data 18, and the segmentation portion 16 is set as the learned segmentation portion 16T. The first learning data 18 is a set of a learning input image 19 and an annotation image 20 . In the second learning, the counting portion 17 is learned using the second learning data 21, and the counting portion 17 is set as the learned counting portion 17T. The second learning data 21 is a set of the learning input image 19 and the counting result 22 .

Trained model 15T includes trained segmentation portion 16T and trained counting portion 17T. The learning device 10 transmits the trained model 15T to the operation device 11 . The operation device 11 receives the trained model 15T from the learning device 10 . The operation device 11 provides the learned model 15T with the input image 25 in which the class and the number of the reflected objects have not yet been determined. The trained model 15T performs semantic segmentation on the input image 25 and counts the number of objects, determines the class and number of objects appearing in the input image 25, and outputs the output image 26 and the count result 27 as the determination result. do. Even after the trained model 15T is installed in the operation device 11, the learned data 18 and 21 may be given to the trained model 15T for learning.

As shown in FIG. 3, the learning input image 19 is, in this example, a cell image obtained by photographing a plurality of cells in culture with a phase-contrast microscope or the like. Differentiated cell DC, undifferentiated cell UDC, dead cell DDC, and medium PL are shown in the learning input image 19 . In the annotation image 20 in this case, differentiated cell DC, undifferentiated cell UDC, dead cell DDC, and medium PL are designated as class 1, class 2, class 3, and class 4, respectively. The designation of each class 1 to 4 is manually performed by the user, for example. Medium PL of class 4 is automatically designated by designating other classes 1-3. Note that the input image 25 given to the trained model 15T is also a cell image obtained by photographing a plurality of cells in culture with a phase-contrast microscope or the like, like the learning input image 19 .

In addition, as shown in FIG. 4, the counting result 22 of the second learning data 21 counts the number of differentiated cell DC, undifferentiated cell UDC, and dead cell DDC of classes 1 to 3 based on the annotation image 20. It is what I did. For example, the user observes the annotation image 20 and counts the numbers of differentiated cell DCs, undifferentiated cell UDCs, and dead cell DDCs. Differentiated cell DC, undifferentiated cell UDC, and dead cell DDC are examples of the “object” according to the technology of the present disclosure.

In FIG. 5, the computer constituting the learning device 10 and the operation device 11 have the same basic configuration, and include a storage device 30, a memory 31, a CPU (Central Processing Unit) 32, a communication unit 33, a display 34, and an input device. A device 35 is provided. These are interconnected via bus lines 36 .

The storage device 30 is a hard disk drive that is built into the computer that constitutes the learning device 10 or the like, or that is connected via a cable or network. Alternatively, the storage device 30 is a disk array in which a plurality of hard disk drives are connected. The storage device 30 stores a control program such as an operating system, various application programs, various data associated with these programs, and the like. A solid state drive may be used instead of the hard disk drive.

The memory 31 is a work memory for the CPU 32 to execute processing. The CPU 32 loads a program stored in the storage device 30 into the memory 31 and executes processing according to the program, thereby comprehensively controlling each section of the computer.

The communication unit 33 is a network interface that controls transmission of various information via the network 12 . The display 34 displays various screens. The computer that constitutes the learning device 10 and the like accepts input of operation instructions from the input device 35 through various screens. The input device 35 is a keyboard, mouse, touch panel, or the like. In the following description, each part of the learning device 10 is given a suffix "A", and each part of the operation device 11 is given a suffix "B" to distinguish them.

In FIG. 6, an operating program 40 is stored in the storage device 30A of the learning device 10. As shown in FIG. The operating program 40 is an application program that causes the computer to function as the learning device 10 . That is, the operating program 40 is an example of the "learning device operating program" according to the technology of the present disclosure. A model group 41, a learning data group 42, and a change schedule 43 are also stored in the storage device 30A.

When the operation program 40 is started, the CPU 32A of the computer constituting the learning device 10 cooperates with the memory 31 and the like to perform read/write (hereinafter abbreviated as RW (Read Write)) control section 50, learning section 51, It functions as a control unit 52 and a transmission control unit 53 .

The RW control unit 50 controls storage of various data in the storage device 30A and reading of various data in the storage device 30A. The RW control unit 50 is an example of an “acquisition unit” according to the technology of the present disclosure.

The learning unit 51 causes the segmentation part 16 to learn using the first learning data 18, and performs the above-described first learning with the segmentation part 16 as the learned segmentation part 16T. The learning unit 51 also makes the counting part 17 learn using the second learning data 21, and performs the above-described second learning with the counting part 17 as the learned counting part 17T.

The control section 52 controls the operation of the learning section 51 . The transmission control unit 53 controls transmission of the trained model 15T to the operation device 11 .

As shown in FIG. 7, the model group 41 includes an intermediate model 15M in addition to the basic model 15 and the trained model 15T described above. The intermediate model 15M is a model in which the segmentation portion 16 of the basic model 15 is the learned segmentation portion 16T after completing the first learning (see FIG. 10). Although not shown, the learning data group 42 includes the first learning data 18 and the second learning data 21 .

As shown in FIG. 8, the basic model 15 has a shared deep neural network (hereinafter abbreviated as DNN (Deep Neural Network)) 60, a segmentation output layer 61, a counting DNN 62, and a counting output layer 63. there is A segmentation output layer 61 and a counting DNN 62 are connected to the shared DNN 60 . Therefore, the output of the shared DNN 60 is input to the segmentation output layer 61 and the counting DNN 62 respectively. A counting output layer 63 is connected to the counting DNN 62 . Therefore, the output of the counting DNN 62 is input to the counting output layer 63 . In the first learning, the segmentation output layer 61 outputs a learning output image 65 (see also FIG. 11). In the second learning, the counting output layer 63 outputs learning counting results 66 (see also FIG. 13).

The shared DNN 60 and the segmentation output layer 61 constitute the segmentation part 16 . The shared DNN 60 , the counting DNN 62 and the counting output layer 63 constitute the counting portion 17 . The shared DNN 60 is, for example, a convolutional neural network such as U-Net, SegNet, ResNet (Residual Network). In contrast, the counting DNN 62 is a fully-connected neural network.

As shown in FIG. 9, the change schedule 43 registers learning contents in a plurality of learning stages. FIG. 9 illustrates a case where the learning stage is divided into two stages, a first stage and a second stage. In the first stage, the learning amount of the second learning is set to 0 and only the first learning is performed. , each of which is registered as an example. Note that the first stage is an example of the “first half learning stage” according to the technology of the present disclosure. Also, the second stage is an example of the “second learning stage” according to the technology of the present disclosure.

The control unit 52 changes the learning amount of the first learning and the learning amount of the second learning according to the change schedule 43 shown in FIG. That is, in the first stage, the control unit 52 sets the learning amount of the second learning to 0, causes the learning unit 51 to perform only the first learning, and sets the learning amount of the first learning to 0 in the second stage. , causes the learning unit 51 to perform only the second learning.

10 to 13 shown below, FIGS. 10 and 11 are related to the first learning of the first stage, and FIGS. 12 and 13 are related to the second learning of the second stage.

As shown in FIG. 10 , in the first learning of the first stage, the learning unit 51 receives from the RW control unit 50 the basic model 15 and the first learning data 18 read by the RW control unit 50 from the storage device 30A. The first learning data 18 is a set of a learning input image 19 and an annotation image 20 . Under the control of the control unit 52, the learning unit 51 uses the first learning data 18 to learn the segmentation part 16 of the basic model 15, and sets the segmentation part 16 to the learned segmentation part 16T. The learning unit 51 outputs the intermediate model 15M including the learned segmentation part 16T to the RW control unit 50. FIG. The RW control unit 50 stores the intermediate model 15M in the storage device 30A.

As shown in FIG. 11 , the learning section 51 has a processing section 70 , an evaluation section 71 and an update section 72 .

In the first stage of the first learning, the processing unit 70 gives the learning input image 19 to the segmentation part 16 of the basic model 15 and causes the segmentation part 16 to output the learning output image 65 . The processing unit 70 outputs the learning output image 65 to the evaluation unit 71 .

The evaluation unit 71 compares the annotation image 20 and the learning output image 65 to evaluate the class discrimination accuracy of the segmentation portion 16 . The evaluation unit 71 evaluates the class discrimination accuracy of the segmentation part 16 using the first loss function 73 . The first loss function 73 is a function representing the degree of difference in class designation between the annotation image 20 and the learning output image 65 . The closer the calculated value of the first loss function 73 to 0, the higher the class discrimination accuracy of the segmentation portion 16 . The evaluation unit 71 outputs an evaluation result of the class discrimination accuracy of the segmentation part 16 by the first loss function 73 to the updating unit 72 .

The update unit 72 updates the segmentation part 16 according to the evaluation result from the evaluation unit 71 . For example, the updating unit 72 changes the parameter values of the convolutional filter of the shared DNN 60 by stochastic gradient descent with learning coefficients or the like. The learning coefficient indicates the variation width of the parameter value. That is, the larger the learning coefficient is, the larger the range of change in parameter values and the like, and the greater the degree of update of the segmentation portion 16 .

The learning unit 51 inputs the learning input image 19 to the basic model 15 by the processing unit 70, outputs the learning output image 65 to the evaluation unit 71, and evaluates the class discrimination accuracy of the segmentation part 16 by the evaluation unit 71. , and updating of the segmentation portion 16 by the updating unit 72 are repeated until the class discrimination accuracy of the segmentation portion 16 reaches a preset level. Then, the learning unit 51 outputs the base model 15 in which the classification accuracy of the class of the segmentation part 16 is at a preset level and the learned segmentation part 16T is output as the intermediate model 15M.

As shown in FIG. 12 , in the second learning of the second stage, the learning unit 51 receives from the RW control unit 50 the intermediate model 15M and the second learning data 21 read by the RW control unit 50 from the storage device 30A. The second learning data 21 is a set of the learning input image 19 and the counting result 22 . Under the control of the control unit 52, the learning unit 51 uses the second learning data 21 to learn the counting part 17 of the intermediate model 15M, and sets the counting part 17 to the learned counting part 17T. The learning unit 51 outputs to the RW control unit 50 a learned model 15T including the learned counting portion 17T and the learned segmentation portion 16T of the first learning. The RW control unit 50 stores the trained model 15T in the storage device 30A.

As shown in FIG. 13, in the second learning of the second stage, the processing unit 70 gives the learning input image 19 to the counting part 17 of the intermediate model 15M, and causes the counting part 17 to output the learning counting result 66. . The processing unit 70 outputs the learning counting result 66 to the evaluation unit 71 .

The evaluation unit 71 compares the counting result 22 of the second learning data 21 and the learning counting result 66 to evaluate the counting accuracy of the number of objects in the counting part 17 . The evaluation unit 71 evaluates the counting accuracy of the number of objects of the counting part 17 using the second loss function 74 . The second loss function 74 is a function representing the degree of difference in the number of the counting results 22 of the second learning data 21 and the learning counting results 66 . As with the first loss function 73 , the closer the calculated value of the second loss function 74 to 0, the higher the accuracy of counting the number of objects in the counting portion 17 . The evaluation unit 71 outputs the evaluation result of the counting accuracy of the number of objects in the counting part 17 by the second loss function 74 to the updating unit 72 .

The updating section 72 updates the counting portion 17 according to the evaluation result from the evaluating section 71 . As in the segmentation part 16, the updating unit 72 updates the parameter values of the filter for convolution operation of the shared DNN 60 and/or the parameter values of the counting DNN 62 by stochastic gradient descent with learning coefficients or the like. etc.

The learning unit 51 receives the input of the learning input image 19 to the intermediate model 15M by the processing unit 70, the output of the learning counting result 66 to the evaluation unit 71, and the counting accuracy of the number of objects in the counting part 17 by the evaluation unit 71. and updating of the counting portion 17 by the updating unit 72 are repeated until the counting accuracy of the number of objects in the counting portion 17 reaches a preset level. Then, the learning unit 51 outputs the intermediate model 15M in which the class discrimination accuracy of the counting portion 17 reaches a preset level and becomes the learned counting portion 17T as the learned model 15T.

Thus, the learning unit 51 uses the first loss function 73 to evaluate the class discrimination accuracy of the segmentation part 16, and uses the second loss function 74 to evaluate the counting accuracy of the number of objects in the counting part 17. . Further, the learning unit 51 sets the learning amount of the second learning to 0, causes the learning unit 51 to perform only the first learning first, sets the segmentation part 16 to the learned segmentation part 16T, and then performs the first learning. The quantity is set to 0, and the learning section 51 is made to perform only the second learning, and the counting portion 17 is set to the learned counting portion 17T.

FIG. 14 is a diagram showing the state of the first learning of the first stage and the second learning of the second stage. In the first stage of the first learning, the learning input image 19 is input to the shared DNN 60 of the basic model 15, as shown in FIG. 14A. A learning output image 65 is output from the segmentation output layer 61 . The learning output image 65 is used together with the annotation image 20 for evaluation of class discrimination accuracy using the first loss function 73 .

In the first stage of the first learning, the DNN 62 for counting and the output layer 63 for counting also output the learning counting result 66 for the time being. However, as shown in FIG. 9, the learning amount of the second learning is set to 0 in the first stage. Therefore, the counting accuracy of the number of objects using the second loss function 74 based on the counting result 22 of the second learning data 21 and the learning counting result 66 is not evaluated.

In the second stage of the second learning, as shown in FIG. 14B, the learning input image 19 is input to the shared DNN 60 of the intermediate model 15M. A learning counting result 66 is output from the counting output layer 63 . The learning counting result 66 is used together with the counting result 22 of the second learning data 21 to evaluate the counting accuracy of the number of objects using the second loss function 74 .

In the second stage of the second learning, the segmentation output layer 61 also outputs the learning output image 65 for the time being. However, as shown in FIG. 9, the learning amount of the first learning is set to 0 in the second stage. Therefore, the accuracy of counting the number of objects using the first loss function 73 based on the annotation image 20 and the learning output image 65 is not evaluated.

The learning unit 51 causes the basic model 15 and the intermediate model 15M to perform mini-batch learning using, for example, mini-batch data. The mini-batch data in the case of the first learning is a plurality of divided images obtained by dividing the learning input image 19 and the annotation image 20 (for example, 10,000 divided images divided by a frame of 1/100 size of the original image). ) (for example, 100 sheets). The mini-batch data in the case of the second learning includes a plurality of divided images obtained by dividing the learning input image 19, and the number of objects appearing in each of a plurality of divided images obtained by dividing the annotation image 20 in the same way as the learning input image 19. It is composed of a part of the set with the counting result 22 . The learning unit 51 creates a plurality of sets (for example, 100 sets) of such mini-batch data, and sequentially gives each set to the basic model 15 and the intermediate model 15M for learning.

As shown in FIG. 15, the second loss function 74 calculates the absolute value of the difference between the learning counting result 66 and the counting result 22 included in the second learning data 21 as the counting result included in the second learning data 21. The counting accuracy is evaluated based on the value obtained by dividing by 22 (hereinafter referred to as the wrong answer rate). For example, when the number of differentiated cell DCs in the learning counting result 66 is 100 and the number of differentiated cell DCs in the counting result 22 of the second learning data 21 is 85, the absolute value of the difference is 15, The error rate is 15/85≈0.18. The absolute value of the difference can also be called the number of incorrect answers.

In FIG. 16, the operating program 80 is stored in the storage device 30B of the operation device 11 . The operating program 80 is an application program for causing the computer to function as the operating device 11 .

A trained model 15T and an input image 25 are also stored in the storage device 30B. The input image 25 is, as described above, an image to be given to the trained model 15T to allow the trained model 15T to discriminate the class and the number of the reflected objects.

When the operation program 80 is activated, the CPU 32B of the computer that constitutes the operation device 11 functions as a RW control section 85, a processing section 86, and a display control section 87 in cooperation with the memory 31 and the like.

The RW control unit 85, like the RW control unit 50 of the learning device 10, controls storage of various data in the storage device 30B and reading of various data in the storage device 30B. The RW control unit 85 stores the trained model 15T transmitted from the learning device 10 in the storage device 30B. The RW control unit 85 also reads out the trained model 15T and the input image 25 from the storage device 30B. The RW control unit 85 outputs the trained model 15T and the input image 25 read out from the storage device 30B to the processing unit 86 . The RW control section 85 also outputs the input image 25 to the display control section 87 .

The processing unit 86 inputs the input image 25 to the trained model 15T, causes the trained model 15T to perform semantic segmentation, and outputs the output image 26 . In addition, the processing unit 86 causes the learned model 15T to count the number of objects, and outputs the counting result 27 . The processing section 86 outputs the output image 26 and the counting result 27 to the display control section 87 .

The display control unit 87 controls display of various screens on the display 34B. The various screens include an evaluation result display screen 90 (see FIG. 17), which is a screen for displaying the output image 26 and the counting result 27, and the like.

As shown in FIG. 17, the input image 25 and the output image 26 are displayed side by side on the evaluation result display screen 90 . A legend 91 for classes 1 to 4 is displayed at the bottom of the output image 26 . Furthermore, the counting result 27 is displayed below the legend 91 . Note that when the OK button 92 is selected, the display of the evaluation result display screen 90 is erased.

Next, the operation of the above configuration will be described with reference to the flow chart of FIG. First, when the operation program 40 is activated in the learning device 10, the CPU 32A of the learning device 10 functions as the RW control unit 50, the learning unit 51, the control unit 52, and the transmission control unit 53, as shown in FIG. be done.

First, as shown in FIG. 10, the RW control unit 50 reads the basic model 15 from the storage device 30A (step ST100). Also, the RW control unit 50 reads the first learning data 18 from the storage device 30A (step ST110). The basic model 15 and the first learning data 18 are output from the RW control section 50 to the learning section 51 . Note that step ST100 is an example of the “acquisition step” according to the technology of the present disclosure.

As shown in FIGS. 10 and 11, learning section 51 performs first learning under the control of control section 52 (step ST120). That is, the learning unit 51 learns the segmentation part 16 of the basic model 15 using the first learning data 18 . As a result, the segmentation portion 16 becomes the learned segmentation portion 16T. The basic model 15 in which the segmentation portion 16 is the learned segmentation portion 16T is output from the learning section 51 to the RW control section 50 as the intermediate model 15M (step ST130). The intermediate model 15M is stored in the storage device 30A by the RW control unit 50. FIG. Note that step ST120 is an example of a "learning step" and a "control step" according to the technology of the present disclosure.

As shown in FIG. 12, the intermediate model 15M is read from the storage device 30A by the RW control unit 50 (step ST140). Also, the RW control unit 50 reads the second learning data 21 from the storage device 30A (step ST150). The intermediate model 15</b>M and the second learning data 21 are output from the RW control section 50 to the learning section 51 .

As shown in FIGS. 12 and 13, learning section 51 performs the second learning under the control of control section 52 (step ST160). That is, the learning unit 51 learns the counting part 17 of the intermediate model 15M using the second learning data 21 . As a result, the counting portion 17 becomes the learned counting portion 17T. Intermediate model 15M in which counting portion 17 is set to learned counting portion 17T is output from learning section 51 to RW control section 50 as learned model 15T (step ST170). The trained model 15T is stored in the storage device 30A by the RW control unit 50. FIG. Note that step ST160, like step ST120, is an example of a "learning step" and a "control step" according to the technology of the present disclosure.

The learned model 15T is read from the storage device 30A by the RW control unit 50 and output from the RW control unit 50 to the transmission control unit 53. FIG. The trained model 15T is transmitted to the operation device 11 by the transmission control unit 53 (step ST180).

In the operation device 11, as shown in FIG. 16, the input image 25 is given to the trained model 15T. Then, an output image 26 obtained by discriminating the classes of objects appearing in the input image 25 and a count result 27 obtained by counting the number of objects appearing in the input image 25 are output from the trained model 15T. The output image 26 and the counting result 27 are displayed on the display 34B of the operation device 11 as the evaluation result display screen 90 as shown in FIG. 17, and are available for viewing by the user.

As described above, the learning device 10 includes the RW control unit 50 as an acquisition unit, the learning unit 51 and the control unit 52 . The RW control unit 50 reads and acquires the basic model 15 including the segmentation portion 16 and the counting portion 17 from the storage device 30A. The learning unit 51 performs first learning for learning the segmentation part 16 using first learning data 18, which is a set of the learning input image 19 and the annotation image 20, and a set of the learning input image 19 and the counting result 22. Second learning for learning the counting part 17 is performed using the second learning data 21, and the basic model 15 is set as a trained model 15T. The control unit 52 makes the learning amount of the first learning larger than the learning amount of the second learning in the first learning stage, and makes the learning amount of the second learning equal to the learning amount of the first learning in the latter learning stage. do more than

In this way, by focusing on the first learning in the first half of the learning stage and increasing the class discrimination accuracy, the second learning in the second half of the learning stage can be efficiently performed. Therefore, it is possible to obtain a trained model 15T with relatively high accuracy in class discrimination in the input image 25 and in counting the number of objects appearing in the input image 25 .

The learning unit 51 uses the first loss function 73 to evaluate the class discrimination accuracy of the segmentation part 16 , and uses the second loss function 74 to evaluate the object count accuracy of the counting part 17 . Therefore, it is possible to appropriately evaluate the class discrimination accuracy and the counting accuracy of the number of objects.

The control unit 52 changes the learning amount of the first learning and the learning amount of the second learning according to the change schedule 43 set in advance. The change schedule 43 sets the learning amount of the second learning to 0, causes the learning unit 51 to perform only the first learning first, sets the segmentation part 16 to the learned segmentation part 16T, and then reduces the learning amount of the first learning to 0, the learning unit 51 is made to perform only the second learning, and the counting part 17 is set to the learned counting part 17T. For this reason, compared to the case where the first learning and the second learning are performed in parallel, the evaluation of the class discrimination accuracy and the counting accuracy of the number of objects by the evaluation unit 71, the segmentation part 16 and the counting part 17 by the update unit 72 Updates can be done easily. Therefore, the trained model 15T can be obtained more easily than when the first learning and the second learning are performed in parallel.

As shown in FIG. 15, the second loss function 74 evaluates the counting accuracy of the number of objects of the counting portion 17 based on the wrong answer rate. This provides the following effects.

Consider a case where the counting accuracy of the number of objects in the counting portion 17 is evaluated based on the absolute value of the difference between the learning counting result 66 and the counting result 22 included in the second learning data 21 . In this case, for example, the number of differentiated cell DCs in the learning counting result 66 is 70 and the number of differentiated cell DCs in the counting result 22 of the second learning data 21 is 100, and the learning counting result 66 The number of differentiated cell DCs in 1 is 7, and the number of differentiated cell DCs in the count result 22 of the second learning data 21 is 10, the counting accuracy of the number of objects is the same. However, the absolute value of the difference is 30 in the former case and 3 in the latter case.

On the other hand, according to the second loss function 74, when the number of differentiated cell DCs in the learning counting result 66 is 70 and the number of differentiated cell DCs in the counting result 22 of the second learning data 21 is 100, , the wrong answer rate is (100-70)/100=0.3, the number of differentiated cell DCs in the learning counting result 66 is 7, and the number of differentiated cell DCs in the counting result 22 of the second learning data 21 is 10. In the case of 1, the wrong answer rate is (10-7)/10=0.3, which is the same. The evaluation of the counting accuracy of the number of objects is also the same between the former and the latter. Therefore, it is possible to ensure the validity of the evaluation of the counting accuracy of the number of objects.

Here, the field of cell culture has recently been in the limelight due to the emergence of iPS (Induced Pluripotent Stem) cells and the like. Therefore, there is a demand for a technique for more accurately determining the class of cells in a cell image and counting the number of cells in the cell image. In the technology of the present disclosure, the input image 25 is a cell image obtained by photographing a plurality of cells in culture. Therefore, it can be said that the technique of the present disclosure is a technique that can meet recent demands.

[Second embodiment]
In the second embodiment shown in FIGS. 19 to 29, learning is performed according to a change schedule that gradually decreases the learning amount of the first learning and gradually increases the learning amount of the second learning.

In FIG. 19, in the second embodiment, a weighted sum 100 is used to evaluate the overall accuracy of the class discrimination accuracy of the segmentation portion 16 and the counting accuracy of the number of objects of the counting portion 17 together. Weighted sum 100 is the sum of first loss function 73 multiplied by a first weighting factor W1 and second loss function 74 multiplied by a second weighting factor W2. The first weighting factor W1 and the second weighting factor W2 are determined so that their sum is 1 (W1+W2=1).

In FIG. 20, the change schedule 105 of the second embodiment registers the first weighting factor W1 and the second weighting factor W2 in a plurality of learning stages. FIG. 20 exemplifies a case where there are six learning stages, ie, the first stage, the second stage, . . . , the fifth stage, and the sixth stage. Then, 1 as the first weighting factor W1 and 0 as the second weighting factor W2 in the first stage, 0.8 as the first weighting factor in the second stage, 0.2 as the second weighting factor W2, and so on. , 0.2 as the first weighting factor W1 and 0.8 as the second weighting factor W2 in the fifth stage, 0 as the first weighting factor in the sixth stage, and 1 as the second weighting factor W2. shows an example.

In this case, since the second weighting factor W2 is 0 in the first stage, the weighted sum 100 is only the first loss function 73 . Therefore, in the first stage, only the class discrimination accuracy of the segmentation portion 16 is evaluated by the first loss function 73, and the counting accuracy of the number of objects of the counting portion 17 is not evaluated by the second loss function 74. do not have. That is, in the first stage, as in the first stage of the first embodiment, only the first learning is performed. On the other hand, in the sixth stage, contrary to the first stage, the first weighting factor is 0, so the weighted sum 100 is only the second loss function 74 . Therefore, in the sixth stage, only the counting accuracy of the number of objects in the counting portion 17 is evaluated by the second loss function 74, and the class discrimination accuracy of the segmentation portion 16 is not evaluated by the first loss function 73. do not have. That is, in the sixth stage, only the second learning is performed, as in the second stage of the first embodiment.

In the second to fifth stages, both the first weighting factor W1 and the second weighting factor W2 are 0 or more. Therefore, in the second to fifth stages, the evaluation of the class discrimination accuracy of the segmentation part 16 by the first loss function 73 and the evaluation of the counting accuracy of the number of objects of the counting part 17 by the second loss function 74 are performed. done. That is, in the second to fifth stages, the first learning and the second learning are performed in parallel.

In the second to fifth stages, as shown in the graph of FIG. 21, the first weighting factor W1 gradually decreases (FIG. 21A) and the second weighting factor W2 gradually increases. (Fig. 21B). Therefore, the weight of learning gradually shifts from the first learning to the second learning as the steps progress.

Thus, in the second embodiment, by changing the first weighting factor W1 of the first loss function 73 and the second weighting factor W2 of the second loss function 74, the learning amount of the first learning and the second learning change the amount of learning In the change schedule 105, the first to third stages are examples of the "first half learning stage" according to the technology of the present disclosure, and the fourth to sixth stages are the "second half of the learning stage" according to the technology of the present disclosure. It is an example of "learning stage".

In FIG. 22, the learning unit 110 of the second embodiment has a processing unit 111, an evaluation unit 112, and an updating unit 113, like the learning unit 51 of the first embodiment. The processing unit 111 gives the learning input image 19 to the segmentation part 16 of the basic model 15 or the intermediate model 15M (only the basic model 15 is shown in FIG. 22), and causes the segmentation part 16 to output the learning output image 65. The processing unit 111 outputs the learning output image 65 to the evaluation unit 112 . Further, the processing unit 111 gives the learning input image 19 to the counting part 17 of the basic model 15 or the intermediate model 15M, and causes the counting part 17 to output the learning counting result 66 . The processing unit 111 outputs the learning counting result 66 to the evaluation unit 112 .

The evaluation unit 112 compares the annotation image 20 and the learning output image 65 to evaluate the class discrimination accuracy of the segmentation portion 16 . The evaluation unit 112 also compares the counting result 22 and the learning counting result 66 to evaluate the accuracy of counting the number of objects in the counting portion 17 . The evaluation unit 112 uses the weighted sum 100 to evaluate the total accuracy of the class discrimination accuracy of the segmentation part 16 and the counting accuracy of the number of objects of the counting part 17 . The closer the calculated value of the weighted sum 100 to 0, the higher the overall accuracy. The evaluation unit 112 outputs the overall accuracy evaluation result based on the weighted sum 100 to the updating unit 113 . In the first stage, the evaluation unit 112 uses the first loss function 73 to evaluate only the class discrimination accuracy of the segmentation part 16, and in the sixth stage, the second loss function 74 is used to evaluate the counting part Only the counting accuracy of the number of 17 objects is evaluated.

The updating unit 113 updates the segmentation part 16 and the counting part 17 according to the evaluation result from the evaluating unit 112 . As with the updating unit 72 of the first embodiment, the updating unit 113 changes the parameter values of the convolutional filter of the shared DNN 60 by stochastic gradient descent with learning coefficients or the like. Further, the updating unit 113 changes parameter values and the like of the counting DNN 62 .

The learning unit 110 inputs the learning input image 19 to the basic model 15 or the intermediate model 15M by the processing unit 111 and outputs the learning output image 65 and/or the learning counting result 66 to the evaluation unit 112, and the evaluation unit Evaluation of the class discrimination accuracy of the segmentation part 16 by 112 and/or evaluation of the counting accuracy of the number of objects of the counting part 17, and updating of the segmentation part 16 and/or the counting part 17 by the updating part 113 are performed at each learning stage. , the class discrimination accuracy of the segmentation portion 16, the counting accuracy of the number of objects of the counting portion 17, or the overall accuracy reaches a preset level. Then, the learning unit 110 selects the basic model 15 or the intermediate model 15M whose class discrimination accuracy in the segmentation part 16, counting accuracy of the number of objects in the counting part 17, or overall accuracy has reached a preset level. is output as an intermediate model 15M or a trained model 15T at the learning stage.

As shown in Table 115 of FIG. 23, the learning unit 110 first learns the basic model 15 in the first stage to obtain a first intermediate model 15M_1 (see FIG. 24B). In the second stage, the first intermediate model 15M_1 is trained to obtain a second intermediate model 15M_2 (see FIG. 25A). In the third stage, the second intermediate model 15M_2 is learned to form a third intermediate model 15M_3 (see FIG. 25B). In the fourth stage, the third intermediate model 15M_3 is learned to be the fourth intermediate model 15M_4 (see FIG. 26A). In the fifth stage, the fourth intermediate model 15M_4 is learned to be the fifth intermediate model 15M_5 (see FIG. 26B). Finally, in the sixth stage, the learning unit 110 learns the fifth intermediate model 15M_5 to make it a trained model 15T.

FIG. 24 is a diagram showing the state of the first learning in the first stage and the first learning and the second learning in the second stage. In the first stage, as shown in FIG. 24A, training input images 19 are input to the shared DNN 60 of the basic model 15 . A learning output image 65 is output from the segmentation output layer 61 . The learning output image 65 is used together with the annotation image 20 for evaluation of class discrimination accuracy using the first loss function 73 . In the first stage, as in the first stage of the first embodiment, the counting accuracy of the number of objects using the second loss function 74 based on the counting result 22 of the second learning data 21 and the learning counting result 66 is not evaluated.

In the second stage, as shown in FIG. 24B, the learning input image 19 is input to the shared DNN 60 of the first intermediate model 15M_1 output in the first stage. A learning output image 65 is output from the segmentation output layer 61 , and a learning counting result 66 is output from the counting output layer 63 . The output image for learning 65 and the counting result for learning 66, together with the annotation image 20 and the counting result 22, are weighted sum 100 (=0.8×first loss function+0.2×second loss function). It is used for accuracy evaluation.

FIG. 25 is a diagram showing the first learning and second learning in the third stage and the first learning and second learning in the fourth stage. In the third stage, as shown in FIG. 25A, the learning input image 19 is input to the shared DNN 60 of the second intermediate model 15M_2 output in the second stage. A learning output image 65 is output from the segmentation output layer 61 , and a learning counting result 66 is output from the counting output layer 63 . The output image for learning 65 and the counting result for learning 66, together with the annotation image 20 and the counting result 22, are the weighted sum 100 (=0.6×first loss function+0.4×second loss function). It is used for accuracy evaluation.

In the fourth stage, as shown in FIG. 25B, the learning input image 19 is input to the shared DNN 60 of the third intermediate model 15M_3 output in the third stage. A learning output image 65 is output from the segmentation output layer 61 , and a learning counting result 66 is output from the counting output layer 63 . The output image for learning 65 and the counting result for learning 66, together with the annotation image 20 and the counting result 22, are the weighted sum 100 (=0.4×first loss function+0.6×second loss function). It is used for accuracy evaluation.

FIG. 26 is a diagram showing the first learning and second learning in the fifth stage and the second learning in the sixth stage. In the fifth stage, as shown in FIG. 26A, the learning input image 19 is input to the shared DNN 60 of the fourth intermediate model 15M_4 output in the fourth stage. A learning output image 65 is output from the segmentation output layer 61 , and a learning counting result 66 is output from the counting output layer 63 . The output image for learning 65 and the counting result for learning 66, together with the annotation image 20 and the counting result 22, are the overall It is used for accuracy evaluation.

In the sixth stage, as shown in FIG. 26B, the learning input image 19 is input to the shared DNN 60 of the fifth intermediate model 15M_5 output in the fifth stage. A learning counting result 66 is output from the counting output layer 63 . The learning counting result 66 is used together with the counting result 22 to evaluate the accuracy of counting the number of objects using the second loss function 74 . In the sixth stage, as in the second stage of the first embodiment, evaluation of class discrimination accuracy using the first loss function 73 based on the annotation image 20 and the learning output image 65 is not performed.

As shown in FIG. 27, in the second embodiment, first, the basic model 15 is read from the storage device 30A by the RW control unit 50 (step ST200). Also, the RW control unit 50 reads the first learning data 18 from the storage device 30A (step ST210). The basic model 15 and the first learning data 18 are output from the RW control section 50 to the learning section 110 . Note that step ST200 is an example of the “acquisition step” according to the technology of the present disclosure.

According to the change schedule 105 shown in FIGS. 20 and 21, the first weighting factor W1 is set to 1 and the second weighting factor W2 is set to 0 (step ST220). Next, as shown in FIG. 24A, learning section 110 performs first learning under the control of control section 52 (step ST230). The segmentation part 16 is thereby learned. The basic model 15 in which the segmentation portion 16 has been learned is output as the first intermediate model 15M_1 from the learning section 51 to the RW control section 50 (NO in step ST240, step ST250). The first intermediate model 15M_1 is stored in the storage device 30A by the RW control unit 50. FIG. Note that step ST230 is an example of a "learning step" and a "control step" according to the technology of the present disclosure.

The RW control unit 50 reads the first intermediate model 15M_1 from the storage device 30A (step ST200). Also, the RW control unit 50 reads the first learning data 18 and the second learning data 21 from the storage device 30A (step ST210). The first intermediate model 15M_1, the first learning data 18, and the second learning data 21 are output from the RW control section 50 to the learning section 110. FIG.

According to the change schedule 105, the first weighting factor W1 is set to 0.8 and the second weighting factor W2 is set to 0.2 (step ST220). Next, as shown in FIG. 24B, learning section 110 performs first learning and second learning under the control of control section 52 (step ST230). The segmentation part 16 and the counting part 17 are thereby learned. The first intermediate model 15M_1 for which the segmentation portion 16 and the counting portion 17 have been learned is output as the second intermediate model 15M_2 from the learning section 51 to the RW control section 50 (NO in step ST240, step ST250). The second intermediate model 15M_2 is stored in the storage device 30A by the RW control unit 50. FIG. Steps ST200 to ST230 and ST250 are repeated until the final stage, the sixth stage, is reached.

In the case of the sixth step, which is the final step (YES in step ST240), the fifth intermediate model 15M_5 in which the segmentation portion 16 and the counting portion 17 have been trained is output from the learning section 51 to the RW control section 50 as the trained model 15T. (Step ST260). The trained model 15T is stored in the storage device 30A by the RW control unit 50. FIG.

The learned model 15T is read from the storage device 30A by the RW control unit 50 and output from the RW control unit 50 to the transmission control unit 53. FIG. The trained model 15T is transmitted to the operation device 11 by the transmission control unit 53 (step ST270).

As described above, in the second embodiment, learning is performed according to the change schedule 105 in which the learning amount of the first learning is gradually decreased and the learning amount of the second learning is gradually increased. In the first embodiment, there is no learning stage other than the first stage for increasing the class discrimination accuracy of the segmentation part 16, and there is no learning stage other than the second stage for increasing the counting accuracy of the number of objects in the counting part 17. On the other hand, in the second embodiment, the class discrimination accuracy of the segmentation portion 16 and the counting accuracy of the number of objects of the counting portion 17 can be increased over a plurality of stages.

The learning unit 110 uses the weighted sum 100 of the first loss function 73 and the second loss function 74 to obtain the total accuracy of the class discrimination accuracy of the segmentation part 16 and the counting accuracy of the number of objects of the counting part 17. evaluate. Therefore, it is possible to appropriately evaluate the class discrimination accuracy and the counting accuracy of the number of objects.

Further, in the second embodiment, by changing the first weighting factor W1 of the first loss function 73 and the second weighting factor W2 of the second loss function 74, the learning amount of the first learning and the learning amount of the second learning changing the amount. Therefore, the learning amount of the first learning and the learning amount of the second learning can be easily changed.

Note that the change schedule of gradually decreasing the learning amount of the first learning and gradually increasing the learning amount of the second learning is not limited to the change schedule 105 illustrated in FIGS. 20 and 21 . For example, like the change schedule 117 shown in FIG. 28, both the first weighting factor W1 and the second weighting factor W2 are the same 0.5, and the learning amount of the first learning and the learning amount of the second learning are the same. may contain. Also, as in the change schedule 119 shown in FIG. 29, the first weighting factor W1 in the first stage is set to a value smaller than 1, and the second weighting factor W2 in the final stage (here, the ninth stage) is set to a value smaller than 1. may be In short, the learning amount of the first learning should be larger than the learning amount of the second learning in the first learning stage, and the learning amount of the second learning should be larger than the learning amount of the first learning in the latter learning stage.

[Third embodiment]
In the third embodiment shown in FIGS. 30 to 33, the learning amount of the first learning is temporarily increased in the process of gradually decreasing, and in the process of gradually increasing the learning amount of the second learning, Learning is performed in accordance with a change schedule that reduces the amount of learning in the first learning in accordance with the timing of temporarily increasing the amount of learning.

The change schedule 125 shown in FIGS. 30 and 31 and the change schedule 127 shown in FIGS. 32 and 33 are basically the same as the change schedule 105 shown in FIGS. The content is to gradually decrease the amount of learning in the first learning and gradually increase the amount of learning in the second learning. However, the change schedules 125 and 127 temporarily increase the first weighting factor W1 to increase the learning amount of the first learning as shown in FIGS. 31A and 33A when shifting to the next learning stage. , along with this, as shown in FIGS. 31B and 33B, the second weighting factor W2 is temporarily reduced to reduce the amount of learning in the second learning.

The change schedule 125 shown in FIGS. 30 and 31 raises the first weighting factor W1 when shifting to the next learning stage to the first weighting factor W1 of the immediately preceding learning stage, and shifts to the next learning stage. The content is that the second weighting factor W2 at the time is reduced to the second weighting factor W2 at the immediately preceding learning stage. The change schedule 127 shown in FIGS. 32 and 33 uniformly raises the first weighting factor W1 when shifting to the next learning stage to the maximum value of 1, and increases the second weighting factor W1 when shifting to the next learning stage. The content is that the coefficient W2 is uniformly reduced to the minimum value of 0.

Thus, in the third embodiment, the learning amount of the first learning is temporarily increased in the process of gradually decreasing the learning amount of the first learning, and the learning amount of the second learning is gradually increased in the process of the first learning. Learning is performed according to change schedules 125 and 127, which are to reduce the amount of learning of . Therefore, it is possible to review the first learning, which has fewer chances to be implemented as it progresses, at appropriate timings. A trained model 15T with higher class discrimination accuracy and higher object count accuracy can be obtained.

In addition, the learning amount of the first learning is temporarily increased in the process of gradually decreasing the learning amount of the first learning, and the learning amount of the first learning is temporarily increased in the process of gradually increasing the learning amount of the second learning. The change schedules of content of reducing the amount according to the timing of the change are not limited to the change schedules 125 and 127 exemplified in FIGS. Instead of moving to the next learning stage, only once between the first learning stage and the second half learning stage, the amount of learning in the first learning is increased compared to the previous learning stage, and the amount of learning in the second learning is increased. It can be less. Also, not only when moving to the next learning stage, but at some timing in one learning stage, even if the learning amount of the first learning is increased and the learning amount of the second learning is decreased good.

[Fourth embodiment]
In the fourth embodiment shown in FIG. 34, user designation of a change schedule is accepted.

In FIG. 34 , a change schedule designation screen 130 is displayed on the display 34 of the study device 10 . The change schedule designation screen 130 has a change schedule designation area 131 . Input boxes 132 and 133 , a check box 134 , and a step addition button 135 are provided in the change schedule designation area 131 . The input box 132 receives the first weighting factor W1 in each learning stage. The input box 133 receives the second weighting factor W2 in each learning stage. The check box 134 is selected when it is desired to temporarily increase the learning amount of the first learning as in the third embodiment. The step addition button 135 is selected when adding a step.

The user inputs the first weighting factor W1 and the second weighting factor W2 in the input boxes 132 and 133, selects the check box 134 as necessary, and then selects the designation button 136. FIG. Accepting unit 138 accepts designation of a change schedule that reflects the input contents of input boxes 132 and 133 and the selection contents of check box 134 . The accepting unit 138 outputs the change schedule for which the designation has been accepted to the RW control unit 50 . The RW control unit 50 stores the change schedule from the receiving unit 138 in the storage device 30A. Learning is performed according to the change schedule stored in this storage device 30A. Note that when the cancel button 137 is selected, the display of the change schedule designation screen 130 is erased.

As described above, in the fourth embodiment, the reception unit 138 receives designation of a change schedule by the user. Therefore, it becomes possible for the user to customize the change schedule, and it is possible to perform learning suitable for the user's preference.

Various modifications are possible for the hardware configuration of the computer that constitutes the machine learning system 2 . For example, the learning device 10 and the operation device 11 may be integrated into one computer. Also, at least one of the learning device 10 and the operation device 11 can be configured with a plurality of computers separated as hardware for the purpose of improving processing capability and reliability. For example, the function of the processing unit 70 of the learning unit 51 of the learning device 10 and the functions of the evaluation unit 71 and update unit 72 are distributed to two computers. In this case, the learning device 10 is composed of two computers.

In this way, the hardware configuration of the computer of the machine learning system 2 can be appropriately changed according to required performance such as processing power, safety, and reliability. Furthermore, not only the hardware but also application programs such as the operation program 40 can of course be duplicated or distributed and stored in multiple storage devices for the purpose of ensuring safety and reliability. be.

In each of the above embodiments, the first weighting factor W1 and the second weighting factor W2 are changed stepwise, but the present invention is not limited to this. For example, like a change schedule 145 shown in FIG. 35, the first weighting factor W1 and the second weighting factor W2 may be changed linearly. Alternatively, like a change schedule 147 shown in FIG. 36, the first weighting factor W1 and the second weighting factor W2 may be changed in a curved line.

Note that the DNN 62 for counting may be connected to the output layer 61 for segmentation, and the output image 65 for learning from the output layer 61 for segmentation may be output to the DNN 62 for counting. Also, both the output data from the shared DNN 60 and the learning output image 65 from the segmentation output layer 61 may be output to the counting DNN 62 .

In each of the above-described embodiments, the input image 25 is a cell image obtained by photographing a plurality of cells in culture, and the classes are cells, medium, and the like, but the input image 25 is not limited to this. For example, a satellite moving image of a road may be used as the input image 25, and the number of cars may be counted using cars as classes. Alternatively, the number of people may be counted by using a moving image of the street taken as the input image 25 and taking people as a class.

In each of the above embodiments, for example, RW control units 50 and 85, learning units 51 and 110 (processing units 70 and 111, evaluation units 71 and 112, update units 72 and 113), control unit 52, transmission control unit 53, processing As a hardware structure of a processing unit (processing unit) that executes various processes such as the unit 86, the display control unit 87, and the reception unit 138, the following various processors can be used. Various processors include, as described above, FPGAs (Field Programmable Gate Arrays) in addition to CPUs 32A and 32B, which are general-purpose processors that execute software (operation programs 40 and 80) and function as various processing units. Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed after manufacture, ASIC (Application Specific Integrated Circuit), etc. It has a circuit configuration specially designed to execute specific processing. A dedicated electrical circuit, such as a processor, is included.

One processing unit may be configured with one of these various processors, or a combination of two or more processors of the same or different type (for example, a combination of a plurality of FPGAs and/or a CPU and combination with FPGA). Also, a plurality of processing units may be configured by one processor.

As an example of configuring a plurality of processing units with a single processor, first, as represented by computers such as clients and servers, a single processor is configured by combining one or more CPUs and software. There is a form in which a processor functions as multiple processing units. Secondly, as typified by System On Chip (SoC), etc., there is a mode of using a processor that realizes the function of the entire system including multiple processing units with a single IC (Integrated Circuit) chip. be. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

Further, as the hardware structure of these various processors, more specifically, an electric circuit in which circuit elements such as semiconductor elements are combined can be used.

From the above description, the invention described in the following supplementary item 1 can be grasped.

[Appendix 1]
A segmentation part that performs semantic segmentation for determining a class, which is the type of an object appearing in an input image, on a pixel-by-pixel basis; and a counting part that performs counting of the number of objects whose class is determined by the semantic segmentation. an acquisition processor that acquires an underlying machine learning model;
first learning for learning the segmentation portion using first learning data that is a set of an input image for learning and an annotation image in which the class is specified for the input image for learning; and the input for learning. Using second learning data, which is a set of an image and a count result of the number of the objects appearing in the input image for learning, second learning is performed to learn the counting part, and the basic machine learning model is learned. a learning processor as a finished machine learning model;
A control processor for controlling the operation of the learning processor, wherein the learning amount of the first learning is larger than the learning amount of the second learning in the first half learning stage, and the second learning amount is increased in the latter half learning stage. a control processor that makes the learning amount of learning larger than the learning amount of the first learning;
A learning device with

The technology of the present disclosure can also appropriately combine various embodiments and various modifications described above. Moreover, it is needless to say that various configurations can be employed without departing from the scope of the present invention without being limited to the above embodiments. Furthermore, the technology of the present disclosure extends to storage media that non-temporarily store programs in addition to programs.

The above description and illustration are detailed descriptions of the parts related to the technology of the present disclosure, and are merely examples of the technology of the present disclosure. For example, the above descriptions of configurations, functions, actions, and effects are descriptions of examples of configurations, functions, actions, and effects of portions related to the technology of the present disclosure. Therefore, unnecessary parts may be deleted, new elements added, or replaced with respect to the above-described description and illustration without departing from the gist of the technology of the present disclosure. Needless to say. In addition, in order to avoid complication and facilitate understanding of the portion related to the technology of the present disclosure, the descriptions and illustrations shown above require no particular explanation in order to enable implementation of the technology of the present disclosure. Descriptions of common technical knowledge, etc., that are not used are omitted.

As used herein, "A and/or B" is synonymous with "at least one of A and B." That is, "A and/or B" means that only A, only B, or a combination of A and B may be used. In addition, in this specification, when three or more matters are expressed by connecting with "and/or", the same idea as "A and/or B" is applied.

All publications, patent applications and technical standards mentioned herein are expressly incorporated herein by reference to the same extent as if each individual publication, patent application and technical standard were specifically and individually noted to be incorporated by reference. incorporated by reference into the book.

Claims (12)

A segmentation part that performs semantic segmentation for determining a class, which is the type of an object appearing in an input image, on a pixel-by-pixel basis; and a counting part that performs counting of the number of objects whose class is determined by the semantic segmentation. an acquisition unit that acquires a basic machine learning model;
first learning for learning the segmentation portion using first learning data that is a set of an input image for learning and an annotation image in which the class is specified for the input image for learning; and the input for learning. Using second learning data, which is a set of an image and a count result of the number of the objects appearing in the input image for learning, second learning is performed to learn the counting part, and the basic machine learning model is learned. a learning unit as a finished machine learning model;
A control unit for controlling the operation of the learning unit, wherein the learning amount of the first learning is larger than the learning amount of the second learning in the first learning stage, and the learning amount of the second learning is increased in the latter learning stage. a control unit that makes the learning amount of learning larger than the learning amount of the first learning;
A learning device with 2. The method according to claim 1, wherein the learning unit evaluates the class discrimination accuracy of the segmentation part using a first loss function, and evaluates the counting accuracy of the number of the objects of the counting part using a second loss function. A learning device as described. 3. The learning device according to claim 2, wherein the control unit changes the learning amount of the first learning and the learning amount of the second learning according to a preset change schedule. The change schedule sets the learning amount of the second learning to 0, causes the learning unit to perform only the first learning first, sets the segmentation part to a learned segmentation part, and then performs the learning of the first learning. The first learning and the second learning are divided into two steps, such that the learning unit is caused to perform only the second learning and the counting portion is set to the learned counting portion with the amount set to 0. 4. A learning device as claimed in claim 3. 4. The learning device according to claim 3, wherein the change schedule gradually decreases the learning amount of the first learning and gradually increases the learning amount of the second learning. In the change schedule, the learning amount of the first learning is temporarily increased in the process of gradually decreasing the learning amount of the second learning, and the learning amount of the first learning is increased in the process of gradually increasing the learning amount of the second learning. 4. The learning device according to claim 3, wherein the content is that the amount is decreased in accordance with the timing of increasing the amount temporarily. The learning unit evaluates the overall accuracy including the discrimination accuracy and the counting accuracy by a weighted sum of the first loss function and the second loss function,
The control unit changes the learning amount of the first learning and the learning amount of the second learning by changing a first weighting factor of the first loss function and a second weighting factor of the second loss function. 7. The learning device according to claim 5 or 6. 8. The learning device according to any one of claims 3 to 7, further comprising a reception unit that receives designation of the change schedule by a user. The second loss function is a learning counting result, which is the result of counting the number of the objects appearing in the learning input image by the counting part, and a counting result of the number of the objects included in the second learning data. 9. The counting accuracy according to any one of claims 2 to 8, wherein the counting accuracy is evaluated based on a value obtained by dividing an absolute value of the difference by a counting result of the number of the objects included in the second learning data. learning device. 10. The learning device according to any one of claims 1 to 9, wherein the input image is a cell image obtained by photographing a plurality of cells in culture. A segmentation part that performs semantic segmentation for determining a class, which is the type of an object appearing in an input image, on a pixel-by-pixel basis; and a counting part that performs counting of the number of objects whose class is determined by the semantic segmentation. an acquisition step of acquiring an underlying machine learning model;
first learning for learning the segmentation portion using first learning data that is a set of an input image for learning and an annotation image in which the class is specified for the input image for learning; and the input for learning. Using second learning data, which is a set of an image and a count result of the number of the objects appearing in the input image for learning, second learning is performed to learn the counting part, and the basic machine learning model is learned. a learning step as a completed machine learning model;
In the first learning stage, the learning amount of the first learning is made larger than the learning amount of the second learning, and in the latter learning stage, the learning amount of the second learning is made larger than the learning amount of the first learning. a control step that increases the
A method of operating a learning device comprising: A segmentation part that performs semantic segmentation for determining a class, which is the type of an object appearing in an input image, on a pixel-by-pixel basis; and a counting part that performs counting of the number of objects whose class is determined by the semantic segmentation. an acquisition unit that acquires a basic machine learning model;
first learning for learning the segmentation portion using first learning data that is a set of an input image for learning and an annotation image in which the class is specified for the input image for learning; and the input for learning. Using second learning data, which is a set of an image and a count result of the number of the objects appearing in the input image for learning, second learning is performed to learn the counting part, and the basic machine learning model is learned. a learning unit as a finished machine learning model;
A control unit for controlling the operation of the learning unit, wherein the learning amount of the first learning is larger than the learning amount of the second learning in the first learning stage, and the learning amount of the second learning is increased in the latter learning stage. As a control unit that makes the learning amount of learning larger than the learning amount of the first learning,
An operating program for a learning device that makes a computer work.

Images