JP7167356B2 - LEARNING APPARATUS, LEARNING APPARATUS OPERATING METHOD, LEARNING APPARATUS OPERATING PROGRAM - Google Patents

Description

The technology of the present disclosure relates to a learning device, a learning device operating method, and a learning device operating program.

Semantic segmentation is known that distinguishes between multiple classes in an input image on a pixel-by-pixel basis. A class is the type of object appearing in the input image. A machine learning model (hereinafter simply referred to as a model) that implements semantic segmentation includes a U-shaped convolutional neural network (U-Net).

Japanese Patent Application Laid-Open No. 2019-016298 describes that all of a plurality of relatively detailed classes such as grass, flowers, skin, and hair are initially set to learn a model. In Japanese Patent Application Laid-Open No. 2019-016298, it is determined for each class whether or not class identification is difficult based on input image information such as the amount of blur and image magnification. Classes that are difficult to identify are integrated into higher-level classes, such as "grass and flowers" and "face" for skin and hair, and then the model is retrained.

In learning a model that implements semantic segmentation, there is a problem that high discrimination accuracy cannot be obtained if all of a plurality of classes are first learned at once, as in Japanese Patent Application Laid-Open No. 2019-016298.

The technology of the present disclosure provides a learning device, a learning device operation method, and a learning device operation program that can obtain a machine learning model with higher class discrimination accuracy than when learning all of a plurality of classes at once. intended to provide

In order to achieve the above object, the learning device of the present disclosure is a set of an input image for learning and an annotation image in which three or more classes to be subjected to semantic segmentation are specified for the input image for learning. an acquisition unit that acquires certain learning data; and an image generation unit that integrates at least two classes out of three or more classes specified in the annotation image and generates a temporary annotation image having fewer classes than the annotation image. , using temporary learning data, which is a set of an input image for learning and a temporary annotation image, to learn a temporary machine learning model having a configuration corresponding to the number of classes of temporary annotation images, and train the temporary machine learning model as a trained temporary machine A temporary learning unit as a learning model; a machine learning model generating unit that generates a machine learning model having a configuration corresponding to the number of classes of annotation images using at least a part of the trained temporary machine learning model; and a main learning unit that uses the machine learning model to learn and sets the machine learning model as a trained machine learning model.

It is preferable to include a temporary machine learning model generation unit that generates the temporary machine learning model.

When the number of classes of annotation images is M and the number of classes of temporary annotation images is N, the image generation unit and the temporary learning unit set the processing for generating temporary annotation images and the temporary machine learning model as a trained temporary machine learning model. It is preferable to repeat the process multiple times while gradually increasing the class number N of the temporary annotation images until the class number N of the temporary annotation images reaches M−1.

It is preferable that the temporary machine learning model generation unit generates the temporary machine learning model to be used this time, using at least part of the temporary machine learning model used last time.

It is preferable that the provisional learning unit changes the loss function used to evaluate the class discrimination accuracy of the provisional machine learning model. In this case, the provisional learning unit preferably makes the weight of the loss function for the class common to the previous time smaller than the weight of the loss function for the class newly appearing this time.

It is preferable that the provisional learning unit and the main learning unit do not update a pre-specified part in the process of converting the temporary machine learning model into the trained temporary machine learning model and the process of converting the machine learning model into the trained temporary machine learning model. In this case, it is preferable to provide a first reception unit that receives the user's designation of the portion not to be updated.

It is preferable that the image generator generates the temporary annotation image according to image generation conditions specified in advance.

It is preferable to include a second reception unit that receives designation of image generation conditions by the user.

It is preferable that the image generation conditions are such that the area of each class of the temporary annotation image is not biased. In addition, it is preferable that the image generation condition is such that the complexity of each class of the temporary annotation image is not biased.

The user selects and instructs whether the image generation condition is such that the area of each class of the temporary annotation image is balanced or the image generation condition is such that the complexity of each class of the temporary annotation image is balanced. It is preferable to provide a third receiving section for receiving.

It is preferable that the image generation condition is such that the class having the largest area is left as one class without being merged. Moreover, it is preferable that the image generation condition is such that the class with the lowest complexity is left as one class without being integrated.

The image generation condition is that the class with the largest area is left as one class without being merged, or the class with the lowest complexity is left as one class without being merged. It is preferable to include a fourth reception unit that receives a selection instruction from the user as to whether the content is to be the image generation condition.

It is preferable to include a display control unit that controls display of the temporary annotation image.

The image generation unit generates multiple types of temporary annotation images of different integrated classes, the display control unit performs control to display the multiple types of temporary annotation images, and selects one of the multiple types of temporary annotation images. It is preferable that a fifth accepting unit that accepts a user's instruction to select a temporary annotation image is provided, and the temporary learning unit uses the temporary annotation image for which the fifth accepting unit has accepted the selection instruction as the temporary learning data.

The learning input image is preferably a cell image obtained by photographing a plurality of cells being cultured.

The operating method of the learning device of the present disclosure acquires learning data that is a set of an input image for learning and an annotation image in which three or more classes to be subjected to semantic segmentation are specified for the input image for learning. an image generation step of integrating at least two classes among three or more classes specified in the annotation image to generate a temporary annotation image having fewer classes than the annotation image; and an input image for learning and a temporary annotation image, a temporary machine learning model having a configuration corresponding to the number of classes of the temporary annotation image is trained, and the temporary machine learning model is a trained temporary machine learning model. A learning step, a machine learning model generating step of generating a machine learning model having a configuration corresponding to the number of annotation image classes using at least a part of the trained temporary machine learning model, and a machine learning model using the learning data and a main learning step of learning the machine learning model as a learned machine learning model.

The operation program of the learning device of the present disclosure acquires learning data that is a set of an input image for learning and an annotation image in which three or more classes to be subjected to semantic segmentation are specified for the input image for learning. an acquisition unit that integrates at least two classes out of three or more classes specified in the annotation image, and an image generation unit that generates a temporary annotation image having fewer classes than the annotation image; and an input image for learning and a temporary annotation image, a temporary machine learning model having a configuration corresponding to the number of classes of the temporary annotation image is trained, and the temporary machine learning model is a trained temporary machine learning model. A learning unit, a machine learning model generating unit that generates a machine learning model having a configuration corresponding to the number of annotation image classes using at least a part of the trained temporary machine learning model, and a machine learning model using the learning data. is learned, and the computer functions as a main learning unit that uses the machine learning model as a trained machine learning model.

According to the technology of the present disclosure, a learning device capable of obtaining a machine learning model with higher class discrimination accuracy than when learning all of a plurality of classes at once, a method of operating the learning device, and a learning device An operating program can be provided.

1 illustrates a machine learning system; FIG. It is a figure which shows the outline|summary of the process in a machine-learning system. FIG. 10 is a diagram showing an input image for learning and an annotation image; FIG. 4 is a diagram showing a model; 1 is a block diagram showing a computer that constitutes a learning device; FIG. 3 is a block diagram showing a processing section of a CPU of the learning device; FIG. FIG. 4 is a diagram showing a group of images; It is a figure which shows a model group. It is a table showing the number of classes in provisional learning and main learning. FIG. 4 is a diagram showing image generation conditions; FIG. 10 is a diagram showing an image generation condition specification screen; It is a figure which shows the process of the image production|generation part in 1st temporary learning. It is a figure which shows the process of the temporary model production|generation part in 1st temporary learning. It is a figure which shows the process of the provisional learning part in 1st provisional learning. It is a figure which shows the process of the image production|generation part in the 2nd provisional learning. It is a figure which shows the process of the temporary model production|generation part in 2nd temporary learning. It is a figure which shows the process of the provisional learning part in 2nd provisional learning. It is a figure which shows the process of a model generation part. It is a figure which shows the process of this learning part. FIG. 10 is a diagram showing annotation images and provisional annotation images used in the first provisional learning; It is a figure which shows the detail of the process of the temporary model production|generation part in 1st temporary learning. FIG. 10 is a diagram showing annotation images and provisional annotation images used in the second provisional learning; It is a figure which shows the detail of the process of the temporary model production|generation part in 2nd temporary learning. It is a figure which shows the detail of a provisional learning part. It is a figure which shows the 1st loss function used for 1st temporary learning. It is a figure which shows the 1st loss function used for the 2nd provisional learning. It is a figure which shows the detail of the process of a model production|generation part. It is a figure which shows the detail of this learning part. It is a figure which shows the 2nd loss function used for this learning. FIG. 30A is a diagram showing the transition of each learning class of the first provisional learning, the second provisional learning, and the main learning using a provisional annotation image and an annotation image, FIG. 30A is a provisional annotation used for the first provisional learning; 30B shows a temporary annotation image used in the second temporary learning, and FIG. 30C shows an annotation image used in the main learning. 4 is a flow chart showing a processing procedure of the learning device; It is a figure which shows a non-updated part designation|designated screen. FIG. 10 is a diagram showing how the first updating unit of the provisional learning unit does not update the non-updated part; FIG. 10 is a diagram showing how the second updating unit of the learning unit does not update non-updated portions; FIG. 10 is a diagram showing image generation conditions such that the area of each class of a temporary annotation image is not biased; FIG. 10 is a diagram showing image generation conditions such that the degree of complexity of each class of temporary annotation images is not biased. FIG. 10 is a diagram showing an image generation condition specification screen; FIG. 10 is a diagram showing image generation conditions such that the class having the largest area is left as one class without being integrated; FIG. 10 is a diagram showing an image generation condition that the class with the lowest complexity is left as one class without being integrated; FIG. 10 is a diagram showing an image generation condition specification screen; It is a figure which shows a temporary annotation image display screen. FIG. 42A is a diagram showing how an image generation unit generates a plurality of types of provisional annotation images with different integrated classes. FIG. 42A is a provisional annotation image in which differentiated cells of class 1 and undifferentiated cells of class 2 are integrated. 42B shows how to generate a temporary annotation image in which class 1 differentiated cells and class 3 dead cells are integrated, and FIG. 42C shows how class 2 undifferentiated cells and class 3 dead cells are generated. , respectively, generate a temporary annotation image in which are integrated. It is a figure which shows a temporary annotation image selection screen.

[First embodiment]
In FIG. 1, the machine learning system 2 is a system that uses a model for implementing semantic segmentation that distinguishes between multiple classes in an input image on a pixel-by-pixel basis. A machine learning system 2 includes a learning device 10 and an operation device 11 . The learning device 10 and the operation device 11 are, for example, desktop personal computers. Learning device 10 and operation device 11 are connected via network 12 so as to be able to communicate with each other. The network 12 is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet or a public communication network.

In FIG. 2, the learning device 10 performs provisional learning and main learning. In the provisional learning, the provisional model 16 is learned using the provisional learning data 15, and the provisional model 16 is set as the trained provisional model 16T. The provisional learning data 15 is a set of a learning input image 17 and a provisional annotation image 18 . A temporary annotation image 18 is generated from the annotation image 19 . In this learning, the learning data 20 is used to learn the model 21, and the model 21 is a trained model 21T. The model 21 is generated from the trained temporary model 16T. The learning data 20 is a set of the learning input image 17 and the annotation image 19 . Temporary model 16 and model 21 include, for example, a convolutional neural network such as U-Net (see FIG. 4).

The learning device 10 transmits the trained model 21T to the operation device 11 . The operation device 11 receives the trained model 21T from the learning device 10 . The operation device 11 provides the learned model 21T with the input image 22 in which the class of the reflected object and its outline have not yet been determined. The trained model 21T performs semantic segmentation on the input image 22 to discriminate classes and outlines of objects appearing in the input image 22, and outputs an output image 23 as the discrimination result. Even after the trained model 21T is installed in the operation device 11, the learned model 21T may be given the learning data 20 to be learned.

As shown in FIG. 3, the learning input image 17 is, in this example, a cell image obtained by photographing a plurality of cells in culture with a phase-contrast microscope or the like. Differentiated cell DC, undifferentiated cell UDC, dead cell DDC, and medium PL are shown in the learning input image 17 . In this case, the annotation image 19 designates differentiated cell DC, undifferentiated cell UDC, dead cell DDC, and medium PL as class 1, class 2, class 3, and class 4, respectively. The designation of each class 1 to 4 is manually performed by the user, for example. Medium PL of class 4 is automatically designated by designating other classes 1-3. Note that the input image 22 given to the trained model 21T is also a cell image obtained by photographing a plurality of cells in culture with a phase-contrast microscope or the like, like the learning input image 17 .

As shown in FIG. 4, the temporary model 16 and the model 21 have a plurality of layers for analyzing the input image. It is a hierarchical model composed of a convolutional neural network such as Net. In the case of this example, there are five hierarchies: the first hierarchy, the second hierarchy, the third hierarchy, the fourth hierarchy, and the fifth hierarchy. In the following description, an example will be described in which the learning input image 17 is given to the model 21 as an input image, and the model 21 outputs the main learning output image 25 (see also FIG. 28).

The model 21 includes an encoder network (hereinafter abbreviated as EN) 26 , a decoder network (hereinafter abbreviated as DN) 27 and an output layer 28 . The EN 26 performs convolution processing using a filter for each layer to extract an image feature map CMP. DN27 expands step by step the image size of the minimum image feature map CMP output from EN26. Then, the stepwise enlarged image feature map CMP and the image feature map CMP output at each layer of EN 26 are combined to generate final output data 29 having the same image size as the learning input image 17 .

Input data having a plurality of pixel values arranged two-dimensionally is input to the EN 26 for each layer. The EN 26 extracts the image feature map CMP by performing convolution processing on the input data in each layer. The learning input image 17 is input as input data to the first layer of the EN 26 . In the first layer, convolution processing is performed on the learning input image 17 and, for example, an image feature map CMP having the same image size as that of the learning input image 17 is output. In the second layer and below, the image feature map CMP output in each upper layer is input as input data. In the second layer and below, convolution processing is performed on the image feature map CMP, and for example, an image feature map CMP having the same image size as the input image feature map CMP is output. The convolution process is shown as "conv (abbreviation for convolution)" in FIG.

In the convolution process, for example, a 3×3 filter is applied to the input data to obtain the pixel value e of the target pixel in the input data and the pixel values a, b, c, d, and d of eight pixels adjacent to the target pixel. By convolving f, g, h, and i, output data in which pixel values are arranged two-dimensionally is obtained in the same manner as the input data. When the coefficients of the filter are r, s, t, u, v, w, x, y, and z, the pixel value k of the pixel of the output data, which is the result of the convolution operation on the pixel of interest, is given by, for example, the following (formula It is obtained by calculating 1).
k=az+by+cx+dw+ev+fu+gt+hs+ir (equation 1)

In the convolution process, each pixel of the input data is subjected to the convolution operation as described above, and the pixel value k is output. In this way, output data having pixel values k arranged two-dimensionally is output. One output data is output for one filter. When multiple filters of different types are used, output data is output for each filter. The image feature map CMP is composed of such output data.

The output data is data in which pixel values k are arranged two-dimensionally, and has width and height. Further, when a plurality of different types of filters are applied and a plurality of output data are output, the image feature map CMP becomes a set of the plurality of output data. In the image feature map CMP, the number of filters is called the number of channels.

The numbers 64, 128, 256, 512, and 1024 shown above each image feature map CMP indicate the number of channels each image feature map CMP has. 1/1, 1/2, 1/4, 1/8, and 1/16 with parentheses attached to the first to fifth layers are images of the learning input image 17 which is the highest input image. Based on the size, the image size handled in each layer is shown.

In the first layer of the EN 26 of this example, the input image 17 for learning is subjected to convolution processing twice. First, the learning input image 17 is subjected to convolution processing applying 64 filters, and a 64-channel image feature map CMP is output. Then, this image feature map CMP is further subjected to convolution processing applying 64 filters, and in the first layer, a 64-channel image feature map CMP is finally output.

In EN 26 , the image size, which is the width and height of the image feature map CMP output by the first layer, is the same as the image size of the learning input image 17 . Therefore, the image size handled by the first layer is the same as that of the learning input image 17, that is, 1/1 representing the same size.

In the first layer of the EN 26, a pooling process indicated as "pool (abbreviation of pooling)" in FIG. 4 is performed on the image feature map CMP extracted by the two convolution processes. The pooling process is a process of calculating local statistics of the image feature map CMP and compressing the image feature map CMP. As the local statistic, for example, the maximum value or average value of pixel values within a block of 2×2 pixels is used. The pooling process for calculating the maximum value is called maximum value pooling, and the pooling process for calculating the average value is called average value pooling. That is, the pooling process can be said to be a process of reducing the image size of the image feature map CMP by reducing the resolution of the image feature map CMP by selecting a local representative value from the pixel values of each pixel of the image feature map CMP. For example, if the pooling process of selecting a representative value from a block of 2×2 pixels is shifted by one pixel, the image feature map CMP is reduced to half the original image size. In the model 21, in the first layer, for example, pooling processing is performed to halve the image size of the image feature map CMP. Therefore, in the second layer of EN 26, an image feature map CMP that is reduced to 1/2 the size of the input image for learning 17 is input as input data.

In the second layer, convolution processing applying 128 filters is performed twice to output a 128-channel image feature map CMP. Then, a pooling process for halving the image size is performed on the 128-channel image feature map CMP. As a result, the 128-channel image feature map CMP reduced to 1/4 the image size of the learning input image 17 is input to the second to third layers as input data. .

In the third layer, two convolution processes applying 256 filters are performed, and a 256-channel image feature map CMP is output. A pooling process is performed to As a result, the 256-channel image feature map CMP reduced to 1/8 the image size of the learning input image 17 is input to the third to fourth layers as input data.

Similarly, in the fourth layer, two convolution processes applying 512 filters are performed to output a 512-channel image feature map CMP. is further halved. As a result, the 512-channel image feature map CMP reduced to 1/16 of the image size of the learning input image 17 is input to the fourth to fifth layers as input data.

In the fifth layer, which is the lowest layer, two convolution processes applying 1024 filters are performed. However, in the fifth layer, the pooling process is not performed on the image feature map CMP extracted by the convolution process.

In EN26, the input data (learning input image 17 or image feature map CMP) input to each layer is reduced in image size step by step from the first layer, which is the highest layer, to the fifth layer, which is the lowest layer. and the resolution is lowered. In this example, based on the image size of the learning input image 17 input to the first layer, the first layer is 1/1 (same size), the second layer is 1/2, and the third layer is 1/2. 4, input data of 1/8 image size is input to the fourth layer, and input data of 1/16 image size is input to the fifth layer.

Each layer of the EN 26 performs convolution processing by applying a filter to input data with different resolutions for each layer. In the first layer, convolution processing is performed on the learning input image 17 having the highest resolution among the input data of each layer. Therefore, the image feature map CMP extracted in the first layer represents the features of the finest structure having the frequency band with the highest spatial frequency in the learning input image 17 . In the second layer and the third layer, convolution processing is performed on input data whose resolution is lower than that of the learning input image 17 . For this reason, the image feature maps CMP extracted in the second and third hierarchies represent mid-range structure features having frequency bands with lower spatial frequencies than in the first hierarchies. Since the resolution of the input data is further lowered in the fourth and fifth layers, the image feature map CMP extracted in the fourth and fifth layers has a frequency band with a lower spatial frequency, and features of the global structure. show.

The EN 26 outputs image features of different frequency bands included in the learning input image 17 for each layer from the first layer, which is the highest layer, to the fifth layer, which is the lowest layer. Each image size from 1/1 of the first layer to 1/16 of the fifth layer indicates a frequency band that can be analyzed by each layer. That is, 1/1 indicates the frequency band with the highest spatial frequency, and 1/16 indicates the frequency band with the lowest spatial frequency. The reason why the number of filters is increased to 64, 128, 256, . This is for extracting various features that can be

The first to fourth hierarchies of EN 26 transmit the image feature map CMP extracted by each to DN 27 . The process of transmitting this image feature map CMP from EN 26 to DN 27 is called skip layer process, and is indicated by "skip" in FIG. In each layer of DN27, the hatched image feature map CMP is the image feature map CMP transmitted from EN26.

The DN 27 repeats upsampling and merging. The upsampling process is shown as "upsmp" (abbreviation for upsampling) in FIG. The upsampling process is a process of stepwise increasing the image size of the minimum image size image feature map CMP output from the EN 26 . The merging process is a process of combining the image feature map CMP that has been stepwise enlarged by the upsampling process and the image feature map CMP that is output for each layer in the EN 26 and has the same image size. The DN 27 outputs final output data 29 through these upsampling processing and merging processing.

The DN 27 has 1st to 5th hierarchies corresponding to the respective hierarchies of the EN 26 . In the upsampling process performed in each layer of DN27, the image feature map CMP is enlarged so as to have the same size as the image size of each corresponding layer of EN26.

Also, the upsampling process of this example involves a convolution process that applies a filter in addition to enlarging the image size. Upsampling processing accompanied by such convolution processing is called upconvolution processing. In each layer of DN 27, merge processing and further convolution processing are performed after the up-convolution processing is completed.

The 4th hierarchy of DN27 first receives the image feature map CMP of the minimum image size of 1/16 from the lowest 5th hierarchy of EN26. This image feature map CMP has 1024 channels. The fourth layer of the DN 27 expands the image feature map CMP of 1/16 image size to 1/8 image size by doubling, and performs convolution processing to apply 512 filters to reduce the number of channels. is reduced by half to 512.

In the fourth layer of DN27, merge processing is performed to combine the image feature map CMP received from the fifth layer of EN26 and the image feature map CMP transmitted from the fourth layer of EN26 by skip layer processing. The image feature maps CMP combined in the fourth layer are each 1/8 image size and 512 channels. Therefore, in the fourth layer, an image feature map CMP of 1/8 image size and 1024 channels (512+512) is generated by merge processing.

Furthermore, in the fourth layer, the convolution process of applying 512 filters to the 1024-channel image feature map CMP is performed twice, resulting in a 1/8 image size and a 512-channel image feature map. CMP is generated. In the fourth layer, the image feature map CMP of 1/8 image size is doubled in size to 1/4 and the number of channels is halved to 256 channels by upconvolution processing. is done. As a result, an image feature map CMP of 1/4 image size and 256 channels is output for the fourth to third hierarchies.

In the third layer of DN27, merge processing is performed to combine the image feature map CMP received from the fourth layer and the image feature map CMP transmitted from the third layer of EN26 by skip layer processing. The image feature maps CMP combined in the third layer are each 1/4 image size and 256 channels. Therefore, in the third hierarchy, an image feature map CMP with a 1/4 image size and 512 channels (256+256) is generated by merge processing.

Furthermore, in the third layer, the convolution process of applying 256 filters to the 512-channel image feature map CMP is performed twice, resulting in a 1/4 image size and a 256-channel image feature map. CMP is generated. In the third layer, the image feature map CMP of the 1/4 image size is doubled in size to 1/2 and the number of channels is halved to 128 channels by upconvolution processing. is done. As a result, an image feature map CMP with half the image size and 128 channels is output from the third hierarchy to the second hierarchy.

In the second layer of DN27, merge processing is performed to combine the image feature map CMP received from the third layer and the image feature map CMP transmitted from the second layer of EN26 by skip layer processing. The image feature maps CMP combined in the second layer are each half the image size and 128 channels. Therefore, in the second layer, an image feature map CMP with half the image size and 256 channels (128+128) is generated by the merging process.

Furthermore, in the second layer, the convolution process of applying 128 filters to the 256-channel image feature map CMP is performed twice, resulting in a 1/2 image size and a 128-channel image feature map. CMP is generated. In the second layer, up-convolution processing is performed to double the image size to 1/1 and to halve the number of channels to 64 channels for the image feature map CMP of this 1/2 image size. is done. As a result, an image feature map CMP of 1/1 image size and 64 channels is finally output from the second layer to the first layer.

In the first layer of DN27, merge processing is performed to combine the image feature map CMP received from the second layer and the image feature map CMP transmitted from the first layer of EN26 by skip layer processing. The image feature maps CMP combined in the first hierarchy are each 1/1 image size and 64 channels. Therefore, in the first layer, an image feature map CMP of 1/1 image size and 128 channels (64+64) is generated by the merging process.

Furthermore, in the first layer, convolution processing is performed by applying 64 filters to the 128-channel image feature map CMP, and then convolution processing is performed by applying one filter. As a result, the final output data 29 having an image size 1/1 that of the learning input image 17 is generated.

In DN27, the image size of the minimum image size image feature map CMP output from EN26 is enlarged step by step. Then, while enlarging the image feature map CMP, the EN 26 combines the image feature maps CMP extracted for each layer to generate the final output data 29 . The image feature map CMP with the smallest image size represents the feature of the global structure with the lowest spatial frequency of the learning input image 17 . In DN27, by enlarging the image feature map CMP of the minimum image size, the features of the global structure are expanded, and by combining the image feature maps CMP from EN26, the features from the midrange structure to the fine structure are expanded. take in.

The output layer 28 generates, from the final output data 29 , a main learning output image 25 obtained by segmenting the regions for each class in the learning input image 17 .

In FIG. 5, the computer that constitutes the learning device 10 includes a storage device 30 , a memory 31 , a CPU (Central Processing Unit) 32 , a communication section 33 , a display 34 and an input device 35 . These are interconnected via bus lines 36 .

The storage device 30 is a hard disk drive built into the computer constituting the learning device 10 or connected via a cable or network. Alternatively, the storage device 30 is a disk array in which a plurality of hard disk drives are connected. The storage device 30 stores a control program such as an operating system, various application programs, various data associated with these programs, and the like. A solid state drive may be used instead of the hard disk drive.

The memory 31 is a work memory for the CPU 32 to execute processing. The CPU 32 loads a program stored in the storage device 30 into the memory 31 and executes processing according to the program, thereby comprehensively controlling each section of the computer.

The communication unit 33 is a network interface that controls transmission of various information via the network 12 . The display 34 displays various screens. The computer that constitutes the learning device 10 receives input of operation instructions from the input device 35 through various screens. The input device 35 is a keyboard, mouse, touch panel, or the like.

In FIG. 6, the storage device 30 stores an operating program 40 . The operating program 40 is an application program that causes the computer to function as the learning device 10 . That is, the operating program 40 is an example of the "learning device operating program" according to the technology of the present disclosure. The storage device 30 also stores an image group 41 , a model group 42 , and image generation conditions 43 .

When the operation program 40 is started, the CPU 32 of the computer constituting the learning device 10 cooperates with the memory 31 and the like to operate a read/write (hereinafter abbreviated as RW (Read Write)) control unit 45 and an image generation unit 46. , a temporary model generation unit 47 , a temporary learning unit 48 , a model generation unit 49 , a main learning unit 50 , a display control unit 51 , a reception unit 52 , and a transmission control unit 53 .

The RW control unit 45 controls storage of various data in the storage device 30 and reading of various data in the storage device 30 . The RW control unit 45 is an example of an “acquisition unit” according to the technology of the present disclosure.

The image generator 46 generates the temporary annotation image 18 from the annotation image 19 according to the image generation conditions 43 . The temporary model generation unit 47 generates the temporary model 16 . The temporary learning unit 48 learns the temporary model 16 using the temporary learning data 15, and performs the above-described temporary learning with the temporary model 16 as the learned temporary model 16T.

The model generator 49 generates the model 21 from the trained temporary model 16T. The main learning unit 50 causes the model 21 to learn using the learning data 20, and performs the above-described main learning with the model 21 as the trained model 21T.

The display control unit 51 controls the display of various screens on the display 34 . The various screens include an image generation condition designation screen 65 (see FIG. 11) for designating the image generation conditions 43, and the like. The accepting unit 52 accepts various operation instructions from the user through various screens. Various operation instructions include designation of the image generation condition 43 by the user through the image generation condition designation screen 65 . The transmission control unit 53 controls transmission of the trained model 21T to the operation device 11 .

As shown in FIG. 7, the image group 41 includes the learning input image 17, the temporary annotation image 18, and the annotation image 19 described above. Further, as shown in FIG. 8, the model group 42 includes the above-described provisional model 16, trained provisional model 16T, model 21, and trained model 21T, and a basic model 60 that is the basis of these models.

As shown in Table 63 of FIG. 9, in this example, provisional learning is divided into first provisional learning and second provisional learning. The first temporary learning is performed using the temporary annotation image 18_1 (see FIG. 20) with the number of classes N=2. The second provisional learning is performed using the provisional annotation image 18_2 (see FIG. 22) with the number of classes N=3 (=M−1), which is increased by one from the first provisional learning. Needless to say, this learning is performed using the annotation image 19 with the number of classes M=4, in which classes 1 to 4 are designated.

As shown in FIG. 10, in the image generation condition 43, which class among the classes 1 to 4 specified in the annotation image 19 is to be integrated is registered for each time of provisional learning. In FIG. 10, in the first provisional learning, classes 1 to 3 (differentiated cell DC, undifferentiated cell UDC, dead cell DDC) are integrated, and in the second provisional learning, classes 1 and 2 (differentiated cell DC , undifferentiated cells (UDC) are registered respectively.

In FIG. 11, an image generation condition designation screen 65 is displayed on the display 34 by the display control unit 51 when learning is started. The image generation condition designation screen 65 has a first designation area 66 and a second designation area 67 . In the first designation area 66, check boxes 68 are arranged for designating the classes to be integrated in the first provisional learning among the classes 1 to 4. FIG. The checkboxes 68 are designed so that only up to three classes can be checked. In the second designation area 67, a check box 69 for designating a class to be integrated in the second provisional learning among the classes 1 to 4 is arranged. Check boxes 69 are designed so that only two classes out of the three classes checked in check boxes 68 can be checked.

The user appropriately selects the check boxes 68 and 69 of the designated areas 66 and 67 and selects the designated button 70 . The receiving unit 52 receives the image generation condition 43 according to the selection state of the check boxes 68 and 69 . That is, the reception unit 52 is an example of the "second reception unit" according to the technology of the present disclosure. FIG. 11 illustrates a case where classes 1 to 3 are selected in the first provisional learning, and classes 1 and 2 are selected in the second provisional learning. Note that when the cancel button 71 is selected, the display of the image generation condition designation screen 65 is erased.

The reception unit 52 outputs the image generation conditions 43 to the RW control unit 45 . The RW control unit 45 stores the image generation conditions 43 in the storage device 30 .

12 to 19 shown below, FIGS. 12 to 14 are for the first provisional learning, FIGS. 15 to 17 are for the second provisional learning, and FIGS. 18 and 19 are for the main learning.

As shown in FIG. 12 , the image generation unit 46 receives from the RW control unit 45 the annotation image 19 and the image generation conditions 43 read from the storage device 30 by the RW control unit 45 . The image generator 46 generates the temporary annotation image 18_1 used for the first temporary learning from the annotation image 19 according to the image generation condition 43 . The image generator 46 outputs the temporary annotation image 18_1 to the RW controller 45 . The RW control unit 45 stores the temporary annotation image 18_1 in the storage device 30 .

As shown in FIG. 13 , the temporary model generation unit 47 receives from the RW control unit 45 the basic model 60 read from the storage device 30 by the RW control unit 45 . The temporary model generation unit 47 generates the temporary model 16_1 used for the first temporary learning from the basic model 60 . The temporary model generator 47 outputs the temporary model 16_1 to the RW controller 45 . The RW control unit 45 stores the temporary model 16_1 in the storage device 30 .

As shown in FIG. 14 , the temporary learning unit 48 receives the temporary learning data 15_1 and the temporary model 16_1 read from the storage device 30 by the RW control unit 45 from the RW control unit 45 . The provisional learning data 15_1 is a set of the learning input image 17 and the provisional annotation image 18_1. The provisional learning unit 48 learns the provisional model 16_1 using the provisional learning data 15_1, and sets the provisional model 16_1 as a trained provisional model 16T_1. The temporary learning unit 48 outputs the trained temporary model 16T_1 to the RW control unit 45 . The RW control unit 45 stores the trained temporary model 16T_1 in the storage device 30 .

As shown in FIG. 15, the image generator 46 receives from the RW controller 45 the annotation image 19 and the image generation conditions 43 read from the storage device 30 by the RW controller 45, as in the case of FIG. The image generator 46 generates the temporary annotation image 18_2 used for the second temporary learning from the annotation image 19 according to the image generation condition 43 . The image generator 46 outputs the temporary annotation image 18_2 to the RW controller 45 . The RW control unit 45 stores the temporary annotation image 18_2 in the storage device 30. FIG.

As shown in FIG. 16 , the temporary model generation unit 47 receives from the RW control unit 45 the trained temporary model 16T_1 read from the storage device 30 by the RW control unit 45 . The temporary model generation unit 47 generates a temporary model 16_2 to be used for the second temporary learning from the trained temporary model 16T_1. The temporary model generator 47 outputs the temporary model 16_2 to the RW controller 45 . The RW control unit 45 stores the temporary model 16_2 in the storage device 30 .

As shown in FIG. 17 , the temporary learning unit 48 receives from the RW control unit 45 the temporary learning data 15_2 and the temporary model 16_2 read from the storage device 30 by the RW control unit 45 . The provisional learning data 15_2 is a set of the learning input image 17 and the provisional annotation image 18_2. The temporary learning unit 48 learns the temporary model 16_2 using the temporary learning data 15_2, and sets the temporary model 16_2 as a trained temporary model 16T_2. The temporary learning unit 48 outputs the trained temporary model 16T_2 to the RW control unit 45 . The RW control unit 45 stores the trained temporary model 16T_2 in the storage device 30. FIG.

In this way, the image generation unit 46 and the temporary learning unit 48 gradually increase the class number N of the temporary annotation image 18 by performing the process of generating the temporary annotation image 18 and the process of making the temporary model 16 into the trained temporary model 16T. and until the number of classes N of the temporary annotation image 18 reaches M-1.

As shown in FIG. 18 , the model generation unit 49 receives from the RW control unit 45 the trained temporary model 16T_2 read from the storage device 30 by the RW control unit 45 . The model generator 49 generates the model 21 from the trained temporary model 16T_2. The model generator 49 outputs the model 21 to the RW controller 45 . The RW control unit 45 stores the model 21 in the storage device 30 .

As shown in FIG. 19 , the main learning unit 50 receives the learning data 20 and the model 21 read from the storage device 30 by the RW control unit 45 from the RW control unit 45 . The main learning unit 50 causes the model 21 to learn using the learning data 20, and sets the model 21 as a trained model 21T. The main learning unit 50 outputs the learned model 21T to the RW control unit 45 . The RW control unit 45 stores the learned model 21T in the storage device 30. FIG.

The provisional learning unit 48 and the main learning unit 50 cause the provisional model 16 and the model 21 to perform mini-batch learning using, for example, mini-batch data. The mini-batch data is divided into a plurality of divided images obtained by dividing the input image for learning 17 and the temporary annotation image 18, or the input image for learning 17 and the annotation image 19 (for example, divided by a frame of 1/100 size of the original image). 10,000 divided images) (for example, 100 images). The provisional learning unit 48 and the main learning unit 50 create a plurality of sets (for example, 100 sets) of such mini-batch data, and sequentially apply each set to the provisional model 16 and the model 21 for learning.

In FIG. 20, the provisional annotation image 18_1 of this example is an image in which Classes 1 to 3 of differentiated cell DC, undifferentiated cell UDC, and dead cell DDC are integrated as an integrated class 1 . That is, the temporary annotation image 18_1 is an image in which three classes 1 to 3 out of the four classes 1 to 4 designated in the annotation image 19 are integrated, and the number of classes is smaller than that of the annotation image 19 .

In FIG. 21, the basic model 60 is composed of EN 26, DN 27, and output layer 28, like the temporary model 16 and model 21. FIG.

The output layer 28 has a presence/absence probability map generation layer 80 and an activation layer 81 . A presence/absence probability map generation layer 80 generates a presence/absence probability map PMP from the final output data 29 output by the DN 27 . A numerical value indicating the presence/absence probability of a class in the input image is registered for each pixel in the presence/absence probability map PMP. The pixels of the presence/absence probability map PMP have a one-to-one correspondence with the pixels of the output image. The presence/absence probability map generation layer 80 generates presence/absence probability maps PMP1 and PMP2 for two classes.

The activation layer 81 outputs authorization data AVD based on the presence/absence probability maps PMP1 and PMP2. The activation layer 81 selects, for example, the maximum value (highest probability) among the pixel values of the pixels of the presence/absence probability maps PMP1 and PMP2 corresponding to the specific pixel of the output image. certified as a class. The certified data AVD output in this manner is data that identifies the class to which each pixel of the output image belongs.

Thus, the output layer 28 of the base model 60 is for two classes. In the first temporary learning, as shown in FIG. 9, the number of classes of the temporary annotation image 18_1 is N=2. That is, the basic model 60 has a configuration corresponding to the number of classes of the temporary annotation image 18_1. Therefore, the temporary model generator 47 outputs the basic model 60 itself as the temporary model 16_1.

Note that the output layer 28 of the base model 60 is not limited to two classes. The output layer 28 may be the base model 60 for the three classes. When the number of classes N of the temporary annotation image 18_1 is 2 and the output layer 28 of the basic model 60 is for three classes, the temporary model generation unit 47 converts the output layer 28 of the basic model 60 into two classes. Replace with output layer 28 .

In FIG. 22 , the provisional annotation image 18_2 of this example is an image in which class 1 and 2 differentiated cell DCs and undifferentiated cell UDCs are integrated as an integrated class 2 . That is, the temporary annotation image 18_2 is an image in which two classes 1 and 2 out of the four classes 1 to 4 designated in the annotation image 19 are integrated, and the number of classes is smaller than that of the annotation image 19 .

In FIG. 23, when the temporary model generation unit 47 generates a temporary model 16_2 used in the second temporary learning from the trained temporary model 16T_1 of the first temporary learning, the temporary model generation unit 47 sets EN26 and DN27 of the trained temporary model 16T_1 to Carry over to the temporary model 16_2. That is, the temporary model generating unit 47 generates the temporary model 16_2 used this time by using at least part of the trained temporary model 16T_1 used last time.

On the other hand, the temporary model generation unit 47 replaces the output layer 28 of the trained temporary model 16T_1 with the output layer 28_2 without carrying over it. The output layer 28_2 has a presence/absence probability map generation layer 86 and an activation layer 87 . The presence/absence probability map generation layer 86 generates presence/absence probability maps PMP1, PMP2, and PMP3 for three classes. Activation layer 87 outputs recognition data AVD from presence/absence probability maps PMP1-PMP3 in the same manner as activation layer 81. FIG. That is, the output layer 28_2 is for three classes.

In the second temporary learning, as shown in FIG. 9, the number of classes of the temporary annotation image 18_2 is N=3. However, the output layer 28 of the trained temporary model 16T_1 is for two classes. Therefore, the temporary model generation unit 47 replaces the output layer 28 for two classes with the output layer 28_2 for three classes. By doing so, the temporary model generation unit 47 generates a temporary model 16_2 having a configuration corresponding to the number of classes of the temporary annotation image 18_2.

As shown in FIG. 24 , the provisional learning section 48 has a first processing section 90 , a first evaluation section 91 and a first updating section 92 . The first processing unit 90 gives the learning input image 17 to the temporary model 16 and causes the temporary model 16 to output a temporary learning output image 93 . The first processing unit 90 outputs the temporary learning output image 93 to the first evaluation unit 91 .

The first evaluation unit 91 compares the temporary annotation image 18 and the temporary learning output image 93 to evaluate the class discrimination accuracy of the temporary model 16 . The first evaluation unit 91 uses the first loss function 94 to evaluate the class discrimination accuracy of the temporary model 16 . The first loss function 94 is a function representing the degree of difference in class designation between the temporary annotation image 18 and the temporary learning output image 93 . The closer the calculated value of the first loss function 94 to 0, the higher the class discrimination accuracy of the temporary model 16 . The first evaluation unit 91 outputs an evaluation result of the class discrimination accuracy of the temporary model 16 by the first loss function 94 to the first updating unit 92 .

The first updating section 92 updates the temporary model 16 according to the evaluation result from the first evaluating section 91 . For example, the first updating unit 92 changes the coefficient values of the filters EN26 and DN27 of the temporary model 16 by stochastic gradient descent with learning coefficients or the like. The learning coefficient indicates the width of change in the value of the coefficient of the filter. That is, the larger the learning coefficient, the greater the change in the value of the filter coefficient, and the greater the degree of update of the temporary model 16 .

The temporary learning unit 48 inputs the learning input image 17 to the temporary model 16 by the first processing unit 90 , outputs the temporary learning output image 93 to the first evaluation unit 91 , outputs the temporary model by the first evaluation unit 91 . The evaluation of the discrimination accuracy of the 16 classes and the updating of the temporary model 16 by the first updating unit 92 are repeated until the class discrimination accuracy of the temporary model 16 reaches a preset level. Then, the temporary learning unit 48 outputs the temporary model 16 whose class discrimination accuracy has reached a preset level as a trained temporary model 16T.

As shown in FIG. 25, the first loss function 94_1 used for the first provisional learning is a loss function for integrated class 1 (differentiated cell DC, undifferentiated cell UDC, dead cell DDC) that integrates classes 1 to 3. WA multiplied by a factor WA and the loss function for class 4 (medium PL) multiplied by a weighting factor WB. In this case, the same value such as 0.5 is set to the weighting factors WA and WB.

On the other hand, as shown in FIG. 26, the first loss function 94_2 used for the second provisional learning is the weighting coefficient WC , the loss function for class 3 (dead cell DDC) multiplied by a weighting factor WD, and the loss function for class 4 (medium PL) multiplied by a weighting factor WB. In this case, the weighting factor WB is set to a smaller value than the weighting factors WC and WD. For example, the weighting factors WC and WD are set to 0.4, and the weighting factor WB is set to 0.2.

In this manner, the temporary learning unit 48 changes the first loss function 94 used to evaluate the class discrimination accuracy of the temporary model 16 . In addition, the provisional learning unit 48 makes the weight of the loss function for the class common to the previous time smaller than the weight of the loss function for the class newly appearing this time. In the example of FIG. 26, class 4 corresponds to "the class common to the previous time", and the weighting factor WB corresponds to "the weight of the loss function for the class common to the previous time". Further, integrated class 2 and class 3 correspond to "the class newly appearing this time", and the weighting coefficients WC and WD correspond to "the weight of the loss function for the class newly appearing this time".

In FIG. 27, the model generation unit 49 carries over the EN26 and DN27 of the trained temporary model 16T_2 to the model 21 when generating the model 21 used for the main learning from the trained temporary model 16T_2 of the second temporary learning. That is, the model generator 49 generates the model 21 using at least part of the trained temporary model 16T_2.

On the other hand, the model generating unit 49 replaces the output layer 28_2 of the trained temporary model 16T_2 with the output layer 28_3 without carrying over it. The output layer 28_3 has a presence/absence probability map generation layer 101 and an activation layer 102 . The presence/absence probability map generation layer 101 generates presence/absence probability maps PMP1, PMP2, PMP3, and PMP4 for four classes. The activation layer 102 outputs authorization data AVD from the presence/absence probability maps PMP1 to PMP4 in the same manner as the activation layers 81 and 87. FIG. That is, the output layer 28_3 is for four classes.

In this learning, as shown in FIG. 9, the number of classes of the annotation image 19 is M=4. However, the output layer 28_2 of the trained temporary model 16T_2 is for three classes. Therefore, the model generator 49 replaces the output layer 28_2 for three classes with an output layer 28_3 for four classes. By doing so, the model generation unit 49 generates the model 21 having a configuration corresponding to the number of classes of the annotation image 19 .

As shown in FIG. 28 , the main learning unit 50 has a second processing unit 105 , a second evaluation unit 106 and a second updating unit 107 . The second processing unit 105 provides the learning input image 17 to the model 21 and causes the model 21 to output the main learning output image 25, like the first processing unit 90 of the provisional learning unit 48 does. The second processing unit 105 outputs the main learning output image 25 to the second evaluation unit 106 .

The second evaluation unit 106 compares the annotation image 19 with the main learning output image 25 and evaluates the class discrimination accuracy of the model 21 in the same manner as the first evaluation unit 91 of the provisional learning unit 48 . The second evaluation unit 106 evaluates the class discrimination accuracy of the model 21 using the second loss function 109 . The second loss function 109 is a function representing the degree of difference in class designation between the annotation image 19 and the main learning output image 25 . The closer the calculated value of the second loss function 109 to 0, the higher the class discrimination accuracy of the model 21 . The second evaluation unit 106 outputs the evaluation result of the class discrimination accuracy of the model 21 by the second loss function 109 to the second updating unit 107 .

The second updating section 107 updates the model 21 according to the evaluation result from the second evaluating section 106, like the first updating section 92 of the provisional learning section 48. FIG.

Similar to the temporary learning unit 48, the main learning unit 50 inputs the learning input image 17 to the model 21 by the second processing unit 105, outputs the main learning output image 25 to the second evaluation unit 106, The evaluation of the class discrimination accuracy of the model 21 by the second evaluation unit 106 and the update of the model 21 by the second update unit 107 are repeated until the class discrimination accuracy of the model 21 reaches a preset level. Then, the main learning unit 50 outputs the model 21 whose class discrimination accuracy reaches a preset level as a trained model 21T.

As shown in FIG. 29, the second loss function 109 is obtained by multiplying the loss function for class 1 (differentiated cells DC) by a weighting factor WE and the loss function for class 2 (undifferentiated cells UDC) by multiplying the weighting factor WF. , the loss function for class 3 (dead cell DDC) multiplied by a weighting factor WD, and the loss function for class 4 (medium PL) multiplied by a weighting factor WB. In this case, the weighting factors WB and WD are set to values smaller than the weighting factors WE and WF. For example, the weighting coefficients WE and WF are set to 0.4, and the weighting coefficients WB and WD are set to 0.1.

In this manner, the main learning unit 50 changes the second loss function 109 used for evaluating the class discrimination accuracy of the model 21 from the first loss function 94 used for evaluating the class discrimination accuracy of the temporary model 16 . Further, the main learning unit 50 makes the weight of the loss function for the class common to the provisional learning smaller than the weight of the loss function for the class newly appearing in the main learning. In the example of FIG. 29, classes 3 and 4 correspond to "classes shared with temporary learning", and weighting coefficients WB and WD correspond to "loss function weights for classes shared with temporary learning". Classes 1 and 2 correspond to "classes newly appearing in the main learning", and weighting coefficients WE and WF correspond to "loss function weights for classes newly appearing in the main learning".

FIG. 30 shows transition of each learning class of the first temporary learning, the second temporary learning, and the main learning using the temporary annotation images 18_1, 18_2, and the annotation image 19. FIG. FIG. 30A shows a provisional annotation image 18_1 with the number of classes N=2 (integrated class 1 and class 4) used for the first provisional learning. FIG. 30B shows a provisional annotation image 18_2 with the number of classes N=3 (integrated class 2, class 3, class 4) used for the second provisional learning. FIG. 30C shows an annotation image 19 with the number of classes M=4 (classes 1 to 4) used for this learning. While the number of classes is increased by one in this way, each of the first provisional learning, the second provisional learning, and the main learning is advanced.

Next, the operation of the above configuration will be described with reference to the flow chart of FIG. First, when the operation program 40 is activated in the learning device 10, as shown in FIG. , model generation unit 49 , main learning unit 50 , display control unit 51 , reception unit 52 , and transmission control unit 53 .

First, the image generation condition designation screen 65 shown in FIG. 11 is displayed on the display 34 by the display control unit 51 (step ST100). When the user selects the check boxes 68 and 69 of the designated areas 66 and 67 and selects the designation button 70, the reception unit 52 receives the image generation condition 43 corresponding to the selection state of the check boxes 68 and 69. (Step ST110). The image generation conditions 43 are output from the reception unit 52 to the RW control unit 45 and stored in the storage device 30 by the RW control unit 45 .

As shown in FIG. 12, the annotation image 19 and the image generation conditions 43 are read from the storage device 30 by the RW control unit 45 (step ST120). The annotation image 19 and the image generation conditions 43 are output from the RW control section 45 to the image generation section 46 . Note that step ST120 is an example of the “acquisition step” according to the technology of the present disclosure.

As shown in FIG. 20, the image generation unit 46 generates a provisional annotation image 18_1 used for the first provisional learning from the annotation image 19 according to the image generation condition 43 (step ST130). The temporary annotation image 18_1 is output from the image generator 46 to the RW controller 45 and stored in the storage device 30 by the RW controller 45 . Note that step ST130 is an example of the "image generation step" according to the technology of the present disclosure.

Next, as shown in FIG. 13, the basic model 60 is read from the storage device 30 by the RW control unit 45 . The basic model 60 is output from the RW control unit 45 to the temporary model generation unit 47 .

As shown in FIG. 21, the temporary model generation unit 47 generates the temporary model 16_1 used for the first temporary learning from the basic model 60 (step ST140). In this example, the temporary model 16_1 is a model that uses the basic model 60 as it is, and has an output layer 28 for two classes. The temporary model 16_1 is output from the temporary model generation unit 47 to the RW control unit 45 and stored in the storage device 30 by the RW control unit 45 .

As shown in FIG. 14, the RW control unit 45 reads the temporary learning data 15_1 and the temporary model 16_1, which are a set of the learning input image 17 and the temporary annotation image 18_1, from the storage device 30 (step ST150). The provisional learning data 15_1 and the provisional model 16_1 are output from the RW control section 45 to the provisional learning section 48 . Note that step ST150, like step ST120, is an example of the "acquisition step" according to the technology of the present disclosure.

The temporary learning unit 48 performs temporary learning of the temporary model 16_1 using the temporary learning data 15_1 (step ST160). More specifically, as shown in FIG. 24, in the first processing unit 90, the learning input image 17 is given to the temporary model 16_1, and the temporary learning output image 93 is output from the temporary model 16_1. Next, in the first evaluation unit 91, the temporary annotation image 18_1 and the temporary learning output image 93 are compared, and the class discrimination accuracy of the temporary model 16_1 is evaluated using the first loss function 94_1 shown in FIG. be. Then, the provisional model 16_1 is updated by the first updating unit 92 . The input of the learning input image 17 to the provisional model 16_1 by the first processing unit 90, the output of the provisional learning output image 93 to the first evaluation unit 91, and the class discrimination accuracy of the provisional model 16_1 by the first evaluation unit 91 and the updating of the temporary model 16_1 by the first update unit 92 are repeated until the class discrimination accuracy of the temporary model 16_1 reaches a preset level. The provisional model 16_1 whose class discrimination accuracy has reached a preset level is output from the provisional learning unit 48 to the RW control unit 45 as a trained provisional model 16T_1, and is stored in the storage device 30 by the RW control unit 45 . Note that step ST160 is an example of a "provisional learning step" according to the technology of the present disclosure.

Subsequently, as shown in FIGS. 15 and 22, the image generator 46 generates a provisional annotation image 18_2 to be used for the second provisional learning from the annotation image 19 (NO in step ST170, step ST130). Next, as shown in FIGS. 16 and 23, the temporary model generator 47 generates a temporary model 16_2 to be used for the second temporary learning from the trained temporary model 16T_1 for the first temporary learning (step ST140). . The temporary model 16_2 is a model in which EN26 and DN27 of the trained temporary model 16T_1 are carried over, and has an output layer 28_2 for three classes. Then, as shown in FIGS. 17 and 24, the provisional learning unit 48 performs provisional learning of the provisional model 16_2 using the provisional learning data 15_2 that is a set of the learning input image 17 and the provisional annotation image 18_2. (Step ST160). The first loss function 94_2 shown in FIG. 26 is used to evaluate the class discrimination accuracy of the temporary model 16_2. The provisional model 16_2 whose class discrimination accuracy has reached a preset level through the second provisional learning is output as a trained provisional model 16T_2 from the provisional learning unit 48 to the RW control unit 45, and is stored by the RW control unit 45. stored in the device 30;

After completing the second provisional learning (YES in step ST170), the RW control unit 45 reads the trained provisional model 16T_2 of the second provisional learning from the storage device 30 as shown in FIG. The trained temporary model 16T_2 is output from the RW control unit 45 to the model generation unit 49. FIG.

As shown in FIG. 27, the model generator 49 generates the model 21 used for the main learning from the trained provisional model 16T_2 (step ST180). The model 21 is a model in which EN26 and DN27 of the trained temporary model 16T_2 are carried over, and has an output layer 28_3 for four classes. The model 21 is output from the model generator 49 to the RW controller 45 and stored in the storage device 30 by the RW controller 45 . Note that step ST180 is an example of the "machine learning model generation step" according to the technology of the present disclosure.

As shown in FIG. 19, the RW control unit 45 reads the learning data 20, which is a set of the learning input image 17 and the annotation image 19, and the model 21 from the storage device 30 (step ST190). The learning data 20 and the model 21 are output from the RW control section 45 to the main learning section 50 . Note that step ST190, like steps ST120 and ST150, is an example of the "acquisition step" according to the technology of the present disclosure.

In the main learning unit 50, the main learning of the model 21 is performed using the learning data 20 (step ST200). More specifically, as shown in FIG. 28, in the second processing unit 105, the learning input image 17 is given to the model 21, and the model 21 outputs the main learning output image 25. FIG. Next, the annotation image 19 is compared with the main learning output image 25 in the second evaluation unit 106, and the class discrimination accuracy of the model 21 is evaluated using the second loss function 109 shown in FIG. Then, the model 21 is updated by the second updating unit 107 . The input of the learning input image 17 to the model 21 by the second processing unit 105, the output of the main learning output image 25 to the second evaluation unit 106, and the evaluation of the class discrimination accuracy of the model 21 by the second evaluation unit 106 , and the update of the model 21 by the second update unit 107 are repeated until the class discrimination accuracy of the model 21 reaches a preset level. The model 21 whose class discrimination accuracy has reached a preset level is output from the main learning unit 50 to the RW control unit 45 as a trained model 21T, and is stored in the storage device 30 by the RW control unit 45 . Note that step ST200 is an example of the "main learning step" according to the technology of the present disclosure.

The learned model 21T is read from the storage device 30 by the RW control unit 45 and is output from the RW control unit 45 to the transmission control unit 53 . The trained model 21T is transmitted to the operation device 11 by the transmission control unit 53 (step ST210).

In the operation device 11, as shown in FIG. 2, the input image 22 is given to the trained model 21, and the learned model 21 outputs an output image 23 obtained by discriminating the class of the object appearing in the input image 22 and its outline. be done. The output image 23 is displayed on the display of the operation device 11 and is provided for viewing by the user. In addition, the output image 23 is used for cell culture evaluation such as counting the number of differentiated cell DCs.

As described above, the learning device 10 includes the RW control unit 45 as an acquisition unit, the image generation unit 46, the provisional learning unit 48, the model generation unit 49, and the main learning unit 50. The RW control unit 45 generates learning data 20, which is a set of an input image for learning 17 and an annotation image 19 in which three or more classes to be subjected to semantic segmentation are specified for the input image for learning 17, It is obtained by reading from the storage device 30 . The image generation unit 46 integrates at least two classes among the three or more classes designated in the annotation image 19 to generate the temporary annotation image 18 having fewer classes than the annotation image 19 . The temporary learning unit 48 uses the temporary learning data 15, which is a set of the learning input image 17 and the temporary annotation image 18, to learn the temporary model 16 having a configuration corresponding to the number of classes of the temporary annotation image 18, and learns the temporary model 16. Let the model 16 be a trained provisional model 16T.

The model generation unit 49 uses at least part of the trained temporary model 16T to generate the model 21 having a configuration corresponding to the number of classes of the annotation image 19 . The main learning unit 50 causes the model 21 to learn using the learning data 20, and sets the model 21 as a trained model 21T. In this way, the learning device 10 trains the model 21 with a smaller number of classes than the classes specified in the annotation image 19 . Therefore, it is possible to obtain a trained model 21T with high class discrimination accuracy compared to the case where all of a plurality of classes are learned at once.

The learning device 10 further includes a temporary model generator 47 that generates the temporary model 16 . Therefore, the temporary model 16 can be generated without bothering the user.

Assuming that the number of classes of the annotation image 19 is M and the number of classes of the temporary annotation image 18 is N, the image generation unit 46 and the temporary learning unit 48 perform processing for generating the temporary annotation image 18 and the temporary model 16 that has been trained. The processing for 16T is repeated multiple times while gradually increasing the class number N of the temporary annotation image 18 until the class number N of the temporary annotation image 18 reaches M−1. Therefore, provisional learning can be divided into a plurality of stages and performed finely, and a model 21 with higher class discrimination accuracy can be obtained.

The temporary model generating unit 47 generates the temporary model 16 to be used this time by using at least part of the temporary model 16 used last time. Therefore, the results of the previous provisional learning can be incorporated into the current provisional learning and, by extension, the main learning.

The provisional learning unit 48 changes the first loss function 94 used to evaluate the class discrimination accuracy of the provisional model 16 . Specifically, the provisional learning unit 48 makes the weight of the loss function for the class common to the previous time smaller than the weight of the loss function for the class newly appearing this time. Classes common to the last time have already been tentatively learned last time. On the other hand, the class that newly appeared this time will be provisionally learned for the first time this time. For this reason, if the weight of the loss function for the class common to the previous time is made smaller than the weight of the loss function for the class newly appearing this time, the class newly appearing this time will be given more weight than the class common to the previous time. can be provisionally learned.

The image generator 46 generates the temporary annotation image 18 according to the image generation conditions 43 specified in advance. Therefore, the temporary annotation image 18 can be easily generated.

The learning device 10 also includes a reception unit 52 as a second reception unit that receives designation of the image generation condition 43 by the user. Therefore, it is possible to generate the provisional annotation image 18 reflecting the user's thoughts, and provisional learning can be performed as specified by the user.

Note that the type of the first loss function 94 may be changed. For example, the first loss function 94 using the Dice coefficient and the first loss function 94 using the squared error are selectively used. Similarly, the types of first loss function 94 and second loss function 109 may be changed.

Here, the field of cell culture has recently been in the limelight due to the emergence of iPS (Induced Pluripotent Stem) cells and the like. Therefore, there is a demand for a technique for more accurately discriminating the class of cells in a cell image. In the technology of the present disclosure, the input image 22 is a cell image obtained by photographing a plurality of cells in culture. Therefore, it can be said that the technique of the present disclosure is a technique that can meet recent demands.

[Second embodiment]
In the second embodiment shown in FIGS. 32 to 34, in the provisional learning process by the provisional learning section 48 and the main learning process by the main learning section 50, the previously specified portion is not updated.

FIG. 32 shows a non-updated portion designation screen 120 displayed on the display 34 by the display control unit 51. As shown in FIG. The non-updated portion designation screen 120 is a screen for accepting designation by the user of portions of the temporary model 16 and the model 21 that are not to be updated (hereinafter referred to as non-updated portions). The non-updated portion designation screen 120 has a first designation area 121 and a second designation area 122 . A check box 123 for designating each layer or all layers of the EN 26 as non-updated portions is arranged in the first designation area 121 . A check box 124 for designating each layer or all layers of the DN 27 as non-updated portions is arranged in the second designation area 122 .

The user appropriately selects the check boxes 123 and 124 of the designated areas 121 and 122 and selects the designated button 125 . The accepting unit 52 accepts a non-updated portion designation condition 127 according to the selection state of the check boxes 123 and 124 . That is, the reception unit 52 is an example of the "first reception unit" according to the technology of the present disclosure. FIG. 32 illustrates a case where all layers of EN26 are selected as non-updated portions. Note that when the cancel button 126 is selected, the display of the non-updated portion designation screen 120 is erased.

The receiving unit 52 outputs the non-updated portion designation condition 127 to the RW control unit 45 . The RW control unit 45 stores the non-updated portion designation condition 127 in the storage device 30 .

As shown in FIG. 33 , the non-updated portion designation condition 127 is read from the storage device 30 by the RW control unit 45 and output from the RW control unit 45 to the first updating unit 92 . The first update unit 92 does not update the non-updated portion of the temporary model 16 (here, the temporary model 16_1 used for the first temporary learning) according to the non-updated portion specifying condition 127 .

In addition, as shown in FIG. 34, the non-updated portion designation condition 127 is also output to the second updating unit 107. FIG. The second update unit 107 does not update the non-updated portions of the model 21 according to the non-updated portion designation condition 127 .

33 and 34 exemplify the case where all layers of EN26 are designated as non-updated portions, as in the case of FIG. In this case, the first updating unit 92 and the second updating unit 107 update the DN 27 of the temporary model 16 and the model 21, but do not update the EN 26. FIG.

Thus, in the second embodiment, the accepting unit 52 accepts user designation of non-updated portions. The provisional learning unit 48 and the main learning unit 50 do not update the non-updated part in the process of converting the temporary model 16 into the trained temporary model 16T and the process of changing the model 21 into the trained model 21T. In this way, when performing so-called transfer learning, in which part of a model learned by another learning device is diverted to the original basic model 60, learning by another learning device diverted to the basic model 60 is performed. The updated part can be specified as a non-updated part so as not to be updated. Therefore, the results of transfer learning can be effectively incorporated.

[Third embodiment]
In the third embodiment shown in FIG. 35, the image generation condition 43 is such that the area of each class of the temporary annotation image 18 is not biased.

In FIG. 35, area information 130 is stored in the storage device 30 in the third embodiment. In the area information 130, the area ratio in the annotation image 19 is registered for each class. The area ratio is a value obtained by counting the number of pixels belonging to each class based on the recognition data AVD and dividing the counted number of pixels by the number of all pixels of the annotation image 19 . In FIG. 35, the area ratio of class 1 differentiated cell DC is 46%, the area ratio of class 2 undifferentiated cell UDC is 11%, the area ratio of class 3 dead cell DDC is 6%, and the area ratio of class 4 medium PL A case where the area ratio is 37% is illustrated.

In this case, the image generation condition 43 is that classes 2 to 4 (undifferentiated cell UDC, dead cell DDC, medium PL) are integrated in the first provisional learning, and classes 2 and 3 in the second provisional learning. (Undifferentiated cell UDC, dead cell DDC) are respectively registered. In the first provisional learning, the area ratio of differentiated cells DC in class 1 was 46%, and the area ratio of integrated classes of classes 2 to 4 was 11 + 6 + 37 = 54%, compared to the case where the other three classes were integrated. The area ratio is not biased. In the second provisional learning, the area ratio of differentiated cells DC of class 1 is 46%, the area ratio of medium PL of class 4 is 37%, and the area ratio of integrated classes of classes 2 and 3 is 11 + 6 = 17%. , and the area ratio is not biased compared to when classes 3 and 4 or classes 2 and 4 are integrated.

Thus, in the third embodiment, the image generation condition 43 is such that the area of each class of the temporary annotation image 18 is not biased. Therefore, the temporary learning load of each class of the temporary annotation image 18 can be averaged.

[Fourth Embodiment]
In the fourth embodiment shown in FIG. 36, the image generation condition 43 is set so that the complexity of each class of the temporary annotation image 18 is balanced.

In FIG. 36, the complexity information 135 is stored in the storage device 30 in the fourth embodiment. The complexity information 135 registers the complexity of the annotation image 19 for each class. The complexity is a level value set according to the area of each class and/or the pitch of adjacent zigzag peaks of the boundary line of each class. The higher the complexity level value, the more complex the class. In FIG. 36, the complexity of class 1 differentiated cell DC is level 5, the complexity of class 2 undifferentiated cell UDC is level 6, the complexity of class 3 dead cell DDC is level 7, and the complexity of class 4 medium PL A case of complexity level 4 is illustrated.

In this case, the image generation condition 43 is that classes 2 to 4 (undifferentiated cell UDC, dead cell DDC, medium PL) are integrated in the first provisional learning, and that classes 3 and 4 are integrated in the second provisional learning. (Dead cell DDC, medium PL) are registered respectively. In the first provisional learning, the complexity of the differentiated cell DC of class 1 is level 5, the complexity of the integrated class of classes 2 to 4 is level 5.7 (≈ (6 + 7 + 4) / 3), and the other 3 The complexity is not biased compared to the integration of two classes. In the second provisional learning, the complexity of class 1 differentiated cell DC is level 5, the complexity of class 2 undifferentiated cell UDC is level 6, and the complexity of class 3 and 4 combined class is level 5. .5 (=(7+4)/2), and the complexity is not biased compared to when classes 2 and 3 or classes 2 and 4 are combined.

Thus, in the fourth embodiment, the image generation condition 43 is such that the complexity of each class of the temporary annotation image 18 is balanced. Therefore, as in the case of the third embodiment, the temporary learning load of each class of the temporary annotation image 18 can be averaged.

As shown in FIG. 37, the image generation condition 43 of the third embodiment is such that the area of each class of the temporary annotation image 18 is not biased, or the complexity of each class of the temporary annotation image 18 is set to A user's selection instruction may be accepted as to whether the image generation conditions 43 of the fourth embodiment are set so as not to be unbiased.

In FIG. 37, on the image generation condition designation screen 140, the image generation condition 43 is set so that the area of each class of the provisional annotation image 18 is not biased, or the complexity of each class of the provisional annotation image 18 is not biased. A radio button 141 is provided for alternatively selecting whether or not the content should not be included. The user selects the radio button 141 and selects the designation button 142 . The receiving unit 52 selects the image generation condition 43 so that the area of each class of the temporary annotation image 18 is not biased, or sets the image generation condition 43 to the complexity of each class of the temporary annotation image 18 . Receives a selection instruction that the contents should not be biased. That is, the reception unit 52 is an example of the "third reception unit" according to the technology of the present disclosure. Note that when the cancel button 143 is selected, the display of the image generation condition designation screen 140 is erased.

In this way, in the receiving unit 52, the image generation condition 43 is set so that the area of each class of the provisional annotation image 18 is not biased, or the content is set so that the complexity of each class of the provisional annotation image 18 is not biased. If the user selects the image generation condition 43, the image generation unit 46 can generate the provisional annotation image 18 based on the image generation condition 43 that is suitable for the user. It should be noted that whether the image generation condition 43 is set such that the area of each class of the temporary annotation image 18 is not biased, or the image generation condition 43 is set such that the complexity of each class of the provisional annotation image 18 is not biased. can be selected instead of alternatively.

[Fifth embodiment]
In the fifth embodiment shown in FIG. 38, the image generation condition 43 is such that the class having the largest area is left as one class without being integrated.

In FIG. 38, in the fifth embodiment, area information 130 is stored in the storage device 30 as in the third embodiment. In FIG. 38, the area ratio of class 1 differentiated cell DC is 23%, the area ratio of class 2 undifferentiated cell UDC is 7%, the area ratio of class 3 dead cell DDC is 2%, and the area ratio of class 4 medium PL A case where the area ratio is 68% is illustrated. That is, the class with the largest area is class 4 medium PL.

In this case, the image generation condition 43 is that classes 1 to 3 (differentiated cell DC, undifferentiated cell UDC, dead cell DDC) are integrated in the first provisional learning, and class 2, 3 (undifferentiated cell UDC, dead cell DDC) are registered respectively. The medium PL of class 4, which is the class with the largest area, is left as one class without being integrated in each provisional learning. In addition, the differentiated cell DC of class 1, which is the class having the largest area among classes 1 to 3, is not integrated into one class in the second provisional learning.

As described above, in the fifth embodiment, the image generation condition 43 is such that the class having the largest area is left as one class without being integrated. In this way, by temporarily learning the class with the largest area independently at an early stage, it is possible to naturally create a flow of temporary learning from the global structure to the detailed structure. Therefore, it is possible to obtain a trained model 21T with higher class discrimination accuracy.

[Sixth Embodiment]
In the sixth embodiment shown in FIG. 39, the image generation condition 43 is such that the class with the lowest complexity is left as one class without being integrated.

39, in the sixth embodiment, the complexity information 135 is stored in the storage device 30 as in the fourth embodiment. In FIG. 39, the complexity of class 1 differentiated cell DC is level 8, the complexity of class 2 undifferentiated cell UDC is level 7, the complexity of class 3 dead cell DDC is level 6, and the complexity of class 4 medium PL A case of complexity level 3 is illustrated. That is, the class with the lowest complexity is class 4 medium PL.

The image generation condition 43 in this case is that classes 1 to 3 (differentiated cell DC, undifferentiated cell UDC, dead cell DDC) are integrated in the first provisional learning, and class 1, 2 (differentiated cell DC, undifferentiated cell UDC) are respectively registered. Class 4 medium PL, which is the class with the lowest complexity, is left as one class without being merged in each trial. In addition, the dead cell DDC of class 3, which is the class with the lowest complexity among classes 1 to 3, is not integrated into one class in the second provisional learning.

Thus, in the sixth embodiment, the image generation condition 43 is such that the class with the lowest complexity is left as one class without being merged. In this way, the class with the minimum complexity is provisionally learned independently at an early stage, thereby naturally creating a provisional learning flow from the global structure to the fine structure, as in the fifth embodiment. , and it is possible to obtain a trained model 21T with higher class discrimination accuracy.

As shown in FIG. 40, the image generation condition 43 of the fifth embodiment is set such that the class having the largest area is left as one class without being integrated, or the minimum complexity is set to A user's selection instruction may be accepted as to whether the image generation condition 43 of the sixth embodiment is to remain as one class without merging the existing classes.

In FIG. 40, on the image generation condition specification screen 150, the image generation condition 43 is set such that the class having the maximum area is left as one class without being integrated, or the class having the minimum complexity is left as it is. A radio button 151 is provided for alternatively selecting whether the classes should be kept as one class without being integrated. The user selects the radio button 151 and selects the designation button 152 . The reception unit 52 sets the image generation condition 43 to a selection instruction that the class having the largest area is left as one class without merging, or sets the image generation condition 43 to the minimum complexity. A selection instruction is accepted to say that the classes having the . That is, the reception unit 52 is an example of the "fourth reception unit" according to the technology of the present disclosure. Note that when the cancel button 153 is selected, the display of the image generation condition designation screen 150 is erased.

In this way, in the receiving unit 52, the image generation condition 43 is set such that the class with the largest area is not integrated and is left as one class, or the class with the minimum complexity is not integrated. 37, based on the image generation condition 43 suitable for the user, as in the case of FIG. The temporary annotation image 18 can be generated in the image generator 46 . The image generation condition 43 is such that the class with the maximum area is left as one class without being integrated, or the class with the minimum complexity is left as one class without being integrated. The image generation condition 43 with the content of "Yes" may be selected not alternatively but both.

[Seventh embodiment]
In the seventh embodiment shown in FIG. 41, a temporary annotation image 18 is displayed.

In FIG. 41 , a temporary annotation image display screen 160 is displayed on the display 34 by the display control section 51 . The temporary annotation image display screen 160 is displayed instead of the image generation condition designation screen 65 when the image generation condition 43 is designated on the image generation condition designation screen 65 shown in FIG. 11, for example. The temporary annotation image display screen 160 displays a temporary annotation image 18_1 used for the first temporary learning and a temporary annotation image 18_2 used for the second temporary learning. The user selects the confirmation button 161 if the displayed temporary annotation images 18_1 and 18_2 are acceptable. When the confirmation button 161 is selected, the temporary annotation image display screen 160 is cleared. On the other hand, if the user is not satisfied with the displayed temporary annotation images 18_1 and 18_2, the user selects the re-specify button 162 . When the redesignation button 162 is selected, the display control unit 51 displays the image generation condition designation screen 65 shown in FIG.

Thus, in the seventh embodiment, the temporary annotation image 18 is displayed. Therefore, the user can confirm the temporary annotation image 18 and, in some cases, redesignate the image generation condition 43 .

Although the temporary annotation images 18_1 and 18_2 are displayed side by side on the temporary annotation image display screen 160 in FIG. 41, the present invention is not limited to this. The temporary annotation images 18_1 and 18_2 may be displayed one by one. However, it is preferable to display the temporary annotation images 18 used for each temporary learning in parallel as shown in FIG. 41 because the transition of the temporary annotation images 18 can be confirmed.

[Eighth embodiment]
In the eighth embodiment shown in FIGS. 42 and 43, multiple types of temporary annotation images 18 with different integrated classes are generated and displayed, and one of the multiple types of temporary annotation images 18 is displayed. 18, and the temporary annotation image 18 for which the selection instruction is received is used as the temporary learning data 15 .

As shown in FIG. 42, the image generator 46 generates a plurality of types of temporary annotation images 18 with different integrated classes. FIG. 42 illustrates a case where a plurality of types of provisional annotation images 18_2 used for the second provisional learning are generated.

More specifically, as shown in FIG. 42A , the image generation unit 46 performs class 1 differentiated cells according to an image generation condition 43A that integrates class 1 differentiated cells DC and class 2 undifferentiated cells UDC. A provisional annotation image 18_2A is generated in which cell DC and undifferentiated cell UDC of class 2 are integrated as an integrated class. In addition, as shown in FIG. 42B, the image generation unit 46 performs class 1 differentiated cell DC and class 3 differentiated cell DC in accordance with image generation condition 43B that integrates class 1 differentiated cell DC and class 3 dead cell DDC. dead cells DDC are integrated as an integrated class to generate a temporary annotation image 18_2B. Furthermore, as shown in FIG. 42C , the image generation unit 46 combines the class 2 undifferentiated cell UDC with the class 2 undifferentiated cell UDC according to the image generation condition 43C that integrates the class 2 undifferentiated cell UDC and the class 3 dead cell DDC. A temporary annotation image 18_2C in which the dead cell DDC of class 3 is integrated as an integrated class is generated.

The display control unit 51 displays a temporary annotation image selection screen 170 shown in FIG. 43 on the display 34 . A plurality of types of temporary annotation images 18_2A to 18_2C generated by the image generator 46 are displayed on the temporary annotation image selection screen 170. FIG. Radio buttons 171 are provided below the plurality of types of temporary annotation images 18_2A to 18_2C. The radio button 171 is a button for selecting one of the temporary annotation images 18_2A to 18_2C.

The user selects the radio button 171 and selects the designation button 172 . The receiving unit 52 receives an instruction to select one temporary annotation image 18 from among the plurality of types of temporary annotation images 18_2A to 18_2C. That is, the reception unit 52 is an example of the "fifth reception unit" according to the technology of the present disclosure. The provisional learning unit 48 uses the provisional annotation image 18_2 selected by the radio button 171 as the provisional learning data 15_2 for the second provisional learning. Note that when the cancel button 173 is selected, the display of the temporary annotation image selection screen 170 is erased.

FIG. 43 illustrates a case where the provisional annotation image 18_2B is selected with the radio button 171 . When the designation button 172 is selected in this state, the provisional learning unit 48 uses the provisional annotation image 18_2B as the provisional learning data 15_2 for the second provisional learning.

Thus, in the eighth embodiment, the image generator 46 generates multiple types of temporary annotation images 18 with different integrated classes. The display control unit 51 controls the display of multiple types of temporary annotation images 18 . The accepting unit 52 accepts a user's instruction to select one of a plurality of types of temporary annotation images. The provisional learning unit 48 uses the provisional annotation image 18 for which the selection instruction is received by the reception unit 52 as the provisional learning data 15 . Therefore, the user can select the temporary annotation image 18 to be used as the temporary learning data 15 while actually checking the multiple types of temporary annotation images 18 .

Note that in the eighth embodiment, selecting the temporary annotation image 18 is equivalent to setting the image generation condition 43 . Therefore, it is not necessary to designate the image generation condition 43 using the image generation condition designation screen 65 shown in FIG. 11 or the like.

In the first provisional learning, it is not always necessary to integrate the three classes as illustrated above. For example, class 1 differentiated cell DC and class 2 undifferentiated cell UDC, and class 3 dead cell DDC and class 4 culture medium PL may be combined to form two classes in total.

The number of classes N of the annotation image 19 is not limited to four as long as it is three or more. Therefore, the number of provisional learnings is not limited to two as illustrated.

Various modifications are possible for the hardware configuration of the computer that constitutes the machine learning system 2 . For example, the learning device 10 and the operation device 11 may be integrated into one computer. Also, at least one of the learning device 10 and the operation device 11 can be configured with a plurality of computers separated as hardware for the purpose of improving processing capability and reliability. For example, the functions of the image generating unit 46, the temporary model generating unit 47, and the temporary learning unit 48 of the learning device 10, and the functions of the model generating unit 49 and the main learning unit 50 are distributed to two computers. . In this case, the learning device 10 is composed of two computers.

In this way, the hardware configuration of the computer of the machine learning system 2 can be appropriately changed according to required performance such as processing power, safety, and reliability. Furthermore, not only the hardware but also application programs such as the operation program 40 can of course be duplicated or distributed and stored in multiple storage devices for the purpose of ensuring safety and reliability. be.

In each of the above-described embodiments, the learning input image 17 is a cell image obtained by photographing a plurality of cells in culture, and the class is a cell, medium, or the like, but the class is not limited to this. For example, MRI (Magnetic Resonance Imaging) images may be used as learning input images 17, and organs such as liver and kidney may be used as classes.

The model is not limited to U-Net, but may be other convolutional neural networks such as SegNet and ResNet (Residual Network).

In each of the above embodiments, for example, the RW control unit 45, the image generation unit 46, the temporary model generation unit 47, the temporary learning unit 48 (first processing unit 90, first evaluation unit 91, first update unit 92), model generation 49, main learning unit 50 (second processing unit 105, second evaluation unit 106, second update unit 107), display control unit 51, reception unit 52, and transmission control unit 53. As the hardware structure of (Processing Unit), various processors shown below can be used. In addition to the CPU 32, which is a general-purpose processor that executes software (operation program 40) and functions as various processing units, as described above, the various processors include FPGAs (Field Programmable Gate Arrays), etc. Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed, ASIC (Application Specific Integrated Circuit), etc. It includes electric circuits and the like.

One processing unit may be configured with one of these various processors, or a combination of two or more processors of the same or different type (for example, a combination of a plurality of FPGAs and/or a CPU and combination with FPGA). Also, a plurality of processing units may be configured by one processor.

As an example of configuring a plurality of processing units with a single processor, first, as represented by computers such as clients and servers, a single processor is configured by combining one or more CPUs and software. There is a form in which a processor functions as multiple processing units. Secondly, as typified by System On Chip (SoC), etc., there is a mode of using a processor that realizes the function of the entire system including multiple processing units with a single IC (Integrated Circuit) chip. be. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

Further, as the hardware structure of these various processors, more specifically, an electric circuit in which circuit elements such as semiconductor elements are combined can be used.

From the above description, the invention described in the following supplementary item 1 can be grasped.

[Appendix 1]
an acquisition processor that acquires learning data that is a set of a learning input image and an annotation image in which three or more classes to be subjected to semantic segmentation are specified for the learning input image;
an image generation processor that integrates at least two classes out of three or more classes specified in the annotation image to generate a temporary annotation image having fewer classes than the annotation image;
A provisional machine learning model having a configuration corresponding to the number of classes of the provisional annotation image is learned using provisional learning data that is a set of the learning input image and the provisional annotation image, and the provisional machine learning model is learned. a provisional learning processor as a completed provisional machine learning model;
a machine learning model generation processor that uses at least part of the trained temporary machine learning model to generate a machine learning model having a configuration corresponding to the number of classes of the annotation image;
a learning processor that trains the machine learning model using the learning data and sets the machine learning model as a trained machine learning model;
A learning device with

The technology of the present disclosure can also appropriately combine various embodiments and various modifications described above. Moreover, it is needless to say that various configurations can be employed without departing from the scope of the present invention without being limited to the above embodiments. Furthermore, the technology of the present disclosure extends to storage media that non-temporarily store programs in addition to programs.

The above description and illustration are detailed descriptions of the parts related to the technology of the present disclosure, and are merely examples of the technology of the present disclosure. For example, the above descriptions of configurations, functions, actions, and effects are descriptions of examples of configurations, functions, actions, and effects of portions related to the technology of the present disclosure. Therefore, unnecessary parts may be deleted, new elements added, or replaced with respect to the above-described description and illustration without departing from the gist of the technology of the present disclosure. Needless to say. In addition, in order to avoid complication and facilitate understanding of the portion related to the technology of the present disclosure, the descriptions and illustrations shown above require no particular explanation in order to enable implementation of the technology of the present disclosure. Descriptions of common technical knowledge, etc., that are not used are omitted.

As used herein, "A and/or B" is synonymous with "at least one of A and B." That is, "A and/or B" means that only A, only B, or a combination of A and B may be used. In addition, in this specification, when three or more matters are expressed by connecting with "and/or", the same idea as "A and/or B" is applied.

All publications, patent applications and technical standards mentioned herein are expressly incorporated herein by reference to the same extent as if each individual publication, patent application and technical standard were specifically and individually noted to be incorporated by reference. incorporated by reference into the book.

Claims (21)

an acquisition unit that acquires learning data that is a set of an input image for learning and an annotation image in which three or more classes to be subjected to semantic segmentation are specified for the input image for learning;
an image generating unit that integrates at least two classes out of three or more classes specified in the annotation image and generates a temporary annotation image having a smaller number of classes than the annotation image;
A provisional machine learning model having a configuration corresponding to the number of classes of the provisional annotation image is learned using provisional learning data that is a set of the learning input image and the provisional annotation image, and the provisional machine learning model is learned. a provisional learning unit as a completed provisional machine learning model;
a machine learning model generation unit that uses at least part of the trained temporary machine learning model to generate a machine learning model having a configuration corresponding to the number of classes of the annotation image;
a main learning unit that learns the machine learning model using the learning data and sets the machine learning model as a trained machine learning model;
A learning device with The learning device according to claim 1, comprising a temporary machine learning model generation unit that generates the temporary machine learning model. When the number of classes of the annotation image is M, and the number of classes of the temporary annotation image is N, the image generating unit and the temporary learning unit perform the process of generating the temporary annotation image and the temporary machine learning model. Claim 1 or claim 2, wherein the process of using a completed temporary machine learning model is repeated a plurality of times while gradually increasing the number N of classes of the temporary annotation image until the number N of classes of the temporary annotation image reaches M-1. The learning device according to . In the learning device according to claim 3 citing claim 2,
The temporary machine learning model generation unit is a learning device that generates the temporary machine learning model to be used this time by using at least part of the trained temporary machine learning model used last time. 5. The learning device according to claim 3, wherein the provisional learning unit changes a loss function used to evaluate class discrimination accuracy of the provisional machine learning model. 6. The learning device according to claim 5, wherein the provisional learning unit weights a loss function for a class common to the previous time less than a weight for a loss function for a class newly appearing this time. The temporary learning unit and the main learning unit, in the process of turning the temporary machine learning model into the trained temporary machine learning model and the process of turning the machine learning model into the trained machine learning model, 7. The learning device according to any one of claims 1 to 6, which does not update. 8. The learning device according to claim 7, further comprising a first receiving unit that receives a user's designation of the portion not to be updated. 9. The learning device according to any one of claims 1 to 8, wherein the image generation unit generates the temporary annotation image according to image generation conditions specified in advance. 10. The learning device according to claim 9, further comprising a second reception unit that receives designation of the image generation condition by a user. 11. The learning device according to claim 9, wherein the image generation condition is such that the area of each class of the temporary annotation image is not biased. 12. The learning device according to any one of claims 9 to 11, wherein the image generation condition is such that complexity of each class of the temporary annotation image is not biased. In the learning device according to claim 12 citing claim 11,
A user who selects whether the image generation condition is such that the area of each class of the temporary annotation image is balanced, or the image generation condition is such that the complexity of each class of the temporary annotation image is balanced. A learning device comprising a third reception unit that receives a selection instruction from. 14. The learning device according to any one of claims 9 to 13, wherein the image generation condition is that the class having the largest area is left as one class without being integrated. 15. The learning device according to any one of claims 9 to 14, wherein the image generation condition is such that the class with the lowest complexity is left as one class without being merged. In the learning device according to claim 15 citing claim 14,
The image generation condition is such that the class with the largest area is left as one class without being merged, or the class with the lowest complexity is left as one class without being merged. A learning device comprising a fourth reception unit that receives a user's selection instruction as to whether or not the image generation condition is to be set. 17. The learning device according to any one of claims 1 to 16, further comprising a display control unit that controls display of the temporary annotation image. The image generation unit generates a plurality of types of temporary annotation images with different integrated classes,
The display control unit performs control to display a plurality of types of the temporary annotation images,
A fifth reception unit that receives a user's selection instruction for one of the plurality of types of temporary annotation images,
18. The learning device according to claim 17, wherein the provisional learning unit uses the provisional annotation image for which the selection instruction is received by the fifth reception unit as the provisional learning data. 19. The learning device according to any one of claims 1 to 18, wherein the learning input image is a cell image obtained by photographing a plurality of cells in culture. an acquisition step of acquiring learning data that is a set of an input image for learning and an annotation image in which three or more classes to be subjected to semantic segmentation are specified for the input image for learning;
an image generating step of integrating at least two classes out of three or more classes specified in the annotation image and generating a temporary annotation image having a smaller number of classes than the annotation image;
A provisional machine learning model having a configuration corresponding to the number of classes of the provisional annotation image is learned using provisional learning data that is a set of the learning input image and the provisional annotation image, and the provisional machine learning model is learned. a provisional learning step as a completed provisional machine learning model;
A machine learning model generation step of generating a machine learning model having a configuration corresponding to the number of classes of the annotation image using at least part of the trained temporary machine learning model;
A main learning step of learning the machine learning model using the learning data and making the machine learning model a trained machine learning model;
A method of operating a learning device comprising: an acquisition unit that acquires learning data that is a set of an input image for learning and an annotation image in which three or more classes to be subjected to semantic segmentation are specified for the input image for learning;
an image generating unit that integrates at least two classes out of three or more classes specified in the annotation image and generates a temporary annotation image having a smaller number of classes than the annotation image;
A provisional machine learning model having a configuration corresponding to the number of classes of the provisional annotation image is learned using provisional learning data that is a set of the learning input image and the provisional annotation image, and the provisional machine learning model is learned. a provisional learning unit as a completed provisional machine learning model;
a machine learning model generation unit that generates a machine learning model having a configuration corresponding to the number of classes of the annotation image using at least part of the trained temporary machine learning model;
As a main learning unit that learns the machine learning model using the learning data and sets the machine learning model as a trained machine learning model,
An operating program for a learning device that makes a computer work.

Images