JP2024025181A - Information processing device, information processing method, and program - Google Patents

Abstract

An object of the present invention is to reduce differences in feature amounts caused by differences in imaging conditions.
An information processing device includes an image acquisition section and a machine learning section. The image acquisition unit acquires a first image obtained by photographing a pathological specimen, and a second image obtained by photographing the pathological specimen under photographing conditions different from photographing conditions for the first image. The machine learning unit extracts the feature amount of the first image outputted by a first feature amount extractor that outputs a feature amount representing the feature of the image, and the feature amount outputted by the first feature amount extractor. The loss value indicating the error with the feature amount of the second image is set to a value corresponding to the degree of coincidence between the feature of the pathological specimen in the first image and the feature of the pathological specimen in the second image. Then, the parameters of the first feature extractor are updated.
[Selection diagram] Figure 1

Description

Embodiments disclosed in this specification and drawings relate to an information processing device, an information processing method, and a program.

Conventionally, in the field of pathology, tissue diagnosis has been performed using pathological images of tissues and specimens of subjects. In recent years, research and development have been conducted on techniques to improve the efficiency of tissue diagnosis by presenting information obtained by analyzing pathological images to doctors and the like.

Image analysis technology in the field of pathology includes, for example, image classification technology that determines whether a certain region in a pathological image contains an abnormality by analyzing its characteristics, and image segmentation technology that extracts abnormal regions. etc.

However, pathological images may differ in the manner in which they are depicted even for the same subject due to differences in imaging conditions such as staining and imaging equipment, and accordingly, the results of image analysis may also differ.

JP 2021-065293 Publication

One of the problems that the embodiments disclosed in this specification and drawings are intended to solve is to reduce differences in feature amounts caused by differences in imaging conditions. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problems. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems.

The information processing device according to the embodiment includes an image acquisition section and a machine learning section. The image acquisition unit acquires a first image obtained by photographing a pathological specimen, and a second image obtained by photographing the pathological specimen under photographing conditions different from photographing conditions for the first image. The machine learning unit uses the feature amount of the first image outputted by a first feature amount extractor that outputs a feature amount representing the feature of the image, and the feature amount outputted by the first feature amount extractor. The loss value indicating the error with the feature amount of the second image is set to a value corresponding to the degree of matching between the feature of the pathological specimen in the first image and the feature of the pathological specimen in the second image. Then, the parameters of the first feature extractor are updated.

FIG. 1 is a diagram showing an example of the configuration of a pathological information processing system according to the first embodiment. FIG. 2 is a diagram illustrating an example of a flow for updating the parameters of the first feature extractor, which is executed by the information processing apparatus according to the first embodiment. FIG. 3 is a flowchart illustrating an example of a learning process executed by the information processing apparatus according to the first embodiment. FIG. 4 is a diagram illustrating an example of the configuration of a pathological information processing system according to the second embodiment. FIG. 5 is a diagram illustrating an example of a flow for applying the first feature amount extractor to the first neural network, which is executed by the information processing apparatus according to the second embodiment. FIG. 6 is a flowchart illustrating an example of a learning process executed by the information processing apparatus according to the second embodiment. FIG. 7 is a diagram illustrating an example of the configuration of a pathological information processing system according to the third embodiment. FIG. 8 is a diagram illustrating an example of a flow of adjustment such as positioning of a specimen image performed by the information processing apparatus according to the third embodiment. FIG. 9 is a flowchart illustrating an example of a learning process executed by the information processing apparatus according to the third embodiment. FIG. 10 is a diagram illustrating an example of the configuration of a pathological information processing system according to the fourth embodiment. FIG. 11 is a diagram illustrating an example of a sample image extraction flow executed by the information processing apparatus according to the fourth embodiment. FIG. 12 is a flowchart illustrating an example of a learning process executed by the information processing apparatus according to the fourth embodiment. FIG. 13 is a diagram illustrating an example of the configuration of a pathological information processing system according to the fifth embodiment. FIG. 14 is a diagram illustrating an example of a flow for generating a specimen image executed by the information processing apparatus according to the fifth embodiment. FIG. 15 is a flowchart illustrating an example of a learning process executed by the information processing apparatus according to the fifth embodiment.

Hereinafter, an information processing apparatus, an information processing method, and a program related to the present embodiment will be described with reference to the drawings. In the following embodiments, parts with the same reference numerals perform similar operations, and redundant explanations will be omitted as appropriate.

(First embodiment)
FIG. 1 is a diagram showing an example of the configuration of a pathological information processing system 1 according to the first embodiment. As shown in FIG. 1, the pathological information processing system 1 includes a first imaging device 10, a second imaging device 20, and an information processing device 30. The first photographing device 10, the second photographing device 20, and the information processing device 30 are connected to each other via a network 40 so as to be able to communicate with each other. Note that the configuration shown in FIG. 1 is an example, and the number of each device may be changed arbitrarily. Furthermore, devices not shown in FIG. 1 may be connected to the network 40.

The first imaging device 10 photographs a pathological specimen. For example, the first imaging device 10 may be a camera that photographs a pathological specimen, a digital slide scanner that photographs a pathological specimen, or another device. The first imaging device 10 then transmits a specimen image obtained by photographing the pathological specimen to the information processing device 30.

The second imaging device 20 photographs a pathological specimen. For example, the second imaging device 20 may be a camera that photographs pathological specimens, a digital slide scanner that photographs pathological specimens, or another device. The second imaging device 20 then transmits a specimen image obtained by photographing the pathological specimen to the information processing device 30.

The information processing device 30 is computer equipment such as a personal computer, a server device, or a workstation. The information processing device 30 acquires specimen images from the first imaging device 10 and the second imaging device 20. The information processing device 30 extracts feature amounts representing the characteristics of the image from the sample image acquired from the first imaging device 10 or the second imaging device 20. Furthermore, the information processing device 30 may include a neural network that executes various processes based on the extracted feature amounts.

The information processing device 30 will be explained in detail.

For example, the information processing device 30 includes an NW (network) interface 310, an input interface 320, a display 330, a storage circuit 340, and a processing circuit 350.

The NW interface 310 is connected to the processing circuit 350 and controls the transmission and communication of various data with each device connected via the network 40. For example, the NW interface 310 is implemented by a network card, network adapter, NIC (Network Interface Controller), or the like.

The input interface 320 is connected to the processing circuit 350 , converts an input operation received from an operator into an electrical signal, and outputs the electrical signal to the processing circuit 350 . Specifically, the input interface 320 converts an input operation received from an operator into an electrical signal and outputs it to the processing circuit 350. For example, the input interface 320 may include a trackball, a switch button, a mouse, a keyboard, a touchpad that performs input operations by touching the operation surface, a touchscreen that integrates a display screen and a touchpad, and a non-control device that uses an optical sensor. This is realized by a touch input circuit, a voice input circuit, etc. Note that in this specification, the input interface 320 is not limited to one that includes physical operation parts such as a mouse and a keyboard. For example, an example of the input interface 320 includes an electrical signal processing circuit that receives an electrical signal corresponding to an input operation from an external input device provided separately from the device and outputs this electrical signal to a control circuit.

The display 330 is connected to the processing circuit 350 and displays various information and various image data output from the processing circuit 350. For example, the display 330 is realized by a liquid crystal display, a CRT (Cathode Ray Tube) display, an organic EL display, a plasma display, a touch panel, or the like.

The storage circuit 340 is connected to the processing circuit 350 and stores various data. Furthermore, the storage circuit 340 stores various programs for realizing various functions by being read and executed by the processing circuit 350. For example, the storage circuit 340 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, a hard disk, an optical disk, or the like.

The processing circuit 350 controls the overall operation of the information processing device 30. The processing circuit 350 has, for example, an image acquisition function 351 and a machine learning function 352. In the embodiment, each processing function performed by the component image acquisition function 351 and machine learning function 352 is stored in the storage circuit 340 in the form of a computer-executable program. The processing circuit 350 is a processor that reads programs from the storage circuit 340 and executes them to implement functions corresponding to each program. In other words, the processing circuit 350 in a state where each program is read has each function shown in the processing circuit 350 of FIG.

Although the image acquisition function 351 and the machine learning function 352 are described as being implemented in a single processor in FIG. 1, the processing circuit 350 is configured by combining a plurality of independent processors, and each processor The functions may be implemented by executing a program. In addition, in FIG. 1, a single storage circuit such as the storage circuit 340 stores programs corresponding to each processing function, but a plurality of storage circuits may be distributed and arranged, and the processing circuit 350 , a configuration may be adopted in which the corresponding program is read from an individual storage circuit.

The word "processor" used in the above explanation refers to, for example, a CPU (Central Processing Unit), a GPU (Graphical Processing Unit), an Application Specific Integrated Circuit (ASIC), a programmable logic device (for example, a simple Refers to circuits such as a programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA). The processor realizes its functions by reading and executing programs stored in the storage circuit 340. Note that instead of storing the program in the storage circuit 340, the program may be directly incorporated into the circuit of the processor. In this case, the processor realizes its functions by reading and executing a program built into the circuit.

The image acquisition function 351 acquires a first image obtained by photographing a pathological specimen, and a second image obtained by photographing the pathological specimen under photographing conditions different from the photographing conditions for the first image. The image acquisition function 351 is an example of an image acquisition unit. The imaging conditions include the type of imaging device, the settings of the imaging device, the imaging time, the position of the photographed subject, the orientation of the photographed subject, the environment during imaging, and the like.

More specifically, the image acquisition function 351 acquires specimen images obtained by photographing a pathological specimen from the first imaging device 10 and the second imaging device 20. Further, the image acquisition function 351 may acquire a plurality of specimen images under different imaging conditions. Note that the image acquisition function 351 is not limited to the first imaging device 10 and the second imaging device 20, and may acquire images from a server device that has an image of a pathological specimen.

The machine learning function 352 combines the feature amount of the first image outputted by the feature amount extractor that outputs the feature amount representing the feature of the image and the feature amount of the second image outputted by the feature amount extractor. Update the parameters of the feature extractor so that the loss value indicating the error becomes a value corresponding to the degree of matching between the features of the pathological specimen in the first image and the features of the pathological specimen in the second image. . The machine learning function 352 is an example of a machine learning section.

More specifically, the machine learning function 352 learns so that as the degree of matching between the features of the pathological specimens of the two specimen images input to the feature extractor increases, the features extracted from the respective specimen images become similar. . In other words, the machine learning function 352 reduces the loss value as the degree of matching between the features of the pathological specimen in the first image input to the feature extractor and the characteristics of the pathological specimen in the second image increases. Update the parameters of the feature extractor as follows.

The characteristics that the pathological specimen has include the shape of the pathological specimen depicted in the specimen image, the color of the pathological specimen, the orientation of the pathological specimen, and the size of the pathological specimen. Note that the characteristics that a pathological specimen has are not limited to these characteristics, and may include other characteristics, or may not include some of these characteristics. The machine learning function 352 updates the parameters of the feature amount extractor so that even if some of the features of the pathological specimens are different, the loss value becomes smaller as the degree of matching of other features becomes higher.

Furthermore, when the positions of the pathological specimens are close in the two specimen images input to the feature extractor, the degree of matching between the features of the pathological specimens becomes high. On the other hand, in the two specimen images input to the feature extractor, if the location of the pathological specimen is far away, the degree of matching between the features of the pathological specimen will be low.

Therefore, the machine learning function 352 changes the feature amount of the first image and the feature amount of the second image as the position of the pathological specimen in the first image and the position of the pathological specimen in the second image become closer. The parameters of the feature extractor may be updated so that the loss value indicating the error between In other words, the machine learning function 352 determines that the position of the pathological specimen is close when the degree of coincidence between the features of the pathological specimen is high in the two specimen images, and the feature amount of the first image and the second The parameters of the feature extractor are updated so that the loss value indicating the error with the feature of the image becomes smaller.

In addition to the value output by the final layer of the feature extractor, the feature value may be an output value of a hidden layer of the feature extractor, or a combination of a part of the output values of the hidden layer. A portion of the output value of the hidden layer is an output value that is selectively cut out along each dimensional axis with respect to the spatial shape of the output value of the hidden layer.

Here, a flow for updating the parameters of the first feature extractor E1 will be described. FIG. 2 is a diagram illustrating an example of a flow of updating the parameters of the first feature extractor E1 executed by the information processing device 30 according to the first embodiment.

The image acquisition function 351 acquires, for example, a first specimen image G1 and a third specimen image G3 photographed by the first imaging device 10. Further, the image acquisition function 351 acquires, for example, the second specimen image G2 photographed by the second photographing device 20. The image acquisition function 351 acquires a second specimen image G2, which is an image of the same position of the same pathological specimen as the first specimen image G1. Furthermore, the image acquisition function 351 acquires a third specimen image G3 in which at least one of the locations is different from that of the pathological specimen in the first specimen image G1.

The first specimen image G1 and the second specimen image G2 depict the same position of the same pathological specimen. In other words, the pathological specimen in the first specimen image G1 and the pathological specimen in the second specimen image G2 have the same characteristics. Further, the first specimen image G1 and the second specimen image G2 are photographed under different photographing conditions. For example, the first specimen image G1 and the second specimen image G2 have different staining.

The first specimen image G1 and the third specimen image G3 differ in at least one of the pathological specimen and the position. In other words, the pathological specimen in the first specimen image G1 and the pathological specimen in the third specimen image G3 have different characteristics. The pathological specimen in the third specimen image G3 has different characteristics from the pathological specimen in the second specimen image G2.

The first feature extractor E1 is a feature extractor that extracts a feature from the specimen image. The first feature extractor E1 extracts the first feature Z1 from the first specimen image G1. Further, the first feature extractor E1 extracts the second feature Z2 from the second specimen image G2. Further, the first feature extractor E1 extracts the third feature Z3 from the third specimen image G3.

The machine learning function 352 calculates a loss value using a loss function representing an error between feature amounts. For example, cross entropy, distance function, etc. are used as the loss function. For example, the machine learning function 352 calculates a first loss value L1 representing an error between the first feature amount Z1 and the second feature amount Z2 using a loss function. Furthermore, the machine learning function 352 calculates a second loss value L2 representing an error between the first feature amount Z1 and the third feature amount Z3 using a loss function. Furthermore, the machine learning function 352 calculates a third loss value L3 representing an error between the second feature amount Z2 and the third feature amount Z3 using a loss function.

The machine learning function 352 updates the parameters of the first feature extractor E1 based on the calculated loss value so that the loss value decreases as the degree of matching of the pathological specimen in each specimen image increases. For example, the machine learning function 352 updates the parameters of the first feature extractor E1 so that the following formula (1) is satisfied.

L(Z1, Z2)<L(Z1, Z3)...(1)

In formula (1), L represents a loss function. Z1 indicates the first feature amount Z1. Z2 indicates the second feature amount Z2. Z3 indicates the third feature amount Z3.

The machine learning function 352 updates the parameters of the first feature amount extractor E1 so that as the degree of matching of the features of the pathological specimen in the specimen image increases, the loss value indicating the error between the feature amounts becomes smaller. More specifically, the machine learning function 352 performs learning to update the parameters of the first feature extractor E1 as the final loss considering the relative magnitude of the loss value calculated by combining each feature. . For example, machine learning functionality 352 performs loss calculations due to cross-entropy errors.

In the case of using the cross-entropy error, the machine learning function 352 uses the first feature amount Z1 and the second feature amount Z2, the first feature amount Z1 and the third feature amount Z3, and the second feature amount Z2 and the second feature amount Z2 shown in FIG. A first loss value L1, a second loss value L2, and a third loss value L3 are calculated for each combination of the three feature quantities Z3. The machine learning function 352 converts the first loss value L1, second loss value L2, and third loss value L3 into an estimated probability distribution.

For example, the machine learning function 352 converts the smaller the loss value to 1 and the larger the loss value to 0. Then, the machine learning function 352 calculates the cross-entropy error by assuming that the true probability distribution is "1, 0, 0." In other words, the machine learning function 352 determines that the cross-entropy error is minimized when the estimated probability corresponding to the first feature Z1 and the second feature Z2, which are a combination of features with a high degree of matching in the pathological specimen, is maximized. The final loss value is output and used for learning.

Further, the first specimen image G1 and the second specimen image G2 are images obtained by photographing the same position of the same pathological specimen. Therefore, the machine learning function 352 updates the parameters so that the loss value is smaller than that of a combination of sample images at different positions. That is, when the first specimen image G1 and the second specimen image G2 are a combination of specimen images in which the same position of the same pathological specimen was photographed, the machine learning function 352 performs a combination of specimen images photographed in different positions. Update the parameters so that the loss value is smaller than the combination.

Next, a learning process executed by the information processing device 30 according to the first embodiment will be described. FIG. 3 is a flowchart illustrating an example of a learning process executed by the information processing device 30 according to the first embodiment.

The image acquisition function 351 acquires a plurality of specimen images (step S11).

The machine learning function 352 inputs the plurality of specimen images acquired by the image acquisition function 351 to the feature quantity extractor, thereby acquiring the feature quantity of each of the plurality of specimen images (step S12).

The machine learning function 352 calculates a loss value for each combination of a plurality of feature amounts (step S13).

The machine learning function 352 determines whether a condition for terminating learning is satisfied based on the loss value (step S14). For example, the machine learning function 352 determines whether each loss value has a value that corresponds to the matching degree of the pathological specimen of each combined specimen image.

If the conditions for terminating learning are not met (step S14; No), the machine learning function 352 updates the parameters of the feature extractor (step S15). Then, the machine learning function 352 moves to step S12.

If the conditions for terminating the learning are satisfied (step S14; Yes), the machine learning function 352 terminates the learning process.

As described above, the information processing device 30 according to the first embodiment has a first image obtained by photographing a pathological specimen, and a second image obtained by photographing the pathological specimen under photographing conditions different from the photographing conditions for the first image. Get the image of. The information processing device 30 has a loss value indicating an error between the feature amount of the first image outputted by the feature amount extractor and the feature amount of the second image outputted by the feature amount extractor. The parameters of the feature amount extractor are updated so that the parameters of the feature extractor become values corresponding to the degree of matching between the features of the pathological specimen in the image and the features of the pathological specimen in the second image. In this way, the information processing device 30 causes the feature extractor to learn so that the error between the feature amount of the first image and the feature amount of the second image becomes small. Therefore, the information processing device 30 can reduce differences in feature amounts caused by differences in imaging conditions.

(Second embodiment)
FIG. 4 is a diagram showing an example of the configuration of a pathological information processing system 1a according to the second embodiment.

The processing circuit 350a of the information processing device 30a has a transfer learning function 353.

The transfer learning function 353 applies the feature extractor generated by the machine learning function 352 to the trained model by transfer learning. That is, the transfer learning function 353 updates the parameters of a feature extractor that is a reconfigured feature extractor generated by the machine learning function 352. The transfer learning function 353 is an example of a transfer learning unit. Thereby, the transfer learning function 353 executes transfer learning of the feature extractor generated by the machine learning function 352. For example, the transfer learning function 353 performs transfer learning in which the feature extractor generated by the machine learning function 352 is applied to a neural network that executes a final task such as a classification task or a segmentation task.

Here, the neural network may perform any kind of machine learning. For example, a neural network may classify an input image, output a segmentation result that divides a region for each object included in the input image, or It may be a device that detects objects that appear in the image, or a device that outputs an image generated from an input image.

The transfer learning function 353 reconstructs the feature extractor by applying the feature extractor to the neural network by transfer learning. After transfer learning, the neural network uses a feature extractor that has been trained to extract the same features from images of the same pathological specimen and at the same position, even if the aspect of the specimen image is different. It is highly likely that the ability to extract features has been inherited. Therefore, a neural network after transfer learning can be expected to output the same results when inputted with specimen images of the same pathological specimen and at the same position, even if the specimen images have different aspects. can.

There are two possible methods for learning neural networks. Among the new reconfigured neural network structures, there is a learning method in which the parameters inherited from the feature extractor are fixed (Freeze) and the parameters are updated only for the new structure, and a method in which all parameters are learned. The former is superior in that learning can be completed in a short time and that it is difficult to overfit the training data. On the other hand, the latter is superior in that it is more likely to result in a neural network with higher performance when there is a sufficient amount of training data.

Here, a flow for applying the feature extractor to the neural network will be described. FIG. 5 is a diagram illustrating an example of a flow for applying the first feature extractor E1 to the first neural network M1, which is executed by the information processing device 30a according to the second embodiment.

In the machine learning function 352, as in the first embodiment, the loss value decreases as the degree of matching of the features of the pathological specimen in the specimen image increases, and the degree of matching of the characteristics of the pathological specimen in the specimen image decreases. The first feature extractor E1 is trained so that the loss value increases according to the following.

The transfer learning function 353 executes transfer learning that applies the first feature extractor E1 to the first neural network M1. Thereby, the transfer learning function 353 generates the first neural network M1 having the second feature extractor E2 reconstructed by transfer learning. In addition, in the transfer learning of the transfer learning function 353, some or all of the parameters of the first feature extractor E1 are used as the initial values of the parameters of the second feature extractor E2, but the parameters updated by the transfer learning procedure By selecting , the first neural network M1 may be generated by adjusting the degree of adaptation to the teacher data used for transfer learning. Specifically, in the transfer learning of the transfer learning function 353, if all parameters of the second feature extractor E2 are learned to be updated, there is a high possibility that the second feature extractor E2 is more likely to be adapted to the teacher data used in the transfer learning. On the other hand, in transfer learning of the transfer learning function 353, when learning to update only the parameters of the output layer among the parameters of the second feature extractor E2, the degree of adaptation to the teacher data used for transfer learning decreases. However, it is possible to generate the first neural network M1 with high accuracy using unknown data that is not used for learning.

The first neural network M1 outputs an execution result when the first specimen image G1 is input. Note that in FIG. 5, the first specimen image G1 is input to the first neural network M1 as an example, but any specimen image may be input.

Next, a learning process executed by the information processing device 30a according to the second embodiment will be described. FIG. 6 is a flowchart illustrating an example of a learning process executed by the information processing device 30a according to the second embodiment.

The image acquisition function 351 acquires a plurality of specimen images (step S21).

The machine learning function 352 inputs the plurality of specimen images acquired by the image acquisition function 351 to the feature quantity extractor, thereby acquiring the feature quantity of each of the plurality of specimen images (step S22).

The machine learning function 352 calculates a loss value for each combination of a plurality of feature amounts (step S23).

The machine learning function 352 determines whether a condition for terminating learning is satisfied based on the loss value (step S24).

If the conditions for terminating learning are not met (step S24; No), the machine learning function 352 updates the parameters of the feature extractor (step S25). The machine learning function 352 then proceeds to step S22.

If the conditions for terminating the learning are satisfied (step S24; Yes), the transfer learning function 353 executes transfer learning to apply the feature extractor to the neural network (step S26).

With the above, the information processing device 30a ends the learning process.

As described above, the information processing device 30a according to the second embodiment applies the feature extractor generated by the machine learning function 352 to an arbitrary model by transfer learning. As a result, when two or more specimen images of the same pathological specimen taken at the same position are input, the model to which the feature extractor is applied will produce approximately the same feature values even if the imaging conditions are different. Processing can be performed based on the

(Third embodiment)
FIG. 7 is a diagram showing an example of the configuration of a pathological information processing system 1b according to the third embodiment.

The processing circuit 350b of the information processing device 30b has an image adjustment function 354.

The image adjustment function 354 matches at least one of the position, orientation, and shape of the pathological specimen in the first image and the second image. The image adjustment function 354 is an example of an adjustment section. For example, when specimen images are captured under different imaging conditions, the size of the specimen images and the position, orientation, and shape of the pathological specimen depicted in the specimen images may differ. In such a case, the image adjustment function 354 performs adjustment processing such as movement, enlargement, reduction, rotation, inversion, and keystone correction of the specimen image.

For example, when the target task for transfer learning is a segmentation task, it is often useful if the features extracted by the feature extractor include features related to position and orientation. Therefore, at the stage of learning the feature extractor, it is desirable that adjustments such as positioning of pairs of specimen images are performed before inputting them to the feature extractor.

Therefore, the image adjustment function 354 determines whether the pair of specimen images are images of the same pathological specimen at a close position. For example, the image adjustment function 354 determines whether the paired specimen images are images located close to each other based on whether the characteristics of the pathological specimen have a high degree of matching. Alternatively, the image adjustment function 354 determines whether the image is located closer to the imaging conditions acquired from the first imaging device 10 or the second imaging device 20.

The image adjustment function 354 executes adjustment processing such as alignment when the paired specimen images are images of the same pathological specimen at close positions. For example, the image adjustment function 354 performs adjustment processing such as movement, enlargement, reduction, rotation, inversion, and trapezoidal correction of paired specimen images.

The machine learning function 352b reduces the loss value as the degree of matching of the features of the pathological specimen of the image acquisition function 351, the acquired specimen image, and the specimen image on which the image adjustment function 354 has performed adjustment processing such as alignment increases. The first feature extractor E1 is trained so that the loss value increases as the degree of matching of the features of the pathological specimen in the specimen image decreases.

Here, a flow for executing adjustment processing such as alignment with respect to a specimen image will be described. FIG. 8 is a diagram illustrating an example of a flow of adjustment such as positioning of a specimen image performed by the information processing device 30b according to the third embodiment.

The image acquisition function 351 acquires, for example, a first specimen image G1 and a third specimen image G3 photographed by the first imaging device 10. Further, the image acquisition function 351 acquires, for example, the second specimen image G2 photographed by the second photographing device 20.

The image adjustment function 354 determines whether the first specimen image G1, the second specimen image G2, and the third specimen image G3 acquired by the image acquisition function 351 are images of the same pathological specimen at close positions. judge.

In the flow shown in FIG. 8, the image adjustment function 354 determines whether the first specimen image G1 and the second specimen image G2 acquired by the image acquisition function 351 are images of the same pathological specimen at close positions. Furthermore, the image adjustment function 354 determines that there is no image of the same pathological specimen in a nearby position in the third specimen image G3. The image adjustment function 354 executes adjustment processing such as alignment of the pathological specimen between the first specimen image G1 and the second specimen image G2.

The first feature extractor E1 receives the first specimen image G1 and the second specimen image G2 on which the adjustment process has been performed. Furthermore, since it is determined that the image is not a close-position image of the same pathological specimen, the first feature amount extractor E1 receives the third specimen image G3 on which the adjustment process has not been performed.

As in the first embodiment, the machine learning function 352b has a loss value that decreases as the degree of matching of the features of the pathological specimen in the specimen image increases, and the degree of matching of the characteristics of the pathological specimen in the specimen image decreases. The first feature extractor E1 is trained so that the loss value increases according to the following.

Next, a learning process executed by the information processing device 30b according to the third embodiment will be described. FIG. 9 is a flowchart illustrating an example of a learning process executed by the information processing device 30b according to the third embodiment.

The image acquisition function 351 acquires a plurality of specimen images (step S31).

The image adjustment function 354 determines whether or not a pair of specimen images selected from a plurality of specimen images are images of the same pathological specimen at close positions (step S32). If the images are not images of the same pathological specimen at a close position (step S32; No), the image adjustment function 354 moves to step S34.

In the case of images of the same pathological specimen at close positions (step S32; Yes), the image adjustment function 354 performs adjustment processing such as alignment on the pair of specimen images (step S33).

The machine learning function 352b inputs the sample image to the feature extractor to acquire the feature amount of each of the plurality of sample images (step S34).

The machine learning function 352b calculates a loss value for each combination of a plurality of feature amounts (step S35).

The machine learning function 352b determines whether a condition for terminating learning is satisfied based on the loss value (step S36).

If the conditions for terminating learning are not met (step S36; No), the machine learning function 352b updates the parameters of the feature extractor (step S37). The machine learning function 352b then proceeds to step S34.

If the conditions for terminating the learning are satisfied (step S36; Yes), the information processing device 30b terminates the learning process.

As described above, the information processing device 30b according to the third embodiment matches at least one of the position, orientation, and shape of the pathological specimen in the first image and the second image. Therefore, in the learning stage, the information processing device 30b can handle a model that is better learned using a pair of aligned sample images.

(Fourth embodiment)
FIG. 10 is a diagram showing an example of the configuration of a pathological information processing system 1c according to the fourth embodiment.

The processing circuit 350c of the information processing device 30c has an image adjustment function 354 and a cutting function 355.

Similar to the image adjustment function 354 according to the third embodiment, the image adjustment function 354 executes adjustment processing such as alignment when the paired sample images are images of the same pathological sample at close positions. . That is, the image adjustment function 354 performs adjustment processing such as movement, enlargement, reduction, rotation, inversion, and trapezoidal correction of paired specimen images.

Here, the image adjustment function 354 needs to control whether the paired partial area images include the same features or different features with respect to the partial area to be cut out by the cutting function 355, so the image adjustment function 354 performs alignment before cutting out. Perform adjustment processing such as In this way, the image adjustment function 354 can simplify the steps of the cutting process by keeping the positional relationship of the pathological specimen consistent between the respective specimen images.

The cutting function 355 cuts out a partial image from an image. The cutting function 355 is an example of a cutting unit that cuts out. The partial image is a sample image obtained by cutting out a portion of the sample image.

Here, the positional relationship of the pathological specimens in the paired specimen images is adjusted by the image adjustment function 354. Therefore, the cropping function 355 can acquire a pair of partial images that include the same features by cropping partial images at the same position and the same size of the specimen images to be paired. On the other hand, the cropping function 355 can acquire a pair of partial images including different features by cropping partial images at different positions or different sizes of the paired specimen images.

Furthermore, the cutout function 355 calculates and stores in advance parameters used for adjustment processing such as position alignment when the positional relationship of the pathological specimens in paired specimen images is not adjusted by the image adjustment function 354. Then, the cutout function 355 cuts out a partial image using the stored parameters when executing the cutout process. Thereby, the cutout function 355 can obtain a pair of partial images that include the same feature or a pair of partial images that include different features.

The machine learning function 352c uses the feature amount of the first partial image, which is a part of the first image outputted by the feature amount extractor, and the second image, which is a part of the second image outputted by the feature amount extractor. The loss value indicating the error with the feature amount of the partial image is set to a value corresponding to the degree of matching between the features of the pathological specimen in the first partial image and the characteristics of the pathological specimen in the second partial image. , update the parameters of the feature extractor.

Here, a flow of cutting out a sample image will be described. FIG. 11 is a diagram illustrating an example of a sample image extraction flow executed by the information processing device 30c according to the fourth embodiment.

The image acquisition function 351 acquires, for example, a first specimen whole image G4 photographed by the first photographing device 10 and a second specimen whole image G5 photographed by the second photographing device 20. The first entire specimen image G4 and the second entire specimen image G5 are images in which substantially the entire pathological specimen is depicted. In FIG. 11, the first whole specimen image G4 and the second whole specimen image G5 are images of the same pathological specimen but have different magnification ratios, but they may also be images taken of different pathological specimens. .

The image adjustment function 354 determines whether the first whole specimen image G4 and the second whole specimen image G5 are the same pathological specimen. In other words, the image adjustment function 354 determines whether or not the image is to be subjected to adjustment processing such as alignment. When it is determined that the first whole specimen image G4 and the second whole specimen image G5 are the same pathological specimen, the image adjustment function 354 performs the following on the first whole specimen image G4 and the second whole specimen image G5: Perform adjustment processing such as alignment. If it is determined that the first whole specimen image G4 and the second whole specimen image G5 are not the same pathological specimen, the image adjustment function 354 aligns the first whole specimen image G4 and the second whole specimen image G5. Do not perform adjustment processing such as

The cropping function 355 can be used to extract a first specimen whole image G4 and a second specimen whole image G5 in which the same pathological specimen has been photographed and adjustment processing such as alignment has been performed, or a second specimen whole image G5 in which a different pathological specimen has been photographed. A partial image is cut out from the first specimen whole image G4 and the second specimen whole image G5. The first specimen image G1 is cut out as a partial image from the first specimen whole image G4 on which adjustment processing such as alignment has been performed, and as a partial image from the second specimen whole image G5 on which adjustment processing such as registration has been performed. , when the second specimen image G2 is cut out from the same position as the first specimen image G1, the features of the pair of the first specimen image G1 and the second specimen image G2 match, so the first feature amount extractor E1 are positive example pairs from which substantially the same feature quantities should be extracted. When the third specimen image G3 is cut out from a different position from the second specimen image G2 as a partial image from the first specimen whole image G4 that has undergone adjustment processing such as alignment, the second specimen image G2 and the third specimen image G3 Since the pair with image G3 does not match in features, it becomes a negative example pair from which the first feature amount extractor E1 should extract different feature amounts. When the first specimen image G1 is cut out from the first whole specimen image G4 in which different pathological specimens are photographed, and the first specimen image G1 is cut out from the second whole specimen image G5, the first specimen image G1 and the second Since the pair with the specimen image G2 does not match in features, it becomes a negative example pair from which the first feature amount extractor E1 should extract different feature amounts.

As in the first embodiment, the machine learning function 352c has a loss value that decreases as the degree of matching of the features of the pathological specimen in the specimen image increases, and the degree of matching of the characteristics of the pathological specimen in the specimen image decreases. The first feature extractor E1 is trained so that the loss value increases according to the following.

Next, a learning process executed by the information processing device 30c according to the fourth embodiment will be described. FIG. 12 is a flowchart illustrating an example of a learning process executed by the information processing device 30c according to the fourth embodiment.

The image acquisition function 351 acquires a plurality of specimen images (step S41).

The image adjustment function 354 determines whether or not a pair of specimen images selected from a plurality of specimen images are images of the same pathological specimen (step S42). If the images are not of the same pathological specimen (step S42; No), the image adjustment function 354 moves to step S45.

In the case of images of the same pathological specimen (step S42; Yes), the image adjustment function 354 performs adjustment processing such as alignment on the pair of specimen images (step S43).

The cutout function 355 cuts out a partial image from a sample image that has undergone adjustment processing such as alignment, or a sample image obtained by photographing a different sample (step S44).

The machine learning function 352c acquires the feature amount of each of the plurality of partial images by inputting the partial image as a sample image to the feature amount extractor (step S45).

The machine learning function 352c calculates a loss value for each combination of a plurality of feature amounts (step S46).

The machine learning function 352c determines whether a condition for terminating learning is satisfied based on the loss value (step S47).

If the conditions for terminating learning are not met (step S47; No), the machine learning function 352c updates the parameters of the feature extractor (step S48). The machine learning function 352c then proceeds to step S45.

If the conditions for terminating the learning are satisfied (step S47; Yes), the information processing device 30c terminates the learning process.

As described above, the information processing device 30c according to the fourth embodiment matches at least one of the position, orientation, and shape of the pathological specimen in the first image and the second image. The information processing device 30c cuts out a portion of the aligned image. At this time, the information processing device 30c can obtain a pair of partial images including the same features by cutting out the images from the same position. Further, the information processing device 30c can obtain a pair of partial images including different features by cutting out images from different positions. Therefore, the information processing device 30c can easily collect images used for learning.

(Fifth embodiment)
FIG. 13 is a diagram showing an example of the configuration of a pathological information processing system 1d according to the fifth embodiment.

The processing circuit 350d of the information processing device 30d has an image generation function 356.

The image generation function 356 generates a specimen image from the specimen image acquired by the image acquisition function 351. Further, the image generation function 356 may generate a sample image from the partial sample image cut out by the cutout function 355. That is, the image generation function 356 generates a specimen image from at least one of the specimen image acquired by the image acquisition function 351 and the partial specimen image cut out by the cutting function 355. The image generation function 356 is an example of a generation unit.

More specifically, the image generation function 356 generates a specimen image through data augmentation processing. That is, the image generation function 356 generates a specimen image by performing geometric operations such as rotation, movement, scaling, deformation, inversion, and keystone correction. Alternatively, the image generation function 356 generates a specimen image through image processing operations such as blurring, noise addition, brightness correction, contrast correction, and gamma correction.

Furthermore, if the same geometric operation is not performed on a pair of specimen images for which a correlation value is to be calculated, the structural features such as the position and orientation of the pathological specimen will change, and therefore the characteristics of the feature amount extractor will change. Specifically, when a specimen image is generated by geometric operations, the feature values used to determine whether the position and orientation of the pathological specimens are the same may be lost from the feature values output by the feature extractor. Highly sexual.

In a neural network where a feature extractor is applied using transfer learning, if features such as the position and orientation of the pathological specimen are not required, the feature extractor uses a sample image generated by geometric operations. This may improve the final robustness.

For example, in classification tasks, feature quantities such as the position and orientation of pathological specimens are often unnecessary. On the other hand, in segmentation tasks, feature quantities such as the position and orientation of pathological specimens are often required.

In this way, the task of the neural network to which the feature extractor is applied determines whether a specimen image is generated using a geometric operation, an image processing operation, or a combination of a geometric operation and an image processing operation. determined by.

Therefore, the image generation function 356 generates a specimen image from the specimen image acquired by the image acquisition function 351 by a method according to the operation, setting, etc. indicating the task of the neural network to which the feature extractor is applied.

Similar to the first embodiment, the machine learning function 352d has a loss value that decreases as the degree of matching of the features of the pathological specimen in the specimen image increases, and the degree of matching of the characteristics of the pathological specimen in the specimen image decreases. The feature extractor is trained so that the loss value increases according to .

If the characteristics of the specimen image generated by the image generation function 356 have changed from the characteristics of the pathological specimen of the specimen image that is the source of this specimen image, the machine learning function 352d changes the characteristics of the pathological specimen of the specimen image. The feature quantity extractor is trained so that the loss value becomes smaller as the degree of matching of the features of the pathological specimen in the specimen image becomes higher, and the loss value becomes larger as the degree of matching of the features of the pathological specimen in the specimen image becomes lower.

Note that even if the characteristics of the specimen image generated by the image generation function 356 are different from the characteristics of the pathological specimen of the specimen image that is the source of this specimen image, the pathological specimen is the same and If the positions are the same, the machine learning function 352d may cause the feature extractor to learn so that the loss value is small. That is, the machine learning function 352d may update the parameters of the feature amount extractor so that the loss value becomes smaller.

For example, the image generation function 356 generates a new specimen image by rotating the specimen image. In this case, when the original unrotated specimen image and the rotated specimen image are input, the feature extractor extracts approximately the same feature amounts from each of them because the pathological specimens are the same and the positions are the same. Output. That is, the feature extractor outputs a feature that does not include information regarding the orientation of the pathological specimen.

A neural network to which this feature extractor is applied by transfer learning has robustness against rotation of the specimen image. That is, the neural network outputs the processing results without considering the rotation of the specimen image.

However, the orientation of the specimen image is an important feature depending on the task of the neural network.

Therefore, the machine learning function 352d makes the feature extractor learn so that the loss value is small, or the loss value is large, depending on the operation or settings indicating the task of the neural network to which the feature extractor is applied. Train the feature extractor to

In other words, the machine learning function 352d recognizes the pathological specimen even if the characteristics of the specimen image generated by the image generation function 356 have changed from the characteristics of the pathological specimen of the specimen image that is the source of this specimen image. are the same and the positions are the same, whether or not to reduce the loss value is determined according to operations, settings, etc.

Here, a flow for executing sample image generation will be described. FIG. 14 is a diagram illustrating an example of a flow for generating a specimen image executed by the information processing device 30d according to the fifth embodiment.

The image acquisition function 351 acquires, for example, a first specimen image G1 photographed by the first imaging device 10 and a second specimen image G2 photographed by the second imaging device 20.

The image generation function 356 copies the first specimen image G1 and generates a fourth specimen image G6 by rotating the copied image.

The machine learning function 352d is configured such that the loss value becomes smaller as the degree of matching of the features of the pathological specimen in the specimen image becomes higher, and the loss value becomes larger as the degree of matching of the features of the pathological specimen in the specimen image becomes lower. 1. Train the feature extractor E1.

The machine learning function 352d determines whether the pathological specimens are the same even if the characteristics of the specimen image generated by the image generation function 356 have changed from the characteristics of the pathological specimen of the specimen image that is the source of this specimen image. , and when the positions are the same, whether or not to reduce the loss value is determined according to operations, settings, etc.

In other words, the machine learning function 352d extracts the first feature Z1 generated from the first specimen image G1 and the third specimen image according to the operation, setting, etc. indicating the task of the neural network to which the feature extractor is applied. The parameters of the first feature extractor E1 are updated so that the second loss value L2 corresponding to the third feature Z3 generated from G3 is made smaller or larger.

Next, a learning process executed by the information processing device 30d according to the fifth embodiment will be described. FIG. 15 is a flowchart illustrating an example of a learning process executed by the information processing device 30d according to the fifth embodiment.

The image acquisition function 351 acquires a plurality of specimen images (step S51).

The image generation function 356 generates one or more specimen images by data expansion from the specimen image acquired by the image acquisition function 351 (step S52).

The machine learning function 352d inputs the plurality of specimen images acquired by the image acquisition function 351 and the specimen image generated by the image generation function 356 to the feature quantity extractor, thereby extracting the characteristic quantities of each of the plurality of specimen images. Acquire (step S53).

The machine learning function 352d calculates a loss value for each combination of a plurality of feature amounts (step S54).

The machine learning function 352d determines whether a condition for terminating learning is satisfied based on the loss value (step S55).

If the conditions for terminating learning are not met (step S55; No), the machine learning function 352d updates the parameters of the feature extractor (step S56). The machine learning function 352d then proceeds to step S53.

If the conditions for terminating the learning are satisfied (step S55; Yes), the machine learning function 352d terminates the learning process.

As described above, the information processing device 30d according to the fifth embodiment generates an image by data expansion from an image photographed by the first photographing device 10 or the second photographing device 20. Therefore, the information processing device 30d can easily collect images used for learning.

(Modification 1)
It has been explained that the image acquisition function 351 acquires specimen images from the first imaging device 10 and the second imaging device 20. However, the pathological information processing systems 1, 1a, 1b, 1c, and 1d do not need to include the second imaging device 20. In this case, the image acquisition function 351 may acquire a plurality of specimen images under different imaging conditions from the first imaging device 10.

(Modification 2)
It has been explained that the first photographing device 10 and the second photographing device 20 are cameras or scanners. However, the first imaging device 10 and the second imaging device 20 are an OCT (Optical Coherence Tomography) device, a MicroCT (Computed Tomography) device, a MicroMRI (Magnetic Resonance Imaging) device, which can capture a three-dimensional tomographic image of a specimen, Alternatively, it may be an X-ray CT device, an MRI (Magnetic Resonance Imaging) device, or the like. That is, the information processing devices 30, 30a, 30b, 30c, and 30d are capable of processing not only two-dimensional images such as specimen images in the field of pathology, but also three-dimensional images such as CT images and MRI images. good.

(Modification 2)
The information processing devices 30, 30a, 30b, 30c, and 30d have an image acquisition function 351, a machine learning function 352, 352b, 352c, and 352d, a transfer learning function 353, an image adjustment function 354, a cutting function 355, and an image generation function 356. I explained that I would be prepared. However, all or part of these functional units may be included in devices other than the information processing devices 30, 30a, 30b, 30c, and 30d. In this case, the information processing devices 30, 30a, 30b, 30c, and 30d may transmit the sample images to other devices.

(Modification 3)
The information processing devices 30, 30a, 30b, 30c, and 30d execute the programs stored in the storage circuit 340 to provide an image acquisition function 351, a machine learning function 352, 352b, 352c, and 352d, a transfer learning function 353, It has been explained that the image adjustment function 354, the cutout function 355, and the image generation function 356 are realized. However, the information processing devices 30, 30a, 30b, 30c, and 30d have an image acquisition function 351, a machine learning function 352, 352b, 352c, and 352d, a transfer learning function 353, an image adjustment function 354, a cutting function 355, and an image generation function. All or part of 356 may be realized by hardware such as a semiconductor circuit.

According to at least one embodiment described above, it is possible to reduce differences in feature amounts caused by differences in imaging conditions.

Although several embodiments have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, changes, and combinations of embodiments can be made without departing from the gist of the invention. These embodiments and their modifications are included within the scope and gist of the invention as well as within the scope of the invention described in the claims and its equivalents.

1, 1a, 1b, 1c, 1d pathological information processing system 10 first imaging device 20 second imaging device 30, 30a, 30b, 30c, 30d information processing device 40 network 310 NW (network) interface 320 input interface 330 display 340 Memory circuit 350, 350a, 350b, 350c, 350d Processing circuit 351 Image acquisition function 352, 352b, 352c Machine learning function 353 Transfer learning function 354 Image adjustment function 355 Cutting function 356 Image generation function E1 First feature extractor E2 2 feature extractor G1 First sample image G2 Second sample image G3 Third sample image G4 First sample entire image G5 Second sample entire image G6 Fourth sample image Z1 First feature Z2 Second feature Z3 Third Feature L1 First loss value L2 Second loss value L3 Third loss value M1 First neural network

Claims (12)

an image acquisition unit that acquires a first image of a pathological specimen and a second image of the pathological specimen under imaging conditions different from imaging conditions of the first image;
A feature amount of the first image outputted by a first feature amount extractor that outputs a feature amount representing the feature of the image, and a feature amount of the second image outputted by the first feature amount extractor. the first image so that the loss value indicating the error with respect to the amount of a machine learning unit that updates parameters of the feature extractor;
An information processing device comprising: The machine learning unit is configured to calculate the loss value from the first image so that the loss value decreases as the degree of coincidence between the characteristics of the pathological specimen in the first image and the characteristics of the pathological specimen in the second image increases. Update the parameters of the feature extractor of
The information processing device according to claim 1. The machine learning unit controls the first feature extractor so that the loss value decreases as the position of the pathological specimen in the first image and the position of the pathological specimen in the second image become closer. update the parameters of,
The information processing device according to claim 1. further comprising a transfer learning unit that updates parameters of a second feature extractor that is the reconfigured first feature extractor;
The information processing device according to claim 1. The neural network having the second feature extractor extracts the execution result of classifying the input image, the segmentation result of dividing the area for each object included in the input image, and the detection result of detecting the object included in the input image. , or output an image generated from an input image,
The information processing device according to claim 4. further comprising an adjustment unit that adjusts at least one of the position, orientation, and shape of the pathological specimen in the first image and the second image;
The information processing device according to claim 1. It further includes a cutting section that cuts out a part of the image,
The machine learning unit is configured to calculate the feature amount of a first partial image that is a part of the first image output by the first feature amount extractor, and the feature amount of the first partial image that is a part of the first image output by the first feature amount extractor. The loss value between the feature amount of the second partial image, which is a part of the second image, is the difference between the feature of the pathological specimen of the first partial image and the characteristic of the pathological specimen of the second partial image. updating the parameters of the first feature extractor so that the values correspond to the degree of matching;
The information processing device according to claim 1. At least one of the first image, the second image, a first partial image that is a part of the first image, and a second partial image that is a part of the second image. further comprising a generation unit that generates an image;
The information processing device according to claim 1. The first image and the second image are images taken at the same position of the same pathological specimen;
The information processing device according to claim 1. When the first image and the second image are a combination of images taken at the same position of the same pathological specimen, the machine learning unit determines that when the first image and the second image are a combination of images taken at the same position of the same pathological specimen, the machine learning unit updating the parameters so that the loss value becomes smaller;
The information processing device according to claim 1. A first image obtained by photographing a pathological specimen and a second image obtained by photographing the pathological specimen under photographing conditions different from the photographing conditions for the first image, and outputting a feature amount representing the characteristics of the image. a loss value indicating an error between the feature amount of the first image outputted by the first feature amount extractor and the feature amount of the second image outputted by the first feature amount extractor; updating the parameters of the first feature extractor so that the parameters of the first feature extractor become values corresponding to the degree of matching between the features of the pathological specimen in the first image and the characteristics of the pathological specimen in the second image;
Information processing methods that include computer,
an image acquisition unit that acquires a first image of a pathological specimen and a second image of the pathological specimen under imaging conditions different from imaging conditions of the first image;
A feature amount of the first image outputted by a first feature amount extractor that outputs a feature amount representing the feature of the image, and a feature amount of the second image outputted by the first feature amount extractor. the first image so that the loss value indicating the error with respect to the amount of a machine learning unit that updates parameters of the feature extractor;
A program to make it work.

Images