Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/0002—Inspection of images, e.g. flaw detection
  • G06T7/0012—Biomedical image inspection
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/762—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using clustering, e.g. of similar faces in social networks
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
  • G06V10/7715—Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H10/00—ICT specially adapted for the handling or processing of patient-related medical or healthcare data
  • G16H10/40—ICT specially adapted for the handling or processing of patient-related medical or healthcare data for data related to laboratory analysis, e.g. patient specimen analysis
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H30/00—ICT specially adapted for the handling or processing of medical images
  • G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H30/00—ICT specially adapted for the handling or processing of medical images
  • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
  • G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
  • G16H40/67—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for remote operation
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
  • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
  • G16H50/70—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for mining of medical data, e.g. analysing previous cases of other patients
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10056—Microscopic image
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20081—Training; Learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20084—Artificial neural networks [ANN]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30004—Biomedical image processing
  • G06T2207/30024—Cell structures in vitro; Tissue sections in vitro

Definitions

• the technology of the present disclosure relates to an image processing device, an operating method for the image processing device, and an operating program for the image processing device.
• U.S. Patent Application Publication No. 2021/0342570 discloses that a machine learning model such as an autoencoder is used to extract feature quantities from patch images obtained by segmenting a specimen image of a tissue specimen such as an animal's liver. Based on the extracted features, determine whether morphological abnormalities (hyperplasia, infiltration, stasis, inflammation, tumor, canceration, proliferation, bleeding, glycogen reduction, etc.) have occurred in the tissue specimen shown in the patch image.
• morphological abnormalities hyperplasia, infiltration, stasis, inflammation, tumor, canceration, proliferation, bleeding, glycogen reduction, etc.
• patch images determined to have a morphological abnormality are clustered into one of a plurality of clusters (hard clustering) based on feature amounts.
• One embodiment of the technology of the present disclosure provides an image processing device capable of clustering patch images of sample images in which the distribution of feature values is not discrete, an operating method for the image processing device, and an operating program for the image processing device. I will provide a.
• the image processing device of the present disclosure includes a processor, the processor acquires a first specimen image in which a tissue specimen of a subject is photographed, and uses a machine learning model to create a first patch obtained by subdividing the first specimen image. A first feature quantity is extracted from the image, and based on the first feature quantity, it is determined whether or not a morphological abnormality has occurred in the tissue specimen shown in the first patch image, and it is determined whether a morphological abnormality has occurred in the tissue specimen.
• a manual clustering process that receives a user's designation as to which cluster the first patch image that has been determined to belong to, and clusters the first patch image into multiple clusters based on the designation; Any one of soft clustering processing is performed to calculate the degree of belonging of the first patch image determined to have occurred to each of a plurality of clusters.
• the processor performs control to display the results of the manual clustering process or the results of the soft clustering process.
• the results are displayed as multiple cluster images generated by processing the first sample image, and the multiple cluster images are made distinguishable by a display format set in advance for each cluster.
• a display format set in advance for each cluster Preferably it is an image.
• the processor displays at least one of the plurality of cluster images in a superimposed manner on the first sample image.
• the processor receives from the user a designation of a cluster image to be superimposed and displayed on the first sample image.
• the processor displays statistical information based on the results.
• the processor reduces the number of dimensions of the first feature amount to two or three dimensions, displays a graph in which the first feature amount with the reduced number of dimensions is plotted in a two-dimensional space or a three-dimensional space, It is preferable to accept specifications on the graph.
• the machine learning model is based on a second patch that is obtained by subdividing a second sample image of tissue samples from multiple subjects that constituted a control group to which no candidate substance was administered in past evaluation tests of drug candidate substances.
• the model is trained using images as training data.
• the training data also includes a patch image of a tissue specimen in which a morphological abnormality has occurred.
• the machine learning model is a model that is responsible for the task of identifying the type of morphological abnormality.
• the processor obtains information on the distribution of the second feature extracted using the machine learning model from the second patch image obtained by subdividing the second sample image, calculates the distance between the distribution and the first feature, It is preferable to make the determination based on distance.
• the operating method of the image processing device of the present disclosure includes acquiring a first specimen image in which a tissue specimen of a subject is photographed, and using a machine learning model to create a first patch image obtained by subdividing the first specimen image. determining whether or not a morphological abnormality has occurred in the tissue sample appearing in the first patch image based on the first feature amount; A manual clustering process that receives a user's designation as to which cluster the first patch image that has been determined to belong to, and clusters the first patch image into multiple clusters based on the designation, and detects morphological abnormalities in the tissue specimen.
• the method includes performing any one of soft clustering processing for calculating the degree of belonging of the first patch image, which is determined to have occurred, to each of a plurality of clusters.
• the operating program of the image processing device of the present disclosure includes acquiring a first specimen image in which a tissue specimen of a subject is photographed, and using a machine learning model to create a first patch image obtained by subdividing the first specimen image. determining whether or not a morphological abnormality has occurred in the tissue sample appearing in the first patch image based on the first feature amount; A manual clustering process that receives a user's designation as to which cluster the first patch image that has been determined to belong to, and clusters the first patch image into multiple clusters based on the designation, and detects morphological abnormalities in the tissue specimen.
• the computer is caused to perform a process including performing any one of soft clustering processes for calculating the degree of belonging of a first patch image determined to have occurred to each of a plurality of clusters.
• an image processing device an operating method for the image processing device, and an operating program for the image processing device that can cluster patch images of specimen images in which the distribution of feature values is not discrete. Can be done.
• FIG. 2 is a block diagram showing a computer that constitutes an image processing device.
• FIG. 2 is a block diagram showing a processing unit of a CPU of the image processing device. It is a figure which shows the 1st patch image which subdivided the 1st specimen image.
• FIG. 6 is a diagram showing how a first feature quantity is extracted from a first patch image by a feature quantity extractor.
• FIG. 2 is a diagram showing the structure of a feature quantity extractor.
• FIG. 3 is a diagram showing processing in a learning phase of an autoencoder.
• FIG. 7 is a diagram showing the formation of a past control group and a second patch image for learning.
• FIG. 7 is a diagram showing how a second feature is extracted from a second patch image by a feature extractor.
• FIG. 7 is a diagram showing a graph in which a second feature quantity is plotted in a feature quantity space, and second feature quantity distribution information.
• FIG. 7 is a diagram showing a distance between a position of a first feature amount and a representative position of a second feature amount. It is a figure which shows the process of a determination part. It is a figure which shows the process of a determination part.
• FIG. 3 is a diagram showing a detailed configuration of a clustering processing section. It is a figure showing a target designation screen. It is a figure which shows the process of a dimension reduction part. It is a figure showing a cluster designation screen.
• FIG. 7 is a diagram showing how a second feature is extracted from a second patch image by a feature extractor.
• FIG. 7 is a diagram showing a graph in which a second feature quantity is plotted in a feature quantity space, and second
• FIG. 7 is a diagram illustrating the process of the designation reception unit when the drug discovery staff designates a cluster on the cluster designation screen.
• FIG. 3 is a diagram showing a cluster image and a superimposed image. It is a figure showing a clustering result display screen.
• FIG. 6 is a diagram showing a clustering result display screen on which a superimposed image in which only one cluster image is superimposed on the target first sample image is displayed.
• 3 is a flowchart showing a processing procedure of the image processing device.
• 3 is a flowchart showing a processing procedure of the image processing device. It is a figure which shows the process of the dimension reduction part of a modification. It is a figure which shows the cluster designation screen of a modification.
• FIG. 6 is a diagram illustrating processing of a membership degree calculation unit and a clustering information generation unit. It is a flowchart which shows the processing procedure of the image processing device of 1st_2nd embodiment. It is a figure which shows the clustering result display screen of 1st_2 embodiment.
• FIG. 3 is a diagram showing a clustering result display screen on which statistical information is displayed.
• FIG. 7 is a diagram illustrating processing in a learning phase of an autoencoder according to a third embodiment. It is a figure which shows the structure of a past administration group and the 3rd patch image for learning.
• FIG. 6 is a diagram illustrating processing of a membership degree calculation unit and a clustering information generation unit. It is a flowchart which shows the processing procedure of the image processing device of 1st_2nd embodiment. It is a figure which shows the clustering result display screen of 1st_2 embodiment.
• FIG. 3 is a diagram showing a clustering result display screen on which statistical information is displayed.
• FIG. 7 is a diagram showing the structure of a feature extractor according to a fourth embodiment.
• FIG. 12 is a diagram showing a fifth embodiment in which specimen images obtained by photographing a slide specimen on which tissue specimens of multiple types of organs are mounted are handled.
• FIG. 6 is a diagram illustrating how a first feature quantity is extracted from a first patch image obtained by subdividing a heart sample by a feature quantity extractor for a heart sample.
• an image processing apparatus 10 of the present disclosure is used to evaluate the efficacy and toxicity of a drug candidate substance 11.
• the image processing device 10 is, for example, a desktop personal computer, and includes a display 12 that displays various screens, and an input device 13 such as a keyboard, a mouse, a touch panel, and/or a microphone for voice input.
• the image processing device 10 is installed, for example, in a drug development facility, and is operated by drug discovery staff DS who are involved in drug development at the drug development facility.
• the drug discovery staff DS also includes pathologists.
• the drug discovery staff DS is an example of a "user" according to the technology of the present disclosure.
• a first specimen image 151 is input to the image processing device 10.
• the first specimen image 151 is an image for evaluating the medicinal efficacy and toxicity of the candidate substance 11.
• the first specimen image 151 is generated, for example, by the following procedure. First, a subject S such as a rat prepared for the evaluation of candidate substance 11 is autopsied, and multiple tissue specimens of a cross section of the organ of the subject S, here the liver LV (hereinafter referred to as liver specimen) LVS. Collect. Next, after pasting the collected liver specimens LVS one by one onto a slide glass 16, the liver specimens LVS are stained, here, with hematoxylin and eosin dye.
• the stained liver specimen LVS is covered with a cover glass 17 to complete a slide specimen 18.
• the slide specimen 18 is set on a photographing device 19 such as a digital optical microscope, and the first specimen image 151 is photographed by the photographing device 19.
• the first specimen image 151 obtained in this way includes a subject ID (Identification Data) for uniquely identifying the subject S, a specimen image ID for uniquely identifying the first specimen image 151, and the photographing date and time. is attached.
• a tissue specimen is also called a tissue section.
• the staining may be performed by staining with hematoxylin dye alone, staining with nuclear fast red dye, or the like.
• the administration group is composed of a plurality of subjects S to whom the candidate substance 11 has been administered.
• the control group is comprised of a plurality of subjects S to whom the candidate substance 11 was not administered, contrary to the administration group.
• the number of subjects S constituting the administration group and the number of subjects S constituting the control group are both, for example, about 5 to 10.
• the subjects S constituting the administration group and the subjects S constituting the control group have the same attributes and are placed under the same breeding environment.
• the same attributes include, for example, the same age in weeks and/or the same gender.
• the same attributes also include having the same age composition ratio and/or the same sex composition ratio (for example, 5 males and 5 females).
• the same breeding environment means, for example, that the feed is the same, the temperature and humidity of the breeding space are the same, and/or the size of the breeding space is the same.
• “Same” in the same rearing environment means not only the exact same, but also includes errors that are generally acceptable in the technical field to which the technology of the present disclosure belongs and that do not go against the spirit of the technology of the present disclosure. Refers to the same meaning.
• the dosage of the candidate substance 11 is different.
• the dosage of the candidate substance 11 is varied in three stages: high dosage group, medium dosage group, and small dosage group. In this way, the influence on the subject S due to the dose of the candidate substance 11 can be determined.
• the first specimen image 151 may be an image showing the liver specimen LVS of the subject S in the administration group, or may be an image showing the liver specimen LVS of the subject S in the control group. In FIG. 1, a case is illustrated in which the first specimen image 151 is an image in which the liver specimen LVS of the subject S in the administration group is photographed.
• the computer configuring the image processing apparatus 10 includes a storage 30, a memory 31, a CPU (Central Processing Unit) 32, and a communication section 33 in addition to the display 12 and input device 13 described above. We are prepared. These are interconnected via a bus line 34.
• the storage 30 is a hard disk drive built into the computer that constitutes the image processing apparatus 10 or connected through a cable or network.
• the storage 30 is a disk array in which a plurality of hard disk drives are connected in series.
• the storage 30 stores control programs such as an operating system, various application programs, and various data accompanying these programs. Note that a solid state drive may be used instead of the hard disk drive.
• the memory 31 is a work memory for the CPU 32 to execute processing.
• the CPU 32 loads the program stored in the storage 30 into the memory 31 and executes processing according to the program. Thereby, the CPU 32 centrally controls each part of the computer.
• the CPU 32 is an example of a "processor" according to the technology of the present disclosure. Note that the memory 31 may be built into the CPU 32.
• the communication unit 33 controls transmission of various information to and from external devices such as the photographing device 19.
• an operation program 40 is stored in the storage 30 of the image processing device 10.
• the operating program 40 is an application program for causing a computer to function as the image processing device 10. That is, the operating program 40 is an example of "an operating program for an image processing device" according to the technology of the present disclosure.
• the storage 30 also stores a feature extractor 41, second feature distribution information 42, and the like.
• the feature extractor 41 is an example of a "machine learning model" according to the technology of the present disclosure.
• the second feature quantity distribution information 42 is an example of "second feature quantity distribution information" according to the technology of the present disclosure.
• the CPU 32 of the computer constituting the image processing device 10 cooperates with the memory 31 and the like to control the read/write (hereinafter abbreviated as RW) control unit 50 and feature extraction. It functions as a section 51, a determining section 52, and a clustering processing section 53.
• RW read/write
• the RW control unit 50 controls storage of various data in the storage 30 and reading of various data in the storage 30.
• the RW control unit 50 stores the first specimen image 151 from the imaging device 19 in the storage 30. Note that since a plurality of first specimen images 151 are actually obtained from one subject S, a plurality of first specimen images 151 for one subject S are stored in the storage 30.
• the RW control unit 50 acquires the first specimen image 151 according to the designation of the drug discovery staff DS through the input device 13 by reading it from the storage 30.
• the RW control unit 50 outputs the read first sample image 151 to the feature amount extraction unit 51 and the clustering processing unit 53.
• the first sample image 151 outputted from the RW control unit 50 to the feature extracting unit 51 and the like is a target for determining whether or not a morphological abnormality has occurred in the liver sample LVS.
• the first specimen image 151 of the target for determining whether or not morphological abnormality has occurred in the liver specimen LVS will be referred to as the target first specimen image 151T (see FIG. 4, etc.).
• the RW control unit 50 reads the feature extractor 41 from the storage 30 and outputs the read feature extractor 41 to the feature extractor 51. Further, the RW control unit 50 acquires the second feature amount distribution information 42 by reading it from the storage 30. The RW control unit 50 outputs the read second feature amount distribution information 42 to the determination unit 52.
• the feature extraction unit 51 uses the feature extractor 41 to extract the first feature 601 from the target first sample image 151T.
• the feature extraction unit 51 outputs the first feature 601 to the determination unit 52.
• the determination unit 52 determines whether a morphological abnormality has occurred in the liver specimen LVS reflected in the target first specimen image 151T.
• Morphological abnormalities include lesions that are not seen in normal liver specimen LVS, such as hyperplasia, infiltration, stasis, inflammation, tumor, canceration, proliferation, bleeding, and glycogen reduction.
• the determination unit 52 outputs a determination result 61 as to whether or not a morphological abnormality has occurred in the liver specimen LVS shown in the target first specimen image 151T to the clustering processing unit 53.
• the clustering processing unit 53 performs manual clustering processing in this embodiment.
• the manual clustering process will be described later.
• the feature extracting unit 51 recognizes the liver specimen LVS appearing in the target first specimen image 151T using a well-known image recognition technique, and uses the recognized liver specimen LVS in a plurality of first patches.
• the image is subdivided into 651 images.
• the first patch image 651 has a preset size that can be handled by the feature extractor 41. Further, the first patch image 651 has a size that covers not only the abnormal shape portion but also the surrounding area.
• the feature extraction unit 51 assigns a patch image ID 85 (see FIG. 12, etc.) to the first patch image 651.
• the feature extracting unit 51 extracts information indicating which position of the first sample image 151T the first patch image 651 is extracted from, that is, position information 86 of the first patch image 651 (see FIG. 12, etc.). ) is associated with patch image ID85. Note that in FIG. 4, the first patch image 651 does not have an area that overlaps with other first patch images 651, but the first patch image 651 partially overlaps with other first patch images 651. Good too.
• the feature extractor 51 uses the feature extractor 41 to extract a first feature 601 for each of a plurality of first patch images 651 obtained by subdividing the target first sample image 151T. Extract. Therefore, the number of first feature amounts 601 is the same as the number of first patch images 651.
• the encoder section 71 of the autoencoder 70 is used as the feature extractor 41.
• the autoencoder 70 includes a decoder section 72 in addition to an encoder section 71.
• a patch image 65 such as a first patch image 651 is input to the encoder section 71 .
• the encoder unit 71 converts the patch image 65 into a feature amount 60.
• the encoder section 71 passes the feature amount 60 to the decoder section 72.
• the decoder unit 72 generates a restored image 73 of the patch image 65 from the feature amount 60.
• the encoder unit 71 includes a convolution layer that performs convolution processing using a filter, a pooling layer that performs pooling processing such as maximum value pooling processing, and the like. The same applies to the decoder section 72.
• the encoder unit 71 extracts the feature amount 60 by repeating the convolution process using the convolution layer and the pooling process using the pooling layer on the input patch image 65 multiple times.
• the extracted feature amount 60 represents the shape and texture features of the liver specimen LVS shown in the patch image 65.
• the feature amount 60 is a set of multiple numerical values. In other words, the feature amount 60 is multidimensional data.
• the number of dimensions of the feature amount 60 is, for example, 512, 1024, or 2048.
• a first feature amount 601 and a second feature amount 602 (see FIG. 9), which will be described later, have the same number of dimensions and can be compared in the same feature amount space 81 (see FIG. 10, etc.).
• the autoencoder 70 is trained by receiving the second patch image 652L for learning in a learning phase before the encoder unit 71 is used as the feature extractor 41.
• the autoencoder 70 outputs a restored learning image 73L for the second learning patch image 652L.
• a loss calculation of the autoencoder 70 using a loss function is performed.
• various coefficients (coefficients of convolutional layer filters, etc.) of the autoencoder 70 are updated in accordance with the results of the loss calculation, and the autoencoder 70 is updated in accordance with the update settings.
• the second patch image for learning 652L is an example of "teacher data" according to the technology of the present disclosure.
• Such learning of the autoencoder 70 may be performed by the image processing device 10, or may be performed by a device separate from the image processing device 10. In the latter case, the feature extractor 41 is sent from another device to the image processing device 10, and the RW control unit 50 stores the feature extractor 41 in the storage 30.
• the second patch image 652L for learning is supplied from a plurality of second patch images 652 obtained by subdividing the second sample image 152.
• the second specimen image 152 is an image in which the liver specimen LVS of the subject S of the past control group 75 is photographed.
• the past control group 75 is composed of a plurality of subjects S to whom no candidate substance was administered in past evaluation tests. Therefore, the number of subjects S forming the past control group 75 is significantly larger than the number of subjects S forming the administration group and the control group, for example, about several hundred to several thousand.
• a plurality of second specimen images 152 are obtained from one subject S, so from the past control group 75, the number obtained from one subject S is multiplied by the number of subjects S.
• a number of second sample images 152 are obtained.
• a plurality of second feature amounts 602 are extracted from each of the two-patch images 652.
• a graph 80 shown in FIG. 10 is obtained by plotting a plurality of second feature amounts 602 extracted in FIG. 9 in the feature amount space 81.
• the second feature distribution information 42 includes coordinates (hereinafter referred to as representative position coordinates) 82 in the feature space 81 of the representative position of the second feature 602 indicated by an x mark.
• the representative position is, for example, the center point or the average point of the distribution 83 of the second feature amount 602.
• the dimensions of the feature space 81 are two dimensions having the D1 axis and the D2 axis, but the actual dimensions of the feature space 81 may be 512 dimensions, etc., as described above. be.
• FIG. 11 that follows, for convenience of explanation, the dimensions of the feature amount space 81 are expressed in two dimensions.
• the second feature distribution information 42 may be created by the image processing device 10, or may be created by a device different from the image processing device 10. In the latter case, the second feature distribution information 42 is transmitted from another device to the image processing device 10, and the RW control unit 50 stores the second feature distribution information 42 in the storage 30. Further, the second feature quantity distribution information 42 may be the plurality of second feature quantities 602 themselves. In this case, the representative position coordinates 82 are derived by the image processing device 10.
• the determination unit 52 determines the feature value between the representative position of the second feature value 602 and the position of the first feature value 601 represented by the representative position coordinates 82 of the second feature value distribution information 42.
• a distance D in space 81 is calculated.
• the determination unit 52 calculates the distance D between the plurality of first feature amounts 601 extracted for each of the plurality of first patch images 651 obtained by subdividing one target first sample image 151T.
• the distance D is the Mahalanobis distance. Note that as the distance D, one of the average value, median value, and maximum value of the Euclidean distance between the position of the k-nearest sample of the distribution 83 of the second feature amount 602 and the position of the first feature amount 601 may be calculated. .
• a value obtained by subtracting the cosine similarity between the vector representing the representative position of the second feature amount 602 and the vector representing the position of the first feature amount 601 from 1.0 may be calculated.
• the cosine similarity takes a value between ⁇ 1.0 and 1.0, and it can be said that the larger the value, the more similar the directions of the vectors are.
• the determination unit 52 compares the calculated distance D with a preset threshold, and if the distance D is less than the threshold, the determination unit 52 It is determined that no morphological abnormality has occurred.
• the determination unit 52 outputs a determination result 61 indicating that no morphological abnormality has occurred in the liver specimen LVS shown in the first patch image 651.
• the determination result 61 in this case includes a patch image ID 85 and position information 86.
• the determination unit 52 determines that a morphological abnormality has occurred in the liver specimen LVS shown in the first patch image 651.
• the determination unit 52 outputs a determination result 61 indicating that a morphological abnormality has occurred in the liver specimen LVS shown in the first patch image 651.
• the determination result 61 in this case includes the first feature amount 601 in addition to the patch image ID 85 and the position information 86.
• the threshold value may be configured to be changeable by the drug discovery staff DS. Instead of the distance D, it may be determined whether a morphological abnormality has occurred in the liver specimen LVS shown in the first patch image 651 by comparing the cosine similarity described above and a threshold value.
• the second feature amount 602 is extracted from the second patch image 652 obtained by subdividing the second specimen image 152 in which the liver specimen LVS of the subject S of the past control group 75 is captured. This is the feature quantity. Since the subject S of the past control group 75 has not been administered the candidate substance, the liver specimen LVS shown in the second specimen image 152 has at least no morphological abnormality caused by the toxicity of the candidate substance. . Therefore, the representative position of the second feature amount 602 is regarded as the representative position of the feature amount of the specimen image in which the normal liver specimen LVS is photographed.
• the distance D between the position of the first feature amount 601 and the representative position of the second feature amount 602 indicates how much the liver sample LVS shown in the first patch image 651 deviates from the normal liver sample LVS. It serves as an indicator to show. Therefore, the determining unit 52 determines that the first patch image 651 for which the distance D is less than the threshold value does not deviate from the normal liver specimen LVS, and that no morphological abnormality has occurred in the liver specimen LVS. On the other hand, the determination unit 52 determines that the first patch image 651 for which the distance D is equal to or greater than the threshold value is deviated from the normal liver specimen LVS, and that a morphological abnormality has occurred in the liver specimen LVS. .
• the clustering processing section 53 includes a dimension reduction section 90, a display control section 91, and a specification reception section 92. These dimension reduction unit 90, display control unit 91, and specification reception unit 92 perform manual clustering processing as described below.
• the determination result 61 from the determination unit 52 is input to the dimension reduction unit 90 .
• the target first specimen image 151T from the RW control unit 50 is input to the display control unit 91.
• the display control unit 91 controls displaying various screens on the display 12.
• the designation receiving unit 92 receives various designations of the drug discovery staff DS through the input device 13.
• the display control unit 91 controls displaying a target designation screen 95 on the display 12.
• the display control unit 91 displays the target designation screen 95 when the drug discovery staff DS instructs to display the first specimen image 151 through the input device 13 and the designation reception unit 92 accepts the display command.
• On the target designation screen 95 a plurality of first specimen images 151 are displayed side by side.
• These plurality of first specimen images 151 are first specimen images 151 obtained from one subject S among the plurality of subjects S constituting the administration group.
• FIG. 15 shows an example in which ten first specimen images 151 with specimen image IDs “SI00001” to “SI00010” obtained from a subject S with a subject ID “R001” are displayed side by side.
• the subject ID is a pull-down menu 96, and it is possible to switch the subject S for which the first specimen image 151 is displayed on the target specification screen 95.
• the target designation screen 95 is a screen for designating one target first specimen image 151T from among the plurality of first specimen images 151.
• the target designation screen 95 is provided with one selection frame 97 that can be moved between the first specimen images 151.
• an analysis button 98 is provided at the bottom of the target specification screen 95. After aligning the selection frame 97 with the desired first specimen image 151, the drug discovery staff DS selects the analysis button 98.
• the first sample image 151 to which the selection frame 97 is aligned is set as the target first sample image 151T
• the feature quantity extracting unit 51 extracts the first feature quantity 601
• the determining unit 52 determines whether a morphological abnormality has occurred. A determination, etc. will be made.
• the dimension reduction unit 90 calculates the number of dimensions of the first feature amount 601 included in the determination result 61 of the first patch image 651 in which it is determined that the liver specimen LVS has a morphological abnormality. Reduce to two dimensions.
• the dimension reduction unit 90 generates dimension-reduced data 100 based on the determination result 61.
• the dimension-reduced data 100 includes a dimension-reduced first feature amount 601R, a patch image ID 85, and position information 86.
• the dimension reduction unit 90 outputs the dimension-reduced data 100 to the display control unit 91.
• methods for dimension reduction include Principal Component Analysis (PCA), t-distributed Stochastic Neighbor Embedding (t-SNE), or UMAP. (Uniform Manifold Approximation and Projection) etc. can be used.
• PCA Principal Component Analysis
• t-SNE t-distributed Stochastic Neighbor Embedding
• UMAP Uniform Manifold Approximation
• the display control unit 91 When the display control unit 91 receives the dimension-reduced data 100 from the dimension reduction unit 90, it controls displaying a cluster designation screen 105 shown in FIG. 17 on the display 12, as an example. A graph 106 is displayed on the cluster designation screen 105.
• the graph 106 is a graph in which the dimension-reduced two-dimensional first feature amount 601R included in the dimension-reduced data 100 is plotted in the two-dimensional feature space 107.
• the first patch image 651 has a size that covers not only the abnormal shape part but also the surrounding area. Therefore, the distribution of the first feature amount 601R does not become discrete.
• a cluster designation toolbar 108 is provided on the right side of the graph 106.
• the cluster designation toolbar 108 is used when the drug discovery staff DS designates on the graph 106 to which cluster the first patch image 651 to which it has been determined that morphological abnormality has occurred in the liver specimen LVS belongs.
• This is a toolbar that aggregates icons for the various tools you use.
• the tools include a tool for enclosing a plot of a group of first feature amounts 601R, which is considered to be a cluster, with a rectangle, circle, ellipse, or free curve.
• the tools also include tools for erasing a rectangle, circle, ellipse, or free curve surrounding the plot of the first feature amount 601R, and tools for changing the specified cluster.
• the drug discovery staff DS makes full use of various tools on the cluster designation toolbar 108 to designate which cluster the first patch image 651 belongs to on the graph 106, as shown in FIG. 18 as an example.
• FIG. 18 illustrates a case where three clusters, clusters 1, 2, and 3, are specified by surrounding them with an ellipse. Note that, as shown in the example, some of the plots of the first feature amount 601R do not belong to any cluster.
• a designation completion button 109 is provided at the bottom of the cluster designation screen 105.
• the drug discovery staff DS selects a specification completion button 109.
• the designation receiving unit 92 receives a designation by the drug discovery staff DS as to which cluster the first patch image 651, which is determined to have a morphological abnormality in the liver specimen LVS, belongs to.
• the designation receiving unit 92 generates clustering information 112 based on the received designation.
• the clustering information 112 is information in which clusters to which each patch image ID 85 of the first patch image 651 belongs are registered.
• the first patch image 651 whose plot of the first feature amount 601R does not belong to any cluster is not registered in the clustering information 112.
• the designation reception unit 92 outputs the clustering information 112 to the display control unit 91.
• the display control unit 91 generates cluster images 115, 116, and 117 by processing the target first specimen image 151T according to the clustering information 112 from the specification reception unit 92.
• Cluster image 115 is an image corresponding to cluster 1 shown in FIG.
• Cluster image 116 is an image corresponding to cluster 2 shown in FIG.
• Cluster image 117 is an image corresponding to cluster 3 shown in FIG.
• the display control unit 91 generates cluster images 115 to 117 according to a display format 118 preset for each cluster 1 to 3.
• the display format 118 is such that, for example, cluster 1 is displayed in indigo, cluster 2 is displayed in yellow-green, and cluster 3 is displayed in gray.
• the display control unit 91 fills in indigo the position of the first patch image 651 of the patch image ID 85 in which cluster 1 is registered in the clustering information 112 (as can be seen from the position information 86) in the target first specimen image 151T. , generates a cluster image 115.
• the display control unit 91 controls the cluster image 116 by filling in the position of the first patch image 651 of the patch image ID 85 in which cluster 2 is registered in the clustering information 112 with yellow-green in the target first specimen image 151T. generate. Furthermore, the display control unit 91 generates a cluster image 117 by filling in gray the position of the first patch image 651 of the patch image ID 85 in which cluster 3 is registered in the clustering information 112 in the target first sample image 151T. do. By changing the displayed colors in this way, cluster images 115 to 117 become images in which clusters 1 to 3 can be identified. Cluster images 115 to 117 are examples of "results of manual clustering processing" according to the technology of the present disclosure. Note that the display format 118 may be configured to be freely changeable by the drug discovery staff DS.
• the display control unit 91 generates a superimposed image 119 in which the first target sample image 151T and at least one of the cluster images 115 to 117 are superimposed.
• FIG. 19 illustrates a superimposed image 119 in which all the cluster images 115 to 117 are superimposed on the target first sample image 151T.
• the display control unit 91 controls displaying a clustering result display screen 125 including the superimposed image 119 on the display 12.
• the clustering result display screen 125 is provided with legends 126 for clusters 1 to 3, and display switching buttons 127, 128, and 129.
• the display switching button 127 is a button for selecting whether to display the cluster image 115 superimposed on the target first sample image 151T.
• the display switching button 128 is a button for selecting whether to display the cluster image 116 superimposed on the target first sample image 151T.
• the display switching button 129 is a button for selecting whether to display the cluster image 117 superimposed on the target first sample image 151T.
• the designation receiving unit 92 receives an instruction to select the display switching buttons 127 to 129 from the drug discovery staff DS.
• the display control unit 91 displays the cluster image 115 superimposed on the target first specimen image 151T. On the other hand, if the display switching button 127 is not selected, the display control unit 91 does not display the cluster image 115 superimposed on the target first specimen image 151T. Similarly, if the display switching button 128 is selected, the display control unit 91 displays the cluster image 116 superimposed on the target first specimen image 151T, and if the display switching button 128 is not selected, the display control unit 91 does not display the cluster image 116 superimposed on the target first sample image 151T.
• the display control unit 91 displays the cluster image 117 superimposed on the target first specimen image 151T, and if the display switching button 129 is not selected, the display control unit 91 displays the cluster image 117 superimposed on the target first specimen image 151T. , the cluster image 117 is not displayed superimposed on the target first sample image 151T.
• the display control unit 91 displays a superimposed image 119 in which all the cluster images 115 to 117 are superimposed on the target first sample image 151T. Display.
• the display control unit 91 changes only the cluster image 115 to the target first specimen image 151T.
• the superimposed image 119 is displayed. In this way, the display control unit 91 displays at least one of the plurality of cluster images 115 to 117 in a superimposed manner on the target first sample image 151T.
• the designation receiving unit 92 receives a designation of cluster images 115 to 117 to be superimposed and displayed on the target first specimen image 151T from the drug discovery staff DS.
• the clustering result display screen 125 is initially displayed with all display switching buttons 127 to 129 selected. Furthermore, the clustering result display screen 125 disappears by selecting the confirmation button 130 at the bottom.
• the operation program 40 when the operation program 40 is started in the image processing device 10, as shown in FIG. It functions as a section 53.
• the clustering processing section 53 functions as a dimension reduction section 90, a display control section 91, and a specification reception section 92.
• the photographing device 19 photographs a first specimen image 151 in which the liver specimen LVS of the subject S is photographed.
• the first specimen image 151 is output from the photographing device 19 to the image processing device 10 .
• the image processing device 10 the first specimen image 151 from the imaging device 19 is stored in the storage 30 by the RW control unit 50.
• the RW control unit 50 reads out the first specimen image 151 specified in the display instruction from the storage 30. (step ST100).
• the first sample image 151 is output from the RW control section 50 to the display control section 91 of the clustering processing section 53.
• the first specimen image 151 is displayed on the display 12 through the target designation screen 95 under the control of the display control section 91 (step ST105).
• the RW control unit 50 selects the selection frame 97.
• the first specimen image 151 with 97 combined is read out from the storage 30 as the target first specimen image 151T and acquired (step ST115).
• the target first sample image 151T is output from the RW control section 50 to the feature amount extraction section 51 and the display control section 91.
• the feature extractor 41 is read out from the storage 30 by the RW control unit 50, and the read feature extractor 41 is output to the feature extractor 51. Further, the second feature distribution information 42 is acquired by being read from the storage 30 by the RW control unit 50, and the read second feature distribution information 42 is output to the determination unit 52.
• the feature extracting unit 51 subdivides the target first sample image 151T into a plurality of first patch images 651 (step ST120). Subsequently, as shown in FIG. 5, the feature amount extractor 51 extracts the first feature amount 601 from the first patch image 651 using the feature amount extractor 41 (step ST125). The first feature amount 601 is output from the feature amount extraction section 51 to the determination section 52.
• the determination unit 52 calculates the distance D between the representative position of the second feature amount 602 and the position of the first feature amount 601. Then, as shown in FIGS. 12 and 13, the determination unit 52 compares the distance D with a preset threshold value to determine whether or not morphological abnormality has occurred in the liver specimen LVS shown in the first patch image 651. It is determined whether (step ST130). A determination result 61 as to whether or not a morphological abnormality has occurred in the liver specimen LVS shown in the first patch image 651 is output from the determination unit 52 to the dimension reduction unit 90 of the clustering processing unit 53.
• steps ST125 and ST130 is performed on all first patch images 651. After the processes in steps ST125 and ST130 are performed on all first patch images 651 (YES in step ST135), the process moves to step ST140 shown in FIG. 23.
• the dimension reduction unit 90 the number of dimensions of the first feature amount 601 of the first patch image 651 determined to have an abnormal shape is reduced to two dimensions (step ST140). .
• the cluster designation screen 105 shown in FIG. 17 is displayed on the display 12 (step ST145).
• the cluster designation screen 105 includes a graph 106 in which the two-dimensional first feature quantity 601R after dimension reduction is plotted in the two-dimensional feature quantity space 107.
• the drug discovery staff DS specifies to which cluster the first patch image 651 belongs, and the specification completion button 109 is selected.
• the designation reception unit 92 accepts the designation by the drug discovery staff DS as to which cluster the first patch image 651 belongs to (step ST150).
• clustering information 112 is generated in the designation reception unit 92. This completes the manual clustering process of clustering the first patch image 651 into a plurality of clusters (step ST155). Clustering information 112 is output from designation receiving section 92 to display control section 91 .
• the display control unit 91 processes the target first sample image 151T according to the clustering information 112 to generate cluster images 115 to 117 in which a plurality of clusters 1 to 3 can be identified.
• the clustering result includes a superimposed image 119 in which at least one of the cluster images 115 to 117 is superimposed on the target first sample image 151T under the control of the display control unit 91.
• Display screen 125 is displayed on display 12 (step ST165).
• the drug discovery staff DS views the superimposed image 119 through the clustering result display screen 125 and evaluates the drug efficacy and toxicity of the candidate substance 11. At this time, the drug discovery staff DS operates the display switching buttons 127 to 129 as necessary to switch the cluster images 115 to 117 to be displayed superimposed on the target first specimen image 151T.
• the CPU 32 of the image processing device 10 includes the RW control section 50, the feature extraction section 51, the determination section 52, and the clustering processing section 53.
• the RW control unit 50 acquires the target first specimen image 151T by reading it from the storage 30.
• the target first specimen image 151T is an image in which the liver specimen LVS of the subject S is photographed.
• the feature extractor 51 uses the feature extractor 41 to extract the first feature 601 from the first patch image 651 obtained by subdividing the target first sample image 151T.
• the determining unit 52 determines, based on the first feature amount 601, whether or not a morphological abnormality has occurred in the liver specimen LVS shown in the first patch image 651.
• the clustering processing unit 53 receives a designation from the drug discovery staff DS as to which cluster the first patch image 651 to which it has been determined that a morphological abnormality has occurred in the liver specimen LVS belongs. Then, manual clustering processing is performed to cluster the first patch image 651 into a plurality of clusters based on the specification. Therefore, it is possible to cluster the first patch image 651 of the target first sample image 151T in which the distribution of the first feature amount 601 is not discrete.
• the display control unit 91 controls displaying on the display 12 a superimposed image 119 that includes cluster images 115 to 117 that are the results of the manual clustering process. Therefore, the drug discovery staff DS can easily know the results of the manual clustering process.
• the results of the manual clustering process are displayed by a plurality of cluster images 115 to 117 generated by processing the target first sample image 151T.
• the plurality of cluster images 115 to 117 are images in which the plurality of clusters 1 to 3 can be identified by the display format 118 set in advance for each of the plurality of clusters 1 to 3. Therefore, the drug discovery staff DS can easily grasp the results of the manual clustering process.
• the display control unit 91 displays at least one of the plurality of cluster images 115 to 117 in a superimposed manner on the target first sample image 151T. For this reason, the drug discovery staff DS can easily determine which parts of the liver specimen LVS belong to which clusters, and by extension, what morphological abnormalities have occurred in which parts of the liver specimen LVS. can be grasped.
• the type of morphological abnormality can be estimated with a certain degree of certainty depending on the part where it occurs. Therefore, the drug discovery staff DS determines whether the morphological abnormality is due to the toxicity of the candidate substance 11 such as hyperplasia or whether it is a naturally occurring abnormality that is not due to the toxicity of the candidate substance 11 such as glycogen reduction. It is possible to identify whether Therefore, the drug discovery staff DS can consider the medicinal efficacy and toxicity mechanism of the candidate substance 11 based on the superimposed image 119.
• the designation receiving unit 92 receives a designation of cluster images 115 to 117 to be superimposed and displayed on the target first specimen image 151T from the drug discovery staff DS. Therefore, it is possible to superimpose only one cluster image that the drug discovery staff DS wants to focus on on the first target sample image 151T, or display all cluster images 115 to 117 superimposed on the first target sample image 151T. can. Evaluation of drug efficacy and toxicity of candidate substance 11 by drug discovery staff DS is progressing.
• the dimension reduction unit 90 reduces the number of dimensions of the first feature amount 601 to two dimensions.
• the display control unit 91 displays on the display 12 a cluster designation screen 105 that includes a graph 106 in which the first feature quantity 601R whose dimension number has been reduced to two dimensions is plotted in a two-dimensional feature quantity space 107.
• the designation accepting unit 92 accepts designations on the graph 106. Therefore, the drug discovery staff DS can easily specify which cluster the first patch image 651 belongs to.
• the feature extractor 41 extracts the livers of a plurality of subjects S forming a past control group 75 to which no candidate substance was administered in past evaluation tests of drug candidate substances.
• This model is trained using a second patch image 652 obtained by subdividing the second sample image 152 in which the sample LVS is captured as training data. Therefore, it is possible to easily extract the first feature amount 601 that well represents the shape and texture characteristics of the liver sample LVS.
• a pathologist or the like distinguishes whether or not morphological abnormalities have occurred in the liver specimen LVS, and selectively adopts the second patch image 652 in which morphological abnormalities have not occurred in the liver specimen LVS as training data.
• the second patch image 652 can be unconditionally employed as training data without requiring extensive preprocessing. Therefore, a large amount of training data can be easily obtained.
• the RW control unit 50 generates second feature distribution information, which is information about the distribution 83 of the second feature 602 extracted using the feature extractor 41 from the second patch image 652 obtained by subdividing the second sample image 152. 42 is acquired by reading it from the storage 30.
• the determining unit 52 calculates the distance D between the distribution 83 and the first feature amount 601, more specifically, the distance D between the representative position of the distribution 83 and the position of the first feature amount, and makes a determination based on the distance D. .
• the distance D indicates the degree of deviation between the target first sample image 151T and the second sample image 152.
• the liver specimen LVS shown in the second specimen image 152 has at least no morphological abnormality caused by the toxicity of the candidate substance. Therefore, if the determination is made based on the distance D, a valid determination result 61 can be obtained.
• the number of dimensions of the first feature quantity 601R after dimension reduction is not limited to the two dimensions illustrated in FIG. 16 .
• the dimension reduction unit 135 shown in FIG. 24 may output dimension reduction data 136 in which the number of dimensions of the first feature amount 601 is reduced to three dimensions.
• the display control unit 91 displays a cluster designation screen 142 on the display 12 including a graph 141 in which the first feature quantity 601R after dimension reduction is plotted in the three-dimensional feature quantity space 140. Control what is displayed on the screen.
• a tool is prepared that surrounds a plot of a group of first feature amounts 601R, which is considered to be a cluster, with a cube, a sphere, an ellipsoid, or a free curved body.
• the drug discovery staff DS can easily specify which cluster the first patch image 651 belongs to.
• the clustering processing unit 145 of this embodiment includes a membership degree calculation unit 146, a clustering information generation unit 147, and a display control unit 91.
• the membership degree calculation unit 146 performs soft clustering processing described below.
• the determination result 61 from the determination unit 52 is input to the degree of belonging calculation unit 146 .
• the degree of belonging calculation unit 146 applies a well-known soft clustering method to the first feature amount 601 included in the determination result 61, thereby assigning each cluster of the first patch image 651 to each cluster.
• the degree of belonging calculation unit 146 calculates the degrees of belonging to three clusters, clusters 1 to 3, and the probability of not belonging to any of clusters 1 to 3 (indicated as "does not belong" in FIG. 27). calculate.
• the total of the degree of belonging to clusters 1 to 3 and the probability of not belonging to any of clusters 1 to 3 is 100%.
• soft clustering methods include Gaussian Mixture Model (GMM), Probabilistic Latent Semantic Analysis (PLSA), and Non-negative Matrix Factorization (NMF). tive Matrix Factorization), or The FCM (Fuzzy c-Means) method or the like can be used.
• GMM Gaussian Mixture Model
• PLSA Probabilistic Latent Semantic Analysis
• NMF Non-negative Matrix Factorization
• FCM Fuzzy c-Means
• the degree of belonging calculation unit 146 generates degree of belonging information 160 in which the degree of belonging and the probability of non-belonging are registered for each patch image ID 85 of the first patch image 651.
• the membership degree calculation unit 146 outputs the membership degree information 160 to the clustering information generation unit 147.
• the clustering information generation unit 147 identifies the cluster to which each of the plurality of first patch images 651 belongs based on the membership degree information 160. More specifically, the clustering information generation unit 147 identifies the cluster having the maximum value among the degrees of belonging registered in the degree of belonging information 160 as the cluster to which the first patch image 651 belongs. If the probability of non-belonging is the maximum value, the clustering information generation unit 147 determines that the first patch image 651 does not belong to any cluster.
• the clustering information generation unit 147 generates clustering information 161 representing the result of the above recognition.
• the clustering information 161 is information in which clusters to which each patch image ID 85 of the first patch image 651 belongs are registered, similar to the clustering information 112 of the above-described 1_1 embodiment.
• the clustering information generation section 147 outputs the clustering information 161 to the display control section 91. Thereafter, the display control unit 91 generates a plurality of cluster images by processing the target first sample image 151T according to the clustering information 161, similarly to the above-described 1_1 embodiment. Then, a superimposed image is generated by superimposing the target first sample image 151T and the cluster image, and control is performed to display a clustering result display screen including the superimposed image on the display 12.
• the clustering processing unit 145 calculates the degree of belonging and the probability of not belonging to a plurality of clusters of the first patch image 651, which is determined to have a morphological abnormality in the liver sample LVS (step ST200).
• the degree of belonging calculation unit 146 generates degree of belonging information 160 that summarizes the calculation results of the degree of belonging and the probability of non-belonging for each first patch image 651.
• the membership degree information 160 is output from the membership degree calculation section 146 to the clustering information generation section 147. This completes the soft clustering process.
• the clustering information generation unit 147 generates clustering information 161 based on the membership degree information 160. Thereby, the first patch image 651 is clustered into a plurality of clusters (step ST205). Clustering information 161 is output from clustering information generation section 147 to display control section 91.
• the subsequent processing is the same as that in the 1_1 embodiment, so the explanation will be omitted.
• the clustering processing unit 145 of the present embodiment performs soft clustering to calculate the degree of membership of each of the first patch images 651 in which it is determined that morphological abnormality has occurred in the liver specimen LVS to a plurality of clusters. Perform processing. Therefore, according to this embodiment as well, it is possible to cluster the first patch image 651 of the target first sample image 151T in which the distribution of the first feature amount 601 is not discrete. Further, since the drug discovery staff DS does not need to specify which cluster the first patch image 651 belongs to, the burden on the drug discovery staff DS can be reduced.
• the degree of belonging may be calculated by the degree of belonging calculation unit 146 after reducing the dimension of the first feature amount 601.
• FIG. 29 illustrates a case where only the cluster image 115 of cluster 1 is displayed superimposed on the target first sample image 151T. Further, a case is illustrated in which the shade of color is changed into three levels depending on the degree of belonging.
• the drug discovery staff DS can easily grasp the degree of attribution based on the shade of color. This further facilitates evaluation of the drug efficacy and toxicity of candidate substance 11.
• the display control unit 91 displays statistical information 171 based on the results of the clustering process, as shown in a clustering result display screen 170 shown in FIG. 30 as an example.
• the statistical information 171 is the number and proportion of first patch images 651 belonging to each cluster 1 to 3.
• the ratio is calculated by dividing the number of first patch images 651 belonging to each cluster 1 to 3 by the total number of first patch images 651.
• the drug discovery staff DS can easily grasp how many first patch images 651 belong to each cluster 1 to 3. Note that instead of or in addition to the number and proportion of first patch images 651 belonging to each cluster 1 to 3, the area of the first patch images 651 belonging to each cluster 1 to 3 is displayed as statistical information. Good too.
• the third patch image for learning 653L in which the liver specimen LVS in which the morphological abnormality has occurred is also automatically It is used as training data for the encoder 70.
• the third patch image 653L for learning is supplied from a plurality of third patch images 653 obtained by subdividing the third sample image 153.
• the third specimen image 153 is an image of the liver specimen LVS of the subject S in the past administration group 175.
• the past administration group 175 is composed of a plurality of subjects S to whom candidate substances were administered in past evaluation tests. Similar to the past control group 75, the number of subjects S forming the past administration group 175 is, for example, about several hundred to several thousand. Similar to the first sample image 151, a plurality of third specimen images 153 are obtained from one subject S, so from the past administration group 175, the number obtained from one subject S is multiplied by the number of subjects S.
• the number of third sample images 153 obtained is as follows.
• the third patch image 653L for learning is obtained by selecting the third patch image 653 in which the liver specimen LVS with morphological abnormality is captured from among the plurality of third patch images 653.
• the third learning patch image 653L is selected manually by, for example, a pathologist.
• the third patch image 653L for learning can be selected by applying the method shown in the 1_1 embodiment above, extracting the third feature from the third patch image 653, and selecting the liver sample LVS based on the third feature. This may be done by determining whether or not a morphological abnormality has occurred.
• the autoencoder 70 and, by extension, the feature extractor 41 can learn about the liver sample LVS having more diverse shape and texture features. As a result, it is possible to extract the first feature amount 601 that better represents the shape and texture characteristics of the liver sample LVS.
• the patch image in which the liver specimen LVS in which morphological abnormalities have occurred is not limited to the third learning patch image 653L obtained from the subject S forming the illustrated past administration group 175. Even in the subjects S forming the past control group 75, morphological abnormalities may occur. For this reason, it does not matter whether the subject S is in the past control group 75 or the past administration group 175, as long as the patch image depicts the liver specimen LVS in which morphological abnormalities have occurred. In other words, the second learning patch image 652L in which the liver specimen LVS with abnormal morphology is captured may be used as the teacher data.
• the patch image in which the liver specimen LVS in which morphological abnormalities have occurred may be an image obtained from the subject S who has been intentionally induced to develop morphological abnormalities by applying various types of stress.
• it may be an image in which a patch image of a normal liver specimen LVS is processed to artificially create a morphological abnormality.
• the encoder unit 181 of the convolutional neural network (hereinafter referred to as CNN) 180 extracts the first feature amount 601 from the first patch image 651.
• This feature is used as a feature extractor 182.
• the feature extractor 182 is an example of a "machine learning model" according to the technology of the present disclosure.
• the CNN 180 has an output section 183 in addition to an encoder section 181.
• a patch image 65 such as a first patch image 651 is input to the encoder unit 181 .
• the encoder unit 181 converts the patch image 65 into a feature amount 60.
• the encoder section 181 passes the feature amount 60 to the output section 183.
• the output unit 183 outputs a prediction result 184 based on the feature amount 60.
• the prediction result 184 is a prediction of one type of morphological abnormality occurring in the liver specimen LVS photographed in the patch image 65 from among multiple types such as hyperplasia, infiltration, stasis, and inflammation.
• the patch image showing the liver specimen LVS with morphological abnormalities shown in the third embodiment is mainly used as training data.
• morphological abnormalities may also occur in the subjects S forming the past control group 75. Therefore, there is a possibility that the second patch image for learning 652L also includes a liver specimen LVS in which a morphological abnormality has occurred. Therefore, in the learning phase, it is preferable to perform label smoothing on the correct data and add an error rate instead of distinguishing the type of morphological abnormality as 1 or 0.
• regions that are likely to be mistakenly detected as morphological abnormalities may be masked before being used for learning. Areas that are likely to be erroneously detected as morphological abnormalities include dust and the like that adhered when the slide specimen 18 was prepared.
• the CNN 180 and, by extension, the feature extractor 182 are a model that takes on the task of identifying the type of morphological abnormality. Therefore, it is possible to extract the first feature amount 601 that better represents the shape and texture characteristics of the liver sample LVS.
• the machine learning model used as the feature extractor is not limited to the autoencoder 70 and CNN 180 as illustrated.
• a generator of generative adversarial networks (GAN) may be used as a feature extractor.
• a machine learning model that does not have a convolution layer, such as Vision Transformer (ViT), may be used as the feature extractor.
• Contrastive learning may be performed in which the distance between feature quantities derived from the same image is made closer in the feature quantity space, and the distance between feature quantities derived from different images is made greater in the feature quantity space.
• learning methods such as SimCLR (A Simple Framework for Contrastive Learning of Visual Representations) are known.
• a learning method such as BYOL (Bootstrap Your Own Latent), which does not use the above-mentioned pairs of different images (also called negative samples), may be used.
• the distribution of the extracted feature amounts may be constrained such as distribution on the unit sphere or distribution following a standard normal distribution.
• the fifth embodiment deals with a slide specimen 190 in which tissue specimens of a plurality of types of organs are placed on one slide glass 16.
• FIG. 34 illustrates a case where a heart specimen HS, a brain specimen BS, and a bone marrow specimen BMS are placed in addition to the liver specimen LVS.
• the first specimen image 151 includes a heart specimen HS, a brain specimen BS, and a bone marrow specimen BMS in addition to the liver specimen LVS.
• the CPU 32 of the image processing device 10 of the fifth embodiment functions as the identification unit 191 in addition to each of the processing units 50 to 53 of the first embodiment.
• the identification unit 191 identifies the tissue specimen of each organ from the first specimen image 151 using, for example, a template for identifying the tissue specimen of each organ or a machine learning model.
• the identification unit 191 outputs coordinate information of frames 192 to 195 surrounding the tissue specimens of each organ as identification results.
• a frame 192 is a frame surrounding the heart sample HS
• a frame 193 is a frame surrounding the liver sample LVS.
• a frame 194 is a frame surrounding the brain sample BS
• a frame 195 is a frame surrounding the bone marrow sample BMS.
• the feature extraction unit 51 subdivides the heart sample HS in the frame 192 into a plurality of first patch images 651, and the feature extractor 200 for the heart sample HS subdivides the heart sample HS from the first patch image 651 into a plurality of first patch images 651.
• a state in which a feature amount 601 is extracted is shown.
• the feature amount extraction unit 51 also extracts the first feature amount 601 for the liver sample LVS in the frame 193, the brain sample BS in the frame 194, and the bone marrow sample BMS in the frame 195. That is, the feature extraction unit 51 extracts the first feature 601 of each tissue specimen of each organ identified by the identification unit 191.
• the subsequent processing of the determination unit 52 is the same as the processing shown in the above-mentioned 1_1 embodiment etc., except that the determination is made for each tissue specimen of each organ shown in the target first specimen image 151T. The explanation will be omitted.
• the first specimen image 151 is an image obtained by photographing the slide specimen 190 on which tissue specimens of multiple types of organs are placed.
• the identification unit 191 identifies the tissue specimen of each organ from such first specimen image 151.
• the feature extraction unit 51 extracts the first feature 601 of each identified tissue sample of each organ.
• the determination unit 52 performs determination on each tissue specimen of each organ shown in the target first specimen image 151T based on the first feature amount 601. Therefore, it is possible to handle slide specimens 190 on which tissue specimens of multiple types of organs are mounted.
• the slide specimen is not the slide specimen 18 on which a tissue specimen of one organ is placed as in the first embodiment, but rather the slide specimen 190 on which tissue specimens of multiple types of organs are placed as in the present embodiment. is more common. Therefore, it is possible to perform processing that is more suitable for general operations.
• the drug discovery staff DS may manually define a frame indicating the tissue specimen of each organ in the first specimen image 151.
• the feature amount 60 is not limited to that extracted by the feature amount extractor 41.
• the average value, maximum value, minimum value, mode, or variance of the pixel values of the patch image 65 may be used.
• the organ is not limited to the illustrated liver LV, etc.
• the stomach, lungs, small intestine, large intestine, etc. may be used.
• the subject S is not limited to rats. It may also be a mouse, guinea pig, sand mouse, hamster, ferret, rabbit, dog, cat, or monkey.
• the image processing device 10 may be a personal computer installed in a drug development facility as shown in FIG. 1, or a server computer installed in a data center independent from the drug development facility.
• the first specimen image 151 is sent from a personal computer installed in each drug development facility to the server computer via a network such as the Internet.
• the server computer delivers various screens such as the target specification screen 95 to the personal computer in the form of screen data for web distribution created using a markup language such as XML (Extensible Markup Language).
• the personal computer reproduces the screen displayed on the web browser based on the screen data and displays this on the display.
• JSON JavaScript (registered trademark) Object Notation
• the image processing device 10 according to the technology of the present disclosure can be widely used throughout all stages of drug development, from the initial stage of drug discovery target setting to the final stage of clinical testing.
• the image processing device 10 may be configured with a plurality of computers separated as hardware for the purpose of improving processing power and reliability.
• the functions of the feature extracting section 51 and the determining section 52 and the functions of the clustering processing section 53 or 145 are distributed between two computers.
• the image processing device 10 is composed of two computers.
• the hardware configuration of the computer of the image processing device 10 can be changed as appropriate depending on the required performance such as processing capacity, safety, and reliability.
• application programs such as the operating program 40 can be duplicated or distributed and stored in multiple storages for the purpose of ensuring safety and reliability. .
• the RW control unit 50, the feature amount extraction unit 51, the determination unit 52, the clustering processing units 53 and 145, the dimension reduction unit 90, the display control unit 91, the specification reception unit 92, the degree of belonging calculation unit 146 , the clustering information generation section 147, and the identification section 191, which execute various processes, can have a hardware structure of the following various processors.
• various processors include the CPU 32, which is a general-purpose processor that executes software (operating program 40) and functions as various processing units, as well as FPGA (Field Programmable Gate Array), etc.
• Dedicated processors are processors with circuit configurations specifically designed to execute specific processes, such as programmable logic devices (PLDs), which are processors whose circuit configurations can be changed, and ASICs (Application Specific Integrated Circuits). Includes electrical circuits, etc.
• PLDs programmable logic devices
• ASICs Application Specific Integrated Circuits
• One processing unit may be composed of one of these various processors, or a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs and/or a CPU and (in combination with FPGA). Further, the plurality of processing units may be configured with one processor.
• one processor is configured with a combination of one or more CPUs and software, as typified by computers such as clients and servers.
• a processor functions as multiple processing units.
• SoC system-on-chip
• various processing units are configured using one or more of the various processors described above as a hardware structure.
• an electric circuit that is a combination of circuit elements such as semiconductor elements can be used.
• the processor includes: Obtaining a first specimen image showing a tissue specimen of the subject; Extracting a first feature amount from a first patch image obtained by subdividing the first sample image using a machine learning model, determining whether a morphological abnormality has occurred in the tissue specimen reflected in the first patch image based on the first feature amount; A designation is received from the user as to which cluster the first patch image to which it has been determined that the morphological abnormality has occurred in the tissue specimen belongs, and the first patch image is divided into a plurality of clusters based on the designation.
• the processor includes: The image processing apparatus according to supplementary note 1, which performs control to display a result of the manual clustering process or a result of the soft clustering process.
• the result is displayed by a plurality of cluster images generated by processing the first sample image, The image processing device according to appendix 2, wherein the plurality of cluster images are images in which the plurality of clusters can be identified by a display format set in advance for each of the plurality of clusters.
• the processor includes: The image processing device according to Supplementary Note 3, wherein at least one of the plurality of cluster images is displayed in a superimposed manner on the first sample image.
• the processor includes: The image processing device according to supplementary note 4, which receives from the user a designation of the cluster image to be superimposed and displayed on the first sample image.
• the processor includes: The image processing device according to any one of Supplementary Notes 2 to 5, which displays statistical information based on the results.
• the processor includes: In the manual clustering process, the number of dimensions of the first feature quantity is reduced to two or three dimensions, and a graph is displayed in which the first feature quantity with the reduced number of dimensions is plotted in a two-dimensional space or a three-dimensional space;
• the image processing apparatus according to any one of Supplementary Notes 1 to 6, which accepts the specification on the graph.
• the machine learning model is based on a second sample image that is obtained by subdividing a second sample image in which tissue samples of a plurality of subjects constituting a control group to which the candidate substance was not administered in a past evaluation test of a drug candidate substance.
• the image processing device according to any one of Supplementary Notes 1 to 7, which is a model learned using a 2-patch image as teacher data.
• the image processing apparatus according to appendix 8, wherein the teacher data also includes a patch image of the tissue specimen in which the morphological abnormality has occurred.
• the machine learning model is a model responsible for the task of identifying the type of the morphological abnormality.
• the processor includes: obtaining information on the distribution of a second feature extracted using the machine learning model from a second patch image obtained by subdividing the second sample image; Calculating the distance between the distribution and the first feature amount, The image processing device according to any one of Supplementary Notes 8 to 10, wherein the determination is made based on the distance.
• a and/or B has the same meaning as “at least one of A and B.” That is, “A and/or B” means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, even when three or more items are expressed in conjunction with “and/or”, the same concept as “A and/or B" is applied.

Landscapes

• Engineering & Computer Science (AREA)
• Health & Medical Sciences (AREA)
• Medical Informatics (AREA)
• General Health & Medical Sciences (AREA)
• Theoretical Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Public Health (AREA)
• General Physics & Mathematics (AREA)
• Physics & Mathematics (AREA)
• Databases & Information Systems (AREA)
• Evolutionary Computation (AREA)
• Epidemiology (AREA)
• Primary Health Care (AREA)
• Computing Systems (AREA)
• Multimedia (AREA)
• Software Systems (AREA)
• Artificial Intelligence (AREA)
• Biomedical Technology (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Radiology & Medical Imaging (AREA)
• Data Mining & Analysis (AREA)
• Quality & Reliability (AREA)
• Pathology (AREA)
• Business, Economics & Management (AREA)
• General Business, Economics & Management (AREA)
• Image Analysis (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Related Child Applications (1)

Publications (1)

Family

ID=89706331

Family Applications (1)

Country Status (5)

Cited By (1)

Families Citing this family (1)

Citations (3)

Family Cites Families (2)

• 2023
  • 2023-07-19 WO PCT/JP2023/026383 patent/WO2024024587A1/ja not_active Ceased
  • 2023-07-19 CN CN202380056570.XA patent/CN119790427A/zh active Pending
  • 2023-07-19 EP EP23846318.6A patent/EP4564287A4/en active Pending
  • 2023-07-19 JP JP2024537627A patent/JPWO2024024587A1/ja active Pending
• 2025
  • 2025-01-23 US US19/035,368 patent/US20250173866A1/en active Pending

Patent Citations (3)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events