WO2024247990A1 - プログラム、生成方法及びシステム - Google Patents

プログラム、生成方法及びシステム Download PDF

Info

Publication number
WO2024247990A1
WO2024247990A1 PCT/JP2024/019506 JP2024019506W WO2024247990A1 WO 2024247990 A1 WO2024247990 A1 WO 2024247990A1 JP 2024019506 W JP2024019506 W JP 2024019506W WO 2024247990 A1 WO2024247990 A1 WO 2024247990A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
label
information
region
images
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2024/019506
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
弘幸 小鮒
皓一 福田
聡志 板倉
修二郎 奥田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Denka Co Ltd
Original Assignee
Denka Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Denka Co Ltd filed Critical Denka Co Ltd
Priority to JP2025524101A priority Critical patent/JPWO2024247990A1/ja
Priority to EP24815473.4A priority patent/EP4723054A1/en
Publication of WO2024247990A1 publication Critical patent/WO2024247990A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—Two-dimensional [2D] image generation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7235—Details of waveform analysis
    • A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
    • A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30096—Tumor; Lesion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/03—Recognition of patterns in medical or anatomical images

Definitions

  • the present invention relates to a program, a generation method, and a system for generating training data for machine learning algorithms.
  • the data generation device in Patent Document 1 generates training data for a machine learning algorithm.
  • the training data consists of multiple images and labels associated with each image.
  • the present invention provides a program, a generation method, and a system that provide a new and highly convenient technology for generating training data.
  • a program for generating learning data from medical images the program causing a computer to execute a generation process in which a plurality of pieces of information are associated with a region image, which is an image of at least one region in the medical image, and the generation process causes the computer to associate labels with the region image using the plurality of pieces of information through manual operations received via an operation input unit or automatic processing by a computer, thereby generating the learning data from the medical image.
  • the plurality of pieces of information include first and second information
  • the region images include a first region image and a second region image
  • the generation process causes a computer to associate a label determined for the first region image by the first information with the second region image selected by the second information, thereby generating the learning data.
  • the classification result of the region image by the machine inference is the second information, and the generation process causes a computer
  • [6] The program according to [4] or [5], wherein the medical image includes a pathology image.
  • the first information includes at least one of a gene mutation type, a tumor mutation-high (TMB-H) type, a result of a biochemical analysis, or a patient's age, sex, medical history, a result of an immunohistochemical analysis, a lifestyle, or a prognosis prediction.
  • TMB-H tumor mutation-high
  • a method for generating learning data from medical images comprising the steps of: causing a computer to execute a generation step in which a plurality of pieces of information are associated with a region image, which is an image of at least one region in the medical image; and causing the computer to associate labels with the region image using the plurality of pieces of information through manual operations received via an operation input unit or through automatic processing by a computer, thereby generating the learning data from the medical image.
  • a system for generating learning data from medical images comprising a generation unit, wherein a plurality of pieces of information are associated with a region image, which is an image of at least one region in the medical image, and the generation unit generates the learning data from the medical image by associating labels with the region image using the plurality of pieces of information through manual operations received via an operation input unit or automatic processing by a computer.
  • the present invention provides a novel technique for associating labels with area images using multiple pieces of information associated with the area images. This makes it possible to generate learning data with high ease.
  • Fig. 1 is a diagram showing an example of a system 1 (server 1), each user terminal 3, and a diagnostic device 4 according to an embodiment.
  • Fig. 2A is a block diagram showing an example of a hardware configuration of the server 1.
  • Fig. 2B is a block diagram showing an example of a hardware configuration of the user terminal 3.
  • Fig. 3A is a block diagram showing the functional configuration of the server 1 according to the first embodiment.
  • Fig. 3B is a functional block diagram of the generating unit 23 according to the first embodiment.
  • 13 is a first example (grid method, manual confirmation process) of a process flow illustrating a control operation of the first embodiment.
  • 13 is a second example (grid method, automatic determination process) of the process flow illustrating the control operation of the first embodiment.
  • FIG. 13 is a third example (designated frame method, manual confirmation process) of a process flow illustrating a control operation of the first embodiment.
  • Fig. 7A shows an example of a state where grid display is turned on in the selection screen 27 of Fig. 6 (designation frame method).
  • Fig. 7B shows an example of the result of region image conversion performed according to the grid of Fig. 7A.
  • 13 is a fourth example (designated frame method, automatic confirmation process) of the process flow illustrating the control operation of the first embodiment.
  • Fig. 9A illustrates an example in which there are three designated frames in Fig. 7A.
  • Fig. 9B illustrates an example of the result of performing region image conversion according to the grid in Fig. 9A.
  • FIG. 9A illustrates an example in which there are three designated frames in Fig. 7A.
  • Fig. 9B illustrates an example of the result of performing region image conversion according to the grid in Fig. 9A.
  • FIG. 10A is an example of an image whose labels comprise index values
  • FIG. 10B is an example of an image whose labels are index values.
  • Fig. 11A is a block diagram showing the functional configuration of the server 1 according to the second embodiment.
  • Fig. 11B is a functional block diagram of the generating unit 23 according to the second embodiment.
  • 13 is a first example of a process flow illustrating a control operation of the second embodiment.
  • 13 is a second example (processing using index values) of a process flow illustrating a control operation of the second embodiment.
  • annotation refers to the act of labeling an image or a group of labels, depending on the context.
  • Annotation as an annotation act means "the act of associating annotation information as correct answer data with target information (electronic information) for machine learning".
  • target information electronic information
  • image annotation includes object detection, area image extraction, and image classification, for example.
  • Image annotation associates a label with an annotation target.
  • the annotation target may be each image (the entire image) or an area image in the image (including an area, the entire image).
  • the label is also called a tag or metadata.
  • the label represents the classification content (classification result) and becomes the correct answer data during supervised learning.
  • "annotation” refers to "associating a label with an area image, which is an image of at least one area specified in an image," except when a different definition is specified or when a clear contradiction occurs in the context.
  • the images to be annotated include at least “medical imaging”.
  • Medical imaging includes “images that visualize the internal structure and function of the human body as images, mainly for the diagnosis and treatment of diseases".
  • medical images may include plain X-rays, computed tomography (CT), magnetic resonance imaging (MRI), ultrasound tomography (US), nuclear medicine examinations, and angiograms.
  • medical images according to the present disclosure may further include "endoscopic images” and "pathological images”.
  • Endoscopic images may include, for example, images of the esophagus, stomach, intestines, and other digestive system images.
  • pathological images may be "(microscopic) images of tissues collected by endoscopy or surgery, (1) fixed to prevent deterioration, (2) stained to facilitate observation, and (3) sliced into a few microns, which are then enlarged and photographed”.
  • the present invention is applicable to any medical image in the same manner, and any label classification type and classification content can be set.
  • the label classification type is, for example, information on the type of pathology.
  • the pathology type may be either "tumor” or "hypermutation".
  • the pathology type may be set to "tumor”, and in that case, the label classification content can be "tumor” or "non-tumor”.
  • the pathology type may be set to "hypermutation", and in that case, the label classification content can be "hypermutation” or "non-hypermutation”.
  • a pathology image is used as an example of a medical image
  • the label classification type is, for example, a classification of whether it is a tumor or not
  • the label classification content is "tumor" or "non-tumor”.
  • a pathology image is used as an example of a medical image
  • the classification type of the label is a classification of whether it is a tumor or not
  • the classification content of the label is "tumor" or "non-tumor.”
  • the "grid method” specifies an area at a desired position and range in an image using a grid (i.e., a grid-like boundary line).
  • a grid i.e., a grid-like boundary line.
  • each square of the grid can define one area.
  • the grid can be uniformly spaced, and each area can be specified as a rectangle of uniform size, and such areas can be called "tiles.”
  • the grid interval can be either fixed or arbitrarily specified by the user.
  • the "grid” referred to here has a broad meaning, and is not limited to a rectangular lattice, but can be a lattice in which any polygon that can be tiled on a plane is periodically arranged.
  • the grid can be, for example, a honeycomb grid.
  • honeycomb grid referred to here has a broad meaning, and can be, for example, a regular hexagonal honeycomb, but can also be, for example, a honeycomb structure of an equilateral triangle or parallelogram, or a honeycomb structure of a parallel hexagon (a hexagon with three pairs of opposite sides parallel to each other but with different lengths of sides between each pair).
  • the grid-based annotation is adopted in the examples of Figures 4 and 5.
  • a region is specified at a desired position and range in an image using a designated frame that is arbitrarily specified by the user.
  • the designated frame is sometimes called a bounding box.
  • the designated frame can be set by the user to any size and shape.
  • the designated frame may be, for example, a rectangle, but is not limited to this, and may be any polygon or closed curve.
  • the designated frame method of annotation is adopted in the examples of Figures 6 and 8.
  • the area specified by a grid may be referred to as a "grid area” to facilitate understanding.
  • the area specified by a designation frame may be referred to as a "designated frame area” to facilitate understanding.
  • area when simply saying "area,” it is intended to include both the grid area and the designated frame area, except when it is clear from the context that one of the grid area and the designated frame area is being referred to.
  • the image corresponding to that area is also called an "area image.”
  • the area image is an image specified as a uniform rectangular size using a grid method, this area image may be called a tile image.
  • the size of the area and area image (number of pixels vertically and horizontally) may be the same as the learning data that can be input to the learning model.
  • the program, method, and system of the embodiment can employ annotation using either or both of the grid method and the designated frame method. If both are employed, the method to be used for the current annotation may be set by a user setting operation and/or by automatic computer processing.
  • the learning dataset is composed of a "learning data image” cut out from a medical image and a label associated with the learning data image.
  • the size of the learning data image (the number of pixels in the vertical and horizontal directions) is set to a size that can be input to the learning model.
  • the "learning data image” may be a medical image cut out along a grid, or a part of the medical image cut out with a designated frame (bounding frame).
  • the learning data image may be a medical image cut out according to a region image at the time of annotation, that is, the region image at the time of annotation and the learning data image may match.
  • the learning data image may be different from the region image at the time of annotation.
  • the learning data image may be generated by cutting it out with a grid (e.g., a rectangular grid with uniform intervals).
  • the learning data image may be generated by cutting it out with another grid (e.g., a rectangular grid with uniform intervals).
  • images of each region of a medical image may be displayed on the work screen 26 and selection screen 27 described below, but it is not necessary to cut out the medical image at this display stage.
  • the medical image may be cut out.
  • First embodiment (2-1. Overview of System 1 and Diagnostic Device 4)
  • the system 1, user terminal 3, and diagnostic device 4 of the first embodiment can communicate with each other via a wireless or wired communication network 2.
  • the system 1 in Fig. 1 can generate learning data from medical images.
  • the system 1 is a computer, specifically a server.
  • the system 1 is provided by the server executing a program.
  • the system 1 may be referred to as the server 1.
  • Users U001 and U002 who use the system 1 are, for example, pathologists.
  • the system 1 includes a generation unit 23 (see FIG. 3A) that has the function of generating learning data.
  • the generation unit 23 can generate learning data using an input image (specifically, a medical image).
  • the learning data can be used as teacher data for training a learning model included in the diagnostic device 4.
  • the diagnostic device 4 uses a trained model trained using the generated learning data to determine whether or not a pathological abnormality is present in the medical image.
  • the system 1 can display on the screen a medical image (in this embodiment, a pathological image) stored in the storage unit 20 superimposed on a grid (boundaries between multiple regions).
  • a medical image in this embodiment, a pathological image
  • One example of the screen is a screen on the display device 34 (see FIG. 2B) of the user terminal 3.
  • the system 1 accepts annotation operations on the region images by the user. In other words, the system 1 accepts label input from users U001 and U002 for each region image displayed on the screen. As a GUI for label input operations, the system 1 generates a work screen 26 in FIG. 4 described below, and displays it on the user terminal 3.
  • the system 1 may generate learning data from one pathology image, or may generate learning data from multiple pathology images.
  • the system 1 repeats the process of dividing the pathology image into multiple region images and accepting label input for each pathology image.
  • the system 1 When the user has completed labeling all pathology images, the system 1 generates learning data. For example, the system 1 generates a learning data image by cutting out a medical image according to a grid, and generates learning data by matching the image data of this learning data image with the labels assigned to each region image.
  • the generated learning data is sent to the diagnostic device 4.
  • the diagnostic device 4 trains a learning model using the learning data sent from the system 1.
  • the learning model may be, for example, a neural network that can be given a predetermined capability through machine learning.
  • the diagnostic device 4 inputs a pathological image of the diagnosis target into the trained model generated by machine learning, and determines whether or not a pathological abnormality is present in the pathological image based on the output result from the trained model.
  • the system 1 can provide a selection screen 27 as a support function for supporting annotation. This support function will be explained in detail later.
  • FIG. 1 is a diagram showing an example of a hardware configuration of the system 1.
  • the system 1 includes a processor 11 such as a central processing unit (CPU) or a graphical processing unit (GPU), a storage device 12 such as a memory, a hard disk drive (HDD) and/or a solid state drive (SSD), and a communication interface (IF) 13 for wired or wireless communication.
  • the system 1 receives user input operations from a user terminal 3 via the communication IF 13 and displays various screens on a display device 34 (see FIG. 2B) of the user terminal 3.
  • the user terminal 3 includes, as an example, a control unit 31 including a processor, a storage unit 32 including a non-volatile memory, a communication unit 33 for wireless communication or/and wired communication, a display device 34, a speaker 35, a microphone 36, a camera 37, and an operation device 38.
  • the operation device 38 is an example of an operation input unit, and is any input device such as a keyboard or a mouse.
  • the specific configuration of the user terminal 3 is not limited, and may be various mobile terminals, smartphones, tablet terminals, or notebook PCs.
  • the display device 34 and the operation device 38 may be integrated into the touch panel display.
  • the operation device 38 and the microphone 36 are input interfaces (or input means) for the user to perform various input operations on the user terminal 3.
  • FIG. 3A is a diagram showing an example of a functional block configuration of the system 1.
  • the system 1 includes a storage unit 20, a display control unit 21, an input unit 22, and a generation unit 23.
  • the storage unit 20 can be realized using a storage device 12 included in the system 1.
  • the display control unit 21, the input unit 22, and the generation unit 23 are provided by the processor 11 of the system 1 executing a program stored in the storage device 12.
  • the program may be provided on a network or may be stored in a storage medium.
  • the storage medium storing the program may be a non-transitory computer readable storage medium.
  • the non-transitory storage medium is not particularly limited, and may be, for example, a storage medium such as a USB memory or a CD-ROM.
  • the storage unit 20 includes various databases (DBs).
  • DBs databases
  • the storage unit 20 stores an account DB, an image DB, an area DB, an annotation DB, a label DB, and a learning data DB.
  • Each DB has an identification ID assigned to the stored data, and data related to each other can be cross-referenced.
  • the account DB stores each user's user ID, user details, login history, etc.
  • the user details may include an email address, a name associated with the account (nickname, avatar icon, etc.), and/or more personal, specific information such as name and affiliation.
  • the image DB stores one or more medical images (e.g., pathology images) used to generate learning data.
  • the region DB stores information that identifies the region image in each medical image.
  • the region DB stores the region image ID, the medical image ID to which the region image is set, and image coordinate data.
  • the image coordinate data specifies what range of the medical image each region image occupies.
  • the annotation DB stores annotation information for distinguishing annotations made using the system 1 together with annotation IDs.
  • the annotation information is preferably information that can identify which medical image was annotated, when, by whom, and what type of annotation was made.
  • the annotation information can preferably store "annotation subject information.”
  • the annotation subject information is identification information for identifying "who made the annotation (i.e., the annotation subject)."
  • the annotation subject information may be, for example, a user ID (e.g., a login account), or may be any one or more pieces of information in the user details information stored in the account DB.
  • the annotation information may include whether the annotation can be disclosed (i.e., whether it can be disclosed to the selection screen 27 of others with different user IDs), and may include the disclosure conditions of the annotation (i.e., under what conditions it should be disclosed to others).
  • the label DB stores information about labels associated with region images, which are images of regions, by each annotation, in association with the annotation ID.
  • the label DB stores each label ID in association with label detail information.
  • the label detail information includes the classification type and classification content of the label (for example, the type is tumor classification, and the label classification content is tumor, etc.).
  • the training data DB stores the generated training data. Users can download any training data by referencing the training data DB.
  • At least the image DB, area image ID, annotation ID, and label ID are linked to each other. This makes it possible to easily call up the contents of any annotation (i.e. medical image, area image, label, etc.) from the annotation DB by specifying the annotation ID.
  • annotation ID i.e. medical image, area image, label, etc.
  • the above data management and DB configurations are merely examples. Any data management and DB configurations can be adopted depending on the specifications of the actual hardware and/or software, etc. Note that if management is not performed on an annotation basis, the annotation DB can be omitted.
  • the display control unit 21 executes display control to display various screens on the display device 34 of the user terminal 3.
  • the display control unit 21 may be capable of displaying a work screen 26 (see FIG. 4) in which a grid (boundary line of an area) is superimposed on a medical image.
  • a screen in which a medical image Q is divided into four parts by a grid is illustrated in a balloon extending from the work screen 26 in FIG. 4.
  • An area image ID is associated with each area image.
  • the top left square of the medical image Q is also referred to as area image Q1 .
  • the display control unit 21 may be capable of displaying a designated frame superimposed on the medical image Q.
  • the input unit 22 accepts various inputs from the operation device 38 of the user terminal 3.
  • the input unit 22 accepts a label input operation (i.e., an operation of the operation device 38 when the user performs annotation).
  • the label input operation is an input operation for specifying which label is to be associated with each of a plurality of area images (area images).
  • the input unit 22 stores the labels associated with each area image in the label DB.
  • the input unit 22 may store an area image ID that uniquely identifies each area image in association with the label associated with each area image in the label DB.
  • the label input may be performed, for example, by accepting a selection of an area image for which a label is to be input from among the area images displayed on the screen, and accepting a designation of a label to be associated with the area image.
  • the generating unit 23 generates learning data for training the learning model. For example, the generating unit 23 acquires a region image ID and a label from the label DB, and generates a learning data image by cutting out a region image corresponding to the region image ID from a medical image stored in the image DB. The generating unit 23 generates learning data by combining the learning data image data with a label stored in association with the region image ID.
  • FIG. 3B shows the functional block configuration of the generation unit 23.
  • the generation unit 23 includes a data acquisition unit 23a, a label determination unit 23b, and a learning data construction unit 23c. The operation of each functional block will be explained in the process flows of FIG. 4 and FIG. 5.
  • FIG. 4 (2-5. Processing flow)
  • FIG. 4 (2-5. Processing flow)
  • FIG. 4 is a specific example of the processing flow of the first embodiment, and provides a program and a method for generating learning data.
  • the processing flows of Figures 4 and 5 use grid-based annotation.
  • the processing flows of Figures 6 and 8 use frame-based annotation.
  • the generation process is executed by a computer (system 1) (see S103 to S106 in FIG. 4, or S103, S115, and S106 in FIG. 5).
  • This generation process is performed by manually or automatically causing the computer (system 1) to associate area images with labels using a plurality of pieces of information associated with the area images, thereby generating learning data.
  • first and second information are associated with each area image. Label association of the area image can be performed using a plurality of pieces of information associated with the area image, so that learning data can be generated with high convenience.
  • the "plurality of information” used here differs between the first and second embodiments.
  • the above “plurality of information” includes first and second information, and first and second labels are used as the first and second information.
  • a computer (system 1) is caused to execute a process of acquiring a first label based on the first annotation and a second label based on the second annotation. This process is comprehensively described in step S100 in FIG. 4, and will be described in detail below.
  • multiple labels associated with a region image in a medical image may have the same classification content.
  • label matching multiple labels associated with a region image and with the same classification content are also referred to as “matching labels.”
  • labels with different classification content are associated with a region image, that is, where judgments based on multiple annotations differ.
  • label mismatch multiple labels associated with a region image and with different classification content are also referred to as “mismatching labels.”
  • the user U002 who has logged in to the system 1 receives the work screen 26 from the system 1 (S101b).
  • the user U002 performs a "second annotation" via the work screen 26, thereby inputting labels to each area image (S301b).
  • the second label Lb1 and the area image Q1 are associated with each other by the operation of the user U002, and the system 1 registers the association in the label DB (S102b). Similar label associations are also performed for the other grid area images Q2 , Q3 , ... of the medical image Q.
  • a label is input to the grid area image in step S100.
  • a designated frame area image is set by the designation frame in step S100, and a label is input to this designated frame area image.
  • the label classification content is displayed with the letters "T" and "N" (S301a, S30b).
  • the letter "T” is displayed.
  • the label “N” is displayed.
  • the label content added in the first and second annotations is different, and a case is illustrated in which the judgments of users U001 and U002 differ.
  • one grid area image Q1 is illustrated as an example for convenience of explanation.
  • the first annotation associates the first labels La1 to Lan with each of one or more area images Q1 to Qn .
  • the second annotation associates the second labels Lb1 to Lbn with each of one or more area images Q1 to Qn .
  • area image Qk first label Lak
  • second label Lbk second label
  • FIG. 4 is a first example (manual confirmation process) of a process flow showing the control operation of the first embodiment.
  • label confirmation is performed by accepting a manual selection operation by a user.
  • the first and second labels by the first and second annotations have already been registered in the label DB by the above-mentioned step S100.
  • a user who wants learning data (user U002 in the example of FIG. 4) issues a learning data generation command to the system 1 via the operation device 38 (S302).
  • this learning data generation command input data required for this learning data generation can be specified.
  • a medical image ID may be specified as a target for adopting learning data
  • a range of area image IDs may be specified
  • an annotation call mode may be set.
  • the system 1 starts a process of generating learning data (S103).
  • the generation process includes steps S103 to S106.
  • the data acquisition unit 23a (see FIG. 3B) of the generation unit 23 refers to each DB of the storage unit 20.
  • the data acquisition unit 23a reads out each area image Q k to be included in the current learning data according to the content of the learning data generation command, and at the same time, reads out each label associated with each area image Q k by searching the label DB by the area image ID.
  • the labels read out here include the first and second labels La k and Lb k .
  • the data acquisition unit 23a performs different data acquisition processes depending on the annotation call mode.
  • the annotation call mode a specific call mode, a full call mode, or a layered call mode can be adopted, and the user may be able to selectively use two or more of these.
  • the desired annotation is specified by the annotation ID, etc., and the area image ID and label ID, etc. of the specified annotation are referenced.
  • the full call mode all annotations associated with the medical image at the time of the call are referenced, and the area image IDs and label IDs, etc. of all the annotations are referenced.
  • stratification In the stratified call mode, only annotations that meet the specified conditions are extracted, and the area image ID and label ID of the extracted annotation are referenced.
  • stratification may be based on whether or not the annotation subject information meets a specified condition. For example, only annotations whose performance values are equal to or greater than a specified value may be extracted, thereby making it possible to use only annotations that are highly proven and reliable.
  • a period may be specified as one of the call conditions, and only annotations that were registered (specifically, new registration or last edited registration) within the specified period may be extracted.
  • index values may be used, and for example, only annotations whose summary statistics of all labels in the annotation (for example, the average index value or the median index value of all labels) are equal to or greater than a specified threshold may be extracted, thereby making it possible to use only annotations that the annotation subject created with a certain degree of confidence.
  • the label determination unit 23b executes a label determination process (S104, S105).
  • the label determination unit 23b generates a selection screen 27 based on the data acquired in step S103 and transmits it to the user terminal 3 (S104).
  • the selection screen 27 is a GUI for allowing the user U002 to select which of the multiple labels associated with each area image Qk should be set as a confirmed label.
  • the system 1 accepts an operation to select either the first or second label La1 , Lb1 in the area image Q1 .
  • the user U002 can determine the desired label by clicking with the mouse pointer P (S303).
  • multiple area images Qk may be comprehensively arranged on the selection screen 27.
  • FIG. 4 the example of FIG.
  • the label determination unit 23b determines the label selected by the user (in this example, the second label Lb 1 ) as a determined label, and associates this determined label with the region image Q 1. Such operations and processes are repeated for each region image Q k required for the learning data.
  • step S106 the learning data construction unit 23c of the generation unit 23 cuts out the medical image Q according to the grid, and generates a learning data image of the same size as the region image by the grid.
  • the learning data construction unit 23c constructs learning data in which the learning data image and the label are associated with each other, and registers the learning data in the DB (S106).
  • the user U001 can download the learning data at will.
  • the process flow of FIG. 4 causes the computer (system 1) to execute the generation process.
  • the computer (system 1) determines a definite label for each region image Q k using a plurality of labels La k , Lb k associated with the region image Q k (S105) according to a manual selection operation (S303) received via the operation device 38, and generates learning data from the medical image by associating the definite label with the region image (S106).
  • step S104 the user may select a label for each area image Q k without distinguishing between label matches and label mismatches, but as a modified example, a batch confirmation process may be provided.
  • the batch confirmation process is a process in which, based on a user instruction (e.g., clicking a predetermined button, etc.), each area image with a matching label is confirmed collectively to the classification content of the matching label.
  • a learning data generation command (S302) is sent to the system 1 by a second user U002 who has previously performed annotation work.
  • this is just one example and is not limiting.
  • a third user may perform only the confirmation operation (S303) without performing a label input operation (S301a, S301b) for medical image Q.
  • the label selection for each area image by the confirmation operation (S303) corresponds to the annotation work.
  • Fig. 5 shows a second example (automatic determination process) of the process flow showing the control operation of the first embodiment.
  • the computer the system 1, specifically the label determination unit 23b
  • the label determination process S115
  • S100 S101a to S102b
  • S103 S103
  • S106 S106
  • step S103 the label determination unit 23b executes the label determination process (S115).
  • the label determination process when the labels match may be, for example, as follows. Assume that there is a region image Q k in which the first and second labels La k and Lb k show the same result. A computer (system 1) executes a process to set the determined label to the same content as the first and second labels La k and Lb k . Specifically, if the first and second labels La 2 and Lb 2 (both not shown) associated with the region image Q 2 are both "T" (i.e., tumor), the determined label of the region image Q 2 is set to "T" in this case.
  • the label determination process when the labels do not match may be as follows, for example. Assume that there is an area image Q k in which the first and second labels La k and Lb k show different results. In this case, a process set in advance as the label determination process is executed. As an example, the computer (system 1) may execute a process in which the result of one of the first and second labels La k and Lb k is set as the determined label according to a predetermined rule. As a specific example, as in the example of FIG. 5, assume that the first and second labels La 1 and Lb 1 associated with the area image Q 1 are different, that is, "T" and "N".
  • the label determination unit 23b may determine the determined label based on information of the annotation subject (users U001 and U002 in the first embodiment), for example.
  • An example of the annotation subject information is "annotation performance value".
  • the label determination unit 23b may read out the annotation result value of the user U001 who assigned the first label La1 and the annotation result value of the user U002 who assigned the second label Lb1 from the storage unit 20 and compare these result values. As a result of the comparison, the label assigned by the user with the relatively higher annotation result value may be set as the determined label.
  • the computer when there is a region image Qk with a mismatched label, the computer (system 1) may be caused to execute a process of excluding the region image Qk from the learning data.
  • a manual operation selection process (for example, S104 and S105 in FIG. 4 can be used) may be adopted for the region image Qk with a mismatched label, and the computer (system 1) may be caused to execute a process of accepting a label selection from a user.
  • the process flow of FIG. 5 causes a computer to execute the generation process.
  • the computer (system 1) determines a definite label for each region image Q k using a plurality of labels (first and second labels La k , Lb k ) associated with the region image Q k according to the automatic process (label determination process of S115) of the computer (system 1), and generates learning data from the medical image by associating the definite label with the region image Q k .
  • the definite label for each region image can be automatically determined based on the labels of the first and second annotations. Learning data that takes into account a plurality of annotations can be generated efficiently and with a small workload, thereby improving convenience.
  • step S101a the work screen 26 displays a medical image Q without a grid.
  • steps S301a and S301b the users U001 and U002 set designated frame area images Q1a and Q1b in the medical image Q using a designated frame via the work screen 26, and input labels La1 and Lb1 .
  • the selection screen 27 is generated so that the designated frame area images Q1a , Q1b and the labels La1 , Lb1 are displayed superimposed on the medical image Q.
  • the confirmation operation S303
  • the user U002 places the mouse pointer P on one of the designated frames of the designated frame area images Q1a , Q1b and clicks. This operation results in the adoption of one of the set of the designated frame area image Q1a and the label La1 , and the set of the designated frame area image Q1b and the label Lb1 .
  • the selection operation of the user U002 is performed in the same manner for all the designated frames associated with the medical image Q.
  • step S106 processing for constructing learning data is executed.
  • the learning data constructing unit 23c can cut out the medical image Q according to the specified frame and generate a learning data image of the same size as the area image defined by the specified frame.
  • the computer may execute the area image conversion process described below in step S104.
  • the "area image conversion” here refers to conversion between the image of the designated frame area and the image of the grid area.
  • a grid is superimposed on the designated frame.
  • a label of the designated frame image is associated with each area image (width Wg x height Hg) defined by the grid in Fig. 7A within the range covered by the designated frame.
  • the grid area image may be associated with the same label as the designated frame area image.
  • the "predetermined overlap amount” is, for example, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, or 100% of the area of the grid area, and may be within a range between any two of the numerical values illustrated here.
  • a grid area and a designated frame area completely overlap.
  • FIG. 7B after the area image conversion, a group of T-labeled areas with 3 rows and 2 columns and a group of N-labeled areas with 2 rows and 3 columns are generated.
  • the two grid area images in FIG. 7B are logical product (AND) area images of the two designated frames, and both the T label and the N label are associated with each other.
  • the user U002 may perform a confirmation operation (S303) for each grid area image in FIG.
  • step S106 a learning data image can be generated according to the grid in FIG. 7B.
  • FIG. 9A and FIG. 9B show an example in which there are three designated frames, so please refer to these as well.
  • step S125 the label of the designated frame area image set in the medical image Q is determined.
  • the process of step S125 can be the same as that of step S115 by compensating for the difference between the grid area and the designated frame area.
  • the designated frame may be automatically selected without converting the area image.
  • the designated frame image and label with the higher annotation performance value may be automatically adopted.
  • a learning data image can be generated according to the designated frame image.
  • the generation unit 23 may perform region image conversion. After the region image conversion, a state is created in which the label is associated with the grid region image, as shown in FIG. 7B. After this, the process of step S115 can be similarly applied. Then, in step S106, a learning data image can be generated according to the grid in FIG. 7B.
  • the computer may execute the label determination process described below in step S125.
  • the overlap between the designated frame area image and the grid area image may differ for each designated frame.
  • T and N labels are associated with two grid area images, but referring to FIG. 7A, in the grid area image on the right side of the figure, the designated frame area image with the N label covers the entire grid area image, while the designated frame area image with the T label does not extend to the entire grid area image.
  • Such differences in overlap between the grid area image and the designated frame area image may be taken into account in the label determination process.
  • the label of the designated frame area image with the largest overlap area may be adopted as the label of that grid area image.
  • the difference in overlap area between the designated frame area image with the largest overlap area and the other designated frame area images is equal to or greater than a threshold, the label of the designated frame area image with the largest overlap area may be used as the label of the grid area image, and if the difference in overlap area is less than the threshold, label determination based on the overlap area may not be used and determination processing may be performed based on other criteria. This makes it possible to lower the importance of designated frame area images with a relatively small overlap area.
  • the process described here can also be used in cases where there are three designated frame images as in Figures 9A and 9B, or four or more designated frame images.
  • the first embodiment can be modified in various ways, at least as described below.
  • index value may be incorporated into the control of the first embodiment.
  • the index value is a parameter that can be associated with each of the multiple pieces of information.
  • the index value indicates the likelihood of the content of each of the multiple pieces of information. This has the advantage that more accurate learning data can be generated based on the index value.
  • the multiple pieces of information may be index values themselves, or may be provided with index values. This will be specifically described below.
  • FIG. 10A is an example of a label image in which the label has an index value
  • FIG. 10B is an example of a label image in which the label is an index value.
  • the first label La k and/or the second label La k associated with the region image as the annotation target has an index value or may be an index value.
  • the index value in this example indicates how likely the label classification result is.
  • the index value may be manually input by the user during annotation work on the work screen 26, for example.
  • the index value can be adopted in both grid-based and designated frame-based annotations.
  • the second label Lb k in FIG. 10A is written as "N70", which indicates that the label classification content is "N" (non-tumor) and the index value (probability) is 70 (%).
  • the index value may be manually input by the users U001 and U002 in steps S301a and S302b in Fig. 4 and Fig. 5.
  • the index value may be arbitrarily set by the user in any increment (for example, a fixed increment such as 1% increment or 5% increment), or the index value may be presented as, for example, a plurality of options (for example, an arbitrary number of stages such as three or four stages), from which the user may select the desired index value.
  • the three-stage index value may be "high” (for example, 70% or more), “medium” (for example, 31% to 69%), and "low” (for example, 30% or less). This has the advantage of improving the convenience of the user in inputting the index value.
  • labels with index values may be associated with area images in the selection screen 27 as shown in FIG. 10A and FIG. 10B.
  • the user can visually recognize and compare the index values of each label.
  • the user can determine the final label for each area image Q k while taking into consideration the high and low index values.
  • the label with the highest index value among the multiple labels may be highlighted and displayed.
  • the method of highlighting may be, for example, attaching an arbitrary badge to the label, highlighting (emphasis display), or displaying the label itself relatively large.
  • a label determination process based on the index value may be executed by the computer.
  • a label having the largest index value among a plurality of labels may be adopted as a determined label.
  • a difference in index value between mismatched labels may be calculated, and if the difference is equal to or greater than a predetermined reference value, the label having the relatively larger reference value may be adopted as a determined label.
  • the area image to which the label is associated may not be included in the learning data, or the user may decide which label to adopt for the area image in a manual operation selection process (for example, S104 and S105 in FIG. 4 may be used).
  • a manual operation selection process for example, S104 and S105 in FIG. 4 may be used.
  • the first and second information are adopted as an example of the "plurality of information".
  • the first to third pieces of information may be used to associate the region image with the label.
  • a third label may be associated with the region image Q k by a third annotation of another third user, and this third label may be acquired together with the first and second labels in step S103.
  • the computer (system 1) may be caused to execute a process of determining a definite label by the first to third labels for each region image Q k (steps S104, S105 in FIG. 4 or S115 in FIG. 5), and the computer (system 1) may be caused to execute a process (S106) of generating learning data in which the definite label is associated with the region image.
  • the statement "determine the definite label from the first to third labels” includes the meaning of "determine the definite label from the options ('T' or 'N') presented by the first to third labels.”
  • the first to third annotations are given as examples for convenience, and the first to third labels are assumed to be applicable to "multiple pieces of information". However, there is no upper limit to the number of "multiple pieces of information" that can be adopted in the first embodiment. Any number of annotations by any number of users may be adopted in the process flows of Figures 4 to 6 and 8. If the number of annotations that can be adopted in the process flow of Figure 4 or Figure 5 in system 1 is "Z", the value of Z may be, for example, 2 to 100, for example, 3 to 20, or for example, 4 to 10. Specifically, the value of Z may be, for example, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 30, 40, 50, 75, or 100, and may be within the range of any two of the numerical values given as examples here.
  • the system 1 of the embodiment may be used by limited members, for example, only members of a certain company, laboratory, or research group. However, without being limited to this, the system 1 may be capable of accepting user account registrations from members of many more various research institutes, universities, or companies, and may be accessible to users in all regions of the world, and therefore may be made widely available to the general public via the communication network 2. This has the advantage that annotation data can be collected from many more various users. However, in such cases, not all users are necessarily willing to disclose their own user information. To address this issue, the system 1 of the embodiment has a function for restricting annotation disclosure and/or a "display/hide function for annotation subject information" described below.
  • the system 1 may adopt all the labels associated with the area image Q k and execute the determination process (S104, S105 in FIG. 4 and FIG. 6, S115 in FIG. 5, and S125 in FIG. 8), but is not limited thereto.
  • the learning data generation command S302
  • the user may be able to specify a group of labels to be adopted in the learning data generation process, that is, the user may be able to specify who made the annotations and when to adopt them.
  • the system 1 may display on the user terminal 3 the user IDs of the labels associated with the area image IDs to be used for generating learning data.
  • the user terminal 3 may accept a selection of the user ID and transmit the specified user ID to the system 1.
  • a process of displaying the first to third labels in one grid region image Q k on the selection screen 27 of the process flow in Fig. 4 may be executed by a computer (the system 1, specifically the display control unit 21). A user can select one of the first to third labels. The same applies to the case where four or more pieces of information are adopted.
  • the selection screen 27 may be presented in a form that allows the user to visually recognize the difference in the number of labels belonging to one of the label classification contents (e.g., "T") and the other (e.g., "N") among three or more labels.
  • one "T” label and one "N” label may be displayed superimposed on each grid area image Q k , and the difference in the number of labels may be expressed in each label image.
  • the number of "T” labels and the number of "N” labels may be presented as numerical values or the like (the numerical values may be displayed superimposed on the grid area image), or the label with the larger number may be displayed prominently.
  • the method of highlighting may be the same as the example described in "1-6-1-2. Example of application to the processing flow of FIG. 4" above.
  • the three designated frame area images may each have a first to third label (see FIG. 9A).
  • each of the application examples of FIG. 4 above can be adopted for the grid area image after area image conversion (see FIG. 9B).
  • a majority decision process may be executed by a computer (the system 1, specifically, the label decision unit 23b).
  • the majority decision process for example, the number of "T" labels and the number of "N" labels in the first to third labels in each area image may be counted, and the larger number of "T"s or "N”s in each area image may be adopted as the confirmed label.
  • the majority decision process is also applicable when area image conversion is performed in the process flow of Fig. 8 (see Fig. 9B).
  • the majority process can also be adopted when four or more pieces of information (i.e., first to fourth labels, etc.) are used. However, if the number of "plural pieces of information" is an even number, the mismatched labels may be balanced at half each.
  • the annotation performance value may be used to determine the final label, for example. Specifically, for example, the classification content of the label with the largest annotation performance value may be determined as the final label.
  • the total annotation performance value of the user who assigned the "T" label may be compared with the total annotation performance value of the user who assigned the "N" label, and the label with the larger total value may be determined as the final label.
  • the area image with the mismatched labels may not be included in the learning data.
  • the computer (system 1) may be caused to execute a process of accepting a label selection from the user for the area image Q k with the mismatched labels in a manual operation selection process (for example, S104 and S105 in FIG. 4 may be diverted).
  • a computer may execute a calculation process based on index values.
  • the calculation process based on index values may be, for example, arithmetic operations (e.g., addition, etc.), or summary statistics calculations (e.g., average value calculation or maximum value calculation).
  • the index values of the "T" label among the first to third labels may be added to obtain a first sum
  • the index values of the "N" label may be added to obtain a second sum
  • the larger of the first and second sums may be used as the determined label.
  • the first and second average values may be calculated instead of the first and second sums, and the larger of the first and second average values may be used as the determined label.
  • the label with the largest index value among the first to third labels may be used as the determined label. The same applies to four or more pieces of information (labels).
  • the calculation process based on index values is also applicable when area image conversion is performed in the process flow of FIG. 8 (see FIG. 9B).
  • a plurality of area images Q k may include an area image with a missing label (i.e., an annotation omission). Specifically, for example, although many other (e.g., the majority) area images have N labels associated with them, some area images may have less than N labels associated with them or the number of labels may be zero. In this case, as an example, if at least one label is associated with an area image with a missing label, a process of determining a definite label may be performed using the at least one label.
  • the area image may be excluded from the target of learning data generation, or the user may be prompted to input a label replenishment by an annotation reception process (e.g., S101a, S301a, and S102a in FIG. 4 may be used) for the area image.
  • an annotation reception process e.g., S101a, S301a, and S102a in FIG. 4 may be used
  • the annotation subject information may be displayed on the selection screen 27.
  • the annotation subject information is information that can identify who is the annotation subject who inputted the labels of each label.
  • the annotation subject information may be a user ID or the like, or may be the user's name, affiliation, nickname, avatar icon, or the like. However, on the other hand, the annotation subject information may be hidden. As an example, the annotation subject information of the annotation called up may not be displayed on the selection screen 27.
  • the user may be able to set the annotation to "disclosure allowed" or "non-disclosure allowed".
  • the annotation may be displayed on the selection screen 27 under certain conditions (i.e., on the condition that the annotation subject information is not displayed). By hiding the annotation subject information, the annotation subject can be kept secret. Even if a user is not favorable to disclosing his/her user information, there is an advantage that annotation data can be utilized while ensuring anonymity.
  • the subject that performs the first and second annotations is not limited to a human being and may include a computer.
  • a computer performs image detection processing using rule-based classification processing or machine inference classification of a learned discriminant model, the classification results are associated with each area image, which may be considered as an annotation. Multiple annotations may be made by different or the same subject.
  • a user ID for the computer may be set in the account DB, and an annotation subject ID unique to each computer may be assigned when registered in the label DB.
  • Second embodiment> the description of the configuration that is the same as or similar to the first embodiment will be omitted or simplified, and the following description will focus on the differences.
  • the system 1 of the second embodiment shown in Fig. 11A includes a storage unit 20, a display control unit 21, an input unit 22, a generation unit 23, and a discrimination unit 25.
  • the system 1 of the first embodiment differs from the system 1 of the first embodiment in that it includes the discrimination unit 25.
  • the generation unit 23 of the second embodiment shown in Fig. 11B includes a data acquisition unit 23a, a label determination unit 223b, and a learning data configuration unit 23c.
  • the second embodiment differs from the system 1 of the first embodiment (see Fig. 3A) in that it includes a label determination unit 223b that executes an automatic labeling process instead of the label determination unit 23b that determines the label selected by the user as the determined label.
  • the discrimination unit 25 is a trained discrimination model constructed by an arbitrary machine learning algorithm.
  • the machine learning algorithm for constructing the discrimination unit 25 there is no limitation on the machine learning algorithm for constructing the discrimination unit 25, and various known techniques can be adopted, such as a neural network, a decision tree, a support vector machine, or a logistic regression analysis, or may be, for example, an arbitrary deep learning algorithm (DNN), or may be, for example, an arbitrary CNN model.
  • DNN deep learning algorithm
  • the discrimination unit 25 has already completed learning using other learning data.
  • the discrimination unit 25 is capable of executing image recognition processing on input image data, and is constructed to be capable of executing machine inference to classify the input image data.
  • (3-3. Processing flow) 12 and 13 are specific examples of the processing flow of the second embodiment, and provide a program and a method for generating learning data.
  • a generation process different from that of the first embodiment is executed by a computer (system 1) (see S152 to S157 in FIG. 12, or S152, S153, S164, S156, and S157 in FIG. 13).
  • the generation process of the second embodiment is an automatic process in which the computer (system 1) associates the area image with a label using a plurality of pieces of information (first and second information) associated with the area image, and generates learning data.
  • the first and second information are also associated with the area image in each area image. Since the label association can be performed automatically using a plurality of pieces of information associated with the area image, learning data can be generated with high convenience.
  • the above “plurality of information” includes first and second information, “image-attached information G” is adopted as the first information (see FIG. 12), and “inference classification result by a learned discrimination model” is adopted as the second information (illustrated by labels Lc k and Ld k in FIG. 12).
  • the image-attached information G is information that is attached to a medical image (the entire medical image). As a typical example, it may be various types of medical information that explain the details of the medical image, or it may be information on the patient from whom the medical image was obtained, and specific variations will be described later.
  • Step S151 12 and 13 have in common that a process of receiving input data including a plurality of region images is executed by a computer (system 1) (S151).
  • the storage unit 20 stores a first medical image Q and a second medical image R.
  • the plurality of region images QM k , RM k can be generated from each of the first medical image Q and the second medical image R.
  • the plurality of region images QM k , RM k may be obtained by cutting out each of the first and second medical images Q, R to a uniform size according to a grid.
  • the plurality of region images QM k , RM k may be obtained by cutting out each of the first and second medical images Q, R according to one or more designated frames set in each of the first and second medical images Q, R.
  • the first and second medical images Q, R are associated with first information (image-attached information G).
  • a gene mutation type is adopted as an example of the image-attached information G.
  • the genetic mutation type of each medical image registered in the image DB is known in advance at the stage of registration in the storage unit 20 (image DB) by various tests called genetic mutation tests, cancer panel tests, etc.
  • the storage unit 20 stores the genetic mutation type (A type applicable/not applicable, B type applicable/not applicable, etc.) in association with each medical image.
  • the first information is "gene mutation type," and the predetermined criterion (labeling criterion) is "whether or not it corresponds to gene mutation type A.”
  • the first medical image Q is an image in which the first information satisfies the predetermined criterion.
  • the second medical image R is an image in which the first information does not satisfy the criterion.
  • FIG. 12 shows the image-related information G as a schematic table, in reality the image-related information G is stored in the image DB in association with the image ID.
  • the type of gene mutation may be, for example, EGFR gene mutation, HER2 gene amplification (ERBB2 copy number abnormality), RAS gene mutation (KRAS and NRAS), BRAF gene mutation (V600E or V600K), ALK fusion gene, ROS1 fusion gene, NTRK fusion gene, high frequency microsatellite instability (MSI-High), BRCA gene mutation, or gene mutation burden (TMB).
  • EGFR gene mutation HER2 gene amplification (ERBB2 copy number abnormality), RAS gene mutation (KRAS and NRAS), BRAF gene mutation (V600E or V600K), ALK fusion gene, ROS1 fusion gene, NTRK fusion gene, high frequency microsatellite instability (MSI-High), BRCA gene mutation, or gene mutation burden (TMB).
  • Gene mutation type A it is sufficient to predetermine what specific types of gene mutations are qualitatively and/or quantitatively considered to be "gene mutation type A", thereby making it possible to distinguish whether or not they are type A.
  • Each type (A, B, C, etc.) may be registered in the storage unit 20, and it may be possible to selectively set whether or not each type applies when registering a medical image in the storage unit 20.
  • the user U001 specifies the first and second medical images Q and R, specifies "relevant/non-relevant to gene mutation type A" as the first information, and specifies “tumor/non-tumor discrimination” as the second information, and the region images QM k and RM k generated from these are processed.
  • any medical image or any area image may be specified, and/or the first and second information may be specified arbitrarily. This further improves convenience.
  • the user U001 may be requested to enter several setting items.
  • the user may be allowed to arbitrarily make at least one of a request to specify a medical image or area image, a request to specify the first information, and a request to specify the second information to the user U001, and the content of the request may be transmitted from the user terminal 3 to the system 1.
  • step S153 the computer (system 1) executes a process of making the learned discrimination model (discrimination unit 25) perform inference to classify the region images QM k and RM k of the input data.
  • the inference classification result for each region image is "T” or "N”, and this inference classification result is the "second information" of the second embodiment.
  • each region image may simply be classified into "T” or "N”, and the classification result of each region image may be provided to the subsequent processing (S154).
  • step S154 the label determination unit 223b executes a selection process for the area images QM k and RM k .
  • the label determination unit 223b executes a selection process for the area images QM k and RM k .
  • only area images to which the "T" label is assigned are extracted.
  • area images QM k and RM k to which the "T" label is assigned are passed to the next step S156.
  • area images to which the "N" label is assigned are exemplarily described as "QM j , RM m ", and labels associated with these are exemplarily described as Lc j , Ld m (j, m are integers).
  • Lc j area images to which the "N” label is assigned
  • Ld m exemplarily described as integers.
  • step S156 the label determination unit 223b executes an automatic labeling process. That is, in the process of step S156, the computer (system 1, specifically the label determination unit 223b) associates a label Le k with one or more region images (region images QM k after the selection in S154) generated from the first medical image Q and classified into the first classification group ("T") as a classification result, and associates a label Lf k with one or more region images (region images RM k after the selection in S154) generated from the second medical image R and classified into the first classification group ("T") as a classification result.
  • Information such as which of the first and second medical images QM k and R each region image RM k was generated from has already been stored in the image DB (see S151).
  • the label Le k indicates that the first information of the region image QM k satisfies the above-mentioned predetermined criterion (corresponding to gene mutation type A).
  • the label Lf k indicates that the first information of the region image RM k does not satisfy the criterion.
  • the labels Le k and Lf k become correct answer data.
  • the region image of the input data is classified by the learned discrimination model, and the label determined by the first information (image-attached information G) can be automatically associated with the region image classified into the first group as the classification result (second information).
  • image-attached information G image-attached information
  • the first information (image-attached information G) can be set in various ways.
  • the medical image employed in the second embodiment is a pathological image
  • the first information may be a gene mutation type, a TMB-H (high tumor mutation load) type, the results of a biochemical analysis, or a patient prognosis prediction.
  • the classification type and classification content of the label determined by the first information in step S156 can be specified in various ways.
  • the classification content of the label may be, for example, whether the gene mutation type corresponds to a predetermined type (e.g., type A), whether it corresponds to a TMB-H (high tumor mutation load) type, whether the results of the biochemical analysis correspond to a predetermined standard, or whether the patient prognosis prediction is a predetermined standard value or more.
  • the patient prognosis prediction may be, for example, whether the life prognosis is a predetermined period or more.
  • the first information may be the patient's age, sex, medical history, the results of the immunohistochemical analysis, or lifestyle.
  • step S157 the computer (system 1) is caused to execute a process of generating learning data using the area images and labels associated in step S156.
  • the generated learning data is registered in the learning data DB, and user U001 can download it at will.
  • the process flow in Fig. 12 causes a computer (system 1) to execute a generation process.
  • the computer (system 1) generates learning data by associating a label (see labels Le k and Lf k in S156 of Fig. 12) determined based on the first information (presence or absence of a gene mutation type A: S151) with a region image (see region images QM k and RM k in S154) that has been sorted based on the second information (tumor/non-tumor inference classification result: S153).
  • Learning data in which a label has been determined based on the first information and sorted based on the second information can be automatically generated.
  • Fig. 13 is a second example of a process flow showing the control operation of the second embodiment (processing using index values). Specifically, the difference between the flows in Fig. 12 and Fig. 13 is that step S154 is replaced with step S164. The other steps S151 to S153, S156, and S157 are the same as those in the flow in Fig. 12.
  • the discrimination unit 25 outputs a probability value in step S153.
  • Some known image recognition models using machine learning algorithms can output the likelihood (i.e., probability or degree of agreement) of a classification result by image recognition.
  • the discrimination unit 25 is a neural network model constructed using a deep learning algorithm.
  • the output layer of the neural network outputs a probability value representing the probability that the input image data belongs to each classification result as an inference result. In the example of FIG. 13, this probability value is used as the index value.
  • the trained discrimination model outputs a probability value representing the likelihood of the classification results of the region images QM k and RM k .
  • step S164 the computer (the system 1, specifically the label determination unit 223b) executes a selection process for the region images QM k and RM k .
  • the computer (the system 1) is caused to extract only region images for which the classification result of the inference classification process (S153) is "T" (tumor) and the index value (probability value) is inferred to be equal to or greater than a predetermined value.
  • the predetermined value is 0.80.
  • This predetermined value is, for example, 0.70 to 1.00, and may be, for example, 0.80 to 1.00. Specifically, this predetermined value may be, for example, 0.50, 0.55, 0.60, 0.65, 0.70, 0.75, 0.80, 0.85, 0.90, or 0.95, and may be within a range between any two of the values exemplified here.
  • the numerical values exemplified here may be expressed as a percentage after converting 1.00 to 100%.
  • the discrimination unit 25 is built into the system 1, but this is merely an example and is not limited to such a form.
  • the discrimination unit 25 does not have to be built into the system 1.
  • the external system may be used as the discrimination unit 25 to execute the process of step S153.
  • the following criteria for substantial identity of images and database management can be adopted for the identity between multiple medical images.
  • Examples of cases in which two medical images are substantially identical include at least the following first and second situations. In the first situation, the two medical images are completely identical and have the same image data. In the second situation, even if there are differences between the medical images (e.g., differences in image data and/or image IDs), the differences can be regarded as identical for annotation purposes.
  • a medical image is obtained by photographing the same medical image subject.
  • a new medical image is generated by duplicating the medical image data, changing the image format after duplication (e.g., saving in a compressed format), or applying image processing after duplication (resizing, changing brightness, contrast, saturation, color tone, etc.).
  • the original medical image and the new medical image are substantially identical even if there are differences in the image ID, image data, etc. This is because the subject photographed in the medical image is the same, and the differences can be ignored for annotation purposes.
  • the same applies to area images, and the substantially identical standard can be adopted for area images generated from substantially identical medical images.
  • a mechanism known as a "virtual slide” may be used.
  • the same subject pathological specimen, i.e. tissue
  • pathological images are registered in an image DB. Since digitized pathological specimen data can be used, this has the advantage of making it easier to improve the pathologist's observation accuracy.
  • a group of pathological images in a virtual slide are essentially identical, since they merely represent the same pathological specimen at different magnifications. This concept is similar for any medical images in which the same subject is photographed at different magnifications.
  • a substantially identical standard can also be adopted for area images generated from these substantially identical medical images.
  • a pathological slice may have an almost identical structure to a pathological slice directly above or below it.
  • the pathological images of each of these pathological slices are substantially identical.
  • Similar cases are also anticipated for all medical images other than pathological images. For example, when a three-dimensional structure is sliced, it is anticipated that the planar images of each adjacent layer are substantially identical. In these cases as well, multiple medical images are substantially identical. The substantially identical standard can also be adopted for regional images generated from these substantially identical medical images.
  • the embodiment can employ at least the "criteria for substantial image identity" described in each of the above examples.
  • the image IDs of medical images that are to be treated as substantially identical can be linked to each other by manual user operation or automatic computer processing so that they can be treated as a group together in the image DB of the storage unit 20.
  • a first user e.g., U001
  • a second user e.g., U002
  • the fourth medical image is a copy of the third medical image, or a copy of the third medical image that has been subjected to image processing.
  • the third and fourth medical images are linked in the image DB, and the two images may be treated as the same.
  • each region image of a substantially identical medical image when multiple labels (e.g., first and second labels) are associated with each region image of a substantially identical medical image, several examples of ID management in each DB of the storage unit 20 are possible.
  • multiple substantially identical medical images may be annotated using a grid method, or a grid may be set using a designated frame method of annotation.
  • the grids for each medical image may be set to the same size and with the same coordinates.
  • the DB in the storage unit 20 may associate a label ID for each of a plurality of annotations with one area image ID.
  • a group of area image IDs may be commonly adopted for a plurality of medical images.
  • an area image ID associated with a certain label first label
  • second label second label
  • a second example of ID management may be as follows. There may be cases where a third medical image (master medical image) is copied to generate one or more fourth medical images (copy medical images). Furthermore, there may be cases where image processing is performed on the fourth medical image. In at least these cases, the third and fourth medical images are essentially the same image, but have different image data and may be assigned different image IDs. Thus, as a second example of ID management, the image IDs of the third and fourth medical images may be linked in the image DB.
  • an image group consisting of a master medical image ID and a group of one or more copy medical image IDs
  • this association may be followed and the same label ID may be automatically associated with the medical images of the remaining image IDs.
  • a medical image ID, a region image ID, and a label ID may be associated directly as in the first example, or indirectly as in the second example. This allows multiple annotations to be performed on substantially identical medical images, and multiple labels may be associated with one region image using multiple annotations.
  • first and second medical images Q and R in the second embodiment are different images and are not substantially identical.
  • the system 1 is illustrated as one information processing device, but the embodiment is not limited thereto.
  • the system 1 may be configured with one or more physical servers, may be configured with a virtual server operating on a hypervisor, or may be configured with a cloud server.
  • the processing of the program related to the generation and display of the selection screen 27 can be arbitrarily shared between the system 1 (server 1) and the user terminal 3. Please note that the way of sharing the processing in each processing flow of FIG. 4 to FIG. 6, FIG. 8, FIG. 12, and FIG. 13 is one example.
  • a computer should be understood to mean “at least one computer” unless otherwise expressly limited.
  • a “computer” may include any one or more of a user terminal and/or a server.
  • the entire execution process may be completed by a single computer, or a part of the execution process may be executed by a first computer (e.g., a server) and the rest of the process may be executed by one or more other computers (e.g., a user terminal or other servers).
  • the other one or more computers may be any one or more of the second to nth computers (n is any integer of 3 or more).
  • the selection screen generation process may be executed by a user terminal.
  • the selection screen generation process may be executed by a server, in which case the completed selection screen data may be sent to the user terminal.
  • the selection screen generation process may be executed by a part of the process by the user terminal and the remaining process by the server.

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Theoretical Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Medical Informatics (AREA)
  • Radiology & Medical Imaging (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Computing Systems (AREA)
  • Evolutionary Computation (AREA)
  • Databases & Information Systems (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Epidemiology (AREA)
  • Primary Health Care (AREA)
  • Public Health (AREA)
  • Quality & Reliability (AREA)
  • Medical Treatment And Welfare Office Work (AREA)
PCT/JP2024/019506 2023-05-31 2024-05-28 プログラム、生成方法及びシステム Ceased WO2024247990A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2025524101A JPWO2024247990A1 (cg-RX-API-DMAC7.html) 2023-05-31 2024-05-28
EP24815473.4A EP4723054A1 (en) 2023-05-31 2024-05-28 Program, generation method, and system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2023-090399 2023-05-31
JP2023090399 2023-05-31

Publications (1)

Publication Number Publication Date
WO2024247990A1 true WO2024247990A1 (ja) 2024-12-05

Family

ID=93657381

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2024/019506 Ceased WO2024247990A1 (ja) 2023-05-31 2024-05-28 プログラム、生成方法及びシステム

Country Status (3)

Country Link
EP (1) EP4723054A1 (cg-RX-API-DMAC7.html)
JP (1) JPWO2024247990A1 (cg-RX-API-DMAC7.html)
WO (1) WO2024247990A1 (cg-RX-API-DMAC7.html)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021095698A1 (ja) * 2019-11-15 2021-05-20 ソニーグループ株式会社 生成装置および生成方法
WO2021206053A1 (ja) * 2020-04-06 2021-10-14 国立大学法人 新潟大学 データ生成装置、データ生成方法及びプログラム
WO2022209501A1 (ja) * 2021-03-31 2022-10-06 富士フイルム株式会社 情報処理装置、情報処理装置の作動方法、情報処理装置の作動プログラム

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021095698A1 (ja) * 2019-11-15 2021-05-20 ソニーグループ株式会社 生成装置および生成方法
WO2021206053A1 (ja) * 2020-04-06 2021-10-14 国立大学法人 新潟大学 データ生成装置、データ生成方法及びプログラム
WO2022209501A1 (ja) * 2021-03-31 2022-10-06 富士フイルム株式会社 情報処理装置、情報処理装置の作動方法、情報処理装置の作動プログラム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4723054A1 *

Also Published As

Publication number Publication date
JPWO2024247990A1 (cg-RX-API-DMAC7.html) 2024-12-05
EP4723054A1 (en) 2026-04-08

Similar Documents

Publication Publication Date Title
Ladbury et al. Integration of artificial intelligence in lung cancer: Rise of the machine
Balagurunathan et al. Requirements and reliability of AI in the medical context
Yousefzadeh et al. ai-corona: Radiologist-assistant deep learning framework for COVID-19 diagnosis in chest CT scans
Prior et al. Open access image repositories: high-quality data to enable machine learning research
Turner et al. Society of toxicologic pathology digital pathology and image analysis special interest group article*: opinion on the application of artificial intelligence and machine learning to digital toxicologic pathology
US20220114213A1 (en) System and method for clinical decision support for therapy planning using case-based reasoning
US20210133231A1 (en) Control method and non-transitory computer-readable recording medium for comparing medical images
JP6275876B2 (ja) 医療データ処理のための進化型文脈的臨床データエンジン
Bizzego et al. Integrating deep and radiomics features in cancer bioimaging
JP5317716B2 (ja) 情報処理装置および情報処理方法
JP5843414B2 (ja) 医療記録ソフトウエアと高度画像処理の統合
US11195610B2 (en) Priority alerts based on medical information
Xie et al. Deep learning for image analysis: Personalizing medicine closer to the point of care
US20220037018A1 (en) Automatically determining a medical recommendation for a patient based on multiple medical images from multiple different medical imaging modalities
CN120584363A (zh) 使用深度基础模型处理电子图像的系统和方法
US11626209B2 (en) Diagnosis support apparatus, diagnosis support system, diagnosis support method, and non-transitory storage medium
JP5677521B2 (ja) 情報処理装置、情報処理方法、プログラムおよび記憶媒体
US20230146840A1 (en) Method and apparatus utilizing image-based modeling in clinical trials and healthcare
US12136484B2 (en) Method and apparatus utilizing image-based modeling in healthcare
EP4465300A1 (en) Predicting medical outcome via artificial intelligence for use by a randomization algorithm
US20250372224A1 (en) Machine learning predictive models of treatment response
US20250259750A1 (en) Multi-modal machine learning to determine risk stratification
WO2023232758A1 (en) Machine learning predictive models of treatment response
US11037659B2 (en) Data-enriched electronic healthcare guidelines for analytics, visualization or clinical decision support
Levy et al. Artificial intelligence, bioinformatics, and pathology: emerging trends part I—an introduction to machine learning technologies

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24815473

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2025524101

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 2024815473

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2024815473

Country of ref document: EP

Effective date: 20260102

ENP Entry into the national phase

Ref document number: 2024815473

Country of ref document: EP

Effective date: 20260102

ENP Entry into the national phase

Ref document number: 2024815473

Country of ref document: EP

Effective date: 20260102

ENP Entry into the national phase

Ref document number: 2024815473

Country of ref document: EP

Effective date: 20260102

ENP Entry into the national phase

Ref document number: 2024815473

Country of ref document: EP

Effective date: 20260102

ENP Entry into the national phase

Ref document number: 2024815473

Country of ref document: EP

Effective date: 20260102

WWP Wipo information: published in national office

Ref document number: 2024815473

Country of ref document: EP