Classifications

• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H30/00—ICT specially adapted for the handling or processing of medical images
  • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H15/00—ICT specially adapted for medical reports, e.g. generation or transmission thereof
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H30/00—ICT specially adapted for the handling or processing of medical images
  • G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
  • G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
  • G16H40/63—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for local operation
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
  • G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
  • G16H40/67—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for remote operation
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
  • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
  • G16H50/70—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for mining of medical data, e.g. analysing previous cases of other patients

Definitions

• the present invention relates to a program, method, and system that supports annotation of medical images.
• the data generation device in Patent Document 1 generates training data for a machine learning algorithm.
• the training data consists of multiple images and labels associated with each image.
• annotation can refer to the act of labeling the image itself or a specified area within the image, or it can refer to a group of assigned labels.
• the present invention provides a program, support method, and system that are configured to present information that is useful when performing annotation.
• a program for supporting annotation of medical images the program causing a computer to execute a presentation process to generate a presentation screen, the presentation screen presenting to a user the classification content of at least one label associated with at least one region image in the medical image by at least one of a plurality of annotations.
• the multiple annotations include a first and a second annotation
• the first annotation is made by a first user
• the second annotation is made by a second user or made by the first user earlier than the first annotation
• the presentation process causes a computer to generate the presentation screen so as to present to the first user the classification content of the at least one label associated with the at least one area image by the first and second annotations while the first user is making the first annotation or after the first annotation is completed.
• the at least one area image includes a label-comparable area image, the label-comparable area image being an area image to which a plurality of labels are associated by the plurality of annotations, and the presentation process causes a computer to generate the presentation screen so as to present the classification contents of the plurality of labels associated with the label-comparable area image by the plurality of annotations in a manner that allows a user to visually compare them.
• a program according to any one of [1] to [5], wherein the at least one area image includes a label-comparable area image, the label-comparable area image being an area image to which a plurality of labels are associated by the plurality of annotations, the program causing a computer to execute a process of calculating a comparison result of comparing classification contents of the plurality of labels, and the presentation screen presents the comparison result to a user, thereby presenting the classification contents of the plurality of labels associated with the label-comparable area image.
• the presentation screen has a region image and a label image, the region image is an image of the at least one region in the medical image, and the label image is an image corresponding to a label associated with the at least one region image, and the program causes a computer to execute a process of changing a display mode of the region image and/or the label image on the presentation screen according to classification content and/or according to a comparison result between a plurality of labels when the plurality of labels are associated with the region image.
• a program according to any one of [1] to [8], wherein the classification content of the label includes an index value or is an index value, and the index value indicates how likely the classification content of the label is.
• a method for supporting annotation of medical images comprising: causing a computer to execute a presentation process for generating a presentation screen, the presentation screen presenting to a user the classification content of at least one label associated with at least one region image in the medical image by at least one of a plurality of annotations.
• a system for supporting annotation of medical images comprising a presentation unit that generates a presentation screen, the presentation screen presenting to a user the classification content of at least one label that is associated with at least one region image in the medical image by at least one of a plurality of annotations.
• the presentation screen can present information that is useful when performing annotation.
• FIG. 1 is a diagram illustrating an example of a system 1 (server 1), each user terminal 3, and a diagnostic device 4 according to an embodiment.
• Fig. 2A is a block diagram showing an example of a hardware configuration of the server 1.
• Fig. 2B is a block diagram showing an example of a hardware configuration of the user terminal 3.
• Fig. 3A is a block diagram showing the functional configuration of the server 1.
• Fig. 3B is a functional block diagram of a presentation unit.
• 4 is a control flow showing a control operation of the embodiment.
• 1 shows an example of a work screen 26 for performing grid-based annotation.
• 13 shows an example of a scene in which input on the presentation screen 27 is accepted to issue a presentation instruction SN1 on the work screen 26 in the case of grid-type annotation.
• FIG. 1 shows an example of a work screen 26 for performing grid-based annotation.
• 13 shows an example of a scene in which input on the presentation screen 27 is accepted to issue a presentation instruction SN1 on the
• FIG. 6 shows an example of a presentation screen 27 in parallel presentation mode (number of calls: 1) and layer display.
• FIG. 6 shows an example of a presentation screen 27 in parallel presentation mode (number of calls: 2) and layer display.
• FIG. 6 shows an example of a presentation screen 27 in a parallel presentation mode (number of calls: 2) and in a window display.
• FIG. 6 shows an example of a presentation screen 27 in a simple presentation mode (number of calls: 1) and in a window display.
• FIG. 6 shows an example of a presentation screen 27 in a simple presentation mode (number of calls: 2) and in a window display.
• FIG. 6 shows an example of a presentation screen 27 in a simple presentation mode (number of calls: 2) and in a window display.
• FIG. 6 shows an example of the presentation screen 27 in the comparison presentation mode (number of calls: 2) and in layer display, in which the annotation being worked on by the user is compared with one other annotation.
• FIG. 6 shows an example of a presentation screen 27 in a comparative presentation mode (number of calls: 2) and in a window display.
• FIG. 6 shows an example of a presentation screen 27 in a comparison presentation mode (number of calls: 2) and window display, in which annotations of two other users (U001, U002) other than the logged-in user (U010) are compared.
• FIG. 6 shows an example of the presentation screen 27 in the comparison presentation mode (number of calls: 2) and in layer display, in which two annotations other than that of the logged-in user are compared.
• FIG. 6 shows an example of the presentation screen 27 in the selective presentation mode (number of calls: 2, displaying only mismatched labels) and in a window display.
• FIG. 6 shows an example of the presentation screen 27 in the selective presentation mode (number of calls: 2, displaying only tumor labels) and layer display.
• FIG. 6 shows an example of a presentation screen 27 in parallel presentation mode (number of calls: 2) and in a pop-up display.
• an arbitrary designation frame type (bounding type) annotation an example of a scene in which input of the presentation screen 27 is accepted to issue a presentation instruction SN1 on the work screen 26 is shown.
• FIG. 19 shows an example of a presentation screen 27 in simple presentation mode (number of calls: 1) and layer display.
• FIG. 21A illustrates a part of the presentation screen 27 in the simple presentation mode (number of calls: 1) with the grid setting ON in Fig. 19 and layer display.
• Fig. 21B illustrates the parallel presentation mode (number of calls: 1) in the case of Fig. 21A.
• Fig. 21C illustrates the comparative presentation mode (number of calls: 1) in the case of Fig. 21A.
• Fig. 22B illustrates the comparative presentation mode (number of calls: 1) in the case of Fig. 22A.
• Fig. 22C illustrates an example of the presentation screen 27 in the parallel presentation mode of Fig.
• Fig. 23A illustrates a part of the presentation screen 27 in the simple presentation mode (number of calls: 2) with the grid setting ON in Fig. 19 and layer display.
• Fig. 23B illustrates the parallel presentation mode (number of calls: 2) in the case of Fig. 23A.
• Fig. 23C illustrates the comparative presentation mode (number of calls: 2) in the case of Fig. 23A.
• Fig. 24A is a diagram showing another example of the grid size
• Fig. 24B is a diagram showing another example of the arbitrarily specified frame (bounding frame).
• Fig. 25A is an image display variation when comparing or contrasting annotations between self and others.
• Fig. 25B is an image display variation when comparing or contrasting multiple annotations of others.
• Fig. 25C is an additional image display variation.
• Fig. 25D is an example of a label with an index value.
• annotation refers to the act of labeling an image or a group of labels, depending on the context.
• Annotation as an annotation act means "the act of associating annotation information as correct answer data with target information (electronic information) for machine learning".
• target information electronic information
• image annotation includes object detection, region extraction, and image classification, for example.
• Image annotation associates a label with an annotation target.
• the annotation target may be each image (the entire image) or an image of a region (area) in the image.
• the label is also called a tag or metadata.
• the label represents the classification content (classification result) and becomes the correct answer data during supervised learning.
• "annotation” refers to "associating a label with a region image, which is an image of at least one region specified in an image," except when a different definition is specified or when a clear contradiction occurs in the context.
• the images to be annotated include at least “medical imaging”.
• Medical imaging includes “images that visualize the internal structure and function of the human body as images, mainly for the diagnosis and treatment of diseases".
• medical images may include plain X-ray photography, computed tomography (CT), magnetic resonance imaging (MRI), ultrasound tomography (US), nuclear medicine examination, and angiography.
• CT computed tomography
• MRI magnetic resonance imaging
• US ultrasound tomography
• nuclear medicine examination nuclear medicine examination
• angiography nuclear medicine examination
• the medical images according to the present disclosure may further include "endoscopic images” and "pathological images”. Endoscopic images may include, for example, images of the esophagus, stomach, intestines, and other digestive system images.
• pathological images may be "(microscopic) images of tissues collected by endoscopy or surgery, (1) fixed to prevent deterioration, (2) stained to facilitate observation, and (3) sliced into a few microns, and then magnified".
• the present invention is similarly applicable to any medical image, and any label classification type and classification content can be set.
• a pathological image is used as an example of a medical image
• the classification type of the label is a classification of whether or not it is a tumor
• the classification content of the label is "tumor" or "non-tumor”.
• honeycomb grid has a broad meaning, and can be, for example, a regular hexagonal honeycomb, but can also be, for example, a honeycomb structure of equilateral triangles or parallelograms, or a honeycomb structure of parallel hexagons (hexagons with three pairs of opposite sides parallel but with different lengths between each pair).
• the area specified by the grid may be called a "grid area.”
• grid-based annotation is used in the examples of Figures 5 to 18.
• the user designates an area in a desired position and range in an image using an arbitrary designated frame.
• the designated frame is sometimes called a bounding box.
• the designated frame can be set to any size and any shape, and may be, for example but not limited to, a rectangle, or may be any polygon or closed curve.
• the area designated by the designated frame may be called the "designated frame area.”
• the designated frame method annotation is adopted in the examples of Figures 19 to 24B.
• the image corresponding to that area is also called an "area image.”
• the area image is an image specified as a uniform rectangular size using a grid method, this area image may be called a tile image.
• the size of the area and area image (number of pixels vertically and horizontally) may be the same as the learning data that can be input to the learning model.
• the program, method, and system of the embodiment can employ either or both of the grid method and the designated frame method. If both are employed, the method to be used for the current annotation may be set by a user's setting operation and/or by a computer's automatic recommendation process.
• the learning dataset is composed of a "learning data image” cut out from a medical image and a label associated with the learning data image.
• the size of the learning data image (the number of pixels in the vertical and horizontal directions) is set to a size that can be input to the learning model.
• the "learning data image” may be a medical image cut out along a grid, or a part of the medical image cut out with a designated frame (bounding frame).
• the learning data image may be a medical image cut out according to a region image at the time of annotation, that is, the region image at the time of annotation and the learning data image may match.
• the learning data image may be different from the region image at the time of annotation.
• the learning data image may be generated by cutting it out with a grid (e.g., a rectangular grid with uniform intervals).
• the learning data image may be generated by cutting it out with another grid (e.g., a rectangular grid with uniform intervals).
• the system 1 has a learning data generation function (learning data generation unit).
• the learning data generation unit can generate learning data using an input image (specifically, a medical image).
• the learning data can be used as teacher data for training a learning model provided in the diagnostic device 4.
• the diagnostic device 4 uses a trained model trained using the generated learning data to determine whether or not features such as pathological abnormalities are present in the medical image.
• the system 1 may generate learning data from one medical image, or may generate learning data from multiple medical images. When generating learning data from multiple medical images, the system 1 repeats the process of specifying one or more region images for the medical image and accepting label input for each medical image. When the user has completed labeling of all medical images, the system 1 is ready to generate learning data. To generate learning data, the system 1 first cuts out the medical image to generate a "learning data image”. The system 1 generates learning data by associating the image data of each learning data image with the label assigned to each learning data image (i.e., the region image). The generated learning data is sent to the diagnostic device 4.
• the diagnostic device 4 trains a learning model using the learning data sent from the system 1.
• the learning model is, for example, a neural network that can be given a certain ability through learning.
• the diagnostic device 4 inputs a medical image to be diagnosed into the trained model generated by learning, and determines whether or not an abnormality exists in the medical image based on the output results from the trained model.
• the system 1 can provide a presentation screen 27 as a support function for supporting annotation. This support function will be explained in detail later.
• FIG. 1 is a diagram showing an example of a hardware configuration of the system 1.
• the system 1 includes a processor 11 such as a central processing unit (CPU) or a graphical processing unit (GPU), a storage device 12 such as a memory, a hard disk drive (HDD) and/or a solid state drive (SSD), and a communication interface (IF) 13 for wired or wireless communication.
• the system 1 receives user input operations from a user terminal 3 via the communication IF 13 and displays various screens on a display device 34 (see FIG. 2B) of the user terminal 3.
• the user terminal 3 includes, for example, a control unit 31 including a processor, a storage unit 32 including a non-volatile memory, a communication unit 33 for wireless communication or/and wired communication, a display device 34, a speaker 35, a microphone 36, a camera 37, and an operation device 38.
• the operation device 38 can be any input device such as a keyboard, a mouse, and/or a touch pen. There is no limitation on the specific configuration of the user terminal 3, and it may be various mobile terminals, smartphones, tablet terminals, or notebook PCs.
• the display device 34 and the operation device 38 may be integrated into the touch panel display.
• the operation device 38 and the microphone 36 serve as an input interface (or input means) for the user to perform various input operations on the user terminal 3.
• FIG. 3A is a diagram showing an example of a functional block configuration of the system 1.
• the system 1 includes a storage unit 20, a display control unit 21, an input unit 22, a generation unit 23, and a presentation unit 25.
• the storage unit 20 can be realized using a storage device 12 included in the system 1.
• the display control unit 21, the input unit 22, the generation unit 23, and the presentation unit 25 are provided by the processor 11 of the system 1 executing a program stored in the storage device 12.
• the program may be provided on a network or may be stored in a storage medium.
• the storage medium storing the program may be a non-transitory computer readable storage medium.
• the non-transitory storage medium is not particularly limited, and may be, for example, a storage medium such as a USB memory or a CD-ROM.
• the storage unit 20 includes various databases (DBs).
• DBs databases
• the storage unit 20 stores an account DB, an image DB, an area DB, an annotation DB, a label DB, and a learning data DB.
• Each DB has an identification ID assigned to the stored data, and data related to each other can be cross-referenced.
• the account DB stores each user's user ID, user details, login history, etc.
• the user details may include an email address, a name associated with the account (nickname, avatar icon, etc.), and/or more personal, specific information such as name and affiliation.
• the image DB stores one or more medical images (e.g., pathology images) used to generate learning data.
• the region DB stores information that identifies the region image in each medical image.
• the region DB stores the region image ID, the medical image ID to which the region image is set, and image coordinate data.
• the image coordinate data specifies what range of the medical image each region image occupies.
• the annotation DB stores annotation information for distinguishing annotations made using the system 1 together with annotation IDs.
• the annotation information is preferably information that can identify which medical image was annotated, when, by whom, and what type of annotation was made.
• the annotation information can preferably store "annotation subject information.”
• the annotation subject information is identification information for identifying "who made the annotation (i.e., the annotation subject)."
• the annotation subject information may be, for example, a user ID (e.g., a login account), or may be any one or more pieces of information in the user detailed information stored in the account DB.
• the annotation information may include whether the annotation can be disclosed (i.e., whether it can be disclosed on the presentation screen 27 of another person with a different user ID), and may include the disclosure conditions of the annotation (i.e., under what conditions if it is disclosed to another person).
• the label DB stores information about the labels associated with the region images by each annotation, linked to the annotation ID.
• the label DB stores each label ID in association with label detail information.
• the label detail information includes the classification type and classification content of the label (for example, the type is tumor classification, and the label classification content is tumor, etc.).
• the training data DB stores the generated training data. Users can download any training data by referencing the training data DB.
• At least the image DB, area image ID, annotation ID, and label ID are linked to each other. This makes it possible to easily call up the contents of any annotation (i.e. medical image, area image, label, etc.) from the annotation DB by specifying the annotation ID.
• annotation i.e. medical image, area image, label, etc.
• the above data management and DB configurations are merely examples. Any data management and DB configurations can be adopted depending on the specifications of the actual hardware and/or software, etc.
• the display control unit 21 executes display control to display various screens on the display device 34 of the user terminal 3. For example, the display control unit 21 displays a work screen 26 in which the boundary line of a region image is superimposed on a medical image. For example, the display control unit 21 performs control to provide a presentation screen 27 (see FIGS. 4, 6 to 18) generated by the presentation unit 25 (see FIG. 3B) together with the work screen 26 or independently from the work screen 26.
• a presentation screen 27 see FIGS. 4, 6 to 18
• the input unit 22 accepts various inputs from the operation device 38 of the user terminal 3.
• the input unit 22 accepts annotations (label inputs) from the operation device 38 of the user terminal 3.
• Label input is an input operation for associating a label with each of a plurality of area images.
• the input unit 22 stores the labels associated with each area image in the label DB.
• the input unit 22 may store an area image ID that uniquely identifies each area image in association with a label associated with each area image in the label DB.
• the label input may be performed, for example, by selecting an area image to which a label is to be input from among the area images displayed on the screen, and accepting a label designation to be associated with the area image.
• the generation unit 23 generates learning data for training a learning model by associating each of a plurality of area images with a label associated with each of the plurality of area images. For example, the generation unit 23 acquires an area image ID and a label from the label DB, and extracts image data of image coordinates corresponding to the area image ID from a medical image stored in the image DB. This extracted image data is the area image. The generation unit 23 can generate learning data by combining the image data of the extracted area image with a label stored in association with the area image ID.
• FIG. 3B shows the functional block configuration of the presentation unit 25.
• the presentation unit 25 includes a data acquisition unit 25a, a presentation screen generation unit 25b, a comparison unit 25c, and a selection unit 25d.
• the data acquisition unit 25a can acquire information required for generating a presentation screen 27 (see FIGS. 4 to 18) from the storage unit 20.
• the presentation screen generation unit 25b can generate the presentation screen 27 based on the information acquired by the data acquisition unit 25a.
• the comparison unit 25c and the selection unit 25d can calculate and/or select the information acquired by the data acquisition unit 25a in a specific presentation mode (comparison presentation mode, selection presentation mode) and supply the information subjected to the calculation, etc. to the presentation screen generation unit 25b.
• Fig. 5 and Fig. 19 are an example of the work screen 26 provided by the system 1.
• Fig. 5 is an annotation work screen in a grid system
• Fig. 19 is an annotation work screen in a designated frame system.
• the work screen 26 is a GUI when a user performs annotation work.
• the work screen 26 includes selection menus M10, M11, and M12, a display area W10, an overview display area W11, a display frame V11, a display frame T10, a button B10 for generating learning data, and a button B11 for referring to annotations.
• Selection menu M10 displays the type of medical image (pathological image as an example) currently being displayed.
• Selection menu M11 is a menu for specifying the medical image (pathological image as an example) to be displayed on the work screen 26 when multiple medical images have been input. In the examples of Figures 5 and 19, 40 pathological images relating to tumor cells have been input, and the third pathological image is currently being displayed.
• Selection menu M12 is a menu that allows the user who is annotating to select the disclosure setting (access rights) of the annotation to users other than the user himself/herself.
• Selection menu M12 allows at least one of disclosure allowed and disclosure not allowed. When disclosure not allowed, only the user who made the annotation can use the annotation. Usage includes, for example, one or more of calling, referencing, copying, editing, and creating learning data. When disclosure allowed, users other than the user who made the annotation can use the annotation. When disclosure allowed, disclosure conditions can be further selected. The disclosure conditions that can be selected are "user disclosure" and "user confidentiality".
• the selection result of selection menu M12 is registered in the annotation DB. When user disclosure is selected, when another user refers to the annotation via the presentation screen 27, the annotation subject information of the user who made the annotation is displayed on the presentation screen 27.
• the overview display area W11 displays the entire medical image and a display frame V11.
• the display frame V11 specifies the area image of the medical image that is enlarged and displayed in the display area W10. The user can change the position and size of the display frame V11 as desired.
• a grid (grid-shaped boundary line) is displayed superimposed on an image obtained by enlarging a part of a medical image in the display area W10.
• one square of the grid corresponds to one "region image".
• one region image has a width Wg and a height Hg .
• a computer i.e., the system 1 executes a process of generating a display area W10 in which a grid (boundaries of a plurality of region images) is superimposed on a medical image.
• Each square of the grid is stored in the region DB in association with the position coordinates of each region image in the medical image.
• a display frame T10 is displayed in the display area W10.
• the display frame T10 indicates the position of the region image that accepts input of a label from the user.
• the position of the display frame T10 can be arbitrarily changed by the operation device 38.
• the display frame T10 can be placed in an arbitrary position and a label can be input by additional operation of the operation device 38.
• the operation device 38 includes a pointing device (a mouse, for example).
• the designation frame may be set by mouse operation, for example, and the click position in the display area W10 may be set as the vertex of the designation frame.
• the designation frame is stored in the area DB in association with the position coordinates of each area image in the medical image. As an example, the designation frame becomes active (selected state) by clicking on it, and a label can be input by additional operation of the operation device 38.
• the label classification is displayed with the letters "T" and "N.” That is, when the area image is determined to be an image of a tumor cell, and as a result, a label indicating a tumor cell is assigned to the area image, the letter "T" is displayed in a predetermined position (as an example, the upper left) for that area image.
• the area image is determined not to be an image of a tumor cell, and as a result, a label indicating that it is not a tumor cell is assigned to the area image, the letter "N" is displayed in a predetermined position (as an example, the upper left) for that area image.
• the display control unit 21 may display an actual value of the number of area images to which labels have been assigned in the display area image N11 of the work screen 26. This actual value may be associated with each account and stored in a user account DB in the storage unit 20. As an example, the storage unit 20 may add up the actual values of all medical images for each user. As an example, the storage unit 20 may store the actual values in association with each medical image, thereby making it possible to visually recognize the difference in actual values between users for the same medical image.
• FIG. 1 A first example of a processing flow of the embodiment will be described with reference to Figures 4 to 7.
• the first example is a grid-based annotation.
• the server 1 executes a program constructed according to the control flow of Figure 4, thereby making it possible to support annotation of medical images, and a method for supporting annotation is provided.
• annotation work (S201, S101, S202, S102) is performed using the work screen 26 in FIG. 5.
• the system 1 presents the presentation screen 27 to the user (S203, S204, S103, S104, S205, FIGS. 5 and 6).
• the user can refer to the classification content of at least one label associated with at least one area image in a medical image by multiple annotations.
• user U010 logs in to system 1 (S200, S100).
• system 1 S200, S100
• user U010 may be referred to simply as "user.”
• the input unit 22 accepts from the user a selection of medical images to be used to generate learning data.
• the system 1 may set access rights to each medical image for each user account in the image DB 20. The user can select any medical image stored in the image DB, or can select only medical images to which access rights have been granted according to the login account.
• step S101 in response to an annotation work screen request from the user, the system 1 outputs a work screen 26 (see FIG. 5) corresponding to the medical image and label classification type selected by the user.
• the display control unit 21 displays the work screen 26 in which a grid is superimposed on one medical image selected by the user.
• the grid is the boundary line of an area.
• a rectangular lattice with orthogonal vertical and horizontal axes is used, which defines multiple rectangular squares, and one rectangular square defines one area image.
• the label classification type is, for example, information on the type of pathology.
• the type of pathology may be either "tumor” or "hypermutation".
• the pathology type may be set to "tumor”, and in that case, the label classification content may be "tumor” or "non-tumor”.
• the pathology type may be set to "hypermutation", and in that case, the label classification content may be "hypermutation” or "non-hypermutation”.
• the label classification type to be applied to the selected medical image may be automatically set by the input unit 22 in accordance with the menu M10. Alternatively, the input unit 22 may receive a selection of the label classification type to be applied to the selected medical image from the user.
• the user operates the user display frame T10 to input labels to each area image.
• the user U010 has finished inputting labels to some of the area images.
• the input unit 22 can accept label input for each area image.
• step S202 the user terminal 3 transmits the user's labeling operation from the operation device 38 to the system 1 each time.
• the label assigned by the user to each region image is either a label indicating that the region image is an image suspected of the presence of a tumor, or a label indicating that the region image is not an image suspected of the presence of a tumor.
• the label assigned by the user to each region image is either a label indicating that the region image is an image suspected of the presence of a hypermutation, or a label indicating that the region image is not an image suspected of the presence of a hypermutation.
• step S102 the input unit 22 associates the label with the area image ID based on the label assignment operation information from the user terminal 3, and stores the association as provisional registration information.
• the user U010 may want to refer to other annotations.
• the user can click the annotation reference button B11 on the user terminal 3.
• step S203 When button B11 is clicked, the determination result in step S203 becomes positive (YES), and the presentation screen 27 illustrated in FIG. 6 is launched. At this launch stage, label classification information has not yet been presented.
• a presentation target frame V12 is displayed on the work screen 26.
• the display control unit 21 displays the presentation target frame V12 in the display area W10.
• the presentation target frame V12 determines the target range of the presentation screen 27 (i.e., the area image in which labels based on other annotations are presented). The user can select one or more areas over any range within the display area W10 by operating the operation device 38, and instruct the system 1 to set or update the presentation target frame V12.
• this presentation instruction SN1 includes, as an example, a presentation range, a presentation mode, and call information.
• the "presentation range” can be set by the user arbitrarily selecting an area image using the presentation target frame V12 in FIG. 6, for example.
• the "presentation mode” can be selected from, for example, simple presentation, parallel presentation, comparison presentation, and selection presentation.
• the "call information” specifies "which annotation to refer to” and “how many annotations to refer to”, and specifically includes "call number” and "call annotation ID”.
• the presentation screen 27 in FIG. 6 has three input boxes (selection menus) for inputting the presentation mode, call number, and call annotation ID.
• the annotation ID "U001-1” makes it possible to call up the label from the first annotation by user U001. Setting the call annotation ID to "U001-1", "U001-2", etc. makes it possible to call up the first, second, etc. annotations by user U001.
• the number of calls can be specified as any number of two or more. An example in which the number of calls is two will be explained later.
• the system 1 starts processing (S103, S104) to generate the presentation screen 27.
• the data acquisition unit 25a refers to the storage unit 20 according to the call annotation ID based on the call information of the presentation instruction SN1.
• the data acquisition unit 25a refers to at least the image DB, area DB, and label DB of the storage unit 20, and thereby reads out the labels of each area image to be presented on the presentation screen 27 (S103).
• the data acquisition unit 25a sequentially refers to the area image ID of each area image in the presentation target frame V12.
• the data acquisition unit 25a can extract all labels linked to one area image ID by searching the label DB using the area image ID as a search key.
• the data acquisition unit 25a compares the annotation ID of each extracted label with each call annotation ID of the presentation instruction SN1.
• the data acquisition unit 25a retains labels that match between both accounts as they are to be presented, and excludes mismatched labels. This makes it possible to extract only the labels to be presented from the label DB.
• step S104 the presentation screen generating unit 25b generates a presentation screen 27 for displaying the label classification contents of each area image based on the information acquired by the data acquiring unit 25a.
• the presentation screen generating unit 25b transmits data for displaying the presentation screen 27 to the user terminal 3.
• Figure 7 shows an example of the presentation screen 27 after the update.
• the presentation screen 27 in the example of Figure 7 displays each label in layers over each area image within the presentation target frame V12 of the display area W10. This allows the user U010 to compare the labels that he or she is currently annotating with the labels that have been annotated by the other user U001. Note that only the labels of the user U001 are displayed on area images that have not yet been labeled by the user U010 currently annotating.
• the user U010 applies a predetermined number (or a predetermined range) of labels to the medical image, and the time arrives to end the annotation work.
• the work ends when the user presses button B10, for example.
• the work clicks button B10 the work ends (S206), a label confirmation instruction is sent to system 1 (S207), and system 1 registers all labels that were provisionally registered in step S102 in the label DB (S105).
• the generation unit 23 generates learning data and stores it in the learning data DB.
• the medical image may be cut out according to the grid of the work screen 26, so that the size of the region image and the size of the learning data image match.
• the learning data is composed of each learning data image and each label (see Figure 5) that the logged-in user U010 has associated with each region image on the work screen 26.
• FIG. 19 A second example of the processing flow of the embodiment will be described with reference to Figures 19 and 20.
• the second example is a designated frame type annotation.
• the computer system 1 performs processing according to the processing flow of Figure 4, but there are differences in steps S101, S202, and S104.
• step S101 the work screen 26 of Figure 19 is provided to the user U010.
• step S202 the user U010 sets a designated frame and inputs a label to the medical image via the work screen 26 of Figure 19.
• the annotation can be disclosed, and the disclosure condition is set to be user confidential.
• the user can launch the presentation screen 27 and input the necessary information into the input fields of the presentation screen 27, thereby enabling the user to send a presentation instruction SN1 (see step S204 in Fig. 4) to the system 1.
• the presentation screen 27 in Fig. 20 is presented to the user in step S104.
• the presentation screen 27 in the example of FIG. 20 displays several designation frames in layers within the display area W10. Each designation frame is labeled, and the user can visually recognize the classification content of the label associated with the area image designated by each designation frame in the medical image.
• the generation unit 23 After that, as in the first example, when the process reaches step S105, the generation unit 23 generates learning data and stores it in the learning data DB.
• the medical image may be cut out according to each designation frame on the work screen 26 in FIG. 19, and the cut-out image may be used as the learning data image.
• a grid may be further superimposed on the designation frame, and the area image within the designation frame may be cut out along the grid. In this way, learning data can be constructed in the second example as well.
• Variations of Presentation Screen 27 and Presentation Processing, etc. 7 (grid method) and 20 (designated frame method) are merely examples of the presentation screen 27, and at least Fig. 8 to Fig. 18 and Fig. 21 to Fig. 24B are further examples.
• the presentation mode of the embodiment can be at least any one of a simple presentation mode, a parallel presentation mode, a comparison presentation mode, and a selection presentation mode. Of these, only one mode may be adopted, or a plurality of modes may be adopted in a switchable manner.
• Figs. 7 to 9, 18, 21B, 22A, 22C, and 23B Examples of the "parallel presentation mode" are shown in Figs. 7 to 9, 18, 21B, 22A, 22C, and 23B.
• This mode multiple labels associated with area images are grouped together and arranged in parallel for each area.
• the arrangement direction may be vertical, horizontal, or diagonal, and the labels may partially overlap.
• FIG. 21A is an enlarged view of the lower left area of FIG. 20.
• area image conversion may be performed by further superimposing a grid on the specified frame.
• This "area image conversion” refers to conversion between the designated frame area and the grid area.
• the label of the designated frame is associated with the area defined by the grid within the range of the designated frame.
• the grid area image may be associated with the same label as the designated frame area image.
• the "predetermined overlap amount” is, for example, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90% or 100% of the area of the grid area, and may be within a range between any two of the numerical values exemplified here. When the predetermined overlap amount is 100%, a certain grid area and the designated frame area completely overlap. In FIG.
• FIG. 22A as an example, only the overlapping area where multiple designation frames overlap each other is set as the area to be presented.
• FIG. 22C is similar, but as an example, three or more designation frames overlap.
• the position of the mouse pointer T110 is set as the active area, and labels are displayed in parallel only in the active area, which may improve visibility.
• FIG. 23B shows the situation in FIG. 23A where three designation frames overlap, and after area image conversion has been applied to FIG. 23A.
• Figures 10, 11, 20, 21A, 23A, and 24A-24B Examples of the "simple presentation mode" are shown in Figures 10, 11, 20, 21A, 23A, and 24A-24B.
• this mode a set of area groups and labels is generated for each annotation, and these sets are arranged separately for each annotation.
• Figure 10 displays area images within the presentation target frame V12 in a separate window above the display area W10, and displays labels based on annotations by the second user U001.
• Figure 11 displays two area images within the presentation target frame V12 side by side, and displays labels based on annotations by the second user U001 and third user U002.
• Fig. 21A, Fig. 23A, and Fig. 24A-Fig. 24B are also in the simple presentation mode, but Fig. 21A, Fig. 23A, and Fig. 24A have the grid display turned ON. Note that there is no limitation on the grid size here, and as an example, it may be set smaller as shown in Fig. 24A, or may be adjustable by the user as desired.
• FIG. 24B shows a specified frame other than a rectangle, and an example is a polygonal specified frame. Grid overlay and area image conversion can also be applied to FIG. 24B. If the parallel presentation mode is adopted for FIG. 24B, only the overlapping areas of multiple specified frames may be presented, as in FIG. 22A and FIG. 22C.
• a "label-comparable area image” is an area image to which multiple labels are associated by multiple annotations.
• a “label-comparable area image” is an image of an overlapping portion of multiple area images to which multiple labels are associated, or the same area image to which multiple labels are associated, and it is possible to compare multiple associated labels.
• the computer may be caused to execute a presentation process that generates a presentation screen 27 so that the classification contents of multiple labels associated with the label-comparable area image are displayed so that the user can compare and visually recognize them.
• the computer may execute a process (processing of the comparison unit 25c) of calculating a comparison result obtained by comparing the classification contents of multiple labels associated with the label-comparable area image.
• the presentation screen 27 may present the comparison result to the user, thereby presenting the classification contents of multiple labels associated with the label-comparable area image.
• the labels themselves are not displayed, but the comparison results obtained by comparing multiple labels are presented to present the classification contents of the labels to the user.
• the “comparison” may be a match or mismatch for homogeneous classification labels, or a relevance/non-relevance determined according to a predetermined rule for heterogeneous classification labels.
• comparison results such as unanimity, mismatch, majority in favor, majority against, or balance may be presented.
• Figures 12 to 15, 21C, 22B, and 23C as an example, when the labels match, "COMT” is displayed if all the labels are "T”, and "COMN” is displayed if all the labels are "N”.
• "DIFF" is displayed.
• the selection unit 25d selects the labels to be presented according to the selection items specified by the user, and the presentation screen generation unit 25b generates the presentation screen 27.
• the selection may be based on the label classification content, or the presentation screen 27 may be generated for only area images with mismatched labels (FIG. 16).
• Selection may be based on the label type.
• the presentation screen 27 may be generated for only area images with tumor labels ("T" labels) from other annotations (FIG. 17).
• the selective presentation mode can also be adopted for annotations using the designated frame method. For example, in FIGS.
• only area images with mismatched labels may be presented, or only designated frames with specific label classification content (e.g., only T labels) or specific label classification types (e.g., only tumor/non-tumor labels) may be presented.
• specific label classification content e.g., only T labels
• specific label classification types e.g., only tumor/non-tumor labels
• the presentation mode does not have to be selectable from multiple modes, and may be only one, in which case the function of specifying the presentation mode can be omitted.
• the call information (call annotation ID) does not have to be selectively set.
• the call information setting function may be omitted, all annotations for the medical image to be presented may be automatically called, or all labels associated with the area image to be presented may be automatically called. In this case, the specification of the annotation ID can be omitted.
• the number of labels assigned for each label content may be expressed numerically (see FIG. 25C).
• the display method of the embodiment can adopt at least any one of a layer display, a window display, and a pop-up display. Only one display method may be adopted, or a plurality of display methods may be adopted in a switchable manner. Any presentation mode and any display method can be combined.
• FIGS 7, 8, 12, 15, 17, and 20 to 24B Examples of “layer display” are shown in Figures 7, 8, 12, 15, 17, and 20 to 24B.
• the presentation screen 27 is overlaid as a layer on the display area W10 of the work screen 26.
• the presentation screen 27 displays labels for each area image overlaid on the display area W10.
• Each label image may have transparency so that the area image is easily visible, but may also be non-transparent.
• FIG. 9 Examples of “window display” are shown in Figures 9 to 11, 13, 14, and 16.
• Presentation screen 27 is displayed as a separate window on top of work screen 26.
• display area W10 is a photograph
• the area image of presentation screen 27 is shown in an illustration style, but this is merely for convenience of illustration and does not limit the present invention.
• the area image of presentation screen 27 may be the same as the photograph in display area W10. This also applies to the pop-up display described below.
• a window display can also be adopted for annotations that use the designated frame method, and the medical image, designated frame, and label of another annotation can be displayed in a separate window.
• FIG. 18 An example of a "pop-up display" is shown in FIG. 18.
• the label classification content for only the selected area image is displayed in a pop-up window near the selected area image.
• the pop-up display in FIG. 20 may be used.
• a display such as that shown in FIG. 21B, FIG. 22A or FIG. 22B may be displayed in a pop-up window.
• the presentation screen 27 presents to the user the classification content of at least one label associated with at least one area image by a plurality of annotations.
• the terms "at least one area image” and "at least one label” are described with the intention of including the following aspects.
• any combination of annotation methods grid method or designated frame method
• presentation modes presentation modes
• display methods and other various modified examples
• the number of area images to be presented by the presentation screen 27 can be changed in various ways depending on the combination.
• the number of labels also changes depending on whether or not there is a missing label. A missing label is caused, for example, by a missing label input in the annotation.
• the pop-up display of the embodiment can present the classification contents of at least one label to the user for one area image.
• the number of labels associated with one area image may be less than two.
• the presentation screen 27 employing the pop-up display at least one label classification information can be presented for one area image.
• the presentation target frame V12 selects only one area image in the layer display or window display, it is the same as the pop-up display.
• two designation frames are provided in substantially the same position and range in FIG.
• each designation frame in FIG. 21A may be smaller than shown, and may be substantially as small as the grid size.
• a situation may occur in which two designation frames surround one area image based on the grid, and a label is associated with only that area image. In these cases, a missing label may occur as well. Even in this situation, the presentation screen 27 can present at least one piece of label classification information for one area image.
• the presentation screen 27 can present to the user the classification contents of multiple labels associated with multiple area images in a medical image by multiple annotations.
• the presentation screen 27 can be generated when multiple area images are selected in the presentation target frame V12, or for the entire display area W10 including the multiple area images, or by a pop-up display for the multiple area images.
• the classification contents of "multiple labels" associated with "multiple area images” can be presented to the user.
• the presentation screen 27 can present to the user the classification contents of the multiple labels associated with the multiple area images through the multiple annotations.
• the computer (system 1) may notify the user that a label is missing, for example, at step S104 in the processing flow of FIG. 4. For example, there may be cases where a label is missing in a specific area image for some or all of the multiple annotations. In that case, for example, at step S104, the computer (system 1) may present a notification signal (such as an error message) to the user instead of/or in addition to the presentation screen 27, or may display the specific area image prominently.
• a notification signal such as an error message
• the presentation target frame V12 may be omitted.
• the entire area image of the display area W10 may always be the target.
• the presentation screen 27 can present to the user the classification contents of multiple labels associated with multiple area images by multiple annotations.
• the first to third annotations are shown as examples for convenience. However, there is no upper limit to the number of annotations that can be presented on the presentation screen 27, and there is no upper limit to the number of labels that can be presented in association with one area image on the presentation screen 27.
• the system 1 can store any number of annotations by any number of users in the storage unit 20, and thus any number of labels are associated with one area image ID in the label DB.
• the system 1 may read out any number of annotations, three or more, (sets of area images and labels) from the image DB and label DB, and thus present the classification contents of any number of labels, three or more, for each area image on the presentation screen 27.
• the maximum number of annotations that the system 1 can present on the presentation screen 27 is "Z".
• the system 1 of the embodiment may be used by limited members, for example, only members of a certain company, laboratory, or research group. However, without being limited to this, the system 1 may be capable of accepting user account registrations from members of many more various research institutes, universities, or companies, and may be accessible to users in all regions of the world, and therefore may be made widely available to the general public via the communication network 2. This has the advantage that annotation data can be collected from many more various users. However, in such cases, not all users are necessarily willing to disclose their own user information. To address this issue, the system 1 of the embodiment has a function for restricting annotation disclosure and/or a "display/hide function for annotation subject information" described below.
• the "achievement value” may be reflected on the presentation screen 27.
• the achievement value is the cumulative number of labels attached for each user.
• the “achievement value” may be incorporated in any presentation mode. For example, labels associated with a user having a high achievement value may be highlighted more than labels associated with a user having a low achievement value. In a selective presentation mode, only the labels of users having achievement values above a certain value may be selectively presented.
• the annotation subject information may be displayed on the presentation screen 27.
• the annotation subject information is information that can identify who the annotation subject is who entered each label. For example, in FIG. 8, "U001-1" and "U002-1" are displayed in the input box for the call annotation ID, and this functions as the annotation subject information. For example, in FIG. 11, there are two sets of area images and label display area images in the presentation screen 27, and "U001-1" is added to one area image and "U002-1" is added to the other area image, making the annotation subject information easily distinguishable. This is one example, and as other examples of annotation subject information, the user's name, affiliation, nickname, avatar icon, etc. may be displayed on the presentation screen 27.
• the annotation subject information may be hidden.
• the subject information of the called annotation may not be displayed on the presentation screen 27.
• the annotation subject information may not be displayed at all.
• Annotations set to "not discloseable" in the selection menu M12 in FIG. 5 and the like may be made callable on condition that the annotation subject information is not displayed.
• the annotation subject can be kept secret. Even if a user is not willing to disclose his/her user information, there is an advantage in that the annotation data of that user can be utilized while ensuring anonymity. An example of utilizing annotation data while ensuring anonymity will be explained later in the variation of the call mode.
• Images L1a to L4b shown in FIG. 25A can be used to compare and contrast self and other annotations, or multiple self annotations.
• Image L1a is an example of a self (logged-in user) label display
• image L1b is an example of a other person's label display, and both can be displayed side-by-side, shifted vertically. Note that the self label and other person's label may adopt the same display format, and do not need to be different in color, etc.
• Images L2a and L2b have other person's labels of the same color, while images L3a and L3b have other person's labels of different colors (T is black, N is white).
• Images L4a and L4b are comparison labels (match/mismatch), with different characters and colors.
• FIG. 25B shows an example of comparing or contrasting multiple annotations of others.
• multiple other-person labels may be displayed in parallel below the self-person labels.
• comparison result labels may be displayed below the self-person labels, and COMT and COMN may be displayed in different colors.
• the classification contents of the labels associated with the area images may be displayed so that they can be visually distinguished. For example, they may be expressed as numbers, as in image L9, and in this example there are 13 "T" labels and 7 "N" labels.
• images L11a to L11c display only matching labels highlighted with a thick frame, while images L12a to L12c display only mismatching labels highlighted with a thick frame.
• images L13a and L13b any number of labels may be displayed partially overlapping.
• Image L16 is a modified version of image L9.
• the "difference in display mode” may, for example, statically and/or dynamically change the appearance of an area image or a label image.
• a specific area image or a specific label may be highlighted or colored differently, or as another example, a specific area image may be overlaid with any additional image other than the label.
• the "additional image” may be an image representing any symbol, color, and/or pattern.
• the "symbol” may be, for example, a character symbol, a mathematical symbol, or a graphic symbol. A difference in the shape, color, and/or pattern of the label may be applied.
• the highlight is to make the area image or label stand out by any method such as a thick frame or emphasis display.
• a difference in color and/or transparency may be used for the area image.
• the color difference may be, for example, monochrome/color color coding, or color coding in any multiple colors.
• the transparency difference may be, for example, by displaying one area image or label image with transparency while displaying the remaining area image or label image with a non-transparent display, so that the user can distinguish and view these area images.
• an animation display movement
• the animation may include blinking, repeated transparency changes, and/or repeated size changes.
• the display mode of the area image and/or the label image may be changed on the presentation screen 27.
• the computer (server 1) may execute a process (S104) of changing the display mode of the area image and/or the label image according to the classification content of the label and/or according to the comparison result of multiple labels associated with one area image.
• the classification content of the label associated with the region image as the annotation target may include an index value, and for example, the index value may be displayed as a subscript to "T" and "N" in the images L17a and L17b.
• the index value indicates how likely the classification content of the label is.
• the index value may be manually input by the user during annotation work on the work screen 26.
• the index value can be adopted in both the grid method and the designated frame method of annotation.
• the classification content of the label may be an index value, and for example, in the display example of the images L17a and L17b, the closer the value is to 100, the 100% possibility of a tumor, and the closer the value is to 0, the 0% possibility of a tumor, that is, a non-tumor.
• the index value may be arbitrarily set by the user in any increment width (for example, a fixed width such as 1% increments or 5% increments), or the index value may be presented as, for example, a plurality of options (for example, an arbitrary number of stages such as three stages or four stages), from which the user may select a desired index value.
• index values may be used, such as “high” (e.g., 70% or more), “medium” (e.g., 31% to 69%), and “low” (e.g., 30% or less).
• high e.g., 70% or more
• medium e.g., 31% to 69%)
• low e.g., 30% or less.
• a calculated value e.g., a total value, an average value, etc.
• the calculated value may be displayed on the label.
• the similarity of multiple labels may be calculated.
• the difference in similarity between the labels may be displayed on the label as a numerical value, or the result of determining whether the difference in index value between the labels is equal to or less than a predetermined value may be displayed on the label.
• the system 1 of the embodiment can be equipped with various call modes.
• the call mode of the embodiment can adopt any one of an annotation designation call mode, a full call mode, and a layered call mode. A user may selectively use two or more of these modes. Any call mode can be adopted together with the annotation method (grid method or designation frame method), each presentation mode, each display method, and various other modified examples of the embodiment.
• the annotation specification call mode in which an annotation ID is input to the presentation screen 27 is mainly described (see, for example, FIG. 7 and FIG. 20).
• the all call mode is a mode in which all annotations associated with a medical image at the time of calling are automatically called and presented on the presentation screen 27.
• the stratified call mode is a mode in which only annotations that meet specified conditions are extracted and presented on the presentation screen 27.
• stratification may be based on whether or not the annotation subject information meets a specified condition. For example, only annotations whose performance values are equal to or greater than a specified value may be extracted, making it possible to present only annotations that are highly proven and reliable. For example, a period may be specified as one of the call conditions, and only annotations that were registered (specifically, new registration or last edited registration) within a specified period may be extracted.
• index values may be used, and for example, only annotations whose summary statistics of all labels in the annotation (for example, the average index value or the median index value of all labels) are equal to or greater than a specified threshold may be extracted, making it possible to extract only annotations that the annotation subject created with a certain degree of confidence.
• the system 1 of the embodiment is configured to be switchable between a "disclosure mode" in which annotation subject information is displayed on the presentation screen 27, and a "non-disclosure mode" in which annotation subject information is not displayed.
• any call mode can be adopted. Note that in the disclosure mode, annotations set to "disclosure allowed/user confidential" in the selection menu M12 of FIG. 5, etc., may be prohibited from being called up in the annotation DB.
• non-disclosure mode there are limitations to the call modes that can be used.
• non-disclosure mode at least a mechanism that can recognize annotation subject information, such as annotation-specific call mode that requires input of an annotation ID that includes a user ID, cannot be used. All call mode and hierarchical call mode can be used in non-disclosure mode, and in these call modes, it is sufficient to prohibit the display of annotation subject information on the presentation screen 27, etc.
• the computer (system 1) executes the process (S103, S104) of generating the presentation screen 27.
• the presentation screen 27 presents to the user U010 the classification content (e.g., "T" or "N") of at least one label associated with at least one region image in the medical image by at least one of the multiple annotations. This makes it possible to present useful information for decision-making during annotation work, etc., by the presentation screen 27. Labels based on multiple annotations can be easily compared.
• user U010 may be referred to as the "first user”
• user U001 may be referred to as the "second user”
• user U002 may be referred to as the "third user”.
• the multiple annotations may include a first and a second annotation.
• the first annotation is made by a first user U010.
• the second annotation may be made by a second user U001 different from the first user U010, or may be made by the first user U010 prior to the first annotation.
• the presentation process (S104) may cause the computer (system 1) to generate a presentation screen 27 so as to present to the first user U010 the classification content of at least one label associated with at least one region image in the medical image by the first and second annotations while the first user U010 is making the first annotation or after the first annotation is completed.
• the presentation screen 27 may disclose information that can identify who the second user is, or may not disclose such information.
• the computer executes a process (S103, S104) of generating a presentation screen 27 to present to the first user U010 the classification content of the label associated with at least one area image in the medical image by the first annotation and the classification content of the label associated with at least one area image in the medical image by a different second annotation (annotation by the second user U001).
• This allows the first user (user U010) and the second user (user U001 and/or U002) to compare the labels created by their respective annotations.
• the computer (system 1) may execute a process (S103, S104) to generate a presentation screen 27 to present to the first user U010 the classification content of the label associated with at least one area image in the medical image by the first annotation (U010-2) and the classification content of at least one label associated with at least one area image in the medical image by a second annotation (U010-1) previously made by the user U010. This allows the user to compare one of his/her own annotations (U010-1) with the other one (U010-2).
• second annotation is distinguishable from any one annotation of the first user U010.
• Multiple annotations e.g. U010-1, U010-2 saved by the same user (e.g. the first user U010) with different annotation IDs may be treated as separate annotations.
• the "second annotation” may include any one or more annotations of another user (the second user U001 in this embodiment) and/or may include the first user U010's past annotation (U010-1).
• the multiple annotations may include a second and a third annotation.
• the second and third annotations may be as follows.
• the second annotation may be made by the second user U001, or may be made by the first user U010 earlier than the first annotation.
• the third annotation may be made by the third user U002, may be made by the first user U010 earlier than the first annotation, or may be made by the second user U002 earlier than the second annotation.
• the second and third annotations may be two annotations made by the first user U010 earlier than the first annotation, but the two annotations are considered to have been made at different times.
• the presentation process (S104) may cause the computer (system 1) to generate a presentation screen 27 to present to the first user U010 the classification content of at least one label associated with at least one region image in the medical image by the second and third annotations while the first user U010 is making the first annotation or after the first annotation is completed.
• the presentation screen 27 may disclose or not disclose information that can identify at least one of the second user and the third user.
• the computer may execute a process (S103, S104) to generate a presentation screen 27 so as to present to the user U010 the classification contents of the labels associated with at least one region image in a medical image by "each of the annotations of the other users (the second user U001 and the third user U002)." Labels by the annotations of the other users (the second user U001 and the third user U002) can be compared.
• the computer (system 1) may execute a process (S103, S104) of generating a presentation screen 27 so as to present to the user U010 the classification contents of the labels associated with at least one region image in the medical image by "each of the first and second annotations (U001-1, U001-2) of the second user U001.”
• the labels by the second and third annotations (U001-1, U001-2) made by the second user U001 can be compared.
• the timing of presenting the presentation screen 27 is not limited to when the first user U010 is performing the first annotation work.
• the computer (system 1) may be caused to present the presentation screen 27 when the first user U010 is not performing the first annotation (e.g., after the work is finished).
• a "reference mode" may be set on the menu screen after logging in, and in this reference mode, the computer (system 1) may be caused to execute a process of generating the presentation screen 27 for the first user U010 to refer to his/her own and/or others' annotations without calling up the work screen 26.
• the following criteria for "substantially identical" can be adopted for the identity between multiple medical images. Examples of cases in which two medical images are substantially identical include at least the following first and second situations. In the first situation, the two medical images are completely identical and have the same image data. In the second situation, even if there are differences between the medical images (e.g., differences in image data and/or image IDs), the differences can be regarded as identical for annotation purposes.
• a medical image is obtained by photographing the same medical image subject.
• a new medical image is generated by duplicating the medical image data, changing the image format after duplication (e.g., saving in a compressed format), or applying image processing after duplication (resizing, changing brightness, contrast, saturation, color tone, etc.).
• the original medical image and the new medical image are substantially identical even if there are differences in the image ID, image data, etc. This is because the subject photographed in the medical image is the same, and the differences can be ignored for annotation purposes.
• the same applies to area images, and the substantially identical standard can be adopted for area images generated from substantially identical medical images.
• a mechanism known as a "virtual slide” may be used.
• the same subject pathological specimen, i.e. tissue
• pathological images are registered in an image DB. Since digitized pathological specimen data can be used, this has the advantage of making it easier to improve the pathologist's observation accuracy.
• a group of pathological images in a virtual slide are essentially identical, since they merely represent the same pathological specimen at different magnifications. This concept is similar for any medical images in which the same subject is photographed at different magnifications.
• a substantially identical standard can also be adopted for area images generated from these substantially identical medical images.
• a pathological slice may have an almost identical structure to a pathological slice directly above or below it.
• the pathological images of each of these pathological slices are substantially identical.
• Similar cases are also anticipated for all medical images other than pathological images. For example, when a three-dimensional structure is sliced, it is anticipated that the planar images of each adjacent layer are substantially identical. In these cases as well, multiple medical images are substantially identical. The substantially identical standard can also be adopted for regional images generated from these substantially identical medical images.
• the embodiment can employ at least the "criteria for substantial image identity" described in each of the above examples.
• the image IDs of medical images that are to be treated as substantially identical can be linked to each other by manual user operation or automatic computer processing so that they can be treated as a group together in the image DB of the storage unit 20.
• a first user e.g., U010
• a second user and/or a third user e.g., U001 and/or U002
• the second medical image is a copy of the first medical image, or a copy of the first medical image that has been subjected to image processing.
• the first and second medical images are linked in the image DB, and both images are treated as the same. Even if one assumes such a case and refers to Figures 7 to 18 and Figures 20 to 24, it can be understood that multiple annotations have associated multiple labels with each region image in the substantially identical first and second medical images.
• each region image of a substantially identical medical image when multiple labels (e.g., first and second labels) are associated with each region image of a substantially identical medical image, several examples of ID management in each DB of the storage unit 20 are possible.
• multiple substantially identical medical images may be annotated using a grid method, or a grid may be set using a designated frame method of annotation.
• the grids for each medical image may be set to the same size and with the same coordinates.
• the DB in the storage unit 20 may associate a label ID for each of a plurality of annotations with one area image ID.
• a group of area image IDs may be commonly adopted for a plurality of medical images.
• an area image ID associated with a certain label first label
• second label second label
• a second example of ID management may be as follows. There may be cases where a first medical image (master medical image) is copied to generate one or more second medical images (copy medical images). Furthermore, there may be cases where image processing is performed on the second medical image. In at least these cases, the first and second medical images are essentially the same image, but have different image data and may be assigned different image IDs. Thus, as a second example of ID management, the image IDs of the first and second medical images may be linked in the image DB.
• an image group consisting of a master medical image ID and a group of one or more copy medical image IDs
• this association may be followed and the same label ID may be automatically associated with the medical images of the remaining image IDs.
• a medical image ID, a region image ID, and a label ID may be associated directly as in the first example, or indirectly as in the second example. This allows multiple annotations to be performed on substantially identical medical images, and multiple labels may be associated with one region image using multiple annotations.
• the subject that performs the annotation is not limited to a human being and may also include a computer.
• a computer classifies an image by a rule-based classification process or by machine inference of a learned discrimination model, the classification content (classification result) is associated with each image, and this may be considered as an annotation.
• the multiple annotations that contrast or compare labels may be of different or the same subject.
• a user ID for the computer may be set in the account DB, and an annotation subject ID unique to each computer may be assigned when registered in the label DB.
• the system 1 is illustrated as one information processing device, but the present embodiment is not limited thereto.
• the system 1 may be configured with one or more physical servers, may be configured with a virtual server operating on a hypervisor, or may be configured with a cloud server.
• the processing of the program related to the generation and display of the presentation screen 27 can be arbitrarily shared between the system 1 (server 1) and the user terminal 3. Please note that the way of sharing the processing in the control flow of FIG. 4 is one example.
• the presentation screen generation process may be executed by a server, in which case the completed presentation screen data may be sent to the user terminal.
• the presentation screen generation process may be executed by a part of the process by the user terminal and the remaining process by the server.
• a computer should be interpreted differently from the above, then that statement shall be used as the basis for the interpretation.

Landscapes

• Health & Medical Sciences (AREA)
• Engineering & Computer Science (AREA)
• Medical Informatics (AREA)
• Public Health (AREA)
• Biomedical Technology (AREA)
• Primary Health Care (AREA)
• General Health & Medical Sciences (AREA)
• Epidemiology (AREA)
• Data Mining & Analysis (AREA)
• Databases & Information Systems (AREA)
• Pathology (AREA)
• Business, Economics & Management (AREA)
• General Business, Economics & Management (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Radiology & Medical Imaging (AREA)
• Medical Treatment And Welfare Office Work (AREA)

Priority Applications (2)

Applications Claiming Priority (2)

Publications (1)

Family

ID=93657899

Family Applications (1)

Country Status (3)

Citations (4)

• 2024
  • 2024-05-28 JP JP2025524102A patent/JPWO2024247991A1/ja active Pending
  • 2024-05-28 EP EP24815474.2A patent/EP4723125A1/en active Pending
  • 2024-05-28 WO PCT/JP2024/019507 patent/WO2024247991A1/ja not_active Ceased

Patent Citations (4)

Also Published As

Similar Documents

Legal Events