JP7066491B2 - Medical image processing device, teacher data creation program and teacher data creation method - Google Patents

Description

Embodiments of the present invention relate to a medical image processing apparatus, a teacher data creation program, and a teacher data creation method.

In order to create machine learning teacher data related to image recognition, annotations defining the image recognition target area of the medical image are manually drawn on the medical image by the user. Since a large amount of medical images are required as teacher data, the work of drawing annotations is very troublesome for the user.

Japanese Unexamined Patent Publication No. 2015-095043

The problem to be solved by the invention is to reduce the trouble of creating teacher data for machine learning related to image recognition.

The medical image processing apparatus according to the embodiment has a display on which medical image data used as teacher data for machine learning related to image recognition is displayed, and an annotated area that defines a target area for image recognition in the medical image data. A matching determination unit that outputs a comparison result between the position of the imparted region in the medical image data and clinical information associated with the medical image data, and the comparison result. A display control unit for displaying on the display is provided.

FIG. 1 is a diagram showing a configuration of a medical image processing apparatus according to the present embodiment. FIG. 2 is a diagram showing a typical flow of teacher data creation processing by the processing circuit of FIG. FIG. 3 is a diagram schematically showing the positional relationship between the original display cross section of the imaging axis reference and the subject. FIG. 4 is a diagram showing an example of a display screen of a display image relating to the original display cross section in step S3 of FIG. FIG. 5 is a diagram schematically showing the necessity of rotation of the display cross section. FIG. 6 is a diagram schematically showing the rotation process of the display cross section in step S4 of FIG. FIG. 7 is a diagram showing an example of an execution screen of the annotation processing in step S7 of FIG. FIG. 8 is a diagram showing a typical flow of annotation processing in step S7 of FIG. FIG. 9 is a diagram showing an example of a display screen displayed in step S72 of FIG. FIG. 10 is a diagram showing an example of a display screen of the comparison result of the matching determination function displayed in step S76 of FIG. FIG. 11 is a diagram showing a display screen on which the image thickness adjustment tool and the annotation thickness adjustment tool are displayed, which are displayed on the display of FIG. 1. FIG. 12 is a diagram schematically showing the slab thickness of the display image of FIG. 11 and the slab thickness of the annotation. FIG. 13 is a diagram showing another display example of the image thickness adjustment tool and the annotation thickness adjustment tool displayed on the display of FIG. 1. FIG. 14 is a diagram schematically showing an annotation interpolation process by the grant area determination function of FIG. 1. FIG. 15 is a diagram schematically showing an application of annotation interpolation processing by the grant area determination function of FIG. 1.

Hereinafter, the medical image processing apparatus according to the present embodiment will be described with reference to the drawings.

The medical image processing apparatus according to the present embodiment is a computer that performs image processing on medical image data. The medical image processing device may be a computer used in a medical facility such as a workstation, an image viewing device (viewer), an image interpretation report creation device, or a PACS (Picture Archiving and Communication System), and is included in the medical diagnostic imaging device. It may be a computer. The medical image diagnostic device can be applied to any medical modality such as an X-ray computer tomography device, an X-ray diagnostic device, a magnetic resonance imaging device, an ultrasonic diagnostic device, and a nuclear medical diagnostic device. Hereinafter, the medical image processing apparatus according to the present embodiment is assumed to be a workstation that performs image processing in order to give a specific explanation.

The medical image data according to the present embodiment is image data for a subject generated by a medical image diagnostic apparatus. The medical image data may be two-dimensional image data which is a set of a plurality of pixels arranged in a two-dimensional manner, or may be three-dimensional image data which is a set of a plurality of voxels arranged in a three-dimensional shape. However, in order to make the following description concrete, it is assumed that the medical image data is three-dimensional image data. Pixels and voxels are collectively referred to as pixels.

As shown in FIG. 1, the medical image processing apparatus 1 according to the present embodiment includes a processing circuit 11, a memory 13, an input interface 15, a communication interface 17, and a display 19.

The processing circuit 11 has a processor such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). When the processor activates the teacher data creation program installed in the memory 13 or the like, the image reading function 111, the text reading function 112, the given area determination function 113, the matching determination function 114, the image processing function 115, and the display control function 116 , The memory control function 117, the machine learning function 118, and the like are executed. It should be noted that each function 111 to 118 is not limited to the case where it is realized by a single processing circuit. A processing circuit may be configured by combining a plurality of independent processors, and each processor may execute a program to realize each function 111 to 118.

In the image reading function 111, the processing circuit 11 reads the medical image data to be processed from the memory 13. The medical image data to be processed is medical image data used as teacher data for machine learning related to image recognition.

In the text reading function 112, the processing circuit 11 reads the text data associated with the medical image data to be processed from the memory 13. The text data is textual data representing clinical information regarding a subject of medical image data.

In the imparting area determination function 113, the processing circuit 11 determines an annotation addition area for the medical image data read by the image reading function 111 according to an instruction by the user via the input interface 15. The annotation according to the present embodiment is graphical data attached to the medical image data and defining a target area for image recognition.

In the matching determination function 114, the processing circuit 11 outputs a comparison result between the annotation area of the annotation determined by the addition area determination function 113 and the clinical information of the text read by the text reading function 112. That is, in the matching determination function 114, the consistency between the annotation area and the clinical information indicated by the text is determined.

In the image processing function 115, the processing circuit 11 aligns the display conditions including the image position and the display image type with the medical image data or the like read by the image reading function 111 in order to continuously work under a constant image reading condition. Various image processing is performed. Typically, the processing circuit 11 applies MPR (Multi-Planer Reconstruction) processing to three-dimensional medical image data as image processing to generate two-dimensional display image data relating to a predetermined display cross section (MPR cross section). .. In addition to MPR processing, the processing circuit 11 can also perform image processing such as volume rendering, surface volume rendering, pixel value projection processing, and CPR (Curved MPR) processing.

In the display control function 116, the processing circuit 11 displays various information on the display 19. For example, the processing circuit 11 displays on the display 19 a display image based on the medical image data read by the image reading function 111 and a text corresponding to the text data read by the text reading function 112. Further, the processing circuit 11 draws annotations on the display image data.

In the storage control function 117, the processing circuit 11 transfers various information to the memory 13 for storage in the memory 13. For example, the processing circuit 11 stores the display image data corresponding to the display image displayed on the display 19 in the memory 13.

In the machine learning function 118, the processing circuit 11 performs machine learning related to image recognition on the annotated medical image data. For example, the processing circuit 11 inputs the annotated medical image data to the DNN (Deep Neural Network), and outputs the classification result of the anatomical part of the annotated region. CNN (Convolutional Neural Network) is one of the DNNs related to image recognition.

The memory 13 is a storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), or an integrated circuit storage device that stores various information. In addition to the above storage device, the memory 13 is a drive device for reading and writing various information between a portable storage medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), and a flash memory, a semiconductor memory element, and the like. May be. Further, the memory 13 may be in another computer connected to the medical image processing device 1 via a network. For example, the memory 13 stores a teacher data creation program. Further, the memory 13 stores medical image data in association with clinical information.

For example, medical image data is associated with text data such as interpretation reports and electronic medical records. Specifically, the text data is text data representing clinical information of a subject, which is the subject of medical image data, such as findings, medical records, interview results, examination orders, and examination instructions. The clinical information according to this embodiment is information that serves as a clue for a lesion generated in a subject. That is, text data is a form of clinical information, and is a word, phrase, or sentence indicating the type and location of a lesion. A plurality of combinations of medical image data and clinical information are stored in the memory 13.

The input interface 15 receives various input operations from the user, converts the received input operations into electric signals, and outputs the received input operations to the processing circuit 11. Specifically, the input interface 15 is connected to an input device such as a mouse, keyboard, trackball, switch, button, joystick, touch pad, and touch panel display. The input interface 15 outputs an electric signal corresponding to an input operation to the input device to the processing circuit 11. Further, the input device connected to the input interface 15 may be an input device provided in another computer connected via a network or the like.

The communication interface 17 is an interface for data communication with another computer. For example, the communication interface 17 receives a plurality of combinations of medical image data and text data used as teacher data from PACS via a network.

The display 19 displays various information according to the display control function 116 of the processing circuit 11. For example, the display 19 displays a display image based on the medical image data read by the image reading function 111 and a text corresponding to the text data read by the text reading function 112. As the display 19, for example, a liquid crystal display (LCD), a CRT (Cathode Ray Tube) display, an organic EL display (OELD), a plasma display or any other display can be appropriately used. be.

Hereinafter, an operation example of the medical image processing apparatus 1 according to the present embodiment will be described.

FIG. 2 is a diagram showing a typical flow of teacher data creation processing by the processing circuit 11 of the medical image processing apparatus 1 according to the present embodiment. The process shown in FIG. 2 is realized by the processing circuit 11 reading the teacher data creation program from the memory 13 and executing the program. In the first stage of the process of FIG. 2, a plurality of combinations of medical image data and text data, which are candidates for teacher data, are previously acquired from PACS or the like via the communication interface 17 or the like and stored in the memory 13. And. In the creation of teacher data, a large number of medical image data related to the cases to be created are prepared, and annotations are added to each of the medical image data. That is, it is assumed that it is known which case the medical image data stored in the memory 13 is related to.

As shown in FIG. 2, first, the processing circuit 11 executes the image reading function 111 (step S1). In step S1, the processing circuit 11 reads out one volume of medical image data regarding the case for which the teacher data is to be created from the memory 13.

When step S1 is performed, the processing circuit 11 executes the image processing function 115 (step S2). In step S2, the processing circuit 11 generates a display image relating to the first display cross section of the imaging axis reference based on the medical image data read in step S1. Hereinafter, the first display cross section of the imaging axis reference will be referred to as an original display cross section.

When step S2 is performed, the processing circuit 11 executes the display control function 116 (step S3). In step S3, the processing circuit 11 displays the display image of the original display cross section generated in step S2 on the display 19.

FIG. 3 is a diagram schematically showing the positional relationship between the original display cross section 31 based on the imaging axis A1 and the subject P. As shown in FIG. 3, the imaging axis A1 is an imaging reference line of a medical modality in which medical image data is collected. For example, when the medical modality is an X-ray computer tomography apparatus, the imaging axis A1 is parallel to the rotation center axis of the rotating portion of the gantry, in other words, the long axis of the top plate 33 on which the subject P is placed. It is stipulated. In medical imaging, the subject axis A2 is aligned with the imaging axis A1. The subject axis A2 is a reference line of the imaging target site, and is typically defined on the body axis of the whole body of the subject P. The medical image data is defined by a coordinate system with reference to the imaging axis A1. Under the assumption that the subject axis A2 coincides with the imaging axis A1, three orthogonal cross sections (axial cross section, sagittal cross section, and coronal cross section) with respect to the subject axis A2 are defined. As the original display cross section, a display image of three orthogonal cross sections with respect to the subject axis A2 is generated. For example, the axial cross section 31 is defined as a cross section orthogonal to the photographing axis A1. Although not shown, the coronal cross section is defined as a cross section parallel to the imaging axis A1 and orthogonal to the vertical axis, and the sagittal cross section is defined as a cross section parallel to the imaging axis A1 and the vertical axis. The display image relating to the axial cross section (hereinafter referred to as the axial image), the display image relating to the sagittal cross section (hereinafter referred to as the sagittal image), and the display image relating to the coronal cross section (hereinafter referred to as the coronal image) are subjected to MPR processing on the medical image data read in step S1. Is generated by applying.

FIG. 4 is a diagram showing an example of a display screen IS1 of a display image relating to an original display cross section. As shown in FIG. 4, the display screen IS1 displays an axial image I1-1, a coronal image I1-2, and a sagittal image I1-3 with reference to the photographing axis A1 as a display image I1 relating to the original display cross section. To. The main display image is displayed in a large size on the display screen IS1, and the other sub display images are displayed in a smaller size than the main display image. In FIG. 4, the axial image I1-1 is displayed as the main display image, and the coronal image I1-2 and the sagittal image I1-3 are displayed as the sub display images. The main display image can be arbitrarily changed from the axial image I1-1, the coronal image I1-2, and the sagittal image I1-3 according to the user's instruction via the input interface 15 and the like. Further, the processing circuit 11 may generate a display image relating to an arbitrary cross section (oblique cross section) according to a user instruction via an input interface 15 or the like as a main display image based on medical image data.

As shown in FIG. 4, the processing circuit 11 displays the rotation button B1 and the fixed button B2 on the display screen IS1. The rotation button B1 is a GUI (Graphical User Interface) button for instructing rotation of the display cross section. The rotation button B1 is provided to change the display target on the display screen IS1 from the original display cross section based on the imaging axis to the second display cross section based on the subject axis. Hereinafter, the second display cross section will be referred to as a secondary display cross section. The fixed button B2 is a GUI button for stopping the rotation of the display cross section.

FIG. 5 is a diagram schematically showing the necessity of rotation of the display cross section. As shown in FIG. 5, the subject axis (specifically, the body axis of the subject) A2 may not coincide with the imaging axis A1. If the subject axis A2 does not coincide with the imaging axis A1, the original axial cross section 31 orthogonal to the imaging axis A1 is not orthogonal to the subject axis A2 and cannot be said to be a true axial cross section. In the teacher data creation process for drawing annotations on a specific anatomical part of a large number of display images, the display cross section is always oriented in a fixed direction with respect to the subject axis A2 from the viewpoint of annotation drawing efficiency and the like. Is desired. This is because if the orientation of the display cross section with respect to the subject axis A2 is not constant, it is difficult for the user to find the anatomical site to which the annotation is to be added. Therefore, the image processing function 115 according to the present embodiment generates a display image relating to the second display cross section with respect to the subject axis A2 in step SA4.

When the rotation button B1 is pressed by the user via the input interface 15, the processing circuit 11 executes the image processing function 115 (step S4), and in step S4, the processing circuit 11 rotates the display cross section and is covered. Generate a display image for the second display cross section of the sample axis reference.

FIG. 6 is a diagram schematically showing the rotation process of the display cross section in step S4. As shown in FIG. 6, it is assumed that the subject axis A2 does not coincide with the imaging axis A1 and the axial cross section, which is the original display cross section 31, is non-orthogonal to the subject axis A2. When the rotation button B1 is pressed by the user via the input interface 15, the processing circuit 11 automatically rotates the display cross section 31 to the display cross section 35 orthogonal to the subject axis A2.

When step S4 is performed, the processing circuit 11 waits for the fixed button to be pressed (step S6). After completing the rotation of the display cross section, the user presses the fixing button via the input interface 15. After pressing the rotation button, annotation processing (step S7) becomes possible due to the pressing of the fixed button.

When the fixed button is pressed in step S6 (step S6: YES), the processing circuit 11 executes annotation processing (step S7).

FIG. 7 is a diagram showing an example of the execution screen I2 of the annotation processing. As shown in FIG. 7, the display screen IS2 displays an axial image I2-1, a coronal image I2-2, and a sagittal image I2-3 with respect to the subject axis A2. The user observes the axial image I2-1, the coronal image I2-2, and the sagittal image I2-3 to find out the anatomical site for which the teacher data is to be created. Then, the user designates an image region corresponding to the anatomical part of the display image I2 such as the axial image I2-1 via the input interface 15. The processing circuit 11 identifies a medical image area in the medical image data corresponding to the designated display image area via the input interface 15, and annotates the specified medical image area.

Examples of the method for designating the annotation area include the following methods.
1) The user specifies the area of the recognition target area (eg, tumor) itself by drawing a circle or the like. 2) The area to be recognized is specified by the user by specifying "other than" the area to be recognized. 3) The recognition target is specified by a "point" or "line" instead of an area. 4) When the user specifies one point inside the area to be recognized, the workstation detects an area having the same pixel value, and the area is designated as the area to be recognized. 5) By designating the recognition target for each frame of the moving image, the recognition target whose area changes with time is designated.

The processing circuit 11 according to the present embodiment can execute various processes for simplifying and facilitating the annotation work. These processes will be described later.

When step S7 is performed, the processing circuit 11 executes the memory control function 117 (step S8). In step S8, the processing circuit 11 stores the data of the display image in the memory 13 together with the orientation of the display cross section of the secondary. For example, when the save button is displayed on the display screen and the save button is pressed via the input interface 15 or the like, the processing circuit 11 saves the data of the display image in the memory 13 together with the orientation of the secondary display cross section. do. By saving the data of the display image together with the orientation of the display cross section, it is possible to display the secondary display image without executing the rotation process in step S4.

When step S8 is performed, the processing circuit 11 determines whether or not to process other medical image data (step S9). When the medical image data relating to the same case for which the teacher data is to be created exists in the memory 13, it is determined that the other medical image data is processed (step S9: YES), and the processing circuit 11 determines the other medical image data. Steps S1-S9 are repeated again. When it is determined that the medical image data relating to the same case for which the teacher data is to be created does not exist in the memory 13 and the other medical image data is not processed (step S9: NO), the processing circuit 11 creates the teacher data. End the process.

As a result, the teacher data creation process according to the present embodiment is completed.

The procedure of the teacher data creation process according to the present embodiment is not limited to the procedure shown in FIG. For example, a display image relating to a secondary display cross section relative to a subject axis may be generated and displayed without generating and displaying a display image relating to the original display cross section. When the subject axis coincides with the photographing axis, the rotation process (step S4) of the displayed cross section may not be performed.

When the teacher data creation process is completed, the processing circuit 11 executes the machine learning function 118. In the machine learning function 118, the processing circuit 11 applies the annotated medical image data to a DNN related to image recognition such as CNN, performs forward propagation processing, and outputs an estimated recognition result of the annotation region. Next, the processing circuit 11 applies the difference (error) between the estimated recognition result and the correct answer recognition result to the DNN to perform back propagation processing, and calculates the gradient vector. Next, the processing circuit 11 updates parameters such as the weighted matrix and bias of the DNN based on the gradient vector.

The machine learning function 118 does not need to be implemented by the same processing circuit 11 as the image reading function 111, the text reading function 112, the imparting area determination function 113, and the matching determination function 114. That is, the machine learning function 118 may be mounted not on the medical image processing device 1 but on another computer for DNN learning (hereinafter, referred to as a DNN learning device). In this case, the annotated medical image data is transmitted to the DNN learning device via the communication interface 17 for machine learning. Then, the machine learning function 118 is executed by the DNN learning device.

Next, the details of the annotation processing in step S7 of FIG. 2 will be described. In the following explanation, it is assumed that the case for which teacher data is created is cerebral ischemia.

FIG. 8 is a diagram showing a typical flow of annotation processing in step S7 of FIG. As shown in FIG. 8, the processing circuit 11 executes the text reading function 112 (step S71). In step S71, the processing circuit 11 reads out the text data associated with the medical image data read out in step S1. As the text data, for example, the text data of the findings associated with the medical image data is read out. In this case, the processing circuit 11 individually extracts text data from the text data of the entire findings, typically in units of predefined clinically meaningful words, phrases, or sentences. For example, text data corresponding to "cerebral ischemia" and "left hemiplegia" are individually extracted.

When step S71 is performed, the processing circuit 11 executes the display control function 116 (step S72). In step S72, the processing circuit 11 displays the display image and the text corresponding to the text data read in step S71.

FIG. 9 is a diagram showing an example of the display screen IS3 displayed in step S72. As shown in FIG. 9, the display image I3 and the text T3 corresponding to the text data extracted in step S71 are displayed side by side on the display screen IS3. In FIG. 9, as the text T3 related to the display image I3, the text “Side of symptoms” T3-1, the text “Time since onset” T3-2, and the text “Neurological defect” regarding the signs of stroke derived from cerebral ischemia. T3-3 is displayed. The text T3 is displayed to assist the user in finding the site of the case for which the teacher data is to be created. For example, if the left half of the body is paralyzed, there should be symptoms in the right brain. By displaying the text "Side of symptoms" T3-1, the user knows that he or she should have symptoms in either the right or left brain. Also, by displaying the text "Time since on set" T3-2, it can be seen that the image is after the appearance of signs such as stroke, and therefore the symptoms should be seen in the right brain. Further, by displaying the text "Neurological defect" T3-3, it can be understood that the optic nerve disorder should be depicted in the image. In this way, the user can easily find the site of the case for which the teacher data is to be created by referring to the clinical information represented by the text T3 as well as the displayed image I3. The display form of the text T3 is not particularly limited.

When step S72 is performed, the processing circuit 11 waits for the annotation drawing area to be designated via the input interface 15 (step S73). The user finds the cerebral ischemia site for which the teacher data is created while observing the display image and the text displayed in step S72. When the cerebral ischemia site is found, the user designates the display image area corresponding to the cerebral ischemia site of the display image as the drawing area via the input interface 15.

When the drawing area of the annotation is specified (step S73: YES), the processing circuit 11 executes the grant area determination function 113 (step S74). In step S74, the processing circuit 11 determines an image area (hereinafter, referred to as an imparting area) in the medical image data corresponding to the drawing area designated in step S73. Specifically, the processing circuit 11 calculates the coordinates of the medical image area of the medical image data corresponding to the drawing area of the displayed image based on the position of the display cross section in the medical image data, and the medical circuit 11 corresponds to the coordinates. The image area is determined as the imparting area.

When step S74 is performed, the processing circuit 11 executes the matching determination function 114 (step S75). In step S75, the processing circuit 11 compares the position of the imparted region determined in step S74 with the clinical information indicated by the text displayed in step S72. More specifically, the processing circuit 11 determines whether the anatomical site of the imparted region determined in step S74 is consistent with the clinical information indicated by the text displayed in step S72. For example, the processing circuit 11 determines the consistency by using a table (LUT: Look Up Table) in which the clinical information indicated by the text and the position condition of the anatomical site corresponding to the clinical information are associated with each other. The positional condition is the position of the anatomical site corresponding to the clinical information. In the table, for example, the text "Side of symptoms" is associated with the anatomical site "right brain". In this case, the processing circuit 11 first identifies the position of the imparted region in the medical image data and determines whether the identified position matches the anatomical site associated with the text in the table. do. The anatomical site at the position is specified as the position of the imparted area in the medical image data. The anatomical site corresponding to the imparted region may be specified by using, for example, an anatomical landmark. For example, the memory 13 stores data of a human anatomical chart in which a plurality of anatomical reference points are registered in advance. The human anatomical chart is data in which the data of a standard morphology of the human body is associated with an anatomical reference point which is an identifier of an anatomical site corresponding to the morphology. The processing circuit 11 aligns the medical image data and the data of the human anatomical chart, identifies the position of the human anatomical chart corresponding to the imparted region, and identifies the anatomical portion associated with the position. If the identified anatomical site matches the anatomical site associated with the text in the table, processing circuit 11 states that the grant area is consistent with the clinical information in the text. Judgment is made, and if they do not match, it is determined that they do not match.

The position conditions of the table may be graded according to the strictness. For example, if the clinical information is "ischemia in the left brain", for example, the positional condition with mild strictness is defined as "left brain", and the positional condition with high strictness is defined as "left brain other than the ventricular". Is regulated. The source density to be adopted can be arbitrarily set by the user.

The consistency determination process may be performed individually for all the displayed texts, or the displayed texts may be determined in order whether or not they are consistent, and the process ends when it is determined that there is no consistency. Is also good.

When step S75 is performed, the processing circuit 11 executes the display control function 116 (step S76). In step S76, the processing circuit 11 displays the comparison result in step S75 on the display 19. Various modes can be considered for the display form of the comparison result.

FIG. 10 is a diagram showing an example of the display screen IS4 of the comparison result of the matching determination function 114. As shown in FIG. 10, the display image I3 and the text T3 are displayed side by side on the display screen IS4. The annotation IA is displayed in the drawing area of the annotation after the display image I3. Annotation IA is displayed in a specific color such as red.

As a comparison result, the processing circuit 11 highlights and displays the text determined to be "unmatched" or "matched" among the text T3-1, the text T3-2, and the text T3-3. For example, as shown in FIG. 10, when the annotation IA is specified in the left brain even though the text “Side of symptoms” T3-1 is displayed, the drawing area (that is, the imparting area) of the annotation IA is It is determined that the text "Side of symptoms" is inconsistent with the clinical information represented by T3-1. In this case, the text "Side of symptoms" T3-1 is highlighted as a warning with a different visual effect than the other texts "Time since onset" T3-2 and "Neurological defect" T3-3. As a visual effect of the warning, for example, as shown in FIG. 10, when the text determined to be "inconsistent" is displayed with a thick line and the text determined to be "consistent" is displayed with a line of normal thickness. good. As the visual effect of emphasis, the line type of the text may be changed, the color of the background or the character may be changed, the enlargement or reduction may be performed, the blinking may be performed, or the like.

By emphasizing and displaying the text determined to be "inconsistent" or "inconsistent" in this way, the user can clearly indicate that the annotation area and the clinical information represented by the text are not consistent. Can be grasped. This makes it possible to reduce annotation errors.

As another display form of the comparison result of "no matching", the processing circuit 11 may display a warning message such as "annotation does not match the text!" On the display 19. In addition, the processing circuit 11 may emit a voice or a warning sound of the warning message via a speaker or the like. Further, the processing circuit 11 may change the visual effect of the annotation depending on whether it is determined to be "matched" or "unmatched". For example, if it is determined to be "consistent", the annotation may be displayed in yellow, and if it is determined to be "unmatched", the annotation may be displayed in red. Further, the determination result of the presence / absence of matching may be attached to the annotation as text data and stored in the memory 13.

In addition, the display form of the comparison result of other matching will be described. For example, the processing circuit 11 does not have to draw the annotation in the drawing area of the annotation specified by the user when it is determined that there is no matching. In other words, the processing circuit 11 draws the annotation in the drawing area only when the drawing area of the annotation is specified in the anatomical part matching the clinical information of the text. In this way, the annotation is not displayed even though the drawing area of the annotation is specified, so that the user can clearly understand that the drawing area of the annotation and the clinical information represented by the text are not consistent. can do. This makes it possible to reduce annotation errors.

As a result, the annotation processing according to the present embodiment is completed.

The above annotation processing is an example and can be modified in various ways. For example, in the above description, it is assumed that steps S71 to S76 in FIG. 8 are performed as step S7 in FIG. However, the process of FIG. 8 may be performed independently. That is, the text may be displayed side by side with respect to the display image of the original cross section of the shooting axis reference, or the drawing area of the annotation may be specified.

In the above description, it is assumed that only the medical image data relating to the case for which the teacher data is created is stored in the memory 13. However, this embodiment is not limited to this. For example, medical image data relating to both the case to be created and the case to be non-created as teacher data may be stored in the memory 13. In this case, the processing circuit 11 reads only the text data related to the case for which the teacher data is created from the text data associated with the medical image data in the text reading function 112 of step S71 in FIG. Is good. Further, the memory 13 may store medical image data relating to a case for which teacher data is to be created and medical image data relating to a case to which teacher data is not created. In this case, the processing circuit 11 may read only the medical image data related to the case for which the teacher data is to be created in the image reading function 111 in step S1 of FIG. By implementing this function, a large amount of medical image data and text data can be stored in the memory 13 regardless of the target for creating the teacher data. Then, the processing circuit 11 can read out only the data related to the case for which the teacher data is to be created ex post facto from the large amount of medical image data and text data.

In the above description, it is assumed that the display image and the text indicating clinical information are displayed on the display 19 in step S72. However, this embodiment is not limited to this. For example, in step S72, the text indicating clinical information may not be displayed, and only the displayed image may be displayed. In the embodiment, the user can determine whether or not the annotation region of the annotation is correct in light of clinical information by referring to the comparison result in step S76.

Next, the efficiency and accuracy of the annotation area will be described.

The processing circuit 11 according to the present embodiment makes it possible to independently adjust the slab thickness related to the display image and the slab thickness related to the annotation in order to improve the efficiency and accuracy of annotation addition. Therefore, the processing circuit 11 includes a GUI tool for adjusting the slab thickness of the display image (hereinafter referred to as an image thickness adjustment tool) and a GUI tool for adjusting the slab thickness of the annotation (referred to as an annotation thickness adjustment tool). Is displayed on the display screen.

FIG. 11 is a diagram showing a display screen IS5 displayed on the display 19 in which the image thickness adjusting tool GI and the annotation thickness adjusting tool GA are displayed. The display screen IS is displayed, for example, in step S7 of FIG. 3 or step S72 of FIG. 9 for adding annotation IA.

As shown in FIG. 11, the display image I5 is displayed on the display screen IS5. Annotation IA is drawn on the display image I5. The image thickness adjustment tool GI and the annotation thickness adjustment tool GA are displayed alongside the display image I5. The image thickness adjustment tool GI and the annotation thickness adjustment tool GA are visually distinguished by color, line type, or the like. The image thickness adjustment tool GI is a GUI tool for adjusting the slab thickness of the displayed image. The image thickness adjusting tool GI includes a knob GI1, a pair of bar GI2, and a center bar GI3. The spacing between the pair of bars GI2 indicates the slab thickness of the displayed image I5. The center bar GI3 indicates the center of the slab thickness of the display cross section of the display image I5. For example, by moving the GI1 upward while dragging the upper GI1 with a mouse or the like, the distance between the pair of bars GI2 is widened. When the drawing area to be annotated is designated for the display image I5, the processing circuit 11 calculates the slab thickness according to the interval between the pair of bars GI2 in real time, and displays the display indicated by the center bar GI3 in the medical image data. The position of the cross section is specified, the image area of the medical image data including the specified position and having the slab thickness is specified, the display image I5 is generated based on the specified image area, and the display image I5 is displayed in real time on the display 19. do.

As shown in FIG. 11, the annotation thickness adjusting tool GA includes a knob GA1, a pair of bars GA2, and a center bar GA3. The spacing between the pair of bars GA2 indicates the slab thickness of annotation IA. The center bar GI3 indicates the center of the slab thickness of the annotation IA. For example, by moving the GA1 upward while dragging the upper GA1 with a mouse or the like, the distance between the pair of bars GA2 is widened. The processing circuit 11 calculates the slab thickness according to the interval between the pair of bars GA2 in real time, specifies the center position of the annotation IA indicated by the center bar GA3 in the medical image data, includes the specified center position, and said the same. An image area of medical image data having a slab thickness is specified, and the image area is set as an annotation IA imparting area.

FIG. 12 is a diagram schematically showing the slab thickness SI of the display image I5 of FIG. 11 and the slab thickness SA of the annotation IA. The annotation area RA of the annotation IA is set only in the image area corresponding to the slab thickness SA of the annotation IA, not in the image area corresponding to the slab thickness of the display image I5. By providing the image thickness adjusting tool GI and the annotation thickness adjusting tool GA, the processing circuit 11 can individually set the slab thickness SI of the display image I5 and the slab thickness SA of the annotation IA. The magnitude relationship between the slab thickness SI of the display image I5 and the slab thickness SA of the annotation IA is not particularly limited. For example, as shown in FIG. 12, by making the slab thickness SA narrower than the slab thickness SI, the region to which the annotation IA is added can be determined with a higher resolution than the displayed image. Further, by expanding the slab thickness SA more than the slab thickness SI, the region to which the annotation IA is applied can be determined easily and with high efficiency.

FIG. 13 is a diagram showing another display example of the image thickness adjusting tool GI and the annotation thickness adjusting tool GA displayed on the display 19. As shown in FIG. 13, the display screen IS6 displays an axial image I6-1 as a main image, and a coronal image I6-2 and a sagittal image I6-3 as sub-images.

As shown in FIG. 13, each image I6 displays a cross-section line L indicating a display cross-section of another image. Specifically, the cross-section line LAC showing the coronal cross section of the coronal image I6-2 and the cross-section line LAS showing the sagittal cross-section of the sagittal image I6-3 are superimposed on the axial image I6-1, and are superimposed on the coronal image I6-2. Is superimposed the cross-section line LCA showing the axial cross-section of the axial image I6-1 and the cross-section line LCS showing the sagittal cross-section of the sagittal image I6-3, and the sagittal image I6-3 shows the axial cross-section of the axial image I6-1. The cross-section line LSA and the cross-section line LSC showing the coronal cross-section of the coronal image I6-2 are superimposed. It is also possible to update the cross-sectional image in conjunction with the movement operation of each cross-sectional line L.

As shown in FIG. 13, the image thickness adjustment tool GI-1 for adjusting the slab thickness of the axial cross section adjacent to the axial image I6-1 and the slab thickness of the coronal cross section adjusted adjacent to the coronal image I6-2. The image thickness adjusting tool GI-2 for adjusting the image thickness and the image thickness adjusting tool GI-3 for adjusting the slab thickness of the sagittal cross section are displayed adjacent to the sagittal image I6-3. For the axial image I6-1 which is the main image, the annotation thickness adjustment tool GA is displayed adjacent to the image thickness adjustment tool GI-1. As shown in FIG. 13, the image thickness adjusting tool GI is preferably displayed on an extension line of each cross-sectional line L. More specifically, the center bar of the image thickness adjustment tool GI is displayed so as to coincide with the section line L. This makes it possible to intuitively understand the relationship between the display cross section indicated by the cross section line L and the slab thickness of the display cross section indicated by the image thickness adjustment tool GI. The annotation slab thickness can be adjusted by the annotation thickness adjustment tool GA. Similarly, the annotation thickness adjusting tool GA may be displayed on an extension line of each section line L, and more specifically, the center bar of the annotation thickness adjusting tool GA is displayed so as to coincide with the section line L. This makes it possible to intuitively understand the relationship between the display cross section indicated by the cross section line L, the slab thickness of the display cross section indicated by the image thickness adjustment tool GI, and the slab thickness of the annotation indicated by the annotation thickness adjustment tool GA. ..

For example, the user specifies the drawing area of the annotation IA in the axial image I6-1 which is the main image via the input interface 15. The annotation IA drawn on the axial image I6-1 is also reflected in other cross sections. That is, as described above, when the drawing area of the annotation IA is specified, the processing circuit 11 determines the area to which the annotation IA is added in the medical image data corresponding to the drawing area. Further, the processing circuit 11 calculates the drawing area in the coronal cross section and the sagittal cross section based on the position of the coronal cross section and the sagittal cross section and the imparted area in the medical image data, and draws the annotation in the drawing area. In FIG. 14, the coronal cross section has a drawing area, but the sagittal cross section does not have a drawing area. In this case, the processing circuit 11 draws the annotation IA in the drawing area of the coronal cross section.

The form of the image thickness adjusting tool GI and the annotation thickness adjusting tool GA is not particularly limited as long as each slab thickness can be adjusted. For example, the image thickness adjustment tool GI and the annotation thickness adjustment tool GA may be realized by a GUI tool such as a slider.

When determining the annotation area three-dimensionally, the user slides a display section having a given slab thickness and specifies a drawing area at a plurality of display section positions. For example, a drawing area is specified at multiple axial cross-section positions while sliding an axial cross-section with a given slab thickness. At this time, the processing circuit 11 may slide the axial cross section stepwise along the subject axis without overlapping with each other, or may slide smoothly while overlapping.

Further, when the granting area is designated by two display images separated from each other in the granting area determination function 113, the processing circuit 11 annotates the image area including the two granting areas corresponding to the two granting areas. Set in the grant area of. This process is called annotation interpolation process.

FIG. 14 is a diagram schematically showing an annotation interpolation process by the grant area determination function 113. In FIG. 14, for the sake of simplicity, the slab thickness of the display image I7 and the slab thickness of the annotation are assumed to be the same. First, when the drawing area of the annotation is specified in the first display image I7-1, the given area RA1 corresponding to the drawing area is determined in the medical image data DI. Next, the cross-sectional movement is performed according to the instruction via the input interface 15 or the like, and the second display image I7-2 separated from the first display image is set. Similarly, in the second display cross section 17-2, the annotation region RA2 is determined in the medical image data DI.

Next, the processing circuit 11 performs interpolation processing. In the interpolation process, the processing circuit 11 determines an image area including the given area RA1 and the given area RA2 in the medical image data DI, and sets the image area in the given area RAI1 after interpolation. Specifically, all the pixels on the straight line connecting the imparting region RA1 and the granting region RA2 are specified. The specified set of pixels can be referred to as an annotation interpolation region RA3. The set of the interpolation region RA3, the grant region RA1 and the grant region RA2 is set in the interpolation region RAI1 after interpolation. The interpolation is not limited to the above-mentioned linear interpolation, and may be arbitrary curve interpolation or interpolation that combines a straight line and a curve. By performing the annotation interpolation processing in this way, the annotation addition processing can be performed easily and with high efficiency.

In the case of simple interpolation processing, the annotation interpolation region RA3 may include an image region that should be excluded from the annotation region, that is, an image region that is not the target of image recognition. For example, when it is desired to set an annotation region in the ischemic portion of the brain parenchyma, the ventricles that are not the target of image recognition may be included in the interpolated region. The processing circuit 11 sets a region excluding the non-target area of image recognition from the interpolated imparted region RAI1 as the interpolated imparted region. This process is called annotation interpolation & exclusion process.

FIG. 15 is a diagram schematically showing the application of annotation interpolation processing by the grant area determination function 113. As shown in FIG. 14, as in FIG. 14, in the medical image data DI, the annotation region RA1 derived from the first display image I7-1 and the annotation region RA2 derived from the second display image I7-2 are added. Is decided.

Next, the processing circuit 11 performs interpolation & exclusion processing. In the interpolation & exclusion process, the processing circuit 11 first extracts the exclusion target area RB3 included in the medical image data DI. The excluded area corresponds to any anatomical site such as the ventricle. The processing circuit 11 can extract the exclusion target region RB by a threshold value or the like, or can extract it by using an anatomical reference point. The extraction process of the exclusion target area RB is not particularly limited. Next, the processing circuit 11 determines an image area including the given area RA1 and the given area RA2 in the medical image data DI, and sets the image area excluding the exclusion target area RB3 from the image area into the interpolated giving area RAI2. Set. Specifically, an interpolation region (similar to the interpolation region RA3 in FIG. 14) connecting the grant region RA1 and the grant region RA2 is specified, and the exclusion target region RB is excluded from the interpolation region. The interpolation area after exclusion of the exclusion target area RB is set in the interpolation area RA4 after interpolation & exclusion. The set of the interpolation region RA4, the grant region RA1 and the grant region RA2 after the interpolation & exclusion is set in the interpolation region RAI2 after the interpolation. By performing the interpolation & exclusion processing in this way, the exclusion target area RB can be automatically excluded from the annotation addition area, so that the annotation addition processing can be performed accurately and easily.

In the above embodiment, the clinical information is defined as the clinical content indicated by words, phrases or sentences included in the findings and the like. However, this embodiment is not limited to this. For example, the clinical information may be the position of the annotation region of the annotation given to the past medical image data regarding the same subject as the subject of the medical image data read by the image reading function 111. The past medical image data may be medical image data used as teacher data, or may be simply annotated medical image data. In this case, the processing circuit 11 executes the matching determination function 114, and the position of the annotation region included in the current medical image data read by the image reading function 111 and the annotation included in the past medical image data. Compare with the position of the grant area. The position of the imparting region may be the coordinates in the medical image data or the anatomical portion of the imparting region. As a display form of the comparison result, for example, when the position of the annotation region of this time and the position of the annotation region of the past are significantly different from each other, the processing circuit 11 may display a warning on the display 19. Further, if the position of the annotation area of this time and the position of the annotation area of the past annotation are not significantly different, a message or the like to that effect may be displayed on the display 19.

As another example, the clinical information is the recognition result of image recognition for the past medical image data regarding the same subject as the subject of the medical image data read by the image reading function 111. The past medical image data is preferably medical image data collected by another modality in which the same recognition target as the recognition target of this time is clearly visualized. The recognition result of image recognition is an image area related to a recognition target. In this case, the processing circuit 11 executes the matching determination function 114, and the position of the annotation region included in the current medical image data read by the image reading function 111 and the image recognition result for the past medical image data. Compare with the position of. For example, the current medical image data is CT image data collected by an X-ray computer tomography device, the past medical image data is NM image data collected by a nuclear medicine diagnostic device, and the myocardial ischemic site is recognized. It is assumed that the subject is an image region recognized as a myocardial ischemic site by image recognition. The position of the imparted region and the position of the image recognition result may be the coordinates in the medical image data or may be the anatomical portion of the imparted region. As a display form of the comparison result, for example, when the position of the annotation region of this time and the position of the past image recognition result are significantly different from each other, the processing circuit 11 may display a warning on the display 19. Further, if the position of the region to which the annotation is added this time and the position of the past image recognition result are not significantly different, a message or the like to that effect may be displayed on the display 19.

As described above, the medical image processing apparatus 1 according to the present embodiment has at least a display 19 and a processing circuit 11. The display 19 displays medical image data used as teacher data for machine learning related to image recognition. The processing circuit 11 realizes the grant area determination function 113, the matching determination function 114, and the display control function 116. In the grant area determination function 113, the processing circuit 11 determines the annotation region of the medical image data that defines the target region for image recognition according to the instruction by the user. In the matching determination function 114, the processing circuit 11 outputs a comparison result between the position of the imparted region in the medical image data and the clinical information associated with the medical image data. In the display control function 116, the processing circuit 11 displays the comparison result by the matching determination function 114 on the display 19.

With the above configuration, when the area to be annotated is specified, the position of the area is compared with the clinical information and the comparison result is displayed, so that the user can immediately determine whether or not the specified area is correct. You can judge. In order to create teacher data for machine learning related to image recognition, it is necessary to perform the work of annotating a large number of images, but according to the present embodiment, this work can be performed easily and accurately.

According to at least one embodiment described above, it is possible to reduce the trouble of creating teacher data for machine learning related to image recognition.

The term "processor" used in the above description refers to, for example, a CPU, a GPU, or an application specific integrated circuit (ASIC), a programmable logic device (for example, a simple programmable logic device (Simple Programmable Logic Device)). : SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)). The processor realizes the function by reading and executing the program stored in the storage circuit. Instead of storing the program in the storage circuit, the program may be directly embedded in the circuit of the processor. In this case, the processor realizes the function by reading and executing the program embedded in the circuit. Further, instead of executing the program, a function corresponding to the program may be realized by a combination of logic circuits. It should be noted that each processor of the present embodiment is not limited to the case where each processor is configured as a single circuit, and a plurality of independent circuits may be combined to form one processor to realize its function. good. Further, the plurality of components in FIG. 1 may be integrated into one processor to realize the function.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and variations thereof are included in the scope of the invention described in the claims and the equivalent scope thereof, as are included in the scope and gist of the invention.

1 Medical image processing device 11 Processing circuit 13 Memory 15 Input interface 17 Communication interface 19 Display 111 Image reading function 112 Text reading function 113 Granted area determination function 114 Consistency judgment function 115 Image processing function 116 Display control function 117 Storage control function 118 Machine learning function

Claims (18)

A display that displays medical image data and
In order to generate machine learning teacher data related to image recognition by annotating the medical image data, a determination unit that determines an annotation region of the medical image data according to a user's instruction. ,
A matching determination unit that outputs a comparison result between the position of the imparted region in the medical image data and clinical information associated with the medical image data, and a matching determination unit.
A display control unit that displays the comparison result on the display,
Equipped with
The display control unit displays a plurality of texts corresponding to the plurality of anatomical position conditions related to the clinical information together with the medical image data on the display screen in the annotation processing.
The matching determination unit determines whether or not the position of the imparting region satisfies the plurality of anatomical position conditions.
When the display control unit determines that the position of the imparted region does not satisfy any of the plurality of anatomical position conditions, the display control unit determines that the anatomical position of the plurality of texts does not satisfy the anatomical position. Distinguish the text corresponding to the positional condition,
Medical image processing equipment. The plurality of texts are texts relating to the case for which the teacher data is to be created, and are extracted from the findings which are the clinical information.
The medical image processing apparatus according to claim 1. The medical image processing apparatus according to claim 1, wherein each of the plurality of anatomical position conditions is graded according to the degree of rigor. The matching determination unit
Identify the anatomical reference point corresponding to the anatomical site indicated by the clinical information.
Whether or not the position of the imparted region satisfies the plurality of anatomical position conditions is determined via the anatomical reference point.
The medical image processing apparatus according to claim 1. The medical image processing device according to claim 1, wherein the display control unit draws the annotation in an image area corresponding to the imparted area in a display image based on the medical image data. The medical image processing apparatus according to claim 5 , wherein the display control unit draws the annotation on the image region only when the imparted region is designated for the anatomical site corresponding to the clinical information. Further, an image generation unit that generates a display image based on the medical image data is provided.
The image generation unit generates a first display image relating to the original display cross section with the imaging axis of the medical modality used for collecting the medical image data as a reference line.
The display control unit displays the first display image on the display.
When the image generation unit receives a change instruction, based on the medical image data, the image generation unit displays a second display image relating to a secondary display cross section having the reference axis of the patient region included in the medical image data as the reference line. Generate and
The display control unit displays the second display image on the display in place of the first display image.
The medical image processing apparatus according to claim 1. Further provided with a storage unit for storing the data of the display image,
The display control unit displays a save button for saving the data of the display image on the display.
The storage unit stores the data of the second display image and the grant area when the save button is pressed after the image area corresponding to the grant area is designated in the second display image. ,
The medical image processing apparatus according to claim 7 . The medical image processing device according to claim 8 , wherein the display control unit displays another first display image based on other medical image data used as teacher data on the display when the save button is pressed. .. The display control unit has a first GUI tool for adjusting a first slab thickness for a display image based on the medical image data, and a second slab thickness for adjusting the second slab thickness for the imparted region of the annotation. The medical image processing apparatus according to claim 1, wherein the second GUI tool is displayed on the display. When the first image area to be annotated is specified in the display image, the determination unit determines the position of the display cross section of the display image in the medical image data, the second slab thickness, and the first. 10. The medical image processing apparatus according to claim 10 , wherein a second image region in the medical image data is determined based on the image region of the above, and the second image region is set as the imparted region. The medical image processing apparatus according to claim 10 , wherein the display control unit visually distinguishes and displays the first GUI tool and the second GUI tool. When the first image area to be annotated is designated by the first display cross section and the second display cross section, the determination unit obtains the first display cross section and the second display cross section. The medical image processing apparatus according to claim 10 , wherein a second image region to be included is set in the imparted region. The medical image processing apparatus according to claim 13 , wherein the determination unit sets a region excluding a region not subject to image recognition from the second image region as the imparted region. The medical image processing apparatus according to claim 1, wherein the clinical information is the position of an annotation region given to past medical image data relating to the same subject as the subject of the medical image data. The medical image processing apparatus according to claim 1, wherein the clinical information is a recognition result of image recognition for past medical image data relating to the same subject as the subject of the medical image data. To a computer that can communicate with a display that displays medical image data,
In order to generate machine learning teacher data related to image recognition by annotating the medical image data, a function of determining the area to be annotated in the medical image data according to a user's instruction.
A function to output a comparison result between the position of the imparted region in the medical image data and clinical information associated with the medical image data, and
The function of displaying the comparison result on the display and
Realized ,
The display function displays a plurality of texts corresponding to a plurality of anatomical position conditions related to the clinical information together with the medical image data on the display screen in the annotation processing.
The output function determines whether or not the position of the imparted region satisfies the plurality of anatomical position conditions.
When it is determined that the position of the imparted region does not satisfy any of the plurality of anatomical position conditions, the function to be displayed is an anatomical determination that the position of the plurality of texts is not satisfied. Distinguish the text corresponding to the positional condition,
Teacher data creation program. It is a method of creating teacher data by a computer that can communicate with the display on which medical image data is displayed.
In order to generate machine learning teacher data related to image recognition by annotating the medical image data, a step of determining an annotation region of the medical image data according to a user's instruction.
A step of outputting a comparison result between the position of the imparted region in the medical image data and clinical information associated with the medical image data, and
The step of displaying the comparison result on the display and
Equipped with
In the step of displaying, a plurality of texts corresponding to a plurality of anatomical position conditions relating to the clinical information are displayed together with the medical image data on the display screen in the annotation processing.
In the output step, it is determined whether or not the position of the imparting region satisfies the plurality of anatomical position conditions.
In the step of displaying, when it is determined that the position of the imparting region does not satisfy any of the plurality of anatomical position conditions, the anatomical determination of the plurality of texts is not satisfied. Distinguish the text corresponding to the positional condition,
How to create teacher data.

Images