JP2020010805A - Specification device, program, specification method, information processing device, and specifier - Google Patents

Abstract

To provide a specification device capable of specifying a state of an imaging object on the basis of an image.SOLUTION: A specification device 20 acquires each of a plurality of kinds of images imaged by different methods for an imaging object. The specification device discriminates a state of the imaging object in each of the acquired images by using a learning model of the learning based on teacher data including the image and the state of the imaging object in the image for each kind of the image. The specification device specifies the state of the imaging object on the basis of a result of the discrimination for each kind of the image.SELECTED DRAWING: Figure 7

Description

  The present disclosure relates to a specifying device, a program, a specifying method, an information processing device, and a specifying device.

  In the medical field, imaging such as X-ray photography, CT (Computed Tomography), MRI (Magnetic Resonance Imaging), PET (Positron Emission Tomography), PET-CT, etc. 2. Description of the Related Art A patient is photographed using an apparatus, and a lesion is diagnosed based on the obtained medical image. Since the imaging device can observe the biological tissue from different viewpoints depending on the type and the provided imaging means, the doctor makes a diagnosis while comparing medical images captured by each device. Patent Literature 1 discloses an apparatus that generates an integrated image in which X-ray CT image data and MRI image data are integrated so as to overlap each other. In the device disclosed in Patent Literature 1, two types of medical images can be integrated and displayed, so that information on a desired biological tissue included in each medical image can be confirmed as one integrated image.

JP-A-2014-182

  The apparatus disclosed in Patent Literature 1 integrates and displays two images of the same object captured by different imaging units, but does not specify the state of the observation target based on a plurality of images.

  The present disclosure has been made in view of such circumstances, and a purpose thereof is to provide a specification device and the like that can specify a state of a shooting target based on an image.

  A specific device according to an aspect of the present disclosure is a specific device that specifies a state of an imaging target, and includes a plurality of image acquisition units that respectively acquire a plurality of types of images imaged by different methods for the imaging target. Using a learning model that is provided for each type of the image and that is learned based on teacher data including the state of the image and the shooting target in the image, and that the image obtaining unit obtains the shooting target in the image. The image processing apparatus includes a plurality of determination units for determining states, and a specification unit for specifying the state of the imaging target based on a determination result by each of the determination units.

  According to the present disclosure, a state of a shooting target can be specified based on an image.

It is a schematic diagram which shows the example of a structure of a diagnosis support system. It is a block diagram showing an example of composition of a learning device. It is a schematic diagram which shows the example of a mammography image. It is a schematic diagram which shows the example of the image after a filter process. It is a block diagram showing a function realized by the control unit of the learning device, FIG. 9 is an explanatory diagram of a learning process performed by the learning device. It is a block diagram showing an example of composition of a specific device. It is a block diagram showing a function realized by a control unit of a specific device. FIG. 9 is an explanatory diagram of a specific process performed by a specific device. It is a flowchart which shows the procedure of the learning process by a learning device. It is a flowchart which shows the procedure of the specific processing by a specific device. It is a schematic diagram which shows the example of a T2-weighted image. It is a schematic diagram which shows the example of a T1CE image. FIG. 13 is a block diagram illustrating functions realized by a control unit of the learning device according to the third embodiment. FIG. 14 is an explanatory diagram of a learning process performed by the learning device according to the third embodiment. FIG. 14 is an explanatory diagram of a learning process performed by the learning device according to the third embodiment. FIG. 13 is a block diagram illustrating functions realized by a control unit of a specific device according to a third embodiment. FIG. 14 is an explanatory diagram of a specifying process performed by a specifying device according to a third embodiment. FIG. 14 is an explanatory diagram of a specifying process performed by a specifying device according to a third embodiment. FIG. 14 is an explanatory diagram of a specifying process performed by a specifying device according to a third embodiment. 13 is a flowchart illustrating a procedure of a learning process performed by the learning device according to the third embodiment. 13 is a flowchart illustrating a procedure of a specific process performed by a specific device according to a third embodiment. 13 is a flowchart illustrating a procedure of a specific process performed by a specific device according to a third embodiment. 4 is a chart showing a configuration of test data. 6 is a table showing the accuracy of determination by a specific device. It is a schematic diagram which shows the example of a mammography image. It is a schematic diagram which shows the example of a difference image. It is a block diagram showing the function realized by the control part of the specific device of Embodiment 5. 15 is a flowchart illustrating a part of a procedure of a specific process performed by a specific device according to a fifth embodiment.

  Hereinafter, regarding the specific device, the program, the specific method, the information processing device, and the specific device of the present disclosure, a medical image obtained by capturing an affected part of a patient using an imaging device such as X-ray imaging, CT, MRI, PET, or PET-CT. A detailed description will be given with reference to the drawings showing an embodiment applied to a diagnosis support system that supports a diagnosis process of determining a state of an affected part of a patient based on the diagnosis. Note that the present disclosure is preferably applicable to a system that supports a diagnosis process of determining a state of a lesion based on medical images obtained by capturing various lesions such as a breast mass, a lung cancer, and a brain tumor. Nor is it limited to the application to a diagnosis support system.

(Embodiment 1)
A diagnosis support system for determining whether or not a breast is a tumor based on a medical image (X-ray image) obtained by imaging a breast of a patient suspected of having a breast tumor by mammography will be described. FIG. 1 is a schematic diagram illustrating a configuration example of a diagnosis support system. The diagnosis support system of the present embodiment includes a learning device 10 and a specific device 20 installed in a medical institution, and the learning device 10 and the specific device 20 can be connected to a network such as the Internet. The learning device 10 is a device that learns a classifier (learning model) that determines whether or not it is a breast mass based on a medical image using teacher data described later. An apparatus that determines (identifies) whether or not the subject is a breast mass based on a medical image of a patient by using a classifier that has been trained in 10. When acquiring the classifier that has been learned from the learning device 10, the identifying device 20 acquires the learned classifier via a network, or using a portable storage medium such as a USB (Universal Serial Bus) memory or a CD-R (compact disc recordable). I do.

  FIG. 2 is a block diagram illustrating a configuration example of the learning device 10. The learning device 10 is a personal computer, a server computer, or the like. The learning device 10 includes a control unit 11, a storage unit 12, a display unit 13, an input unit 14, a communication unit 15, and the like, and these units are mutually connected via a bus. The control unit 11 includes one or a plurality of processors such as a central processing unit (CPU), a micro-processing unit (MPU), or a graphics processing unit (GPU). The control unit 11 performs various types of information processing, control processing, and the like to be performed by the learning device 10 by appropriately executing a control program stored in the storage unit 12.

  The storage unit 12 includes a random access memory (RAM), a flash memory, a hard disk, a solid state drive (SSD), and the like. The storage unit 12 stores in advance a control program to be executed by the control unit 11 and various data necessary for executing the control program. The storage unit 12 temporarily stores data and the like generated when the control unit 11 executes the control program. The control program stored in the storage unit 12 includes a learning program 12a for executing a learning process of the first classifier 12b and the second classifier 12d. Further, the storage unit 12 stores a first classifier 12b and a second classifier 12d, which are CNN (Convolution Neural Network) models constructed by machine learning processing, for example. The data stored in the storage unit 12 includes a first teacher data DB 12c storing first teacher data for learning the first classifier 12b and a second teacher data for learning the second classifier 12d. The stored second teacher data DB 12e is included.

  The control program and data stored in the storage unit 12 are acquired from an external device via a network via the communication unit 15 and stored in the storage unit 12, for example. When the learning device 10 includes a reading unit that reads information stored in the portable storage medium, the control program and data stored in the storage unit 12 are read from the portable storage medium and stored in the storage unit 12. It may be stored. Further, the first teacher data DB 12c and the second teacher data DB 12e may be stored in an external storage device connected to the learning device 10, or stored in a storage device that can communicate with the learning device 10 via a network. Is also good.

  The display unit 13 is, for example, a liquid crystal display, an organic EL display, or the like, and displays various types of information according to instructions from the control unit 11. The input unit 14 receives an operation input by the user and sends a control signal corresponding to the operation content to the control unit 11. The display unit 13 and the input unit 14 may be an integrated touch panel. The communication unit 15 is an interface for connecting to a network by wire communication or wireless communication, and transmits and receives information to and from an external device via the network.

  Hereinafter, a medical image used in the diagnosis support system of the present embodiment will be described. Different types of medical images can be obtained by performing a filter process using a different image processing filter on a captured image (hereinafter, referred to as a mammographic image) obtained by mammography. In the diagnosis support system according to the present embodiment, an image after applying a Gabor filter (hereinafter referred to as a Gabor image) obtained by performing a filtering process using a Gabor filter on a mammography image and an image obtained after applying an iris filter using a filtering process using an iris filter (Hereinafter referred to as an iris image) to determine (determine) whether or not the state of the imaging target is a tumor. The medical image used for the determination is not limited to these, and a medical image that has been subjected to filter processing using another filter may be used. Also, the plurality of types of medical images used for the determination may be captured images of the same imaging target (patient's breast) captured in the same state, and are captured by the same imaging device at the same imaging timing with different parameters. Or an image taken by a different imaging device.

  FIG. 3 is a schematic diagram showing an example of a mammography image, and FIG. 4 is a schematic diagram showing an example of an image after filtering. FIG. 3A shows an example of a mammography image, and FIG. 3B shows an example of a tumor image T including a tumor region and a normal image N including a normal region extracted from the mammography image. In the present embodiment, when generating teacher data, a tumor image T including a tumor region and a normal image N including a normal region without a tumor are extracted from a mammography image of a patient having a tumor. Then, a filtering process using a Gabor filter is performed on each of the extracted tumor image T and normal image N to generate a Gabor image, and a filtering process using an iris filter is performed to generate an iris image. FIG. 4A shows an example of a Gabor image generated from the tumor image T and the normal image N shown in FIG. 3B, and FIG. 4B shows an example of an iris image generated from the tumor image T and the normal image N shown in FIG. 3B. Show. The size of the Gabor image and the iris image can be, for example, 128 pixels × 128 pixels, but is not limited thereto. In the diagnosis support system of the present embodiment, as the type of the state to be determined, it is determined whether the state of the lesion to be imaged is a tumor or a normal state (whether or not it is a tumor). Therefore, the first classifier 12b is a classifier that determines whether the state of the lesion to be imaged is a tumor or a normal based on the Gabor image of the patient, and the second classifier 12d is a iris of the patient. It is a classifier that determines whether the state of a lesion to be imaged is a tumor or a normal based on an image.

  In the first teacher data DB 12c, a large number of Gabor images as shown on the left side of FIG. 4A are accumulated as images of the tumor region (tumor images), and as images of normal regions (normal images), Many Gabor images as shown are accumulated. In the first teacher data DB 12c, a set of one Gabor image and information indicating a tumor or normal associated with the Gabor image is referred to as first teacher data. In the second teacher data DB 12e, many iris images as shown on the left side of FIG. 4B are accumulated as images of the tumor region, and many iris images as shown on the right side of FIG. 4B are accumulated as images of the normal region. Have been. In the second teacher data DB 12e, a set of one iris image and information indicating a tumor or normal associated with the iris image is referred to as second teacher data. The learning device 10 causes the first classifier 12b to learn based on the first teacher data based on the Gabor image as illustrated in FIG. 4A, and based on the second teacher data based on the iris image as illustrated in FIG. 4B. Train 12d.

  Next, a function realized by the control unit 11 of the learning device 10 executing the learning program 12a will be described. FIG. 5 is a block diagram illustrating functions realized by the control unit 11 of the learning device 10, and FIG. 6 is an explanatory diagram of a learning process performed by the learning device 10. When the control unit 11 of the learning device 10 executes the learning program 12a stored in the storage unit 12, the first teacher data acquisition unit 101, the first learning unit 104, the second teacher data acquisition unit 111, and the second learning The functions of the unit 114 are realized. In the present embodiment, each of these functions is realized by the control unit 11 executing the learning program 12a, but a part of them may be realized by a dedicated hardware circuit.

The first teacher data acquisition unit 101 sequentially acquires the first teacher data stored in the first teacher data DB 12c. The first teacher data includes information indicating a tumor or normal, and a Gabor image as shown in FIG. 4A.
The first learning unit 104 causes the first classifier 12b to learn based on the information indicating the tumor or normal acquired by the first teacher data acquisition unit 101 from the first teacher data DB 12c and the Gabor image. Here, the first classifier 12b will be described. FIG. 6 is a schematic diagram illustrating a configuration example of the first classifier 12b of the present embodiment. The first classifier 12b shown in FIG. 6 is configured by a convolutional neural network model (CNN). The configuration of the first classifier 12b is not limited to a multilayer neural network (deep learning) as shown in FIG. 6, but may use another machine learning algorithm.

  As shown in FIG. 6, the first classifier 12b includes an input layer, a middle layer, and an output layer. The intermediate layer includes a convolutional layer, a pooling layer, and a fully connected layer. In the first classifier 12b of the present embodiment, the number of nodes in the input layer is 16,384 (the number of channels is 1), the number of nodes in the output layer is 2, and the Gabor image as described above has been trained. It is provided as input data of the first classifier 12b. The Gabor image given to the node of the input layer is input to the intermediate layer, and in the intermediate layer, an image feature amount is extracted by a filtering process or the like in a convolutional layer to generate a feature map, which is compressed in a pooling layer to reduce the information amount. Be reduced. A plurality of convolutional layers and pooling layers are provided repeatedly, and a feature map generated by the plurality of convolutional layers and pooling layers is input to a fully connected layer. The total connection layer is provided in a plurality of layers (two layers in FIG. 6), and based on the input feature map, calculates the output values of the nodes of each layer using weights and various functions, and calculates the calculated output values. Input to the node of the layer after the order. The fully connected layer finally gives each node of the output layer a respective output value by sequentially inputting the output value of the node of each layer to the node of the subsequent layer. The number of each of the convolutional layer, the pooling layer, and the total connection layer is not limited to the example shown in FIG. Each node in the output layer outputs a classification probability for each of a tumor and a normal. For example, the node 0 outputs the probability that the input Gabor image is the image of the tumor region, and the node 1 outputs the probability that the image is the image of the normal region. The output value of each node in the output layer is, for example, a value of 0 to 1.0, and the total of the probabilities output from the two nodes is 1.0 (100%).

  The first learning unit 104 inputs the Gabor image acquired by the first teacher data acquiring unit 101 from the first teacher data DB 12c to a node of the input layer of the first classifier 12b, and outputs a tumor of the Gabor image in the output layer. Alternatively, the first classifier 12b is trained so that the output value of the node corresponding to the information indicating normality approaches 1.0 and the output value of the other node approaches 0. That is, when the Gabor image input to the node in the input layer is a tumor image, the first learning unit 104 determines that the output value of the node 0 approaches 1.0 and the output value of the node 1 approaches 0. The one classifier 12b is trained. The first learning unit 104 learns the first classifier 12b by optimizing, for example, a weight and a function for connecting the nodes of each layer of the fully connected layer by a learning algorithm. The first learning unit 104 trains the first classifier 12b using all the first teacher data stored in the first teacher data DB 12c. Thereby, the learned first classifier 12b is generated.

  Although each of the second teacher data acquisition unit 111 and the second learning unit 114 has a different image to be processed than the first teacher data acquisition unit 101 and the first learning unit 104, they perform the same processing. The second classifier 12d has the same configuration as the first classifier 12b. Therefore, a detailed description thereof will be omitted. The second teacher data acquisition unit 111 sequentially acquires the second teacher data stored in the second teacher data DB 12e. The second teacher data includes information indicating a tumor or normal, and an iris image as shown in FIG. 4B. The second learning unit 114 makes the second classifier 12d learn using all the second teacher data stored in the second teacher data DB 12e. Thereby, the learned second classifier 12d is generated.

  Hereinafter, using the first classifier 12b and the second classifier 12d trained by the learning device 10 as described above, the state of the lesion to be photographed is a tumor based on the medical image of the patient to be diagnosed. The identification device 20 that determines whether the data is normal or not (is not a tumor) will be described. FIG. 7 is a block diagram illustrating a configuration example of the specifying device 20. The specifying device 20 is a personal computer, a server computer, or the like, and has a configuration similar to that of the learning device 10, and thus the details are omitted. The identification device 20 includes a control unit 21, a storage unit 22, a display unit 23, an input unit 24, a communication unit 25, and the like, and these units are mutually connected via a bus. The storage unit 22 of the identifying device 20 stores an identifying program 22a that is a program of the present disclosure, and a first classifier 22b and a second classifier 22c that have been learned by the learning device 10. The first classifier 12b and the second classifier 12d that have been trained by the learning device 10 are collectively provided as a specifying device.

  The first classifier 22b and the second classifier 22c are the first classifier 12b and the second classifier 12d learned by the learning device 10. The identifying device 20 acquires the first classifier 22b and the second classifier 22c from the learning device 10 via the network via the communication unit 25, for example, and stores them in the storage unit 22. When the specifying device 20 includes a reading unit that reads information stored in the portable storage medium, the specifying device 20 reads the first classifier 22b and the second classifier 22c from the portable storage medium and reads the first classifier 22b and the second classifier 22c from the storage unit 22. May be stored. The specifying device 20 may obtain the specifying program 22a together with the first classifier 22b and the second classifier 22c via a network, or may obtain the specific program 22a using a portable storage medium.

  Next, functions realized by the control unit 21 executing the specific program 22a in the specific device 20 will be described. FIG. 8 is a block diagram illustrating functions realized by the control unit 21 of the specifying device 20, and FIG. 9 is an explanatory diagram of a specifying process performed by the specifying device 20. When the specific program 22a stored in the storage unit 22 is executed, the control unit 21 of the specific device 20 executes the first image acquisition unit 201, the first tumor region extraction unit 202, the first determination unit 205, and the second image acquisition The functions of the unit 211, the second tumor region extracting unit 212, the second determining unit 215, the specifying unit 221, and the output unit 222 are realized. In the present embodiment, each of these functions is realized by the control unit 21 executing the specific program 22a. However, a part of these functions may be realized by a dedicated hardware circuit.

  The first image acquisition unit 201 acquires a Gabor image of a patient to be diagnosed suspected of having a breast mass. The Gabor image here is a Gabor image generated by performing a filtering process using a Gabor filter on a mammography image of a patient. The identifying device 20 previously acquires the Gabor image of the patient via a network or a portable storage medium and stores the Gabor image in the storage unit 22 in advance, and the first image acquisition unit 201 stores the Gabor image of the diagnosis target in the storage unit. 22. When the identification device 20 acquires a mammography image of a patient via a network or a portable storage medium and stores the acquired mammography image in the storage unit 22, the first image acquisition unit 201 stores the mammography image to be diagnosed in the storage unit. 22. Then, the first image acquiring unit 201 generates a Gabor image by performing a filtering process using a Gabor filter on the read mammography image, and acquires a Gabor image of the patient.

  The first tumor region extraction unit 202 extracts a tumor region from the Gabor image acquired by the first image acquisition unit 201. The first tumor region extracting unit 202 extracts a tumor region from a Gabor image of the entire breast based on, for example, a designation made by a user (a doctor or the like) through the input unit 24. Thereby, a tumor region (tumor image) as shown on the left side of FIG. 4A is extracted. In addition, the first tumor region extracting unit 202 may recognize and extract a tumor region in the Gabor image acquired by the first image acquiring unit 201 using a recognition model that has been learned by deep learning, for example.

  The first determination unit 205 determines, using the first classifier 22b, whether the state of the imaging target is a tumor or a normal based on the tumor region (tumor image) extracted by the first tumor region extraction unit 202. I do. Specifically, the first determination unit 205 inputs the extracted tumor region to the first classifier 22b, and obtains an output value from the first classifier 22b. The output value of the first classifier 22b is a classification probability for a tumor or normal. The first determination unit 205 stores the determination result in, for example, the storage unit 22.

  Each of the second image acquisition unit 211, the second tumor region extraction unit 212, and the second determination unit 215 is an image to be processed by the first image acquisition unit 201, the first tumor region extraction unit 202, and the first determination unit 205. , But perform the same processing. Therefore, the detailed description is omitted. The Gabor image acquired by the first image acquisition unit 201 and the iris image acquired by the second image acquisition unit 211 are images of the breast (imaging target) of the same patient.

  The second image acquisition unit 211 acquires an iris image of a patient to be diagnosed. Also in this case, the identification device 20 acquires the iris image of the patient via a network or via a portable storage medium and stores the iris image in the storage unit 22, and the second image acquisition unit 211 stores the iris image to be diagnosed in the storage unit. 22. When the identification device 20 stores the mammography image of the patient in the storage unit 22, the second image acquisition unit 211 reads out the mammography image to be diagnosed from the storage unit 22, performs a filtering process using an iris filter, and performs iris filtering. An image may be generated. The second tumor region extraction unit 212 extracts a tumor region from the iris image acquired by the second image acquisition unit 211. The second determination unit 215 inputs the tumor region (tumor image) extracted by the second tumor region extraction unit 212 to the second classifier 22c, and calculates the classification probability for the tumor or normal output from the second classifier 22c. get. The second determination unit 215 stores the determination result in, for example, the storage unit 22.

  The identification unit 221 includes a classification probability for the tumor or normal acquired by the first determination unit 205 based on the Gabor image of the patient, and a classification probability for the tumor or normal acquired by the second determination unit 215 based on the iris image of the patient. Is used to determine whether the state of the lesion in this patient is a tumor or a normal state. Specifically, the identification unit (calculation unit) 221 integrates the classification probability for the tumor or normal acquired by the first determination unit 205 and the classification probability for the tumor or normal acquired by the second determination unit 215, The classification probability (discrimination probability) for the tumor or normal in this patient is calculated. For example, the specifying unit 221 calculates the average value of the classification probability obtained based on the Gabor image and the classification probability obtained based on the iris image for each of the tumor or normal to be determined, and is integrated. Probability. The identification unit 221 identifies the determination result having the high classification probability in the integrated classification probability for the tumor or the normal as the lesion state of this patient.

  The output unit 222 outputs the specified result by the specifying unit 221. For example, the output unit 222 generates display information for displaying the specified result by the specifying unit 221 and displays the specified result screen on the display unit 23 based on the generated display information. FIG. 9 shows an example of the specification result screen. The screen shown in FIG. 9 displays the classification probability (discrimination probability) for the tumor or normal in this patient finally calculated by the specification unit 221. The determination result identified by 221 is underlined and displayed prominently. The screen shown in FIG. 9 displays a Gabor image and an iris image used for the diagnosis. Therefore, the output unit 222 generates the identification result screen that displays the Gabor image and the iris image and the classification probability for the tumor or the normal calculated by the identification unit 221.

  The screen displayed on the display unit 23 is not limited to the example illustrated in FIG. 9, and may be, for example, a screen that displays only the classification probability with respect to the tumor or normal finally calculated by the specifying unit 221. In addition, for example, an image (Gabor image or iris image) with a higher classification probability obtained by the first determination unit 205 and the second determination unit 215 for the determination result specified by the specification unit 221 may be displayed. Further, an image (Gabor image or iris image) having a larger size of the tumor region may be displayed. Such an image is likely to be an image in which it is easy to determine that the image is the determination result specified by the specifying unit 221. When displaying the Gabor image and the iris image, the tumor region in the Gabor image and the iris image may be displayed prominently. The output unit 222 may output the result of the identification performed by the identifying unit 221 on the display unit 23 by voice, or may transmit the result to a predetermined terminal device.

  Next, a learning process performed by the learning device 10 in the diagnosis support system will be described with reference to a flowchart. FIG. 10 is a flowchart illustrating a procedure of a learning process performed by the learning device 10. The following processing is executed by the control unit 11 according to a control program including a learning program 12a stored in the storage unit 12 of the learning device 10. The learning device 10 causes the first classifier 12b to learn using the first teacher data of the Gabor image stored in the first teacher data DB 12c, and the second teacher of the iris image stored in the second teacher data DB 12e. The second classifier 12d is trained using the data. The learning process of the first classifier 12b and the learning process of the second classifier 12d are the same process, and FIG. 10 shows only the learning process of the first classifier 12b.

  The control unit 11 of the learning device 10 acquires one piece of first teacher data of a Gabor image from the first teacher data DB 12c (S11). The first teacher data includes information indicating a tumor or normal, and a Gabor image as shown in FIG. 4A. The control unit 11 causes the first classifier 12b to learn using the information indicating the tumor or normal included in the first teacher data and the Gabor image (S12). Here, the control unit 11 inputs the Gabor image to a node in the input layer of the first classifier 12b, and in the output layer, when the Gabor image is a tumor image (or a normal image), the node 0 (or the node 1) ) Is approached to 1.0, and the first classifier 12b is trained so that the output value of the node 1 (or the node 0) approaches 0.

  The control unit 11 determines whether the processing based on all the first teacher data stored in the first teacher data DB 12c has been completed (S13). If it is determined that the processing based on all the first teacher data has not been completed (S13: NO), the control unit 11 returns to the processing of step S11 and acquires one unprocessed first teacher data (S11). ). The control unit 11 performs the process of step S12 based on the acquired first teacher data. When it is determined that the processing based on all the first teacher data has been completed (S13: YES), the control unit 11 ends the learning processing of the first classifier 12b.

The control unit 11 of the learning device 10 causes the second classifier 12d to learn using the second teacher data of the iris image stored in the second teacher data DB 12e by a process similar to the process illustrated in FIG. In the learning process of the second classifier 12d, the control unit 11 acquires the second teacher data of the iris image from the second teacher data DB 12e in step S11 in FIG. In step S12, the control unit 11 causes the second classifier 12d to learn using the information indicating the tumor or normal included in the second teacher data and the iris image.
By the above-described processing, the first classifier 12b that determines whether the state of the patient's lesion is a tumor based on the Gabor image and whether the state of the patient's lesion is a tumor based on the iris image is determined. The second classifier 12d to be determined can be learned. Therefore, the learned first classifier 12b and second classifier 12d are obtained.

  Next, the identification processing by the identification device 20 in the diagnosis support system will be described based on a flowchart. FIG. 11 is a flowchart illustrating the procedure of the specifying process performed by the specifying device 20. The following processing is executed by the control unit 21 according to a control program including the specific program 22a stored in the storage unit 22 of the specific device 20. The storage unit 22 of the identification device 20 stores the first classifier 22b and the second classifier 22c that have been learned by the learning device 10. The storage unit 22 stores a Gabor image and an iris image of the breast of the patient to be diagnosed. Note that a mammography image may be stored in the storage unit 22 instead of the Gabor image and the iris image. In the processing illustrated in FIG. 11, the control unit 21 of the identification device 20 executes the processing of steps S21 to S24 and the processing of steps S31 to S34 in parallel, but executes one processing and then executes the other processing. You may.

  The control unit 21 of the identification device 20 acquires a Gabor image of the patient to be diagnosed (S21). When a mammography image is stored in the storage unit 22, the control unit 21 performs a filtering process using a Gabor filter on the mammography image to obtain a Gabor image. The control unit 21 extracts a tumor region from the acquired Gabor image (S22), and the first classifier 22b determines whether the state of the lesion in the tumor region is normal or normal for the extracted tumor region. The determination is performed (S23), and the determination result is stored in the storage unit 22 (S24). Specifically, the first classifier 22b outputs classification probabilities for each of the determination result that the lesion state in the Gabor image is a tumor and the determination result that the lesion state is normal, and the control unit 21 The classification probability for the tumor or normal output from the first classifier 22b is stored in the storage unit 22.

  On the other hand, the control unit 21 acquires an iris image of the patient (S31), extracts a tumor region from the acquired iris image (S32), and determines whether the extracted tumor region is a tumor or normal by the second classifier 22c. The determination is performed (S33), and the determination result is stored in the storage unit 22 (S34). Here, the second classifier 22c outputs classification probabilities for each of the determination result that the lesion state in the iris image is a tumor and the determination result that the lesion state is normal. The classification probability for a tumor or normal output from the classifier 22c is stored in the storage unit 22.

  The control unit 21 integrates the classification probability for the tumor or normal acquired based on the Gabor image in step S23 and the classification probability for the tumor or normal acquired based on the iris image in step S33, and calculates the mass or A classification probability for normality is calculated (S41). The control unit 21 generates display information for displaying the calculated classification probability of the patient as a tumor or normal as a determination result (S42). For example, the control unit 21 generates display information for displaying a classification probability of the patient regarding a mass or normal. And the control part 21 displays the determination result by the specific device 20 on the display part 23 based on the generated display information (S43). Thereby, for example, a screen as shown in FIG. 9 is displayed on the display unit 23, and the result of the determination by the identifying device 20 is notified.

  In the present embodiment, based on the Gabor image and the iris image of the patient, it is possible to calculate the classification probability indicating that the imaging target lesion is a tumor mass and the classification probability indicating that the imaging target lesion is normal. Therefore, it is possible to notify the possibility that the state of the lesion is a mass or normal, so that the doctor can consider when diagnosing the state of the lesion based on the mammography image, and can assist the diagnosis by the doctor. Although it is difficult for a doctor to visually and accurately read information of a medical image, the identification device 20 makes a determination based on the medical image, so that the medical image can be used effectively.

  In the present embodiment, the discrimination result based on the Gabor image using the first classifier 22b and the discrimination result based on the iris image using the second classifier 22c are integrated to determine whether the lesion of the patient is a tumor or not. Is determined. Thereby, the discrimination accuracy is improved. The applicant has conducted an evaluation experiment for evaluating the accuracy of the discrimination processing by the identification device 20. As a comparison target, when it is determined whether the state of the lesion is a tumor based on the Gabor image using the first classifier 22b, for example, it can be determined that the test data of 48 Gabor images is correct. The ratio (classification accuracy) was 64.6%. Further, when it is determined whether the state of the lesion is a tumor based on the iris image using the second classifier 22c, for example, the percentage of the test data of 48 iris images determined to be correct (Classification accuracy) was 62.5%. Further, the applicant inputs two images of a Gabor image and an iris image, and performs a discriminating process using a classifier for discriminating whether or not the state of a lesion to be photographed is a tumor based on the two images. . In this case, for example, the ratio (classification accuracy) at which the test data of 48 Gabor images and iris images could be determined to be correct for each was 58.3%. On the other hand, the classification accuracy in the discrimination processing by the specifying device 20 of the present embodiment was 66.7%. As described above, the first classification is compared with the classification accuracy by the classification process using only one of the first classifier 22b and the second classifier 22c and the classification accuracy by the classification process of inputting and discriminating two images. The classification accuracy obtained by the classification processing in which the classification results obtained by the classifier 22b and the second classifier 22c are integrated is high. Therefore, classification accuracy is improved by performing a classification process using a classifier for each medical image on different types of medical images and integrating the respective classification results to specify a final classification result. Can be done.

  In the present embodiment, a Gabor image and an iris image obtained by filtering from a mammography image are used as images used for diagnosis, but the present invention is not limited to these. The objects to be determined by the classifiers 22b and 22c are not limited to the two types: tumor mass and normal. For example, for various lesions such as breast cancer, fibroadenoma of the breast, mastosis, mastitis, and phyllodes tumor, the type of the lesion can be determined by learning the classifiers 22b and 22c using the respective teacher data.

  In the present embodiment, instead of using the Gabor image and the iris image of the teacher data as they are to train the classifiers 12b and 12d, the learning device 10 uses a predetermined number of pixels from the Gabor image and the iris image of the teacher data. May be extracted, and the classifiers 12b and 12d may be learned using the patch images. In this case, the identifying device 20 extracts a plurality of patch images from the Gabor image and the iris image instead of performing the discriminating process by the classifiers 22b and 22c using the Gabor image and the iris image of the patient to be diagnosed as they are, and The classification process by the classifiers 22b and 22c may be performed by. When a patch image extracted from an image is used for teacher data, a plurality of patch images (teacher data) can be generated from one image, so that the number of teacher data can be increased, and as a result, learning accuracy can be improved. it can. Further, in the case where the discrimination processing is performed using the patch image extracted from the image, the number of data to be discriminated can be increased, so that the discrimination accuracy can be improved.

  In the present embodiment, it may be configured to determine the state of the lesion of the patient using three or more types of images. In this case, the learning device 10 trains a classifier (learning model) using teacher data for each type of image, and the identifying device 20 performs a discriminating process using the classifier for each type of image, The determination result for each type is integrated to determine the state of the lesion of the patient. In the case of such a configuration, further improvement of the discrimination accuracy can be expected.

(Embodiment 2)
Diagnosis support for determining, based on medical images (PET image and CT image) obtained by imaging the lungs of a patient suspected of having lung cancer by PET-CT, whether or not the state of a predetermined part of the lung is lung cancer. The system will be described. In this embodiment, in order to acquire different types of images, images (PET images and CT images) based on different imaging methods are used. Since each device of the diagnosis support system of the present embodiment has the same configuration as each device of the first embodiment, the description of the configuration will be omitted. In the present embodiment, it is determined (determined) whether or not the state of the imaging target is lung cancer using a PET image and a CT image obtained by imaging with PET-CT. The medical images used for the determination are not limited to these, and other types of medical images may be used. Further, the plurality of types of medical images used for the determination may be images captured of the same imaging target (patient's lung), and may be images simultaneously captured by PET-CT. The images may be shot separately by each device.

  In the present embodiment, the first classifiers 12b and 22b are classifiers that determine whether the state of the lesion to be imaged is lung cancer or normal based on the PET image of the patient, and the second classifier 12d. , 22c are classifiers that determine whether the state of the lesion to be imaged is lung cancer or normal based on the CT image of the patient. Therefore, the first teacher data DB12c includes first teacher data including a tumor image including a tumor region (lung cancer region) extracted from a PET image of a lung cancer patient and a normal image including a normal region having no tumor (lung cancer). The second teacher data DB 12e stores a large number of tumor images including a tumor region (lung cancer region) extracted from a CT image of a lung cancer patient and a normal image including a normal region without a tumor (lung cancer). 2. Many teacher data are accumulated. Then, the learning device 10 of the present embodiment makes the first classifier 12b learn based on the first teacher data based on the PET image, and learns the second classifier 12d based on the second teacher data based on the CT image.

  In the learning device 10 of the present embodiment, the control unit 11 realizes each function illustrated in FIG. 5 by executing the learning program 12a. Note that, in the present embodiment, the first teacher data acquisition unit 101 acquires the first teacher data based on the PET image from the first teacher data DB 12c. The first learning unit 104 makes the first classifier 12b learn based on the information indicating the lung cancer or normal included in the first teacher data and the PET image. Further, the second teacher data acquiring unit 111 acquires second teacher data based on a CT image from the second teacher data DB 12e. Further, the second learning unit 114 makes the second classifier 12d learn based on the information indicating the lung cancer or normal included in the second teacher data and the CT image.

  In the identification device 20 of the present embodiment, the control unit 21 realizes each function shown in FIG. 8 by executing the identification program 22a. In the present embodiment, the first image acquisition unit 201 acquires a PET image of a patient to be diagnosed, and the first tumor region extraction unit 202 extracts a tumor region from the PET image. The first determination unit 205 acquires the classification probability for lung cancer or normal output from the first classifier 22b for the tumor region extracted from the PET image. Further, the second image acquisition unit 211 acquires a CT image of a patient to be diagnosed, and the second tumor region extraction unit 212 extracts a tumor region from the CT image. The second determination unit 215 acquires the classification probability for lung cancer or normal output from the second classifier 22c for the tumor region extracted from the CT image. The other functions are the same as those of the first embodiment, and a description thereof will be omitted.

  In the present embodiment, the learning device 10 performs the same processing as the processing illustrated in FIG. In the present embodiment, at step S11 in FIG. 10, the control unit 11 acquires the first teacher data based on the PET image from the first teacher data DB 12c, and at step S12, the PET image included in the first teacher data. The first classifier 12b is learned using the information indicating the lung cancer or normal. Other processes are the same as those in the first embodiment, and a description thereof will not be repeated. In addition, the control unit 11 of the learning device 10 of the present embodiment executes the second classifier using the second teacher data of the CT image stored in the second teacher data DB 12e by the same process as the process illustrated in FIG. Train 12d. Thereby, the learned first classifier 12b and second classifier 12d are obtained.

  In the present embodiment, the identification device 20 performs the same processing as the processing illustrated in FIG. In the present embodiment, the control unit 21 acquires a PET image of a patient to be diagnosed in step S21 in FIG. 11, and extracts a tumor region from the acquired PET image in step S22. In step S23, the control unit 21 uses the first classifier 22b to determine whether the lesion state is lung cancer or normal for the tumor region extracted from the PET image. Further, the control unit 21 acquires a CT image of the patient to be diagnosed in step S31, and extracts a tumor region from the acquired CT image in step S32. In step S33, the control unit 21 uses the second classifier 22c to determine whether the state of the lesion is lung cancer or normal for the tumor region extracted from the CT image. Other processes are the same as those in the first embodiment, and a description thereof will not be repeated.

  In the present embodiment, the same effects as those of the above-described first embodiment can be obtained. Therefore, since it is possible to notify the possibility that the condition of the lesion of the patient is lung cancer and the possibility that the condition is not lung cancer (normal), the doctor can consider when diagnosing the condition of the lesion based on the PET image and the CT image, We can assist doctors in diagnosis. Also, the determination result based on the PET image using the first classifier 22b and the determination result based on the CT image using the second classifier 22c are integrated to determine whether the patient's lesion is lung cancer. Therefore, the discrimination accuracy is improved.

  Also in the present embodiment, medical images used for diagnosis are not limited to PET images and CT images. Further, the discrimination targets by the classifiers 22b and 22c are not limited to the two types, lung cancer and normal. For example, in addition to lung cancer, various types of lesions such as metastatic lung tumors, hamartomas, and sclerosing hemangiomas can be used to discriminate the types of lesions by learning the classifiers 22b and 22c using the respective teacher data. Becomes Also, in the present embodiment, the modification described in the first embodiment can be applied. That is, the learning device 10 may extract a plurality of patch images from the PET image and the CT image of the teacher data, and train the classifiers 12b and 12d using the patch images. A plurality of patch images may be extracted from the PET image and the CT image, and the classifiers 22b and 22c may perform the discriminating process on the patch images. Further, the diagnosis support system according to the present embodiment may be configured to determine the state of a lesion of a patient using three or more types of images.

(Embodiment 3)
A diagnosis support system that determines the state (type) of a brain tumor to be imaged based on a medical image (MRI image) obtained by imaging the brain of a brain tumor patient with MRI will be described. Since each device of the diagnosis support system of the present embodiment has the same configuration as each device of the first embodiment, the description of the configuration will be omitted. In the present embodiment, the type of brain tumor to be imaged is determined using an MRI image obtained by imaging with MRI.

  In MRI, different types of MRI images can be obtained by changing parameters at the time of imaging. In the diagnosis support system of the present embodiment, a T2-weighted image obtained by imaging using a method in which water, fat, and a tumor appear white without using a contrast agent, and a fat and a contrast agent appear white using a contrast agent, Then, the type of the brain tumor to be imaged is determined using the T1CE image obtained by imaging the tumor in a black image. The medical image used for determining the type of the brain tumor is not limited to these, and another type of medical image may be used. MRI is an apparatus that acquires a cross-sectional image (slice image) cut by a plane in the front-rear and left-right directions while scanning a patient's head (brain to be imaged) in a vertical direction. For example, about 20 images are shot for a T2-weighted image, and about 200 images are shot for a T1CE image. The number of images taken by MRI is not limited to these numbers.

  FIG. 12 is a schematic diagram illustrating an example of a T2-weighted image, and FIG. 13 is a schematic diagram illustrating an example of a T1CE image. The size of the T2-weighted image and the T1CE image can be, for example, 512 pixels × 512 pixels, but is not limited thereto. In the T2-weighted image, the lesion (tumor) region is easily recognized, and in the T1CE image, the outline of the lesion (tumor) is easily recognized. FIG. 12 shows an example of a T2-weighted image of five types of brain tumors of glioblastoma, malignant lymphoma, meningioma, metastatic tumor, and schwannoma. FIG. 13 shows glioblastoma, malignant lymphoma, and marrow tumor. 5 shows an example of T1CE images of five types of brain tumors: a membranous tumor, a metastatic tumor, and a schwannoma. In the diagnosis support system of the present embodiment, it is determined which of the five types of brain tumor to be determined is glioblastoma, malignant lymphoma, meningioma, metastatic tumor, and schwannoma. Therefore, the first classifiers 12b and 22b are classifiers that determine the type of the brain tumor with respect to the T2 weighted image based on the T2 weighted image of the patient, and the second classifiers 12d and 22c generate the T1CE images of the patient. Based on this T1CE image, it is a classifier that determines the type of brain tumor.

  Accordingly, in the first teacher data DB 12c of the present embodiment, a large number of T2-weighted images as shown in FIG. 12 are associated with each of glioblastoma, malignant lymphoma, meningioma, metastatic tumor, and schwannoma. Has been accumulated. Further, a large number of T1CE images as shown in FIG. 13 are stored in the second teacher data DB 12e in association with each type of brain tumor. In the first teacher data DB 12c, one T2-weighted image (slice image) and the type of brain tumor (glioblastoma, malignant lymphoma, meningioma, metastatic tumor, or neural sheath) associated with this T2-weighted image The set with the information indicating the tumor is referred to as first teacher data. In the second teacher data DB 12e, a set of one T1CE image (slice image) and information indicating the type of brain tumor associated with the T1CE image is referred to as second teacher data. The learning device 10 of the present embodiment makes the first classifier 12b learn based on the first teacher data based on the T2-weighted image as illustrated in FIG. 12, and based on the second teacher data based on the T1CE image as illustrated in FIG. To make the second classifier 12d learn.

  FIG. 14 is a block diagram illustrating functions realized by the control unit 11 of the learning device 10 according to the third embodiment. FIGS. 15 and 16 are diagrams illustrating a learning process performed by the learning device 10 according to the third embodiment. In the learning device 10 of the present embodiment, when the control unit 11 executes the learning program 12a, the first teacher data acquisition unit 101, the first tumor region extraction unit 102, the first patch image extraction unit 103, the first learning unit 104 , The second teacher data acquisition unit 111, the second tumor region extraction unit 112, the second patch image extraction unit 113, and the second learning unit 114. Some of these functions may be realized by a dedicated hardware circuit. The first teacher data acquisition unit 101, the first learning unit 104, the second teacher data acquisition unit 111, and the second learning unit 114 perform the same processing as each function of the first embodiment illustrated in FIG.

  In the present embodiment, the first teacher data acquisition unit 101 acquires the first teacher data using a T2-weighted image from the first teacher data DB 12c. The first teacher data includes information indicating the type of brain tumor and a T2-weighted image (slice image). The first tumor region extraction unit 102 extracts a tumor region (brain tumor region) from the T2-weighted image acquired by the first teacher data acquisition unit 101. The T2-weighted image stored in the first teacher data DB 12c is an image of the entire brain, and the first tumor region extracting unit 102 extracts a brain tumor region based on, for example, a designation by a user's input operation via the input unit 14. . Further, the first tumor region extracting unit 102 may recognize and extract a brain tumor region in the T2-weighted image acquired by the first teacher data acquiring unit 101, for example, using a recognition model learned by deep learning. . In the example shown in FIG. 15A, a region R surrounded by a broken closed curve is extracted as a brain tumor region.

  The first patch image extracting unit 103 extracts a plurality of rectangular patch images (pixel blocks) including a predetermined number of pixels from the tumor region extracted by the first tumor region extracting unit 102. The size of the patch image can be, for example, 128 pixels × 128 pixels, but is not limited thereto. The first patch image extraction unit 103 converts a patch image whose center position is at a predetermined interval in the tumor region extracted by the first tumor region extraction unit 102 into one T2-weighted image (tumor region) For example, about 100 to 200 sheets are extracted. In the example shown in FIG. 15B, the area P surrounded by each rectangle is extracted as a patch image.

  The first learning unit 104 generates a first classifier 12b based on information indicating the type of brain tumor included in the first teacher data and a plurality of patch images (patch images of the T2-weighted image) extracted from the T2-weighted image. To learn. Here, the first classifier 12b of the present embodiment will be described. FIG. 16 shows a first classifier 12b of the present embodiment, and the first classifier 12b shown in FIG. 16 is configured by a CNN similarly to the first classifier 12b of the first embodiment. As shown in FIG. 16, in the first classifier 12b of the present embodiment, the number of nodes in the input layer is 16,384 (the number of channels is 1), and the number of nodes in the output layer is 5. The above-described patch image is provided as input data to the first classifier 12b of the present embodiment. Each node in the output layer outputs a classification probability for each of the five types of brain tumors. For example, node 0 outputs the probability that the input patch image is a glioblastoma image, node 1 outputs the probability of being an image of malignant lymphoma, and node 2 outputs the probability of being an image of meningioma. The node 3 outputs the probability of being an image of a metastatic tumor, and the node 4 outputs the probability of being an image of a schwannoma. The output value of each node in the output layer is, for example, a value from 0 to 1.0, and the total of the probabilities output from the five nodes is 1.0 (100%). The second classifier 12d of the present embodiment also has the same configuration as the first classifier 12b.

  The first learning unit 104 of the present embodiment inputs the patch image extracted from the T2-weighted image to the node of the input layer of the first classifier 12b, and outputs the patch image (T2-weighted image) of the brain tumor in the output layer. The first classifier 12b is made to learn so that the output value of the node corresponding to approaches 1.0 and the output values of the other nodes approach 0. That is, when the patch image input to the node in the input layer is an image of glioblastoma, the first learning unit 104 approaches the output value of the node 0 to 1.0 and sets the output values of the nodes 1 to 4 to 0. The first classifier 12b is made to learn so as to approach.

  In the present embodiment, each of the second teacher data acquisition unit 111, the second tumor region extraction unit 112, the second patch image extraction unit 113, and the second learning unit 114 includes the first teacher data acquisition unit 101, the first tumor region Although the image to be processed is different from the extraction unit 102, the first patch image extraction unit 103, and the first learning unit 104, they perform the same processing. The second classifier 12d has the same configuration as the first classifier 12b. Therefore, a detailed description thereof will be omitted. The second teacher data acquisition unit 111 acquires the second teacher data based on the T1CE image from the second teacher data DB 12e, and the second tumor region extraction unit 112 extracts a tumor region (brain tumor region) from the T1CE image. In addition, the second patch image extraction unit 113 extracts a plurality of patch images from the brain tumor region extracted from the T1CE image. The second learning unit 114 makes the second classifier 12d learn based on the information indicating the type of the brain tumor included in the second teacher data and the plurality of patch images extracted from the T1CE image.

  FIG. 17 is a block diagram illustrating functions realized by the control unit 21 of the specifying device 20 according to the third embodiment. FIGS. 18 to 20 are diagrams illustrating a specifying process performed by the specifying device 20 according to the third embodiment. When executing the specific program 22a stored in the storage unit 22, the control unit 21 of the specific device 20 according to the present embodiment, the first image acquisition unit 201, the first tumor region extraction unit 202, the first patch image extraction unit 203, a first patch determination unit 204, a first determination unit 205, a second image acquisition unit 211, a second tumor region extraction unit 212, a second patch image extraction unit 213, a second patch determination unit 214, a second determination unit 215 , The functions of the specifying unit 221 and the output unit 222 are realized. In the present embodiment, some of these functions may be realized by a dedicated hardware circuit. The first image acquiring unit 201, the first tumor region extracting unit 202, the second image acquiring unit 211, the second tumor region extracting unit 212, the specifying unit 221, and the output unit 222 are the respective functions of the first embodiment shown in FIG. The same processing is performed.

  In the present embodiment, the first image acquisition unit 201 acquires a T2-weighted image (slice image) of a patient to be diagnosed. The T2-weighted image is a cross-sectional image taken while scanning the patient's brain in the vertical direction, and a plurality of T2-weighted images are processed as one set for one patient. Therefore, the first image acquisition unit 201 acquires a plurality of T2-weighted images obtained from one patient. The first tumor region extraction unit 202 extracts a brain tumor region from each of the T2-weighted images acquired by the first image acquisition unit 201.

  The first patch image extraction unit 203 performs the same processing as the first patch image extraction unit 103 realized by the control unit 11 of the learning device 10. That is, the first patch image extraction unit 203 extracts a plurality of patch images from each of the tumor regions extracted by the first tumor region extraction unit 202. The first patch discriminating unit 204 discriminates the type of brain tumor for each patch image using the first classifier 22b based on the patch image (the patch image of the T2-weighted image) extracted by the first patch image extracting unit 203. I do. Specifically, the first patch determination unit 204 inputs each of the patch images to the first classifier 22b, and acquires an output value from the first classifier 22b. The output value of the first classifier 22b is the classification probability for each of the five types of brain tumors (glioblastoma, malignant lymphoma, meningioma, metastatic tumor, schwannoma). The first patch determination unit 204 performs a determination process on each of the patch images extracted by the first patch image extraction unit 203 from each of the plurality of T2-weighted images of one patient. The first patch determination unit 204 stores a determination result for each patch image in, for example, the storage unit 22. FIG. 18 is a schematic diagram illustrating a determination result by the first patch determination unit 204. The table shown in FIG. 18 stores a patient ID, a T2-weighted image ID, a patch ID, and a classification probability in association with each other. The patient ID is identification information assigned to each patient, the T2-weighted image ID is identification information assigned to the T2-weighted image of the patient, and the patch ID is assigned to each of the patch images extracted from the T2-weighted image. This is assigned identification information. The classification probability is a classification probability for each type of brain tumor output by the first classifier 22b based on each patch image.

  The first determination unit 205 calculates a classification probability for each type of brain tumor for each T2-weighted image (slice image) based on the determination result by the first patch determination unit 204. That is, based on the classification probabilities for each type of brain tumor output from the first classifier 22b for each patch image by the first classifier 22b, the first determination unit 205 determines the brain tumor in the T2-weighted image (slice image) from which this patch image has been extracted. The classification probability for each type of is calculated. For example, the first determination unit 205 calculates the average value of the classification probabilities for each type of brain tumor output for each of the patch images extracted from one T2-weighted image, and calculates the brain tumor in the T2-weighted image. May be the classification probability for each type. The method of calculating the classification probability for each type of brain tumor in the T2-weighted image is not limited to this. For example, the average value of the classification probabilities for each type of brain tumor in some patch images may be used as the classification probabilities for each type of brain tumor in the T2-weighted image. The first determination unit 205 calculates a classification probability for each type of brain tumor for all T2-weighted images of one patient, and stores the classification probability in, for example, the storage unit 22.

  Then, the first determination unit 205 of the present embodiment determines the type of the brain tumor of this patient based on the classification probability for each type of the brain tumor calculated for each T2-weighted image (slice image). Specifically, the first determination unit 205 calculates the classification probability for each type of brain tumor in this patient based on the classification probability for each type of brain tumor in each T2-weighted image. Here, the first determination unit 205 performs, for example, weighting according to the size (area) of the brain tumor region in each T2-weighted image, and a weighted average value (weighted average) of classification probabilities for each type of brain tumor. Is calculated as the classification probability for each type of brain tumor in the patient.

  FIG. 19 is a schematic diagram for explaining weighting according to the size of the brain tumor region in the T2-weighted image. 19A to 19D show a T2-weighted image and a brain tumor region (tumor region) in each T2-weighted image. In the T2-weighted images shown in FIGS. 19A to 19D, it is assumed that the areas of the respective brain tumor regions are a1 to a4 and the maximum area is a2. In this case, the coefficients (weighting factors) according to the size of the brain tumor region in the T2-weighted images shown in FIGS. 19A to 19D can be a1 / a2, 1, a3 / a2, and a4 / a2, respectively. Then, for each type of brain tumor, the first determination unit 205 multiplies the classification probability in each T2-weighted image by a coefficient corresponding to the size of the brain tumor region, and adds the result. And the obtained quotient is taken as the classification probability for this patient. The method for calculating the classification probability of the patient from the classification probabilities in the T2-weighted image is not limited to this. For example, the classification probability of a patient may be calculated using the classification probabilities in some T2-weighted images. The first determination unit 205 calculates a classification probability in a patient for each of the brain tumors to be determined.

  The second image acquiring unit 211, the second tumor region extracting unit 212, the second patch image extracting unit 213, the second patch discriminating unit 214, and the second discriminating unit 215 each include a first image acquiring unit 201 and a first tumor region. The extraction unit 202, the first patch image extraction unit 203, the first patch discrimination unit 204, and the first discrimination unit 205 differ in the image to be processed, but perform the same processing. Therefore, the detailed description is omitted. Note that the T2-weighted image acquired by the first image acquisition unit 201 and the T1CE image acquired by the second image acquisition unit 211 are cross-sectional images of the same patient's brain (imaging target) at the same location. Such a T2-weighted image and a T1CE image may be captured using the same imaging device at the same imaging timing with different parameters, or may be captured using different imaging devices. Alternatively, it may be generated by performing different image processing after photographing with one imaging device.

  The second image acquisition unit 211 acquires a plurality of T1CE images (slice images) obtained from one patient. The second tumor region extraction unit 212 extracts a brain tumor region from each of the T1CE images acquired by the second image acquisition unit 211. The second patch image extraction unit 213 extracts a plurality of patch images from each of the brain tumor regions extracted by the second tumor region extraction unit 212.

  In the present embodiment, the second patch discriminating unit 214 inputs each of the patch images extracted from the T1CE image (brain tumor region) to the second classifier 22c, and outputs five types of brain tumors that are output from the second classifier 22c. Obtain the classification probabilities for each of the. The second patch determination unit 214 performs determination processing on each of the patch images extracted from each of the plurality of T1CE images of one patient, and stores the determination result in a table having the same configuration as the table illustrated in FIG. In the table in which the second patch determination unit 214 stores the determination result, the T1CE image ID is stored instead of the T2-weighted image ID in the table shown in FIG.

  The second determining unit 215 determines, for each type of brain tumor in the T1CE image (slice image) from which the patch image is extracted, based on the classification probability for each type of brain tumor output from the second classifier 22c for each patch image. Is calculated. Then, the second determination unit 215 calculates the classification probability for each type of brain tumor in this patient based on the classification probability for each type of brain tumor calculated for each T1CE image (slice image). Also here, the second determination unit 215 may calculate the classification probability for each type of brain tumor in the patient using a weight coefficient according to the size of the brain tumor region in the T1CE image. The second determination unit 215 calculates the classification probability for each type of brain tumor to be determined for one patient, and stores the classification probability in, for example, the storage unit 22.

  The identification unit 221 according to the present embodiment includes a classification probability for each type of brain tumor calculated by the first determination unit 205 based on the T2-weighted image of the patient, and a brain tumor calculated by the second determination unit 215 based on the T1CE image of the patient. The type of brain tumor in this patient is specified based on the classification probability for each type. Specifically, the identification unit 221 integrates the classification probability for each type of brain tumor calculated by the first determination unit 205 and the classification probability for each type of brain tumor calculated by the second determination unit 215, and , The classification probability (discrimination probability) for each type of brain tumor is calculated. For example, the specifying unit 221 calculates, for each type of brain tumor, an average value of the classification probability calculated based on the T2 weighted image and the average value of the classification probability calculated based on the T1CE image, and sets the average value as an integrated probability. . The identification unit 221 identifies the brain tumor having the highest classification probability among the classification probabilities for each type of integrated brain tumor as the brain tumor of this patient.

  The output unit 222 of the present embodiment generates display information for displaying the specified result by the specifying unit 221 and displays a specified result screen as shown in FIG. 20 on the display unit 23 based on the generated display information. . On the screen shown in FIG. 20, the classification probability (discrimination probability) for each type of brain tumor in this patient, which is finally calculated by the identification unit 221, is displayed, and the brain tumor identified by the identification unit 221 is underlined. It is displayed more prominently than other brain tumors. The screen illustrated in FIG. 20 includes, for example, a classification probability calculated by the first determination unit 205 for a brain tumor (a metastatic tumor in FIG. 20) identified by the identification unit 221 in a T2-weighted image to be diagnosed. Are displayed. Such a T2-weighted image is likely to be an image in which it is easy to determine that the brain tumor has been identified by the identifying unit 221. Similarly, on the screen shown in FIG. 20, among the T1CE images to be diagnosed, a T1CE image with the highest classification probability calculated by the second determination unit 215 is displayed for the brain tumor identified by the identification unit 221. ing. Therefore, the output unit 222 selects the T2-weighted image and the T1CE image as described above, and generates a specific result screen to be displayed together with the classification probabilities calculated for each type of brain tumor in this patient.

  The screen displayed on the display unit 23 is not limited to the example illustrated in FIG. 20, and may be, for example, a screen that displays only the classification probabilities for each type of brain tumor in this patient. In addition, for example, a T2 weighted image having the largest size of the brain tumor region may be displayed among the T2 weighted images having a high classification probability calculated by the first determination unit 205 for the brain tumor specified by the specifying unit 221. Similarly, the T1CE image having the largest size of the brain tumor region may be displayed among the T1CE-emphasized images having the high classification probability calculated by the second determination unit 215 for the brain tumor identified by the identification unit 221. In addition, for example, a patch image of a T2-weighted image whose classification probability calculated by the first patch determination unit 204 is equal to or more than a predetermined value may be displayed for the brain tumor identified by the identification unit 221. Similarly, for the brain tumor identified by the identifying unit 221, a patch image of a T1CE image whose classification probability calculated by the second patch determining unit 214 is equal to or greater than a predetermined value may be displayed. Furthermore, a brain tumor region may be highlighted on each of the displayed slice images (T2-weighted image and T1CE image).

  FIG. 21 is a flowchart illustrating a procedure of a learning process performed by the learning device 10 according to the third embodiment. The learning device 10 of the present embodiment makes the first classifier 12b learn using the first teacher data of the T2-weighted image stored in the first teacher data DB 12c, and the T1CE stored in the second teacher data DB 12e. The second classifier 12d is trained using the second teacher data of the image. The learning process of the first classifier 12b and the learning process of the second classifier 12d are the same process, and FIG. 21 shows only the learning process of the first classifier 12b.

  In the learning device 10 of the present embodiment, the control unit 11 acquires one piece of the first teacher data of the T2-weighted image from the first teacher data DB 12c (S91). The first teacher data includes information indicating the type of brain tumor and a T2-weighted image (slice image). The control unit 11 extracts a tumor region (brain tumor region) from the acquired T2-weighted image (S92), and extracts a patch image including a predetermined number of pixels from the extracted tumor region (S93). Then, the control unit 11 makes the first classifier 12b learn using one patch image and information indicating the type of the brain tumor included in the first teacher data (S94). Here, the control unit 11 inputs the patch image to the node of the input layer of the first classifier 12b, and in the output layer, the output value of the node corresponding to the type of the brain tumor of this patch image approaches 1.0, The first classifier 12b is trained so that the output values of the other nodes approach zero.

  The control unit 11 determines whether or not the processing has been completed for all the patch images extracted from the T2-weighted image in step S93 (S95). If it is determined that the processing has not been completed (S95: NO), the process proceeds to step S93. Returning to the processing of S94, the first classifier 12b is learned using the unprocessed patch image (S94). When it is determined that the processing has been completed for all patch images (95: YES), the control unit 11 determines whether the processing based on all the first teacher data stored in the first teacher data DB 12c has been completed. Is determined (S96). If it is determined that the processing based on all the first teacher data has not been completed (S96: NO), the control unit 11 returns to the processing of step S91 and acquires one unprocessed first teacher data (S91). ). The control unit 11 performs the processing of steps S92 to S95 based on the acquired first teacher data. When determining that the processing based on all the first teacher data has been completed (S96: YES), the control unit 11 ends the learning processing of the first classifier 12b.

  The control unit 11 of the learning device 10 of the present embodiment performs the same process as the process illustrated in FIG. 21 to generate the second classifier 12d using the second teacher data of the T1CE image stored in the second teacher data DB 12e. Let them learn. In the learning process of the second classifier 12d, the control unit 11 acquires the second teacher data of the T1CE image from the second teacher data DB 12e in step S91 in FIG. In step S94, the control unit 11 causes the second classifier 12d to learn using the patch image extracted from the T1CE image and information indicating the type of brain tumor included in the second teacher data. By the above-described processing, the first classifier 12b that determines the type of the brain tumor of the patient based on the T2-weighted image and the second classifier 12d that determines the type of the brain tumor of the patient based on the T1CE image can be learned. it can. Therefore, the learned first classifier 12b and second classifier 12d are obtained.

  FIG. 22 and FIG. 23 are flowcharts illustrating the procedure of the specifying process by the specifying device 20 of the third embodiment. The storage unit 22 of the identification device 20 stores the first classifier 22b and the second classifier 22c that have been learned by the learning device 10. The storage unit 22 stores a plurality of T2-weighted images and a plurality of T1CE images of the brain of the patient to be diagnosed. In the processing illustrated in FIG. 22, the control unit 21 of the identification device 20 executes the processing of steps S51 to S59 and the processing of steps S61 to S69 in parallel, but executes one processing and then executes the other processing. You may.

  In the present embodiment, the control unit 21 of the identification device 20 acquires one of the T2-weighted images of the patient to be diagnosed (S51). The control unit 21 extracts a brain tumor region from the acquired T2-weighted image (S52), and extracts a plurality of patch images from the extracted brain tumor region (S53). Then, the control unit 21 determines the type of the brain tumor for one patch image by the first classifier 22b (S54), and stores the determination result in the storage unit 22 (S55). The discrimination result is the classification probability for each type of brain tumor output from the first classifier 22b. The control unit 21 determines whether or not the processing has been completed for all the patch images extracted from the T2-weighted image in step S53 (S56). If it is determined that the processing has not been completed (S56: NO), the process proceeds to step S53. Returning to the processing of S54, the processing of steps S54 to S55 is performed on the unprocessed patch image.

  If it is determined that the processing has been completed for all patch images (S56: YES), the control unit 21 determines in step S51 based on the classification probability for each type of brain tumor for each patch image stored in the storage unit 22. The classification probability for each type of brain tumor with respect to the acquired T2-weighted image (slice image) is calculated (S57). For example, the control unit 21 calculates an average value of classification probabilities based on each patch image for each type of brain tumor, and sets the average value as the classification probability for the slice image (T2-weighted image). The control unit 21 stores in the storage unit 22 the classification probabilities for each type of brain tumor calculated for one T2-weighted image.

  The control unit 21 determines whether the processing has been completed for all T2-weighted images (S58). When it is determined that the processing has not been completed for all the T2-weighted images (S58: NO), the control unit 21 returns to the processing of step S51, and acquires one unprocessed T2-weighted image (S51). . The control unit 21 performs the processing of steps S52 to S57 on the acquired T2-weighted image. If it is determined that the processing has been completed for all T2-weighted images (S58: YES), the control unit 21 determines the type of brain tumor for this patient based on the classification probability for each type of brain tumor for each T2-weighted image. A classification probability is calculated for each (S59). Here, the control unit 21 calculates a weighted average value using a weight coefficient according to the size of the brain tumor region in each T2-weighted image from the classification probability based on each T2-weighted image for each type of brain tumor. And the classification probability for the patient.

  On the other hand, the control unit 21 acquires one of the T1CE images of the patient (S61), extracts a brain tumor region from the acquired T1CE image (S62), and extracts a plurality of patch images from the extracted brain tumor region (S61). S63). Then, the control unit 21 determines the type of the brain tumor for one patch image by the second classifier 22c (S64), and stores the determination result in the storage unit 22 (S65). The control unit 21 determines whether or not the processing has been completed for all the patch images (S66). If it is determined that the processing has not been completed (S66: NO), the control unit 21 returns to the processing of step S64 and returns to the unprocessed state The processing of steps S64 to S65 is performed on the patch image.

  If it is determined that the processing has been completed for all patch images (S66: YES), the control unit 21 determines in step S61 based on the classification probability of each type of brain tumor for each patch image stored in the storage unit 22. The classification probability for each type of brain tumor with respect to the acquired T1CE image (slice image) is calculated (S67). The control unit 21 determines whether the processing has been completed for all T1CE images (S68). When it is determined that the processing has not been completed for all the T1CE images (S68: NO), the control unit 21 returns to the processing of step S61 and acquires one unprocessed T1CE image (S61). The control unit 21 performs the processing of steps S62 to S67 on the acquired T1CE image. When it is determined that the processing has been completed for all T1CE images (S68: YES), the control unit 21 classifies the brain tumor for each patient based on the classification probability for each type of brain tumor for each T1CE image. The probability is calculated (S69).

  The control unit 21 integrates the classification probability for each type of brain tumor calculated based on the T2-weighted image in step S59 and the classification probability for each type of brain tumor calculated based on the T1CE image in step S69, and Then, the classification probability for each type of brain tumor is calculated (S71). The control unit 21 generates display information for displaying the calculated classification probability of each type of brain tumor for the patient as a determination result (S72). And the control part 21 displays the determination result by the specific device 20 on the display part 23 based on the generated display information (S73). Thereby, for example, a screen as shown in FIG. 20 is displayed on the display unit 23, and the result of the determination by the identifying device 20 is notified.

  In the present embodiment, the same effects as in the first embodiment can be obtained. That is, in this embodiment, based on the T2-weighted image and the T1CE image of the patient, it is possible to notify the possibility of the brain tumor of this patient for each of at least five types of brain tumors. Therefore, the doctor can consider when diagnosing what kind of brain tumor is based on the T2-weighted image and the T1CE image, and can support the diagnosis by the doctor. Although it is difficult for a doctor to visually check all medical images, the identification device 20 performs determination using all medical images, so that the medical images can be used effectively. In addition, a slice image (T2-weighted image or T1CE image) that makes it easy to determine that the brain tumor is likely to be a patient's brain tumor can be notified as the determination result by the specifying device 20. Therefore, when the diagnosis can be performed based on the notified slice image, the doctor does not need to check all the images, and the burden of the diagnosis on the doctor can be reduced.

  The identification device 20 of the present embodiment integrates the determination result based on the T2 weighted image using the first classifier 22b and the determination result based on the T1CE image using the second classifier 22c, and obtains the brain tumor of the patient. Determine the type. Thereby, the discrimination accuracy is improved. The applicant has conducted an evaluation experiment for evaluating the accuracy of the discrimination processing by the identification device 20. FIG. 24 is a chart showing the configuration of the test data, and FIG. 25 is a chart showing the discrimination accuracy of the identification device 20. In the evaluation experiment, for each of the brain tumors of glioblastoma, malignant lymphoma, meningioma, metastatic tumor, and schwannoma, the T2 weighted images and the T1CE images (slice images) of the number of slice images shown in FIG. Used as test data. Also, patch images of the number of patch images shown in FIG. 24 were extracted from each of the T2-weighted image and the T1CE image of the test data and used for the discrimination processing.

  The identification device 20 performs the determination process using the test data as shown in FIG. 24, and the result as shown in FIG. 25 is obtained. The sensitivity is the ratio of the correct brain tumor to the correct brain tumor based on the T2-weighted image and T1CE image (test data) of each patient (the highest classification probability is output for the correct brain tumor) Percentage). Based on the T2-weighted image and T1CE image of each patient, the specificity is the percentage of incorrect brain tumors determined to be incorrect brain tumors (low classification probabilities can be output for incorrect brain tumors). Percentage). The classification accuracy is, for each test data patient, the proportion of the subject that could be determined to be a correct brain tumor based on the T2-weighted image, the proportion that the subject could be determined to be the correct brain tumor based on the T1CE image, and the first classifier. The percentages at which it is possible to determine that the brain tumor is a correct brain tumor based on the classification probabilities obtained by integrating the determination results obtained by the second classifier 22b and the second classifier 22c are shown. In the results shown in FIG. 25, the first classifier 22b and the second classifier were compared by comparing the rate at which a correct brain tumor could be determined based on the T2-weighted image and the rate at which a correct brain tumor could be determined based on the T1CE image. A high value is obtained by integrating the discrimination results by 22c and finally discriminating a correct brain tumor. Therefore, discrimination processing using different medical image classifiers is performed on different medical images, and the final discrimination is performed by integrating the respective discrimination results, thereby improving the discrimination accuracy. .

  In the present embodiment, the classification probability for each type of brain tumor calculated based on each of the plurality of slice images (T2-weighted image / T1CE image) is weighted according to the size of the brain tumor region in each image. Then, classification probabilities for each type of brain tumor for the patient were calculated. It is considered that the larger the brain tumor region in the image, the higher the reliability of the classification probability for each type of brain tumor calculated from this slice image. Therefore, the reliability of the classification probability for each type of brain tumor in the calculated patient can be improved by using a larger weighting factor as the brain tumor region in the image is larger.

  Also in the present embodiment, the medical images used for diagnosis are not limited to the T2-weighted images and the T1CE images. The brain tumors to be determined by the classifiers 22b and 22c are not limited to five types: glioblastoma, malignant lymphoma, meningioma, metastatic tumor, and schwannoma. Brain tumors other than these can be discriminated by learning the classifiers 22b and 22c. Also, diseases other than brain tumors can be discriminated by learning the classifier using the respective teacher data. Also in the present embodiment, the learning device 10 may cause the classifiers 12b and 12d to learn using the T2 weighted image and the T1CE image of the teacher data as they are, and the identification device 20 determines the T2 weighted value of the patient to be diagnosed. The classification process by the classifiers 22b and 22c may be performed using the image and the T1CE image as they are. Further, the diagnosis support system of the present embodiment may be configured to determine the type of the brain tumor of the patient using three or more types of images.

(Embodiment 4)
A diagnosis support system that determines whether a breast is a tumor based on a medical image (mammography image) obtained by mammography of a breast of a patient suspected of having a breast mass will be described. Since each device of the diagnosis support system of the present embodiment has the same configuration as each device of the first embodiment, the description of the configuration will be omitted. In the present embodiment, two types of mammography images (hereinafter, referred to as a left breast image and a right breast image) obtained by imaging left and right breasts of one and the same patient, respectively, and a difference image between the two mammography images are provided. Using the image, it is determined whether the state of the imaging target is a tumor.

  FIG. 26 is a schematic diagram illustrating an example of a mammography image, and FIG. 27 is a schematic diagram illustrating an example of a difference image. FIG. 26 shows a left breast image and a right breast image of one patient. The left breast image and the right breast image are shown on the left and right sides of FIG. 26, respectively. In the present embodiment, when generating teacher data, in a breast image of a patient having a tumor in one of the left and right breasts, a tumor image T1 including a tumor region from a breast image having a tumor (left breast image or right breast image) Is extracted, and a corresponding image T2 corresponding to the tumor image T1 is extracted from a breast image without a tumor (right breast image or left breast image). The tumor image T1 and the corresponding image T2 are images of a region which is symmetrical with respect to the center axis of the patient's body in the vertical direction. In the present embodiment, a normal image N1 including a normal region in a breast image having a tumor is extracted, and a corresponding image N2 corresponding to the normal image N1 is extracted from a breast image having no tumor. The normal image N1 and the corresponding image N2 are also images of a symmetrical region in the patient's body. In the example shown in FIG. 26, since there is a tumor in the right breast, the tumor image T1 and the normal image N1 are extracted from the right breast image, and the corresponding images T2 and N2 are extracted from the left breast image. The size of the tumor image T1, the normal image N1, and the corresponding images T2 and N2 can be, for example, 128 pixels × 128 pixels, but is not limited thereto.

In the present embodiment, a difference image T3 between the extracted tumor image T1 and the corresponding image T2 is generated, and a difference image N3 between the extracted normal image N1 and the corresponding image N2 is generated. For example, the absolute value of each value obtained by subtracting each pixel value in the corresponding images T2 and N2 from each pixel value in the tumor image T1 and the normal image N1 is set as each pixel value of the difference images T3 and N3. FIG. 27A shows an example of a tumor image T1, a corresponding image T2, and a difference image T3 of the tumor image T1 and the corresponding image T2, and FIG. 27B shows a normal image N1, a corresponding image N2, and a difference image N3 of the normal image N1 and the corresponding image N2. Here is an example.
In the first teacher data DB 12c, many tumor images T1 (mammography images) as shown on the left side of FIG. 27A are accumulated as images of the tumor region, and as images of normal regions, as shown on the left side of FIG. 27B. Many normal images N1 (mammography images) are accumulated. In the first teacher data DB 12c, a set of one mammography image and information indicating a tumor or normal associated with the image is referred to as first teacher data. In the second teacher data DB 12e, a large number of difference images T3 as shown on the right side of FIG. 27A are accumulated as images of the tumor region, and a difference image N3 as shown on the right side of FIG. Many are accumulated. In the second teacher data DB 12e, a set of one difference image and information indicating a tumor or normal associated with the difference image is referred to as second teacher data. The learning device 10 makes the first classifier 12b learn based on the first teacher data based on the mammography image, and makes the second classifier 12d learn based on the second teacher data based on the difference image generated from the mammography image.

  In the diagnosis support system of the present embodiment, it is determined whether the lesion to be imaged is a tumor or a normal tumor (not a tumor) as a determination target state. Therefore, the first classifiers 12b and 22b are classifiers that determine whether the lesion to be imaged is a tumor or a normal lesion based on the mammographic image of the patient. The second classifiers 12d and 22c are classifiers that determine whether the lesion to be imaged is a tumor or a normal lesion based on the difference image of the patient.

  In the learning device 10 of the present embodiment, the control unit 11 realizes each function illustrated in FIG. 5 by executing the learning program 12a. In the present embodiment, the first teacher data obtaining unit 101 obtains the first teacher data based on the mammography image from the first teacher data DB 12c. The first classifier 12b is trained on the basis of the information indicating the above and the mammography image. Further, the second teacher data acquiring unit 111 acquires second teacher data based on the difference image from the second teacher data DB 12e, and the second learning unit 114 compares the second teacher data with the information indicating the tumor or normal contained in the second teacher data. The second classifier 12d is trained based on the image.

  In the identification device 20 of the present embodiment, the control unit 21 realizes each function shown in FIG. 8 by executing the identification program 22a. In the present embodiment, the first image acquisition unit 201 acquires a mammography image of a patient to be diagnosed, and the first tumor region extraction unit 202 extracts a tumor region from the mammography image. The first determination unit 205 acquires the classification probability for the tumor or normal output by the first classifier 22b for the tumor region extracted from the mammography image. Further, the second image acquisition unit 211 acquires a difference image of the patient to be diagnosed, and the second tumor region extraction unit 212 extracts a tumor region from the difference image. The second determination unit 215 acquires the classification probability for the tumor or normal output from the second classifier 22c for the tumor region extracted from the difference image. The other functions are the same as those of the first embodiment, and a description thereof will be omitted.

  In the present embodiment, the learning device 10 performs the same processing as the processing illustrated in FIG. In the present embodiment, at step S11 in FIG. 10, the control unit 11 acquires first teacher data based on a mammography image from the first teacher data DB 12c, and at step S12, calculates a tumor or The first classifier 12b is trained using the information indicating normality and the mammography image. Other processes are the same as those in the first embodiment, and a description thereof will not be repeated. In addition, the control unit 11 of the learning device 10 of the present embodiment executes the second classifier using the second teacher data of the difference image stored in the second teacher data DB 12e by the same process as the process illustrated in FIG. Train 12d. Thereby, the learned first classifier 12b and second classifier 12d are obtained.

  In the present embodiment, the identification device 20 performs the same processing as the processing illustrated in FIG. In the present embodiment, the control unit 21 acquires a mammography image of a patient to be diagnosed in step S21 in FIG. 11, and extracts a tumor region from the acquired mammography image in step S22. In step S23, the control unit 21 uses the first classifier 22b to determine whether the state of the lesion is a tumor or a normal lesion in the tumor region extracted from the mammography image. The control unit 21 acquires a difference image of the patient to be diagnosed in step S31, and extracts a tumor region from the acquired difference image in step S32. In step S33, the control unit 21 uses the second classifier 22c to determine whether the state of the lesion is a tumor or a normal lesion in the tumor region extracted from the difference image. Other processes are the same as those in the first embodiment, and a description thereof will not be repeated.

  In the present embodiment, the same effects as those of the above-described first embodiment can be obtained. Therefore, it is possible to notify the possibility that the state of the lesion of the patient is a breast cancer mass and the possibility that the state is normal, so that the doctor can consider when diagnosing the state of the lesion based on the mammography image and can assist the diagnosis by the doctor . Further, the determination result based on the mammography image using the first classifier 22b and the determination result based on the difference image using the second classifier 22c are integrated to determine whether the patient's lesion is a tumor. Therefore, the discrimination accuracy is improved.

(Embodiment 5)
A modification of the specifying device 20 according to the first embodiment will be described. Since each device of the diagnosis support system of the present embodiment has the same configuration as each device of the first embodiment, the description of the configuration will be omitted. FIG. 28 is a block diagram illustrating functions realized by the control unit 21 of the specifying device 20 according to the fifth embodiment. When the specific program 22a is executed, the control unit 21 of the specific device 20 according to the present embodiment realizes the functions of the first mass detection unit 206 and the second mass detection unit 216 in addition to the functions illustrated in FIG. . In FIG. 28, functions other than the first image acquisition unit 201, the first mass region extraction unit 202, the second image acquisition unit 211, the second mass region extraction unit 212, the first mass detection unit 206, and the second mass detection unit 216 Are not shown.

  In the identification device 20 of the present embodiment, the first tumor detection unit 206 detects the presence or absence of a tumor region suspected of being a tumor in the Gabor image acquired by the first image acquisition unit 201. The tumor region is a region of a tumor in which the first classifier 22b can determine whether or not the tumor is a tumor. The first tumor detection unit 206 detects a tumor region in the Gabor image acquired by the first image acquisition unit 201, for example, using a recognition model that has been learned by deep learning. Therefore, the first tumor region extraction unit 202 of the present embodiment extracts the tumor region detected by the first tumor detection unit 206 from the Gabor image acquired by the first image acquisition unit 201. Similarly, the second tumor detection unit 216 detects the presence or absence of a tumor region in the iris image acquired by the second image acquisition unit 211. Therefore, the second tumor region extraction unit 212 of the present embodiment extracts the tumor region detected by the second tumor detection unit 216 from the iris image acquired by the second image acquisition unit 211.

  FIG. 29 is a flowchart illustrating a part of the procedure of the specifying process by the specifying device 20 of the fifth embodiment. The processing shown in FIG. 29 is different from the processing of the first embodiment shown in FIG. 11 in that steps S81 and S82 are added between steps S21 and S22, and steps S83 and S83 are added between steps S31 and S32. This is an added step of S84. 29, illustration of steps other than steps S21, S22, S31, and S32 in FIG. 11 is omitted.

  In the identification device 20 of the present embodiment, the control unit 21 acquires a Gabor image of a patient to be diagnosed (S21), and executes a detection process for detecting an area of a breast mass on the acquired Gabor image (S81). ). The control unit 21 determines whether or not the tumor area has been detected by the detection processing (S82). If it is determined that the tumor area has been detected (S82: YES), the control unit 21 extracts the tumor area detected in step S81 from the acquired Gabor image. Then (S22), the processing after step S23 is performed. When it is determined that the tumor region cannot be detected in the Gabor image (S82: NO), the control unit 21 ends the subsequent processing on the Gabor image. Further, the control unit 21 acquires an iris image of the patient to be diagnosed (S31), and executes a detection process for detecting a region of a breast mass on the acquired iris image (S83). The control unit 21 determines whether or not the tumor area has been detected by the detection processing (S84). If it is determined that the tumor area has been detected (S84: YES), the control unit 21 extracts the tumor area detected in step S83 from the acquired iris image. Then (S32), the processing after step S33 is performed. When it is determined that the tumor region cannot be detected in the iris image (S84: NO), the control unit 21 ends the subsequent processing on the iris image.

  In the present embodiment, the same effects as those of the above-described first embodiment can be obtained. Further, in the present embodiment, it is determined whether or not a tumor region is present in the medical image (Gabor image and iris image) to be diagnosed, and if so, a determination process based on the tumor region (medical image) is performed. Therefore, for example, before the physician checks a huge medical image taken from a large number of patients by a medical examination, a brain dock, or the like, a patient having a possibility of having a certain mass is extracted by the determination processing by the identification device 20. Can be. Therefore, a doctor can preferentially check a medical image of a patient who has a high possibility of having some kind of mass, and can reduce the burden of diagnosis by the doctor and suppress omission of diagnosis. The configuration of this embodiment can be applied to the second to fourth embodiments, and the same effects can be obtained even when applied to the second to fourth embodiments.

  In the above-described first to fifth embodiments, the system for specifying the state of the lesion of the patient based on the medical image obtained by imaging the patient using the imaging device has been described. However, the present disclosure can be applied to other systems. For example, the present invention can be applied to a system that detects (specifies) the state of the outer wall of a building. In this case, a plurality of types of images are acquired by photographing the outer wall of the building, the state of the photographing target is determined for each image using a learning model corresponding to each image, the determination results are integrated, and the outer wall of the outer wall is integrated. By specifying the state, the state of the outer wall can be accurately detected.

  The embodiment disclosed this time is an example in all points and should be considered as not being restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

10 Learning device (information processing device)
Reference Signs List 20 identification device 21 control unit 22 storage unit 101 first teacher data acquisition unit (teacher data acquisition unit)
103 First patch image extraction unit (extraction unit)
104 1st learning part (learning part)
111 Second teacher data acquisition unit (teacher data acquisition unit)
113 Second Patch Image Extraction Unit (Extraction Unit)
114 second learning unit (learning unit)
201 first image acquisition unit (image acquisition unit)
202 first tumor region extraction unit 203 first patch image extraction unit (extraction unit)
204 first patch determining unit (block determining unit)
205 First discriminator (discriminator)
211 Second image acquisition unit (image acquisition unit)
212 second tumor region extraction unit 213 second patch image extraction unit (extraction unit)
214 second patch determining unit (block determining unit)
215 Second discriminator (discriminator)
221 specifying part 222 output part 12b, 22b first classifier (learning model)
12d, 22c Second classifier (learning model)

Claims (12)

A specifying device for specifying a state of a shooting target,
A plurality of image acquisition units that respectively acquire a plurality of types of images imaged by different methods with respect to a shooting target,
Using the learning model provided based on the teacher data including the image and the state of the imaging target in the image provided for each type of the image, the state of the imaging target in the image acquired by each of the image acquisition units. A plurality of determination units for determining
A specifying unit configured to specify a state of the imaging target based on a determination result by each of the determination units. Each of the determination units, for the image acquired by the image acquisition unit, calculates a probability indicating the possibility of the state for each state,
Based on the probability that each of the determination units calculated for each state, further comprising a calculation unit that calculates the determination probability for each state,
The identification device according to claim 1, wherein the identification unit identifies the state of the imaging target based on the determination probability calculated for each state by the calculation unit. The identification device according to claim 2, wherein the calculation unit calculates an average value of the probabilities calculated for each of the states by each of the determination units as a determination probability for each of the states. From the image acquired by each of the image acquisition unit, a plurality of extraction units to extract a plurality of pixel blocks composed of a predetermined number of pixels,
A plurality of block determination units that determine the state of the imaging target in the pixel block extracted by each of the plurality of extraction units using the learning model provided for each type of the image,
4. The image processing apparatus according to claim 1, wherein each of the determination units determines a state of a shooting target in the image from which the pixel block is extracted, based on a determination result for each pixel block by each of the block determination units. Specific device according to. Each of the block determination unit, for the pixel block extracted by the extraction unit, calculates a probability indicating the possibility of the state for each state,
Each of the block determination units further includes a plurality of average calculation units that calculate an average value of the probability calculated for each state,
The identification device according to claim 4, wherein each of the determination units determines a state of a shooting target in the image from which the pixel block is extracted, based on the average value calculated by each of the average calculation units. An output unit that outputs an image having a high probability of indicating the possibility of being in the state specified by the specifying unit among the images determined by the determination units as being in the state specified by the specifying unit. Item 6. The specifying device according to any one of Items 1 to 5. The identification device according to any one of claims 1 to 6, wherein the plurality of types of images are images obtained by imaging the same portion of the imaging target by different methods. A program that causes a computer to execute a process of identifying a state of an imaging target,
On the computer,
Acquire multiple types of images that are imaged by different methods for the shooting target,
For each type of the image, using a learning model learned based on the image and teacher data including the state of the imaging target in the image, determine the state of the imaging target in the acquired image,
A program for executing a process of specifying a state of the imaging target based on a determination result determined for each type of the image. A specifying method by a specifying device that specifies a state of a shooting target,
The specific device,
Acquire multiple types of images that are imaged by different methods for the shooting target,
For each type of the image, using a learning model learned based on the image and teacher data including the state of the imaging target in the image, determine the state of the imaging target in the acquired image,
A specifying method for performing a process of specifying the state of the imaging target based on a determination result determined for each type of the image. A teacher data acquisition unit that acquires teacher data including a plurality of types of images imaged by a different method for an imaging target and the state of the imaging target in the image, for each type of the image,
An information processing apparatus comprising: a learning unit that learns, based on the teacher data obtained by the teacher data obtaining unit, a learning model that determines a state of a shooting target in the image for each type of the image. For each type of the image, further comprising an extraction unit that extracts a plurality of pixel blocks composed of a predetermined number of pixels from the image acquired by the teacher data acquisition unit,
The information processing device according to claim 10, wherein the learning unit learns the learning model based on the pixel block extracted by the extraction unit and a state of the imaging target in the image from which the pixel block is extracted. . A specifying device for specifying a state of a shooting target,
Learning is performed based on teacher data including a plurality of types of images formed by different methods for the imaging target and the state of the imaging target in the image, and the state of the imaging target in each type of the image is determined. A specifying device including a learning model for each type of the image.