JP2023129024A - Computer program, learning model generation method, image processing device, image processing system, and image processing method - Google Patents

Abstract

To provide a computer program, a learning model generation method, an image processing device, an image processing system, and an image processing method, which can accurately identify the portion of a muscle.SOLUTION: A computer program causes a computer to execute the processing of: acquiring a medical image including a muscle part; inputting the acquired medical image into a second learning model learned to output region information for the whole muscle part when the medical image is input, and outputting the region information for the whole muscle part; generating a whole muscle part image on the basis of the output region information for the whole muscle part; and inputting the acquired medical image and the generated whole muscle part image into a first learning model learned to output region information for each muscle portion of the muscle part when the medical image and the whole muscle part image are input, and outputting the region information for each muscle portion.SELECTED DRAWING: Figure 1

Description

The present invention relates to a computer program, a learning model generation method, an image processing device, an image processing system, and an image processing method.

The muscles in your thighs are larger than those in your forearms and lower legs, and increasing muscle mass can increase your basal metabolism. Furthermore, the thigh is an important part for maintaining the balance of the pelvis and supporting the knee joint, and understanding the condition of these muscles is essential for maintaining health.

Patent Document 1 discloses an ultrasonic diagnostic apparatus that performs texture analysis on an ultrasound echo image obtained by irradiating a muscle to be measured with ultrasound to derive muscle weakness in the muscle to be measured.

Japanese Patent Application Publication No. 2020-130596

Muscles are composed of multiple parts, each of which plays an important role, and it is desirable to identify each part with high accuracy.

The present invention was made in view of the above circumstances, and an object of the present invention is to provide a computer program, a learning model generation method, an image processing device, an image processing system, and an image processing method that can accurately identify muscle parts. do.

The present application includes a plurality of means for solving the above problems. To give one example, a computer program acquires a medical image including a muscle part, and when the medical image is input to a computer, the computer program acquires a medical image including a muscle part. The acquired medical image is input to the second learning model that has been trained to output region information, and region information of the entire muscle region is outputted. Based on the output region information of the entire muscle region, the second learning model is trained to output region information. When an image is generated and a medical image and a whole muscle image are input, the obtained medical image and the generated first learning model are trained to output region information for each muscle part of the muscle part. The image of the whole muscle part thus obtained is input, and a process is executed to output area information of each muscle part.

According to the present invention, a muscle region can be identified with high accuracy.

1 is a diagram illustrating an example of the configuration of an image processing apparatus according to an embodiment. It is a figure showing an example of composition of a control part. FIG. 3 is a diagram illustrating an example of part detection by an image processing device. It is a figure which shows an example of the generation method of a 1st learning model and a 2nd learning model. FIG. 3 is a diagram illustrating an example of femoral contour detection by the image processing device. It is a figure which shows an example of the generation method of the whole quadriceps image. It is a figure which shows the 1st example of the generation method of a fascia image. It is a figure which shows the 2nd example of the generation method of a fascia image. It is a figure which shows an example of the area|region correction screen of each quadriceps image (region information). It is a figure which shows an example of the area|region correction screen of a fascia image (region information). It is a figure which shows an example of the area|region information of the whole quadriceps muscle output by the 2nd learning model using a different algorithm. FIG. 3 is a diagram showing another example of the configuration of the image processing device according to the present embodiment. It is a figure which shows an example of the detection process of each quadriceps muscle by an image processing device. FIG. 3 is a diagram illustrating an example of learning model generation processing performed by the image processing device.

Embodiments of the present invention will be described below. FIG. 1 is a diagram showing an example of the configuration of an image processing apparatus 100 of this embodiment. The image processing device 100 includes a control section 10 that controls the entire device, a communication section 11, a memory 12, a display section 13, an operation section 14, and a storage section 19. The storage unit 19 stores a computer program 20, a first learning model 21, a second learning model 22, a third learning model 23, and the like. Note that the image processing device 100 may be configured by distributing each functional unit to a plurality of devices and configured by the plurality of devices to form an image processing system, or may be configured by one or more clouds (servers). . For example, each part of this system may be implemented in the processor of one computer, or each part may be distributed and implemented in multiple computers or a cloud (server). That is, this embodiment may be implemented using one or more processors.

FIG. 2 is a diagram showing an example of the configuration of the control section 10. As shown in FIG. The control unit 10 includes an acquisition unit 31, an output unit 32, a reduction/enlargement processing unit 15, a contour extraction unit 16, an image generation unit 24, a relearning processing unit 17, a reliability index calculation unit 18, and a matching degree calculation unit 33. Be prepared. The reduction/enlargement processing section 15 includes an interval determination section 151. Note that in this specification, the acquisition unit 31, the output unit 32, the reduction/enlargement processing unit 15, the contour extraction unit 16, the image generation unit 24, the relearning processing unit 17, the reliability index calculation unit 18, and the degree of matching calculation unit 33 are also collectively referred to as the control section 10.

The control unit 10 includes a required number of CPUs (Central Processing Units), MPUs (Micro-Processing Units), GPUs (Graphics Processing Units), and the like. In addition, by executing the computer program 20, the control unit 10 also controls the acquisition unit 31, the output unit 32, the reduction/enlargement processing unit 15, the contour extraction unit 16, the image generation unit 24, the relearning processing unit 17, the reliability index At least one function of the calculating section 18 and the matching degree calculating section 33 can be realized by software instead of hardware. The control unit 10 performs processing using the first learning model 21, the second learning model 22, and the third learning model 23. Further, the control unit 10 can execute processing determined by the computer program 20. That is, the processing by the control unit 10 is also the processing by the computer program 20.

The communication unit 11 includes, for example, a communication module and has a communication function with an external device via a communication network. The communication unit 11 can acquire medical images from a device such as an ultrasound diagnostic device, a computed tomography device, a magnetic resonance diagnostic device, an X-ray imaging device, an angiography X-ray diagnostic device, a PET examination device, or an electron microscope. The communication unit 11 stores the acquired medical image in the storage unit 19 and outputs it to the control unit 10. Note that in this specification, an ultrasound image will be described as an example of a medical image. Further, the explanation will be given by taking the quadriceps femoris muscle as an example of a muscle part, each quadriceps muscle as an example of a muscle part, and the femur as an example of a predetermined bone in the vicinity of the muscle part.

The memory 12 can be configured with a semiconductor memory such as an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), or a flash memory. By loading the computer program 20 into the memory 12, the control unit 10 can execute the computer program 20.

The display unit 13 can be configured with a liquid crystal panel, an organic EL (Electro Luminescence) display, or the like.

The operation unit 14 is configured with, for example, a hardware keyboard, a mouse, etc., and is capable of operating icons displayed on the display unit 13, moving and operating a cursor, inputting characters, etc. Note that the operation unit 14 may be configured with a touch panel.

The acquisition unit 31 of the control unit 10 acquires a thigh image including the thigh. The thigh image may be acquired from an external device (for example, an ultrasound diagnostic device, etc.) via the communication unit 11, or may be acquired from the storage unit 19. The thigh image is an image including the quadriceps femoris (rectus femoris, vastus medialis, vastus intermedius, and vastus lateralis).

FIG. 3 is a diagram showing an example of part detection by the image processing device 100. In the example of FIG. 3, each quadriceps femoris (rectus femoris, vastus medialis, vastus intermedius, and vastus lateralis) is detected as a region. The control unit 10 inputs the thigh image to the second learning model 22. The second learning model 22 is trained to output region information of the entire quadriceps muscle when a thigh image is input. The output unit 32 of the control unit 10 can acquire area information of the entire quadriceps muscle from the second learning model 22 and output it to the image generation unit 24.

More specifically, the second learning model 22 can output segmentation data (region information) when a thigh image is input. The segmentation data is obtained by classifying each pixel of the thigh image into classes. The second learning model 22 can classify pixels of each input thigh image into two classes, eg, classes 0 and 1. Class 0 indicates the background and indicates the area outside the quadriceps muscle. Class 1 indicates the quadriceps muscle.

The image generation unit 24 generates an image of the entire quadriceps muscle based on the region information of the entire quadriceps output by the second learning model 22. The entire quadriceps muscle image is, for example, an image in which regions classified as class 0 are set to black, and regions classified as class 1 are set to a specific color other than black.

Next, the control unit 10 inputs the thigh image and the generated entire quadriceps image to the first learning model 21. The first learning model 21 outputs region information of each quadriceps muscle (rectus femoris, vastus medialis, vastus intermedius, and vastus lateralis) when inputting a thigh image and an entire quadriceps image. This is how it is learned. The output unit 32 of the control unit 10 can acquire region information for each quadriceps muscle from the first learning model 21 and output it to the image generation unit 24.

More specifically, the first learning model 21 can output segmentation data (region information) when the thigh image and the whole quadriceps image are input. The segmentation data is obtained by classifying each pixel of the thigh image into classes. The first learning model 21 calculates the pixels of each input thigh image within the range of the entire quadriceps region indicated by the entire quadriceps image, for example, in classes 0, 2, 3, 4, and 5. can be classified into two classes. Class 0 indicates the background and indicates the area outside the quadriceps muscle. Classes 2 to 5 indicate each part. For example, class 2 indicates rectus femoris, class 3 indicates vastus intermedius, class 4 indicates vastus lateralis, and class 5 indicates vastus medialis.

Since not only the thigh image but also the whole quadriceps image is input to the first learning model 21, each quadriceps muscle is can be trained to output area information. Thereby, the muscle region (each quadriceps muscle region) can be identified with high accuracy.

The image generation unit 24 generates each quadriceps image based on the region information of each quadriceps outputted by the first learning model 21. In each quadriceps muscle image, for example, areas classified as class 0 are set to black, areas (parts) classified as classes 2 to 5 are set to colors other than black, and each quadriceps muscle image has a different display mode (for example, color). (See FIGS. 7 and 8). Muscle mass may be calculated based on the size of each part. By accurately detecting the region of each quadriceps muscle, the size of each quadriceps muscle can be accurately determined, and the muscle mass can be calculated accurately.

Each of the first learning model 21 and the second learning model 22 may use semantic segmentation such as U-Net, Attention-Unet, or SegNet, or may use instance segmentation such as Mask R-CNN, DeepMask, or FCIS. It may use segmentation or may be a GAN (Generative Adversarial Network).

FIG. 4 is a diagram showing an example of a method for generating the first learning model 21 and the second learning model 22. The second learning model 22 may be generated, for example, as follows. The control unit 10 acquires training data including a thigh image and area information of the entire quadriceps muscle. For example, the training data may be collected and stored in an external server or the like, and then acquired from the server. The control unit 10 inputs the thigh image included in the training data to the second learning model 22, and inputs the region information of the entire quadriceps muscle outputted by the second learning model 22 and the teacher data included in the training data. The parameters of the second learning model 22 may be adjusted so that the value of the loss function based on the area information of the entire quadriceps muscle is minimized.

Furthermore, the first learning model 21 may be generated in the following manner, for example. The control unit 10 acquires training data including a thigh image, an entire quadriceps image, and region information of each quadriceps muscle. For example, the training data may be collected and stored in an external server or the like, and then acquired from the server. The control unit 10 inputs the thigh image and the whole quadriceps image included in the training data to the first learning model 21, and inputs the area information of each quadriceps muscle output from the first learning model 21 and the training data. The parameters of the first learning model 21 may be adjusted so that the value of the loss function based on the area information of each quadriceps muscle as the included teacher data is minimized.

FIG. 5 is a diagram illustrating an example of femoral contour detection by the image processing device 100. The control unit 10 inputs the thigh image to the third learning model 23. The third learning model 23 is trained to output femoral region information when a femoral image is input. The output unit 32 of the control unit 10 can acquire region information of the femur from the third learning model 23 and output it to the image generation unit 24.

More specifically, the third learning model 23 can output segmentation data (region information) when a thigh image is input. The segmentation data is obtained by classifying each pixel of the thigh image into classes. The third learning model 23 can classify pixels of each input thigh image into two classes, classes 10 and 11, for example. Class 10 indicates the background and indicates the lateral region of the femur. Class 11 indicates the femur.

The image generation unit 24 generates a femur image based on the femur region information output by the third learning model 23. The femur image is, for example, an image in which a region classified into class 10 is set to black, and a region classified into class 11 is set to a specific color other than black.

The contour extraction section 16 extracts the contour of the femur based on the femur image generated by the image generation section 24, and generates a contour image in which the contour of the femur is drawn. Contour extraction can be performed, for example, as follows. The thigh image is converted into a binarized image (0: background, 1: figure pixel), the binarized image is raster scanned, and the first detected figure pixel is set as the starting point (first pixel). Eight neighboring pixels around the pixel at the starting point are checked, for example, in a counterclockwise direction, and if there is a graphic pixel, it is determined as the second pixel. Eight neighboring pixels around the second pixel are examined, and if there is a graphic pixel, it is determined as the third pixel. Thereafter, by repeating the same process, pixels forming the contour line can be extracted. Furthermore, a library such as OpenCV may be used for contour extraction.

The third learning model 23 may be generated, for example, as follows. The control unit 10 acquires training data including a thigh image and femur region information. The region information of the femur may be obtained by labeling the entire region of the femur, for example. The outline of the femur may be labeled, but if the outline is labeled and learned, part of the outline may be interrupted when detecting the outline of the femur. By labeling the entire region of the femur, the third learning model 23 detects a specific region. Therefore, by extracting the contour from the detected region, it is possible to prevent part of the contour from being cut off. The control unit 10 inputs the thigh image included in the training data to the third learning model 23, and inputs the femur region information outputted by the third learning model 23 and the femur as teacher data included in the training data. What is necessary is to adjust the parameters of the third learning model 23 so that the value of the loss function based on the region information and the region information is minimized.

The third learning model 23 may use semantic segmentation such as U-Net, Attention-Unet, or SegNet, or may use instance segmentation such as Mask R-CNN, DeepMask, or FCIS. It may also be a GAN (Generative Adversarial Network).

The first learning model 21, the second learning model 22, and the third learning model 23 may be generated by the image processing device 100, or an external learning processing device may generate each learning model and the image processing device 100 may generate the first learning model 21, the second learning model 22, and the third learning model 23. However, each learning model may be acquired from the learning processing device.

FIG. 6 is a diagram illustrating an example of a method for generating an entire quadriceps image. In the case of FIG. 3, the image generation unit 24 is configured to generate an image of the entire quadriceps muscle based on area information of the entire quadriceps muscle. In the case of FIG. 6, the image generation unit 24 generates an entire image of the quadriceps muscle, further taking into account the outline image of the femur. In this case, the image generation unit 24 generates the entire quadriceps image as follows. That is, an entire image of the quadriceps muscle is generated using, as a landmark, the outline of the femur drawn in the outline image of the femur generated based on the region information of the femur output by the third learning model 23. For example, if a part of the entire quadriceps muscle area is included inside the femoral contour line, that part is removed from the entire quadriceps area. In addition, if the distance between the contour line and the entire area of the quadriceps muscle is larger than a predetermined threshold (number of pixels), the distance between the outline line and the area of the entire quadriceps muscle is adjusted to be within the threshold. A portion of the entire muscle region that faces the contour line may be expanded (enlarged). The entire quadriceps image serves as the standard for detecting area information for each quadriceps (detect each quadriceps individually without extending beyond the range of the entire quadriceps), so It is preferable that the region is wide (has a large area) within an appropriate range.

As described above, the control unit 10 inputs the acquired thigh image to the third learning model 23, which is trained to output femur region information when the thigh image is input. , output the region information of the femur, extract the contour of the femur based on the output region information of the femur, and extract the entire quadriceps muscle image based on the output region information of the entire quadriceps muscle and the extracted contour. can be generated.

FIG. 7 is a diagram showing a first example of a method for generating a fascia image. The fascia image is an image obtained by removing each quadriceps muscle (muscle) from the thigh image or the entire quadriceps image. Let the parts of each quadriceps image be A, B, C, and D. For the quadriceps muscle, regions A to D are the rectus femoris, vastus medialis, vastus intermedius, and vastus lateralis, respectively. The example in FIG. 7 shows a case where the distance G between the region A and the region B is narrower than a predetermined range (predetermined number of pixels), and for example, a part of the region A and a part of the region B are in contact with each other. In this case, the fascia that originally exists between site A and site B does not exist. Therefore, the reduction/enlargement processing unit 15 performs a process of reducing at least one of the region A and the region B so that the distance G between the region A and the region B falls within a predetermined range. The reduction process can be performed, for example, by reducing the number of pixels outside the region by about 1 to 2 pixels. Note that the number of pixels to be reduced during the reduction process may be set to a required number depending on the position of the part.

The image generation unit 24 can generate a fascia image by removing each quadriceps muscle image that has been subjected to the reduction process from the thigh image or the entire quadriceps image. This allows a user such as a doctor to generate an appropriate fascia image while viewing the image.

As described above, the interval determination unit 151 of the reduction/enlargement processing unit 15 determines whether the interval between each quadriceps muscle is within a predetermined range based on the area information of each quadriceps muscle. When the distance between the quadriceps muscles is narrower than a predetermined range, the reduction/enlargement processing unit 15 corrects the area information by reducing at least one of the quadriceps muscles. The output unit 32 can output the fascia region information based on the quadriceps whole image and the region information of each quadriceps muscle including the corrected region information.

FIG. 8 is a diagram showing a second example of a method for generating a fascia image. The example in FIG. 8 shows a case where the interval G between the region A and the region D is wider than a predetermined range (predetermined number of pixels). In this case, the area of the fascia existing between site A and site D becomes large. Therefore, the reduction/enlargement processing unit 15 performs a process of enlarging (expanding) at least one of the region A and the region D, so that the distance G between the region A and the region D falls within a predetermined range. The enlargement process can be performed, for example, by increasing the number of pixels outside the region by about 1 to 2 pixels. Note that the number of pixels to be increased during the enlargement process may be set to a required number depending on the position of the part.

The image generation unit 24 can generate a fascia image by removing each quadriceps image that has been subjected to the enlargement process from the thigh image or the entire quadriceps image. This allows a user such as a doctor to generate an appropriate fascia image while viewing the image.

As described above, the interval determination unit 151 of the reduction/enlargement processing unit 15 determines whether the interval between each quadriceps muscle is within a predetermined range based on the area information of each quadriceps muscle. When the distance between the quadriceps muscles is wider than a predetermined range, the reduction/enlargement processing unit 15 enlarges (expands) at least one of the quadriceps muscles to correct the region information. The output unit 32 can output the fascia region information based on the quadriceps whole image and the region information of each quadriceps muscle including the corrected region information.

FIG. 9 is a diagram showing an example of a region correction screen 210 for each quadriceps image (region information). On the area correction screen 210 displayed on the display unit 13, an image display field 214, an editing tool 211, an "edit" icon 212, and a "save" icon 213 are displayed. The image display column 214 displays each quadriceps image generated by the image generation unit 24 based on the region information of each quadriceps outputted by the first learning model 21 (see FIG. 3). The parts of each quadriceps muscle are represented by A to D. Sites A to D are rectus femoris, vastus medialis, vastus intermedius, and vastus lateralis, respectively.

A user such as a doctor operates the "edit" icon 212, moves the cursor 215 to the position of a desired region, and uses the editing tools in the editing tool 211 to make corrections to the region of the region. Editing tools include, for example, a brush to draw free lines by dragging on the image, shape selection icons to draw lines, curves, squares, circles, or arrows on the image, pen tip size, and patterns. It also has icons for various tools, such as a palette icon for changing colors and an eraser for erasing desired parts of an image.

As shown in FIG. 9, for example, the boundary 216 between the region D and the region C can be modified to become a boundary 217. When the modification is completed, the user can save each modified quadriceps image by operating the "save" icon 213. The storage location may be the storage unit 19, an external data server, or the like. The region information of each quadriceps muscle after correction can be used as training data for relearning the first learning model 21. The relearning processing unit 17 can acquire the corrected region information of each quadriceps muscle and relearn the first learning model 21.

As described above, the control unit 10 accepts corrections to the area information of each quadriceps displayed on the display unit 13, and based on the area information for which the correction has been accepted, the thigh image and the entire quadriceps image, the control unit 10 The learning model 21 can be retrained. Thereby, the detection accuracy of each quadriceps muscle region by the first learning model 21 can be further improved.

FIG. 10 is a diagram showing an example of a region correction screen 220 of a fascia image (region information). On the area correction screen 220 displayed on the display unit 13, an image display field 221, an editing tool 211, an "edit" icon 212, and a "save" icon 213 are displayed. The fascia image generated by the image generation unit 24 is displayed in the image display column 221 (see FIGS. 7 and 8). A user such as a doctor operates the "edit" icon 212, moves the cursor 215 to a desired position, and uses the editing tool in the editing tool 211 to modify the fascia image.

As shown in FIG. 10, for example, a portion of fascia 222 may be modified as indicated at 223. This allows areas of fascia to be corrected. When the modification is completed, the user can save the modified fascia image (region information) by operating the "save" icon 213. The storage location may be the storage unit 19, an external data server, or the like. The corrected fascia region information can be used as training data for relearning the first learning model 21. The relearning processing unit 17 can acquire the corrected fascia region information and relearn the first learning model 21 by also taking into consideration the fascia region information. In this case, the region information of each quadriceps muscle can be updated based on the corrected fascia region information.

As described above, the control unit 10 accepts corrections to the fascia area information displayed on the display unit 13, and displays the thigh image and the area information for which the correction has been accepted (the updated area information of each quadriceps muscle). The first learning model 21 can be retrained based on the whole quadriceps image. Thereby, the detection accuracy of each quadriceps muscle region by the first learning model 21 can be further improved.

Next, the reliability index of the detection results of muscle parts by the image processing device 100 will be explained.

FIG. 11 is a diagram showing an example of area information of the entire quadriceps muscle outputted by the second learning model 22 using a different algorithm. The different algorithms are designated P1, P2, P3. Examples of the algorithms include U-Net, Attention-Unet, SegNet, Mask R-CNN, DeepMask, FCIS, and GAN (Generative Adversarial Network). As shown in FIG. 11A, the degree of coincidence (concordance rate) of the region information of the whole quadriceps outputted by the second learning model 22 using algorithms P1 and P2 is high. On the other hand, as shown in FIG. 11B, the degree of matching (matching rate) of the region information of the entire quadriceps muscle outputted by the second learning model 22 using the algorithms P1 and P3 is low. The reliability index calculation unit 18 calculates the ratio of the number of pixels occupying the entire area of the common quadriceps muscle to the number of pixels occupying the area of the entire quadriceps muscle outputted by each of the second learning models 22 using algorithms P1 and P2. can be calculated as the degree of matching. When both regions completely match, the degree of match is 100%.

The coincidence calculation unit 33 of the control unit 10 inputs the acquired thigh images to each of the plurality of types of second learning models 22 with different algorithms, and calculates the total quadriceps muscle output from each of the second learning models 22. Calculate the degree of matching of area information. If the calculated degree of matching is equal to or greater than a predetermined threshold, the image generation unit 24 may generate an image of the entire quadriceps muscle based on the output region information of the entire quadriceps muscle.

In addition, if the degree of matching of the region information of the entire quadriceps muscle outputted by each of the plurality of types of second learning models 22 is equal to or higher than a predetermined threshold (for example, 85%, 90%, 95%, etc.), Area information of the entire high quadriceps muscle may be used. Furthermore, among the degree of coincidence of the region information of the whole quadriceps muscle outputted by each of the plurality of types of second learning models 22, the region information with the lowest degree of coincidence is excluded, and the region information including all the remaining region information is used. You can also do this.

In addition, the confidence level (reliability of each pixel) of the area information of the entire quadriceps muscle output by the second learning model 22 is averaged, and if the average confidence level is lower than a predetermined threshold, another algorithm is applied. It is also possible to switch to the second learning model 22 used and output area information of the entire quadriceps muscle. Alternatively, the reliability of each pixel may be weighted and averaged for each of the plurality of types of second learning models 22, and the averaged model with a high reliability may be used.

FIG. 12 is a diagram showing another example of the configuration of the image processing device 50 of this embodiment. As shown in FIG. 12, the image processing device 50 can be, for example, a personal computer. The image processing device 50 can include a CPU 51, a ROM 52, a RAM 53, an input section 54, an output section 55, a recording medium reading section 56, and the like. A computer program (computer program product) recorded on a recording medium 1 (for example, an optically readable disk storage medium such as a CD-ROM) is read by a recording medium reading unit 56 (for example, an optical disk drive) and is stored on a hard disk (not shown). Can be stored. Here, the computer program includes processing procedures described in FIGS. 13 and 14, which will be described later. A computer program can be loaded into the RAM 53 and executed by the CPU 51.

FIG. 13 is a diagram illustrating an example of detection processing for each quadriceps muscle by the image processing device 100. For convenience, the main body of processing will be explained below as the control unit 10. The control unit 10 acquires a thigh image (S11), inputs the thigh image to the second learning model 22, and acquires area information of the entire quadriceps muscle output by the second learning model 22 (S11). (see FIG. 3) (S12).

The control unit 10 inputs the thigh image to the third learning model 23, acquires the region information of the femur output from the third learning model 23 (S13), and, based on the acquired region information of the femur, A contour image of the femur is generated (see FIG. 5) (S14).

The control unit 10 generates an image of the entire quadriceps muscle based on the area information of the entire quadriceps muscle and the contour image of the femur (see FIG. 6) (S15). Note that the processing in steps S13 to S15 is not essential. In this case, the process in step S15 may be changed to generate the entire quadriceps image based on the area information of the entire quadriceps.

The control unit 10 inputs the thigh image and the entire quadriceps image to the first learning model 21, and acquires the region information of each quadriceps output from the first learning model 21 (see FIG. 3). S16).

The control unit 10 generates each quadriceps image (S17), and determines whether the interval between each quadriceps falls within a predetermined range (S18). If the distance between each quadriceps muscle is within a predetermined range (YES in S18), a fascia image is generated (S19), and the process of step S21 described later is performed.

If the distance between each quadriceps muscle is not within the predetermined range (NO in S18), the control unit 10 reduces or expands at least one of the adjacent quadriceps muscles (see FIGS. 7 and 8) (S20), and in step S19 Process. The control unit 10 displays each quadriceps muscle image and fascia image on the display unit 13 (S21), and ends the process.

FIG. 14 is a diagram illustrating an example of a learning model generation process by the image processing apparatus 100. In FIG. 14, a method of generating the first learning model 21 and the second learning model 22 will be described. The control unit 10 acquires training data including a thigh image, area information of the entire quadriceps, and area information of each quadriceps (S31).

When a thigh image is input, the control unit 10 updates the internal parameters of the second learning model 22 so that the region information of the entire quadriceps muscle outputted by the second learning model 22 approaches the teacher data (Fig. 4 Reference) (S32). The control unit 10 determines whether the value of the loss function is within the allowable range (S33), and if the value of the loss function is not within the allowable range (NO in S33), performs the processing from step S32 onwards.

If the value of the loss function is within the allowable range (YES in S33), the control unit 10 generates a whole image of the quadriceps muscle based on the area information of the whole quadriceps muscle (S34), When the entire head muscle image is input, the internal parameters of the first learning model 21 are updated so that the region information of each quadriceps muscle output by the first learning model 21 approaches the teacher data (see FIG. 4) (S35). .

The control unit 10 determines whether the value of the loss function is within the allowable range (S36), and if the value of the loss function is not within the allowable range (NO in S36), performs the processing from step S35 onwards. If the value of the loss function is within the allowable range (YES in S36), the control unit 10 stores the generated first learning model 21 and second learning model 22 in the storage unit 19 (S37), and ends the process. do.

The computer program of this embodiment acquires a medical image including a muscle part into a computer, and when the medical image is input, the second learning model is trained to output area information of the entire muscle part. When the medical image is input, region information of the entire muscle region is outputted, and a whole muscle region image is generated based on the output region information of the entire muscle region, and the medical image and the whole muscle region image are input. The acquired medical image and the generated whole muscle image are input to a first learning model trained to output area information for each muscle part of the muscle part, and area information for each muscle part is inputted. Output or execute processing.

In the computer program of this embodiment, when a medical image including a muscle part is input to the computer, a third learning model trained to output area information of a predetermined bone near the muscle part is acquired. input the medical image, output predetermined bone region information, extract the outline of the bone based on the output predetermined bone region information, and extract the entire muscle part output by the second learning model. A process is executed to generate an entire muscle image based on the region information and the extracted contour.

The computer program of this embodiment inputs acquired medical images into each of the plurality of second learning models with different algorithms into a computer, and matches the area information of the entire muscle part outputted by each of the second learning models. If the calculated degree of coincidence is equal to or greater than a predetermined threshold, a process of generating the entire muscle part image based on the output region information of the entire muscle part is executed.

The computer program of the present embodiment causes the computer to receive corrections to the output region information of each muscle region, and to generate the first learning model based on the medical image and the entire muscle region image together with the region information for which the correction has been accepted. Re-learn and execute the process.

The computer program of the present embodiment causes the computer to determine whether or not the interval between the muscle parts is within a predetermined range based on the area information of each muscle part, and determines whether the interval between the muscle parts is within the predetermined range. If so, at least one of the muscle parts is reduced to correct the region information, and fascia region information is output based on the region information of each of the muscle parts including the whole muscle part image and the corrected region information. , execute the process.

The computer program of the present embodiment causes a computer to enlarge at least one of the muscle parts to correct area information, when the interval between the muscle parts is wider than the predetermined range, and to correct the area information of the whole muscle part image and the corrected area. A process is executed to output fascia region information based on the region information of each of the muscle parts including information.

The computer program of the present embodiment causes the computer to receive corrections to the output fascia region information, and re-learns the first learning model based on the medical image and the whole muscle image together with the region information for which the correction has been accepted. , execute the process.

In the computer program of this embodiment, the muscle section includes the thigh, the entire muscle section includes the entire quadriceps muscle, and the muscle region includes the quadriceps muscle.

The learning model generation method of the present embodiment acquires first training data including a medical image including a muscle part, a whole muscle part image showing the whole muscle part, and region information of each muscle part of the muscle part, and Based on the first training data, a first learning model is generated so as to output region information of each muscle part of the muscle part when a medical image and a whole muscle part image are input.

The learning model generation method of the present embodiment acquires first training data including a medical image including a muscle part, area information of the entire muscle part, and area information of each muscle part of the muscle part, and acquires the first training data. , a second learning model is generated so as to output region information of the entire muscle region when a medical image is input, an image of the entire muscle region is generated based on the region information of the entire muscle region, and the second learning model is generated based on the region information of the entire muscle region. Based on the first training data, a first learning model is generated so as to output region information of each muscle part of the muscle part when the medical image and the generated whole muscle part image are input.

The image processing device of this embodiment includes an acquisition unit that acquires a medical image including a muscle part, and a second learning model that is trained to output region information of the entire muscle part when a medical image is input. a first output section that inputs the medical image obtained and outputs region information of the entire muscle region; an image generation section that generates an image of the entire muscle region based on the output region information of the entire muscle region; Inputting the obtained medical image and the generated whole muscle image into a first learning model that is trained to output region information for each muscle part of the muscle part when the whole muscle image is input. and a second output unit that outputs area information for each muscle site.

The image processing system of this embodiment includes an acquisition unit that acquires a medical image including a muscle part, and a second learning model that is trained to output region information of the entire muscle part when a medical image is input. a first output unit that inputs the medical image and outputs area information of the entire muscle area; an image generation unit that generates an image of the entire muscle area based on the output area information of the entire muscle area; Inputting the obtained medical image and the generated whole muscle image into a first learning model that is trained to output region information for each muscle part of the muscle part when the whole muscle image is input. and a second output unit that outputs area information for each muscle site.

In the image processing method of the present embodiment, when a medical image including a muscular part is acquired and the medical image is input, the acquired medical An image is input, area information of the entire muscle area is output, and an image of the entire muscle area is generated based on the output area information of the entire muscle area, and when a medical image and an image of the entire muscle area are input, The acquired medical image and the generated whole image of the muscle part are input to a first learning model that has been trained to output region information of each muscle part of the body, and the region information of each muscle part is outputted. do.

In this embodiment, the quadriceps femoris of the thigh has been described as a muscle part, and each quadriceps muscle has been described as a muscle part, but the muscle part is not limited to the muscles of the thigh. For example, the muscle part may be the upper arm, abdomen, forearm, or lower leg, the muscle part may be each muscle constituting each muscle part, and the predetermined bones near the muscle part are each It may also be a bone near the muscular part of the body.

1 Recording medium 10 Control unit 11 Communication unit 12 Memory 13 Display unit 14 Operation unit 15 Reduction/enlargement processing unit 151 Interval determination unit 16 Contour extraction unit 17 Relearning processing unit 18 Reliability index calculation unit 19 Storage unit 20 Computer program 21 1 Learning model 22 2nd learning model 23 3rd learning model 24 Image generation section 31 Acquisition section 32 Output section 33 Matching degree calculation section 51 CPU
52 ROM
53 RAM
54 input section 55 output section 56 recording medium reading section 100, 50 image processing device

Claims (13)

to the computer,
Obtain medical images including muscles,
When a medical image is input, inputting the acquired medical image to a second learning model trained to output area information of the entire muscle part, and outputting area information of the entire muscle part,
Generates an image of the entire muscle area based on the output area information of the entire muscle area,
When a medical image and a whole muscle part image are input, the obtained medical image and the generated whole muscle part are input to a first learning model that is trained to output region information for each muscle part of the muscle part. Inputs an image and outputs region information for each muscle region.
A computer program that performs a process. to the computer,
When a medical image including a muscle part is input, the acquired medical image is input to the third learning model, which is trained to output region information of a predetermined bone near the muscle part, and Output bone area information,
Extracting the outline of the bone based on the output predetermined bone region information,
generating an image of the entire muscle region based on the region information of the entire muscle region outputted by the second learning model and the extracted contour;
The computer program according to claim 1, which causes the computer program to execute a process. to the computer,
Inputting the acquired medical image into each of the plurality of second learning models with different algorithms, calculating the degree of matching of the area information of the entire muscle part output by each of the second learning models,
If the calculated degree of matching is greater than or equal to a predetermined threshold, generating the entire muscle part image based on the output region information of the entire muscle part;
The computer program according to claim 1 or 2, which causes the computer program to execute a process. to the computer,
Accept corrections to the output area information of each muscle part,
relearning the first learning model based on the medical image and the whole muscle image together with the area information for which the correction has been accepted;
The computer program according to any one of claims 1 to 3, which causes the computer program to execute a process. to the computer,
Determining whether the interval between each muscle part is within a predetermined range based on the area information of each muscle part,
when the distance between the muscle parts is within the predetermined range, reducing at least one of the muscle parts to correct the area information;
outputting fascia region information based on the region information of each of the muscle parts including the whole muscle part image and corrected region information;
The computer program according to any one of claims 1 to 4, which causes the computer program to execute a process. to the computer,
If the distance between the muscle parts is wider than the predetermined range, at least one of the muscle parts is enlarged to correct the area information;
outputting fascia region information based on the region information of each of the muscle parts including the whole muscle part image and corrected region information;
The computer program according to claim 5, which causes the computer program to execute a process. to the computer,
Accepts corrections to the output fascia area information,
relearning the first learning model based on the medical image and the whole muscle image together with the area information for which the correction has been accepted;
The computer program according to any one of claims 1 to 6, which causes the computer program to execute a process. The muscular part includes the thigh,
The entire muscle part includes the entire quadriceps muscle,
The muscle region includes a quadriceps muscle,
A computer program according to any one of claims 1 to 7. Obtaining first training data including a medical image including a muscle part, a whole muscle part image showing the whole muscle part, and area information of each muscle part of the muscle part,
Generating a first learning model based on the first training data so as to output region information of each muscle part of the muscle part when a medical image and an entire muscle part image are input;
Learning model generation method. obtaining first training data including a medical image including a muscle part, area information of the entire muscle part, and area information of each muscle part of the muscle part;
Generating a second learning model based on the first training data so as to output region information of the entire muscle part when a medical image is input;
Generates an image of the entire muscle area based on area information of the entire muscle area,
Generating a first learning model based on the first training data so as to output region information for each muscle part of the muscle part when the medical image and the generated whole muscle part image are input;
Learning model generation method. an acquisition unit that acquires a medical image including a muscle part;
A first output unit that inputs the obtained medical image to a second learning model trained to output region information of the entire muscle region when a medical image is input, and outputs region information of the entire muscle region. and,
an image generation unit that generates an image of the entire muscle region based on the output region information of the entire muscle region;
When a medical image and a whole muscle part image are input, the obtained medical image and the generated whole muscle part are input to a first learning model that is trained to output region information for each muscle part of the muscle part. a second output unit that inputs an image and outputs region information of each muscle region;
Image processing device. an acquisition unit that acquires a medical image including a muscle part;
A first output unit that inputs the obtained medical image to a second learning model trained to output region information of the entire muscle region when a medical image is input, and outputs region information of the entire muscle region. and,
an image generation unit that generates an image of the entire muscle region based on the output region information of the entire muscle region;
When a medical image and a whole muscle part image are input, the obtained medical image and the generated whole muscle part are input to a first learning model that is trained to output region information for each muscle part of the muscle part. a second output unit that inputs an image and outputs region information of each muscle region;
Image processing system. Obtain medical images including muscles,
When a medical image is input, inputting the acquired medical image to a second learning model trained to output area information of the entire muscle part, and outputting area information of the entire muscle part,
Generates an image of the entire muscle area based on the output area information of the entire muscle area,
When a medical image and a whole muscle part image are input, the obtained medical image and the generated whole muscle part are input to a first learning model that is trained to output region information for each muscle part of the muscle part. Inputs an image and outputs region information for each muscle region.
Image processing method.

Images