CN109102501B

CN109102501B - Joint image processing method and image processing equipment

Info

Publication number: CN109102501B
Application number: CN201810888096.8A
Authority: CN
Inventors: 夏清; 谢帅宁
Original assignee: Beijing Sensetime Technology Development Co Ltd
Current assignee: Beijing Sensetime Technology Development Co Ltd
Priority date: 2018-08-06
Filing date: 2018-08-06
Publication date: 2021-05-28
Anticipated expiration: 2038-08-06
Also published as: CN109102501A

Abstract

The invention discloses a joint image processing method and image processing equipment. The method comprises the following steps: acquiring a joint image; processing the joint image by using an image processing model to obtain state information of the ligament in the joint image, wherein the state information is used for describing whether the ligament is torn or not; generating a processing report containing the status information. A corresponding image processing apparatus is also disclosed. Whether the ligament is torn or not is detected by processing the joint image through the image processing model, so that the processing efficiency of the joint image is effectively improved.

Description

Joint image processing method and image processing equipment

Technical Field

The present invention relates to the field of image processing technologies, and in particular, to a joint image processing method and an image processing apparatus.

Background

When people do strenuous exercises such as sports competition, dance and acrobatics, if the movement exceeding the joint moving range is rapidly performed, ligaments at the ankle joint, the knee joint, the palm joint or the finger joint can be passively pulled to cause tearing or complete breakage. Among them, the most common is the rupture of the posterior cruciate ligament, the anterior cruciate ligament and the medial collateral ligament caused by sudden violent backward or forward dislocation of the tibia in the knee flexion position.

Generally, small blood vessels rupture to bleed after the ligament injury, so that local swelling of joints, internal bleeding of tissues and the like can be found through external examination, obvious pain is caused during traction, and even the stability of the joints is reduced if the ligament is completely broken. When the ligament is damaged, in addition to primarily determining the injury of the patient through external examination, in order to make more precise examination, a method of Magnetic Resonance Imaging (MRI) may be used to obtain an image of the joint inside the joint, and then the tearing condition of the ligament may be determined through reading by a doctor.

However, since there are a lot of patients and the area of the anterior cruciate ligament on the magnetic resonance image of the joint is small, the doctor needs to observe the sectional images one by one to find the ligament, and the state of the ligament can not be detected accurately because of the difference between the skill level and experience of different doctors. In general, even if a lot of manpower and time are invested, the processing efficiency of the joint image is still low.

Disclosure of Invention

The embodiment of the application provides a joint image processing method and image processing equipment, and the application processes a joint image through an image processing model to detect whether a ligament is torn or not, so that the processing efficiency of the joint image is effectively improved.

In a first aspect, an embodiment of the present application provides a joint image processing method, including:

acquiring a joint image;

processing the joint image by using an image processing model to obtain state information of the ligament in the joint image, wherein the state information is used for describing whether the ligament is torn or not;

generating a processing report containing the status information.

With reference to the first aspect, in a first implementation of the first aspect, the image processing model includes a target segmentation module and a state classification module.

With reference to the first implementation manner of the first aspect, in a second implementation manner of the first aspect, the object segmentation module is configured to perform object segmentation on the joint image to segment a ligament image including a ligament in the joint image.

With reference to the first implementation or the second implementation of the first aspect, in a third implementation of the first aspect, the state classification module is configured to classify a state of a ligament in the ligament image to obtain state information of the ligament, where the state information includes no tear, partial tear, and fracture.

With reference to the first aspect, in a fourth implementation manner of the first aspect, the image processing model includes a target detection module, a region clipping module, and a state classification module.

With reference to the fourth implementation manner of the first aspect, in a fifth implementation manner of the first aspect, the object detection module is configured to perform object detection on the joint image to detect a ligament attachment point in the joint image.

With reference to the fourth implementation manner of the first aspect or the fifth implementation manner of the first aspect, in a sixth implementation manner of the first aspect, the area clipping module is configured to clip the ligament image in the joint image, where the ligament image takes the ligament attachment point as a base point.

With reference to any one implementation manner of the fourth implementation manner of the first aspect to the sixth implementation manner of the first aspect, in a seventh implementation manner of the first aspect, the state classification module is configured to classify states of ligaments in the ligament image to obtain state information of the ligaments, where the state information includes no tear, partial tear, and fracture.

With reference to the fourth implementation manner of the first aspect, in an eighth implementation manner of the first aspect, the target detection module is a first neural network, and the state classification module is a second neural network.

With reference to the eighth implementation manner of the first aspect, in a ninth implementation manner of the first aspect, before the processing the ligament image using the image processing model, the method further includes:

constructing a first neural network; obtaining a first training sample comprising the joint image, bone segmentation results, and/or the ligament attachment points; training the first neural network using the first training sample.

With reference to the eighth implementation manner of the first aspect, in a tenth implementation manner of the first aspect, before the processing the ligament image using the image processing model, the method further includes:

constructing a second neural network; obtaining a second training sample, the second training sample including the ligament image and the status information; training the second neural network using the second training samples.

With reference to any one of the first implementation manner of the first aspect to the tenth implementation manner of the first aspect, in an eleventh implementation manner of the first aspect, before the generating a processing report including the status information, the method includes:

obtaining feedback information and a loss function; substituting the feedback information and the state information into the loss function, and calculating to obtain loss; optimizing the state classification module using the loss; generating a processing report containing the feedback information.

In a second aspect, an embodiment of the present application provides an image processing apparatus including means for performing the method of the first aspect, the image processing apparatus including:

an acquisition unit configured to acquire a joint image;

the processing unit is used for processing the joint image by using an image processing model to obtain the state information of the ligament in the joint image, wherein the state information is used for describing whether the ligament is torn or not;

a generating unit configured to generate a processing report including the status information.

With reference to the second aspect, in a first implementation manner of the second aspect, the image processing model includes a target segmentation module and a state classification module; the processing unit comprises a segmentation unit and a classification unit.

With reference to the first implementation manner of the second aspect, in a second implementation manner of the second aspect, the segmentation unit is configured to perform target segmentation on the joint image by using the target segmentation module to segment a ligament image including a ligament in the joint image.

With reference to the first implementation manner of the second aspect or the second implementation manner of the second aspect, in a third implementation manner of the second aspect, the classifying unit is configured to classify a state of a ligament in the ligament image by using the state classification module to obtain state information of the ligament, where the state information includes no tear, partial tear, and fracture.

With reference to the second aspect, in a fourth implementation manner of the second aspect, the image processing model includes a target detection module, a region clipping module, and a state classification module; the processing unit comprises a detection unit, a cutting unit and a classification unit.

With reference to the fourth implementation manner of the second aspect, in a fifth implementation manner of the second aspect, the detection unit is configured to perform object detection on the joint image by using the object detection module to detect a ligament attachment point in the joint image.

With reference to the fourth implementation manner of the second aspect or the fifth implementation manner of the second aspect, in a sixth implementation manner of the second aspect, the cutting unit is configured to cut out the ligament image in the joint image by using the area cutting module, where the ligament image has the ligament attachment point as a base point.

With reference to the fourth implementation manner of the second aspect to the sixth implementation manner of the second aspect, in a seventh implementation manner of the second aspect, the classifying unit is configured to classify the state of the ligament in the ligament image by using the state classification module to obtain state information of the ligament, where the state information includes no tear, partial tear, and fracture.

With reference to the fourth implementation manner of the second aspect, in an eighth implementation manner of the second aspect, the target detection module is a first neural network, and the state classification module is a second neural network.

With reference to the eighth implementation manner of the second aspect, in a ninth implementation manner of the second aspect:

a construction unit for constructing a first neural network; the acquisition unit is further used for obtaining a first training sample, wherein the first training sample comprises the joint image, a bone segmentation result and/or the ligament attachment point; and the training unit is used for training the first neural network by using the first training sample.

With reference to the eighth implementation manner of the second aspect, in a tenth implementation manner of the second aspect, the method further includes: a construction unit for constructing a second neural network; the acquiring unit is used for acquiring a second training sample, and the second training sample comprises the ligament image and the state information; and the training unit is used for training the second neural network by using the second training sample.

With reference to any one of the first implementation manner of the second aspect to the tenth implementation manner of the second aspect, in an eleventh implementation manner of the second aspect: the obtaining unit is further configured to obtain feedback information and a loss function; the calculating unit is used for substituting the feedback information and the state information into the loss function to calculate and obtain the loss; an optimization unit for optimizing the state classification module using the loss; the generating unit is used for generating a processing report containing the feedback information.

With reference to the second aspect, in a twelfth implementation manner of the second aspect: the image processing apparatus includes the image processing model; or, the image processing apparatus further comprises a receiving unit for receiving the image processing model.

In a third aspect, an embodiment of the present application provides another image processing apparatus, including a processor, a communication interface, and a memory, where the processor, the communication interface, and the memory are connected to each other, where the memory is used to store a computer program, and the communication interface is used to perform data interaction with other terminal devices, and the computer program includes program instructions, and the processor is configured to call the program instructions to perform the method according to any one of the implementation manners of the first aspect to the first aspect.

In a fourth aspect, the present application provides a computer-readable storage medium, where a computer program is stored, where the computer program includes program instructions, where the program instructions are executed by a processor to perform the method of any one of the implementation manners of the first aspect to the first aspect.

In the present application, an image processing apparatus first acquires a joint image; then processing the joint image by using an image processing model to obtain state information of the ligament in the joint image, wherein the state information is used for describing whether the ligament is torn or not; and finally generating a processing report containing the state information. Therefore, whether the ligament in the joint image is torn or not is detected by using the image processing model, and compared with a traditional manual film reading method, the processing efficiency of the joint image can be greatly improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.

Fig. 1 is a schematic flow chart of a joint image processing method provided in an embodiment of the present application;

FIG. 2 is a schematic flow chart diagram of another joint image processing method provided in the embodiments of the present application;

FIG. 3 is a schematic diagram of an image processing model provided by an embodiment of the present application;

FIG. 4 is a schematic flow chart diagram of another joint image processing method provided by the embodiment of the application;

FIG. 5 is a schematic diagram of another image processing model provided by an embodiment of the present application;

fig. 6 is a schematic block diagram of an image processing apparatus provided in an embodiment of the present application;

fig. 7 is a structural block diagram of an image processing apparatus according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to a determination" or "in response to a detection". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".

In particular implementations, the image processing devices described in embodiments of the present application include, but are not limited to, terminal devices and servers having touch sensitive surfaces (e.g., touch screen displays and/or touch pads). Where the terminal device is a device such as a mobile phone, laptop or tablet computer, the server includes an image processing apparatus having a touch sensitive surface (e.g., a touch screen display and/or touch pad) and a desktop computer.

In the discussion that follows, an image processing device is described that includes a display and a touch sensitive surface. However, it should be understood that the image processing device may include one or more other physical user interface devices such as a physical keyboard, mouse, and/or joystick.

The improper and violent movement can lead to the ligament at the joint to be passively pulled to cause tearing or complete breakage, particularly the posterior cruciate ligament, the anterior cruciate ligament and the medial collateral ligament at the knee joint of a person, which are extremely easy to be injured. When a ligament is injured, although whether the ligament is injured or not can be preliminarily judged through external inspection, when a specific injured condition of the ligament is to be checked relatively accurately, a joint image of the inside of a joint is obtained by using a Magnetic Resonance Imaging (MRI) method, and then the torn condition of the ligament is judged through reading by a doctor. However, such manual film reading is inefficient, time-consuming and labor-consuming, and is affected by the experience and subjective judgment of the doctor, so that it is not guaranteed that the ligament state can be accurately detected. In general, even if a lot of manpower and time are invested, the processing efficiency of the joint image is still low.

The principle of magnetic resonance imaging is to place the human body in a special magnetic field, excite hydrogen nuclei in the human body with radio frequency pulses, cause the hydrogen nuclei to resonate, and absorb energy. After stopping the radio frequency pulse, the hydrogen atomic nucleus sends out radio signals according to specific frequency, releases absorbed energy, and is recorded by a receiver outside the body, and then the electronic computer images different tissues according to the intensity of the received energy and displays different gray scales. Generally, the nuclear magnetic resonance phenomenon directly reflects the surrounding environment state of protons in water molecules in a human body and the position in a molecular structure, so that biochemical pathological state and information on a molecular level are provided, the contrast to soft tissues is large, the resolution is high, and the range of pathological changes such as inflammation, edema and tumor is determined very definitely.

In order to solve the above problem, an embodiment of the present application provides a joint image processing method, which uses an image processing model to detect whether a ligament in a joint image is torn, and compared with a conventional method of manually reading a piece, the method of the present application can greatly improve the processing efficiency of the joint image.

In order to better understand the embodiment of the present application, a method applying the embodiment of the present application will be described below, and the embodiment of the present application may be applied to a scene in which an image processing model processes a joint image.

The image processing model comprises a target segmentation module and a state classification module, firstly, the target segmentation is carried out on the joint image by utilizing the target segmentation to segment the ligament image containing the ligament in the joint image, then, the state classification module is utilized to carry out state classification on the ligament image to identify the tearing state of the ligament in the ligament image, wherein the tearing state comprises no tearing, partial tearing and fracture.

Alternatively, in addition to directly performing the target segmentation on the joint image by using the target segmentation module, another image processing model may be used to perform the target segmentation on the joint image. The image processing model comprises an object detection module, an area cutting module and a state classification module, wherein the object detection module is used for carrying out object detection on a joint image to detect key points in the joint image, such as ligament attachment points and the like connecting ligaments and bones. And then, performing region clipping on the joint image based on the key point in the joint image by using a region clipping module, and clipping out a ligament image containing the ligament in the joint image.

Therefore, the joint image processing method realizes accurate positioning and tearing classification of the ligament in the joint image of the joint, and compared with an artificial film reading method, the accuracy of positioning and classification is greatly improved, so that the processing efficiency of the joint image is improved.

Referring to fig. 1, fig. 1 is a schematic flowchart illustrating a joint image processing method according to an embodiment of the present application, where the method includes:

101: an image of the joint is acquired.

In the embodiment of the present application, the image processing apparatus acquires a joint image, which is an image of a ligament therein obtained by a Magnetic Resonance Imaging (MRI) technique, wherein the joint image is Magnetic Resonance Imaging at the entire joint.

102: processing the joint image by using an image processing model to obtain state information of the ligament in the joint image, wherein the state information is used for describing whether the ligament is torn or not.

In the embodiment of the application, the image processing device processes the joint image by using the image processing model, so as to obtain the state information of the ligament in the joint image, wherein the state information comprises no tearing, partial tearing, breaking and the like, and whether the ligament is torn or not can be known from the state information of the ligament. Wherein the image processing model is obtained by the image processing device from a local storage or other terminal devices.

The image processing model is a model capable of performing image processing on the joint image, and is used for identifying the characteristics of the image to obtain a characteristic image of the joint image, and the characteristic image includes color characteristics, texture characteristics, shape characteristics, spatial relationship characteristics, and the like. The image processing model comprises an image processing model based on a neural network, an image processing model based on wavelet moments and an image processing model based on fractal characteristics.

Specifically, the image processing model in the embodiment of the present application includes a target segmentation module and a state classification module, where the processing of the joint image by using the image processing model refers to performing target segmentation on the joint image by using the target segmentation module to segment a ligament image including ligaments in the joint image, and then performing state classification on the ligaments in the ligament image by using the state classification module to obtain state information of the ligaments, so that whether the ligaments are torn or not can be known according to the state information of the ligaments.

Optionally, the image processing model includes an object detection module, an area clipping module and a state classification module, where the processing of the joint image by the image processing model includes firstly performing object detection on the joint image by the object detection module to detect a ligament attachment point in the joint image, then clipping the ligament image in the joint image by the area clipping module with the ligament attachment point as a base point, for example, clipping a preset size area as a ligament image with the ligament attachment point as a center, and finally performing state classification on the ligament image clipped by the area clipping module by the state classification module to detect a torn state of the ligament.

The image processing apparatus according to the embodiment of the present application detects a ligament attachment point in a joint image by performing object detection on the joint image by using an object detection module, and then cuts out a ligament image from the joint image by using the ligament attachment point as a base point by using an area clipping module. Compared with the image processing device in the previous application embodiment, the image processing device in the present application embodiment directly segments and segments the ligament image in the joint image by using the target, and acquires the ligament image by using the target detection module and the area clipping module.

Further, the target detection module, the state classification module and the target segmentation module are respectively a first neural Network, a second neural Network and a third neural Network, the first neural Network is a neural Network such as R-CNN, SPP-NET, Fast R-CNN or R-FCN, and the second neural Network and the third neural Network are a neural Network such as full convolution Network (FCN, full probabilistic Network), U-NET or V-Net.

In the embodiment of the present application, the first neural network is a multitask learning neural network, the multitask neural network can perform bone segmentation and ligament attachment point positioning simultaneously, specifically, the multitask neural network performs bone segmentation on a joint image, and then predicts the position of the ligament attachment point on a bone according to the segmented bone.

It should be noted that, due to the similarity of human body structures, there is usually a certain positional relationship between bones and ligament attachment points of most people, and this is helpful for accurately predicting ligament attachment points on the basis of knowing images of bones. Compared with the single-task learning for detecting the ligament attachment point, firstly, in the multi-task learning, the bone segmentation is easier, and secondly, the ligament attachment point can be more easily and accurately predicted on the basis that the bone is segmented, so that the multi-task learning is beneficial to sharing resources and parameters among multiple tasks, the training time is shortened, and the accuracy of target detection is improved. Since the first neural network learns the positional relationship between the bone and the ligament attachment point through multitask learning, the first neural network can predict the relationship between the ligament attachment points based on the segmented bone after the bone segmentation of the joint image, and predict the position of the ligament attachment point on the bone according to the segmented bone.

Further, before processing the joint image using an image processing model, constructing a first neural network; obtaining a first training sample, wherein the first training sample comprises joint images, bone segmentation results and/or ligament attachment points; training the first neural network with a first training sample.

In the embodiment of the present application, a framework of a first neural network is first constructed, for example, a composition of several layers of neural networks, each layer of neural network may be a convolutional layer, a pooling layer, a batch normalization layer and/or an activation function layer, wherein the convolutional layer may use Conv3D or Conv2D to convolve input image data to obtain a feature image, the pooling may use a maximum pooling method or an average pooling method to simplify data, the batch normalization layer is used to increase a training speed of an image processing model, and the activation function layer may introduce a non-linear factor to improve an ability of the image processing model to solve a non-linear problem. Here, Conv3D refers to a convolution operation for a stereo image, and Conv2D refers to a convolution operation for a plane image. The feature image describes features of the image, including color features, texture features, shape features, spatial relationship features, and the like.

After the first neural network is constructed, a large number of training samples are input into the first neural network, so that the first neural network is trained, and the accuracy of processing the joint image can be improved by training parameters in the first neural network to be continuously modified and optimized.

It should be noted that the first training sample includes a negligible number of joint images, bone segmentation results and/or ligament attachment points, i.e. training sets. The training of the first neural network by using the first training sample means that the first training sample is input into the network, the difference between the actual output and the expected output of the network is calculated by using a loss function, and then the parameter in the first neural network is adjusted by using an optimization algorithm, so that the first neural network is optimized. The loss function includes a DICE loss function, an IOU loss function, a regression loss function, a cross entropy function, a Logloss function and the like, and the optimization function includes a back-propagation (back-propagation) algorithm and the like.

For example, before training, the parameters in the first neural network are random numbers, and the parameters in the first neural network can be adjusted after training, so that a mature target detection module capable of correctly extracting the features in the joint image is obtained. In particular, one sample in the training setBook (A)_i,B_i,C_i) Joint image data a in (1)_iInputting the skeleton image into the first neural network to obtain a skeleton image Y of the first neural network_iAnd ligament attachment point Z_i. Wherein A is_iImage data for the ith joint image, B_iActual bone image for the ith joint image, C_iThe ligament attachment point for the ith joint image. Then D ═ B (B) is calculated_i-Y_i)-y(C_i-Z_i) D is the error between the predicted value and the actual value, b and y are arbitrary constants, and b + y is 1. Parameters in the first neural network are then adjusted using a back propagation algorithm based on the magnitude of the error D. And repeating the process for each sample until the error D does not exceed the preset error, thereby indicating that the training of the first neural network is completed. The preset error may be any set value.

Further, before processing the joint image using an image processing model, constructing a second neural network; obtaining a second training sample, wherein the second training sample comprises the ligament image and the state information; the second neural network is trained using the second training samples.

In the embodiment of the application, a framework of a second neural network is firstly constructed, such as five convolutional neural network layers, one pooling layer and one classification layer, wherein the five convolutional neural network layers comprise a plurality of convolutional layers, batch normalization layers, activation function layers and pooling layer operations, the size of a characteristic image output by each layer is 64 x 32, 32 x 16, 16 x 8, 8 x 4 and 4 x 4 2 respectively through the five neural network layers, and meanwhile, the number of characteristic channels is increased from 1 to 2048. In order to extract the features of the image sufficiently, the second neural network performs feature extraction on the same image for a plurality of times by using a plurality of convolution kernels at the convolution layer, so that a plurality of images are generated, the plurality of images are also regarded as a plurality of channels of the same image, so that more images are generated after the image processing, the images are stacked to increase the dimension of the image, and the number of channels of the image is increased.

It should be noted that the convolutional layer may use Conv3D or Conv2D to convolve the input image data to obtain a feature image, the pooling may use a maximum pooling method or an average pooling method to simplify the data, the batch normalization layer is used to increase the training speed of the image processing model, and the activation function layer may introduce a non-linear factor to improve the capability of the image processing model to solve the non-linear problem. Here, Conv3D refers to a convolution operation for a stereo image, and Conv2D refers to a convolution operation for a plane image. The feature image describes features of the image, including color features, texture features, shape features, spatial relationship features, and the like.

The pooling layer is used for pooling feature images according to channels, and outputs depth feature descriptors with length of 2048, the classification layer comprises a full connection layer and a softmax layer, and the depth feature descriptors output by the pooling layer are input into the full connection layer and then output through the softmax layer to obtain probabilities of torn states of the ligament.

For example, the classification layer output p ═ 0.1, 0.2, 0.7, which represents probabilities of ligament "no tear", "partial tear", and "rupture", respectively, i.e., the result represents that the torn state of ligament is most likely to be rupture.

It should be noted that the number of layers and functions of the second neural network described above by way of example do not limit the embodiments of the present application.

After the second neural network is constructed, a large number of training samples are input into the second neural network, so that the image processing model is trained, and the parameters in the training second neural network are continuously modified and optimized, so that the second neural network can improve the accuracy of processing the joint image.

It should be noted that the second training sample includes a negligible number of the plurality of ligament images and the state information, i.e., a training set (training set). The training of the second neural network using the second training samples means inputting the second training samples into the second neural network, calculating a difference between an actual output and a desired output of the second neural network using a loss function, and then adjusting parameters of the second neural network using an optimization algorithm, thereby optimizing the second neural network. The loss function includes a DICE loss function, an IOU loss function, a regression loss function, a cross entropy function, a Logloss function and the like, and the optimization function includes a back-propagation (back-propagation) algorithm and the like.

For example, before training, the parameters in the image processing model are random numbers, and data in the image processing model can be adjusted after training to obtain a mature graphic processing model capable of correctly extracting features in the joint image. Specifically, one sample (A) in the training set is used_i,B_i) Joint image data a in (1)_iInputting into the first neural network to obtain actual state information Y of ligament_i. Wherein A is_iImage data for the ith ligament image, B_iIs the expected state information of the ith joint image. Then calculating D ═ B_i-Y_iAnd D is the error between the predicted value and the actual value. Parameters in the second neural network are then adjusted using a back propagation algorithm based on the magnitude of the error D. And repeating the process for each sample until the error D does not exceed the preset error, thereby representing that the training of the second neural network is completed. The preset error may be any set value.

103: generating a processing report containing the status information.

In the embodiment of the present application, after the status information is obtained, a processing report template is acquired, and a processing report including the status information is generated by combining the obtained status information.

Further, before generating a processing report containing the state information, obtaining feedback information and a loss function; substituting the feedback information and the state information into the loss function, and calculating to obtain loss; optimizing the state classification module using the loss; generating a processing report containing the feedback information.

In the embodiment of the present application, after the state information is obtained by processing the joint image by the state classification module, an operator (for example, a doctor) may view the state information and manually input the feedback result of the manual film reading, and if the feedback result of the manual film reading is inconsistent with the state information obtained by processing the joint image by the state classification module, it indicates that the film reading by the state classification module is inaccurate and needs to be optimized. Then, the feedback information is compared with the state information obtained by the state classification module, the loss is calculated by using a loss function, and parameters of the image processing model are updated by using a back-propagation algorithm according to the loss, so that the image processing model is optimized. Accordingly, a processing report containing the feedback information is generated, and the processing report is used for displaying the processing result in a standardized way. Wherein the loss function includes a DICE loss function, an IOU loss function, a regression loss function, a cross entropy function and a Logloss function.

For example, the state information includes non-torn, partially torn, and broken states, and the result of the state classification module classifying the state of the ligament is p ═ 0.1, 0.2, 0,7, which indicates that the ligament is most likely to be broken, and then the result of the state classification in the feedback information is compared, for example, y ═ 0, 0, 1, which indicates that the result of the diagnosis of the ligament by the doctor in the feedback information is broken, so that it can be seen that the prediction of the state classification is correct, but the parameters in the state classification module are not perfect enough, which can be further optimized to make the probability that the state classification module judges that the ligament is broken as close to 1 as possible.

The classification accuracy of the image processing model is then described using a cross-entropy function as a classification loss function, which may be expressed as L_i＝-log(p_i) Wherein L is_iIs the loss of state information of class i, p_iThe probability that the state of the ligament is of the ith class is predicted for the state classification module. According to the above-mentioned example, the Loss ═ log (0.7) for correctly classifying the image processing model can be obtained from the classification Loss function.

It should be further noted that, the above optimizing the image processing model by using a back-propagation algorithm according to the loss means that derivation is performed by using a chain rule, so as to propagate the loss back, and then the weight parameters in the image processing model are updated, so as to ensure that the predicted result and the actual result of the final image processing model are kept within a certain error range, that is, the error is zero, which is the final target.

In the embodiment of the present application, the image processing apparatus first acquires a joint image; then processing the joint image by using an image processing model to obtain state information of the ligament in the joint image, wherein the state information is used for describing whether the ligament is torn or not; and finally generating a processing report containing the state information. Therefore, whether the ligament in the joint image is torn or not is detected by using the image processing model, and compared with a traditional manual film reading method, the processing efficiency of the joint image can be greatly improved.

Referring to fig. 2, fig. 2 is a schematic flowchart illustrating a joint image processing method according to an embodiment of the disclosure, where the method includes:

201: an image of the joint is acquired.

202: and performing target segmentation on the joint image by using a target segmentation module so as to segment a ligament image containing ligaments in the joint image.

In an embodiment of the present invention, the joint image is subjected to object segmentation by an object segmentation module to segment a ligament image including a ligament in the joint image. The target segmentation module comprises an image processing model based on a neural network, an image processing model based on a wavelet moment, an image processing model based on a fractal characteristic and the like.

Further, the target segmentation module is a third neural Network, and the third neural Network is a Full Convolution Network (FCN), a U-NET, or a V-NET.

It should be noted that, in the embodiment of the present application, as shown in fig. 3, the image processing model includes an image processing model containing an object segmentation module and a state classification module. The object segmentation module is used for performing object segmentation on the joint image to segment a ligament image containing ligaments in the joint image, and the state classification module is used for performing state classification on the ligaments in the ligament image to obtain state information of the ligaments, so that whether the ligaments are torn or not can be known according to the state information of the ligaments.

203: and carrying out state classification on the ligaments in the ligament image by using a state classification module so as to obtain the state information of the ligaments.

In the embodiment of the application, the ligament in the ligament image is subjected to state classification by using a state classification module to obtain state information of the ligament, wherein the state information includes no tearing, partial tearing and breakage. The target segmentation module comprises an image processing model based on a neural network, an image processing model based on a wavelet moment and an image processing model based on a fractal characteristic.

Further, the state classification module is used for classifying the state of the first neural Network into a state of a.

204: and generating a processing report containing the state information.

In the embodiment of the application, the image processing model comprises a target segmentation module and a state grading module, so that the image processing device firstly utilizes the target segmentation module to perform target segmentation on the joint image to segment out a ligament image comprising ligaments in the joint image, and then utilizes the state grading module to perform state classification on the ligaments in the ligament image to obtain state information of the ligaments, so that whether the ligaments are torn or not can be known according to the state information of the ligaments. Therefore, in the embodiment of the application, the image processing device can detect whether the ligament in the joint image is torn or not by using the target segmentation module and the state classification module in the image processing model, and compared with the traditional manual image reading method, the application can greatly improve the processing efficiency of the joint image.

401: an image of the joint is acquired.

402: and performing target detection on the joint image by using a target detection module so as to detect the ligament attachment point in the joint image.

In the embodiment of the application, the target detection module is used for carrying out target detection on the joint image, and ligament attachment points in the joint image are detected. The target detection module comprises an image processing model based on a neural network, an image processing model based on a wavelet moment, an image processing model based on a fractal characteristic and the like.

It should be noted that, in the embodiment of the present application, as shown in fig. 5, the image processing model includes an image processing model including an object detection module, a region clipping module, and a state classification module. The method comprises the steps of firstly utilizing an object detection module to carry out object detection on a joint image, detecting a ligament attachment point in the joint image, then utilizing an area cutting module to cut the ligament image in the joint image by taking the ligament attachment point as a base point, for example, taking the ligament attachment point as a center, cutting an area with a preset size as a ligament image, and finally utilizing a state classification module to carry out state classification on the ligament image obtained by cutting the area cutting module, thereby detecting the tearing state of the ligament.

Furthermore, the target detection module is a first neural network, and the first neural network is a neural network such as R-CNN, SPP-net, Fast R-CNN, Faster R-CNN or R-FCN.

For example, before training, the parameter in the first neural network is a random number, and after passing trainingParameters in the first neural network can be adjusted to obtain a mature target detection module capable of correctly extracting features in the joint image. Specifically, one sample (A) in the training set is used_i,B_i,C_i) Joint image data a in (1)_iInputting the skeleton image into the first neural network to obtain a skeleton image Y of the first neural network_iAnd ligament attachment point Z_i. Wherein A is_iImage data for the ith joint image, B_iActual bone image for the ith joint image, C_iThe ligament attachment point for the ith joint image. Then D ═ B (B) is calculated_i-Y_i)-y(C_i-Z_i) D is the error between the predicted value and the actual value, b and y are arbitrary constants, and b + y is 1. Parameters in the first neural network are then adjusted using a back propagation algorithm based on the magnitude of the error D. And repeating the process for each sample until the error D does not exceed the preset error, thereby indicating that the training of the first neural network is completed. The preset error may be any set value.

403: and utilizing an area cutting module to cut out the ligament image in the joint image, wherein the ligament image takes the ligament attachment point as a base point.

In the embodiment of the present application, the area clipping module clips the ligament image in the joint image with the ligament attachment point as a base point, for example, clipping an area with a preset size as the ligament image with the ligament attachment point as a center.

404: and carrying out state classification on the ligaments in the ligament image by using a state classification module so as to obtain the state information of the ligaments.

In the embodiment of the application, the ligament images obtained by the region clipping module are classified by the state classification module, and the tearing state of the ligament is detected. The state classification module comprises an image processing model based on a neural network, an image processing model based on a wavelet moment, an image processing model based on a fractal characteristic and the like.

Further, the state classification module is a second neural network. The second neural Network is a Full Convolution Network (FCN), a U-NET or a V-Net.

405: and generating a processing report containing the state information.

In this embodiment, the image processing module includes an object detection module, an area clipping module and a state classification module, so that the image processing apparatus firstly uses the object detection module to perform object detection on the joint image, and detects a ligament attachment point in the joint image, then the area clipping module clips the ligament image in the joint image with the ligament attachment point as a base point, for example, using the ligament attachment point as a center, and clips a preset size area as the ligament image, and finally uses the state classification module to perform state classification on the ligament image clipped by the area clipping module, and detect a torn state of the ligament. Therefore, in the embodiment of the application, the image processing device can detect whether the ligament in the joint image is torn or not by using the target detection module, the region cutting module and the state classification module in the image processing model, and compared with a traditional manual film reading method, the application can greatly improve the processing efficiency of the joint image.

The embodiment of the present application also provides an image processing apparatus, which is used for executing the units of the method of the foregoing first embodiment. Specifically, referring to fig. 6, a schematic block diagram of an image processing apparatus provided in an embodiment of the present application is shown. The image processing apparatus of the present embodiment includes: the obtaining unit 610, the processing unit 620, and the generating unit 630 specifically:

an acquisition unit 610 for acquiring a joint image;

a processing unit 620, configured to process the joint image using an image processing model to obtain status information of a ligament in the joint image, where the status information is used to describe whether the ligament is torn or not;

a generating unit 630, configured to generate a processing report including the status information.

Specifically, the image processing model comprises a target segmentation module and a state classification module; the processing unit 620 includes a dividing unit 621 and a classifying unit 622; the segmentation unit 621 is configured to perform target segmentation on the joint image by using the target segmentation module to segment a ligament image including a ligament in the joint image; a classifying unit 622, configured to classify the ligament state in the ligament image by using the state classifying module to obtain state information of the ligament, where the state information includes no tear, partial tear, and fracture.

Optionally, the image processing model includes a target detection module, a region clipping module, and a state classification module; the processing unit 620 includes a detecting unit 623, a clipping unit 624, and a classifying unit 622; the detection unit 623 is configured to perform target detection on the joint image by using the target detection module to detect a ligament attachment point in the joint image; a cutting unit 624, configured to cut out the ligament image in the joint image by using the region cutting module, where the ligament image uses the ligament attachment point as a base point; a classifying unit 622, configured to classify the ligament state in the ligament image by using the state classifying module to obtain state information of the ligament, where the state information includes no tear, partial tear, and fracture.

Further, the object detection module is a first neural network, the state classification module is a second neural network, and the object segmentation module is a third neural network.

Further, the image processing apparatus further includes a constructing unit 640 configured to construct a first neural network; the obtaining unit 610 is further configured to obtain a first training sample, where the first training sample includes the joint image, the bone segmentation result, and/or the ligament attachment point; a training unit 650, configured to train the first neural network by using the first training sample.

Optionally, the constructing unit 640 is configured to construct a second neural network; the acquiring unit 610 is configured to acquire a second training sample, where the second training sample includes the ligament image and the status information; a training unit 650 for training the second neural network using the second training samples.

Further, the obtaining unit 610 is further configured to obtain feedback information and a loss function; the image processing apparatus further includes a calculating unit 660, configured to substitute the feedback information and the state information into the loss function, and calculate a loss; an optimization unit 670 for optimizing the state classification module using the loss; the generating unit is further configured to generate a processing report including the feedback information.

Optionally, the image processing apparatus includes the image processing model; or the image processing device further comprises a receiving unit 680, which is used for receiving the image processing model.

In the present application, an image processing apparatus first acquires a joint image with an acquisition unit; then processing the joint image by using a processing unit to obtain the state information of the ligament in the joint image, wherein the state information is used for describing whether the ligament is torn or not; finally, the generating unit generates a processing report containing the state information. Therefore, whether the ligament in the joint image is torn or not is detected by using the image processing model, and compared with a traditional manual film reading method, the processing efficiency of the joint image can be greatly improved.

Referring to fig. 7, another image processing apparatus provided in the embodiment of the present application includes one or more processors 710, a communication interface 720 and a memory 730, where the processors 710, the communication interface 720 and the memory 730 are connected to each other, where the memory 730 is used to store a computer program, the communication interface 720 is used to perform data interaction with other terminal apparatuses, the computer program includes program instructions, and the processors 710 are configured to call the program instructions to perform the method according to the embodiment of the present invention as described above, specifically:

a processor 710 for performing the functions of the acquisition unit 610 for acquiring a joint image; is further configured to perform the function of the processing unit 620 for processing the joint image using an image processing model to obtain status information of a ligament in the joint image, the status information describing whether the ligament is torn or not; and is further configured to perform the function of the generating unit 630 for generating a processing report containing the status information.

Specifically, the processor 710 is configured to execute a function of the segmentation unit 621, configured to perform target segmentation on the joint image by using the target segmentation module to segment a ligament image including a ligament in the joint image; and is further configured to perform a function of the classifying unit 622, configured to perform a state classification on the ligament in the ligament image by using the state classification module to obtain state information of the ligament, where the state information includes no tear, partial tear, and fracture.

Optionally, the processor 710 is configured to execute a function of the detection unit 623, configured to perform target detection on the joint image by using the target detection module to detect a ligament attachment point in the joint image; further configured to execute a function of the cropping unit 624, configured to crop out the ligament image in the joint image by using the region cropping module, wherein the ligament image takes the ligament attachment point as a base point; and is further configured to perform a function of the classifying unit 622, configured to perform a state classification on the ligament in the ligament image by using the state classification module to obtain state information of the ligament, where the state information includes no tear, partial tear, and fracture.

Further, the processor is further configured to execute the functions of the constructing unit 640 for constructing the first neural network; further for obtaining a first training sample comprising the joint image, bone segmentation results, and/or the ligament attachment point; and further for performing the function of a training unit 650 for training the first neural network with the first training samples.

Optionally, the processor 710 is further configured to construct a second neural network; also for obtaining a second training sample, the second training sample containing the ligament image and the status information; and further configured to train the second neural network with the second training samples.

Further, the processor 710 is further configured to obtain feedback information and a loss function; is further configured to perform a function of the calculating unit 660, configured to substitute the feedback information and the state information into the loss function, and calculate a loss; and for performing the function of an optimization unit 670 for optimizing the state classification module with the loss; and also for generating a processing report containing the feedback information.

Optionally, the image processing apparatus includes the image processing model.

Optionally, the communication interface is configured to perform the function of the receiving unit 680, for receiving the image processing model.

It should be understood that in the embodiments of the present Application, the Processor 510 may be a Central Processing Unit (CPU), and the Processor may be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 530 may include both read-only memory and random access memory, and provides instructions and data to the processor 510. A portion of memory 530 may also include non-volatile random access memory. For example, memory 530 may also store device type information.

In a specific implementation, the processor 510, the communication interface 520, and the memory 530 described in this embodiment of the present application may execute the implementation manners described in the first embodiment and the second embodiment of the joint image processing method provided in this embodiment of the present application, and may also execute the implementation manner of the image processing apparatus described in this embodiment of the present application, which is not described herein again.

The embodiment of the present application further provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, and the computer program includes program instructions, which are executed by a processor, to perform the method according to the embodiment of the present invention.

The computer readable storage medium may be an internal storage unit of the image processing apparatus of any of the foregoing embodiments, such as a hard disk or a memory of the image processing apparatus. The computer-readable storage medium may also be an external storage device of the image processing apparatus, such as a plug-in hard disk provided on the image processing apparatus, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like. Further, the computer-readable storage medium may also include both an internal storage unit of the image processing apparatus and an external storage apparatus. The computer-readable storage medium is used to store a computer program and other programs and data required by the image processing apparatus. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

The embodiment of the present application further provides a computer program product, which includes a computer readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute the method of the embodiment of the present invention.

Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the server and the unit described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed server and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of a unit is merely a logical division, and an actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may also be an electric, mechanical or other form of connection.

Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiments of the present application.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention essentially or partially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

Claims

1. A method of processing a joint image, the method comprising:

acquiring a joint image;

processing the joint image by using an image processing model to obtain state information of the ligament in the joint image, wherein the state information is used for describing whether the ligament is torn or not; the image processing model comprises a target detection module, a region cutting module and a state classification module; the target detection module is a first neural network, the first neural network comprises a multitask learning network, the first neural network is used for carrying out bone segmentation on the joint image and predicting the position of a ligament attachment point in the joint image according to the segmented bone and the learned position relationship between the bone and the ligament attachment point; the area cutting module is used for cutting out a ligament image in the joint image according to the position of the ligament attachment point, wherein the ligament image takes the ligament attachment point as a base point, and the base point is a center; the state classification module is a second neural network, the second neural network comprises five layers of convolutional neural networks, one layer of pooling layer and one layer of classification layer, and the second neural network is used for carrying out state classification on ligaments in the ligament images so as to obtain state information of the ligaments;

generating a processing report containing the status information.

2. The method of claim 1, wherein the status information comprises no tear, partial tear, and break.

3. The method of claim 1, wherein prior to processing the ligament image using an image processing model, further comprising:

constructing a first neural network;

obtaining a first training sample comprising the joint image, bone segmentation results, and/or the ligament attachment points;

training the first neural network using the first training sample.

4. The method of claim 1, wherein prior to processing the ligament image using an image processing model, further comprising:

constructing a second neural network;

obtaining a second training sample, the second training sample including the ligament image and the status information;

training the second neural network using the second training samples.

5. The method of any of claims 1 to 4, wherein prior to generating the process report containing the status information, comprising:

obtaining feedback information and a loss function;

substituting the feedback information and the state information into the loss function, and calculating to obtain loss;

optimizing the state classification module using the loss;

generating a processing report containing the feedback information.

6. An image processing apparatus characterized by comprising:

an acquisition unit configured to acquire a joint image;

the processing unit is used for processing the joint image by using an image processing model to obtain the state information of the ligament in the joint image, wherein the state information is used for describing whether the ligament is torn or not; the image processing model comprises a target detection module, a region cutting module and a state classification module; the target detection module is a first neural network, the first neural network comprises a multitask learning network, the first neural network is used for carrying out bone segmentation on the joint image and predicting the position of a ligament attachment point in the joint image according to the segmented bone and the learned position relationship between the bone and the ligament attachment point; the area cutting module is used for cutting out a ligament image in the joint image according to the position of the ligament attachment point, wherein the ligament image takes the ligament attachment point as a base point, and the base point is a center; the state classification module is a second neural network, the second neural network comprises five layers of convolutional neural networks, one layer of pooling layer and one layer of classification layer, and the second neural network is used for carrying out state classification on ligaments in the ligament images so as to obtain state information of the ligaments;

7. The apparatus according to any one of claim 6, wherein the state information includes no tear, partial tear, and break.

8. The image processing apparatus according to claim 6, further comprising:

a construction unit for constructing a first neural network;

the acquisition unit is further used for obtaining a first training sample, wherein the first training sample comprises the joint image, a bone segmentation result and/or the ligament attachment point;

and the training unit is used for training the first neural network by using the first training sample.

9. The image processing apparatus according to claim 6, further comprising:

a construction unit for constructing a second neural network;

the acquiring unit is used for acquiring a second training sample, and the second training sample comprises the ligament image and the state information;

and the training unit is used for training the second neural network by using the second training sample.

10. The image processing apparatus according to any one of claims 6 to 9, further comprising:

the obtaining unit is further configured to obtain feedback information and a loss function;

the calculating unit is used for substituting the feedback information and the state information into the loss function to calculate and obtain the loss;

an optimization unit for optimizing the state classification module using the loss;

the generating unit is used for generating a processing report containing the feedback information.

11. The image processing apparatus according to claim 6, wherein the image processing apparatus contains the image processing model; or

The image processing apparatus further comprises a receiving unit for receiving the image processing model.

12. An image processing apparatus, comprising a processor, a communication interface and a memory, the processor, the communication interface and the memory being interconnected, wherein the memory is configured to store a computer program, the communication interface is configured to interact with other terminal devices, the computer program comprises program instructions, and the processor is configured to invoke the program instructions to perform the method according to any one of claims 1 to 5.

13. A computer-readable storage medium, characterized in that the computer storage medium stores a computer program comprising program instructions for execution by a processor for performing the method according to any one of claims 1-5.