CN116433560A

CN116433560A - Muscle disease identification method, device and equipment based on ultrasonic bimodal images

Info

Publication number: CN116433560A
Application number: CN202111670824.6A
Authority: CN
Inventors: 肖杨; 张湘楠; 袁乐萌; 李彦明; 郑海荣
Original assignee: Shenzhen National Research Institute of High Performance Medical Devices Co Ltd
Current assignee: Shenzhen National Research Institute of High Performance Medical Devices Co Ltd
Priority date: 2021-12-31
Filing date: 2021-12-31
Publication date: 2023-07-14

Abstract

The application provides a muscle disease identification method, device and equipment based on ultrasonic bimodal images, wherein the method comprises the following steps: acquiring a gray image sequence and an elastic image sequence of detected skeletal muscles of an object to be identified; preprocessing the gray level image sequence and the elastic image sequence to obtain a gray level image and an elastic image; and inputting the gray level image and the elastic image into a preset muscle disease recognition network to recognize whether the object to be recognized has muscle disease. The scheme is convenient and quick, has instantaneity, not only can acquire the anatomical structure of the muscle, but also can quantitatively evaluate the hardness of the muscle, and solves the problem that the information of the muscle tissue acquired by a single mode is not comprehensive enough.

Description

Muscle disease identification method, device and equipment based on ultrasonic bimodal images

Technical Field

The invention belongs to the technical field of medical evaluation, and particularly relates to a muscle disease identification method, device and equipment based on ultrasonic bimodal images.

Background

Muscle disorders include muscle atrophy. Muscle atrophy refers to the decrease in muscle volume caused by striated muscular dystrophy, attenuation or even loss of muscle fibers, and the like. The etiology of muscle atrophy can be divided into three types: 1) physiological, 2) neurogenic, 3) pathological muscular atrophy. Physiological muscle atrophy is caused by insufficient muscle usage, and can be reversed by exercise and better nutritional supplementation. Neurogenic muscular atrophy is the most severe type of muscular atrophy, atrophy resulting from nerve injury or disease connecting muscles. Such as rare deadly Amyotrophic Lateral Sclerosis (ALS), 80% -90% of ALS patients die within 3 to 5 years. Pathological muscular atrophy is common in muscle inflammation, hyperglycemia, and the like. Muscle inflammation leads to catabolism and impairs anabolic signals, resulting in reduced muscle mass. Patients with sepsis accompanied by systemic inflammation, develop muscle atrophy of the upper and lower limbs; in addition, multiple organ failure can also lead to significant muscle atrophy. The syndrome cachexia associated with malignant neoplastic disease is mainly manifested by progressive skeletal muscle atrophy, with loss of muscle mass being a key feature, and death of up to 20% of cancer patients is associated with cachexia. Muscle diseases increase the risk of morbidity, disability and mortality of individuals, have a great impact on the quality of life of patients and their families, and also bring a great economic burden to the medical system and the whole society. If the muscle diseases can be diagnosed timely and effectively, the early screening of acute and chronic diseases such as diabetes, cancer and the like can be facilitated, so that the treatment can be carried out pertinently, the prognosis of patients can be improved, and the survival rate can be improved.

The current clinical diagnostic methods widely used for muscle wasting diseases are: (1) physical examination: the doctor evaluates the muscle function by observing the volume and appearance of the muscle, the extent of muscular atrophy, whether there is muscle tendon tremor, and the like. Most doctors clinically judge the muscle tension by touching the hardness of the muscle due to muscular atrophy accompanied by muscle strength reduction, but the evaluation method is larger in subjective factor image of the doctors and introduces uncertain factors; (2) Electromyography, which is the earliest method for evaluating skeletal muscle, is based on recording bioelectric signals generated when skeletal muscle contracts and is in a resting state, and thus evaluating the functional state of muscle. The electromyographic signals can be used to aid in the diagnosis of motor and neuromuscular diseases, and can also be used as signals to aid in motor rehabilitation and control prostheses. However, the electromyographic signals are also susceptible to neighboring muscles, the environment and noise, which limit the application of the electromyographic signals in muscle evaluation. (3) Imaging means, mainly nuclear magnetic resonance (Magnetic Resonance Imaging, MRI) and computed tomography (Computed Tomography, CT) and Ultrasound (US). MRI can image in multiple directions, has high spatial resolution, and can detect small changes in muscle size and structure; multiple measurements and lateral comparisons are required to obtain a diagnosis of muscle atrophy; MRI elastography can measure muscle stiffness, but can only be examined in the prone position, which is time consuming and not amenable to claustrophobia and metal implanted patients; namely, MRI is long in time consumption, lacks real-time performance and is high in price, and has a certain limitation in clinical examination of amyotrophy. CT has good evaluation on the shape of the muscle, and the imaging of the muscle outline is clearer, but due to certain requirements on radiation dose and insufficient contrast of soft tissue imaging of CT, the CT is more suitable for the examination of bone diseases. Although the traditional ultrasonic elastography technique can also evaluate the mechanical properties of muscle tissue, it has the following drawbacks: the one-dimensional elastography technology can only measure the average elastic modulus value of the tissue and can not acquire the elastic distribution information of the tissue; the quasi-static elastography technology can only give a relative two-dimensional pseudo-color displacement/strain map, and cannot give specific values of local hardness of tissues.

Disclosure of Invention

An object of an embodiment of the present disclosure is to provide a method, an apparatus and a device for identifying muscle diseases based on ultrasound bimodal images.

In order to solve the technical problems, the embodiments of the present application are implemented in the following manner:

in a first aspect, the present application provides a method for identifying muscle disorders based on ultrasound bimodal images, the method comprising:

acquiring a gray image sequence and an elastic image sequence of detected skeletal muscles of an object to be identified;

preprocessing the gray level image sequence and the elastic image sequence to obtain a gray level image and an elastic image;

and inputting the gray level image and the elastic image into a preset muscle disease recognition network to recognize whether the object to be recognized has muscle disease.

In one embodiment, the preset muscle disease recognition network comprises:

a first convolution layer group comprising two convolution layers;

a first max pooling layer;

a second convolution layer group comprising two convolution layers;

a second maximum pooling layer;

a third convolution layer group comprising three convolution layers;

a third maximum pooling layer;

a fourth convolution layer group comprising three convolution layers;

a fourth max pooling layer;

a full connection layer having 512 nodes;

softmax layer.

In one embodiment, the preset muscle disease recognition network further comprises: and a batch normalization algorithm, so that before each layer of input of the preset muscle disease identification network, the output data distribution of the previous layer is normalized by using the batch normalization algorithm.

In one embodiment, a preset muscle disease recognition network introduces a learnable parameter gamma and beta, calculates the average value and variance of all neurons of a feature map corresponding to all samples, and performs normalization processing on the neurons of the whole feature map.

In one embodiment, calculating the average value and variance of all neurons of the feature map corresponding to all samples, and normalizing the neurons of the whole feature map includes:

introducing initial values of parameters beta, gamma and epsilon to obtain data B= { x before inputting the activation function ₁ ,x ₂ ,…x _n }；

Finding the mean value of the input mu _B Sum of variances

And (3) carrying out normalization processing on the input data:

training parameters β, γ:

output data y after batch normalization operation _i ：

In one embodiment, a dropout mechanism is included in the preset muscle disease recognition network.

In one embodiment, preprocessing a gray scale image sequence and an elastic image sequence to obtain a gray scale image and an elastic image includes:

Framing the gray level image sequence and the elastic image sequence, extracting preset frames at equal intervals, and cutting out an imaging region of gray level imaging and an imaging region of shear wave elastic imaging;

and subtracting pixel values of the imaging region of shear wave elastography and the imaging region of gray scale imaging one to obtain an elastography and a gray scale image.

In a second aspect, the present application provides a muscle disease identification device based on ultrasound bimodal imaging, the device comprising:

the acquisition module is used for acquiring a gray image sequence and an elastic image sequence of the detected skeletal muscle of the object to be identified;

the preprocessing module is used for preprocessing the gray image sequence and the elastic image sequence to obtain a gray image and an elastic image;

the identification module is used for inputting the gray level image and the elastic image into a preset muscle disease identification network and identifying whether the object to be identified has muscle disease.

In a third aspect, the present application provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method for identifying muscle disorders based on ultrasound bimodal images as in the first aspect when the program is executed by the processor.

In a fourth aspect, the present application provides a readable storage medium having stored thereon a computer program which when executed by a processor implements the ultrasound bimodal image based muscle disease recognition method as in the first aspect.

The technical scheme provided by the embodiment of the present specification can be seen from the following scheme:

starting from the characteristic of muscle ultrasound, muscle atrophy can cause the anatomical structure of the muscle to change, and the gray scale ultrasound mode can image the anatomical structure of the muscle; muscle atrophy results in increased stiffness due to a decrease in the number of muscle fibers and the composition of connective tissue, and shear wave elastography modalities are capable of deriving the elastic modulus of muscle tissue. The data of the fusion of the anatomical structure and the elastic information of the gray ultrasonic imaging is used as the input of the neural network, so that the problem of low single-mode diagnosis performance is solved.

Compared with electromyography, CT and MRI, the method is convenient and rapid by adopting two diagnosis modes of an ultrasonic gray mode and an elastic mode, has instantaneity, can acquire the anatomical structure of the muscle, can quantitatively evaluate the hardness of the muscle, and solves the problem that the information of the muscle tissue acquired by a single mode is not comprehensive.

Drawings

In order to more clearly illustrate the embodiments of the present description or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some of the embodiments described in the present description, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of a method for identifying muscle diseases based on ultrasound bimodal images provided by the present application;

fig. 2 is a schematic diagram of an image sequence preprocessing flow provided in the present application;

fig. 3 is a flow chart of construction of a preset muscle disease recognition network provided in the present application;

fig. 4 is a schematic structural diagram of a preset muscle disease recognition network provided in the present application;

FIG. 5 is a comparison of the structure of a neural network without dropout and with dropout;

FIG. 6 is a graph of contrast for different imaging modalities based on MAVGG-BN;

FIG. 7 is a schematic structural diagram of a device for identifying muscle diseases based on ultrasound bimodal images provided by the present application;

fig. 8 is a schematic structural diagram of an electronic device provided in the present application.

Detailed Description

In order to make the technical solutions in the present specification better understood by those skilled in the art, the technical solutions in the embodiments of the present specification will be clearly and completely described below with reference to the drawings in the embodiments of the present specification, and it is obvious that the described embodiments are only some embodiments of the present specification, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are intended to be within the scope of the present disclosure.

In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

It will be apparent to those skilled in the art that various modifications and variations can be made in the specific embodiments of the present disclosure without departing from the scope or spirit of the disclosure. Other embodiments will be apparent to the skilled person from the description of the present application. The specification and examples are exemplary only.

As used herein, the terms "comprising," "including," "having," "containing," and the like are intended to be inclusive and mean an inclusion, but not limited to.

The "parts" in the present application are all parts by mass unless otherwise specified.

In the related art, ultrasonic imaging has the advantages of no radiation, low cost, real time and the like, and is a preferred imaging method for diagnosing amyotrophy. Ultrasound imaging is currently the only non-invasive diagnostic technique for muscle diseases. Ultrasonic energy well characterizes muscle structure to aid in understanding muscle atrophy or injury, and is currently the most widespread method for quantitatively assessing muscle structure characteristics. The muscle structure characteristics comprise muscle thickness, subcutaneous fat thickness, feather angle, muscle bundle length and the like, and the change of pathological structure characteristics of the muscle can be detected by gray ultrasonic imaging; in addition to the above structural feature parameters, gray scale ultrasound can also characterize the condition of muscles by measuring echoes and non-uniformities obtained as gray scale differences. Various types of muscular dystrophies and neurogenic diseases, such as spinal muscular atrophy, often exhibit echo enhancement on gray scale ultrasound images. The muscles of ICU patients lying in bed for a long period of time are atrophic, lose their normal echogenic texture, gradually blur the texture and have a uniform frosted glass appearance. Normally healthy muscle tissue is formed by alternately arranging black muscle fiber tissue and white muscle bundle membrane, and is shown as a star pattern on a transverse muscle image; the inside of the atrophic muscle may be damaged by fat infiltration, and the fasciculi membrane is also degenerated to lose its original compact structure, partially "penetrating" into the muscle, which exhibits a frostlike change. It is because the muscles have a specific texture structure, and the lesion of the muscles can be found by observing the ultrasonic image of the muscles by using the gray-scale ultrasonic. However, when the anatomical structure of the muscle is often displayed on the gray-scale ultrasonic image to change, the atrophy degree of the muscle is very serious, and a single ultrasonic gray-scale mode has the problem of insufficient sensitivity, so that the basis can not be provided for early diagnosis.

Many studies have found that muscle atrophy results in a decrease in the number of muscle fibers and a decrease in the relative concentration of connective tissue in the endometrium, perimuscular and epicardial tissues, resulting in significant changes in the structure and mechanisms of the muscle bundles and significant changes in the biomechanical properties of the muscle. Ultrasonic shear wave elastography for quantitatively detecting mechanical properties of biological tissues is an emerging ultrasonic imaging technology in recent years, and the basic principle is that acoustic pulses emitted by an ultrasonic probe are focused at different depths of tissues to generate transverse shear waves, and a physical quantity-Young modulus value of the propagation speed of the shear waves in the reaction tissues can be calculated through a quantitative analysis system. The relation between Young's modulus value and shear wave propagation velocity is: e=3ρc ² (E: young's modulus; c: shear wave propagation velocity; ρ: tissue density). The harder the tissue, the greater the density, the greater the shear wave propagation velocity, and the greater the Young's modulus value. The technology breaks through the limitation of the traditional ultrasonic elastography technology, and realizes the real quantitative measurement of the tissue elasticity. The novel revolutionary technology breaks through the limitations of the traditional ultrasonic elastography technology, and enables quantitative analysis of biomechanical characteristics of human tissues to be possible. The method is quantitative reproduction of palpation of clinicians, and can be regarded as a relevant tool for diagnosis in places where palpation has clinical value, and the method has good effect and high sensitivity in breast tumor characterization, liver fibrosis stage and thyroid diseases. However, the muscle has a different structure from that of the breast, liver, thyroid, etc., and the hardness of the muscle changes with the change of the muscle strength. The elasticity of the muscle as measured by elastography is influenced by changes in the structure of the muscle on the one hand and by the longitudinal shear modulus parallel to the muscle fibres in vivo and in vitro being linearly related to the passive and active muscle forces, e.g. different joint angles or muscle activation The resulting elasticity value of the muscle will also change. Since muscle is an active tissue, structural parameters should also be characterized when the muscle is passively stretched. Thus, the use of a single modality may be deficient in diagnosing the extent of muscle atrophy. Different ultrasonic modes can reflect different properties of muscle tissues, and can complement each other by matching with application, so that more available information is provided for identifying and diagnosing the atrophy of the muscle tissues. The ultrasonic elastography can reflect biomechanical information of muscle tissues, and the combined use of the ultrasonic elastography and the gray level ultrasound can improve the diagnosis accuracy, make up the defect of single-mode diagnosis and provide a richer basis for doctors to diagnose muscular atrophy.

In recent years, with the rapid development of artificial intelligence and deep learning, in the medical field, the artificial intelligence can rapidly identify pictures and images and extract important information therefrom, thereby helping doctors to rapidly acquire information required for diagnosis. Because of the traditional machine learning method, the characteristics are required to be manually extracted and screened, human subjective factors are inevitably doped in the whole characteristic extraction and dimension reduction screening process, and the extracted characteristics cannot fully represent the information of the image. The deep learning forms more abstract high-level features to represent attribute categories or features by combining low-level features, and uses a neural network to perform end-to-end training from the original pixel values to the output of the classifier, thereby making up the characteristics of machine learning that the features need to be manually designed and the like and depend on professionals.

In the technology of detecting muscle tissue lesions using deep learning, 1) Diagnosing Muscle Atrophy by Use of a Comprehensive Method of Assessing the Elastic Properties of Muscle During Passive Stretching proposes to estimate a hardness index of muscle tissue from a measured muscle extension length value and a corresponding young's modulus value sequence, the index parameter being closely related to the degree of muscle atrophy, but the calculation of the parameter is very complex; 2) The full convolution deep learning algorithm is adopted in Development of an automatic muscle atrophy measuring algorithm to calculate the ratio of supraspinatus in supraspinous fossa using deep learning to measure the I phase and the II phase of supraspinatus muscular dystrophy to quantitatively detect supraspinatus rotator cuff tear, so that the diagnosis efficiency of doctors is improved; 3) The Automated diagnosis of myositis from muscle ultrasound: exploring the use of machine learning and deep learning methods combines machine learning and deep learning technologies to diagnose a patient with myositis in a muscle ultrasonic image, proposes that a conventional random forest machine learning method needs to be used for manually drawing muscle boundaries, and classification based on a convolutional neural network does not need to be used for manually drawing, thereby providing a full-automatic ultrasonic myositis classification method; 4) The Early diagnosis of Alzheimer's disease with deep learning proposes a deep learning structure consisting of a stacked automatic encoder and a Softmax output regression layer, and the early diagnosis of Alzheimer's disease caused by hippocampal atrophy is carried out, so that the accuracy reaches 88.58%; 5) The method is characterized in that a spectrogram and continuous wavelet transformation are adopted to perform time-frequency representation on electromyographic signals in a Classification of amyotrophic lateral sclerosis disease based on convolutional neural network and reinforcement sample learning algorithm article, and a convolutional neural network consisting of two convolutional layers, two pooling layers and one full-connection layer is used for classifying time-frequency characteristics, so that the identification accuracy of amyotrophic lateral sclerosis is 96.80%; 6) The A novel deep learning-based multi-model ensemble method for the prediction of neuromuscular disorders uses Bhattacharya coefficients and genetic algorithms for feature selection and convolutional neural networks to group classifiers such as decision trees, k-nearest neighbors, random forests, support vector machines, etc., thereby constructing an integrated model for diagnosing neuromuscular normal and disease samples. The related researches of the deep learning are concentrated on the classification of myositis, tendon laceration and neurodegenerative muscle diseases, so that good effects are obtained, and the classification accuracy is high.

The parameter closely related to the degree of muscular atrophy in the above document 1) is relatively complex to calculate, and requires a physician to have a solid mathematical background; in documents 2) to 5), the current deep learning application is focused on myositis, tendon lacerations and neurodegenerative muscle diseases, and the related documents related to muscular atrophy classification are relatively few, and related algorithm research is lacking in the muscular atrophy field. After the ultrasonic image data is acquired, the traditional doctor obtains a diagnosis result through a visual reading mode, which is highly dependent on long-term clinical experience and medical background knowledge, the mode is interfered by subjective factors, the resolution of human eyes on the image is far lower than that of the image, and a large amount of characteristic information is still contained in the image and is not discovered. At present, a machine learning method is widely applied, and the traditional machine learning method generally extracts texture characteristic parameters from an image, and then carries out mathematical modeling based on the parameters so as to achieve the aim of image recognition. However, the method needs to manually extract the characteristics and screen and reduce the dimension, and partial characteristic information is lost, and artificial subjective factors are added.

Because the gray-scale ultrasonic mode can image the anatomical structure of the muscle and can acquire the structural parameters such as the volume, the thickness, the cross-sectional area and the like of the muscle; the shear wave elastography mode can obtain the hardness value of the muscle tissue in a noninvasive and quantitative mode. Therefore, aiming at the defects of the existing clinical diagnosis technology and the lack of wide application of deep learning in the field of amyotrophy, the method utilizes the unique biomechanical characteristics of skeletal muscle, acquires a real-time ultrasonic bimodal (gray scale and elasticity) image sequence during passive stretching, carries out channel fusion on images of two modes, adopts a convolutional neural network algorithm, and identifies muscle diseases, thereby achieving the purpose of diagnosing the amyotrophy diseases.

The invention is described in further detail below with reference to the drawings and examples.

Referring to fig. 1, a flow chart of a method for identifying muscle diseases based on ultrasound dual-mode images, which is applicable to the embodiment of the application, is shown. It will be appreciated that muscle disorders include muscle atrophy and other disorders, and that muscle atrophy is exemplified in the examples below. Muscle atrophy may include thigh muscle atrophy, calf muscle atrophy, scapular band muscle atrophy, myogenic facial muscle atrophy, interosseous and thenar muscle atrophy, and the like.

As shown in fig. 1, the method for identifying muscle diseases based on ultrasound bimodal images can include:

s110, acquiring a gray level image sequence and an elastic image sequence of the detected skeletal muscle of the object to be identified.

In particular, the object to be identified may be a person to be actually suffering from a muscle disorder.

The skeletal muscle examined corresponds to a muscle atrophy site. For example, the calf muscle is atrophic and the skeletal muscle examined is gastrocnemius. The skeletal muscle to be examined below will be described by taking gastrocnemius muscle as an example.

When the gray image sequence and the elastic image sequence of the gastrocnemius are acquired, the object to be identified adopts a supine position, the right foot (or the left foot) of the object to be identified is fixed on a pedal of a constant-speed muscle strength training system, and the constant-speed muscle strength training system drives the ankle to perform constant-speed passive motion from plantar flexion to dorsiflexion at 40 degrees, and the detected skeletal muscle is pulled to cause passive stretching. During the whole ankle joint uniform passive rotation process, the subject is required to completely relax the calf muscle and not autonomously provide any principal force or resistance. Continuously imaging the gastrocnemius structure through a B-mode of an ultrasonic imaging system, displaying the structural morphological change of the muscle during passive stretching, and obtaining a gray image sequence of the detected skeletal muscle. And synchronously acquiring an elastic image sequence of the skeletal muscle through a shear wave elastic imaging mode of an ultrasonic imaging system. Different elastic image sequences of the muscle tissue in the passive stretching process are obtained, the imaging area is 13mm x 7mm, and the sequence duration is 28 s-32 s. The gray image sequence and the elastic image sequence are the ultrasonic bimodal image sequence.

S120, preprocessing the gray level image sequence and the elastic image sequence to obtain a gray level image and an elastic image, which may include:

Specifically, the preset frame may be set according to actual requirements, for example, 9 frames.

As shown in fig. 2, framing operation is completed on the acquired bimodal image sequence by using opencv, 9 frames are uniformly and equidistantly extracted, an ROI (region of interest ) region (including an imaging region of gray scale imaging and an imaging region of shear wave elastography) is cut out, pixel values of an imaging region of SWE (shear wave elastography, shear wave elastrography) and a pixel value of a B-mode ultrasonic imaging region are subtracted one to one, an original ROI elastography and a gray scale image are obtained, and the size of the processed image is normalized to 224 x 224. In fig. 2, the SWE original image is a color image, and the SWE original image is represented by gray.

S130, inputting the gray level image and the elastic image into a preset muscle disease recognition network, and recognizing whether the object to be recognized has muscle disease.

The preset muscle disease recognition network is a pre-trained and optimized network model, and is used for outputting whether an object corresponding to the gray level image and the elastic image has muscle disease or not when the gray level image and the elastic image are input.

According to the embodiment of the application, from the characteristic of muscle ultrasound, muscle atrophy can cause the anatomical structure of the muscle to change, and the gray-scale ultrasound mode can image the anatomical structure of the muscle; muscle atrophy results in increased stiffness due to a decrease in the number of muscle fibers and the composition of connective tissue, and shear wave elastography modalities are capable of deriving the elastic modulus of muscle tissue. The data of the fusion of the anatomical structure and the elastic information of the gray ultrasonic imaging is used as the input of the neural network, so that the problem of low single-mode diagnosis performance is solved.

The method adopts two diagnosis modes of an ultrasonic gray mode and an elastic mode, is convenient and quick compared with electromyography, CT and MRI, has instantaneity, can not only acquire the anatomical structure of the muscle, but also quantitatively evaluate the hardness of the muscle, and solves the problem that the acquisition of muscle tissue information by a single mode is not comprehensive enough.

As shown in fig. 3, the construction of the preset muscle disease recognition network may be performed by:

the specific acquisition process may refer to step S110 by acquiring a bimodal image sequence of muscular tissue of a plurality of subjects, and the object to be identified in step S110 may be executed several times as one subject.

The bimodal image sequence is preprocessed to obtain a gray image and an elastic image, and the specific preprocessing process can refer to step S120.

Labeling the processed data, and inputting the labeled data into a convolutional neural network for training and classification. It will be appreciated that the data may be made into a training set, a validation set and a test set, the trained model of the training set requiring the validation set to determine parameters for the network structure or controlling the complexity of the model, the test set to verify the performance of the model.

Among the best 5 models of VGG in a network architecture that classifies 1000 categories of input images in the world's best known computer vision competition ImageNet classification, VGG is the model that has the highest accuracy in classifying input images. The network contains 16 weight layers and is therefore referred to as a "deep network". Every two or three convolution layers are connected with one pooling layer, and the number of convolution kernels is 2 times of the relation, so that the structure of the network is irregular. The original VGG-16 model, with a total of 5 sets of convolutions, is increased in depth by using the smaller 3*3 convolution kernel architecture, the first and second sets each containing two convolutional layers, the third through fifth sets each containing 3 convolutional layers, and the last 3 fully-connected layers, with the parameters of the network concentrated on the last three fully-connected layers. Because the VGG-16 network is classified for 1000 input categories, but in the application, only two categories of diseases and health are provided, the data volume is relatively small, and therefore, the VGG-16 network is improved in the application.

In one embodiment, as shown in fig. 4, the preset muscle disease recognition network adopts an improved VGG16 model, and the structure is as follows:

a first convolution layer group comprising two convolution layers;

a first max pooling layer;

a second convolution layer group comprising two convolution layers;

a second maximum pooling layer;

a third convolution layer group comprising three convolution layers;

a third maximum pooling layer;

a fourth convolution layer group comprising three convolution layers;

a fourth max pooling layer;

a full connection layer having 512 nodes;

softmax layer.

Specifically, the improvement of the VGG16 model eliminates the fifth group of convolution layers and the last three full connection layers of the VGG16 model, and adds one full connection layer with 512 nodes, so that the network is more simplified, and the parameter quantity and the calculation quantity are reduced. After 10-layer convolution and corresponding pooling operations, the data is flattened using the flat operation to become a one-dimensional vector.

Specifically, since the input data has been artificially normalized for each sample, and the nature of the training network is to update parameters, the updating of the weights of the previous layer of network will result in the change of the distribution of the output data, so that the distribution of the input data of each layer of the rest of network layers except the input layer is changed, and the training speed of the network is greatly reduced. When the batch normalization algorithm (Batch Normalization, BN) aims at training, before each layer of the network is input, the BN is used for carrying out normalization processing on the output data distribution of the previous layer before entering the next layer of the network, so that the problem that the hidden layer data distribution changes in the process of training the network can be solved.

However, when normalizing the output data of the previous layer, the features learned by the network of the previous layer cannot be destroyed, so that the preset muscle disease recognition network introduces the learnable parameters gamma and beta, calculates the average value and variance of all neurons of the feature map corresponding to all samples, and performs normalization processing on the neurons of the whole feature map, including:

Finding the mean value of the input mu _B Sum of variances

And (3) carrying out normalization processing on the input data:

training parameters β, γ:

output data y after batch normalization operation _i ：

The constant change of the weight parameters in the network training process causes the distribution of the input of each subsequent layer to change, and the learning process needs to adapt each layer to the input distribution, so that the learning rate has to be reduced. After the BN algorithm is introduced, higher learning rate is allowed, and training is quickened. And by introducing the learnable reconstruction parameters gamma and beta, the network can learn and recover the feature distribution to be learned of the original network.

According to the characteristic of smaller clinical data volume, the VGG network is introduced and applied to optimize, the simple and portable network is designed, and the characteristic value output by the upper layer is prevented from excessively large distribution change before the input of the lower layer by adding the normalization algorithm, so that the generalization capability of the network is enhanced.

The convolution neural network is adopted to automatically analyze the characteristic information of the ultrasonic double modes, and the characteristics are not required to be manually extracted and screened, so that the classification result is more objective and accurate.

The full connection layer maps the distributed feature description learned by convolution, pooling, activation and other operations to a sample marking space, so that the influence of the feature position on classification is greatly reduced, and the function of a classifier is realized in the whole convolution neural network. But its main disadvantage is the large number of parameters involved, and the complex calculations to be performed during the training process. Nodes and connections can be eliminated using dropout technology.

The main function of dropout is to make some neurons stop working randomly, eliminate joint adaptability among neuron nodes and enhance generalization capability of the network. Because of the small clinical data set, it is difficult to train a large number of parameters when training a deep convolutional neural network, which is prone to overfitting problems that perform well on the training set but have poor classification effect on the test set. The dropout mechanism is similar to a voting strategy, some neurons are randomly discarded according to a certain probability, and a large number of experimental researches show that the method can effectively prevent the network from being over fitted, enhance the sparsity of the network and reduce the network parameters. Fig. 5 is a comparison diagram of the neural network structure with or without dropout, where dropout is not added in fig. 5 (a), and dropout is set to 0.5 in fig. 5 (b), that is, a certain neuron stops working with a probability of 0.5, so that generalization of the model is enhanced, and the model is not too dependent on certain local features.

The generalization capability of the model is enhanced by adding a BN algorithm and a dropout mechanism to the simplified network, the improved network can be named MAVGG-BN, and detailed parameters are shown in a table 1.

TABLE 1 detailed parameter Table of MAVGG-BN model

/>

The following is a data set prepared by acquiring image data in gray scale ultrasound and elastography modes of gastrocnemius muscle of healthy volunteers and amyotrophic patients, preprocessing the data, training the gray scale ultrasound data, the elastance data, and the bimodal data fused with the two respectively using a modified VGG16 model (i.e., MAVGG-BN, structure see fig. 4), and comparing the performance of each method.

The image was initialized to 224 x 224 size and input into MAVGG-BN for training, the first layer convolutional layer size was 3 x 3, the stride size was 1, the batch size was set to 16, and the number of iterations (epoch) was 50. The settings of the experimental parameters are shown in table 2.

Table 2 experimental parameter set table

The diagnosis results of the muscular atrophy pathological examination are compared, and the prediction results of the model are compared with the diagnosis results given by doctors, so that the test results are divided into real classes (TP), false Positive classes (FP), true Negative classes (TN) and False Negative classes (FN). The accuracy, sensitivity and f1-score are calculated by the four conditions to measure the prediction result of the model. The calculation formula is as follows:

The test classification results of the gray scale ultrasonic mode, the ultrasonic elastic imaging mode and the two fused double modes on the MAVGG-BN neural network after training are shown in table 3 and figure 6. According to experimental results, the accuracy of the model classification corresponding to the bimodal mode is 95.56%, which is obviously better than that of the gray ultrasonic mode, and is 9.45%,2.78% higher than that of the gray ultrasonic mode and the elastic mode, and the sensitivity and the index value of the F1 fraction are higher than those of the elastic mode and the gray ultrasonic mode.

TABLE 3 comparison of MAVGG-BN based different imaging modalities

To verify the feasibility and effectiveness of the present application, 120 normal subjects, 80 muscle wasting patients, and 200 samples were collected, and 9 frames were equally spaced from each sample, for a total of 1800 images. 1080 images are randomly extracted from the test set to serve as a training set, the number of images of the verification set and the test set is 180, and the accuracy rate on the test set reaches 95.56%.

Referring to fig. 7, a schematic structural diagram of a muscle disease recognition device based on ultrasound dual-mode imaging is shown, according to one embodiment of the present application.

As shown in fig. 7, the ultrasound dual modality image based muscle disease recognition device 700 may include:

An acquisition module 710 for acquiring a gray image sequence and an elastic image sequence of the detected skeletal muscle of the object to be identified;

the preprocessing module 720 is configured to preprocess the gray image sequence and the elastic image sequence to obtain a gray image and an elastic image;

the identifying module 730 is configured to input the gray level image and the elastic image into a preset muscle disease identifying network, and identify whether the object to be identified has a muscle disease.

Optionally, the preset muscle disease recognition network includes:

a first convolution layer group comprising two convolution layers;

a first max pooling layer;

a second convolution layer group comprising two convolution layers;

a second maximum pooling layer;

a third convolution layer group comprising three convolution layers;

a third maximum pooling layer;

a fourth convolution layer group comprising three convolution layers;

a fourth max pooling layer;

a full connection layer having 512 nodes;

softmax layer.

Optionally, the preset muscle disease recognition network further includes: and a batch normalization algorithm, so that before each layer of input of the preset muscle disease identification network, the output data distribution of the previous layer is normalized by using the batch normalization algorithm.

Optionally, the preset muscle disease recognition network introduces the learnable parameters gamma and beta, calculates the average value and variance of all neurons of the feature map corresponding to all samples, and performs normalization processing on the neurons of the whole feature map.

Optionally, calculating the average value and variance of all neurons of the feature map corresponding to all samples, and performing normalization processing on the neurons of the whole feature map, including:

initial values of the introduction parameters beta, gamma, epsilonObtain the data B= { x before inputting the activation function ₁ ,x ₂ ,…x _n }；

Finding the mean value of the input mu _B Sum of variances

And (3) carrying out normalization processing on the input data:

training parameters β, γ:

output data y after batch normalization operation _i ：

Optionally, a dropout mechanism is included in the preset muscle disease recognition network.

Optionally, the preprocessing module 720 is further configured to:

The embodiment of the method can be implemented by the muscle disease identification device based on the ultrasound bimodal image, and the implementation principle and the technical effect are similar and are not repeated here.

Fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. As shown in fig. 8, a schematic structural diagram of an electronic device 300 suitable for use in implementing embodiments of the present application is shown.

As shown in fig. 8, the electronic device 300 includes a Central Processing Unit (CPU) 301 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 302 or a program loaded from a storage section 308 into a Random Access Memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the device 300 are also stored. The CPU 301, ROM 302, and RAM 303 are connected to each other through a bus 304. An input/output (I/O) interface 305 is also connected to bus 304.

The following components are connected to the I/O interface 305: an input section 306 including a keyboard, a mouse, and the like; an output portion 307 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 308 including a hard disk or the like; and a communication section 309 including a network interface card such as a LAN card, a modem, or the like. The communication section 309 performs communication processing via a network such as the internet. The driver 310 is also connected to the I/O interface 306 as needed. A removable medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed on the drive 310 as needed, so that a computer program read therefrom is installed into the storage section 308 as needed.

In particular, according to embodiments of the present disclosure, the process described above with reference to fig. 1 may be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program comprising program code for performing the ultrasound dual modality image based muscle disease identification method described above. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 309, and/or installed from the removable medium 311.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units or modules described in the embodiments of the present application may be implemented by software, or may be implemented by hardware. The described units or modules may also be provided in a processor. The names of these units or modules do not in some way constitute a limitation of the unit or module itself.

The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a notebook computer, a mobile phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

As another aspect, the present application also provides a storage medium, which may be a storage medium contained in the foregoing apparatus in the foregoing embodiment; or may be a storage medium that exists alone and is not incorporated into the device. The storage medium stores one or more programs for use by one or more processors in performing the ultrasound dual modality image based muscle disorder recognition method described herein.

Storage media, including both permanent and non-permanent, removable and non-removable media, may be implemented in any method or technology for storage of information. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises an element.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.

Claims

1. A method for identifying muscle diseases based on ultrasound bimodal images, the method comprising:

inputting the gray level image and the elastic image into a preset muscle disease recognition network, and recognizing whether the object to be recognized has muscle disease.

2. The method according to claim 1, wherein the preset muscle disease recognition network comprises:

a first convolution layer group comprising two convolution layers;

a first max pooling layer;

a second convolution layer group comprising two convolution layers;

A second maximum pooling layer;

a third convolution layer group comprising three convolution layers;

a third maximum pooling layer;

a fourth convolution layer group comprising three convolution layers;

a fourth max pooling layer;

a full connection layer having 512 nodes;

softmax layer.

3. The method of claim 2, wherein the preset muscle disease recognition network further comprises: and a batch normalization algorithm, so that before each layer of input of the preset muscle disease identification network, the output data distribution of the previous layer is normalized by using the batch normalization algorithm.

4. A method according to claim 3, wherein the preset muscle disease recognition network introduces a learnable parameter γ, β, calculates the mean and variance of all neurons of the feature map corresponding to all samples, and normalizes the whole feature map neurons.

5. The method of claim 4, wherein calculating the mean and variance of all neurons of the feature map corresponding to all samples and normalizing the neurons of the entire feature map comprises:

Finding the mean value of the input mu _B Sum of variances

And (3) carrying out normalization processing on the input data:

training parameters β, γ:

output data y after batch normalization operation _i ：

6. The method according to any one of claims 2-5, wherein a dropout mechanism is included in the preset muscle disease recognition network.

7. The method according to any one of claims 1-5, wherein the preprocessing the gray scale image sequence and the elastic image sequence to obtain a gray scale image and an elastic image comprises:

and subtracting pixel values of the imaging region of the shear wave elastography and the imaging region of the gray scale imaging one to obtain the elastography and the gray scale image.

8. A muscle disorder recognition device based on ultrasound bimodal images, the device comprising:

the preprocessing module is used for preprocessing the gray level image sequence and the elastic image sequence to obtain a gray level image and an elastic image;

The identification module is used for inputting the gray level image and the elastic image into a preset muscle disease identification network and identifying whether the object to be identified has muscle disease or not.

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the ultrasound bimodal image based muscle disease recognition method as claimed in any one of claims 1 to 7 when executing the program.

10. A readable storage medium having stored thereon a computer program, which when executed by a processor implements the ultrasound dual modality image based muscle disorder recognition method of any one of claims 1 to 7.