KR102436035B1 - Apparatus and method for diagnosing skeletal muscle condition from ultrasound images using deep neural network - Google Patents

Abstract

The apparatus for diagnosing the state of skeletal muscle according to the present invention includes a preprocessing unit that processes the ultrasound image according to a predetermined format when an ultrasound image of the skeletal muscle is input, an input layer, at least one pair of alternately repeated convolutional layers; By performing pixel unit operation on the processed ultrasound image through a prediction network including a pooling layer, at least one fully connected layer, and an output layer, it is calculated as a probability whether Sarcopenia is in progress in the skeletal muscle, It includes a predictor for predicting whether sarcopenia progresses according to the calculated probability.

Description

Apparatus and method for diagnosing skeletal muscle condition from ultrasound images using deep neural network}

The present invention relates to a technique for diagnosing a skeletal muscle state, and more particularly, to an apparatus for diagnosing a skeletal muscle state from an ultrasound image using a deep neural network (DNN) and a method therefor.

Sarcopenia was introduced in 1988 by Rosenberg, and its original definition was muscle loss to cecal muscle mass in the elderly. In 2010, the definition of sarcopenia was changed to be a decrease in muscle function as well as low muscle mass. And sarcopenia may be due to differences in individual muscle strength and decreases with age. To evaluate sarcopenia, muscle mass must be measured, and common medical imaging modalities for measuring muscle mass include dual-energy X-ray absorptiometry (DXA) and computed tomography (CT). ), magnetic resonance imaging (MRI), and ultrasonography (USG).

Korean Patent Publication No. 2019-0113089 published on October 08, 2019 (Title: Image processing apparatus employing artificial neural network-based human body shape analysis method for sarcopenia analysis support and image processing method using the same)

An object of the present invention is to provide an apparatus for diagnosing a state of skeletal muscle from an ultrasound image using a deep neural network and a method therefor.

An apparatus for diagnosing the state of skeletal muscle according to a preferred embodiment of the present invention for achieving the above object includes a preprocessing unit that processes the ultrasound image according to a predetermined format when an ultrasound image of the skeletal muscle is input; By performing a pixel unit operation on the processed ultrasound image through a prediction network including an input layer, at least a pair of alternately repeated convolutional and pooling layers, at least one fully connected layer, and an output layer, sarcopenia in the skeletal muscle (Sarcopenia) includes a predictor for calculating whether or not progressing with a probability, and predicting whether sarcopenia progresses according to the calculated probability.

When the prediction unit predicts sarcopenia, the apparatus compares the output value and the gradient of the convolutional layer of the prediction network to distinguish and display the region causing sarcopenia in the ultrasound image from other regions. It further includes a visualization unit for generating.

에 따라 뉴런 중요도 가중치를 산출하고, 수학식

에 따라 상기 시각화 영상을 생성하며, 상기

는 컨벌루션층에서 컨벌루션 연산을 수행하고 활성화함수에 의한 연산을 수행하기 전, 출력의 c번째 값이고, 상기

는 컨벌루션층의 k번째 특징지도의 좌표(i, j)의 값이고, 상기

는

가

에 대해 가지는 경사도(gradient)이고, 상기

는 상기 뉴런 중요도 가중치이고, 상기

는 상기 시각화 영상의 좌표(i, j)의 값인 것을 특징으로 한다. After the visualization unit collects the results of the calculation of the prediction network,

Calculate the neuron importance weight according to the formula

to generate the visualization image according to the

is the c-th value of the output before performing the convolution operation in the convolutional layer and performing the operation by the activation function,

is the value of the coordinates (i, j) of the k-th feature map of the convolutional layer,

Is

go

is the gradient with respect to , and

is the neuron importance weight, and

is the value of the coordinates (i, j) of the visualization image.

When the preprocessor extracts a plurality of radioactive features from the ultrasound image of the skeletal muscle, the apparatus performs an operation on the plurality of radioactive features through an identification network including an input layer, a hidden layer, and an output layer to cause sarcopenia in the skeletal muscle (Sarcopenia) further includes an identification unit for calculating whether or not progressing with a probability, and identifying whether sarcopenia is progressing according to the calculated probability.

In the device, when the preprocessor processes the ultrasound image of the skeletal muscle in a format different from the predetermined format, an input layer, at least a pair of convolutional layers and a pooling layer that are repeated alternately, and at least one complete connection A pixel unit operation is performed on the ultrasound image through a classification network including a layer and an output layer to calculate the probability that the skeletal muscle belongs to each class of muscle echogenicity, and according to the calculated probability, It further includes a classification unit for classifying the grade of the muscle echogenicity.

The output layer of the prediction network prepares an initial model of the prediction network including an output node that performs a softmax operation, and performs initial learning using an image stored in a public database for the initial model as training data, When the initial learning is completed, the last fully connected layer of the prediction network is replaced with an initialized fully connected layer, and the output node performing the softmax operation of the output layer is replaced with an output node performing the sigmoid operation. It further includes a learning unit that prepares a regular model of the prediction network, and performs regular learning on the regular model by using an ultrasound image as training data.

를 이용하여 상기

을 0으로 설정한 후, 상기 예측망의 출력값과 분류 레이블의 차이인 분류 손실이 최소가 되도록 예측망의 파라미터를 수정하는 분류 손실 최적화를 수행하고, 상기

을 0.5로 설정한 후, 상기 예측망의 출력값과 분류 레이블의 차이인 분류 손실과 상기 예측망의 은닉층의 마지막층의 출력값과 은닉 레이블과의 차이인 은닉 손실을 포함하는 복합 손실이 최소가 되도록 예측망의 파라미터를 수정하는 복합 손실 최적화를 수행하고, 상기 E(y, f)는 손실함수이고, 상기 yi는 출력층의 출력값이고, 상기 ei는 출력층의 출력값에 대응하는 분류 레이블이고, 상기 gij는 은닉층의 마지막층의 출력값이고, 상기 cij는 은닉층의 마지막층의 출력값에 대응하는 은닉 레이블이고, 상기 i는 출력층의 노드에 대응하는 인덱스이고, 상기 j는 은닉층의 마지막층의 노드에 대응하는 인덱스이고, 상기

는 하이퍼파라미터인 것을 특징으로 한다. When the learning unit performs the regular learning on the prediction network, a loss function

above using

is set to 0, and classification loss optimization is performed to correct the parameters of the prediction network so that the classification loss, which is the difference between the output value of the prediction network and the classification label, is minimized, and

is set to 0.5, and the composite loss including the classification loss, which is the difference between the output value of the prediction network and the classification label, and the concealment loss, which is the difference between the output value and the hidden label of the last layer of the hidden layer of the prediction network, is predicted to be minimal. Perform complex loss optimization by modifying network parameters, where E(y, f) is a loss function, yi is an output value of an output layer, ei is a classification label corresponding to an output value of an output layer, and gij is a hidden layer is the output value of the last layer of , cij is the hidden label corresponding to the output value of the last layer of the hidden layer, i is the index corresponding to the node of the output layer, and j is the index corresponding to the node of the last layer of the hidden layer, remind

is a hyperparameter.

In an apparatus for diagnosing a state of skeletal muscle according to a preferred embodiment of the present invention for achieving the above object, when an ultrasound image of skeletal muscle is input, the ultrasound image is processed according to two different formats, and the A preprocessor for extracting a plurality of radioactive features from an ultrasound image, and a classification network including a plurality of layers perform a pixel unit operation on an ultrasound image processed in one of the two formats through a classification network including a plurality of layers A classification unit that calculates a grade of echogenicity as a probability, classifies a grade of a muscle echogenicity of the skeletal muscle according to the calculated probability, and a prediction network including a plurality of layers. A prediction unit that calculates whether sarcopenia is progressing in the skeletal muscle by performing pixel unit operation on the ultrasound image processed in the form with a probability, and predicts whether sarcopenia is progressing according to the calculated probability; and a plurality of layers. It includes an identification unit that calculates whether sarcopenia is in progress in the skeletal muscle by performing an operation on the plurality of radioactive features through an identification network including a probability, and identifies whether sarcopenia is in progress according to the calculated probability.

The prediction network includes an input layer, at least a pair of alternately repeated convolutional and pooling layers, at least one fully connected layer and an output layer, and when the prediction unit predicts sarcopenia, the The method further includes a visualization unit that compares an output value and a gradient to generate a visualization image that distinguishes and displays a region causing sarcopenia in the ultrasound image from other regions.

에 따라 뉴런 중요도 가중치를 산출하고, 수학식

에 따라 상기 시각화 영상을 생성하며, 상기

는 컨벌루션층에서 컨벌루션 연산을 수행하고 활성화함수에 의한 연산을 수행하기 전, 출력의 c번째 값이고, 상기

는 컨벌루션층의 k번째 특징지도의 좌표(i, j)의 값이고, 상기

는

가

에 대해 가지는 경사도(gradient)이고, 상기

는 상기 뉴런 중요도 가중치이고, 상기

는 상기 시각화 영상의 좌표(i, j)의 값인 것을 특징으로 한다. After the visualization unit collects the results of the calculation of the prediction network,

Calculate the neuron importance weight according to the formula

to generate the visualization image according to the

is the c-th value of the output before performing the convolution operation in the convolutional layer and performing the operation by the activation function,

is the value of the coordinates (i, j) of the k-th feature map of the convolutional layer,

Is

go

is the gradient with respect to , and

is the neuron importance weight, and

is the value of the coordinates (i, j) of the visualization image.

The apparatus prepares an initial model of the prediction network in which the output node of the output layer of the prediction network performs a softmax operation, and performs initial learning using an image stored in a public database for the initial model as training data, When the initial learning is completed, the last layer of the hidden layer of the classification network and the prediction network is replaced with an initialized fully connected layer, and an output node that performs the softmax operation of the output layer performs a sigmoid operation. It further includes a learning unit that prepares a regular model of the prediction network by replacing it with a node, and performs regular learning on the regular model by using an ultrasound image as training data.

를 이용하여 상기

을 0으로 설정한 후, 상기 예측망의 출력값과 분류 레이블의 차이인 분류 손실이 최소가 되도록 예측망의 파라미터를 수정하는 분류 손실 최적화를 수행하고, 상기

을 0.5로 설정한 후, 상기 예측망의 출력값과 분류 레이블의 차이인 분류 손실과 상기 예측망의 은닉층의 마지막층의 출력값과 은닉 레이블과의 차이인 은닉 손실을 포함하는 복합 손실이 최소가 되도록 예측망의 파라미터를 수정하는 복합 손실 최적화를 수행한다. When the learning unit performs the regular learning on the prediction network, a loss function

above using

is set to 0, and classification loss optimization is performed to correct the parameters of the prediction network so that the classification loss, which is the difference between the output value of the prediction network and the classification label, is minimized, and

is set to 0.5, and the composite loss including the classification loss, which is the difference between the output value of the prediction network and the classification label, and the concealment loss, which is the difference between the output value and the hidden label of the last layer of the hidden layer of the prediction network, is predicted to be minimal. Perform complex loss optimization that modifies network parameters.

는 하이퍼파라미터인 것을 특징으로 한다. where E(y, f) is a loss function, yi is an output value of the output layer, ei is a classification label corresponding to an output value of the output layer, gij is an output value of the last layer of the hidden layer, and cij is the output value of the hidden layer is a hidden label corresponding to the output value of the last layer of , wherein i is an index corresponding to a node of the output layer, and j is an index corresponding to a node of the last layer of the hidden layer, wherein

is a hyperparameter.

In the method for diagnosing the state of skeletal muscle according to a preferred embodiment of the present invention for achieving the above object, when the prediction unit receives an ultrasound image of skeletal muscle, an input layer and at least one pair of convolutional layers alternately repeated and performing a pixel unit operation on the ultrasound image through a prediction network including a pooling layer, at least one fully connected layer, and an output layer to calculate with a probability whether sarcopenia is in progress in the skeletal muscle; and predicting whether sarcopenia progresses according to the additionally calculated probability.

In the method, when the prediction unit predicts sarcopenia, the visualization unit compares the output value and the gradient of the convolutional layer of the prediction network to distinguish and display the region causing sarcopenia in the ultrasound image from other regions The method further includes generating a visualization image.

에 따라 뉴런 중요도 가중치를 산출하는 단계와, 상기 시각화부가 수학식

에 따라 상기 시각화 영상을 생성하는 단계를 포함한다. 여기서, 상기

는 컨벌루션층에서 컨벌루션 연산을 수행하고 활성화함수에 의한 연산을 수행하기 전, 출력의 c번째 값이고, 상기

는 컨벌루션층의 k번째 특징지도의 좌표(i, j)의 값이고, 상기

는 상기

가 상기

에 대해 가지는 경사도(gradient)이고, 상기

는 상기 뉴런 중요도 가중치이고, 상기

는 상기 시각화 영상의 좌표(i, j)의 값인 것을 특징으로 한다. The generating of the visualization image includes: the visualization unit collecting the results of the calculation of the prediction network;

calculating the neuron importance weight according to

and generating the visualization image according to the Here, the

is the c-th value of the output before performing the convolution operation in the convolutional layer and performing the operation by the activation function,

is the value of the coordinates (i, j) of the k-th feature map of the convolutional layer,

is said

is reminded

is the gradient with respect to , and

is the neuron importance weight, and

is the value of the coordinates (i, j) of the visualization image.

In the method, when a plurality of radioactive features extracted from the ultrasound image of the skeletal muscle are input by the identification unit, an operation is performed on the plurality of radioactive features through an identification network including an input layer, a hidden layer, and an output layer. The method further includes: calculating whether or not sarcopenia is progressing with a probability; and identifying whether sarcopenia is progressing according to the probability calculated by the identification unit.

In the method, when the classification unit receives the ultrasound image of the skeletal muscle, the ultrasound image is passed through a classification network including an input layer, at least a pair of alternately repeated convolutional and pooling layers, at least one fully connected layer, and an output layer. calculating the probability that the skeletal muscle belongs to each grade of the muscle echogenicity by performing pixel unit operation on include more

In the method, before calculating with probability whether sarcopenia is in progress in the skeletal muscle, a learning unit provides an initial model of the prediction network including an output node in which the output layer of the prediction network performs a softmax operation and performing initial learning by the learning unit using an image stored in a public database for the initial model as training data, and the learning unit replacing the last fully connected layer of the prediction network with an initialized fully connected layer, preparing a regular model of the prediction network by replacing an output node that performs the softmax operation of the output layer with an output node that performs a sigmoid operation; It further includes the step of performing regular learning using.

를 이용하여 상기

을 0으로 설정한 후, 상기 예측망의 출력값과 분류 레이블의 차이인 분류 손실이 최소가 되도록 예측망의 파라미터를 수정하는 분류 손실 최적화를 수행하는 단계와, 상기 학습부가 상기

을 0.5로 설정한 후, 상기 예측망의 출력값과 분류 레이블의 차이인 분류 손실과 상기 예측망의 은닉층의 마지막층의 출력값과 은닉 레이블과의 차이인 은닉 손실을 포함하는 복합 손실이 최소가 되도록 예측망의 파라미터를 수정하는 복합 손실 최적화를 수행하는 단계를 포함한다. 여기서, 상기 E(y, f)는 손실함수이고, 상기 yi는 출력층의 출력값이고, 상기 ei는 출력층의 출력값에 대응하는 분류 레이블이고, 상기 gij는 은닉층의 마지막층의 출력값이고, 상기 cij는 은닉층의 마지막층의 출력값에 대응하는 은닉 레이블이고, 상기 i는 출력층의 노드에 대응하는 인덱스이고, 상기 j는 은닉층의 마지막층의 노드에 대응하는 인덱스이고, 상기

는 하이퍼파라미터인 것을 특징으로 한다. In the step of performing the regular learning, the learning unit is a loss function

above using

is set to 0, and then performing classification loss optimization by modifying the parameters of the prediction network so that the classification loss, which is the difference between the output value of the prediction network and the classification label, is minimized;

is set to 0.5, and the composite loss including the classification loss, which is the difference between the output value of the prediction network and the classification label, and the concealment loss, which is the difference between the output value and the hidden label of the last layer of the hidden layer of the prediction network, is predicted to be minimal. and performing complex loss optimization to modify parameters of the network. where E(y, f) is a loss function, yi is an output value of the output layer, ei is a classification label corresponding to an output value of the output layer, gij is an output value of the last layer of the hidden layer, and cij is the output value of the hidden layer is a hidden label corresponding to the output value of the last layer of , wherein i is an index corresponding to a node of the output layer, and j is an index corresponding to a node of the last layer of the hidden layer, wherein

is a hyperparameter.

According to the present invention, it is possible to derive various types of information about the state of a muscle captured by an ultrasound image using a plurality of deep neural networks. This is achieved by analyzing the radiomics features extracted from two different types of ultrasound images (B-mode USG and shear-wave elastography (SWE) USG) and ultrasound images (B-mode USG). It includes indirect information such as muscle echogenicity and visualization images as well as intuitive information for diagnosis, such as the probability of acquired sarcopenia. Accordingly, it is possible to diagnose the condition of skeletal muscle in a multifaceted and comprehensive manner.

1 is a diagram for explaining the configuration of a system for diagnosing a state of skeletal muscle from an ultrasound image using a deep neural network according to an embodiment of the present invention.
2 is a view for explaining the detailed configuration of an apparatus for diagnosing a state of skeletal muscle from an ultrasound image using a deep neural network according to an embodiment of the present invention.
3 is a view for explaining a preprocessor of an apparatus for diagnosing a state of a skeletal muscle from an ultrasound image using a deep neural network according to an embodiment of the present invention.
4 to 6 are diagrams for explaining a classification unit of an apparatus for diagnosing a state of skeletal muscle from an ultrasound image using a deep neural network according to an embodiment of the present invention.
7 and 8 are diagrams for explaining a prediction unit of an apparatus for diagnosing a state of skeletal muscle from an ultrasound image using a deep neural network according to an embodiment of the present invention.
9 is a view for explaining an identification unit of an apparatus for diagnosing a state of skeletal muscle from an ultrasound image using a deep neural network according to an embodiment of the present invention.
10 is a flowchart illustrating a method of training an identification network to diagnose a state of a skeletal muscle according to an embodiment of the present invention.
11 is a flowchart illustrating a method for training a classification network and a prediction network to diagnose a state of a skeletal muscle according to an embodiment of the present invention.
12 is a flowchart illustrating a method of performing regular learning on a classification network and a prediction network to diagnose a state of a skeletal muscle according to an embodiment of the present invention.
13 is an example of a vector space for explaining a method of performing regular learning for a classification network and a prediction network to diagnose a state of a skeletal muscle according to an embodiment of the present invention.
14 is a flowchart illustrating a method for diagnosing a state of skeletal muscle from an ultrasound image using a deep neural network according to an embodiment of the present invention.

Prior to the detailed description of the present invention, the terms or words used in the present specification and claims described below should not be construed as being limited to their ordinary or dictionary meanings, and the inventors should develop their own inventions in the best way. It should be interpreted as meaning and concept consistent with the technical idea of the present invention based on the principle that it can be appropriately defined as a concept of a term for explanation. Accordingly, the embodiments described in this specification and the configurations shown in the drawings are only the most preferred embodiments of the present invention, and do not represent all the technical ideas of the present invention, so various equivalents that can replace them at the time of the present application It should be understood that there may be water and variations.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this case, it should be noted that the same components in the accompanying drawings are denoted by the same reference numerals as much as possible. In addition, detailed descriptions of well-known functions and configurations that may obscure the gist of the present invention will be omitted. For the same reason, some components are exaggerated, omitted, or schematically illustrated in the accompanying drawings, and the size of each component does not fully reflect the actual size.

First, a system for diagnosing a state of skeletal muscle from an ultrasound image using a deep neural network according to an embodiment of the present invention will be described. 1 is a diagram for explaining the configuration of a system for diagnosing a state of skeletal muscle from an ultrasound image using a deep neural network according to an embodiment of the present invention. Referring to FIG. 1 , a diagnostic system according to an embodiment of the present invention includes a diagnostic apparatus 10 and a diagnostic imaging apparatus 20 .

The diagnostic imaging apparatus 20 is basically for capturing an ultrasonography (USG) image. In the embodiment of the present invention, the diagnostic imaging apparatus 20 will be described as one device capable of photographing both a B-mode ultrasound image and an SWE ultrasound image, but the diagnostic imaging apparatus 20 is a device for photographing a B-mode ultrasound image. It is divided into a device for capturing an SWE ultrasound image and can be implemented as two devices.

The diagnostic apparatus 10 receives an ultrasound image from the diagnostic imaging apparatus 20, and uses a deep neural network, for example, a deep convolutional neural network (DCNN), a multi-layer perceptron (MLP). By analyzing the different types of ultrasound images (B-mode USG and SWE USG) and from the ultrasound images (B-mode USG) of the species (B-mode USG radiomics features), various types of It is intended to derive information and provide it. The diagnostic apparatus 10 includes an interface unit 11 , an input unit 12 , a display unit 13 , a storage unit 14 , and a control unit 15 .

The interface unit 11 is for receiving an ultrasound image from the diagnostic imaging apparatus 20 . The interface unit 11 may be connected to the diagnostic imaging apparatus 20 through wireless communication, wired communication, etc. as necessary to receive an ultrasound image.

The input unit 12 receives a user's key manipulation for controlling various functions and operations of the diagnostic apparatus 10 , generates an input signal, and transmits the generated input signal to the control unit 15 . The input unit 12 may be a keyboard, a mouse, or the like. The input unit 12 may include a power key for power on/off, a character key, a number key, a direction key, and the like. The function of the input unit 12 may be performed on the display unit 13 when the display unit 13 is implemented as a touch screen, and when all functions can be performed only with the display unit 13, the input unit 12 may be omitted. have.

The display unit 13 visually provides a menu of the diagnostic apparatus 10 , input data, function setting information, and other various information to the user. In particular, according to an embodiment of the present invention, the controller 15 analyzes the input ultrasound image and provides the analysis result. In this case, the display unit 13 may display the analysis result through the display unit 13 under the control of the control unit 15 . The display unit 13 performs a function of outputting a boot screen, a standby screen, a menu screen, and the like of the diagnostic apparatus 10 . The display unit 13 may be formed of a liquid crystal display (LCD), an organic light emitting diode (OLED), an active matrix organic light emitting diode (AMOLED), or the like.

The storage unit 14 serves to store various types of data necessary for the operation of the diagnostic apparatus 10 , applications, and various types of data configured according to the operation of the diagnostic apparatus 10 . The storage unit 14 may largely include a program area and a data area. The program area may store an operating system (OS) for booting and operation of the diagnostic apparatus 10 , an application according to an embodiment of the present invention, and the like. The data area may store various types of data for diagnosis according to an embodiment of the present invention. Various types of data stored in the storage unit 14 may be deleted, changed, or added according to a user's operation.

The controller 15 may control the overall operation of the diagnostic apparatus 10 and signal flow between internal blocks of the diagnostic apparatus 10 , and perform a data processing function of processing data. The control unit 15 may be a central processing unit (CPU), a digital signal processor (DSP), or the like. Also, the controller 15 may further include an image processor or a graphic processing unit (GPU). The specific operation of the control unit 15 will be described in more detail below.

Next, a configuration of an apparatus for diagnosing a state of skeletal muscle from an ultrasound image using a deep neural network according to an embodiment of the present invention will be described in more detail. 2 is a view for explaining the detailed configuration of an apparatus for diagnosing a state of skeletal muscle from an ultrasound image using a deep neural network according to an embodiment of the present invention. 3 is a view for explaining a preprocessor of an apparatus for diagnosing a state of a skeletal muscle from an ultrasound image using a deep neural network according to an embodiment of the present invention. 4 to 6 are diagrams for explaining a classification unit of an apparatus for diagnosing a state of skeletal muscle from an ultrasound image using a deep neural network according to an embodiment of the present invention. 7 and 8 are diagrams for explaining a prediction unit of an apparatus for diagnosing a state of skeletal muscle from an ultrasound image using a deep neural network according to an embodiment of the present invention. And FIG. 9 is a view for explaining an identification unit of an apparatus for diagnosing a state of skeletal muscle from an ultrasound image using a deep neural network according to an embodiment of the present invention.

First, referring to FIG. 2 , the control unit 15 includes a learning unit 100 , a preprocessing unit 200 , a classification unit 300 , a prediction unit 400 , a visualization unit 500 , an identification unit 600 , and a report. part 700 .

The learning unit 100 performs a series of processes for learning a deep neural network (DNN) according to an embodiment of the present invention. Meanwhile, the classification unit 300 , the prediction unit 400 , and the identification unit 600 are deep neural networks and include a classification network 310 , a prediction network 410 , and an identification network 610 , respectively. The learning unit 100 trains the classification network 310 , the prediction network 410 , and the identification network 610 . This learning process will be described in more detail below.

Referring to FIG. 3 , when the preprocessor 200 receives an ultrasound image from the diagnostic imaging apparatus 20 through the interface unit 11 , the classification network 310 of the classification unit 300 and the prediction unit 400 are The received ultrasound image is processed according to a format corresponding to at least one of the prediction network 410 and the identification network 610 of the identification unit 600 , or information is extracted to provide the processed ultrasound image or information. To this end, the preprocessor 200 includes an augmentation unit 210 , an image processing unit 220 , and a feature processing unit 230 .

The augmentation unit 210 is for augmenting the number of learning data according to an embodiment of the present invention. According to an embodiment, when an ultrasound image prepared for learning is input, the augmentation unit 210 performs transformations such as flipping, shifting, shearing, enlargement, reduction, and rotation of the learning data. number can be increased.

When the image processing unit 220 receives an ultrasound image from the diagnostic imaging apparatus 20 through the interface unit 11 , one of the classification network 310 of the classification unit 300 and the prediction network 410 of the prediction unit 400 . The received ultrasound image is processed according to a format corresponding to at least one, and the processed ultrasound image is provided. When an ultrasound image in any one format of the B mode and SWE is input, the image processing unit 220 adjusts the size of the ultrasound image to a size of, for example, 112x112 according to the size of the input layer (GIL) of the classification network 310, The DICOM (Digital Imaging and Communications in Medicine) format may be converted into an image file, for example, a format such as PNG, JPEG, or GIF. As described above, the image processing unit 220 provides the processed ultrasound image to the classification network 310 of the classification unit 300 after processing such as size adjustment and format conversion on the ultrasound image. Also, when an ultrasound image is input, when an ultrasound image in any one of the B mode and SWE is input, the image processing unit 220 cuts out a plurality of images to a predetermined size, for example, a size of 84x84 from the input ultrasound image. It is possible to extract and convert the extracted image from the DICOM format to the format of an image file. Because ultrasound images have different heights and widths, the performance of classifying sarcopenia is significantly degraded when resizing to make an input. Accordingly, a plurality of ultrasound images cut out from the ultrasound image to a predetermined size are reconstructed according to the size of the input layer EIL of the prediction network 410 . Then, the image processing unit 220 provides the plurality of ultrasound images processed as described above to the classification network 410 of the prediction unit 400 .

The feature processing unit 230 is for extracting radioactive features from the ultrasound image. First, the feature processing unit 230 extracts all of the plurality of candidate radiomics features from the B-mode ultrasound image. Then, the feature processing unit 230 from the plurality of extracted candidate radioactive features through the LASSO (least absolute shrinkage and selection operator) algorithm, the Rousseau coefficient (LASSO regression coefficients) in the order of the highest size of the input layer (PIL) of the identification network In accordance with , a predetermined number, eg, 30, radioactive features were derived. These radioactive characteristics are shown in Table 1 below.

Radiomic Feature Coefficients Radiomic Feature Coefficients original_FirstOrder_Skewness 5.991432e-01 wavelet-LH_glszm_SmallAreaHighGrayLevelEmphasis 1.496917e-01 wavelet-LL_FirstOrder_Skewness 5.418939e-01 original_gldm_LargeDependenceLowGrayLevelEmphasis 1.229492e-01 wavelet-LL_glszm_GrayLevelVariance 5.129526e-01 original_glcm_Autocorrelation 1.110921e-01 original_glszm_GrayLevelVariance 4.527734e-01 wavelet-LH_glszm_GrayLevelVariance 1.080109e-01 original_ngtdm_Complexity 4.306312e-01 wavelet-LH_glrlm_LongRunEmphasis 1.019890e-01 original_glszm_SmallAreaHighGrayLevelEmphasis 3.301892e-01 wavelet-HL_glcm_Autocorrelation 9.387824e-02 wavelet-LH_FirstOrder_Skewness 2.804855e-01 wavelet-HH_gldm_LargeDependenceEmphasis 9.138463e-02 original_glrlm_RunVariance 2.628373e-01 wavelet-HL_glrlm_HighGrayLevelRunEmphasis 7.951094e-02 wavelet-LL_gldm_SmallDependenceHighGrayLevelEmphasis 2.403814e-01 wavelet-LL_glrlm_GrayLevelVariance 7.916654e-02 wavelet-LL_FirstOrder_RobustMeanAbsoluteDeviation 2.086700e-01 wavelet-LL_FirstOrder_InterquartileRange 7.887526e-02 wavelet-HH_glszm_SmallAreaHighGrayLevelEmphasis 1.927658e-01 wavelet-LL_gldm_LargeDependenceEmphasis 7.362614e-02 wavelet-LH_glrlm_RunVariance 1.841436e-01 wavelet-HH_gldm_LargeDependenceLowGrayLevelEmphasis 7.329982e-02 wavelet-LH_gldm_LargeDependenceEmphasis 1.569288e-01 original_glrlm_LongRunEmphasis 6.850687e-02 wavelet-LH_glrlm_LongRunLowGrayLevelEmphasis 1.565633e-01 original_glrlm_ShortRunHighGrayLevelEmphasis 6.227870e-02 wavelet-HL_gldm_LargeDependenceLowGrayLevelEmphasis 1.552139e-01 original_FirstOrder_InterquartileRange 6.074799e-02

Next, the classification unit 300 will be described with reference to FIGS. 4 to 6 . As described above, the classification unit 300 includes a classification network 310 , and the classification network 310 may be a deep convolutional neural network (DCNN) among deep neural networks. The classification network 310 of the present invention includes a plurality of layers. That is, the classification network 310 includes an input layer (GIL), at least a pair of alternately repeated convolutional layers (GCL) and a pooling layer (GPL), at least one fully connected layer (GFL1, GFL2), and an output layer (GOL). includes When the classification unit 300 receives the ultrasound image of the skeletal muscle from the image processing unit 220 of the pre-processing unit 200, the skeletal muscle generates a muscle echo by performing a pixel unit operation on the ultrasound image through the classification network 310. The probability of belonging to each grade of the degree (Muscle echogenicity) is calculated, and the grade of the muscle echogenicity of the skeletal muscle is classified according to the calculated probability. According to an embodiment, as shown in FIG. 4 , the classification network 310 sequentially includes an input layer (GIL), a convolution layer (GCL), a pooling layer (GPL), a first fully connected layer (GFL1), It includes a second fully connected layer (GFL2) and an output layer (GOL).

The convolutional layer (GCL) and the pooling layer (GPL) are composed of at least one feature map (FM). The feature image (FM) is generated as a result of performing an operation in which weights are applied to the values of the previous layer. These weights are applied through a filter or kernel (W). In the embodiment of the present invention, the first filter W1 is used for the convolution operation of the convolutional layer GCL, and the second filter W2 is used for the pooling operation of the pooling layer PL.

When an ultrasound image is input from the preprocessor 100 to the input layer GIL, the convolutional layer GCL performs a convolution operation and activation on the ultrasound image of the input layer GIL using the first filter W1. At least one first feature image FM1 is derived by performing an operation by a function.

Subsequently, the pooling layer GPL performs a pooling or sub-sampling operation using the second filter W2 on at least one first feature image FM1 of the convolutional layer GCL to obtain at least one A second feature image FM2 is derived.

The first complete connection layer GFL1 includes a plurality of first nodes f1 to fm. The plurality of first nodes f1 to fm of the first finalized connection layer GFL1 is formed by calculating the activation function with respect to at least one second feature image FM2 of the pooling layer GPL. Calculate the value.

The second complete connection layer GFL2 includes a plurality of second nodes g1 to gn. The plurality of second nodes f1 to fm of the second fully connected layer GFL2 has a plurality of second node values through an operation using an activation function on the plurality of first node values of the first fully connected layer GFL1. to calculate

The activation functions used in the above-described convolutional layer (GCL) and the first and second complete connection layers (GFL1, GFL2) are sigmoid, hyperbolic tangent (tanh), and exponential linear unit (ELU). , ReLU (Rectified Linear Unit), Leakly ReLU, Maxout, Minout, Softmax, etc. may be exemplified. Any one of these activation functions can be selected and used.

5 and 6 , the output layer GOL includes two output nodes H and L. Referring to FIG. 5 , each of the plurality of second nodes g1 to gn of the second complete connection layer GFL2 is a channel (indicated by a dotted line) having a weight W, and an output node ( H, L) are connected. In other words, a weight is applied to the plurality of second node values of the plurality of second nodes g1 to gn and respectively input to the output nodes H and L. Accordingly, the output nodes H and L of the output layer GOL are converted to the input layer GIL through an operation by an activation function for a plurality of second node values to which the weight of the second complete connection layer GFL2 is applied. The probability that the skeletal muscle of the input ultrasound image belongs to each grade of muscle echogenicity is calculated.

Meanwhile, a node N according to an embodiment of the present invention is illustrated in FIG. 6 . This node N will be described assuming any one of the output nodes H and L of the classification network 310 , but the description of this node N will be described with the first node f1 included in the classification network 310 . to fm), the second nodes f1 to fm, and the output nodes (H, L) and other deep neural networks 410 and 610 of the present invention may be commonly applied to all nodes included.

The node N is the input signal x=[x1, x2, ... , xn] with weight w=[w1, w2, … , wn], and take a function F on the result. Here, the function F is an activation function. At this time, even when the input is the same, the output has a different value according to the weight W. That is, the output of each node is as shown in Equation 1 below.

의 값이 임계치 보다 작을 때 해당 노드가 활성화되지 않도록 하는 역할을 한다. 예를 들면, 노드(N)의 이전 계층의 노드가 3개라고 가정한다. 이에 따라, 해당 노드에 대해 3개의 입력(n=3) X1, X2, X3과 3개의 가중치 W1, W2, W3이 존재한다. 노드(N)는 3개의 입력 X1, X2, X3에 대응하는 가중치 W1, W2, W3을 곱한 값을 입력받고, 모두 합산한 후, 합산된 값을 전달 함수에 대입하여 출력을 산출한다. 구체적으로, 입력 [X1, X2, X3] = 0.5, -0.3, 0이라고 가정하고, 가중치 w=[W1, W2, W3] = 4, 5, 2라고 가정한다. 또한, 설명의 편의를 위하여 전달 함수는 'sgn()'이라고 가정하면, 다음과 같이 출력값이 산출된다. Among the unexplained parameters, b is a threshold, and this threshold is

It serves to prevent the corresponding node from being activated when the value of is less than the threshold. For example, it is assumed that there are three nodes in the previous layer of the node N. Accordingly, there are three inputs (n=3) X1, X2, X3 and three weights W1, W2, and W3 for the corresponding node. The node N receives a value obtained by multiplying the weights W1, W2, and W3 corresponding to the three inputs X1, X2, and X3, sums them all up, and substitutes the summed value into the transfer function to calculate an output. Specifically, it is assumed that the input [X1, X2, X3] = 0.5, -0.3, 0, and the weight w = [W1, W2, W3] = 4, 5, 2. Also, for convenience of explanation, assuming that the transfer function is 'sgn()', the output value is calculated as follows.

x1 × w1 = 0.5 × 4 = 2

x2 × w2 = - 0.3 × 5 = -1.5

x3 × w3 = 0 × 2 = 0

2 + (-1.5) + 0 = 0.5

sgn(0.5) = 1

In this way, any one node of the deep neural network (310, 410, 610) of the present invention receives a value obtained by applying a weight to the node value of the previous layer, sums them up, performs an operation according to the activation function, and displays these results It is passed as input to the next layer.

Referring again to FIG. 5 , each of the two output nodes H and L of the output layer GOL corresponds to a high grade and a low grade of muscle echogenicity. That is, the first output node H corresponds to a high grade, and the second output node L corresponds to a low grade. Accordingly, the output value of the first output node (H) is the probability that the skeletal muscle of the ultrasound image input to the input layer (GIL) belongs to a high grade of muscle echogenicity, and the output value of the second output node (L) denotes the probability that the skeletal muscle of the ultrasound image input to the input layer (GIL) belongs to the low grade of the muscle echogenicity.

Accordingly, referring to FIGS. 5 and 6 , each of the output nodes H and L has a plurality of second node values x=[x1, x2, ... , xn] with weight w=[w1, w2, … , wn] is applied, and the output value is calculated by taking the activation function F on the result. For example, if the output values of the first and second output nodes H and L are 0.789 and 0.211, respectively, the probability that the skeletal muscle belongs to the high grade of the muscle echogenicity is 79%, and the low grade This indicates that there is a 21% chance of belonging. As such, when the classification network 310 outputs probabilities (0.789, 0.211), the classification unit 300 determines that the corresponding skeletal muscle belongs to a high grade of muscle echogenicity according to these probabilities (0.789, 0.211). can judge In addition, the classification unit 300 provides a probability that is the output of the classification network 310 and a determination according to the probability, that is, “muscle echogenicity: high grade” to the report unit 700 .

Next, the prediction unit 400 will be described with reference to FIGS. 7 and 8 . As described above, the prediction unit 400 includes a prediction network 410 , and the prediction network 410 may be a deep convolutional neural network (DCNN) among deep neural networks. The prediction network 410 of the present invention includes a plurality of layers. That is, the prediction network 410 includes an input layer (EIL), at least a pair of alternately repeated convolutional layers (ECL) and a pooling layer (EPL), at least one fully connected layer (EFL1, EFL2), and an output layer (EOL). includes When an ultrasound image of skeletal muscle is received from the image processing unit 220 of the pre-processing unit 200, a pixel unit operation is performed on the ultrasound image through the prediction network 410 to cause sarcopenia in the skeletal muscle of the ultrasound image. It is calculated by probability, and whether or not sarcopenia progresses is predicted according to the calculated probability.

According to an embodiment, the prediction network 410 sequentially includes an input layer (EIL), a convolution layer (ECL), a pooling layer (EPL), a first fully connected layer (EFL1), It includes a second fully connected layer (EFL2) and an output layer (EOL).

The convolutional layer (ECL) and the pooling layer (EPL) are composed of at least one feature map (FM). The feature image (FM) is generated as a result of performing an operation in which weights are applied to the values of the previous layer. These weights are applied through a filter or kernel (W). In the embodiment of the present invention, the first filter W1 is used for the convolution operation of the convolutional layer ECL, and the second filter W2 is used for the pooling operation of the pooling layer PL.

When an ultrasound image is input from the preprocessor 100 to the input layer EIL, the convolutional operation and activation of the convolutional layer ECL using the first filter W1 on the ultrasound image of the input layer EIL At least one first feature image FM1 is derived by performing an operation by a function.

Subsequently, the pooling layer EPL performs a pooling or sub-sampling operation using the second filter W2 on at least one first feature image FM1 of the convolution layer ECL to obtain at least one A second feature image FM2 is derived.

The first complete connection layer EFL1 includes a plurality of first nodes f1 to fm. The plurality of first nodes f1 to fm of the first complete connection layer EFL1 is formed by calculating the activation function with respect to at least one second feature image FM2 of the pooling layer EPL. Calculate the value.

The second complete connection layer EFL2 includes a plurality of second nodes g1 to gn. The plurality of second nodes f1 to fm of the second fully connected layer EFL2 has a plurality of second node values through an operation using an activation function on the plurality of first node values of the first fully connected layer EFL1. to calculate

The activation functions used in the above-described convolutional layer (ECL) and the first and second complete connection layers (EFL1, EFL2) are sigmoid, hyperbolic tangent (tanh), and exponential linear unit (ELU). , ReLU (Rectified Linear Unit), Leakly ReLU, Maxout, Minout, Softmax, etc. may be exemplified. Any one of these activation functions can be selected and used.

7 and 8 , the output layer EOL includes two output nodes P and N. As shown in FIG. 8 , each of the plurality of second nodes g1 to gn of the second complete connection layer EFL2 outputs the output layer OL to a channel (indicated by a dotted line) having a weight W: It is connected to the nodes (P, N). In other words, a weight is applied to the plurality of second node values of the plurality of second nodes g1 to gn and respectively input to the output nodes P and N. Accordingly, the output nodes P and N of the output layer EOL are converted to the input layer EIL through an activation function operation for a plurality of second node values to which the weight of the second finalized connection layer EFL2 is applied. Whether sarcopenia is progressing in the skeletal muscle of the input ultrasound image is calculated as a probability.

Referring to FIG. 8 , each of the two output nodes P and N of the output layer EOL corresponds to positive and negative for sarcopenia. That is, the first output node P corresponds to positive, and the second output node N corresponds to negative. Accordingly, the output value of the first output node P is a probability indicating that there is sarcopenia in the skeletal muscle of the ultrasound image input to the input layer EIL, and the output value of the second output node L is the input layer EIL. It is the probability that the skeletal muscle of the ultrasound image input as , indicates that there is no sarcopenia.

Accordingly, with reference to FIGS. 7 and 8 , each of the output nodes P and N has a weight w=[w1, w2, ... , wn] is applied, and the output value is calculated by taking the activation function F on the result. For example, if the output values of the first and second output nodes P and N are 0.089 and 0.911, respectively, the probability indicating that the skeletal muscle has sarcopenia is 9%, and the probability that the corresponding skeletal muscle does not have sarcopenia is 91%. As such, when the prediction network 410 outputs probabilities (0.089, 0.911), the prediction unit 400 may determine that there is no sarcopenia in the corresponding skeletal muscle according to these probabilities (0.089, 0.911). In addition, the prediction unit 400 provides a probability that is an output of the prediction network 410 and a determination according to the probability, that is, “sarcopenia: voice” to the report unit 700 .

Meanwhile, referring to FIGS. 2 and 7 , the visualization unit 500 is for generating a visualization image using the prediction network 410 . The visualization image refers to an image in which a region causing sarcopenia is derived from an input ultrasound image and displayed to be distinguished from other regions. Such an image may be a CAM (Class Activation Map), Grad-CAM, or the like. Accordingly, when the prediction unit 400 predicts sarcopenia according to the probability that it is the output of the prediction network 410 , the output value and the gradient of the convolutional layer (ECL) of the prediction network 410 are compared in the ultrasound image. A region causing sarcopenia is derived, and the region causing sarcopenia in the ultrasound image is displayed separately from other regions. Thereby, a visualization image is generated. When the region causing sarcopenia is displayed separately from other regions, it can be displayed using the type and brightness of the color. This method will be described in more detail.

The visualization unit 500 first collects the results of the operation of the prediction network 410 and then calculates a neuron importance weight according to Equation 2 below.

는 뉴런 중요도 가중치이고,

는 컨벌루션층(ECL)에서 컨벌루션 연산을 수행하고 활성화함수에 의한 연산을 수행하기 전, 출력의 c번째 값이고,

는 컨벌루션층(ECL)의 k번째 특징지도의 좌표(i, j)의 값을 의미한다. 특히,

는

가

에 대해 가지는 경사도(gradient)를 의미한다. here,

is the neuron importance weight,

is the c-th value of the output before performing the convolution operation in the convolutional layer (ECL) and performing the operation by the activation function,

is the value of the coordinates (i, j) of the k-th feature map of the convolutional layer (ECL). Especially,

Is

go

It means the gradient with respect to .

Next, the visualization unit 500 generates a visualization image according to Equation 3 below.

는 시각화 영상의 좌표(i, j)의 값을 나타낸다. 또한,

는 뉴런 중요도 가중치이고,

는 컨벌루션층(ECL)의 k번째 특징영상(FM)의 좌표(i, j)의 값을 의미한다. 이와 같이, 시각화부(500)는

를 산출한 후,

값에 따라 색을 적용함으로써, 시각화 영상을 생성할 수 있다. here,

denotes the value of the coordinates (i, j) of the visualization image. In addition,

is the neuron importance weight,

is the value of the coordinates (i, j) of the k-th feature image FM of the convolutional layer ECL. In this way, the visualization unit 500

After calculating

By applying a color according to a value, a visualization image can be generated.

The identification unit 600 will be described with reference to FIG. 9 . As described above, the identification unit 600 includes an identification network 610, and the identification network 610 may be a multilayer perceptron (MLP) among deep neural networks. The identification network 610 includes a plurality of layers (PIL, PHL, POL). The plurality of layers includes an input layer (PIL), a plurality of hidden layers (PHL: HL1 to HLk), and an output layer (POL).

In addition, each of the plurality of layers (PIL, PHL, and POL) includes a plurality of nodes. For example, as illustrated, the input layer PIL may include n input nodes i1 to in, and the output layer OL may include two output nodes P and N. In addition, among the hidden layers (PHL), the first hidden layer (PHL1) includes a number of nodes (h11 to h1a), the second hidden layer (PHL2) includes b nodes (h21 to h2b), and the kth hidden layer (h21 to h2b) The layer PHLk may include c nodes hk1 to hkc.

All of the plurality of nodes of the plurality of layers have operations. In particular, a plurality of nodes of different layers are connected by a channel (indicated by a dotted line) having a weight (W). In other words, the calculation result of one node is weighted and becomes the input of the next layer node. This connection relationship has been described above with reference to FIG. 6 . That is, any one node of any one layer of the identification network 610 receives a value obtained by applying a weight to the input from the node of the previous layer, sums them up to obtain a transfer function, and transfers the result to the input of the next layer. do. Accordingly, when a plurality of radiomics features extracted from the ultrasound image of skeletal muscle from the feature processing unit 230 of the preprocessor 200 are input to the input layer PIL of the identification network 610, the identification network ( 610) calculates whether sarcopenia is in progress in skeletal muscle with a probability by performing a plurality of calculations in which weights of a plurality of layers (PIL, PHL, POL) are applied to a plurality of input radioactive features, According to the calculated probability, it is identified whether sarcopenia is progressing.

More specifically, when each of a plurality of radiomics features is input to a plurality of input nodes i1 to in of the input layer PIL of the identification network 610, a plurality of radiomics features of the first hidden layer PHL1 Each of the first hidden nodes (h11 to h1a) receives a value in which a weight is applied to each of a plurality of radiomics features of a plurality of input nodes (i1 to in) (indicated by a dotted line), and sums all the input values Then, a plurality of first hidden node values are calculated by performing an operation according to the activation function on the summed values. Then, each of the plurality of second hidden nodes h21 to h2b of the second hidden layer PHL2 receives a value in which a weight is applied to each of the plurality of first hidden node values of the plurality of first hidden nodes h11 to h1a. (indicated by a dotted line), all input values are summed, and a plurality of second hidden node values are calculated by performing an operation according to the activation function on the summed values. In this way, a weight is applied to a previous node value and transmitted in the hidden layer (PHL), and a current node value is calculated through calculation. By repeating this process, a plurality of k-th hidden node values of the plurality of k-th hidden nodes hk1 to hkc of the k-th hidden layer PHLk may be calculated.

Accordingly, referring to FIG. 9 , each of the output nodes P and N has a weight w = [ w1, w2, … , wc×2] is received as input (indicated by a dotted line), and after summing all the input values, an operation is performed on the summed values according to the activation function to calculate an output value. Each of the two output nodes P and N of the output layer POL corresponds to positive and negative for Sarcopenia. That is, the first output node P corresponds to positive, and the second output node N corresponds to negative. Accordingly, the output value of the first output node (P) is a probability indicating that sarcopenia is present in skeletal muscle having radiomics features input to the input layer (PIL), and the output value of the second output node (N) is It is the probability that skeletal muscle with radioactive features input into the input layer (PIL) indicates the absence of sarcopenia.

For example, if the output values of the first and second output nodes P and N are 0.089 and 0.911, respectively, the probability indicating that the skeletal muscle has sarcopenia is 9%, and the probability that the corresponding skeletal muscle does not have sarcopenia is 91%. As such, when the identification network 610 outputs probabilities (0.089, 0.911), the identification unit 600 may determine that there is no sarcopenia in the corresponding skeletal muscle according to these probabilities (0.089, 0.911). And the identification unit 600 provides the output of the identification network 610 with a probability and a determination according to the probability, for example, “sarcopenia: voice” to the report unit 700 .

Referring back to FIG. 2 , the report unit 700 receives information from each of the classification unit 300 , the prediction unit 400 , the visualization unit 500 , and the identification unit 600 , and synthesizes the received information and displays the information. It can be indicated through (13). Specifically, the classification unit 300 determines the probability that the skeletal muscle belongs to a high grade and a low grade of the muscle echogenicity, that is, a determination according to the probability, that is, "muscle echogenicity: high level ( High grade)" or "muscle echogenicity: low grade" is provided to the report unit 700 . The prediction unit 400 provides the report unit 700 with a probability of having sarcopenia in the skeletal muscle and a probability of not having sarcopenia, and a determination based on the probability, "sarcopenia: positive" or "sarcopenia: negative". The visualization unit 500 provides a visualization image to the report unit 700 . In addition, the identification unit 600 provides the report unit 700 with a probability that skeletal muscle has sarcopenia and a probability that there is no sarcopenia, and a determination according to the probability, “sarcopenia: positive” or “sarcopenia: negative” to the report unit 700 . can

Next, a method for diagnosing a state of skeletal muscle from an ultrasound image using a deep neural network according to an embodiment of the present invention will be described. In order to diagnose the state of the skeletal muscle, learning of the deep neural networks 310 , 410 , and 610 is required. These methods will be described. First, a method for learning the identification network 610 will be described. 10 is a flowchart illustrating a method of training an identification network to diagnose a state of a skeletal muscle according to an embodiment of the present invention.

Referring to FIG. 10 , the learning unit 100 prepares a radioactive feature extracted from an ultrasound image of a patient whose sarcopenia is known in step S110 as learning data, that is, a radioactive feature for learning. Then, the learning unit 100 sets a label according to whether or not sarcopenia known to the radioactive characteristic for learning in step S120. At this time, a label [1, 0] is given to the radioactive feature extracted from the ultrasound image of a patient with sarcopenia in correspondence with the output node (P, N). On the other hand, radioactive features extracted from ultrasound images of patients without sarcopenia are assigned labels [0, 1] corresponding to the output nodes (P, N).

Next, the learning unit 100 inputs the radioactive characteristic for learning corresponding to the step S130 into the identification network 610 . Then, the identification network 610 performs a plurality of calculations in which the weights of a plurality of layers (PIL, PHL, POL) are applied to the radioactive features for learning input in step S140 to determine whether sarcopenia is in progress in the skeletal muscle. An output value indicating whether or not there is a probability is calculated.

Then, the learning unit 100 uses the loss function in step S150 to minimize the binary cross entropy loss that is the difference between the output value and the label. You can perform optimizations to correct it.

Steps S120 to S150 described above are repeatedly performed using a plurality of different learning data, and such repetition may be performed until a desired accuracy is reached through an evaluation index.

Next, it is a flowchart for explaining a method of training a classification network and a prediction network to diagnose a state of a skeletal muscle according to an embodiment of the present invention. 11 is a flowchart illustrating a method for training a classification network and a prediction network to diagnose a state of a skeletal muscle according to an embodiment of the present invention.

Referring to FIG. 11 , the learning unit 100 prepares initial models of a classification network 310 and a prediction network 410 that are deep neural networks in step S210 . The initial model included an input layer (GIL/EIL), at least one pair of alternating convolutional layers (GCL/ECL) and a pooling layer (GPL/EPL), at least one fully connected layer (GFL/EFL) and an output layer (GOL). /EOL). In particular, in the initial model, a softmax is selected as an activation function of an output node of the output layer (GOL/EOL). That is, the initial model includes an output node whose activation function is a softmax.

Then, the learning unit 100 performs initial learning using the image stored in the public database for the initial model as training data in step S220. Here, the image stored in the public database may use an image of the database of ImageNet. Initial learning uses the image stored in the public database as training data to set a label, calculates the output value of the initial model for the training data, and according to the loss function, a loss that is the difference between the output value of the initial model and the label, e.g., binary It is a procedure to modify the parameters of the initial model so that the cross-entropy loss is minimized.

When the initial learning is completed, the learning unit 100 replaces the last fully connected layer, which is the last layer of the hidden layer of the prediction network of the initial model in step S230, with an initialized fully connected layer (eg, GFL2/EFL2), and The initial model is converted into a regular model by replacing the output node that performs the softmax operation with the output node that performs the sigmoid operation.

Next, the learning unit 100 performs regular learning using the ultrasound image as training data for the regular model in step S240 . In regular learning, an ultrasound image for training is generated from an ultrasound image, that is, a B-mode ultrasound image or a SWE ultrasound image through the preprocessor 100, a label is set for the ultrasound image for training, and the normal model is applied to the ultrasound image for training. It is a procedure of calculating the output value and modifying the parameters of the regular model so that the loss, which is the difference between the output value of the initial model and the label, is minimized according to the loss function.

Then, the learning method for the aforementioned regular model will be described in more detail. 12 is a flowchart illustrating a method of performing regular learning on a classification network and a prediction network to diagnose a state of a skeletal muscle according to an embodiment of the present invention. 13 is an example of a vector space for explaining a method of performing regular learning for a classification network and a prediction network to diagnose a state of a skeletal muscle according to an embodiment of the present invention.

Referring to FIG. 12 , the learning unit 100 prepares learning data, that is, an ultrasound image for learning in step S310 . In the case of the classification network 310 , the ultrasound image for learning is an ultrasound image of a patient whose level of muscle echogenicity is known, and in the case of the prediction network 410 , it is an ultrasound image of a patient whose sarcopenia is known. When such an ultrasound image for learning is prepared, the learning unit 100 performs learning using a loss function as shown in Equation 4 below.

는 하이퍼파라미터를 나타낸다. In Equation 4, E(y, f) represents a loss function. yi is an output value of the output layer, and ei is a classification label corresponding to the output value of the output layer. gij is the output value of the last layer of the hidden layer, and cij is the hidden label corresponding to the output value of the last layer of the hidden layer. i is the index corresponding to the node of the output layer, and j is the index corresponding to the node of the last layer of the hidden layer. and

is a hyperparameter.

는 0으로 설정한다. 이에 따라, 분류 손실을 최적화하는 학습에서 수학식 4의 손실함수는 다음의 수학식 5와 같다. The learning unit 100 performs settings for optimizing classification loss in step S320. At this time, the learning unit 100 sets a hyperparameter and a classification label. In this step S320, the learning unit 100 is a hyperparameter

is set to 0. Accordingly, in the learning to optimize the classification loss, the loss function of Equation 4 is the following Equation 5.

Also, in step S320, the learning unit 100 sets a classification label. In the case of the classification network 310, a classification label [1, 0] is assigned to an ultrasound image of a patient known to have a high level of muscle echogenicity in response to the output nodes (H, L), and ultrasound of a patient known as a low level A classification label [0, 1] is given to an image corresponding to the output node (H, L). In addition, in the case of the prediction network 410, a classification label [1, 0] is assigned to an ultrasound image of a patient known to have sarcopenia in response to the output nodes (P, N), and A classification label [0, 1] is given to the ultrasound image corresponding to the output nodes (P, N).

When the hyperparameter and the classification label are set as described above, the learning unit 100 performs learning to optimize the classification loss in step S330 . At this time, the learning unit 100 inputs the training ultrasound image to the regular model of the classification network 310 and the identification network 410 . Then, the regular model calculates an output value through a plurality of operations in which a plurality of inter-layer weights are applied to the training ultrasound image. Then, the learning unit 100 is normalized so that the classification loss of the loss function of Equation 5 indicating the difference between the previously set classification label and the output value of the output layer is minimized by using the loss function as in Equation 5 in which the hyperparameter is set to 0. A classification loss optimization that modifies the parameters of the model is performed. This classification loss optimization is repeatedly performed using a plurality of different ultrasound images for training, and such repetition may be performed until a desired accuracy is reached through an evaluation index.

When the learning according to the classification loss optimization as described above is completed, the learning unit 100 selects a representative vector from among the hidden vectors for each classification target class of the regular model in step S340 . The method of selecting a representative vector is as follows. By re-inputting the training ultrasound image used above (S330) into the regular model with classification loss optimization, output values of the last layers (eg, GFL2, EFL2) of the hidden layer of the regular model are derived. 5 and 8, the output of the last layer of the hidden layer (eg, GFL2, EFL2) is the hidden vector G[g1, g2, g3, ... , gn]. The learning unit 100 embeds a plurality of hidden vectors in a predetermined vector space. 13A shows an example in which a plurality of hidden vectors are embedded in a predetermined vector space. Basically, some degree of classification is made for each class of a plurality of hidden vectors by the classification line Div in the vector space by classification loss optimization. Here, in the case of the classification network 310, the class to be classified includes a class having a high level (H) and a class having a low level (L), and in the case of the prediction network 410, sarcopenia positive ( It includes a class that is P) and a class that is negative (N). That is, in (A) of FIG. 13 , the first class is a class having a high level (H) of muscle echogenicity in the case of the classification network 310, and in the case of the prediction network 410, sarcopenia is positive (P). ) can be a class. In addition, in FIG. 13A , the second class is a class with a low level (L) of muscle echogenicity, and in the case of the prediction network 410, a class in which sarcopenia is negative (N). The learning unit 100 may select a representative vector for each class from among a plurality of hidden vectors embedded in such a vector space. As the representative vector, as shown in (A) of 13, it is preferable to select an intermediate value among the hidden vectors in the same class.

는 0.5로 설정한다. 이에 따라, 분류 손실을 최적화하는 학습에서 수학식 4의 손실함수는 다음의 수학식 6과 같다. Next, the learning unit 100 performs settings for optimizing the complex loss in step S350 . At this time, the learning unit 100 sets a hyperparameter, a classification label, and a hidden label. In this step S350, the learning unit 100 is a hyperparameter

is set to 0.5. Accordingly, in the learning to optimize the classification loss, the loss function of Equation 4 is as shown in Equation 6 below.

Also, in step S350, the learning unit 100 sets a classification label and a hidden label. As for the classification label setting, in the case of the classification network 310, a classification label [1, 0] is given to the ultrasound image of a patient known to have a high level of muscle echogenicity in response to the output nodes (H, L), A classification label [0, 1] is assigned to an ultrasound image of a patient known as a low level corresponding to the output nodes (H, L). In addition, in the case of the prediction network 410, a classification label [1, 0] is assigned to an ultrasound image of a patient known to have sarcopenia in response to the output nodes (P, N), and A classification label [0, 1] is given to the ultrasound image corresponding to the output nodes (P, N).

In particular, the setting of the hidden label uses the representative vector selected above (S340). Accordingly, a representative vector for each class is set as a hidden label.

When the hyperparameter, classification label, and hidden label are set as described above, when the parameter is set as described above, the learning unit 100 performs learning to optimize the complex loss in step S360. At this time, the learning unit 100 inputs the training ultrasound image to the regular model of the classification network 310 and the identification network 410 . Then, the regular model calculates a hidden vector and an output value through a plurality of operations in which a plurality of inter-layer weights are applied to the training ultrasound image. Then, the learning unit 100 performs complex loss optimization by modifying the parameters of the regular model so that the complex loss of the loss function of Equation 6 is minimized by using the loss function as in Equation 6 in which the hyperparameter is set to 0.5. . The complex loss includes a classification loss indicating a difference between a classification label and an output value of an output layer, and a hiding loss indicating a difference between a hidden vector and a representative vector. The complex loss optimization is repeatedly performed using a plurality of different ultrasound images for training, and such repetition may be performed until a desired accuracy is reached through an evaluation index.

When the learning by the complex loss is completed, a plurality of hidden vectors in the vector space may be classified as shown in FIG. 13B. That is, the plurality of hidden vectors are moved toward the representative vector in the vector space. Accordingly, it can be seen that the classification for each class becomes clearer. That is, the classification performance of the classification network 310 and the prediction network 410 may be improved by learning by complex loss.

Next, a method of diagnosing the state of skeletal muscle using the classification network 310 , the prediction network 410 , and the identification network 610 that has been trained in the manner described above will be described. 14 is a flowchart illustrating a method for diagnosing a state of skeletal muscle from an ultrasound image using a deep neural network according to an embodiment of the present invention.

Referring to FIG. 14 , the preprocessor 200 of the diagnostic apparatus 10 may receive an ultrasound image from the diagnostic imaging apparatus 20 through the interface 11 in operation S410 . The input ultrasound image may be any one of a B-mode ultrasound image and an SWE ultrasound image.

Then, the preprocessor 200 performs preprocessing on the ultrasound image in step S420 . In this case, the preprocessor 200 may adjust the size of the ultrasound image or cut it to a predetermined size, and then convert it into an image file format. Also, the preprocessor 200 may extract radial features from the ultrasound image. When the preprocessing is completed, the preprocessing unit 200 provides the preprocessed ultrasound image or radiomics features to the classifying unit 300 , the predicting unit 400 , and the identifying unit 600 .

The classification unit 300 , the prediction unit 400 , the visualization unit 500 , and the identification unit 600 perform evaluation on the skeletal muscle in step S430 to derive an evaluation result. That is, the classification unit 300 calculates the probability that the skeletal muscle belongs to each class of muscle echogenicity by performing pixel unit operation on the ultrasound image through the classification network 310 , and according to the calculated probability Classify the muscle echogenicity of skeletal muscle. In addition, the prediction unit 400 calculates with a probability whether sarcopenia is progressing in the skeletal muscle by performing a pixel unit operation on the ultrasound image through the prediction network 410 , and determines whether sarcopenia is progressing according to the calculated probability. predict And the visualization unit 500 compares the output value and the gradient of the convolutional layer (ECL) of the prediction network 410 to generate a visualization image that displays the region causing sarcopenia in the ultrasound image by distinguishing it from other regions. do. In addition, the identification unit 600 calculates with a probability whether sarcopenia is in progress in the skeletal muscle by performing an operation on a plurality of radioactive features through the identification network 610, and determines whether sarcopenia is in progress according to the calculated probability. identify

Next, the report unit 700 receives information from each of the classification unit 300 , the prediction unit 400 , the visualization unit 500 , and the identification unit 600 in step S440 , and a comprehensive report synthesizing the received information may be displayed through the display unit 13 . Here, the comprehensive report indicates the probability of belonging to a high grade and a probability of belonging to a low grade of muscle echogenicity, and a judgment based on the probability, that is, "muscle echogenicity: high grade" or " Muscular echogenicity: Low grade, and the probability of having sarcopenia predicted based on ultrasound imaging and the probability of not having sarcopenia, and judging according to the probability, "Sarcopenia: benign" or "Sarcopenia: negative", the probability of having sarcopenia and the absence of sarcopenia identified on the basis of the visual image and the radial features, and the judgment according to the probability, "sarcopenia: positive" or "sarcopenia: negative" .

Meanwhile, the above-described method according to an embodiment of the present invention may be implemented in the form of a program readable by various computer means and recorded in a computer readable recording medium. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. For example, the recording medium includes magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floppy disks ( magneto-optical media) and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions may include not only machine language wires such as those generated by a compiler, but also high-level language wires that can be executed by a computer using an interpreter or the like. Such hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Although the present invention has been described above using several preferred embodiments, these examples are illustrative and not restrictive. As such, those of ordinary skill in the art to which the present invention pertains will understand that various changes and modifications can be made in accordance with the doctrine of equivalents without departing from the spirit of the present invention and the scope of rights set forth in the appended claims.

10: control device 20: diagnostic imaging device
100: learning unit 200: preprocessing unit
210: augmentation unit 220: image processing unit
230: feature processing unit 300: classification unit
310: classification network 400: prediction unit
410: prediction network 500: visualization unit
600: identification unit 610: identification network
700: report unit

Claims (19)

In the apparatus for diagnosing the condition of skeletal muscle,
When an ultrasound image of skeletal muscle is input, a preprocessor processing the ultrasound image according to a predetermined format; and
Through a prediction network comprising an input layer, at least a pair of alternating convolutional and pooling layers, at least one fully connected layer, and an output layer
By performing a pixel unit operation on the processed ultrasound image, it is calculated as a probability whether Sarcopenia is in progress in the skeletal muscle,
a predictor for predicting whether sarcopenia progresses according to the calculated probability;
includes,
When the preprocessor extracts a plurality of radioactive features from the ultrasound image of the skeletal muscle,
Calculating with a probability whether Sarcopenia is in progress in the skeletal muscle by performing an operation on the plurality of radioactive features through an identification network including an input layer, a hidden layer, and an output layer,
an identification unit for identifying whether sarcopenia progresses according to the calculated probability;
characterized in that it further comprises
A device for diagnosing the condition of skeletal muscle. According to claim 1,
If the prediction unit predicts sarcopenia,
Comparing the output value and the gradient of the convolutional layer of the prediction network to generate a visualization image displaying a region causing sarcopenia in the ultrasound image by distinguishing it from other regions
visualization unit;
characterized in that it further comprises
A device for diagnosing the condition of skeletal muscle. 3. The method of claim 2,
The visualization unit
After collecting the results of the calculation of the prediction network,
formula

Calculate the neuron importance weight according to
formula

to generate the visualization image according to
remind

is the c-th value of the output before performing the convolution operation in the convolutional layer and performing the operation by the activation function,
remind

is the value of the coordinates (i, j) of the k-th feature map of the convolutional layer,
remind

Is

go

is the gradient with respect to
remind

is the neuron importance weight,
remind

is the value of the coordinates (i, j) of the visualization image
A device for diagnosing the condition of skeletal muscle. delete According to claim 1,
When the preprocessor processes the ultrasound image of the skeletal muscle in a format different from the predetermined format,
Through a classification network comprising an input layer, at least a pair of alternating convolutional and pooling layers, at least one fully connected layer, and an output layer
calculating the probability that the skeletal muscle belongs to each class of muscle echogenicity by performing pixel unit operation on the ultrasound image,
a classification unit for classifying the degree of muscle echogenicity of the skeletal muscle according to the calculated probability;
characterized in that it further comprises
A device for diagnosing the condition of skeletal muscle. According to claim 1,
The output layer of the prediction network prepares an initial model of the prediction network consisting of an output node that performs a softmax operation, and performs initial learning using an image stored in a public database for the initial model as training data,
When the initial learning is completed, the last fully connected layer of the prediction network is replaced with an initialized fully connected layer, and the output node performing the softmax operation of the output layer is replaced with an output node performing the sigmoid operation. a learning unit that prepares a regular model of the prediction network and performs regular learning on the regular model by using an ultrasound image as training data;
characterized in that it further comprises
A device for diagnosing the condition of skeletal muscle. 7. The method of claim 6,
the learning unit
When performing the regular learning on the prediction network,
loss function

using
remind

is set to 0, and classification loss optimization is performed to correct the parameters of the prediction network so that the classification loss, which is the difference between the output value of the prediction network and the classification label, is minimized,
remind

is set to 0.5, and the composite loss including the classification loss, which is the difference between the output value of the prediction network and the classification label, and the concealment loss, which is the difference between the output value and the hidden label of the last layer of the hidden layer of the prediction network, is predicted to be minimal. performing complex loss optimization that modifies the parameters of the network;
The E(y, f) is a loss function,
Wherein yi is the output value of the output layer,
Wherein ei is a classification label corresponding to the output value of the output layer,
The gij is the output value of the last layer of the hidden layer,
Wherein cij is a hidden label corresponding to the output value of the last layer of the hidden layer,
wherein i is an index corresponding to the node of the output layer,
Where j is an index corresponding to the node of the last layer of the hidden layer,
remind

is a hyperparameter
A device for diagnosing the condition of skeletal muscle. In the apparatus for diagnosing the condition of skeletal muscle,
When an ultrasound image of skeletal muscle is input, the preprocessor processes the ultrasound image according to two different formats and extracts a plurality of radioactive features from the ultrasound image; and
Through a classification network including a plurality of layers, a pixel unit operation is performed on an ultrasound image processed in any one of the two formats, and the grade of the muscle echogenicity of the skeletal muscle is calculated as a probability, and the calculated probability a classification unit for classifying the degree of muscle echogenicity of the skeletal muscle according to the
Through a prediction network including a plurality of layers, a pixel unit operation is performed on the ultrasound image processed in the other one of the two formats to calculate with a probability whether sarcopenia is in progress in the skeletal muscle, and the calculated a predictor for predicting whether sarcopenia progresses according to a probability; and
Identification for calculating whether sarcopenia is in progress in the skeletal muscle by performing calculations on the plurality of radioactive features through an identification network including a plurality of layers, and identifying whether sarcopenia is in progress according to the calculated probability wealth;
characterized in that it comprises
A device for diagnosing the condition of skeletal muscle. 9. The method of claim 8,
The forecasting network is
an input layer, at least a pair of alternating convolutional and pooling layers, at least one fully connected layer and an output layer;
If the prediction unit predicts sarcopenia,
Comparing the output value and the gradient of the convolutional layer of the prediction network to generate a visualization image displaying a region causing sarcopenia in the ultrasound image by distinguishing it from other regions
visualization unit;
characterized in that it further comprises
A device for diagnosing the condition of skeletal muscle. 10. The method of claim 9,
The visualization unit
After collecting the results of the calculation of the prediction network,
formula

Calculate the neuron importance weight according to
formula

to generate the visualization image according to
remind

is the c-th value of the output before performing the convolution operation in the convolutional layer and performing the operation by the activation function,
remind

is the value of the coordinates (i, j) of the k-th feature map of the convolutional layer,
remind

Is

go

is the gradient with respect to
remind

is the neuron importance weight,
remind

is the value of the coordinates (i, j) of the visualization image
A device for diagnosing the condition of skeletal muscle. 9. The method of claim 8,
An output node of the output layer of the prediction network prepares an initial model of the prediction network that performs softmax operation, and performs initial learning using an image stored in a public database for the initial model as training data,
When the initial learning is completed, the last layer of the hidden layer of the classification network and the prediction network is replaced with an initialized fully connected layer,
A regular model of the prediction network is prepared by replacing an output node that performs the softmax operation of the output layer with an output node that performs a sigmoid operation, and regular learning is performed using an ultrasound image as training data for the regular model. a learning unit to perform;
characterized in that it further comprises
A device for diagnosing the condition of skeletal muscle. 12. The method of claim 11,
the learning unit
When performing the regular learning on the prediction network,
loss function

using
remind

is set to 0, and classification loss optimization is performed to correct the parameters of the prediction network so that the classification loss, which is the difference between the output value of the prediction network and the classification label, is minimized,
remind

is set to 0.5, and the composite loss including the classification loss, which is the difference between the output value of the prediction network and the classification label, and the concealment loss, which is the difference between the output value and the hidden label of the last layer of the hidden layer of the prediction network, is predicted to be minimal. performing complex loss optimization that modifies the parameters of the network;
The E(y, f) is a loss function,
Wherein yi is the output value of the output layer,
Wherein ei is a classification label corresponding to the output value of the output layer,
The gij is the output value of the last layer of the hidden layer,
Wherein cij is a hidden label corresponding to the output value of the last layer of the hidden layer,
wherein i is an index corresponding to the node of the output layer,
Where j is an index corresponding to the node of the last layer of the hidden layer,
remind

is a hyperparameter
A device for diagnosing the condition of skeletal muscle. In the method for diagnosing the condition of skeletal muscle,
When the prediction unit receives an ultrasound image of skeletal muscle, the ultrasound image is processed in pixel units through a prediction network including an input layer, at least a pair of alternately repeated convolutional and pooling layers, at least one fully connected layer, and an output layer. calculating with a probability whether sarcopenia is in progress in the skeletal muscle by performing an operation; and
predicting whether sarcopenia progresses according to the probability calculated by the prediction unit;
includes,
When the classification unit receives the ultrasound image of the skeletal muscle, the ultrasound image is processed through a classification network including an input layer, at least a pair of alternately repeated convolutional and pooling layers, at least one fully connected layer, and an output layer. calculating a probability that the skeletal muscle belongs to each class of muscle echogenicity by performing a unit operation;
classifying, by the classification unit, the degree of muscle echogenicity of the skeletal muscle according to the calculated probability;
characterized in that it further comprises
A method for diagnosing a condition of skeletal muscle. 14. The method of claim 13,
When the prediction unit predicts sarcopenia, the visualization unit compares the output value and the gradient of the convolutional layer of the prediction network to distinguish and display a region causing sarcopenia in the ultrasound image from other regions. generating;
characterized in that it further comprises
A method for diagnosing a condition of skeletal muscle. 15. The method of claim 14,
The step of generating the visualization image is
collecting, by the visualization unit, a result of the calculation of the prediction network;
the visualization unit
formula

calculating a neuron importance weight according to
the visualization unit
formula

generating the visualization image according to
includes,
remind

is the c-th value of the output before performing the convolution operation in the convolutional layer and performing the operation by the activation function,
remind

is the value of the coordinates (i, j) of the k-th feature map of the convolutional layer,
remind

is said

is reminded

is the gradient with respect to
remind

is the neuron importance weight,
remind

is the value of the coordinates (i, j) of the visualization image
A method for diagnosing a condition of skeletal muscle. 14. The method of claim 13,
When the identification unit receives a plurality of radioactive features extracted from the ultrasound image of the skeletal muscle, sarcopenia progresses in the skeletal muscle by performing an operation on the plurality of radioactive features through an identification network including an input layer, a hidden layer, and an output layer. calculating whether or not it is being performed with a probability; and
identifying whether sarcopenia progresses according to the probability calculated by the identification unit;
characterized in that it further comprises
A method for diagnosing a condition of skeletal muscle. delete 14. The method of claim 13,
Before the step of calculating with a probability whether sarcopenia is in progress in the skeletal muscle,
preparing, by a learning unit, an initial model of the prediction network including an output node in which the output layer of the prediction network performs a softmax operation; and
performing, by the learning unit, initial learning with respect to the initial model by using an image stored in a public database as training data;
The learning unit replaces the last fully connected layer of the prediction network with an initialized fully connected layer, and replaces the output node performing the softmax operation of the output layer with an output node performing the sigmoid operation, and preparing a canonical model; and
performing, by the learning unit, regular learning on the regular model using an ultrasound image as training data;
characterized in that it further comprises
A method for diagnosing a condition of skeletal muscle. 19. The method of claim 18,
The step of performing the regular learning is
the learning department
loss function

using
remind

after setting to 0, performing classification loss optimization by modifying parameters of the prediction network so that the classification loss, which is the difference between the output value of the prediction network and the classification label, is minimized;
the learning unit

is set to 0.5, and the composite loss including the classification loss, which is the difference between the output value of the prediction network and the classification label, and the concealment loss, which is the difference between the output value and the hidden label of the last layer of the hidden layer of the prediction network, is predicted to be minimal. performing complex loss optimization to modify parameters of the network;
includes,
The E(y, f) is a loss function,
Wherein yi is the output value of the output layer,
Wherein ei is a classification label corresponding to the output value of the output layer,
The gij is the output value of the last layer of the hidden layer,
Wherein cij is a hidden label corresponding to the output value of the last layer of the hidden layer,
wherein i is an index corresponding to the node of the output layer,
Where j is an index corresponding to the node of the last layer of the hidden layer,
remind

is a hyperparameter
A method for diagnosing a condition of skeletal muscle.

Images