KR20230069287A - Apparatus and method for measuring amniotic fluid index based on ultrasound images - Google Patents

Abstract

The present invention provides an apparatus and method for measuring an AFI, which can accurately detect amniotic fluid sacs robustly even against reverberation artifacts that occur in an ultrasound image, wherein the apparatus comprises: an ultrasound image input unit which acquires an input image that is an ultrasound image taken of an amniotic fluid sac; a first area estimation unit which performs a neural network operation on the input image using a pre-trained artificial neural network to detect an area of the amniotic fluid sac and extract the same as a first area; a second area estimation unit which performs a neural network operation on the input image using a pre-trained artificial neural network to detect an abdominal wall area, detects an area where reverberation artifacts occur in the input image based on the detected abdominal wall area and the direction of ultrasound movement, and extracts the same as a second area; and an amniotic fluid sac extraction unit which merges the first area and the second area to obtain a complementary amniotic fluid sac area.

Description

Apparatus and method for measuring amniotic fluid index based on ultrasound images}

The present invention relates to an amniotic fluid indicator measuring device and method, and more particularly to an ultrasound image-based amniotic fluid indicator measuring device and method.

Amniotic fluid (AF) is a protective fluid contained in the amniotic cavity and is an essential component for fetal development and maturation during pregnancy. AF volume (hereafter referred to as AFV) is an important index that reflects pregnancy progress and fetal development, and AFV evaluation is essential during ultrasound examination of pregnant women before childbirth. estimated by measuring Conventionally, a measurer such as a doctor manually measured AFI while performing an ultrasound examination.

1 is a diagram for explaining an AFI index measurement method according to the existing manual work.

As shown in FIG. 1, the AFI measurement generally divides the amniotic fluid pocket area into four quadrants (Q1, Q2, Q3, Q4), and as shown on the right, divided into four quadrants. AFI (AFI = _{y Q1} + y _Q2 + y _Q3 ) by summing the amniotic fluid index (y Q1 , y _Q2 , y _Q3 , y _Q4 ) + y _Q4 ). However, such manual measurement is not only time consuming, but also has a problem in that errors occur greatly depending on the skill level of the measurer, and erroneous measurement of AFI causes an erroneous diagnosis of the condition of the fetus, making it difficult to respond early.

In order to overcome these problems, research on a method for automatically measuring AFI from ultrasound images using deep learning techniques has been conducted recently. However, due to factors such as the amorphous nature of the amniotic sac, ultrasound reverberation, AF mimic area, suspended matter, and incomplete or missing borders of the ultrasound image, the accuracy is still not high enough to be applied clinically.

In particular, reverberation artifacts of ultrasound, which frequently appear in maternal obesity and advanced gestational age, make it difficult to distinguish the amniotic sac and become a major factor hindering the measurement of AFI.

Korean Patent Publication No. 10-2021-0002198 (published on January 7, 2021)

An object of the present invention is to provide an AFI measuring device and method capable of automatically and accurately measuring AFI.

Another object of the present invention is to provide an AFI measurement apparatus and method capable of improving AFI measurement performance by accurately estimating an amniotic sac area in an ultrasound image despite reverberation artifacts.

An AFI measuring device according to an embodiment of the present invention for achieving the above object includes an ultrasound image input unit for obtaining an input image that is an ultrasound image taken of an amniotic sac; a first area estimator for performing a neural network operation on the input image using a pre-learned artificial neural network to detect an area of the amniotic sac and extract it as a first area; A neural network operation is performed on the input image using a pre-learned artificial neural network to detect the abdominal wall area, and based on the detected abdominal wall area and the direction of ultrasound waves, an area in which reverberation artifacts occur in the input image is detected to detect the second area. a second area estimator for extracting as; and an amniotic fluid bag extraction unit configured to obtain a supplementary amniotic bag region by merging the first region and the second region.

an abdominal wall area estimating unit implemented as a pre-learned artificial neural network and detecting the abdominal wall area by performing a neural network operation on the input image; a reverberation range setting unit configured to set a range corresponding to the thickness of the abdominal wall region from the boundary of the estimated abdominal wall region in the traveling direction of the ultrasonic waves as a reverberation artifact possible region where reverberation artifacts may occur; a reverberation feature map acquisition unit acquiring a reverberation feature map in which reverberation artifacts are emphasized in a region corresponding to the reverberation artifact possible region of the input image according to the generation characteristics of the reverberation artifact; and a second area detection unit implemented as a pre-learned artificial neural network and detecting the second area by performing a neural network operation on the input image and the reverberation feature map.

The reverberation feature map obtaining unit may acquire the reverberation feature map by performing a Hadamard product of the input image and the reverberation artifact possible region, and differentiating the reverberation feature map along a traveling direction of ultrasonic waves.

The second region detection unit may include a scale conversion input unit configured to receive the input image and the reverberation feature map, and down-scaling the applied reverberation feature map to different sizes to obtain a plurality of scaled reverberation feature maps; Features are extracted by encoding each of the input image and the scaled reverberation feature map, and the features extracted from the input image are pooled and scaled down, combined with the features extracted from the scaled reverberation feature map of the corresponding lower scale, and re-encoded a reverberation enhancement feature extraction unit that extracts a feature and decodes a feature extracted at a lower level together with a feature extracted at an upper level to detect a reverberation artifact region; and a second area obtaining unit configured to acquire the second area by dividing the reverberation artifact area from the input image.

The first region estimator may include a multi-scaling unit configured to obtain a plurality of scaled input images by down-scaling the input image to different sizes; The input image and each of the plurality of scaled input images are encoded by performing a neural network operation according to the learned method, but at a lower scale, a plurality of scale features according to each scale are encoded by pooling and combining features encoded at the upper scale. A feature map extractor that obtains and decodes the encoded features by neural network operation, but obtains a plurality of scale feature maps according to each scale by combining and decoding the feature maps obtained by decoding at the upper scale and the lower scale; a scale restorer configured to obtain a plurality of sub AF maps by upsampling the plurality of scale feature maps to the size of the input image; and a feature map combiner configured to obtain the first area by combining the plurality of sub AF maps.

The amniotic bag extraction unit may include an estimated region merging unit that merges the first region and the second region to obtain the supplementary amniotic bag region; and a post-processing unit that removes minute errors included in the supplementary amniotic bag area based on the morphological lung structure of the amniotic bag.

The AFI measuring device may obtain the AFI by measuring the thickness of the supplementary amniotic sac region.

An AFI measurement method according to another embodiment of the present invention for achieving the above object includes acquiring an input image that is an ultrasound image taken of an amniotic sac;

detecting an area of the amniotic sac by performing a neural network operation on the input image using a pre-learned artificial neural network and extracting the area as a first area; A neural network operation is performed on the input image using a pre-learned artificial neural network to detect the abdominal wall area, and based on the detected abdominal wall area and the direction of ultrasound waves, an area in which reverberation artifacts occur in the input image is detected to detect the second area. Extracting with; and merging the first area and the second area to obtain a supplementary amniotic bag area.

Therefore, the AFI measuring apparatus and method according to an embodiment of the present invention, when determining the amniotic sac area for AFI measurement from an ultrasound image, constructs a dual-path network to accurately estimate the area of the amniotic sac including the area where reverberation artifacts occur, thereby providing AFI to be able to estimate accurately.

1 is a diagram for explaining an AFI index measurement method according to the existing manual work.
2 shows a schematic structure of an AFI measuring device according to an embodiment of the present invention.
FIG. 3 is a diagram for explaining a schematic operation of each component of the AFI measuring device of FIG. 2 .
FIG. 4 shows an example of a detailed configuration of the first area estimation unit of FIG. 2 .
FIG. 5 shows an implementation example of the first area estimator of FIG. 4 .
6 is a diagram for explaining misdetection factors when detecting an amniotic fluid sac from an ultrasound image using a single neural network.
7 shows an example of an amniotic sac area incorrectly detected due to reverberation artifacts.
8 and 9 are diagrams for explaining regions where reverberation artifacts occur according to reverberation characteristics of an ultrasound beam.
FIG. 10 shows an example of a detailed configuration of the second area estimation unit of FIG. 2 .
FIG. 11 is a diagram for explaining a schematic operation of each component of the second area estimation unit of FIG. 6 .
FIG. 12 shows an example of a detailed configuration of the second area detection unit of FIG. 10 .
FIG. 13 illustrates an implementation example of the second area detection unit of FIG. 12 .
14 shows an example of a detailed configuration of the amniotic bag extraction unit of FIG. 2 .
FIG. 15 is a diagram for explaining the operation of each component of the amniotic bag extractor of FIG. 14 .
16 shows an AFI measurement method according to an embodiment of the present invention.

In order to fully understand the present invention and its operational advantages and objectives achieved by the practice of the present invention, reference should be made to the accompanying drawings illustrating preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, the present invention will be described in detail by describing preferred embodiments of the present invention with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the described embodiments. And, in order to clearly describe the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when a part "includes" a certain component, it means that it may further include other components, not excluding other components unless otherwise stated. In addition, terms such as "... unit", "... unit", "module", and "block" described in the specification mean a unit that processes at least one function or operation, which is hardware, software, or hardware. And it can be implemented as a combination of software.

2 shows a schematic structure of an AFI measuring device according to an embodiment of the present invention, and FIG. 3 is a diagram for explaining a schematic operation of each component of the AFI measuring device of FIG. 2 .

Referring to FIG. 2 , the AFI measuring device according to the present embodiment includes an ultrasound image input unit 100, a first area estimation unit 200, a second area estimation unit 300, an amniotic sac extraction unit 400, and an AFI acquisition unit. It may include section 500 .

The ultrasound image input unit 100 acquires an ultrasound image as an input image I through ultrasound examination of a pregnant woman, and inputs the ultrasound image to the first region estimator 200 and the second region estimator 300, respectively.

The ultrasound image input unit 100 may be implemented as an ultrasound examination device, but may also be implemented as a storage device in which an ultrasound image obtained from the ultrasound examination device is previously stored or a communication module that receives ultrasound images.

At this time, the ultrasound image input unit 100 removes an unnecessary area from the obtained ultrasound image or converts the size into an input image I having a size that is easy to be processed by the first and second area estimation units 200 and 300. You can also do preprocessing. For example, as shown in FIG. 3 , the ultrasound image input unit 100 may obtain an input image I having a size of 512×512 by receiving an ultrasound image having a size of 600 × 1000.

The first and second area estimators 200 and 300 are each implemented as a pre-learned artificial neural network to estimate and detect an area corresponding to the amniotic sac in the input image I transmitted from the ultrasound image input unit 100. It works as an amniotic sac detection unit.

)을 검출한다. 즉 본 실시예의 AFI 측정 장치는 입력 이미지(I)로부터 양수 주머니 영역을 검출하기 위한 이중 신경망 경로(dual path network)로 구현된다. 여기서는 제1 영역 추정부(200)는 입력 이미지(I)에서 양수 주머니 전체 영역을 추정하는 주경로(Primary path: f^*)라 하고, 제2 영역 추정부(300)는 제1 영역 추정부(200)에서 추정한 양수 주머니 영역에서 누락된 영역을 보완하기 위한 2차 경로(Secondary path: f^**) 또는 보조 경로(auxiliary path)라 할 수 있다.The first area estimator 200 performs a neural network operation on the input image I according to the learned method, and detects the first area Ω _D corresponding to the entire area of the amniotic sac. In addition, the second area estimator 200 includes a second area corresponding to a reverberation artifact area that is difficult for the first area estimator 200 to detect in the entire area of the amniotic bag (

) is detected. That is, the AFI measuring device of this embodiment is implemented as a dual path network for detecting the amniotic sac area from the input image I. Here, the first area estimating unit 200 is referred to as a primary path (f ^* ) for estimating the entire area of the amniotic bag in the input image I, and the second area estimating unit 300 is the first area estimating unit ( 200), it can be referred to as a secondary path (f ^** ) or an auxiliary path to compensate for the missing area in the amniotic sac area.

)을 병합함으로써, 입력 이미지(I)에서 양수 주머니에 해당하는 양수 주머니 영역을 추출한다. 그리고 AFI 획득부(500)는 양수 주머니 추출부(400)가 획득한 양수 주머니 영역에서 AFI를 기지정된 방식으로 측정하여 획득한다.Meanwhile, the amniotic bag extractor 400 determines the first area Ω D estimated by the first area estimating unit 200 and the second area Ω _D estimated by the second area estimating unit 300 (

), the amniotic sac region corresponding to the amniotic sac is extracted from the input image I. The AFI obtaining unit 500 obtains the AFI by measuring the AFI in the amniotic bag area acquired by the amniotic bag extracting unit 400 in a predetermined manner.

FIG. 4 shows an example of a detailed configuration of the first area estimation unit of FIG. 2 , and FIG. 5 shows an implementation example of the first area estimation unit of FIG. 4 .

The first area estimator 200 may be implemented with various artificial neural networks, but here, as an example, the AF-net (H. C. Cho, S. Sun, C. M. Hyun) designed to distinguish the amniotic sac area by modifying the existing U-net , J.-Y. Kwon, B. Kim, Y. Park, and J. K. Seo, "Automated ultrasound assessment of amniotic fluid index using deep learning," Medical Image Analysis, vol. 69, p. 101951, 2021.) 5 shows AF-net as an implementation example.

When the first area estimator 200 is implemented as an AF-net, the first area estimator 200 includes a multiple scaling unit 210, a feature map extraction unit 220, and a scale as shown in FIG. A restoration unit 230 and a feature map combination unit 240 may be included.

Referring to FIG. 5, each component of the first area estimator 200 of FIG. 4 will be described. First, the multi-scaling unit 210 receives an input image I, down-scales it to different sizes, and multi-scales it. Acquire input images η ₁ (I), η ₂ (I), η ₃ (I). For example, the multi-scaling unit 210 may perform downscaling by performing average pooling.

Further, the feature map extractor 220 receives the input image I and a plurality of scaled input images η ₁ (I), η ₂ (I), and η ₃ (I), and converts the applied input image (I) to and a plurality of scaled input images (η ₁ (I), η ₂ (I), η ₃ (I)), and performs neural network operation by atrous convolution according to the learned method. As shown in the lower left of FIG. 5, the atrus convolution performs a convolution operation by inserting the number of 0s according to a specified rate between adjacent weight elements in the weight matrix of the convolution kernel in the artificial neural network. It is a technique to In FIG. 5, as an example, assuming that an atrus convolution operation with a ratio of 4 is performed using a weight matrix having a size of 3 × 3, the convolution kernel has a weight for every 4th row and 4th column in the row and column directions, respectively. It can be seen that all elements of the remaining rows and columns are filled with 0. Here, the feature map extractor 220 has a structure based on U-net and is composed of a plurality of scale encoders and a plurality of scale decoders, and the highest scale encoder receiving the input image (I) among the plurality of scale encoders is an input A feature is extracted by performing a neural network operation only on the image (I). In addition, the remaining scale encoder performs maximum pooling on the scale input image (η ₁ (I), η ₂ (I), η ₃ (I)) according to the corresponding scale and the features obtained in the upper scale. The features that match the size are applied together and the features are extracted. Meanwhile, the lowest scale decoder having the smallest size among the plurality of scale decoders decodes the feature applied from the corresponding lowest scale encoder and outputs the lowest scale feature map. Then, each of the remaining scale decoders performs average unpooling of the features applied from the corresponding scale encoder and the features applied from the lower scale decoder to obtain the corresponding scale feature map by decoding the unpooling features of which sizes match together. do.

)(여기서 k는 스케일 인덱스이며 도 5의 예에서는 k = {1, …, 4})을 획득한다. 여기서 활성화 함수는 일 예로 시그모이드(sigmoid) 함수가 이용될 수 있다.The scale restoration unit 230 performs up-sampling so that feature maps obtained with different scales from each of the plurality of scale decoders of the feature map extraction unit 220 have the same size. Here, the up-sampled feature map may be up-sampled to the size of the input image I. In addition, the scale restoration unit 230 applies a predetermined activation function to each of the upsampled feature maps to sub AF map (

) (where k is a scale index and in the example of FIG. 5 k = {1, ..., 4}) is obtained. Here, as an example, a sigmoid function may be used as the activation function.

)을 결합하여 입력 이미지(I)에서 양수 주머니에 해당하는 영역을 구분함으로써 제1 영역(Ω_D)을 획득한다.The feature map combining unit 240 includes a plurality of sub AF maps (

) to obtain the first region Ω _D by dividing the region corresponding to the amniotic sac in the input image I.

As described above, learning about the operation (f ^* ) of the first area estimator 200 that is implemented as AF-net and extracts the amniotic sac area from the input image I may be performed using Equation 1.

는 실제 양수 주머니 영역(

)이 레이블된 입력 이미지(I)인 학습 데이터에서 획득된 서브 AF 맵(

)과 실제 양수 주머니 영역(

) 사이에 계산되는 교차 엔트로피 손실 함수로서, 수학식 2로 계산된다.here

is the actual amniotic sac area (

) is the sub-AF map (

) and the actual amniotic sac area (

) as the cross entropy loss function calculated between Equation 2.

Here, Ω represents a set of pixels (I ^(j) ) in the input image (I).

Since AF-net is a well-known technology, it is not described in detail here.

)을 검출한다. 본 실시예에서 AFI 측정 장치가 제1 영역 추정부(200)뿐만 아니라 제2 영역 추정부(300)를 더 구비하여 양수 주머니 영역을 추정하는 것은, 양수 주머니 영역 전체를 검출하도록 학습된 제1 영역 추정부(200)만으로는 입력 이미지(I)인 초음파 이미지에 포함된 아티팩트(artifact)로 인해 정확한 양수 주머니 영역을 추정할 수 없기 때문이다.Meanwhile, as described above, the second area estimator 300 includes a second area (which corresponds to a reverberation artifact area that is difficult for the first area estimator 200 to detect).

) is detected. In this embodiment, the AFI measuring device further includes the first area estimating unit 200 and the second area estimating unit 300 to estimate the amniotic bag area, which is the first area learned to detect the entire amniotic bag area. This is because the estimation unit 200 alone cannot accurately estimate the amniotic sac area due to artifacts included in the ultrasound image that is the input image I.

For this reason, although the first area estimator 200 is implemented as an AF-net specialized for detecting the amniotic sac area and learned in advance to detect the amniotic sac area, various artifacts included in the input image I Therefore, it is well known from previous studies that the first area estimator 200 has limitations in accurately estimating the amniotic sac area. That is, it is very difficult to prevent erroneous detection of the amniotic sac area due to artifacts using only a single neural network path.

6 is a diagram for explaining misdetection factors when detecting an amniotic fluid sac from an ultrasound image using a single neural network.

As shown in FIG. 6 , an ultrasound image may include various artifacts. In FIG. 6, (a) shows reverberation artifacts according to ultrasonic characteristics, (b) shows artifacts according to AF mimicking region, and (c) shows artifacts according to floating matters. Indicates an artifact. And (d) represents an artifact due to an incomplete or missing boundary of the ultrasound image.

Among the various artifacts shown in FIG. 6 , in particular, reverberation artifacts, which are different from the characteristics of ultrasound examination, become the biggest obstacle that makes it difficult to accurately detect the amniotic sac area.

7 shows an example of an amniotic sac area incorrectly detected due to reverberation artifacts.

7 shows amniotic sac regions incorrectly detected in cases (Case 1 and Case 2) for two input images (I). In FIG. 7, (a) shows the input image (I), (b) shows the real area of the amniotic bag and the reverberation artifact area incorrectly detected due to the reverberation artifact, and (c) shows the first area estimation unit 200 represents the estimated amniotic sac area. In (b) of FIG. 7, the orange outline represents the actual amniotic bag area, the cyan dotted line represents the area where reverberation artifacts occur, and the green area in (c) represents the amniotic bag area estimated by the first area estimator 200. indicate

Comparing (b) and (c) of FIG. 7 , even if the first area estimator 200 implemented as an artificial neural network is trained to estimate the amniotic sac, the area where reverberation artifacts are generated cannot be detected as the area of the amniotic sac. It can be seen that there is no

)을 검출하는 제2 신경망 경로인 제2 영역 추정부(300)가 더 구비된다.Accordingly, in the present embodiment, a second region (a region in which reverberation artifacts occur in the amniotic sac) separate from the first neural network path of the first region estimator 200 (

) is further provided with a second area estimator 300 that is a second neural network path for detecting.

8 and 9 are diagrams for explaining regions where reverberation artifacts occur according to reverberation characteristics of an ultrasound beam.

In FIG. 8, (a) is a diagram for explaining the cause of reverberation artifacts, (b) shows the relationship between an abdominal wall area and a reverberation artifact area in an actual ultrasound image, and FIG. 9 shows a reverberation artifact represents an area with a high probability of contamination by

Severe reverberation artifacts occur when the ultrasound beam encounters a strong reflector, such as the fat-muscle interface of the abdominal wall, approximately orthogonal to the direction of beam propagation. As shown in (a) of FIG. 8, these reverberation artifacts are formed in the form of a plurality of parallel curves (rev1 to rev3) at regular intervals (d) perpendicular to the ultrasonic propagation direction. Therefore, as shown in (b) of FIG. 8 , the reverberation artifact appears on the opposite side of the direction in which the ultrasonic wave is applied, that is, on the side of the ultrasonic wave propagation direction.

In FIG. 9 , areas where reverberation artifacts or appeared in several ultrasound images are indicated by dotted lines. Since reverberation artifacts are generated by the fat-muscle boundary, it can be seen that, as shown in (a) to (d) of FIG. 9 , if the ultrasound images are different, the regions where reverberation artifacts occur also appear differently.

However, as described above, since the reverberation artifact is formed in the form of a plurality of parallel curves (rev1 to rev3) at uniform intervals (d) perpendicular to the ultrasonic propagation direction, the reverberation artifact is the thickness (d _abw ) of the abdominal wall area (Ω _abw ) It appears in the range of thickness (d _rvb ) almost equal to (d _rvb ≒ d _abw ). Therefore, when the thickness (d _abw ) is confirmed by detecting the abdominal wall area (Ω _abw ), it can be seen that reverberation artifacts are generated within the range of the thickness (d _rvb ) corresponding to the abdominal wall thickness (d _abw ) from the boundary of the abdominal wall. . That is, a reverberation artifact possible region (Ω _rvb ) in which reverberation artifacts may occur may be set. In (b) of FIG. 8 , purple arrows, white arrows, and blue arrows indicate reverberation artifacts of fascia, amniotic sac, and peritoneum-fascia layer, respectively.

Therefore, in this embodiment, the second area estimator 300 detects the abdominal wall area and easily detects the reverberation artifact area based on the detected abdominal wall area.

FIG. 10 shows an example of a detailed configuration of the second area estimation unit of FIG. 2 , and FIG. 11 is a diagram for explaining a schematic operation of each configuration of the second area estimation unit of FIG. 6 .

Referring to FIG. 10 , the second area estimation unit 300 may include an abdominal wall area estimation unit 310, a reverberation range setting unit 320, a reverberation mask acquisition unit 330, and a second area detection unit 340. .

The abdominal wall area estimator 310 receives the input image I and performs a neural network operation according to a previously learned method to detect the abdominal wall area Ω _abw . When the second area estimator 300 includes the abdominal wall area estimator 310 to detect the abdominal wall area Ω _abw , the abdominal wall area Ω _abw is more distinct than the reverberation artifact possible area Ω _rvb , as described above. This is because it has morphological characteristics and is easy to extract.

The abdominal wall area estimator 310 may be implemented with various artificial neural networks, but here it is assumed that it is implemented with a U-net pretrained to detect the abdominal wall area as an example. Similar to AF-net of the first area estimator 200, U-net accurately extracts the abdominal wall area (Ω _abw ) by converting the input image (I) into various scales and extracting and combining features for each scale. do.

)은 수학식 3에 따라 미리 학습될 수 있다.Here, the operation of the abdominal wall area estimation unit 310 implemented by U-net (

) may be learned in advance according to Equation 3.

는 입력 이미지(I)에 실제 복벽 영역(

)이 레이블된 학습 데이터와 추정된 복벽 영역(

= Ω_abw) 사이에 계산되는 교차 엔트로피 손실 함수이다.here

Is the actual abdominal wall area in the input image (I) (

) is the labeled training data and the estimated abdominal wall area (

= Ω _abw ) is the cross-entropy loss function calculated between

The reverberation range setting unit 320 measures the thickness (d _abw ) of the abdominal wall area (Ω _abw ) estimated by the abdominal wall area estimating unit 310, and measures the direction (r) of the ultrasonic wave from the boundary of the abdominal wall area (Ω _abw ). A range of the thickness (Ω _rvb ) equal to the abdominal wall thickness (d _abw ) measured in the direction is set as the reverberation artifact possible region (Ω _rvb ). Here, the reverberation artifact possible region Ω _rvb is a kind of mask, and pixel values of the remaining region except for the reverberation artifact possible region Ω _rvb may be set to 0.

)하고, 초음파의 진행 방향(r)을 따라 미분(

)하여 잔향 특징맵을 획득할 수 있다.The reverberation feature map acquisition unit 330 obtains a reverberation artifact feature map based on the input image I and the set possible region for derived reverberation artifacts (Ω _rvb ). The reverberation feature map acquisition unit 330 calculates the Hadamard product (Hadamard product) of the input image (I) and the set reverberation artifact possible region (Ω _rvb ).

), and differentiated along the direction (r) of the ultrasonic wave (

) to obtain a reverberation feature map.

The area where reverberation artifacts are possible (Ω _rvb ) merely represents a region where reverberation artifacts can occur, but does not represent a region where reverberation artifacts occur in the amniotic fluid bag. Accordingly, the reverberation feature map acquisition unit 330 acquires a reverberation feature map that highlights and displays a region where a reverberation artifact occurs in an actual amniotic sac by differentiating the reverberation artifact in the traveling direction (r) of the ultrasonic waves due to the nature of the reverberation artifact.

)을 검출하고, 검출된 잔향 아티팩트 영역(

)을 입력 이미지로부터 분할하여 제2 영역(

)을 획득한다.The second region detector 340 receives the input image I and the reverberation feature map, and determines a reverberation artifact region in which a reverberation artifact occurs in the input image I based on the applied reverberation feature map (

) is detected, and the detected reverberation artifact area (

) is divided from the input image to form a second region (

) to obtain

FIG. 12 shows an example of a detailed configuration of the second area detection unit of FIG. 10 , and FIG. 13 shows an implementation example of the second area detection unit of FIG. 12 .

Referring to FIG. 12 , the second area detector 340 according to the present embodiment may include a scale conversion input unit 341 , a reverberation enhancement feature extractor 342 and a second area obtainer 343 .

Referring to FIG. 13, the operation of each component of the second area detection unit 340 will be described. First, the scale conversion input unit 341 receives an input image I and a reverberation feature map, and converts the applied reverberation feature map into By down-scaling to different sizes, multiple scale reverberation feature maps are obtained.

)을 검출하도록 수정된 인공 신경망으로 구현될 수 있으며, 여기서는 이를 RVB-net라고 한다.The reverberation enhancement feature extractor 342 is implemented as an artificial neural network and identifies and detects a reverberation artifact region in the input image I according to a learned method. The reverberation enhancement feature extractor 342 extracts, for example, a reverberation artifact area based on U-net (

) can be implemented as an artificial neural network modified to detect , which is referred to herein as RVB-net.

)을 검출한다. 이때 디코딩 과정에서 도 13에 도시된 바와 같은 주의 게이트(attention gate: AG)를 적용하여, 잔향 강조 특징 추출부(342)가 잔향 특징맵의 영역에만 집중하여 잔향 아티팩트 영역(

)을 검출하도록 할 수 있다.As shown in FIG. 13, the reverberation enhancement feature extractor 342 encodes the input image I and the scaled reverberation feature map to extract features, and pools the maximum value for the features extracted from the input image I. While scaling down by applying , the feature is extracted by encoding again by combining with the feature extracted from the corresponding scaled reverberation feature map. That is, the features to be extracted at the upper scale are transferred to the lower scale and combined to extract the features. And the features extracted from the lower part are decoded together with the features extracted from each scale to obtain a reverberation artifact area (

) is detected. At this time, in the decoding process, by applying an attention gate (AG) as shown in FIG. 13, the reverberation enhancement feature extractor 342 concentrates only on the region of the reverberation feature map,

) can be detected.

)은 수학식 4에 따라 학습될 수 있다.Operation of the reverberation enhancement feature extraction unit 342 implemented as U-net based RVB-net (

) can be learned according to Equation 4.

는 실제 잔향 아티팩트 영역(

)이 레이블된 입력 이미지(I_rvb)의 학습 데이터와 스케일(k)에 따라 추출된 잔향 아티팩트 영역(

) 사이에 계산되는 교차 엔트로피 손실 함수이다.here

is the actual reverberation artifact area (

) is extracted according to the training data and scale (k) of the labeled input image (I _rvb ).

) is the cross-entropy loss function calculated between

The reverberation enhancement feature extractor 342 may also be configured with other artificial neural networks, and since an implementation example of the attention gate AG is also shown in FIG. 13 , a detailed description thereof will be omitted here.

)을 검출하면, 검출된 잔향 아티팩트 영역(

)을 구분하여 제2 영역(

)을 획득한다.The reverberation enhancement feature extractor 342 obtains a reverberation artifact area (

), the detected reverberation artifact area (

) by dividing the second area (

) to obtain

FIG. 14 shows an example of a detailed configuration of the amniotic bag extractor of FIG. 2 , and FIG. 15 is a diagram for explaining the operation of each component of the amniotic bag extractor of FIG. 14 .

)을 병합하여, 도 15의 왼쪽과 같이, 보완 양수 주머니 영역을 획득한다.The amniotic bag extraction unit 400 may include an estimated region merging unit 410, a post-processing unit 420, and a size adjustment unit 430. The estimated area merging unit 410 divides the area estimated as the amniotic sac by the first area estimating unit 200 to obtain the first area Ω _D and the second area estimating unit 300 to generate reverberation artifacts among the amniotic sacs. The second area obtained by dividing into the area where

) are merged to obtain a complementary amniotic sac area, as shown on the left of FIG. 15 .

)의 병합에도 발생한 미세한 오류가 제거되었음을 알 수 있다.Further, the post-processing unit 420 removes holes indicated by orange arrows and false positive errors indicated by yellow arrows in FIG. 15 based on the morphological closed structure of the amniotic sac. Comparing (a) and (b), the first region (Ω _D ) and the second region (

), it can be seen that the subtle errors that occurred in the merge were removed.

The size adjusting unit 430 converts the image of the supplementary amniotic sac area from which errors are removed into the size of the original ultrasound image and outputs the converted image.

Further, the AFI acquisition unit 500 obtains the AFV by measuring the AFI representing the amniotic fluid depth in the corresponding quadrant as in FIG. 1 by measuring the width or thickness of the amniotic bag region, as shown in the right figure in FIG. 15 .

As a result, the AFI measurement method according to the present embodiment determines the area where reverberation artifacts can occur based on the abdominal wall based on the principle that reverberation artifacts occur in ultrasound images together with the first area estimator 200 that estimates the entire amniotic sac area. A second estimator 300 for detecting and estimating an area in which reverberation artifacts actually occur within the detected reverberation artifact possibility area is provided, and the area estimated by the first and second area estimators 200 and 300 is By merging, the amniotic sac area can be extracted from the ultrasound image with high accuracy, and the AFI can be measured from the extracted amniotic sac area.

16 shows an AFI measurement method according to an embodiment of the present invention.

)을 추출하는 단계(S30)를 수행한다.Referring to the AFI measurement method of FIG. 16 with reference to FIGS. 1 to 15, first, an input image I is obtained based on an ultrasound image of a pregnant woman in order to measure AFI (S10). Then, a neural network operation is performed on the acquired input image I using an artificial neural network pretrained to detect the amniotic sac region, and a first region Ω _D is extracted (S20). The second area (with the step of detecting the first area)

) Performs the step (S30) of extracting.

)하고, 초음파의 진행 방향(r)에 따라 미분(

)하여 잔향 아티팩트가 강조된 잔향 특징맵을 획득한다(S33). 입력 이미지(I)와 잔향 특징맵에 대해 학습된 방식에 따라 신경망 연산을 수행하여 입력 이미지(I)에서 잔향 특징맵에 의해 강조된 잔향 아티팩트 영역을 검출함으로써 제2 영역(

)을 검출한다.In the step of extracting the second region (S30), a neural network operation is performed on the input image (I) using an artificial neural network pretrained to extract the abdominal wall region from the input image (I) to obtain the abdominal wall region (Ω _abw ). Detect (S31). Then, _an area equal to the thickness (d abw ) of the abdominal wall area (Ω _abw ) in the traveling direction (r) of the ultrasound from the boundary of the detected abdominal wall area (Ω _abw ) is set as a reverberation artifact possible area (Ω _rvb ) (S32). . Then, the Hadamard product of the input image (I) and the possible region for reverberation artifacts (Ω _rvb ) (

), and differentiated according to the traveling direction (r) of the ultrasonic wave (

) to obtain a reverberation feature map in which reverberation artifacts are emphasized (S33). The second area (

) is detected.

)이 추출되면, 추출된 제1 영역(Ω_D)과 제2 영역(

)을 병합하여 보완 양수 주머니 영역(Ω)을 획득한다. 그리고 획득된 양수 주머니 영역(Ω)의 두께, 즉 양수 주머니의 깊이를 나타내는 AFI를 측정함으로써 AFV를 계산한다(S50).The first region (Ω _D ) and the second region (

) is extracted, the extracted first region (Ω _D ) and the second region (

) to obtain the complementary amniotic sac area (Ω). Then, AFV is calculated by measuring the thickness of the obtained amniotic sac area (Ω), that is, AFI representing the depth of the amniotic sac (S50).

The method according to the present invention may be implemented as a computer program stored in a medium for execution on a computer. Here, computer readable media may be any available media that can be accessed by a computer, and may also include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, including read-only memory (ROM) dedicated memory), random access memory (RAM), compact disk (CD)-ROM, digital video disk (DVD)-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

Although the present invention has been described with reference to the embodiments shown in the drawings, this is only exemplary, and those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom.

Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

100: ultrasound image input unit 200: first area estimation unit
210: multi-scaling unit 220: feature map extraction unit
230: scale restoration unit 240: feature map combining unit
300: second area estimation unit 310: abdominal wall area estimation unit
320: reverberation range setting unit 330: reverberation feature map acquisition unit
340: second area detection unit 341: scale conversion input unit
342: reverberation enhancement feature extraction unit 343: second area acquisition unit
400: amniotic sac extraction unit 410: estimation area merging unit
420: post-processing unit 430: size adjustment unit
500: AFI acquisition unit

Claims (19)

an ultrasound image input unit that obtains an input image that is an ultrasound image of the amniotic sac;
a first area estimator for performing a neural network operation on the input image using a pre-learned artificial neural network to detect an area of the amniotic sac and extract it as a first area;
A neural network operation is performed on the input image using a pre-learned artificial neural network to detect the abdominal wall area, and based on the detected abdominal wall area and the direction of ultrasound waves, an area in which reverberation artifacts occur in the input image is detected to detect the second area. a second area estimator for extracting as; and
and an amniotic sac extractor configured to merge the first area and the second area to obtain a supplementary amniotic bag area. The method of claim 1, wherein the second area estimation unit
an abdominal wall area estimator implemented with a pre-learned artificial neural network and performing a neural network operation on the input image to detect the abdominal wall area;
a reverberation range setting unit configured to set a range corresponding to the thickness of the abdominal wall region from the boundary of the estimated abdominal wall region in the traveling direction of the ultrasonic waves as a reverberation artifact possible region where reverberation artifacts may occur;
a reverberation feature map acquisition unit acquiring a reverberation feature map in which reverberation artifacts are emphasized in a region corresponding to the reverberation artifact possible region of the input image according to the generation characteristics of the reverberation artifact; and
AFI measuring device comprising a second area detection unit implemented as a pre-learned artificial neural network and detecting the second area by performing a neural network operation on the input image and the reverberation feature map. The method of claim 2, wherein the reverberation feature map obtaining unit
An AFI measurement device for acquiring the reverberation feature map by performing a Hadamard product of the input image and the reverberation artifact possible region, and differentiating along a traveling direction of ultrasonic waves. The method of claim 2, wherein the abdominal wall area estimation unit
math formula

(here

Is the actual abdominal wall area in the input image (I) (

) is the labeled training data and the estimated abdominal wall area (

= Ω _abw ) is the cross-entropy loss function calculated between.)
An AFI measurement device that learns according to The method of claim 2, wherein the second area detection unit
a scale conversion input unit configured to receive the input image and the reverberation feature map, and obtain multiple scale reverberation feature maps by down-scaling the applied reverberation feature map to different sizes;
Features are extracted by encoding each of the input image and the scaled reverberation feature map, and the features extracted from the input image are pooled and scaled down, combined with the features extracted from the scaled reverberation feature map of the corresponding lower scale, and re-encoded a reverberation enhancement feature extraction unit that extracts a feature and decodes a feature extracted at a lower level together with a feature extracted at an upper level to detect a reverberation artifact region; and
and a second area acquiring unit configured to acquire the second area by dividing the reverberation artifact area from the input image. The method of claim 5, wherein the reverberation enhancement feature extraction unit
math formula

(here

is the actual reverberation artifact area (

) is extracted according to the training data and scale (k) of the labeled input image (I _rvb ).

) is the cross-entropy loss function calculated between.)
An AFI measurement device that learns according to The method of claim 1, wherein the first area estimation unit
a multi-scaling unit receiving the input image and down-scaling it to different sizes to obtain a plurality of scaled input images;
The input image and each of the plurality of scaled input images are encoded by performing a neural network operation according to the learned method, but at a lower scale, a plurality of scale features according to each scale are encoded by pooling and combining features encoded at the upper scale. A feature map extractor that obtains and decodes the encoded features by neural network operation, but obtains a plurality of scale feature maps according to each scale by combining and decoding the feature maps obtained by decoding at the upper scale and the lower scale;
a scale restorer configured to obtain a plurality of sub AF maps by upsampling the plurality of scale feature maps to the size of the input image; and
and a feature map combiner configured to acquire the first area by combining the plurality of sub AF maps. 8. The method of claim 7, wherein the first area estimation unit
math formula

(here

is the actual amniotic sac area (

) is the sub-AF map (

) and the actual amniotic sac area (

) is the cross-entropy loss function calculated between.)
An AFI measurement device that learns according to The method of claim 1, wherein the amniotic bag extraction unit
an estimated region merging unit merging the first region and the second region to obtain the supplemental amniotic bag region; and
An AFI measuring device including a post-processing unit that removes minute errors included in the supplementary amniotic bag region based on the morphological lung structure of the amniotic bag. The method of claim 9, wherein the ultrasound image input unit
Acquiring the input image by receiving the ultrasound image and adjusting it to a predetermined size;
The amniotic sac extraction unit
AFI measurement device further comprising a size adjusting unit for restoring the post-processed supplementary amniotic sac area to the size of the ultrasound image. The method of claim 1, wherein the AFI measuring device
AFI measurement device for obtaining the AFI by measuring the thickness of the supplementary amniotic sac region. acquiring an input image that is an ultrasound image of the amniotic sac;
detecting an area of the amniotic sac by performing a neural network operation on the input image using a pre-learned artificial neural network and extracting the area as a first area;
A neural network operation is performed on the input image using a pre-learned artificial neural network to detect the abdominal wall area, and based on the detected abdominal wall area and the direction of ultrasound waves, an area in which reverberation artifacts occur in the input image is detected to detect the second area. Extracting with; and
and acquiring a supplementary amniotic sac area by merging the first area and the second area. 13. The method of claim 12, wherein the step of extracting to the second area
estimating the abdominal wall area by performing a neural network operation on the input image using a pre-learned artificial neural network;
setting a range equal to the thickness of the abdominal wall region from the boundary of the estimated abdominal wall region in the traveling direction of the ultrasonic waves as a reverberation artifact possible region where reverberation artifacts may occur;
obtaining a reverberation feature map in which reverberation artifacts are emphasized in a region corresponding to the reverberation artifact possible region of the input image according to the occurrence characteristic of the reverberation artifact; and
AFI measuring method comprising detecting the second area by performing a neural network operation on the input image and the reverberation feature map by implementing a pre-learned artificial neural network. 14. The method of claim 13, wherein obtaining the reverberation feature map comprises:
performing a Hadamard product of the input image and the possible reverberation artifact area;
AFI measurement method comprising the step of differentiating a result of Hadamard multiplication along a traveling direction of ultrasound to obtain the reverberation feature map. 15. The method of claim 14, wherein detecting the second area comprises:
receiving the input image and the reverberation feature map and down-scaling the applied reverberation feature map to different sizes to obtain multiple scale reverberation feature maps;
Features are extracted by encoding each of the input image and the scaled reverberation feature map, and the features extracted from the input image are pooled and scaled down, combined with the features extracted from the scaled reverberation feature map of the corresponding lower scale, and re-encoded extracting a feature and decoding a feature extracted at a lower level together with a feature extracted at an upper level to detect a reverberation artifact region; and
and distinguishing the reverberation artifact area from the input image to obtain the second area. 13. The method of claim 12, wherein the step of extracting to the first area
obtaining a plurality of scaled input images by down-scaling the input image to different sizes;
The input image and each of the plurality of scaled input images are encoded by performing a neural network operation according to the learned method, but at a lower scale, a plurality of scale features according to each scale are encoded by pooling and combining features encoded at the upper scale. Obtaining and decoding the encoded features by neural network operation, but obtaining a plurality of scale feature maps according to each scale by combining and decoding the feature maps obtained by decoding at the upper scale at the lower scale;
obtaining a plurality of sub AF maps by upsampling the plurality of scale feature maps to the size of the input image; and
and acquiring the first area by combining the plurality of sub AF maps. 13. The method of claim 12, wherein the obtaining of the supplementary amniotic sac area comprises:
merging the first area and the second area to obtain the supplementary amniotic bag area; and
An AFI measurement method comprising the step of removing minute errors included in the supplementary amniotic sac area based on the morphological pulmonary structure of the amniotic sac. 18. The method of claim 17, wherein acquiring the input image comprises:
Acquiring the input image by receiving the ultrasound image and adjusting it to a predetermined size;
The step of acquiring the supplementary amniotic sac area
The AFI measurement method further comprising restoring the post-processed supplementary amniotic sac region to the size of the ultrasound image. The method of claim 12, wherein the AFI measurement method
The AFI measurement method further comprising obtaining the AFI by measuring a thickness of the supplementary amniotic sac region.

Images