KR102631689B1 - Apparatus and method for measuring amniotic fluid index based on ultrasound images - Google Patents

Abstract

The present invention includes an ultrasound image input unit that acquires an input image, which is an ultrasound image taken of an amniotic fluid sac, and a neural network operation is performed on the input image using a pre-trained artificial neural network to detect the amniotic fluid sac area and extract it as a first area. A first area estimation unit, detects the abdominal wall area by performing a neural network operation on the input image using a pre-trained artificial neural network, and an area where reverberation artifacts occur in the input image based on the detected abdominal wall area and the direction of ultrasound movement. It includes a second area estimation unit that detects and extracts as a second area, and an amniotic sac extraction unit that merges the first area and the second area to obtain a complementary amniotic fluid sac area, and is robust even to reverberation artifacts generated in ultrasound images. Provides an AFI measurement device and method that can accurately detect an amniotic fluid sac.

Description

Apparatus and method for measuring amniotic fluid index based on ultrasound images}

The present invention relates to an amniotic fluid index measuring device and method, and to an ultrasound image-based amniotic fluid index measuring device and method.

Amniotic Fluid (AF) is a protective fluid contained in the amniotic cavity and is an essential component for the development and maturation of the fetus during pregnancy. Amniotic fluid volume (AF volume: AFV) is an important indicator that reflects the progress of pregnancy and fetal development. AFV evaluation is essential during ultrasound examination of pregnant women before childbirth, and AFV is generally measured by AF index (AF index). It is estimated by measurement. Previously, AFI was measured manually while an operator, such as a doctor, performed an ultrasound examination.

Figure 1 is a diagram for explaining the existing manual AFI index measurement method.

As shown in Figure 1, AFI measurement generally divides the amniotic fluid pocket area into four quadrants (Q1, Q2, Q3, Q4), divided into four quadrants as shown on the right. The depth of the amniotic fluid sac measured in each region, that is, the four quadrant amniotic fluid index (y _Q1 , y _Q2 , y _Q3 , y _Q4 ) is summed to obtain AFI (AFI = y _Q1 + y _Q2 + y _Q3) . + y _Q4 ) is obtained. However, such manual measurement not only takes a lot of time, but also has the problem of large errors depending on the skill of the measurer, and mismeasurement of AFI causes incorrect diagnosis of the fetal condition, making early response difficult.

To overcome these problems, research has recently continued on methods to automatically measure AFI from ultrasound images using deep learning techniques. However, due to factors such as the amorphous nature of the amniotic fluid sac, ultrasound reverberation, AF mimic area, suspended matter, and incomplete or missing boundaries of the ultrasound image, there are still limitations in that the accuracy is not high enough to be applied clinically.

In particular, the reverberation artifact of ultrasound, which frequently appears in maternal obesity and advanced gestational age, makes it difficult to distinguish between amniotic fluid sacs and is a major factor that interferes with the measurement of AFI.

Korean Patent Publication No. 10-2021-0002198 (published on January 7, 2021)

The purpose of the present invention is to provide an AFI measurement device and method that can automatically and accurately measure AFI.

Another object of the present invention is to provide an AFI measurement device and method that can improve AFI measurement performance by accurately estimating the amniotic fluid sac area in ultrasound images despite reverberation artifacts.

An AFI measurement device according to an embodiment of the present invention for achieving the above object includes an ultrasound image input unit that acquires an input image that is an ultrasound image taken of an amniotic fluid sac; a first area estimation unit that performs a neural network operation on the input image using a pre-trained artificial neural network to detect the amniotic fluid sac area and extract it as a first area; A neural network operation is performed on the input image using a pre-trained artificial neural network to detect the abdominal wall area, and based on the detected abdominal wall area and the direction of ultrasound, the area where reverberation artifacts occur in the input image is detected to create a second area. a second region estimation unit that extracts; and an amniotic fluid sac extraction unit that merges the first region and the second region to obtain a complementary amniotic fluid sac region.

The second area estimation unit includes an abdominal wall area estimation unit implemented with a pre-trained artificial neural network to detect the abdominal wall area by performing a neural network operation on the input image; a reverberation range setting unit that sets a range corresponding to the thickness of the abdominal wall area from the estimated boundary of the abdominal wall area in the direction in which the ultrasonic wave travels as a reverberation artifact possible area where reverberation artifacts can occur; a reverberation feature map acquisition unit that acquires a reverberation feature map in which reverberation artifacts are emphasized in a region corresponding to the reverberation artifact possible region of the input image according to the occurrence characteristics of the reverberation artifact; and a second area detection unit implemented with a pre-trained artificial neural network to detect the second area by performing a neural network operation on the input image and the reverberation feature map.

The reverberation feature map acquisition unit may obtain the reverberation feature map by performing a Hadamard product on the input image and the reverberation artifact possible area and differentiating the result along the direction of ultrasonic waves.

The second area detection unit includes a scale conversion input unit that receives the input image and the reverberation feature map and downscales the applied reverberation feature map to different sizes to obtain a plurality of scale reverberation feature maps; Features are extracted by encoding each of the input image and the scaled reverberation feature map, the features extracted from the input image are pooled and scaled down, and re-encoded by combining them with the features extracted from the scaled reverberation feature map of the corresponding subscale. a reverberation emphasis feature extraction unit that extracts features and detects a reverberation artifact area by decoding the features extracted at the lower level together with the features extracted at each upper scale; and a second area acquisition unit configured to obtain the second area by distinguishing the reverberation artifact area from the input image.

The first area estimation unit includes a multi-scaling unit that receives the input image and downscales it to different sizes to obtain multiple scale input images; A neural network operation is performed and encoded according to a learned method for each of the input image and the multiple scale input images, and at the lower scale, the features encoded at the higher scale are pooled and combined to encode the multiple scale features for each scale. A feature map extraction unit that acquires and decodes the encoded features by performing a neural network operation, but combines and decodes the feature maps obtained by decoding the lower scale at the higher scale to obtain a plurality of scale feature maps according to each scale; a scale restoration unit configured to obtain a plurality of sub-AF maps by up-sampling the plurality of scale feature maps to the size of the input image; and a feature map combining unit that combines the plurality of sub-AF maps to obtain the first area.

The amniotic fluid sac extraction unit includes an estimated area merging unit that merges the first area and the second area to obtain the complementary amniotic fluid sac area; and a post-processing unit that removes microscopic errors contained in the supplementary amniotic fluid sac area based on the morphological closed structure of the amniotic fluid sac.

The AFI measurement device may obtain the AFI by measuring the thickness of the complementary amniotic fluid sac area.

To achieve the above object, an AFI measurement method according to another embodiment of the present invention includes the steps of acquiring an input image, which is an ultrasound image taken of an amniotic sac;

performing a neural network operation on the input image using a pre-trained artificial neural network to detect the amniotic fluid sac area and extract it as a first area; A neural network operation is performed on the input image using a pre-trained artificial neural network to detect the abdominal wall area, and based on the detected abdominal wall area and the direction of ultrasound movement, the area where reverberation artifacts occur in the input image is detected to create a second area. Extracting with; and merging the first region and the second region to obtain a complementary amniotic fluid sac region.

Therefore, the AFI measurement device and method according to an embodiment of the present invention constructs a dual path network when determining the amniotic fluid sac area for AFI measurement from an ultrasound image, thereby accurately estimating the amniotic fluid sac area including the area where the reverberation artifact occurred, thereby measuring the AFI Allows for accurate estimation.

Figure 1 is a diagram for explaining the existing manual AFI index measurement method.
Figure 2 shows a schematic structure of an AFI measurement device according to an embodiment of the present invention.
FIG. 3 is a diagram for explaining the schematic operation of each component of the AFI measurement device of FIG. 2.
FIG. 4 shows an example of the detailed configuration of the first region estimation unit of FIG. 2.
FIG. 5 shows an implementation example of the first region estimation unit of FIG. 4.
Figure 6 is a diagram illustrating the factors of misdetection when detecting an amniotic fluid sac from an ultrasound image using a single neural network.
Figure 7 shows an example of an amniotic fluid sac area misdetected due to reverberation artifacts.
Figures 8 and 9 are diagrams for explaining areas where reverberation artifacts occur according to the reverberation characteristics of an ultrasonic beam.
FIG. 10 shows an example of the detailed configuration of the second region estimation unit of FIG. 2.
FIG. 11 is a diagram illustrating a schematic operation of each component of the second region estimation unit of FIG. 6.
FIG. 12 shows an example of the detailed configuration of the second area detection unit of FIG. 10.
FIG. 13 shows an implementation example of the second area detection unit of FIG. 12.
Figure 14 shows an example of the detailed configuration of the amniotic fluid bag extractor of Figure 2.
Figure 15 is a diagram for explaining the operation of each component of the amniotic fluid bag extractor of Figure 14.
Figure 16 shows an AFI measurement method according to an embodiment of the present invention.

In order to fully understand the present invention, its operational advantages, and the objectives achieved by practicing the present invention, reference should be made to the accompanying drawings illustrating preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, the present invention will be described in detail by explaining preferred embodiments of the present invention with reference to the accompanying drawings. However, the present invention may be implemented in many different forms and is not limited to the described embodiments. In order to clearly explain the present invention, parts that are not relevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when a part "includes" a certain element, this does not mean excluding other elements, unless specifically stated to the contrary, but rather means that it may further include other elements. In addition, terms such as "... unit", "... unit", "module", and "block" used in the specification refer to a unit that processes at least one function or operation, which is hardware, software, or hardware. and software.

FIG. 2 shows a schematic structure of an AFI measurement device according to an embodiment of the present invention, and FIG. 3 is a diagram for explaining the schematic operation of each component of the AFI measurement device of FIG. 2.

Referring to FIG. 2, the AFI measurement device according to this embodiment includes an ultrasound image input unit 100, a first area estimation unit 200, a second area estimation unit 300, an amniotic fluid sac extraction unit 400, and AFI acquisition. It may include unit 500.

The ultrasound image input unit 100 acquires an ultrasound image as an input image (I) through an ultrasound examination of a pregnant woman, and inputs it to the first area estimation unit 200 and the second area estimation unit 300, respectively.

The ultrasonic image input unit 100 may be implemented as an ultrasonic inspection device, but may also be implemented as a storage device that pre-stores ultrasonic images obtained from the ultrasonic inspection device or a communication module that receives the ultrasonic images.

At this time, the ultrasound image input unit 100 removes unnecessary areas from the acquired ultrasound image or converts the size into an input image (I) of a size that is easy to process in the first and second area estimation units 200 and 300. Preprocessing tasks can also be performed. As an example, as shown in FIG. 3, the ultrasound image input unit 100 may receive an ultrasound image with a size of 600

The first and second area estimation units 200 and 300 are each implemented with a pre-trained artificial neural network to estimate and detect the area corresponding to the amniotic fluid sac in the input image I transmitted from the ultrasound image input unit 100. It operates as an amniotic fluid sac detection unit.

The first area estimation unit 200 performs a neural network operation on the input image I according to a learned method and detects the first area Ω _D corresponding to the entire area of the amniotic sac. And the second area estimation unit 200 selects a second area corresponding to a reverberation artifact area that is difficult for the first area estimation unit 200 to detect in the entire area of the amniotic fluid sac ( ) is detected. That is, the AFI measurement device of this embodiment is implemented with a dual path network for detecting the amniotic fluid sac area from the input image (I). Here, the first area estimation unit 200 is referred to as a primary path (f ^* ) that estimates the entire area of the amniotic fluid sac in the input image (I), and the second area estimation unit 300 is referred to as the first area estimation unit ( It can be called a secondary path (f ^** ) or auxiliary path to make up for the missing area in the amniotic fluid sac area estimated in 200).

Meanwhile, the amniotic fluid sac extractor 400 uses the first area (Ω _D ) estimated by the first area estimation unit 200 and the second area estimated by the second area estimation unit 300 ( ), extract the amniotic fluid sac area corresponding to the amniotic fluid sac from the input image (I). Additionally, the AFI acquisition unit 500 measures and obtains the AFI in the amniotic fluid sac area obtained by the amniotic fluid sac extraction unit 400 in a predetermined manner.

FIG. 4 shows an example of the detailed configuration of the first area estimation unit of FIG. 2, and FIG. 5 shows an example of implementation of the first area estimation unit of FIG. 4.

The first area estimation unit 200 may be implemented with various artificial neural networks, but here, as an example, AF-net (H. C. Cho, S. Sun, C. M. Hyun) is designed to distinguish the amniotic fluid sac area by modifying the existing U-net. , J.-Y. Kwon, B. Kim, Y. Park, and J. K. Seo, "Automated ultrasound assessment of amniotic fluid index using deep learning," Medical Image Analysis, vol. 69, p. 101951, 2021.) It is assumed that this is the case, and Figure 5 shows AF-net as an implementation example.

When the first region estimation unit 200 is implemented as AF-net, the first region estimation unit 200 includes a multi-scaling unit 210, a feature map extraction unit 220, and a scale, as shown in FIG. 4. It may include a restoration unit 230 and a feature map combination unit 240.

When explaining each configuration of the first region estimation unit 200 of FIG. 4 with reference to FIG. 5, first, the multi-scaling unit 210 receives the input image (I) and downscales it to different sizes to generate multiple scales. Obtain input images (η ₁ (I), η ₂ (I), η ₃ (I)). For example, the multi-scaling unit 210 may perform downscaling by performing average pooling.

And the feature map extractor 220 receives the input image (I) and a plurality of scale input images (η ₁ (I), η ₂ (I), η ₃ (I)), and receives the applied input image (I) A neural network operation is performed using atrous convolution according to the learned method for each of the multiple scale input images (η ₁ (I), η ₂ (I), η ₃ (I)). As shown in the lower left corner of Figure 5, Atrus convolution performs a convolution operation by inserting a number of 0s according to a specified rate between adjacent weight elements in the weight matrix of the convolution kernel in an artificial neural network. This is a technique. In Figure 5, as an example, assuming that an atrus convolution operation with a ratio of 4 is performed using a weight matrix of size 3 It can be seen that all elements of the remaining rows and columns are filled with 0. Here, the feature map extractor 220 has a U-net-based structure and is composed of multiple scale encoders and multiple scale decoders, and the highest scale encoder that receives the input image (I) among the multiple scale encoders is the input image (I). Features are extracted by performing a neural network operation only on the image (I). And the remaining scale encoder performs maximum value pooling on the scale input image (η ₁ (I), η ₂ (I), η ₃ (I)) according to the corresponding scale and the features obtained at the upper scale. The features of the same size are received together and the features are extracted. Meanwhile, the smallest scale decoder among the plurality of scale decoders decodes the features applied from the corresponding lowest scale encoder and outputs the lowest scale feature map. And each of the remaining scale decoders performs average unpooling on the features applied from the corresponding scale encoder and the features applied from the lower scale decoder, and decodes the unpooled features with matching sizes to obtain the corresponding scale feature map. do.

The scale restoration unit 230 up-samples the feature maps obtained at different scales from each of the multiple scale decoders of the feature map extraction unit 220 so that each of them has the same size. Here, the up-sampled feature map may be up-sampled to the size of the input image (I). And the scale restoration unit 230 applies a predetermined activation function to each of the up-sampled feature maps to map the sub-AF map ( ) (where k is the scale index, and in the example of Figure 5, k = {1, ..., 4}) is obtained. Here, as an example, a sigmoid function may be used as the activation function.

The feature map combining unit 240 includes a plurality of sub AF maps ( ) is combined to obtain the first area (Ω _D ) by distinguishing the area corresponding to the amniotic fluid sac in the input image (I).

As described above, learning about the operation (f ^* ) of the first region estimation unit 200, which is implemented as AF-net and extracts the amniotic fluid sac region from the input image (I), can be performed using Equation 1.

here is the actual amniotic fluid sac area ( ) is the sub-AF map ( ) and the actual amniotic fluid sac area ( ) is a cross-entropy loss function calculated between Equation 2.

Here, Ω represents the set of pixels (I ^(j) ) in the input image (I).

Since AF-net is a known technology, it will not be described in detail here.

Meanwhile, as described above, the second area estimation unit 300 generates a second area corresponding to a reverberation artifact area that is difficult for the first area estimation unit 200 to detect. ) is detected. In this embodiment, the AFI measurement device further includes not only the first area estimation unit 200 but also the second area estimation unit 300 to estimate the amniotic fluid sac area. The first area learned to detect the entire amniotic fluid sac area This is because the estimation unit 200 alone cannot accurately estimate the amniotic fluid sac area due to artifacts included in the ultrasound image, which is the input image (I).

Because of this, even though the first region estimation unit 200 is implemented as an AF-net specialized for detecting the amniotic fluid sac region and is trained in advance to detect the amniotic fluid sac region, various artifacts included in the input image (I) Therefore, it is well known from existing research that the first area estimation unit 200 has limitations in accurately estimating the amniotic fluid sac area. In other words, it is very difficult to prevent misdetection of the amniotic fluid sac area due to artifacts using only a single neural network path.

Figure 6 is a diagram illustrating the factors causing misdetection when detecting an amniotic fluid sac from an ultrasound image using a single neural network.

As shown in FIG. 6, ultrasound images may include various artifacts. In Figure 6, (a) shows reverberation artifacts according to ultrasound characteristics, (b) shows artifacts according to the AF mimicking region, and (c) shows artifacts due to floating matters. Represents an artifact. And (d) represents artifacts caused by incompleteness or missing boundaries of the ultrasound image.

Among the various artifacts shown in FIG. 6, reverberation artifacts that are different from the characteristics of ultrasound examination are the biggest obstacle that makes it difficult to accurately detect the amniotic fluid sac area.

Figure 7 shows an example of an amniotic fluid sac area misdetected due to reverberation artifacts.

Figure 7 shows the amniotic fluid sac area incorrectly detected in cases (Case 1 and Case 2) for two input images (I). In Figure 7, (a) represents the input image (I), (b) represents the actual area of the amniotic fluid sac and the reverberation artifact area misdetected due to the reverberation artifact, and (c) represents the first area estimation unit 200. represents the estimated amniotic fluid sac area. In (b) of Figure 7, the orange outline represents the actual amniotic fluid sac area, the cyan dotted line represents the area where reverberation artifacts occurred, and the green area in (c) represents the amniotic fluid sac area estimated by the first area estimation unit 200. indicates.

Comparing (b) and (c) of FIG. 7, even though the first area estimation unit 200 implemented with an artificial neural network is trained to estimate the amniotic fluid sac, the area where the reverberation artifact occurred can be detected as the amniotic fluid sac area. You can see that there is no.

Accordingly, in this embodiment, separately from the first neural network path of the first area estimation unit 200, a second area (which is an area where reverberation artifacts occurred in the amniotic fluid sac) is used. ) is further provided with a second region estimation unit 300, which is a second neural network path that detects .

Figures 8 and 9 are diagrams for explaining areas where reverberation artifacts occur according to the reverberation characteristics of an ultrasonic beam.

In Figure 8, (a) is a diagram for explaining the cause of reverberation artifacts, (b) shows the relationship between the abdominal wall area and the reverberation artifact area in an actual ultrasound image, and Figure 9 shows the reverberation artifact area. Indicates areas where contamination is likely to occur.

Significant reverberation artifacts occur when the ultrasound beam encounters a strong reflector, such as the fat-muscle interface of the abdominal wall, approximately orthogonal to the direction of beam propagation. As shown in (a) of FIG. 8, these reverberation artifacts are formed in the form of a plurality of parallel curves (rev1 to rev3) at uniform intervals (d) perpendicular to the ultrasonic propagation direction. Therefore, as shown in (b) of FIG. 8, the reverberation artifact appears on the abdominal wall opposite to the direction in which the ultrasound waves are applied, that is, on the side in the direction in which the ultrasound waves travel.

In Figure 9, areas where reverberation artifacts appear in several ultrasound images are indicated with dotted lines. Since reverberation artifacts are generated by the fat-muscle boundary, as shown in Figures 9 (a) to (d), it can be seen that when the ultrasound images are different, the area where the reverberation artifacts occur also appears differently.

However, as described above, since the reverberation artifact is formed in the form of multiple parallel curves (rev1 to rev3) at uniform intervals (d) perpendicular to the ultrasound propagation direction, the reverberation artifact is the thickness (d _abw ) of the abdominal wall region (Ω _abw ). It appears in a range of thickness (d _rvb ) that is almost the same as (d _rvb ≒ d _abw ). Therefore, if the thickness (d _abw ) is confirmed by detecting the abdominal wall area (Ω _abw ), it can be seen that reverberation artifacts occur within the range of the thickness (d _rvb ) corresponding to the abdominal wall thickness (d _abw ) from the boundary of the abdominal wall. . In other words, a reverberation artifact possible area (Ω _rvb ) in which reverberation artifacts may occur can be set. In Figure 8(b), the purple arrows, white arrows, and blue arrows represent reverberation artifacts of the fascia, amniotic sac, and peritoneal-fascial layer, respectively.

Accordingly, in this embodiment, the second area estimation unit 300 detects the abdominal wall area and easily detects the reverberation artifact area based on the detected abdominal wall area.

FIG. 10 shows an example of the detailed configuration of the second area estimation unit of FIG. 2, and FIG. 11 is a diagram for explaining the schematic operation of each component of the second area estimation unit of FIG. 6.

Referring to FIG. 10, the second area estimation unit 300 may include an abdominal wall area estimation unit 310, a reverberation range setting unit 320, a reverberation feature map acquisition unit 330, and a second area detection unit 340. there is.

The abdominal wall area estimation unit 310 receives the input image (I) and performs a neural network operation according to a previously learned method to detect the abdominal wall area (Ω _abw ). As described above, the second area estimation unit 300 is provided with the abdominal wall area estimation unit 310 to detect the abdominal wall area (Ω _abw ) because the abdominal wall area (Ω _abw ) is clearer than the reverberation artifact possible area (Ω _rvb ). This is because it has morphological characteristics and is easy to extract.

The abdominal wall area estimation unit 310 may be implemented with various artificial neural networks, but here, for example, it is assumed to be implemented with a U-net that has been previously trained to detect the abdominal wall area. Similar to the AF-net of the first area estimation unit 200 described above, U-net accurately extracts the abdominal wall area (Ω _abw ) by converting the input image (I) to various scales and extracting and combining features for each scale. do.

Here, the operation of the abdominal wall area estimation unit 310 implemented in U-net ( ) can be learned in advance according to Equation 3.

here is the actual abdominal wall area in the input image (I) ( ) labeled training data and the estimated abdominal wall area ( = Ω _abw ) is the cross-entropy loss function calculated between

The reverberation range setting unit 320 measures the thickness (d _abw ) of the abdominal wall area (Ω _abw ) estimated by the abdominal wall area estimation unit 310, and determines the direction (r) of ultrasonic waves from the boundary of the abdominal wall area (Ω _abw ). The range of thickness (Ω _rvb ) equal to the abdominal wall thickness (d _abw ) measured in the direction is set as the reverberation artifact possible area (Ω _rvb ). Here, the reverberation artifact possible area (Ω _rvb ) is a kind of misk, and the pixel values of the remaining areas excluding the reverberation artifact possible area (Ω _rvb ) may be set to 0.

The reverberation feature map acquisition unit 330 acquires a reverberation feature map containing reverberation artifacts based on the input image (I) and the set derived reverberation artifact possible area (Ω _rvb ). The reverberation feature map acquisition unit 330 converts the input image (I) and the set reverberation artifact possible area (Ω _rvb ) into a Hadamard product ( ), and differentiate along the direction (r) of ultrasonic waves ( ) to obtain a reverberation feature map.

The reverberation artifact possible area (Ω _rvb ) simply represents an area where reverberation artifacts can occur and does not represent the area where reverberation artifacts are actually generated in the amniotic fluid sac. Accordingly, the reverberation feature map acquisition unit 330 obtains a reverberation feature map that highlights and displays the area where the reverberation artifact occurs in the actual amniotic fluid sac by differentiating it in the direction (r) of the ultrasound due to the nature of the reverberation artifact.

The second area detector 340 receives the input image (I) and the reverberation feature map, and based on the applied reverberation feature map, determines the reverberation artifact area ( ) is detected, and the detected reverberation artifact area ( ) is divided from the input image into a second region ( ) to obtain.

FIG. 12 shows an example of the detailed configuration of the second area detection unit of FIG. 10, and FIG. 13 shows an implementation example of the second area detection unit of FIG. 12.

Referring to FIG. 12 , the second area detection unit 340 according to this embodiment may include a scale conversion input unit 341, a reverberation enhancement feature extraction unit 342, and a second area acquisition unit 343.

Referring to FIG. 13, when explaining the operation of each component of the second area detection unit 340, first, the scale conversion input unit 341 receives the input image (I) and the reverberation feature map, and receives the applied reverberation feature map. Multiple scale reverberation feature maps are obtained by downscaling to different sizes.

The reverberation emphasis feature extractor 342 is implemented as an artificial neural network and identifies and detects the reverberation artifact area in the input image (I) according to a learned method. As an example, the reverberation emphasis feature extraction unit 342 extracts a reverberation artifact area ( ) can be implemented as an artificial neural network modified to detect ), herein referred to as RVB-net.

As shown in FIG. 13, the reverberation emphasis feature extraction unit 342 extracts features by encoding each of the input image (I) and the scaled reverberation feature map, and performs maximum value pooling on the features extracted from the input image (I). While scaling down by applying , the features are extracted by combining them with the features extracted from the corresponding scale reverberation feature map and re-encoding them. In other words, the features to be extracted from the upper scale are transferred to the lower scale and combined to extract the features. Then, the features extracted at the lower level are decoded together with the features extracted at each scale to determine the reverberation artifact area ( ) is detected. At this time, by applying an attention gate (AG) as shown in FIG. 13 during the decoding process, the reverberation emphasis feature extractor 342 focuses only on the area of the reverberation feature map and creates a reverberation artifact area ( ) can be detected.

Operation of the reverberation emphasis feature extraction unit 342 implemented with U-net-based RVB-net ( ) can be learned according to Equation 4.

here is the actual reverberation artifact area ( ) is the training data of the labeled input image (I _rvb ) and the reverberation artifact area ( ) is a cross-entropy loss function calculated between

The reverberation emphasis feature extraction unit 342 may be composed of another artificial neural network, and an implementation example of the attention gate (AG) is also shown in FIG. 13, so detailed description is omitted here.

The second area acquisition unit 343 allows the reverberation emphasis feature extraction unit 342 to extract the reverberation artifact area (I) from the input image (I). ), the detected reverberation artifact area ( ) is divided into a second area ( ) to obtain.

FIG. 14 shows an example of the detailed configuration of the amniotic fluid bag extractor of FIG. 2, and FIG. 15 is a diagram for explaining the operation of each component of the amniotic fluid bag extractor of FIG. 14.

The amniotic fluid sac extraction unit 400 may include an estimated region merging unit 410, a post-processing unit 420, and a size adjusting unit 430. The estimated area merge unit 410 divides the area estimated to be the amniotic fluid sac in the first area estimation unit 200 and obtains a first region (Ω _D ) and a reverberation artifact among the amniotic fluid sac in the second area estimation unit 300. The second area obtained by dividing into the area where occurred ( ) are merged to obtain a complementary amniotic fluid sac area, as shown on the left of Figure 15.

And the post-processing unit 420 removes holes indicated by orange arrows and false positive errors indicated by yellow arrows in FIG. 15 based on the morphological closed structure of the amniotic fluid sac. Comparing (a) and (b), the first area (Ω _D ) and the second area ( It can be seen that minor errors that occurred even in the merge of ) have been removed.

The size adjustment unit 430 converts the image of the complementary amniotic fluid sac area from which the error has been removed to the size of the original ultrasound image and outputs it.

And the AFI acquisition unit 500 acquires the AFV by measuring the width, that is, the thickness, of the amniotic fluid sac area, as shown on the right in FIG. 15, and measuring the AFI indicating the amniotic fluid depth in the corresponding quadrant, as in FIG. 1.

As a result, the AFI measurement method according to this embodiment includes a first area estimation unit 200 that estimates the entire amniotic fluid sac area, and an area in which reverberation artifacts can occur based on the abdominal wall based on the principle that reverberation artifacts occur in ultrasound images. It is provided with a second estimation unit 300 that detects and estimates the area where the actual reverberation artifact occurs within the detected area where the reverberation artifact can occur, and the area estimated by the first and second area estimation units 200 and 300 is provided. By merging, the amniotic fluid sac area can be extracted from the ultrasound image with high accuracy, and AFI can be measured from the extracted amniotic fluid sac area.

Figure 16 shows an AFI measurement method according to an embodiment of the present invention.

Referring to FIGS. 1 to 15 , the AFI measurement method of FIG. 16 will be described. First, in order to measure AFI, an input image (I) is acquired based on an ultrasound image of a pregnant woman (S10). Then, a neural network operation is performed on the input image (I) obtained using an artificial neural network previously trained to detect the amniotic fluid sac area to extract the first area (Ω _D ) (S20). Along with the step of detecting the first area, the second area ( ) is performed (S30).

In the step of extracting the second area (S30), first, a neural network operation is performed on the input image (I) using an artificial neural network previously trained to extract the abdominal wall area from the input image (I) to extract the abdominal wall area (Ω _abw ). Detect (S31). Then, an area equal to the thickness (d _abw ) of the abdominal wall area (Ω _abw ) in the direction (r) of the ultrasound from the boundary of the detected abdominal wall area (Ω _abw ) is set as the reverberation artifact possible area (Ω _rvb ) (S32) . Afterwards, the input image (I) and the possible reverberation artifact area (Ω _rvb ) are combined with the Hadamard product ( ), and differentiated according to the direction (r) of ultrasonic waves ( ) to obtain a reverberation feature map in which reverberation artifacts are emphasized (S33). A neural network operation is performed according to a method learned for the input image (I) and the reverberation feature map to detect the reverberation artifact area emphasized by the reverberation feature map in the input image (I), thereby creating a second area ( ) is detected.

The first region (Ω _D ) and the second region ( ) is extracted, the extracted first area (Ω _D ) and the second area (Ω D) are extracted. ) are merged to obtain the complementary amniotic fluid sac area (Ω). Then, the AFV is calculated by measuring the thickness of the acquired amniotic fluid sac area (Ω), that is, the AFI indicating the depth of the amniotic fluid sac (S50).

The method according to the present invention can be implemented as a computer program stored on a medium for execution on a computer. Here, computer-readable media may be any available media that can be accessed by a computer, and may also include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data, including read-only memory (ROM). It may include dedicated memory), RAM (random access memory), CD (compact disk)-ROM, DVD (digital video disk)-ROM, magnetic tape, floppy disk, optical data storage device, etc.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely illustrative, and those skilled in the art will understand that various modifications and other equivalent embodiments are possible therefrom.

Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the attached claims.

100: Ultrasound image input unit 200: First area estimation unit
210: Multi-scaling unit 220: Feature map extraction unit
230: scale restoration unit 240: feature map coupling unit
300: Second area estimation unit 310: Abdominal wall area estimation unit
320: Reverberation range setting unit 330: Reverberation feature map acquisition unit
340: Second area detection unit 341: Scale conversion input unit
342: Reverberation emphasis feature extraction unit 343: Second area acquisition unit
400: Amniotic fluid sac extraction unit 410: Estimated area merging unit
420: Post-processing unit 430: Size control unit
500: AFI Acquisition Department

Claims (19)

An ultrasound image input unit that acquires an input image that is an ultrasound image taken of the amniotic fluid sac;
a first area estimation unit that performs a neural network operation on the input image using a pre-trained artificial neural network to detect the amniotic fluid sac area and extract it as a first area;
A neural network operation is performed on the input image using a pre-trained artificial neural network to detect the abdominal wall area, and based on the detected abdominal wall area and the direction of ultrasound movement, the area where reverberation artifacts occur in the input image is detected to create a second area. a second region estimation unit extracted by; and
An amniotic fluid index (AFI) measuring device comprising an amniotic fluid sac extraction unit that merges the first area and the second area to obtain a complementary amniotic fluid sac area. The method of claim 1, wherein the second area estimation unit
an abdominal wall area estimation unit implemented with a pre-trained artificial neural network to detect the abdominal wall area by performing a neural network operation on the input image;
a reverberation range setting unit that sets a range corresponding to the thickness of the abdominal wall area from the estimated boundary of the abdominal wall area in the direction in which the ultrasonic wave travels as a reverberation artifact possible area where reverberation artifacts can occur;
a reverberation feature map acquisition unit that acquires a reverberation feature map in which reverberation artifacts are emphasized in a region corresponding to the reverberation artifact possible region of the input image according to the occurrence characteristics of the reverberation artifact; and
An AFI measurement device including a second area detection unit implemented with a pre-trained artificial neural network and detecting the second area by performing a neural network operation on the input image and the reverberation feature map. The method of claim 2, wherein the reverberation feature map acquisition unit
An AFI measurement device that obtains the reverberation feature map by performing a Hadamard product on the input image and the reverberation artifact possible area and differentiating it along the direction of ultrasonic waves. The method of claim 2, wherein the abdominal wall area estimation unit
math equation

(here is the actual abdominal wall area in the input image (I) ( ) labeled training data and the estimated abdominal wall area ( = Ω _abw ) is the cross-entropy loss function calculated between
AFI measurement device trained according to. The method of claim 2, wherein the second area detection unit
a scale conversion input unit that receives the input image and the reverberation feature map and downscales the applied reverberation feature map to different sizes to obtain a plurality of scale reverberation feature maps;
Features are extracted by encoding each of the input image and the scaled reverberation feature map, the features extracted from the input image are pooled and scaled down, and re-encoded by combining them with the features extracted from the scaled reverberation feature map of the corresponding subscale. a reverberation emphasis feature extraction unit that extracts features and detects a reverberation artifact area by decoding the features extracted at the lower level together with the features extracted at each upper scale; and
An AFI measurement device comprising a second area acquisition unit configured to obtain the second area by distinguishing the reverberation artifact area from the input image. The method of claim 5, wherein the reverberation enhancement feature extractor
math equation

(here is the actual reverberation artifact area ( ) is the training data of the labeled input image (I _rvb ) and the reverberation artifact area ( ) is a cross-entropy loss function calculated between ).
AFI measurement device trained according to. The method of claim 1, wherein the first area estimation unit
a multi-scaling unit that receives the input image and downscales it to different sizes to obtain multiple scale input images;
A neural network operation is performed and encoded according to a learned method for each of the input image and the multiple scale input images, and at the lower scale, the features encoded at the higher scale are pooled and combined to encode the multiple scale features for each scale. A feature map extraction unit that acquires and decodes the encoded features by performing a neural network operation, but combines and decodes the feature maps obtained by decoding the lower scale at the higher scale to obtain a plurality of scale feature maps according to each scale;
a scale restoration unit configured to obtain a plurality of sub-AF maps by up-sampling the plurality of scale feature maps to the size of the input image; and
An AFI measurement device including a feature map combining unit that combines the plurality of sub AF maps to obtain the first area. The method of claim 7, wherein the first area estimation unit
math equation

(here is the actual amniotic fluid sac area ( ) is the sub-AF map ( ) and the actual amniotic fluid sac area ( ) is a cross-entropy loss function calculated between ).
AFI measurement device trained according to. The method of claim 1, wherein the amniotic fluid bag extractor
an estimated area merging unit that merges the first area and the second area to obtain the complementary amniotic fluid sac area; and
An AFI measurement device including a post-processing unit that removes microscopic errors contained in the supplementary amniotic fluid sac area based on the morphological lung structure of the amniotic fluid sac. The method of claim 9, wherein the ultrasound image input unit
Obtaining the input image by receiving the ultrasound image and adjusting it to a predetermined size,
The amniotic fluid sac extraction unit
An AFI measurement device further comprising a size adjustment unit that restores the post-processed complementary amniotic fluid sac area to the size of the ultrasound image. The method of claim 1, wherein the AFI measurement device
An AFI measurement device that obtains the AFI by measuring the thickness of the complementary amniotic fluid sac area. Acquiring an input image, which is an ultrasound image taken of the amniotic sac;
performing a neural network operation on the input image using a pre-trained artificial neural network to detect the amniotic fluid sac area and extract it as a first area;
A neural network operation is performed on the input image using a pre-trained artificial neural network to detect the abdominal wall area, and based on the detected abdominal wall area and the direction of ultrasound, the area where reverberation artifacts occur in the input image is detected to create a second area. Extracting with; and
Amniotic Fluid Index (AFI) measuring method comprising the step of merging the first area and the second area to obtain a complementary amniotic fluid sac area. The method of claim 12, wherein the step of extracting to the second area is
Estimating the abdominal wall area by performing a neural network operation on the input image using a pre-trained artificial neural network;
Setting a range equal to the thickness of the abdominal wall region from the estimated boundary of the abdominal wall region in the direction of travel of the ultrasound as a reverberation artifact possible region where reverberation artifacts can occur;
Obtaining a reverberation feature map in which reverberation artifacts are emphasized in a region corresponding to the reverberation artifact possible region of the input image according to the occurrence characteristics of the reverberation artifacts; and
An AFI measurement method implemented with a pre-trained artificial neural network and comprising detecting the second area by performing a neural network operation on the input image and the reverberation feature map. The method of claim 13, wherein the step of acquiring the reverberation feature map is
performing a Hadamard product between the input image and the possible reverberation artifact area;
AFI measurement method comprising differentiating the Hadamard multiplication result along the direction of ultrasonic waves to obtain the reverberation feature map. The method of claim 14, wherein detecting the second area includes
Obtaining a plurality of scaled reverberation feature maps by receiving the input image and the reverberation feature map and downscaling the applied reverberation feature map to different sizes;
Features are extracted by encoding each of the input image and the scaled reverberation feature map, the features extracted from the input image are pooled and scaled down, and re-encoded by combining them with the features extracted from the scaled reverberation feature map of the corresponding subscale. Extracting features and decoding the features extracted at the lower level together with the features extracted at each upper scale to detect a reverberation artifact area; and
AFI measurement method comprising dividing the reverberation artifact area in the input image to obtain the second area. The method of claim 12, wherein the step of extracting into the first area is
Obtaining a plurality of scaled input images by receiving the input image and downscaling it to different sizes;
A neural network operation is performed and encoded according to a learned method for each of the input image and the multiple scale input images, and at the lower scale, the features encoded at the higher scale are pooled and combined to encode the multiple scale features for each scale. Obtaining and decoding the encoded features by performing a neural network operation, but combining and decoding the feature maps obtained by decoding at the lower scale at the higher scale to obtain a plurality of scale feature maps according to each scale;
Upsampling the plurality of scale feature maps to the size of the input image to obtain a plurality of sub AF maps; and
AFI measurement method comprising obtaining the first area by combining the plurality of sub AF maps. The method of claim 12, wherein obtaining the complementary amniotic fluid sac area comprises:
Obtaining the complementary amniotic fluid sac area by merging the first area and the second area; and
A method of measuring AFI comprising the step of removing microscopic errors contained in the complementary amniotic fluid sac area based on the morphological lung structure of the amniotic fluid sac. The method of claim 17, wherein the step of acquiring the input image is
Obtaining the input image by receiving the ultrasound image and adjusting it to a predetermined size,
The step of obtaining the complementary amniotic fluid sac area is
AFI measurement method further comprising restoring the post-processed complementary amniotic fluid sac area to the size of the ultrasound image. The method of claim 12, wherein the AFI measurement method is
AFI measurement method further comprising obtaining the AFI by measuring the thickness of the complementary amniotic fluid sac area.