KR20240018125A - Method for providing standard planar information about object and device for performing the same - Google Patents

Abstract

The present invention is a method of providing standard planar information about an object, the method comprising the steps of acquiring an ultrasound image of a fetus, determining the head of the fetus in the ultrasound image, and determining the head area in the ultrasound image of the head. Predicting a preset head region classification result among the ultrasound images of the head using a first prediction model learned to classify, a second learning to predict a standard plane probability using the ultrasound image of the head region as input Inputting an ultrasound image for a head region classified according to the classification result into a prediction model to predict a standard plane probability of the ultrasound image, and determining a standard plane for the head of the fetus based on the standard plane probability. It is structured to include steps.

Description

Method for providing standard plan information for an object and device for performing the same {METHOD FOR PROVIDING STANDARD PLANAR INFORMATION ABOUT OBJECT AND DEVICE FOR PERFORMING THE SAME}

The present invention relates to a method for providing standard plane information to an object and a device for performing the same.

Fetal Central Nervous System (CNS) malformations are one of the major causes of prenatal mortality and childhood morbidity rates. Accordingly, all pregnant women typically undergo ultrasound evaluation of the fetus for central nervous system malformations.

Ultrasound is mainly used to evaluate central nervous system malformations in the fetus, and examination and evaluation are performed manually by medical staff. However, because ultrasound examination is complicated and requires various key inputs and probe movements, there is a large variation in evaluation accuracy depending on the skill level of the medical staff.

Additionally, in order to evaluate central nervous system malformations, three standard planes (transthalamic plane, transventricular plane, and transcerebellar plane) must be explored. However, since the position or direction of the fetal head is not fixed, it is difficult to obtain the desired plane. Scans must be performed repeatedly, which takes a lot of time even for experienced medical staff.

The technology behind the invention has been written to facilitate easier understanding of the invention. It should not be understood as an admission that matters described in the technology underlying the invention exist as prior art.

Accordingly, a method has been disclosed to automatically obtain a cross-section of a diagnostic object (e.g., the subject's head) using a conventional machine-learned model, but the conventional method is configured to find a suitable cross-section using only a single frame of an ultrasound image. Therefore, there are cases where there are discrepancies with the actual test results of medical staff.

Accordingly, there is a need for a method that can more accurately determine the standard plane of an object during an ultrasound examination.

As a result, the inventors of the present invention have developed a method for determining the standard plane of an object by considering structural changes (context information) of the object obtained based on successive frames in addition to the anatomical structure clearly visible in a single frame of an ultrasound image. and attempted to develop a device that performs this.

In particular, the inventors of the present invention have developed a method that can accurately determine the standard plane of an object by learning the structural characteristics of the object according to the imaging direction of the probe while an ultrasound image of the object is captured through ultrasound examination.

The problems of the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

In order to solve the problems described above, a method of providing standard plane information about an object according to an embodiment of the present invention is provided. The method includes obtaining an ultrasound image of the fetus, determining the head of the fetus in the ultrasound image, using a first prediction model learned to classify the head region in the ultrasound image of the head, Predicting a preset head region classification result among ultrasound images for the head region, using the ultrasound image for the head region as input to a second prediction model trained to predict a standard plane probability, and applying the head region classified according to the classification result to the second prediction model. It is configured to include a step of predicting a standard plane probability of the ultrasound image by inputting an ultrasound image, and determining a standard plane for the head of the fetus based on the standard plane probability.

According to a feature of the present invention, the step of determining the head includes detecting the head of the fetus by inputting the ultrasound image of the fetus into a third prediction model learned to detect the head by inputting the ultrasound image. It may further include determining an ellipse corresponding to the head of the fetus using the results of ellipse fitting or curve fitting for the boundary line of the detected head.

According to another feature of the present invention, the step of determining the head may further include extracting a patch including the determined ellipse from the ultrasound image.

According to another feature of the present invention, the step of predicting the head region classification result includes an index map (Structured Dissimilarity Index Measure Map) indicating structural changes in the head based on a plurality of frames constituting the ultrasound image of the head. , DISSM Map) and inputting the ultrasound image of the head and the index map into the first prediction model to predict a classification result of the head region of the fetus.

According to another feature of the present invention, the first prediction model includes two different feature extraction layers configured to extract feature maps from each of the ultrasound image of the head and the index map, the two features It may include an integration layer configured to integrate two different feature maps extracted from the extraction layer and a classification layer configured to output a classification vector for classifying the head region of the fetus based on the results output from the integration layer. .

According to another feature of the present invention, the head region classification result includes a first region including a standard TV plane (Transventricular plane) and a standard TT plane (Transthalamic plane), and a second region including a standard TC plane (Transcerebella plane). It may include a classification result for at least one area among the area and a third area excluding the first area and the second area.

According to another feature of the present invention, the step of predicting the standard plane probability includes predicting the standard plane probability centered on different points based on ultrasound images of head regions classified into a plurality according to the classification result. It may be a step.

According to another feature of the present invention, the step of predicting the standard plane probability includes probabilities for two or more standard planes among the standard TV plane (Transventricular plane), the standard TT plane (Transthalamic plane), and the standard TC plane (Transcerebella plane). It may be a step to predict.

According to another feature of the present invention, the second prediction model includes a feature extraction layer configured to extract a feature map from an ultrasound image for the classified head region, and forward and reverse outputs of the feature map extracted from the feature extraction layer. An integration layer configured to integrate the results, an attention layer configured to extract structural feature points of the fetus based on the results output from the integration layer, and configured to output a probability for the standard plane based on the output results and the feature points. May include a classification layer.

According to another feature of the present invention, the integration layer includes a recurrent neural network, and the recurrent neural network may be composed of a bi-directional gated recurrent unit (GRU) or a bi-directional long short term memory (LSTM). You can.

In order to solve the problems described above, a device for providing standard planar information according to another embodiment of the present invention is provided. The device includes a communication interface, a memory, and a processor operably connected to the communication interface and the memory, wherein the processor acquires an ultrasound image of the fetus, determines the head of the fetus from the ultrasound image, and Predict a preset head region classification result among the ultrasound images of the head using a first prediction model learned to classify the head region in the ultrasound image of the head, and use the ultrasound image of the head region as an input to create a standard plane An ultrasound image for the head region classified according to the classification result is input to a second prediction model trained to predict the probability, the standard plane probability of the ultrasound image is predicted, and the fetal head is based on the standard plane probability. It is configured to determine the standard plane for .

In order to solve the problems described above, a method of providing standard plane information about an object according to another embodiment of the present invention is provided. The method includes obtaining an ultrasound image for an area of an object, determining a diagnosis object in the ultrasound image, and using a first prediction model learned to classify a diagnosis region in an ultrasound image for the diagnosis object. Predicting the result of classifying a preset diagnostic area among ultrasound images for a diagnosis object, a diagnosis classified according to the classification result to a second prediction model learned to predict a standard plane probability using the ultrasound image for the diagnosis area as input. It is configured to include predicting a standard plane probability of the diagnostic object by inputting an ultrasound image for a region, and determining a standard plane for the diagnostic object based on the standard plane probability.

Specific details of other embodiments are included in the detailed description and drawings.

The present invention can predict the standard plane of an object more accurately than a method of predicting the standard plane simply using a single frame of an ultrasound image by learning changes in structural characteristics according to the dynamic movement of the object.

Additionally, the present invention can assist medical staff in quick evaluation (diagnosis) by providing probability information that one frame corresponds to a standard plane through a learned prediction model. In particular, it is possible to minimize the deviation in evaluation accuracy depending on the skill level of the medical staff.

In addition, the present invention can provide highly accurate standard plane information for early prenatal detection of disorders/deformities/expected diseases of fetal organs, and in addition, standards for various body parts and diagnostic areas that can be examined through an ultrasound device. Can provide planar information.

The effects according to the present invention are not limited to the details exemplified above, and further various effects are included within the present invention.

1 is a schematic diagram illustrating an outline of a method for providing standard plane information according to an embodiment of the present invention.
Figure 2 is a schematic diagram of a standard plan information providing system according to an embodiment of the present invention.
Figure 3 is a block diagram showing the configuration of a medical staff device according to an embodiment of the present invention.
Figure 4 is a block diagram showing the configuration of a device for providing standard planar information according to an embodiment of the present invention.
Figure 5 is a schematic flowchart of a method for providing standard plane information of an object according to an embodiment of the present invention.
Figure 6 is a schematic diagram illustrating a method for determining the head area of a fetus according to an embodiment of the present invention.
7A and 7B are schematic diagrams exemplarily showing the structure of a first prediction model used in a method of providing standard plane information of an object according to an embodiment of the present invention.
8A and 8B are schematic diagrams exemplarily showing the structure of a second prediction model used in a method of providing standard plane information of an object according to an embodiment of the present invention.
9A and 9B are schematic diagrams exemplarily showing the results of a method for providing standard plane information of an object according to an embodiment of the present invention.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various different forms, but the present embodiments only serve to ensure that the disclosure of the present invention is complete and are within the scope of common knowledge in the technical field to which the present invention pertains. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. In connection with the description of the drawings, similar reference numbers may be used for similar components.

In this document, expressions such as “have,” “may have,” “includes,” or “may include” refer to the existence of the corresponding feature (e.g., a numerical value, function, operation, or component such as a part). , and does not rule out the existence of additional features.

In this document, expressions such as “A or B,” “at least one of A or/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, “A or B,” “at least one of A and B,” or “at least one of A or B” includes (1) at least one A, (2) at least one B, Or (3) it may refer to all cases including both at least one A and at least one B.

Expressions such as “first,” “second,” “first,” or “second,” used in this document can modify various components regardless of order and/or importance, and can refer to one component. It is only used to distinguish from other components and does not limit the components. For example, a first user device and a second user device may represent different user devices regardless of order or importance. For example, the first component may be renamed as the second component without departing from the scope of rights described in this document, and similarly, the second component may also be renamed as the first component.

A component (e.g., a first component) is “(operatively or communicatively) coupled with/to” another component (e.g., a second component). When referred to as being “connected to,” it should be understood that any component may be directly connected to the other component or may be connected through another component (e.g., a third component). On the other hand, when a component (e.g., a first component) is said to be “directly connected” or “directly connected” to another component (e.g., a second component), It may be understood that no other component (e.g., a third component) exists between other components.

As used in this document, the expression “configured to” depends on the situation, for example, “suitable for,” “having the capacity to.” ," can be used interchangeably with "designed to," "adapted to," "made to," or "capable of." The term “configured (or set to)” may not necessarily mean “specifically designed to” in hardware. Instead, in some contexts, the expression “a device configured to” may mean that the device is “capable of” working with other devices or components. For example, the phrase "processor configured (or set) to perform A, B, and C" refers to a processor dedicated to performing the operations (e.g., an embedded processor), or by executing one or more software programs stored on a memory device. , may refer to a general-purpose processor (e.g., CPU or application processor) capable of performing the corresponding operations.

Terms used in this document are merely used to describe specific embodiments and may not be intended to limit the scope of other embodiments. Singular expressions may include plural expressions, unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by a person of ordinary skill in the technical field described in this document. Among the terms used in this document, terms defined in general dictionaries may be interpreted to have the same or similar meaning as the meaning they have in the context of related technology, and unless clearly defined in this document, have an ideal or excessively formal meaning. It is not interpreted as In some cases, even terms defined in this document cannot be interpreted to exclude embodiments of this document.

Each feature of the various embodiments of the present invention can be partially or fully combined or combined with each other, and as can be fully understood by those skilled in the art, various technical interconnections and operations are possible, and each embodiment may be implemented independently of each other. It may be possible to conduct them together due to a related relationship.

For clarity of interpretation of this specification, terms used in this specification will be defined below.

As used herein, the term “subject” may refer to any subject who wishes to receive information about a diagnostic area through ultrasound examination. For example, the subject may be a mother and the diagnostic area may be the uterus. For another example, the subject may be a patient, and the diagnostic area may be the abdomen or leg joints. Meanwhile, the entity disclosed in this specification may be any mammal except human, but is not limited thereto.

The term “ultrasound image” used in this specification may be a video consisting of a plurality of cuts (frames). Accordingly, the present invention can provide standard planar information prediction results in the form of a streaming service at the same time as receiving an ultrasound image from a device for providing standard planar information.

As used herein, the term “first prediction model” may be a model learned to predict a preset head region classification result using an ultrasound image of the head and an index map as input. For example, the head region classification result may be the result of predicting whether any one frame constituting the ultrasound image corresponds to any one of the first to third regions. However, the preset region classification results here may differ depending on the object and diagnosis region.

As used herein, the term “second prediction model” may be a model learned to predict the standard plane probability using an ultrasound image of the head region as input. For example, the second prediction model may predict the standard plane probability for each preset head region, and one frame with the highest probability value may be determined as the standard plane for the head.

The term “third prediction model” used herein may be a model learned to detect the head using ultrasound images as input. For example, the third prediction model can segment the oval-shaped fetal head within the frames that make up the ultrasound image.

Meanwhile, the first and second prediction models may be models including a recurrent neural network such as a bi-directional gated recurrent unit (GRU) or a bi-directional long short term memory (LSTM). Additionally, the third prediction model may be U-net, a model learned to infer (detect) objects in images, but is not limited thereto. For example, prediction models include VGG net, DenseNet, FCN (Fully Convolutional Network) with encoder-decoder structure, SegNet, DeconvNet, deep neural networks (DNN) such as DeepLAB V3+, SqueezeNet, Alexnet, ResNet18, MobileNet-v2 It may be based on at least one network model selected from , GoogLeNet, Resnet-v2, Resnet50, RetinaNet, Resnet101, and Inception-v3. Furthermore, the first to third prediction models may be ensemble models based on at least two network models among the above-described algorithms.

Hereinafter, the present invention will be described in detail by explaining preferred embodiments of the present invention with reference to the accompanying drawings.

1 is a schematic diagram illustrating an outline of a method for providing standard plane information according to an embodiment of the present invention.

Referring to FIG. 1, the method for providing a standard plane according to an embodiment of the present invention can be largely divided into three steps. Specifically, the method for providing a standard plane of the present invention can extract only the diagnostic area from a plurality of frames (input frames) constituting an ultrasound image of an object (step 1). Here, extracting only the diagnostic area can be understood as detecting a diagnostic area corresponding to a region of interest (ROI) in an image frame and cutting out only the corresponding area to create a patch.

Based on the diagnostic area, the ultrasound image can be classified into multiple zones (step 2), and in this process, supplementary contextual information (DSSIM map) about the object can be applied. Contextual clues may refer to information that predicts how the structure of an object changes by considering the movement of the object and the movement of the probe during an ultrasound examination. In other words, the present invention allows medical staff to actually perform an ultrasound examination while the head of the fetus, for example, Falx/Cranium, Lateral ventricle/Choroid plexus, Glomus/Lateral ventricle, Ambient cistern/Thalamus, Ambient cistern/Cerebellum, Cerebellum/ It is possible to classify the area by creating a map of how the conversion occurred within the area containing Cisterna magna and the aspects of the conversion.

And finally, the present invention can calculate the probability that the frames for each region correspond to the standard plane and determine the standard plane based on the probability value (step 3). In one embodiment of the present invention, the standard plane includes a standard TV plane (Transventricular plane), a standard TT plane (Transthalamic plane), and a standard TC plane (Transcerebella plane). In addition, the standard plane includes various planes depending on the object and diagnostic area. It can be determined based on the axis.

So far, the outline of the method for providing standard plane information according to an embodiment of the present invention has been briefly described, and the system that performs this will now be described.

Figure 2 is a schematic diagram of a standard plan information providing system according to an embodiment of the present invention.

Referring to FIG. 2, the standard plane information providing system 1000 (hereinafter referred to as the “plane providing system”) may be a system configured to provide standard plane information about an object. The plane providing system 1000 includes a medical staff device 100 that receives and outputs standard plane information, an ultrasound device 200 that acquires an ultrasound image of the diagnostic area, and a standard plane information that generates standard plane information based on the ultrasound image. It may include a provision device 300.

The medical staff device 100 is a device owned by a medical staff who checks information related to the diagnostic area and performs a diagnosis on an object, and may include a smartphone, tablet PC, PC, laptop, etc. Specifically, the medical staff device 100 may provide a user interface that outputs information related to the diagnostic area of the subject.

The ultrasound device 200 may be a device that can acquire a medical image of a diagnostic area of an object. In the present invention, for convenience of explanation, it is assumed that the subject is a mother and the diagnosis area is the uterus-fetus.

The device 300 for providing standard plane information may predict the standard plane for the head of the fetus based on the ultrasound image provided from the ultrasound device 200. For example, the device 300 for providing standard plane information may include a general-purpose computer, a laptop, and a data server that can receive an image and perform inference on any one of head region detection, region classification, and standard plane probability prediction. You can.

In various embodiments, the device 300 for providing standard planar information acquires an ultrasound image of the fetus, determines the head of the fetus from the ultrasound image, predicts a preset head region classification result using a first prediction model, and , the standard plane probability of the ultrasound image can be predicted using the second prediction model, and then the standard plane for the fetal head can be determined based on the standard plane probability.

In addition, the standard flat information providing device 300 and the medical staff device 100 each correspond to a client-service providing server, and accordingly, the standard flat information providing device 300 is connected to the medical staff device through a web or mobile application or program. (100) can provide standard plane information.

In various embodiments, functions performed by the standard planar information providing device 300 may be performed by the medical staff device 100. That is, the planar provision system 1000 consists only of the medical staff device 100 and the ultrasound device 200, and the medical staff device 100 provides a standard for the fetal head based on the ultrasound image and contextual clues for the examination of the fetus. Plane information can also be determined.

So far, the plane providing system 1000 according to an embodiment of the present invention has been described, and hereinafter, a device for obtaining and providing standard plane information about an object will be described with reference to FIGS. 3 and 4.

Figure 3 is a block diagram showing the configuration of a medical staff device according to an embodiment of the present invention.

Referring to FIG. 3 , the medical staff device 100 may include a memory interface 110, one or more processors 120, and a peripheral interface 130. The various components within identifier device 100 may be connected by one or more communication buses or signal lines.

The memory interface 110 is connected to the memory 150 and can transmit various data to the processor 120. Here, the memory 150 is a flash memory type, hard disk type, multimedia card micro type, card type memory (for example, SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, network storage, and cloud. , It may include at least one type of storage medium among the blockchain database.

In various embodiments, the memory 150 may include a first prediction model for classifying each of a plurality of frames constituting an ultrasound image of the head into preset regions, and a probability that some frames constituting the ultrasound image correspond to a standard plane. A second prediction model for predicting and a third prediction model for detecting the fetal head in an ultrasound image can be stored. Additionally, the memory 150 may store training data for the first to third models and ultrasound images received from the ultrasound device 200.

In various embodiments, memory 150 includes operating system 151, communications module 152, graphical user interface module (GUI) 153, sensor processing module 154, telephony module 155, and application module 156. ) can be stored. Specifically, the operating system 151 may include instructions for processing basic system services and instructions for performing hardware tasks. The communication module 152 may communicate with at least one of one or more other devices, computers, and servers. The graphical user interface module (GUI) 153 can process a graphical user interface. Sensor processing module 154 may process sensor-related functions (e.g., processing voice input received through one or more microphones 192). The phone module 155 can process phone-related functions. Application module 156 may perform various functions of a user application, such as electronic messaging, web browsing, media processing, navigation, imaging, and other processing functions. In addition, the identifier device 100 may store one or more software applications 156-1 and 156-2 (eg, a standard flat information providing application) associated with one type of service in the memory 150.

In various embodiments, memory 150 may store a digital assistant client module 157 (hereinafter referred to as DA client module), thereby storing various user data 158 and instructions for performing client-side functions of the digital assistant. (e.g. user-customized vocabulary data, preference data, other data such as the user's electronic address book, etc.) may be stored.

Meanwhile, the DA client module 157 uses the user's voice input, text input, touch input, and/or gesture input through various user interfaces (e.g., I/O subsystem 140) provided in the medical staff device 100. It can be obtained.

Additionally, the DA client module 157 can output data in audiovisual and tactile forms. For example, the DA client module 157 may output data consisting of a combination of at least two or more of voice, sound, notification, text message, menu, graphics, video, animation, and vibration. In addition, the DA client module 157 can communicate with a digital assistant server (not shown) using the communication subsystem 180.

In various embodiments, the DA client module 157 may collect additional information about the surrounding environment of the medical staff device 100 from various sensors, subsystems, and peripheral devices to construct a context associated with user input. . For example, the DA client module 157 may infer the user's intention by providing context information along with user input to the digital assistant server. Here, context information that may accompany the user input may include sensor information, for example, lighting, ambient noise, ambient temperature, images of the surrounding environment, video, etc. As another example, the situation information may include the physical state of the medical staff device 100 (e.g., device orientation, device location, device temperature, power level, speed, acceleration, motion pattern, cellular signal strength, etc.). As another example, the context information may include information related to the software status of the medical staff device 100 (e.g., processes running on the medical staff device 100, installed programs, past and present network activity, background services, error logs, resource usage). etc.) may be included.

In various embodiments, the memory 150 may include added or deleted instructions, and further, the medical staff device 100 may also include additional components other than those shown in FIG. 3 or exclude some components.

The processor 120 can control the overall operation of the medical staff device 100 and executes various commands to implement a user interface that provides information related to the standard plane by running an application or program stored in the memory 150. You can.

The processor 120 may correspond to a computing device such as a Central Processing Unit (CPU) or an Application Processor (AP). In addition, the processor 120 may be implemented in the form of an integrated chip (IC) such as a system on chip (SoC) that integrates various computing devices that perform machine learning, such as a neural processing unit (NPU). .

In various embodiments, the processor 120 acquires an ultrasound image of the fetus, determines the head of the fetus from the ultrasound image, predicts a preset head region classification result using a first prediction model, and uses a second prediction model. After predicting the standard plane probability of the ultrasound image, the standard plane for the fetal head can be determined based on the standard plane probability.

The peripheral interface 130 is connected to various sensors, subsystems, and peripheral devices, and can provide data so that the medical staff device 100 can perform various functions. Here, any function performed by the medical staff device 100 may be understood as being performed by the processor 120.

The peripheral interface 130 may receive data from the motion sensor 160, the light sensor (light sensor) 161, and the proximity sensor 162, through which the medical staff device 100 can determine orientation, light, and proximity. It can perform detection functions, etc. For another example, the peripheral interface 130 may receive data from other sensors 163 (positioning system-GPS receiver, temperature sensor, biometric sensor), through which the medical staff device 100 may use other sensors. Functions related to (163) can be performed.

In various embodiments, the medical staff device 100 may include a camera subsystem 170 connected to the peripheral interface 130 and an optical sensor 171 connected thereto, through which the medical staff device 100 can take pictures and video. You can perform various shooting functions such as clip recording.

In various embodiments, medical staff device 100 may include a communication subsystem 180 coupled to a peripheral interface 130 . The communication subsystem 180 consists of one or more wired/wireless networks and may include various communication ports, radio frequency transceivers, and optical transceivers.

In various embodiments, medical staff device 100 includes an audio subsystem 190 coupled to a peripheral interface 130, which includes one or more speakers 191 and one or more microphones 192. By including them, medical staff device 100 can perform voice-activated functions, such as voice recognition, voice duplication, digital recording, and telephony functions.

In various embodiments, medical staff device 100 may include an I/O subsystem 140 coupled with a peripheral interface 130. For example, the I/O subsystem 140 may control the touch screen 143 included in the identifier device 100 through the touch screen controller 141.

For example, the touch screen controller 141 uses any one of a plurality of touch sensing technologies such as capacitive, resistive, infrared, surface acoustic wave technology, proximity sensor array, etc. to detect the user's touch and movement or contact. and cessation of movement can be detected. As another example, the I/O subsystem 140 may control other input/control devices 144 included in the identifier device 100 through other input controller(s) 142. As an example, other input controller(s) 142 may control one or more buttons, rocker switches, thumb-wheels, infrared ports, USB ports, and pointer devices such as a stylus.

Figure 4 is a block diagram showing the configuration of a device for providing standard planar information according to an embodiment of the present invention.

Referring to FIG. 4, the device 300 for providing standard flat information may include a communication interface 310, a memory 320, an I/O interface 330, and a processor 340, and each component includes one or more They can communicate with each other through communication buses or signal lines.

The communication interface 310 can be connected to the medical staff device 100 and the ultrasound device 200 through a wired/wireless communication network to exchange data. For example, the communication interface 310 may transmit standard plane information about the fetal head to the medical staff device 100 . For another example, the communication interface 310 may receive an ultrasound image of the mother's uterus from the ultrasound device 200 in real time.

Meanwhile, the communication interface 310 that enables transmission and reception of such data includes a wired communication port 311 and a wireless circuit 312, where the wired communication port 311 is one or more wired interfaces, for example, Ethernet. , Universal Serial Bus (USB), Firewire, etc. Additionally, the wireless circuit 312 can transmit and receive data with an external device through RF signals or optical signals. Additionally, wireless communications may use at least one of a plurality of communication standards, protocols and technologies, such as GSM, EDGE, CDMA, TDMA, Bluetooth, Wi-Fi, VoIP, Wi-MAX, or any other suitable communication protocol.

The memory 320 can store various data used in the device 300 for providing standard planar information. For example, the memory 320 may store information about the object, image frames of ultrasound images used as learning data, standard plane prediction results, etc. For example, the memory 320 may include a first prediction model for classifying each of the plurality of frames constituting the ultrasound image of the head into preset regions, and a probability that some frames constituting the ultrasound image correspond to a standard plane. A second prediction model for prediction and a third prediction model for detecting the fetal head in an ultrasound image may be stored. Additionally, the memory 150 may store training data for the first to third models and ultrasound images received from the ultrasound device 200.

In various embodiments, memory 320 may include volatile or non-volatile recording media capable of storing various data, instructions, and information. For example, the memory 320 may be a flash memory type, hard disk type, multimedia card micro type, card type memory (e.g. SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, network storage storage. , cloud, or blockchain database may include at least one type of storage medium.

In various embodiments, memory 320 may store configuration of at least one of operating system 321, communication module 322, user interface module 323, and one or more applications 324.

Operating system 321 (e.g., embedded operating system such as LINUX, UNIX, MAC OS, WINDOWS, VxWorks, etc.) is a variety of software to control and manage common system tasks (e.g., memory management, storage device control, power management, etc.) It may contain components and drivers and may support communication between various hardware, firmware, and software components.

The communication module 323 may support communication with other devices through the communication interface 310. The communication module 220 may include various software components for processing data received by the wired communication port 311 or the wireless circuit 312 of the communication interface 310.

The user interface module 323 may receive a user's request or input from a keyboard, touch screen, keyboard, mouse, microphone, etc. through the I/O interface 330 and provide a user interface on the display.

Applications 324 may include programs or modules configured to be executed by one or more processors 340 . Here, an application for providing standard planar information about the fetal head can be implemented on a server farm.

The I/O interface 330 may connect at least one of input/output devices (not shown) of the standard flat information providing device 300, such as a display, keyboard, touch screen, and microphone, to the user interface module 323. The I/O interface 330 may receive user input (eg, voice input, keyboard input, touch input, etc.) together with the user interface module 323 and process commands according to the received input.

The processor 340 is connected to the communication interface 310, the memory 320, and the I/O interface 330 to control the overall operation of the device 300 for providing standard planar information, and stores the information stored in the memory 320. Various commands can be executed to extract standard plane information from ultrasound images through an application or program.

The processor 340 may correspond to a computing device such as a Central Processing Unit (CPU) or an Application Processor (AP). Additionally, the processor 340 may be implemented in the form of an integrated chip (IC) such as a system on chip (SoC) in which various computing devices are integrated. Alternatively, the processor 340 may include a module for calculating an artificial neural network model, such as a Neural Processing Unit (NPU).

Hereinafter, with reference to FIGS. 5 to 9B, a method by which the processor 340 of the standard plane information providing device 300 provides information about the standard plane of the fetal head in an ultrasound image will be described.

Figure 5 is a schematic flowchart of a method for providing standard plane information of an object according to an embodiment of the present invention.

Referring to FIG. 5, the processor 340 may acquire an ultrasound image of the fetus (S110). For example, the processor 340 may receive an ultrasound image of the mother's uterus from the ultrasound device 200 through the communication interface 310.

In various embodiments, the processor 340 may be connected to a probe (not shown) of the ultrasound device 200 through the wired communication port 311 to receive ultrasound images of the mother's uterus in real time.

After step S110, the processor 340 may determine the head of the fetus from the ultrasound image (S120). Specifically, the processor 340 may determine a region of interest (ROI) including the head of the fetus in the ultrasound image.

In relation to this, Figure 6 is a schematic diagram illustrating a method for determining the head area of a fetus according to an embodiment of the present invention.

Referring to FIG. 6, the processor 340 may detect the head of the fetus by inputting the ultrasound image of the fetus into a prediction model learned to detect the head using the ultrasound image as input. The processor 340 may input a plurality of frames 11 constituting the ultrasound image into a prediction model. Additionally, the processor 340 may perform ellipse fitting or curve fitting on the detected boundary line of the head (step 1-1).

Accordingly, the processor 340 may determine the borderline 12 of the fetal head having an oval shape. Here, each of the long axis and short axis of the ellipse may correspond to the anteroposterior diameter (occipitofrontal diameter, OFD) and the biparietal diameter (BPD) of the fetal head.

The processor 340 can horizontally align the frame 11 based on the long axis of the ellipse and, for example, rotate the frame 11 so that the front of the fetus' head is placed on the left (step 1-2).

The processor 340 can obtain only the rectangular area 13 containing the head of the fetus, that is, an oval, and extract the patch 14 (steps 1-3). That is, the processor 340 may extract a frame including the fetal head into a uniformly sized patch (I ^ROI ) 14 that can be input to the first and second prediction models.

Referring again to FIG. 5, after step S120, the processor 340 uses a first prediction model learned to classify the head region in the ultrasound image of the head to classify a preset head region classification result among the ultrasound images of the head. It is predictable. Specifically, in the present invention, it is possible to classify the head region by taking into account the structural transition aspect of the head that is considered during the ultrasound examination process, beyond simply using only the frames of the ultrasound image. For example, the head region classification result may be a first region containing the standard TV plane (Transventricular plane) and a standard TT plane (Transthalamic plane), a second region including the standard TC plane (Transcerebella plane), and It may include classification results for at least one of the remaining third areas excluding the second area.

To this end, the processor 340 may obtain an index map (Structured Dissimilarity Index Measure Map, DISSM Map) indicating structural changes in the fetal head based on a plurality of frames constituting the ultrasound image of the head. Specifically, the index map (DISSM Map) is a map that displays the differences between frames (I ₁ ^ROI )(I _1+s ^ROI ),,,(I _1+(L-1)s ^ROI ) by pixel, It can be used as data to detect structural turning points (areas where structural transitions are prominent) in continuous cross-sectional images. For example, the DISSM Map can acquire index maps at intervals of 10 and 20 seconds from ultrasound images with a sequential frame length (L) of 10, and the index maps are DSM _1,1 and DSM _1, respectively. _{, 2} , … It can be defined as DSM _{1, L-1} .

In relation to this, FIGS. 7A and 7B are schematic diagrams exemplarily showing the structure of a first prediction model used in a method of providing standard plane information of an object according to an embodiment of the present invention.

Referring to FIG. 7A, the processor 340 processes one frame 14 (e.g., I ₁ ^ROI ) constituting an ultrasound image and an index map 16 (e.g., DSM _{1, 1} ) can be input into the first prediction model 15 to predict the head region classification result. For example, the processor 340 may predict that the frame 14 (eg, I ₁ ^ROI ) is a frame corresponding to the first region (z ₁ ).

In addition, the first prediction model 15 may be composed of multiple networks as shown in FIG. 7B. Specifically, the first prediction model 15 uses each of the ultrasound image and index map for the head as input to generate different feature maps ( )( ), a feature extraction layer (1511, f _CNN1 ) (1512, f _CNN2 ) configured to extract two different feature maps (1511, f CNN2), and two different feature maps ( )( ) and a classification vector for classifying the head region of the fetus based on the result (O _t ) (one map connecting two feature maps) output from the integration layer 152 and the integration layer 152 configured to integrate the It may include a classification layer 153 configured to output.

At this time, the integration layer 152 that integrates the two feature maps includes a recurrent neural network. For example, the recurrent neural network may be a bi-directional gated recurrent unit (GRU) or a bi-directional long short term memory (LSTM). It can be composed of: For example, the integration layer 152 is composed of a forward ConvLSTM (f ⁺ _ConvLSTM ) and a reverse ConvLSTM (f ^- _ConvLSTM ) as shown in [Equation 1] and [Equation 2] below, and each network model has different Two feature maps ( )( ) Each can be used as input data.

Here, h _t ⁺ and h _t ^- are the hidden state of the forward and reverse LSTM, respectively, and c _t ⁺ and c _t ^- are the cell state of the forward and reverse LSTM, respectively.

In this way, as the feature map acquired through the ultrasound image is input bidirectionally, the structural transition aspect during the ultrasound examination can be taken into consideration, and the classification layer 153 determines which of the plurality of frames constituting the ultrasound image. Whether it corresponds to the region (z _t ), the head region classification results can be output.

In addition, during the learning process of the first prediction model 15, a weight based on a loss function can be applied to the classification layer 153 as shown in [Equation 3], and the process of classifying the region can be optimized accordingly. there is.

here, is the training data labeled for each region, and M is the number of training data.

Referring again to FIG. 5, after step S130, the processor 340 inputs the ultrasound image of the head region into a second prediction model that has been trained to predict the standard plane probability, and the head region classified according to the classification result. By inputting an ultrasound image, the standard plane probability of the ultrasound image can be predicted (S140). Specifically, the processor 340 may predict a standard plane centered on different points based on ultrasound images of head regions classified into a plurality of head regions according to the classification result.

In relation to this, FIGS. 8A and 8B are schematic diagrams illustrating the structure of a second prediction model used in a method of providing standard plane information of an object according to an embodiment of the present invention.

Referring to FIG. 8A, the processor 340 inputs a plurality of frames (patch) 17 in the ultrasound image classified into the first region into the second prediction model 18, so that each of the plurality of frames is, for example, a ventricle. The probability (y ₁ ) corresponding to the TV plane (transventricular plane) passing through can be predicted. That is, the processor 340 calculates the respective probabilities that a plurality of patches classified into the first region correspond to the standard TV plane, and a plurality of patches classified into the second region correspond to the TT plane (transthalamic plane) passing through the thalamus. Each probability that a plurality of patches classified into the third region correspond to the TC plane (Transcerebella plane) passing through the cerebellum can be predicted.

In this way, the processor 340 is configured to provide information about two or more standard planes among the standard TV plane (Transventricular plane) passing through the ventricle, the standard TT plane (Transthalamic plane) passing through the thalamus, and the standard TC plane (Transcerebella plane) passing through the cerebellum. Probabilities can be predicted. However, in addition to this, the processor 340 can predict probabilities for various reference planes passing through the object.

In addition, the second prediction model 18 may be composed of multiple networks as shown in Figure 8b. Specifically, the second prediction model 18 has a network structure similar to the first prediction model 15, and the previously classified head A feature extraction layer 181 configured to extract a feature map from an ultrasound image (patch) for a region, an integration layer 182 configured to integrate the forward and reverse output results of the feature map extracted from the feature extraction layer 181, integration It may include an attention layer 183 configured to extract structural feature points of the fetus based on the results output from the layer, and a classification layer 184 configured to output a probability for the standard plane based on the output results and feature points.

At this time, the integration layer 182 includes a recurrent neural network. For example, the recurrent neural network may be composed of a bi-directional gated recurrent unit (GRU) or a bi-directional long short term memory (LSTM).

In addition, during the learning process of the second prediction model 18, a weight based on a loss function can be applied to the classification layer 184, as shown in [Equation 4], and the process of classifying the region can be optimized accordingly. there is.

Here, M is the number of learning data, A = {A ₁ , .., A _L } is anatomical attention heat maps, and A ^* means the correct answer with the data label corresponding to A.

Again, referring to FIG. 5, the processor 340 may output standard plane probabilities for a plurality of patches included in the region through the second prediction model 18, and based on the standard plane probabilities, the fetal head The standard plane for can be determined (S150).

In relation to this, FIGS. 9A and 9B are schematic diagrams exemplarily showing the results of a method for providing standard plane information of an object according to an embodiment of the present invention.

Referring to FIG. 9A, the processor 340 can output the probability that each of dozens of frames (patches) classified by region corresponds to any one of the standard TT plane, standard TV plane, and standard TC plane. In addition, the processor 340 may provide probability values in the following graph form to the medical staff device 100 through the communication interface 310.

Additionally, referring to FIG. 9B, the processor 340 considers the frame (patch) with the highest probability value among dozens of frames (patches) as the frame (patch) corresponding to the standard TT plane, standard TV plane, and standard TC plane. and you can highlight it. For example, the medical staff device 100 may output a highlighted frame (patch) corresponding to a standard plane among dozens of frames (patches), as shown in FIG. 9B.

Meanwhile, the standard plane prediction results determined by the method above are simulation test results, and have higher accuracy (Acc) and precision than the conventional method that simply uses ultrasound images as input data, as shown in [Table 1] below. ) (Pre), recall factor (Rec), and harmonic mean value (F1-Score) (F1). Here, Comparative Examples 1, 2, and 3 may be planar prediction methods using a single CNN-based model and CNN-RNN-based models, respectively.

Through these results, it can be understood that the method for providing standard plane information according to an embodiment of the present invention has plane prediction capabilities by considering changes in anatomical structures in learning during the ultrasound imaging process.

Example Comparative Example 1 Comparative Example 2 Comparative Example 3 TV Pre 0.906 0.691 0.860 0.928 Rec 0.980 0.653 0.900 0.819 F1 0.942 0.671 0.880 0.875 TT Pre 0.965 0.813 0.721 0.752 Rec 0.875 0.727 0.816 0.875 F1 0.918 0.768 0.766 0.809 TC Pre 0.860 0.664 0.845 0.881 Rec 0.910 0.818 0.701 0.885 F1 0.884 0.733 0.839 0.883 nstd Pre 0.901 0.787 0.783 0.813 Rec 0.844 0.736 0.701 0.853 F1 0.872 0.761 0.740 0.853 Average-age ACC 0.899 0.741 0.806 0.851 Pre 0.908 0.739 0.803 0.853 Rec 0.902 0.734 0.846 0.813 F1 0.904 0.733 0.806 0.849

So far, we have outlined the standard plane information providing device 300 and the standard plane information providing method for an object performed by the processor 340 of the standard plane information providing device 300 according to an embodiment of the present invention. explained. According to the present invention, by learning changes in structural characteristics according to the movement of a dynamic object, the standard plane of the object can be predicted more accurately than a method of predicting the standard plane simply with a single frame of an ultrasound image.

Although embodiments of the present invention have been described in more detail with reference to the accompanying drawings, the present invention is not necessarily limited to these embodiments, and may be modified and implemented in various ways without departing from the technical spirit of the present invention. there is. Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but are for illustrative purposes, and the scope of the technical idea of the present invention is not limited by these embodiments. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. The scope of protection of the present invention should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be construed as being included in the scope of rights of the present invention.

1000: Standard flat information provision system
100: Medical staff device
200: Ultrasound device
110: memory interface 120: processor
130: peripheral interface 140: I/O subsystem
141: Touch screen controller 142: Other input controller
143: touch screen
144: Other input control devices
150: Memory 151: Operating system
152: Communication module 153: GUI module
154: sensor processing module 155: phone module
156: Applications
156-1, 156-2: Application
157: Digital assistant client module
158: User data
160: motion sensor 161: light sensor
162: Proximity sensor 163: Other sensors
170: Camera subsystem 171: Optical sensor
180: Communication subsystem
190: Audio subsystem
191: Speaker 192: Microphone
300: Device for providing standard flat information
310: communication interface
311: wired communication port 312: wireless circuit
320: memory
321: operating system 322: communication module
323: user interface module 324: application
330: I/O interface 340: Processor

Claims (21)

Obtaining an ultrasound image of the fetus;
determining the head of the fetus from the ultrasound image;
predicting a preset head region classification result among the ultrasound images of the head using a first prediction model learned to classify the head region in the ultrasound images of the head;
Predicting the standard plane probability of the ultrasound image by inputting the ultrasound image of the head region classified according to the classification result into a second prediction model learned to predict the standard plane probability by inputting the ultrasound image of the head region. step; and
determining a standard plane for the fetal head based on the standard plane probability; A method of providing standard plane information about an object including. According to paragraph 1,
The step of determining the head is,
Detecting the head of the fetus by inputting the ultrasound image of the fetus into a third prediction model learned to detect the head by inputting the ultrasound image, and
A method of providing standard plane information for an object further comprising determining an ellipse corresponding to the head of the fetus using ellipse fitting or curve fitting results for the detected boundary line of the head. According to paragraph 2,
The step of determining the head is,
A method of providing standard plane information about an object further comprising extracting a patch including an ellipse determined from the ultrasound image. According to paragraph 1,
The step of predicting the head region classification result is,
Obtaining an index map (Structured Dissimilarity Index Measure Map, DISSM Map) indicating structural changes in the head based on a plurality of frames constituting the ultrasound image of the head, and
Predicting a head region classification result of the fetus by inputting the ultrasound image of the head and the index map into the first prediction model. According to paragraph 4,
The first prediction model is,
Two different feature extraction layers configured to extract feature maps from each of the ultrasound image of the head and the index map;
an integration layer configured to integrate two different feature maps extracted from the two feature extraction layers; and
a classification layer configured to output a classification vector for classifying the head region of the fetus based on the results output from the integrated layer; A method of providing standard plane information about an object including. According to paragraph 4,
The head region classification result is,
A first region including a standard TV plane (Transventricular plane) and a standard TT plane (Transthalamic plane), a second region including a standard TC plane (Transcerebella plane), and a third region excluding the first region and the second region. A method of providing standard planar information about an object, including a classification result for at least one of the regions. According to paragraph 1,
The step of predicting the standard plane probability is,
A method of providing standard plane information about an object, which is a step of predicting standard plane probabilities centered on different points based on ultrasound images of head regions classified into a plurality of head regions according to the classification results. In clause 7,
The step of predicting the standard plane probability is,
A method of providing standard plane information for an object, which is a step of predicting the probability of two or more standard planes among the standard TV plane (Transventricular plane), the standard TT plane (Transthalamic plane), and the standard TC plane (Transcerebella plane). According to paragraph 1,
The second prediction model is,
a feature extraction layer configured to extract a feature map from the ultrasound image for the classified head region;
an integration layer configured to integrate forward and backward output results of the feature map extracted from the feature extraction layer;
an attention layer configured to extract structural feature points of the fetus based on the results output from the integration layer; and
a classification layer configured to output a probability for the standard plane based on the output result and the feature points; A method of providing standard plane information about an object including. According to clause 5 or 9,
The integration layer includes a recurrent neural network,
The recurrent neural network is comprised of a bi-directional gated recurrent unit (GRU) or a bi-directional long short term memory (LSTM). communication interface;
Memory; and
a processor operably connected to the communication interface and the memory; Including,
The processor,
Obtaining an ultrasound image of the fetus, determining the head of the fetus from the ultrasound image, and using a first prediction model learned to classify the head region in the ultrasound image of the head, Predict the set head region classification result and input the ultrasound image of the head region classified according to the classification result into a second prediction model that has been trained to predict the standard plane probability using the ultrasound image of the head region as input. A device for providing standard plane information for a subject, configured to predict a standard plane probability of an ultrasound image and determine a standard plane for the head of the fetus based on the standard plane probability. According to clause 11,
The processor,
The head of the fetus is detected by inputting the ultrasound image of the fetus into a third prediction model learned to detect the head using the ultrasound image as input, and an ellipse fitting or curve is performed on the boundary line of the detected head. A device for providing standard plane information about an object, configured to determine an ellipse corresponding to the head of the fetus using a curve fitting result. According to clause 12,
The processor,
A device for providing standard plane information about an object, configured to extract a patch including an ellipse determined from the ultrasound image. According to clause 11,
The processor,
An index map (Structured Dissimilarity Index Measure Map, DISSM Map) indicating structural changes in the head is obtained based on a plurality of frames constituting the ultrasound image of the head, and the ultrasound image of the head and the index map are obtained. A device for providing standard planar information about an object, configured to predict a head region classification result of the fetus by inputting it into a first prediction model. According to clause 14,
The first prediction model is,
Two different feature extraction layers configured to extract feature maps from each of the ultrasound image of the head and the index map;
an integration layer configured to integrate two different feature maps extracted from the two feature extraction layers; and
a classification layer configured to output a classification vector for classifying the head region of the fetus based on the results output from the integrated layer; A device for providing standard flat information about an object including a. According to clause 14,
The head region classification result is,
A first region including a standard TV plane (Transventricular plane) and a standard TT plane (Transthalamic plane), a second region including a standard TC plane (Transcerebella plane), and a third region excluding the first region and the second region. A device for providing standard planar information about an object, including a classification result for at least one of the regions. According to clause 11,
The processor,
A device for providing standard plane information about an object, configured to predict standard plane probabilities centered on different points based on ultrasound images of head regions classified into a plurality of head regions according to the classification results. According to clause 17,
The processor,
A device for providing standard plane information about a subject, configured to predict probabilities for two or more standard planes among a standard TV plane (Transventricular plane), a standard TT plane (Transthalamic plane), and a standard TC plane (Transcerebella plane). According to clause 11,
The second prediction model is,
a feature extraction layer configured to extract a feature map from the ultrasound image for the classified head region;
an integration layer configured to integrate forward and backward output results of the feature map extracted from the feature extraction layer;
an attention layer configured to extract structural feature points of the fetus based on the results output from the integration layer; and
a classification layer configured to output a probability for the standard plane based on the output result and the feature points; A device for providing standard flat information about an object including a. According to claim 15 or 19,
The integration layer includes a recurrent neural network,
The recurrent neural network is a device for providing standard planar information about an object, consisting of a bi-directional gated recurrent unit (GRU) or a bi-directional long short term memory (LSTM). Obtaining an ultrasound image of an area of an object;
determining a diagnosis target from the ultrasound image;
Predicting a preset diagnostic area classification result among the ultrasound images of the diagnostic object using a first prediction model learned to classify the diagnostic region in the ultrasound image of the diagnostic object;
Predicting the standard plane probability of the diagnosis object by inputting the ultrasound image of the diagnostic region classified according to the classification result into a second prediction model learned to predict the standard plane probability by inputting the ultrasound image of the diagnostic region. step; and
determining a standard plane for the diagnostic object based on the standard plane probability; A method of providing standard plane information about an object including.