KR20200006016A - Apparatus and method for analyzing image - Google Patents

Abstract

Proposed are an image analysis device and an image analysis method. According to one embodiment, the image analysis device can comprise: a control part for extracting a landmark point on the image to perform reliability analysis on the extracted landmark point and generating an analysis image based on the reliability analyzed landmark point; and a memory for storing the generated analysis image.

Description

Image analysis device and image analysis method {APPARATUS AND METHOD FOR ANALYZING IMAGE}

Embodiments disclosed herein relate to an image analyzing apparatus and method, and more particularly, automatically detects a landmark of an input image to calculate a reliability of the detected landmark, and is modified according to the calculated reliability. An apparatus and method for analyzing an image through learning of a landmark are provided.

For the dental, jaw and facial cosmetic surgery, data of the skull or face of the patient is needed. For this purpose, before the surgeon proceeds with the surgery, the medical staff analyzes an image of the head of the patient using an X-ray apparatus or a computed tomography (CT) diagnostic device and predicts the patient through the analyzed file. Determine the surgical site. At this time, the medical staff analyzes the main points on the image by marking them. The method of analysis is based on the experiences of the medical staff accumulated for a long time.

However, this method has a problem that it takes a long time for the medical staff to guide the accurate diagnosis results to the patient because the medical staff must select the point directly, so that the analysis data can not be obtained in real time. In addition, the method can not only rely on the experience of the medical staff, but if the non-skilled medical staff to select a point, the problem is generated because the wrong selection of the point to create a distorted analysis data and the operation based on the wrong analysis data There is no problem.

In this regard, Korean Patent Publication No. 10-2000-0054062, a prior art document, digitizes all the information, such as the results of medical examinations of patients, so that doctors can describe the contents that can be easily treated at places where the Internet is available, such as at home or office. In addition, it does not describe the contents that provide medical staff with the results of analyzing medical images so that medical staff can provide improved medical services to patients. Therefore, there is a need for a technique for solving the above problems.

On the other hand, the background art described above is technical information that the inventors possess for the derivation of the present invention or acquired in the derivation process of the present invention, and is not necessarily a publicly known technique disclosed to the general public before the application of the present invention. .

Embodiments disclosed herein are aimed at presenting an image analyzing apparatus and method.

In addition, the embodiments disclosed herein are intended to provide an accurate analysis result based on the image.

In addition, the embodiments disclosed herein are intended to quickly provide a result of analyzing the image.

In addition, the embodiments disclosed herein are intended to provide a report that is easy to understand by the medical staff as well as the patient by processing the results of analyzing the image once.

As a technical means for achieving the above-described technical problem, according to an embodiment of the present invention, an apparatus for analyzing an image, extracts a landmark point on the image to perform a reliability analysis on the extracted landmark point, the reliability analysis It may include a control unit for generating an analysis image based on the landmark point and a memory for storing the generated analysis image.

According to another exemplary embodiment of the present invention, a method for analyzing an image by an image analyzing apparatus includes extracting landmark points on an image, and performing reliability analysis on the extracted landmark points. And generating an analysis image based on the reliability analyzed landmark point.

As a technical means for achieving the above-described technical problem, according to another embodiment may be a computer-readable recording medium in which a program for performing the image analysis method is recorded, the image analysis method, extracting a landmark point on the image The method may include performing reliability analysis on the extracted landmark points and generating an analysis image based on the reliability analyzed landmark points.

As a technical means for achieving the above-described technical problem, according to another embodiment is performed by the image analysis device, may be a computer program stored in a medium for performing the image analysis method, the image analysis method, land on the image The method may include extracting a mark point, performing a reliability analysis on the extracted landmark point, and generating an analysis image based on the reliability analyzed landmark point.

According to any one of the aforementioned problem solving means, it is possible to provide an image analyzing apparatus and method.

According to any one of the aforementioned problem solving means, by providing an accurate analysis result based on the image, it can help the medical staff to determine the expected surgical site of the patient.

According to any one of the above-described problem solving means, it is possible to prevent the delay of the necessary surgery by providing the result of analyzing the image quickly.

According to any one of the above-mentioned means for solving the problem, by processing the result of analyzing the image once, it is possible to provide a report that is easily understood by the medical staff as well as the patient.

The effects obtainable in the disclosed embodiments are not limited to the effects mentioned above, and other effects not mentioned above are apparent to those skilled in the art to which the embodiments disclosed from the following description belong. Can be understood.

1 and 2 are block diagrams illustrating an image analyzing apparatus according to an exemplary embodiment.
3 is an exemplary diagram for describing an image analyzing apparatus according to an exemplary embodiment.
4 to 6 are flowcharts for describing an image analysis method, according to an exemplary embodiment.
7 to 8 are exemplary diagrams for describing an image analysis method, according to an exemplary embodiment.

Hereinafter, various embodiments will be described in detail with reference to the accompanying drawings. The embodiments described below may be embodied in various different forms. In order to more clearly describe the features of the embodiments, detailed descriptions of the matters well known to those skilled in the art to which the following embodiments belong are omitted. In the drawings, parts irrelevant to the description of the embodiments are omitted, and like reference numerals designate like parts throughout the specification.

Throughout the specification, when a configuration is "connected" to another configuration, this includes not only 'directly connected' but also 'connected' between different configurations. In addition, when a configuration "includes" a certain configuration, this means that, unless specifically stated otherwise, it may further include other configurations other than the other configuration.

Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings.

However, before describing this, the meanings of the terms used below are first defined.

An 'image' is an image of a patient's cranial or facial image that can be used for head observation or head measurement, for example dimensions of the anatomical site of the head or the anatomical location of the head. Can be represented.

The image may typically be a radiograph or computed tomography (CT) diagnosis of the front or side of the patient's head, but the method of obtaining the image is not limited to the example described above. In addition, the image may be a two-dimensional image or a three-dimensional image.

In addition to the terms defined above, terms that need explanation are explained separately below.

1 and 2 are block diagrams illustrating an image analyzing apparatus according to an exemplary embodiment.

The image analysis apparatus 100 may generate an analysis image combining the identification image with respect to the acquired image.

The image analysis apparatus 100 may be implemented as an electronic terminal or may be implemented as a server-client system (or a cloud system), and the system may include an electronic terminal in which an application for online service for interaction with a user is installed. can do.

In this case, the electronic terminal may be implemented as a computer, a portable terminal, a television, a wearable device, or the like, connected to a remote server through a network N or connected to other terminals and servers. Here, the computer includes, for example, a laptop, desktop, laptop, etc., which is equipped with a web browser, and the portable terminal is, for example, a wireless communication device that ensures portability and mobility. , Personal Communication System (PCS), Personal Digital Cellular (PDC), Personal Handyphone System (PHS), Personal Digital Assistant (PDA), Global System for Mobile communications (GSM), International Mobile Telecommunication (IMT) -2000, Code CDMA Division Multiple Access (2000), all types of handhelds such as W-Code Division Multiple Access (W-CDMA), Wireless Broadband Internet (Wibro), Smart Phones, and Mobile Worldwide Interoperability for Microwave Access (WiMAX). It may include a (Handheld) based wireless communication device. In addition, the television may include an Internet Protocol Television (IPTV), an Internet Television (Internet Television), a terrestrial TV, a cable TV, or the like. Further, the wearable device is, for example, an information processing device of a type that can be worn directly on a human body such as a watch, glasses, accessories, clothes, shoes, etc., and is connected to a remote server or another terminal via a network directly or through another information processing device. It can be connected with.

The image analysis apparatus 100 according to an embodiment may include a controller 110 and a memory 120.

The controller 110 controls the overall operation of the image analysis apparatus 100 and may include a processor such as a CPU. For example, the controller 110 may control other components included in the image analysis apparatus 100 to perform an operation corresponding to a user input received through the input / output unit 130.

For example, the controller 110 may execute a program stored in the memory 120, read a file stored in the memory 120, or store a new file in the memory 120.

According to one embodiment described herein, the controller 110 may acquire an image.

For example, the controller 110 may include a device (not shown) capable of radiography or computed tomography scan of the head of the patient, or communicate with the device to obtain an image from the device.

In addition, according to another exemplary embodiment, the controller 110 may determine that the preprocessed image is obtained as an image after performing the preprocessing on the acquired image.

For example, the controller 110 may apply a CLAHE parameter, Gaussian Blur, or warping on the acquired image, or scale on the X and / or Y axis. Preprocessing of an image can be performed by increasing the resolution by scaling, cropping or cropping the learning image, or changing a pixel value.

According to another embodiment, the controller 110 may acquire an image from the raw image when the acquired image is a raw image. To this end, the controller 110 may perform preprocessing for the low image.

In this regard, the low image may be an image including an image, for example, an x-ray image of the upper body of the patient, including an image of another body part of the patient including the image.

The controller 110 may search whether a predetermined identifier is located in a predetermined area on the row image. For example, it may be searched whether an identifier recognized as "orbital" is located in a predetermined area on the low image. In this case, the image analysis apparatus 100 may set an area within the window as a predetermined area when a window formed of a closed curve having an arbitrary size and shape is positioned on the low image, and the window moves on the low image. You can set a range of areas.

Also, for example, the image analysis apparatus 100 may use machine learning (or deep learning or CNN) to detect whether a predetermined identifier is located in a predetermined area of a window. For example, the image analysis apparatus 100 may perform CNN on a predetermined region to extract coordinates in which an identifier within the region is located. The coordinates may consist of one or more coordinates, may be coordinates within a predetermined area, or may be coordinates on a row image.

As such, the controller 110 extracting the coordinates of the predetermined identifier on the row image may obtain an image based on the extracted coordinates.

For example, the controller 110 may extract three identifiers of 'anwa', 'mandible' and 'nasal' which can be specified as an image in the low image. In order to extract three identifiers, the controller 110 may move a window and extract coordinates corresponding to the identifiers. The controller 110 may acquire an image by setting an area determined as an image on the low image based on the extracted coordinates. For example, the image analysis apparatus 100 may set a region including the above three identifiers and a predetermined range from each identifier as an image.

In this case, three identifiers are merely examples, and the number of identifiers is not limited. In this regard, the identifier may consist of one coordinate, but may consist of a plurality of coordinates to be a specific organ of the body. Thus, the identifier may be, for example, the central point of the 'orwa', but may be the body organ itself, such as the 'anwa', 'nose', 'spine 1', 'chin', and the like.

By additionally standardizing the acquired image, the controller 110 may process the image to facilitate extraction of the landmark spot. That is, since the controller 110 learns from the learning image, the controller 110 may standardize the image to be the same as or similar to the set value set for the learning image. For example, the image analyzing apparatus 100 may perform standardization processing such as changing the resolution, brightness, etc. of the image, or enlarging / reducing the size of the image. The controller 110 may determine the image after the normalization process is obtained as the acquired image.

When it is determined that the image is acquired as described above, the controller 110 may extract the landmark point on the image.

At this time, the 'landmark point' refers to a point of interest identified on the image of the patient as a point that the medical staff should know in order to perform orthodontics, medicine, and surgery. The landmark points are variously combined and used as reference points to determine the head condition of the patient. According to the related art, an experienced medical staff selects and displays landmark points directly on an image, but according to the embodiments described herein, image analysis Device 100 may extract landmark points on the image.

According to an embodiment, the controller 110 may extract the landmark point on the image through a geometric operation. That is, a landmark point having no feature point or a geometrically defined landmark point on the image may be extracted through geometric calculation. For example, a point derived by calculating a bisection of a mandibular plane and a ramus may be extracted as a landmark point of a chinion point (gonion).

According to another embodiment, the controller 110 may extract landmark points on the image using machine learning.

According to another exemplary embodiment, the controller 110 may extract landmark points on the image using deep learning.

According to another embodiment, the controller 110 may extract a landmark point on the image using a convolutional neural network (CNN). To this end, the image analysis apparatus 100 may acquire one or more images in which the landmark points are displayed, learn CNNs using the one or more images, and extract landmark points on the images using the learned CNNs. .

To this end, the controller 110 creates a feature map through convolution, for example, from an image acquired for learning of the CNN, and obtains through subsampling the feature map. Performing convolution and subsampling on a feature obtained by performing convolution and subsampling on a local feature to acquire a feature, and performing convolution and subsampling on the obtained feature again The global feature obtained by repeatedly performing the step may be connected to an input of a general neural network to produce an optimal recognition result.

That is, the controller 110 may learn using the learning image in which the landmark point is displayed (or the identification image is combined with the landmark mark) in order to extract the landmark point on the image as described above. Pre-processing may be performed on the training image for the above-described CNN training, for example, applying CLAHE parameters, Gaussian blur, warping, or X-axis and / or The resolution can be increased by scaling on the Y axis, reducing or cropping the training image, or changing pixel values.

In addition, the controller 110 may extract a landmark point on the image by performing segmentation using, for example, the CNN. Segmentation is a technique for distinguishing meaningful parts on an image, and various disclosed segmentation methods may be applied to the embodiments described herein. For example, after checking the distribution of pixels using a histogram, a threshold value of an appropriate value can be set to distinguish by pixel, or a meaningful edge can be extracted and distinguished, and also has homegenety. It can be divided into areas.

According to another embodiment, the controller 110 may set a region of interest on the image and extract a landmark point in the region of interest. The image analysis apparatus 100 may use machine learning (or deep learning or CNN) to set a region of interest, and may use machine learning (or deep learning or CNN) to extract landmark points in the region of interest. .

3 is an exemplary diagram for describing an image analyzing apparatus 100 setting a region of interest on an image and extracting landmark points.

As shown in FIG. 3A, the image analysis apparatus 100 may set one or more regions of interest 310, 320, 330, 340, and 350 on the image 300, and each region of interest may have a different shape, It may have different sizes or may have the same shape or the same size.

When setting the ROIs 310, 320, 330, 340, and 350, the image analysis apparatus 100 may detect an area determined as a medically significant point by using machine learning. In other words, the location of the object of interest may be identified by machine learning to be divided into a bounding box. For example, the image analysis apparatus 100 may detect an area determined to be “chin” by using machine learning, and define the jaw. The encompassing area can be set as the region of interest. Published techniques related to detection can be applied to the embodiments described herein. In addition, when setting the ROIs 310, 320, 330, 340, and 350, the image analysis apparatus 100 may set an area within the window as the ROI when the window is positioned on the image 300. Accordingly, the image analysis apparatus 100 may set the region of interest as the region of interest by changing the region of interest while moving the window or by placing the window at a fixed position.

In this case, as shown in (a) of FIG. 3, the ROI may be divided into a region of interest in a solid line and a lower region of interest divided by a dotted line in the region of interest. The lower region of interest is a region determined to be a major point for acquiring a landmark point. For example, the lower region of interest may be set as a lower region of chin, or a region around the corner of the lower region of chin. A region of interest may be additionally set based on a center point to surround the lower region of interest. Through this, it is possible to more accurately grasp the landmark spot by minimizing the error of landmark spot extraction due to the incorrect setting of the region of interest.

Landmark points can be extracted from the selected ROI, and there is a learning model for the points to be extracted as landmark points, and segmentation is performed on the models to generate and generate probability image maps. A blob may be set in the probability map to extract the midpoint of the blob as a landmark point. Such landmark point extraction may be performed for each region of interest 310, 320, 330, 340, 350. For example, for the region of interest 310 in FIG. As shown on the ROI 311 of FIG. 3B, the landmark points may be extracted and the extracted landmark points may have a high level of accuracy.

As described above, the portion of the image to be the main analysis target may be set as the region of interest using machine learning (or deep learning or CNN), and landmark points within the region of interest may be extracted. As a result, it is possible to extract landmark points quickly and accurately by solving a problem that it is difficult to extract the landmark points when the angle or position of the image is not aligned.

Meanwhile, according to another exemplary embodiment, the controller 110 may extract the landmark point to be extracted by using machine learning (or deep learning or CNN) and extract it by geometric calculation. Thus, for example, when extracting a tuck corner point as a landmark point, the controller 110 may determine one point as a tuck corner point among the points extracted through the machine learning and the geometric calculation, respectively. When the reliability of the landmark point extracted using machine learning is less than or equal to a predetermined value, the 110 may extract the point obtained by the geometric operation as the tuck corner point.

According to another exemplary embodiment, when the landmark point is to be extracted, the controller 110 may also verify the extracted point by geometric calculation when extracted using machine learning (or deep learning or CNN).

In this case, it may be verified by determining whether the landmark point extracted by machine learning is the same position as the point extracted by geometric calculation. In addition, it is possible to geometrically calculate and verify whether the surrounding points or the engine structure of the landmark points extracted by machine learning are located. In this process, Bayesian (or likelihood maximization) can be used to calculate the reliability of the position of the landmark and its relative position with the points corresponding to the surrounding structure or organ structure. For example, a landmark point extracted from the ear canal of the ear canal is the upper left of the posterior cranial base surface and the condylar head or neck (Ar; articulare). We can verify that we are located at and express the result according to the reliability. In addition, the size of the landmark point extracted by machine learning can be geometrically calculated and verified. In addition, around landmark points extracted by machine learning, whether or not a particular point or organ structure is located is determined by the size of the particular point (including both relative or absolute size with other points) or the size of the organ structure (with other organs). Can be verified by calculating relative or absolute size).

According to another exemplary embodiment, the controller 110 may extract the landmark points to be extracted by geometric operations and verify the extracted points by using machine learning (or deep learning or CNN).

For example, the controller 110 extracts a point derived by calculating a bisector of a mandibular plane and a ramus as a landmark point of a chin corner point (gonion) to extract a chin corner point. By geometrically extracting and performing the analysis using machine learning around the extracted points, it is possible to verify whether the extracted points fit the chin corner points.

In addition, the controller 110 may verify whether the ROI is properly set by machine learning (or deep learning or CNN) or geometrical calculation. For example, the controller 110 may verify whether or not the region of interest extracted by the detection of machine learning is the intended region of interest by classifying an image of the region of interest or express it with reliability. Alternatively, for example, the controller 110 may verify whether a region of interest extracted by the detection of machine learning is an intended region of interest with a point or an engine structure around the region of interest.

As described above, by performing machine learning and geometric operations together, the accuracy and reliability of the extracted landmark points can be improved.

Meanwhile, when the landmark point is extracted from the image as described above, the controller 110 may analyze the reliability of the extracted landmark.

That is, the controller 110 may calculate the reliability based on a blob in which the similarity between the landmark point on the image and the feature map generated according to the learning exceeds a preset value. This is expressed as a formula below.

[Equation 1]

Reliability =

α: weight for the longest diameter of the blob, β: weight for the kurtosis of the blob

γ: weight for the maximum probability in the blob

The controller 110 may use information on the longest diameter of the blob, the kurtosis of the blobs, the maximum probability within the blob, which are geometric information represented by the blob, as a factor for calculating reliability.

First, the controller 110 may calculate the longest diameter of the blob.

For example, the controller 110 may calculate the similarity between the landmark points extracted from the image and the feature map generated through learning from the learning image as a probability. The controller 110 may generate a blob composed of closed curves connecting coordinates having the same probability value among coordinates having a probability value exceeding a predetermined value among the calculated probability values, which is the distance among the coordinates constituting the blob. The longest diameter can be calculated.

The controller 110 may calculate Kurtosis according to the following equation based on the probability value of each point of the blob.

[Equation 2]

Kurtosis =

X: probability value of the coordinate point in the blob, U: probability mean in the blob, б: standard deviation

For example, the probability values for coordinates (0,1), (1,0), (1,1), (0,0), (2,1) are assigned to 0.6, 0.5, 0.8, 0.9, 0.3 If the preset probability value is 0.5, the controller 110 coordinates (0,1), (1,0), (1,1), (0,0) having a probability value of 0.5 or more of the preset probability value among the coordinates. We can calculate the probability mean 0.7 for, and calculate the standard deviation of 0.15. The controller 110 may calculate the kurtosis 0.444 according to Equation 2.

In addition, the controller 110 may identify a maximum probability value in the blob and calculate a reliability of the extracted landmark point according to Equation 1.

In this case, the controller 110 may determine the weight based on the values of the factors when calculating reliability.

For example, the shorter the diameter of the blob indicates that the accuracy of the landmark point is higher, the control unit 110 can increase the reflection of the longest diameter value of the blob when calculating reliability by increasing the weight α as the longest diameter of the blob is shorter. have.

In addition, for example, the higher the kurtosis of the blobs, the higher the accuracy of the landmark point. Thus, the controller 110 may increase the reflection value of the blobs in the reliability calculation by increasing the weight β as the blob kurtosis value is larger.

For example, since the larger the maximum probability value of the blob indicates that the extracted landmark point is higher in accuracy, the controller 110 increases the weight γ as the maximum probability value of the blob increases to reflect the maximum probability value of the blob when calculating reliability. Can increase.

Thereafter, the controller 110 may update the landmark point for the image based on the calculated reliability of the extracted landmark point.

For example, the controller 110 may display and provide the calculated reliability of the extracted landmark point, and may update the landmark point for the image based on a user input.

Alternatively, for example, the controller 110 may move the landmark points having a predetermined reliability or less at predetermined intervals based on the calculated reliability of the extracted landmark points at a predetermined interval, and calculate the reliability of the moved landmark points. When the set value is exceeded, the landmark point of the image may be updated with the moved landmark point.

In addition, the controller 110 may perform the learning as the learning image on the image of which the landmark point is updated.

For example, the controller 110 may learn the image of which the landmark point is updated based on the reliability as a learning image by using machine learning (or deep learning or CNN).

Meanwhile, when the landmark point is extracted from the image as described above, the controller 110 may generate an analysis image by combining the identification image with the landmark point.

In this case, according to an exemplary embodiment, the controller 110 may analyze the image in which the landmark point is updated according to the reliability of the landmark point on the image.

That is, the controller 110 may synthesize the identification image on the image when the image of the landmark point is updated to the analysis image based on the reliability of the extracted landmark point, and the land may be synthesized when the identification image is synthesized. The identification image can be synthesized at the mark point.

The controller 110 may provide an analysis image to a user. In addition, the controller 110 may determine whether to provide a report including an analysis image.

At this time, according to an embodiment, the controller 110 may generate and provide a report to form a part of the report as it is.

According to another embodiment, the control unit 110 may provide a report processing the analysis image.

Meanwhile, various types of data such as files, applications, and programs may be installed and stored in the memory 120. The controller 110 may access and use data stored in the memory 120 or store new data in the memory 120. In addition, the controller 110 may execute a program installed in the memory 120. Referring to FIG. 1, the memory 120 may be provided with a computer readable program for performing a method for analyzing an image. In addition, the memory 120 may store an identification image for performing an image analysis, an analysis image according to the image analysis, or a report.

According to an embodiment, upon receiving an input for requesting analysis of an image from a user, the controller 110 executes a program for executing an image analysis method stored in the memory 120 to perform image analysis.

Meanwhile, as illustrated in FIG. 2, the image analysis apparatus 100 may further include an input / output unit 130.

The input / output unit 130 may include a component supporting various types of input / output for acquiring an image or providing an analysis image to a user. It may include an output unit for displaying information such as the state of (100). For example, the input / output unit 130 may include an operation panel for receiving a user input and a display panel for displaying a screen.

In detail, the input unit may include devices capable of receiving various types of user input such as a keyboard, a physical button, a touch screen, a camera, or a microphone. In addition, the output unit may include a display panel or a speaker. However, the present invention is not limited thereto, and the input / output unit 110 may include a configuration that supports various input / output.

The image analyzing apparatus 100 may further include a communication unit 140.

The communication unit 140 may perform wired or wireless communication with another device or a network. To this end, the communication unit 140 may include a communication module supporting at least one of various wired and wireless communication methods. For example, the communication module may be implemented in the form of a chipset.

Wireless communication supported by the communication unit 140 may be, for example, Wi-Fi (Wireless Fidelity), Wi-Fi Direct, Bluetooth, UWB (Ultra Wide Band), or NFC (Near Field Communication). In addition, the wired communication supported by the communication unit 140 may be, for example, USB or High Definition Multimedia Interface (HDMI).

4 to 6 are flowcharts for describing an image analysis method, according to an exemplary embodiment.

The image analysis method according to the embodiment illustrated in FIGS. 4 to 6 includes steps processed in time series in the image analysis apparatus 100 illustrated in FIG. 1 or 2. Therefore, even if omitted below, the above descriptions of the image analysis apparatus 100 illustrated in FIG. 1 or 2 may be applied to the image analysis method according to the embodiments illustrated in FIGS. 4 to 6. .

4 to 6 will be described below with reference to FIGS. 7 to 8. 7 to 8 are exemplary diagrams for describing an image analysis method, according to an exemplary embodiment.

The image analyzing apparatus 100 may learn by using a learning image in which landmark points are displayed, in order to perform the image analyzing method according to an exemplary embodiment of the present invention. In this case, the image analyzing apparatus 100 may perform preprocessing on the learning image for learning, for example, applying a CLAHE parameter, applying a Gaussian blur, or applying warping to the learning image, or Scale on the X-axis and / or Y-axis, reduce or crop the training image, change the pixel value, or increase the resolution. The image analysis apparatus 100 trained using the preprocessed learning image may extract landmark points from the image.

As illustrated in FIG. 4, the image analysis apparatus 100 may acquire an image (S410).

The image analyzing apparatus 100 may include a device (not shown) capable of performing a radiography scan or computed tomography scan of the head, or communicate with the device, to acquire an image through the device.

In addition, the image analysis apparatus 100 may receive an image through an external device (not shown) and may acquire the received image as an image.

In relation to this, the image analysis apparatus 100 may determine whether the acquired image is an image when the image is acquired.

That is, according to an exemplary embodiment, the image analysis apparatus 100 may determine whether the acquired image is an image and determine that the image has been acquired only when it is determined that the image is an image.

For example, if an image constituting the tofu is not included by analyzing the acquired image, it may be determined that the image is not an image. That is, for example, when the acquired image is not included by analyzing the acquired image, it may be determined that the image is not an image.

In addition, according to another embodiment, the image analysis apparatus 100 may determine whether the acquired image may be analyzed as an image, and may determine that the image has been acquired only when it is determined that the image can be analyzed.

For example, the image analyzing apparatus 100 may determine whether or not it is possible to analyze by analyzing the resolution, size, etc. of the acquired image. Therefore, for example, when the acquired image has a resolution lower than the predetermined or higher resolution or smaller than the size of the predetermined or higher, the image may be determined to be difficult to analyze and may be determined to have not acquired the image.

In addition, according to another exemplary embodiment, the image analysis apparatus 100 may determine a preprocessed image as an image after performing preprocessing on the acquired image.

For example, if it is determined that the acquired image does not satisfy a predetermined criterion, the image analyzing apparatus 100 may modify the image or combine it with another image to satisfy the criterion. For example, the image analysis apparatus 100 may perform preprocessing such as increasing the resolution, adjusting the brightness, increasing or decreasing the size of the acquired image.

According to another exemplary embodiment, when the acquired image is a low image, the image analysis apparatus 110 may acquire an image from the low image.

In addition, according to another exemplary embodiment, the image analyzing apparatus 110 may additionally standardize the acquired image. The controller 110 may determine the image after the normalization process is obtained as the acquired image.

If it is determined that the image has been acquired as described above, the image analysis apparatus 100 may extract the landmark point on the image (S420).

For example, the image analyzing apparatus 100 may extract the landmark point on the image through a geometric calculation. Alternatively, for example, the image analyzing apparatus 100 may extract landmark points on the image using machine learning, deep learning, or CNN.

Alternatively, for example, a region of interest on the image may be extracted and a landmark point in the region of interest may be extracted. The image analysis apparatus 100 may use machine learning, deep learning, or CNN to extract a region of interest, and may use machine learning, deep learning, or CNN to extract a landmark point in the region of interest.

In relation to step S420, as shown in FIG. 5, when the first point is analyzed by analyzing the image (S510), the first point is verified based on the second point, and when it is determined that the first point is extracted correctly, the first point is determined. One point may be determined as a landmark point (S520).

In this case, each of the first point and the second point refers to a point on the image and is merely a name for distinguishing from each other, and does not refer to a specific point.

According to various embodiments of the present disclosure, the image analysis apparatus 100 may identify a first point estimated as a landmark point on an image according to various embodiments.

According to an embodiment, the image analysis apparatus 100 may extract a first point, which is estimated as a landmark point, on the image through a geometric calculation. That is, a landmark point having no feature point on the image or a landmark point defined geometrically may be extracted through geometric calculation. For example, a first point derived by calculating a bisection of a mandibular plane and a ramus may be extracted as a landmark point of a chinion point (gonion).

According to another exemplary embodiment, the image analyzing apparatus 100 may extract a first point estimated as a landmark point on the image by using machine learning.

According to another exemplary embodiment, the image analyzing apparatus 100 may extract the first point estimated as a landmark point on the image by using deep learning.

According to another exemplary embodiment, the image analyzing apparatus 100 may extract a first point estimated as a landmark point on the image by using the CNN. To this end, the image analysis apparatus 100 may obtain one or more images in which the landmark points are displayed, and extract the landmark points on the image using the one or more images and the CNN.

To this end, the image analysis apparatus 100 creates a feature map through convolution, for example, from an image acquired for learning of the CNN, and through subsampling the feature map. Perform convolution and subsampling on the acquired local feature to perform convolution and subsampling on the acquired feature to acquire the feature, and then perform convolution and subsampling on the obtained feature again. By performing the steps repeatedly performed, the obtained global feature may be connected to an input of a general neural network to produce an optimal recognition result.

Also, the image analyzing apparatus 100 may extract a landmark point on an image by performing segmentation using, for example, a CNN. Segmentation is a technique for distinguishing meaningful parts on an image, and various disclosed segmentation methods may be applied to the embodiments described herein. For example, after checking the distribution of pixels using a histogram, a threshold value of an appropriate value can be set to distinguish by pixel, or a meaningful edge can be extracted and distinguished, and also has homegenety. It can be divided into areas.

According to another exemplary embodiment, the image analyzing apparatus 100 may extract a region of interest on an image and extract a first point estimated as a landmark point in the region of interest. The image analysis apparatus 100 may use machine learning, deep learning, or CNN to extract a region of interest, and may also use machine learning, deep learning, or CNN to extract a first point estimated as a landmark point in the region of interest. Can be.

According to another embodiment, the image analysis apparatus 100 may set an area within the window as the ROI by moving the window on the image, and extract a first point that is assumed to be a landmark point in the ROI. If the number of landmark points to be extracted on the image is N (N is an integer greater than or equal to 1), the image analysis apparatus 100 may extract a first point in the ROI in order to extract each of the N landmark points. N times may be performed for each mark point.

Meanwhile, according to another exemplary embodiment, the image analysis apparatus 100 may extract the first point estimated as the landmark point to be extracted by using machine learning and also extract it by geometric calculation. Therefore, for example, when extracting the corner of the jaw as a landmark point, the image analysis apparatus 100 may extract one point from among the points extracted through machine learning and geometric calculation, respectively, as a first point. For example, the image analysis apparatus 100 may extract a point obtained by a geometric operation as a first point when the reliability of the landmark point extracted using machine learning is equal to or less than a predetermined value.

Meanwhile, the image analysis apparatus 100 may verify the identified first point. According to an embodiment, the image analysis apparatus 100 may be extracted when extracted using machine learning, deep learning, or CNN. You can also verify geometrically on points.

In this case, the first point identified by machine learning, deep learning, or CNN can be verified by determining whether it is the same position as the second point extracted by geometric operations, or the first point extracted by machine learning, deep learning, or CNN. It can be geometrically computed and verified whether the periphery or tracheal structure of is positioned as the second point. For example, the first point as a landmark point of the pores of the ear canal is defined as the articulare of the posterior cranial base surface and the condylar head or neck. When it is determined as the second point, it can be verified whether it is located at the upper left of the second point, and the result can be expressed with reliability.

In addition, the image analysis apparatus 100 may verify whether the ROI is properly set by machine learning, deep learning, or CNN or geometrical calculation. For example, the image analysis apparatus 100 may verify whether or not the region of interest extracted by the detection of machine learning is the intended region of interest by classifying an image of the region of interest or express it with reliability. Alternatively, for example, the image analysis apparatus 100 may verify whether a region of interest extracted by the detection of machine learning is an intended region of interest using a point or an organ structure around the region of interest.

According to another exemplary embodiment, the image analysis apparatus 100 may verify the landmark point to be extracted by using machine learning, deep learning, or CNN on the geometrically extracted first point. For example, the image analysis apparatus 100 may determine whether the first point is located within a predetermined radius based on the second point obtained by performing machine learning, deep learning, or CNN on the image, and thus the first point. Whether it is appropriate as this landmark point can be verified.

If the extracted landmark point is not the intended landmark point or the intended landmark point is not extracted at all when the verification is performed according to the above-described method, the image analysis apparatus 100 may determine the landmark point. Can be extracted again.

In this case, the image analysis apparatus 100 may re-extract the landmark point with respect to the existing image. Alternatively, the image analyzing apparatus 100 may extract a landmark point by analyzing a new image by acquiring a new image instead of an existing image. To this end, the image analyzing apparatus 100 may re-acquire an image from a low image or perform standardization on the image. You can do it again.

In operation S403, the image analyzing apparatus 100 may calculate reliability of the landmark point extracted in step S420.

First, the image analysis apparatus 100 may generate a feature map obtained through convolution from an image for learning, and may generate the same probability value based on a probability value for similarity between the generated feature map and landmark points extracted from the image. You can create a blob that follows the coordinates with.

In this case, the image analysis apparatus 100 may generate the blob using only a probability value equal to or greater than a preset probability value when generating the blob.

For example, the probability values for coordinates (0,1), (1,0), (1,1), (0,0), and (2,1) are 0.6, 0.5, 0.8, 0.9, and 0.3, respectively. In this case, the image analysis apparatus 100 may generate a blob connecting the coordinates (0,1), (1,0), (1,1), (0,0) having a probability value of 0.5 or more, which is a preset probability value, in a closed curve form. Can be generated.

The image analysis apparatus 100 may calculate the maximum diameter of the blob and the kurtosis of the blob, which are geometric values of the generated blob.

For example, when the coordinates constituting the blob are (0,1), (1,0), (1,1), (0,0), the image analysis apparatus 100 may determine the distance between the coordinates constituting the blob. The longest distance √2 can be calculated as the maximum diameter of the blob. The image analysis apparatus 100 may calculate a kurtosis kurt 0.444 of a value obtained by dividing the probability value of each coordinate in the blob and the probability mean of 0.7 by the square of the difference by the fourth square of 0.15, which is the standard deviation.

 The image analysis apparatus 100 may calculate the reliability of the landmark point based on the calculated maximum diameter of the blob, kurtosis of the blob, and the maximum probability of the blob.

In this case, the image analysis apparatus 100 may change the reflected intensity according to the reliability indicated by each of the maximum diameter of the blob, the kurtosis of the blob, and the maximum probability of the blob when calculating the reliability.

For example, since the smaller the maximum diameter of the blob, the higher the reliability, the image analysis apparatus 100 may increase the weight for the maximum diameter of the blob as the maximum diameter of the blob is smaller. The higher the kurtosis value of the blob, the higher the reliability. The image analysis apparatus 100 may increase the weight of the blob kurtosis as the kurtosis value of the blob is larger.

Thereafter, the image analysis apparatus 100 may determine whether the reliability value of the landmark point is greater than or equal to a preset value (S440). If the reliability is less than or equal to the preset value, the image analysis apparatus 100 determines the landmark point. Can be updated (S450).

For example, the image analysis apparatus 100 may display a landmark point whose reliability is less than a preset value, and may update the landmark point by setting the landmark point from the user.

Alternatively, for example, the image analysis apparatus 100 may calculate reliability by randomly changing a landmark point within a predetermined radius from a landmark point whose reliability is less than a preset value, and setting the position having the highest reliability as the landmark point. To update the landmark point of the image.

Thereafter, the image analyzing apparatus 100 may perform learning as the learning image on the image of which the landmark point is updated (S460).

For example, the image analysis apparatus 100 may accurately learn a region to be extracted as a landmark point in the image by performing CNN learning based on the image of which the landmark point is updated based on the reliability.

In addition, the image analysis apparatus 100 may repeatedly learn the CNN while repeating steps S410 to S460, according to an embodiment, and extracting by repeatedly performing landmark learning by updating a landmark point according to the reliability as described above. The accuracy and reliability of the landmark point can be increased.

Meanwhile, when the image analysis apparatus 100 extracts the landmark point on the image as described above, the image analysis device 100 may generate the analysis image by combining the identification image with the landmark point (S470).

That is, the image analyzing apparatus 100 may synthesize the identification image on the image when the image is changed into the analysis image, and may synthesize the identification image at the landmark point when the identification image is synthesized.

In this case, the image analyzing apparatus 100 may synthesize the identification image having the same shape at the landmark point.

Alternatively, the image analysis apparatus 100 may store a plurality of identification images and change the identification image synthesized according to the location of the landmark spot.

On the other hand, the image analysis apparatus 100 may determine whether to provide a report including the analysis image (S460).

For example, the image analysis apparatus 100 may provide a report only when the user requests to receive the report.

According to another example, the image analysis apparatus 100 may set a report including a analyzed image as a default in the system and provide a report.

In addition, the image analysis apparatus 100 may provide a report only for an analysis image having a value equal to or greater than a predetermined threshold, for example, at a position outside a predetermined range than a position where a specific point is generally displayed. The report may be provided only for the analysis image of the patient having the head to which the identification image corresponding to the specific point is displayed.

As such, when the image analysis apparatus 100 determines to provide the report, the image analysis apparatus 100 may include the analysis image as it is or process the analysis image to generate and provide the report (S490).

In this case, according to an exemplary embodiment, the image analyzing apparatus 100 may generate and provide a report so that the analysis image may form part of the report.

According to another exemplary embodiment, the image analyzing apparatus 100 may provide a report in which the analyzed image is processed.

Meanwhile, according to an image analyzing method according to another exemplary embodiment, the image analyzing apparatus 100 may extract a landmark point by receiving a low image before acquiring an image, which will be described later with reference to FIG. 6.

As illustrated in FIG. 6, the image analysis apparatus 100 may perform preprocessing on a low image (S610). That is, the image analysis apparatus 100 may apply a CLAHE parameter, apply Gaussian blur, or warp to a low image, scale on the X axis and / or Y axis, or reduce or crop the training image ( Preprocessing for the low image may be performed by increasing the resolution by cropping or by changing the pixel value.

In operation S620, the image analyzing apparatus 100 may obtain an image from a low image.

According to an exemplary embodiment for acquiring an image, the image analysis apparatus 100 may search whether a predetermined identifier is located in a predetermined region on a low image input as an analysis target.

For example, it may be searched whether an identifier recognized as "orbital" is located in a predetermined area on the low image. In this case, the image analysis apparatus 100 may set an area within a window formed of a closed curve having an arbitrary size and shape on the low image as a predetermined area, and the window may set an area of various ranges while moving on the low image. Can be.

Also, for example, the image analysis apparatus 100 may use machine learning (or deep learning or CNN) to detect whether a predetermined identifier is located in a predetermined area of a window. For example, the image analysis apparatus 100 may perform CNN on a predetermined region to extract coordinates in which an identifier within the region is located. The coordinate may be a coordinate within a predetermined area or may be a coordinate on a row image.

As such, the image analyzing apparatus 100 extracting the coordinates of the predetermined identifier on the row image may obtain an image based on the extracted coordinates.

For example, if the low image is an x-ray image of the upper body of the patient, the image analyzing apparatus 100 may identify three identifiers of the orbit, the mandible, and the nasal cavity, which can be specified as the image in the low image. Can be extracted. In order to extract three identifiers, the image analysis apparatus 100 may extract coordinates corresponding to the identifiers by moving a window. The image analyzing apparatus 100 may obtain an image by setting a region determined as an image on the low image based on the extracted coordinates. For example, the image analysis apparatus 100 may set a region including the above three identifiers and a predetermined range from each identifier as an image.

The image analyzing apparatus 100 may process the image to facilitate extraction of the landmark spot by standardizing the acquired image. For example, the image analyzing apparatus 100 may perform standardization processing such as changing the resolution, brightness, etc. of the image, or enlarging / reducing the size of the image. Alternatively, for example, the image analyzing apparatus 100 may enlarge and scale the image in the case of an image of a pediatric or reduce and scale the image in the case of an image of a patient with a large head.

As described above, the image analyzing apparatus 100 may obtain an image and extract a landmark candidate point with respect to the image (S630).

The landmark candidate point means a point before being verified as a point extracted as a landmark point.

To extract the landmark candidate points, the image analysis apparatus 100 may extract a region of interest on the image and extract a landmark candidate point in the region of interest. The image analysis apparatus 100 may use machine learning (or deep learning or CNN) to extract a region of interest, and may also use machine learning (or deep learning or CNN) to extract landmark candidate points in the region of interest. have.

The region of interest can be set by moving the window or positioning it at a fixed point on the image.

If the number of landmark points to be extracted on the image is N (N is an integer greater than or equal to 1), the image analysis apparatus 100 sets the region of interest to extract each of the N landmark points, and selects the landmark candidate points within the region of interest. The extraction step may be performed N times. In this case, when it is estimated that the landmark point is located at a specific position of the image, the image analysis apparatus 100 may extract the landmark candidate point by fixing the window to the specific position. For example, in the case of a landmark point located around the jaw, the image analysis apparatus 100 may set a region including the jaw as a region of interest and extract a landmark candidate point in the region of interest.

The image analysis apparatus 100 may verify the extracted landmark candidate points (S640).

According to an embodiment, the image analysis apparatus 100 may verify whether the landmark point acquired by using machine learning (or deep learning or CNN) is out of a predetermined position by performing geometric calculations together. According to another embodiment, the image analysis device 100 is a point determined from the uppermost point or point A of the ear canal, the head of the bent bone portion from the anterior tip of the nose to the maxillary pterygium. It is possible to verify whether the extracted landmark candidate point is extracted correctly by verifying whether the projected portion) is extracted as a landmark point.

If the landmark candidate points extracted through the above-described verification are not the extracted points, or there are no intended landmark points even when all of the extracted landmark candidate points are verified, the image analysis apparatus 100 may determine a low image. The image may be re-obtained from, or the image may be normalized again to obtain a new image, and the candidate candidate point may be extracted for the new image to perform verification. That is, the accurate landmark point may be extracted by performing the steps S620 to S640 again.

Meanwhile, when the image analysis apparatus 100 extracts the landmark point on the image as described above, the image analysis device 100 may provide the analysis image or report by combining the identification image with the landmark point to generate the analysis image. In this regard, since the same reference numerals are used for the same reference numerals in each step of FIG. 4, descriptions of steps S430 to S490 of FIG. 6 will be omitted.

7 to 8 are exemplary views showing what the image analysis device 100 provides as a report.

For example, the head state of the patient may be intuitively represented by connecting segments between the identification images displayed on the analysis image, for example, to analyze the dental alveolar-lower incisor on the head of the patient. As shown in FIG. 2, the identification images may be connected by line segments to provide a report analyzing the lower teeth state on the head of the patient.

In addition, a report can be provided to the user by displaying the information corresponding to each identification image displayed on the analysis image. For example, the result of measuring the tofu using the identification image can be provided. As shown, the interincisal angle of the patient may be calculated and displayed along with the mean value.

For example, as shown in FIG. 8, a report including an image obtained by synthesizing another image on the analysis image may be provided.

The term '~' used in the above embodiments refers to software or a hardware component such as a field programmable gate array (FPGA) or an ASIC, and '~' serves a part. However, '~' is not meant to be limited to software or hardware. '~ Portion' may be configured to be in an addressable storage medium or may be configured to play one or more processors. Thus, as an example, '~' means components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, procedures, and the like. Subroutines, segments of program patent code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

The functionality provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or separated from additional components and 'parts'.

In addition, the components and '~' may be implemented to play one or more CPUs in the device or secure multimedia card.

The image analysis method according to the embodiments described with reference to FIGS. 4 to 6 may also be implemented in the form of a computer readable medium for storing instructions and data executable by a computer. In this case, the command and data may be stored in the form of program code, and when executed by the processor, a predetermined program module may be generated to perform a predetermined operation. In addition, computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer readable medium may be a computer recording medium, which is volatile and non-implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. It can include both volatile, removable and non-removable media. For example, the computer recording medium may be a magnetic storage medium such as HDD and SSD, an optical recording medium such as CD, DVD and Blu-ray Disc, or a memory included in a server accessible through a network.

In addition, the image analysis method according to the embodiments described with reference to FIGS. 4 to 6 may be implemented as a computer program (or computer program product) including instructions executable by a computer. The computer program includes programmable machine instructions processed by the processor and may be implemented in a high-level programming language, an object-oriented programming language, an assembly language, or a machine language. . The computer program may also be recorded on tangible computer readable media (eg, memory, hard disks, magnetic / optical media or solid-state drives, etc.).

Accordingly, the image analysis method according to the embodiments described with reference to FIGS. 4 to 6 may be implemented by executing the computer program as described above by the computing device. The computing device may include at least a portion of a processor, a memory, a storage device, a high speed interface connected to the memory and a high speed expansion port, and a low speed interface connected to the low speed bus and the storage device. Each of these components are connected to each other using a variety of buses and may be mounted on a common motherboard or otherwise mounted in a suitable manner.

Here, the processor may process instructions within the computing device, such as to display graphical information for providing a graphical user interface (GUI) on an external input, output device, such as a display connected to a high speed interface. Instructions stored in memory or storage. In other embodiments, multiple processors and / or multiple buses may be used with appropriately multiple memories and memory types. The processor may also be implemented as a chipset consisting of chips comprising a plurality of independent analog and / or digital processors.

The memory also stores information within the computing device. In one example, the memory may consist of a volatile memory unit or a collection thereof. As another example, the memory may consist of a nonvolatile memory unit or a collection thereof. The memory may also be other forms of computer readable media, such as, for example, magnetic or optical disks.

In addition, the storage device can provide a large amount of storage space to the computing device. The storage device may be a computer readable medium or a configuration including such a medium, and may include, for example, devices or other configurations within a storage area network (SAN), and may include a floppy disk device, a hard disk device, an optical disk device, Or a tape device, flash memory, or similar other semiconductor memory device or device array.

The above-described embodiments are for illustrative purposes, and those of ordinary skill in the art to which the above-described embodiments belong may easily change to other specific forms without changing the technical spirit or essential features of the above-described embodiments. I can understand. Therefore, it is to be understood that the above-described embodiments are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope to be protected by the present specification is represented by the following claims rather than the above description, and should be construed to include all changes or modifications derived from the meaning and scope of the claims and their equivalents. .

100: image analysis device
110: control unit
120: memory
130: input and output unit
140: communication unit

Claims (14)

An apparatus for analyzing an image,
A controller configured to extract landmark points on the image to perform reliability analysis on the extracted landmark points and generate an analysis image based on the reliability analyzed landmark points; And
And a memory for storing the generated analysis image. The method of claim 1,
The control unit,
And acquires a learning image of which the landmark point is updated based on the calculated reliability of the extracted landmark point, and performs learning. The method of claim 2,
The control unit,
And calculating the reliability based on a blob in which the similarity between the landmark point on the image and the feature map generated according to the learning exceeds a preset value. The method of claim 3, wherein
The control unit,
And calculating kurtosis based on each point probability value of the blob. The method of claim 4, wherein
The control unit,
And calculate the reliability based on the longest diameter of the blob, the kurtosis, and the maximum probability value in the blob. The method of claim 2,
The control unit,
And analyzing and updating the landmark point for the image based on the reliability calculated for the extracted landmark point. As the image analysis device analyzes an image,
Extracting landmark points on the image;
Performing reliability analysis on the extracted landmark points; And
Generating an analysis image based on the reliability analyzed landmark point. The method of claim 7, wherein
Performing the reliability analysis,
Obtaining a learning image of which the landmark point is updated based on the reliability calculated for the extracted landmark point; And
And performing learning based on the updated learning image. The method of claim 8,
The image analysis method,
And calculating the reliability based on a blob in which the similarity between the landmark point on the image and the feature map generated according to the learning exceeds a preset value. The method of claim 9,
The step of calculating the reliability,
Calculating Kurtosis based on each point probability value of the blob. The method of claim 10,
The image analysis method,
Calculating the reliability based on the longest diameter of the blob, the kurtosis and the maximum probability value in the blob. The method of claim 8,
Extracting the landmark point,
And updating and analyzing the landmark points for the image based on the reliability calculated for the extracted landmark points. A computer-readable recording medium having recorded thereon a program for performing the method of claim 7. A computer program executed by an image analyzing apparatus and stored in a medium for performing the method of claim 7.

Images