KR102097396B1 - Method for facial image registration - Google Patents

Abstract

A facial image registration method is provided in the present invention. The facial image registration method comprises the steps of: obtaining, by a computer, medical image data; dividing a skin surface from the obtained medical image data to obtain first skin surface data; obtaining, by the computer, second skin surface data extracted from the photographed facial surface image data; reducing, by the computer, the number of first surface coordinate data by adaptive sampling to the first skin surface data; and performing, by the computer, surface registration of the first skin surface data to the second skin surface data using the number of the first surface coordinate data.

Description

METHOD FOR FACIAL IMAGE REGISTRATION

The present invention relates to a method for matching a facial image, and more particularly, to a method for accurately matching a medical image and a 3D image on the face.

Image registration is a processing technique that transforms different images and displays them in one coordinate system. Through registration, it can be seen how the images obtained through different measurement methods correspond.

In medical image processing, image matching is performed to compare the image of the affected area with time or to compare the image of normal tissue.

The image registration is divided into a single method matching to match one type of images and a multiple method matching to match different types of images. Single-mode matching is to match only images obtained from one scanner or measuring device, and multiple imaging methods are to match images obtained from different scanners or measuring devices.

Multi-method matching is mainly used to match images obtained using various scanners, particularly in medical imaging. For example, matching between magnetic resonance imaging (MRI) and computed tomography (CT) images, matching between positron emission tomography (PET) and computed tomography (CT) images, or matching between ultrasound and computed tomography images It is to match the images acquired in various ways.

Korean Patent Publication No. 10-2015-0088972, 2015.08.04.

The problem to be solved by the present invention is to provide a method of accurately matching a medical image and a 3D image that can be usefully used for simulation in order to predict a result through simulation before surgery and plan surgery.

In addition, the problem to be solved by the present invention is to provide a matching method that further shortens the execution time when matching the surfaces of medical images and 3D images.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

A method of matching a facial image according to an embodiment of the present invention for solving the above-described problem is a step of obtaining a medical image data by a computer, and dividing a skin surface from the obtained medical image data to obtain first skin surface data Step, acquiring the second skin surface data extracted from the facial surface image data photographed by the computer, reducing the number of first surface coordinate data by applying the adaptive sampling to the first skin surface data by the computer. And a computer performing surface registration of the first skin surface data to the second skin surface data using the number of first surface coordinate data.

The face image registration method according to an embodiment of the present invention for solving the above-described problem is the same from each of the first skin surface data and the second skin surface data before the step of reducing the number of first surface coordinate data. Extracting one or more landmark points at a location, and shifting and converting the first skin surface data such that the center point of the landmark point of the first skin surface data matches the center point of the landmark point of the second skin surface data And performing an initial matching.

In the step of acquiring the first skin surface data, selecting a skin region by applying Otsu binarization, selecting a point as a seed point that is a region selected using Otsu binarization and having a brightness value equal to or greater than a predetermined value. Filling the empty space inside with a hole filing method, the mask further comprising filling the empty space inside the mask, which is a divided skin area by applying the selected seed point to a 3D seed region growth method. Includes.

In the step of performing the surface registration, the facial image matching method according to an embodiment of the present invention uses adaptive sampling to use a cheek, forehead, and cheekbone, which are regions with a small curvature, as a reference area. Characterized by matching,

Applying the adaptive sampling is characterized by applying adaptive sampling for each group by grouping the first surface coordinate data of the same y coordinate based on the first skin surface data on the y-axis, but apart at predetermined intervals. Determining a distance difference between the predetermined number of first surface coordinate data and calculating a curvature to select the first surface coordinate data below a predetermined curvature value, and using the selected first surface coordinate data, a predetermined interval And reflecting each curvature by moving and selecting the separated first surface coordinate data.

In the step of performing the surface registration, calculating a 3D Gaussian mask of the second skin surface data, coinciding the center of the 3D Gaussian mask calculated with respect to the second surface coordinate data of the second skin surface data Step of assigning a Gaussian weight to a periphery corresponding to the mask, Computing an average value of a Gaussian weighted distance map with respect to the coordinate values of the first surface coordinate data and the second surface coordinate data using the Gaussian weight and the The method further includes extracting a location where the average value of the weighted distance map becomes the maximum.

The face image matching program according to another embodiment of the present invention for solving the above-described problems is combined with a hardware computer, and is stored in a medium to execute the above method.

Other specific matters of the present invention are included in the detailed description and drawings.

According to the present invention, it is possible to accurately match a medical image and a 3D image by performing surface registration by adaptive sampling.

In addition, according to the present invention, by using only the surface coordinate data spaced apart at regular intervals among the surface coordinate data by adaptive sampling, it is possible to accurately match the image matching time while reducing the time.

In addition, according to the present invention, before adaptive sampling is applied, initial matching using a predetermined number of landmark points is performed, thereby reducing the amount of computation in image matching and reducing the time for converging the optimal position of image matching. .

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a flow chart for explaining a facial image matching method according to an embodiment of the present invention.
2 is a flow chart for explaining a method of dividing a skin surface among face image registration methods.
Figure 3 is a view showing the results of each step of dividing the skin surface from the medical image data.
4 is a flowchart for explaining a method of applying adaptive sampling in a step of reducing the number of first surface coordinate data among the face image matching methods.
5 is a diagram for explaining the principle of adaptive sampling in the step of reducing the number of first surface coordinate data among the face image matching methods.
6 is a diagram showing the results of applying adaptive sampling.
7 is a flowchart for describing a method of performing surface registration among face image matching methods.
8 is a flowchart for explaining a method of matching a facial image after performing initial matching in the facial image matching method according to an embodiment of the present invention.
9 is a view showing a result of matching the facial image by various methods, including a method for matching the facial image according to an embodiment of the present invention.

Advantages and features of the present invention, and methods for achieving them will be clarified with reference to embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only the present embodiments allow the disclosure of the present invention to be complete, and are common in the technical field to which the present invention pertains. It is provided to fully inform the skilled person of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, the singular form also includes the plural form unless otherwise specified in the phrase. As used herein, “comprises” and / or “comprising” does not exclude the presence or addition of one or more other components other than the components mentioned. Throughout the specification, the same reference numerals refer to the same components, and “and / or” includes each and every combination of one or more of the mentioned components. Although "first", "second", etc. are used to describe various components, it goes without saying that these components are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings commonly understood by those skilled in the art to which the present invention pertains. In addition, terms defined in the commonly used dictionary are not ideally or excessively interpreted unless explicitly defined.

The spatially relative terms “below”, “beneath”, “lower”, “above”, “upper”, etc., are as shown in the figure. It can be used to easily describe a correlation between a component and other components. The spatially relative terms should be understood as terms including different directions of components in use or operation in addition to the directions shown in the drawings. For example, if a component shown in the drawing is flipped over, the component described as "below" or "beneath" the other component will be placed "above" the other component. You can. Thus, the exemplary term “below” can include both the directions below and above. Components can also be oriented in different directions, and thus spatially relative terms can be interpreted according to orientation.

In the present specification, 'computer' includes all of various devices capable of performing arithmetic processing and providing a result to a user. For example, the computer is not only a desktop PC, a notebook, but also a smart phone, a tablet PC, a cellular phone, and a personal communication service phone (PCS phone), synchronous / asynchronous. A mobile terminal of an International Mobile Telecommunication-2000 (IMT-2000), a Palm Personal Computer (PC), and a Personal Digital Assistant (PDA) may also be applicable. Further, the computer may also be a medical computer, a medical PC, a medical tablet, a medical device, and a server that receives a request from a client and performs information processing.

In this specification, 'medical image data' includes all medical image data obtained by a medical image processing method using tomography generated by computer processing. For example, medical imaging data includes CT (Computer tomography), Nuclear Magnetic Resonance Computed Tomography (NMR-CT), Positron Emission Tomography (PET), and Conebeam CT (CBCT). It may apply.

In the present specification, 'face surface image data' is face surface image data photographed from at least one of a 3D camera, a 3D scanner, an optical camera, a C-arm device, and an optical coherence tomography. .

In the present specification, 'first surface coordinate data' refers to a surface point in a skin surface image obtained by dividing a skin surface from medical image data.

In the present specification, the “second surface coordinate data” means a surface point in the facial surface image data.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flow chart for explaining a facial image matching method according to an embodiment of the present invention.

Referring to FIG. 1, the method of matching a facial image includes obtaining a medical image data by a computer (S100), obtaining a first skin surface data by dividing a skin surface from the medical image data (S120), and facial surface image data In step (S140) of obtaining the second skin surface data, the computer applies adaptive sampling to the first skin surface data to reduce the number of first surface coordinate data (S160) and the first skin surface data is second And performing a surface registration on the skin surface data (S180).

The step of obtaining the first skin surface data by dividing the skin surface from the medical image data (S120), when performing image registration, the corresponding portion for image registration in the medical image data is skin, so that the skin surface portion is firmly extracted In order to divide the skin surface, the first skin surface data is data obtained by dividing the skin surface from medical image data.

2 is a flow chart for explaining a method of dividing a skin surface among face image registration methods.

Referring to FIG. 2, the method of dividing the skin surface is a step of selecting a skin region by applying Otsu binarization (S122), and selecting a point having a brightness value equal to or greater than a predetermined value as a seed point by using Otsu binarization. Step S124 and filling an empty space inside the mask (S126).

The step of selecting the skin region by applying Otsu binarization (S122) is a step of more accurately dividing the skin layer by reducing noise of image data.

Otsu binarization refers to what gray value to separate pixels from, and sets a cost function in threshold setting and takes a threshold value as a minimum value of the set cost function, that is, it is fixed when Otsu binarization is applied. Instead of using a threshold, it is possible to perform segmentation of the skin region using a threshold optimized for the data.

The mask is selected by using Otsu binarization, and the mask is three-dimensionally selected from the seed point in step S124 of selecting a point having a brightness value equal to or higher than a predetermined value and filling an empty space inside the mask (S126). The skin region is divided by applying the seed region growth method.

The seed region growth method is one of various methods for classifying images, and an identification tag when a user identifies another region of the image is called a seed display. The algorithm of the seed area growth method starts with the seed display and repeatedly increases the size of the seed display area. While the region is expanding, the algorithm must determine which pixels are incorporated into a given seed display region, and determining which pixels are incorporated into a given seed display region is based on similarity measurements.

If an empty space exists inside the mask on the divided skin surface, an unnecessary surface is generated, so that a seed point is selected inside the mask to fill the empty space to prevent the generation of unnecessary surfaces.

When there is an empty space inside the mask on the divided skin surface, there may be various cases, and in particular, it may be a case where metal artifacts are generated.

A representative CT (Computed Tomography) of a medical imaging device that acquires medical image data rotates a source that outputs X-rays and a detector that accepts X-rays that have passed through the object, around the object. It is a device to acquire a cross-sectional image for. The X-rays output from the source pass through the target object and are absorbed by the detector, and the amount of X-rays absorbed by the detector depends on the nature and length of the target object placed in the path where the X-rays pass.

When the target object is rotated in the same way as described above and passed through X-rays, a graph indicating the amount of X-ray absorption by each rotation angle is obtained. The visualization of the sum of the amount of X-rays absorbed for each rotation angle is called a sinogram, and if a certain operation is performed on the sinogram, a tomography image of the target object can be obtained.

However, if a metal object is included in the object to be photographed, the sinogram is abnormally obtained due to the metal material. As a result, even if it is a uniform material in a tomography image, the middle part of the object appears darker than the outer part (cupping artifact), or the dark or light part between the metal materials appears as a streak artifact or metal artifact. There may be a problem.

Therefore, the skin region segmentation by the 3D seed region growth method is performed by selecting a point that is a region selected by using Otsu binarization and having a brightness value equal to or higher than a predetermined value as a seed point, and then filling an empty space inside the mask.

In addition, in the method of matching facial images according to an embodiment of the present invention, the method may further include three-dimensional modeling of the mask in which the skin region is segmented by Otsu binarization and 3D seed region growth using a marching cube algorithm. have.

Figure 3 is a view showing the results of each step of dividing the skin surface from the medical image data.

Fig. 3 (a) is medical image data for the face, and Fig. 3 (b) divides the skin by applying it to the 3D seed region growth method using the selected region as the seed point through Otsu binarization, and the divided skin 3 (c) is a view showing that the empty space generated in the mask of FIG. 3 (b) is filled by using a hole-filing method. FIG. 3 (d) is a diagram showing that skin surface modeling was finally performed.

4 is a flowchart for explaining a method of applying adaptive sampling in a step of reducing the number of first surface coordinate data among the face image matching methods.

Referring to FIG. 4, in the method of applying adaptive sampling, a distance difference between a predetermined number of first surface coordinate data spaced apart at a predetermined interval is checked, and a curvature is calculated to calculate a curvature of a first surface below a predetermined curvature value. And selecting adaptive data (S162) and applying adaptive sampling by reflecting each curvature by moving and selecting first surface coordinate data spaced at predetermined intervals (S164).

In a general surface registration method performed without adaptive sampling, registration is performed by minimizing a distance difference between surface coordinate data of two images to be matched. If the number of surface coordinate data is large, there is a disadvantage in that the image registration execution time is long. In the case of the present invention, after applying adaptive sampling, the number of surface coordinate data is reduced, and there is an effect of reducing the time of performing image registration by matching the images.

Since the area with a large curvature among the first skin surface data is a part affected by facial expressions such as eyes, nose, and mouth, the surface registration is performed using adaptive sampling in the facial image matching method according to an embodiment of the present invention. To do this, the cheeks, forehead, and cheekbones, which are regions with a small curvature, are matched using a reference region. By matching a region having a small curvature as a reference region, the number of first surface coordinate data can be reduced and images can be more accurately matched.

In order to apply adaptive sampling, before applying adaptive sampling, the first skin surface data obtained from medical image data is grouped by the surface points of the same y coordinate based on the y-axis, and then adaptive sampling is applied to each group. .

When grouping on the same position based on the y-axis, the entire outline of the face can be better represented.

5 is a diagram for explaining the principle of adaptive sampling in the step of reducing the number of first surface coordinate data among the face image matching methods.

Referring to FIG. 5, if the distance between p _i-1 and p _i is A, the distance between p _i and p _{i + 1} is B, and the distance between p _i-1 and p _{i + 1} is C, a region having a large curvature In A + B> C + ε, in flat areas A + B <C + ε. ε is an element that can control the curvature and adds it to C. Therefore, pi in the region satisfying the condition of A + B <C + ε is adopted to select the first surface coordinate data in the region having a small curvature.

5 (a) is a region with a large curvature, A + B> C + ε, and it can be confirmed that the condition is not satisfied, and FIG. 5 (b) is a region with a small curvature, A + B <C It can be seen that the condition of + ε is satisfied.

In the face image matching method according to an embodiment of the present invention, in order to apply adaptive sampling, the adjacent three first surface coordinate data in the first skin surface data do not reflect the overall curvature of the surface, so a predetermined constant The distance difference between the predetermined number of first surface coordinate data spaced apart is checked, and the curvature is calculated to select the first surface coordinate data below a predetermined curvature value (S162).

In one embodiment, the predetermined predetermined interval is 10 first surface coordinate data intervals, and the curvature is calculated in advance by checking the distance difference between the three first surface coordinate data spaced apart from the 10 first surface coordinate data intervals. The first surface coordinate data below a predetermined curvature value may be selected, but is not limited to the above example, and the interval and number of the first surface coordinate data may be determined in advance.

After selecting the first surface coordinate data, adaptive sampling is applied by reflecting each curvature (S164) by moving and selecting the first surface coordinate data spaced apart at predetermined intervals using the selected first surface coordinate data (S164). do.

6 is a diagram showing the results of applying adaptive sampling.

FIG. 6 (a) is a view showing first skin surface data divided from medical image data, and FIG. 6 (b) is a view showing a result of applying adaptive sampling with ε of 0.5.

If ε is greater than 0.5, a large portion of the curvature remains, and if ε is less than 0.5, sampling may occur in a flat area. You can.

7 is a flowchart for describing a method of performing surface registration among face image matching methods.

The method of performing surface registration includes calculating a 3D Gaussian mask of the second skin surface data (S182), and matching the center of the 3D Gaussian mask with the second surface coordinate data of the second skin surface data ( S184), assigning a Gaussian weight to the periphery corresponding to the mask (S186), calculating an average value of a Gaussian weighted distance map for coordinate values of the first surface coordinate data and the second surface coordinate data (S188) and the weight And extracting a location where the average value of the distance map becomes the maximum (S190).

In step S182 of calculating the 3D Gaussian mask of the second skin surface data, the 3D Gaussian mask is calculated by Equation 1 below.

<Equation 1>

In Equation 1, G is a Gaussian mask, x, y, and z are second surface coordinate data values, σ is a Gaussian variance, and c is a Gaussian mask center point coordinate.

When the value of the mask calculated by Equation 1 is compared, the larger the size of the mask converges accurately even if there is a shape difference in the local area, and the smaller the size of the mask, the better the image with a smaller shape difference.

Thereafter, the center of the three-dimensional Gaussian mask is matched with respect to the second surface coordinate data of the second skin surface data (S184), and the step of assigning a Gaussian weight to the periphery corresponding to the mask (S186) is performed.

In the case of a location where the weight already exists, it is updated and assigned to a larger value among the existing weight and the weight to be assigned. According to this, in the Gaussian weighted distance map, 1 is assigned to the position of the second surface coordinate data, and as it moves away from the second surface coordinate data, a Gaussian distribution is shown and a small weight is assigned.

In the step of calculating the average value of the Gaussian weighted distance map with respect to the coordinate values of the first surface coordinate data and the second surface coordinate data (S188) and extracting the location where the average value of the weighted distance map is the maximum (S190), the two The degree to which the images are locally matched and the average value of the Gaussian weighted distance map when the second surface coordinate data is similarly transformed is calculated by Equation 2 below.

<Equation 2>

In Equation 2, T is a pseudo-transformation function, GW is a distance map value of second coordinate data corresponding to coordinates, and N is the number of first surface coordinate data.

When the first skin surface data and the second skin surface data are matched, the weight value increases as the distance increases and the weight increases as the distance increases, thereby repeatedly searching and converging the location where the GWD is the maximum, thereby matching the image. Can be done accurately.

8 is a flowchart for explaining a method of matching a facial image after performing initial matching in the facial image matching method according to an embodiment of the present invention.

Referring to FIG. 8, in the method of matching a facial image according to an embodiment of the present invention, after obtaining the second skin surface data from the facial surface image data (S140), the first skin surface data and the second skin surface data are obtained. The step of extracting one or more landmark points at the same location from each (S145) and shifting the first skin surface data so that the center points of the landmark points of the first skin surface data and the second skin surface data coincide to perform initial matching. Further comprising the step of performing (S150)

In the step of performing the initial registration by using the extracted landmark point and extracting the landmark point, the size of the image obtained by the facial surface image data varies depending on the shooting location, and is modality with the medical image data. Since the initial position or the direction of the front face does not coincide with each other, initial matching is performed using a predetermined number of landmarks to match the initial position and the overall size. The landmark selects the location that is least affected by the shooting condition.

By performing the initial matching, in performing the surface matching, the amount of calculation can be reduced, and accordingly, there is an effect that it may take less time to converge to the optimal position.

9 is a view showing a result of matching the facial image by various methods, including a method for matching the facial image according to an embodiment of the present invention.

FIG. 9 (a) is a view of joining images based only on points, and is a view showing results of image joining by initial registration described in FIG. In the description of FIG. 8, a method of matching an image by applying adaptive sampling after performing initial matching by extracting a landmark point has been described, but FIG. 9A shows that only point-based image concatenation is performed. This diagram shows the result of not applying the surface registration including sampling.

9 (b) is a view showing the results of image bonding by surface registration. Unlike the present invention, surface matching is performed using all surface coordinate data without applying sampling.

9 (c) is a diagram showing the results of image bonding by surface registration to which uniform sampling is applied. Unlike the present invention, uniform sampling is applied instead of adaptive sampling, and uniform sampling is performed at coordinates separated by equal distances based on the x-axis and the y-axis from the first surface coordinate data extracted from the first skin surface data. It is by the method of extracting the first surface coordinate data.

9 (d) is a diagram showing the results of image bonding by surface registration to which adaptive sampling is applied according to an embodiment of the present invention.

FIG. 9 is a color map showing the distance difference between the matched surfaces in color, and the smaller the distance difference between the two images is, the white and green are displayed, and the larger the positional error, the red and blue. .

Fig. 9 (a) is entirely displaced, in particular, it can be confirmed that the forehead portion did not fit well, and in Fig. 9 (b) and (c) the forehead and cheekbones fit well, but relatively large in the mouth and chin area You can see that it has an error.

9 (d) by the facial image splicing method according to an embodiment of the present invention, it can be confirmed that the white portion having a distance difference occupies the most area, and the cheeks, jaw, and mouth areas are well matched. have.

The steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, a software module executed by hardware, or a combination thereof. The software modules may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside on any type of computer readable recording medium well known in the art.

The embodiments of the present invention have been described above with reference to the accompanying drawings, but a person skilled in the art to which the present invention pertains may implement the present invention in other specific forms without changing its technical spirit or essential features. You will understand. Therefore, it should be understood that the above-described embodiments are illustrative in all respects and not restrictive.

Claims (7)

A computer obtaining medical image data;
Dividing the skin surface from the obtained medical image data to obtain first skin surface data;
Acquiring second skin surface data extracted from the facial surface image data photographed by the computer;
Reducing the number of first surface coordinate data by adaptive sampling by the computer to the first skin surface data;
And the computer performing surface registration of the first skin surface data to the second skin surface data using the number of first surface coordinate data.
Applying the adaptive sampling,
Characterized in that adaptive sampling is applied to each group by grouping the first surface coordinate data of the same y coordinate based on the first skin surface data on the y-axis.
How to match facial images. According to claim 1,
Before the step of reducing the number of first surface coordinate data,
Extracting one or more landmark points at the same location from each of the first skin surface data and the second skin surface data; And
Performing initial matching by moving and transforming the first skin surface data such that the center point of the landmark point of the first skin surface data matches the center point of the landmark point of the second skin surface data.
Further comprising, a facial image registration method. According to claim 1,
In the step of obtaining the first skin surface data,
Selecting a skin region by applying oats binarization;
Selecting a seed point as a point selected by using Otsu binarization and having a brightness value equal to or greater than a predetermined value; And
Filling the empty space inside the mask by a hole-filing method, the mask further comprising filling the empty space inside the mask, which is a divided skin region, by applying the selected seed point to a 3D seed region growth method. Including,
How to match facial images. According to claim 1,
In the step of performing the surface registration,
Using adaptive sampling, the cheek, forehead, and cheekbones, which are areas with a small curvature, are matched using a reference area.
How to match facial images. According to claim 1,
Applying the adaptive sampling,
Checking a distance difference between a predetermined number of first surface coordinate data spaced apart at predetermined intervals and calculating a curvature to select first surface coordinate data below a predetermined curvature value; And
Reflecting each curvature by moving and selecting the first surface coordinate data spaced apart at predetermined intervals using the selected first surface coordinate data,
How to match facial images. According to claim 1,
In the step of performing the surface registration,
Calculating a three-dimensional Gaussian mask of the second skin surface data;
Matching the center of the three-dimensional Gaussian mask calculated with respect to the second surface coordinate data of the second skin surface data;
Assigning a Gaussian weight to a periphery corresponding to the mask;
Calculating an average value of a Gaussian weighted distance map with respect to the coordinate values of the first surface coordinate data and the second surface coordinate data using the Gaussian weights; And
Further comprising the step of extracting a location where the average value of the weighted distance map is the maximum,
How to match facial images. A facial image matching program stored in a medium to execute the method of any one of claims 1 to 6 in combination with a computer that is hardware.

Images