KR102191542B1 - Oral data provision system - Google Patents

Abstract

The present invention relates to an oral data provision system comprising: an oral scanner for generating a photographed image by scanning the inside of the oral cavity; and an oral data processing device for an oral structure based on the photographed image received from the oral scanner. According to an aspect of the present invention, precise oral modeling data can be generated even if the photographing angle is changed during oral scanning.

Description

Oral data provision system {ORAL DATA PROVISION SYSTEM}

The present invention relates to an oral data providing system, and more particularly, to an oral data providing system capable of modeling precise three-dimensional oral structure data using an image collected from an oral scanner.

In general, oral diseases, if not properly treated at the right time, not only cause great discomfort to daily life, but also cause abnormal appearance, so early treatment is important even for small symptoms, and prevention before onset is more effective if possible. to be.

However, patients with oral diseases do not easily visit the dentist until they feel a toothache due to personal time and cost, so only a few customers regularly refer to the dentist for diagnosis and treatment.

In addition, in order to treat Chinese characters for oral diseases, various photos and data such as X-ray images and oral images are required for diagnosis, treatment, and treatment progress. Recently, the oral cavity is digitally scanned and scanned using an oral scanner. A technology for modeling and displaying the generated oral data in three dimensions has been disclosed.

However, in order to model the structure of the oral cavity, it is necessary to collect the image data scanned inside the oral cavity.In the past, since the doctor takes the image data while manually entering the oral scanner into the oral cavity, If the photographing angle is changed due to a factor, there is a problem in that a different photographed image of the same tooth is generated and the modeling result may be distorted.

In order to solve this problem, image data must be acquired by a mechanical method, but compared to the manual data acquisition method, it takes a lot of time and requires expensive equipment, so there is a cost limit.

On the other hand, the above-described background technology is technical information that the inventor possessed for derivation of the present invention or acquired during the derivation process of the present invention, and is not necessarily a known technology disclosed to the general public prior to filing the present invention. .

Korean Patent Registration No. 10-1902391 Korean Patent Publication No. 10-1825216

One aspect of the present invention provides an oral data providing system capable of modeling precise three-dimensional oral structure data using an image collected from an oral scanner.

The technical problem of the present invention is not limited to the technical problem mentioned above, and other technical problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

An oral data providing system according to an embodiment of the present invention includes an oral scanner for generating a local image by scanning the inside of the oral cavity; And an oral data processing apparatus for the entire internal structure of the oral cavity based on the localized image received from the oral scanner.

The oral data processing device,

A photographed image collection unit sequentially collecting the local photographed images received from the oral scanner;

An image matching unit for matching a plurality of localized images captured from different angles based on any one localized image;

An oral data generation unit for generating oral data representing the entire oral cavity structure based on the localized image matched by the image matching unit; And

Includes a display unit for displaying the oral data as image information,

The image matching unit,

A reference image setting unit for setting a reference image among a plurality of localized images;

A representative for extracting a plurality of feature points from a reference image projected on the virtual object and a local photographed image subsequently received after the reference image, and matching feature points of the local photographed image corresponding to the feature points of the reference image Point extraction unit;

A region dividing unit that divides the reference image into a plurality of regions according to depth information, and extracts representative feature points for each region from a plurality of feature points distributed in each of the divided regions;

A boundary point at a boundary between different regions is calculated based on the representative feature points for each region, and a boundary point of the local photographed image is calculated according to a distance ratio between the representative feature points for each region of the reference image and the boundary, and the calculated local A boundary point calculator for dividing the local photographed image into a plurality of areas based on the boundary point of the photographed image; And

Warping for setting a warping point for each area of the localized image, calculating a transformation matrix of the warping point to warp the localized image for each area, and projecting the localized image for each area to the virtual object in real time Containing wealth

The reference image setting unit,

Locally photographed images taken while maintaining the preset reference 3-axis posture value at predetermined time intervals are set as the reference image,

The image matching unit,

It characterized in that the locally photographed image collected before the subsequent reference image is set is matched with the reference image,

The oral data generation unit,

The reference image is modeled in a 3D virtual space to generate a virtual object, and the 3D coordinates of the virtual object are mapped to the reference image and stored,

The area division unit,

The depth information of the reference image is analyzed based on the 3D coordinate information, and the reference image is divided by regions having the same depth section according to a preset depth section, and from a plurality of feature points distributed in each of the divided regions. Extracting the representative feature points for each region, and calculating a boundary point at the boundary between different regions based on the representative feature points for each region,

The display unit,

The oral data generated by the oral data generator is displayed as image information, and a localized image received in real time from the oral scanner is displayed in a predetermined area, and an area corresponding to the localized image in the oral data is a region of interest. Characterized in that it is marked as.

According to an aspect of the present invention described above, by setting any one of the localized images collected from the oral scanner as a reference image, and matching the remaining localized images by warping to the reference image, even if the shooting angle is changed during oral scanning Precise oral modeling data can be generated.

1 is a diagram showing a schematic configuration of an oral data providing system according to an embodiment of the present invention.
2 is a block diagram showing a detailed configuration of the oral data processing apparatus of FIG. 1.
3 is a diagram showing a detailed configuration of the image matching unit of FIG. 2.
4 to 7 are diagrams for explaining a specific image matching process performed by an image matching unit.
8 is a diagram illustrating an example of image information displayed on a display unit.

DETAILED DESCRIPTION OF THE INVENTION The detailed description of the present invention to be described later refers to the accompanying drawings, which illustrate specific embodiments in which the present invention may be practiced. These embodiments are described in detail sufficient to enable a person skilled in the art to practice the present invention. It is to be understood that the various embodiments of the present invention are different from each other, but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present invention in relation to one embodiment. In addition, it is to be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the detailed description to be described below is not intended to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all scopes equivalent to those claimed by the claims. Like reference numerals in the drawings refer to the same or similar functions over several aspects.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a diagram showing a schematic configuration of a system for providing oral data according to an embodiment of the present invention.

The oral data providing system 1 according to the present invention includes an oral scanner 100 and an oral data processing device 200.

The oral scanner 100 is a device that generates a localized image by locally scanning the inside of the oral cavity, and may be generally in the form of a conventional general oral scanner.

However, the oral scanner 100 according to the present invention is provided with a sensor (eg, a gyro sensor) for measuring the 3-axis rotation amount of the oral scanner 100 to generate information on the shooting angle of the oral scanner 100 I can.

The oral data processing apparatus 200 is a device that generates oral data for the entire internal structure of the oral cavity based on a localized image received from an oral scanner, and outputs it through a display module equipped with itself.

In addition, the oral data processing apparatus 200 displays oral data generated based on the user's input information (touch) sensed through the display module in a 3-axis rotated state or enlarged/reduced to allow the user (medical person) to Make it easy to understand the structure of the oral cavity.

To this end, the oral data processing apparatus 200 according to the present invention may be installed and executed with software (application) for generating oral data according to the method for providing oral data according to the present invention.

The oral data processing apparatus 200 may be movable or fixed. The oral data processing apparatus 200 may be in the form of a server or an engine, and may be a device, an apparatus, a terminal, a user equipment (UE), a mobile station (MS), It may be referred to in other terms, such as a wireless device or a handheld device.

The oral data processing apparatus 200 may execute or manufacture various software based on an operating system (OS), that is, a system. The operating system is a system program for allowing software to use the hardware of the device, and mobile computer operating systems such as Android OS, iOS, Windows Mobile OS, Bada OS, Symbian OS, Blackberry OS, Windows series, Linux series, Unix series, It can include all computer operating systems such as MAC, AIX, and HP-UX.

Referring to FIG. 2, the oral data processing apparatus 200 may include a photographed image collection unit 210, an image matching unit 220, an oral data generation unit 230, and a display unit 240.

The photographed image collection unit 210 collects and stores a local photographed image received in real time from the oral scanner 100.

The image matching unit 220 matches a plurality of localized images captured at different angles with respect to any one localized image.

Referring to FIG. 3 together, the image matching unit 220 includes a reference image setting unit 221, a representative point extracting unit 222, a depth classifying unit 223, a boundary point calculating unit 224, and a local image warping unit 225. ).

The reference image setting unit 221 may set any one of a plurality of local photographed images received from the photographed image collection unit 210 as a reference image.

As described above, the local imaging image is generated by a medical professional using the oral scanner 100, and there may be a case in which the imaging angle is changed during the process of scanning the inside of the oral cavity.

For example, as shown in FIG. 4, a medical practitioner is photographing a corresponding tooth at the position of the first A, and then the photographing angle may be changed due to various factors during the photographing process. In this case, when the photographing angle is changed from A to B, the shape of the tooth is distorted in the two-dimensional localized image due to the change in the photographing angle even if the same tooth is photographed.

Accordingly, the reference image setting unit 221 may set any one of the plurality of localized images as the reference image so that distortion caused by the change of the angle may be minimized even if the angle of photographing of the oral scanner is changed.

Specifically, the reference image setting unit 221 sets a local photographed image captured while maintaining a preset reference 3-axis posture value at predetermined time intervals as a reference image. For example, the reference image setting unit 221 allows the reference 3-axis posture value to be output through the display unit 240 every 10 seconds, so that the medical staff takes a local image at the shooting angle set by the reference image setting unit 221 at that time. You can guide them to collect images.

In this case, the image matching unit 200, as shown in FIG. 5, is characterized in that the localized photographed image collected before a subsequent reference image is set with respect to the reference image, and the oral data generation unit ( 230) is characterized in that the reference image is modeled in a 3D virtual space to generate a virtual object, and the 3D coordinates of the virtual object are mapped to the reference image and stored.

The feature point extraction unit 20 may extract a plurality of feature points by analyzing the projected reference image, and extract a plurality of feature points by analyzing a local photographed image. In addition, the feature point extraction unit 20 may match the feature points of the reference image and the feature points of the local photographed image. 5 is a diagram illustrating an example in which a feature point of a reference image and a feature point of a locally photographed image are matched. The feature point extracting unit 20 may analyze the extracted feature points and match feature points representing similar features. In addition, the feature point extraction unit 20 may calculate a matching value between the matched feature points. The feature point extraction and matching method may be performed in various ways using known techniques such as SIFT technique, SURF technique, ORB technique, and BRISK technique. Therefore, detailed content of extracting and matching feature points will be omitted.

The region dividing unit 30 may divide a region of the reference image projected onto the virtual object. The region dividing unit 30 may divide the region so that regions having similar depth information are divided using the depth information. As described above, when the reference image is projected onto the virtual object, the virtual space generator 10 may store the reference image and 3D coordinate information of the mapped virtual object together. Accordingly, the region dividing unit 30 may compare the depth information stored in the reference image with a preset depth section and divide it into a plurality of regions having the same depth section.

7 illustrates an example in which the region dividing unit 30 divides the reference image into a plurality of regions according to depth information using 3D coordinate information stored in the reference image. As illustrated, the region dividing unit 30 may divide the reference image into a region below D1, a region between D1-D2, a region between D2-D3, and an upper region of D3 according to depth information.

Also, the region dividing unit 30 may extract representative feature points for each divided region. The region dividing unit 30 may extract two or more representative feature points for each divided region among a plurality of feature points distributed in the reference image. The representative feature point may be extracted based on the matching value calculated by the feature point extracting unit 20. For example, the region dividing unit 30 may arrange 10 feature points located in the region 1 in the order of matching values, and select the upper n feature points as representative feature points. It can be seen that two representative feature points are selected for each area. There is no limit to the number of selected feature points, and there is no limit to the number as long as more than two are extracted.

The boundary point calculation unit 40 may calculate the position of the boundary point at the boundary between regions by using the representative feature points extracted for each region of the reference image. As described above, since 3D coordinate information of the projected virtual object is stored in the reference image, the 3D coordinates of each feature point can also be known. The boundary point calculation unit 40 may calculate boundary points and their positions in all areas in the reference image through the same method.

Meanwhile, the boundary point calculating unit 40 may calculate a boundary point position of the local photographed image based on the representative feature point of the reference image and the calculated boundary point. Specifically, the boundary point calculation unit 40 may calculate the boundary point of the local photographed image by using a distance ratio between the representative feature point of the reference image and the calculated boundary point. Unlike the reference image, the local photographed image does not store 3D location information, so the boundary point can be indirectly calculated through the reference image, which is a reference image. As described above, the feature point extraction unit 20 may match the feature points of the reference image and the feature points of the localized image. In other words, the feature point Pnm of the reference image (n: the number of divided regions of the reference image, m: the number of extracted representative feature points. For example, in the case of two extracted representative feature points, m=1 or 2) is a local It may be matched with the feature point P'nm of the captured image.

In a similar way, the boundary point calculating unit 40 may generate a line segment by calculating boundary points of a localized image corresponding to all boundary points calculated from the reference image, and divide the localized image into a plurality of regions. 9 shows an example of a localized photographed image divided into a plurality of regions.

The warping unit 50 may set a warping point for each area of the localized image so as to warp and project the localized image captured by the oral scanner onto the virtual object. The warping point may be one of a plurality of feature points generated by the feature point extracting unit 20 and a plurality of boundary points calculated by the boundary point calculating unit 40. 7 is a diagram illustrating an example of warping points set within a divided area of a localized image. In the illustrated example, the number of warping points is set to 4, but the number of warping points is not limited. As the number of warping points to be set increases, the warping of the local photographed image becomes more accurate, while the amount of computation required increases, so that the image processing speed may decrease. In addition, as the number of set warping points decreases, the image processing speed increases, but there is a disadvantage in that the accuracy of warping may decrease compared to the former. Of course, the administrator can arbitrarily adjust the number of warping points selected according to the use environment.

The warping unit 50 may represent the selected warping points as a warping matrix, and may warp the localized image by calculating a transformation matrix. In this case, the warping unit 50 may calculate a transformation matrix for each region of the localized image. Referring to the above example, since the local photographed image is divided into three regions, the warping unit 50 may calculate the transformation matrix three times. Accordingly, each divided region in the localized photographed image may be warped to a different degree. As a method of calculating the warping matrix, a transformation matrix of the warping matrix may be calculated using any one of known transformation techniques such as affine transformation or projection transformation. In particular, in the case of using projection transformation, the warping unit 50 may set four warping points for each area, and convert a localized image to a trapezoidal shape based on the set four warping points.

Thereafter, if it is confirmed that the warping unit 50 has calculated the warping matrix in all regions of the divided localized image, the oral data generation unit 230 will use the result to project the localized image on the virtual object. I can. Therefore, if the oral data providing system 1 according to the present embodiment is used, even if the angle of photographing of the oral scanner is moved, the local photographed image can be warped in real time and projected onto a virtual object.

On the other hand, when it is determined that all warping points selected by the localized image warping unit 50 are boundary points, the boundary point calculating unit 224 may calculate a new boundary point of the surveillance image through the process using Equation 1 described above. The boundary point calculator 224 may divide the previously divided region again by using the calculated new boundary point.

A diagram illustrating an example of resetting a divided area using a newly calculated boundary point other than the initially calculated boundary point. As shown, two points on the red solid line represent newly calculated boundary points. In addition, the divided area may be reset by a red solid line, which is a line segment to which newly calculated boundary points are connected.

In this way, the image matching unit 220 minimizes distortion of oral data by warping the other localized images based on any one localized image (reference image) even if the localized images are generated from different angles. can do.

The display unit 240 displays the oral data generated by the oral data generating unit 230 as image information, and displays a local photographed image received in real time from the oral scanner in a predetermined area, while It characterized in that the area corresponding to the localized photographed image is displayed as a region of interest (ROI).

9, a schematic flow of a method of providing oral data according to an embodiment of the present invention is shown.

First, a reference image for an oral region to be monitored may be generated and projected onto a virtual object in a 3D virtual space (110). To this end, a virtual object may be implemented by modeling an actual surveillance area in a 3D virtual space, and a reference image, which is an initial image of the real surveillance area, may be received. In addition, the reference image may be projected onto the virtual object, and the reference image projected onto the virtual object and 3D coordinate information of the virtual object may be stored together.

Next, the reference image projected on the virtual object and the localized image may be matched (120). Specifically, feature points may be extracted from each of the reference image and the localized image, and matched with the feature points of the localized image corresponding to the specific feature point of the reference image. Matched feature points may be calculated according to similarity.

The reference image projected on the virtual object may be divided into a plurality of regions according to depth information, and representative feature points may be extracted for each region (130). By comparing the matching values of each of the plurality of feature points distributed in each divided region of the reference image, the representative feature points may be extracted as many as a preset number in the order of matching values.

A boundary point at a boundary between different regions of the reference image may be calculated based on the extracted representative feature points for each region (140). That is, the reference image projected on the virtual object is divided into a plurality of regions according to depth information using depth information of 3D coordinates, and on the boundary line between different regions based on the representative feature points distributed in each divided region. It is possible to calculate the position of the located boundary point.

A boundary point of the localized photographed image may be calculated based on the representative feature points and boundary points of each region of the reference image, and the local photographed image may be divided into a plurality of regions based on the calculated boundary point of the local photographed image (150). The boundary point of the localized image may be determined according to a distance ratio between the representative feature point and the boundary point of the reference image.

A warping point may be set for each area of the localized image, and a transformation matrix of the warping point may be calculated to warp the localized image for each area (160). That is, the localized image can be warped for each area, and if all the warping points set in the process of setting the warping point are boundary points, a process of calculating a new boundary point in addition to the boundary point of the previously calculated local image is further performed, and a new boundary point The divided area of the locally photographed image can be reset by using. Thereafter, a process of setting a warping point for each region of the localized image in which the divided region has been reset may be performed again.

Finally, the localized photographed image warped for each region may be projected onto the virtual object in real time (170). That is, the reference image initially projected on the virtual object may be updated as a surveillance image warped for each region, and the localized photographed image warped on the virtual object may be projected in real time by repeating the above-described steps.

Such a technique for providing an oral data processing method may be implemented as an application or implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.

The program instructions recorded in the computer-readable recording medium may be specially designed and constructed for the present invention, and may be known and usable to those skilled in the computer software field.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magnetic-optical media such as floptical disks. media), and a hardware device specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like.

Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

Although the above has been described with reference to embodiments, those skilled in the art will understand that various modifications and changes can be made to the present invention without departing from the spirit and scope of the present invention described in the following claims. I will be able to.

1: oral data provision system
100: oral scanner
200: oral data processing device

Claims (2)

An oral scanner that scans the inside of the oral cavity locally to generate a localized image; And
In the oral data providing system comprising an oral data processing device for generating oral data for the entire internal structure of the oral cavity based on the local image received from the oral scanner,
The oral data processing device,
A photographed image collection unit sequentially collecting the local photographed images received from the oral scanner;
An image matching unit for matching a plurality of localized images captured at different angles with respect to any one localized image;
An oral data generation unit for generating oral data representing the entire oral cavity structure based on the localized image matched by the image matching unit; And
Includes a display unit for displaying the oral data as image information,
The image matching unit,
A reference image setting unit for setting a reference image among a plurality of localized images;
A representative point extracting unit for extracting a plurality of feature points from the reference image and a local photographed image subsequently received after the reference image, and matching feature points of the local photographed image corresponding to the feature points of the reference image;
A region dividing unit for dividing the reference image into a plurality of regions according to depth information, and extracting representative feature points for each region from a plurality of feature points distributed in each of the divided regions;
A boundary point at a boundary between different regions is calculated based on the representative feature points for each region, and a boundary point of the local photographed image is calculated according to a distance ratio between the representative feature points for each region of the reference image and the boundary, and the calculated local A boundary point calculator for dividing the local photographed image into a plurality of regions based on boundary points of the photographed image; And
A warping unit for setting a warping point for each area of the localized image, calculating a transformation matrix of the warping point to warp the localized image for each area, and projecting the localized image for each area on a virtual object in real time Included
The reference image setting unit,
Locally photographed images taken while maintaining the preset reference 3-axis posture value at predetermined time intervals are set as the reference image,
The image matching unit,
It characterized in that the local photographed image collected before the subsequent reference image is set is matched with the reference image,
The oral data generation unit,
The reference image is modeled in a 3D virtual space to generate a virtual object, and the 3D coordinates of the virtual object are mapped to the reference image and stored,
The area division unit,
The depth information of the reference image is analyzed based on the three-dimensional coordinates, and the reference image is divided by regions having the same depth section according to a preset depth section, and a region from a plurality of feature points distributed in each divided region It is characterized in that a representative feature point of a star is extracted, and a boundary point at a boundary between different regions is calculated based on the representative feature point of each region,
The display unit,
The oral data generated by the oral data generator is displayed as image information, and a localized image received in real time from the oral scanner is displayed in a predetermined area, and an area corresponding to the localized image in the oral data is a region of interest. Characterized in that the display as, oral data providing system.
delete

Images