CN114664454A - Simulation method of jaw face soft tissue three-dimensional model in orthodontic process - Google Patents

Abstract

The application discloses a method for simulating a jaw face soft tissue three-dimensional model in an orthodontic process, which comprises the following steps: step 1: acquiring a jaw face soft tissue three-dimensional model at the moment of T1 in the orthodontic process to form a three-dimensional reference model; step 2: the soft tissue of the maxillofacial region can be changed in the orthodontic treatment process. The method comprises the steps of obtaining a two-dimensional image of jaw face soft tissue at the moment of T2 in an orthodontic process, and obtaining at least one updated image according to an actual jaw face soft tissue scene of a user; and 3, step 3: simulating a jaw and facial soft tissue scene corresponding to the time T2 based on the jaw and facial soft tissue three-dimensional reference model at the time T1 and the two-dimensional image obtained at the time T2 in the step 2, and then forming a three-dimensional simulation model according to the simulation calculation result; and 4, step 4: the display device displays the three-dimensional model of the jaw and facial soft tissue at the time T2 calculated in the step 3, and displays the analysis result of the orthodontic effect, such as three-dimensional morphological change of the jaw and facial soft tissue at the time T2 relative to the time T1. The method can obtain the three-dimensional model of the soft tissue of the maxillofacial region simply, quickly and accurately at low cost; the compliance of the patient to the treatment measures can be effectively monitored, and the orthodontic treatment effect is improved.

Description

Simulation method of jaw face soft tissue three-dimensional model in orthodontic process

Technical Field

The application relates to a method for simulating a jaw face soft tissue three-dimensional model in an orthodontic process.

Background

In orthodontic treatment, patients do not have a desire to correct their teeth but rather to correct their teeth to alter the aesthetic appearance and morphology of their face. The beauty book is the natural requirement of people and the true picture of the heart of a patient. Most orthodontic treatments therefore require, in addition to simple tooth alignment, a more general movement of the dentition, so as to allow a certain reconstruction of the soft tissues attached thereto, and a better change in the facial appearance. For example, the focus on whether orthodontic treatment should be done to extract teeth also focuses on the aesthetic impact of these two means of correction rather than merely achieving the desired bite, i.e., the difference in the impact of extraction and non-extraction on facial morphology. One view is that the face of a patient is fuller after orthodontic extraction, and flattens after orthodontic extraction.

Based on the accumulation, at the beginning of the 21 st century, William r.profsit formally proposes an orthodontic concept of 'guiding soft tissue', emphasizes the improvement of the outline of the soft tissue, and takes the change of the soft tissue as a primary consideration for diagnosis and making a correction target, thereby achieving the maximum improvement of stable occlusion and facial form. The proposal of the concept mainly based on soft tissue is immediately widely accepted and valued by orthodontists and patients, and has profound influence on the orthodontics. From this point on, "look at the face" has become a popular vocabulary in the orthodontic field, and has also brought doctors and patients into closer contact. The research focus of orthodontic correction also focuses on clinical examination of the appearance of soft tissues and research of the change rule of the soft tissues along with orthodontic correction, so that the change rule of the soft tissues along with orthodontic correction is expected to be obtained, better reference is provided for orthodontic correction plan making and clinical development, the requirements of patients can be met, and the aesthetic risk caused by an inappropriate orthodontic scheme in clinic by an orthodontic doctor is avoided.

Tooth movement in orthodontic treatment or jaw movement in orthodontic treatment can cause certain influence on facial soft tissues attached to the outside; and the existence of most of the appliances can influence the convexity of soft tissues. It is therefore important for the orthodontist to know how the appliances affect the thickness of the soft tissue, particularly the lip thickness. Up to now, the study on the influence of the appliance on the soft tissue morphology has been carried out by analyzing two-dimensional pictures, for example, two-dimensional lateral skull positioning pictures are used to evaluate the influence of orthodontic treatment on the facial morphology of a patient. However, with the development of three-dimensional imaging technology, digital image acquisition, superposition and analysis become possible, and for the change analysis measurement of three-dimensional soft tissue, only three-dimensional images need to be shot at different time points, and then specific software is used for superposition analysis, so that the change characteristics of the soft tissue can be obtained.

1) Two-dimensional image-based maxillofacial soft tissue condition analysis method

The application of the X-ray in orthodontics enables orthodontists to have more accurate cognition on craniofacial bones, plays a vital role in the formulation of an orthodontics plan, and leaves a stroke of heavy ink in the development history of orthodontics. However, as the concept of orthodontic treatment shifts from the viewpoint of hard tissues such as a dental jaw and a jaw to the viewpoint of soft tissues such as a maxillofacial region, the participation of patients in orthodontic treatment is enhanced and the demand for the treatment is increased. So that the traditional two-dimensional measuring, analyzing and recording methods such as X-ray measurement, digital photography and the like are difficult to meet the requirements of orthodontists and patients

2) Maxillofacial soft tissue condition analysis method based on three-dimensional image

In recent years, three-dimensional digital technology is continuously developed, and the establishment of a jaw and facial soft tissue three-dimensional model and the research of morphological analysis team on the facial growth, the diagnosis of facial deformity, the prediction of postoperative surface shape and the evaluation of curative effect have important significance. The current three-dimensional information acquisition method of the facial soft tissue comprises Moire moire, three-dimensional laser scanning, structured light technology, three-dimensional stereography, grating projection measurement, spiral CT, cone beam CT computer tomography, MRI and the like. The development of three-dimensional analysis software enables an orthodontist to compare and analyze the soft facial tissues of a patient before and after an orthodontics treatment in three dimensions. Meanwhile, orthodontic patients can be compared with normal people, so that the difference between the orthodontic patients and the facial soft tissues of the normal people can be obtained, and whether different soft tissues of the face are more prominent or more sunken with the change of the normal people along with time in the orthodontic process can be analyzed. The development of three-dimensional analysis software opens a new field for analyzing the influence of the environment on the shape of the face, so that the influence of environmental factors and genetic factors on the face can be distinguished. More and more research is being conducted to observe changes before and after facial treatment using three-dimensional means. These studies include the changes in three-dimensional soft tissues of the face after cleft lip and palate, as well as the three-dimensional effects of different functional appliances on the face of the patient.

In the process of orthodontic treatment of a patient at present, tooth movement of orthodontic treatment or jaw movement of the orthodontic treatment can cause certain influence on external facial soft tissues, so that an orthodontist needs to compare and analyze the conditions of the facial soft tissues of the patient before and after the orthodontic treatment quickly and in real time, and further analyze whether different soft tissue changes on the face are more prominent or sunken in the orthodontic process. In particular, orthodontists compare actual and ideal maxillofacial soft tissue conditions. In particular, the actual position or shape or size of different parts of the soft maxillofacial tissue (infraorbital region, zygomatic region, paranasal region, upper buccal region, infrazygomatic region, upper angular region, lower buccal region, mandibular angular region and temporal region of bilateral soft tissue) is compared with the theoretical position or shape or size. Therefore, there is a need for an analysis method that can quickly, conveniently and inexpensively allow an orthodontist to perform simulation and condition analysis on a three-dimensional model of the patient's jaw and facial soft tissue, thereby effectively monitoring compliance of the patient with treatment measures and improving the treatment effect, and the current analysis methods of the jaw and facial soft tissue condition need to be implemented by means of professional equipment and software. It is an object of the present invention to at least partially address these needs.

Disclosure of Invention

The invention aims to provide a simulation method of a jaw face soft tissue three-dimensional model in an orthodontic process, so as to overcome the defects in the prior art.

In order to achieve the purpose, the invention provides the following technical scheme: a simulation method of a jaw face soft tissue three-dimensional model in an orthodontic process comprises the following steps:

step 1: acquiring a jaw face soft tissue three-dimensional model at the moment of T1 in the orthodontic process to form a three-dimensional reference model;

step 2: the change of the soft tissues of the jaw and the face can be brought in the orthodontic treatment process of the oral cavity. The method comprises the steps of obtaining a two-dimensional image of jaw face soft tissue at the moment of T2 in an orthodontic process, and obtaining at least one updated image according to an actual jaw face soft tissue scene of a user;

and step 3: simulating a jaw and facial soft tissue scene corresponding to the time T2 based on the jaw and facial soft tissue three-dimensional reference model at the time T1 and the two-dimensional image obtained at the time T2 in the step 2, and then forming a three-dimensional simulation model according to the simulation calculation result;

and 4, step 4: the display device displays the three-dimensional model of the jaw and facial soft tissue at the time T2 calculated in the step 3, and displays the analysis result of the orthodontic effect, such as three-dimensional morphological change of the jaw and facial soft tissue at the time T2 relative to the time T1.

Preferably, in step 1: the three-dimensional reference model is acquired by any one of a structured light technique, a laser scanning measurement technique, CBCT, MRI, and a stereographic measurement technique.

Preferably, in step 1, the three-dimensional reference model is a three-dimensional model obtained by combining a common model of soft tissues of the maxillofacial region with actual observation data.

Preferably, in step 2, the update image is a two-dimensional update image obtained by a CCD or CMOS sensor device.

Preferably, the step 3 further comprises the following steps:

semantic segmentation into several subdivided regions based on the three-dimensional reference model at time T1. The segmentation method can be a common algorithm for three-dimensional point cloud segmentation, such as an edge-based segmentation method, a region growth-based segmentation method, a model fitting-based segmentation method, a clustering-based segmentation method, a deep learning-based segmentation method, and the like (e.g., pointet);

semantic segmentation into several subdivided regions based on the two-dimensional updated image acquired at time T2. The segmentation method can be a commonly used segmentation method in a two-dimensional image, such as a threshold method, a segmentation method based on region growing, a segmentation method based on morphology, a segmentation method based on feature or model fitting, a segmentation method based on clustering, a segmentation method based on deep learning, and the like;

-searching for a perspective image of the three-dimensional reference model at time T1 having the greatest degree of matching with the updated image at time T2, based on the three-dimensional reference model at time T1;

sequentially matching the view images with the sub-divided regions of the updated image at the time T2 according to the characteristic information, and modifying the three-dimensional reference model at the time T1 according to the matching result, thereby obtaining the three-dimensional simulation model at the time T2.

Further, the characteristic information includes any one or a combination of several of free contour information, color information, density information, distance information, brightness information, saturation information, and glare information.

Further, the subdivision region includes: infraorbital region, zygomatic arch region, paranasal alar region, upper cheek region, infrazygomatic arch region, upper corner of the mouth region, lower cheek region, mandibular corner region and temporal region of the soft tissues on both sides.

Preferably, in the step 4: the analysis results include at least color, texture, shape, size, and location.

Preferably, in the step 4: the analysis result comprises a three-dimensional contrast model formed by overlaying the three-dimensional simulation model at the T2 moment on the three-dimensional reference model at the T1 moment in a transparent mode.

Preferably, the analysis result comprises: the degree of difference between the three-dimensional reference model at time T1 and the three-dimensional simulation model at time T2.

Compared with the prior art, the simulation method of the jaw face soft tissue three-dimensional model in the orthodontic process at least has the following beneficial effects:

1. by the aid of the simulation method, the jaw and facial soft tissue three-dimensional model at the T2 moment in the orthodontic treatment process can be simply, quickly and accurately obtained based on the three-dimensional reference model at the T1 moment;

2. by the simulation method, a three-dimensional simulation model at the time of T2 can be generated by a low-cost two-dimensional image acquisition device without acquiring a three-dimensional model of the soft tissue of the jaw part by adopting professional equipment for actual sampling as at the time of T1. The practical cost is greatly reduced, and the realization possibility is provided for remote medical treatment;

3. through the simulation method, the change conditions of the jaw and facial soft tissue can be compared quickly, and the quick and convenient remote understanding of the local orthodontic doctor by the doctor is convenient, so that the jaw and facial soft tissue condition of the patient can be monitored and analyzed in real time.

4. Can effectively monitor the compliance of the patient to the treatment measures and improve the effect of orthodontic treatment.

5. The method of the patent is used for simulating the three-dimensional condition of the jaw and facial soft tissue in different time periods, and the previous three-dimensional image is only used as a reference model and is assumed to be at the time T1; obtaining an updated image of the soft tissue of the jaw face at a time T2; between the time T1 and the time T2, due to orthodontic treatment, the three-dimensional situation corresponding to the time T2 and the three-dimensional situation corresponding to the time T1 are changed greatly, and a three-dimensional model corresponding to the time T2 needs to be simulated by a two-dimensional image obtained at the time T2 and a three-dimensional image obtained at the time T1.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 is a flowchart of a simulation method of a three-dimensional model of soft tissues of a jaw face in an orthodontic process according to an embodiment of the invention;

FIG. 2 is a schematic diagram of a three-dimensional reference model at time T1 of a simulation method of a three-dimensional model of soft tissues of a jaw face in an orthodontic process according to an embodiment of the invention;

FIG. 3 is a perspective view of a three-dimensional reference model at time T1 of a method for simulating a three-dimensional model of soft jaw and facial tissue during orthodontic treatment according to an embodiment of the invention;

FIG. 4 is a graph showing characteristic information of a three-dimensional reference model diagram at time T1 of a method for simulating a three-dimensional model of soft tissues of a jaw and a face during orthodontic treatment according to an embodiment of the invention;

FIG. 5 is a two-dimensional updated image at time T2 of a method for simulating a three-dimensional model of soft jaw and facial tissue during orthodontic treatment according to an embodiment of the invention;

FIG. 6 is a characteristic information display diagram of a two-dimensional updated image at time T2 of a simulation method of a three-dimensional model of soft tissue of a jaw and a face during orthodontic treatment according to an embodiment of the invention;

FIG. 7 is a graph showing the analysis result of the simulation method of the three-dimensional model of the soft tissue of the jaw and face in the orthodontic process according to the embodiment of the invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be described in detail below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.

Referring to fig. 1-7, the method for simulating a three-dimensional model of soft tissues of a jaw face in an orthodontic process of the invention comprises the following steps:

i.) at a reference time T1, generating at least one three-dimensional model, i.e. a three-dimensional reference model, which models the patient's maxillofacial soft tissue;

ii.) at update time T2, acquiring at least one two-dimensional update image corresponding to the actual maxillofacial soft tissue scene observed by the operator;

iii.) simulating the actual jaw and facial soft tissue scene corresponding to the two-dimensional updated image at the time T2 by combining with the three-dimensional reference model at the time T1, and then determining the three-dimensional model of the jaw and facial soft tissue corresponding to the updated time T2 according to the simulation calculation result, namely the three-dimensional simulation model;

iv.) displaying the analysis result of the jaw-facial soft tissue corresponding to the updating time on a screen in a specific mode.

In step i.), at a reference time T1, the patient's maxillofacial soft tissue physical three-dimensional reference model is scanned, preferably by means of proprietary equipment, the scanning method not being limited to structured light techniques, laser scanning measurement techniques, CBCT, MRI, stereographic measurement techniques, etc.

Alternatively, the reference model may be generated as follows: or selecting a universal model of soft tissue of the maxillofacial region from a library of universal models, and then correcting the universal model according to data measured or observed for the patient to obtain a three-dimensional reference model.

The three-dimensional reference model at time T1 may be viewed from any angle, as shown in conjunction with fig. 2 and 3, the example of fig. 2 and 3 being the three-dimensional reference model viewed from different viewing perspectives, i.e. the views viewed corresponding to the different viewing perspectives.

The three-dimensional reference model at time T1 observed from the different viewing perspectives is intended to be compared with the updated image at time T2 to determine, for each two-dimensional updated image at time T2, the capture perspective of the actual maxillofacial soft tissue scene with the greatest degree of match thereto. Based on the searched shooting angle and the two-dimensional updating image, the three-dimensional reference model is modified, and then a three-dimensional simulation model is obtained, which provides information about the jaw and facial soft tissue condition specific to the updating time.

The reference time T1 may be close to the update time T2 and may be set to be any period of time long, such as less than 0.5 seconds or 1 day apart from the update time (after or before the update time). The reference time may be separated from the update time by a longer time, e.g. by more than 1 month, more than 3 months or 6 months.

The reference model may be significantly different from the actual situation at the moment of updating, and the method disclosed in the present invention can obtain a three-dimensional simulated model, and through the comparative analysis of the information contained in the model on the three-dimensional reference model, the analysis result reflects the change of the jaw and facial soft tissue conditions between the reference moment and the moment of updating, i.e. they can provide information about the orthodontic treatment effect.

In step ii), at update time T2, an update image is acquired with the camera presenting the actual maxillofacial soft tissue scene. Fig. 5 is an updated image of a patient taken by the imaging device, specifically represented as two-dimensional image data.

The device according to the invention, in which the camera is incorporated in a mobile phone, a laptop, a tablet or a video camera, makes it possible to obtain an updated image that accurately represents the actual maxillofacial soft tissue scene observed by the operator on the screen of the mobile phone, laptop, tablet or video camera, respectively.

In particular, using a virtual reality helmet, the camera may be firmly fixed on the screen or independent of the screen. The independence of the camera and the screen advantageously allows the operator to operate without directly seeing the actual scene, that is, by seeing the actual scene only on the screen.

Preferably, the update image is in color, preferably true color.

In step iii), a three-dimensional simulation model is generated using the following sub-steps:

a.) based on the three-dimensional reference model at time T1 (FIG. 2), semantic segmentation is performed to divide the three-dimensional reference model into a plurality of subdivided regions. The segmentation method can be a common algorithm for three-dimensional point cloud segmentation, such as an edge-based segmentation method, a region growth-based segmentation method, a model fitting-based segmentation method, a clustering-based segmentation method, a deep learning-based segmentation method, and the like (e.g., pointet);

b.) based on the two-dimensional updated image (fig. 5) acquired at time T2, semantic segmentation is performed to divide the image into a plurality of subdivided regions. The segmentation method can be a common segmentation method in a two-dimensional image, such as a threshold method, a segmentation method based on region growing, a segmentation method based on model fitting, a segmentation method based on clustering, a segmentation method based on deep learning and the like;

c.) the shooting perspective of the actual maxillofacial soft tissue scene with the maximum matching degree with the two-dimensional updated image (fig. 5) obtained at the time T2 can be searched by combining the three-dimensional reference model (fig. 2) at the time T1.

c-1: selecting a specific shooting visual angle;

c-2: adjusting the three-dimensional reference model (figure 2) to a specific shooting visual angle and acquiring a corresponding view;

c-3. extracting the same feature information in the corresponding view and the two-dimensional updated image (fig. 5). Preferably, the identification information is selected from the group consisting of contour information, color information, density information, distance information, brightness information, saturation information, information about glare, and combinations thereof. In this example, edge contour line feature information is extracted, and the extracted feature information is superimposed on the three-dimensional reference model and the two-dimensional updated image by using a white line, as shown in fig. 4 and 6;

c-4, matching the two-dimensional updated image at the time of T2 with the characteristic information of each subdivision region in the three-dimensional reference model at the time of T1 (the three-dimensional reference model is presented under a specific shooting visual angle);

and c-5, if the matching degree is smaller than a preset threshold value, modifying the shooting angle of view, and returning to the c-2 substep. If the matching degree is greater than the preset threshold value, it indicates that the matching degree of the view under the shooting angle and the feature information of the two-dimensional updated image is very good, and the current shooting angle is the searched shooting angle (fig. 3).

d.) modifying the three-dimensional reference model at the time of T1 based on the searched shooting visual angle and the two-dimensional updating image at the time of T2, and further obtaining the three-dimensional simulation model at the time of T2.

d-1, adjusting the three-dimensional reference model (figure 4) to a specific shooting visual angle and acquiring a corresponding view;

d-2, comparing the actual position or shape or size of each subdivided region (such as different parts of jaw and facial soft tissues, such as infraorbital region, zygomatic arch region, paranasal region, upper cheek region, infrazygomatic arch region, upper corner region, inferior cheek region, mandibular corner region and temporal region of two-dimensional updated image (figure 6) at the time T2 with the theoretical position or shape or size according to the characteristic information by sequentially matching the obtained view with the actual position or shape or size of each subdivided region;

d-3: and modifying the three-dimensional reference model (figure 2) at the T1 moment according to the matching result, and further obtaining a three-dimensional simulation model at the T2 moment. Preferably, the movement and deformation of the soft tissue model of the jaw face is determined or refined by means of the matching result of the one or more updated images with the acquired view. Preferably, movements and/or deformations of the three-dimensional reference model corresponding to the infraorbital, zygomatic arch, paranasal area, upper cheek area, infrazygomatic area, upper paraoral area, lower cheek area, mandibular area and temporal area of the bilateral soft tissue may be included. The generated three-dimensional simulation model may simulate information of visible or invisible portions of the patient's maxillofacial soft tissue, the information including at least one of color, texture, shape, size, and position.

In step iii), all known methods for producing the maximum degree of matching can be envisaged. The maximum degree of matching can be obtained in particular due to the optimization to minimize said distance ("best fit"). Preferably, the search is performed by means of a meta-heuristic method, preferably an evolutionary methodology, preferably by a simulated annealing method.

The skilled person knows how to process the image to extract feature information of the image and thus create a corresponding map. For example, fig. 2 and 3 are views in which the three-dimensional reference model is located at different angles of view, fig. 4 is a feature information map relating to the contour of the maxillofacial region obtained from the view of fig. 3, and fig. 6 is a feature information map relating to the contour of the maxillofacial region obtained from the updated image of fig. 5.

In step iv), the analysis result of the maxillofacial soft tissue corresponding to the update time is displayed on the screen in a specific manner. The display of the analysis result of the jaw and facial soft tissue corresponding to the update time can be equipment with a display function, and is not limited to a mobile phone, a notebook computer, a virtual reality helmet or glasses, a tablet computer and a video camera.

1) The three-dimensional simulation model corresponding to the update time can be directly projected onto the three-dimensional reference model obtained at the reference time. The operator can thus observe the virtual maxillofacial soft tissue three-dimensional model and its effect superimposed on the three-dimensional reference model acquired at the reference instant through the screen.

For example, the difference between the three-dimensional reference model obtained at the reference time and the three-dimensional simulation model corresponding to the update time may be calculated by comparing the three-dimensional simulation model with the three-dimensional reference model, and based on the difference between the difference, special color blocks with different colors may be projected onto the infraorbital region, the zygomatic bone region, the zygomatic arch region, the paranasal alare region, the upper cheek region, the infrazygomatic arch region, the paraoral horn region, the lower cheek region, the mandibular horn region, and the temporal region of the two soft tissues to represent the movement and/or deformation of the corresponding parts. The analysis results may be displayed on a screen.

2) When the screen is opaque, particularly in embodiments where the screen is that of a cell phone, laptop, virtual reality helmet or tablet computer, the operator can thus view both the updated image and the three-dimensional simulation model on the screen at the same time.

The virtual maxillofacial soft tissue scene may be sufficiently transparent to reveal an updated image of the actual maxillofacial soft tissue scene or below.

Alternatively, the virtual maxillofacial soft tissue scene may be opaque so as not to reveal the actual maxillofacial soft tissue scene or an updated image extending behind or below it. The opacity of the virtual jaw-facial soft tissue scene is particularly advantageous for simulating past or future jaw-facial soft tissue conditions, such as viewing changes in the jaw-facial soft tissue under the influence of orthodontic treatment, or viewing changes in the jaw-facial soft tissue without orthodontic treatment, for example. When the virtual maxillofacial soft tissue scene is rendered on the updated image, it may be added on top of the updated image, or in an equivalent manner, the updated image may be replaced with an image that is reprocessed by the computer to reveal the virtual maxillofacial soft tissue scene.

3) Preferably, in order to present the virtual maxillofacial soft tissue scene in registration on the representation of the actual maxillofacial soft tissue scene or on an updated image, the respective feature marker points are extracted on the updated image, the position of which relative to the virtual maxillofacial soft tissue scene is known. For example, a characteristic shape of a parastomal upper or parastomal lower region represented on both the updated image and the virtual maxillofacial soft tissue scene is identified. In particular, the reference image from which the virtual maxillofacial soft tissue scene is determined may be used to identify the location of the signature in the virtual scene. In step iv), a representation of these signatures is then overlaid on the updated image and in the virtual maxillofacial soft tissue scene, as shown in particular in fig. 7. This allows for accurate positioning of the virtual maxillofacial soft tissue scene relative to the feature labels represented on the updated image.

4) Preferably, by comparing the three-dimensional simulation model corresponding to the update time with the three-dimensional reference model obtained at the reference time, elements of the actual jaw-facial soft tissue scene corresponding to the update time that do not meet the evaluation criteria can be further identified, and the elements are highlighted in the three-dimensional simulation model and/or a message related to the elements is added to the three-dimensional simulation model, preferably the message is an indication to be followed by the patient. The evaluation criterion relates to the health status of the patient and/or the sex and age of the patient and/or the orthodontic treatment measures received by the patient and/or the positioning of the orthodontic appliance worn by the patient and/or the condition of the orthodontic appliance worn by the patient.

Preferably, the process then returns to step ii) preferably less than 30 seconds, less than 10 seconds, less than 5 seconds, preferably less than 6 seconds, preferably less than 3 seconds, less than 0.5 seconds, less than 0.2 seconds, preferably less than 0.1 seconds after step iv). Preferably, the cycles of steps ii) to iv) are carried out in real time without interruption.

Alternatively, in one embodiment, the method is completed at the end of step iv) without returning to step ii). Thus, the method is performed "on-demand" rather than in real-time.

Example 1

In one embodiment, the method is used for analyzing an adult patient, and the method is used for checking the variation trend of the soft tissues of the jaw face of the tooth extraction patient. Firstly, an oral professional acquires a three-dimensional model of the soft tissue of the jaw face of a patient before tooth extraction as a reference model; at any time after tooth extraction, such as 1 month or 3 months, which is concerned by oral professionals, informing a patient to take a two-dimensional maxillofacial picture through a self-owned mobile phone or other related equipment, namely updating the image; based on the three-dimensional reference model and the updated image, a virtual maxillofacial soft tissue scene can be generated through software, and the scene simulates an actual maxillofacial soft tissue scene corresponding to the acquisition time of the updated image. The virtual scene is presented to oral professionals in a virtual jaw and facial soft tissue three-dimensional model mode and is used for analyzing the jaw and facial soft tissue condition change of the patient with tooth extraction, so that better reference is provided for orthodontic treatment plan making and clinical development, meanwhile, the virtual scene can be better communicated with the patient, the clinical aesthetic risk is avoided, and possible doctor-patient disputes are prevented in advance.

Example 2

In one embodiment, the method is used to analyze an adult patient for modeling trends in soft tissue changes in the patient's jaw face after a dental extraction procedure is not performed. Firstly, an oral professional acquires a three-dimensional model of the current soft tissue of the maxillofacial region of a patient as a reference model; modifying the currently shot two-dimensional maxillofacial picture according to a clinical rule at any time after the current time concerned by oral professionals, such as after 1 month or after 3 months, and generating an updated image; based on the three-dimensional reference model and the updated image, a virtual maxillofacial soft tissue scene can be generated through software, and the scene simulates the maxillofacial soft tissue scene corresponding to a certain specific time after the current time. The virtual scene is presented to oral professionals in a virtual jaw and facial soft tissue three-dimensional model mode and used for simulating the jaw and facial soft tissue condition change of the tooth extraction patient, so that better reference is provided for orthodontic correction plan making and clinical development, and meanwhile, the virtual scene can be better communicated with the patient.

Example 3

In one embodiment, the method is used to obtain better acceptance of treatment by a patient. In particular, the method can be used to closely simulate the effects of orthodontic treatment on the soft jaw-facial tissues. Preferably, the image capturing device is a mobile phone, a notebook computer, a tablet computer

By overlapping with the updated images, which follow one another on the screen of a phone, laptop or tablet computer, the patient sees corresponding virtual jaw face soft tissue scenes, which represent different moments of orthodontic treatment. For example, the scene may be a time marking the beginning of the treatment, or a time 1 to 3 months after the treatment, and the virtual maxillofacial soft tissue scene may represent changes in the maxillofacial region that the patient has brought after the current orthodontic treatment. Then, the patient can easily change the viewing direction by turning his or her head or moving a mobile phone, laptop or tablet, thereby adjusting the virtual maxillofacial soft tissue scene in real time.

Preferably, the patient can modify the simulation moment by interacting with the screen, for example by moving a virtual cursor. The virtual jaw face soft tissue scene is adjusted accordingly, especially considering the treatment effect. By acting on the cursor, the patient can then simulate the progress of the treatment. Thus, the movement of the cursor allows him or her to simulate an acceleration of time.

Of course, the invention is not limited to the embodiments described and shown.

The present invention is not limited to a method comprising steps i) to iv). In particular, the search for the reference image is not essential to the implementation of the present invention. In particular, a plurality of reference images may be obtained based on a plurality of corresponding update images acquired simultaneously or at different (e.g., consecutive) update times to determine a virtual maxillofacial soft tissue scene.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

The foregoing is merely a detailed description of the present application, and it should be noted that modifications and embellishments could be made by those skilled in the art without departing from the principle of the present application, and these should also be considered as the protection scope of the present application.

Claims (10)

1. A simulation method of a jaw face soft tissue three-dimensional model in an orthodontic process is characterized by comprising the following steps: the method comprises the following steps:

step 1: acquiring a jaw face soft tissue three-dimensional model at the moment of T1 in the orthodontic process to form a three-dimensional reference model;

step 2: acquiring a two-dimensional image of jaw face soft tissue at the moment of T2 in the orthodontic process, and acquiring at least one updated image according to the actual jaw face soft tissue scene of a user;

and step 3: simulating a jaw and facial soft tissue scene corresponding to the time T2 based on the jaw and facial soft tissue three-dimensional reference model at the time T1 and the two-dimensional image obtained at the time T2 in the step 2, and then forming a three-dimensional simulation model according to the simulation calculation result;

and 4, step 4: and the display device displays the jaw face soft tissue three-dimensional model at the time T2 calculated in the step 3 and displays the analysis result of the orthodontic effect.

2. The method for simulating the three-dimensional model of the soft tissues of the jaw face in the orthodontic process according to claim 1, wherein the method comprises the following steps: in the step 1: the three-dimensional reference model is acquired by any one of a structured light technique, a laser scanning measurement technique, CBCT, MRI, and a stereographic measurement technique.

3. The method for simulating the three-dimensional model of the soft tissues of the jaw and face in the orthodontic process according to claim 1, wherein the method comprises the following steps: in the step 1, the three-dimensional reference model is a three-dimensional model obtained by combining a jaw face soft tissue general model with actual observation data correction.

4. The method for simulating the three-dimensional model of the soft tissues of the jaw and face in the orthodontic process according to claim 1, wherein the method comprises the following steps: in step 2, the update image is a two-dimensional update image obtained by a CCD or CMOS sensor device.

5. The method for simulating the three-dimensional model of the soft tissues of the jaw and face in the orthodontic process according to claim 1, wherein the method comprises the following steps: the step 3 comprises the following steps:

performing semantic segmentation based on the three-dimensional reference model at the time T1 to divide the three-dimensional reference model into a plurality of subdivided regions; the segmentation method can be a common algorithm of three-dimensional point cloud segmentation, and comprises any one of a segmentation method based on edges, a segmentation method based on region growing, a segmentation method based on model fitting, a segmentation method based on clustering and a segmentation method based on deep learning;

semantic segmentation into several subdivided regions based on the two-dimensional updated image acquired at time T2. The segmentation method can be any one of common segmentation methods in two-dimensional images, such as a threshold method, a segmentation method based on region growing, a segmentation method based on morphology, a segmentation method based on feature or model fitting, a segmentation method based on clustering and a segmentation method based on deep learning;

-searching for a perspective image of the three-dimensional reference model at time T1 having the greatest degree of matching with the updated image at time T2, based on the three-dimensional reference model at time T1;

sequentially matching the view angle image with each subdivided region of the updated image at the time T2 according to the characteristic information, and modifying the three-dimensional reference model at the time T1 according to the matching result, thereby obtaining the three-dimensional simulation model at the time T2.

6. The method for simulating the three-dimensional model of the soft tissues of the jaw face in the orthodontic process according to claim 5, wherein the method comprises the following steps: the characteristic information comprises any one or combination of more of free contour information, color information, density information, distance information, brightness information, saturation information and glare information.

7. The method for simulating the three-dimensional model of the soft tissues of the jaw face in the orthodontic process according to claim 5, wherein the method comprises the following steps: the subdivided regions include: infraorbital region, zygomatic arch region, paranasal alar region, upper cheek region, infrazygomatic arch region, upper corner of the mouth region, lower cheek region, mandibular corner region and temporal region of the soft tissues on both sides.

8. The method for simulating the three-dimensional model of the soft tissues of the jaw and face in the orthodontic process according to claim 1, wherein the method comprises the following steps: in the step 4: the analysis results include at least color, texture, shape, size, and location.

9. The method for simulating the three-dimensional model of the soft tissues of the jaw and face in the orthodontic process according to claim 1, wherein the method comprises the following steps: in the step 4: the analysis result comprises a three-dimensional contrast model formed by overlaying the three-dimensional simulation model at the time T2 on the three-dimensional reference model at the time T1 in a transparent mode.

10. The method for simulating the three-dimensional model of the soft tissues of the jaw and face in the orthodontic process according to claim 1, wherein the method comprises the following steps: the analysis result comprises: the degree of difference between the three-dimensional reference model at time T1 and the three-dimensional simulation model at time T2.

Images