US11915378B2 - Method and system for proposing and visualizing dental treatments - Google Patents

Method and system for proposing and visualizing dental treatments Download PDF

Info

Publication number
US11915378B2
US11915378B2 US17/310,278 US202017310278A US11915378B2 US 11915378 B2 US11915378 B2 US 11915378B2 US 202017310278 A US202017310278 A US 202017310278A US 11915378 B2 US11915378 B2 US 11915378B2
Authority
US
United States
Prior art keywords
dimensional
orthodontic treatment
data record
actual teeth
corrected
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US17/310,278
Other languages
English (en)
Other versions
US20220139044A1 (en
Inventor
André Koza
Ronny KUCHARCZYK
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Dentsply Sirona Inc
Original Assignee
Dentsply Sirona Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Dentsply Sirona Inc filed Critical Dentsply Sirona Inc
Publication of US20220139044A1 publication Critical patent/US20220139044A1/en
Assigned to DENTSPLY SIRONA INC. reassignment DENTSPLY SIRONA INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: Koza, André, Kucharczyk, Ronny
Application granted granted Critical
Publication of US11915378B2 publication Critical patent/US11915378B2/en
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61C—DENTISTRY; APPARATUS OR METHODS FOR ORAL OR DENTAL HYGIENE
    • A61C13/00—Dental prostheses; Making same
    • A61C13/0003—Making bridge-work, inlays, implants or the like
    • A61C13/0004—Computer-assisted sizing or machining of dental prostheses
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
    • A61B5/0082—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence adapted for particular medical purposes
    • A61B5/0088—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence adapted for particular medical purposes for oral or dental tissue
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61C—DENTISTRY; APPARATUS OR METHODS FOR ORAL OR DENTAL HYGIENE
    • A61C13/00—Dental prostheses; Making same
    • A61C13/0001—In-situ dentures; Trial or temporary dentures
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61C—DENTISTRY; APPARATUS OR METHODS FOR ORAL OR DENTAL HYGIENE
    • A61C19/00—Dental auxiliary appliances
    • A61C19/04—Measuring instruments specially adapted for dentistry
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61C—DENTISTRY; APPARATUS OR METHODS FOR ORAL OR DENTAL HYGIENE
    • A61C7/00—Orthodontics, i.e. obtaining or maintaining the desired position of teeth, e.g. by straightening, evening, regulating, separating, or by correcting malocclusions
    • A61C7/002—Orthodontic computer assisted systems
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61C—DENTISTRY; APPARATUS OR METHODS FOR ORAL OR DENTAL HYGIENE
    • A61C9/00—Impression cups, i.e. impression trays; Impression methods
    • A61C9/004—Means or methods for taking digitized impressions
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61C—DENTISTRY; APPARATUS OR METHODS FOR ORAL OR DENTAL HYGIENE
    • A61C9/00—Impression cups, i.e. impression trays; Impression methods
    • A61C9/004—Means or methods for taking digitized impressions
    • A61C9/0046—Data acquisition means or methods
    • A61C9/0053—Optical means or methods, e.g. scanning the teeth by a laser or light beam
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
    • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
    • G01J3/46—Measurement of colour; Colour measuring devices, e.g. colorimeters
    • G01J3/50—Measurement of colour; Colour measuring devices, e.g. colorimeters using electric radiation detectors
    • G01J3/508—Measurement of colour; Colour measuring devices, e.g. colorimeters using electric radiation detectors measuring the colour of teeth
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/0007—Image acquisition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H10/00—ICT specially adapted for the handling or processing of patient-related medical or healthcare data
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/50—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for simulation or modelling of medical disorders
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B18/00—Surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body
    • A61B18/18—Surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body by applying electromagnetic radiation, e.g. microwaves
    • A61B18/20—Surgical instruments, devices or methods for transferring non-mechanical forms of energy to or from the body by applying electromagnetic radiation, e.g. microwaves using laser
    • A61B2018/2035—Beam shaping or redirecting; Optical components therefor
    • A61B2018/20351—Scanning mechanisms
    • A61B2018/20353—Scanning in three dimensions [3D]
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B2027/0178—Eyeglass type
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/04—Indexing scheme for image data processing or generation, in general involving 3D image data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30036—Dental; Teeth
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face

Definitions

  • the present application generally relates to a method, a system and a computer readable storage media for orthodontic analysis and, more particularly, to a method, system and computer readable storage media for detecting a misalignment of teeth and proposing and visualizing possible orthodontic/dental treatments, in real time, utilizing augmented reality.
  • Dentists may usually detect a misalignment of teeth of a patient in a standard investigation/clinical check. Patients may then be transferred to an orthodontist for further treatment. In some cases, diagnosis of the condition of the teeth may take place late, for example, if the patient does not go to the dentist on a regular base. Moreover, manual diagnosis by an orthodontist may be time consuming. There may therefore be a need to quickly and/or inexpensively diagnose a misalignment of teeth, propose possible solutions and visualize the proposal.
  • U.S. Patent Application Publication No. US20180249912A1 discloses a preview of how the teeth may look using an image of the teeth that was obtained beforehand.
  • U.S. Pat. No. 9,675,305B2 discloses the use of photos to detect wrong positions of teeth without depth information.
  • US Patent Application No. 2017202633 discloses an imaging and display system for guiding medical interventions comprising a wearable display for viewing by a user wherein the display presents a composite, or combined image that includes pre-operative surgical navigation images, intraoperative images, and in-vivo microscopy images or sensing data.
  • a probe such as a microscopy probe or a sensing probe, may be used to acquire in-vivo imaging/sensing data from the patient and the intraoperative and in-vivo images may be acquired using tracking and registration techniques to align them with the pre-operative image and the patient to form a composite image for display.
  • US Patent Application No. 20020082498 discloses a method for image-guided surgery comprising capturing 3-dimensional (3D) volume data of a portion of a patient, processing the volume data so as to provide a graphical representation of the data, capturing a stereoscopic video view of a scene including a portion of said patient, rendering the graphical representation and the stereoscopic video view in a blended manner so as to provide a stereoscopic augmented image, and displaying said stereoscopic augmented image in a video-see-through display.
  • 3-dimensional (3D) volume data of a portion of a patient processing the volume data so as to provide a graphical representation of the data
  • capturing a stereoscopic video view of a scene including a portion of said patient rendering the graphical representation and the stereoscopic video view in a blended manner so as to provide a stereoscopic augmented image
  • displaying said stereoscopic augmented image in a video-see-through display.
  • US Patent Application Publication No. 20160191887 describes a real-time surgery navigation method and apparatus for displaying an augmented view of a patient from a static or dynamic viewpoint of a surgeon.
  • a surface image, a graphical representation of the internal anatomic structure of the patient processed from preoperative or intraoperative images, and a computer geometrically registering both images may be used.
  • Responsive to geometrically registering the images a head mounted display may present to a surgeon an augmented view of the patient.
  • Orthodontic treatments may hereinafter be used interchangeably with cosmetic treatments to cover various treatments of the tooth such as correction of malocclusions (including for example overjets, overbites, crossbites, etc.), as well as improving the smile of a patient including the use of dental crowns, dental veneers, dental bonding, inlays, onlays, implants and the like.
  • malocclusions including for example overjets, overbites, crossbites, etc.
  • improving the smile of a patient including the use of dental crowns, dental veneers, dental bonding, inlays, onlays, implants and the like.
  • the present invention provides a method for the visualization of an orthodontic/cosmetic treatments, the method comprising: recording a three-dimensional image of a face to obtain a three-dimensional facial data record; determining characteristic points within the three-dimensional facial data record; segmenting a mouth area based on the characteristic points; obtaining a three-dimensional representation of actual teeth to be corrected; determining an orthodontic treatment as a three-dimensional tooth data record based on the obtained three-dimensional representation of actual teeth; and overlaying the orthodontic treatment on a mouth area of the three-dimensional facial data record.
  • the method further comprises one or more of the following steps: (i) determining characteristic points within the three-dimensional facial data record, determining a direction of orientation of the face in the three-dimensional facial data record based on the characteristic points; and orienting said orthodontic treatment such that its direction matches the direction of orientation of the face in the three-dimensional facial data record, (ii) recording said three-dimensional image of the face with a mobile device, (iii) designing a corrective appliance based on the orthodontic treatment, (iv) wherein the three-dimensional representation of actual teeth is obtained through extraction from the three-dimensional image of the face or obtained from a scan, (v) establishing reference objects/reference frames and/or detecting anatomical dental features in the three-dimensional representation of actual teeth to be corrected; and determining misalignments of the three-dimensional representation of actual teeth to be corrected by comparing the reference objects/reference frames and/or anatomical dental features with reference data, (vi) wherein the reference data includes data obtained from ongoing patient treatments, past patient treatments, scientific literature
  • the present invention provides a method for the augmented visualization of an orthodontic treatment, the method comprising: recording a three-dimensional image of a face to obtain a three-dimensional facial data record; segmenting a mouth area based on the characteristic points; obtaining a three-dimensional representation of actual teeth to be corrected; determining an orthodontic treatment as a three-dimensional tooth data record based on the obtained three-dimensional representation of actual teeth; overlaying the orthodontic treatment as an augmentation on a treatment site through a display device for augmented visualization such that the orthodontic treatment appears directly superimposed on said treatment site.
  • the method further comprises one or more of the following steps: (i) wherein the display device for augmented visualization is an augmented reality glass or HUD display, (ii) further comprising designing a corrective appliance based on the orthodontic treatment, (iii) wherein the three-dimensional representation of actual teeth is obtained through extraction from the three-dimensional image of the face or obtained from a scan, (iv) further comprising: establishing reference objects/reference frames and/or detecting anatomical dental features in the three-dimensional representation of actual teeth to be corrected; and determining misalignments of the three-dimensional representation of actual teeth to be corrected by comparing the reference objects/reference frames and/or anatomical dental features with reference data, (v) wherein the reference data includes data obtained ongoing patient treatments, past patient treatments, scientific literature and textbooks, (vi) wherein the anatomical features include at least one of cusps, ridges, grooves and contact points, (vii) further comprising continuously updating the three-dimensional facial data record based on data from a tracking system
  • the present invention provides a system for visualizing an orthodontic treatment, the system comprising: a mobile device having a means for distance measurement, a means for motion tracking and/or a means for color measurement, and at least one processor configured to perform the steps of; recording a three-dimensional image of a face to obtain a three-dimensional facial data record; segmenting a mouth area based on the characteristic points; obtaining a three-dimensional representation of actual teeth to be corrected; determining an orthodontic treatment as a three-dimensional tooth data record based on the obtained three-dimensional representation of actual teeth; and overlaying the orthodontic treatment on a mouth area of the three-dimensional facial data record.
  • the system may further comprise one or more of the following configurations: (i) wherein the processor is further configured to perform the steps of: determining a direction of orientation of the face in the three-dimensional facial data record based on the characteristic points; and orienting said orthodontic treatment such that its direction matches the direction of orientation of the face in the three-dimensional facial data record, (ii) wherein the processor is further configured to perform the step of designing a corrective appliance based on the orthodontic treatment, (iii) wherein the three-dimensional representation of actual teeth is obtained through extraction from the three-dimensional image of the face or obtained from a scan (iv) wherein the processor is further configured to perform the steps of: establishing reference objects/reference frames and/or detecting anatomical dental features in the three-dimensional representation of actual teeth to be corrected; and determining misalignments of the three-dimensional representation of actual teeth to be corrected by comparing the reference objects/reference frames and/or anatomical dental features with reference data, (v) wherein the reference data includes data obtained from ongoing patient treatments, past patient treatments
  • the present invention may provide a system for visualizing an orthodontic treatment, the system comprising: a display device for augmented visualization, and at least one processor configured to perform the steps of; recording a three-dimensional image of a face to obtain a three-dimensional facial data record; determining characteristic points within the three-dimensional facial data record; segmenting a mouth area based on the characteristic points; obtaining a three-dimensional representation of actual teeth to be corrected; determining an orthodontic treatment as a three-dimensional tooth data record based on the obtained a three-dimensional representation of actual teeth; overlaying the orthodontic treatment as an augmentation on a treatment site through the display device for augmented visualization such that the orthodontic treatment appears directly superimposed on said treatment site.
  • the system may further comprise one or more of the following configurations: (i) wherein the display device for augmented visualization is an augmented reality glass or HUD (head-up display) display, (ii) wherein the processor is further configured to perform the steps of designing a corrective appliance based on the orthodontic treatment, (iii) wherein the three-dimensional representation of actual teeth is obtained through extraction from the three-dimensional image of the face or obtained from an intraoral scan, (iv) wherein the processor is further configured to perform the steps of: establishing reference objects/reference frames and/or detecting anatomical dental features in the three-dimensional representation of actual teeth to be corrected; and determining misalignments of the three-dimensional representation of actual teeth to be corrected by comparing the reference objects/reference frames and/or anatomical dental features with reference data, (v) wherein the reference data includes data obtained ongoing patient treatments, past patient treatments, scientific literature and textbooks, (vi) wherein the processor is further configured to perform the steps of continuously updating the three-dimensional facial data record based on data from a tracking
  • the present invention may provide a non-transitory computer-readable storage medium storing a program which, when executed by a computer system, causes the computer system to perform a procedure comprising: recording a three-dimensional image of a face to obtain a three-dimensional facial data record; determining characteristic points within the three-dimensional facial data record; segmenting a mouth area based on the characteristic points; obtaining a three-dimensional representation of actual teeth to be corrected; determining an orthodontic treatment as a three-dimensional tooth data record based on the obtained three-dimensional representation of actual teeth; and overlaying the orthodontic treatment on a mouth area of the three-dimensional facial data record.
  • FIG. 1 is a block diagram illustrating an orthodontic/cosmetic visualization system according to an embodiment of the present invention
  • FIG. 2 illustrates a mobile device according to an exemplary embodiment of the present invention
  • FIG. 3 illustrates a side view of a mobile device and a patient according to an exemplary embodiment of the present invention
  • FIG. 4 is a block diagram showing a computer system according to an exemplary embodiment of the present invention.
  • FIG. 5 is a flow chart showing a method according to an exemplary embodiment of the present invention.
  • FIG. 6 a is a front view of a three-dimensional data record of a patient's face
  • FIG. 6 b is a front view of a three-dimensional data record of a patient's face having characteristic points
  • FIG. 7 a is a front view of a three-dimensional tooth data record
  • FIG. 7 b is a front view of a three-dimensional data record of a patient's face having a proposed orthodontic/cosmetic treatment.
  • a method, system and computer readable storage media may be provided for detecting a misalignment of teeth and proposing possible orthodontic/cosmetic treatments.
  • FIG. 1 illustrates a visualization system 1 comprising a display device 12 for augmented visualization such as head mounted augmented reality glasses, an HUD display, or a stereoscopic display capable of receiving stereoscopic video images, or otherwise display device 12 that may be used for overlaying an orthodontic treatment 42 in an augmented manner on a target/treatment site 14 a or on a stereoscopic video of the target/treatment site 14 a such that the orthodontic treatment 42 appears to be superimposed on the target/treatment site 14 a when viewed by a clinician 10 .
  • the visualization system 1 may also include a camera system 3 , wherein the camera system 3 may preferably be a mobile device 54 such as a smart phone (as shown in FIG. 2 ) equipped with a color, depth and motion tracking sensors.
  • the mobile device 54 may have an RGB-IR camera 62 for color measurement, an IR projector 56 , for distance measurement using infra-red rays, and a fisheye camera 58 for motion tracking.
  • an orthodontic proposal/treatment may be displayed virtually on the display of the mobile device 54 , and preferably over a real-time 3D model of a patient's face for viewing.
  • the mobile device 54 may also have a flash 60 for lighting an object during of the real-time 3D model.
  • the mobile device 54 may also have other means for color measurement, distance measurement and motion tracking of a patient 14 as may be appreciated by a person of ordinary skill in the art.
  • a patient 14 may use a display of the mobile device 54 to view an orthodontic treatment 42 overlaid on a video or 3D model of him/herself.
  • the camera system 3 may be a 3D optical tracking system and/or stereoscopic camera system and may be included in the computer system and/or may form or be a part of the tracking system 2 .
  • the camera system 3 may also be embedded in the display device 12 of the clinician 10 .
  • the camera system may operate under one of several depth sensing principles including, for example, (i) structural light, (ii) Time of Flight (ToF) and/or (iii) stereoscopic principles explained hereinafter.
  • a light source may be used to project a known pattern onto the patient 14 , and a receiver may detect the distortion of the reflected pattern to calculate depth map based on geometry.
  • a light source may send out a pulse, and a sensor may detect a reflection of the pulse from the patient 14 in order to record it's time of flight. Knowing that and the constant speed of light, the system may calculate how far away the patient 14 is.
  • a modulated light source may be sent and a phase change of light reflected from the patient may be detected.
  • multiple cameras may be placed at different positions to capture multiple images of the patient, and a depth map may be calculated based on geometry. This depth information may be used to track the patient's location during treatment (e.g. during dental treatment).
  • the target/treatment site 14 a may be a buccal cavity area of the patient 14 corresponding to locations of the patient's actual teeth 74 ( FIG. 3 ).
  • the orthodontic treatment 42 may be overlaid on a plain background on the display of the mobile device 54 or may be displayed directly on a screen of a smart glass worn by the clinician 10 without being superimposed in an augmented manner.
  • patients' actual teeth 74 may be captured preferably three dimensionally and analyzed to determine any corrections needed as explained in the methods hereinafter.
  • the display device 12 of FIG. 1 may be connected to or form part of a computer system 100 .
  • the computer system 100 may include the tracking system 2 and a processor 122 .
  • the tracking system 2 may alternatively be separate from the computer system 100 and may form at least part of any of the other devices, components, and/or systems discussed herein.
  • the tracking system 2 may be electrically connected to a processor 122 and may offer real-time location data for a precise location and orientation of images and objects in a common coordinate system.
  • the tracking system may be vision based, for example utilizing cameras of the camera system 3 for visual tracking of the face of the patient 14 , or characteristic points 63 ( FIG. 3 ) on the face of the patient 14 or predetermined markers (not shown) placed on the face of the patient, using for example object/pattern recognition.
  • the processor 122 may be configured to receive real time three dimensional patient facial data, to analyze said data to obtain an orthodontic treatment 42 and to overlay said orthodontic treatment 42 on (i) the treatment site 14 a for viewing by the clinician 10 through the display 12 and/or (ii) on a region in a 3D model or stereoscopic video of the patient corresponding to the treatment site 14 .
  • the clinician 10 and/or patient 14 may receive real-time proposed orthodontic treatments that possess depth and that may move/shift accordingly with changing facial expressions or locations of the patient 14 as if they were real teeth. This may be achieved, for example, by monitoring the characteristic points 63 in real time and showing or hiding portions of the displayed orthodontic treatments based on changes in locations of the characteristic points 63 caused by the changing facial expressions.
  • Example facial expressions may include (a) smiling wherein the processor 122 may be configured to show all teeth or most teeth, (b) opening the mouth wherein the processor 122 may be configured to show the upper jaw to not be in contact with the lower jaw, (c) closing the mouth wherein the processor 122 may be configured to show no teeth, (d) biting wherein the processor 122 may be configured to show the upper jaw in contact with the lower jaw) etc.
  • the orthodontic treatment 42 may optionally be overlaid on the treatment site 14 a after a request is received from the clinician 10 through a user interface of the computer system 100 (such as a gesture recognition system and/or a voice recognition system or the like) before or during a treatment procedure.
  • a user interface of the computer system 100 such as a gesture recognition system and/or a voice recognition system or the like
  • Overlaying of the orthodontic treatment 42 on the treatment site 14 a or 3D model or stereo video of the treatment site may be performed dynamically and in real time and may be achieved by the processor 122 working in tandem with the tracking system 2 wherein changes in position of (i) the patient 14 and/or (ii) the clinician 10 , as well as changes in facial expressions of the patient 14 , captured by the tracking system 2 , may be translated into corresponding changes in positions of the overlaid orthodontic treatment 42 such that said orthodontic treatment 42 appears directly superimposed even as the patient 14 and/or or clinician 10 moves.
  • the processor 122 may be configured to provide the further proposals until one is accepted.
  • An accepted proposal may be stored for fabrication of aligners or for use in other orthodontic/dental treatments.
  • FIG. 4 shows a block diagram of a computer system 100 that may be employed in accordance with at least some of the example embodiments herein.
  • FIG. 4 shows a block diagram of a computer system 100 that may be employed in accordance with at least some of the example embodiments herein.
  • various embodiments may be described herein in terms of this exemplary computer system 100 , after reading this description, it may become apparent to a person skilled in the relevant art(s) how to implement the disclosure using other computer systems and/or architectures.
  • the computer system 100 may include at least one computer processor 122 and may include a tracking system 2 , user interface 126 and input unit 130 .
  • the input unit 130 in one exemplary embodiment may be used by the clinician 10 along with a display unit 128 such as a monitor to send instructions or requests about a treatment to the computer processor 122 .
  • the input unit 130 is a finger or stylus to be used on a touchscreen interface (not shown).
  • the input unit 130 may alternatively be a gesture/voice recognition device, a trackball, a mouse or other input device such as a keyboard or stylus.
  • the display unit 128 , the input unit 130 , and the computer processor 122 may collectively form the user interface 126 .
  • the computer processor 122 may include, for example, a central processing unit, a multiple processing unit, an application-specific integrated circuit (“ASIC”), a field programmable gate array (“FPGA”), or the like.
  • the processor 122 may be connected to a communication infrastructure 124 (e.g., a communications bus, or a network).
  • the processor 122 may receive a request for automatically calculating and displaying an orthodontic treatment 42 and may obtain instructions concerning the request from one or more storage units of the computer system 100 .
  • the processor 122 may then load said instructions and execute the loaded instructions such as using a database or artificial intelligence (AI) to obtain possible treatments for display.
  • AI artificial intelligence
  • One or more steps/procedures for proposing the orthodontic treatment 42 may be stored on a non-transitory storage device in the form of computer-readable program instructions.
  • the processor 122 loads the appropriate instructions, as stored on a storage device, into memory and then executes the loaded instructions.
  • the computer system 100 may further comprise a main memory 132 , which may be a random access memory (“RAM”) and also may include a secondary memory 134 .
  • the secondary memory 134 may include, for example, a hard disk drive 136 and/or a removable-storage drive 138 (e.g., a floppy disk drive, a magnetic tape drive, an optical disk drive, a flash memory drive, and the like).
  • the removable-storage drive 138 may read from and/or write to a removable storage unit 140 in a well-known manner.
  • the removable storage unit 140 may be, for example, a floppy disk, a magnetic tape, an optical disk, a flash memory device, and the like, which may be written to and read from by the removable-storage drive 138 .
  • the removable storage unit 140 may include a non-transitory computer-readable storage medium storing computer-executable software instructions and/or data.
  • the secondary memory 134 may include other computer-readable media storing computer-executable programs or other instructions to be loaded into the computer system 100 .
  • Such devices may include a removable storage unit 144 and an interface 142 (e.g., a program cartridge and a cartridge interface); a removable memory chip (e.g., an erasable programmable read-only memory (“EPROM”) or a programmable read-only memory (“PROM”)) and an associated memory socket; and other removable storage units 144 and interfaces 142 that allow software and data to be transferred from the removable storage unit 144 to other parts of the computer system 100 .
  • EPROM erasable programmable read-only memory
  • PROM programmable read-only memory
  • the computer system 100 also may include a communications interface 146 that enables software and data to be transferred between the computer system 100 and external devices.
  • a communications interface 146 may include a modem, a network interface (e.g., an Ethernet card or a wireless interface), a communications port (e.g., a Universal Serial Bus (“USB”) port or a FireWire® port), a Personal Computer Memory Card International Association (“PCMCIA”) interface, Bluetooth®, and the like.
  • Software and data transferred via the communications interface 146 may be in the form of signals, which may be electronic, electromagnetic, optical or another type of signal that may be capable of being transmitted and/or received by the communications interface 146 .
  • Signals may be provided to the communications interface 146 via a communications path 148 (e.g., a channel).
  • the communications path 148 may carry signals and may be implemented using wire or cable, fiber optics, a telephone line, a cellular link, a radio-frequency (“RF”) link, or the like.
  • the communications interface 146 may be used to transfer software or data or other information between the computer system 100 and a remote server or cloud-based storage (not shown).
  • One or more computer programs or computer control logic may be stored in the main memory 132 and/or the secondary memory 134 .
  • the computer programs may also be received via the communications interface 146 .
  • the computer programs may include computer-executable instructions which, when executed by the computer processor 122 , cause the computer system 100 to perform the methods as described hereinafter.
  • the software may be stored in a non-transitory computer-readable storage medium and loaded into the main memory 132 and/or the secondary memory 134 of the computer system 100 using the removable-storage drive 138 , the hard disk drive 136 , and/or the communications interface 146 .
  • Control logic when executed by the processor 122 , causes the computer system 100 , and more generally the system for visualizing an orthodontic treatment 1 , to perform all or some of the some of the methods described herein.
  • the computer system 100 may be a mobile device 54 having an application that may be engaged by a user to propose and visualize orthodontic treatments 42 .
  • FIG. 5 shows a process 200 for proposing an orthodontic treatment.
  • FIGS. 6 a - 7 b show a three-dimensional facial data record 62 and a corresponding three-dimensional tooth data record 61 for correction of misaligned teeth.
  • the process of proposing an orthodontic treatment 42 may start at Step S 100 in FIG. 5 when a mobile device 54 or camera system 3 is engaged to capture a 3D facial data record 62 ( FIG. 6 a ).
  • a stereo video of the patient's face may alternatively be captured and object recognition methods may be used to detect anatomical features in the stereo video.
  • characteristic points 63 FIG. 6 b
  • characteristic points 63 FIG. 6 b
  • the facial recognition methods may also include methods as described in U.S. Pat. No. 8,831,295B2, entitled “Electronic device configured to apply facial recognition based upon reflected infrared illumination and related methods” by Coons, U.S. Pat. No. 7,804,997B2 entitled “Method and system for a three dimensional facial recognition system”, by Geng et al, and U.S. Pat. No. 8,457,367B1, entitled “Facial recognition” by Sipe et al, and are incorporated by reference herein in their entirety, as if set forth fully herein.
  • a three-dimensional direction 65 of the orientation of the patient's face in the three-dimensional facial data record 62 may be determined in Step S 400 .
  • a mouth area 64 between the lips of the patient 14 may be segmented and/or cut from the 3D facial data record 62 in Step S 500 .
  • the three-dimensional representation of the actual teeth 66 of the patient 14 may also be extracted in Step S 600 from the 3D facial data record 62 . From the three-dimensional representation, it may also be determined if there is a misalignment (Step S 700 ), through for example, automated measurements.
  • the establishment of reference objects, reference frames and/or detection of anatomical dental features in the three-dimensional representation of the actual teeth 66 may allow for automatic dental measurements.
  • the establishment of reference frames may allow for determination of how much the teeth are supposed to move or have moved.
  • the detection of anatomical features may comprise automatic detection of landmarks associated with the 3D representation of the patient's actual teeth 66 , the landmarks comprising at least one of cusps, ridges, grooves and contact points.
  • Dental measurements may then be computed based on the reference objects, reference frames and/or anatomical dental features and compared with reference data (for example using artificial intelligence) in order to determine the presence of misalignments.
  • Systems and methods for dental measurements may include those taught in U.S. Pat. No. 8,126,726B2, entitled “System and method for facilitating automated dental measurements and diagnostics”, by Matov et al, which is incorporated by reference herein in its entirety, as if set forth fully herein.
  • the 3D representation of the patient's actual teeth 66 may be received from a previously created database or intraoral scan.
  • Step S 800 a proposed correction for the misaligned teeth may be obtained.
  • instructions such as (i) predetermined instructions for correcting misalignments and/or (ii) instructions based on artificial intelligence may be to propose an orthodontic treatment.
  • artificial intelligence data analysis may be used on the obtained dental measurements of Step S 700 to diagnose an orthodontic condition and generate a set instructions for the processor 122 to propose a correction.
  • the processor 122 may be configured to identify a location and position of a plurality of teeth of the 3D representation of the patient's actual teeth 66 and assign corresponding coordinates.
  • the diagnosis may be based on or a best fit to one or more inputs, that may be compared with the 3D representation of the patient's actual teeth 66 such as inputs being derived from or otherwise based on ongoing patient treatments, past patient treatments (such as diagnostic data sets of a past treatment with each diagnostic data set comprising coordinates that correspond to known locations/positions of a plurality of teeth), scientific literature, textbooks etc.
  • the processor 122 may further be configured to identify at least one treatment approach, a corrective appliance, corrected teeth, a combination thereof or otherwise orthodontic treatment 42 for at least one diagnosis, which may include calculating a set of target coordinates, which may represent a desired or corrected location or position of a tooth, and identifying at least one treatment approach, a corrective appliance, or a combination thereof, which may be at least partially effective in reorienting one or more teeth toward a location or position represented by one or more target coordinates.
  • the method may also include determining a probability value that may be correlated with a relative likelihood of the orthodontic treatment 42 to be effective in reorienting one or more teeth to a location or position represented by a target, and may include employing an application of at least one artificial intelligence or other algorithm.
  • the orthodontic treatment 42 may preferably be a three-dimensional tooth data record 61 ( FIG. 7 a ) corresponding to the corrected teeth which may be stored for use in further treatment procedures such as for designing aligners.
  • the orthodontic treatment 42 may be rotated and overlaid (fully or partially or used to replace the 3D representation of the patient's actual teeth 66 on the mobile device 54 ) on the 3D facial data record 62 in such a way that the orientation of the orthodontic treatment 42 matches the 3D direction 65 of the orientation of the patient's face and the position matches the position of the mouth area 64 .
  • a relative orientation of the orthodontic treatment 42 to 3D direction 65 of the orientation of the patient's face may be established.
  • the relationship may be established, for example, on the basis of existing teeth that will remain or teeth that are to be replaced.
  • the relevant relationship data may be stored so that the orientation of the orthodontic treatment 42 may take place automatically when the patient moves.
  • the 3D facial data record 62 may be colored due to the use of a color camera 62 in the smartphone and may also be updated in real time.
  • the tracking system 2 may be used to continuously orient the orthodontic treatment as discussed hereinafter.
  • the orthodontic treatment 42 may be routed to a screen of the display device 12 such that the orthodontic treatment 42 appears directly superimposed on a treatment site 14 a of the patient 14 in an augmented manner.
  • data from the tracking system 2 including, for example, (i) real time data tracking changes in the patient's facial expressions 44 (e.g. smiling) (ii) real time data tracking clinician movements 46 , and/or (iii) real time data tracking patient movements 50 (e.g. changes in position), the augmented data (orthodontic treatment 42 ) routed to the display device 12 may be dynamically updated in real time for overlay on the treatment site 14 a such that said orthodontic treatment 42 appears directly superimposed on said site.
  • Said treatment site 14 a may be recognized using, for example, object recognition on data obtained from the camera system 3 .
  • example embodiments described herein provide a method, system and computer readable storage media for proposing and visualizing an orthodontic treatment.
US17/310,278 2019-01-30 2020-01-22 Method and system for proposing and visualizing dental treatments Active 2040-08-10 US11915378B2 (en)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
EP19000053.9 2019-01-30
EP19000053 2019-01-30
EP19000053.9A EP3689287B1 (fr) 2019-01-30 2019-01-30 Système de proposition et de visualisation de traitements dentaires
PCT/EP2020/051468 WO2020156896A1 (fr) 2019-01-30 2020-01-22 Procédé et système pour proposer et visualiser des traitements dentaires

Publications (2)

Publication Number Publication Date
US20220139044A1 US20220139044A1 (en) 2022-05-05
US11915378B2 true US11915378B2 (en) 2024-02-27

Family

ID=65275921

Family Applications (1)

Application Number Title Priority Date Filing Date
US17/310,278 Active 2040-08-10 US11915378B2 (en) 2019-01-30 2020-01-22 Method and system for proposing and visualizing dental treatments

Country Status (9)

Country Link
US (1) US11915378B2 (fr)
EP (1) EP3689287B1 (fr)
JP (1) JP7472145B2 (fr)
KR (1) KR20210121050A (fr)
CN (1) CN113164237A (fr)
AU (1) AU2020213847A1 (fr)
BR (1) BR112021008107A2 (fr)
CA (1) CA3117881A1 (fr)
WO (1) WO2020156896A1 (fr)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3689287B1 (fr) * 2019-01-30 2022-07-27 DENTSPLY SIRONA Inc. Système de proposition et de visualisation de traitements dentaires
US11937996B2 (en) * 2019-11-05 2024-03-26 Align Technology, Inc. Face capture and intraoral scanner and methods of use
TWI712396B (zh) * 2020-01-16 2020-12-11 中國醫藥大學 口腔缺陷模型之修補方法及口腔缺陷模型之修補系統
KR102476832B1 (ko) * 2020-08-10 2022-12-13 아주대학교산학협력단 증강현실 의료영상을 제공하기 위한 사용자 단말기 및 증강현실 의료영상 제공 방법
WO2024049918A2 (fr) * 2022-08-30 2024-03-07 Align Technology, Inc. Reconstruction et visualisation faciales 3d dans une planification de traitement dentaire

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170065379A1 (en) * 2014-02-21 2017-03-09 Trispera Dental, Inc. Augmented Reality Dental Design Method and System
US10467815B2 (en) * 2016-12-16 2019-11-05 Align Technology, Inc. Augmented reality planning and viewing of dental treatment outcomes
EP3689287A1 (fr) * 2019-01-30 2020-08-05 DENTSPLY SIRONA Inc. Procédé et système de proposition et de visualisation de traitements dentaires
US20200306010A1 (en) * 2019-03-26 2020-10-01 CVSTOM Co. Methods and systems for orthodontic treatment planning
WO2020232223A1 (fr) * 2019-05-16 2020-11-19 Alta Smiles, Llc Modèle d'analyse et de prédiction pour traitement orthodontique

Family Cites Families (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004538538A (ja) 2000-10-05 2004-12-24 シーメンス コーポレイト リサーチ インコーポレイテツド 強化現実可視化付きの、手術中にイメージガイドされる神経外科手術法及び手術用装置
US9421074B2 (en) * 2001-04-13 2016-08-23 Orametrix, Inc. Unified three dimensional virtual craniofacial and dentition model and uses thereof
JP4287646B2 (ja) 2002-12-26 2009-07-01 株式会社ミツトヨ 画像読取装置
US8126726B2 (en) * 2004-02-27 2012-02-28 Align Technology, Inc. System and method for facilitating automated dental measurements and diagnostics
US7804997B2 (en) 2004-06-10 2010-09-28 Technest Holdings, Inc. Method and system for a three dimensional facial recognition system
GB0707454D0 (en) * 2007-04-18 2007-05-23 Materialise Dental Nv Computer-assisted creation of a custom tooth set-up using facial analysis
EP2408394B1 (fr) * 2009-03-20 2019-05-08 3Shape A/S Système pour une planification, visualisation et optimisation des restaurations dentaires
US9788917B2 (en) 2010-03-17 2017-10-17 ClearCorrect Holdings, Inc. Methods and systems for employing artificial intelligence in automated orthodontic diagnosis and treatment planning
US8831295B2 (en) 2012-03-21 2014-09-09 Authentec, Inc. Electronic device configured to apply facial recognition based upon reflected infrared illumination and related methods
US8457367B1 (en) 2012-06-26 2013-06-04 Google Inc. Facial recognition
WO2015126466A1 (fr) 2014-02-21 2015-08-27 The University Of Akron Système d'imagerie et d'affichage permettant de guider des interventions médicales
US9675305B2 (en) 2014-06-03 2017-06-13 Ortho-Tain System and method for determining an orthodontic diagnostic analysis of a patient
US10154239B2 (en) 2014-12-30 2018-12-11 Onpoint Medical, Inc. Image-guided surgery with surface reconstruction and augmented reality visualization
FR3032282B1 (fr) * 2015-02-03 2018-09-14 Francois Duret Dispositif de visualisation de l'interieur d'une bouche
DE102015222782A1 (de) * 2015-11-18 2017-05-18 Sirona Dental Systems Gmbh Verfahren zur Visualisierung einer Zahnsituation
JP2017221329A (ja) 2016-06-14 2017-12-21 医療法人社団秀信会 ２次元画像を用いて生体の下顎開閉軸とバーチャル咬合器の開閉軸を一致させる方法
EP4241725A3 (fr) * 2017-03-20 2023-11-01 Align Technology, Inc. Génération de représentation virtuelle d'un traitement orthodontique d'un patient

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170065379A1 (en) * 2014-02-21 2017-03-09 Trispera Dental, Inc. Augmented Reality Dental Design Method and System
US10467815B2 (en) * 2016-12-16 2019-11-05 Align Technology, Inc. Augmented reality planning and viewing of dental treatment outcomes
US20200020170A1 (en) * 2016-12-16 2020-01-16 Align Technology, Inc. Augmented reality planning and viewing of dental treatment outcomes
EP3689287A1 (fr) * 2019-01-30 2020-08-05 DENTSPLY SIRONA Inc. Procédé et système de proposition et de visualisation de traitements dentaires
US20220139044A1 (en) * 2019-01-30 2022-05-05 Dentsply Sirona Inc. Method and system for proposing and visualizing dental treatments
US20200306010A1 (en) * 2019-03-26 2020-10-01 CVSTOM Co. Methods and systems for orthodontic treatment planning
WO2020232223A1 (fr) * 2019-05-16 2020-11-19 Alta Smiles, Llc Modèle d'analyse et de prédiction pour traitement orthodontique

Also Published As

Publication number Publication date
US20220139044A1 (en) 2022-05-05
EP3689287B1 (fr) 2022-07-27
JP2022519987A (ja) 2022-03-28
WO2020156896A1 (fr) 2020-08-06
JP7472145B2 (ja) 2024-04-22
CA3117881A1 (fr) 2020-08-06
BR112021008107A2 (pt) 2021-08-03
KR20210121050A (ko) 2021-10-07
CN113164237A (zh) 2021-07-23
AU2020213847A1 (en) 2021-05-27
EP3689287A1 (fr) 2020-08-05

Similar Documents

Publication Publication Date Title
US11915378B2 (en) Method and system for proposing and visualizing dental treatments
CN107735016B (zh) 用于扫描解剖结构且用于显示扫描结果的系统和方法
US11357576B2 (en) Method and system for augmented reality guided surgery
US20220192800A1 (en) Method and system for guiding an intra-oral scan
CN113260335B (zh) 用于测量牙周袋深度的系统
US20220096004A1 (en) System for visualizing patient stress
US11445165B1 (en) Method, system and computer readable storage media for visualizing a magnified dental treatment site
US11771506B2 (en) Method and system for controlling dental machines
KR20220107546A (ko) 표면 영상유도 기반의 환자 위치 정렬 및 모니터링 시스템

Legal Events

Date Code Title Description
FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

AS Assignment

Owner name: DENTSPLY SIRONA INC., PENNSYLVANIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:KOZA, ANDRE;KUCHARCZYK, RONNY;REEL/FRAME:066176/0555

Effective date: 20220114

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE