CN115714842A - AR (augmented reality) glasses-based automatic oral implantation operation recording method - Google Patents

AR (augmented reality) glasses-based automatic oral implantation operation recording method Download PDF

Info

Publication number
CN115714842A
CN115714842A CN202211410672.0A CN202211410672A CN115714842A CN 115714842 A CN115714842 A CN 115714842A CN 202211410672 A CN202211410672 A CN 202211410672A CN 115714842 A CN115714842 A CN 115714842A
Authority
CN
China
Prior art keywords
patient
implant
key
teeth
glasses
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202211410672.0A
Other languages
Chinese (zh)
Other versions
CN115714842B (en
Inventor
邹多宏
施旭
廖寅秀
朱明�
王子硕
王建华
杨驰
林瓊
张志愿
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Xvisio Technology Corp
Ninth Peoples Hospital Shanghai Jiaotong University School of Medicine
Original Assignee
Xvisio Technology Corp
Ninth Peoples Hospital Shanghai Jiaotong University School of Medicine
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xvisio Technology Corp, Ninth Peoples Hospital Shanghai Jiaotong University School of Medicine filed Critical Xvisio Technology Corp
Priority to CN202211410672.0A priority Critical patent/CN115714842B/en
Publication of CN115714842A publication Critical patent/CN115714842A/en
Application granted granted Critical
Publication of CN115714842B publication Critical patent/CN115714842B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Landscapes

  • Dental Tools And Instruments Or Auxiliary Dental Instruments (AREA)
  • Dental Prosthetics (AREA)

Abstract

The application discloses an automatic oral implant surgery recording method based on AR glasses, which is characterized by comprising the following steps of firstly, manufacturing an AR display model of teeth of a patient; marking the planting sites on the AR display model; step three, preassembling a marker on the teeth of the patient, and starting an oral implant operation; identifying the spatial position of the marker through a binocular vision slam algorithm; step five, overlapping and displaying the AR display model of the patient teeth and the actual spatial position of the patient teeth; step six, scanning the key surgical tool, acquiring the spatial position of the key surgical tool, judging whether the key surgical stage is entered, and recording video or photographing if the key surgical stage is entered; and seventhly, picking off the marker from the teeth of the patient, and finishing the oral implant operation. The method provided by the application realizes the automatic recording of the key operation stage, does not need any operation by a doctor in the whole process, and avoids influencing the operation state of the doctor.

Description

AR (augmented reality) glasses-based automatic oral implantation operation recording method
Technical Field
The application relates to an automatic oral implantation operation recording method based on AR glasses, and belongs to the technical field of automatic operation recording.
Background
The oral implant surgery generally refers to implanting artificial teeth in the oral cavity, in the traditional oral implant surgery, when a specific surgery stage is met, a doctor needs to take pictures or record videos so as to perform real-time comparison or later discussion judgment, in the CN109274914A, AR glasses with a camera shooting function are proposed to be used for recording, but when the specific surgery stage is met, the doctor still needs to send an instruction to control the AR glasses to record in a voice or gesture mode. Since the surgical state is dynamic, continuous, and evanescent during the procedure, if the surgeon needs to be distracted to manipulate the AR glasses, there is a possibility that the AR glasses miss recording valuable moments due to delays, on the one hand, and the surgeon's procedure is interrupted, on the other hand, resulting in a reduction in the quality of the procedure.
Disclosure of Invention
The technical problem that this application will be solved is how to realize the specific operation stage of automatic record oral cavity planting operation based on AR glasses.
In order to solve the technical problems, the technical scheme of the application is to provide an automatic oral implant surgery recording method based on AR glasses, which is characterized by comprising the following steps,
step one, acquiring CT image data of a patient oral cavity, and making an AR display model of the patient teeth;
marking the planting sites on the AR display model to obtain a marked AR display model;
step three, preassembling a marker on the alveolar bone of the patient, wearing AR glasses by a doctor, and judging that the oral implant operation starts when the marker is identified by a binocular camera of the AR glasses;
identifying the spatial position of the marker through a binocular vision slam algorithm;
calculating to obtain the actual spatial position of the patient teeth according to the relative position relationship between the marker and the patient teeth, and overlapping and displaying the AR display model of the patient teeth and the actual spatial position of the patient teeth in a display module of the AR glasses;
step six, in the process of the oral implant surgery, scanning key surgical tools through a binocular camera of AR glasses to obtain the spatial positions of the key surgical tools, judging whether the critical surgical stage is entered or not by judging the position relation between the key surgical tools and the teeth of a patient, and recording videos or photographing records if the critical surgical stage is entered;
and seventhly, recognizing that the marker is picked off from the teeth of the patient by the doctor through the binocular camera, judging that the oral implant operation is finished, and storing the recorded operation key stage.
Preferably, the key surgical tool is a tapping drill, the key surgical stage is the preparation of an implant nest, and the sixth step specifically comprises:
s6.1, acquiring an image in an operation visual field by using a binocular camera;
s6.2, continuously inputting the images into the trained neural network for identifying the tapping drill;
s6.3, after the neural network identifies the tapping drill, identifying the spatial position of the tapping drill based on a binocular vision slam algorithm;
s6.4, calculating the distance between the tapping drill and the teeth of the patient, judging to enter a key stage of the operation when the distance is smaller than a threshold value d, and starting to record a video;
s6.5 the binocular vision slam algorithm continuously tracks the tapping drill, and after the tapping drill is judged to be away from the teeth of the patient and the tooth distance of the patient exceeds a threshold value D for a period of time t, the key operation stage is judged to be finished, and video recording is finished.
Preferably, the surgical key tool is an implant, the surgical key stage is implant implantation, and the sixth step specifically includes:
s6.1, acquiring an image in an operation visual field by using a binocular camera;
s6.2, continuously inputting the image into the trained neural network for recognizing the implant;
s6.3, after the neural network identifies the implant, identifying the spatial position of the implant based on a binocular vision slam algorithm;
s6.4, calculating the distance between the implant and the implant site of the patient, judging to enter a key stage of the operation when the distance is smaller than a threshold value d, and starting to record a video;
s6.5, continuously tracking the implant by using a binocular vision slam algorithm, judging that the key stage of the operation is finished and ending the video recording after the neck of the implant and the gingival margin height of a patient are kept consistent for a period of time t.
Further, the third step further comprises marking the dental nerve, the blood vessel and the condylar ostial maxillary sinus on the AR display model.
Furthermore, in the sixth step, when the video recording is performed, the marked AR display model is recorded into the image at the same time.
The method has the advantages that the starting and the ending of the oral implant operation are judged by installing the markers on the teeth of the patient and identifying the markers by using the binocular camera; in the process of the oral implant surgery, the AR display model and the actual spatial position of the oral cavity of the patient are displayed in an overlapping mode on a display module of AR glasses according to the relative position relation between the marker and the implant, and a doctor observes the actual teeth of the patient and the AR display model of the teeth of the patient through the display module at the same time, so that the implant sites marked on the AR display model are used for assisting the doctor in carrying out the oral implant surgery; meanwhile, the binocular camera is used for identifying key surgical tools, judging whether the key surgical tools enter a key surgical stage or not, and recording videos or photographing if the key surgical tools enter the key surgical stage, so that automatic recording of the key surgical stage is achieved, a doctor does not need to perform any operation in the whole recording process, the surgical state of the doctor is prevented from being influenced, the continuity of the surgical process is maintained, the recorded key surgical stage can be used for postoperative analysis, teaching and the like, can also be used for surgical quality evaluation, and provides key evidence to avoid medical disputes.
Drawings
Fig. 1 is a flow chart of a method of automated AR glasses-based oral implant procedure recording provided in an embodiment.
Detailed Description
In order to make the present application more comprehensible, preferred embodiments are described in detail below with reference to the accompanying drawings.
Examples
The embodiment provides an automatic oral implantation surgery recording method based on AR glasses, the AR glasses adopted by the embodiment at least comprise a binocular camera for visual collection, a processor for calculation and a storage module for storage besides a display module for AR display, when in use, a doctor wears the AR glasses, the display module is used for AR display, the binocular camera is used for visual collection, and the processor is used for executing a binocular vision slam algorithm, a neural network algorithm and the like;
specifically, referring to fig. 1, the method includes:
step one, acquiring CT image data of a patient oral cavity, and making an AR display model of the patient teeth;
marking the planting sites on the AR display model to obtain a marked AR display model;
step three, preassembling a marker on the alveolar bone of the patient, wearing AR glasses by a doctor, and judging that the oral implant operation starts when the marker is identified by a binocular camera of the AR glasses;
identifying the spatial position of the marker through a binocular vision slam algorithm;
calculating to obtain the actual spatial position of the patient teeth according to the relative position relation between the marker and the patient teeth, and overlapping and displaying the AR display model of the patient teeth and the actual spatial position of the patient teeth in a display module of AR glasses;
step six, in the process of the oral implant surgery, scanning key surgical tools through a binocular camera of AR glasses to obtain the spatial positions of the key surgical tools, judging whether the critical surgical stage is entered or not by judging the position relation between the key surgical tools and the teeth of a patient, and recording videos or photographing records if the critical surgical stage is entered;
and seventhly, recognizing that the marker is picked off from the teeth of the patient by the doctor through the binocular camera, judging that the oral implant operation is finished, and storing the recorded operation key stage.
According to the method, the starting and the ending of the oral implant operation are judged by installing the markers on the teeth of the patient and identifying the markers by using the binocular camera; in the process of the oral implant operation, the AR display model and the actual spatial position of the teeth of the patient are displayed in an overlapping manner on the display module of the AR glasses according to the relative position relationship between the marker and the teeth of the patient, and a doctor observes the actual teeth of the patient and the AR display model of the teeth of the patient through the display module at the same time, so that the position of the root and mouth of the teeth marked on the AR display model is utilized to assist the doctor in carrying out the oral implant operation; meanwhile, a binocular camera is used for identifying key surgical tools, whether the key surgical tools enter a key surgical stage or not is judged, if the key surgical tools enter the key surgical stage, video recording or photographing recording is carried out, so that automatic recording of the key surgical stages is achieved, a doctor does not need to carry out any operation in the whole recording process, the operation state of the doctor is avoided being influenced, the continuity of the operation process is maintained, meanwhile, the recorded key surgical stages can be used for postoperative analysis, teaching and the like, can also be used for operation quality evaluation, and meanwhile, key evidence is provided to avoid medical disputes.
The whole oral implant operation mainly comprises the following stages: the doctor determines the best planting angle and position through the CT image data of the oral cavity and the AR display model, and lays a foundation for the operation; cutting off alveolar ridges, stripping periosteum flaps at two sides of the cut, fully exposing bone surfaces of a planting area, trimming alveolar bones, and removing soft tissues adhered to the surfaces, residual granulation tissues and sharp bone tips; then preparing an implant pit on the alveolar bone through a tapping drill to form a thread shape; finally, the implant is screwed in, a covering screw or a healing cap is installed, the wound is sutured, and the first-stage operation is finished. Wherein, the key operation worker mainly comprises a tapping drill which is used for preparing an implant nest on the alveolar bone, and an implant which is arranged in the implant nest to serve as an artificial tooth root; the main operation key stage that involves is, tapping bores and prepares the planting nest on the alveolar bone, the whole process needs to record video for later stage teaching and quality inspection; 2. the implant is implanted into the implant nest, the image is needed in the implantation process, the distance between the implant and key areas such as dental nerves, blood vessels and condylar foramen maxillary sinus is recorded, the fact that the implant does not invade the key areas such as the dental nerves, the blood vessels and the condylar foramen maxillary sinus is guaranteed, later-stage quality inspection is facilitated, and key evidence is provided to avoid medical disputes.
Based on the above operation critical stages and operation critical tools, in a preferred embodiment, the method for recording the automatic oral implant operation based on the AR glasses provided in this embodiment specifically includes the following steps:
s6.1, acquiring an image in an operation visual field by using a binocular camera;
s6.2, continuously inputting the images into the trained neural network for identifying the tapping drill (or the implant);
s6.3, after the neural network identifies the tapping drill (or the implant), identifying the spatial position of the tapping drill (or the implant) based on a binocular vision slam algorithm;
s6.4, calculating the distance between the tapping drill and the teeth of the patient (or between the implant and the implant site of the patient), judging to enter a key stage of the operation when the distance is smaller than a threshold value d, and starting to record a video;
s6.5, continuously tracking the tapping drill (or the implant) by using a binocular vision slam algorithm, judging that the key stage of the operation is finished and finishing video recording after the distance between the tapping drill and the teeth of the patient exceeds a threshold D (or the height of the neck of the implant and the gingival margin of the patient is kept consistent) for a period of time t.
In the embodiment, the threshold D is 1cm, the threshold D is 20cm, and the time t is 20s;
in a preferred embodiment, in order to clearly show the interference relationship between the oral implant operation and key areas such as the dental nerve, the blood vessel, the condylar foramina maxillary sinus and the like, the third step further comprises marking the dental nerve, the blood vessel and the condylar foramina maxillary sinus on the AR display model; and step six, recording the marked AR display model into an image during video recording.
The embodiment also provides a system for automatic oral implantation operation record based on AR glasses, which comprises a display module for AR display, a binocular camera for visual collection, a processor for calculation and a storage module for storage.
The binocular camera is installed on the AR glasses and used for conducting visual acquisition on the operation visual field and achieving video recording or shooting.
And executing a binocular vision slam algorithm and a neural network algorithm in the processor, wherein the neural network algorithm comprises a trained neural network and is used for identifying the key surgical tool, and the binocular vision slam algorithm acquires the spatial position of the identified key surgical tool.
The storage module stores the recorded image, and in a preferred embodiment, the system further comprises a communication module for transmitting the recorded image to a terminal or a server.
In a preferred embodiment, the processor is a computer system or a processor chip built in the AR glasses.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.

Claims (7)

1. A method for recording an automatic oral implantation operation based on AR glasses is characterized by comprising the following steps,
step one, acquiring CT image data of a patient oral cavity, and making an AR display model of the patient teeth;
marking the planting sites on the AR display model to obtain a marked AR display model;
step three, preassembling a marker on the alveolar bone of the patient, wearing AR glasses by a doctor, and judging that the oral implant operation starts when the marker is identified by a binocular camera of the AR glasses;
identifying the spatial position of the marker through a binocular vision slam algorithm;
calculating to obtain the actual spatial position of the patient teeth according to the relative position relationship between the marker and the patient teeth, and overlapping and displaying the AR display model of the patient teeth and the actual spatial position of the patient teeth in a display module of the AR glasses;
step six, in the process of the oral implant surgery, scanning key surgical tools through a binocular camera of AR glasses to obtain the spatial positions of the key surgical tools, judging whether the critical surgical stage is entered or not by judging the position relation between the key surgical tools and the teeth of a patient, and recording videos or photographing records if the critical surgical stage is entered;
and seventhly, recognizing that the marker is picked off from the teeth of the patient by the doctor through the binocular camera, judging that the oral implant operation is finished, and storing the recorded operation key stage.
2. The method for automated oral implant procedure records based on AR glasses according to claim 1, wherein the key surgical tool is a tapping drill and the key surgical stage is the preparation of an implant nest.
3. The method for automated AR glasses-based dental implant procedure recording according to claim 2, wherein the sixth step specifically comprises:
s6.1, acquiring an image in an operation visual field by using a binocular camera;
s6.2, continuously inputting the images into the trained neural network for identifying the tapping drill;
s6.3, after the neural network identifies the tapping drill, identifying the spatial position of the tapping drill based on a binocular vision slam algorithm;
s6.4, calculating the distance between the tapping drill and the teeth of the patient, judging to enter a key stage of the operation when the distance is smaller than a threshold value d, and starting to record a video;
s6.5 the binocular vision slam algorithm continuously tracks the tapping drill, and after the distance between the tapping drill and the teeth of the patient exceeds a threshold D for a period of time t, the key stage of the operation is judged to be finished, and the video recording is finished.
4. The method of claim 1, wherein the surgical key tool is an implant and the surgical key stage is implant implantation.
5. The method of claim 4, wherein the step six specifically comprises:
s6.1, acquiring an image in an operation visual field by using a binocular camera;
s6.2, continuously inputting the image into the trained neural network for recognizing the implant;
s6.3, after the neural network identifies the implant, identifying the spatial position of the implant based on a binocular vision slam algorithm;
s6.4, calculating the distance between the implant and the implant site of the patient, judging to enter a key stage of the operation when the distance is smaller than a threshold value d, and starting to record a video;
s6.5, continuously tracking the implant by using a binocular vision slam algorithm, judging that the key operation stage is finished and finishing video recording after the neck of the implant and the gingival margin height of a patient are kept consistent for a period of time t.
6. The method of claim 5, wherein the step three further comprises marking the dental nerve, the blood vessel and the maxillary sinus of the condylar foramen on the AR display model.
7. The method for recording the AR glasses-based automatic oral implant surgery according to claim 3 or 6, wherein in the sixth step, during the video recording, the marked AR display model is recorded into the image.
CN202211410672.0A 2022-11-11 2022-11-11 AR (augmented reality) glasses-based automatic oral implantation operation recording method Active CN115714842B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211410672.0A CN115714842B (en) 2022-11-11 2022-11-11 AR (augmented reality) glasses-based automatic oral implantation operation recording method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211410672.0A CN115714842B (en) 2022-11-11 2022-11-11 AR (augmented reality) glasses-based automatic oral implantation operation recording method

Publications (2)

Publication Number Publication Date
CN115714842A true CN115714842A (en) 2023-02-24
CN115714842B CN115714842B (en) 2024-05-17

Family

ID=85232794

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211410672.0A Active CN115714842B (en) 2022-11-11 2022-11-11 AR (augmented reality) glasses-based automatic oral implantation operation recording method

Country Status (1)

Country Link
CN (1) CN115714842B (en)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2013034764A (en) * 2011-08-10 2013-02-21 Akira Takebayashi Surgical guide device and method for positioning drill
US20150120726A1 (en) * 2013-10-30 2015-04-30 Texas Instruments Incorporated Using Audio Cues to Improve Object Retrieval in Video
KR20190000488A (en) * 2017-06-23 2019-01-03 동의대학교 산학협력단 A tooth mirror for monitering oral
CN109171604A (en) * 2018-08-05 2019-01-11 广州高通影像技术有限公司 A kind of intelligent endoscope operating system having AR function
CN109274914A (en) * 2018-11-06 2019-01-25 北京智能决策医疗科技有限公司 A kind of method and system of the automatic operation interpretation of records based on AR glasses
CN112738443A (en) * 2021-01-07 2021-04-30 浙江德维迪亚数字科技有限公司 Intelligent field recording interaction system based on AR glasses
CN112885436A (en) * 2021-02-25 2021-06-01 刘春煦 Dental surgery real-time auxiliary system based on augmented reality three-dimensional imaging
CN114554179A (en) * 2022-02-24 2022-05-27 北京蜂巢世纪科技有限公司 Automatic shooting method, system, terminal and storage medium based on target model

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2013034764A (en) * 2011-08-10 2013-02-21 Akira Takebayashi Surgical guide device and method for positioning drill
US20150120726A1 (en) * 2013-10-30 2015-04-30 Texas Instruments Incorporated Using Audio Cues to Improve Object Retrieval in Video
KR20190000488A (en) * 2017-06-23 2019-01-03 동의대학교 산학협력단 A tooth mirror for monitering oral
CN109171604A (en) * 2018-08-05 2019-01-11 广州高通影像技术有限公司 A kind of intelligent endoscope operating system having AR function
CN109274914A (en) * 2018-11-06 2019-01-25 北京智能决策医疗科技有限公司 A kind of method and system of the automatic operation interpretation of records based on AR glasses
CN112738443A (en) * 2021-01-07 2021-04-30 浙江德维迪亚数字科技有限公司 Intelligent field recording interaction system based on AR glasses
CN112885436A (en) * 2021-02-25 2021-06-01 刘春煦 Dental surgery real-time auxiliary system based on augmented reality three-dimensional imaging
CN114554179A (en) * 2022-02-24 2022-05-27 北京蜂巢世纪科技有限公司 Automatic shooting method, system, terminal and storage medium based on target model

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
BENEDETTO BARABINO等: "An Offline Framework for Handling Automatic Passenger Counting Raw Data", 《IEEE》, 31 December 2014 (2014-12-31) *
卢晓勇;游斌;林瑜;陈木生;: "基于数字相机和时间心理视觉调制的增强现实技术", 计算机应用, no. 08, 10 August 2017 (2017-08-10) *
汤轶;肖高明;陈跃军;吴冠宇;王志刚;: "虚拟现实、增强现实和混合现实技术在胸壁肿瘤切除及重建手术治疗中的应用", 中华胸部外科电子杂志, no. 02, 28 May 2018 (2018-05-28) *

Also Published As

Publication number Publication date
CN115714842B (en) 2024-05-17

Similar Documents

Publication Publication Date Title
EP3012759B1 (en) Method for planning, preparing, accompaniment, monitoring and/or final control of a surgical procedure in the human or animal body, system for carrying out such a procedure and use of the device
KR101687821B1 (en) Method for dental surgery using augmented reality
CN102170841B (en) A dental bone implant, methods for implanting the dental bone implant and methods and systems for manufacturing dental bone implants
Fortin et al. Computer-assisted dental implant surgery using computed tomography
US8170327B2 (en) Method for (semi-) automatic dental implant planning
US7367801B2 (en) Method for precisely-positioned production of a cavity, especially a bone cavity and instrument therefor
US20110136077A1 (en) Method for making a dental prosthesis and related surgical guide
CN111407443A (en) Accurate positioning and intelligent navigation method for oral implantation robot
CN108210098A (en) Man-machine collaboration machine ethnic group tooth method and man-machine collaboration machine ethnic group tooth system based on augmented reality
WO2004017860A1 (en) Artificial tooth root implantation position determining instrument, artificial tooth root implantation position determining method, guide member manufacturing device, sensor, drill, artificial tooth manufacturing device, computer program, and recorded medium
CN106725936A (en) A kind of dentistry 3D printing system based on cloud
JP3820403B2 (en) Artificial root placement position specifying device, computer program, and recording medium
JP2013034764A (en) Surgical guide device and method for positioning drill
WO2015154125A9 (en) Dental implant guide formation system
KR20200070706A (en) Design method and apparatus for guiding a dental implant surgery that determines a crown position by reflecting a relationship between adjacent teeth and a an antagonistic teeth
CN112972027A (en) Orthodontic micro-implant implantation positioning method using mixed reality technology
KR101653494B1 (en) The method and system for fine operation of tooth
JP2006271986A (en) Apparatus and method for specifying artificial tooth root planting position
CN115714842B (en) AR (augmented reality) glasses-based automatic oral implantation operation recording method
CN113133802B (en) Bone surgery line automatic positioning method based on machine learning
CN110495959A (en) A kind of intelligent medical U.S. tooth kind tooth method and auxiliary diagnosis equipment
CN113143457A (en) Maxillofacial operation auxiliary system and method based on MR head-mounted equipment
CN211863018U (en) Dental surgery auxiliary device
CN210541887U (en) Accurate implant tooth repair operation titanium mesh device of individualized bone ridge regeneration titanium mesh
CN114587657B (en) Auxiliary navigation method and system for oral implantation based on mixed reality technology

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant