JP2019000531A - Oral cavity treatment support system and oral cavity treatment support method - Google Patents

Abstract

To provide an oral cavity treatment support system and the like which can improve efficiency of treatment and increase experience of a practitioner.SOLUTION: An oral cavity treatment support system includes a touch panel-type PC 10, a dental instrument 20, a camera 30, an HMD device 40, and a portable terminal 50. The touch panel-type PC obtains an actual image data which is a two-dimensional static image containing a root canal of an oral cavity of a patient imaged by a camera, displays it on the HMD device, and statistically predicts and obtains a two-dimensional or three-dimensional predicted image data from the past data stored in a predefined storage medium based on an image data of the current teeth of a patient even when a practitioner cannot visually recognize the root canal during operation which has been visible at first. When a tracking angle returns within a range of the above tolerance value, the portable terminal displays each of the images displayed in the HMD device in the same manner, and expand or reduce each image to display upon receiving an operation of the practitioner on an own terminal.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for assisting intraoral treatment by imaging a treatment site.

In the oral cavity treatment, the root canal is difficult to visually recognize for the dentist who is a practitioner, and the direct observation may be difficult.
However, since it is necessary for the practitioner to grasp the image of the treatment site including the root canal and proceed with the treatment, even a practitioner with little experience value can perform various treatments so far. Solutions have been proposed.

  For example, as a related technique, an imaging module and a mounting unit are provided, and when the dental treatment imaging device is mounted on the bottom or side of the head unit by the mounting unit, the imaging axis by the imaging module and the rotational axis of the rotary cutting tool Is substantially parallel and does not use an expensive image guide, and simplifies the configuration, and the imaging module is located at the bottom or side of the head, and the imaging axis by the imaging module and the rotational axis of the rotary cutting tool In addition, even if the imaging axis and the rotation axis are substantially parallel, the imaging range of the imaging module has a finite viewing angle, and the imaging module can image the tip of the rotary cutting tool. There is a technique that can image a treatment site more reliably (see Patent Document 1).

  In addition, as a related technique, a three-dimensional computer model of a patient's teeth, including a pulp chamber and a root canal, a patient's teeth to be treated as at least one 2D gray value image of the teeth and / or visible in the oral cavity of the teeth. There is a technique in which a part of surface information with a small part is combined with a statistically digitized shape model of each type of tooth (see Patent Document 2).

Japanese Patent No. 5085732 JP-T-2014-529451

However, as a problem peculiar to dental root canal treatment, there are cases where the root canal, which was initially visible to the practitioner, cannot be viewed from various causes (for example, covered with gingiva) during the operation. In such a case, the most realistic and accurate image of the root canal that was initially visible was used to the maximum, and the root canal position was predicted from past accumulated data only when necessary. No specific technology for displaying was provided.
Unlike the conventional technique for virtually displaying the invisible root canal location, the present invention makes the best use of the closest real and accurate image of the root canal location that was initially visible. There is a point in doing so, and so-called hindrance factors are inherent in the gazettes and the like that disclose a technique for pursuing conventional mere virtual display.

  The object of the present invention is to install a very small camera on various dental instruments (instruments), and to display the captured intraoral image on an HMD (head mounted display) device to improve the efficiency of treatment, It is to provide an intraoral treatment support system and the like that can raise the experience value of a practitioner by incorporating AI as the support.

The intraoral treatment support system of the present invention includes:
A touch panel PC, a dental instrument, a camera, an HMD device, and a mobile terminal are provided, and the touch panel PC is wired to a camera provided in the dental instrument, and the HMD device is Wired connection with the portable terminal, the touch panel PC is an intraoral treatment support system wirelessly connected with the portable terminal,
The touch panel PC is
Acquiring real image data, which is a two-dimensional still image taken by the camera, including a root canal in the oral cavity of a patient to be treated with the dental instrument;
An image of a real image including a root canal in the oral cavity of the patient photographed by the camera is displayed on the HMD device via the portable terminal,
Accepting the operator's operation on the automatic recognition or the touch panel of the own device, it is possible to set a marker for tracking the real image data displayed superimposed on the real image,
The real image data is displayed on the HMD device via the mobile terminal by superimposing and following the real image image,
Even if the root canal that was initially visible to the practitioner is not visible during the operation, the real image data is superimposed and followed by the image of the real image in a state where the root canal is no longer visible. Display on the HMD device via the portable terminal;
Based on the imaging data of the patient's current teeth imaged by the camera, the prediction data of 2D or 3D is statistically predicted and acquired from the past data accumulated in a predetermined storage medium,
When the real image data is superimposed on and followed by the real image in a state where the root canal is not visible, if the follow angle exceeds an allowable value, the predicted image data is converted to the real image. Overlay and follow the image to display,
When the follow-up angle returns to within the allowable value range, the real image data is displayed again by superimposing and following the real image in a state where the root canal is no longer visible,
The portable terminal is
Each image displayed on the HMD device is displayed in the same manner, and the operator's operation on the terminal is accepted, and the images are enlarged or reduced for display.

By configuring in this way, a very small fiberscope camera or the like is installed on the dental instrument, and the imaged intraoral image is displayed on the HMD device, so that the practitioner can see the work field up close. The treatment can be made more efficient.
In addition, since the HMD device is used, an image is presented in a normal line-of-sight direction, and treatment can be performed in a natural posture without changing the posture in particular to look at the display device like a conventional intraoral camera. it can.
In addition, since the practitioner uses a portable terminal such as a smartphone or a tablet terminal, an operation can be performed by putting it in a white coat pocket, and an enlarged display or the like can be performed using a digital zoom instead of an optical zoom.
In addition, using current imaging data of the patient's teeth, two-dimensional and three-dimensional images are statistically predicted and displayed from past data, so even practitioners with little experience value can By drawing out and effectively utilizing it, the experience value of the practitioner can be substantially raised and treatment with a high experience value can be performed.
In addition, when the root canal that was initially visible to the practitioner cannot be visually recognized due to various causes (for example, covered with gingiva) during the operation, when the follow-up angle exceeds the allowable value, the AI is changed from the viewing mode to the AI. The mode can be switched so that the most realistic and accurate image of the root canal that was initially visible can be used to the maximum, and the root canal position can be predicted and displayed only when necessary. .
Further, since the practitioner can perform tracking marking by finger operation from the touch panel type PC in the browsing mode, tracking (tracking) can be realized easily and reliably.

The intraoral treatment support method of the present invention comprises:
A touch panel type PC, a dental instrument, a camera, an HMD device, and a portable terminal, wherein the touch panel type PC is wired to a camera provided in the dental instrument, and the HMD device Is a wired connection with the mobile terminal, and the touch panel PC is an intraoral treatment support method using a wireless connection with the mobile terminal,
The touch panel PC is
Obtaining real image data, which is a two-dimensional still image taken by the camera and including a root canal in the oral cavity of a patient to be treated with the dental instrument;
Displaying an image of a real image including a root canal in the oral cavity of the patient photographed by the camera on the HMD device via the portable terminal;
A step of setting a marker for following the real image data displayed by being superimposed on the real image image by receiving an operator's operation on the touch panel of the device by automatic recognition or the reality Displaying image data on the HMD device via the portable terminal by superimposing and following the image data on the real image; and
Even if the root canal that was initially visible to the practitioner is not visible during the operation, the real image data is superimposed and followed by the image of the real image in a state where the root canal is no longer visible. Displaying on the HMD device via the portable terminal;
A step of statistically predicting and obtaining predicted two-dimensional or three-dimensional image data from past data stored in a predetermined storage medium based on imaging data of the patient's current teeth imaged by the camera When,
When the real image data is superimposed on and followed by the real image in a state where the root canal is not visible, if the follow angle exceeds an allowable value, the predicted image data is converted to the real image. Superimposing and following the image for display;
A step of superimposing and following the real image data on the real image in a state where the root canal is no longer visible when the follow angle returns to within the allowable range. And
The mobile terminal is
Each of the images displayed on the HMD device is displayed in the same manner, and an operation performed by the practitioner on the terminal is received to display the images in an enlarged or reduced manner.

  According to the present invention, it is possible to provide an intraoral treatment support system that can improve the efficiency of treatment and raise the experience value of a practitioner.

It is a figure which shows the structure of the intraoral treatment assistance system which concerns on embodiment of this invention. It is a functional block diagram of touch panel type PC in the intraoral treatment support system concerning an embodiment of the invention. It is a figure which shows the image of the display image of the HMD apparatus in the intraoral treatment assistance system which concerns on embodiment of this invention. In particular, FIG. 4 is a schematic diagram when the root canal becomes invisible (including a part), and FIG. 5 is a schematic diagram of an image of a display image when the follow-up angle exceeds an allowable value. Is shown at the same angle as FIG. It is a schematic diagram when the root canal which shows the image of the display image of the HMD device in the intraoral treatment support system concerning an embodiment of the invention cannot be visually recognized. It is a figure which shows the image of the display image of the HMD apparatus in the intraoral treatment assistance system which concerns on embodiment of this invention, and is a schematic diagram of the image of a display image when a follow-up angle exceeds an allowable value, Is shown at the same angle as FIG. It is a figure explaining the marking in the intraoral treatment assistance system which concerns on embodiment of this invention. It is a flowchart which shows the processing operation of the intraoral treatment assistance system which concerns on embodiment of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
In the following description, a case where root canal treatment is mainly performed will be described as an example. However, the scope of application of the present invention is not limited to root canal treatment, and can be widely applied to dental treatment and the like.

〔System configuration〕
As shown in FIG. 1, the intraoral treatment support system S in the present embodiment includes a touch panel PC (personal computer) 10, a dental instrument 20, a small camera 30 that images a patient's oral cavity, and the like. 30, a cable 31 connected to 30, an HMD device (head mounted display device) 40 attached to the face of the practitioner, a cable 41 connected to the HMD device 40, and a portable terminal 50.
These hardware may be freely selected from commercially available products that meet certain performance criteria. Therefore, the cost of maintaining the hardware can be reduced by using an existing general-purpose product without purchasing equipment for each treatment unlike the conventional expensive system.

The touch panel PC 10 is composed of an input device such as a central processing unit (CPU), a memory, an HDD, a keyboard, a mouse, a touch panel, a liquid crystal display device, an organic EL display device, and the like, and is based on an image signal output from a control unit. This is a known information processing apparatus including a predetermined display unit that outputs and displays an image. A processing capability of a certain level that does not drop the processing is necessary, and a notebook type is preferable. Note that an information processing apparatus capable of executing the same processing as that of the touch panel PC 10 is widely applicable.
The touch panel PC 10 is realized by installing a predetermined program. The program is stored in the memory. The program can be stored in various known storage media.

FIG. 2 shows a functional block diagram of the touch panel type PC 10.
The touch panel PC 10 includes a real image acquisition unit 11, a real image data acquisition unit 12, a browsing mode realization unit 13, a marking unit 14, a tracking unit 15, a predicted image data acquisition unit 16, a tracking angle determination / mode switching unit 17, and an AI mode implementation. A unit 18 and a grid display unit 19 are provided. Each unit may be realized by a CPU or the like.

The touch panel type PC 10 is a real image of the surgical site in the patient's oral cavity taken by the camera 30 on the predetermined display unit of the own device and the HMD device 40 (hereinafter sometimes simply referred to as “HMD device 40”). 110, “real image data” 120, which is a two-dimensional still image including a root canal in a patient's oral cavity created during or before surgery, is overlaid and followed. Thus, the coordinate conversion process can be performed and displayed. The real-image image 110 includes a predetermined tooth shape portion that serves as a marker for following, and the touch panel type PC 10 has its position coordinates (that is, three-dimensional position coordinates (translation: X, Y, Z)) and The posture at the position coordinate (that is, the posture of the marker, the tilted state, etc.) is measured. Note that the degree of superimposition of the real image 110 and the “real image data” 120 is displayed in a grid display including the matching rate (%) in the X-axis / Y-axis directions, and image position correction for complete superposition is performed. It may be for the convenience of work.
That is, the touch panel type PC, as the “browsing mode”, captures a still image by imaging the root canal that is visible as a hole, for example, during or before the operation, and “ “Real image data” 120 is displayed on the HMD device 40. And even if the root canal that was initially visible to the practitioner is not visible due to various causes (for example, covered with gingiva) during the operation, The root canal “real image data” 120 is continuously superimposed on and followed by the real image 110 of the tooth under operation in a state in which the real root canal is no longer visible (the transparency can be adjusted by the slider icon). May be displayed on the HMD device 40 (see FIG. 3). Therefore, an important index called root canal position information can be displayed superimposed on and followed by a real image of the surgical site.
Further, for superimposition and follow-up display, there is no need for a special marking operation by a practitioner, which will be described later, and tracking (tracking) is based on automatically recognizing a patient's tooth itself as a marker. However, in the “browsing mode”, the touch panel type PC 10 accepts a finger operation of a practitioner or the like on the touch panel of a predetermined display unit of the device itself in the “browsing mode”, for example, and an image of a real image of an intraoperative tooth Marking (setting a marker) may be possible for simply and reliably realizing “real image data” 120 that is transparently displayed superimposed and followed by 110. The marker setting portion may be displayed in a format that is transparent and does not interfere with the display of other existing images.
A marking determination operation is manually performed, and a conversion matrix to a three-dimensional image based on the data of the position and orientation (scale, rotation, movement, etc.) is stored in the touch panel PC 10. Note that this transformation matrix is information (three-dimensional positional relationship information) indicating the relative relationship between the three-dimensional position and orientation of the marker and the three-dimensional position and orientation of the image. I understand (image of skewering).

  Further, the touch panel PC 10 uses statistical data from past data based on imaging data of the current tooth of the patient captured by the camera 30 during or before the operation of the “predicted image data” 130 used in the “AI mode”. A two-dimensional (or three-dimensional) still image including the root canal in the patient's mouth is predicted and prepared. Specifically, the “predicted image data” 130 is obtained by changing the image by HSV inverted image, gray scale, edge image, etc. by image processing, performing automatic extraction characterization from the image, and further by texture analysis of the entire image It is calculated based on matching with past data using SVM (Support Vector Machine) which is one method of machine learning by integrating the extracted feature quantities of 200 or more. Note that a similar image may be simply displayed from past data, and at that time, a plurality of types of data may be combined or a plurality of types of data may be displayed in parallel. Although it may be based on imaging data (including CT images and the like) of the patient's teeth taken in advance before surgery, the camera 30 images the current teeth of the patient during surgery and is based on the imaging data. In this case, the pre-shooting burden is reduced.

  The touch panel type PC 10 is in the “viewing mode”, and even if the root canal that was initially visible to the practitioner is not visible due to various causes (for example, covered with gingiva) during the operation, The HMD device 40 transparently superimposes and follows the “real image data” 120 of the root canal in a state in which the hole is visible to the image 110 of the real image of the tooth in operation in a state in which the real root canal is no longer visible. However, if the follow-up angle exceeds the allowable value, the effectiveness of the “real image data” 120 decreases, so the “view mode” is switched to the “AI mode”, and the follow-up angle is displayed. The “predicted image data” 130 according to the above is superimposed and followed by the real image 110 of the tooth under operation and transparently displayed on the HMD device 40 (see FIG. 5). Conversely, when the follow-up angle returns to within the allowable value, the “AI mode” may be switched to the “browsing mode”, and the “real image data” 120 may be displayed on the HMD device 40 again. The “predicted image data” 130 is adjusted in advance in size and superposition position (including the marker position) so that it can be replaced with “real image data” 120.

  In addition, the touch panel type PC 10 displays on the HMD device 40, for example, grid display (for example, display the green line in two dimensions on the entire screen in advance, make every three thick lines, and display one basic line in red lines) )), The position may be corrected by a finger operation on the touch panel of the predetermined display unit of the device itself, or the pitch may be variable according to the enlargement / reduction of the image.

  Note that an image displayed on a predetermined display unit of the touch panel PC 10 is similarly output to the HMD device 40 as an image. Therefore, the image that the practitioner sees with the HMD device 40 during the operation can be seen by the image displayed on the predetermined display unit by the operation staff other than the practitioner, and the state of the operation can be confirmed. Yes.

  Various types of dental instruments 20 are included, but files that are root canal enlargement instruments include both rotary and ultrasonic types. It is formed in a long shape that can be gripped by the practitioner's hand, and the cable extends from the end.

The camera 30 is a fiberscope camera that is used by being mounted on the dental instrument 20. The camera 30 is detachably provided on the dental instrument 20 through, for example, a mounting portion, and can be customized to a diameter of 1 mm. . The fiberscope camera is freely bent by the glass fiber and deforms flexibly according to the operation of the practitioner's hands and feet, and the camera lens unit is not necessarily located at the bottom or side near the head of the dental instrument 40. May not be in the relationship of being positioned, and may be provided in the upper part, the shaft part or other parts. Moreover, the write function may coexist. Note that the camera is not necessarily detachably mounted in a posture in which the imaging axis of the camera and the rotation axis of the rotary cutting tool (dental instrument 40) are substantially parallel.
The camera 30 captures a real image 110 of the surgical site in the patient's mouth and outputs it to the touch panel PC 10 via the cable 31. Note that the captured image is displayed on the HMD device 40 via the portable terminal 50 wirelessly connected to the touch panel PC 10.

The HMD device 40 superimposes “real image data” 120 and “predicted image data” 130 on the real image 110 of the surgical site in the patient's mouth taken by the camera 30 and output from the touch panel PC 10. This is a binocular glasses type (one eye type) optical transmission type device for viewing what is transparently displayed, and is used by being attached to the face of the practitioner.
During the operation, the practitioner does not need to compare with the monitor image of the PC as in the prior art, and the time is shortened.
In addition, since it is an optical transmission type that can see a live real image of an operation part etc. directly in a see-through manner in the practitioner's real field of view, information by direct vision can be obtained by slightly changing the line of sight. For example, whether or not the patient feels pain can be read from the facial expression of the patient.
The HMD device 40 is connected to the portable terminal 50 by a wire such as a cable 41. Because it is wired, there is no delay due to processing, and real-time performance can be realized. At the current technical level, in the case of wireless, a delay of 0.5 seconds or the like occurs, so this is not adopted. However, it may be wireless when an HW that realizes a high processing capacity suitable for cost in the future appears. . Note that the real-time image only needs to satisfy immediacy within a range that does not hinder the operation when viewing the image, and does not require perfect time coincidence.
If the HMD device 40 has only a display function and does not include a CPU or the like, the weight can be reduced.

  The portable terminal 50 is an information processing apparatus such as a smartphone or a tablet terminal. As described above, wired connection with the HMD device 40 and wireless connection (for example, Wi-Fi) with the touch panel PC 10 are performed. Since the portable terminal 50 is small, it can be conveniently stored in a practitioner's white coat pocket or the like. It is possible to display various images displayed on the HMD device 40 in response to a touch panel operation or the like by the operator's fingers on the portable terminal 50, and to perform enlargement / reduction display or position movement display, which is difficult to identify with the naked eye. Even if the level is 1 mm, the practitioner can recognize. In addition, in order to improve visibility, parameters such as the brightness of the image background and the brightness / transparency / resolution of the image may be adjustable according to the operation on the mobile terminal 50. The start / end and switching of the “browsing mode” and the “AI mode” may be executed in accordance with an operation on the mobile terminal 50. It is assumed that a predetermined application that operates on an arbitrary OS is installed in the portable terminal.

  Next, the processing flow of the intraoral treatment support system S according to the present embodiment will be described in detail with reference to FIG.

  First, the real image acquisition unit 11 of the touch panel PC 10 causes the HMD device 40 to display the real image 110 of the surgical site in the patient's oral cavity taken by the camera 30 (step S501).

  Next, the real image data acquisition unit 12 of the touch panel type PC 10 acquires “real image data” 120 that is a two-dimensional image including the root canal in the patient's mouth taken by the camera 30 during the operation (step S502). The “real image data” 120 may be captured by the camera 30 before surgery.

Next, the browsing mode realization unit 13 of the touch panel type PC 10 displays the “real image data” 120 so as to be superimposed on the real image 110 of the operative site (see FIG. 3). (Step S503). For example, the marking part 18 of the touch panel type PC 10 can be used as a marker for following the automatic recognition of a predetermined tooth shape part or the like, as shown in FIG. Marking for following is possible by finger operation on the touch panel, and the touch panel type PC 10 has its position coordinates (that is, three-dimensional position coordinates (translation: X, Y, Z)) and the posture at the position coordinates (that is, the marker position). Orientation, inclination, etc.) is measured.
During the operation, the follower 15 of the touch panel PC 10 follows (tracks) this marker, and the three-dimensional position and orientation of the marker in the real image 110 of the surgical part in the patient's oral cavity taken by the camera 30 is in real time. The PC 10 repeatedly performs measurement judgment on whether there is no change in the three-dimensional position and orientation of the marker at a very short interval (real time), and the real image 110 of the surgical site in the patient's mouth taken by the camera 30 For example, an image as shown in FIG. 3 is output to the HMD device 40 by superimposing in real time by performing coordinate conversion processing while maintaining the three-dimensional positional relationship with the marker so that the real image data 120 is superimposed. Display.
In other words, even when the position of the marker included in the real image 110 of the surgical site changes due to the body movement of the patient and practitioner, the position of the real image data 120 is also corrected and displayed in accordance with the movement. Is done. For example, the PC 10 calculates a change amount with respect to the reference position, and corrects based on the calculated change amount. When the coordinates of the reference position are (x, y, z) and the amount of change is (Δx, Δy, Δz), the coordinates are corrected to (x + Δx, y + Δy, z + Δz).

  And the browsing mode realization part 13 of touch panel type PC10 is the above-mentioned even if it is a case where the root canal initially visible to the practitioner becomes invisible during various operations (for example, covered with gingiva). The "real image data" 120 of the root canal that is visible as a hole in the HMD device is transparently superposed on and followed by the real image 110 of the tooth during surgery in a state where the real root canal is no longer visible. 40 (see FIG. 4) and remain in the “browsing mode” (step S504). Therefore, an important index called root canal position information can be displayed superimposed on and followed by a real image of the surgical site.

  Next, the predicted image data acquisition unit 16 of the touch panel type PC 10 uses a predetermined (based on imaging data of the current tooth of the patient obtained by the intraoperative camera 30 to capture the “predicted image data” 130 used in the “AI mode” ( A two-dimensional (or three-dimensional) still image is statistically predicted and acquired from past treatment data stored in a server or storage medium (including on the cloud) (step S505). Specifically, the “predicted image data” 130 is obtained by changing the image by HSV inverted image, gray scale, edge image, etc. by image processing, performing automatic extraction characterization from the image, and further by texture analysis of the entire image More than 200 extracted feature quantities are integrated and an approximation based on matching with past data or the like is calculated and acquired using SVM (Support Vector Machine) which is one method of machine learning / AI. Although it may be based on imaging data (including a CT image) of a patient's teeth taken in advance before surgery, the camera 30 captures the current teeth of the patient during surgery and is based on the imaging data. In this case, there is a significant merit that the pre-shooting burden is reduced.

  And touch panel type PC10 is a case where the root canal which was initially visible by the practitioner is not visible from various causes during the operation (for example, covered with gingiva) in the “viewing mode”. In addition, the “real image data” 120 of the root canal in the state where the hole is visible is continuously superimposed on the image 110 of the real image of the tooth in the state where the real root canal is not visible, and the HMD is transparently transmitted. The tracking angle determination / mode switching unit 17 of the touch panel type PC 10 displays the validity of the “real image data” 120 when the tracking angle exceeds an allowable value (for example, an angle of 10 degrees on the Z axis). Therefore, the “viewing mode” is switched to the “AI mode”, and the “predictive image data” 130 corresponding to the follow-up angle is obtained by the AI mode realizing unit 18 of the touch panel PC 10 during the operation. When the real image 110 is superimposed and followed and transparently displayed on the HMD device 40 (see FIG. 5), the “AI mode” is selected. Conversely, when the follow angle returns to the allowable value, the “AI mode” is selected. The “viewing mode” is switched to “real image data” again on the HMD device 40 (see FIG. 3) (step S506). For example, the touch panel PC 10 calculates a change amount with respect to the reference position, and determines whether the follow-up angle (particularly the z axis) is within an allowable value based on the calculated change amount. If the coordinates of the reference position are (x, y, z) and the change amount is (Δx, Δy, Δz), it is determined whether or not the value is within an allowable value (z ′). It should be noted that a plurality of “predicted image data” 130 may be prepared according to the tracking angle, and may be switched to an appropriate one according to the tracking angle. Further, the “predicted image data” 130 may be simply displaying a similar image from past data. In this case, a plurality of types of data may be combined or a plurality of types of data may be displayed in parallel. It may be.

  The grid display unit 19 of the touch panel PC 10 may display the HMD device 40 with a grid (virtual lattice lines) added (step S507). For example, a green line is displayed two-dimensionally on the entire screen in advance, every third line is a thick line, and one basic line is displayed as a red line. The position may be corrected by a finger operation on the touch panel of a predetermined display unit of the device itself, or the pitch / size adjustment and the grid number adjustment may be performed in accordance with the enlargement / reduction of the image. A virtual grid can be displayed in front of the practitioner's eyes, making it easy to check parallelism when using the grid as a guide and cutting teeth.

According to the present embodiment described above, it is possible to improve the efficiency of treatment and raise the experience value of the practitioner. Even a person who has little diagnostic experience (for example, a new oral surgeon) can perform accurate treatment and surgery at a level close to (or higher than) an experienced practitioner.
In addition, since a very small fiberscope camera is installed on the dental instrument and the intraoral image captured is displayed on the HMD device, the practitioner can see the work field close-up and improve the efficiency of the treatment. be able to.
In addition, since the HMD is used, an image is presented in a normal line-of-sight direction, and treatment can be performed in a natural posture without changing the posture in particular to look at the display device like a conventional intraoral camera. .
In addition, since the practitioner uses a portable terminal such as a smartphone or a tablet terminal, an operation can be performed in a pocket of a white robe, and an enlarged display can be performed by digital zoom instead of optical zoom.
In addition, using current imaging data of the patient's teeth, two-dimensional images are statistically predicted and displayed from past data, so even practitioners with little experience can draw out a huge number of previous cases and be effective. By utilizing this, the experience value of the practitioner can be substantially raised, and treatment with a high experience value can be performed.
In addition, when the root canal that was initially visible to the practitioner cannot be visually recognized due to various causes (for example, covered with gingiva) during the operation, when the follow-up angle exceeds the allowable value, the AI is changed from the viewing mode to the AI. The mode can be switched so that the most realistic and accurate image of the root canal that was initially visible can be used to the maximum, and the root canal position can be predicted and displayed only when necessary. .
In addition, since the practitioner can perform tracking marking by finger operation from the touch panel type PC in the AI mode, tracking (tracking) can be realized easily and reliably.

  Each of the above-described embodiments is a preferred embodiment of the present invention, and various modifications can be made without departing from the scope of the present invention. For example, a process for realizing the function of each device or the like may be performed by causing each device or the like to read and execute a program for realizing the function of each device or the like. Further, the program is transmitted to another computer system by a transmission wave via a CD-ROM or magneto-optical disk as a computer-readable recording medium, or via the Internet or a telephone line as a transmission medium. Also good. Also, some systems may be realized through human actions.

S Intraoral treatment support system 10 Touch panel PC
DESCRIPTION OF SYMBOLS 20 Dental instrument 30 Camera 31 Cable 40 HMD apparatus 41 Cable 50 Portable terminal 110 Real image 120 Real image data 130 Predictive image data

The intraoral treatment support system of the present invention includes:
A touch panel PC, a dental instrument, a camera, an HMD device, and a mobile terminal are provided, and the touch panel PC is wired to a camera provided in the dental instrument, and the HMD device is Wired connection with the portable terminal, the touch panel PC is an intraoral treatment support system wirelessly connected with the portable terminal,
The touch panel PC is
Acquiring real image data, which is a two-dimensional still image taken by the camera, including a root canal in the oral cavity of a patient to be treated with the dental instrument;
An image of a real image including a root canal in the oral cavity of the patient photographed by the camera is displayed on the HMD device via the portable terminal,
It receives an operation of the practitioner on the touch panel of the automatic recognition or by the own apparatus, with respect to the real image data to be superimposed on the image of the real image is displayed on the own device, setting the marker for the follow-up Can
The real image data is displayed on the HMD device via the mobile terminal by superimposing and following the real image image,
Even if the root canal that was initially visible to the practitioner is not visible during the operation, the real image data is superimposed and followed by the image of the real image in a state where the root canal is no longer visible. Display on the HMD device via the portable terminal;
Two-dimensional or three-dimensional matching similar to the imaging data of teeth in past cases stored in a predetermined storage medium based on the imaging data of the current teeth of the patient imaged by the camera during or before surgery Select and obtain predicted image data of dimensions,
When superimposing and following the real image data on the real image in a state where the root canal cannot be seen, the following angle determined by the angle formed by the patient's teeth and the camera exceeded an allowable value. In the case, the predicted image data is displayed by superimposing and following the real image,
When the follow-up angle returns to within the allowable value range, the real image data is displayed again by superimposing and following the real image in a state where the root canal is no longer visible,
The portable terminal is
The same image as each image displayed on the HMD device is displayed, and the operator's operation on the terminal is accepted, and the images are enlarged or reduced and displayed.

The intraoral treatment support method of the present invention comprises:
A touch panel type PC, a dental instrument, a camera, a HMD device, I Oh in the mobile terminal, the touch panel-type PC is camera and wired connections provided on the dental instrument, the HMD The device is a wired connection with the mobile terminal, and the touch panel PC is an intraoral treatment support method using a wireless connection with the mobile terminal,
The touch panel PC is
Obtaining real image data, which is a two-dimensional still image taken by the camera and including a root canal in the oral cavity of a patient to be treated with the dental instrument;
Displaying an image of a real image including a root canal in the oral cavity of the patient photographed by the camera on the HMD device via the portable terminal;
The step of accepting an operation practitioner on the touch panel of the automatic recognition or by the own apparatus, with respect to the real image data to be superimposed on the image of the real image is displayed on the own device, sets a marker for the follow-up And displaying the real image data on the HMD device via the portable terminal by superimposing and following the real image data on the real image, and
Even if the root canal that was initially visible to the practitioner is not visible during the operation, the real image data is superimposed and followed by the image of the real image in a state where the root canal is no longer visible. Displaying on the HMD device via the portable terminal;
Two-dimensional or three-dimensional matching similar to the imaging data of teeth in past cases stored in a predetermined storage medium based on the imaging data of the current teeth of the patient imaged by the camera during or before surgery Selecting and obtaining predicted image data of a dimension;
When superimposing and following the real image data on the real image in a state where the root canal cannot be seen, the following angle determined by the angle formed by the patient's teeth and the camera exceeded an allowable value. In the case, the predicted image data is displayed by superimposing and following the image of the real image, and
A step of superimposing and following the real image data on the real image in a state where the root canal is no longer visible when the follow angle returns to within the allowable range. And
The mobile terminal is
Displaying the same image as each image displayed on the HMD device, receiving an operation of a practitioner on the own terminal, and displaying each image enlarged or reduced.

The intraoral treatment support system of the present invention includes:
A touch panel PC, a dental instrument, a camera, an HMD device, and a mobile terminal are provided, and the touch panel PC is wired to a camera provided in the dental instrument, and the HMD device is Wired connection with the portable terminal, the touch panel PC is an intraoral treatment support system wirelessly connected with the portable terminal,
The touch panel PC is
The camera shot, obtains the real image data is a two-dimensional still image including a root canal in the oral cavity of the patient to be treated using the dental instrument,
An image of a real image including a root canal in the oral cavity of the patient photographed by the camera is displayed on the HMD device via the portable terminal,
It is possible to set a marker for tracking the real image data displayed on the self-apparatus superimposed on the real-image image by receiving an operator's operation on the touch panel of the self-apparatus by automatic recognition. Can
The real image data is displayed on the HMD device via the mobile terminal by superimposing and following the real image image,
Even if the root canal that was initially visible to the practitioner is not visible during the operation, the real image data is superimposed and followed by the image of the real image in a state where the root canal is no longer visible. Display on the HMD device via the portable terminal;
Two-dimensional or three-dimensional matching similar to the imaging data of teeth in past cases stored in a predetermined storage medium based on the imaging data of the current teeth of the patient imaged by the camera during or before surgery Select and obtain predicted image data of dimensions,
When superimposing and following the real image data on the real image in a state where the root canal cannot be seen, the following angle determined by the angle formed by the patient's teeth and the camera exceeded an allowable value. In the case, the predicted image data is displayed by superimposing and following the real image,
When the follow-up angle returns to within the allowable value range, the real image data is displayed again by superimposing and following the real image in a state where the root canal is no longer visible,
The portable terminal is
The same image as each image displayed on the HMD device is displayed, and the operator's operation on the terminal is accepted, and the images are enlarged or reduced and displayed.

The intraoral treatment support method of the present invention comprises:
A touch panel type PC, a dental instrument, a camera, an HMD device, and a portable terminal, wherein the touch panel type PC is wiredly connected to a camera provided in the dental instrument, and the HMD device Is a wired connection with the mobile terminal, and the touch panel PC is an intraoral treatment support method using a wireless connection with the mobile terminal,
The touch panel PC is
The method comprising the camera shot, obtains the real image data is a two-dimensional still image including a root canal in the oral cavity of the patient to be treated using the dental instrument,
Displaying an image of a real image including a root canal in the oral cavity of the patient photographed by the camera on the HMD device via the portable terminal;
A step of accepting a surgeon's operation on the touch panel of the own device by automatic recognition and setting a marker for following the real image data displayed on the own device superimposed on the real image. Superimposing and following the real image data on the real image and displaying the real image data on the HMD device via the portable terminal; and
Even if the root canal that was initially visible to the practitioner is not visible during the operation, the real image data is superimposed and followed by the image of the real image in a state where the root canal is no longer visible. Displaying on the HMD device via the portable terminal;
Two-dimensional or three-dimensional matching similar to the imaging data of teeth in past cases stored in a predetermined storage medium based on the imaging data of the current teeth of the patient imaged by the camera during or before surgery Selecting and obtaining predicted image data of a dimension;
When superimposing and following the real image data on the real image in a state where the root canal cannot be seen, the following angle determined by the angle formed by the patient's teeth and the camera exceeded an allowable value. In the case, the predicted image data is displayed by superimposing and following the image of the real image, and
A step of superimposing and following the real image data on the real image in a state where the root canal is no longer visible when the follow angle returns to within the allowable range. And
The mobile terminal is
Displaying the same image as each image displayed on the HMD device, receiving an operation of a practitioner on the own terminal, and displaying each image enlarged or reduced.

Claims (2)

A touch panel PC, a dental instrument, a camera, an HMD device, and a mobile terminal are provided, and the touch panel PC is wired to a camera provided in the dental instrument, and the HMD device is Wired connection with the portable terminal, the touch panel PC is an intraoral treatment support system wirelessly connected with the portable terminal,
The touch panel PC is
Acquiring real image data, which is a two-dimensional still image taken by the camera, including a root canal in the oral cavity of a patient to be treated with the dental instrument;
An image of a real image including a root canal in the oral cavity of the patient photographed by the camera is displayed on the HMD device via the portable terminal,
Accepting the operator's operation on the automatic recognition or the touch panel of the own device, it is possible to set a marker for tracking the real image data displayed superimposed on the real image,
The real image data is displayed on the HMD device via the mobile terminal by superimposing and following the real image image,
Even if the root canal that was initially visible to the practitioner is not visible during the operation, the real image data is superimposed and followed by the image of the real image in a state where the root canal is no longer visible. Display on the HMD device via the portable terminal;
Based on the imaging data of the patient's current teeth imaged by the camera, the prediction data of 2D or 3D is statistically predicted and acquired from the past data accumulated in a predetermined storage medium,
When the real image data is superimposed on and followed by the real image in a state where the root canal is not visible, if the follow angle exceeds an allowable value, the predicted image data is converted to the real image. Overlay and follow the image to display,
When the follow-up angle returns to within the allowable value range, the real image data is displayed again by superimposing and following the real image in a state where the root canal is no longer visible,
The portable terminal is
An intraoral treatment support system that displays each image displayed on the HMD device in the same manner, accepts an operation of a practitioner on the terminal, and displays each image enlarged or reduced. A touch panel type PC, a dental instrument, a camera, an HMD device, and a portable terminal, wherein the touch panel type PC is wired to a camera provided in the dental instrument, and the HMD device Is a wired connection with the mobile terminal, and the touch panel PC is an intraoral treatment support method using a wireless connection with the mobile terminal,
The touch panel PC is
Obtaining real image data, which is a two-dimensional still image taken by the camera and including a root canal in the oral cavity of a patient to be treated with the dental instrument;
Displaying an image of a real image including a root canal in the oral cavity of the patient photographed by the camera on the HMD device via the portable terminal;
A step of setting a marker for following the real image data displayed by being superimposed on the real image image by receiving an operator's operation on the touch panel of the device by automatic recognition or the reality Displaying image data on the HMD device via the portable terminal by superimposing and following the image data on the real image; and
Even if the root canal that was initially visible to the practitioner is not visible during the operation, the real image data is superimposed and followed by the image of the real image in a state where the root canal is no longer visible. Displaying on the HMD device via the portable terminal;
A step of statistically predicting and obtaining predicted two-dimensional or three-dimensional image data from past data stored in a predetermined storage medium based on imaging data of the patient's current teeth imaged by the camera When,
When the real image data is superimposed on and followed by the real image in a state where the root canal is not visible, if the follow angle exceeds an allowable value, the predicted image data is converted to the real image. Superimposing and following the image for display;
A step of superimposing and following the real image data on the real image in a state where the root canal is no longer visible when the follow angle returns to within the allowable range. And
The mobile terminal is
An intraoral treatment support method comprising: displaying each image displayed on the HMD device in the same manner, receiving an operation performed by a practitioner on the own terminal, and displaying each image enlarged or reduced.