JP7109047B2 - INFORMATION DISPLAY DEVICE, INFORMATION DISPLAY SYSTEM, INFORMATION DISPLAY METHOD, AND INFORMATION DISPLAY PROGRAM - Google Patents

Description

The present invention relates to an information display device, an information display system, an information display method, and an information display program. The present invention relates to a display system, an information display method, and an information display program.

Conventionally, as in Patent Document 1, there is known a head-mounted display device that can be mounted on the head of an operator and has a display that comes within the field of view of the operator when mounted on the head. . Information necessary for dental treatment is displayed on this display, and the necessary information can be read simply by moving the operator's line of sight to the display.

JP-A-2000-060876 (claim 1, paragraph [0009])

However, with the head-mounted display device disclosed in Patent Literature 1, it is necessary to proceed with treatment while alternately viewing the oral cavity of the patient undergoing dental treatment and the display. For this reason, there is a possibility that the operator may misidentify the tooth to be treated due to his/her prejudice or mistakenly extract an impacted tooth that cannot be visually observed.

The present invention has been made in view of the circumstances described above, and an object of the present invention is to provide an information display device, an information display system, an information display method, and an information display program that can reduce mistakes in dental treatment. .

In order to solve the above problems, there is provided an information display device for displaying information related to treatment to an operator during dental treatment, the information display device comprising a storage unit, an imaging unit, a display unit, and a control unit. , wherein the storage unit stores an X-ray row-of-teeth image obtained by imaging at least a part of the patient's row of teeth using X-rays, and the control unit is configured to perform treatment imaged by the imaging unit. an X-ray row of teeth image addition unit that extracts the row of teeth of a patient inside and displays the X-ray row of teeth image on the display unit in a state of being superimposed on the row of teeth ; A marker made of a material that does not transmit X-rays is attached to one of the positions, and the X-ray row of teeth image adding unit extracts the marker appearing in the image captured by the imaging unit, The X-ray row of teeth image is positioned so as to be superimposed on the row of teeth by performing matching with the markers appearing on the row of teeth X-ray image, and the storage unit stores the image of the row of teeth with the X-ray row of teeth image for each patient. The control unit further includes a patient recognition unit that recognizes the patient under treatment by reading the patient identification information displayed on the marker captured by the imaging unit. The X-ray row-of-teeth image addition unit displays the X-ray row-of-teeth image associated with the patient recognized by the patient recognition unit in a state of being superimposed on the row of teeth.

The present invention can reduce mistakes in dental treatment.

1 is a schematic configuration diagram of an information display system according to a first embodiment of the present invention; FIG. 1 is an external perspective view of an X-ray CT (Computer Tomography) apparatus according to a first embodiment of the present invention; FIG. It is a figure which shows the imaging|photography situation by an X-ray CT apparatus. 1 is a schematic diagram of an X-ray CT image; FIG. 3 is a block diagram of a management terminal according to the first embodiment of the present invention; FIG. FIG. 4 is a schematic diagram of an X-ray CT image after treatment simulation. It is a figure which shows the condition which is performing treatment. 1 is a block diagram of an information display device according to a first embodiment of the present invention; FIG. It is an illustration of an overlay display of an X-ray CT image. 4 is a flow chart showing the operation of the information display system according to the first embodiment of the present invention; 4 is a flow chart showing the operation of the information display system according to the first embodiment of the present invention; FIG. 2 is a schematic configuration diagram of an information display system according to a second embodiment of the present invention; FIG. It is a figure which shows the imaging|photography situation by an X-ray CT apparatus. 1 is a schematic diagram of an X-ray CT image; FIG. FIG. 8 is a block diagram of a management terminal according to the second embodiment of the present invention; It is a block diagram of an information display device according to a second embodiment of the present invention. It is an illustration of an overlay display of an X-ray CT image. 8 is a flow chart showing the operation of the information display system according to the second embodiment of the present invention; 8 is a flow chart showing the operation of the information display system according to the second embodiment of the present invention;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Each drawing is merely a schematic representation to the extent that the present invention can be fully understood. Accordingly, the present invention is not limited to the illustrated examples only. In addition, in each figure, the same code|symbol is attached|subjected about a common component and a similar component, and those overlapping description is abbreviate|omitted.

[First embodiment]
An information display system 1 according to a first embodiment of the present invention will be described in detail with reference to FIGS. 1 to 11. FIG.

The information display system 1 utilizes augmented reality (AR) technology to display an X-ray image of the oral cavity of a patient using X-rays (particularly, an X-ray row of teeth image of the row of teeth). is overlaid on the patient during treatment. The overlay display here is performed via the information display device 4 worn by the operator during treatment. The X-ray image may be a three-dimensional image or a two-dimensional image, or may be an image obtained by photographing a part of the dentition (for example, a specific tooth).

The information display system 1 can also overlay the content of the treatment simulation in addition to the X-ray image. The treatment simulation here is to determine, for example, whether or not the size of the bolt in implant treatment is appropriate. The content of the treatment simulation may be incorporated in the X-ray image, or may be an image (treatment simulation image) separate from the X-ray image.

<Overall composition>
The information display system 1 includes, for example, as shown in FIG. 1, an X-ray CT (Computer Tomography) device 2 for imaging a patient's oral cavity (especially, a row of teeth), a management terminal 3 for managing patient information, and an information display device 4 for overlay-displaying an X-ray image.

<X-ray CT device>
The X-ray CT apparatus 2 includes, for example, a main unit 21, an elevating unit 22, and a scanner 23, as shown in FIG.

The main body part 21 is a part that becomes the main body of the X-ray CT apparatus 2 . The main body 21 includes a base 21a, a pillar 21b vertically provided at the rear of the base 21a, a patient chair 21c provided at the front of the base 21a, and a head fixing mechanism for fixing the patient's head. 21d. In addition, illustration of the patient is omitted in FIG.

The head fixing mechanism 21d has a structure in which sliding motion and rotational motion are combined, and is configured as a link mechanism having links 21da, 21db, and 21dc. A patient fixture 21dd for fixing the patient's head is provided at the tip of the link 21dc. FIG. 3 shows a state in which the patient's head is fixed by the head fixing mechanism 21d.

The elevating section 22 is a component that elevates the column section 21b. The elevating unit 22 includes a scanner support unit 22a that supports the scanner 23, a positioning unit 22b that positions the scanner 23 in the oral cavity of the patient, a visible light camera 22c that photographs the patient's face using visible light, It has

A rotation mechanism for rotating the scanner 23 is accommodated inside the scanner support portion 22a, and rotates the scanner 23 with the vertical direction as a rotation axis.

The positioning portion 22b is a portion that irradiates an alignment beam (for example, a laser beam) forward. The beam emitted from the positioning unit 22b is parallel to the plane (horizontal plane) formed by the front/rear and left/right line segments, and is used for alignment of the scanner 23 in the vertical direction.

The visible light camera 22c is a component that photographs the patient's face using visible light. The visible light camera 22c, for example, photographs the patient's face from the front and creates a facial image Pa of the patient's facial appearance (normal appearance) viewed from the front. A facial image Pa captured by the visible light camera 22c is used to identify the patient. It should be noted that the facial image Pa only needs to show the shape and arrangement of the patient's eyes, nose, mouth, ears, etc., and the direction and timing of imaging the patient are not particularly limited.

The scanner 23 is a component that takes an image of the patient's oral cavity (in particular, the maxillary and mandibular dentitions) using X-rays. The scanner 23 has an X-ray source and X-ray imaging means inside, and can perform imaging while rotating. By combining image data captured while the scanner 23 is rotating, an X-ray CT image Qa, which is three-dimensional volume data of the maxillary and mandibular dentitions, is obtained (see FIG. 4). This X-ray CT image Qa is composed of a maxillary image Qaa and a mandibular image Qab.

<Management terminal>
The management terminal 3 is a PC (Personal Computer) for management. For example, as shown in FIG. , is equipped with

The input unit 31 is, for example, a keyboard and a mouse, and various information is input by the operator. The display unit 32 is, for example, a liquid crystal display, and displays various information to the operator. The communication unit 33 is, for example, a wired or wireless communication module, and communicates various information with the X-ray CT apparatus 2 and the information display apparatus 4 (see FIG. 1).

The storage unit 34 is, for example, an HDD (hard disk drive) or memory, and stores a patient information DB 34a. The patient information DB 34a stores patient-related information (including information necessary for overlay display of the X-ray CT image Qa). In the patient information DB 34a, for example, information such as the patient's name, age, and sex is registered in association with patient identification information (for example, a unique number). In addition, the patient's facial image Pa, feature points of the facial image, X-ray CT image Qa, feature points of the X-ray CT image, etc. are registered in association with the patient identification information. The patient information DB 34a may register a flag indicating that the treatment simulation has been performed and the contents of the treatment simulation. Also, part or all of this information may be stored in a device other than the management terminal 3, and the management terminal 3 may acquire the information via the network as necessary.

The control unit 35 is, for example, a CPU (Central Processing Unit), and implements a patient information management unit 35a and a treatment simulation processing unit 35b by program execution processing. Programs for realizing these functions are stored in the storage unit 34, for example. Note that these programs may be stored in a device other than the management terminal 3, and the management terminal 3 may obtain the programs via a network as necessary.

The patient information management unit 35a performs processing related to the patient information DB 34a. The patient information management unit 35a registers, for example, patient information input by the operator in the patient information DB 34a. Further, the patient information management unit 35a receives the facial image Pa and the X-ray CT image Qa from the X-ray CT apparatus 2, and registers the facial image Pa and the X-ray CT image Qa in the patient information DB 34a in association with the patient identification information. do. Further, the patient information management unit 35a extracts feature points from the facial image Pa and the X-ray CT image Qa, respectively, and associates the feature points of the facial image Pa and the feature points of the X-ray CT image Qa with the patient identification information. Register in the information DB 34a.

The treatment simulation processing unit 35b realizes simulation of dental treatment. The simulation of dental treatment, for example, determines whether or not the size of the bolt in implant treatment is appropriate, and is performed using the X-ray CT image Qa by the operator's operation. The treatment simulation processing unit 35b registers the contents of the treatment simulation in the patient information DB 34a (in particular, the X-ray CT image Qa). FIG. 6 shows a state in which the treatment simulation image Ra is incorporated into the X-ray CT image Qa. The treatment simulation image Ra here is an artificial tooth (dental implant).

<Information display device>
The information display device 4 superimposes (overlays) the X-ray CT image Qa on the patient being treated. The information display device 4 is not particularly limited as long as it can overlay-display the X-ray CT image Qa using augmented reality technology. The information display device 4 is, for example, a head mounted display (HMD: Head Mounted Display) mounted on the head of the operator or a mobile terminal operated by hand, as shown in FIG. Here, the information display device 4 is assumed to be a head-mounted display of a system (video see-through system) that synthesizes an X-ray CT image Qa with an image at a signal level using video capture.

The information display device 4 includes, for example, an imaging unit 41, a display unit 42, a communication unit 43, a storage unit 44, and a control unit 45, as shown in FIG. Note that the control unit 45 is an example of a "control unit" in the scope of claims.

The imaging unit 41 is a camera including, for example, a CMOS (Complementary metal-oxide-semiconductor) image sensor or a CCD (Charged-coupled devices) image sensor. The photographing unit 41 is installed so as to photograph a video (including an image, the same shall apply hereinafter) in the direction of the operator's line of sight while being worn by the operator.

The display unit 42 is, for example, a liquid crystal display or an organic EL display. The display unit 42 is arranged so as to block the field of view of the operator when worn by the operator. The image captured by the imaging unit 41 is displayed on the display unit 42 . The display unit 42 displays an X-ray CT image Qa overlaid on the image of the patient captured by the imaging unit 41 (see FIG. 9). The overlay display method of the X-ray CT image Qa is not particularly limited as long as the image of the patient captured by the imaging unit 41 and the X-ray CT image Qa are displayed in a distinguishable manner. For example, the display unit 42 overlay-displays the X-ray CT image Qa in a translucent state.

The communication unit 43 is, for example, a wired or wireless communication module, and communicates various information with the management terminal 3 (see FIG. 1).

The storage unit 44 is, for example, a memory, and stores an additional information DB 44a. The additional information DB 44a stores information necessary for overlay display of the X-ray CT image Qa. In the additional information DB 44a, for example, information of patients to be treated among the information stored in the patient information DB 34a is registered.

The control unit 45 is, for example, a CPU (Central Processing Unit), and implements a patient recognition unit 45a, an X-ray row-of-teeth image acquiring unit 45b, and an X-ray row-of-teeth image adding unit 45c by program execution processing. Programs for realizing these functions are stored in the storage unit 44, for example. Note that these programs may be stored in a device other than the information display device 4, and the information display device 4 may obtain the programs via a network as necessary.

The patient recognition unit 45a extracts feature points of the patient's facial features from the facial image Pb (for example, an image at the start of treatment and is part of the video) captured by the imaging unit 41, and uses the information as a key to obtain patient information. The corresponding patient is determined by searching the DB 34a.

The X-ray row of teeth image acquisition unit 45b acquires the X-ray CT image Qa associated with the identified patient and the characteristic points of the X-ray CT image Qa from the management terminal 3, and registers the acquired information in the additional information DB 44a. do.

The X-ray row-of-teeth image adding unit 45c extracts characteristic points of the patient's row-of-teeth shape from the facial image Pc captured by the imaging unit 41 (for example, an image during treatment and part of the video). The range for extracting feature points is not particularly limited. For example, feature points of all teeth may be extracted, or feature points of several teeth may be extracted. When displaying either the upper jaw or the lower jaw superimposed on the patient, the X-ray row of teeth image addition unit 45c extracts characteristic points of the row of teeth shape of the upper or lower jaw to be displayed. Moreover, when both the upper jaw and the lower jaw are superimposed and displayed on the patient, the X-ray row of teeth image addition unit 45c preferably extracts feature points from each of the upper jaw and the lower jaw. Although the feature point of one tooth may be extracted, it is desirable to extract the feature point of a plurality of teeth in consideration of the accuracy of subsequent matching processing.

Further, the X-ray row of teeth image addition unit 45c matches the X-ray CT image Qa with the feature points of the row of teeth appearing in the facial image Pc, and creates the X-ray CT image Qa on the row of teeth of the patient according to the matching result. Overlay display (see FIG. 9). The X-ray row-of-teeth image addition unit 45c may perform overlay display so that the row of teeth reflected in the facial image Pc and the X-ray CT image Qa can be distinguished. For example, the X-ray CT image Qa is displayed in a translucent state. During overlay display, the X-ray row of teeth image adding unit 45c adjusts the positions of the maxilla side image Qaa and the mandible side image Qab in the X-ray CT image Qa according to the opening of the mouth in the facial image Pc. is good. The X-ray row-of-teeth image adding unit 45c may further overlay the treatment simulation image Ra (see FIG. 6) on the row of teeth shown in the facial image Pc. Note that the overlay-displayed X-ray CT image Qa may be switched between display and non-display by the operator's operation, for example.

Although the management terminal 3 and the information display device 4 are described as separate devices in this embodiment, the information display device 4 may include a part or all of the functions of the management terminal 3 . That is, the information display device 4 may have functions such as the patient information DB 34a, the patient information management section 35a, and the treatment simulation processing section 35b.

<Operation of the information display system according to the first embodiment>
Next, the operation of the information display system 1 according to the first embodiment of the present invention will be described in detail with reference to FIGS. 10 to 11 (see FIGS. 1 to 9 as appropriate). Here, the operation of the information display system 1 will be described by dividing it into "pre-processing (see FIG. 10)" and "treatment processing (see FIG. 11)".

≪Pretreatment≫
<S1: Patient information registration>
First, the operator operates the management terminal 3 to register patient information (S1). In this process, for example, information such as the patient's name, age, and sex is input, and patient identification information (for example, a unique number) is associated with this information and registered in the patient information DB 34a of the management terminal 3.

<S2: Face photography>
Next, the operator guides the patient to the X-ray CT apparatus 2 and uses the visible light camera 22c to photograph the patient's facial features (S2). The patient's facial image Pa captured by this process is sent to the management terminal 3 .

<S3: Extraction of Feature Points from Facial Image>
Next, the patient information management unit 35a of the management terminal 3 extracts feature points of the facial image Pa of the patient captured by the visible light camera 22c (S3).

<S4: X-ray CT imaging>
Next, the operator uses the scanner 23 to photograph the patient's oral cavity (S4). The X-ray CT image Qa captured by this process is sent to the management terminal 3 .

<S5: Extraction of feature points from X-ray CT image>
Next, the patient information management unit 35a of the management terminal 3 extracts feature points of the patient's intraoral X-ray CT image Qa (S5).

<S6: Data linking>
Next, the patient information management unit 35a of the management terminal 3 associates the facial image Pa, the feature points of the facial image Pa, the X-ray CT image Qa, and the feature points of the X-ray CT image Qa with the patient identification information. Stored in the information DB 34a. That is, here, the information on the facial image Pa and the information on the X-ray CT image Qa are linked (S6).

<S7, S8: treatment simulation>
Next, the operator determines whether or not it is necessary to perform a treatment simulation, and if necessary, performs simulation using the X-ray CT image Qa taken (S7, S8). Thereby, the treatment simulation image Ra is incorporated into the X-ray CT image Qa. When the treatment simulation is performed, the feature points of the X-ray CT image Qa after the treatment simulation may be extracted again.

The X-ray CT image Qa and the treatment simulation image Ra may be managed as separate information without incorporating the treatment simulation image Ra into the X-ray CT image Qa. In that case, the treatment simulation image Ra is managed in association with the patient identification information.

Here, the case where X-ray CT imaging is performed after imaging the patient's facial features is shown, but the order of these processes is not particularly limited. For example, facial imaging may be performed after X-ray CT imaging, or facial imaging and X-ray CT imaging may be performed simultaneously. Also, the timing for extracting the feature points of the facial image Pa and the X-ray CT image Qa is not limited to those shown here.

≪Treatment during treatment≫
<S11: Patient Confirmation>
Following the pretreatment shown in FIG. 10, the operator guides the patient to dental medical equipment (eg, an examination table) (not shown). Then, the operator looks at the patient's face while wearing the information display device 4 (device). The image capturing unit 41 of the information display device 4 captures an image of the patient's facial features in the line-of-sight direction of the operator (S11).

<S12: Patient discrimination>
Next, the patient recognition unit 45a of the information display device 4 extracts feature points of the patient's facial features from the facial image Pb photographed by the photographing unit 41 of the information display device 4, and uses the information as a key to store the patient information DB 34a. The relevant patient is determined by searching (S12).

The facial image Pb photographed by the photographing unit 41 of the information display device 4 may be transmitted to the management terminal 3, and the management terminal 3 may extract the feature points of the patient's facial features.

<S13: Calling up the X-ray CT image of the corresponding patient>
Next, the X-ray row-of-teeth image acquisition unit 45b of the information display device 4 acquires the X-ray CT image Qa associated with the identified patient from the management terminal 3 and the characteristic points of the X-ray CT image Qa (S13).

<S14: Extraction of characteristic points of tooth row shape>
Next, the operator asks the patient to open his/her mouth, and the imaging unit 41 of the information display device 4 photographs the patient's facial image Pc (including the image of the row of teeth) in the direction of the operator's line of sight. The X-ray row-of-teeth image addition unit 45c of the information display device 4 extracts feature points from the row-of-teeth shape shown in the facial image Pc (S14).

<S15: Matching Feature Points>
Next, the X-ray row of teeth image adding unit 45c of the information display device 4 matches the feature points of the X-ray CT image Qa with the feature points of the patient's row of teeth shown in the facial image Pc (S15).

<S16: Overlay display of X-ray CT image>
Next, the X-ray row of teeth image adding unit 45c of the information display device 4 overlay-displays the X-ray CT image Qa on the row of teeth of the patient according to the matching result (S16).

The information display system 1 according to the first embodiment of the present invention configured as described above has the following effects.
In other words, the information display system 1 uses augmented reality technology to display the X-ray CT image Qa superimposed on the dentition shown in the image (facial image Pc) of the intraoral area of the patient. Therefore, it is possible to reduce mistakes in dental treatment, such as erroneous recognition of a tooth to be treated due to an operator's prejudice and erroneous extraction of an impacted tooth that cannot be visually observed. Also, even when the operator confirms the X-ray CT image Qa, the operator does not need to take his/her line of sight out of the patient's oral cavity, so that the operator can instantly grasp the correspondence between the teeth actually viewed and the teeth shown in the X-ray CT image Qa. can do. Therefore, the burden on the operator can be reduced.

Further, in the information display system 1, the feature points of the patient's dentition appearing in the video (facial image Pc) taken during treatment are matched with the feature points of the X-ray CT image Qa taken in advance, so that the dentition is displayed. The X-ray CT image Qa is superimposed on and displayed. Therefore, advance preparation is simple.

Further, in the information display system 1, when one of the upper jaw and the lower jaw is displayed superimposed on the patient, the X-ray row-of-teeth image addition unit 45c extracts characteristic points of the tooth row shape of the upper jaw or the lower jaw to be displayed. Moreover, when displaying both the upper jaw and the lower jaw superimposed on the patient, the X-ray row-of-teeth image addition unit 45c extracts feature points from each of the upper jaw and the lower jaw. Therefore, by matching the feature points of the row of teeth and the feature points of the X-ray CT image Qa, the X-ray CT image Qa can follow the opening and closing motion of the mouth.

In addition, in the information display system 1, the patient to be treated is identified by comparing the feature points of the patient's facial features shown in the video (facial image Pb) taken at the start of treatment with the feature points of the facial image Pa taken in advance. Identify. Then, an X-ray CT image Qa of the identified patient is acquired, and the acquired X-ray CT image Qa is displayed superimposed on the row of teeth. That is, the operator can accurately identify the patient by photographing the patient's face at the start of treatment. Therefore, the operator does not need to identify the patient, and the burden on the operator can be reduced.

[Second embodiment]
An information display system 1A according to a second embodiment of the present invention will be described in detail with reference to FIGS. 12 to 19. FIG.

In the information display system 1 according to the first embodiment, the feature points of the patient's dentition appearing in the video (facial image Pc) taken during treatment are matched with the feature points of the X-ray CT image Qa taken in advance. By doing so, the X-ray CT image Qa was superimposed on the row of teeth and displayed. In the present embodiment, an X-ray CT image Qa is captured in advance with a marker Mn attached to the patient's face, and the marker Mn appearing in the image (facial image Pc) captured during treatment and the X-ray captured in advance The X-ray CT image Qa is superimposed on the row of teeth and displayed by matching with the marker Mn appearing on the CT image Qa.

Further, in the information display system 1 according to the first embodiment, the feature points of the patient's facial features appearing in the video (facial image Pb) taken at the start of treatment are compared with the feature points of the facial image Pa taken in advance. identified patients for treatment. In this embodiment, it is also possible to specify a patient to be treated by a method similar to that of the first embodiment. The patient is identified by reading the information attached to the marker Mn appearing in the (facial image Pb).

An information display system 1A according to the second embodiment includes, for example, as shown in FIG. 3A and an information display device 4A that overlay-displays an X-ray image. Also, the patient is marked with a marker Mn. The following description focuses on the differences from the first embodiment.

<X-ray CT device>
The X-ray CT apparatus 2 does not capture the patient's facial image Pa with the visible light camera 22c, but uses the scanner 23 to capture only the patient's X-ray CT image Qa with markers Mn. FIG. 13 shows a photographing scene of the X-ray CT image Qa in the second embodiment. FIG. 14 shows an X-ray CT image Qa in the second embodiment.

The marker Mn is preferably made of a material that does not transmit X-rays, and it is desirable that the appearance (shape) of the marker Mn changes depending on the viewing direction. Therefore, the marker Mn is preferably, for example, rectangular parallelepiped rather than spherical.

The number of markers Mn and the positions where the markers Mn are fixed can be set arbitrarily. If the number of markers Mn is one, it cannot correspond to the opening and closing of the mouth (it is not known whether the mouth is open or closed). Preferably, more than one marker Mn is fixed. Moreover, it is preferable that the position where the marker Mn is fixed is a portion where the position relative to the dentition does not move (a portion where the distance is constant) when the mouth is opened and closed, and where it does not interfere with the treatment. Therefore, for example, as shown in FIG. 13, it is preferable to fix one marker Mn to each of the nose and the tip of the chin.

<Management terminal>
The management terminal 3A is a PC (Personal Computer) for management. For example, as shown in FIG. , is equipped with

The storage unit 34 is, for example, an HDD (hard disk drive) or memory, and stores a patient information DB 34aA. The patient information DB 34aA stores patient-related information (including information necessary for overlay display of the X-ray CT image Qa). In the patient information DB 34aA, for example, information such as the patient's name, age, and sex is registered in association with patient identification information (for example, a unique number). Also, the patient's X-ray CT image Qa and the like are registered in association with the patient identification information. The patient information DB 34a may register a flag indicating that the treatment simulation has been performed and the contents of the treatment simulation. Also, part or all of this information may be stored in a device other than the management terminal 3A, and the management terminal 3A may acquire the information via the network as necessary.

The control unit 35 is, for example, a CPU (Central Processing Unit), and realizes a patient information management unit 35aA and a treatment simulation processing unit 35b by program execution processing. Programs for realizing these functions are stored in the storage unit 34, for example. Note that these programs may be stored in a device other than the management terminal 3A, and the management terminal 3A may obtain the programs via a network as necessary.

The patient information management unit 35aA performs processing related to the patient information DB 34aA. The patient information management unit 35aA registers, for example, patient information input by the operator in the patient information DB 34a. Further, the patient information management unit 35aA receives the patient's X-ray CT image Qa with the marker Mn from the X-ray CT apparatus 2, and registers the X-ray CT image Qa in the patient information DB 34aA in association with the patient identification information. .

<Information display device>
The information display device 4A includes, for example, an imaging unit 41, a display unit 42, a communication unit 43, a storage unit 44, and a control unit 45, as shown in FIG. Note that the control unit 45 is an example of a "control unit" in the scope of claims.

The storage unit 44 is, for example, a memory, and stores an additional information DB 44aA. The additional information DB 44aA stores information necessary for overlay display of the X-ray CT image Qa. In the additional information DB 44aA, for example, information of patients to be treated among the information stored in the patient information DB 34aA is registered.

The control unit 45 is, for example, a CPU (Central Processing Unit), and implements a marker recognition unit 45aA, an X-ray row-of-teeth image acquiring unit 45bA, and an X-ray row-of-teeth image addition unit 45cA by program execution processing. Programs for realizing these functions are stored in the storage unit 44, for example. These programs may be stored in a device other than the information display device 4A, and the information display device 4A may obtain the programs via a network as necessary.

The marker recognition unit 45aA extracts the marker Mn from the facial image Pb (for example, the image at the start of treatment and is part of the video) captured by the imaging unit 41, and the information displayed on the marker Mn (for example, the patient identification information). Then, the marker recognition unit 45aA searches the patient information DB 34aA using the read information as a key to determine the corresponding patient.

The X-ray row of teeth image acquisition unit 45bA acquires the X-ray CT image Qa associated with the identified patient from the management terminal 3A, and registers the acquired information in the additional information DB 44aA.

The X-ray row-of-teeth image addition unit 45cA extracts the marker Mn attached to the patient from the facial image Pc captured by the imaging unit 41 (for example, an image during treatment and a part of the video). Further, the X-ray row of teeth image adding unit 45cA matches the marker Mn appearing in the X-ray CT image Qa with the marker Mn appearing in the facial image Pc, and according to the matching result, creates the X-ray CT image Qa on the patient's row of teeth. is overlaid (see FIG. 17). The X-ray row-of-teeth image addition unit 45cA may perform overlay display so that the rows of teeth reflected in the facial image Pc and the X-ray CT image Qa can be distinguished. For example, the X-ray CT image Qa is displayed in a translucent state. The X-ray row-of-teeth image adding unit 45cA may further overlay the treatment simulation image Ra (see FIG. 6) on the row of teeth shown in the facial image Pc. Note that the overlay-displayed X-ray CT image Qa may be switched between display and non-display by the operator's operation, for example.

Although the management terminal 3A and the information display device 4A are described as separate devices in this embodiment, the information display device 4A may include a part or all of the functions of the management terminal 3A. That is, the information display device 4A may have functions such as the patient information DB 34aA, the patient information management section 35aA, and the treatment simulation processing section 35b.

<Operation of the information display system according to the second embodiment>
Next, the operation of the information display system 1A according to the second embodiment of the present invention will be described in detail with reference to FIGS. 18 to 19 (see FIGS. 12 to 17 as appropriate). Here, the operation of the information display system 1A will be described by dividing it into "pre-processing (see FIG. 18)" and "treatment processing (see FIG. 19)".

≪Pretreatment≫
<T1: Patient information registration>
First, the operator operates the management terminal 3A to register patient information (T1). In this process, for example, information such as the patient's name, age, and sex is input, and patient identification information (for example, a unique number) is associated with this information and registered in the patient information DB 34aA of the management terminal 3A.

<T2: Marker fixed>
Next, the operator guides the patient to the X-ray CT apparatus 2 and fixes the marker Mn on a part of the patient's face (T2). The markers Mn are made of a material opaque to X-rays.

<T3: X-ray CT imaging>
Next, the operator uses the scanner 23 to photograph the patient's oral cavity (T3). The X-ray CT image Qa captured by this process is sent to the management terminal 3A.

<T4, T5: treatment simulation>
Next, the operator determines whether or not it is necessary to perform a treatment simulation, and if necessary, performs simulation using the X-ray CT image Qa taken (T4, T5). Thereby, the treatment simulation image Ra is incorporated into the X-ray CT image Qa.

The X-ray CT image Qa and the treatment simulation image Ra may be managed as separate information without incorporating the treatment simulation image Ra into the X-ray CT image Qa. In that case, the treatment simulation image Ra is managed in association with the patient identification information.

≪Treatment during treatment≫
<T11: Marker Recognition>
Following the pretreatment shown in FIG. 18, the operator guides the patient to dental medical equipment (eg, an examination table) (not shown). Then, the operator looks at the patient's face while wearing the information display device 4A (device). The image capturing unit 41 of the information display device 4A captures an image (facial image Pb) of the marker Mn attached to the patient's face in the line-of-sight direction of the operator. Information (for example, patient identification information) displayed on the marker Mn appearing in the image Pb is read (T11).

<T12: Calling up the X-ray CT image of the corresponding patient>
Next, the X-ray row-of-teeth image acquisition unit 45bA of the information display device 4A acquires the X-ray CT image Qa associated with the identified patient from the management terminal 3A (T12).

<T13: Marker matching>
Next, the X-ray row of teeth image addition unit 45cA of the information display device 4A matches the marker Mn of the X-ray CT image Qa with the marker Mn attached to the patient appearing in the facial image Pc taken during treatment (T13 ).

<T14: Overlay display of X-ray CT image>
Next, the X-ray row of teeth image addition unit 45cA of the information display device 4 overlay-displays the X-ray CT image Qa on the row of teeth of the patient according to the matching result (T14).

Here, information (for example, patient identification information) displayed on the marker Mn is read (T11), and the X-ray CT image Qa is acquired using the read information (T12). However, the method of identifying the patient is not limited to this, and the operator may manually input information displayed on the marker Mn and acquire the X-ray CT image Qa using the input information. In that case, the marker Mn may be recognized after reading out the X-ray CT image Qa.

The information display system 1A according to the second embodiment of the present invention configured as described above has the same effects as those of the first embodiment.
That is, the information display system 1A uses augmented reality technology to superimpose and display the X-ray CT image Qa on the dentition shown in the image (facial image Pc) of the intraoral area of the patient. Therefore, it is possible to reduce mistakes in dental treatment, such as erroneous recognition of a tooth to be treated due to an operator's prejudice and erroneous extraction of an impacted tooth that cannot be visually observed. Also, even when the operator confirms the X-ray CT image Qa, the operator does not need to take his/her line of sight out of the patient's oral cavity, so that the operator can instantly grasp the correspondence between the teeth actually viewed and the teeth shown in the X-ray CT image Qa. can do. Therefore, the burden on the operator can be reduced.

In addition, in the information display system 1A, the patient's marker Mn in the image (facial image Pc) captured during treatment is matched with the marker Mn in the X-ray CT image Qa captured in advance, so that X-rays are displayed on the dentition. The CT image Qa is superimposed and displayed. Therefore, the amount of processing can be reduced compared to the case of extracting characteristic points of the dentition and performing matching.

Further, in the information display system 1A, the patient to be treated is specified by reading the patient identification information displayed on the marker Mn appearing in the image (facial image Pb) taken at the start of treatment. Then, an X-ray CT image Qa of the identified patient is acquired, and the acquired X-ray CT image Qa is displayed superimposed on the row of teeth. That is, the operator can accurately identify the patient by photographing the marker Mn attached to the patient's face at the start of treatment. Therefore, the operator does not need to identify the patient, and the burden on the operator can be reduced.

[Modification]
Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and can be implemented with appropriate modifications.

In the information display system 1 of the first embodiment, as shown in FIG. 45 superimposed (overlay display) the X-ray CT image Qa on the patient under treatment. That is, the control unit 45 has a patient recognition unit 45a, an X-ray row-of-teeth image acquisition unit 45b, and an X-ray row-of-teeth image addition unit 45c. However, the configuration of the present invention is not limited to this. For example, the management terminal 3 (an example of the “control device”) shown in FIG. You may In other words, even if a configuration in which a management terminal 3 (control device) separate from the information display device 4 has some or all of the functions of the additional information DB 44a and the control unit 45 of the information display device 4 of the present embodiment, good. In that case, the management terminal 3 (control device) receives, for example, the facial image Pc captured by the imaging unit 41 from the information display device 4, and displays an image in which the row of teeth shown in the facial image Pc is superimposed on the X-ray CT image Qa. It is transmitted to the information display device 4 . The same applies to the information display system 1A according to the second embodiment. The management terminals 3 and 3A (control device) in this case are an example of the "control means" in the scope of claims.

Further, in each embodiment, the image to be overlaid is assumed to be the X-ray CT image Qa, but it may be a panoramic image.

In each embodiment, the information display devices 4 and 4A are assumed to be video see-through head-mounted displays, but the type of head-mounted display is not limited to this. For example, an optical see-through system in which the X-ray CT image Qa is superimposed on a transparent plate-like component (combiner) may be used. Further, it may be of a retinal projection type that scans a weak laser beam on the retina of the operator.

When the information display devices 4 and 4A are of these types, the X-ray row of teeth image adding units 45c and 45cA, for example, determine the position of the patient's row of teeth from the image captured by the imaging unit 41, as seen from the operator's line of sight. Calculation is performed, and the X-ray CT image Qa is displayed at the corresponding position on the display unit 42 . That is, in the overlay display in this case, by displaying only the X-ray CT image Qa on the display unit 42 (transparent plate-like part or retina), the operator can see the X-ray CT image Qa superimposed on the patient's dentition. I feel like there is.

1, 1A Information display system 2 X-ray CT device (X-ray imaging device)
22c visible light camera 3, 3A management terminal (control device, control means)
31 input unit 32 display unit 33 communication unit 34 storage unit 34a, 34aA patient information DB
35 control unit 35a, 35aA patient information management unit 35b treatment simulation processing unit 4, 4A information display device 41 imaging unit 42 display unit 43 communication unit 44 storage unit 44a additional information DB (X-ray row of teeth image)
45 control unit (control means)
45a patient recognition unit 45aA marker recognition unit 45b, 45bA X-ray row of teeth image acquisition unit 45c, 45cA X-ray row of teeth image addition unit Mn marker

Claims (5)

An information display device for displaying information related to treatment to an operator during dental treatment,
The information display device includes a storage unit, an imaging unit, a display unit, and a control unit,
The storage unit stores an X-ray row of teeth image obtained by photographing at least a part of the patient's row of teeth using X-rays,
The control unit extracts the row of teeth of the patient under treatment photographed by the imaging unit, and displays the X-ray row of teeth image on the display unit in a state of being superimposed on the row of teeth. have with
a marker made of a material opaque to X-rays is attached to the position of at least one of the patient's upper jaw and lower jaw;
The X-ray row-of-teeth image addition unit extracts the markers appearing in the image captured by the imaging unit, and performs matching with the markers appearing in the X-ray row-of-teeth image to convert the X-ray row-of-teeth image into the corresponding tooth image. Positioning to overlap the row,
The storage unit associates and stores the X-ray row of teeth image and patient identification information of the patient for each patient,
The control unit
further comprising a patient recognition unit that recognizes a patient under treatment by reading patient identification information displayed on the marker captured by the imaging unit;
The X-ray row of teeth image addition unit displays the X-ray row of teeth image associated with the patient recognized by the patient recognition unit in a state of being superimposed on the row of teeth.
An information display device characterized by: An information display device according to claim 1 ;
an X-ray imaging device for imaging the X-ray row of teeth image;
An information display system comprising: An information display system comprising: an X-ray imaging device for imaging an X-ray row of teeth image; an information display device for displaying information related to treatment to an operator undergoing dental treatment; and a control device,
The information display device includes an imaging unit and a display unit,
The control device comprises a storage unit and a control unit,
The storage unit stores an X-ray row of teeth image obtained by photographing at least a part of the patient's row of teeth using X-rays,
The control unit extracts the row of teeth of the patient under treatment photographed by the imaging unit, and displays the X-ray row of teeth image on the display unit in a state of being superimposed on the row of teeth. have with
a marker made of a material opaque to X-rays is attached to the position of at least one of the patient's upper jaw and lower jaw;
The X-ray row-of-teeth image addition unit extracts the markers appearing in the image captured by the imaging unit, and performs matching with the markers appearing in the X-ray row-of-teeth image to convert the X-ray row-of-teeth image into the corresponding tooth image. Perform positioning to overlap the row,
The storage unit associates and stores the X-ray row of teeth image and patient identification information of the patient for each patient,
The control unit
further comprising a patient recognition unit that recognizes a patient under treatment by reading patient identification information displayed on the marker captured by the imaging unit;
The X-ray row of teeth image addition unit displays the X-ray row of teeth image associated with the patient recognized by the patient recognition unit in a state of being superimposed on the row of teeth.
An information display system characterized by: An information display method for displaying information related to treatment to an operator during dental treatment,
an X-ray dentition image capturing step of creating an X-ray dentition image of the patient by imaging at least a portion of the patient's dentition using X-rays;
an X-ray row of teeth image storage step of correlating and storing the photographed X-ray row of teeth image and the patient identification information of the patient;
an X-ray row of teeth image display step of extracting the row of teeth appearing in an image of the face of the patient being treated and displaying the X-ray row of teeth image superimposed on the row of teeth on a display unit; death,
a marker made of a material opaque to X-rays is attached to the position of at least one of the patient's upper jaw and lower jaw;
In the X-ray row of teeth image displaying step, the markers appearing in the image captured during treatment are extracted, the X-ray row of teeth image linked to the patient identification information displayed on the marker is acquired, and the acquired positioning the X-ray row of teeth image on the row of teeth by matching with the marker reflected in the row of teeth X-ray image;
An information display method characterized by: An information display program for causing a control means to execute the information display method according to claim 4 .

Images