JP2012157692A - Oral imaging and display system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a system which forms lateral panoramic dentition images using a hand-held intraoral camera for understanding conditions of the teeth and gum accurately according to the obtained panoramic image of the lateral dentition, and for suggesting dental treatments easy to understand for a patient.SOLUTION: The system includes: a continuous image sequence-forming means for forming an image sequence by continuous imaging of the surfaces of the dental arch; a lateral dentition image-forming means for forming multiple partial dentition images with the image sequence formed by the continuous image sequence-forming means as partial dentition images and combining, starting with the image that is to become the center of the overall combined image; and a lateral dentition image-combining means for forming the entire dental arch by radially combining the multiple partial dentition images formed by the lateral dentition image-forming means, starting with the image that is to become the center of the overall combined image.

Description

  The present invention relates to a system for photographing the whole oral cavity and displaying the panorama.

Cameras for intraoral photography are used to obtain understanding and consent for patient treatment in dental caries treatment, implant treatment, etc. from high resolution, miniaturization, low price of the camera, etc., recording of oral condition etc. In addition to photographing each affected part, the entire side surface of the dentition may be photographed and used for diagnosis and treatment.
Since the dentition is complicatedly curved, the upper dentition, lower anterior teeth, upper right molar, right lower molar, left upper molar, left lower molar are divided into multiple images. In many cases, an image is formed.
The dentition image may be used as a diagnostic image to determine the number of remaining teeth, caries, and restoration, or to identify the site of gingivitis from the swelling of the gingiva in the image. When a dentition is displayed using a plurality of images, it is difficult to easily recognize which part of the oral cavity each image is.
A technique for obtaining a panoramic image with a real image in the oral cavity is disclosed in, for example, International Publication No. 2007/063980, in which an imaging device is formed on a side surface by forming a frame having a shape approximating a dental arch home approximate virtual curve It is described that a whole three-dimensional panoramic image can be obtained by photographing the entire dentition by using this and taking it in stereo format.

Certainly, it is possible to take a three-dimensional image of the dentition, but it is difficult to say that it is a simple measurement method in that it requires the formation of a frame having an archhome approximate virtual curve.
Japanese Patent Laid-Open No. 2005-144171 describes a configuration in which an imaging region is irradiated with an aiming laser diode light, and describes that a target region is accurately imaged in a narrow space in the oral cavity.

JP 2001-212161 A JP 2005-144171 A International Publication WO2007 / 063980

Many observations in the oral cavity by image display have been proposed, but in the end, it is limited to providing information to the patient on the conventional single and local treatment system. It has not yet reached a system that provides photographed images for the purpose of improving the overall health of the oral cavity.
In order to obtain a panoramic image of the dentition, particularly the tooth side surface, as shown in International Publication WO2007 / 063980, a fixed frame such as a frame having a shape approximated to a phantom arch home approximate similarity virtual shape curve is used. Attempts with handheld intraoral cameras have not yet been proposed, such as the need to prepare possible molds.

  In view of the above, the present invention provides a continuous image sequence forming unit that continuously captures images of a side surface of a dentition to form an image sequence, and an image sequence formed by the continuous image sequence formation unit as a partial dentition image. A side dentition image forming unit that forms a plurality of partial dentition images by synthesizing from the image that is the center of the total synthesis, and a total synthesis between the plurality of partial dentition images formed by the side dentition image formation unit. By providing a side dentition image synthesizing means that forms a whole dentition by combining and synthesizing based on the central image, a clear panoramic dentition can be formed using a hand-held intraoral camera, and X A line panoramic dentition image, a virtual orthodontic or virtual aesthetically colored panoramic dentition can be displayed together with each other, or can be displayed in a superimposed manner, thereby expanding the dental treatment range.

The image shown in the present invention finally shows a digital image, and either a moving image or a still image may be used.
The “image serving as the center of total synthesis” in the present invention indicates, for example, an image that is common to both when two partial panoramic images are synthesized. “Overall” does not show only the final entire panoramic dentition image, but, for example, when three or more partial dentition images are formed, the two partial panoramic dentition images are synthesized first. The panoramic tooth image in the middle of synthesis obtained in this way is also included. “Combine from the image that is the center of the overall synthesis” means, for example, a plurality of images obtained by continuously copying the dentition in the back tooth direction from an image in which a part of the center tooth of the front tooth is in the center. The left partial dentition panoramic image and the right partial dentition panoramic image. “Connected composition” is, for example, combining the left partial dentition panorama image and the right partial dentition panorama image by combining them at a common part or connecting them based on connectable parts. Means that.

In the present invention, there is a case where a mark is set on an image including a central part of the composition. This mark is not easily dissolved by, for example, saliva or water, and is an elongated rectangular shape or a short rectangular shape. For example, a sticker having an adhesive, a pressure-sensitive adhesive, or the like applied to the back surface, which can be peeled off.
Further, the mark may be drawn on the teeth with a colored pen such as green or red, which is not limited to the attached matter, and can remove the color and is easily photographed.
The part where the mark is arranged is preferably arranged so as to cover the upper tooth and the lower tooth. However, for example, when only one of the upper jaw and the lower jaw is photographed, only the person who photographs may be disposed. The predetermined position on the dentition where the mark is placed is, for example, a position where the shooting is interrupted or the movement stops by changing the shooting direction or holding method of the camera, and becomes the center of the composition An image is shown.

What is necessary is just to form with the color (green, blue, etc.) and shape which are easy to recognize on a picked-up image, and a material and a color are selected suitably. Moreover, when obtaining a three-dimensional image, you may use the thing provided with the characteristic three-dimensionality. For example, the mark may be displayed on the surface of the tooth, and the position of the mark is clearly displayed on the captured image. In the case where an irradiating means is provided, or an irradiating means having a corresponding relationship between the illumination distance and the area of the irradiated light, such as a spotlight, may be arranged so as to be able to irradiate the teeth from the intraoral camera.
The mark only needs to be synthesized from an image in which the mark is photographed at a predetermined position, and the photographing direction may not be from the back teeth. The shooting direction and the compositing direction do not necessarily have to be reversed. Based on the image in which the mark at the time of compositing is partially composited left and right from the image displayed at the predetermined position, and finally the mark is displayed at the predetermined position. In some cases, total synthesis may be performed. The predetermined position of the mark for starting the synthesis is exemplified by, for example, the case where the mark comes to the center of the photographed image, but is not limited thereto, and may be a part that can be easily synthesized in partial synthesis and overall synthesis. It ’s fine.

For example, in the case where the mark is partially combined with respect to the right, center, and left side surfaces of the dentition, such as the back side surface of the teeth, the middle tooth between the right side surface and the central side surface, and the center side surface and right side tooth 2 May be necessary, and a plurality of marks may be added. The side surface of the dentition of the present invention may include not only the front side but also the back side and the occlusal surface. Continuous shooting indicates that 30 images or less are automatically captured every second.
The synthesis of images shot at different locations is a known panoramic synthesis method, and the existing method can be selectively used as a synthesis method. Simple synthesis method, simple enumeration method, block matching method, And automatic or manual synthesis methods such as optical flow estimation methods such as Lucas-Kanade method can be used, but in advance image adjustment means such as affine transformation, etc., based on the common part between images, tilt, magnification etc. Is preferably adjusted.
The characteristic part in the present invention is a linear, dot-like, figure-like, or three-dimensional shape indicating a joining position when joining partial dentition panoramic images. For example, two side panoramic images are synthesized. In this case, the boundary line between the central front tooth and the front tooth is shown, but the present invention is not limited to this.

Examples of the oral cavity include regions such as teeth, dentition, gingiva, alveolar bone, lips, hard palate, soft palate, and uvula.
Imaging means conversion to image data that can be output and displayed on a computer monitor (display) device or mobile phone display unit, as well as two-dimensional or three-dimensional display such as a printed or printed state on paper. Indicates a state where
Since the present invention uses a reflecting mirror, the optical path of the irradiation light is relatively long, and an aiming light source such as an LED having a spread based on the directivity angle can be used to clarify the photographing position and the photographing range.

Further, according to the present invention, the angle information of the main body is measured by the gyro sensor for the posture of the intraoral camera that moves vertically and horizontally, the angle of the mirror is derived from the angle of the main body, and the state of the state is grasped. By adjusting the posture of the image from the shooting state, it is possible to display an image that is always easy to see regardless of the state of the camera moving vertically and horizontally.
In the present invention, a position sensor such as an angular velocity sensor (gyro sensor) or an acceleration sensor is used. Specifically, rate gyro that outputs angular velocity, rate integration gyro that outputs angle, attitude gyro, MEMS (micro electro mechanical systems) type, other mechanical, optical angular velocity sensors, piezoresistive type, capacitance type An acceleration sensor such as a heat detection type MEMS sensor is exemplified.
The color of the aiming light in the present invention may be any color that can be distinguished from the color of the illumination light. If the illumination light is white, examples of the aiming light include red and green. Moreover, the timing of irradiation of the aiming light is preferably the timing immediately before the user starts shooting, but there may be a case where irradiation is performed for a short time during shooting.

Furthermore, since the present invention can easily form a panoramic real image on the side surface of the intraoral dentition, the captured panoramic X-ray images can be captured and superimposed, for example, an X-ray image for each tooth can be combined with the panoramic real image. Or by superimposing and displaying, it is possible to form clinical data that is easy to understand for dental workers and patients.
By superimposing and juxtaposing the actual image and the X-ray image on the display means, it becomes possible to realize further understanding of the treatment of the patient.

  The present invention enables a simple panoramic display of an actual image of a part or all of the dentition using an intraoral camera that can be photographed in a handheld state, and in some cases, an X-ray image can be obtained. Since it can be displayed by being overlapped or written together, it is possible to give an easy-to-understand medical explanation to the patient using the display.

FIG. 1 is a diagram showing an embodiment of the present invention. FIG. 2 is a diagram for explaining the embodiment. FIG. 3 is a diagram for explaining the embodiment. FIG. 4 is a diagram for explaining the embodiment. FIG. 5 is a diagram showing another embodiment of the present invention. FIG. 6 is a diagram for explaining the embodiment. FIG. 7 is a diagram for explaining the embodiment. FIG. 8 is a diagram for explaining the embodiment. FIG. 9 is a diagram showing an embodiment of the present invention. FIG. 10 is a diagram for explaining the embodiment. FIG. 11 is a diagram for explaining the embodiment. FIG. 12 is a diagram for explaining the embodiment. FIG. 13 is a diagram showing another embodiment of the present invention.

The present invention is a technique for continuously taking a dentition and synthesizing a partial panoramic image by panoramic synthesis and combining these partial panoramic images to form an entire panoramic dentition, and preferably marks the synthesizing unit. It is possible to easily form a dentition panoramic image by a hand-held camera.
When forming a panoramic dentition image by photographing the lateral direction of the dentition, if the camera uses a reflector, for example, shoot from the back teeth toward the anterior teeth with the cheek meat sandwiched between the tooth sides It is preferable. At a predetermined position, change the direction and take a picture from the opposite back tooth toward the front tooth.
The shooting is exemplified by continuous shooting in which still images are formed at predetermined intervals when the shutter button of the camera is pressed. However, continuous shooting was performed by a method of forming a digital moving image for shooting about 30 still images per second. A still image may be formed, but it may be preferable to photograph at 20 or less per second.
In continuous shooting, it is only necessary to form a still image having at least a common part between images. Therefore, the continuous shooting image may be formed at a lower speed by automatically or manually turning the shutter while shifting the location. In some cases.
In the case of back teeth photography with a hand-held camera, in order to secure the shooting distance, it is often taken while performing an operation to spread the cheek meat using the camera, but in that case, the camera will be shaken etc. Although it can be said that there is little rocking and it is close to a stable state, when moving to the front teeth, it is released from such pushing and spreading operation, and the camera may rock suddenly due to camera shake etc. By detecting the gingiva-to-tooth and tooth-to-teeth boundaries, a reference region may be formed, and a method of correcting and combining the swinging images so as to be combined with the reference may be taken.

Next, embodiments of the present invention will be described in detail with reference to the drawings.
Further, FIGS. 1 to 5 show an example of a panoramic tooth row image forming method in which a panoramic tooth row is clearly synthesized and formed in a state where teeth are engaged.
As shown in Fig. 1, the camera takes a picture taken from the left back to the center, changes the orientation of the intraoral camera, and takes a continuous shot from the right back to the center. In this case, since the photographing state is interrupted in order to change the direction of the camera once, the left and right dentition images cannot be accurately combined and often shift.
For example, in the case of back tooth photography, manual camera movement photography is arranged between the cheek meat on the inner surface of the oral cavity and the side surface of the tooth, and the intraoral camera is moved in such a state as to push the cheek meat away, Since the contact state with the side surface is formed, the cheek meat and the tooth side surface support the reflecting mirror of the intraoral camera and the imaging part of the intraoral camera, but if the intraoral camera is moved toward the front tooth, It is released from pressure by meat and the like, and it becomes a hand-held state, the position of the camera for photographing becomes unstable, and the image is easily disturbed. In particular, there may be a case where the distance between the camera and the tooth side surface that is the subject fluctuates or the photographing direction fluctuates, and the size of the tooth that is the subject fluctuates or the image is distorted.

The intraoral camera 101 using the reflecting mirror shown in FIG. 1 is exemplified by the configuration shown in FIGS. 9 and 10, and has a configuration in which a plurality of light source LEDs are arranged around the modular CCD camera and C-MOS camera. And while illuminating the oral cavity with the LED for the light source through the reflecting mirror, the image of the dentition and the like in the oral cavity is obtained continuously and can be stably photographed from the back teeth. However, the present invention is not limited to this, and there may be a case where the camera for direct photographing does not use a reflecting mirror.
The intraoral camera 101 used here includes a reflecting mirror unit 103 (A10 in FIG. 9) having a planar reflecting mirror 102 (A10K in FIG. 9), an image pickup device such as a CCD camera and a C-MOS camera, and a light emitting diode. A configuration in which the photographing unit 105 (A14 in FIG. 9) combined with about 4 to 8 illuminators arranged around the imager is replaceably mounted on the front end of the main body 104 (A11 in FIG. 9). Is provided as an example.
The shooting unit 105 is exemplified by a unit that outputs a still image digital image by using a continuous shooting method for shooting a still image so that an image can be obtained in a range of 10 to 30 images per second.

Before starting continuous shooting, a mark ML is attached in the vicinity of the center of the dentition 100a in which the upper and lower teeth are meshed in advance. The mark ML temporarily removes the colored sticker and removes the color. It is preferable that the image can be clearly displayed on the image by photographing with a camera such as a description with a coloring pen that can be used. The vicinity of the center when the mark ML is attached to the vicinity of the center of the dentition, for example, indicates a reference location when the left and right dentitions are continuously shot and then combined. A part that is characteristic in image processing may be detected near the center, and the part may be set on the mark image. The mark ML is preferably arranged so as to cover the upper teeth and the lower teeth.
For example, in order to photograph the reflecting mirror 102 of the intraoral camera 101 from the back teeth toward the front teeth, the continuous shooting operation is performed from the state of 106a along the tooth surface as in 106b and 106c, preferably from the dentition surface. The body 104 is held by hand with the same distance apart, and the reflecting mirror 102 is moved to take a picture.

IG is a correction display unit, which is an image correction, for example, a girder, a square, a triangle, a grid, or other figures, on an adhesive member that is attached to the tooth surface in a peelable state, and is an image distortion. Any display that can correct the size of the image based on the distance between the camera and the tooth side surface is acceptable, and the color of the correction display portion IG is also green, and other colors that can be identified in image processing If so, it is not limited to green.
For example, as shown in FIG. 1 and FIG. 3, this sticking member may be attached to the surface of the tooth with the mark ML, and the tooth to be attached is not limited to one tooth. In addition, it may be photographed by pasting a sticking member on a plurality of teeth, or may be attached to a part where other teeth are provided in the oral cavity without any other support and the teeth are manually photographed. .
The means for attaching the correction display part IG to the tooth surface may be the same technique as that of the mark ML. For example, the display is saliva or the like and does not melt or bleed. It is preferable when the image and the right dentition partial panorama image are combined by correcting both image distortion and size based on the correction display portion IG photographed in common for both.

When combining partial panoramic images divided into three parts, that is, the left dentition, the central dentition, and the right dentition, the correction display unit IG may be provided on the common tooth of each partial panoramic image sequence. good.
The correction in each captured still image is also based on, for example, the reference correction display unit IG in an image obtained by continuous shooting, or the correction display unit IG is included in the captured image group. One of the captured images is detected by the block matching method or the template matching method, etc., and this is used as a reference. After the correction display portion IG captured by another image is detected, the distortion is compared with the reference image. Further, correction using affine transformation for detecting enlargement, reduction, rotation, and movement adjustment may be performed by detecting a difference in inclination and size. At the time of photographing in which manual swinging is unavoidable, the correction display portion IG is attached to the teeth, so that composition for stable panoramic tooth row image formation is possible. The affine transformation is a preferable example, and other adjustment methods capable of performing similar image operations may be used.

Such image correction using the correction display unit IG is performed by, for example, camera image calibration such as Zhang's method (IEEE. Transactions on pattern Analysis and Machine Intelligence, 22 (11); 1330-1334, 2000). You may use the method used by.
Further, the correction display unit IG may not be essential due to processing such as a photographing state and affine transformation.
FIG. 1 (b) is a schematic diagram showing a single photographing range when actually taking continuous shots from the back teeth.
A continuous-shot image is obtained as a still image while moving the reflector portion of the intraoral camera 101 over time as in 106a → 106b → 106c → 106d → 106e → 106f → 106g. Since the intraoral camera 101 is configured to include a reflecting mirror 102 having a predetermined angle at the tip, assuming that the dentition surface is photographed from the left back tooth in the figure, the orientation of the main body 104 is near the center. In other words, since the images are taken in order from the right back tooth in the figure, after the surfaces 106e to 106g shown in FIG. 1 are imaged, the intraoral camera 101 is reversed and the image taking from the right back tooth is started.
Since the main body 104 is moved manually, the continuous shooting speed is set to 20 to 30 shots per second in consideration of the effects of shaking such as camera shake, and there is room to remove blurring and out-of-focus images. It is preferable to perform continuous shooting.

This continuous shooting is preferably performed until the mark ML reaches the center of the imaging screen or the reflector. However, there are cases in which the imaging is performed until a certain amount of time has passed, and then the captured image is selected. .
For manual camera operation, it may be preferable to perform affine transformation based on a common part of each image and to adjust the state of the image before synthesis.
For example, a plurality of points that are common to the comparison image by the block matching method are detected based on the image that is the first center of synthesis. The next image is affine transformed based on the plurality of common points. For example, a plurality of pixel coordinates (xb, yb) of the next image corresponding to the pixel coordinates (xa, ya) of the reference image in the common part are selected and substituted into the following equations to obtain the coefficient values a to f. .
With the coefficient value substituted into the following equation, panorama synthesis may be performed after the next image is affine transformed to organize the image or while the image is being prepared.

このように歯列１００ａの図中左奥から正面付近まで、連写して得た画像の一部を図２に示し、図１と併用して当該実施例に基づく口腔内カメラの撮影動作を説明する。
図２（ａ）から（ｅ）は、図１（ｂ）で示す歯列の撮影画像データにおける中心付近１０７ｃから１０７ｅ付近の連写撮影をした際の撮影画像データ列の一例である。
尚、口腔内カメラは、反射鏡１０２を利用して歯列画像を得ていることから、撮影画像は、左右が反転するが、わかりやすくする為に、反転していない状態で図２及び図４に示した。

A part of the image obtained by continuous shooting from the left back of the dentition 100a to the vicinity of the front is shown in FIG. 2, and the photographing operation of the intraoral camera based on the embodiment is described in combination with FIG. To do.
FIGS. 2A to 2E are examples of photographed image data sequences when continuous shooting is performed from the vicinity of the center 107c to the vicinity of 107e in the photographed image data of the dentition shown in FIG. 1B.
In addition, since the intraoral camera obtains a dentition image using the reflecting mirror 102, the captured image is reversed left and right. This is shown in FIG.

2 (a) is taken near 106c in FIG. 1 (b), FIG. 2 (b) is around 106d, FIG. 2 (c) is around 106e, FIG. 2 (d) is around 106f, FIG. 2E shows an image obtained by photographing the vicinity of 106 g. In addition, since shooting is performed manually, there are many cases where the shot images cannot always be shot in the same state. Therefore, when the image is slanted or the like, it is preferable to perform correction using affine transformation in, for example, a portion overlapping with another tooth state, and FIG. 2 shows a diagram after correction.
At the time of correction, a vertical center line CL and a horizontal center line HL that are preset on the reflecting mirror surface on which the reflecting mirror 102 is photographed may be used in addition to the mark ML.

The vertical center line CL and the horizontal center line HL are not necessarily displayed in the image, and marks such as protrusions are respectively provided at the vertical center line at the edge of the reflector and the portions corresponding to the start point and end point of the horizontal center line. It may be the extent to be attached, and may be displayed virtually based on this mark. Further, a process for correcting the distortion caused by the lens of the photographing CCD camera may be performed together.
RM is the contour of the mirror, and the actual captured image is a circular image within the contour RM, but is displayed as a square image for easy explanation of the range.
Since the photographed image is handheld, the distance between the teeth and the camera may be different. In this case, the other photographed images are based on an image in which the mark ML and the vertical center line CL substantially match. You may correct | amend the magnitude | size of.
Note that there may be a case where the obtained image can be stabilized by performing continuous shooting while making the tip of the reflecting mirror 102 lightly contact the tooth surface.

As shown in FIG. 1, the reflecting mirror 102 of the main body 104 is arranged on the back teeth on the left side from the front, and then moved in the front direction to continuously shoot, for example, to acquire the images of FIGS.
In this case, the image obtained up to the image FIG. 2C taken at the timing when the mark ML coincides with the vertical center line (CL) of the reflecting mirror 102 is employed as the composition image. Since the use of FIG. 2 (d) and later may cause a shift during synthesis, it is preferable not to use it for synthesis.
In order to make the image 10 (c) the reference image, the mark ML is used as a reference, and if necessary, the inclination of the image is corrected, that is, the image is corrected using the long side and the short side of the mark ML as a reference image. It is good.

An example of one synthesis operation will be described below.
Images used for the synthesis are shown in FIG. 2 (f) corresponds to the image shown in FIG. 2 (c), FIG. 2 (g) corresponds to FIG. 2 (b), and FIG. 2 (e) corresponds to FIG. 2 (a). The image 106e shown in FIG. 2 (f) and the image 106d shown in FIG. 2 (g) are superimposed and synthesized based on a common portion where the shape of the image 106d matches or approximates with the image 106e as a reference.
A portion protruding from the left side of FIG. 2 (g) when overlapped with reference to FIG. 2 (f) is indicated by 201a. 201a is an image in the back tooth direction.
Next, the synthesized image shown in FIG. 2 (g) and the image 106c shown in FIG. 2 (a) are visually observed at a portion having a common shape or the like with reference to the synthesized image of FIG. 2 (g). Alternatively, they are overlapped by an image processing method. A portion of the superimposed image that protrudes leftward from FIG. 2G is indicated by 201b.
201b is an image in the back tooth direction.
This operation is performed between the next adjacent images, and further overlapped with the next adjacent image at the common portion, so that the left dentition is synthesized from the image where the mark is in a predetermined position in a panoramic image in the direction of the back teeth. Form.

In addition, a method of joining the portions 201a, 201b,... That protrude from the center image shown in FIG.
In addition, for example, the protruding portion is detected by extracting the protruding portion between 100e in FIG. 2C and 100d in FIG. 2B, and further detecting 100d in FIG. 2B and FIG. ) Of 100), and the parts that protrude between the next adjacent images are collected, and finally the reference image figure is 10 (f), and these protruding parts are joined together. A panoramic image may be formed. In some cases, it is preferable to superimpose adjacent images from the center of the image of the protruding portion with the image from the center as a reference.
Note that, even when they are not completely overlapped and do not match or approximate, if a marker portion exists in common, the same composition may be performed by overlapping the portions. In addition, since it is manual, a perspective and inclination may differ for every still image obtained. In this case, it is preferable that the correction is automatically performed using affine transformation or the like, or is arbitrarily enlarged and reduced.

The synthesized state is shown in FIG. As shown in FIG. 5A, a left half panoramic tooth image can be formed.
Next, as shown in FIG. 3A, in a state where the reflecting mirror 102 of the main body 103d is disposed at the right back of the tooth row 100a in a state of being vertically engaged, 301a → 301b → 301c and continuous shooting with still images are performed in the direction approaching the center front tooth.
FIG. 3B schematically shows the positional relationship between the still image captured and the dentition 100a.
While holding the intraoral camera 101 by hand, the user moves the camera in the direction of 302a → 302b → 302c → 302d → 302e → 302f → 302g → 302h, and obtains still images continuously. Since the present embodiment is hand-held, the photographed image is tilted or shifted in the left-right and depth directions. However, the adjacent images obtained by continuous shooting using affine transformation or other contour extraction means are used. You may provide the process of adjusting an image by rotating and moving based on the common part which exists.

Since the intraoral camera 101 shown in FIG. 3 (a) is the same as that shown in FIG. 1, the same number is assigned and the description is omitted.
FIGS. 4A to 4E show typical images in the range from 302a to 302f in FIG. 3B.
4 and FIG. 2 are denoted by the same reference numerals as those shown in FIG.
4, FIG. 4A shows the vicinity of the image 302d in FIG. 3B, FIG. 4B shows the vicinity of the image 302e in FIG. 3B, and FIG. FIG. 3B shows the vicinity of the image 302f in FIG.
4D shows the vicinity of the image 302g in FIG. 3B, and FIG. 4E shows the vicinity of the image 302h in FIG. 3B.

4D and 4E are photographic images when the reflecting mirror 102 is further moved to the left after the vertical center line CL and the mark ML coincide with each other. 4A to 4C are employed instead of adopting this because an overlapping portion of images photographed from the direction increases and image shift occurs.
Among these, FIG. 4F shows an image based on an image (302f in FIG. 4C) in which the mark ML and the vertical center line CL coincide with each other. Next, the common parts of the images 302e and 302f shown in FIG.
401a is a portion that protrudes in a state where the image 302e is overlaid in a range in which the shape matches or approximates from the image 302f on the basis of the image 302f.
Next, the superposed image and the image 401b shown in FIG. 4B are superposed on a portion where the shapes match or substantially coincide with each other using a pattern matching method or the like on the basis of the superposed image.

In the overlapped state, the protruding portion is 401b. In this way, the adjacent teeth are overlapped to form the right dentition. As another synthesis method, the reference image 106e and the images 401a and 401b may be synthesized as shown in FIG. 5B.
In addition, when there is a slight shift when the previous and subsequent images are superimposed, one of the superimposed portions may be deleted.
With the above procedure, continuous shot still images are synthesized, and the center image 106e obtained when the left dentition image is synthesized and the center image 301c obtained when the right dentition image is synthesized are overlapped to match or substantially match. By doing so, the panoramic tooth row 303 shown in FIG. 5C is formed. A panoramic image of the entire dentition is shown in FIG. b101 is a left partial panoramic dentition, and b102 is a right partial panoramic dentition.

The correction display portion IG and the mark MK set on the center tooth are preferably subjected to a process of erasing from the image using general-purpose graphic software. Erasing and adding in image processing are, for example, by adjusting pixel values. For example, in the case of erasing a mark, the brightness value of teeth on the adjacent image around the mark is detected and detected. It may be erased by a process of replacing with a brightness value or converted to a preset brightness value.
For superposition, for example, it is preferable to connect and synthesize the left and right panoramic tooth rows based on the central tooth-to-tooth boundary portion (KL). In addition, in order to cope with the case where the tooth-to-tooth boundary is different between the upper jaw and the lower jaw, the tooth boundary portion of the upper jaw or the lower jaw may be used as the boundary, and synthesis may be performed only for the upper jaw. In some cases, the panoramic dentition and the panoramic dentition of the lower jaw may be combined, and finally the upper jaw and the lower jaw may be combined. Examples of the boundary part detection include a contour extraction process in the two-dimensional Fourier transform process described below and a method of extracting a color component value that emphasizes the boundary value, and the boundary value is a boundary between teeth of the upper jaw or a lower jaw, or The boundary etc. which pass both the upper jaw and the lower jaw in the biting state at the time of occlusion are illustrated.

FIG. 5D shows a diagram in which the boundary KL is extracted from each of the right partial panoramic tooth row b101 and the left partial panoramic tooth row b102, and is connected and coupled at the boundary KL. In some cases, the images may be superposed on each other or may be replaced with one image of the partial panoramic dentition. Further, for example, the center image 106e and the center image 301c shown in FIG. 5 are not necessarily overlapped in advance, and one of the center images CGG may be adopted.
In this case, for example, when the center image 106e is adopted, affine transformation may be performed on the image from the image 401a to the back teeth based on the center image 106e, or the sizes of the 401a and 401b may be slightly modified. When 301c is adopted, the sizes of 201a and 201b may be corrected. In this way, in addition to the method of joining the protruding parts between the images, if the overlapping parts are not extracted and are overlapped as they are in the common part, they are superimposed while adjusting the size of the central images. There is also a case.
In this way, the left and right dentitions are synthesized from the central image to form the left and right synthetic dentitions, and these are synthesized based on the respective central images, so that a panoramic dentition with reduced deviation can be obtained. It can be formed.

For the synthesis based on the center image, for example, it may be preferable to combine the left composite dental row and the right composite dental row using the contact line (edge) between the center tooth and the tooth as a characteristic part. Note that the mark does not necessarily have to be attached to the center tooth, and may be a tooth part photographed at the timing of changing the direction of the camera at the time of photographing. The position of the mark is appropriately determined for other purposes. Selected. Similarly to the mark, the characteristic part is not particularly limited as long as it can be aligned at the time of image synthesis.
Further, by adding a mark to a dentition that is a subject in advance or setting a part corresponding to the mark from an image, more accurate composition is possible.
The above description of the operation is based on operations such as copy paste, drag and drop, image enlargement / reduction, tilt correction, etc. by visual and mouse operations using graphic software. In some cases, known automatic panoramic photo composition software (Photoshop Elements 7 (trademark) (manufactured by Adobe), Photo Stitch (trademark) (manufactured by Canon), or the like may be used.

Creation of 3D panoramic dentition image

The dentition is curved in an arcuate shape with respect to the occlusal plane. When the whole is realistically grasped, the two-dimensional panoramic display shows the state of the individual teeth, but the teeth are planar. Since they are displayed side by side, three-dimensional grasping is insufficient, and therefore a method capable of three-dimensional display is preferable.
Next, an embodiment for forming a three-dimensional panoramic image using a real image will be described with reference to FIG.
FIG. 6 is a block diagram for explaining the three-dimensional panoramic dentition image forming means.
Reference numeral 501 denotes photographing data input means for connecting a stereo image photographing camera having a plurality of cameras as shown in FIG. 7 to simultaneously form the same number of images as the number of cameras.
The shooting data input means 501 has a configuration in which a shutter is pressed for each image and a configuration that enables continuous shooting in which a plurality of photos are shot by pressing the shutter once. May be output continuously.

Reference numeral 502 denotes a correction unit, which is a unit for correcting distortion due to the lens shape, camera shake, and the like, and correcting a long and short distance, and is configured using a known method such as a so-called calibration unit.
The correction unit 502 calibrates each of the images taken at the same time, and sometimes deletes a peripheral portion having a large distortion.
Reference numeral 503 denotes common point detection means, for example, means for detecting common points of a pair of images. The common point detection unit 503 uses the luminance value of one pixel or a group of pixels in one image as a reference luminance value, and uses the luminance value of one pixel or a group of pixels in another image, for example, luminance A sum of differences (SAD) and a sum of squares of differences in brightness (SSD) are taken, and a portion that matches by the minimum or maximum value or a portion estimated by subpixel estimation is output as a common point.

More specifically, for example, Motoki Arai et al., Optimization of correlation function and subpixel designation method in image block matching, Information Processing Society of Japan, 2004, P33-40 and other publicly known configurations A technique may be used.
Reference numeral 504 denotes world coordinate conversion means for converting the coordinates of the obtained images of the common points into the common three-dimensional coordinates as a whole.
The world coordinate transformation means 504 is performed by computer processing by an arithmetic technique such as trigonometry, eight-point algorithm method, triangulation method, etc., and the coordinates of the photographic images of the common points obtained by the common point detection means 503 are used. From the value to the parallax value and camera characteristics (focal length of the lens, which is an internal parameter, image center, pixel size, and position and orientation of two cameras, which are external parameters), world coordinates (X, Y, Z) Is formed and output.
For example, it is generally known from perspective projection matrices P1 and P2 each consisting of different camera internal parameters and external parameters, and local coordinates (u1, v1) and (u2, v2) of a common point M of each captured image. Based on Expression (1), world coordinates (X, Y, Z) are obtained.

カメラの内部パラメータ、外部パラメータを用いて透視投影行列を求めるか、各静止画像から得られる複数の共通点のローカル座標等から透視投影行列Ｐ1、Ｐ2及びω1、ω2を求めて、式（１）にもとづいてワールド座標を得る方法等に関しては、例えば、電子情報通信学会誌 Ｖol．９２、Ｎo.6,2009,463-468に記載されている手法等の公知手法が好適に用いられる。
その他３次元座標を得る手法としては、接写手法とともに例えば、歯科材料・器械 Ｖｏｌ．１９Ｎｏ．３333-338(2000)等に記載されているが、これに限らず、その他一般的な手法をとり得る。

A perspective projection matrix is obtained using internal parameters and external parameters of the camera, or perspective projection matrices P1, P2, and ω1, ω2 are obtained from local coordinates of a plurality of common points obtained from each still image. Regarding the method of obtaining the world coordinates based on, for example, the Journal of the Institute of Electronics, Information and Communication Engineers, Vol. 92, No. 6, 2009, 463-468, and other known methods are preferably used.
Other techniques for obtaining three-dimensional coordinates include, for example, dental materials and instruments, Vol. 19No. 3333-338 (2000), etc., but is not limited thereto, and other general methods can be used.

Reference numeral 505 denotes a three-dimensional image forming means for displaying the world coordinate data in a three-dimensional coordinate space virtually formed on a computer, for example, and displaying the coordinate data connected by straight lines or curves. A wire frame model may be formed, and more realistic panoramic three-dimensional data can be obtained by pasting virtual surface data to a portion surrounded by a line. A curved dentition may be displayed by three-dimensionalizing the three-dimensional panoramic dentition data on the three-dimensional coordinates. Reference numeral 506 denotes a display means, such as a computer monitor, printer, etc. And may be displayed as a virtual three-dimensional image on a normal computer monitor, or may be displayed by projecting a curved panoramic dentition image two-dimensionally.
FIG. 7 shows an example of a probe-like three-dimensional measurement probe 600 having two cameras at the tip.
Reference numeral 601 denotes an imaging unit A, in which a lens 601a is disposed at the center, and has a so-called camera module configuration.
Reference numeral 602 denotes an imaging unit B, in which a lens 602a is disposed at the center, and has a so-called camera module configuration.
Reference numeral 603 denotes an illumination light irradiation unit, which is preferably provided around the imaging unit A 601 and the imaging unit B 602, and emits light emitted from the light emitting means 605 formed inside the support member 606 via the optical path 604. This is the part that irradiates the surface of the tooth that is the subject. It is preferable that the illumination light irradiation unit 603 has a shape that allows more uniform illumination, and is not limited to the shape shown in FIG.

Reference numeral 603a is a light source for indication, which is used to indicate a photographing surface, and is a portion that performs spotlight-like output, and uses LEDs such as red, blue, and white whose periphery is covered with a black cylindrical body. It is done. Since it is spotlight-like light, the area of the irradiation surface varies depending on the distance. Therefore, the probe 600 can be moved at a constant distance by moving the probe 600 while keeping the light circle constant.
Reference numeral 604 denotes an optical path, which is formed inside the support member 606, and its surface is preferably covered with a light reflecting member such as a thin film such as aluminum or silver.
Reference numeral 605 denotes a light emitting means, which is mounted in the support member 606, and includes a white LED and other light sources. In this embodiment, intermittent flash driving such as a strobe light, continuous illumination driving, or the like may be performed.
Reference numeral 606 denotes a support member, which is formed of, for example, a lightweight and hard plastic material, has an imaging unit or the like at the tip, has a shape that is easy to insert into the back of the oral cavity.

Reference numeral 607 denotes an operation switch for performing a shutter operation or the like. The number should just be set according to the operation specification and the purpose, and should just be arbitrary structures.
When used as a shutter, it is possible to adopt a specification in which the shutter is continuously driven at a predetermined interval while being pressed.
Reference numeral 608 denotes a gripping part, which is formed integrally with the support member 606, and is preferably molded from a lightweight, strong plastic material or the like.
Reference numeral 609 denotes an electrical lead wire for connection to an external power source, connection to an external data processing apparatus, or the like, and a cable using a USB connection connector may be used.
Also, it is not necessary when the light source is strobe light emission and continuous shooting data can be temporarily stored inside the camera, or when using Zigbee (registered trademark) or a wireless communication front-end circuit for wireless connection Sometimes it becomes.

The operation of this embodiment shown in FIGS. 6 and 7 will be described.
In the photographing data input means 501 of FIG. 6, the user holds the gripping portion 608 shown in FIG. 7 and the imaging portion A601 and the imaging portion at the tip of the support member 606 with the upper and lower teeth engaged as shown in FIG. While B602 is brought close to the subject part and the size and position of the spotlight indicated by the instruction light source 603a is viewed, the operation switch 607 is pressed to start the continuous shooting operation.
In some cases, the shutter operation switch 607 may be pressed for every shooting, but continuous shooting with fewer pressing operations to prevent camera shake is preferable.
In this pressed state, the image is taken to the same position as in FIG. 1A, and then the same position as in FIG. 3A is taken to the same position.
FIG. 8 shows a pair of images in an image group obtained by one continuous shooting. In addition, when measuring the surface shape of a tooth more accurately in the oral cavity, it is preferable to photograph in a close state, and it is preferable to set the focal length of the camera in a state in which close-up photography is possible.
These captured images are calibrated for distortion caused by the curved surface of the lens by the correction unit 502 in FIG. 6, and are output to the common point detection unit 503.
When performing the three-dimensional processing, it is preferable to perform from the center in consideration of the composition from the center to the left and right as shown in FIGS. 2 and 4, but the method is not particularly limited to this method.
FIG. 8A is an image obtained by photographing a place near the center of the front surface of the tooth, and even data obtained by photographing with the photographing data input means 501 having the probe 600 of FIG. is there.

701a is an image of the imaging unit B602, and 702a is an image of the imaging unit A601. Coordinates having the same central point as the same part are assigned to this image, and for example, an arbitrary point A (x1, y1) of the image 701a in FIG. 8A photographed by the imaging unit B602 is set and the same position is set. The point A ′ (x2, y2) to be shown is searched. For example, the average luminance value is obtained by using the point A as a block of one pixel. A pixel block having the same size as that of the point A is obtained from the vicinity of the position assumed to be the point A ′ of the image data 702a in FIG. 8A, and the sum or square sum of the difference between the luminance values is gradually obtained. 'While moving in the same direction while performing similar calculations. A coincidence evaluation curve is formed.
On the coincidence degree evaluation curve, a subpixel estimation method in which the sum of the difference values or the sum of squares of the differences is set to be the smallest portion or the largest portion as the point A ′ is preferable. It is not particularly specified as long as it is a method to obtain.
Next, the same operation is performed on the pixel block next to the point A of 701a, and a common point is detected from the image 702a. Repeat this operation. The coordinates of the common point are obtained in the common range 703a excluding the occlusion range of the images 701a and 702a.
Also in this case, it is possible to obtain a common point with high accuracy by forming the common point coordinates around the position of the newly added mark ML.

In addition, by reducing the size of this block, common points can be detected with high accuracy, but the processing time becomes longer, so the size of the block and the like are appropriately selected.
Next, in the world coordinate conversion means 504 of FIG. 6, for example, from the coordinate values measured by taking a plurality of the common points, a three-dimensional world coordinate system is obtained based on parameters and a set of parameters such as parallax and focal length. Convert. As this specific method, a normal method shown in the above-mentioned literature or the like is preferably employed.
After obtaining a common point from the pair of images in FIG. 8A, the common range 703b excluding the occlusion range is converted into three-dimensional world coordinates using the image pair 701b and 702b shown in FIG. 8B. Convert. For example, a wire frame shape shown in FIG. 8D is created in the state of the three-dimensional world coordinates.
3A (X, Y, Z) in FIG. 8D is a three-dimensional image converted by using equation (1) based on FIG. 8A, A (x1, y1) and A ′ (x2, y2). World coordinates.

Further, the three-dimensional world coordinates of each common point in the common range 703c excluding the occlusion range of the image pairs 701c and 702c shown in FIG.
Next, in the three-dimensional image forming means 505 shown in FIG.
The three-dimensional coordinates obtained from FIG. 8 (a) converted to world coordinates and FIG. 8 (b) are superimposed in a virtual three-dimensional coordinate space instead of in a planar superposition.
The three-dimensional world coordinates shown in FIG. 8C are further superimposed on this superimposed image.
If this superposition is a superposition in a virtual three-dimensional space, for example, a wire frame-like dentition as shown in FIG. 8D is formed on a computer monitor based on the data converted into the three-dimensional world coordinates. It is possible to display each of them virtually and to superimpose them by visually operating a computer interface such as a mouse. However, if you want to increase the accuracy of the overlay, use either of the three-dimensional values to be compared as a reference. In some cases, such a combination that the difference is the smallest due to the difference comparison or the like is preferable. A common point may be obtained by a sub-pixel estimation method using block matching.

8A to 8C, the dentition is formed based on the three-dimensional world coordinate data of the right dentition toward the front. Next, the left dentition is three-dimensional world coordinated. .
For the world coordinate conversion, the known method described above may be used. For example, after obtaining three-dimensional coordinates based on two images, conversion to world coordinates which are common coordinates may be performed.
In some cases, after obtaining the three-dimensional world coordinates of the left and right dentitions, the three-dimensional virtual display of the computer or approximate matching by numerical superposition of coordinate values may be used.
In that case, it is preferable to superimpose the mark portion on the basis of the three-dimensional coordinate data, and according to this method, the panoramic tooth row display without deviation is displayed by the display means 506 shown in FIG. It becomes possible.
Superposition is a superposition of data values of the same shape part, and if they do not match, a method of superimposing them while taking the average of both coordinate values, a manual method of dragging and dropping the image on the screen An example is a method of obtaining a coordinate value after superposition while operating a computer mouse.

Even if the images have common parts, if the images are in a relationship that does not overlap correctly after correction, it is only necessary to select one of the common images and create a panoramic image. In some cases, non-overlapping partial images necessary for the purpose may be used.
It is possible to display all the dentitions in a so-called denture-like three-dimensional shape by synthesis by three-dimensionalization based on the world coordinates.
Since the two-dimensional panoramic digital image display of the tooth side and the three-dimensional panoramic digital image display of the tooth side allow the patient to easily understand the entire dentition state, for example, 2 For each of the three-dimensional panoramic dentition data and the three-dimensional panoramic dentition data, orthodontic simulation processing and virtual whitening processing are performed on the computer monitor so that the virtual dentition is formed on the computer monitor. You may form the aspect which realizes.

For example, if the orthodontic simulation processing is a two-dimensional panoramic dentition image, a database of various shapes of one tooth is created in advance according to the part, and each tooth image on the two-dimensional panoramic dentition image is obtained. The means by which the orthodontist operates the computer mouse on the graphic software and performs copy paste to form the formed virtual orthodontic panoramic dentition.
In the case of a panoramic 3D dentition image, since 3D coordinates have already been set, orthodontics etc. perform 3D coordinate adjustment of the panoramic 3D dentition image on the existing CAD software, and the virtual A means for forming an orthodontic three-dimensional panoramic dentition is exemplified. These correction panorama display methods are examples, and other methods may be adopted.
The virtual display after the virtual whitening process for the own panoramic 2D and 3D dentition images may be written together with the dentist adjusting the color data for these images on the graphic software. In this way, by providing virtual panoramic correction and virtual coloring processing on the patient's own panoramic dentition on the screen, the patient can deepen their understanding of treatment.

Aiming light irradiation means

FIG. 9 is a diagram showing an embodiment of the present invention.
A10 is a reflecting mirror unit, which is made of hard plastic or the like, and is provided with a reflecting mirror A10K installed at a predetermined angle (for example, 45 degrees) at the front end, and can be connected to the outer periphery of the photographing unit A14 at the rear end. A cylindrical mounting portion A10S is formed, and in the meantime, an open shape is formed.
The connection between the mounting part A10S and the photographing unit A14 is formed by pushing it in, and it is sufficient that it can be detached only by pulling it apart. In order to prevent rotation, both sides are provided with irregularities, or an asymmetrical shape such as an ellipse. You may do it.
A11 is a main body, which is formed of plastic and resin, has a cylindrical shape such as a ballpoint pen with a large aperture and is easy to hold by hand, is arranged so that the photographing unit A14 protrudes at the tip, and at the rear end The cable A15 for connecting to the external processing apparatus is connected.
A12 is an observation direction when a dentist, a dental hygienist, or the like directly observes and observes the reflector A10K arranged at a predetermined angle at the tip of the reflector unit A10.

A13 is an aiming light irradiation surface, which is an example of the irradiation surface of the aiming light when the aiming light output from the aiming light source A142 is reflected by the reflecting mirror A10K to illuminate the tooth surface.
The position of the aiming light source A142 is arbitrary and may be any other position as long as the range of the captured image can be determined. The site is, for example, the tip of the reflector unit A10, and illuminates the imaging range. It may be a part to do. In this case, since the optical path is shortened, the range of the captured image may be illuminated even if the directivity angle is somewhat wide.
FIG. 9C shows an enlarged state of the photographing unit A14.
A141 is a light source for illumination, and an LED or a combination of a lens and an LED having a wide directivity angle is exemplified. A plurality of illumination light sources A141 are arranged around the photographing member A143.
A142 is a light source for aiming, and examples thereof include an LED having a small directivity angle and an output capable of having a predetermined spread on the irradiation surface by a combination of a lens and an LED.

In the case of a light source having a small directivity angle, it is preferable that a plurality of light sources be arranged with a predetermined interval.
A143 is a photographing member, which is composed of a CCD or CMOS camera, and preferably has a larger number of pixels.
A15 is a cable for connecting to an external image display device, and may be formed by a dedicated cable in addition to a general-purpose cable such as a USB cable.
A16a and A16b are a first operation button and a first operation button, which are configured by a push type, a pull type, a rotation type, or the like, and when the first operation button A16a is pressed among these buttons, a predetermined time is used. The light source A142 emits light and illuminates the main part for a certain period of time via the reflector A10K.
The fixed time is a time during which the user can recognize at least the aiming light that illuminates the main part of the oral cavity, and is preferably until the timing of photographing, for example, the first operation button A16a or the like is pressed.
At the time of shooting, illumination parts with different color schemes are formed on the still image, which is not preferable in that it interferes with observation, but if it does not interfere with observation, it is not necessary to turn off the aiming light, and it can be blinked. It may be something that the user recognizes. It is also possible to make the distance between the reflecting mirror and the teeth constant by photographing the side surface of the dentition in the oral cavity while maintaining the irradiation area of the aiming light.

Next, the operation of the embodiment shown in FIG. 9 will be described.
The illumination output of the illumination light source A141 mounted around the imaging member A143 of the imaging unit A14 illuminates the tooth AH1 in the oral cavity via the reflector A10K. A14L is an aiming optical path.
In addition, the illumination light source is also irradiated through the reflecting mirror A10K, and a part of the illumination light surface is in a state where the aiming light irradiation surface A13 is formed.
The imaging member A143 inputs the illuminated intraoral site via the reflector A10K and displays it on the external monitor device via the cable A15.
A user such as a dentist can grasp the shooting position from the image displayed on the external monitor device, but when the medical treatment time is shortened or treatment is included, it is used in the same way as a dental mirror. In many cases, the observation direction may be different from the captured image as indicated by A12. In this case, for example, the first operation button A16a is pressed. When the first operation button A16a is pressed, the aiming light source A142 emits light, and the light emission is performed for a predetermined time, preferably before the start of photographing, and the observer. It emits light with an intensity to confirm the observation position and the shooting position.

The observer moves the reflecting mirror A10K in order to match the photographing position with the observation position, and presses the first operation button A16a again to adjust the photographing position and the observation position.
When the positioning is completed, the still image or the moving image is recorded by pressing the first operation button A16a or the second operation button A16b again. The operation content of the first operation button A16a and the second operation button A16b described above is an example, and is appropriately selected depending on the case.
An example is shown in FIG.
A17 is an example of an image, and a tooth AH1 to be photographed is photographed. The irradiation range of the aiming light is a range of a circle indicated by the aiming light irradiation surface A13, and includes almost all of the main parts intended for photographing.
The range of the aiming light irradiation surface A13 changes when the combination of the main body A11 and the reflector unit A10 moves up and down with respect to the tooth H1, so that the user moves the main body A11 and the reflector unit A10 up and down. However, there is no deviation from the main part of the photographing range, and the main part of the photographing range is more sufficiently shown than the point light source.
By the above operation, the photographing surface and the observation surface are adjusted, and an accurate still image or moving image is recorded.

The body A11 shown in FIG. 9 (a) is held, and the observation part in the oral cavity is moved vertically and horizontally so as to capture the target part of the reflecting mirror A10K and observed and photographed.
The intraoral camera is inserted into a narrow and deep part of the upper jaw or lower jaw of the oral cavity, and the imaging range of the reflector is wide and the intraoral image is taken with the reflector inverted. Since the image is taken even in an oblique state or sideways, the photographed image may be tilted or inverted.
As shown in FIGS. 1 and 3, even when shooting while moving the reflector of the intraoral camera from the back tooth to the front tooth, the camera and the reflector may tilt or rotate because it is manual. The resulting image is similarly tilted and rotated.
In this state, the shooting state is detected by the position sensor, and the image is corrected to the image obtained when shooting in the horizontal movement state that is the ideal movement of the camera with respect to the upside down or oblique state. FIG. 10 to FIG. 12 will be described with reference to FIGS. 10 to 12, in which the image correction means is provided so that the obtained image becomes a posture-controlled image even when the camera is slightly rotated or tilted.

Photographed image correction means

FIG. 10 is an example of an intraoral camera used for explaining an embodiment of the present invention, and a part thereof is shown as a cross-sectional view.
Reference numeral A21 denotes a gripping housing, which has a cylindrical shape with an internal space, and an oval cylindrical photographing unit A23 including a camera and an illumination light source formed at the periphery of the camera at the tip is a housing A21. The connection is made in a more protruding state, and a cable A26 for connecting to an external display device is connected to the rear end.
An example of the lighting unit is shown in FIG.
A22 is a reflecting mirror unit, and a reflecting mirror A22H disposed at a predetermined angle is attached to the front end, and a cylindrical mounting portion A22S that can be attached to cover the periphery of the photographing unit A23 is formed at the rear end. Others are open.
A24 is a circuit board, which is mounted inside the casing A21 and mounted with an image processing IC and the like. A25a and A25b are position sensors, and have an IC chip shape mounted on a circuit board. The number of position sensors and mounting parts are examples, and depending on the type of sensor, they may not be mounted.
Since the intraoral camera can take a wide range of movement states, such as an acceleration sensor and an angular velocity sensor, the position sensors A25a and A25b employ sensor elements that do not become incapable of measurement depending on the angle. Although the number of position sensors is shown as two, it is an example, and there may be one if the number of chips changes according to the number of axes or there is a unit for three axes.
The acceleration sensor and the angular velocity sensor are exemplified by triaxial sensors, and the number of position sensors may be adjusted by the number of axes or the like.

The x-axis, y-axis, and z-axis of the position sensors A25a, A25b,.
When the position sensors A25a and A25b are angular velocity sensors (gyro sensors), the angular velocity sensors are, for example, the amount of change in angle due to movement around the x axis, the amount of change in angle due to movement around the y axis, and the amount of change around the z axis, respectively. The amount of change in angle due to movement is output.
In the case of an angular velocity sensor, the initial state of the x-axis, y-axis, and z-axis is arbitrarily set, and the amount of change of each axis is added to detect the shooting state of the camera.
On the other hand, the acceleration sensor outputs an acceleration component in the x-axis direction, an acceleration component in the y-axis direction, and an acceleration component in the z-axis direction. Further, the combined vector of these acceleration components outputs a posture vector, and when in a stationary state, indicates a gravitational acceleration vector. From this posture vector, the shooting state of the camera can be obtained.
For example, since the acceleration sensor outputs the state of the gravitational acceleration vector A451 as a posture vector when stationary, the vectors in the x-axis direction, the y-axis direction, and the z-axis direction in this state are used as reference postures, and then the angular velocity sensor is used. Because it is possible to use methods such as adding the rotation change amount of the axis of

Various states of the camera may be detected by combining both the acceleration sensor and the angular velocity sensor.
FIG. 11 is a block diagram showing an example of means for correcting the image display state using the position sensor. The configuration shown in the block diagram may have a portion executed by a program as long as it is a computer process.
A31 is photographing means, and means for photographing a moving image and a still image with a camera arranged at the center of the photographing unit A23 in FIG.
A32 is an image correction means, which is composed of an image recording memory, a CPU, etc., temporarily records the image obtained by the photographing means A31, and rotates the image based on the camera angle information of the position detection means A34 so that it is easy to see Image data is formed.
For example, in the case where the reference posture image is rotated by moving the housing A21 with respect to the reference posture of the image taken by the photographing unit A23 and projected on the monitor, the angular velocity sensor or the like is connected to each axis. By detecting the amount of change in the rotation angle from the angular velocity at, and rotating and displaying the image by an angle obtained by subtracting the amount of change in the rotation angle, an easy-to-view image can be formed.

In some cases, the image can be corrected only by rotation around the y-axis shown in FIG.
When photographing an intraoral image reflected by a reflector, the camera of the photographing unit A23 is always facing the direction of the reflector, so the image rotates around the y-axis coordinate shown in FIG. This is because the rotation accompanying the rotation of the housing is the main.
Accordingly, at least in the state where the main body A11 and the reflecting mirror unit A10 are changed in the x-axis, y-axis, and z-axis directions, the image displayed on the xz plane is corrected to be in a certain direction in the image display unit. May be preferred.
Reference numeral 33 denotes image display means, which may be a computer monitor or other dedicated monitor that can display the output image of the image correction means A32.

A34 is a position detection means, which is composed of the position sensors A25a, A25b, etc. of FIG. 10, and specifically, a rate gyro that outputs angular velocity, a rate integration gyro that outputs angles, an attitude gyro, a MEMS type, and other mechanical types An acceleration sensor such as an optical angular velocity sensor, a piezoresistive type, a capacitance type, or a heat detection type MEMS sensor can be used.
Next, the operation of FIG. 11 will be described with reference to FIGS.
The coordinate axes shown in FIG. 12 indicate the case where one position sensor corresponds to three axes. When the position sensor corresponds to one axis, two axes, etc., the coordinate axis corresponding to each part of the position sensor is Is set.
A case A21 shown in FIG. 10 is held, and a reflecting mirror A22H is inserted into the oral cavity to photograph a target site. At that time, the initial posture state is recorded, for example, by operating a switch attached on the casing A21. An example of coordinates in the desired posture state is indicated by A410 in FIG.
A coordinate axis is formed by installing the position sensor. In this embodiment, A421 is the x axis, A431 is the y axis, and A441 is the z axis.

A451 is a gravitational acceleration vector, which is an example of a combined posture vector when the acceleration sensor is stationary. Therefore, the gravitational acceleration vector may not be used when the acceleration sensor is not used.
For example, the intraoral camera shows coordinates in a state close to being upright in order to photograph the side surface of the back tooth.
A422 is an x-axis, A432 is a y-axis, A442 is a z-axis, and when an acceleration sensor is used, A452 can indicate a gravitational acceleration vector.
The reflecting mirror 22H is moved to a target site in the oral cavity. An example of how to move is shown in FIG. The intraoral camera composed of the reflecting mirror 22H and the casing moves vertically and horizontally as indicated by A411, A412, and A413, and the imaging means A31 captures intraoral images in each state as still images and moving images. Output to the image correction means A32.

For example, the position detection unit A34 outputs the initial posture information to the image correction unit A32 in the x, y, and z directions. Note that, in the case of the configuration in which the image of the reflecting mirror is displayed as in this embodiment, the camera is oriented in the direction of the reflecting mirror, so the camera image is reversed or difficult to see due to rotation around the y axis. Since there are many factors, there are cases where only a single-axis type position sensor may be used. The image correction means A32 associates this initial posture information with the image, outputs it to the image display means A33, and displays it.
As shown in FIG. 12, when the intraoral camera is moved like A411, A412, and A413 and the intraoral area is imaged, the imaging unit A31 outputs an image corresponding to each posture. When the camera rotates around the y axis, the image is taken upside down and an image corresponding to the shooting state is output.

The position detection means A34 detects angular velocities around the x-axis, y-axis, and z-axis from the position sensors A25a and A25b (for example, in the case of a gyro sensor), and the amount of change in angle around the x-axis (Δθyz) from this angular velocity. ), An angle change amount around the y-axis (Δθxz), and an angle change amount around the z-axis (Δθxy).
These changes are output to the image correction means A32. The image correction unit A32 rotates the image based on, for example, the amount of change in angle based on the image data input from the imaging unit A31 and the position information output from the position detection unit 34, and the first image Return to the state.
Therefore, even if the image display means A33 rotates the camera and shoots the same intraoral object as a moving image or a still image, the image is always displayed in the initial setting state and the display content image is changed. Can do.

Gingival-tooth boundary detection means

The present invention includes a configuration for making it possible to clearly set and detect the boundary between teeth and gums.
Since many images of intraoral camera images are taken while illuminating a dark and narrow space, for example, as a means for forming a panoramic image by computer image processing shown in FIGS. 1 and 3, a common part between images is detected. When it is performed or when a dental treatment is explained to a patient, it may be difficult to identify due to an approximate color such as the gingival-tooth boundary or the influence of saliva.
Therefore, the contour extraction means for extracting the tooth contour of the real image, the color component image conversion means for converting the real image into component colors and clarifying the shape of the teeth and gums, and the color component image conversion means. The combination of the image and the synthetic image forming means that combines the contour extraction image enables the tooth and gingival contours to be extracted even in places where the teeth are dirty or where illumination is insufficient.
The contour extraction means includes, for example, a means for performing a two-dimensional Fourier transform on an image used in the phase-only correlation method, and an inverse Fourier transform by detecting only a phase signal in a frequency domain indicating a change in light and shade of the image after the Fourier transform. It is comprised by the means to perform. In addition, a Z conversion system or a Laplace conversion system may be used.

  The color component image conversion means includes, for example, RGB color system, La * b * color system, HSV color system, XYZ color system, xyY color system, L * u * v * color system, Munsell color system Image, Ostwald color system, NCS (Natural Color System), DIN color system, etc. Thus, regardless of the means for forming an image based on the component color, or by combining the component colors regardless of the color system, a component color that has a clearer shape is formed to form an image based on the component color. Means. The selection is preferably determined in advance by measuring a component color suitable for an intraoral image. For example, any one of an L component image, an a * component image, or a b * component image of the La * b * color system Any one of the H (Hue) component image, S (Saturation / Chroma) component image, and V (Brightness / Lightness / Value) component image of the HSV color system In other words, it may be preferable to combine component colors of different color systems. Component color selection or combination synthesis may be performed as long as conversion to a component color in which the shape is clearly seen or synthesis of a plurality of component colors is performed.

The component image includes an image that emphasizes a component, for example, obtained by adjusting a numerical value indicating a component value on a program.
In addition to selecting a component image whose shape is clearly recognizable, it is colored using a combination of colors that allows a person to recognize the boundary more. For example, it is shown that an R (red) component image is applied to the gingiva and a G (green) component image is applied to the tooth.
The component colors shown here are not limited to those detected from the image, but may be newly colored using a color that makes the boundary clear.
Then, the composite image forming unit uses the contour image obtained by the contour extraction unit and the image obtained by the color component image conversion unit as the same color system except for the contour part of the contour image, for example, as in the chroma key method. The color component image is synthesized by being transmitted. In addition, after the synthesis, in order to further enhance the color component image, the gingiva is changed to a red color or deeply emphasized so that the boundary can be further visually or mechanically distinguished, and the teeth are changed to green. Alternatively, a means for emphasizing the image may be used. Depending on the color component, if there is one that converts the gingival color to red and the tooth color to green, the color component image may be converted. All of these means are preferably realized by computer software, but may be configured by a custom or semi-custom IC such as a gate array or a PLD (Programmable Logic Device).

Next, the embodiment shown in FIG. 13 will be described.
Reference numeral 2901 denotes an image input means for inputting a still image taken by an intraoral camera, for example. The still image image input by the image input unit is output to the filter unit 2102. Reference numeral 2902 denotes a filter means, which is indicated by an edge enhancement filter such as an unsharp filter. In addition, there may be a case where a filter for enhancing the density of an image may be used.
The image subjected to the filter processing by the filter unit 2902 is output to the contour extraction unit 2903 and the component color image conversion unit 2904.
2903 is a contour extraction means, which comprises a two-dimensional Fourier transform means, a phase signal detection means, and an inverse Fourier transform means. These means are in a software library such as open-CV (manufactured by Intel). An example of executing a combination of program modules is illustrated.
Reference numeral 2904 denotes component color image conversion means for converting into La * b * color system, HSV color system a * component color, b * component color, H component color S component color V component color as described above. Image forming means and component color synthesizing means for synthesizing these component colors. Although the hue changes, an image with clear shapes of teeth and gums is formed. In addition, this change in hue may make it possible to discover tooth stains and decayed teeth.

The contour image obtained by the contour extraction unit 2903 and the image converted into the component color image by the component color image conversion unit 2904 are output to the composite display unit 2905.
Reference numeral 2905 denotes synthesis display means for synthesizing the contour image output from the contour extraction means 2903 and the output component color image of the component color image conversion means. In this synthesis, for example, colors other than the contour of the contour image are made similar colors, the similar colors are made transparent, and synthesized with the component color image that becomes the background image. The synthesized image is displayed on a computer monitor.
In some cases, an image with a clear boundary between the gingiva and the tooth can be obtained by a series of these synthesis processes. Further, for example, the boundary KL between the two teeth at the center is detected as a guide when combining or combining the images in which the mark ML is approximately in the center of the image (for example, the image 906e and the image 301c). In this case, this embodiment is preferably used. Further, the contour image obtained by detecting the phase signal by Fourier transform and the actual image may be synthesized as they are. In addition, since an image with a clear shape can be obtained using only the component color image, an image with a clear boundary between the gingiva and the tooth can be obtained only with the image obtained by the component color image detection means. Such an image in which the boundary between teeth and gingiva is clear can be used as a mark for various synthesis operations, or used for explanation to a patient and for dental treatment.

  The present invention proposes a dental system that can renew patient's intraoral information by providing the patient with easy-to-see intraoral information in order to achieve oral health in dental practice, It is effectively used in the field of dentistry.

101 Intraoral camera 102 Reflector 103 Reflector unit 104 Main body 105 Imaging unit ML Mark IG Correction display unit RM Outline

Claims (4)

Continuous image sequence forming means for continuously capturing images of the side surfaces of the dentition to form an image sequence, and an image that forms the center of the overall synthesis using the image sequence formed by the continuous image sequence forming means as a partial dentition image A side dentition image forming unit that forms a plurality of partial dentition images by combining them from each other, and a plurality of partial dentition images formed by the side dentition image formation unit are connected based on an image that is the center of overall synthesis An intraoral radiographing display system comprising a side dentition image synthesizing unit that synthesizes and forms an entire dentition. Mark setting means for setting a recognizable mark on a photographed image at a predetermined position on the dentition, continuous image sequence forming means for forming a continuous image sequence on one side and the other side of the dentition, For each of the continuous image sequence on the side surface and the continuous image sequence on the other side surface, the one side dentition image and the other side dentition image are formed by synthesizing from the image where the mark is in a predetermined position. Side dentition image forming means,
Side dentition image synthesizing means for linking and synthesizing the one side dentition image and the other side dentition image based on a characteristic part on the dentition in the still image in which the mark is in a predetermined position. The intraoral imaging display system according to claim 1.   X-ray image display means for displaying an X-ray image of a tooth corresponding to the side surface dentition image; virtual dentition display means for displaying a tooth dentition corresponding to the side surface dentition image; The intraoral imaging display system according to claim 1.   The intraoral radiography display system according to claim 1, wherein the handheld radiographing unit that images the intraoral area, and the radiographing unit includes an irradiation unit that irradiates illumination light that indicates an imaging region.

Images