JP2023143585A - Data processing apparatus, data processing method, and data processing program - Google Patents

Abstract

To provide a data processing apparatus, a data processing method, and a data processing program for easily and appropriately extracting three-dimensional data of a predetermined object inside the oral cavity.SOLUTION: In a data processing system, a data processing apparatus 1 includes: an input unit to which three-dimensional data including position information of each point of a point group indicating a surface of at least one object is input; and an arithmetic unit. The arithmetic unit includes: a two-dimensional data generation unit that processes the three-dimensional data input from the input unit and generates two-dimensional data from the three-dimensional data; an identification unit that identifies a predetermined object among the at least one object based on the two-dimensional data; a removal unit that extracts unnecessary three-dimensional data by extracting the position information of each point group indicating the surface of the identified predetermined object; and a combining unit that acquires the three-dimensional data of one scan from the removal unit every time the three-dimensional data of one scan is input, and generates combined three-dimensional data of a plurality of scans by combining accumulated pieces of the three-dimensional data of the plurality of scans.SELECTED DRAWING: Figure 9

Description

The present disclosure relates to a data processing device, a data processing method, and a data processing program that process data including position information of at least one object in the oral cavity.

BACKGROUND ART Conventionally, in the dental field, a technique is known in which three-dimensional data of an object such as a tooth is obtained by scanning the inside of the oral cavity with a three-dimensional scanner. For example, Patent Document 1 discloses a method of recording the shape of a tooth by capturing an image of the tooth using a three-dimensional scanner.

Japanese Patent Application Publication No. 2000-74635

According to the method disclosed in Patent Document 1, a user can measure teeth in the oral cavity using a three-dimensional image generated based on three-dimensional data acquired by a three-dimensional scanner, and While checking the three-dimensional image showing the tooth, students can identify the type of tooth based on their own knowledge. However, there is a need for a technology that can appropriately extract three-dimensional data of a predetermined object in the oral cavity, such as a tooth, without the user identifying the predetermined object in the oral cavity.

The present disclosure has been made to solve such problems, and aims to provide a technology that can easily and appropriately extract three-dimensional data of a predetermined object in the oral cavity.

According to an example of the present disclosure, a data processing device is provided that processes data including position information of at least one object within the oral cavity. The data processing device includes an input section into which three-dimensional data including position information of each point group indicating the surface of at least one object is input, and a calculation section that processes the three-dimensional data input from the input section. . The calculation unit generates two-dimensional data from the three-dimensional data, and based on the two-dimensional data, selects at least one of the at least one object to be inserted into the oral cavity, the tongue, the lips, and the mucous membrane. A predetermined object is identified, predetermined three-dimensional data including position information of each point group indicating the surface of the identified predetermined object is extracted, and predetermined three-dimensional data is extracted from the three-dimensional data input from the input unit. Remove data.

According to an example of the present disclosure, there is provided a data processing method for processing data including position information of at least one object in an oral cavity using a computer. The data processing method includes the steps of inputting three-dimensional data including position information of each of a point group indicating a surface of at least one object, and processing the input three-dimensional data. The processing step includes the step of generating two-dimensional data from the three-dimensional data, and based on the two-dimensional data, at least one of at least one object inserted into the oral cavity, a tongue, a lip, and a mucous membrane. a step of identifying a predetermined object including one; a step of extracting predetermined three-dimensional data including positional information of each point group indicating the surface of the identified predetermined object; and from the input three-dimensional data, and removing predetermined three-dimensional data.

According to an example of the present disclosure, a data processing program is provided that processes data including position information of at least one object within the oral cavity. The data processing program causes the computer to execute the steps of inputting three-dimensional data including position information of each of a group of points indicating the surface of at least one object, and processing the input three-dimensional data. The processing step includes the step of generating two-dimensional data from the three-dimensional data, and based on the two-dimensional data, at least one of at least one object inserted into the oral cavity, a tongue, a lip, and a mucous membrane. a step of identifying a predetermined object including one; a step of extracting predetermined three-dimensional data including positional information of each point group indicating the surface of the identified predetermined object; and from the input three-dimensional data, and removing predetermined three-dimensional data.

According to the present disclosure, a user can easily and appropriately extract three-dimensional data of a predetermined object in the oral cavity.

1 is a diagram illustrating an application example of a data processing system and a data processing device according to Embodiment 1. FIG. 1 is a block diagram showing a hardware configuration of a data processing device according to Embodiment 1. FIG. 1 is a diagram showing the configuration of a three-dimensional scanner according to Embodiment 1. FIG. FIG. 3 is a diagram for explaining an example in which the three-dimensional scanner according to the first embodiment acquires three-dimensional data based on a confocal method. FIG. 3 is a diagram for explaining a scanning method using a three-dimensional scanner. FIG. 3 is a diagram showing objects in each scan range acquired by the three-dimensional scanner according to the first embodiment. FIG. 2 is a diagram showing how an object is scanned using a three-dimensional scanner. FIG. 2 is a diagram showing how an object is scanned using a three-dimensional scanner. 1 is a block diagram showing a functional configuration of a data processing device according to Embodiment 1. FIG. FIG. 3 is a diagram for explaining three-dimensional data input to the data processing device according to the first embodiment. FIG. 3 is a diagram for explaining learning data used when performing machine learning on the estimation model according to the first embodiment. FIG. 3 is a diagram for explaining the correspondence between each of a plurality of objects and a correct label. FIG. 3 is a diagram for explaining combined data after unnecessary object removal generated by the data processing device according to the first embodiment. 3 is a flowchart for explaining an example of processing executed by the data processing device according to the first embodiment. FIG. 7 is a diagram for explaining an example in which the three-dimensional scanner according to the second embodiment acquires three-dimensional data based on a triangulation method. 7 is a diagram showing a two-dimensional image viewed from an arbitrary viewpoint based on three-dimensional data acquired by a three-dimensional scanner according to a second embodiment. FIG.

<Embodiment 1>
Embodiment 1 of the present disclosure will be described in detail with reference to the drawings. Note that the same or corresponding parts in the figures are designated by the same reference numerals, and the description thereof will not be repeated.

[Application example]
An application example of the data processing system 10 and data processing device 1 according to the first embodiment will be described with reference to FIG. FIG. 1 is a diagram showing an application example of a data processing system 10 and a data processing device 1 according to the first embodiment.

As shown in FIG. 1, a user can acquire three-dimensional data of a plurality of objects in the oral cavity by scanning the oral cavity of a subject using the three-dimensional scanner 2. "Users" include operators such as dentists, dental assistants, teachers or students at dental universities, dental technicians, manufacturers' engineers, and manufacturing factory workers, who use the 3D scanner 2 to scan objects such as teeth. Any person who acquires three-dimensional data may be used. The "target person" may be any person who can be scanned by the three-dimensional scanner 2, such as a patient at a dental clinic or a test subject at a dental university. The "object" may be anything that can be scanned by the three-dimensional scanner 2, such as teeth in the subject's oral cavity.

The data processing system 10 includes a data processing device 1 and a three-dimensional scanner 2. A display 3, a keyboard 4, and a mouse 5 are connected to the data processing device 1.

The three-dimensional scanner 2 is an imaging device that images the inside of the oral cavity, and acquires three-dimensional data of an object using a built-in three-dimensional camera. Specifically, by scanning the inside of the oral cavity, the three-dimensional scanner 2 obtains position information (vertical direction, horizontal direction, height) of each point group (multiple points) indicating the surface of an object as three-dimensional data. The coordinates of each axis of the direction) are acquired using an optical sensor or the like. That is, the three-dimensional data is position data that includes position information of each position of a group of points forming the surface of an object.

The measurement range that the 3D scanner 2 can measure at one time is limited, so if the user wants to obtain 3D data of the entire dentition (dental arch) in the oral cavity, the user can move the 3D scanner 2 along the dentition. The inside of the mouth is scanned multiple times by moving the scanner around the mouth.

The data processing device 1 generates two-dimensional image data corresponding to a two-dimensional image seen from an arbitrary viewpoint based on the three-dimensional data acquired by the three-dimensional scanner 2, and displays the generated two-dimensional image on the display 3. By doing so, it is possible to show the user a two-dimensional projection of the surface of the object viewed from a specific direction.

Furthermore, the data processing device 1 outputs three-dimensional data to a dental laboratory. In the dental laboratory, a dental technician creates a tooth model such as a prosthesis based on the three-dimensional data acquired from the data processing device 1. Note that if an automatic manufacturing device capable of automatically manufacturing tooth models, such as a milling machine and a 3D printer, is installed in the dental clinic, the data processing device 1 may output the three-dimensional data to the automatic manufacturing device. .

[Hardware configuration of data processing device]
The hardware configuration of the data processing device 1 according to the first embodiment will be described with reference to FIG. 2. FIG. 2 is a block diagram showing the hardware configuration of the data processing device 1 according to the first embodiment. The data processing device 1 may be realized, for example, by a general-purpose computer or a computer dedicated to the data processing system 10.

As shown in FIG. 2, the data processing device 1 includes a calculation section 11, a storage section 12, a scanner interface 13, a communication section 14, a display interface 15, and a peripheral device interface 16 as main hardware elements. , and a reading section 17.

The calculation unit 11 is a calculation main body (calculation device) that performs various processes by executing various programs, and is an example of a computer. The calculation unit 11 includes, for example, a CPU (Central Processing Unit), an FPGA (Field-Programmable Gate Array), and a GPU (Graphics Processing Unit). Note that the calculation unit 11 may be composed of at least one of a CPU, an FPGA, and a GPU, or may be composed of a CPU and an FPGA, an FPGA and a GPU, a CPU and a GPU, or a CPU, an FPGA, and a GPU. Good too. Further, the calculation unit 11 may be configured with a calculation circuit (processing circuitry). Note that the calculation unit 11 may be configured with one chip or may be configured with a plurality of chips. Further, all or part of the functions of the calculation unit 11 may be provided in a server device (for example, a cloud-type server device) not shown.

The storage unit 12 includes a volatile storage area (for example, a working area) that temporarily stores program codes, work memory, etc. when the calculation unit 11 executes an arbitrary program. For example, the storage unit 12 is configured with a volatile memory device such as a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory). Furthermore, the storage unit 12 includes a nonvolatile storage area. For example, the storage unit 12 is configured with a nonvolatile memory device such as a ROM (Read Only Memory), a hard disk, or an SSD (Solid State Drive).

Note that in this embodiment, an example is shown in which the volatile storage area and the nonvolatile storage area are included in the same storage unit 12, but the volatile storage area and the nonvolatile storage area are They may be included in different storage units. For example, the calculation unit 11 may include a volatile storage area, and the storage unit 12 may include a nonvolatile storage area. The data processing device 1 may include a microcomputer including a calculation section 11 and a storage section 12.

The storage unit 12 stores a data processing program 121 and an estimation model 122. The data processing program 121 includes an identification system in which the calculation unit 11 generates two-dimensional data from the three-dimensional data acquired by the three-dimensional scanner 2, and identifies objects in the oral cavity based on the two-dimensional data and the estimation model 122. Processing is described.

Estimation model 122 includes a neural network 1221 and parameters 1222 used by neural network 1221. The estimation model 122 uses learning data including two-dimensional data including position information of each of a plurality of objects in the oral cavity and correct labels indicating the types of each of the plurality of objects. Machine learning is being used to estimate the type of object.

Specifically, in the learning phase, when the estimation model 122 receives two-dimensional data including position information of each object in the oral cavity, the neural network 1221 extracts the features of each object based on the two-dimensional data. , estimate the type of each object based on the extracted features. Then, based on the estimated type of each object and the correct label indicating the type of each object associated with the two-dimensional data, the estimation model 122 does not update the parameter 1222 if the two match; If they do not match, the parameters 1222 are updated so that they match, thereby optimizing the parameters 1222. In this way, the estimation model 122 undergoes machine learning by optimizing the parameters 1222 based on learning data including two-dimensional data as input data and the type of each object as correct data. Thereby, the estimation model 122 can estimate each of the plurality of objects in the oral cavity based on the two-dimensional data of each of the plurality of objects in the oral cavity.

Note that the estimated model 122 that has been optimized by learning such an estimated model 122 is also particularly referred to as a "trained model." That is, while the pre-learning estimation model 122 and the trained estimation model 122 are collectively referred to as the "estimated model", the trained estimation model 122 is also particularly referred to as the "trained model".

The estimation model 122 includes a program for the calculation unit 11 to execute estimation processing and learning processing. In the first embodiment, programs that perform image-specific processing include, for example, U-Net, SegNet, ENet, ErfNet, VoxNet, 3D ShapeNets, 3D U-Net, Multi-View CNN, RotationNet, OctNet, and PointCNN. , FusionNet, PointNet, PointNet++, SSCNet, MarrNet, VoxelNet, PAConv, VGGNet, ResNet, DGCNN, KPConv, FCGF, ModelNet40, ShapeNet, SemanticKITTI, SunRGB-D, VoteNet, LinkNet, Lambda Network, PREDATOR, 3D Medical Point Transformer, and PCT etc. are used for the estimation model 122 program, but other programs such as a forward propagation neural network, a recurrent neural network, a graph neural network, an attention mechanism, a transformer, etc. are used for the estimation model 122 program. It's okay.

The scanner interface 13 is an interface for connecting the three-dimensional scanner 2 and realizes data input/output between the data processing device 1 and the three-dimensional scanner 2. The data processing device 1 and the three-dimensional scanner 2 are connected via a wire using a cable or wirelessly (WiFi, BlueTooth (registered trademark), etc.).

The communication unit 14 transmits and receives data to and from the above-mentioned dental laboratory or automatic manufacturing device via wired or wireless communication. For example, the data processing device 1 transmits data for generating a prosthesis generated based on three-dimensional data to a dental laboratory or an automatic manufacturing device via the communication unit 14.

The display interface 15 is an interface for connecting the display 3 and realizes data input/output between the data processing device 1 and the display 3.

The peripheral device interface 16 is an interface for connecting peripheral devices such as the keyboard 4 and the mouse 5, and realizes data input/output between the data processing device 1 and the peripheral devices.

The reading unit 17 reads various data stored in the removable disk 20, which is a storage medium. For example, the reading unit 17 may acquire the data processing program 121 from the removable disk 20.

[3D scanner configuration]
The configuration of the three-dimensional scanner 2 according to the first embodiment will be described with reference to FIGS. 3 and 4. FIG. 3 is a diagram showing the configuration of the three-dimensional scanner 2 according to the first embodiment. FIG. 4 is a diagram for explaining an example in which the three-dimensional scanner 2 according to the first embodiment acquires three-dimensional data based on the confocal method.

As shown in FIG. 3, the three-dimensional scanner 2 is a handheld handpiece, and includes a housing 21, a probe 22 detachably connected to the housing 21, and a control device 40.

The probe 22 is inserted into the oral cavity and projects light having a pattern (hereinafter also simply referred to as a "pattern") onto objects within the oral cavity. The probe 22 guides reflected light from the object onto which the pattern is projected into the housing 21 .

The three-dimensional scanner 2 includes a light source 23 , a lens 24 , an optical sensor 25 , a prism 26 , a counterweight 27 , and an opening 29 inside a housing 21 . Note that in FIGS. 3 and 4, for convenience of explanation, plane directions parallel to the opening 29 are defined by the X axis and the Y axis. Further, a direction perpendicular to the X-axis and the Y-axis is defined as the Z-axis.

The light source 23 includes a laser element, an LED (Light Emitting Diode), or the like. The light (optical axis L) from the light source 23 passes through the prism 26 and the lens 24, is reflected by the reflection section 28 provided on the probe 22, and is output from the opening 29. The light output from the opening 29 is irradiated onto an object along the Z-axis direction and reflected by the object. That is, the optical axis direction of the light output from the three-dimensional scanner 2 coincides with the Z-axis direction and is perpendicular to the plane direction consisting of the X-axis and the Y-axis.

The light reflected by the object enters the housing 21 again through the opening 29 and the reflecting section 28, passes through the lens 24, and enters the prism 26. The prism 26 changes the traveling direction of light from the object to the direction in which the optical sensor 25 is located. The light whose traveling direction has been changed by the prism 26 is detected by the optical sensor 25.

When acquiring three-dimensional data of an object using confocal technology, light having a pattern such as a checkerboard pattern that has passed through a pattern generation element (not shown) provided between the lens 24 and the object is scanned. Projected onto an object within range R. When the lens 24 moves linearly back and forth on the same straight line, the focal position of the pattern projected onto the object changes on the Z-axis. The optical sensor 25 detects light from an object every time the focal position changes on the Z-axis.

The control device 40 includes, for example, a CPU, a ROM, a RAM, and the like, and controls the processing performed by the three-dimensional scanner 2. Note that the control device 40 may be configured with an FPGA or a GPU. Further, the control device 40 may be composed of at least one of a CPU, an FPGA, and a GPU, a CPU and an FPGA, a FPGA and a GPU, a CPU and a GPU, or a CPU, an FPGA, and a GPU. Good too. Further, the control device 40 may be configured with processing circuitry. The control device 40 calculates position information for each point group indicating the surface of the object based on the position of the lens 24 and the detection result of the optical sensor 25 at that time.

Thereby, the three-dimensional scanner 2 acquires position information (X coordinate and Y coordinate) of each point group indicating the surface of the object on the XY plane within the scan range R. As shown in FIG. 4, when an object is viewed in the Z-axis direction from a virtual viewpoint between the three-dimensional scanner 2 and the object, a two-dimensional image showing the surface of the object can be expressed in the XY plane. The three-dimensional scanner 2 obtains three-dimensional data (X-coordinate, Y-coordinate, and Z coordinate). One scan corresponds to acquiring three-dimensional data (X coordinate, Y coordinate, and Z coordinate) of an object once while the position of the probe 22 of the three-dimensional scanner 2 is fixed.

More specifically, when one scan is performed in which three-dimensional data is acquired with a fixed optical axis without moving the three-dimensional scanner 2, the control device 40 controls the surface of the object to be scanned. Three-dimensional position information is given to each of the shown point groups, with the optical axis direction as the Z coordinate, and the plane directions orthogonal to the optical axis direction (Z-axis direction) as the X and Y coordinates. When multiple scans are performed by the three-dimensional scanner 2, the control device 40 performs matching of overlapping portions when combining the three-dimensional data of the point cloud acquired by each scan between the multiple scans. Combine 3D data based on the shape of the object. When the connection is completed or at a certain timing, the control device 40 re-assigns X coordinates, Y coordinates, and Z coordinates based on an arbitrary origin to the three-dimensional data of the connected point group. , obtain three-dimensional data of a point cloud that includes position information of a uniform object as a whole.

Three-dimensional data of an object acquired by the three-dimensional scanner 2 is input to the data processing device 1 via the scanner interface 13. Note that the data processing device 1 may include all or some of the functions of the control device 40. For example, the calculation unit 11 of the data processing device 1 may have the functions of the control device 40.

[An example of scanning using a 3D scanner]
An example of scanning by the three-dimensional scanner 2 will be described with reference to FIGS. 5 to 8.

FIG. 5 is a diagram for explaining a scanning method using the three-dimensional scanner 2. The scanning range of the three-dimensional scanner 2 is limited by the size of the probe 22 that can be inserted into the oral cavity. Therefore, the user inserts the probe 22 into the oral cavity and scans the oral cavity multiple times by moving the probe 22 within the oral cavity along the row of teeth.

For example, as shown in FIG. 5, by moving the probe 22 within the oral cavity, the user can sequentially switch the scan range such as R1, R2, R3,...Rn to scan various parts of the oral cavity. Acquire three-dimensional data. More specifically, the user scans some teeth by moving the probe 22 from the lingual side of the tooth, through the occlusal surface, to the labial side of the tooth, and performs such scanning from the left molar side to the anterior tooth side. A plurality of teeth are sequentially scanned by moving the probe 22 toward the right molar side via the . Note that since the way the probe 22 is moved within the oral cavity differs for each user or for each dental practice, the portion of the oral cavity from which three-dimensional data is acquired and the order of acquisition may vary.

FIG. 6 is a diagram showing objects in each scan range acquired by the three-dimensional scanner 2 according to the first embodiment. As shown in FIG. 6, when the user scans an object while moving the probe 22, the three-dimensional scanner 2 can acquire three-dimensional data of the object included in each scan range. For example, the three-dimensional scanner 2 can acquire three-dimensional data in one scan range for each scan, and in the example of FIG. 6, it can acquire three-dimensional data in each of the scan ranges R11 to R15. can. The three-dimensional scanner 2 combines a plurality of three-dimensional data corresponding to each of the plurality of scan ranges R11 to R15 obtained by multiple scans, thereby calculating the entire object included in the plurality of scan ranges R11 to R15. Three-dimensional data can be obtained.

FIGS. 7 and 8 are diagrams showing how the three-dimensional scanner 2 is used to scan an object. As shown in Figures 7 and 8, in the oral cavity, the tongue, the frenulum between the mandibular tooth row and the tongue, the mandibular tooth row, the frenulum between the mandibular tooth row and the lower lip (not shown) (omitted), lower lip, hard palate, frenulum between the maxillary teeth and hard palate, maxillary teeth, frenulum between the maxillary teeth and upper lip (not shown), It includes multiple objects such as the upper lip, gums, mucous membranes, and prosthetics (metal teeth, ceramic teeth, resin teeth). During a scan by the 3D scanner 2, there may be any objects such as the operator's fingers or medical instruments, or the patient's tongue, lips, or mucous membrane (back of the cheek) between the object to be scanned, such as a tooth, and the 3D scanner 2. There are cases where an unnecessary object gets in and the three-dimensional scanner 2 is unable to properly acquire three-dimensional data of the object. For example, in the example shown in FIG. 6, a finger, which is an unnecessary object, has entered the range R12 to R14. Note that the fingers are not limited to the operator's real fingers, but also include fingers worn by the operator with gloves. Furthermore, examples of medical instruments include dental instruments such as vacuums, mouth opening devices, and tongue depressors.

For example, as shown in FIGS. 7 and 8, when the probe 22 of the three-dimensional scanner 2 is inserted into the oral cavity, a finger is inserted between the teeth and the lips to press the soft tissues in the oral cavity. Sometimes.

Specifically, as shown in FIG. 7, when acquiring three-dimensional data of the side surface of the teeth on the labial side, the user inserts the probe 22 into the gap between the teeth and the lips. Insert your finger into the gap between the lips and press down on the soft tissue to prevent it from interfering with the scan. Further, as shown in FIG. 8, when acquiring three-dimensional data of the side surface of teeth on the lingual side, the user inserts the probe 22 into the gap between the tooth row and the tongue. Insert your finger into the gap to press down on the soft tissue to prevent it from interfering with the scan. Note that not only the finger but also a medical instrument for pressing soft tissue may be inserted into the gap between the teeth and the lips or the gap between the teeth and the tongue. In this way, when scanning the inside of the oral cavity with the finger pressing the soft tissue in contact with the dentition, three-dimensional data is generated with the finger, which is an unnecessary object, entering the range R12 to R14, as shown in Figure 6. is obtained.

As described above, in dental treatment, the inside of the oral cavity is usually scanned with an insertion object such as a finger or a medical instrument inserted into the oral cavity. As shown in R12 to R14, the inserted object may enter the scanning range and the three-dimensional scanner 2 may not be able to properly acquire three-dimensional data of the object.

Therefore, the data processing device 1 according to the first embodiment utilizes AI (Artificial Intelligence) to detect teeth, tongue, lips, frenulum, gums, mucous membranes, prosthetics (metal teeth, ceramic teeth, etc.) in the oral cavity. Identify each type of multiple objects such as teeth, resin teeth), and insertion objects inserted into the oral cavity, and extract and delete 3D data of objects that are unnecessary for dental treatment based on the identification results. It is configured as follows. The specific functions of the data processing device 1 will be described below.

[Functional configuration of data processing device]
The functional configuration of the data processing device 1 according to the first embodiment will be described with reference to FIG. 9. FIG. 9 is a block diagram showing the functional configuration of the data processing device 1 according to the first embodiment.

As shown in FIG. 9, the data processing device 1 includes an input section 1101, a two-dimensional data generation section 1106, an identification section 1102, a removal section 1103, a combination section 1104, and an image generation section as main functional sections. 1105 and a storage unit 12.

The input unit 1101 is a functional unit of the scanner interface 13 and acquires three-dimensional data for each scan acquired by the three-dimensional scanner 2. Note that the input unit 1101 may be a functional unit of the communication unit 14, the peripheral device interface 16, or the reading unit 17. For example, when the input unit 1101 is a functional unit of the communication unit 14, the communication unit 14 acquires three-dimensional data from an external device via wired communication or wireless communication. Note that the external device may be a server device installed in a dental clinic, or a cloud-type server device installed in a location other than the dental clinic. When the input unit 1101 is a functional unit of the peripheral device interface 16, the peripheral device interface 16 acquires three-dimensional data input by the user using the keyboard 4 and mouse 5. When the input unit 1101 is a functional unit of the reading unit 17, the reading unit 17 acquires three-dimensional data stored on the removable disk 20.

Here, three-dimensional data input to the data processing device 1 will be explained with reference to FIG. 10. FIG. 10 is a diagram for explaining three-dimensional data input to the data processing device 1 according to the first embodiment. As shown in FIG. 10, the three-dimensional data for each scan input to the input unit 1101 includes the X coordinate, Y coordinate, and Z coordinate associated with each point group indicating the surface of the object within the scan range. It includes position information of coordinates and normal information of X component, Y component, and Z component. Although not shown, the three-dimensional data also includes color information associated with each point group indicating the surface of the object within the scan range.

As explained using FIG. 4, the position information is the X coordinate, Y coordinate, and Z coordinate of each point group indicating the surface of the object included in the scan range. The normal information is the X component, Y component, and Z component of a normal line perpendicular to the tangent line at the point of interest, focusing on one point included in the point group. Note that a known technique such as principal component analysis may be used to generate a normal to one point included in the point group.

Returning to FIG. 9, the two-dimensional data generation unit 1106 is a functional unit of the calculation unit 11. The two-dimensional data generation unit 1106 generates two-dimensional data from one scan worth of three-dimensional data input from the input unit 1101.

Specifically, as explained using FIG. 4, the three-dimensional scanner 2 using the confocal technique detects a point indicating the surface of the object by detecting light from the object included in the scan range. Three-dimensional data including position information for each of the groups can be obtained. As shown in FIG. 10, the three-dimensional data acquired by the three-dimensional scanner 2 and input from the input unit 1101 includes, as position information, the X coordinate, Y coordinate, and Z coordinate of each point group indicating the surface of the object. and color information at that position. The two-dimensional data generation unit 1106 generates two-dimensional data for each point group indicating the surface of the object using only the X and Y coordinates of the position information included in the three-dimensional data input from the input unit 1101. do. The X and Y coordinates are the pixel positions of the two-dimensional data, and the color information is the pixel value at the pixel positions. That is, the two-dimensional data generated based on the three-dimensional data by the two-dimensional data generation unit 1106 is based on at least one object included in the scan range viewed from a position a certain distance away (for example, the virtual viewpoint position in FIG. 4). This is data of a two-dimensional image showing the appearance of the at least one object in the case where the at least one object is viewed from the outside. Two-dimensional data generation section 1106 outputs the generated two-dimensional data to identification section 1102.

Note that the two-dimensional data generation unit 1106 generates each point group indicating the surface of the object using the X coordinate, Y coordinate, and Z coordinate of the position information included in the three-dimensional data input from the input unit 1101. , two-dimensional data may be generated as a distance image. That is, the two-dimensional data generation unit 1106 may use the X coordinate and the Y coordinate as a pixel position of the two-dimensional data, and convert the Z coordinate into a pixel value at that pixel position. A distance image is two-dimensional data whose Z coordinate is expressed by color information including the shading of the image. Furthermore, the two-dimensional data generation unit 1106 uses two-dimensional data representing the surface of the object generated using only the X and Y coordinates, and a distance image generated using the X, Y, and Z coordinates. Both two-dimensional data may be generated. In this way, using the Z coordinate as color information is useful when humans visually view a two-dimensional image (distance image). For example, in a distance image (an image in which the Z coordinate is converted to a pixel value) when scanning back teeth from above, the area near the occlusal surface of the teeth will be close to white, and the back side of the gums will be close to black. . In other words, the height difference in the back teeth can be expressed in black and white. On the other hand, in the case of a normal two-dimensional image such as a color photograph, the shape of the back teeth is represented by color or the outline in the XY plane, and height differences cannot be represented. In particular, in machine learning, when it is difficult to determine whether a scanned object is the gums, mucous membrane (back of the cheek), or lips using only a two-dimensional image such as a color photograph, the above-mentioned height and low Using distance images with differences makes it possible to identify each object. Also, in a computer (AI) such as the calculation unit 11, when using a pixel value for the Z coordinate such as in a distance image, it is easier to calculate the distance between adjacent objects than when simply using the height of the shape as the Z coordinate. Since the relationship becomes easier to understand, it becomes easier to calculate using convolution.

The identification unit 1102 is a functional unit of the calculation unit 11. The identification unit 1102 identifies at least one object among the plurality of objects based on the two-dimensional data input from the two-dimensional data generation unit 1106 and the estimated model 122. Identification section 1102 outputs the identification result to removal section 1103.

Here, machine learning of the estimation model 122 will be explained with reference to FIG. 11. FIG. 11 is a diagram for explaining learning data used when performing machine learning on the estimation model 122 according to the first embodiment. As shown in FIG. 11, the estimation model 122 according to the first embodiment shows two-dimensional data including an X coordinate and a Y coordinate, and each of the plurality of objects as position information of each of the plurality of objects in the oral cavity. Machine learning (supervised learning) is performed to estimate each of a plurality of objects based on one scan's worth of two-dimensional data using one scan's worth of learning data including the correct answer label. That is, machine learning is performed on the estimation model 122 so as to estimate the object based on two-dimensional data including the X and Y coordinates without using the Z coordinate.

Here, the relative positional relationship between a plurality of objects will be explained. In the oral cavity, the position of each of a plurality of objects, such as teeth and tongue, is anatomically determined in advance in relation to arbitrary landmarks, that is, in relative relation. For example, as shown in Figures 7 and 8 above, when a face with an open mouth is viewed from the front, the lower jaw can be seen from the center of the upper or lower jaw (back of the oral cavity) to the outside (in the direction of the mouth opening). In order, the tongue, the frenulum between the mandibular tooth row and the tongue, the mandibular tooth row, the frenulum between the mandibular tooth row and the lower lip (not shown), and the lower lip are located. . In other words, if the mandibular tooth row is the starting point (arbitrary landmark), the tongue is located further back in the mouth than the mandibular tooth row, and the lower lip is located in the front side of the mouth than the mandibular tooth row. do. In addition, when looking at a face with an open mouth from the front, in the upper jaw, from near the center of the upper or lower jaw (back of the oral cavity) to the outside (in the direction of the mouth opening), the hard palate, the maxillary teeth, and the hard palate. The frenulum between the maxillary dentition, the maxillary dentition, the maxillary dentition and the upper lip (not shown), and the upper lip are located. In other words, if the maxillary dentition is the starting point (an arbitrary landmark), the hard palate is located further back in the oral cavity than the maxillary dentition, and the upper lip is located further toward the opening of the oral cavity than the maxillary dentition. To position. That is, within the oral cavity, the relative positional relationship between a plurality of objects such as teeth, tongue, and lips is determined.

In this way, since the relative positional relationship between multiple objects in the oral cavity is determined, the input data of the estimation model 122, which is three-dimensional data including the position information of the objects, and the output data of the estimation model 122, It can be said that there is a correlation between the identification result of the type of the object. That is, since there is a correlation between input data and output data such that the position information of an object included in the three-dimensional data is associated with the type of the object, the estimation model 122 can Based on three-dimensional data including position information, the type of object can be identified by specifying in which region of the oral cavity a position corresponding to the three-dimensional data is included.

FIG. 12 is a diagram for explaining the correspondence between each of a plurality of objects and a correct label. As shown in FIG. 12, for the lower jaw, data indicating "01" as the correct label is associated with the tongue. Data indicating "02" as the correct label is associated with the mandibular first gap. Data indicating "31" to "48" as correct answer labels is associated with each of the plurality of teeth included in the dentition of the lower jaw. Data indicating "04" as the correct label is associated with the second mandibular gap. Data indicating "05" is associated with the lower lip as the correct label.

Regarding the upper jaw, data indicating "06" as the correct label is associated with the hard palate. Data indicating "07" as the correct label is associated with the maxillary first gap. Data indicating "11" to "28" as correct answer labels is associated with each of the plurality of teeth included in the dentition of the upper jaw. Data indicating "09" as the correct label is associated with the second maxillary gap. Data indicating "10" is associated with the upper lip as the correct label.

As mentioned above, in dental practice, insertion objects such as fingers and clinical instruments may be inserted into the oral cavity. Among the insertion objects, the finger is associated with data indicating "51" as the correct label. Furthermore, among the insertion objects, data indicating "52" as the correct label is associated with the medical instrument.

Returning to Figure 9, in the learning data, a correct label indicating the type of object corresponds to each two-dimensional data (position information) of a point cloud indicating the surface of each of multiple objects obtained in one scan. It is attached.

For example, as shown in FIG. 11, data indicating "01" as a correct answer label is associated with two-dimensional data generated from three-dimensional data obtained by scanning the tongue. Data indicating "38" as a correct answer label is associated with two-dimensional data generated from three-dimensional data obtained by scanning a mandibular left third molar. Two-dimensional data generated from three-dimensional data obtained by scanning the mandibular left second molar is associated with data indicating "37" as a correct answer label. Furthermore, data indicating "51" as a correct answer label is associated with two-dimensional data generated from three-dimensional data obtained by scanning a finger inserted into the second mandibular gap.

In this way, in the first embodiment, the two-dimensional data (X coordinate , Y coordinate) is associated with a correct label indicating the type of object.

The estimation model 122 identifies the type of each of the plurality of scanned objects based on one scan's worth of two-dimensional data, and adjusts the parameter 1222 based on the degree of agreement between the identification result and the correct label.

As a result, the estimation model 122 performs machine learning to identify the type of object corresponding to the two-dimensional data based on the correct label associated with the two-dimensional data for one scan. Be able to identify the type of object that corresponds to the object.

Furthermore, since the estimation model 122 performs machine learning based on two-dimensional data whose dimensions have been reduced by the Z coordinate, machine learning can be performed while reducing the burden of calculation processing, compared to using three-dimensional data that includes the Z coordinate. I can do it.

Returning to FIG. 9, the removal unit 1103 is a functional unit of the calculation unit 11. The removal unit 1103 obtains the object type identification result from the identification unit 1102. As shown in Fig. 12, the identification results include the tongue, the first mandibular gap, each tooth included in the mandibular dentition, the mandibular second gap, the lower lip, the hard palate, the maxillary first gap, and the teeth included in the maxillary dentition. In addition to the identification results of each tooth, the upper jaw second gap, and the upper lip, the identification results of inserted objects such as fingers and medical instruments are also included. If the identification result of the identification unit 1102 includes the identification result of an unnecessary object such as a surgeon's finger, a medical instrument, or a patient's tongue, the removal unit 1103 removes the three-dimensional data from the three-dimensional data input from the input unit 1101. By removing unnecessary three-dimensional data including the position information of each point group indicating the surface of the unnecessary object identified by the identification unit 1102, three-dimensional data after unnecessary object removal (hereinafter referred to as "unnecessary three-dimensional data") ) is generated.

Specifically, the removal unit 1103 extracts position information (X coordinates, Y coordinates) in the XY plane directions of each point group indicating the surface of the unnecessary object, and extracts position information (X coordinates, Y coordinates) for each point group indicating the surface of the unnecessary object. By extracting position information (Z coordinate) in the optical axis direction, unnecessary three-dimensional data is extracted. For example, the removing unit 1103 extracts the X and Y coordinates of each point group indicating the surface of the unnecessary object identified by the identifying unit 1102, based on the two-dimensional data generated by the two-dimensional data generating unit 1106. Furthermore, the removal unit 1103 uses the extracted X and Y coordinates of the unnecessary object as a search key, and based on the three-dimensional data acquired by the input unit 1101, removes the Z coordinate associated with the X and Y coordinates of the unnecessary object. Extract. The removal unit 1103 can use the extracted X coordinate, Y coordinate, and Z coordinate of the unnecessary object as unnecessary three-dimensional data. Here, extracting an unnecessary object includes, for example, storing each data of the X coordinate, Y coordinate, and Z coordinate in association with identification data that can identify the unnecessary object. The removal unit 1103 can generate three-dimensional data after removing unnecessary objects by removing unnecessary three-dimensional data from the three-dimensional data input from the input unit 1101. The removing unit 1103 outputs the three-dimensional data after removing unnecessary objects to the combining unit 1104.

The coupling unit 1104 is a functional unit of the calculation unit 11. The combining unit 1104 acquires one scan's worth of three-dimensional data from the removing unit 1103 each time one scan's worth of three-dimensional data is input to the input unit 1101, and combines the accumulated three-dimensional data of multiple scans. By doing so, three-dimensional data for a plurality of combined scans (hereinafter also referred to as "combined data") is generated.

FIG. 13 is a diagram for explaining the combined data after unnecessary object removal generated by the data processing device 1 according to the first embodiment. As shown in FIG. 13, when the identification result of the identification unit 1102 includes the identification result of an unnecessary object such as a surgeon's finger or a medical instrument, or a patient's tongue, lips, or mucous membrane, the removal unit 1103 A removal flag is set for the three-dimensional data (unnecessary three-dimensional data) corresponding to the unnecessary object identified by the identification unit 1102. In the example of FIG. 13, "01" is set as the removal flag for the unnecessary three-dimensional data corresponding to the tongue, lips, mucous membrane, and fingers to be removed. The data processing device 1 is configured not to use unnecessary three-dimensional data whose removal flag is set to “01” in the two-dimensional image displayed on the display 3. The combining unit 1104 generates combined data after unnecessary object removal by combining the three-dimensional data for multiple scans, as shown in FIG. 13, based on the three-dimensional data input from the removal unit 1103.

Returning to FIG. 9, the combining unit 1104 outputs the combined data to the storage unit 12 and the image generating unit 1105. The storage unit 12 stores the combined data input from the combining unit 1104. The image generation unit 1105 is a functional unit of the calculation unit 11. The image generation unit 1105 generates two-dimensional image data corresponding to a two-dimensional image seen from an arbitrary viewpoint based on the combined data input from the combining unit 1104, and outputs the generated two-dimensional image data to the display 3. . At this time, the image generation unit 1105 generates two-dimensional image data without using unnecessary three-dimensional data whose removal flag is set to "01." Thereby, the data processing device 1 can display on the display 3 a two-dimensional image of the oral cavity after the unnecessary object has been removed, and show it to the user.

[Processing flow of data processing device]
15 is a diagram illustrating an example of a process executed by the data processing device 1 with reference to FIG. 14. FIG. FIG. 14 is a flowchart for explaining an example of a process executed by the data processing device 1 according to the first embodiment. Each step (hereinafter referred to as “S”) shown in FIG. 14 is realized by the calculation unit 11 of the data processing device 1 executing the data processing program 121. Furthermore, after the three-dimensional scanner 2 starts scanning, the data processing device 1 executes the process of the flowchart shown in FIG. 14 .

As shown in FIG. 14, the data processing device 1 acquires three-dimensional data of an arbitrary point scanned by the three-dimensional scanner 2 (S11). The data processing device 1 generates two-dimensional data from the acquired three-dimensional data using only the X and Y coordinates (S12). The data processing device 1 identifies each of the plurality of scanned objects based on the two-dimensional data and the estimated model 122 (S13).

The data processing device 1 determines whether an unnecessary object is detected based on the identification result (S14). That is, the data processing device 1 determines, as identification results, data indicating "01" corresponding to the tongue, data indicating "51" corresponding to the finger, data indicating "52" corresponding to the medical instrument, and data indicating "52" corresponding to the lower lip. It is determined whether data indicating the corresponding "05" and data indicating "10" corresponding to the upper lip have been output. When the data processing device 1 detects an unnecessary object (YES in S14), the data processing device 1 extracts unnecessary three-dimensional data corresponding to the detected unnecessary object, and removes the extracted unnecessary three-dimensional data (S15). That is, the data processing device 1 sets a removal flag for unnecessary three-dimensional data.

If the data processing device 1 does not detect an unnecessary object (NO in S14), or after removing unnecessary three-dimensional data in S45, the data processing device 1 generates combined data by combining three-dimensional data for multiple scans. (S16).

The data processing device 1 stores the combined data in the storage unit 12 (S17). Further, the data processing device 1 generates two-dimensional image data corresponding to a two-dimensional image seen from an arbitrary viewpoint based on the combined data, and outputs the generated two-dimensional image data to the display 3 to display the intraoral information. A two-dimensional image of is displayed on the display 3 (S18).

The data processing device 1 determines whether scanning by the three-dimensional scanner 2 has stopped (S19). If the scanning by the three-dimensional scanner 2 has not stopped (NO in S19), the data processing device 1 returns the process to S11. On the other hand, when the three-dimensional scanner 2 stops scanning (YES in S19), the data processing device 1 ends this process.

As described above, the data processing device 1 uses the estimation model 122 to identify unnecessary objects that are unnecessary for dental treatment among the plurality of objects scanned by the three-dimensional scanner 2, and shows the surface of the identified unnecessary object. Unnecessary three-dimensional data including position information of each point group can be extracted. Thereby, the user does not need to extract the three-dimensional data of the unnecessary object himself, and can easily and appropriately extract the three-dimensional data of the unnecessary object.

Furthermore, the data processing device 1 can identify each of a plurality of objects in the oral cavity using the estimation model 122 based on two-dimensional data whose dimension is reduced by the Z coordinate. Compared to identifying each of a plurality of objects in the oral cavity using original data, it is possible to extract three-dimensional data of unnecessary objects while reducing the burden of calculation processing.

Since the data processing device 1 can remove unnecessary 3D data from the 3D data input by the 3D scanner 2, the user can remove the 3D data of the unnecessary object by himself/herself and There is no need to create three-dimensional data, and three-dimensional data after unnecessary objects are removed can be easily obtained.

Since the data processing device 1 outputs image data generated using the three-dimensional data after removing unnecessary three-dimensional data to the display 3, the user can view the two-dimensional image of the oral cavity after unnecessary objects have been removed. There is no need to generate the two-dimensional image by oneself, and a two-dimensional image with unnecessary objects removed can be easily obtained.

<Embodiment 2>
Embodiment 2 of the present disclosure will be described in detail with reference to the drawings. In the second embodiment, only the parts that are different from the first embodiment will be described, and the same parts as in the first embodiment will be given the same reference numerals and the description thereof will not be repeated.

FIG. 15 is a diagram for explaining an example in which the three-dimensional scanner 102 according to the second embodiment acquires three-dimensional data based on the triangulation method. FIG. 16 is a diagram showing a two-dimensional image viewed from an arbitrary viewpoint based on three-dimensional data acquired by the three-dimensional scanner 102 according to the second embodiment. The three-dimensional scanner 102 according to the second embodiment, unlike the three-dimensional scanner 2 according to the first embodiment, acquires three-dimensional data based on triangulation.

As shown in FIG. 15, in the triangulation method, a pattern is projected onto an object by a projector 8, and an image of the pattern projected onto the object is captured by a camera 9 located at a different position from the projector 8. As shown in FIG. 15(A), when the projector 8 projects a straight line pattern onto an object, the angle formed by the line connecting the projector 8 and the object and the line connecting the camera 9 and the object is used. , as shown in FIG. 15(B), the pattern image obtained by imaging with the camera 9 shows a pattern that follows the shape of the object. As shown in FIG. 15(C), the three-dimensional scanner 102 determines the length of the line connecting the projector 8 and the object, the length of the line connecting the camera 9 and the object, and the length of the line connecting the projector 8 and camera 9. Based on the length and the angle of each vertex of the triangle generated by the plurality of lines, the position of the pattern projected onto the object is detected using a known triangulation method.

As shown in FIG. 16, the three-dimensional scanner 102 includes a projector 8 that projects a pattern onto an object within the oral cavity, and a camera 9 that images the pattern projected onto the object. As shown in FIG. 16(A), the three-dimensional scanner 102 images the object with the camera 9 while the pattern is projected onto the object by the projector 8. The three-dimensional scanner 102 uses a known triangulation method to obtain position information (X coordinate, Y coordinate, Z coordinate) of each point group indicating the surface of the object based on the captured image of FIG. 16(A). .

Furthermore, the three-dimensional scanner 102 may image the object with the camera 9 in a state where no pattern is projected onto the object by the projector 8, as shown in FIG. 16(B). As described above, the data processing device 300 according to the third embodiment is configured to identify each of a plurality of objects in the oral cavity based on two-dimensional data, and the data processing device 300 is configured to identify each of a plurality of objects in the oral cavity based on two-dimensional data. Each of the plurality of objects in the oral cavity may be identified based on the captured image of FIG. 16(B) acquired by. Furthermore, the data processing device 300 may use the extracted X and Y coordinates of the unnecessary object as search keys to extract the Z coordinate of the unnecessary object acquired based on the captured image of FIG. 16(A).

Note that the three-dimensional scanner 102 performs processing in which the object is imaged by the camera 9 in a state in which a pattern is projected onto the object by the projector 8, and in which the object is imaged by the camera 9 in a state in which the pattern is not projected onto the object by the projector 8. By switching between the processes, the captured image in FIG. 16(A) and the captured image in FIG. 16(B) may be obtained. Alternatively, the three-dimensional scanner 102 includes a plurality of cameras 9, and a first camera images the object while a pattern is projected onto the object by the projector 8; The captured image in FIG. 16(A) and the captured image in FIG. 16(B) may be obtained by capturing an image of the object with the second camera.

<Modified example>
The present disclosure is not limited to the above embodiments, and various modifications and applications are possible. Modifications applicable to the present disclosure will be described below.

In the embodiment described above, an unnecessary object that is unnecessary for dental treatment is exemplified as a "predetermined object," but the "predetermined object" is not limited to an unnecessary object, but can also be a specific object arbitrarily specified by the user in advance. It's okay. For example, if a user discovers caries in a specific tooth (for example, the left third molar of the mandible) among multiple teeth in the oral cavity, the user can pick up only that specific tooth and set it as a "predetermined object." May be specified. In this case, the data processing device 1 generates two-dimensional data from the intraoral three-dimensional data acquired by the three-dimensional scanner 2, and identifies a specific tooth specified by the user based on the generated two-dimensional data. , is configured to extract predetermined three-dimensional data including position information of each point group indicating the surface of the identified specific tooth.

The identification unit 1102 does not only identify each of a plurality of objects in the oral cavity using the estimation model 122, but also performs pattern matching using predesigned feature amounts, that is, predetermined shapes included in a two-dimensional image. Each of a plurality of objects within the oral cavity may be identified. As described above, the two-dimensional data generated by the two-dimensional data generation unit 1106 is based on at least one object in the oral cavity when viewed from a position a certain distance away (for example, the virtual viewpoint position in FIG. 4). This is two-dimensional image data showing the appearance of one object. The identification unit 1102 identifies the object (for example, a tooth or an unnecessary object) from a two-dimensional image showing the appearance of the object (for example, a tooth or an unnecessary object) in the oral cavity to be identified, which is generated by the two-dimensional data generation unit 1106. Recognizes the image and determines the degree of agreement between the recognized image of the object (for example, a tooth or an unnecessary object) and a pre-designed feature amount, that is, a predetermined shape included in the two-dimensional image. Based on the results, the object to be identified (for example, a tooth or an unnecessary object) may be identified.

The three-dimensional data input to the input unit 1101 includes color information (RGB values) representing the actual color of each point group in addition to position information and normal information for each point group indicating the surface of the object. May be included. Furthermore, the estimation model 122 may undergo machine learning to identify the type of object based on color information (RGB values) associated with the three-dimensional data input to the input unit 1101. Note that the three-dimensional data input to the input unit 1101 may include only position information for each point group, and may not include normal information and color information.

The removal unit 1103 is not limited to removing unnecessary three-dimensional data, and may add color information indicating an unnecessary object to unnecessary three-dimensional data. The image generation unit 1105 is not limited to generating a two-dimensional image of the oral cavity after the unnecessary object has been removed, but also a two-dimensional image in which the portion corresponding to the unnecessary object is colored to indicate the unnecessary object. may be generated and output to the display 3.

As a three-dimensional measurement method, in addition to the above techniques, triangulation methods such as random pattern projection or SfM (Structure From Motion) without pattern projection and SLAM (Simultaneous Localization and Mapping) may be used, and TOF (Time Laser technology such as LIDAR (Light Detection and Ranging) or LIDAR (Light Detection and Ranging) may also be used.

The embodiments disclosed this time should be considered to be illustrative in all respects and not restrictive. The scope of the present disclosure is indicated by the claims rather than the above description, and it is intended that all changes within the meaning and range equivalent to the claims are included. Note that the configurations exemplified in this embodiment and the configurations exemplified in the modified examples can be combined as appropriate.

1,200,300,400 Data processing device, 2,102 Three-dimensional scanner, 3 Display, 4 Keyboard, 5 Mouse, 7 Intraoral camera, 8 Projector, 9 Camera, 10 Data processing system, 11 Arithmetic unit, 12 Storage unit , 13 scanner interface, 14 communication unit, 15 display interface, 16 peripheral device interface, 17 reading unit, 20 removable disk, 21 housing, 22 probe, 23 light source, 24 lens, 25 optical sensor, 26 prism, 27 counterweight, 28 reflection section, 29 aperture, 40 control device, 121 data processing program, 122 estimation model, 1101 input section, 1102 identification section, 1103 removal section, 1104 combination section, 1105 image generation section, 1106 two-dimensional data generation section, 1221 neural Network, 1222 parameters.

Claims (9)

A data processing device that processes data including position information of at least one object in the oral cavity,
an input unit into which three-dimensional data including position information of each point group indicating the surface of the at least one object is input;
a calculation unit that processes the three-dimensional data input from the input unit,
The arithmetic unit is
generating two-dimensional data from the three-dimensional data;
Based on the two-dimensional data, identifying a predetermined object among the at least one object including at least one of an insertion object inserted into the oral cavity, a tongue, a lip, and a mucous membrane;
extracting predetermined three-dimensional data including position information of each point group indicating the surface of the identified predetermined object;
A data processing device that removes the predetermined three-dimensional data from the three-dimensional data input from the input section. The insertion object includes at least one of a finger and a medical instrument that are inserted to press soft tissue in the oral cavity when an imaging device for imaging the inside of the oral cavity is inserted into the oral cavity. , The data processing device according to claim 1. 3. The data processing device according to claim 1, wherein the arithmetic unit outputs image data generated using three-dimensional data after removing the predetermined three-dimensional data. The three-dimensional data input from the input unit is acquired by an imaging device that is inserted into the oral cavity and images the inside of the oral cavity based on a confocal method or a triangulation method,
The two-dimensional data includes position information of each point group indicating the surface of the at least one object in a plane direction perpendicular to the optical axis direction of the imaging device,
The calculation unit extracts position information in the plane direction of each point group indicating the surface of the predetermined object, and extracts position information in the optical axis direction corresponding to each point group indicating the surface of the predetermined object. The data processing device according to claim 1, wherein the predetermined three-dimensional data is extracted by extracting the predetermined three-dimensional data. The two-dimensional data generated based on the three-dimensional data is data of a two-dimensional image showing the appearance of the at least one object when the at least one object is viewed from a position separated by a certain distance,
The data processing device according to claim 1, wherein the calculation unit identifies the predetermined object by recognizing an image of the predetermined object included in the two-dimensional image. The calculation unit extracts positional information in a direction orthogonal to a plane direction indicated by the two-dimensional image, which corresponds to positional information of the predetermined object included in the two-dimensional image. The data processing device according to claim 5, which extracts data. The calculation unit calculates the predetermined object based on the two-dimensional data and an estimation model subjected to machine learning for estimating the predetermined object from among the at least one object based on the two-dimensional data. The data processing device according to claim 1, wherein the data processing device identifies a . A data processing method for processing data including position information of at least one object in an oral cavity by a computer, the method comprising:
inputting three-dimensional data including position information of each point group indicating the surface of the at least one object;
processing the input three-dimensional data,
The processing step includes:
generating two-dimensional data from the three-dimensional data;
Based on the two-dimensional data, identifying a predetermined object among the at least one object including at least one of an insertion object inserted into the oral cavity, a tongue, a lip, and a mucous membrane;
extracting predetermined three-dimensional data including position information of each point group indicating the surface of the identified predetermined object;
A data processing method comprising the step of removing the predetermined three-dimensional data from the input three-dimensional data. A data processing program that processes data including position information of at least one object in the oral cavity,
to the computer,
inputting three-dimensional data including position information of each point group indicating the surface of the at least one object;
processing the input three-dimensional data;
The processing step includes:
generating two-dimensional data from the three-dimensional data;
Based on the two-dimensional data, identifying a predetermined object among the at least one object including at least one of an insertion object inserted into the oral cavity, a tongue, a lip, and a mucous membrane;
extracting predetermined three-dimensional data including position information of each point group indicating the surface of the identified predetermined object;
A data processing program comprising: removing the predetermined three-dimensional data from the input three-dimensional data.

Images