JP7195291B2 - DATA PROCESSING APPARATUS, DATA PROCESSING SYSTEM, DATA PROCESSING METHOD, AND DATA PROCESSING PROGRAM - Google Patents

Description

The present disclosure relates to a data processing device, a data processing system, a data processing method, and a data processing program.

2. Description of the Related Art Conventionally, in the field of dentistry, a three-dimensional scanner with a built-in three-dimensional camera for obtaining a three-dimensional shape of a tooth has been known for digital designing of a prosthesis or the like on a computer. For example, Patent Literature 1 discloses a technique for recording the shape of a tooth by capturing an image of the tooth using a three-dimensional camera. An operator such as a dentist can record the three-dimensional shape of the tooth to be imaged by using the three-dimensional camera disclosed in Patent Document 1, and the recorded three-dimensional shape of the tooth can be recorded. While confirming the displayed three-dimensional image, the user can identify the type of tooth according to his/her own knowledge.

JP-A-2000-74635

In this way, conventionally, the operator has identified the type of the tooth based on his/her own knowledge based on the three-dimensional image including the tooth acquired by the three-dimensional camera, but the level of knowledge varies for each operator. Therefore, there is a problem that the accuracy of the identification result varies depending on the level of knowledge of the operator. For example, central incisors and lateral incisors, canines and first premolars, first premolars and second premolars, second premolars and first molars, first premolars and second molars, etc. The accuracy of discrimination between similar teeth is not particularly high.

The present disclosure is made to solve such problems, and provides a data processing device, a data processing system, a data processing method, and a data processing program that can accurately identify the type of tooth. for the purpose.

According to the present disclosure, a data processing apparatus is provided for processing data related to tooth type. The data processing device includes an input unit for inputting three-dimensional data including three-dimensional positional information on each of a plurality of points constituting a tooth, three-dimensional data input from the input unit, and based on the three-dimensional data. an associating unit that associates relevant data related to the type of tooth with each of a plurality of points corresponding to the three-dimensional data based on an estimation model including a neural network for estimating the type of tooth; and an associating unit. and an output unit that outputs the association result.

According to the present disclosure, a data processing system is provided for processing data related to tooth type. The data processing system includes a three-dimensional scanner that acquires three-dimensional data including three-dimensional positional information on each of a plurality of points that constitute the tooth using a three-dimensional camera, and three-dimensional data acquired by the three-dimensional scanner. and a data processing device for associating relevant data relating to tooth type to each of a plurality of points corresponding to . The data processing device includes an input unit for inputting three-dimensional data acquired by a three-dimensional scanner, three-dimensional data input from the input unit, and a neural network for estimating the type of tooth based on the three-dimensional data. An associating unit that associates relevant data related to tooth types with each of the plurality of points corresponding to the three-dimensional data based on the estimated model including the above, and an output unit that outputs the result of association by the associating unit include.

According to the present disclosure, a data processing method is provided for processing data related to tooth type by a computer. The data processing method includes a step of inputting three-dimensional data including three-dimensional positional information on each of a plurality of points constituting a tooth, the three-dimensional data input by the input step, and the three-dimensional data an estimation model including a neural network for estimating the type of tooth based on the step of associating relevant data related to the type of tooth with each of a plurality of points corresponding to the three-dimensional data; and outputting the results of the association.

According to the present disclosure, a data processing program is provided for processing data related to tooth type. The data processing program comprises a step of inputting three-dimensional data including three-dimensional position information of each of a plurality of points constituting a tooth into a computer; a step of associating relevant data related to the tooth type with each of a plurality of points corresponding to the three-dimensional data, based on an estimation model including a neural network for estimating the tooth type based on the three-dimensional data; , and a step of outputting the result of association by the associating step.

According to the present disclosure, it is possible to accurately identify the type of tooth based on three-dimensional data including three-dimensional position information on each of a plurality of points that constitute the tooth.

1 is a schematic diagram showing an application example of a data processing device according to an embodiment; FIG. 1 is a schematic diagram showing the overall configuration of a system according to an embodiment; FIG. 1 is a schematic diagram showing a hardware configuration of a data processing device according to an embodiment; FIG. 2 is a schematic diagram showing a hardware configuration of a server device according to this embodiment; FIG. 1 is a schematic diagram showing a functional configuration of a data processing device according to an embodiment; FIG. FIG. 4 is a schematic diagram for explaining the contents of three-dimensional data input to the data processing device according to the embodiment; FIG. 4 is a schematic diagram showing an example of a tooth to be estimated by the data processing device according to the present embodiment; FIG. 4 is a schematic diagram for explaining generation of learning data according to the present embodiment; FIG. 4 is a schematic diagram for explaining an example of a learning data set according to the embodiment; FIG. FIG. 4 is a schematic diagram for explaining generation of a trained model based on a learning data set according to the present embodiment; FIG. 10 is a schematic diagram for explaining an example of output of an association result according to the embodiment; FIG. 10 is a schematic diagram for explaining an example of output of an association result according to the embodiment; FIG. 4 is a schematic diagram for explaining an example of output data according to the embodiment; FIG. FIG. 4 is a schematic diagram for explaining an example of output data according to the embodiment; FIG. 5 is a flowchart for explaining an example of learning processing executed by the data processing device according to the embodiment; 6 is a flowchart for explaining an example of learning processing executed by the server device according to the present embodiment; 6 is a flowchart for explaining an example of service providing processing executed by the data processing device according to the embodiment; FIG. 11 is a schematic diagram for explaining an example of output of an association result according to a modification; FIG. 11 is a flowchart for explaining an example of service providing processing executed by a data processing device according to a modification; FIG. FIG. 11 is a schematic diagram for explaining generation of a trained model based on a learning data set according to a modification; FIG. 11 is a flowchart for explaining an example of service providing processing executed by a data processing device according to a modification; FIG. FIG. 11 is a schematic diagram for explaining an example of a learning data set according to a modification; FIG. 11 is a flowchart for explaining an example of service providing processing executed by a data processing device according to a modification; FIG.

Embodiments of the present disclosure will be described in detail with reference to the drawings. The same or corresponding parts in the drawings are given the same reference numerals, and the description thereof will not be repeated.

[Application example]
An application example of the data processing apparatus 100 according to the present embodiment will be described with reference to FIGS. 1 and 2. FIG. FIG. 1 is a schematic diagram showing an application example of a data processing device 100 according to this embodiment. FIG. 2 is a schematic diagram showing the overall configuration of the system according to this embodiment.

As shown in FIG. 1, a user 1 uses a data processing system 10 to obtain three-dimensional intraoral data (hereinafter also referred to as "three-dimensional data") including at least teeth and gums of a subject 2. can be obtained. The three-dimensional data includes position information of each point that constitutes the scanned teeth and gums in the oral cavity. The “user” is a person who uses the data processing system 10, such as an operator such as a dentist, a dental assistant, a teacher or student at a dental university, a dental technician, a manufacturer's engineer, or a factory worker. Either can be used. The “subject” may be any person to be scanned by the data processing system 10, such as a patient at a dental clinic or a subject at a dental college.

Data processing system 10 according to the present embodiment includes three-dimensional scanner 200 , data processing device 100 , and display 300 . The three-dimensional scanner 200 acquires three-dimensional data of a scanning target by a built-in three-dimensional camera. Specifically, the three-dimensional scanner 200 scans the inside of the oral cavity to generate three-dimensional data as positional information of each of a plurality of points that constitute the teeth and gums to be scanned (for example, X axis, Y-axis along the horizontal direction, and Z-axis along the height direction) are acquired using an optical sensor or the like. Alternatively, the three-dimensional scanner 200 scans a model created by taking an impression of the oral cavity, and obtains positional information (for example, longitudinal X-axis along the direction, Y-axis along the horizontal direction, and Z-axis along the height direction) are acquired using an optical sensor or the like. Based on the three-dimensional data acquired by the three-dimensional scanner 200, the data processing device 100 generates a two-dimensional image including a tooth image from an arbitrary viewpoint, or a three-dimensional image including a three-dimensional image such as a hologram. An image is generated and the generated three-dimensional image is displayed on the display 300 . At least one of the three-dimensional scanner 200 and the data processing device 100 may be of a movable portable type or of a fixed stationary type. For example, if the three-dimensional scanner 200 and the data processing device 100 are stationary desktop scanners, the scanned object may be an artificially generated tooth such as a plaster model or a denture.

For example, in order to digitally design on a computer a prosthesis or the like that compensates for missing teeth of the subject 2, the user 1 images the intraoral cavity of the subject 2 with the three-dimensional scanner 200, thereby reshaping the teeth and gums. Acquire at least three-dimensional intraoral data. The three-dimensional data includes at least three-dimensional positional information for each of a plurality of points forming the tooth. Each time the user 1 takes an intraoral image, three-dimensional data is sequentially acquired, and the three-dimensional image of the intraoral cavity is displayed on the display 300 . The user 1 confirms the three-dimensional image displayed on the display 300 and scans the missing part of the three-dimensional data.

Conventionally, based on a three-dimensional image in which three-dimensional data including the tooth acquired by the three-dimensional scanner 200 is visualized, the user 1 identifies the type of tooth that is being scanned or has been scanned based on his/her own knowledge. . However, since the level of knowledge differs for each user 1, depending on the knowledge of the user 1 may result in variations in the accuracy of the identification results. In orthodontics, etc., when designing orthodontic appliances, the teeth are classified by type (segmentation) while distinguishing between the teeth and adjacent teeth, and between the teeth and the gingiva. conversion) is also required.

Therefore, the data processing system 10 according to the present embodiment utilizes AI (Artificial Intelligence) of the data processing apparatus 100, based on the three-dimensional data acquired by the three-dimensional scanner 200, the three-dimensional The tooth type is estimated for each of the plurality of points corresponding to the data, and associated data related to the tooth type including the estimation result is associated with each of the plurality of points corresponding to the three-dimensional data. there is Note that the process of estimating the tooth type by the data processing device 100 is also referred to as "estimating process".

The "types of teeth" are upper right central incisors, lateral incisors, canines, first premolars, second premolars, first molars, second molars, and third molars, upper left side. central incisors, lateral incisors, canines, first premolars, second premolars, first molars, second molars, and third molars, lower right central incisors, lateral incisors, canines, 1st premolar, 2nd premolar, 1st molar, 2nd molar, and 3rd molar, mandibular left central incisor, lateral incisor, canine, 1st premolar, 2nd premolar, 3rd molar It refers to each tooth type such as 1 molar, 2nd molar, and 3rd molar.

Specifically, when the user 1 scans the oral cavity of the subject 2 using the three-dimensional scanner 200 , three-dimensional data of the oral cavity including at least teeth and gums is input to the data processing device 100 . The data processing device 100 executes an estimation process for estimating the type of tooth based on the input three-dimensional data including tooth characteristics and an estimation model including a neural network.

"Estimation model" includes a neural network and parameters used by the neural network, tooth information corresponding to the type of tooth associated with three-dimensional data, and estimation of the type of tooth using the three-dimensional data It is optimized (adjusted) by being learned based on the results.

Specifically, when three-dimensional data including teeth is input to the estimation model, a neural network based on the three-dimensional data extracts features of the teeth for each three-dimensional data, and based on the extracted tooth features, The type of tooth is estimated for each three-dimensional data. Then, the estimation model does not update the parameters based on the type of tooth estimated by itself and the type of tooth (tooth information) associated with the input three-dimensional data, if both match. If they do not match, the parameters are optimized by updating the parameters so that they match. In this way, the estimation model is learned by optimizing parameters using teacher data including three-dimensional data as input data and tooth types (tooth information) as correct data.

Note that processing for learning such an estimation model is also referred to as “learning processing”. In addition, the estimation model optimized by the learning process is particularly called a "learned model". That is, in the present embodiment, pre-learning estimation models and trained estimation models are collectively referred to as "estimation models", while trained estimation models are also particularly referred to as "learned models".

"Tooth information" includes upper right central incisors, lateral incisors, canines, first premolars, second premolars, first molars, second molars, and third molars, upper left central incisors Teeth, lateral incisors, canines, first premolars, second premolars, first molars, second molars, and third molars, mandibular right central incisors, lateral incisors, canines, first minors Molars, second premolars, first molars, second molars, and third molars, mandibular left central incisors, lateral incisors, canines, first premolars, second premolars, first molars , second molars, and third molars. In addition, "tooth information" includes the number 1 assigned to the central incisor, the number 2 assigned to the lateral incisor, the number 3 assigned to the canine, the number 4 assigned to the first premolar, the second small A number assigned to each tooth, such as number 5 assigned to the molar, number 6 assigned to the first molar, number 7 assigned to the second molar, number 8 assigned to the third molar, and so on. (eg tooth numbers commonly used in the dental field). In addition, the "tooth information" may include color information assigned to each tooth, or may include symbol information assigned to each tooth.

When the data processing device 100 executes the estimation process using the trained model, related data related to the tooth type including the estimation result is associated with each of the plurality of points corresponding to the three-dimensional data. The association result obtained in this manner is output to display 300 .

"Associated data" includes any data indicative of an inferred tooth type, such as colors, letters, symbols, and tooth numbers indicating an inferred tooth type. For example, the display 300 uses colors, characters, symbols, tooth numbers, etc. corresponding to tooth types to indicate tooth types for each of a plurality of points corresponding to the three-dimensional data acquired by the three-dimensional scanner 200. .

The estimation result by the data processing apparatus 100 is output as scan information to the server apparatus 500 arranged in the dental laboratory and the management center together with the three-dimensional data used in the estimation process.

For example, as shown in FIG. 2, data processing system 10 is located at each of a plurality of locals AC. For example, local A and local B are dental clinics, and in the dental clinics, an operator and a dental assistant who are user 1 use the data processing system 10 to examine teeth and gums of a patient who is the subject 2. Acquire at least three-dimensional intraoral data. Also, the local C is a dental college, and at the dental college, a teacher or a student who is the user 1 acquires intraoral three-dimensional data of a subject who is the subject 2 . The scan information (three-dimensional data, estimation results) acquired by each of the locals A to C is output via the network 5 to the server device 500 located in the dental laboratory and management center which is the local D. FIG.

In the dental laboratory, a dental technician or the like creates a prosthesis or the like to replace the missing tooth portion of the subject 2 based on the scan information obtained from each of the locals A to C. In the management center, the server device 500 accumulates and stores the scan information obtained from each of the locals A to C, and holds it as big data.

Note that the server device 500 is not limited to being arranged in a management center different from the local one in the dental clinic, and may be arranged locally. For example, the server apparatus 500 may be arranged within any one of the locals A to C. FIG. Also, a plurality of data processing devices 100 may be arranged within one local, and furthermore, a server device 500 capable of communicating with the plurality of data processing devices 100 may be arranged within the one local. Moreover, the server device 500 may be implemented in the form of a cloud service.

In the dental laboratory, scan information is aggregated from various locations, such as local A-C. For this reason, the scan information held at the dental laboratory may be transmitted to the management center via the network 5, or may be sent to a removable disk 550 such as a CD (Compact Disc) and USB (Universal Serial Bus) memory. may be sent to the management center via

The scan information may also be sent to the management center from each of the local A to C via the removable disk 550 without going through the network 5 . Also, between each of the locals A to C, scan information may be sent to each other via the network 5 or the removable disk 550 .

Each local A to C data processing device 100 has its own estimation model, and estimates the type of tooth using the estimation model held by each at the time of estimation processing. Each of the local A to C data processors 100 learns its own estimation model through its own learning process, thereby generating a trained model. Furthermore, in the present embodiment, server device 500 also holds an estimation model. The server device 500 generates a trained model by learning the estimated model by learning processing using the scan information acquired from the data processing device 100 of each local A to C and the dental laboratory, and each local A to C , the learned model is distributed to the data processing devices 100 of .

In the present embodiment, both the data processing devices 100 of the local A to C and the server devices 500 execute the learning process, but only the data processing devices 100 of the local A to C perform the learning process. , or a form in which only the server device 500 executes the learning process. In the case where only the server device 500 executes the learning process, the estimated model (learned model) held by each local A to C data processing device 100 is are made common.

Also, the server device 500 may have the function of the estimation processing in the data processing device 100 . For example, each local A to C transmits the acquired three-dimensional data to the server device 500, and the server device 500 estimates the type of tooth in each based on the three-dimensional data received from each local A to C. Results may be calculated. Then, server device 500 may transmit the respective estimation results to each of locals A to C, and each of locals A to C may output the estimation results received from server device 500 to a display or the like. In this way, each local A to C and the server device 500 may be configured in the form of a cloud service. In this way, as long as server device 500 holds an estimation model (learned model), each of locals A to C can obtain an estimation result without holding an estimation model (learned model). can.

As described above, according to the data processing system 10 according to the present embodiment, the type of tooth is estimated based on the three-dimensional data acquired by the three-dimensional scanner 200 using the AI of the data processing apparatus 100. . Then, based on the estimation result, each of the plurality of points corresponding to the three-dimensional data acquired by the three-dimensional scanner 200 is associated with related data related to the tooth type, and the result of the association is displayed on the display 300. etc. In this way, by using AI, it is possible to find not only tooth features that can be extracted by the user 1, but also tooth features that the user 1 cannot extract. Since the tooth type is identified for each of a plurality of corresponding points, the user 1 can accurately identify the tooth type without relying on his/her own knowledge.

[Hardware Configuration of Data Processor]
An example of the hardware configuration of the data processing device 100 according to this embodiment will be described with reference to FIG. FIG. 3 is a schematic diagram showing the hardware configuration of the data processing device 100 according to this embodiment. Data processing apparatus 100 may be implemented by, for example, a general-purpose computer or a computer dedicated to data processing system 10 .

As shown in FIG. 3, the data processing apparatus 100 includes, as main hardware elements, a scanner interface 102, a display interface 103, a peripheral device interface 105, a network controller 106, a media reader 107, and a PC display 108. , a memory 109 , a storage 110 , and an arithmetic device 130 .

The scanner interface 102 is an interface for connecting the three-dimensional scanner 200 and realizes input/output of data between the data processing device 100 and the three-dimensional scanner 200 .

The display interface 103 is an interface for connecting the display 300 and realizes input/output of data between the data processing device 100 and the display 300 . Display 300 is configured by, for example, an LCD (Liquid Crystal Display) or an organic ELD (Electroluminescence) display.

Peripheral device interface 105 is an interface for connecting peripheral devices such as keyboard 601 and mouse 602, and implements data input/output between data processing apparatus 100 and the peripheral devices.

The network controller 106 communicates with each of the devices located at the dental laboratory, the server device 500 located at the management center, and other locally located data processing devices 100 via the network 5. Send and receive data. The network controller 106 supports arbitrary communication methods such as Ethernet (registered trademark), wireless LAN (Local Area Network), and Bluetooth (registered trademark).

The media reading device 107 reads various data such as scan information stored in a removable disk 550, which is a storage medium.

A PC display 108 is a dedicated display for the data processing apparatus 100 . PC display 108 is configured by, for example, an LCD or an organic EL display. Although PC display 108 is separate from display 300 in the present embodiment, it may be shared with display 300 .

The memory 109 provides a storage area for temporarily storing program codes, work memory, etc. when the arithmetic unit 130 executes an arbitrary program. The memory 109 is composed of, for example, a volatile memory device such as a DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory).

The storage 110 provides storage areas for storing various data necessary for estimation processing, learning processing, and the like. The storage 110 is composed of a non-volatile memory device such as a hard disk or SSD (Solid State Drive), for example.

The storage 110 includes scan information 112, an estimated model 114 (learned model), a learning data set 116, color classification data 118, profile data 119, a data processing program 120, a learning program 121, An OS (Operating System) 127 is stored.

The scan information 112 includes three-dimensional data 122 acquired by the three-dimensional scanner 200 and an estimation result 124 of estimation processing executed based on the three-dimensional data 122 . The estimation result 124 is stored in the storage 110 in association with the three-dimensional data 122 used in the estimation process. The learning data set 116 is a group of learning data used for learning processing of the estimation model 114 . The color classification data 118 is data used for generating the learning data set 116 and for learning processing. The profile data 119 is attribute information about the target person 2, and is data (for example, medical record information) in which profiles such as age, sex, race, height, weight, and place of residence of the target person 2 are summarized. be. The data processing program 120 is a program for executing processing such as associating relevant data with each of a plurality of points constituting a tooth and outputting the result of the association. A program for execution is also included. The learning program 121 is a program for executing the learning process of the estimation model 114, and part of it also includes a program for executing the estimation process.

The arithmetic unit 130 is an arithmetic entity that executes various processes such as an estimation process and a learning process by executing various programs, and is an example of a computer. Arithmetic device 130 includes, for example, a CPU (Central Processing Unit) 132, an FPGA (Field-Programmable Gate Array) 134, a GPU (Graphics Processing Unit) 136, and the like.

Arithmetic device 130 may be configured with at least one of CPU 132, FPGA 134, and GPU 136, or may be configured with CPU 132 and FPGA 134, FPGA 134 and GPU 136, CPU 132 and GPU 136, or CPU 132, FPGA 134, and GPU 136. may be Processing units 130 may also be referred to as processing circuitry.

[Hardware Configuration of Server Device]
An example of the hardware configuration of server device 500 according to the present embodiment will be described with reference to FIG. FIG. 4 is a schematic diagram showing the hardware configuration of server device 500 according to the present embodiment. Server device 500 may be implemented by, for example, a general-purpose computer or a computer dedicated to data processing system 10 .

As shown in FIG. 4, the server device 500 includes, as main hardware elements, a display interface 503, a peripheral device interface 505, a network controller 506, a media reader 507, a memory 509, a storage 510, and a computing device. and a device 530 .

The display interface 503 is an interface for connecting the display 350 and realizes input/output of data between the server device 500 and the display 350 . Display 350 is configured by, for example, an LCD or an organic ELD display.

Peripheral device interface 505 is an interface for connecting peripheral devices such as keyboard 651 and mouse 652, and implements data input/output between server device 500 and the peripheral devices.

The network controller 506 transmits and receives data via the network 5 to and from each of the locally located data processing apparatus 100 and the apparatus located at the dental laboratory. The network controller 506 may support any communication method such as Ethernet (registered trademark), wireless LAN, Bluetooth (registered trademark), or the like.

The media reader 507 reads various data such as scan information stored in the removable disk 550 .

The memory 509 provides a storage area for temporarily storing program codes, work memory, etc. when the arithmetic unit 530 executes an arbitrary program. Memory 509 comprises, for example, a volatile memory device such as a DRAM or SRAM.

The storage 510 provides storage areas for storing various data necessary for learning processing and the like. The storage 510 is configured with a non-volatile memory device such as a hard disk or SSD, for example.

Storage 510 stores scan information 512 , estimated model 514 (learned model), learning data set 516 , color classification data 518 , profile data 519 , learning program 521 , and OS 527 .

The scan information 512 consists of three-dimensional data 522 acquired from the data processing apparatus 100 and the dental laboratory locally arranged via the network 5, and an estimation result 524 from an estimation process executed based on the three-dimensional data 522. include. The estimation result 524 is stored in the storage 510 in association with the three-dimensional data 522 used for the estimation process. A learning data set 516 is a group of learning data used for learning processing of the estimation model 514 . The color classification data 518 is data used for generating the learning data set 516 and for learning processing. The profile data 519 is attribute information about the subject 2, and is data in which the profile of the subject 2 such as age, sex, race, height, weight, place of residence, etc. is summarized (for example, medical record information). . The learning program 521 is a program for executing the learning process of the estimation model 514, and part of it also includes a program for executing the estimation process.

Note that the estimated model 514 (learned model) is transmitted to the local data processing device 100 and is held as the estimated model 114 (learned model) by the data processing device 100 .

The arithmetic device 530 is an arithmetic entity that executes various types of processing such as learning processing by executing various programs, and is an example of a computer. Arithmetic device 530 includes, for example, CPU 532, FPGA 534, GPU 536, and the like.

Note that the arithmetic device 530 may be configured with at least one of the CPU 532, FPGA 534, and GPU 536, or may include the CPU 532 and FPGA 534, the FPGA 534 and GPU 536, the CPU 532 and GPU 536, or the CPU 532, FPGA 534, and GPU 536. may be Processing units 530 may also be referred to as processing circuitry.

[Processing by data processor]
An example of estimation processing by data processing apparatus 100 according to the present embodiment will be described with reference to FIG. FIG. 5 is a schematic diagram showing the functional configuration of the data processing device 100 according to this embodiment. FIG. 6 is a schematic diagram for explaining the contents of three-dimensional data input to the data processing apparatus 100 according to this embodiment. FIG. 7 is a schematic diagram showing an example of a tooth to be estimated by the data processing apparatus 100 according to this embodiment. In FIG. 7, the tooth to be scanned by the three-dimensional scanner 200 is represented by a diagram.

As shown in FIG. 5, the data processing apparatus 100 has an input unit 1102, a profile acquisition unit 1119, an estimation unit 1130, an association unit 1101, and an output unit 1103 as functional units related to data processing. Each of these functions is realized by the arithmetic unit 130 of the data processing device 100 executing the OS 127 and the data processing program 120 .

Three-dimensional data acquired by the three-dimensional scanner 200 is input to the input unit 1102 . Profile acquisition unit 1119 acquires profile data 119 of subject 2 . The estimation unit 1130 uses the estimation model 114 (learned model) based on the three-dimensional data input to the input unit 1102 and the profile data 119 of the subject 2 acquired by the profile acquisition unit 1119 to determine the type of tooth. Perform an estimation process to estimate .

Estimation model 114 includes a neural network 1142 and parameters 1144 used by neural network 1142 . Parameters 1144 include weighting factors used in calculations by neural network 1142 and decision values used to determine identity. The associating unit 1101 associates each of the points corresponding to the three-dimensional data input from the input unit 1102 with related data related to the tooth type based on the tooth type estimation result by the estimating unit 1130 . Output unit 1103 outputs the result of association by association unit 1101 to output devices such as display 300 and server device 500 .

Here, as shown in FIG. 6, the three-dimensional data input to the input unit 1102 includes three-dimensional position information at each point on the tooth and color information at each point on the tooth. The position information includes the coordinates of the absolute position in three dimensions with reference to a predetermined position. For example, the positional information can be obtained by using the center position of each point of the tooth as the origin, the X-axis (for example, the axis along the lateral direction of the tooth), the Y-axis (for example, the axis along the longitudinal direction of the tooth), and the Z-axis. (eg, the axis along the height direction of the tooth). Note that the position information is not limited to coordinates of a three-dimensional absolute position with a predetermined position as the origin, but includes coordinates of a three-dimensional relative position indicating the distance and direction (vector) from an adjacent point, for example. You can stay.

Here, as shown in FIG. 7, when the tooth to be scanned by the three-dimensional scanner 200 is an incisor of the upper jaw, the obtained three-dimensional image is a region on the upper lip side, a region on the palate side, and a region on the incisal side. The oral cavity of a subject 2 is scanned by a user 1 so as to include at least an image of the site. Also, when the teeth to be scanned by the three-dimensional scanner 200 are the canines and molars of the upper jaw, the obtained three-dimensional image should include at least images of the buccal region, the palatal region, and the occlusal region. , the oral cavity of a subject 2 is scanned by a user 1 . When the tooth to be scanned by the three-dimensional scanner 200 is a mandibular incisor, the three-dimensional image obtained includes at least images of the lower lip side region, the lingual region, and the incisal side region. A user 1 scans the oral cavity of a subject 2 . When the teeth to be scanned by the three-dimensional scanner 200 are the canines and molars of the mandible, the resulting three-dimensional image includes at least images of the buccal region, the lingual region, and the occlusal region. 1 scans the oral cavity of a subject 2 .

In general, the teeth of the subject 2 differ in shape and size depending on their type. For example, for maxillary incisors, the upper labial surface is generally U-shaped, whereas for maxillary canines, the buccal surface is generally pentagonal. Each tooth has a characteristic shape and size according to its type, and the estimation unit 1130 uses the estimation model 114 based on three-dimensional data in which these characteristic shapes and sizes are quantified. The type of tooth corresponding to the three-dimensional data is estimated.

As shown in FIG. 5, estimation model 114 includes neural network 1142 . In the neural network 1142, the value of the positional information included in the three-dimensional data input to the input unit 1102 is input to the input layer. In the neural network 1142, for example, the intermediate layer multiplies the value of the input positional information by a weighting factor or adds a predetermined bias, and performs calculation using a predetermined function. The result is compared with the decision value. Then, in the neural network 1142, the result of the calculation and determination is output from the output layer as the estimation result. Any method may be used for the calculation and determination by the neural network 1142 as long as the tooth can be estimated based on the three-dimensional data.

In the neural network 1142 of the estimation model 114, processing by deep learning is performed because the intermediate layer has a multi-layered structure. In the present embodiment, for example, VoxNet, 3D ShapeNets, Multi-View CNN, RotationNet, OctNet, FusionNet, PointNet, PointNet++, SSCNet, MarrNet, etc. are used as programs that perform estimation processing specialized for 3D images. program, but other programs may be used. Also, an existing mechanism may be applied to the mechanism of the neural network 1142 .

With such a configuration, when three-dimensional data corresponding to a plurality of points forming a tooth is input, the data processing apparatus 100 estimates the features of the tooth based on the positional information of the plurality of points. can be extracted using the neural network 1142, and the type of tooth corresponding to each of the plurality of points can be estimated based on the extracted features. Furthermore, the data processing device 100 can associate relevant data related to the tooth type with each of the plurality of points based on the estimation result of the tooth type corresponding to each of the plurality of points. In addition, as shown in FIG. 5, not only the tooth to be estimated but also three-dimensional data including adjacent teeth are input to the data processing apparatus 100, so that the neural network 1142 of the estimation model 114 It is possible to extract features of teeth by taking into consideration positional information of a plurality of points constituting a plurality of teeth. The data processing apparatus 100 can extract not only tooth features that are generally recognized by the user 1 or the like, but also tooth features that are not generally recognized, thereby accurately identifying the type of tooth. can be identified.

The neural network included in estimation model 514 held by server device 500 has the same configuration as neural network 1142 included in estimation model 114 shown in FIG.

[Generation of learning data]
An example of generating the learning data set 116 will be described with reference to FIGS. 8 and 9. FIG. FIG. 8 is a schematic diagram for explaining generation of learning data according to the present embodiment. FIG. 9 is a schematic diagram for explaining an example of the learning data set 116 according to this embodiment.

As shown in FIG. 8, first, three-dimensional data is acquired by the three-dimensional scanner 200 (STEP 1). The three-dimensional data acquired by the three-dimensional scanner 200 includes three-dimensional position information at each point on the teeth and gums corresponding to the three-dimensional data, and color information (RGB values) at each point on the teeth and gums. . When a three-dimensional image is generated based on the three-dimensional data acquired by the three-dimensional scanner 200, a three-dimensional image including teeth and gums with actual colors is generated as shown in FIG. 8(a). .

Next, noise removal processing is performed in preparation for color-coding processing for each tooth, which will be described later. For example, in the present embodiment, a 3D image corresponding to 3D data is grayscaled (STEP 2). Grayscaling of the three-dimensional image is performed by the user 1 (in this case, a manufacturer engineer who generates learning data, an operator at a manufacturing plant, a dentist, a dental technician, or other person with dental knowledge). will be When the 3D image is grayscaled, a 3D image including grayscaled teeth and gums is generated as shown in FIG. 8(b). Further, in accordance with the grayscaling of the three-dimensional image, the color information (RGB values) at each point of the teeth and gums corresponding to the three-dimensional data is changed to a value corresponding to the grayscale (for example, 192192192).

Next, each tooth is color-coded by applying a predetermined color to each tooth included in the three-dimensional image corresponding to the three-dimensional data (STEP 3). For example, as shown in FIG. 9, the color classification data 118 held by the data processing device 100 is provided for each site in the oral cavity such as the left mandible, the right mandible, the left mandible, and the right mandible. FIG. 9 shows color classification data 118 corresponding to the left side of the lower jaw. In each color classification data 118, a tooth number generally used in the dental field and predetermined color information are assigned to each tooth type.

For example, the second molar is assigned number 7 as the tooth number and color a as the color information. The first molar is assigned number 6 as the tooth number and color b as the color information. The second premolar is assigned number 5 as the tooth number and color c as the color information. Thus, in each color classification data 118, tooth numbers and color information are assigned in advance to each tooth type.

The application of color to each tooth is performed by a user 1 (in particular, a person with dental knowledge such as a dentist, dental technician, etc.). Specifically, the user 1 identifies the type of each tooth included in the three-dimensional image based on his/her own knowledge, identifies the color corresponding to the identified tooth type with reference to the color classification data 118, Apply the specified color to the image of the tooth.

For example, when the user 1 identifies that the tooth included in the three-dimensional image is the second molar, the user 1 applies color a to the image of the tooth. Further, when the tooth included in the three-dimensional image is identified as the first molar, the color b is applied to the image of the tooth. When the predetermined color is applied to each tooth included in the three-dimensional image, the predetermined color is applied to each tooth as shown in FIGS. 8(c) and 9(d). A three-dimensional image is generated. For the sake of clarity, each tooth is indicated by hatching in the drawing, but actually each tooth is color-coded for each of a plurality of points constituting the tooth.

Further, according to the color coding of each tooth, the color information (RGB values) at each point of the tooth corresponding to the three-dimensional data is changed to a value corresponding to the color applied to each tooth. For example, for each position coordinate of the second molar painted in color a (red), the color information (RGB value) is "255000000", and for the first molar painted in color b (green) For each position coordinate, the color information (RGB value) is "000255000", and for each position coordinate of the second premolar tooth painted in color c (blue), the color information (RGB value) is " 00000255″. That is, predetermined color information (RGB values) is associated with each point of the tooth corresponding to the three-dimensional data.

When predetermined color information is associated with each tooth, the three-dimensional data includes position information and color information corresponding to the applied color. Adopted as training data. That is, in the learning data according to the present embodiment, color information corresponding to the type of tooth is associated (labeled) with the position information referred to in the estimation process. Further, color information is associated with the three-dimensional data so that each range of a plurality of teeth corresponding to the three-dimensional data can be specified. Specifically, the same color information is associated with each piece of position information corresponding to each tooth. A collection of such learning data is held in the data processing device 100 as a learning data set 116 .

In this way, there are many advantages in labeling correct data by applying colors to each tooth included in the three-dimensional image by the user 1 when generating learning data. For example, if the labeling is done using simple characters or symbols, the user 1 will find it difficult to recognize the range of each tooth, but if the labeling is done by color coding, the user 1 will be able to distinguish between the tooth to be labeled and the adjacent tooth. , and between the tooth and gingiva to be labeled can be easily recognized by the application of color. In addition, the user 1 applies the color while checking the three-dimensional image from various angles during labeling. becomes easier to recognize.

In this embodiment, the user 1 manually applies the color to each tooth included in the three-dimensional image based on his/her own knowledge, but it is also possible to supplement part of the work with software. . For example, the boundary between the tooth to be labeled and the tooth adjacent to it, and the boundary between the tooth to be labeled and the gingiva may be identified by edge detection, such that: Only teeth that are labeling targets can be extracted.

Generating the learning data set 116 shown in FIGS. 8 and 9 can also be applied to generating the learning data set 516 held by the server device 500 . For example, learning data set 116 shown in FIG. 9 may be applied to learning data set 516 held by server device 500, and color classification data 118 shown in FIG. It may be applied to data 518 .

[Generate trained model]
An example of generating a learned model will be described with reference to FIG. FIG. 10 is a schematic diagram for explaining generation of a trained model based on learning data set 116 according to the present embodiment.

As shown in FIG. 10, the learning data set 116 can be classified into categories based on the profile of the subject 2 who was scanned when the learning data set 116 was generated. For example, age (minor, working generation, elderly), gender (male, female), race (Asian, Western, African), height (less than 150 cm, 150 or more), weight (less than 50 kg, 50 kg) above), and place of residence (resident in Japan, residing outside of Japan) can be assigned a learning data set generated from three-dimensional data including the teeth of the corresponding subject 2 . Note that the stratification of each category can be set as appropriate. For example, with respect to age, for each predetermined age difference (every three years in this case), specifically, more detailed information such as 0 to 3 years old, 4 to 6 years old, 7 to 9 years old, etc. can be stratified into

The data processing device 100 generates a trained model by training the estimation model 114 using a plurality of training data sets 116a to 116o that can be classified by category. Note that the learning data may overlap depending on how the categories are classified, but if the learning data overlaps, the estimation model 114 may be trained using only one of the learning data. .

In general, tooth shape differs depending on genetics or living environment such as age, sex, race, height, weight, place of residence, and the like. For example, the permanent teeth of adults are generally larger than the milk teeth of children, and the two have different shapes. In general, male teeth are larger than female teeth, and the two have different shapes. In general, the teeth of Westerners tend to have sharp tips so that they can easily bite off hard meat and bread, whereas the teeth of Japanese people tend to have sharp tips so that they can easily grind soft rice and vegetables. It tends to be smooth. Therefore, by performing learning processing based on profile data as in the present embodiment, it is possible to generate a trained model that can identify the type of tooth in consideration of heredity or living environment.

It should be noted that the generation of the trained model shown in FIG. 10 can also be applied to the generation of the trained model held by the server device 500 . For example, the learning data sets 116a to 116o shown in FIG. It may be applied to model 514 .

[Example of association result output]
An example of the output of association results associated with each point constituting the tooth and gum will be described with reference to FIGS. 11 to 14. FIG. 11 and 12 are schematic diagrams for explaining an example of output of the association result according to the present embodiment. 13 and 14 are schematic diagrams for explaining an example of output data according to this embodiment.

As shown in FIG. 11( a ), at the beginning of intraoral scanning by the three-dimensional scanner 200 , the number of three-dimensional data obtained is small, so the data processing device 100 corresponds to each three-dimensional data. Tooth type cannot be estimated for each point. Further, since the data processing apparatus 100 cannot estimate the tooth for each point, it cannot distinguish between the tooth and the gingiva. Therefore, the data processing apparatus 100 cannot associate color information with each point as related data related to the type of tooth. Each point corresponding to the three-dimensional data obtained by scanning is displayed.

FIG. 13(a) shows an example of output data used for image display by the display 300 shown in FIG. 11(a). As shown in FIG. 13(a), the output data includes positional information of each point obtained by scanning and color information (associated data) associated with the positional information. Since the type of tooth is not estimated, the position information of each point is associated with the value (192192192) corresponding to the gray scale as color information (related data). When a value corresponding to a grayscale is associated with each point as color information, each point is represented by a square as shown in FIG. Points are represented in gray.

As shown in FIG. 11B, as the intraoral scanning by the three-dimensional scanner 200 progresses to some extent, the number of three-dimensional data obtained increases. Focusing on one point, three-dimensional data of other points located around the one point is input to the data processing device 100, and the number of input three-dimensional data of other points gradually increases. In the estimation process using the estimation model 114, the data processing device 100 determines the relationship between the one point and the other point (for example, the distance between the two, the vector from the one point to the other point). Based on this, the type of tooth at the one point is estimated. Therefore, as the number of three-dimensional data input to the data processing apparatus 100 increases, the data processing apparatus 100 can gradually estimate the type of tooth corresponding to each point. can also be distinguished. Therefore, the data processing device 100 can also associate relevant data (color information) with each point, and the three-dimensional data obtained by scanning can be displayed on the screen of the display 300 with the relevant data associated. Each point corresponding to is displayed.

Here, even if the number of three-dimensional data input to the data processing device 100 increases, if the number of three-dimensional data corresponding to the surrounding points is insufficient, the data processing device 100 estimates the type of tooth. sometimes you can't. For example, as shown in FIG. 11(b), for some points, the type of tooth cannot be estimated yet, so it is necessary to associate related data (color information) with those points. I can't do it either. As will be described later, the data processing device 100 can gradually associate related data even for some points that cannot be associated with related data as the number of input three-dimensional data increases.

FIG. 13(b) shows an example of output data used for image display by the display 300 shown in FIG. 11(b). As shown in FIG. 13(b), the point where the tooth type is estimated is associated with the value of color a (255000000) corresponding to the tooth type as color information (related data), and the tooth type is estimated. Points that are not marked are associated with values corresponding to grayscale as color information (related data). When the value of color a corresponding to the type of tooth is associated with each point as color information, each point is represented by a circle as shown in FIG. Each point on the screen is represented by color a (eg, red).

In this way, the data processing apparatus 100, for one input three-dimensional data, based on the relationship (distance, vector) with other three-dimensional data contained within a specific range of the one three-dimensional data, A tooth type is estimated at a point corresponding to one three-dimensional data.

As a more specific example, data processing apparatus 100 cannot estimate the type of tooth for any one point (attention point corresponding to one three-dimensional data) due to a small amount of information, or Even if it is estimated, after that, other points (corresponding to other 3D data If the type of tooth can be estimated for the point (point of interest), the type of tooth corresponding to the point that exists in the largest proportion within the specific range is regarded as the type of tooth corresponding to the point (point of interest). do. In this way, the data processing device 100 focuses on one point, considers the tooth type estimation results for other points existing within a specific range of the one point (point of interest), and Estimate the type of tooth for one point (point of interest). In the present embodiment, since each point constituting the tooth is represented by three-dimensional data, the "within a specific range" is a range drawn with an arbitrary radius centered on one point (point of interest). However, "within a specific range" is not limited to the range within which the sphere is drawn, and may be within the range of a rectangular parallelepiped or cube surrounding one point (point of interest). Alternatively, an arbitrary number may be selected in order of proximity from other points that are three-dimensionally closest to one point (target point) from among the other points. “Within a specific range” means a range surrounding one point (point of interest), and there is a point that may affect the result of estimating the type of tooth for the one point (point of interest). It may be within any range as long as it is within the range.

Then, based on the estimation result, the data processing device 100 associates the related data of the color a corresponding to the estimated tooth type with each of the plurality of points corresponding to the input three-dimensional data. Thereby, the data processing apparatus 100 can identify the tooth to be scanned using the related data including the color information of the color a. Further, the data processing device 100 can distinguish each of a plurality of adjacent teeth by identifying the tooth with the color a, and can represent the boundary between the tooth and its adjacent tooth with the color. Further, the data processing apparatus 100 can distinguish between the tooth and the gum by identifying the tooth with the color a, and can represent the boundary between the tooth and the gum with the color.

As shown in FIG. 11(c), as the intraoral scanning by the three-dimensional scanner 200 progresses further, the number of three-dimensional data obtained further increases. Then, the data processing apparatus 100 can estimate the tooth type even for some points where the tooth type could not be estimated. In this way, the association for one tooth is completed.

FIG. 13(c) shows an example of output data used for image display by the display 300 shown in FIG. 11(c). As shown in FIG. 13(c), the point for which the tooth type is estimated is associated with the color a value corresponding to the tooth type as color information (related data). Furthermore, for some points where the tooth type could not be estimated, the tooth type was estimated, so the associated related data changed from the value corresponding to the gray scale to the color a value corresponding to the tooth type. be done.

In this way, the data processing device 100 associates the point corresponding to the input three-dimensional data with the related data of the value corresponding to the gray scale, and then, within the specific range of the one three-dimensional data Corresponding to the gray scale associated with the point corresponding to the one three-dimensional data based on the related data of the color a corresponding to the tooth type associated with the point corresponding to the other three-dimensional data included The value related data is changed to the color a related data corresponding to the tooth type.

To give a more specific example, the data processing apparatus 100 cannot estimate the type of tooth for any one point (attention point corresponding to one three-dimensional data) due to the small amount of information, and once corresponds to gray scale. Even if a value is associated with the other point (point of interest), other points existing within the range (within a specific range) where a sphere is drawn with an arbitrary radius centered on the point (point of interest) (corresponding to other three-dimensional data If the type of tooth can be estimated for the point (point of interest), the type of tooth corresponding to the point that exists in the largest proportion within the specific range is regarded as the type of tooth corresponding to the point (point of interest). do. In this way, the data processing apparatus 100 focuses on one point, considers the result of estimating the tooth type for points existing within a specific range of the one point (point of interest), and Estimate the type of tooth for a point (point of interest).

As shown in FIG. 12(d), at the beginning of the scan of the next tooth, the number of three-dimensional data obtained is small. On the other hand, the type of tooth cannot be estimated. Further, since the data processing apparatus 100 cannot estimate the tooth for each point, it cannot distinguish between the tooth and the gingiva. Therefore, since the data processing apparatus 100 cannot associate color information with each point as related data related to the type of tooth, the scanned image is displayed on the screen of the display 300 in a state in which the related data is not related. Each point corresponding to the three-dimensional data obtained by is displayed.

FIG. 14(d) shows an example of output data used for image display by the display 300 shown in FIG. 12(d). As shown in FIG. 14(d), since the tooth type has not yet been estimated for each point, the position information of each point is associated with a value corresponding to a grayscale as color information (related data). ing.

As shown in FIG. 12(e), as the intraoral scanning by the three-dimensional scanner 200 progresses to some extent, the number of three-dimensional data obtained increases. Then, when a certain point is focused on, three-dimensional data of other points located around the one point are input to the data processing device 100, and the number of input three-dimensional data of other points increases. come. Therefore, by increasing the number of three-dimensional data input to the data processing device 100, the data processing device 100 can estimate the type of tooth corresponding to each point, and distinguish between the tooth and the gingiva. can also be distinguished. Therefore, the data processing device 100 can also associate relevant data (color information) with each point, and the three-dimensional data obtained by scanning can be displayed on the screen of the display 300 with the relevant data associated. Each point corresponding to is displayed.

Here, even if the number of three-dimensional data input to the data processing device 100 increases, the data processing device 100 may incorrectly estimate the type of tooth depending on the three-dimensional data corresponding to the surrounding points. be. For example, as shown in FIG. 12(e), since the tooth type is incorrectly estimated for some points, the associated data (color information) with incorrect values for those points is generated. Associated. As will be described later, the data processing device 100 can gradually associate correct related data even for some points associated with incorrect related data as the number of input three-dimensional data increases. become.

FIG. 14(e) shows an example of output data used for image display by the display 300 shown in FIG. 12(e). As shown in FIG. 14(e), the points for which the tooth type was correctly estimated are associated with the color b value (000255000) corresponding to the correct tooth type as color information (related data). The point for which the type of is estimated is associated with the color a value corresponding to the wrong tooth type as color information (related data). When the value of color b corresponding to the type of tooth is associated with each point as color information, each point is represented by x as shown in FIG. Each point on the screen is represented by color b (eg, green).

In this way, the data processing apparatus 100, for one input three-dimensional data, based on the relationship (distance, vector) with other three-dimensional data contained within a specific range of the one three-dimensional data, A tooth type is estimated at a point corresponding to one three-dimensional data. Then, based on the estimation result, the data processing device 100 associates the related data of the color b corresponding to the estimated tooth type with each of the plurality of points corresponding to the input three-dimensional data. As a result, the data processing apparatus 100 can identify the tooth to be scanned using the related data including the color information of the color b. Further, the data processing apparatus 100 can distinguish between the tooth and the gum by identifying the tooth with the color b, and can represent the boundary between the tooth and the gum with the color.

As shown in FIG. 12(f), as the intraoral scanning by the three-dimensional scanner 200 progresses further, the number of three-dimensional data obtained further increases. Then, the data processing apparatus 100 can correctly estimate the tooth type even for some points where the tooth type was erroneously estimated.

FIG. 14(f) shows an example of output data used for image display by the display 300 shown in FIG. 12(f). As shown in FIG. 14(f), the point for which the tooth type is estimated is associated with the color b value corresponding to the tooth type as color information (related data). Furthermore, for some points where the tooth type was erroneously estimated, the tooth type was correctly estimated. is changed to the value of color b corresponding to .

In this way, the data processing apparatus 100 associates the point corresponding to the one input three-dimensional data with the related data of the color a corresponding to the wrong tooth type, and then Based on the related data of the color b corresponding to the correct tooth type associated with the point corresponding to the other three-dimensional data included in the specific range, the point corresponding to the one three-dimensional data is associated Change the related data of color a to the related data of color b.

As a more specific example, the data processing device 100 once estimates that the tooth type is A (for example, the tooth corresponding to color a) for any one point (point corresponding to one three-dimensional data). After that, the type of tooth for other points (points corresponding to other three-dimensional data) existing within a range (within a specific range) where a sphere is drawn with an arbitrary radius centered on the one point (point of interest) to estimate Among the points existing within the specific range of one point (target point), the point estimated to be tooth type B (for example, the tooth corresponding to color b) is tooth type A. If there are more points than the estimated points and they exist at the highest ratio, the data processing apparatus 100 changes the tooth type from A to B for the one point (target point). In this way, the data processing apparatus 100 focuses on one point, considers the result of estimating the tooth type for points existing within a specific range of the one point (point of interest), and Correct (change) the tooth type estimation result for the point (point of interest), and change the related data associated with the three-dimensional data corresponding to the one point (point of interest).

In addition, once the type of tooth is erroneously estimated for one point (point of interest), and after associating erroneous related data with the three-dimensional data corresponding to the one point (point of interest), When correcting the estimation result for one point (point of interest) in consideration of the estimation result for points existing within a specific range of (point of interest), data processing device 100 corrects the one point (point of interest). ) may be changed at any timing. For example, at the timing when data processing device 100 determines to correct the estimation result for one point (point of interest) in consideration of the estimation results for points existing within a specific range of the one point (point of interest). , the related data associated with the three-dimensional data corresponding to the one point (target point) may be changed to correct related data. Alternatively, after determining that data processing device 100 corrects the estimation result for one point (point of interest) in consideration of the estimation results for points existing within a specific range of the one point (point of interest), Three-dimensional data corresponding to the one point (point of interest) at the timing when a predetermined amount of time has passed, the timing when a predetermined amount of three-dimensional data is additionally acquired, or the timing when the scanning by the three-dimensional scanner 200 is completed. may be changed to correct related data.

In this way, the data processing apparatus 100 displays a point group in which a plurality of points corresponding to the input three-dimensional data are collected, thereby classifying teeth by type using related data including color information ( segmentation). Further, the data processing apparatus 100 can distinguish between teeth and gums by identifying the teeth for each point using related data including color information, and can represent the boundary between the teeth and gums by color. can. Further, the data processing apparatus 100 identifies teeth for each point using related data including color information, thereby displaying the input three-dimensional data as a two-dimensional illustration image for easy use when working. Become. Thereby, the data processing device 100 can facilitate the digital design of orthodontic appliances in orthodontics.

Furthermore, since the data processing apparatus 100 can estimate the type of tooth for each point that constitutes a plurality of adjacent teeth and gingiva in the oral cavity, it is possible to perform neural calculation in consideration of the relationship with the shape of the adjacent tooth. Since the tooth features can be extracted by the estimation model 114 including the network 1142, the type of tooth can be estimated with high accuracy.

[Learning processing of data processor]
A learning process executed by the data processing apparatus 100 will be described with reference to FIG. 15 . FIG. 11 is a flowchart for explaining an example of the learning process executed by data processing apparatus 100 according to the present embodiment. In the present embodiment, data processing device 100 causes estimation model 114 to learn by supervised learning using learning data. Each step shown in FIG. 15 is implemented by the arithmetic device 130 of the data processing device 100 executing the OS 127 and the learning program 121 .

As shown in FIG. 15, the data processing device 100 selects learning data to be used for learning from the learning data set 116 (S1). Specifically, the data processing device 100 selects one or a plurality of learning data from the learning data sets 116 included in the learning data set group shown in FIG. Note that the data processing apparatus 100 is not limited to automatically selecting learning data, and may use learning data selected by the user 1 for learning processing.

The data processing device 100 inputs the three-dimensional data included in the selected learning data and the profile data of the subject 2 who was scanned when the learning data was generated to the estimation model 114 (S2). At this time, the data processing device 100 does not receive the correct answer data labeled with the three-dimensional data. The data processing device 100 executes an estimation process of estimating the type of tooth using the estimation model 114 based on the characteristics of the tooth corresponding to the three-dimensional data (S3). In the estimation process, the data processing device 100 estimates the type of tooth using the estimation model 114 based on profile data in addition to three-dimensional data. At this time, the data processing apparatus 100 estimates the type of tooth using the estimation model 114 for each of a plurality of points corresponding to the three-dimensional data input by scanning.

The data processing device 100 calculates the parameters 1144 of the estimation model 114 based on the error between the tooth type estimation result at each point estimated by the estimation process and the correct data at each point corresponding to the learning data used in the learning process. is updated (S4).

For example, the data processing device 100 estimates color information corresponding to a specific tooth type for each of a plurality of points based on three-dimensional data corresponding to a plurality of points forming the specific tooth. The data processing device 100 compares the color information (correct data) for each of the plurality of points corresponding to the specific tooth included in the learning data with the color information for each of the plurality of points estimated by itself, and matches. For the points, the parameters 1144 of the estimation model 114 are maintained, while for the points that do not match, the parameters 1144 of the estimation model 114 are updated so that they match.

Alternatively, the data processing device 100 estimates color information corresponding to a specific tooth type for each of a plurality of points based on three-dimensional data corresponding to a plurality of points forming the specific tooth, and performs color classification data 118. Based on this, the tooth type and tooth number (correct data) corresponding to the color information are specified for each of a plurality of points. The data processing device 100 stores the tooth type and tooth number (correct data) assigned to the color information for each of the plurality of points corresponding to the specific tooth included in the learning data, and the plurality of points estimated by itself. The tooth type and tooth number are compared for each tooth, and while maintaining the parameters 1144 of the estimation model 114 for the points that match, the parameters 1144 of the estimation model 114 are updated so that the two match for points that do not match. do.

Next, the data processing device 100 determines whether or not learning has been performed based on all the learning data (S5). If the data processing apparatus 100 has not learned based on all the learning data (NO in S5), the process returns to S1.

On the other hand, if the data processing device 100 has learned based on all the learning data (YES in S5), the data processing device 100 stores the learned estimation model 114 as a learned model (S6), and terminates this process.

In this way, the data processing device 100 uses the tooth information (color information, tooth name, tooth number, etc.) corresponding to the tooth type associated with the three-dimensional data included in the learning data as correct data, A learned model can be generated by causing the estimation model 114 to learn based on the estimation result of the tooth type using the three-dimensional data obtained by the estimation process.

In addition, the data processing device 100 uses three-dimensional data including tooth types associated with each of a plurality of points constituting the tooth and three-dimensional positional information of each of the plurality of points to determine the tooth type. By making the estimation model 114 learn based on the estimation result of the type, the type of tooth can be estimated for each of a plurality of points forming the tooth.

Furthermore, in the learning process, the data processing device 100 learns the estimation model 114 in consideration of the profile data in addition to the learning data.

[Learning processing of the server device]
The learning process executed by the server device 500 will be described with reference to FIG. 16 . FIG. 16 is a flowchart for explaining an example of learning processing executed by server device 500 according to the present embodiment. In the present embodiment, server device 500 learns estimation model 514 by supervised learning using learning data. Each step shown in FIG. 16 is implemented by the arithmetic device 530 of the server device 500 executing the OS 527 and the learning program 521 .

As shown in FIG. 16, the server device 500 selects learning data to be used for learning from the learning data set 516 (S501). Here, the learning data may be generated using big data accumulated and stored by the server device 500 . For example, the server device 500 generates learning data in advance using three-dimensional data included in the scan information acquired from the data processing devices 100 of each of the local A to C and the dental laboratory, and the generated learning data A learning process may be performed using the data. Note that the server device 500 is not limited to automatically selecting learning data, and may use learning data selected by the user 1 for learning processing.

The server device 500 inputs the three-dimensional data included in the selected learning data and the profile data of the subject 2 who was scanned when the learning data was generated to the estimation model 514 (S502). At this time, the correct answer data labeled with the three-dimensional data is not input to the server device 500 . The server apparatus 500 executes an estimation process of estimating the type of tooth using the estimation model 514 based on the characteristics of the tooth corresponding to the three-dimensional data (S503). In the estimation process, server device 500 estimates the type of tooth using estimation model 514 based on profile data in addition to three-dimensional data. At this time, server device 500 estimates the type of tooth using estimation model 514 for each of a plurality of points corresponding to the three-dimensional data input by scanning.

The server device 500 updates the parameters of the estimation model 514 based on the error between the tooth type estimation result at each point estimated by the estimation process and the correct data at each point corresponding to the learning data used in the learning process. (S504).

For example, server device 500 estimates color information corresponding to a specific tooth for each of a plurality of points based on three-dimensional data corresponding to a plurality of points forming the specific tooth. The server device 500 compares the color information (correct data) for each of a plurality of points corresponding to the specific tooth included in the learning data with the color information for each of a plurality of points estimated by itself, and finds the matching points. While maintaining the parameters of the estimation model 514 for the points that do not match, the parameters of the estimation model 514 are updated so that both match.

Alternatively, server device 500 estimates color information corresponding to a specific tooth type for each of a plurality of points based on three-dimensional data corresponding to a plurality of points forming a specific tooth, and stores the color information in color classification data 518. Based on this, the tooth type and tooth number (correct data) corresponding to the color information are specified for each of a plurality of points. The server device 500 stores the tooth type and tooth number (correct data) assigned to the color information for each of the plurality of points corresponding to the specific tooth included in the learning data, and the color information for each of the plurality of points estimated by itself. The tooth type and tooth number are compared with each other, and while maintaining the parameters of the estimated model 514 for the points that match, the parameters of the estimated model 514 are updated so that both match for the points that do not match.

Next, the server apparatus 500 determines whether or not learning has been performed based on all the learning data (S505). If the server apparatus 500 has not learned based on all the learning data (NO in S505), the process returns to S501.

On the other hand, if the server device 500 has learned based on all the learning data (YES in S505), the server device 500 stores the learned estimation model 514 as a learned model (S506). After that, the server device 500 transmits the generated learned model to each local data processing device 100 (S507), and ends this processing.

In this way, the server device 500 uses the tooth information (color information, tooth name, tooth number) corresponding to the type of tooth associated with the three-dimensional data included in the learning data as correct data, and performs estimation processing. A learned model can be generated by causing the estimation model 514 to learn based on the result of estimating the type of tooth using the three-dimensional data.

In addition, the server device 500 uses three-dimensional data including tooth types associated with each of a plurality of points forming a tooth and three-dimensional positional information of each of the plurality of points to determine the tooth type. By making the estimation model 514 learn based on the estimation results of , the type of tooth can be estimated for each of a plurality of points forming the tooth.

In addition, in the learning process, the server device 500 learns the estimation model 514 in consideration of the profile data in addition to the learning data.

Furthermore, since the server device 500 uses the three-dimensional data included in the scan information acquired from the data processing devices 100 of the local A to C and the dental laboratory as learning data used for learning processing, the data processing The learning process can be performed based on more learning data than the learning process performed for each device 100, and a trained model that can identify the type of tooth with higher accuracy can be generated. can.

[Service provision processing of data processor]
A service providing process executed by the data processing apparatus 100 will be described with reference to FIG. 17 . FIG. 17 is a flowchart for explaining an example of service providing processing executed by data processing apparatus 100 according to the present embodiment. Note that the service providing processing shown in FIG. 17 and FIGS. 19, 21, and 23, which will be described later, is an embodiment of "data processing." Each step shown in FIG. 17 is realized by the arithmetic device 130 of the data processing device 100 executing the OS 127 and the data processing program 120 .

As shown in FIG. 17, the data processing device 100 determines whether three-dimensional data corresponding to a predetermined point has been input (S41). If three-dimensional data has not been input (NO in S41), the data processing device 100 ends this process.

On the other hand, when the three-dimensional data is input (YES in S41), the data processing device 100 determines whether or not the profile data of the subject 2 is input by the user 1 (S42). If profile data has not been input (NO in S42), data processing device 100 inputs three-dimensional data to the learned model (S43). On the other hand, if profile data has been input (YES in S42), data processing device 100 inputs the three-dimensional data and profile data to the learned model (S44). The trained model used at this time is not limited to the trained model generated by the data processing device 100 in the learning process shown in FIG. 15, and the trained model generated by the server device 500 in the learning process shown in FIG. may be

After S43 and S44, the data processing apparatus 100 executes an estimation process of estimating the type of tooth using a learned model based on the characteristics of the tooth corresponding to the three-dimensional data (S45). At this time, if the profile data has been input to the learned model in S44, the data processing device 100 estimates the type of the tooth using the learned model based on the profile data in addition to the three-dimensional data. In this case, the tooth type can be estimated with higher accuracy than when the tooth type is estimated using a trained model based only on three-dimensional data.

Based on the estimation result obtained by the estimation process, the data processing device 100 associates related data (for example, color information) indicating the estimated tooth type with the three-dimensional data input in S41 (S46). Specifically, as shown in FIGS. 13 and 14, position information included in the three-dimensional data is associated with color information corresponding to the estimated tooth type. As a result, the point corresponding to the input three-dimensional data is associated with related data (color information) as a result of estimating the type of tooth corresponding to the point.

Data processing device 100 outputs the association result to display 300, server device 500, etc. (S47). Specifically, as shown in FIGS. 11 and 12, the point corresponding to the input three-dimensional data is associated with related data (color information) as a result of estimating the type of tooth corresponding to the point. and the result of the association is displayed on the display 300. FIG. After that, the data processing device 100 terminates this process.

The data processing apparatus 100 executes the service providing process shown in FIG. 17 for each of a plurality of points that constitute the tooth, thereby distinguishing the intraoral tooth that is the scanning target for each type and and gingiva.

Further, when the data processing apparatus 100 cannot estimate the type of tooth by the service providing process executed for the predetermined point, the data processing apparatus 100 skips the processes of S45 to S47 and ends this process. After that, when the data processing device 100 estimates the type of tooth by the service providing process executed for another point and associates the relevant data, based on the relevant data associated with the other point, Related data is also associated with the predetermined point.

Furthermore, if the data processing device 100 erroneously estimates the type of tooth and associates erroneous related data with the service provision processing executed for a predetermined point, the data processing device 100 subsequently performs service provision processing for other points. When the tooth type is estimated and the related data is associated, the correct related data is associated with the predetermined point based on the related data associated with the other points.

As described above, the data processing apparatus 100 estimates the type of the tooth using the learned model based on the characteristics of the tooth corresponding to the input three-dimensional data. The type of tooth can be estimated with higher accuracy than estimating .

In addition, the data processing device 100 estimates the type of tooth for each of a plurality of points forming the tooth to be scanned, and associates the estimation result with color information as related data for each of the plurality of points. , it is possible to show the user the estimation result of the tooth type in a more detailed range. The data processing device 100 can distinguish between teeth and gingiva by associating relevant data with teeth, and can also classify (segment) teeth according to their types, enabling digital design of orthodontic appliances. become easier.

Furthermore, in the estimation process, the data processing apparatus 100 considers the profile data in addition to the input three-dimensional data to identify the type of tooth, so that the type of tooth can be identified with higher accuracy.

[Main Disclosure]
As described above, the present embodiment includes the following disclosures.

The data processing apparatus 100 includes an input unit 1102 to which three-dimensional data including three-dimensional positional information on each of a plurality of points constituting a tooth is input, the three-dimensional data input from the input unit 1102, and the three-dimensional data input from the input unit 1102. an estimation model 114 including a neural network 1142 for estimating the type of tooth based on the data; 1101 and an output unit 1103 that outputs the result of association by the association unit 1101 .

As a result, the data processing device 100 inputs three-dimensional data including three-dimensional position information of each of a plurality of points that constitute the tooth to the estimation model 114 (learned model) including the neural network 1142, Since each of a plurality of points corresponding to three-dimensional data can be associated with related data related to the type of tooth, the tooth can be classified (segmented) for each of the plurality of points that constitute the tooth, The type of tooth can be identified with high accuracy.

The data processing apparatus 100 includes an estimation unit 1130 that estimates the type of tooth for each of a plurality of points corresponding to the three-dimensional data based on the three-dimensional data input from the input unit 1102 and the estimation model 114. Unit 1101 associates related data with each of a plurality of points corresponding to three-dimensional data input from input unit 1102 based on the estimation result by estimation unit 1130 .

As a result, the user 1 inputs three-dimensional data including three-dimensional positional information for each of a plurality of points forming the tooth into the estimation model 114 (learned model) including the neural network 1142, thereby can be estimated, the type of tooth can be identified with higher accuracy than when identifying the type of tooth depending on the user's own knowledge. Further, since the user 1 can estimate the type of tooth for each of a plurality of points forming the tooth, it is possible to obtain the result of tooth type estimation in a more detailed range.

The estimating unit 1130 estimates the one three-dimensional data input from the input unit 1102 based on the relationship with other three-dimensional data included in the specific range of the one three-dimensional data. Estimate the tooth type at the corresponding point.

With this, the user 1 can consider other three-dimensional data corresponding to other points included within a specific range of one point among the plurality of points constituting the tooth, and can Since the type can be estimated, each point can be classified by tooth type with higher accuracy.

The associating unit 1101 associates the related data with the point corresponding to the one three-dimensional data input from the input unit 1102, and then associates the other three-dimensional data included in the specific range of the one three-dimensional data with Based on the relevant data associated with the corresponding point, the relevant data associated with the point corresponding to the one three-dimensional data is changed.

With this, the user 1 considers other three-dimensional data corresponding to other points included in a specific range of one point among the plurality of points constituting the tooth, and associates the point with the one point. Since erroneous related data can be changed to correct related data, each point can be classified by tooth type with higher accuracy.

The estimation model 114 estimates the type of the tooth using three-dimensional data including the type of tooth associated with each of a plurality of points constituting the tooth and three-dimensional positional information of each of the plurality of points. It is learned based on results.

This allows the user 1 to learn the estimation model 114 for each of a plurality of points forming the tooth.

The three-dimensional data input from the input unit 1102 includes three-dimensional positional information for each of a plurality of points forming teeth and gums in the oral cavity.

As a result, the user 1 can extract the characteristics of the teeth by the estimation model 114 including the neural network 1142 in consideration of the shapes of the teeth and gums in the oral cavity, so that the type of the teeth can be accurately identified. . In addition, the user 1 can distinguish between teeth and gingiva and classify (segment) teeth according to their types, thereby facilitating digital design of orthodontic appliances.

The related data includes at least one of colors, letters, numbers, and symbols corresponding to tooth types.

Thereby, the user 1 uses at least one of colors, letters, numbers, and symbols to associate relevant data related to the estimated tooth type with each of the plurality of points that constitute the tooth. The teeth can be classified (segmented) in a more comprehensible manner. Also, the user 1 can intuitively recognize the result of association, which improves convenience.

The output unit 1103 outputs the result of the association by the association unit 1101 by point cloud display representing a plurality of points corresponding to the three-dimensional data input from the input unit 1102 .

As a result, the user 1 can display the result of associating the relevant data with each of the plurality of points that constitute the tooth by the point cloud display, so that the tooth can be classified (segmented) for each of the plurality of points in a more comprehensible manner. )can do.

The data processing system 10 uses a three-dimensional camera to acquire three-dimensional data including three-dimensional position information at each of a plurality of points that constitute the tooth, and a three-dimensional scanner 200 acquired by the three-dimensional scanner 200. a data processing device 100 for associating relevant data related to tooth types with each of a plurality of points corresponding to three-dimensional data, wherein the data processing device 100 stores the three-dimensional data acquired by the three-dimensional scanner 200 as: Based on an input unit 1102 to be input, three-dimensional data input from the input unit 1102, and an estimation model 114 including a neural network 1142 for estimating the type of tooth based on the three-dimensional data, the three-dimensional data and an output unit 1103 for outputting the result of association by the associating unit 1101 .

As a result, the user 1 inputs three-dimensional data including three-dimensional positional information of each of a plurality of points forming the tooth into the estimation model 114 (learned model) including the neural network 1142, thereby obtaining a three-dimensional Since each of the multiple points corresponding to the data can be associated with related data related to the type of tooth, it is possible to classify (segment) the tooth for each of the multiple points that make up the tooth with high accuracy. Tooth type can be identified.

The data processing method includes a step (S41) of inputting three-dimensional data including three-dimensional positional information of each of a plurality of points constituting a tooth, and a step of inputting (S41) the input three-dimensional data. , an estimation model 114 including a neural network 1142 for estimating the tooth type based on the three-dimensional data, and related data related to the tooth type for each of a plurality of points corresponding to the three-dimensional data. and a step (S47) of outputting the result of the association by the associating step (S46).

As a result, the user 1 inputs three-dimensional data including three-dimensional positional information of each of a plurality of points forming the tooth into the estimation model 114 (learned model) including the neural network 1142, thereby obtaining a three-dimensional Since each of the multiple points corresponding to the data can be associated with related data related to the type of tooth, it is possible to classify (segment) the tooth for each of the multiple points that make up the tooth with high accuracy. Tooth type can be identified.

The data processing program 120 performs a step (S41) of inputting three-dimensional data including three-dimensional position information of each of a plurality of points forming a tooth into the arithmetic device 130, and a step of inputting (S41). Based on the input three-dimensional data and an estimation model 114 including a neural network 1142 for estimating the type of tooth based on the three-dimensional data, a tooth is extracted for each of a plurality of points corresponding to the three-dimensional data. and a step (S47) of outputting the result of association by the step of associating (S46).

As a result, the user 1 inputs three-dimensional data including three-dimensional positional information of each of a plurality of points forming the tooth into the estimation model 114 (learned model) including the neural network 1142, thereby obtaining a three-dimensional Since each of the multiple points corresponding to the data can be associated with related data related to the type of tooth, it is possible to classify (segment) the tooth for each of the multiple points that make up the tooth with high accuracy. Tooth type can be identified.

[Modification]
The present disclosure is not limited to the above embodiments, and various modifications and applications are possible. Modifications applicable to the present invention will be described below.

(Output of association result)
As shown in FIGS. 11 and 12, data processing apparatus 100 according to the present embodiment obtains the result of associating relevant data with respect to a plurality of points corresponding to input three-dimensional data. Although the point cloud is displayed to the user 1, the association result may be displayed to the user 1 by a method other than the point cloud display.

For example, FIG. 18 is a schematic diagram for explaining an example of an association result output according to the modification. As shown in FIG. 18, the data processing device 100 may display the results of association of related data with a plurality of points corresponding to the input three-dimensional data on the display 300 using polygon meshes. A polygon mesh is a method of displaying an object by arranging polygons such as triangles and quadrilaterals on the screen. Data processing device 100 replaces a plurality of points corresponding to the input three-dimensional data with polygons, and displays on display 300 a result in which related data is associated with the plurality of points.

For example, as shown in FIGS. 18(a) and 18(b), when data processing device 100 cannot associate related data with respect to some points, the result of association is different from the case where related data can be associated. display using a polygon mesh. Further, as shown in FIGS. 18A and 18B, when the data processing device 100 erroneously associates the related data with respect to some points, the result of the association is changed to the other It is displayed using a polygon mesh in a manner different from the point of .

In this way, the output unit 1103 of the data processing apparatus 100 outputs the result of the association by the association unit 1101 using polygon meshes representing the tooth composed of a plurality of points corresponding to the three-dimensional data input from the input unit 1102. You may

As a result, the user 1 can represent the result of associating the relevant data with each of the plurality of points that constitute the tooth by the polygon mesh, so that the tooth can be classified (segmented) in a more comprehensible manner using a three-dimensional image. ) and can also distinguish between teeth and gums.

Furthermore, the data processing device 100 may display the result of association of related data with a plurality of points corresponding to the input three-dimensional data on the display 300 using a three-dimensional model such as a surface model or a solid model. .

(Learning processing during service provision processing)
As shown in FIG. 17, the data processing device 100 according to the present embodiment does not execute learning processing in the service providing processing, but as shown in FIG. The learning process may be executed in the service providing process. The data processing device 100 may perform so-called federated learning. FIG. 19 is a flowchart for explaining an example of service providing processing executed by the data processing device 100a according to the modification. 19 are the same as the processes of S41 to S47 shown in FIG. 17, detailed description of the processes of S41 to S47 is omitted in FIG.

As shown in FIG. 19, the data processing device 100a outputs the results of associating related data based on the estimation results by the processes of S41 to S47, and then executes the learning process at the time of service provision. Specifically, after S47, the data processing device 100a determines whether correct data for error correction has been input (S61a). For example, when the type of tooth estimated in S45 is different from the type of tooth that was actually scanned, the data processing device 100a allows the user 1 to input the type of tooth that was actually scanned. Determine whether the error has been corrected.

If the correct data for error correction is not input (NO in S61a), the data processing device 100a ends this process. On the other hand, when the correct data for error correction is input (YES in S61a), the data processing device 100a gives a reward based on the estimation result and the correct data (S62a).

For example, the smaller the degree of dissociation between the estimation result and the correct answer data, the smaller the negative points given as a reward, and the greater the degree of dissociation between the two, the larger the negative points given as a reward. Just do it. Specifically, if the tooth output as the estimation result and the tooth input as correct data are adjacent to each other, the data processing device 100a gives a small negative point, and if the two are separated, a negative point is given. give a big minus point. In this way, the data processing device 100a gives different rewards depending on the degree of dissociation between the estimation result and the correct data. Note that the reward is not limited to minus points, and may be plus points.

The data processing device 100a updates the parameters 1144 of the trained model based on the given reward (S63a). For example, the data processing device 100a updates the parameter 1144 of the learned model so that the negative points given as a reward approach zero. After that, the data processing device 100a terminates this process.

In this way, the data processing device 100a according to the modification executes learning processing even in the service providing processing. Therefore, the more the user 1 uses the data processing device 100a, the more the accuracy of the estimation processing improves, and the more accurately the type of tooth is identified. can do. Such processing is a kind of so-called collaborative learning.

(Generation of trained models for each category)
As shown in FIG. 10, the data processing device 100 according to the present embodiment learns the estimation model 114 using a learning data set group including a plurality of learning data sets 116a to 116o classified by category. However, as shown in FIG. 20, the data processing device 100b according to the modification generates each of a plurality of learning data sets classified by category. A trained model for each category may be generated by training the estimation model 114 using each category. FIG. 20 is a schematic diagram for explaining generation of a trained model based on the learning data set according to the modification.

As shown in FIG. 20, the learning data set 116 is classified and held for each category based on the profile of the subject 2 who was scanned when the learning data set 116 was generated. For example, learning data sets are assigned to six categories based on age (minors, working generation, elderly) and sex (male, female).

The data processing device 100b according to the modification causes the estimation model 114 to learn using each of the plurality of learning data sets 116p to 116u classified for each category, thereby learning the trained models 114p to 114p for each category. 114u.

In this way, the data processing device 100b according to the modification can generate a plurality of trained models 114p to 114u classified by category. The type of tooth can be identified with high accuracy.

The generation of the trained models 114p to 114u shown in FIG. 20 can also be applied to the generation of trained models held by the server device 500. FIG. For example, the learning data sets 116p-116u shown in FIG. It may be applied to the retained trained model.

(Service provision processing using trained model for each category)
A service providing process executed by the data processing device 100 using the trained models 114p to 114u for each category will be described with reference to FIG. FIG. 21 is a flowchart for explaining an example of service providing processing executed by the data processing device 100b according to the modification. 21 are the same as the processes of S41, S42, and S45 to S47 shown in FIG. 17, therefore, in FIG. A detailed description of is omitted.

As shown in FIG. 21, data processing device 100b receives three-dimensional data corresponding to a predetermined point (YES in S41), and when user 1 inputs profile data of subject 2 (YES in S42). YES), a trained model corresponding to the profile data is selected from the trained model group shown in FIG. 20 (S61b). For example, if the subject 2 is an elderly woman, the data processing device 100b selects the learned model 114u.

After that, the data processing device 100b inputs the three-dimensional data to the learned model (S62b). The data processing device 100b executes an estimation process for estimating the type of the tooth using the learned model based on the characteristics of the tooth corresponding to the three-dimensional data (S45), and based on the estimation result obtained by the estimation process , associated data indicating the estimated tooth type with three-dimensional data (S46), and the result of the association is output (S47).

In this way, the data processing device 100b according to the modification can execute the estimation process using the trained model most suitable for the profile of the subject 2, so that more detailed analysis according to the profile of the subject 2 Therefore, the type of tooth can be identified with higher accuracy.

(Profile output)
The data processing apparatus 100 according to the present embodiment identifies the type of tooth by estimation processing. However, as shown in FIGS. 10 and 22, in view of the fact that the trained model is generated based on the learning data set considering the profile of the subject 2, the three-dimensional data is input to the trained model in the estimation process. By doing so, the profile of the owner of the tooth may be output as an estimation result based on the characteristics of the tooth corresponding to the three-dimensional data. In this way, the profile of the unidentified subject 2 found in a disaster or incident can be specified from the three-dimensional data including teeth.

(learning process)
Data processing apparatus 100 according to the present embodiment updates parameter 1144 of estimation model 114 by learning processing, but is not limited to updating parameter 1144, and neural network 1142 is updated by learning processing. (eg, the algorithm of neural network 1142 is updated). Server device 500 according to the present embodiment updates the parameters of estimation model 514 by learning processing, but is not limited to updating the parameters, and the neural network is updated by learning processing (for example, , the algorithm of the neural network is updated).

(identification using normal and/or color information)
FIG. 22 is a schematic diagram for explaining an example of a learning data set according to a modification. The data processing device 100c according to the modified example inputs the actual color information of the tooth into the estimation model 114 in addition to the position information included in the three-dimensional data acquired by the three-dimensional scanner 200, thereby making the estimation model 114 You can learn.

For example, as described with reference to FIG. 9, the learning data set includes position information, which is input data for the estimation model 114, and color information after color coding associated with tooth types, which is correct data. In addition to this, as shown in FIG. 22, tooth color information before color-coding may also be included. In the learning process, in addition to the positional information, the color information of the teeth before color-coding is input to the estimation model 114, so that a learned model is generated in consideration of the actual color information of the teeth. good too.

Furthermore, in the learning data set, in addition to position information, which is input data for the estimation model 114, and color information after color coding associated with tooth types, which is correct data, as shown in FIG. Normal information that can be calculated based on the position information may also be included. Then, in the learning process, by inputting the normal information in addition to the position information to the estimation model 114, the learned model may be generated in consideration of the normal information.

Normal information can be calculated, for example, as follows. For example, one of a plurality of points forming a tooth is focused on, and a normal to the point of interest is generated based on a plurality of points belonging to a predetermined range near the point of interest. Specifically, the normal to the point of interest can be generated using principal component analysis for a plurality of points belonging to a predetermined range near the point of interest. Principal component analysis can generally be performed by computing a variance-covariance matrix. Eigenvectors are calculated in the calculation of the variance-covariance matrix, and the principal component direction is generated as the normal to the point of interest. Since the method of generating a normal to a point in the point group is well known, other generally known techniques may be used.

In this way, by adding the normal information to the learning data set, the data processing device can also learn which side of the tooth that each point constitutes is the surface for each of the plurality of points that constitute the tooth. can. In addition, the data processing device can learn features of a shape such as a depression based only on a small number of point groups belonging to a predetermined range near the point of interest.

The learning data set may contain both color information and normal line information of the tooth before color-coding, or may contain only one of them.

Next, a service providing process for performing an estimation process using a trained model trained based on a learning data set containing tooth color information and normal information before color-coding will be described with reference to FIG. 23 . FIG. 23 is a flowchart for explaining an example of service providing processing executed by the data processing device 100c according to the modification. Note that the processes of S41, S42, S45 to S47 shown in FIG. 23 are the same as the processes of S41, S42, S45 to S47 shown in FIG. A detailed description of is omitted.

As shown in FIG. 23, the data processing device 100c according to the modification additionally executes the processing of S61c, unlike the service providing processing executed by the data processing device 100 shown in FIG. That is, after the three-dimensional data corresponding to the predetermined point is input (YES in S41), the data processing device 100c determines the normal line of the point forming the tooth based on the position information included in the input three-dimensional data. is generated (S61c). In addition to the position information, the input three-dimensional data includes color information of the teeth before color-coding.

After that, if the profile data has not been input (NO in S42), the data processing device 100c inputs the normal information to the learned model in addition to the three-dimensional data (S62c). On the other hand, if the profile data has been input (YES in S42), the data processing device 100c inputs the normal information to the learned model in addition to the three-dimensional data and the profile data (S63c). After S62c and S63c, the data processing device 100c executes an estimation process of estimating the type of tooth using the learned model (S45).

In this way, the data processing device 100c according to the modification further calculates the color information of the tooth before color-coding and the normal lines generated for each of the plurality of points forming the tooth corresponding to the three-dimensional data. The type may be identified. Note that the data processing device 100c may be input with color information of teeth before color separation, but without input of normal line information. Alternatively, the data processing device 100c may receive normal line information but not tooth color information before color separation.

In this way, the data processing device 100c can identify the type of tooth based on the color information of the tooth before color-coding and/or the normal lines generated for each of the plurality of points. Able to identify types.

It should be considered that the embodiments disclosed this time are illustrative in all respects and not restrictive. The scope of the present disclosure is indicated by the scope of claims rather than the above description, and is intended to include all modifications within the meaning and scope of equivalents of the scope of claims. Note that the configurations exemplified in this embodiment and the configurations exemplified in the modifications can be combined as appropriate.

1 user, 2 subject, 5 network, 10 data processing system, 100, 100a, 100b data processing device, 102 scanner interface, 103, 503 display interface, 105, 505 peripheral device interface, 106, 506 network controller, 107, 507 Media reader, 108 PC display, 109,509 memory, 110,510 storage, 112,512 scan information, 114,514 estimation model, 114a, 514a trained model, 116,516 learning data set, 118,518 color classification data, 119,519 profile data, 120 data processing program, 121,521 learning program, 122,522 three-dimensional data, 124,524 estimation result, 127,527 OS, 130,530 arithmetic device, 200 three-dimensional scanner, 300,350 display, 500 server device, 550 removable disk, 601,651 keyboard, 602,652 mouse, 1102 input unit, 1103 output unit, 1119 profile acquisition unit, 1130 estimation unit, 1142 neural network, 1144 parameters.

Claims (13)

A data processing device for processing data related to tooth type, comprising:
an input unit for inputting three-dimensional data including three-dimensional positional information for each of a plurality of points forming a tooth;
Based on the three-dimensional data input from the input unit and an estimation model including a neural network for estimating the type of tooth based on the three-dimensional data, for each of a plurality of points corresponding to the three-dimensional data an associating unit for associating relevant data associated with tooth types;
and an output unit that outputs a result of association by the association unit. an estimation unit for estimating the type of tooth for each of a plurality of points corresponding to the three-dimensional data based on the three-dimensional data input from the input unit and the estimation model;
2. The data processing device according to claim 1, wherein said associating unit associates said related data with each of a plurality of points corresponding to three-dimensional data input from said input unit based on an estimation result by said estimating unit. . The estimation unit converts the one three-dimensional data input from the input unit to the one three-dimensional data based on the relationship with other three-dimensional data included in a specific range of the one three-dimensional data 3. A data processing apparatus according to claim 2, for estimating tooth type at corresponding points. 3. The estimating unit further estimates the type of tooth for each of the plurality of points based on normal lines generated for each of the plurality of points corresponding to the three-dimensional data input from the input unit. Item 4. The data processing device according to item 3. The associating unit, after associating the related data with a point corresponding to the one three-dimensional data input from the input unit, other three-dimensional data included within a specific range of the one three-dimensional data According to any one of claims 1 to 4, the relevant data associated with the point corresponding to the one three-dimensional data is changed based on the relevant data associated with the point corresponding to Data processing apparatus as described. The estimation model estimates the type of tooth using three-dimensional data including tooth types associated with each of a plurality of points constituting a tooth and three-dimensional position information of each of the plurality of points. 6. The data processing device according to any one of claims 1 to 5, wherein the learning is performed based on the results. The three-dimensional data input from the input unit according to any one of claims 1 to 6, wherein the three-dimensional data includes three-dimensional position information at each of a plurality of points that constitute teeth and gums in the oral cavity. Data processing equipment. The data processing device according to any one of claims 1 to 7, wherein said related data includes at least one of colors, letters, numbers, and symbols corresponding to tooth types. 9. The output unit according to any one of claims 1 to 8, wherein the output unit outputs the result of association by the association unit using a point cloud display representing a plurality of points corresponding to the three-dimensional data input from the input unit. A data processing device according to any one of the preceding paragraphs. 1 to 9, wherein the output unit outputs the result of the association by the association unit using a polygon mesh representing a tooth composed of a plurality of points corresponding to the three-dimensional data input from the input unit. The data processing device according to any one of . A data processing system for processing data related to tooth type, comprising:
A three-dimensional scanner that acquires three-dimensional data including three-dimensional position information at each of a plurality of points that constitute the tooth using a three-dimensional camera;
a data processing device that associates relevant data related to tooth types with each of a plurality of points corresponding to the three-dimensional data acquired by the three-dimensional scanner;
The data processing device is
an input unit into which three-dimensional data acquired by the three-dimensional scanner is input;
Based on the three-dimensional data input from the input unit and an estimation model including a neural network for estimating the type of tooth based on the three-dimensional data, for each of a plurality of points corresponding to the three-dimensional data an associating unit for associating relevant data associated with tooth types;
and an output unit that outputs a result of association by the association unit. A computer data processing method for processing data relating to tooth type, comprising:
a step of inputting three-dimensional data including three-dimensional positional information of each of a plurality of points constituting the tooth;
Each of a plurality of points corresponding to the three-dimensional data based on the three-dimensional data input in the input step and an estimation model including a neural network for estimating the type of tooth based on the three-dimensional data associating relevant data relating to tooth type to
and outputting a result of association by the associating step. A data processing program for processing data related to tooth type,
The data processing program is a computer,
a step of inputting three-dimensional data including three-dimensional positional information of each of a plurality of points constituting the tooth;
Each of a plurality of points corresponding to the three-dimensional data based on the three-dimensional data input in the input step and an estimation model including a neural network for estimating the type of tooth based on the three-dimensional data associating relevant data relating to tooth type to
A data processing program causing execution of a step of outputting a result of association by the associating step.

Images