JP7178016B2 - Image processing device and its image processing method - Google Patents

Description

The present invention is a neural network (NN) based image processing device capable of processing tooth object images, identifying and classifying individual teeth , arranging them, arranging them, and identifying them, and , an image processing apparatus capable of accurately and quickly locating, arranging, and identifying an object image using the imaging apparatus, and determining a dental formula, etc. , and an image processing method using the same. In particular, the present invention relates to an image processing apparatus and an image processing method using the same, which are suitable for determining dental formulas from dental X-ray digital photographs .

In recent years, generally referred to as artificial intelligence (AI), a technology that attempts to make a computer perform human intellectual ability or higher ability has developed rapidly, agriculture, forestry and fisheries, various manufacturing industries, It is being used in all industries such as construction, information and communication, electricity, gas, water, transportation, postal, wholesale and retail, finance and insurance, various services, medical and welfare, and public affairs.

The definition of AI is not clearly defined even by experts, but deep learning (DL), which is a model of nerve cells (neurons) in the brain of living organisms, is an algorithm that deepens the hierarchy of NN. It is thought that a computing system that analyzes all kinds of data such as sales, finance, environment, voice, and images and outputs them as information and knowledge is utilized.

The basic configuration of the NN includes an input layer, a hidden layer, and an output layer. Each layer has a structure in which a plurality of nodes are connected by edges. Those with a large number of layers are called DL. In such a DL, each layer is given a function called an ``activation function'', and an ``edge'' is given a ``weight''. It is calculated from the weight value and the activation function that the layer has. At present, it has evolved into a wide variety of configurations and countless algorithms have been developed (for example, Non-Patent Document 1).

Among such DLs, a CNN (Convolutional Neural Network), which has a particularly successful track record in the field of image processing, has a hidden layer composed of a convolutional layer and a pooling layer (for example, Non-Patent Document 1). The convolutional layer filters the nodes of the previous layer to obtain a "feature map", and the pooling layer further reduces the feature map output from the convolutional layer into a new feature map, so the image's The image is abstracted by significantly compressing the amount of data in the image while maintaining its features. In other words, this abstracted image is image data in which the amount of data is reduced while maintaining the characteristics of the input image. Classification can now be processed at high speed, greatly improving the performance of image recognition. Therefore, various image processing algorithms using CNN have been developed and still play an important role.

However, for example, in the international image recognition competition called ILSVRC (ImageNet Large Scale Visual Recognition Challenge) 2012, 1.2 million annotated images were used as training data. There was a problem that a long learning time was required to analyze a huge amount of learning data just by using it.

In addition, since an image usually contains a plurality of objects to be detected, the performance required for image recognition consists of the task of classifying one object image to be detected (classification) and the task of classifying one object image thus classified. It is necessary to solve not only the task (localization) of identifying the position of an object image, but also the task (detection) of classifying and locating a plurality of detection target object images present in the image.

Therefore, the development of an image processing algorithm capable of quickly and accurately classifying and locating images of objects to be detected has been studied using CNN. The impetus for this was an image processing algorithm that successfully reduced the amount of processing image data for the object image to be detected. After that, a CNN-based network can be used to extract features and accurately identify the main object in the captured image as a bounding box, thereby performing object image classification and localization. This is an object detection algorithm called R-CNN (Region-based Convolutional Neural Network). This R-CNN greatly improves the object image detection accuracy for classifying and locating object images.

As a result, in image processing, object detection algorithms have been developed one after another for the purpose of faster and more accurate object image detection using CNN as DL. For example, Fast R-CNN, Faster R-CNN, Mask R-CNN, and R-FCN (Region-based Fully Convolutional Network), and SPP-net (Spatial Pyramid Pooling-Network), etc. can be mentioned ( For example, Non-Patent Document 2).

These are heavily influenced by R-CNN, and as described above, networks that extract region candidates for detected object images in an image and CNN-based networks that identify detected object images for region candidates. It is a two-stage object detection algorithm that runs in series. For this reason, although the accuracy was relatively high, there was a problem in terms of high speed, and improvements were made in order to maintain the accuracy.

As a result, many One-Stage object detection algorithms have been developed in which extraction of region candidates and their identification are performed by DL in one network including CNN, and are still being developed today.その代表例が、Оｖｅｒｆｅａｔ、ＤＰＭ（Ｄｅｆｏｒｍａｂｌｅ Ｐａｒｔｓ Ｍｏｄｅｌ）、ＳＳＤ（Ｓｉｎｇｌｅ Ｓｈｏｔ ＭｕｌｔｉＢｏｘ Ｄｅｔｅｃｔｏｒ）、ＤＳＳＤ（Ｄｅｃｏｎｖｏｌｕｔｉｏｎａｌ Ｓｉｎｇｌｅ Ｓｈｏｔ Ｄｅｔｅｃｔｏｒ）、ＥＳＳＤ（Ｅｘｔｅｎｄ ｔｈｅ ｓｈａｌｌｏｗ ｐａｒｔ ｏｆ Ｓｉｎｇｌｅ Ｓｈｏｔ ＭｕｌｔｉＢｏｘ Ｄｅｔｅｃｔｏｒ）、ＲｅｆｉｎｅＤｅｔ（Ｓｉｎｇｌｅ－Ｓｈｏｔ Ｒｅｆｉｎｅｍｅｎｔ Ｎｅｕｒａｌ Network for Object Detection), RetinaNet, M2Det, YOLO, and EfficientDet (for example, Non-Patent Document 2). In particular, the evolution of the one-step object detection algorithm is remarkable, and many improved algorithms, such as YOLO, have been produced with versions added to these names.

On the other hand, in both the two-step method and the one-step method, a CNN-based network that identifies object images is built in as a component (backbone), and various training image data are used in this network. The DL result is called transfer learning, and has been a major factor in increasing the accuracy and speed of classifying and identifying the position of the object image to be detected.このようなバックボーンとしての物体分類アルゴリズムには、例えば、ＡｌｅｘＮｅｔ、ＧＰｉｐｅ（Ｇｉａｎｔ Ｎｅｕｒａｌ Ｎｅｔｗｏｒｋｓ ｕｓｉｎｇ Ｐｉｐｅｌｉｎｅ Ｐａｒａｌｌｅｌｉｓｍ）、Ｉｎｃｅｐｔｉｏｎ、ＳＥＢ（Ｓｑｕｅｅｚｅ－ａｎｄ－Ｅｘｃｉｔａｔｉｏｎ Ｂｌｏｃｋ）－Ｉｎｃｅｐｔｉｏｎ、Ｘｅｐｔｉｏｎ、ＤｅｎｓｅＮｅｔ（Ｄｅｎｓｅｌｙ Ｃｏｎｎｅｃｔｅｄ Ｃｏｎｖｏｌｕｔｉｏｎａｌ Ｎｅｔｗｏｒｋ）、 ＲｅｓＮｅｔ（Ｒｅｓｉｄｕａｌ Ｎｅｔｗｏｒｋ）、ＳＥＢ－ＲｅｓＮｅｔ、Ｉｎｃｅｐｔｉｏｎ－ＲｅｓＮｅｔ、ＳＥＢ－Ｉｎｃｅｐｔｉｏｎ－ＲｅｓＮｅｔ、ＲｅｓＮｅＸｔ、ＮＡＳＮｅｔ（Ｎｅｕｒａｌ Ａｒｃｈｉｔｅｃｔｕｒｅ Ｓｅａｒｃｈ Ｎｅｔｗｏｒｋ）、ＶＧＧ（Ｖｉｓｕａｌ Ｇｅｏｍｅｔｒｙ Ｇｒｏｕｐ）、ＳＥＢ－ＶＧＧ、ＭｏｂｉｌｅＮｅｔ、ＭｎａｓＮｅｔ、ＡｍｏｅｂａＮｅｔ、ＣＳＰＮｅｔ（Ｃｒｏｓｓ Stage Partial Network), CBNet (Composite Backbone Network), Darknet, and EfficientNet. Similar to the object detection algorithm, this object classification algorithm also has versions added to these names, and many improved algorithms have been produced.

Image processing devices equipped with such advanced object detection algorithms are generally referred to as AI, and are widely used in the agriculture, forestry and fisheries industries, various manufacturing industries, construction industries, information and communication industries, electricity, gas and water industries, transportation and It is used in all industries such as postal business, wholesale/retail business, finance/insurance business, various service businesses, medical care/welfare, public affairs, etc. Possibility of making computers perform human intellectual ability or higher ability. is rising dramatically.

In particular, in the medical field, from the perspective of standardization of image data and information, image diagnosis using AI, and the development of telemedicine, the digitization of medical care is spreading endlessly (for example, Non-Patent Documents 3 and 4). . Among them, conventionally, detection of cancer from X-ray CT (Computed Tomography) images, detection of cerebral infarction from head MRI (Magnetic Resonance Imaging), detection of polyps from endoscopic images, and exposure dose Computer aided diagnosis/detection (Computer -Aided Diagnosis/Detection, CAD) system, the development of applying image processing technology using DL, which is excellent in image recognition among AI, is the most noticeable and expected. As mentioned above, advances in AI based on DL are producing great results in all industries, and the reason for this is the dramatic improvement in AI image recognition accuracy due to advances in image recognition algorithms using DL. There is

In particular, when limited to the field of dentistry, the first application of AI to an automatic image diagnosis support system and the second application of AI to a personal identification system are attracting attention.

The automatic image diagnosis support system automatically analyzes DPR such as osteoporosis, dental caries, periapical lesions, tartar, and cysts with AI so that the burden on dentists can be reduced, quickly and with high accuracy. (For example, Non-Patent Documents 5 and 6). Conventionally, in order for a dentist to create an image finding from DPR, it requires specialized knowledge and requires a long time, which is a heavy burden for the dentist and subjective images left to experience. It was an observation. AI is expected to have a high possibility of quickly and accurately detecting imaging findings that tend to be overlooked by inexperienced dentists and objective imaging findings. DPR is the most widely used image with a large amount of data, with extremely low exposure dose during imaging, and it is the center of development using it because it visualizes all the characteristics and lesions of the teeth and jawbone. . Of course, the images that can be used in the automatic image diagnosis system by AI are not limited as long as they are dental digital images, and various intraoral X-ray images, dental CBCT (Cone beam X-ray) that can obtain three-dimensional images Computed Tomograph), Cephalogram, which is a head X-ray photograph, and the like can also be used.

A personal identification system identifies a dead body in a large-scale disaster or the like (for example, Non-Patent Documents 7 and 8). Physical characteristics, fingerprints, genetic information, dental information, etc. are used for personal identification, but if the corpse is severely damaged, it is difficult to identify by physical characteristics or fingerprints, and genetic information is used to identify information from before life. The importance of dental personal identification is recognized because it is often not obtained. In addition, there is a track record of clarifying the identities of 1,250 people from dental findings in the Great East Japan Earthquake. However, although such dental personal identification is performed based on dental digital X-ray images, it requires specialized knowledge and time as with the above-mentioned image diagnosis. To do so requires a great deal of effort. Also, the mental burden of autopsy work cannot be compared with diagnosis. For this reason, there are growing expectations for AI-based personal identification systems as a method that enables rapid and accurate identification without the intervention of a dentist. is underway.

As can be seen from the development of image recognition algorithms, expectations for AI in dentistry are higher overseas than in Japan, and image recognition technology using DL has been extensively studied for dental digital X-ray image analysis. (eg, Non-Patent Documents 9-15).

A general consideration of these conventional techniques reveals that in tooth image processing, identifying each tooth, specifying the tooth number, and determining the tooth formula is an extremely difficult task.

First, as an object detection algorithm, there are many rectangle detections based on R-CNN, which has greatly improved the detection accuracy of object images for classifying and locating object images. Patent Documents 9, 11, 13, and 14 all use Faster R-CNN, and Non-Patent Document 10 uses Mask R-CNN. It is presumed that this is because the teeth are closely arranged with similar objects, so many research results of the R-CNN system, which is a two-stage object detection algorithm that emphasizes accuracy, have been reported. . In Non-Patent Document 14, the recall, specificity, and precision of Faster R-CNN exceed those of SSD, a one-step object detection algorithm, suggesting this speculation. ing.

Secondly, in Non-Patent Documents 10 and 12, although the purpose is different from object detection, an algorithm suitable for image recognition called segmentation that clearly separates the boundary of the object instead of the rectangle that encloses the object and specifies it in units of pixels. We are considering Mask R-CNN and Deeplabv3. Specifying in pixel units has the feature that the tooth size and the pixel value of the tooth can be measured more accurately, and has features not found in the above rectangle detection, but the output amount in DL is large and the calculation cost is high. There is the problem of being big.

Third, as is clear from Non-Patent Document 9 and Non-Patent Documents 11 to 14, the accuracy of the predicted tooth formula cannot be improved without some kind of correction. Although heuristic empirical methods are often used as opposed to algorithms in computer programming, Non-Patent Document 12 provides expert corrections. In addition, in Non-Patent Document 15, since it is not tooth image recognition using DL, in the final prediction of tooth expression, dynamic programming for elucidating protein and genetic information in the bioinformatics field , DP method), here a simplified Smith-Waterman algorithm, has been applied to the local alignment.

Fourth, in Non-Patent Document 11, an object detection algorithm and an object classification algorithm of the two-step method, and in Non-Patent Document 12, an algorithm suitable for segmentation and an object classification algorithm are connected in series and image processing is executed. That is.

In this way, in image processing of teeth in which similar objects are closely arranged, it is an extremely difficult task to identify each tooth, specify the tooth number, and determine the tooth formula. . Therefore, in such image processing, it is necessary to emphasize accuracy and perform heuristic correction, and an image processing apparatus capable of quickly and accurately performing image recognition using DL has not been found. In particular, there are only examples of dental image recognition studies limited to adults, and it is difficult to identify and predict more complex dentition with deciduous teeth and mixed dentition with permanent and deciduous teeth, such as in infants and children. in a situation.

TickTack World, “Easy Introduction to Machine Learning”, [online], [searched on January 12, 2021], Internet <http://gagbot.net/machine-learning> Hiromasa Fujiwara, “General Object Recognition by Deep Learning and Its Business Application,” Image Lab, January 2019, pp. 57-67 Akitoshi Katsumata, "Current status and future prospects of dental image information", Journal of Japanese Dentistry Conservation, Vol. 62, No. 5, pp. 238-242 (October 2019) Akitoshi Katsumata, “Do you know panoramic X-ray photography?”, NL Dayori, February 2018 issue, No. 482, pp. 1-2 Tatsuro Hayashi, Ryu Takahashi, Yosuke Tsuji, Hironobu Tsuji, "Osteoporosis screening using artificial intelligence technology", Journal of the Society of Medical Image Information, Vol. 36, No. 2, pp. 114-116 (2019) Medihome Co., Ltd., "(Industry's first) development of diagnostic AI for dental X-rays-diagnosis speed is about 6000 times faster than doctors-", [online], [searched January 12, 2021], Internet <HTTPS: //prtimes.jp/main/html/rd/p/000000001.000034901. html> Chisako Muramatsu, "Application of deep learning technology to dental personal identification," JCR News, No. 217, pp. 10-11, (2017) Hideyuki Takano, Yukihiro Momota, Kenji Terada, “-Learning from the Past, Preparing for the Future-Rapid Identity Confirmation Using AI and Image Analysis,” Dental Diamond, March 2020, pp. 88-93 Ｈｕ Ｃｈｅｎ，Ｋａｉｌａｉ Ｚｈａｎｇ，Ｐｅｉｊｕｎ Ｌｙｕ，Ｈｏｎｇ Ｌｉ，Ｌｕｄａｎ Ｚｈａｎｇ，Ｊｉ Ｗｕ ａｎｄ Ｃｈｉｎ－Ｈｕｉ Ｌｅｅ，“Ａ ｄｅｅｐ ｌｅａｒｎｉｎｇ ａｐｐｒｏａｃｈ ｔｏ ａｕｔｏｍａｔｉｃ ｔｅｅｔｈ ｄｅｔｅｃｔｉｏｎ ａｎｄ ｎｕｍｂｅｒｉｎｇ ｂａｓｅｄ ｏｎ ｏｂｊｅｃｔ ｄｅｔｅｃｔｉｏｎ ｉｎ ｄｅｎｔａｌ ｐｅｒｉａｐｉｃａｌ ｆｉｌｍｓ”，Ｓｃｉｅｎｔｉｆｉｃ Ｒｅｐｏｒｔｓ ｖｏｌｕｍｅ ９，Ａｒｔｉｃｌｅ number: 3840 (2019), [online], [searched January 12, 2021], Internet <https://www.nature.com/articles/s41598-019-40414-y> Ｇｉｌ Ｊａｄｅｒ，Ｊｅｆｆｅｒｓｏｎ Ｆｏｎｔｉｎｅｌｅ，Ｍａｒｃｏ Ｒｕｉｚ，Ｋａｌｙｆ Ａｂｄａｌｌａ，Ｍａｔｈｅｕｓ Ｐｉｔｈｏｎ ａｎｄ Ｌｕｃｉａｎｏ Ｏｌｉｖｅｉｒａ，“Ｄｅｅｐ Ｉｎｓｔａｎｃｅ Ｓｅｇｍｅｎｔａｔｉｏｎ ｏｆ Ｔｅｅｔｈ ｉｎ Ｐａｎｏｒａｍｉｃ Ｘ－Ｒａｙ Ｉｍａｇｅｓ”，［ｏｎｌｉｎｅ］，［２０２１年１月１２日検索］，インターネット＜ｆｉｌｅ： ///C::/Users/User/Downloads/tooth_segmentation%20(1). pdf> Dmitry V. Tuzoff, Lyudmila N.; Tuzova, Michael M.; Bornstein, Alexey S.; Krasnov, Max A.; Kharchenko, Sergey I.; Nikolenko, Mikhail M.; Sveshnikov and Georgy B. Ｂｅｄｎｅｎｋｏ，“Ｔｏｏｔｈ ｄｅｔｅｃｔｉｏｎ ａｎｄ ｎｕｍｂｅｒｉｎｇ ｉｎ ｐａｎｏｒａｍｉｃ ｒａｄｉｏｇｒａｐｈｓ ｕｓｉｎｇ ｃｏｎｖｏｌｕｔｉｏｎａｌ ｎｅｕｒａｌ ｎｅｔｗｏｒｋｓ”，Ｄｅｎｔｏｍａｘｉｌｌｏｆａｃｉａｌ Ｒａｄｉｏｌｏｇｙ Ｖｏｌｕｍｅ ４８，ＩＳＳＵＥ ４，２０１９，２０１８００５１，［ｏｎｌｉｎｅ］，［２０２１年１月１２日検索］，インターネット＜Ｈｔｔｐｓ：／／ｗｗｗ． publications. org/doi/full/10.1259/Dmfr. 20180051> Ａｎｄｒｅ Ｆｅｒｒｅｉｒａ Ｌｅｉｔｅ１，Ａｄｒｉａａｎ Ｖａｎ Ｇｅｒｖｅｎ，Ｈｏｌｇｅｒ Ｗｉｌｌｅｍｓ，Ｔｈｏｍａｓ Ｂｅｚｎｉｋ，Ｐｉｅｒｒｅ Ｌａｈｏｕｄ，Ｈｕｇｏ Ｇａｅｔａ－Ａｒａｕｊｏ，Ｍｙｒｔｈｅｌ Ｖｒａｎｃｋｘ ａｎｄ Ｒｅｉｎｈｉｌｄｅ Ｊａｃｏｂｓ，“Ａｒｔｉｆｉｃｉａｌ ｉｎｔｅｌｌｉｇｅｎｃｅ－ｄｒｉｖｅｎ ｎｏｖｅｌ ｔｏｏｌ ｆｏｒ ｔｏｏｔｈ ｄｅｔｅｃｔｉｏｎ ａｎｄ ｓｅｇｍｅｎｔａｔｉｏｎ ｏｎ ｐａｎｏｒａｍｉｃ ｒａｄｉｏｇｒａｐｈｓ”，［ｏｎｌｉｎｅ］，［ Retrieved January 12, 2021], Internet <HTTPS://omfsimpath. be/onewebmedia/Artificial% 20intelligence-driven% 20novel% 20tool% 20for% 20tooth% 20detection% 20and% 20segmentation% 20on% 20panoramic% 20radiographs. pdf> Ｆａｈａｄ Ｐａｒｖｅｚ Ｍａｈｄｉ，Ｋｏｔａ Ｍｏｔｏｋｉ ａｎｄ Ｓｙｏｊｉ Ｋｏｂａｓｈｉ，“Ｏｐｔｉｍｉｚａｔｉｏｎ ｔｅｃｈｎｉｑｕｅ ｃｏｍｂｉｎｅｄ ｗｉｔｈ ｄｅｅｐ ｌｅａｒｎｉｎｇ ｍｅｔｈｏｄ ｆｏｒ ｔｅｅｔｈ ｒｅｃｏｇｎｉｔｉｏｎ ｉｎ ｄｅｎｔａｌ ｐａｎｏｒａｍｉｃ ｒａｄｉｏｇｒａｐｈｓ”，Ｓｃｉｅｎｔｉｆｉｃ Ｒｅｐｏｒｔｓ ｖｏｌｕｍｅ １０，Ａｒｔｉｃｌｅ ｎｕｍｂｅｒ：１９２６１（２０２０），［ｏｎｌｉｎｅ］，［２０２１年１月１２ date search], Internet <https://www.nature.com/articles/s41598-020-75887-9> Changgyun Kim, Donghyun Kim, HoGul Jeong, Suk-Ja Yoon and Sekyung Youm, "Automatic Tooth Detection and Numbering Using a Combination of a CNN and Algorithmic." Sci. 2020, 10(16), 5624, [online], [searched January 12, 2021], Internet <file:///C:/Users/User/Downloads/applsci-10-05624-v2. pdf> Ａｎｎｙ Ｙｕｎｉａｒｔｉ，Ａｎｉｎｄｈｉｔａ Ｓｉｇｉｔ Ｎｕｇｒｏｈｏ，Ｂｉｌｑｉｓ Ａｍａｌｉａｈ ａｎｄ Ａｇｕｓ Ｚａｉｎａｌ Ａｒｉｆｉｎ，“Ｃｌａｓｓｉｆｉｃａｔｉｏｎ ａｎｄ Ｎｕｍｂｅｒｉｎｇ ｏｆ Ｄｅｎｔａｌ Ｒａｄｉｏｇｒａｐｈｓ ｆｏｒ ａｎ Ａｕｔｏｍａｔｅｄ Ｈｕｍａｎ Ｉｄｅｎｔｉｆｉｃａｔｉｏｎ Ｓｙｓｔｅｍ”，ＴＥＬＫＯＭＮＩＫＡ，Ｖｏｌ． 10, No. 1, March 2012, pp. 137-146, [online], [retrieved January 12, 2021], Internet <file:///C:/Users/Owner/Downloads/Classification_and_Numbering_of_Dental_Radiographs. pdf>

In the image recognition of teeth , which are similar objects that exist densely, it is possible to specify the position of each object image and identify each object image quickly with the DL image recognition algorithm developed so far. And it is extremely difficult to do it accurately. In particular, when identifying and predicting the tooth formula, in addition to the two-stage object detection algorithm that prioritizes accuracy over speed, after applying the object classification algorithm, the tooth number is corrected based on heuristic empirical methods. There is a need to. Moreover, although there are studies on the identification of adult teeth, it is difficult to identify and predict more complex tooth patterns with deciduous teeth, such as infants and children, and mixed dentition with permanent and deciduous teeth. be.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an image processing apparatus and an image processing method capable of rapidly and accurately recognizing tooth images. In particular, it is an object of the present invention to provide an image processing apparatus and an image processing method capable of quickly and accurately identifying dental formulas of not only adults but also infants and children.

Therefore, the present inventors connect YOLOv3, a one-stage object detection algorithm that emphasizes speed, and EfficientNet, an existing object classification algorithm, in series, identify the position of each tooth rectangle, and Rapidly and accurately identify complex dentitions with deciduous and mixed dentitions with permanent and deciduous teeth by DP alignment correction for dental inconsistent arrangements after identification of teeth I found that it can be done. Based on this knowledge, as a result of various further studies, it was found that there is no need to be limited to such an object detection algorithm and an object classification algorithm, and similar results can be obtained by adding an appropriate DP alignment correction to the connection of both algorithms. The present invention has been completed.

That is, the present invention includes an input unit capable of inputting object image data, and an object feature extraction unit for extracting object feature amounts from an existing object image data set executed by an object classification algorithm having at least a CNN as a module. as a backbone, learns the input first teacher image data, the learning image data set, and the extended image data of this learning image data set, and learns the learning model of the object feature extraction unit A first learning model can be created by performing transfer learning and fine-tuning using an object image locator capable of identifying the position of the first rectangle obtained by the object classification algorithm, the data of the first rectangle and/or the global data of the first rectangle; Second teacher image data different from the first teacher image data, the learning image data set, and extended image data of the learning image data set are learned, transfer learning using a learning model of the object feature extraction unit, and An object image capable of fine-tuning to create a second learning model, and adding a unique information tag to the first rectangle specified by the object image placement unit to classify and identify the detection target object image. An identification unit, an object image correction unit capable of correcting a result output from the object image identification unit, and an output unit capable of outputting the processing result of the object image to be detected. It is an image processing device characterized by:

Here, the object classification algorithm is not particularly limited, but it is fast and accurate, AlexNet, GPipe (Giant Neural Networks using Pipeline Parallelism), Inception, SEB (Squeeze-and-Excitation Block) －Ｉｎｃｅｐｔｉｏｎ、Ｘｅｐｔｉｏｎ、ＤｅｎｓｅＮｅｔ（Ｄｅｎｓｅｌｙ Ｃｏｎｎｅｃｔｅｄ Ｃｏｎｖｏｌｕｔｉｏｎａｌ Ｎｅｔｗｏｒｋ）、ＲｅｓＮｅｔ（Ｒｅｓｉｄｕａｌ Ｎｅｔｗｏｒｋ）、ＳＥＢ－ＲｅｓＮｅｔ、Ｉｎｃｅｐｔｉｏｎ－ＲｅｓＮｅｔ、ＳＥＢ－Ｉｎｃｅｐｔｉｏｎ－ＲｅｓＮｅｔ、ＲｅｓＮｅＸｔ、ＮＡＳＮｅｔ（Ｎｅｕｒａｌ Ａｒｃｈｉｔｅｃｔｕｒｅ Ｓｅａｒｃｈ Ｎｅｔｗｏｒｋ）、ＶＧＧ（Ｖｉｓｕａｌ Ｇｅｏｍｅｔｒｙ Ｇｒｏｕｐ）、 It is preferable to use at least one or more selected from SEB-VGG, MobileNet, MnasNet, AmoebaNet, CSPNet (Cross Stage Partial Network), CBNet (Composite Backbone Network), Darknet, EfficientNet, and NFNet. As mentioned above, the object classification algorithm has a number of versions attached to these names, and many improved algorithms have been produced. The same is true for all object classification algorithms that

In particular, the object classification algorithms include SEB, RB (Residual Block), DConv (Depthwise Convolution Layer), PConv (Pointwise Convolution Layer), MixConv (Mixed Depthwise Convolution Layer), and GAP (Globalization). An algorithm that includes at least one or more selected modules and/or blocks is preferable because it can improve accuracy, and is not particularly limited as long as it is an object classification algorithm that includes these modules and/or blocks. Examples include ResNet, ResNeXt, MobileNet, MnasNet, Darknet, and EfficientNet.

The object image correction unit can be executed by an algorithm called dynamic programming (DP method) for elucidating protein and gene information in the bioinformatics field, which is opposed to the heuristic empirical method. , it is possible to perform quick and high-speed correction, which is preferable. Since this DP method cannot determine the overall sequence, but can be applied to local alignments when determining locally similar alignments, and to global alignments when determining the overall sequence, It is necessary to use properly according to the purpose. In particular, when identifying the dental formula, it is difficult to correct the entire dental formula when applied to the local local alignment, and when applied to the overall global alignment, the presence or absence of wisdom teeth and the presence of missing teeth continuing from the wisdom teeth. It is more preferable to apply to semi-global alignment because correction for .

On the other hand, as the object detection algorithm, either a two-step object detection algorithm or a one-step object detection algorithm can be applied.

In particular, two-stage object detection algorithms include R-CNN (Region-based Convolutional Neural Network), Fast R-CNN, Faster R-CNN, Mask R-CNN, and R-FCN (Region-based Fully Convolutional Network) is preferably adopted.

また、一段階法の物体検出アルゴリズムとしては、Оｖｅｒｆｅａｔ、ＤＰＭ（Ｄｅｆｏｒｍａｂｌｅ Ｐａｒｔｓ Ｍｏｄｅｌ）、ＳＳＤ（Ｓｉｎｇｌｅ Ｓｈｏｔ ＭｕｌｔｉＢｏｘ Ｄｅｔｅｃｔｏｒ）、ＤＳＳＤ（Ｄｅｃｏｎｖｏｌｕｔｉｏｎａｌ Ｓｉｎｇｌｅ Ｓｈｏｔ Ｄｅｔｅｃｔｏｒ）、ＥＳＳＤ（Ｅｘｔｅｎｄ ｔｈｅ ｓｈａｌｌｏｗ ｐａｒｔ ｏｆ Ｓｉｎｇｌｅ Ｓｈｏｔ ＭｕｌｔｉＢｏｘ Ｄｅｔｅｃｔｏｒ）、 It is preferable to adopt at least one or more selected from RefineDet (Single-Shot Refinement Neural Network for Object Detection), RetinaNet, M2Det, YOLO, Scaled YOLO, and EfficientDet. As already mentioned, this object detection algorithm also has versions attached to these names, and many improved algorithms have been produced, but the object detection algorithm includes all of these, The same is true for all of the object detection algorithms described below.

It is generally believed that the two-step method is superior in accuracy and the one-step method is superior in speed. We found that the combination of object detection algorithm and object classification algorithm running the image processor is the key factor. As a result, the object detection algorithm is at least one selected from M2Det, YOLO, and EfficientDet, and the object classification algorithm is selected from ResNet, ResNeXt, MobileNet, MnasNet, Darknet, and EfficientNet It was found that it is preferable to select at least one or more. Furthermore, in order to perform image recognition more quickly and accurately, the object detection algorithm of YOLO and EfficientDet after YOLOv3, ResNet after ResNet-101, MobileNet after MobileNet V3, and Darknet object after Darknet-53 A combination with a classification algorithm is preferred.

The image processing apparatus of the present invention has been described above. The image processing method of the present invention is effective in tooth image processing, and is capable not only of quickly and accurately specifying the dental formula of adults, but also of infants and children. It is characterized by being able to quickly and accurately identify even more complex dental formulas such as those in which primary teeth and permanent teeth are mixed.

First, the first image processing method of the present invention learns input first teacher image data, a learning image data set, and extension data of this learning image data set, and learns a learning model of an object feature extraction unit. Using the first learning model learned by performing transfer learning and fine tuning using the input detection target image, a first rectangular information tag containing the image of each detection target object on the input detection target image, and this information specifying the position of the first rectangle to which the tag is added; and the first rectangle data and/or the first rectangle wide-area data, the input first teacher image data different from the second The second acquired by learning the teacher image data, the learning image data set, and the extension data of the learning image data set, and performing transfer learning and fine tuning using the learning model of the object feature extraction unit Using the learning model, a process of adding a unique information tag to the first rectangle whose position is specified, classifying and identifying the detection target object image, and identifying the detection target object image whose identification result violates the laws of nature. It is characterized by passing through a step of correcting and a step of outputting the processing result of the object image to be detected.

The object placement unit of the image processing apparatus of the present invention is a two-step method or a one-step object detection method incorporating an object classification algorithm with a built-in object feature extraction unit that extracts the feature amount of an object from a known learning data set as a backbone. Since it is executed by the algorithm, it is possible to position and identify the object image, but the information tag annotated in the first teacher image data is limited, and the position and position of the rectangle containing each object image to be detected. It is characterized by its specialization in accurately specifying identification.

Different from the first teacher image data, the second teacher image data contains information in which each detection object can be identified based on the position of the rectangle containing each detection target object image accurately specified in this way. Information tags are annotated, and using an object classification algorithm specialized for classifying objects, a unique information tag is added to each to accurately identify the position of the detection target object image, and the type of the detection target object image is specified. correctly identified.

However, in recognizing images of teeth that are densely present, due to overlapping of object images, predictions that go against the laws of nature, such as duplicate predictions that adjacent different objects are in the same arrangement, are performed. may occur. Therefore, it is effective to go through the process of correcting the object to be detected as the final inspection. As a method of correction, it is preferable to use a DP alignment algorithm.

As a specific example, when such an image processing method is used to identify a dental formula, it can be carried out as follows.

The first training image data consists of training image data, verification data, and test data annotated by dental radiologists using dental X-ray digital images taken at many medical institutions. is executed by a two-step or one-step object detection algorithm with built-in object feature extractor as a backbone that extracts the feature quantity of an object from an existing object image data set, which is executed by an object classification algorithm equipped with CNN as a module. is input to the object image placement unit. Here, the information tag defined in the annotation distinguishes between, for example, the maxillary tooth and the mandibular tooth, the crown portion, the root portion, the vicinity of the boundary between the crown portion and the root portion, and the entire crown portion and the root portion. Rectangle data defined and set as rectangles of common portions of individual teeth are input as first teacher image data. On the other hand, the large number of dental X-ray digital images are subjected to so-called augmentation considering aspect ratio fluctuation, resolution scaling, cropping, translation, rotation, left-right reversal, random erasure, random noise addition, and brightness. , an image with data extension is created and input to the object image placement unit.

Then, learning is performed using these image data, and transfer learning is performed using the feature values extracted by learning in the object feature extraction unit, and the first learning model is created as a result of fine tuning. Then, using this learning model, each information tag of the first rectangle that includes each tooth, which is the detection target object on the input detection target image, and each detection target object to which the information tag is added are obtained. A first rectangle containing the teeth of is located.

Next, the learning image set input to the object image placement unit and the data-extended images thereof are the same, but second teacher image data different from the first teacher image data are created, and these are used as object image data. Input to the object image identification section which is executed by the classification algorithm. Here, the second training image data are individual teeth that are objects to be detected, that is, image data for detecting the target teeth, and annotated training image data for classifying and identifying the target teeth. Use Examples of the former include image data of the target tooth, image data showing the relative positions of the target tooth and other teeth, wide-area images centering on the target tooth, and gradient and angle images of the target tooth. can. In particular, a wide-area image including at least adjacent teeth centered on the target tooth is preferably used. For the latter, the classification of maxillary and mandibular teeth, the classification of right and left teeth, the classification of permanent and deciduous teeth, the classification of tooth types (incisors, canines, premolars, molars, etc.), wisdom teeth and non-wisdom teeth Use a teacher signal for the classification of As a result of learning the above teacher signal, a second learning model is created, and the tooth formula of the tooth to be detected is inferred using this learning model. In creating this learning model, both single-task learning and multi-task learning can be used, but multi-task learning is preferred in the present invention. This is because it is generally preferable to use multi-task learning in terms of speed during reasoning, but without the worrying degradation of accuracy for similar objects such as teeth. Also, in inference, a general inference module can be used, and OpenVINO (registered trademark) can be used.

In the tooth formula inferred in this way, similar teeth are densely present and the relationship between adjacent teeth is not taken into consideration, so an inferred result of an anatomically impossible arrangement may be obtained. In this case, a step of correcting with an object image corrector executed with the DP alignment algorithm is obtained. In particular, the characteristic of the process of correcting the tooth formula executed by this DP alignment algorithm is that the correction of the tooth formula is processed in the step of classifying and identifying the tooth, which is the object to be detected, and the result inferred from the DP alignment. It is that it can be performed with high accuracy by applying it to the algorithm. However, of course, an erroneous inference result is not necessarily obtained, so this step is included in the image processing method of the present invention, but it does not necessarily mean that it is necessary to go through this step. .

Furthermore , in order to perform tooth image recognition more accurately, after further specifying the rectangle that contains the set of teeth , after specifying the position of each rectangle to which the information tag is added, each tooth We have found that image processing methods that classify and identify are more preferable.

That is, a more preferable image processing method of the present invention is to process third teacher image data different from the input first and second teacher image data, a learning image data set, and extension data of this learning image data set. Using a third learning model acquired by performing transfer learning and fine tuning using a learning model of the object feature extraction unit while learning, a third learning model that includes all detection target object images on the input detection target image a step of specifying the positions of two rectangles, the second rectangle data and/or the second rectangle wide area data, the input first teacher image data, the learning image data set, and the learning image data A set of extended data is learned, and using a fourth learning model acquired by performing transfer learning and fine tuning using the learning model of the object feature extraction unit, each detection target object on the input detection target image is used. identifying a third rectangular information tag enhanced with the first rectangle containing the image of; identifying the position of the third rectangular to which the information tag is added; data of the third rectangle and/or the Learning the third rectangular wide area data, the input second teacher image data, the learning image data set, and the extension data of the learning image data set, and transferring using the learning model of the object feature extraction unit Using the fifth learning model acquired through learning and fine-tuning, adding a unique information tag to the third rectangle, classifying and identifying the detection target object image, and correcting the detection target object image. and a step of outputting the processing result of the object image to be detected.

In this image processing method, the second rectangle is a rectangle containing a set of teeth . This is not only because he found it important to grasp the relative positional relationship between them.

In particular, in specifying the dental formula, the second rectangle enclosing this set of teeth is a rectangle enclosing all of the individual teeth. A third teacher image data was applied, in which a rectangle containing all teeth was set as a teacher image. Except for the addition of this step, it can be performed in the same manner as in the first image processing method, but it was recognized that this step improves the accuracy of specifying the tooth formula.

Such an image processing apparatus and image processing method of the present invention are particularly suitable for image recognition of teeth , which are objects to be detected in which similar objects are densely packed, and are not limited in any way as long as they are digital images. Dental digital X-ray images, such as intraoral X-ray images, DPR, CBCT, cephalograms, etc., can also be used in identifying the dental formula given as a specific example. However, DPR is most preferable because it has the lowest radiation dose, is the most widely used and has the most data, and renders all the major features and lesions occurring in the teeth and jawbone.

According to the present invention, image processing is performed by serially connecting an object detection algorithm and an object classification algorithm of a two-step method or a one-step method, and recognition of tooth images can be performed quickly and accurately. In addition, since the object image correction of the present invention employs the DP alignment algorithm, it is possible to logically and quickly resolve contradictions in prediction that go against the law of nature and tend to occur in densely existing tooth images. became. Therefore, the image processing apparatus and image processing method of the present invention can perform the arrangement, classification, and identification of very complicated similar objects that are close to each other, which has been difficult with conventional techniques. In particular, it was possible to improve the prediction speed in identifying the dental formula of adults, and it is also possible to identify the dental formula of infants and children with primary teeth, which has been difficult and has never been reported. rice field.

FIG. 1 shows an image processing apparatus in which an input unit, an object image placement unit, an object image identification unit, an object image correction unit, and an output unit are connected in series according to an embodiment of the present invention. YOLOv3, a one-step object detection algorithm; the object image identification unit, an object classification algorithm; EfficientNet; and the object image correction unit, a DP alignment algorithm. It is a figure which shows the outline|summary of a processing apparatus. FIG. 2 shows an image processing apparatus in which an input unit, an object image placement unit, an object image identification unit, an object image correction unit, and an output unit are connected in series, according to an embodiment of the present invention. the object image identification unit using the one-step object detection algorithm EfficientDet, the object image identification unit using the object classification algorithm EfficientNet, and the object image correction unit using the DP alignment algorithm. It is a figure which shows the outline|summary of a processing apparatus. In the tooth image processing method using the first image processing device according to one embodiment of the present invention, FIG. is identified, an information tag is added to the rectangle, and an outline is shown up to the process of identifying the position. In the tooth image processing method using the first image processing apparatus according to one embodiment of the present invention, FIG. 10 is a diagram showing an outline of a process of outputting a detected object image, which is an image processing result, through a process of correcting an erroneous identified image that is input to an object image correcting unit and is in violation of the law of nature; FIG. In the tooth DPR image processing method using the first image processing device according to one embodiment of the present invention, FIG. FIG. 10 is a diagram showing an outline of a process of specifying a rectangle to be processed, adding an information tag to each rectangle, and specifying a position; In the tooth DPR image processing method using the first image processing apparatus according to one embodiment of the present invention, FIG. From the process of classifying rectangular images and adding tooth numbers to generate tooth formulas, through the process of correcting erroneous tooth formula images that violate natural laws input to the object image corrector, the tooth formulas that are the results of image processing are generated. FIG. 4 is a diagram showing an outline of a process of outputting an image; In the tooth DPR image processing method using the first imaging device according to one embodiment of the present invention, FIG. It is a figure which shows the outline|summary to the process of identifying a rectangle. In the tooth DPR image processing method using the first imaging device according to one embodiment of the present invention, FIG. FIG. 10 is a diagram showing an outline of a process of identifying rectangles of individual teeth in a rectangular image including teeth of teeth, adding an information tag to each rectangle, and identifying a position; In the tooth DPR image processing method using the first image processing apparatus according to one embodiment of the present invention, FIG. From the process of classifying rectangular images and adding tooth numbers to generate tooth formulas, through the process of correcting erroneous tooth formula images that violate natural laws input to the object image corrector, the tooth formulas that are the results of image processing are generated. FIG. 4 is a diagram showing an outline of a process of outputting an image;

Regarding the image processing apparatus and the image processing method using the same of the present invention, an embodiment will be described in detail, mainly assuming that it is used for identifying a dental formula using a DPR digital image. The processing device and the image processing method using it are not limited to this. Also, the configuration of the image processing apparatus and the steps of the image processing method of the present invention are not limited to this, and various changes can be made without departing from the spirit of the present invention. It is limited only by the technical ideas described in the scope of.

FIG. 1 shows an input unit 1100, an object image placement unit 1200, an object image identification unit 1300, an object image correction unit 1400, and an output unit 1500 connected in series according to an embodiment of the present invention. 1200 is a one-step object detection algorithm YOLOv3 (1210), an object image identification unit 1300 is an object classification algorithm EfficientNet 1310, and an object image correction unit 1400 is characterized by using a DP alignment algorithm 1410. 1 shows an outline of a first image processing apparatus 1000 having

In FIG. 2, as in FIG. 1, an input unit 2100, an object image placement unit 2200, an object image identification unit 2300, an object image correction unit 2400, and an output unit 2500 are connected in series. The second image processing apparatus 2000 is characterized by using EfficientDet 2210, which is a one-step method object detection algorithm, and an object image correction unit 2400 executing a DP alignment algorithm 2410 for semi-global alignment. This second image processing device 2000 uses EfficientDet 2210 as a one-step object detection algorithm in the object image placement unit 2200 to increase speed and accuracy, and uses a semi-global alignment method 2410 to increase accuracy. It is preferable to the first image processing apparatus 1000 because it can This semi-global alignment method 2410 is also effective in applying to the first image processing apparatus 1000 and increasing the accuracy of the first image processing apparatus 1000 .

Additionally, non-limiting one-step object detection algorithms and object classification algorithms can be used. In particular, the object classification algorithm is provided with at least one or more modules and/or blocks selected from SEB, RB, DConv, PConv, MixConv, and GAP, especially for high accuracy preferable.

Figures 3 and 4 outline a method for image processing teeth using a first image processing apparatus 1000, according to an embodiment of the present invention. FIG. 3 shows an overview of the process of identifying a rectangle containing each object from the image data 1110 input from the input unit 1100 to the object image placement unit 1200, adding an information tag to the rectangle, and identifying the position. FIG. 4 is a diagram showing; Further, FIG. 4 shows the process of classifying and identifying the rectangular image of the object input to the object image identification unit 1300 from the image processing step of FIG. FIG. 15 is a diagram showing an outline of a process of outputting a detected object image, which is an image processing result 1510, through a process of correcting the image;

In this way, the invention of the image processing apparatus and the image processing method using the image processing apparatus of the present invention was made possible by the dramatic improvement in AI image recognition accuracy due to advances in image recognition algorithms using DL. It was created as a result of the active development of applying AI image processing technology to CAD systems against the backdrop of the fact that AI image processing technology is producing great results in all industries. (Non-Patent Documents 3-14). That is, the present invention finds an image processing apparatus and an image processing method with sufficient speed and accuracy because it is difficult to quickly and accurately identify the dental model only with the existing object detection algorithm using DL. It starts with not being.

Therefore, the present invention will be described more specifically using examples used for identification of tooth formulas. First, FIGS. 5 and 6 show an example of applying the image processing method shown in FIGS. 3 and 4 to identify a dental model.

Regarding the image processing method for tooth DPR using the first image processing apparatus 1000 according to an embodiment of the present invention, FIG. FIG. 10 is a diagram showing an outline of a process of identifying rectangles containing teeth of a tooth, adding an information tag to each rectangle, and identifying a position; FIG. 6 shows the process of classifying rectangular images of individual teeth input to the object image identification unit 1300 from the image processing step of FIG. 15 is a diagram showing an outline of a process of outputting a tooth formula image as an image processing result 1511 through a process of correcting an erroneous tooth formula image that violates the law of nature.

More specifically, in FIG. 5, the learning image data 1111 identifies a first rectangle containing each tooth to be detected, adds an information tag to the first rectangle, and identifies the position of the first rectangle. In order to identify it, the following steps are taken. Defined first teacher image data 1132, first annotated learning image data 1142 for learning in the object image placement unit, and augmented learning image data augmented extended image data 1151 as object images It is input to the placement unit 1200 . Here, as the first teacher image data 1132, the tooth crown and tooth root that distinguish the upper and lower teeth of individual teeth common to all teeth, the boundary between the tooth crown and the tooth root, and , It is preferable to use the entire tooth crown and root. Augmentation includes aspect ratio fluctuation, resolution scaling, cropping, translation, rotation, left-right inversion, random erasure, random noise addition, and brightness etc. is preferable. In addition, these image data are executed and learned by YOLOv3 (1210), a one-step object detection algorithm, and Darknet-53 (object feature extraction unit) 1211, an object classification algorithm built into YOLOv3 (1210). Transfer learning and fine tuning are performed using a learning model generated from a huge amount of image data, and as a result a first learning model 1231 is generated. On the other hand, the detection image data 1121 input to the object image arrangement unit 1200 undergoes feature quantity extraction 1213 in YOLOv3 (1210), and feature quantity extraction data 1241 is generated. Then, an inference program makes an inference from the first learning model 1231 and the feature quantity extraction data 1241, identifies a first rectangle containing each tooth, adds an information tag to each rectangle, and determines the position. identified.

FIG. 6 shows in detail the process of classifying, tooth numbering and identifying the first rectangular teeth containing individual teeth, which is the image data generated in this way. Unlike the first teacher image data, the image data for detecting the target tooth and the annotated second teacher image data 1133 for classifying and identifying the target tooth are used. Examples of the former include image data of the target tooth, image data showing the relative positions of the target tooth and other teeth, wide-area images centering on the target tooth, and gradient and angle images of the target tooth. can. In particular, a wide-area image including at least adjacent teeth centering on the target tooth is preferably used. Specifically, when the length of the long axis of the target tooth is L, and a rectangle having a length of L is set vertically from the center of the target tooth, L is preferably 1 to 3, and 1.5 to 2.5. 5 is more preferred. For the latter, the classification of maxillary and mandibular teeth, the classification of right and left teeth, the classification of permanent and deciduous teeth, the classification of tooth types (incisors, canines, premolars, molars, etc.), wisdom teeth and non-wisdom teeth Use image data that can be classified into Also, since the teacher image data is changed, the second annotated learning image data 1143 different from the first annotated image data shown in FIG. 5 and the aspect ratio fluctuation, resolution scaling, and cropping of the learning image data , translation, rotation, left-right inversion, random erasure, random noise addition, and augmented image data 1151 such as brightness are used. The second annotated training image data 1143 may be the first annotated image data 1131 . Such image data is run through the object classification algorithm EfficientNet 1310 to generate a second learned model 1321 . On the other hand, the detected image data 1121 generates the feature amount extraction data 1331 by the feature amount extraction 1312 of the EfficientNet 1310 . Then, an inference program makes an inference from the second learning model 1321 and the feature amount extraction data 1331, and classifies and identifies the first rectangle.

However, since this inference result may infer anatomically impossible tooth number duplication, in the object image correction unit 1400, it is corrected by the DP alignment algorithm 1411 and the first image processing device is used. A tooth formula image to be detected is generated by the output unit 1500 as an image processing result 1511 by the tooth image processing method.

Furthermore, as a result of examination of image processing methods using teeth, an image processing method capable of increasing accuracy was found. Therefore, a more detailed description will be given using FIGS. 7 to 9, including evaluation results. This image processing method is characterized by the addition of a step of specifying the position of a second rectangle containing all the teeth, which is the detection target object image on the input detection target image. Such a second rectangle is a rectangle that contains a set of teeth, which are densely similar objects. This is because it is important to grasp not only the shape of each tooth but also their relative positional relationship in order to identify the tooth. This is because the second rectangle containing all the teeth can be used as a relative reference position for each tooth.

FIG. 7 is a diagram showing an overview up to a process of identifying a rectangle containing all teeth from image data input from the input unit 1100 to the object image placement unit 1200. FIG. Therefore, 1,000 DPR cases photographed at many medical institutions were used as learning image data. The training image data includes third training image data 1134 defined by a dental radiologist for generating a second rectangle containing all teeth, first training image data 1134 defined by a dental radiologist. of annotation image data 1142 . The former is a rectangle that encloses all of the individual teeth, and representative examples of the latter are the crown and root portions of individual teeth that are common to all teeth, distinguishing the upper and lower teeth, and , line of alveolar bone, and the like. On the other hand, from images of 1,000 DPR cases, extended image data by fluctuation of aspect ratio, resolution scaling, cropping, translation, rotation, left-right inversion, random elimination, addition of random noise, and augmentation such as brightness (shading) 1151 was created. These learning data are input to the object image placement unit 1200, and 720 cases for training, 80 cases for verification, and 200 cases for testing are used to perform 5-fold cross-validation to obtain a one-step object detection algorithm. It was run and trained on YOLOv3 (1210).

At the same time, using these learning data, the object classification algorithm Darknet-53 (1211) built in YOLOv3 (1210) performs transfer learning and fine tuning using a learning model generated from a huge amount of image data, As a result, a third learning model 1232 was generated.

On the other hand, the detection image data 1121 input to the object image arrangement unit 1200 undergoes feature quantity extraction 1213 in YOLOv3 (1210), and feature quantity extraction data 1241 is generated. Then, the third learning model 1232 and the feature amount extraction data 1241 are inferred by the inference program, the second rectangles containing all the teeth are specified, information tags are added to the respective rectangles, and the positions are determined. identified. As a result, the detection performance of a rectangle containing all teeth was evaluated by AP (Average Precision) calculated from precision and recall, which are generally used as indicators of precision in object detection. IоU (Intersection over Union), which is called the overlap rate (Jaccard coefficient) for calculating the ratio, is 1.000 and 0.997 at 50% and 75%, respectively, and GT (Ground Truth , accuracy rate) was 75% or more, which was a good result.

FIG. 8 shows an image processing step executed in the object image placement unit 1200 following FIG. It specifies three rectangles, adds an information tag to each rectangle, and shows an overview up to the process of specifying the position.

What is important here is that, although not shown, the second rectangle containing all the teeth is magnified 1.1 to 1.5 times horizontally and 1.1 to 1.5 times vertically from the center. It is used as a wide-area image with a magnification of 1.8. This is to avoid excluding wisdom teeth, deciduous teeth and apexes from the second rectangle.

Again, 1,000 DPR cases were used as training image data. As teacher image data from this learning image data, the first teacher data 1132 shown in FIG. 5, that is, all The crown part that separates the maxillary and mandibular teeth, the root part that separates the maxillary and mandibular teeth, the boundary between the crown and the root, and the entire crown and root At least one or more was used from The first annotation image data defined by the dental radiologist was used as training data 1142, and the augmentation data 1151 was also used, as was the case for identifying a second rectangle containing all teeth. Then, these learning data are 720 cases for training, 80 cases for verification, and 200 cases for testing, and 5-fold cross-validation is performed, and the one-step object detection algorithm YOLOv3 (1210) is executed and learned. At the same time, Darknet-53 (1211), an object classification algorithm built into YOLOv3 (1210), performs transfer learning and fine tuning using a learning model generated from a huge amount of image data, and as a result, the fourth learning A model 1233 was generated.

7, the detection image data 1121 input to the object image placement unit 1200 is subjected to feature extraction 1213 in YOLOv3 (1210), feature extraction data 1241 is generated, and the fourth learning model An inference program inferred from 1233 and feature amount extraction data 1241 to identify a third rectangle containing each tooth, and an information tag was added to each rectangle to identify the position. The results were evaluated using a general object detection index called MAP (Mean Average Precision), which is the average AP of the maxillary and mandibular teeth. And the average values of MAP for the entire crown and root are 0.727, 0.699, 0.682, and 0.777, respectively, and 97% or more of the target teeth can be detected. Good results were obtained.

Next, FIG. 9 shows the step of classifying the third rectangular image containing each tooth input to the object image identification unit 1300 from the image processing step of FIG. 15 is a diagram showing an outline of a process of outputting a tooth formula image, which is an image processing result 1512, through a process of correcting an anatomically incorrect tooth formula image that violates the law of nature and is input to an object image correction unit 1400. FIG.

Here, as in FIG. 6, unlike the first teacher image data, the image data for detecting the target tooth and the annotated second teacher image data for classifying and identifying the target tooth 1133 is used. Examples of the former include image data of the target tooth, image data showing the relative positions of the target tooth and other teeth, wide-area images centering on the target tooth, and gradient and angle images of the target tooth. Although possible, wide area images are particularly preferred. Specifically, when the length of the long axis of the target tooth is L, and a rectangle having a length of L is set vertically from the center of the target tooth, L is preferably 1 to 3, and 1.5 to 2.5. 5 is more preferred. The latter includes the classification of maxillary and mandibular teeth, the classification of right and left teeth, the classification of permanent and deciduous teeth, the classification of tooth types (incisors, canines, premolars, molars, etc.), and the classification of wisdom teeth and non-wisdom teeth. Use image data that can be used. Also, since the teacher image data was changed, the second annotated learning image data 1143 and the augmented extended image data 1151 different from the first annotated image data were used, as in FIG. The second annotated training image data 1143 may be the first annotated image data 1142 . Such image data was run through the object classification algorithm EfficientNet 1310 to generate a fifth learned model 1311 . For the detected image data 1121, feature amount extraction data 1331 is generated by the feature amount extraction 1312 of the EfficientNet 1310 as in FIG. Then, an inference program was used to make an inference from the fifth learning model 1311 and the feature amount extraction data 1331, and the third rectangle was classified and identified.

Here, OpenVINO (registered trademark) was used as an inference program, and as a result of classifying multi-task processing with a single model, results comparable to the results of classifying single-task processing with multiple models were obtained. For speed considerations, it is preferable to use multitasking. It is believed that this is because the teeth have mutually similar shapes.

As a result, in the general object detection index of tooth number, maxillary and mandibular teeth, right and left teeth, permanent and deciduous teeth, tooth types (incisors, canines, premolars, molars), wisdom teeth and non-wisdom teeth Certain precision, recall, and F value (harmonic mean of precision and recall), as shown in Table 1, were very good results. It should be noted that the accuracy rate for tooth numbering was 97%, while the accuracy for other tasks exceeded 99%. is. This identification is probably the first in the world.

However, as can be seen from the matching rate of the tooth number, this inference result may infer an anatomically impossible duplication of the tooth number. For example, 26, that is, two double upper left first molars were detected in the tooth formula based on the FDI (Federation Dentaire International) system (Two-digit system). Therefore, in applying the DP alignment algorithm 1412 to the semi-global alignment in the object image correction unit 1400, the identification results other than the tooth numbers in Table 1, for example, the results of the upper and lower teeth and the permanent teeth and deciduous teeth are used for correction. As a result, correct correction was performed, and a dental formula image to be detected was generated as an image processing result 1512 by the output unit.

As described above, according to the image processing method using the image processing apparatus of the present invention, it is possible to quickly and accurately generate a dental formula, and it is possible to detect deciduous teeth from the DPR of infants and children in which permanent teeth and deciduous teeth are mixed. became clear.

The image processing apparatus of the present invention and the image processing method using it are suitable for recognizing images of teeth in which similar objects are densely present. Various trees growing together, shapes piled up in a warehouse, many cardboard boxes of similar materials, schools of fish in the sea, various aerial photographs, and images in which various similar objects are densely present , such as spectators at stadiums. Industrial applicability is extremely high in that it is thought that there is a possibility that it can be used for identification of

1000 first image processing device 1100 input unit 1110 learning image data 1111 tooth learning image data 1120 detection image data 1121 tooth image learning data 1130 first teacher image data 1131 second teacher image data 1132 tooth first Teacher data 1133 Second teacher image data of teeth 1134 Third teacher image data of teeth 1140 First annotated image data 1141 Second annotated image data 1142 First annotated image data of teeth 1143 Teeth 1150 Augmented Augmented Data 1151 Augmented Augmented Data for Teeth 1200 Object Image Placer 1210 One Step Object Detection Algorithm (YOLOv3)
1211 Built-in object classification algorithm (Darknet-53) as the backbone of YOLOv3
1212 Learning by YOLOv3 1213 Feature extraction by YOLOv3 1214 Inference by YOLOv3
1220 Feature quantity extraction data by Darknet-53 1230 First learning model by YOLOv3 1231 First learning model of teeth by YOLOv3 1232 Third learning model of teeth by YOLOv3 1233 Fourth learning model of teeth by YOLOv3 1240 By YOLOv3 Feature quantity extraction data 1241 Tooth feature quantity extraction data by YOLOv3 1250 Addition of information tag to object image by YOLOv3 and identification of position 1251 Addition of first rectangular information tag containing each tooth by YOLOv3 and identification of position 1252 Identification of the second rectangle containing all teeth by YOLOv3 1253 Addition and location of information tag of the third rectangle containing each tooth by YOLOv3 1300 Object identification unit 1310 Object classification algorithm (EfficientNet)
1311 Learning by EfficientNet 1312 Feature extraction by EfficientNet 1313 Inference by EfficientNet 1320 Second learning model by EfficientNet 1321 Second learning model of teeth by EfficientNet 1322 Fifth learning model of teeth by EfficientNet 1330 Feature extraction data by EfficientNet 1331 Tooth Feature Amount Extraction Data by EfficientNet 1340 Object Classification and Identification by EfficientNet 1341 First Rectangle Classification and Identification by EfficientNet 1342 Third Rectangle Classification and Identification by EfficientNet 1400 Object Image Corrector 1410 Object DP Alignment Algorithm 1411 One Rectangle DP Alignment Algorithm 1412 Third Rectangle DP Alignment Algorithm (Semi-Global Alignment)
1500 Output unit 1510 Object image processing result by the first image processing device 1511 Tooth image processing result by the first image processing device (identification of tooth formula)
1512 Tooth image processing result using the second rectangle containing all teeth by the first image processor (identification of tooth formula)
2000 second image processing device 2100 input unit 2200 object image placement unit 2210 one-step object detection algorithm (EfficientDet)
Built-in object classification algorithm (EfficientNet) as the backbone of 2211 EfficientDet
2300 Object Identification Unit 2310 Object Classification Algorithm (EfficientNet)
2400 object image correction unit 2410 DP alignment algorithm (semi-global alignment)
2500 output unit

Claims (15)

An image processing device for identifying a dental formula with a tooth as an object,
an input unit capable of inputting object image data;
Executed by an object classification algorithm that includes at least a CNN (Convolutional Neural Network) as a module Executed by an object detection algorithm that incorporates an object feature extraction unit that extracts the feature amount of an object from an existing object image data set as a backbone and learns the input first teacher image data, learning image data set, and extended image data of the learning image data set, and performs transfer learning and fine tuning using the learning model of the object feature extraction unit. identifying a first rectangular information tag surrounding an image of each detection target object on the input detection target image and the position of the first rectangle to which the information tag is added; an object image placement unit capable of
Executed by the object classification algorithm, the first rectangular data and/or the first rectangular wide area data, second teacher image data different from the input first teacher image data, and the learning image A second learning model can be created by learning the data set and the extended image data of the learning image data set, and performing transfer learning and fine tuning using the learning model of the object feature extraction unit, and the object image arrangement. an object image identification unit capable of classifying and identifying the detection target object image by adding a unique information tag to the first rectangle identified by the unit;
The result output from the object image identification unit isBy DP (Dynamic Programming) alignment algorithman object image corrector capable of correcting;
an output unit capable of outputting a processing result of the detection target object image;
An image processing device comprising: The object classification algorithm comprises:
ＡｌｅｘＮｅｔ、ＧＰｉｐｅ（Ｇｉａｎｔ Ｎｅｕｒａｌ Ｎｅｔｗｏｒｋｓ ｕｓｉｎｇ Ｐｉｐｅｌｉｎｅ Ｐａｒａｌｌｅｌｉｓｍ）、Ｉｎｃｅｐｔｉｏｎ、ＳＥＢ（Ｓｑｕｅｅｚｅ－ａｎｄ－Ｅｘｃｉｔａｔｉｏｎ Ｂｌｏｃｋ）－Ｉｎｃｅｐｔｉｏｎ、Ｘｅｐｔｉｏｎ、ＤｅｎｓｅＮｅｔ（Ｄｅｎｓｅｌｙ Ｃｏｎｎｅｃｔｅｄ Ｃｏｎｖｏｌｕｔｉｏｎａｌ Ｎｅｔｗｏｒｋ）、ＲｅｓＮｅｔ（Ｒｅｓｉｄｕａｌ Ｎｅｔｗｏｒｋ）、ＳＥＢ－ＲｅｓＮｅｔ、Ｉｎｃｅｐｔｉｏｎ－ＲｅｓＮｅｔ、 ＳＥＢ－Ｉｎｃｅｐｔｉｏｎ－ＲｅｓＮｅｔ、ＲｅｓＮｅＸｔ、ＮＡＳＮｅｔ（Ｎｅｕｒａｌ Ａｒｃｈｉｔｅｃｔｕｒｅ Ｓｅａｒｃｈ Ｎｅｔｗｏｒｋ）、ＶＧＧ（Ｖｉｓｕａｌ Ｇｅｏｍｅｔｒｙ Ｇｒｏｕｐ）、ＳＥＢ－ＶＧＧ、ＭｏｂｉｌｅＮｅｔ、ＭｎａｓＮｅｔ、ＡｍｏｅｂａＮｅｔ、ＣＳＰＮｅｔ（Ｃｒｏｓｓ Ｓｔａｇｅ Ｐａｒｔｉａｌ Ｎｅｔｗｏｒｋ）、ＣＢＮｅｔ（Ｃｏｍｐｏｓｉｔｅ Ｂａｃｋｂｏｎｅ Ｎｅｔｗｏｒｋ）、Ｄａｒｋｎｅｔ、 2. The image processing apparatus according to claim 1, wherein at least one or more selected from EfficientNet and NFNet. The object classification algorithm comprises:
SEB, RB (Residual Block), DConv (Depthwise Convolution Layer), PConv (Pointwise Convolution Layer), MixConv (Mixed Depthwise Convolution Layer), and GAP (Global Average) block and/or modules selected from 2. The image processing apparatus according to claim 1, comprising at least one or more. The object classification algorithm comprises:
2. The image processing apparatus according to claim 1, wherein the network is at least one selected from ResNet, ResNeXt, MobileNet, MnasNet, Darknet, EfficientNet, and NFNet. 5. The image processing apparatus according to claim 1 , wherein said DP alignment algorithm is applied to semi-global alignment. The image processing apparatus according to any one of claims 1 to 5 , wherein the object detection algorithm is a two-stage object detection algorithm. The image processing apparatus according to any one of claims 1 to 5 , wherein the object detection algorithm is a one-step object detection algorithm. The two-stage object detection algorithm comprises:
At least one or more selected from R-CNN (Region-based Convolutional Neural Network), Fast R-CNN, Faster R-CNN, and R-FCN (Region-based Fully Convolutional Network) 7. The image processing apparatus according to claim 6 , characterized by: The one-step object detection algorithm comprises:
Оｖｅｒｆｅａｔ、ＤＰＭ（Ｄｅｆｏｒｍａｂｌｅ Ｐａｒｔｓ Ｍｏｄｅｌ）、ＳＳＤ（Ｓｉｎｇｌｅ Ｓｈｏｔ ＭｕｌｔｉＢｏｘ Ｄｅｔｅｃｔｏｒ）、ＤＳＳＤ（Ｄｅｃｏｎｖｏｌｕｔｉｏｎａｌ Ｓｉｎｇｌｅ Ｓｈｏｔ Ｄｅｔｅｃｔｏｒ）、ＥＳＳＤ（Ｅｘｔｅｎｄ ｔｈｅ ｓｈａｌｌｏｗ ｐａｒｔ ｏｆ Ｓｉｎｇｌｅ Ｓｈｏｔ ＭｕｌｔｉＢｏｘ Ｄｅｔｅｃｔｏｒ）、ＲｅｆｉｎｅＤｅｔ（Ｓｉｎｇｌｅ－Ｓｈｏｔ Ｒｅｆｉｎｅｍｅｎｔ Ｎｅｕｒａｌ Ｎｅｔｗｏｒｋ ｆｏｒ Ｏｂｊｅｃｔ Ｄｅｔｅｃｔｉｏｎ） , RetinaNet , M2Det, YOLO, Scaled YOLO, and EfficientDet. In the image processing device according to any one of claims 1 to 9 ,
learning the input first teacher image data, the learning image data set, and the extension data of the learning image data set, and performing transfer learning and fine tuning using the learning model of the object feature extraction unit; a first rectangular information tag containing an image of each of the detection target objects on the input detection target image; locating a rectangle of
the first rectangular data and/or the first rectangular wide area data, second teacher image data different from the input first teacher image data, the learning image data set, and the learning Using the second learning model learned by learning the extended data of the image data set and performing transfer learning and fine tuning using the learning model of the object feature extraction unit, the first rectangle, the adding a unique information tag to classify and identify the detection target object image;
a step of correcting the result of classification and identification of the detection target object image by the DP alignment algorithm ;
a step of outputting a processing result of the detection target object image;
An image processing method characterized by passing through. In the image processing device according to any one of claims 1 to 9 ,
learning third teacher image data different from the input first and second teacher image data, the learning image data set, and extension data of the learning image data set; Using a third learning model acquired by performing transfer learning and fine tuning using a learning model, the position of a second rectangle containing all of the detection target object images on the input detection target image is specified. and
learning the second rectangular data and/or the second rectangular wide area data, the input first teacher image data, the learning image data set, and extension data of the learning image data set; , using a fourth learning model acquired by performing transfer learning and fine tuning using the learning model of the object feature extraction unit, and including the image of each of the detection target objects on the input detection target image. identifying a third rectangular information tag and the position of the third rectangle to which the information tag is attached;
learning the third rectangular data and/or the third rectangular wide area data, the input second teacher image data, the learning image data set, and extension data of the learning image data set; , using a fifth learning model acquired by performing transfer learning and fine tuning using the learning model of the object feature extraction unit, adding the unique information tag to the third rectangle to add the detection target object image classifying and identifying the
a step of correcting the result of classification and identification of the detection target object image by the DP alignment algorithm ;
a step of outputting a processing result of the detection target object image;
An image processing method characterized by passing through. applying said DP alignment algorithm to a semi-global alignment characterized byClaim 10 or 11The image processing method described in . 13. The image processing method according to any one of claims 10 to 12 , wherein the step of correcting by the DP alignment algorithm applies a result of processing in the step of classifying and identifying the detection target object image. . the object image is Claims 1 to 3, characterized in that they are dental digital photographs9The image processing device according to any one of . The image processing method according to any one of claims 10 to 13 , wherein said object is a tooth and said object image is a dental digital photograph.

Images