JP2023099273A - Oral cavity image learning/classification device and method of the same - Google Patents

Abstract

To provide an oral cavity image learning/classification device which can determine whether or not an image is an image related to an oral cavity and guide a determination result with image analysis using a machine learning algorithm when an image obtained by imaging the oral cavity is uploaded and a method of the same.SOLUTION: An oral cavity image learning/classification device includes a user terminal which guides determination information obtained by determining whether or not an oral cavity photograph directly captured by a user is an oral cavity image including an image of an oral cavity, collects the oral cavity photograph provided from the user terminal, learns and analyzes the collected oral cavity photograph with a CNN algorithm, and can determine whether or not the oral cavity photograph is the oral cavity image.SELECTED DRAWING: Figure 1

Description

The present invention relates to an oral image learning/classification method, and more particularly, an oral image learning/classifying device that trains an image classification model based on training images for oral and non-oral types and classifies uploaded images. and its method.

Appropriate preventive treatment and continuous management of oral diseases can provide a high preventive effect, but many patients with oral diseases do not seek medical advice or seek treatment because they do not experience oral health problems. As such, there is still a lack of awareness of oral prevention.

Oral diseases are different from other diseases in that preventive methods are very important. Even simple treatments such as scaling can prevent periodontal diseases, which is a simple preventive measure for children who are expected to have malocclusion. If treatment is given, there is no need to receive orthodontic treatment after the growth period, which is very advantageous not only in terms of oral health management but also in terms of cost.

In predicting the current oral condition of the patient, accurate examination and diagnosis results and objective numerical information are required, not the subjective opinion of the dentist. There is no system to calculate it.

Oral diseases include tooth decay and periodontal disease.

In particular, periodontal diseases, including periodontitis and gingivitis, do not cause pain in the early stages and progress chronically. be.

Therefore, various methods for promoting oral health and preventing oral diseases have been researched and proposed based on the preliminary diagnosis of such oral diseases.

For example, Patent Literatures 1 to 3 below disclose conventional techniques for oral health promotion and oral disease prevention.

In Patent Document 1, user condition information including information on at least one of the user's gender, age, periodontal disease, and goal setting information and treatment information at the user's member medical institution are sent from the user terminal. is entered. Also, information on oral care products is input from an oral care product member store. Thereafter, information on specific oral care products included in a specific category corresponding to the user's condition information and treatment information is provided to the user terminal to provide oral care services suitable for the user.

In Patent Document 2, the patient's oral condition is examined, and the oral examination index calculation reference data including the patient's oral examination data, residence, and age information are converted into big data, and then analyzed. The patient's place of residence and age determine the impact of the oral health examination index calculation standard data on oral health. Then, a weight is given to the oral examination index calculation reference data to quantify the reference data for oral examination index calculation, and oral health is managed.

In Patent Document 3, user's personal information, oral hygiene behavior data, oral interview data, and oral examination data generated by the user visiting a specialized medical institution are collected. Each information collected in this way is integrated and analyzed to evaluate the user's risk of oral disease and provide personalized service.

On the other hand, prior art oral examination devices accept all images taken by the user for oral analysis, which may lead to erroneous oral examination results.

Therefore, there is a demand for development of a technique that can confirm whether the photograph taken by the user is a photograph of the oral cavity before performing further analysis, and guide the user to take only the photograph of the oral cavity.

Korean Patent Publication No. 10-2017-0050467 Korean Patent No. 10-1868979 Korean Patent No. 10-1788030

The object of the present invention is to solve the above-mentioned problems. When a user uploads an image of the oral cavity, it analyzes the image using a machine learning algorithm to obtain an image of the oral cavity. It is an object of the present invention to provide an oral cavity image learning/classifying device and method capable of determining whether or not and guiding the determination result.

In order to achieve the above objects, the oral cavity image learning/classifying device according to the present invention provides an oral cavity photograph taken directly by a user, and determines whether the oral cavity photo is an oral cavity image containing an intraoral image. Collect the user terminal that guides the determination information and the oral cavity photograph provided from the user terminal, learn and analyze the collected oral cavity photo with the CNN algorithm, and determine whether the relevant oral cavity photo is an oral cavity image and an analysis server that determines whether or not the user terminal has the determination information and provides the determination information to the user terminal.

Further, in order to achieve the above-described objects, the oral cavity image learning/classification method according to the present invention includes (a) a step of collecting oral cavity photographs taken using a user terminal in an analysis server; (c) using the split training data to design a classification model; (d) split testing using data to test and evaluate the performance of the designed classification model; and (e) using the tested and evaluated analysis model to determine whether the collected oral photographs are oral images. and a step of determining whether the

According to the oral cavity image learning/classifying apparatus and method according to the present invention, when a user uploads oral cavity photos, the oral cavity photos are analyzed in real time using machine learning algorithms to determine whether the intraoral images are included in the corresponding oral cavity photos. It is possible to accurately determine whether or not the image is an oral cavity image.

Thus, according to the present invention, before performing further analysis for determining oral health conditions using oral photographs, it is determined whether or not the photographed oral photographs are oral images, and determination information is obtained. It can guide and provide oral images for further analysis.

1 is a block diagram of an oral image learning/classifying device according to a preferred embodiment of the present invention; FIG. 1 is a flow chart explaining step by step an oral cavity image learning/classification method according to a preferred embodiment of the present invention; FIG. 4 is a diagram illustrating a process of analyzing and classifying input oral cavity photographs with the CNN algorithm; FIG. 4 is a diagram illustrating a process of convolving an input image using a filter and extracting features; FIG. 10 is a diagram illustrating a process of classifying an oral image and a non-oral image by connecting layers transformed by a one-dimensional vector into one vector; FIG. 10 is a diagram showing a classification model for recognizing whether or not an oral cavity image exists; FIG. 4 is a diagram showing the process of judging an input image using a classification model;

Hereinafter, an oral image learning/classifying apparatus and method according to preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a block diagram of an oral image learning/classifying device according to a preferred embodiment of the present invention.

The oral image learning/classifying device 10 according to the preferred embodiment of the present invention includes a user terminal 20 and an analysis server 30, as shown in FIG.

The user terminal 20 and the analysis server 30 are connected by various wired/wireless networks and have real-time communication interfaces.

The user terminal 20 is a terminal used by a user who wants to check the oral health condition, and can take and provide oral photographs of the user.

Here, the user uses the user terminal 20 to provide the user's personal information, interview data, general data, and the like.

Through the user terminal 20, the analysis server 30 receives determination information corresponding to the result of determination as an oral image (oral image) or a non-oral image (non-oral image). Whether it is an oral image or a non-oral image can be confirmed. Therefore, when the user determines that the image is a non-oral image, the user can again take and provide an intraoral photograph.

Such a user terminal 20 can be implemented as a mobile device such as a smart phone and a smart pad, a desktop computer and a laptop computer capable of communication, etc. In this embodiment, it is implemented as a smart phone.

The analysis server 30 collects the oral cavity photos provided from the user terminal 20, analyzes the collected oral cavity photos with a machine learning algorithm (CNN, DNN, etc.), and the oral cavity photo contains the oral cavity image It has the function of determining whether or not

Such an analysis server 30 analyzes the collected oral cavity photographs to determine whether the relevant oral cavity photographs are oral images or non-oral images, and provides determination information to the user terminal 20 .

For this reason, the analysis server 30 includes a data collection unit 31 that collects oral cavity photographs taken from the user terminal 20, a data analysis unit 32 that learns and analyzes the collected oral cavity photographs with the CNN algorithm, and an analyzed result Based on, an image determination unit 33 that classifies and determines an oral image and a non-oral image containing an intraoral image, and a determination information providing unit 34 that transfers the determination result to the user terminal 20 as determination information include.

That is, in this embodiment, the analysis server 30 trains an image classification model to classify oral and non-oral images.

In order to train such a classification model, the data collection unit 31 of the analysis server 30 collects a huge amount of image data regarding oral images and non-oral images including the intraoral area.

The data analysis unit 32 designs CNN models using Python-based libraries such as Tensorflow and Keras. To this end, the data analysis unit 32 preprocesses the collected oral photographs and randomly divides them into training data and test data.

The data analysis unit 32 also tests and tests the classification model using the split test data to see if the classification model designed using the split training data provides the best performance. evaluate.

When the performance of the classification model is optimized from the evaluation results, the optimized classification model is stored, and the image determination unit 33 uses the optimized classification model to classify and produce random input images.

FIG. 2 is a flow chart explaining step by step an oral image learning/classification method according to a preferred embodiment of the present invention.

The oral cavity image learning/classifying method according to the preferred embodiment of the present invention includes the steps of (a) collecting taken oral cavity photographs using a user terminal in an analysis server; (c) using the split training data to design a classification model; (d) using the split test data to design (e) using the tested and evaluated analysis model to determine whether the collected oral photograph is an oral image; include.

Also, the oral cavity image learning/classification method according to the preferred embodiment of the present invention further includes the step of (f) transferring the determination information determined in step (e) to the user terminal.

First, the user downloads an oral health management application and stores it in the user terminal 20 in order to check and manage his/her oral health condition.

Then, the user executes the oral health management application and requests an oral health examination while providing oral photos, personal information, etc., in order to check the oral health status. Here, if necessary, interview data for oral health examination is created and provided. The medical interview data is a medical questionnaire in a check format. The medical questionnaire can include items for confirming whether or not correction is possible.

The oral cavity photograph is preferably a photograph of the entire oral cavity, and a photograph of only a part of the oral cavity can also be used.

Further, the oral cavity photograph is an oral cavity image including the intraoral area, or a non-oral image not including the intraoral area.

Analysis server 30 thereby determines whether the collected oral photographs are oral or non-oral images before performing further analysis of the oral photographs to ascertain oral health.

More specifically, in step S<b>10 , the user terminal 20 takes an oral cavity photograph by user operation and uploads the taken oral cavity photograph to the analysis server 30 .

Then, the data collection unit 31 of the analysis server 30 collects the oral cavity photographs received from the user terminal 20 .

Thereby, the data analysis unit 32 learns the collected oral cavity photographs by the CNN algorithm and determines whether or not the collected oral cavity photographs are oral cavity images.

Deep learning is a type of machine learning that focuses on the structure of the human brain and consists of a series of algorithms that imitate its activity.

CNN is a deep learning technique that uses convolutional operations to transform.

Such a CNN includes a convolution layer, a pooling layer, and a fully-connected layer as basic layers. A deep structure of the CNN model includes an activation layer, a normalization layer, and a drop-out layer that improve the overall model performance.

In this example, for the purpose of training a classification model, we develop a model/algorithm that trains images for oral and non-oral labels/classes containing a large number of training images for each label. This is done by a CNN that learns and extracts various features present in oral and non-oral images.

Developing CNN algorithms is time consuming and requires huge amounts of data, but once developed, the process is automated and fast.

This access method can also search the analysis server 30 for the developed solution and generate an immediate response to the user terminal 20 .

When the user takes and uploads oral photographs, the classification model can analyze the images taken by the analysis server 30, determine the taken oral photographs as oral or non-oral images, and notify the user.

The classification model accepts the oral picture for further analysis if the oral picture is an oral picture.

In addition, if the oral cavity photograph is a non-oral image, the classification model guides the user that it is not an oral cavity image, so that the user can take and upload the oral cavity image again.

FIG. 3 is a diagram illustrating the process of analyzing and classifying input oral photographs using the CNN algorithm, and FIG. 4 is a diagram illustrating the process of extracting features by convolving the input image using a filter. FIG. 5 is a diagram for explaining the process of classifying an oral image and a non-oral image by connecting layers converted by one-dimensional vectors into one vector.

In FIG. 3, for a large number of oral photographs including oral and non-oral images, after going through multiple convolution layers and pooling layers with the CNN algorithm, a fully-connected layer ) to finally derive the determination result.

FIG. 4 shows the process of max pooling and average pooling of the input image, that is, the oral cavity photograph and the filter combined and multiplied and convoluted.

FIG. 5 shows a process of judging an oral image or a non-oral image through a fully connected layer with a previous layer transformed into a one-dimensional vector by flattening.

As shown in FIGS. 3-5, when oral or non-oral images are input to the CNN and transferred, the CNN algorithm passes through a number of convolutional and pooling layers and then through fully connected layers. image recognition result, that is, correction information for determining whether or not it is an oral cavity image.

Python packages such as Tensorflow and Keras can be used to build CNN models and match input sizes and output classes so that the system GPU can be fully utilized. Thus, the system GPU can be used to increase training speed and reduce training time.

On the other hand, it is important to design a strong CNN model to guide the user to take oral images only. However, CNN models do not always guarantee the best results.

Thus, in the present example, to improve performance, pre-trained models such as ResNet50 can be leveraged, with initial weights value can be added.

Thus, using a pre-trained model can increase prediction accuracy and reduce performance time. If it is a non-oral image, a guidance message can be forwarded to the user to take an oral image again.

Pretrained models such as ResNet50 are based on transfer learning access methods.

Transfer learning refers to leveraging pre-trained models by fine-tuning fully connected layers and user-defining models for all types of tasks.

Such transfer learning essentially uses any model as a starting point for new work.

A deep residual network, known as ResNet, builds on the core concept denoted by skip connections or shortened connections, where data and gradations are combined with the actual network flow. , through the network, can provide alternate routes that can be traveled.

Specifically, in step S12 of FIG. 2, the data analysis unit 32 preprocesses the collected oral photographs and randomly divides them into training data and test data.

In step S14, the data analysis unit 32 uses the divided training data to design a classification model based on a Python-based library.

In step S16, the data analysis unit 32 trains and tests the classification model using the split test data to see if the designed classification model provides the best performance. Evaluate the results.

When the performance of the classification model is optimized through the process of continuously upgrading and tuning the classification model through such an evaluation process, the optimized classification model is saved.

In step S18, the image determination unit 33 determines whether the oral cavity photograph collected using the optimized classification model is an oral image or a non-oral image, and the determination information providing unit 34 The determination result is transferred to the user terminal (S20).

Here, if the non-oral image is determined, the user terminal 20 displays a guidance message to prompt the user to take an oral image again.

On the other hand, if it is determined to be an oral image, the analysis server 30 provides further analysis information for confirming the oral health condition from the corresponding oral image.

FIG. 6 is a diagram showing a classification model for recognizing whether or not an oral cavity image exists, and FIG. 7 is a diagram showing a process of judging an input image using the classification model.

Oral photographs, ie, image data consisting of oral and non-oral images, are input and transferred to the CNN block, and the CNN block extracts and learns various features from the oral photographs in a training step.

During CNN training, convolutional layers extract features or feature maps from input images, pooling layers perform non-linear downsampling to reduce the size of feature maps, and fully connected layers perform final convolutions or Call the output feature map of the pooling layer to classify the image.

Here, it can be clearly seen that the size of the input image comparison output value is significantly reduced, as shown in FIG.

As the pre-trained model, ResNet 50 makes the classification model with skip connections not shallow, as shown in FIG. 7, for high density. Such skip connections help extract more features and improve the performance of the classification model.

The classification model is then optimized through a training and testing process, and the optimized classification model is stored for production and future use where unknown images are tested.

The classification model is part of the model deployment API and allows the user to take oral cavity images and upload the taken oral cavity photographs to the analysis server 30 .

According to the above-described process, when a user uploads an oral cavity photograph, the present invention uses a machine learning algorithm to analyze the oral cavity photo in real time, and obtains an oral cavity image containing an intraoral image in the corresponding oral cavity photo. can be determined correctly.

Thereby, the present invention uses the oral photograph to determine whether the photographed oral photograph is an oral image before performing further analysis to determine the oral health condition, and guides the determination information. can provide oral images for further analysis.

The invention made by the present inventors has been specifically described above with reference to examples, but the present invention is not limited to the above-described examples, and can be variously modified without departing from the scope of the invention. It goes without saying.

10 Oral image learning/classification device 20 User terminal 30 Analysis server 31 Data collection unit 32 Data analysis unit 33 Image judgment unit 34 Judgment information provision unit

Claims (5)

An oral image learning/classifying device that analyzes oral photographs with a machine learning algorithm to learn and classify,
A user terminal that provides an oral cavity photograph directly taken by a user and guides determination information for determining whether the oral cavity photo is an oral cavity image including an intraoral image;
Collect oral photos provided from the user terminal, learn and analyze the collected oral photos with the CNN algorithm, determine whether the relevant oral photos are oral images, and the determination information an analysis server provided to the user terminal; and an oral image learning/classification device. The analysis server is
a data collection unit that collects the oral cavity photograph taken from the user terminal;
a data analysis unit that learns and analyzes the collected oral photographs with the CNN algorithm;
an image determination unit that classifies and determines the oral image and the non-oral image based on the analysis result;
The oral cavity image learning/classification device according to claim 1, further comprising a determination information providing unit that transfers a determined result as the determination information to the user terminal. The data analysis unit preprocesses the collected oral photographs and divides them into training data and test data, uses the divided training data to design a classification model, and uses the divided test data. to test and evaluate the performance of the designed classification model;
3. The oral cavity according to claim 2, wherein the image determination unit determines whether or not the collected oral cavity photograph is the oral cavity image using the analytical model subjected to the testing and evaluation. Image learning/classifier. An oral image learning / classification method for analyzing, learning and classifying by analyzing the oral cavity photograph with the machine learning algorithm using the oral cavity image learning / classification device according to any one of claims 1 to 3,
(a) collecting the photographed oral cavity photograph using a user terminal in an analysis server;
(b) preprocessing the collected oral photographs to divide them into training data and test data;
(c) using the split training data to design a classification model;
(d) testing and evaluating the performance of the designed classification model using split test data;
(e) using the tested and evaluated analytical model to determine whether the collected oral photograph is the oral image. Method. (f) forwarding the determination information determined in step (e) to the user terminal;
The analysis server provides further analysis information for confirming oral health status when the oral photograph is determined to be the oral image;
5. The oral cavity image learning/classifying method according to claim 4, wherein, when the oral cavity photograph is determined to be a non-oral image, the user terminal displays a guidance message to take an intraoral image again.

Images