CN113436734B

CN113436734B - Tooth health assessment method, equipment and storage medium based on face structure positioning

Info

Publication number: CN113436734B
Application number: CN202010209040.2A
Authority: CN
Inventors: 罗冠; 游强; 殷晓珑; 田勇
Original assignee: Beijing Haola Technology Co ltd
Current assignee: Beijing Haola Technology Co ltd
Priority date: 2020-03-23
Filing date: 2020-03-23
Publication date: 2024-03-05
Anticipated expiration: 2040-03-23
Also published as: CN113436734A

Abstract

The invention discloses a tooth health assessment method, equipment and storage medium based on face structure positioning. The method comprises the following steps: collecting an effective face image; determining the attitude angle of a target face in the effective face image; if the attitude angle of the target face is within the preset attitude angle range, extracting a tooth area image of the target face from the effective face image; dividing a tooth image corresponding to each tooth in the tooth area image; and determining the tooth health grade of the target face according to the tooth image corresponding to each tooth. The embodiment of the invention can solve the problems of high cost, complex process, complete participation of specialized staff and the like in the comprehensive evaluation of the tooth health by means of instruments by utilizing an image processing technology without the help of specialized medical instruments.

Description

Tooth health assessment method, equipment and storage medium based on face structure positioning

Technical Field

The present invention relates to the field of image processing technologies, and in particular, to a method, an apparatus, and a storage medium for tooth health assessment based on face structure positioning.

Background

With the continuous development of economy, people pay more and more attention to the health level of teeth. On the one hand, dental health is an important index of human health, and the oral cavity is used as a first gateway of the digestive system of the human body, and the "good mouth" often indicates the "good stomach". On the other hand, the tooth with a clean and whitening effect has a great effect on improving the image of the individual, and can lead the person to be more confident and face work and life from the beginning.

Currently, more and more users are focusing on their own dental health problems, and the current dental health assessment methods often need to use a professional doctor and/or a professional instrument, which causes time and region limitations when the users perform dental health assessment, for example: the user can only go to a hospital or dental office for a dental health assessment, which can result in a costly dental health assessment.

Disclosure of Invention

The embodiment of the invention mainly aims to provide a tooth health assessment method, equipment and storage medium based on face structure positioning, so as to solve the problem of high cost of tooth health assessment at present.

Aiming at the technical problems, the embodiment of the invention is solved by the following technical scheme:

The embodiment of the invention provides a tooth health assessment method based on face structure positioning, which comprises the following steps: collecting an effective face image; determining the attitude angle of a target face in the effective face image; if the attitude angle of the target face is within the preset attitude angle range, extracting a tooth area image of the target face from the effective face image; dividing a tooth image corresponding to each tooth in the tooth area image; and determining the tooth health grade of the target face according to the tooth image corresponding to each tooth.

Wherein, gather effective face image, include: collecting a user environment image; determining an average brightness value of the user environment image; if the average brightness value of the user environment image is within a preset brightness value range, face detection is carried out on the user environment image; if a face is detected in the user environment image, determining that the user environment image is a valid face image; and if the average brightness value of the user environment image is not within the brightness value range, or no human face is detected in the user environment image, carrying out user environment image re-acquisition prompt.

Before the face detection is performed on the user environment image, the method further comprises the following steps: determining an image brightness standard deviation of the user environment image; and if the image brightness standard deviation is smaller than a preset image brightness standard deviation threshold, performing image enhancement processing on the user environment image by using a gamma conversion algorithm.

The determining the attitude angle of the target face in the effective face image comprises the following steps: marking the effective face image by marking points aiming at the target face; acquiring a preset three-dimensional human body head portrait model; wherein, the face of the three-dimensional human body head portrait model is marked with mark points, and the number of the mark points marked on the face of the three-dimensional human body head portrait model and the number of the mark points marked on the target human face are the same as the types in the same dimension space; and determining the attitude angle of the target face according to the mark points in the three-dimensional human body head portrait model and the mark points aiming at the target face in the effective face image.

Wherein before extracting the tooth area image of the target face in the effective face image, the method further comprises: marking the effective face image by marking points aiming at the target face; determining the opening and closing angle of the oral cavity of the target face according to the marking points used for marking the oral cavity area in the marking points of the target face; if the opening and closing angle of the oral cavity is larger than a preset opening and closing angle threshold, extracting a tooth area image of the target face from the effective face image; otherwise, the effective face image is acquired again and prompted.

Wherein, in the effective face image, extracting the tooth area image of the target face includes: marking points respectively used for positioning a non-tooth area, a candidate non-tooth area, a tooth area and a candidate tooth area in a face are preset; the non-tooth area, the candidate non-tooth area, the tooth area and the candidate tooth area which are positioned by the marking points are used as parameters in a preset image segmentation algorithm; performing image segmentation processing on the effective face image through the image segmentation algorithm to obtain an initial tooth region image; and screening out pixel points with the color space in a preset tooth color space range from the initial tooth area image, and forming a tooth area image according to the screened pixel points.

Wherein segmenting the tooth image corresponding to each tooth in the tooth region image comprises: extracting edge lines from the tooth region image according to a preset edge extraction algorithm; calculating the length of each edge line and the average length of all edge lines; determining a length threshold and a distance threshold according to the average lengths of all the edge lines; for each edge line, if the length of the edge line is greater than the length threshold and the distance between two endpoints of the edge line and the nearest line segment is less than the distance threshold, determining the edge line as a tooth dividing line; extracting contour images of each candidate tooth in the tooth region image according to the tooth dividing line; and respectively matching the contour image of each candidate tooth with a plurality of preset tooth shapes, if the tooth shapes matched with the contour image of the candidate tooth exist, determining the candidate tooth as a tooth, and dividing the tooth image according to the contour image of the candidate tooth.

Wherein determining the tooth health level of the target face according to the tooth image corresponding to each tooth comprises: determining a tooth area center line according to the mark points of the target face; determining the left-right contact ratio of the teeth according to the tooth region center line and the tooth image corresponding to each tooth segmented in the tooth region image; determining the overall uniformity of the teeth according to a preset standard model of each tooth and a tooth image corresponding to each tooth segmented in the tooth area image; determining the overall yellowing degree of the teeth according to the tooth images corresponding to each tooth which are segmented in the tooth area images; and determining the tooth health grade of the target face according to the left-right contact ratio of the teeth, the overall uniformity of the teeth and the overall yellowing degree of the teeth.

The embodiment of the invention also provides a tooth health evaluation device based on the face structure positioning, which comprises a processor and a memory; the processor is configured to execute a dental health evaluation program based on facial structure localization stored in the memory, so as to implement any one of the above dental health evaluation methods based on facial structure localization.

The embodiment of the invention also provides a storage medium, wherein one or more programs are stored in the storage medium, and the one or more programs can be executed by one or more processors to realize the tooth health assessment method based on the face structure positioning.

The embodiment of the invention has the following beneficial effects:

the embodiment of the invention collects the effective face image, identifies the tooth area image in the effective face image, and carries out tooth health assessment according to the tooth area image. The embodiment of the invention can solve the problems of high cost, complex process, complete participation of specialized staff and the like in the comprehensive evaluation of the tooth health by means of instruments by utilizing an image processing technology without the help of specialized medical instruments.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiments of the invention and together with the description serve to explain the invention and do not constitute a limitation on the invention. In the drawings:

FIG. 1 is a flow chart of a method of tooth health assessment based on face structure localization in accordance with an embodiment of the present invention;

FIG. 2 is a flowchart of steps for acquiring a valid face image according to one embodiment of the present invention;

FIG. 3 is a flowchart of steps of an image enhancement process according to an embodiment of the present invention;

FIG. 4 is a flowchart of the steps for attitude angle determination according to one embodiment of the present invention;

FIG. 5 is a schematic diagram of coordinate system conversion according to an embodiment of the invention;

FIG. 6 is a flow chart of the verification of the effectiveness of the opening and closing angle of the oral cavity according to an embodiment of the present invention;

FIG. 7 is a flowchart of the steps for extracting an image of a dental region in accordance with one embodiment of the present invention;

FIG. 8 is a flowchart of steps for tooth region image segmentation in accordance with one embodiment of the present invention;

FIG. 9 is a flowchart of the steps for tooth health level assessment according to one embodiment of the present invention;

fig. 10 is a block diagram of a tooth health assessment device based on face structure localization in accordance with an embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the drawings and the embodiments, in order to make the objects, technical solutions and advantages of the present invention more apparent.

According to an embodiment of the invention, a tooth health assessment method based on face structure positioning is provided. Referring to fig. 1, a flowchart of a method for evaluating tooth health based on face structure localization according to an embodiment of the invention is shown.

Step S110, collecting effective face images.

The effective face image refers to an image containing a face and having an average luminance value within a preset average luminance value range.

Step S120, determining the attitude angle of the target face in the effective face image.

The target face refers to the face of the user whose teeth health level is to be evaluated.

The pose angles (θ, ψ, Φ) of the target face include: pitch angle θ, yaw angle ψ, and rotation angle φ.

In this embodiment, the pose angle of the target face is determined from the face image of the target face.

The effective face image may include a plurality of faces, and one face is selected as a target face in the effective face image.

Step S130, if the attitude angle of the target face is within the preset attitude angle range, extracting the tooth area image of the target face from the effective face image.

The tooth region image is an image of the teeth of the target face.

And if the attitude angle of the target face is within the preset attitude angle range, the target face is basically a front face. The pitch angle range epsilon-25 DEG, 25 DEG in the attitude angle range, the deflection angle range epsilon-25 DEG, 25 DEG and the rotation angle range epsilon-35 DEG, 35 DEG can be set. When θ=0, ψ=0, and Φ=0, it means that the current target face is a standard face. The attitude angle of the target face is within the range of the attitude angle, namely: and judging that the target face is effective if the pitch angle of the target face is within the pitch angle range, the deflection angle is within the deflection angle range and the rotation angle is within the rotation angle range.

If the attitude angle of the target face is not within the preset attitude angle range, the target face is not the front face, and the re-acquisition prompt is carried out, so that the user re-acquires the user environment image according to the re-acquisition prompt. Further, comparing the attitude angle of the target face with a preset attitude angle range, if the attitude angle range is exceeded, invalidating the target face, and sending a re-acquisition prompt to the user so as to prompt the user to upload an image containing the front face.

And step S140, segmenting out a tooth image corresponding to each tooth of the tooth area image.

The tooth region image comprises a plurality of tooth images of teeth, and the tooth region image is subjected to image segmentation processing by utilizing a preset image segmentation algorithm to segment the tooth image corresponding to each tooth.

And step S150, determining the tooth health grade of the target face according to the tooth image corresponding to each tooth.

Tooth health grade, is used for measuring the tooth health degree of target human face.

Determining the left-right contact ratio, the overall uniformity and the overall yellowing degree of the teeth according to the corresponding tooth image of each tooth; and determining the tooth health grade of the target face according to the left-right contact ratio of the teeth, the overall uniformity of the teeth and the overall yellowing degree of the teeth.

The right-left tooth overlap ratio refers to the overlap ratio between the tooth area on the left side and the tooth area on the right side in the tooth area image.

The overall uniformity of teeth refers to the average value of the shape similarity of each tooth in the image of the tooth area and the standard model of the corresponding position.

Overall tooth yellowing refers to the average of all teeth in the image of the tooth area.

In this embodiment, the tooth health level of the target face may be compared with a preset health interval, and if the tooth health level is not within the health interval, early warning of the tooth health state may be performed. The healthy area may be an empirical value or a value obtained through experimentation. For example, the health interval is a tooth health rating of greater than 0.8; when the tooth health grade is more than 0.8, the overall condition of the tooth is better; when the tooth health grade is less than or equal to 0.8, the overall health condition of the teeth is poor, and the user corresponding to the target face needs to be prompted to go to a special dental medical institution for more deep examination and treatment. Of course, a plurality of sections can be divided, each section corresponds to a prompt message, and corresponding prompt messages are popped up according to the section where the tooth health grade is located.

The embodiment can solve the problems of high cost, complex process, need of comprehensive participation of professional staff and the like when the comprehensive evaluation of the tooth health is carried out by means of instruments by utilizing an image processing technology without the aid of professional medical instruments. According to the embodiment, the face image processing is utilized to carry out tooth health grading screening, processing comments are individually given to the health problems of the teeth of the user, the method has the advantages of being low in cost, free of time and region limitation, capable of completing tooth health assessment anytime and anywhere, and capable of being conveniently accessed into an online medical scene.

In the embodiment, teeth in the effective face image can be detected by using a machine learning method, and the basic idea is to mark out the tooth parts in the existing face sample to form a data set of the teeth, and then perfect the detection and segmentation flow of the teeth according to a Mask R-CNN model learning mode; the method can also use a plurality of segmentation methods for fusion when the teeth are segmented according to the effective face image so as to improve the accuracy of the teeth segmentation; the evaluation mode of the tooth health grade can be increased, because the index of the tooth health analysis can be more, for example, the tooth is blacker due to the existence of some calculus or long-term smoking, and more health indexes can be completely analyzed according to the flow of the scheme.

The steps for acquiring a valid face image are described in detail below.

Fig. 2 is a flowchart illustrating steps for acquiring an effective face image according to an embodiment of the present invention.

Step S210, collecting a user environment image.

The user environment image refers to an image in the field of view of the camera acquired by the camera.

The user environment image may invoke a camera of the user device or the dental health assessment device to capture the user environment image, or to obtain a user environment image uploaded by the user. For example: the user equipment is utilized to collect the user environment image in real time, and the user can be prompted to upload the user environment image.

In the user environment image, one or more faces may be included. Of course, no face may be included in the user environment image.

Step S220, determining an average brightness value of the user environment image.

In this embodiment, I (x, y) may be used to represent a user environment image, where the width of the user environment image is w and the height is h; wherein x is E [0,w ]],y∈[0,h]；I _xy The value of (2) represents the brightness value of a pixel point with (x, y) position coordinates in the user environment image, I _xy ∈[0,255]。

The calculation formula of the average brightness value of the user environment image is as follows:

Further, if the user environment image is a color image, I _xy ＝[I _R ,I _G ,I _B ]Wherein I _R ，I _G And I _B The luminance values of the three channels of red, yellow and blue respectively, and the average luminance value of the user environment image can be replaced by the average value of the luminance average values of the three channels, namely: average luminance value of user environment image = (luminance average of red channel + luminance average of yellow channel + luminance average of blue channel)/(3), luminance average = sum of luminance values of all pixels/(number of all pixels).

Step S230, judging that the average brightness value of the user environment image is within a preset brightness value range; if yes, go to step S240; if not, step S270 is performed.

Presetting the brightness value range as [ I ] ⁰ ,I ¹ ]. The end value I of the brightness value range ⁰ And I ¹ May be an empirical value or a value obtained through experimentation. When (when)The average brightness value representing the user environment image is too dark; when->The average luminance value representing the user environment image is too bright.

In this embodiment, in order to reduce the number of times of capturing the user environment image, a more extreme case is simulated in advance, for example, the average brightness value of the user environment image in the night environment and the high-power light source direct face scene is simulated, and the average brightness value of the user environment image in the night environment is taken as the lower limit I of the brightness value range ⁰ Taking the average brightness value of the user environment image in the high-power light source direct-irradiation face scene as the upper limit I of the brightness value range ¹ . Further, the lower limit I of the luminance value range may be set ⁰ And upper limit I ¹ Which are set to 25 and 230 in sequence. The extreme average brightness value of the image taken in the daily case is difficult to appear, and once the extreme case appears, the image is hardly representedFor this purpose, a predetermined reject operation may be performed. The rejection operation may be a re-acquisition prompt. By judging the brightness of the user environment image, the accuracy of the subsequent face detection can be improved.

Step S240, if the average brightness value of the user environment image is within the brightness value range, performing face detection for the user environment image.

The manner in which face detection is performed for the user environment image will be described in detail later.

Step S250, judging whether a human face is detected in the user environment image; if yes, go to step S260; if not, step S270 is performed.

Step S260, if a face is detected in the user environment image, determining that the user environment image is a valid face image.

After a face is detected in the user environment image, a face region is identified in the user environment image, and the identified face region is taken as a face image.

In this embodiment, the area where the face is located may be identified in the user environment image by using a face detection frame. If a plurality of faces are detected in the user environment image, the areas of each detected face are respectively identified by using a plurality of face detection frames.

Step S270, if the average brightness value of the user environment image is not within the brightness value range, or no face is detected in the user environment image, performing a user environment image re-acquisition prompt.

In this embodiment, before face detection is performed on the user environment image, in order to ensure that the user environment image has good contrast, image enhancement processing may be performed on the user environment image.

The contrast of a user-environment image refers to a measure of the different brightness levels between the brightest white and darkest black of the bright-dark areas in the user-environment image, i.e. the magnitude of the user-environment image brightness contrast (difference). The larger the brightness contrast, the larger the contrast, and the smaller the brightness contrast, the smaller the contrast.

In the present embodiment, the manner of image enhancement processing includes, but is not limited to: gamma conversion and logarithmic conversion. The image enhancement processing is performed on the user environment image with a smaller contrast, and will be described in detail.

As shown in fig. 3, a flowchart of steps of image enhancement processing according to an embodiment of the present invention is shown.

Step S310, determining an image brightness standard deviation of the user environment image.

In order to determine whether the user environment image needs to be subjected to an image enhancement operation, an image brightness standard deviation of the user environment image, which may be referred to as root mean square contrast, may be calculated.

In the present embodiment, the calculation formula of the image brightness standard deviation σ is as follows:

the larger the contrast of the user environment image is, the larger the standard deviation sigma of the image brightness is; the smaller the contrast of the user environment image, the smaller the image brightness standard deviation sigma.

Step S320, if the standard deviation of the image brightness is smaller than the preset standard deviation threshold of the image brightness, performing image enhancement processing on the user environment image by using a gamma transformation algorithm.

For the user environment image with smaller contrast ratio, the image enhancement processing can be performed by adopting a gamma conversion algorithm. The gamma transformation algorithm has the following standard form:

Wherein I (x, y) is a user environment image before image enhancement, O (x, y) is a user environment image after image enhancement, and γ is a control parameter. Wherein γ is greater than 0. That is, the following operation is performed for each pixel point in the user environment image:

wherein, the brightness value of the pixel point after the image enhancement is obtained.

When γ is greater than 1, the user environment image will darken as a whole, which stretches the areas of the image where the brightness is higher, while compressing the portions where the brightness is lower.

When γ is equal to 1, the user environment image is unchanged.

When γ is greater than 0 and less than 1, the user environment image will be overall illuminated, which stretches the lower brightness areas of the image while compressing the higher brightness portions.

In this embodiment, the average brightness value of the user environment image is combinedThe optimal brightness value range of the user environment image is between 165 and 175, and 170 can be taken as an average brightness value threshold.

Wherein, the empirical formula of gamma is as follows:

when (when)When gamma is equal to 1, the user environment image is unchanged; when->When tending to 0, γ tends to 0, the user environment image becomes bright as a whole, and the contrast increases; when->When it goes to 255, γ tends to be endless, the user environment image becomes dark as a whole, and the contrast ratio becomes large.

After the image enhancement processing is performed on the user environment image, denoising processing may also be performed on the user environment image after the image enhancement processing.

After the image enhancement processing is performed on the user environment image, face detection may be performed on the user environment image. Face detection is described further below.

The face detection method may be performed by a sliding window method. Specifically, the sliding window moves in a preset step in the user environment image, the classifier performs face recognition on the image area in the sliding window based on the external contour of the face, and when a shape matched with the external contour of the face exists in the image area, the image area is classified into the face, which represents that the face is detected.

The sliding window may be considered a face detection frame. Because of the different dimensions of faces, the size of the sliding window is scaled to match the dimensional changes of different faces. In the process of detecting a face by using the sliding window, a face detection method based on a gradient histogram (Histogram of Gradients) can be adopted to detect the face in the user environment image; face detection methods based on Harr-like features can also be used to detect faces in the user environment images.

Of course, because the face has special structure and texture characteristics, the embodiment of the invention can also use the deep neural network to detect the face in the user environment image.

Types of deep neural networks include, but are not limited to: a Multi-tasking cascade convolutional neural network (Multi-Task Convolution Neural Network, MTCNN for short) and a MobileNet-SSD.

In the embodiment of the invention, the MTCNN can be used for carrying out face detection on the input user environment image. The MTCNN can detect a human face in the user environment image, and the region where the detected human face is located is identified in the user environment image by using a human face detection frame.

The MTCNN is a face detection deep learning model based on the multitasking cascade CNN, and face frame regression and face key point detection are comprehensively considered in the model. The user environment images input into the MTCNN can be scaled into user environment images with different dimensions according to different scaling ratios, so that a feature pyramid of the images is formed, and faces with different dimensions can be detected. The MTCNN contains three cascaded subnetworks, called PNet, RNet and ONet, respectively. Wherein, for each scale of user environment image, PNet, RNet and ONet are used for:

The PNet generates regression vectors of candidate windows and boundary boxes for identifying the face areas according to the input user environment images; calibrating the generated candidate window by using the regression vector of the boundary box; and performing first de-duplication processing on the calibrated candidate frame openings through a first Non-maximum suppression (NMS) algorithm to obtain candidate windows subjected to PNet de-duplication.

Firstly, using the regression vector of the boundary box by RNet to calibrate the candidate window subjected to PNet deduplication; and performing secondary deduplication processing on the calibrated candidate window by using a second NMS algorithm to obtain a candidate window subjected to RNet deduplication. In this way, further screening of candidate windows that have been de-duplicated by PNet is achieved.

The function of the ONet is similar to the function of the RNet, and the regression vector of the bounding box is utilized to calibrate the candidate window subjected to RNet deduplication; and performing a third deduplication process on the calibrated candidate window by using a third NMS algorithm, and simultaneously generating five mark point positions while removing the overlapped candidate windows. Thus, the ONet detects five marker points on the face framed by each candidate window while further screening the candidate windows that have undergone RNet deduplication. The mark points refer to feature points marked at preset positions of the face. The five marker points include: the mark points marked on the two pupils respectively, the mark points marked on the nose and the mark points marked on the two corners of the mouth respectively.

The overlapping degree (Intersection over Union, abbreviated as IOU) set in the first NMS algorithm, the second NMS algorithm and the third NMS algorithm is different, the IOU is from big to small, the first NMS algorithm, the second NMS algorithm and the third NMS algorithm are sequentially arranged, and thus PNet, RNet and ONet can finish the weight removal of the candidate window from thick to thin.

Since the user environment image input into the MTCNN is scaled according to different scaling ratios to form an image pyramid, that is, a plurality of scale images, and then the PNet, the RNet and the ONet respectively perform face detection on the user environment image of each scale, all candidate windows need to be normalized to the user environment image of the original size after face detection. For example: if the scale of the user environment image is twice that of the original user environment image, the candidate window needs to be normalized to the original size when the user environment image is returned to the original size, i.e. the size of the candidate window is divided by 2. The candidate windows on multiple scales are normalized to the original scale for comparability.

In this embodiment, before a face is detected in a user environment image based on a deep neural network, a face detection network MTCNN for face detection needs to be trained. Further, training of MTCNN includes: pre-training the MTCNN by using an open-source face data set so as to pre-train weights in the MTCNN; and retraining the MTCNN by using a pre-acquired directional face data set so as to perform fine-tune training on weights in the MTCNN, so that the MTCNN can better detect face images similar to face type distribution of the directional face data set. Face types, including but not limited to: the age layer of the face, the sex of the face and the skin color of the face.

Open source face datasets, including but not limited to: VGG-Face, FDDB. The open source data set is characterized by strong face universality, but lack of accuracy, and faces of various species are included, wherein the faces of white people are taken as the main materials. The directional face data set is a face image of a preset face type acquired according to characteristics of an application scene, for example: the images in the directed face dataset are dominated by the faces of the yellow race people.

Whether pre-training or fine-tuning training is performed, face images of a face data set (an open source face data set and a directional face data set) are input into the MTCNN, so that the MTCNN detects faces in the face images, the detection result is compared with a result marked in advance for the face images, if the detection result of the MTCNN is the same as the result marked in advance for the face images, the fact that the trained MTCNN classifies samples (face images) correctly (namely, the recognition is accurate) is indicated, and when the recognition accuracy of the MTCNN is not improved any more, the MTCNN is considered to be converged. Recognition accuracy = number of recognition accuracy ++number of recognition errors.

After the MTCNN converges, the MTCNN can perform face detection on the user environment image after the image enhancement.

The user environment image is input into the MTCNN which has been trained. The user environment image input to the MTCNN network may or may not contain a face. When the user environment image does not contain a human face, the result output by the MTCNN network is null; when the user environment image contains a face, the MTCNN network outputs the user environment image containing a face detection box (face region is identified). When a face appears in the user environment image, the face is framed by a face detection frame. When a plurality of faces appear in the user environment image is detected, each face is framed by a face detection frame.

If a face is detected in the user environment image, and the average brightness value of the user environment image is within the brightness value range, the user environment image is determined to be an effective face image, and then the attitude angle of the target face in the effective face image can be determined.

As shown in fig. 4, a flowchart of the steps for determining the attitude angle according to an embodiment of the present invention is shown.

In step S410, in the effective face image, a mark point is marked for the target face.

The pose of a face includes a pitch angle (pitch angle) of a low head-up of the face in a three-dimensional space, a yaw angle (yaw angle) of the face toward the left or right, and an angle (rotation angle) of whether the face rotates counterclockwise or clockwise in a plane. To complete the estimation of the attitude angle of the target face, the more and more fine the mark points are, the more accurate the estimated attitude angle is depending on the mark points of each part of the target face.

In this embodiment, when determining the pose angle of the target face, the 5-point mark point model may be used to mark the target face in the effective face image based on 5 mark points of the output of MTCNN, or based on the 5-point mark point model used in the open-source machine learning library (dlib). Of course, to improve the accuracy of pose estimation, a 68-point mark point model in dlib, that is, 68 mark points on the target face, may also be used.

Step S420, a preset three-dimensional human body head portrait model is obtained; wherein, the mark points are marked on the face of the three-dimensional human body head model, and the number of the mark points marked on the face of the three-dimensional human body head model and the number of the mark points marked on the target human face are the same as the types in the same dimension space.

The type of the mark point may represent the position of the mark point on the face. For example: the marked point located at the eyebrow may represent the point between the eyebrows.

The types of the mark points marked on the face of the three-dimensional human body head model and the mark points marked on the target human face in the same dimension space are the same, and the mark points are: after the marking points of the target face are converted into the three-dimensional space, the marking points of the target face are the same as the marking points of the face of the three-dimensional human body head portrait model; or after the mark points of the face of the three-dimensional human body head portrait model are converted into the two-dimensional space, the mark points of the face of the three-dimensional human body head portrait model and the mark points of the target human face are the same in type. Thus, each marking point marked on the target human face has a corresponding marking point at a corresponding position of the face of the three-dimensional human body head portrait model.

If the face of the three-dimensional human body head portrait model is marked with 5 marking points, the target human face can be marked with 5 marking points; if 68 mark points are marked on the face of the three-dimensional human body head model, 68 mark points are marked on the target human face.

Step S430, determining the attitude angle of the target face according to the mark points in the three-dimensional human head portrait model and the mark points aiming at the target face in the effective face image.

The three-dimensional human body head portrait model is rotated in three directions, so that N marking points of the target human face are overlapped (or approximately overlapped) with N marking points in the three-dimensional human body head portrait model, and the gesture of the three-dimensional human body head portrait model is the gesture of the target human face.

In this way, the pose angle estimation problem of the target face can be converted into the following optimization problem:

assuming that the attitude angles of the three-dimensional human body head portrait model are (theta, phi and phi), the pitch angle, the deflection angle and the rotation angle are correspondingly and sequentially arranged. As shown in fig. 5, in the case where the camera (camera) parameters are fixed, the rotation matrix R and the translation vector t in the world coordinate system down to the camera coordinate system are solved. The world coordinate system is a three-dimensional coordinate system where the three-dimensional human head portrait model is located, and the camera coordinate system is a plane coordinate system (two-dimensional coordinate system) where the target face is located in the effective face image.

After the rotation matrix R and the translation vector t are obtained, the rotation matrix R and the translation vector t are subjected to Euler angle conversion, and the pitch angle, the deflection angle and the rotation angle of the target face are obtained.

Specifically, after marking N mark points on the target face, each mark point on the target face is a projection point of one mark point of the face of the three-dimensional human body head portrait model. The three-dimensional coordinates of the marking point P of the face of the three-dimensional human body head portrait model are P _i The imaging coordinate (two-dimensional coordinate) of the mark point P on the plane of the target face is f (P _i The method comprises the steps of carrying out a first treatment on the surface of the R, t), the two-dimensional coordinates of the true projection point p are p _i In order to find the rotation matrix R and the translation vector t, only the following minimum projection mean square error problem needs to be solved.

Wherein, the expression of the minimum projection mean square error can be:

thus, the minimum projection mean square error can be approximately solved by a Levenberg-Marquardt optimization method, which is based on the idea that: and (3) minutely adjusting the three-dimensional human body head portrait model to obtain the coordinates of the projection of the mark points on the three-dimensional human body head portrait model on an image plane (the plane of the target human face) until the projection mean square error reaches a minimum value. In practical engineering application, firstly, a coordinate set of a mark point on the face of a three-dimensional human body head portrait model on an image plane is obtained through a standard camera, then, internal parameters (initial R and t) of the camera and the focal length of the camera are calibrated, and the pose estimation of a target human face can be completed by using functions such as open-source computer vision library OpenCV to call a solvePnP.

After the attitude angle of the target face is obtained, comparing the attitude angle of the target face with a preset attitude angle range, if the attitude angle of the target face is within the preset attitude angle range, considering the target face to be effective, cutting the target face in the effective face image, and only reserving the face area of the target face to obtain the face image of the target face.

In this embodiment, before extracting the tooth area image of the target face, a face alignment operation is performed on the target face. The face alignment operation includes: the compensation of the pose angle is performed by affine transformation so that the face is transformed into a front face or an approximate front face, which operations are called face alignment.

In this embodiment, before extracting the dental region image of the target face, validity verification may be performed on the dental region, where the validity verification is used to verify the open and closed states of the mouth of the target face in the valid face image, because in order to better evaluate the health level of the teeth, the teeth must be displayed in the valid face image as much as possible, and if the mouth of the user is in a closed state, no information of the teeth will be extracted in the dental region image, so that subsequent evaluation cannot be performed.

Fig. 6 is a flowchart of verifying the effectiveness of an opening and closing angle of an oral cavity according to an embodiment of the present invention.

In step S610, in the effective face image, a mark point is marked for the target face.

The step of marking the face region of the target face with marking points is similar to the step of marking the face region of the target face with marking points when determining the pose angle, but in order to better mark the structural information of the target face, the model used in this embodiment is 68 marking point models in dlib, and these 68 marking points can outline each part of the target face, for example, these 68 marking points can outline eyebrows, eyes, nose, mouth and face contours.

Step S620, determining an opening and closing angle of the oral cavity of the target face according to the marking points used for marking the oral cavity area in the marking points of the target face.

A marking point for marking an area of an oral cavity, comprising: marking points corresponding to the upper lip and the lower lip.

The oral cavity opening and closing angle comprises: the opening and closing angle of the left mouth angle and the opening and closing angle of the right mouth angle.

The judgment of the opening and closing states of the mouth depends on the positioning of the mouth in the front face mark point. The above-mentioned mark points of the human face 68 points mark the upper and lower lips, including the mark points of the upper and lower lips, and the open and closed states of the mouth are detected based on these mark points, the detection process is as follows: the mark points 61, 62, 63 on the lower edge of the upper lip and the mark points 67, 66, 65 on the upper edge of the lower lip are extracted, and the two mark points 48, 60 on the left mouth corner and the two mark points 54, 64 on the right mouth corner are extracted, and the opening angle of the mouth is calculated from the figures formed by these mark points 61, 62, 63, 67, 66, 65, 48, 60, 54, and 64.

Further, considering that there may be a deviation in the positions of the mark points, in order to improve robustness, the mean coordinate point a 'of the mark points 61, 62, 63, the mean coordinate point B' of the mark points 67, 66, 65, and the mean coordinate point C 'of the mark points 48, 60, the mean coordinate point D' of the mark points 54, 64 are calculated, and the angle of opening of the left mouth angle a 'C' B 'and the angle of opening of the right mouth angle a' C 'D' are calculated, respectively. The mean coordinate refers to a coordinate average value of a plurality of coordinates. The abscissa of the mean coordinate point is the abscissa mean value of the plurality of coordinate points, and the ordinate is the ordinate mean value of the plurality of coordinate points.

Step S630, judging whether the opening and closing angle of the oral cavity is larger than a preset opening and closing angle threshold value; if yes, go to step S640; if not, step S650 is performed.

Judging whether the opening angle of the left mouth angle and the opening angle of the right mouth angle are larger than a preset opening and closing angle threshold value or not; if both are greater than the opening and closing angle threshold, step S640 is performed, otherwise step S650 is performed.

The opening and closing angle threshold may be an empirical value or a value obtained through experimentation. The opening and closing angle of the oral cavity is larger than the opening and closing angle threshold value, so that teeth in the oral cavity can be well displayed. The opening and closing angle threshold value can be obtained through statistical analysis by collecting images of the mouth opening face. In this embodiment, the opening and closing angle threshold may be 25 degrees. Only when the mouth is opened at an angle of at least 25 degrees, it is meaningful to conduct dental health analysis through the face. Namely: the extracted tooth region images in the face are considered to be valid, with angle a ' C ' B ' >25 and + ' a ' C ' D ' > 25.

Step S640, if the opening and closing angle of the oral cavity is greater than a preset opening and closing angle threshold, extracting a tooth area image of the target face from the effective face image.

And extracting the tooth area image of the target face from the effective face image by using an image segmentation algorithm of preset supervision information. The specific extraction method will be described in detail later.

Step S650, if the opening and closing angle of the oral cavity is smaller than or equal to the preset opening and closing angle threshold, performing effective facial image reacquisition prompt.

The step of extracting the image of the tooth region is further described below.

Fig. 7 is a flowchart illustrating the steps for extracting an image of a dental region according to an embodiment of the present invention.

Step S710, preset marking points for positioning a non-tooth area, a candidate non-tooth area, a tooth area and a candidate tooth area in a face; and taking the non-tooth area, the candidate non-tooth area, the tooth area and the candidate tooth area with the marking points positioned as parameters in a preset image segmentation algorithm.

Non-dental areas refer to areas that are not necessarily teeth.

Candidate non-dental regions refer to regions that may not be teeth.

The tooth region is a region which is necessarily a tooth.

Candidate tooth regions refer to regions that may be teeth. Further, the candidate tooth region may include a tooth region.

Image segmentation algorithms, including but not limited to: grabCut algorithm.

Based on the mark points of the target face, areas that may be teeth, areas that may not be teeth, areas that are not teeth, and areas that are not teeth may be marked approximately. And determining the marking points for marking the non-tooth area, the candidate non-tooth area, the tooth area and the candidate tooth area in the marking points of the target face, taking the determined marking points as supervision information, namely parameters in an image segmentation algorithm, and enabling the segmented tooth area image to be more accurate through the supervision information.

In particular, in order to reduce the influence of the mark point position deviation on the tooth region image extraction as much as possible, the range of the initial region setting may be made larger. The peripheral rectangular region of the marker points numbered from 48 to 67 is extracted as a candidate tooth region, that is: extracting the coordinates of the mark points 48 to 67, determining the coordinates (x _min ，y _min ) And the coordinates (x) _max ，y _max ) This is exactly the coordinates of the two anchor points that make up the peripheral rectangle; will be calculated from the abscissa and ordinate minimum (x _min ，y _min ) And maximum value (x _max ，y _max ) The peripheral rectangle representing the localization is determined as the region PR FGD that is likely to be a tooth.

The area outside the polygon formed by the mark points numbered 60 to 67 is determined as the area pr_bgd that may not be a tooth. Regions that are not necessarily teeth are extracted from pr_bgd.

The areas where the mark points numbered 1, 28, 29, and 2 are formed are areas BGD which are not necessarily teeth.

The pr_fgd is extracted from a region that is necessarily a tooth, and the connecting line that marks the midpoints of the dot pairs (61,67), (62, 66), (63, 65) is regarded as a region that is a region FGD that is necessarily a tooth.

The regions PR_FGD and PR_ BGD, BGD, FGD marked by the marking points are taken as parameters into an image segmentation algorithm, so that the image segmentation algorithm can obtain a preliminary tooth region image.

Step S720, performing image segmentation processing on the effective face image by using the image segmentation algorithm, so as to obtain an initial tooth region image.

The initial dental region image refers to a candidate dental region, i.e. a region image of a tooth.

In this embodiment, the image segmentation algorithm may employ a semi-supervised based GrabCut algorithm.

In particular, the GrabCut algorithm is generally interactive so as to add part of the supervision information to the GrabCut algorithm, which may be region division information, for example: a region that may be a foreground, a region that may be a background, a foreground region, or a background region; if no supervision information is specified, the foreground and background division method is used by default based on unsupervised graph segmentation, the graph segmentation is carried out by constructing a similarity graph between areas, and then clustering analysis is carried out on the areas based on spectral clustering, so that the foreground and the background are separated. In this embodiment, the marked points assist the positioning of the region accurately.

Further, the initial dental region image segmented by the marker points may have certain problems, such as: in order to avoid deviation of the segmentation result of the single tooth, the embodiment can adjust the initially segmented tooth region, fill the possible holes to a certain extent, and fill the holes in the initial tooth region image by using a hole filling algorithm in morphological operation to obtain the initial tooth region image.

In step S730, in the initial tooth area image, the pixel points with the color space in the preset tooth color space range are screened out, and the tooth area image is formed according to the screened pixel points.

The tooth region image is an image of a tooth region of a target face. Specifically, the dental region image is a collection of dental pixels of the target face.

The teeth pixels are screened according to the color space range of the teeth, so that the area which is really the teeth, namely the teeth area image, can not be regarded as teeth by lips, other areas in the oral cavity after mouth opening, such as gums, tongue and the like.

In general, the color of the teeth has a distinct characteristic compared to other areas inside the mouth, especially in the HSV (Hue, saturation, value, hue, saturation and brightness) color space, where the hue of the teeth does not fall in the red or purple areas. The general red or purple tone is in the range of [0,8] and [158,180], each pixel point in the initial tooth area image is screened through the tone, and the red or purple pixel points converted into HSV space are eliminated, so that a set of tooth pixel points is obtained.

After the tooth area image is obtained, an image of each tooth may be segmented in the tooth area image. Further, fine segmentation of the teeth is performed according to the screened tooth pixel set, namely, a segmentation line between the teeth is found out, and the area where each tooth is located is obtained.

In this embodiment, the tooth-to-tooth segmented edges can be obtained according to an adaptive edge extraction algorithm, and the obtained edges are numerous and disordered, which is most likely to calculate the brightness change of the same tooth as the edges, and the edges are short, small and disordered. Thus, the edge screening can be completed through heuristic processing flow, and the fine segmentation result of the teeth is finally obtained.

As shown in FIG. 8, a flowchart of steps for segmentation of a dental region image is provided according to one embodiment of the present invention.

Step S810, extracting edge lines from the tooth area image according to a preset edge extraction algorithm.

In step S820, the length of each of the extracted edge lines and the average length of all the edge lines are calculated.

Calculating the length L of each edge line _e The method comprises the steps of carrying out a first treatment on the surface of the Counting the average length of all edge lines

In step S830, a length threshold and a distance threshold are determined according to the average lengths of all edge lines.

The length threshold may be a preset first proportion of the average length. For example: length threshold value of

The distance threshold may be a preset second proportion of the average length. For example: distance threshold

Step S840, for each edge line, determining the edge line as a tooth dividing line if the length of the edge line is greater than the length threshold and the distances between two end points of the edge line and the nearest line segment are respectively smaller than the distance threshold.

In this embodiment, the reserved length is greater than the length thresholdAnd completely unconnected isolated edge lines (the starting and ending positions of the edge lines are further from the surrounding edge, at a distance of +.>Above) other edge lines are deleted.

The line segment closest to one end point of the edge line and the line segment closest to the other end point of the edge line may be different line segments, and separate judgment is required.

Step S850, extracting a contour image of each candidate tooth in the tooth region image according to the tooth dividing line.

The candidate teeth refer to suspected teeth in the image of the tooth region.

Connecting the edges of teeth which are closer to each other through morphological closing operation; and finally, extracting the contour of each candidate tooth based on the contour of the tooth.

Step S860, for each candidate tooth profile image, matching the candidate tooth profile image with a plurality of preset tooth shapes, respectively, if there is a tooth shape matching the candidate tooth profile image, determining the candidate tooth as a tooth, and dividing the tooth image according to the candidate tooth profile image.

The teeth of a person often have a certain shape, and through the preset tooth shape, the teeth which are not in line with each other can be screened out, and finally the segmentation result of each tooth is obtained.

Further, a tooth shape is provided for the teeth at each position in the oral cavity. The contour image of each candidate tooth is sequentially matched with the tooth shape of the tooth at each position, and if the contour image of the candidate tooth matches the tooth shape of the tooth at one position, the candidate tooth can be determined as the tooth at the position, and the contour image of the candidate tooth is the tooth image of the tooth.

After the tooth segmentation is completed, a tooth health grade assessment may be performed. The degree of alignment and yellowing of the user's teeth is evaluated based on the segmented tooth regions. Firstly, judging whether a tooth has a clear midline (the midline generally passes through a gap between two incisors), and judging whether the left half part and the right half part of the tooth positioned at the two sides of the midline have better contact ratio after being overturned, wherein the contact ratio reflects the integral uniformity of the tooth; secondly, whether each tooth has a good shape fit with the teeth on the standard tooth model reflects the uniformity of each tooth; again, the degree of yellowing of each tooth reflects whether or not there are dirty problems such as calculus.

As shown in FIG. 9, a flowchart of the steps for tooth health level assessment according to one embodiment of the present invention is shown.

Step S910, determining a tooth area center line according to the mark points of the target face; and determining the left-right contact ratio of the teeth according to the center line of the tooth area and the tooth image corresponding to each tooth segmented in the tooth area image.

The tooth region center line refers to a line dividing the tooth region into left and right parts in the tooth region image.

Determining a tooth region midline from the marked points marked on the target face, comprising: a midpoint connecting line of the marking point pair (21, 22), (31, 35) is determined, the connecting line extends downwards to the tooth area, the tooth area is divided into a left part and a right part, and the connecting line is taken as a tooth area central line.

The left-right overlap ratio of the teeth is obtained based on the overlap ratio after the two sides of the midline are overturned. Let the tooth area of the left part be T _L The area T is turned right by the center line _LR The method comprises the steps of carrying out a first treatment on the surface of the The right half of the tooth area is T _R The area T is turned left by the midline _RL Degree of coincidence S between left and right teeth ₀ ∈[0,1]The method can be calculated by the following formula:

here, area (T) represents the area of the region T, and the area of the region T may be replaced with the number of pixels included in the region T, and T represents a region variable.

Step S920, determining the overall uniformity of the teeth according to the preset standard model of each tooth and the tooth image corresponding to each tooth segmented in the tooth area image.

Determining the uniformity of each tooth according to a preset standard model of each tooth and a tooth image corresponding to each tooth segmented in the tooth area image; and determining the overall uniformity of the teeth according to the uniformity of all the teeth in the teeth area image.

Determining the uniformity of each tooth, comprising: and determining the shape similarity of each tooth in the tooth area image and the standard model of the tooth at the corresponding position, and taking the shape similarity as the uniformity of the tooth. Further, for teeth in the image of the tooth regionNormalizing the tooth shape to ensure that the size of a standard model of the tooth is consistent with the size of the tooth in the tooth area image in scale, wherein the specific method comprises the following steps of; for each tooth in the tooth area image, determining the center point of the tooth according to the tooth image of the tooth, and calculating the distance between the center point of the tooth and each point (pixel point) at the edge of the tooth; in each distance, acquiring the median of the distances as a reference distance; determining a standard model of the corresponding position of the tooth; determining a center point of the standard model; calculating the distance from the center point of the standard model to each pixel point at the edge of the standard model, and determining the median of the distance; the center point of the tooth is overlapped with the center point of the standard model, so that the median of the tooth is equal to the median of the standard model; then, calculating the uniformity R of the single tooth according to a preset single tooth uniformity formula _Ti 。

Wherein the area of tooth i is TO _i The area of the standard tooth corresponding to the tooth i is T _Si 。

Assuming that m teeth are in the teeth area image, the teeth overall uniformity R can be calculated according to a preset teeth overall uniformity calculation formula ₁ 。

In this embodiment, the concept of the median is used because the user's teeth may have defects, and in order to better evaluate the problem of tooth uniformity due to defects, a median distance insensitive to tooth defects is selected as the normalization scale.

Step S930, determining overall yellowness of the teeth according to the tooth image corresponding to each tooth segmented in the tooth area image.

Overall tooth yellowing is used to measure overall tooth yellowing. The degree of yellowing of the teeth can be determined according to a standard color chart. The standard color chart may be a 20 color tooth color chart. A level 1 indicates the whitest teeth and a level 20 indicates the yellow teeth.

The color of each tooth in the tooth area image is calculated through similarity with the color in the standard color card, so that the yellowing degree Y' epsilon {1,2, …,20} of the individual tooth is determined.

After determining the yellowness of each tooth, the average of the yellowness of all teeth in the image of the tooth area may be calculated and taken as the overall yellowness of the tooth.

In order to improve the accuracy of grading the degree of yellowing of teeth, the color of the teeth can be simply calibrated by using the color of a preset reference object, because the color of the teeth is greatly influenced by ambient light, and based on the constancy of the color in perception (namely, when the color of an object changes due to the change of illumination conditions, the perception of the color of an individual to a familiar object still tends to be consistent), the original color of the teeth can be perceived and analyzed by naked eyes, while the image shot by an image system does not have the characteristic, and the sensor of the image system can faithfully record the absolute color value of the influence of the environment on the teeth. For example: in a warm light environment, the color of the teeth in the image is more yellow than the actual color, and in other environments, the color of the teeth in the image is different, which is not beneficial to the judgment of the overall yellowing degree of the teeth.

Extracting a color average value of a preset reference object from the effective face image, and determining a superposition color value of the environment to the teeth according to the color average value of the reference object; after determining the average value of the tooth color according to the tooth image, subtracting the superimposed color value from the average value of the tooth color to obtain the overall yellowing degree of the tooth. Further, the reference may be an eye white region in the effective face image because the average color of the eye white under natural conditions is fixed.

In this embodiment, the white region may be extracted according to the mark point of the target face, and the average value of the color of the white region may be determined. Further, the area surrounded by the mark points 36 to 41 may be referred to as a left eye area, and the area surrounded by the mark points 42 to 47 may be referred to as a right eye area.

Because the brightness of the eye white is obviously different from the brightness of other areas, the area with the eye white can be extracted from the effective face image according to the brightness range of the eye white; from the area of the eye white, the color average of the eye white is calculated. Presetting the average value of the color of the eye white of a normal person (without any eye diseases) in a standard environment (such as a natural light environment), and obtaining the difference value of the average value of the color of the extracted eye white and the average value of the color of the eye white in the standard environment as a superimposed color value based on the constancy of the color.

Further, in calculating the superimposed color values, since the color space exhibited by the RGB (red green blue) color representation is not uniform, i.e., the distance reflected by the color cannot be calculated simply by the euclidean distance of two points in the RGB color space. Thus, the RGB space with non-uniform colors can be converted into a uniform XYZ color space, and the XYZ color space is expressed in relation to the color development device, so that another Lab color space with non-uniform colors is often used instead in actual calculation, and the difference between colors can be obtained through subtraction of color vectors.

The process of calculating the superimposed color values is as follows:

the average RGB value (color average value) of the eye white under the standard environment is assumed to be [ R0, G0, B0], and the color average value of the eye white in the effective face image is assumed to be [ R1, G1, B1]. The conversion formula and the inverse conversion formula from the RGB space to the Lab space are shown as formula (1) and formula (2), respectively, the average color value of the eye white in the Lab space under the standard environment is [ L0, a0, b0] and the average value of the eye white in the Lab space in the effective face image is [ L1, a1, b1] can be obtained through the formula (1).

/>

The difference between Lab spaces is [ L1-L0, a1-a0, b1-b0]. Assuming that the average value of the color of the tooth image of the single tooth extracted from the effective face image is [ RT, GT, BT ] and the value obtained by transforming the color image into the Lab space is [ LT, aT, BT ], then the value of the overall tooth yellowing degree in the Lab space is [ LT-l1+l0, aT-a1+a0, BT-b1+b0] in the standard environment, and then the overall tooth yellowing degree of the tooth image in the RGB space in the standard environment can be obtained based on the formula (2).

Step S940, determining the tooth health level of the target face according to the center line roll-over contact ratio, the overall tooth uniformity and the overall tooth yellowing.

The tooth health grade S can be found by referring to the following formula:

Wherein, the super parameters alpha and beta are weight factors of overall tooth yellowing degree and overall tooth uniformity respectively. In practical application, the values of alpha and beta can be adjusted according to the actual scene.

Generally, teeth are yellowish (e.g., Y'>16 Representing the possible presence of insufficient cleaning of the teeth, the presence of some plaque and calculus, improved by rinsing, whitening, etc., and for irregularities of the teeth (e.g. S ₀ <0.8 Even with a few deletions (e.g. S) ₁ <0.8 If the result is that the teeth are not good, the operation operations such as correction of tooth deformity and tooth filling are required. Generally, α may be between 0.1 and 0.3, and β may be between 0.3 and 0.5, for example: let α=0.2, β=0.4.

In the embodiment, the method can be implemented by equipment comprising a camera without medical instruments, the method is convenient for collecting effective face images, the time consumption for tooth health assessment is short, the cost is low, the image processing technology used in the whole process is simple, the face images are preprocessed, the mouth opening and closing detection is carried out, the teeth are segmented and the tooth health assessment is carried out, the tooth health assessment process is simple and easy, and the result has good interpretability.

Aiming at common problems of tooth health, such as uneven teeth, missing teeth or unclean teeth, the embodiment can simply evaluate the tooth health of a user through a mobile phone or a computer camera, and treat the problems facing the tooth health of the user according to the tooth health grade in a distinguishing way: for users with lighter dental health problems, online suggestions for dental health are given; and for some users with more dental problems, the related stomatologists or hospitals are recommended for professional follow-up treatment.

The embodiment provides tooth health assessment equipment based on face structure positioning. Referring to fig. 10, a structure diagram of a tooth health assessment apparatus based on face structure localization according to an embodiment of the present invention is shown.

In this embodiment, the tooth health assessment device based on face structure positioning includes, but is not limited to: a processor 1010, and a memory 1020.

The processor 1010 is configured to execute a dental health evaluation program based on facial structure localization stored in the memory 1020 to implement the above-mentioned dental health evaluation method based on facial structure localization.

Specifically, the processor 1010 is configured to execute a dental health assessment program stored in the memory 1020 and based on facial structure localization, so as to implement the following steps: collecting an effective face image; determining the attitude angle of a target face in the effective face image; if the attitude angle of the target face is within the preset attitude angle range, extracting a tooth area image of the target face from the effective face image; dividing a tooth image corresponding to each tooth in the tooth area image; and determining the tooth health grade of the target face according to the tooth image corresponding to each tooth.

Wherein segmenting each tooth in the tooth region image comprises: extracting edge lines from the tooth region image according to a preset edge extraction algorithm; calculating the length of each edge line and the average length of all edge lines; determining a length threshold and a distance threshold according to the average lengths of all the edge lines; for each edge line, if the length of the edge line is greater than the length threshold and the distance between two endpoints of the edge line and the nearest line segment is less than the distance threshold, determining the edge line as a tooth dividing line; extracting contour images of each candidate tooth in the tooth region image according to the tooth dividing line; and respectively matching the contour image of each candidate tooth with a plurality of preset tooth shapes, if the tooth shapes matched with the contour image of the candidate tooth exist, determining the candidate tooth as a tooth, and dividing the tooth image according to the contour image of the candidate tooth.

The embodiment of the invention also provides a storage medium. The storage medium here stores one or more programs. Wherein the storage medium may comprise volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid state disk; the memory may also comprise a combination of the above types of memories.

The one or more programs, when executed by the one or more processors, implement the facial structure localization-based dental health assessment method described above.

Specifically, the processor is configured to execute a dental health evaluation program stored in the memory and based on facial structure localization, so as to implement the following steps: collecting an effective face image; determining the attitude angle of a target face in the effective face image; if the attitude angle of the target face is within the preset attitude angle range, extracting a tooth area image of the target face from the effective face image; dividing a tooth image corresponding to each tooth in the tooth area image; and determining the tooth health grade of the target face according to the tooth image corresponding to each tooth.

The above description is only an example of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for assessing dental health based on facial structure localization, comprising:

collecting an effective face image;

determining the attitude angle of a target face in the effective face image;

if the attitude angle of the target face is within the preset attitude angle range, extracting a tooth area image of the target face from the effective face image; wherein before extracting the tooth area image of the target face in the effective face image, the method further comprises: marking the effective face image by marking points aiming at the target face; determining the opening and closing angle of the oral cavity of the target face according to the marking points used for marking the oral cavity area in the marking points of the target face; if the opening and closing angle of the oral cavity is larger than a preset opening and closing angle threshold, extracting a tooth area image of the target face from the effective face image; otherwise, performing effective face image re-acquisition prompt; extracting the tooth area image of the target face from the effective face image comprises the following steps: marking points respectively used for positioning a non-tooth area, a candidate non-tooth area, a tooth area and a candidate tooth area in a face are preset; the non-tooth area, the candidate non-tooth area, the tooth area and the candidate tooth area which are positioned by the marking points are used as parameters in a preset image segmentation algorithm; performing image segmentation processing on the effective face image through the image segmentation algorithm to obtain an initial tooth region image; screening out pixel points with color space in the range of the preset tooth color space from the initial tooth area image, and forming a tooth area image according to the screened pixel points;

Dividing a tooth image corresponding to each tooth in the tooth area image;

and determining the tooth health grade of the target face according to the tooth image corresponding to each tooth.

2. The method of claim 1, wherein the acquiring valid face images comprises:

collecting a user environment image;

determining an average brightness value of the user environment image;

if the average brightness value of the user environment image is within a preset brightness value range, face detection is carried out on the user environment image;

if a face is detected in the user environment image, determining that the user environment image is a valid face image;

and if the average brightness value of the user environment image is not within the brightness value range, or no human face is detected in the user environment image, carrying out user environment image re-acquisition prompt.

3. The method of claim 2, further comprising, prior to said face detection for said user environment image:

determining an image brightness standard deviation of the user environment image;

and if the image brightness standard deviation is smaller than a preset image brightness standard deviation threshold, performing image enhancement processing on the user environment image by using a gamma conversion algorithm.

4. The method of claim 1, wherein the determining the pose angle of the target face in the valid face image comprises:

marking the effective face image by marking points aiming at the target face;

acquiring a preset three-dimensional human body head portrait model; wherein, the face of the three-dimensional human body head portrait model is marked with mark points, and the number of the mark points marked on the face of the three-dimensional human body head portrait model and the number of the mark points marked on the target human face are the same as the types in the same dimension space;

and determining the attitude angle of the target face according to the mark points in the three-dimensional human body head portrait model and the mark points aiming at the target face in the effective face image.

5. The method of claim 1, wherein segmenting the tooth image corresponding to each tooth in the tooth region image comprises:

extracting edge lines from the tooth region image according to a preset edge extraction algorithm;

calculating the length of each edge line and the average length of all edge lines;

determining a length threshold and a distance threshold according to the average lengths of all the edge lines;

For each edge line, if the length of the edge line is greater than the length threshold and the distance between two endpoints of the edge line and the nearest line segment is less than the distance threshold, determining the edge line as a tooth dividing line;

extracting contour images of each candidate tooth in the tooth region image according to the tooth dividing line;

and respectively matching the contour image of each candidate tooth with a plurality of preset tooth shapes, if the tooth shapes matched with the contour image of the candidate tooth exist, determining the candidate tooth as a tooth, and dividing the tooth image according to the contour image of the candidate tooth.

6. The method of claim 5, wherein determining the dental health level of the target face from the dental image corresponding to each of the teeth comprises:

determining a tooth area center line according to the mark points of the target face; determining the left-right contact ratio of the teeth according to the tooth region center line and the tooth image corresponding to each tooth segmented in the tooth region image;

Determining the overall uniformity of the teeth according to a preset standard model of each tooth and a tooth image corresponding to each tooth segmented in the tooth area image;

determining the overall yellowing degree of the teeth according to the tooth images corresponding to each tooth which are segmented in the tooth area images;

and determining the tooth health grade of the target face according to the left-right contact ratio of the teeth, the overall uniformity of the teeth and the overall yellowing degree of the teeth.

7. A tooth health assessment device based on face structure positioning, which is characterized by comprising a processor and a memory; the processor is configured to execute a dental health evaluation program based on facial structure localization stored in the memory, so as to implement the dental health evaluation method based on facial structure localization according to any one of claims 1 to 6.

8. A storage medium storing one or more programs executable by one or more processors to implement the face structure localization-based dental health assessment method of any one of claims 1-6.