CN112163552A - Labeling method and device for key points of nose, electronic equipment and storage medium - Google Patents

Labeling method and device for key points of nose, electronic equipment and storage medium Download PDF

Info

Publication number
CN112163552A
CN112163552A CN202011100012.3A CN202011100012A CN112163552A CN 112163552 A CN112163552 A CN 112163552A CN 202011100012 A CN202011100012 A CN 202011100012A CN 112163552 A CN112163552 A CN 112163552A
Authority
CN
China
Prior art keywords
key point
point
key
nose
labeling
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011100012.3A
Other languages
Chinese (zh)
Inventor
张博宁
张国鑫
马里千
刘晓强
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Dajia Internet Information Technology Co Ltd
Original Assignee
Beijing Dajia Internet Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Dajia Internet Information Technology Co Ltd filed Critical Beijing Dajia Internet Information Technology Co Ltd
Priority to CN202011100012.3A priority Critical patent/CN112163552A/en
Publication of CN112163552A publication Critical patent/CN112163552A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/005—General purpose rendering architectures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Computer Graphics (AREA)
  • Processing Or Creating Images (AREA)
  • Image Analysis (AREA)

Abstract

The disclosure relates to a labeling method and device for key points of a nose, electronic equipment and a storage medium. The method comprises the steps of obtaining a rendering image of a face generated based on rendering data, wherein the rendering image comprises key points to be marked on a nose, and the rendering data comprises three-dimensional positions of the key points and a plurality of preset areas of the nose; responding to the fact that the projection position of the three-dimensional position of the key point projected to the two-dimensional image is invisible in the rendered image, and obtaining marking parameters, wherein the marking parameters at least comprise the position of a reference outline of the key point, the reference outline is an outline of a preset area corresponding to the key point, and the preset area corresponding to the key point is a preset area with the probability of the key point being larger than a first threshold value; and marking the positions of the key points in the rendered image according to the marking parameters. Therefore, the method can automatically label the positions of the key points in the rendered image, and can obtain a uniform labeling result for the same key point due to the objectivity of the parameters, thereby being beneficial to reducing the error of the model obtained by training.

Description

Labeling method and device for key points of nose, electronic equipment and storage medium
Technical Field
The present disclosure relates to the field of image processing technologies, and in particular, to a method and an apparatus for labeling key points of a nose, an electronic device, and a storage medium.
Background
Face keypoint detection is a key step in the field of face recognition and analysis, which is a prerequisite and breakthrough for problems such as automatic face recognition, expression analysis, three-dimensional face reconstruction, and other face-related problems.
The human face key point detection model based on deep learning needs a large amount of human face images and labeling data for training, so at present, a method of manually labeling the key points of the nose is usually adopted to obtain the nasal labeling data, but the manual labeling method has high labor cost, and the labeling results of different labeling personnel on the same key point may not be uniform, so that the error of the trained model is large.
Disclosure of Invention
The disclosure provides a labeling method and device for key points of a nose, electronic equipment and a storage medium, which are used for at least solving the problems that the existing labeling method for key points of the nose is high in labor cost and large in error of a trained model due to the fact that labeling results are not uniform. The technical scheme of the disclosure is as follows:
according to a first aspect of the embodiments of the present disclosure, a method for labeling a nose key point is provided, including:
obtaining a rendering image of a face, wherein the rendering image comprises key points to be marked on a nose, the rendering image is generated based on rendering data, and the rendering data comprises three-dimensional positions of the key points and a plurality of preset areas of the nose;
responding to that the projection position of the key point is invisible in the rendered image, acquiring a labeling parameter, wherein the labeling parameter at least comprises the position of a reference outline of the key point, the reference outline is the outline of the preset area corresponding to the key point, the preset area corresponding to the key point is the preset area with the probability of the key point being greater than a first threshold value, and the projection position is the position obtained by projecting the three-dimensional position of the key point to a two-dimensional image space;
and marking the positions of the key points in the rendered image according to the marking parameters.
Optionally, the labeling parameters further include: the reference line is a straight line which passes through the projection position of the key point and has the direction of the axis in the nose or the direction vertical to the axis in the nose;
according to the labeling parameters, labeling the positions of the key points in the rendered image, including:
and marking the position of the key point in the rendered image according to the intersection point of the reference line and the reference outline.
Optionally, a key point is located on the nasal medial axis, and the direction of the reference line of the key point is the direction of the nasal medial axis;
marking the position of the key point in the rendered image according to the intersection point of the reference line and the reference contour, including:
selecting one intersection point from the intersection points of the reference line and the reference contour as the position of the key point according to the position relation of the key point and a target line in the axial direction of the nose; the target line is a straight line perpendicular to the nasal medial axis and passing through a midpoint of the nasal medial axis.
Optionally, the key point is located on the nasal axis;
the direction of the reference line of the key point is the direction of the axis in the nose;
marking the position of the key point in the rendered image according to the intersection point of the reference line and the reference contour, including:
sequentially inquiring whether the reference line and the reference contour have the intersection point or not according to the sequence of the reference contour of the key point from large to small according to the probability until the intersection point is inquired;
and selecting one intersection point from the intersection points according to the position relation of the key point and a target preset area in the axial direction of the nose, wherein the intersection point is marked as the position of the key point in the rendered image, and the target preset area is the preset area with the reference contour of the inquired intersection point.
Optionally, there is a preconfigured associated point for a keypoint, the associated point being symmetric about the nasal axis to the keypoint and not visible in the rendered image;
the direction of the reference line of the key point is the direction of the axis in the nose;
marking the position of the key point in the rendered image according to the intersection point of the reference line and the reference contour, including:
and marking the position of the key point in the rendered image according to the coordinate of the intersection point of the reference line and the reference contour in the axial direction in the nose.
Optionally, there is a preconfigured associated point for the keypoints, the associated point being symmetric about the nasal axis with the keypoints and visible in the rendered image;
the direction of the reference line of the key point is the vertical direction of the axis in the nose;
marking the position of the key point in the rendered image according to the intersection point of the reference line and the reference contour, including:
and selecting one intersection point from the plurality of intersection points of the reference line and the reference contour as the position of the key point according to the position relation of the key point and the axis in the nose in the direction of the reference line.
Optionally, before the labeling, according to the labeling parameter, the position of the keypoint in the rendered image, the method further includes:
obtaining a conversion image of the rendering image by converting the flip angle of the rendering image into zero;
according to the labeling parameters, labeling the positions of the key points in the rendered image, including:
selecting a reference contour corresponding to a pitch angle and a yaw angle of the converted image from the reference contours as a target reference contour; the reference contour corresponding to the pitch angle and the yaw angle of the converted image is a contour of the preset area, wherein the contour comprises the key point with the probability larger than the first threshold value under the pitch angle and the yaw angle;
selecting the position of a pixel point from the target reference contour as the target position of the key point according to the position relation between the key point and the axis in the nose in the target direction; the target direction is a perpendicular direction to the nasal axis;
and converting the target position to a position in the rendering image to obtain the position of the key point in the rendering image.
Optionally, the method further comprises:
in response to the projected locations of the keypoints being visible in the rendered image, labeling the projected locations of the keypoints as the locations of the keypoints in the rendered image.
Optionally, after the labeling the position of the keypoint in the rendered image according to the labeling parameter, the method further includes:
obtaining a contour line between a first key point and a second key point in the marked key points; the first key point and the second key point are the key points which are positioned on the contour line of the nose and have the curvature larger than a second threshold value in the marked key points;
and respectively marking the positions of N equal division points on the contour line as the positions of other key points, wherein N is an integer greater than 1.
Optionally, after the labeling the position of the keypoint in the rendered image according to the labeling parameter, the method further includes:
acquiring a connecting line between a first key point and a second key point in the labeled key points, wherein the first key point and the second key point are the key points positioned on the axis in the nose;
and respectively marking the positions of N equal division points on the connecting line as the positions of other key points, wherein N is an integer greater than 1.
According to a second aspect of the embodiments of the present disclosure, there is provided a nasal key point labeling device, including:
the rendering image obtaining module is used for obtaining a rendering image of a face, the rendering image comprises key points to be marked on a nose, the rendering image is generated based on rendering data, and the rendering data comprises three-dimensional positions of the key points and a plurality of preset areas of the nose;
a labeling parameter obtaining module, configured to obtain a labeling parameter in response to that a projection position of the key point is invisible in the rendered image, where the labeling parameter at least includes a position of a reference contour of the key point, the reference contour is a contour of the preset region corresponding to the key point, the preset region corresponding to the key point is the preset region where a probability of including the key point is greater than a first threshold, and the projection position is a position obtained by projecting a three-dimensional position of the key point to a two-dimensional image space;
and the marking module is used for marking the position of the key point in the rendered image according to the marking parameters.
Optionally, the labeling parameters further include: a reference line which passes through the projection position and is a straight line with the direction of the nasal axis or the direction perpendicular to the nasal axis;
the labeling module is configured to label the position of the keypoint in the rendered image according to the labeling parameter, and includes:
the labeling module is specifically configured to label the position of the key point in the rendered image according to an intersection of the reference line and the reference outline.
Optionally, the key point is located on the nasal middle axis, and the direction of the reference line of the key point is the direction of the nasal middle axis;
the labeling module is configured to label the position of the keypoint in the rendered image according to an intersection of the reference line and the reference outline, and includes:
the labeling module is specifically configured to select one intersection point from the plurality of intersection points of the reference line and the reference contour as the position of the key point according to the position relationship between the key point and the target line in the axial direction of the nose; the target line is a straight line perpendicular to the nasal medial axis and passing through a midpoint of the nasal medial axis.
Optionally, the key point is located on the nasal middle axis, and the direction of the reference line of the key point is the direction of the nasal middle axis;
the labeling module is configured to label the position of the keypoint in the rendered image according to an intersection of the reference line and the reference outline, and includes:
the marking module is specifically configured to sequentially query whether the reference line and the reference contour have the intersection point according to the sequence of the reference contour of the key point until the intersection point is queried; the sequence of the reference outline is obtained by arranging the reference outlines from large to small according to the probability; and selecting one intersection point from the intersection points according to the position relation of the key point and a target preset area in the axial direction of the nose, wherein the intersection point is marked as the position of the key point in the rendered image, and the target preset area is the preset area with the reference contour of the inquired intersection point.
Optionally, there is a preconfigured associated point for the key point, the associated point is symmetric with the key point about the nasal axis, and the associated point is not visible in the rendered image;
the direction of the reference line of the key point is the direction of the axis in the nose;
the labeling module is configured to label the position of the keypoint in the rendered image according to an intersection of the reference line and the reference outline, and includes:
the labeling module is specifically configured to label the position of the key point in the rendered image according to the coordinate of the intersection point of the reference line and the reference contour in the nasal axis direction.
Optionally, there is a preconfigured associated point for the key point, the associated point is symmetric with the key point about the nasal axis, and the associated point is visible in the rendered image;
the direction of the reference line of the key point is the vertical direction of the axis in the nose;
the labeling module is configured to label the position of the keypoint in the rendered image according to an intersection of the reference line and the reference outline, and includes:
the labeling module is specifically configured to select one intersection point from the plurality of intersection points of the reference line and the reference contour as the position of the key point according to the position relationship between the key point and the axis in the nose in the direction of the reference line.
Optionally, the method further includes:
the conversion module is used for converting the turning angle of the rendering image into zero to obtain a conversion image of the rendering image before the position of the key point in the rendering image is marked according to the marking parameter;
the labeling module is configured to label the position of the keypoint in the rendered image according to an intersection of the reference line and the reference outline, and includes:
the labeling module is specifically used for selecting a reference contour corresponding to the pitch angle and the yaw angle of the converted image from the reference contours as a target reference contour; the reference contour corresponding to the pitch angle and the yaw angle of the converted image is a contour of the preset area, wherein the contour comprises the key point with the probability larger than the first threshold value under the pitch angle and the yaw angle; selecting the position of a pixel point from the target reference contour as the target position of the key point according to the position relation between the key point and the axis in the nose in the target direction; the target direction is a perpendicular direction to the nasal axis; and converting the target position to a position in the rendering image to obtain the position of the key point in the rendering image.
Optionally, the labeling module is further configured to:
in response to the projected locations of the keypoints being visible in the rendered image, labeling the projected locations of the keypoints as the locations of the keypoints in the rendered image.
Optionally, the labeling module is further configured to:
after the positions of the key points in the rendered image are marked according to the marking parameters, obtaining a contour line between a first key point and a second key point in the marked key points; the first key point and the second key point are the key points which are positioned on the contour line of the nose and have the curvature larger than a second threshold value in the marked key points; and respectively marking the positions of N equal division points on the contour line as the positions of other key points, wherein N is an integer greater than 1.
Optionally, the labeling module is further configured to:
after the positions of the key points in the rendered image are labeled according to the labeling parameters, acquiring a connecting line between a first key point and a second key point in the labeled key points, wherein the first key point and the second key point are the key points on a nasal medial axis; and respectively marking the positions of N equal division points on the connecting line as the positions of other key points, wherein N is an integer greater than 1.
According to a third aspect of the embodiments of the present disclosure, there is provided an electronic apparatus including:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the method for labeling nose key points disclosed in the first aspect.
According to a fourth aspect of the embodiments of the present disclosure, there is provided a storage medium, wherein instructions of the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the method for labeling nose keypoints disclosed in the first aspect.
According to a fifth aspect of embodiments of the present disclosure, there is provided a computer program product, which, when executed by a processor of an electronic device, enables the electronic device to perform the method of labeling nasal keypoints as described above.
The technical scheme provided by the embodiment of the disclosure at least brings the following beneficial effects:
the labeling method, the labeling device, the electronic equipment and the storage medium of the key points of the nose part are used for obtaining a rendering image of a face generated based on rendering data, wherein the rendering image comprises the key points to be labeled of the nose part, and the rendering data comprises three-dimensional positions of the key points and a plurality of preset areas of the nose part. And responding to the fact that the projection position of the key point is invisible in the rendered image, and obtaining marking parameters, wherein the marking parameters at least comprise the position of a reference outline of the key point, the reference outline is an outline of a preset area corresponding to the key point, and the preset area corresponding to the key point is a preset area with the probability of the key point being larger than a first threshold value. And marking the positions of the key points in the rendered image according to the marking parameters. Therefore, the positions of the key points in the rendered image can be automatically labeled, and the labeling parameters are generated according to objective data and values, so that the influence of subjective factors on the labeling result is avoided, a uniform labeling result can be obtained for the same key point, and the error of the trained model is favorably reduced. In conclusion, the technical scheme provided by the disclosure not only reduces the labor cost, but also avoids the problem of large model errors caused by non-uniform labeling of the same key point by different labeling personnel.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure and are not to be construed as limiting the disclosure.
FIG. 1 is a flow diagram illustrating a method for labeling nose keypoints, according to an exemplary embodiment;
FIG. 2 is a schematic diagram illustrating a contour of a preset region of a nose according to an exemplary embodiment;
FIG. 3 is a schematic diagram illustrating a rendered image including keypoints to be annotated, according to an exemplary embodiment;
FIG. 4 is a flowchart illustrating a specific implementation of a method for labeling nose key points according to an exemplary embodiment;
FIG. 5 is a flowchart illustrating a specific implementation of yet another method for labeling nose key points according to an exemplary embodiment;
FIG. 6 is a block diagram illustrating a nasal keypoint tagging device in accordance with an exemplary embodiment;
FIG. 7 is a block diagram illustrating an electronic device in accordance with an example embodiment.
Detailed Description
In order to make the technical solutions of the present disclosure better understood by those of ordinary skill in the art, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.
It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in sequences other than those illustrated or otherwise described herein. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
An optional application scenario of the present disclosure is to label key points located in a nose in a rendered image of a human face, and the obtained rendered image and label data are used for training a machine learning model.
Fig. 1 is a flowchart illustrating a method for labeling nose key points according to an exemplary embodiment, as shown in fig. 1, including the following steps:
and S101, obtaining a rendering image of the human face.
In the present disclosure, the rendered image of the face includes a key point to be labeled in the nose, and the rendered image is generated based on rendering data. Methods of acquiring rendering data and generating rendered images may be found in the prior art. The key points to be labeled may be manually pre-designated pixel points of some noses commonly used by training models, such as a nose tip point, a left alar point, a right alar point, and the like.
In the present disclosure, rendering data includes: the three-dimensional position of the keypoint and a plurality of preset zones of the nose. In computer rendering techniques, rendering data is applied to render a three-dimensional image, the three-dimensional position of a keypoint is indicative of the position of the keypoint in the three-dimensional image, and generally, the representation of the three-dimensional position of the keypoint is in the form of three-dimensional coordinate values. A plurality of preset regions of the nose are pre-divided, and the method for acquiring rendering data and dividing the preset regions can be referred to in the prior art.
Taking the rendering image shown in fig. 2 as an example, the preset area of the nose portion respectively includes, from top to bottom: the upper region 201 of the nose (denoted as the first predetermined region C)up) A base area 202 of the nose (denoted as a second predetermined area C)base) A middle area 203 of the nose (marked as a third preset area C)mid) A point area 204 of the nose (marked as a fourth preset area C)tip) And a bottom region 205 of the nose (denoted as a fifth predetermined region C)bom)。
The outline of each preset region in the rendering data is composed of pixel points at the junctions of the preset region and other regions (other preset regions or regions at other positions in the rendering image). For example, the fourth preset area CtipThe outline of the first pixel point (the fourth preset area C)tipAnd a third predetermined area CmidPixel point at the boundary) and the second type of pixel point (the fourth preset area C)tipAnd a fifth predetermined area CbomThe pixel points at the boundary of (a).
And S102, responding to the fact that the projection positions of the key points are visible in the rendered image, and marking the projection positions of the key points as the positions of the key points in the rendered image.
In the present disclosure, the projection position is a position obtained by projecting the three-dimensional position of the key point to the two-dimensional image space. The projected positions of the keypoints are visible in the rendered image, and the projected positions of the keypoints are represented to be coincident with the positions of the keypoints in the rendered image. Therefore, the present disclosure labels the projected locations of the keypoints as the locations of the keypoints in the rendered image.
The method for determining whether the projection position of the key point is visible in the rendered image may refer to the prior art, for example, the method may be performed by using a depth map, and details are not repeated here.
FIG. 3 illustrates some of the key points of the nose to be labeled, including key points 63-74 and 97-100 to be labeled.
Taking the keypoint (hereinafter, referred to as P)63 as an example, suppose that the projection position of P63 is visible in the rendered image, i.e., the projection position of P63 coincides with the position of P63 in the rendered image, so the projection position of P63 is labeled as the position of P63 in the rendered image.
According to the method and the device, the positions of the key points in the rendered image can be automatically marked in response to the fact that the projection positions of the key points are visible in the rendered image, and when the projection positions of the key points are visible in the rendered image, the projection positions of the key points are shown to be overlapped with the positions of the key points in the rendered image, so that the projection positions of the key points are directly marked as the positions of the key points in the rendered image, the positions of the key points in the rendered image do not need to be obtained according to rendering data, and therefore the marking efficiency is improved on the premise that the accuracy of marking results is guaranteed.
S103, responding to the fact that the projection position of the key point is invisible in the rendered image, and obtaining the marking parameter.
In the present disclosure, the annotation parameter includes at least the location of the reference contour of the keypoint. The position of the reference contour may specifically be a position of a pixel point constituting the reference contour in the rendered image.
In the present disclosure, a preset region of the nose, which includes a preset region in which the probability of the key point is greater than the first threshold, is used as a preset region corresponding to the key point. And taking the contour of the preset area corresponding to the key point as a reference contour. For example, the preset regions where the key points are located in each rendered image are artificially counted to obtain statistical data, and then the probability that each preset region includes each key point is determined according to the statistical data.
And S104, marking the positions of the key points in the rendered image according to the marking parameters.
In the present disclosure, for different keypoints, a specific method for labeling the positions of the keypoints in the rendered image according to the labeling parameters of the keypoints may be referred to in the following embodiments.
According to the process, the annotation parameters are obtained in response to the fact that the projection positions of the key points are invisible in the rendered image, and the positions of the key points in the rendered image are annotated according to the annotation parameters. It can be seen that the present disclosure can automatically label the locations of keypoints in rendered images. Because the labeling parameters at least include the positions of the reference contours of the key points, the reference contours are the contours of the preset regions corresponding to the key points, and the preset regions corresponding to the key points are as follows: the preset region with the probability of the key point being greater than the first threshold value is included, and therefore the marking parameters are generated according to objective data and values, the influence of subjective factors on the marking result is avoided, a uniform marking result can be obtained for the same key point, and the error of the model obtained through training is favorably reduced. In conclusion, the technical scheme provided by the disclosure not only reduces the labor cost, but also avoids the problem of large model errors caused by non-uniform labeling of the same key point by different labeling personnel.
The above labeling method will be described in more detail below with respect to different key points, taking the flow shown in fig. 4 and 5 as an example respectively.
Fig. 4 illustrates a specific implementation flow of a selectable nasal key point labeling method, as shown in fig. 4, specifically including S401 to S403, as follows:
s401, obtaining a rendering image of the human face.
In the present disclosure, the rendered image is generated based on rendering data, the rendering data including: the three-dimensional position of the keypoint and a plurality of preset zones of the nose. See S101 for a specific implementation.
S402, responding to the fact that the projection position of the key point is invisible in the rendered image, and obtaining the position of the reference outline of the key point and the reference line as marking parameters.
In the present disclosure, the reference line is a straight line passing through the projected position of the key point and having a direction of the nasal axis or a direction perpendicular to the nasal axis.
In the present disclosure, the mathematical expression of the direction of the axis in the nose is determined according to the attitude angle of the human face in the rendered image, the attitude angle includes a pitch angle, a yaw angle, and a roll-over angle, and the rendered data generally includes the attitude angle, so the attitude angle can be obtained from the rendered data.
Specifically, the direction of the nasal axis is denoted by v ═ sin roll, and the perpendicular direction of the nasal axis is denoted by v ═ sin roll, where roll denotes the flip angle.
And S403, marking the position of the key point in the rendered image according to the intersection point of the reference line and the reference outline.
The specific implementation manner of S403 is different according to different key points, and will be further described in the following examples.
It can be seen from the above flow that, in this embodiment, the annotation parameter includes a reference line in addition to the position of the reference contour, and the position of the key point in the rendered image is annotated according to the intersection point of the reference line and the reference contour, because the annotation parameter increases the objective parameter of the reference line, not only can the accuracy of the annotation result be improved, but also different annotation requirements of the key point can be adapted, specifically, the accuracy requirement of some key points on the coordinate of the axis direction in the nose is higher, some key points need to be combined with the face posture to obtain a more accurate annotation result, and the introduction of the reference line can better satisfy the above requirements.
The flow shown in fig. 4 will be exemplified with respect to the key points of the above-mentioned different requirements.
1. The key points located on the axis of the middle of the nose, such as P97 and P100 in fig. 3. The accuracy requirement of the key points on the coordinate in the nasal central axis direction is higher.
Thus, for such key points, the direction of the reference line is chosen to be the direction of the axis in the nose.
According to the intersection point of the reference line and the reference contour, the specific implementation manner of labeling the position of the key point in the rendered image (i.e. S403) is as follows: and selecting one intersection point from a plurality of intersection points of the reference line and the reference contour as the position of the key point according to the position relation of the key point and the target line in the axial direction of the nose, wherein the target line is a straight line which is perpendicular to the axis of the nose and passes through the middle point of the axis of the nose.
Specifically, for P97: the reference contour is the first predetermined area C in FIG. 2upThe contour of (a), a second predetermined area CbaseThe contour of (1), the third predetermined area CmidAnd a fourth predetermined area CtipThe profile of (a). The reference line is a straight line passing through the projected position of P97 and oriented in the direction of the axis in the nose. Since P97 is located above the target line, the reference line of P97 is aligned with the first predetermined area CupThe contour of (a), a second predetermined area CbaseThe contour of (1), the third predetermined area CmidAnd a fourth predetermined area CtipThe uppermost (e.g., the largest or smallest coordinate in the axial direction in the nose) intersection of the respective intersections of the contour of (a) is denoted as the position of P97.
For P100, the reference profile is the second predetermined region C in FIG. 2baseThe contour of (1), the third predetermined area CmidContour of (1), fourth predetermined area CtipAnd a fifth predetermined area CbomThe profile of (a). The reference line is a straight line passing through the projected position of P100 and oriented in the direction of the axis in the nose. Since P100 is located below the target line, the reference line of P100 is aligned with the second predetermined area CbaseThe contour of (1), the third predetermined area CmidContour of (1), fourth predetermined area CtipAnd a fifth predetermined area CbomEach intersection of the contours of (1)The intersection point of the lowest end (e.g., the smallest or largest coordinate in the axial direction in the nose) in (b) is labeled as the position of P100.
It can be understood that the reference line is used as one of the labeling parameters, which is beneficial to more accurately positioning the coordinate in the direction of the reference line, and further, the labeling point is selected from a plurality of possible intersection points according to the position relationship between the key point and the target line, so that the accuracy of the labeling position can be further improved.
2. For key points located on the axis in the nose, the following can be used for labeling in addition to 1:
the direction of the reference line is selected from the direction of the axis in the nose.
According to the intersection point of the reference line and the reference contour, the specific implementation manner of labeling the position of the key point in the rendered image (i.e. S403) is as follows: and sequentially inquiring whether the reference line and the reference contour have intersection points or not according to the sequence of the reference contour of the key points from large to small until the intersection points are inquired. And selecting one intersection point from the intersection points according to the position relation of the key point and the target preset region in the axis direction in the nose, and marking the intersection point as the position of the key point in the rendered image, wherein the reference contour is obtained by arranging the reference contours from large to small according to the probability. The probability of any reference contour is the probability that the preset region with the reference contour comprises the key point. The target preset area is a preset area having a reference contour where the intersection point is found. Optionally, the position relationship between the key point and the target preset region in the nasal axis direction may be obtained by counting the position relationship between the key point and the target preset region in the rendered images of multiple faces in the nasal axis direction to obtain the most likely position relationship.
Taking P99 in fig. 2 as an example, the reference contour is the second predetermined area C in fig. 2baseContour of (1), fourth predetermined area CtipAnd a fifth predetermined area CbomThe profile of (a). The reference line is a straight line passing through the projected position of P99 and oriented in the direction of the axis in the nose. The reference contour of P99 is the fourth predetermined area C according to the sorting result of the probabilities from large to smalltipOutline of > fifth Preset area CbomOutline of > second Preset area CbaseOf (2) a. Then, specifically, the reference line and the fourth preset area C are queriedtipWhether there is an intersection of the contours of (1), if any, due to P99 and CtipThe position relation in the axial direction in the nose is that P99 is positioned in a fourth preset area CtipSo that the reference line and the fourth predetermined area C are selectedtipIs marked as the position of P99.
If the reference line is in the fourth predetermined area CtipDoes not have an intersection point, and inquires the reference line and the fifth preset area CbomWhether there is an intersection of the contours of (1), if any, due to P99 and CtipThe position relation in the axial direction in the nose is that P99 is positioned in a fifth preset area CbomSo that the reference line and the fifth predetermined area C are selectedbomIs marked as the position of P99.
If the reference line is the fifth predetermined area CbomDoes not have an intersection point, queries the reference line and the second preset area CbaseWhether there is an intersection of the contours of (1), if any, due to P99 and CbaseThe position relation in the axial direction in the nose is that P99 is positioned in a second preset area CbaseSo that the reference line and the second predetermined area C are selectedbaseIs marked as the position of P99.
It can be understood that the intersection point sequential query method can save computing resources, and further, according to the position relationship between the key point and the target preset region in the axial direction of the nose, the marking point is selected from a plurality of possible intersection points, so that the accuracy of the marking position can be further improved.
3. Key points with associated points that are symmetrical with respect to the axis in the nose. The applicant finds out through research that: the accuracy of the labeling result of the key points is related to the pose of the face in the rendered image, so that the pose of the face needs to be determined according to the key points and the associated points, and the direction of the reference line needs to be further determined, so as to ensure the accuracy of the labeling result.
Taking P74 in fig. 3 as an example, the association point of P74 is P63.
(1) In the case where neither P74 nor P63 is visible,the gesture of the human face is a large-angle head, in this case, the direction of the reference line is selected as the direction of the axis in the nose by referring to the manual labeling mode. The reference contour is a second predetermined area CbaseThe contour of (1), the third predetermined area CmidContour of (1), fourth predetermined area CtipAnd a fifth predetermined area CbornThe profile of (a).
According to the intersection point of the reference line and the reference contour, the specific implementation manner of labeling the position of the key point in the rendered image (i.e. S403) is as follows: and marking the position of the key point in the rendered image according to the coordinate of the intersection point of the reference line and the reference contour in the axial direction in the nose.
For P74, the reference line is a straight line passing through the projection position of P74 and the direction is the direction of the axis in the nose, and since neither P74 nor P63 can see the head of a person who is in a large-angle posture, the reference line of P74 and the second preset area C are combinedbaseThe contour of (1), the third predetermined area CmidContour of (1), fourth predetermined area CtipAnd a fifth predetermined area CbornThe uppermost (e.g., the largest or smallest coordinate in the axial direction in the nose) intersection of the respective intersections of the contour of (a) is denoted as the position of P74.
(2) When the P74 is invisible in the rendered image and the P63 is visible in the rendered image, the pose of the human face is the twisting of the head to the P74 side, in which case the direction of the reference line is selected to be the vertical direction of the axis in the nose with reference to the manual labeling mode.
According to the intersection point of the reference line and the reference contour, the specific implementation manner of labeling the position of the key point in the rendered image (i.e. S403) is as follows: and selecting one intersection point from a plurality of intersection points of the reference line and the reference contour as the position of the key point according to the position relation between the key point and the axis in the nose in the direction of the reference line.
For P74, the reference line is a straight line passing through the projected position of P74 and oriented in a direction perpendicular to the axis in the nose, and the reference contour is the first preset region CupThe contour of (a), a second predetermined area CbaseThe contour of (1), the third predetermined area CmidContour of (1), fourth predetermined area CtipWheel (D)Outline and fifth preset area CbornThe profile of (a). Since P74 is to the right of the nasal axis, the reference line and the first predetermined area C are selectedupThe contour of (a), a second predetermined area CbaseThe contour of (1), the third predetermined area CmidContour of (1), fourth predetermined area CtipAnd a fifth predetermined area CbornThe rightmost (e.g., the coordinate in the vertical direction of the axis in the nose is largest or smallest) intersection of the respective intersections of the contours of (a) is labeled as the position of P74.
From (1) and (2), it can be seen that the direction of the reference line is determined based on the visible or invisible states of the associated points and the key points which can reflect the human face posture, and the habit of manual labeling is fitted, so that the accuracy of the labeling result can be further improved. Furthermore, the intersection point is selected according to the position relation of the key point and the axis in the nose in the direction of the reference line, and the accuracy of the labeling result can be further improved from another angle.
Note that, for the keypoints that do not need to consider the pose, as in the above-described P97, P99, and P100, if there is no intersection between the reference line of the keypoint and the reference outline, the projection position of the keypoint is labeled as the position of the keypoint in the rendered image. For example, if the reference line of P99 and the fourth predetermined area CtipProfile of (2), fifth preset area CbornAnd a second predetermined area CbaseIf there is no intersection in the contours of (a), the projected position of P99 is labeled as the position of P99 in the rendered image. For the key points needing to consider the posture, such as P74, the reference lines of the key points and the reference outline have no intersection points, which shows that the angle of the face posture is too large, no label and no practical value exist, and the rendering graph is abandoned.
In addition to marking the locations of keypoints according to the reference line and the reference outline as shown in fig. 4, the locations of keypoints may be marked only according to the reference outline.
Fig. 5 illustrates a specific implementation flow of another optional method for labeling a nose key point, and as shown in fig. 5, the method specifically includes steps S501 to S506:
s501, obtaining a rendering image of the human face.
In this disclosure, the rendering image includes a key point to be labeled at the nose, the rendering image is generated based on rendering data, and the rendering data includes: the three-dimensional position of the keypoint and a plurality of preset zones of the nose.
S502, responding to the fact that the projection position of the key point is invisible in the rendered image, and obtaining the position of the reference outline of the key point as a marking parameter.
In this disclosure, the reference contour is a contour of a preset region corresponding to the key point, and the preset region corresponding to the key point is: the preset region including the keypoint has a probability greater than the first threshold, and it should be noted that the method for obtaining the position of the reference contour of the keypoint is described in S103 above.
And S503, converting the flip angle of the rendering image into zero to obtain a conversion image of the rendering image.
It should be noted that, the specific method for obtaining the converted image of the rendered image can be referred to in the prior art.
And S504, selecting a reference contour corresponding to the pitch angle and the yaw angle of the converted image from the reference contours as a target reference contour.
In the present disclosure, the reference profile corresponding to the pitch angle and the yaw angle of the converted image is a profile of a preset region in which the probability of including the key point is greater than the first threshold value under the pitch angle and the yaw angle.
It should be noted that the converted image is obtained by converting the flip angle of the rendered image into zero, that is, the face in the converted image is the front face, the method for obtaining the target reference contour may refer to S103, and the rendered image may be replaced by the converted image.
And S505, selecting the position of one pixel point from the target reference contour as the target position of the key point according to the position relation between the key point and the axis in the nose in the target direction.
In the present disclosure, the target direction is the perpendicular direction of the axis in the nose. The key points are positioned on two sides of the nasal central axis, and the position relation between the key points and the nasal central axis in the target direction comprises: the keypoint is on a first side of the nasal medial axis or the keypoint is on a second side of the nasal medial axis. For example, the first side is designated as the left side and the second side is designated as the right side.
The above steps are exemplified by P66 and P71 in fig. 3.
Assume that the projected position of P66 is not visible in the rendered image, and that P66 is located to the left of the nasal axis. The method for selecting the position of one pixel point as the target position of P66 comprises the following steps of A1-A2:
a1, if the pitch angle of the converted image is less than-51 and the yaw angle of the converted image is greater than 53, the target reference profile of P66 is the fourth preset area CtipThe profile of (a). Since P66 is located to the left of the nasal axis, the fourth preset area C will betipThe position of the leftmost pixel point on the outline of (1) is taken as the target position of P66.
A2, if the pitch angle of the converted image is not less than-51 or the yaw angle of the converted image is not more than 53, the target reference profile of P66 is the fourth preset area CtipAnd a fifth predetermined area CbomThe profile of (a). A fourth preset area CtipAnd a fifth predetermined area CbomThe position of the leftmost pixel point on the outline of (1) is taken as the target position of P66.
It should be noted that, the position of the leftmost pixel point on the target reference contour is obtained according to the coordinates of the pixel point on the target reference contour.
Assume that the projected position of P71 is not visible in the rendered image, and that P71 is located to the right of the nasal axis. The method for selecting the position of one pixel point as the target position of P71 comprises the following steps of B1-B2:
b1, if the pitch angle of the converted image is larger than 51 and the yaw angle of the converted image is larger than 53, the target reference profile of the P71 is the fourth preset area CtipThe profile of (a). Since P71 is located to the right of the nasal axis, the fourth preset area C will betipThe position of the rightmost pixel point on the outline of (1) is set as the target position of P71.
B2, if the pitch angle of the converted image is not less than-51 or the yaw angle of the converted image is not greater than 53, then the target reference profile of P71 includes a fourth preset area CtipThe contour and the fifth preSetting region CbomThe profile of (a). A fourth preset area CtipAnd a fifth predetermined area CbomThe position of the rightmost pixel point on the outline of (1) is set as the target position of P71.
S506, converting the target position to a position in the rendering image to obtain the position of the key point in the rendering image.
It should be noted that the target position is a position of the key point in the converted image, and a method for converting the target position to a position in the rendered image may be referred to in the prior art.
It can be seen from the flow shown in fig. 5 that, by using the transformation of the reference contour and the attitude angle parameter in combination to label the key points, another automatic labeling manner is provided besides the reference line and the reference contour, which not only can save labor cost, but also has higher accuracy because the attitude angle is also an objective numerical value, and can avoid the problem that the error of the trained model is larger due to subjective influence.
In this way, when the labeling positions of some key points are obtained, the labeling positions of other key points can be determined according to the obtained labeling positions.
Specifically, the method comprises the following steps: and acquiring a contour line between a first key point and a second key point in the marked key points, and respectively marking the positions of N equal division points on the contour line as the positions of other key points.
Specifically, N is an integer greater than 1, a value of N may be preset according to an actual requirement, and the first and second key points are key points that are located on the contour line of the nose and have a curvature greater than a second threshold value, among the labeled key points, that is, inflection points on the contour line of the nose. Such as P63, P66, P71, P74, or P100, as noted in the labeling methods above.
Taking N as an example, the method for labeling the positions of other key points, i.e. trisection points, includes:
the first keypoint is P66 and the second keypoint is P100, and of the trisections of the contour line between P66 and P100, the position close to P66 is denoted as the position of P67 in the rendered image and the position close to P100 is denoted as the position of P68 in the rendered image.
The first keypoint is P63 and the second keypoint is P66, with the trisection point of the contour line between P63 and P66, the position close to P63 being denoted as the position of P64 in the rendered image and the position close to P66 being denoted as the position of P65 in the rendered image.
The first keypoint is P71 and the second keypoint is P100, and of the trisections of the contour line between P71 and P100, the position close to P71 is denoted as the position of P70 in the rendered image, and the position close to P100 is denoted as the position of P69 in the rendered image.
The first keypoint is P71 and the second keypoint is P74, with the trisection point of the contour line between P71 and P74, the position near P71 being denoted as the position of P72 in the rendered image and the position near P74 being denoted as the position of P73 in the rendered image.
It can be understood that, for the first keypoint and the second keypoint marked on the contour line of the nose region, the positions of other keypoints on the contour line of the nose region in the rendered image can be obtained by equally dividing the contour line N between the first keypoint and the second keypoint, and obviously, the method for obtaining the marked positions by equally dividing the contour line is highly efficient and simple in calculation.
Alternatively, for key points located on the axis in the nose, other key points may be labeled using the labeled key points. Specifically, the method comprises the following steps: and acquiring a connecting line between a first key point and a second key point in the marked key points, and marking the positions of N equally divided points on the connecting line as the positions of other key points respectively.
Specifically, N is an integer greater than 1, and the first and second keypoints are keypoints located on the nasal medial axis. Such as P97 and P99, noted by the labeling methods described above.
Taking N-3 as an example, the method for labeling the positions of other key points includes:
the first keypoint is P97 and the second keypoint is P99, and the position near P99 in the trisection of the connecting line of P97 and P99 is labeled as the position of P98 in the rendered image.
It can be understood that, for the first and second key points marked on the nasal central axis, the positions of other key points on the nasal central axis in the rendered image can be obtained by equally dividing the connecting line N between the first and second key points.
FIG. 6 is a block diagram illustrating a nasal keypoint labeling apparatus, according to an exemplary embodiment. Referring to fig. 6, the apparatus includes:
a rendering image obtaining module 601, configured to obtain a rendering image of a face, where the rendering image includes a key point to be labeled on a nose, the rendering image is generated based on rendering data, and the rendering data includes: the three-dimensional position of the key point and a plurality of preset regions of the nose;
a labeling parameter obtaining module 602, configured to obtain a labeling parameter in response to that a projection position of the key point is not visible in the rendered image, where the labeling parameter at least includes a position of a reference contour of the key point, the reference contour is a contour of the preset area corresponding to the key point, and the preset area corresponding to the key point is: the preset region with the probability of the key point being greater than a first threshold value is included, and the projection position is a position obtained by projecting the three-dimensional position of the key point to a two-dimensional image space;
and the labeling module 603 is configured to label the position of the key point in the rendered image according to the labeling parameter.
Optionally, the labeling parameters further include: the reference line is a straight line which passes through the projection position of the key point and has the direction of the axis in the nose or the direction vertical to the axis in the nose;
optionally, the labeling module is configured to label, according to the labeling parameter, a position of the keypoint in the rendered image, and includes: the labeling module is specifically configured to: and marking the position of the key point in the rendered image according to the intersection point of the reference line and the reference outline.
Optionally, a key point is located on the nasal medial axis, and the direction of the reference line of the key point is the direction of the nasal medial axis;
optionally, the labeling module is configured to label, according to an intersection of the reference line and the reference contour, a position of the keypoint in the rendered image, and includes: the labeling module is specifically configured to: selecting one intersection point from the intersection points of the reference line and the reference contour as the position of the key point according to the position relation of the key point and a target line in the axial direction of the nose; the target line is a straight line perpendicular to the nasal medial axis and passing through a midpoint of the nasal medial axis.
Optionally, the key point is located on the nasal axis; the direction of the reference line of the key point is the direction of the axis in the nose;
optionally, the labeling module is configured to label, according to an intersection of the reference line and the reference contour, a position of the keypoint in the rendered image, and includes: the labeling module is specifically configured to: sequentially inquiring whether the reference line and the reference contour have the intersection point or not according to the sequence of the reference contour of the key point from large to small according to the probability until the intersection point is inquired; the sequence of the reference outline is obtained by arranging the reference outlines from large to small according to the probability; and selecting one intersection point from the intersection points according to the position relation of the key point and a target preset area in the axial direction of the nose, wherein the intersection point is marked as the position of the key point in the rendered image, and the target preset area is the preset area with the reference contour of the inquired intersection point.
Optionally, there is a preconfigured associated point for a keypoint, the associated point being symmetric about the nasal axis to the keypoint and not visible in the rendered image;
the direction of the reference line of the key point is the direction of the axis in the nose;
optionally, the labeling module is configured to label, according to an intersection of the reference line and the reference contour, a position of the keypoint in the rendered image, and includes: the labeling module is specifically configured to: and marking the position of the key point in the rendered image according to the coordinate of the intersection point of the reference line and the reference contour in the axial direction in the nose.
Optionally, there is a preconfigured associated point for the keypoints, the associated point being symmetric about the nasal axis with the keypoints and visible in the rendered image;
the direction of the reference line of the key point is the vertical direction of the axis in the nose;
optionally, the labeling module is configured to label, according to an intersection of the reference line and the reference contour, a position of the keypoint in the rendered image, and includes: the labeling module is specifically configured to: and selecting one intersection point from the plurality of intersection points of the reference line and the reference contour as the position of the key point according to the position relation of the key point and the axis in the nose in the direction of the reference line.
Optionally, the labeling device for nasal key points further includes: and the transformed image acquisition module is used for converting the turning angle of the rendered image into zero to obtain a transformed image of the rendered image before the positions of the key points in the rendered image are marked according to the marking parameters.
Optionally, the labeling module is configured to label, according to the labeling parameter, a position of the keypoint in the rendered image, and includes: the labeling module is specifically configured to: selecting a reference contour corresponding to a pitch angle and a yaw angle of the converted image from the reference contours as a target reference contour; the reference contour corresponding to the pitch angle and the yaw angle of the converted image is a contour of the preset area, wherein the contour comprises the key point with the probability larger than the first threshold value under the pitch angle and the yaw angle;
selecting the position of a pixel point from the target reference contour as the target position of the key point according to the position relation between the key point and the axis in the nose in the target direction; the target direction is a perpendicular direction to the nasal axis;
and converting the target position to a position in the rendering image to obtain the position of the key point in the rendering image.
Optionally, the annotation module is further configured to: in response to the projected locations of the keypoints being visible in the rendered image, labeling the projected locations of the keypoints as locations of the keypoints in the rendered image.
Optionally, the annotation module is further configured to:
after the positions of the key points in the rendered image are marked according to the marking parameters, obtaining a contour line between a first key point and a second key point in the marked key points; the first key point and the second key point are the key points which are positioned on the contour line of the nose and have the curvature larger than a second threshold value in the marked key points;
and respectively marking the positions of N equal division points on the contour line as the positions of other key points, wherein N is an integer greater than 1.
Optionally, the annotation module is further configured to:
after the positions of the key points in the rendered image are labeled according to the labeling parameters, acquiring a connecting line between a first key point and a second key point in the labeled key points, wherein the first key point and the second key point are the key points on a nasal medial axis;
and respectively marking the positions of N equal division points on the connecting line as the positions of other key points, wherein N is an integer greater than 1.
With regard to the labeling device of the nose key points in the above embodiments, the specific manner in which each module performs the operation has been described in detail in the embodiments related to the method, and will not be elaborated here.
Fig. 7 is a block diagram illustrating a hardware structure of an electronic device, and referring to fig. 7, the hardware structure of the electronic device may include: a processor 1 and a memory 2 storing instructions executable by the processor, optionally a communication interface 3, and a communication bus 4.
The processor is configured to execute the executable instructions to implement the labeling method of the nose key points described in the above embodiments. Optionally, the refinement and extension functions of the instructions may be as described above.
Specifically, in the embodiment of the present application, the number of the processor 1, the memory 2, the communication interface 3, and the communication bus 4 is at least one, and the processor 1, the memory 2, and the communication interface 3 complete mutual communication through the communication bus 4;
the processor 1 may be a central processing unit CPU, or an application Specific Integrated circuit asic, or one or more Integrated circuits configured to implement embodiments of the present invention, etc.;
the memory 2 may include a high-speed RAM memory, and may further include a non-volatile memory (non-volatile memory) or the like, such as at least one disk memory;
in an exemplary embodiment, there is also provided a storage medium comprising instructions, such as a memory 2 comprising instructions, executable by a processor 1 of an electronic device to perform the above method. Alternatively, the storage medium may be a non-transitory computer readable storage medium, which may be, for example, a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (10)

1. A labeling method for key points of a nose is characterized by comprising the following steps:
obtaining a rendering image of a face, wherein the rendering image comprises key points to be marked on a nose, the rendering image is generated based on rendering data, and the rendering data comprises three-dimensional positions of the key points and a plurality of preset areas of the nose;
responding to that the projection position of the key point is invisible in the rendered image, acquiring a labeling parameter, wherein the labeling parameter at least comprises the position of a reference outline of the key point, the reference outline is the outline of the preset area corresponding to the key point, the preset area corresponding to the key point is the preset area with the probability of the key point being greater than a first threshold value, and the projection position is the position obtained by projecting the three-dimensional position of the key point to a two-dimensional image space;
and marking the positions of the key points in the rendered image according to the marking parameters.
2. The method of claim 1, wherein the labeling parameters further comprise: a reference line which passes through the projection position and is a straight line with the direction of the nasal axis or the direction perpendicular to the nasal axis;
the marking the position of the key point in the rendered image according to the marking parameter comprises:
and marking the position of the key point in the rendered image according to the intersection point of the reference line and the reference outline.
3. The method of claim 2, wherein the keypoint is located on the nasal middle axis and the direction of the reference line of the keypoint is the direction of the nasal middle axis;
the marking the position of the key point in the rendered image according to the intersection point of the reference line and the reference contour comprises:
selecting one intersection point from the intersection points of the reference line and the reference contour as the position of the key point according to the position relation of the key point and a target line in the axial direction of the nose; the target line is a straight line perpendicular to the nasal medial axis and passing through a midpoint of the nasal medial axis.
4. The method of claim 2, wherein the keypoint is located on the nasal middle axis and the direction of the reference line of the keypoint is the direction of the nasal middle axis;
the marking the position of the key point in the rendered image according to the intersection point of the reference line and the reference contour comprises:
sequentially inquiring whether the reference line and the reference contour have the intersection points or not according to the sequence of the reference contour of the key points until the intersection points are inquired; the sequence of the reference outline is obtained by arranging the reference outlines from large to small according to the probability;
and selecting one intersection point from the intersection points according to the position relation of the key point and a target preset area in the axial direction of the nose, wherein the intersection point is marked as the position of the key point in the rendered image, and the target preset area is the preset area with the reference contour of the inquired intersection point.
5. The method of claim 2, wherein there is a preconfigured associated point for the keypoints, the associated point being symmetric about the nasal axis with the keypoint and not visible in the rendered image;
the direction of the reference line of the key point is the direction of the axis in the nose;
the marking the position of the key point in the rendered image according to the intersection point of the reference line and the reference contour comprises:
and marking the position of the key point in the rendered image according to the coordinate of the intersection point of the reference line and the reference contour in the axial direction in the nose.
6. The method of claim 2, wherein there is a preconfigured associated point for the keypoints, the associated point being symmetric about the nasal axis with the keypoint and visible in the rendered image;
the direction of the reference line of the key point is the vertical direction of the axis in the nose;
the marking the position of the key point in the rendered image according to the intersection point of the reference line and the reference contour comprises:
and selecting one intersection point from the plurality of intersection points of the reference line and the reference contour as the position of the key point according to the position relation of the key point and the axis in the nose in the direction of the reference line.
7. The method of claim 1, further comprising, prior to said labeling the location of the keypoint in the rendered image in accordance with the labeling parameter:
obtaining a conversion image of the rendering image by converting the flip angle of the rendering image into zero;
the marking the position of the key point in the rendered image according to the marking parameter comprises:
selecting a reference contour corresponding to a pitch angle and a yaw angle of the converted image from the reference contours as a target reference contour; the reference contour corresponding to the pitch angle and the yaw angle of the converted image is a contour of the preset area, wherein the contour comprises the key point with the probability larger than the first threshold value under the pitch angle and the yaw angle;
selecting the position of a pixel point from the target reference contour as the target position of the key point according to the position relation between the key point and the axis in the nose in the target direction; the target direction is a perpendicular direction to the nasal axis;
and converting the target position to a position in the rendering image to obtain the position of the key point in the rendering image.
8. A nasal keypoint labeling device, comprising:
the rendering image obtaining module is used for obtaining a rendering image of a face, the rendering image comprises key points to be marked on a nose, the rendering image is generated based on rendering data, and the rendering data comprises three-dimensional positions of the key points and a plurality of preset areas of the nose;
a labeling parameter obtaining module, configured to obtain a labeling parameter in response to that a projection position of the key point is invisible in the rendered image, where the labeling parameter at least includes a position of a reference contour of the key point, the reference contour is a contour of the preset region corresponding to the key point, the preset region corresponding to the key point is the preset region where a probability of including the key point is greater than a first threshold, and the projection position is a position obtained by projecting a three-dimensional position of the key point to a two-dimensional image space;
and the marking module is used for marking the position of the key point in the rendered image according to the marking parameters.
9. An electronic device, comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the executable instructions to implement the nasal keypoint labeling method of any of claims 1-7.
10. A storage medium in which instructions, when executed by a processor of an electronic device, enable the electronic device to perform a method of labeling nasal keypoints as claimed in any one of claims 1 to 7.
CN202011100012.3A 2020-10-14 2020-10-14 Labeling method and device for key points of nose, electronic equipment and storage medium Pending CN112163552A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011100012.3A CN112163552A (en) 2020-10-14 2020-10-14 Labeling method and device for key points of nose, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011100012.3A CN112163552A (en) 2020-10-14 2020-10-14 Labeling method and device for key points of nose, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN112163552A true CN112163552A (en) 2021-01-01

Family

ID=73867003

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011100012.3A Pending CN112163552A (en) 2020-10-14 2020-10-14 Labeling method and device for key points of nose, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN112163552A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113657307A (en) * 2021-08-20 2021-11-16 北京市商汤科技开发有限公司 Data labeling method and device, computer equipment and storage medium

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107451950A (en) * 2016-05-30 2017-12-08 北京旷视科技有限公司 Face image synthesis method, human face recognition model training method and related device
CN109034131A (en) * 2018-09-03 2018-12-18 福州海景科技开发有限公司 A kind of semi-automatic face key point mask method and storage medium
CN109063678A (en) * 2018-08-24 2018-12-21 北京字节跳动网络技术有限公司 The method, apparatus and storage medium of face image identification
US20190318152A1 (en) * 2018-04-12 2019-10-17 Omron Corporation Image analysis apparatus, method, and program
CN110348344A (en) * 2019-06-28 2019-10-18 浙江大学 A method of the special facial expression recognition based on two and three dimensions fusion
CN111652086A (en) * 2020-05-15 2020-09-11 汉王科技股份有限公司 Face living body detection method and device, electronic equipment and storage medium
CN111696172A (en) * 2019-03-12 2020-09-22 北京京东尚科信息技术有限公司 Image labeling method, device, equipment and storage medium

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107451950A (en) * 2016-05-30 2017-12-08 北京旷视科技有限公司 Face image synthesis method, human face recognition model training method and related device
US20190318152A1 (en) * 2018-04-12 2019-10-17 Omron Corporation Image analysis apparatus, method, and program
CN110378182A (en) * 2018-04-12 2019-10-25 欧姆龙株式会社 Image analysis apparatus, method for analyzing image and recording medium
CN109063678A (en) * 2018-08-24 2018-12-21 北京字节跳动网络技术有限公司 The method, apparatus and storage medium of face image identification
CN109034131A (en) * 2018-09-03 2018-12-18 福州海景科技开发有限公司 A kind of semi-automatic face key point mask method and storage medium
CN111696172A (en) * 2019-03-12 2020-09-22 北京京东尚科信息技术有限公司 Image labeling method, device, equipment and storage medium
CN110348344A (en) * 2019-06-28 2019-10-18 浙江大学 A method of the special facial expression recognition based on two and three dimensions fusion
CN111652086A (en) * 2020-05-15 2020-09-11 汉王科技股份有限公司 Face living body detection method and device, electronic equipment and storage medium

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
KYUNG-MIN LEE: ""Three-dimensional prediction of the nose for facial reconstruction using cone-beam computed tomography"", 《FORENSIC SCIENCE INTERNATIONAL》, pages 1 - 5 *
鞠磊: ""稠密人脸关键点：数据集和标注工具"", 《中国优秀硕士学位论文全文数据库信息科技辑》, pages 29 - 43 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113657307A (en) * 2021-08-20 2021-11-16 北京市商汤科技开发有限公司 Data labeling method and device, computer equipment and storage medium

Similar Documents

Publication Publication Date Title
CN108734120B (en) Method, device and equipment for labeling image and computer readable storage medium
CN110414477B (en) Image scale detection method and device
CN109145904A (en) A kind of character identifying method and device
CN111652301B (en) Femoral lesser trochanter identification method and device based on deep learning and electronic equipment
CN109657675B (en) Image annotation method and device, computer equipment and readable storage medium
CN111340054A (en) Data labeling method and device and data processing equipment
WO2021189827A1 (en) Method and apparatus for recognizing blurred image, and device and computer-readable storage medium
CN110796135A (en) Target positioning method and device, computer equipment and computer storage medium
CN111208998B (en) Method, device and storage medium for automatic layout of data visualization large screen
CN105139007A (en) Positioning method and apparatus of face feature point
CN109871743B (en) Text data positioning method and device, storage medium and terminal
CN111159451B (en) Power line point cloud dynamic monomer method based on spatial database
CN112163552A (en) Labeling method and device for key points of nose, electronic equipment and storage medium
CN111932552A (en) Aorta modeling method and device
TW201317815A (en) System and method for processing information of measurement numbers
CN117237681A (en) Image processing method, device and related equipment
CN112241749A (en) Character recognition model training method, device and equipment
CN111144466A (en) Image sample self-adaptive depth measurement learning method
CN112396057A (en) Character recognition method and device and electronic equipment
CN115280374A (en) Labeling method and device
CN114792295B (en) Method, device, equipment and medium for correcting blocked object based on intelligent photo frame
CN110443251A (en) Instrument image recognition methods and device
CN112529038B (en) Method and device for identifying main board material and storage medium
CN117764647B (en) Automatic advertisement management method, system, equipment and storage medium
CN112597328B (en) Labeling method, device, equipment and medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination